#  QACheck: A Demonstration System
for Question-Guided Multi-Hop Fact-Checking

Liangming Pan1,2 Xinyuan Lu3 Min-Yen Kan3 Preslav Nakov1
1MBZUAI 2University of California, Santa Barbara 3 National University of
Singapore

<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Fact-checking real-world claims often requires complex, multi-step reasoning
due to the absence of direct evidence to support or refute them. However,
existing fact-checking systems often lack transparency in their decision-
making, making it challenging for users to comprehend their reasoning process.
To address this, we propose the Question-guided Multi-hop Fact-Checking
(QACheck) system, which guides the model’s reasoning process by asking a
series of questions critical for verifying a claim. QACheck has five key
modules: a claim verifier, a question generator, a question-answering module,
a QA validator, and a reasoner. Users can input a claim into QACheck, which
then predicts its veracity and provides a comprehensive report detailing its
reasoning process, guided by a sequence of (question, answer) pairs. QACheck
111QACheck is public available at https://github.com/XinyuanLu00/QACheck. A
recorded video is at https://www.youtube.com/watch?v=ju8kxSldM64 also provides
the source of evidence supporting each question, fostering a transparent,
explainable, and user-friendly fact-checking process.

##  1 Introduction

In our age characterized by large amounts of both true and false information,
fact-checking is not only crucial for counteracting misinformation but also
plays a vital role in fostering trust in AI systems. However, the process of
validating real-world claims is rarely straightforward. Unlike the simplicity
of supporting or refuting a claim with a single piece of direct evidence,
real-world claims often resemble multi-layered puzzles that require complex
and multi-step reasoning to solve Jiang et al. (2020); Nguyen et al. (2020);
Aly and Vlachos (2022); Chen et al. (2022); Pan et al. (2023).

Figure 1: An example of question-guided reasoning for fact-checking complex
real-world claims.

As an example, to verify the claim “Sunlight can reach the deepest part of the
Black Sea.”, it may be challenging to find direct evidence on the web that
refutes or supports this claim. Instead, a human fact-checker needs to
decompose the claim, gather multiple pieces of evidence, and perform step-by-
step reasoning Pan et al. (2023). This reasoning process can be formulated as
question-guided reasoning, where the verification of the claim is guided by
asking and answering a series of relevant questions, as shown in Figure 1. In
this example, we sequentially raise two questions: “What is the greatest depth
of the Black Sea?” and “How far can sunlight penetrate water?”. After
independently answering these two questions by gathering relevant information
from the Web, we can assert that the initial claim is false with simple
reasoning.

While several models Liu et al. (2020); Zhong et al. (2020); Aly and Vlachos
(2022) have been proposed to facilitate multi-step reasoning in fact-checking,
they generally lack transparency in their reasoning processes. These models
simply take a claim as input, then output a veracity label without an explicit
explanation. Recent attempts, such as Quin+ Samarinas et al. (2021) and
WhatTheWikiFact Chernyavskiy et al. (2021), have aimed to develop more
explainable fact-checking systems, by searching and visualizing the supporting
evidence for a given claim. However, these systems primarily validate a claim
from a single document, and do not provide a detailed, step-by-step
visualization of the reasoning process as shown in Figure 1.

We introduce the Question-guided Multi-hop Fact-Checking (QACheck) system,
which addresses the aforementioned issues by generating multi-step
explanations via question-guided reasoning. To facilitate an explainable
reasoning process, QACheck manages the reasoning process by guiding the model
to self-generate a series of questions vital for claim verification. Our
system, as depicted in Figure 2, is composed of five modules: 1) a claim
verifier that assesses whether sufficient information has been gathered to
verify the claim, 2) a question generator to generate the next relevant
question, 3) a question-answering module to answer the raised question, 4) a
QA validator to evaluate the usefulness of the generated (Q, A) pair, and 5) a
reasoner to output the final veracity label based on all collected contexts.

QACheck offers enough adaptability, allowing users to customize the design of
each module by integrating with different models. For example, we provide
three alternative implementations for the QA component: the retriever–reader
model, the FLAN-T5 model, and the GPT3-based reciter–reader model.
Furthermore, we offer a user-friendly interface for users to fact-check any
input claim and visualize its detailed question-guided reasoning process. The
screenshot of our user interface is shown in Figure 4. We will discuss the
implementation details of the system modules in Section 3 and some evaluation
results in Section 4. Finally, we present the details of the user interface in
Section 5. and conclude and discuss future work in Section 6.

##  2 Related Work

#### Fact-Checking Systems.

The recent surge in automated fact-checking research aims to mitigate the
spread of misinformation. Various fact-checking systems, for example,
Tanbih222https://www.tanbih.org/about Zhang et al. (2019),
PRTA333https://propaganda.qcri.org/ Martino et al. (2020), and
WhatTheWikiFact444https://www.tanbih.org/whatthewikifact Chernyavskiy et al.
(2021) predominantly originating from Wikipedia and claims within political or
scientific domains, have facilitated this endeavor. However, the majority of
these systems limit the validation or refutation of a claim to a single
document, indicating a gap in systems for multi-step reasoning Pan et al.
(2023). The system most similar to ours is Quin+ Samarinas et al. (2021),
which demonstrates evidence retrieval in a single step. In contrast, our
QACheck shows a question-led multi-step reasoning process with explanations
and retrieved evidence for each reasoning step. In summary, our system 1)
supports fact-checking real-world claims that require multi-step reasoning,
and 2) enhances transparency and helps users have a clear understanding of the
reasoning process.

#### Explanation Generation.

Simply predicting a veracity label to the claim is not persuasive, and can
even enhance mistaken beliefs Guo et al. (2022). Hence, it is necessary for
automated fact-checking methods to provide explanations to support model
predictions. Traditional approaches have utilized attention weights, logic, or
summary generation to provide post-hoc explanations for model predictions Lu
and Li (2020); Ahmadi et al. (2019); Kotonya and Toni (2020); Jolly et al.
(2022); Xing et al. (2022). In contrast, our approach employs question–answer
pair based explanations, offering more human-like and natural explanations.

Figure 2: The architecture of our QACheck system.

##  3 System Architecture

Figure 2 shows the general architecture of our system, comprised of five
principal modules: a Claim Verifier $\mathcal{D}$, a Question Generator
$\mathcal{Q}$, a Question-Answering Model $\mathcal{A}$, a Validator
$\mathcal{V}$, and a Reasoner $\mathcal{R}$. We first initialize an empty
context $\mathcal{C}=\emptyset$. Upon the receipt of a new input claim $c$,
the system first utilizes the claim verifier to determine the sufficiency of
the existing context to validate the claim, i.e.,
$\mathcal{D}(c,\mathcal{C})\rightarrow\\{\texttt{True},\texttt{False}\\}$. If
the output is False, the question generator learns to generate the next
question that is necessary for verifying the claim, i.e.,
$\mathcal{Q}(c,\mathcal{C})\rightarrow q$. The question-answering model is
then applied to answer the question and provide the supported evidence, i.e.,
$\mathcal{A}(q)\rightarrow a,e$, where $a$ is the predicted answer, and $e$ is
the retrieved evidence that supports the answer. Afterward, the validator is
used to validate the usefulness of the newly-generated (Q, A) pair based on
the existing context and the claim, i.e.,
$\mathcal{V}(c,\\{q,a\\},\mathcal{C})\rightarrow\\{\texttt{True},\texttt{False}\\}$.
If the output is True, the $(q,a)$ pair is added into the context $C$.
Otherwise, the question generator is asked to generate another question. We
repeat this process of calling
$\mathcal{D}\rightarrow\mathcal{Q}\rightarrow\mathcal{A}\rightarrow\mathcal{V}$
until the claim verifier returns a True indicating that the current context
$C$ contains sufficient information to verify the claim $c$. In this case, the
reasoner module is called to utilize the stored relevant context to justify
the veracity of the claim and outputs the final label, i.e.,
$\mathcal{R}(c,\mathcal{C})\rightarrow\\{\texttt{Supported},\texttt{Refuted}\\}$.
The subsequent sections provide a comprehensive description of the five key
modules in QACheck.

###  3.1 Claim Verifier

The claim verifier is a central component of QACheck, with the specific role
of determining if the current context information is sufficient to verify the
claim. This module is to ensure that the system can efficiently complete the
claim verification process without redundant reasoning. We build the claim
verifier based on InstructGPT Ouyang et al. (2022), utilizing its powerful in-
context learning ability. Recent large language models such as InstructGPT
Ouyang et al. (2022) and GPT-4 OpenAI (2023) have demonstrated strong few-shot
generalization ability via in-context learning, in which the model can
efficiently learn a task when prompted with the instruction of the task
together with a small number of demonstrations. We take advantage of
InstructGPT’s in-context learning ability to implement the claim verifier. We
prompt InstructGPT with ten distinct in-context examples as detailed in
Appendix A.1, where each example consists of a claim and relevant
question–answer pairs. We then prompt the model with the claim, the context,
and the following instruction:

> Claim = CLAIM
> We already know the following:
> CONTEXT
> Can we know whether the claim is true or false now? Yes or no?

If the response is ‘no’, we proceed to the question generator module.
Conversely, if the response is ‘yes’, the process jumps to call the reasoner
module.

###  3.2 Question Generator

The question generator module is called when the initial claim lacks the
necessary context for verification. This module aims to generate the next
relevant question needed for verifying the claim. Similar to the claim
verifier, we also leverage InstructGPT for in-context learning. We use
slightly different prompts for generating the initial question and the follow-
up questions. The detailed prompts are in Appendix A.2. For the initial
question generation, the instruction is:

> Claim = CLAIM
> To verify the above claim, we can first ask a simple question:

For follow-up questions, the instruction is:

> Claim = CLAIM
> We already know the following:
> CONTEXT
> To verify the claim, what is the next question we need to know the answer
> to?

###  3.3 Question Answering Model

After generating a question, the Question Answering (QA) module retrieves
corresponding evidence and provides an answer as the output. The system’s
reliability largely depends on the accuracy of the QA module’s responses.
Understanding the need for different QA methods in various fact-checking
scenarios, we introduce three different implementations for the QA module, as
shown in Figure 3.

Figure 3: Illustrations of the three different implementations of the Question
Answering module in QACheck.

#### Retriever–Reader.

We first integrate the well-known retriever–reader framework, a prevalent QA
paradigm originally introduced by Chen et al. (2017). In this framework, a
retriever first retrieves relevant documents from a large evidence corpus, and
then a reader predicts an answer conditioned on the retrieved documents. For
the evidence corpus, we use the Wikipedia dump provided by the Knowledge-
Intensive Language Tasks (KILT) benchmark Petroni et al. (2021), in which the
Wikipedia articles have been pre-processed and separated into paragraphs. For
the retriever, we apply the widely-used sparse retrieval based on BM25
Robertson and Zaragoza (2009), implemented with the Pyserini toolkit Lin et
al. (2021). For the reader, we use the RoBERTa-large Liu et al. (2019) model
fine-tuned on the SQuAD dataset Rajpurkar et al. (2016), using the
implementation from PrimeQA555https://github.com/primeqa/primeqa Sil et al.
(2023).

#### FLAN-T5.

While effective, the retriever–reader framework is constrained by its reliance
on the evidence corpus. In scenarios where a user’s claim is outside the scope
of Wikipedia, the system might fail to produce a credible response. To enhance
flexibility, we also incorporate the FLAN-T5 model Chung et al. (2022), a
Seq2Seq model pretrained on more than 1.8K tasks with instruction tuning. It
directly takes the question as input and then generates the answer and the
evidence, based on the model’s parametric knowledge.

#### GPT Reciter–Reader.

Recent studies Sun et al. (2023); Yu et al. (2023) have demonstrated the great
potential of the GPT series, such as InstructGPT Ouyang et al. (2022) and
GPT-4 OpenAI (2023), to function as robust knowledge repositories. The
knowledge can be retrieved by properly prompting the model. Drawing from this
insight, we introduce the GPT Reciter–Reader approach. Given a question, we
prompt the InstructGPT to “recite” the knowledge stored within it, and
InstructGPT responds with relevant evidence. The evidence is then fed into a
reader model to produce the corresponding answer. While this method, like
FLAN-T5, does not rely on a specific corpus, it stands out by using
InstructGPT. This offers a more dependable parametric knowledge base than
FLAN-T5.

The above three methods provide a flexible and robust QA module, allowing for
switching between the methods as required, depending on the claim being
verified and the available contextual information. In the following, we use
GPT Reciter–Reader as the default implementation for our QA module.

Figure 4: The screenshot of the QACheck user interface showing its key
annotated functions. First, users have the option to select a claim or
manually input a claim that requires verification. Second, users can start the
verification process by clicking the Submit button. Third, the system shows a
step-by-step question-answering guided reasoning process. Each step includes
the reasoning depth, the generated question, relevant retrieved evidence, and
the corresponding predicted answer. Finally, it presents the final prediction
label with the supporting rationale.

###  3.4 QA Validator

The validator module ensures the usefulness of the newly-generated QA pairs.
For a QA pair to be valid, it must satisfy two criteria: 1) it brings
additional information to the current context $\mathcal{C}$, and 2) it is
useful for verifying the original claim. We again implement the validator by
prompting InstructGPT with a suite of ten demonstrations shown in Appendix
A.3. The instruction is as follows:

> Claim = CLAIM
> We already know the following:
> CONTEXT
> Now we further know:
> NEW QA PAIR
> Does the QA pair have additional
> knowledge useful for verifying
> the claim?

The validator acts as a safeguard against the system producing redundant or
irrelevant QA pairs. Upon validation of a QA pair, it is added to the current
context $\mathcal{C}$. Subsequently, the system initiates another cycle of
calling the claim verifier, question generator, question answering, and
validation.

###  3.5 Reasoner

The reasoner is called when the claim verifier determines that the context
$\mathcal{C}$ is sufficient to verify the claim or the system hits the maximum
allowed iterations, set to 5 by default. The reasoner is a special question-
answering model which takes the context $\mathcal{C}$ and the claim $c$ as
inputs and then answers the question “Is the claim true or false?”. The model
is also requested to output the rationale with the prediction. We provide two
different implementations for the reasoner: 1) the end-to-end QA model based
on FLAN-T5, and 2) the InstructGPT model with the prompts given in Appendix
A.4.

##  4 Performance Evaluation

To evaluate the performance of our QACheck, we use two fact-checking datasets
that contain complex claims requiring multi-step reasoning: HOVER Jiang et al.
(2020) and FEVEROUS Aly et al. (2021), following the same experimental
settings used in Pan et al. (2023). HOVER contains 1,126 two-hop claims, 1,835
three-hop claims, and 1,039 four-hop claims, while FEVEROUS has 2,962 multi-
hop claims. We compare our method with the baselines of directly applying
InstructGPT with two different prompting methods: (_i_) direct prompting with
the claim, and (_ii_) CoT Wei et al. (2022) or chain-of-thought prompting with
few-shot demonstrations of reasoning explanations. We also compare with
ProgramFC Pan et al. (2023), FLAN-T5 Chung et al. (2022), and Codex Chen et
al. (2021). We use the reported results for the baseline models from Pan et
al. (2023).

The evaluation results are shown in Table 1. Our QACheck system achieves a
macro-F1 score of 55.67, 54.67, and 52.35 on HOVER two-hop, three-hop, and
four-hop claims, respectively. It achieves a 59.47 F1 score on FEVEROUS. These
scores are better than directly using InstructGPT, Codex, or FLAN-T5. They are
also on par with the systems that apply claim decomposition strategies, i.e.,
CoT, and ProgramFC. The results demonstrate the effectiveness of our QACheck
system. Especially, the QACheck has better improvement over the end-to-end
models on claims with high reasoning depth. This indicates that decomposing a
complex claim into simpler steps with question-guided reasoning can facilitate
more accurate reasoning.

Model | HOVER | FEVEROUS
---|---|---
2-hop | 3-hop | 4-hop
InstructGPT |  |  |  |
\- Direct | 56.51 | 51.75 | 49.68 | 60.13
\- CoT | 57.20 | 53.66 | 51.83 | 61.05
Codex | 55.57 | 53.42 | 45.59 | 57.85
FLAN-T5 | 48.27 | 52.11 | 51.13 | 55.16
ProgramFC | 54.27 | 54.18 | 52.88 | 59.66
QACheck | 55.67 | 54.67 | 52.35 | 59.47

Table 1: Evaluation of F1 scores for different models. The bold text shows the
best results for each setting.

##  5 User Interface

We create a demo system based on
Flask666https://flask.palletsprojects.com/en/2.3.x/ for verifying open-domain
claims with QACheck, as shown in Figure 4. The QACheck demo is designed to be
intuitive and user-friendly, enabling users to input any claim or select from
a list of pre-defined claims (top half of Figure 4).

Upon selecting or inputting a claim, the user can start the fact-checking
process by clicking the “Submit” button. The bottom half of Figure 4 shows a
snapshot of QACheck’s output for the input claim “Lars Onsager won the Nobel
prize when he was 30 years old”. The system visualizes the detailed question-
guided reasoning process. For each reasoning step, the system shows the index
of the reasoning step, the generated question, and the predicted answer to the
question. The retrieved evidence to support the answer is shown on the right
for each step. The system then shows the final veracity prediction for the
original claim accompanied by a comprehensive rationale in the “Prediction
with rationale” section. This step-by-step illustration not only enhances the
understanding of our system’s fact-checking process but also offers
transparency to its functioning.

QACheck also allows users to change the underlying question–answering model.
As shown at the top of Figure 4, users can select between the three different
QA models introduced in Section 3.3, depending on their specific requirements
or preferences. Our demo system will be open-sourced under the Apache-2.0
license.

##  6 Conclusion and Future Works

This paper presents the QACheck system, a novel approach designed for
verifying real-world complex claims. QACheck conducts the reasoning process
with the guidance of asking and answering a series of questions and answers.
Specifically, QACheck iteratively generates contextually relevant questions,
retrieves and validates answers, judges the sufficiency of the context
information, and finally, reasons out the claim’s truth value based on the
accumulated knowledge. QACheck leverages a wide range of techniques, such as
in-context learning, document retrieval, and question-answering, to ensure a
precise, transparent, explainable, and user-friendly fact-checking process.

In the future, we plan to enhance QACheck 1) by integrating additional
knowledge bases to further improve the breadth and depth of information
accessible to the system Feng et al. (2023); Kim et al. (2023), and 2) by
incorporating a multi-modal interface to support image Chakraborty et al.
(2023), table Chen et al. (2020); Lu et al. (2023), and chart-based fact-
checking Akhtar et al. (2023), which can broaden the system’s utility in
processing and analyzing different forms of data.

## Limitations

We identify two main limitations of QACheck. First, several modules of our
QACheck currently utilize external API-based large language models, such as
InstructGPT. This reliance on external APIs tends to prolong the response time
of our system. As a remedy, we are considering the integration of open-source,
locally-run large language models like LLaMA Touvron et al. (2023). Secondly,
the current scope of our QACheck is confined to evaluating True/False claims.
Recognizing the significance of also addressing Not Enough Information claims,
we plan to devise strategies to incorporate these in upcoming versions of the
system.

## Ethics Statement

The use of large language models requires a significant amount of energy for
computation for training, which contributes to global warming. Our work
performs few-shot in-context learning instead of training models from scratch,
so the energy footprint of our work is less. The large language model
(InstructGPT) whose API we use for inference consumes significant energy.

## Acknowledgement

This project is supported by the Ministry of Education, Singapore, under its
MOE AcRF TIER 3 Grant (MOE-MOET32022-0001). The computational work for this
article was partially performed on resources of the National Supercomputing
Centre, Singapore.

## References

  * Ahmadi et al. (2019) Naser Ahmadi, Joohyung Lee, Paolo Papotti, and Mohammed Saeed. 2019.  Explainable fact checking with probabilistic answer set programming.  In _Proceedings of the 2019 Truth and Trust Online Conference (TTO)_.
  * Akhtar et al. (2023) Mubashara Akhtar, Oana Cocarascu, and Elena Simperl. 2023.  Reading and reasoning over chart images for evidence-based automated fact-checking.  In _Findings of the Association for Computational Linguistics (EACL)_ , pages 399–414.
  * Aly et al. (2021) Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021.  FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information.  In _Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks_.
  * Aly and Vlachos (2022) Rami Aly and Andreas Vlachos. 2022.  Natural logic-guided autoregressive multi-hop document retrieval for fact verification.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6123–6135.
  * Chakraborty et al. (2023) Megha Chakraborty, Khusbu Pahwa, Anku Rani, Adarsh Mahor, Aditya Pakala, Arghya Sarkar, Harshit Dave, Ishan Paul, Janvita Reddy, Preethi Gurumurthy, Ritvik G, Samahriti Mukherjee, Shreyas Chatterjee, Kinjal Sensharma, Dwip Dalal, Suryavardan S, Shreyash Mishra, Parth Patwa, Aman Chadha, Amit P. Sheth, and Amitava Das. 2023.  FACTIFY3M: A benchmark for multimodal fact verification with explainability through 5w question-answering.  _CoRR_ , abs/2306.05523.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 1870–1879.
  * Chen et al. (2022) Jifan Chen, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. 2022.  Generating literal and implied subquestions to fact-check complex claims.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3495–3516.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating large language models trained on code.  _ArXiv preprint_ , abs/2107.03374.
  * Chen et al. (2020) Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020.  Tabfact: A large-scale dataset for table-based fact verification.  In _Proceedings of 8th International Conference on Learning Representations (ICLR)_.
  * Chernyavskiy et al. (2021) Anton Chernyavskiy, Dmitry Ilvovsky, and Preslav Nakov. 2021.  Whatthewikifact: Fact-checking claims against wikipedia.  In _Proceedings of the 30th ACM International Conference on Information and Knowledge Management (CIKM)_ , pages 4690–4695.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.  _CoRR_ , abs/2210.11416.
  * Feng et al. (2023) Shangbin Feng, Vidhisha Balachandran, Yuyang Bai, and Yulia Tsvetkov. 2023.  Factkb: Generalizable factuality evaluation using language models enhanced with factual knowledge.  _CoRR_ , abs/2305.08281.
  * Guo et al. (2022) Zhijiang Guo, Michael Sejr Schlichtkrull, and Andreas Vlachos. 2022.  A survey on automated fact-checking.  _Transactions of the Association for Computational Linguistics (TACL)_ , 10:178–206.
  * Jiang et al. (2020) Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020.  HoVer: A dataset for many-hop fact extraction and claim verification.  In _Findings of the Association for Computational Linguistics (EMNLP)_ , pages 3441–3460.
  * Jolly et al. (2022) Shailza Jolly, Pepa Atanasova, and Isabelle Augenstein. 2022.  Generating fluent fact checking explanations with unsupervised post-editing.  _Information_ , 13:500.
  * Kim et al. (2023) Jiho Kim, Sungjin Park, Yeonsu Kwon, Yohan Jo, James Thorne, and Edward Choi. 2023\.  Factkg: Fact verification via reasoning on knowledge graphs.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 16190–16206.
  * Kotonya and Toni (2020) Neema Kotonya and Francesca Toni. 2020.  Explainable automated fact-checking for public health claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7740–7754.
  * Lin et al. (2021) Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Frassetto Nogueira. 2021.  Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations.  In _Proceedings of International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)_ , pages 2356–2362.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Liu et al. (2020) Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020.  Fine-grained fact verification with kernel graph attention network.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 7342–7351.
  * Lu et al. (2023) Xinyuan Lu, Liangming Pan, Qian Liu, Preslav Nakov, and Min-Yen Kan. 2023.  SCITAB: A challenging benchmark for compositional reasoning and claim verification on scientific tables.  _CoRR_ , abs/2305.13186.
  * Lu and Li (2020) Yi-Ju Lu and Cheng-Te Li. 2020.  GCAN: graph-aware co-attention networks for explainable fake news detection on social media.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 505–514.
  * Martino et al. (2020) Giovanni Da San Martino, Shaden Shaar, Yifan Zhang, Seunghak Yu, Alberto Barrón-Cedeño, and Preslav Nakov. 2020.  Prta: A system to support the analysis of propaganda techniques in the news.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL)_ , pages 287–293.
  * Nguyen et al. (2020) Van-Hoang Nguyen, Kazunari Sugiyama, Preslav Nakov, and Min-Yen Kan. 2020.  FANG: leveraging social context for fake news detection using graph representation.  In _Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM)_ , pages 1165–1174.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _CoRR_ , abs/2303.08774.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS)_.
  * Pan et al. (2023) Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min-Yen Kan, and Preslav Nakov. 2023.  Fact-checking complex claims with program-guided reasoning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 6981–7004.
  * Petroni et al. (2021) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick S. H. Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021.  KILT: a benchmark for knowledge intensive language tasks.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , pages 2523–2544.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100, 000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2383–2392.
  * Robertson and Zaragoza (2009) Stephen E. Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: BM25 and beyond.  _Journal of Foundations and Trends in Information Retrieval_ , 3(4):333–389.
  * Samarinas et al. (2021) Chris Samarinas, Wynne Hsu, and Mong-Li Lee. 2021.  Improving evidence retrieval for automated explainable fact-checking.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: System Demonstrations, (NAACL-HLT)_ , pages 84–91.
  * Sil et al. (2023) Avi Sil, Jaydeep Sen, Bhavani Iyer, Martin Franz, Kshitij Fadnis, Mihaela Bornea, Sara Rosenthal, J. Scott McCarley, Rong Zhang, Vishwajeet Kumar, Yulong Li, Md. Arafat Sultan, Riyaz Bhat, Jürgen Broß, Radu Florian, and Salim Roukos. 2023.  Primeqa: The prime repository for state-of-the-art multilingual question answering research and development.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL)_ , pages 51–62.
  * Sun et al. (2023) Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2023.  Recitation-augmented language models.  In _Proceedings of the 11th International Conference on Learning Representations (ICLR)_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.  _CoRR_ , abs/2302.13971.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _CoRR_ , abs/2201.11903.
  * Xing et al. (2022) Rui Xing, Shraey Bhatia, Timothy Baldwin, and Jey Han Lau. 2022.  Automatic explanation generation for climate science claims.  In _Proceedings of the The 20th Annual Workshop of the Australasian Language Technology Association (ALTA)_ , pages 122–129.
  * Yu et al. (2023) Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023.  Generate rather than retrieve: Large language models are strong context generators.  In _Proceedings of the 11th International Conference on Learning Representations (ICLR)_.
  * Zhang et al. (2019) Yifan Zhang, Giovanni Da San Martino, Alberto Barrón-Cedeño, Salvatore Romeo, Jisun An, Haewoon Kwak, Todor Staykovski, Israa Jaradat, Georgi Karadzhov, Ramy Baly, Kareem Darwish, James R. Glass, and Preslav Nakov. 2019.  Tanbih: Get to know what you are reading.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing: System Demonstrations (EMNLP-IJCNLP)_ , pages 223–228.
  * Zhong et al. (2020) Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020.  Reasoning over semantic-level graph for fact checking.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 6170–6180.

##  Appendix A Prompts

###  A.1 Prompts for Claim Verifier

⬇

Claim = Superdrag and Collective Soul are

both rock bands.

We already know the following:

Question 1 = Is Superdrag a rock band?

Answer 1 = Yes

Can we know whether the claim is

true or false now? Yes or no?

Prediction = No, we cannot know.

Claim = Superdrag and Collective Soul are

both rock bands.

We already know the following:

Question 1 = Is Superdrag a rock band?

Answer 1 = Yes

Question 2 = Is Collective Soul a rock band?

Answer 2 = Yes

Can we know whether the claim is

true or false now? Yes or no?

Prediction = Yes, we can know.

<10 demonstrations in total>

\--------

Claim = [[CLAIM]]

Claim = CLAIM

We already know the following:

[[QA_CONTEXTS]]

Can we know whether the claim is

true or false now? Yes or no?

Prediction =

###  A.2 Prompts for Question Generation

Prompts for the initial question generation

⬇

Claim = Superdrag and Collective Soul are

both rock bands.

To verify the above claim, we can

first ask a simple question:

Question = Is Superdrag a rock band?

<10 demonstrations in total>

\--------

Claim = [[CLAIM]]

To verify the above claim, we can

first ask a simple question:

Question =

Prompts for the follow-up question generation

⬇

Claim = Superdrag and Collective Soul are

both rock bands.

We already know the following:

Question 1 = Is Superdrag a rock band?

Answer 1 = Yes

To verify the claim, what is the

next question we need to know the

answer to?

Question 2 = Is Collective Soul a rock band?

<10 demonstrations in total>

\--------

Claim = [[CLAIM]]

We already know the following:

[[QA_CONTEXTS]]

To verify the claim, what is the

next question we need to know the

answer to?

Question [[Q_INDEX]] =

###  A.3 Prompts for Validator

⬇

Claim = Superdrag and Collective Soul are

both rock bands.

We already know the following:

Question = Is Superdrag a rock band?

Answer = Yes

Now we further know:

Question = Is Collective Soul a rock band?

Answer = Yes

Does the QA pair have additional

knowledge useful for verifying the claim?

The answer: Yes

<10 demonstrations in total>

\--------

Claim = [[CLAIM]]

We already know the following:

[[QA_CONTEXTS]]

Now we further know:

[[NEW_QA_PAIR]]

Does the QA pair have additional

knowledge useful for verifying the claim?

The answer:

###  A.4 Prompts for Reasoner

⬇

Contexts:

Q1: When Lars Onsager won the Nobel Prize?

A1: 1968

Q2: When was Lars Onsager born?

A2: 1903

Claim = Lars Onsager won the Nobel Prize

when he was 30 years old.

Is this claim true or false?

Answer:

Lars Onsager won the Nobel Prize in 1968.

Lars Onsager was born in 1903.

Therefore, the final answer is: False.

<10 demonstrations in total>

\--------

Contexts:

[[CONTEXTS]]

Claim = [[CLAIM]]

Is this claim true or false?

Answer:

Therefore, the final answer is


* [455] Ishan Durugkar, Elad Liebman, and Peter Stone.  Balancing individual preferences and shared objectives in multiagent reinforcement learning.  In Proceedings of the International Joint Conference on Artificial Intelligence, pages 2505–2511, 2020.
  * [456] Willem Röpke.  Reinforcement learning in multi-objective multi-agent systems.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 2999–3001, 2023.
  * [457] Roxana Radulescu, Timothy Verstraeten, Yijie Zhang, Patrick Mannion, Diederik M. Roijers, and Ann Nowé.  Opponent learning awareness and modelling in multi-objective normal form games.  Neural Computing and Applications, 34(3):1759–1781, 2022.
  * [458] Javier Garcıa and Fernando Fernández.  A comprehensive survey on safe reinforcement learning.  Journal of Machine Learning Research, 16(1):1437–1480, 2015.
  * [459] Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll.  A review of safe reinforcement learning: Methods, theory and applications.  preprint arXiv:2205.10330, 2022.
  * [460] Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic.  Provably efficient generalized lagrangian policy optimization for safe multi-agent reinforcement learning.  In Learning for Dynamics and Control Conference, pages 315–332, 2023.
  * [461] Donghao Ying, Yunkai Zhang, Yuhao Ding, Alec Koppel, and Javad Lavaei.  Scalable primal-dual actor-critic method for safe multi-agent rl with general utilities.  preprint arXiv:2305.17568, 2023.
  * [462] Zhili Zhang, Yanchao Sun, Furong Huang, and Fei Miao.  Safe and robust multi-agent reinforcement learning for connected autonomous vehicles under state perturbations.  preprint arXiv:2309.11057, 2023.
  * [463] Charles Dawson, Sicun Gao, and Chuchu Fan.  Safe control with learned certificates: A survey of neural lyapunov, barrier, and contraction methods for robotics and control.  IEEE Transactions on Robotics, 2023.
  * [464] Marc G Bellemare, Will Dabney, and Mark Rowland.  Distributional reinforcement learning.  MIT Press, 2023.
  * [465] Noa Agmon and Peter Stone.  Leading ad hoc agents in joint action settings with multiple teammates.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 341–348, 2012.
  * [466] Stefano V. Albrecht and Peter Stone.  Reasoning about hypothetical agent behaviours and their parameters.  In Proceedings of the Conference on Autonomous Agents and MultiAgent Systems, pages 547–555, 2017.
  * [467] Samuel Barrett, Avi Rosenfeld, Sarit Kraus, and Peter Stone.  Making friends on the fly: Cooperating with new teammates.  Artificial Intelligence, 242:132–171, 2017.
  * [468] Manish Ravula, Shani Alkoby, and Peter Stone.  Ad hoc teamwork with behavior switching agents.  In Proceedings of the International Joint Conference on Artificial Intelligence, pages 550–556, 2019.
  * [469] William Macke, Reuth Mirsky, and Peter Stone.  Expected value of communication for planning in ad hoc teamwork.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11290–11298, 2021.
  * [470] Samuel Barrett, Noa Agmon, Noam Hazon, Sarit Kraus, and Peter Stone.  Communicating with unknown teammates.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 1433–1434, 2014.
  * [471] Arrasy Rahman, Ignacio Carlucho, Niklas Höpner, and Stefano V Albrecht.  A general learning framework for open ad hoc teamwork using graph-based policy learning.  preprint arXiv:2210.05448, 2022.
  * [472] Ted Fujimoto, Samrat Chatterjee, and Auroop Ganguly.  Ad hoc teamwork in the presence of adversaries.  preprint arXiv:2208.05071, 2022.
  * [473] Gerald Tesauro.  Td-gammon, a self-teaching backgammon program, achieves master-level play.  Neural computation, 6(2):215–219, 1994.
  * [474] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  Science, 362(6419):1140–1144, 2018.
  * [475] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster.  “other-play” for zero-shot coordination.  In Proceedings of the International Conference on Machine Learning, pages 4399–4410, 2020.
  * [476] Johannes Heinrich, Marc Lanctot, and David Silver.  Fictitious self-play in extensive-form games.  In Proceedings of the International Conference on Machine Learnin, pages 805–813, 2015.
  * [477] Xihuai Wang, Shao Zhang, Wenhao Zhang, Wentao Dong, Jingxiao Chen, Ying Wen, and Weinan Zhang.  Quantifying zero-shot coordination capability with behavior preferring partners.  preprint arXiv:2310.05208, 2023.
  * [478] Rui Zhao, Jinming Song, Hu Haifeng, Yang Gao, Yi Wu, Zhongqian Sun, and Yang Wei.  Maximum entropy population based training for zero-shot human-AI coordination.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6145–6153, 2023.
  * [479] Darius Muglich, Christian Schroeder de Witt, Elise van der Pol, Shimon Whiteson, and Jakob Foerster.  Equivariant networks for zero-shot coordination.  In Advances in Neural Information Processing Systems, pages 6410–6423, 2022.
  * [480] Lebin Yu, Yunbo Qiu, Quanming Yao, Xudong Zhang, and Jian Wang.  Improving zero-shot coordination performance based on policy similarity.  preprint arXiv:2302.05063, 2023.
  * [481] Yang Li, Shao Zhang, Jichen Sun, Yali Du, Ying Wen, Xinbing Wang, and Wei Pan.  Cooperative open-ended learning framework for zero-shot coordination.  preprint arXiv:2302.04831, 2023.
  * [482] Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn.  One solution is not all you need: Few-shot extrapolation via structured maxent RL.  In Advances in Neural Information Processing Systems, pages 8198–8210, 2020.
  * [483] Takayuki Osa, Voot Tangkaratt, and Masashi Sugiyama.  Discovering diverse solutions in deep reinforcement learning by maximizing state–action-based mutual information.  Neural Networks, 152:90–104, 2022.
  * [484] Jean-Baptiste Gaya, Laure Soulier, and Ludovic Denoyer.  Learning a subspace of policies for online adaptation in reinforcement learning.  In International Conference on Learning Representations, 2022.
  * [485] Arash Ajoudani, Andrea Maria Zanchettin, Serena Ivaldi, Alin Albu-Schäffer, Kazuhiro Kosuge, and Oussama Khatib.  Progress and prospects of the human–robot collaboration.  Autonomous Robots, 42:957–975, 2018.
  * [486] Federico Vicentini.  Collaborative robotics: a survey.  Journal of Mechanical Design, 143(4):040802, 2021.
  * [487] Niels Van Berkel, Mikael B Skov, and Jesper Kjeldskov.  Human-ai interaction: intermittent, continuous, and proactive.  Interactions, 28(6):67–71, 2021.
  * [488] Linda Onnasch and Eileen Roesler.  A taxonomy to structure and analyze human–robot interaction.  International Journal of Social Robotics, 13(4):833–849, 2021.
  * [489] Rose E Wang, Sarah A Wu, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman-Weiner.  Too many cooks: Coordinating multi-agent collaboration through inverse planning.  In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 2032–2034, 2020.
  * [490] Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba.  Watch-and-help: A challenge for social perception and human-ai collaboration.  In International Conference on Learning Representations, 2020.
  * [491] Adam Lerer and Alexander Peysakhovich.  Learning existing social conventions via observationally augmented self-play.  In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 107–114, 2019.
  * [492] Mycal Tucker, Yilun Zhou, and Julie Shah.  Adversarially guided self-play for adopting social conventions.  preprint arXiv:2001.05994, 2020.
  * [493] Andy Shih, Arjun Sawhney, Jovana Kondic, Stefano Ermon, and Dorsa Sadigh.  On the critical role of conventions in adaptive human-ai collaboration.  In International Conference on Learning Representations, 2020.
  * [494] Mengxi Li, Minae Kwon, and Dorsa Sadigh.  Influencing leading and following in human–robot teams.  Autonomous Robots, 45:959–978, 2021.
  * [495] Yang Li, Shao Zhang, Jichen Sun, Wenhao Zhang, Yali Du, Ying Wen, Xinbing Wang, and Wei Pan.  Tackling cooperative incompatibility for zero-shot human-ai coordination.  preprint arXiv:2306.03034, 2023.
  * [496] Arun Kumar AV, Santu Rana, Alistair Shilton, and Svetha Venkatesh.  Human-ai collaborative bayesian optimisation.  In Advances in Neural Information Processing Systems, pages 16233–16245, 2022.
  * [497] Jakob Thumm, Felix Trost, and Matthias Althoff.  Human-robot gym: Benchmarking reinforcement learning in human-robot collaboration.  preprint arXiv:2310.06208, 2023.
  * [498] Maxence Hussonnois, Thommen George Karimpanal, and Santu Rana.  Controlled diversity with preference: Towards learning a diverse set of desired skills.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 1135–1143, 2023.
  * [499] Luyao Yuan, Xiaofeng Gao, Zilong Zheng, Mark Edmonds, Ying Nian Wu, Federico Rossano, Hongjing Lu, Yixin Zhu, and Song-Chun Zhu.  In situ bidirectional human-robot value alignment.  Science robotics, 7(68):eabm4183, 2022.
  * [500] Meng Guo and Mathias Bürger.  Interactive human-in-the-loop coordination of manipulation skills learned from demonstration.  In 2022 International Conference on Robotics and Automation, pages 7292–7298, 2022.
  * [501] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al.  A survey of large language models.  preprint arXiv:2303.18223, 2023.
  * [502] Weirui Ye, Yunsheng Zhang, Mengchen Wang, Shengjie Wang, Xianfan Gu, Pieter Abbeel, and Yang Gao.  Foundation reinforcement learning: towards embodied generalist agents with foundation prior assistance.  preprint arXiv:2310.02635, 2023.
  * [503] Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans.  Foundation models for decision making: Problems, methods, and opportunities.  preprint arXiv:2303.04129, 2023.
  * [504] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al.  A generalist agent.  Transactions on Machine Learning Research, 2022.
  * [505] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap.  Mastering diverse domains through world models.  preprint arXiv:2301.04104, 2023.
  * [506] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  In Advances in neural information processing systems, pages 15084–15097, 2021.
  * [507] Shengchao Hu, Li Shen, Ya Zhang, Yixin Chen, and Dacheng Tao.  On transforming reinforcement learning by transformer: The development trajectory.  preprint arXiv:2212.14164, 2022.
  * [508] Zhuoran Li, Ling Pan, and Longbo Huang.  Beyond conservatism: Diffusion policies in offline multi-agent reinforcement learning.  preprint arXiv:2307.01472, 2023.
  * [509] Tao Li, Juan Guevara, Xinghong Xie, and Quanyan Zhu.  Self-confirming transformer for locally consistent online adaptation in multi-agent reinforcement learning.  preprint arXiv:2310.04579, 2023.
  * [510] Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein.  Generative agents: Interactive simulacra of human behavior.  preprint arXiv:2304.03442, 2023.
  * [511] Ziluo Ding, Wanpeng Zhang, Junpeng Yue, Xiangjun Wang, Tiejun Huang, and Zongqing Lu.  Entity divider with language grounding in multi-agent reinforcement learning.  In Proceedings of the International Conference on Machine Learning, pages 8103–8119, 2023.
  * [512] Hengyuan Hu and Dorsa Sadigh.  Language instructed reinforcement learning for human-ai coordination.  In Proceedings of the International Conference on Machine Learning, pages 13584–13598, 2023.
  * [513] Ceyao Zhang, Kaijie Yang, Siyi Hu, Zihao Wang, Guanghe Li, Yihang Sun, Cheng Zhang, Zhaowei Zhang, Anji Liu, Song-Chun Zhu, et al.  Proagent: Building proactive cooperative ai with large language models.  preprint arXiv:2308.11339, 2023.
  * [514] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi.  Encouraging divergent thinking in large language models through multi-agent debate.  preprint arXiv:2305.19118, 2023.
  * [515] Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu.  Chateval: Towards better llm-based evaluators through multi-agent debate.  preprint arXiv:2308.07201, 2023.
  * [516] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang.  Autogen: Enabling next-gen llm applications via multi-agent conversation framework.  preprint arXiv:2308.08155, 2023.
  * [517] Nuoya Xiong, Zhihan Liu, Zhaoran Wang, and Zhuoran Yang.  Sample-efficient multi-agent rl: An optimization perspective.  preprint arXiv:2310.06243, 2023.
  * [518] Wanpeng Zhang and Zongqing Lu.  Rladapter: Bridging large language models to reinforcement learning in open worlds.  preprint arXiv:2309.17176, 2023.
  * [519] Yuxi Li.  Reinforcement learning applications.  preprint arXiv:1908.06973, 2019.