# CFEVER: A Chinese Fact Extraction and VERification Dataset

Ying-Jia Lin, Chun-Yi Lin, Chia-Jen Yeh, Yi-Ting Li,
Yun-Yu Hu, Chih-Hao Hsu, Mei-Feng Lee, Hung-Yu Kao

###### Abstract

We present CFEVER, a Chinese dataset designed for Fact Extraction and
VERification. CFEVER comprises 30,012 manually created claims based on content
in Chinese Wikipedia. Each claim in CFEVER is labeled as “Supports”,
“Refutes”, or “Not Enough Info” to depict its degree of factualness. Similar
to the FEVER dataset, claims in the “Supports” and “Refutes” categories are
also annotated with corresponding evidence sentences sourced from single or
multiple pages in Chinese Wikipedia. Our labeled dataset holds a Fleiss’ kappa
value of 0.7934 for five-way inter-annotator agreement. In addition, through
the experiments with the state-of-the-art approaches developed on the FEVER
dataset and a simple baseline for CFEVER, we demonstrate that our dataset is a
new rigorous benchmark for factual extraction and verification, which can be
further used for developing automated systems to alleviate human fact-checking
efforts. CFEVER is available at https://ikmlab.github.io/CFEVER.

## Introduction

Fact verification involves assessing the truthfulness of claims presented in
text or speech. In recent years, the popularization of media platforms has
accelerated the spread of misinformation, making fact verification a critical
task to prevent the public from being exposed to false information. However,
the process of fact verification typically involves extensive searches and
assessments from many potential sources conducted by journalists, which is
time-consuming and labor-intensive (Guo, Schlichtkrull, and Vlachos 2022).
Consequently, it is imperative to develop automated fact verification systems
to speed up the verification process.

In recent years, researchers have been building fact-verification systems with
deep neural networks (Ma et al. 2016), where the models are provided with a
claim and required to determine whether the claim is true or false. A well-
established dataset for fact verification is FEVER (Fact Extraction and
VERification) (Thorne et al. 2018a) and its shared task (Thorne et al. 2018b).
FEVER asks models to extract factual sentences as evidence from the fixed
Wikipedia database and provide a verdict for a given claim. Therefore, the
task serves for both verification and evidence extraction for claims. Though
FEVER has been widely used as a benchmark for fact verification, the dataset
and most of the other fact verification datasets (Wang 2017; Hanselowski et
al. 2019; Schuster, Fisch, and Barzilay 2021) are created in English. The
dissemination of rumors and fake news is a serious problem in East Asia,
especially in China. Compared to English, text written in Chinese tends to be
more ambiguous and nuanced, making it more difficult for people to identify
misinformation. Therefore, it is critical to create a clean and high-quality
dataset as supervised data for Chinese fact verification. To achieve this
goal, Hu et al. (2022) proposed the CHEF dataset from the sources of the fact-
checking and news websites. However, compared to English fact verification
datasets (Augenstein et al. 2019; Kotonya and Toni 2020), the size of CHEF is
relatively small, making training and evaluating fact verification systems
challenging. A potential alternative for building Chinese fact verification
systems is to take advantage of the multilingual fact-checking dataset
(Nielsen and McConville 2022). Nevertheless, the Chinese claims in the dataset
are still limited in size.

Table 1: An example from the CFEVER dataset. Underlined words indicate
knowledge that should be verified from other pages in Wikipedia.

Dataset | #Claims | Source | Language | Task
---|---|---|---|---
Document | Evidence | Claim
Retrieval | Retrieval | Verification
Weibo-16 (Ma et al. 2016) | 4,664 | Weibo | Chinese | No | No | Yes*
Weibo-20 (Zhang et al. 2021) | 6,362 | Weibo | Chinese | No | No | Yes*
MuMiN (Nielsen and McConville 2022) | 1,283† | Tweets | Multi‡ | No | No | Yes
CHEF (Hu et al. 2022) | 10,000 |  Fact-checking/ News websites | Chinese | No | Yes | Yes
CFEVER | 30,012 | Wikipedia | Chinese | Yes | Yes | Yes

* Rumor detection task. † Including Chinese and other languages. ‡ Multi-lingual dataset.

Table 2: Comparison of CFEVER with other fact verification datasets in
Chinese.

In this work, we present CFEVER, a Chinese dataset for Factual Extraction and
VERification. Following FEVER’s construction process (Thorne et al. 2018a), we
first built a fact database with the fixed version of the Chinese Wikipedia
dump. Then, we hired several workers from our university to alter the
extracted sentences from the Wikipedia pages and label each claim with
“Supports”, “Refutes”, or “Not Enough Info” based on the Wikipedia pages in
the fact database. The relevant factual sentences from the fact database were
also annotated by our workers as evidence for the claims belonging to the
first two categories. Our CFEVER dataset includes 30,012 claims in total,
which are three times the size of CHEF (Hu et al. 2022). To evaluate our
labeling quality, we report the five-way inter-annotator agreement of 0.7934
in Fleiss $\kappa$ (Fleiss 1971) using 6.45% claims of our dataset, while the
scores in FEVER and CHEF are 0.6841 and 0.74, measured from 4% and 3% of
claims from the datasets, respectively.

To thoroughly assess the challenges of CFEVER, following the FEVER task
(Thorne et al. 2018b), we conduct experiments on the three stages: document
retrieval, sentence retrieval, and claim verification. We test the performance
for each stage and the full-pipeline setting with the state-of-the-art
approaches (Stammbach 2021; DeHaven and Scott 2023) developed on FEVER and our
simple baseline (Robertson, Zaragoza et al. 2009; Soleimani, Monz, and Worring
2020) along with oracle experiments to dive into the challenges of CFEVER.
Further extensive analysis even reveals the characteristics and difficulty of
CFEVER. In summary, we list our contributions as follows:

  * •

We present CFEVER, the currently largest Chinese dataset for evidence-based
fact verification.

  * •

The five-way inter-annotator agreement of CFEVER in Fleiss $\kappa$ indicates
that the dataset was built with high label consistency among claims.

  * •

Extensive experiments on CFEVER show that the proposed dataset can serve as a
challenging benchmark for future research or development on Chinese fact
verification.

## Related Work

### FEVER

Thorne et al. (2018b) created the task of Fact Extraction and VERification
(FEVER) along with the dataset of the same name (Thorne et al. 2018a). FEVER
is the currently largest fact verification dataset that contains 185,445
claims. Each of the claims was first sampled from the sentences in the
introductory sections of approximately 50,000 popular pages and then revised
by human annotators. In another round of annotation, the annotators label the
claims as “Supports”, “Refutes”, or “Not Enough Information”, and discover
evidence sentences from Wikipedia pages. The following year, Thorne et al.
(2019) introduced the FEVER 2.0 task to first generate adversarial claims to
fool the existing verification systems built with the FEVER dataset (Thorne et
al. 2018a). Then the task required participants to improve the systems to
prevent such adversarial attacks. More recently, Aly et al. (2021) proposed
FEVEROUS and extended the fact verification task for verifying claims with the
information in structured data, such as tables in Wikipedia.

Nørregaard and Derczynski (2021) follow the annotation process of FEVER and
propose a new FEVER dataset (Thorne et al. 2018a) in Danish. Jiang et al.
(2020) report that 87% of the claims in the FEVER dataset (Thorne et al.
2018a) require only a single Wikipedia page for verification, which does not
support real-world situations where misinformation may come from multiple
articles. Thus, Jiang et al. (2020) propose a new dataset containing 26K
claims for multi-hop reasoning based on the building process of FEVER.

### Chinese Fact Verification

We compare CFEVER with other Chinese fact verification datasets in Table 2.
Existing Chinese fact verification datasets mainly focus on rumor detection,
such as the Weibo-16 (Ma et al. 2016) and Weibo-20 datasets (Zhang et al.
2021). In this work, we focus on both the fact extraction and verification
tasks, which are different from the binary claim detection in rumor detection
(Guo, Schlichtkrull, and Vlachos 2022). The dataset closest to our work is
CHEF (Hu et al. 2022), which is a pilot Chinese dataset for evidence-based
fact-checking. There are two main differences between CHEF and our dataset.
First, CHEF is created from the sources of fact-checking websites, while our
dataset is created from Wikipedia. Second, there is no document retrieval
process in the task of CHEF, with candidate evidence sentences provided for
each claim in the dataset. In contrast, we follow FEVER (Thorne et al. 2018a)
to provide a fixed fact database and ask models to first extract relevant
documents from Wikipedia before verifying the claims with the evidence
sentences. We consider our approach, along with FEVER (Thorne et al. 2018a),
to be more realistic for real-world fact verification scenarios.

## Dataset Construction

We followed the labeling approach of FEVER (Thorne et al. 2018a) to create the
CFEVER dataset and adapted the annotation
platform111https://github.com/awslabs/fever/tree/master/fever-annotations-
platform publicly released by Thorne et al. (2018a) for our construction task.
The annotation process consists of two stages: claim generation and claim
annotation. Both stages are conducted by human workers recruited from our
university. The two stages of annotation are conducted separately. To
distinguish the two stages, we refer to the workers in the claim generation
stage as writers and the ones in the claim annotation stage as annotators,
based on the characteristics of their tasks. Before the construction process,
we first describe our method to prepare the Wikipedia data and the fact
database.

### Preparation for the Wikipedia Data

We used the December 2022 dump of the Chinese Wikipedia and extracted the text
from the introductory section from each page following the pre-processing
method of FEVER (Thorne et al. 2018a). The raw Wikipedia comprises articles in
both Traditional and Simplified Chinese. To unify the data, we processed the
text with the open-source software OpenCC222https://github.com/BYVoid/OpenCC
to convert the text to Traditional Chinese. Then, the processed data
containing 1,187,751 pages were fixed to serve as the fact database for
CFEVER. For the next stage of data construction, we created a source page pool
based on the processed Wikipedia pages. The source page pool includes the 500
most visited Chinese Wikipedia
pages333https://pageviews.wmcloud.org/topviews/?project=zh.wikipedia.org
worldwide in 2022, 10,000 Taiwanese pages, and 3,000 random pages. All the
claims in our dataset were created based on the pages in the source page pool.

### Claim Generation

At this stage, writers are responsible for writing claims based on the
Wikipedia pages in the source page pool. At each time, each writer was given
one extracted sentence randomly sampled from the introductory section of a
page in the source page pool. In addition, Wikipedia pages related to the
given sentence based on the hyperlinks in the raw Wiki data were also provided
to the writer. Then, we asked a writer to first generate a TRUE claim based on
the given extracted sentence and the information in the relevant pages without
any learned human knowledge. The provided relevant pages were used to help
writers come up with diverse claims, which may also result in complex claims
that require multi-step reasoning from different pages in the verification
task. After that, the writer was asked to generate six variants of the TRUE
claim:

  * •

Rephrasing: A TRUE claim should be rephrased to a different sentence with the
same meaning.

  * •

Negation: A TRUE claim should be negated without simple negation words, such
as “not”.

  * •

Entity substitution at a similar level: An entity in a TRUE claim should be
substituted with another one similar to the original entity.

  * •

Entity substitution at a disjointed level: An entity in a TRUE claim should be
substituted with another one disjointed to the original entity.

  * •

Specification: A TRUE claim should be narrowed down with more specific
concepts.

  * •

Generalization: A TRUE claim should be generalized with more abstract
concepts.

During claim generation, the writers were asked not to generate claims with
their own learned knowledge. They should write claims solely based on the
information in the given extracted sentence and the relevant pages. The reason
behind this step is to help generate verifiable claims and maintain the
quality of the generated claims among different writers. We also measured the
domains of the claims in CFEVER. To prevent misclassification based on the
domains of source Wiki pages, we asked the writers to select a category from
the pre-defined domains for each claim they generated. The final domain
distribution of the generated claims is shown in Figure 1.

Figure 1: Domain distribution of the generated claims in our dataset. H&S
refers to the domain of Humanities & Social Sciences.

### Claim Annotation

Once a claim was generated, an annotator was asked to label the claim as
“Supports”, “Refutes”, or “Not Enough Info”. For the first two categories, the
annotator must also find the sentences as evidence from the fact database. To
achieve such a process, at each time of annotation, four kinds of materials
are provided to the annotator in default:

  * •

Claim: The claim generated by the writer in the claim generation stage.

  * •

Page name: The title of the original page from which the claim was generated.

  * •

Original sentences: The sentences in the introductory section of the original
page which the claim was generated from.

  * •

Relevant pages: The pages related to the original page based on the hyperlinks
in the raw Wiki data.

If none of the sentences provided in default can be selected as evidence for
the claim, an annotator can also search the fact database with a Wiki page
name as a keyword. After passing the keyword to the annotation platform, the
sentences in the introductory section of the Wikipedia page will show up at
the annotation interface to be selectable as evidence by an annotator. Note
that a claim may become “Supports” or “Refutes” based on multiple sentences.
We encouraged the annotators to find as many sentences as possible to support
or refute a claim from the fact database. Once the annotator considered no
factual sentences in our fact database could support or refute a claim, the
claim was labeled as “Not Enough Info.”

Some claims were generated based on the original extracted sentence along with
the content from the relevant pages in the claim generation stage. These
claims require evidence referenced from multiple pages. For the example in
Table 1, the term “Middle East” in the claim cannot be inferred directly from
the sentence mentioning “Iraq.” Another sentence in the “Iraq” page should be
selected as a part of the evidence, even though the relationship between the
two terms is common knowledge. Therefore, for these claims, we asked
annotators to combine two or more sentences from different pages as complete
evidence to support or refute the claim using the annotation platform.

### Workers

We recruited nine writers and annotators from our university in total. All of
the workers were native Chinese speakers. Among them, three workers were from
the College of Liberal Arts, with two of them focused on the claim generation
task only, and the remaining six were from the College of Engineering. The
workers were trained by the authors for the annotation tasks and guidelines
until they were able to undergo the annotation process correctly and
independently.

Split | Training | Development | Test
---|---|---|---
Total Claims | 24,012 | 3,000 | 3,000
Num of SUP | 11,085 | 1,000 | 1,000
Num of REF | 7,113 | 1,000 | 1,000
Num of NEI | 5,814 | 1,000 | 1,000
Avg. Claim Length | 33.19 | 34.04 | 34.04
Avg. Evidence | 1.57 sents | 1.53 sents | 1.55 sents
per Claim
Avg. Evidence | 1.13 | 1.14 | 1.12
Pages per Claim | (88.41%) | (87.35%) | (89.45%)

Table 3: Dataset statistics of CFEVER in different splits. SUP indicates the
“Supports” class, REF stands for “Refutes”, and NEI represents “Not Enough
Information”. The Avg. Claim Length is the average number of characters in a
claim. The Avg. Evidence per Claim is the average number of evidence sentences
for a claim in the “Supports” or “Refutes” class, and the ratio in the
parentheses is the proportion of claims with evidence from one single page.

### Data Validation

To evaluate the consistency of the class labels among the annotators, we
randomly sampled 1,936 claims (6.45% of our total dataset) and asked five
annotators from our workers to label them. The five-way inter-annotator
agreement over the 1,936 claims shows a score of 0.7934 in Fleiss $\kappa$
(Fleiss 1971), which is higher than the score of 0.6841 measured with 7,506
claims (4% from total) in FEVER (Thorne et al. 2018a) and the one of 0.74 with
310 claims (3% from total) in the CHEF dataset (Hu et al. 2022). To evaluate
the correctness of the evidence sentences, we randomly sampled another 700
claims (2.33% of the dataset) for being reviewed by the authors. We discovered
that 84.4% of the claims are annotated correctly with correct evidence
sentences.

### Dataset Statistics

There are 30,012 claims in the CFEVER dataset. We split 80%, 10%, and 10% of
the claims into the training, development, and test sets, respectively. The
statistics of the dataset are shown in Table 3. The number of claims in the
three categories is balanced in the development and test sets to ensure that
the performance of the models is not biased towards any category during
evaluations. The average claim length (character level) and the average number
of evidence sentences per claim are also similar among the three splits. We
also report the ratio of claims with evidence from a single page in Table 3,
with 88.41%, 87.35%, and 89.45% of the claims whose evidence can be found in a
single page in the three sets. These ratios are close to the statistic of
FEVER (Thorne et al. 2018a) reported by Jiang et al. (2020), where 87% of the
claims require one page for verification.

## Baseline Systems

Following Thorne et al. (2018b), our task requires a model to retrieve
evidence from the Wikipedia fact database and perform verification for each
claim. This section introduces the approaches we test for CFEVER in three
stages: document retrieval, sentence retrieval, and recognizing textual
entailment (RTE) for claim verification. The systems we build involves two
full-pipeline methods444BERT (https://huggingface.co/hfl/chinese-bert-wwm-ext)
is used for BEVERS and our baseline unless otherwise noted. for the three
stages: one simple baseline proposed by ourselves and the state-of-the-art
approach (DeHaven and Scott 2023) developed for FEVER (Thorne et al. 2018a).
We also test CFEVER with the sentence retrieval approach proposed by Stammbach
(2021). We provide essential details in this section, and more information is
available on the CFEVER website.

### Our Baseline

To understand the difficulty and behaviors of CFEVER, we first design a
baseline with simple components to explore the dataset.

#### Document Retrieval

For evidence-based claim verification, relevant pages should be discovered for
each claim to extract evidence. Many studies for the FEVER task (Thorne et al.
2018b) adopt TF-IDF (Thorne et al. 2018a) or the search with the MediaWiki API
(Hanselowski et al. 2018) for document retrieval. Following Jiang et al.
(2020), we use BM25 (Robertson, Zaragoza et al. 2009) for retrieving relevant
pages for each claim. Our implementation was based on
Elasticsearch555https://github.com/elastic/elasticsearch, and the
representations were built with the Wikipedia pages from our fact database.

#### Sentence Retrieval

After retrieving relevant pages for each claim, we perform sentence retrieval
to select evidence sentences from the pages. Inspired by Hanselowski et al.
(2018) and Soleimani, Monz, and Worring (2020), we implement a pointwise
approach for sentence retrieval with BERT (Devlin et al. 2019) to classify
each claim-sentence pair in binary. Positive pairs are created using claims
and their corresponding gold evidence sentences. In contrast, negative pairs
consist of claims paired with non-evidence sentences, which are sampled from
the predicted pages acquired during the document retrieval phase.

#### Recognizing Textual Entailment

We verify each claim with the evidence sentences by recognizing textual
entailment (RTE). Following Hanselowski et al. (2018); Nie, Chen, and Bansal
(2019), we first concatenate a claim with its top five evidence sentences and
then fine-tune the BERT model (Devlin et al. 2019) for the three-class RTE
task.

### BEVERS

#### Document Retrieval

The second approach is based on the BEVERS (DeHaven and Scott 2023), which is
the state-of-the-art full-pipeline system for FEVER (Thorne et al. 2018b).
BEVERS uses a hybrid approach to include both the search results from
Wikisearch (Hanselowski et al. 2018) and the TF-IDF method (Thorne et al.
2018a). Such a hybrid approach was also adopted by Stammbach (2021). Note that
BEVERS replaced the MediaWiki API in the approach of Hanselowski et al. (2018)
with a fuzzy string search system.

#### Sentence Retrieval

Besides the document retrieval approach, DeHaven and Scott (2023) also
proposed a competitive sentence retrieval approach. BEVERS extends the binary
pointwise approach (Hanselowski et al. 2018) with an additional ternary
classification task for classifying a claim-sentence pair into “Supports”,
“Refutes”, or “Not Enough Info”, as an initial set of predicted evidence
sentences. Then, BEVERS used the results from the initial set to explore more
evidence sentences from the hyperlinks in a Wikipedia article, which was
called “re-retrieval.” Finally, all extracted sentences from these two steps
are ranked to yield the final evidence sentences for each claim.

#### Recognizing Textual Entailment

In addition to the concatenation-based approach (Hanselowski et al. 2018; Nie,
Chen, and Bansal 2019) for performing claim verification with concatenated
evidence, the singleton-based approach (Malon 2018; Soleimani, Monz, and
Worring 2020) was also proposed to classify each claim-evidence pair
individually. In this setting, each claim will have multiple scores for each
evidence sentence. Then, the scores are aggregated based on the rules (Malon
2018; Soleimani, Monz, and Worring 2020) to obtain the final prediction.
BEVERS adopts a mixture of both approaches (DeHaven and Scott 2023). They
first fine-tune DeBERTa-V2-XL (He et al. 2021) pre-trained on MNLI (Williams,
Nangia, and Bowman 2018) for each of the approaches and train an additional
gradient boosting classifier (Friedman 2001) for aggregating the final
predictions.

### Stammbach

Stammbach (2021) treats sentence retrieval as a token-level classification
problem, where a model must predict 1 for each token within evidence sentences
and 0 for the tokens belonging to non-evidence sentences. Such an approach
requires a model to process long sequences from input claim-article pairs.
Thus, Stammbach (2021) adopts BigBird (Zaheer et al. 2020) as the encoder.
Since this approach was proposed for sentence retrieval, we only test this
approach for retrieving evidence with the ground-truth documents.

## Evaluation Metrics

For document retrieval and sentence retrieval, we report the performance in
recall (%). Our recall evaluation metric is designed to assess the model’s
ability to correctly predict at least one complete set of evidence pages
during document retrieval and, similarly, at least one complete set of
evidence sentences during sentence retrieval, for each data instance. For
claim verification in RTE, following Thorne et al. (2018b), we report
performance in accuracy (%) and FEVER Score (%). The latter is a strict
measure of accuracy, requiring a model to correctly predict at least one
complete evidence set for each claim. For implementing the evaluation metrics,
we use the script from DeHaven and Scott (2023) for document retrieval. For
sentence retrieval and RTE, we utilize the official scoring tool from Thorne
et al. (2018b).

## Results and Analysis

### Results in Different Stages

To thoroughly analyze CFEVER, we list the results for the different stages in
Table 4. For document retrieval, we observe that BEVERS achieves more than 90%
of recall, demonstrating that the approach is able to find correct articles
for most of the given claims. The performance difference between our simple
baseline (BM25) and BEVERS is primarily due to the hybrid approach employed in
BEVERS, which combines the search predictions from the TF-IDF method and the
fuzzy string search.

For sentence retrieval, there’s a huge performance gap between our simple
baseline and BEVERS. This may result from the training with an additional
ternary classification task and the employment of the re-retrieval technique
in BEVERS, whose score is also far ahead of the other baselines close to our
sentence selection method on FEVER. However, BEVERS scored 94.41% in recall
reported in their paper (DeHaven and Scott 2023) for FEVER, showing that our
dataset remains a challenge for BEVERS.

For claim verification and the full-pipeline setting, BEVERS obtains only
69.73% for the label accuracy, which is much lower than their score (80.24%)
reported for the FEVER dataset (DeHaven and Scott 2023). Since the
classification is based on the evidence extracted in the sentence retrieval
stage, the scores for claim verification will be affected if a model cannot
identify correct evidence sentences. Still, BEVERS significantly outperforms
our simple baseline by 8.56% in label accuracy and 12.33% in FEVER Score.

We also test the performance of GPT-3.5 (Ouyang et al. 2022) and GPT-4 (OpenAI
2023)666The prompts we used can be found from the CFEVER website. for claim
verification with the test set using the zero-shot and few-shot settings. For
the few-shot setting, we sample three labeled claims from the training set
with the same domain as the input claim for each class. From Table 4, we find
that the claims in CFEVER are challenging for both models, and the performance
can be slightly improved with the few-shot setting.

### Oracle Results

To analyze the difficulty of CFEVER further, we also report the oracle results
for the last two stages in Table 5. For the oracle setting in sentence
retrieval, the gold documents of each claim are provided for the models. As
for the oracle setting in claim verification, the models take gold evidence
sentences for each claim as inputs. The results show that both methods achieve
more than 95% in recall for the oracle setting in sentence retrieval. However,
there are still 15% and 10% error rates in claim verification for our baseline
and BEVERS
($\textrm{RoBERTa}_{\textrm{Large}}$777https://huggingface.co/hfl/chinese-
roberta-wwm-ext-large), showing that certain claims in our dataset pose
verification challenges. We notice that Stammbach performs worse than the
other two baselines for sentence retrieval. The results may be affected by the
Chinese BigBird we used888https://huggingface.co/Lowin/chinese-bigbird-
base-4096, since there is no official Chinese version of BigBird (Zaheer et
al. 2020).

Task (metric) | System | Score (%)
---|---|---
Doc retrieval | Our baseline | 87.65
(Recall) | BEVERSa | 92.60
Sent retrieval | Our baseline | 76.65
(Recall) | BEVERSa | 86.60
RTE (Accuracy) | Our baseline | 61.17
BEVERSa | 69.73
GPT-3.5 (zeroshot) | 43.17
GPT-3.5 (3-shot) | 44.20
GPT-4 (zeroshot) | 47.23
GPT-4 (3-shot) | 48.40
Full pipeline | Our baseline | 52.47
(FEVER Score) | BEVERSa | 64.80
Table 4: Results on the different stages. $a$: DeHaven and Scott (2023). Task (metric) | System | Score (%)
---|---|---
Sent retrieval (Recall) | Our baseline | 95.90
BEVERSa | 95.20
Stammbachb | 83.55
RTE (Accuracy) | Our baseline | 85.10
BEVERSa ($\textrm{BERT}_{\textrm{Base}}$) | 88.50
BEVERSa ($\textrm{RoBERTa}_{\textrm{Large}}$) | 90.33
Table 5: Oracle results on sentence retrieval and the task of recognizing
textual entailment (RTE). $a$: DeHaven and Scott (2023). $b$: Stammbach
(2021).

### Analysis for Claims with Evidence from Multiple Pages

After testing the two methods in different stages, we further analyze the
performance on claims with evidence labeled from different numbers of
Wikipedia pages for the full-pipeline setting in accuracy and FEVER Score. The
results in Table 6 show that the performance of both methods degrades
significantly for the claims with evidence from more pages. For the claims
with more than three pages of evidence, the two methods achieve unsatisfactory
performance in FEVER Score. In summary, we identify that about 10% claims with
evidence from multiple pages in our dataset are challenging to verify. Even
BEVERS can only get about 70% of the FEVER Score for the claims with evidence
from single pages.

Methods | #Pages (ratio)
---|---
1 (89.45%) | 2 (9.35%) |  $\geq$ 3 (1.20%)
Ours | 67.75 / 57.97 | 55.08 / 15.51 | 50.00 / 0.00
BEVERSa | 73.95 / 70.54 | 63.10 / 21.93 | 58.33 / 16.67

Table 6:  Full-pipeline results in accuracy / FEVER Score (%) for the claims
with evidence from different number of pages. “Ratio” indicates the proportion
of the claims from the test set (not counted for the claims of “Not Enough
Information”) in each group. $a$: DeHaven and Scott (2023).

### Analysis for Claims with Different Numbers of Evidence Sentences

Methods | #Evidence sentences (ratio)
---|---
1 | 2 | 3 |  $\geq$ 4
(61.75%) | (26.40%) | (10.15%) | (1.70%)
Ours | 64.1 / 55.9 | 72.2 / 56.3 | 65.5 / 36.9 | 61.8 / 11.8
BEVERSa | 70.8 / 67.8 | 75.6 / 65.3 | 80.3 / 57.1 | 55.9 / 26.5

Table 7:  Full-pipeline results in accuracy / FEVER Score (%) for the claims
with different numbers of evidence sentences. “Ratio” indicates the proportion
of the claims from the test set (not counted for the claims of “Not Enough
Information”) in each group. a: DeHaven and Scott (2023).

Since each claim in our dataset can have multiple evidence sentences, we also
analyze the performance on claims with different numbers of evidence
sentences. The results in Table 7 show that both methods perform better on the
claims with two and three evidence sentences in the full-pipeline setting.
This may result from the fact that the claims with one evidence sentence are
usually shorter ones, containing less information to be verified.
Additionally, the claims with more than three evidence sentences are usually
much longer, requiring more correct evidence sentences to be identified and
thus more difficult for the models.

### Analysis for Claims from Different Domains

As the chart in Figure 1 shows, the claims in our dataset were labeled into 11
domains by our writers during the claim generation process. Figure 2 shows the
performance of the two methods in FEVER Score on claims from different
domains. We discover that both of the methods have lower performance for the
domains with fewer training claims, such as “Sports,” “Technology,” and
“Politics.” We also observe that the number of Wikipedia pages of the evidence
for claims in the “Sports” domain is higher than the numbers of all the other
domains, which may result in lower performance and match our findings in Table
6.

Figure 2: Performance comparisons for the claims of different domains in the
full-pipeline setting. H&S refers to the domain of Humanities & Social
Sciences. Values in the parentheses are the average number of evidence pages.

### Analysis for Claims in Different Lengths

In this section, we analyze the performance of the two methods on claims with
different lengths in the full-pipeline setting. We divide the claims in the
test set into five groups according to the number of characters in the claims
and show the results in Figure 3. We observe that our simple baseline performs
worse on the claims longer than 51 characters, while BEVERS remains stable.
Furthermore, the two methods perform better on the medium length claims with
31-40 characters. These results again indicate that the length of the claims
is an important factor in the claim verification task.

Figure 3: Performance comparisons with different lengths of claims in the
full-pipeline setting. “Ratio” indicates the proportion for each group of the
claims from the test set.

## Discussion

We discuss two limitations of our dataset in this section. First, although
CFEVER is currently the largest Chinese dataset for fact extraction and
verification, it is still much smaller than the English FEVER dataset. Data
size is an important factor for model performance and generalization. We hope
to increase the data scale in the future while maintaining the high quality of
annotation. Second, CFEVER may not be a perfect dataset for training models to
handle complex reasoning tasks. As we show in Table 2, most of the claims in
CFEVER require evidence extracted from only one Wikipedia page. This issue has
also been reported for FEVER (Jiang et al. 2020). We hope to expand our
dataset with more complex claims that require evidence from multiple pages in
the future.

## Conclusion

This paper introduces CFEVER, a new Chinese dataset for Fact Extraction and
VERification. Following the FEVER task (Thorne et al. 2018b), CFEVER forms a
verification task that requires models to verify the claims into “Supports”,
“Refutes”, and “Not Enough Information”. In addition, for the first two
categories, models are also required to extract the evidence sentences from
our fact database composed of Chinese Wikipedia pages. We carefully validate
the quality of the dataset and obtain an inter-annotator agreement of 0.7934
in Fleiss $\kappa$ for the class label consistency. Though the experiments
with the simple baseline designed by ourselves and the state-of-the-art method
(DeHaven and Scott 2023) developed on FEVER (Thorne et al. 2018a), we believe
that CFEVER is a challenging dataset to serve as a benchmark on Chinese claim
verification and fact extraction.

## Acknowledgements

This work was supported by the National Science and Technology Council of
Taiwan, under Grant NSTC 112-2223-E-006-009. We thank the anonymous reviewers
for their insightful comments. We also extend our deepest gratitude to all of
our data annotators for their hard work and dedication during the data
construction process.

## References

  * Aly et al. (2021) Aly, R.; Guo, Z.; Schlichtkrull, M. S.; Thorne, J.; Vlachos, A.; Christodoulopoulos, C.; Cocarascu, O.; and Mittal, A. 2021.  FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
  * Augenstein et al. (2019) Augenstein, I.; Lioma, C.; Wang, D.; Chaves Lima, L.; Hansen, C.; Hansen, C.; and Simonsen, J. G. 2019.  MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 4685–4697. Hong Kong, China: Association for Computational Linguistics.
  * DeHaven and Scott (2023) DeHaven, M.; and Scott, S. 2023.  BEVERS: A General, Simple, and Performant Framework for Automatic Fact Verification.  In _Proceedings of the Sixth Fact Extraction and VERification Workshop (FEVER)_ , 58–65. Dubrovnik, Croatia: Association for Computational Linguistics.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics.
  * Fleiss (1971) Fleiss, J. L. 1971.  Measuring nominal scale agreement among many raters.  _Psychological bulletin_ , 76(5): 378.
  * Friedman (2001) Friedman, J. H. 2001.  Greedy function approximation: a gradient boosting machine.  _Annals of statistics_ , 1189–1232.
  * Guo, Schlichtkrull, and Vlachos (2022) Guo, Z.; Schlichtkrull, M.; and Vlachos, A. 2022.  A Survey on Automated Fact-Checking.  _Transactions of the Association for Computational Linguistics_ , 10: 178–206.
  * Hanselowski et al. (2019) Hanselowski, A.; Stab, C.; Schulz, C.; Li, Z.; and Gurevych, I. 2019.  A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , 493–503. Hong Kong, China: Association for Computational Linguistics.
  * Hanselowski et al. (2018) Hanselowski, A.; Zhang, H.; Li, Z.; Sorokin, D.; Schiller, B.; Schulz, C.; and Gurevych, I. 2018.  UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification.  In _Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)_ , 103–108. Brussels, Belgium: Association for Computational Linguistics.
  * He et al. (2021) He, P.; Liu, X.; Gao, J.; and Chen, W. 2021.  DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION.  In _International Conference on Learning Representations_.
  * Hu et al. (2022) Hu, X.; Guo, Z.; Wu, G.; Liu, A.; Wen, L.; and Yu, P. 2022.  CHEF: A Pilot Chinese Dataset for Evidence-Based Fact-Checking.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 3362–3376. Seattle, United States: Association for Computational Linguistics.
  * Jiang et al. (2020) Jiang, Y.; Bordia, S.; Zhong, Z.; Dognin, C.; Singh, M.; and Bansal, M. 2020.  HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 3441–3460. Online: Association for Computational Linguistics.
  * Kotonya and Toni (2020) Kotonya, N.; and Toni, F. 2020.  Explainable Automated Fact-Checking for Public Health Claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 7740–7754. Online: Association for Computational Linguistics.
  * Ma et al. (2016) Ma, J.; Gao, W.; Mitra, P.; Kwon, S.; Jansen, B. J.; Wong, K.-F.; and Cha, M. 2016\.  Detecting Rumors from Microblogs with Recurrent Neural Networks.  In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence_ , IJCAI’16, 3818–3824. AAAI Press.  ISBN 9781577357704.
  * Malon (2018) Malon, C. 2018.  Team Papelo: Transformer Networks at FEVER.  In _Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)_ , 109–113. Brussels, Belgium: Association for Computational Linguistics.
  * Nie, Chen, and Bansal (2019) Nie, Y.; Chen, H.; and Bansal, M. 2019.  Combining fact extraction and verification with neural semantic matching networks.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, 6859–6866.
  * Nielsen and McConville (2022) Nielsen, D. S.; and McConville, R. 2022.  MuMiN: A Large-Scale Multilingual Multimodal Fact-Checked Misinformation Social Network Dataset.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’22, 3141–3153. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450387323.
  * Nørregaard and Derczynski (2021) Nørregaard, J.; and Derczynski, L. 2021.  DanFEVER: claim verification dataset for Danish.  In _Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)_ , 422–428. Reykjavik, Iceland (Online): Linköping University Electronic Press, Sweden.
  * OpenAI (2023) OpenAI, R. 2023.  GPT-4 technical report.  _arXiv_ , 2303–08774.
  * Ouyang et al. (2022) Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P. F.; Leike, J.; and Lowe, R. 2022.  Training language models to follow instructions with human feedback.  In Koyejo, S.; Mohamed, S.; Agarwal, A.; Belgrave, D.; Cho, K.; and Oh, A., eds., _Advances in Neural Information Processing Systems_ , volume 35, 27730–27744. Curran Associates, Inc.
  * Robertson, Zaragoza et al. (2009) Robertson, S.; Zaragoza, H.; et al. 2009.  The probabilistic relevance framework: BM25 and beyond.  _Foundations and Trends® in Information Retrieval_ , 3(4): 333–389.
  * Schuster, Fisch, and Barzilay (2021) Schuster, T.; Fisch, A.; and Barzilay, R. 2021.  Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 624–643. Online: Association for Computational Linguistics.
  * Soleimani, Monz, and Worring (2020) Soleimani, A.; Monz, C.; and Worring, M. 2020.  BERT for Evidence Retrieval and Claim Verification.  In _Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part II_ , 359–366. Berlin, Heidelberg: Springer-Verlag.  ISBN 978-3-030-45441-8.
  * Stammbach (2021) Stammbach, D. 2021.  Evidence Selection as a Token-Level Prediction Task.  In _Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER)_ , 14–20. Dominican Republic: Association for Computational Linguistics.
  * Thorne et al. (2018a) Thorne, J.; Vlachos, A.; Christodoulopoulos, C.; and Mittal, A. 2018a.  FEVER: a Large-scale Dataset for Fact Extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 809–819. New Orleans, Louisiana: Association for Computational Linguistics.
  * Thorne et al. (2018b) Thorne, J.; Vlachos, A.; Cocarascu, O.; Christodoulopoulos, C.; and Mittal, A. 2018b.  The Fact Extraction and VERification (FEVER) Shared Task.  In _Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)_ , 1–9. Brussels, Belgium: Association for Computational Linguistics.
  * Thorne et al. (2019) Thorne, J.; Vlachos, A.; Cocarascu, O.; Christodoulopoulos, C.; and Mittal, A. 2019\.  The FEVER2.0 Shared Task.  In _Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER)_ , 1–6. Hong Kong, China: Association for Computational Linguistics.
  * Wang (2017) Wang, W. Y. 2017.  “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 422–426. Vancouver, Canada: Association for Computational Linguistics.
  * Williams, Nangia, and Bowman (2018) Williams, A.; Nangia, N.; and Bowman, S. 2018.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 1112–1122. New Orleans, Louisiana: Association for Computational Linguistics.
  * Zaheer et al. (2020) Zaheer, M.; Guruganesh, G.; Dubey, K. A.; Ainslie, J.; Alberti, C.; Ontanon, S.; Pham, P.; Ravula, A.; Wang, Q.; Yang, L.; and Ahmed, A. 2020.  Big Bird: Transformers for Longer Sequences.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., _Advances in Neural Information Processing Systems_ , volume 33, 17283–17297. Curran Associates, Inc.
  * Zhang et al. (2021) Zhang, X.; Cao, J.; Li, X.; Sheng, Q.; Zhong, L.; and Shu, K. 2021.  Mining Dual Emotion for Fake News Detection.  In _Proceedings of the Web Conference 2021_ , WWW ’21, 3465–3476. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450383127.

##  Appendix A Appendix

### Implementation Details

For our baseline, we retrieved the top ten documents for each claim using BM25
in document retrieval; we retrieved the top five sentences as evidence for
each claim in sentence retrieval. For the former, we mainly chose the setting
with the highest recall score, which we provide a comparison in Table 8. For
the latter, we followed the statistics of CFEVER (Table 7) to include
potential evidence for all claims with the top five sentences. To implement
our baseline in the sentence retrieval and RTE stages, we use HuggingFace
Transformers999https://huggingface.co/docs/transformers/ and
PyTorch101010https://pytorch.org (version: 1.13.1+cu117) to build the models.
The hyperparameter sets are present in Table 9 for our baseline. The final
scores were determined based on the performance of the development set using
grid searches.

Top-$k$ | Recall | Precision | F1 Score
---|---|---|---
1 | 69.05 | 75.25 | 72.02
3 | 80.10 | 29.67 | 43.30
5 | 83.85 | 18.73 | 30.62
7 | 86.00 | 13.74 | 23.69
10 | 87.65 | 9.79 | 17.62

Table 8: Document retrieval performance on the test set using BM25 with
different top-$k$ values.

Stage | Hyperparameter | Value
---|---|---
Sent Retrieval | learning rate | {2e-5, 3e-5, 5e-5}
num of epochs | {1, 2}
batch size | 64
negative samples | 50%
warmup ratio | 10% training steps
RTE | learning rate | {3e-5, 5e-5, 7e-5}
num of epochs | {2, 3}
batch size | 32
warmup ratio | 10% training steps

Table 9: Hyperparameters we used for our baseline.

For implementing BEVERS, we followed the instructions and ran their code
available on GitHub111111https://github.com/mitchelldehaven/bevers. To make
fair comparisons between the two baseline systems, we used the same pre-
trained BERT model121212https://huggingface.co/hfl/chinese-bert-wwm-ext in the
sentence retrieval and RTE stages for BEVERS and our baseline. For Stammbach
(Stammbach 2021), we used the official
code131313https://github.com/dominiksinsaarland/document-level-FEVER with the
Chinese BigBird141414https://huggingface.co/Lowin/chinese-bigbird-base-4096
for the sentence retrieval stage.

### Details for ChatGPT

We used the GPT-3.5 Turbo and the GPT-4 Turbo
models151515https://platform.openai.com/docs/models/ with the following
versions for the experiments in Table 4:

  * •

GPT-3.5: gpt-3.5-turbo

  * •

GPT-4: gpt-4-1106-preview

The prompts we used for GPT-3.5 and GPT-4 are shown in Table 10 and Table 11.
Note that we asked GPT-4 to use Chinese Wikipedia for verification in the
prompt, since we found that GPT-4 performed better with the requirement
(underlined in Table 11). Also note that though we provide the translation of
the prompts in Table 10 and Table 11, the English part was not included in the
experiments.

Table 10: Prompt for the zero-shot and few-shot settings with GPT-3.5
(gpt-3.5-turbo). Note that the English part was not included in the
experiments. Table 11: Prompt for the zero-shot and few-shot settings with
GPT-4 (gpt-4-1106-preview). Note that the English part was not included in the
experiments.

### Additional Metrics for the Retrieval Tasks

In the main content, we provide the recall scores for the document and
sentence retrieval results in Table 4 and Table 5. The corresponding precision
and F1 scores are shown in Table 12. We find that BEVERS has lower precision
and F1 scores than our baseline in document retrieval. The reason is that
BEVERS combines the retrieved documents from TF-IDF and the fuzzy string
search system (DeHaven and Scott 2023). The average number of the predicted
pages for the claims in the test set is 67.8 in BEVERS’s document retrieval
results, while we only use the top ten documents for our baseline. Thus,
BEVERS’ precision and F1 scores are lower than our baseline scores in document
retrieval.

Task | System | Recall | Precision | F1
---|---|---|---|---
Doc | Our baseline | 87.65 | 9.79 | 17.62
Retrieval | BEVERSa | 92.60 | 2.29 | 4.47
Sent | Our baseline | 76.65 | 25.36 | 38.11
Retrieval | BEVERSa | 86.60 | 27.00 | 41.17
Sent | Our baseline | 95.90 | 40.26 | 56.71
Retrieval | BEVERSa | 95.20 | 38.98 | 55.31
(Oracle) | Stammbachb | 83.55 | 36.12 | 50.44
Table 12: Scores for the document and sentence retrieval tasks. $a$: DeHaven
and Scott (2023). $b$: Stammbach (2021).


# MQE: Unleashing the Power of Interaction with Multi-agent Quadruped
Environment

Ziyan Xiong∗ 0009-0001-3319-4679 Bo Chen  Shiyu Huang† 0000-0003-0500-0141
Wei-Wei Tu  Zhaofeng He  Yang Gao Ziyan Xiong and Yang Gao are with the
Institute for Interdisciplinary Information Sciences, Tsinghua University,
Beijing, China (e-mail<EMAIL_ADDRESS>gaoyangiiis@mail.tsinghua.edu.cn).Bo Chen and Zhaofeng He are with AI
Institute, Beijing University of Posts and Telecommunications, Beijing, China
(e-mail: {chenbo, zhaofenghe}@bupt.edu.cn).Shiyu Huang is with Zhipu AI,
Beijing 100084, China (e-mail: shiyu.huang@aminer.cn).Wei-Wei Tu is with
4Paradigm Inc., Beijing 100084, China (e-mail: tuweiwei@4paradigm.com).Yang
Gao is with Shanghai Artificial Intelligence Laboratory and Shanghai Qi Zhi
Institute, Shanghai, China.∗Work was done during the intern at 4Paradigm
Inc..†Work was done while at 4Paradigm Inc..

###### Abstract

The advent of deep reinforcement learning (DRL) has significantly advanced the
field of robotics, particularly in the control and coordination of quadruped
robots. However, the complexity of real-world tasks often necessitates the
deployment of multi-robot systems capable of sophisticated interaction and
collaboration. To address this need, we introduce the Multi-agent Quadruped
Environment (MQE), a novel platform designed to facilitate the development and
evaluation of multi-agent reinforcement learning (MARL) algorithms in
realistic and dynamic scenarios. MQE emphasizes complex interactions between
robots and objects, hierarchical policy structures, and challenging evaluation
scenarios that reflect real-world applications. We present a series of
collaborative and competitive tasks within MQE, ranging from simple
coordination to complex adversarial interactions, and benchmark state-of-the-
art MARL algorithms. Our findings indicate that hierarchical reinforcement
learning can simplify task learning, but also highlight the need for advanced
algorithms capable of handling the intricate dynamics of multi-agent
interactions. MQE serves as a stepping stone towards bridging the gap between
simulation and practical deployment, offering a rich environment for future
research in multi-agent systems and robot learning. For open-sourced code and
more details of MQE, please refer to https://ziyanx02.github.io/multiagent-
quadruped-environment/.

###### Index Terms:

quadrupedal locomotion, multi-agent reinforcement learning, hierarchical
reinforcement learning

##  I Introduction

In recent years, the field of robotics has witnessed remarkable advancements
in the locomotion control of quadruped robots. These strides have been largely
propelled by the utilization of deep reinforcement learning (DRL) in
simulation environments [1, 2], enabling quadrupeds to perform a wide array of
tasks with unprecedented agility and efficiency, from trespassing [3, 4, 5]
rugged terrains to carrying out manipulative tasks [6, 7, 8], the capabilities
of current quadrupeds represent a significant leap forward in robotic autonomy
and versatility.

Despite these advancements, the intricacy and variety of real-world tasks
often surpass the capability of single-robot systems. These systems encounter
fundamental constraints in performing tasks in situations such as large-scale
environmental monitoring [9, 10] and intricate logistical challenges within
warehousing environments [11, 12, 13], highlighting the critical need for the
development and deployment of multi-robot systems. Such scenarios demand a
level of coordination, collaboration, and perception that single-robot
systems, no matter how complex they are, cannot achieve on their own.

Addressing the need for coordinated multi-robot systems, the research
community has done much research on multi-agent reinforcement learning in
different multi-agent environments, including the Multi-Agent Particle
Environment (MPE) [14, 15], Google Research Football [16, 17], and the
StarCraft Multi-Agent Challenge (SMAC) [18, 19]. While these environments have
significantly contributed to the advancement of MARL by providing platforms
for developing and evaluating algorithms, we still lack tangible
intermediaries to apply multi-agent policies to real-world tasks. The
simplified or even frictional interactions in these environments often fall
short of replicating the complex interference and dynamics present in real-
world scenarios, such as imperfect action execution caused by direct or
indirect interactions among real robots. These issues highlight a gap between
the current research efforts and practical real-world applications.

Figure 1: Agents learning to herd sheep in hundreds of parallel environments.

In response to these limitations, we propose the Multi-agent Quadruped
Environment to bridge this gap, emphasizing 1) complex and dynamic
interactions between objects and robots, 2) a hierarchical structure of
policies that transforms complex control problems into simple multi-agent
tasks, and 3) realistic and challenging evaluation scenarios of MARL focusing
on either assessing the viability of simplifying complex multi-robot tasks
through the hierarchical structure of policies or exploring robust locomotion
policies amidst the unpredictable interactions among robots.

To explore the capabilities of our environment and the effectiveness of
current MARL methodologies, we experiment with multi-agent reinforcement
learning algorithms, comparing those with a hierarchical structure of policies
intended to streamline the learning against those without such a structure.
Our findings reveal that even advanced algorithms struggle to solve the more
difficult tasks presented in our environment. This underscores not only the
complexity and realism of the challenges we introduce but also the pressing
need for our environment as a tool for developing more sophisticated and
capable RL algorithms. As inspiration for future works, we also showcase
several promising research directions in this environment, i.e. hierarchical
RL and RL enhanced by self-play strategies.

##  II Related Works

###  II-A Reinforcement Learning for Quadruped Control

The exploration of learning-based locomotion control has significantly
advanced the capabilities of quadruped robots, enabling them to perform
dynamic tasks such as robust walking on complex terrains [3, 20, 21], high-
speed running [22, 23, 24, 25, 26], executing parkour abilities [27], climbing
on obstacles [20, 28, 29, 30], and playing football [31, 32, 6]. These
achievements highlight the importance of simulation environments [33, 34] in
facilitating the development of these complex locomotion strategies. Various
simulators such as Gazebo, PyBullet, IsaacGym, and MATLAB have been employed,
each offering different levels of fidelity, ease of use, and computational
efficiency. Notably, IsaacGym, with its GPU-accelerated architecture, stands
out for its enhanced sample efficiency, which is crucial for the rapid
training of robots.

Despite these advancements, the focus has largely been on single-agent
scenarios. However, as tasks become more complex and demand higher levels of
interaction, the need for multi-agent simulation environments becomes evident.
The emergence of multi-robot simulation environments in other domains, such as
drones [35, 36, 37] and dexterous manipulators [38, 39] highlights the
necessity for similar advancements in quadruped robotics. Addressing this, our
work introduces a novel multi-agent quadruped environment, aiming to bridge
this gap and facilitate the exploration of cooperative and competitive
dynamics among robots, thus expanding the scope of quadruped research into
multi-robot operations and opening new avenues for the application of these
technologies.

###  II-B Multi-agent Reinforcement Learning

Multi-agent reinforcement learning has increasingly received much research
interest, propelled by its potential to tackle complex tasks through the
collaborative and competitive interactions of multiple agents. The field
distinguishes between two principal categories of MARL algorithms: value-based
and policy-based. Value-based algorithms, including QMIX [40], VDN[41], and
QPLEX[42], focus on estimating the value of actions in given states to
determine the best course of action. On the other hand, policy-based
algorithms like MADDPG[14], MAPPO[43], and MAT[44] directly learn a policy
that maps states to actions, potentially offering more flexibility in
continuous action spaces.

The development of MARL algorithms is established on the environments used for
training and testing. Current MARL environments, such as MPE and SMAC, often
present simplified, idealized scenarios, lacking complicated dynamics, i.e. in
robotics simulation, to evaluate the capability of MARL algorithms. However,
existing multi-robot environments primarily assess MARL algorithms through
navigation tasks [45, 46, 47], which significantly differ from the simplified
settings commonly employed in MARL research. This discrepancy highlights the
necessity for integrating both kinds of environments.

##  III Multi-agent Quadruped Environment

The Multi-agent Quadruped Environment is developed atop the widely recognized
reinforcement learning framework for legged robots, _legged_gym_ , which
operates on NVIDIA’s Isaac Gym. An overview of MQE can be found in Fig. 1,
accompanied by a comparative analysis of MQE against existing legged robot
simulations in Table I. In addition to the wide range of quadruped models
available through _legged_gym_ , MQE introduces several notable features:

TABLE I: Comparison between Quadruped Environments | Multi-agent | Hierarchical policy | Simulator
---|---|---|---
champ | ✓ | ✓ | Gazebo
Rex-gym | ✓ | ✗ | PyBullet
Genloco | ✗ | ✗ | PyBullet
_legged_gym_ | ✗ | ✓ | IsaacGym
MQE (ours) | ✓ | ✓ | IsaacGym

#### Multi-robot interaction

Our environment is designed to seamlessly incorporate actuated robots,
stationary and manipulable objects, and non-player characters (NPCs) which are
objects that adhere to predetermined policies within each parallel-simulated
environment. The inclusion of multiple robots and interactive objects
intensifies the complexity of learning as each robot, or agent must navigate
interactions with others, necessitating collaboration or evasion to mitigate
the impacts of those interactions.

#### Modular terrain registration

Similar to many other legged robot environments, the simulated ground is
segmented into parallel tracks to facilitate domain randomization of terrain
or to implement curriculum learning strategies. Notably, our environment
further divides each track into blocks, simplifying the creation and
adaptation of terrain for various tasks. These blocks can be efficiently
repurposed and reconfigured to meet diverse requirements.

Figure 2: Demonstration of benchmarking tasks. Blue and green robots are
assigned to accomplish collaborative tasks, while red and orange robots will
play against them. Arrows of different colors illustrate the intended
movements of agents and objects in each task. Generally, tasks demonstrated
lower are harder due to the rising demands of advanced locomotion control and
environment awareness.

#### Alternative hierarchical policy

Recognizing the challenges associated with learning a locomotion policy from
scratch in a multi-agent task setting, our environment incorporates a robust
walking policy as an optional foundational layer. This low-level policy
responds to commands from the high-level policy. This structure allows for the
simplification of some tasks as straightforward multi-agent tasks with complex
movements, given that robot dynamics might not strictly follow the commands
due to potential collisions, sliding, or command alterations.

#### Parallel Simulation

Leveraging the GPU-accelerated capabilities of Isaac Gym, our environment
supports the execution of numerous simulations in parallel. This feature
significantly enhances the sample efficiency of reinforcement learning,
facilitating the achievement of high-performance policy within a constrained
time.

##  IV Benchamarking Tasks

Based on the previously outlined environment, we have designed 12 tasks with
varying degrees of complexity and distinct characteristics for benchmarking
purposes. These tasks are evenly divided into collaborative and competitive
categories, with their details illustrated in Figure 2. The objective of
collaborative tasks includes: 1) requiring agents to adopt asymmetric policies
to achieve their goals, i.e. creating space to prevent potential collisions,
and 2) necessitating cooperation among agents, i.e. passing and stopping a
football. On the other hand, competitive tasks are designed to foster self-
improvement among agents through self-play, as opposed to learning in a
solitary agent setting, i.e. dribbling a football under the opposing
defender’s defense.

Actions are torques of the robot’s actuated Degrees of Freedom (DoFs), or
target positions that will be converted to torques through PD control. To
accelerate the learning of high-level planning, we forgo the extensive
training of a locomotion policy in favor of incorporating a pre-trained walk
policy specifically designed for the Unitree Go1 robot [48]. This advanced
locomotion policy takes 18 dimensions of command and robot states,
subsequently determining the target positions for each actuated DoF, which
enables precise control over the Go1 robot’s movement, ensuring it walks
according to the specified gait and commands. For uniformity and simplicity in
our benchmarking results, we will exclusively utilize the Unitree Go1 robot.

All collaborative benchmarking tasks are formulated as decentralized partially
observable Markov Decision Processes (Dec-POMDPs) with selected observation
and shared rewards, although the simulator itself is fully observable. We
provide all available observations within the simulator to facilitate
privileged observation in actor-critic algorithms or policy distillation.
Proprioception is provided to the aforementioned low-level locomotion policy
and thereby is omitted from the observations delivered to the high-level
policy within the benchmarking tasks. This omission aims to evaluate the
effectiveness of a hierarchical policy framework, where the high-level policy
operates independently from low-level locomotion control.

###  IV-A Collaborative Tasks

#### Narrow Gate

A narrow gate that permits only one robot to go through splits a room.
Positioned on the same side, two robots are instructed to walk through this
gate sequentially, avoiding any collisions. This task necessitates an
understanding of potential collisions and the management of routine to avoid
collisions, requiring different agents to learn asymmetric policies of the
timing and order for optimal rewards.

#### Climb on Seesaw

Positioned next to a suspended platform, a flat seesaw’s design allows one end
to be elevated to the platform’s height, allowing the robot to climb onto the
platform. Due to the mechanism of the seesaw, successful ascension requires
coordination wherein one robot occupies the farther end of the seesaw,
maintaining its stability, while the second robot ascends along the seesaw.
The absence of this synchrony results in the seesaw collapsing.

#### Sheep Dog (Easy)

In _Sheep Dog_ tasks, we introduce sheep as non-player characters (NPCs) that
exhibit autonomous movement based on the following principles: 1) sheep always
maintain a distance from dogs, which are the robots controlled by agents, and
will accelerate to keep away from the dogs, 2) sheep always tends to get close
to the central of the herd, and 3) sheep always move with randomness in
velocity. Similar to _Narrow Gate_ , _Sheep Dog-Easy_ introduces a sheep being
placed on the same side as the dogs. The objective of this task is to guide
the sheep through the gate by utilizing the movement features of sheep.

#### Sheep Dog (Hard)

Building on _Sheep Dog (Easy)_ , _Sheep Dog-Hard_ escalates the challenge by
requiring the dogs to herd nine sheep, significantly increasing the complexity
of the sheep’s movements. This complexity challenges agents’ perception of the
position and movement of sheep and their corresponding ability to really
”understand” the situation. It necessitates the agents follow the sheep’s
behavior patterns rather than relying on the memorization of the specific
trajectories of sheep and dogs.

#### Push Box

Similar to the design of the _Sheep Dog_ tasks, _Push Box_ utilizes the
terrain established in _Narrow Gate_ , with the sheep replaced by a heavy box
that requires the collaborative effort of two robots to push. This task
challenges to navigate and push the box through the gate. Due to the weight of
the box, a dedicated locomotion policy for the pushing task will simplify the
tasks. Furthermore, the task asks for cooperative behavior from both robots to
maintain the box’s trajectory and prevent its rotation.

#### Football (2 vs 1)

We create a football field enclosed by walls which replace sidelines and goals
for _Football_ tasks. A defender, which is a robot serving as an NPC in this
task, is programmed to position itself at the midpoint between the ball and
the goal. Two robots are placed on the opposite side against the defender with
a football. The target of the two robots is to kick the football into the
goal. With the defender maintaining a position between the goal and the ball,
this task not only requires the two dogs to possess dribbling skill but also
necessitates strategic passing to circumvent the defender’s blockades and
successfully score, demanding both refined locomotion policies and effective
cooperative policy formulation.

###  IV-B Competitive Tasks

Figure 3: Learning curves of 6 tasks under 2 settings: results using pre-
trained locomotion policy is illustrated left and results learning from
scratch is illustrated right. Rewards of tasks related to sheep and boxes are
jiggling on a small scale due to the reset of corresponding objects.

#### Push Cylinder

A cylinder centrally placed in an arena can only move sideways. Two quadruped
robots, positioned on opposite sides, compete by pushing the cylinder towards
each other. The robot that pushes the cylinder towards the opponent’s side
earns a reward. This task challenges the robots to identify optimal directions
and positions for pushing, leveraging the cylinder’s circular cross-section
for axial force.

#### Revolving Door

Centered on a wall, a revolving gate permits rotation and passage from both
sides, albeit one at a time. Two robots placed on the same side compete to be
the first to pass through the door by manipulating the door’s rotation and the
wall close to the door. This task demands that the robots understand the
revolving door’s mechanics and strategically counter their opponent’s moves.

#### Sumo Wrestling

Two robots are placed in a sumo wrestling match on a featured platform. The
objective for each robot is twofold: to destabilize and cause their opponent
to fall or to push them out of the designated circle. This task challenges the
robots’ ability to maintain balance when subjected to external forces, as well
as their capability to tactically disrupt the opponent’s movement without
compromising their own stability.

#### Traverse Bridge

With the intention of crossing a narrow bridge to the other side, two robots
are placed at opposite ends. The bridge’s limited width permits only one robot
to pass at a time. Thus, the primary objective for each robot is to push its
opponent off the bridge to secure passage, asking the robots to make use of
every inch of space in maintaining equilibrium while attempting to displace
the opposing robot.

#### Football (1 vs 1)

On the football field introduced in _Football (2 vs 1)_ , we ask two robots to
compete for control of the football and try to goal, requiring the robots with
capabilities to dribble, tackle and shoot. Goals only happen when there is a
huge strength gap between two robots.

#### Football (2 vs 2)

By increasing the number of robots on both sides, _Football (1 vs 1)_ becomes
a simple football match. The target of each side is simply winning the match
by a goal, utilizing the ability of dribbling, passing, tackling, shooting and
cohesive team collaboration. There are many complex strategies in a 2 vs 2
football game, making this task a challenge for both locomotion control and
strategies.

##  V Experiments

With the help of the pre-trained locomotion policy, our environment serves as
a fair comparative basis for different MARL algorithms, as we get rid of the
possible variance coming from different locomotion policies and focus fully on
the planning capability provided by MARL algorithms. In the following
sections, we showcase the performance of different multi-agent reinforcement
learning algorithms using the 6 collaborative tasks with and without the pre-
trained locomotion policy.

###  V-A Simulation Performance

We select 4 tasks with different objects to illustrate the efficient
simulation capabilities of our environment, which are _Narrow Gate_ , _Push
Box_ , _Sheep Dog (Easy)_.

TABLE II: Simulation Performance of Our Environment in FPS # Envs |  _Narrow Gate_ ∗ |  _Climb on Seesaw_ ∗ |  _Sheep Dog (Easy)_ ∗
---|---|---|---
2 agents | 2 agents & 1 object | 2 agents & 1 NPC
500 | $18307\pm 154$ | $14083\pm 63$ | $16164\pm 68$
1000 | $27720\pm 80$ | $22856\pm 80$ | $25129\pm 82$
∗with the pre-trained locomotion policy running. |

Thanks to Isaac Gym’s high-performance parallel simulating capabilities, our
environment can achieve $10000+$ frames per second with pre-trained locomotion
policy running, which contains two MLPs with 4 layers and 3 layers. Means and
standard deviations of FPS (frame per second) are shown in Table II, and all
the results are obtained on a server with NVIDIA RTX3090, CUDA 11.4, and Isaac
Gym Preview 4. The high speed of simulation extremely accelerates the learning
process in the wall-clock time, with all data processes accomplished on GPUs.

###  V-B Environment Setup

Currently, we evaluate the performance of MAPPO, JRPO, and MAT implementation
in OpenRL [49] on all collaborative tasks. Reward specifications are listed in
Table III, while two settings of actions are 1) commands of horizontal
velocity and rotation velocity of yaw or 2) target positions of actuated DoFs,
aiming to simulate two reinforcement learning settings: 1) utilizing pre-
trained locomotion policy and hierarchical structure to fully focus on the
high-level collaboration, and 2) learning from scratch. Subtle rewards
designed for learning a walking policy are excluded because such rewards
require sampling of target velocity [48] that is impossible to achieve in a
task-oriented environment. We train each algorithm in 500 parallel
environments for 40 Million environment steps with 5 different random seeds.

###  V-C Quantitative Results

The performance of each algorithm on different tasks is shown in Figure 3. By
comparing the results under two settings, it’s undeniable that hierarchical
structure greatly simplifies tasks that can be solved by utilizing pre-trained
locomotion policy through high-level commands, i.e. _Narrow Gate_ and _Climb
on Seesaw_ which only requires different agents to simply walk around, but
it’s impossible to solve tasks that demand more specialized locomotion
policies like _Push Box_ and _Football-Defender_ that, intuitively, requires a
policy for pushing instead of walking and policies for dribbling, passing and
stopping footballs to successfully solve the tasks. And the poor performance
of RL from scratch demonstrates that end-to-end learning in multi-robot tasks
is impossible. Because of the intricate NPC introduced in _Sheep Dog_ tasks,
MAT achieves notably better performance compared to JRPO and PPO, illustrating
that environment awareness takes an important role in realistic multi-agent
settings.

According to the results, we can divide the 6 collaborative tasks into 3
categories with rising complexity: 1) tasks that require the agents to
perceive other agents’ positions and collaborate according to the positions,
including _Narrow Gate_ and _Climb on Seesaw_ ; 2) tasks that require the
agents to perceive both other agents’ positions and have a general knowledge
of how the environment will change according to all agents’ involvement, i.e.
_Sheep Dog-Easy_ and _Sheep Dog-Hard_ ; and 3) tasks that demand specialized
locomotion policy as well as collaboration capabilities required by previous 2
categories, including _Push Box_ and _Football-Defender_. The different
complexity between categories implies that a more sophisticated method of
hierarchical RL other than simply outputting state-unrelated commands is
necessary.

##  VI Conclusion and Future Work

In this paper, we introduced the Multi-agent Quadruped Environment, our new
reinforcement learning environment that supports the convenient design of
multi-agent tasks for quadruped robots, along with twelve tasks predefined,
covering cooperative, adversarial, low-level control, and high-level planning
learning. Leveraging the powerful parallel computing capabilities of IsaacGym,
our environment is capable of simulating hundreds of environments in parallel,
significantly accelerating the process of reinforcement learning. Within our
environment, we explored the feasibility of hierarchical RL in the control of
quadruped robots and easily completed some cooperative tasks with the help of
a low-level locomotion policy. This demonstrates the viability of hierarchical
RL in the field of quadruped robots and its relative speed advantage over end-
to-end learning. However, the naive hierarchical methods we employed were
insufficient for solving complex tasks, prompting us to explore and discuss
some viable future work here.

_a) Environment awareness_ plays a critical role in tasks featuring with
complex dynamics. However, the lack of observations in the real world as well
as the difficulty in understanding dynamics, highlights the importance of
enhancing environment perception and prediction capabilities in multi-agent
tasks, marking it as a promising area for future research.

_b) Varied low-level policies_ will greatly simplify the process of solving
any task through a hierarchical structure. We aspire to see an increasing
number of low-level policies proven to be effective in the hierarchical
structure as well as novel tasks demanding higher agility of quadrupeds.

## Appendix

Rewards for collaborative tasks are listed in Table III, where quadratic terms
will be labeled by 2 and exponential terms will be labeled by $\exp$.

TABLE III: Reward Specification Task | Description of Reward | Scale
---|---|---
_Narrow Gate_ | agent walked through the gate | 5.
decreased distance to the gate | 1.
distance between agents (clipped)2 | 0.025
| collision penalty | -2
_Climb on Seesaw_ | agent reached the destination | 10.
height of agent | 1.
agent’s movement along the seesaw | 5
agent’s distance from the seesaw2 | -0.5
collision penalty | -2.
falling penalty | -2.
distance between agents (clipped)2 | 0.25
_Sheep Dog (Easy)_ | sheep moved through the gate | 1.
decreased sheep’s distance to the gate | 2.
_Sheep Dog (Hard)_ | sheep moved through the gate | 1.
$\exp$(-sheep’s distance to the gate) | 1.
_Push Box_ | box moved through the gate | 1.
decreased box’s distance to the gate | 10.
_Football (2 vs 1)_ | goal | 10.
$\exp$(distance between ball and the goal) | 3.

There are two kinds of rewards: 1) rewards calculated based on change of
states, i.e. Distance to Target Reward based on the decreased distance between
agents and a specified target point behind the gate in _Narrow Gate_ , are
designed to guide the agent at the beginning stage of learning as they are
easy to acquire, and 2) rewards calculated based on states, i.e. Success
Reward in _Narrow Gate_ provided when the agent has crossed the gate, are
always closely related to success states of the tasks and result in much
higher returns as they are provided repeatedly. Intuitively, the agent will
first follow the change-of-state-based rewards, which may not provide an
optimal guide to the target state, to discover the state space including
success states, and then the state-based rewards illustrating the success
states will dominate the learning. Figure 4 shows the effectiveness of the
first kind of reward at the beginning stage and then the domination of the
second kind of reward.

Figure 4: Learning curves of two selected rewards in PPO learning of _Narrow
Gate_ within first 20M environment steps (average of 5 seeds).

## References

  * [1] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa _et al._ , “Isaac gym: High performance gpu-based physics simulation for robot learning,” _arXiv preprint arXiv:2108.10470_ , 2021.
  * [2] N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in _Conference on Robot Learning_. PMLR, 2022, pp. 91–100.
  * [3] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” _Science robotics_ , vol. 5, no. 47, p. eabc5986, 2020.
  * [4] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” _Science Robotics_ , vol. 7, no. 62, p. eabk2822, 2022.
  * [5] S. Gai, S. Lyu, H. Zhang, and D. Wang, “Continual reinforcement learning for quadruped robot locomotion,” _Entropy_ , vol. 26, no. 1, p. 93, 2024.
  * [6] Y. Ji, Z. Li, Y. Sun, X. B. Peng, S. Levine, G. Berseth, and K. Sreenath, “Hierarchical reinforcement learning for precise soccer shooting skills using a quadrupedal robot,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 1479–1486.
  * [7] Y. Ji, G. B. Margolis, and P. Agrawal, “Dribblebot: Dynamic legged manipulation in the wild,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 5155–5162.
  * [8] Z. Fu, X. Cheng, and D. Pathak, “Learning a unified policy for whole-body control of manipulation and locomotion,” in _6th Annual Conference on Robot Learning_ , 2022.
  * [9] A. Nayyar, V. Puri, N. G. Nguyen, and D. N. Le, “Smart surveillance robot for real-time monitoring and control system in environment and industrial applications,” in _Information Systems Design and Intelligent Applications: Proceedings of Fourth International Conference INDIA 2017_. Springer, 2018, pp. 229–243.
  * [10] T. N. Nguyen, T. B. Nguyen, T. Van Chien, and T. H. Nguyen, “Utilizing deep reinforcement learning to control uav movement for environmental monitoring,” _International Journal of Electrical and Electronic Engineering & Telecommunications_, vol. 12, no. 5, pp. 317–325, 2023.
  * [11] A. P. Pandian, “Artificial intelligence application in smart warehousing environment for automated logistics,” _Journal of Artificial Intelligence_ , vol. 1, no. 02, pp. 63–72, 2019.
  * [12] S. Li, W. Guo _et al._ , “Supervised reinforcement learning for ulv path planning in complex warehouse environment,” _Wireless Communications and Mobile Computing_ , vol. 2022, 2022.
  * [13] H. Lee and J. Jeong, “Mobile robot path optimization technique based on reinforcement learning algorithm in warehouse environment,” _Applied sciences_ , vol. 11, no. 3, p. 1209, 2021.
  * [14] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [15] K. Liu, Y. Zhao, G. Wang, and B. Peng, “Sa-matd3: Self-attention-based multi-agent continuous control method in cooperative environments,” _arXiv preprint arXiv:2107.00284_ , 2021.
  * [16] K. Kurach, A. Raichuk, P. Stanczyk, M. Zajkac, O. Bachem, L. Espeholt, C. Riquelme, D. Vincent, M. Michalski, O. Bousquet _et al._ , “Google research football: A novel reinforcement learning environment,” _arXiv preprint arXiv:1907.11180_ , 2019.
  * [17] F. Lin, S. Huang, T. Pearce, W. Chen, and W.-W. Tu, “Tizero: Mastering multi-agent football with curriculum learning and self-play,” _arXiv preprint arXiv:2302.07515_ , 2023.
  * [18] M. Samvelyan, T. Rashid, C. S. De Witt, G. Farquhar, N. Nardelli, T. G. Rudner, C.-M. Hung, P. H. Torr, J. Foerster, and S. Whiteson, “The starcraft multi-agent challenge,” _arXiv preprint arXiv:1902.04043_ , 2019.
  * [19] M. Kim, J. Oh, Y. Lee, J. Kim, S. Kim, S. Chong, and S.-Y. Yun, “The starcraft multi-agent challenges+: Learning of multi-stage tasks and environmental factors without precise reward functions,” _arXiv preprint arXiv:2207.02007_ , 2022.
  * [20] A. Agarwal, A. Kumar, J. Malik, and D. Pathak, “Legged locomotion in challenging terrains using egocentric vision,” in _Conference on robot learning_. PMLR, 2023, pp. 403–415.
  * [21] V. Tsounis, M. Alge, J. Lee, F. Farshidian, and M. Hutter, “Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 3699–3706, 2020.
  * [22] Y. Jin, X. Liu, Y. Shao, H. Wang, and W. Yang, “High-speed quadrupedal locomotion by imitation-relaxation reinforcement learning,” _Nature Machine Intelligence_ , vol. 4, no. 12, pp. 1198–1208, 2022.
  * [23] X. B. Peng, E. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine, “Learning agile robotic locomotion skills by imitating animals,” _arXiv preprint arXiv:2004.00784_ , 2020.
  * [24] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” _Science Robotics_ , vol. 4, no. 26, p. eaau5872, 2019.
  * [25] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal, “Rapid locomotion via reinforcement learning,” _The International Journal of Robotics Research_ , p. 02783649231224053, 2022.
  * [26] M. Hutter, C. Gehring, D. Jud, A. Lauber, C. D. Bellicoso, V. Tsounis, J. Hwangbo, K. Bodie, P. Fankhauser, M. Bloesch _et al._ , “Anymal-a highly mobile and dynamic quadrupedal robot,” in _2016 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2016, pp. 38–44.
  * [27] X. Cheng, K. Shi, A. Agarwal, and D. Pathak, “Extreme parkour with legged robots,” _arXiv preprint arXiv:2309.14341_ , 2023.
  * [28] P. Fankhauser, M. Bjelonic, C. D. Bellicoso, T. Miki, and M. Hutter, “Robust rough-terrain locomotion with a quadrupedal robot,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 5761–5768.
  * [29] H. Lee, Y. Shen, C.-H. Yu, G. Singh, and A. Y. Ng, “Quadruped robot obstacle negotiation via reinforcement learning,” in _Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006._ IEEE, 2006, pp. 3003–3010.
  * [30] M. P. Austin, M. Y. Harper, J. M. Brown, E. G. Collins, and J. E. Clark, “Navigation for legged mobility: dynamic climbing,” _IEEE Transactions on Robotics_ , vol. 36, no. 2, pp. 537–544, 2019.
  * [31] F. B. Martins, M. G. Machado, H. F. Bassani, P. H. M. Braga, and E. S. Barros, “rsoccer: A framework for studying reinforcement learning in small and very small size robot soccer,” 2021.
  * [32] JiDi, “Jidi olympics football,” https://github.com/jidiai/ai_lib/blob/master/env/olympics_football.py, 2022.
  * [33] N. Heess, D. Tb, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. Eslami _et al._ , “Emergence of locomotion behaviours in rich environments,” _arXiv preprint arXiv:1707.02286_ , 2017.
  * [34] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine, “Learning to walk via deep reinforcement learning,” _arXiv preprint arXiv:1812.11103_ , 2018.
  * [35] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” in _Field and Service Robotics: Results of the 11th International Conference_. Springer, 2018, pp. 621–635.
  * [36] Y. Song, S. Naji, E. Kaufmann, A. Loquercio, and D. Scaramuzza, “Flightmare: A flexible quadrotor simulator,” in _Conference on Robot Learning_. PMLR, 2021, pp. 1147–1157.
  * [37] W. Koch, R. Mancuso, R. West, and A. Bestavros, “Reinforcement learning for uav attitude control,” _ACM Transactions on Cyber-Physical Systems_ , vol. 3, no. 2, pp. 1–21, 2019.
  * [38] Y. Chen, Y. Geng, F. Zhong, J. Ji, J. Jiang, Z. Lu, H. Dong, and Y. Yang, “Bi-dexhands: Towards human-level bimanual dexterous manipulation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [39] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _2012 IEEE/RSJ international conference on intelligent robots and systems_. IEEE, 2012, pp. 5026–5033.
  * [40] T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson, “Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 4295–4304.
  * [41] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls _et al._ , “Value-decomposition networks for cooperative multi-agent learning,” _arXiv preprint arXiv:1706.05296_ , 2017.
  * [42] J. Wang, Z. Ren, T. Liu, Y. Yu, and C. Zhang, “Qplex: Duplex dueling multi-agent q-learning,” _arXiv preprint arXiv:2008.01062_ , 2020.
  * [43] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of ppo in cooperative multi-agent games,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 24 611–24 624, 2022.
  * [44] M. Wen, J. G. Kuba, R. Lin, W. Zhang, Y. Wen, J. Wang, and Y. Yang, “Multi-agent reinforcement learning is a sequence modeling problem,” _arXiv preprint arXiv:2205.14953_ , 2022.
  * [45] L. Wen, Y. Liu, and H. Li, “Cl-mapf: Multi-agent path finding for car-like robots with kinematic and spatiotemporal constraints,” _Robotics and Autonomous Systems_ , vol. 150, p. 103997, 2022.
  * [46] X. Yang, S. Huang, Y. Sun, Y. Yang, C. Yu, W.-W. Tu, H. Yang, and Y. Wang, “Learning graph-enhanced commander-executor for multi-agent navigation,” _arXiv preprint arXiv:2302.04094_ , 2023.
  * [47] H. Surmann, C. Jestel, R. Marchel, F. Musberg, H. Elhadj, and M. Ardani, “Deep reinforcement learning for real autonomous mobile robot navigation in indoor environments,” _arXiv preprint arXiv:2005.13857_ , 2020.
  * [48] G. B. Margolis and P. Agrawal, “Walk these ways: Tuning robot control for generalization with multiplicity of behavior,” in _Conference on Robot Learning_. PMLR, 2023, pp. 22–31.
  * [49] S. Huang, W. Chen, Y. Sun, F. Bie, and W.-W. Tu, “Openrl: A unified reinforcement learning framework,” _arXiv preprint arXiv:2312.16189_ , 2023.