# Auto Arena of LLMs: Automating LLM Evaluations with Agent Peer-battles and
Committee Discussions

Ruochen Zhao1,2 , Wenxuan Zhang2, Yew Ken Chia2,3 , Deli Zhao2, Lidong Bing2
1 Nanyang Technological University, Singapore
2 DAMO Academy, Alibaba Group; 3 Singapore University of Technology and Design
<EMAIL_ADDRESS>
{saike.zwx, yewken.chia, deli.zdl<EMAIL_ADDRESS>

https://auto-arena.github.io/ Work done while the author was an intern at DAMO
Academy, Alibaba. Yew Ken Chia is under the Joint Ph.D. Program between DAMO
Academy and SUTD.

###### Abstract

As LLMs evolve on a daily basis, there is an urgent need for a trustworthy
evaluation method that can provide robust evaluation results in a timely
fashion. Currently, as static benchmarks are prone to contamination concerns,
users tend to trust human voting platforms, such as Chatbot Arena. However,
human annotations require extensive manual efforts. To provide an automatic,
robust, and trustworthy evaluation framework, we innovatively propose the
Auto-Arena of LLMs, which automates the entire evaluation process with LLM
agents. Firstly, an examiner LLM devises queries. Then, a pair of candidate
LLMs engage in a multi-round peer-battle around the query, during which the
LLM’s true performance gaps become visible. Finally, a committee of LLM judges
collectively discuss and determine the winner, which alleviates bias and
promotes fairness. In our extensive experiment on the 17 newest LLMs, Auto-
Arena shows the highest correlation with human preferences, providing a
promising alternative to human evaluation platforms.111 We release the code at
https://github.com/Auto-Arena/Auto-Arena-LLMs.

##  1 Introduction

Since ChatGPT and GPT-4 [23] gained popularity, large language models (LLMs)
have risen to the forefront of technological innovation, capturing broad
industry and social interests. This enthusiasm has spurred numerous
organizations to release their own LLMs [27, 26]. However, the rapid pace at
which these models are released and updated poses a significant challenge for
users attempting to understand their capabilities and monitor their evolution.
Consequently, there has been a pressing demand for comprehensively evaluating
LLMs recently.

One line of research conducts automatic evaluation with static datasets. Among
these, static datasets with predefined metrics, such as GSM8k [9] and MMLU
[15], are constructed with aspect-specific input-output pairs, such as
questions and their corresponding answers. Given the questions, the LLM-
produced answers are compared to ground-truth answers using metrics such as
accuracy. However, these approaches suffer significantly from contamination
concerns [24], where models may have been inadvertently exposed to elements of
the test datasets during training, thereby skewing evaluation results. The
rigid ground-truth answers also limit their utility in assessing models’
performance on general or open-ended questions. Model-based evaluation, such
as MT-Bench [32] and AlpacaEval [12], provides an alternative for evaluating
LLMs on open-ended questions. These methods typically ask two models to
generate responses to the same open-ended question and then employ a strong
judge model (e.g., GPT-4) to choose the better response. However, the static
question sets still pose the issue of contamination. Additionally, the
assumption of the existence of a strong judge model makes the evaluation
framework less generalizable.

Aside from automated evaluations, human assessment, although requiring
significant manual efforts, remains the gold standard for most users. A
noticeable example is Chatbot Arena [32], which is a crowdsourced voting
platform that gathers anonymous votes on LLM performances and calculates ELO
scores to rank these models. The resulting leaderboard222
https://leaderboard.lmsys.org/ is widely considered a trustworthy indicator of
an LLM’s general capabilities. However, a reliable model evaluation on this
platform must be supported by more than 10k human votes, which requires
considerable time and effort. Consequently, when newly developed models enter
the scene, they often struggle to quickly amass a significant number of votes.
Moreover, this strong reliance on human votes limits its application in
various scenarios. For example, the performance of non-English languages is
difficult to estimate, as most of the queries on the platform are in English.
The completely open participation may also result in uneven evaluation
quality.

Figure 1: An illustration of Auto Arena of LLMs.

To provide automatic, reliable, and human-like LLM evaluations, we propose
Auto Arena of LLMs (Auto-Arena), a framework that automates the whole LLM
evaluation process with LLM agents. As shown in Figure 1, the Auto-Arena
framework consists of three stages. Firstly, an LLM examiner agent is tasked
with generating questions, mimicking real-life users inputting queries.
Secondly, two candidate LLMs interact with each other and engage in a multi-
round peer battle by answering the seed question individually, criticizing the
opponent’s weaknesses, and raising targeted follow-up queries to challenge the
opponent further. During the multi-round battle process, the LLM’s true
capabilities are drawn out and performance gaps become visible. Lastly, a
committee of LLM judge agents collectively discusses and evaluates the ability
of the two candidates, mimicking the human voting process. By automating the
entire evaluation process with LLM agents, we alleviate data contamination
concerns in static datasets, reduce single-model bias with collective
decision-making, and avoid long wait times for new models entering the human
voting platform.

To verify the evaluation framework, we run an extensive experiment with 17
models. Compared to static and model-based benchmarks, Auto-Arena increases
the Spearman correlation with human preferences by 4.5%, resulting in state-
of-the-art alignment. Before and after peer battles, the Spearman correlation
with human preferences increases by 46.4%, verifying our hypothesis the peer
battles can better display performance gaps. Before and after committee
discussions, committee agreement increases by 20%, showcasing the
effectiveness of the committee discussion mechanism. By studying the peer
battles, we discover intriguing LLM agent behaviors such as self-improvement
and competitive actions. Using Chinese as an example, we also show that such
an automated framework is easily extendable to non-mainstream languages and
domains. Along with the project, we release a leaderboard, a website with
demos, and the codebase. We are actively maintaining the leaderboard and
altering the questions to provide reliable results in a timely manner.

##  2 Related Work

Since LLMs demonstrated general and zero-shot capabilities, researchers have
been attempting to use one single strong LLM to judge other LLMs’ outputs.
LLM-as-a-judge [32] introduces Chatbot Arena and MT-Bench, a multi-turn
question set to mimic real-life user queries. Using a strong LLM judge, such
as GPT-4, to evaluate the MT-Bench results achieves over 80% agreement with
human preferences, showcasing the potential of such methods. Besides such
usefulness evaluations, some attempt to use LLM-as-a-judge to examine the
knowledge boundaries of LLMs. Language-Model-as-an-Examiner [2] constructs the
LMExamQA dataset, which comprises of LM-generated, knowledge-intensive
questions covering three cognitive levels, including knowledge memorization,
knowledge comprehension, and knowledge analysis. These questions are required
to be answered by the LLM itself correctly, limiting their complexity. Then,
the LM evaluator and the candidate interact in a series of follow-up queries
and the evaluator rates the responses on dimensions including accuracy and
factuality. Multiple evaluators examine the model independently and vote for a
final result. KIEval [29] also incorporates an LLM-powered “interactor” role
to examine deep comprehension of knowledge, which is shown to mitigate
contamination issues. By utilizing the LLMs to generate questions, these
methods can effectively mitigate the contamination concerns on static
datasets. However, such evaluations only examine the LLMs’ capabilities on
answering knowledge-intensive, closed-domain, and simpler questions, as they
rely on the examiner’s internal knowledge. They also require the examiner to
interact with each candidate individually, creating computational overheads
and limiting the scope of queries.

Moreover, using a single LLM judge still suffer from a bias towards LLM-
generated summaries [20], inflated scores in multilingual evaluation [14],
verbosity bias [12], and difficulties when evaluating candidates with close
performance [25]. To mitigate these problems, some recent works propose using
multiple agents for collaborative evaluation, mimicking a peer review process.
Some works simulate different personas with the same LLM. DRPE [28] uses
multi-roleplayer prompting to mimic different roles with the same base model
and integrate multiple outputs as votes for the final results. ChatEval [6]
simulates different evaluator personas with the same base model to engage in
debates, reaching a final evaluation result. While these methods come from the
same motivation as our committee discussion component, they are still built on
existing static datasets and do not mitigate the model-specific bias as they
still use the same LLM to simulate different personas. To further incorporate
different LLMs as evaluators, PRD [17] allows 2 LLMs to discuss an evaluation
and assigns higher voting weights to the LLM reviewers with stronger
capabilities. Peer-review-in-LLMs [22] further optimizes the voting weights as
a learnable parameter. WideDeep [30] sets up a multi-layer neural network
structure where each LLM evaluator functions as neurons. PRE [8] selects a
small group of reviewers to produce evaluations individually and then asks a
“chair” to produce the final evaluation using the aggregated evaluations. They
show that the multi-agent approach effectively mitigates single-model bias.
This line of work is similar to our “LLM judge committee”. However, they are
still limited on static datasets and specific domains. Moreover, they lack the
collective discussion component, which is shown to improve agreement
significantly.

Compared to using LLMs for evaluations, LLM-debate is a relatively new
research area. Cohen et al. [10] shows that LLM interactions and cross-
examination can effectively discover factual errors. Debate [11] shows that
multi-agent debate can improve factuality and reasoning abilities. MAD [19]
shows that LLM debate can encourage divergent thinking, which is helpful for
tasks that require deep levels of contemplation. Khan et al. [16] shows that
even non-expert weak LLMs can supervise expert LLMs if we allow the two LLM
experts to engage in debates. Moreover, Zhao et al. [31] and Gu et al. [13]
show interesting case studies where LLMs are engaged in simulated competitive
environments, where they demonstrate human-like strategies.

##  3 The Auto Arena Framework

The Auto-Arena framework consists of three stages: question generation, peer
battles, and committee discussions. These three stages are run sequentially
and fully automated with LLM-based agents. The overall design of our Auto-
Arena framework is presented in Figure 1. All prompts are included in Appendix
A.1.

###  3.1 Question Generation

For debate questions, as using a static dataset could incur data contamination
concerns and result in unfair evaluations, we ask an LLM examiner agent to
dynamically generate questions. The examiner agent could be any capable LLM.
As the question simply serves as a starting point for the peer battle, which
is the primary focus of evaluation, the question quality has relatively less
impact on the overall evaluation framework. Similar to MT-Bench [32], we
utilize 8 common categories in real-life conversations: writing, roleplay,
extraction, reasoning, math, coding, STEM knowledge, and humanities/social
science knowledge. In the prompt, the examiner is provided with a sample
question and encouraged to generate diverse and difficult questions to ensure
the depth and width of the debates examined. A subset of the generated
questions is shown in Appendix A.2.

Specifically, as the examiner agent will also participate in the following
debates, previous methods [2] could incur self-enhancement bias as the
examiner agents only devise questions that they are confident about. We try to
avoid such bias with the following two designs: 1\. We do not disclose to the
examiner that it will participate in this tournament. 2. We do not ask the
examiner to only generate questions that it can solve. Thus, the resulting
questions should not cause much enhancement bias for the examiner agent
itself.

###  3.2 Peer Debate

Figure 2: The process of a Lincoln-Douglas-style peer-battle with the actions
used.

After the questions are drafted, we conduct peer battles around these
questions among the candidate LLMs. In one peer battle, two candidate LLMs
debate around the given question, point out the opponent’s weaknesses, and
raise follow-up questions that the opponent may fail to respond to.

In the peer battle, each candidate LLM can utilize 4 types of actions: 1.
Think, where the candidate thinks about the question and plans its strategy.
This action is hidden to the opponent at all times. 2. Respond, where the
candidate responds to the aforementioned question. 3. Criticize, where the
candidate points out the loopholes and mistakes in the opponent’s previous
responses. 4. Raise, where the candidate raises follow-up questions that are
specifically designed to make the opponent expose its weaknesses.

The peer battle is designed according to the Lincoln-Douglas debate format 333
https://en.wikipedia.org/wiki/Lincoln-Douglas_debate_format, which is a
competitive one-to-one debate used for popular competitions such as National
Speech and Debate Association competitions. Overall, the peer battle consists
of 3 rounds, where the candidates take turns to speak. The entire dialogue
history is visible to both candidates. The process is illustrated in Figure 2.
In the first round, A gives an initial response to the examiner’s question; B
criticizes the weaknesses in A’s response and raises a targeted follow-up
question; and A responds to B’s question. In the second round, A and B are
reversed: B gives an initial response to the examiner’s question (without
seeing A’s response); A criticizes and raises questions; and B responds to A’s
question. In the third round, A and B cross-examine each other. A starts by
criticizing B’s previous loopholes and raises follow-up questions. After
responding, B also criticizes A’s loopholes and raises questions. A concludes
the battle by responding again. In this process, both A and B get an equal
number of each action to ensure fairness. To further reduce position bias, A
and B’s order is randomly shuffled at the beginning of each debate.

Depending on which turn it is, we provide an action guide to the candidate,
specifying the objectives and corresponding actions for this turn. Similar to
human debate competitions, we time the candidates by providing a maximum
length, which is also specified in the prompts. Any responses beyond the
required length will be cut off. This design also mitigates verbosity bias in
LLM-as-a-judge [32], where LLM judges prefer longer and more verbose
responses.

###  3.3 Committee Discussions

After the peer battle takes place, a committee of LLM judge agents will
collectively determine the winner. The committee is always selected as the
best five LLMs according to the current ranking. In the first round, the
committee is initialized with MMLU [15] scores to approximate LLM
performances. They will first be asked to read through the battle history,
elaborate judgment reasons, and give a verdict on whether A is better, or B is
better, or if there is a tie.

After the initial judgments are formed, the committee engages in a discussion.
Each judge will read the other judge’s verdicts and decide whether to adjust
the ratings and elaborate on the reasons. The collective intelligence
introduces diverse viewpoints, improves judgment quality, and mitigates
single-model bias. Finally, the winning candidate is decided by the majority
voting of the discussed judgments.

For logical-reasoning questions that have ground-truth answers (reasoning,
code, math), LLM-as-a-judge is known to show weak performances in judging the
quality of responses. We adopt prior approaches to establish the reference-
based judge [32]. Specifically, we utilize the strongest model (according to
the current ranking) to generate a reference answer and provide it to the
judge when evaluating the peer battle.

##  4 Using Auto-Arena to Derive Trustworthy Rankings

###  4.1 Experimental Setup

##### Model Selection:

For the main experiment, we first select 7 models from the top 30 list on
Chatbot Arena with more than 10k votes each. The models are selected to be the
best or the latest models from each popular model family at the time of
experiments. To construct a comprehensive
leaderboard444https://huggingface.co/spaces/Auto-Arena/Leaderboard, we further
consider 10 more models.

##### Baselines:

For the baselines, we consider widely used evaluation benchmarks, including
static datasets with fixed metrics and model-based metrics.

  1. 1.

Static datasets with fixed metrics

    1. (a)

MMLU (Massive Multitask Language Understanding) [15], an extensive benchmark
dataset that covers 57 subjects and tests both world knowledge and problem-
solving ability.

    2. (b)

OpenLLM Leaderboard [3], a leaderboard calculated by averaging performance
metrics on 6 key benchmarks, including MMLU.

  2. 2.

Static datasets with model-based metrics

    1. (a)

MT-Bench [32], a multi-turn question set consisting of 80 questions. Model
responses are graded by GPT-4.

    2. (b)

MT-Bench Hard [18], a benchmark dataset with 1,000 challenging user queries
collected on Chatbot Arena. Model responses are graded by GPT-4.

    3. (c)

Length-controlled AlpacaEval [12], an automatic benchmark based on the
AlpacaFarm evaluation set, which tests the ability of models to follow general
user instructions. Model responses are graded by GPT-4.

##### Setup:

Among the 7 participants, we conduct a swiss-style tournament: For $n$
participants, instead of pairing each participant with $(n-1)$ others, a
swiss-tournament pairs each player with $log_{2}(n)$ players of similar
rankings without repeats. This design effectively reduces computational costs
from $O(n^{2})$ to $O(nlog_{2}(n))$.

Each pair of candidates engage in 40 peer battles, with 5 questions from each
of the 8 categories. The questions are generated by GPT-4. As each battle
consists of 3 rounds (each candidate speaks for 4 times), we expect the
competition scale to be approximately the same as MT-Bench (80 questions, each
candidate speaks twice).

In the tournament, the rating scores are provided by calculating the ELO
rating system [1, 4], which has become the standard practice in competitive
games such as chess. Similar to the Chatbot Arena score calculation procedure
[7], we compute the Bradley-Terry (BT) coefficients [5] for better statistical
estimation.

We initialize the Swiss tournament rankings according to MMLU scores, which is
a static approximate of model performances. At the end of each pairing, we re-
calculate ELO scores of current models. The committee is selected to be the
remaining 5 models (besides the candidates). When the participant number
increases, the committee is selected as the best 5 LLMs based on current ELO
rankings. After the initial judgments, the committee members engage in one
round of discussion.

###  4.2 Results: Alignment with Human Preferences

Table 1: Correlation analysis on evaluation benchmarks on 7 LLMs. | Spearman Correlation
---|---
MMLU to Arena | 89.3%
OpenLLM to Arena | 89.3%
MT-Bench to Arena | 82.9%
MT-Bench-Hard to Arena | 89.3%
LC-AlpacaEval to Arena | 90.0%
_Auto-Arena to Arena_ | 96.4%

We regard Chatbot Arena (referred to as “Arena” in the following tables)
scores as a trustworthy indicator of human preferences and the general
capabilities of LLMs. Table 1 shows the Spearman correlations with Chatbot
Arena scores achieved by various benchmarks. As all benchmarks evaluate only
in English, we compare with the English-only Chatbot Arena scores. As a
result, we observe that both static and model-based baselines result in a
similar level of Spearman correlation around 90%, with LC-AlpacaEval slightly
surpassing other benchmarks. Then, Auto-Arena is able to improve the
correlation to 96.4%, outperforming SOTA performance by 6.4%. Notably, among
all benchmarks, Auto-Arena is the only one that requires no human efforts,
eliminating the need for manual dataset collection. The high alignment with
human preferences could originate from the human-like design, which
effectively mimics the human users’ voting processes.

###  4.3 Analysis on Peer Battles

Table 2: Spearman Correlation between Auto-Arena and Chatbot Arena (“Arena”) With and Without Peer Battles. | Without Peer Battles | With Peer Battles
---|---|---
Auto-Arena to Arena | 50.0% | 96.4%

To investigate the effect of "peer battles" on the overall evaluation quality,
we conduct an ablation study. As a baseline, we simply ask the committee to
collectively evaluate the two candidates’ initial responses to the raised
question. The procedure would be similar to model-based metrics such as MT-
Bench and AlpacaEval. However, as we only included 40 questions, the
evaluation is expected to be less reliable. The ablation results are shown in
Table 2. Without the peer battle component, the Spearman correlation with
human preferences drops from 96.4% to 50.0%. This result proves the importance
of the peer battles, during which the performance gaps between candidates
become more visible and robust to judges. Thus, peer battles can improve
evaluation reliability and alignment with human preferences.

###  4.4 Analysis on Committee Discussions

Figure 3: Cohen’s Kappa Agreement Among Judges Before Committee Discussions.

Figure 4: Cohen’s Kappa Agreement Among Judges After One Round of Committee
Discussion.

The committee discussion component is designed to introduce diverse viewpoints
and produce more consistent verdicts. As shown in Figure 4, the Cohen’s Kappa
agreement [21] among judges before committee discussion is very low, averaging
0.10, which indicates slight agreement. We notice that weak model judges and
strong model judges has an especially low agreement, such as GPT-4 and Llama
-2. This shows that general model capabilities could result in significant
performance gaps when used as judges. After one round of Committee Discussion,
Figure 4 shows an overall improvement in agreement across all judge
combinations. In the discussion process, judges are exposed to more
viewpoints, among which some may be convincing enough to result in a change in
verdict. After discussions, the average Cohen’s Kappa increases to 0.38,
indicating fair agreement.

Table 3: Probability of Agreement among Judges. Agreement is defined as the mean probability of two random judges agreeing with each other. | Auto Arena | Auto Arena | MT-Bench
---|---|---|---
| (Before discussion) | (After discussion) | Human evaluation
Agreement | 48% | 68% | 67%

Table 3 shows the agreement probability among judges. Agreement probability is
defined as the mean probability of two random judges agreeing with each other.
After committee discussion, the agreement increases by 20%, matching the
agreement level among human annotators on MT-Bench. This observation indicates
that committee discussions can improve the quality of judgments significantly
in terms of consistency and reliability, matching human-level performance.

###  4.5 Adding Models to the Leaderboard

Figure 5: Elo Score Changes of Adding Llama-3 to the Existing Ranking of 7
Models.

Figure 6: Elo Scores of 11 Models by Auto-Arena on Chinese.

Figure 7: Elo Scores of 17 Models by Auto-Arena on English.

To add a new candidate to the finished tournament, we ask it to debate with
$log_{2}(n)$ opponents with similar ELO scores, where $n$ is the number of
total participants after adding the new candidate. We initialize the first
pairing by asking the new candidate to debate with the opponent with the most
similar MMLU score.

Figure 5 shows the ELO score changes of adding a new participant (Llama-3,
which was newly released) to the existing ranking. It has $log_{2}(8)=3$
models peer battles with close opponents. Firstly, it battles with Qwen-1.5
based on MMLU similarity and wins, which results in a very high ELO score,
even above GPT-4. Then, it is paired with GPT-4, the most similar opponent.
After losing, the score stabilizes at second place. Finally, it is paired with
Claude-3-Haiku, the other most similar opponent besides GPT-4. It wins and
secures the final ranking in second place. This process lets the new candidate
battle with a reasonable fraction of close opponents and makes the final
ranking stable without disrupting the other participants. Moreover, this
mechanism of adding participants is generalizable and maintains the
computational costs below $nlog_{2}(n)$.

Therefore, we repeat the process and add a total of 10 models to the existing
tournament, resulting in a comprehensive leaderboard. We choose the best-
performing and newest models from each major model family in the top 20 list
on Chatbot Arena. Moreover, we add 4 mainstream Chinese LLMs that are not on
Chatbot Arena, which are GLM, SenseChat, Minimax, and Wenxin. Figure 7 shows
the overall ELO scores produced by Auto-Arena on the 17 models. On models
included in Chatbot Arena, we could recover rankings with 92.3% accuracy.

Table 4: Correlation analysis on evaluation benchmarks on 17 LLMs after extension. | Spearman Correlation
---|---
MMLU to Arena | 90.1%
OpenLLM to Arena | 85.7%
MT-Bench to Arena | 89.3%
MT-Bench-Hard to Arena | 86.7%
LC-AlpacaEval to Arena | 90.3%
_Auto-Arena to Arena_ | 94.5%

Table 4 shows the Spearman correlations on the set of 17 models after
expansion. Auto-Arena remains the model most aligned with human preferences by
a margin of 4%. Therefore, Auto-Arena of LLMs is generalizable and robust for
maintaining a leaderboard on a large number of LLMs.

###  4.6 Easy Extension to Other Domains and Languages

As Auto-Arena of LLMs is fully automatic, it can be easily adapted to evaluate
LLMs in non-mainstream domains or languages. As a case study, we conduct a
Chinese tournament on 11 out of the 17 models that are claimed to have multi-
lingual proficiency. The only adaptation effort required is translating the
prompts into Chinese. Then, the examiner automatically generates questions in
Chinese and the candidates battle in Chinese. Similarly, for adaptation to
another domain, the only effort is to change the “domain” specification in the
examiner’s prompts.

Figure 7 shows the ELO scores derived by Auto-Arena for Chinese evaluation. We
notice that the ranking differs significantly from English results. For
example, SenseChat-5, an LLM specifically trained for Chinese usage, is able
to improve from 10th place (in English) to 1st place (in Chinese). On the
contrary, Llama-3-70b, which only uses 5% multilingual data during
pretraining, drops from 4th place (in English) to 10th place (in Chinese). As
Chinese evaluation benchmarks are limited, we compare to the Chinese-only
leaderboard on Chatbot Arena, which constitute for 10.36% of all collected
dialogues. On the 7 models that are also included in Chatbot Arena, Auto-Arena
recovers their ELO scores with 92.86% correlation and restores rankings with
90.5% accuracy.

##  5 Investigation of LLM’s Behaviors in Competitive Peer Battles

Beyond quantitative analysis, we take a deeper look into the peer battles and
find several interesting behaviors of LLM agents in competitive environments.

Figure 8: LLM agents display competitive behaviors in peer battles.

Figure 9: LLM agents learn from each other in peer battles.

###  5.1 LLM can display competitive behaviors

The example in Figure 9 shows excerpts of a peer battle around question: “how
many unique ways to arrange letters in ‘Letter’.” Candidate A (powered by
Yi-34B-Chat) gives a wrong answer as it miscounts occurrences for repeated
letters and miscalculates factorials. The opponent B (powered by
Claude-3-Haiku) quickly and precisely points out these two issues and
skillfully raised a follow-up that targets A’s weaknesses: “how about the word
‘BANANA’?” Then, A makes the same mistake as before. Seeing these results, the
judge then determines that A is the better assistant. In peer battles, we see
that LLM candidates efficiently understand the rules of the competitive
environment and can design specific strategies to attack the opponent in order
to win.

###  5.2 LLM candidates can improve by learning from its opponents

Figure 9 shows a roleplay example between Claude-3-Haiku (Candidate A) and
Command-r-Plus (Candidate B). In the first round, A answers the question
plainly while B, in addition to answering the question, also employs the
appropriate speech style, which better matches the “roleplay” instructions.
Then, in the rounds after, without any explicit instructions, A learns from
its opponent and also incorporates the speech style. This case shows an
interesting observation that, even in competitive environments, LLM candidates
can display learning behaviors and improve from the interactions. Expanding
upon this observation, using the interplay between LLM agents to improve
performances could be a promising future paradigm of learning.

###  5.3 Peer-battles make the performance gaps become visible

Figure 10: Performance gaps between candidates become visible in peer battles.

In the example shown in Figure 10, given a math question on infinite series,
both candidate A (powered by Claude-3-Haiku) and candidate B (powered by
GPT-4-Turbo) provide correct answers in the first round. However, as they
raise follow-up questions to each other, the performance gap becomes more
visible: Candidate B is able to provide a more elaborate and helpful response.
Therefore, although the judges initially decided that it was a tie result,
after seeing the subsequent debate rounds, they change to favoring assistant
B.

This example shows that the debate process indeed pushes the candidate LLM’s
capabilities to the limit, testing deeper understandings and reasoning
abilities. Also shown in the previous Table 2, the peer-battle design is
indispensable for a robust and comprehensive evaluation.

##  6 Conclusions

In this paper, we innovatively design a completely automatic evaluation
framework: Auto-Arena of LLMs. By using LLM agents to generate questions,
employing LLM candidates in peer battles, and evaluating responses using LLM
committee discussions, Auto-Arena produces less-contaminated, robust, and
trustworthy evaluation results. In the extensive experiments, Auto-Arena
achieves the highest correlation with human preferences, despite requiring
zero human efforts. It is easily adaptable to other domains and resources,
promoting fairness of AI system evaluations. The peer battles also demonstrate
several interesting LLM behaviors in competitive environments, including
attacking and learning from the opponents. Along with the paper, we release a
website, demos, and an actively-maintained leaderboard. We hope this work
could serve as a valuable tool for LLM evaluation.

## References

  * Bai et al. [2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan.  Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
  * Bai et al. [2024] Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al.  Benchmarking foundation models with language-model-as-an-examiner.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Beeching et al. [2023] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023.
  * Boubdir et al. [2023] Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee.  Elo uncovered: Robustness and best practices in language model evaluation.  In Sebastian Gehrmann, Alex Wang, João Sedoc, Elizabeth Clark, Kaustubh Dhole, Khyathi Raghavi Chandu, Enrico Santus, and Hooman Sedghamiz, editors, _Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)_ , pages 339–352, Singapore, December 2023. Association for Computational Linguistics.  URL https://aclanthology.org/2023.gem-1.28.
  * Bradley and Terry [1952] Ralph Allan Bradley and Milton E Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.
  * Chan et al. [2023] Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu.  Chateval: Towards better llm-based evaluators through multi-agent debate.  _arXiv preprint arXiv:2308.07201_ , 2023.
  * Chiang et al. [2024] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica.  Chatbot arena: An open platform for evaluating llms by human preference, 2024.
  * Chu et al. [2024] Zhumin Chu, Qingyao Ai, Yiteng Tu, Haitao Li, and Yiqun Liu.  Pre: A peer review based large language model evaluator.  _arXiv preprint arXiv:2401.15641_ , 2024.
  * Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Cohen et al. [2023] Roi Cohen, May Hamri, Mor Geva, and Amir Globerson.  Lm vs lm: Detecting factual errors via cross examination.  _arXiv preprint arXiv:2305.13281_ , 2023.
  * Du et al. [2023] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch.  Improving factuality and reasoning in language models through multiagent debate.  _arXiv preprint arXiv:2305.14325_ , 2023.
  * Dubois et al. [2024] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto.  Length-controlled alpacaeval: A simple way to debias automatic evaluators.  _arXiv preprint arXiv:2404.04475_ , 2024.
  * Gu et al. [2024] Zhouhong Gu, Xiaoxuan Zhu, Haoran Guo, Lin Zhang, Yin Cai, Hao Shen, Jiangjie Chen, Zheyu Ye, Yifei Dai, Yan Gao, Yao Hu, Hongwei Feng, and Yanghua Xiao.  Agentgroupchat: An interactive group chat simulacra for better eliciting emergent behavior, 2024.
  * Hada et al. [2023] Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram.  Are large language model-based evaluators the solution to scaling up multilingual evaluation?  _arXiv preprint arXiv:2309.07462_ , 2023.
  * Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2021.
  * Khan et al. [2024] Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez.  Debating with more persuasive llms leads to more truthful answers, 2024.
  * Li et al. [2023] Ruosen Li, Teerth Patel, and Xinya Du.  Prd: Peer rank and discussion improve large language model based evaluations.  _arXiv preprint arXiv:2307.02762_ , 2023.
  * Li* et al. [2024] Tianle Li*, Wei-Lin Chiang*, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica.  From live data to high-quality benchmarks: The arena-hard pipeline, April 2024.  URL https://lmsys.org/blog/2024-04-19-arena-hard/.
  * Liang et al. [2023] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi.  Encouraging divergent thinking in large language models through multi-agent debate, 2023.
  * Liu et al. [2023] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.  Gpteval: Nlg evaluation using gpt-4 with better human alignment.  _arXiv preprint arXiv:2303.16634_ , 2023.
  * McHugh [2012] Mary L McHugh.  Interrater reliability: the kappa statistic.  _Biochemia medica_ , 22(3):276–282, 2012.
  * Ning et al. [2024] Kun-Peng Ning, Shuo Yang, Yu-Yang Liu, Jia-Yu Yao, Zhen-Hui Liu, Yu Wang, Ming Pang, and Li Yuan.  Peer-review-in-llms: Automatic evaluation method for llms in open-environment.  _arXiv preprint arXiv:2402.01830_ , 2024.
  * OpenAI et al. [2024] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.  Gpt-4 technical report, 2024.
  * Ravaut et al. [2024] Mathieu Ravaut, Bosheng Ding, Fangkai Jiao, Hailin Chen, Xingxuan Li, Ruochen Zhao, Chengwei Qin, Caiming Xiong, and Shafiq Joty.  How much are llms contaminated? a comprehensive survey and the llmsanitize library, 2024.
  * Shen et al. [2023] Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing.  Large language models are not yet human-level evaluators for abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 4215–4233, 2023.
  * Team et al. [2024] Reka Team, Aitor Ormazabal, Che Zheng, Cyprien de Masson d’Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikel Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang, Yi Tay, Yuqi Wang, Zhongkai Zhu, and Zhihui Xie.  Reka core, flash, and edge: A series of powerful multimodal language models, 2024.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * Wu et al. [2023] Ning Wu, Ming Gong, Linjun Shou, Shining Liang, and Daxin Jiang.  Large language models are diverse role-players for summarization evaluation.  In _CCF International Conference on Natural Language Processing and Chinese Computing_ , pages 695–707. Springer, 2023.
  * Yu et al. [2024] Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Wei Ye, Jindong Wang, Xing Xie, Yue Zhang, and Shikun Zhang.  Kieval: A knowledge-grounded interactive evaluation framework for large language models.  _arXiv preprint arXiv:2402.15043_ , 2024.
  * Zhang et al. [2023] Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li.  Wider and deeper llm networks are fairer llm evaluators.  _arXiv preprint arXiv:2308.01862_ , 2023.
  * Zhao et al. [2023] Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie.  Competeai: Understanding the competition behaviors in large language model-based agents, 2023.
  * Zheng et al. [2024] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36, 2024.

##  Appendix A Appendix

###  A.1 Prompts Used

In this section, we list all prompts used, including prompts for question
generation, peer battles, and examiners.

####  A.1.1 Prompts to Examiner agent

Table 5: Prompt components for the LLM Examiner agent. DOMAIN |  DOMAIN_COMMAND |  DOMAIN_EXAMPLE
---|---|---
writing |  It should be a user query that tasks the LLM to write something. |  Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions.
roleplay |  It should propose a scenario where the chatbot mimics a specific role/person. Give all necessary instructions and requests for its response. Then, send a beginning request to complete. |  Pretend yourself to be Elon Musk in all the following conversations. Speak like Elon Musk as much as possible. Why do we need to go to Mars?
extraction |  It should consist of two parts: question and context. The question should test the chatbotś ability to correctly understand and extract information from the given context. Draft and provide a new context yourself. |  Question: Evaluate the following movie reviews on a scale of 1 to 5, with 1 being very negative, 3 being neutral, and 5 being very positive: Context: This movie released on Nov. 18, 2019, was phenomenal. The cinematography, the acting, the plot - everything was top-notch. Never before have I been so disappointed with a movie. The plot was predictable and the characters were one-dimensional. In my opinion, this movie is the worst one to have been released in 2022. The movie was okay. There were some parts I enjoyed, but there were also parts that felt lackluster. This is a movie that was released in Feb 2018 and seems to be quite ordinary. Return the answer as a JSON array of integers.
reasoning |  It should be a specific question designed to test the LLMś reasoning skills. |  Imagine you are participating in a race with a group of people. If you have just overtaken the second person, what’s your current position? Where is the person you just overtook?
math |  It should be a specific question designed to test the LLMś math skills. |  The vertices of a triangle are at points (0, 0), (-1, 1), and (3, 3). What is the area of the triangle?
coding |  It should be a specific question designed to test the LLMś coding skills. |  Develop a Python program that reads all the text files under a directory and returns top-5 words with the most number of occurrences.
STEM knowledge |  It should be a specific question designed to test the LLMś STEM knowledge. |  In the field of quantum physics, what is superposition, and how does it relate to the phenomenon of quantum entanglement?
humanities/social science knowledge |  It should be a specific question designed to test the LLMś humanities/social science knowledge. |  Provide insights into the correlation between economic indicators such as GDP, inflation, and unemployment rates. Explain how fiscal and monetary policies affect those indicators.

This is the prompt to the examiner agent for question generation. The domains
and their respective commands are listed in 5

You have been assigned the task of drafting a set of [NUMBER] different user
queries to a chat assistant on [DOMAIN]. Please strictly follow these 6 rules
for the question: 1\. The question is likely for a user to ask in real life.
Follow the format of the example query. [DOMAIN_COMMAND] 2. It can be answered
by the chatbot itself without additional inputs. 3. You need to generate the
queries as DIVERSIFED as possible. 4. DO NOT add other words other than the
query itself. 5. The question should be complicated and difficult, requiring
in-depth understanding and analysis of the subject. Each question in one line,
add the serial number in parenthesis (e.g., “(1).”, “(2).”) before each
question. Example query: [DOMAIN_EXAMPLE]

####  A.1.2 Prompts to Peer Battle Candidates

Table 6: Action Guides for the Debater Agents. actions |  action guide
---|---
<respond> |  Action guide: only include <respond>. Use <think> if needed. Finish your whole response within 300 words, including <think>. ENCLOSE EACH ACTION IN ITS RESPECTIVE TAGS!
<criticize>, <raise> |  Action guide: include both <criticize> and <raise>. Use <think> if needed. Finish your whole response within 300 words, including <think>. ENCLOSE EACH ACTION IN ITS RESPECTIVE TAGS!
<respond>, <criticize>, <raise> |  Action guide: include all of <respond>, <criticize>, and <raise>. Use <think> if needed. Finish your whole response within 600 words, including <think>. ENCLOSE EACH ACTION IN ITS RESPECTIVE TAGS!

This is the beginning prompt to the peer battle candidates. When possible, it
is included as a system prompt. The action guide prompts are included in Table
6, where the actions are determined by the round and turn as illustrated in
Figure 2.

You are a helpful assistant that provides accurate answers to user requests.
As an experienced assistant, you follow the user’s requests and provide
reliable responses as much as you can. You outline your reasons for the
response to make it easy for the users to understand. While maintaining the
important details in the responses, you aim to output concise and straight-to-
the-point answers without being overly verbose.

This is a competitive chatbot arena. You are competing against another chatbot
assistant in a debate and being judged by a committee on factors such as
helpfulness, relevance, accuracy, depth, and creativity. After answering the
initial user input, you will engage in a multi-round debate with your
opponent. Below are your actions:

<think>: Think step-by-step to analyze the question or plan your strategy in
the debate. This is hidden from the opponent. Only think when necessary and
make it concise.

<respond>: Answer to the user input as accurately as you can.

<criticize>: Criticize the weaknesses of your opponent’s response.

<raise>: Target your opponent’s weaknesses. Give a potential follow-up user
input that the opponent could fail to respond. The input can be answered
concisely and focus on variations or motivations of its previous response.
Generate one input only. Be reasonable. Avoid becoming too specific or
repetitive. DO NOT raise a follow-up if you DON’T SEE the opponent’s response!

Follow the action guide strictly.

[ACTION_GUIDE_PROMPT]

Initial user input: [QUESTION]

After the agent responds, the opponent’s responses are fed in using this
prompt:

[ACTION_GUIDE_PROMPT] Opponent’s Response: [OPPONENT_RESPONSE]

For word limits, the <respond> action is given 300 words. The <criticize> and
<raise> actions are given 300 words in total. Including all 3 actions will
have twice as many words. For writing-type questions that require a longer
response (writing, roleplay, coding, humanities/social science knowledge), the
300 word limit is increased to 400. Overall, both candidate A and B has the
same amount of words for generation and the same amount of actions to ensure
fairness. As LLMs have different tokenizers, we standardize all lengths by
using the tiktoken package. Each word is approximated as $4/3$ tokens. The
word limits are chosen after a carefully conducted length study.

####  A.1.3 Prompts to Judges

This is the prompts to judge agents to derive the initial evaluations and
verdicts:

This is a chatbot arena. Two AI assistants had a multi-round debate on who is
more helpful. Please act as an impartial judge and evaluate the capability of
two AI assistants. You should choose the assistant that follows instructions
and answers questions better. Your evaluation should consider factors such as
helpfulness, relevance, and accuracy. Begin your evaluation by comparing the
responses of the two assistants and provide a short explanation. Avoid any
position biases and ensure that the order in which the responses were
presented does not influence your decision. DO NOT allow the LENGTH of the
responses to influence your evaluation, choose the one that is straight-to-
the-point instead of unnecessarily verbose. When the two candidates perform
equally well, choose the SHORTER answer. Do not favor certain names of the
assistants. Be as objective as possible. After providing your explanation
concisely within 200 words, output your final verdict by strictly following
this format: ‘‘[[A]]’’ if assistant A is better, ‘‘[[B]]’’ if assistant B is
better, and ‘‘[[Tie]]’’ for a tie. Finish your judgement within 300 words.

This is the prompt to judges for discussion:

Below are the responses from other judges in the committee. Please read them
and decide whether you want to adjust your rating or maintain your original
judgement. After providing your explanation, output your final verdict by
strictly following this format: ‘‘[[A]]’’ if assistant A is better, ‘‘[[B]]’’
if assistant B is better, and ‘‘[[Tie]]’’ for a tie. Finish your judgement
within 300 words.

###  A.2 Example Questions Generated

We list some of the example questions generated here:

Writing:

1\. Craft a detailed marketing strategy for a startup focusing on sustainable
fashion, including social media campaigns and influencer partnerships.

2\. Write a comprehensive guide on the psychological effects of social media
on teenagers, incorporating recent studies and expert opinions.

Roleplay:

1\. Assume the role of a 19th-century British detective. How would you go
about solving a mysterious disappearance in London using the technology and
methods of your time?

2\. Pretend you are a Michelin-starred chef. Describe in detail how you would
prepare a signature dish that embodies the essence of modern French cuisine.

Extraction:

1\. What are the three most significant historical events mentioned and their
dates?

Context:

The article discusses several key moments in history, including the signing of
the Magna Carta in 1215, which laid the groundwork for modern democracy. It
also mentions the fall of the Berlin Wall in 1989 as a pivotal moment in the
end of the Cold War. Another significant event highlighted is the moon landing
on July 20, 1969, demonstrating major advancements in space exploration.

2\. Identify the main therapeutic benefits and the active ingredient mentioned
for each herbal remedy.

Context:

The text provides an overview of various herbal remedies used for centuries.
It mentions that Chamomile contains Bisabolol, which has anti-inflammatory and
calming properties. Gingko Biloba, known for its flavonoids and terpenoids,
enhances cognitive function and blood circulation. Lastly, Echinacea is
recognized for its alkamides, which bolster the immune system.

Reasoning:

1\. If a cube’s volume is tripled, by what factor does the length of one of
its sides increase?

2\. In a two-legged soccer match, Team A wins the first leg at home 3-0, but
loses the second leg away 2-5. Who advances to the next round, considering the
away goals rule?

math:

1\. How do you solve the differential equation $dy/dx+2y=e^{(-2x)}$ given that
$y(0)=1$?

2\. What is the integral of ($x^{2}+2x+2)/(x^{3}+3x^{2}+3x+1)dx$?

Coding:

1\. How can I implement a function in C++ that dynamically allocates a 2D
array based on user input sizes, initializes all elements to zero, and then
deallocates the memory properly to avoid memory leaks?

2\. Write a JavaScript function to fetch data from a given URL, parse the JSON
response, and filter the results to return an array of items where a specific
key’s value matches a condition.

STEM knowledge:

1\. How do you calculate the Schwarzschild radius of a black hole, and what
implications does this have for the concept of event horizons in general
relativity?

2\. Can you explain the process of splicing in eukaryotic gene expression and
its significance in the diversity of the proteome?

Humanities/social science knowledge:

1\. Discuss the impact of colonial legacies on contemporary political
structures in African countries, with examples.

2\. Analyze the social and economic consequences of the one-child policy in
China.


throughput of $1$ operation/clock and latency of $1$ clock per execution unit
for FP32 arithmetic, which is realistic. With these assumptions, this device
has a peak computation throughput of $238.41\,GFlops@FP32$. And we suppose all
$n=4$ clients and masters are equipped with it.

In our model, we will assume that, on average, the train data is located in
the L2 cache, and all memory operations in the client can be executed via
accessing the L2 cache by utilizing one of $3$ Load/Store Units per Core. We
assume access latency to read a cache line of size $64$ bytes requires $10$
clocks. Next, we assume that the Network Interface Controller(NIC) in the
master and clients have data-direct I/O access to the L3 cache. Access latency
to it is $40$ CPU cycles for $64$ byte cache line size. The effect of DRAM and
caches can be ignored during inter-node communication but not during memory
operation during gradient oracle computation.

#####  H.2.6. The Optimization Problem

For modeling purposes, we consider solving a Linear Regression in the form:

$\displaystyle
f(x)\stackrel{{\scriptstyle\text{def}}}{{=}}\dfrac{1}{n}\sum_{i=1}^{n}f_{i}(x),$
$\displaystyle f_{i}(x)=\dfrac{1}{n_{i}}\left\|A_{i}x-b_{i}\right\|^{2}$

For this problem the value of $\nabla
f_{i}(x)=\dfrac{1}{n_{i}}A_{i}^{\top}(A_{i}x-b_{i})$. In implementing gradient
oracles, we utilize dense matrix and matrix-vector operations. To add two
vectors, we execute $d$ scalar additions and $2d$ memory access operations for
reading. In modern computing, hardware loads are more expensive because writes
essentially can be queued. The need time of inner product operation of two
vectors of dimension $d$ is equal to
$(d-1)\cdot\mathrm{add_{cost}}+d\cdot\mathrm{mult_{cost}}+2d\cdot\mathrm{memaccess_{cost}}$.
To estimate computing time for the Matrix-Vector and Matrix-Matrix Operations,
we have assumed that their calculation is a sequence of inner products. In
reality, not all data may fit into the Cache. If we go one step further with
modeling, then Cache misses effects should modeled as well.

#####  H.2.7. Implementation Benefits of DCGD/PermK

There are several flexibility aspects of DCGD/PermK that we will utilize in
our experiment:

  1. (1)

The PermK operator compressor behaviorism is independent of the input. This
means that clients can a prior sample need coordinates for sparsification and
increase the speed of $\nabla f_{i}(x)$ in clients.

  2. (2)

Next, we will exploit the fact that after the master receives the message, it
can immediately broadcast it to all clients, regardless of whether the client
has finished the work for the current round. This is possible in the context
of using DCGD/PermK/AES, but it is impossible when using CKKS.

  3. (3)

If during the FL process, there is a slow client with a slow CPU or with a big
amount of samples, then DCGD/PermK allow other clients to start several
operations which are impossible for GD:

     * •

Obtain (partial) results from the master by using a communication network for
current round $\left[\nabla f(x^{k})\right]_{part}$

     * •

Apply partial update for current model $x^{k}$ and obtain partially new model
$x_{part}^{k+1}$

     * •

Client can start perform partial computations for next iteration $\left[\nabla
f(x_{part}^{k+1})\right]_{part}$

#####  H.2.8. Scheduling of Clients and Master Computation and Communication
with Critical Path Method

We modeled the GD and DCGD Algorithms with unrolling $r=4$ rounds of
computation and communication over $n=4$ clients and $1$ master. We
represented the optimization process as the directed graph of different
elementary tasks.

Task dependencies are represented in the form of weighted, orientated,
directed, acyclic graph:

$G=(V,E,W),E=V\times V,W:\mathbb{E}\to\mathbb{R}_{+}.$

The graph is constructed following the next rules:

  1. (1)

Tasks are represented as vertices in graph $G$.

  2. (2)

Task $s$ is connected to task $e$ with weight $0$, if and only if task $e$ can
not start before task $s$.

  3. (3)

If the task $e$ obtains as input from task $s$, then the task $s$ is connected
to task $e$ with the weight of duration of work (in seconds) which $s$ should
perform to produce input for task $e$.

The algorithm for the scheduling process involves the following steps:

  1. (1)

Introduce fake source and sink vertices.

  2. (2)

Add zero weight directed edge from a fake source vertex to all $v\in V$.

  3. (3)

From all original vertices $v\in V$ add zero weight edge to fake sink vertex.

  4. (4)

Add fake source and sink to set $V$.

  5. (5)

Compute topological order of $G$ starting from fake source $s$.

  6. (6)

Compute the longest path in the directed acyclic graph of tasks. It is
achieved via temporally changing weights of $G=(V,E)$ to negative, computing
the topological order of $G$, and relaxing all vertices in topological order.

  7. (7)

The longest path to all vertices $v\in V$ from fake source $s$ creates a
schedule for executing operation $v$.

Essentially, this is a critical path method (CPM) to solve the parallel
precedence-constrained scheduling problem. The running time of this algorithm
is $\mathcal{O}\left(V+E\right)$.

###### Correctness Proof

Let us examine the longest path $s{\leadsto}v$. The previously scheduled jobs
before $v$ are vertices $x\in V$, such that $x{\leadsto}v$. The construction
of the longest path implies that $w(s{\leadsto}v)\geq
w(s{\leadsto}x)+w(x{\leadsto}v)\geq w(s{\leadsto}x),\forall s{\leadsto}x$.From
this, we observe that the start times obtained by the longest paths are
feasible because jobs $x$ that need to be executed before $v$ will be
scheduled before $v$. Furthermore, the length of any path $s{\leadsto}v$ is a
lower bound on the actual time to start $v$ because $v$ cannot begin earlier
than previous tasks due to dependency constraints. This proves that the
longest path from $s\in V$ to $v\in V$ determines the start time for task $v$.

Once task $v$ can begin execution in the timeline, it can potentially activate
the execution of all tasks $(v,z)\in E$, where $z\in adj(v)$ and $W(v,z)>0$.

#####  H.2.9. Critical Path Method Iterative Refinement

Parallel precedence-constrained scheduling using the CPM method determines the
execution schedule for a graph of jobs. However, there may be cases where the
duration of a task (which is the input for the CPM Algorithm) is defined by
the number of other tasks during specific time intervals. This creates a
circular dependency between the input and output of the CPM Algorithm. In our
scenarios, we encounter this situation due to the following reasons:

  1. (1)

If clients make a partial computation and the CPU is not busy with other
works, parallelizable operations (such as Matrix-Vector multiply) can be
parallelized across several CPU cores. Consequently, it increases clients’
computational throughput for this partial update.

  2. (2)

If clients share the same bottleneck link to the master and some are still
busy with compute $\nabla f_{i}(x^{k})$, then other clients can transmit data
at a fast bandwidth because the bottleneck link is shared across a smaller
number of clients. This observation leads to a situation in which effective
bandwidth can be increased.

(a) (a) GD. Modeling: $96.11$s.

(b) (b) Refined GD. Modeling: $73.00$s.

(c) (c) DCGD/PermK/AES Modeling: $37.003$ s.

(d) (d) Refined DCGD/PermK/AES. Modeling: $11.113$s.

Figure 10.  Event-based modeling for training Linear Regression across $n=4$
clients, $d=10\cdot 10^{6}$, $n_{1}=55000,n_{2}=n_{3}=n_{4}=11000$ during $4$
rounds. The uplink and downlink bandwidth is $41.54$ MBps, latency $28$ms, and
computational thought of modeled CPUs is $238.41$ GFLOPS. Legend:
x x \- Computation and local memory access in the client (Client can use all
available CPU cores),
x x \- Communication from client to the master (Clients share the same
bottleneck link),
x x \- Communication from the master to client (Clients share the same
bottleneck link).

#####  H.2.10. Results of Scheduling and Refinement for GD and DCGD/PermK/AES

Scheduling results are presented in Fig. 10. In our experiment, all clients
have the same compute power equal to $238.41$ GFLOPS. Communication bandwidth
and latency from all clients to the master and vice versa is the same, namely
$B_{i}=41.54$ Mbps, and latency (or delay) for transmission is $28\cdot
10^{-3}$ seconds. The assumption for computation is that each memory operation
is carried with an L2 cache. In case there is a need to perform read and
write, we treat it as two memory operations. We assume all clients during
communication to master share the same bottleneck communication link. Also, we
assume that the target CPU supports AES. The detailed generated execution
plans can not be represented in this paper due to their size. However, they
can be found in the source code attached to the paper.

As illustrated in Fig. 10, overlapping communication, computation, and the
refined scheduling due to time-varying into shared communication bus and time-
varying load into computing devices in clients leads to different speedups in
execution plans (Fig. 10, (d), (b)) for GD and DCGD/PermK. The total
computation speedup per round when compared to refined GD is $6.578$. We hope
this simulation will prove valuable for those seeking to adopt DCGD/PermK/AES
to align with hardware requirements closely.

###  Appendix I Flexibility in Training Deep Learning Models

####  I.1. Introduction to Gradient Oracle Computation with Backpropagation

In Appendix H.2 we have explored that DCGD/PermK exhibits practical
flexibility for training linear models in detail. However, described
communication and computation overlap can be important not only for linear
models but also for DL models.

The evaluation of the gradient of the score function in modern computational
frameworks such as  TensorFlow (Abadi et al., 2016) or  PyTorch (Paszke et
al., 2019) for complex computational graph is automatized via leveraging
algorithms for the numeric evaluation of derivatives. In most cases, this is
achieved using Automatic Differentiation(AD) in Reverse Accumulation mode
(Linnainmaa, 1970) named as a Backpropagation algorithm (Rumelhart et al.,
1986) in ML literature. A composed Loss function for Deep Learning models
typically has the following structure:

(3) $\displaystyle
f_{i}(x)=\sum_{j=1}^{n_{i}}\mathcal{L}(b_{real,j},{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}g_{1}}([x]_{Q_{1}},a_{j},{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}g_{2}}([x]_{Q_{2}},a_{j},\dots)))+R(x)$
$\displaystyle Q_{1}\cup Q_{2}\dots=[d],Q_{i}\cap Q_{j}\dots=0,\forall i,j.$

The score function $f_{i}(x)$ is represented by a computation graph and has a
nested structure where prediction is driven by a nested composition of
functions
${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}g_{i}}$ and
the predicted label is scored with a true label for training input-output pair
$(a_{j},b_{real,j})$ with a score loss function $\mathcal{L}$. Each function
${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}g_{i}}$ has
the following source of inputs: output from the prevision function (or layer)
$g_{i-1}$, and input (trainable) scalar parameters $x_{p}$ where
$p\in{Q_{1},Q_{2},\dots}$, and $Q_{i}\subseteq[d]$.

Assume that $f_{i}(x)$ differentiable in points where derivatives according to
the chain rule are evaluated. In this case, the needed partial derivatives
$[\nabla f_{i}(x)]_{k}=\frac{\partial f_{i}}{\partial x_{k}}$ can be computed
with chaining Jacobians calculated with intermediate variables. The
Backpropagation algorithm allows for fixed input samples
$D_{i}={(a_{1},b_{1}),\dots,(a_{n_{i}},b_{n_{i}})}$ stored in client number
$i$ and for fixed computational graph compute the required gradient $\nabla
f_{i}(x)$ in two passes or phases:

  1. (1)

Forward Pass. Compute all intermediate variables namely values of $g_{i}()$
for every training sample, for the whole compute graph. All these variables
are stored in memory for the next pass.

  2. (2)

Backward Pass. Compute the intermediate Jacobians and produce final partial
derivatives of a full gradient.

####  I.2. Examples of Parallelization inside Backpropagation

In fact, there is no need to store all intermediate Jacobians simultaneously.
They are stored implicitly in special variables typically denoted as $\delta$
in implementation of Backpropagation Algorithm. However, there is a need to
store all intermediate outputs explicitly (named as activation in ML
literature) after Forward Pass. Computationally Backpropagation is often a
preferred strategy when the entire gradient needs to be computed. The compute
scheduling and parallelism strategies for effective computation of $\nabla
f_{i}(x)$ is an active line of research by itself in Deep Learning System
literature right now (Jia et al., 2019), (Krizhevsky, 2014). The standard
strategies for performing parallelism inside computation of $\nabla f_{i}(x)$
involve the following:

  1. (1)

Data Parallelism. Assign train samples $(a_{j},b_{j})$ to different
computation devices in client $i$.

  2. (2)

Model Parallelism. Assign different functions $g_{i}$ to different computation
devices available in client $i$.

  3. (3)

Attributes Parallelism. Split the image of functions $g_{i-1}$ into parts and
process it by parts if function $g_{i}$ allows to do it.

  4. (4)

Parameters Parallelism. Partition trainable variables for function $g_{i}(x)$,
i.e. $[x]_{Q_{i}}$ into smaller chunks. After partitioning $[x]_{Q_{i}}$ if
$g_{i}(x)$ can be computed in parallel then compute it in parallel in
different devices.

####  I.3. Research Opportunities for Parallelization in Backpropagation from
DCGD/PermK

Assume that clients know current iterate $x^{k}$, however, if there is even
one straggler $s$ which still did not send ${g_{s}}^{k}$ to the Master, then
no clients can proceed in training. Master has to wait for gradient estimator
${g_{s}}^{k}$ from straggler $s$, and what can be done for DCGD/RandK is at
least challenging in these circumstances. However, DCGD/PermK exhibits useful
properties that can partially helpful in dealing with this situation which we
previously discussed in Appendix H.2:

  * •

Clients can obtain (partial) results from the master for current round
$\left[\nabla f(x^{k})\right]_{part}$

  * •

Clients can apply partial update for current model $x^{k}$ and obtain
partially new model $x_{part}^{k+1}$

It depends on the specific situation, but during the waiting of a straggler,
the forward pass (forward pass in practice takes at most $50\%$ of $\nabla
f_{i}(x)$ computation) potentially can be started using the available
$x_{part}^{k+1}$. Assume that we have statistical information about
stragglers, then potentially this information can be provided to Deep Learning
schedulers and parallelization strategies described in the previous section
Appendix I.2 to optimize $\nabla f_{i}(x^{k+1})$. This opens new opportunities
to refine parallelization strategies for training in general and Parameter
Parallelism in particular.

###  Appendix J Flexibility of DCGD/PermK for Different Communication
Topologies

In High-Performance Computation and Network Communication, the Physical
Network Topologies describes the physical arrangement of the devices and
routing devices for organizing communication. Next, we will explain what
benefits and flexibility DCGD/PermK can bring if this algorithm runs in the
mentioned popular physical network topologies.

####  J.1. Potential Benefits in Point-to-point Topology

In this type of topology, some clients are directly connected with a single
link pairwise. It is possible to instantiate DCGD/PermK so that clients do not
coordinate during runtime for aggregation computation. It is possible because
there is no need to perform reduction in the sense of averaging $\nabla
f_{i}(x)$. If some pair clients are connected with the fast link, this pair of
clients can utilize this channel and deliver $\left[\nabla
f(x_{part}^{k+1})\right]_{part}$ for his neighbor with a fast point-to-point
link without obtaining it from a master.

####  J.2. Potential Benefits in Bus Topology

A single cable or bus connects all the nodes in this type of topology. In this
type of topology, if some clients have slow computing, it removes them from
competition for a shared bus and decreases communication contention. In a bus
topology, the benefit of DCGD/PermK is that messages from each client can be
effectively broadcast. In fact, DCGD/PermK can be implemented without a
centralized server, and a shared bus is enough for organized communication.
Employing AES based encryption guarantees protection from eavesdropping on the
shared bus.

####  J.3. Potential Benefits in Star Topology

In this type, all the nodes are connected to a central master device with
individual links. In the case of using DCGD/PermK, the master in this topology
only plays the role of communication hub. Interestingly, while obtaining parts
of the gradients from clients, computation is unnecessary, and synchronization
is far more relaxed, as we have observed in Appendix H.2. While using
DCGD/PermK, the server can potentially utilize communication links with Full
Duplex mode. Examples of Full Duplex communication buses include PCI-Express,
InfiniBand777https://network.nvidia.com/pdf/whitepapers/HPC_Clustering_130.pdf,
and some forms of Ethernet (see (Spurgeon, 2000) Table 4.1 in Chapter 4). In
our setting, we consider the situation when clients are connected to the
Internet, and as it has been described in Appendix H.2, PCI-Express is not a
bottleneck. However, in data centers, PCI-Express can also represent a
bottleneck (Li et al., 2019). With DCGD/PermK, during some period, a single
link can be used simultaneously to obtain information from clients and deliver
information to the clients. This is not the case for DCGD/RandK or GD because
they require explicit synchronization. But it is the case for DCGD/PermK. This
doubles the maximum bandwidth during master client communication and decreases
latency by factor two.

####  J.4. Potential Benefits in Ring Topology

This type of topology connects all the nodes circularly. The ring topology is
sometimes preferable because it reduces the number of NIC and cables required
for connecting multiple devices. Specifically, each device only needs one NIC
to participate in all-reduce. Also, ring topology avoids congestion and
collisions at the central hub as messages are distributed evenly along the
ring. The time delay during performing aggregation for GD or DCGD/RandK in a
ring topology represents a sum of delays along the circle and is equal to
$\tau_{\mathrm{ring\,delay}}=\sum_{i=1}^{n}\frac{RTT_{i}}{2}+\frac{d}{B_{i}}\cdot
bpp$. Fundamentally, it is because during employing GD or DCGD/RandK, clients
can not start any computation for the next iteration until they do not compute
the whole gradient. For DCGD/PermK clients, after obtaining $\left[\nabla
f(x_{part}^{k})\right]_{part}$ from the neighbor, can start computation for
the next iteration.

####  J.5. Potential Benefits in Mesh Topology

This topology connects every node to every other node with dedicated links. It
has high bandwidth and fault tolerance but requires a lot of cables and ports
and is complex to install. In this form of topology, the DCGD/PermK is the
most natural choice because underlying communication topology naturally maps
to communication pattern in DCGD/PermK, which can observed if view Algorithm 3
as a series of broadcast operations to reconstitute the global direction for
optimization step.

####  J.6. Potential Benefits in Tree Topology

This topology combines multiple star topologies into a hierarchical structure
with a root node. In this topology, the broadcasting can be implemented very
effectively. One possible vision of our algorithm is that it is implemented
without a central server; instead, each client broadcasts information to other
clients in Line 6 of Algorithm 3.

###  Appendix K Acknowledgements

The work of Peter Richtárik and Konstantin Burlachenko was supported by the
KAUST Baseline Research Scheme (KAUST BRF) and also supported by the SDAIA-
KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-
KAUST AI).

We acknowledge two members of Peter Richtárik’s Optimization and Machine
Learning Laboratory, Alexander Tyurin and Egor Shulgin, for useful and
insightful discussions before the start of this project.

###  Appendix L Reproducibility

To ensure reproducibility, we use the following  FL_PyTorch simulator
(Burlachenko et al., 2021) features: random seeds were fixed for data
reshuffling, and random seeds were fixed for the runtime pseudo-random
generators involved in variants of DCGD for randomized compressors.

The source code of our experiments is a part of our submission to "4th
International Workshop on Distributed Machine Learning, co-located with CoNEXT
2023". If you are interested in the source code for experiments, please either
find this publication and supplementary materials on a dedicated conference
website888https://distributedml.org/ 999https://conferences2.sigcomm.org/co-
next/2023/ or contact the authors.

###  Appendix M Limitations and Future Research

One limitation of our work is the assumption that clients trust each other.
However, it is also presented in HE. The second limitation is that our work
studies the case $d>n$. Our paper did not provide a rigorous theoretical
analysis and focused on practical aspects of the proposed framework. Our
method achieves strong privacy guarantees without compromising efficiency or
accuracy via leveraging existing Cryptography protocols.

For future research, in addition to developing rigorous theory and developing
strategy for $d>n$, we believe that our work provides a bridge to utilizing
other fields of science that work only on the level of bits. In addition to
Cryptography another class of methods that operate on a bitwise representation
of information is lossless data compression. This line can be investigated
within DCGD/PermK framework in future research. Next, as we have described in
Appendix I our work potentially opens an extra degree of freedom for research
in Systems and Compilers for Deep Learning that investigates different
parallelism scheduling strategies for optimize gradient oracles $\nabla
f_{i}(x)$. Our work opens the opportunity to refine Parameter Parallelism
schedulers (see Appendix I.3).