# Improving Factuality and Reasoning in Language Models through Multiagent
Debate

Yilun Du
MIT CSAIL
<EMAIL_ADDRESS>
&Shuang Li
MIT CSAIL
<EMAIL_ADDRESS>
&Antonio Torralba
MIT CSAIL
<EMAIL_ADDRESS>
&Joshua B. Tenenbaum
MIT CSAIL, BCS, CBMM
<EMAIL_ADDRESS>
&Igor Mordatch
Google Brain
<EMAIL_ADDRESS>

###### Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in
language generation, understanding, and few-shot learning in recent years. An
extensive body of work has explored how their performance may be further
improved through the tools of prompting, ranging from verification, self-
consistency, or intermediate scratchpads. In this paper, we present a
complementary approach to improve language responses where multiple language
model instances propose and debate their individual responses and reasoning
processes over multiple rounds to arrive at a common final answer. Our
findings indicate that this approach significantly enhances mathematical and
strategic reasoning across a number of tasks. We also demonstrate that our
approach improves the factual validity of generated content, reducing
fallacious answers and hallucinations that contemporary models are prone to.
Our approach may be directly applied to existing black-box models and uses
identical procedure and prompts for all tasks we investigate. Overall, our
findings suggest that such "society of minds" approach has the potential to
significantly advance the capabilities of LLMs and pave the way for further
breakthroughs in language generation and understanding. Project website at
https://composable-models.github.io/llm_debate/.

##  1 Introduction

Large language models (LLMs) have demonstrated remarkable language generation,
understanding, and few-shot learning capabilities in recent years. These
methods are trained on a massive corpus of text on the internet, where the
quality and accuracy of extracted natural language may not be ensured. Thus,
current models may suffer from confidently hallucinating facts or making
implausible jumps in chains of reasoning. An extensive body of recent work has
focused on improving factual accuracy and reasoning in language models. These
range from prompting models with few or zero-shot chain-of-thought
demonstrations, use of verification, self-consistency, or intermediate
scratchpads.

We note that these techniques are applied over a single model instance.
Instead, we propose a complementary approach inspired by _The Society of Mind_
[19] and multi-agent settings, where multiple language model instances (or
agents) individually propose and jointly debate their responses and reasoning
processes to arrive at a single common answer. More specifically, given a
query, multiple instances of a language model first generate individual
candidate answers to a query. Then each individual model instance reads and
critiques the responses of all other models and uses this content to update
its own answer. This step is then repeated over several rounds. This process
induces models to construct answers that are consistent with both their
internal critic as well as sensible in light of the responses of other agents.
The resulting quorum of models can hold and maintain multiple chains of
reasoning and possible answers simultaneously before proposing the final
answer.

We find that our debate approach outperforms single model baselines such as
zero-shot chain of thought [11] and reflection [26, 18] on a variety of six
reasoning, factuality, and question-answering tasks. Using both multiple model
agents and multiple rounds of debate are important to achieve the best
performance. Given an initial query, we find that individual model instances
propose a diverse range of answers despite being the same model class
(although we also investigate the case of mixing different model types, such
as chatGPT [21] and Bard [23]). After debating and examining the responses of
other model instances, we find that the population almost always converges on
a single and more accurate common answer. Debate results are also less likely
to include false facts that models are internally uncertain of. This is
because as the debate progresses, individual model instances tend to disagree
on uncertain facts and omit them from the answer (Figure 7). Lastly, we find
that debate does not just act to amplify one correct answer in a model quorum
- we find many cases where all the models initially make incorrect
predictions, but then arrive at the correct answer as debate progresses
(Figure 4,11).

We use the same methodology and prompt templates for all our tasks and require
only black-box access to language model generations – no model-internal
information such as likelihoods or gradients is needed. This allows our method
to be used with common public models serving interfaces. The method is also
orthogonal to other model generation improvements such as retrieval or prompt
engineering (in fact, we combine our debate method with zero-shot chain of
thought). While the debate process is more costly, requiring multiple model
instances and rounds, it arrives at significantly improved answers and may be
used to generate additional model training data, effectively creating a model
self-improvement loop.

To help evaluate the effect of our approach on factual accuracy, we introduce
a new benchmark and dataset evaluating factual accuracy of famous computer
scientist biographies. We find that contemporary language models have an
especially high tendency to hallucinate factually incorrect biographies, often
misrepresenting the relevant institutions and dates. Moreover, these facts
often inconsistent across different language model instances. By asking models
to come to a consensus across their answers, such inconsistent facts may be
either removed or corrected.

In summary, our work contributes the following. First, we present a novel
approach to improving factual correctness and reasoning accuracy in
contemporary language models, leveraging a multi-agent debate process between
models. Second, we introduce a new benchmark of factual correctness which
contemporary language models struggle with. Finally, we evaluate the
performance of our debate procedure in language generation, both in terms of
the number of agents, the underlying rounds of debate, and the prompts that
elicit such behavior across a set of six different reasoning and factual
accuracy tasks.

Figure 1: Multiagent Debate Improves Reasoning and Factual Accuracy. Accuracy
of traditional inference and our multi-agent debate over six benchmarks (chess
move optimality reported as a normalized score)

##  2 Language Generation through Multiagent Debate

We present an approach to generate language responses through multiagent
debate. We provide an overview of our approach in Section 2.1. We further
discuss convergence to consensus in the debate process in Section 2.2. The
overall overview of our approach is shown in Figure 2.

Figure 2: Illustration of Debate. Illustration of the debate procedure.

###  2.1 Multiagent Language Generation

Consider your work process when solving the following math question on an
exam: “What is the area of a triangle with side lengths of 3, 4, 5?". In one
thread of work, you may recognize that the triangle side-lengths directly
correspond to a right triangle, and thus directly compute the area as
$0.5\times 3\times 4=64$. To make sure that you have the right answer, you may
then try to solve the problem differently by estimating an angle $\theta$ in
the triangle using the Law of Cosines, and then obtain the area by using the
formula $0.5\times 3\times 4\times\sin(\theta)$, arriving at another answer to
the given exam problem.

When these lines of work give the same answer, your confidence about the
answer increases. In contrast, when these answers are different, individual
lines of work may engage in a mental “debate" procedure, where you closely
cross-examine the reasoning and assumptions of each line of work and refine
solutions until a consistent answer.

Similarly, consider writing a biography of a historical figure. To ensure the
factuality of the biography, you may consult multiple different sources on
each fact. Facts that are consistent in each source increase your confidence
about the fact. In contrast, facts that are inconsistent require careful
cross-examination between sources to determine the final consistent data.

To mimic the above multi-threaded reasoning process and multi-source
factuality checking processes, we propose to generate answers subject to a
multi-agent debate procedure between multiple instances of large language
models. Given a question, multiple agents represented as copies of a large
language model, generate answers to the question. Each response serves as a
possible thought process or source of information which agents may re-examine
to find consistent final answers.

After initial responses are generated from different agents, we initiate a
round of debate between agents. Individual responses from other agents are
concatenated and given as context to each agent, with each agent instructed to
construct a new response based on such responses. Each language agent is thus
responsible for both verifying the collection of responses given by other
agents, and refining its own response based on other agents’ responses. We
iteratively repeat this debate procedure over multiple rounds for improved
performance.

Concretely, we first prompt each agent to independently solve the given
problem or task. After each agent generates a response, we feed each agent a
consensus prompt, illustrated in Figure 3, where each agent is instructed to
update their responses based on the responses of other agents. This resultant
consensus prompt may then be repeatedly given, using the updated responses of
each agent. We illustrate an overview of this multiagent debate procedure in
Figure 2.

Note that our proposed approach operates in an orthogonal manner to existing
approaches to prompt language models. Given a question, we may apply
additional techniques for prompting language models to further improve our
debate procedure by eliciting additional more detailed responses from language
models. We illustrate the synergy of our approach with existing approaches to
prompting language models in Figure 6 and directly apply zero-shot chain-of-
thought reasoning in our evaluations.

Debate Length | Prompt
---|---
Short | _" These are the solutions to the problem from other agents: [other answers]_
_Based off the opinion of other agents, can you give an updated response
$\ldots$"_
Long | _" These are the solutions to the problem from other agents: [other answers]_
_Using the opinion of other agents as additional advice, can you give an
updated response $\ldots$"_

Figure 3: Prompts to induce long and short form debate. Responses of other
agents to questions are are inserted in the middle of the prompt (indicated
with _[other answers]_)

###  2.2 Consensus in Debates

Given multiple rounds of debate, how can we ensure that a set of language
model agents will converge to a final consensus answer? In general, debate can
be seen as a multi-agent game, where convergence is not guaranteed.
Empirically, however, we find that language models are able to converge on a
single shared answer after multiple rounds of debate (Figure 4).

We found that we could control the duration of debates by how changing how
much a language model trusts its own outputs over those generated by other
models through different prompts. We illustrate two prompts below in Figure 3,
which we use to induce different debate durations between language models, and
illustrate the effect of such prompts in Figure 12. In general, we found that
prompts that encouraged models to be more “stubborn’ based on their own
solutions led to longer debates and better final solutions. Overall, we
observed that language model agents were relatively "agreeable", perhaps as a
result of instruction tuning or reinforcement learning based on human feedback
[22].

##  3 Experiments

In our experiments, we evaluate our multiagent debate procedure and answer the
following questions: (1) To what extent does multiagent debate improve
reasoning? (2) To what extent does multiagent debate improve factual validity?
(3) What design choices enable multiagent debate to improve language
generation performance?

Figure 4: Illustration of Solving Math. Reasoning between agents is omitted. Figure 5: Illustration of Solving Grade School Math. Reasoning between agents omitted. Model | Arithmetic (%) $\uparrow$ | Grade School Math (%) $\uparrow$ | Chess ($\Delta$PS) $\uparrow$
---|---|---|---
Single Agent |  67.0 $\pm$ 4.7 |  77.0 $\pm$ 4.2 |  91.4 $\pm$ 10.6
Single Agent (Reflection) |  72.1 $\pm$ 4.5 |  75.0 $\pm$ 4.3 |  102.1 $\pm$ 11.9
Multi-Agent (Majority) |  69.0 $\pm$ 4.6 |  81.0 $\pm$ 3.9 |  102.2 $\pm$ 6.2
Multi-Agent (Debate) | 81.8 $\pm$ 2.3 | 85.0 $\pm$ 3.5 | 122.9 $\pm$ 7.6
Table 1: Multiagent Debate Improves Reasoning Multi-agent debate improves the
reasoning abilities of language models. Multi-agent results in the table are
run with 3 agents and two rounds of debate.

###  3.1 Improving Reasoning with Multiagent Debate

We first evaluate the extent to which multiagent debate improves the
underlying reasoning process in language models.

#### Tasks.

We evaluate our approach on three reasoning tasks of increasing difficulty:

  * •

Arithmetic. We first evaluate the ability of models to correctly evaluate an
arithmetic expression (containing addition, multiplication, and subtraction)
consisting of six different two-digit numbers. For example: What is the result
of 12+15*21+0-3*27?

  * •

GSM8K. Next, we consider harder mathematical reasoning tasks. Using the GSM8K
dataset [3], the models must correctly solve grade school mathematical
reasoning tasks.

  * •

Chess Move Prediction. Finally, we consider the strategic reasoning of the
ability of models, and ask models to predict the best next move in a game of
chess, given the first 14 moves of a chess game between two chess grand-
masters described in PGN notation [6].

We report the accuracy of final answers in arithmetic and GSM8K tasks and
report the pawn score (advantage) of predicted moves, as estimated by
Stockfish in the Chess move prediction tasks. Additional details may be found
in the Appendix.

Figure 6: Synergy with Other Methods. Performance of debate increases with use
of Chain of Thought prompting. Figure 7: Illustration of Generating
Biographies. Illustration of generating bullet biographies of computer
scientists. For brevity, only the first 3 generated bullets are shown. Figure
8: Illustration of MMLU. Illustration of debate when answering factual tasks.
Reasoning omitted.

#### Baselines.

We compare our approach to three alternative approaches to generate responses
for reasoning problems. First, we ask agents to directly generate responses
(single agent). Next, we consider asking language models to generate and then
"self-reflect" on the responses generated [26, 18]. Finally, we consider
generating responses using multiple agents and performing majority voting [15,
3]. As the focus of our experiments is to verify the effectiveness of
multiagent agent debate, we run both baselines and our approach, using the
identical starting prompt and language model across all evaluations. We
evaluate models in a zero-shot setting, with prompts found in the Appendix of
the paper. We use chatGPT-based language model [21] in all our experiments
except those in Figure 11 where we compare multiple language models.

Due to computational expense, we evaluate our approach across benchmarks
mainly using three agents with two rounds of debates, although we found
further gains with both more agents and rounds of debate (Figure 10).
Additional evaluation details are found in the Appendix.

#### Quantitative Results.

In Table 1, we report the results of each approach on arithmetic, grade school
math, and chess reasoning task. In each task, we observe that utilizing
multiple different agents to generate solutions improves performance over
using a single language model agent to generate a solution. Simultaneously, we
also see that reflection, where a language model is asked to critique its
early generation, generally gives a modest boost in performance. Multiagent
debate, which may be seen as a combination of both reflection and multiagent
generation, gives a substantial boost in reasoning across each of the tasks.

#### Qualitative Results.

In Figure 4, 5, we provide qualitative illustrations of the debate procedure
between models. Interestingly, we find cases in which all models initially
give an incorrect response, yet the result of debate still obtains the correct
answer as agents critique each others’ reasoning. Thus, the purpose of our
debate isn’t just to amplify a correct answer – all models can initially be
wrong but arrive at the correct answer through the debate process.

#### Compatibility with other reasoning methods.

Our multiagent generation procedure operates orthogonally approach to other
prompting methods which focus on single-agent generation. In Figure 6, we
illustrate the performance of multi-agent debate with and without zero-shot
chain-of-thought prompting [11] on GSM8K. In both settings, multiagent
generation is beneficial.

Model | Biographies | MMLU | Chess Move Validity
---|---|---|---
Single Agent |  66.0 $\pm$ 2.2 |  63.9 $\pm$ 4.8 |  29.3 $\pm$ 2.6
Single Agent (Reflection) |  68.3 $\pm$ 2.9 |  57.7 $\pm$ 5.0 |  38.8 $\pm$ 2.9
Multi-Agent (Debate) | 73.8 $\pm$ 2.3 | 71.1 $\pm$ 4.6 | 45.2 $\pm$ 2.9
Table 2: Multiagent Debate Improves Factual Accuracy Multi-agent debate
improves the factual accuracy. Figure 9: Expressing Uncertainty with Multiple
Answers. For facts that a language model is uncertain about, different
language agents generate different facts. Debate causes agents to converge to
one fact that is more accurate, but not necessarily always factually correct.
Figure 10: (a) Performance with Increased Agents. Performance improves as the
number of underlying agents involved in debate increases. (b) Performance with
Increased Rounds. Performance rises as the number of rounds of underlying
debate increases.

###  3.2 Extracting Factual Information from Multiagent Debate

We next evaluate the extent to which multiagent debate improves the underlying
factuality in language models.

#### Tasks.

We evaluate the factuality of language models in three different settings:

  * •

Biographies. To evaluate the factuality of language models, we introduce a
novel task of accurately generating historical biographies of people. In
preliminary testing, we found that existing language models had a tendency to
hallucinate many facts on this task. We constructed ground truth bullet point
biographies of 524 well-known computer scientists. We then asked language
models to generate bullet point biographies for each person, and evaluated the
accuracy at which each ground truth bullet point agreed with generated
bullets. We report additional evaluation details in the Appendix.

  * •

MMLU. Next, we assess the factuality of language models in responding to
different factual knowledge questions typically learned and assessed in
different exams. We utilize the existing MMLU dataset [8] to benchmark the
accuracy of responses.

  * •

Chess Move Validity. Lastly, we study the hallucinations in language models
when planning under to the given rules of an existing environment or game.
Specifically, we measure the validity of possible moves in a game of Chess
given by BIG-Bench Chess-State Tracking Benchmark [27] task of chess-move
prediction. In this task, an agent is given a set of next moves, and must make
a valid next move of a piece on a board.

#### Baselines.

We use the same baselines as in Section 3.1. The multiagent (majority) is not
directly applicable in this setting as individual responses are not easily
comparable, and so we omit baseline comparison with the majority voting in
this setting.

#### Results.

We analyze the performance of each method in Table 2. We found that approaches
based on reflection led to poor performance in the factuality setting. In
contrast, debate gives the best performance in this setting also, and
significantly outperforms each baseline. We illustrate a debate between agents
on the biography task in Figure 7 and on MMLU in Figure 8. We found that
multiagent debate improved and settled on bullets that were more consistent
across agents.

We found that different language agents tended to give different answers when
the underlying language model was uncertain about the question. However,
directly asking each agent about their confidence [10] of the answer led to
high confidence assessments on each answer. However, when these different
language agents were asked to communicate with each other, each agent would
quickly change their opinion to a consensus answer which was more accurate. We
illustrate this in Figure 9. Interestingly, we found that on facts that the
language model was confident in (i.e. many instances of the same model all
gave the same answer), it was very difficult to convince an agent to change
their opinion, suggesting that “ease of persuasion” may be a method to assess
factual confidence.

Figure 11: Debate Between chatGPT and Bard Illustration of debate between
different models.

###  3.3 Analysis: Understanding Multiagent Debate

Finally, we analyze how multiagent debate improves the underlying language
generation procedure in language models.

#### Number of Agents.

First, we analyze the impact of agents number in debate. In Figure 10(a), we
increase the number of agents used in debate, while fixing the debate length
to be two. On arithmetic, performance monotonically increases with the
increased number of agents. For larger number of agents, we first summarize
all agent responses with chatGPT instead of directly concatenating responses
due to context length error.

#### Rounds of Debate

Next, we analyze the impact of the number of rounds of debate in multiagent
debate. In Figure 10(b), we increase the debate length between agents, while
fixing the number of agents to three. We find that on the arithmetic task, the
performance also monotonically increases with debate length. However, we found
that additional debate rounds above four led to a similar final performance to
4 rounds of debate.

Figure 12: Performance vs Debate Length. Prompts which induce longer debate
improve performance.

#### Effect of Debate Length on Accuracy

As discussed in Section 2.2, the underlying convergence time in the debate
between agents can be controlled by the extent to which agents are encouraged
to maintain their opinions. In Figure 12, we consider the effect of short and
long-form prompts discussed in Figure 3. We find that debates using longer
prompts lead to slower convergence to correct answers, but also lead to a
better final consensus on the correct answer. We provide an analysis of
consensus between agents in Figure 14.

#### Using Different Initialization Prompts

In our experiments we use the same prompts for all agents. We also consider
the effect of using different questions, where we first instruct each language
model to behave like a different persona (professor, doctor, mathematician) on
the MMLU dataset. We found that improved performance on MMLU from 71.1 to 74.2
with different agents, suggesting further gains can be obtained with different
initialization prompts.

Figure 13: Effect of Summarization. When there are many agents in a debate,
responses from other agents may be first summarized and then given as context,
reducing context length. This operation improves performance.

#### Summarization.

While in the majority of experiments in the paper we directly concatenate the
responses of other agents as context for an agent to generate a new response,
this is expensive when the number of agents involved in debate gets large. We
may alternatively first summarize the responses from all other agents into a
single response that we provide to agent at each round for more efficient
debate. We apply this strategy in Figure 10 to enable the use of five or more
agents in debate. In Figure 13, we analyze the effect compared to directly
concatenating the responses of other agents. We find this improves the
performance of debate, suggesting that summarization is another tool that can
further improve multiagent debate.

#### Utilizing Different Language Models

Our existing debate results are reported using multiple instances of a chatGPT
language model. We further assess the impact of using two different language
models, where we ask chatGPT and Bard [23] language models to debate with each
other on a set of 20 GSM8K math problems. In this set, we find that multi-
agent debate improves the performance of both agents, with Bard solving 11
problems, chatGPT solving 14 problems, and joint multi-agent debate solving 17
problems. We qualitatively illustrate a debate between agents in Figure 11.
While both agents initially provide incorrect answers to the problem, chatGPT
is able to utilize the incorrect response given by Bard to generate the final
correct answer.

##  4 Related Work

#### Reasoning and Factuality in Language Models.

A wide range of work has explored how to enable reasoning and factuality in
language models. To improve reasoning, approaches have relied on prompting
techniques such as scratchpads [20], verification [3], chain-of-thought
demonstrations [30, 11, 25], and intermediate self-reflection [26, 18] and
finetuning [13, 24, 31]. To improve factuality, approaches have relied on
training techniques such as RLHF [33, 16, 2], pruning truthful datasets [12],
external knowledge retrieval [7] and training-free methods based off
likelihood estimation [10].

Our work provides an alternative way to obtain reasoning and factuality in
language models using multiagent debates, which only requires black-box access
to a language generator. Prior work also has explored how to take the majority
vote across different models [15, 3, 29, 28] while in this work, we use the
power of a language model to combine different answers. Most similar to our
work, Irving et al. [9] also proposes a debate procedure to verify the
accuracy and safety of powerful AI agents. In contrast to our approach, in
their work, agents are asked to alternatively provide proof of a input, and
humans are tasked with assessing these debates and determining safety.

#### Compositional Generation.

Our work is also related to existing works that focus on text generation by
combining different models [4, 17, 32, 1, 5]. Most similar to our work, [14,
32] propose to combine multiple different large pretrained models together for
multimodal reasoning. In contrast, in our work, we aim to use communication
between different language models to enable more effective reasoning and
factuality in language models.

##  5 Limitations and Discussion

In this paper, we present an orthogonal approach to improve the performance of
language models using multi-agent debate. We find that the approach is simple
and effective across a wide set of different reasoning and validity language
modeling tasks.

#### Limitations.

In comparison to other prompting techniques, our multiagent debate procedure
is more computationally expensive, as it requires both multiple language
generations, and an underlying debate procedure. However, we believe that this
approach may be seen as a method to generate additional data that may be
distilled back to self-improve the original base model.

Further, we observed that as debates became longer in duration, current
language models sometimes struggled to fully process the entire debate input,
and typically only focused on the most recent generations. We believe that
this performance will be alleviated with longer-context and improved language
models or by summarizing early portions of the debate.

Finally, we found that while debates typically converged into single final
answers, these answers were not necessarily correct. Despite answers being
incorrect, language models would confidently affirm that their answer is
correct and consistent with all other agent responses. We believe this result
is in part due to the fact that LMs do not correctly express their uncertainty
when generating responses, and believe that other orthogonal approaches to
improve this performance would improve the results of multiagent debate.

## References

  * Alayrac et al. [2022] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al.  Flamingo: A visual language model for few-shot learning.  _NeurIPS_ , 2022.  URL https://arxiv.org/abs/2204.14198.
  * Christiano et al. [2017] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei.  Deep reinforcement learning from human preferences.  In _Neural Information Processing Systems_ , 2017.
  * Cobbe et al. [2021] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Du et al. [2020] Y. Du, S. Li, and I. Mordatch.  Compositional visual generation with energy based models.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Du et al. [2023] Y. Du, C. Durkan, R. Strudel, J. B. Tenenbaum, S. Dieleman, R. Fergus, J. Sohl-Dickstein, A. Doucet, and W. Grathwohl.  Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc.  _arXiv preprint arXiv:2302.11552_ , 2023.
  * [6] Fsmosca.  Fsmosca/pgn-standard: Portable game notation specification and implementation guide.  URL https://github.com/fsmosca/PGN-Standard.
  * Guu et al. [2020] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang.  REALM: Retrieval-augmented language model pre-training.  _arXiv preprint arXiv:2002.08909_ , 2020.
  * Hendrycks et al. [2020] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_ , 2020.
  * Irving et al. [2018] G. Irving, P. Christiano, and D. Amodei.  Ai safety via debate.  _arXiv preprint arXiv:1805.00899_ , 2018.
  * Kadavath et al. [2022] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. H. Dodds, N. DasSarma, E. Tran-Johnson, et al.  Language models (mostly) know what they know.  _arXiv preprint arXiv:2207.05221_ , 2022.
  * Kojima et al. [2022] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa.  Large language models are zero-shot reasoners.  _arXiv preprint arXiv:2205.11916_ , 2022.
  * Lee et al. [2022] N. Lee, W. Ping, P. Xu, M. Patwary, P. N. Fung, M. Shoeybi, and B. Catanzaro.  Factuality enhanced language models for open-ended text generation.  _Advances in Neural Information Processing Systems_ , 35:34586–34599, 2022.
  * Lewkowycz et al. [2022] A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al.  Solving quantitative reasoning problems with language models.  _arXiv preprint arXiv:2206.14858_ , 2022.
  * Li et al. [2022a] S. Li, Y. Du, J. B. Tenenbaum, A. Torralba, and I. Mordatch.  Composing ensembles of pre-trained models via iterative consensus.  _arXiv preprint arXiv:2210.11522_ , 2022a.
  * Li et al. [2022b] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al.  Competition-level code generation with alphacode.  _Science_ , 378(6624):1092–1097, 2022b.
  * Liu et al. [2022a] H. Liu, L. Lee, K. Lee, and P. Abbeel.  Instruction-following agents with jointly pre-trained vision-language models.  _arXiv preprint arXiv:2210.13431_ , 2022a.
  * Liu et al. [2022b] N. Liu, S. Li, Y. Du, A. Torralba, and J. B. Tenenbaum.  Compositional visual generation with composable diffusion models.  _arXiv preprint arXiv:2206.01714_ , 2022b.
  * Madaan et al. [2023] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al.  Self-refine: Iterative refinement with self-feedback.  _arXiv preprint arXiv:2303.17651_ , 2023.
  * Minsky [1988] M. Minsky.  _Society of mind_.  Simon and Schuster, 1988.
  * Nye et al. [2021] M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al.  Show your work: Scratchpads for intermediate computation with language models.  _arXiv preprint arXiv:2112.00114_ , 2021.
  * OpenAI [2022] OpenAI.  Chatgpt: Optimizing language models for dialogue, Dec 2022.  URL https://openai.com/blog/chatgpt/.
  * Ouyang et al. [2022] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_ , 2022.
  * Pichai [2023] S. Pichai.  An important next step on our ai journey, Feb 2023.  URL https://blog.google/technology/ai/bard-google-ai-search-updates/.
  * Rajani et al. [2019] N. F. Rajani, B. McCann, C. Xiong, and R. Socher.  Explain yourself! leveraging language models for commonsense reasoning.  _arXiv preprint arXiv:1906.02361_ , 2019.
  * Reynolds and McDonell [2021] L. Reynolds and K. McDonell.  Prompt programming for large language models: Beyond the few-shot paradigm.  In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_ , pages 1–7, 2021.
  * Shinn et al. [2023] N. Shinn, B. Labash, and A. Gopinath.  Reflexion: an autonomous agent with dynamic memory and self-reflection.  _arXiv preprint arXiv:2303.11366_ , 2023.
  * Srivastava et al. [2022] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _arXiv preprint arXiv:2206.04615_ , 2022.
  * Thoppilan et al. [2022] R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, et al.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_ , 2022.
  * Wang et al. [2022] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_ , 2022.
  * Wei et al. [2022] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ , 2022.
  * Zelikman et al. [2022] E. Zelikman, Y. Wu, J. Mu, and N. Goodman.  Star: Bootstrapping reasoning with reasoning.  _Advances in Neural Information Processing Systems_ , 35:15476–15488, 2022.
  * Zeng et al. [2022] A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, et al.  Socratic models: Composing zero-shot multimodal reasoning with language.  _arXiv preprint arXiv:2204.00598_ , 2022.  URL https://arxiv.org/abs/2204.00598.
  * Ziegler et al. [2019] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.

##  Appendix A Appendix

In this appendix, we provide additional analysis and visualizations of the
debates used in the main paper in Section A.1. We further provide detailed
experimental details on each dataset in Section A.2.

###  A.1 Additional Results

Figure 14: Effect of Prompts on Consensus. Using a short debate prompt induces
faster consensus between agents

#### Consensus Between Agents.

In Figure 14, we illustrate the consensus between agents using either short or
long consensus prompts discussed in Figure 3. The use of debate prompts that
encourage agents to adapt more to the opinions of other agents improves
consensus.

#### Additional Qualitative Visualizations.

We added additional qualitative visualizations of the debate process. In
Figure 16, Figure 17, Figure 18, Figure 19, Figure 20, we illustrate debates
between agents in the GSM8K dataset which result in the correct answer. In
Figure 21, Figure 22, Figure 23, we further illustrate debates in GSM8K which
lead to the incorrect answer. We further provide an example illustration of
debate in arithmetic in Figure 24, arithmetic with summarization of individual
responses of agents in Figure 25, MMLU in Figure 26, a debate with the full
contents biographies in Figure 27, and debate in chess in Figure 28. In
general, we found that debate improved the performance of final generated
answers, though sometimes answers would converge to the incorrect value.

###  A.2 Evaluation Details

We provided detailed evaluation details for each setting in the paper. We run
all experiments using the gpt-3.5-turbo-0301 model. We provide a table listing
the prompts used to prompt models and initialize debate in Table 15.

#### Arithmetic.

To evaluate the arithmetic task, we generated six random integers for each
task between 0 and 30. We then evaluated the extent to which the correct
integer answer was correctly obtained. We evaluated models on one hundred
generated arithmetic tasks.

#### Grade School Math.

To evaluate the GSM8K task, we evaluated the accuracy at which models were
able to obtain the final correct answer, as extracted from a box. We evaluated
models on one hundred grade school math problems.

#### Chess.

To evaluate the chess reasoning task, we used chess games from
https://www.pgnmentor.com/players/Adams.zip. We asked chatGPT to predict the
next move for white to move at turn 14 and reported the relative Stockfish
pawn score with search depth 20 after executing the suggested move from
chatGPT. We evaluated models on three hundred selected chess games.

#### Biographies.

To evaluate the biographies task, we compare each generated bullet point
biography for a person with a ground truth set of facts about the person
extracted from Wikipedia. We iteratively loop through each ground truth fact,
and validate the extent to which the generated biography matches a particular
bullet by prompting chatGPT with the prompt: _Consider the following biography
of <person>: <generated biography> Is the above biography above consistent
with the fact below? <ground truth bullet> Give a single-word answer, yes, no,
or uncertain._ We then evaluate and report the percentage of ground bullets
that chatGPT returns either yes or no on. We ignored ground truth bullets that
chatGPT returns returned uncertain.

We found this evaluation metric provided a fast way to evaluate how relatively
correct a generated bullet point biography is. However, we found that
generated facts could contain incorrect information that was not captured in
the ground truth bullet and thus could not be validated through this metric.
Nevertheless, we believe this evaluation scheme estimates the relative
accuracy of a generated biography.

#### MMLU.

To evaluate MMLU, we measured the accuracy in which models were able to select
the correct multiple-choice answer in each problem. We evaluated models on one
hundred selected MMLU questions randomly distributed across each of the
subject areas.

#### Chess Validity.

To evaluate chess validity, we consider the BIG-Bench Chess-State Tracking
Benchmark [27], where we used the hardest reported task in the benchmark
synthetic_short. Each generated answer was deemed correct as long as it was
one of the valid answers in the sequence. We evaluated models of one hundred
selected chess validity tasks.

Task | Type | Prompt
---|---|---
Arithmetic | Starting | _What is the result of{}+{}*{}+{}-{}*{}? Make sure to state your answer at the end of the response._
Debate | _These are the recent/updated opinions from other agents: <other agent responses> Use these opinions_
_carefully as additional advice, can you provide an updated answer? Make sure
to state your answer_
_at the end of the response._
GSM8K | Starting | _Can you solve the following math problem? <Problem> Explain your reasoning. Your final answer_
_should be a single numerical number, in the form \boxed{{answer}}, at the end
of your response._
Debate | _These are the solutions to the problem from other agents: <other agent responses> Using the solutions_
_from other agents as additional information, can you provide your answer to
the math problem? The original_
_math problem is <Problem>. Your final answer should be a single numerical
number, in the form_
_\boxed{{answer}}, at the end of your response._
Chess | Starting | _Here is the current sequence of moves in a chess game: <moves>. What is the best chess move I should_
_execute next? Give a single move suggestion of the form 14. <XXX> and make
sure the chess move_
| _is valid in the current board state._
| Debate | _Here are other chess move suggestions from other agents: <other agent responses> Using the chess suggestions_
| _from other agents as additional advice and your earlier generated solution,
can you give me your updated thoughts_
| _on the best next chess move I should play given the chess sequence ? Give a
single move suggestion of the form_
|  | _14. <XXX> and make sure the chess move is valid in the current board state._
Biographies | Starting | _Give a bullet point biography of highlighting their contributions and achievements as a computer scientist,_
_with each fact separated with a new line character._
Debate | _Here are some bullet point biographies of <person> given by other agents: <other agent response> Closely_
_examine your biography and the biography of other agents and provide an
updated bullet point biography._
MMLU | Starting | _Can you answer the following question as accurately as possible? : A) , B) , C) , D) Explain your answer,_
_putting the answer in the form (X) at the end of your response._
Debate | _These are the solutions to the problem from other agents: <other agent responses> Using the reasoning_
_from other agents as additional advice, can you give an updated answer?
Examine your solution and_
_that other agents. Put your answer in the form (X) at the end of your
response._
Chess Validity | Starting | _Given the chess game , give one valid destination square for the chess piece at . State the destination square_
_in the form (X), where X follows the regex [a-h][1-8], for example (e5). Give
a one line explanation_
_of why your destination square is a valid move._
Debate | _Here are destination square suggestions from other agents: <other agent responses> Can you double_
_check that your destination square is a valid move? Check the valid move
justifications from other agents._
_State your final answer in a newline with a 2 letter response following the
regex [a-h][1-8]._

Figure 15: Prompts in each task. List of prompts used in each task

Figure 16: Example of a correct GSM8K Debate. Figure 17: Example of Correct
GSM8K Debate. Figure 18: Example of Correct GSM8K Debate. Figure 19: Example
of Correct GSM8K Debate. Figure 20: Example of Correct GSM8K Debate. Figure
21: Example of Incorrect GSM8K Debate. Figure 22: Example of Incorrect GSM8K
Debate. Figure 23: Example of Incorrect GSM8K Debate. Figure 24: Example of
Arithmetic Debate. Figure 25: Example of Arithmetic Debate with Summarization.
Four separate agents participate in debate, with two illustrated above.
Instruction contains the summarized responses across agents. Figure 26:
Example of MMLU Debate. Figure 27: Example of Biography Debate. While we found
that generated biographies after debate to be more accurate, many facts remain
incorrect. Figure 28: Example of Chess Debate.


* Shin et al. (2019) Shin, T., Adhikari, S., Baxter, E. J., et al. 2019, MNRAS, 487, 2900
  * Smith et al. (2017) Smith, A., Cole, S., Baugh, C., et al. 2017, MNRAS, 470, 4646
  * Soares-Santos et al. (2011) Soares-Santos, M., de Carvalho, R. R., Annis, J., et al. 2011, ApJ, 727, 45
  * Somerville & Primack (1999) Somerville, R. S., & Primack, J. R. 1999, MNRAS, 310, 1087
  * Somerville et al. (2021) Somerville, R. S., Olsen, C., Yung, L. Y. A., et al. 2021, MNRAS, 502, 4858
  * Sousbie et al. (2008) Sousbie, T., Courtois, H., Bryan, G., & Devriendt, J. 2008, ApJ, 678, 569
  * Springel et al. (2005) Springel, V., White, S. D. M., Jenkins, A., et al. 2005, Nature, 435, 629
  * Stein et al. (2020) Stein, G., Alvarez, M. A., Bond, J. R., van Engelen, A., & Battaglia, N. 2020, J. Cosmology Astropart. Phys, 2020, 012
  * Sutherland (2012) Sutherland, W. 2012, in Science from the Next Generation Imaging and Spectroscopic Surveys
  * Szepietowski et al. (2014) Szepietowski, R. M., Bacon, D. J., Dietrich, J. P., et al. 2014, MNRAS, 440, 2191
  * Tassev et al. (2013) Tassev, S., Zaldarriaga, M., & Eisenstein, D. J. 2013, Journal of Cosmology and Astro-Particle Physics, 2013, 036
  * Tinker et al. (2012) Tinker, J. L., Sheldon, E. S., Wechsler, R. H., et al. 2012, ApJ, 745, 16
  * To et al. (2020) To, C.-H., Reddick, R. M., Rozo, E., Rykoff, E., & Wechsler, R. H. 2020, ApJ, 897, 15
  * To et al. (2021) To, C.-H., Krause, E., Rozo, E., et al. 2021, MNRAS, 502, 4093
  * Tröster et al. (2019) Tröster, T., Ferguson, C., Harnois-Déraps, J., & McCarthy, I. G. 2019, MNRAS, 487, L24
  * Troxel et al. (2018) Troxel, M. A., MacCrann, N., Zuntz, J., et al. 2018, Phys. Rev. D, 98, 043528
  * Vale & Ostriker (2004) Vale, A., & Ostriker, J. P. 2004, MNRAS, 353, 189
  * Vale & Ostriker (2006) —. 2006, MNRAS, 371, 1173
  * van den Bosch & Ogiya (2018) van den Bosch, F. C., & Ogiya, G. 2018, MNRAS, 475, 4066
  * van den Bosch et al. (2018) van den Bosch, F. C., Ogiya, G., Hahn, O., & Burkert, A. 2018, MNRAS, 474, 3043
  * van den Bosch et al. (2007) van den Bosch, F. C., Yang, X., Mo, H. J., et al. 2007, MNRAS, 376, 841
  * VanderPlas et al. (2012) VanderPlas, J. T., Connolly, A. J., Jain, B., & Jarvis, M. 2012, ApJ, 744, 180
  * Varga et al. (2019a) Varga, T. N., DeRose, J., Gruen, D., et al. 2019a, MNRAS, 489, 2511
  * Varga et al. (2019b) —. 2019b, MNRAS, 489, 2511
  * Vogelsberger et al. (2020) Vogelsberger, M., Marinacci, F., Torrey, P., & Puchwein, E. 2020, Nature Reviews Physics, 2, 42
  * Watson et al. (2015) Watson, D. F., Hearin, A. P., Berlind, A. A., et al. 2015, MNRAS, 446, 651
  * Wechsler (2004) Wechsler, R. H. 2004, in Clusters of Galaxies: Probes of Cosmological Structure and Galaxy Evolution, ed. J. S. Mulchaey, A. Dressler, & A. Oemler
  * Wechsler & Tinker (2018) Wechsler, R. H., & Tinker, J. L. 2018, Annual Review of Astronomy and Astrophysics, 56, 435
  * Weinberg et al. (2013) Weinberg, D. H., Mortonson, M. J., Eisenstein, D. J., et al. 2013, Phys. Rep., 530, 87
  * Wetzel & White (2010) Wetzel, A. R., & White, M. 2010, MNRAS, 403, 1072
  * White et al. (2014) White, M., Tinker, J. L., & McBride, C. K. 2014, MNRAS, 437, 2594
  * White & Frenk (1991) White, S. D. M., & Frenk, C. S. 1991, ApJ, 379, 52
  * Yamamoto et al. (2015) Yamamoto, M., Masaki, S., & Hikage, C. 2015, arXiv e-prints, arXiv:1503.03973
  * Yan et al. (2004) Yan, R., White, M., & Coil, A. L. 2004, ApJ, 607, 739
  * Yang et al. (2020) Yang, S., Du, X., Benson, A. J., Pullen, A. R., & Peter, A. H. G. 2020, MNRAS, 498, 3902
  * Yang et al. (2003a) Yang, X., Mo, H. J., & van den Bosch, F. C. 2003a, MNRAS, 339, 1057
  * Yang et al. (2003b) —. 2003b, MNRAS, 339, 1057
  * Zehavi et al. (2011a) Zehavi, I., Zheng, Z., Weinberg, D. H., et al. 2011a, ApJ, 736, 59
  * Zehavi et al. (2011b) —. 2011b, ApJ, 736, 59
  * Zhang et al. (2019) Zhang, X., Wang, Y., Zhang, W., et al. 2019, arXiv e-prints, arXiv:1902.05965
  * Zheng et al. (2007) Zheng, Z., Coil, A. L., & Zehavi, I. 2007, ApJ, 667, 760
  * Zheng et al. (2005) Zheng, Z., Berlind, A. A., Weinberg, D. H., et al. 2005, ApJ, 633, 791
  * Zu & Mandelbaum (2015) Zu, Y., & Mandelbaum, R. 2015, MNRAS, 454, 1161


Jakub Kabat and Beata Strycharz-Szemberg
Diminished Fermat-type arrangements
and unexpected curves

The purpose of this note is to present and study a new series of the so-called unexpected curves.
They enjoy a surprising property to the effect that their degree grows to infinity,
whereas the multiplicity at a general fat point remains constant, equal $3$, which
is the least possible number appearing as the multiplicity of an unexpected curve
at its singular point. We show that additionally the BMSS dual curves inherits the
same pattern of behaviour.

14C20, 14N10, 14N20

§ INTRODUCTION

The notion of unexpected curves has been introduced by Cook II, Harbourne, Migliore and Nagel in [3].
Motivated by an example described first by Di Gennaro, Ilardi, and Vallés in [4], they observed
that there exist configurations $Z$ of points in $\P^2$ such that imposing an additional
point $P$ of multiplicity $m$ on the linear system of curves of degree $d$ vanishing at all points of $Z$ imposes
less conditions than expected. If this happens, we say that $Z$ admits an unexpected curve of degree $d$
with a point of multiplicity $m$, or shortly, that $Z$ has the $U(2;d,m)$ property. If the dimension of
the ambient space is clear to be $2$, in order to alleviate notation, we drop the first index and speak simply
of the $U(d,m)$ property.

After the appearance of [3] in 2017, the subject has attracted a lot of attention.
The foundational article [3] studies sets $Z$ with the $U(d,d-1)$ property. Bauer, Malara, Szemberg, and Szpond
in [1] discovered the first example of an unexpected surface in $\P^3$, more precisely,
they identified a set $Z$ with the $U(3;4,3)$ property. In the same paper, they observed a phenomenon
which is now known as the BMSS duality. Harbourne, Migliore, Nagel and Teitler study in [7]
more systematically higher dimensional unexpected hypersurfaces, in particular a series of such hypersurfaces attached to root systems. Unexpected cones are studied in [8]. Other constructions of unexpected hypersurfaces appear in [2, 5, 6].
In a series of papers  [9, 11, 10], Szpond shows examples
of unexpected hypersurfaces with multiple fat points and studies series of examples
building upon Fermat-type arrangements of hyperplanes. Our study is related to Fermat-type
arrangements as well.

§ INITIAL EXAMPLE

In this section, we analyse in detail an example underlying our construction.
Let $F$ be the Fermat cubic curve given by the defining equation

It is well-known that $F$ has $9$ inflection points. The coordinates of these points
can be computed explicitly intersecting $F$ with its Hessian curve $H$, which in this case splits in the union of lines given by the equation

Thus the inflection points on $F$ have coordinates

$$A_1=(1:-1:0),\;\; A_2=(1:-\eps:0),\;\ A_3=(1:-\eps^2:0),$$

$$A_4=(1:0:-1),\;\; A_5=(1:0:-\eps),\;\ A_6=(1:0:-\eps^2),$$

$$A_7=(0:1:-1),\;\; A_8=(0:1:-\eps),\;\ A_9=(0:1:-\eps^2),$$

where $\eps$ is a primitive root of the unity of order $3$.
The lines tangent to $F$ at points $A_1,\ldots,A_9$ are given by equations:

$$\ell_1:\; x+y=0,\;\; \ell_2:\; x+\eps^2y=0,\;\; \ell_3:\; x+\eps y=0,$$

$$\ell_4:\; x+z=0,\;\; \ell_5:\; x+\eps^2z=0,\;\; \ell_6:\; x+\eps z=0,$$

$$\ell_7:\; y+z=0,\;\; \ell_8:\; y+\eps^2z=0,\;\; \ell_9:\; y+\eps z=0.$$

Note that the product of these lines is the polynomial

\begin{equation}\label{eq: nine lines product}
\end{equation}

Consider the arrangement $\call_3=\left\{\ell_1,\ldots,\ell_9\right\}$. It has $3$ triple points in the coordinate points

$$X_1=(1:0:0),\;\; X_2=(0:1:0),\;\; X_3=(0:0:1)$$

and $27$ points where only $2$ configuration lines meet.
We denote the set of these $27$ points by $Y$. The union of the coordinate points is denoted by $X$.

The points in $Y$ form an almost
complete intersection. The ideal $I(Y)$ of $Y$ is generated in degree $6$ by the following three polynomials

$$f_1=x^6-y^6,\;\; f_2=x^6-z^6,\;\; f_3=(x^3+z^3)(y^3+z^3).$$

The first two polynomials define a complete intersection $W_6$ of $36$ points of the form

where $\tau$ is a primitive root of unity of order $6$ and $\alpha,\beta=1,\ldots,6$.
Our $27$ points are the set difference of $W_6$ and the complete intersection $W_3$ defined by

$$g_1=x^3-y^3,\;\; \mbox{ and }\;\; g_2=x^3-z^3.$$

Obviously, none of points in $W_3$ belongs to the set of zeroes of $f_3$, whereas
$f_3$ vanishes at all points of $Y=W_6\setminus W_3$.
We are interested in the set

\begin{equation}\label{eq: Z}
Z=Y\cup X,
\end{equation}

which associated ideal $I(Z)$ is

$$I(Z)=I(Y)\cap (xy,xz,yz).$$

The ideal $I(Z)$ is generated in degree $7$ by the following polynomials

$$h_1=x(y^6-z^6),\;\; h_2=y(x^6-z^6),\;\; h_3=z(x^3+y^3)(y^3+z^3),$$

$$h_4=z(x^3+y^3)(x^3+z^3),\;\; h_5=y(x^3+z^3)(y^3+z^3),\;\; h_6=x(x^3+z^3)(y^3+z^3).$$

Let $J$ be the ideal generated by $h_1,\ldots,h_6$. It is easy to see that

$$J\subset I(Z),$$

i.e., the polynomials $h_i$ for $i=1,\ldots, 6$ vanish at all points of $Z$.
For the reverse inclusion, let $F\in I(Z)$ be an arbitrary polynomial.
Since, in particular, $F\in I(Y)$, there are polynomials $\alpha, \beta, \gamma\in\C[x,y,z]$ such that

\begin{equation}\label{eq: def of F}
F=\alpha f_1+\beta f_2+ \gamma f_3.
\end{equation}

From $F\in I(X)$ we obtain, evaluating at $X_1$, that $\beta(1:0:0)=0$, hence
there are polynomials $\beta_1, \beta_2 \in\C[x,y,z]$ such that

\begin{equation}\label{eq: beta}
\beta=y\beta_1+z\beta_2.
\end{equation}

Analogously, evaluating at $X_2$ and $X_3$, we obtain

\begin{equation}\label{eq: alpha gamma}
\alpha=x\alpha_1 + z\alpha_2\;\mbox{ and }\; -\alpha+\gamma-\beta=x\gamma_1+y\gamma_2,
\end{equation}

for some polynomials $\alpha_1,\alpha_2,\gamma_1,\gamma_2\in\C[x,y,z]$.
Substituting (<ref>) and (<ref>) to (<ref>), we obtain

\begin{align*}
F &= x\alpha_1(y^6-z^6)+z\alpha_2(y^6-z^6)+y\beta_1(x^6-z^6)+z\beta_2(x^6-z^6)\\
&\qquad +(x\gamma_1+y\gamma_2+x\alpha_1+z\alpha_2+y\beta_1+z\beta_2)(x^3+z^3)(y^3+z^3)\\
&= \alpha_1 h_1+\beta_1 h_2+ \gamma_1 h_6+\gamma_2 h_5+\alpha_1 h_6 +\beta_1 h_5\\
&\qquad +z\alpha_2(y^3+z^3)(y^3-z^3+x^3+z^3)+z\beta_2(x^3+z^3)(x^3-z^3+y^3+z^3)\\
&= \alpha_1 h_1+\beta_1 h_2+ \gamma_1 h_6+\gamma_2 h_5+\alpha_1 h_6 +\beta_1 h_5 +\alpha_2h_3 +\beta_2 h_4.
\end{align*}

Thus $F\in J$ which completes the proof.
Lemma <ref> yields the following, immediate, consequence.

The set $Z$ defined in (<ref>) imposes independent conditions of forms of degree $7$.

Indeed, the space of homogeneous polynomials in $\C[x,y,z]$ of degree $7$ has dimension $\binom{9}{2}=36$.
There are $30$ points in $Z$, so the expected dimension of the vector space of homogeneous
polynomials of degree $7$ vanishing along $Z$ is $36-30=6$. This is equal to the actual
dimension established in Lemma <ref>.
The main result in this part is the following statement.

The set $Z$ defined in (<ref>) has the $U(7,3)$ property.

Let $P=(a:b:c)$ be a general point in $\P^2$ and let $\Gamma$ be the curve defined

\begin{align}\label{eq: gamma}
\begin{split}
\gamma=\gamma_P(x:y:z) &= a^2(5c^3-a^3)h_1 + b^2(b^3-5c^3)h_2\\
&\qquad + c^2(c^3-5a^3)h_3 + c^2(5b^3-c^3)h_4\\
&\qquad + 5b^2(a^3-c^3)h_5 + 5a^2(c^3-b^3)h_6.
\end{split}
\end{align}

Obviously $\gamma\in I(Z)$. Moreover $\mult_P\Gamma\geq 3$. This can be checked
directly computing partial derivatives of $\gamma$ of order $2$ and checking that
they all vanish at $P$. By the multiple use of the Euler formula this justifies
the claim. We omit the simple calculations.

§.§ The BMSS dual of $\Gamma$

The idea of the BMSS duality is to consider $\gamma$ as a polynomial in variables $a,b,c$
with parameters $x,y,z$ viewed as coordinates of a general point $Q=(x:y:z)$ in the
projective plane with the $a,b,c$ coordinates. Thus, reorganizing terms we have

\begin{align*}
\gamma=\gamma_Q(a:b:c) &= x(z^6-y^6)a^5+ y(x^6-z^6)b^5+ z(y^6-x^6)c^5 \\
&\qquad + 5y(x^3+z^3)(y^3+z^3)a^3b^2- 5x(x^3+z^3)(y^3+z^3)a^2b^3+ \\
&\qquad - 5z(x^3+y^3)(z^3+y^3)a^3c^2+ 5x(x^3+y^3)(z^3+y^3)a^2c^3+ \\
&\qquad + 5z(x^3+y^3)(x^3+z^3)b^3c^2- 5y(x^3+y^3)(x^3+z^3)b^2c^3.
\end{align*}

We claim that $\mult_Q\Gamma\geq 3$. This is again easy to check verifying vanishing
of all partial derivatives of order $2$ of $\gamma$ taken, this time, with respect
to variables $a,b,c$.

Let $\Lambda$ be the linear system of curves of degree $5$ generated by

$$u_1=a^2(5c^3-a^3),\;\; u_2=b^2(b^3-5c^3),\;\; u_3=c^2(c^3-5a^3),$$

$$u_4=c^2(5b^3-c^3),\;\; u_5=5b^2(a^3-c^3),\;\; u_6=5a^2(c^3-b^3).$$

These are the coefficients in (<ref>).

We establish the following easy fact.

The system $\Lambda$ is base point free.

The polynomial $u_6$ vanishes if $a=0$ or $c=\eps^\alpha b$ for some $\alpha\in\left\{0,1,2\right\}$.

Assume first that $a=0$. Then $u_3=c^5$, so it must be $c=0$. But then $u_2=b^5$ and it must be $b=0$,
which is not possible for coordinates of a point in the projective plane.

Now we turn to the second case $c=\eps^\alpha b$. Then vanishing of $u_2$ implies that $b=0$ and
proceeding as in the previous case we conclude that all coordinates vanish.
Thus $\Lambda$ is not defined by vanishing along a specific set of points in $\P^2$.
Nevertheless it has the unexpected property of having a member with a point of multiplicity $3$
in a general point, which does not follow by a naive dimension count. It would be very interesting
to understand better how $\Lambda$ appears and to investigate properties of the companion surface,
i.e., the image of $\P^2$ in $\P^5$ under the morphism defined by $\Lambda$, see <cit.>
for this path of thought.

§ GENERAL CASE

In the present section we generalize the construction exhibited in Section <ref>.
Recall, that the Fermat arrangement of lines is defined by linear factors of the polynomial

The singular points $S_m$ of this arrangement (i.e., points where $2$ or more lines intersect) are the union of
a complete intersection grid $W_m$ defined by the ideal $I(W_m)=\left((x^m-y^m),(x^m-z^m)\right)$
and the three coordinate points $X=\left\{X_1,X_2,X_3\right\}$.
We consider sets of points $Z_m$ defined as the set difference of $S_{2m}$ and $W_m$ (therefore the name diminished Fermat-type arrangements). It turns out that these sets behave surprisingly

The points in $Y_m=W_{2m}\setminus W_m$ form an almost
complete intersection. The ideal $I(Y_m)$ of $Y_m$ is generated in degree $2m$ by the following $3$ polynomials

$$f_1=x^{2m}-y^{2m},\;\; f_2=x^{2m}-z^{2m},\;\; f_3=(x^m+z^m)(y^m+z^m).$$

The proof is the same (with obvious exponent changes) as of Lemma <ref>
and therefore omitted here.

The ideal $I(Z_m)$ is generated in degree $2m+1$ by the following polynomials

$$h_1=x(y^{2m}-z^{2m}),\;\; h_2=y(x^{2m}-z^{2m}),\;\; h_3=z(x^m+y^m)(y^m+z^m),$$

$$h_4=z(x^m+y^m)(x^m+z^m),\;\; h_5=y(x^m+z^m)(y^m+z^m),\;\; h_6=x(x^m+z^m)(y^m+z^m).$$

The proof is almost verbatim as that of Lemma <ref>.
We omit it here.
The main result of this work is the following.

For $m\geq 3$ the sets $Z_m$ have the $U(2m+1,3)$ property.

It is enough to produce the equation of the unexpected curve of degree $2m+1$ explicitly.
To this end, let $P=(a:b:c)$ be a general point in $\P^2$ and let $\Gamma_m$ be the curve defined

\begin{align*}
\gamma_m=\gamma_{m,P}(x:y:z) &= a^{m-1}((2m-1)c^m-a^m)h_1 + b^{m-1}(b^m-(2m-1)c^m)h_2\\
&\qquad + c^{m-1}(c^m-(2m-1)a^m)h_3 + c^{m-1}((2m-1)b^m-c^m)h_4\\
&\qquad + (2m-1)b^{m-1}(a^m-c^m)h_5 + (2m-1)a^{m-1}(c^m-b^m)h_6.
\end{align*}

Obviously $\gamma_m\in I(Z_m)$. Moreover $\mult_P\Gamma_m\geq 3$. This can be checked
directly computing partial derivatives of $\gamma_m$ up to order $2$ and checking that
they all vanish at $P$. By a multiple use of the Euler formula we can justify
that claim – we omit here simple calculations.

§.§ The BMSS dual curve

Of course, also in the general case, we can look for the equation of $\Gamma_m$
from the perspective of coordinates $(a:b:c)$. We obtain a curve of degree $2m-1$

\begin{align*}
\gamma_m=\gamma_Q(a:b:c) &= x(z^{2m}-y^{2m})a^{2m-1}+ y(x^{2m}-z^{2m})b^{2m-1}+ z(y^{2m}-x^{2m})c^{2m-1} \\
&\qquad + (2m-1)y(x^m+z^m)(y^m+z^m)a^mb^{m-1} \\
&\qquad - (2m-1)x(x^m+z^m)(y^m+z^m)a^{m-1}b^m \\
&\qquad - (2m-1)z(x^m+y^m)(z^m+y^m)a^mc^{m-1} \\
&\qquad + (2m-1)x(x^m+y^m)(z^m+y^m)a^{m-1}c^m \\
&\qquad + (2m-1)z(x^m+y^m)(x^m+z^m)b^mc^{m-1} \\
&\qquad - (2m-1)y(x^m+y^m)(x^m+z^m)b^{m-1}c^m
\end{align*}

with a point of multiplicity $3$ at $Q=(x:y:z)$.

As in the initial case, there is a full analogy with Lemma <ref>.

The linear system $\Lambda_m$ generated by

$$u_1=a^{m-1}((2m-1)c^m-a^m),\;\; u_2=b^{m-1}(b^m-(2m-1)c^m),\;\; u_3=c^{m-1}(c^m-(2m-1)a^m),$$

$$u_4=c^{m-1}((2m-1)b^m-c^m),\;\; u_5=(2m-1)b^{m-1}(a^m-c^m),\;\; u_6=(2m-1)a^{m-1}(c^m-b^m)$$

is base point free.

The existence of a member of $\Lambda_m$ vanishing to order $3$ at a general point is unexpected.
It would be interesting to explore linear systems of this kind in a more systematic way. We formulate
it as an open problem.

Find more examples of base points free (even very ample) linear systems admitting members
with exceptional high multiplicity at a general point.

The authors thank Piotr Pokora, Tomasz Szemberg and Justyna Szpond for introducing them to this circle
of ideas and for helpful conversations. The first author was supported by the National Science Centre
grant UMO 2018/31/N/ST1/02101.

[1]
T. Bauer, G. Malara, T. Szemberg, and J. Szpond.
Quartic unexpected curves and surfaces.
Manuscripta Math., 161(3-4):283–292, 2020.

[2]
L. Chiantini and J. Migliore.
Sets of points which project to complete intersections,

[3]
D. Cook II, B. Harbourne, J. Migliore, and U. Nagel.
Line arrangements and configurations of points with an unexpected
geometric property.
Compos. Math., 154(10):2150–2194, 2018.

[4]
R. Di Gennaro, G. Ilardi, and J. Vallès.
Singular hypersurfaces characterizing the Lefschetz properties.
J. Lond. Math. Soc. (2), 89(1):194–212, 2014.

[5]
M. Dumnicki, L. u. Farnik, B. Harbourne, G. Malara, J. Szpond, and
H. Tutaj-Gasińska.
A matrixwise approach to unexpected hypersurfaces.
Linear Algebra Appl., 592:113–133, 2020.

[6]
G. Favacchio, E. Guardo, B. Harbourne, and J. Migliore.
Expecting the unexpected: quantifying the persistence of unexpected
hypersurfaces, arXiv:2001.10366.

[7]
B. Harbourne, J. Migliore, U. Nagel, and Z. Teitler.
Unexpected hypersurfaces and where to find them, arXiv:1805.10626,
accepted for publication in Michigan Math. J.

[8]
B. Harbourne, J. Migliore, and H. Tutaj-Gasińska.
New constructions of unexpected hypersurfaces in ${\mathbb P}^N$,

[9]
J. Szpond.
Unexpected curves and Togliatti–type surfaces.
Math. Nachr., 293:158–168, 2020.

[10]
J. Szpond.
Unexpected hypersurfaces with multiple fat points, arXiv:1812.04032.

[11]
J. Szpond.
Fermat-type arrangements, arXiv:1909.04089.

Jakub Kabat

Department of Mathematics, Pedagogical University of Cracow,
Podchora̧żych 2,
PL-30-084 Kraków, Poland.

E-mail address:

Beata Strycharz-Szemberg

Department of Mathematics,
Cracow University of Technology,
Warszawska 24,
PL-31-155 Kraków, Poland.

E-mail address: