# Emergence of anti-coordinated patterns in snowdrift game by reinforcement
learning

Zhen-Wei Ding***Co-first authors. Ji-Qiang Zhang†††Co-first authors.
<EMAIL_ADDRESS>Guo-Zhong Zheng  Wei-Ran Cai  Chao-Ran Cai  Li Chen  Xu-
Ming Wang  School of Physics, Ningxia University, Yinchuan 750021, P. R. China
School of Physics and Information Technology, Shaanxi Normal University,
Xi’an, 710062, P. R. China  School of Computer Science, Soochow University,
Suzhou 215006, P. R. China  School of Physics, Northwest University, Xi’an
710127, P. R. China

###### Abstract

Patterns by self-organization in nature have garnered significant interest in
a range of disciplines due to their intriguing structures. In the context of
the snowdrift game (SDG), which is considered as an anti-coordination game,
but the anti-coordination patterns are counterintuitively rare. In the work,
we introduce a model called the Two-Agents, Two-Action Reinforcement Learning
Evolutionary Game ($2\times 2$ RLEG), and apply it to the SDG on regular
lattices. We uncover intriguing phenomena in the form of Anti-Coordinated
domains (AC-domains), where different frustration regions are observed and
continuous phase transitions at the boundaries are identified. To understand
the underlying mechanism, we develop a perturbation theory to analyze the
stability of different AC-domains. Our theory accurately partitions the
parameter space into non-anti-coordinated, anti-coordinated, and mixed areas,
and captures their dependence on the learning parameters. Lastly, abnormal
scenarios with a large learning rate and a large discount factor that deviate
from the theory are investigated by examining the growth and nucleation of AC-
domains. Our work provides insights into the emergence of spatial patterns in
nature, and contributes to the development of theory for analysing their
structural complexities.

###### keywords:

Nonlinear dynamics, Anti-coordinate pattern, Evolutionary game, Reinforcement
learning

††journal: Chaos, Solitons $\&$ Fractals

s

Anti-coordinated patterns naturally emerge in snowdrift game by reinforcement
learning.

A perturbation theory is successfully developed

The effect of memory and frustration greatly influence these patterns.

##  1 Introduction

The spatial patterns by self-organization are ubiquitous in ecosystems [1, 2,
3, 4, 5], physics [6, 7, 8, 9, 10], biology [11, 12, 13, 14] and other fields
[15, 16]. The emergence of large-scale intriguing patterns in space is rooted
in the operation of positive feedback mechanisms at small scales that disrupt
uniform states, which has been extensively studied in the field of nonlinear
dynamics. As a pioneer, Turing first proposed the activator-inhibitor
principle that relies on local feedback as an explanation for the pattern
formation [17]. This principle has since been widely accepted to explain
patterns observed across various disciplines [18, 19, 20, 21]. Along this
direction, significant progress has been made in revealing various mechanisms
underlying spatial self-organization: localized disturbance-recovery processes
that give rise to power laws in cluster geometry [22], oscillating consumer-
resource interactions leading to spiral waves [23, 24], as well as scale-
dependent feedback resulting in regular patterns [25, 26]. Another direction
stems from non-equilibrium phase transitions [6, 7, 8, 27], where several
theoretical methods are also developed to analyze pattern formation [28, 29,
6, 7, 27].

In parallel, over the past fifty years, evolutionary dynamics have been
introduced and adopted as one of the main theoretical frameworks to
investigate ecosystems and human behaviors [30, 31, 32]. It is therefore
natural to utilize this framework to explore the emergence of various patterns
in both systems, where two approaches are generally utilized: continuum
partial differential equations [33, 34, 35, 36] and discrete agent-based
models [37, 38, 39]. Some researchers focus on the analysis of general
mechanisms, the stability, and diversity of these patterns [3, 4, 38].
Meanwhile, other researchers pay attention to the emergence of patterns in
specific contexts, such as cooperation [40, 41] and fairness [42, 43], among
other emergent phenomena. With the booming of network science, researchers now
have been broadening their focus from classical spatiotemporal systems to
different types of networked systems [44, 45, 46], and even systems on
hypergraphs [47, 48, 49].

In classical agent-based models, agents evolve according to the predetermined
rules, such as imitation, replication [40, 45, 50], or majority rule [51, 52],
among others [53, 54, 55]. Inspired by psychology and neuroscience,
reinforcement learning (RL) has been gradually replacing these fixed rules as
it captured the agents’ adaptivity by learning [56, 57, 58, 59]. Nonetheless,
works in this regards are still in its infancy to understand puzzling
phenomena, such as cooperation, fairness and resource optimalization [60],
aiming to provide mechanistic explanations for them. Several recent studies
have shown that RL better explains conditional cooperation and its temporal
patterns [61, 62, 63, 64, 65].

However, there is a knowledge gap of how spatial patterns could be understood
within this new framework, and what’s the underlying mechanisms in such agent-
based systems [66]. In particular, the snowdrift game (SDG) is a well-known
anti-coordination game, the emergence of anti-coordinated patterns are
expected through self-organization. Though, it is surprisingly rare to observe
such patterns within the classical models imitation. Within the RL framework,
most of studies monitor the evolution of cooperation and concentrate on the
impact of various factors on the cooperative prevalence [67, 68, 69, 70].
Therefore, we are particularly interested in the following questions: _Can
anti-coordination patterns emerge in the evolutionary RL game framework? If
so, what factors determine the types of patterns and formation processes? And
can we develop a theory of pattern formation within this framework?_
Addressing these questions is of paramount significance because it helps us
understand the emergence of patterns from the learning perspective, and could
be a cornerstone for developing theory for more complex scenarios.

This paper is structured as follows: In Sec. 2, we introduce a model called
Two-Agents Two-Actions Reinforcement Learning Evolutionary Game ($2\times 2$
RLEG), which combines the Q-learning algorithm with the framework of
evolutionary game theory. In Sec. 3, we apply the RLEG to SDG on three regular
lattices and show the emergence of Anti-Coordinated domains (AC-domains). Upon
the preliminary analysis in Sec. 4.1, we propose a perturbation theory to
analyze the stability of these domains in Sec. 4.2. Using this theory, we
partition the parameter space based on the emergence of AC-domains, and
uncover potentially continuous phase transitions at their boundaries. In Sec.
4.3, we investigate the evolution of AC-domains and uncover abnormal scenarios
due to agents’ high learning rate and expectation of future. Lastly, in Sec.
4.4, we explore the composition of agents, which determines the cooperation
preference within the population. Our conclusion and discussion are presented
in Sec. 5.

##  2 A general reinforcement learning model

Here, we present a general Two-player, Two-action Reinforcement Learning
Evolutionary Game ($2\times 2$ RLEG) model in a structured population
$\mathcal{N}$, where each agent plays the game with their nearest-neighbors
following the Q-learning algorithm [71]. Let’s denote the action set
$\mathcal{A}=\\{1\mathchar 44\relax\penalty 00\\}$, where $1$ and $0$
respectively represent cooperation (C) and defection (D). The state set for a
given player $i$ is the Cartesian product between set of neighbors action
combinations and action set, denoted as $\mathcal{S}^{i}=\\{00\mathchar
44\relax\penalty 010\mathchar 44\relax\penalty 0\cdots\mathchar
44\relax\penalty 0|\Omega^{i}|0\mathchar 44\relax\penalty 001\mathchar
44\relax\penalty 011\mathchar 44\relax\penalty 0\cdots\mathchar
44\relax\penalty 0|\Omega^{i}|1\\}$, where $\Omega^{i}$ is its neighborhood.
In Q–learning algorithm, each player has a Q-table in hand to guide their
choice of action, which is a matrix on Cartesian product for states – actions
$\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$

$\displaystyle{\bf
Q}^{i}(\tau)=\left(\begin{array}[]{cc}Q^{i}_{00,0}(\tau)&Q^{i}_{00,1}(\tau)\\\
Q^{i}_{10,0}(\tau)&Q^{i}_{10,1}(\tau)\\\ \vdots&\vdots\\\
Q^{i}_{|\Omega_{i}|0,0}(\tau)&Q^{i}_{|\Omega_{i}|0,1}(\tau)\\\
Q^{i}_{01,0}(\tau)&Q^{i}_{01,1}(\tau)\\\
Q^{i}_{11,0}(\tau)&Q^{i}_{11,1}(\tau)\\\ \vdots&\vdots\\\
Q^{i}_{|\Omega_{i}|1,0}(\tau)&Q^{i}_{|\Omega_{i}|1,1}(\tau)\end{array}\right).$
(9)

The Q-value $Q^{i}_{s,a}$ for each item is the action-value function,
measuring the value of the corresponding action $a$ for a given state $s$; a
larger value of an item $Q^{i}_{s,a}>Q^{i}_{s,\hat{a}}$ means that the
corresponding action $a$ is more preferred than the other $\hat{a}$. The
Q-table can then be interpreted as the policy the player used to make their
choice.

Initially, all players are randomly assigned C or D, and each items in their
Q-tables are also randomly and independently initialised within the range of
$[0,1]$. The evolution follows synchronous updating scheme. At each time step
$\tau$, the state is computed according to
$s^{i}(\tau)=n^{i}(\tau)a^{i}(\tau)$, where
$n^{i}(\tau)=\sum_{j\in\Omega^{i}}a^{j}(\tau-1)$. The evolutionary protocol is
divided into two processes: _gaming_ and _learning_ processes. In the gaming
process, with probability $\epsilon$, an action is randomly chosen within
$a^{i}(\tau)\in\mathcal{A}$. Otherwise, player $i$ takes action following the
guidance of its Q-table

$\displaystyle a^{i}(\tau)\rightarrow h({\bf
Q}^{i},s^{i})=\arg\max\limits_{a^{\prime}}\\{Q^{i}_{s^{i},a^{\prime}}(\tau)\\},a^{\prime}\in\mathcal{A},$
(10)

where $\arg\max\limits_{a^{\prime}}\\{Q^{i}_{s^{i},a^{\prime}}(\tau)\\}$ is
the action corresponding to the maximum Q-value in the row of state $s^{i}$.
The parameter $0<\epsilon\ll 1$ is the exploration rate, to introduce some
randomness besides the exploitation of the Q-table.

When all agents make their decisions, the rewards can then be computed
according to the payoff matrix predefined by the game as

$\displaystyle{\bf\Pi}=\left(\begin{array}[]{cc}\Pi_{11}&\Pi_{10}\\\
\Pi_{01}&\Pi_{00}\\\ \end{array}\right)=\left(\begin{array}[]{cc}R&S\\\ T&P\\\
\end{array}\right).$ (15)

Here, $\Pi_{aa^{\prime}}$ denotes $i$’s payoff if $i$ with action
$a^{i}(\tau)=a$ is against one of neighbours with the action
$a^{j}(\tau)=a^{\prime}$. Specifically, the reward received by $i$ is

$\displaystyle r^{i}(\tau)$ $\displaystyle=$
$\displaystyle\mathbbm{1}_{a_{i}=1}\left\\{\sum\limits_{j\in\Omega_{i}}a^{j}\cdot\Pi_{11}+\sum\limits_{j\in\Omega_{i}}[1-a^{j}]\cdot\Pi_{10}\right\\}+$
(16)
$\displaystyle\mathbbm{1}_{a_{i}=0}\left\\{\sum\limits_{j\in\Omega_{i}}a^{j}\cdot\Pi_{01}+\sum\limits_{j\in\Omega_{i}}[1-a^{j}]\cdot\Pi_{00}\right\\},$

in which $\mathbbm{1}_{predicate}$ denotes the random variable that is $1$ if
_predicate_ is true and 0 if it is not. By then, the gaming process is then
completed, and the new state for each player can be computed as
$s^{\prime}\equiv s^{i}(\tau+1)=(n^{i}(\tau+1),a^{i}(\tau))$, where
$n^{i}(\tau+1)=\sum_{j\in\Omega^{i}}a^{j}(\tau)$.

In the learning process, the element $Q^{i}_{s,a}$ is to revised as follows

$\displaystyle Q^{i}_{s,a}(\tau+1)$ $\displaystyle=$ $\displaystyle g({\bf
Q}^{i}(\tau),\bar{r}^{i}(\tau))$ (17) $\displaystyle=$
$\displaystyle(1-\alpha)Q^{i}_{s,a}(\tau)+\alpha\left[\gamma
Q_{s^{\prime},a^{\prime}}^{i\max}(\tau)+\bar{r}^{i}(\tau)\right].$

Here, we adopt the average reward defined as
$\bar{r}^{i}(\tau)=r^{i}(\tau)/|\Omega^{i}|$. $\alpha\in(0,1]$ is the learning
rate and $\gamma\in[0,1)$ is the discount factor. The former reflects the
strength of memory effect since a large value of $\alpha$ means that the
player is forgetful. The latter determines the importance of future rewards
since $Q_{s^{\prime},a^{\prime}}^{\max}$ is the maximum element that one can
expect within the new state $s^{\prime}$.

Input: Learning parameters: $\alpha$, $\gamma$, $\epsilon$; Payoff matrix:
$\Pi$; Population:$\mathcal{N}$

1 Initialization;

2 for  _$i$ in $\mathcal{N}$_ do

3 Pick an action randomly from $\mathcal{A}$;

4 Create a Q-table with each item in the matrix near zero;

5

6for  _$i$ in $\mathcal{N}$_ do

7 Generate state:$s^{i}\rightarrow n^{i}a^{i}$;

8

9repeat

10 for  _$i$ in $\mathcal{N}$_ do

11 Generate a random number $p$;

12 if  _$p <\epsilon$_ then

13 Pick an action randomly from $\mathcal{A}$

14 else

15 Take an action according to state, Q-table and Eq. (10)

16

17 for  _$i$ in $\mathcal{N}$_ do

18 Get reward $\bar{r}_{i}$ according to their actions and payoff matrix
$\Pi$;

19 Update Q-table according to Eq. (17) and state $s^{i}\rightarrow
n^{i}a^{i}$;

20

21

22until  _t he system becomes statistically stable or evolves for the desired
time duration_;

Algorithm 1 The Algorithm of Reinforcement Learning Evolutionary Game

The two processes repeat until the system becomes statistically stable or
evolves for the desired time duration. To summarize, the pseudo code is
provided in Algorithm 1.

##  3 Simulation results for snowdrift game

In this work, players are placed on a regular lattice, where
$|\Omega^{i}|=|\Omega|$ for any agent. They play SnowDrift Game (SDG) [72, 73]
with their nearest-neighbors within our RLRG framework. The payoff matrix for
the SDG is as follows

$\displaystyle{\bf\Pi}=\left(\begin{array}[]{cc}R&S\\\ T&P\\\
\end{array}\right)=\left(\begin{array}[]{cc}1&1-b\\\ 1+b&0\\\
\end{array}\right),$ (22)

where $b\in(0,1)$ is a tunable game parameter. Here we focus on the average
cooperation preference $\langle\bar{f}_{c}\rangle$ of the population within
the stable stage, defined as

$\displaystyle\langle\bar{f}_{c}\rangle:=\sum_{\tau=t_{0}}^{t}\frac{\sum_{i\in\mathcal{N}}\mathbbm{1}_{a^{i}(\tau)=1}}{(t-t_{0})|\mathcal{N}|},$
(23)

in which $t$ is the overall steps used for simulations and $t_{0}$ denotes the
transient time step. In this study, we investigate the cooperation preference
in different types of lattice, and examine the impact of the game parameter
$b$ and the learning parameters $\alpha$ and $\gamma$.

Figure 1: The average cooperation preference as the function of game parameter
$b$ on three different regular lattices and under combinations of learning
parameters. In (a1-a3), (b1-b3) and (c1-c3), the regular lattices are von
Neumann lattices ($|\Omega|=4$), triangular lattices ($|\Omega|=6$) and Moore
lattices ($|\Omega|=8$) respectively. Theoretically predicted Anti-Coordinated
areas (AC-areas), Non-Anti-Coordinated areas (NAC-areas) and Mixed areas
(M-areas) are depicted in blue, gray, and orange shading, respectively. In the
top panels, we identify and label the type of Anti-Coordinated domain (AC-
domain) in the corresponding AC-area, as well as the types of coexisting AC-
domains observed in the corresponding M-area. Other parameters:
$\epsilon=0.01$, and the population size $|\mathcal{N}|=10000$.

We first fix different learning parameter combination$(\alpha\mathchar
44\relax\penalty 0\gamma)$ and calculate $\langle\bar{f}_{c}\rangle$ as the
function of the game parameter $b$ for three types of lattices, as shown in
Fig. 1. These include the von Neumann lattice [Fig. 1(a1-a3)], the triangular
lattice [Fig. 1(b1-b3)], and the Moore lattice [Fig. 1(c1-c3)] [74, 75], with
the periodic boundary condition being used in all cases. Fig. 1 shows a
decline of $\langle\bar{f}_{c}\rangle$ as $b$ is increased, which is in line
with our expectation. But interestingly, there are also flat regions, where
the decrease of cooperation prevalence becomes very slow as $b$ is varied.
Detailed examination shows that the number, position, and volatility of these
flat regions are influenced by both the type of lattice and the learning
parameter combination of $(\alpha\mathchar 44\relax\penalty 0\gamma)$. These
include: (1) a larger degree $|\Omega|$ yields more flat regions; (2) the
learning rate $\alpha$ is found to have more significant impact on the
position of flat regions than $\gamma$ for the given lattice type; (3) a
larger $\alpha$ generally narrows the flat regions and makes them more
volatile.

To develop some intuition, we provide some typical snapshots as shown in Fig.
2. In most panels, we observe that locally stable domains are enclosed by
“domain walls”; Within the locally stable domains, referred to as “anti-
coordinated domains” (AC-domains), the action preference is opposite to that
of their neighbors, in analogous to spin configurations in the anti-
ferromagnetic system. Due to the different topologies of different lattices,
the unit cells $\mathcal{U}$ that make up AC-domains are different. But even
on the same topology, the configurations in AC-domains are still could be
different [see Fig. 2(c1-c3)], and we refer to as “Anti-Coordinated areas”
(AC-areas) for distinction. These snapshots immediately explain why flat
regions are seen in Fig. 1. This is because the variation in parameter $b$
only shift the domain walls whereas the configurations of AC-domains remain
largely unchanged.

Figure 2: Typical snapshots on three lattices when the evolution becomes
stable. The snapshots of actions in (a1-a3) are on the von Neumann lattices,
with game parameter being $b=0.25,0.4,0.65$ respectively. The snapshots of
actions in (b1-b3) are on the triangular lattices, with game parameter being
$b=0.2,0.52,0.9$respectively. The snapshots of actions in (c1-c3) are on the
Moore lattices, with game parameter being $b=0.2,0.65,0.9$ respectively. The
entire domains for a snapshot is made up Anti-Coordinated domains (AC-domains)
enclosed by domain walls, and disordered domains (D-domains). Here the
cooperators and defectors are encoded by  and  within the AC-domains, by  and
on the walls, and by  and  within the D-domains, respectively. Parameters:
$(\alpha\mathchar 44\relax\penalty 0\gamma)=(0.1\mathchar 44\relax\penalty
00.1)$, $\epsilon=0.01$, the population size $|\mathcal{N}|=10000$.

Different from the von Neumann lattice, there are triangular motifs in the
triangular and Moore lattices, which interestingly result in frustration in
AC-domains due to anti-coordination [see Fig. 2 (b1-b3) and (c1-c3)]. And, the
frustrated agents in AC-domains prefer cooperation for low $b$, but this
choice is gradually replaced by defection as $b$ increases. The flip of the
preference brings the diversity of AC-domains and the emergence of AC-areas on
triangular and Moore lattices as shown.

Besides AC-domains, we also observe localized _“Disordered domains”_
(D-domains) [see Fig. 2 (b2)], which is in between two AC-areas of Fig. 1. For
the combination of parameters between two AC-areas of Fig. 1, we observe the
presence of coexisting AC-domains, along with localized D-domains [see Fig. 2
(b2)]. And, the coexisting AC-domains are the dominating structure in the two
AC-areas, respectively. We call these areas as _“Mixed areas”_ (M-areas)
because domains within their snapshot consistently exhibit mixed. While the
area, where cooperation preference in the population is high, apart from the
AC-areas and M-areas, is appoint to the _“Non-Anti-Coordinated areas”_ (NAC-
areas). In sum, there are three types of configurations in the domain, namely
NAC-area, AC-areas, and M-areas. And there are subclass of AC-areas, depending
on their configuration of AC-domains.

##  4 Mechanism analysis

###  4.1 The preliminary

Figure 3:  The unit cells and basic blocks of anti-coordinated domains on
different lattices. (a) shows the unique basic unit cell on the von Neumann
lattice, with the ratio cooperators to defector $R_{c}=1:1$, appointed to
type-I. (b1-b2) give the two basic unit cells on the triangular lattice, with
$R_{c}=2:1$ and $1:2$, appointed to type-I and -II. (c1-c3) display three
basic unit cells on the Moore lattice, with $R_{c}=3:1$, $1:1$, and $1:3$,
appointed to type-I, -II, and -III. Here, the cooperator-centred block
$\mathcal{B}_{c}$ and defector-centred block $\mathcal{B}_{d}$ are indicated
with blue and yellow shadows, respectively.

In the subsection, we present a brief preliminary as the foundation for the
following subsections. The flat regions and different types of AC-domains in
Fig. 2 are due to the presence of unit cell, see Fig. 3. In Fig. 3(a),
(b1-b2), and (c1-c3), we present the unit cell for each domain type on the von
Neumann lattice, triangular lattice, and Moore lattice, respectively. For a
given type of unit cell, one can define the ratio of the number of cooperators
to the number of defectors within it,

$\displaystyle
R_{c}(\mathcal{U}_{X}):=\frac{\sum_{i\in\mathcal{U}_{X}}\mathbbm{1}_{a^{i}=1}}{\sum_{i\in\mathcal{U}_{X}}\mathbbm{1}_{a^{i}=0}},$

which reflects the cooperation preference in this unit.

Obviously, the action of a given individual is influenced by the actions of
their neighbors in the cell, which also influences the stability of the
engaged AC-domains. Here we designate $\mathcal{B}_{c}$ and $\mathcal{B}_{d}$
for the central agents in the blocks are $1$ (cooperator) and $0$ (defector),
respectively. Together with its neighbors, they form the basic blocks. In the
following study, we are going to investigate how the perturbations arising
from agents’ exploration affect the stability of basic blocks.

###  4.2 The partitioning of parameter space

The above simulations illustrate that the parameter space can be partitioned
into NAC-area, AC-areas, and M-areas, based on the character of the snapshots.
And, the locations of AC-areas play a crucial role in determining the
locations of other areas. Hence, in the subsequent analysis, we prioritize our
attention towards examining the location of different AC-areas in space.

For any AC-area, there is always a specific type of stable AC-domain. The
stability of this AC-domain is evidently dependent on the stability of its
constituent internal basic blocks $\mathcal{B}_{c}$ and $\mathcal{B}_{d}$.
Hence, our principal focus is to investigate the necessary conditions that
$\mathcal{B}_{c}$ and $\mathcal{B}_{d}$ must satisfy to attain internal
stability during exploration. These conditions also dictate the location of
the AC-area in parameter space. In our analysis, we appoint an agent’s
exploratory action as the action that differs from the action obtained from
its Q-table, i.e., $a(\tau)=1-h({\bf Q},s)$. The stability conditions for
$\mathcal{B}_{c}$ and $\mathcal{B}_{d}$ require that the structure of any
agent’s Q-table remains intact and is not disrupted by the perturbation from
its neighboring agents exploratory actions. In other words, the stability of
these blocks is contingent upon the preservation of Q-table structure under
the influence of neighboring exploratory actions.

Without loss of generality, we investigate two adjacent blocks
$\mathcal{B}_{c}$ and $\mathcal{B}_{d}$ within any specific type-$X$ AC-
domain. In this context, we assign the central cooperator in $\mathcal{B}_{c}$
to “$\tilde{1}$” and the central defector in $\mathcal{B}_{d}$ to
“$\tilde{0}$”. And for $\tilde{1}$ and $\tilde{0}$, the number of cooperators
in their neighbours are $m$ and $n$, respectively. In our analysis, we make
the assumption that $\mathcal{B}_{c}$ and $\mathcal{B}_{d}$ have achieved
stability at $\tau$. By making this assumption, we are able to establish a
self-consistent relation for stability that in turn enables us to derive
condition on stability.

Given that $\mathcal{B}_{c}$ and $\mathcal{B}_{d}$ are stable, we get the
stable Q-elements for $\tilde{1}$ and $\tilde{0}$ in $\mathcal{B}_{c}$ and
$\mathcal{B}_{d}$ at $\tau$ that are

$\displaystyle\left\\{\begin{array}[]{l}Q^{\tilde{1}}_{m1,1}(\tau)=Q^{\tilde{1}*}_{m1,1}=\displaystyle{\frac{m\Pi_{11}+\left(|\Omega|-m\right)\Pi_{10}}{|\Omega|\left(1-\gamma\right)}}\\\
Q^{\tilde{0}}_{n0,0}(\tau)=Q^{\tilde{0}*}_{n0,0}=\displaystyle{\frac{n\Pi_{01}+\left(|\Omega|-n\right)\Pi_{00}}{|\Omega|\left(1-\gamma\right)}}\end{array}\right.$
(26)

Additionally, in terms of stability, it is imperative that any exploratory
actions taken by either $\tilde{1}$ or $\tilde{0}$ at $\tau+1$ do not disrupt
the structure of their own Q-tables, as well as the Q-tables of the central
agent $\tilde{0}$’s or $\tilde{1}$’s of their adjacent block in the future.
Rather, the original configurations of $\mathcal{B}_{c}$ and $\mathcal{B}_{d}$
are expected to be recovered over steps, even if initially disrupted by the
exploratory action of $\tilde{1}$ or $\tilde{0}$. Here, we additionally
propose an assumption, $1$-step recovery, stating that following their
respective exploratory actions, the original configurations of
$\mathcal{B}_{c}$ and $\mathcal{B}_{d}$ will be recovered at the next step
$\tau+2$ [see Fig. 4]. Then, the Q-element associated with $\tilde{1}$’s or
$\tilde{0}$’s exploratory action at $\tau+1$ and $\tau+2$ can be obtained as
follows

$\displaystyle\left\\{\begin{array}[]{l}Q^{\tilde{1}*}_{m1,0}=\displaystyle{\frac{m\Pi_{01}+\left(|\Omega|-m\right)\Pi_{00}}{|\Omega|}}+\gamma
Q^{\tilde{1}*}_{m0,1}\\\
Q^{\tilde{0}*}_{n0,1}=\displaystyle{\frac{n\Pi_{11}+\left(|\Omega|-n\right)\Pi_{10}}{|\Omega|}}+\gamma
Q^{\tilde{0}*}_{n1,0}\\\ \end{array}\right.$ (29)

and

$\displaystyle\left\\{\begin{array}[]{l}Q^{\tilde{1}*}_{m0,1}=\displaystyle{\frac{m\Pi_{11}+\left(|\Omega|-m\right)\Pi_{10}}{|\Omega|}}+\gamma
Q^{\tilde{1}*}_{m1,1}=Q^{\tilde{1}*}_{m1,1}\\\
Q^{\tilde{0}*}_{n1,0}=\displaystyle{\frac{n\Pi_{01}+\left(|\Omega|-n\right)\Pi_{00}}{|\Omega|}}+\gamma
Q^{\tilde{0}*}_{n0,0}=Q^{\tilde{0}*}_{n0,0}\end{array}\right.$ (32)

Figure 4: Schematic for the assumption to $1$-step recovery of basic blocks
after exploratory actions. In (a) and (b), both blocks $\mathcal{B}_{c}$
(highlighted with a yellow shadow) and $\mathcal{B}_{d}$ (highlighted with a
blue shadow) has become stable at $\tau$ step. At $\tau+1$, the central agent
“$\tilde{1}$” or “$\tilde{0}$” takes an exploratory action, but neither
configuration of $\mathcal{B}_{c}$ and $\mathcal{B}_{d}$ can be broken by this
action in the future. As a result, both configurations will be recovered at
$\tau+2$ and sustained at $\tau+3$ in the assumption of the 1-step recovery.
In the schematic, the exploratory action for $\tilde{1}$ is represented by a
red circle with an orange border denoting defection, while the exploratory
action for $\tilde{0}$ is represented by an orange circle with a red border
denoting cooperation.

Furthermore, the assumption to $1$-step recovery lead us to conclude that the
exploratory action executed by either $\tilde{1}$ or $\tilde{0}$ will also not
break the structure of the Q-tables belonging to their neighboring agents,
$\tilde{0}$ and $\tilde{1}$ at $\tau+1$, respectively. Then, we have

$\displaystyle\left\\{\begin{array}[]{l}\begin{aligned}
&Q^{\tilde{0}}_{n0,0}(\tau+1)=(1-\alpha)Q^{\tilde{0}*}_{n0,0}+\alpha\left[\gamma
Q^{\tilde{0}*}_{(n-1)0,0}+\right.\\\
&\quad\left.\displaystyle{\frac{(n-1)\Pi_{01}+(|\Omega|-n+1)\Pi_{00}}{|\Omega|}}\right]\geq
Q^{\tilde{0}*}_{n0,1}\\\
&Q^{\tilde{1}}_{m1,1}(\tau+1)=(1-\alpha)Q^{\tilde{1}*}_{m1,1}+\alpha\left[\gamma
Q^{\tilde{1}*}_{(m+1)1,1}+\right.\\\
&\quad\left.\displaystyle{\frac{(m+1)\Pi_{11}+(|\Omega|-m-1)\Pi_{10}}{|\Omega|}}\right]\geq
Q^{\tilde{1}*}_{m1,0},\end{aligned}\end{array}\right.$ (34)

which guarantee the reconstructed blocks $\mathcal{B}_{d}$ and
$\mathcal{B}_{c}$ will be sustained in the future. Here, the
$Q^{\tilde{0}*}_{(n-1)0,0}$ and $Q^{\tilde{1}*}_{(m+1)1,1}$ can be gained
through our assumptions that are

$\displaystyle\left\\{\begin{array}[]{l}Q^{\tilde{1}*}_{(n+1)1,1}=\displaystyle{\frac{n\Pi_{11}+\left(|\Omega|-n\right)\Pi_{10}}{|\Omega|}}+\gamma
Q^{\tilde{1}*}_{n1,1}=Q^{\tilde{1}*}_{n1,1}\\\
Q^{\tilde{0}*}_{(m-1)0,0}=\displaystyle{\frac{m\Pi_{01}+\left(|\Omega|-m\right)\Pi_{00}}{|\Omega|}}+\gamma
Q^{\tilde{1}*}_{m0,0}=Q^{\tilde{0}*}_{m0,0}\\\ \end{array}\right.$ (37)

In the Fig. 4, we present a schematic representation illustrating the
schematic for the assumption to $1$-step recovery for $\mathcal{B}_{c}$ and
$\mathcal{B}_{d}$ on von Neumann networks according to the updating protocol.

For the SDGs, the payoff matrix meets $\Pi_{01}>\Pi_{11}>\Pi_{10}>\Pi_{00}$.
By substituting it into Eqs. (34-37), we obtain the conditions to the
stability for $\mathcal{B}_{c}$ and $\mathcal{B}_{d}$ that are

$\displaystyle\left\\{\begin{array}[]{l}\alpha\geq\displaystyle{\frac{m(\Pi_{01}-\Pi_{11}+\Pi_{10}-\Pi_{00})+|\Omega|(\Pi_{00}-\Pi_{10})}{\Pi_{11}-\Pi_{10}}}\\\
\alpha\leq\displaystyle{\frac{n(\Pi_{01}-\Pi_{11}+\Pi_{10}-\Pi_{00})+|\Omega|(\Pi_{00}-\Pi_{10})}{\Pi_{01}-\Pi_{00}}}\end{array}\right.$
(40)

Our findings suggest that, given our assumption, the conditions are contingent
upon both $\alpha$ and $\bf{\Pi}$, but not upon $\gamma$. The results are
similar to previous studies, which suggest that the specific type of AC-domain
that emerges depends on the memory of the agents [55]. Under our game
settings, the conditions for a given $\alpha$ can be further rewritten as

$\frac{|\Omega|-n+\alpha}{|\Omega|-\alpha}=b^{*}_{l}\leq b\leq
b^{*}_{u}=\frac{|\Omega|-m}{|\Omega|-\alpha}$ (41)

with $b\in(0,1)$, $\alpha\in(0,1)$ and $m\leq n$. Here, the lower limit
$b^{*}_{l}$ and the upper limit $b^{*}_{u}$ correspond to the minimum and
maximum game parameter required to maintain stability in the type-$X$ AC-
domain, given a specified $\alpha$. The area between $b^{*}_{l}$ and
$b^{*}_{u}$ in Eq. (41) is also the type-$X$ AC-areas in parameter space.

For an M-area between type-$X$ and type-$X^{\prime}$ AC-areas, the parameters
in space meet

$\displaystyle\frac{|\Omega|-m}{|\Omega|-\alpha}<b<\frac{|\Omega|-n^{\prime}+\alpha}{|\Omega|-\alpha}\quad\text{with}\quad
n^{\prime}=m$ (42)

in which $m$ denotes the number of neighboring cooperators surrounding the
central defector in type- $X$ AC-domain, while $n^{\prime}$ denotes the number
of neighboring cooperators surrounding the central defector in
type-$X^{\prime}$ AC-domain. Based on Eq. (42), we observe that in the M-area
context with $b^{*}_{l}<b\nleq b^{*}_{u}$ and $b^{\prime*}_{l}\nleq
b<b^{\prime*}_{u}$, type-$X$ AC-domain remains stable when the cooperators
take exploratory actions but becomes unstable when the defectors take
exploratory actions. Conversely, type-$X^{\prime}$ AC-domains remains stable
when the defectors take exploratory actions but becomes unstable when the
cooperators take exploratory actions. Therefore, we refer to both type-$X$ and
type-$X^{\prime}$ AC-domains as _“semi-stable”_ due to their dynamics under
exploratory actions. However, it’s worth noting that under the exploratory
action of cooperators, the instability of type-$X$ increases with increasing
game parameter $b$. While, under the exploratory actions of defectors, the
instability of type-$X^{\prime}$ decreases with increasing $b$.

Figure 5: The cooperation preference by simulations and boundaries predicted
by theory. The cooperation preference $\langle\bar{f}_{c}\rangle$ is color-
coded in $\alpha-b$ space and the dashed lines are the theoretical boundaries
of different _Anti-Coordinated Areas_ (AC-areas) and _Non-Anti-Coordinated
Areas_ (NAC-area), (a-c) are for von Neumann lattice, triangular lattice and
Moore lattice, respectively. The types of AC-domains in AC-areas and the types
of semi-stable AC-domains in M-areas are labeled in the corresponding areas.
These areas are delineated by dashed lines marking $b_{l}^{*}$ and $b_{r}^{*}$
in Eq. (41). Other parameters: $\gamma=0.5$, $\epsilon=0.01$ and
$|\mathcal{N}|=10000$.

Based on the analysis provided above, we can classify the domain wall and
D-domains collectively as “NonAC-domains”, denoted as $\hat{\mathcal{D}}$.
Then, the cooperation preference in the population is

$\displaystyle\langle\bar{f}_{c}\rangle=\sum_{X}p(\mathcal{D}_{X})f_{c}(\mathcal{D}_{X})+p(\hat{\mathcal{D}})f_{c}(\tilde{D})$
(43)

Here, $p(\mathcal{D}_{X})$ and $p(\hat{\mathcal{D}})$ are the proportions of
type-$X$ AC-domains and NonAC-domains respectively, and meet
$\sum_{X}p(\mathcal{D}_{X})+p(\hat{\mathcal{D}})=1$. Additionally,
$f_{c}(\mathcal{D}_{X})=R_{c}(\mathcal{U}_{X})/(1+R_{c}(\mathcal{U}_{X}))$ and
$f_{c}(\tilde{D})$ are the cooperation preferences for the agents belonging to
the corresponding domains. So, it can be inferred that the cooperation
preference $\langle\bar{f}_{c}\rangle$ for type-$X$ AC-area approximates
$f_{c}(\mathcal{D}_{X})$. Hence, the cooperation preference
$\langle\bar{f}_{c}\rangle$ can be utilized in the $\alpha-b$ space to
evaluate the validation of our analysis as Figs. 5 and 1 shows. The outcomes
confirm that the theoretical partitioning hold true in a broad sense,
encompassing von Neumann lattices, triangular lattices, and Moore lattices in
$\alpha-b$. Besides, the results in Fig. 5 and Eqs. (41-42) indicate that high
$\alpha$ leads to a narrowing of AC-areas but an expansion of M-areas, similar
to the observations in Figure 1.

###  4.3 The evolution of anti-coordinated domain

In the previous analysis, we focus exclusively on the stability of the AC-
domain, assuming that it is already formed. Yet, similar to the process of
crystallization, the formation of an AC-domain consists of two distinct
processes: nucleation and growth. There, it is also possible for the
nucleation fail to develop, or fail to grow when in a competition with the
D-domains. In our study, we will refer to the case as the “ _normal scenario_
” when the composition of the domains is consistent with our theoretical
predictions, and when the composition contradicts our theory, we term it “
_abnormal scenario_ ”. In order to monitor the evolution of nucleation and
growth, we present time series illustrating the number of AC-domains and their
average sizes in both normal and abnormal scenarios in the following.

####  4.3.1 Normal Scenarios

Figure 6: Statistics of anti-coordinated domains in the normal scenario on a
triangular lattice. We show the time series of the number and the average size
of AC-domains within the NAC-area, the AC-area, and M-area, respectively shown
in the left, middle and right panels. Parameters: $(\alpha\mathchar
44\relax\penalty 0\gamma)=(0.5,0.1)$, $|\mathcal{N}|=10000$.

In the NAC-area, the number of AC-domains decreases over time and eventually
stabilizes at a very low level, as shown in the left panels of Fig. 6(d).
Meanwhile, the average size of the AC-domains shows a slight increase but
remains largely unchanged [Fig. 6(a)]. This means that the AC-domain nucleus
fail to grow against invasion and become extremely scarce and nearly invisible
in the system. By contrast, in the AC-areas, the number of AC-domains
increases initially and then decreases, while the average size of AC-domains
steadily increases, see Fig. 6(b,e). These observations indicate that a
continuous formation of new AC-domain nuclei initially, followed by gradual
growth and merging. This means the AC-domains is able to overcome the invasion
of D-domain and dominate the population.

Within M-areas, both the number of AC-domains and the average size of AC-
domains show a consistent increase over time, and eventually stabilise as
shown in right panels [Fig. 6(c,f)]. The results indicate the continual
formation of new patches within M-areas, which also expand in size. However,
unlike the scenario in AC-areas, these semi-stable AC-domains establish a
competitive balance with D-domains during their growth. As a result, the
isolated AC-domains are surrounded by D-domains and cannot merge with each
other. This leads to the coexistence of two types of AC-domains along with
D-domains.

Figure 7: Distribution of primary anti-coordinated domain size around lower
and upper boundaries of game parameter. (a), (b1-b3), and (c1-c4) respectively
show the distributions of the size of AC-Domains on the von Neumann lattice,
triangular lattice, and Moore lattice within double-logarithmic coordinates.
The parameters of the panels are marked with the black circles in the
corresponding panels of Fig. 5. In (a), the game parameter $b=0.143$, which is
in close proximity to the lower boundary of game parameter $b_{l}^{*}$ for
type-I. In (b1-b3), $b=0.09$, $0.58$ and $0.62$, which are proximate to
$b_{l}^{*}$ and upper threshold $b_{u}^{*}$ for type-I, and $b_{l}^{*}$ for
type-II, respectively. In (c1-c4), $b=0.305$, $0.334$, $0.82$ and $0.8665$,
which are in close proximity to $b_{u}^{*}$ for type-I, $b_{l}^{*}$ and
$b_{u}^{*}$ for type-II, and $b_{l}^{*}$ for type-III. Other parameters:
$\alpha=\gamma=0.5$, $\epsilon=0.01$, and $|\mathcal{N}|=10000$.

Finally, we also provide the size distribution for primary AC-domains near
$b_{l}^{*}$ and $b_{r}^{*}$ of different AC-areas in normal scenarios. These
size distribution all follows either a power-law or an exponentially truncated
power-law distribution, regardless of the lattice structure, see Fig. 7.
Furthermore, the scaling exponent for a given $\alpha$ and the lattice type
remains the same across different $\gamma$ for most cases. This observation
may suggests that continuous phase transitions occur at $b_{l}^{*}$ and
$b_{r}^{*}$ [22], which is different from the discontinue phase transition in
the previous works [55].

####  4.3.2 Abnormal Scenarios

When both the learning rate $\alpha$ and the discount factor $\gamma$ are
large, or a large $\alpha$ combined with the frustrated lattice, abnormal
scenarios are more likely to occur according to Figs. 1 and 5. To investigate
these abnormal scenarios, we present the time series of the number of AC-
domains and their average sizes in these scenarios, as shown in Fig. 8.

Figure 8: Statistics of anti-coordinated domains in the abnormal scenarios on
a triangular lattice. We show the time series of the number of AC-domains and
of the size of AC-domains within the NAC-area, the AC-areas, and M-areas
respectively shown in the left, middle and right panels. The learning
parameters are $(\alpha\mathchar 44\relax\penalty 0\gamma)=(0.9,0.1)$,
$(0.1,0.9)$ and $(0.5,0.9)$ from left to right, respectively. Parameters:
$|\mathcal{N}|=10000$.

For a large $\alpha$ on the frustrated lattice, the temporal evolution of both
average size and number in AC-areas are similar to those in NAC-areas of
normal scenarios, showing an increase in average size and a decrease in
number, as Fig. 8 (a,b) show. The results also reveal that the AC-domain
nuclei exhibit instability, fail to maintain the domain growth. This
observation may explain the similarity in the trend of average cooperation
preference with the increase of $b$ between these scenarios and the normal
scenarios of NAC-areas. The reason why a large $\alpha$ leads to abnormal
scenarios is that agents in this case tend to be more responsive to the
actions of their neighbors. This increased sensitivity further amplifies the
challenge of nucleation and growth of AC-domains, especially in the frustrated
lattices.

For the case where a large $\gamma$ is combined with large $b$, we
respectively select the parameters in the theoretical AC-areas and M-areas as
depicted in the middle panels [Fig. 8 (b) and (e)] and right panels [(c) and
(f)]. Similar to the normal scenarios of M-areas, both the number and average
size are increased till stabilized. The observations demonstrate the nucleus
of AC-domains are able to form and grow in the competition with D-domains. As
a result, the AC-domains are unable to assert their dominance over the system.
The results indicate a large $\gamma$ combined with a large $b$ may enhance
the competitive advantage of D-domains. In fact, the abnormal scenarios being
attributed to the fact that our assumptions regarding $1$-step recovery may
not hold true, as the $\gamma$ determines the significance of future rewards.
However, $1$-step recovery assumes agents to be short-sighted.

###  4.4 The compositions of population

As Eq. (43) shows, the average cooperation preference
$\langle\bar{f}_{c}\rangle$ in population results from the compositions of
different AC-domains and NonAC-domains. Here, we further provide detailed
compositions of different AC-domains, and compositions of cooperators and
defectors for NonAC-domains, see Fig. 9. The top panels in Figure 9(a1-c1)
depict the normal scenarios that align with our theoretical predictions. These
scenarios can be summarized as follows: 1) The AC-domains are nearly invisible
in the NAC-area; 2) Each AC-area is dominated by a specific type-X AC-domains;
3) Within an M-area between type-$X$ and type-$X^{\prime}$, which act as
transition area between AC-areas, two types of AC-domains coexist alongside
NonAC-domains due to the semi-stability of both corresponding AC-domains.
Besides, as the parameter $b$ increases in M-areas, one type-$X^{\prime}$ AC-
domain gradually replaces type-$X$.

Figure 9: Compositions of AC-domains and cooperators/defectors in Non-Anti-
coordinated domains. Here we distinguish different AC-domains as
$\mathcal{D}_{X}$, where $X$ represents the type of the corresponding AC-
domain. The individuals who cooperate and defect in the Non-AC domain are
denoted as $\hat{\mathcal{D}}_{c}$ and $\hat{\mathcal{D}}_{d}$, respectively.
Their proportions in the population correspond to the filling proportions in
each panel. The combinations of learning rate and discounting rate,
$(\alpha\mathchar 44\relax\penalty 0\gamma)$, from top to bottom are
$(0.5\mathchar 44\relax\penalty 00.1)$, $(0.9\mathchar 44\relax\penalty 00.1)$
and $(0.5\mathchar 44\relax\penalty 00.9)$. From left to right panel, the
population topology corresponds to the von Neumann lattice, triangular
lattice, and Moore lattice, respectively. Other parameters: $\epsilon=0.01$
and $|\mathcal{N}|=10000$.

The middle panels, especially Fig. 9 (b2-c2), highlight the abnormal scenarios
arising from large learning rates $\alpha$ in frustrated lattices. In such
abnormal scenarios, the AC-domains fail to survive or flourish when confronted
with the invasion of D-domains. Therefore, the proportion of any specific type
of domain is very low within the population. In addition, the results suggest
that frustrated lattices may be a necessary condition for the emergence of
such abnormal scenarios, as no such scenario is observed in the von Neumann
lattice.

But for the combination of large $\gamma$ and large game parameters $b$, the
abnormal scenarios appear to be irrelevant to whether the lattice is
frustrated or not. This observation is highlighted in the bottom panels [Fig.
9 (a3-c3)]. The occurrence of this phenomenon may be attributed to the
violation of the assumption of $1$-step recovery, which is independent of the
lattice as mentioned before. Furthermore, the results also show that the
cooperation preference in Non-AC domains exhibits a linear decrease as $b$
increases.

##  5 CONCLUSION AND DISCUSSION

In this study, we propose a model of Two-Agent Two-Action Reinforcement
Learning Evolutionary Game ($2\times 2$ RLEG) and is applied to the snowdrift
game (SDG) on three lattices. Interestingly, we observe the emergence of
distinct and diverse anti-coordinated domains (AC-domains), which are expected
but rarely seen in previous studies. The emergence and types of these AC-
domains depend on the learning parameters and the game parameter.
Specifically, we can observe non-anti-coordination, anti-coordination, and
mixed cases within the Non-Anti-Coordinated area (NAC-area), Anti-Coordinated
areas (AC-areas), and Mixed areas (M-areas) of the parameter space,
respectively. Furthermore, AC-areas can be further categorized based on their
types of AC-domains. In the NAC-area, the occurrence of AC-domains is scarce.
In contrast, within the AC-areas, only a specific type of AC-domain
dominating. Whereas, within the M-areas located between AC-areas, we observe
the coexistence of AC-domains accompanied by the presence of Disorder domains
(D-domains).

To understand these observations, we developed a perturbation theory to
investigate the stability of different types of AC-domains. Our theory
successfully predicts different areas observed in simulations, and
specifically we have the following findings: 1) Within the NAC-areas, no
single type of AC-domain is stable. 2) In each AC-area, only a specific type
of AC-domain exhibits stability. 3) However, within the M-areas located
between AC-areas, we observe the presence of two semi-stable types of ACDs.
Besides, in the “ _normal scenarios_ ” conforming to the theory, our
simulations further reveal there may be continuous phase transitions at the
partitioning points between the areas [22], which is different from the
discontinuous phase transition in the previous works [55].

Our study also reveals the specific “ _abnormal scenarios_ ”, where the stable
AC-domains are expected to dominate theoretically, but this is not the case or
even fail to emerge in simulations. Based on the analysis of the nucleation
and growth of AC-domains, we discover that agents with a large learning rate
are more sensitive and responsive to the actions of their neighbors. This
further makes the nucleation difficult, particularly with frustrated lattices.
Moreover, the agents’ high expectation about the future deviates from our
assumptions in theory and ultimately give rise to some abnormal scenarios.

Although our study provides new insights within the RLEG frame, there are
still several open questions remained. Specifically, in order to fully
understand the behavior of RLEG systems, it will be necessary to develop more
sophisticated theories that incorporate the complex topologies of the real
population [76, 77]. Furthermore, the computational complexity of RLEG is
significantly higher compared to classical evolutionary games. This poses a
challenge in identifying the type of phase transition and computing the
associated critical exponents, especially conducting finite-size scaling [78,
79]. Therefore, improving the computational efficiency is crucial for
simulating large-scale system of RLEGs. Finally, our work may act as a good
starting point for studying spatiotemporal pattern emergence in RLEGs, but
many important questions remain for future research.

## Acknowledgments

We are supported by the Natural Science Foundation of China under Grants No.
12165014 and 12075144, and the Key Research and Development Program of Ningxia
Province in China under Grant No. 2021BEB04032.

##  Appendix A The comparison between normal scenarios and abnormal scenarios

In this section, we compare the time series of the number and average size of
Anti-Coordinated Domains (AC-domains) on different regular lattices between
normal and abnormal scenarios, as shown in Figs. 10 and 11. In normal
scenarios, regardless of the type of lattice, the panels show an initial
increase followed by a decrease in the number of AC-domains over time.
However, the average size of AC-domains consistently increases and remains
unchanged at the end. These results confirm our analysis in Fig. 6 of Sec.
4.3.1 that AC-domains undergo three processes: nucleation, growth, and
eventual merger.

In abnormal scenarios where agents exhibit a high learning rate $\alpha$ on
the frustrated lattice, both the number and average size of AC-domains remain
unchanged over time [referring to (b2-b3) and (c1-c2)]. These results
reinforce the findings in Sec. 4.3.2 that randomly generated nuclei of AC-
domains lack stability. Agents with a high learning rate are overly sensitive
to their neighbors’ actions, compromising the stability of their Q-table
structure.

In another abnormal scenario where agents have high expectations for the
future ($\gamma$), the number of AC-domains initially increases followed by a
small decrease or remaining unchanged, while the average size of AC-domains
consistently increases and remains unchanged at the end. The presence of high
expectations for the future may lead to decreased stability of AC-domains when
faced with the invasion of Disordered domains (D-domains). However, over time,
a balance is established to counteract the invasion and maintain stability.
The abnormal scenarios deviating theory is due to the invalid of the
assumption in our theory.

Figure 10: The comparisons of the time series of the number of AC-domains
between normal scenarios and abnormal scenarios on different lattices. In
Panel (a), the comparison is made on von Neumann lattices with a game
parameter $b=0.95$. In Panels (b1-b3), the comparisons are on a triangular
lattice with corresponding $b=0.8$, $0.28$ and $0.75$. In Panels (c1-c2), the
comparisons are on a Moore lattice with corresponding $b=0.2$, $0.32$. In each
panel, the upper part corresponds to the results in normal scenarios, while
the lower part represents the results in abnormal scenarios. The sharing
learning parameters in all panels are exploration rate: $\epsilon=0.01$ and
the scale of the population: $|\mathcal{N}|=10000$. Figure 11: The comparisons
of the time series of the average size of AC-domains between normal scenarios
and abnormal scenarios on different lattices. The parameters in the panels are
identical to the corresponding panels in Fig. 10. In each panel, the upper
part corresponds to the results in normal scenarios, while the lower part
represents the results in abnormal scenarios.

In Fig. 12, we present additional snapshots of abnormal scenarios on different
lattices after the system has reached stability. As depicted in the top panels
(a1-c1), the coexistence of AC-domains and D-domains is observed in the
abnormal scenarios caused by the high future expectations of agents. Besides,
we further notice that the average size of AC-domains on the von Neumann
lattice is larger compared to the triangular and Moore lattices. This finding
reinforces the notion of a competitive balance between AC-domains and
D-domains, but it also suggests that the frustration effect might contribute
to the increased instability of AC-domains.

In abnormal scenarios resulting from the high learning rate of agents, we do
not observe the actual patterns of AC-domains. Instead, we only see randomly
generated nuclei of AC-domains in the snapshots, as depicted in panels (a2-c2)
and (b3-c3). This indicates that the nuclei of AC-domains are unstable and
cannot grow into their intended patterns in such scenarios.

Figure 12: The final snapshots of abnormal scenarios on different lattices. In
panels (a1-a3), the snapshots show the agent’s actions on von Neumann
lattices. The corresponding panel parameters are $(\alpha\mathchar
44\relax\penalty 0\gamma\mathchar 44\relax\penalty 0b)=(0.1\mathchar
44\relax\penalty 00.9\mathchar 44\relax\penalty 00.95)$, $(0.9\mathchar
44\relax\penalty 00.1\mathchar 44\relax\penalty 00.1)$, and $(0.9\mathchar
44\relax\penalty 00.1\mathchar 44\relax\penalty 00.75)$. In panels (b1-b3),
the snapshots show the agent’s actions on triangular lattices. The
corresponding panel parameters are $(\alpha\mathchar 44\relax\penalty
0\gamma\mathchar 44\relax\penalty 0b)=(0.1\mathchar 44\relax\penalty
00.9\mathchar 44\relax\penalty 00.8)$, $(0.9\mathchar 44\relax\penalty
00.1\mathchar 44\relax\penalty 00.28)$ and $(0.9\mathchar 44\relax\penalty
00.1\mathchar 44\relax\penalty 00.75)$. In panels (c1-c3), the snapshots show
the agent’s actions on Moore lattices. The corresponding panel parameters are
$(\alpha\mathchar 44\relax\penalty 0\gamma\mathchar 44\relax\penalty
0b)=(0.1\mathchar 44\relax\penalty 00.9\mathchar 44\relax\penalty 00.9)$,
$(0.9\mathchar 44\relax\penalty 00.1\mathchar 44\relax\penalty 00.2)$ and
$(0.9\mathchar 44\relax\penalty 00.1\mathchar 44\relax\penalty 00.32)$. In the
panels, the cooperators and defectors are marked by  and  , respectively,
within the AC-domains, by  and  on the walls, and by  and  within the
D-domains. The sharing learning parameter in (a-c) are exploration rate:
$\epsilon=0.01$ and the scale of the population: $|\mathcal{N}|=10000$.

##  Appendix B The robustness of anti-coordinated pattern

Figure 13: The comparisons of the time series of the average size of AC-
domains between normal scenarios and abnormal scenarios on different lattices.
The parameters in the panels are identical to the corresponding panels in Fig.
10. In each panel, the upper part corresponds to the results in normal
scenarios, while the lower part represents the results in abnormal scenarios.

Here, we further investigate the robustness of the anti-coordination pattern
across different structured populations. For comparison with the von Neumann
lattice, we assign agents to regular random networks [80, 81], ensuring each
agent has the same number of neighbors as in the von Neumann lattice. As shown
in Figure 13 (a1), we observe that the average cooperation preference, as a
function of game parameter, exhibits some “ _flat areas_ ” when the agents
have a low learning rate. Based on the analysis in the main text, we can
reasonably conclude that these flat areas result from the anti-coordinated
effects, indicating the robustness of the anti-coordination phenomenon across
different populations. However, unlike the regular lattice, these flat areas
disappear as the learning rate equipped by agents increases [see (a2) in Fig.
13 and (a2) in Fig. 1]. This suggests that the anti-coordination phenomenon is
more vulnerable to the learning rate on regular random networks.

In addition, we also observed some abnormal scenarios similar to those
described in the main text. Specifically, we found that a high value of
$\alpha$ leads to increased volatility of the flat areas as game parameter $b$
changes, given a low expectation of future $\gamma$. Furthermore, a
combination of high $\gamma$ and high game parameter $b$ can exacerbate the
deterioration of the anti-coordination phenomenon [see Fig. 13 (a1) and (a3)].
The results demonstrate that phenomena observed on the lattice may be general
in structured population.

## References

  * Rietkerk and Van de Koppel [2008] M. Rietkerk, J. Van de Koppel,  Regular pattern formation in real ecosystems,  Trends in Ecology & Evolution 23 (2008) 169–175. doi:10.1016/j.tree.2007.10.013.
  * Meron [2012] E. Meron,  Pattern-formation approach to modelling spatially extended ecosystems,  Ecological Modelling 234 (2012) 70–82. doi:10.1016/j.ecolmodel.2011.05.035.
  * Dobramysl et al. [2018] U. Dobramysl, M. Mobilia, M. Pleimling, U. C. Täuber,  Stochastic population dynamics in spatially extended predator–prey systems,  Journal of Physics A: Mathematical and Theoretical 51 (2018) 063001. doi:10.1088/1751-8121/aa95c7.
  * Reichenbach et al. [2007] T. Reichenbach, M. Mobilia, E. Frey,  Mobility promotes and jeopardizes biodiversity in rock–paper–scissors games,  Nature 448 (2007) 1046–1049. doi:10.1038/nature06095.
  * Widder et al. [2016] S. Widder, R. J. Allen, T. Pfeiffer, T. P. Curtis, C. Wiuf, W. T. Sloan, O. X. Cordero, S. P. Brown, B. Momeni, W. Shou, et al.,  Challenges in microbial ecology: building predictive understanding of community function and dynamics,  The ISME Journal 10 (2016) 2557–2568. doi:10.1038/ismej.2016.45.
  * Cross and Hohenberg [1993] M. C. Cross, P. C. Hohenberg,  Pattern formation outside of equilibrium,  Reviews of Modern Physics 65 (1993) 851. doi:10.1103/RevModPhys.65.851.
  * Gollub and Langer [1999] J. P. Gollub, J. S. Langer,  Pattern formation in nonequilibrium physics,  Reviews of Modern Physics 71 (1999) S396. doi:10.1103/RevModPhys.71.S396.
  * Lebon et al. [2008] G. Lebon, D. Jou, J. Casas-Vázquez, Understanding non-equilibrium thermodynamics, volume 295, Springer, 2008.
  * Staliunas et al. [2002] K. Staliunas, S. Longhi, G. J. De Valcárcel,  Faraday patterns in bose-einstein condensates,  Physical Review Letters 89 (2002) 210406. doi:10.1103/PhysRevLett.89.210406.
  * Battiston et al. [2020] F. Battiston, G. Cencetti, I. Iacopini, V. Latora, M. Lucas, A. Patania, J.-G. Young, G. Petri,  Networks beyond pairwise interactions: Structure and dynamics,  Physics Reports 874 (2020) 1–92. doi:10.1016/j.physrep.2020.05.004.
  * Koch and Meinhardt [1994] A.-J. Koch, H. Meinhardt,  Biological pattern formation: from basic mechanisms to complex structures,  Reviews of Modern Physics 66 (1994) 1481. doi:10.1103/RevModPhys.66.1481.
  * Wong et al. [2016] R. Wong, S. Geyer, W. Weninger, J.-C. Guimberteau, J. K. Wong,  The dynamic anatomy and patterning of skin,  Experimental Dermatology 25 (2016) 92–98. doi:10.1111/exd.12832.
  * Meinhardt and Meinhardt [1982] H. Meinhardt, H. Meinhardt, Models of biological pattern formation, volume 6, Academic Press London, 1982\.
  * Chen et al. [2017] L. Chen, F. Ghanbarnejad, D. Brockmann,  Fundamental properties of cooperative contagion processes,  New Journal of Physics 19 (2017) 103041. doi:10.1088/1367-2630/aa8bd2.
  * Walgraef [2012] D. Walgraef, Spatio-temporal pattern formation: with examples from physics, chemistry, and materials science, Springer Science & Business Media, 2012\.
  * Resnik [1997] M. D. Resnik, Mathematics as a Science of Patterns, Oxford University Press, 1997\.
  * Turing [1990] A. M. Turing,  The chemical basis of morphogenesis,  Bulletin of Mathematical Biology 52 (1990) 153–197. doi:10.1016/s0092-8240(05)80008-4.
  * Rovinsky and Menzinger [1993] A. B. Rovinsky, M. Menzinger,  Self-organization induced by the differential flow of activator and inhibitor,  Physical Review Letters 70 (1993) 778. doi:10.1103/physrevlett.70.778.
  * Klausmeier [1999] C. A. Klausmeier,  Regular and irregular patterns in semiarid vegetation,  Science 284 (1999) 1826–1828. doi:10.1126/science.284.5421.1826.
  * Sun [2012] G.-Q. Sun,  Pattern formation of an epidemic model with diffusion,  Nonlinear Dynamics 69 (2012) 1097–1104. doi:10.1007/s11071-012-0330-5.
  * Chen [2019] L. Chen,  Persistent spatial patterns of interacting contagions,  Physical Review E 99 (2019) 022308. doi:10.1103/PhysRevE.99.022308.
  * Pascual and Guichard [2005] M. Pascual, F. Guichard,  Criticality and disturbance in spatial ecological systems,  Trends in Ecology & Evolution 20 (2005) 88–95. doi:10.1016/j.tree.2004.11.012.
  * Hassell et al. [1991] M. P. Hassell, H. N. Comins, R. M. Mayt,  Spatial structure and chaos in insect population dynamics,  Nature 353 (1991) 255–258. doi:10.1038/353255a0.
  * Thompson et al. [2012] R. M. Thompson, U. Brose, J. A. Dunne, R. O. Hall, S. Hladyz, R. L. Kitching, N. D. Martinez, H. Rantala, T. N. Romanuk, D. B. Stouffer, et al.,  Food webs: reconciling the structure and function of biodiversity,  Trends in Ecology & Evolution 27 (2012) 689–697. doi:10.1016/j.tree.2012.08.005.
  * Levin [1998] S. A. Levin,  Ecosystems and the biosphere as complex adaptive systems,  Ecosystems 1 (1998) 431–436. doi:10.1007/s100219900037.
  * Preiser et al. [2018] R. Preiser, R. Biggs, A. De Vos, C. Folke,  Social-ecological systems as complex adaptive systems,  Ecology and Society 23 (2018). doi:10.5751/ES-10558-230446.
  * Brown and Pleimling [2017] B. L. Brown, M. Pleimling,  Coarsening with nontrivial in-domain dynamics: Correlations and interface fluctuations,  Physical Review E 96 (2017) 012147. doi:10.1103/physreve.96.012147.
  * Langer [1980] J. S. Langer,  Instabilities and pattern formation in crystal growth,  Reviews of Modern Physics 52 (1980) 1. doi:10.1103/revmodphys.52.1.
  * Sattinger [1978] D. Sattinger,  Group representation theory, bifurcation theory and pattern formation,  Journal of Functional Analysis 28 (1978) 58–101. doi:10.1016/0022-1236(78)90080-0.
  * Alberti et al. [2020] M. Alberti, E. P. Palkovacs, S. D. Roches, L. D. Meester, K. I. Brans, L. Govaert, N. B. Grimm, N. C. Harris, A. P. Hendry, C. J. Schell, et al.,  The complexity of urban eco-evolutionary dynamics,  BioScience 70 (2020) 772–793. doi:10.1093/biosci/biaa079.
  * Dietz and Burns [1992] T. Dietz, T. R. Burns,  Human agency and the evolutionary dynamics of culture,  Acta Sociologica 35 (1992) 187–200. doi:10.1177/000169939203500302.
  * Rogalski et al. [2017] M. A. Rogalski, C. D. Gowler, C. L. Shaw, R. A. Hufbauer, M. A. Duffy,  Human drivers of ecological and evolutionary dynamics in emerging and disappearing infectious disease systems,  Philosophical Transactions of the Royal Society B: Biological Sciences 372 (2017) 20160043\. doi:10.1098/rstb.2016.0043.
  * Neuhauser [2001] C. Neuhauser,  Mathematical challenges in spatial ecology,  Notices of the AMS 48 (2001) 1304–1314.
  * Holmes et al. [1994] E. E. Holmes, M. A. Lewis, J. Banks, R. Veit,  Partial differential equations in ecology: spatial interactions and population dynamics,  Ecology 75 (1994) 17–29. doi:10.2307/1939378.
  * Turner et al. [2001] M. G. Turner, R. H. Gardner, R. V. O’neill, R. V. O’Neill, Landscape ecology in theory and practice, volume 401, Springer, 2001.
  * Cantrell and Cosner [2004] R. S. Cantrell, C. Cosner, Spatial ecology via reaction-diffusion equations, John Wiley & Sons, 2004.
  * Packard and Wolfram [1985] N. H. Packard, S. Wolfram,  Two-dimensional cellular automata,  Journal of Statistical Physics 38 (1985) 901–946. doi:10.1007/bf01010423.
  * Esmaeili et al. [2018] S. Esmaeili, B. L. Brown, M. Pleimling,  Perturbing cyclic predator-prey systems: How a six-species coarsening system with nontrivial in-domain dynamics responds to sudden changes,  Physical Review E 98 (2018) 062105. doi:10.1103/PhysRevE.98.062105.
  * Szolnoki et al. [2014] A. Szolnoki, M. Mobilia, L.-L. Jiang, B. Szczesny, A. M. Rucklidge, M. Perc,  Cyclic dominance in evolutionary games: a review,  Journal of the Royal Society Interface 11 (2014) 20140735. doi:10.1098/rsif.2014.0735.
  * Nowak and May [1992] M. A. Nowak, R. M. May,  Evolutionary games and spatial chaos,  Nature 359 (1992) 826–829. doi:10.1038/359826a0.
  * Nowak and May [1993] M. A. Nowak, R. M. May,  The spatial dilemmas of evolution,  International Journal of Bifurcation and Chaos 3 (1993) 35–78. doi:10.1142/s0218127493000040.
  * Lamba and Mace [2013] S. Lamba, R. Mace,  The evolution of fairness: Explaining variation in bargaining behaviour,  Proceedings of the Royal Society B: Biological Sciences 280 (2013) 20122028\. doi:10.1098/rspb.2012.2028.
  * André and Baumard [2011] J.-B. André, N. Baumard,  The evolution of fairness in a biological market,  Evolution 65 (2011) 1447–1456. doi:10.1111/j.1558-5646.2011.01232.x.
  * Asllani et al. [2014] M. Asllani, D. M. Busiello, T. Carletti, D. Fanelli, G. Planchon,  Turing patterns in multiplex networks,  Physical Review E 90 (2014) 042814. doi:10.1103/physreve.90.042814.
  * Allen et al. [2017] B. Allen, G. Lippner, Y.-T. Chen, B. Fotouhi, N. Momeni, S.-T. Yau, M. A. Nowak,  Evolutionary dynamics on any population structure,  Nature 544 (2017) 227–230. doi:10.1038/nature21723.
  * Nakao and Mikhailov [2010] H. Nakao, A. S. Mikhailov,  Turing patterns in network-organized activator–inhibitor systems,  Nature Physics 6 (2010) 544–550. doi:10.1038/nphys1651.
  * Carletti et al. [2020] T. Carletti, D. Fanelli, S. Nicoletti,  Dynamical systems on hypergraphs,  Journal of Physics: Complexity 1 (2020) 035006. doi:10.1088/2632-072x/aba8e1.
  * Carletti and Fanelli [2022] T. Carletti, D. Fanelli,  Pattern formation on hypergraphs,  in: Higher-Order Systems, Springer, 2022, pp. 163–180. doi:10.1007/978-3-030-91374-8_5.
  * Wang and Liu [2023] W. Wang, S. Liu,  Spatio-temporal patterns of non-autonomous systems on hypergraphs: Turing and benjamin–feir mechanisms,  New Journal of Physics 25 (2023) 023008. doi:10.1088/1367-2630/acb5e1.
  * Ma et al. [2023] L. Ma, J. Zhang, G. Zheng, R. Liang, L. Chen,  Emergence of cooperation in a population with bimodal response behaviors,  Chaos, Solitons & Fractals 171 (2023) 113452. doi:10.1016/j.chaos.2023.113452.
  * Lima et al. [2022] J. Lima, F. Lima, T. Alves, G. Alves, A. Macedo-Filho,  Diffusive majority-vote model,  Physical Review E 105 (2022) 034105. doi:10.1103/PhysRevE.105.034105.
  * Crokidakis and de Oliveira [2012] N. Crokidakis, P. M. C. de Oliveira,  Impact of site dilution and agent diffusion on the critical behavior of the majority-vote model,  Physical Review E 85 (2012) 041147. doi:10.1103/physreve.85.041147.
  * Zhang et al. [2016] J.-Q. Zhang, Z.-G. Huang, Z.-X. Wu, R. Su, Y.-C. Lai,  Controlling herding in minority game systems,  Scientific Reports 6 (2016) 20925. doi:10.1038/srep20925.
  * Huang et al. [2012] Z.-G. Huang, J.-Q. Zhang, J.-Q. Dong, L. Huang, Y.-C. Lai,  Emergence of grouping in multi-resource minority game dynamics,  Scientific Reports 2 (2012) 703. doi:10.1038/srep00703.
  * Wang et al. [2006] W.-X. Wang, J. Ren, G. Chen, B.-H. Wang,  Memory-based snowdrift game on networks,  Physical Review E 74 (2006) 056113. doi:10.1103/physreve.74.056113.
  * Song et al. [2022] Z. Song, H. Guo, D. Jia, M. Perc, X. Li, Z. Wang,  Reinforcement learning facilitates an optimal interaction intensity for cooperation,  Neurocomputing 513 (2022) 104–113. doi:10.1016/j.neucom.2022.09.109.
  * Wang et al. [2022] L. Wang, D. Jia, L. Zhang, P. Zhu, M. Perc, L. Shi, Z. Wang,  Lévy noise promotes cooperation in the prisoner’s dilemma game with reinforcement learning,  Nonlinear Dynamics 108 (2022) 1837–1845. doi:10.1007/s11071-022-07289-7.
  * Bertsekas [2019] D. Bertsekas, Reinforcement learning and optimal control, Athena Scientific, 2019\.
  * Zhang et al. [2020] J.-Q. Zhang, S.-P. Zhang, L. Chen, X.-D. Liu,  Understanding collective behaviors in reinforcement learning evolutionary games via a belief-based formalization,  Physical Review E 101 (2020) 042402. doi:10.1103/PhysRevE.101.042402.
  * Zheng et al. [2023] G. Zheng, W. Cai, G. Qi, J. Zhang, L. Chen,  Optimal coordination in minority game: A solution from reinforcement learning,  arXiv preprint arXiv:2312.14970 (2023).
  * Ezaki et al. [2016] T. Ezaki, Y. Horita, M. Takezawa, N. Masuda,  Reinforcement learning explains conditional cooperation and its moody cousin,  PLoS Computational Biology 12 (2016) e1005034. doi:10.1371/journal.pcbi.1005034.
  * Izquierdo et al. [2007] L. R. Izquierdo, S. S. Izquierdo, N. M. Gotts, J. G. Polhill,  Transient and asymptotic dynamics of reinforcement learning in games,  Games and Economic Behavior 61 (2007) 259–276. doi:10.1371/journal.pcbi.1005034.
  * Izquierdo et al. [2008] S. S. Izquierdo, L. R. Izquierdo, N. M. Gotts,  Reinforcement learning dynamics in social dilemmas,  Journal of Artificial Societies and Social Simulation 11 (2008) 1.
  * Masuda and Nakamura [2011] N. Masuda, M. Nakamura,  Numerical analysis of a reinforcement learning model with the dynamic aspiration level in the iterated prisoner’s dilemma,  Journal of Theoretical Biology 278 (2011) 55–62. doi:10.1016/j.jtbi.2011.03.005.
  * Horita et al. [2017] Y. Horita, M. Takezawa, K. Inukai, T. Kita, N. Masuda,  Reinforcement learning accounts for moody conditional cooperation behavior: experimental results,  Scientific Reports 7 (2017) 39275. doi:10.1038/srep39275.
  * Fan et al. [2022] L. Fan, Z. Song, L. Wang, Y. Liu, Z. Wang,  Incorporating social payoff into reinforcement learning promotes cooperation,  Chaos: An Interdisciplinary Journal of Nonlinear Science 32 (2022). doi:10.1063/5.0093996.
  * Zhang et al. [2012] J.-j. Zhang, H.-y. Ning, Z.-y. Yin, S.-w. Sun, L. Wang, J.-q. Sun, C.-y. Xia,  A novel snowdrift game model with edge weighting mechanism on the square lattice,  Frontiers of Physics 7 (2012) 366–372. doi:10.1007/s11467-011-0208-x.
  * Ni et al. [2009] Y. Ni, C. Xu, P. Hui, N. Johnson,  Cooperative behavior in evolutionary snowdrift game with bounded rationality,  Physica A: Statistical Mechanics and its Applications 388 (2009) 4856–4862. doi:10.1016/j.physa.2009.07.045.
  * Jia and Ma [2013] N. Jia, S. Ma,  Evolution of cooperation in the snowdrift game among mobile players with random-pairing and reinforcement learning,  Physica A: Statistical Mechanics and its Applications 392 (2013) 5700–5710. doi:10.1016/j.physa.2013.07.049.
  * Xu and Hui [2022] C. Xu, P. M. Hui,  Enhanced cooperation in multiplayer snowdrift games with random and dynamic groupings,  Physical Review E 105 (2022) 054309. doi:10.1103/physreve.105.054309.
  * Watkins and Dayan [1992] C. J. C. H. Watkins, P. Dayan,  Technical note: q-learning,  Machine Learning 8 (1992) 279–292. doi:10.1007/BF00992698.
  * Sugden et al. [2004] R. Sugden, et al., The economics of rights, co-operation and welfare, Springer, 2004\.
  * Smith [1982] J. M. Smith,  Evolution and the theory of games,  in: Did Darwin get it right? Essays on games, sex and evolution, Springer, 1982, pp. 202–215.
  * Flores et al. [2022] L. S. Flores, M. A. Amaral, M. H. Vainstein, H. C. Fernandes,  Cooperation in regular lattices,  Chaos, Solitons & Fractals 164 (2022) 112744. doi:10.1016/j.chaos.2022.112744.
  * Szabó et al. [2005] G. Szabó, J. Vukov, A. Szolnoki,  Phase diagrams for an evolutionary prisoner’s dilemma game on two-dimensional lattices,  Physical Review E 72 (2005) 047107. doi:10.1103/PhysRevE.72.047107.
  * de Oliveira [1992] M. J. de Oliveira,  Isotropic majority-vote model on a square lattice,  Journal of Statistical Physics 66 (1992) 273–281. doi:10.1007/bf01060069.
  * Liu et al. [2019] J. Liu, Y. Fan, J. Zhang, Z. Di,  Coevolution of agent’s behavior and noise parameters in majority vote game on multilayer networks,  New Journal of Physics 21 (2019) 015007. doi:10.1088/1367-2630/ab00aa.
  * Stanley [1971] H. E. Stanley, Phase transitions and critical phenomena, volume 7, Clarendon Press, Oxford, 1971.
  * Domb [2000] C. Domb, Phase transitions and critical phenomena, Elsevier, 2000.
  * Newman et al. [2002] M. E. Newman, D. J. Watts, S. H. Strogatz,  Random graph models of social networks,  Proceedings of the National Academy of Sciences 99 (2002) 2566–2572. doi:10.1073/pnas.012582999.
  * Erdős et al. [1960] P. Erdős, A. Rényi, et al.,  On the evolution of random graphs,  Publications of the Mathematical Institute of the Hungarian Academy of Sciences 5 (1960) 17–60.


# Algebraic de Rham theorem and Baker-Akhiezer function

Igor Krichever† and  Leon Takhtajan  Department of Mathematics, Stony Brook
University, Stony Brook, NY 11794 USA;
Euler International Mathematical Institute, Pesochnaya Nab. 10, Saint
Petersburg 197022 Russia

###### Abstract.

For the case of algebraic curves — compact Riemann surfaces — it is shown that
de Rham cohomology group $H^{1}_{\mathrm{dR}}(X,\mathbb{C})$ of a genus $g$
Riemann surface $X$ has a natural structure of a symplectic vector space.
Every choice of a non-special effective divisor $D$ of degree $g$ on $X$
defines a symplectic basis of $H^{1}_{\mathrm{dR}}(X,\mathbb{C})$, consisting
of holomorphic differentials and differentials of the second kind with poles
on $D$. This result, the algebraic de Rham theorem, is used to describe the
tangent space to Picard and Jacobian varieties of $X$ in terms of
differentials of the second kind, and to define a natural vector fields on the
Jacobian of $X$ that move points of the divisor $D$. In terms of the Lax
formalism on algebraic curves, these vector fields correspond to the Dubrovin
equations in the theory of integrable systems, and the Baker-Akhierzer
function is naturally obtained by the integration along the integral curves.

###### Key words and phrases:

Riemann surfaces, divisors, line bundles, Riemann-Roch theorem, differentials
of the second kind, algebraic de Rham theorem, Picard and Jacobian varieties,
vector fields on the Jacobian variety, Lax representation, Dubrovin equation,
Baker-Akhiezer function

###### 2020 Mathematics Subject Classification:

14F40, 14H40, 14H70

†October 8, 1950 – December 1, 2022

##  1\. Introduction

Let $X$ be a smooth algebraic variety over $\mathbb{C}$ with the classical
topology of a complex manifold. According to Atiyah and Hodge [1], closed
meromorphic $p$-form $\varphi$ on $X$ is called _differential of a second
kind_ , if it has zero residues on open subsets $U=X\setminus D$ for
sufficiently large divisors $D$. A far-reaching generalization of Atiyah and
Hodge results was given by Grothendieck [8]. The quotient groups

$\frac{\\{\text{$p$-forms of the second kind}\\}}{\\{\text{exact forms}\\}}$

have a natural interpretation in terms of a spectral sequence of certain
complex of sheaves of meromorphic forms on $X$ [6, Ch. 3, Sect. 5]. In
particular, one gets the statement

$H_{\mathrm{dR}}^{1}(X,\mathbb{C})\simeq\frac{\\{\text{$1$-forms of the second
kind}\\}}{\\{\text{exact forms}\\}}.$

When $X$ is a smooth algebraic curve of genus $g$, this isomorphism follows
from the Riemann-Roch theorem. It turns out that in this case the space of
differentials of the second kind carries a natural skew-symmetric bilinear
form, which is non-degenerate when taking a quotient by the subspace of exact
forms. Using this bilinear form, in Theorem 1 we give a more explicit
formulation of the algebraic de Rham theorem. Specifically, we show that each
non-special effective divisor $D$ of degree $g$ on $X$ defines a symplectic
basis of $H_{\mathrm{dR}}^{1}(X,\mathbb{C})$, which makes it possible to
explicitly describe a complement to the Lagrangian subspace of holomorphic
$1$-forms on $X$ as the subspace of differentials of the second kind with the
poles in $D$.

In Section 4 it is shown that every non-special effective divisor $D$ of
degree $g$ defines an explicit isomorphism between the vector space
$H^{0,1}(X,\mathbb{C})$ and the Lagrangian subspace of differentials of the
second kind with poles in $D$. This allows us to explicitly describe the
tangent space to Picard variety (and its incarnations, Albanese and Jacobian
varieties) in pure algebro-geometric terms.

It is quite remarkable that this formalism is connected with the theory of
integrable systems. In the standard approach (see, e.g., [5]) the integrable
system is described by the zero curvature equation

$\frac{\partial L}{\partial t}-\frac{\partial M}{\partial x}+LM-ML=0,$

where $L(x,t,\lambda)$ and $M(x,t,\lambda)$ are certain $r\times r$ matrix-
valued rational functions of the spectral parameter $\lambda$, varying on
$\mathbb{C}\mathbb{P}^{1}$, and also depend on extra arguments $x$ and $t$
(physical space and time variables). In papers [12, 13] by the first author
(I.K.), the zero curvature formalism was extended to the case when spectral
parameter varies on an algebraic curve.

Namely, it was shown in [12] that a natural framework for this generalization
is provided by the Hitchin system, expressed in terms of Tyurin parameters for
stable holomorphic vector bundles of rank $r$ and degree $rg$ on an algebraic
curve. Correspondingly, the rational functions $L$ on
$\mathbb{C}\mathbb{P}^{1}$ become special $r\times r$ meromorphic matrix
$1$-forms $L(z)dz$ on an algebraic curve, and the set of such matrices is
parameterized by the moduli space (or rather its Zariski open subset) of
stable holomorphic vector bundles of rank $r$ and degree $rg$ (see [12] for
details). Similar explicit description is given for $r\times r$ meromorphic
matrix-valued functions $M(z)$.

It turns out that the simplest case $r=1$ this formalism is still non-trivial
and is naturally connected with Theorem 1 and related discussion in Section 3.
Namely, as shown in Section 5, the meromorphic 1-forms $L(z)dz$ become
differentials of the first kind on an algebraic curve $X$, while analogs of
$M(z)$ are meromorphic functions $f$, defined using two non-special effective
divisors $D$ and $D_{0}$ of degree $g$ on $X$. Varying divisors $D$
parameterize Jacobian of $X$ with the base point $D_{0}$, and the vector
fields describing the motion of points of $D$ are naturally expressed in terms
of the meromorphic functions $f$.

Remarkably, in the case when $X$ is a hyperelliptic curve, the equations for
the integral curves of these vector fields, equations (7), coincide with the
_Dubrovin equations_ , arising in the theory of finite-gap integration of the
Korteweg-de Vries equation [3]. Moreover, integrating meromorphic functions
$f$ along these integral curves and using Dubrovin’s equations, we naturally
obtain the _Baker-Akhiezer function_ , a fundamental object in the algebro-
geometric approach to integrable systems, introduced by the first author
(I.K.) in [11]!

###  1.1. Acknowledgments

The second author (L.T.) is grateful to the referee for constructive remarks
and suggestions.

##  2\. Differentials of the second kind

Let $X$ be connected, compact Riemann surface of genus $g$ with the classical
topology. Denote by $\mathcal{O}_{X}$ the sheaf of germs of holomorphic
functions on $X$, by $\mathcal{M}_{X}$ — the sheaf of germs of meromorphic
functions on $X$, and by $\mathcal{M}$ — the vector space of meromorphic
functions on $X$. For every divisor $D$ on $X$ denote by $L=\mathcal{O}(D)$
the holomorphic line bundle associated with $D$, and by $H^{0}(X,L)$ — the
vector space of holomorphic sections of $L$ over $X$. The following
isomorphism

$H^{0}(X,L)\simeq\mathcal{L}_{D}=\\{f\in\mathcal{M}:(f)+D\geq 0\\}$

is very useful.

The Riemann-Roch theorem, together with the Kodaira-Serre duality, is the
formula

$h^{0}(L)-h^{0}(K_{X}-L)=\deg L+1-g,$

where $h^{0}(L)=\dim_{\mathbb{C}}H^{0}(X,L)$, $\deg L$ is the degree of $L$,
and $K_{X}$ is the canonical class of $X$ — the holomorphic cotangent bundle
to $X$.

Let $\mathrm{d}$ be the exterior derivative on $X$. The sheaf
$\mathrm{d}\mathcal{M}_{X}$ is a sheaf of germs of differentials of the second
kind on $X$ and $\Omega^{(\mathrm{2nd})}=H^{0}(X,\mathrm{d}\mathcal{M}_{X})$
is the infinite-dimensional vector space of the differentials of the second
kind — meromorphic $1$-forms on $X$ with zero residues.

The infinite-dimensional vector space $\Omega^{(\mathrm{2nd})}$ has a natural
skew-symmetric bilinear form111There are also analogs of the skew-symmetric
bilinear form $\omega_{X}$ and of algebraic de Rham theorem for meromorphic
quadratic differentials, to appear in a paper by the second author (L.T.).

$\omega_{X}(\theta_{1},\theta_{2})=\sum_{P\in
X}\mathrm{Res}_{P}(\mathrm{d}^{-1}\theta_{1}\theta_{2}),\quad\theta_{1},\theta_{2}\in\Omega^{(\mathrm{2nd})},$

where $\mathrm{d}^{-1}\theta_{1}$ denotes any locally defined function $f$,
such that $\mathrm{d}f=\theta_{1}$, called the local antiderivative. The
ambiguity in the choice of $f$ does not matter.

Indeed, it is clear that bilinear form $\omega_{X}$ is defined by a finite sum
and the choice of an additive constant in the definition of a local
antiderivative is irrelevant. The skew-symmetry of $\omega_{X}$ follows from
the basic property

$\mathrm{Res}_{P}(f_{1}\mathrm{d}f_{2})=-\mathrm{Res}_{P}(f_{2}\mathrm{d}f_{1}),$

where meromorphic functions $f_{1}$ and $f_{2}$ are local antiderivatives of
$\theta_{1}$ and $\theta_{2}$ in a neighborhood of $P\in X$.

##  3\. Algebraic de Rham theorem

In an abstract form, algebraic de Rham theorem is the following statement

(1)
$H^{1}_{\mathrm{dR}}(X,\mathbb{C})\simeq\Omega^{(\mathrm{2nd})}/\mathrm{d}\mathcal{M},$

which is easily proved using a sheaf-theoretic de Rham isomorphism

$H^{1}_{\mathrm{dR}}(X,\mathbb{C})\simeq H^{1}(X,\underline{\mathbb{C}}),$

where $\underline{\mathbb{C}}$ is the locally constant sheaf.

Indeed, consider the following short exact sequence of sheaves

$\begin{CD}0@>{}>{}>\underline{\mathbb{C}}@>{i}>{}>\mathcal{M}_{X}@>{\mathrm{d}}>{}>\mathrm{d}\mathcal{M}_{X}@>{}>{}>0\\\
\end{CD}$

and the corresponding exact sequence in the cohomology

$\begin{CD}H^{0}(X,\mathcal{M}_{X})@>{\mathrm{d}}>{}>H^{0}(X,\mathrm{d}\mathcal{M}_{X})@>{\delta}>{}>H^{1}(X,\underline{\mathbb{C}})@>{}>{}>H^{1}(X,\mathcal{M}_{X}).\end{CD}$

It follows from the Riemann-Roch theorem that $H^{1}(X,\mathcal{O}(D))=0$ if
$\deg D>2g-2$, which implies (see, e.g., [7, Ch. 2, §17.7])

$H^{1}(X,\mathcal{M}_{X})=\\{0\\},$

and (1) is proved.

Using bilinear form $\omega_{X}$, we can make isomorphism (1) more concrete.
Namely, we have the following statement result (see [2, Ch. 6, §8], [4, Ch.
III, §§5.3-5.4] and [14, Theorem 4]).

######  Theorem 1.

The following statements hold.

  1. (i)

The restriction of the bilinear form $\omega_{X}$ to
$\Omega^{(\mathrm{2nd})}/\mathrm{d}\mathcal{M}$ is non-degenerate and

$\dim_{\mathbb{C}}\Omega^{(\mathrm{2nd})}/\mathrm{d}\mathcal{M}=2g.$

  2. (ii)

Each degree $g$ non-special effective divisor $D$ on $X$ defines the
isomorphism

$\Omega^{(\mathrm{2nd})}/\mathrm{d}\mathcal{M}\simeq\Omega^{(\mathrm{2nd})}\cap
H^{0}(X,K_{X}+2D).$

  3. (iii)

Let $D=P_{1}+\cdots+P_{g}$ be a non-special divisor of degree $g$ with
distinct points. For every choice of local coordinates in the neighborhoods of
$P_{i}$, the vector space $\Omega^{(\mathrm{2nd})}\cap H^{0}(X,K_{X}+2D)$ has
the basis $\\{\vartheta_{i},\tau_{i}\\}_{i=1}^{g}$, symplectic with respect to
the bilinear from $\omega_{X}$,

$\omega_{X}(\vartheta_{i},\vartheta_{j})=\omega_{X}(\tau_{i},\tau_{j})=0,\;\;\omega_{X}(\vartheta_{i},\tau_{j})=\delta_{ij},\quad
i,j=1,\dots,g.$

This basis consists of differentials of the first kind $\vartheta_{i}$ and
differentials of the second kind $\tau_{i}$, uniquely characterized by the
conditions

$\vartheta_{i}=\left(\delta_{ij}+O(z-z_{j})\right)\mathrm{d}z\;\;\text{and}\;\;\tau_{i}=\left(\frac{\delta_{ij}}{(z-z_{j})^{2}}+O(z-z_{j})\right)\mathrm{d}z,$

where $z_{j}=z(P_{j})$ for a local coordinate $z$ at $P_{j}$ and
$i,j=1,\dots,g$.

###### Proof.

Let $(\theta)_{\infty}=\sum_{i=1}^{l}n_{i}Q_{i}$ be the polar divisor of
$\theta\in\Omega^{(\mathrm{2nd})}$, $n_{i}\geq 2$. Since $D$ is non-special,
$h^{0}(K_{X}-D)=0$ and by Riemann-Roch formula we have $h^{0}(D+nQ_{i})=n+1$
for $n\geq 0$. Thus if $Q_{i}$ is not a point of $D$, there is
$f_{i}\in\mathcal{L}_{D+(n_{i}-1)Q_{i}}$ such that

$\mathrm{ord}_{Q_{i}}(\theta-df_{i})\geq 0.$

If $Q_{i}$ is a point of $D$, there is
$f_{i}\in\mathcal{L}_{D+(n_{i}-1)Q_{i}}$ such that

$\mathrm{ord}_{Q_{i}}(\theta-df_{i})\geq-2.$

(In this case, because $h^{0}(D)=1$, one can not adjust the principle part of
$df_{i}$ at $Q_{i}$ to cancel posible second order pole of $\theta$). Thus for
$f=\sum_{i=1}^{l}f_{i}$ we have

$(\theta-\mathrm{d}f)\geq-2D,$

which proves part (ii).

The dimension formula in part (i) easily follows from part (ii) since

$\displaystyle\dim_{\mathbb{C}}\Omega^{(\mathrm{2nd})}\cap H^{0}(X,K_{X}+2D)$
$\displaystyle=h^{0}(X,K_{X}+2D)-h^{0}(X,K_{X}+D)+h^{0}(X,K_{X})$
$\displaystyle=(3g-1)-(2g-1)+g=2g.$

To prove part (iii), and the remaining statement in part (i), consider the
linear map

$L:\Omega^{(\mathrm{2nd})}\cap H^{0}(X,K_{X}+2D)\to\mathbb{C}^{2g},$

defined as follows. For each $\theta\in\Omega^{(\mathrm{2nd})}\cap
H^{0}(X,K_{X}+2D)$, let $\alpha_{i}(\theta),\beta_{i}(\theta)\in\mathbb{C}$ be
such that near $P_{i}$

$\frac{\theta}{\mathrm{d}z}-\alpha_{i}(\theta)-\frac{\beta_{i}(\theta)}{(z-z_{i})^{2}}=O(z-z_{i}),$

and put

$L(\theta)=(\alpha_{1}(\theta),\beta_{1}(\theta),\dots,\alpha_{g}(\theta),\beta_{g}(\theta)).$

Since $D$ is non-special, the map $L$ is injective and hence is an
isomorphism, and we define $\vartheta_{i}$ and $\tau_{i}$ to have only non-
zero components of $L$ to be, respectively, $\alpha_{i}=1$ and $\beta_{i}=1$.
∎

######  Remark 1.

The choice of a non-special effective divisor $D$ on $X$ with $g$ distinct
points $P_{i}$ and local coordinates is as an algebraic analogue of the choice
of $a$-cycles on a Riemann surface. Correspondingly, differentials $\tau_{i}$
are analogues of differentials of the second kind with second-order poles,
zero $a$-periods and normalized $b$-periods. The symplectic property of the
basis $\\{\vartheta_{i},\tau_{i}\\}_{i=1}^{g}$ is an analogue of the
reciprocity laws for differentials of the first kind and the second kind
(see[9, Ch. 5, §1], and [10, Ch. VI, §3]).

######  Remark 2.

Denote by $\Omega^{(\mathrm{2nd})}(2D)$ the subspace in
$\Omega^{(\mathrm{2nd})}$ spanned by $\tau_{i}$,

$\Omega^{(\mathrm{2nd})}(2D)=\mathbb{C}\tau_{1}\oplus\cdots\oplus\mathbb{C}\tau_{g}.$

Then $\Omega^{(\mathrm{2nd})}(2D)$ and $H^{0}(X,K_{X})$ are Lagrangian
subspaces in $\Omega^{(\mathrm{2nd})}/\mathrm{d}\mathcal{M}$, dual with
respect to the pairing given by the symplectic form $\omega_{X}$.

##  4\. Tangent space to Picard variety

We have a decomposition

(2) $H^{1}_{\mathrm{dR}}(X,\mathbb{C})=H^{1,0}(X,\mathbb{C})\oplus
H^{0,1}(X,\mathbb{C})$

with the natural pairing

$H^{1,0}(X,\mathbb{C})\otimes
H^{0,1}(X,\mathbb{C})\ni\alpha\otimes\beta\mapsto(\alpha,\beta)=\int_{X}\alpha\wedge\beta\in\mathbb{C}.$

The period map

$H^{1,0}(X,\mathbb{C})\ni\vartheta\mapsto\int_{c}\vartheta\in\mathbb{C},\quad\text{where}\quad
c\in H_{1}(X,\mathbb{Z}),$

gives canonical inclusion of the lattice $H_{1}(X,\mathbb{Z})$ into
$H^{1,0}(X,\mathbb{C})^{\vee}$, the dual space to $H^{1,0}(X,\mathbb{C})$, and
defines the Albanese variety

$\mathrm{Alb}(X)=H^{1,0}(X,\mathbb{C})^{\vee}/H_{1}(X,\mathbb{Z}).$

Using the Dolbeault isomorphism and the exponential exact sequence of sheaves
on $X$, we have for the Picard variety of line bundles of degree $0$

$\mathrm{Pic}^{0}(X)=H^{0,1}(X,\mathbb{C})/H^{1}(X,\mathbb{Z}).$

Thus holomorphic tangent space to $\mathrm{Pic}^{0}(X)$ can be identified with
the vector space $H^{0,1}(X,\mathbb{C})$.

However, Theorem 1 allows to describe the tangent space to Picard variety in
purely algebro-geometric terms. Namely, we have the following simple result.

######  Proposition 1.

Each non-special effective divisor $D$ of degree $g$ defines an isomorphism

$H^{0,1}(X,\mathbb{C})\simeq\Omega^{(\mathrm{2nd})}(2D).$

###### Proof.

It follows from Theorem 1 part (iii), that the mapping

$H^{0,1}(X,\mathbb{C})\ni\beta\mapsto\psi(\beta)=\sum_{i=1}^{g}(\vartheta_{i},\beta)\tau_{i}\in\Omega^{(\mathrm{2nd})}(2D)$

satisfies

$(\vartheta,\beta)=\omega_{X}(\vartheta,\psi(\beta))$

for any $\vartheta\in H^{0}(X,K_{X})$, and is an isomorphism. ∎

Identifying $H^{1,0}(X,\mathbb{C})^{\vee}$ with $\Omega^{(\mathrm{2nd})}(2D)$,
we get an inclusion of $H_{1}(X,\mathbb{Z})$ into
$\Omega^{(\mathrm{2nd})}(2D)$, defined as follows. Let $\theta_{c}$ be the
$(0,1)$-component of the Poincaré dual of a cycle $c\in H_{1}(X,\mathbb{Z})$,
so

$\int_{c}\vartheta=\int_{X}\vartheta\wedge\theta_{c}=(\vartheta,\theta_{c})\quad\text{for
all}\quad\vartheta\in H^{1,0}(X,\mathbb{C}).$

Then

(3) $H_{1}(X,\mathbb{Z})\ni
c\mapsto\tau_{c}=\psi(\theta_{c})=\sum_{i=1}^{g}\int_{c}\vartheta_{i}\cdot\tau_{i}\in\Omega^{(\mathrm{2nd})}(2D),$

so

(4) $\mathrm{Alb}(X)=\Omega^{(\mathrm{2nd})}(2D)/H_{1}(X,\mathbb{Z}).$

Thus a choice of a non-special effective divisor $D$ of degree $g$ allows to
identify holomorphic tangent spaces to
$\mathrm{Alb}(X)\simeq\mathrm{Pic}^{0}(X)\simeq\mathrm{Jac}(X)$ with the
vector space $\Omega^{(\mathrm{2nd})}(2D)$ of the differentials of the second
kind with poles in $D$. Correspondingly, the holomorphic cotangent space is
naturally identified with the vector space of $H^{1,0}(X,\mathbb{C})$ of
differentials of the first kind, and the pairing with
$\Omega^{(\mathrm{2nd})}(2D)$ is given by the symplectic form $\omega_{X}$.

##  5\. The Baker-Akhiezer function

Fix a non-special effective divisor $D_{0}=Q_{1}+\cdots+Q_{g}$ of degree $g$
and let $\\{\vartheta_{i}\\}_{i=1}^{g}$ be the basis of $H^{0}(X,K_{X})$ from
Theorem 1, specialized to the divisor $D_{0}$. Consider the Abel-Jacobi map

$X^{(g)}\ni D\to\mu^{(g)}(D)\in\mathrm{Jac}(X),$

where $\mu^{(g)}$ is the Abel sum: for varying $D=P_{1}+\cdots+P_{g}$

(5)
$\mu^{(g)}(D)=\left(\sum_{i=1}^{g}\int_{Q_{i}}^{P_{i}}\vartheta_{1},\dots,\sum_{i=1}^{g}\int_{Q_{i}}^{P_{i}}\vartheta_{g}\right).$

Choose local coordinates at $P_{i}$ and put $z_{i}=z(P_{i})$. It follows from
(5) that $1$-forms $dz_{i}$ on $\mathrm{Jac}(X)$ at the base point
$\mu^{g}(D_{0})$ correspond to differentials $\vartheta_{i}$, and the vector
fields $\dfrac{\partial}{\partial z_{i}}$ to the differentials of the second
kind $\tau_{i}$ from Theorem 1. If divisor $D$ is also non-special, it follows
from the group law on the Jacobian and Theorem 1 that $\mathrm{d}z_{i}$ and
$\dfrac{\partial}{\partial z_{i}}$ at a point $\mu^{(g)}(D)$ are given by the
symplectic basis of $\Omega^{(\mathrm{2nd})}\cap H^{0}(X,K_{X}+2D)$ from
Theorem 1.

Equivalently, these vector fields on $\mathrm{Jac}(X)$ can be described using
the formalism of Lax equations on algebraic curves, developed by the first
author in [12, 13]. The main ingredients in [12, 13] are stable vector bundles
of rank $r$ and degree $rg$ and Lax operators, certain meromorphic $r\times r$
matrix-valued $1$-forms $L(z)dz$ on a Riemann surface and $r\times r$
meromorphic matrix-valued functions $M(z)$.

Specialization to the Jacobian corresponds to the case $r=1$ and simplifies
construction in [12, 13] dramatically. Namely, meromorphic $1$-forms
$L(z)\mathrm{d}z$ become differentials of the first kind $\vartheta\in
H^{0}(X,K_{X})$, while analogs of meromorphic functions $M(z)$ are defined as
follows.

Consider the vector space

$\mathcal{L}_{D+D_{0}}=\\{f\in\mathcal{M}:(f)+D+D_{0}\geq 0\\}.$

It follows from the Riemann-Roch theorem that
$\dim_{\mathbb{C}}\mathcal{L}_{D+D_{0}}=g+1$. Thus for any fixed choice of
principal parts of $f$ at the points of $D_{0}$, not all of them are equal to
zero, there is a unique, up to an inessential additive constant,
$f\in\mathcal{L}_{D+D_{0}}$ satisfying

(6) $f(z)=\frac{\alpha_{i}}{z-z_{i}}+O(1),\quad z_{i}=z(P_{i})$

at all points of the divisor $D=P_{1}+\cdots+P_{g}$. Functions $f$,
parametrized by their fixed principal parts at $D_{0}$, play the role of
meromorphic functions $M(z)$ in case $r=1$; coefficients $\alpha_{i}$ depend
on the principal parts at $D_{0}$.

We have a unique decomposition

$\mathrm{d}f=\tau-\tau_{0},$

where $\tau\in\Omega^{(\mathrm{2nd})}(2D)$ (see Remark 2) and
$(\tau_{0})+2D_{0}\geq 0$. By the residue theorem,

$-\sum_{i=1}^{g}\mathrm{Res}_{P_{i}}(f\vartheta)=\omega_{X}(\vartheta,\tau)=\omega_{X}(\vartheta,\tau_{0}),\quad\vartheta\in
H^{0}(X,K_{X}),$

so the pairing (2.22) in [12], given by the Krichever-Phong form, coincides
with the pairing given by the symplectic form $\omega_{X}$.

A choice of a symplectic basis of $\Omega^{(\mathrm{2nd})}\cap
H^{0}(X,K_{X}+2D)$ establishes a correspondence

$f\mapsto\mathscr{L}_{f}=-\sum_{i=1}^{g}\alpha_{i}\dfrac{\partial}{\partial
z_{i}}$

between rational functions $f\in\mathcal{L}_{D+D_{0}}$ and vector fields on
$\mathrm{Jac}(X)$. Along an integral curve $D(t)=P_{1}(t)+\cdots+P_{g}(t)$ of
$\mathscr{L}_{f}$, where $D(0)=D$, we have

(7) $\dot{z}_{i}(t)=-\alpha_{i}(t),\quad i=1,\dots,g,$

where the dot stands for the $t$-derivative. In case when $X$ is a
hyperelliptic curve, equations (7) are classical _Dubrovin equations_ ,
arising in the theory of finite-gap integration for the Korteweg-de Vries
equation [3], written in terms of the Abel transform. Using Dubrovin
equations, we see that along the integral curve equations (6) take the form

(8) $f_{t}(z)=-\frac{\dot{z}_{i}(t)}{z-z_{i}(t)}+O(1),\quad i=1,\dots,g.$

Thus integrating and introducing

$\Psi(z)=\exp\left\\{\int_{0}^{T}f_{t}(z)dt\right\\},$

we see from (8) that $\Psi$ is a meromorphic function on $X\setminus D_{0}$
having simple poles only at $D$, simple zeros only at $D(T)$, and essential
singularities at the points of $D_{0}$. The function $\Psi$ is nothing but the
celebrated _Baker-Akhiezer function_ , introduced by the first author in [11]!

We leave it to the interested reader to describe by explicit formulas this
connection between algebraic de Rham theorem and the integrable systems.

## References

  * [1] M.F. Atiyah and W.V.D. Hodge, Integrals of the second kind on an algebraic variety, Annals of Math., 62 (1955), 56–91.
  * [2] C. Chevalley, Introduction to the theory of algebraic functions of one variable, American Mathematical Society, Providence, R.I., 1963.
  * [3] B.A. Dubrovin, _Periodic problems for the Korteweg-de Vries equation in the class of finite band potentials_ , Funkt. Anal. Priloz., 9:3 (1975), 41–51; English transl. Funct. Anal. Appl., 9:3 (1975), 215–223.
  * [4] M. Eichler, Introduction to the theory of algebraic numbers and functions, Academic Press, New York, 1966.
  * [5] L.D. Faddeev and L.A. Takhtajan, _Hamiltonian methods in the theory of solitons_ , Springer “Classics in Mathematics” 2007, xiv + 592 pp, reprint of 1987 original.
  * [6] Ph. Griffiths and J. Harris, Principles of algebraic geometry, Wiley-Interscience, New York, 1978.
  * [7] O. Forster, _Lectures on Riemann surfaces_ , Springer-Verlag, New York, 1981.
  * [8] A. Grothendieck, _On the de Rham cohomology of algebraic varieties_ , Publications mathématiques de I.H.É.S., 29 (1966), 95–103.
  * [9] K. Iwasawa, Algebraic functions, Transl. Math. Monogr., vol. 118, Amer. Math. Soc., Providence, RI 1993.
  * [10] I. Kra, Automorphic forms and Kleinian groups, Benjamin, Reading, MA 1972.
  * [11] I.M. Krichever, _Integration of nonlinear equations by the methods of algebraic geometry_ , Funkt. Anal. Priloz., _11_ :1 (1977), 15–31; English transl. Funct. Anal. Appl., 11:1 (1977), 12–26.
  * [12] I.M. Krichever, _Vector Bundles and Lax Equations on Algebraic Curves_ , Commun. Math. Phys. 229 (2002), 229–269.
  * [13] I. M. Krichever, _Isomonodromy equations on algebraic curves, canonical transformations and Whitham equations_ , Mosc. Math. J., 2:4 (2002), 717–752.
  * [14] L.A. Takhtajan, _Quantum field theories on algebraic curves. I. Additive bosons_ Izv. Ross. Akad. Nauk Ser. Mat. 77 (2013), no. 2, 165–196; English transl. Izv. Math. 77 (2013), no. 2, 378–406.