# Improving Input-label Mapping with Demonstration Replay for
In-context Learning

Zhuocheng Gong1111Work done during an internship at Meituan., Jiahao Liu2,
Qifan Wang3
Jingang Wang2, Xunliang Cai2, Dongyan Zhao1,4,5222Corresponding authors:
Dongyan Zhao<EMAIL_ADDRESS>and Rui Yan (ruiyan@ruc.edu.cn)., Rui
Yan6222Corresponding authors: Dongyan Zhao<EMAIL_ADDRESS>and Rui Yan
(ruiyan@ruc.edu.cn).
1Wangxuan Institute of Computer Technology, Peking University
2Meituan; 3Meta AI; 4National Key Laboratory of General Artificial
Intelligence
5Beijing Institute for General Artificial Intelligence
6Gaoling School of Artificial Intelligence, Renmin University of China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

In-context learning (ICL) is an emerging capability of large autoregressive
language models where a few input-label demonstrations are appended to the
input to enhance the model’s understanding of downstream NLP tasks, without
directly adjusting the model parameters. The effectiveness of ICL can be
attributed to the strong language modeling capabilities of large language
models (LLMs), which enable them to learn the mapping between input and labels
based on in-context demonstrations. Despite achieving promising results, the
causal nature of language modeling in ICL restricts the attention to be
backward only, i.e., a token only attends to its previous tokens, failing to
capture the full input-label information and limiting the model’s performance.
In this paper, we propose a novel ICL method called Repeated Demonstration
with Sliding Causal Attention, (RdSca). Specifically, we duplicate later
demonstrations and concatenate them to the front, allowing the model to
‘observe’ the later information even under the causal restriction. Besides, we
introduce sliding causal attention, which customizes causal attention to avoid
information leakage. Experimental results show that our method significantly
improves the input-label mapping in ICL demonstrations. We also conduct an in-
depth analysis of how to customize the causal attention without training,
which has been an unexplored area in previous research.

##  1 Introduction

Large language models (LLMs) have become the backbone of various natural
language processing tasks in different fields. One of the most remarkable
abilities of LLMs is in-context learning (ICL) (Brown et al., 2020). By
providing a few demonstrations and instructions into the input, along with the
input queries, LLMs can perform well in new tasks without requiring fine-
tuning. The secret of ICL is to formulate the input as the natural language
generation task, then the LLM can be activated to prompt knowledge learned in
the pre-training stage.

Despite the promising results demonstrated by existing ICL models (Gonen et
al., 2022; Wei et al., 2022), their causal nature in language modeling
restricts each token’s attention solely to its preceding tokens. As a result,
these models fail to capture the complete input-label information, thereby
limiting their overall performance. Specifically, the pre-training objective
of the current autoregressive LLMs focuses on predicting future tokens based
on past ones Radford et al. (2018), implemented with causal attention. While
this approach works well for modeling regular sequences, it becomes less
effective when applied to ICL tasks. The limitation of causal attention
restricts ICL demonstrations to having only left context, which hampers the
model’s ability to fully comprehend and exploit the input-label relationship.

Unlike tokens in a sentence that possess sequential dependencies, there is no
inherent sequential relationship between the demonstrations in the ICL input.
Therefore, it is desirable for these demonstrations to interact with one
another comprehensively, rather than relying solely on later demonstrations
attending to earlier ones, while the reverse is not possible. Intuitively, if
we can enable each demonstration to attend to all the others, we can
potentially obtain a more sophisticated context for the ICL query. However,
achieving this on an LLM pre-trained with the objective of causal language
modeling is not straightforward. Simply removing the causal restriction and
allowing the model to have access to the right context during inference is not
feasible, as it would result in a significant disparity between training and
inference conditions.

In this work, we focus on capturing full input-label mapping information from
demonstrations. To achieve this target, we propose two techniques. The first
is Repeated Demonstration, where we replicate later demonstrations and
concatenate them to the front. This allows the model to ‘observe’ the later
information even under the causal restriction. For example, if we consider
four demonstrations represented by $d_{1}d_{2}d_{3}d_{4}$, the input sequence
after replication becomes
$d_{2}^{\prime}d_{3}^{\prime}d_{4}^{\prime}d_{1}d_{2}d_{3}d_{4}$. However,
simply duplicating demonstrations brings about a new problem: we do not want
to attend a demonstration twice, as this may cause the model to take shortcuts
by learning to repeat the answer of its first encounter, rather than learning
the input-label mapping. To address this, we propose the second technique, the
Sliding Causal Attention, which customizes the original causal attention by
restricting the attention window so that each demonstration can only attend to
all other demonstrations once. In the case of four demonstrations, attention
windows are $d_{2}^{\prime}d_{3}^{\prime}d_{4}^{\prime}d_{1}$,
$d_{3}^{\prime}d_{4}^{\prime}d_{1}d_{2}$, $d_{4}^{\prime}d_{1}d_{2}d_{3}$, and
$d_{1}d_{2}d_{3}d_{4}$, respectively. Through experiments, we demonstrate that
our proposed method (Repeated Demonstrations with Sliding Causal Attention,
RdSca) significantly enhances the ability to learn input-label mapping from
ICL demonstrations.

Our proposed sliding casual attention is the first attempt that customizes the
causal attention in the inference stage without further training. We
investigate a number of different designs for customizing causal attention and
reach some principles for the success of ICL. For example, we find that the
first <SOS> token plays an essential role. It should always be available to
attend to no matter where the attention window slides. Besides, the size of
the attention window determines the richness of the semantic context, thus it
affects the performance greatly. Our contributions are summarized as:

  * •

To the best of our knowledge, we are the first to identify the limitation of
causal language modeling in the ICL and to introduce a novel approach for
enabling effective interactions between demonstrations.

  * •

We validate the feasibility of customizing causal attention during the
inference stage without further training and conduct further analysis on
causal attention customization. We believe this idea has great potential and
sheds new light on optimizing ICL and other large model inference scenarios.

  * •

We conduct experiments on several text classification datasets to evaluate the
effectiveness of our proposed method. The experimental results clearly
demonstrate that our approach significantly enhances the input-label mapping
in ICL demonstrations.

Figure 1: An illustration for regular ICL and our RdSca ICL. Regular ICL uses
vanilla causal attention, causing the uneven interaction between
demonstrations. In RdSca, each demonstration can attend to all the others
thanks to demonstration repetition and sliding causal attention.

##  2 Backgrounds

###  2.1 Causal Language Modeling

Most of the current decoder-only LLMs employ causal language modeling as the
pre-training objective (Radford et al., 2018), which aims to predict future
tokens based on the past.

$\operatorname*{arg\,min}_{\theta}\sum\limits_{i}P_{\theta}(u_{i}|u_{0},u_{1},...,u_{i-1})$
(1)

In causal language modeling, the model only attends to tokens that occur
before (the left context), resulting in a unidirectional attention scheme
known as causal attention. This approach enables the model to process each
token in the input sequence in order, without accessing any information from
the future (the right context).

#### Causal Attention Mask

Practically, a causal attention mask is used to implement causal attention,
which guarantees unidirectionality by masking all right-to-left attention
connections and only allowing right-to-left connections. Formally, the
attention mask is a binary-valued matrix $M\in\\{0,1\\}^{n\times n}$, where
$n$ is the total sequence length. The element $m_{ij}$ in $M$ indicates
whether the $j$-th token in the sequence can attend to the $i$-th token, with
a value of $1$ for yes and $0$ for no. Therefore, the causal attention mask
matrix is a lower triangular matrix where $m_{ij}=0,\forall i<j$.

###  2.2 In-context Learning

Now we formally introduce the definition and basic notations of in-context
learning. We focus on classification tasks and causal language models. Given
$K$ input-label pairs $\\{x_{i},y_{i}\\}_{i=1}^{K}$ and a query $x_{q}$, the
objective is to predict the label of the query $y_{q}$ by predicting the next
token. Formally,

$\operatorname*{arg\,min}_{y_{q}\in\mathcal{C}}P(y_{q}|(x_{1},y_{1}),(x_{2},y_{2}),...,(x_{K},y_{K}),x_{q})$
(2)

where $\mathcal{C}$ is the label set.

To perform ICL successfully, we transform the classification task into a
natural language generation task by adding templates to the query and
demonstrations. Additionally, discrete labels are mapped to label words, such
as "positive" and "negative" for sentiment classification. We denote the
demonstration with templates and label words as $d_{i}=T(x_{i},y_{i})$. The
entire input to the model is formed by concatenating the demonstrations and
the query $q=T(x_{q},\\_)$, resulting in $d_{1}d_{2}...d_{k}q$.

#### Semantically-unrelated label ICL (SUL-ICL)

In regular ICL, natural language words that are closely related to the task
objective are used as label words. This allows the model to utilize semantic
prior knowledge. For example, when conducting sentiment classification, label
words such as "positive" and "negative" are highly semantically related to the
sentiment labels of the input. During pre-training, the model is exposed to
similar patterns and can learn to associate these label words with the
corresponding sentiment labels. In this paper, we eliminate the contribution
of semantic priors and perform semantically-unrelated label ICL (Wei et al.,
2023). In this setting, natural language labels are replaced with
semantically-unrelated labels. This approach forces the model to rely solely
on input-label mappings to perform ICL.

##  3 Method

###  3.1 Defect of Traditional ICL

During few-shot ICL, we combine the labeled demonstrations with the query by
concatenating them and feeding them into the LLM. The labeled demonstrations
provide valuable in-context input-label information, which is included in the
few-shot ICL input. As a result, few-shot ICL consistently achieves better
performance compared to zero-shot ICL.

However, few-shot ICL under the current scheme of causal language modeling has
a defect. Due to the restriction of causal attention, each demonstration only
has access to half of the full context (_i.e._ , the left context). As a
result, it cannot ‘observe’ demonstrations that follow it. For example, in
4-shot ICL, if we consider the third demonstration as the last query (which
makes sense because tokens that come after it have no influence on it), then
the first two demonstrations serve as its context while the fourth
demonstration is ignored. In this sense, we can regard predicting the label of
the first demonstration as a zero-shot ICL task, the second as a one-shot ICL,
the third as a two-shot ICL, and so on, as shown in Figure 1.

While restricting the model from accessing the right context makes sense when
modeling regular natural language sequences to prevent information leakage, it
is unnecessary for few-shot ICL since there is no dependency between
demonstrations. The causal restriction limits earlier demonstrations from
accessing later information, resulting in only half of the available
information being utilized.

###  3.2 Repeated Demonstrations

Enabling each demonstration to ‘observe’ later demonstrations is not a trivial
task. Simply replacing causal attention with full attention to allow the model
to receive both left and right context is not feasible, as the model is pre-
trained with causal language modeling. Switching to full attention during
inference would result in a significant loss in performance, as we have
verified through experiments (Section 4.4). Therefore, we propose the Repeated
Demonstration method to establish sufficient interactions among demonstrations
while still adhering to the premise of causal attention. The method is based
on a simple idea: duplicating all the other demonstrations except the first
one and concatenating them to the front of the sequence. By doing so, we
expand the input sequence $d_{1}d_{2}...d_{K}q$ to
$d_{2}^{\prime}...d_{K}^{\prime}d_{1}d_{2}...d_{K}q$ where $d_{i}^{\prime}$ is
the duplication of $d_{i}$. This operation allows each demonstration to attend
to all the others by adding the later demonstrations to its left context. As a
result, the model can make full use of demonstrations while still obeying the
causal attention restriction.

###  3.3 Sliding Causal Attention

The Repeated Demonstration method alone is not sufficient to achieve our
target, as it introduces a new problem of duplicated information in the
context. Under causal attention, a token can attend to all tokens that precede
it. When combined with repeated demonstrations, this can result in the same
information being duplicated in the context. For example, when performing
4-shot ICL with repeated demonstrations, the input is
$d_{2}^{\prime}d_{3}^{\prime}d_{4}^{\prime}d_{1}d_{2}d_{3}d_{4}q$. The context
of $d_{3}$ is $d_{2}^{\prime}d_{3}^{\prime}d_{4}^{\prime}d_{1}d_{2}d_{3}$
where some demonstrations appear twice. Repetitive information can cause the
model to learn a shortcut to predict the label by repeating the label of the
same demonstration that appeared for the first time, rather than learning the
input-label mapping as expected. We will provide a detailed explanation of
this phenomenon in the experimental section (Section 4.4).

To tackle this issue, we propose Sliding Causal Attention, which utilizes a
sliding attention window to limit the range of tokens that each demonstration
can attend to. This approach effectively prevents the occurrence of repetitive
information and ensures that each demonstration has a non-repetitive context.
Specifically, the attention window is defined as follows:

$window(x)=\begin{cases}d_{i+1}^{\prime}...d_{K}^{\prime}d_{1}...d_{i}&x=d_{i},\forall
i\leq K\\\ d_{1}d_{2}...d_{K}q&x=q\end{cases}$ (3)

Then we explain how sliding causal attention works. We use the window size $W$
to represent the number of demonstrations contained within it. In our main
setting, we use $W=K$, indicating each attention window contains all the
demonstrations. As the window slides, a new demonstration enters while an old
one exits. This ensures that there are always consistent $K$ different
demonstrations within the context.

Additionally, we find that the <SOS> token, which represents the first token
of the sequence, is crucial to the model’s performance. Therefore, we add
<SOS> to every attention window to ensure that each token can attend to <SOS>.
We will provide further explanations on this in the experimental section 4.4.

Figure 2: Averaging performance of RdSca on LLAMA of different scales. Left:
ICL with random-selected demonstrations. We report the results of both 4-shot
and 7-shot ICL for this setting. Middle: We combine our method with the
demonstration retrieval technique. We employ KATE (Liu et al., 2022) to
retrieve target demonstrations. Right: we combine with Prototypical
Calibration Han et al. (2022) to further improve performance. Full results can
be found in Appendix A

##  4 Experiments

###  4.1 Setup

#### Models

We conducted experiments on decoder-only causal language models. We utilized
the LLAMA (Touvron et al., 2023) model family with varying scales, including
7B, 13B, 30B, and 65B.

#### Tasks

We evaluate on classification tasks, including SST-2 (Socher et al., 2013), CB
(De Marneffe et al., 2019), RTE (Dagan et al., 2005; Wang et al., 2019a),
AGNews (Zhang et al., 2015), QQP (Wang et al., 2019b), and QNLI (Wang et al.,
2019b). If the dataset includes a validation split, we evaluate the model’s
performance on the validation set. Otherwise, we evaluate on the test set.
Datasets are obtained from Huggingface Datasets
library***https://huggingface.co/docs/datasets/index.

#### Other Details

For all experiments, we use $K$ = 4 demonstrations by default. Demonstrations
are uniformly sampled from the training data. We utilize prompt templates from
PromptSource†††https://github.com/bigscience-workshop/promptsource (Bach et
al., 2022). For each dataset, we use four different templates and select a set
of $K$ training examples using 4 different random seeds. Therefore, the
reported results are the average of 16 different runs. We would like to
emphasize that we run all methods using the same random seeds, ensuring that
the demonstrations used by each method are identical. This was done to
eliminate any potential bias caused by differences in the demonstrations seen
by each method and ensure a fair comparison. Due to limited computing
resources, we evaluate a random sample of 200 examples for tasks with a large
validation set, instead of the entire dataset. We have found that this sample
size is sufficient to obtain stable and reliable results.

###  4.2 Main Results

Figure 2 presents a comparison between our method and regular ICL. We observe
a clear correlation between the performance of semantically unrelated label
ICL and the model scale: increasing the model scale improves performance for
all methods. In the case of LLAMA-7B, the performance is poor, and the
performance gap between different methods is small, indicating that it is
challenging for small models to learn the input-label mapping from
demonstrations while the semantic meaning of labels is removed. As the model
size scales up to 30B and 65B, the models begin to exhibit strong ICL
capabilities, and the performance of different methods varies. This phenomenon
is consistent with the findings of Wei et al. (2023), which suggest that
larger language models learn the input-label mapping differently because the
input-label mapping learning ability only emerges when the model is large
enough. In comparison with ICL with random demonstrations, we find that
retrieving demonstrations that are more relevant to the query can
significantly improve the performance on all scales. Furthermore, calibration
has a positive influence on the ICL results, showing that our method can be
combined with other techniques to further enhance the overall performance.

####  RdSca improves input-label mapping

We observe a significant performance boost over regular ICL in both 4-shot and
7-shot settings on LLAMA-30B and LLAMA-13B. Our method shows an average
improvement of 8.4% on LLAMA-30B and 10.5% on LLAMA-65B compared to regular
ICL, indicating a significant advantage in learning the input-label mapping
from demonstrations. As mentioned in previous sections, we believe that in the
scheme of causal language modeling, regular ICL only utilizes half of the
context (left context) for the demonstrations. In contrast, our method makes
use of all the available information, which enhances the representations of
demonstrations and provides richer semantics for predicting the query’s
target.

###  4.3 Results on MMLU

| Regular ICL | RdSca
---|---|---
humanities | 65.99 | 68.44
STEM | 51.84 | 51.57
social sciences | 72.29 | 73.60
other | 63.34 | 64.65
average | 62.20 | 63.66

Table 1: LLAMA-65B 4-shot ICL results on MMLU.

We also validate LLAMA-65B on MMLU Hendrycks et al. (2021), which is a more
advanced benchmark for evaluating the abilities of LLMs. The results are shown
in Table 1.

Figure 3: Illustration of the attention mask, which we divide into multiple
regions and label with numbers. The specific meanings are as follows: ① query-
to-demonstrations attention; ② inter-demonstrations right-to-left attention; ③
inter-demonstrations(repeat) right-to-left attention; ④ attention to the
start-of-sentence token; ⑤ attention to redundant context; ⑥ inter-
demonstrations left-to-right attention;⑦ demonstrations-to-query attention;⑧
other left-to-right attention.

Methods | Attention Regions | Is causal? | LLAMA-65B | LLAMA-30B
---|---|---|---|---
Random Guess | - | - | $43.06$ |  | $43.06$ |
Regular ICL | ①+② | Yes | $59.83$ | ($+16.77$) | $52.71$ | ($+9.65$)
Full Attn. (v1) | ①+②+⑥ | No | 45.15 | ($+2.09$) | 47.84 | ($+4.78$)
Full Attn. (v2) | ①+②+⑥+⑦ | No | $44.32$ | ($+1.26$) | $47.36$ | ($+4.30$)
Rd | ①+②+③+④+⑤ | Yes | $60.18$ | ($+17.12$) | $54.02$ | ($+10.96$)
RdSca w/o <SOS> | ①+②+③ | Yes | $32.20$ | ($-10.86$) | $12.96$ | ($-30.10$)
RdSca | ①+②+③+④ | Yes | $64.96$ | ($+\textbf{21.90}$) | 56.55 | ($+\textbf{13.49}$)

Table 2: Ablations on causal attention customization. We report the averaging
performance on all tasks. We implement two versions of the Full Attn., which
differ in whether including left-to-right attention of the query.

###  4.4 Looking Deeper into Causal Attention Customization

In this section, we conduct a further investigation to look deeper into the
effectiveness of sliding causal attention. How and why does sliding causal
attention work and are there alternatives when customizing causal attention?
To answer these questions, we implement the following ablations:

  * •

Full Attn. This uses full attention instead of causal attention so the model
can attend to both the left and right context.

  * •

Rd. This is the ablation of RdSca that uses original causal attention without
sliding causal attention.

  * •

RdSca w/o <SOS>. In this ablation of RdSca, when the attention window moves
forward, latter tokens cannot attend to the beginning of the sentence, _i.e._
, the <SOS> token.

Figure 3 shows how we divide the whole attention matrix into various regions.
The above-mentioned methods have unique access to these regions. Details and
experimental results are shown in Table 2.

#### Can we break the causal restriction?

First, we are concerned about the limitations of causal attention.
Intuitively, if we can remove the causal restriction during inference by using
non-causal attention and providing the model with the full context, we can
fully utilize the demonstration information. However, our experimental results
indicate that this simple idea is not feasible. The performance of Full Attn.
is no better than random guessing, but why? There is a simple explanation: the
gap between causal attention training and non-causal attention inference is
too huge. Since the model is trained using causal language modeling, it has no
inductive bias toward data with the correct context. Therefore, using full
attention during inference is out-of-distribution for a causal model.

#### Sliding attention window matters.

Based on the results, it is evident that Rd has little improvement over
regular ICL. Although Rd provides richer interaction between demonstrations
than regular ICL, the lack of attention restrictions causes the same
demonstration to attend to its first appearance. Consequently, the model takes
a shortcut and repeats the answer of the first appearance instead of learning
the input-label mapping to predict the result. Therefore, the improvement
brought by simply repeating the demonstrations is limited. However, the
situation changes when we add the sliding causal window, which blocks repeated
information and forces the demonstrations to attend only to other unseen
demonstrations, ensuring that there is no information leakage. According to
the evaluation results, this approach successfully enables the model to better
capture the input-label mapping information, as expected.

This ablation study further indicates another insight: the default causal
attention may not be always optimal in ICL. Previous research aimed at
improving ICL has mainly focused on template construction and demonstration
selection, with little attention paid to the causal attention masks. Our study
shows that customizing attention masks can be a new technique for enhancing
ICL, which is worthy of further research.

#### Attending to the <SOS> token is essential.

Next, we examine the role of the first token <SOS> in causal attention
customization. If not treating the <SOS> token separately, as the attention
window moves, the first token slides out of the scope of the attention window.
This means that the latter demonstrations and the query cannot attend to
<SOS>. Our experiments show that in this setting the ICL ability is severely
affected (①+②+③ in Table 2). However, when we manually add the <SOS> token to
the attention window (as in the case of RdSca), we observe a significant
improvement in performance. This comparison demonstrates that the <SOS> token
is crucial for the model to perform correctly. But why does such a seemingly
insignificant token have such a significant impact on performance? We believe
that the significance of the <SOS> token lies in allowing the model to treat
the tokens under the attention window as a valid sequence rather than a
fragment of the whole sequence.

As the model is pre-trained on sequences that all start with <SOS>, it does
not know how to handle sequences that do not start with <SOS>. Therefore, when
customizing causal attention, we need to make sure that the model regards the
tokens in the attention window as a valid sequence starting with <SOS>. Only
in this way, we can activate the model’s language modeling ability correctly.

Figure 4: Visualization of attention masks for different window sizes.

Window | n-shots | LLAMA-65B | LLAMA-30B
---|---|---|---
Regular | 0-1-2-3-4 | $69.87$ | $63.10$
W=1 | 0-0-0-0-4 | $60.83$ | $52.78$
W=2 | 1-1-1-1-4 | $71.33$ | $60.83$
W=3 | 2-2-2-2-4 | $74.01$ | $65.70$
W=4 | 3-3-3-3-4 | $74.27$ | $68.73$
W=4 | 3-3-3-3-3 | $70.99$ | $64.44$

Table 3: Average performance on all tasks for different attention window
sizes. We also report the number of demonstrations contained in the context of
each demonstration. For example, "0-1-2-3-4" means that, for regular ICL, the
first demonstration has no context, the second sees one demonstration, the
third sees two, and so on. In this experiment, we use KATE to retrieve
demonstrations instead of random-selected ones. In this way, we save
computational resources because there is no need to run experiments over
different random seeds.

###  4.5 Ablation on Attention Window Size

In this section, we discuss the impact of window size. We denote the attention
window size of the original RdSca as $W=K$, meaning that there are $K$
demonstrations in each window. In this case, each demonstration can attend to
all other ones, so each demonstration can be considered as performing
($K-1$)-shot ICL. As shown in Figure 4, we employ smaller window sizes $W$ on
RdSca and see what happens to the ICL ability. Intuitively, reducing the size
of the sliding window means reducing the context (the number of
demonstrations) that the current one is able to attend to. Thus the model’s
ability to learn the input-label mapping from context would be affected.
Especially, when $W=1$, each demonstration can only attend to itself, which is
equivalent to zero-shot ICL. The results are shown in Table 3. As expected,
the window size is highly correlated with ICL performance: the larger the
window size, the better the performance. Surprisingly, we notice that when
$W=2$, _i.e._ , one-shot ICL for all demonstrations, RdSca is already
comparable with regular ICL on both 30B and 65B LLAMA. This indicates that our
ICL method makes more efficient exploitation of the demonstrations in ICL. The
last row of Table 3 shows when the query not seeing all 4 demonstrations, the
performance drops.

###  4.6 What If Adding More Demonstrations?

Figure 5: Increasing the number of demonstrations for LLAMA-65B on SST-2.

In this section, we investigate how our method performs with more
demonstrations. We start from $K=4$ demonstrations and gradually add more. As
shown in Figure 5, all methods receive significant performance gains when the
number of demonstrations is relatively small. However, as the number of
demonstrations increases, the benefits of adding more demonstrations gradually
reduce. The ICL performance reaches a plateau at around $K=16$. Before
reaching this plateau, RdSca consistently outperforms the vanilla-repeat and
regular ICL, indicating that our method is robust to the increase in the
number of demonstrations. However, this advantage gradually diminishes when
adding more demonstrations. All methods seem to perform comparably at around
$K=16$, suggesting that the ability to learn from context is close to its
upper bound at this point.

In addition, we observe an interesting phenomenon in the experiment. We find
that the performance of Rd sharply declines when there are too many
demonstrations (decreasing by 12.02 points from $K=24$ to $30$, and 25.24
points from $K=30$ to $40$.), while RdSca does not suffer such huge losses. We
explain this phenomenon from two aspects: First, as mentioned earlier, if not
customizing the causal attention, the model learns shortcuts from repeated
examples rather than input-label mapping, which leads to a decline in ICL
performance. Second, due to the repetition of demonstrations, the context
becomes too long when adding too many demonstrations, and some studies have
shown that LLMs still have difficulty modeling long sequences. Our method can
effectively solve these problems by customizing attention. Therefore, we
believe that our exploration of customizing causal attention highlights a
possible solution for tackling the long sequence modeling issue, which is
worth further research in the future.

###  4.7 Discussions on Efficiency Issues

As repeating demonstrations involves expanding the sequence lengths, the model
is required to process more tokens when compared with regular ICL, which may
lead to computational inefficiency and extra memory consumption. However, we
argue that these efficiency issues can be neglected practically. Thanks to the
auto-regressive nature of Large Language Models LLMs, the representations of
previous tokens are not dependent on later tokens. Therefore, in the inference
phase, the representations of demonstrations can be pre-computed and stored
offline, thereby mitigating the added computational burden. Moreover, our
adaptation of the attention mask allows the LLM to focus only on tokens within
the sliding window for key-value (KV) caching, rather than retaining
information on all previous tokens. This optimization reduces the memory
requirements of RdSca. Consequently, the memory consumption of RdSca remains
on par with that of a standard ICL setup.

##  5 Related Works

###  5.1 In-context Learning

In-context learning (ICL) is an effective approach for adapting pre-trained
LLMs to downstream tasks Brown et al. (2020). This is achieved by adding task-
specific templates and demonstrations before the test query, without updating
model parameters. Recent works focus on enhancing ICL with various techniques.
For example, Liu et al. (2022); Rubin et al. (2022); Ram et al. (2023); Luo et
al. (2023) propose that choosing demonstrations more carefully results in
better ICL performance. Some studies try improving ICL with more sophisticated
prompt templates, either by hand-craft or by automation (Gonen et al., 2022).
Some propose chain-of-thoughts (CoT) to elicit the reasoning abilities of LLMs
by augmenting each demonstration with a chain of reasoning steps (Wei et al.,
2022). Subsequent studies have continued to build upon this approach,
achieving further improvements (Wang et al., 2022). Some studies explore the
mechanism of ICL. Min et al. (2022) shows that randomly replacing labels in
the demonstrations barely hurts performance so ground truth demonstrations are
not required. (Wei et al., 2023) claims that the success of ICL relies on both
semantic prior knowledge and input-label mappings. They find that larger
models can better capture input-label mappings from demonstrations.

###  5.2 Attention Customization

There has been a number of work done on attention customization since
Transformer was proposed (Vaswani et al., 2017). many of them focus on
modeling long sequences (Kitaev et al., 2020; Beltagy et al., 2020) or
improving efficiency through sparse attention (Tay et al., 2023). Some works
have explored customizing attention to block certain dependencies. For
example, Mu et al. (2023) prevents later tokens from attending to the prompt
to compress context. These attention customizations can be viewed as some kind
of modifications to the model architecture, which requires additional
training. However, our study is the first to investigate the possibility of
attention customization during inference without further training.

##  6 Conclusion

In this study, we introduce RdSca, a novel ICL framework that enhances the
learning of the input-label mapping from demonstrations. We propose that the
causal attention mechanism of decoder-only LLMs restricts the model from fully
exploiting the input-label mapping from demonstrations. To address this, we
suggest repeating demonstrations to allow each demonstration to have full
context and customizing the vanilla causal attention mechanism to prevent
information leakage. Experimental results show that our method consistently
improves the input-label mapping ability of ICL on LLMs of different scales.
Furthermore, we delve deeper into causal attention customization and show how
different attention settings affect ICL performance. Additionally, this work
is the first to customize the causal attention of a pre-trained autoregressive
LLM without further training, which may pave the way for further research in
this direction.

## Limitations

Although RdSca has demonstrated outstanding performance in capturing the full
input-label mapping from demonstrations, there are some limitations to our
work in this section that we must acknowledge. First, our method expands the
sequence length by duplicating demonstrations, which leads to increased
computation costs and inference latency. This may limit its practicality in
computation-constrained scenarios. Moreover, we have not evaluated our method
on more complex tasks, nor have we determined its performance with chain-of-
thought scenarios. Therefore, in the future, we need to investigate how it can
be extended to a broader range of task scenarios.

## Acknowledgments

This work is supported by National Key R&D Program of China (No.
2021YFC3340304) and National Natural Science Foundation of China (NSFC Grant
No. 62122089). Jingang Wang is funded by Beijing Nova Program (Grant No.
20220484098). We sincerely thank all reviewers for their valuable comments and
suggestions, which are crucial for improving our work.

## References

  * Bach et al. (2022) Stephen H. Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M. Saiful Bari, Thibault Févry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged Saeed AlShaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir R. Radev, Mike Tian-Jian Jiang, and Alexander M. Rush. 2022.  Promptsource: An integrated development environment and repository for natural language prompts.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 - System Demonstrations, Dublin, Ireland, May 22-27, 2022_ , pages 93–104. Association for Computational Linguistics.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _CoRR_ , abs/2004.05150.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005.  The PASCAL recognising textual entailment challenge.  In _Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers_ , volume 3944 of _Lecture Notes in Computer Science_ , pages 177–190. Springer.
  * De Marneffe et al. (2019) Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019.  The commitmentbank: Investigating projection in naturally occurring discourse.  In _proceedings of Sinn und Bedeutung_ , volume 23, pages 107–124.
  * Gonen et al. (2022) Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. 2022\.  Demystifying prompts in language models via perplexity estimation.  _CoRR_ , abs/2212.04037.
  * Han et al. (2022) Zhixiong Han, Yaru Hao, Li Dong, and Furu Wei. 2022.  Prototypical calibration for few-shot learning of language models.  _CoRR_ , abs/2205.10183.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Kitaev et al. (2020) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020.  Reformer: The efficient transformer.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022.  What makes good in-context examples for gpt-3? In _Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO@ACL 2022, Dublin, Ireland and Online, May 27, 2022_ , pages 100–114. Association for Computational Linguistics.
  * Luo et al. (2023) Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Seyed Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Y. Zhao. 2023.  Dr.icl: Demonstration-retrieved in-context learning.  _CoRR_ , abs/2305.14128.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , pages 11048–11064. Association for Computational Linguistics.
  * Mu et al. (2023) Jesse Mu, Xiang Lisa Li, and Noah D. Goodman. 2023.  Learning to compress prompts with gist tokens.  _CoRR_ , abs/2304.08467.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.
  * Ram et al. (2023) Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023.  In-context retrieval-augmented language models.  _CoRR_ , abs/2302.00083.
  * Rubin et al. (2022) Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022.  Learning to retrieve prompts for in-context learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , pages 2655–2671. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL_ , pages 1631–1642. ACL.
  * Tay et al. (2023) Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2023.  Efficient transformers: A survey.  _ACM Comput. Surv._ , 55(6):109:1–109:28.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.  _CoRR_ , abs/2302.13971.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 3261–3275.
  * Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, and Denny Zhou. 2022\.  Self-consistency improves chain of thought reasoning in language models.  _CoRR_ , abs/2203.11171.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  In _NeurIPS_.
  * Wei et al. (2023) Jerry W. Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023.  Larger language models do in-context learning differently.  _CoRR_ , abs/2303.03846.
  * Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 649–657.

##  Appendix A Full results

Methods | 65B | 30B | 13B | 7B
---|---|---|---|---
Regular ICL | 59.82 | 55.18 | 47.79 | 49.72
Rd | 63.66 | 59.91 | 49.68 | 49.34
RdSca | 75.76 | 69.32 | 53.03 | 50.92
with demonstration retrieval |  |  |  |
Regular ICL | 74.75 | 65.62 | 58.00 | 56.88
Rd | 77.13 | 73.63 | 56.38 | 59.25
RdSca | 87.38 | 82.13 | 68.88 | 64.63

Table 4: Full evaluation results on SST-2

Methods | 65B | 30B | 13B | 7B
---|---|---|---|---
Regular ICL | 76.79 | 58.15 | 44.98 | 50.67
Rd | 76.12 | 56.49 | 42.94 | 48.34
RdSca | 76.90 | 56.17 | 46.97 | 50.66
with demonstration retrieval |  |  |  |
Regular ICL | 78.12 | 64.73 | 45.54 | 57.59
Rd | 79.72 | 60.38 | 49.06 | 59.91
RdSca | 79.72 | 60.38 | 52.83 | 57.08

Table 5: Full evaluation results on CB

Methods | 65B | 30B | 13B | 7B
---|---|---|---|---
Regular ICL | 53.67 | 49.13 | 61.50 | 59.62
Rd | 53.20 | 53.37 | 62.54 | 57.96
RdSca | 59.68 | 53.44 | 60.05 | 57.23
with demonstration retrieval |  |  |  |
Regular ICL | 60.17 | 53.85 | 67.94 | 63.47
Rd | 61.79 | 62.95 | 70.38 | 64.20
RdSca | 64.87 | 61.54 | 66.79 | 64.36

Table 6: Full evaluation results on QQP

Methods | 65B | 30B | 13B | 7B
---|---|---|---|---
Regular ICL | 59.38 | 53.45 | 47.84 | 46.56
Rd | 58.27 | 52.28 | 47.77 | 47.63
RdSca | 60.51 | 54.49 | 45.33 | 47.29
with demonstration retrieval |  |  |  |
Regular ICL | 63.21 | 60.90 | 51.79 | 52.56
Rd | 62.03 | 65.67 | 51.66 | 53.14
RdSca | 64.43 | 66.15 | 51.27 | 52.98

Table 7: Full evaluation results on QNLI

Methods | 65B | 30B | 13B | 7B
---|---|---|---|---
Regular ICL | 36.97 | 36.33 | 29.69 | 28.19
Rd | 37.86 | 36.49 | 30.31 | 27.64
RdSca | 43.85 | 40.09 | 31.12 | 30.89
with demonstration retrieval |  |  |  |
Regular ICL | 70.75 | 70.38 | 67.12 | 67.50
Rd | 74.63 | 76.12 | 72.00 | 74.12
RdSca | 75.13 | 76.25 | 70.12 | 70.75

Table 8: Full evaluation results on AG-News

Methods | 65B | 30B | 13B | 7B
---|---|---|---|---
Regular ICL | 72.36 | 63.99 | 55.4 | 53.76
Rd | 72.0 | 65.56 | 55.4 | 55.6
RdSca | 73.07 | 65.8 | 57.07 | 53.95
with demonstration retrieval |  |  |  |
Regular ICL | 72.25 | 63.12 | 56.75 | 52.75
Rd | 72.39 | 65.27 | 55.7 | 54.66
RdSca | 74.07 | 65.92 | 56.74 | 53.23

Table 9: Full evaluation results on RTE


# Ising Model on the Fibonacci Sphere

A. S. Pochinok  Pacific Quantum Center, Far Eastern Federal University, 690950
Vladivostok, Russia  A. V. Molochkov  Pacific Quantum Center, Far Eastern
Federal University, 690950 Vladivostok, Russia  M. N. Chernodub  Institut
Denis Poisson UMR 7013, Université de Tours, 37200 Tours, France

###### Abstract

We formulate the ferromagnetic Ising model on a two-dimensional sphere using
the Delaunay triangulation of the Fibonacci covering. The Fibonacci approach
generates a uniform isotropic covering of the sphere with approximately equal-
area triangles, thus potentially supporting a smooth thermodynamic limit. In
the absence of magnetic field, the model exhibits a spontaneous magnetization
phase transition at a critical temperature that depends on the connectivity
properties of the underlying lattice. While in the standard triangular
lattice, every site is connected to 6 neighboring sites, the triangulated
Fibonacci lattice of the curved surface contains a substantial density of the
5- and 7-vertices. As the number of sites in the Fibonacci sphere increases,
the triangular cover of the sphere experiences a series of singular
transitions that reflect a sudden change in its connectivity properties. These
changes substantially influence the statistical features of the system leading
to a series of first-order-like discontinuities as the radius of the sphere
increases. We found that the Ising model on a uniform, Fibonacci-triangulated
sphere in a large-radius limit possesses the phase transition at the critical
temperature
$T^{\large\mathbin{\mathchoice{\ooalign{$\displaystyle\vbox{\hbox{\scalebox{0.6}{$\displaystyle\bigcirc$}}}$\cr$\displaystyle*$\cr}}{\ooalign{$\textstyle\vbox{\hbox{\scalebox{0.6}{$\textstyle\bigcirc$}}}$\cr$\textstyle*$\cr}}{\ooalign{$\scriptstyle\vbox{\hbox{\scalebox{0.6}{$\scriptstyle\bigcirc$}}}$\cr$\scriptstyle*$\cr}}{\ooalign{$\scriptscriptstyle\vbox{\hbox{\scalebox{0.6}{$\scriptscriptstyle\bigcirc$}}}$\cr$\scriptscriptstyle*$\cr}}}}_{c}\simeq
3.33(3)J$, which is slightly lower than the thermodynamic result for an
equilaterally triangulated planar lattice. This mismatch is a memory effect:
the planar Fibonacci lattice remembers its origin from the curved space.

##  I Introduction

The two-dimensional Ising model is one of the simplest statistical models that
describe ferromagnetism in planar lattices associated with $2d$ crystalline
atomic systems [1]. The spin variable $S_{\boldsymbol{x}}$ represents the
magnetic moment of an atom at the site $\boldsymbol{x}$. This variable can
take two values, $S_{\boldsymbol{x}}=+1$ or $S_{\boldsymbol{x}}=-1$,
associated, respectively, with up or down orientations of the magnetic moment
with respect to the atomic crystal plane. The Hamiltonian of the system is:

$H=-J\sum\limits_{<{\boldsymbol{x}},{\boldsymbol{y}}>}S_{\boldsymbol{x}}S_{\boldsymbol{y}}-h\sum\limits_{\boldsymbol{x}}S_{\boldsymbol{x}}\,,$
(1)

where $J$ is the interaction energy and $h$ is the external magnetic field.
The summation in Eq. (1) is going over the pairs
$<{\boldsymbol{x}},{\boldsymbol{y}}>$ of nearest-neighbor sites.

At low temperatures, the Ising model (1) with positive spin coupling $J>0$
possesses a ferromagnetic phase characterized by the ordered (co-aligned)
spins of the same sign. At high temperatures, the spins reside in a
disordered, paramagnetic phase. The model describes a class of natural
ferromagnetic systems.

In the absence of the background magnetic field, $h=0$, the model (1)
possesses the global ${\mathbb{Z}}_{2}$ symmetry, which corresponds to the
invariance of the $h=0$ Hamiltonian with respect to a simultaneous flip of all
spins, $S_{\boldsymbol{x}}\to-S_{\boldsymbol{x}}$. The ordered ferromagnetic
ground state spontaneously breaks this symmetry,
${\mathbb{Z}}_{2}\to{\mathchoice{\rm 1\mskip-4.0mul}{\rm 1\mskip-4.0mul}{\rm
1\mskip-4.5mul}{\rm 1\mskip-5.0mul}}$. The ordered and disordered phases are
separated by a second-order thermodynamic phase transition at a critical
temperature $T_{c}$. The transition temperature and the associated critical
exponents depend on the type (connectivity properties) of the underlying
lattice. For example, the critical temperature on the square lattice [2, 3]

$\displaystyle\frac{T^{\Box}_{c}}{J}=\frac{2}{\ln\left(1+\sqrt{2}\right)}\simeq
2.2692\,,$ (2)

is substantially lower than the transition temperature at the equilateral
triangular lattice [3]:

$\displaystyle\frac{T^{\triangle}_{c}}{J}=\frac{4}{\ln 3}\simeq 3.6410\,,$ (3)

(here, we work in the units $k_{B}=1$). For the first time, the Ising model
was considered on a triangular lattice in works [4, 5]. A detailed calculation
of the critical parameters of the phase transition of the model has been
implemented in Ref. [6].

At a negative spin coupling, $J<0$, the nearest spins tend to anti-align, thus
forming an (ordered) antiferromagnetic phase. The structure of the ground
state depends on the connectivity features of the underlying lattice. For
bipartite lattices (the ones that can be divided into two sub-lattices with
sites from one sub-lattice interacting with sites from the other sub-lattice
only), the ground state corresponds to the opposite orientation of the spins
at fully polarized sub-lattices. The ground state structure is less
straightforward for non-bipartite lattices (for example, for a triangular
lattice). The antiferromagnetic Ising model possesses plenty of interesting
phenomena associated with frustrated ground states [7]. While in our study, we
concentrate on the ferromagnetic sector of the phase diagram with $J>0$, the
connectivity of the lattice will also play an essential role, as we will see
later (in our paper, we consider non-bipartite lattices).

We investigate the phase diagram of the Ising model in the spherical topology.
To tackle this problem, we triangulate the spherical surface via its covering
by the Fibonacci lattice. The Fibonacci sphere is characterized by a uniform
distribution of points over the whole area without unnatural inhomogeneities
that emerge, for example, around the poles in a naive latitude–longitude
triangulation in a spherical coordinate system [8]. The Fibonacci sphere has
already proven itself to be helpful in magnetic resonance modeling [9], image
processing [10], climate models [11], and others. Moreover, in lattice spin
(gauge) models, the Fibonacci covering has potentially a well-defined
thermodynamic limit due to approximately equal-area regions between three
adjacent sites (triangular plaquettes). This fact makes it attractive for
formulating problems of lattice quantum field theory in spaces with a
spherical topology and, potentially, at other surfaces (the Fibonacci lattice
has been recently employed in the investigation of the XY model in Ref. [12]).
In Section II, we discuss the generation of a Fibonacci lattice on the sphere
and its triangulation via the Delaunay method.

The thermodynamic properties of the model on the Fibonacci sphere are
discussed in Section III. We consider various quantities, such as
magnetization and energy, as well as the corresponding susceptibilities. We
determine the position of the phase transition in the absence of the magnetic
field and investigate the thermodynamic limit of the model. The thermodynamic
limit corresponds to the infinite number of points which, for a uniformly
triangulated sphere, implies the limit of the sphere of an infinite radius,
$R\to\infty$.

The infinite-radius limit of the Fibonacci sphere corresponds to a locally
flat surface of an infinite area. Naively, one could expect the properties of
the Ising model formulated on the $R\to\infty$ Fibonacci sphere and the
infinite-area triangular planar lattice should be the same. We show that,
unexpectedly, this expectation is not realized since, even for a
thermodynamically substantial number of sites, the Fibonacci lattice contains
a significant density of points with abnormal 5-vertex or 7-vertex
connectivities that influence the thermodynamic properties of the system. In
other words, the Fibonacci lattice in the flat, infinite-radius limit still
“remembers” its spherical origin and differs from the standard equilateral
lattice.

Our conclusions are summarized in Section IV.

##  II Triangulated Fibonacci sphere

###  II.1 Fibonacci lattice

The Fibonacci lattice is characterized by an almost uniform and isotropic
distribution of the lattice sites. These properties play an essential role in
the triangulation of the sphere, which is challenging to triangulate uniformly
otherwise [8, 13].

A Fibonacci number $F_{i}$ is a part of the infinite sequence in which every
term, from the third term onward ($i\geqslant 3$), is the sum of the previous
two terms, $F_{i}=F_{i-1}+F_{i-2}$. The first few numbers in the sequence are
as follows:

$\displaystyle\\{0,1,1,2,3,5,8,13,21,\dots\\}\,.$ (4)

The ratio of two adjacent numbers gives, in the large-number limit
$i\to\infty$, the golden ratio

$\displaystyle
g=\lim_{i\to\infty}\frac{F_{i+1}}{F_{i}}=\frac{1+\sqrt{5}}{2}\approx
1.618\approx 137.5^{\circ}\,,$ (5)

defined by the relationship $g=1+1/g$.

The structure of the Fibonacci lattice is most straightforward to understand
in a flat space. The Fibonacci covering of a unit square by a set of points is
as follows [14]:

$\displaystyle x_{k}=\left\\{\frac{k}{g}\,\right\\},\qquad
y_{k}=\frac{k}{n}\,,$ (6)

where the integer number $k=1,\dots,n$ labels the sites with the coordinates
${\boldsymbol{x}}=(x_{k},y_{k})$ and $\left\\{\dots\right\\}$ stands for the
fractional part of a number. The resulting uniform distribution of the sites
over a unit square is shown in Fig. 1.

Figure 1: A flat Fibonacci lattice made of 99 sites using scheme (6). Ordinal
numbering is used. The sequences with the Fibonacci numbers 3, 5, 8, and 13
are shown in colors.

It is easy to conclude from Fig. 1 that the distribution of the points is
reasonably uniform. Moreover, the sites are structured in various straight
lines, inclined clockwise and counterclockwise, with the difference between
the adjacent elements given by a Fibonacci number (4). The latter property
gives the name of the whole construction, called “the Fibonacci lattice”.
Notice that, in general, each site is a part of more than one structured line
with distinct lines labeled by a fixed Fibonacci number. Some of these lines
are easy to spot in Fig. 1.

A similar construction on a disk is called the Fibonacci spiral. This
structure is obtained by imposing a grid on a disk using the following
parametrization of the points in the polar coordinate system [11, 15]:

$\displaystyle x$ $\displaystyle=$ $\displaystyle r\cos\varphi\,,\qquad\qquad
y=r\sin\varphi\,,$ (7a) $\displaystyle r$ $\displaystyle=$ $\displaystyle
r_{0}\sqrt{k-\frac{1}{2}}\,,\qquad\varphi=\frac{2\pi k}{g},$ (7b)

where $r_{0}$ is an arbitrary scale parameter and $k=1,2,\dots$ is an integer
number that labels the lattice sites.

Figure 2: (a) Fibonacci spiral a disk with 700 sites constructed with scheme
(7). In general, the spiral structure of the Fibonacci lattice is not evident
since the distance between the points of the sequence is bigger than the
distance between the geometrically closest points. Here, every 13’s, 21’s,
34’s, and 55’s sites, corresponding to subsequent Fibonacci numbers, are
highlighted in color to visualize the spiral structure. (b) The Fibonacci
lattice on a sphere made with 2001 sites using construction (8).

The Fibonacci spiral is shown in Fig. 2(a). This structure can be represented
in two sets of spiral arcs labeled by the Fibonacci number (4), with the
latter representing the distance between the consecutive points. The
construction is pretty similar to the case of the disk with the difference
given by the curved nature of the spirals that are often called “dominant”
[11]. Some sets of spirals are twisted clockwise, while in the other set, they
are twisted in an opposite, counterclockwise direction. With an increase in
the lattice radius (total number of points), the number of the dominant
spirals increases. One can also show that the Fibonacci lattice is, in fact,
the result of a sequential arrangement of elements in the form of a single,
“generative” spiral [16]. The angular distance between the elements of the
generative spiral corresponds to the golden ratio (5). Since the golden ratio
is the most irrational number [17], the appearance of this number in the
construction of the Fibonacci spiral is an essential factor that serves to
optimize the packing efficiency in avoiding periodicities or quasi-
periodicities of the elements.

In order to obtain a uniform Fibonacci covering of a sphere, it is enough to
project a flat uniform lattice on a disk (the Fibonacci spiral) onto a sphere
to preserve equal areas between adjacent sites:

$\displaystyle x$ $\displaystyle=$ $\displaystyle
r\cos\vartheta\sin\varphi,\quad y=r\sin\vartheta\sin\varphi,\quad
z=r\cos\varphi\,,\quad$ (8a) $\displaystyle\varphi$ $\displaystyle=$
$\displaystyle\frac{2\pi k}{g};\qquad\vartheta=\arccos{\frac{k}{n}}\,.$ (8b)

The methodology of construction and review of some properties of the Fibonacci
sphere can be found in Refs. [18, 19, 20].

###  II.2 Delaunay triangulation of a sphere

Triangulation is the process of the construction of a graph from triangular
elements. For curved two-dimensional manifolds, triangulation serves as a way
to find an approximation of the smooth curved surface by a set of triangles.
Furthermore, triangulation is helpful in computer modeling as it allows
discretizing continuous curved objects. We will also use triangulation to
discretize the sphere and define the Ising model on a spherical surface.

In order to obtain a well-defined and self-consistent thermodynamic limit of
the statistical model on a curved surface, we need to approximate this surface
as a collection of a connected set of triangles of approximately equal areas
and shapes. In other words, the triangulation should be globally and locally
uniform regarding the spatial distribution of the triangles and the local
shapes of their elements.

We have already constructed the uniform distribution of points over a sphere
using the Fibonacci construction. The next step is to triangulate the sphere
using these points as the vertices of the triangles. For a given set of
points, triangulation is generally not a unique procedure since a general set
of points can be connected with a set of disjoint lines in more than one way.
For example, four points shown in Fig. 3(a) could be triangulated in two
different ways using triangles of two different shapes.

In practice, a smooth, physically reasonable triangulation is achieved with
the help of the Delaunay triangulation [21, 22]. The rule determines the
Delaunay triangulation of a set of points that a circle circumscribed around
any triangle does not contain any point from this set in the interior of the
circle. An example of the Delaunay triangulation, along with the circumscribed
circles, is shown in Fig. 3(a). There are two ways to make the triangulation,
and only one of those, shown in the figure, satisfies the Delaunay criterion.
One of the advantages of the Delaunay triangulation is that it maximizes the
minimal angle in every triangle. The Delaunay method allows obtaining the
triangles closest in their shapes and areas for a uniform collection of
points.

|
---|---
(a) | (b)
Figure 3: Delaunay triangulation (a) for four points on a plane and (b) on a
sphere of a lattice with 100 sites.

For a particular case of a sphere, the Delaunay condition can be reformulated
as a requirement that a plane that crosses three sites of any triangle should
contain the remaining set of points from the same triangulation either above
or below the plane. In other words, each such plane separates the space into
two parts, with one part containing all other points of the set while the
other part possessing no points from the same set. The illustration of the
Delaunay triangulation of the sphere is shown in Fig. 3(b). The procedure
details, defined uniquely, are presented in Appendix A. The statistical
properties of the Delaunay triangulation of the Fibonacci sphere are discussed
in Appendix B. These triangulation features will be important for the
thermodynamic limit, which we discuss in the next section.

##  III Ising model on Fibonacci sphere

###  III.1 Energy and magnetization

After constructing the triangulated Fibonacci sphere, we define the Ising
model with the Hamiltonian (1). While the triangles in our model are not
equilateral, we consider the constant couplings, which do not depend on the
length of the links. In this sense, our construction is similar to the
implementation of the spin XY model on the Fibonacci sphere [12]. To simplify
notations, we consider dimensionful variables in units of the coupling $J=1$.

First, we studied the basic properties of the Ising model on small Fibonacci
spheres to ensure that our understanding of the model was correct. To this
end, we calculated the mean energy as the expectation value of the
Hamiltonian, $E={\left\langle H\right\rangle}$, and the total magnetization of
the system:

$M=\sum\limits_{{\boldsymbol{x}}}S_{\boldsymbol{x}}\,.$ (9)

We used the Monte Carlo method to generate configurations using the standard
Metropolis algorithm. The numerical results on the Fibonacci lattice with
$1000$ sites shown below were calculated using $2\cdot 10^{6}$ Monte Carlo
steps.

Figure 4: Energy (the blue color with the axis on the left) and magnetization
(the red color with the axis on the right) of the system on the Fibonacci
sphere with $N=1000$ sites as the function of the external magnetic field $h$
at fixed temperature $T=3.5$.

In Fig. 4, we show the energy and magnetization of the system as the function
of the external magnetic field, $h\neq 0$. At a nonvanishing magnetic field,
the spin-reflection ${\mathbb{Z}}_{2}$ symmetry is broken as the spins prefer
to polarize along the direction of the magnetic field. Therefore, no critical
behavior associated with the spontaneous symmetry breaking occurs at $h\neq
0$, and the transition is characterized by a smooth crossover. With increasing
magnetic field $h$, the magnetization $M$ quickly reaches the saturation point
(with all spins aligned along the background magnetic field) while the mean
energy $E$ smoothly decreases due to ferromagnetic coupling.

Figure 5: The same as in Fig. 4 but as functions of temperature $T$ at
vanishing external magnetic field, $h=0$.

At vanishing magnetic field, $h=0$, the behavior is more nontrivial as the
system in thermodynamic limit experiences a phase transition at varying
temperatures. On a finite lattice, the energy and magnetization exhibit a
pseudo-critical behavior signaling a close presence of a thermodynamic
singularity. In Fig. 5, we show these quantities as functions of temperature
at $h=0$.

Both figures 4 and 5 show a standard behavior for the Fibonacci lattices,
which is qualitatively similar to the properties of the Ising model on flat
lattices [3]. These features encourage us to search for the critical
temperature in the thermodynamic limit similar to the standard methods used on
the planar lattices. In the rest of the paper, we work at vanishing magnetic
field, $h=0$.

###  III.2 Locating transition point

An accurate way to estimate the location of the thermodynamic phase transition
utilizes a Binder cumulant, a higher-order generalization of susceptibility.
We use the 4th-order cumulant, $U\equiv U_{4}$, expressed via the
magnetization (9):

$U(T,N)=1-\frac{\langle M^{4}(T,N)\rangle}{3\langle M^{2}(T,N)\rangle^{2}},$
(10)

where $N$ is the number of lattice sites. The intersection of the Binder
curves for different lattice sizes determines the position of phase transition
in the thermodynamic limit.

Figure 6: The Binder cumulant (10) as the function of temperature $T$ at the
Fibonacci lattices with different numbers of the lattice sites $N$.

In Fig. 6, we show the Binder cumulant (10), calculated with the number of
Monte Carlo steps of the order of $10^{6}$, for two sets of lattices: with the
rare filling (for $N=100,200,300$ sites) and a denser filling (with
$N=1000,2000,3000$ sites). While the numerical data in the figure show a
seemingly good crossing of all these curves, it does not allow us to determine
the phase transition point with good precision. Surprisingly, an increase in
the accuracy of Monte Carlo simulations does not help. The phase transition
temperature approximately corresponds to $T_{c}\approx 3.35$ for the small
filling of the sphere and $T_{c}\approx 3.32$ for the denser filling,
respectively. Thus, we arrive at a contradiction that shows some inconsistent
behavior of the system when we approach the thermodynamic limit.

Figure 7: (top) Heat capacity and (bottom) magnetic susceptibility for a
Fibonacci sphere with $N=100,300,1000$ sites. The red curves show the best
fits by a Gaussian function and the region of the fits.

To illustrate the level of inconsistency further, we calculate the heat
capacity

$C(T,N)=\frac{\langle E^{2}(T,N)\rangle-\langle E(T,N)\rangle^{2}}{T^{2}}\,.$
(11)

and the magnetic susceptibility

$\chi(T,N)=\frac{\langle M^{2}(T,N)\rangle-\langle M(T,N)\rangle^{2}}{T}\,,$
(12)

in the pseudocritical region of couplings for a series of lattices with an
increased total number of sites $N$. So far, these quantities have the
standard behavior: their peaks with the height, which increases with rising
$N$, can be fit by a Gaussian function. The examples of the data and the fits
are shown in Fig. 7.

Figure 8: The extrapolation of pseudocritical temperatures obtained from the
heat capacity (11) and the magnetic susceptibility (12) towards the
thermodynamic limit, $1/N\to 0$.

The maxima of the fits point out to the pseudocritical temperatures, which are
then plotted together as functions of an inverse number of sites of the
Fibonacci sphere, $1/N$, in Fig. 8. The pseudocritical values, obtained from
different quantities, do not coincide with each at finite $N$. According to
our experience with planar lattices, this expected finite-volume effect should
disappear in the thermodynamic limit.

For the Fibonacci spheres, the pseudocritical temperatures can be extrapolated
nicely towards the thermodynamic limit, but, surprisingly, the extrapolated
values do not coincide with each other, Fig. 8. Although the difference
between the extrapolated critical temperatures is about 1%, the magnitude of
this difference is substantial enough to be seen in our accurate calculations.
This observation deepens the puzzle and gives us a quantitative counterpart of
a similar effect that we observed on a qualitative level while studying the
Binder cumulants.

###  III.3 Thermodynamic limit

Does our result mean that the Fibonacci sphere, with the subsequent Delaunay
triangulation, has no well-defined thermodynamic limit? That would be a
surprising conclusion given that the Fibonacci construction provides us with
the most uniform coverage of the sphere, which has no singular points and
possesses other essential advantages with respect to the other triangulation
schemes [8]. To clarify this intriguing property, we plot in Fig. 9 the
extrapolation of the pseudocritical temperatures – obtained from the magnetic
susceptibility peaks – towards the continuum limit using a more extensive set
of Fibonacci spheres.

Figure 9: The scaling of the pseudocritical temperature obtained from the
magnetic susceptibility of the Ising model at the Fibonacci sphere as the
function of an inverse number of sites, $1/N$. Five series of data, labeled
$1,\dots,5$, are fitted separately by a linear dependence. The shaded regions
at the horizontal axis mark the data ranges and the fits. The shaded regions
at the vertical axis give the extrapolated values of the critical temperature
and the corresponding extrapolation errors. The insets show zoom in on the
regions 1 and 2.

Surprisingly, the data of Fig. 9 shows that pseudo-critical temperatures can
be separated into groups of different lattice sizes characterized by an almost
perfect linear behavior. The slopes of the linear curves, however, differ from
each other. Consequently, the extrapolated critical temperatures obtained from
different size groups of the Fibonacci spheres give us slightly different
critical temperatures in the continuum limit.

This unusual phenomenon, the segmentation of the extrapolation region into
separate intervals, can be related to sudden changes in the connectivity
properties of the Fibonacci spheres. As we discuss in detail in Appendix B,
the Fibonacci lattice consists of the sites connected to 5, 6, and 7
neighbors, with the 6-fold connectivity being dominant at large lattice sizes,
$N\sim 10^{3}$. As we increase the lattice size, the proportion of the
6-connected sites increases up to a specific critical lattice size,
$N=N^{(i)}_{\mathrm{cr}}$, when it abruptly diminishes singularly, and the
rise of the fraction of the 6-connected sites starts over again.

The critical values of the lattice size, $N=N^{(i)}_{\mathrm{cr}}$, correspond
to the sudden restructuring of the connectivity properties of the Fibonacci
sphere. In between the critical points,
$(N^{(i)}_{\mathrm{cr}},N^{(i+1)}_{\mathrm{cr}})$, the connectivity changes
smoothly as the number of sites $N$ increases. The linear patches in Fig. 9
correspond precisely to these smooth regions, while the change of the linear
slopes occurs close to the critical value of $N$.

The extrapolated critical values are shown in Fig. 9. The change in the
connectivity properties of the Fibonacci lattice can be attributed to the
systematic effect of the triangulation, which decreases as the lattice size
increases. The combined region for extrapolated critical temperature thus
gives us a consolidated error which includes statistical, fitting, and
systematic effects. Our final result for the critical transition temperature
of the Ising model on the Fibonacci sphere extrapolated to the thermodynamic
limit is as follows:

$\displaystyle\frac{T^{\large\mathbin{\mathchoice{\ooalign{$\displaystyle\vbox{\hbox{\scalebox{0.6}{$\displaystyle\bigcirc$}}}$\cr$\displaystyle*$\cr}}{\ooalign{$\textstyle\vbox{\hbox{\scalebox{0.6}{$\textstyle\bigcirc$}}}$\cr$\textstyle*$\cr}}{\ooalign{$\scriptstyle\vbox{\hbox{\scalebox{0.6}{$\scriptstyle\bigcirc$}}}$\cr$\scriptstyle*$\cr}}{\ooalign{$\scriptscriptstyle\vbox{\hbox{\scalebox{0.6}{$\scriptscriptstyle\bigcirc$}}}$\cr$\scriptscriptstyle*$\cr}}}}_{c}}{J}\simeq
3.33(3)\simeq\frac{10}{3}\,,$ (21)

where
${\large\mathbin{\mathchoice{\ooalign{$\displaystyle\vbox{\hbox{\scalebox{0.6}{$\displaystyle\bigcirc$}}}$\cr$\displaystyle*$\cr}}{\ooalign{$\textstyle\vbox{\hbox{\scalebox{0.6}{$\textstyle\bigcirc$}}}$\cr$\textstyle*$\cr}}{\ooalign{$\scriptstyle\vbox{\hbox{\scalebox{0.6}{$\scriptstyle\bigcirc$}}}$\cr$\scriptstyle*$\cr}}{\ooalign{$\scriptscriptstyle\vbox{\hbox{\scalebox{0.6}{$\scriptscriptstyle\bigcirc$}}}$\cr$\scriptscriptstyle*$\cr}}}}$
denotes the Delaunay-triangulated Fibonacci sphere.

Interestingly, the extrapolated critical temperature of the Ising model at the
Fibonacci sphere lies in between the critical temperatures for the square
lattice $T^{\Box}_{c}$ and that of the equilateral triangular lattice
$T^{\triangle}_{c}$, which are given in Eqs. (2) and (3), respectively:

$\displaystyle
T^{\Box}_{c}<T^{\large\mathbin{\mathchoice{\ooalign{$\displaystyle\vbox{\hbox{\scalebox{0.6}{$\displaystyle\bigcirc$}}}$\cr$\displaystyle*$\cr}}{\ooalign{$\textstyle\vbox{\hbox{\scalebox{0.6}{$\textstyle\bigcirc$}}}$\cr$\textstyle*$\cr}}{\ooalign{$\scriptstyle\vbox{\hbox{\scalebox{0.6}{$\scriptstyle\bigcirc$}}}$\cr$\scriptstyle*$\cr}}{\ooalign{$\scriptscriptstyle\vbox{\hbox{\scalebox{0.6}{$\scriptscriptstyle\bigcirc$}}}$\cr$\scriptscriptstyle*$\cr}}}}_{c}\lesssim
T^{\triangle}_{c}\,.$ (30)

Naively, one could expect that
$T^{\large\mathbin{\mathchoice{\ooalign{$\displaystyle\vbox{\hbox{\scalebox{0.6}{$\displaystyle\bigcirc$}}}$\cr$\displaystyle*$\cr}}{\ooalign{$\textstyle\vbox{\hbox{\scalebox{0.6}{$\textstyle\bigcirc$}}}$\cr$\textstyle*$\cr}}{\ooalign{$\scriptstyle\vbox{\hbox{\scalebox{0.6}{$\scriptstyle\bigcirc$}}}$\cr$\scriptstyle*$\cr}}{\ooalign{$\scriptscriptstyle\vbox{\hbox{\scalebox{0.6}{$\scriptscriptstyle\bigcirc$}}}$\cr$\scriptscriptstyle*$\cr}}}}_{c}$
should coincide with $T^{\triangle}_{c}$ because both models are defined on
the triangular lattices while the triangulated sphere in the thermodynamic
limit $N\to\infty$ becomes a planar lattice. However, we cannot confirm this
expectation with definiteness since in the large-$N$ limit, the Fibonacci
lattice contains a significant (at a level of 1%) portion of 5- and
7-connected sites as one can see in Fig. 11. The critical temperature of the
Ising model on the Fibonacci lattice is thus much closer to its triangular
(6-connected) planar counterpart $T^{\triangle}_{c}$ than the critical
temperature $T^{\Box}_{c}$ at the (4-connected) square lattice. The closeness
of the two critical temperatures,
$T^{\large\mathbin{\mathchoice{\ooalign{$\displaystyle\vbox{\hbox{\scalebox{0.6}{$\displaystyle\bigcirc$}}}$\cr$\displaystyle*$\cr}}{\ooalign{$\textstyle\vbox{\hbox{\scalebox{0.6}{$\textstyle\bigcirc$}}}$\cr$\textstyle*$\cr}}{\ooalign{$\scriptstyle\vbox{\hbox{\scalebox{0.6}{$\scriptstyle\bigcirc$}}}$\cr$\scriptstyle*$\cr}}{\ooalign{$\scriptscriptstyle\vbox{\hbox{\scalebox{0.6}{$\scriptscriptstyle\bigcirc$}}}$\cr$\scriptscriptstyle*$\cr}}}}_{c}$
and $T^{\triangle}_{c}$, is well explained by the similarity of the
connectivity properties of the Fibonacci triangulation in the large-$N$ limit
and the planar equilateral triangle lattices.

##  IV Conclusions

We studied the Ising model on a two-dimensional sphere triangulated using the
Delaunay method applied to a uniform Fibonacci covering. This construction
provides us with a uniform isotropic covering of the sphere with approximately
equal-area triangles, thus potentially supporting a smooth thermodynamic
limit.

We found, however, that as the number of sites $N$ of the triangulated
Fibonacci sphere increases, the connectivity properties of the triangular
lattice experience discontinuous transitions at specific critical lattice
sizes $N=N_{\mathrm{cr}}^{(i)}$, $i=1,2,\dots$ which influence its statistical
features. While in the standard triangular lattice, every site is connected to
6 neighboring sites, the Fibonacci lattice contains a substantial density of
the 5- and 7-vertices. The relative number of sites with different
connectivity changes at critical sizes of the spheres, thus systematically
affecting the extrapolation of the finite-volume results toward the
thermodynamic limit.

We uncovered that the Ising model on Fibonacci-triangulated spheres up to
large sizes (with $2\times 10^{5}$ sites) possesses a single (pseudo)critical
temperature associated with a spontaneous magnetization of the system. In the
thermodynamic limit, the extrapolated critical temperature is given in Eq.
(21). Its value slightly differs from the critical temperature of the Ising
model on an equilateral triangular planar lattice (3). We interpret this
result as a signature of a memory effect that can survive in the thermodynamic
limit: the planar lattice corresponding to the Fibonacci sphere of an infinite
radius remembers its curved origin imprinted in the irregular (but
thermodynamically important) connectivity of the lattice sites.

###### Acknowledgements.

The numerical calculations were performed on computing cluster Vostok-1 of Far
Eastern Federal University. The work of AVM and ASP was supported by Grant No.
0657-2020-0015 of the Ministry of Science and Higher Education of Russia.

##  Appendix A Delaunay triangulation of a sphere

A triangulation corresponds to a procedure that connects a set of sites
(nodes) by non-intersecting links, thus covering the surface with a set of
triangles. The Delaunay triangulation is a particular kind of procedure that
is identified by the following criteria.

For a triangulation of a plane, the global criterion requires that a link
belongs to the Delaunay triangulation if and only if one can represent this
link as a chord of a circle that contains, in its interior, no other sites of
the given set of sites. The local criterion demands that a link belongs to the
Delaunay triangulation if and only if a circle that goes through all three
vertices of any triangle does not contain, in its interior, the sites (nodes)
of any adjacent triangle.

For the Delaunay triangulation for a non-planar surface, for example, for a
sphere, one can formulate the following condition that incorporates both the
global and local criteria mentioned above. For every triangle of the set, we
put in correspondence an auxiliary sphere such that this sphere incorporates
all three nodes of the triangle while the sphere’s center belongs to the plane
of the triangle. For a correct Delaunay triangulation of the sphere, the
interior of each auxiliary sphere does not contain any vertices of any other
triangles.

We implement the Delaunay triangulation using the following numerical
algorithm:

  1. 1.

select a site $A$ from the set;

  2. 2.

take randomly two sites $B$ and $C$ from the same set as candidates for a
possible triangle $ABC$ of the valid triangulation;

  3. 3.

verify whether any other site of the complete set belongs to the interior of
the auxiliary sphere spanned on the triangle $ABC$ with the center located in
the plane parallel to the plane of the $ABC$ triangle.

  4. 4.

If the response is negative, then the site $ABC$ belongs to the Delaunay
triangulation. Otherwise, for site $A$, we choose other sites $B$ and $C$ from
the set and repeat the procedure.

Step 3 is performed using the following construction. First, we need to find
the position ${\boldsymbol{x}}_{0}=(x_{0},y_{0},z_{0})$ of the center of a
circle described around a triangle and belonging to the triangle plane.

The equation of a plane has the following form:

$\displaystyle ax+by+cz+d=0,$ (31)

where $a$, $b$, $c$, and $d$ are certain coefficients. Let the vertices of the
triangle be given by the coordinates
${\boldsymbol{x}}_{i}=(x_{i},y_{i},z_{i})$, where $i=1,2$, and $3$ stand for
the points $A$, $B$, and $C$, respectively. Then, the equation of the plane
which passes through these three sites is defined by the following
determinant:

$\begin{vmatrix}x&y&z&1\\\ x_{1}&y_{1}&z_{1}&1\\\ x_{2}&y_{2}&z_{2}&1\\\
x_{3}&y_{3}&z_{3}&1\\\ \end{vmatrix}=0\,,$ (32)

which is equivalent to Eq. (31). A comparison of Eqs. (31) and (32) gives us
the coefficients that determine the plane:

$\displaystyle a$
$\displaystyle=y_{1}(z_{2}-z_{3})+y_{2}(z_{3}-z_{1})+y_{3}(z_{1}-z_{2}),$ (33)
$\displaystyle b$
$\displaystyle=z_{1}(x_{2}-x_{3})+z_{2}(x_{3}-x_{1})+z_{3}(x_{1}-x_{2}),$ (34)
$\displaystyle b$
$\displaystyle=x_{1}(y_{2}-y_{3})+x_{2}(y_{3}-y_{1})+x_{3}(y_{1}-y_{2}),$ (35)
$\displaystyle d$ $\displaystyle=-\epsilon^{ijk}x_{i}y_{j}z_{k}\,,$ (36)

where $\epsilon^{ijk}$ is Levi-Civita symbol.

The auxiliary sphere intersects the plane at a circle. The sphere and the
circle share the same center ${\boldsymbol{x}}_{0}$, which is equally
elongated from positions of the vertices ${\boldsymbol{x}}_{i}$, $i=1,2,3$ of
the triangle. This requirement gives us the following three equations:

$\displaystyle({\boldsymbol{x}}_{0}-{\boldsymbol{x}}_{i})^{2}=R^{2}\,,\qquad
i=1,2,3\,,$ (37)

where

$R=\frac{AB\cdot BC\cdot CA}{4S}\,,$ (38)

is the radius of the circle that belongs to the $ABC$ plane and goes through
the points $A$, $B$, and $C$. Here $AB$ is the length of the link $AB$ (and
similarly for $BC$ and $CA$),

$S\equiv S_{ABC}{=}\sqrt{p\cdot(p{-}AB)\cdot(p{-}BC)\cdot(p{-}CA)}\,.$ (39)

is the area of the triangle and

$\displaystyle p\equiv p_{ABC}=\frac{AB+BC+CA}{2},$ (40)

is its half-perimeter.

Setting ${\boldsymbol{x}}={\boldsymbol{x}}_{0}$ in Eq. (31) and taking a
difference between Eqs. (37) with $i=1$ and $i=2$, as well as $i=1$ and $i=3$,
we get three independent linear equations that can be combined into the single
matrix equation of the following form:

$\displaystyle{\hat{M}}{\boldsymbol{x}}_{0}+{\boldsymbol{d}}=0\,,$ (41)

with

$\displaystyle{\hat{M}}=\begin{pmatrix}a&b&c\\\ a_{1}&b_{1}&c_{1}\\\
a_{2}&b_{2}&c_{2}\\\ \end{pmatrix}\,,\qquad d=\begin{pmatrix}d\\\ d_{1}\\\
d_{2}\\\ \end{pmatrix}$ (42)

where

$\displaystyle a_{1}$ $\displaystyle=2(x_{2}-x_{1}),\quad$ $\displaystyle
a_{2}=2(x_{3}-x_{1}),$ (43) $\displaystyle b_{1}$
$\displaystyle=2(y_{2}-y_{1}),\quad$ $\displaystyle b_{2}=2(y_{3}-y_{1}),$
(44) $\displaystyle b_{1}$ $\displaystyle=2(z_{2}-z_{1}),\quad$ $\displaystyle
c_{2}=2(z_{3}-z_{1}),$ (45) $\displaystyle d_{1}$
$\displaystyle=({\boldsymbol{x}_{1}})^{2}-({\boldsymbol{x}_{2}})^{2},\quad$
$\displaystyle\quad
d_{2}=({\boldsymbol{x}_{1}})^{2}-({\boldsymbol{x}_{3}})^{2}.$ (46)

The center of the circle is given by the solution of the linear equation (41),
${\boldsymbol{x}}_{0}=-{\hat{M}}{\boldsymbol{d}}$, for which all coefficients
are known and given in Eqs. (36), (42) and (46). The radius of the circle (38)
allows us to check the validity of the Delaunay triangulation, which implies
that the inequality

$\displaystyle({\boldsymbol{x}}_{i}-{\boldsymbol{x}}_{0})^{2}>R^{2}\,,$ (47)

is satisfied for all points from the set other than the original $i=1,2,3$.
Repeating this procedure for all points allows us to cover the sphere
completely with triangular packages without empty areas and overlaps.

Finally, to confirm that our triangulation does not produce artifacts such as
intersections of links or incomplete coverage of the sphere with plaquettes,
it is necessary to prove that the Fibonacci lattice topologically corresponds
to the sphere. For this purpose, we calculate the Euler characteristic number
$\chi=V-R+G$ of the discretized surface (here $V$, $R$, and $G$ are,
respectively, the numbers of its vertices, edges, and faces of a polyhedron
corresponding to the triangulation). Our calculation shows that the Euler
characteristic for each triangulated Fibonacci lattice is $\chi=2$, implying
its topological equivalence to the sphere.

Figure 10: Statistical distributions of the plaquette areas (left) and the
link lengths (right) for the the Delaunay triangulation of the Fibonacci
spheres of the unit radii for different total numbers of sites $N$. Notice the
logarithmic scale of the area plots on the left. Figure 11: The statistical
connectivity properties of the Fibonacci spheres with the Delaunay
triangulation: the percentage of (a) 5- and 7-vertex sites and (b) 6-vertex
sites for lattices with a different number of sites $N$. The inset shows the
region with a relatively small number of points (shown in the light blue
color).

##  Appendix B Statistical properties of Delaunay triangulation of Fibonacci
sphere

One of the main advantages of the Fibonacci lattice is the partitioning of the
surface into triangular plaquettes of approximately equal areas. To verify
this property for the Delaunay triangulation of the sphere used in our work,
we plot in Fig. 10(left) the distribution of plaquettes for the Delaunay
triangulation for lattices of various sizes $N$. The plots show that our
construction is characterized by a remarkably narrow distribution of the area
of triangular plaquettes, which keeps its narrowness with increasing lattice
size $N$.

However, the area distribution is not smooth: instead of the expected Gaussian
profile, the area distribution of Fig. 10(left) shows an Empire State
Building-like profile with several clearly visible steps. Such steps are
probably related to inhomogeneous profiles of the statistical distributions of
the length of the lattice links shown in Fig. 10(right). Thus, the approximate
equality of the areas of the triangular plaquettes does not imply an equality
of the lengths of the links connecting neighboring lattice nodes. In other
words, the statistical distribution of link lengths is not characterized by a
narrow distribution. However, this fact does not affect our construction
because the Ising Hamiltonian (1) does not consider the length of the links
between adjacent sites.

Another attractive characteristic of the Fibonacci sphere is the connectivity
of the lattice, which does affect the Ising Hamiltonian (1). We already
mentioned in the Introduction that the physical properties of the Ising model
defined on planar lattices with different connectivities (for example, square
and triangular lattices) differ. Each site of the square lattice possesses (is
connected with) four nearest neighbors, while the number of neighbors of the
equilateral triangular lattice is six. The number of nearest neighbors of the
Fibonacci lattice sites varies from 5 to 7, as illustrated in Fig. 3(b). The
most common number of neighbors (the “connectivity”) is 6.

The significant heterogeneity of neighbors affects the statistical properties
of the Ising model at Fibonacci lattices. The statistical connectivity
properties of the Fibonacci spheres with the Delaunay triangulation are shown
in Fig. 11. One can observe that the percentage of such 6-connected sites
generally increases as the number of sites increases. However, the uniform
monotonic increase gets broken in isolated critical points
$N^{(i)}_{\mathrm{cr}}$, $i=1,2,\dots$ where the number of sites with the
6-point connectivity experiences sudden jumps towards lower values. At the
same time, the number of sites with 5- and 7-point connectivities shows
discontinuities towards higher values at the same set of critical points. We
observed that the jumps are related to the formation of new Fibonacci spirals,
Fig. 2, in the covering of the sphere.

The discontinuities in connectivity are pronounced at small sizes of the
Fibonacci sphere with $O(30\%)$ jumps at $N\sim 10^{2}$ and $O(10\%)$
discontinuities at $N\sim 10^{3}$, while at larger lattices, $N\sim 10^{5}$,
the jumps become much smoother, reducing to the $O(1\%)$ order of magnitude.

## References

  * Landau and Lifshitz [2013] L. D. Landau and E. M. Lifshitz,  _Statistical Physics: Volume 5_ , Vol. 5 (Elsevier, 2013).
  * Kramers and Wannier [1941] H. A. Kramers and G. H. Wannier, Phys. Rev. 60, 252 (1941).
  * Baxter [2016] R. J. Baxter,  _Exactly solved models in statistical mechanics_ (Elsevier, 2016).
  * Newell [1950] G. F. Newell, Physical Review 79, 876 (1950).
  * Wannier [1950] G. H. Wannier, Physical Review 79, 357 (1950).
  * Zhi-Huan  _et al._ [2009] L. Zhi-Huan, L. Mushtaq, L. Yan, and L. Jian-Rong, Chinese Physics B 18, 2696 (2009).
  * Collins and Petrenko [1997] M. Collins and O. Petrenko, Canadian journal of physics 75, 605 (1997).
  * González [2010] Á. González, Mathematical Geosciences 42, 49 (2010).
  * Saliba and Barnes [2019] E. P. Saliba and A. B. Barnes, The Journal of Chemical Physics 151, 114107 (2019).
  * Marques  _et al._ [2013] R. Marques, C. Bouville, M. Ribardière, L. P. Santos, and K. Bouatouch, Computer Graphics Forum 32, 134 (2013).
  * Swinbank and Purser [2006] R. Swinbank and R. J. Purser, Quarterly Journal of the Royal Meteorological Society 132, 1769 (2006).
  * Song  _et al._ [2022] C.-H. Song, Q.-C. Gao, X.-Y. Hou, X. Wang, Z. Zhou, Y. He, H. Guo, and C.-C. Chien, Physical Review Research 4, 023005 (2022).
  * Saff and Kuijlaars [1997] E. B. Saff and A. B. J. Kuijlaars, The Mathematical Intelligencer 19, 5 (1997).
  * Niederreiter and H. [1994] H. Niederreiter and S. I. H., Journal of Computational and Applied Mathematics 51, 57 (1994).
  * Vogel [1979] H. Vogel, Mathematical Biosciences 44, 179 (1979).
  * Bravais and Bravais [1837] L. Bravais and A. Bravais, Ann. Sci. Nat.Bot. Biol. Vég. 7, 11–42, 42–110, 193–221, 291–348. (1837).
  * Weisstein [2002] E. W. Weisstein,  _CRC concise encyclopedia of mathematics_ (Chapman and Hall/CRC, 2002).
  * González [2009] Á. González, Mathematical Geosciences 42, 49 (2009).
  * Hannay and Nye [2004] J. H. Hannay and J. F. Nye, Journal of Physics A: Mathematical and General 37, 11591 (2004).
  * Keinert  _et al._ [2015] B. Keinert, M. Innmann, M. Sänger, and M. Stamminger, ACM Transactions on Graphics 34, 1 (2015).
  * Delaunay [1934] B. Delaunay, Bulletin de l’Académie des Sciences de l’URSS. Classe des sciences mathématiques et naturelles 6, 793–800 (1934).
  * Skvortsov [2002] A. Skvortsov, Publishing House of Tomsk State University , 128 (2002).