# Exploring Spatial Schema Intuitions in Large Language and Vision Models

Philipp Wicke*
Ludwig-Maximilian-University, Munich
Munich Center for Machine Learning (MCML)
<EMAIL_ADDRESS>
&Lennart Wachowiak*
King’s College London
Imperial College London
<EMAIL_ADDRESS>
Both authors contributed equally.

###### Abstract

Despite the ubiquity of large language models (LLMs) in AI research, the
question of embodiment in LLMs remains underexplored, distinguishing them from
embodied systems in robotics where sensory perception directly informs
physical action. Our investigation navigates the intriguing terrain of whether
LLMs, despite their non-embodied nature, effectively capture implicit human
intuitions about fundamental, spatial building blocks of language. We employ
insights from spatial cognitive foundations developed through early
sensorimotor experiences, guiding our exploration through the reproduction of
three psycholinguistic experiments. Surprisingly, correlations between model
outputs and human responses emerge, revealing adaptability without a tangible
connection to embodied experiences. Notable distinctions include polarized
language model responses and reduced correlations in vision language models.
This research contributes to a nuanced understanding of the interplay between
language, spatial experiences, and the computations made by large language
models.111Project site: https://cisnlp.github.io/Spatial_Schemas/

##  1 Introduction

Large language models (LLMs) excel in varied NLP tasks like text generation,
sentiment analysis, or summarization. Nonetheless, an underexplored facet in
the study of LLMs pertains to the concept of embodiment. Unlike embodied
systems in robotics, where the physical form plays a central role in shaping
the system’s abilities, LLMs lack a direct connection between sensory
perception and physical action. Within this context, we investigate the extent
to which LLMs, despite their lack of direct embodiment, might capture the
implicit, often sensory-derived, conceptual structures that underlie human
language and cognition. In our analysis, we make use of image schema theory,
which provides a set of spatio-temporal cognitive building blocks that are
learned in early infancy based on reoccurring sensorimotor experiences Lakoff
and Johnson (2008); Barsalou (2008); Johnson (2013). For example, the image
schema Support is learned by observing and experiencing objects like tables or
chairs supporting other objects. According to theory, the same image schema is
reused to structure our language and thought, even in regard to abstract
topics. For instance, when talking about emotional support, we say to support
a friend.

Figure 1: Overview of the three experiments

Given that LLMs lack grounding, our research centers around a fundamental
question: Can LLMs encode people’s intuitions about the image schematic basis
of words and phrases? In the subsequent sections, we present the rationale
behind our inquiry and the methodologies for investigating the intricate
relationship between language, embodiment, and the core aspects of human
cognition. In summary, we make the following contributions:

  * •

We use LLMs and vision language models (VLMs) to reproduce three
psycholinguistics experiments (Fig. 1) that connect language to humans’
spatial intuitions

  * •

We find that, in many instances, the answers of the largest models and of
human participants show moderate to strong correlations even though the
model’s language use is not grounded in an embodied or enacted sense

  * •

Crucial differences to human answers remain: the output of small and base
models often shows low correlations, LLMs tend to give polarized responses
(selecting either 1 or 7 on a scale), and VLMs show lowest correlations with
open-source VLMs showing none

##  2 Background

###  2.1 Image Schemas in Cognition & Language

Image schemas, rooted in cognitive linguistics and embodied cognition, have
emerged as a foundational concept in our understanding of human language and
cognition. These dynamic mental structures, originally proposed by Lakoff and
Johnson (2008), serve as the cognitive building blocks for our
conceptualization of the world. Image schemas are pre-linguistic and
perceptual in nature, providing a fundamental means for humans to ground
abstract concepts in concrete sensory and motor experiences Johnson (2013);
Barsalou (2008). This grounding is pivotal in comprehending and communicating
complex ideas, as it bridges the gap between sensorimotor experiences and the
vast array of abstract concepts that are integral to human thought and
language. The centrality of image schemas in cognitive processes underscores
their profound influence on language, from shaping our metaphors and
linguistic expressions to facilitating our ability to reason, plan, and
understand the world Di Paolo et al. (2018); Hampe (2005). An example of an
image schema is Verticality. We can physically experience the image schema
Verticality by standing upright or seeing one object positioned above another.
In turn, these learned schemas can help us comprehend and communicate abstract
concepts such as complex emotions (I feel down) or power dynamics (she ranks
high).

Evidence for image schemas is provided by various psycholinguistic studies
Mandler (1992); Gibbs et al. (1994); Boroditsky (2000); Richardson et al.
(2001); Gibbs (2005). For example, Richardson et al. (2001) has participants
pick one of four arrows ($\uparrow,\downarrow,\leftarrow,\rightarrow$) that
best represent a concrete or abstract action on horizontal or vertical
dimension (e.g. concrete, horizontal: pushed / abstract, vertical: obeyed).
The results indicate a common correlation that points towards underlying
schemas in language. But how are these schematic intuitions represented by
computational models of language?

###  2.2 Image Schemas in Language Models?

LLMs are trained on vast amounts of text and code in order to construct a
model of language that can be used for a variety of NLP tasks without having
been specifically trained on these tasks Radford et al. (2019). The ability of
these emergent properties scales with the size of these models Wei et al.
(2022). Moreover, VLMs, e.g. GPT-4 OpenAI (2023), are trained on images, text
and code. Yet, neither LLMs nor VLMs are embodied systems in the sense that
they never connect “perception to action directly” Brooks (1991). This leads
to the symbol grounding problem, as posed by Harnad (1990), questioning
whether symbols can derive meaning just from other symbols alone (as would be
the case for text-based LLMs) or whether they would need to be connected in a
bottom-up fashion to sensory representations. To this day, such questions are
being critically discussed in the NLP community Bender and Koller (2020).
Research with embodied computational systems (e.g. robots) often works with
the implicit premise (or challenge) that a system’s physical form contributes
to its technical capabilities, its affordances Brohan et al. (2023). One way
to look at the effect of embodiment in LLMs is provided by Wicke (2023), who
shows that the degree of perceived embodiment of an action word can have a
positive effect on an LLM’s capability to interpret figurative language.

##  3 Related Work

###  3.1 Human Behavior Experiments with LLMs

Using LLMs as human stand-in participants for psychology experiments has
recently gained attention Dillion et al. (2023); Harding et al. (2023); Aher
et al. (2023). Such use can be motivated by wanting to generate initial
hypotheses for an experiment, pilot a new design, and gain insight into human
cognition based on the assumption that LLMs trained on a large amount of
human-generated text will produce similar output to that of human participants
Dillion et al. (2023). For instance, Dillion et al. (2023), who propose such a
use, report a high correlation of 0.95 between human answers and GPT-3.5
answers on moral judgment tasks. At the same time, they acknowledge that
current LLMs are bad at capturing variation and diversity present in human
responses and are biased towards responses of people from certain countries,
economic backgrounds, genders, etc. Harding et al. (2023) critique the use of
LLMs to replace human participants and question the informativeness of the
LLM’s output.

Another motivation to simulate psychological experiments with LLMs is to gain
insights not into human cognition but into the capabilities of language models
themselves. Reproducing various experiments with LLMs, one can compare the LLM
output with how humans behaved in the real experiment, thereby establishing
the “human-likeness” of the model’s text generations. The usefulness of such
experiments has also been suggested with respect to psycholinguistics, where
experiments can show what properties of language can be successfully
processed, reproduced, or generated by LLMs Houghton et al. (2023). Dentella
et al. (2023), for example, show that LLMs fail at distinguishing grammatical
and ungrammatical sentences in a similar way to people. Other studies find
that on many psychology tasks, the LLM output is comparable to human answers,
even showing similar cognitive biases Hagendorff et al. (2022); Dasgupta et
al. (2022). Hagendorff et al. (2022) show that these cognitive biases tend to
vanish when experimenting with the most recent models, such as ChatGPT and
GPT-4. Aher et al. (2023) extend the idea of repeating prominent experiments
with LLMs. Specifically, they not only look at a single output of an LLM given
some experiment prompt but try to simulate different demographics by prompting
the model multiple times with different personas attached to each prompt. In
general, one needs to be aware that the same or similar experiments or tests
might have been part of the training data.

###  3.2 NLP for Image Schemas

LLMs have only sparingly been used for image schema-related tasks. Initial
research on the computational processing of image schemas was restricted to
spectral cluster analysis Gromann and Hedblom (2017), whereas more recent work
Wachowiak et al. (2022); Wachowiak and Gromann (2022) uses the language model
XLM-RoBERTa Conneau et al. (2020). Wachowiak and Gromann (2022) show that
language models can be fine-tuned to classify sentences based on eight image
schema classes, with an accuracy between 57% and 80% depending on the
language. Although their model requires first seeing more than 1,000 correctly
annotated samples, these results indicate that, in principle, it is possible
for a neural model trained only on text to pick up the pattern indicating a
specific image schema in natural language. However, in contrast to the
experiments in our work, the language samples they use are manually collated
or created by experts. The psycho-linguistic experiments that build the
foundation for the present work take a different approach, letting multiple
people annotate phrases based on intuitions and felt relatedness. Accordingly,
the expert-annotated image schemas are annotated with a single discrete label
per sample, while the psycho-linguistic experiments inspiring this study lead
to fine-grained annotations using ordinal scales for five different image
schemas per phrase. Kamath et al. (2023) test various VLMs for whether they
can correctly classify simple spatial configurations, such as A being left,
right, under, or over B. Thus, they test for image schematic relations in
their original physical form rather than the abstract extensions. They find
models to perform poorly on the task, with the best model only achieving an
accuracy of 60%, while humans achieve 99%, one of the reasons being that
prepositions occur infrequently and in an ambiguous manner in the training
data. Similar work by Jassim et al. (2023) shows shortcomings of VLMs’
understanding of spatial configurations and intuitive physics when given
different images or video inputs of simulated spatial scenes.

##  4 Method

This section provides an overview of the general model and prompt selection,
while following subsections detail the setup for individual experiments.

#### Model Selection.

We recreate the experiments using closed and open-source language models. The
open-source LLaMA-2 instruction-tuned models Touvron et al. (2023) are chosen
in two sizes222Results of further experiments with non-instruct LLaMA models
and their 7b version are included in the repository.: LLaMA-2-13b and
LLaMA-2-70b. The closed source models include: GPT-3 base (davinci-002),
GPT-3.5 instruction-tuned (text-davinci-003) and GPT-4 (gpt-4-0613).
Instruction-tuned versions have been chosen over base models because
instruction-tuned models tend to perform better Touvron et al. (2023). This
difference can be observed across all of our experiments in varying degrees.
For the third experiment, we include three VLMs: the two open-source models
IDEFICS-80b Laurençon et al. (2023) in base and instruction-tuned variants, as
well as the GPT-4 Vision model (gpt-4-vision-preview).

Model | Model | Model | API | Open
---|---|---|---|---
Type | Name | Size | Endpoint | Source
LLM | GPT-3base | 175b | davinci-003 | -
LLM | GPT-3inst | N/A | text-davinci-003 | -
LLM | GPT-4 | N/A | gpt-4-0613 | -
LLM | LLaMA-2-13b | 13b | Llama-2-13b-chat-hf | $\checkmark$
LLM | LLaMA-2-70b | 70b | Llama-2-70b-chat-hf | $\checkmark$
VLM | GPT-4vision | N/A | gpt-4-vision-preview | -
VLM | IDEFICS-80bbase | 80b | idefics-80b | $\checkmark$
VLM | IDEFICS-80binst | 80b | idefics-80b-instruct | $\checkmark$

Table 1: Models selected for experiments. LLM: large language model, VLM:
vision–language model.

#### Prompt Selection.

Regarding their format, we keep the prompts as close to each original
experiment as possible. Given the information provided in each paper, we reuse
the wording of the image schema definitions and the items being evaluated.
Besides these, we write our own instructive sentence that prompts the LLM to
rate each item since the original instructions given to the human participants
are unfortunately not provided in any of the papers. As suggested by Aher et
al. (2023), we optimize the instructive sentences by choosing a sentence that
maximizes the fraction of valid model answers for each task. In experiments 1
and 2, valid answers consist of the numbers 1 to 7, while in experiment 3, the
valid answers consist of the four possible directions. Given a set of valid
answers $V$ and a prompt $k$, the validity score is computed as:

$\displaystyle\sum_{a\in V}p\left(a|k\right)$ (1)

This way of finding a prompt allows us to get valid answers by only looking at
their form but not at their content. Thus, we adopt this method that prevents
overfitting caused by prompt-engineering.

Recreating a prompt that closely mirrors the approach of the original paper
would involve consolidating all the stimuli into a single list and instructing
the models to associate each stimulus with one of the image schemas. However,
all preliminary experiments revealed that this comprehensive prompt yielded
impractical and unmeaningful responses. Frequently, models redundantly
reproduced identical outputs for each item in the list. Consequently, we opted
to refine our approach by providing the models with a single stimulus per
input prompt.

#### Evaluation.

To evaluate how well the models can predict the human participant’s judgment,
we compute the Spearman correlation coefficient Spearman (1904) between the
human and the LLM ratings for each image schema per experiment. For
interpretation, we use the labels weak, moderate ($>$0.4) and strong ($>$0.7),
common in psychology literature Akoglu (2018). In each of the three
independent experiments, we addressed the challenge of multiple testing by
employing the Benjamini-Hochberg correction for False Discovery Rate (FDR)
Benjamini and Hochberg (1995). This correction was applied to account for the
evaluation of multiple correlations within each experiment, ensuring a
controlled rate of false positives and bolstering the statistical reliability
of our findings. We use the statsmodels python package Seabold and Perktold
(2010) to evaluate the corrected p-values. All reported p-values are corrected
and marked with * for $p<0.05$ as statistically significant.

###  4.1 Experiment 1 - Gibbs et al. (1994)

#### The Original Experiment.

Firstly, we reproduce experiments by Gibbs et al. (1994), testing people’s
intuitions about the image schemas that underlie various uses of the verb to
stand. Given 32 phrases and the definitions of five relevant image schemas,
they asked 27 participants to rate the relatedness between each image schema
and each phrase on a Likert scale from 1 (“not at all related”) to 7 (“very
strongly related”). The image schemas used in this experiment are: Balance,
Verticality, Center–Periphery, Resistance, and Linkage. Participants have to
rate the relatedness of all 32 phrases to a single image schema before moving
to the next (we refer to this data as Gibbsstand). Before giving the rating
for a particular image schema, the schema is introduced with a short
definition. For instance, Verticality is introduced as referring “to the sense
of an extension along an up–down orientation”. The order in which the image
schemas have to be rated is counterbalanced using five different orders
overall. Additionally, Gibbs et al. repeat the experiment using the same 32
phrases but with stand being replaced by a word with a synonymous meaning (we
refer to this data as Gibbssyn). For example, “to stand the test of time” is
substituted by “to pass the test of time”.

#### The LLM Experiment.

To extract ratings from an LLM, we retrieve the most likely answer generated,
i.e. the answer received with a temperature of 0 in the OpenAI API or a top_k
of 1 with HuggingFace. Alternatively, one could consider the probability for
each valid answer, the numbers between 1 and 7, and compute the sum of each
number weighted by their likelihood, normalized by the sum of all seven
numbers’ likelihoods. While this way takes the LLM’s uncertainty into account,
it also requires seven times the amount of compute as simply taking the top
answer. Only minimal result variations led us to choose the cost-effective
methodology of selecting only the top answer.

#### Prompting.

For experimenting, we started with a basic input text as close as possible to
the original experiment, for example:

> Consider the notion of Verticality. Verticality refers to the sense of an
> extension along an up–down orientation. How strongly is the phrase “stand at
> attention” related to this notion on a scale from 1 (not at all related) to
> 7 (very strongly related)?

The different image schema definitions used in these prompts can be found in
Appendix A1. Given such a text as the start of the input, we try to find a way
to end the prompt so that the model’s output probabilities for the next token
converge towards 100% when summed for all valid 7 numbers. This final bit of
the prompt depends on the model. Our prompt ending choices are described in
Section 5.1, and all used phrases are listed in the Appendix A3.

###  4.2 Experiment 2 - Beitel et al. (2001)

#### The Original Experiment.

Beitel et al. (2001) repeat the experimental paradigm established by Gibbs et
al. (1994), however, with a new set of phrases, all containing the preposition
on. Given the focus on the word on instead of to stand, they also select a
different set of image schemas that are more relevant in this case, namely:
Support, Pressure, Constraint, Covering, Visibility. Instead of having access
to a general definition of each image schema, the participants can now check
an example sentence for which five introduction statements explain how each
image schema relates to it. For example, the experiment introduces Support by
saying, “In the case of the use of “on” in the “the book is on the desk”: the
Support relation refers to the desk supporting the book”. A list of
introductions for all five image schemas is available in the Appendix Table
A2.

#### The LLM Experiment.

Given its identical structure, the experiment is conducted in the same manner
as with the data from Gibbs et al. (1994).

###  4.3 Experiment 3 - Richardson et al. (2001)

#### The Original Experiment.

Richardson et al. (2001) provide experimental evidence for image schemas via
two different experiments. The first experiment presents human participants
with lists of 30 verbs ranked by concreteness (based on the MRC
psycholinguistic database Coltheart (1981)). The verbs are represented using
an agent-patient relation through a circle-square depiction, e.g. $\bigcircle$
offended $\square$. Distinguished by abstractness versus concreteness of the
verbs, those verbs were further divided into three groups based on their
primary direction: horizontal, vertical and neutral. This resulted in a 2x3
factor design of concreteness by directionality. The task for each participant
was to choose one of four images that best represents the action. The images
use the same circle-square relation, but with an arrow representing the
directionality on the horizontal and vertical axis
($\leftarrow,\rightarrow,\uparrow,\downarrow$). The choice of images is
depicted in Figure 2. As an example, for the horizontal, concrete item
$\bigcircle$ offended $\square$ the participant needs to select one of the A-D
images (Fig. 2). It is important to note that Richardson et al. (2001) analyse
the results with respect to the primary direction (horizontal/vertical). The
second experiment by Richardson et al. (2001) requires participants to draw a
schematic representation of items from the first experiment. Since we do not
include visual generation, we do reproduce the second experiment in the
presented work.

Figure 2: Target images form the original study by Richardson et al. (2001).
Each participant was asked to match 30 verbs to one of the images (A-D).

#### The LLM Experiment.

The original experiment involved the use of four visual depictions (see Fig.2)
alongside a list of verbs. In this setup, participants were exposed to all the
images and words simultaneously. The word presentation order was randomised,
and the images were labeled A-D. To replicate the experiment, which
necessitates a visual input, we also employ VLMs. In order to enable
comparisons with our selected (text-only) LLMs, we design a tripartite
experimentation.

In the first phase, we opted to translate the visual depictions into textual
representations of the underlying schemas, specifically the words up, down,
left, and right. For the second phase, we conducted the experiment again, but
this time with pseudo-visual renderings using Unicode arrows (↑, ↓, ←, →). In
the final phase, we turned to a limited selection of VLMs to rerun the
experiment with textual and visual input, showing the actual images from Fig.
2. This tripartite approach allows us to investigate the impact of visual
input on language models, compare it to text-only models, and explore the
utility of different forms of visual and textual representations across three
conditions.

Importantly, we compare the model responses with the results from the
Richardson et al. (2001) experiment. Initially, our experiments included the
analysis of left, right, up and down decisions, which require an additional
encoding of the agent-patient relation through a circle-square depiction. This
encoding diffused any correlations and we identified that a restrictions to
primary directions (horizontal/vertical) by grouping up/down and left/right
options in the analysis yields a better reflection of the the models
correlations, since it does not require an agent-patient encoding.

#### Prompting.

Detailed lists of prompts for each condition can be found in Appendix Table
A5. We observed a significant influence on the model’s response based on the
order in which the four options were presented, whether as concept words,
Unicode arrows or images. This is in-line with findings by Pezeshkpour and
Hruschka (2023). Regardless of different option orders (e.g., ↑, ↓, ←, →), the
model most often favoured the first option (e.g., ↑) as a default answer for
most items. To mitigate this effect and to acquire a distribution for each
verb, we conducted all 24 possible permutations of the choices per word (4!
permutations of 30 verbs).

VLMs require a label attached to each image, which allows them to formulate a
choice, e.g. “Image A”. We observed that labels with an implicit (alphabetical
or numerical) order, e.g. A, B, C or 1, 2, 3, introduce a selection bias.
Hence, we use arbitrary labels (VMBR, WJZX, XQHL, YGPK).

Lastly, the GPT-4 models are a commercial product that include safety
guardrails in their system prompt in order to avoid hate speech, abuse or
disinformation. Often, GPT-4 models would tend to refuse to answer subjective
questions. Consequently, we modified instructions for GPT-4 in order to force
subjective model answers (Fig. A5).

##  5 Results

Exp. | Image Schema | LLaMA-2 | GPT | Avg.
---|---|---|---|---
|  | 13b-chat | 70b-chat | GPT-3base | GPT-3inst | GPT-4 |
Gibbsstand | Verticality | 0.26 | 0.41* | N/A | 0.53* | 0.69* | 0.47
Balance | 0.27 | 0.38 | -0.05 | 0.37 | 0.49* | 0.29
Center–Periphery | 0.20 | 0.36 | N/A | 0.82* | 0.56* | 0.49
Linkage | 0.24 | -0.06 | N/A | 0.46* | 0.61* | 0.31
Resistance | 0.41* | 0.48* | N/A | 0.71* | 0.82* | 0.60
Gibbssyn | Verticality | 0.22 | 0.57* | N/A | 0.49* | 0.70* | 0.49
Balance | 0.36 | 0.50* | 0.17 | 0.50* | 0.54* | 0.41
Center–Periphery | 0.22 | 0.32 | 0.16 | 0.67* | 0.67* | 0.41
Linkage | 0.61* | 0.32 | N/A | 0.61* | 0.24 | 0.45
Resistance | 0.30 | 0.54* | N/A | 0.61* | 0.77* | 0.56
Beitelon | Support | 0.19 | 0.32 | N/A | 0.48* | 0.62* | 0.40
Pressure | 0.37* | 0.72* | N/A | 0.79* | 0.37* | 0.56
Constraint | 0.49* | 0.37* | N/A | 0.60* | 0.47* | 0.48
Covering | -0.15 | 0.41* | N/A | 0.46* | 0.68* | 0.35
Visibility | 0.24 | 0.38* | N/A | 0.69* | 0.62* | 0.48
Table 2: Spearman correlation: model answers and human answers. * for
$p<0.05$, bold = highest correlation

###  5.1 Experiment 1 — Gibbs et al. (1994)

#### Prompt Selection.

Depending on the chosen model, different prompt endings worked well in making
the respective LLM generate valid answers, that is, a number between 1 and 7.
After considering the output probabilities over a subset of 15 input samples,
we ended up using either prompt ending “Only answer with the score:” or “I
choose the number”. In each case, the chosen prompt ending guarantees that
more than 99% of the probability mass is allocated to the valid answers. For
GPT-4, the log-probabilities are not accessible through the API, which is why
we reused the prompt ending chosen for davinci-003. Based on the 15 tested
samples, we also assessed whether the models are very sensitive to these
scores in relation to the number they output. Fortunately, we found that the
scores are relatively stable and, on average, differ only around half a point
on the 7-point scale.

#### Answer Correlations.

First, we compare the two evaluation methods presented in Section 4.3, i.e.,
simply extracting the most likely answer or computing the average of all valid
answers based on their likelihood. Using davinci-003, we find that, on
average, the thereby extracted answers are only 0.39 points apart. Thus, we
decided to use the simpler method of using the most likely answer when
presenting the results from hereinafter since it has the advantage of
incurring fewer computational costs.

Table 2 shows the Spearman correlations between human and model answers for
the five different image schemas. For 70% of the image schemas, GPT-4
generates the answers most similar to those of human participants, with
GPT-3inst generating the most similar answers for the remaining 30%. On
average, GPT-4 shows a correlation of 0.61, which can be interpreted as
moderate. In comparison, the LLaMA-2 models produce answers that are more
dissimilar to those of humans, as indicated by some moderate correlations and
many correlations that are not statistically significant. The 13b variant
achieves an average correlation of 0.31, and the 70b variant of 0.38. These
lower correlations can be partly attributed to the fact that the LLaMA-2
models tend to answer with either 4 or 7, largely ignoring other options on
the scale. The only model that is trained without reinforcement learning from
human feedback, GPT-3base, fails to generate insightful answers, nearly always
generating a 4.

###  5.2 Experiment 2 — Beitel et al. (2001)

Results are similar to the previous experiment, with the answers of GPT-4 and
GPT-3instruct showing the highest similarity to those of human participants,
ranging between correlations of 0.4 and 0.8.

###  5.3 Experiment 3 — Richardson et al. (2001)

TEXT | Llama-2-13bchat | Llama-2-70bchat | GPT-3base | GPT-3inst | GPT-4 | Avg.
---|---|---|---|---|---|---
Horiz. / Vert. | 0.53* / 0.54* | 0.59* / 0.59* | 0.68* / 0.68* |  0.72* / 0.72* | 0.58* / 0.58* | 0.57 / 0.59
PSEUDO-VISUAL | Llama-2-13bchat | Llama-2-70bchat | GPT-3base | GPT-3inst | GPT-4 | Avg.
Horiz. / Vert. | 0.68* / 0.79* | 0.51* / 0.51* | 0.17 / 0.16 | 0.68* / 0.68* |  0.82* / 0.83* | 0.61 / 0.66
VISUAL | IDEFICS-80bbase | IDEFICS-80binst | GPT-4vision |  |  | Avg.
Horiz. / Vert. | 0.00 / 0.00 | -0.01 / -0.01 |  0.57* / 0.56* |  |  | 0.19 / 0.18

Table 3: Spearman correlation: model answers and human answers. * for
$p<0.05$, bold = highest correlation

#### Prompt Selection.

In contrast to the previous experiments, the model output is not an ordinal
measure (1-7), but a nominal classification. Analogously, we use a subset of
input samples along with various different prompt endings to identify that the
chosen prompt ending guarantees that a high probability mass is allocated to
the labels.

Notable results can be reported for text and vision prompting. For text-based
models, the addition of quotation marks around the label (e.g. 'up') increases
the probability of the model to choose a valid label if prompted with a quote
at the end. In general, using a “Question:” and “Answer:” structure improves
label likelihood. For VLMs we cannot obtain the log-probabilities, therefore
we follow the examples provided by the model
developers333https://huggingface.co/docs/transformers/main/en/tasks/idefics.

#### Results of Primary Directionality.

For all models and all conditions, we summarise the Spearman correlations in
Table 3 (detailed results for all choices are listed in the Appendix Tab. A6).
We can observe strong correlations (> 0.7) in the textual and pseudo-visual
conditions, but not in the visual one. In the textual condition, all models
show a significant correlation with the human choices with respect to their
choice of a primary directionality (horizontal/vertical). Here, GPT-3inst
shows a strong correlation of 0.72. Correlations for pseudo-visual conditions
are higher except for one outlier: GPT-3base, which has mostly selected the ↑
for all items irrespective of the order of choices (i.e. order of arrows in
the prompt). On the contrary, GPT-4 has the highest correlation across all
models and conditions for the pseudo-visual task with 0.82 (horizontal) and
0.83 (vertical). In the visual condition, both open-source versions of IDEFICS
do not show any correlation with the human responses. GPT-4vision achieves a
significant, but moderate correlation with 0.57 (horizontal) and 0.56
(vertical). On average, correlations between model and human answers are
higher in the pseudo-visual condition despite the outlier of the GPT-3base
model.

##  6 Discussion

Figure 3: Distribution of image schema choice for items “bombed” and “lifted”
by humans (bold) and GPT-4 (light).

We explored if LLMs capture human intuitions about image schematic basis in
language through three psycholinguistic experiments comparing LLMs and VLMs.
Model responses often correlate with people’s, especially in larger models,
although discrepancies exist for certain image schemas. Nevertheless, the
models reflect spatial primitive intuitions, potentially stemming from their
ability to model words, their contextual use, and their relation to schema
definitions. Another possibility is that the original experiment papers,
serving as training data, might contribute to the models’ reproduction of
observed patterns, although parsing the original papers’ results effectively
is unlikely.

At the same time, one might wonder why, in some cases, the model answers are
so far apart from human answers. Besides the explanation of the lack of
embodied experience, one aspect to consider is that the LLMs only had access
to one item at a time when rating stimuli. Thus, models are unable to rate
items relative to each other — a strategy a human participant is likely to
adopt. For instance, given the two stimuli “stand in awe” and “the clock
stands on the mantle” separately, GPT-4 gives scores of 2 and 1 for the image
schema Balance. People, on the other hand, tend to relate Balance much more
strongly to a clock standing on a mantle. This intuition is captured when
prompting the model with both stimuli at once, thus allowing the model to
provide relative scores; in this case, the model provides a score of 2 for the
first phrase but a score of 4 for the clock on the mantel, thus being much
closer to human scores. However, prompting the model to provide all answers at
once is currently not feasible, as described in Section 4. This missing
comparison between items is also reflected in the fact that the models tend to
use the extremes of the scale, i.e., rating items with 1 or 7. Another
possible answer for the differences is related to the partially small
participant pool used in the original experiments. The three experiments
recruited 27, 24, and 173 participants. Therefore, the originally recorded
human answers might not be very robust, and future experiments should rerun
similar setups not only with novel stimuli but also with larger participant
pools to enable a more robust data set for comparison.

The overall weakest correlations are observed for VLMs, particularly with
open-source model IDEFICS, displaying meaningless responses. For instance,
IDEFICS-80binst consistently selects the left-arrow image, disregarding the
item, image order, or randomized labels. This failure is attributed to VLMs
being trained on natural images, unsuitable for interpreting the highly
abstract line drawings. Even GPT-4vision, despite showing moderate
correlations (0.56-0.57), occasionally provides random answers, deviating from
human consensus (see Fig. 3). This discrepancy raises questions about the
models’ alignment with human intuitions.

##  7 Conclusion

This exploratory study is the first of its kind to reproduce psycholinguistic
experiments in order to explore spatial schema intuitions. Moreover, it
provides evidence that LLMs are able to reflect those intuitions in different
tasks and setups. The results also point out that despite the duo-modality,
VLMs do not encode the spatial understanding as effectively as their textual
origins. Future research aims to assess models using novel, collected stimuli,
ensuring no prior exposure during training. Our current findings provide a
foundation for formulating precise, testable hypotheses in subsequent
experiments. Additionally, we would like to extend this line of research to
include a multilingual analysis.

##  8 Limitations

While our investigation into the applications of proprietary models, such as
GPT-3 and GPT-4, offers valuable insights, it is essential to acknowledge the
inherent limitations associated with their use. The opaque nature of the
underlying mechanisms in proprietary models poses a significant limitation.
The understanding is constrained by the lack of detailed information on the
architectural intricacies, leaving us to make assumptions based on analogies
to open-source models. It is crucial to recognize that variations in
architecture, data, and parameters between proprietary and open-source models
impact the generalizability of our findings.

Additionally, our reliance on psycholinguistic data introduces notable
limitations. Firstly, the temporal aspect of the data is a concern, given its
age range of 30 to 23 years. Language evolves over time, and the potential
disparities between our data and contemporary linguistic trends may affect the
applicability of our results. Therefore, we plan to replicate the studies to
gauge temporal robustness.

Secondly, the incorporation of original papers into the training data of
proprietary models, particularly the LLM and VLM, poses challenges. This
integration may introduce biases, potentially influencing the outcomes of our
experiments. Yet, the original papers’ results are presented in formats that
are unlikely to have undergone parsing during the training procedures,
especially within the context of the VLM, yet this remains hard to prove.

##  9 Considerations and Impact of the Work

#### Environmental Impact

Overall, text generations with the OpenAI API cost 25.85$. Text generations
with LLaMA models were run on two clusters. In those experiments, we utilized
8 NVIDIA RTX A6000 (48GB) GPUs for a 4-hour runtime, each with a power
consumption of 300 W, resulting in an estimated total power consumption of 9.6
kWh and a CO2 emission of approximately 3.984 kg. Additionally, we used 2
NVIDIA A100 (40GB) GPUs for a 39-hour runtime, also consuming 300 W each,
contributing to a combined total power consumption of 33 kWh and a CO2
emission of approximately 13.7 kg.

## References

  * Aher et al. (2023) Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai. 2023.  Using large language models to simulate multiple humans and replicate human subject studies.  In _International Conference on Machine Learning_ , pages 337–371. PMLR.
  * Akoglu (2018) Haldun Akoglu. 2018.  User’s guide to correlation coefficients.  _Turkish journal of emergency medicine_ , 18(3):91–93.
  * Barsalou (2008) Lawrence W Barsalou. 2008.  Grounded cognition.  _Annu. Rev. Psychol._ , 59:617–645.
  * Beitel et al. (2001) Dinara A Beitel, Raymond W Gibbs Jr, and Paul Sanders. 2001.  The embodied approach to the polysemy of the spatial preposition on.  In _Polysemy in cognitive linguistics_ , pages 241–260. John Benjamins.
  * Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020.  Climbing towards NLU: On meaning, form, and understanding in the age of data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5185–5198, Online. Association for Computational Linguistics.
  * Benjamini and Hochberg (1995) Yoav Benjamini and Yosef Hochberg. 1995.  Controlling the false discovery rate: a practical and powerful approach to multiple testing.  _Journal of the Royal statistical society: series B (Methodological)_ , 57(1):289–300.
  * Boroditsky (2000) Lera Boroditsky. 2000.  Metaphoric structuring: Understanding time through spatial metaphors.  _Cognition_ , 75(1):1–28.
  * Brohan et al. (2023) Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. 2023.  Do as i can, not as i say: Grounding language in robotic affordances.  In _Conference on Robot Learning_ , pages 287–318. PMLR.
  * Brooks (1991) Rodney A Brooks. 1991.  Intelligence without representation.  _Artificial intelligence_ , 47(1-3):139–159.
  * Coltheart (1981) Max Coltheart. 1981.  The mrc psycholinguistic database.  _The Quarterly Journal of Experimental Psychology Section A_ , 33(4):497–505.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Dasgupta et al. (2022) Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022.  Language models show human-like content effects on reasoning.  _arXiv preprint arXiv:2207.07051_.  DeepMind.
  * Dentella et al. (2023) Vittoria Dentella, Fritz Günther, and Evelina Leivada. 2023.  Systematic testing of three language models reveals low language accuracy, absence of response stability, and a yes-response bias.  _Proceedings of the National Academy of Sciences_ , 120(51):e2309583120.
  * Di Paolo et al. (2018) Ezequiel A Di Paolo, Elena Clare Cuffari, and Hanne De Jaegher. 2018.  _Linguistic bodies: The continuity between life and language_.  MIT press.
  * Dillion et al. (2023) Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023.  Can ai language models replace human participants?  _Trends in Cognitive Sciences_.
  * Gibbs (2005) Raymond W Gibbs. 2005.  The psychological status of image schemas.  _From perception to meaning: Image schemas in cognitive linguistics_ , 29:113–136.
  * Gibbs et al. (1994) Raymond W Gibbs, Dinara A Beitel, Michael Harrington, and Paul E Sanders. 1994.  Taking a stand on the meanings of stand: Bodily experience as motivation for polysemy.  _Journal of semantics_ , 11(4):231–251.
  * Gromann and Hedblom (2017) Dagmar Gromann and Maria M Hedblom. 2017.  Kinesthetic mind reader: A method to identify image schemas in natural language.  _Proceedings of Advancements in Cogntivie Systems_.
  * Hagendorff et al. (2022) Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. 2022.  Machine intuition: Uncovering human-like intuitive decision-making in gpt-3.5.  _arXiv preprint arXiv:2212.05206_.
  * Hampe (2005) Beate Hampe. 2005.  Image schemas in cognitive linguistics: Introduction.  _From perception to meaning: Image schemas in cognitive linguistics_ , 29:1–12.
  * Harding et al. (2023) Jacqueline Harding, William D’Alessandro, NG Laskowski, and Robert Long. 2023\.  Ai language models cannot replace human research participants.  _AI & SOCIETY_, pages 1–3.
  * Harnad (1990) Stevan Harnad. 1990.  The symbol grounding problem.  _Physica D: Nonlinear Phenomena_ , 42(1-3):335–346.
  * Houghton et al. (2023) Conor Houghton, Nina Kazanina, and Priyanka Sukumaran. 2023.  Beyond the limitations of any imaginable mechanism: large language models and psycholinguistics.  _arXiv preprint arXiv:2303.00077_.
  * Jassim et al. (2023) Serwan Jassim, Mario Holubar, Annika Richter, Cornelius Wolff, Xenia Ohmer, and Elia Bruni. 2023.  Grasp: A novel benchmark for evaluating language grounding and situated physics understanding in multimodal language models.  _arXiv preprint arXiv:2311.09048_.
  * Johnson (2013) Mark Johnson. 2013.  _The body in the mind: The bodily basis of meaning, imagination, and reason_.  University of Chicago press.
  * Kamath et al. (2023) Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023.  What’s ”up” with vision-language models? investigating their struggle with spatial reasoning.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
  * Lakoff and Johnson (2008) George Lakoff and Mark Johnson. 2008.  _Metaphors we live by_.  University of Chicago press.
  * Laurençon et al. (2023) Hugo Laurençon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. 2023.  OBELICS: An open web-scale filtered dataset of interleaved image-text documents.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Mandler (1992) Jean M Mandler. 1992.  How to build a baby: Ii. conceptual primitives.  _Psychological review_ , 99(4):587.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Pezeshkpour and Hruschka (2023) Pouya Pezeshkpour and Estevam Hruschka. 2023.  Large language models sensitivity to the order of options in multiple-choice questions.  _arXiv preprint arXiv:2308.11483_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Richardson et al. (2001) Daniel C Richardson, Michael J Spivey, Shimon Edelman, and Adam J Naples. 2001.  " language is spatial": Experimental evidence for image schemas of concrete and abstract verbs.  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ , volume 23.
  * Seabold and Perktold (2010) Skipper Seabold and Josef Perktold. 2010.  statsmodels: Econometric and statistical modeling with python.  In _9th Python in Science Conference_.
  * Spearman (1904) C. Spearman. 1904.  The proof and measurement of association between two things.  _The American Journal of Psychology_ , 15(1):72–101.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wachowiak and Gromann (2022) Lennart Wachowiak and Dagmar Gromann. 2022.  Systematic analysis of image schemas in natural language through explainable multilingual neural language processing.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 5571–5581, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Wachowiak et al. (2022) Lennart Wachowiak, Dagmar Gromann, and Chao Xu. 2022.  Drum up SUPPORT: Systematic analysis of image-schematic conceptual metaphors.  In _Proceedings of the 3rd Workshop on Figurative Language Processing (FLP)_ , pages 44–53, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022.  Emergent abilities of large language models.  _Transactions on Machine Learning Research_.  Survey Certification.
  * Wicke (2023) Philipp Wicke. 2023.  Lms stand their ground: Investigating the effect of embodiment in figurative language interpretation by language models.  _Findings of the Association for Computational Linguistics: ACL2023_.

Appendix Image Schema |  Definition/Introduction
---|---
Balance |  Consider the notion of Balance. Balance refers to your sense of symmetry or stability relative to some point within your body.
Verticality |  Consider the notion of Verticality. Verticality refers to the sense of an extension along an up–down orientation.
Center–Periphery |  Consider the notion of Center–Periphery. Center–periphery refers to the experience of some objects or events as central while surrounding objects and events are peripheral or to the outside.
Resistance |  Consider the notion of Resistance. Resistance refers to the experience of your body opposing some external force.
Linkage |  Consider the notion of Linkage. Linkage refers to the perception of a connection between objects or events.
Table A1: Image schema definitions provided in the prompts (Experiment 1) Image Schema |  Definition/Introduction
---|---
Support |  In the case of the use of “on” in “the book is on the desk”: the Support relation refers to the desk supporting the book.
Pressure |  In the case of the use of “on” in “the book is on the desk”: the Pressure relation refers to the book exerting some pressure on the desk.
Constraint |  In the case of the use of “on” in “the book is on the desk”: the Constraint relation refers to the desk constraining the possible motions of the book.
Covering |  In the case of the use of “on” in “the book is on the desk”: the Covering relation refers to the book concealing the part of the desk that is under the book.
Visibility |  In the case of the use of “on” in “the book is on the desk”: the Visibility relation refers to the book being visible on the desk.
Table A2: Image schema definitions provided in the prompts (Experiment 2)

Stimuli with stand (Experiment 1) | Stimuli with synonym (Experiment 1) | Stimuli with on (Experiment 2)
---|---|---
stand at attention | be at attention | The family depends on the father
stand out in several sports | be distinguished in several sports | There is a physician on call
to stand firm | to hold firm | All books are on sale
don’t stand for such treatment | don’t allow such treatment | The band is on tour
to stand the test of time | to pass the test of time | The boat is on course
united we stand | united we are strong | The bus is on schedule
we stand on 30 years of experience | we are backed up by 30 years experience | Jeff is on time
let the issue stand | let the issue remain as is | Sam is on his way home
let the mixture stand | leave the mixture undisturbed | She puts the blame on my actions
get stood up for a date | have a date with someone who didn’t show up | He pulled a gun on me
he stands six-foot nine | he measures six-foot nine | Pat has been on sick leave
the clock stands on the mantle | the clock is on the mantle | There is a parade on Sunday
one-night stand | one-night fling | The program will be broadcast on CBS
to stand to profit | to be in the position to make a profit | Joan works on the committee
to stand in someone else’s shoes | to be in someone else’s shoes | Linda is very knowledgeable on this subject
on the witness stand | on the witness platform | The factory workers are on strike for a second day
stand in awe | be in aw | These cold nights are very hard on the homeless
the police told them to stand back | the police told them to get back | They ordered a court martial to be held on him
stand by your man | support your man | Pam is on a diet
the engine can’t stand the constant wear | the engine can’t endure the constant wear | The dog is on the leash
stand on shaky ground | to be on shaky ground | She is on the pill
to stand accused | to be accused | The ship is on the anchor
the house stands in the clearing | the house is in the clearing | The lunch is on George
the barometer stands at 30 inches | the barometer is at 30 inches | He lives on a pension
as the matter now stands | as the matter now exists | The boss is on my neck
the part stands for the whole | the part represents the whole | There are fifty nurses on the hospital staff
it stands to reason | it conforms with reason | There is ten cent interest on the dollar
they did nothing but stand around | they did nothing but hang around | The vase is on the table
to stand against great odds | to face great odds | The fish is on the surface
a standing ovation | a roaring ovation | The fly is on the ceiling
the boss always stands over me | the boss always hovers over me | She has a beautiful ring on her finger
he stands committed | he remains committed | The house rests on the foundation
|  | They hung a picture on the wall
|  | I have a cut on my finger
|  | I have a rash on my back
|  | The actor is on stage
|  | He resides on the continent

Table A3: Items the LLMs had to rate in experiments 1 and 2 Action words
(Experiment 3)
---
fled, pointed at, pulled, pushed, walked, hunted, impacted, perched, showed,
smashed, bombed, flew, floated, lifted, sank, argued with, gave to, offended,
rushed, warned, owned, regretted, rested, tempted, wanted, hoped, increased,
obeyed, respected, succeeded
Table A4: List of action words from Richardson et al. (2001). These words are
used for Experiment 3.
Condition |  Prompt
---|---
Textual |  Given the concepts: [DIRECTION WORDS]. For the concept that best represents the event '[ACTION WORD]', I would choose '
Pseudo-Visual |  Given the event '[ACTION WORD]', which of the following arrows best represents this event: [UNICODE ARROWS]. A research participant would choose the arrow:
Visual |  [Instruction: You are shown four images. Select one of the four images to answer the question.] Image [IMAGE LABEL]:[IMAGE] Image [IMAGE LABEL]:[IMAGE] Image [IMAGE LABEL]:[IMAGE] Image [IMAGE LABEL]:[IMAGE] Question: Which is the best image to describe the event '[ACTION WORD]'? Answer: For '[ACTION WORD]' it is Image
GPT-4chat |  [TEXTUAL or PSEUDO-VISUAL PROMPT] You are a participant in a research experiment. Even if the answer is subjective, provide it. Do not say it is subjective.
Table A5: Prompts of the three different modality-conditions used in Experiment 3. DIRECTION WORDS: 'up', 'down', 'left', 'right'. UNICODE ARROWS: ↑, ↓, ←, →. IMAGE LABEL: VMBR, WJZX, XQHL, YGPK. IMAGE see Fig. 1 (without letter labels).  Condition | Image Schema | LLaMA-2 | GPT | Avg.
---|---|---|---|---
|  | 13b-chat | 70b-chat | GPT-3base | GPT-3inst | GPT-4 |
TEXTUAL | up | 0.49 | 0.66 | 0.67 | 0.63 | 0.66 | 0.62
down | 0.21 | 0.23 | 0.34 | 0.31 | 0.33 | 0.28
left | -0.12 | 0.25 | 0.23 | 0.37 | 0.24 | 0.24
right | 0.57 | 0.56 | 0.53 | 0.56 | 0.41 | 0.53
PSEUDO-VISUAL | $\uparrow$ | 0.44 | 0.61 | -0.09 | 0.49 | 0.70 | 0.43
$\downarrow$ | 0.44 | 0.14 | 0.31 | 0.42 | 0.49 | 0.36
$\leftarrow$ | 0.18 | 0.29 | 0.03 | 0.31 | 0.18 | 0.20
$\rightarrow$ | 0.31 | 0.43 | N/A | 0.56 | 0.69 | 0.50
Table A6: Spearman correlation between model answers and human answers. bold = highest correlation | Image | IDEFICS | GPT | Avg.
---|---|---|---|---
|  | 80b | 80b-inst | GPT-4 |
VISUAL |  | 0.12 | 0.10 | 0.51 | 0.24
| -0.12 | 0.01 | 0.27 | 0.05
| -0.23 | 0.02 | 0.11 | -0.03
| 0.49 | 0.30 | 0.53 | 0.44
Table A7: Spearman correlation between model answers and human answers. bold =
highest correlation


# Correlations of spin splitting and orbital fluctuations due to 1/f charge
noise in the Si/SiGe Quantum Dot

M. Kepa  Institute of Physics, Polish Academy of Sciences, al. Lotników 32/46,
PL 02-668 Warsaw, Poland  Ł. Cywiński  Institute of Physics, Polish Academy of
Sciences, al. Lotników 32/46, PL 02-668 Warsaw, Poland  J. A. Krzywda
<EMAIL_ADDRESS>Institute of Physics, Polish Academy of Sciences, al.
Lotników 32/46, PL 02-668 Warsaw, Poland  Lorentz Institute and Leiden
Institute of Advanced Computer Science, Leiden University, P.O. Box 9506, 2300
RA Leiden, The Netherlands

###### Abstract

Fluctuations of electric fields can change the position of a gate-defined
quantum dot in a semiconductor heterostructure. In the presence of magnetic
field gradient, these stochastic shifts of electron’s wavefunction lead to
fluctuations of electron’s spin splitting. The resulting spin dephasing due to
charge noise limits the coherence times of spin qubits in isotopically
purified Si/SiGe quantum dots. We investigate the spin splitting noise caused
by such process caused by microscopic motion of charges at the semiconductor-
oxide interface. We compare effects of isotropic and planar displacement of
the charges, and estimate their densities and typical displacement magnitudes
that can reproduce experimentally observed spin splitting noise spectra. We
predict that for defect density of $10^{10}$ cm-2, visible correlations
between noises in spin splitting and in energy of electron’s ground state in
the quantum dot, are expected.

Due to the weakness of spin-orbit coupling in conduction band of silicon, and
possibility of isotopic enrichment leading to removal of spinful 29Si nuclei,
a spin of a single electron in an isotopically purified silicon quantum dot
(QD) has the longest coherence time of all the QD-based spin qubits Chan  _et
al._ (2018); Yoneda  _et al._ (2018); Tanttu  _et al._ (2019); Zajac  _et al._
(2018); Mills  _et al._ (2022); Philips  _et al._ (2022) However, this
relative isolation of the spin degree of freedom from external influence makes
it hard to perform spin rotations, i.e. single-qubit quantum gates. The
currently commonly accepted solution is to expose such qubits to magnetic
field gradients generated by nearby nanomagnets.Tokura  _et al._ (2006);
Neumann and Schreiber (2015); Aldeghi, Allenspach, and Salis (2023) The
gradients of magnetic fields transverse to a global quantization axis, due to
constant external $B$ field, allow then for dot-selective electron spin
resonance operations driven by ac voltages on gates defining the QDs.Takeda
_et al._ (2016); Yoneda  _et al._ (2018); Zhao  _et al._ (2019); Corna  _et
al._ (2018); Struck  _et al._ (2020) The gradients of the longitudinal
components of the magnetic field lead to qubit-specific spin splittings,
further diminishing the addressing errors, when one of two nearby qubits is
driven by ac voltages with appropriately chosen frequency.Li  _et al._ (2018);
Heinz and Burkard (2021) However, the presence of the latter gradient makes
the splitting of a given spin sensitive to fluctuations of electric fields, as
charge-noise-induced fluctuations of an electron position translate into noise
in its spin splitting.

Charge noise affecting electron’s orbital energy levels has been recently
characterized for many silicon-based QDs in heterostructures that are
considered as platforms for large-scale semiconductor-based quantum
computer.Connors  _et al._ (2019, 2022); Yoneda  _et al._ (2022); Rojas-Arias
_et al._ (2023); Struck  _et al._ (2020) This spurred theoretical
investigations of microscopic models of $1/f$ charge noise that could
reproduce the observed levels of electron orbital energy fluctuations in SiMOS
Shehata  _et al._ (2022) and Si/SiGe QDs.Kępa  _et al._ (2023) It is commonly
assumed that the charge noise in such metal-insulator-semiconductor structures
is due to dynamics of charged defects located at the semiconductor/oxide
interface.Culcer and Zimmerman (2013); Shehata  _et al._ (2022); Wuetz  _et
al._ (2023) Calculations in Shehata  _et al._ , 2022 and Kępa  _et al._ , 2023
have established that in order to reproduce the typical order of magnitude of
$1/f$ charge noise seen in relevant QDs, the charges should remain trapped at
the interface, switching between positions separated by $\delta r\\!<\\!1$ nm,
i.e. acting as two-level fluctuators (TLFs). Ranges of densities, $\rho$ of
such TLFs, and values of $\delta r$ that lead to reproduction of observed
noise, were described in these papers. Here we follow up on the study from
Kępa  _et al._ , 2023 by adding a magnetic field gradient, and considering
fluctuations of spin splitting due to dynamics of sources of charge noise.
Comparison of the results of the simulations with measured data on spin
dephasing in Si/SiGe QDs exposed to $B$ field gradient, leads to further
narrowing down of ranges of parameters of the microscopic models of charge
noise that are consistent with state-of-the art experiments.

We use the model of Si/SiGe QD device from Kępa  _et al._ , 2023, and start
with the model of TLFs described there. We consider charges localized at
random positions near the semiconductor-oxide interface, with $\rho$ being
their interface-averaged planar density, and assume that each of them switches
between two positions, $\mathbf{r}_{n}$ and
$\mathbf{r}_{n}+\delta\mathbf{r}_{n}$, where the components of
$\delta\mathbf{r}_{n}$ are drawn from independent Gaussian distributions of
zero average and equal variances, such that
$\langle|\mathbf{r}_{n}|\rangle\\!=\\!\langle\sqrt{\delta x^{2}+\delta
y^{2}+\delta z^{2}}\rangle\equiv\delta$r is the rms of charge displacement.

The model of the device consists of 600x600 nm
$\text{Si}_{0.7}\text{Ge}_{0.3}$-Si quantum well (QW) where electrons are
trapped, located about 80 nm below the metallic electrodes, and has been
implemented in COMSOL (See Kępa  _et al._ , 2023 for device description).
Dashed overlay on Fig. 1a shows the outline of metallic gates and the channel.
We compute the ground and first excited state of the stationary Schroedinger
equation in a 10 nm high Si layer of QW, using a built-in COMSOL eigenvalue
solver. In this study, the voltage on the plunger gate in the middle of the
device is set to 0.25 V, while keeping other gates grounded, giving a
realistic orbital gap $\hbar\omega=E_{1}-E_{0}\approx 2$ meV. To study the
effect of charged defects, we define a 31x31 regular grid in XY plane. At each
point of the grid, and at four different heights
$z_{0}=101.75,102.0,102.25,102.5$nm, close to semiconductor-oxide interface,
we place a single charge with $q=-|e|$. For each defect position, we extract
the expectation values of position operator
$\mathbf{R}\equiv\langle\psi|\hat{\mathbf{r}}|\psi\rangle=\int_{V}\mathbf{r}|\psi|^{2}dv$,
where $\psi$ is wavefunction of the ground state and $V$ represents the volume
of Si layer. This procedure is followed by the interpolation.

In presence of longitudinal magnetic field gradient, the displacement of the
electron wavefunction is translated to a change in Zeeman splitting,

$\delta\Omega(\mathbf{r}_{n})=g\mu_{\text{B}}\,\Delta\mathbf{B}_{\parallel}\,\cdot\delta\mathbf{R}(\mathbf{r}_{n}),$
(1)

where we used electronic g-factor $g$, Bohr magneton $\mu_{B}$,
$\Delta\mathbf{B}_{\parallel}$ as gradient of longitudinal component of the
magnetic field at the location of the QD, and
$\delta\mathbf{R}(\mathbf{r}_{n})\\!=\\![X(\mathbf{r}_{n}),Y(\mathbf{r}_{n}),Z(\mathbf{r}_{n})]^{T}$
as the shift of QD position due to presence of the defect at $\mathbf{r}_{n}$.
For simplicity, in this paper we will assume that the longitudinal field
gradient is along the x-axis only, $\Delta\mathbf{B}_{\parallel}=(\Delta
B_{\parallel},0,0)^{T}$, i.e. in the direction parallel to the channel and
perpendicular to metallic gates (See Fig. 1). To calibrate our results against
experimental results we use here $\Delta B_{\parallel}=0.2\,$mT/nm from Yoneda
_et al._ (2022, 2018), which in Si is equivalent to the gradient in Zeeman
splitting $\Delta\Omega/\Delta X\approx 0.025\,\mu$eV/nm. Gradient along a
single direction means that the shift in spin splitting is sensitive to the
change in $x$-component of QD position only, i.e.
$\delta\Omega(\mathbf{r}_{n})=g\mu_{B}\Delta B_{\parallel}\delta
X(\mathbf{r}_{n})$.

The raw data are obtained using finite element method implemented in COMSOL
software, as described above, and presented in Fig. 1a for a single plane
$z_{0}=102$nm. The resulting shift in Zeeman splitting, $\delta\Omega$, is
about four orders of magnitude smaller than the shift in orbital energy,
$\delta E$ computed for the same device in Kępa  _et al._ (2023). In contrast
to those results, we observe both increase and decrease of $\Omega$, which can
be understood as a motion of quantum dot to the right (along the gradient) and
to the left (against the gradient) respectively, see Fig. 1b for the cartoon
of the energy shift.

Figure 1: a) Modification of spin splitting $\delta\Omega(x,y,z_{0})$ due to
presence of a charge defect located close to semiconductor-oxide interface,
i.e. at $\mathbf{r}=(x,y,z_{0})$. b) Illustration of $\delta\Omega$ caused by
the shift of electron wavefunction $\Delta R(\mathbf{r})$ in presence of
magnetic field gradient $\Delta B_{\parallel}=0.2$mT/nm . c) Derivatives along
$x$, $y$, $z$ directions used later to compute the TLF-spin coupling
$\zeta_{n}=\nabla\delta\Omega\cdot\delta\mathbf{r}_{n}$.

As in Kępa  _et al._ , 2023, we employ now the $1/f$ noise model of many
independent TLFs corresponding to charges jumping between $\mathbf{r}_{n}$ and
$\mathbf{r}_{n}+\delta\mathbf{r}_{n}$, each being a source Random Telegraph
Noise characterized by switching rate $\gamma_{n}$ sampled from a log-normal
distribution Schriefl  _et al._ (2006); You, Clerk, and Koch (2021); Shnirman
_et al._ (2005). We consider switching rates
$f_{\text{min}}<\gamma_{n}<f_{\text{max}}$, with experimentally relevant
frequencies $f_{\text{min}}=10^{-6}$Hz and $f_{\text{max}}=10^{6}$Hz. We
define the coupling between the TLF and the single electron spin qubit as

$\zeta_{n}=\delta\Omega(\mathbf{r}_{n}+\delta\mathbf{r}_{n})-\delta\Omega(\mathbf{r}_{n})\approx\nabla\delta\Omega(\mathbf{r}_{n})\cdot\delta\mathbf{r}_{n}\,\,.$
(2)

We assume the contribution from multiple defects is additive, i.e.
$\delta\Omega(\\{\sum_{n}\mathbf{r}_{n}\\})=\sum_{n}\delta\Omega(\mathbf{r}_{n})$.
We have tested this assumption against the case of two charges located at
random positions, which produced relatively small error
$\delta\Omega(\mathbf{r}_{1},\mathbf{r_{2}})-\delta\Omega(\mathbf{r}_{1})-\delta\Omega(\mathbf{r}_{2})\ll
0.1\,$neV. This allow us to compute the Power Spectral Density (PSD) of the
noise in Zeeman splitting for various charge densities $\rho$, and individual
TLFs parameters: displacement size $\delta r_{n}$, and switching rates
$\gamma_{n}$, without further finite-element simulations. Since each TLF is
characterized by a Lorentzian spectrum, the total PSD reads:

$S_{\text{spin}}(f)=\sum_{n=1}^{N}\frac{2\zeta_{n}^{2}}{\gamma_{n}+(2\pi
f)^{2}/\gamma_{n}}\,\,.$ (3)

where the $n$-the TLF is characterized by coupling $\zeta_{n}$ and switching
rate $\gamma_{n}$. For each realisation, we first use uniform spatial
distribution to draw positions of $N$ charges at $z_{0}=102$nm plane, where
$N=A\rho$ is related to charge density $\rho$ and considered area $A$. Next,
in each realisation, and for each charge we draw the components of
displacement vector $\delta\mathbf{r}_{n}$ from the independent Gaussian
distribution of zero average, and the equal variances such that
$\langle|\mathbf{r}_{n}|\rangle=\langle\sqrt{\delta x^{2}+\delta y^{2}+\delta
z^{2}}\rangle\equiv\delta r$ is the displacement size.

Using interpolated data from Fig. 1a, we compute the gradients of spin
splitting shifts in Fig. 1c. In all directions we observe that depending on
the initial location, the motion of the defect can both increase and decrease
$\Omega$. By comparing the relative value of the gradient components, we see
the same displacement of a charge along z-direction results in noise larger by
about an order of magnitude in comparison to the in-plane displacements. This
is due to specific to motion along z, reduction of dipole moment, that is
caused by decrease of the distance between the metal (inside which a mirror
charge is placed) and the actual charge. It can be also seen from similarity
(with negative sign) between $\delta\Omega$ and $\partial_{z}\delta\Omega$. In
the planar direction, the $x$-derivative is about two times larger than y-,
which shows that the motion of the charge along the $x$ direction is more
effective at moving the electron’s wavefunction along this direction (which is
the direction of the gradient of the longitudinal field).

We now translate motion of multiple TLFs into spin splitting noise. We
concentrate on slow fluctuations of spin splitting, that directly affect
measured coherence time of spin qubit,
$T_{2}^{*}=\sqrt{2}/\sigma_{\text{spin}}$, that is parameterized by the noise
amplitude

$\sigma_{\text{spin}}^{2}=2\int_{f_{\text{min}}}^{f_{\text{max}}}S_{\text{spin}}(f)\text{d}f\equiv\sum_{n}\zeta_{n}^{2}.$
(4)

Since typically $T_{2}^{*}$ varies between different QDs in isotopically
purified Si/SiGe, as a reference we use both the longest-observed coherence
time of $T_{2}^{*}\\!\approx\\!20\mu$s from Yoneda  _et al._ (2018); Struck
_et al._ (2020) and the much shorter $T_{2}^{*}\\!=\\!1\,\mu$s from Yoneda
_et al._ (2022). Assuming $\Delta B_{\parallel}\\!=\\!0.2\,$mT/nm these values
can be related to $\sigma_{\text{spin}}=1\,$neV, (Ref. Yoneda  _et al._ ,
2022), $0.05\,$neV (Ref. Yoneda  _et al._ , 2018) and
$\sigma_{\text{spin}}=0.1\,$neV, (Ref. Struck  _et al._ , 2020), for which we
have increased noise amplitude to compensate for twice smaller gradient. In
Fig. 2a we show the statistics of the noise amplitude obtained from 1000
realisations of the isotropic model, for three different charge densities
$\rho$ (colors) as a function of displacement size $\delta$r (x-axis). We mark
the values of $\sigma_{\text{spin}}$ from references above using dashed lines.
For none of the selected pairs of $\rho$ and $\delta r$ it is possible to
reconstruct all of these values. Additionally,
$\sigma_{\text{spin}}\\!\leqslant\\!0.1\,$neV corresponding to two Yoneda  _et
al._ , 2018; Struck  _et al._ , 2020, has been reached only for extreme
parameters, i.e. the smallest $\delta r=0.1\,$nm, and $\rho=5\times
10^{9}\,$nm. For larger densities, not more than a single reference value has
been reached, independently of $\delta$r.

Figure 2: Amplitude of spin splitting noise as a function of typical
displacement size for three different charge densities in the isotropic model
(a) from Kępa  _et al._ (2023) and the planar model (b). Violins for given
$\delta r$ are slightly offset to increase the legibility of the figure. The
numbers on the right correspond to noise amplitude from experimental Ref.
Yoneda  _et al._ , 2018, 2022; Struck  _et al._ , 2020.

Figure 3: a) Fitted spin splitting noise amplitude
$\langle\sigma_{\text{spin}}^{(xy)}\rangle$ in the XY model, as a function of
charge density $\rho$ and displacement size $\delta$r. Using solid lines we
draw a cut through parameter space that corresponds to
$\sigma_{\text{orb}}=10\mu$eV of orbital energy fluctuations. Filled (hollow)
dots correspond to a parameters used in panel b) for planar (isotropic)
models. b) 10 realisations of Power Spectral Densities of the spin splitting
noise for four parameter pairs corresponding to colored dots in a). For each
dot, using the same color, we plot ideal 1/f spectra corresponding to
$\sigma_{\text{spin}}=0.05,0.1,1$neV using dashed lines.

For this reason we consider now an alternative model of charge motion
constrained only to the $z_{0}=102$nm plane. We show the result for such a
planar motion model in Fig. 2b. In comparison to the isotropic motion, we
observe a small increase of the data spread and, more importantly, a decrease
in the average noise amplitude by a factor of about 2. Those changes makes the
$\sigma_{\text{spin}}\\!\leqslant\\!0.1\,$neV much easier to achieve for a
range of parameters, i.e. $\rho\\!\leqslant\\!10^{10}\,$cm-2 and $\delta
r\\!\leqslant\\!0.5\,$nm. Additionally, for $\delta r\,\geqslant\,0.3$nm we
expect non-zero probability of reaching the largest
$\sigma_{\text{spin}}=1\,$neV, which puts this model in less tension with
experimental data.

In each model, both ensemble averages and standard deviations of
$\sigma_{\text{spin}}$ follows linear trend with respect to $\delta r$, with
their proportionality factor having polynomial dependence on $\rho$ (See Ref.
Kępa  _et al._ , 2023 for analogous analysis, but for orbital noise). For the
average $\sigma_{\text{spin}}$ we find effective expressions:

$\displaystyle\langle\sigma_{\text{spin}}^{\text{(xy)}}\rangle$
$\displaystyle\approx\bigg{[}0.91\big{(}\frac{\rho}{10^{10}\text{cm}^{-2}}\big{)}^{0.54}+0.030\bigg{]}\frac{\delta\text{r}}{\text{nm}}\,\text{neV}.$
$\displaystyle\langle\sigma_{\text{spin}}^{\text{(iso)}}\rangle$
$\displaystyle\approx\bigg{[}2.1\big{(}\frac{\rho}{10^{10}\text{cm}^{-2}}\big{)}^{0.53}+0.025\bigg{]}\frac{\delta\text{r}}{\text{nm}}\,\text{neV}.$
(5)

We use the fit from above in Fig. 3a, where we plot
$\langle\sigma_{\text{spin}}^{(xy)}\rangle$ against $\delta r$ and $\rho$.

We also perform a cross-check, and compute the amplitude of fluctuation of
ground orbital level $\delta E$ within the isotropic and planar model of
charge motion. For the isotropic model we use the final result of Kępa  _et
al._ , 2023, and additionally compute the planar version using $xy$-gradients
presented therein. In Fig. 3a we use the solid lines to plot the cut through
parameter space that matches $\langle\sigma_{\text{orb}}\rangle=10\mu$eV,
corresponding to $T_{2}^{*}\\!\approx\\!0.1$ns of the charge qubit,Shi  _et
al._ (2013) that has been obtained in the isotropic (black) and planar (blue)
models of charge motion. On each of them we chose two pairs of parameters
$(\rho,\delta r)$. We mark then in Fig. 3a using colored dots, where hollow
(filled) dots corresponds to an isotropic (planar) model. For each of those
points, in Fig. 3b, we plot the corresponding PSD of the spin splitting noise,
and compare it with a PSD of ideal 1/f-shape:

$S_{\text{spin}}(f)=\frac{\sigma_{\text{spin}}^{2}}{2\ln(f_{\text{max}}/f_{\text{min}})}\times(1\text{Hz}/f).$
(6)

calculated for reference values of $\sigma_{\text{spin}}=0.05,0.1,1\,$neV,
Yoneda  _et al._ (2022, 2018); Struck  _et al._ (2020).

In Fig. 3b we show exemplary 10 realisations of random: defect positions
$\\{\mathbf{r}_{n}\\}$, their (isotropic/in-plane) displacement
$\\{\delta\mathbf{r}_{n}\\}$ and switching rates
$f_{\text{min}}<\\{\gamma_{n}\\}<f_{\text{max}}$. We report qualitative
agreement between the obtained shapes and experimentally measured
spectra,Yoneda  _et al._ (2018); Struck  _et al._ (2020); Yoneda  _et al._
(2022) which can be seen from visible deviations from 1/f trend due to
distinct single TLF features (see also the discussion in Güngördü and Kestner
(2019)). Those features are more prominent in the isotropic model, which can
be seen be the direct comparison of the PSDs obtained with $\rho=10^{10}$cm-2
(two middle panels). On the other hand, such a single TLF features are still
present after an increase of $\rho$ (last panel). Finally, we observe that
quantitatively in isotropic model, parameters $(\rho,\delta r)$ corresponding
to $\sigma_{\text{orb}}=10\,\mu$eV, gives smaller spin splitting than in the
planar model. Also in the latter case, the resulting PSDs are a closer match
to the dashed lines, with the prominent example of $\rho=10^{10}$cm-2 and
$\delta r=0.5\,$nm, the realisations of which spreads between all reference
spectra (dashed lines).

We finally study the correlation between the orbital and spin splitting
fluctuations. In analogy to Eq. (3) and Eq. (4) we define the amplitude of
low-frequency cross-correlated noise as

$\sigma_{\Omega,E}^{2}=\sum_{n}\zeta_{n}\eta_{n},$ (7)

where $\zeta_{n}$ ($\eta_{n}$) denotes coupling between the n-th TLF and the
spin (ground orbital energy) of the electron. The contribution of each TLF to
the correlated noise is then given by the product $\zeta_{n}\eta_{n}$. In Fig.
4a we show such a product for a single TLF located at
$\mathbf{r}_{n}=(x,y,z_{0})$ with displacement size $\delta r=0.3$nm along
x-axis (a), y-axis (b), and z-axis (c). We see that for the motion along these
three primary directions, both positive and negative correlation of the noise
processes are equally probable, which can be seen from the comparison between
the sizes of red and blue regions. However, a non-negligible correlation
requires the charge to be located in the central region around the QD.

Figure 4: Product of the couplings between a single defect at location
$(x,y,z_{0}=102$nm$)$ and spin ($\zeta_{n}$) and orbital level ($\eta_{n}$)
for defect displacement size of $\delta r=0.3$nm along x (a), y (b), z (c)
axes. d) Correlation coefficient $C_{\Omega,E}$ between the fluctuations of
ground orbital level $E$ and spin splitting $\Omega$ as a function of defect
density $\rho$ and for two different displacement sizes $\delta r=0.1,0.5$nm.
For low-densities two quantities are correlated, and the correlation decreases
with the increase of $\rho$.

We now analyze relative correlation between orbital and spin splitting noise,
defined via the normalized covariance:

$C_{\Omega,E}=\frac{\sigma^{2}_{\Omega,E}}{\sigma_{\Omega}\sigma_{E}}.$ (8)

where $C_{\Omega,E}=\pm 1$ means perfectly correlated and anti-correlated
noises, respectively, and $C_{\Omega,E}=0$ means that the two fluctuations are
uncorrelated. In Fig. 4b we plot $C_{\Omega,E}$ as a function of charge
density $\rho$ and for two displacement sizes, $\delta r\\!=\\!0.1$, $0.5$nm.
From their comparison, we conclude that the correlation very weakly depends on
$\delta$r. We also confirm that positively- and negatively- correlated noises
in $\Omega$ and $E$ are equally probable (note the symmetry with respect to
$C_{\Omega,E}=0$). The noise is strongly correlated if the number of TLFs is
low, with the extreme case of $N\approx 4$ for $\rho=10^{9}\,$cm-2, and
becomes weakly correlated if the number of TLFs is large, with $N=400$ at
$\rho=10^{11}\,$cm-2 as an example. In the former case this is because the
noise is typically dominated by a few or even a single TLF located in the
central region of Fig 4a, for which the correlations are large. As $\rho$
increases, both red and blue regions of Fig. 4a become equally populated, and
also contributions from the regions in which the cross-correlations are weak
increase. We report that correlations obtained in the isotropic and planar
model of charge motion are similar. We also highlight that at the expected
densities of $\rho=10^{10}$cm-2, a relatively wide distribution of
$C_{\Omega,E}$ is predicted, which means that correlation between
$\delta\Omega$ and $\delta E$ is expected to vary visibly between similar
devices, or spatially distant regions of the same device.

In summary, building upon the model of the device and $1/f$ charge noise
previously introduced in Kępa  _et al._ , 2023 we have simulated the influence
of charge noise on the spin splitting of an electron in a single quantum dot
in a Si/SiGe device, in presence of magnetic field gradient of magnitude used
in spin qubit manipulation experiments. While in previous modelingKępa  _et
al._ (2023) isotropic motion of TLF charges was assumed, here we have
introduced a planar model, in which defects were allowed to move only in the
plane parallel to the interface. We have found that the planar motion of
defects allowed for better agreement with experimentally measured spin
splitting noise. We have predicted values of density of TLFs, $\rho\approx
10^{10}$cm-2 and the typical magnitude of motion of charges associated with
them, $\delta\\!r\\!\approx 0.5$ nm, that give results that are consistent
with measured orbital energy fluctuations, as well as with other theoretical
works.Rojas-Arias  _et al._ (2023); Shalak, Delerue, and Niquet (2022);
Shehata  _et al._ (2022) For this density, noise in spin splitting and in
orbital energy should exhibit visible correlations.

###### Acknowledgements.

This work was supported by funds from PRELUDIUM grant of the Polish National
Science Centre (NCN), Grant No. 2021/41/N/ST3/02758.

## AUTHOR DECLARATIONS

### Conflict of interest

The authors have no conflicts to disclose.

## Data Availability Statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * Chan  _et al._ (2018) K. W. Chan, W. Huang, C. H. Yang, J. C. Hwang, B. Hensen, T. Tanttu, F. E. Hudson, K. M. Itoh, A. Laucht, A. Morello, and A. S. Dzurak, “Assessment of a Silicon Quantum Dot Spin Qubit Environment via Noise Spectroscopy,” Physical Review Applied 10, 1 (2018).
  * Yoneda  _et al._ (2018) J. Yoneda, K. Takeda, T. Otsuka, T. Nakajima, M. R. Delbecq, G. Allison, T. Honda, T. Kodera, S. Oda, Y. Hoshi, N. Usami, K. M. Itoh, and S. Tarucha, “A quantum-dot spin qubit with coherence limited by charge noise and fidelity higher than 99.9%,” Nature Nanotechnology 13, 102 (2018).
  * Tanttu  _et al._ (2019) T. Tanttu, B. Hensen, K. W. Chan, C. H. Yang, W. W. Huang, M. Fogarty, F. Hudson, K. Itoh, D. Culcer, A. Laucht, A. Morello, and A. Dzurak, “Controlling spin-orbit interactions in silicon quantum dots using magnetic field direction,” Phys. Rev. X 9, 021028 (2019).
  * Zajac  _et al._ (2018) D. M. Zajac, A. J. Sigillito, M. Russ, F. Borjans, J. M. Taylor, G. Burkard, and J. R. Petta, “Resonantly driven CNOT gate for electron spins,” Science 359, 439–442 (2018).
  * Mills  _et al._ (2022) A. R. Mills, C. R. Guinn, M. J. Gullans, A. J. Sigillito, M. M. Feldman, E. Nielsen, and J. R. Petta1, “Two-qubit silicon quantum processor with operation fidelity exceeding 99%,” Sci. Adv. 8, eabn5130 (2022).
  * Philips  _et al._ (2022) S. G. J. Philips, M. T. Madzik, S. V. Amitonov, S. L. de Snoo, M. Russ, N. Kalhor, C. Volk, W. I. L. Lawrie, D. Brousse, L. Tryputen, B. P. Wuetz, A. Sammak, M. Veldhorst, G. Scappucci, and L. M. K. Vandersypen, “Universal control of a six-qubit quantum processor in silicon,” Nature 609, 919–924 (2022).
  * Tokura  _et al._ (2006) Y. Tokura, W. G. van der Wiel, T. Obata, and S. Tarucha, “Coherent single electron spin control in a slanting zeeman field,” Phys. Rev. Lett. 96, 047202 (2006).
  * Neumann and Schreiber (2015) R. Neumann and L. R. Schreiber, “Coherent single electron spin control in a slanting zeeman field,” J. Appl. Phys. 117, 193903 (2015).
  * Aldeghi, Allenspach, and Salis (2023) M. Aldeghi, R. Allenspach, and G. Salis, “Modular nanomagnet design for spin qubits confined in a linear chain,” Applied Physics Letters 122 (2023), 10.1063/5.0139670.
  * Takeda  _et al._ (2016) K. Takeda, J. Kamioka, T. Otsuka, J. Yoneda, T. Nakajima, M. R. Delbecq, S. Amaha, G. Allison, T. Kodera, S. Oda, and Seigo Tarucha, “A fault-tolerant addressable spin qubit in a natural silicon quantum dot,” Science Advances 2, e1600694 (2016).
  * Zhao  _et al._ (2019) R. Zhao, T. Tanttu, K. Y. Tan, B. Hensen, K. W. Chan, J. C. C. Hwang, R. C. C. Leon, C. H. Yang, W. Gilbert, F. E. Hudson, K. M. Itoh, A. A. Kiselev, T. D. Ladd, A. Morello, A. Laucht, and A. S. Dzurak, “Single-spin qubits in isotopically enriched silicon at low magnetic field,” Nature Communications 10, 5500 (2019).
  * Corna  _et al._ (2018) A. Corna, L. Bourdet, R. Maurand, A. Crippa, D. Kotekar-Patil, H. Bohuslavskyi, R. Laviéville, L. Hutin, S. Barraud, X. Jehl, M. Vinet, S. De Franceschi, Y.-M. Niquet, and M. Sanquer, “Electrically driven electron spin resonance mediated by spin–valley–orbit coupling in a silicon quantum dot,” npj Quantum Information 4, 6 (2018).
  * Struck  _et al._ (2020) T. Struck, A. Hollmann, F. Schauer, O. Fedorets, A. Schmidbauer, K. Sawano, H. Riemann, N. V. Abrosimov, Ł. Cywiński, D. Bougeard, and L. R. Schreiber, “Low-frequency spin qubit detuning noise in highly purified 28Si/SiGe,” npj Quantum Inf. 6, 40 (2020).
  * Li  _et al._ (2018) R. Li, L. Petit, D. P. Franke, J. P. Dehollain, J. Helsen, M. Steudtner, N. K. Thomas, Z. R. Yoscovits, K. J. Singh, S. Wehner, L. M. K. Vandersypen, J. S. Clarke, and Menno Veldhorst, “A crossbar network for silicon quantum dot qubits,” Science Advances 4, eaar3960 (2018).
  * Heinz and Burkard (2021) I. Heinz and G. Burkard, “Crosstalk analysis for single-qubit and two-qubit gates in spin qubit arrays,” Phys. Rev. B 104, 045420 (2021).
  * Connors  _et al._ (2019) E. J. Connors, J. Nelson, H. Qiao, L. F. Edge, and J. M. Nichol, “Low-frequency charge noise in Si/SiGe quantum dots,” Phys. Rev. B 100, 165305 (2019).
  * Connors  _et al._ (2022) E. J. Connors, J. Nelson, L. F. Edge, and J. M. Nichol, “Charge-noise spectroscopy of si/sige quantum dots via dynamically-decoupled exchange oscillations,” Nat. Comm. 13, 940 (2022).
  * Yoneda  _et al._ (2022) J. Yoneda, J. S. Rojas-Arias, P. Stano, K. Takeda, A. Noiri, T. Nakajima, D. Loss, and S. Tarucha, “Noise-correlation spectrum for a pair of spin qubits in silicon,” arXiv:2208.14150 (2022).
  * Rojas-Arias  _et al._ (2023) J. S. Rojas-Arias, A. Noiri, P. Stano, T. Nakajima, J. Yoneda, K. Takeda, T. Kobayashi, A. Sammak, G. Scappucci, D. Loss, and S. Tarucha, “Spatial noise correlations beyond nearest-neighbor in 28si/sige spin qubits,” arXiv:2302.11717 (2023).
  * Shehata  _et al._ (2022) M. M. E. K. Shehata, G. Simion, R. Li, F. A. Mohiyaddin, D. Wan, M. Mongillo, B. Govoreanu, I. Radu, K. De Greve, and P. Van Dorpe, “Modelling semiconductor spin qubits and their charge noise environment for quantum gate fidelity estimation,” arXiv:2210.04539 (2022).
  * Kępa  _et al._ (2023) M. Kępa, N. Focke, Łukasz Cywiński, and J. A. Krzywda, “Simulation of 1/f charge noise affecting a quantum dot in a si/sige structure,” (2023), arXiv:2303.13968 .
  * Culcer and Zimmerman (2013) D. Culcer and N. M. Zimmerman, “Dephasing of Si singlet-triplet qubits due to charge and spin defects,” Applied Physics Letters 102, 232108 (2013).
  * Wuetz  _et al._ (2023) B. P. Wuetz, D. D. Esposti, A.-M. J. Zwerver, S. V. Amitonov, M. Botifoll, J. Arbiol, A. Sammak, L. M. K. Vandersypen, M. Russ, and G. Scappucci, “Reducing charge noise in quantum dots by using thin silicon quantum wells,” Nature Communications 14 (2023).
  * Schriefl  _et al._ (2006) J. Schriefl, Y. Makhlin, A. Shnirman, and G. Schön, “Decoherence from ensembles of two-level fluctuators,” New Journal of Physics 8, 1 (2006).
  * You, Clerk, and Koch (2021) X. You, A. A. Clerk, and J. Koch, “Positive- and negative-frequency noise from an ensemble of two-level fluctuators,” Phys. Rev. Res. 3, 013045 (2021).
  * Shnirman  _et al._ (2005) A. Shnirman, G. Schön, I. Martin, and Y. Makhlin, “Low- and high-frequency noise from coherent two-level systems,” Phys. Rev. Lett. 94, 127002 (2005).
  * Shi  _et al._ (2013) Z. Shi, C. B. Simmons, D. R. Ward, J. R. Prance, R. T. Mohr, T. S. Koh, J. K. Gamble, X. Wu, D. E. Savage, M. G. Lagally, M. Friesen, S. N. Coppersmith, and M. A. Eriksson, “Coherent quantum oscillations and echo measurements of a si charge qubit,” Phys. Rev. B 88, 075416 (2013).
  * Güngördü and Kestner (2019) U. Güngördü and J. P. Kestner, “Indications of a soft cutoff frequency in the charge noise of a si/sige quantum dot spin qubit,” Phys. Rev. B 99, 081301 (2019).
  * Shalak, Delerue, and Niquet (2022) B. Shalak, C. Delerue, and Y.-M. Niquet, “Modelling of spin decoherence in a Si hole qubit perturbed by a single charge fluctuator,” arXiv:2210.10476 (2022).