11institutetext: Department of Electrical Engineering, City University of Hong
Kong, Kowloon, Hong Kong SAR, China
22institutetext: Huawei Noah’s Ark Lab, Sha Tin, Hong Kong SAR, China

# SBoRA: Low-Rank Adaptation with Regional Weight Updates

Lai-Man Po 11 Yuyang Liu 11 Haoxuan Wu 11 Tianqi Zhang 11 Wing-Yin Yu 22 Zeyu
Jiang 11 Kun Li 11

###### Abstract

This paper introduces Standard Basis LoRA (SBoRA), a novel parameter-efficient
fine-tuning approach for Large Language Models that builds upon the pioneering
works of Low-Rank Adaptation (LoRA) and Orthogonal Adaptation. SBoRA further
reduces the computational and memory requirements of LoRA while enhancing
learning performance. By leveraging orthogonal standard basis vectors to
initialize one of the low-rank matrices, either $\mathbf{A}$ or $\mathbf{B}$,
SBoRA enables regional weight updates and memory-efficient fine-tuning. This
approach gives rise to two variants, SBoRA-FA and SBoRA-FB, where only one of
the matrices is updated, resulting in a sparse update matrix
$\mathrm{\Delta}\mathbf{W}$ with a majority of zero rows or columns.
Consequently, the majority of the fine-tuned model’s weights
$(\mathbf{W}_{0}+\mathrm{\Delta}\mathbf{W})$ remain unchanged from the pre-
trained weights. This characteristic of SBoRA, wherein regional weight updates
occur, is reminiscent of the modular organization of the human brain, which
efficiently adapts to new tasks. Our empirical results demonstrate the
superiority of SBoRA-FA over LoRA in various fine-tuning tasks, including
commonsense reasoning and arithmetic reasoning. Furthermore, we evaluate the
effectiveness of QSBoRA on quantized LLaMA models of varying scales,
highlighting its potential for efficient adaptation to new tasks. Code is
available at https://github.com/cityuhkai/SBoRA

###### Keywords:

Large Language Models Parameter-Efficient Fine-Tuning LoRA

##  1 Introduction

Large language models (LLMs) and diffusion models have become essential
components of natural language processing [1, 25] and multimodal AI
applications [14, 16]. Full fine-tuning (FFT) has been shown to significantly
improve their performance in various downstream tasks [28] or introduce new
concepts. However, fine-tuning pre-trained models, especially LLMs, by
updating all parameters is computationally costly. As depicted in Fig. 1(a),
FFT involves directly updating the high-dimensional pre-trained weight matrix
$\mathbf{W}_{0}$.

To address this issue, parameter-efficient fine-tuning (PEFT) methods [9] have
gained momentum, focusing on updating only a small fraction of parameters,
like adapter [7, 9, 19], prompt tuning [13, 21, 26]. Among these methods, Low-
Rank Adaptation (LoRA) has emerged as a groundbreaking technique for LLMs.
LoRA [10] introduces a parallel low-rank adapter to the weights of linear
layers as shown in Fig. 1(b), reducing memory overhead and computational costs
during fine-tuning. However, there are still limitations to LoRA, including
activation memory consumption and a performance gap compared to FFT. These
limitations have led to the development of many LoRA variants [2, 3, 4, 6, 17,
22, 23, 27, 29, 31, 32].

Figure 1: Four fine-tuning strategies: (a) Full Fine-Tuning (FFT), (b) LoRA,
(c) SBoRA-FA, and (d) SBoRA-FB.

In this work, we introduce Standard Basis LoRA (SBoRA), a novel approach that
further reduces the computational and memory requirements while enhancing the
learning capacity. SBoRA achieves this by selectively updating specific rows
or columns of the pre-trained weight matrix $\mathbf{W}_{0}$, preserving most
of the base model’s weights. The resulting update matrix
$\mathrm{\Delta}\mathbf{W}=\mathbf{BA}$ mainly consists of zero rows or
columns, indicating that most of the fine-tuned model’s weights
$(\mathbf{W}^{\prime}=\mathbf{W}_{0}+\mathrm{\Delta}\mathbf{W})$ remain
identical to the pre-trained weights. This localized learning process
parallels the modular organization of the human brain, where specific
cognitive functions are localized in distinct regions [5]. For example, the
hippocampus is responsible for episodic memory [30] while the prefrontal
cortex is crucial for working memory and decision-making [12]. By mirroring
the modular functionality of the human brain, SBoRA is expected to improve
knowledge retention and facilitate efficient adaptation.

Let’s assume that the pre-trained weight matrix $\mathbf{W}_{0}$, the
projection-down matrix $\mathbf{A}$, and the projection-up matrix $\mathbf{B}$
are given. SBoRA adopts a unique approach, utilizing orthogonal standard basis
vectors (one-hot vectors) to construct its projection matrices. These
matrices, designated as $\mathbf{A}_{sb}$ for SBoRA-FA and $\mathbf{B}_{sb}$
for SBoRA-FB, are illustrated in Figures 1(c) and 1(d), respectively. This
efficiency stems from the elimination of the need to store coefficients for
one of the projection matrices, resulting in approximately 50% reduction in
memory usage. In SBoRA-FA, besides the frozen weight matrix $\mathbf{W}_{0}$,
the $\mathbf{A}_{sb}$ is fixed and only updates $\mathbf{B}$ during fine-
tuning. The weight changes ($\mathrm{\Delta}\mathbf{W}=\mathbf{BA}_{sb}$)
reside in a low-rank subspace spanned by $\mathbf{A}_{sb}$, and the
projection-down transformation can be efficiently achieved by selecting a
subset of input vector samples. SBoRA-FB operates similarly, using a pre-
defined standard basis projection-up matrix $\mathbf{B}_{sb}$ and updating
only matrix $\mathbf{A}$. Both SBoRA-FA/FB align with LoRA, reducing
computational overhead during fine-tuning and introducing no extra inference
latency.

We extensively validated SBoRA across diverse tasks, including evaluations on
floating-point 16-bit precision models and quantized models in the QLoRA
framework [4]. For 16-bit models, we focused on commonsense reasoning and
arithmetic reasoning tasks. Quantized models were evaluated using the MMLU
benchmarks [8]. Experimental results consistently demonstrate SBoRA’s
superiority over other PEFT baselines. For instance, with similar number of
trainable parameters (SBoRA: rank 64, LoRA/DoRA: rank 32), SBoRA-FA
demonstrates significant improvements in commonsense reasoning (+2.9%/1.7% on
LLaMA-7B/LLaMA3-8B) compared to LoRA, and arithmetic reasoning (+2.8%/+2.0% on
LLaMA-7B/LLaMA3-8B) compared to DoRA. Furthermore, with approximately half the
trainable parameters (QSBoRA/QLoRA/QDoRA: rank 64), QSBoRA-FA exhitbs notable
enhancements on MMLU benchmarks, such as +4.3%/+6.9% on quantized
LLaMA-13B/LLaMA3-8B compared to QLoRA.

The summary of our contributions is as follows:

  * •

Introduces regional weight updates in PEFT that mirroring the human brain’s
ability to store specific skills in distinct, specialized regions.

  * •

Enables memory-efficient fine-tuning by storing either the standard basis low-
rank matrix $\mathbf{A}_{\mathit{sb}}$ or $\mathbf{B}_{\mathit{sb}}$ as a
1-dimensional vector, reducing storage needs, and requiring only half of the
gradients, thereby saving memory.

  * •

Introduces sample-based multiplication to realize the efficient SBoRA-FA
projection-down and SBoRA-FB projection-up transformations that reduce
computational complexity by eliminating one matrix multiplication operation.

##  2 Related Work

In deep learning, fine-tuning is a fundamental concept that enables leveraging
pre-trained models as a starting point for new tasks [7, 9]. This approach is
based on the idea that a model trained on one task can be adapted to perform
well on another related task.

###  2.1 Full Fine-Tuning (FFT)

FFT involves updating all of the parameters of a pre-trained model on a new
dataset for a specific task as shown in Fig. 1(a). This process adjusts the
model’s weights to fit the new task, leading to improved performance. However,
the FFT approach has several limitations. Notably, it necessitates storing and
deploying numerous copies of large models tailored to different tasks, which
can be prohibitively storage-intensive. Furthermore, FFT is computationally
costly, particularly for LLMs, and can be overly time-consuming.

###  2.2 Parameter-Efficient Fine-Tuning (PEFT)

To address the limitations of FFT, researchers have proposed PEFT techniques
[9]. These methods adapt only a small subset of a model’s parameters to a new
task, aiming to achieve comparable performance to FFT while reducing
computational and memory requirements. Early approaches include the use of
adapter layers, which add small, trainable modules to each Transformer layer.
Another approach is prefix tuning [15], which optimizes a few continuous
"prefix" vectors prepended to the input sequence. Both of these approaches
significantly reduce the memory and computation needed to fine-tune large
models. However, they also have their limitations, such as extra inference
latency in the case of adapter layers, and difficulty in optimization for
prefix tuning.

###  2.3 Low-Rank Adaptation (LoRA)

LoRA [10], a prominent PEFT method, has garnered significant attention due to
its effectiveness and efficiency. As depicted in Fig. 1(b), the fundamental
concept underlying LoRA involves approximating full-rank updates with low-rank
updates within the FFT domain. Specifically, given a pre-trained parameter
matrix $\mathbf{W}_{0}\in\mathbb{R}^{d\times k}$, LoRA utilizes two low-rank
matrices, $\mathbf{A}\in\mathbb{R}^{r\times k}$ and
$\mathbf{B}\in\mathbb{R}^{d\times r}$, to compute the weight update
$\mathrm{\Delta}\mathbf{W}\in\mathbb{R}^{d\times k}$. The output vector
$\mathbf{h}\in\mathbb{R}^{d\times 1}$ of a LoRA linear layer with input
$\mathbf{x}\in\mathbb{R}^{k\times 1}$ can be expressed as

$\mathbf{h}=\mathbf{W}_{0}\mathbf{x}+\mathrm{\Delta}\mathbf{Wx}=\mathbf{W}_{0}\mathbf{x}+\mathbf{BAx}$
(1)

LoRA ensures $\mathrm{\Delta}\mathbf{W}$ is initialized to zero by
initializing $\mathbf{A}$ with a uniform distribution and $\mathbf{B}$ with
zero. The low-rank decomposition of $\mathrm{\Delta}\mathbf{W}$ into
$\mathbf{BA}$ indicates that the rank of $\mathrm{\Delta}\mathbf{W}$ is at
most $r$, significantly lower than full-rank updating in FFT, with $r$ being
much smaller than the minimum of $d$ and $k$.

LoRA’s low-rank updating approach has demonstrated comparable performance to
full-rank updating in tasks like text classification and instruction tuning.
It reduces memory usage and computational requirements by decreasing the
number of trainable parameters needed. LoRA injects low-rank matrices directly
into existing layers without adding extra layers, resulting in no additional
inference latency. Switching out different LoRA modules is easy, simplifying
deployment in multi-task scenarios. Empirical studies show that LoRA can match
or exceed FFT performance using only a fraction of the total parameters,
reducing the need for high-performance hardware.

###  2.4 Variants of LoRA

LoRA has sparked significant research in PEFT methods, including its own
variants such as QLoRA [4], QA-LoRA [29], LongLoRA [3], S-LoRA [23], LQ-LoRA
[6], MultiLoRA [27], LoRA-FA [32], Tied-LoRA [22], GLoRA [2], and DoRA [17].
QLoRA is an industry-standard technique for PEFT of LLMs, it employs 4-bit
quantization on pretrained weights and trains LoRA modules on this quantized
representation. Techniques like 4-Bit NormalFloat (NF4) Format, Double
Quantization, and Paged Optimizers further minimize memory usage. QA-LoRA
reduces the computational burden with group-wise quantization. LongLoRA
enables fine-tuning for longer context lengths using sparse local attention.
S-LoRA presents a scalable strategy for deploying multiple LoRA modules
efficiently. LQ-LoRA refines the quantization scheme for improved performance.
MultiLoRA handles complex multi-task learning, LoRA-FA reduces memory
overhead, and Tied-LoRA leverages weight tying. GLoRA adapts both weights and
activations, and DoRA decomposes weights for enhanced learning capacity.
Recently, Orthogonal Adaptation enables efficient merging of customized models
without sacrificing fidelity or incurring additional computational costs.

##  3 Standard Basis Low-Rank Adaptation (SBoRA)

SBoRA, inspired by Orthogonal Adaptation [20], utilizes a predefined
orthogonal basis $\mathbf{O}_{A}$ to generate orthogonal subspaces. These
subspaces, represented as low-rank projection-down matrices
$\mathbf{A}_{\mathit{i}}$, enable independent LoRA fine-tuning for multi-
concept customization in diffusion models. The linear layer combining $c$
custom concepts, given a pre-trained weight $\mathbf{W}_{0}$, can be expressed
as

$Linear(\mathbf{x})=(\mathbf{W}_{0}+\sum_{i=1}^{c}\lambda_{i}\cdot\mathbf{B}_{i}\mathbf{A}_{i})\mathbf{x}$
(2)

where $i$ denotes the index of the $i$-th LoRA, and $\lambda_{i}$ are scalar
factors determined through empirical tuning. The input is a column vector
$\mathbf{x}\in\mathbb{R}^{k\times 1}$. $\mathbf{A}_{i}\in\mathbb{R}^{r\times
k}$ matrices are constructed by selecting $r$ non-overlapping orthogonal basis
vectors from the shared orthogonal basis
$\mathbf{O}_{\mathit{A}}\in\mathbb{R}^{k\times k}$. $\mathbf{O}_{\mathit{A}}$
consists of $k$ orthogonal basis vector $\mathbf{o}_{p}\in\mathbb{R}^{1\times
k}$, where $p=1,2,\ldots,k$. It can be represented as

$\mathbf{O}_{\mathit{A}}=\left[\begin{array}[]{c}\mathbf{o}_{1}\\\
\mathbf{o}_{2}\\\ \vdots\\\ \mathbf{o}_{k}\end{array}\right]\mathrm{\ with\
}\mathbf{o}_{p}\mathbf{o}_{q}^{T}=0\mathrm{\ if\ }p\neq q$ (3)

To maintain orthogonality between projection-down matrices, these basis
vectors must be non-overlapping from $\mathbf{O}_{A}$, ensuring
$\mathbf{A}_{p}\mathbf{A}_{q}^{T}=0$ if $p\neq q$. And to minimize crosstalk,
$\mathbf{B}_{j}\mathbf{A}_{j}\mathbf{x}=\mathbf{0}$ if $j\neq i$ when
$\mathbf{x}$ belongs to subspace $\mathbf{A}_{i}$. During fine-tuning, the
orthogonal matrices $\mathbf{A}_{i}$ are frozen while updating the projection-
up matrices $\mathbf{B}_{i}$ to learn different concepts.

Figure 2: The diagram illustrates the regional weight update process of SBoRA,
showcasing distinct $\mathbf{W}_{0}+\mathrm{\Delta}\mathbf{W}$ computing
procedures of SBoRA-FA(upper) and SBoRA-FB (lower). The diagram employs
different colors to represent frozen, trainable, and zero parameters.

###  3.1 Orthogonal Standard Basis

The orthogonal adaptation approach offers not only effective multi-concept
merging but also memory efficiency. During LoRA fine-tuning, the
$\mathbf{A}_{i}$ matrices are frozen, which significantly reduces memory
requirements for gradient and intermediate activation storage. This is similar
to the LoRA-FA [32], where projection-down matrix $\mathbf{A}$ is randomly
initialized and then frozen during training. However, we can further improve
this orthogonal adaptation idea by utilizing a standard basis with one-hot
vectors as basis vectors. In this case, the orthogonal basis $\mathbf{O}_{A}$
becomes a $k\times k$ identity matrix $\mathbf{I}$. Specifically,
$\mathbf{O}_{A}$ can be represented as:

$\mathbf{O}_{A}=\mathbf{I}=\left[\begin{array}[]{c}\mathbf{e}_{1}\\\
\mathbf{e}_{2}\\\ \vdots\\\
\mathbf{e}_{k}\end{array}\right]=\left[\begin{array}[]{cccc}1&0&\cdots&0\\\
0&1&\cdots&0\\\ \vdots&\vdots&\ddots&\vdots\\\ 0&0&0&1\end{array}\right]$ (4)

where $\mathbf{e}_{p}\mathbf{e}_{q}^{T}=0$ if $p\neq q$. The standard basis
vector $\mathbf{e}_{k}\in\mathbb{R}^{1\times k}$ are one-hot row vectors, each
having a single non-zero entry of 1 at index $p$. For instance,
$\mathbf{e}_{p}$ can be represented as

$\displaystyle\mathbf{e}_{p}=[0\ \cdots\ 0$ $\displaystyle\ 1\ 0\ \cdots\ 0]$
(5) $\displaystyle\uparrow$ $\displaystyle\ p$

where the non-zero component with value one at the index of $p$. In SBoRA-FA,
we don’t need to randomly pre-generate an orthogonal-basis matrix
$\mathbf{O}_{A}$. Instead, we randomly select $r$ non-overlapping indices
between 1 and $k$ to construct a standard subspace basis matrix
$\mathbf{A}_{sb}$ with $r$ standard basis row vectors of $\mathbf{e}_{p}$.
This approach eliminates the need to store weights for the projection-down
matrix, as only the indices of the standard basis vectors are required to
represent $\mathbf{A}_{sb}$. The basic structure of SBoRA-FA is depicted in
Fig. 1(c), where both $\mathbf{W}_{0}$ and $\mathbf{A}_{sb}$ are frozen, and
only the matrix $\mathbf{B}$ is updated with initial values of zero that
ensure that the pre-trained models do not alter the model prediction before
fine-tuning. Similarity, SBoRA-FB freezes the projection-up as standard-basis
low-rank matrix $\mathbf{B}_{sb}$ and only update the projection-down matrix
$\mathbf{A}$ during the fine-tuning, in which orthogonal basis matrix
$\mathbf{O}_{B}$ is a $d\times d$ identity matrix $\mathbf{I}$, the standard
basis vectors are one-hot column vectors instead of row vectors.

$\mathbf{O}_{B}=\mathbf{I}=\left[\begin{array}[]{cccc}\mathbf{e}_{1}&\mathbf{e}_{2}&\cdots&\mathbf{e}_{d}\end{array}\right]=\left[\begin{array}[]{cccc}1&0&\cdots&0\\\
0&1&\cdots&0\\\ \vdots&\vdots&\ddots&\vdots\\\ 0&0&0&1\end{array}\right]$ (6)

where $\mathbf{e}_{p}^{T}\mathbf{e}_{q}=0$ if $p\neq q$. The standard basis
vector $\mathbf{e}_{p}\in\mathbb{R}^{d\times 1}$ are one-hot column vectors,
each having a single non-zero entry of 1 at index $p$. For instance,
$\mathbf{e}_{p}$ can be represented as

$\displaystyle\mathbf{e}_{p}=[0\ \cdots\ 0$ $\displaystyle\ 1\ 0\ \cdots\
0]^{T}$ (7) $\displaystyle\uparrow$ $\displaystyle\ p$

The basic structure of SBoRA-FB is depicted in Fig. 1(d), where both
$\mathbf{W}_{0}$ and $\mathbf{B}_{sb}$ are frozen, and only the matrix
$\mathbf{A}$ is updated with initial values of zero that ensure that the pre-
trained models do not alter the model prediction before fine-tuning.

###  3.2 Regional Weight Update

An interesting property of the proposed SBoRA is that the merged weight matrix
$\mathbf{W}^{\prime}$ of the fine-tuned model is only regionally updated, with
most of the weights remaining unchanged from the original pre-trained weight
$\mathbf{W}_{0}$ as depicted in Fig. 2. Specifically, the fine-tuned weight
$\mathbf{W}^{\prime}$ of SBoRA-FA can be represented as:

$\mathbf{W}^{\prime}=\mathbf{W}_{0}+\mathrm{\Delta}\mathbf{W}=\mathbf{W}_{0}+\mathbf{BA}_{sb}$
(8)

In which, the update matrix $\mathrm{\Delta}\mathbf{W}=\mathbf{BA}_{sb}$ is
very sparse, with most of the columns having zero weights due to the one-hot
nature of the standard basis subspace matrix $\mathbf{A}_{sb}$ as shown in the
upper part of Fig. 2. For example, when $r=2$ and $k=4$ with
$\mathbf{A}_{sb}=[\mathbf{e}_{1}\ \mathbf{e}_{4}]^{T}$, the
$\mathrm{\Delta}\mathbf{W}$ will only have two non-zero column as

$\mathrm{\Delta}\mathbf{W}=\mathbf{BA}_{sb}=\left[\begin{array}[]{cc}b_{11}&b_{12}\\\
b_{21}&b_{22}\\\ b_{31}&b_{32}\\\
b_{41}&b_{42}\end{array}\right]\left[\begin{array}[]{cccc}1&0&0&0\\\
0&0&0&1\end{array}\right]=\left[\begin{array}[]{cccc}b_{11}&0&0&b_{12}\\\
b_{21}&0&0&b_{22}\\\ b_{31}&0&0&b_{32}\\\ b_{41}&0&0&b_{42}\end{array}\right]$
(9)

The fine-tuned weight $\mathbf{W}^{\prime}$ is given by

$\mathbf{W}^{\prime}=\mathbf{W}_{0}+\mathrm{\Delta}\mathbf{W}=\left[\begin{array}[]{cccc}w_{11}+b_{11}&w_{12}&w_{13}&w_{14}+b_{12}\\\
w_{21}+b_{21}&w_{22}&w_{23}&w_{24}+b_{22}\\\
w_{31}+b_{31}&w_{32}&w_{33}&w_{34}+b_{32}\\\
w_{41}+b_{41}&w_{42}&w_{43}&w_{44}+b_{42}\end{array}\right]$ (10)

For SBoRA-FB, when $r=2$ and $d=4$ with $\mathbf{B}_{sb}=[\mathbf{e}_{1}\
\mathbf{e}_{4}]$, the $\mathrm{\Delta}\mathbf{W}$ will only have two non-zero
row as

$\mathrm{\Delta}\mathbf{W}=\mathbf{B}_{sb}\mathbf{A}=\left[\begin{array}[]{cc}1&0\\\
0&0\\\ 0&0\\\
0&1\end{array}\right]\left[\begin{array}[]{cccc}b_{11}&b_{12}&b_{13}&b_{14}\\\
b_{12}&b_{22}&b_{23}&b_{24}\end{array}\right]=\left[\begin{array}[]{cccc}b_{11}&b_{12}&b_{13}&b_{14}\\\
0&0&0&0\\\ 0&0&0&0\\\ b_{21}&b_{22}&b_{23}&b_{24}\end{array}\right]$ (11)

The fine-tuned weight $\mathbf{W}^{\prime}$ is given by

$\mathbf{W}^{\prime}=\mathbf{W}_{0}+\mathrm{\Delta}\mathbf{W}=\left[\begin{array}[]{cccc}w_{11}+b_{11}&w_{12}+b_{12}&w_{13}+b_{13}&w_{14}+b_{14}\\\
w_{21}&w_{22}&w_{23}&w_{24}\\\ w_{31}&w_{32}&w_{33}&w_{34}\\\
w_{41}+b_{21}&w_{42}+b_{22}&w_{43}+b_{23}&w_{44}+b_{24}\end{array}\right]$
(12)

In these examples, only half of the weight matrix $\mathbf{W}^{\prime}$
updated, specifically two columns or two rows $(r=2)$, while the remaining
weights remain unchanged. In practice, $r\ll$ min$(k,d)$, therefore only a
small portion of the original weights will be updated. More details of the
regional weight update property of SBoRA-FA and SBoRA-FB can be visualized in
Fig. 2. It resembles the localized learning process observed in neuroscience,
where specific cognitive functions are associated with distinct brain regions.
SBoRA efficiently adapts to new tasks by updating specific rows of the weight
matrix, preserving existing knowledge, similar to how the brain reorganizes
and refines neural connections in response to new experiences.

##  4 Complexity Analysis of SBoRA

The implementation of SBoRA offers a high degree of flexibility, and we
propose sampling-based multiplication, sampleMul, to achieve lower memory and
computational requirements. Essentially, both SBoRA-FA/FB can be implemented
similarly to conventional LoRA through matrix multiplication, with just frozen
the standard basis matrix $\mathbf{A}_{sb}$/$\mathbf{B}_{sb}$. As a result,
the computational complexity and memory usage of SBoRAs are similar to those
of LoRA. However, we can optimize memory usage by representing the standard
basis matrix $\mathbf{A}_{sb}$/$\mathbf{B}_{sb}$ as a $r$-dimensional vector
with integer coefficients corresponding to the indices of the selected
subspace basis vectors. The basis vector indices are initialized by randomly
sampling $r$ unique numbers between 1 and $k$ for SBoRA-FA or between 1 and
$d$ for SBoRA-FB. This implementation of SBoRA achieves a significant
reduction in parameter storage, approximately 50%, while also saving around
half of the gradient storage due to the standard basis matrix
$\mathbf{A}_{sb}$/$\mathbf{B}_{sb}$ being frozen. Table 1 provides an analysis
of the memory requirements for SBoRA compared to LoRA.

Table 1: Analysis of extra memory storage bring by a single LoRA module and SBoRA-FA/FB. Through the utilization of a standard basis and index-based updates SBoRA enables the replacement of an entire matrix with a 1-dimensional vector. This reduction leads to a significant decrease of approximately 50% total parameters.  Method | Trainable Params | Total Params | Gradient Memory
---|---|---|---
LoRA | $(k+d)\times r$ | $(k+d)\times r$ | $(k+d)\times r$
SBoRA-FA | $d\times r$ | $r+k\times r$ | $d\times r$
SBoRA-FB | $k\times r$ | $r+d\times r$ | $k\times r$

To reduce computational complexity, we can leverage one-hot standard basis
vectors to simplify the projection transformation, effectively converting it
into a component sampling process that eliminates the need for multiplication
operations. Specifically, we select components of the input vector based on
the indices of the standard basis vectors. This allows us to sample the input
vector into the projection-down vector based on the corresponding indices in
$\mathbf{A}_{sb}$. The forward process of SBoRA-FA with sampleMul can be
expressed as:

$\displaystyle\text{SBoRA-FA:
}\begin{cases}\mathbf{B}(\mathbf{A}_{sb}\mathbf{x})=\mathbf{B}(\mathbf{x}[seq,\mathbf{A}_{sb}])\\\
\mathbf{h}=\mathbf{W}_{0}\mathbf{x}+\mathbf{BA}_{sb}\mathbf{x}\end{cases}$
(13) where $\mathbf{x}[seq,\mathbf{A}_{sb}]$ is a component sampling process
method for implementing $\mathbf{A}_{sb}\mathbf{x}$. This approach is also
applied to the case of SBoRA-FB in the projection-up transformation. We can
utilize the $index\\_add$($\cdot$) operation to incorporate the non-zero
values from the output projection-up vector into the pre-trained weight output
vector ($\mathbf{W}_{0}\mathbf{x}$) based on the corresponding indices in
$\mathbf{B}_{sb}$. The forward process of SBoRA-FB with sampleMul can be
expressed as: $\displaystyle\text{SBoRA-FB:
}\begin{cases}\mathbf{z}=\mathbf{Ax}\\\
\mathbf{h}=(\mathbf{W}_{0}\mathbf{x})\text{.}\textit{index\\_add}(\mathbf{B}_{sb},\mathbf{z})\end{cases}$
(14)

Notably, for SBoRA-FB, projection-down vector $\mathbf{z}$ is directly added
into the pre-trained weight output vector through index addition method,
eliminating the need for matrix multiplication operations. During this
process, two matrix multiplication operations
$\mathbf{B}(\mathbf{A}_{sb}\mathbf{x})$ / $\mathbf{Ax}$ and
$\mathbf{W}_{0}\mathbf{x}$, and one matrix addition operation of
$\mathbf{W}_{0}\mathbf{x}+\mathbf{BA}_{sb}\mathbf{x}$ /
$\mathbf{W}_{0}\mathbf{x}+\mathbf{B}_{sb}\mathbf{Ax}$ are involved.

It reduces computation complexity by saving one matrix multiplication
operation compared to LoRA[10]. For our experiments, we used the SBoRA
sampleMul implementation due to its lower computational complexity and memory
usage compared to traditional LoRA methods. Table 2 provides an analysis of
computational requirements for SBoRA using sampleMul compared to LoRA,
assuming input $\mathbf{x}$ in the shape of $[d\times 1]$.

Table 2: Computational analysis and comparison of LoRA and SBoRA’s sampleMul implementation, with a pre-trained weight shape of $k\times d$ and a LoRA rank of $r$. Notably, in the SBoRA_sampleMul implementations, the number of multiplication and addition operations is significantly reduced. Method | Multiplication | Addition
---|---|---
LoRA |  $k\times d$; $(k+d)\times r$ |  $k\times(d-1)$; $r\times(d-1)+k\times(r-1)$; $1\times k$
SBoRA-FA |  $k\times d$; $k\times r$ |  $k\times(d-1)$; $k\times(r-1)$; $1\times k$
SBoRA-FB |  $k\times d$; $d\times r$ |  $k\times(d-1)$; $r\times(d-1)$; $1\times r$

##  5 Experiment

We conducted a comprehensive evaluation of SBoRA’s effectiveness across a
range of tasks. To begin with, we compared the performance of SBoRA-FA and
SBoRA-FB to that of several other well-known PEFT methods by fine-tuning LLMs,
specifically LLaMA-7B, on both commonsense reasoning tasks and arithmetic
reasoning tasks. Additionally, given the recent release of LLaMA3 [18], we
expanded our evaluation to assess SBoRA’s performance using LLaMA3-8B.
Furthermore, recognizing the growing interest in quantized models, we also
evaluated SBoRA on the quantized versions of LLaMA-7B LLaMA-13B and LLaMA3-8B
models, utilizing the techniques introduced in QLoRA [4].

###  5.1 Evaluating SBoRA on Commonsense Reasoning Tasks

We evaluated SBoRA-FA/FB against LoRA, and DoRA approach on the LLaMA-7B and
LLaMA3-8B language models for commonsense reasoning tasks. We followed the
evaluation method introduced in DoRA [17] and LLM-Adapter [11] which used a
comprehensive training dataset created by aggregating eight tasks, evaluating
models on individual testing datasets for each task. In addition, the results
of GPT-3.5 are included for reference as shown in Table 3.

For a comprehensive comparison, we initially fine-tuned models with SBoRA-FA
and SBoRA-FB, both using a double rank of 64 to maintain a nearly identical
number of trainable parameters (TP) as LoRA (rank=32), with only additional
storage for standard basis indices. We then conducted experiments with rank 32
for SBoRA-FA/FB and 64 for other baselines to investigate if SBoRA could
achieve better performance with half the trainable parameters. Consistency was
maintained by using one training epoch and learning rate of 2e-4 for all
experiments.

In LLaMA-7B, SBoRA-FA (rank=64) outperforms all baseline methods with a
similar number of trainable parameters, including LoRA (rank=32), and DoRA
(rank=32). SBoRA-FA (rank=32) achieves commendable performance, matching the
accuracy of DoRA (rank=32) and trailing behind the best accuracy of LoRA
(rank=32) by only 1.1%. Despite having approximately half the trainable
parameters, SBoRA performs comparably to models with the same rank. While in
LLaMA3-8B, the top two results are achieved by SBoRA-FA with rank 64 and 32,
respectively.

Importantly, under identical settings and rank with NVIDIA Geforce RTX 4090,
SBoRA-FA/FB significantly reduces training time by half compared to DoRA.
Fine-tuning with DoRA takes around 7-8 hours, while achieving comparable or
better performance with SBoRA can be done in just 3-4 hours. The notable
computational efficiency of SBoRA saves a substantial amount of time.

Table 3: Comparison of LLaMA-7B and LLaMA3-8B with different PEFT methods, and evaluating on the commonsense reasoning task. The first row list the results of GPT-3.5 for reference. We report the accuracy (%) results for each of the eight sub-tasks as well as the average accuracy (%), higher is better for all metrics. The column headers indicates TP for the number of trainable parameters and $r$ for the rank. Model | Method | $r$ | TP | BoolQ | PIQA | SIQA | HS | WG | ARC-e | ARC-c | OBQA | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---
GPT-3.5 | - | - | - | 73.1 | 85.4 | 68.5 | 78.5 | 66.1 | 89.8 | 79.9 | 74.8 | 77.0
LLaMA-7B | LoRA | 32 | 56.1M | 66.8 | 81.1 | 78.4 | 53.5 | 80.5 | 81.1 | 61.9 | 79.4 | 72.8
DoRA | 57.0M | 68.8 | 82.0 | 70.6 | 57.6 | 73.2 | 79.4 | 64.2 | 78.2 | 71.7
SBoRA-FA | 28.0M | 68.0 | 79.7 | 76.2 | 54.4 | 79.1 | 79.8 | 61.3 | 75.0 | 71.7
SBoRA-FB | 28.0M | 66.1 | 64.2 | 74.8 | 57.2 | 71.5 | 80.4 | 62.4 | 75.8 | 64.9
LoRA | 64 | 112.2M | 62.1 | 81.8 | 78.2 | 62.9 | 78.6 | 79.8 | 63.7 | 81.2 | 73.5
DoRA | 113.1M | 68.7 | 82.8 | 78.2 | 64.8 | 62.9 | 79.7 | 64.8 | 80.0 | 72.7
SBoRA-FA | 56.1M | 68.2 | 81.3 | 77.6 | 74.7 | 81.1 | 80.8 | 62.8 | 79.4 | 75.7
SBoRA-FB | 56.1M | 66.5 | 79.2 | 76.7 | 59.2 | 76.5 | 76.8 | 59.0 | 74.4 | 71.0
LLaMA3-8B | LoRA | 32 | 56.6M | 71.9 | 86.7 | 80.4 | 94.0 | 85.6 | 87.8 | 75.9 | 83.6 | 83.2
DoRA | 57.4M | 73.6 | 87.1 | 80.8 | 94.4 | 86.1 | 88.8 | 78.3 | 84.2 | 84.2
SBoRA-FA | 25.2M | 73.3 | 87.8 | 79.1 | 93.9 | 85.2 | 89.9 | 80.0 | 86.0 | 84.4
SBoRA-FB | 31.5M | 72.9 | 86.3 | 78.8 | 92.6 | 83.0 | 88.8 | 76.3 | 85.0 | 83.0
LoRA | 64 | 113.2M | 72.5 | 87.8 | 80.3 | 94.4 | 86.4 | 88.7 | 79.3 | 85.2 | 84.3
DoRA | 114.0M | 70.5 | 86.0 | 80.3 | 91.8 | 83.7 | 86.2 | 74.7 | 83.2 | 82.1
SBoRA-FA | 50.3M | 74.0 | 88.3 | 80.8 | 94.3 | 86.3 | 89.9 | 78.7 | 86.6 | 84.9
SBoRA-FB | 62.9M | 71.8 | 85.2 | 79.2 | 91.4 | 82.9 | 86.7 | 74.0 | 83.4 | 81.8

###  5.2 Evaluating SBoRA on Arithmetic Reasoning

We also evaluated the effectiveness of SBoRA on the Arithmetic Reasoning task
using the LLaMA-7B and LLaMA3-8B models. The fine-tuning and evaluation
datasets are constructed following the dataset settings in LLM-Adapters [11].
We performed tests with ranks 32 and 64. As the training set for arithmetic
reasoning is smaller (10k) compared to the commonsense reasoning training set,
we increased the number of training epochs to 3 for better convergence and set
the learning rate to 3e-4.

Table 4 presents the performance of PEFT methods, including GPT-3.5. On
average, for LLaMA-7B, GPT-3.5 (175B) outperforms adapter-based PEFT LLMs in
accuracy. However, for simpler math reasoning tasks like MultiArith, adapter-
based methods outperform GPT-3.5, and SBoRA-FA achieves the best performance.
Initially, both SBoRA-FA and SBoRA-FB used a rank of 64 for consistency in
trainable parameters. SBoRA-FA achieves the best average accuracy. Increasing
the rank of other baselines to 64 improves accuracy but still lags behind
SBoRA-FA. Reducing SBoRA’s rank to 32 maintains comparable performance to
rank-32 baselines. Notably, for the challenging task AQuA, SBoRA-FA with rank
32 excels. Moving to LLaMA3-8B, SBoRA-FA with rank 32 achieves the best
performance, followed by SBoRA-FA with rank 64.

SBoRA-FA/FB achieves comparable or superior results to other baseline PEFT
methods with half the trainable parameters for the same rank. Arithmetic
reasoning tasks are particularly well-suited for SBoRA due to the specialized
knowledge required in mathematics. SBoRA’s regional weight update property
makes it more suitable for higher ranks like 64, impacting more pre-trained
weight regions.

Table 4: Comparison of LLaMA-7B and LLaMA3-8B with different PEFT methods, and evaluating on the arithmetic reasoning task. The first row list out the results of GPT-3.5 for reference. We report the accuracy results for each of the sub-tasks as well as the average accuracy, higher is better for all metrics. The number of trainable parameters (TP) can be found in each row. LLM | Method | Rank | TP | MultiArith | GSM8K | AddSub | AQuA | SingleEq | SVAMP | Avg
---|---|---|---|---|---|---|---|---|---|---
GPT-3.5 | - | - | - | 83.8 | 56.4 | 85.3 | 38.9 | 88.1 | 69.9 | 70.4
LLaMA-7B | LoRA | 32 | 56.1M | 94.5 | 36.3 | 81.8 | 15.0 | 82.7 | 45.6 | 59.3
DoRA | 57.0M | 95.7 | 36.2 | 78.7 | 15.4 | 81.7 | 46.6 | 59.1
SBoRA-FA | 28.0M | 95.5 | 34.6 | 79.7 | 20.1 | 78.9 | 44.8 | 58.9
SBoRA-FB | 28.0M | 92.2 | 31.0 | 77.5 | 15.7 | 78.5 | 41.8 | 56.1
LoRA | 64 | 112.2M | 94.0 | 36.8 | 84.3 | 17.3 | 82.3 | 44.7 | 59.9
DoRA | 113.1M | 95.0 | 35.5 | 84.1 | 20.1 | 85.0 | 47.1 | 61.1
SBoRA-FA | 56.1M | 97.8 | 36.6 | 85.1 | 19.3 | 83.9 | 48.5 | 61.9
SBoRA-FB | 56.1M | 94.8 | 33.1 | 77.5 | 16.9 | 78.5 | 40.6 | 56.9
LLaMA3-8B | LoRA | 32 | 56.6M | 68.3 | 50.5 | 83.3 | 35.8 | 87.2 | 71.2 | 66.1
DoRA | 57.4M | 97.3 | 62.0 | 90.9 | 25.6 | 94.9 | 73.4 | 74.0
SBoRA-FA | 25.2M | 99.5 | 66.0 | 91.9 | 30.3 | 97.4 | 75.8 | 76.8
SBoRA-FB | 31.5M | 98.0 | 57.2 | 92.2 | 33.9 | 94.1 | 69.6 | 74.2
LoRA | 64 | 113.2M | 97.2 | 56.3 | 92.7 | 22.8 | 92.3 | 69.3 | 71.8
DoRA | 114.0M | 97.8 | 55.2 | 91.1 | 24.0 | 94.7 | 72.0 | 72.5
SBoRA-FA | 50.3M | 99.2 | 64.7 | 94.4 | 24.8 | 98.0 | 75.0 | 76.0
SBoRA-FB | 62.9M | 98.2 | 50.9 | 87.1 | 28.0 | 91.7 | 63.0 | 69.8

###  5.3 QSBoRA Evaluation on MMLU

We explored the effectiveness of SBoRAs in the memory-efficient QLoRA
framework [4]. QLoRA utilizes techniques including 4-bit NormalFloat (NF4)
double quantization, gradient checkpointing, and paged optimizer to reduce
memory usage. Following the experiments in QLoRA, we fine-tuned the NF4
quantized versions of LLaMA-7B/13B and LLaMA3-8B using the Alpaca [24] and
sampled FLAN v2 [28] datasets. Model performance was evaluated on the MMLU
benchmark [8] by comparing the mean 5-shot MMLU test accuracy.

To ensure fairness, we used the same hyperparameter settings for all
experiments. For the Alpaca dataset, we applied a batch size (bs) of 16 and
10k training steps. To maintain a similar number of training epochs with
Alpaca, for FLAN v2, we used batch size 8 and 20k training steps. In the QLoRA
framework, we compared SBoRA with LoRA and DoRA, using rank 64 for all
methods. Adapters were added to linear layers in both self-attention and MLP
modules.

The results shown in Table 5 reveal that within LLaMA-7B, both QSBoRA-FA/FB
achieve comparable accuracy results to QLoRA. In the case of LLaMA-13B and
LLaMA3-8B, both QSBoRA-FA/FB outperform QLoRA and QDoRA in terms of accuracy,
indicating its compatibility with larger-scale models.

Table 5: Finetune LLaMA-7B/13B and LLaMA3-8B on Alpaca and Flan v2. We measured the performance using MMLU benchmark and report the 5-shot average accuracy. The training settings and the number of trainable parameters (TP) are included. NFloat4 | LLaMA-7B | LLaMA-13B | LLaMA3-8B
---|---|---|---
PEFT | TP | Alpaca | Flanv2 | TP | Alpaca | Flanv2 | TP | Alpaca | Flanv2
QLoRA | 80.0M | 37.9 | 44.4 | 125.2M | 45.4 | 46.7 | 83.9M | 51.9 | 49.5
QDoRA | 80.6M | 38.0 | 42.8 | 126.2M | 46.7 | 48.8 | 84.6M | 53.0 | 51.9
QSBoRA-FA | 43.5M | 36.5 | 43.1 | 68.2M | 49.0 | 51.0 | 44.0M | 56.5 | 56.4
QSBoRA-FB | 36.4M | 36.9 | 43.4 | 57.0M | 48.3 | 50.5 | 39.8M | 54.5 | 55.0

##  6 Conclusion

We introduce SBoRA, a novel approach for parameter-efficient fine-tuning of
LLMs. Our proposed SBoRA-FA and SBoRA-FB methods utilize the low-rank
projection-down and projection-up matrices constructed from standard basis
vectors, leveraging one-dimensional index representations and sample-based
multiplication. This innovative approach reduces memory and computational
requirements while enhancing learning capacity, outperforming LoRA and
matching the performance of DoRA. Empirical results demonstrate that SBoRA-FA
is the preferred method for single-task LLM fine-tuning, conserving activation
memory while achieving superior performance.

The SBoRA approach enables regional weight updates, preserving most of the
pre-trained model weights while efficiently adapting to new tasks. This
localized learning process draws parallels with the modular organization of
the brain, where distinct cognitive functions are localized to specific brain
regions. This analogy highlights the potential of SBoRA to inspire AI
architectures that mimic the efficiency and adaptability of biological neural
systems.

SBoRA holds immense potential for further development, particularly in multi-
task training. The introduction of Multi-SBoRA would create a powerful
framework for efficient adaptation to multiple tasks. Each task would undergo
independent, non-overlapping weight fine-tuning with SBoRA, allowing the
integration of task-specific knowledge while minimizing interference and
maximizing the model’s capacity to leverage shared information. This approach
enables the maintenance of distinct capabilities for each task within a single
model, paving the way for more efficient and effective AI systems

## References

  * [1] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
  * [2] Chavan, A., Liu, Z., Gupta, D., Xing, E., Shen, Z.: One-for-all: Generalized lora for parameter-efficient fine-tuning. arXiv preprint arXiv:2306.07967 (2023)
  * [3] Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., Jia, J.: Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307 (2023)
  * [4] Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems 36 (2024)
  * [5] Genon, S., Reid, A., Langner, R., Amunts, K., Eickhoff, S.B.: How to characterize the function of a brain region. Trends in cognitive sciences 22(4), 350–364 (2018)
  * [6] Guo, H., Greengard, P., Xing, E.P., Kim, Y.: Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311.12023 (2023)
  * [7] He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., Neubig, G.: Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366 (2021)
  * [8] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., Steinhardt, J.: Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020)
  * [9] Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for nlp. In: International conference on machine learning. pp. 2790–2799. PMLR (2019)
  * [10] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
  * [11] Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.P., Bing, L., Xu, X., Poria, S., Lee, R.K.W.: Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933 (2023)
  * [12] Lara, A.H., Wallis, J.D.: The role of prefrontal cortex in working memory: a mini review. Frontiers in systems neuroscience 9, 173 (2015)
  * [13] Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021)
  * [14] Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International conference on machine learning. pp. 12888–12900. PMLR (2022)
  * [15] Li, X.L., Liang, P.: Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021)
  * [16] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36 (2024)
  * [17] Liu, S.Y., Wang, C.Y., Yin, H., Molchanov, P., Wang, Y.C.F., Cheng, K.T., Chen, M.H.: Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353 (2024)
  * [18] Llama, M.: Meta llama 3. https://llama.meta.com/llama3/ (2024)
  * [19] Mahabadi, R.K., Ruder, S., Dehghani, M., Henderson, J.: Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint arXiv:2106.04489 (2021)
  * [20] Po, R., Yang, G., Aberman, K., Wetzstein, G.: Orthogonal adaptation for modular customization of diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7964–7973 (2024)
  * [21] Razdaibiedina, A., Mao, Y., Hou, R., Khabsa, M., Lewis, M., Ba, J., Almahairi, A.: Residual prompt tuning: Improving prompt tuning with residual reparameterization. arXiv preprint arXiv:2305.03937 (2023)
  * [22] Renduchintala, A., Konuk, T., Kuchaiev, O.: Tied-lora: Enhacing parameter efficiency of lora with weight tying. arXiv preprint arXiv:2311.09578 (2023)
  * [23] Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., et al.: S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:2311.03285 (2023)
  * [24] Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., Hashimoto, T.B.: Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca (2023)
  * [25] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
  * [26] Wang, Y., Wu, J., Dabral, T., Zhang, J., Brown, G., Lu, C.T., Liu, F., Liang, Y., Pang, B., Bendersky, M., et al.: Non-intrusive adaptation: Input-centric parameter-efficient fine-tuning for versatile multimodal modeling. arXiv preprint arXiv:2310.12100 (2023)
  * [27] Wang, Y., Lin, Y., Zeng, X., Zhang, G.: Multilora: Democratizing lora for better multi-task learning. arXiv preprint arXiv:2311.11501 (2023)
  * [28] Wei, J., Bosma, M., Zhao, V.Y., Guu, K., Yu, A.W., Lester, B., Du, N., Dai, A.M., Le, Q.V.: Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021)
  * [29] Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., Tian, Q.: Qa-lora: Quantization-aware low-rank adaptation of large language models. arXiv preprint arXiv:2309.14717 (2023)
  * [30] Yonelinas, A., Hawkins, C., Abovian, A., Aly, M.: The role of recollection, familiarity, and the hippocampus in episodic and working memory. Neuropsychologia 193, 108777 (2024)
  * [31] Yuan, Z., Chen, R., Li, Z., Jia, H., He, L., Wang, C., Sun, L.: Mora: Enabling generalist video generation via a multi-agent framework. arXiv preprint arXiv:2403.13248 (2024)
  * [32] Zhang, L., Zhang, L., Shi, S., Chu, X., Li, B.: Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.03303 (2023)


# Quantum Magnetic Skyrmions on the Kondo Lattice

Zhen Zhao<EMAIL_ADDRESS>Division of Mathematical Physics and ETSF,
Lund University, PO Box 118, 221 00 Lund, Sweden  Emil Östberg
<EMAIL_ADDRESS>Division of Mathematical Physics and ETSF, Lund
University, PO Box 118, 221 00 Lund, Sweden  Ferdi Aryasetiawan
<EMAIL_ADDRESS>Division of Mathematical Physics and ETSF,
Lund University, PO Box 118, 221 00 Lund, Sweden  LINXS Institute of advanced
Neutron and X-ray Science (LINXS), IDEON Building: Delta 5, Scheelevägen 19,
223 70 Lund, Sweden  Claudio Verdozzi<EMAIL_ADDRESS>Division
of Mathematical Physics and ETSF, Lund University, PO Box 118, 221 00 Lund,
Sweden  LINXS Institute of advanced Neutron and X-ray Science (LINXS), IDEON
Building: Delta 5, Scheelevägen 19, 223 70 Lund, Sweden

###### Abstract

A quantum description is given of nanoskyrmions in 2D textures with localised
spins and itinerant electrons, isolated or coupled to leads, in or out of
equilibrium. The spin-electron exchange is treated at the mean-field level,
while Tensor Networks and exact diagonalization or nonequilibrium Green’s
functions are used for localised spins and itinerant electrons. We motivate
our scheme via exact benchmarks, then show by several examples that itinerant
electrons distinctly affect the properties of quantum nanoskyrmions. Finally,
we mention lines of future work and improvement of the approach.

Magnetic skyrmions are topologically protected, long-lived spin textures which
can be manipulated by ultralow currents [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11],
and of great potential for applications in spintronics and quantum computing
[12, 13, 14, 15, 16]. Skyrmions originate from the interplay of different
types of magnetic interactions in solids [17], such as Heisenberg exchange
[18], Dzyaloshinskii-Moriya interaction (DMI) [19, 20], and anisotropy [21,
22], to mention some.

Theoretical descriptions of skyrmions often treat spins classically [23, 24,
25, 26], via the Landau-Lifshitz-Gilbert [27, 28] and Thiele equations [29].
While very successful, these formulations, where itinerant electrons
(i-electrons) enter implicitly or as a source of renormalization of the
classical spin dynamics, are not always adequate.

For example, a classical description of the spins is reliable for skyrmion
sizes of hundreds of lattice constants and large spin ($S\gtrsim 2$), where
quantum fluctuations are small. However, as experiments motivate, for
nanometer-scale skyrmions [23] (and, in general, for $S\lesssim 1$) a quantum
description [30, 31, 32, 33, 34] is in order. This is usually via exact
diagonalization (ED) and Tensor Networks [35, 36, 37, 38] (for recent
analytical and dynamical mean-field-theory studies, see [33, 34]).

Furthermore, describing explicitly the interaction (via Kondo exchange) of the
i-electrons with the localised spins (l-spins) can also be important, to e.g.
address interface phenomena like current-driving and optical generation of
nanoskyrmions [39]. At the atomistic level, most i-electrons+l-spins schemes
treat the spins classically and the electrons quantum mechanically [40, 41,
42, 43, 44, 45, 46, 47, 48]. Computations are then viable also for fairly
large samples, but the outcome can be at variance with exact quantum results
[49], obtained by ED or by mapping the (2D) spin-electron system to mixed
chains to use the Density Matrix Renormalization Group (DMRG). Since both
spins and electrons are explicitly treated, such exact results are from small
clusters and mostly for benchmark uses.

Thus, it is highly desirable (in fact necessary in several cases) to have a
framework including i-electrons and electronic reservoirs/leads, but not
neglecting the quantum effects on nanoskyrmions. However, to proceed,
approximations are needed, since in essence one deals with a Kondo-lattice
problem, further complicated by Heisenberg and DMI interactions for the
l-spins.

In this work, we propose a practically viable quantum treatment for both
skyrmions and electrons, in- and out-of-equilibrium. The approach rests on a
mean-field treatment of the Kondo-exchange. Yet, it describes at the quantum
level both spin-texture and electron dynamics, to access intrinsically quantum
physical quantities, e.g. spin-entanglement and quantum spin chirality.
Furthermore, our numerical results for different system and situations
illustrate the merit of the approach and show that including explicitly
i-electrons has a very noticeable effect on the properties of quantum
nanoskyrmions.

System and Hamiltonian. We consider three setups of increasing complexity,
shown in Fig. 1a. By denoting as $C$ ($L_{E}$) the regions where the l-spins
(i-electrons) reside, we have i) both l-spins and i-electrons in the same
finite lattice $C\equiv L_{E}$; ii) l-spins in $C$, and i-electrons in an
enlarged finite lattice $L_{E}=L+C+R$, where region $L$ ($R$) is coupled to
$C$ from the left (right); iii) same as ii), but with semi-infinite $L$ and
$R$ regions. We describe i-iii) via a Kondo lattice Hamiltonian plus
Heisenberg and DMI, i.e. $\hat{H}=\hat{H}^{s}+\hat{H}^{d}+\hat{H}^{sd}$ with

$\displaystyle\hat{H}^{s}=-t_{h}\\!\\!\\!\\!\\!\\!\sum_{\langle ij\rangle\in
L_{E};\sigma}\\!\\!\\!\big{(}\hat{c}_{i\sigma}^{\dagger}\hat{c}_{j\sigma}+\text{h.c.}\big{)},\quad\hat{H}^{sd}=-g\sum_{i\in
C}\hat{\mathbf{s}}_{i}\cdot\hat{\mathbf{S}}_{i}$
$\displaystyle\hat{H}^{d}=\\!\\!\\!\\!\\!\\!\sum_{\langle ij\rangle\in
C}\Big{[}\mathbf{D}_{ij}\cdot\hat{\mathbf{S}}_{i}\times\hat{\mathbf{S}}_{j}-J_{ij}\hat{\mathbf{S}}_{i}\cdot\hat{\mathbf{S}}_{j}\Big{]}-\sum_{i\in
C}\mathbf{B}_{i}\cdot\hat{\mathbf{S}}_{i}.$ (1)

Here, $\hat{H}^{s}$ is the i-electrons part, with hopping parameter $t_{h}$
between nearest-neighbour (n.n.) sites, and $\hat{c}_{i\sigma}^{\dagger}$
creating an electron with spin projection $\sigma$ at site $i$. The l-spins
texture is described by $\hat{H}^{d}$, where $\mathbf{\hat{S}}_{i}$ is the
spin operator at site $i$, and $J$, $D$, and $\mathbf{B}$ respectively denote
the Heisenberg exchange, a Neel-type DMI, and the external magnetic field.
Finally, $\hat{H}^{sd}$ is the Kondo exchange term , with
$\mathbf{\hat{s}}_{i}=(1/2)\sum_{\tau\tau^{\prime}}\hat{c}^{\dagger}_{i\tau}\boldsymbol{\sigma}_{\tau\tau^{\prime}}\hat{c}_{i\tau^{\prime}}$
and $\boldsymbol{\sigma}\equiv(\sigma^{x},\sigma^{y},\sigma^{z})$ the vector
of Pauli matrices. In $\hat{H}^{s}$, there is no spin-orbit interaction (SOI).
Usually small, SOI can sometimes affect skyrmion stabilisation [50, 39, 51,
46]), but it is not considered here to limit the number of model parameters.

Figure 1: a) Sketch of the setups considered, where i-electrons and l-spins
interact via local Kondo exchange of strength $g$. b) Difference between exact
and mean-field ground-state expectation values when
$D\\!=\\!0.2,J\\!=\\!0.1,B^{z}\\!=\\!0.1,t_{h}\\!=\\!1,g\\!=\\!2$.

Approach and method of solution. We will treat $\hat{H}^{sd}$ at the mean-
field level. Hence, by defining $\hat{V}^{s}=\sum_{i\in
C}\langle\hat{\mathbf{S}}_{i}\rangle\cdot\hat{\mathbf{s}}_{i}$ and
$\hat{V}^{d}=\sum_{i\in
C}\langle\hat{\mathbf{s}}_{i}\rangle\cdot\hat{\mathbf{S}}_{i}$, we have
$\hat{H}\rightarrow(\hat{H}^{s}_{MF},\hat{H}^{d}_{MF})$, with

$\displaystyle\hat{H}^{s/d}_{MF}=\hat{H}^{s/d}-g\hat{V}^{s/d}.$ (2)

The results in the paper are obtained using Eq. (2).
Isolated/enlarged system.- Starting with $\langle\mathbf{s}_{i}\rangle=0$, we
use a Matrix Product State (MPS) algorithm [52, 53, 54] from the ITensor
library [55] to find the ground state of $\hat{H}^{d}_{MF}$ and update the
averages $\langle\mathbf{S}_{i}\rangle$. These then enter as parameters when
solving via ED for $\hat{H}^{s}_{MF}$, while reproducing
$\langle\mathbf{s}_{i}\rangle$. As the iterations converge, the mean-field
ground state is reached (see the supplementary material, SM, for details). For
the enlarged case, the procedure is carried out with $\epsilon_{i\sigma}=0$ at
$t=0$). The time evolution for the enlarged system is discussed later in the
paper.
Open system.- Starting with the ground state of the isolated region $C$, the
tunnelling matrix elements between $C$ and the leads $R$, $L$ are switched on
in time adiabatically up to a final value, so that $L+C+R$ reaches a steady
state (chosen as the initial/ground state of the open system). In the process,
the quantum l-spins evolve in time via time-evolution block-decimation (TEBD)
[56], and the l-electrons via the nonequilibrium Green’s function (NEGF)
approach [57, 58, 59, 60] in the equal-time formulation. In this scheme, the
key quantity is the one-particle density matrix $\rho$ for region C, related
to the full one-particle NEGF $G(t,t^{\prime})$ via $\rho(t)=-iG(t,t^{+})$.
With leads, $\rho$ obeys

$\displaystyle i\frac{d}{dt}\rho(t)=[h(t)\rho(t)-iI_{col.}(t)]-\text{h.c.}$
(3)

Here, $h(t)$ is the (possibly time-dependent) single particle term and
$I_{col.}$ is the collision integral [61], that in general describes
interaction-induced correlation effects and lead-induced embedding effects.
The i-electrons do not interact in our model, i.e. $I_{col.}$ has only the
embedding part. We also i) use the so-called wide-band limit (WBL), where the
hopping term $t_{l}$ in $L,R$ tends to infinity, whilst
$\gamma=t_{l}^{2}/t_{h}$ remain fixed ($\gamma$ thus sets the magnitude of the
$L$-$C$ and $R$-$C$ couplings); ii) further approximate the WBL by simplifying
the calculation of $I_{col.}$. Steps i,ii) are a good trade-off between
reduced computation time and accuracy [48] (see the SM).

Figure 2: Ground state results for $D=0.2,J=0.1,B^{z}=0.1,t_{h}=1,g=2$. a)
Spin expectation-value (left) and concurrences $\mathcal{C}$ (right) heatmaps
in a $21\times 21$ isolated rhombus cluster, with l-spins (i-electrons)
chiralities $Q_{\text{sp}}=-6.37$ ($Q_{\text{el}}=-0.030$). In the spin map,
the top (bottom) half shows the expectation value $\langle S^{x,y,z}\rangle$
($\langle s^{x,y,z}\rangle$) of the l-spins (the i-electrons). The values of
the $z$\- and $xy$ spin-components are indicated by colours and arrows,
respectively. In the concurrence map, the value of $\mathcal{C}$ for n.n.
(upper half) and next nearest neighbour (2nn, lower half) sites is represented
by the bond colour. The missing half of each map is recovered using rhombus
symmetry. b) Logarithm of the static structure factors $\ln S^{zz}$ (left) and
$\ln S^{+-}$ (right) of the l-spins. c) $\langle S^{x,y,z}\rangle$ (left half)
and $\langle s^{x,y,z}\rangle$ for a $6\times 6$ square cluster contacted to
leads. Figure 3: Nonequilibrium dynamics of a rhombus cluster $C$ of $4\times
10$ sites with an enlarged electron region $L+C+R$ of $4\times(100+10+100)$
sites, for parameters
$D\\!=\\!0.2,J\\!=\\!0.05,B^{z}\\!=\\!0.06,t_{h}\\!=\\!1,g\\!=\\!2$. The
dynamics is induced by a spin-up polarized bias $\epsilon_{L}=-\epsilon_{R}$
in the enlarged region, switched-on at $t\\!=\\!0$ and ramped to maximum
strength 0.5 at $\\!t=\\!10$. a) Spin-up current $I_{\langle ij\rangle}$ along
the bonds $\langle ij\rangle$ labeled by A-F (sketch on the right). b) Time
snapshots of the spin expectation values $\langle S^{x,y,z}(t)\rangle$ in $C$,
$z$\- and $xy$-spin components indicated by colours and arrows, respectively.
The current-driven motion of one meron is highlighted by the green semicircle.

Exact vs mean field Kondo exchange, and quantum vs classical spins. To see if
a quantum treatment of l-spins+i-electrons (albeit within a mean-field account
of Kondo exchange) is beneficial, we consider a square plaquette with 4 spins
and 4 electrons and, via Eq. (1) and the Lanczos method, we determine for
reference its full quantum ground state. Then, we move around the plaquette
while increasing the number $n_{cl}$ of classical l-spins from 0 to 4. This
results in a mean field, semiclassical (SC) coupling scheme between l-spins
and electrons, and between classical and quantum l-spins. To characterize the
SC treatment, we use $\eta_{n_{cl}}=N^{-1}\sum_{i\in
C}|\langle{\hat{\bm{S}}^{\text{Ex.}}_{i}}\rangle-\langle{\hat{\bm{S}}^{\text{SC}}_{i}(n_{cl})}\rangle|\big{/}|\langle{\hat{\bm{S}}^{\text{Ex.}}_{i}}\rangle|$,
which depends on the number $n_{cl}$ of l-spins treated classically. In Fig.
1b), $\eta$ grows as a function of $n_{cl}$ but gets smaller when the spin
value $S$ increases from 1/2 to 5/2, a sign that the classical regime is being
approached.

Next, to discuss the i-electrons’ role, we move to a spin-electron (sp-el) and
a spin-only (sp) system in a rhombus-shaped $11\times 11$ finite triangular
lattice. We again compare their respective ground states using
$\eta^{\prime}=N^{-1}\sum_{i\in
C}\big{|}\langle\hat{\mathbf{S}}_{i}^{\text{sp-
el}}\rangle-\langle\hat{\mathbf{S}}_{i}^{\text{sp}}\rangle\big{|}\big{/}\big{|}\langle\hat{\mathbf{S}}_{i}^{\text{sp}}\rangle\big{|}$.
We choose
$D\\!\\!=\\!\\!0.2,J\\!\\!=\\!\\!0.15,B^{z}\\!\\!=\\!\\!0.06,t_{h}\\!\\!=\\!\\!1,g\\!\\!=\\!\\!2$,
for which both systems exhibit a single-skyrmion texture. This is confirmed by
the value of the quantum scalar chirality [31] $Q=\pi^{-1}\sum_{<ijk>\in
C}\langle\hat{\mathbf{S}}_{i}\cdot[\hat{\mathbf{S}}_{j}\times\hat{\mathbf{S}}_{k}]\rangle$,
where the sum runs over all non overlapping triangles formed by neighboring
sites $i$, $j$, $k$. For $S=1/2$ l-spins, $|Q|\sim 1$ signals the presence of
a skyrmion. For the case at hand we respectively find
$Q_{\text{sp}}^{\text{sp-el}}=-0.618$, and $Q^{\text{sp}}=-0.624$. Even in
this strong Kondo regime ($g/t_{h}=2$), the mean-field electron-spin
expectation values $\langle\hat{\mathbf{s}}_{i}\rangle$ are only few percents
of the corresponding $\langle\hat{\mathbf{S}}_{i}^{\text{sp-el}}\rangle$. Yet,
since $\eta^{\prime}=0.14$, the sp-el and sp ground states noticeably differ.
Altogether, these comparisons confirm that i-electrons markedly affect the
quantum nanoskyrmion and that it is beneficial, even with a mean-field
treatment of Kondo exchange, to have quantum rather than classical l-spins.

Ground state. To simulate nanosized skyrmions in closed/open systems we take
$D/J\sim 2-5$. These are large values, but consistent with the giant DMI ones
from some monolayer materials [62]. The other parameters used, $J/t_{h}\sim
0.1-0.5$ and $g/t_{h}\sim 1-2$, are also congruous with typical values in the
literature [49].
Isolated system.- We consider is a $21\times 21$ rhombus cluster (which
permits to simulate a skyrmion lattice) with open boundary conditions and
parameters $D=0.2,J=0.1,B^{z}=0.1,t_{h}=1,g=2$. The ground state profile of
such spin+electron system is shown in Fig. 2a, where the skyrmions exhibit an
approximate periodic alignment, whilst the sites with $S^{z}\simeq\frac{1}{2}$
state (black circles) can be thought of as domain walls. Since the l-spins are
described quantum mechanically, we can look at entanglement between spin pairs
at sites $i\neq j$ [63, 64, 37], quantified by the concurrence
$\mathcal{C}_{ij}=\mathcal{C}[\hat{\rho}_{ij}]$, and where $\hat{\rho}_{ij}$
is the reduced density matrix. Similarly to what found in [37], $\mathcal{C}$
shows a lack of long-range entanglement for the skyrmion (crystal) texture.
However, since the distance $\eta^{\prime}$ between sp-el and sp systems is
$10\%$, the concurrences with and without i-electrons discernibly differ. In
turn, the i-electrons are affected by the skyrmion texture (for example, the
local density of states for spin-up and spin-down i-electrons show large
(small) imbalance at (far away from) the skyrmions-core sites, see SM).

Further insight comes from the spin structure factor, defined as
$S^{\alpha\beta}(\mathbf{k})=N^{-1}\sum_{ij\in
C}\langle\hat{S}_{i}^{\alpha}\hat{S}_{j}^{\beta}\rangle
e^{i\mathbf{k}\cdot(\mathbf{r}_{i}-\mathbf{r}_{j})}$, and with $\alpha\beta$
denoting either the $zz$ or the $+-$ components. In Fig. 2b (for clarity, we
show $\ln S^{\alpha\beta}$ rather than $S^{\alpha\beta}$), both $S^{zz}({\bf
k})$ and $S^{+-}({\bf k})$ exhibit sixfold intensity patterns, consistent with
neutron scattering results (from e.g. the B20 compound MnSi [3]). Overall, the
signal is stronger for $S^{zz}$ than for $S^{+-}$; in particular, $S^{zz}$ has
a bright spot at ${\bf k}=0$, due to the strong degree of spin orientation
along $z$. Additionally, the logarithmic display reveals additional but much
less intense, lower symmetry features, that we ascribe to size and the open-
boundary-conditions effects in the rhombus cluster.

Ground state for an open system. Using NEGF for the i-electrons and Tensor
networks for the l-spins, we have also considered the case of a $6\times 6$
sp-el square cluster attached to leads. The isolated central region ground
state at $t=0$ (determined in the mean-field Kondo-exchange) is connected to
the leads adiabatically for $0<t<50$. The evolution of the sp-el system can
then be continued to $t_{0}=125$, which establishes steady spin-polarised
currents through the central region in equilibrium. These currents are weak
and essentially present only at the boundary sites of $C$ (see the SM). The
ground state average spin distributions for this setup, with
$D=1,J=0.2,B^{z}=0.5,t_{h}=1,g=1$, are shown in Fig. 2d. They result in a
value $\eta=0.06$, i.e. a non-strong average influence of the electrons on the
spin-texture. Still, a noticeable and interesting feature is that the spins of
the i-electrons are tilted compared to the isolated case (not shown). Since
for the open case there is a spin-polarized electric current flowing (and a
corresponding induced magnetic field), we attribute the spin tilt to an
induced effective spin-orbit interaction.

Dynamics. For out-of-equilibrium situations, the natural option would be to
consider the open system iii) of Fig. 1c where, after reaching equilibrium at
$t=t_{0}$ (as discussed above, via a slow-ramping dynamics to connect the
leads and dissipate fluctuations), time-dependent currents are injected in the
electron subsystem via a bias. For example, for spin-up polarization and with
reference to Eq. (2),
$\hat{H}^{s}\rightarrow\hat{H}^{s}(t)\equiv\hat{H}^{s}+\hat{w}_{L}(t)+\hat{w}_{R}(t)$,
where $\hat{w}_{L}(t)=\sum_{i\in
L}\epsilon_{L}(t)\hat{c}^{\dagger}_{i\uparrow}\hat{c}_{i\uparrow}$ and similar
for $\hat{w}_{R}(t)$. The time evolution would then be performed via NEGF for
the electrons and via e.g. TEBD for the spins.

We defer the study of the NEGF+TEBD dynamics of a l-spins+i electrons system
connected to semi-infinite leads to a forthcoming paper. Here, as proof of
concept of our approach, we use the enlarged isolated setup of Fig. 1b, with a
region $C$ consisting of a $4\times 10$ rhombus cluster, and an enlarged
electron region of $4\times(100+10+100)$ sites, to delay the reflection of the
currents by the boundaries, and to have steady, stable currents established
within the time of interest. In our simulations, we set
$t_{0}\\!=\\!0,D\\!=\\!0.2,J\\!=\\!0.05,B^{z}\\!=\\!0.06,t_{h}\\!=\\!1,g\\!=\\!2$.
In this case, the (Kondo-exchange mean-field) ground state
$|\Psi^{\text{el}}(0)\rangle$ (obtained without time propagation, and shown at
$t\\!=\\!0$ in Fig. 3b) corresponds to a spin texture with multiple meron-like
(i.e. half-skyrmion-like) structures. These (henceforth called merons for
simplicity) are centered at sites $i$ of the cluster boundaries where
$\langle\hat{S}_{i}^{z}\rangle<0$, with
$\langle\hat{S}_{i^{\prime}}^{z}\rangle>0$ for their second and third
neighbours sites $i^{\prime}$. To support this interpretation and address the
individual behavior of merons, we use a modified, “local” scalar chirality
$Q_{i}$ calculated over all the triangles including sites $i$ and $i^{\prime}$
(for example, at $t\\!=\\!0$, $Q_{i}\sim-0.16$ for the green shaded meron in
Fig. 3b, see SM for details).

Then, starting from $|\Psi^{\text{el}}(0)\rangle$, we introduce for $t>0$ a
bias $\epsilon_{L}(t)=0.5[1-\theta(10-t)\cos^{2}(\pi t/20)]$ with
$\epsilon_{R}(t)=-\epsilon_{L}(t)$. At each time step of the time evolution,
$\hat{H}^{d}_{MF}$ is constructed with input from the time-evolved spin
expectation $\langle\hat{\mathbf{s}}_{i}\rangle_{t}$, and the l-spins are
evolved via TEBD. Subsequently, an updated
$\langle\hat{\mathbf{S}}_{i}\rangle_{t}$ and the bias $\epsilon_{L/R}(t)$
drive the electrons via $\hat{H}^{s}_{MF}(t)$, and the full spin-electron
system is thus evolved for one time step.

The time evolution of selected spin-up bond currents $I_{\langle
ij\rangle}=-2t_{h}\text{Im}\langle\hat{c}_{j\uparrow}^{\dagger}\hat{c_{i\uparrow}}\rangle$,
is shown in Fig. 3a. After the transient regime ($t\lesssim 10$), the currents
at/near the edges ($I_{AF,AE,AB}$) stay rather stable till $t\approx 80$;
then, they start to be reflected at the outer boundaries of the $L,R$ regions.
By contrast, currents away from the edges ($I_{BE,BD,BG}$) undergo reflection
sooner (i.e., $t\approx 50$). In Fig. 3a, the currents at different bonds
differ for magnitude and/or sign (also for parallel adjacent bonds). This is
because of the merons: these are driven in a direction opposite (see the
shaded area in the time snapshots) to the average flow of the spin-up
electrons (also, small size and nonequilibrium quantum fluctuation make the
merons to slightly change their shape in time). The i-electrons, in turn,
conform their flow to the presence of the moving merons, which results in the
observed behavior of the bond spin-up currents. Furthermore, in the ground
state ($t=0$), the i-electron’s spins texture mirrors the l-spins one, but in
a much weaker pattern. The latter extends outside region $C$, while fading as
the distance from $C$ increases. Yet, with currents (i.e. $t>0$), the outer
parts of the pattern move towards $C$, affecting the l-spins texture. This is
why, for example, $\langle\hat{S}^{z}\rangle$ at the top-left site in Fig. 3b
changes sign during the time evolution.

Conclusions. We have proposed an approach to describe quantum nanoskyrmions in
two-dimensional itinerant-electron+localised-spin systems. It combines Matrix
Product States methods for the localised-spins, with exact diagonalization or
nonequilibrium Green’s function methods for the itinerant electrons, while
treating the spin-electron interaction at the mean field level. Our results
from different systems and physical setups show that the approach is
computationally viable to study in- and out-of-equilibrium quantum skyrmions
with itinerant electrons, with a scope different from that of classical-
spin+quantum-electron treatments. As future developments, we plan to go beyond
a mean-field treatment of the spin-electron interaction via non perturbative
local approximations, and to include photon-skyrmion interactions via recent
developments in Tensor Networks [65, 66] and nonequilibrium Green’s function
formalisms [67].

###### Acknowledgements.

Z.Z, E.Ö and C.V. gratefully acknowledge financial support from the Swedish
Research Council (Vetenskapsrådet, VR, Grant No. 2017-03945 and No.
2022-04486). F.A. gratefully acknowledges financial support from the Swedish
Research Council (Vetenskapsrådet, VR, Grant No. 2021-04498_3).

## References

  * Bogdanov and Hubert [1994] A. Bogdanov and A. Hubert, Thermodynamically stable magnetic vortex states in magnetic crystals, J. Magn. Magn. Mater. 138, 255 (1994).
  * Rößler  _et al._ [2006] U. K. Rößler, A. N. Bogdanov, and C. Pfleiderer, Spontaneous skyrmion ground states in magnetic metals, Nature 442, 797 (2006).
  * Mühlbauer  _et al._ [2009] S. Mühlbauer, B. Binz, F. Jonietz, C. Pfleiderer, A. Rosch, A. Neubauer, R. Georgii, and P. Böni, Skyrmion lattice in a chiral magnet, Science 323, 915 (2009).
  * Yu  _et al._ [2010] X. Z. Yu, Y. Onose, N. Kanazawa, J. H. Park, J. H. Han, Y. Matsui, N. Nagaosa, and Y. Tokura, Real-space observation of a two-dimensional skyrmion crystal, Nature 465, 901 (2010).
  * Jonietz  _et al._ [2010] F. Jonietz, S. Mühlbauer, C. Pfleiderer, A. Neubauer, W. Münzer, A. Bauer, T. Adams, R. Georgii, P. Böni, R. A. Duine, K. Everschor, M. Garst, and A. Rosch, Spin transfer torques in mnsi at ultralow current densities, Science 330, 1648 (2010).
  * Schulz  _et al._ [2012] T. Schulz, R. Ritz, A. Bauer, M. Halder, M. Wagner, C. Franz, C. Pfleiderer, K. Everschor, M. Garst, and A. Rosch, Emergent electrodynamics of skyrmions in a chiral magnet, Nat. Phys. 8, 301 (2012).
  * Huang and Chien [2012] S. X. Huang and C. L. Chien, Extended Skyrmion Phase in Epitaxial $\mathrm{FeGe}(111)$ Thin Films, Phys. Rev. Lett. 108, 267201 (2012).
  * Nagaosa and Tokura [2013] N. Nagaosa and Y. Tokura, Topological properties and dynamics of magnetic skyrmions, Nature Nanotechnology 8, 899 (2013).
  * Fert  _et al._ [2013] A. Fert, V. Cros, and J. Sampaio, Skyrmions on the track, Nat. Nanotechnol. 8, 152 (2013).
  * Fert  _et al._ [2017] A. Fert, N. Reyren, and V. Cros, Skyrmion qubits: Challenges for future quantum computing applications, Nature Reviews Materials 2, 17031 (2017).
  * Bhowal and Spaldin [2022] S. Bhowal and N. A. Spaldin, Magnetoelectric classification of skyrmions, Phys. Rev. Lett. 128, 227204 (2022).
  * Zhang  _et al._ [2020] X. Zhang, Y. Zhou, K. M. Song, T.-E. Park, J. Xia, M. Ezawa, X. Liu, W. Zhao, G. Zhao, and S. Woo, Skyrmion-electronics: writing, deleting, reading and processing magnetic skyrmions toward spintronic applications, J. Phys.: Condens. Matter 32, 143001 (2020).
  * Amoroso  _et al._ [2020] D. Amoroso, P. Barone, and S. Picozzi, Spontaneous skyrmionic lattice from anisotropic symmetric exchange in a ni-halide monolayer, Nature Communications 11, 5784 (2020).
  * Yang  _et al._ [2021] S.-H. Yang, R. Naaman, Y. Paltiel, and S. S. P. Parkin, Chiral spintronics, Nature Reviews Physics 3, 328 (2021).
  * Psaroudaki and Panagopoulos [2021] C. Psaroudaki and C. Panagopoulos, Skyrmion qubits: A new class of quantum logic elements based on nanoscale magnetization, Phys. Rev. Lett. 127, 067201 (2021).
  * Psaroudaki  _et al._ [2023] C. Psaroudaki, E. Peraticos, and C. Panagopoulos, Skyrmion qubits: Challenges for future quantum computing applications, Appl. Phys. Lett. 123, 260501 (2023).
  * Szilva  _et al._ [2023] A. Szilva, Y. Kvashnin, E. A. Stepanov, L. Nordström, O. Eriksson, A. I. Lichtenstein, and M. I. Katsnelson, Quantitative theory of magnetic interactions in solids, Rev. Mod. Phys. 95, 035004 (2023).
  * Heisenberg [1928] W. Heisenberg, Zur Theorie des Ferromagnetismus, Z. Phys. 49, 619 (1928).
  * Dzyaloshinsky [1958] I. Dzyaloshinsky, A thermodynamic theory of “weak” ferromagnetism of antiferromagnetics, J. Phys. Chem. Solids 4, 241 (1958).
  * Moriya [1960] T. Moriya, Anisotropic superexchange interaction and weak ferromagnetism, Phys. Rev. 120, 91 (1960).
  * Vaz  _et al._ [2008] C. A. F. Vaz, J. A. C. Bland, and G. Lauhoff, Magnetism in ultrathin film structures, Rep. Prog. Phys. 71, 056501 (2008).
  * Dieny and Chshiev [2017] B. Dieny and M. Chshiev, Perpendicular magnetic anisotropy at transition metal/oxide interfaces and applications, Rev. Mod. Phys. 89, 025008 (2017).
  * Heinze  _et al._ [2011] S. Heinze, K. von Bergmann, , M. Menzel, J. Brede, A. Kubetzka, R. Wiesendanger, G. Bihlmayer, and S. Blügel, Spontaneous atomic-scale magnetic skyrmion lattice in two dimensions, Nat. Phys. 7, 713 (2011).
  * Makhfudz  _et al._ [2012] I. Makhfudz, B. Krüger, and O. Tchernyshyov, Inertia and chiral edge modes of a skyrmion magnetic bubble, Phys. Rev. Lett. 109, 217201 (2012).
  * Schütte  _et al._ [2014] C. Schütte, J. Iwasaki, A. Rosch, and N. Nagaosa, Inertia, diffusion, and dynamics of a driven skyrmion, Phys. Rev. B 90, 174434 (2014).
  * Psaroudaki  _et al._ [2017] C. Psaroudaki, S. Hoffman, J. Klinovaja, and D. Loss, Quantum Dynamics of Skyrmions in Chiral Magnets, Phys. Rev. X 7, 041045 (2017).
  * Landau and Lifshitz [1935] L. D. Landau and E. Lifshitz, On the theory of the dispersion of magnetic permeability in ferromagnetic bodies, Phys. Z. Sowjet. 8, 153 (1935).
  * Gilbert [2004] T. Gilbert, A phenomenological theory of damping in ferromagnetic materials, IEEE Transactions on Magnetics 40, 3443 (2004).
  * Thiele [1973] A. A. Thiele, Steady-State Motion of Magnetic Domains, Phys. Rev. Lett. 30, 230 (1973).
  * Ozawa  _et al._ [2017] R. Ozawa, S. Hayami, and Y. Motome, Zero-field skyrmions with a high topological number in itinerant magnets, Phys. Rev. Lett. 118, 147205 (2017).
  * Sotnikov  _et al._ [2021] O. M. Sotnikov, V. V. Mazurenko, J. Colbois, F. Mila, M. I. Katsnelson, and E. A. Stepanov, Probing the topology of the quantum analog of a classical skyrmion, Phys. Rev. B 103, L060404 (2021).
  * Mazurenko  _et al._ [2023] V. V. Mazurenko, I. A. Iakovlev, O. M. Sotnikov, and M. I. Katsnelson, Estimating patterns of classical and quantum skyrmion states, J. Phys. Soc. Jpn 92, 081004 (2023).
  * Haller  _et al._ [2024] A. Haller, S. A. Díaz, W. Belzig, and T. L. Schmidt, Quantum magnetic skyrmion operator (2024), arXiv:2403.10347 [cond-mat.str-el] .
  * Peters  _et al._ [2023] R. Peters, J. Neuhaus-Steinmetz, and T. Posske, Quantum skyrmion hall effect in $f$-electron systems, Phys. Rev. Res. 5, 033180 (2023).
  * Cirac  _et al._ [2021] J. I. Cirac, D. Pérez-García, N. Schuch, and F. Verstraete, Matrix product states and projected entangled pair states: Concepts, symmetries, theorems, Rev. Mod. Phys. 93, 045003 (2021).
  * Lohani  _et al._ [2019] V. Lohani, C. Hickey, J. Masell, and A. Rosch, Quantum skyrmions in frustrated ferromagnets, Phys. Rev. X 9, 041063 (2019).
  * Haller  _et al._ [2022] A. Haller, S. Groenendijk, A. Habibi, A. Michels, and T. L. Schmidt, Quantum skyrmion lattices in heisenberg ferromagnets, Phys. Rev. Res. 4, 043113 (2022).
  * Joshi  _et al._ [2024] A. Joshi, R. Peters, and T. Posske, Quantum skyrmion dynamics studied by neural network quantum states (2024), arXiv:2403.08184 [cond-mat.dis-nn] .
  * Hellman  _et al._ [2017] F. Hellman, A. Hoffmann, Y. Tserkovnyak, G. S. D. Beach, E. E. Fullerton, C. Leighton, A. H. MacDonald, D. C. Ralph, D. A. Arena, H. A. Dürr, P. Fischer, J. Grollier, J. P. Heremans, T. Jungwirth, A. V. Kimel, B. Koopmans, I. N. Krivorotov, S. J. May, A. K. Petford-Long, J. M. Rondinelli, N. Samarth, I. K. Schuller, A. N. Slavin, M. D. Stiles, O. Tchernyshyov, A. Thiaville, and B. L. Zink, Interface-induced phenomena in magnetism, Rev. Mod. Phys. 89, 025006 (2017).
  * Viñas Boström and Verdozzi [2019] E. Viñas Boström and C. Verdozzi, Steering magnetic skyrmions with currents: A nonequilibrium green’s functions approach, physica status solidi (b) 256, 1800590 (2019).
  * Reyes-Osorio and Nikolic [2023] F. Reyes-Osorio and B. K. Nikolic, Anisotropic skyrmion mass induced by surrounding conduction electrons: A Schwinger-Keldysh field theory approach, arXiv.2302.04220 10.48550/arXiv.2302.04220 (2023).
  * Reyes-Osorio and Nikolić [2024] F. Reyes-Osorio and B. K. Nikolić, Gilbert damping in metallic ferromagnets from Schwinger-Keldysh field theory: Intrinsically nonlocal, nonuniform, and made anisotropic by spin-orbit coupling, Phys. Rev. B 109, 024413 (2024).
  * Ghosh  _et al._ [2022] S. Ghosh, F. Freimuth, O. Gomonay, S. Blügel, and Y. Mokrousov, Driving spin chirality by electron dynamics in laser-excited antiferromagnets, Communications Physics 5, 69 (2022).
  * Ghosh  _et al._ [2023] S. Ghosh, S. Blügel, and Y. Mokrousov, Ultrafast optical generation of antiferromagnetic meron-antimeron pairs with conservation of topological charge, Phys. Rev. Res. 5, L022007 (2023).
  * Viñas Boström  _et al._ [2022] E. Viñas Boström, A. Rubio, and C. Verdozzi, Microscopic theory of light-induced ultrafast skyrmion excitation in transition metal films, npj Computational Materials 8, 62 (2022).
  * Sahu  _et al._ [2022] P. Sahu, B. R. K. Nanda, and S. Satpathy, Formation of the skyrmionic polaron by rashba and dresselhaus spin-orbit coupling, Phys. Rev. B 106, 224403 (2022).
  * Wang and Batista [2023] Z. Wang and C. D. Batista, Skyrmion crystals in the triangular Kondo lattice model, SciPost Phys. 15, 161 (2023).
  * Östberg  _et al._ [2023] E. Östberg, E. Viñas Boström, and C. Verdozzi, Microscopic theory of current-induced skyrmion transport and its application in disordered spin textures, Frontiers in Physics 11 (2023).
  * Mondal  _et al._ [2021] P. Mondal, A. Suresh, and B. K. Nikolić, When can localized spins interacting with conduction electrons in ferro- or antiferromagnets be described classically via the landau-lifshitz equation: Transition from quantum many-body entangled to quantum-classical nonequilibrium states, Phys. Rev. B 104, 214401 (2021).
  * Manchon  _et al._ [2015] A. Manchon, H. C. Koo, J. Nitta, S. M. Frolov, and R. A. Duine, New perspectives for Rashba spin–orbit coupling, Nature Materials 14, 871 (2015).
  * Rowland  _et al._ [2016] J. Rowland, S. Banerjee, and M. Randeria, Skyrmions in chiral magnets with rashba and dresselhaus spin-orbit coupling, Phys. Rev. B 93, 020404(R) (2016).
  * Fannes  _et al._ [1992] M. Fannes, B. Nachtergaele, and R. F. Werner, Finitely correlated states on quantum spin chains, Commun. Math. Phys. 144, 443 (1992).
  * Östlund and Rommer [1995] S. Östlund and S. Rommer, Thermodynamic limit of density matrix renormalization, Phys. Rev. Lett. 75, 3537 (1995).
  * Rommer and Östlund [1997] S. Rommer and S. Östlund, Class of ansatz wave functions for one-dimensional spin systems and their relation to the density matrix renormalization group, Phys. Rev. B 55, 2164 (1997).
  * Fishman  _et al._ [2022] M. Fishman, S. R. White, and E. M. Stoudenmire, The ITensor Software Library for Tensor Network Calculations, SciPost Phys. Codebases , 4 (2022).
  * Paeckel  _et al._ [2019] S. Paeckel, T. Köhler, A. Swoboda, S. R. Manmana, U. Schollwöck, and C. Hubig, Time-evolution methods for matrix-product states, Annals of Physics 411, 167998 (2019).
  * Kadanoff and Baym [1962] L. P. Kadanoff and G. Baym,  _Quantum statistical mechanics : Green’s function methods in equilibrium and nonequilibrium problems_ (W.A. Benjamin, 1962).
  * Keldysh [1964] L. V. Keldysh, Diagram technique for nonequilibrium processes, Sov. Phys. JETP 20, 1018 (1964).
  * Balzer and Bonitz [2012] K. Balzer and M. Bonitz,  _Nonequilibrium Green’s Functions Approach to Inhomogeneous Systems_ (Springer Berlin, Heidelberg, 2012).
  * Stefanucci and van Leeuwen [2013] G. Stefanucci and R. van Leeuwen,  _Nonequilibrium Many-Body Theory of Quantum Systems: A Modern Introduction_ (Cambridge University Press, 2013).
  * Latini  _et al._ [2014] S. Latini, E. Perfetto, A.-M. Uimonen, R. van Leeuwen, and G. Stefanucci, Charge dynamics in molecular junctions: Nonequilibrium green’s function approach made fast, Phys. Rev. B 89, 075306 (2014).
  * Zhang  _et al._ [2023] S. Zhang, X. Li, H. Zhang, P. Cui, X. Xu, and Z. Zhang, Giant Dzyaloshinskii-Moriya interaction, strong XXZ-type biquadratic coupling, and bimeronic excitations in the two-dimensional CrMnI6 magnet, npj Quantum Materials 8, 38 (2023).
  * Wootters [1998] W. K. Wootters, Entanglement of formation of an arbitrary state of two qubits, Phys. Rev. Lett. 80, 2245 (1998).
  * Horodecki  _et al._ [2009] R. Horodecki, P. Horodecki, M. Horodecki, and K. Horodecki, Quantum entanglement, Rev. Mod. Phys. 81, 865 (2009).
  * Huggins  _et al._ [2019] W. Huggins, P. Patil, B. Mitchell, K. B. Whaley, and E. M. Stoudenmire, Towards quantum machine learning with tensor networks, Quantum Sci. and Technol. 4, 024001 (2019).
  * Shinaoka  _et al._ [2023] H. Shinaoka, M. Wallerberger, Y. Murakami, K. Nogaki, R. Sakurai, P. Werner, and A. Kauch, Multiscale space-time ansatz for correlation functions of quantum systems based on quantics tensor trains, Phys. Rev. X 13, 021015 (2023).
  * Pavlyukh  _et al._ [2022] Y. Pavlyukh, E. Perfetto, D. Karlsson, R. van Leeuwen, and G. Stefanucci, Time-linear scaling nonequilibrium Green’s function methods for real-time simulations of interacting electrons and bosons. I. Formalism, Phys. Rev. B 105, 125134 (2022).