# LLM-Pruner: On the Structural Pruning
of Large Language Models

Xinyin Ma Gongfan Fang Xinchao Wang
National University of Singapore
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Corresponding author

###### Abstract

Large language models (LLMs) have shown remarkable capabilities in language
understanding and generation. However, such impressive capability typically
comes with a substantial model size, which presents significant challenges in
both the deployment, inference, and training stages. With LLM being a general-
purpose task solver, we explore its compression in a task-agnostic manner,
which aims to preserve the multi-task solving and language generation ability
of the original LLM. One challenge to achieving this is the enormous size of
the training corpus of LLM, which makes both data transfer and model post-
training over-burdensome. Thus, we tackle the compression of LLMs within the
bound of two constraints: being task-agnostic and minimizing the reliance on
the original training dataset. Our method, named LLM-Pruner, adopts structural
pruning that selectively removes non-critical coupled structures based on
gradient information, maximally preserving the majority of the LLM’s
functionality. To this end, the performance of pruned models can be
efficiently recovered through tuning techniques, LoRA, in merely _3 hours_ ,
requiring only _50K_ data. We validate the LLM-Pruner on three LLMs, including
LLaMA, Vicuna, and ChatGLM, and demonstrate that the compressed models still
exhibit satisfactory capabilities in zero-shot classification and generation.
The code is available at: https://github.com/horseee/LLM-Pruner

##  1 Introduction

Recently, Large Language Models (LLMs) OpenAI (2023); Touvron et al. (2023);
Thoppilan et al. (2022); Scao et al. (2022); Xue et al. (2020); Chiang et al.
(2023); Zeng et al. (2022) have demonstrated remarkable proficiency in
language understanding and generation. With the increase in model size, they
are better equipped to handle complex tasks Brown et al. (2020); Chowdhery et
al. (2022); Wei et al. (2022b) and even exhibit emergent abilities Wei et al.
(2022a). However, notwithstanding their impressive performance, LLMs pose
challenges in deployment and inference. Their extensive scale engenders
substantial computational demands, and the multitude of parameters involved
can induce long latencies and other related issues. Several techniques are
proposed to solve these problems, like model pruning Wang et al. (2019b); Xia
et al. (2022); Zafrir et al. (2021); Kurtic et al. (2022), knowledge
distillation Sun et al. (2019); Pan et al. (2020b); Sun et al.
(2020a),quantization Bai et al. (2020); Frantar et al. (2022) within the
context of pre-trained language model (PLM).

While previous methods have effectively maintained model performance amidst
parameter reduction, they primarily target compression within specialized
domains or for designated tasks in the context of task-specific compression.
For instance, a PLM is fine-tuned on a particular dataset, such as one of the
classification tasks in the GLUE benchmark Wang et al. (2018), after which
these models are distilled into a smaller classification model Sun et al.
(2019); Hou et al. (2020). Although this paradigm could potentially be
employed for LLM compression, it compromises the LLM’s capacity as a versatile
task solver, rendering it suited to a single task exclusively.

Thus, we strive to compress the LLM in a new setting: to reduce the LLM’s size
while preserving its diverse capabilities as general-purpose task solvers, as
depicted in Figure 1. This introduces the task-agnostic compression of LLMs,
which presents two key challenges:

  * •

The size of the training corpus of the LLM is enormous. Previous compression
methods heavily depend on the training corpus. The LLM has escalated the
corpus scale to 1 trillion tokens or more Hoffmann et al. (2022); Touvron et
al. (2023). The extensive storage needs and protracted transmission times make
the dataset difficult to acquire. Furthermore, if the dataset is proprietary,
acquisition of the training corpus verges on impossibility, a situation
encountered in Zeng et al. (2022); OpenAI (2023).

  * •

The unacceptably long duration for the post-training of the pruned LLM.
Existing methods require a substantial amount of time for post-training the
smaller model Wang et al. (2020); Liang et al. (2023). For instance, the
general distillation in TinyBERT takes around 14 GPU days Jiao et al. (2020).
Even post-training a task-specific compressed model of BERT demands around 33
hours Xia et al. (2022); Kwon et al. (2022). As the size of both the model and
corpus for LLMs increases rapidly, this step will invariably consume an even
more extensive time.

Figure 1:  Illustration of LLM-Pruner. (i) Task-specific compression: the
model is fine-tuned then compressed on a specific task. (ii) TinyBERT: First
distill the model on unlabeled corpus and then fine-tune it on the specific
task. (iii) LLM-Pruner: Task-agnostic compression within 3 hours.

To tackle the aforementioned challenges associated with the task-agnostic
compression of LLMs, we introduce a novel approach called LLM-Pruner. Since
our goal is to compress LLMs with reduced data dependency and expedited post-
training, how to prune model with the minimal disruption to the origin is
crucial. To accomplish this, we propose a dependency detection algorithm that
identifies all the dependent structures within the model. Once the coupled
structure is identified, we employ an efficient importance estimation strategy
to select the optimal group for pruning under the task-agnostic setting, where
the first-order information and an approximated hessian information is taken
into account. Finally, a rapid recovery stage is executed to post-train the
pruned model with limited data.

#### Contribution.

In this paper, we propose a novel framework, LLM-Pruner, for the task-agnostic
compression of the large language model. To the best of our knowledge, LLM-
Pruner is the first framework designed for structured pruning of LLMs. We
conclude the advantages of the LLM-Pruner as (i) Task-agnostic compression,
where the compressed language model retains its ability to serve as a multi-
task solver. (ii) Reduced demand for the original training corpus, where only
50k publicly available samples are needed for compression, significantly
reducing the budget for acquiring the training data (iii) Quick compression,
where the compression process ends up in three hours. (iv) An automatic
structural pruning framework, where all the dependent structures are grouped
without the need for any manual design. To evaluate the effectiveness of LLM-
Pruner, we conduct extensive experiments on three large language models:
LLaMA-7B, Vicuna-7B, and ChatGLM-6B. The compressed models are evaluated using
nine datasets to assess both the generation quality and the zero-shot
classification performance of the pruned models. The experimental results
demonstrate that even with the removal of 20% of the parameters, the pruned
model maintains 93.6% of the performance of the original model.

##  2 Related Work

#### Compression of Language Model.

Language models Devlin et al. (2018); Liu et al. (2019); Lewis et al. (2019)
have gained much attention and increase the need to reduce the size of
parameters and reduce the latency Lan et al. (2019); Sun et al. (2020b). To
compress the language model, previous works can be divided into several
categories: network pruning Kurtic et al. (2022); Xu et al. (2021); Liu et al.
(2021); Guo et al. (2019), knowledge distillation Sun et al. (2019, 2020a);
Pan et al. (2020a), quantization Yao et al. (2022); Bai et al. (2020); Zafrir
et al. (2019) and other techniques, like early exit Xin et al. (2020) or
dynamic token reduction Ye et al. (2021). We focus on the pruning of the
language models, especially structural pruning Li et al. (2016). Structural
pruning removes the entire filter from the neural network, which is more
hardware friendly. There are several ways to remove the structure, such as
l1-dependent pruning Han et al. (2015); Zafrir et al. (2021), first-order
importance estimation Hou et al. (2020), hessian-based estimation Kurtic et
al. (2022); Wang et al. (2019a) or the optimal brain surgeon LeCun et al.
(1989); Kurtic et al. (2022). As for the pruning unit in structural pruning,
some works adopt the entire layer Fan et al. (2019) as the minimal unit, and
others take the multi-head attention Voita et al. (2019) or the feed-forward
layers Hou et al. (2020); McCarley et al. (2019) as the basic structure to
prune. CoFi Xia et al. (2022) studies the pruning unit in different
granularity.

#### Efficient and Low Resource Compression.

With the growing size of models, there is an increasing demand for efficient
LLM compression and compression is independent of the original training data.
As for the efficient compression, Kwon et al. (2022) accelerate the post-
training by defining the reconstruction error as a linear least squares
problem. Frantar et al. (2022); Frantar and Alistarh (2023) propose the layer-
wise optimal brain surgeon. As for the constraint of availability of the
training corpus, data-free pruning Srinivas and Babu (2015); Yvinec et al.
(2022) come up with several strategies to prune the model by measuring
neurons’ similarity. Besides, Ma et al. (2022, 2020); Rashid et al. (2020)
proposes methods that distill the model without reliance on the training
corpus of the model. However, those methods are too time-consuming, involving
synthesizing samples by backpropagating the pre-trained language models.

##  3 Methods

In this section, we provide a detailed explanation of LLM-Pruner. Following
the conventional model compression pipelineKwon et al. (2022), LLM-Pruner
consists of three steps: (1) Discovery Stage (Section 3.1). This step focuses
on identifying groups of interdependent structures within LLMs. (2) Estimation
Stage (Section 3.2). Once the coupled structures are grouped, the second step
entails estimating the contribution of each group to the overall performance
of the model and deciding which group to be pruned. (3) Recover Stage (Section
3.3). This step involves fast post-training that alleviates potential
performance degradation caused by the removal of structures.

###  3.1 Discover All Coupled Structure in LLMs

In light of the limited availability of data for post-training, it becomes
imperative to prioritize the removal of structures with minimal damage when
compressing the model. This underscores the dependency-based structural
pruning, which ensures coupled structures are pruned in unison. We provide an
experiment in Section 4.3 to show the importance of dependency-based
structural pruning when compressing the large language model.

#### Structure Dependency in LLMs.

Similar to Fang et al. (2023), the pruning begins by building the dependency
for LLMs. Assume $N_{i}$ and $N_{j}$ are two neurons in the model,
$\operatorname{In}(N_{i})$ and $\operatorname{Out}(N_{i})$ represents all the
neurons that point towards or point from $N_{i}$. The dependency between
structures can be defined as:

$N_{j}\in\operatorname{Out}(N_{i})\wedge\operatorname{Deg}^{-}(N_{j})=1\Rightarrow
N_{j}\text{ is dependent on }N_{i}$ (1)

where $\operatorname{Deg}^{-}(N_{j})$ represents the in-degree of neuron
$N_{j}$. Noting that this dependency is directional, we can therefore
correspondingly obtain another dependency:

$N_{i}\in\operatorname{In}(N_{j})\wedge\operatorname{Deg}^{+}(N_{i})=1\Rightarrow
N_{i}\text{ is dependent on }N_{j}$ (2)

where $\operatorname{Deg}^{+}(N_{i})$ represents the out-degree of neuron
$N_{i}$. The principle of dependency here is, if a current neuron (e.g.,
$N_{i}$) depends solely on another neuron (e.g., $N_{j}$), and the neuron
$N_{j}$ is subjected to pruning, it follows that the neuron $N_{i}$ must also
undergo pruning.

#### Trigger the Dependency Graph.

By having the definition of dependency, the coupled structures in the LLM can
be analyzed automatically. Considering any neuron within the LLM as the
initial trigger, it possesses the capability to activate neurons that depend
on it. Subsequently, these newly triggered neurons can serve as the subsequent
triggers to identify the dependency and activate their respective dependent
neurons. This iterative process continues until no new neurons are detected.
Those neurons then form a group for further pruning. Taking LLaMA as an
example, by searching over all the neurons as the initial trigger, we can
locate all the coupled structures, as shown in Figure2.

Given the diversity in the structure of different LLMs, manual analysis and
removal of coupled structures in each LLM could be extremely time-consuming.
However, by employing LLM-Pruner, all coupled structures can be automatically
identified and extracted.

Figure 2:  Illustration of the coupled structures in LLaMA. We simplify the
neurons in each layer to make the dependent group clear. The trigger neuron,
marked as a circle with a bell, cause weights with dependency pruned (dashed
lines), which may propagate (red dashed lines) to coupled neurons (dashed
circles). A group can be triggered by a variety of trigger neurons. Taking
Group B as an example, the trigger for this group involves (i) the attention
head, (ii) the output neuron in Query, Key or Value, and (iii) the input
neuron in the final output projection.

###  3.2 Grouped Importance Estimation of Coupled Structure

Till now, all coupled structures are grouped within the model. Weights within
the same group should be pruned simultaneously, as partial pruning not only
increases parameter size but also introduces inaccurate intermediate
representations into the model. Therefore, we estimate the importance of a
group as a whole, as opposed to evaluating the importance of individual
structures. Given the limited access to the training dataset, we explore the
use of available public datasets or manually created samples as alternative
resources. Although the domains of these datasets may not perfectly align with
the training set, they still provide valuable information for assessing the
importance of the structural groups.

#### Weight Importance.

Suppose that given a dataset $\mathcal{D}=\\{x_{i},y_{i}\\}_{i=1}^{N}$, where
N is the number of samples. In our experiments, we set N equal to 10 and we
use some public datasets as the source of $\mathcal{D}$. A group (as
previously defined as a set of coupled structures) can be defined as
$\mathcal{G}=\\{W_{i}\\}_{i=1}^{M}$, where M is the number of coupled
structures in one group and $W_{i}$ is the weight for each structure. While
pruning, our goal is to remove the structure that has the least impact on the
model’s prediction, which can be indicated by the deviation in the loss.
Specially, to estimate the importance of $W_{i}$, the change in loss can be
formulated as LeCun et al. (1989):

$I_{W_{i}}=|\Delta\mathcal{L}(\mathcal{D})|=|\mathcal{L}_{W_{i}}(\mathcal{D})-\mathcal{L}_{W_{i}=0}(\mathcal{D})|=|\underbrace{\frac{\partial\mathcal{L}^{\top}(\mathcal{D})}{\partial
W_{i}}W_{i}}_{\neq
0}+\frac{1}{2}{W_{i}}^{\top}HW_{i}+\mathcal{O}\left(\|W_{i}\|^{3}\right)|$ (3)

where $H$ is the hessian matrix. Here, $\mathcal{L}$ represents the next-token
prediction loss. The first term is typically neglected in prior work LeCun et
al. (1989); Wang et al. (2019a); Frantar and Alistarh (2023), as the model has
already converged on the training dataset, where
${\partial\mathcal{L}^{\top}}/{\partial W_{i}}\approx 0$. However, since
$\mathcal{D}$ here is not extracted from the original training data, which
means that ${\partial\mathcal{L}^{\top}}/{\partial W_{i}}\not\approx 0$. This
presents a desirable property for determining the importance of $W_{i}$ by the
gradient term under LLMs, since computation of the second term, the Hessian
matrix, on the LLM is impractical with $\mathcal{O}\left(N^{2}\right)$
complexity.

#### Parameter Importance.

The above can be considered as an estimate for the weight $W_{i}$. We can
derive another measure of importance at a finer granularity, where each
parameter within $W_{i}$ is assessed for its significance:

$I_{W_{i}^{k}}=|\Delta\mathcal{L}(\mathcal{D})|=|\mathcal{L}_{W_{i}^{k}}(\mathcal{D})-\mathcal{L}_{W_{i}^{k}=0}(\mathcal{D})|=|\frac{\partial\mathcal{L}(\mathcal{D})}{\partial
W_{i}^{k}}W_{i}^{k}+\frac{1}{2}{W_{i}^{k}}H_{kk}W_{i}^{k}+\mathcal{O}\left(\|W_{i}^{k}\|^{3}\right)|$
(4)

Here, $k$ represents the k-th parameter in $W_{i}$. The diagonal of the
hessian $H_{kk}$ can be approximated by the Fisher information matrix, and the
importance can be defined as:

$I_{W_{i}^{k}}=|\mathcal{L}_{W_{i}^{k}}(\mathcal{D})-\mathcal{L}_{W_{i}^{k}=0}(\mathcal{D})|\approx|\frac{\partial\mathcal{L}(\mathcal{D})}{\partial
W_{i}^{k}}W_{i}^{k}+\frac{1}{2}\left(\frac{\partial\mathcal{L}(\mathcal{D})}{\partial
W_{i}^{k}}W_{i}^{k}\right)^{2}+\mathcal{O}\left(\|W_{i}^{k}\|^{3}\right)|$ (5)

#### Group Importance.

By utilizing either $I_{W_{i}^{k}}$ or $I_{W_{i}}$, we estimate the importance
at the granularity of either a parameter or a weight. Remembering that our
goal is to estimate the importance of $\mathcal{G}$, we aggregate the
importance scores in four ways: (i) Summation:
$I_{\mathcal{G}}=\sum_{i=1}^{M}I_{W_{i}}$ or
$I_{\mathcal{G}}=\sum_{i=1}^{M}\sum_{k}I_{W_{i}^{k}}$, (ii) Production:
$I_{\mathcal{G}}=\prod_{i=1}^{M}I_{W_{i}}$ or
$I_{\mathcal{G}}=\prod_{i=1}^{M}\sum_{k}I_{W_{i}^{k}}$, (iii) Max:
$I_{\mathcal{G}}=\max_{i=1}^{M}I_{W_{i}}$ or
$I_{\mathcal{G}}=\max_{i=1}^{M}\sum_{k}I_{W_{i}^{k}}$; (iv) Last-Only: Since
deleting the last executing structure in a dependency group is equivalent to
erasing all the computed results within that group, we assign the importance
of the last executing structure as the importance of the group:
$I_{\mathcal{G}}=I_{W_{l}}$ or $I_{\mathcal{G}}=\sum_{k}I_{W_{l}^{k}}$, where
$l$ is the last structure. After assessing the importance of each group, we
rank the importance of each group and prune the groups with lower importance
based on a predefined pruning ratio.

###  3.3 Fast Recovery with Low-rank Approximation

In order to expedite the model recovery process and improve its efficiency
under limited data, it is crucial to minimize the number of parameters that
need optimization during the recovery phase. To facilitate this, we employ the
low-rank approximation, LoRAHu et al. (2021), to post-train the pruned model.
Each learnable weight matrix in the model, denoted as $W$, encompassing both
pruned and unpruned linear projection in the LLM, can be represented as $W$.
The update value $\Delta W$ for $W$ can be decomposed as $\Delta
W=PQ\in\mathbb{R}^{d^{-}\times d^{+}}$, where $P\in\mathbb{R}^{d^{-}\times d}$
and $Q\in\mathbb{R}^{d\times d^{+}}$. The forward computation can now be
expressed as:

$f(x)=(W+\Delta W)X+b=(WX+b)+(PQ)X$ (6)

where $b$ is the bias in the dense layer. Only training $P$ and $Q$ reduces
the overall training complexity, reducing the need for large-scale training
data. Besides, the extra parameters $P$ and $Q$ can be reparameterized into
$\Delta W$, which would not cause extra parameters in the final compressed
model.

Table 1: Zero-shot performance of the compressed LLaMA-7B. The average is
calculated among seven classification datasets. ‘Underline’ indicates the best
pruning-only performance, while ‘bold’ represents the overall best performance
with the same pruning ratio, considering both pruning and post-training.
Method ‘Channel’ utilizes the pruning group of ‘Channel’ in Table 2. For all
other methods, the ‘Block’ group is used. Since Touvron et al. (2023) did not
provide its prompt, the evaluation of the result with ⋆ is performed under
different prompts, which is lower than the official results.

Pruning Ratio | Method | WikiText2$\color[rgb]{0,.5,.5}\downarrow$ | PTB$\color[rgb]{0,.5,.5}\downarrow$ | BoolQ | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Average
---|---|---|---|---|---|---|---|---|---|---|---
Ratio = 0% | LLaMA-7BTouvron et al. (2023) | - | - | 76.5 | 79.8 | 76.1 | 70.1 | 72.8 | 47.6 | 57.2 | 68.59
LLaMA-7B⋆ | 12.62 | 22.14 | 73.18 | 78.35 | 72.99 | 67.01 | 67.45 | 41.38 | 42.40 | 63.25
Ratio = 20% w/o tune | l2 | 582.41 | 1022.17 | 59.66 | 58.00 | 37.04 | 52.41 | 33.12 | 28.58 | 29.80 | 42.65
random | 27.51 | 43.19 | 61.83 | 71.33 | 56.26 | 54.46 | 57.07 | 32.85 | 35.00 | 52.69
Channel | 74.63 | 153.75 | 62.75 | 62.73 | 41.40 | 51.07 | 41.38 | 27.90 | 30.40 | 45.38
Weight | 22.28 | 41.78 | 61.44 | 71.71 | 57.27 | 54.22 | 55.77 | 33.96 | 38.40 | 53.25
$\text{Parameter}^{2}$ | 20.89 | 36.66 | 62.05 | 73.88 | 57.80 | 54.85 | 52.86 | 32.08 | 36.80 | 52.90
$\text{Parameter}^{1}$ | 19.09 | 34.21 | 57.06 | 75.68 | 66.80 | 59.83 | 60.94 | 36.52 | 40.00 | 56.69
$\text{Parameter}^{1+2}$ | 19.24 | 34.09 | 62.54 | 75.41 | 65.99 | 60.30 | 61.57 | 36.69 | 39.20 | 57.39
Ratio = 20% w/ tune | Channel | 22.02 | 38.67 | 59.08 | 73.39 | 64.02 | 60.54 | 57.95 | 35.58 | 38.40 | 55.57
Weight | 18.84 | 33.05 | 65.75 | 74.70 | 64.52 | 59.35 | 60.65 | 36.26 | 39.40 | 57.23
$\text{Parameter}^{2}$ | 18.07 | 30.60 | 64.83 | 76.28 | 65.53 | 59.75 | 61.53 | 36.01 | 38.20 | 57.45
$\text{Parameter}^{1}$ | 17.58 | 30.11 | 64.62 | 77.20 | 68.80 | 63.14 | 64.31 | 36.77 | 39.80 | 59.23
$\text{Parameter}^{1+2}$ | 17.39 | 30.20 | 66.79 | 77.58 | 68.48 | 64.96 | 64.06 | 37.88 | 39.00 | 59.82
Ratio = 50% w/o tune | l2 | 39266.42 | 48867.85 | 55.11 | 53.59 | 27.03 | 49.49 | 26.43 | 29.01 | 34.40 | 39.29
random | 3887.90 | 4337.27 | 46.79 | 53.37 | 27.50 | 50.59 | 28.07 | 27.90 | 30.00 | 37.75
Channel | 13891.92 | 16114.91 | 40.37 | 52.18 | 25.72 | 48.86 | 25.72 | 28.24 | 30.40 | 35.93
Weight | 141.06 | 236.24 | 62.17 | 55.11 | 27.25 | 49.88 | 29.00 | 25.77 | 34.00 | 40.45
$\text{Parameter}^{2}$ | 115.58 | 278.36 | 59.72 | 59.47 | 27.40 | 50.12 | 29.12 | 24.57 | 31.00 | 40.20
$\text{Parameter}^{1}$ | 112.44 | 255.38 | 52.32 | 59.63 | 35.64 | 53.20 | 33.50 | 27.22 | 33.40 | 42.13
$\text{Parameter}^{1+2}$ | 130.97 | 302.15 | 48.56 | 59.96 | 33.98 | 52.09 | 34.30 | 26.88 | 34.00 | 41.40
Ratio = 50% w/ tune | Channel | 1122.15 | 1092.26 | 40.76 | 54.84 | 26.94 | 49.41 | 27.86 | 25.43 | 32.20 | 36.77
Weight | 43.47 | 68.51 | 62.11 | 64.96 | 40.52 | 51.54 | 46.38 | 28.33 | 32.40 | 46.61
$\text{Parameter}^{2}$ | 40.13 | 72.75 | 60.37 | 66.97 | 42.31 | 52.01 | 46.17 | 26.54 | 34.60 | 47.00
$\text{Parameter}^{1}$ | 38.12 | 66.35 | 60.28 | 69.31 | 47.06 | 53.43 | 45.96 | 29.18 | 35.60 | 48.69
$\text{Parameter}^{1+2}$ | 39.02 | 71.45 | 61.56 | 68.72 | 46.62 | 52.64 | 47.94 | 29.27 | 35.40 | 48.88

##  4 Experiments

###  4.1 Experimental Settings

#### Foundation Large Language Model.

To showcase the effectiveness and versatility of LLM-Pruner, we test it over
three open-source large language models with two kinds of structure: LLaMA-7B
Touvron et al. (2023), Vicuna-7B Chiang et al. (2023) and ChatGLM-6B Zeng et
al. (2022).

#### Evaluation and Datasets.

To assess the performance of the model in the task-agnostic setting, we follow
LLaMa’s evaluation to perform zero-shot task classification on common sense
reasoning datasets: BoolQ Clark et al. (2019), PIQA Bisk et al. (2020),
HellaSwag Zellers et al. (2019), WinoGrande Sakaguchi et al. (2019), ARC-easy
Clark et al. (2018), ARC-challenge Clark et al. (2018) and OpenbookQA Mihaylov
et al. (2018). Follow Gao et al. (2021), the model ranks the choices in the
multiple choice tasks or generates the answer in the open-ended generation
111https://github.com/EleutherAI/lm-evaluation-harness. Additionally, we
complement our evaluation with a zero-shot perplexity (PPL) analysis on
Wikitext2 Merity et al. (2016) and PTB Marcus et al. (1993). However, for the
experiment on ChatGLM, we do not estimate the PPL on these two datasets. This
is attributed to the inherent nature of ChatGLM as a dialogue model, which
necessitates a query prompt and often recycles partial words from the query
within the response. This pattern of interaction diverges significantly from
the formats in Wikitext2 and PTB.

#### Implementation Details.

In the model pruning process, we use 10 randomly selected samples from
Bookcorpus Zhu et al. (2015), each truncated to a sequence length of 128, as
the input for establishing dependency and calculating the gradient for both
LLaMA and Vicuna. For ChatGLM, we select 10 random samples from DailyDialog Li
et al. (2017). During the recovery phase, we utilize the cleaned version of
Alpaca Taori et al. (2023), which comprises approximately 50k samples.
Remarkably, tuning these samples requires merely 3 hours on a single GPU with
only 2 epochs. More hyper-parameters of pruning and training can be found in
Appendix.

Table 2: Statistic for LLaMA, Vicuna and ChatGLM.

Model | Group | Ratio | #Params | #MACs | Memory
---|---|---|---|---|---
LLaMA-7B Vicuna-7B | - | - | 6.74B | 424.02G | 12884.5MiB
Channel | 20% | 5.47B | 323.79G | 10488.4MiB
Block | 20% | 5.42B | 367.52G | 10375.5MiB
Channel | 50% | 3.37B | 178.22G | 6556.3MiB
Block | 50% | 3.35B | 277.99G | 6533.9MiB
ChatGLM | - | - | 6.17B | 426.03G | 11775.3MiB
Block | 10% | 5.57B | 387.37G | 10622.8MiB

#### Statistics of the Compressed Model.

Table 2 presents the statistic of the models that are used in our experiments:
the parameter count, MACs, and memory requirements for running each model. The
statistical evaluation is conducted using the inference mode, where the model
is fed a sentence consisting of 64 tokens. Here, the ‘Block’ strategy implies
that the pruned unit in the model consists of Group A and Group B as
illustrated in Figure 2, whereas ‘Channel’ indicates that the unit to be
pruned is Group C. We delve into an analysis of these two choices in Section
4.2. The pruning ratio stated here denotes the ratio of parameters to be
pruned, and the pruning ratio of structural pruning may not align exactly with
the predefined ratio.

###  4.2 Zero-shot Performance

Table 1,3 and 4 shows the zero-shot performance of the pruned model. Based on
the evaluation conducted on LLaMA, employing a 20% parameter reduction without
post-training, the pruned model manages to retain 90.7% of the performance
exhibited by the unpruned model. Furthermore, through the efficient post-
training, the classification accuracy further improves to 59.23%, achieving
93.64% of the accuracy attained by the original model. This demonstration
proves the feasibility of using LLM-Pruner to effectively compress the model,
even without relying on training data, and within a remarkably short period of
time. Surprisingly, we discover that on most datasets, the pruned model with
5.4B LLaMA even outperformed chatGLM-6B. This highlights the superiority of
the LLM-Pruner: if a smaller model with a customized size is required, LLM-
Pruner is more cost-effective compared to retraining another model with a
satisfying performance. However, with 50% parameters pruned, a large accuracy
degradation is observed. Compressing LLMs under high compression rates still
remains a large challenge.

Table 3: Zero-shot performance of the compressed Vicuna-7B

Pruned Model | Method | WikiText2 $\color[rgb]{0,.5,.5}\downarrow$ | PTB$\color[rgb]{0,.5,.5}\downarrow$ | BoolQ | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Average
---|---|---|---|---|---|---|---|---|---|---|---
Ratio = 0% | Vicuna-7B | 16.11 | 61.37 | 76.57 | 77.75 | 70.64 | 67.40 | 65.11 | 41.21 | 40.80 | 62.78
Ratio = 20% w/o tune | l2 | 3539.98 | 5882.21 | 55.90 | 56.15 | 32.37 | 51.85 | 30.01 | 28.41 | 28.20 | 40.41
random | 34.63 | 112.44 | 61.47 | 70.89 | 54.67 | 56.27 | 55.60 | 31.74 | 34.60 | 52.18
Channel | 71.75 | 198.88 | 51.77 | 63.93 | 42.58 | 55.17 | 43.94 | 29.27 | 33.40 | 45.72
Weight | 27.03 | 92.51 | 62.17 | 71.44 | 55.80 | 53.43 | 55.77 | 33.28 | 37.80 | 52.81
$\text{Parameter}^{2}$ | 24.84 | 89.69 | 61.90 | 72.96 | 56.78 | 53.04 | 52.57 | 33.28 | 36.80 | 52.47
$\text{Parameter}^{1}$ | 25.74 | 92.88 | 61.70 | 75.30 | 63.75 | 56.20 | 63.22 | 36.60 | 37.00 | 56.25
$\text{Parameter}^{1+2}$ | 26.51 | 92.87 | 62.97 | 74.76 | 63.40 | 55.88 | 64.23 | 38.14 | 36.60 | 56.57
Ratio = 20% w/ tune | Weight | 19.94 | 74.66 | 63.15 | 74.59 | 61.95 | 60.30 | 60.48 | 36.60 | 39.40 | 56.64
$\text{Parameter}^{2}$ | 19.07 | 74.74 | 56.97 | 76.28 | 63.76 | 60.30 | 60.94 | 35.67 | 38.00 | 55.99
$\text{Parameter}^{1}$ | 19.69 | 78.25 | 63.33 | 76.17 | 65.13 | 60.22 | 62.84 | 37.12 | 39.20 | 57.71
$\text{Parameter}^{1+2}$ | 19.47 | 76.55 | 66.45 | 75.84 | 65.05 | 60.38 | 62.37 | 36.43 | 39.80 | 58.05

The compression results of Vicuna-7B align with those of LLaMA, as pruning 20%
of parameters on Vicuna-7B maintains performance at 92.47% of the original
model. We test a smaller pruning rate of 10% on chatGLM-7B, where the pruned
model only experiences a marginal performance decrease of 0.89%, which can be
recovered through post-training. Despite the pruned model outperforming the
uncompressed model, we don’t assert it is better than the original model. This
is largely because chatGLM-6B, a bilingual model, has limited English pre-
training exposure. Post-training, however, introduces it to more English
corpus, albeit limited, improving its English comprehension.

Figure 3: Layer sensitivity for Pruning: Removing Groups in only one layer.
The PPL of the first layer for Wikitext2 and PTB is 4474.9 and 7435.8
correspondingly.

#### Ablation: Impact of Importance Estimation.

We conduct tests on all proposed importance estimation techniques mentioned in
Section 3.2. Here, $\textit{Parameter}^{\text{n}}$ represents the importance
evaluation utilizing the n-th order term in Eq.5.
$\textit{Parameter}^{\text{1+2}}$ combines first-order and second-order
information, which corresponds to the precise calculation of Equation Eq.5.
Weight represents the result corresponding to Eq.3. Based on the results
obtained from LLaMA-7B and Vicuna-7B, pruning algorithms achieved the best
average performance by leveraging the first-order derivatives and the
approximated second-order information for each parameter. However, the results
on chatGLM-7B differed significantly from these findings. The importance
estimation on each parameter fails, performing even worse than l2, while the
importance estimation on the weight matrix reaches the best performance.

#### Channel Group vs. Block Group.

From the results presented in Table 1, it is evident that pruning ‘Channel’
significantly deteriorates performance compared to pruning ‘Block’. Our
conclusion is that this discrepancy arises because the layers within the
stacked transformer do not evenly distribute their importance. As shown in
Figure 3, the first and last layers have a profound impact on the model’s
performance, and pruning them results in more substantial performance
degradation compared to other layers. However, due to the uniform treatment of
the ‘Channel’ group across all layers, it becomes inevitable to prune the
first and last layers, leading to a significant decline in performance.

Table 4: Zero-shot Performance of the compressed ChatGLM-6B

Pruned Model | Method | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Average
---|---|---|---|---|---|---|---|---
Ratio = 0% | ChatGLM-6B | 67.95 | 46.37 | 52.33 | 48.36 | 29.95 | 37.40 | 47.05
Ratio = 10% w/o tune | l2 | 61.97 | 37.22 | 49.72 | 42.05 | 28.24 | 35.40 | 42.43
random | 65.29 | 43.18 | 51.30 | 47.52 | 29.52 | 34.60 | 45.24
Weight | 66.32 | 43.51 | 53.04 | 47.56 | 30.72 | 35.80 | 46.16
$\text{Parameter}^{1}$ | 54.35 | 28.07 | 50.59 | 27.82 | 24.66 | 33.20 | 36.45
w/ tune | Weight | 67.74 | 46.35 | 53.99 | 51.01 | 29.95 | 35.00 | 47.34

Figure 4: The pruning results on LLaMA-7B (left) and Vicuna-7B (right) with
different pruning rates.

Figure 5: Perplexity on zero-shot datasets across varyhing steps.

Table 5: Effect of the dependency-based structural pruning. Average represents
the average performance on 7 classification datasets.

| Method | WikiText2$\color[rgb]{1,0,0}\downarrow$ | PTB$\color[rgb]{1,0,0}\downarrow$ | Average$\color[rgb]{0,.5,.5}\uparrow$
---|---|---|---|---
w/o Tuning | w/o dependency | 68378.42 | 79942.47 | 38.32
w/ dependency | 19.09 | 34.21 | 56.69
w/ Tuning | w/o dependency | 13307.46 | 13548.08 | 38.10
w/ dependency | 17.58 | 30.11 | 59.23

Table 6: Impact of different aggregation strategies on group importance
estimation. Experiments are performed on LLaMA-7B.

Method | WikiText2$\color[rgb]{1,0,0}\downarrow$ | PTB$\color[rgb]{1,0,0}\downarrow$ | ARC-e$\color[rgb]{0,.5,.5}\uparrow$ | PIQA$\color[rgb]{0,.5,.5}\uparrow$ | OBQA$\color[rgb]{0,.5,.5}\uparrow$
---|---|---|---|---|---
Summation | 66.13 | 164.25 | 40.70 | 63.49 | 34.80
Max | 62.59 | 144.38 | 39.60 | 63.71 | 34.60
Production | 77.63 | 192.88 | 37.84 | 62.08 | 35.00
Last-only | 130.00 | 170.875 | 41.92 | 64.75 | 35.20

###  4.3 More Analysis

#### Impact of Dependency-based Structured Pruning.

To study the importance of dependency-based structural pruning, we conduct an
experiment to disrupt dependencies within groups, where each weight matrix
$W_{i}$ is pruned solely based on the importance score estimated on itself.
Table 6 presents the results demonstrating the impact of dependencies in
structural pruning. In the absence of dependencies, the model nearly fails in
the zero-shot generation and classification tasks. Even with tuning, the model
fails to recover, showing a substantial difference compared to the results in
dependency-based pruning.

#### Impact of Different Pruning Rates.

We investigate the impact of pruning the LLM at various pruning ratios in
Figure 5. We compare our pruning results with the L2 strategy because L2 is
also a data-free pruning algorithm. It is observed in the experiment of LLaMA
that when the pruning ratio reaches approximately 20%, the magnitude-dependent
algorithm experiences a rapid collapse, leading to the loss of information.
Conversely, by employing LLM-Pruner, we are able to increase the pruning ratio
to around 60% while achieving an equivalent perplexity level. Furthermore, in
the case of Vicuna-7B, removing 10% parameters results in a performance
decline equivalent to that of LLM-Pruner with 60%. The utilization of LLM-
Pruner enables a significant increase in the number of model parameters that
can be pruned, thereby substantially reducing computational overhead.

#### Tuning on the External Dataset.

To tune the pruned model, we utilize the external dataset Alpaca Taori et al.
(2023). The evaluation curves of the pruned model on two zero-shot datasets
during the post-training process are depicted in Figure 5. The results
demonstrate a rapid decrease in the perplexity of the pruned model within 300
steps, followed by a gradual increase. It is important to note that if the
model is trained for an excessive number of steps, it runs the risk of
overfitting the external dataset, potentially compromising its performance in
other general-purpose tasks.

#### Impact of Different Aggregation Strategies.

We conduct tests on the aggregation algorithms proposed in Section 3.2. Our
experimental results unveil notable discrepancies in model performance across
different aggregation strategies, with particular emphasis on the ‘Last-only’
strategy. Among the evaluated approaches, the ‘Max’ strategy attains the most
favorable outcomes in terms of perplexity, signifying enhanced coherence and
fluency in sentence generation. However, it is important to note that the
‘Max’ strategy exhibits the poorest zero-shot classification results compared
to all four strategies. Conversely, the ‘Last-only’ strategy showcases
superior classification performance but suffers from the poorest generation
quality. In our experiments, we make a trade-off by selecting the ‘Sum’
strategy since it shows both good generalization quality and classification
performance.

#### Case Study.

We provide some examples of sentences generated by the model compressed using
LLM-Pruner in Table 7. We made efforts to ensure a minimal overlap between
these generated sentences and the information contained in the tuning corpus,
which demonstrates that the information originates from the original model
rather than the tuning corpus. We provide additional examples in the Appendix,
including the generated sentences of the model without post-training. From the
cases in Table 7, it is evident that the sentences generated by the compressed
model are comparable to those produced by the original model. They exhibit
fluency, relevance, and informativeness regarding the given topic.
Nevertheless, during our experiments, we observed that the pruned model’s
performance deviates from that of the original model, particularly when
generating lengthy sentences. Occasionally, it may generate sentences that are
meaningless or contain repetitive tokens.

Table 7: Some examples of generations obtained with the original model and the
pruned model.

Model |  Original Model |  Pruned Model
---|---|---
LLaMa |  The universe is the entirety of space, time, matter, and energy that exists. It includes all matter and energy, whether visible or not, and in fact the whole universe is invisible. There are many theories regarding its origin and ultimate fate… |  The universe is the entirety of space, time, matter, and energy that exists. The laws that make up physics — electrons orbiting atoms, magnets attracting other objects, planets orbiting the Sun—have been consistent over …
Vicuna |  10 steps to build an ios app: 1\. Research and develop the idea. 2. Decide on the platform you want to use (iOS, Android, or both). 3. Create a plan and budget. 4. Hire a team of developers. 5. Design the user interface (UI). 6. Develop the user interface (UI). 7. Test and iterate on the app. 8. Submit the app to the app store. 9. Optimize the app for performance and user experience (UX). 10. Market and promote the app. |  10 steps to build an ios app: 1\. Plan your app: decide what it should do, what it will look like, and what it will cost to build it. 2. Hire a developer: A good iOS developer will help you design the user interface, create the app, and make sure it’s bug-free. 3. Create a user interface: This is where the magic happens. Your app’s user interface will be designed and mocked up at this point…
ChatGLM |  Write a poetry about love and peace. In the midst of life’s hustle and bustle, I find your love a soothing peace. It is a feeling that cannot be conveyed, A connection that cannot be distanced. With your smile, your eyes, your embrace, I feel a sense of belonging that only you can give. It is a love that lasts forever, A love that cannot be defeat… |  Write a poetry about love and peace. Love and peace, two such wonderful things\ A pure and endless desire in my heart\ And both of them, I must seek for\ A long, long time, I know..\ Love, I know, is a feeling of being\ A perfect partner, in every sense\ And peace, I need it, so much, one day\ A long, long way, my heart will go..

##  5 Conclusion

In this paper, we propose LLM-Pruner, a structured pruning approach for large
language models. LLM-Pruner aims to compress sizable language models in a
task-agnostic manner while minimizing the dependency on the original training
corpus and preserving the linguistic capabilities of LLMs. LLM-Pruner
accomplishes this by iteratively examining each neuron within the model as a
trigger for identifying dependency groups, thereby constructing the LLM’s
dependency graph. Subsequently, LLM-Pruner assesses the importance of these
groups using both parameter-wise and weight-wise estimation. Finally, we
utilize LoRA for fast recovery and adjustment of the pruned model. We evaluate
the efficacy of LLM-Pruner on three distinct models—LLaMA, Vicuna, and
ChatGLM—utilizing various zero-shot datasets. Our experimental results
indicate that LLM-Pruner successfully prunes the model, reducing computational
burden while retaining its zero-shot capabilities. Nevertheless, considerable
performance degradation occurs when employing high pruning rates, such as the
removal of 50% of LLaMA’s parameters, resulting in a substantial decline in
model performance. Additionally, we observe instances in which the model
generates incoherent sentences. Addressing the challenges associated with
compressing LLMs at higher pruning rates remains a challenging task.

## References

  * Bai et al. [2020] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King.  Binarybert: Pushing the limit of bert quantization.  _arXiv preprint arXiv:2012.15701_ , 2020.
  * Bisk et al. [2020] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.  Piqa: Reasoning about physical commonsense in natural language.  In _Thirty-Fourth AAAI Conference on Artificial Intelligence_ , 2020\.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality, March 2023.  URL https://lmsys.org/blog/2023-03-30-vicuna/.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Clark et al. [2019] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2924–2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1300.  URL https://aclanthology.org/N19-1300.
  * Clark et al. [2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv:1803.05457v1_ , 2018.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Fan et al. [2019] Angela Fan, Edouard Grave, and Armand Joulin.  Reducing transformer depth on demand with structured dropout.  _arXiv preprint arXiv:1909.11556_ , 2019.
  * Fang et al. [2023] Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang.  Depgraph: Towards any structural pruning, 2023.
  * Frantar and Alistarh [2023] Elias Frantar and Dan Alistarh.  Massive language models can be accurately pruned in one-shot.  _arXiv preprint arXiv:2301.00774_ , 2023.
  * Frantar et al. [2022] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.  Gptq: Accurate post-training quantization for generative pre-trained transformers.  _arXiv preprint arXiv:2210.17323_ , 2022.
  * Gao et al. [2021] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al.  A framework for few-shot language model evaluation.  _Version v0. 0.1. Sept_ , 2021.
  * Guo et al. [2019] Fu-Ming Guo, Sijia Liu, Finlay S. Mungall, Xue Lin, and Yanzhi Wang.  Reweighted proximal pruning for large-scale language representation.  _CoRR_ , abs/1909.12486, 2019.  URL http://arxiv.org/abs/1909.12486.
  * Han et al. [2015] Song Han, Jeff Pool, John Tran, and William Dally.  Learning both weights and connections for efficient neural network.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc., 2015.  URL https://proceedings.neurips.cc/paper_files/paper/2015/file/ae0eb3eed39d2bcef4622b2499a05fe6-Paper.pdf.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_ , 2022.
  * Hou et al. [2020] Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu.  Dynabert: Dynamic bert with adaptive width and depth.  _Advances in Neural Information Processing Systems_ , 33:9782–9793, 2020.
  * Hu et al. [2021] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models, 2021.
  * Jiao et al. [2020] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.  Tinybert: Distilling bert for natural language understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4163–4174, 2020.
  * Kurtic et al. [2022] Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh.  The optimal bert surgeon: Scalable and accurate second-order pruning for large language models.  _arXiv preprint arXiv:2203.07259_ , 2022.
  * Kwon et al. [2022] Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami.  A fast post-training pruning framework for transformers.  _arXiv preprint arXiv:2204.09656_ , 2022.
  * Lan et al. [2019] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_ , 2019.
  * LeCun et al. [1989] Yann LeCun, John Denker, and Sara Solla.  Optimal brain damage.  _Advances in neural information processing systems_ , 2, 1989.
  * Lewis et al. [2019] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ , 2019.
  * Li et al. [2016] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf.  Pruning filters for efficient convnets.  _arXiv preprint arXiv:1608.08710_ , 2016.
  * Li et al. [2017] Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu.  Dailydialog: A manually labelled multi-turn dialogue dataset.  In _Proceedings of The 8th International Joint Conference on Natural Language Processing (IJCNLP 2017)_ , 2017.
  * Liang et al. [2023] Chen Liang, Haoming Jiang, Zheng Li, Xianfeng Tang, Bin Yin, and Tuo Zhao.  Homodistil: Homotopic task-agnostic distillation of pre-trained transformers.  _arXiv preprint arXiv:2302.09632_ , 2023.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Liu et al. [2021] Zejian Liu, Fanrong Li, Gang Li, and Jian Cheng.  Ebert: Efficient bert inference with dynamic structured pruning.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4814–4823, 2021.
  * Ma et al. [2020] Xinyin Ma, Yongliang Shen, Gongfan Fang, Chen Chen, Chenghao Jia, and Weiming Lu.  Adversarial self-supervised data-free distillation for text classification.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6182–6192, 2020.
  * Ma et al. [2022] Xinyin Ma, Xinchao Wang, Gongfan Fang, Yongliang Shen, and Weiming Lu.  Prompting to distill: Boosting data-free knowledge distillation via reinforced prompt.  _arXiv preprint arXiv:2205.07523_ , 2022.
  * Marcus et al. [1993] Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz.  Building a large annotated corpus of English: The Penn Treebank.  _Computational Linguistics_ , 19(2):313–330, 1993\.  URL https://www.aclweb.org/anthology/J93-2004.
  * McCarley et al. [2019] JS McCarley, Rishav Chakravarti, and Avirup Sil.  Structured pruning of a bert-based question answering model.  _arXiv preprint arXiv:1910.06360_ , 2019.
  * Merity et al. [2016] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models, 2016.
  * Mihaylov et al. [2018] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _EMNLP_ , 2018.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Pan et al. [2020a] Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li, and Jun Huang.  Meta-kd: A meta knowledge distillation framework for language model compression across domains.  _CoRR_ , abs/2012.01266, 2020a.  URL https://arxiv.org/abs/2012.01266.
  * Pan et al. [2020b] Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li, and Jun Huang.  Meta-kd: A meta knowledge distillation framework for language model compression across domains.  _arXiv preprint arXiv:2012.01266_ , 2020b.
  * Rashid et al. [2020] Ahmad Rashid, Vasileios Lioutas, Abbas Ghaddar, and Mehdi Rezagholizadeh.  Towards zero-shot knowledge distillation for natural language processing, 2020.
  * Sakaguchi et al. [2019] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Winogrande: An adversarial winograd schema challenge at scale, 2019.
  * Scao et al. [2022] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_ , 2022.
  * Srinivas and Babu [2015] Suraj Srinivas and R Venkatesh Babu.  Data-free parameter pruning for deep neural networks.  _arXiv preprint arXiv:1507.06149_ , 2015.
  * Sun et al. [2019] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu.  Patient knowledge distillation for bert model compression.  _arXiv preprint arXiv:1908.09355_ , 2019.
  * Sun et al. [2020a] Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, and Jingjing Liu.  Contrastive distillation on intermediate representations for language model compression.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 498–508, Online, November 2020a. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.36.  URL https://aclanthology.org/2020.emnlp-main.36.
  * Sun et al. [2020b] Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou.  Mobilebert: a compact task-agnostic bert for resource-limited devices.  _arXiv preprint arXiv:2004.02984_ , 2020b.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Thoppilan et al. [2022] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_ , 2022.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Voita et al. [2019] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5797–5808, 2019.
  * Wang et al. [2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_ , 2018.
  * Wang et al. [2019a] Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang.  Eigendamage: Structured pruning in the kronecker-factored eigenbasis.  In _International conference on machine learning_ , pages 6566–6575. PMLR, 2019a.
  * Wang et al. [2020] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  _Advances in Neural Information Processing Systems_ , 33:5776–5788, 2020.
  * Wang et al. [2019b] Ziheng Wang, Jeremy Wohlwend, and Tao Lei.  Structured pruning of large language models.  _arXiv preprint arXiv:1910.04732_ , 2019b.
  * Wei et al. [2022a] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ , 2022a.
  * Wei et al. [2022b] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ , 2022b.
  * Xia et al. [2022] Mengzhou Xia, Zexuan Zhong, and Danqi Chen.  Structured pruning learns compact and accurate models.  _arXiv preprint arXiv:2204.00408_ , 2022.
  * Xin et al. [2020] Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin.  DeeBERT: Dynamic early exiting for accelerating BERT inference.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2246–2251, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.204.  URL https://aclanthology.org/2020.acl-main.204.
  * Xu et al. [2021] Dongkuan Xu, Ian En-Hsu Yen, Jinxi Zhao, and Zhibin Xiao.  Rethinking network pruning–under the pre-train and fine-tune paradigm.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2376–2382, 2021.
  * Xue et al. [2020] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mt5: A massively multilingual pre-trained text-to-text transformer.  _arXiv preprint arXiv:2010.11934_ , 2020.
  * Yao et al. [2022] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He.  Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.  _Advances in Neural Information Processing Systems_ , 35:27168–27183, 2022.
  * Ye et al. [2021] Deming Ye, Yankai Lin, Yufei Huang, and Maosong Sun.  TR-BERT: Dynamic token reduction for accelerating BERT inference.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5798–5809, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.463.  URL https://aclanthology.org/2021.naacl-main.463.
  * Yvinec et al. [2022] Edouard Yvinec, Arnaud Dapogny, Matthieu Cord, and Kevin Bailly.  Red++: Data-free pruning of deep neural networks via input splitting and output merging.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(3):3664–3676, 2022.
  * Zafrir et al. [2019] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat.  Q8bert: Quantized 8bit bert.  In _2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS)_ , pages 36–39. IEEE, 2019.
  * Zafrir et al. [2021] Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, and Moshe Wasserblat.  Prune once for all: Sparse pre-trained language models.  _arXiv preprint arXiv:2111.05754_ , 2021.
  * Zellers et al. [2019] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019.
  * Zeng et al. [2022] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al.  Glm-130b: An open bilingual pre-trained model.  _arXiv preprint arXiv:2210.02414_ , 2022.
  * Zhu et al. [2015] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _The IEEE International Conference on Computer Vision (ICCV)_ , December 2015.


# Connecting Structure, Conformation and Energetics of Human
Telomere G-quadruplex Multimers

###### Abstract

G-quadruplexes (G4s) are helical four-stranded structures forming from
guanine-rich nucleic acid sequences, which are thought to play a role in
cancer development and malignant transformation. Most current studies focus on
G4 monomers, yet under suitable and biologically relevant conditions G4s
undergo multimerization. Here, we address the structural, conformational and
energetic features of G4 multimers formed in solutions by the human telomere
sequence. A novel multi-technique approach is used combining Small Angle X-ray
Scattering (SAXS) and circular dichroism experiments with coarse-grained
simulations, to provide quantitative information about large-scale structural
features and the stability of G4 multimers. The latter show a significant
polydispersity with an exponential distribution of contour lengths, suggesting
a step-growth polymerization. On increasing DNA concentration, the strength of
G4 stacking interaction increases, as well as the number of the units in the
aggregates, with dimers and trimers as the most probable forms. At the same
time, a variation of G4 conformation is observed. Our findings provide a
quantitative picture of human telomere G4 multimers, which must be accounted
for to achieve a rational design of anticancer drugs targeting DNA structures.

###### keywords:

DNA, Small Angle X-ray Scattering, Coarse-Grained Simulations, Circular
Dichroism, Structural Biology

Benedetta Petra Rosi Valeria Libera Andrea Orecchini Silvia Corezzi Giorgio
Schirò Petra Pernot Caterina Petrillo Lucia Comez* Cristiano De Michele*
Alessandro Paciaroni*

Dr. B. P. Rosi, Dr. V. Libera, Prof. A. Orecchini, Prof. S. Corezzi, Prof. C.
Petrillo, Prof. A. Paciaroni
Dipartimento di Fisica e Geologia
Università degli Studi di Perugia
Perugia 06123, Italy
E-mail<EMAIL_ADDRESS>

Prof. A. Orecchini, Dr. L. Comez
IOM-CNR c/o Dipartimento di Fisica e Geologia
Università degli Studi di Perugia
Perugia 06123, Italy
E-mail<EMAIL_ADDRESS>

C. De Michele
Dipartimento di Fisica
Università degli Studi di Roma “La Sapienza”
Rome 00185, Italy
E-mail<EMAIL_ADDRESS>

G. Schirò
Institut de Biologie Structurale
CNRS
Grenoble 38000, France

P. Pernot
European Synchrotron Radiation Facility (ESRF)
Grenoble 38043, France

##  1 Introduction

Deciphering the relationship between structure and function of biological
molecules is a difficult but crucial task. This appears even more complex, if
one considers that in the physiological context the role played by a
biomolecule not only depends on the behavior in its monomeric state, but also
on the interaction among different biomolecules and/or biomolecular units
giving rise to higher-order structures. A paradigmatic case where the higher-
order structure is crucial is represented by G-quadruplexes (G4s). These
biomolecules are noncanonical nucleic acid structures formed by G-rich
oligonucleotides which fold into four-stranded helical structures consisting
of multiple stacked planar arrays of four guanine bases associated through
cyclic Hoogsteen-like hydrogen bonds (G-tetrads) [1]. G4s display three main
topologies (parallel, antiparallel and hybrid) that differ in the relative
orientation of the four guanine runs and in the arrangement of loop regions
[2, 3, 4]. They are highly polymorphic, as their structure depends both on the
specific oligonucleotide sequence and on environmental factors, such as the
type and concentration of cations, molecular crowding and/or dehydration
conditions [5, 6, 7, 8, 9, 10]. In the genomes of higher eukaryotes, sequences
with the ability to form G4s are abundant [11, 12, 13] and concentrated in the
telomeric regions (up to 25 % of all G4 DNA) [14]. G4s have also been detected
in cells [15, 14], where they are thought to regulate transcription,
translation, DNA replication, RNA localization and other biological functions
[16, 17, 18]. Because of such biological importance, G4s have received a lot
of attention as drug-design targets [1, 19, 20]. In particular, since G4s have
been demonstrated to block telomerase and HIV integrase, there is cause to
believe that specific G4-stabilizing ligands could be used as anti-cancer or
anti-viral drugs [21, 22]. In addition to this, G4s have been widely
investigated as promising building blocks and functional elements in fields
such as synthetic biology and nanotechnology [23, 24], mostly because of their
high stability, structural versatility, and functional diversity.
The majority of studies on G4s have focused on their monomeric state, but
there is evidence that G4s can take many different multimeric forms [25].
Aggregation has been shown to depend on the length of loops and likely to
occur through the stacking of external G-tetrads of parallel folds, with
dimers and trimers as the most probable aggregated forms [9]. It has been
recently proposed that multimeric G4s may have an important biological role in
the case of telomeric DNA, where the 3’ single-stranded overhang has the
capability to form higher-order structures containing several G4 units linked
by TTA spacers [26]. An effective design of anticancer drugs targeting
telomere G4s [27] clearly demands for a detailed knowledge of the spatial
arrangement and stability of their multimeric structure. Here we investigate
the structural, conformational and energetic features of multimers formed by
units of the human telomere sequence Tel22 [AG3(T2AG3)3], by exploiting a
unique biophysical multi-technique approach that combines small angle X-ray
scattering (SAXS), circular dichroism (CD) and coarse-grained simulations. The
comparison of experimental and numerical data allows a quantitative
description of the multimerization of Tel22 G4s at increasing DNA
concentrations and unveils how the aggregation process affects the structure,
the conformation and stacking interactions of Tel22 G4s units.

##  2 Results and Discussion

###  2.1 Small-angle X-ray Scattering

To experimentally investigate the structural properties of concentrated G4
samples, we performed SAXS measurements on G4 solutions at different DNA
concentrations. In the high-$Q$ region, $Q>1$ nm-1, the signal accounts mainly
for the structural features of the G4 unit (Tel22), i.e. shape and
characteristic dimensions, through the so-called form factor function $P(Q)$.
On the other hand, in the intermediate- and low-Q regions, the intensity
$I(Q)$ reflects the higher-order structure arising from the reversible
multimerization process of G4 units. In particular, the contribution of Tel22
multimers is quite visible, as a clear deviation below $Q<1$ nm-1 in Figure 1,
where the scattering intensity of a solution of Tel22 monomers at DNA
concentration $C=0.5$ mM (data taken from Reference [28]) is compared with
that of the present study. It is worth noting that the monomeric sample was
prepared in such a way to rule out the presence of possible aggregates (see
the Methods section). Actually, describing the complex mechanism of G4s self-
assembly and deriving quantitative information on structural and
conformational features of the G4 aggregates from SAXS measurements is a very
difficult task, as we expect a significant degree of polydispersity in the
length distribution of the aggregates [29]. Also in the high-Q region, where
the SAXS signal is dominated by the form factor, the interpretation of the
experimental data deserves great care. In a previous study [28], we found the
form factor of Tel22 monomers in solution to be well described by that of a
hard cylinder (HC). In this case the topology of Tel22 consists of a mixture
of antiparallel and hybrid conformers, however, an increase of concentration
above $\approx 1$ mM drives a progressive shift toward the parallel
(propeller) state, with concomitant decrease of the anti-parallel and hybrid
forms [30, 31, 32, 33]. This shift in the conformational populations entails a
different description of the average G4 unit, as each conformer is
characterized by a different shape and volume [34]. Since the only form factor
known for the Tel22 sequence in solution is the one in the antiparallel
conformation (143D structure in the PDB database [35]), we exploited numerical
simulations in conjuction with experimental data to get new insights into the
basic structural features of the different conformers.

Figure 1: Logarithmic plot of the SAXS intensities of G4 solutions at
different DNA concentrations. Data are reported in absolute scale and
normalized to the molar concentration $C$ of DNA. For comparison, also the
SAXS intensities of a Tel22 sample at $C$=0.5 mM are reported from Reference
[28]. In the inset, the same SAXS profiles are shown on a linear-log scale.

###  2.2 Simulation model

Figure 2: Simulation model for the G4 unit: (a) each G4 is modeled as a HC
characterized by a diameter $D$ and a length $L$. (b) Each cylinder is
decorated with two attractive sites at the basis. Sites belonging to different
cylinders interact through the SW potential $u(r)$.

To obtain more quantitative information on the G4 ensembles, we developed a
novel approach comparing SAXS experimental intensities with those
computationally estimated through Monte Carlo (MC) simulations, by modeling
the G4 unit as a hard cylinder (HC) with two attractive sites at the basis, as
schematized in Figure 2 (for more details see the Methods section). Scattering
intensities from simulations of HCs have been obtained following a procedure
similar to that discussed in Reference [36]. Within this procedure each
cylinder is replaced with a set of scattering points randomly placed inside
its volume with a fixed number density, and by using these points the
structure factor is then calculated. This procedure ensures that the numerical
scattering intensity also includes the form factor of HCs so that it can be
directly compared with the experimental one. This simple yet effective coarse-
grained model has proven to successfully describe reversible self-aggregation
processes in other DNA-based systems [37, 38], and it is especially suitable
for describing the hydrophobic (stacking) forces acting between G4 units. An
extensive campaign of simulations was carried out so as to reproduce SAXS
results for all the concentrations studied experimentally. In particular, to
find the best agreement between simulations and experiments we explored the
phase space corresponding to different G4 shapes and different strength of
stacking interaction between units. As to the former, the starting point was a
HC with a diameter of $D_{0}=2.0$ nm and a length of $L_{0}=3.7$ nm that best
reproduces the monomeric form of Tel22 sequences [28]. To modulate the shape
of the cylinder we introduced the parameter $K$, so that $D=D_{0}\cdot K$ and
$L=L_{0}/K$ ($K\geq 1$). Finally, the stacking interaction between G4s was
varied through the effective temperature $T^{*}=k_{B}T/u_{0}$ (where $u_{0}$
is the binding energy of the HC attractive sites).

###  2.3 Comparison between SAXS data and simulations

Figure 3: Scattering intensities for the G4 solution at $C$ = 0.6 mM, obtained
from SAXS measurements and simulations (panels b – f). The accordance between
experimental and simulated data has been evaluated through the residual sum of
squares ($RSS$), calculated at different values of $T^{*}$ and $K$ (panel a).

The experimental $I(Q)$ can be suitably reproduced by finely changing the $K$
and $T^{*}$ simulation parameters, as shown in Figure 3 for the selected
concentration $C$=0.6 mM. In this case an excellent agreement between
simulation and experiment is obtained for $K$=1 and $T^{*}$=0.095 (panel b).
Similar plots for $C$=1.2 mM and $C$=4.5 mM are reported in Figure SI1 and SI2
of the Supporting Information, respectively.

Figure 4: Panel a, b, and c of the figure show the best accordance between
experimental and simulated scattering intensities, respectively for the G4
solution at $C$ = 0.6 mM, 1.2 mM, and 4.5 mM.

Best matches (according to the residual sum of squares $RSS$) between
experiments and simulations are reported in Figure 4. Experimental and
simulated curves remarkably superimpose onto each other over the whole
explored $Q$-range. As expected, in the region of the highest $Q$ values
(above $\approx 3$ nm-1) the simulated SAXS intensity is less effective in
describing the experimental data, since it does not take into account the fine
structural details of the actual G4 folds numerically modeled as HCs. On the
other hand, simulations are able to reproduce strikingly well the experimental
data in the low- and intermediate-$Q$ region, and can therefore effectively
describe the aggregation processes.

Table 1: List of parameters associated with the best representative state
point ($K$,$T^{*}$) for each of the investigated concentration $C$.

$C$ [mM] | $T^{*}$ | $K$ | $M^{a}$ | Ð |  $G^{0}_{ST}$ [kcal mol-1]b |  $H^{0}_{ST}$ [kcal mol-1]c |  $S^{0}_{ST}$ [cal mol-1K-1]d
---|---|---|---|---|---|---|---
0.6 | 0.095 | 1 | 2.5 $\pm$ 0.3 | 1.60 $\pm$ 0.04 | -0.76 $\pm$ 0.17 | -6.12 $\pm$ 0.16 | -18.3 $\pm$ 0.8
1.2 | 0.1 | 1.05 | 2.4 $\pm$ 0.2 | 1.58 $\pm$ 0.04 | -0.70 $\pm$ 0.15 | -5.82 $\pm$ 0.15 | -17.5 $\pm$ 0.7
4.5 | 0.09 | 1.1 | 3.4 $\pm$ 0.5 | 1.70 $\pm$ 0.04 | -1.21 $\pm$ 0.19 | -6.47 $\pm$ 0.18 | -18.0 $\pm$ 0.9

aAverage chain length. Errors were estimated by considering the statistical
error on $\epsilon$ and the error associated with the finite value $\Delta
T^{*}$ by which temperatures were sampled in the simulations. bStacking free
energy calculated for a standard concentration 1 M of G4s and $T=293$ K.
cEnergy contribution to $G^{0}_{ST}$. d Entropic contribution to $G^{0}_{ST}$.

Once validated, simulations can be exploited to obtain quantitative
information on stacking and aggregation of G4 solutions as the concentration
varies. In particular, the adopted numerical model is consistent with an
exponential distribution $\nu(l)$ of G4 multimer chain lengths $l$:
$\nu(l)=\rho M^{-(l+1)}(M-1)^{(l-1)}$, where
$\rho=\sum_{l=1}^{\infty}l\nu(l)=N/V$ is the number of G4s per unit volume,
and $M$ is the average number of stacked units. The value of $M$ can be
directly obtained from the simulation, since it is related to the average
potential energy per G4 monomer, $\epsilon$, through the relationship
$M=(1+\epsilon/u_{0})^{-1}$ [39]. The simulations thus provide an estimate of
the average length of G4 multimers at each of the investigated concentrations,
as reported in Table 1. Quite interestingly, $M$ values are in the range 2.4
$\div$ 3.4, in agreement with the experimental results suggesting dimers and
trimers as the main multimeric forms [9].
It is worth noting that using an exponential distribution to describe the
length of multimer chains is equivalent to assume that multimerization of G4s
complies with a step-growth mechanism. This is analogous to the case of the
self-assembly of DNA-encoded nanoparticles into chain-like superstructures
[40], and at variance with the chain-growth mechanism which is less commonly
applied to supramolecular biopolymers [41, 42]. In addition, as Tel22
multimers follow an exponential distribution, we can easily estimate the
spread of their molecular mass distribution as quantified by the ratio between
the weight average molecular weight and the number average molecular weight,
i.e. the so called dispersity Ð$=2-1/M$ [43]. The parameter Ð, which is
reported in Table 1, indicates a moderate degree of polidispersity as it is
higher than the monodisperse limit Ð$=1$ but smaller than the highest
attainable value Ð$=2$.

The knowledge of the average chain length $M$ allows to calculate also the
coaxial stacking free energy of the system $G_{ST}=-k_{B}T\ln[M(M-1)]$ [44]
(Table 1). The standard free energy $G^{0}_{ST}$, calculated for a standard
temperature of 293 K and concentration of 1 M, increases from about -0.7 kcal
mol-1 at the lowest concentrations to -1.2 kcal mol-1 at $C=4.5$ mM. The
associated contributions of the standard bonding energy $H^{0}_{ST}$ and the
standard bonding entropy $S^{0}_{ST}$ are of the order of $\approx-6$ kcal
mol-1 and $\approx-18$ cal mol-1K-1, respectively. We note that our estimate
for $G^{0}_{ST}$ is much lower than other values reported in literature for
the stacking free energies of G4 structures. In a recent computational work,
for instance, the dimerization free energy of parallel Tel22 G4s was estimated
to be of the order of $-20$ kcal mol-1 in the most stable 5’-5’ configuration
[45]. However, we point out that this disagreement can arise from the fact
that free energy estimates based on all-atom molecular dynamics simulations
may be quite inaccurate in calculating the entropic contribution to the
stacking free energy, as it happens in the case of self-assembled short DNA-
duplexes [44, 46, 47, 48]. For example, a system composed of $90$ ultra-short
DNA duplexes of 5 base-pairs each has been studied through all-atom
simulations in Reference [49] and authors note that an overestimate of
stacking attractions leads to unrealistically long aggregates. For comparison,
computational studies in which only the enthalpic term is considered return
G4s stacking energies between -34 and -8 kcal mol-1 [50], which are in line
with our values. As current estimates of intermolecular G4 stacking energies
mainly come from numerical studies, our findings call for future experimental
work on this subject.

As for the geometry of the G4 units, the experimental curves are better
reproduced using slightly larger values of $K$ on increasing concentration (as
reported in Table 1), corresponding to a variation of the simulated HCs
towards a more disk-like shape. This trend suggests an increase of the
fraction of G4 with parallel conformation, that is known to have a more oblate
shape compared to the antiparallel and hybrid folds [34]. This is consistent
with previous literature where the fraction of parallel G4s was proposed to
progressively increase for DNA concentrations above $\approx 1$ mM [30, 31,
33]. It is also worth of note that the parallel-stranded propeller topology
might favour the stacking of G4 units [51], thus further favouring the
formation of longer multimers on increasing concentration.

From the cylinder length $L$ and the average number of stacked units $M$ we
can readily estimate the gyration radius of the G4 multimers, that turns out
to be in the range 2.5 nm $\div$ 3.5 nm (see Supporting Information). To
further support the picture provided by our method, in Figure SI3 gyration
radii are shown to be in excellent agreement with those obtained by fitting
the SAXS data with a phenomenological function proposed by Beaucage to
describe systems with several characteristic lengths [52].

###  2.4 Circular dichroism

Figure 5: CD spectra of G4 solutions at different DNA concentrations
(circles). In each panel, the fit resulting from the spectral deconvolution
through the software by del Villar-Guerra et al. is reported (solid lines),
together with the corresponding residues (dashed lines). Pie charts in the
bottom of each panel report the fraction of quadruplex units with antiparallel
(AP), hybrid (H), and parallel (P) conformation, according to the spectral
deconvolution.

To shed light on the possible change of the G4 secondary structure, we
performed CD measurements as a function of the DNA concentration. As shown in
Figure 5, at $C=0.6$ mM (panel a) the spectrum displays the typical features
of G4 hybrid topology, with a maximum at about 295 nm, a shoulder at 270 nm
and a minimum at 240 nm. As $C$ increases, the maximum progressively
downshifts and a new peak appears at 265 nm (panel b and c), suggesting an
increase in the parallel fraction consistent with the findings reported above
[33]. A method based on the principal component analysis and singular value
decomposition was used to quantify the fraction of the major G4 topologies
[53]. In Figure 5 the fit of experimental data obtained from the deconvolution
into the three main folded topologies (parallel, antiparallel and hybrid), is
reported, with the corresponding results represented in the pie charts. A
population transfer is evident, as the hybrid component halves at the highest
concentration, while the parallel component grows from 10% to 40%.

By combining the values of the G4 stacking energy and aspect ratio from SAXS
and simulations with the fractions of topologies from CD, we can easily obtain
specific information on the shape and energy parameters of each G4
conformation (see Table SI2 of the Supporting Information). As anticipated, a
more elongated shape is associated with the antiparallel conformer, whereas a
flattened cylinder represents the parallel one. As for the stacking
interaction, it turns out that antiparallel G4s are much less prone to form
multimers than either hybrid or parallel G4s (see Table SI2). This is
consistent with previous results indicating that the parallel [45] and the
hybrid [26] conformations are prevalent in human telomere multimers.

##  3 Conclusions

This work provides an unprecedented quantitative description of the self-
assembly process of Tel22 G4s in concentrated solutions. Coarse-grained
simulations were exploited to extensively explore the parameter space related
to interaction stacking strength between G4s units, multimer length and the G4
unit shape. The coarse-grained simulation model was refined by successfully
reproducing the experimental SAXS data, thus determining how the average
length and the stability of G4 multimers changes as a function of DNA
concentration. In the emerging picture, on increasing the self-crowding of
G4s, the attractive interactions between different units become more and more
important and at the same time the G4 conformation shifts toward the parallel
state.
We show that our integrated approach allows a direct interpretation of low-
resolution structural data on a challenging system consisting of polydisperse
aggregates of polymorphic G4s, while also determining their thermodynamic
features. In the light of this, we plan to adopt the same method in the future
to provide a detailed description of G4s multimers even more closely related
to biological issues, such as those forming from long telomeric sequences, in
the presence of ligands of potential therapeutic interest. This knowledge will
be crucial to design and optimize new drugs specifically targeting interunit
junctions between adjacent G4s.

##  4 Methods

###  4.1 Experimental Methods

####  4.1.1 Sample preparation

Human telomere Tel22 AG3(T2AG3)3 ($M_{w}$: 6966.6 g mol-1) was purchased from
Eurogentec and used as received. Samples were prepared by dissolving the
lyophilized powder in a 50 mM phosphate buffer at pH 7, 0.3 mM EDTA and 150 mM
KCl. The high concentration of the starting solution ($\approx$ 13 mM) is
appropriate to ensure the formation of aggregates, that remain stable even
after dilution. The solution was heated up to 95°C, slowly cooled down to room
temperature and left at room temperature overnight. After this procedure, the
solution was centrifugated for 120 s at 15°C and 15000 rpm. From the
centrifugated solution, samples at 3 different DNA concentrations were
prepared, namely, $C$ = 0.6 mM, 1.2 mM, and 4.5 mM. The molarity of the
solutions was determined from UV absorption measurements at 260 nm, using a
molar extinction coefficient of 228 500 M-1 cm-1. Both experimental and
computational investigations were performed at these concentrations. Before
measurements, samples where further annealed and left at room temperature
overnight.

It should be noted that the sample in monomeric state taken from Reference
[28] was prepared starting from a stock solution with a lower concentration of
$C=1$ mM. Such a different procedure avoided the formation of aggregates.

####  4.1.2 Circular Dichroism

CD experiments were performed using a Jasco J810 spectropolarimeter in 0.01 to
0.1 mm path-length quartz cells. Spectra were collected in the range from 220
nm to 330 nm with a scan speed of 50 nm/min. All spectra were acquired at room
temperature. CD data were expressed as the difference in the molar absorption
$\Delta\epsilon$ [M-1 cm-1] of the right- and left-handed circularly polarized
light.

####  4.1.3 Small-angle X-ray Scattering

Small-angle X-ray Scattering (SAXS) experiments were performed at the BM29
beamline of the European Synchrotron Radiation Facility (ESRF) in Grenoble,
France. The incident energy was 12.5 keV, corresponding to an incident
wavelength of 0.99 Å-1. The scattering vector range was between $Q$ = 0.0044
Å-1 and 0.521 Å-1. All patterns were collected at 20° C. Analogous patterns of
the buffer were collected before and after every collection on the samples and
used to subtract any contribution from the solvent and the sample environment.

###  4.2 Computational Methods

###  4.3 Simulation model

The simulation model consisted of hard cylinders (HCs) characterized by a
length $L$ and a diameter $D$, with two attractive sites at the bases. The
attractive sites were located along the symmetry axis at a distance $L/2$ \+
$0.15D/2$ from the HC center of mass. Sites belonging to distinct particles
interact via the square-well (SW) potential, i.e., $\beta u_{SW}=-\beta
u_{0}$, if $r>\delta$, and $\beta u_{SW}$ = 0, if $r>\delta$, where $r$ is the
distance between the interacting sites, $\delta=0.25D$ is the interaction
range (i.e., the diameter of the attractive sites), and $\beta u_{0}$ is the
ratio between the binding energy and the thermal energy $k_{B}T$, where
$k_{B}$ i s the Boltzmann constant. The temperature was expressed as the
adimensional parameter $T^{*}=k_{B}T/u_{0}$.

####  4.3.1 Monte Carlo simulations

Simulations were performed in the canonical ($NVT^{*}$) ensemble, leveraging a
recently developed algorithm for checking the overlap between HCs [54] which
relies on a novel and very efficient algorithm for finding the roots of a
quartic equation [55]. For each value of concentration, we simulated a
suitable number of particles $N$ in a cubic box with volume $V$ using standard
periodic boundary conditions. Values of $N$ and $V$ where chosen so that the
density $\rho=N/V=N_{av}C$, where $N_{av}$ is tha Avogadro’s number,
reproduced each of the experimentally-investigated values of DNA concentration
$C$. The number of particles used in the simulations was $N=$ 5808, 6292 and
6776 for $C=$0.6 mM, 1.2 mM and 4.5 mM respectively. The aspect ratio of the
HCs was varied from that associated with the starting values of $D_{0}=2$ nm
and $L_{0}=3.7$ nm, by introducing the parameter $K$: $D=D_{0}\cdot K$ and
$L=L_{0}/K$. For each value of $C$, five different values of $K$ ($K$ =1,
1.05, 1.10, 1.15, and 1.2) were considered, thereby obtaining a total of 12
starting configurations. Each starting configuration was thermalised for at
least $10^{6}$ MC steps at six different values of $T^{*}$, namely,
$T^{*}$=0.085, 0.9, 0.095, 0.1, 0.105, and 0.11. The initial configuration
used for the equilibration was obtained by placing the hard cylinders on a
ortorhombic lattice.

Supporting Information

Supporting Information is available from the Wiley Online Library or from the
author.

Acknowledgements

B.P.R. and C.P. acknowledges the support from MIUR-PRIN (Grant No.
2017RX9XXY). C.D.M. acknowledges the support from MIUR-PRIN (Grant No.
2017Z55KCW).

## References

  * [1] S. Neidle,  _Nat. Rev. Chem._ 2017, _1_ , 0041.
  * [2] M. Webba da Silva,  _Chem. Eur. J._ 2007, _13_ 9738\.
  * [3] S. Neidle, G. N. Parkinson,  _Curr. Opin. Struct. Biol._ 2003, _13_ , 3 275.
  * [4] A. I. Karsisiotis, N. M. Hessari, E. Novellino, G. P. Spada, A. Randazzo, M. Webba da Silva,  _Angew. Chem. Int. Ed._ 2011, _50_ , 45 10645.
  * [5] J. L. Huppert,  _The FEBS Journal_ 2010, _277_ , 17 3452.
  * [6] A. T. Phan, V. Kuryavyi, D. J. Patel,  _Curr. Opin. Struct. Biol._ 2006, _16_ , 3 288.
  * [7] V. Víglaský, K. Tlučková, L. Bauer,  _Eur. Biophys. J._ 2011, _40_ 29\.
  * [8] J. B. Chaires,  _The FEBS Journal_ 2010, _277_ , 5 1098.
  * [9] N. Smargiasso, F. Rosu, W. Hsia, P. Colson, E. S. Baker, M. T. Bowers, E. De Pauw, V. Gabelica,  _J. Am. Chem. Soc._ 2008, _130_ , 31 10208.
  * [10] B. Heddi, A. T. Phan,  _J. Am. Chem. Soc._ 2011, _133_ , 25 9824.
  * [11] J. L. Huppert, S. Balasubramanian,  _Nucleic Acids Res._ 2005, _33_ , 9 2908.
  * [12] A. K. Todd, M. Johnston, S. Neidle,  _Nucleic Acids Res._ 2005, _33_ , 9 2901.
  * [13] R. Hänsel-Hertsch, M. Di Antonio, S. Balasubramanian,  _Nat. Rev. Mol. Cell Biol._ 2017, _18_ 279\.
  * [14] G. Biffi, D. Tannahill, J. McCafferty, S. Balasubramanian,  _Nat. Chem._ 2013, _5_ 182\.
  * [15] E. Y. N. Lam, D. Beraldi, D. Tannahill, S. Balasubramanian,  _Nat. Commun._ 2013, _4_ 1796\.
  * [16] J. L. Huppert, S. Balasubramanian,  _Nucleic Acids Res._ 2006, _35_ , 2 406.
  * [17] S. Kendrick, L. H. Hurley,  _Pure Appl. Chem._ 2010, _82_ , 8 1609.
  * [18] D. Rhodes, H. J. Lipps,  _Nucleic Acids Res._ 2015, _43_ , 18 8627.
  * [19] F. Bianchi, L. Comez, R. Biehl, F. D’Amico, A. Gessini, M. Longo, C. Masciovecchio, C. Petrillo, A. Radulescu, B. Rossi, F. Sacchetti, F. Sebastiani, N. Violini, A. Paciaroni,  _Nucleic Acids Res._ 2018, _46_ , 22 11927.
  * [20] L. Comez, F. Bianchi, V. Libera, M. Longo, C. Petrillo, F. Sacchetti, F. Sebastiani, F. D’Amico, B. Rossi, A. Gessini, C. Masciovecchio, H. Amenitsch, C. Sissi, A. Paciaroni,  _Phys. Chem. Chem. Phys._ 2020, _22_ 11583\.
  * [21] S. Neidle,  _The FEBS Journal_ 2010, _277_ , 5 1118.
  * [22] R. Perrone, E. Butovskaya, D. Daelemans, G. Palù, C. Pannecouque, S. N. Richter,  _J. Antimicrob. Chemother._ 2014, _69_ , 12 3248.
  * [23] L. A. Yatsunyk, O. Mendoza, J.-L. Mergny,  _Acc. Chem. Res._ 2014, _47_ , 6 1836.
  * [24] J.-L. Mergny, D. Sen,  _Chem. Rev._ 2019, _119_ , 10 6290.
  * [25] S. Kolesnikova, E. A. Curtis,  _Molecules_ 2019, _24_ , 17.
  * [26] R. C. Monsen, S. Chakravarthy, W. L. Dean, J. B. Chaires, J. O. Trent,  _Nucleic Acids Res._ 2021, _49_ , 3 1749.
  * [27] J. Zhao, Q. Zhai,  _Bioorg. Chem._ 2020, _103_ 104229\.
  * [28] V. Libera, E. A. Andreeva, A. Martel, A. Thureau, M. Longo, C. Petrillo, A. Paciaroni, G. Schirò, L. Comez,  _J. Phys. Chem. Lett._ 2021, _12_ , 33 8096.
  * [29] E. J. Baldassarri, M. G. Ortore, F. Spinozzi, A. Round, C. Ferrero, P. Mariani,  _Nanomaterials_ 2020, _10_ , 4.
  * [30] I. Kejnovská, M. Vorlíčková, M. Brázdová, J. Sagi,  _Biopolymers_ 2014, _101_ , 4 428.
  * [31] M. Vorlíčková, I. Kejnovská, J. Sagi, D. Renčiuk, K. Bednářová, J. Motlová, J. Kypr,  _Methods_ 2012, _57_ , 1 64.
  * [32] J. Palacký, M. Vorlíčková, I. Kejnovská, P. Mojzeš,  _Nucleic Acids Res._ 2013, _41_ , 2 1005.
  * [33] D. Renčiuk, I. Kejnovská, P. Školáková, K. Bednářová, J. Motlová, M. Vorlíčková,  _Nucleic Acids Res._ 2009, _37_ , 19 6625.
  * [34] N. An, A. M. Fleming, E. G. Middleton, C. J. Burrows,  _PNAS_ 2014, _111_ , 40 14325.
  * [35] Y. Wang, D. J. Patel,  _Structure_ 1993, _1_ , 4 263.
  * [36] A. Pal, C. A. De Filippo, T. Ito, M. A. Kamal, A. V. Petukhov, C. De Michele, P. Schurtenberger,  _ACS Nano_ 2022, _16_ , 2 2558.
  * [37] K. T. Nguyen, A. Battisti, D. Ancora, F. Sciortino, C. De Michele,  _Soft Matter_ 2015, _11_ 2934\.
  * [38] K. T. Nguyen, F. Sciortino, C. De Michele,  _Langmuir_ 2014, _30_ , 16 4814.
  * [39] F. Sciortino, E. Bianchi, J. F. Douglas, P. Tartaglia,  _J. Chem. Phys._ 2007, _126_ , 19 194903.
  * [40] M. Gu, X. Ma, L. Zhang, J. Lin,  _J. Am. Chem. Soc._ 2019, _141_ , 41 16408.
  * [41] S. Ogi, K. Sugiyasu, S. Manna, S. Samitsu, M. Takeuchi,  _Nature Chem._ 2014, _6_ 188–195.
  * [42] H. Zhang, Y. Wang, H. Zhang, X. Liu, A. Lee, Q. Huang, F. Wang, J. Chao, H. Liu, J. Li, J. Shi, X. Zuo, L. Wang, X. Cao, C. Bustamante, Z.-Q. Tian, C. Fan,  _Nat. Comm._ 2019, _10_ , 1006.
  * [43] I. Teraoka,  _Models of Polymer Chains_ , chapter 1, 1–67,  John Wiley and Sons, Ltd,  ISBN 9780471224518, 2002.
  * [44] C. De Michele, T. Bellini, F. Sciortino,  _Macromolecules_ 2012, _45_ , 2 1090.
  * [45] M. Kogut, C. Kleist, J. Czub,  _PLoS Comput. Bio._ 2019, _15_ , e1007383.
  * [46] C. Maffeo, B. Luan, A. Aksimentiev,  _Nucleic Acids Res._ 2012, _40_ 3812\.
  * [47] C. Maffeo, J. Yoo, J. Comer, D. B. Wells, B. Luan, A. Aksimentiev,  _J. Condens. Matter Phys._ 2014, _26_ , 41 413101.
  * [48] A. A. Chen, A. E. García,  _PNAS_ 2013, _110_ , 42 16820.
  * [49] S. Saurabh, Y. Lansac, Y. H. Jang, M. A. Glaser, N. A. Clark, P. K. Maiti,  _Phys. Rev. E_ 2017, _95_ 032702\.
  * [50] N. Špačková, I. Berger, J. Šponer,  _J. Am. Chem. Soc._ 1999, _121_ , 23 5519.
  * [51] S. Haider, G. N. Parkinson, S. Neidle,  _Biophys. J._ 2008, _95_ , 1 296.
  * [52] G. Beaucage,  _J. Appl. Crystallog._ 1996, _29_ , 2 134.
  * [53] R. del Villar-Guerra, J. O. Trent, J. B. Chaires,  _Angew. Chem. Int. Ed._ 2018, _57_ , 24 7171.
  * [54] A. Orellana, E. Romani, C. De Michele,  _Eur. Phys. J. E_ 2018, _41_ , 4.
  * [55] A. G. Orellana, C. D. Michele,  _ACM Trans. Math. Softw._ 2020, _46_ , 2.