# Dependency-Aware Semi-Structured Sparsity of
GLU Variants in Large Language Models

Zhiyu Guo  Hidetaka Kamigaito  Taro Wanatnabe

###### Abstract

The rapid advancement in Large Language Models (LLMs) has markedly enhanced
the capabilities of language understanding and generation. However, the
substantial model size poses hardware challenges, affecting both memory size
for serving and inference latency for token generation. To address those
challenges, we propose Dependency-aware Semi-structured Sparsity (DaSS), a
novel method for the recent prevalent SwiGLU-based LLMs pruning. Our approach
incorporates structural dependency into the weight magnitude-based
unstructured pruning. We introduce an MLP-specific pruning metric that
evaluates the importance of each weight by jointly considering its magnitude
and its corresponding MLP intermediate activation norms. DaSS facilitates a
balance between the adaptability offered by unstructured pruning and the
structural consistency inherent in dependency-based structured pruning.
Empirical evaluations on Mistral and LLaMA2 model families demonstrate that
DaSS not only outperforms both SparseGPT and Wanda in achieving hardware-
friendly N:M sparsity patterns but also maintains the computational efficiency
of Wanda.

Machine Learning, ICML

##  1 Introduction

Recent years have witnessed the great success of Transformer-based Large
Language Models (LLMs) (Brown et al., 2020; Chowdhery et al., 2023; OpenAI,
2023) across various challenging tasks, such as mathematical reasoning (Cobbe
et al., 2021), code generation (Chen et al., 2021). However, the practical use
of these models for inference has faced a major obstacle due to the
substantial computational resources they consume. To tackle this, several
methods for efficient LLMs inference have been applied, including training
compact models using more data (Touvron et al., 2023a), model quantization
(Dettmers et al., 2022; Frantar et al., 2023), and grouped query attention
(Shazeer, 2019; Ainslie et al., 2023).

Neural network pruning (Han et al., 2015; Wen et al., 2016) is a widely
adopted technique for compressing models, leading to a substantial reduction
in model size and inference latency. Pruning can be categorized into two main
approaches: unstructured pruning (Sun et al., 2023; Frantar & Alistarh, 2023),
which involves the removal of specific weights, and structured pruning (Zhang
et al., 2023; Ma et al., 2023), which entails the removal of complete rows or
columns of weights. In contrast to structured pruning, which struggles with
performance in LLMs even at low sparsity levels, unstructured pruning (Sun et
al., 2023; Frantar & Alistarh, 2023) exhibits promising results without
additional retraining, and achieves practical speedup in Nvidia GPU through
more stringent N:M sparsity pattern (Mishra et al., 2021; Kurtic et al.,
2023). SparseGPT (Frantar & Alistarh, 2023) tackles the issue of pruning LLMs
by considering it as a problem of layerwise reconstruction. It exclusively
depends on weight updates aimed at maintaining the input-output relationship
for each layer. Motivated by the emergent large magnitude activations in LLMs
(Dettmers et al., 2022), Wanda (Sun et al., 2023) further eases the
computational load in SparseGPT by solely employing the product derived from
weights and input activation magnitudes.

(a) Dependency-based Structured Sparsity

(b) Wanda Unstructured Sparsity

(c) Dependency-aware Semi-structured Sparsity

Figure 1: Illustration of our proposed Dependency-aware Semi-structured
Sparsity (DaSS). In (a) dependency-based structured pruning (Ma et al., 2023),
all the weights connecting to the same intermediate neuron are removed or
remain simultaneously. In (b) Wanda unstructured pruning (Sun et al., 2023),
it assigns greater emphasis to the weights corresponding to large input
activations. For Gate-Proj and UP-Proj, the same number of weights are removed
for each MLP neuron, regardless of whether some neurons have much larger
activation norms. For Down-Proj, the weights corresponding to larger
activation norms are more likely to be pruned. This can lead to a structural
mismatch. In (c), all the weights corresponding to large intermediate
activations are more likely to be reserved.

As the SwiGLU-based MLP (Shazeer, 2020) module accounts for more than 80%
parameters in the recent LLMs that use multi/grouped query attention
(Chowdhery et al., 2023; Touvron et al., 2023b)111In LLaMA2-70B, the dimension
of key and value is $\frac{1}{8}d$, MLP intermediate dimension is
$\frac{7}{2}d$., its pruning emerges as a pivotal factor in determining the
overall compression efficacy of LLMs. In dependency-aware structured pruning,
it’s critical to consider that pruned parameters have dependencies with other
parameters, owing to their interconnected nature (Ma et al., 2023; Fang et
al., 2023). In the context of MLP pruning, all the weights connected to each
intermediate neuron should be preserved or pruned simultaneously. The precise
coordination involved in pruning is crucial for upholding the model’s
structural integrity and its functional capabilities. Although current
unstructured pruning methods (Sun et al., 2023; Frantar & Alistarh, 2023)
effectively remove a significant number of redundant weights, they operate
entirely locally within each linear layer without considering inter-
dependencies to other layers. This can lead to a structural mismatch, which is
more evident in Wanda as shown in Figure 1(b): In Gate and Up projections, the
same amount of parameters are pruned for each MLP neuron. However, the
intermediate activation norms of SwiGLU are not uniformly distributed and some
neurons have much larger norms than others. Based on Wanda pruning metric,
more weights connected to neurons with large activation norms are preserved.
At high sparsity, this problem gets magnified in Wanda causing significant
drops in performance by the broken network.

In order to overcome the limitations present in current pruning methodologies,
we introduce a new paradigm, namely, Dependency-aware Semi-structured Sparsity
(DaSS). This approach is specifically designed to navigate the middle ground
between the flexibility of unstructured pruning and the structural consistency
of dependency-based structured pruning. We examine the effectiveness of
magnitude pruning in Gate and Up projections of SwiGLU, and we find that
grouping weights per-input is surprisingly effective compared with Wanda (Sun
et al., 2023). Leveraging these newly acquired understandings, we present a
new MLP pruning metric that assesses each weight’s importance based on the
product of its magnitude and the norm of the corresponding MLP intermediate
activations. Our proposed DaSS method, illustrated in Figure 1(c), embodies a
semi-structured pattern that retains a degree of the adaptability inherent in
unstructured pruning while incorporating the dependency-aware aspect of
structured pruning. This balance allows for more precise pruning. DaSS can be
easily extended to hardware-friendly N:M sparsity patterns.

We perform extensive experiments on LLaMA2 and Mistral to evaluate DaSS across
various tasks from language modeling, commonsense reasoning, and MMLU
(Hendrycks et al., 2021). In achieving hardware-friendly N:M sparsity
patterns, DaSS consistently excels beyond the existing LLM pruning methods
SparseGPT (Frantar & Alistarh, 2023) and Wanda (Sun et al., 2023), while
maintaining the computational efficiency akin to Wanda. Impressively, DaSS
outperforms SparseGPT at high sparsity even without weight update. Our work
contributes fresh insights into the nuanced understanding of sparsity within
large language models.

##  2 Preliminary Exploration

###  2.1 Wanda Pruning Method

In the context of LLM pruning, we denote a linear layer weight matrix
$\mathbf{W}\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$ and input
activations $\mathbf{X}\in\mathbb{R}^{L\times d_{\text{in}}}$, where
$d_{\text{in}}$, $d_{\text{out}}$ and $L$ represent weight input dimension,
weight output dimension, and input sequence length, respectively. Wanda (Sun
et al., 2023) evaluates weight importance through the multiplication of the
weight magnitude with the norm of the input feature. Specifically, the
importance score for a given weight $\mathbf{W}_{i,j}$ is determined as
follows:

$\mathbf{I}_{i,j}=\left|\mathbf{W}_{i,j}\right|\cdot\left\|\mathbf{X}_{j}\right\|_{2}$
(1)

Here, $\mathbf{I}_{i,j}$ is the weight importance score for a given weight
$\mathbf{W}_{i,j}$ in the weight matrix. $\left\|\mathbf{X}_{j}\right\|_{2}$
represents the $\ell_{2}$ norm of the $j^{th}$ feature of input $\mathbf{X}$,
which is calculated across all $L$ tokens to produce a scalar. In Wanda
pruning, importance scores for weights are compared for each output
individually, corresponding to each row in the matrix $\mathbf{W}$. We call
this as output-balanced granularity.

###  2.2 GLU Variants for Transformer

Gated Linear Units (GLU) (Dauphin et al., 2017) are formed by the element-wise
multiplication of two linear projections, with a sigmoid function applied to
one projection before the multiplication. Shazeer (2020) suggests an
alternative design for the Transformer’s MLP layer that incorporates GLU
variants, effectively replacing the conventional first linear transformation
and activation function. More formally, we denote the $d_{\text{hidden}}$ as
the dimension of the model’s hidden state and $d_{\text{int}}$ as intermediate
dimension of the MLP module. For
$\mathbf{x}\in\mathbb{R}^{d_{\text{hidden}}}$,
$\mathbf{W_{1}}\in\mathbb{R}^{d_{\text{hidden}}\times d_{\text{int}}}$,
$\mathbf{W_{2}}\in\mathbb{R}^{d_{\text{hidden}}\times d_{\text{int}}}$,
$\mathbf{W_{3}}\in\mathbb{R}^{d_{\text{int}}\times d_{\text{hidden}}}$, and
activation function $\sigma(\cdot)$, each MLP layer produces
$\mathbf{z}\in\mathbb{R}^{d_{\text{hidden}}}$ by evaluating

$\mathbf{y}=\sigma(\mathbf{x}\mathbf{W_{1}})\otimes\mathbf{x}\mathbf{W_{2}}$
(2) $\mathbf{z}=\mathbf{y}\mathbf{W_{3}}$ (3)

We call $\mathbf{W_{1}},\mathbf{W_{2}},\mathbf{W_{3}}$ linear projections as
Gate-Proj, Up-Proj, and Down-Proj, respectively. GLU variants that use Swish
(Ramachandran et al., 2017) and ReLU (Glorot et al., 2011) activation
functions in Eq.(2) are called SwiGLU and ReGLU, respectively. SwiGLU is most
widely used in the recent LLMs.

###  2.3 Revisiting the Effectiveness of Magnitude Pruning

Sun et al. (2023) conducted ablation studies among different granularities of
magnitude pruning, including input-balanced granularity, which sorts weights
connecting to the same input neuron. Compared with Wanda pruning, input-
balanced magnitude pruning is just slightly worse.

Here, we further investigate the effectiveness of input-balanced magnitude
pruning for the MLP module. We divide the SwiGLU-based MLP module into two
groups, input projections (Gate-Proj and Up-Proj) and output projection (Down-
Proj). We prune those two groups separately and compare Wanda pruning with
input-balanced magnitude pruning. Table 1 shows that input-balanced magnitude
pruning is surprisingly better than Wanda pruning at higher sparsity ratios
for input projections pruning. For the output projection, input-balanced
magnitude pruning is consistently worse than Wanda pruning. This suggests the
diminished importance of outlier features within input projections compared to
the output projection. We analyze the layerwise intermediate activation
outlier distribution in Appendix B. However, in the older GeLU-based MLP
expansion projection, we observe Wanda can achieve much better performance as
shown in Appendix C, suggesting this is a special property of GLU variants.

Table 1:  Comparison of Wanda pruning and Input-balanced Magnitude pruning, perplexity with LLaMA2-7B on WikiText | Up+Gate Projection | Down Projection
---|---|---
Sparsity | 50% | 60% | 70% | 50% | 60% | 70%
Wanda | 6.05 | 7.12 | 12.42 | 5.76 | 6.11 | 6.97
Magnitude | 6.12 | 7.10 | 10.52 | 5.87 | 6.56 | 9.03

###  2.4 Output-balanced Pruning Limitations

N:M sparsity pattern offers notable speed improvements on recent NVIDIA GPUs
(Mishra et al., 2021; Kurtic et al., 2023). It is specified that every group
of M consecutive weights must include N zeros. SparseGPT (Frantar & Alistarh,
2023) is not explicitly designed for output-balanced pruning granularity. When
converting into an N:M sparsity pattern, SparseGPT forces every M consecutive
weight in each row to have exactly N zeros. In such cases, SparseGPT (Frantar
& Alistarh, 2023) and Wanda (Sun et al., 2023) unanimously remove the same
amount of weights for each output. Such methods often emphasize the
significance of individual components within the weight matrix and overlook
inter-dependencies to other layers in the network. For MLP input projections
pruning, an equal amount of weights corresponding to each intermediate neuron
are removed. However, for output projection based on SparseGPT and Wanda
pruning metrics, the weights connecting to intermediate neurons with larger
activations are more likely to be reserved leading to a structural mismatch.

##  3 Dependency-aware Semi-structured Sparsity

In this section, we introduce Dependency-aware Semi-structured Sparsity (DaSS)
for pruning MLP, which incorporates structural dependency into weight
magnitude-based unstructured pruning method. An overview of DaSS and its
comparison with the existing pruning method is shown in Figure 1.

Here, we denote the transposed weight matrices
$\mathbf{W_{1}},\mathbf{W_{2}},\mathbf{W_{3}}$ in Eqs. (2) and (3) as
$\mathbf{W^{(1)}}$, $\mathbf{W^{(2)}}$ and $\mathbf{W^{(3)}}$ , respectively.
For $\mathbf{W^{(1)}}$ and $\mathbf{W^{(2)}}$, the shapes are
$(d_{\text{int}},d_{\text{hidden}})$. For $\mathbf{W^{(3)}}$, the shape is
$(d_{\text{hidden}},d_{\text{int}})$.

#### Structure Dependency in MLP.

In dependency-based structured pruning (Ma et al., 2023; Fang et al., 2023),
the initial step is dedicated to recognizing groups of interconnected
structures within the model. In terms of SwiGLU-based MLP module pruning,
there are three projection matrices, all weights connecting to an identical
intermediate neuron are collectively classified into the same dependency
group. When pruning weights in $\mathbf{W^{(1)}_{i,:}}$, it is essential to
also prune all the weights in $\mathbf{W^{(2)}_{i,:}}$ and
$\mathbf{W^{(3)}_{:,i}}$ to maintain the structural consistency of the pruned
neural network. In DaSS pruning, instead of aggressively pruning all the
weights in less important dependency groups, we incorporate such structural
coordination into unstructured pruning. If all weights in
$\mathbf{W^{(3)}_{:,i}}$ are emphasized importance through additional
importance indicator, such as augmenting corresponding input activations in
Eq.(1), then the weights in $\mathbf{W^{(1)}_{i,:}}$ and
$\mathbf{W^{(2)}_{i,:}}$ should also be emphasized similar importance.

#### Incorporating Dependency into Weight Importance.

In dependency-based pruning, we assess the importance of each weight and then
aggregate the importance scores within the same group as the group importance
score. Consequently, each weight within the same dependency group shares a
consistent importance score, ensuring their simultaneous retention or
elimination. The importance score of each weight is equal to the group
importance score. In DaSS pruning, we take both group importance and weight
magnitude into consideration to evaluate the importance of each weight. LLM-
pruner (Ma et al., 2023) evaluates the group importance using gradient-based
methods. However, computing gradient for LLMs will introduce a significant
amount of memory cost and it is less practical for larger models. The existing
unstructured pruning methods (Sun et al., 2023; Frantar & Alistarh, 2023) are
more efficient than gradient-based methods. By prioritizing the weights linked
to outliers in intermediate activations, Wanda outperforms magnitude pruning
in Down-Proj pruning. To minimize the impact on these critical outliers, it
would be beneficial to also give greater significance to the weights that lead
to outlier generation in both Gate-Proj and Up-Proj projections. Thus, we use
the norm of intermediate activations $\|\mathbf{y}\|_{2}$ as the group
importance indicator. The computation of activation norms is straightforward
without introducing more computation and memory overhead than computing
gradient. To assign larger importance scores on weights corresponding to more
important groups, we evaluate weight importance based on the product of weight
magnitude and corresponding group importance score. More specifically, for the
Gate-Proj and Up-Proj , the importance score attributed to a specific weight
$\mathbf{W}_{i,j}^{k}$ is determined as follows:

$\mathbf{I}_{i,j}^{k}=\left|\mathbf{W}_{i,j}^{k}\right|\cdot\|\mathbf{y}_{i}\|_{2}^{\alpha}$
(4)

Where $k=1,2$. We introduce a hyper-parameter group importance strength
$\alpha$. Our empirical results show that $\alpha=0.5$ is a well-balanced
point for different models and datasets. For the Down-Proj projection pruning,
we directly augment group importance into weight magnitude without additional
hyper-parameter. For Down-Proj matrix, the importance score of weight
$\mathbf{W}_{i,j}^{3}$ is determined as follows:

$\mathbf{I}_{i,j}^{3}=\left|\mathbf{W}_{i,j}^{3}\right|\cdot\|\mathbf{y}_{j}\|_{2}$
(5)

In Down-Proj pruning, the pruning metric is the same as done with Wanda (Sun
et al., 2023). By augmenting intermediate activations into all three weight
importance matrices, DaSS inherently assigns greater emphasis to weights
corresponding to intermediate activation outliers, thereby facilitating more
nuanced structural coordination among the entire MLP module.

#### Pruning Granularity.

Pruning LLaMA models in finer granularity can improve the performance (Sun et
al., 2023). To incorporate intermediate activations into SwiGLU-based MLP
pruning, each weight in the same comparison group should correspond to
different intermediate activations. In Section 2.3, we show that input-
balanced magnitude pruning can still achieve competitive results for Gate-Proj
and Up-Proj pruning. In such pruning granularity, we can augment intermediate
activations into Gate-Proj and Up-Proj pruning metric. In input-balanced
pruning, we remove $s\%$ of the weights linked to each input for a
predetermined sparsity ratio of $s\%$ based on weight importance scores. DaSS
uses output-balanced sparsity for Down-Proj pruning, which is the same as
Wanda.

#### Extension to N:M Sparsity.

The DaSS pruning design allows for easy adaptation to the N:M sparsity
pattern. For Gate-Proj and Up-Proj the N: M sparsity pattern is formed on an
input-balanced basis. This means that for weights connecting to each input
neuron, out of every group of M consecutive weights, there are exactly N zeros
included. For Down-Proj the N: M sparsity pattern is formed on an output-
balanced basis.

#### Discussion.

In summary, our DaSS method offers multiple appealing aspects for pruning
LLMs:

  1. 1.

It retains the fundamental simplicity inherent in Wanda pruning method.
Without weight updating, it still matches the performance of SparseGPT even at
high sparsity as demonstrated in Section 4.4. This demonstrates the
consistently effective and efficient capability of the DaSS method in
identifying sparse neural networks.

  2. 2.

Unlike SparseGPT and Wanda that use input + intermediate activations for MLP
pruning, DaSS only uses intermediate activations. By using intermediate
activations as group importance indicator, DaSS prunes MLP module in a more
comprehensive view that captures the collective importance of all the weights
connecting to each intermediate neuron.

  3. 3.

DaSS effectively explores the balance between unstructured pruning’s
flexibility and the structural coherence in dependency-based structured
pruning.

##  4 Experiments

###  4.1 Settings

#### Models.

DaSS’s performance is evaluated over popular LLMs using SwiGLU, including the
LLaMA2 model family (Touvron et al., 2023b), which has models with parameters
ranging between 7 billion and 70 billion, and also the Mistral-7B model (Jiang
et al., 2023). Among them, LLaMA2-70B and Mistral-7B use grouped-query
attention (Ainslie et al., 2023), the MLP module parameter accounts for around
80% of the total model parameters. To test the generalization ability to ReGLU
models, we use ReluLLaMA (Team, 2023), which is fine-tuned using ReGLU variant
(Shazeer, 2020; Mirzadeh et al., 2023) based on LLaMA2 with small accuracy
loss. The model configuration details are in Appendix 7. We access the public
checkpoints of the involved models provided by the HuggingFace Transformers
library (Wolf et al., 2019).

#### Baseline Approaches.

As shown in Sun et al. (2023), the pre-LLM methods become less effective for
LLMs. We compare the performance with two LLM-specific one-shot pruning
approaches, SparseGPT (Frantar & Alistarh, 2023) and Wanda (Sun et al., 2023).
Those baseline methods utilize uniform layerwise sparsity that can be easily
converted into hardware-friendly N:M sparsity pattern. We used the same
calibration data set as SparseGPT and Wanda in their model pruning processes,
consisting of 128 sequences of 2048 tokens each, randomly selected from the
first shard of the C4 dataset (Raffel et al., 2020).

#### Evaluation.

To comprehensively evaluate the efficacy of our proposed method, three unique
metrics are utilized to evaluate the performance of the pruned models: (1)
perplexity (PPL) of language modeling (2) zero-shot accuracy on 5 commonsense
reasoning tasks (3) 5-shot accuracy on Massive Multitask Language
Understanding (MMLU) (Hendrycks et al., 2021). Although perplexity has been
regarded as a consistent and reliable metric for measuring compressed models
in previous works (Dettmers & Zettlemoyer, 2023; Frantar & Alistarh, 2023), a
recent study indicates that perplexity fails to capture the change in
capabilities of compressed LLMs on performing knowledge-intensive MMLU tasks
(Jaiswal et al., 2023). For perplexity evaluation, we use the validation
dataset of WikiText2 (Merity et al., 2017). For zero-shot commonsense
reasoning tasks, we choose five widely used tasks for accuracy evaluation: ARC
(Easy and Challenge) (Clark et al., 2018), HellaSwag (Zellers et al., 2019),
PiQA (Bisk et al., 2020), and WinoGrande (Sakaguchi et al., 2021), implemented
in the Lm-Evaluation-Harness (Gao et al., 2021). For 5-shot MMLU evaluation,
we use Chain-of-Thought Hub (Fu et al., 2023) which is based on the official
implementation of MMLU (Hendrycks et al., 2021). The MMLU encompasses 57
tasks, spanning from STEM, Humanities, Social Sciences, among others and we
report the mean accuracy of 57 tasks. We evaluate the (1) perplexity of all
the aforementioned models. To fully demonstrate the task-wise performance in
different sparsity patterns, we report the (2) and (3) task performance of the
largest LLaMA2-70B model.

Table 2: WikiText perplexity of pruned LLaMA-2 and Mistral models

Model Family | Mistral | Llama2
---|---|---
Size | 7B | 7B | 13B | 70B
Method | MLP Sparsity |  PPL ($\downarrow$)
Dense | - | 5.25 | 5.47 | 4.88 | 3.32
SparseGPT | 4:8 | 7.36 | 7.33 | 6.29 | 4.66
Wanda | 4:8 | 7.38 | 7.63 | 6.42 | 4.59
DaSS | 4:8 | 7.06 | 7.26 | 6.16 | 4.41
SparseGPT | 2:4 | 8.86 | 8.72 | 7.30 | 5.32
Wanda | 2:4 | 9.24 | 9.55 | 7.68 | 5.35
DaSS | 2:4 | 8.39 | 8.48 | 6.90 | 4.91
SparseGPT | 50% | 6.20 | 6.38 | 5.60 | 4.06
Wanda | 50% | 6.25 | 6.50 | 5.67 | 4.07
DaSS | 50% | 6.15 | 6.44 | 5.59 | 4.00

Table 3: Downstream tasks performance of LLaMA2-70B model in different sparsity pattern. The MMLU scores of dense LLaMA2-70B and LLaMA2-34B reported in Touvron et al. (2023b) are 68.9 and 62.6 respectively. Sparsity | Methods |  Commonsense Reasoning (0 shot) ($\uparrow$) |  World Knowledge ($\uparrow$)
---|---|---|---
PIQA | HellaSwag | Winogrande | ARC-e | ARC-c | Average | MMLU (5 shot)
Dense | - | 82.15 | 66.05 | 77.98 | 82.55 | 54.35 | 72.62 | 69.10
4:8 | SparseGPT | 80.52 | 61.00 | 77.03 | 79.85 | 50.60 | 69.80 | 60.24
Wanda | 80.47 | 61.85 | 75.45 | 80.10 | 50.00 | 69.57 | 59.69
DaSS | 80.79 | 62.70 | 76.09 | 81.20 | 51.19 | 70.39 | 60.86
DaSS+skip 1/4 | 81.34 | 63.10 | 76.09 | 80.05 | 50.68 | 70.25 | 65.82
2:4 | SparseGPT | 79.00 | 59.00 | 76.64 | 78.50 | 47.87 | 68.20 | 56.99
Wanda | 79.22 | 59.25 | 74.66 | 78.90 | 47.01 | 67.81 | 56.41
DaSS | 79.70 | 60.00 | 74.82 | 79.65 | 49.15 | 68.66 | 57.53
DaSS+skip 1/4 | 80.63 | 62.05 | 74.51 | 80.00 | 49.40 | 69.32 | 64.36
50% | SparseGPT | 81.50 | 64.20 | 78.45 | 81.90 | 52.73 | 71.76 | 64.52
Wanda | 81.01 | 64.30 | 77.35 | 80.95 | 52.05 | 71.13 | 62.72
DaSS | 81.18 | 64.60 | 77.90 | 81.35 | 51.71 | 71.35 | 63.27
DaSS+skip 1/4 | 81.23 | 64.45 | 77.27 | 81.35 | 52.81 | 71.42 | 66.81

#### Sparsity.

In the less interpretable perplexity evaluation, we only prune the MLP layers.
In the task-wise evaluation of LLaMA2-70B model, we prune both attention and
MLP modules in accordance with previous works to better understand the
performance gap between the pruned and original models. For attention module
pruning, we use Wanda method, which is more efficient than SaprseGPT. In
LlaMA2-70B, as the MLP module accounts for more than 80% total parameters, we
observe the choice of attention pruning has almost no impact on the final
performance. We apply a uniform sparsity ratio across all the pruned layers
and conduct evaluations using three sparsity types: unstructured sparsity, and
semi-structured sparsities of 4:8 and 2:4.

###  4.2 Language Modeling

This evaluation involved a detailed examination of all the opened LLaMA2
models and Mistral 7B model. In perplexity evaluation, as shown in Table 2,
our method consistently achieves better performance than SparseGPT and Wanda
in more constrained and practical N:M sparsity pattern. As indicated by Sun et
al. (2023), where weight updates can improve the performance in N:M sparsity
pattern, our method shows superior performance even without computationally
expensive weight updates. For Mistral 7B and LLaMA2 70B models with larger MLP
layers, our method also outperforms SparseGPT in unstructured sparsity.

###  4.3 Downstream Tasks

While perplexity has been a popular metric in earlier LLM compression works
(Frantar & Alistarh, 2023; Dettmers & Zettlemoyer, 2023), it essentially
measures the confidence of a language model in text prediction and doesn’t
always reflect its proficiency in performing downstream tasks (Jaiswal et al.,
2023). Apart from assessing perplexity, we comprehensively evaluate the
performance of pruned LLaMA2-70B models in downstream tasks.

Comparison with Baselines. In Table 3, we present the performance of different
sparse LLaMA2-70B models on downstream tasks with prompting. The results show
that our method outperforms SparseGPT and Wanda in most tasks at semi-
structured N:M sparsity pattern including the aggregated task MMLU. The only
exception is that SparseGPT outperforms both Wanda and DaSS in Winogrande
task. For unstructured sparsity, our method outperforms Wanda sharing the same
complexity level. It is noteworthy that the improvement of DaSS over Wanda
becomes more pronounced in the challenging MMLU task, where it achieves an
increase in accuracy of 1.17 and 1.12 for the 4:8 and 2:4 sparsity patterns,
respectively.

Partial N:M Sparsity & Layer Sensitivity. In accordance with the observations
with Jaiswal et al. (2023), we see a considerable performance degradation in
knowledge-intensive MMLU task, in which only unstructured 50% sparsity models
outperform the dense LLaMA2-34B model. As GPU plays a more important role in
larger model inference, it is essential to improve the performance of pruned
models in hardware-friendly N:M sparsity. Uniform reduction of the overall
sparsity level is not feasible for N:M sparsity, Frantar & Alistarh (2023)
suggests a specific subset of layers can be chosen for full N:M
sparsification. Here we skip pruning 1/4 consecutive layers (20 layers) and
result in a final 37.5% sparsity ratio. We use the first 10 tasks of MMLU to
study pruning sensitivity. We divide the model into 4 consecutive parts and
study skipping pruning each part. As shown in Table 4, the earlier layers are
more sensitive than the later ones in knowledge-intensitive tasks, which is
contradictory to the findings in Frantar & Alistarh (2023) using perplexity.

Table 4: MMLU subset accuracy after skipping pruning 20 layers at various start indices in 4:8 sparsity. Start Index | 0 | 10 | 20 | 40 | 60 | Dense
---|---|---|---|---|---|---
Acc (%) | 60.12 | 61.72 | 59.75 | 56.54 | 56.83 | 64.86

We continue to search for the better skipping layers with the start layer
index range in [0,20]. We found that starting skipping from layer 10 can
achieve the best performance in the subset. Then we test its results in the
full MMLU tasks. As shown in Table 3, skipping sensitive 1/4 layers can
significantly improve the performance of pruned models, especially for N:M
sparsity. We can achieve sparse models that perform better than LLaMA2-34B.
Although partial N:M sparsity models have more parameters than smaller dense
models, training many smaller dense models like LLaMA2-34B is still
computationally expensive. Efficient post-training pruning enables us to
easily adjust the accuracy-efficiency trade-off in real-world applications.

An extensive line of work (Haviv et al., 2023; Meng et al., 2022; Geva et al.,
2023) discussed how factual knowledge is stored in Transformer parameters, and
early to middle MLP sublayers are demonstrated to be crucial for memorized
predictions. Interestingly, the layer sensitivity results have a strong
correlation with the findings in those works. We also apply the searched layer
index in MMLU tasks to commonsense reasoning tasks, and we see fewer
performance gains, which highlights the importance of task-specific pruning.

###  4.4 Generalization to ReGLU

We test the generalization ability to another GLU variant, ReGLU. In Table 5,
we also observed DaSS outperform SparseGPT and Wanda in N:M sparsity pattern.
This demonstrates the generalization ability of our method to other GLU
variants. Since augmenting input activations is important for the expanding
projection of non-GLU MLP modules, our methods cannot achieve better
performance than Wanda in older LLMs that do not use GLU variants.

Table 5: WikiText perplexity of ReluLLaMA models Size | 7B | 13B
---|---|---
Method | MLP Sparsity |  PPL ($\downarrow$)
Dense | - | 6.15 | 5.50
SparseGPT | 4:8 | 8.44 | 7.45
Wanda | 4:8 | 9.21 | 7.74
DaSS | 4:8 | 8.19 | 7.16
SparseGPT | 2:4 | 10.26 | 8.98
Wanda | 2:4 | 12.68 | 9.87
DaSS | 2:4 | 9.61 | 8.18
SparseGPT | 50% | 7.22 | 6.39
Wanda | 50% | 7.52 | 6.53
DaSS | 50% | 7.24 | 6.40

###  4.5 Performance Analysis

#### Sparsity Variation.

Figure 2 illustrates a comparison of the mean zero-shot task accuracy at
varying levels of sparsity for the MLP component of the LLaMA2-70B model. It’s
evident that DaSS pruning maintains competitive performance which closely
matches that of SparseGPT across the entire range of sparsity ratios tested.
Notably, DaSS achieves such performance without the need for weight updates,
suggesting its effectiveness and efficiency in locating sparse neural
networks. On the other hand, output-balanced Wanda pruning shows a significant
decline in accuracy as the sparsity ratio increases. This suggests that Wanda
pruning may suffer from structural mismatch issues within the MLP layer, which
become more pronounced at higher sparsity levels. As a result, the neural
network’s performance deteriorates, potentially leading to a dysfunctional
model at extreme sparsity ratios.

Figure 2: Mean zero-shot tasks performance at different MLP sparsity ratios.

#### Robustness to calibration samples.

Ashkboos et al. (2023) observes that the intermediate activations of SwiGLU-
based MLP layers exhibit high variance, primarily caused by the Hadamard
product of the preceding two outputs. This leads to diminished accuracy in
4-bit quantization. In Figure 3, we present how varying the number of
sequences sampled for calibration affects the performance of pruning methods.
Even though only using intermediate activations, our method demonstrates
minimal sensitivity to changes in the number of calibration samples.

Figure 3: Robustness to calibration samples.

###  4.6 Running Time Analysis

#### Pruning speed

As observed in our preceding study on layer sensitivity, the efficiency of the
pruning process is crucial when numerous iterations are necessary to reach the
desired performance for specific tasks. For DaSS and Wanda, the computational
complexity is quantified as $O(d^{2})$, while SparseGPT exhibits a higher
complexity of $O(d^{3})$. We recorded the overall time taken for pruning MLP
layers, not including the forward pass process, in accordance with the
approach described by Sun et al. (2023). We use a single A6000 48GB GPU to
prune the 7B and 13B models, and use 8 A100 40GB GPUs to prune the larger 70B
model.

As demonstrated in Table 6, the computational overhead incurred by DaSS is
minimal, especially when compared to SparseGPT. While the processing speed of
DaSS is marginally slower than that of Wanda, this can be attributed to
Wanda’s more efficient approach of sorting weights exclusively along the last
dimension. Nonetheless, given the substantial improvements in accuracy
achieved by DaSS, this additional computational time is justifiable and
beneficial.

Table 6: Pruning speed (in seconds) comparison. Method | Mistral | LLaMA-2
---|---|---
7B | 7B | 13B | 70B
SparseGPT | 155 | 185 | 206 | 1092.76
Wanda | 0.60 | 0.54 | 0.85 | 14.13
DaSS | 0.81 | 0.81 | 1.02 | 20.70

##  5 Related Work

Pruning LLM. Neural network pruning in LLM can be broadly categorized into two
groups: structured pruning (Ma et al., 2023; Zhang et al., 2023) and
unstructured pruning (Frantar & Alistarh, 2023; Sun et al., 2023). Ma et al.
(2023) proposes a dependency detection algorithm to detect and prune non-
critical grouped structures followed by LoRA fine-tuning. Although structured
pruning can usually have better hardware efficiency, the accuracy drops a lot
even at a low compression rate. Unstructured pruning can yield a higher
compression rate and achieve acceleration on Nvidia’s GPUs by employing a
hardware-friendly N:M sparsity pattern. SparseGPT (Frantar & Alistarh, 2023)
leverages the Hessian inverse for pruning and reduces reconstruction error of
dense and sparse weights by subsequent weight updates. Wanda (Sun et al.,
2023) employs an efficient method that augments input activations into weight
magnitudes, and matches the performance of SparseGPT at medium sparsity. Our
work incorporates dependency information into unstructured pruning, achieving
a novel pruning paradigm.

Inherent Sparsity of Transformer MLP. Interestingly, sparsity within the MLP
activations of trained Transformer-based models occurs innately even without
applying explicit regularizations or constraints (Zhang et al., 2022; Li et
al., 2023; Dong et al., 2023). Such a phenomenon is prevalent in learned
Transformers, including other zero-saturating functions. Liu et al. (2023);
Mirzadeh et al. (2023); Zhang et al. (2022) achieve actual LLM inference
speedup by only performing computation corresponding to the activating neuron
for a given input. They do not actually reduce the model size since they
mainly reduce I/O and computation latency in a selective weights loading
manner, and thus, these methods are less applicable in large batch-size
inference settings. Our work investigates the weight sparsity in MLP module by
considering corresponding intermediate activations.

Outlier-dependent LLM Compression. Outlier features, defined as features with
magnitudes substantially larger than others, are a notable characteristic of
LLMs (Dettmers et al., 2022). Despite making up only a small fraction of all
feature dimensions, these outliers play a critical role in attention and
predictive performance. Such observation has impeded the development of some
LLM-specific quantization methods (Dettmers et al., 2022; Xiao et al., 2023;
Lin et al., 2023; Ashkboos et al., 2023) to handle outliers more effectively.
Wanda (Sun et al., 2023) broadens these insights, revealing that outlier
features are significant in deciding weight importance when pruning LLMs. Our
analysis diverges from conventional wisdom by demonstrating that, in the
context of SwiGLU-based MLP input projections, the influence of outlier
features is not as pronounced as previously assumed, prompting a reevaluation
of their role in LLM compression strategies.

##  6 Conclusion

We propose the Dependency-aware Semi-structured Sparsity (DaSS) method which
effectively addresses the challenges of pruning SwiGLU-based MLP modules LLMs.
DaSS strikes a unique balance between the adaptability of unstructured pruning
and the orderliness of structured pruning. By leveraging the MLP intermediate
activation norms as the group importance indicator, we develop a novel pruning
metric that assesses weight importance in a more structurally consistent
manner. Empirical evaluations on the Mistral and LLaMA2 model families
demonstrate that DaSS surpasses state-of-the-art LLM pruning methods like
SparseGPT and Wanda in achieving hardware-friendly N:M sparsity patterns. Our
research further reveals that selectively skipping the pruning of earlier
layers can lead to significant performance improvements in knowledge-intensive
tasks. The efficiency of the pruning process becomes paramount when multiple
iterations are essential to tailor and optimize the sparsity of LLMs for
varied tasks. This underscores the potential of our DaSS method in
facilitating rapid and task-specific model pruning to achieve optimal
performance.

## Impact Statements

The present study is focused on advancing the field of large language model
deployment with an emphasis on efficiency. While there are numerous societal
implications of our work, we believe none require specific emphasis in this
context.

## References

  * Ainslie et al. (2023) Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S.  Gqa: Training generalized multi-query transformer models from multi-head checkpoints.  _arXiv preprint arXiv:2305.13245_ , 2023.
  * Ashkboos et al. (2023) Ashkboos, S., Markov, I., Frantar, E., Zhong, T., Wang, X., Ren, J., Hoefler, T., and Alistarh, D.  Towards end-to-end 4-bit inference on generative large language models.  _the 3rd NeurIPS Workshop on Efficient Natural Language and Speech Processing (ENLSP)_ , 2023.
  * Bisk et al. (2020) Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al.  Piqa: Reasoning about physical commonsense in natural language.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pp. 7432–7439, 2020.
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chen et al. (2021) Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ , 2021.
  * Chowdhery et al. (2023) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstern, K., Eck, D., Dean, J., Petrov, S., and Fiedel, N.  Palm: Scaling language modeling with pathways.  _Journal of Machine Learning Research_ , 24(240):1–113, 2023.  URL http://jmlr.org/papers/v24/22-1144.html.
  * Clark et al. (2018) Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_ , 2018.
  * Cobbe et al. (2021) Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Dauphin et al. (2017) Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D.  Language modeling with gated convolutional networks.  In _International conference on machine learning_ , pp. 933–941. PMLR, 2017.
  * Dettmers & Zettlemoyer (2023) Dettmers, T. and Zettlemoyer, L.  The case for 4-bit precision: k-bit inference scaling laws.  In _International Conference on Machine Learning_ , pp. 7750–7774. PMLR, 2023.
  * Dettmers et al. (2022) Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L.  GPT3.int8(): 8-bit matrix multiplication for transformers at scale.  In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=dXiGWqBoxaD.
  * Dong et al. (2023) Dong, H., Chen, B., and Chi, Y.  Towards structured sparsity in transformers for efficient inference.  In _Workshop on Efficient Systems for Foundation Models @ ICML2023_ , 2023.  URL https://openreview.net/forum?id=c4m0BkO4OL.
  * Fang et al. (2023) Fang, G., Ma, X., Song, M., Mi, M. B., and Wang, X.  Depgraph: Towards any structural pruning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 16091–16101, 2023.
  * Frantar & Alistarh (2023) Frantar, E. and Alistarh, D.  SparseGPT: Massive language models can be accurately pruned in one-shot.  In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 10323–10337. PMLR, 23–29 Jul 2023.  URL https://proceedings.mlr.press/v202/frantar23a.html.
  * Frantar et al. (2023) Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D.  OPTQ: Accurate quantization for generative pre-trained transformers.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=tcbBPnfwxS.
  * Fu et al. (2023) Fu, Y., Ou, L., Chen, M., Wan, Y., Peng, H., and Khot, T.  Chain-of-thought hub: A continuous effort to measure large language models’ reasoning performance.  _arXiv preprint arXiv:2305.17306_ , 2023.
  * Gao et al. (2021) Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., Phang, J., Reynolds, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A.  A framework for few-shot language model evaluation, September 2021.  URL https://doi.org/10.5281/zenodo.5371629.
  * Geva et al. (2023) Geva, M., Bastings, J., Filippova, K., and Globerson, A.  Dissecting recall of factual associations in auto-regressive language models.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023.  URL https://openreview.net/forum?id=F1G7y94K02.
  * Glorot et al. (2011) Glorot, X., Bordes, A., and Bengio, Y.  Deep sparse rectifier neural networks.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pp. 315–323. JMLR Workshop and Conference Proceedings, 2011.
  * Han et al. (2015) Han, S., Pool, J., Tran, J., and Dally, W.  Learning both weights and connections for efficient neural network.  _Advances in neural information processing systems_ , 28, 2015.
  * Haviv et al. (2023) Haviv, A., Cohen, I., Gidron, J., Schuster, R., Goldberg, Y., and Geva, M.  Understanding transformer memorization recall through idioms.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pp. 248–264, 2023.
  * Hendrycks et al. (2021) Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=d7KBjmI3GmQ.
  * Jaiswal et al. (2023) Jaiswal, A., Gan, Z., Du, X., Zhang, B., Wang, Z., and Yang, Y.  Compressing llms: The truth is rarely pure and never simple.  _arXiv preprint arXiv:2310.01382_ , 2023.
  * Jiang et al. (2023) Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_ , 2023.
  * Kurtic et al. (2023) Kurtic, E., Kuznedelev, D., Frantar, E., Goin, M., and Alistarh, D.  Sparse finetuning for inference acceleration of large language models.  _arXiv preprint arXiv:2310.06927_ , 2023.
  * Li et al. (2023) Li, Z., You, C., Bhojanapalli, S., Li, D., Rawat, A. S., Reddi, S. J., Ye, K., Chern, F., Yu, F., Guo, R., and Kumar, S.  The lazy neuron phenomenon: On emergence of activation sparsity in transformers.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=TJ2nxciYCk-.
  * Lin et al. (2023) Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S.  Awq: Activation-aware weight quantization for llm compression and acceleration.  _arXiv preprint arXiv:2306.00978_ , 2023.
  * Liu et al. (2023) Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., et al.  Deja vu: Contextual sparsity for efficient llms at inference time.  In _International Conference on Machine Learning_ , pp. 22137–22176. PMLR, 2023.
  * Ma et al. (2023) Ma, X., Fang, G., and Wang, X.  Llm-pruner: On the structural pruning of large language models.  _Advances in Neural Information Processing Systems_ , 2023.
  * Meng et al. (2022) Meng, K., Bau, D., Andonian, A., and Belinkov, Y.  Locating and editing factual associations in gpt.  _Advances in Neural Information Processing Systems_ , 35:17359–17372, 2022.
  * Merity et al. (2017) Merity, S., Xiong, C., Bradbury, J., and Socher, R.  Pointer sentinel mixture models.  In _International Conference on Learning Representations_ , 2017.  URL https://openreview.net/forum?id=Byj72udxe.
  * Mirzadeh et al. (2023) Mirzadeh, I., Alizadeh, K., Mehta, S., Del Mundo, C. C., Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M.  Relu strikes back: Exploiting activation sparsity in large language models.  _arXiv preprint arXiv:2310.04564_ , 2023.
  * Mishra et al. (2021) Mishra, A., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P.  Accelerating sparse deep neural networks.  _arXiv preprint arXiv:2104.08378_ , 2021.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Raffel et al. (2020) Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Ramachandran et al. (2017) Ramachandran, P., Zoph, B., and Le, Q. V.  Swish: a self-gated activation function.  _arXiv: Neural and Evolutionary Computing_ , 2017.  URL https://api.semanticscholar.org/CorpusID:196158220.
  * Sakaguchi et al. (2021) Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y.  Winogrande: An adversarial winograd schema challenge at scale.  _Communications of the ACM_ , 64(9):99–106, 2021.
  * Shazeer (2019) Shazeer, N.  Fast transformer decoding: One write-head is all you need.  _arXiv preprint arXiv:1911.02150_ , 2019.
  * Shazeer (2020) Shazeer, N.  Glu variants improve transformer.  _arXiv preprint arXiv:2002.05202_ , 2020.
  * Sun et al. (2023) Sun, M., Liu, Z., Bair, A., and Kolter, J. Z.  A simple and effective pruning approach for large language models.  _arXiv preprint arXiv:2306.11695_ , 2023.
  * Team (2023) Team, S.  Sparse large language models with relu activation, 2023.
  * Touvron et al. (2023a) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G.  Llama: Open and efficient foundation language models, 2023a.
  * Touvron et al. (2023b) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T.  Llama 2: Open foundation and fine-tuned chat models, 2023b.
  * Wen et al. (2016) Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H.  Learning structured sparsity in deep neural networks.  _Advances in neural information processing systems_ , 29, 2016.
  * Wolf et al. (2019) Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ , 2019.
  * Xiao et al. (2023) Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S.  Smoothquant: Accurate and efficient post-training quantization for large language models.  In _International Conference on Machine Learning_ , pp. 38087–38099. PMLR, 2023.
  * Yin et al. (2023) Yin, L., Wu, Y., Zhang, Z., Hsieh, C.-Y., Wang, Y., Jia, Y., Pechenizkiy, M., Liang, Y., Wang, Z., and Liu, S.  Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity.  _arXiv preprint arXiv:2310.05175_ , 2023.
  * Zellers et al. (2019) Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y.  Hellaswag: Can a machine really finish your sentence?  _arXiv preprint arXiv:1905.07830_ , 2019.
  * Zhang et al. (2023) Zhang, M., Shen, C., Yang, Z., Ou, L., Yu, X., Zhuang, B., et al.  Pruning meets low-rank parameter-efficient fine-tuning.  _arXiv preprint arXiv:2305.18403_ , 2023.
  * Zhang et al. (2022) Zhang, Z., Lin, Y., Liu, Z., Li, P., Sun, M., and Zhou, J.  MoEfication: Transformer feed-forward layers are mixtures of experts.  In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), _Findings of the Association for Computational Linguistics: ACL 2022_ , pp. 877–890, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-acl.71.  URL https://aclanthology.org/2022.findings-acl.71.

##  Appendix A Model Configurations

Here are the configurations of the models used in the paper. We don’t use
LLaMA2-34B since it was not released. ReluLLaMA uses the same configuration as
LLaMA2, the only difference is the activation function. Here is the link to
ReluLLaMA:

ReluLLaMA-7B: https://huggingface.co/SparseLLM/ReluLLaMA-7B
ReluLLaMA-13B: https://huggingface.co/SparseLLM/ReluLLaMA-13B

Table 7: Model configurations of Llama2 and Mistral models. Model | Param | Layers | Hidden | Intermediate | Query Heads | KV Head
---|---|---|---|---|---|---
LlaMa2-7B | 7B | 32 | 4096 | 11008 | 32 | 32
LlaMa2-13B | 13B | 40 | 5120 | 13824 | 40 | 40
LlaMa2-70B | 70B | 80 | 8192 | 28672 | 64 | 8
Mistral-7B | 7B | 32 | 4096 | 14336 | 32 | 8

##  Appendix B Activation Outlier in SwiGLU

Figure 4: Layerwise Activation Outlier distribution of LLaMA2-7B

In Section 2.3, we show the importance of augmenting input magnitude is
useful. In this part, we target to identify the existence of outliers in the
intermediate activations of SwiGLU. Directly analyzing it like model hidden
states Dettmers et al. (2022); Xiao et al. (2023) is challenging since it
demonstrates high variance. (Ashkboos et al., 2023). Wanda (Sun et al., 2023)
shows $\ell_{2}$ norm can be a metric for evaluating activations, it shows
effectiveness in Down-Proj pruning. Motivated by Yin et al. (2023), we define
the outlier in MLP intermediate activations by identifying activations whose
$\ell_{2}$ norm magnitude is $T$ times greater than the average MLP activation
norm in the corresponding layer. We set $T$ as an empirical value 7. The
layerwise distribution of the SwiGLU intermediate activation outlier is shown
in Figure 4.

##  Appendix C GeLU-based MLP Input projection

In the older MLP module, unlike GLU variants, there are only two projection
matrices. We investigate magnitude pruning in different granularity for the
MLP input projection matrix. We use the Falcon-7B model, which uses GeLU-based
MLP. ”Layer Magnitude” means we group weights in the entire weight matrix
using magnitude pruning. In Table 8, we surprisingly find the input-balanced
magnitude pruning achieves much worse results. This is in contrast to the
results in GLU variants. The results show the huge significance of input
activations for GeLU-based input projection pruning. Since our DaSS method
needs to group weights per input, our method is less applicable to older MLP
modules. However, GeLU-based MLP is rarely used in the current state-of-the-
art LLMs.

Table 8:  The Wikitext perplexity results of GeLU input projection at different sparsity ratio Sparsity | 50% | 60% | 70%
---|---|---|---
Wanda | 7.17 | 8.72 | 18.95
Output-balanced Magnitude | 8.46 | 15.27 | 498.77
Input-balanced Magnitude | 1708 | 10587 | 13706
Layer Magnitude | 21.82 | 1895 | 12896