# ReLU Strikes Back:
Exploiting Activation Sparsity in Large Language Models

Iman Mirzadeh222 &Keivan Alizadeh &Sachin Mehta &Carlo C Del Mundo Oncel Tuzel
&Golnoosh Samei &Mohammad Rastegari &Mehrdad Farajtabar222

###### Abstract

Large Language Models (LLMs) with billions of parameters have drastically
transformed AI applications. However, their demanding computation during
inference has raised significant challenges for deployment on resource-
constrained devices. Despite recent trends favoring alternative activation
functions such as GELU or SiLU, known for increased computation, this study
strongly advocates for reinstating ReLU activation in LLMs. We demonstrate
that using the ReLU activation function has a negligible impact on convergence
and performance while significantly reducing computation and weight transfer.
This reduction is particularly valuable during the memory-bound inference
step, where efficiency is paramount. Exploring sparsity patterns in ReLU-based
LLMs, we unveil the reutilization of activated neurons for generating new
tokens and leveraging these insights, we propose practical strategies to
substantially reduce LLM inference computation up to three times, using ReLU
activations with minimal performance trade-offs.

$\dagger$$\dagger$footnotetext: Corresponding authors:
<EMAIL_ADDRESS>
##  1 Introduction

The widespread excitement surrounding Large Language Models (LLMs) has sparked
significant interest in leveraging AI across diverse domains [5, 9, 6].
However, realizing the potential of LLMs is challenged by their significant
computational and memory requirements during inference [60, 40, 3]. To enhance
the inference efficiency111In this work, we use FLOPS as a proxy for inference
efficiency. In Appendix B, we demonstrate that for LLMs with activation
sparsity, FLOPS can serve as a good approximation of real-world efficiency due
to the structure inherent in activation sparsity (e.g., skipping the entire
row corresponding to zero activations)., various techniques have been
explored, including quantization [12, 50], speculative decoding [41], pruning
[53, 71], and weight sparsification [20, 15]. Among these techniques,
achieving activation sparsity offers a compelling advantage by providing a
favorable balance between accuracy and speedup, especially on modern hardware
like GPUs [51].

Notably, employing the Rectified Linear Unit (ReLU) activation function [22]
in neural networks is recognized for inducing sparse activations and has been
adopted in various prior works [27, 44, 48, 69]. To reaffirm this property, we
employ the OPT model [80], utilizing ReLU, and measure the sparsity of
activations in the Feed Forward Network (FFN) between the fully connected
layers. As illustrated in Fig. 1(a), all layers exhibit sparsity exceeding
$90\%$. On average, across all layers, this activation sparsity results in
substantial weight transfer (I/O) savings between the GPU and CPU, impacting
$95\%$ of the rows of the down projection layer’s weights (Fig. 1(b)). This
reduction directly translates to computation savings, as for these rows, the
result of the matrix multiplication operation will be zero. Furthermore,
unlike unstructured sparsity (e.g., weight pruning), this type of sparsity is
more hardware-friendly due to zeroing more extensive and structured chunks,
such as rows or columns [36, 51]. For OPT models, this sparsity reduces the
computation required for inference from 6.6G FLOPS (Floating Point Operations
Per Second) to 4.5G FLOPS per token, resulting in a $32\%$ computation saving
(Fig. 1(c)).

However, a recent trend has emerged, favoring variations of ReLU that are
smoother but more complex [28, 64]. These alternatives have gained popularity
due to their slightly faster convergence and improved final accuracy [66]. For
example, PaLM [9] and Llama models [73] adopt SiLU222To be more precise, the
mentioned models use SwiGLU activation function, but in this work, we focus on
the gating module that uses SiLU (Swish) function. [28, 17, 64], while MPT
[56] and Falcon models [2] use GELU [28]. Nonetheless, as demonstrated in Fig.
1(c), when we finetune several pretrained LLMs with different activation
functions, their performance does not change significantly (within a specific
model), while ReLU models require much less computation.

In this paper, we re-evaluate using ReLU for LLMs. We are motivated by the
pragmatic consideration that, in many real-world applications and
computational platforms capable of supporting sparse vector-matrix
multiplications, computational efficiency during _inference_ outweighs the
one-time computational cost incurred during training. We make the following
contributions:

  * •

We demonstrate that when trained from scratch, there is no significant
difference in terms of performance between different activation functions.
However, in terms of computational requirements during inference, ReLU
activations prove significantly lighter (Sec. 3).

  * •

Considering that many modern LLMs (e.g., Llama and Falcon) have been trained
with non-ReLU activations, and it is not cost-effective to train them from
scratch, we investigate fine-tuning these models with ReLU activations. We
show that the models quickly regain their original performance across various
reasoning and reading comprehension tasks (Sec. 4.1). Moreover, we show that
by leveraging the activation sparsity of ReLU layers and inserting additional
ReLU layers after normalization layers, we can further reduce inference FLOPS
by up to threefold (Sec. 4.2).

  * •

In addition to their computational benefits, we present two promising
applications of activation sparsity that can inspire future work. Firstly, we
demonstrate that LLMs with ReLU activations reuse a significant portion of
already activated neurons during token generation, a phenomenon we term
_aggregated sparsity_ (Sec. 5.1). This reusability leads to an inference
speedup for speculative decoding (Sec. 5.2). Additionally, we show that
studying the pre-activations of pretrained LLMs can guide the selection of
unconventional activation functions (e.g., _shifted ReLU_), achieving up to
90% sparsity while maintaining performance similar to ReLU activation (Sec.
5.3).

Overall, we believe our work represents a significant step toward leveraging
the potential of sparse activation functions for faster and more efficient
inference in large language models.

(a) Sparsity of different models

(b) Sparsity for Efficiency

(c) Accuracy vs. Computation

Figure 1:  (a) Activation Sparsity of different pretrained models: ReLU-based
OPTs show significantly higher sparsity. (b) Zeroed out entries after ReLU
save compute in large semi-structured chunks (e.g., rows). (c) Comparison of
inference efficiency and performance of the different models with different
activation functions after fine-tuning: The choice of activation function does
not significantly impact the accuracy, as any of GELU, SiLU, or ReLU can be
used on all three models and achieve the same level of accuracy as the
original activation function. However, using ReLU can provide an additional
benefit of leading to activation sparsity and faster inference.

##  2 Related Works

Activation Functions in Transformers. The original Transformer architecture
[74] was proposed with the ReLU activation function [22], following the
popularity of ReLU at the time. Later, several studies aimed to improve the
ReLU activation function by increasing its smoothness [28] and/or including
parameterized gating mechanisms, such as GELU, SiLU, GLU, and SwiGLU [11, 64].
Earlier studies demonstrated the benefits of these alternatives to ReLU for
transformers [66, 57], but on a small scale (e.g., they trained models up to a
couple of 100M parameters with at most 35B tokens, while in this work, we
train 1B parameter models on more than 100B tokens). However, we believe the
impact of activation functions on performance is marginal, following scaling
laws [37, 31], which state that architectural changes do not significantly
impact performance.

Activation Sparsity. Existing research shows increased sparsity reduces
inference and training times [44, 25, 70, 81, 47, 51]. For instance, Jaszczur
et al. [36] uses ReLU and added a controller to both promote and predict
sparsity, while other works only use prediction modules to predict the
activation masks [51]. We note that the mentioned works assume the pretrained
model has already been using a sparse ReLU activation, and hence, only
training a separate module to predict sparsity could be enough. However, we
note that most LLMs pretrained these days do not use ReLU, and we aim to
bridge this gap. Moreover, these works focus only on a single transformer
architecture while we focus on various architectures so our findings can be
practical. Finally, we show that there is no need to train a separate
prediction module that complicates the computation graph, and using efficient
ReLU layers can be enough.

Speculative Decoding and Sparsity. Speculative decoding combats latency under
memory constraints using a smaller model for token prediction and a larger
model for verification [46, 41]. Investigating its integration with sparsity,
we find activation sparsity exhibits a temporal pattern, enhancing speculative
decoding. We provide guidelines for parameter selection when incorporating
sparsity.

We defer other lines of related works that are orthogonal to our work, such as
model compression techniques, sparse attention methods, and Mixture of Experts
(MoE) to Appendix A.

##  3 Does the Activation Function Impact Performance?

This section first overviews our experimental setup, including models, data,
and evaluations. Then, by training various models from scratch with different
activation functions, we demonstrate that changing activation functions
minimally impacts performance. However, the impact on inference efficiency is
substantial.

###  3.1 Experimental setup

Models. We use open source pretrained models such as OPT [80], Llama (v1)
[73], and Falcon [2] as they use different architectures and pretraining setup
(e.g., attention/FFN structure/normalization, activation functions), allowing
our study covers a wider range of models.

Datasets. We use the RefinedWeb dataset [59], for our pretraining in Sec. 3.2
and finetuning pretrained models in Sec. 4. We chose RefinedWeb because it is
a high-quality subset of Common Crawl, which is often used in the pretraining
phase of LLMs, including Llama, Falcon, and OPT. We also use the validation
split of WikiText [54] for measuring the sparsity and recording preactivation
distributions of various pretrained models. However, our conclusions hold for
other datasets we have tested.

Training and Finetuning. For finetuning the pretrained models, we follow the
original pretraining recipe, except we use a fixed learning rate of 1.5e-5 for
Llama 7B, Falcon 7B, and OPT 6.7B models. In addition, we use the AdamW
optimizer [52] for our finetuning with ZeRO stage 1 [62], where we shard the
optimizer states across different GPUs. For pretraining OPT 1.3B models from
scratch in Sec. 3.2, we follow the OPT training recipe.

Evaluation. For our _performance_ evaluation, we use the few-shot tasks from
Language Model Evaluation Harness [23]. We select these tasks such that they
can measure various abilities of the models (e.g., reading comprehension,
reasoning, etc.), and we aim to be consistent with other works in the
literature to make the comparison easier. Consistent with the other sections,
we compare activation sparsity as a measure of _efficiency_.

Further details regarding the relationship between activation sparsity, FLOPS,
and inference efficiency are discussed in Appendix B.

###  3.2 Training from scratch: performance and sparsity

While the previous literature suggests that non-ReLU variants can improve the
performance of transformers [66, 57], we argue the impact is marginal at best.
To support our claim, we train the OPT 1.3B model from scratch on a hundred
billion tokens of the RefinedWeb datasets with different activation functions,
including ReLU, SiLU, and GELU. All these activation functions can be viewed
as $\text{f(x)}=x\cdot\sigma(\beta x)$, where $\beta$ controls the gating part
(smoothed cutoff threshold) of the activation function (see Fig. 2(a)). For
$\beta=1$, we will have SiLU($x\cdot\sigma(x)$), and $\beta=1.7$ is a good
approximation of GELU. Finally, as $\beta\to\infty$, the activation function
becomes closer to ReLU. To further explore the spectrum of ReLU to SiLUwe add
another one with $\beta=8$.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2:  (top) (a) Shapes of different gating functions over [-5, 5]; (b)
Continuation of (a) where SiLU is comparably larger compared to others; (c)
Sparsity of the FFN with different activations: increasing $\beta$ increases
sparsity. (bottom) when trained from scratch, OPT 1.3 B models using different
activation functions achieve similar performance.

As shown in the bottom row of Fig. 2, the performance of the models is very
similar when using different activation functions. This is consistent with the
scaling laws literature ([37, 31]), which suggests that the performance of
sufficiently large models trained on sufficiently large data depends heavily
on compute and data, not architectural details.

While the performance levels of the different activations are similar, their
activation sparsity levels differ. Here, we define sparsity as the average
sparsity level across all layers for each model. As shown in Fig. 2(c), as we
transition from SiLUto ReLU (increasing $\beta$), the sparsity also increases.
This results from the different gating thresholds, as ReLU drops significantly
more values compared to GELU and SiLU (see Fig. 2(b)). In Appendix D, we
illustrate the evolution of the pre-activation distribution throughout
training.

Overall, the results support our initial claim: non-ReLU activations result in
a negligible performance gain (if any) but a substantial loss in sparsity and
efficiency. While, at times, the performance of GeLU or SiLU might be slightly
higher, ReLU can match it with slightly longer training. We acknowledge that
to compensate for the small gap in performance, we need to pay the one-time
cost of longer training. However, in return, we get a significantly more
sparsity.

Figure 3: Architectural surgeries for _relufication_. In stage 1 we keep the
existing ReLUs (in the case of OPT) or replace the activation function between
up projection and down projections from GELU (Falcon) and SiLU (Llama) to
ReLU. In stage 2, we insert new ReLUs after normalization layers.

##  4 Relufication

While in the previous section, we have seen that the performance does not
depend on the activation function, we note that most of the available
pretrained LLMs are trained with activation functions other than ReLU. Hence,
to incorporate the computational benefits of ReLU activations at inference
time, we perform various architectural surgeries and study the consequences of
such changes.

We present our findings about incorporating ReLU activations into the
pretrained LLMs, a process we refer to as _relufication_. More specifically,
we show that replacing the activation functions of pretrained LLMs with ReLU
is possible, and the performance can be recovered very rapidly during
finetuning. Moreover, we show that we can exploit the sparse ReLU activations,
and by inserting additional ReLU layers after normalization layers, we can
improve inference efficiency, as FLOPS indicates. Finally, we show these
modifications, which are easy to implement, lead to lighter models at
inference time while maintaining comparable performance to the original
pretrained models.

###  4.1 Stage 1: replacing non-ReLU activations

The process of relufication for different pretrained architectures is shown in
Fig. 3. This process can be done in multiple stages, as we describe here. The
first and more intuitive stage replaces non-ReLU activations with ReLU in the
FFN layer. For the Falcon and Llama models, this means replacing GELU and
SiLU, respectively. We note that since OPT models already use ReLU
activations, we keep those unchanged. After finetuning on 30 billion tokens of
the RefinedWeb, Fig. 4 shows that the modified models have significantly more
sparsity in their activations.

(a) Falcon 7B (GELU)

(b) Falcon 7B (ReLU)

(c) Llama 7B (SiLU)

(d) Llama 7B (ReLU)

Figure 4:  Activation sparsity of Falcon and Llama models improves
significantly after _relufication_.

(a) Falcon 7B (GELU)

(b) Falcon 7B (ReLU)

(c) Llama 7B (SiLU)

(d) Llama 7B (ReLU)

Figure 5: The preactivation distribution of pretrained models for Falcon and
Llama does not change significantly during the short finetuning stage of
relufication. The dashed line shows the cutoff point before which the output
is almost zero. Figure 6: Evolution of zero-shot accuracy during finetuning:
The model quickly recovers most of its lost performance due to the
architecture surgery.

In addition to the drastic improvement in activation sparsity, we can make
several notable observations. First, while the shape of preactivation depends
on the pretraining dynamics and architecture, in Fig. 5, we show that it does
not change significantly during the relatively short finetuning stage. As a
result, we can predict the activation sparsity before finetuning, knowing it
will not change significantly. Later in Sec. 5.3 we build on this observation
and propose shifting the preactivation values before applying ReLU and further
increasing the activation sparsity. The stability of the preactivation
distribution may suggest that the behavior of the network does not change
while creating sparse representations. Indeed, we show that after replacing
the activation function with ReLU, finetuned models quickly recover their
performance in Fig. 6. We believe optimizing this process even further (e.g.,
using better finetuning data) is an exciting follow-up direction.

###  4.2 Stage 2: Pushing for more sparsity

In the previous stage, we replaced non-ReLU activations to gain more sparsity.
This leads to the input of _down projection_ layer being sparse, roughly
$30\%$ of the total computation. However, there are other matrix-vector
multiplications in the decoder layer of transformers besides the down
projection. For instance, before the _up projection_ and _gate projections_ of
FFN layer, and _QKV projections_ in the attention layer (see Fig. 3).
Together, the mentioned matrix-vector multiplications consume about $55\%$ of
the total computation.

To this end, we utilize the fact that in modern transformer layers, the input
to both the attention and FFN layers come from a normalization layer, e.g.,
LayerNorm [4] or RMSNorm [78]. These layers can be viewed as a specific form
of MLP, where, instead of applying arbitrary learnable parameters, they learn
to scale inputs. Therefore, we apply ReLU to obtain sparse activations after
normalization layers which we call the _second stage_ of relufication in Fig.
3.

Table 1: Comparing zero-shot performance across several tasks: After
_relufication_ , the activation sparsity of models increases significantly,
hence increased efficiency measured by FLOPS. Within each group, the
performance levels are comparable.

Model (stage) | Input Sparsity (%) |  FLOPS (G) | Zero-Shot Accuracy (%)
---|---|---|---
QKV | DownProj | UpProj | Avg | Arc-E | Arc-C | Hellaswag | BoolQ | PIQA | LAMBADA | TriviaQA | WinoGrande | SciQ
OPT 1.3B | 0 | 96 | 0 | 1.3 | 50.7 | 57.3 | 22.9 | 41.3 | 57.0 | 71.8 | 56.0 | 6.1 | 58.9 | 84.6
OPT 2.7B (s2) | 50 | 96 | 35 | 1.1 | 53.1 | 60.3 | 26.8 | 44.9 | 55.4 | 73.9 | 57.6 | 12.4 | 59.6 | 86.7
OPT 2.7B | 0 | 96 | 0 | 1.8 | 54.5 | 63.3 | 29.2 | 45.8 | 57.6 | 74.2 | 61.4 | 12.3 | 60.8 | 85.9
OPT 6.7B (s2) | 50 | 97 | 40 | 2.8 | 58.6 | 66.5 | 32.2 | 49.1 | 63.0 | 76.4 | 63.3 | 23.8 | 63.1 | 90.3
OPT 6.7B | 0 | 97 | 0 | 4.5 | 59.8 | 68.0 | 32.4 | 50.2 | 68.4 | 75.8 | 67.2 | 20.9 | 65.3 | 90.2
Falcon 7B (s2) | 56 | 95 | 56 | 2.2 | 64.8 | 73.6 | 38.6 | 55.3 | 68.4 | 78.9 | 67.6 | 40.4 | 67.1 | 93.4
Falcon 7B (s1) | 0 | 94 | 0 | 4.1 | 65.2 | 72.2 | 39.1 | 55.4 | 70.6 | 78.4 | 69.2 | 40.5 | 67.5 | 93.1
Falcon 7B | 0 | 1 | 0 | 6.6 | 66.8 | 74.6 | 40.2 | 57.7 | 73.5 | 79.4 | 74.5 | 40.4 | 67.2 | 94.0
Llama 7B (s2) | 51 | 65 | 67 | 2.9 | 66.4 | 73.8 | 39.6 | 54.8 | 69.9 | 77.9 | 70.7 | 48.5 | 68.6 | 93.8
Llama 7B (s1) | 0 | 62 | 0 | 4.8 | 67.1 | 75.2 | 40.1 | 55.2 | 73.4 | 77.7 | 71.5 | 49.6 | 67.1 | 94.2
Llama 7B | 0 | 0 | 0 | 6.6 | 68.4 | 75.5 | 42.1 | 69.9 | 74.8 | 78.7 | 73.1 | 49.9 | 69.8 | 95.4

Table 2: MMLU five-shot accuracy. Models finetuned with different activation
functions have similar performance.* Denotes we replace the SiLU function in
Llama’s SwiGLU activation function with ReLU.

Model | Activation | FLOPS(%) | Avg | Humanities | STEM | Social Sciences | Other
---|---|---|---|---|---|---|---
Falcon 7B | SiLU | 100 | 26.4 | 24.8 | 27.4 | 27.2 | 26.2
Falcon 7B | GELU | 100 | 27.7 | 28.1 | 26.0 | 28.0 | 29.4
Falcon 7B | ReLU | 62 | 27.9 | 26.0 | 26.5 | 31.8 | 27.9
Llama 7B | SiLU* | 100 | 35.1 | 37.9 | 30.2 | 37 | 37.1
Llama 7B | GELU | 100 | 35.9 | 38.4 | 29.4 | 37.6 | 39.5
Llama 7B | ReLU | 72 | 34.7 | 34.8 | 31.2 | 36.3 | 37.8

Tab. 1 shows that different stages of the relufication process do not
significantly reduce zero-shot accuracy while using significantly less
compute. The sparsity is broken down into three categories: up, down, and QKV
projections. Notably, the input to QKV is less sparse than FFN projections,
which opens an interesting avenue for future research. We note that the small
gap in performance between the original vs. relufied models may be partially
due to the finetuning process and not necessarily the activation function. Our
finetuning is applied only for 30B and 50B tokens for stages 1 and 2,
respectively. Putting into prospect and comparing it with 1T tokens of Llama,
for example, this is equivalent to 3-5% of the original training duration. As
discussed in Sec. 3.2, according to the scaling properties of LLMs, the gap
will be further bridged by additional finetuning steps.

We also assess the in-context learning ability of the relufied models with the
Massive Multitask Language Understanding (MMLU) [29] benchmark in Tab. 2. Our
results show that when we augment the original LLMs with different activations
and finetune, the few-shot performance does not change significantly either.
Moreover, Sec. E in the appendix shows that a larger but relufied model
performs better than an original smaller model of the same FLOPS. Overall, the
results affirm that the proposed relufication procedure can decrease the
inference FLOPS at various stages and rates while maintaining on-par
performance on various tasks.

##  5 Applications

In this section, we discuss promising directions motivated by our
investigation in Sec. 4. First, we introduce _aggregated sparsity_ , showing
that ReLU networks reuse previously activated neurons when generating tokens.
Hence, we can leverage this to increase the generation speed. Next, we relate
aggregated sparsity with speculative decoding to further improve speculative
decoding’s inference time. Finally, we briefly discuss a promising direction
of using the _shifted ReLU_ activation function to improve the sparsity
further.

###  5.1 Aggregated Sparsity: reusing previously activated neurons

A consequence of using only a small subset of neurons for each token is that
if these neurons are shared to some degree, the model still does not use all
of the neurons until many tokens are processed. We refer to this as
_aggregated sparsity_ , which we defined as the ratio of neurons that have not
been used up to processing the first $t$ token. Note that this metric will
always be non-increasing. Intuitively, it measures the unused capacity of
feed-forward neurons for processing a specific prompt.

Here in Fig. 7(a) we show that for the OPT-6.7B model, on average, about 50%
of all the neurons will be unused across the first 150 tokens of prompts
coming from the WikiText dataset. Our empirical results hold for other ReLU
models and other datasets. Additionally, in Fig. 7(b), we show that this
pattern is far from random activation of neurons during the token generation
with a rate equal to the average rate of activation usage per token. Let
$s_{i}$ be the activation sparsity of layer $i$ averaged over all tokens.
Then, the probability of an activation not used in generating the first $t$
tokens in uniformly random selection is $s_{i}^{t}$ . Fig. 7(b) shows this
quantity for two layers $i=8,24$ for the first $256$ tokens in dashed line. It
also shows the real (observed) number of activations being used in the solid
line. The fact that the random aggregated sparsity (referred to as random
sparsity) is lower than the observed aggregated sparsity (we refer to it as
aggregated sparsity) shows a clear pattern of reusing activations.

We can benefit from the overlapping activations by utilizing previously loaded
weights from the down projection layer for upcoming tokens. To test this, we
initiate with reading 128 tokens. For the subsequent 128 tokens, we
intermittently avoid loading new weights for every $\gamma$ token. Using
$\gamma=16$ as an example, tokens 129-145 are generated conventionally.
However, for tokens 146-161, we retain the existing weight without introducing
any new weight. This pattern continues, with every next set of $\gamma$ tokens
alternating between conventional generation and weight reuse. In Fig. 7(c), we
observe only a slight increase in perplexity when using this approximation to
address the memory and I/O-intensive nature of LLM inference. This figure
contrasts the perplexity obtained from reused activations and random
selections. The reuse strategy aligns well with the baseline, whereas random
selection notably increases perplexity, highlighting the effectiveness of
reusing the already loaded activations for subsequent tokens.

(a)

(b)

(c)

(d)

Figure 7: (a) Aggregated sparsity of different layers and their mean. (b)
Aggregated sparsity during token generation and comparison with a random
sparsity. (c) Perplexity, based on the number of tokens for which loaded
weights from previous tokens are reused. The dashed line represents no reuse,
the solid blue line shows the case with activation reuse according to
aggregated sparsity, and the orange line depicts the perplexity when
activations are reused according to a random sparsity. (d) The inference
speedup of speculative decoding with aggregated sparsity and with random
sparsity. Speedup equal to 1.0 is the standard version of speculative
decoding.

###  5.2 Activation sparsity and speculative decoding

As highlighted in Sec. 5.1, activation reuse happens for multiple consecutive
tokens. When multiple consecutive tokens are processed together, we can save
the I/O (i.e., transferring weights to GPU/CPU as discussed in Appendix B)
associated with activations that are not used in any of them. If the reuse was
not happening, and the sparsity of all tokens was purely random, the
aggregated sparsity would shrink exponentially and quickly diminish.
Speculative decoding [46] is a related technique that uses a smaller model
$M_{q}$ to propose $\gamma$ tokens and a larger model $M_{p}$ to verify those
tokens and select matching ones. It improves the runtime of the model by
avoiding running $M_{p}$ sequentially.

To improve speculative decoding, aggregated sparsity can trim down the portion
of the model that needs to be run. Instead of running the full model, only the
non-sparse parts need to be evaluated, which will reduce I/O and compute
latency. Suppose the average aggregated sparsity of $M_{p}$ for $\gamma$
tokens is $\bar{s}_{\text{agg}}(\gamma)$, and cost of running $M_{q}$ over
$M_{p}$ is $c$. Then the expected latency speedup when going from standard
speculative decoding to sparse speculative decoding is
$\frac{c\gamma+1}{c\gamma+(1-\bar{s}_{\text{agg}}(\gamma))}$.

Fig. 7(d) compares sparse speculative decoding to the standard version for OPT
6.7B model. As a case study, for $\gamma=16$, the sparse version has a 1.27x
speedup over the standard speculative decoding. If the aggregated sparsity was
random over different tokens, the speedup would have been only 1.20x. Note
that even random sparsity will lead to speedup over standard speculative
decoding. This further shows the value of relufication. However, the speedup
due to random sparsity would diminish quickly in comparison to aggregated
sparsity as we go for larger $\gamma$. For example, for $\gamma=64$ the
speedup is almost negligible, while the speedup for the aggregated sparsity is
around 1.14x. Further discussion and details are postponed to Appendix C,
where we compare sparse speculative decoding, standard speculative decoding,
and autoregressive decoding and discuss optimal $\gamma$ in the case of sparse
speculative decoding.

###  5.3 The shifted ReLU activation

Our work in this section is motivated by the observation from Sec. 4, where,
comparing Fig. 4(d) with Fig. 4(b) revealed that the relufied Llama has much
less sparsity (65%) than the relufied Falcon model (95%). In addition, we
build on two of our previous findings. First, the preactivation distribution
of the relufied Llama (Fig. 5(c)) includes a considerable mass after the
cutoff value at zero. Second, the shape of the preactivation distribution does
not change before and after the relufication process (Fig. 5(c) and Fig.
5(d)).

Therefore, we may be able to shift the preactivation distribution to the left
to put more volume before the cutoff at 0. To this end, for preactivation
input $x$, rather than applying $\text{ReLU}(x)$, we use $\text{ReLU}(x-b)$
where $b\in\mathbb{R}$ is a constant scalar. We propose to set the value $b$
based on the preactivation distribution. For instance, based on the
distribution in Fig. 5(d), setting $b=1$ and hence using $\text{ReLU}(x-1)$ as
our activation function will result in dropping $95\%$ of the preactivations
and make it significantly sparser. Another benefit of this approach is
simplicity, as this does not require changing the loss function or the
training regime.

Figure 8(a) shows that the shifted ReLU activation function has on-par
accuracy with the ReLU activation function. Moreover, similar to our
observation in Sec. 4, the shifted ReLU activation quickly recovers the lost
performance due to the drastic change of activation function, while it also
maintains a very high-level activation sparsity during the finetuning stage.
The gap between shifted ReLU and ReLU is wider in the early stages of
training, and it narrows down when more tokens are seen.

A deeper investigation of ReLU-variants that can promote sparsity without
sacrificing performance is an appealing future direction. Moreover, it will be
interesting to study the impact of the shifted ReLU for stage-2 of our
relufication process where the sparsity level is usually not very high.

(a)

(b)

Figure 8: The effect of shifted ReLU on Llama model. (a) The performance is
almost the same as the original ReLU. (b) Shifted ReLU (i.e.,
$\text{ReLU}(x-1)$) is much sparser than the original ReLU.

##  6 Conclusion

In this study, we conducted a large-scale investigation of the activation
functions, and we have shown that the choice of activation functions during
pretraining and finetuning does not have a significant impact on performance
while using ReLU can provide an additional benefit of leading to activation
sparsity and more efficient inference. To bridge the gap between existing pre-
trained models and our work, we have _relufied_ several models to incorporate
the ReLU activation function into the architecture of these already pre-
trained models. We have shown that across several zero-shot and few-shot
tasks, the ReLU-based LLMs perform similarly to their non-ReLU models at a
significantly reduced computation. In addition, after observing sparsity
patterns in ReLU LLMs, we explored a few promising directions to improve the
token generation speed through _aggregated sparsity_ and achieve greater
efficiency using ReLU-based activation functions like _shifted ReLU_.

We believe our work is among the few studies that investigate changes in the
architectural components of LLMs on a large scale. We hope our findings
motivate the community to further investigate the advantages of well-
structured activation sparsity, ultimately enhancing the efficiency of these
models.

## Acknowledgments

The authors would like to thank Fartash Faghri, Minsik Cho, Thomas Merth, and
Mohammad Samragh for their invaluable discussions and feedback on this
project.

## References

  * Agarwal et al. [2023] Rishabh Agarwal, Nino Vieillard, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem.  Gkd: Generalized knowledge distillation for auto-regressive sequence models.  _CoRR_ , 2023.
  * Almazrouei et al. [2023] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Maitha Alhammadi, Mazzotta Daniele, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.  The falcon series of language models: Towards open frontier models.  2023\.
  * Aminabadi et al. [2022] Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al.  Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale.  In _SC22: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–15. IEEE, 2022.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization, 2016.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Bubeck et al. [2023] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_ , 2023.
  * Chee et al. [2023] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa.  Quip: 2-bit quantization of large language models with guarantees.  _CoRR_ , abs/2307.13304, 2023.  doi: 10.48550/arXiv.2307.13304.
  * Chen et al. [2022] Tianyu Chen, Shaohan Huang, Yuan Xie, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei.  Task-specific expert pruning for sparse mixture-of-experts.  _ArXiv_ , abs/2206.00277, 2022.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Clevert et al. [2016] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus).  In Yoshua Bengio and Yann LeCun, editors, _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ , 2016.
  * Dauphin et al. [2017] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier.  Language modeling with gated convolutional networks.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, page 933–941. JMLR.org, 2017.
  * Dettmers et al. [2022] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.  Llm. int8 (): 8-bit matrix multiplication for transformers at scale.  _arXiv preprint arXiv:2208.07339_ , 2022.
  * Dettmers et al. [2023a] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms.  _CoRR_ , 2023a.
  * Dettmers et al. [2023b] Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh.  Spqr: A sparse-quantized representation for near-lossless LLM weight compression.  _CoRR_ , abs/2306.03078, 2023b.  doi: 10.48550/arXiv.2306.03078.
  * Dong and Chen [2023] Gaochen Dong and Wei Chen.  Blockwise compression of transformer-based models without retraining.  _arXiv preprint arXiv:2304.01483_ , 2023.
  * Du et al. [2022] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P. Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui.  Glam: Efficient scaling of language models with mixture-of-experts.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 5547–5569. PMLR, 2022.
  * Elfwing et al. [2018] Stefan Elfwing, Eiji Uchibe, and Kenji Doya.  Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.  _Neural networks_ , 107:3–11, 2018.
  * Fedus et al. [2022a] William Fedus, Jeff Dean, and Barret Zoph.  A review of sparse expert models in deep learning.  _ArXiv_ , abs/2209.01667, 2022a.
  * Fedus et al. [2022b] William Fedus, Barret Zoph, and Noam Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _J. Mach. Learn. Res._ , 23:120:1–120:39, 2022b.
  * Frantar and Alistarh [2023] Elias Frantar and Dan Alistarh.  Sparsegpt: Massive language models can be accurately pruned in one-shot.  In _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 10323–10337. PMLR, 2023.  URL https://proceedings.mlr.press/v202/frantar23a.html.
  * Frantar et al. [2022] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.  GPTQ: accurate post-training quantization for generative pre-trained transformers.  _CoRR_ , abs/2210.17323, 2022.  doi: 10.48550/arXiv.2210.17323.
  * Fukushima [1969] Kunihiko Fukushima.  Visual feature extraction by a multilayered network of analog threshold elements.  _IEEE Trans. Syst. Sci. Cybern._ , 5:322–333, 1969.
  * Gao et al. [2021] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, September 2021.  URL https://doi.org/10.5281/zenodo.5371628.
  * Gu et al. [2023] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang.  Knowledge distillation of large language models.  _CoRR_ , 2023.
  * Han et al. [2023] Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William J. Dally.  Retrospective: Eie: Efficient inference engine on sparse and compressed neural network, 2023.
  * Hazimeh et al. [2021] Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed H. Chi.  Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning.  In _Neural Information Processing Systems_ , 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hendrycks and Gimpel [2016] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.  URL https://openreview.net/forum?id=d7KBjmI3GmQ.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _CoRR_ , 2015.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training compute-optimal large language models.  _CoRR_ , abs/2203.15556, 2022.  doi: 10.48550/arXiv.2203.15556.
  * Hron et al. [2020] Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak.  Infinite attention: NNGP and NTK for deep attention networks.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 4376–4386. PMLR, 2020\.
  * Hsieh et al. [2023] Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister.  Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.  In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_ , pages 8003–8017. Association for Computational Linguistics, 2023.  URL https://doi.org/10.18653/v1/2023.findings-acl.507.
  * Hwang et al. [2023] Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, Mao Yang, and Minsoo Rhu.  Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference.  _arXiv preprint arXiv:2308.12066_ , 2023.
  * Jaiswal et al. [2023] Ajay Jaiswal, Shiwei Liu, Tianlong Chen, and Zhangyang Wang.  The emergence of essential sparsity in large pre-trained models: The weights that matter.  _CoRR_ , 2023.
  * Jaszczur et al. [2021] Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser, Wojciech Gajewski, Henryk Michalewski, and Jonni Kanerva.  Sparse is enough in scaling transformers.  In _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=-b5OSCydOMe.
  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _CoRR_ , abs/2001.08361, 2020.  URL https://arxiv.org/abs/2001.08361.
  * Kim et al. [2023a] Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee.  Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization.  _CoRR_ , abs/2305.14152, 2023a.  doi: 10.48550/arXiv.2305.14152.  URL https://doi.org/10.48550/arXiv.2305.14152.
  * Kim et al. [2023b] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer.  Squeezellm: Dense-and-sparse quantization.  _CoRR_ , abs/2306.07629, 2023b.  doi: 10.48550/arXiv.2306.07629.
  * Kim et al. [2023c] Sehoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael W Mahoney, et al.  Full stack optimization of transformer inference.  In _Architecture and System Support for Transformer Models (ASSYST@ ISCA 2023)_ , 2023c.
  * Kim et al. [2023d] Sehoon Kim, Karttikeya Mangalam, Suhong Moon, John Canny, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer.  Speculative decoding with big little decoder, 2023d.
  * Klambauer et al. [2017] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter.  Self-normalizing neural networks.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 971–980, 2017.
  * Kong et al. [2023] Rui Kong, Yuanchun Li, Qingtian Feng, Weijun Wang, Linghe Kong, and Yunxin Liu.  Serving moe models on resource-constrained edge devices via dynamic expert swapping, 2023.
  * Kurtz et al. [2020] Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh.  Inducing and exploiting activation sparsity for fast inference on deep neural networks.  In _International Conference on Machine Learning_ , pages 5533–5543. PMLR, 2020.
  * Lee et al. [2023] Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park.  OWQ: lessons learned from activation outliers for weight quantization in large language models.  _CoRR_ , abs/2306.02272, 2023.  doi: 10.48550/arXiv.2306.02272.
  * Leviathan et al. [2023] Yaniv Leviathan, Matan Kalman, and Yossi Matias.  Fast inference from transformers via speculative decoding.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 19274–19286. PMLR, 2023.
  * Li et al. [2022] Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, et al.  Large models are parsimonious learners: Activation sparsity in trained transformers.  _arXiv preprint arXiv:2210.06313_ , 2022.
  * Li et al. [2023] Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix X. Yu, Ruiqi Guo, and Sanjiv Kumar.  The lazy neuron phenomenon: On emergence of activation sparsity in transformers.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net, 2023\.
  * Lin et al. [2023] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han.  AWQ: activation-aware weight quantization for LLM compression and acceleration.  _CoRR_ , abs/2306.00978, 2023.  doi: 10.48550/arXiv.2306.00978.
  * Liu et al. [2023a] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra.  Llm-qat: Data-free quantization aware training for large language models.  _CoRR_ , 2023a.
  * Liu et al. [2023b] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al.  Deja vu: Contextual sparsity for efficient llms at inference time.  In _International Conference on Machine Learning_ , pages 22137–22176. PMLR, 2023b.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.  URL https://openreview.net/forum?id=Bkg6RiCqY7.
  * Ma et al. [2023] Xinyin Ma, Gongfan Fang, and Xinchao Wang.  Llm-pruner: On the structural pruning of large language models.  _arXiv preprint arXiv:2305.11627_ , 2023.
  * Merity et al. [2017] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.
  * Mirzadeh et al. [2020] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh.  Improved knowledge distillation via teacher assistant.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 5191–5198, 2020.
  * MosaicML [2023] NLP Team MosaicML.  Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023.  URL www.mosaicml.com/blog/mpt-7b.
  * Narang et al. [2021] Sharan Narang, Hyung Won Chung, Yi Tay, Liam Fedus, Thibault Févry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, and Colin Raffel.  Do transformer modifications transfer across implementations and applications?  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 5758–5773. Association for Computational Linguistics, 2021.
  * Park et al. [2023] Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee.  Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models.  _CoRR_ , 2023.
  * Penedo et al. [2023] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The refinedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only.  _CoRR_ , abs/2306.01116, 2023.  doi: 10.48550/arXiv.2306.01116.
  * Pope et al. [2023] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean.  Efficiently scaling transformer inference.  _Proceedings of Machine Learning and Systems_ , 5, 2023.
  * Puigcerver et al. [2023] Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby.  From sparse to soft mixtures of experts.  _arXiv preprint arXiv:2308.00951_ , 2023.
  * Rajbhandari et al. [2020] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.  Zero: memory optimizations toward training trillion parameter models.  In Christine Cuicchi, Irene Qualters, and William T. Kramer, editors, _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020_ , page 20. IEEE/ACM, 2020.
  * Rajbhandari et al. [2022] Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He.  Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 18332–18346. PMLR, 2022.
  * Ramachandran et al. [2017] Prajit Ramachandran, Barret Zoph, and Quoc V Le.  Searching for activation functions.  _arXiv preprint arXiv:1710.05941_ , 2017.
  * Santacroce et al. [2023] Michael Santacroce, Zixin Wen, Yelong Shen, and Yuanzhi Li.  What matters in the structured pruning of generative language models?  _CoRR_ , 2023.
  * Shazeer [2020] Noam Shazeer.  Glu variants improve transformer, 2020.
  * Shazeer et al. [2017] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.
  * Shen et al. [2023] Kai Shen, Junliang Guo, Xu Tan, Siliang Tang, Rui Wang, and Jiang Bian.  A study on relu and softmax in transformer.  _CoRR_ , 2023.
  * Sheng et al. [2023] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang.  Flexgen: High-throughput generative inference of large language models with a single GPU.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 31094–31116. PMLR, 2023.
  * Song et al. [2021] Zhao Song, Lichen Zhang, and Ruizhe Zhang.  Training multi-layer over-parametrized neural network in subquadratic time, 2021.
  * Sun et al. [2023] Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter.  A simple and effective pruning approach for large language models.  _CoRR_ , 2023.
  * Team et al. [2022] NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang.  No language left behind: Scaling human-centered machine translation, 2022\.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  _CoRR_ , abs/2302.13971, 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008, 2017.
  * Wortsman et al. [2023] Mitchell Wortsman, Jaehoon Lee, Justin Gilmer, and Simon Kornblith.  Replacing softmax with relu in vision transformers, 2023.
  * Xiao et al. [2023] Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han.  Smoothquant: Accurate and efficient post-training quantization for large language models.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 38087–38099. PMLR, 2023.
  * Yi et al. [2023] Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu.  Edgemoe: Fast on-device inference of moe-based large language models.  _arXiv preprint arXiv:2308.14352_ , 2023.
  * Zhang and Sennrich [2019] Biao Zhang and Rico Sennrich.  Root mean square layer normalization.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 12360–12371, 2019.
  * Zhang et al. [2023] Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang.  Pruning meets low-rank parameter-efficient fine-tuning.  _CoRR_ , 2023.
  * Zhang et al. [2022a] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  OPT: open pre-trained transformer language models.  _CoRR_ , abs/2205.01068, 2022a.
  * Zhang et al. [2022b] Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou.  Moefication: Transformer feed-forward layers are mixtures of experts.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 877–890. Association for Computational Linguistics, 2022b.
  * Zhu et al. [2023] Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang.  A survey on model compression for large language models.  _CoRR_ , 2023.
  * Zoph et al. [2022] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus.  St-moe: Designing stable and transferable sparse expert models.  _arXiv preprint arXiv:2202.08906_ , 2022.

## Appendix

##  Appendix A Extended Related Works

Activation Functions. ReLU, introduced by [22], remains a predominant
activation function for deep neural networks and was notably utilized in the
original transformers work [74]. SwiGLU [66] has been shown to enhance
performance when replacing ReLU in feedforward layers and is a feature in
models like Llama [73]. Narang et al. [57] conducted an extensive comparison
of various activation functions, such as GeLU, SiLU [28, 17], ELU [10], SeLU
[42], and GLU variants [11], identifying certain advantages over ReLU. Our
paper and results differ from theirs by training billion scale models and data
as opposed to their smaller scale ones. Furthermore, our results indicate that
extended training can diminish the performance gap between ReLU and these
other functions, also leading to savings in computational costs.

Activation Sparsity. A body of prior research [44, 25, 70] has demonstrated
that increasing sparsity can lead to reductions in both inference and training
times. Dejavu [51] and [47] observed pronounced sparsity in activations when
using the ReLU function in feedforward layers. These studies propose that
predicting this sparsity can further boost inference speeds. Similarly, [36]
employed ReLU activations and introduced a controller to actively promote
sparsity. Notably, these studies predominantly focus on networks employing
ReLU activations, leaving out those with alternative activation functions. In
contrast, our approach modifies networks by substituting other activation
functions with ReLU. We then fine-tune these networks to achieve activation
sparsity in the MLP layer post-ReLUfication. We further illustrate that
inserting ReLU prior to the QKV and Feedforward layers can substantially
reduce FLOPS, albeit at a minor cost to accuracy. Unlike the aforementioned
studies, we do not utilize a sparsity predictor to minimize FLOPS.

ReLU in Attention Mechanisms. Beyond the activation function in the MLPs of
large language models, a softmax activation is often employed within the
attention module. Prior studies have indicated that it’s feasible to replace
this softmax with ReLU without compromising accuracy [75, 68, 32]. This avenue
of research is distinct from our approach of Relufication, which specifically
focuses on activations preceding weight multiplications.

Model compression for efficient inference Quantization, pruning and
distillation are the main three techniques for compressing neural networks
[82]. Quantization has been used to reduce model size and faster inference
[13, 50, 58, 12, 49, 45, 14, 39, 7, 76]. The quantized model occupies less
space reducing the memory latency [21, 38]. Reluification is orthogonal to
quantization and reduces the amount of memory required to be loaded and can
further decrease the memory latency. Distillation [33, 30, 24, 55, 1] is
another technique to train smaller models. This is orthogonal to using ReLU
activations as any activation can be used in distillation methods. Sparsifying
or pruning weights of neural networks [20, 35, 79, 71, 65, 53] can reduce
computation and inference time. Weight sparsity is usually unstructured and
hard to implement for hardware, but the sparsity induced by ReLU can easily be
implemented as a matrix multiplication of non zero rows. Weight sparse models
can be combined with our relufication for further decrease in compute.

Mixture of Experts. Mixture of Experts (MoE) LLMs usually subdivide the feed-
forward layer into multiple experts. A router is then employed to selectively
and sparsely activate these experts [67, 19, 72]. Similar to our work, MoE is
a form of activation sparsity but in a group form and can be seen as a subset
of sparse activation. Subsequent studies have further refined the inference
and training methodologies for MoE models [61, 34, 77, 16, 43, 63, 83, 8, 26].
MoE can be also combined with Relufication, having sparsity inside FFN of each
expert.

Another line of work is MoEfication of networks that have sparse activations
by subdividing neurons [81]. Relufication can also help MoEfication be
applicable to a wider range of networks by increasing sparsity of FFNs. For a
more in depth review of mixture of expert models we refer the reader to [18].

Speculative Decoding and Sparsity. Speculative decoding is a method that aims
to improve model latency when faced with memory bandwidth constraints [46,
41]. It involves using a smaller model to predict the next tokens, with a
larger model subsequently verifying multiple tokens in a single operation. In
this work, we examine the direct effects of incorporating sparsity into
speculative decoding. We show that adding sparsity can lead to performance
improvements in speculative decoding. Additionally, we provide guidelines on
selecting parameters for speculative decoding when sparsity is introduced.

##  Appendix B Discussion on Activation Sparsity and Inference Efficiency

The primary motivation behind our work is to enhance _efficiency_ , and we
believe it is essential to provide a precise definition of this term.
Throughout the main text, we predominantly use FLOPS as our efficiency metric.
In this section, we argue why, in the presence of _activation sparsity_ ,
FLOPS can serve as a suitable proxy for measuring various efficiency metrics.

Firstly, it is important to be reminded of the two major factors contributing
to the efficiency of Large Language Models (LLMs): (1) the total amount of
computation and (2) input/output (IO) transfer—i.e., transferring parameters
from RAM to CPU/GPU for calculations. Notably, for today’s large models,
factor (2) acts as the major bottleneck during the inference phase. We refer
the reader to the detailed analysis by Liu et al. [51]. Ultimately, for a
specific target device and assuming an efficient implementation, we believe
that the most practical measure of efficiency is _latency_ (e.g., the average
time to generate a token). However, each device possesses its unique
properties, necessitating a more ubiquitous proxy metric.

(a) sparse vector, dense matrix multiplication: by skipping rows, we reduce
both the weight transfer (i.e., loading these rows for computation) and
computation (i.e., the result will be zero).

(b) Comparing FLOPS versus real latency for OPT model (FFN).

Figure 9: For LLMs that have sparse activations, FLOPS is a good approximation
of the real latency.

We argue that the way we calculated _FLOPS_ in our paper and is greatly
influenced by _activation sparsity_ can reasonably approximate _efficiency_.
Here are our reasons:

  * •

Reduced Computation: As shown in Fig. 9(a), with activation sparsity, we have
a sparse vector-dense matrix multiplication at inference, while this will be a
sparse-matrix-dense matrix multiplication during training. It is important to
note that this is a semi-structured sparsity (unlike magnitude weight
pruning), and we can leverage the fact that we are transferring weights in
large chunks (i.e., _rows_). Modern accelerators already support sparse
operations333For example, both cuSPARSE on NVIDIA CUDA® and Accelerate on
Apple devices. and we can build on these existing tools.

  * •

Reduced IO Transfer: During inference, weights need to be transferred to the
device cache for computation (e.g., from RAM to CPU cache or GPU VRAM to GPU
cache). This step constitutes the main bottleneck during token generation. For
instance, approximately $99.3\%$ of the total latency is attributed to IO, as
indicated by Liu et al. [51]. However, by storing matrices in a row-major
order, we can _skip loading_ unnecessary rows as the output will be zero.

Overall, as depicted in Figure 9(b) based on the calculations by Liu et al.
[51], we demonstrate that for the OPT model on an NVIDIA A100 node, counting
FLOPS provides a reasonable approximation to and is highly correlated with the
time needs to generate tokens, especially, for LLMs with activation sparsity.

##  Appendix C Activation Sparsity and Speculative Decoding

Speculative decoding [46] is a technique that uses a smaller model $M_{q}$ to
propose $\gamma$ tokens and a larger model $M_{p}$ to verify those tokens and
selects matching ones. This technique improves the runtime of the model by
avoiding running $M_{p}$ sequentially per all tokens. To further improve the
speculative decoding inference, we can leverage sparsity as follows.

Latency model.  We assume a simple conceptual model for latency in speculative
decoding. Following Deja Vu [51] latency can be broken down into compute and
I/O latency. The compute latency is negligible to I/O. Also, notice that the
Speculative decoding is meant for the constraints that memory bandwidth is the
bottleneck. Therefore we only compare I/O latency between sparse and non-
sparse models. If the average aggregated sparsity of $M_{p}$ for $\gamma$
tokens is $\bar{s}_{\text{agg}}(\gamma)$, and runtime of $M_{p}$ is $T$, we
approximate the latency of running $M_{p}$ for $\gamma$ consecutive tokens
with $(1-\bar{s}_{\text{agg}}(\gamma))T$. As discussed in the previous
section, this is a good approximation of real latency.

Theoretical latency improvement. Assume the smaller model $M_{q}$ operates $c$
times faster than the cumbersome model $M_{p}$. As per the primary text, token
acceptances are assumed to follow an independent and identically distributed
(i.i.d.) behavior. Denote $\alpha$ as the expected probability of a token
generated by $M_{q}$ being approved by $M_{p}$. The following theorems hold:

######  Theorem 1.

The expected improvement factor in latency for speculative decoding with
sparsity, over standard speculative decoding is
$\frac{c\gamma+1}{c\gamma+(1-\bar{s}_{\text{agg}}(\gamma))}$.

###### Proof.

The amount of time required to run sparsified model is quantified as
$Tc\gamma+(1-\bar{s}_{\text{agg}}(\gamma))T$. It is the time of running a
smaller model plus a larger model’s non-sparse portion. Run time of
speculative decoding without sparsity is $Tc\gamma+T$. The number of generated
tokens in both is the same. Therefore the relative speedup of using sparsity
is given by: $\frac{Tc\gamma+T}{Tc\gamma+(1-\bar{s}_{\text{agg}}(\gamma))T}$.
∎

######  Theorem 2.

The expected improvement factor in latency, when combining sparsity with
speculative decoding against normal (autoregressive) decoding using only
$M_{p}$, is
$\frac{1-\alpha^{\gamma+1}}{(c\gamma+(1-\bar{s}_{\text{agg}}(\gamma)))(1-\alpha)}$.

###### Proof.

Similar to theorem above $Tc\gamma+(1-\bar{s}_{\text{agg}}(\gamma))T$ gives
the time required for sparse speculative decoding. According to the original
paper, the standard speculative decoding yields an average of
$\frac{1-\alpha^{\gamma+1}}{1-\alpha}$ tokens generated per each run [46].
Thus, the anticipated run time when generating tokens with sparsity during
speculative decoding becomes
$\frac{(c\gamma+(1-\bar{s}_{\text{agg}}(\gamma)))(1-\alpha)}{1-\alpha^{\gamma+1}}T$.
Given the runtime for producing a single token via an autoregressive approach
is $T$, the inverse of this fraction gives the desired results. ∎

Optimal $\gamma$. The optimal $\gamma$ for speculative decoding can be found
by optimizing the speedup factor equation in Theorem 2. When sparsity is not
present, the equation can be solved numerically, but for reluified networks,
the aggregated sparsity for different $\gamma$’s will affect the final
results. We have found optimal $\gamma$s based on
$\bar{s}_{\text{agg}}(\gamma)$ for OPT 6.7B and presented the results in
figure 10(a). The chosen $\gamma$ for sparse speculative decoding is smaller
than standard speculative decoding since higher $\gamma$ will result in less
sparsity. The gap in $\gamma$ is always less than 20%. Also, in figure 10(b),
it can be seen for the specific case of $\alpha=0.8,c=0.02$, the sparse
speculative decoding has the highest speed-up factor over autoregressive at
$\gamma=10$s vs standard version’s optimal point which happens for
$\gamma=12$. Sparse speculative decoding at $\gamma=12$ is better than
standard speculative decoding at $\gamma=12$, while sparse speculative
decoding at $\gamma=10$ beats both. Another observation from 10(b) is for the
case of purely random sparsity, the benefit of sparse speculative decoding
would diminish over standard speculative decoding in higher $\gamma$s. In
contrast, the benefits of aggregated sparsity would last for larger values of
$\gamma$.

(a)

(b)

Figure 10:  (a) optimal $\gamma$ for sparse speculative decoding (b) speed up
of sparse speculative decoding and standard speculative decoding over
autoregressive decoding when $\alpha=0.8$ and $c=0.02$

##  Appendix D Pre-activation Distribution of OPT models trained from scratch

The distribution of pre-activation inputs is suspected to be the main factor
determining the amount of sparsity. As we saw in Sec. 4.1, the pre-activation
distribution for Llama and Falcon differs a lot. One may wonder that if we
control for the training data and optimization algorithm, would the shapes of
the distribution differ? To this end, we train OPT 1.3B models from scratch
using our four variants of the activation function and depicted the pre-
activation distribution along the training in Fig. 11. They start from the
exact figure but gradually diverge. From SiLU to ReLU (increasing $\beta$),
the pre-activation distribution becomes more concentrated around 0 and would
be almost uni-modal. Further investigations on the dynamics of pre- and post-
activations and their relation to efficiency and accuracy are left as an
exciting future direction for research.

Figure 11: Pre-activation distributions of various OPT 1.3B models with all
four types of activations trained from scratch at various number of seen
tokens during training.

##  Appendix E Is a sparsified large model better than dense smaller ones?

When it comes to deploying efficient models, one may naturally use an original
smaller-size (dense) model. The argument would be the performance of the
relufied larger model might be already equal to or less than the smaller dense
model. To study the above question, we plotted the performance vs. efficiency
of the original and the relufied OPT models in Fig. 12. Taking the relufied
OPT 6.7B model as an example, it operates at 2.8 GFLPOPs per token.
Interpolating the blue line (that can be seen as a scaling plot of the OPT
model), a dense model with equivalent FLOPS falls more than 2% short in zero-
shot performance.

Similarly, compared to the relufied OPT 2.7B model, the equivalent (in FLOPS)
dense model performs almost 2% lower. Indeed, the fact that the relufied
models lie well above the scaling graph of the original OPT models, shows the
effectiveness of relufication processes as a method to get better but more
efficient models. As a side benefit, it makes the efficiency spectrum of the
available LLMs more continuous. For example, consider a combination of
hardware and use case that only allows deploying LLMs with lower than 3 GFLOPS
during inference. Going with standard pretrained models, the only available
option is OPT 2.7B with almost 1 GFLOPS, as the 6.7B does not satisfy the
hardware constraint. In this situation, our relufied model not only falls in
the limited inference budget but is also very close to the next largest
available model in terms of accraucy. An exciting and timely direction for
future research is finding methods, that, given an LLM (or a family of LLMs),
are able to produce the best performing model matching the specified inference
computation budget.

Figure 12: Performance of sparse large models vs. dense smaller models: The
relufied large models (red stars) are above the scaling curve of original
dense models (blue circles and dashed line).


# Segment Anything Model (SAM) Meets Glass: Mirror and Transparent Objects
Cannot Be Easily Detected

Dongsheng Han
Kyung Hee University
Chaoning Zhang
Kyung Hee University
Yu Qiao
Kyung Hee University
Maryam Qamar
Kyung Hee University
Yuna Jung
Kyung Hee University
SeungKyu Lee
Kyung Hee University
Sung-Ho Bae
Kyung Hee University
Choong Seon Hong
Kyung Hee University
Correspondence Author<EMAIL_ADDRESS>
###### Abstract

Meta AI Research has recently released SAM (Segment Anything Model) which is
trained on a large segmentation dataset of over 1 billion masks. As a
foundation model in the field of computer vision, SAM (Segment Anything Model)
has gained attention for its impressive performance in generic object
segmentation. Despite its strong capability in a wide range of zero-shot
transfer tasks, it remains unknown whether SAM can detect things in
challenging setups like transparent objects. In this work, we perform an
empirical evaluation of two glass-related challenging scenarios: mirror and
transparent objects. We found that SAM often fails to detect the glass in both
scenarios, which raises concern for deploying the SAM in safety-critical
situations that have various forms of glass.

_Keywords_ Segment Anhting Model $\cdot$ Glass $\cdot$ Morror Object $\cdot$
Transparent Object

##  1 Introduction

In the past few years, generative AI Zhang et al. (2023a) has caught
significant attention with interesting applications like ChatGPT Zhang et al.
(2023b), text-to-image Zhang et al. (2023c), text-to-speech Zhang et al.
(2023d) and graph generation Zhang et al. (2023e). A key factor that drives
the development of generative AI is foundation model Bommasani et al. (2021)
that at inference can generalize to tasks and data distributions different
from training. With the success of ChatGPT Zhang et al. (2023b), GPT-3 (Brown
et al., 2020) has been widely recognized as one of the most widely recognized
foundation models for NLP.

Very recently, Meta AI research team has recent released a segment anything
project Kirillov et al. (2023) that introduces a promotable segmentation task
for training a vision foundation model. The resulting segment anything model
(SAM) has been recognized as the GPT-3 moment for vision. The model was
trained on over 1 billion masks on 11 million licensed and privacy-respecting
images. It represents a significant step towards achieving cognitive
recognition for all objects in the world, aiming to handle interactive
segmentation tasks while addressing real-world constraints. Segmentation tasks
encompass a wide range of challenging areas, and the detection of transparent
objects and mirror regions constitutes one of the most representative and
challenging tasks in this field. Given the ubiquitous appearance of such glass
objects in daily life and the unique challenges in detecting, localizing, and
reconstructing them from color images in computer vision(Lin et al., 2021).
This diccicult is due to the fact that most glass objects exhibit a visual
appearance that includes both the transmitted background scene and reflected
objects(Maeno et al., 2013)(He et al., 2021), while the visual appearance
observed from a mirror surface is entirely from reflected objects. These
issues led to many incidents, such as collisions of autonomous mobile robots
with transparent front doors or mirror walls and robot arms struggling to grip
a transparent bottle.

In this work, we examine the performance of the SAM foundation model in
recognizing and segmenting transparent objects and mirror surfaces and
evaluate its segmentation results. We aim to explore the challenges that arise
for foundation models in computer vision tasks when dealing with the
reflection and refraction phenomena that occur on glass and mirror objects, as
well as the faint object boundaries that result from these effects. After
extensive testing on large glass and mirror dataset benchmarks, we find that
while SAM segments general objects well in natural images well but often fail
to detect mirror and transparent objects. Failing to recognize glass can cause
serious issues when deploying the vision foundation model in safety-critical
setups.

##  2 Experimental Evaluations

Datasets: As the SAM model is designed to recognize and segment all objects in
an image, we select benchmark datasets where only one transparent or mirror
object exists in a given scene. This allows us to evaluate the model’s
capability to identify and segment these specific objects accurately. We
conduct experiments on two glass segmentation datasets GDD (Mei et al., 2020),
GSD (Lin et al., 2021) and two mirror segmentation datasets MSD (Yang et al.,
2019), PMD (Lin et al., 2020). GDD contains 2980 training and 936 test images.
GSD is transparent object dataset collected and labeled through networking and
photography, consisting of 4102 annotated glass images with close-up, medium,
and long shots from diverse scenes. MSD is a large mirror segmentation dataset
with 4018 images. PMD is the latest mirror segmentation dataset collected from
both indoor and outdoor scenes.

In our final additional visualization experiment, we included a multi-class
dataset of transparent objects called Trans10k (Xie et al., 2020) in addition
to the previous benchmark datasets. Trans10k is a dataset of transparent
objects categorized into two groups based on objects properties: movable small
objects (things) and non-movable objects (stuff).

Implementation Details: We conduct the test using the SAM best pre-trained
model VIT-H, which was pre-trained on a large segmentation dataset (SA-1B) and
generously provided as open source by Meta AI Research. As SAM does not
specify the object categories for segmentation, it outputs all possible
objects in the image. To test SAM’s capability to recognize glass and mirror
objects, we calculated the Intersection over Union (IoU) between the predicted
results and the ground truth for each object outputted by SAM. We then
selected the result with the highest IoU with the ground truth for glass
objects and used it as the detection result for SAM. We conducted different
evaluations based on this result.

Evaluation Metrics: We employed five commonly used evaluation metrics in
semantic and glass surface segmentation tasks: Intersection over Union (IoU),
pixel accuracy (ACC), weighted F-measure ($F_{\beta}$) Margolin et al. (2014),
mean absolute error (MAE), and balance error rate (BER). ${F}_{\beta}$ is a
harmonic mean of average precision and average recall defined as follows.

${\textit{F}_{\beta}}=\frac{(1+{\beta^{2}})(Precision\times{Recall})}{{\beta^{2}}Precision+Recall},$
(1)

where $\beta^{2}$ is set to 0.3 as suggested in Achanta et al. (2009). Mean
absolute error (MAE) is widely used in foreground-background segmentation
tasks where average pixel-wise error between predicted mask P and ground truth
mask G are calculated.

${\textit{MAE}}=\frac{1}{H\times{W}}\sum_{i=1}^{H}\sum_{j=1}^{W}\mid
P(i,j)-G(i,j)\mid,$ (2)

where P(i, j) indicates predicted probability at location (i, j). We employ
the balance error rate (BER) as an evaluation metric, which takes into account
the imbalanced regions in glass and mirror object segmentation tasks. This
metric provides a quantitative measure for evaluating the performance of glass
surface segmentation.

${\textit{BER}}=(1-\frac{1}{2}(\frac{TP}{\textit{N}_{p}}+\frac{TN}{\textit{N}_{n}}))\times{100},$
(3)

where TP, TN, Np, and Nn represent the numbers of true positives, true
negatives, glass pixels, and non-glass pixels.

###  2.1 Experiment on glass objects

We compare SAM with the state-of-the-art semantic segmentation, shadow
detection, saliency object detection, and glass segmentation methods such as
MirrorNet, Translab, GDNet, EBLNet as shown in Table 1, 2.

As shown in Table 1 and 2, We observed that the segmentation results of glass
objects in two benchmarks were poor for the SAM model.

As shown in Figure 1 and 4, We find that the SAM model struggle to recognize
and segment the boundary regions, particularly when the boundaries of glass
objects are distorte by light. These challenges in glass objects segmentation
are caused by the fact that transparent surfaces exhibit a visual appearance
that includes both the transmitted background scene and any objects behind
them, making it difficult to distinguish and accurately segment them from
their surroundings. The phenomenon of transmission tends to cause SAM to
detect objects behind glass objects, rather than the glass objects themselves.

Method | IoU $\uparrow$ | Acc $\uparrow$ |  ${F}_{\beta}$ $\uparrow$ | mAE $\downarrow$ | BER $\downarrow$
---|---|---|---|---|---
PSPNet Zhao et al. (2017) | 84.06 | 0.916 | 0.906 | 0.084 | 8.79
PointRend Kirillov et al. (2020) | 86.51 | 0.933 | 0.928 | 0.067 | 6.50
PiCANet Liu et al. (2018) | 83.73 | 0.916 | 0.909 | 0.093 | 8.26
DSC Hu et al. (2018) | 83.56 | 0.914 | 0.911 | 0.090 | 7.97
BDRAR Zhu et al. (2018) | 80.01 | 0.902 | 0.902 | 0.098 | 9.87
MirrorNet Yang et al. (2019) | 85.07 | 0.918 | 0.903 | 0.083 | 7.67
GDNet Mei et al. (2020) | 87.63 | 0.939 | 0.937 | 0.063 | 5.62
EBLNet He et al. (2021) | 88.16 | 0.941 | 0.939 | 0.059 | 5.58
SAM | 48.47 | 0.732 | 0.798 | 0.268 | 26.08
Table 1: Experimental Comparison on GDD Method | IoU $\uparrow$ |  ${F}_{\beta}$ $\uparrow$ | mAE $\downarrow$ | BER $\downarrow$
---|---|---|---|---
BASNetQin et al. (2019) | 69.79 | 0.808 | 0.106 | 13.54
MINet Pang et al. (2020) | 77.29 | 0.879 | 0.077 | 9.54
BDRARPang et al. (2020) | 75.92 | 0.860 | 0.081 | 8.61
PSPNetPang et al. (2020) | 7.30 | 0.834 | 0.110 | 10.66
MirrorNet Yang et al. (2019) | 74.20 | 0.828 | 0.090 | 10.76
SINet Fan et al. (2020) | 77.04 | 0.875 | 0.077 | 9.25
GDNet Mei et al. (2020) | 79.01 | 0.869 | 0.069 | 7.72
TransLabXie et al. (2020) | 74.05 | 0.837 | 0.088 | 11.35
GlassNet Lin et al. (2021) | 83.64 | 0.903 | 0.055 | 6.12
GlassSemNetLin et al.  | 85.60 | 0.920 | 0.044 | 5.60
SAM | 50.60 | 0.799 | 0.213 | 23.91
Table 2: Experimental Comparison on GSD.  Figure 1: Sample qualitative
comparison results on GDD and GSD.

###  2.2 Experiment on mirror objects

We compare SAM with the state-of-the-art semantic segmentation, shadow
detection, saliency objects detection, and mirror objects segmentation methods
such as BDRAR, MirrorNet, EBLNet, PMD-Net, LSA as shown in Table 3, 4.

Segmenting mirror objects is a challenging task in semantic segmentation due
to their strong reflective properties, which cause them to have a variable
appearance. To evaluate the performance of SAM model, we conducted tests on
the MSD and PMD benchmark datasets, (Lin et al., 2020).

Through our experiments, we found that the PMD benchmark, with more images of
mirrors captured from a distance, offers a clearer view of the boundaries of
mirror regions compared to the MSD benchmark, which has more images captured
from close range. As shown in Table 4, SAM performs comparably to state-of-
the-art methods on the PMD benchmark benchmark. However, as shown in Table3
and Figure2, SAM’s performance on the MSD benchmark is somewhat
unsatisfactory, as it tends to segment objects inside the mirror rather than
recognizing the mirror itself.

Figure 2: Sample qualitative comparison results on MSD and PMD. Method | IoU $\uparrow$ | Acc $\uparrow$ |  ${F}_{\beta}$ $\uparrow$ | mAE $\downarrow$ | BER $\downarrow$
---|---|---|---|---|---
ICNet Zhao et al. (2018) | 57.25 | 0.694 | 0.710 | 0.124 | 18.75
DSSHou et al. (2017) | 59.11 | 0.665 | 0.743 | 0.125 | 18.81
RASChen et al. (2018) | 60.48 | 0.845 | 0.758 | 0.111 | 17.60
BDRARZhu et al. (2018) | 67.43 | 0.821 | 0.792 | 0.093 | 12.41
DSCHu et al. (2018) | 69.71 | 0.816 | 0.812 | 0.087 | 11.77
PSPNet Zhao et al. (2017) | 68.01 | 0.922 | 0.846 | 0.079 | 12.08
MirrorNetYang et al. (2019) | 78.95 | 0.935 | 0.857 | 0.065 | 6.39
LSA Guan et al. (2022) | 79.85 | 0.946 | 0.889 | 0.055 | 7.12
EBLNet He et al. (2021) | 80.33 | 0.951 | 0.883 | 0.049 | 8.63
SAM | 51.57 | 0.876 | 0.817 | 0.124 | 23.17
Table 3: Experimental Comparison on MSD Method | IoU $\uparrow$ |  ${F}_{\beta}$ $\uparrow$ | mAE $\downarrow$ | Acc $\uparrow$
---|---|---|---|---
CPNET Yu et al. (2020) | 56.36 | 0.734 | 0.051 | 94.85
GloRe Chen et al. (2019) | 61.25 | 0.774 | 0.044 | 95.61
BDRARZhu et al. (2018) | 58.43 | 0.7433 | 0.043 | 95.66
PSPNet Zhao et al. (2017) | 60.44 | 0.806 | 0.039 | 96.13
MirrorNet Yang et al. (2019) | 62.50 | 0.778 | 0.041 | 96.27
PMD-Net Lin et al. (2020) | 62.40 | 0.827 | 0.055 | 96.80
LSA Guan et al. (2022) | 66.84 | 0.844 | 0.049 | 96.82
SAM | 64.75 | 0.861 | 0.0525 | 94.75
Table 4: Experimental Comparison on PMD

###  2.3 Additional visualization results

We conducted additional visualization experiments on the glass and mirror
object benchmarks to facilitate observation of the results.

In contrast to the previous visualizations, we present the detection results
of all objects in the predicted images of glass and mirror objects. To enhance
the visibility of the results obtained from the demonstration, we applied a
pseudo-binary technique to improve the clarity of the output results. Our
output images are generated as pseudo-binary images, where the predicted
regions of objects are represented in shades of gray, and the non-object
regions are displayed as black. Furthermore, if overlapping objects are
detected, the corresponding overlapping regions will be entirely displayed as
white.

As shown in Figures 3 and 4, when recognizing glass objects, the SAM model
successfully identified objects behind the transparent ones, but failed to
recognize the glass objects themselves. And many regions inside the glass were
not segmented as any object, resulting in these regions being unlabeled and
displayed as black areas. In Figure 5 the model correctly identified most
objects reflected in the mirror, but neglected to recognize the mirror itself.

Figure 3: Some challenging scenarios involving objects categorized as glass
"things" and "stuff" in the Trans10k test sets resulted in poor testing
results. Please zoom in to see the details. Figure 4: Some bad testing results
obtain from challenging scenarios that involve glass objects. Figure 5: Some
bad testing results obtain from challenging scenarios that involve mirror
objects.

##  3 Conclusion

This work is the first of its kind to perform a comprehensive study on whether
SAM can segment glass-related objects. We find that SAM often fails to detect
transparent objects. On two benchmark datasets, the performance of SAM is
significantly worse than those models that are specifically trained to detect
transparent objects.

## References

  * Zhang et al. [2023a] Chaoning Zhang, Chenshuang Zhang, Sheng Zheng, Yu Qiao, Chenghao Li, Mengchun Zhang, Sumit Kumar Dam, Chu Myaet Thwal, Ye Lin Tun, Le Luang Huy, et al.  A complete survey on generative ai (aigc): Is chatgpt from gpt-4 to gpt-5 all you need?  _arXiv preprint arXiv:2303.11717_ , 2023a.
  * Zhang et al. [2023b] Chaoning Zhang, Chenshuang Zhang, Chenghao Li, Yu Qiao, Sheng Zheng, Sumit Kumar Dam, Mengchun Zhang, Jung Uk Kim, Seong Tae Kim, Jinwoo Choi, et al.  One small step for generative ai, one giant leap for agi: A complete survey on chatgpt in aigc era.  _arXiv preprint arXiv:2304.06488_ , 2023b.
  * Zhang et al. [2023c] Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon.  Text-to-image diffusion models in generative ai: A survey.  _arXiv preprint arXiv:2303.07909_ , 2023c.
  * Zhang et al. [2023d] Chenshuang Zhang, Chaoning Zhang, Sheng Zheng, Mengchun Zhang, Maryam Qamar, Sung-Ho Bae, and In So Kweon.  A survey on audio diffusion models: Text to speech synthesis and enhancement in generative ai.  _arXiv preprint arXiv:2303.13336_ , 2023d.
  * Zhang et al. [2023e] Mengchun Zhang, Maryam Qamar, Taegoo Kang, Yuna Jung, Chenshuang Zhang, Sung-Ho Bae, and Chaoning Zhang.  A survey on graph diffusion models: Generative ai in science for molecule, protein and material.  _arXiv preprint arXiv:2304.01565_ , 2023e.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Lin et al. [2021] Jiaying Lin, Zebang He, and Rynson WH Lau.  Rich context aggregation with reflection prior for glass surface detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13415–13424, 2021.
  * Maeno et al. [2013] Kazuki Maeno, Hajime Nagahara, Atsushi Shimada, and Rin-ichiro Taniguchi.  Light field distortion feature for transparent object recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2786–2793, 2013.
  * He et al. [2021] Hao He, Xiangtai Li, Guangliang Cheng, Jianping Shi, Yunhai Tong, Gaofeng Meng, Véronique Prinet, and LuBin Weng.  Enhanced boundary learning for glass-like object segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15859–15868, 2021.
  * Mei et al. [2020] Haiyang Mei, Xin Yang, Yang Wang, Yuanyuan Liu, Shengfeng He, Qiang Zhang, Xiaopeng Wei, and Rynson WH Lau.  Don’t hit me! glass detection in real-world scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3687–3696, 2020.
  * Yang et al. [2019] Xin Yang, Haiyang Mei, Ke Xu, Xiaopeng Wei, Baocai Yin, and Rynson WH Lau.  Where is my mirror?  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8809–8818, 2019.
  * Lin et al. [2020] Jiaying Lin, Guodong Wang, and Rynson WH Lau.  Progressive mirror detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3697–3705, 2020.
  * Xie et al. [2020] Enze Xie, Wenjia Wang, Wenhai Wang, Mingyu Ding, Chunhua Shen, and Ping Luo.  Segmenting transparent objects in the wild.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16_ , pages 696–711. Springer, 2020.
  * Margolin et al. [2014] Ran Margolin, Lihi Zelnik-Manor, and Ayellet Tal.  How to evaluate foreground maps?  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 248–255, 2014.
  * Achanta et al. [2009] Radhakrishna Achanta, Sheila Hemami, Francisco Estrada, and Sabine Susstrunk.  Frequency-tuned salient region detection.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 1597–1604. IEEE, 2009.
  * Zhao et al. [2017] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2881–2890, 2017.
  * Kirillov et al. [2020] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick.  Pointrend: Image segmentation as rendering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9799–9808, 2020.
  * Liu et al. [2018] Nian Liu, Junwei Han, and Ming-Hsuan Yang.  Picanet: Learning pixel-wise contextual attention for saliency detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3089–3098, 2018.
  * Hu et al. [2018] Xiaowei Hu, Lei Zhu, Chi-Wing Fu, Jing Qin, and Pheng-Ann Heng.  Direction-aware spatial context features for shadow detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7454–7462, 2018.
  * Zhu et al. [2018] Lei Zhu, Zijun Deng, Xiaowei Hu, Chi-Wing Fu, Xuemiao Xu, Jing Qin, and Pheng-Ann Heng.  Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 121–136, 2018.
  * Qin et al. [2019] Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jagersand.  Basnet: Boundary-aware salient object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 7479–7489, 2019.
  * Pang et al. [2020] Youwei Pang, Xiaoqi Zhao, Lihe Zhang, and Huchuan Lu.  Multi-scale interactive network for salient object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9413–9422, 2020.
  * Fan et al. [2020] Deng-Ping Fan, Ge-Peng Ji, Guolei Sun, Ming-Ming Cheng, Jianbing Shen, and Ling Shao.  Camouflaged object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2777–2787, 2020.
  * [26] Jiaying Lin, Yuen Hei Yeung, and Rynson WH Lau.  Exploiting semantic relations for glass surface detection.  In _Advances in Neural Information Processing Systems_.
  * Zhao et al. [2018] Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia.  Icnet for real-time semantic segmentation on high-resolution images.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 405–420, 2018.
  * Hou et al. [2017] Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, and Philip HS Torr.  Deeply supervised salient object detection with short connections.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3203–3212, 2017.
  * Chen et al. [2018] Shuhan Chen, Xiuli Tan, Ben Wang, and Xuelong Hu.  Reverse attention for salient object detection.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 234–250, 2018.
  * Guan et al. [2022] Huankang Guan, Jiaying Lin, and Rynson WH Lau.  Learning semantic associations for mirror detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5941–5950, 2022.
  * Yu et al. [2020] Changqian Yu, Jingbo Wang, Changxin Gao, Gang Yu, Chunhua Shen, and Nong Sang.  Context prior for scene segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12416–12425, 2020.
  * Chen et al. [2019] Yunpeng Chen, Marcus Rohrbach, Zhicheng Yan, Yan Shuicheng, Jiashi Feng, and Yannis Kalantidis.  Graph-based global reasoning networks.  In _CVPR_ , pages 433–442, 2019.