# Augmenters at SemEval-2023 Task 1: Enhancing CLIP in Handling
Compositionality and Ambiguity for Zero-Shot Visual WSD through Prompt
Augmentation and Text-To-Image Diffusion

Jie S. Li  University of Maryland, College Park  Yow-Ting Shiue  University of
Maryland, College Park  Yong-Siang Shih  Duolingo, Inc.  Jonas Geiping
University of Maryland, College Park

###### Abstract

This paper describes our zero-shot approaches for the Visual Word Sense
Disambiguation (VWSD) Task in English. Our preliminary study shows that the
simple approach of matching candidate images with the phrase using CLIP
suffers from the many-to-many nature of image-text pairs. We find that the
CLIP text encoder may have limited abilities in capturing the compositionality
in natural language. Conversely, the descriptive focus of the phrase varies
from instance to instance. We address these issues in our two systems,
Augment-CLIP and Stable Diffusion Sampling (SD Sampling). Augment-CLIP
augments the text prompt by generating sentences that contain the context
phrase with the help of large language models (LLMs). We further explore CLIP
models in other languages, as the an ambiguous word may be translated into an
unambiguous one in the other language. SD Sampling uses text-to-image Stable
Diffusion to generate multiple images from the given phrase, increasing the
likelihood that a subset of images match the one that paired with the text.

##  1 Introduction

The Task of Visual Word Sense Disambiguation, as set out in SemEval-2023 Task
1 Overview Paper Raganato et al. (2023), can be described as follows: Given a
target word (the target word) in the context of a two or more word phrase (the
full phrase) and ten candidate images, pick the image (the gold image) among
the ten candidate images that correctly corresponds to the target word. This
competition was run in three languages, English, Farsi and Italian. We
participated in the English version of the task. This task is in line with
previous tasks connecting images to text, such as Elliott et al. (2017). We
explore two distinct systems to tackle this task. Both systems use Contrastive
Language-Image Pre-training (CLIP) (Radford et al., 2021) as a foundation.
CLIP was trained to associate text and related images, through increasing the
cosine similarity (CLIP-similarity) between the normalized text-embedding and
image-embedding of related text and image pairs and decreasing that for
unrelated pairs. Our first system (Augment-CLIP) augments the CLIP text
embedding by introducing additional context (through key-to-text) and
accessing CLIP text and image embedding in other languages, through third-
party implementation of CLIP for various languages. The second system (SD
Sampling) samples Stable Diffusion (Rombach et al., 2021) to generate multiple
images illustrating the semantics of the full phrase and then applies a
distance metric to select the candidate image that is closest to the generated
images.

As standalone systems, Augment-CLIP and SD Sampling do not outperform Base-
CLIP as the additional context may not correctly extend the target word
meaning, but they offer complementary benefits and improve Base-CLIP through
ensembling. We ensemble models by first calculating a new probability (or
score) for each candidate image by taking the equally weighted average of the
probability calculated from the underlying models. Each individual model can
output a probability for a candidate image. For CLIP-based models, the
probability is the softmax of the candidate image logits. We then rank the
candidate images based on the new probability in descending order, with the
highest probability candidate image being the predicted image from the
ensembled model. See Table 1.

##  2 Systems Overview

###  2.1 Augment-CLIP

We look at two methods to create the Augment-CLIP system. Both methods attempt
to disambiguate the full phrase containing the target word. The first does
this through introducing additional text and the second does this through
accessing additional languages.

####  2.1.1 Augment-CLIP with key-to-text

Our baseline approach, referred to as the Base-CLIP approach (or Base-CLIP
model) is the approach of encoding the full phrase using the CLIP text encoder
and encoding the candidate images using the CLIP image encoder, followed by
choosing the candidate image whose encoding has the largest similarity to the
full phrase encoding. Base-CLIP models, regardless of their specific
underlying architecture, suffer from a weakness in compositionality.
Compositionality is the change of word meaning in the presence of other words.
For example, the meaning of "baby powder" is not the average of "baby" and
"powder", and "powder" means different things in "baby powder" versus "milk
powder". This is a general problem with embeddings beyond CLIP, such as text
embeddings (Sutor et al., 2019). CLIP is trained with image and caption pairs
only, with captions consisting of shorter texts and of less diversity than
texts used to train language models, so the complex syntactic and semantic
relationships among words, including compositionality, is not well-captured by
CLIP. In comparison, standard language models trained on larger text corpora
are composed of longer texts, from a larger variety of sources. We utilize
this idea to augment Base-CLIP with key-to-text completion
111https://github.com/gagan3012/keytotext to leverage additional language
knowledge through the key-to-text system. We use the key-to-text systems "k2t"
(k2t 1), "k2t-base" (k2t 2), and "mrm8488/t5-base-finetuned-common_gen" (k2t
3).

For example, for the target word "administration" and the full phrase
"administration prime minister" from the trial data, we created three
additional sets of context texts:

  1. 1.

"The administration prime minister is the official title of the leader."

  2. 2.

"The Administration Prime Minister is the leader of the country."

  3. 3.

"prime minister speaks to the media during his visit."

These texts further reinforce the semantic meaning of "administration". The
CLIP text-embedding of the augmented context text is used to measure the CLIP-
similarity to the candidate images.

To keep the focus on the benefit of additional text context rather than
optimizing the context itself, we use a greedy method to sample key-to-text
and do not evaluate alternative sampling methods.

####  2.1.2 Augment-CLIP through additional languages

The second way to augment Base-CLIP is to resolve the ambiguity of the full
phrase in the source language by translating the full phrase into a different
language via a translation model (we leverage Google
Translate222https://translate.google.com/) and then use the other language’s
CLIP text-embedding of the translation to measure the distance to the
candidate images. We evaluate this idea with Chinese translations. Chinese
Augment-CLIP does not outperform Base-CLIP, often due to poor translation,
but, interestingly, it offers sufficient complementarity to Base-CLIP or other
Augment-CLIP that it improves performance through ensembling. See results in
Table 1.

Table 1: Augment-CLIP systems performance. | Trial Data |  | Test Data |
---|---|---|---|---
Systems | hit rate | mrr | hit rate | mrr
Base-CLIP (ViT-B/32) | 56.25 | 73.44 | 57.88 | 72.43
Aug-CLIP: zh | 43.75 | 62.14 | 51.40 | 67.60
Aug-CLIP: k2t 1 | 50.00 | 68.75 | 57.24 | 71.71
Aug-CLIP: k2t 2 | 56.25 | 70.42 | 58.10 | 71.98
Aug-CLIP: k2t 3 | 43.75 | 68.08 | 53.78 | 69.07
submit1=ensemble(B-CLIP, k2t 2) | 62.50 | 76.56 | 59.18 | 73.21
ensemble(B-CLIP, zh, k2t 2) | 68.75 | 81.25 | 63.71 | 76.11
ensemble(B-CLIP, zh, k2t 1, k2t 2, k2t 3) | 68.75 | 81.25 | 63.93 | 76.45
baseline: |  |  |  |
Base-CLIP (ViT-Large-Patch14-336) | 56.25 | 73.12 | 60.48 | 73.88

####  2.1.3 Base-CLIP model differences

For Base-CLIP, the performance difference in the two versions of CLIP that we
used, ViT-B/32 and Vit-L/14, is notable. ViT-B/32 in fact gave better
performance on trial and test data. This is unexpected as ViT-L/14 is a larger
model and has more training and more data (Radford et al., 2021). Further the
organizers’ baseline uses CLIP-ViT-large-patch14-336, an even larger model
which improved performance in test data. See Table 1. This leads to the
question of how different Base-CLIP embeddings affect performance on this
task, which is outside the scope of this paper as we take the Base-CLIP
embedding as a given in our systems.

###  2.2 Stable Diffusion Sampling

The second system samples text-to-image Stable Diffusion-v1.4 (SD) to generate
multiple images after inputting the full phrase as the text prompt. Then the
system outputs the candidate image with the closest distance to any of the
generated SD images. There are two advantages of this system: first is the
access to the larger training data of Stable Diffusion, which includes
LAION2B-en (Schuhmann et al., 2022), a 2.32 billion common crawl image-text
pairs dataset. Second, evaluating multiple images for a given text input
resolves the text ambiguity of the input text and also the pictorial ambiguity
in its image representation. As an example of text ambiguity, "angora" can
mean a type of fiber or less frequently a specific city, as in "Angora City".
Sampling several images allows the possibility that a subset of the images
correctly express the meaning of the target word. Even for an unambiguous
word, there may be pictorial diversity in its representation, and sampling
multiple images allows for broader coverage of this diversity than a single
image.

We evaluate two sampling methods of Stable Diffusion, text-to-image and text-
and-image-to-image. For each, two similarity metrics were used: CLIP-
similarity and $l_{2}$ distance between InceptionV3 (Szegedy et al., 2015)
features of candidate image and InceptionV3 features of SD sampled image. Of
these four, text-to-image sampling of Stable Diffusion with CLIP-similarity
performs the best on trial data and a subset of train data - this is
designated SD Sampling and is our submission 2 for the task.

For text-to-image sampling, we input the full phrase to SD and generate 50
output images (independent of any candidate images). We then calculate the
maximum CLIP-similarity (CLIP ViT-L/14) between a candidate image and the 50
output images and associate that largest CLIP-similarity to that candidate
image (candidate image distance). We then output the candidate image with the
largest CLIP-similarity.

##  3 Experimental setup

The trial, train, and test datasets consist of multiple instances. An instance
is a target word and a full phrase (containing the target word) and ten
candidate images, with one image (the gold image) capturing the semantic
meaning of the target word as exemplified in the full phrase. Train, trial,
and test have $12869$, $16$, and $463$ instances, respectively. For the test
data, there are two versions of the dataset provided by the task organizers,
differing in the image size Raganato et al. (2023). We perform our predictions
on the dataset with the smaller image size.

We do not train or fine-tune our models on training data to demonstrate the
zero-shot property of our approach, although we do use the training data in
part to inform us of which Augment-CLIP system and which SD Sampling system to
select for task submissions. Based on trial data performance, among the three
k2t systems, we choose k2t 2, and among the SD Sampling systems, we choose
text-to-image with CLIP-similarity.

As measurements of the performance of the models, hit rate and mean reciprocal
rank (mrr) are applied to the model predictions on the trial dataset and test
dataset. Based on the inputs, the model assigns a score to each candidate
image. The model can output one predicted image, with the highest score, or it
can output a list of images ordered in decreasing order of score. Hit rate is
the percentage of instances where the predicted image is the gold image. Mean
reciprocal rank is the average of the reciprocal of the rank of the gold image
in the list of images, ordered based on score.

##  4 Results

###  4.1 Augment-CLIP through key-to-text

While standalone Augment-CLIP through key-to-text does not outperform Base-
CLIP, it does reveal that adding context can improve performance. The
additional context, when correctly augmenting the meaning of the target word,
can indeed improve performance on the test set. In the best-case scenario, the
additional context is an extension and explanation of meaning of the target
word. In the worst-case, an incorrect extension of context dilutes the meaning
of the target word. In the former case, Augment-CLIP is likely to correctly
predict the gold image. In the group of instances in which both Augment-CLIP
and Base-CLIP correctly predict the gold image, the CLIP-similarity score is
higher in Augment-CLIP than in Base-CLIP, showing the effectiveness of added
context. This depends on the quality of the context extension process: if the
augmented context does not aid in conveying the correct semantic meaning of
the target word, then the incorrect additional context may degrade performance
in a standalone system. This is analogous to the performance of a language
model with in-context learning, where the performance depend on the quality of
in-context examples (Liu et al., 2021).

Table 2: Mean of CLIP similarity values of an instance of the full phrase to gold image versus text to all candidate images in the test data. Given the scale of similarity values shown in this table, increasing the similarity value between text and its associated image by values as small as $0.01$ could noticeably improve the system performance.  | Test Data | Test Data
---|---|---
Systems | sim(text, gold img) | sim(text, all imgs)
Base-CLIP | 0.2932 | 0.2276
Aug-CLIP: kt2 2 | 0.2929 | 0.2276

Adding a k2t system can improve performance of the Base-CLIP. This can be seen
in Table 3. For each instance in the dataset, consider the Base-CLIP
similarity score for the full phrase and the gold image and consider the
Augment-CLIP through k2t similarity score for the full phrase and the gold
image. The difference between the Augment-CLIP similarity score and the Base-
CLIP similarity score is calculated and shown in Table 3. This difference
shows whether Augment-CLIP would have done better or worse than Base-CLIP. It
also shows the potential of Augment-CLIP to improve Base-CLIP’s performance.

Extra steps can be taken to improve the quality of the k2t text completion,
but our focus is not to improve the performance of the k2t system but to show
that reasonable additional context offers complementary benefits to Base-CLIP.

Table 3: Test dataset confusion matrix: Aug-CLIP k2t 2 similarity(text, gold image) minus Base-CLIP similarity(text, gold image) and count of instances. Aug-CLIP can increase performance of Base-CLIP when Base-CLIP is incorrect by increasing the similarity(text, gold image). | difference | Aug-CLIP: k2t 2 |
---|---|---|---
| in similarity | correct | incorrect
Base-CLIP | correct | 0.0015 | -0.0248
| incorrect | 0.0107 | -0.0025
| instance | Aug-CLIP: k2t 2 |
| count | correct | incorrect
Base-CLIP | correct | 234 | 34
| incorrect | 36 | 159

###  4.2 Augment-CLIP through other languages

We evaluate another method to disambiguate the full phrase by translating the
full phrase in English to another language and exploring the CLIP text
embedding and image embedding in that foreign language. Direct translation to
a foreign language (through taking the first result of Google Translate), with
that language chosen to be Chinese (Aug-CLIP: zh) in our evaluation, does not
increase performance, and this is partly due to incorrect translations. Here,
identical round-trip translations can serve as a proxy for correct translation
from English to Chinese. The test instances can be divided into two groups,
one being identical in round-trip translation and the other group containing
all other instances. For instance, starting with the English full phrase and
translating to Chinese and then translating that result back to English
(English_1 $\rightarrow$ Chinese $\rightarrow$ English_2 with English_1 and
English_2 being the same, up to capitalization) and another group that does
not have identical round-trip translation, the first group has a higher
foreign-language CLIP-similarity score than the second.

As a standalone system, direct translation does not improve performance, but
ensembling with a direct translation system does improve performance. By
adding Chinese translation to the ensemble (ensemble(B-CLIP, zh, k2t 2)), test
data hit rate increases from $59.18$ to $63.71$ and test data mrr increases
from $73.21$ to $76.11$. See Table 1.

###  4.3 SD Sampling

SD Sampling does not outperform Base-CLIP. It’s worth noting that the
instances where SD Sampling correctly selects the gold image is different from
those of Base-CLIP, showing a potential to gain from accessing the SD Sampling
system. See Table 4.

Table 4: SD Sampling versus Base-CLIP test dataset confusion matrix: count of test instances. | instance | SD Sampling |
---|---|---|---
| count | correct | incorrect
Base-CLIP | correct | 190 | 78
| incorrect | 56 | 139

There is pictorial diversity in the SD samples, and often that diversity
includes the correct image expression of the target word in the full phrase,
as intended. There is diversity in viewpoint, proximity and style of the
object presented. See images of various cityscapes outputted by Stable
Diffusion for the full phrase "angora city" in Figure 2, and see images of
various views of different models of "internet routers" in Figure 3. There is
also diversity in the semantic interpretation of the full phrase: see for
example Figure 1 for interpretation of the word "breaking wheel" as both a
torture device and a music group. This shows that the first goal of the SD
System of producing a diversity of pictorial representation of the desired
object is met. But the subsequent application of the distance metric fails to
match the sampled SD image to the gold image. At times, incorrect candidate
images have larger CLIP-similarity to the correct sampled SD image than the
gold image, due to a coincidence of similar style, lighting, or material. This
is not a shortcoming of CLIP-similarity as it is intended to be applied to
(text, image) pairs and not (image, image) pairs (Radford et al., 2021). As an
alternative, we evaluate metrics such as $l_{2}$ distance between InceptionV3
features of the sampled image and InceptionV3 features of the candidate image.
Using $l_{2}$ metric underperforms the CLIP-similarity metric as shown Table
5. We do not evaluate other image-to-image similarity metrics and leave for
future work the search for an effective metric.

Figure 1: Top row: "wheel" in "breaking wheel" with gold image and SD Sampling
predicted candidate image. Bottom row: Examples from SD Sampling of "breaking
wheel", as both an object and a music band. Figure 2: Top row: "angora" in
"angora city" with gold image and SD Sampling predicted candidate image.
Bottom row: Examples from SD Sampling, with various views of a city. A view of
a side street of a city matches more closely an incorrect candidate image of a
natural wall. Figure 3: Top row: "router" in "internet router" with gold image
and SD Sampling predicted candidate image. Bottom row: Examples from SD
Sampling, with various views of a router. A broad view of a router does not
match the close up view presented in the gold image.

An issue with SD sampling on this dataset is the domain shift between the
dataset on which SD was trained, a common crawl of text and image pairs in
English, and the more scientific and technical focus of the full phrases in
the test data. For example, the full phrase "breaking wheel" is a historical
term and meant to be unambiguous and to resolve to mean a medieval torture
device and the gold image is of such a device. On the other hand, to the
layperson, "breaking wheel" sounds like the name of a band, akin to Stone
Sour, Breaking Benjamin, Nickelback, and this popular understanding of
"breaking wheel" is evidenced in Stable Diffusion sampled images, which
include images of band groups. Similarly, other instances with the full phrase
being technical and scientific terms that are not well-known to the general
public are expressed in Stable Diffusion output images in terms of how a
layperson would interpret such a term, instead of the correct technical
meaning.

Table 5: SD Sampling systems performance. | Trial Data |
---|---|---
Systems | hit rate | mrr
(text)2img inception | 31.25 | 48.24
(text,img)2img inception | 31.25 | 53.39
submit2=(text)2img CLIP | 31.25 | 56.00
(text,img)2img CLIP | 37.50 | 55.00
| Test Data |
Systems | hit rate | mrr
(text)2img inception | 34.77 | 52.59
(text,img)2img inception | 28.08 | 49.41
submit2=(text)2img CLIP | 53.35 | 68.75
(text,img)2img CLIP | 44.28 | 63.14

##  5 Conclusion

The Base-CLIP system is a strong solution to the task challenge. Our system
Augment-CLIP complements Base-CLIP to resolve text ambiguity and improves text
compositionality. Our system SD Sampling provides pictorial diversity in
ambiguous and unambiguous text interpretation. These two methods offer
additional ways to connect text and images.

## References

  * Elliott et al. (2017) Desmond Elliott, Stella Frank, Loïc Barrault, Fethi Bougares, and Lucia Specia. 2017.  Findings of the second shared task on multimodal machine translation and multilingual image description.  In _Proceedings of the Second Conference on Machine Translation_ , pages 215–233, Copenhagen, Denmark. Association for Computational Linguistics.
  * Liu et al. (2021) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021.  What makes good in-context examples for gpt-3?  _ArXiv_ , abs/2101.06804.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_.
  * Raganato et al. (2023) Alessandro Raganato, Iacer Calixto, Asahi Ushio, Jose Camacho-Collados, and Mohammad Taher Pilehvar. 2023.  SemEval-2023 Task 1: Visual Word Sense Disambiguation.  In _Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)_ , Toronto, Canada. Association for Computational Linguistics.
  * Rombach et al. (2021) Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2021.  High-resolution image synthesis with latent diffusion models.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10674–10685.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _ArXiv_ , abs/2210.08402.
  * Sutor et al. (2019) Peter Sutor, Yiannis Aloimonos, Cornelia Fermuller, and Douglas Summers-Stay. 2019\.  Metaconcepts: Isolating context in word embeddings.  In _2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)_ , pages 544–549.
  * Szegedy et al. (2015) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2015.  Rethinking the inception architecture for computer vision.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2818–2826.


# Exploring Activation Patterns of Parameters
in Language Models

Yudong Wang, Damai Dai, Zhifang Sui
MOE Key Lab of Computational Linguistics, School of Computer Science, Peking
University
<EMAIL_ADDRESS>{daidamai<EMAIL_ADDRESS>

###### Abstract

Most work treats large language models as black boxes without in-depth
understanding of their internal working mechanism. In order to explain the
internal representations of LLMs, we propose a gradient-based metric to assess
the activation level of model parameters. Based on this metric, we obtain
three preliminary findings. (1) When the inputs are in the same domain,
parameters in the shallow layers will be activated densely, which means a
larger portion of parameters will have great impacts on the outputs. In
contrast, parameters in the deep layers are activated sparsely. (2) When the
inputs are across different domains, parameters in shallow layers exhibit
higher similarity in the activation behavior than deep layers. (3) In deep
layers, the similarity of the distributions of activated parameters is
positively correlated to the empirical data relevance. Further, we develop
three validation experiments to solidify these findings. (1) Firstly, starting
from the first finding, we attempt to configure different prune ratios for
different layers, and find this method can benefit model pruning. (2)
Secondly, we find that a pruned model based on one calibration set can better
handle tasks related to the calibration task than those not related, which
validate the second finding. (3) Thirdly, Based on the STS-B and SICK
benchmark, we find that two sentences with consistent semantics tend to share
similar parameter activation patterns in deep layers, which aligns with our
third finding. Our work sheds light on the behavior of parameter activation in
LLMs, and we hope these findings will have the potential to inspire more
practical applications.

##  1 Introduction

Ever since the emergence of GPT-4 achiam2023gpt , there has been a surge of
interest in Large Language Models (LLMs). As these LLMs continue to advance
and their capabilities strengthen, there remains a noticeable gap in research
dedicated to their interpretability. In this study, we aim to investigate the
coexistence of different capabilities within the model. More specifically,
when faced with inputs that are across different domains, we observe
variations in the internal representation of Large Language Models (LLMs).
There have been some explorations into the functions of specific layers and
parameters in LLMs azaria2023internal ; geiger2024finding . It is generally
recognized that some significantly different capabilities in LLMs cannot fully
coexist within a limited scale. However, there is still no targeted research
that analyzes the operational patterns of different capabilities within large
models in a more general sense.

Recent work has found that there may be some parameters within the model that
exist for specific tasks. Fu et al. fu2023specializing  revealed that while
the distilled model excels in the specific task it was designed for, the
original, more general model experiences a decline in performance in other
tasks it was previously proficient. This observation suggests that different
tasks may tap into distinct capacities within a model, and these capacities
seem to be mutually exclusive to some extent. In another study, Zhang et al.
zhang2021moefication  introduced the notion that LLMs inherently evolve into a
Mixture of Experts (MoE) within themselves. This concept implies that
different sections of the network are tasked with handling different inputs,
further strengthening the idea of internal specialization within the model.

Building on the insights gleaned from the aforementioned phenomena, this study
seeks to unravel the following questions: Which parameters within the network
are activated to determine the outputs and does the distribution of these
activated weights exhibit distinct patterns when faced with inputs across
different domains? In essence, we aim to explore whether the degree of
parameter activation varies in response to non-homogenous input scenarios, and
if so, to what extent.

Drawing on methods from network pruning ma2023llm , we assess the influence of
a parameter by comparing the original output of a model to that of a model in
which the parameter is set to $0$. Specifically, we employ the first-order
term of the Taylor expansion of the model to gauge the impact of a parameter
on the outputs. Given two inputs $x,y$, whether homogenous or not, we derive
two vectors $v_{x},v_{y}$ that characterize the influence of the internal
parameters of the model. By examining the cosine similarity between these two
vectors from LLM with different data (LLMDcos), we observe three phenomena:

  * •

For inputs in the same domain, parameters in the shallow layers of the model
are activated densely, while parameters in the deep layers are activated
sparsely.

  * •

For inputs from different domains, the similarity of the activation patterns
of parameters in the shallow layers of the model is higher than deep layers.

  * •

In deep layers, the similarity of the distributions of activated parameters is
positively correlated to the empirical data relevance.

To validate our observed results, we designed three experiments, which include
model pruning and semantic similarity tasks. The pruning method improved based
on our analytical results outperformed the original pruning method. We
validated our second finding by comparing the performance changes caused by
the different calibration sets of the pruning method. The proposed LLMDcos was
also validated to be related to semantic similarity.

Our contributions are listed in the following:

  * •

We employ a novel approach to analyze the internal capabilities of the model.

  * •

We observed the different capabilities of different layers in LLMs, summarized
three phenomena, and designed experiments to validate each respectively.

  * •

We optimized the pruning method, providing a reference for other pruning
methods.

  * •

We proposed a new method for calculating data similarity based on gradient
information.

##  2 Background and Motivation

###  2.1 Motivation

Our motivation for this study stems from a phenomenon observed in distillation
fu2023specializing : when one capability of a general model is enhanced
through distillation, there tends to be a corresponding decline in other
evaluations. This observation prompts us to investigate the nature of the
relationship between different capabilities within a model - are they mutually
reinforcing or mutually exclusive? And if both, under what circumstances does
each scenario occur?

A study by zhang2021moefication  proposed the idea that a model internally
generates a Mixture of Experts (MoE), which suggests that the model handling
of different tasks could be attributed to the spontaneous formation of a
sparse structure during training. This structure, in turn, might harbor
distinct capabilities that are mutually exclusive to some extent.

Our work is primarily related to two areas: model pruning and data similarity.
The parameter scoring method from model pruning serves as a valuable tool to
explore which parameters within the model are most responsive to a given
input. Meanwhile, the variation in the model’s performance during evaluation
due to different pruning settings can also validate our conclusions. On the
other hand, we have utilized non-homologous data to investigate whether there
would be different activation distributions within the model. From the
results, the different activation distributions are related to data relevance.
Through this lens, we aim to shed light on the internal dynamics of large
language models and their response to varying inputs.

###  2.2 Causal Inference

In line with our intention, many techniques in causal inference are also aimed
at exploring the mechanisms and patterns within the network. These techniques
include probing, attribution methods, and causal abstraction.

Probes are essentially models that are trained with the internal
representations of a neural network as input, aiming to explore the inherent
semantics within the model hupkes2018visualisation ; peters2018dissecting ;
tenney2019bert ; clark2019does . A plethora of studies employing probes have
delved into internal information related to aspects such as time, space, and
inferential variables.

Attribution methods shrikumar2016not ; sundararajan2017axiomatic , in line
with our objectives, strive to quantify the degree to which a representation
contributes to the output of the model for a specific sample or set of
samples. Using gradient information, attribution methods inherently offer
explanations, thereby demystifying the inner mechanics of the neural network.

Causal abstraction geiger2021causal ; geiger2024finding , on the other hand,
concentrates more on the specific implications within the network. It
evaluates the effect of a weight or neuron by fixing, disturbing, or setting
it to zero and observing the difference in the outputs. Like probes, causal
abstraction requires extensive prior knowledge, which limits its ability to
examine more general situations and information.

However, the aforementioned techniques, probe and causal abstraction, are
aimed at verifying whether specific information from human reasoning processes
exists within the network. Meanwhile, attribution methods concentrate on
specific data within the dataset. In contrast, our work primarily focuses on
explaining the internal mechanisms of the model by studying the differences in
the model’s internal state when facing inputs from different domains.

###  2.3 Model Pruning

The crux of model pruning lies in identifying the crucial parameters within
the network. From the perspective of model pruning, we can derive insights
into the significant role scoring of parameters.

Model pruning techniques lecun1989optimal ; hassibi1993optimal ;
han2015learning  for LLMs can be broadly categorized into two types
zhu2023survey : structured pruning frantar2023sparsegpt ; zhang2023pruning ;
sun2023simple  and unstructured pruning santacroce2023matters ; ma2023llm .
Structured pruning aims to reduce the hidden state size by removing entire
rows or columns from the weight matrix, which can lead to actual acceleration
and pruning benefits. However, this method often results in a significant loss
of performance. Unstructured pruning, on the other hand, involves eliminating
individual connections, i.e., specific elements within the weight matrix. This
approach can maintain model performance even at high pruning ratios but does
not inherently lead to computational speedup unless a substantial proportion
of connections is pruned within specific regions.

Regardless of the type of pruning, both methods focus on identifying which
components of the network have the least impact on the output. Many studies
have utilized Taylor expansion to define the rank of weights in terms of their
influence on the network’s structure, thereby guiding the pruning process by
removing weights with minimal impact.

In this work, we draw upon the Taylor expansion to define the degree to which
internal weights are activated, thereby investigating the underlying
mechanisms within the model.

##  3 Preliminary Findings

In all the content of this paper, unless specifically marked, all model
results are analysis results of Llama2-7b-hf touvron2023llama . This paper
provides results from more models in subsequent sections.

###  3.1 Definition of Activation and LLMDcos

We begin with a standard deep learning problem in an empirical scenario. Given
two data sources $X,Y$, our aim is to quantify the influence of the parameters
$w$ within the model $D$. The activation of $w_{i}$ is defined as

$\mathcal{A}(X,w_{i})=|D(X,w_{i})-D(X,0)|=|w_{i}\cdot\frac{\partial
D(X,w_{i})}{\partial w_{i}}+O(w_{i}^{2})|\approx|w_{i}\cdot\frac{\partial
D(X,w_{i})}{\partial w_{i}}|$ (1)

By concatenating all the $\mathcal{A}(w_{i})$, we derive $\mathcal{A}(w)\in
R^{n}$, where $n$ denotes the number of parameters within the model. We
examine $\mathcal{A}(w)$ across different sentences from various data sources.
We define the cosine metric of a Data pair based on the Large Language Model
(LLMDcos):

$LLMDcos(X_{1},X_{2})=\frac{A(X_{1},w)\cdot
A(X_{2},w)}{\sqrt{\|A(X_{1},w)\|^{2}\cdot\|A(X_{2},w)\|^{2}}}$ (2)

Where $X_{1},X_{2}$ are two inputs from the same domain or different domains.
With LLMDcos, our aim is to analyze different layers within LLMs, as well as
the differences between different inputs.

layer 1

layer 2

layer 9

layer 16

layer 24

layer 31

Figure 1: Activated Parameter Statistics. The x-axis represents the value of
$A(w_{i})$, and the y-axis represents the quantity. For convenience of
statistics, we have performed statistical processing, so the numerical value
on the y-axis is an estimate of one-thousandth of the actual value. The first
row is the statistical image for Boolq, the second row is for HumanEval, and
the third row is the activation statistics for MMLU.

###  3.2 Finding 1: Parameter Activation Patterns for Inputs in the Same
Domain

In this section, we attempt to analyze the distinct behaviors within Large
Language Models (LLMs). Using our defined $\mathcal{A}(w_{i})$, we analyze the
distribution of activated parameters in different layers when faced with a
single input.

In Figure 1, we present the statistical results from three data sources: Boolq
clark2019boolq , HumanEval chen2021codex , and MMLU hendrycks2020measuring .
For each dataset, we selected 64 samples and averaged the activation status of
each parameter facing different samples. When calculating the activation
status of data in different layers, we collectively consider all the tunable
parameters within a layer. Specifically, this includes parameters from seven
parts: the fully connected linear layers of Q, K, V, O, and the three fully
connected linear layers of the MLP layer. To facilitate the statistics, we
sort the $\mathcal{A}(w_{i})$ values of the parameters, take the average every
1000 units, reducing the original 20,000 parameters to 20, and then perform
the distribution statistics.

From the statistical results, we have selected the distribution
characteristics of representative layers 1, 2, 9, 16, 24, and 31 for display.
The full results can be referred to in the appendix. As shown in the figure,
we can find that fewer parameters are activated in the first layer, meaning
that only a small portion of parameters have a significant impact on the
results. In layers 2-9, the parameters that have a greater impact on the
results gradually increase. In the relatively deeper layers, the parameters
that have a significant impact on the results decrease, concentrating on
specific parts. This phenomenon is consistent across the three data sets
representing different abilities. This leads us to speculate that for a single
task, apart from the first layer, many parameters in the shallow layers are
involved in the calculation of the results. Conversely, in the deep layers and
the first layer, only a few parameters have a significant impact on the
results.

Boolq-Boolq

MMLU-MMLU

HumanEval-HumanEval

Boolq-MMLU

Boolq-HumanEval

MMLU-HumanEval

Figure 2: Statistics of LLMDcos values for different layers. The x-axis
represents the layer number, and the y-axis represents the LLMDcos values of
64 samples. The two data sources are indicated below the image. All images are
results from Llama2-7b.

###  3.3 Finding 2: Parameter Activation Patterns for Inputs in the Different
Domains

In order to observe the functionality of different layers within Large
Language Models (LLMs), we have documented the activation scenarios of various
layers when faced with input from different data domains. In response to
inputs from two data domains, we calculated the LLMDcos for each layer each
time.

As depicted in Figure 2, we conducted an analysis on data from three data
domains: Boolq, HumanEval, and MMLU. Apart from HumanEval-HumanEval where we
only experimented with 16 sample groups, we statistically analyzed 64 sample
groups in all other experiments. Consistent with the previous section, within
the same layer, we included the parameters from seven parts in our statistics.

From the results, we observed that when faced with the same data source, the
activation levels of different layers were similar. When faced with different
data sources, the first 12 layers of the network had higher LLMDcos, while the
later layers had lower similarity. Notably, the second layer had a high degree
of similarity in any analysis between two data sources. Moreover, when faced
with empirically similar data sources, the similarity in activation levels of
the later layers was relatively high, that is, MMLU and Boolq showed
relatively high similarity, while the activation distribution of both datasets
had a significant difference from the HumanEval dataset. Combining the
conclusions from the previous section, we can speculate that for different
tasks, the shallow end of the network has a more general understanding
ability. When faced with different tasks, similar parameters are activated to
understand the problem, especially in the second layer. In the deeper layers,
the network has relatively dispersed parameters, that is, some parameters are
activated for specific tasks.

###  3.4 Finding 3: Observing Data through Activated Distribution

Through the aforementioned experiments, we observed that for different
datasets, the similarity of the deep layers in the model significantly
decreases, while for the same dataset, the similarity between the shallow and
deep layers of the model remains consistent. This leads us to hypothesize that
the similarity in the later layers may be related to semantic similarity. To
validate this hypothesis, we test the similarity of more datasets in the
Section 4.3, as well as conduct tests on benchmarks for semantic similarity.

###  3.5 Generality Across Different Models

Llama-7B
MMLU-MMLU

Qwen-7B
MMLU-MMLU

Llama2-13B
MMLU-MMLU

Llama-7B
MMLU-HumanEval

Qwen-7B
MMLU-HumanEval

Llama2-13B
MMLU-HumanEval

Figure 3: Statistics of LLMDcos values for different models. The x-axis
represents the layer number, and the y-axis represents the LLMDcos values of
64 samples. The two data sources and the model are indicated below the image.
The images are consistent with the results from Llama2-7B.

To determine the universality of our results, rather than their specificity to
the Llama series or the 7B models, we performed experiments analogous to those
in Section 3.2 on Qwen-7b bai2023qwen , Llama2-13b and Llama-7b
touvron2023llama1 . The results are illustrated in the Figure 3.

It is noteworthy that the results from llama-7b are remarkably similar to
those from llama2-7b. Although the outcomes from qwen-7b and llama2-13b
deviate somewhat from those of llama2-7b, the overall trend remains
consistent, i.e., there is an extremely high degree of similarity in certain
layers at the shallow part, with the shallow end generally exhibiting a higher
degree of similarity and the deep half showing a lower degree of similarity.

##  4 Validation Experiments

Given that the assessments in Section 3.1 are entirely based on our definition
of parameter activation levels. This paper, drawing upon the preceding
analysis, suggests several application approaches. The success of these
applications substantiates the validity of our analytical results.

###  4.1 Validation 1: Pruning LLMs with Different Sparasity According to the
Activation Level

Based on the analysis results from Section 3.2, we can observe the following
phenomena: For a 32-layer Llama-2-7b-hf network, most of the parameters in the
first layer and the deep half of the network have little impact on the
results. However, in the parameters of the 2-17 layers of the network, there
are relatively more parameters that have a significant impact on the results.
To validate this conclusion, we will prune the model, making the 2-17 layers
of the network more sparse, while the 1st and 18-32 layers of the network have
a smaller degree of sparsity.

We employed the unstructured pruning method proposed by sun2023simple , and
while keeping all other settings unchanged, we set the 2-17 layers (2-21 for
Llama2-13B) to be pruned by only 45%, while the 1st and 18-32 (1st and 22-40
for Llama2-13B) layers were pruned by 55% in the setting where 50% of the
whole network was pruned. To compare the test results of the network, we
conducted tests on two different metrics on six datasets based on the original
settings. All the calibtration dataset is C4 raffel2020exploring . The
evaluation datasets include Wikitext2 merity2016pointer , Boolq, SIQA
sap2019socialiqa , PIQA Bisk2020 , Hellaswag zellers2019hellaswag , and MMLU.

As can be seen from the results in the Table 1, we observe that our method
consistently improves the Perplexity (PPL) value on Wikitext2 across all model
results, suggesting that our approach can generally enhance the language
modeling capability of the models. In the zero-shot results, it is worth
noting that the improvements in Hellaswag are universal, while Boolq generally
experiences a decline. In conjunction with the results from the Figure 4, we
find that Hellaswag has the highest correlation with C4, while Boolq is
relatively lower. Therefore, the results of using C4 as the calibration set
are less satisfactory in Boolq and other evaluations.

###  4.2 Validation 2: Pruning LLMs with Different Calibration Set

Models | pruning method | wikitext2 | zero-shot | MMLU
---|---|---|---|---
Boolq | SIQA | PIQA | hellaswag
Llama-7B | Wanda | 7.26 | 66.41 | 35.94 | 52.73 | 29.30 | 20.70
Wanda(ours) | 7.19 | 62.50 | 35.94 | 52.34 | 29.69 | 25.78
Llama2-7B | Wanda | 6.46 | 77.73 | 40.23 | 51.56 | 29.30 | 37.11
Wanda(ours) | 6.38 | 73.05 | 41.41 | 51.17 | 34.38 | 35.55
Llama2-13B | Wanda | 5.58 | 81.64 | 55.47 | 53.91 | 50.00 | 42.58
Wanda(ours) | 5.52 | 80.08 | 55.47 | 55.86 | 55.08 | 41.41
Table 1:  Pruning experiment results. All models have an overall pruning ratio of 50%, with c4 as the calibration set. The evaluation metric for wikitext2 is perplexity, and for MMLU it is 5-shot. Wanda (ours) refers to our Wanda model after layer-by-layer adjustment of the pruning ratio, ensuring the overall pruning ratio remains the same.  Models | Calibration set | wikitext2 | zero-shot | MMLU
---|---|---|---|---
Boolq | SIQA | PIQA | hellaswag
Llama-7B | Boolq | 7.22 | 63.67 | 39.84 | 52.73 | 24.61 | 25.78
SIQA | 7.44 | 60.94 | 34.38 | 52.34 | 25.78 | 24.61
Llama2-7B | Boolq | 6.44 | 73.83 | 42.97 | 51.56 | 32.42 | 35.16
SIQA | 6.67 | 71.88 | 46.48 | 53.13 | 32.03 | 36.33
Llama2-13B | Boolq | 5.57 | 80.08 | 57.03 | 53.91 | 53.91 | 43.36
SIQA | 5.70 | 75.39 | 57.03 | 57.81 | 50.39 | 47.27
Table 2:  Pruning experiment results. In the figure, all models have an
overall pruning ratio of 50%, and the pruning method used is Wanda (ours). The
evaluation metric for wikitext2 is perplexity, and for MMLU it is 5-shot.

Based on the results from Section 3.3, we observe the following: For a
32-layer Llama-2-7b-hf network, when faced with different data domains, the
majority of parameters in the deep part of the network exhibit a lower degree
of activation similarity. In contrast, in the shallow layers of the network,
there is a relatively higher degree of similarity in the distribution of
parameter activation. This leads us to hypothesize that the shallow layers of
the network consist of more generic parameters, while the deep layers are
discretely composed of parameters that address different problems. To validate
this conclusion, we will modify the calibration set to specifically prune for
specialized tasks, under the premise of pruning different layers of the model
at varying proportions as outlined in the previous section. We will then
verify whether this results in a decrease in performance on other test
results.

We adopted the pruning method from Section 4.1: Wanda (ours), ensuring all
other settings remained unchanged. We adjusted the calibration set to be the
same length as Boolq and SIQA. To ensure the sentence length of the
calibration set is the same, we concatenated different Boolq and SIQA data,
including answers, into long sentences and cut them to a specific length (2048
for the 7b model, 4096 for the 13b model).

The results are shown in the Table 2. We observe that compared to the C4
dataset, the language modeling capability of the pruning results using Boolq
as the calibration set continues to decline, while the PPL value of SIQA is
even lower. This might be due to Boolq serving as reading material, which
encompasses a wider range of knowledge, while the content of SIQA is
relatively singular by comparison. For Llama-7b, the results on most zero-shot
tasks declined due to the substantial loss of language modeling capability in
SIQA. However, the results for Llama-2 align with our expectations. The
pruning results using Boolq as the calibration set consistently outperform
those based on C4 and SIQA on Boolq. Moreover, on PIQA, MMLU, and SIQA, three
evaluations with stronger correlation with SIQA, the pruning model results
using SIQA as the calibration set always outperform those of C4 and Boolq.
This further validates our hypothesis.

###  4.3 Validation 3: Semantic Similarity with LLMDcos

In the analysis presented in Section 3.4, we observed that the similarity of
the deep layers of the network decreases for different data domains.

To validate the relationship between LLMDcos and data relevance, we tested its
performance on a semantic similarity benchmark STS-B cer2017semeval  and SICK
marelli-etal-2014-sick .

As shown in the Table 3, the LLMDcos calculated by Llama2-7B yielded the best
results. As evident from the results, LLama2-7b achieved the best performance,
while the results of Llama2-13B were relatively inferior. We believe that,
compared to semantic similarity, LLMDcos assesses more of the similarity in
the capabilities of the Large Language Models (LLMs) required by the inputs.
For 13B models, these capabilities may be more densely represented in the
parameters, leading to these deviations.

In addition to this, we calculated the similarity relationships across nine
datasets, including Boolq, C4, GSM8K cobbe2021gsm8k , Hellaswag, HumanEval,
MMLU, PIQA, SIQA, and Wikitext2. The results are shown in Figure 4

Model | STS-B | SICK
---|---|---
Llama-7B | 0.30 | 0.52
Llama2-7B | 0.66 | 0.51
Llama2-13B | 0.43 | 0.52

Table 3: Spearman correlation between LLMDcos and semantic similarity. We
sampled 256 examples on each dataset, and all p-values were far less than
0.001. We use the LLMDcos of 20-30 layers (20-38 for Llama2-13B)

Figure 4: Dataset relevance. The figure shows the similarity calculated based
on the mean LLMDcos of layers 16-29 in Llama-2-7b.

##  5 related work

###  5.1 data similarity

Our work reflects data relevance. In the field of NLP, data similarity mainly
refers to text similarity. Text similarity in early machine learning was
primarily based on statistical methods, such as word frequency and sentence
length. However, in the era of deep learning, semantic similarity has become
of greater interest, leading to the proposal of a series of models. Recently,
as data training efficiency has been increasingly recognized, many data
selection works that employ clustering ideas are based on data similarity.

Most of the relatively recent work has directly calculated relevance using the
hidden state after the embedding layer. laskar2020contextualized  proposed two
methods for calculating data similarity, one based on the average pooling of
the token dimension of BERT’s hidden state, and the other through training
with the CLS token. Both methods assisted in the main task of answer
selection. li2020sentence  improved the effect of similarity calculation by
mapping data similarity to a Gaussian distribution based on a kernel
distribution.

In addition, there has been little new development in evaluation datasets. The
STS-B dataset has not been updated since 2017, and it consists only of similar
short sentences. In the era of large language models, we may need a more
generalized text similarity.

###  5.2 Role of Each Layer in LLMs

In many causality inference papers, the functionality of different layers of
the model is discussed. These studies employ probes, causal alignment, and
some even more drastic measures, such as directly skipping certain layers to
observe the difference in results. However, most of the work focuses on a
specific function, rather than a macroscopic discussion of the functions of
different layers.

For example, Zhao et al. zhao2023causality  found that the third layer in
Llama may have a significant impact on whether the model outputs toxic
information. Azaria et al. azaria2023internal  verified that the later layers
of the model are more aware of whether they contain false information. Mcgrath
et al. mcgrath2023hydra  indicated that different layers in the model may have
different information about numbers.

##  6 limitation and future work

###  6.1 limitations

We identify two main limitations of this study. On one hand, there are
constraints due to GPU limitations, and on the other hand, there is a lack of
theoretical proof.

Due to GPU memory limitations, we only have eight A40s for experimentation.
These memory constraints prevent us from verifying the results of Llama-70b.

In terms of theory, our activation degree algorithm is a very rough
calculation. However, more refined calculations are constrained by time
complexity and space complexity, and likewise cannot be performed on our
machine. With more effective mathematical reasoning, we believe we can obtain
more refined results.

###  6.2 Future Work

In this paper, we find that the degree of activation may be related to data
similarity, which inspires us to think about a new dimension of data
relevance: What kind of capability is used to model the current sentence? Data
similarity based on capability may play different roles in the pre-training of
large models and SFT.

##  7 Conclusion

This paper proposes a metric for measuring the activation patterns of internal
parameters in language models, and subsequently introduces LLMDcos to
calculate the similarity of internal network activations when facing inputs
from two different data domains. Based on this metric, we made three
discoveries, each reflecting that the shallow layers of the network are more
generic, the deep layers possess more specific capabilities, and the deep
layer’s LLMDcos is related to data similarity. To validate our findings, we
designed three corresponding verification experiments. Two pruning experiments
respectively verified the differences in the distribution of internal
parameter activations in different model layers. Furthermore, the results on
the semantic similarity benchmark also reflected that the deep layer’s LLMDcos
can represent data similarity. We hope that our work can advance researchers’
understanding of LLM.

## References

  * (1) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * (2) Amos Azaria and Tom Mitchell.  The internal state of an llm knows when its lying.  arXiv preprint arXiv:2304.13734, 2023.
  * (3) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  arXiv preprint arXiv:2309.16609, 2023.
  * (4) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.  Piqa: Reasoning about physical commonsense in natural language.  In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
  * (5) Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia.  Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation.  arXiv preprint arXiv:1708.00055, 2017.
  * (6) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code.  2021\.
  * (7) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  arXiv preprint arXiv:1905.10044, 2019.
  * (8) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning.  What does bert look at? an analysis of bert’s attention.  arXiv preprint arXiv:1906.04341, 2019.
  * (9) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  arXiv preprint arXiv:2110.14168, 2021.
  * (10) Elias Frantar and Dan Alistarh.  Sparsegpt: Massive language models can be accurately pruned in one-shot.  In International Conference on Machine Learning, pages 10323–10337. PMLR, 2023.
  * (11) Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot.  Specializing smaller language models towards multi-step reasoning.  In International Conference on Machine Learning, pages 10421–10430. PMLR, 2023.
  * (12) Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts.  Causal abstractions of neural networks.  Advances in Neural Information Processing Systems, 34:9574–9586, 2021.
  * (13) Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman.  Finding alignments between interpretable causal variables and distributed neural representations.  In Causal Learning and Reasoning, pages 160–187. PMLR, 2024.
  * (14) Song Han, Jeff Pool, John Tran, and William Dally.  Learning both weights and connections for efficient neural network.  Advances in neural information processing systems, 28, 2015.
  * (15) Babak Hassibi, David G Stork, and Gregory J Wolff.  Optimal brain surgeon and general network pruning.  In IEEE international conference on neural networks, pages 293–299. IEEE, 1993.
  * (16) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  arXiv preprint arXiv:2009.03300, 2020.
  * (17) Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema.  Visualisation and’diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure.  Journal of Artificial Intelligence Research, 61:907–926, 2018.
  * (18) Md Tahmid Rahman Laskar, Xiangji Huang, and Enamul Hoque.  Contextualized embeddings based transformer encoder for sentence similarity modeling in answer selection task.  In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5505–5514, 2020.
  * (19) Yann LeCun, John Denker, and Sara Solla.  Optimal brain damage.  Advances in neural information processing systems, 2, 1989.
  * (20) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li.  On the sentence embeddings from pre-trained language models.  arXiv preprint arXiv:2011.05864, 2020.
  * (21) Xinyin Ma, Gongfan Fang, and Xinchao Wang.  Llm-pruner: On the structural pruning of large language models.  Advances in neural information processing systems, 36:21702–21720, 2023.
  * (22) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli.  A SICK cure for the evaluation of compositional distributional semantic models.  In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 216–223, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA).
  * (23) Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg.  The hydra effect: Emergent self-repair in language model computations.  arXiv preprint arXiv:2307.15771, 2023.
  * (24) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models, 2016.
  * (25) Matthew E Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih.  Dissecting contextual word embeddings: Architecture and representation.  arXiv preprint arXiv:1808.08949, 2018.
  * (26) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of machine learning research, 21(140):1–67, 2020.
  * (27) Michael Santacroce, Zixin Wen, Yelong Shen, and Yuanzhi Li.  What matters in the structured pruning of generative language models?  arXiv preprint arXiv:2302.03773, 2023.
  * (28) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi.  Socialiqa: Commonsense reasoning about social interactions.  arXiv preprint arXiv:1904.09728, 2019.
  * (29) Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje.  Not just a black box: Learning important features through propagating activation differences.  arXiv preprint arXiv:1605.01713, 2016.
  * (30) Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter.  A simple and effective pruning approach for large language models.  arXiv preprint arXiv:2306.11695, 2023.
  * (31) Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In International conference on machine learning, pages 3319–3328. PMLR, 2017.
  * (32) Ian Tenney, Dipanjan Das, and Ellie Pavlick.  Bert rediscovers the classical nlp pipeline.  arXiv preprint arXiv:1905.05950, 2019.
  * (33) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * (34) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * (35) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
  * (36) Mingyang Zhang, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, Bohan Zhuang, et al.  Pruning meets low-rank parameter-efficient fine-tuning.  arXiv preprint arXiv:2305.18403, 2023.
  * (37) Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou.  Moefication: Transformer feed-forward layers are mixtures of experts.  arXiv preprint arXiv:2110.01786, 2021.
  * (38) Wei Zhao, Zhe Li, and Jun Sun.  Causality analysis for evaluating the security of large language models.  arXiv preprint arXiv:2312.07876, 2023.
  * (39) Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang.  A survey on model compression for large language models.  arXiv preprint arXiv:2308.07633, 2023.

##  Appendix A Appendix

###  A.1 Experiments Setting

We use the implementation of Wanda from https://github.com/locuslab/wanda. We
only use unstructed pruning with 50% sparasity.

All models and datasets used in this paper are sourced from huggingface.co.

All experiments in this paper were conducted on a single machine equipped with
eight 50G A40 GPU. All the single experiment can be finished with in half an
hour.

All the model are loaded with torch.float16 except for the validation
experiments 3, which we use float 32 instead.

###  A.2 Statistics for Finding 1 on Boolq

layer 1

layer 2

layer 3

layer 4

layer 5

layer 6

layer 7

layer 8

layer 9

layer 10

layer 11

layer 12

layer 13

layer 14

layer 15

layer 16

layer 17

layer 18

layer 19

layer 20

layer 21

layer 22

layer 23

layer 24

layer 25

layer 26

layer 27

layer 28

layer 29

layer 30

layer 31

layer 32

Figure 5: Activated Parameter Statistics. The x-axis represents the value of
$A(w_{i})$, and the y-axis represents the quantity. For convenience of
statistics, we have performed statistical processing, so the numerical value
on the y-axis is an estimate of one-thousandth of the actual value.

###  A.3 Statistics for Finding 1 on HumanEval

layer 1

layer 2

layer 3

layer 4

layer 5

layer 6

layer 7

layer 8

layer 9

layer 10

layer 11

layer 12

layer 13

layer 14

layer 15

layer 16

layer 17

layer 18

layer 19

layer 20

layer 21

layer 22

layer 23

layer 24

layer 25

layer 26

layer 27

layer 28

layer 29

layer 30

layer 31

layer 32

Figure 6: Activated Parameter Statistics. The x-axis represents the value of
$A(w_{i})$, and the y-axis represents the quantity. For convenience of
statistics, we have performed statistical processing, so the numerical value
on the y-axis is an estimate of one-thousandth of the actual value.

###  A.4 Statistics for Finding 1 on MMLU

layer 1

layer 2

layer 3

layer 4

layer 5

layer 6

layer 7

layer 8

layer 9

layer 10

layer 11

layer 12

layer 13

layer 14

layer 15

layer 16

layer 17

layer 18

layer 19

layer 20

layer 21

layer 22

layer 23

layer 24

layer 25

layer 26

layer 27

layer 28

layer 29

layer 30

layer 31

layer 32

Figure 7: Activated Parameter Statistics. The x-axis represents the value of
$A(w_{i})$, and the y-axis represents the quantity. For convenience of
statistics, we have performed statistical processing, so the numerical value
on the y-axis is an estimate of one-thousandth of the actual value.

###  A.5 Statistics for Finding 2

humaneval-hellaswag

wikitext2-humaneval

piqa-piqa

c4-hellaswag

piqa-hellaswag

wikitext2-boolq

humaneval-gsm8k

siqa-boolq

boolq-mmlu

boolq-hellaswag

siqa-mmlu

piqa-mmlu

gsm8k-gsm8k

c4-gsm8k

humaneval-mmlu

gsm8k-mmlu

wikitext2-wikitext2

c4-c4

wikitext2-siqa

c4-mmlu

wikitext2-c4

piqa-gsm8k

hellaswag-hellaswag

c4-siqa

wikitext2-mmlu

siqa-gsm8k

siqa-humaneval

wikitext2-gsm8k

boolq-humaneval

piqa-siqa

wikitext2-hellaswag

siqa-siqa

gsm8k-hellaswag

humaneval-humaneval

mmlu-mmlu

wikitext2-piqa

siqa-hellaswag

c4-boolq

piqa-humaneval

boolq-gsm8k

boolq-boolq

c4-humaneval

hellaswag-mmlu

piqa-boolq

c4-piqa

Figure 8: Statistics of LLMDcos values for different layers. The x-axis
represents the layer number, and the y-axis represents the LLMDcos values of
64 samples. The two data sources are indicated below the image. All images are
results from Llama2-7b.

## NeurIPS Paper Checklist

  1. 1.

Claims

  2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

  3. Answer: [Yes]

  4. Justification: contributions and scope are included in Section 1.

  5. Guidelines:

     * •

The answer NA means that the abstract and introduction do not include the
claims made in the paper.

     * •

The abstract and/or introduction should clearly state the claims made,
including the contributions made in the paper and important assumptions and
limitations. A No or NA answer to this question will not be perceived well by
the reviewers.

     * •

The claims made should match theoretical and experimental results, and reflect
how much the results can be expected to generalize to other settings.

     * •

It is fine to include aspirational goals as motivation as long as it is clear
that these goals are not attained by the paper.

  6. 2.

Limitations

  7. Question: Does the paper discuss the limitations of the work performed by the authors?

  8. Answer: [Yes]

  9. Justification: Limitations are discussed in Section 6

  10. Guidelines:

     * •

The answer NA means that the paper has no limitation while the answer No means
that the paper has limitations, but those are not discussed in the paper.

     * •

The authors are encouraged to create a separate "Limitations" section in their
paper.

     * •

The paper should point out any strong assumptions and how robust the results
are to violations of these assumptions (e.g., independence assumptions,
noiseless settings, model well-specification, asymptotic approximations only
holding locally). The authors should reflect on how these assumptions might be
violated in practice and what the implications would be.

     * •

The authors should reflect on the scope of the claims made, e.g., if the
approach was only tested on a few datasets or with a few runs. In general,
empirical results often depend on implicit assumptions, which should be
articulated.

     * •

The authors should reflect on the factors that influence the performance of
the approach. For example, a facial recognition algorithm may perform poorly
when image resolution is low or images are taken in low lighting. Or a speech-
to-text system might not be used reliably to provide closed captions for
online lectures because it fails to handle technical jargon.

     * •

The authors should discuss the computational efficiency of the proposed
algorithms and how they scale with dataset size.

     * •

If applicable, the authors should discuss possible limitations of their
approach to address problems of privacy and fairness.

     * •

While the authors might fear that complete honesty about limitations might be
used by reviewers as grounds for rejection, a worse outcome might be that
reviewers discover limitations that aren’t acknowledged in the paper. The
authors should use their best judgment and recognize that individual actions
in favor of transparency play an important role in developing norms that
preserve the integrity of the community. Reviewers will be specifically
instructed to not penalize honesty concerning limitations.

  11. 3.

Theory Assumptions and Proofs

  12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

  13. Answer: [N/A]

  14. Justification: There are no theoretical results.

  15. Guidelines:

     * •

The answer NA means that the paper does not include theoretical results.

     * •

All the theorems, formulas, and proofs in the paper should be numbered and
cross-referenced.

     * •

All assumptions should be clearly stated or referenced in the statement of any
theorems.

     * •

The proofs can either appear in the main paper or the supplemental material,
but if they appear in the supplemental material, the authors are encouraged to
provide a short proof sketch to provide intuition.

     * •

Inversely, any informal proof provided in the core of the paper should be
complemented by formal proofs provided in appendix or supplemental material.

     * •

Theorems and Lemmas that the proof relies upon should be properly referenced.

  16. 4.

Experimental Result Reproducibility

  17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

  18. Answer: [Yes]

  19. Justification: We will provide code in supplementary material and release it after review.

  20. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

If the paper includes experiments, a No answer to this question will not be
perceived well by the reviewers: Making the paper reproducible is important,
regardless of whether the code and data are provided or not.

     * •

If the contribution is a dataset and/or model, the authors should describe the
steps taken to make their results reproducible or verifiable.

     * •

Depending on the contribution, reproducibility can be accomplished in various
ways. For example, if the contribution is a novel architecture, describing the
architecture fully might suffice, or if the contribution is a specific model
and empirical evaluation, it may be necessary to either make it possible for
others to replicate the model with the same dataset, or provide access to the
model. In general. releasing code and data is often one good way to accomplish
this, but reproducibility can also be provided via detailed instructions for
how to replicate the results, access to a hosted model (e.g., in the case of a
large language model), releasing of a model checkpoint, or other means that
are appropriate to the research performed.

     * •

While NeurIPS does not require releasing code, the conference does require all
submissions to provide some reasonable avenue for reproducibility, which may
depend on the nature of the contribution. For example

       1. (a)

If the contribution is primarily a new algorithm, the paper should make it
clear how to reproduce that algorithm.

       2. (b)

If the contribution is primarily a new model architecture, the paper should
describe the architecture clearly and fully.

       3. (c)

If the contribution is a new model (e.g., a large language model), then there
should either be a way to access this model for reproducing the results or a
way to reproduce the model (e.g., with an open-source dataset or instructions
for how to construct the dataset).

       4. (d)

We recognize that reproducibility may be tricky in some cases, in which case
authors are welcome to describe the particular way they provide for
reproducibility. In the case of closed-source models, it may be that access to
the model is limited in some way (e.g., to registered users), but it should be
possible for other researchers to have some path to reproducing or verifying
the results.

  21. 5.

Open access to data and code

  22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

  23. Answer: [Yes]

  24. Justification: All the data are from public datasets and code will be provided.

  25. Guidelines:

     * •

The answer NA means that paper does not include experiments requiring code.

     * •

Please see the NeurIPS code and data submission guidelines
(https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

     * •

While we encourage the release of code and data, we understand that this might
not be possible, so “No” is an acceptable answer. Papers cannot be rejected
simply for not including code, unless this is central to the contribution
(e.g., for a new open-source benchmark).

     * •

The instructions should contain the exact command and environment needed to
run to reproduce the results. See the NeurIPS code and data submission
guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more
details.

     * •

The authors should provide instructions on data access and preparation,
including how to access the raw data, preprocessed data, intermediate data,
and generated data, etc.

     * •

The authors should provide scripts to reproduce all experimental results for
the new proposed method and baselines. If only a subset of experiments are
reproducible, they should state which ones are omitted from the script and
why.

     * •

At submission time, to preserve anonymity, the authors should release
anonymized versions (if applicable).

     * •

Providing as much information as possible in supplemental material (appended
to the paper) is recommended, but including URLs to data and code is
permitted.

  26. 6.

Experimental Setting/Details

  27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

  28. Answer: [Yes]

  29. Justification: in the appendix

  30. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The experimental setting should be presented in the core of the paper to a
level of detail that is necessary to appreciate the results and make sense of
them.

     * •

The full details can be provided either with the code, in appendix, or as
supplemental material.

  31. 7.

Experiment Statistical Significance

  32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

  33. Answer: [Yes]

  34. Justification: shown in statistics figures.

  35. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The authors should answer "Yes" if the results are accompanied by error bars,
confidence intervals, or statistical significance tests, at least for the
experiments that support the main claims of the paper.

     * •

The factors of variability that the error bars are capturing should be clearly
stated (for example, train/test split, initialization, random drawing of some
parameter, or overall run with given experimental conditions).

     * •

The method for calculating the error bars should be explained (closed form
formula, call to a library function, bootstrap, etc.)

     * •

The assumptions made should be given (e.g., Normally distributed errors).

     * •

It should be clear whether the error bar is the standard deviation or the
standard error of the mean.

     * •

It is OK to report 1-sigma error bars, but one should state it. The authors
should preferably report a 2-sigma error bar than state that they have a 96%
CI, if the hypothesis of Normality of errors is not verified.

     * •

For asymmetric distributions, the authors should be careful not to show in
tables or figures symmetric error bars that would yield results that are out
of range (e.g. negative error rates).

     * •

If error bars are reported in tables or plots, The authors should explain in
the text how they were calculated and reference the corresponding figures or
tables in the text.

  36. 8.

Experiments Compute Resources

  37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  38. Answer: [Yes]

  39. Justification: in the appendix

  40. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The paper should indicate the type of compute workers CPU or GPU, internal
cluster, or cloud provider, including relevant memory and storage.

     * •

The paper should provide the amount of compute required for each of the
individual experimental runs as well as estimate the total compute.

     * •

The paper should disclose whether the full research project required more
compute than the experiments reported in the paper (e.g., preliminary or
failed experiments that didn’t make it into the paper).

  41. 9.

Code Of Ethics

  42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  43. Answer: [Yes]

  44. Justification: [N/A]

  45. Guidelines:

     * •

The answer NA means that the authors have not reviewed the NeurIPS Code of
Ethics.

     * •

If the authors answer No, they should explain the special circumstances that
require a deviation from the Code of Ethics.

     * •

The authors should make sure to preserve anonymity (e.g., if there is a
special consideration due to laws or regulations in their jurisdiction).

  46. 10.

Broader Impacts

  47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  48. Answer: [N/A]

  49. Justification: This work only try to explain the model and verify the explaination.

  50. Guidelines:

     * •

The answer NA means that there is no societal impact of the work performed.

     * •

If the authors answer NA or No, they should explain why their work has no
societal impact or why the paper does not address societal impact.

     * •

Examples of negative societal impacts include potential malicious or
unintended uses (e.g., disinformation, generating fake profiles,
surveillance), fairness considerations (e.g., deployment of technologies that
could make decisions that unfairly impact specific groups), privacy
considerations, and security considerations.

     * •

The conference expects that many papers will be foundational research and not
tied to particular applications, let alone deployments. However, if there is a
direct path to any negative applications, the authors should point it out. For
example, it is legitimate to point out that an improvement in the quality of
generative models could be used to generate deepfakes for disinformation. On
the other hand, it is not needed to point out that a generic algorithm for
optimizing neural networks could enable people to train models that generate
Deepfakes faster.

     * •

The authors should consider possible harms that could arise when the
technology is being used as intended and functioning correctly, harms that
could arise when the technology is being used as intended but gives incorrect
results, and harms following from (intentional or unintentional) misuse of the
technology.

     * •

If there are negative societal impacts, the authors could also discuss
possible mitigation strategies (e.g., gated release of models, providing
defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms
to monitor how a system learns from feedback over time, improving the
efficiency and accessibility of ML).

  51. 11.

Safeguards

  52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  53. Answer: [N/A]

  54. Justification: This work try to explain the model and do not release any new model or datasets.

  55. Guidelines:

     * •

The answer NA means that the paper poses no such risks.

     * •

Released models that have a high risk for misuse or dual-use should be
released with necessary safeguards to allow for controlled use of the model,
for example by requiring that users adhere to usage guidelines or restrictions
to access the model or implementing safety filters.

     * •

Datasets that have been scraped from the Internet could pose safety risks. The
authors should describe how they avoided releasing unsafe images.

     * •

We recognize that providing effective safeguards is challenging, and many
papers do not require this, but we encourage authors to take this into account
and make a best faith effort.

  56. 12.

Licenses for existing assets

  57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  58. Answer: [Yes]

  59. Justification: All the codes, models and datasets are cited

  60. Guidelines:

     * •

The answer NA means that the paper does not use existing assets.

     * •

The authors should cite the original paper that produced the code package or
dataset.

     * •

The authors should state which version of the asset is used and, if possible,
include a URL.

     * •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

     * •

For scraped data from a particular source (e.g., website), the copyright and
terms of service of that source should be provided.

     * •

If assets are released, the license, copyright information, and terms of use
in the package should be provided. For popular datasets,
paperswithcode.com/datasets has curated licenses for some datasets. Their
licensing guide can help determine the license of a dataset.

     * •

For existing datasets that are re-packaged, both the original license and the
license of the derived asset (if it has changed) should be provided.

     * •

If this information is not available online, the authors are encouraged to
reach out to the asset’s creators.

  61. 13.

New Assets

  62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

  63. Answer: [N/A]

  64. Justification: All the models/codes/datasets are public.

  65. Guidelines:

     * •

The answer NA means that the paper does not release new assets.

     * •

Researchers should communicate the details of the dataset/code/model as part
of their submissions via structured templates. This includes details about
training, license, limitations, etc.

     * •

The paper should discuss whether and how consent was obtained from people
whose asset is used.

     * •

At submission time, remember to anonymize your assets (if applicable). You can
either create an anonymized URL or include an anonymized zip file.

  66. 14.

Crowdsourcing and Research with Human Subjects

  67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  68. Answer: [N/A]

  69. Justification: [N/A]

  70. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Including this information in the supplemental material is fine, but if the
main contribution of the paper involves human subjects, then as much detail as
possible should be included in the main paper.

     * •

According to the NeurIPS Code of Ethics, workers involved in data collection,
curation, or other labor should be paid at least the minimum wage in the
country of the data collector.

  71. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  73. Answer: [N/A]

  74. Justification: [N/A]

  75. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Depending on the country in which research is conducted, IRB approval (or
equivalent) may be required for any human subjects research. If you obtained
IRB approval, you should clearly state this in the paper.

     * •

We recognize that the procedures for this may vary significantly between
institutions and locations, and we expect authors to adhere to the NeurIPS
Code of Ethics and the guidelines for their institution.

     * •

For initial submissions, do not include any information that would break
anonymity (if applicable), such as the institution conducting the review.