# JUSTICE: A Benchmark Dataset for Supreme Court’s Judgment Prediction

Mohammed Alsayed, Shaayan Syed, Mohammad Alali, Smit Patel, Hemanth Bodala
University of Southern California
{alsayedm, shaayans, alalim, smitp<EMAIL_ADDRESS>
##  1 Introduction

Over 7,000 Supreme Court cases are requested to be reviewed each year in the
United States; however, only 100 cases on average are heard. [4] Most of this
data is raw, and while there are existing datasets comprised of this kind of
information, they were of other countries’ court cases such as the Chinese
Supreme Court and the European Human Rights court. [5, 1] There was no
existing dataset that was created to perform the task we have outlined in this
paper on the Supreme Court of the United States.

Our goal is to compile a well-annotated dataset that can be readily used in
natural language processing (NLP) and other data-driven applications. While
court cases have an abundance of information detailing them, the ones that
concern us the most include the absolute neutral facts of the case and the
details of the decisions made by the presiding justices, such as the voting
counts and the winning party.

Furthermore, we train a model to predict a court’s judgment based on the facts
provided by both the petitioner and respondent. In other words, the model is
simulating a human jury by generating a final verdict. Recently, advancements
in NLP and machine learning provide us with the tools to build predictive
models that can be used to reveal patterns that influence Supreme Court case
decisions. This modeling problem has the potential to be solved using the
dataset we have created.

Datasets similar to this already exist. However, as pointed out earlier, they
are directed towards a specific category of cases, such as those involving
antitrust laws, rather than the entire spectrum we wish to analyze. One
related research paper that is an example of this is titled “CAIL2018: A
Large-Scale Legal Dataset for Judgement Prediction” by [5]. [5] Their goal of
predicting judgement results is comparable to ours, but the big difference is
that their 2.6 million cases dataset only consists of criminal cases. [5] In
addition, these cases were published by the Supreme People’s Court of China
which are vastly greater in number than the cases in our dataset which pertain
to the Supreme Court of the United States.

Given that there have been a significant number of Supreme Court cases filed
over the years, we looked to the databases hosted by the Caselaw Access
Project and Oyez as our source of raw information. Due to the challenges
involving missing data and a scarcity of relevant entries, we also had to
locate other sources to patch these holes; the methodology for this will be
detailed further on. Following that, we trained a few models to gauge the
dataset’s versatility and quality. By analyzing the results of this study, we
gained a better understanding of how the dataset could be modified to improve
and train more accurate models in the future.

Column |  Description
---|---
ID |  Unique case identifier.
Name |  The name of the case.
HREF |  The Oyez’s API URL for the case.
Docket ID |  A special identifier of the case used by the legal system.
Term |  The year when the Court received the case.
First Party |  The name of the first party (petitioner).
Second Party |  The name of the second party (respondent).
Facts |  The absolute, neutral facts of the case written by the court clerk.
Majority Vote |  The number of justices voting for the majority opinion.
Minority Vote |  The number of justices voting for the minority opinion.
Winning Party |  The name of the party that won the case.
First Party Winner |  True if the first party won the case, otherwise False and the second party won the case.
Decision Type |  The type of the decision decided by the court, e.g.: per curiam, equally divided, opinion of the court.
Disposition |  The treatment the Supreme Court accorded the court whose decision it reviewed; e.g.: affirmed, reversed, vacated.
Issue Area |  The pre-defined legal issue category of the case; e.g.: Civil Rights, Criminal Procedure, Federal Taxation.
Table 1:  Dataset columns definition.

##  2 Methods

###  2.1 Dataset Preparation

A large portion of our time was focused on researching which source of Supreme
Court cases would provide the most appropriate data for our purposes. Given
our goal, we needed to find one that provided a large enough text summary of
the facts of each case, as well as the final decision made by the judges.
Additional information regarding the case decision was welcome. Our top two
candidates were the Caselaw Access Project (CAP) and Oyez. While both sources
contain the facts of the case and the outcome, the facts from CAP were usually
short and contained little-to-no information to base a decision on. Hence, due
to the fact that Oyez had richer facts and a simple API to retrieve cases, we
decided to use it as our main source of data.

Unfortunately, a major downside of Oyez was that only a total of 8,209 cases
were available, which was reasonable considering that only 100 cases are heard
annually by the Supreme Court. [4] Nevertheless, we ran into further issues
with this database which reduced this number even further.

####  2.1.1 Dataset Definition

The most important part of a dataset is the way it’s structured and the data
it contains. Hence, after much deliberation, we landed into the dataset with
the following columns as listed in Table 1 after filtering the data from Oyez.
Additionally, we kept the Oyez URL that the case was obtained from for future
reference.

####  2.1.2 Data Complications

As mentioned earlier, the biggest problem we encountered was the fact that a
significant portion of the cases we collected had either missing or incorrect
information. The following are examples of the issues found while analyzing
the dataset. Firstly, there were some ambiguous winning party names that could
not be identified whether it’s a substring of the first or second party as
shown in Table 2.

Case Name | Party1 | Party2 | Winner
---|---|---|---
Boggs v. Boggs | Boggs | Boggs | Boggs
Table 2:  Case with ambiguous winning party.

Secondly, the dataset had swapped values between the majority and minority
votes for some instances since it didn’t make sense to have a larger minority
vote than a majority vote. An example of a swapped vote is illustrated below
in Table 3.

Case Name | Majority Vote | Minority Vote
---|---|---
Taylor v. Barkes | 0 | 9
Table 3:  Case with incorrectly swapped votes.

Thirdly, some of the data had an unmatching decision type with its voting
values. An example of a case where its votes do not reflect the decision type
is shown in Table 4.

Name | Maj. Vote | Min. Vote | Decision
---|---|---|---
Sears vs. LA | 0 | 8 | equally divided
Table 4:  Case with incorrect decision type.

Finally, some of the cases had a total of zero votes. All of these issues have
been solved by comparing them with the Washington University Law database
http://scdb.wustl.edu/ which is a more robust database of the supreme court
that contains solely the decisions. However, some cases were not found so they
have been kept as is or dropped if they lacked information, and the ones
matched have been updated with the correct values.

###  2.2 Dataset Preprocessing

####  2.2.1 Data Cleaning

Once the dataset has been populated, the next step was to enrich the data by
removing unnecessary information to ensure a high-quality dataset. Primarily,
the data cleaning process was focused on the textual aspect of the dataset,
which included the party names and the facts. First, all HTML tags and URLs
were removed from the text as it served no significance to the models
utilizing the dataset. Second, the excess spaces and all non-UTF8 characters
were removed. Third, the contractions in the dataset were expanded to their
normal form; for example, the word “can’t” was converted into “cannot”.
Finally, the stopwords in text, such as “the”, were discarded as they added
hardly any information. This process not only improved the quality of text in
the dataset by discarding irrelevant text, it also shrank the size of the
average facts by 20% which could facilitate faster learning and processing.

####  2.2.2 Data Augmentation

Even though we managed to preserve most of our dataset, we still had a limited
number of cases to work with. Furthermore, we observed that in a majority of
our entries, the judges favored the second party in only one out of eight
cases, while the rest of our entries showed a majority of the judges favoring
the first party; i.e. a 88-12 class imbalance in the dataset. To combat this
problem, we have augmented the cases where the second party won by generating
similar cases. Data augmentation was performed using the nlpaug library, which
provides textual augmentation for machine learning applications. There are
three levels of data augmentation, i.e. characters, words, and sentences. In
this work, we performed data augmentation on words, by replacing some of the
words in the text with their synonyms or similar words.

The first approach was back-translation, where a phrase is translated into
another language and translated back to English. This approach replaced very
few words in the sentence, and we found that the augmented sentences were
nearly the same as the original ones. In the second technique, we replaced the
words with their synonyms, but it still did not allow us to create a new
sentence that maintained the meaning but was different syntactily. In the last
two techniques, we used word embeddings to find the words which are similar to
the word and replaced it; out of these two techniques, we adapted the
contextual word embeddings technique which was the best one among the ones we
tried. This method not only replaces the word based on similarity but also
substitutes the best-fit words based on the contextual word embeddings; using
this technique, we successfully replaced 50% of the words in the facts without
affecting the nouns. Using the bert-base-cased pre-trained model, we replaced
the words with their four most similar words from the embeddings. This model
has the advantage of persisting the case sensitivity of our data and the
essence of the case while paraphrasing the facts into something novel. This
process was repeated until the dataset had a 60-40 class balance.

####  2.2.3 Data Mirroring

While the Data Augmentation step lessened the dataset imbalance issue, we
complemented it with another procedure to mitigate the imbalanced data even
further. We noticed that the two classes in the dataset were interchangeable;
class 1 is when the first party wins and class 2 is when the second party
wins. Consequently, being the first party or the second party has no bearing
on the outcome of the case. Thus, we mirrored each case in the dataset, where
the first and second party names were swapped. Not only did this double our
dataset’s size and end the class imbalance by having a 50-50 class balance, it
also emphasized that the outcome of a case is invariant to the position of the
parties.

##  3 Experiments

While our main priority was for the dataset to be used to predict the final
decision for a Supreme Court case, we prepared several models and data to also
solve another task as well in order to demonstrate its versatility. So along
with the decision prediction task, we also used the dataset to train a model
that predicts how controversial each court case would be.

###  3.1 Decision Prediction Task

The primary task to evaluate the dataset was to predict the judicial judgment
given the absolute neutral facts of the case written by the court clerks. The
judgment was broken into a binary classification; the winner of the case was
mapped to 0 when it was the first party, or 1 otherwise. To accomplish this
task, instead of having a single model predict the outcome, we opted for an
ensemble model which encompasses multiple diverse models that each vote for an
outcome and then the majority vote wins; exactly the same way the Supreme
Court works with 9 justices. However, due to time limitations, we trained only
7 models to be a part of the ensemble model rather than the thematic 9. The
models used are perceptron, support vector machines (SVM), logistic
regression, naive bayes, multi-layer perceptron (MLP), k-nearest neighbors
(KNN), and calibrated classifier. The scikit-learn package was utilized to
train the models, complemented by the usage of GridSearchCV to obtain decent
hyperparameters, which are listed in Table 5.

Model | Hyperparameters
---|---
SVM | max_iter=5
Logistic Regression | default
Naive Bayes | alpha=3
Calibrated Classifier | default
Perceptron |  max_iter=5 penalty=’l1’
KNN |  n_neighbors=3 weights=’distance’
MLP |  early_stopping=True beta_2=0 max_iter=10
Table 5:  Hyperparameters for the models used in the paper.

As the models do not explicitly take in raw textual format, we had to
preprocess all textual input and vectorize it. We used the TF-IDF vectorizer
to transform the document into an array of numbers which is more suitable for
the model’s input. TF-IDF is a statistical measurement used to weigh words by
their importance to a document; however, it does not take into account the
context nor its position in the document. [2] Next, the models were fed the
vectorized first party name, second party name, and the facts as input. The
party names were a part of the input so that the model can distinguish between
the entities mentioned in the facts of the case; furthermore, since there is a
mirrored case where the party names were swapped around, the model should
learn that the party name’s position has no bearing on the outcome. Moreover,
other models utilizing LSTMs were attempted in our experiments; however, due
to the size of the facts for each case, which on average is 189 words after
preprocessing, it became difficult to correctly predict and train due to the
vanishing and/or exploding gradients. [3]

###  3.2 Controversialness Prediction Task

Another task we liked to accomplish with the constructed dataset was to
determine, based on the facts presented, if a particular case would be
controversial among the judges. In this context, we determine how
controversial a case is by how close the final decisions by the Supreme Court
Justices are to an even split. We determine whether a case is controversial
based on the following expression: $1-\frac{majority\ votes}{total\ votes}$,
where the class 1 represents cases with a controversy score of 0.3 or greater.
The threshold value 0.3 was chosen to get a roughly even split of
controversial to non-controversial cases.

Once every case is assigned a score representing how controversial it is, the
next step is to generate word embeddings for the entire corpus of facts in
each case. We construct a Word2Vec model based on the facts of all cases we
have available, and do so with minimal preprocessing to ensure a model that is
as large as possible. We opted to construct a completely novel Word2Vec model
rather than use a premade model using data from search engines or other text
sources because we believed that having a model constructed out of legal text
would capture lexical similarity better in this context specifically. The
model was constructed with a vector size of 100 items and a window size of 11.
Once the Word2Vec embeddings were constructed, we have both an input and
labels and can now begin training models to classify how controversial a case
may be.

For this instance, we constructed both an SVM classifier as well as an LSTM to
determine the performance of the dataset using different classifiers. To train
the SVM, we calculated the average Word2Vec vector for the facts of every case
and used that as input for the classifier, with the labels calculated earlier
as our targets. To train the LSTM, we passed an array of vectors for each word
of the facts of a case. Every array of vectors was padded to match the length
of the longest set of facts that existed in our dataset.

##  4 Results

###  4.1 Decision Prediction Task

Before data augmentation and case mirroring, we had 2384 cases of which are
2114 class 1 (first party won) and 270 class 2 (second party won). Data
augmentation was done only on class 2 which increased the dataset size to 3464
cases of which are 2114 class 1 and 1350 class 2. Next, each case was mirrored
which doubled the number of cases to 6928 while having 3464 of each case; a
50-50 perfectly balanced dataset. This case mirroring ensured us that the
position of the party is invariant to the outcome. Hence, while these data
augmentation methods synthetically alleviated the class imbalance issue, it is
still preferable to obtain a dataset that is naturally balanced; however, that
is rarely the case. Table 6 below depicts the result of the models against a
testing dataset, which was a 20% split of the entire dataset. The models were
evaluated using three metrics: accuracy (Acc%), macro precision (MP%), macro
recall (MR%), and macro F1 score (MF1). Collectively, these metrics should
give us an insight of the performance of the models.

Model | Acc% | MP% | MR% | MF1
---|---|---|---|---
Perceptron | 65% | 65% | 65% | 0.65
SVM | 60% | 60% | 60% | 0.60
Logistic Reg | 61% | 61% | 61% | 0.61
Naive Bayes | 59% | 59% | 59% | 0.59
MLP | 64% | 64% | 64% | 0.64
KNN | 68% | 69% | 68% | 0.67
Calib. Classifier | 62% | 63% | 62% | 0.62
Ensemble | 62% | 62% | 62% | 0.62
Table 6:  Evaluation metric for each trained model.

As shown in Table 6, the ensemble achieved an average accuracy of 62%. It is
decent enough to show that the model is not randomly choosing between the
classifications with a 50% chance for each. However, we were unable to achieve
an accuracy higher than this throughout the variations of models that we
experimented with. This shows us that the problem most likely lies in the
dataset. The data augmentation methods we used earlier improved the problems
of the limited entries, as well as the imbalance towards one type of
classification, but it did not completely solve the dilemma. We hypothesize
that this could be accounted for by incorporating other sources of Supreme
Court cases in order to fill out this dataset with a more balanced ratio, thus
giving us a higher accuracy if used to train these models again.

The lack of data and balanced outcomes played a part in the inability to
provide compelling results for the Controversial task. Although, an additional
problem also lies in the content of the input itself. The summarized facts of
the court cases occasionally provided a comprehensive look at the case, but a
significant amount of them contained almost no information at all. The average
character length was 1,197, with the minimum being 95 and the maximum being
6108. For word length, the average measurement was 189 words. Given this
knowledge, and the fact that the minimum number of words was 13, we can see
that the summarized facts have very little content.

###  4.2 Controversialness Prediction Task

Unfortunately, we were unable to produce meaningful results from either the
LSTM or the SVM in completing the task to determine if a case is controversial
or not. The SVM classifier was able to produce an accuracy of 61%, however,
upon further inspection, this is because the model will guess that every case
is non-controversial. And since the non-controversial cases make up
approximately 60% of the dataset, this accuracy value is meaningless. As for
the LSTM, we were unable to train the model in a way that produced quality
results. After training the model with all of the facts for several hundred
epochs with various hyper-parameter configurations, we were still unable to
improve the model’s cross entropy loss past 0.6.

We believe that these results came about due to a flaw in the task itself. In
other text-based binary labelling tasks such as sentiment analysis, there are
clear targets for a model to pick up in order to make an accurate prediction.
In the case of sentiment analysis these can be words such as ‘outstanding’ or
‘terrible’. These features are all present within the input text to begin so
there is no external knowledge needed. However, for legal cases, the facts of
the case itself may not immediately give hints as to how controversial a case
may be. Typically, a case is controversial due to prior cases and laws that
set a precedence favoring one party over another. This means to get an
accurate picture of how controversial a case is, we must train it with
extensive knowledge about prior legal cases and laws that could impact the
case at hand. This is no fault of the dataset in itself, but rather a fault of
a task that could never be accomplished with this dataset alone.

##  5 Conclusions

Based on the results produced by the models described above, we have concluded
that there are still improvements to be made to this dataset. First and
foremost is to increase the number of entries that are included in the data.
This can be done by searching through other sources for the Supreme Court
cases in order to account for incorrect or incomplete data. Another important
priority is to make sure the dataset is not heavily skewed towards one
specific outcome. Even though the data imbalance issue is lessened by the data
augmentation and data mirroring procedures mentioned above, it is still
preferable to have novel data that is naturally balanced. Again, this can be
accounted for by increasing the number of entries.

The versatility of this database should also be considered when updating it,
which is why one improvement could also be to split the facts of the case
based on whether it corresponds to the first or second party. This type of
format could be useful for a number of varying models, one example being to
predict which party was the petitioner and which one was the respondent.
Similar adjustments to the dataset like this one would no doubt broaden its
usage.

An assumption we made during the data preparation stage was that the size of
the summarized facts would be sufficient for our models to accurately predict
the outcomes. Unfortunately, it is difficult to increase the length without
using artificial augmentation methods. One solution we can use to improve the
content itself, however, is to manually annotate the text, similar to named-
entity recognition (NER) tagging. For example, we would replace every
reference to the first party in the text with <p1>, and the second party with
<p2>. This has the potential to work especially well considering that we
primarily used BERT to obtain the word embeddings, which bases the embedding
on the context in which the word is used. Furthermore, removing the party
names from the learning ensures that the model is learning to deduce the
correct judgement based solely on the facts of the case and less about who is
in the case.

Given all these adjustments, the next step would be to test this updated
dataset on the same models used in this report so that a clear improvement, or
decline, can be ascertained.

## References

  * [1]  Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preoţiuc-Pietro and Vasileios Lampos  “Predicting judicial decisions of the European Court of Human Rights: a Natural Language Processing perspective”  In _PeerJ Comput. Sci._ 2.e93 PeerJ, 2016, pp. e93
  * [2]  Juan Ramos  “Using TF-IDF to determine word relevance in document queries”, 2003
  * [3]  Alex Sherstinsky  “Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network”  In _Physica D: Nonlinear Phenomena_ 404 Elsevier BV, 2020, pp. 132306  DOI: 10.1016/j.physd.2019.132306
  * [4]  Supreme Court United States  “Granted & noted list: October term 2020 cases for argument”, 2020
  * [5]  Chaojun Xiao et al.  “CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction”, 2018  arXiv:1807.02478 [cs.CL]


# MCSD: An Efficient Language Model with Diverse Fusion

Hua Yang
<EMAIL_ADDRESS>
&Duohai Li
<EMAIL_ADDRESS>
&Shiman Li
<EMAIL_ADDRESS>

###### Abstract

Transformers excel in Natural Language Processing (NLP) due to their prowess
in capturing long-term dependencies but suffer from exponential resource
consumption with increasing sequence lengths. To address these challenges, we
propose MCSD model, an efficient language model with linear scaling and fast
inference speed. MCSD model leverages diverse feature fusion, primarily
through the multi-channel slope and decay (MCSD) block, to robustly represent
features. This block comprises slope and decay sections that extract features
across diverse temporal receptive fields, facilitating capture of both local
and global information. In addition, MCSD block conducts element-wise fusion
of diverse features to further enhance the delicate feature extraction
capability. For inference, we formulate the inference process into a recurrent
representation, slashing space complexity to $O(1)$ and time complexity to
$O(N)$ respectively. Our experiments show that MCSD attains higher throughput
and lower GPU memory consumption compared to Transformers, while maintaining
comparable performance to larger-scale language learning models on benchmark
tests. These attributes position MCSD as a promising base for edge deployment
and embodied intelligence.

MCSD: An Efficient Language Model with Diverse Fusion

Hua Yang<EMAIL_ADDRESS>Duohai Li<EMAIL_ADDRESS>Shiman Li
<EMAIL_ADDRESS>

##  1 Introduction

Recent years have witnessed significant strides in Natural Language Processing
(NLP), notably the emergence of Large Language Models (LLMs) Gesmundo and
Maile (2023), transforming machine-human interaction by mimicking human-like
language comprehension and generation. Among them, Transformer dominates in
NLP due to its powerful performance Vaswani et al. (2017). Benefiting from its
self-attention mechanism, Transformer has long-range dependencies that
substantially improve the ability to process language. Transformer-based LLMs
trained on extensive datasets sourced from the web have achieved remarkable
success Brown et al. (2020); OpenAI (2023); Team et al. (2024). However, it
suffers from the disadvantage of high computational resource consumption,
accompanied by $O(N^{2})$ computational complexity and $O(N)$ space complexity
at inference Shazeer (2019). The computational requirements of the model scale
quadratically with the length of the input sequence $N$ during inference. This
limits its application to some scenarios, such as time-sensitive or resource-
limited environments, typical of edge devices.

Many attempts have been made to address the above drawbacks. Some methods
simplify the query mechanism by replacing multi-head attention with multi-
query attention Shazeer (2019) and group query attention Ainslie et al.
(2023). The memory requirement is reduced by sharing the keys $K$ and values
$V$ among all heads or grouped heads, providing a flexible trade-off between
computational efficiency and model representation capability. Other approaches
focus on improving the computational efficiency of attention, such as AFT Zhai
et al. (2021) and RWKV Peng et al. (2023). They obviate computing and storing
expensive attention matrices by optimizing the matrix multiplication,
resulting in linear computational complexity. However, these methods still use
a QKV-based querying mechanism as multi-head attention does, which requires
global interactions between queries and values. Recently, alternatives like
Mamba Gu and Dao (2024), rooted in state-space model (SSM) evolution, have
gained traction in the reseach community. Mamba extends input information into
a higher-dimensional hidden space, demonstrating efficiency and lightness.
Yet, empirical evidences suggest scaling challenges for Mamba De et al.
(2024), indicating ongoing hurdles in its widespread adoption.

To this paper, we propose MCSD, an efficient language model that achieves a
trade-off between performance and computational efficiency through diverse
fusion. Specifically, our method facilitates both local and global feature
fusion through the innovative MCSD block. In the MCSD block, it contains a
slope section and a decay section, each adept at integrating long-range and
short-range information, respectively. Subsequently, combining the outputs of
two sections can empower the network with rich feature extraction
capabilities. We use multi-channel slope and multi-channel decay to integrate
historical information across varying temporal receptive fields, and leverage
diverse perturbations for fine-grained element-wise integration. In addition,
our approach uses multiple predefined linear operations instead of traditional
dot product token interactions, thereby achieving a linear computational
complexity and significantly reduced memory footprint during inference. Our
contributions in this paper are as follows:

  * •

A new MCSD model is proposed to achieve a balance between computational
consumption and representation ability via series of linear fusion, helping to
address the scaling and deployment challenges.

  * •

To enhance the feature extraction capability, Multi-Channel Slope and Decay
(MCSD) block is proposed to achieve rich feature extraction and diverse
fusion, which ensure the diversity of feature extraction and fine-grained
feature interaction.

  * •

A recurrent representation is proposed during inference stage to accelerate
the inference speed. This simplified approach enjoys a computational
complexity linear to the length of the sequence and has a low and stable
memory complexity.

  * •

Experiments show that the MCSD approach outperforms Transformer on three
metrics: GPU memory, latency, and throughput, showing the robust scalability
of our method. The results confirm its competitive edge in delivering high-
performance outcomes at low computational costs, making it a viable solution
for resource-constrained edge-device deployments.

##  2 Methodology

###  2.1 Architecture

The MCSD model’s architecture, depicted in Fig. 1 (left), is a fully-connected
feed-forward network comprising $N$ stacked identical layers, as shown in Fig.
1 (left side). Input tokens are initially embedded to map discrete symbols
into a continuous space. After embedding, the data flows into the network for
subsequent feature learning and prediction of network outputs. Within each
layer, two residual connections encapsulate the MCSD Block and Gated MLP
Block, each preceded by RMSNorm Zhang and Sennrich (2019) for training
stability enhancement. The MCSD Block, detailed in Section 2.2, constitutes
the cornerstone of our innovation. The Gated MLP Block mirrors the design
outlined in Dauphin et al. (2017), utilizing GeGLU Shazeer (2020) as the
activation function in conjunction with linear layers.

Figure 1: The architecture of our MCSD model (left). The proposed MCSD block
(right) with decay and slope sections.

###  2.2 MCSD block

Figure 2: The slope section comprises multi-channel slope and slope
perturbation, integrating past positional information via distinct slope
matrices and conveying historical data to current features through element-
wise multiplication, respectively. A gating mechanism filters this output,
predominantly preserving current information.

The internal structure of MCSD block is showed on the right side of Fig. 1.
MCSD block integrates slope and decay sections, which capture local and global
feature extraction via different pre-defined weight matrix, respectively. Each
section employs a dual-branch design. One branch extracts multi-channel
historical features by aggregating the context from preceding tokens, another
branch conducts linear projection. Two branches are followed by a perturbation
operation, which facilitates the interaction between two branches.
Specifically, given input $X$ to the MCSD block, a dimensional transformation
operation is first performed $X$, ${X\in{{\mathbb{R}}^{N\times D}}}\to
X\in{{\mathbb{R}}^{C\times N\times{{D}_{c}}}}$, where $N$ denotes sequence
length, $D$ denotes feature dimension, $D=C\times{{D}_{c}}$, $C$ represents
the number of channels, and ${D}_{c}$ represents the feature dimension of each
channel.

There are two main differences between slope and decay sections: 1\. Divergent
Pre-defined Weights. This determines a long or short context-awareness ability
of the two in extracting historical information. 2\. Contrasting Perturbation
Directions. This design ensures that the slope section perturbs current
features by local historical information, whereas the decay section perturbs
global historical features in response to current inputs, fostering a nuanced
interplay between past and current contexts.

Figure 3: The decay section encompasses multi-channel decay and decay
perturbation, integrating past positional data via distinct decay matrices and
updating historical information through element-wise multiplication with
current features. A gating mechanism selectively filters this output,
primarily conserving historical information.

Slope section The detail of the slope section is shown in Figure 2. Given the
transformed input $X\in{{\mathbb{R}}^{C\times N\times{{D}_{c}}}}$, the slope
section yields the matrix $U$,$V$ by linear projection as illustrated in Eq.
(1), where
${{W}_{U}},{{W}_{V}}\in{{\mathbb{R}}^{C\times{{D}_{C}}\times{{D}_{C}}}}$;
$U,V\in{{\mathbb{R}}^{C\times N\times{{D}_{c}}}}$.

$U=X{{W}_{U}},V=X{{W}_{V}}$ (1)

In the multi-channel slope operation on matrix $V$, we set different weight
matrices for different channels to realize diverse contextual feature fusion.
The corresponding weight $\beta_{i}$ for each channel is defined as follows:

${{\beta}_{i}}={{\left({{2}^{{}^{-8}/{}_{C}}}\right)}^{i+1}},i\in\left\\{0,1,...C-2,C-1\right\\}$
(2)

where $\beta_{i}$ is the weight applied to the $i$-th channel pair. Then, we
pre-define the slope matrix $s_{i}$ for the $i$-th channel as shown in Eq. 3.
In order to integrate the extracted historical information, we pre-define
$s_{i}$ as a lower triangular matrix with decreasing weights in the lower
triangular direction, which assigns different weights to different positions
in the $N$ dimension. The upper-triangular portion of the slope matrix $s_{i}$
is negatively infinite. We normalize the matrix by a softmax function $\delta$
to ensure stable feature extraction under the pre-defined matrix. Since
$e^{-\infty}\rightarrow 0$, the upper triangular portion of matrix $s_{i}$
tends to $0$ after the softmax operation, which is equivalent to mask the
current and subsequent information for the historical information that does
not include the current information. To prevent the denominator of the first
row from approaching 0, set the first number to 1.

${{s}_{i}}=\delta\left(\left[\begin{matrix}1&\cdots&\cdots&-\infty\\\
-{{\beta}_{i}}&-\infty&\ddots&\vdots\\\
\vdots&-{{\beta}_{i}}&-\infty&\vdots\\\
\left(-N+1\right){{\beta}_{i}}&\cdots&-{{\beta}_{i}}&-\infty\\\
\end{matrix}\right]\right)$ (3)

The set of $C$ channel $s_{i}$ matrices forms the weight matrix $W_{s}$ for
subsequent computation. Subsequently, matrix $W_{s}$ and matrix $V$ undergo
the matrix multiplication. Here, the essence of matrix multiplication lies in
the weighting and manipulation of the historical feature of input $V$, thereby
imbuing the output feature with information from historical tokens. The
specific operation is shown in Eq. (5), where
${V}^{\prime}\in{{\mathbb{R}}^{C\times N\times{{D}_{c}}}}$ denotes the matrix
$V$ output after the multi-channel slope operation. This operation yields
features ${V}^{\prime}$ containing historical information.

${{W}_{s}}=\left({{s}_{0}},{{s}_{1}},...,{{s}_{C-2}},{{s}_{C-1}}\right)$ (4)

${V}^{\prime}={{W}_{s}}V$ (5)

${{\Theta}_{s}}=\sigma\left({{V}^{\prime}}\right)\odot U$ (6)

Then, a slope perturbation is employed on the linear projection output $U$ and
the feature output of multi-channel slope $V^{\prime}$. The slope perturbation
includes a gating mechanism to control the direction of perturbation and a
element-wise dot-multiplication to fuse the information. The historical
information $V^{\prime}$ first pass the gating function and then is used to
perturb feature $U$ via the dot-multiplication. The specific formula is shown
in Eq. (6). In this equation, $\sigma$ represents the SiLU function Elfwing et
al. (2017), and $\Theta_{s}\in{{\mathbb{R}}^{C\times N\times{{D}_{c}}}}$
represents the output of the slope section. The slope perturbation bring the
feature with historical information to the current linear input $U$ to obtain
feature with adjacent information interaction.

After multi-channel slope and slope perturbation, we can get the short-range
context-aware features, which incorporate the adjacent history information to
the current information.

Decay section The detail of the decay section is shown in Figure 3. Similarly,
given the input data $X$, the matrics $F,E$ are obtained by the linear
projection, as illustrated in Eq. (7), where
${{W}_{F}},{{W}_{E}}\in{{\mathbb{R}}^{C\times{{D}_{C}}\times{{D}_{C}}}}$;
$F,E\in{{\mathbb{R}}^{C\times N\times{{D}_{c}}}}$.

$F=X{{W}_{F}},E=X{{W}_{E}}$ (7)

One branch performs a multi-channel decay operation on the matrix $E$, which
is analogous to the slope section, with a different weight matrix. The multi-
channel decay operation is illustrated in Eq. (8-11). Inspired by Sun et al.
(2023), we define different $\alpha_{i}$ values for each channel. We then
incorporate these $\alpha$ values to build the triangular matrix and the
triangular matrix is normalized by the activation function $\varepsilon$ by
RMSNorm to obtain decay matrix $de_{i}\in{{{\mathbb{R}}}^{N\times N}}$ matrix.
The formulas for $\alpha_{i}$ values of each channel and decay matrix $de_{i}$
are shown in equations (8-9).

${{\alpha}_{i}}=1-{{2}^{-5-i}},i\in\left\\{0,1,...,C-2,C-1\right\\}$ (8)

${{de}_{i}}=\varepsilon\left(\left[\begin{matrix}1&-\infty&\cdots&\cdots&-\infty\\\
\alpha_{i}^{1}&-\infty&\ddots&\ddots&\vdots\\\
\vdots&\ddots&-\infty&\ddots&\vdots\\\
\alpha_{i}^{N-2}&\alpha_{i}^{N-3}&\ddots&-\infty&-\infty\\\
\alpha_{i}^{N-1}&\alpha_{i}^{N-2}&\cdots&\alpha_{i}^{1}&-\infty\\\
\end{matrix}\right]\right)$ (9)

Then ${E}^{\prime}$ can be defined as shown in Eq. (10). The $de_{i}$ matrices
of the different channels are concatenated to obtain the
${W}_{d}\in{{\mathbb{R}}^{C\times N\times N}}$ matrix for the multi-channel
decay operation. As shown in Eq. (11), a multi-channel decay operation is
performed on the matrix $E$ to extract and normalize the history information
of the input data. Unlike the multi-channel slope, the multi-channel decay
operation assigns relatively high weights to the more distant historical
information, thus ensuring that the matrix
${E}^{\prime}\in{{\mathbb{R}}^{C\times N\times{{D}_{c}}}}$ contains as much
global historical information as possible. The $W_{d}$ is calculated by:

${{W}_{d}}=\left({{de}_{0}},{{de}_{1}},...,{{de}_{C-2}},{{de}_{C-1}}\right)$
(10)

${E}^{\prime}={W}_{d}E$ (11)

The output of the decay section $\Theta_{d}$ is defined by Eq.(12). A decay
perturbation acts on linear projection output $F$ and multi-channel decay
feature output $E^{\prime}$, incorporating a gating mechanism to steer
perturbation direction and an element-wise multiplication for information
fusion. The gating function applied to $F$ modulates its impact on
$E^{\prime}$ through dot-multiplication. Here, $\sigma$ represents the sigmoid
function, and $\Theta_{d}\in{{\mathbb{R}}^{C\times N\times{{D}_{c}}}}$
represents the output of the $E^{\prime}$ section. The perturbation bring the
input feature $F$ to feature with global information $E^{\prime}$ to obtain
long-dependency information.

${{\Theta}_{d}}={E}^{\prime}\odot\sigma\left(F\right)$ (12)

$\Theta=Concat\left({{\Theta}_{s}},{{\Theta}_{d}}\right)$ (13)

The MCSD block’s output, denoted by $\Theta\in{{\mathbb{R}}^{C\times
N\times{2{D}_{c}}}}$, emerges from concatenating slope and decay section
outputs (Eq. 13). The combination of the two sections allows the MCSD block to
extract information from closer histories without losing information from more
distant histories, enhancing local and global feature extraction. Meanwhile,
the pre-defined multi-channel matrix enriches the representation on the
feature subspace.

###  2.3 Fast inference for the MCSD block

The matrix multiplication in the current algorithm can benefit from the high
speed parallel computation of GPUs in training. However, in inference, it
increases with input length $N$ causes $O(N^{2})$ algorithmic complexity and
$O(N)$ space complexity consumption. This is detrimental to the efficient
deployment of the model in end devices. To address this problem, we simplify
the computation of model inference by a recursive form. This transformation
results in a space complexity of $O(1)$ and a time complexity of $O(N)$ for
the inference process, which greatly reduces the inference time. The optimized
inference step in multi-channel ramp is shown in Eq. (14-16), while the
optimized inference step in multi-channel decay is shown in Eq. (17-19).

${{{V}^{\prime}}_{n}}=\frac{\sum\limits_{j=1}^{n-1}{{{e}^{\left(-j\right){{\beta}_{i}}}}{{V}_{n-j}}}}{\sum\limits_{j=1}^{n-1}{{{e}^{\left(-j\right){{\beta}_{i}}}}}}$
(14)

${{{V}^{\prime}}_{n+1}}=\frac{\sum\limits_{j=1}^{n}{{{e}^{\left(-j\right){{\beta}_{i}}}}{{V}_{n+1-j}}}}{\sum\limits_{j=1}^{n}{{{e}^{\left(-j\right){{\beta}_{i}}}}}}$
(15)

$\begin{split}S_{n+1}^{slope}&={{{V}^{\prime}}_{n+1}}={{e}^{\left(-1\right){{\beta}_{i}}}}{{{{V}^{\prime}}}_{n}}\frac{\sum\limits_{j=1}^{n-1}{{{e}^{\left(-j\right){{\beta}_{i}}}}}}{\sum\limits_{j=1}^{n}{{{e}^{\left(-j\right){{\beta}_{i}}}}}}+\frac{{{e}^{\left(-1\right){{\beta}_{i}}}}{{V}_{n}}}{\sum\limits_{j=1}^{n}{{{e}^{\left(-j\right){{\beta}_{i}}}}}}\\\
&=(1-\frac{1}{\sum\limits_{j=0}^{n-1}{{{e}^{\left(-j\right){{\beta}_{i}}}}}})S_{n}^{slope}+\frac{1}{\sum\limits_{j=0}^{n-1}{{{e}^{\left(-j\right){{\beta}_{i}}}}}}{{V}_{n}}\end{split}$
(16)

where $E_{j}\in{{\mathbb{R}}^{1\times{{D}_{c}}}}$ and
$V_{j}\in{{\mathbb{R}}^{1\times{{D}_{c}}}}$ represent the $j_{th}$ vector
representation in sequence, while $n$ denotes the number of currently inputted
vector representations in sequence.
$S_{n+1}^{slope}\in{{\mathbb{R}}^{1\times{{D}_{c}}}}$ and
$S_{n+1}^{decay}\in{{\mathbb{R}}^{1\times{{D}_{c}}}}$ represent the next
vector representation(with weight $0$ at position $n$) that is inferred under
$n$ $E_{i}$ and $V_{i}$ inputs, respectively. The $i_{th}$ channel weights,
$\beta_{i}$ and $\alpha_{i}$ represent the channel weights under multi-channel
slope and multi-channel decay, respectively.

${{{E}^{\prime}}_{n}}=\sum\limits_{j=1}^{n-1}{\alpha_{i}^{j}}{{E}_{n-j}}$ (17)

${{{E}^{\prime}}_{n+1}}=\sum\limits_{j=1}^{n}{\alpha_{i}^{j}}{{E}_{n+1-j}}$
(18)

$\begin{split}S_{n+1}^{decay}&={{{{E}^{\prime}}}_{n+1}}={{\alpha}_{i}}{{{{E}^{\prime}}}_{n}}+{{\alpha}_{i}}{{E}_{n}}\\\
&={{\alpha}_{i}}S_{n}^{decay}+{{\alpha}_{i}}{{E}_{n}}\end{split}$ (19)

In the original form (e.g., Eq. (14-15), Eq.(17-18)), when reasoning about the
data of the next token, it is necessary to compute all the data, which results
in a significant increase in computational complexity. In optimized
formulations such as Eq. (16), Eq. (19), the next output token is only related
to the input of the adjacent previous token, which greatly reduces the
computational complexity.

$\begin{split}\theta_{n+1}^{i}=&U\odot\sigma\left(S_{n+1}^{slope}\right)+\\\
&\sigma\left(F\right)\odot RMSNorm\left(S_{n+1}^{decay}\right)\end{split}$
(20)

$\Theta_{n+1}=\left(\theta_{n+1}^{0},\theta_{n+1}^{1},...,\theta_{n+1}^{C-1}\right)$
(21)

The final output is presented in Eq. (20) and Eq. (21). In these equations,
$\theta_{n+1}^{i}\in{{\mathbb{R}}^{1\times{{D}_{c}}}}$ represents the final
output of the $(i+1)_{th}$ channel, while
$\Theta_{n+1}\in{{\mathbb{R}}^{C\times 1\times{{D}_{c}}}}$ denotes the
$(i+1)_{th}$ output of the merged multi-channel sum, where
$i\in\left\\{0,1,...,C-2,C-1\right\\}$.

##  3 Experiment

###  3.1 Experimental Details

We train three models with different parameter size: 1.6B, 3B, 10B. The detail
hyperparameters of our method is summarized in Table 1. Three models were
trained, each with a different number of parameters: 1.6B, 3B and 10B. All
three models were trained using a batchsize of 1152 and a learning rate of
8e-4. All experiments were performed using the AdamW optimizer Hoffmann et al.
(2022).

Table 1: The hyperparameters of the proposed MCSD.

Size | Sequence Length | Hidde Size | Channel | Layers
---|---|---|---|---
1.6B | 4096 | 2560 | 10 | 12
3B | 4096 | 2560 | 10 | 20
10B | 4096 | 3072 | 12 | 60

###  3.2 Scaling Curves

Figure 4: Scaling curves for MCSD illustrate a linear decline in loss value
with growing training token volume, culminating in convergence. Larger model
parameter counts correlate with diminished converged loss values.

The scaling results, which demonstrate the relationship between the number of
training tokens and the training loss, are presented in Figure 4. The figure
illustrates three MCSD models for 1.6B, 3B and 10B, all using 4096 for the
sequence length. The results present a positive relationship between training
loss and model size in the case of training convergence, which is consistent
with Chinchilla’s scaling law Loshchilov and Hutter (2019). All three models
exhibit convergence when the number of training tokens reaches 10B, and the
models show relatively stable performance throughout the training process. The
result demonstrates that the MCSD model can be trained with stability and
efficiency, exhibiting good scalability.

###  3.3 Inference Cost

As illustrated in Figure (5-7), a comparison is presented between the GPU
memory, latency, and throughput of the Transformer and the proposed MCSD
during the inference phase. In these inference experiments, the 1.6B model was
evaluated on an RTX-3090 24G GPU. The sequence length represents the number of
output tokens and batch size represents the number of parallel inference
processes. The prompt length is 128 tokens. Transformers employ the reuse of
the key-value (KV) caches of previously decoded tokens. MCSD utilizes a
simplified representation, as illustrated in Eq. (20-21). The plots
demonstrate that MCSD outperforms the same parametric number of Transformer
models in terms of three inference metrics.

Figure 5: GPU memory versus sequence length curves for MCSD and Transformer.

GPU Memory As illustrated in Figure 5, the memory cost of the Transformer
increases linearly during inference due to the presence of the KV cache. In
contrast, the memory consumption of the proposed MCSD remains almost constant
even for long sequences, and MCSD has a lower memory footprint compared to the
Transformer of sequence length from 2048 to 8192. This helps to achieve long
sequence inference on end device with low memory footprint.

Figure 6: Latency versus batch size curves for MCSD and Transformers.

Latency Latency is a crucial metric in deployment, as it can significantly
impact user experience. Figure 6 illustrates the latency comparison between
MCSD and Transformer. The experimental results demonstrate that increasing the
batch size leads to a notable increase in the delay of Transformer, which is
approximately linear. Furthermore, the latency increases more rapidly when the
sequence length increases. This severely constrains the applicability of
Transformer to long sequence output. For a given sequence length, the delay of
MCSD is considerably less than that of the Transformer, and remains relatively
consistent across different batch sizes.

Throughput As illustrated in Figures (7-8), we examine the impact of sequence
length and batch size on throughput separately. As shown in Figure 7, we
adjust the batch size for each curve, and the throughput of Transformer
gradually declines with the increase in sequence length, assuming a constant
batch size. In the case of a batch size of at least 16, the presence of the KV
cache results in an out-of-memory issue. Conversely, the throughput of the
proposed MCSD reaches a bottleneck after a slight increase in sequence length,
remaining almost unchanged in the absence of out-of-memory issues.
Furthermore, under the same batch size, MCSD exhibits a higher throughput
performance than Transformer.

Figure 7: Throughput versus sequence length curves for MCSD and Transformers.
Figure 8: Throughput versus batch size curves for MCSD and Transformers. Table
2: Comparison of the accuracy of MCSD and other models in downstream
experiments with 5-shot.

Model type | Model size | ARC-C | ARC-E | WinoGrande | MMLU | Hellaswag | Average
---|---|---|---|---|---|---|---
Llama3 | 8B | 79.6 | 92.3 | 67.1 | 66.3 | 69.5 | 75.0
Pythia | 2.8B | 32.9 | 64.1 | 59.7 | 26.8 | 59.3 | 48.6
Mamba | 2.8B | 36.3 | 69.7 | 63.5 | 26.2 | 66.1 | 52.4
RWKV4 | 3B | 33.1 | 67.8 | 59.6 | 25.6 | 59.6 | 49.1
MCSD | 3B | 36.8 | 70.2 | 63.8 | 29.3 | 67.2 | 53.5

The curves in Figure 7 illustrate the relationship between throughput and
sequence length. The throughput of Transformer reaches a maximum and then
declines with an increase in batch size. The throughput of Transformer is
easily constrained and rapidly exhausts memory resources. With a fixed
sequence length, the throughput of MCSD increases with the increase of batch
size, and there is no out-of-memory phenomenon. Furthermore, the throughput of
MCSD is higher than that of Transformer with the same sequence length.
Additionally, the throughput of MCSD is several times higher than that of
Transformer in the case of the same sequence length.

###  3.4 Downstream Task Comparison

A series of evaluations were conducted on the downstream tasks of baseline
pre-trained models. The external baseline models we compared were
Llama3-8BTouvron et al. (2023), Mamba-2.8BGu and Dao (2024), RWKV4-3BPeng et
al. (2023), Pythia-2.8BBiderman et al. (2023). Among these models, Mamba-2.8B
and RWKV4-3B are the most robust small recurrent models reported in the latest
literature. Pythia-2B is the most robust transformer-structured small model.
Llama3 is a widely used state-of-the-art open transformer model. The proposed
MCSD is trained with 3B parameters. The evaluated performance is shown in
Table 2. It displays the results of the 5-shot evaluation.

In Table 2, a comparison was conducted between external baseline models and
MCSD on MMLU, HellaSwang, ARC-E, and ARC-C, as well as WinoGrande datasets.
All models were pre-trained model. The llama3-8B model is of the largest
number of parameters among the models, and it outperforms the other models in
all the metrics. Compared to other pre-trained model with similar model size,
MCSD-3B demonstrates a high level of performance. MCSD significantly
outperforms Pythia-2.8B and RWKV-3B on all metrics for all three datasets,
with average metrics exceeding 4.9 and 4.4, respectively. Compared to
Mamba-2.8B model, MCSD similarly outperforms it on all metrics, with an
average improvement of 1.1. The above results show that among models of
similar size, our proposed MCSD model has the leading performance.

###  3.5 Ablation Study

Ablation experiments were conducted for both the slope section and the decay
section. We established four sets of ablation experiments by either removing
the decay section or the slope section, or by using two decay or two slope
sections. The ablation experiment for our proposed MCSD is shown in Figure 9.
The training loss is plotted as a function of the number of training tokens.
The results show that deleting or replacing any of these sections increases
the training loss, weakening the model effect to varying degrees. The loss of
the slope section has a more pronounced impact on MCSD, while the loss of the
decay section has a comparatively smaller effect. Nevertheless, this does not
imply that the decay section is superfluous. Since most of the training data
are short texts, it is difficult to assess the effect of the decay component
on long texts.

Figure 9: Ablation experiment of the proposed MCSD.

##  4 Conclusion

This work introduces MCSD, a novel architecture that extracts the contextual
information using MCSD block consisting of the slope and decay sections. This
approach has been shown to enable very fast inference speed and low inference
cost. During inference, MCSD exhibits better scalability and low resource
consumption than Transformers with the same size. MCSD exhibits relatively
superior performance compared to language modeling at all scales, showing the
possibility of MCSD for end-side deployment and embodied intelligence.

##  5 Limitations

Notwithstanding advancements, limitations persist. Our method relies on
publicly available corpora, which may limit its generalizability to complex
professional domains such as law and medicine. Additionally, the potential
presence of unmitigated toxic content within these corpora underscores
toxicity mitigation as a critical area for future consideration. Furthermore,
while showcasing reduced resource consumption, extensive deployment on edge
devices awaits empirical scrutiny, demanding further assessment on hardware-
limited platforms. Moreover, the current iteration of the MCSD module does not
incorporate mechanisms for synchronous learning, i.e., concurrent training and
inference. Enhancing our model with such capabilities could significantly
broaden its applicability in real-world scenarios. Tailoring the architecture
to support training-inference synchronization represents a promising avenue
for future research.

## References

  * Ainslie et al. (2023) Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023.  Gqa: Training generalized multi-query transformer models from multi-head checkpoints.  _Preprint_ , arXiv:2305.13245.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  _Preprint_ , arXiv:2304.01373.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Dao and Gu (2024) Tri Dao and Albert Gu. 2024.  Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.  _Preprint_ , arXiv:2405.21060.
  * Dauphin et al. (2017) Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017.  Language modeling with gated convolutional networks.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, page 933–941. JMLR.org.
  * De et al. (2024) Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. 2024.  Griffin: Mixing gated linear recurrences with local attention for efficient language models.  _Preprint_ , arXiv:2402.19427.
  * Elfwing et al. (2017) Stefan Elfwing, Eiji Uchibe, and Kenji Doya. 2017.  Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.  _Preprint_ , arXiv:1702.03118.
  * Gesmundo and Maile (2023) Andrea Gesmundo and Kaitlin Maile. 2023.  Composable function-preserving expansions for transformer architectures.  _Preprint_ , arXiv:2308.06103.
  * Gu and Dao (2024) Albert Gu and Tri Dao. 2024.  Mamba: Linear-time sequence modeling with selective state spaces.  _Preprint_ , arXiv:2312.00752.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022.  Training compute-optimal large language models.  _Preprint_ , arXiv:2203.15556.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  _Preprint_ , arXiv:1711.05101.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _Preprint_ , arXiv:2303.08774.
  * Peng et al. (2023) Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. 2023.  Rwkv: Reinventing rnns for the transformer era.  _Preprint_ , arXiv:2305.13048.
  * Shazeer (2019) Noam Shazeer. 2019.  Fast transformer decoding: One write-head is all you need.  _Preprint_ , arXiv:1911.02150.
  * Shazeer (2020) Noam Shazeer. 2020.  Glu variants improve transformer.  _Preprint_ , arXiv:2002.05202.
  * Sun et al. (2023) Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. 2023.  Retentive network: A successor to transformer for large language models.  _Preprint_ , arXiv:2307.08621.
  * Team et al. (2024) Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, and Amelia Glaese et al. 2024.  Gemini: A family of highly capable multimodal models.  _Preprint_ , arXiv:2312.11805.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.  _Preprint_ , arXiv:2302.13971.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Zhai et al. (2021) Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. 2021.  An attention free transformer.  _Preprint_ , arXiv:2105.14103.
  * Zhang and Sennrich (2019) Biao Zhang and Rico Sennrich. 2019.  Root mean square layer normalization.  _Preprint_ , arXiv:1910.07467.