[table]capposition=top

# DeBERTaV3: Improving DeBERTa using
ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

Pengcheng He1, Jianfeng Gao2, Weizhu Chen1
1 Microsoft Azure AI
2 Microsoft Research
<EMAIL_ADDRESS>
###### Abstract

This paper presents a new pre-trained language model, DeBERTaV3, which
improves the original DeBERTa model by replacing masked language modeling
(MLM) with replaced token detection (RTD), a more sample-efficient pre-
training task. Our analysis shows that vanilla embedding sharing in ELECTRA
hurts training efficiency and model performance, because the training losses
of the discriminator and the generator pull token embeddings in different
directions, creating the “tug-of-war” dynamics. We thus propose a new
_gradient-disentangled embedding sharing_ method that avoids the tug-of-war
dynamics, improving both training efficiency and the quality of the pre-
trained model. We have pre-trained DeBERTaV3 using the same settings as
DeBERTa to demonstrate its exceptional performance on a wide range of
downstream natural language understanding (NLU) tasks. Taking the GLUE
benchmark with eight tasks as an example, the DeBERTaV3 Large model achieves a
91.37% average score, which is 1.37% higher than DeBERTa and 1.91% higher than
ELECTRA, setting a new state-of-the-art (SOTA) among the models with a similar
structure. Furthermore, we have pre-trained a multilingual model mDeBERTaV3
and observed a larger improvement over strong baselines compared to English
models. For example, the mDeBERTaV3 Base achieves a 79.8% zero-shot cross-
lingual accuracy on XNLI and a 3.6% improvement over XLM-R Base, creating a
new SOTA on this benchmark. Our models and code are publicly available at
https://github.com/microsoft/DeBERTa.

##  1 Introduction

Recent advances in Pre-trained Language Models (PLMs) have created new state-
of-the-art results on many natural language processing (NLP) tasks. While
scaling up PLMs with billions or trillions of parameters (Raffel et al., 2020;
Radford et al., 2019; Brown et al., 2020; He et al., 2020; Fedus et al., 2021)
is a well-proved way to improve the capacity of the PLMs, it is more important
to explore more energy-efficient approaches to build PLMs with fewer
parameters and less computation cost while retaining high model capacity.

Towards this direction, there are a few works that significantly improve the
efficiency of PLMs. The first is RoBERTa (Liu et al., 2019) which improves the
model capacity with a larger batch size and more training data. Based on
RoBERTa, DeBERTa (He et al., 2020) further improves the pre-training
efficiency by incorporating disentangled attention which is an improved
relative-position encoding mechanism. By scaling up to 1.5B parameters, which
is about an eighth of the parameters of xxlarge T5 (Raffel et al., 2020),
DeBERTa surpassed human performance on the SuperGLUE (Wang et al., 2019a)
leaderboard for the first time. The second new pre-training approach to
improve efficiency is Replaced Token Detection (RTD), proposed by ELECTRA
(Clark et al., 2020). Unlike BERT (Devlin et al., 2019), which uses a
transformer encoder to predict corrupt tokens with masked language modeling
(MLM), RTD uses a generator to generate ambiguous corruptions and a
discriminator to distinguish the ambiguous tokens from the original inputs,
similar to Generative Adversarial Networks (GAN). The effectiveness of RTD is
also verified by several works, including CoCo-LM (Meng et al., 2021), XLM-E
(Chi et al., 2021), CodeBERT(Feng et al., 2020) and SmallBenchNLP (Kanakarajan
et al., 2021).

In this paper, we explore two methods of improving the efficiency of pre-
training DeBERTa. Following ELECTRA-style training, we replace MLM in DeBERTa
with RTD where the model is trained as a discriminator to predict whether a
token in the corrupt input is either original or replaced by a generator. We
show that DeBERTa trained with RTD significantly outperforms the model trained
using MLM. The second is a new embedding sharing method. In ELECTRA, the
discriminator and the generator share the same token embeddings. However, our
analysis shows that embedding sharing hurts training efficiency and model
performance, since the training losses of the discriminator and the generator
pull token embeddings into opposite directions. This is because the training
objectives between the generator and the discriminator are very different. The
MLM used for training the generator tries to pull the tokens that are
semantically similar close to each other while the RTD of the discriminator
tries to discriminate semantically similar tokens and pull their embeddings as
far as possible to optimize the binary classification accuracy, causing a
conflict between their training objectives. In other words, this creates the
“tug-of-war” dynamics that reduces the training efficiency and the model
quality, as illustrated in Hadsell et al. (2020). On the other hand, we show
that using separated embeddings for the generator and the discriminator
results in significant performance degradation when we fine-tune the
discriminator on downstream tasks, indicating the merit of embedding sharing,
e.g., the embeddings of the generator are beneficial to produce a better
discriminator, as argued in Clark et al. (2020). To balance these tradeoffs,
we propose a new _gradient-disentangled embedding sharing_ (GDES) method where
the generator shares its embeddings with the discriminator but stops the
gradients from the discriminator to the generator embeddings. This way, we
avoid the tug-of-war effect and preserve the benefits of embedding sharing. We
empirically demonstrate that GDES improves both pre-training efficiency and
the quality of the pre-trained models.

We pre-train four variants of DeBERTaV3 models, i.e., DeBERTaV3large,
DeBERTaV3base, DeBERTaV3small and DeBERTaV3xsmall. We evaluate them on various
representative natural language understanding (NLU) benchmarks and set new
state-of-the-art numbers among models with a similar model structure. For
example, DeBERTaV3large surpasses previous SOTA models with a similar model
structure on GLUE (Wang et al., 2019b) benchmark with an average score over
+1.37%, which is significant. DeBERTaV3base achieves a 90.6% accuracy score on
the MNLI-matched (Williams et al., 2018) evaluation set and an 88.4% F1 score
on the SQuAD v2.0 (Rajpurkar et al., 2018) evaluation set. This improves
DeBERTabase by 1.8% and 2.2%, respectively. Without knowledge distillation,
DeBERTaV3small and DeBERTaV3xsmall surpasses previous SOTA models with a
similar model structure on both MNLI-matched and SQuAD v2.0 evaluation set by
more than 1.2% in accuracy and 1.3% in F1, respectively. We also train
DeBERTaV3base on the CC100 (Conneau et al., 2020) multilingual data using a
similar setting as XLM-R (Conneau et al., 2020) but with only a third of the
training passes. We denote the model as mDeBERTaV3base. Under the cross-
lingual transfer setting, mDeBERTaV3base achieves a 79.8% average accuracy
score on the XNLI (Conneau et al., 2018) task, which outperforms XLM-Rbase and
mT5base (Xue et al., 2021) by 3.6% and 4.4%, respectively. This makes
mDeBERTaV3 the best model among multi-lingual models with a similar model
structure. All these results strongly demonstrate the efficiency of DeBERTaV3
models and set a good base for future exploration towards more efficient PLMs.

##  2 Background

###  2.1 Transformer

A Transformer-based language model is composed of $L$ stacked Transformer
blocks (Vaswani et al., 2017). Each block contains a multi-head self-attention
layer followed by a fully connected positional feed-forward network. The
standard self-attention mechanism lacks a natural way to encode word position
information. Thus, existing approaches add a positional bias to each input
word embedding so that each input word is represented by a vector whose value
depends on both its content and position. The positional bias can be
implemented using absolute position embedding (Vaswani et al., 2017; Brown et
al., 2020; Devlin et al., 2019) or relative position embedding (Huang et al.,
2018; Yang et al., 2019). Several studies have shown that relative position
representations are more effective for natural language understanding and
generation tasks (Dai et al., 2019; Shaw et al., 2018; He et al., 2020).

###  2.2 DeBERTa

DeBERTa improves BERT with two novel components: DA (Disentangled Attention)
and an enhanced mask decoder. Unlike existing approaches that use a single
vector to represent both the content and the position of each input word, the
DA mechanism uses two separate vectors: one for the content and the other for
the position. Meanwhile, the DA mechanism’s attention weights among words are
computed via disentangled matrices on both their contents and relative
positions. Like BERT, DeBERTa is pre-trained using masked language modeling.
The DA mechanism already considers the contents and relative positions of the
context words, but not the absolute positions of these words, which in many
cases are crucial for the prediction. DeBERTa uses an enhanced mask decoder to
improve MLM by adding absolute position information of the context words at
the MLM decoding layer.

###  2.3 ELECTRA

####  2.3.1 Masked Language Model

Large-scale Transformer-based PLMs are typically pre-trained on large amounts
of text to learn contextual word representations using a self-supervision
objective, known as MLM (Devlin et al., 2019). Specifically, given a sequence
$\bm{X}=\left\\{x_{i}\right\\}$, we corrupt it into $\bm{\tilde{X}}$ by
masking 15% of its tokens at random and then train a language model
parameterized by $\theta$ to reconstruct $\bm{X}$ by predicting the masked
tokens $\tilde{x}$ conditioned on $\bm{\tilde{X}}$:

$\max_{\theta}\text{log}\;p_{\theta}(\bm{X}|\bm{\tilde{X}})=\max_{\theta}\sum_{i\in\mathcal{C}}{\text{log}\;p_{\theta}(\tilde{x}_{i}=x_{i}|\bm{\tilde{X}})}$
(1)

where $\mathcal{C}$ is the index set of the masked tokens in the sequence. The
authors of BERT propose to keep 10% of the masked tokens unchanged, another
10% replaced with randomly picked tokens and the rest replaced with the [MASK]
token.

####  2.3.2 Replaced token detection

Unlike BERT, which uses only one transformer encoder and trained with MLM,
ELECTRA was trained with two transformer encoders in GAN style. One is called
generator trained with MLM; the other is called discriminator trained with a
token-level binary classifier. The generator is used to generate ambiguous
tokens to replace masked tokens in the input sequence. Then the modified input
sequence is fed to the discriminator. The binary classifier in the
discriminator needs to determine if a corresponding token is either an
original token or a token replaced by the generator. We use $\theta_{G}$ and
$\theta_{D}$ to represent the parameters of the generator and the
discriminator, respectively. The training objective in the discriminator is
called RTD (Replaced Token Detection). The loss function of the generator can
be written as,

$L_{MLM}=\mathbbm{E}\left(-\sum_{i\in\mathcal{C}}{\text{log}\;p_{\theta_{G}}\left(\tilde{x}_{i,G}=x_{i}|\bm{\tilde{X}}_{G}\right)}\right)$
(2)

, where $\bm{\tilde{X}}_{G}$ is the input to the generator by randomly masking
$15\%$ tokens in $\bm{X}$.

The input sequence of the discriminator is constructed by replacing masked
tokens with new tokens sampled according to the output probability from the
generator:

$\tilde{x}_{i,D}=\left\\{\begin{array}[]{cr}\tilde{x}_{i}\sim
p_{\theta_{G}}\left(\tilde{x}_{i,G}=x_{i}|\bm{\tilde{X}}_{G}\right),&i\in\mathcal{C}\\\
x_{i},&i\notin\mathcal{C}\end{array}\right.$ (3)

The loss function of the discriminator is written as,

$L_{RTD}=\mathbbm{E}\left(-\sum_{i}{\text{log}\;p_{\theta_{D}}\left(\mathbbm{1}\left(\tilde{x}_{i,D}=x_{i}\right)|\bm{\tilde{X}}_{D},i\right)}\right)$
(4)

, where $\mathbbm{1}(\cdot)$ is the indicator function and
$\bm{\tilde{X}}_{D}$ is the input to the discriminator constructed via
equation 3. In ELECTRA, $L_{MLM}$ and $L_{RTD}$ are optimized jointly,
$L=L_{MLM}+\lambda L_{RTD}$, where $\lambda$ is the weight of the
discriminator loss $L_{RTD}$, which was set to 50 in ELECTRA.

##  3 DeBERTaV3

This section describes DeBERTaV3, which improves DeBERTa by using the RTD
training loss of Clark et al. (2020) and a new weight-sharing method.

###  3.1 DeBERTa with RTD

Since RTD in ELECTRA and the disentangled attention mechanism in DeBERTa have
proven to be sample-efficient for pre-training, we propose a new version of
DeBERTa, referred to as _DeBERTaV3_ , by replacing the MLM objective used in
DeBERTa with the RTD objective to combine the strengths of the latter.

In this implementation, Wikipedia and the bookcorpus (Zhu et al., 2015) are
used as training data, following the base model configuration of Devlin et al.
(2019). The generator is the same width as the discriminator but is half the
depth. The batch size is set to 2048, and the model is trained for 125,000
steps with a learning rate of 5e-4 and warmup steps of 10,000. Following Clark
et al. (2020), we use $\lambda=50$ with the same optimization hyperparameters.
We validate the effectiveness of DeBERTaV3 on two representative NLU tasks,
i.e., MNLI and SQuAD v2.0. The results, presented in
\raisebox{-0.9pt}{\small1}⃝ of Table 2, show that DeBERTaV3 significantly
outperforms DeBERTa, i.e., +2.5% on the MNLI-m accuracy and +3.8% on the SQuAD
v2.0 F1.

In the next two subsections, we will show that the performance of DeBERTaV3
can be further improved by replacing token Embedding Sharing (ES) used for
RTD, originally proposed in Clark et al. (2020), by a new Gradient-
Disentangled Embedding Sharing (GDES) method. We start with an analysis of ES
in Section 3.2.

###  3.2 Token Embedding Sharing in ELECTRA

To pre-train ELECTRA, we use a generator and a discriminator that share token
embeddings, as shown in Figure 1 (a). This method, called Embedding Sharing
(ES), allows the generator to provide informative inputs for the discriminator
and reduces the number of parameters to learn. However, it also creates a
multitask learning problem, where the generator’s and the discriminator’s
objectives interfere with each other and slow down the training convergence.

Let $\bm{E}$ and $g_{\bm{E}}$ be the token embeddings and their gradients,
respectively. In each training step of ELECTRA, we compute $g_{\bm{E}}$ by
back-propagating the errors from both the generator’s Masked Language Modeling
(MLM) loss and the discriminator’s Replaced Token Detection (RTD) loss, as
$g_{\bm{E}}=\frac{\partial L_{MLM}}{\partial\bm{E}}+\lambda\frac{\partial
L_{RTD}}{\partial\bm{E}}$. This equation means that the token embeddings are
updated by balancing the gradients from the two tasks, which can be seen as a
tug-of-war procedure (Hadsell et al., 2020). The procedure can eventually
converge if we control the update speed carefully (e.g., using a small
learning rate or gradient clipping), but it can be very inefficient if the two
tasks have different optimal directions for the embeddings. This is indeed the
case for MLM and RTD, because they have opposite effects on the token
embeddings. MLM encourages the embeddings of semantically similar tokens to be
close to each other, while RTD tries to separate them to make the
classification easier.

To verify our hypothesis, we implement a variant of ELECTRA that does not
share token embeddings between the generator and the discriminator, which we
refer to as No Embedding Sharing (NES), as illustrated in Figure 1 (b). In
NES, we update the generator and the discriminator alternately in each
training step. First, we run the generator to generate inputs for the
discriminator, and then we update the generator’s parameters, including its
token embeddings $\bm{E}_{G}$, by back-propagating the MLM loss. Next, we run
the discriminator using the inputs from the generator and update the
discriminator’s parameters, including its token embeddings $\bm{E}_{D}$, by
back-propagating the RTD loss.

((a))

((b))

((c))

Figure 1: Illustration of different embedding sharing methods. (a) ES:
$\bm{E}$, $\theta_{G}$ and $\theta_{D}$ will be jointly updated in a single
backward pass with regards to $L_{MLM}+\lambda L_{RTD}$ . (b) NES:
$\bm{E}_{G}$ and $\theta_{G}$ will first be updated via the backward pass with
regards to $L_{MLM}$, then $\bm{E}_{D}$ and $\theta_{D}$ will be updated via
the backward pass with regards to $\lambda L_{RTD}$ . (c) GDES: $\bm{E}_{G}$
and $\theta_{G}$ will first be updated in the backward pass with regards to
$L_{MLM}$, then $\bm{E}_{\Delta}$ and $\theta_{D}$ will be updated via the
backward pass with regards to $\lambda L_{RTD}$ and $\bm{E}_{G}$ . $sg$ is the
stop gradient operator that prevents the discriminator from updating
$\bm{E}_{G}$

[2]  Word Embedding Sharing $\bm{E}_{G}$ $\bm{E}_{D}$ $\bm{E}_{\Delta}$
\raisebox{-0.9pt}{\small1}⃝ ES 0.02 0.02 - \raisebox{-0.9pt}{\small2}⃝ NES
0.45 0.02 - \raisebox{-0.9pt}{\small3}⃝ GDES 0.45 0.29 0.02 Model MNLI-m/mm
SQuAD v2.0 Acc F1/EM BERTbase 84.3/84.7 76.3/73.7 ELECTRAbase 85.8/- -/-
DeBERTabase 86.3/86.2 82.5/79.3 DeBERTa+RTDbase \raisebox{-0.9pt}{\small1}⃝ ES
88.8/88.4 86.3/83.5 \raisebox{-0.9pt}{\small2}⃝ NES 88.3/87.9 85.3/82.7
\raisebox{-0.9pt}{\small3}⃝ GDES 89.3/89.0 87.2/84.5

Table 1:  Average cosine similarity of word embeddings of the generator and
the discriminator with different embedding sharing methods.  Table 2:  Fine-
tuning results on MNLI and SQuAD v2.0 tasks of base models trained with
different embedding sharing methods.

We compare ES and NES on three aspects: convergence speed, quality of the
token embeddings, and performance on downstream Natural Language Understanding
(NLU) tasks. Figure 2 shows that NES converges faster than ES, as expected,
because it avoids the conflicting gradients between the two downstream tasks.
We then measure the average cosine similarity scores of the token embeddings
111We sample 10% of the word pieces from the vocabulary and calculate the
average cosine similarity of every pair of two word pieces. . Table 2 shows
that NES produces two distinct embedding models, with $\bm{E}_{G}$ being more
semantically coherent than $\bm{E}_{D}$. This confirms our hypothesis.
However, the embeddings learned by NES do not lead to any significant
improvement on two representative downstream NLU tasks (i.e., MNLI and SQuAD
v2.0), as shown in Table 2. This result supports the argument of Clark et al.
(2020) that ES has the advantage of making the discriminator benefit from the
generator’s embeddings, in addition to saving parameters. In the next
subsection, we propose a new embedding sharing method that combines the
strengths of ES and NES, while avoiding their drawbacks.

###  3.3 Gradient-Disentangled Embedding Sharing

We propose the Gradient-Disentangled Embedding Sharing (GDES) method to
overcome the drawbacks of ES and NES while retaining their advantages. As
shown in Figure 1 (c), GDES shares the token embeddings between the generator
and the discriminator, which enables the two models to learn from the same
vocabulary and leverage the rich semantic information encoded in the
embeddings. However, unlike ES, GDES does not allow the RTD loss to affect the
gradients of the generator, thus avoiding the interference and inefficiency
caused by the conflicting objectives. Instead, GDES only updates the generator
embeddings with the MLM loss, which ensures the consistency and coherence of
the generator output. As a result, GDES can achieve the same converging speed
as NES, but without sacrificing the quality of the embeddings.

To implement GDES, we re-parameterize the discriminator embeddings as
$\bm{E}_{D}=sg(\bm{E}_{G})+\bm{E}_{\Delta}$, where the stop gradient operator
$sg$ prevents the gradients from flowing through the generator embeddings
$\bm{E}_{G}$ and only updates the residual embeddings $\bm{E}_{\Delta}$. We
initialize $\bm{E}_{\Delta}$ as a zero matrix and train the model following
the NES procedure. In each iteration, we first generate the inputs for the
discriminator using the generator and update both $\bm{E}_{G}$ and
$\bm{E}_{D}$ with the MLM loss. Then, we run the discriminator on the
generated inputs and update $\bm{E}_{D}$ with the RTD loss, but only through
$\bm{E}_{\Delta}$. After training, we add $\bm{E}_{\Delta}$ to $\bm{E}_{G}$
and save the resulting matrix as $\bm{E}_{D}$ for the discriminator.

We conduct extensive experiments to evaluate the effectiveness of GDES
compared to ES and NES. We measure the converging speed, the quality of the
token embeddings, and the performance on downstream tasks. The results in
Figure 2, Tables 2 and 2 demonstrate that GDES outperforms both ES and NES in
various aspects. First, Figure 2 reveals that GDES converges faster than ES
and matches the efficiency of NES. As GDES, ES and NES only differ at the way
of embedding sharing, the computation cost of each step is the same.

Figure 2: MLM training loss of the generator with different word embedding
sharing methods.

Second, Table 2 indicates that GDES produces two distinctive token embedding
matrices, with the generator embeddings having higher average similarity
scores than the discriminator embeddings. However, the difference is smaller
than that in NES, suggesting that GDES preserves more semantic information in
the discriminator embeddings through the partial weight sharing. Third, Table
2 shows that after fine-tuning, the model pre-trained with GDES achieves the
best performance on two downstream tasks, MNLI and SQuAD v2.0. These results
confirm that GDES is an effective weight-sharing method for language model
pre-trained with MLM and RTD.

##  4 Experiment

###  4.1 Main Results on NLU tasks

To further verify the effectiveness of those technologies, we combine RTD,
GDES and DA (Disentangled Attention) to train models of different sizes(i.e.,
large, base and small) using standard pre-training settings. Since all of our
experiments are modified based on DeBERTa code base and follow most of the
settings of DeBERTa, we denote the new models as DeBERTaV3large,
DeBERTaV3base, and DeBERTaV3small. The discriminator part of DeBERTaV3large
and DeBERTaV3base are the same as DeBERTalarge and DeBERTabase, respectively.
The discriminator of DeBERTaV3small has the same width and attention heads as
DeBERTabase and half the depth of DeBERTabase, i.e., 6 layers with 768 hidden
size and 12 attention heads. The generator of DeBERTaV3 has the same width as
the discriminator and half the depth of the discriminator. We train those
models with 160GB data, which is the same as DeBERTaV2 and RoBERTa, and use
the same SentencePiece (Kudo, 2018; Sennrich et al., 2016) vocabulary as
DeBERTaV2 (He et al., 2020) which contains 128,000 tokens. All the models are
trained for 500,000 steps with a batch size of 8192 and warming up steps of
10,000. The learning rate for base and small model is 5e-4, while the learning
rate for large model is 3e-4. Following the DeBERTa setting, we use the AdamW
(Loshchilov & Hutter, 2018) optimizer which is a fixed version of Adam (Kingma
& Ba, 2014) with weight decay, and set $\beta_{1}=0.9$, $\beta_{2}=0.98$ for
the optimizer. After pre-training, the discriminators of those models are used
for downstream task fine-tuning following the same paradigm as Transformer
PLMs, such as BERT, RoBERTa, ELECTRA, and DeBERTa. We provide more details on
the hyper parameters of pre-training and fine-tuning in the Appendix.

####  4.1.1 Performance on Large Models

Following previous studies on PLMs, we first evaluate our model on the eight
NLU tasks in GLUE (Wang et al., 2019b), which are the most representative
sentence classification tasks. We fine tune the pre-trained models on those
tasks by plugging a classification head on top of the hidden states of the
[CLS] token at the last layer. We summarize the results in Table 3, where
DeBERTaV3 is compared with previous Transformer-based PLMs of similar
structures (i.e., 24 layers with hidden size of 1024), including BERT,
RoBERTa, XLNet (Yang et al., 2019), ALBERT (Lan et al., 2019), ELECTRA and
DeBERTa. Compared to previous SOTA models, DeBERTaV3 performs consistently
comparable or mostly better across all the tasks. Meanwhile, DeBERTaV3
outperforms XLNet in seven out of eight tasks. In terms of average GLUE score,
DeBERTaV3 outperforms other SOTA PLMs with a large margin ($>1.3\%$).
Particularly, compared with previous best numbers, there are big jumps on the
performance of low resource tasks (i.e., RTE ($+4.4\%$), CoLA ($+4.8\%$)).
This indicates DeBERTaV3 is more data efficient and has a better
generalization performance. We also note that the improvements on SST-2,
STS-B, and MRPC are relatively small ($<0.3\%$). We conjecture this is due to
the performance on those tasks is close to be saturated, and thus even small
but consistent improvements on them are valuable.

Model | CoLA | QQP | MNLI-m/mm | SST-2 | STS-B | QNLI | RTE | MRPC | Avg.
---|---|---|---|---|---|---|---|---|---
Mcc | Acc | Acc | Acc | Corr | Acc | Acc | Acc |
#Train | 8.5k | 364k | 393k | 67k | 7k | 108k | 2.5k | 3.7k |
BERTlarge | 60.6 | 91.3 | 86.6/- | 93.2 | 90.0 | 92.3 | 70.4 | 88.0 | 84.05
RoBERTalarge | 68.0 | 92.2 | 90.2/90.2 | 96.4 | 92.4 | 93.9 | 86.6 | 90.9 | 88.82
XLNetlarge | 69.0 | 92.3 | 90.8/90.8 | 97.0 | 92.5 | 94.9 | 85.9 | 90.8 | 89.15
ELECTRAlarge | 69.1 | 92.4 | 90.9/- | 96.9 | 92.6 | 95.0 | 88.0 | 90.8 | 89.46
DeBERTalarge | 70.5 | 92.3 | 91.1/91.1 | 96.8 | 92.8 | 95.3 | 88.3 | 91.9 | 90.00
DeBERTaV3large | 75.3 | 93.0 | 91.8/91.9 | 96.9 | 93.0 | 96.0 | 92.7 | 92.2 | 91.37
Table 3:  Comparison results on the GLUE development set.

To further evaluate the model performance, in addition to GLUE, DeBERTaV3large
is evaluated on three categories of representative NLU benchmarks: (1)
Question Answering: SQuAD v2.0, RACE (Lai et al., 2017), ReCoRD (Zhang et al.,
2018), and SWAG (Zellers et al., 2018); (2) Natural Language Inference: MNLI;
and (3) NER: CoNLL-2003 (Sang & De Meulder, 2003). Among those tasks, RACE,
SWAG, and MNLI are fine-tuned using a same way as sentence classification
tasks. SQuAD v2.0, ReCoRD and NER are fine-tuned as sequence tagging tasks,
where a token classification head is plugged on top of the hidden states of
each token at the last layer. For comparison, we include ALBERTxxlarge 222 The
hidden dimension of ALBERTxxlarge is 4 times of DeBERTa and the computation
cost is about 4 times of DeBERTa., DeBERTalarge, DeBERTa1.5B, and Megatron
(Shoeybi et al., 2019) with three different model sizes, denoted as
Megatron336M, Megatron1.3B and Megatron3.9B, which are trained using the same
dataset as RoBERTa. Note that Megatron336M has a similar model size as other
models mentioned above333T5 (Raffel et al., 2020) has more parameters (11B).
Raffel et al. (2020) only reported the test results of T5 which are not
comparable with models mentioned above..

Model | MNLI-m/mm | SQuAD v2.0 | RACE | ReCoRD | SWAG | NER
---|---|---|---|---|---|---
Acc | F1/EM | Acc | F1/EM | Acc | F1
BERTlarge | 86.6/- | 81.8/79.0 | 72.0 | - | 86.6 | 92.8
ALBERTlarge | 86.5/- | 84.9/81.8 | 75.2 | - | - | -
RoBERTalarge | 90.2/90.2 | 89.4/86.5 | 83.2 | 90.6/90.0 | 89.9 | 93.4
XLNetlarge | 90.8/90.8 | 90.6/87.9 | 85.4 | - | - | -
ELECTRAlarge | 90.9/- | -/88.1 | - | - | - | -
Megatron336M | 89.7/90.0 | 88.1/84.8 | 83.0 | - | - | -
DeBERTalarge | 91.1/91.1 | 90.7/88.0 | 86.8 | 91.4/91.0 | 90.8 | 93.8
DeBERTaV3large | 91.8/91.9 | 91.5/89.0 | 89.2 | 92.3/91.8 | 93.4 | 93.9
ALBERTxxlarge | 90.8/- | 90.2/87.4 | 86.5 | - | - | -
Megatron1.3B | 90.9/91.0 | 90.2/87.1 | 87.3 | - | - | -
Megatron3.9B | 91.4/91.4 | 91.2/88.5 | 89.5 | - | - | -
DeBERTa1.5B | 91.7/91.9 | 92.2/89.7 | 90.8 | 94.5/94.0 | 92.3 | -
Table 4:  Results on MNLI in/out-domain, SQuAD v2.0, RACE, ReCoRD, SWAG, CoNLL
2003 NER development set. Note that missing results in literature are
signified by “-”.

We summarize the results in Table 4. Compared to the previous SOTA PLMs with a
similar model size (i.e., BERT, RoBERTa, XLNet, ALBERTlarge, Megatron336M and
DeBERTalarge), DeBERTaV3large shows superior performance on all six tasks. We
see a big performance jump on RACE (+2.4%) and SWAG (+2.6%), which require the
models to have non-trivial reasoning capability and common-sense knowledge
(Lai et al., 2017; Zellers et al., 2018). We conjecture those improvements
indicate DeBERTaV3large has a better capability of reasoning and common sense
knowledge. Although it is well proved to improve the model capacity by
increasing the number of parameters (Raffel et al., 2020; Fedus et al., 2021),
compared with larger models, DeBERTaV3large outperforms ALBERTxxlarge and
Megatron1.3B by a large margin on all three tasks, as well as outperform
Megatron3.3B on both MNLI and SQuAD v2.0. Compared with DeBERTa1.5B, which
used to be the SOTA NLU models on GLUE and SuperGLUE leaderboards,
DeBERTaV3large is still on par with it on MNLI but outperforms it on SWAG.

####  4.1.2 Performance on Base and Smaller Models

We evaluate DeBERTaV3base, DeBERTaV3small, and DeBERTaV3xsmall on two
representative tasks, i.e., MNLI and SQuAD v2.0, and summarize the results in
Table 5. DeBERTaV3base consistently outperforms DeBERTabase and ELECTRAbase by
a larger margin than that in the Large models. For example, on MNLI-m,
DeBERTaV3base obtains an improvement of +1.8(90.6% vs. 88.8%) over both
DeBERTabase and ELECTRAbase. On SQuAD v2.0 in terms of the EM score,
DeBERTaV3base achieves an improvement of +4.9% (85.4% vs. 80.5%) over
ELECTRAbase and +2.3% (85.4% vs 83.1%) over DeBERTabase.

Model | Vocabulary | Backbone | MNLI-m/mm | SQuAD v2.0
---|---|---|---|---
Size(K) | #Params(M) | ACC | F1/EM
Base models:12 layers,768 hidden size,12 heads
BERTbase | 30 | 86 | 84.3/84.7 | 76.3/73.7
RoBERTabase | 50 | 86 | 87.6/- | 83.7/80.5
XLNetbase | 32 | 92 | 86.8/- | -/80.2
ELECTRAbase | 30 | 86 | 88.8/- | -/80.5
DeBERTabase | 50 | 100 | 88.8/88.5 | 86.2/83.1
DeBERTaV3base | 128 | 86 | 90.6/90.7 | 88.4/85.4
Small models:6 layers,768 hidden size,12 heads
TinyBERTsmall | 30 | 44 | 84.5/- | 77.7/-
MiniLMv2small | 30 | 44 | 87.0/- | 81.6/
BERTsmall | 30 | 44 | 81.8/- | 73.2/-
DeBERTaV3small | 128 | 44 | 88.2/87.9 | 82.9/80.4
XSmall models:12 layers,384 hidden size,6 heads
MiniLMv2xsmall | 30 | 22 | 86.9/- | 82.3/
DeBERTaV3xsmall | 128 | 22 | 88.1/88.3 | 84.8/82.0
Table 5:  Results on MNLI in/out-domain (m/mm) and SQuAD v2.0 development set.
TinyBERTsmall(Jiao et al., 2019), MiniLMv2small and MiniLMv2xsmall models are
pre-trained with knowledge distillation while BERTsmall, DeBERTaV3small and
DeBERTaV3xsmall are trained from scratch with MLM and RTD objective,
respectively.

Compared with smaller size models with similar model structures,
DeBERTaV3small outperforms BERTsmall (Wang et al., 2020b) by a large margin on
those two tasks (i.e., a 6.4% improvement on MNLI-m and a 9.7% F1 score
improvement on SQuAD v2.0). Surprisingly, even though DeBERTaV3xsmall has only
half the parameters of DeBERTaV3small, it performs on par or even better than
DeBERTaV3small on these two tasks. We conjecture that this is due to
DeBERTaV3xsmall has deeper layers which allows to extract better semantic
features. Without knowledge distillation, DeBERTaV3small outperforms
MiniLMv2small (Wang et al., 2020a) by 1.2% and 1.3% on MNLI-m and SQuAD v2.0,
respectively. DeBERTaV3xsmall outperforms MiniLMv2xsmall (Wang et al., 2020a)
by 1.2% and 2.5% on MNLI-m and SQuAD v2.0, respectively. It’s worth noting
that, even though DeBERTaV3xsmall has only 1/4 backbone parameters of
RoBERTabase and XLNetbase, the former significantly outperforms both models on
these two representative tasks (i.e., 0.5% improvement on MNLI-m and 1.5% EM
score improvement on SQuAD v2.0). This further demonstrates the efficiency of
the DeBERTaV3 models.

###  4.2 Multilingual Model

As an important extension, we extend DeBERTaV3 to multi-lingual. We train the
multi-lingual model with the 2.5T CC100 multi-lingual dataset which is the
same as XLM-R. We denote the model as mDeBERTaV3base. We use the same
SentencePiece vocabulary as mT5 which has 250k tokens. The model structure is
the same as our base model, i.e., 768 hidden size, 12 layers, and 12 attention
heads. It is worth noting that, unlike XLM or XLM-E, we have not trained our
model with any parallel data 444we expect to have a better model by
continually pre-training on parallel data and will release a new model when
available. The pre-training settings are similar to XLM-R, except that we only
train the model with 500k steps instead of 1.5M steps.

Model en fr es de el bg ru tr ar vi th zh hi sw ur Avg Cross-lingual transfer
XLM 83.2 76.7 77.7 74.0 72.7 74.1 72.7 68.7 68.6 72.9 68.9 72.5 65.6 58.2 62.4
70.7 mT5base 84.7 79.1 80.3 77.4 77.1 78.6 77.1 72.8 73.3 74.2 73.2 74.1 70.8
69.4 68.3 75.4 XLM-Rbase 85.8 79.7 80.7 78.7 77.5 79.6 78.1 74.2 73.8 76.5
74.6 76.7 72.4 66.5 68.3 76.2 mDeBERTaV3base 88.2 82.6 84.4 82.7 82.3 82.4
80.8 79.5 78.5 78.1 76.4 79.5 75.9 73.9 72.4 79.8 Translate train all XLM 84.5
80.1 81.3 79.3 78.6 79.4 77.5 75.2 75.6 78.3 75.7 78.3 72.1 69.2 67.7 76.9
mT5base 82.0 77.9 79.1 77.7 78.1 78.5 76.5 74.8 74.4 74.5 75.0 76.0 72.2 71.5
70.4 75.9 XLM-Rbase 85.4 81.4 82.2 80.3 80.4 81.3 79.7 78.6 77.3 79.7 77.9
80.2 76.1 73.1 73.0 79.1 mDeBERTaV3base 88.9 84.4 85.3 84.8 84.0 84.5 83.2
82.0 81.6 82.0 79.8 82.6 79.3 77.3 73.6 82.2

Table 6:  Results on XNLI test set under the cross-lingual transfer and the
translate-train-all settings.

As XNLI is one of the major benchmarks to measure multi-lingual model
generalization performance, we evaluate the performance of mDeBERTaV3 on XNLI
across 15 languages. Following previous multi-lingual PLMs, we report both the
zero-shot cross-lingual transfer performance and the translate-train-all
performance. Zero-shot cross-lingual transfer is to fine-tune the model with
English data only and evaluate it on multi-lingual test sets. translate-train-
all is to fine-tune the model with English data and multi-lingual data
translated from English data which is provided together with the XNLI
dataset(Conneau et al., 2018), and then evaluate the model on multi-lingual
test sets. As shown in Table 6, mDeBERTaV3base significantly outperforms
previous SOTA model XLM-Rbase on all languages under both settings. With
regards to the average score, mDeBERTaV3base obtains an improvement of +3.6%
(79.8% v.s. 76.2%) compared with XLM-Rbase in the cross-lingual transfer
setting, as well as achieves an improvement of +3.1% (82.2% v.s. 79.1%)
compared with XLM-Rbase under the translate-train-all setting. These results
clearly show the effectiveness of DeBERTaV3 and the disentanglement is
simultaneously valuable to multi-lingual pre-training.

All this clearly demonstrates the efficiency of the DeBERTaV3 models. The
consistent improvements over a large range of the downstream tasks also show
the huge value of improving pre-trained language models.

##  5 Conclusions

In this paper, we propose a novel pre-training paradigm for language models
based on the combination of DeBERTa and ELECTRA, two state-of-the-art models
that use relative position encoding and replaced token detection (RTD)
respectively. We show that simply combining these two models leads to pre-
training instability and inefficiency, due to a critical interference issue
between the generator and the discriminator in the RTD framework which is well
known as the “tug-of-war” dynamics. To address this issue, we introduce a
novel embedding sharing paradigm called GDES, which is the main innovation and
contribution of this work. GDES allows the discriminator to leverage the
semantic information encoded in the generator’s embedding layer without
interfering with the generator’s gradients and thus improves the pre-training
efficiency. GDES defines a new way of sharing information between the
generator and the discriminator in the RTD framework, which can be easily
applied to other RTD-based language models. We conduct extensive analysis and
experiments to compare GDES with other alternatives to verify its
effectiveness.

Furthermore, we show that DeBERTaV3 with GDES achieves significant
improvements over previous state-of-the-art (SOTA) models on various NLU tasks
that cover different aspects of natural language understanding. For example,
DeBERTaV3Large surpasses other models with a similar architecture by more than
1.37% on the GLUE average score and mDeBERTaV3base beats XLM-Rbase by 3.6% on
the cross lingual transfer accuracy of the XNLI task. These results highlight
the effectiveness of all the DeBERTaV3 models and establish DeBERTaV3 as the
new SOTA pre-trained language models (PLMs) for natural language understanding
at different model scales, i.e., Large, Base, Small and XSmall. Meanwhile,
this work clearly shows huge potential to further improve model’s parameter
efficiency and provide some direction for future studies of far more
parameter-efficient pre-trained language models.

## References

  * Bar-Haim et al. (2006) Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo.  The second PASCAL recognising textual entailment challenge.  In _Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment_ , 01 2006.
  * Bentivogli et al. (2009) Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini.  The fifth pascal recognizing textual entailment challenge.  In _In Proc Text Analysis Conference (TAC’09_ , 2009.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ , 2020.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia.  Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation.  _arXiv preprint arXiv:1708.00055_ , 2017.
  * Chi et al. (2021) Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, and Furu Wei.  Xlm-e: Cross-lingual language model pre-training via electra.  _arXiv preprint arXiv:2106.16138_ , 2021.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning.  ELECTRA: Pre-training text encoders as discriminators rather than generators.  In _ICLR_ , 2020.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov.  Xnli: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 2475–2485, 2018.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 8440–8451, 2020.
  * Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini.  The pascal recognising textual entailment challenge.  In _Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment_ , MLCW’05, Berlin, Heidelberg, 2006.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 2978–2988, 2019.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4171–4186, 2019.
  * Dolan & Brockett (2005) William B Dolan and Chris Brockett.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_ , 2005.
  * Fedus et al. (2021) William Fedus, Barret Zoph, and Noam Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _arXiv preprint arXiv:2101.03961_ , 2021.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al.  Codebert: A pre-trained model for programming and natural languages.  _arXiv preprint arXiv:2002.08155_ , 2020.
  * Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan.  The third PASCAL recognizing textual entailment challenge.  In _Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing_ , pp. 1–9, Prague, June 2007. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/W07-1401.
  * Hadsell et al. (2020) Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu.  Embracing change: Continual learning in deep neural networks.  _Trends in Cognitive Sciences_ , 24(12):1028–1040, 2020.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.  Deberta: Decoding-enhanced bert with disentangled attention.  In _International Conference on Learning Representations_ , 2020.
  * Huang et al. (2018) Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck.  Music transformer: Generating music with long-term structure.  2018\.
  * Jiao et al. (2019) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.  Tinybert: Distilling bert for natural language understanding.  _arXiv preprint arXiv:1909.10351_ , 2019.
  * Kanakarajan et al. (2021) Kamal Raj Kanakarajan, Bhuvana Kundumani, and Malaikannan Sankarasubbu.  Small-bench nlp: Benchmark for small single gpu trained models in natural language processing.  _ArXiv_ , abs/2109.10847, 2021.
  * Kingma & Ba (2014) Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kudo (2018) Taku Kudo.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 66–75, 2018.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy.  Race: Large-scale reading comprehension dataset from examinations.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pp. 785–794, 2017.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations.  In _International Conference on Learning Representations_ , 2019.
  * Levesque et al. (2012) Hector Levesque, Ernest Davis, and Leora Morgenstern.  The winograd schema challenge.  In _Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning_ , 2012.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Loshchilov & Hutter (2018) Ilya Loshchilov and Frank Hutter.  Fixing weight decay regularization in adam.  2018\.
  * Meng et al. (2021) Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song.  Coco-lm: Correcting and contrasting text sequences for language model pretraining.  _arXiv preprint arXiv:2102.08473_ , 2021.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8), 2019.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67, 2020.  URL http://jmlr.org/papers/v21/20-074.html.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , November 2016.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang.  Know what you don’t know: Unanswerable questions for squad.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pp. 784–789, 2018\.
  * Sang & De Meulder (2003) Erik Tjong Kim Sang and Fien De Meulder.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pp. 142–147, 2003.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguistics.  doi: 10.18653/v1/P16-1162.  URL https://aclanthology.org/P16-1162.
  * Shaw et al. (2018) Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani.  Self-attention with relative position representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pp. 464–468, 2018.
  * Shoeybi et al. (2019) Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.  Megatron-lm: Training multi-billion parameter language models using gpu model parallelism.  _arXiv preprint arXiv:1909.08053_ , 2019.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pp. 1631–1642, 2013.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pp. 5998–6008, 2017.
  * Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Advances in neural information processing systems_ , pp. 3266–3280, 2019a.
  * Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _7th International Conference on Learning Representations, ICLR 2019_ , 2019b.
  * Wang et al. (2020a) Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei.  Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers.  _arXiv preprint arXiv:2012.15828_ , 2020a.
  * Wang et al. (2020b) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  _arXiv preprint arXiv:2002.10957_ , 2020b.
  * Warstadt et al. (2018) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman.  Neural network acceptability judgments.  _arXiv preprint arXiv:1805.12471_ , 2018.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pp. 1112–1122. Association for Computational Linguistics, 2018.  URL http://aclweb.org/anthology/N18-1101.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mt5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 483–498, 2021.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in neural information processing systems_ , pp. 5754–5764, 2019.
  * Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi.  Swag: A large-scale adversarial dataset for grounded commonsense inference.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 93–104, 2018.
  * Zhang et al. (2018) Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme.  ReCoRD: Bridging the gap between human and machine commonsense reading comprehension.  _arXiv preprint 1810.12885_ , 2018.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 19–27, 2015.

##  Appendix A Appendix

###  A.1 Dataset

Corpus | Task | #Train | #Dev | #Test | #Label | Metrics
---|---|---|---|---|---|---
General Language Understanding Evaluation (GLUE)
CoLA | Acceptability | 8.5k | 1k | 1k | 2 | Matthews corr
SST | Sentiment | 67k | 872 | 1.8k | 2 | Accuracy
MNLI | NLI | 393k | 20k | 20k | 3 | Accuracy
RTE | NLI | 2.5k | 276 | 3k | 2 | Accuracy
WNLI | NLI | 634 | 71 | 146 | 2 | Accuracy
QQP | Paraphrase | 364k | 40k | 391k | 2 | Accuracy/F1
MRPC | Paraphrase | 3.7k | 408 | 1.7k | 2 | Accuracy/F1
QNLI | QA/NLI | 108k | 5.7k | 5.7k | 2 | Accuracy
STS-B | Similarity | 7k | 1.5k | 1.4k | 1 | Pearson/Spearman corr
Question Answering
SQuAD v1.1 | MRC | 87.6k | 10.5k | 9.5k | - | Exact Match (EM)/F1
SQuAD v2.0 | MRC | 130.3k | 11.9k | 8.9k | - | Exact Match (EM)/F1
ReCoRD | MRC | 101k | 10k | 10k | - | Exact Match (EM)/F1
RACE | MRC | 87,866 | 4,887 | 4,934 | 4 | Accuracy
SWAG | Multiple choice | 73.5k | 20k | 20k | 4 | Accuracy
Token Classification
CoNLL 2003 | NER | 14,987 | 3,466 | 3,684 | 8 | F1
Multi-lingual Natural Language Inference(XNLI)
XNLIcross-lingual | NLI | 393k | 37k | 75k | 3 | Accuracy
XNLItranslate train | NLI | 5.9M | 37k | 75k | 3 | Accuracy
Table 7: Summary information of the NLP application benchmarks.

$\bullet$ GLUE. The General Language Understanding Evaluation (GLUE) benchmark
is a collection of nine natural language understanding (NLU) tasks. As shown
in Table 7, it includes question answering (Rajpurkar et al., 2016),
linguistic acceptability (Warstadt et al., 2018), sentiment analysis (Socher
et al., 2013), text similarity (Cer et al., 2017), paraphrase detection (Dolan
& Brockett, 2005), and natural language inference (NLI) (Dagan et al., 2006;
Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009;
Levesque et al., 2012; Williams et al., 2018). The diversity of the tasks
makes GLUE very suitable for evaluating the generalization and robustness of
NLU models.

$\bullet$ RACE is a large-scale machine reading comprehension dataset
collected from English examinations in China designed for middle school and
high school students (Lai et al., 2017).

$\bullet$ SQuAD v1.1/v2.0 is the Stanford Question Answering Dataset (SQuAD)
v1.1 and v2.0 (Rajpurkar et al., 2016; 2018), two popular machine reading
comprehension benchmarks from approximately 500 Wikipedia articles with
questions and answers obtained by crowdsourcing. The SQuAD v2.0 dataset
includes unanswerable questions about the same paragraphs.

$\bullet$ SWAG is a large-scale adversarial dataset for the task of grounded
commonsense inference, which unifies natural language inference and physically
grounded reasoning (Zellers et al., 2018). SWAG consists of 113k multiple
choice questions about grounded situations.

$\bullet$ CoNLL 2003 (Sang & De Meulder, 2003) is an English dataset
consisting of text from a wide variety of sources. It has 4 types of named
entities.

$\bullet$ XNLI (Conneau et al., 2018) comes with ground truth dev and test
sets in 15 languages, and a ground-truth English training set which is same as
MNLI training set. The training set has been machine-translated to the
remaining 14 languages, providing synthetic training data for these languages
as well.

###  A.2 Pre-training Dataset

For DeBERTaV3 pre-training, we use same data as RoBERTa and DeBERTa1.5B, which
is a combination of Wikipedia, Bookcorpus, CCNews, Stories and OpenWebText.
The multi-lingual version of DeBERTaV3 is trained with 2.5TB CC100 data which
is the same as XLM-R. For pre-training, we also sample 5% of the training data
as the validation set to monitor the training process. Table 8 compares
datasets used in different pre-trained models.

Model | Wiki+Book | OpenWebText | Stories | CC-News | Giga5 | ClueWeb | Common Crawl | CC100
---|---|---|---|---|---|---|---|---
| 16GB | 38GB | 31GB | 76GB | 16GB | 19GB | 110GB | 2.5TB
BERT | ✓ |  |  |  |  |  |  |
XLNet | ✓ |  |  |  | ✓ | ✓ | ✓ |
ELECTRA | ✓ |  |  |  | ✓ | ✓ | ✓ |
RoBERTa | ✓ | ✓ | ✓ | ✓ |  |  |  |
DeBERTa | ✓ | ✓ | ✓ |  |  |  |  |
DeBERTa1.5B | ✓ | ✓ | ✓ | ✓ |  |  |  |
DeBERTaV3 | ✓ | ✓ | ✓ | ✓ |  |  |  |
mDeBERTaV3base |  |  |  |  |  |  |  | ✓

Table 8:  Comparison of the pre-training data.

###  A.3 Generality of GDES

To demonstrate the generality of GDES as an enhancement for RTD, we applied it
to the ELECTRA setting, which we reproduced with a base model of 12 layers for
the discriminator and 6 layers for the generator, both with the same hidden
size. We used wikipedia+bookcorpus data to pre-train the model from scratch
with a learning rate of 5e-4 and a batch size of 2k for 125k steps. We then
evaluated the pre-trained models on MNLI and SQuAD v2.0 using the same setting
as in Table 2. The results in Table 9, show that GDES can also improve the
training efficiency of RTD in this setting, consistent with the findings in
Table 2.

Model | MNLI-m/mm | SQuAD v2.0
---|---|---
Acc | F1/EM
ELECTRAbase | 85.8/- | -/-
ELECTRAbase |  |
\raisebox{-0.9pt}{\small1}⃝ Reimplemented (ES) | 87.9/87.4 | 85.0/82.3
\raisebox{-0.9pt}{\small2}⃝ NES | 86.3/85.6 | 81.7/78.9
\raisebox{-0.9pt}{\small3}⃝ GDES | 88.3/87.8 | 85.9/83.1
Table 9:  Fine-tuning results on MNLI and SQuAD v2.0 tasks of base ELECTRA
models trained with different embedding sharing methods.

###  A.4 Implementation Details

Our pre-training almost follows the same setting as DeBERTa (He et al., 2020).
The generators are trained with MLM where we randomly replace 15% input tokens
with [MASK] tokens. The discriminator is trained with RTD which is the same as
ELECTRA. The experiments in Section 3 are trained using Wikipedia English data
and Bookcorpus data with a batch size of 2k for 125k steps. The experiments in
Section 4 are trained using data listed in Table 8 with a batch size of 8k for
500k steps. We list the detailed hyper parameters of pre-training in Table 10.
For pre-training, we use Adam (Kingma & Ba, 2014) as the optimizer with weight
decay (Loshchilov & Hutter, 2018). For fine-tuning, we use Adam (Kingma & Ba,
2014) as the optimizer for a fair comparison. For fine-tuning, we train each
task with a hyper-parameter search procedure, each run taking about 1-2 hours
on a DGX-2 node. All the hyper-parameters are presented in Table 11. The model
selection is based on the performance on the task-specific development sets.

Our code is implemented based on DeBERTa (He et al.,
2020)555https://github.com/microsoft/DeBERTa and ELECTRA (Clark et al.,
2020)666https://github.com/google-research/electra.

Hyper-parameter | DeBERTaV3large | DeBERTaV3base | DeBERTaV3small | mDeBERTaV3base | DeBERTaV3base-analysis
---|---|---|---|---|---
Number of Layers | 24 | 12 | 6 | 12 | 12
Hidden size | 1024 | 768 | 768 | 768 | 768
FNN inner hidden size | 4096 | 3072 | 3072 | 3072 | 3072
Attention Heads | 12 | 12 | 12 | 12 | 12
Attention Head size | 64 | 64 |  | 64 | 64
Dropout | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
Warmup Steps | 10k | 10k | 10k | 10k | 10k
Learning Rates | 3e-4 | 6e-4 | 6e-4 | 6e-4 | 5e-4
Batch Size | 8k | 8k | 8k | 8k | 2k
Weight Decay | 0.01 | 0.01 | 0.01 | 0.01 | 0.01
Max Steps | 500k | 500k | 500k | 500k | 125k
Learning Rate Decay | Linear | Linear | Linear | Linear | Linear
Adam $\epsilon$ | 1e-6 | 1e-6 | 1e-6 | 1e-6 | 1e-6
Adam $\beta_{1}$ | 0.9 | 0.9 | 0.9 | 0.9 | 0.9
Adam $\beta_{2}$ | 0.98 | 0.98 | 0.98 | 0.98 | 0.999
Gradient Clipping | 1.0 | 1.0 | 1.0 | 1.0 | 1.0

Table 10:  Hyper-parameters for pre-training DeBERTaV3.

Hyper-parameter | DeBERTaV3large | DeBERTaV3base | DeBERTaV3small | mDeBERTaV3base
---|---|---|---|---
Dropout of task layer | {0,0.15,0.3} | {0,0.1,0.15} | {0,0.1,0.15} | {0,0.1,0.15}
Warmup Steps | {50,100,500,1000} | {50,100,500,1000} | {50,100,500,1000} | {50,100,500,1000}
Learning Rates | {5e-6, 8e-6, 9e-6, 1e-5} | {1.5e-5,2e-5, 2.5e-5, 3e-5} | {1.5e-5,2e-5, 3e-5, 4e-5} | {1.5e-5,2e-5, 2.5e-5, 3e-5}
Batch Size | {16,32,64} | {16,32,48,64} | {16,32,48,64} | {16,32,48,64}
Weight Decay | 0.01 | 0.01 | 0.01 | 0.01
Maximun Training Epochs | 10 | 10 | 10 | 10
Learning Rate Decay | Linear | Linear | Linear | Linear
Adam $\epsilon$ | 1e-6 | 1e-6 | 1e-6 | 1e-6
Adam $\beta_{1}$ | 0.9 | 0.9 | 0.9 | 0.9
Adam $\beta_{2}$ | 0.999 | 0.999 | 0.999 | 0.999
Gradient Clipping | 1.0 | 1.0 | 1.0 | 1.0

Table 11:  Hyper-parameters for fine-tuning DeBERTaV3 on down-streaming tasks.


# An explicit upper bound on the number of subgroups of a finite group

Pablo Spiga  Pablo Spiga, Dipartimento di Matematica e Applicazioni,
University of Milano-Bicocca,
Via Cozzi 55, 20125 Milano, Italy<EMAIL_ADDRESS>
###### Abstract.

In this paper we prove that a finite group of order $r$ has at most

$7.3722\cdot r^{\frac{\log_{2}r}{4}+1.5315}$

subgroups.

###### Key words and phrases:

subgroups, bound

###### 2010 Mathematics Subject Classification:

20D99

##  1\. Introduction

Let $R$ be a finite group of cardinality $r$. Since a chain of subgroups of
$R$ has length at most $\log_{2}r$, we deduce that every subgroup $H$ of $R$
has a generating set of cardinality at most $\lfloor\log_{2}r\rfloor$. In
particular, since the generators of $H$ are elements of $R$, we have at most

$r^{\lfloor\log_{2}r\rfloor}\leq r^{\log_{2}r}$

choices for $H$. In other words, $R$ has at most $r^{\log_{2}r}$ subgroups.
This is the typical argument for bounding the number of subgroups of a finite
group $R$.

This bound is not too far off from the best possible. In fact, an elementary
abelian $2$-group of order $r:=2^{a}$ has

$\genfrac{[}{]}{0.0pt}{}{a}{\lfloor\frac{a}{2}\rfloor}_{2}:=\frac{2^{a}-1}{2^{\lfloor\frac{a}{2}\rfloor}-1}\frac{2^{a-1}-1}{2^{\lfloor\frac{a}{2}\rfloor-1}-1}\cdots\frac{2^{a-\lfloor\frac{a}{2}\rfloor+1}-1}{2-1}$

subgroups of order $2^{\lfloor a/2\rfloor}$. Since
$\genfrac{[}{]}{0.0pt}{}{a}{\lfloor\frac{a}{2}\rfloor}_{2}\geq 2^{a^{2}/4}$,
$R$ has at least $2^{a^{2}/4}=r^{\log_{2}r/4}$ subgroups.

Borovik, Pyber and Shalev [3, Corollary $1.6$] have shown that a finite group
of order $r$ has at most

$r^{\log_{2}(r)\cdot\left(\frac{1}{4}+o(1)\right)}$

subgroups. Therefore, in the light of the previous paragraph, this bound is
somehow best possible.

In applications, however, it is sometimes useful to have an explicit upper
bound rather than an asymptotic result. For instance, in proving the Babai-
Godsil conjecture [6] on the asymptotic enumeration of Cayley digraphs, the
authors have used numerous times the trivial bound $r^{\log_{2}r}$ in their
argument. However, there are applications where the naive bound
$r^{\log_{2}r}$ does not suffice and the $1/4$ improvement in the exponent
might give considerable help. For instance, this turns out useful in
investigating the asymptotic enumeration of Haar graphs [8].

The fundamental part of the argument in [3] comes from an estimate on the
number of maximal subgroups of a finite group. The papers [2, 5] do obtain
very useful information on the number of maximal subgroups in a finite group;
however, the implicit constants make it difficult to extract explicit bounds.
Analogously, the bounds in [1] are very explicit, but they do depend on the
number of generators of $R$. Hence, it seems cumbersome to use [1] for
obtaining an explicit upper bound on the number of subgroups of a finite
group.

In this paper we prove the following result.

######  Theorem 1.1.

A finite non-identity group $R$ has at most
$7.3722\cdot|R|^{\frac{\log_{2}|R|}{4}+1.5315}$ subgroups.

Our group theoretic argument is entirely elementary and we give a sketch of
the proof in Section 2. This part is essentially as in [3]. However, rather
than reducing to certain maximal subgroups, we make some detailed arithmetic
considerations that allow us to prove Theorem 1.1. We do believe that similar
considerations can prove the upper bound $7.3722\cdot|R|^{\log_{2}|R|}$;
however, this seems to require (at least with our method) some very long
arithmetic arguments.

##  2\. A group theoretic argument

In this section we give a sketch of our proof of Theorem 1.1.

###### Sketch of the proof of Theorem 1.1: part I.

Let $R$ be a finite group and let $r$ be the order of $R$. Without loss of
generality, we may suppose that $r\geq 2$. Now, we factorize

$r=\prod_{i=1}^{\ell}p_{i}^{a_{i}}$

into prime factors; in particular, $p_{1},\ldots,p_{\ell}$ are distinct primes
and $a_{i}\geq 1$ for each $i\in\\{1,\ldots,\ell\\}$. Relabeling the indexed
set if necessary, we may suppose that $p_{1}<\cdots<p_{\ell}$.

Let $H$ be a subgroup of $R$. Then $H$ is uniquely determined by a family
$(Q_{i})_{i}$ of Sylow $p_{i}$-subgroups of $H$, for each
$i\in\\{1,\ldots,\ell\\}$. Each of these subgroups $Q_{i}$ is contained in a
Sylow $p_{i}$-subgroup $P_{i}$ of $R$. Now, from Sylow’s theorem, all Sylow
$p_{i}$-subgroups of $R$ are conjugate and hence $R$ has at most
$r/p_{i}^{a_{i}}$ Sylow $p_{i}$-subgroups. Therefore, we have at most

$\prod_{i=1}^{\ell}\frac{r}{p_{i}^{a_{i}}}=r^{\ell-1}$

choices for the $\ell$-tuple $(P_{i})_{i}$.

In (3.2) we define a function $S(p,a)$ such that every $p$-group of order
$p^{a}$ has at most $S(p,a)$ subgroups, see Remark 3.1. At this point, let us
ignore what this function is and let us see what we may deduce. When
$(P_{i})_{i}$ is given, since $Q_{i}$ is a subgroup of $P_{i}$ and since
$P_{i}$ is a $p_{i}$-group, from the previous paragraph, we see that we have
at most

$\prod_{i=1}^{\ell}S(p_{i},a_{i})$

choices for the $\ell$-tuple $(Q_{i})_{i}$.

From above, $R$ has at most

(2.1) $\displaystyle
r^{\ell-1}\cdot\prod_{i=1}^{\ell}S(p_{i},a_{i})=r^{-1}\prod_{i=1}^{\ell}r\cdot
S(p_{i},a_{i})$

subgroups.

The bulk of the argument in Section 3 is proving that

(2.2) $rS(p_{i},a_{i})\leq p_{i}^{a_{i}\frac{\log_{2}r}{4}},$

unless $p_{i}\leq 23$. Actually, Section 3 proves much more than that and
hence there is room for hoping for an improvement on $7.3722\cdot
r^{\log_{2}r/4+1.5315}$. For instance, in the particular case that $p_{1}>23$,
from (2.1) and (2.2) we deduce the stronger upper bound

$r^{-1}\prod_{i=1}^{\ell}rS(p_{i},a)\leq
r^{-1}\prod_{i=1}^{\ell}p_{i}^{a_{i}\frac{\log_{2}r}{4}}=r^{\frac{\log_{2}r}{4}-1}.$

Our weaker bound $7.3722\cdot r^{\frac{\log_{2}r}{4}+1.5315}$ arises from
dealing with small primes in the factorization of $r$.

We postpone the rest of the proof after Section 3. ∎

##  3\. Arithmetical observations

For each prime number $p$, we let

(3.1) $\displaystyle C(p)$ $\displaystyle:=\prod_{i\geq
1}\frac{1}{1-\frac{1}{p^{i}}},$ $\displaystyle c(p)$
$\displaystyle:=2.129\cdot C(p).$

Now, let $p$ be a prime number and let $a$ be a positive integer. We define

(3.2) $S(p,a):=\begin{cases}2&\textrm{when }a:=1,\\\ p+3&\textrm{when
}a:=2,\\\ 2p^{2}+2p+4&\textrm{when }a:=3,\\\
p^{4}+3p^{3}+4p^{2}+3p+5&\textrm{when }a:=4,\\\
2p^{6}+2p^{5}+6p^{4}+6p^{3}+6p^{2}+4p+6&\textrm{when }a:=5,\\\
c(p)p^{\frac{a^{2}}{2}}&\textrm{when }a\geq 6.\end{cases}$

Strictly speaking we give more details than it is barely necessary for the
proof of Theorem 1.1. We hope that this information can be used in the future
for improving the bound in Theorem 1.1.

######  Remark 3.1.

Let $P$ be a $p$-group of order $p^{a}$. We observe here that $P$ has at most
$S(p,a)$ subgroups, where $S(p,a)$ is defined in (3.2).

Let $k\in\\{0,\ldots,a\\}$. Corollary $4.2$ in [7] shows that the number of
subgroups of $P$ having index $p^{k}$ is at most

$\genfrac{[}{]}{0.0pt}{}{a}{k}_{p}\leq C(p)p^{a(n-k)},$

where $C(p)$ is defined in (3.1). In particular, the number of subgroups of
$P$ is at most

$\sum_{k=0}^{a}\genfrac{[}{]}{0.0pt}{}{a}{k}_{p}.$

When $a\leq 5$, we see with a computation that this summation is exactly
$S(p,a)$. For instance, when $a=4$, we have

$\displaystyle\genfrac{[}{]}{0.0pt}{}{a}{0}_{p}+\genfrac{[}{]}{0.0pt}{}{a}{1}_{p}+\genfrac{[}{]}{0.0pt}{}{a}{2}_{p}+\genfrac{[}{]}{0.0pt}{}{a}{3}_{p}+\genfrac{[}{]}{0.0pt}{}{a}{4}_{p}$
$\displaystyle=1+\frac{p^{4}-1}{p-1}+\frac{(p^{4}-1)(p^{3}-1)}{(p^{2}-1)(p-1)}+\frac{p^{4}-1}{p-1}+1$
$\displaystyle=2(p^{3}+p^{2}+p+1+2)+p^{4}+p^{3}+2p^{2}+p+1$
$\displaystyle=S(p,a).$

Assume now that $a\geq 6$.

Suppose that $a$ is even. Then the number of subgroups of $R$ is at most

$\displaystyle C(p)\sum_{k=0}^{a}p^{k(a-k)}$
$\displaystyle=C(p)\cdot\left(p^{\frac{a^{2}}{4}}+2\sum_{k=0}^{\frac{a}{2}-1}p^{k(a-k)}\right)=C(p)p^{\frac{a^{2}}{4}}\left(1+2\sum_{k=1}^{\frac{a}{2}}\frac{1}{p^{k^{2}}}\right)$
$\displaystyle\leq
C(p)p^{\frac{a^{2}}{4}}\left(-1+2\sum_{k=0}^{\infty}\frac{1}{p^{k^{2}}}\right)\leq
2.129C(p)p^{\frac{a^{2}}{4}}=S(p,a),$

where the value $2.129$ is obtained by taking $p:=2$ in the infinite sum.
Suppose that $a$ is odd. Then the number of subgroups of $R$ is at most

$\displaystyle C(p)\sum_{k=0}^{a}p^{k(a-k)}$ $\displaystyle=C(p)\cdot
2\sum_{k=0}^{\frac{a-1}{2}}p^{k(a-k)}=C(p)\cdot
2p^{\frac{a^{2}-1}{4}}\sum_{k=0}^{\frac{a-1}{2}}\frac{1}{p^{k(k+1)}}$
$\displaystyle\leq C(p)\cdot
2p^{\frac{a^{2}-1}{4}}\sum_{k=0}^{\infty}\frac{1}{p^{k(k+1)}}\leq
2.53175C(p)p^{\frac{a^{2}-1}{4}}$ $\displaystyle\leq
2.129C(p)p^{\frac{a^{2}}{4}}=S(p,a),$

where the value $2.53175$ is obtained by taking $p:=2$ in the infinite sum and
where $2.129$ is obtained by multiplying $2.53175$ with $2^{-1/4}$. Summing
up, regardless of whether $a$ is odd or even, $R$ has at most $S(p,a)$
subgroups.

In this section we prove various upper bounds on $S(p,a)$. We give here the
first lemma that in part explains the role of $7.3722\cdot$ in the upper bound
in Theorem 1.1.

######  Lemma 3.2.

Let $p$ be a prime number, let $a$ be a positive integer and let $r$ be a
multiple of $p^{a}$. Then $S(p,a)\leq 7.3722\cdot p^{a\frac{\log_{2}r}{4}}$.

###### Proof.

Since $r\to\log_{2}r$ is monotone increasing, we may suppose that $r=p^{a}$.
In particular, $7.3722\cdot p^{a\frac{\log_{2}r}{4}}=7.3722\cdot
p^{\frac{a^{2}}{4}\log_{2}p}$. Now the proof follows by distinguishing various
possibilities for $a$. When $a=1$, we have $S(p,1)=2$ and

$7.3722\cdot p^{\frac{a^{2}}{4}\log_{2}p}=7.3722\cdot
p^{\frac{\log_{2}p}{4}}\geq 7.3722\cdot 2^{1/4}=9.5136.$

When $a=2$, $S(p,2)=p+3$ and

$7.3722\cdot p^{\frac{a^{2}}{4}\log_{2}p}=7.3722\cdot p^{\log_{2}p}\geq
7.3722\cdot p;$

clearly, $p+3\geq 7.3722\cdot p$. The cases $a\in\\{3,4,5\\}$ are entirely
similar.

Suppose $a\geq 6$. Now, $c(p)p^{\frac{a^{2}}{4}}=S(p,a)\leq 7.3722\cdot
p^{\frac{a^{2}}{4}}$ if and only if $c(p)\leq 7.3722$. From (3.1), $p\mapsto
c(p)$ is a monotone decreasing function and hence $c(p)\leq
c(2)=7.372187<7.3722$. ∎

###  3.1. Dealing with one prime

######  Lemma 3.3.

Let $p$ be a prime number and let $r$ be a positive multiple of $p$ with
$\gcd(p,r/p)=1$. Then $r\cdot S(p,1)\leq p^{\frac{\log_{2}r}{4}}$ unless one
of the following holds

  1. (1)

$p=23$ and $1\leq r/p\leq 8$,

  2. (2)

$p=19$ and $1\leq r/p\leq 3\,784$,

  3. (3)

$p\leq 17$.

###### Proof.

Here $S(p,1)=2$. The proof follows from easy computations, assisted with the
computer. When $p\geq 29$, it can be verified that $p^{\log_{2}(p)/4}>2p$ and
$p^{1/4}>2$. Therefore

$\displaystyle p^{\frac{\log_{2}r}{4}}$
$\displaystyle=p^{\frac{\log_{2}p}{4}}\cdot
p^{\frac{\log_{2}(r/p)}{4}}>2p\cdot p^{\frac{\log_{2}(r/p)}{4}}$
$\displaystyle>2p\cdot 2^{\log_{2}(r/p)}=2p\cdot(r/p)=2r=rS(p,1).$

Suppose now $p<29$. If $p\leq 17$, then we obtain part (3). If $p>17$, then
$p\in\\{19,23\\}$ and parts (1) and (2) follow with a computer assisted
computation. ∎

######  Lemma 3.4.

Let $p$ be a prime number and let $r$ be a positive multiple of $p^{2}$ with
$\gcd(p,r/p^{2})=1$. Then $r\cdot S(p,2)\leq p^{2\frac{\log_{2}r}{4}}$ unless
one of the following holds

  1. (1)

$p=7$ and $1\leq r/p^{2}\leq 6$,

  2. (2)

$p=5$ and $1\leq r/p^{2}\leq 16\,314$,

  3. (3)

$p\in\\{2,3\\}$.

###### Proof.

Here $S(p,2)=p+3$. The proof is very similar to the proof of Lemma 3.3 and it
basically follows from straightforward computations. When $p\geq 19$, it can
be verified that $p^{\log_{2}(p)/2}>(p+3)p$ and $p^{1/2}>2$. Therefore

$\displaystyle p^{2\frac{\log_{2}r}{4}}$
$\displaystyle=p^{\frac{\log_{2}p}{2}}\cdot
p^{\frac{\log_{2}(r/p)}{2}}>(p+3)p\cdot p^{\frac{\log_{2}(r/p)}{2}}$
$\displaystyle>(p+3)p\cdot 2^{\log_{2}(r/p)}=(p+3)p\cdot(r/p)=rS(p,2).$

Suppose now $p<19$. If $p\leq 3$, then we obtain part (3). If $p>3$, then
$p\in\\{5,7,11,13\\}$ and part (1) and (2) follow with computer assisted
computations by dealing with each case at the time. ∎

######  Lemma 3.5.

Let $p$ be a prime number and let $r$ be a positive multiple of $p^{3}$ with
$\gcd(p,r/p^{3})=1$ and $r/p^{3}>1$. Then $r\cdot S(p,3)\leq
p^{3\frac{\log_{2}r}{4}}$ unless one of the following holds

  1. (1)

$p=5$ and $1\leq r/p^{3}\leq 2$,

  2. (2)

$p\in\\{2,3\\}$.

###### Proof.

The proof is very similar to the proof of Lemmas 3.3 and 3.4 and we omit it. ∎

######  Lemma 3.6.

Let $p$ be a prime number and let $r$ be a positive multiple of $p^{4}$ with
$\gcd(p,r/p^{4})=1$. Then $r\cdot S(p,4)\leq p^{4\frac{\log_{2}r}{4}}$ unless
one of the following holds

  1. (1)

$p=3$ and $1\leq r/p^{4}\leq 116$,

  2. (2)

$p=2$.

###### Proof.

The proof is omitted. ∎

######  Lemma 3.7.

Let $p$ be a prime number and let $r$ be a positive multiple of $p^{5}$ with
$\gcd(p,r/p^{5})=1$. Then $r\cdot S(p,5)\leq p^{5\frac{\log_{2}r}{4}}$ unless
one of the following holds

  1. (1)

$p=3$ and $1\leq r/p^{5}\leq 11$,

  2. (2)

$p=2$.

###### Proof.

The proof is omitted. ∎

######  Lemma 3.8.

Let $p$ be a prime number, let $a\geq 6$ be an integer and let $r$ be a
positive multiple of $p^{a}$ with $\gcd(p,r/p^{a})=1$. Then $r\cdot S(p,a)\leq
p^{a\frac{\log_{2}r}{4}}$ unless one of the following holds

  1. (1)

$p=3$ and $r\in\\{729,1\,458,2\,187,2\,916\\}$,

  2. (2)

$p=2$.

###### Proof.

The proof is omitted. ∎

######  Corollary 3.9.

Let $R$ be a finite group having order $r$ divisible by $p^{a}$, where $p$ is
a prime number, $a$ is a positive integer and $\gcd(r/p^{a},p)=1$. Then either
$rS(p,a)\leq p^{a\frac{\log_{2}r}{4}}$ or one of the following holds

  1. (1)

$R$ satisfies Theorem 1.1,

  2. (2)

$p\in\\{5,7,11,13,17\\}$ and $a=1$,

  3. (3)

$p=3$ and $a\leq 3$,

  4. (4)

$p=2$.

###### Proof.

Suppose that $rS(p,a)>p^{a\frac{\log_{2}r}{4}}$. In particular, $(p,a)$
satisfies the conclusions in Lemmas 3.3–3.8. We consider in turn each of these
cases.

When $a=1$, Lemma 3.3 holds. In particular, we only need to deal with part (1)
and (2) of Lemma 3.3, because when $p\leq 17$ we see that parts (2)–(4) are
satisfied. In particular, we only have a finite number of cases to consider.
Let $r=p_{1}^{a_{1}}\cdots p_{\ell}^{a_{\ell}}$ be the factorization of $r$
into distinct prime powers. Let us consider the function

$f(r):=r^{\ell-1}\prod_{i=1}^{\ell}S(p_{i},a_{i}).$

By Section 2, if $f(r)\leq 7.3722\cdot r^{\log_{2}r/4+1.5315}$, then Theorem
1.1 holds and hence part (1) is satisfied. Therefore, we may suppose that
$f(r)>7.3722\cdot r^{\log_{2}r/4+1.5315}$. We have implemented this function
in a computer and we have checked that no $r$ in the range described in Lemma
3.3 parts (1) and (2) satisfies $f(r)>7.3722r^{\log_{2}r/4+1.5315}$.

When $a=2$, Lemma 3.4 holds. In particular, we only need to deal with part (1)
and (2) of Lemma 3.4, because when $p\leq 3$ we see that parts (2)–(4) are
satisfied. In particular, we only have a finite number of cases to consider.
We have checked that no $r$ in the range described in Lemma 3.4 parts (1) and
(2) satisfies $f(r)>7.3722\cdot r^{\log_{2}r/4+1.5315}$.

When $a=3$, Lemma 3.5 holds. In particular, we only need to deal with part (1)
of Lemma 3.5, because when $p\leq 3$ we see that parts (3)–(4) are satisfied.
In particular, we only have a finite number of cases to consider. We have
checked that no $r$ in the range described in Lemma 3.5 part (1) satisfies
$f(r)>7.3722\cdot r^{\log_{2}r/4+1.5315}$.

Finally, the cases $a\geq 4$ are analogous. ∎

##  4\. Proof of Theorem 1.1

In this section, we complete the proof of Theorem 1.1 that we have begun in
Section 2. We argue by induction on $r$. Write $\varepsilon:=1.5315$.

Let $\mathcal{I}$ be the collection of indices $i\in\\{1,\ldots,\ell\\}$ with
$rS(p_{i},a_{i})>p_{i}^{a_{i}\log_{2}r/4}$ and let $\mathcal{P}:=\\{p_{i}\mid
i\in\mathcal{I}\\}$. From Lemmas 3.3–3.8, we have
$\mathcal{P}\subseteq\\{2,3,5,7,11,13,17,19,23\\}$. Actually, from Corollary
3.9, either Theorem 1.1 holds or $\mathcal{P}\subseteq\\{2,3,5,7,11,13,17\\}$.
Moreover, if $i\in\mathcal{I}$ with $p_{i}\in\\{5,7,11,13,17\\}$, then
$a_{i}=1$.

Assume there exists $i\in\mathcal{I}$ with $p_{i}\in\\{5,7,11,13,17\\}$ or
with $p_{i}=3$ and $a_{i}=2$ such that ${\bf N}_{R}(P_{i})={\bf
C}_{R}(P_{i})$. Let $P_{i}$ be a Sylow $p_{i}$-subgroup of $R$ and observe
that $P_{i}$ is cyclic of prime order when $p_{i}>3$ and $P_{i}$ is abelian
when $p_{i}=3$. Then, from the Burnside $p$-complement theorem, $R_{i}$
contains a normal subgroup $N$ with $R=NP_{i}$ and $N\cap P_{i}=1$. Thus $R$
is the semidirect product of $N$ with $P_{i}$. For the moment, let us suppose
that $(p_{i},a_{i})\neq(3,2)$ and we do come back to this case later. Let $x$
be the number of subgroups of $N$. We claim that $R$ has at most
$x(1+r/p_{i})$ subgroups. Indeed, the number of subgroups of $R$ contained in
$N$ is $x$ and, if $H$ is a subgroup of $R$ not contained in $N$, then
$H=\langle K,P_{i}^{g}\rangle$ for some subgroup $K$ of $N$ and for some $g\in
R$. Observe that $g$ can be chosen in a transversal of $P_{i}$ in $R$ and
hence we have $r/p_{i}$ choices for $g$. Therefore the number of subgroups of
$R$ is at most

$\displaystyle x\left(1+\frac{r}{p_{i}}\right)$ $\displaystyle\leq 2\cdot
7.3722\cdot\left(\frac{r}{p_{i}}\right)^{\frac{\log_{2}(r/p_{i})}{4}+\varepsilon}\left(1+\frac{r}{p_{i}}\right).$

Moreover,

$\displaystyle r^{\frac{\log_{2}r}{4}+\varepsilon}$
$\displaystyle=\left(\frac{r}{p_{i}}\right)^{\frac{\log_{2}r}{4}+\varepsilon}p_{i}^{\frac{\log_{2}r}{4}+\varepsilon}=\left(\frac{r}{p_{i}}\right)^{\frac{\log_{2}(r/p_{i})}{4}+\varepsilon}\left(\frac{r}{p_{i}}\right)^{\frac{\log_{2}p_{i}}{4}}p_{i}^{\frac{\log_{2}r}{4}+\varepsilon}$
$\displaystyle=\left(\frac{r}{p_{i}}\right)^{\frac{\log_{2}(r/p_{i})}{4}+\varepsilon}\left(\frac{r}{p_{i}}\right)^{\frac{\log_{2}p_{i}}{4}}r^{\frac{\log_{2}p_{i}}{4}}p_{i}^{\varepsilon}$
$\displaystyle=\left(\frac{r}{p_{i}}\right)^{\frac{\log_{2}(r/p_{i})}{4}+\varepsilon}r^{\frac{\log_{2}p_{i}}{2}}p_{i}^{\varepsilon-\frac{\log_{2}p_{i}}{4}}.$

When $p_{i}\geq 5$, going through the various possibilities for $p_{i}$, it is
not hard to verify that $r^{\log_{2}p_{i}/2}>r$ and
$p^{\varepsilon-\log_{2}p_{i}/4}>2$ and hence

$r^{\frac{\log_{2}p_{i}}{2}}p_{i}^{\varepsilon-\frac{\log_{2}p_{i}}{4}}>1+\frac{r}{p_{i}}$

and the theorem follows in this case. When $(p_{i},a_{i})=(3,2)$, $P_{i}$ is
no longer cyclic of prime order. However, $P_{i}$ has at most $5$ non-identity
subgroups. Thus arguing as above, we deduce that the number of subgroups of
$R$ is at most $x(1+5\cdot r/9)$. Now, we may repeat the computations above
(with minor modifications) and we obtain that the theorem follows.

For the rest of the argument we may suppose that, for every $i\in\mathcal{I}$
with $p_{i}\in\\{3,5,7,11,13,17\\}$, either ${\bf N}_{R}(P_{i})>{\bf
C}_{R}(P_{i})$ or $(p_{i},a_{i})\in\\{(3,1),(3,3)\\}$. When ${\bf
N}_{R}(P_{i})>P_{i}$, the number of Sylow $p_{i}$-subgroups of $R$ is at most
$|R|/2p_{i}^{a_{i}}$. Let

$\displaystyle\mathcal{J}$ $\displaystyle:=\\{i\in\mathcal{I}\mid
p_{i}\in\\{5,7,11,13,17\\}\\},$ $\displaystyle\mathcal{J}^{\prime}$
$\displaystyle:=\\{i\in\mathcal{I}\mid p_{i}\in\\{2,3\\}\\}.$

In the particular case that there exists $i\in\mathcal{I}$ with $p_{i}=3$ and
$a_{i}=2$, we do include the index $i$ in $\mathcal{J}$ and remove it from
$\mathcal{J}^{\prime}$.

With this slight improvement and with this notation, we may go back to (2.1)
and deduce that $R$ has at most

(4.1) $r^{-1}\prod_{i\in\mathcal{J}^{\prime}}r\cdot
S(p_{i},a_{i})\prod_{i\in\mathcal{J}}\frac{r}{2}\cdot
S(p_{i},a_{i})\prod_{\begin{subarray}{c}i=1\\\
i\notin\mathcal{I}\end{subarray}}^{\ell}r\cdot S(p_{i},a_{i})$

subgroups. Let us call $A$ this product.

Let $i\in\mathcal{J}$ with $a_{i}=1$. Observe that here we are only excluding
the possibility that $p_{i}=3$ and $a_{i}=2$. We have

$\displaystyle\frac{r}{2}\cdot S(p_{i},a_{i})$ $\displaystyle=\frac{r}{2}\cdot
2=r=r^{1-\frac{\log_{2}p_{i}}{4}}r^{\frac{\log_{2}p_{i}}{4}}=r^{1-\frac{\log_{2}p_{i}}{4}}p_{i}^{a_{i}\frac{\log_{2}r}{4}}.$

In the case that $p_{i}=3$ and $a_{i}=2$, we have

$\displaystyle\frac{r}{2}\cdot S(3,2)$ $\displaystyle=\frac{r}{2}\cdot
6=3r=3r^{1-2\frac{\log_{2}p_{i}}{4}}r^{2\frac{\log_{2}p_{i}}{4}}=3r^{1-2\frac{\log_{2}p_{i}}{4}}p_{i}^{a_{i}\frac{\log_{2}r}{4}}$
$\displaystyle\leq
r^{1-\frac{\log_{2}p_{i}}{4}}p_{i}^{a_{i}\frac{\log_{2}r}{4}}.$

The last inequality follows from a computation and is only valid when $r\geq
16$; however, when $r<16$, the veracity of Theorem 1.1 can be easily checked
with a direct inspection.

From the previous paragraph and (4.1), we obtain

$\displaystyle A$ $\displaystyle\leq
r^{-1}\prod_{i\in\mathcal{J}^{\prime}}S(p_{i},a_{i})\prod_{i\in\mathcal{J}}r^{1-\frac{\log_{2}p_{i}}{4}}p_{i}^{a_{i}\frac{\log_{2}r}{4}}\prod_{i\notin\mathcal{I}}p_{i}^{a_{i}\frac{\log_{2}r}{4}}$
$\displaystyle=r^{-1}\prod_{i\in\mathcal{J}^{\prime}}rS(p_{i},a_{i})\cdot
r^{\sum_{j\in\mathcal{J}}1-\frac{\log_{2}p_{i}}{4}}\cdot\left(\prod_{i\notin\mathcal{J}^{\prime}}p_{i}^{a_{i}}\right)^{\frac{\log_{2}r}{4}}.$

The maximum of $\sum_{i\in\mathcal{J}}1-\log_{2}(p_{i})/4$ is
$1.5315=\varepsilon$ and is obtained when $\\{p_{i}\mid
i\in\mathcal{J}\\}=\\{3,5,7,11,13\\}$.

If $\mathcal{J}^{\prime}=\emptyset$, then

$A\leq
r^{\frac{\log_{2}r}{4}-1+\sum_{j\in\mathcal{J}}1-\frac{\log_{2}p_{i}}{4}}\leq
r^{\frac{\log_{2}r}{4}-1+\varepsilon}.$

Assume $|\mathcal{J}^{\prime}|=1$. Let $i\in\mathcal{J}^{\prime}$. By Lemma
3.2, we have $S(p_{i},a_{i})\leq 7.3722\cdot p_{i}^{a_{i}\log_{2}r/4}$ and
hence

$A\leq 7.3722\cdot
r^{\frac{\log_{2}r}{4}+\sum_{j\in\mathcal{J}}1-\frac{\log_{2}p_{i}}{4}}\leq
7.3722r^{\frac{\log_{2}r}{4}+\varepsilon}.$

Finally assume $|\mathcal{J}^{\prime}|=2$. Thus
$\mathcal{J}^{\prime}=\\{1,2\\}$, $p_{1}=2$ and $p_{2}=3$. Recall that
$a_{2}\in\\{1,3\\}$. Now, since the index corresponding to the prime $3$ is
not in $\mathcal{J}$, the maximum of
$\sum_{i\in\mathcal{J}}1-\log_{2}(p_{i})/4$ is $0.9278$ and is obtained when
$\\{p_{i}\mid i\in\mathcal{J}\\}=\\{5,7,11,13\\}$. When $a_{2}=3$, it can be
verified that

$rS(3,a_{i})\leq r^{\varepsilon-0.9278}\cdot 3^{a_{i}\frac{\log_{2}r}{4}},$

for every $r\geq 68$. Therefore, when $r\geq 68$, using (4.1), we get

$A\leq 7.3722\cdot r^{\frac{\log_{2}r}{4}+\varepsilon}.$

The veracity of Theorem 1.1 for smaller values can be checked with a computer.

Finally suppose $a_{2}=1$. When ${\bf N}_{R}(P_{2})>P_{2}$, we may refine the
factor $rS(3,a_{1})=2r$ in (4.1) with simply $r$. Now

$r=r^{1-\log_{2}(3)}\cdot 3^{a_{i}\frac{\log_{2}r}{4}}.$

Therefore, using (4.1), we get again

$A\leq 7.3722\cdot r^{\frac{\log_{2}r}{4}+\varepsilon}.$

Assume then ${\bf N}_{R}(P_{2})=P_{2}$. From Burnside $p$-complement theorem,
there exists a normal subgroup $N$ of $R$ with $R=NP_{i}$ and $N\cap P_{i}=1$.
As $N$ has order relatively prime to $N$, by applying the argument above to
$N$, we deduce that $N$ has at most

$7.3722\cdot\left(\frac{r}{3}\right)^{\frac{\log_{2}(r/3)}{4}+0.9278}$

subgroups. Now $R$ has at most

$7.3722\cdot\left(\frac{r}{3}\right)^{\frac{\log_{2}(r/3)}{4}+0.9278}\left(1+\frac{r}{3}\right)$

subgroups. It is not hard to verify that this number is at most $7.3722\cdot
r^{\log_{2}r/4+\varepsilon}$.

## References

  * [1] A. Ballester-Bolinches, R. Esteban-Romero, P. Jimènez-Seral, Bounds on the Number of Maximal Subgroups of Finite Groups: Applications, Mathematics 10 (2022), 1–25.
  * [2] A. V. Borovik, On the Number of Maximal Soluble Subgroups of a Finite Group, Comm. Algebra 26, 4041–4050.
  * [3] A. V. Borovik, L. Pyber, A. Shalev, Maximal subgroups in finite and profinite groups, Trans. Amer. Math. Soc. 348 (1996), 3745–3761.
  * [4] W. Bosma, C. Cannon, C. Playoust, The MAGMA algebra system I: The user language, J. Symbolic Comput. 24 (1997), 235–265.
  * [5] M. W. Liebeck, L. Pyber, A. Shalev, On a conjecture of G. E. Wall, J. Algebra 317 (2007), 184–197.
  * [6] J. Morris, P. Spiga, Asymptotic enumeration of Cayley digraphs, Israel J. Math. 242 (2021), 401–459.
  * [7] A. Shalev, Growth functions, p-adic analytic groups, and groups of finite coclass, J. London Math. Soc. (2) 46 (1992), 111–122.
  * [8] P. Spiga, Bipartite and Haar graphical representations of finite groups and their asymptotic enumeration, in preparation.