11institutetext: Pioneer Alpha, 11email<EMAIL_ADDRESS>22institutetext: Mississippi State University, USA

# Bangla Image Caption Generation through CNN-Transformer based Encoder-
Decoder Network

Md Aminul Haque Palash 11 0000-0002-4794-8531 MD Abdullah Al Nasim 11
0000-0001-5594-607X Sourav Saha 11 0000-0003-0483-0581 Faria Afrin 11
0000-0001-5603-8647 Raisa Mallik 11 0000-0002-0043-8649 Sathishkumar Samiappan
22 0000-0002-8443-883X

###### Abstract

Automatic Image Captioning is the never-ending effort of creating
syntactically and validating the accuracy of textual descriptions of an image
in natural language with context. The encoder-decoder structure used
throughout existing Bengali Image Captioning (BIC) research utilized abstract
image feature vectors as the encoder’s input. We propose a novel transformer-
based architecture with an attention mechanism with a pre-trained ResNet-101
model image encoder for feature extraction from images. Experiments
demonstrate that the language decoder in our technique captures fine-grained
information in the caption and, then paired with image features, produces
accurate and diverse captions on the BanglaLekhaImageCaptions dataset. Our
approach outperforms all existing Bengali Image Captioning work and sets a new
benchmark by scoring 0.694 on BLEU-1, 0.630 on BLEU-2, 0.582 on BLEU-3, and
0.337 on METEOR.

###### Keywords:

Bengali Image Captioning ResNet-101 Transformer.

##  1 Introduction

Depiction aims to describe an image’s substance in text. Plenty of the
captioning algorithms now follow an architecture of an encoder-decoder where a
decoder infrastructure may anticipate words using the function retrieved via a
method of attention from the encoder network. Image subtitling in studies has
been focused on a translation technique, including a visual encoder as well as
a language decoder [13]. The method of producing picture titles may be used
for automating driving automobiles, implementing face recognition systems,
describing persons with visual impairment, improving the quality of the photo
query, etc. At the junction of computer vision and natural language
interpretation, the challenge of producing natural language descriptions of
the data of an image resides. In some domains, for example, image search,
picture captioning has already had substantial implications. It may have a
positive impact in many fields, including disability software, video
surveillance, and security, the interaction between humans and computers.

Image captioning seems to be the function in the natural language to describe
the visual data. An algorithm is required to comprehend and represent the
links underlying visual and textual information, and to create a series of the
text output. The most important studies are based on the encoder-decoding
framework [13] which is quite close to the machine transcription sequence-to-
sequence paradigm. The architecture of merge or mixing, which is mostly
utilized for linguistic encoding. This same coded language information and
visual characterize complete transformer connectivity in the Caption
Transformer (CPTR) [13] replaces a CNN with a fully convolution-free
transformer encoder throughout the encoder portion. We directly sequence raw
pictures as an input when compared to the traditional subtitled models that
take the feature derived by CNN or feature detector as the source. Influenced
by the studies above, we explore a new sequence-to-sequence approach for the
resolution of the picture subtitling issue. Statistics are then fused into one
multi-modal layer which predicts the term of the title by term [2].

Open-domain submission seems to be a very difficult process, for it demands a
thorough knowledge of the global as well as local image domains. A new
operating system to evaluate image subscribing systems is a recent release of
the MSCOCO project. Deep sequence model methodologies have produced amazing
results. It was demonstrated recently that exposure differentiation and non-
differentiable task metric questions may be solved by the incorporation of
reinforcement learning approaches. Designers propose a novel, basic
transformer network design, based only on principles of attention. At the WMT
2014, BLEU will be delivered to the model [11]. Elements and other prominent
areas of the image are a lot better natural foundation of attention to create
more human-like descriptions and inquiry responses. Researchers offer a joint
visual recognition mechanism from the bottom up to the top down. The findings
on the MSCOCO [11] production servers provide a new cutting edge for the
captioning challenge. To summarize, the paper will contribute to these factors
proceeding ahead:

  * •

We propose a novel architecture of transformer base which will be implemented
for image captioning, with attention mechanism. It is tailored to the complex
structure of the whole image regions [11].

  * •

The model architecture will be designed using the ResNet-101 [8] for feature
extraction from images.

  * •

We will perform the image depiction by using the train set and validation set
of the BanglaLekhaImageCaptions dataset [14].

  * •

On the BanglaLekhaImageCaptions dataset, we evaluate our approach both
qualitatively and quantitatively.

##  2 Related Works

A rudimentary method to creating the image title may be to anticipate and
connect words from image areas. Researchers built one of the earliest systems
of picture subtitling in 1999. The underlying job of picture translation was
subsequently retranslated in 2002. However, for various reasons, it has been
shown out that the approach fails to correctly map an image to a word that
entirely misses all kinds of interactions between the elements. Earlier
subtitles based on basic templates generated, filled out by an extracted
features [9] or predictor attributes. With its introduction, most subtitling
systems used RNNs as speech models for encoding visual input on one or more
levels of a CNN [9]. New careful paradigms have been suggested, and state-of-
the-art outcomes for machine interpretation and language comprehension work
have been accomplished. Self-attention would be a technique that relates
several places in a specific setting to calculate a sequence presentation. It
is the backbone for the Extended Neural models GPU, ByteNet, and ConvS2S [11],
all used as fundamental building blocks for deep convolutional neural network
models. Throughout the Transformer, all of that is dropped to a constant
number of stages. At the same time, the average attention-weighted places are
lowered at the cost of such effective resolution describe the steps.
In the field of image subscription, a search-based method was prominent. The
system demonstrates that the spatial link [26] between items enhances the
quality and the positioning of the produced annotation. One proposed approach
is to integrate the pictures and text in the same place in the vector to
facilitate searches. Similar efforts include using a recurring visual
representation of a photograph and using deep reinforcement education [26] for
picture submission. Image extraction is the second collection of data in the
BanglaLekha series [15] focused on pictures of solitary characteristics in
Bangla. Specific direct techniques were also suggested, including the query
extension method Yagcioglu et al. to cope with picture description issues [3].
These techniques are creative, and each has its features, but they have the
same problem of not intuitively observing the items or activities in the
image. In this study, we examine the process of developing techniques of
picture description in past years and describe the essential framework and
improvements. The task of the decoder is to need future words with the
absolute most excellent chance from all vocabulary words within the previously
created title and to produce endings of the term ken. The first is the
injection architecture, where the RNN is employed as a generator for the
caption conditioning of the picture characteristics. Based on these fusion
model accomplishments and the NLP CNN [8] success story, we propose an
encoder-decoder-based model for Bengali Image Titling based on the fusion
architecture [25]. Contrary to the CNN used in our study, a pooling layer is
used to capture essential and relevant information. We also compared our CNN
with qualitative and quantitative data mixture model, and thus the CNN-LSTM
[11] data based mixture model was proposed. The CNN-based language encoder is
used to improve the proposed model’s overall performance. Introducing
compassionate models in which the recurring connection for self-attention is
discontinued gives unparalleled set-up and sequence modeling possibilities.
Multi-modal structure of the sub-titling of images for specific architectures,
differing from the need for a single modality. This same transformer paradigm
underlying machine translation inspires our design and contains two crucial
innovations in all prior image description methods. M2 Transformer with
Meshed-Memory features [6] consists of an encoder that includes a self-
attention with a forward feeding layer stack and then a decoder that utilizes
autonomous attention to the signs and awareness to the output of the final
encoder phase. Tested on the COCO reference point, the ”Karpathy” testing data
[6], the model attained a new method of state-of-the-art in both individual
and set configurations. Significantly, it exceeds current online test server
suggestions and ranks first between the methods provided.

Figure 1: The overview of the framework of our proposed model

##  3 Methodology

We employed an Encoder-Decoder approach where the image extraction process was
carried out in this research utilizing the ResNet-101 model. In earlier
models, ResNets executed other functions, including object detection [15] and
semantical picture segmentation [3]. In computer vision, experts have
progressively substituted the conventional extractors for ResNet-101. The
underlying mechanism for the efficiency of ResNets remains to be defined,
though. ResNets are initially used to remove great features from the input
data. Last but not least, the state sequence is employed to recognize the
picture elements. The ResNet encoder comprises Ne stacked equivalent layers,
each consisting of the sublayer and the positional forward-feed substitute.
Then Positional encoding is being used to retain location coordinates as a
shared component. The basic Transformer encoding methodology, [23] employs
sinusoidal oscillations to fill out the weighted network model of positioning
encoding. An attention layer ties all places to a predetermined sequential
number of stages, whereas a recurring layer involves sequential operations.
Autoattention might be limited to investigate the input data focused on the
position of the respective component. The proposed image captioning algorithm
is illustrated in Fig. 1 as an overview. It is comprised of two parts: an
encoder and a decoder.

###  3.1 Encoder

All input sequences must be translated into a consistent abstract
representation that comprises all the learned information for this type of
sequence by the Encoder layer. The Encoder consists of two sub-modules: multi-
headed attention and another is a completely linked network. Layer
normalization creates multiple residual connections between the two sublayers
(see Fig. 2).

Figure 2: Encoder architecture

####  3.1.1 Feature Extraction:

As an image feature extractor, we utilized the pre-trained ResNet-101 [4]. The
ImageNet dataset [20] was used to train it. Pattern recognition has always
been a strong suit for neural networks with several layers. There are
overfitting problems and they are difficult to optimize. Residual CNNs are
built with fast connections between layers in mind. Identity mapping is
performed via these relationships. ResNets are simple to tune, and their
implementation improves as the network depth grows. We discarded the
ResNet-101’s final output layer, which contains the picture classification
output, and only used the encoded image characteristics created by the hidden
layers.

####  3.1.2 Positional Encoding:

We must provide data about the relative or absolute positions of the tokens in
the series for the model to use the sequence order because ours lacks
recurrence and convolution. To accomplish so, we add ”positional threescore
encodings” to the bottoms of the input embeddings of the encoder and decoder
stacks.

There are a variety of learned and fixed positional encodings from which to
choose. We utilize sine and cosine functions of various frequencies in this
work: Let t stand for the desired location in an input sentence, be the
encoding that corresponds to it, and d stand for the encoding dimension,
$\overrightarrow{P_{t}}\;\epsilon\;\mathbb{R}^{d}$ be its corresponding
encoding, and d be the encoding dimension (where $d\equiv 20$) then
$f:\mathbb{N\rightarrow}\mathbb{R}^{d}$ will be the function that produces the
output vector $\overrightarrow{P_{t}}$ and it defined as follows:

$\overrightarrow{p_{t}}^{(i)}=f(t)^{(i)}$ (1)

Here, the limit is between
$\sin(w_{k}.t),\mathbf{if}\;\mathit{i}=2k\;and\;cos(w_{k}.t),\mathbf{if}\;\mathit{i}=2k+1$
where

$\omega_{k}=\frac{1}{10000^{2k/d}}$ (2)

The frequencies decrease along the vector dimension, as can be seen from the
function description. As a result, the wavelengths follow a geometric
progression from $2\pi$ to 10000$\cdot 2\pi$ on the wavelengths. The
positional embedding pt may alternatively be thought of as a vector containing
pairs of sines and cosines positional $embedding\rightarrow pt$ for each
frequency (note that d is divisible by 2).

####  3.1.3 Multi-Headed Attention:

Multi-headed attention in the encoder uses an unique attention technique
called self-attention model. Self-attention, in our case, allows models to
connect each pixel in the input three-three to other pixels (see Fig. 3).
Where the Query, Key, and Value Vectors are Q, K, and V, respectively. To
achieve self-attention, we feed the input into three distinct, ultimately
linked layers to produce the query, key, and value vectors.

Figure 3: Multi-Headed Attention mechanism

####  3.1.4 Dot Product of Query and Key:

The questions and keys are multiplied by a dot product matrix to produce a
scoring matrix after passing the query, key, and value vectors through a
linear layer. The scoring matrix indicates how much importance a pixel should
obtain in comparison to other pixels. As a result, each pixel in the time step
will be assigned a score related to the scores of other pixels in the time
step (see Fig. 5). The higher the score, the greater the concentration.

####  3.1.5 Scaling Down the Attention Scores:

After that, the scores are scaled-down by dividing them by the square root of
the query and critical dimensions (see Fig. 5). This score allows for more
steady gradients because multiplying numbers can have explosive results.After
that, the scores are scaled-down by dividing them by the square root of the
query and critical dimensions. This score allows for more steady gradients
because multiplying numbers can have explosive results.

####  3.1.6 Softmax of the Scaled Scores Multiply Softmax Output with Value
vector:

The softmax of the scaled score is then used to determine the attention
weights, yielding probability values ranging from 0 to 1. Your higher scores
rise while your lower scores decline when you do a softmax. It makes choosing
which pixel to focus on for the model much more accessible.

$Z=softmax(\frac{Q*K^{T}}{\sqrt{d_{k}}})*V$ (3)

Where Z is the self-attention matrix, Q is the query matrix, K is the
fundamental matrix, V is the value matrix, and dk is the primary matrix’s
dimension. [23] improved the self-attention layer by incorporating a ”Multi-
Head” attention mechanism. The self-attention computation is implemented eight
times with various weight matrices in Multi-Head attention.

Figure 4: Dot product with query and key metrics

Figure 5: Scaled score metrics

####  3.1.7 The Residual Connections, Layer Normalization, and Feed Forward
Network:

The residual connection is formed when the multi-headed attention output
vector is cumulated to the original positional input embedding. The output of
the residual link is normalized at the layer level before being projected for
further processing using a pointwise feed-forward network. A ReLU activation
separates two linear layers in the pointwise feed-forward network. The output
is then normalized by adding it to the input of the pointwise feed-forward
network. By allowing gradients to pass straight through the networks, the
remaining links improve network training. The grid is stabilized using layer
normalizations, resulting in a considerable reduction in training time. The
pointwise feed-forward layer is used to project better possible attention
outcomes.

$FFN(x)=FC_{2}(Dropout(ReLU(FC_{1}(x))))$ (4)

A sublayer connection in each sublayer is made up of a residual link proceeded
by layer normalization.

$x^{out}=LayerNorm(x^{in}+Sublayer(x^{in}))$ (5)

Where $x^{in}$, $x^{out}$ are the input and output of one sublayer,feed-
forward and the sublayer can be the attention feed-forward-forward layer.

###  3.2 Decoder

On the decoder side, we combine the word embedding features with sinusoid
positional embedding and use both the extra findings and encoder output
features as input. The decoder is made up of Nd identical layers, each of
which has a mask multi-head self-attention sublayer and a multi-head cross
attention sublayer, as well as a positional feed-forward sublayer in that
order. A linear layer with an output dimension identical to the vocabulary
size uses the output feature throughout the last decoder layer to figure out
the meaning of the next word. Given a ground truth sentence $y_{1:T}^{*}$ and
the prediction $y_{t}^{*}$ of the captioning model with parameters $\Theta$,
we minimize the subsequent cross-entropy loss:

$L_{XE}(\Theta)=-\sum_{t=1}^{T}log(p_{\Theta}(y_{t}^{*}|y_{1:t-1}^{*}))$ (6)

We used the sahajBERT model from the Hugging Face library for word embedding
in the decoding stage. Like other encoding processes, we have used multi-head
self-attention sublayer, multi-head cross attention sublayer, and positional
feed-forward sublayer in the decoding stage to finetune our model.

##  4 Experiments

###  4.1 Datasets

This part mainly focuses on methods for analyzing open-source datasets and
generated sentences in this domain, where a good dataset might help an
algorithm or model perform better.

We have tested and trained our model on the BanglaLekhaImageCaptions dataset
[14].There are 9,154 images in the dataset presented are collected from the
public domain. Besides, it is relevant to Bengali culture to some extent.
While this collected data is small in comparison to other datasets where the
prominent image captioning datasets such as MSCOCO [5], PASCAL 1K [19],
Flickr8k [10], Flickr30k [17] contain a strong western cultural bias, with
annotations written in English.

To train our model, we split the data set into two parts: train and validation
where we have used 2 captions per image. We used 7323 x 2 images for the
training. The remaining 1831 x 2 images were used in during validation.

###  4.2 Data Preprocessinng

A considerable amount of human bias exists in the BanglaLekhaImageCaptions
dataset [14]. Any model’s capacity to describe non-human subjects is hampered
because of this bias. In certain circumstances, the captions aren’t as
descriptive as they should be, resulting in inaccurate training and evaluation
of any model. For this reason, we have done some preprocessing to ensure that
our dataset is noise-free and that it can perform well.

Even though the Hugging Face library has a large number of pre-trained Bangla
models and for preprocessing we have used sahajBERT [21] which is a pre-
trained language model that consists of

  * •

A tokenizer created specifically for Bengali and

  * •

An ALBERT architecture that has been cooperatively pre-trained on a dump of
Wikipedia in Bengali and the Bengali section of OSCAR.

By fine-tuning their pre-trained models three times on two downstream tasks in
Bengali. Besides, we assess the quality of the sahajBERT model and two other
model benchmarks (XLM-R-large and IndicBert).

##  5 Evaluation

###  5.1 Quantitative Analysis

For Quantitative results, there are five evaluation indexes, namely, BLEU
[16], METEOR [7], ROUGE [12], CIDEr [24], and SPICE [1] to evaluate the
models’ predicted caption.

Among all five indicators, BLEU [16] is the most often used criterion for
evaluation. BLEU considers translation length, word order, and word choice
when determining how near a machine translation is to a human reference
translation. It is used for image captioning, machine translation, etc.

METEOR [7] is also used to evaluate machine translation, in which the model’s
generated translation is compared to the reference translation. It’s intended
to address some of the issues with BLEU.

ROUGE [12] is an excellent metric for assessing the performance of machine
translation tasks and automatic summarization. The better the performance, the
higher the RUGE score.

CIDEr [24] was designed primarily to solve image annotation issues. It aims to
address the case of a weak correlation between previous metrics and human
judgments.

SPICE [1] is a systematic metric for automatic evaluation comparing
propositional semantic content in image captions. Rather than using the
traditional n-gram metrics, SPICE is better at capturing human opinions about
the model’s subtitles on the dataset of raw image captions.

Here, higher metric results suggested superior performance in the BLEU and
METEOR indexes. In Table LABEL:table_1, the scores have been highlighted with
the highest accuracy in boldface where a comparative study among our model,
CNN language encoder with a merged-based architecture [11], LSTM language
encoder with a mixture-based architecture [22], and Bi-LSTM language encoder
with an inject-based architecture [18] can be found here.

Table 1: Comparing the performance of various models quantitatively Model | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR
---|---|---|---|---|---
Our proposed model | 0.694 | 0.580 | 0.505 | 2.22e-308 | 0.337
CNN + ResNet-50 [merged] [11] | 0.651 | 0.426 | 0.278 | 0.175 | 0.297
CNN + LSTM [mixture] [22] | 0.632 | 0.414 | 0.269 | 0.168 | 0.291
CNN + Bi-LSTM [inject] [18] | 0.619 | 0.403 | 0.261 | 0.163 | 0.296

As can be seen, our proposed model achieved better results than the
traditional architecture. It’s also worth mentioning that the BLEU and METEOR
scores are higher, indicating that our captions are of higher quality.

###  5.2 Qualitative Analysis

Fig. 6 depicts a qualitative example of our model’s performance. While the
assessment metrics provide a numerical indication of the validity of the
captions, any result can be misinterpreted. Detecting subtle differences in
the generated captions compared to the original human language description can
be evaluated via qualitative analysis. This concept will be bolstered even
more by the qualitative assessment offered below.

---
Figure 6: Qualitative examples of Bengali captions generated by our proposed
model with untouched captions and BLEU score

For non-native Bengali speakers, we provide the predicted Bengali caption and
the associated English Translated caption. Our algorithm outperformed the
competitive edge not only in terms of scoring but also in terms of caption
quality. In comparison to other models, our model correctly identified the
most appropriate caption as well as very well-detailed.

##  6 Conclusion and Future Work

This paper introduces a novel Transformer-based architecture using a Deep
Neural Network for automatically developing captions in Bengali from an input
image, with a comprehensive test on the BanglaLekhaImageCaptions dataset,
which yielded excellent results, justifies the feasibility of our proposed
approach. Besides, the BLEU and METEOR scores were used to evaluate the
proposed architecture’s performance and the results, which were analyzed using
both quantitative and qualitative methods. The result shows that image
captioning in the Bengali language can be improved shortly. Furthermore, we
will aim to adapt the visual attention and transformer models instead of
ResNet-101 for improved feature extraction and more precise captions. We
anticipate that our approach will encourage others to develop more efficient
transformer-based architectures to aid image captioning as well as other
computer vision tasks requiring relational attention.

## References

  * [1] Anderson, P., Fernando, B., Johnson, M., Gould, S.: Spice: Semantic propositional image caption evaluation. In: European conference on computer vision. pp. 382–398. Springer (2016)
  * [2] Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6077–6086 (2018)
  * [3] Bernardi, R., Cakici, R., Elliott, D., Erdem, A., Erdem, E., Ikizler-Cinbis, N., Keller, F., Muscat, A., Plank, B.: Automatic description generation from images: A survey of models, datasets, and evaluation measures. Journal of Artificial Intelligence Research 55, 409–442 (2016)
  * [4] Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., Gao, W.: Pre-trained image processing transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12299–12310 (2021)
  * [5] Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollár, P., Zitnick, C.L.: Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015)
  * [6] Cornia, M., Stefanini, M., Baraldi, L., Cucchiara, R.: Meshed-memory transformer for image captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10578–10587 (2020)
  * [7] Denkowski, M., Lavie, A.: Meteor universal: Language specific translation evaluation for any target language. In: Proceedings of the ninth workshop on statistical machine translation. pp. 376–380 (2014)
  * [8] Ghosal, P., Nandanwar, L., Kanchan, S., Bhadra, A., Chakraborty, J., Nandi, D.: Brain tumor classification using resnet-101 based squeeze and excitation deep neural network. In: 2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP). pp. 1–6. IEEE (2019)
  * [9] Herdade, S., Kappeler, A., Boakye, K., Soares, J.: Image captioning: Transforming objects into words. arXiv preprint arXiv:1906.05963 (2019)
  * [10] Hodosh, M., Young, P., Hockenmaier, J.: Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research 47, 853–899 (2013)
  * [11] Khan, M.F., Sadiq-Ur-Rahman, S., Islam, M.S.: Improved bengali image captioning via deep convolutional neural network based encoder-decoder model. In: Proceedings of International Joint Conference on Advances in Computational Intelligence. pp. 217–229. Springer (2021)
  * [12] Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out. pp. 74–81 (2004)
  * [13] Liu, W., Chen, S., Guo, L., Zhu, X., Liu, J.: Cptr: Full transformer network for image captioning. arXiv preprint arXiv:2101.10804 (2021)
  * [14] Mansoor, N., Kamal, A.H., Mohammed, N., Momen, S., Rahman, M.M.: Banglalekhaimagecaptions (2019), Mendeley Data, V2, doi: 10.17632/rxxch9vw59.2
  * [15] Naim, F.A.: Bangla handwritten digit recognition based on different pixel matrices. In: International Conference on Innovative Computing and Communications. pp. 325–341. Springer (2022)
  * [16] Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics. pp. 311–318 (2002)
  * [17] Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In: Proceedings of the IEEE international conference on computer vision. pp. 2641–2649 (2015)
  * [18] Rahman, M., Mohammed, N., Mansoor, N., Momen, S.: Chittron: An automatic bangla image captioning system. Procedia Computer Science 154, 636–642 (2019)
  * [19] Rashtchian, C., Young, P., Hodosh, M., Hockenmaier, J.: Collecting image annotations using amazon’s mechanical turk. In: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. pp. 139–147 (2010)
  * [20] Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V.: Self-critical sequence training for image captioning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7008–7024 (2017)
  * [21] Saifullah, K.: neuropark/sahajbert (2021), https://github.com/khalidsaifullaah/BERTify
  * [22] Tanti, M., Gatt, A., Camilleri, K.P.: What is the role of recurrent neural networks (rnns) in an image caption generator? arXiv preprint arXiv:1708.02043 (2017)
  * [23] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998–6008 (2017)
  * [24] Vedantam, R., Lawrence Zitnick, C., Parikh, D.: Cider: Consensus-based image description evaluation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4566–4575 (2015)
  * [25] Wang, H., Zhang, Y., Yu, X.: An overview of image caption generation methods. Computational intelligence and neuroscience 2020 (2020)
  * [26] Wu, X., Jackson, D.J., Chen, H.C.: Novel fractal image-encoding algorithm based on a full-binary-tree searchless iterated function system. Optical Engineering 44(10), 107002 (2005)


# You Only Compress Once: Towards Effective and Elastic BERT Compression via
Exploit-Explore Stochastic Nature Gradient

Shaokun Zhang 1∗ Xiawu Zheng 1,4 Chenyi Yang 1 Yuchao Li 2
Yan Wang 5 Fei Chao 1 Mengdi Wang 2 Shen Li 2 Yang Jun 2 Rongrong Ji 1,3,4
1Media Analytics and Computing Lab, Department of Artificial Intelligence,
School of Informatics, Xiamen University 2Alibaba Group
3Institute of Artificial Intelligence, Xiamen University
4Peng Cheng Laboratory, Shenzhen, China.
5Pinterest, USA
Equal Contribution.Corresponding Author<EMAIL_ADDRESS>
###### Abstract

Despite superior performance on various natural language processing tasks,
pre-trained models such as BERT are challenged by deploying on resource-
constraint devices. Most existing model compression approaches require re-
compression or fine-tuning across diverse constraints to accommodate various
hardware deployments. This practically limits the further application of model
compression. Moreover, the ineffective training and searching process of
existing elastic compression paradigms [27, 4] prevents the direct migration
to BERT compression. Motivated by the necessity of efficient inference across
various constraints on BERT, we propose a novel approach, YOCO-BERT, to
achieve _compress once and deploy everywhere_. Specifically, we first
construct a huge search space with $10^{13}$ architectures, which covers
nearly all configurations in BERT model. Then, we propose a novel stochastic
nature gradient optimization method to guide the generation of optimal
candidate architecture which could keep a balanced trade-off between
explorations and exploitation. When a certain resource constraint is given, a
lightweight distribution optimization approach is utilized to obtain the
optimal network for target deployment without fine-tuning. Compared with
state-of-the-art algorithms, YOCO-BERT provides more compact models, yet
achieving 2.1$\%$-4.5$\%$ average accuracy improvement on the GLUE benchmark.
Besides, YOCO-BERT is also more effective, _e.g.,_ the training complexity is
$\mathcal{O}(1)$ for $N$ different devices. Code is available
https://github.com/MAC-AutoML/YOCO-BERT.

##  1 Introduction

While BERT [7] has driven advances in various natural language processing
tasks [31, 18], it is still notoriously challenging to deploy on resource-
limited devices, due to its large demands on computational power and memory
footprint. To alleviate this issue, there has been an exponential increase in
research concerning compressing the model, such as knowledge distillation [23,
20, 12], quantization [21], weight factorization [13] and pruning [9, 16].
Given an expected budget on computational power or memory, these methods
compress the full-fledged BERT into a smaller and/or faster model, with
acceptable compromise on model performance. While effective, these methods are
not sufficiently scalable to accommodate a wide range of devices, as re-
compression or an expensive fine-tuning process is usually required for even a
small change on the budget. It is thus impractical to adapt BERT or similar
Transformer-based models to a reasonably large number of types of devices.

Recently, there have been pioneer works aiming to improve the flexibility of
BERT compression. DynaBERT [11] compresses BERT into smaller BERT-like sub-
networks with adaptive widths and heights by training them iteratively using
distillation. AdaBERT and NAS-BERT [6, 30] exploits a NAS method to compress
the BERT model in a huge convolution-based space, to achieve more flexibility
in architecture. While these two directions achieve deployment flexibility by
serving different sub-networks at runtime, if viewed from an explore-exploit
perspective, they lie in the two extremes of the spectrum. By constraining the
architecture of sub-networks to be the same as the model to be compressed,
DynaBERT focuses on exploitation with tweaks on the width and height of
Transformers. On the contrary, AdaBERT and NAS-BERT nearly completely discard
the Transformer architecture but perform an architectural search in a general
search space based on convolution.

However, neither of these extreme approaches is optimal. While the pure
exploration approaches are able to achieve more flexibility from the huge
search space, conventional NAS methods tend to converge to local minimas, and
thus result in sub-optimal model performance. The pure exploiting approaches
strictly stick to the prior design of Transformers, and the limited search
space soon becomes the bottleneck of an effective trade-off between model size
and accuracy. We argue that a proper balance between exploitation and
exploration is the key to effective BERT model compression. And this can be
achieved with a huge but dedicatedly designed architectural search space,
which is then explored and exploited by an effective search algorithm.
Different from the traditional sub-network-based compression approaches [11,
4, 27], which randomly sample (i.e. explore) and train sub-networks to update
the weights in super models, we formulated the search algorithm in a
probabilistic manner. This is inspired by stochastic natural gradient methods
[22, 1]. These methods formulate the NAS process, which is traditionally
thought non-differentiable, as a continuous likelihood maximization process.
On one hand, this allows us to effectively explore the search space to achieve
faster convergence. On the other hand, we can also easily plug in different
distributions to model exploitation.

Therefore, in this paper, we propose an efficient and elastic BERT compression
algorithm, YOCO-BERT, to enable maximal flexibility across a wide range of
devices and minimal performance compromise, thus to achieve _compress once and
deploy everywhere_. We first construct a huge search space with $10^{13}$
architectures. Comparing with Dynabert[11] only has a dozen options, our
search space covers nearly all configurations in BERT model. Then, a novel
exploit-explore balanced stochastic natural gradient optimization algorithm is
proposed to efficiently explore the search space. Specifically, there are two
sequential stages in YOCO-BERT. We decouple the compression process into a
“super mode” training process, which does the core heavy lifting, and a
lightweight model adaptation process, which directly inherits the weight from
the “super model” _without any finetuning process_.

In the Exploit-Explore Stochastic Natural Gradient (EE-SNG) optimization
algorithm, a probability distribution group containing an exploration
distribution and an exploitation distribution is introduced in the super-BERT
training process to sample potential optimal sub-BERTs architectures at each
training step. Joint optimizations are then adopted on the weight across
architectures and the parameters of the exploitation distribution. Thus, the
“promising” architectures will be more likely to be sampled. We further
introduce an online learnable controller to dynamically determine whether
exploitation or exploration is more important. More specifically, the
controller is determined by the information entropy of the exploitation
distribution, _i.e.,_ the probability the exploitation is increasing with the
information entropy. In the deployment stage, we propose a novel objective
function to obtain the optimal architecture given any constraints based on the
optimized distribution. Thanks to the EE-SNG in the training stage, weights of
the optimal architectures under different constraints are well optimized. In
other words, we do not need any fine-tuning process to construct the target
model, which leads to an effective and elastic BERT compression.

Figure 1: The proposed YOCO-BERT. We separate YOCO-BERT into two stages. Given
expanded training space, we first propose a EE-SNG to optimize the probability
distribution $P_{\theta}$ and super-BERT. In the second stage, searching
algorithm is supervised by the optimized $P_{\theta}$ to obtain the optimal
architecture across diverse constraints.

The main contributions are listed as:

1) We design a novel and efficient BERT compression algorithm, YOCO-BERT,
which enables scalability to a wide range of hardware.

2) We propose a novel stochastic nature gradient optimization method called
EE-SNG to explicitly model exploitation and exploration in the model
compression.

3) Comparing to previous methods [23, 20, 29, 11, 8], experiments show that
our compressed models achieve the best compression rate and performance trade-
off on GLUE[26] benchmark.

##  2 Related Work

Since most of the BERT compression methods have been discussed in Sec. 1, we
introduce the two most related fields in this section.

Neural Architecture Search. NAS technologies are proposed to automatically
discover suitable network architectures over a vast architecture search space.
Early works exploit the paradigms of reinforcement learning (RL) [35] to reach
that goal. However, the RL-based NAS methods require intensive computational
and memory costs. _e.g.,_ [35] needs to train and evaluate more than $20,000$
neural architectures across $500$ GPUs over four days. In order to alleviate
this burden, one-shot NAS methods [14, 28, 15, 4, 34] proposed to construct a
super-BERT that share the weights with all the sub-network in the search
space. In this way, the search time will be reduced to few GPU hours and
retaining a comparable accuracy since the sub-network could be evaluated
efficiently.

Distribution Optimization. In terms of distribution optimization, various
approaches have been proposed. For discrete optimization, there is population-
based incremental learning (PBIL) [3], and the compact genetic algorithm (cGA)
[10]. Optimizing the parameters of the probability distribution is based on
the IGO [19]. There is a research direction of transforming NAS into a
distribution optimization problem. Specifically, following [22, 1, 33],
considering a probability distribution that generates network architecture and
optimizing the parameters of the distribution instead of directly optimizing
the network architecture. In this way, both the weight and the architecture
will be differentiable which could be optimized simultaneously. However, such
a method easily falls into local optimization due to reducing the probability
of exploring the entire search space.

##  3 YOCO-BERT

In this section, we describe the proposed framework. Specifically, we first
define the optimization problem in Sec. 3.1. Then, we discuss the search space
in Sec. 3.2, the exploit-explore stochastic natural gradient in Sec. 3.3 and
the adaption of the optimized super-BERT for diverse constraints in Sec. 3.4.

###  3.1 Problem Definition

Our primary proposal is to optimize one super-BERT containing all sub-BERTs,
enables the direct searching of candidate across diverse constraints. To this
end, given a specific dataset, we address the following optimization problems:

$\displaystyle
W^{*}=\arg\min_{W}\mathcal{L}_{train}\left(N\left(M,W\right)\right).$ (1)
$\displaystyle m^{*}=\mathop{\arg\max}_{m\in
M}(\text{Eval}_{\text{ACC}}\left(m,W^{*}\left(m\right)\right),\,s.t.\;\mathcal{T}\left(m\right)<{\Omega}.$
(2)

As demonstrated in the above equations, the problem is separated into two
aspects. Specifically, Eq. 1 optimized the weight $W$ of the super-BERT
$N\left(M,W\right)$ to minimize the training loss
$\mathcal{L}_{train}\left(.\right)$ of all possible candidate architectures in
search space $M$. The candidate architecture is denoted as $m\in M$. After the
super-BERT is well optimized, the optimal architecture $m^{*}$ with satisfied
constraint could be directly searched with weight $W^{*}\left(m\right)$
inherited from super-BERT. In Eq. 2, $\mathcal{T}\left(m\right)$ denotes the
constraints for sub-BERT $m$ and $\Omega$ denotes the target constraints given
for any devices. Note that, the optimal sub-BERT architecture $m^{*}$ directly
inherited the weight from super-BERT to acquire the performance without any
fine-tuning process, demonstrated in Tab. 5. Eq. 1 and Eq. 2 is solved by two
corresponding stages, which are summarized as follows:

Stage I: Joint Optimization (Sec. 3.3). We first solve the optimization
problem of super-BERT (Eq. 1) in an optimization of a differentiable objective
$\mathcal{G}(\cdot)$, the expectation of the loss on the training set.
Formally,

$\mathcal{G}\left(W,\theta\right)=E_{m\sim\mathcal{P}_{\theta}}\left(\mathcal{L}\left(N\left(w,m\right)\right)\right),$
(3)

where the function $\mathcal{L}\left(N\left(w,m\right)\right))$ is defined as
the loss of the sub-BERT $N\left(w,m\right)$ on training set. To this end, the
search space is considered as a probability distribution
$\mathcal{P}_{\theta}$ , which is parametrized by the hyper-parameter
$\theta$. The expected loss $\mathcal{G}\left(\cdot\right)$ is minimized
through optimizing the hyper-parameter $\theta$ and the weight $W$ of the
super-BERT model. In this paper, we propose a new Exploit-Explore Stochastic
Natural Gradient (EE-SNG) method to optimize the aforementioned distribution.

Stage II: Hardware Search (Sec. 3.4). After the weight of super-BERT and the
distribution are well optimized, the optimal sub-BERT could be directly
obtained from super-BERT under specific constraints ${\Omega}$. This process
is formally expressed in Eq. 2. In our algorithm, an efficient searching
process could be achieved since the searching process is supervised by
optimized distribution $\mathcal{P}_{\theta}$.

###  3.2 Search Space

Search Space Design. YOCO-BERT automatically searches optimal transformer
based architectures for BERT compression. In contrast to the convolutional
neural networks, the transformer-based architecture consists of a word
embedding layer and several transformer encoder layers, while each encoder
layer is fully characterized by a multi-head self-attention mechanism and a
position-wise feed-forward network. Instead of providing limited options of
architecture settings [11], we specially design a more general search space
for BERT, covering three main dimensions, _i.e._ , encoder layers number $d$,
intermediate size for two feed-forward layers $k$ and heads for each multi-
head attention module $h$. In this way, we built a huge search space with
$10^{13}$ architectures, allowing elastic head number, intermediate size,
layer number to accommodate almost all hardware constraints.

Super-BERT Model. To bypass the repeated training of candidate architectures,
we train a parameters-shared network, super-BERT model, subsuming all sub-BERT
models. Specifically, the weight of all candidate sub-BERT models are directly
inherited from the front portion of its corresponding part in super-BERT after
training super-BERT once. In this way, the optimized super-BERT is determined
as an evaluation proxy for all sub-BERTs. Intuitively, the introduce of super-
BERT model successfully decouple the parameters-shared super-BERT training and
the candidate architecture search which greatly reduces the cost of sub-BERTs
evaluation. Moreover, this decoupling enables to the acquisition of the sub-
BERTs by extracting corresponding weights without any fine-tuning.

Progressive Expanding Training Space. Inspired by the success of OFA [4] in
CNN-based space, we propose a progressive expanding training space in
transformer-based space to reduce the interference between sub-BERTs.
Specifically, the above dimensions are progressively expanded to train the
super-BERT. We first train the largest candidate, $\text{BERT}_{\text{base}}$
model ($d=12$, $k=3,072$ and $h=12$). Then, progressively shrink this
candidate to support the small architectures to generate an elastic training
space in each dimension (_i.e.,_ attention head number, intermediate size for
two feed-forward layers and layer number). Specifically, the head numbers in
each transformer encoder layer are relaxed to optional dimensions from a fixed
size in step-by-step processing, then the same relaxation is applied for the
intermediate size and the layer number. (_i.e.,_ the optional dimension of the
head number is expanded from a fixed $\\{12\\}$ to $\\{12,8\\}$, and then
expanded to $\\{12,8,4\\}$ at each layer. In each training step, all sub-BERTs
are sampled from the exploit-explore probability distribution group of the
expanding training space to optimize the corresponding part in the super-BERT.

###  3.3 Exploit-Explore Stochastic Natural Gradient Optimization

In this section, we demonstrate the optimization of probability distribution
$\mathcal{P}_{\theta}$ in Eq. 3 and how to alleviate the dilemma of
exploitation and exploration. Our goal is to minimize the loss expect of all
candidates in training space, and thus the optimal architectures are more
likely to be directly sampled from the optimized distribution. To this end, we
formulate the training space as a probability distribution
$\mathcal{P}_{\theta}$. Intuitively, we introduce a family of multi-
dimensional probability distribution $P_{\theta}\left(m\right)$ defined on the
training search space where $m$ denotes an sub-BERT architecture. Each sub-
BERTs $m=(o,d)$ could be defined by dimensions selections of each operations.
Intuitively, every architecture has a corresponding probability
$P_{\theta}(m)$. The candidate architectures are obtained through a sequential
process:
$\mathcal{P}_{\theta}\left(m\right)=p_{\theta_{1}}\left(o_{1}\right)\times
p_{\theta_{2}}(o_{2})...p_{\theta_{i}}(o_{i})$. Notice that, different
architecture variables are independent from each other. $o_{i}$ denotes an
one-hot vector that represents the dimension selection of operation $i$. Based
on previous works [22, 1], the expectation of loss is differentiable for $w$
and ${\theta}$, thus the distribution $\mathcal{P}_{\theta}$ could be
optimized using _stochastic relaxation_ :

$\displaystyle\nabla_{w}\mathcal{G}(W,\theta)=\mathbb{E}_{m\sim\mathcal{P}_{\theta}}[\nabla_{w}\mathcal{L}(w,m)],$
(4)
$\displaystyle\tilde{\nabla_{\theta}}\mathcal{G}(W,\theta)=\mathbb{E}_{m\sim\mathcal{P}_{\theta}}[\mathcal{L}(w,m)\nabla_{\theta}\ln(P_{\theta}(m))].$

Where $\mathcal{G}(W,\theta)$ denotes the expected loss for super-BERT with
weight $W$ and distribution with hyper-parameter $\theta$. The
$\tilde{\nabla}_{\theta}\ln(P_{\theta_{i}}(m))=F_{-1}\ln(P_{\theta_{i}}(m))$
is the so-called natural gradient of the log-likelihood introduced in [2]. In
practice, the gradients are estimated by the Monte-Carlo with samples
$m^{i}\sim\mathcal{P}_{\theta}(i=1,...,\lambda)$.

$\displaystyle\nabla_{w}\mathcal{G}(W,\theta)=\frac{1}{\lambda}\sum_{i=1}^{\lambda}\nabla_{w}\mathcal{L}(w^{i},m^{i}),$
(5)
$\displaystyle\tilde{\nabla_{\theta}}\mathcal{G}(W,\theta)=\frac{1}{\lambda}\sum_{i=1}^{\lambda}\mathcal{L}(w^{i},m^{i})\tilde{\nabla}_{\theta}\ln\mathcal{P}_{\theta}(m^{i}).$

With sufficient statistics $U(m)$, the expectation parameterization
$\theta=E(U(m))$ admits the natural gradient of the log-likelihood
$\tilde{\nabla}_{\theta}\ln(P_{\theta}(m))=U(x)-\theta$. The weight $W$ and
the parameters of the distribution $\theta$ will be optimized with appropriate
learning-rates.

However, the greedy nature of the SNG [22] based methods inevitably introduces
bias to ${\theta}$ and w during optimization, result a dilemma of exploitation
and exploration. Intuitively, this bias leave a portion of architectures well
trained and thus sampling more times, gradually leading a more serious biased
sampling. Reduced probability of exploring the entire search space limited the
upper bound of our optimization. Therefore, we propose exploit-explore
probability distribution group (EE-probability distribution group) to
alleviate this issue, which is presented in Alg. 1. In the following, we
discuss the collaboration of exploration and exploitation distribution with an
online-learnable controller.

Exploit-Explore Probability Distribution Group. We propose a probability
distribution group to merge the benefits of both deterministic and stochastic
predictions, significantly improving efficiency and effectiveness. The main
distribution, the exploitation distribution $\mathcal{P}_{\theta}$, is
optimized during the whole process. The parameters $\theta$ of the
exploitation distribution and the weight $w$ of the sampled architectures are
both optimized with sampled architectures using the gradient as shown in Eq.
5. However, such an optimization algorithm easily falls into a local optimum
as state earlier. Specifically, we introduce a novel distribution $\pi$
defined on the whole search space. $\pi$ is always a uniform distribution
which is considered as an exploration distribution. When the further
optimization of $P_{\theta}$ is constraint by the aforementioned bias, the
samplings from the distribution $\pi$ will supervise the optimization to
explore the architectures which have a small probability sampled by
$P_{\theta}$. In this way, the exploitation distribution avoids falling into
local optima since introducing the uncertainty and randomness from the entire
search space. A critical problem of the explore-exploit probability
distribution group is how to balance them in the optimization process, which
is described in the following contents.

Dynamic Controller. An online-learnable controller is proposed to determine
whether to accept the exploitation distribution or to introduce the stochastic
behavior for further optimization. More specifically, a Bernoulli distribution
is determined by the information entropy of exploitation distribution to
decide the sampling options.

$H\sim B\left(N,K\right),K=\frac{\rho}{\rho_{max}},$ (6)

where $H$ has the possibility of $K$ to be the exploitation distribution, and
the possibility of $1-K$ to be the exploration distribution, $N$ represents
the number of times to select $H$. In our algorithm, the Bernoulli
distribution $B$ is dynamic, and the controller could adapt to the appropriate
state during the super-BERT optimizing process based on the information
entropy $\rho$ of the exploitation distribution as shown in Eq. 6. On one
hand, when the exploitation distribution $P_{\theta}$ tends to convergence,
and the corresponding information entropy will decrease. In this case, the
Bernoulli distribution $H$ introduces a penalty to the local exploration,
i.e., the possibility of sampling from the entire training space will be
enhanced. On the other hand, when $P_{\theta}$ has a high information entropy,
the distribution $H$ will let the optimization process to exploit in
$P_{\theta}$. This simple but effective method could keep a good balance
between exploration and exploitation. We demonstrate the effectiveness of our
method in Tab. 4. Note that, in the optimization process, the sub-BERTs
sampling will only happen on the current training space based on the
distributions.

###  3.4 Search For Optimal Architecture

After the optimization process in Sec. 3.3, we obtain a probability
distribution $\mathcal{P}_{\theta}$ and an optimized super-BERT model. There
are various works trying to deploy the searched architectures in NAS process
to different devices [5, 32] . However, such methods are computationally
expensive. Inspired by [24], together with the optimized probability, we
design a constraint sensitive reward function Eq. 7 to further optimize the
exploitation distribution $\mathcal{P}_{\theta}$ using stochastic natural
gradient optimization in Eq. 5, which means $\mathcal{P}_{\theta}$ will
concentrate on the sub-BERTs with superior performance under certain
constraints. Noted that the optimized weight of super-BERT is fixed in this
stage.

$\displaystyle Reward=\begin{cases}\quad\quad\quad
ACC\left(m\right),&\mathcal{T}\left(m\right)<{\Omega\vspace{0.8ex}}\\\
ACC\left(m\right)\times\left(\frac{\mathcal{T}\left(m\right)-\Omega}{\mathcal{T}\left({M}\right)-\Omega}\right)^{\alpha},&otherwise\end{cases}$
(7)

Given a sub-BERT $m$, $ACC(m)$ denotes its accuracy on the target tasks. The
function $\mathcal{T}\left(m\right)$ and $\mathcal{T}\left(M\right)$ measure
the related property of sub-BERT $m$ and super-BERT $M$ (_e.g.,_ FLOPs, model
size). $\Omega$ means the target constraints under certain devices. In our
algorithm, we empirically use $\alpha=2$. We employ a simple but effective
method to search the target sub-BERT. During the searching process, we
randomly sample sub-BERTs from the exploitation distribution
$\mathcal{P}_{\theta}$, and evaluate using the weight inheriting from the
super-BERT. The reward function in Eq. 6 is used to optimize
$\mathcal{P}_{\theta}$. This process will repeat until reaching max searching
steps. Since we only need to train one super-BERT for various devices, _e.g.,_
we could gain the target architecture using our searching algorithm. For $N$
different devices, the training complexity is $\mathcal{O}(1)$.

Algorithm 1 Exploit-Explore Stochastic Natural Gradient Optimization

0: Training Data $D$, Update Interval $I$

0: Optimized parameters of $W$ and $\theta$

1: Initialize $W$ and $\theta$; $t\leftarrow 0$;

2: for $t$ = 1,…,$t_{max}$ epoch do

3: if $t\,\%\,I=0$ then

4: Update the controller using Eq. 6;

5: end if

6: Get a batch of samples from $D$;

7: Decide the distribution by dynamic controller;

8: Sample $i$ architectures $m_{0},...,m_{i}$ according to the
exploitation/exploration distribution;

9: Compute the loss $\mathcal{L}\left(w_{i},m_{i}\right)$ for each $m_{i}$;

10: Update $\theta$ and $W$ using the gradient in Eq. 5;

11: Expand training search space; $t\leftarrow t+1$;

12: end for

##  4 Experiments

In this section, the extensive experiments of YOCO-BERT are conducted against
various BERT compression methods. More accurate performance is reported,
associating with reduced searching cost. The effectiveness of our method is
further demonstrated in ablation analysis.

Datasets. The experiments are conducted on the GLUE benchmark[26], which is
widely used to evaluate pre-trained language models. Following BERT [7], we do
not consider the controversial WNLI dataset. Regarding SST-2, MNLI, QNLI QQP
and RTE, the accuracy is utilized as the metric. While the CoLA is evaluated
on matthew’s correlation. In terms of MRPC, the F1 and accuracy are evaluated.
The average result of MNLI-m and MNLI-mm is reported on MNLI . In particular,
the experiments conducted on Fig. 2 utilize the accuracy as the metric for all
datasets. In this paper, if not specifically mentioned, the experimental data
is the development set of GLUE by default.

Experimental Setup. For training the super-BERT, the initial learning rates
are $1\times 10^{-5}$ for MNLI and QNLI and $2\times 10^{-5}$ for the other
datasets of GLUE respectively. Super-BERT is trained from 1 to 10 epochs
according to the dataset size. The Adam optimizer is utilized for weight
optimization scheduled by a linear annealing. Weight is not decay during
training. The epsilon for adam is set to be $1\times 10^{-8}$. Since fine-
tuning on BERT easily destroys the pre-trained weight if a high learning rate
is set, we perform a complete learning rate decay procedure, in which the
learning rate decays to 0. For the most tasks, our total training epoch does
not exceed 20 except QNLI and MRPC to avoid over-training. All our experiments
are conducted on one Nvidia V100 16GB GPU.

Search Space Setups. To support a scalable super-BERT, our architectures
consist of a stack of transformer encoder layers, searching the layer number
$d\in\\{6,8,10,12\\}$. For each layer, we sample from the intermediate size
$k\in\\{512,768,1024,3072\\}$ and head number $h\in\\{4,8,12\\}$. In this
case, we construct a huge search space with $10^{13}$ architectures.

Supe-BERT Setups. The initial super-BERT weight is directly inherited from the
pre-trained $\text{BERT}_{\text{base}}$ model ($d=12$, $k=3,072$ and $h=12$),
which composed of 12 identical transformer encoder layers. To speed up the
performance evaluation of the compressed model, we directly conducted training
process on super-BERT.

###  4.1 Results on GLUE benchmark

Results Under Different Constraints. We first compare the performance of the
searched sub-BERTs with several constraints against BERT [7], where the
results are reported in Tab. 1. Compared to $\text{BERT}_{\text{base}}$,
architectures compressed by YOCO-BERT achieve superior performance with
reduced model sizes on most datasets. In particular,
$\text{Searched}_{\text{C}}$ achieves $10\%$ accuracy increasing with $64\%$
parameter size compressing on COLA. Moreover, most architectures searched by
YOCO-BERT outperforms $\text{BERT}_{\text{base}}$ while maintaining a more
compact size. This further demonstrates the necessity of BERT compressing, as
redundant parameters of BERT clearly resulting in significant resource cost
[17].

Table 1: Development set results of the GLUE benchmark on the optimal architectures searched by YOCO-BERT in different parameter constraints. Model Name | Params | SST-2 | MRPC | CoLA | RTE | MNLI | QQP | QNLI
---|---|---|---|---|---|---|---|---
$\text{BERT}_{base}$ | 110m | 92.7 | 89.5 | 54.3 | 71.1 | 83.5 | 89.8 | 91.2
$\text{Searched}_{A}$ | 20m $\sim$ 40m | 84.3 | 81.2 | 15.2 | 65.0 | 71.8 | 88.8 | 69.8
$\text{Searched}_{B}$ | 40m $\sim$ 60m | 92.1 | 88.5 | 55.6 | 69.3 | 81.7 | 89.9 | 85.1
$\text{Searched}_{C}$ | 60m $\sim$ 80m | 92.8 | 90.3 | 59.8 | 72.9 | 82.6 | 90.5 | 87.2
$\text{Searched}_{D}$ | 80m $\sim$ 100m | 93.6 | 89.5 | 61.3 | 74.4 | 83.8 | 90.4 | 87.5

Comparisons with Previous Compression Work. We compare the performance and
parameters of YOCO-BERT against existing BERT compression methods, including
BERT-PKD [23], DistilBERT [20], DynaBERT[11], BERT-of-Theseus [29], PD-BERT
[25], MINILM [17] and LayerDrop [8]. As fine-tuning and data augmentation (DA)
are not utilized on YOCO-BERT, NAS-BER[30] is not compared in our experiments.
For a fair comparison, the model accuracy is compared maintaining a similar
FLOPs and parameters. To compare the results of methods that have released the
code, we performance the experiments by running the official repository with
the same random seed. Otherwise, the results are obtained from the original
paper. The corresponding results reported in Tab. 2. In evidence that YOCO-
BERT outperforms all previous STOA models compression methods on almost all
datasets with a same level constraint (67M). Notably, YOCO-BERT reports a
$14.4\%$ higher accuracy on CoLA and a $7.7\%$ improvement on RTE compared
with LayerDrop. In terms of DistilBERT[20] which used large external data, the
average accuracy of our method outperforms it by $3.7\%$.

We further demonstrate the effectiveness of YOCO-BERT under different
constraints as shown in Fig. 2. The comparisons demonstrate that architectures
obtained by YOCO-BERT achieve better trade-off between accuracy and model
size. Compared with DynaBERT[11], YOCO-BERT achieves $6.2\%$, $5.4\%$ accuracy
improvement on MRPC and RTE with more compact models respectively. Moreover,
YOCO-BERT achieves a $0.5\%$ accuracy improvement with a $53\%$ parameter size
reduction on SST-2. Regarding different datasets, the training time span from
30 minutes (MRPC) to 20 hours (QNLI) depending on the size of the datasets.
Compared with DistilBERT[20] which costs 720 GPU hours to train, our algorithm
is extremely efficient.

Table 2: Comparision of YOCO-BERT with previous works under the same constraints on the development set of GLUE. Method | Params | SST-2 | MRPC | CoLA | RTE | MNLI | QQP | QNLI | Average
---|---|---|---|---|---|---|---|---|---
$\text{BERT}_{12}$ | 110m | 92.7 | 89.5 | 54.3 | 71.1 | 83.5 | 89.8 | 91.2 | 81.7
LayerDrop[8] | 67m | 90.7 | 85.9 | 45.4 | 65.2 | 80.7 | 88.3 | 88.4 | 77.8
DistilBERT[20] | 67m | 91.3 | 87.5 | 51.3 | 59.9 | 82.2 | 88.5 | 89.2 | 78.6
BERT-PKD[23] | 67m | 91.3 | 85.7 | 45.5 | 66.5 | 81.3 | 88.4 | 88.4 | 78.2
PD-BERT[25] | 67m | 91.1 | 87.2 | - | 66.7 | 82.5 | 89.1 | 89.0 | -
BERT-of-Theseus[29] | 67m | 91.5 | 89.0 | 51.1 | 68.2 | 82.3 | 89.6 | 89.5 | 80.2
MINILM[17] | 67m | 92.0 | 88.4 | 49.2 | 71.5 | - | 91.0 | 91.0 | -
YOCO-BERT | 59m$\sim$67m | 92.8 | 90.3 | 59.8 | 72.9 | 82.6 | 90.5 | 87.2 | 82.3
Figure 2: Comparison of parameters, FLOPs between YOCO-BERT and previous works
under different datasets. We use accuracy as the metric for the three
datasets. Since YOCO-BERT does not use external data, Dyanbert’s results are
obtained without using data augmentation (DA).

Comparing with HAT[27] and OFA[4]. HAT[27] demonstrate a transformer-based
compression on machine translation tasks. OFA[4] compresses CNN-based model
elastically on a wide range of devices. For a fair comparison, we re-implement
these two methods and search optimal architectures in our search space. The
results are reported in Tab. 3. As we can see, YOCO-BERT outperforms HAT[27]
and OFA[4] on all datasets in achieving higher accuracy on both compression
ratios. Regarding $0.75$ compression ratio on RTE dataset, YOCO-BERT reports
10.1$\%$ and 12.3$\%$ improvements for HAT and OFA respectively. Since HAT and
OFA require additional fine-tuning process, there is a linear growing of the
total costs concerning these methods, preventing the directly migration of
these methods. In contrast to these methods, the weight is directly inherited
from super-BERT in our method, reducing the cost of specialized BERT
compression from $\mathcal{O}(N)$ to $\mathcal{O}(1)$.

Table 3: Comparison with HAT[27] and OFA[4] | MRPC | SST-2 | RTE
---|---|---|---
Compression Ratio | 0.75x | 0.5x | 0.75x | 0.5x | 0.75x | 0.5x
HAT-BERT | 82.2 | 82.6 | 88.6 | 88.6 | 65.0 | 64.6
OFA-BERT | 87.6 | 85.2 | 89.3 | 89.8 | 62.8 | 65.3
YOCO-BERT | 90.4 | 87.6 | 92.9 | 91.9 | 75.1 | 69.3

###  4.2 Ablation Study

Impact of Exploit-Explore Stochastic Natural Gradient Optimization. To analyse
the effectiveness of EE-SNG optimization algorithm, three experiments are
conducted under different constraints: (a) Exploit-Only BERT compression; (b)
Explore-Only BERT compression; (c) YOCO-BERT. The results are reported in Tab.
4, compared to (a) and (b), YOCO-BERT provides better performance. When
compressing BERT with $0.75x$ compression ratio, YOCO-BERT outperforms
Explore-Only 9.8$\%$ accuracy on RTE, while 5.3$\%$ accuracy increasing is
achieved on MRPC. This proves the importance of balancing exploration and
exploitation.

Table 4: Ablation study of the Exploit-Explore Stochastic Natural Gradient Optimization | Exploit-Only | Explore-Only | YOCO-BERT
---|---|---|---
Compression Ratio | 0.75x | 0.5x | 0.75x | 0.5x | 0.75x | 0.5x
MRPC | $88.7_{{\color[rgb]{1,0,0}-1.7}}$ | $84.1_{{\color[rgb]{1,0,0}-3.5}}$ | $85.1_{{\color[rgb]{1,0,0}-5.3}}$ | $85.5_{{\color[rgb]{1,0,0}-2.1}}$ | 90.4 | 87.6
RTE | $72.5_{{\color[rgb]{1,0,0}-2.6}}$ | $66.4_{{\color[rgb]{1,0,0}-2.9}}$ | $65.3_{{\color[rgb]{1,0,0}-9.8}}$ | $67.1_{{\color[rgb]{1,0,0}-2.2}}$ | 75.1 | 69.3
Figure 3: Comparison of evolutionary search, random search and YOCO-BERT in
the search stage.

Impact of the Searching Algorithm. To analyse the impact of the probability
distribution-based search algorithm, we compared our algorithm against random
search and evolutionary search on the MRPC dataset, while other experimental
conditions are fixed. Results are shown in Fig. 3, ten optimal sub-
architectures are recorded every steps in each algorithms. Our algorithm
outperforms all other methods in achieving both highest accuracy and
convergent rate. In fact, the probability distribution has been well optimized
during the super-BERT training process. Therefore, the optimal architecture
could be identified more efficiently from optimized distribution during the
searching process.

Fine-tuning Searched Architectures. In the previous experiments, we reported
the results without fine-tuning stage, _i.e._ directly inheriting the weights
from super-BERT and evaluating. In this section, we explore the impact of the
fine-tuning to the searched architectures. Fine-tuning is performed regarding
different learning rates for additional ten epochs on the three datasets
(MRPC, SST-2, RTE). As shown in Tab. 5, the fine-tuning stage obviously is not
able to capture a significant improving and even acts a negative impact on the
sampled sub-BERTs. In NAS, this phenomenon has been mentioned in BigNAS[32] as
well.

Table 5: Ablation study of the fine-tuning process for sampled architecture from super-BERT model. We compare the performance of sub-architectures fine-tuned with various learning rates (FT at different LR) and directly inheriting the weight from super-BERT (w/o Fine-tuning). Dataset | FT | FT | FT | w/o
---|---|---|---|---
lr = $2*e^{-5}$ | lr = $2*e^{-6}$ | lr = $2*e^{-7}$ | Finetuning
MRPC | $85.7_{{\color[rgb]{1,0,0}-4.6}}$ | $88.3_{{\color[rgb]{1,0,0}-2.0}}$ | $89.4_{{\color[rgb]{1,0,0}-0.9}}$ | 90.3
SST-2 | $89.2_{{\color[rgb]{1,0,0}-3.6}}$ | $91.3_{{\color[rgb]{1,0,0}-1.5}}$ | $92.7_{{\color[rgb]{1,0,0}-0.1}}$ | 92.8
RTE | $67.9_{{\color[rgb]{1,0,0}-5.0}}$ | $67.5_{{\color[rgb]{1,0,0}-5.4}}$ | $72.2_{{\color[rgb]{1,0,0}-0.7}}$ | 72.9

##  5 Conclusion

In this paper, we propose a BERT compression algorithm, YOCO-BERT, that
automatically compresses transformer-based models to satisfy resource
constraints in an elastic and effective way. We first perform NAS in a huge
search space with $10^{13}$ architectures within the BERT family. Under the
guidance of the probability distribution, the training is conducted in a
progressive way such that the sub-BERTs sampled from the final super-BERT
model are able to provide good performance. Then we use a lightweight search
based on a probability distribution to obtain the optimal model. In this way,
we only need to compress once for different hardware constraints. Extensive
experiments on GLUE demonstrate that YOCO-BERT shows better accuracy and
compression ratio compared with previous compression methods. Future works may
include extending the approach for more tasks and broader models.

## References

  * [1] Youhei Akimoto, Shinichi Shirakawa, Nozomu Yoshinari, Kento Uchida, Shota Saito, and Kouhei Nishida.  Adaptive stochastic natural gradient method for one-shot neural architecture search.  In ICML, 2019.
  * [2] Shun-Ichi Amari.  Natural gradient works efficiently in learning.  Neural computation, 10(2):251–276, 1998.
  * [3] Shumeet Baluja.  Population-based incremental learning. a method for integrating genetic search based function optimization and competitive learning.  Technical report, Carnegie-Mellon Univ Pittsburgh Pa Dept Of Computer Science, 1994.
  * [4] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han.  Once-for-all: Train one network and specialize it for efficient deployment.  In International Conference on Learning Representations, 2020.
  * [5] Han Cai, Ligeng Zhu, and Song Han.  Proxylessnas: Direct neural architecture search on target task and hardware.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  * [6] Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, and Jingren Zhou.  Adabert: Task-adaptive bert compression with differentiable neural architecture search.  In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 2463–2469. International Joint Conferences on Artificial Intelligence Organization, 7 2020.
  * [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding".  In NAACL, pages 4171–4186, 2019.
  * [8] Angela Fan, Edouard Grave, and Armand Joulin.  Reducing transformer depth on demand with structured dropout.  In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
  * [9] Mitchell Gordon, Kevin Duh, and Nicholas Andrews.  Compressing BERT: Studying the effects of weight pruning on transfer learning.  In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 143–155, Online, 2020. Association for Computational Linguistics.
  * [10] Georges R Harik, Fernando G Lobo, and David E Goldberg.  The compact genetic algorithm.  IEEE transactions on evolutionary computation, 3(4):287–297, 1999\.
  * [11] Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu.  Dynabert: Dynamic BERT with adaptive width and depth.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [12] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.  TinyBERT: Distilling BERT for natural language understanding.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174, Online, November 2020. Association for Computational Linguistics.
  * [13] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  ALBERT: A lite BERT for self-supervised learning of language representations.  In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
  * [14] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy.  Progressive neural architecture search, 2018.
  * [15] Hanxiao Liu, Karen Simonyan, and Yiming Yang.  Darts: Differentiable architecture search, 2019.
  * [16] J. Scott McCarley.  Pruning a bert-based question answering model.  ArXiv, abs/1910.06360, 2019.
  * [17] Paul Michel, Omer Levy, and Graham Neubig.  Are sixteen heads really better than one?, 2019.
  * [18] Rodrigo Nogueira and Kyunghyun Cho.  Passage re-ranking with bert.  arXiv preprint arXiv:1901.04085, 2019.
  * [19] Yann Ollivier, Ludovic Arnold, Anne Auger, and Nikolaus Hansen.  Information-geometric optimization algorithms: A unifying picture via invariance principles.  Journal of Machine Learning Research, 18(18):1–65, 2017.
  * [20] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  arXiv preprint arXiv:1910.01108, 2019.
  * [21] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer.  Q-bert: Hessian based ultra low precision quantization of bert.  In AAAI, pages 8815–8821, 2020.
  * [22] Shinichi Shirakawa, Yasushi Iwata, and Youhei Akimoto.  Dynamic optimization of neural network structures using probabilistic modeling, 2018.
  * [23] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu.  Patient knowledge distillation for bert model compression.  In EMNLP, 2019.
  * [24] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le.  Mnasnet: Platform-aware neural architecture search for mobile, 2019.
  * [25] Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Well-read students learn better: On the importance of pre-training compact models.  arXiv preprint arXiv:1908.08962, 2019.
  * [26] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  * [27] Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han.  Hat: Hardware-aware transformers for efficient natural language processing.  In Annual Conference of the Association for Computational Linguistics, 2020.
  * [28] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin.  SNAS: stochastic neural architecture search.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  * [29] Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou.  BERT-of-theseus: Compressing BERT by progressive module replacing.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7859–7869, Online, November 2020\. Association for Computational Linguistics.
  * [30] Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Li Jian, Tao Qin, and Tie-Yan Liu.  Task-agnostic and adaptive-size BERT compression, 2021.
  * [31] Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin.  End-to-end open-domain question answering with BERTserini.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 72–77, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [32] Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel M. Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc V. Le.  Bignas: Scaling up neural architecture search with big single-stage models.  2020\.
  * [33] Xiawu Zheng, Rongrong Ji, Yuhang Chen, Qiang Wang, Baochang Zhang, Qixiang Ye, Jie Chen, Feiyue Huang, and Yonghong Tian.  Migo-nas: Towards fast and generalizable neural architecture search.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2021.
  * [34] Xiawu Zheng, Rongrong Ji, Lang Tang, Baochang Zhang, Jianzhuang Liu, and Qi Tian.  Multinomial distribution learning for effective neural architecture search.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1304–1313, 2019.
  * [35] Barret Zoph and Quoc V. Le.  Neural architecture search with reinforcement learning.  2017\.


# Screening in gated bilayer graphene

L.A. Falkovsky  L.D. Landau Institute for Theoretical Physics, Moscow 117334,
Russia  Institute of the High Pressure Physics, Troitsk 142190, Russia

###### Abstract

The tight-binding model of a graphene bilayer is used to find the gap between
the conduction and valence bands, as a function of both the gate voltage and
as the doping by donors or acceptors. The total Hartree energy is minimized
and the equation for the gap is obtained. This equation for the ratio of the
gap to the chemical potential is determined only by the screening constant.
Thus the gap is strictly proportional to the gate voltage or the carrier
concentration in the absence of donors or acceptors. In the opposite case,
where the donors or acceptors are present, the gap demonstrates the
asymmetrical behavior on the electron and hole sides of the gate bias.

###### pacs:

73.20.At, 73.21.Ac, 73.43.-f, 81.05.Uw

Bilayer graphene has attracted much interest partly due to the opening of a
tunable gap in its electronic spectrum by an external electrostatic field.
Such a phenomenon was predicted by McCann and Fal’ko McF  and can be observed
in optical studies controlled by applying a gate bias OBS ; ZBF ; KHM ; LHJ ;
ECNM ; NC . In Refs. Mc ; MAF , within the self-consistent Hartree
approximation, the gap was derived as a near-linear function of the carrier
concentration injected in the bilayer by the gate bias. Recently, this problem
was numerically considered GLS  using the density functional theory (DFT) and
including the external charge doping involved with impurities. The DFT
calculation gives the gap which is roughly half of the gap obtained in the
Hartree approximation. This disagreement was explained in Ref. GLS  as a
result of both the inter- and intralayer correlations.

In this Brief Report, we study this problem within the same Hartree
approximation as in Refs. Mc ; MAF , but including the external doping. We
consider the case, where the carrier concentration in the bilayer is less than
1013 cm-2, calculating the carrier concentration on both layers. Then, we
minimize the total energy of the system and find self-consistently both the
chemical potential and the gap induced by the gate bias. Our results
completely differ from those obtained in Refs. Mc ; MAF  even for the range
where the external doping is negligible. The dependence of the gap on the
carrier concentration, i. e., on the gate voltage, exhibits an asymmetry at
the electron and hole sides of the gate bias.

The graphene bilayer lattice is shown in Fig. 1. Atoms in one layer, i. e.,
$a$ and $b$ in the unit cell, are connected by solid lines, and in the other
layer, e. g., $a_{1}$ and $b_{1}$, by the dashed lines. The atom $a$ ($a_{1}$)
differs from $b$ ($b_{1}$) because it has a neighbor just below in the
adjacent layer, whereas the atom $b$ ($b_{1}$) does not.

Let us recall the main results of the Slonchewski–Weiss–McClure model SW ;
McCl . In the tight-binding model, the Bloch functions of the bilayer are
written in the form

$\displaystyle\psi_{a}=\frac{1}{\sqrt{N}}\sum_{j}e^{i{\bf
ka}_{j}}\psi_{0}({\bf a}_{j}-{\bf r})$
$\displaystyle\psi_{b}=\frac{1}{\sqrt{N}}\sum_{j}e^{i{\bf
ka}_{j}}\psi_{0}({\bf a}_{j}+{\bf a}-{\bf r})$ (1)
$\displaystyle\psi_{a1}=\frac{1}{\sqrt{N}}\sum_{j}e^{i{\bf
ka}_{j}}\psi_{0}({\bf a}_{j}+{\bf c}-{\bf r})$
$\displaystyle\psi_{b1}=\frac{1}{\sqrt{N}}\sum_{j}e^{i{\bf
ka}_{j}}\psi_{0}({\bf a}_{j}+{\bf c}+{\bf a}-{\bf r}),$

where the sums are taken over the lattice vectors ${\bf a}_{j}$ and $N$ is the
number of unit cells. Vectors ${\bf a}$ and ${\bf c}$ connect the nearest
atoms in the layer and in the neighbor layers, correspondingly.

Figure 1: Bilayer lattice

For the nearest neighbors, the effective Hamiltonian in the space of the
functions (Screening in gated bilayer graphene) contains the hopping integrals
$\gamma_{0},\gamma_{1},\gamma_{3},\gamma_{4},$ and $\Delta$ PP . The largest
of them, $\gamma_{0}$, determines the band dispersion near the $K$ point in
the Brillouin zone with a velocity parameter $v$. The parameters $\gamma_{3}$
and $\gamma_{4}$ giving a correction to the dispersion are less than
$\gamma_{0}$ by a factor of 10 (see Refs. KHM ; LHJ ). The parameters
$\gamma_{1}$ and $\Delta$ result in the displacements of the levels at $K$,
but $\Delta$ is much less than $\gamma_{1}$. Besides, there is the parameter
$U$ induced by the gate and describing the asymmetry of two layers in the
external electrostatic field. This parameter simply presents the potential
energy between two layers, $2U=-edE$, where $d$ is the interlayer distance and
$E$ is the electric field induced both by the gate voltage and the external
charge dopants in the bilayer. In the simplest case, the effective Hamiltonian
can be written as

$H(\mathbf{k})=\left(\begin{array}[]{cccc}U&vk_{+}&\gamma_{1}&0\\\
vk_{-}&U&0&0\\\ \gamma_{1}&0&-U&vk_{-}\\\ 0&0&vk_{+}&-U\end{array}\right),$
(2)

where the matrix elements are expanded in the momentum $k_{\pm}=\mp
ik_{x}-k_{y}$ near the $K$ points.

Figure 2: Band structure of bilayer

The Hamiltonian gives four energy bands:

$\displaystyle\varepsilon_{1,4}(q)=\pm\left(\frac{\gamma_{1}^{2}}{2}+U^{2}+q^{2}+W\right)^{1/2}\,,$
(3)
$\displaystyle\varepsilon_{2,3}(q)=\pm\left(\frac{\gamma_{1}^{2}}{2}+U^{2}+q^{2}-W\right)^{1/2}\,,$

where

$W=\left(\frac{\gamma_{1}^{4}}{4}+(\gamma_{1}^{2}+4U^{2})q^{2}\right)^{1/2}$

and we denote $q^{2}=(vk)^{2}$.

The band structure is shown in Fig. 2. The minimal value of the upper energy
$\varepsilon_{1}$ is $\sqrt{U^{2}+\gamma_{1}^{2}}$. The $\varepsilon_{2}$ band
takes the maximal value $|U|$ at $k=0$ and the minimal value
$\tilde{U}=\gamma_{1}|U|/\sqrt{\gamma_{1}^{2}+4U^{2}}$ at
$q^{2}=2U^{2}(\gamma_{1}^{2}+2U^{2})/(\gamma_{1}^{2}+4U^{2}).$ Because the
value of $U$ is much less than $\gamma_{1}$, the distinction between $U$ and
$\tilde{U}$ is small and the gap between the bands $\varepsilon_{2}$ and
$\varepsilon_{3}$ takes the value $2|U|$.

Figure 3: Electrostatic model; $d$ is the interlayer distance, $d_{w}$ is the
wafer thickness.

The eigenfunctions ${\mathbf{C}}$ of the Hamiltonian (2) have the form

${\mathbf{C}}=\frac{1}{C}\left(\begin{array}[]{c}(U-\varepsilon_{n})[(\varepsilon_{n}+U)^{2}-q^{2}]\\\
-q_{-}[(\varepsilon_{n}+U)^{2}-q^{2}]\\\
\gamma_{1}(U^{2}-\varepsilon_{n}^{2})\\\
\gamma_{1}q_{+}(U-\varepsilon_{n})\end{array}\right),$ (4)

where the ${\mathbf{C}}$ norm squared is

$\displaystyle
C^{2}=[(\varepsilon_{n}+U)^{2}-q^{2}]^{2}[(\varepsilon_{n}-U)^{2}+q^{2}]$
$\displaystyle+\gamma_{1}^{2}(\varepsilon_{n}-U)^{2}[(\varepsilon_{n}+U)^{2}+q^{2}]\,.$

The probability to find an electron, for instance, on the first layer is
$|C_{1}|^{2}+|C_{2}|^{2}$, as seen from Eqs. (Screening in gated bilayer
graphene).

We assume, that carriers occupy only the bands $\varepsilon_{2,3}$, so the
chemical potential $\mu$ and the gap $2|U|$ are less than the distance between
the bands $\varepsilon_{1}$ and $\varepsilon_{2}$, i. e.,
$(|\mu|,2|U|)<\gamma_{1}.$ The electron dispersion for the $\varepsilon_{2,3}$
bands can be expanded in powers of $q^{2}$:

$\varepsilon_{2,3}^{2}(q)=U^{2}-4\frac{U^{2}}{\gamma_{1}^{2}}q^{2}+\frac{q^{4}}{\gamma_{1}^{2}}.$

Then, for $q^{2}\gg 4U^{2}$, we can use the simple relations:

$\displaystyle\varepsilon_{2,3}(q)=\pm\sqrt{U^{2}+q^{4}/\gamma_{1}^{2}}\,,$
$\displaystyle|C_{1}|^{2}+|C_{2}|^{2}=q^{4}/[q^{4}+\gamma_{1}^{2}(\varepsilon_{2,3}-U)^{2}]$
(5) $\displaystyle=(\varepsilon_{2,3}+U)/2\varepsilon_{2,3}\,.$

Within such the approximation, many observable effects can be analytically
evaluated for the intermediate carrier concentration,
$4U^{2}\ll\gamma_{1}\sqrt{\mu^{2}-U^{2}}\ll\gamma_{1}^{2}$, where we neglect
the effect of the ”mexican hat”.

At zero temperature, for the total carrier concentration $n$ and the carrier
concentrations $n_{1,2}$ on the layers, we obtain

$\displaystyle
n=\frac{\gamma_{1}}{\pi\hbar^{2}v^{2}}\sqrt{\mu^{2}-U^{2}}=\frac{n_{0}U}{\gamma_{1}}\sqrt{x^{2}-1}\,,$
(6) $\displaystyle
n_{1,2}=\frac{\gamma_{1}}{2\pi\hbar^{2}}\int_{U}^{\mu}\sqrt{\frac{\varepsilon+U}{\varepsilon-U}}d\varepsilon$
(7)
$\displaystyle=\frac{n_{0}U}{2\gamma_{1}}[\sqrt{x^{2}-1}\pm\ln{(x+\sqrt{x^{2}-1})}]\,,$

where $n_{0}=\gamma_{1}^{2}/\pi\hbar^{2}v^{2}=1.2\times 10^{13}$ cm-2 and
$x=\mu/U$.

In order to find the chemical potential $\mu$ and the gap $2|U|$ at the given
gate voltage, we minimize the total energy containing both the energy $V$ of
the carriers and the energy $V_{f}$ of the electrostatic field. Instead of the
chemical potential, it is convenient to use the variable $x$ along with $U$.
Electrons in the $\varepsilon_{2}$ band or holes in the $\varepsilon_{3}$ band
contribute in the total energy of the system the energy

$\displaystyle V=\frac{2}{\pi\hbar^{2}v^{2}}\int\varepsilon_{2}(q)qdq=$
$\displaystyle\frac{n_{0}U^{2}}{2\gamma_{1}}[x\sqrt{x^{2}-1}+\ln{(x+\sqrt{x^{2}-1})}]\,.$
(8)

The energy of the electrostatic field,

$V_{f}=\frac{1}{8\pi}(dE^{2}+\epsilon_{w}d_{w}E^{2}_{w})$ (9)

can be written in terms of the carrier concentrations with the help of
relations (see Fig. 3):

$4\pi e(n_{1}-n_{d}/2)=E\,\text{ and}\,4\pi e(n-n_{d})=\epsilon_{w}E_{w}\,,$
(10)

where $\epsilon_{w}$ is the dielectric constant of the wafer, the negative
(positive) $n_{d}$ is the acceptor (donor) concentration and we suppose that
the donors/acceptors are equally divided between two layers.

We seek the minimum of the total energy as a function of two variables, $U$
and $x$, under the gate bias constraint

$eV_{g}=-e^{2}dE-e^{2}d_{w}E_{w}\,.$ (11)

Excluding the Lagrange multiplier and assuming the interlayer distance to be
much less than the thickness $d_{w}$ of the dielectric wafer, we obtain the
following equation:

$4\pi
e^{2}d\left(n_{2}-\frac{n_{d}}{2}\right)\left(\frac{n_{1x}}{n_{x}}-\frac{n_{1u}}{n_{u}}\right)=\frac{V_{x}}{n_{x}}-\frac{V_{u}}{n_{u}}\,.$
(12)

Let us emphasize, that this equation is invariant only with respect to the
simultaneous sign change in $n_{1,2}$ and $n_{d}$, that expresses the charge
invariance of the problem. At the fixed sign of the external doping $n_{d}$,
the gap on the electron and hole sides of the gate bias is not symmetrical.

The derivatives in Eq. (12) are calculated with the help of Eqs. (6)–(10). As
a result, Eq. (12) becomes

$\displaystyle\frac{\gamma_{1}n_{d}}{Un_{0}}=\sqrt{x^{2}-1}\pm\left\\{f(x)+\frac{xf(x)}{\Lambda[xf(x)-\sqrt{x^{2}-1}]}\right\\}$
(13)

with the function $f(x)=\ln{(x+\sqrt{x^{2}-1})}$ and the dimensionless
screening constant

$\Lambda=\frac{e^{2}\gamma_{1}d}{(\hbar v)^{2}}\,.$

For the parameters of graphene $d=3.35\,\AA\,,\gamma_{1}=0.4$ eV, and
$v=10^{8}$ cm/s, we get $\Lambda=0.447$.

First, consider an ideal undoped bilayer with $n_{d}=0$, namely,
$|\gamma_{1}n_{d}/Un_{0}|\ll 1$. We obtain a solution, as $x_{0}=6.2784$, only
for one sign in Eq. (13) determining the polarity of the layers [see Eq. (7)].
This value gives $2|U/\mu|=2/x_{0}=0.3186$ for the ratio of the gap to the
chemical potential. According to Eq. (6), the gap as a function of the carrier
concentration takes a very simple form:

$2|U/n|=\frac{2\gamma_{1}}{n_{0}\sqrt{x_{0}^{2}-1}}=1.08\times
10^{-11}\text{meV}\cdot\text{cm}^{2}\,,$ (14)

where the right-hand side does not depend on the gate bias at all, but only on
the screening constant $\Lambda$.

Figure 4: The gap in units of $\gamma_{1}=0.4$ eV versus the carrier
concentration for the hole doping with concentration $n_{d}=-5\times 10^{12}$
cm-2; the positive (negative) values of $n$ correspond to the electron (hole)
conductivity.

We can compare Eq. (14) with the corresponding result of Ref. Mc :

$2|U/n|=\frac{e^{2}d}{2\epsilon_{0}}\left[1+2\Lambda\frac{|n|}{n_{0}}+\Lambda\ln\frac{n_{0}}{|n|}\right]^{-1}\,.$
(15)

Both equations are in the numerical agreement at $|n|=0.1n_{0}\simeq 10^{12}$
cm-2. However, contrary to Eq. (14), Eq. (15) contains the carrier
concentration in the right-hand side giving rise to the more rapid increase in
the gap with $|n|\ll n_{0}$. This increase also contradicts to the DFT
calculations GLS .

For the bilayer with acceptors or donors, $n_{d}\neq 0$, Eq. (13) presents a
solution $w=Un_{0}/\gamma_{1}n_{d}$ as a function of $x$. We obtain,
evidently, the large values of $w$ for $x$ close to $x_{0}=6.28$. In this
region of the relatively large $|U|$, we find again with the help of Eqs. (6)
and (13) the linear dependence

$\displaystyle
2|U|=2\gamma_{1}\frac{|n-n_{d}|}{n_{0}x_{0}}=1.08\,|n-n_{d}|\times
10^{-11}\text{ meV}\cdot\text{cm}^{2}.$

For the small gap, $|Un_{0}/\gamma_{1}n_{d}|<1$, we obtain different results
for the electron and hole types of conductivity. For instance, if the bilayer
contains acceptors (see Fig. 4) with concentration $n_{d}$, the gap decreases
linearly with the hole concentration and vanishes, when the gate bias is not
applied and the hole concentration equals $n_{d}$. Starting from this point,
the gap increases and, thereafter, becomes again small (equal to zero in Fig.
4) at the carrier concentration corresponding to the minimal value of the dc
conductivity. Therefore, the difference observed in Ref. MLS  between these
two values of carrier concentrations, at the zero bias and at the minimal
conductivity, gives directly the donor/acceptor concentration in the bilayer.
Then, for the gate bias applied in order to increase the electron
concentration, the gap is rapidly opening with the electron appearance.

We see, that the asymmetry arises between the electron and hole sides of the
gate bias. This asymmetry can simulate a result of the hopping integral
$\Delta$ in the electron spectrum CNM . In order to obtain the gap dependence
for the case of electron doping, $n_{d}>0$, the reflection transformation
$n\rightarrow-n$ has to be made in Fig. 4.

The gap in the vicinity of the minimal conductivity point reaches indeed a
finite value due to several reasons. One of them is the form of the ”mexican
hat” shown in Fig. 2. Second, the trigonal warping is substantial at low
carrier concentrations. Finally, the graphene electron spectrum is unstable
with respect to the Coulomb interaction at the low momentum values. For the
graphene monolayer as shown in Ref. Mi , the logarithmic corrections appear at
the small momentum. In the case of the bilayer, the electron self-energy
contains the linear corrections, as can be found using the perturbation
theory. The similar linear terms resulting in a nematic order were also
obtained in the framework of the renormalization group VY .

In conclusions, the gap $2U$ opening in the gated graphene bilayer has an
intriguing behavior as a function of carrier concentration. In the presence of
the external doping charge, i. e. donors or acceptors, this function is
asymmetric on the hole and electron sides of the gate bias and it is linear
only for the large gate bias.

This work was supported by the Russian Foundation for Basic Research (grant
No. 07-02-00571). The author is grateful to the Max Planck Institute for the
Physics of Complex Systems for hospitality in Dresden.

## References

  * (1) E. McCann, V.I. Fal’ko, Phys. Rev. Lett. 96, 086805 (2006).
  * (2) T. Ohta, A. Bostwick, T Seyller, K. Horn, and E. Rotenberg, Science 313, 951 (2006).
  * (3) L.M. Zhang, Z.Q. Li, D.N. Basov, M.M. Foger, Z. Hao, and M.C. Martin, Phys. Rev. B 78, 235408 (2008).
  * (4) A.B. Kuzmenko, E. van Heumen, D. van der Marel, P. Lerch, P. Blake, K.S. Novoselov, A.K. Geim, Phys. Rev. B 79, 115441 (2009).
  * (5) Z.Q. Li, E.A. Henriksen, Z. Jiang, Z. Hao, M.C. Martin, P. Kim, H.L. Stormer, and D.N. Basov, Phys. Rev. Lett. 102, 037403 (2009).
  * (6) E.V. Castro, K.S. Novoselov. S.V. Morozov, N.M.R. Peres, J.M.B. Lopes dos Santos, Johan Nilsson, F. Guinea, A.K. Geim, and A.H. Castro Neto, Phys. Rev. Lett. 99, 216802 (2007).
  * (7) E.J. Nicol, J.P. Carbotte, Phys. Rev. B 77, 155409 (2008).
  * (8) E. McCann, Phys. Rev. B 74, 161403(R) (2006).
  * (9) E. McCann, D.S.L. Abergel, V.I. Fal’ko, Sol. St. Comm. 143, 110 (2007).
  * (10) P. Gava, M. Lazzeri, A.M. Saitta, and F. Mauri, arXiv:0902.4615 (2009).
  * (11) J.C. Slonchewski and P.R. Weiss, Phys. Rev. 109, 272 (1958).
  * (12) J.W. McClure, Phys. Rev. 108, 612 (1957).
  * (13) B. Partoens, F.M. Peeters, Phys. Rev. B 74, 075404 (2006).
  * (14) E.V. Castro, K.S. Novoselov. S.V. Morozov, N.M.R. Peres, J.M.B. Lopes dos Santos, Johan Nilsson, F. Guinea, A.K. Geim, and A.H. Castro Neto, arXiv:0807.3348 (2008).
  * (15) K.F. Mak, C.H. Lui, J. Shan, and T.F. Heinz, (2009).
  * (16) E.G. Mishchenko, Phys. Rev. Lett. 98, 216801 (2007).
  * (17) O. Vafek and K. Yang, arXiv:0906.2483 (2009).


# Plasma-based Control of Supersonic Nozzle Flow

Datta V. Gaitonde

Computational Sciences Branch, AFRL/RBAC
Air Vehicles Directorate
Air Force Research Laboratory, WPAFB, OH 45433, USA

###### Abstract

The flow structure obtained when Localized Arc Filament Plasma Actuators
(LAFPA) are employed to control the flow issuing from a perfectly expanded
Mach 1.3 nozzle is elucidated by visualizing coherent structures obtained from
Implicit Large-Eddy Simulations. The computations reproduce recent
experimental observations at the Ohio State University to influence the
acoustic and mixing properties of the jet. Eight actuators were placed on a
collar around the periphery of the nozzle exit and selectively excited to
generate various modes, including first and second mixed ($m=\pm 1$ and $m=\pm
2$) and axisymmetric ($m=0$). In this fluid dynamics video (mpeg-1, mpeg-2),
unsteady and phase-averaged quantities are displayed to aid understanding of
the vortex dynamics associated with the $m=\pm 1$ and $m=0$ modes excited at
the preferred column-mode frequency (Strouhal number $0.3$). The unsteady flow
in both contains a broad spectrum of coherent features. For $m=\pm 1$, the
phase-averaged flow reveals the generation of successive distorted elliptic
vortex rings with axes in the flapping plane, but alternating on either side
of the jet axis. This generates a chain of structures where each interacts
with its predecessor on one side and its successor on the other. Through self
and mutual interaction, the leading segment of each loop is pinched and passes
through the previous ring before rapidly breaking up, and the mean jet flow
takes on an elliptic shape. The $m=0$ mode exhibits relatively stable roll-up
events, with vortex ribs in the braid regions connecting successive large
coherent structures. Results with other modes are described in Ref. 1.

##  1 Introduction

Jet flow behavior and control has significant consequences for many
applications, including mitigation of aircraft noise, and mixing for various
industrial concerns such as combustion and pollution. Numerous types of
control methods have been employed, including passive (chevrons, tabs) and
active (fluidic and plasma) techniques. The present effort uses numerical
simulations to explore the dynamics of plasma-based open-loop flow control of
a Mach 1.3 jet, described by Samimy et al [2]. The plasma devices, denoted
Localized Arc Filament Plasma Actuators (LAFPA), strike an electric arc at a
specified frequency between two closely placed pin electrodes. Their
advantages include rapid on-off capability, low inertia and superior high
frequency performance. The full 3-D Navier-Stokes equations are solved with an
implicit LES approach. The effect of the actuators is simulated with a surface
heating model that successfully reproduces the principal effects observed in
experimental flow visualizations. Since the actuators can be either on or off,
the excitation is applied in a discrete fashion and can be described by a
collective frequency and duty cycle. The fluid dynamics video may be found in
mpeg-1 and mpeg-2 formats. The visualizations explore first mixed (flapping)
and axisymmetric modes at a Strouhal number of $0.3$, corresponding to
$4618Hz$ and a duty cycle of $20\%$. The spreading rate of the jet with the
$m=\pm 1$ mode is significantly increased in the flapping plane, but is
reduced in the non flapping-plane relative to the no-control case. Phase-
averaged observations visualized with isolevels of the Q-criterion colored by
vorticity magnitude, and vorticity magnitude colored by velocity magnitude,
indicate staggered and unstaggered structures on the two planes respectively.
The video shows these to be consistent with rings whose axes lie in the
flapping plane, but alternate about the jet centerline. The resulting
interaction between successive such events results in elongation and partial
pushing through of the leading segment of each ring through the trailing
segment of the previous ring. The jet displays an elliptic cross-section in
the mean. Axisymmetric excitation yields successive roll-up events subjected
to azimuthal disturbances, whose rapid evolution leads to breakdown. Although
not shown in the video, the $m=\pm 2$ mode (see Ref. [1]) also generates
elliptic vortex structures but with major axes being successively aligned
along the two symmetry planes. The minor axes regions move downstream faster
because of the higher velocity near the centerline, yielding rings which
stretch in the streamwise direction and ultimately breakdown.

## References

[1] Gaitonde, D.V., “Simulation of Supersonic Nozzle Flows with Plasma-based
Control,” AIAA-2009-4187, 39th AIAA Fluid Dynamics Conference, San Antonio,
Texas, June 22-25, 2009.
[2] Samimy, M., Kim, J.-H., Kastner, J., Adamovich, I., and Utkin, Y., “Active
Control of High-speed and High-Reynolds-number Jets Using Plasma Actuators,”
J. Fluid Mech., Vol. 578, 2007, pp. 305-330