# Born for Auto-Tagging: Faster and better with new objective functions

Chiung-ju Liu  awoo Intelligence AI  Huang-Ting Shieh We thank Tung-Chi Tsai
and Liang-Wei Liu for the technical support and discussion during the
preparation of the paper. awoo Intelligence AI

###### Abstract.

Keyword extraction is a task of text mining. It is applied to increase search
volume in SEO and ads. Implemented in auto-tagging, it makes tagging on a mass
scale of online articles and photos efficiently and accurately. BAT is
invented for auto-tagging which served as awoo’s AI marketing platform (AMP).
awoo AMP not only provides service as a customized recommender system but also
increases the converting rate in E-commerce. The strength of BAT converges
faster and better than other SOTA models, as its 4-layer structure achieves
the best F scores at 50 epochs. In other words, it performs better than other
models which require deeper layers at 100 epochs. To generate rich and clean
tags, awoo creates new objective functions to maintain similar ${\rm F_{1}}$
scores with cross-entropy while enhancing ${\rm F_{2}}$ scores simultaneously.
To assure the even better performance of F scores awoo revamps the learning
rate strategy proposed by Transformer [24] to increase ${\rm F_{1}}$ and ${\rm
F_{2}}$ scores at the same time.

###### Key words and phrases:

Text classification objective functions contrastive learning.

##  1\. Introduction

Keywords are widely used for site search, ads, and personalized
recommendations on E-commence, news sites, streaming media, etc. In
particular, keyword search dominates the performance in search engine
optimization (SEO). Valuable words such as high search volume are used to
write articles with strong content or target popular issues. Automatically
keyword extraction is helpful in online business.

Keyword extraction is an essential task in text mining, information retrieval,
and natural language processing (NLP). However, it is challenging when it is
implemented on web data or industry documents. The difficulty is not only due
to the scale of data but also the noisy signals including unstructured
sentences, emotional symbols, network language, etc. Manual labeling for
keywords is time-consuming. The consistency of labeling differs from labelers
and needs rules to supervise. Machine learning emerges to solve the problem.
Unsupervised algorithms based on statistic models are known as TF-IDF, BM25,
YAKE [1]. Graph-based models include TextRank [11], Rake [17], Singlerank [25]
etc. Supervised learning with neural networks has various development
recently. Some approaches are based on ranking the matching scores of queries
with documents [8, 6, 27, 13]. Others are performed as semantic analysis
classification [28, 22], etc.

awoo majors in MarTech. It serves as SaaS by combining OMO (Online Merge
Offline) and awoo AMP (AI marketing platform). It is successful in utilizing
valuable tags in E-commerce, retail stores, AdTech, SEO, retargeting service,
and business intelligence. awoo AMP uses tags to help customers to find
products efficiently and increase conversion rates. Tagging for product pages
does not only advantage in keyword search but also benefits the online stores
in managing products. Most personal recommendation systems are based on
collaborative filtering. It focuses either on items that one bought in the
past or other people bought in common. However, transfer of interest or
overlapping with popular items with lacking enough personal information may
cause poor recommendation. Understanding products and focusing on the message
of the items by tags may capture the intention of the customers. awoo AMP uses
the A/B test to show that it increases 71% search volume, 230% time on page,
200% conversion rates and 100% per customer transaction on average. It does
induce great shopping serendipity.

To generate tags for large E-commerce platforms, we investigate extracting
tags automatically. Tags range over categories, goods, brands, and features
which vary in different industries. They are complicated and technical to
establish rules. However, we have extensive experience in managing tags.

BAT (Born for Auto-Tagging) is reconstructed from the encoder of Evolved
Transformer [20]. This model is created for auto-tagging. It converges faster
and better than other SOTA models. Its 4-layer structure obtains the best F
scores at 50 epochs and the scores exceed the other deep layers at 100 epochs.
To generate rich (higher ${\rm F_{2}}$) and clean (higher ${\rm F_{1}}$) tags,
we create new objective functions $L_{\rm PBP}$ (1) and $L_{\rm CECLA}$ (3) to
maintain similar ${\rm F_{1}}$ scores with cross-entropy $L_{\rm CE}$ but also
increase ${\rm F_{2}}$ scores simultaneously.

Figure 1. CECLA vs PBP vs CE

Industry aspects such as SEO and ads value ${\rm F_{2}}$ scores more than
${\rm F_{1}}$ scores. On the other hand, security-related businesses such as
fingerprint recognition or medical’s object detection concern more about
precision or ${\rm F_{1}}$ scores. Since we want to find not only plentiful
but also neat tags, we look for the relative maximum of macro ${\rm F_{2}}$
curves, not the lowest loss value. Since the training is to minimize the loss
function, the loss value decreases along with the training. However, the
optimal point may not occur at the end or a certainly given time step. To
overcome this problem and enhance a better convergence, we revamp the learning
rate strategy (10) proposed by Transformer [24] to increase ${\rm F_{1}}$ and
${\rm F_{2}}$ scores at the same time. See the detail in section 4.3.

In the classification task on EC-Zh data, BAT gets 1.7%, 3.1%, 3.3% and 2.7%
more than Transformer via $L_{\rm CE}$, weighted cross-entropy $L_{\rm WCE}$,
cross-entropy with contrastive learning $L_{\rm CECLA}$ and punish bad
prediction $L_{\rm PBP}$ objective functions with respectively in macro ${\rm
F_{1}}$ at 50 epochs. BAT obtains 2.9%, 2.2%, 2.3% and 2.2% more than
Transformer by $L_{\rm CE}$, $L_{\rm WCE}$, $L_{\rm CECLA}$ and $L_{\rm PBP}$
loss functions with respectively in macro ${\rm F_{2}}$ on EC-Zh data. Most
SOTA models maintain almost the same ${\rm F_{1}}$ scores via new loss
functions and gain 2% ${\rm F_{2}}$ in average by comparing to $L_{\rm CE}$.
Moreover, on EC-JP dataset, BAT gets 1.4%, 2.5%, 3.5%, and 2.8% in macro ${\rm
F_{1}}$ via $L_{\rm CE}$, $L_{\rm WCE}$, $L_{\rm CECLA}$ and $L_{\rm PBP}$
with respectively. BAT gets 1.7%, 0.8%, 1.6%, and 1.5% in macro ${\rm F_{2}}$
via $L_{\rm CE}$, $L_{\rm WCE}$, $L_{\rm CECLA}$ and $L_{\rm PBP}$ with
respectively. See the experiment of comparison in section 5.2.

Figure 2. CECLA loss SOTA comparison

Figure 3. PBP loss SOTA comparison

Compared to the previously published works in the tasks of text classification
and translation, the contribution of this paper is as follows:

  * •

Tournament selection: Evolved Transformer [20] first implements this idea by
involving two convolution branches that cascade with a self-attention
sublayer. MUSE [29] lays out the multiple scale attention in parallel. We
build the multiple feature self-attention (MFSA) and the residual connection
in parallel. The two branches compete through backpropagation while MFSA races
with constant speed. In particular, in the feed-forward tournament selection
block (FFTS) FFNN enters the competition with a weighted residual connection.
It learns to balance the speed between the two branches and converges fast to
increase its F scores.

  * •

New objective functions: We create two types of objective functions for
imbalanced data in multiple classification tasks. One is PBP loss (punish bad
prediction):

$L_{\rm PBP}=-\frac{1}{N}\sum_{i=1}^{N}\left(\boldsymbol{\alpha}{\bf
y}_{i}({\bf 1}-{\bf p}_{i})^{\gamma}\log{\bf p}_{i}+\boldsymbol{\beta}(({\bf
1}-{\bf y})\odot\hat{{\bf y}}){\bf p}_{i}^{\gamma}\log({\bf 1}-{\bf
p}_{i})\right),$ (1)

where ${\bf y}_{i},{\bf p}_{i},\hat{{\bf
y}},\boldsymbol{\alpha},\boldsymbol{\beta}\in\mathbb{R}^{C}$. ${\bf y}_{i}$
denotes the label of a token $w_{i}$, ${\bf p}_{i}$ denotes the probability of
its prediction, $\hat{{\bf y}}$ denotes the predict class of ${\bf p}$, $C$
denotes the number of categories, $\boldsymbol{\alpha}$ and
$\boldsymbol{\beta}$ denote the balance weights and $N$ denotes the total
samples. $L_{\rm PBP}$ produces more loss value if the model predict incorrect
category. It makes models to converge faster. BAT obtains macro ${\rm
F_{1}}=85.5$ and ${\rm F_{2}}=88.0$ with $L_{\rm PBP_{1,8}}$ at 30 epochs for
a learning rate $\alpha=6/11$ and $\lambda=0.999$ in (11). $L_{\rm PBP}$
indeed saves training cost.

Let $N_{1},N_{2},\cdots,N_{C}$ denote the number of samples of each category.
The EE (equal expectation) weights are

$\begin{cases}\begin{array}[]{ll}\boldsymbol{\alpha}_{EE}&=(\frac{N}{N_{1}},\cdots,\frac{N}{N_{C}})\\\
\boldsymbol{\beta}_{EE}&=(\frac{N}{(N-N_{1})},\cdots,\frac{N}{(N-N_{C})})\end{array}\end{cases}.$
(2)

Another one is CECLA (Cross-Entropy by Contrastive Learning with Array weight)
loss

$L_{\rm CECLA}=-\frac{1}{N}\sum_{i=1}^{N}{\bf a}_{i}\bigg{(}{\bf y}_{i}({\bf
1}-{\bf p}_{i})^{\gamma}\log{\bf p}_{i}+({\bf 1}-{\bf y}_{i}){\bf
p}_{i}^{\gamma}\log({\bf 1}-{\bf p}_{i})\bigg{)},$ (3)

where

${\bf a}_{i}=A{\bf y}_{i},\quad
A=(A_{ij})=\begin{cases}\begin{array}[]{cl}\frac{N}{N_{i}}&\text{if $i=j$}\\\
\frac{N}{(C-1)N_{j}}&\text{if $i\neq j$}\end{array}\end{cases}.$

In particular, $A{\bf y}_{i}$ extracts the $j$-th column of $A$ if ${\bf
y}_{i}$ is in class $j$. $L_{\rm CECLA}$ also induces fast convergence. It
obtains its optimal (macro ${\rm F_{1}}=85.7$ and ${\rm F_{2}}=88.1$) with
learning rate $\alpha=7/15$, $\beta=-1/30$, and $\lambda=1.001$ in (11) at 40
epochs . It increases accuracy not only for BAT but also other SOTA models.

  * •

Best weight for weighted cross-entropy $L_{\rm WCE}$:

$L_{\rm WCE}=-\frac{1}{N}\sum_{i=1}^{N}\boldsymbol{\alpha}{\bf y}_{i}\log{\bf
p}_{i}$ (4)

For imbalanced data, cross-entropy $L_{\rm CE}$ can not obtain the optimal
parameters. We find out that it achieves the optimal with the coefficients

$\boldsymbol{\alpha}=(\frac{N}{N_{1}},\cdots,\frac{N}{N_{C}})$

according to equation (15) in appendix. We compare the performance of 9 loss
functions through F scores in section 5.3.

  * •

Best performance in multi-class classification in E-commerce data: We compare
BAT with 6 SOTA models via 4 objective functions in E-commence data with F
scores in section 5.2. SOTA models include Transformer, Admin, Switch
Transformer, MUSE, Performers, and Informer. The 4-layer structure of BAT
achieves the best F scores among all SOTA models with loss function $L_{\rm
CE}$, $L_{\rm WCE}$, $L_{\rm PBP}$, and $L_{\rm CECLA}$ at 50 epochs. Its
4-layer structure also beat other SOTA models’ deep structure at 100 epochs.
It saves training time and performs better.

  * •

Best learning strategy for your model: Models with large parameters tend to
converge slower and the other way around. We develop a method of adjusting the
triangle learning rate that introduces by Transformer [24]. It can change the
convergence of learning. We find the best learning rate for BAT and it obtains
the best ${\rm F_{1}}$ and ${\rm F_{2}}$ scores among all models. You may
formulate the best learning rate for your model introduced in section 4.3.

We select these SOTA models due to their performance in the task of machine
translation. However, we only extract the encoder of SOTA models and focus on
the task of text classification in this paper. We arrange the rest of the
paper as follows: Section 2 introduces the related work; Section 3 provides
the architecture of BAT; Section 4 indicates the training of BAT including the
loss functions and learning rate strategy; Section 5 shows the data set
description and experimental results; Appendix contains the proof and details
of loss functions.

##  2\. Related Work

###  2.1. Keyword Extraction

Keyword extraction is a task of NLP that automatically identify terms that
describe a document the most. Unsupervised methodologies that are based on a
static algorithm such as TF-IDF and BM25 calculate the relevance between the
query and documents. Recently, YAKE [1] pays more attention to the context and
sentence structure of the query and achieves a better result. Other
unsupervised methods such as graph-based models includes TextRank [11], RAKE
[17], SingleRank [25], etc. They are developed based on the linking-structure
design of PageRank.

On the other hand, early supervised methods such as the Naive Bayes machine
learning algorithm are used in KEA [26]. Deep learning models emerge neural IR
since [8]. They rank the relevance scores of the given query and documents by
their similarity through DNN. Various deep learning models dedicate to finding
more relations between the query and documents in semantic space.
Representation-based models such as [19] use CNN to obtain feature vectors.
Ranking models rely deeply on feature engineering these days.

Interaction-based models such as DRMM [6] focus on the relevance matching,
[27] use a kernel-based model to rank the scores and [14] combine CNN and RNN
to find the relevance representation. Recently, [13] use a pre-train model [4]
to fine-tune the task. In particular, models build upon classification tasks
that originate from similar ideas. [28] use BLSTM to output key phrases and
[22] fine-tuned BERT with an attention layer for keyword extraction.

###  2.2. NLP representation learning

The feature engineering of NLP had evolved rapid-ly since AI emerged in the
1960s. It was successfully applied to machine translation via BLSTM until
word2vec [12] was released. It transforms the vocabulary space into the
feature representation space and makes NLP training possible with deep
learning models. In addition, embedding vectors of word2vec have cosine
similarity if words have similar meanings. GloVe [15] calculates the co-
occurrence between tokens inside a certain window size. Similar words are
frequently surrounded by the same words. Like word cloud. ELMo [23]
concatenates forward and backward LSTM to capture the long sequence bi-
direction relation. Moreover, BERT [4] surprises the world by sweeping all the
performance in NLP tasks. The masking token technics induce a cloze test
effect for the machine learning. It has different influences between
languages. It can be regarded as an NLU learning in English. In mandarin,
terms or phrases with comment words have similar vector embeddings. However,
this is not always true. OpenAI uses the decoder of Transformer which only
extracts the forward sequential relation. GPT2 uses Pre-LN and obtains a slice
better result in semantic tasks. Admin [10] analyzes the influence between
Post-LN and Pre-LN. Post-LN has a gradient vanishing problem in the decoder
which may cause the performance in machine translation.

In E-commence, especially in Mandarine, there are a lot of new terms. BERT is
built by character-level which assembles characters to get new terms. This is
very convenient for most users. However, terms that have no-repeating
characters may have similar meanings but embeddings may not have cosine
similarity. If a vector embedding model can produce new terms easily and has
cosine similarity corresponding to their literal, then it may be an
outstanding embeddings system. We have some effort in this direction and will
publish it soon.

In our experiments, we use word2vec [12] as our embeddings. It performs
slightly better than BERT [4].

###  2.3. Contrastive Learning

An objective function plays an important role in supervised learning. It
directly LeCun etc. introduced the contrastive loss function in [7]. It is
well known as triplet loss [18] in image classification. Given a pair
$\\{x_{+},x_{-}\\}$ of a positive and negative sample and a training sample
$x$, the object is to make $x$ look similar to $x_{+}$ and not similar to
$x_{-}$ if $x$ is labeled as positive ($y=1$). The triplet loss is formulated
as

$L=\max(0,\|x-x_{+}\|^{2}-\|x-x_{-}\|^{2}+\alpha)$

for binary classification. The model is supervised to learn to draw close to
$x_{+}$ and push away from $x_{-}$ with a marginal distance $\alpha$. The
feature space is measured in $L_{2}$-norm. It is applied to multiple
classification [21]. The $(N+1)$-tuplet loss is to optimize identifying a
positive sample from $N$ negative samples. It is

$L(x_{i},x_{j})={\bf 1}\\{y_{i}=y_{j}\\}\|x_{i}-x_{j}\|^{2}+{\bf
1}\\{y_{i}\neq y_{j}\\}\max(0,\alpha-\|x_{i}-x_{j}\|^{2})$

When $N=2$, it is equivalent to triplet loss. Since it is not easy to train an
NLP model by using $L_{2}$-norm, we prefer to use cross-entropy in binary
classification:

$L=y\log p+(1-y)\log(1-p),$

where $y=1$ if $x$ is positive and $y=0$ if $x$ is negative, and $p$ is the
prediction of $x$. For multiple classification, we formulate it in equation 7.
We discuss the objective functions in detail in section 4.1

##  3\. Model Architecture

BAT is a model reconstructed from Evolved Transformer’s architecture [20]. We
use the encoder for text classification. The complete encoder-decoder
structure is designed for machine translation and other tasks that match
different lengths of input and output sequences. Attention mechanisms in RNN
models [2] integrate the total inference of the input sequence with each
output which accelerates the development of progress in NLP. Transformer
adopts full attention and exempts from sequential input which makes a leap of
development in deep learning. It dominates machine learning in all aspects.
Many models are developed from it.

The architecture of BAT is demonstrated in Figure 4.

###  3.1. Input

The encoder imports a sequence of tokens $\\{w_{1},\cdots,w_{n}\\}$ with
vector embedding $\\{{\bf x}_{1}$, $\cdots,{\bf x}_{n}\\}$ respectively. Since
Transformer uses position encoding to replace the sequential input, the
encoder can input the data $X=[{\bf x}_{1},\cdots,{\bf x}_{n}]^{T}$
$\in\mathbb{R}^{n\times d_{e}}$ and output the probability $[{\bf
p}_{1},\cdots,{\bf p}_{n}]^{T}\in\mathbb{R}^{n\times C}$ for each token at the
same time.

Transformer [24] adds up the position encoding and embedding vectors to be the
feature representation. One can change it to a linear or affine linear
combination for richer inputs. GPT2 [16] moves the layer normalization (LN)
before the self-attention block and obtains a better result. Admin [10] shows
that Pre-LN maintains steady gradient distribution through each layer in the
decoder but does not affect much in the encoder. We have an ablation
experiment for a text classification task with a 4-layer encoder. Pre-LN
obtains a slice better result.

Figure 4. Encoder of BAT

###  3.2. Encoder

The encoder is composed of a stack of identical layers (4 layers for
commercial requirements). Each layer consists of two sublayers: attention and
feed-forward tournament selection sublayers (ATS and FFTS). The competition
forms differently: MFSA races with constant residual connection and FFNN races
with a weighted residual connection. The two parallel branches compete through
backpropagation to update parameters for better performance.

####  3.2.1. ATS: Attention Tournament Selection sublayer

ATS sublayer contains a multi-feature self-attention block (MFSA) and a
residual connection in parallel. Inspired by [20], we extract multiple feature
from the input: multi-head self-attention, $1\times 1$-convolution, and
$3\times 1$-convolution in parallel. It can formulated as

${\rm MFSA}=f_{\rm LN}\left({\rm MultiHead}(Q,K,V)+f_{1\times 1{\rm
conv}}(X)+f_{3\times 1{\rm conv}}(X)\right).$ (5)

####  3.2.2. MFSA: Multi-feature self-attention

MFSA adds up three features:

  * •

The Multi-head attention is directly followed from Transformer [24]. It
captures the entire bidirectional relation of the whole sequence. We quote it
to recall the importance.

The model space is divided and transformed to $h$ multi-head subspaces to
extract different features. Each subspace is performed with the scaled dot-
product attention [24] separately: Let $d_{e}$ denote the dimension of the
model space (embedding space), $d_{\rm k}=d_{e}/h$ denote the dimension of
each head and $h$ be the head number.

${\rm Attention}(Q,K,V)={\rm softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V,$ (6)

where $Q=XW_{Q},K=XW_{K},V=XW_{Q}\in\mathbb{R}^{n\times d_{\rm k}}$ and $X$ is
the input feature map with sequence length $n$. The compressing weights
$W_{Q}$, $W_{K}$, $W_{V}$, which transform the data from the model space to
each head-subspace, are learnable parameters for the attention mechanism.The
Multi-head attention consists $h$ scaled dot-product attention blocks running
in parallel. The output is

${\rm MultiHead}(Q,K,V)={\rm concat}({\rm head}_{1},\cdots,{\rm
head}_{h})W_{O},$

where ${\rm head}_{i}={\rm Attention}(Q_{i},K_{i},V_{i})$ for $i=1,\cdots,h$
and $W_{O}\in\mathbb{R}^{d_{e}\times d_{e}}$.

  * •

The $1\times 1$-convolution followed by the activation ReLU captures the
relation of oneself. The activation ReLU plays a role as a dropout.

  * •

The $3\times 1$-convolution followed by the activation ReLU captures the
relation between itself and the context words. This represents a local
feature.

####  3.2.3. FFTS: Feed-Forward Tournament Selection sublayer

FFTS is created for the competition between two branches: FFNN and weighted
residual connection. An ablation experiment shows that FFNN needs support from
the weighted res-connection to get better performance.

Based on our training results, we discover that the dropout does not affect
much in our network (0.2%). It is removed from our model. The other
architecture remains the same as [24].

##  4\. Training

Suppose that training data $\\{({\bf x}_{i},{\bf y}_{i})\\}_{i=1}^{N}$ is a
set of pairwise vectors, where ${\bf x}_{i}\in\mathbb{R}^{d_{e}}$ is the
embedding of word $w_{i}$ and ${\bf y}_{i}\in\mathbb{R}^{C}$ is its label.
Labels are defined by humans.

###  4.1. Training loss

An objective function affects the performance of an ML model. We compare PBP,
CECL, and CECLA loss with CE loss as followings:

  * •

The categorical cross-entropy is

$L_{\rm CE}=-{\bf y}\log{\bf p}.$

It does not punish for false prediction. For example, suppose that GT is ${\bf
y}=[1,0,0]$ with the two predictions ${\bf p}_{1}=[0.4,0.3,0.3]$ and ${\bf
p}_{2}=[0.4,0.5,0.1]$. They have the same loss $-\log(0.4)$. Though ${\bf
p}_{1}$ gets the right class and ${\bf p}_{2}$ does not. It has less chance
for the network to update the parameters to get the right prediction.
Furthermore, in imbalanced data, categorial cross-entropy can not achieve the
optimal. See Appendix A.1 for details.

  * •

The categorical cross-entropy with contrastive learning:

$L_{\rm CECL}=-\left({\bf y}\log{\bf p}+({\bf 1}-{\bf y})\log({\bf 1}-{\bf
p})\right).$ (7)

This loss function helps the model to learn more efficiently. In the previous
example, it obtains different values $L_{\rm CECL}({\bf
p}_{1})=-\log(0.4)-\log(0.3)-\log(0.3)=0.71$ and $L_{\rm CECL}({\bf
p}_{2})=-\log(0.4)-\log(0.5)-\log(0.3)=0.74$. The wrong prediction gets larger
loss. Especially when one calculates the gradient of it, it modifies each
prediction in each category. Please see Appendix A.1 for details.

  * •

The PBP loss is given as

$L_{\rm PBP}=-\left({\bf y}\log{\bf p}+({\bf 1}-{\bf y})\odot\hat{{\bf
y}}\log({\bf 1}-{\bf p})\right),$

where $\hat{{\bf y}}$ the predict class of ${\bf p}$, which is a one-hot
vector. It focuses on punishing the bad prediction. We get $L_{\rm PBP}({\bf
p}_{1})=-\log 0.4=0.4$ and $L_{\rm PBP}({\bf p}_{2})=-\log 0.4-\log 0.5=0.7$.
This loss converges faster then $L_{\rm CE}$ and $L_{\rm CECL}$.

###  4.2. Why new balanced weights

Appropriate weight for a loss function is the key to supervising a model,
especially in an imbalanced dataset. Suppose that the numbers of each category
in the dataset are $N_{1}:N_{2}:\cdots:N_{C}$ and $\sum_{i=1}^{C}N_{i}=N$. For
example, if the distribution of the whole data is $N_{1}:N_{2}:N_{3}=20:2:1$,
then the cross-entropy is

$L_{\rm CE}=-\sum_{i=1}^{20}\log p_{i,1}-\sum_{i=21}^{22}\log
p_{i,2}-\sum_{i=23}^{23}\log p_{i,3}.$

That means that the model gets more loss and gradient from the major data.
Therefore, it learns faster and predicts more accurately than the major data.
Especially in keyword extraction, valuable words are much fewer in the whole
data. It encourages us to find appropriate coefficients on the objective
function for imbalanced data.

  * •

The weighted categorical cross-entropy is given by

$L_{\rm WCE}=-\frac{1}{N}\sum_{i=1}^{N}\boldsymbol{\alpha}{\bf y}_{i}\log{\bf
p}_{i},$

which is a method to control the fair distribution of gradient in each
category for imbalanced data. Equation (15) in Appendix A.1 shows that the
optimal weight is

$\boldsymbol{\alpha}=(\frac{N}{N_{1}},\cdots,\frac{N}{N_{C}}).$

It can be normalized by any scaling factor. However, we notice that training
cost for a single sample from the micro data might be more than it from the
major data. For example, if the distribution of the whole data is
$N_{1}:N_{2}:N_{3}=20:2:1$ and the prediction is ${\bf p}=(0.1,0.5,0.4)$, then

$\displaystyle-\frac{N}{N_{1}}(\log p_{1})=1.15\quad$ if $y=(1,0,0)$
$\displaystyle<-\frac{N}{N_{2}}(\log p_{2})=3.45\quad$ $\displaystyle\text{if
$y=(0,1,0)$}.$

Since the cost for getting a correct prediction in the second category is
higher than getting a false prediction in the first category, the model is
easier to get $\\{T=1,P=2\\}$ than $\\{T=2,P=2\\}$ if the parameters updated
in a batch that the data distribution is different from the whole data. From
Figure 5, we may see that $L_{\rm WCE}$ predicts more counts on microdata,
especially in the early stage. In this case, its $F_{2}$ scores in microdata
are relatively high. Thus $L_{\rm WCE}$ needs to train on a certain amount of
batches such that the distribution is close to the whole data or train on
certain epochs to obtain higher $F_{1}$ scores on microdata.

Figure 5. prediction on category numbers

  * •

Weighted cross-entropy with contrastive learning: In our setup, we have two
weight vectors $\boldsymbol{\alpha},\boldsymbol{\beta}$:

$L_{\rm WCECL}=-\frac{1}{N}\sum_{i=1}^{N}\left(\boldsymbol{\alpha}{\bf
y}_{i}\log{\bf p}+\boldsymbol{\beta}({\bf 1}-{\bf y}_{i})\log({\bf 1}-{\bf
p}_{i})\right).$

The standard weight for contrastive learning is

$\begin{cases}\begin{array}[]{ll}\boldsymbol{\alpha}_{\rm
st}&=(\frac{N-N_{1}}{N},\cdots,\frac{N-N_{C}}{N})\\\ \boldsymbol{\beta}_{\rm
st}&=(\frac{N_{1}}{N},\cdots,\frac{N_{C}}{N})\end{array}\end{cases},$ (8)

if we realize the weights are in an inverse ratio with the data. That is, for
class $i$, the ratio of positive samples and negative samples is
$N_{i}:N-N_{i}$ and the weights in an inverse ratio are
$\alpha_{i}:\beta_{i}=N-N_{i}:N_{i}$. We can normalize it to get (8).

If we consider the gradient for updating the parameters (16) in A.1 for
$\gamma=0$, then we get

$\frac{\partial L}{\partial
p_{j}}=-\frac{\alpha_{j}N_{j}}{N}\frac{1}{p_{j,+}}+\frac{\beta_{j}(N-N_{j})}{N}\frac{1}{(1-p_{j,-})}=0$

for each category. In particular, $L_{\rm WCECL}$ suggest that the EE (Equal-
Expectation) weights

$\begin{cases}\begin{array}[]{ll}\boldsymbol{\alpha}_{EE}&=(\frac{N}{N_{1}},\cdots,\frac{N}{N_{C}})\\\
\boldsymbol{\beta}_{EE}&=(\frac{N}{N-N_{1}},\cdots,\frac{N}{N-N_{C}})\end{array}\end{cases}$

for $\frac{\partial L}{\partial p_{j}}=0$ with $p_{j}=\frac{1}{2}$. This
weight advantages in imbalanced data. In fact, it has equal expectation for
each category both in positive samples and negative samples if one views
$\boldsymbol{\alpha}_{EE}$ and $\boldsymbol{\beta}_{EE}$ as probability of
samples.

However, suppose that $N_{i}<N_{j}<N-N_{j}<N-N_{i}$. That is, the data in
class $i$ is more imbalanced than class $j$. Compared to the standard weights,
the proportion of the positive weight with negative weight is the same

$(\alpha_{EE,i}:\beta_{EE,i})=\frac{N}{(N-N_{i})N_{i}}(\alpha_{\rm
st,i}:\beta_{\rm st,i}).$

$\frac{N}{N_{i}(N-N_{i})}>\frac{N}{N_{j}(N-N_{j})}$ reveals that a single
token from microdata gets more punishment than major data, although the
proportion $\alpha_{i}:\beta_{i}$ is the same. The model predicts more numbers
on microdata via EE weights than standard weights in the early stage but the
other around at 100 epochs. It obtains almost the same $F_{1}$ scores but
higher $F_{2}$ scores. See the results of the experiments 5.4.

  * •

Array weight:
For $L_{\rm WCECL}$, all negative samples get the same cost even if they
belong to different categories. Moreover, if ${\bf y}$ is in class $c$, it
receives different loss for $p_{j}$, $j\neq c$ in $L_{\rm WCECL}$.

To make them get different amounts of loss value according to their proportion
to the whole data, let the array weights

$A=\begin{pmatrix}\alpha_{1}&\beta_{1,2}&\cdots&\beta_{1,C}\\\
\beta_{2,1}&\alpha_{2}&\cdots&\beta_{2,C}\\\ \vdots&&\ddots&\\\
\beta_{C,1}&&&\alpha_{C}\end{pmatrix},$

where $A_{ii}=\alpha_{i}=\frac{N}{N_{i}}$,
$A_{ij}=\beta_{i,j}=\frac{N}{(C-1)N_{j}}$ for $j\neq i$. $L_{\rm CECLA}$
obtains higher $F_{1}$ and $F_{2}$ scores than $L_{\rm WCECL}$.

  * •

Multiplication with $\boldsymbol{\alpha}$:
In order to enlarge the range of probability $p_{j}$ such that gradient
$\frac{\partial L}{\partial p_{j}}<0$ keeps descending, we multiply a factor
$\lambda$ with $\boldsymbol{\alpha}$ in equation (16) (focal factor
$\gamma=0$, array weight)

$\frac{\partial L}{\partial
p_{j}}=-\frac{\lambda}{p_{j,+}}+\frac{1}{1-p_{j,-}}.$ (9)

If $\lambda=9$, $\frac{\partial L}{\partial p_{j}}<0$ for $0<p_{j}<0.9$. If
$\lambda=12$, $\frac{\partial L}{\partial p_{j}}<0$ for
$0<p_{j}<\frac{12}{13}$.

###  4.3. Learning rate strategy

Transformer adapts the learning rate according to the following formula [24]:

$lrate=d_{\rm model}^{-0.5}\cdot\min({\rm step\\_num}^{-1/2},{\rm
step\\_num}\cdot{\rm warmup\\_steps}^{-1.5}),$ (10)

with ${\rm warmup\\_step}=4000$. Although BAT is a model with large
parameters, it converges faster than other SOTA models. For such a model, we
may not need to change its learning rate. However, for a model that converges
too slow or needs a best result at a certain epoch, it may choose to adjust
(10) for its convergence.

For a single category, any F score is formulated as

$F_{\beta}=\frac{(1+\beta^{2})pr}{\beta^{2}p+r},$

where $p$ denotes the precision and $r$ denotes the recall. If we maintain
${\rm F_{1}}$ (on the level curves of ${\rm F_{1}}$), then $\frac{\partial
r}{\partial p}=\frac{2r-F_{1}}{F_{1}-2p}<0$. Under this condition, precision
and recall are negative relative to each other. Moreover, we have

$F_{1}-F_{2}=\frac{3pr(p-r)}{(p+r)(4p+r)}.$

${\rm F_{2}}$ is negative relative to ${\rm F_{1}}$ scores. Models get
increasing ${\rm F_{1}}$ scores and decreasing ${\rm F_{2}}$ scores along with
training. However, beyond the overfitting point, both F scores decrease. We
may see Figure 1 for this trend.

For different industry requirements, one can choose a learning rate strategy
to satisfy the target. The different industries may pay attention to different
F scores. Medical business or safety-related domains may prefer perfect
precision. Ads or SEO may prefer higher recall scores. Different industries
need different loss functions to supervise their models. We favor higher ${\rm
F_{2}}$ scores and people care about ${\rm F_{1}}$ scores in usual
classification tasks. We pick the relative maximum of ${\rm F_{2}}$, which is
not the lowest value of the loss functions. If one needs this point happens at
certain epochs, he may adjust the learning rate to make the model converges
faster or slower.

Here are some tips for getting higher F scores.

  * •

Let $x={\rm step\\_num}$ and $S={\rm warmup\\_steps}$. Given $\alpha>0$, we
adjust the learning rate

$lrate=\lambda\cdot d_{\rm model}^{-1/2}\cdot\min(S^{\beta}\cdot
x^{-\alpha},S^{-3/2}\cdot x),$ (11)

by finding a $\beta$ to satisfy the continuous property

$S^{\beta}\cdot x^{-\alpha}=S^{-3/2}\cdot x.$

  * •

The larger warmup steps one obtains, the lower learning rate one gets in the
warming up steps. A lower learning rate induces a finer convergence. However,
if it is too low, you may not reach the goal in an efficient time. However,
it’s not that the higher the better. It depends on the models.

  * •

A relative higher $\alpha$ in (11) leads to faster convergence and higher
${\rm F_{1}}$ scores.

  * •

$\lambda$ controls the entire value of the learning rate. We believe that a
lower learning rate induces finer convergence.

For the text classification task, we choose ${\rm warmup\\_steps}=4000$ for
our 4-layer encoder. Here are some examples: For $L_{\rm CECLA_{1,20}}$, 50
epochs

model | v1 | v2 | v3 | v3$\ast 0.999$ | v4$\ast 1.001$ | v5
---|---|---|---|---|---|---
Micro F1 | 94.7 | 94.7 | 94.8 | 94.6 | 94.7 | 94.6
Macro F1 | 85.3 | 85.5 | 85.6 | 85.3 | 85.4 | 85.0
Macro F2 | 87.8 | 87.7 | 87.6 | 88.1 | 87.9 | 87.3

  * •

v1 represents the original learning rate (10).

  * •

v2 represents $\alpha=6/11$, $\beta=1/22$. Roughly speaking, v2 converges
slower than v1 due to the learrning rate power $6/11<1/2$.

  * •

v3 represents $\alpha=6/13$, $\beta=-1/22$. v3 converges faster than v1. v3 at
50 epochs is almost same with v1 at 60 epochs. ${\rm v3\ast 0.999}$ converges
slower than v3 and has higher ${\rm F_{2}}$ scores.

  * •

v4 represents $\alpha=7/15$, $\beta=-1/30$. v4 converges slower than v1. When
v4 multiply a factor $1.001$, it forces v4 to converges faster than itself.
${\rm v4\ast 1.001}$ obtains its optimal at 40 epochs with macro ${\rm
F_{1}}=85.7$ and ${\rm F_{2}}=88.1$.

  * •

v5 represents $\alpha=11/20$, $\beta=1/20$. v5 coverges faster than v1. It
obtains its optimal at 40 epochs with macro ${\rm F_{1}}=85.5$ and ${\rm
F_{2}}=88.1$.

Figure 6. Learning rate comparison for CECLA loss

  * •

${\rm v3\ast 0.999}$ has almost the same ${\rm F_{1}}$ with v1 but better
${\rm F_{2}}$ than v1. ${\rm v4\ast 1.001}$ performs much better then v1 most
of the time.

  * •

The best result for $L_{\rm CECLA_{1,20}}$ is ${\rm v4\ast 1.001}$ at 40
epochs.

For PBP loss at 50 epochs:

model | v1 | v2*1.001 | v4$\ast 1.001$ | v4$\ast 1.0011$ | v5
---|---|---|---|---|---
Micro F1 | 94.8 | 94.8 | 94.6 | 94.6 | 94.9
Macro F1 | 85.6 | 85.7 | 85.5 | 85.2 | 85.8
Macro F2 | 87.7 | 87.7 | 87.8 | 87.9 | 87.3

  * •

At 50 epochs, ${\rm v2\ast 1.001}$ performs better than v1.

  * •

v5 obtains the best macro ${\rm F_{1}}$ scores.

  * •

${\rm v4\ast 1.0011}$ obtains the optimal for macro ${\rm F_{1}}=85.6$ and
${\rm F_{2}}=88.0$ at 40 epochs.

  * •

For an even faster convergence, by adding focal factor $\gamma=1$, ${\rm
v3\ast 0.999}$ obtains macro ${\rm F_{1}}=85.5$ and ${\rm F_{2}}=88.1$ at 30
epochs.

Figure 7. Learning rate comparison for PBP loss

Play with functions (11) to obtain the best learning rate for your own model.

##  5\. Experiments

Variations on the BAT architecture

N | $d_{\rm model}$ | $d_{ff}$ | h | $d_{k}$ | $d_{v}$ | $p_{\rm drop}$ | params
---|---|---|---|---|---|---|---
4 | 128 | 512 | 4 | 32 | 32 | 0 | 0$1.316\times 10^{6}$

$N$ is the number of encoders.

###  5.1. Training Data

We have several training datasets. The EC dataset contains product pages which
consist of titles and descriptions. We clean the data with a list of
regulations and stop words for our embedding system.

  1. (1)

EC-Zh, which is collected from 5 E-commence platforms including cosmetics,
furniture, fashion, shoes, and large retail store. The product pages are
written in traditional Chinese. It contains 2000 product pages from each
platform and is divided into 80% for training and 20% for testing. The average
length is 46.6 words per page and the longest page is 368 in the training set.
This dataset has high variance and is challenging for text classification.

  2. (2)

EC-JP, which is collected from 11 E-commence platforms including fashion,
decorate, farm, and multiple categories. The product pages are written in
Japanese. We remove page lengths over 400 due to the efficiency of model
training. The training data is twice of EC-Zh. The average length is 30 words
per page.

####  5.1.1. Hardware

GPU: 1080 Ti and 1080.

####  5.1.2. Optimizer

All experiments are trained with optimizer Adam [9] with $\beta_{1}=0.9$,
$\beta_{2}=0.98$, and $\epsilon=10^{-9}$. We adapt the base learning rate (10)
with ${\rm warmup\\_steps}=4000$.

####  5.1.3. Regularization

All dropout are removed.

###  5.2. Accuracy Comparison with SOTA

First, we compare the performance of BAT with Transformer [24], Switch
Transformer [5], Admin [10], MUSE [29], Performer [3], and Informer [30] in
EC-Zh dataset with three loss functions. We use the official or open-source
from the third place for a fair comparison. In the task of multi-class
classification, we extract only extract the encoder and add a classification
layer classification on top of it.

The distribution in our training data (EC-Zh) is
$N_{1}:N_{2}:N_{3}=21.2:1.9:1$. All models are built with 4-layer, 4 heads,
and no dropout. The following experiments are trained under the same
conditions: Adam optimizer, learning rate strategy, batch=8, 50 epochs. We use
micro-${\rm F_{1}}$ score, macro ${\rm F_{1}}$, macro ${\rm F_{2}}$ as the
metric of accuracy comparison. The setup for the SOTA models is as follows:

  * •

Switch [5]: The best performance is $(E,C)=(50,1)$, which $E$ is close to the
average page length.

  * •

MUSE: We adapt the complete version of MUSE.

  * •

Performers: We use the positive random kernel with redraw projection in the
official release code. Let the dimension of the kernel space $m=16$.

  * •

Informer: Redraw per page for each head to calculate the multi-head ProbSparse
attention, since we run experiments on the NLP dataset. We also remove the
time-stamp input and the distilling layer to not reduce the attention map to
predict each token.

The accuracy comparison for the classification task is as following:

####  5.2.1. Accuracy comparison micro ${\rm F_{1}}$

At 50 epochs

model | BAT | Admin | Switch | MUSE | Transformer | Performers | Informer
---|---|---|---|---|---|---|---
${\rm CE}$ | 95.2 | 94.9 | 94.1 | 94.8 | 94.6 | 94.8 | 94.9
${\rm WCE}$ | 94.7 | 94.2 | 91.7 | 92.3 | 93.8 | 93.6 | 92.6
${\rm PBP_{0,8}}$ | 94.8 | 94.2 | 91.2 | 93.8 | 93.7 | 93.3 | 92.5
${\rm CECLA_{1,20}}$ | 94.7 | 94.5 | 88.5 | 93.7 | 93.5 | 92.8 | 92.4

####  5.2.2. Accuracy comparison macro ${\rm F_{1}}$

At 50 epochs

model | BAT | Admin | Switch | MUSE | Transformer | Performers | Informer
---|---|---|---|---|---|---|---
CE | 85.7 | 84.5 | 82.1 | 84.2 | 84.0 | 83.9 | 84.6
WCE | 85.5 | 84.0 | 79.6 | 81.2 | 82.9 | 83.0 | 80.8
${\rm PBP_{0,8}}$ | 85.6 | 84.0 | 79.1 | 84.0 | 82.9 | 82.7 | 81.2
${\rm CECLA_{1,20}}$ | 85.3 | 84.3 | 75.0 | 83.7 | 82.4 | 82.1 | 80.9

####  5.2.3. Accuracy comparison macro ${\rm F_{2}}$

At 50 epochs

model | BAT | Admin | Switch | MUSE | Transformer | Performers | Informer
---|---|---|---|---|---|---|---
CE | 85.7 | 83.7 | 81.8 | 83.4 | 83.8 | 82.8 | 84.0
WCE | 87.9 | 86.2 | 84.6 | 87.2 | 85.5 | 86.4 | 85.6
${\rm PBP_{0,8}}$ | 87.7 | 86.4 | 84.5 | 87.6 | 85.5 | 86.4 | 86.0
${\rm CECLA_{1,20}}$ | 87.8 | 86.0 | 82.5 | 87.7 | 85.3 | 86.2 | 86.2

Figure 8. CECLA loss SOTA comparison

Figure 9. PBP loss SOTA comparison

Figure 10. WCE loss SOTA comparison

Figure 11. CE loss SOTA comparison

BAT is a model that converges faster than other SOTA models for the overall
performance. BAT obtains the best F scores at 50 epochs and the scores are
better than other SOTA models either with their 4-layer or deep layer
structure at 100 epochs. See Figure 8, 9, 10, and 11 for the complete
information of 4 loss functions.

####  5.2.4. EC-JP dataset

The distribution in the EC-JP dataset is $N_{1}:N_{2}:N_{3}=9.57:1:1.01$. All
models are built with 4-layer, 4 heads, and no dropout. The following
experiments are trained under the same conditions: Adam optimizer, learning
rate strategy, batch=8, and trained for 50 epochs. We use micro ${\rm F_{1}}$,
macro ${\rm F_{1}}$, macro ${\rm F_{2}}$ scores as the metric of accuracy
comparison.
Micro ${\rm F_{1}}$

model | BAT | Admin | MUSE | Transformer | Performers
---|---|---|---|---|---
${\rm CE}$ | 93.3 | 93.0 | 92.7 | 92.7 | 92.6
${\rm WCE}$ | 92.4 | 91.9 | 91.3 | 91.7 | 89.0
${\rm PBP_{0,12}}$ | 92.3 | 91.0 | 90.6 | 91.2 | 88.5
${\rm CECLA_{1,20}}$ | 92.6 | 91.3 | 89.9 | 89.9 | 88.6

Macro ${\rm F_{1}}$

model | BAT | Admin | MUSE | Transformer | Performers
---|---|---|---|---|---
CE | 85.8 | 84.8 | 84.7 | 84.4 | 84.8
WCE | 85.4 | 84.6 | 84.8 | 83.9 | 81.2
${\rm PBP_{0,12}}$ | 85.4 | 83.7 | 83.2 | 83.6 | 80.9
${\rm CECLA_{1,20}}$ | 85.6 | 83.9 | 82.6 | 82.1 | 80.9

Macro ${\rm F_{2}}$

model | BAT | Admin | MUSE | Transformer | Performers
---|---|---|---|---|---
CE | 86.1 | 84.2 | 85.1 | 84.4 | 85.6
WCE | 88.3 | 87.7 | 87.3 | 87.5 | 86.5
${\rm PBP_{0,12}}$ | 88.4 | 87.6 | 87.8 | 86.9 | 86.4
${\rm CECLA_{1,20}}$ | 88.3 | 87.5 | 87.5 | 86.7 | 86.3

We adopt the learning rate (10) by multiplying 0.93 for CE loss. For $L_{\rm
CECLA_{1,20}}$ loss, we adopts $\alpha=6/11$ and $\beta=1/22$ for (11).

####  5.2.5. Layers comparison on EC-Zh

Experiments are done with the same conditions: loss function $L_{\rm
CECLA_{1,20}}$, batch=8, 100 epochs. The Macro ${\rm F_{1}}$ are as following:

model | BAT | Admin | Switch | MUSE | Transformer | Performers | Informer
---|---|---|---|---|---|---|---
4-layer | 85.7 | 83.9 | 79.1 | 84.4 | 83.0 | 82.7 | 82.8
6-layer | 85.4 | 82.9 | 76.1 | 84.2 | 82.9 | 83.7 | 83.6
12-layer | 85.5 | 83.6 | 80.2 | 85.4 | 82.1 | 82.9 | 83.4
24-layer | 85.3 | 83.4 | 71.0 | 85.2 | 33.9 | 84.3 | 82.3

The Macro ${\rm F_{2}}$ are as following:

model | BAT | Admin | Switch | MUSE | Transformer | Performers | Informer
---|---|---|---|---|---|---|---
4-layer | 87.2 | 85.5 | 84.1 | 86.9 | 85.0 | 85.3 | 86.5
6-layer | 87.1 | 85.5 | 82.7 | 86.7 | 84.2 | 85.3 | 86.4
12-layer | 86.9 | 85.5 | 85.1 | 86.7 | 85.3 | 86.0 | 86.3
24-layer | 86.8 | 86.2 | 78.3 | 86.2 | 37.1 | 87.0 | 86.5

The learning rate for 4-layer and 6-layer uses the original learning rate
(10). The 4-layer structure model balances the accuracy and efficiency well.
It serves as our online model.

We adjust the learning rate policy for BAT in higher layers. They are as
follows:

  * •

BAT: 12-layer: warmup=8000, ${\rm lrate}/2.999$; 24-layer: warmup=8000,
$\alpha=4/9$, $\beta=-1/18$, $\lambda=1/5.2$ in (11)

  * •

Transformer: 12-layer warmup=8000, ${\rm lrate}/3$. This setup is to reduce
the slope of the warmup stage lower than it in (10). 24-layer warmup=12000,
${\rm lrate}/10$ However, the model seems not to converge for 24-layer.

  * •

Admin: 12-layer; 24-layer: warmup=8000, ${\rm lrate}\ast\sqrt{2}$.

  * •

MUSE; Performers; Informer: 12-layer; 24-layer: warmup=8000, ${\rm lrate}/3$.

  * •

Switch: 12-layer: warmup=8000, ${\rm lrate}/3$; 24-layer: warmup=8000, ${\rm
lrate}/10$.

The structure of a model with a 60-layer structure is more sensitive to the
learning rate due to the deep-stacked structure. We might not get the best
policy for each model but we have tried several learning rates with a high
possibility of successful training.

Admin adds the initialization parameters and improves the performance in
higher layers. Switch and Performers maintain stable performance in all layers
structure. MUSE especially has ${\rm F_{1}}$ scores in 12 and 24 layers.

####  5.2.6. Speed comparison

training time per epochs under 4-layer, 4-heads

model | BAT | Transformer | Admin | MUSE | Switch | Performer | Informer
---|---|---|---|---|---|---|---
params$\times 10^{6}$ | 1.316 | 0.79 | 0.791 | 0.89 | 26.3 | 0.79 | 0.79
s/epoch | 44 | 26 | 45 | 58 | 74 | 20 | 90

###  5.3. Accuracy Comparison of loss functions

The experiments are done with batch =8 and 100 epochs. The distribution in our
training data (EC-Zh) is $N_{1}:N_{2}:N_{3}=21.2:1.9:1$.
50 epochs

model | CE | FCE | WCE | ${\rm PBP_{0,s}}$ | ${\rm PBP_{0,8}}$ | ${\rm PBPA_{0,8}}$ | ${\rm CLA_{1,20}}$ | ${\rm CL_{1,20}}$ | ${\rm CL_{1,s}}$
---|---|---|---|---|---|---|---|---|---
Micro F1 | 95.2 | 95.1 | 94.7 | 94.8 | 94.8 | 94.8 | 94.7 | 94.5 | 94.9
Macro F1 | 85.7 | 85.7 | 85.5 | 85.3 | 85.6 | 85.4 | 85.3 | 85.0 | 85.5
Macro F2 | 85.7 | 86.0 | 87.9 | 86.6 | 87.7 | 87.4 | 87.8 | 87.8 | 86.9

100 epochs

model | CE | FCE | WCE | ${\rm PBP_{0,s}}$ | ${\rm PBP_{0,8}}$ | ${\rm PBPA_{0,8}}$ | ${\rm CLA_{1,20}}$ | ${\rm CL_{1,20}}$ | ${\rm CL_{1,s}}$
---|---|---|---|---|---|---|---|---|---
Micro F1 | 95.0 | 95.1 | 94.8 | 94.8 | 94.7 | 94.7 | 94.8 | 94.5 | 94.8
Macro F1 | 85.7 | 85.6 | 85.7 | 85.2 | 85.3 | 85.3 | 85.7 | 85.0 | 85.3
Macro F2 | 85.8 | 85.4 | 87.1 | 86.0 | 86.8 | 86.8 | 87.2 | 86.9 | 86.9

${\rm CLA}_{1,20}$ represents focal CECL loss with array weight, $\gamma=1$
and multiplication factor $\lambda=20$. ${\rm CL}_{1,20}$ represents focal
WCECL loss with EE weight, $\gamma=1$ and multiplication factor $\lambda=20$.

${\rm PBP}_{0,8}$ stands for PBP loss with EE weight, $\gamma=0$ and
$\lambda=8$. ${\rm PBP_{0,s}}$ is the PBP loss with standard weight and
$\gamma=0$.

  * •

$L_{\rm CE}$ obtains the highest $F_{1}$ scores. However, it is more accurate
on the major data. Its macro $F_{1}$ on the micro data is almost the same with
$L_{\rm WCE}$ but $2\%\sim 4\%$ less than $L_{\rm WCE}$.

  * •

From equation (15), $L_{\rm WCE}$ achieves the optimal with weights
$\boldsymbol{\alpha}=(\frac{N}{N_{1}},\frac{N}{N_{2}},\frac{N}{N_{3}})$.
Appropriate loss weights help the model achieve better performance. It is much
more efficient than the focal weight.

  * •

$L_{\rm PBP}$ loss learns better with EE weight. $L_{\rm PBP_{0,8}}$ obtains
higher macro $F_{1}$ than $L_{\rm WCE}$ at 50 epochs. It converges faster than
other weighted loss functions. Its $F$ scores also drop fast at 100 epochs.

  * •

$L_{\rm CECL}$ loss learns better with array weight and especially for
$\gamma=1$ of focal loss.

###  5.4. Accuracy Comparison with $\boldsymbol{\beta}$ and weights

$L_{\rm WCE}$ attends to predict more counts and also has higher $F_{2}$
scores on the microdata, especially in the early stage. Since its gradient
only values the positive samples, microdata obtains more contribution. It
learns to predict more on macro data and reduce predicts on microdata along
with training. Refer to figure 5, 13, and 14 for detail. For contrastive
learning type of functions (with $\boldsymbol{\beta}$), their gradients have
more bounce back between positive and negative samples. They predict fewer
counts on the major data than $L_{\rm CE}$ loss but more counts than $L_{\rm
WCE}$ loss and the opposite on the microdata. $L_{\rm CECL}$ has more portion
with EE weight than standard weight on the microdata and it has less
prediction on the counts. Since the negative samples of $L_{\rm CECLA}$ depend
on categories, it predicts more counts than EE weight on the microdata.

$L_{\rm PBP}$ learns on the contrary due to its punishment effect. $L_{\rm
CECL}$ and $L_{\rm PBP}$ do not perform their best for $\lambda=1$ due to
contrastive learning. We find appropriate $\lambda$ by (18) and the
experiments are in section 5.5.

For $\gamma=0$ at 100 epochs, we have counts prediction:

loss | CE | WCE | ${\rm CL_{0,s}}$ | ${\rm CL_{0,1}}$ | ${\rm CLA_{0,1}}$ | ${\rm PBP_{0,s}}$ | ${\rm PBP_{0,1}}$ | ${\rm PBPA_{0,1}}$
---|---|---|---|---|---|---|---|---
$N_{1}=80448$ | 80378 | 79222 | 79247 | 79818 | 79512 | 79716 | 79383 | 79091
$N_{2}=7396$ | 7352 | 8352 | 8111 | 7715 | 7995 | 8035 | 8049 | 8421
$N_{3}=3801$ | 3915 | 4071 | 4287 | 4112 | 4138 | 3894 | 4213 | 4133

We may see that when an objective function predicts more counts on one
category it tends to get higher $F_{2}$ score on it. However, its precision
may not high as well.

model | CE | WCE | ${\rm CL_{0,s}}$ | ${\rm CL_{0,1}}$ | ${\rm CLA_{0,1}}$ | ${\rm PBP_{0,s}}$ | ${\rm PBP_{0,1}}$ | ${\rm PBPA_{0,1}}$
---|---|---|---|---|---|---|---|---
category1 $F_{1}$ | 97.3 | 97.2 | 97.2 | 97.2 | 97.0 | 97.1 | 97.1 | 97.1
category 1 $F_{2}$ | 97.3 | 96.7 | 96.7 | 96.9 | 96.5 | 96.9 | 96.7 | 96.6
category 2 $F_{1}$ | 76.5 | 77.1 | 77.1 | 76.6 | 76.1 | 76.4 | 76.2 | 76.6
category 2 $F_{2}$ | 76.3 | 80.0 | 79.3 | 77.6 | 78.6 | 78.4 | 78.2 | 79.1
category 3 $F_{1}$ | 83.2 | 82.7 | 81.6 | 81.7 | 81.5 | 82.2 | 82.0 | 81.4
category 3 $F_{2}$ | 83.9 | 84.5 | 84.7 | 83.7 | 85.3 | 82.8 | 84.6 | 83.5

###  5.5. Accuracy Comparison for different multiplication factor of
$\boldsymbol{\alpha}$

For ${\rm L_{CECLA}}$ loss (3), we multiply $\boldsymbol{\alpha}$ by a factor
$\lambda$ for $1\leq\lambda\leq 30$. This experiment is to verify that
enlarging the range of prediction (18) in appendix helps the improving the
accuracy. Experiments are done under batch=8 and 100 epochs.

model | ${\rm L_{CECLA_{1}}}$ | ${\rm L_{CECLA_{12}}}$ | ${\rm L_{CECLA_{18}}}$ | ${\rm L_{CECLA_{20}}}$ | ${\rm L_{CECLA_{22}}}$ | ${\rm L_{CECLA_{24}}}$ | ${\rm L_{CECLA_{30}}}$
---|---|---|---|---|---|---|---
Micro F1 | 94.6 | 94.4 | 94.7 | 94.8 | 94.7 | 94.9 | 94.7
Macro F1 | 85.1 | 84.8 | 85.2 | 85.7 | 85.3 | 85.5 | 85.2
Macro F2 | 86.7 | 86.8 | 86.8 | 87.2 | 86.9 | 86.7 | 86.6

${\rm L_{CECLA}}$ obtains best F scores for $\lambda=20$. These experiments
show that we do not need large $\lambda$ and they perform well for
$20\leq\lambda\leq 24$ .
For ${\rm L_{PBP}}$ loss and $\gamma=0$, we also multiply
$\boldsymbol{\alpha}$ by a factor $\lambda$ for $1\leq\lambda\leq 30$.
Experiments are done under batch=8 and 100 epochs.

model | ${\rm L_{PBP_{1}}}$ | ${\rm L_{PBP_{8}}}$ | ${\rm L_{PBP_{9}}}$ | ${\rm L_{PBP_{10}}}$ | ${\rm L_{PBP_{12}}}$ | ${\rm L_{PBP_{14}}}$ | ${\rm L_{PBP_{20}}}$ | ${\rm L_{PBP_{24}}}$ | ${\rm L_{PBP_{30}}}$
---|---|---|---|---|---|---|---|---|---
Micro F1 | 94.7 | 94.7 | 94.7 | 94.7 | 94.7 | 94.5 | 94.7 | 94.8 | 84.5
Macro F1 | 85.1 | 85.2 | 85.1 | 85.1 | 85.4 | 85.0 | 85.3 | 85.4 | 84.9
Macro F2 | 86.5 | 86.8 | 86.4 | 86.5 | 87.2 | 86.9 | 87.0 | 87.0 | 86.8

${\rm L_{PBP}}$ obtains its best F scores for $\lambda=12$ at 100 epochs and
$\alpha=8$ at 50 epochs. ${\rm L_{PBP}}$ also does not need large $\lambda$.

##  Appendix A Gradient of loss functions

###  A.1. Cross-Entropy with contrastive learning

The basic setup for the loss of categorical weighted focal cross-entropy with
contrastive learning is

$L_{\rm WFCECL}=-\frac{1}{N}\sum_{i=1}^{N}(\boldsymbol{\alpha}{\bf y}_{i}({\bf
1}-{\bf p}_{i})^{\gamma}\log{\bf p}_{i}+\boldsymbol{\beta}({\bf 1}-{\bf
y}_{i}){\bf p}_{i}^{\gamma}\log({\bf 1}-{\bf p}_{i})),$

where $\boldsymbol{\alpha},\boldsymbol{\beta},{\bf y}_{i},{\bf
p}_{i}\in\mathbb{R}^{C}$ and $C$ is number os categories. If we calculate the
gradient for a single token for which ${\bf y}$ is in class $c$ and
$\boldsymbol{\alpha}={\bf 1}$ and $\boldsymbol{\beta}={\bf 1}$, then we get

$\displaystyle\frac{\partial L}{\partial p_{c}}$
$\displaystyle=\gamma(1-p_{c})^{\gamma-1}\log
p_{c}-\frac{(1-p_{c})^{\gamma}}{p_{c}}<0$ if $y_{c}=1$ (12)
$\displaystyle\frac{\partial L}{\partial p_{j}}$ $\displaystyle=-\gamma
p_{j}^{\gamma-1}\log(1-p_{j})+\frac{p_{j}^{\gamma}}{1-p_{j}}>0$
$\displaystyle\text{if $y_{j}=0$}.$ (13)

The objective function teaches the model to increase $p_{c}$ in the correct
class and decrease $p_{j}$ for $j\neq c$. For $\boldsymbol{\beta}\neq{\bf 0}$,
this loss function supervise the model learn on a single token faster.

However, the backpropagation is performed per batch and the proportion of
numbers in different categories varies in batches. The distribution is also
different from the whole dataset. This affects the performance of the models
and becomes challenging in the task of classification.

Suppose that the numbers of each category in the dataset are
$N_{1}:N_{2}:\cdots:N_{C}$ and $\sum_{i=1}^{C}N_{i}=N$. Furthermore, suppose
that all predictions for positive and negative categories are the same to
simplify the analysis.

We get

$L=-\frac{1}{N}\sum_{j=1}^{C}\left(\alpha_{j}N_{j}(1-p_{j,+})^{\gamma}\log
p_{j,+}+\beta_{j}(N-N_{j})p_{j,-}^{\gamma}\log(1-p_{j,-})\right),$ (14)

where $p_{j,+}$ is the prediction for positive samples in class $j$ and
$p_{j,-}$ is the prediction for negative samples for class $j$, $1\leq c\leq
C$. The loss function is a decreasing function if $\boldsymbol{\beta}={\bf 0}$
and is a increasing function if $\boldsymbol{\alpha}={\bf 0}$.

  * •

For categorical cross-entropy $L_{\rm CE}$ ($\boldsymbol{\beta}={\bf 0}$,
$\boldsymbol{\alpha}={\bf 1}$ and $\gamma=0$), solve by Lagrange multiplier
under the condition $p_{1}+p_{2}+\cdots+p_{C}=1$ (this condition comes from
the prediction of softmax)

$\frac{\partial L}{\partial p_{j}}=-\frac{N_{j}}{N}\frac{1}{p_{j}}$

for $j=1,\cdots,C$. If the data is very unbalanced, this loss can not get the
optimal. For example: If $N_{1}=20N_{2}$, then $p_{1}=20p_{2}=1$ and
$p_{2}=0.05$ can’t get the right class.

  * •

For weighted cross-entropy $L_{\rm WCE}$ ( $\boldsymbol{\beta}={\bf 0}$ and
$\gamma=0$), Lagrange Multiplier gives

$\frac{\partial L}{\partial p_{j}}=-\frac{\alpha_{j}N_{j}}{N}\frac{1}{p_{j}}.$
(15)

It concludes that
$\boldsymbol{\alpha}=(\frac{N}{N_{1}},\cdots,\frac{N}{N_{c}})$ may get the
best prediction. It maintains almost the same ${\rm F_{1}}$ scores, but
increase ${\rm F_{2}}$ scores.

  * •

For $L_{\rm WFCECL}$ ( $\boldsymbol{\beta}\neq{\bf 0}$ and
$\boldsymbol{\alpha}\neq{\bf 0}$) , the gradient is

$\frac{\partial L}{\partial p_{j}}=\frac{\partial L}{\partial
p_{j,+}}+\frac{\partial L}{\partial p_{j,-}},$

where

$\displaystyle\frac{\partial L}{\partial p_{j,+}}$
$\displaystyle=\alpha_{j}\frac{N_{j}}{N}\gamma(1-p_{j,+})^{\gamma-1}\log
p_{j,+}-\alpha_{j}\frac{N_{j}}{N}\frac{(1-p_{j,+})^{\gamma}}{p_{j,+}}<0$
$\displaystyle\frac{\partial L}{\partial p_{j,-}}$
$\displaystyle=-\beta_{j}\frac{N-N_{j}}{N}\gamma
p_{j,-}^{\gamma-1}\log(1-p_{j,-})+\beta_{j}\frac{N-N_{j}}{N}\frac{p_{j,-}^{\gamma}}{1-p_{j,-}}>0.$
(16)

The gradient from positive samples is negative in order to minimize the loss.
Models learn to increase $p_{+}$. The gradient from negative samples is
positive to decrease $p_{-}$. However, when they mix together, models may not
know how to distinguish these two parts. Simplify the problem under two basic
setup:

    1. (1)

standard weight: $\alpha_{j}=\frac{N-N_{j}}{N}$ and
$\beta_{j}=\frac{N_{j}}{N}$ for $j=1,\cdots,C$.

    2. (2)

ee weight (Equal Expectation): $\alpha_{j}=\frac{N}{N_{j}}$ and
$\beta_{j}=\frac{N}{N-N_{j}}$ for $j=1,\cdots,C$.

In either case, the gradient becomes

$\frac{\partial L}{\partial p_{j}}=\left(\gamma(1-p_{j,+})^{\gamma-1}\log
p_{j,+}-\frac{(1-p_{j,+})^{\gamma}}{p_{j,+}}-\gamma
p_{j,-}^{\gamma-1}\log(1-p_{j,-})+\frac{p_{j,-}^{\gamma}}{1-p_{j,-}}\right)$
(17)

up to scaling. Notice that $\frac{\partial L}{\partial p_{j}}<0$ if
$p_{j}<\frac{1}{2}$ and $\frac{\partial L}{\partial p_{j}}>0$ if
$p_{j}>\frac{1}{2}$. To enlarge the range of prediction to get a negative
gradient, we multiply a constant $\lambda\geq 1$ with $\boldsymbol{\alpha}$.
In order to make the model to learn by decreasing the loss, we prefer to
maintain the gradient negative.

For $\gamma=1$, the gradient is

$\frac{\partial L}{\partial p_{j}}=\lambda\log
p_{j,+}-\lambda\frac{(1-p_{j,+})}{p_{j}}-\log(1-p_{j,-})+\frac{p_{j,-}}{1-p_{j,-}}.$
(18)

If $\lambda=20$, then $\frac{\partial L}{\partial p_{j}}<0$ for $p_{j}<0.831$.
From the experiments 5.5, we may not need large $\lambda$ and it performs well
for $18\leq\lambda\leq 24$.

###  A.2. Cross-Entropy with contrastive learning and array weight

In (14), we assume that all the prediction in the negative samples are the
same. Suppose that the prediction is corresponding to the distribution of
negative categories. In this case, we develop the loss weights that depend on
the distribution not only to its positive and negative samples but also to its
negative distribution in each category.

Let CECL loss with array weight be

$L_{\rm CECLA}=-\frac{1}{N}\sum_{i=1}^{N}{\bf a}_{i}\bigg{(}{\bf y}_{i}({\bf
1}-{\bf p}_{i})^{\gamma}\log{\bf p}_{i}+({\bf 1}-{\bf y}_{i}){\bf
p}_{i}^{\gamma}\log({\bf 1}-{\bf p}_{i})\bigg{)},$

where ${\bf a}_{i}=A{\bf y}_{i}$, which extracts the $j$-th column if ${\bf
y}_{i}$ is in class $j$. In particular, we set

$A=(A_{ij})=\begin{cases}\begin{array}[]{cl}\frac{N}{N_{i}}&\text{if $i=j$}\\\
\frac{N}{(C-1)N_{j}}&\text{if $i\neq j$}\end{array}\end{cases}.$ (19)

For example, If we have ten samples in three categories and the proportion is
$7:2:1$, then we get

$A=10\cdot\begin{pmatrix}1/7&1/4&1/2\\\ 1/14&1/2&1/2\\\
1/14&1/4&1\end{pmatrix}.$

The array according to $L_{\rm WCECL}$ is

$A=10\cdot\begin{pmatrix}1/7&1/3&1/3\\\ 1/8&1/2&1/8\\\
1/9&1/9&1\end{pmatrix}.$

For $\gamma=0$, if a token is in class $2$ and its prediction is ${\bf
p}=(1/4,1/2,1/4)$, then we get

$L_{\rm WCECL}=-1/2\log(1/2)-1/3\log(3/4)-1/9\log(3/4)$

and

$L_{\rm CECLA}=-1/2\log(1/2)-1/4\log(3/4)-1/4\log(3/4).$

$L_{\rm WCECL}$ obtains a different amount of loss from negative samples of
different categories. From experiments in section 5.4, $L_{\rm CECLA}$ has
better $F_{2}$ in microdata. In section 5.3, $L_{\rm CECLA{1,20}}$ also
performs better than $L_{\rm WCECL_{1,20}}$. Therefore, $L_{\rm CECLA}$ might
be more sensitive in the data distribution.

Let ${\bf p}=(p_{1,j},\cdots,p_{C,j})$ for $y_{j}=1$. Then

$L=-\frac{1}{N}\sum_{i=1}^{C}\left(A_{ii}N_{i}(1-p_{i,i})^{\gamma}\log
p_{i,i}+\sum_{j\neq i}N_{i}A_{ji}p_{j,i}^{\gamma}\log(1-p_{j,i})\right).$

If $A_{ij}$ is set by (19), then its gradient is the same as (17).

###  A.3. Punish bad prediction loss

$L_{\rm PBP}=-\frac{1}{N}\sum_{i=1}^{N}\left(\boldsymbol{\alpha}{\bf
y}_{i}({\bf 1}-{\bf p}_{i})^{\gamma}\log{\bf p}_{i}+\boldsymbol{\beta}(({\bf
1}-{\bf y})\odot\hat{{\bf y}}){\bf p}_{i}^{\gamma}\log({\bf 1}-{\bf
p}_{i})\right),$

where ${\bf y}_{i},{\bf p}_{i},\hat{{\bf
y}},\boldsymbol{\alpha},\boldsymbol{\beta}\in\mathbb{R}^{C}$. Let ${\bf
y}_{i}$ be the label of word $w_{i}$, ${\bf p}_{i}$ be the probability of
prediction, $\hat{{\bf y}}$ be the predict class of ${\bf p}$, $C$ be the
number of categories and $\boldsymbol{\alpha},\boldsymbol{\beta}$ be balanced
weights.

###  A.4. Comparison of $L_{\rm WCE}$, $L_{\rm CECLA}$ and $L_{\rm PBP}$

The differences between $L_{\rm WCE}$ and $L_{\rm CECLA}$:

Figure 12. CECLA vs PBP vs WCE

  * •

Since $L_{\rm CECLA}$ is based on contrastive learning, the loss has a
contribution from positive and negative samples. The gradient (16) of $L_{\rm
CECLA}$ causes the pull and push between them. The F scores have more up and
down. It has more chance to obtain the optimal point. From figure 12, it
achieves best macro $F_{2}$ score at 40 epochs. On the other hand, $L_{\rm
WCE}$ only has the contribution from positive samples. Therefore, its three F
scores curves are more steady increasing or decreasing.

  * •

Figure 5 shows that $L_{\rm CE}$ loss predicts category numbers more close to
ground truth. However, its precision is not good enough. On the micro data
(category 2 and 3), $L_{\rm CECLA}$, $L_{\rm PBP}$ and $L_{\rm WCE}$ obtain
higher $F_{1}$ than $L_{\rm CE}$ after 50 epochs on category 2. Moreover,
$L_{\rm CECLA}$ performs better than $L_{\rm WCE}$ after 50 epochs. $L_{\rm
CECLA}$ obtains better $F_{2}$ than $L_{\rm WCE}$ on category2 during $40\sim
50$ epochs.

  * •

In general, $L_{\rm PBP}$ converges faster than other loss functions. It has
higher $F_{1}$ than $L_{\rm WCE}$ on category 2. At 40 epochs, $L_{\rm PBP}$
performs better than $L_{\rm WCE}$.

Figure 13. CECLA vs PBP, WCE, CE on category2

  * •

On category 3, $L_{\rm CECLA}$ gets better $F_{1}$ score than $L_{\rm CE}$ at
40 epochs. $L_{\rm PBP}$ obtains best ${\rm F_{2}}$ among all loss functions
at 40 epochs.

Figure 14. CECLA vs PBP, WCE, CE on category3

## References

  * [1] R. Campos, V. Mangaravite, A. Pasquali, A. M. Jorge, C. Nunes, and A. Jatowt.  Yake! collection-independent automatic keyword extractor.  In G. Pasi, B. Piwowarski, L. Azzopardi, and A. Hanbury, editors, Advances in Information Retrieval - 40th European Conference on IR Research, ECIR 2018, Grenoble, France, March 26-29, 2018, Proceedings, volume 10772 of Lecture Notes in Computer Science, pages 806–810. Springer, 2018.
  * [2] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio.  On the properties of neural machine translation: Encoder-decoder approaches.  CoRR, abs/1409.1259, 2014.
  * [3] K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlós, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, D. Belanger, L. Colwell, and A. Weller.  Rethinking attention with performers.  CoRR, abs/2009.14794, 2020.
  * [4] J. Devlin, M. Chang, K. Lee, and K. Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  CoRR, abs/1810.04805, 2018.
  * [5] W. Fedus, B. Zoph, and N. Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  CoRR, abs/2101.03961, 2021.
  * [6] J. Guo, Y. Fan, Q. Ai, and W. B. Croft.  A deep relevance matching model for ad-hoc retrieval.  CoRR, abs/1711.08611, 2017.
  * [7] R. Hadsell, S. Chopra, and Y. LeCun.  Dimensionality reduction by learning an invariant mapping.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742, 2006.
  * [8] P. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. P. Heck.  Learning deep structured semantic models for web search using clickthrough data.  In Q. He, A. Iyengar, W. Nejdl, J. Pei, and R. Rastogi, editors, 22nd ACM International Conference on Information and Knowledge Management, CIKM’13, San Francisco, CA, USA, October 27 - November 1, 2013, pages 2333–2338. ACM, 2013.
  * [9] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [10] L. Liu, X. Liu, J. Gao, W. Chen, and J. Han.  Understanding the difficulty of training transformers.  CoRR, abs/2004.08249, 2020.
  * [11] R. Mihalcea and P. Tarau.  TextRank: Bringing order into texts.  In Proceedings of EMNLP-04and the 2004 Conference on Empirical Methods in Natural Language Processing, July 2004.
  * [12] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean.  Distributed representations of words and phrases and their compositionality.  CoRR, abs/1310.4546, 2013.
  * [13] R. Nogueira and K. Cho.  Passage re-ranking with BERT.  CoRR, abs/1901.04085, 2019.
  * [14] L. Pang, Y. Lan, J. Guo, J. Xu, J. Xu, and X. Cheng.  Deeprank: A new deep architecture for relevance ranking in information retrieval.  CoRR, abs/1710.05649, 2017.
  * [15] J. Pennington, R. Socher, and C. D. Manning.  Glove: Global vectors for word representation.  In EMNLP, volume 14, pages 1532–1543, 2014.
  * [16] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever.  Improving language understanding with unsupervised learning.  https://openai.com/blog/language-unsupervised/, 2018.
  * [17] S. Rose, D. Engel, N. Cramer, and W. Cowley.  Automatic keyword extraction from individual documents.  In M. W. Berry and J. Kogan, editors, Text Mining. Applications and Theory, pages 1–20. John Wiley and Sons, Ltd, 2010.
  * [18] F. Schroff, D. Kalenichenko, and J. Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  * [19] Y. Shen, X. He, J. Gao, L. Deng, and G. Mesnil.  Learning semantic representations using convolutional neural networks for web search.  In C. Chung, A. Z. Broder, K. Shim, and T. Suel, editors, 23rd International World Wide Web Conference, WWW ’14, Seoul, Republic of Korea, April 7-11, 2014, Companion Volume, pages 373–374. ACM, 2014.
  * [20] D. R. So, C. Liang, and Q. V. Le.  The evolved transformer.  CoRR, abs/1901.11117, 2019.
  * [21] K. Sohn.  Improved deep metric learning with multi-class n-pair loss objective.  In D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, editors, NIPS, pages 1849–1857, 2016.
  * [22] M. Tang, P. Gandhi, M. A. Kabir, C. Zou, J. Blakey, and X. Luo.  Progress notes classification and keyword extraction using attention-based deep learning models with BERT.  CoRR, abs/1910.05786, 2019.
  * [23] S. Tseng, P. G. Georgiou, and S. S. Narayanan.  Multimodal embeddings from language models.  CoRR, abs/1909.04302, 2019.
  * [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need.  CoRR, abs/1706.03762, 2017.
  * [25] X. Wan and J. Xiao.  Collabrank: Towards a collaborative approach to single-document keyphrase extraction.  In D. Scott and H. Uszkoreit, editors, COLING 2008, 22nd International Conference on Computational Linguistics, Proceedings of the Conference, 18-22 August 2008, Manchester, UK, pages 969–976, 2008.
  * [26] I. H. Witten, G. W. Paynter, E. Frank, C. Gutwin, and C. G. Nevill-Manning.  KEA: practical automatic keyphrase extraction.  CoRR, cs.DL/9902007, 1999.
  * [27] C. Xiong, Z. Dai, J. Callan, Z. Liu, and R. Power.  End-to-end neural ad-hoc ranking with kernel pooling.  CoRR, abs/1706.06613, 2017.
  * [28] Q. Zhang, Y. Wang, Y. Gong, and X. Huang.  Keyphrase extraction using deep recurrent neural networks on twitter.  In J. Su, X. Carreras, and K. Duh, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 836–845. The Association for Computational Linguistics, 2016.
  * [29] G. Zhao, X. Sun, J. Xu, Z. Zhang, and L. Luo.  MUSE: parallel multi-scale attention for sequence to sequence learning.  CoRR, abs/1911.09483, 2019.
  * [30] H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang.  Informer: Beyond efficient transformer for long sequence time-series forecasting.  CoRR, abs/2012.07436, 2020.