# BLISS: Robust Sequence-to-Sequence Learning via
Self-Supervised Input Representation

Zheng Zhang†, Liang Ding‡, Dazhao Cheng‡,
Xuebo Liu, Min Zhang, and Dacheng Tao,  Z. Zhang and D. Cheng are with the
Institute of Artificial Intelligence, School of Computer Science, Wuhan
University, Wuhan, China<EMAIL_ADDRESS>dcheng@whu.edu.cn). L.
Ding and D. Tao are with the JD Explore Academy at JD.com, Beijing, China
(e-mail<EMAIL_ADDRESS>dacheng.tao@jd.com). X. Liu and M. Zhang are with
the Institute of Computing and Intelligence, Harbin Institute of Technology,
Shenzhen, China (e-mail<EMAIL_ADDRESS>zhangminmt@hotmail.com). †Work
was done when Zheng Zhang interning at JD Explore Academy; ‡Corresponding
Authors: Liang Ding (e-mail<EMAIL_ADDRESS>and Dazhao Cheng (e-mail:
dcheng@whu.edu.cn).

###### Abstract

Data augmentations (DA) are the cores to achieving robust sequence-to-sequence
learning on various natural language processing (NLP) tasks. However, most of
the DA approaches force the decoder to make predictions conditioned on the
perturbed input representation, underutilizing supervised information provided
by perturbed input. In this work, we propose a framework-level robust
sequence-to-sequence learning approach, named BLISS, via self-supervised input
representation, which has the great potential to complement the data-level
augmentation approaches. The key idea is to supervise the sequence-to-sequence
framework with both the supervised (“input$\rightarrow$output”) and self-
supervised (“perturbed input$\rightarrow$input”) information. We conduct
comprehensive experiments to validate the effectiveness of BLISS on various
tasks, including machine translation, grammatical error correction and text
summarization. The results show that BLISS outperforms significantly the
vanilla Transformer and consistently works well across tasks than the other
five contrastive baselines. Extensive analyses reveal that BLISS learns robust
representations and rich linguistic knowledge, confirming our claim. Source
code will be released upon publication.

###### Index Terms:

sequence-to-sequence learning, self-supervised learning, machine translation,
text summarization, grammatical error correction

##  I Introduction

Sequence-to-sequence learning [1] has advanced the state-of-the-art in various
natural language processing (NLP) tasks, such as machine translation [2, 3,
4], grammatical error correction [5, 6] and text summarization [7, 8].
Sequence-to-sequence learning models are generally implemented with an
encoder-decoder framework, in which the encoder summarizes the input sentence,
and predictions of the decoder are correspondingly supervised by matching the
cross-entropy of ground truth. That is, the existing sequence-to-sequence
learning frameworks are supervised by the direct correlation between the input
and the output.

To achieve robust sequence-to-sequence learning, many data augmentation
methods [9, 10, 11, 12, 13, 14] are proposed to enrich the training datasets
by automatically or manually creating the perturbed input. For example, EDA
[15] introduces some simple data augmentation strategies, e.g. insert, swap
and deletion, working well for the low-resource settings. To avoid the
generated data deviating from the original context, language models are
employed to generate the substitutions for the subset of the input sentence
[9, 10, 11]. Besides, the adversarial techniques are also adopted to generate
the adversarial samples to enhance the model generalization [12, 13, 14].
Although those data-level approaches are straightforward and easy to use, all
the above methods force the decoder to make lexical choices conditioned on the
perturbed input representation, which underutilizes supervised information
provided by perturbed input.

In response to this problem, we propose a framework-level robust approach to
make the most of the perturbed input in sequence-to-sequence learning via
self-supervised input representation. The key idea is to supervise the
sequence-to-sequence framework with both the transformation from inputs to
outputs, and the correlation between the perturbed input and its original
input. In particular, we employ two extremely simple and effective data
augmentation techniques, i.e. shuffle and replacement, as the input perturbing
function. Then, we propose a smoothness controller to harness the perturbing
degree. Based on the perturbed input, we correspondingly design a self-
supervised mechanism upon the top of the encoder, where we choose the token
prediction and position prediction as two objectives to restore the perturbed
subset. By doing so, we can achieve robust sequence-to-sequence learning by
fully exploiting the supervised (“input$\rightarrow$output”) and self-
supervised (“perturbed input$\rightarrow$input”) information.

We validated our approach on several sequence-to-sequence NLP tasks in Section
IV-D, including machine translation [16, 4], grammatical error correction [7,
17] and text summarization [5, 6], across five datasets. The experimental
results show that our proposed BLISS significantly outperforms the vanilla
Transformer and consistently works well across tasks than other five
competitive baselines. Experiments on translation show that our proposed BLISS
yields consistent improvements, ranging from 0.6 up to 0.8 BLEU points. As for
correction and summarization tasks, we achieve +2.0 F0.5 and +0.5 Rouge-L
improvements against strong Transformer models, demonstrating the
effectiveness and universality of our approach. In addition, we conducted
comprehensive analyses in Section IV-E to understand when and why our BLISS
works. Furthermore, we showed that our framework-level self-supervised BLISS
is complementary to some existing augmentation approach, e.g. SwitchOut [18].
Also, our BLISS is robust to inference noises and hyper-parameters compared to
baselines. Importantly, through probing task [19], we found that our model
could preserve significantly rich linguistic knowledge against vanilla
Transformer. Our main contributions can be summarized as:

  * •

We introduce a robust sequence-to-sequence learning framework via self-
supervised input representation, which has the potential to complement
existing data augmentation approaches.

  * •

Our approach provides a unified framework to make the most of existing
supervised signals, i.e. correlation between input and output, and self-
supervised signals, i.e. self-supervisions between perturbed input and
original input.

  * •

We empirically validate the the effectiveness and universality on extensive
experiments across tasks and datasets.

##  II Related Work

Our work is inspired by two lines of research: i) self-supervised learning and
ii) data augmentation.

#### Self-Supervised Learning

Self-supervision signals have been widely investigated in language model
pretraining and unsupervised learning. BERT [20] propose the mask language
model, where they substitute a subset of tokens in the input sentence by a
special symbol [MASK], and then predicts the missing tokens by the residual
ones. MASS [21] presents a sequence-to-sequence pre-training framework, which
takes non-mask tokens as the encoder input and leverages masked tokens as the
decoder input as well as the to-be-predicted target. STRUCTBERT [22] extends
BERT by leveraging the structural information: word-level ordering and
sentence level ordering. SpanBERT [23] masks random contiguous spans rather
than individual tokens and additionally introduces span-boundary objective.
Different from these works that apply self-supervisions to the cost pre-train
stage and fine-tune them on the down-stream tasks, we design the self-
supervision objectives for input sentence to complement the existing MLE
generation objectives to achieve further improvement.

Similar to our work, there exists several works that combine self-supervisions
with from-scratch sequence-to-sequence model training. JM-S2S [24] introduce
mask task to non-autoregressive translation model to fully exploit the
undertrained encoder. Monolingual data is used by self-supervisions in
multilingual translation [25]. Self-supervised and supervised learning are
combined to optimize the machine translation models especially for the rich-
resource settings [26]. Different from these works, we propose a plug-and-play
self-supervised input representation approach for general sequence-to-sequence
tasks, which could be used to complement any data augmentation approaches and
consistently enhance the model performance.

Figure 1: Illustration of the Transformer based encoder-decoder model. Figure
2: Illustration of the proposed smooth augmented data generator in Section
III-B, which consists of two components, i.e. perturbing functions and
smoothness controller, represented by dashed rounded rectangles, respectively.
The blue block represent tokens been shuffled while the green block represent
tokens been replaced.

#### Data Augmentation

There exists some easy augmentation method, including randomly shuffling the
words within a fixed window size to construct the perturbed sentence [27, 28],
dropping some words randomly in the source sentence for learning an auto-
encoder to help train the unsupervised NMT model [29], replacing the word with
a placeholder token or a word sampled from the frequency distribution of
vocabulary [30], mixing the token representations to encourage the diversified
generation [31] and other common NLP data scaling approaches [32, 33]. These
methods are usually useful in small datasets. However, some studies [15]
experiments with easy data augmentation methods like randomly insert, swap and
delete and they found that these simple methods take little effect with full
datasets. Besides, SwitchOut [18] replaces words in the source/target
sentences with other words form the source/target vocabulary. SeqMix [34] mixs
up pairs of source sentences or decoder input sentences. Our work
significantly differs from these work. We do not predict the target lexicons
conditioned on these perturbed input directly. Rather, we propose to recover
the noised input with encoder, thus the conditional representation for decoder
preserve much linguistic knowledge (See Section IV-E).

##  III Self-Supervised Input Representation

In this section, we first review the sequence-to-sequence learning in Section
III-A. Then we introduce the smoothed data augmentation technique, namely
Smooth Augmented Data Generator in Section III-B. Finally Section III-C
elaborates our proposed Self-Supervised Input Representation approach.

###  III-A Preliminaries

#### Sequence-to-Sequence Learning

Given the target sequence $\mathbb{y}=\\{y_{1},y_{2},...,y_{t}\\}$ conditioned
on a source sequence $\mathbb{x}=\\{x_{1},x_{2}\,...,x_{s}\\}$, the objective
of Seq2Seq model can be formulated as $\mathbb{\hat{y}}=argmax\
logP(\mathbb{y}|\mathbb{x})$. Up to now, Transformer [4] advanced the state of
art results compared to other architectures. Thus, we employ Transformer as
the strong baseline and the test bed. Transformer consists of an encoder
equipped with several identical layers mapping the source sequence
$\mathbb{x}$ into intermediate representation $\mathbb{h}$ and a decoder
equipped with several identical layers taking $\mathbb{h}$ and shifted target
sequence as input and generates $\mathbb{y}$ target sequences
autoregressively:

$\displaystyle\bf h$ $\displaystyle=enc(\bf x+pos\\_emb(\bf x))$
$\displaystyle\bf{\hat{y}}_{\leq t}$
$\displaystyle=dec(\bf{\hat{y}}_{<t}+pos\\_emb(\mathbb{\hat{y}}_{<t}),\bf h)$

Encoder and decoder consists of position-wise feed-forward network, multi-head
dot-product attention network and so on, details can be seen from Figure 1.
Noticeable, tokens and position embeddings calculated by $pos\\_emb(\cdot)$
are taken as one of the inputs of encoder or decoder, which provides supports
for the existence and necessity of position information. So, we design
position auxiliary task to restore position information in encoder
representation. Decoder generate target sentence autoregressively until
meeting special symbol $<eos>$. Finally, the output of the decoder
$\hat{\mathbb{y}}$ is projected into the probability
$P(\mathbb{y}|\mathbb{x})$, and the optimization objective can be formulated
as:

$\text{argmax}_{\theta}\text{log}P(\mathbb{y}|\mathbb{x};\theta_{enc},\theta_{dec})$
(1)

where $\theta_{enc}$ and $\theta_{dec}$ denote the parameters of the encoder
and decoder respectively.

###  III-B Smooth Augmented Data Generator

As shown in Figure 2, our smooth augmented data generator contains two parts,
perturbing functions and smoothness controller.

#### Perturbing Functions

As shown in Figure 2, we feed the source sentences into two perturbing
functions, shuffle function and replace function sequentially. For each
function, we randomly select $\gamma$ percentage of source sentences for
generating augmented data. Specifically, we randomly shuffle tokens within a
certain sized window by the shuffle function, and randomly replace several
tokens in the source sentences with other words form the source vocabulary by
the replace function.

Figure 3: The illustration of our proposed self-supervised input
representation (Section III-C) in sequence-to-sequence learning framework. We
add two classifier to predict the token and position of perturbed tokens
synthesized by the smooth augmented data generator in Section III-B. The
meaning of blue rectangle and green rectangle is the same as in Figure 2. The
red rectangles represent disturbed tokens’ intermediate representation
produced by the top layer of encoder.

#### Smoothness Controller

We set $\alpha_{shu},\alpha_{rep}$ to control the maximum number of shuffled
and replaced tokens respectively. Without smoothness controller, models can
only see augmented data with $\alpha_{shu}L$ shuffle tokens of $\alpha_{rep}L$
replaced tokens, where $L$ is the length of sequence. To balance the diversity
and similarity of augmented data, we design a smoothness controller to get a
smooth distribution of augmented data with different perturbing tokens.
Specifically, we hypothesis sampling the number of perturbed tokens $l$ from
geometric distribution $l|_{l<=\alpha L}\sim Geometric(p)$, where p is between
0 and 1111after preliminary studies, we set p=0.2 for all tasks. Then, since
$l$ is limited by a upper bound, we normalize the distribution of $l$. Finally
we sample $l$ according to the probability distribution expressed in Equation
2. For shuffle function and replacement function, we repeat the above
procedures individually and obtain $l_{shu}$ and $l_{rep}$ for perturbing
operations.

$P(l)=\frac{p(1-p)^{l-1}}{\sum_{i=1}^{\alpha L}p(1-p)^{i-1}}\delta_{1-\alpha
L}(l)$ (2)

$\delta_{1-\alpha L}(l)$ equal 1 when $l$ is equal or greater than 0 and equal
or smaller than $\alpha L$ otherwise 0.

###  III-C Self-Supervised Input Representation

Inspired by mask language models [20], which mask tokens in source sentences
and predicts masked tokens on the output, we take similar procedure but two
differences distinguish us between them. First, our method is applied in the
down-stream tasks with labeled datasets while mask language models take
effects in pre-training tasks with unlabeled datasets, so our method works in
parallel with mask language models and is complementary with them. Second,
prior studies only take token as ground truth label to supervise output. We
also take positions as additional labels.

Specifically, we design two softmax classifiers to predict token and position
by $\tilde{\bf h}$ respectively. Token classifier are responsible to predict
the origin tokens of $\tilde{\bf x}$ while the position classifier predicts
the position of perturbed tokens. And corresponding self-supervision loss
functions $L_{token},L_{pos}$ is expressed as Equation 3 and Equation 4, where
$x_{i},p_{i}$ denote the origin tokens and absolute position,
$W_{token}\in\mathbf{R}^{e\times v}$ and $W_{pos}\in\mathbf{R}^{e\times
p_{m}}$ represent the parameters of softmax classifier, and $e,v,p_{m}$ denote
embedding dimension, vocabulary size and maximum position index. Following the
preliminary trials, we set $p_{m}=400$.

$\displaystyle\mathcal{L}_{token}=\sum_{i}logP(x_{i}|\tilde{h}_{i},W_{token},\theta_{enc})$
(3)
$\displaystyle\mathcal{L}_{pos}=\sum_{i}logP(p_{i}|\tilde{h}_{i},W_{pos},\theta_{enc})$
(4)
$\displaystyle\mathcal{L}_{nll}(\mathbb{\tilde{x}},\mathbb{y})=\text{log}P(\mathbb{y}|\mathbb{\tilde{x}};\theta_{enc},\theta_{dec})$
(5)

By integrating the above two loss functions with the traditional negative log-
likelihood loss function as Equation 5, the complete objective function of our
model is expressed as Equation 6:

$\underset{\theta}{\text{argmax}}\
\mathcal{L}(x,y)=\mathcal{L}_{nll}+\lambda_{token}\mathcal{L}_{token}+\lambda_{pos}\mathcal{L}_{pos}$
(6)

where $\theta=\\{\theta_{enc},\theta_{dec},W_{token},W_{pos}\\}$,
$\lambda_{token}$ and $\lambda_{pos}$ are hyper-parameters that balance the
weights of different self-supervision objectives.

In conclusion, we add smooth augmented data generator for source sentences,
and restore it’s original token and position information on the encoder
output. The basic intuition behind is that although the augmented sequence
information is distorted locally, but the contextual information is still
preserved, a robust encoder should own the ability to restore correct
information from the distorted sentences. Besides the tokens of sequence, the
positions of tokens in the sequence play an importance role of the sequence
information. So, we design the encoder to predict the positions of swapped
tokens to help encoder understand the position information of sequence.

##  IV Experiments

TABLE I: Statistics of the datasets and hyperparameters for the experiments.
“Batch” denotes the number of source tokens and target tokens used in each
training step. “DP” denotes the dropout value [35]. “LP” denotes the length
penalty [36]. For GEC and text summarization tasks, we chose the checkpoint
with best validation score for testing, for translation tasks, we choose the
average of last five checkpoints for testing.

| Vocab | Sents | Training | Testing
---|---|---|---|---
Src/Tgt | Train | Dev | Test | Batch | Step | DP | Beam | LP
WMT14 En-De | 32768 | 4.5M | 3K | 3K | 64K | 300K | 0.2 | 5 | 0.6
WMT16 En-Ro | 34976 | 0.6M | 2K | 2K | 160K | 15K | 0.3 | 5 | 1.0
IWSLT14 De-En | 10148 | 160215 | 7282 | 6750 | 32K | 20K | 0.3 | 5 | 1.0
CNN/DM | 50264 | 0.3M | 13K | 11K | 64K | 70K | 0.1 | 4 | 2.0
CONLL | 33352 | 1.3M | 5K | 1K | 64K | 85K | 0.2 | 6 | 0.6

TABLE II: Hyper parameters of our methods in different tasks. | $\gamma$ | $\alpha_{shu}$ | $\alpha_{rep}$
---|---|---|---
WMT14 En-De | 0.3 | 0.1 | 0.1
WMT16 En-Ro | 0.4 | 0.1 | 0.1
IWSLT14 De-En | 0.3 | 0.12 | 0.15
CNN/DM | 0.4 | 0.08 | 0.15
CONLL | 0.3 | 0.12 | 0.1

###  IV-A Tasks and Datasets

To validate the effectiveness of BLISS, we conducted experiments on three
representative tasks, which vary in the distance between input and output
domains and the scale of training data:

TABLE III: Experimental results of the proposed BLISS method on the Seq2Seq
tasks. Results marked with † are statistically significant compared to vanilla
transformer, with ‡ are statistically significant compared to best baseline.

| Translation | Correction | Summarization
---|---|---|---
De-En | En-Ro | En-De | Prec. | Recall | F0.5 | RG-1 | RG-2 | RG-L
Vanilla | 35.1 | 34.7 | 26.7 | 58.7 | 33.8 | 51.2 | 40.1 | 17.6 | 36.8
Dropout | 35.4 | 35.2 | 26.5 | 60.8 | 34.2 | 52.6 | 40.4 | 17.7 | 37.1
Blank | 35.6 | 35.4 | 27.0 | 59.3 | 32.7 | 51.0 | 40.0 | 17.5 | 36.8
Shuffle | 34.9 | 34.4 | 26.5 | 52.5 | 33.9 | 47.3 | 40.1 | 17.3 | 36.8
SeqMix | 35.4 | 35.5 | 26.8 | 58.3 | 33.5 | 50.8 | 40.2 | 17.6 | 36.9
Switchout | 35.6 | 35.5 | 26.9 | 60.3 | 34.1 | 52.3 | 40.6 | 17.9 | 37.1
BLISS (ours) | 35.7† | 35.5† | 27.3†‡ | 60.2† | 36.3†‡ | 53.2†‡ | 40.6† | 17.9† | 37.3†

Machine Translation takes a sentence in one language as input, and outputs a
semantically-equivalent sentence in another language. We evaluate our method
on three widely-used benchmarks: IWSLT14 German$\rightarrow$English (IWSLT14
De-En222https://wit3.fbk.eu/, [37]), WMT16 English$\rightarrow$Romanian (WMT16
En-Ro333https://www.statmt.org/wmt16/translation-task, [38]), and WMT14
English-German (WMT14 En-De444new://www.statmt.org/wmt14/translation-task,
[4]). We strictly follow the dataset configurations of previous works for a
fair comparison. For the IWSLT14 De-En task, we train the model on its
training set with 160K training samples and evaluate on its test set. For the
WMT14 En-De task, we train the model on the training set with 4.5M training
samples, where newstest2013 and newstest2014 are used as the validation and
test set respectively. As for the WMT16 En-Ro task which has 610K training
pairs, we utilize newsdev2016 and newstest2016 as the validation and test set.
For each dataset, we tokenize the sentences by Moses [39] and segment each
word into subwords using Byte-Pair Encoding (BPE, [40]), resulting in a 32K
vocabulary shared by source and target languages. All the translation tasks
are evaluated with sacreBLEU [41] score.

Grammatical Error Correction takes a sentence with grammatical errors as input
and generates a corrected sentence. We evaluate our method on CONLL14
dataset555https://www.comp.nus.edu.sg/~nlp/conll14st.html, which has 1.4M
training samples. We closely follow [42] to preprocess the data. The MaxMatch
($\text{M}^{2}$) scores [43] were used for evaluation with Precision, Recall,
and $F_{0.5}$ values.

Text Summarization takes a long-text document as input, and generates a short
and adequate summary in the same language. We evaluate our method on the the
most representative summarization benchmark CNN/Daily Mail
corpus666https://huggingface.co/datasets/cnn_dailymail, which contains 300K
training samples. We follow [44] to preprocess the data. During testing, the
minimum length was set to 55 and the maximum length was set to 140, which were
tuned on the development data. We also follow [45] to disallow repeating the
same trigram. We evaluate the summarization performance with the standard
ROUGE metric [46], i.e. Rouge-1, Rouge-2, and Rouge-L.

The machine translation task has distant input/output domains (i.e. in
different languages), while the other tasks has similar input/output domains
(i.e. in the same language). Details of the datasets777We use the same
datasets with [47]. are listed in Table I.

###  IV-B Implementation

Our model is based on the Transformer [4] sequence-to-sequence architecture
due to its state-of-the-art performance and all the models are implemented by
the open-source toolkit fairseq888https://github.com/pytorch/fairseq [44]. For
better reproduction, we employ the base Transformer ($\text{d}_{model}$ =
$\text{d}_{hidden}$ = 512, $\text{n}_{layer}$ = 6, $\text{n}_{head}$ = 8) for
all tasks in this paper. All models were trained on NVIDIA DGX A100 cluster.
Table I gives more details of the benchmarks. It is noted that other
unmentioned hyperparameters keep the same with the original paper of
Transformer [4]. The hyper-parameters of training of different tasks and
datasets are listed in Table I. We set $\lambda_{token}=\lambda_{pos}$=0.005
and $p=0.2$ for all tasks, other hyper parameters varying in tasks as shown in
Table II.

###  IV-C Baselines

To validate the effectiveness of our methods, we compare our approach with
following baselines:

  * •

Vanilla [4]: The original sequence-to-sequence training strategy without any
data augmentation strategies.

  * •

Dropout [29, 28]: Randomly dropping tokens with their best drop ratio 0.1.

  * •

Blank [30]: Randomly replacing word tokens with a placeholder, we leave their
best setting ratio=0.1 as default.

  * •

Shuffle [27, 28]: Randomly swapping words in nearby positions within a window
size K=3.

  * •

SeqMix [34]: Mixing sentence pairs on both the source and target side. We
reimplement according to their public
code999https://github.com/dguo98/SeqMix/tree/main.

  * •

SwithOut [18]: Replacing tokens with other tokens in vocabulary on the source
side and target side. We re-implement according to the Appendix A.5 of their
paper.

TABLE IV: Effects of removing each component. The metrics and datasets are
same as that of Table III. Bold represents the settings with the most
performance degradation for each corresponding task.

| Translation | Correction | Summarization
---|---|---|---
De-En | En-Ro | En-De | Prec. | Recall | F0.5 | RG-1 | RG-2 | RG-L
Vanilla | 35.1 | 34.7 | 26.7 | 58.7 | 33.8 | 51.2 | 40.1 | 17.6 | 36.8
BLISS | 35.7 | 35.5 | 27.3 | 60.2 | 36.3 | 53.2 | 40.6 | 17.9 | 37.3
-aug-smooth | 35.5 | 35.1 | 26.8 | 60.8 | 32.3 | 52.3 | 40.4 | 17.6 | 36.8
-smooth | 35.6 | 35.2 | 26.9 | 60.2 | 34.9 | 52.6 | 40.4 | 17.7 | 37.0
-token | 35.6 | 35.1 | 26.8 | 60.5 | 32.0 | 51.3 | 40.4 | 17.7 | 37.0
-pos | 35.6 | 35.3 | 26.9 | 60.3 | 35.0 | 52.7 | 40.5 | 17.8 | 37.1

###  IV-D Main Results

Table III lists the performances of our models as well as strong baseline
models on different tasks. Clearly, the proposed self-supervised input
representation approach (“BLISS”) significantly outperforms the vanilla
Transformer in all cases, while there are still considerable differences among
model variations. Specifically, on translation task, our BLISS equipped models
achieve the best among all contrasted approaches, and encouragingly outperform
the vanilla transformer by averaged +0.7 BLEU points. As for the grammatical
error correction task, we achieve the +2.0 F0.5 scores improvement against the
vanilla model, and notably, our robust self-supervised input representation
approach recalls significantly more potential grammatical errors, i.e. +2.5
percent. On the contrary, the existing data augmentation approaches, e.g.
Shuffle, Blank and SeqMix, slightly undermine the GEC performance. We
conjecture that such performance degradation for previous approaches is due to
the lack of generalization across tasks, i.e. they are proposed for MT. As for
summarization task, the results also show a promising trend against all
baseline methods. All those findings demonstrate that our proposed robust
self-supervised input representation approach (“BLISS”) is effective and
universal across language pairs and tasks.

###  IV-E Analysis

In this section, we provide some insights into when and why our BLISS works.

#### Effects of Each Component

There are four carefully designed components: i) perturbing functions named as
“aug” performs shuffling and replacement operations to generate augmented data
sequentially. ii) Smoothness controller named as “smooth” generates augmented
data of various degrees. iii) Token self-supervision loss named as “token”
supervises the lexical information of augmented input, which helps the encoder
capture robust token representation. iv) Position self-supervision loss named
as “pos” supervises the position information of augmented input. To verify the
reasonableness of those components, we remove different components in Table
IV, e.g. “-aug-smooth”, “-smooth”, “-token” and “-pos”, as the ablation
settings. Takeaway: Our proposed BLISS performs worse when either component is
removed, demonstrating the effectiveness of four proposed components.

#### Complementary to Related Works

TABLE V: Complementary to other work, i.e. SwitchOut [18]. BLISS with
SwitchOut-style augmentation function is denoted as “BLISS w/ S.”.
Translation, Correction and Summarization are evaluated with BLEU, F0.5 and
RG-L, respectively.

| WMT14 En-De | CoNLL14 | CNN/DM
---|---|---|---
SwitchOut | 26.9 | 52.3 | 37.1
BLISS w/ S. | 27.1 | 52.9 | 37.3

Our proposed BLISS enables self-supervisions from the structure-level, thus
BLISS has the great potential to complement existing strong data-level
methods. Here we choose SwitchOut [18] due to its competitive performance in
main experiments. We replace the vanilla simple augmentation function in
BLISS, i.e. shuffle and replacement, with SwitchOut and the results is
reported in Table V. Takeaway: Our proposed structure-level self-supervised
approach BLISS achieves further improvement across different sequence-to-
sequence tasks with advanced data augmentation functions, e.g. SwitchOut,
showing its appealing expandability.

#### BLISS is Robust to the Inference Noises

Figure 4: The performance drop when inference on noised testing data, where we
test on WMT14 En-De and report the scaled BLEU scores. The noise types for the
left and right figures are shuffling and replacing, respectively.

Our self-supervised input representation is expected to tolerate the inputting
noises to some extent. To validate our hypothesis, we follow [48, 49] to
inject two types of artificial noise, e.g. shuffle and replacement, into the
test samples with different ratios ranging from {2%, 4%, 8% and 16%}. For
shuffle noise, we select a span whose length is $\alpha l$ ($l$ is the length
of source sentence) and shuffle the order of words within the span. As for the
replacement noise, we follow our replacement function, where we randomly
replace $\alpha l$ tokens with other tokens in the vocabulary. Figure 4 shows
different models’ performance on noisy data about WMT14 En-De task. Takeaway:
Compared with vanilla Transformer and existing contrastive variants, as noise
increases, our model “BLISS” is significantly robust to both noise,
demonstrating the robustness of our approach.

#### BLISS is Robust to the Hyper-Parameters

Figure 5: $F_{0.5}$ scores on CONLL14 dataset with different hyper parameters.
Left: $\alpha_{shu},\alpha_{rep}$. Right: $\gamma$.

Data augmentation approaches are always sensitive to hyper-parameters. To
dispel the doubt, we investigate whether our approach is robust to different
hyper-parameters. We empirically study the effect of hyper parameters
$\alpha_{shu},\alpha_{rep},\gamma$ on GEC task. We can observe from Figure 5
that although the performance varies with hyper-parameters, the extreme values
of the results are not significant, still outperforming the baseline approach.
To further validate that our model is not hyper-parameter sensitive, we do
experiments with different values of hyper-parameters sampling from half of
the optimal value to 1.5 times the optimal value. For example, the optimal
value of $\gamma$ is 0.04, so we test on values 0.02, 0.03, 0.04, 0.05, 0.06.
The violin plot graph is shown in Figure 6, where the minimum values of each
hyper-parameters are higher than baseline, proving the insensitivity of our
hyper-parameters.

Takeaway: Our proposed BLISS is not sensitive to hyper-parameters, all hyper-
parameters’ variants outperform the baseline.

TABLE VI: Performance on 10 probing tasks to evaluate the linguistic
properties. Note that we train the model on WMT14 En-De.

Task | vanilla | BLISS
---|---|---
Surface | SeLen | 93.1 | 94.0
WC | 42.7 | 41.9
Syntactic | TrDep | 41.7 | 44.0
ToCo | 73.5 | 75.3
BShif | 69.3 | 71.8
Semantic | Tense | 77.0 | 77.5
SubN | 77.3 | 78.4
ObjN | 75.0 | 75.2
SoMo | 50.4 | 50.6
CoIn | 62.2 | 63.3

Figure 6: The violin plot of three hyper parameters,
$\gamma,\alpha_{shu},\alpha_{hep}$. The ‘violin’ of each hyper show the
distribution of GEC $F_{0.5}$ scores with different value of hyper parameters.

#### BLISS Captures Better Linguistic Representation

Intuitively, our proposed robust self-supervised input representation approach
bringing the capability to correct artificial errors by restoring the token
and position information, may help the encoder capture more linguistic
knowledge. To verify this hypothesis, we quantitatively investigate it with 10
probing
tasks101010https://github.com/facebookresearch/SentEval/tree/master/data/probing
[19] to study what linguistic properties are captured by the encoders.

Each probing task is a classification problem that focuses on simple
linguistic properties of sentences. The 10 probing tasks are categorized into
three groups: (1) “Surface” focuses on the simple surface properties learned
from the sentence embedding. (2) “Syntactic” quantifies the syntactic
reservation ability; and (3) “Semantic” assesses the deeper semantic
representation ability. More detailed information about the 10 tasks can refer
to the original paper [19]. For each task, we trained the classifier on the
train set, and validated the classifier on the validation set.

Following [50] and [51], we first extract the sentence representations of
input sentences by taking average of encoder output. The classifier we used as
the sentence as a Multi-Layer Perceptron(MLP) with a hidden dimention of 256.
We optimized the model using the Adam optimizer with a leaning rate of 0.001
in 70 epochs for ‘WC’ and ‘SoMo’ task and 10 epochs for other tasks.

To evaluate the representation ability of our BLISS, we compare the pretrained
vanilla Transformer [4] and BLISS equipped machine translation model encoders,
followed by a MLP classifier. Sepcifically, the mean of the top encoding
layer, as sentence representation, will be passed to the classifier. Table VI
lists the results. Takeaway: The proposed BLISS could preserve significant
better surface, syntactic and semantic knowledge (Vanilla vs. BLISS = 65.1 vs.
66.2), confirming our hypothesis.

Figure 7: The influence on robustness of models of self-supervision loss
functions. We test on IWSLT14 De-En and report the scaled BLUE scores. The x
axis $\gamma$ represents the degree of noised data. The noise types for the
left and right figures are shuffling and replacing, respectively. Green and
orange line represent BLISS models removing position loss and token loss
individually. Figure 8: The influence on robustness of models of self-
supervision loss functions. We test on WMT16 En-Ro and report the scaled BLUE
scores. The meaning of axis is the same as Figure 7.

#### Auxiliary Self-Supervision Loss Improves Model Robustness

We introduced two auxiliary self-supervision loss functions (token denoising
loss and position denoising loss) to help the encoders learn more robust
representation of the source sequences. Given the noised input, a vanilla
transformer model learn a robust representation from noised data by the joint
effort of encoder and decoder, while our method encourages the encoder to take
more responsibility to not only understand the noised input but also
distinguish the noises (with token and position denoising losses). To
illustrate the effects of the auxiliary denoising losses, we conduct ablation
studies to observe the performance degradation without token and position
self-supervised losses on IWSLT14 De-En (in Figure 7) and WMT16 En-Ro (in
Figure 8). As expected, removing the self-supervision loss function will cause
significant performance degradation when performing the noise attacks.
However, the improvement of self-supervision loss functions is relatively
small on IWSLT14 De-En task and when the noise is replace. The potential
reason may be that the IWSLT14 De-En task contains fewer sentences and the
replace denoising task is relatively easy for the model to handle. So the
encoder is not necessary to be enhanced by self supervision method.

##  V Conclusion

In this paper, we investigate how to achieve robust sequence-to-sequence
learning with self-supervised input representation. To achieve it, we propose
to make the most of supervised signals and self-supervised signals with our
proposed BLISS, which consists of a smooth augmented data generator and
corresponding self-supervised objectives upon the top of the encoder.
Experiments show that BLISS consistently outperforms the vanilla Transformer
and other five data augmentation approaches in several datasets. Additional
analyses show that BLISS indeed learns robust input representation and better
linguistic information, confirming our hypothesis.

Future directions include validating our findings on more sequence-to-sequence
tasks (e.g. dialogue and speech recognition) and model architectures (e.g.
DynamicConv [36]). Also, its worthy to explore our method to large scale
sequence-to-sequence language model pretraining (e.g. BART [52]).

## References

  * [1] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in _NIPS_ , 2014.
  * [2] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in _ICLR_ , 2015.
  * [3] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey _et al._ , “Google’s neural machine translation system: Bridging the gap between human and machine translation,” _In arXiv_ , 2016.
  * [4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _NIPS_ , 2017, pp. 5998–6008. [Online]. Available: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
  * [5] S. Kiyono, J. Suzuki, M. Mita, T. Mizumoto, and K. Inui, “An empirical study of incorporating pseudo data into grammatical error correction,” in _EMNLP-IJCNLP_ , 2019.
  * [6] M. Kaneko, M. Mita, S. Kiyono, J. Suzuki, and K. Inui, “Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction,” in _ACL_ , D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, Eds., 2020.
  * [7] L. Wang, W. Zhao, R. Jia, S. Li, and J. Liu, “Denoising based sequence-to-sequence pre-training for text generation,” in _EMNLP_ , K. Inui, J. Jiang, V. Ng, and X. Wan, Eds., 2019.
  * [8] J. Zhang, Y. Zhao, M. Saleh, and P. J. Liu, “Pegasus: Pre-training with extracted gap-sentences for abstractive summarization,” in _ICML_ , 2020\.
  * [9] S. Kobayashi, “Contextual augmentation: Data augmentation by words with paradigmatic relations,” in _NAACL_ , 2018.
  * [10] X. Wu, S. Lv, L. Zang, J. Han, and S. Hu, “Conditional BERT contextual augmentation,” in _ICCS_ , 2019.
  * [11] F. Gao, J. Zhu, L. Wu, Y. Xia, T. Qin, X. Cheng, W. Zhou, and T. Liu, “Soft contextual data augmentation for neural machine translation,” in _ACL_ , A. Korhonen, D. R. Traum, and L. Màrquez, Eds., 2019.
  * [12] Y. Cheng, L. Jiang, W. Macherey, and J. Eisenstein, “Advaug: Robust adversarial augmentation for neural machine translation,” in _ACL_ , D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, Eds., 2020.
  * [13] G. Chen, K. Fan, K. Zhang, B. Chen, and Z. Huang, “Manifold adversarial augmentation for neural machine translation,” in _Findings of ACL_ , 2021\.
  * [14] J. X. Morris, E. Lifland, J. Y. Yoo, J. Grigsby, D. Jin, and Y. Qi, “Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP,” in _EMNLP_ , 2020.
  * [15] J. W. Wei and K. Zou, “EDA: easy data augmentation techniques for boosting performance on text classification tasks,” in _EMNLP-IJCNLP_ , K. Inui, J. Jiang, V. Ng, and X. Wan, Eds., 2019.
  * [16] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in _ICLR_ , Y. Bengio and Y. LeCun, Eds., 2015.
  * [17] J. Zhang, Y. Zhao, M. Saleh, and P. J. Liu, “PEGASUS: pre-training with extracted gap-sentences for abstractive summarization,” in _ICML_ , 2020\.
  * [18] X. Wang, H. Pham, Z. Dai, and G. Neubig, “Switchout: an efficient data augmentation algorithm for neural machine translation,” in _EMNLP_ , 2018\.
  * [19] A. Conneau and D. Kiela, “Senteval: An evaluation toolkit for universal sentence representations,” in _Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018_ , N. Calzolari, K. Choukri, C. Cieri, T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, and T. Tokunaga, Eds. European Language Resources Association (ELRA), 2018.
  * [20] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in _NAACL-HLT_ , 2019\.
  * [21] K. Song, X. Tan, T. Qin, J. Lu, and T. Liu, “MASS: masked sequence to sequence pre-training for language generation,” in _ICML_ , K. Chaudhuri and R. Salakhutdinov, Eds., 2019.
  * [22] W. Wang, B. Bi, M. Yan, C. Wu, J. Xia, Z. Bao, L. Peng, and L. Si, “Structbert: Incorporating language structures into pre-training for deep language understanding,” in _ICLR_ , 2020.
  * [23] M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, and O. Levy, “Spanbert: Improving pre-training by representing and predicting spans,” _Trans. Assoc. Comput. Linguistics_ , vol. 8, pp. 64–77, 2020.
  * [24] J. Guo, L. Xu, and E. Chen, “Jointly masked sequence-to-sequence model for non-autoregressive neural machine translation,” in _ACL_ , D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, Eds., 2020.
  * [25] A. Siddhant, A. Bapna, Y. Cao, O. Firat, M. X. Chen, S. R. Kudugunta, N. Arivazhagan, and Y. Wu, “Leveraging monolingual data with self-supervision for multilingual neural machine translation,” in _ACL_ , D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, Eds., 2020\.
  * [26] Y. Cheng, W. Wang, L. Jiang, and W. Macherey, “Self-supervised and supervised joint training for resource-rich machine translation,” in _ICML_ , M. Meila and T. Zhang, Eds., 2021.
  * [27] M. Artetxe, G. Labaka, E. Agirre, and K. Cho, “Unsupervised neural machine translation,” in _ICLR_ , 2018.
  * [28] G. Lample, A. Conneau, L. Denoyer, and M. Ranzato, “Unsupervised machine translation using monolingual corpora only,” in _ICLR_ , 2018.
  * [29] M. Iyyer, V. Manjunatha, J. L. Boyd-Graber, and H. D. III, “Deep unordered composition rivals syntactic methods for text classification,” in _ACL_ , 2015.
  * [30] Z. Xie, S. I. Wang, J. Li, D. Lévy, A. Nie, D. Jurafsky, and A. Y. Ng, “Data noising as smoothing in neural network language models,” in _ICLR_ , 2017.
  * [31] Y. Cao, L. Ding, Z. Tian, and M. Fang, “Towards efficiently diversifying dialogue generation via embedding augmentation,” _ICASSP_ , 2021.
  * [32] L. Ding, D. Wu, and D. Tao, “Improving neural machine translation by bidirectional training,” in _EMNLP_ , 2021.
  * [33] B. Wang, L. Ding, Q. Zhong, X. Li, and D. Tao, “A contrastive cross-channel data augmentation framework for aspect-based sentiment analysis,” _arXiv preprint_ , 2022.
  * [34] D. Guo, Y. Kim, and A. M. Rush, “Sequence-level mixed sample data augmentation,” in _EMNLP_ , B. Webber, T. Cohn, Y. He, and Y. Liu, Eds., 2020\.
  * [35] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _J. Mach. Learn. Res._ , vol. 15, pp. 1929–1958, 2014.
  * [36] F. Wu, A. Fan, A. Baevski, Y. Dauphin, and M. Auli, “Pay less attention with lightweight and dynamic convolutions,” in _ICLR_ , 2018.
  * [37] X. Nguyen, S. R. Joty, W. Kui, and A. T. Aw, “Data diversification: An elegant strategy for neural machine translation,” in _NeurIPS_ , 2020.
  * [38] J. Gu, J. Bradbury, C. Xiong, V. O. Li, and R. Socher, “Non-autoregressive neural machine translation,” in _ICLR_ , 2018.
  * [39] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst, “Moses: Open source toolkit for statistical machine translation,” in _ACL_ , 2007.
  * [40] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in _ACL_ , 2016.
  * [41] M. Post, “A call for clarity in reporting BLEU scores,” in _Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018, Belgium, Brussels, October 31 - November 1, 2018_. Association for Computational Linguistics, 2018, pp. 186–191. [Online]. Available: https://doi.org/10.18653/v1/w18-6319
  * [42] S. Chollampatt and H. T. Ng, “A multilayer convolutional encoder-decoder neural network for grammatical error correction,” in _AAAI_ , 2018.
  * [43] D. Dahlmeier and H. T. Ng, “Better evaluation for grammatical error correction,” in _NAACL_ , 2012.
  * [44] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, “fairseq: A fast, extensible toolkit for sequence modeling,” in _NAACL-HLT_ , W. Ammar, A. Louis, and N. Mostafazadeh, Eds., 2019.
  * [45] R. Paulus, C. Xiong, and R. Socher, “A deep reinforced model for abstractive summarization,” in _ICLR_ , 2018.
  * [46] C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in _Text summarization branches out_ , 2004, pp. 74–81.
  * [47] X. Liu, L. Wang, D. F. Wong, L. Ding, L. S. Chao, and Z. Tu, “Understanding and improving encoder layer fusion in sequence-to-sequence learning,” in _ICLR_ , 2021.
  * [48] L. Ding, L. Wang, and D. Tao, “Self-attention with cross-lingual position representation,” in _ACL_ , 2020.
  * [49] Q. Zhong, L. Ding, J. Liu, B. Du, H. Jin, and D. Tao, “Knowledge graph augmented network towards multiview representation learning for aspect-based sentiment analysis,” _arXiv preprint arXiv:2201.04831_ , 2022.
  * [50] J. Hao, X. Wang, B. Yang, L. Wang, J. Zhang, and Z. Tu, “Modeling recurrence for transformer,” in _NAACL-HLT_ , J. Burstein, C. Doran, and T. Solorio, Eds., 2019.
  * [51] L. Ding, L. Wang, D. Wu, D. Tao, and Z. Tu, “Context-aware cross-attention for non-autoregressive translation,” in _COLING_ , 2020.
  * [52] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in _ACL_ , 2020\.


#  3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low Bitwidth Quantization, and
Ultra-Low Latency Acceleration

Yao Chen<EMAIL_ADDRESS>
Advanced Digital Sciences Centre, Singapore ,  Cole Hawkins
<EMAIL_ADDRESS>
University of California, Santa Barbara, USA ,  Kaiqi Zhang<EMAIL_ADDRESS>
University of California, Santa Barbara, USA ,  Zheng Zhang
<EMAIL_ADDRESS>
University of California, Santa Barbara, USA and  Cong Hao
<EMAIL_ADDRESS>
Georgia Institute of Technology

(2021)

###### Abstract.

The deep neural network (DNN) based AI applications on the edge require both
low-cost computing platforms and high-quality services. However, the limited
memory, computing resources, and power budget of the edge devices constrain
the effectiveness of the DNN algorithms. Developing edge-oriented AI
algorithms and implementations (e.g., accelerators) is challenging. In this
paper, we summarize our recent efforts for efficient on-device AI development
from three aspects, including both training and inference. First, we present
on-device training with ultra-low memory usage. We propose a novel rank-
adaptive tensor-based tensorized neural network model, which offers orders-of-
magnitude memory reduction during training. Second, we introduce an ultra-low
bitwidth quantization method for DNN model compression, achieving the state-
of-the-art accuracy under the same compression ratio. Third, we introduce an
ultra-low latency DNN accelerator design, practicing the software/hardware co-
design methodology. This paper emphasizes the importance and efficacy of
training, quantization and accelerator design, and calls for more research
breakthroughs in the area for AI on the edge.

Edge AI, on-device training, DNN quantization, DNN acceleration

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
Great Lakes Symposium on VLSI 2021; June 22–25, 2021; Virtual Event,
USA††booktitle: Proceedings of the Great Lakes Symposium on VLSI 2021 (GLSVLSI
’21), June 22–25, 2021, Virtual Event, USA††price: 15.00††doi:
10.1145/3453688.3461738††isbn: 978-1-4503-8393-6/21/06††ccs: Hardware
Reconfigurable logic and FPGAs††ccs: Computing methodologies Machine learning

##  1\. Introduction

Deep neural networks (DNNs) are becoming attractive solutions for many edge AI
applications and have made remarkable progress in various areas such as
computer vision, natural language processing, health care, autonomous driving,
and surveillance. Meanwhile, with the increase of the size and complexity of
the neural networks, training and deploying a DNN with a large number of
parameters and complex data transmission on small and power-constrained edge
devices, such as smart phones and wearable devices, becomes increasingly
challenging (Hao et al., 2021; Han et al., 2016; Zhang et al., 2017). In this
work, we focus on three primary challenges: ultra-low memory training, ultra-
low bitwidth quantization, and ultra-low latency acceleration, and discuss our
solutions for each of them.

First, there is an increasing demand for on-device machine learning model
training, to preserve data privacy, enable model personalization and lifelong
learning, and to improve energy efficiency to avoid the massive data
transmission to the cloud (Teerapittayanon et al., 2017; Wang et al., 2019).
However, model training has a much larger memory requirement than inference,
exposing additional challenges for on-device training, where the edge-devices
are usually equipped with limited memory capacity. Therefore, ultra-low memory
training method must be explored to enable on-device training. To this end, we
present an end-to-end low-precision tensorized neural network training
framework with orders-of-magnitude memory reduction (Zhang et al., 2021). The
rank-adaptive tensorized training method employs a Bayesian method for
automatic tensor rank determination and model compression in the training
process.

Second, to implement DNNs on the memory-constrained edge devices, pruning and
quantization are promising to reduce the number of weights and the data bit-
width in DNN models, with an extreme case that quantizes the weights down to
binary/ternary representations (Han et al., 2016; Li and Liu, 2016;
Courbariaux et al., 2016). These methods can dramatically reduce the network
size as well as number of the multiplications during the execution of the
model. Given the tight memory and computing resource budget on the edge,
ultra-low bitwidth quantization methods are especially attractive. However,
ultra-low bitwidth quantization can easily cause significant degradation on
the model accuracy, making such aggressive quantization methods challenging.
To address such challenges, we present a novel ternary weight quantization
method by proposing a vectorized loss function, achieving the state-of-the-art
accuracy under the same compression ratio (Gong et al., 2020).

Third, for efficient DNN deployment on the edge-devices, FPGAs are becoming
attractive platforms comparing with CPUs, GPUs and digital signal processors
(DSPs) (Liu et al., 2011; Zhang et al., 2018; Qiu et al., 2016). FPGAs can
provide the flexibility to be configured as domain specific architecture that
can meet various implementation requirements such as ultra-low latency on the
edge-devices. In addition, modern SoC FPGAs integrate low power processors and
sufficient interfaces that can support widely used sensors for Internet-of-
things (IoT) applications. We present the first instruction based ternarized
low-latency deep learning accelerator with high performance, low resource
utilization, and high flexibility for different DNN models (Chen et al.,
2019b).

The remaining of this paper is organized as follows. Section 2 introduces our
low-memory rank-adaptive on-device training framework; Section 3 introduces
our low-bitwidth DNN quantization solution; Section 4 introduces our low-
latency DNN accelerator design. In Section 5 we demonstrate the effectiveness
of our proposed methods, followed by the conclusions and future work in
Section 6.

##  2\. Ultra-Low Memory Training

The large amount of model parameters consume massive computing and memory
resources, which prevents direct training of neural networks on edge devices.
A promising technique of reducing model parameters is low-rank tensor
decomposition (Kolda and Bader, 2009; Oseledets, 2011). This method has
achieved great success in post-training compression and fixed-rank training
(Zhou et al., 2019; Calvi et al., 2019; Yin et al., 2020; Tjandra et al.,
2017; Lebedev et al., 2014; Novikov et al., 2015; Garipov et al., 2016).
However, several fundamental issues need to be addressed in on-device one-shot
training:

  * •

Firstly, a rank-adaptive training framework is needed to avoid combinatorial
search of tensor ranks and multiple training runs.

  * •

Secondly, hardware-friendly tensor algorithms should be developed to
facilitate their implementation on edge devices.

In this section, we summarize our recent work on the algorithm (Hawkins and
Zhang, 2019; Hawkins et al., 2020) and hardware (Zhang et al., 2021) levels to
address these challenges.

(a)

(b)

(c)

(d)

Figure 1. (a): An order-3 tensor. (b) and (c): CP and Tucker representations,
respectively. (d): TT representation, where the gray lines and squares
indicate a slice of the TT core by fixing its mode index. This figure is
reproduced from (Hawkins et al., 2020).

###  2.1. Bayesian Tensorized Training Models

####  2.1.1.  Low-rank tensor representation.

In many cases we can describe a neural network with much less parameters via
low-rank tensors. Consider a weight matrix $\mathbf{W}\in\mathbb{R}^{J\times
I}$ for example (and other parameters such as convolutional filters and
embedding tables can be handled similarly). We can firstly fold $\mathbf{W}$
to a high-dimensional tensor $\mathcal{W}$ of size $J_{1}\times\dots\times
J_{d}\times I_{1}\times\dots\times I_{d}$, where
$I=\prod_{n=1}^{d}I_{n},J=\prod_{n=1}^{d}J_{n}$. Then, we can describe the
tensor $\mathcal{W}$ with some low-rank tensor factors $\boldsymbol{\Phi}$.
This can be done with various low-rank tensor decomposition formats as shown
in Fig. 1 (Hawkins et al., 2020). In various tensor decompositions,
$\boldsymbol{\Phi}$ denotes the associated tensor factors. For large fully
connected layers and embedding tables, the tensor-train matrix (TTM) format
turns to be highly effective (Hawkins et al., 2020). In the TTM format,
$\boldsymbol{\Phi}=\\{\mathcal{G}^{(n)}\\}_{n=1}^{d}$, and each
$\mathcal{G}^{(n)}\in\mathbb{R}^{R_{n-1}\times J_{n}\times I_{n}\times R_{n}}$
is an order-$4$ TTM core. The vector $\mathbf{R}=(R_{0},R_{1},\cdots,R_{d})$
with $R_{0}=R_{d}=1$ is the tensor ranks that determine the model complexity.
With low-rank tensors, one may reduce the number of model parameters from an
exponential function of $d$ to a linear one.

####  2.1.2. Bayesian Tensorized End-to-End Training.

Despite the high compression ratio via tensor methods, determining the tensor
rank in advance is very hard (Hillar and Lim, 2013). This is further
complicated by the nonlinear forward model in neural networks, which has
prevented tensorized one-shot on-device training in previous works. We have
developed two Bayesian models to address this issue:

  * •

Stein Variational Inference for TTM Format. In (Hawkins and Zhang, 2019), we
have considered TTM format. We model each slice of $\mathcal{G}^{(k)}$ with a
zero-mean Gaussian prior density. We further control the variance by two
tunable Gamma hyper-priors to enforce low tensor ranks. The actual tensor rank
is decided jointly by the training data and rank-controlling hyper-parameters.
Starting from an initial rank parameter $R_{k}$, we can learn an actual rank
$\hat{R}_{k}\leq R_{k}$, leading to further model compression in the training
process. This method uses a Stein variational inference (Liu and Wang, 2016)
to compute the posteior density for small- or medium-size neural networks.

  * •

Scalable SVI for One-Shot Tensorized Training.  In (Hawkins et al., 2020), we
have developed a more generic and efficient Bayesian model for tensorized
training. This work can handle CP, Tucker, TT and TTM formats. It uses
Gaussian priors to model low-rank tensor factors, and uses Half-Cauchy or Log-
Uniform hyper-priors to control tensor ranks. We have improved the stochastic
variational inference (SVI) (Hoffman et al., 2013) by two steps. Firstly, we
simplify the posterior density of rank-controlling hyper-parameters to a Delta
function to avoid gradient explosion. Secondly, we use a hybrid
numerical/analytical update rule inside SVI. This highly scalable method can
perform one-shot training of very large-scale neural networks with billions of
model parameters.

####  2.1.3. Performance Summary.

  * •

Our first method (Hawkins and Zhang, 2019) has been tested on a two-layer
fully connected neural network, a 6-layer CNN and a 110-layer residual neural
network. Our work has produced $7.4\times$ to $137\times$ more compact neural
networks directly from the training with little or no accuracy loss.

  * •

Our recent work (Hawkins et al., 2020) has been tested on a practical CNN, a
large-scale NLP model (Khrulkov et al., 2019) and an extremely large deep
learning recommendation model (DLRM) (Naumov et al., 2019) from Facebook.
Orders-of-magnitude parameter reduction has been achieved in the training
process. As shown in Table 1, training the DLRM with a standard method
involves $4.25\times 10^{9}$ variables. Our proposed method only trains
$2.36\times 10^{6}$ variables due to low-rank tensorization, and it further
reduce the model parameters to $164$K in the training process due to the
automatic rank determination. The overall parameter reduction ratio in the
training process is $2.6\times 10^{4}$.

Table 1. Performance of our tensorized training (Hawkins et al., 2020) on the Facebook DLRM model. | standard | tensorization | rank-adaptive training
---|---|---|---
# parameters | 4.25B | 2.36M | 164K
compression | N/A | $1,800\times$ | $26,000\times$

###  2.2. One-Shot On-Device Tensorized Training

To demonstrate on-device training, we have developed a low-precision
tensorized training algorithm and its FPGA prototype (Zhang et al., 2021).

####  2.2.1. Low-Precision Tensorized Training.

We consider the maximum a posteriori probability (MAP) estimate of the
Bayesian model (Hawkins et al., 2020). In this case, the training loss
function includes two parts: the cross-entropy loss of a neural network
classifier dependent on TTM factors $\\{\mathcal{G}^{(k)}\\}_{k=1}^{d}$, and a
regularization term caused by the Gaussian priors of TTM factors as well as
the Log-Uniform hyper-priors for rank-controlling parameters
$\boldsymbol{\lambda}_{k}$’s. In the training process, both TTM factors and
rank-controlling parameters will be computed. To reduce the training cost on
hardware, a low-precision tensorized training algorithm is developed based on
the following key ideas:

  * •

We use BinaryConnect (Courbariaux et al., 2015) to compute low-precision TTM
factors. BinaryConnect keeps the real values of all low-precision parameters
in a buffer. In each iteration, the gradients are accumulated in the buffer,
and the low-precision parameters are updated by quantizing the buffer. To
handle the non-differentiable quantization function in the training process,
we use the straight-through estimator (STE) (Bengio et al., 2013) to
approximate its gradient.

  * •

We use different precisions for different variables in the training process.
Specifically, we use 4 bits to represent TT factors, 8 bits for activations
and bias, and 16 bits for the gradients.

####  2.2.2. On-FPGA Training.

To demonstrate our training algorithms on edge devices, we have implemented an
FPGA accelerator as shown in Fig. 2 for the low-precision tensorized training
framework.

  * •

Since our low-rank tensorization can greatly reduce the training variables,
all model parameters may be stored in the on-chip BRAM. The data samples,
activations, and gradients are stored in the off-chip DRAM during the training
process.

  * •

The forward and backward propagations are run on the FPGA programmable logic.
The TTM factors and rank-controlling parameters are updated on the embedded
ARM core.

  * •

Three processing elements (PEs) are designed for the forward and backward
propagation. PE1 and PE2 are shared by the forward and backward propagations,
and they handle tensor contractions. PE1 is used for a two-index tensor
contraction which contains the last dimension of two tensor variables. In
contrast, PE2 performs a tensor contraction along a single dimension that is
not the last. PE3 computes the outer products in a backward propagation.

Figure 2. Our FPGA accelerator for the end-to-end tensorized training.
Reproduced from (Zhang et al., 2021).

##  3\. Ultra-low Bitwidth Quantization

Neural network quantization employs low precision (bitwidth) data for
efficient model execution. Especially, ultra-low bitwidth quantization leads
to much less memory usage, lower complexity of the multiply-accumulate
operations, and higher efficiency of model execution, making it an appealing
technology for enabling AI at edge devices. However, aggressively lowering the
data bitwidth (e.g., lower than 4-bit) is very challenging:

  * •

It can easily result in large accuracy degradation (Hao et al., 2019; Chen et
al., 2019b; Gysel and other, 2016), requiring a careful balance between the
computing efficiency and the final model accuracy.

  * •

Minimizing the quantization loss, i.e., the L2 distance between the original
and the quantized values, is an appealing method (Han et al., 2016; Leng et
al., 2018a; Wang et al., 2018; Cheng et al., 2019; Li and Liu, 2016; Leng et
al., 2018b) but have major drawbacks such as easily falling into local optima
and neglecting the distribution and correlations of the weights (Gong et al.,
2020).

To address such challenges and achieve high-accuracy ultra-low bitwidth
quantization, we have proposed a quantization method, namely VecQ (Gong et
al., 2020), with a novel vectorized loss function and an open-sourced training
flow. VecQ can quantize the model weights into 1-bit to 16-bit and shows
exceptional performance especially under ultra-low bitwidth, e.g., ternary
values.

Figure 3. The quantization angle between $\omega_{f}$ and $\omega_{q}$.

Vectorized Loss Function. We organize the weights within one DNN layer into a
vector $\omega\in\mathbb{R}^{l}$ where $l$ is the number of weights, and
denote the original floating point weight vector by $\omega_{f}$ and the
quantized weight vector by $\omega_{q}$. Typically, there is a scaling factor
$\alpha$ such that $\omega_{q}=\alpha\boldsymbol{v}$, where each element in
$\boldsymbol{v}$ is a low-bitwidth representation. Based on the vector
representations, we define the quantization angle between $\omega_{f}$ and
$\omega_{q}$, denoted by $\theta$. Figure 3 shows an example when $l=2$. The
objective is to find optimal $\alpha$ and $\boldsymbol{v}$ such that
$\omega_{q}$ is as close to $\omega_{f}$ as possible.

We propose the vectorized loss to describe the quantization loss, denoted by
$J_{v}$, and we minimize $J_{v}$ during the quantization. $J_{v}$ is defined
as the summation of the orientation loss, denoted by $J_{o}$, and the modulus
loss, denoted by $J_{m}$ as follows:

(1)
$\begin{split}J_{o}&=1-\cos\theta,\quad\text{where}\;\cos\theta=\frac{\alpha\boldsymbol{v}\cdot\omega_{f}}{||\alpha\boldsymbol{v}||\cdot||\omega_{f}||}\\\
J_{m}&=||\omega_{f}-\alpha\boldsymbol{v}||_{2}^{2}\\\
J_{v}&=J_{o}+J_{m}\end{split}$

The orientation loss describes the angle between two vectors, while the
modulus loss describes the squared distance between $\omega_{f}$ and
$\omega_{q}$. Notably, by minimizing $J_{v}$, we usually achieve lower
quantization loss comparing with directly minimizing $J_{m}$. More details can
be found in the VecQ paper (Gong et al., 2020).

Vectorized Loss Minimization. We minimize $J_{v}$ in two steps, namely
steering and driving, as shown in Figure 4. First, the steering step minimizes
the orientation loss $J_{o}$ to find the best $\boldsymbol{v}$, since $J_{o}$
is independent of $\alpha$. Second, the driving step minimizes the modulus
loss $J_{m}$ to find the best scaling factor $\alpha$. For a convolution
layer, all the weights within the same layer share the same scaling factor;
for a depth-wise convolution layer, each kernel has its own scaling factor for
a better representation of the less number of weights for it. We quantize the
activations to fixed point values during the training to further reduce the
memory utilization.

Figure 4. The data quantization flow of VecQ, reproduced from VecQ (Gong et
al., 2020).

Training Flow Integration. We integrate our VecQ solution into the Tensorflow
and Pytorch DNN training frameworks. For each layer, in the forward
propagation, we first quantize the weights from $\omega_{f}$ to $\omega_{q}$,
and then use $\omega_{q}$ to compute the output activations, which is also
quantized into fixed point. In the backward propagation, the gradients are
also calculated using $\omega_{q}$ to update $\omega_{f}$.

##  4\. Ultra-low Latency Acceleration

The effectiveness of a dedicated FPGA accelerator for DNN models have been
widely demonstrated (Qiu et al., 2016; Chen et al., 2019a). However, ultra-low
latency accelerators for edge devices with an extremely limited resource
budget still require careful design considerations.

Benefiting from our quantization solution VecQ for ultra-low bitwidth, we have
proposed T-DLA, a light-weight ternarized accelerator overlay under strict
resource constraint, to achieve ultra-low latency on edge devices (Chen et
al., 2019b). The key features of T-DLA include:

  * •

An optimized and expressive single instruction multiple data (SIMD)
instruction set.

  * •

A novel memory sub-system supporting effective data access of the computation
modules.

  * •

An efficient execution pipeline with low-latency computation modules.

Table 2. The format of the instruction word in T-DLA.

Byte Idx | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0
---|---|---|---|---|---|---|---|---
Load | OP | FS | SAM | SAL | DAM | DAL | KS | CC

11footnotemark: 1

OP: operation code 22footnotemark: 2FS: input feature size
33footnotemark: 3SAM/SAL: source address most/least significant byte
44footnotemark: 4DAM/DAL: destination address most/least significant byte
55footnotemark: 5KS: kernel size 66footnotemark: 6CC:
in/out/activation/pooling selection

SIMD Instruction Set. To support the task scheduling for various DNN models,
the instruction set of T-DLA is designed as simple yet expressive enough for a
large variety of DNNs. Each instruction is a 64-bit word (8 bytes) with the
format shown in Table 2. The payloads of the different bytes are generated
according to the layer configurations.

Memory Sub-system. The memory subsystem contains two levels of storage to
provide low latency data fetching to the computation units: a simple input
buffer and a variable-length line buffer. The simple input buffer is a BRAM
buffer for temporary input feature storage; the variable line buffer serves
for the efficient data streaming into the ternary computation array, as shown
in Figure 5. It is designed to support variable kernel size $k$ and variable
buffer depth $d$, which are specified by the instruction to reduce the data
transmission latency caused by fixed hardware paths. Once configured by the
instruction, it provides an output of $k\times k$ data each clock cycle to the
computation array.

Figure 5. The variable-length line buffer. The kernel size and the depth of
the buffers are both configurable.

Execution Pipeline and Computation Modules. T-DLA has four major computation
modules: 1) a ternary computation array, 2) a set of adder trees, 3)
activation and scaling modules, and 4) pooling modules.

Ternary computation array. With our ternarized model training via VecQ, the
weights are represented by 2 bits using two’s complement encoding, so that the
multiplication in the convolution layer is simplified to selection and
inversion logic. Benefiting from such simplified logic, we can achieve
parallelism along the input channel, the output channel, and the kernel
dimensions. The computation array is constructed by $T_{n}\times T_{m}\times
L_{K}^{2}$ computation units, which can process this number of input data
simultaneously. $T_{n}$ and $T_{m}$ are the maximum numbers of the input and
output channel that can be processed by the computation array, and $L_{K}$ is
the pre-defined maximum allowable kernel size. The values of
$T_{n},T_{m},L_{K}$ and the length of the line buffer $L_{D}$ are all
configurable and could be determined based on the on-chip resource
availability.

Adder tree. Since the computation array is built only using LUTs and FFs, we
use DSPs to construct adder trees. We take advantage of the SIMD mode of the
DSPs where the internal carry propagation between segments is blocked to
ensure independent operation. Therefore, we split the 48-bit input of a DSP
into four 12-bit independent accumulation channels, so that a single DSP can
perform addition for 8 pieces of input data and provide 4 outputs. Benefiting
from the SIMD mode, the DSPs can provide outputs in every single clock once
the internal register lines are filled up. Furthermore, the clock frequency of
the DSPs are configured to be higher than other logic parts with the help of
input/output asynchronous FIFOs, which further reduces the processing latency.

Other modules. The ReLU activation module, the linear scaling module, and the
max pooling module are all designed to process in a single clock cycle to
reduce the depth of the execution pipeline.

##  5\. Experimental Results

In this section we demonstrate the effectiveness of our methods, including the
ultra-low memory training framework, the ultra-low bitwidth VecQ quantization,
and the ultra-low latency T-DLA design. Notbly, all these works are open-
sourced.

Table 3. The on-device ultra-low memory training method on the Fashion MNIST dataset. Method |  Training accuracy |  Testing accuracy |  Model parameters |  Memory in bits |  Memory reduction
---|---|---|---|---|---
Vanilla | 95.75% | 89.27% | $4.67\times 10^{5}$ | $1.49\times 10^{7}$ | N/A
Floating, w/o prior | 92.54% | 88.03% | $1.48\times 10^{4}$ | $4.74\times 10^{5}$ |  31.4$\times$
Fixed, w/o prior | 88.31% | 86.67% | $1.48\times 10^{4}$ | $6.13\times 10^{4}$ |  243$\times$
Floating, w/ prior | 90.17% | 87.88% | $1.08\times 10^{4}$ | $3.46\times 10^{5}$ | $43.1\times$
Fixed, w/ prior | 85.45% | 84.86% | $1.22\times 10^{4}$ | $5.11\times 10^{4}$ |  292$\times$

###  5.1. On-device Training

We implement our low-precision rank-adaptive tensorized training on an Avnet
Ultra96-V2 FPGA board and use it to train a two-layer neural network for a
classification task on the FashionMNIST dataset. There are 512 neurons in the
hidden layer, folded into $4\times 4\times 2\times 16$ for the first layer,
and $32\times 16$ for the second layer. We use the Pytorch and Tensorly
modules to implement our training algorithm on the embedded processor. For the
FPGA we set the clock rate to 100MHz. We compare the training methods with or
without the low rank TT priors. As shown in Table 3, our method achieves
$294\times$ memory reduction for the model parameters compared with the
standard non-tensorized training. This on-FPGA training has achieved
$59\times$ speedup and $123\times$ energy reduction than the training on an
embedded CPU.

###  5.2. Ultra-low Bitwidth Quantization

Table 4. VecQ: top-1 classification accuracy Dataset | MNIST | CIFAR10 | CIFAR10 | ImageNet
---|---|---|---|---
Model | Lenet-5 | Cifarnet | VGG-like | Resnet-18
Floating | 99.41 | 80.54 | 93.49 | 69.60
IJCNN’17 (Alemdar et al., 2017) | 98.33 | - | 87.89 | -
NIPS’16 (Li and Liu, 2016) | 99.35 | - | 92.56 | 61.8
Ours | 99.5 | 78.7 | 92.94 | 68.23

We use MNIST, Cifar10 and ImageNet to evaluate the ultra-low bitwidth
quantization, VecQ. The evaluated DNN models include Lenet-5, Cifarnet, a VGG-
like network (Prost-Boucle et al., 2017), and Resnet-18.

Table 5. VecQ: model size reduction Model | Lenet-5 | Cifarnet | VGG-like | Resnet-18
---|---|---|---|---
Param. Total (M) | 0.43 | 0.279 | 5.35 | 11.69
Param. Conv (M) | 0.025 | 0.258 | 1.114 | 11.177
Floating (MB) | 1.644 | 1.065 | 20.408 | 44.594
Ours (MB) | 0.393 | 0.081 | 4.284 | 3.154
Mem.Reduc.(%) | 76.09 | 92.39 | 79.01 | 92.93
Table 6. T-DLA resource and performance Configuration Parameters $<T_{n},T_{m},L_{K},L_{D},D_{w}>$ | $<4,16,5,32,12>$
---|---
Resource Utilization(%) $<LUT/FF/BRAM/DSP>$ | 79 / 47.47 / 68.93 / 91.82
Clock Frequency Logic / Adder (MHz) | 125 / 250
Peak Performance (GOPS) | 400
DNN Model | Lenet-5 | Cifarnet | VGG-like | Resnet-18
latency(ms) | 0.016 | 0.063 | 2.12 | 48.8
Table 7. T-DLA: Comparison with the state-of-the-art implementations.

Dataset | Design | Model | Acc.(%) | F., W. (bits) | fps | platform
---|---|---|---|---|---|---
MNIST | (Umuroglu et al., 2017) | MFC-max | 97.69 | 1, 1 | 6238000 | ZC706
MNIST | (Park and Sung, 2016) | Lenet-5 | - | 8, 3 | 70000 | ZC706
MNIST | Ours | Lenet-5 | 99.5 | 8, 2 | 62051.1 | Zedboard
CIFAR10 | (Umuroglu et al., 2017) | VGG-like | 80.1 | 24, 1 | 21900 | ZC706
CIFAR10 | (Nakahara et al., 2017) | VGG-like | 81.8 | 1, 1 | 420 | Zedboard
CIFAR 10 | (Prost-Boucle et al., 2017) | VGG-like | 86.71 | 8, 2 | 27043 | VC709
CIFAR 10 | (Zhao et al., 2017) | VGG-like | 88.68 | 1, 1 | 168 | Zedboard
CIFAR 10 | Ours | VGG-like | 89.08 | 8,2 | 457 | Zedboard
ImageNet | (Li and Liu, 2016) | Resnet-18 | 65.44 | FP32,FP32 | 1.545 |  Xeon11footnotemark: 1
ImageNet | (Li and Liu, 2016) | Resnet-18 | 65.44 | FP32,FP32 | 387.597 |  1080Ti22footnotemark: 2
ImageNet | Ours | Resnet-18 | 68.23 | 8, 2 | 20.48 | Zedboard

11footnotemark: 1

Xeon: Xeon E5-2630 v3; 22footnotemark: 21080Ti: Nvidia 1080Ti

####  5.2.1. Classification accuracy

The classification accuracy on different datasets are shown in Table 4. For
simplicity, we only show the top-1 accuracy. Comparing to the floating point
models (Floating in the table), the classification accuracy using ternary
weights and quantized scalars and activations shows negligible degradation.
VecQ also achieves superior accuracy comparing to the recent works (Alemdar et
al., 2017; Li and Liu, 2016), in which only the weights are ternarized but not
the scalars and activations. Our proposed method shows better accuracy for
Resnet-18 on ImageNet data set. This result demonstrates the scalability and
stability of VecQ, especially in aggressive low-bitwidth quantization
scenarios.

####  5.2.2. Model size reduction

VecQ also greatly reduces the memory footprint (Mem. Reduc.) as shown in Table
5. Ternary weight occupies only 2 bits whereas the original floating point
requires 32 bits. As shown in Table 5, for convolution layers, VecQ compresses
the parameters nearly to the theoretical limit (almost $16\times$ reduction).
We quantize the last FC layer to 12-bit to maintain accuracy, so that the
networks with less or no FC layers have higher compression ratio, such as
Cifarnet and Resnet-18. Specifically, VecQ reduces up to 92.93%
(14.14$\times$) size of Resnet-18 in floating point.

###  5.3. Ultra-low Latency Acceleration

We use the models quantized by VecQ to evaluate our T-DLA accelerator design
in terms of accuracy and frame per second (fps). The measurements of the
original models are on a server with two Intel Xeon E5-2630 v3 CPUs and one
Nvidia 1080 Ti GPU. T-DLA is implemented on a Xilinx Zedboard FPGA, which is
suitable for edge applications with very limited logic resources. It has an
on-chip dual-core ARM Cortex A9, and has 53.2K LUTs, 106.4K FFs, 140 BRAM
blocks of 36Kb each, and 220 DSPs. Vivado System Design Suite 2019.2 is used
for system implementation.

####  5.3.1. Hardware Resource and Processing Latency Evaluation

We choose an accelerator configuration that fully utilizes the given
resources, shown in Table 6, together with the execution latency of the
different models with this configuration. We only show the most important
configuration parameters including $T_{n},T_{m},L_{K},L_{D}$, and the
quantized bitwidth of the activations ($D_{w}$). As can be seen in Table 6,
T-DLA with customized configuration can almost use all the resources,
especially the DSPs. The targeted FPGA can support up to 250MHz for the DSPs,
which is twice of the frequency of other logic benefiting from the ternary
computation array and independent clock design of the adder trees.

####  5.3.2. Performance Comparison

We compare T-DAL in terms of accuracy and fps with existing designs, either
using the same DNN model or the same dataset. The results are shown in Table
7. For MNIST dataset, the design in (Umuroglu et al., 2017) shows higher fps
because of the DNN model they used is simpler and the ZC706 platform has
almost $4\times$ more resources than ours. However, our implementation on
Zedboard has a comparable fps (62051) to a design (Park and Sung, 2016)
(70000) with 3-bit weights on the ZC706 platform. On CIFAR10 dataset, our
design shows dominating accuracy advantage among all the VGG-like models. On
ImageNet dataset, we directly compare our results with the floating point
version. T-DLA shows longer execution latency than the GPU but outperforms the
CPU by $9.2\times$.

##  6\. Conclusions

In this paper, we summarized our recent efforts for efficient on-device AI
development including both training and inference. We mainly focused on three
major challenges of edge AI development. First, we presented on-device
training with ultra-low memory usage by proposing a novel rank-adaptive
tensor-based tensorized neural network model, which offers orders-of-magnitude
memory reduction during training. Second, we introduced VecQ, a novel
quantization method that supports ultra-low bitwidth quantization with
negligible accuracy degradation. Third, we presented T-DLA, an ultra-low
latency DNN accelerator design for ternarized DNNs achieving the state-of-the-
art performance. On top of the achievements in this paper, we expect more
research breakthroughs to boost the development and deployment for the edge
AI.

## References

  * (1)
  * Alemdar et al. (2017) H. Alemdar et al. 2017\.  Ternary neural networks for resource-efficient AI applications. In _IJCNN_.
  * Bengio et al. (2013) Yoshua Bengio et al. 2013\.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _arXiv preprint arXiv:1308.3432_ (2013).
  * Calvi et al. (2019) Giuseppe G Calvi et al. 2019\.  Tucker tensor layer in fully connected neural networks.  _arXiv preprint arXiv:1903.06133_ (2019).
  * Chen et al. (2019a) Yao Chen et al. 2019a.  Cloud-DNN: An Open Framework for Mapping DNN Models to Cloud FPGAs. In _FPGA_.
  * Chen et al. (2019b) Yao Chen et al. 2019b.  T-DLA: An Open-source Deep Learning Acceleratorfor Ternarized DNN Models on Embedded FPGA.  _ISVLSI_ (2019).
  * Cheng et al. (2019) Gong Cheng et al. 2019\.  $\mathrm{\mu}$L2Q: An Ultra-Low Loss Quantization Method for DNN Compression.  (2019).
  * Courbariaux et al. (2016) M. Courbariaux et al. 2016\.  Binarynet: training deep neural networks with weights and activations constrained to +1 or -1.  _arXiv_ (2016).
  * Courbariaux et al. (2015) Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2015\.  Binaryconnect: Training deep neural networks with binary weights during propagations. In _Advances in neural information processing systems_. 3123–3131.
  * Garipov et al. (2016) Timur Garipov et al. 2016\.  Ultimate tensorization: compressing convolutional and fc layers alike.  _arXiv preprint arXiv:1611.03214_ (2016).
  * Gong et al. (2020) Cheng Gong et al. 2020\.  VecQ: Minimal loss dnn model compression with vectorized weight quantization.  _IEEE Trans. Comput._ (2020).
  * Gysel and other (2016) Philipp Gysel and other. 2016.  Hardware-oriented Approximation of Convolutional Neural Networks.  _CoRR_ abs/1604.03168 (2016).
  * Han et al. (2016) Song Han et al. 2016\.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding.  (2016).
  * Hao et al. (2019) Cong Hao et al. 2019\.  FPGA/DNN Co-Design: An Efficient Design Methodology for IoT Intelligence on the Edge. In _DAC_.
  * Hao et al. (2021) C. Hao et al. 2021\.  Enabling Design Methodologies and Future Trends for Edge AI: Specialization and Co-design.  _IEEE Design Test_ (2021), 1–1.
  * Hawkins et al. (2020) Cole Hawkins, Xing Liu, and Zheng Zhang. 2020.  Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination.  _arXiv preprint arXiv:2010.08689_ (2020).
  * Hawkins and Zhang (2019) Cole Hawkins and Zheng Zhang. 2019.  Bayesian Tensorized Neural Networks with Automatic Rank Selection.  _arXiv preprint arXiv:1905.10478_ (2019).
  * Hillar and Lim (2013) Christopher J Hillar and Lek-Heng Lim. 2013.  Most tensor problems are NP-hard.  _Journal of the ACM (JACM)_ 60, 6 (2013), 45\.
  * Hoffman et al. (2013) Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. 2013\.  Stochastic variational inference.  _The Journal of Machine Learning Research_ 14, 1 (2013), 1303–1347.
  * Khrulkov et al. (2019) Valentin Khrulkov, Oleksii Hrinchuk, Leyla Mirvakhabova, and Ivan Oseledets. 2019.  Tensorized embedding layers for efficient model compression.  _arXiv preprint arXiv:1901.10787_ (2019).
  * Kolda and Bader (2009) Tamara G Kolda and Brett W Bader. 2009.  Tensor decompositions and applications.  _SIAM review_ 51, 3 (2009), 455–500.
  * Lebedev et al. (2014) Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. 2014.  Speeding-up convolutional neural networks using fine-tuned cp-decomposition.  _arXiv preprint arXiv:1412.6553_ (2014).
  * Leng et al. (2018a) Cong Leng et al. 2018a.  Extremely Low Bit Neural Network: Squeeze the Last Bit Out With ADMM. In _AAAI_.
  * Leng et al. (2018b) Cong Leng et al. 2018b.  Extremely low bit neural network: Squeeze the last bit out with admm. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 32.
  * Li and Liu (2016) F Li and B Liu. 2016\.  Ternary Weight Networks.  _Proc. NIPS Workshop Efficient Methods Deep Neural Network_ (2016).
  * Liu and Wang (2016) Qiang Liu and Dilin Wang. 2016.  Stein variational Gradient descent: a general purpose Bayesian inference algorithm. In _Proceedings of the 30th International Conference on Neural Information Processing Systems_. 2378–2386.
  * Liu et al. (2011) S. Liu et al. 2011\.  Real-time object tracking system on FPGAs. In _SAAHPC_.
  * Nakahara et al. (2017) H. Nakahara et al. 2017\.  A fully connected layer elimination for a binarizec convolutional neural network on an FPGA. In _FPL_.
  * Naumov et al. (2019) Maxim Naumov et al. 2019\.  Deep learning recommendation model for personalization and recommendation systems.  _arXiv preprint arXiv:1906.00091_ (2019).
  * Novikov et al. (2015) Alexander Novikov et al. 2015\.  Tensorizing neural networks. In _Advances in neural information processing systems_. 442–450.
  * Oseledets (2011) Ivan V Oseledets. 2011\.  Tensor-train decomposition.  _SIAM Journal on Scientific Computing_ 33, 5 (2011), 2295–2317.
  * Park and Sung (2016) Jinhwan Park and Wonyong Sung. 2016.  FPGA Based Implementation of Deep Neural Networks Using On-chip Memory Only.  _CoRR_ (2016).
  * Prost-Boucle et al. (2017) A. Prost-Boucle et al. 2017\.  Scalable high-performance architecture for convolutional ternary neural networks on FPGA. In _FPL_.
  * Qiu et al. (2016) Jiantao Qiu et al. 2016\.  Going deeper with embedded fpga platform for convolutional neural network. In _FPGA_. 26–35.
  * Teerapittayanon et al. (2017) Surat Teerapittayanon et al. 2017\.  Distributed Deep Neural Networks Over the Cloud, the Edge and End Devices. In _ICDCS_. 328–339.
  * Tjandra et al. (2017) Andros Tjandra et al. 2017\.  Compressing recurrent neural network with tensor train. In _2017 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 4451–4458.
  * Umuroglu et al. (2017) Yaman Umuroglu et al. 2017\.  FINN: A Framework for Fast, Scalable Binarized Neural Network Inference. In _FPGA_.
  * Wang et al. (2018) Peisong Wang et al. 2018\.  Two-step quantization for low-bit neural networks. In _CVPR_.
  * Wang et al. (2019) Yue Wang et al. 2019\.  E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings. In _NeurIPS_.
  * Yin et al. (2020) Miao Yin et al. 2020\.  Compressing Recurrent Neural Networks Using Hierarchical Tucker Tensor Decomposition.  _arXiv preprint arXiv:2005.04366_ (2020).
  * Zhang et al. (2021) Kaiqi Zhang et al. 2021\.  On-FPGA Training with Ultra Memory Reduction: A Low-Precision Tensor Method.  _ICLR Workshop of Hardware Aware Efficient Training_ (2021).
  * Zhang et al. (2017) Xiaofan Zhang et al. 2017\.  Machine learning on FPGAs to face the IoT revolution. In _ICCAD_.
  * Zhang et al. (2018) Xiaofan Zhang et al. 2018\.  DNNBuilder: an automated tool for building high-performance Dnn hardware accelerators for FPGAs. In _ICCAD_.
  * Zhao et al. (2017) Ritchie Zhao et al. 2017\.  Accelerating Binarized Convolutional Neural Networks with Software-Programmable FPGAs. In _FPGA_.
  * Zhou et al. (2019) Mingyi Zhou et al. 2019\.  Tensor rank learning in CP decomposition via convolutional neural network.  _Signal Processing: Image Communication_ 73 (2019), 12–21.