# Task-Independent Knowledge Makes for Transferable Representations for
Generalized Zero-Shot Learning

Chaoqun Wang1 2, Xuejin Chen111Corresponding Author1 2, Shaobo Min2, Xiaoyan
Sun2, Houqiang Li1 2

###### Abstract

Generalized Zero-Shot Learning (GZSL) targets recognizing new categories by
learning transferable image representations. Existing methods find that, by
aligning image representations with corresponding semantic labels, the
semantic-aligned representations can be transferred to unseen categories.
However, supervised by only seen category labels, the learned semantic
knowledge is highly task-specific, which makes image representations biased
towards seen categories. In this paper, we propose a novel Dual-Contrastive
Embedding Network (DCEN) that simultaneously learns task-specific and task-
independent knowledge via semantic alignment and instance discrimination.
First, DCEN leverages task labels to cluster representations of the same
semantic category by cross-modal contrastive learning and exploring semantic-
visual complementarity. Besides task-specific knowledge, DCEN then introduces
task-independent knowledge by attracting representations of different views of
the same image and repelling representations of different images. Compared to
high-level seen category supervision, this instance discrimination supervision
encourages DCEN to capture low-level visual knowledge, which is less biased
toward seen categories and alleviates the representation bias. Consequently,
the task-specific and task-independent knowledge jointly make for transferable
representations of DCEN, which obtains averaged $4.1\%$ improvement on four
public benchmarks.

## Introduction

Deep learning-based methods are highly successful on various computer vision
tasks, such as image classification (He et al. 2016), object detection
(Girshick et al. 2014), and semantic segmentation (Badrinarayanan, Kendall,
and Cipolla 2017). However, these models require a huge demand for manually
labelled training data for numerous classes. To this end, Generalized Zero-
Shot Learning (GZSL), which aims to recognize either seen or unseen categories
thereby reduces manual annotation labors, recently has attracted great
interests.

Figure 1: Motivation of this paper. (a) Existing methods focus on using task
labels to learn semantic-aligned representations, which can be transferred to
unseen categories. (b) Besides, this paper further learns task-independent
knowledge via instance discrimination supervision, which significantly
improves the representation transferability.

Due to unavailable unseen category data during training, GZSL puts a high
demand on transferability of image representations (Jiang et al. 2018; Tong et
al. 2019). Previous methods tackle this problem by aligning image
representations with corresponding category attributes (Xian et al. 2018a), as
shown in Fig. 1 (a). Since seen and unseen domain categories share a common
attribute space, the semantic-aligned representation can be transferred to
unseen categories, and the recognition becomes a nearest neighbour searching
problem, _i.e.,_ images and category attributes serve as queries and anchors,
respectively. Based on this paradigm, most GZSL methods focus on designing
elaborate semantic-visual alignment mechanism, such as visual and attribute
attentions (Zhu et al. 2019; Xie et al. 2019; Huynh and Elhamifar 2020),
semantic auto-encoders (Felix et al. 2018; Min et al. 2019), and domain
detectors (Atzmon and Chechik 2019). However, trained with only seen category
labels, the learned image representations are highly task-specific and biased
towards seen categories, which makes an unseen image tend to be recognized as
seen categories.

To alleviate this issue, we introduce task-independent knowledge that is
learned without seen category concepts and shared by both seen and unseen
images. We propose a novel Dual-Contrastive Embedding Network (DCEN) that
simultaneously learns task-specific and task-independent knowledge to obtain
transferable representations in GZSL. Specifically, we design two modules in
DCEN,  _i.e.,_ Semantic Contrastive Module (SCM) and Visual Contrastive Module
(VCM), to enhance the semantic-visual alignment knowledge via task labels and
introduce annotation-free visual knowledge via instance discrimination,
respectively.

First, SCM is developed to leverage semantic labels to learn task-specific
knowledge. By constructing cross-modal triplets, SCM constrains the image
representation to be not only aligned with the corresponding category but also
far away from the most confusing category, which improves the inter-class
representation discrimination. Besides, we design masked attribute prediction
to complete the missing category attributes from corresponding image
representations. This renders the image representation to preserve the
semantic-visual complementarity, resulting in better semantic-visual
alignment. With cross-modal contrastive learning and masked attribute
prediction, SCM can better bridge the semantic-visual gap than previous
methods to learn semantic-aligned representation.

Second, VCM is designed to learn task-independent knowledge by distinguishing
each image as an individual category, as shown in Fig. 1 (b). This forces the
image representation to capture as many detailed visual cues as possible for
each individual image, so that all images can be separated apart. Without task
labels, the learned low-level visual knowledge is only related to intra-image
invariance and inter-image difference, which is irrelevant to seen category
concepts. Thus, the learned knowledge is less biased towards the seen
categories than the task-specific knowledge from seen category labels, and
thereby significantly improves the representation transferability.

Consequently, the task-independent knowledge from instance discrimination and
task-specific knowledge from semantic alignment jointly make for transferable
representations of our DCEN, which achieves averaged $4.1\%$ improvement on
four widely-used GZSL benchmarks. Our overall contribution is threefold:

  * •

We propose a novel Dual-Contrastive Embedding Network (DCEN) that
simultaneously learns task-specific and task-independent knowledge to produce
transferable representations in GZSL. To the best of our knowledge, this is
the first work that introduces task-independent knowledge from instance
discrimination to boost representation transferability in GZSL.

  * •

DCEN learns task-independent knowledge by exploring inter-image visual
distinction and intra-image transformation invariance. Extensive experiments
are conducted to search important low-level invariance concepts for GZSL.

  * •

DCEN learns task-specific knowledge by constructing cross-modal contrastive
learning and exploring semantic-visual complementarity to better bridge the
semantic-visual gap.

## Related Works

### Generalized Zero-Shot Learning

This paper belongs to Inductive GZSL (Xian et al. 2016; Kodirov, Xiang, and
Gong 2017; Min et al. 2020), where the unseen domain data is unavailable
during training. A general solution is to learn a joint embedding space, where
the image representations and semantic labels,  _e.g.,_ category attributes
(Farhadi et al. 2009; Morgado and Vasconcelos 2017) or text descriptions (Ba
et al. 2015), are aligned (Xian et al. 2018a; Zhu, Wang, and Saligrama 2019).

Based on this motivation, most existing methods focus on boosting the
semantic-visual alignment by carefully designing the embedding space. For
example, previous methods (Shigeto et al. 2015; Zhu, Wang, and Saligrama 2019;
Min et al. 2019) constrain the embedding space to be spanned by high-
dimensional visual representations, instead of semantic labels, to improve the
space discrimination. This can prevent a few seen categories from being the
anchors for most input images (Annadani and Biswas 2018), but it also weakens
the semantic relationship in the embedding space. To this end, recent methods
(Kodirov, Xiang, and Gong 2017; Tong et al. 2019) use auto-encoders to enhance
the semantic relationship in the embedding space. For example, DSEN (Min et
al. 2019) designs domain-specific cyclic encoders between semantic embeddings
and category attributes to explore semantic domain differences, and SP-AEN
(Chen et al. 2018) uses cyclic constraints between visual representations and
images to capture detailed visual clues. Recently, attention mechanisms are
leveraged for images and semantic labels to improve representation
discrimination. In VSE (Zhu, Wang, and Saligrama 2019) and SGMA (Zhu et al.
2019), multi-head attentions are designed to infer important local regions to
boost semantic-visual alignment. LFGAA (Liu et al. 2019), and DAZLE (Huynh and
Elhamifar 2020) use attribute attention to generate clean semantic embeddings.
Although being effective, the previous methods focus on better leveraging seen
domain labels to improve semantic-visual alignment. Thus, it inevitably makes
the image representation highly task-specific and biased towards seen
categories, which limits the transferability to unseen categories.

Figure 2: The diagram of Dual-Contrastive Embedding Network. Visual
Contrastive Module learns task-independent knowledge via instance
discrimination, and Semantic Contrastive Module learns task-specific knowledge
via semantic alignment.

### Self-Supervised Learning

Without human annotations, self-supervised learning methods (Tian, Krishnan,
and Isola 2020) have proved that learning task-independent knowledge can
produce representations with strong generalization on various downstream
tasks, which even surpass fully-supervised methods. Specifically, they usually
construct some proxy tasks as supervision to learn image representations. For
example, RotNet (Gidaris, Singh, and Komodakis 2018) constrains the model to
predict the rotation angles of an input image. CFN (Noroozi and Favaro 2016)
regards an image as a jigsaw and predicts the shuffled patch order. Deep
clustering methods (Yang et al. 2018; Caron et al. 2018; Yang et al. 2020)
separate different classes via clustering. Recently, contrastive learning
(Tian, Krishnan, and Isola 2020; He et al. 2020) has attracted increasing
attention, which aims to minimize the mutual information between different
images and regards each image as an individual category. Some works extend
contrastive learning to cross-modal tasks. AVID (Morgado, Vasconcelos, and
Misra 2020) utilizes the correspondence between audio and visual
representations. CMC (Tian, Krishnan, and Isola 2020) contrasts across
multiple views, such as luminance, chrominance RGB, and depth. Inspired by the
generalized self-supervised representation, task-independent knowledge has
been introduced in many other fields, such as few-shot learning (Gidaris et
al. 2019) and landmark detection (Cheng, Su, and Maji 2020).

In this paper, we explore the task-independent knowledge from instance
discrimination to boost representation transferability in GZSL. Notably, SDGN
(Wu et al. 2020) also uses self-supervised learning in GZSL. However, it is
much different from our DCEN, because: (a) SDGN follows Transductive ZSL
setting that uses unseen domain images for training, while our DCEN only uses
data from seen domain during the training stage; (b) SDGN applies contrastive
loss to triplet data of two domains to alleviate domain confusion, while we
utilize instance discrimination knowledge of the seen domain; (c) SDGN is a
generative method that utilizes powerful GANs for unseen domain feature
generation.

## Dual-Contrastive Embedding Network

### Problem Formulation

The target of GZSL is to recognize images from either seen or unseen
categories using the model trained only with seen domain data. Here, we define
seen domain data as
$\mathcal{S}=\\{I_{i},y_{i},\boldsymbol{a}_{y_{i}}|I_{i}\in\mathcal{I}_{s},y_{i}\in\mathcal{Y}_{s},\boldsymbol{a}_{y_{i}}\in\mathcal{A}_{s}\\}$,
where $I_{i}$ is an image of a seen category, $y_{i}$ is the corresponding
category, and $\boldsymbol{a}_{y_{i}}$ is the semantic description of $y_{i}$,
such as category attributes. Unseen domain data are similarly defined as
$\mathcal{U}=\\{I_{i},y_{i},\boldsymbol{a}_{y_{i}}|I_{i}\in\mathcal{I}_{u},y_{i}\in\mathcal{Y}_{u},\boldsymbol{a}_{y_{i}}\in\mathcal{A}_{u}\\}$,
where $\mathcal{Y}_{s}\cap\mathcal{Y}_{u}=\emptyset$.

One main stream of GZSL methods is to learn a joint embedding space, where the
image representations and category descriptions can be aligned by minimizing:

$\mathcal{L}_{zsl}=\sum_{I_{i}\in\mathcal{I}_{s}}d\big{(}f(I_{i}),\boldsymbol{a}_{y_{i}}\big{)},$
(1)

where $f(\cdot)$ is the feature extractor to produce image representations.
$d(\cdot,\cdot)$ is a distance function, such as negative cosine similarity.
Since $\mathcal{A}_{s}$ and $\mathcal{A}_{u}$ usually share a common semantic
space, the semantic-visual relationship captured by $f(\cdot)$ can be
transferred to the unseen domain by:

$\hat{y}=\arg\min_{y\in\mathcal{Y}_{s}\cup\mathcal{Y}_{u}}d\big{(}f(I),\boldsymbol{a}_{y}\big{)},$
(2)

where $I\in\mathcal{I}_{s}\cup\mathcal{I}_{u}$.

However, due to unavailable $\mathcal{U}$ during training, it puts a high
demand on transferability of learned $f(\cdot)$ in Eq. (1).

### Semantic Contrastive Module

In order to better align semantic and visual embeddings, we introduce a
Semantic Contrastive Module (SCM) to better bridge the semantic-visual gap via
cross-modal contrastive learning and exploring semantic-visual
complementarity. Specifically, in the SCM, we construct semantic-visual
triplets to prevent image representations from being biased towards confusing
category and design masked attribute prediction to complete the missing
attributes from image representations.

As shown in Fig. 2, given an image $I_{i}$ and its category attribute
$\boldsymbol{a}_{y_{i}}$, SCM learns the semantic-aligned representation
$f(I_{i})$ by minimizing:

$\mathcal{L}_{sa}=\sum_{I_{i}\in\mathcal{I}_{s}}d\big{(}f(I_{i}),h(\widehat{\boldsymbol{a}}_{y_{i}})\big{)}-\min\limits_{y_{k}\neq
y_{i}}\\{d\big{(}f(I_{i}),h(\widehat{\boldsymbol{a}}_{y_{k}})\big{)}\\}.$ (3)

Different from the general objective function Eq. (1), SCM first randomly
masks some elements of $\boldsymbol{a}_{y_{i}}$, which generates
$\widehat{\boldsymbol{a}}_{y_{i}}$ to introduce some semantic variance and
improve model robustness to input noises. As attributes contain abstract
semantic information, such as wing colors, it is hard for visual feature
extractor $f(\cdot)$ to extract such highly abstract concepts from the images.
Thus, instead of directly aligning $f(I_{i})$ with
$\widehat{\boldsymbol{a}}_{y_{i}}$, SCM uses $K$ fully-connected (FC) modules,
consisting of FC, Batch normalization (BN), and ReLU, to produce high-level
semantic embeddings $h(\widehat{\boldsymbol{a}}_{y_{i}})$, which can better
bridge the semantic-visual gap. Besides, SCM constructs cross-modal triplet
$\\{f(I_{i}),h(\widehat{\boldsymbol{a}}_{y_{i}}),h(\widehat{\boldsymbol{a}}_{y_{k}})\\}$,
where $f(I_{i})$ is the anchor image representation,
$h(\widehat{\boldsymbol{a}}_{y_{i}})$ is the positive category semantic
embedding, and $h(\widehat{\boldsymbol{a}}_{y_{k}})$ is the most confusing
negative category semantic embedding. Triplet loss has been demonstrated
useful in previous works of face recognition (Schroff, Kalenichenko, and
Philbin 2015). Thus, $\mathcal{L}_{sa}$ can not only align $f(I_{i})$ with the
corresponding $h(\widehat{\boldsymbol{a}}_{y_{i}})$, but also punish the most
confusing category. Compared to $\mathcal{L}_{zsl}$, $\mathcal{L}_{sa}$ can
produce more discriminative semantic-aligned $f(I_{i})$ via cross-modal
contrastive learning.

Figure 3: Architecture for masked attribute prediction.

To better preserve semantic relationships among categories and encode the
semantic knowledge into $f(I_{i})$, masked attribute prediction
$\widehat{h}(\cdot)$ is designed by predicting the missed elements in
$\widehat{\boldsymbol{a}}_{y_{i}}$ from the image representation $f(I_{i})$.
The architecture is shown in Fig. 3. we first fuse the image representation
$f(I_{i})$ and semantic embedding $h(\widehat{\boldsymbol{a}}_{y_{i}})$ via FC
blocks and concatenation. Then, a decoder module, consisting of $K$ FC blocks,
is used to predict the intact attribute $\boldsymbol{a}^{p}_{y_{i}}$. The
depths of $h(\cdot)$ and $\widehat{h}(\cdot)$ are kept consistent for better
attribute reconstruction. Notably, since the attribute masking is random and
$h(\widehat{\boldsymbol{a}}_{y_{i}})$ has no masking knowledge, $f(I_{i})$
should capture accurate and entire semantic information from $I_{i}$ to
complete $\widehat{\boldsymbol{a}}_{y_{i}}$, thereby enhancing the semantic-
visual alignment. The loss function for masked attribute prediction is:

$\mathcal{L}_{sp}=\sum_{I_{i}\in\mathcal{I}_{s}}\left\|\widehat{h}\big{(}f(I_{i}),h(\widehat{\boldsymbol{a}}_{y_{i}})\big{)}-\boldsymbol{a}_{y_{i}}\right\|_{2}.$
(4)

With cross-modal contrastive learning and masked attribute prediction, SCM
learns better semantic alignment knowledge than naive $\mathcal{L}_{zsl}$,
which makes image representation $f(\cdot)$ better transferable to unseen
categories.

### Visual Contrastive Module

Since only $\mathcal{S}$ is available for training, the learned semantic
knowledge in $f(\cdot)$ from SCM is inevitably task-specific to the seen
categories. Thus, a visual contrastive module (VCM) is proposed to learn task-
independent knowledge without human annotations, which further improves the
transferability of $f(\cdot)$. To this end, VCM regards each image $I$ as an
individual category, to push different image representations far away from
each other. This enforces $f(\cdot)$ to capture more detailed visual clues to
represent each individual image so that they can be separated apart well.
Besides, VCM learns low-level visual invariance concepts, such as rotation and
cropping invariance, that lead to strong generalization.

Given the $i$-th image $I_{i}$ in $\mathcal{I}_{s}$, VCM first transforms
$I_{i}$ into two different counterparts $I_{i}^{v1}=\varphi_{v}^{1}(I_{i})$
and $I_{i}^{v2}=\varphi_{v}^{2}(I_{i})$. $\varphi_{v}(\cdot)$ is a composition
of random low-level transformations,  _e.g._ , cropping, rotation, blurring,
or color jittering. Then, $I_{i}^{v1}$ and $I_{i}^{v2}$ are fed into two CNN
models $f(\cdot)$ and $g(\cdot)$ to extract visual representations
$f(I_{i}^{v1})$ and $g(I_{i}^{v2})$, respectively. Finally, a visual
contrastive loss is used to attract representations of two counterparts
generated from the same image and repel representations from different images
by:

$\mathcal{L}_{id}=-\sum_{I_{i}\in\mathcal{I}_{s}}\log\frac{\exp\\{d(f(I_{i}^{v1}),g(I_{i}^{v2}))/\tau\\}}{\sum_{j}\exp\\{d(f(I_{i}^{v1}),g(I_{j}^{v2}))/\tau\\}},$
(5)

where $d(\cdot,\cdot)$ is the cosine similarity between two representations,
and $\tau$ is a temperature parameter. The positive data pair is defined as
{$I_{i}^{v1}$, $I_{i}^{v2}$} that are two views from the same image, while the
negative data pair is defined as {$I_{i}^{v1}$, $I_{j}^{v2}$} for different
images when $i\neq j$. Data from previous batches serve as negative data. The
numerator term of $\mathcal{L}_{id}$ constrains $f(\cdot)$ to be invariant to
pre-defined image transformations $\varphi_{v}(\cdot)$. By designing different
image transformations $\varphi_{v}(\cdot)$, VCM can induce different low-level
invariance concepts that are useful in GZSL tasks. The denominator term of
$\mathcal{L}_{id}$ constrains $f(\cdot)$ to be discriminative among different
images.

In $\mathcal{L}_{id}$, the weights of $g(\cdot)$ are momentum-updated (Laine
and Aila 2017) from $f(\cdot)$ by:

$W_{g}=m*W_{g}+(1-m)*W_{f},$ (6)

where $W_{g}$ and $W_{f}$ are weights of $g(\cdot)$ and $f(\cdot)$. $m$ is a
momentum parameter. The momentum-updated $g(\cdot)$ has two main advantages:
a) compared to a new trainable CNN, $g(\cdot)$ has less memory cost, and b)
compared to sharing weights of $f(\cdot)$, $g(\cdot)$ is the temporal ensemble
of different checkpoints from $f(\cdot)$ and contains temporal momentum
information.

Consequently, through instance discrimination $\mathcal{L}_{id}$, $f(\cdot)$
can gather augmented views of the same image and push different images apart.
Thus, $f(\cdot)$ can not only capture the unique information contained in each
individual image but also learn useful invariance concepts. More importantly,
this low-level visual knowledge is irrelevant to any category concept of
$\mathcal{Y}_{s}$, thereby being well-transferable to the unseen images.

### Overall Objective

Finally, the overall objective function of DCEN is :

$\mathcal{L}_{all}=\lambda_{1}\mathcal{L}_{id}+\mathcal{L}_{sa}+\lambda_{2}\mathcal{L}_{sp},$
(7)

where $\lambda_{1}$ and $\lambda_{2}$ are hyper-parameters to balance
$\mathcal{L}_{id}$ and $\mathcal{L}_{sp}$. The task-specific knowledge of
$\mathcal{L}_{sa}+\lambda_{2}\mathcal{L}_{sp}$ and task-independent knowledge
of $\lambda_{1}\mathcal{L}_{id}$ jointly make for transferable visual
representation $f(I)$.

The testing phase is viewed as a nearest neighbour searching process by:

$\hat{y}=\arg\min_{y\in\mathcal{Y}_{s}\cup\mathcal{Y}_{u}}d\big{(}f(I),h(\boldsymbol{a}_{y})\big{)},$
(8)

where $I\in\mathcal{I}_{s}\cup\mathcal{I}_{u}$ is the query image, and
$\boldsymbol{a}_{y}$ is the anchor category attribute from both the seen and
unseen domains.

## Experiments

In this section, we first evaluate each component of DCEN and then compare
DCEN with state-of-the-art GZSL methods. Please refer to the supplementary for
implementation details of DCEN.

### Experimental Settings

#### Datasets.

We adopt four widely-used GZSL benchmarks, which are Caltech-USCD
Birds-200-2011 (CUB) (Wah et al. 2011), SUN (Patterson and Hays 2012), Animals
with Attributes2 (AWA2) (Xian et al. 2018a), and Attribute Pascal and Yahoo
(aPY) (Farhadi et al. 2009) for the following experiments. Detailed statistics
of datasets are listed in Table 1. Category attributes provided by datasets
are used as semantic labels.

Dataset | Seen/Unseen | Attributes | Train | Val | Test
---|---|---|---|---|---
CUB | 150/50 | 312 | 7,057 | 1,764 | 2,967
AWA2 | 40/10 | 85 | 23,527 | 5,882 | 7,913
aPY | 20/12 | 64 | 5,932 | 1,483 | 7,924
SUN | 645/72 | 102 | 10,320 | 2,580 | 1,440

Table 1: Detailed statistics of datasets.

#### Metrics.

For GZSL, the harmonic mean $H=(2MCA_{u}\times MCA_{s})/(MCA_{u}+MCA_{s})$ is
widely used to evaluate the model performance, where $MCA_{s}$ and $MCA_{u}$
are the Mean Class Top-1 Accuracy for seen and unseen domains, respectively.

### Ablation Study

In this part, we analyze each component of DCEN on CUB and aPY datasets for
fast validation. The baseline model is denoted as Basic-ZSL, which only learns
$f(\cdot)$ via $\mathcal{L}_{ZSL}$ in Eq. (1) and is test via Eq. (2).

Figure 4: Effects of different $\lambda_{1}$ and $\lambda_{2}$ for
$\mathcal{L}_{id}$ and $\mathcal{L}_{sp}$, respectively.

#### Effects of instance discrimination.

The task-independent knowledge from instance discrimination is a major
contribution of this paper, which is learned by Visual Contrastive Module
(VCM). Thus, we first analyze how the task-independent knowledge boosts the
transferability of semantic-aligned representation in GZSL by varying
$\lambda_{1}$ to influence the visual contrastive loss $\mathcal{L}_{id}$. The
results are given in Fig. 4 (a). In this experiment, we add VCM to Basic-ZSL
with varying $\lambda_{1}$, thus $\lambda_{1}=0$ denotes no task-independent
knowledge introduced. We can see that when increasing $\lambda_{1}$ from $0$
to $0.1$ for CUB and aPY, $\mathcal{L}_{id}$ begins introducing instance
discriminative knowledge, and $H$ gradually increases. The best recognition
performance is obtained at around $\lambda_{1}=0.1$ for both CUB and aPY. This
demonstrates that the task-independent knowledge brought by instance
discrimination $\mathcal{L}_{id}$ enables DCEN to produce more transferable
representations, thereby recognizing unseen domain images more accurately.
When $\lambda_{1}$ increases from $0.1$ to $1.0$, the performance $H$ drops.
Specifically, for CUB, the result with $\lambda_{1}=1$ is even worse than that
with $\lambda_{1}=0$. This reveals that, when $\lambda_{1}$ is large, the
instance discrimination knowledge $\mathcal{L}_{id}$, which targets pushing
all representations far away, becomes too sensitive to intra-class variance of
different images and hard to preserve the semantic relationship, thereby being
unable to cluster images correctly. Thus, a trade-off between task-independent
knowledge and task-specific knowledge is critical to make for strong
transferable representations in GZSL. We find $\lambda_{1}=0.1$ is suitable
for most cases, which is used for the following experiments.

Aug. | Crop | Flip | Gray | CJ | Blur | Rotation | Swap | H
---|---|---|---|---|---|---|---|---
prob. | 1.0 | 0.5 | 0.2 | 0.8 | 0.5 | 0.5 | 0.2
|  |  |  |  |  |  |  | 65.3
$\mathcal{L}_{id}$ | ✓ |  |  |  |  |  |  | 65.8$\uparrow$
✓ | ✓ |  |  |  |  |  | 66.0$\uparrow$
✓ | ✓ | ✓ |  |  |  |  | 65.5$\downarrow$
✓ | ✓ |  |  ✓(v1) |  |  |  | 63.8$\downarrow$
✓ | ✓ |  |  ✓(v2) |  |  |  | 65.3$\downarrow$
✓ | ✓ |  |  ✓(v3) |  |  |  | 65.2$\downarrow$
✓ | ✓ |  |  | ✓ |  |  | 67.1$\uparrow$
✓ | ✓ |  |  |  |  ✓(90) |  | 64.4$\downarrow$
✓ | ✓ |  |  |  |  ✓(60) |  | 64.7$\downarrow$
✓ | ✓ |  |  |  |  ✓(30) |  | 66.5$\uparrow$
✓ | ✓ |  |  |  |  |  ✓(7) | 64.2$\downarrow$
✓ | ✓ |  |  |  |  |  ✓(5) | 65.8$\downarrow$
✓ | ✓ |  |  |  |  |  ✓(3) | 66.7$\uparrow$
✓ | ✓ |  |  | ✓ |  ✓(30) |  | 67.9$\uparrow$
✓ | ✓ |  |  | ✓ |  ✓(30) |  ✓(3) | 68.5$\uparrow$

Table 2: Evaluating different visual augmentations on CUB by successively
adding operations. When a certain operation brings positive effects, it is
retained, otherwise, it is removed. The operation probability is
experimentally determined. For v1, v2, and v3 color jittering, the parameters
are respectively (0.4, 0.4, 0.4, 0.4), (0.4, 0.4, 0.4, 0.1), and (0.8, 0.8,
0.8, 0.2) for brightness, contrast, saturation, and hue. For rotation and
swap, the parameters in parentheses indicate rotation angle and jigsaw number,
respectively.

#### Effects of different invariance concepts.

In the task-independent knowledge brought by instance discrimination, the low-
level invariance concepts, induced by visual augmentations, are important. In
this part, we conduct extensive experiments to explore useful invariance
concepts in GZSL by combining different augmentations as $\varphi_{v}(\cdot)$
in VCM on CUB. The experiments are carried out based on Basic-ZSL with
$\lambda_{1}=0.1$. The results are listed in Table 2, where we explore seven
widely-used low-level augmentations, which contain random cropping, horizontal
flipping, grayscale, color jittering, Gaussian blurring, rotation, and
swapping. Specifically, the swapping augmentation (Chen et al. 2019) regards
an image as a jigsaw and randomly shuffles the patches, which requires the
representation to be invariant to patch-swapped images. From the results,
adding cropping, horizontal flipping, Gaussian blurring, rotation, and swap
augmentations into $\varphi_{v}(\cdot)$ can benefit the performance, while
adding grayscale and color jittering harms the results. This is because that
the grayscale and color augmentations will change the color space severely,
which may prevent the model from inferring correct color attributes,  _e.g.,_
wing color, to recognize unseen categories. Among the useful augmentations,
Gaussian blurring brings the largest improvement because it helps the model
neglect high-frequent noises. For random rotation and swapping, too large
angles and jigsaw numbers bring negative effects, which may destroy the image
contents severely. Finally, we combine all positive low-level augmentations in
Table 2 as $\varphi_{v}(\cdot)$, which obtains an impressive improvement of
$3.2\%$ over the baseline. The useful low-level invariance concepts explored
in this experiment may provide an insight for the following GZSL studies.

Figure 5: Visualization of representation on CUB and aPY. Dotted circles
denote unseen domain representations.

#### Visualization of representation.

To intuitively analyze the effect of task-independent knowledge from instance
discrimination, we visualize the image representations before and after adding
VCM using t-SNE (Van der Maaten and Hinton 2008) on randomly selected
categories. The results are given in Fig. 5. After introducing task-
independent knowledge into image representations, the category margin is
obviously enlarged between seen and unseen categories, and the distribution of
each individual cluster becomes more compact. This means that image
representations are more distinguishable between categories of the two
domains, thus the unseen image representation is less biased towards the seen
categories. In other words, the visual representation can be better
transferred to represent unseen image characteristics. Besides, Table 3 gives
quantitative results after adding VCM. It shows that VCM can bring an
impressive gain of $5\%$ on $H$ and $6.1\%$ on $MCA_{u}$, which demonstrates
that VCM helps the model better transfer to the unseen domain. In summary,
both quantitative and qualitative results prove that the task-independent
knowledge from instance discrimination can improve the representation
transferability in GZSL.

Methods | MCAu | MCAs | $H$
---|---|---|---
Basic-ZSL | 56.3 | 72.8 | 63.5
Basic-ZSL with VCM | 62.4 | 75.9 | 68.5
+attribute masking | 62.5 | 78.3 | 69.5
+cross-modal triplet | 63.5 | 77.7 | 69.9
+$\mathcal{L}_{sp}$ | 63.8 | 78.4 | 70.4
Table 3: Effects of each component of DCEN on CUB. Figure 6: Effects of
different $\sigma$ and $K$.

| Methods | CUB | AWA2 | aPY | SUN
---|---|---|---|---|---
| MCAu | MCAs | $H$ | MCAu | MCAs | $H$ | MCAu | MCAs | $H$ | MCAu | MCAs | $H$
GEN | FGN(Xian et al. 2018b) | 43.7 | 57.7 | 49.7 | - | - | - | - | - | - | 42.6 | 36.6 | 39.4
SABR-I(Paul, Krishnan, and Munjal 2019) | 55.0 | 58.7 | 56.8 | 30.3 | 93.9 | 46.9 | - | - | - | 50.7 | 35.1 | 41.5
f-VAEGAN-D2(Xian et al. 2019) | 63.2 | 75.6 | 68.9 | - | - | - | - | - | - | 50.1 | 37.8 | 43.1
RFF-GZSL(Han, Fu, and Yang 2020) | 59.8 | 79.9 | 68.4 | - | - | - | - | - | - | 58.8 | 45.3 | 51.2
OCD-CVAE(Keshari, Singh, and Vatsa 2020) | 44.8 | 59.9 | 51.3 | 59.5 | 73.4 | 65.7 | - | - | - | 44.8 | 42.9 | 43.8
IZF-Softmax(Shen, Qin, and Huang 2020) | 52.7 | 68.0 | 59.4 | 60.6 | 77.5 | 68.0 | 42.3 | 60.5 | 49.8 | 52.7 | 57.0 | 54.8
TF-VAEGAN(Narayan et al. 2020) | 63.8 | 79.3 | 70.7 | - | - | - | - | - | - | 62.4 | 47.1 | 53.7
NON-GEN | SYNC(Changpinyo et al. 2016) | 11.5 | 70.9 | 19.8 | 10.0 | 90.5 | 18.0 | 7.4 | 66.3 | 13.3 | 7.9 | 43.3 | 13.4
CDL(Jiang et al. 2018) | 23.5 | 55.2 | 32.9 | - | - | - | 19.8 | 48.6 | 28.1 | 21.5 | 34.7 | 26.5
PSR-ZSL(Annadani and Biswas 2018) | 24.6 | 54.3 | 33.9 | 20.7 | 73.8 | 32.2 | 13.5 | 51.4 | 21.4 | 20.8 | 37.2 | 26.7
SP-AEN(Chen et al. 2018) | 34.7 | 70.6 | 46.6 | 23.3 | 90.9 | 37.1 | 13.7 | 63.4 | 22.6 | 24.9 | 38.6 | 30.3
DLFZRL(Tong et al. 2019) | - | - | 37.1 | - | - | 45.1 | - | - | 31.0 | - | - | 24.6
MLSE(Ding and Liu 2019) | 22.3 | 71.6 | 34.0 | 23.8 | 83.2 | 37.0 | 12.7 | 74.3 | 21.7 | 20.7 | 36.4 | 26.4
TripletLoss(Cacheux, Borgne, and Crucianu 2019) | 55.8 | 52.3 | 53.0 | 48.5 | 83.2 | 61.3 | - | - | - | 47.9 | 30.4 | 36.8
COSMO(Atzmon and Chechik 2019) | 44.4 | 57.8 | 50.2 | - | - | - | - | - | - | 44.9 | 37.7 | 41.0
PREN(Ye and Guo 2019) | 32.5 | 55.8 | 43.1 | 32.4 | 88.6 | 47.4 | - | - | - | 35.4 | 27.2 | 30.8
VSE-S(Zhu, Wang, and Saligrama 2019) | 33.4 | 87.5 | 48.4 | 41.6 | 91.3 | 57.2 | 24.5 | 72.0 | 36.6 | - | - | -
AREN(Xie et al. 2019) | 63.2 | 69.0 | 66.0 | 54.7 | 79.1 | 64.7 | 30.0 | 47.9 | 36.9 | 40.3 | 32.3 | 35.9
CosineSoftmax(Kampffmeyer et al. 2019) | 47.4 | 47.6 | 47.5 | 56.4 | 81.4 | 66.7 | 26.5 | 74.0 | 39.0 | 36.3 | 42.8 | 39.3
DAZLE(Huynh and Elhamifar 2020) | 56.7 | 59.6 | 58.1 | 60.3 | 75.7 | 67.1 | - | - | - | 52.3 | 24.3 | 33.2
| DCEN | 63.8 | 78.4 | 70.4 | 62.4 | 81.7 | 70.8 | 37.5 | 61.6 | 46.7 | 43.7 | 39.8 | 41.7

Table 4: Results of GZSL on four classification benchmarks. Generative methods
(GEN) utilize extra synthetic unseen domain data for training. The best result
is bolded, and the second best is underlined.

#### Analysis of semantic contrastive module.

In this part, we analyze each component of SCM, and the results are given in
Table 3. Based on Basic-ZSL with VCM, we add random attribute masking, cross-
modal triplet loss, and masked attribute prediction sequentially. First, we
evaluate the effect of masking $\sigma\%$ attributes in SCM, which can improve
model robustness to semantic noises. In training, attributes are randomly
chosen with $p=0.5$, and $\sigma\%$ elements of these chosen attributes are
masked. Random masking is not used in testing. Fig. 6 (a) shows the effect of
varying $\sigma$ on CUB and aPY. When $\sigma$ is too large which means many
elements of attributes are missed, the semantic alignment becomes weak,
thereby the performance drops. Notably, the reason for small optimal $\sigma$
on aPY is that there are much fewer attributes in aPY than in CUB. Thus,
suitable semantic masking is a simple way to improve semantic-visual
robustness. Then, adding cross-modal triplet loss $\mathcal{L}_{sa}$ brings
about $0.4\%$ improvement on CUB in Table 3. Specifically, $\mathcal{L}_{sa}$
contrasts cross-modal triplets to make visual representation not only close to
corresponding semantic anchors but also far away from confusing negative
categories, which improves representation discrimination. Finally, we evaluate
the effect of masked attribute prediction by varying $\lambda_{2}$ of
$\mathcal{L}_{sp}$, and the results are given in Fig. 4 (b). We can see that
setting appropriate weights of $\lambda_{2}$ for $\mathcal{L}_{sp}$ can boost
the final results. It proves that, by constraining the visual representation
to predict the missed attributes, more semantic information from category
attributes is explicitly encoded and preserved in visual representation. In
summary, our SCM can generate better semantic-aligned representation via
cross-modal triplet loss and masked attribute prediction.

#### Analysis for other hyper-parameters.

In DCEN, $K$, $\tau$, and $m$ are three minor hyper-parameters. $K$ determines
the architecture $h(\cdot)$ in Eq. (3). Here, we evaluate the effects of
different $K$ in Fig. 6 (b). It can be seen that $K=2$ is suitable for most
cases. $\tau$ and $m$ control the cosine similarity scaling in Eq. (5) and
momentum updating of $g(\cdot)$ in Eq. (6). In this paper, we set $\tau=0.07$
and $m=0.999$, which are commonly used in contrastive learning (He et al.
2020).

### Comparison with State-of-the-Art Methods

Finally, we compare our DCEN with the state-of-the-art methods on four
datasets,  _i.e.,_ CUB, AWA2, aPY, and SUN. Results are given in Table 4,
which shows that DCEN surpasses existing methods on four datasets by a large
margin.

First, we compare DCEN with related embedding-based methods, which use only
seen domain data for training and no generative models. DCEN surpasses the
best related method by respectively $4.4\%$, $3.7\%$, $7.7\%$, $0.7\%$ on CUB,
AWA2, aPY, and SUN datasets in the term of $H$, demonstrating its good
generalizability to different image domains. Notably, among non-generative
methods, AREN (Xie et al. 2019) obtains similar performance to DCEN, but it
utilizes a model ensemble of two separate recognition branches. Our
improvements are impressive, because DCEN introduces extra task-independent
knowledge into image representations via instance discrimination learning, and
it is proved that appropriate task-independent knowledge can significantly
improve representation transferability in GZSL.

Then, we compare DCEN with recent generative methods. Notably, the generative
methods utilize prior unseen domain semantics to synthesize extra unseen
visual data for training,  _e.g.,_ powerful GANs, while DCEN only uses the
seen domain data. From Table 4, DCEN outperforms most generative methods by a
large margin, which is encouraging for embedding-based methods because no
synthesized unseen domain data is used for training. Compared to recent TF-
VAEGAN (Narayan et al. 2020), DCEN obtains comparable results on CUB using
only seen domain data. This is due to introducing task-independent knowledge
from instance discrimination, and it proves that the embedding-based methods
have much potential.

Finally, we can conclude that: a) task-independent knowledge and task-specific
semantic knowledge jointly make for strong transferable representations in
GZSL, which enables DCEN to surpass all related works; and b) task-independent
knowledge from instance discrimination is useful, and other task-independent
knowledge remains to be explored.

### Discussion

Compared to previous embedding-based methods, the improvement of DCEN on SUN
is much less than the other three benchmarks. The reason is that the number of
images in each category of SUN is small,  _e.g.,_ averaged 16 images per
category. Thus, learning instance discrimination has a similar effect on image
representation, compared to category discrimination. This also reveals a
limitation of DCEN,  _i.e.,_ the task-independent knowledge from instance
discrimination may be not suitable for tail categories with inadequate
samples.

## Conclusion

In this paper, we propose a novel Dual-Contrastive Embedding Network (DCEN)
that utilizes task-independent and task-specific knowledge to jointly make for
transferable representation in GZSL. Specifically, a semantic contrastive
module is developed to learn task-specific knowledge by performing cross-modal
contrastive learning and exploring semantic-visual complementarity with
category labels. Besides, a visual contrastive module is designed to learn
annotation-free task-independent knowledge via instance discrimination
supervision, which gathers representations of the same image and pushes
different image representations apart. Compared to seen category knowledge,
the task-independent knowledge from instance discrimination is less biased,
which can improve the representation transferability to unseen categories.
Extensive experiments show that our DCEN achieves superior performance on four
GZSL benchmarks.

In the future, the effects of more task-independent knowledge, such as
rotation angle prediction and jigsaw order prediction, will be explored for
GZSL.

## Acknowledgements

This work was supported by the National Key R&D Program of China with grant
No. 2020AAA0108602, National Natural Science Foundation of China (NSFC) under
Grants 61632006 and 62076230, and Fundamental Research Funds for the Central
Universities under Grants WK3490000003.

## Ethics Statement

All datasets used in this paper are publicly available and have no relations
with personal privacy information.

## References

  * Annadani and Biswas (2018) Annadani, Y.; and Biswas, S. 2018.  Preserving Semantic Relations for Zero-Shot Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 7603–7612.
  * Atzmon and Chechik (2019) Atzmon, Y.; and Chechik, G. 2019.  Adaptive Confidence Smoothing for Generalized Zero-Shot Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 11671–11680.
  * Ba et al. (2015) Ba, J. L.; Swersky, K.; Fidler, S.; and salakhutdinov, R. 2015.  Predicting Deep Zero-Shot Convolutional Neural Networks Using Textual Descriptions.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 4247–4255.
  * Badrinarayanan, Kendall, and Cipolla (2017) Badrinarayanan, V.; Kendall, A.; and Cipolla, R. 2017.  Segnet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 39(12): 2481–2495.
  * Cacheux, Borgne, and Crucianu (2019) Cacheux, Y. L.; Borgne, H. L.; and Crucianu, M. 2019.  Modeling Inter and Intra-Class Relations in the Triplet Loss for Zero-Shot Learning.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 10333–10342.
  * Caron et al. (2018) Caron, M.; Bojanowski, P.; Joulin, A.; and Douze, M. 2018.  Deep Clustering for Unsupervised Learning of Visual Features.  In _Proceedings of the European Conference on Computer Vision_ , 132–149.
  * Changpinyo et al. (2016) Changpinyo, S.; Chao, W.-L.; Gong, B.; and Sha, F. 2016.  Synthesized Classifiers for Zero-Shot Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 5327–5336.
  * Chen et al. (2018) Chen, L.; Zhang, H.; Xiao, J.; Liu, W.; and Chang, S.-F. 2018.  Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Network.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 1043–1052.
  * Chen et al. (2019) Chen, Y.; Bai, Y.; Zhang, W.; and Mei, T. 2019.  Destruction and Construction Learning for Fine-Grained Image Recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 5157–5166.
  * Cheng, Su, and Maji (2020) Cheng, Z.; Su, J.-C.; and Maji, S. 2020.  Unsupervised Discovery of Object Landmarks via Contrastive Learning.  _arXiv preprint arXiv:2006.14787_ .
  * Ding and Liu (2019) Ding, Z.; and Liu, H. 2019.  Marginalized Latent Semantic Encoder for Zero-Shot Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 6191–6199.
  * Farhadi et al. (2009) Farhadi, A.; Endres, I.; Hoiem, D.; and Forsyth, D. 2009.  Describing Objects by Their Attributes.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 1778–1785.
  * Felix et al. (2018) Felix, R.; Kumar, V. B.; Reid, I.; and Carneiro, G. 2018.  Multi-Modal Cycle-Consistent Generalized Zero-Shot Learning.  In _Proceedings of the European Conference on Computer Vision_ , 21–37.
  * Gidaris et al. (2019) Gidaris, S.; Bursuc, A.; Komodakis, N.; Pérez, P.; and Cord, M. 2019.  Boosting Few-Shot Visual Learning with Self-Supervision.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 8059–8068.
  * Gidaris, Singh, and Komodakis (2018) Gidaris, S.; Singh, P.; and Komodakis, N. 2018.  Unsupervised Representation Learning by Predicting Image Rotations.  In _International Conference on Learning Representations_ , 1–16.
  * Girshick et al. (2014) Girshick, R.; Donahue, J.; Darrell, T.; and Malik, J. 2014.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 580–587.
  * Han, Fu, and Yang (2020) Han, Z.; Fu, Z.; and Yang, J. 2020.  Learning the Redundancy-Free Features for Generalized Zero-Shot Object Recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 12865–12874.
  * He et al. (2020) He, K.; Fan, H.; Wu, Y.; Xie, S.; and Girshick, R. 2020.  Momentum Contrast for Unsupervised Visual Representation Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 9729–9738.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep Residual Learning for Image Recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 770–778.
  * Huynh and Elhamifar (2020) Huynh, D.; and Elhamifar, E. 2020.  Fine-Grained Generalized Zero-Shot Learning via Dense Attribute-Based Attention.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 4483–4493.
  * Jiang et al. (2018) Jiang, H.; Wang, R.; Shan, S.; and Chen, X. 2018.  Learning Class Prototypes via Structure Alignment for Zero-Shot Recognition.  In _Proceedings of the European Conference on Computer Vision_ , 118–134.
  * Kampffmeyer et al. (2019) Kampffmeyer, M.; Chen, Y.; Liang, X.; Wang, H.; Zhang, Y.; and Xing, E. P. 2019\.  Rethinking Knowledge Graph Propagation for Zero-Shot Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 11487–11496.
  * Keshari, Singh, and Vatsa (2020) Keshari, R.; Singh, R.; and Vatsa, M. 2020.  Generalized Zero-Shot Learning via Over-Complete Distribution.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 13300–13308.
  * Kodirov, Xiang, and Gong (2017) Kodirov, E.; Xiang, T.; and Gong, S. 2017.  Semantic Autoencoder for Zero-Shot Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 3174–3183.
  * Laine and Aila (2017) Laine, S.; and Aila, T. 2017.  Temporal Ensembling for Semi-supervised Learning.  In _International Conference on Learning Representations_ , 1–13.
  * Liu et al. (2019) Liu, Y.; Guo, J.; Cai, D.; and He, X. 2019.  Attribute Attention for Semantic Disambiguation in Zero-Shot Learning.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 6698–6707.
  * Min et al. (2019) Min, S.; Yao, H.; Xie, H.; Zha, Z.-J.; and Zhang, Y. 2019.  Domain-Specific Embedding Network for Zero-Shot Recognition.  In _Proceedings of ACM International Conference on Multimedia_ , 2070–2078.
  * Min et al. (2020) Min, S.; Yao, H.; Xie, H.; Zha, Z.-J.; and Zhang, Y. 2020.  Domain-Oriented Semantic Embedding for Zero-Shot Learning.  _IEEE Transactions on Multimedia_ .
  * Morgado and Vasconcelos (2017) Morgado, P.; and Vasconcelos, N. 2017.  Semantically Consistent Regularization for Zero-Shot Recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 6060–6069.
  * Morgado, Vasconcelos, and Misra (2020) Morgado, P.; Vasconcelos, N.; and Misra, I. 2020.  Audio-Visual Instance Discrimination with Cross-Modal Agreement.  _arXiv preprint arXiv:2004.12943_ .
  * Narayan et al. (2020) Narayan, S.; Gupta, A.; Khan, F. S.; Snoek, C. G.; and Shao, L. 2020.  Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification.  In _Proceedings of the European Conference on Computer Vision_ , 1–23.
  * Noroozi and Favaro (2016) Noroozi, M.; and Favaro, P. 2016.  Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles.  In _Proceedings of the European Conference on Computer Vision_ , 69–84.
  * Patterson and Hays (2012) Patterson, G.; and Hays, J. 2012.  Sun Attribute Database: Discovering, Annotating, and Recognizing Scene Attributes.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2751–2758.
  * Paul, Krishnan, and Munjal (2019) Paul, A.; Krishnan, N. C.; and Munjal, P. 2019.  Semantically Aligned Bias Reducing Zero Shot Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 7056–7065.
  * Schroff, Kalenichenko, and Philbin (2015) Schroff, F.; Kalenichenko, D.; and Philbin, J. 2015.  FaceNet: A Unified Embedding for Face Recognition and Clustering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 815–823.
  * Shen, Qin, and Huang (2020) Shen, Y.; Qin, J.; and Huang, L. 2020.  Invertible Zero-Shot Recognition Flows.  In _Proceedings of the European Conference on Computer Vision_ , 614–631.
  * Shigeto et al. (2015) Shigeto, Y.; Suzuki, I.; Hara, K.; Shimbo, M.; and Matsumoto, Y. 2015.  Ridge Regression, Hubness, and Zero-Shot Learning.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , 135–151.
  * Tian, Krishnan, and Isola (2020) Tian, Y.; Krishnan, D.; and Isola, P. 2020.  Contrastive Multiview Coding.  In _Proceedings of the European Conference on Computer Vision_ , 776–794.
  * Tong et al. (2019) Tong, B.; Wang, C.; Klinkigt, M.; Kobayashi, Y.; and Nonaka, Y. 2019.  Hierarchical Disentanglement of Discriminative Latent Features for Zero-Shot Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 11467–11476.
  * Van der Maaten and Hinton (2008) Van der Maaten, L.; and Hinton, G. 2008.  Visualizing data using t-SNE.  _Journal of Machine Learning Research_ 9(11): 2579–2605.
  * Wah et al. (2011) Wah, C.; Branson, S.; Welinder, P.; Perona, P.; and Belongie, S. 2011.  The Caltech-UCSD Birds-200-2011 Dataset.  In _California Institute of Technology_ , 1–8.
  * Wu et al. (2020) Wu, J.; Zhang, T.; Zha, Z.-J.; Luo, J.; Zhang, Y.; and Wu, F. 2020.  Self-Supervised Domain-Aware Generative Network for Generalized Zero-Shot Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 12767–12776.
  * Xian et al. (2016) Xian, Y.; Akata, Z.; Sharma, G.; Nguyen, Q.; Hein, M.; and Schiele, B. 2016.  Latent Embeddings for Zero-Shot Classification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 69–77.
  * Xian et al. (2018a) Xian, Y.; Lampert, C. H.; Schiele, B.; and Akata, Z. 2018a.  Zero-shot learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 41(9): 2251–2265.
  * Xian et al. (2018b) Xian, Y.; Lorenz, T.; Schiele, B.; and Akata, Z. 2018b.  Feature Generating Networks for Zero-Shot Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 5542–5551.
  * Xian et al. (2019) Xian, Y.; Sharma, S.; Schiele, B.; and Akata, Z. 2019.  F-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 10275–10284.
  * Xie et al. (2019) Xie, G.-S.; Liu, L.; Jin, X.; Zhu, F.; Zhang, Z.; Qin, J.; Yao, Y.; and Shao, L. 2019.  Attentive Region Embedding Network for Zero-shot Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 9384–9393.
  * Yang et al. (2018) Yang, X.; Deng, C.; Liu, X.; and Nie, F. 2018.  New l2, 1-norm Relaxation of Multi-Way Graph Cut for Clustering.  In _AAAI Conference on Artificial Intelligence_ , 1–8.
  * Yang et al. (2020) Yang, X.; Deng, C.; Wei, K.; Yan, J.; and Liu, W. 2020.  Adversarial Learning for Robust Deep Clustering.  In _Advances in Neural Information Processing Systems_ , 9098–9108.
  * Ye and Guo (2019) Ye, M.; and Guo, Y. 2019.  Progressive Ensemble Networks for Zero-Shot Recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 11728–11736.
  * Zhu, Wang, and Saligrama (2019) Zhu, P.; Wang, H.; and Saligrama, V. 2019.  Generalized Zero-Shot Recognition based on Visually Semantic Embedding.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2995–3003.
  * Zhu et al. (2019) Zhu, Y.; Xie, J.; Tang, Z.; Peng, X.; and Elgammal, A. 2019.  Semantic-Guided Multi-Attention Localization for Zero-Shot Learning.  In _Advances in Neural Information Processing Systems_ , 1–11.


#  Drone delivery: Reliable Cellular UAV Communication Using Multi-Operator
Diversity††thanks: This research is supported by the Research Foundation
Flanders (FWO), projects no. S003817N (OmniDrone) and G098020N.

Achiel Colpaert, Michaël Raes, Evgenii Vinogradov, and Sofie Pollin
<EMAIL_ADDRESS>KU Leuven, ESAT - Department of Electrical
Engineering, Kasteelpark Arenberg 10, 3001 Heverlee, Belgium
E-mail: {achiel.colpaert, evgenii.vinogradov<EMAIL_ADDRESS>
###### Abstract

The market size of Unmanned Aerial Vehicles (UAVs, a.k.a drones) can reach up
to 10% of the global market value. In particular, drone delivery is one of the
most attractive applications. The growing number of drones requires
appropriate traffic management systems that will rely on cellular networks.
However, it has been shown in the literature that these networks cannot
provide reliable communication due to low coverage probability and frequent
handovers. This article presents a potential solution targeting these problems
while requiring no modifications of the existing infrastructure. Namely,
equipping the UAV with multiple cellular modems to connect to different
providers’ networks introduces network diversity resulting in 98% coverage
probability at the flight altitude of 100 meters. In contrast, one network
ensures only 80% coverage. At the same time, the size of the outage zones
becomes up to ten times smaller and the frequency of harmful handovers is
reduced to zero. The results are obtained with a physical-layer simulator
utilizing a real urban 3D environment, cellular network parameters (e.g., site
locations, antenna orientation and gains), and specific aerial channel models.

###### Index Terms:

UAV, UTM, Urban Air Mobility, Reliability, Cellular networks

##  I Introduction

In May 2021, Morgan Stanley released a forecast [1] stating that by 2050 the
total market of Urban Air Mobility (UAM, including drone delivery, air taxi,
patrolling drones, to name a few) will reach up to 11% of the projected global
Gross Domestic Product (GDP). The same document claims that the large-scale
deployments of drone-based urban delivery are expected to be in place by 2030.

Apart from the mobility, Unmanned Aerial Vehicles and Systems (UAVs, UASs) are
expected to play an essential role in future 6G networks where aerial Base
Stations (BSs) will provide connectivity to ground users [2, 3, 4]. Shorter-
term predictions say that by 2025, the drone services market will be worth a
total of USD 63.6 billion [5] and the UAV fleet, both recreational and
commercial, is projected to reach 2 to 3 million by 2023 [6].

Managing such a large fleet requires the design of UAS Traffic Management
(UTM) solutions in order to ensure expected levels of safety, security,
operation transparency, and airspace usage efficiency [7]. The UTM framework
formulated by International Civil Aviation Organization (ICAO) in [8] includes
cellular technology as a critical enabler of large-scale drone deployments.
Several competing UTM implementations have been proposed [9]. All of them rely
on the existence of a reliable command and control (C2) link. Indeed, cellular
networks are an ideal candidate as they provide ubiquitous coverage and remove
the need for the UTM provider to deploy expensive dedicated wireless
infrastructure. 3GPP has taken the first steps to introduce aerial vehicles
some time ago now [10], but many questions remain open [2, 4].

###  I-A State of the Art: overview of open problems

In an effort to evaluate the performance of UAVs in cellular networks many
different types of studies have been performed, from analysis to simulator
based, to actual experimental work. Several relevant overviews can be found in
[2, 3]. In this article, let us provide just a few of the most relevant works.

An in-depth analysis of the performance of UAVs in a cellular network based on
stochastic geometry provided in [11]. Authors of [12] and [13] characterized
the UAV channels through dedicated measurement campaigns. All works listed
above conclude that the antenna configuration at BSs should be considered when
introducing aerial users to a cellular network as the antennas are pointed
towards the ground generally.

Other simulation-based research has proven that current LTE networks do not
provide sufficient coverage at altitudes above building height, mainly due to
interference problems [14] and high handover rates [15]. Several measurement
campaigns [16, 17] showed satisfying results both in terms of coverage and
handover rates. However, they were performed in rural areas where the BS
density is much lower compared to an urban scenario.

Takeaways: A high density of base stations results in high signal strengths
but simultaneously high interference levels due to line-of-sight conditions.
Moreover, the BS antennas are downtilted to optimize the ground coverage.
Consequently, the problem of achieving a highly reliable connection with UAVs
in an urban environment remains open due to i) limited coverage caused by high
interference (growing with increasing flight altitude); ii) frequent
handovers.

###  I-B Multi-Operator Diversity as a Potential Solution

This article suggests a solution to increase the reliability of UAV
communication links while requiring no modifications of the terrestrial
cellular networks. Several studies have shown through network level
measurements that equipping a UAV with multiple LTE modems to connect to
different providers’ networks will improve reliability and Quality of Service
[18, 19, 20]. Thus, we can assume that introducing network diversity improves
the coverage and handover rates experienced by UAVs. Indeed, network
infrastructure from different providers is deployed at different base station
sites in the city111Even when the sites are shared (for instance the same mast
can be used), the sector antennas are usually pointed differently. This is due
to the need to optimize the performance under different underlying network
topology.. This feature reduces the probability of having a bad connection to
all BSs at a given location of UAV.

The price to pay is a slightly higher payload weight since the modern modems
are quite compact. The size and weight can be reduced even further if a
dedicated multi-connection module is designed. Another factor is an increased
power consumption. However, a more stable connection results in less frequent
modifications of the flight path [21, 22]. Note that power consumption of the
communication modems is negligible in comparison with the propulsion energy
[21].

###  I-C Contributions

The main contributions of this work are:

  1. 1.

We designed a realistic simulator taking into account

     * •

real 3D environment including information about i) ground surface, ii)
buildings and infrastructure;

     * •

real cellular network parameters (e.g., BS locations, sector orientations,
used power, etc.) reported by the Belgian operators to the government;

     * •

3D antenna patterns (with adaptable sidelobes);

     * •

specific UAV channel models;

     * •

users’ mobility (UAVs).

  2. 2.

We assessed potential effects of multi-operator diversity on i) coverage and
ii) handovers.

Note that in this work, we focus on a promising use case of drone delivery.
UAV delivery will probably be performed at higher altitudes than other popular
operations (e.g., patrolling) due to safety reasons. Though the higher
altitude can result in more severe damage in case of malfunction, it gives a
better time margin for the safety systems to act. For this reason we use
altitudes higher than foreseen in [8]. Of course, the presented results are
useful for other UAV applications when only the appropriate altitudes are
analyzed.

##  II Model

To model the scenario of drone deliveries, we consider drones flying in an
urban environment at a fixed speed. They travel at constant altitudes ranging
from just above rooftop level up to a maximum of 300 m above ground level.
They fly in a straight line starting at a random point $A$ in the city to
another random point $B$ representing the delivery address. The requirements
for a network supporting these drones are twofold. On one side, only a low
throughput link is necessary to monitor and control the UAV from a remote
location. The requirement for this is a minimum Signal-to-Noise-Ratio (SINR).
On the other hand, the link needs to be very reliable. Due to safety concerns,
one cannot afford to lose connection to the UAV. The amount of handovers (HO)
between sectors and radio link failures (RLF) will affect the time that the
UAV is not connected to the network.

For the location of our study, we choose the city of Leuven as the environment
as this is the location of our university, and this allows us to verify our
results even further in the future.

We will evaluate the performance of the network using several metrics based on
the SINR which is calculated at any 3D point $\vec{p}$ in the simulator as
follows:

$SINR(\vec{p})=\frac{P_{r}(\vec{p},a)}{\sum^{A}_{i\neq a}P_{r,i}(\vec{p})+N},$
(1)

where $A$ represents the set of all sectors in the environment and $a$
represents the currently assigned sector, $N$ represents the noise power and
is calculated as follows $N=BN_{0}F$, where $B$ is the bandwidth, $N_{0}$ is
the noise density and $F$ is the noise figure.

The metrics considered are the following: the first being coverage probability
$P_{cov}$ calculated as the ratio of the area where the Signal-to-
Interference-and-Noise-Ratio (SINR) is larger than a threshold (T) based on
throughput requirements, $SINR>T$, divided by the total area. The second
metric is the size of the largest continuous zone where no coverage exists in
square kilometers, called the maximum outage zone $OUT_{max}$. This metric
represents the importance of continuous coverage, large regions of outage
results in the UAV being disconnected for long periods of time which is
detrimental in a reliable network.

##  III Simulations

To simulate the received powers at any 3D point, an improved version of the
physical layer simulator developed in [14, 15] is used. It uses a real 3D
environment based on surface scans to create a virtual environment where
assets like base stations and users can be deployed in a 3D space. The
modified version includes realistic cellular network configurations as we
detail in the following subsection.

###  III-A Environment

The simulator requires several parameters based on the real world environment
as input. First of all, terrain and surface information is required, as this
affects the heights of the antennas and, more importantly, allows us to
determine if a sector is in line of sight with a point. For this, the DHMV-
dataset, see [14], is used and the considered environment is displayed in Fig.
1.

Figure 1: Digital Surface Model with sectors and simulation area indicated in
red. The lines represent the sectors and their azimuth rotation and each color
represents a different network.

Secondly, information about the location of the different base stations, their
sectors, tilts, transmit powers and antennas needs to be known. For this,
information from the Belgian Antenna Registry of the BIPT [23], the
organisation managing the spectrum in this environment, is combined with the
database of _declarations of conformity_ issued by operators for each antenna
site. These declaration documents contain all information about the hardware
deployed at each antenna site and are used as input for the simulator.

Thirdly, we used a 3D antenna pattern corresponding to the antennas used on
the sites in the evaluated area [24]. It is important to note that just like
the real antenna, this antenna pattern has sidelobe suppression implemented
meaning that the actual gain of the sidelobes is further reduces to improve
power efficiency by reducing energy emitted towards the sky.

Lastly, the used channel model is defined in [10] specifically for aerial
vehicles.

###  III-B Scenarios

Two different scenarios are simulated. One static scenario where all points on
the map are evaluated at different altitudes, generating statistics of the
area at different heights. Secondly, a mobile scenario is evaluated where a
UAV travels a random path over the city populated by static ground users. In
this scenario, we can evaluate handover behaviour and Radio-Link-Failure (RLF)
occurrences of the network.

####  III-B1 Static scenario

The first scenario’s goal is to evaluate the statistics of the chosen city of
Leuven and characterize the cellular network at different altitudes above
ground level to give us some insight on how well a UAV can utilize the
cellular network. Using the info of the environment and the general parameters
used in a typical LTE deployment, see Table I, the simulator evaluates the
channel model for every 3D point in the simulation area, and as such can
calculate the received power from each sector at each 3D point. This happens
for three different network operators (Op1, Op2 & Op3). The base station sites
of these operators can coincide. The received power from all sectors can be
used to determine cell allocation and SINR levels. SINR levels can then be
further used to calculate coverage.

This characterization of the environment is performed for different operators.
After combining all data, we can calculate the SINR at any point in a multi-
operator network, as in [25]:

$SINR(\vec{p})=\max\limits_{k}\Bigg{(}\frac{P_{r}(\vec{p},a)}{\sum^{A}_{i_{k}\neq
a}P_{r,i_{k}}(\vec{p})+N}\Bigg{)},$ (2)

where $k$ is the operator index. These SINR values can then be used to
calculate the coverage as described in Section II. Using all the datapoints at
a specific altitude we can also evaluate the area size of continuous outage
zones and we can calculate the largest outage zone, $OUT_{max}$.

TABLE I: Simulation Parameters Parameter | Value
---|---
Simulation area, | $9~{}km^{2}$
Resolution, | $5~{}m$
Carrier frequency, $f_{c}$ | $1800~{}MHz$
Bandwidth, $B$ | $20~{}MHz$
Outage-threshold, $T$ | $-6~{}dB$
Noise density, $N_{0}$ [26] | $-174~{}dBm/Hz$
Noise number, $F$ [26] | $9~{}dB$

####  III-B2 Mobile scenario

Finally a second scenario is simulated to get more insight into the mobility
characteristics of a multi-network approach. In [15], the simulator was
extended with the possibility to consider a moving drone that travels over a
certain trajectory in three dimensional space and can hence experience
changing network circumstances. They also implemented a simplified handover
mechanism based on the `A3`-event from the 4G- and 5G standards [27].

Alongside the handover event, there exists another event that could cause
temporary disconnection and latency issues, the RLF event. When the UE detects
problems in the connection it will wait for a specific timer (`T310` [27]) to
run out and when the problem is not resolved at that time the UE will consider
itself in RLF and begin the cell selection procedure again. The handover
threshold and the RLF timer are network parameters that the operator can tweak
on a per-UE basis. Because we want very low latency for drone applications,
the following simulation uses a `T310` of $200~{}ms$. For the A3-threshold we
take $-2~{}dB$ as in [10].

In a multi-network context the definitions are slightly different. First, a
RLF is defined as an event when the user is not connected to any network.
Secondly, a handover is defined as:

  * •

One of the used networks has a HO and the other used networks are in RLF at
that time;

  * •

All used networks experience a handover within 1 second.

This period of $1~{}s$ is self-defined and is a measure for how robust one
wants to protect the combined link against near simultaneous handovers. A
longer period gives more margin, a shorter period assumes that the handover is
correctly resolved when the other network starts a handover. The remainder of
the simulation parameters can be found in Table II, refer to [15] for the full
explanation of all the parameters.

TABLE II: Handover Simulation Parameters Parameter | Value
---|---
Heights (AGL) $h$ |  $20$ m, $40$ m, $80$ m, $160$ m
Drone velocity $v$ | $15~{}m/s$
Outage-threshold $T$ | $-6~{}dB$
A3 - Offset $T_{A3}$ | $2~{}dB$
RLF-timer `T310` | $200~{}ms$
Simulation timestep $\delta_{t}$ | $100~{}ms$
Number simulations | $5000$

##  IV Results

In this section, we look at the results generated by the simulations described
above. We evaluate the metrics at different altitudes to get a better insight
into the effect of drone height. We also draw the evaluations under different
network loads, but we focus on the $100~{}\%$ network load as this is the
worst-case scenario but the most important scenario when looking at the
reliability of the communication link.

First, we evaluate the coverage situation, followed by a study of the effects
of using multiple operators. Next, we evaluate the handover results and verify
whether using multiple networks results in an improvement in terms of
handovers.

###  IV-A Coverage

$-10$$-5$$0$$5$$10$$15$$20$$0$$0.2$$0.4$$0.6$$0.8$$1$SINR
[$dB$]CDFhg$100\%$hghg$50\%$hghghg$20m$hghg$40m$hghg$80m$hghg$160m$

Figure 2: CDF of the SINR of Operator 1 at different heights for a half or
full network load. The vertical red line represent the coverage threshold.
SINR levels below this threshold are considered as network outage. A network
at half load has almost 100% coverage up to an altitude of $160~{}m$.

The resulting statistics of the SINR for Operator 1 can be seen in Fig. 2
where the CDF of the SINR for different altitudes and loads is shown. To meet
the requirements for downlink communications, UAVs must have a minimum SINR of
$-6~{}dB$ [14] for a command-and-control link. This is the coverage threshold
$T$, below which the drone experiences a SINR that is too low to achieve
successful communication. This is indicated by a vertical red line.

The figure shows that as the drone flies higher, we can see that the density
function shifts to the left, indicating that SINR values drop. One can also
see that the curve becomes steeper indicating that the variance keeps
decreasing with height: a very wide distribution at $20~{}m$ becomes much more
concentrated at $160~{}m$.

The effect of choosing a different outage threshold can be seen in Fig. 3,
using the suggested threshold of $-6~{}dB$ results in a coverage probability
of $0.74$ at an altitude of $160~{}m$. If a larger threshold is chosen the
coverage probability drops drastically with $P_{cov}$ reaching zero at a
threshold of $2~{}dB$ for an altitude of $160~{}m$. These results indicate the
bad aerial coverage situation.

$-6$$-4$$-2$$0$$2$$4$$6$$0$$0.2$$0.4$$0.6$$0.8$$1$Coverage Threshold, $T$
[$dB$]Coverage Probability,
$P_{cov}$hg$100\%$hghg$50\%$hghghg$20~{}m$hghg$40~{}m$hghg$80~{}m$hghg$160~{}m$

Figure 3: Impact of the outage threshold, $T$, on the coverage probability,
$P_{cov}$ at different altitudes under a $50\%$ or a $100\%$ network load.

In an effort to solve this coverage problem, we consider a connection with
multiple networks at the same time. The SINR values in this network are
calculated using (2). The coverage probability as a function of the drone
altitude can be seen in Fig. 4(a). When using one network operator, the
probability stays one up until around $20~{}m$ height as the network is
designed this way. However, we can see the coverage drops as soon as the UAV
flies above rooftop height. In a multi-operator network we can see that the
100% coverage reaches a much higher altitude. At an altitude of $100~{}m$ the
coverage probability in a multi-operator network is still 0.99. This altitude
is much more likely to be used by a drone network. Above this altitude the
coverage drops again, albeit less than with one operator. The increase in
coverage probability from using two compared to using three operators is less
but can be considered for ultra-reliable systems.

$0$$50$$100$$150$$200$$250$$300$$0.5$$0.6$$0.7$$0.8$$0.9$$1$Height
[$m$]Coverage Probability, $P_{cov}$op1op1 & op2op1 & op2 & op3 (a) Coverage
probability $P_{cov}$

$0$$50$$100$$150$$200$$250$$300$$0.0$$1.0$$2.0$$3.0$Height [$m$]Max Outage
Zone, $OUT_{max}$ [$km^{2}$]op1op1 & op2op1 & op2 & op3 (b) Maximum outage
zone $OUT_{max}$

Figure 4: Impact of multi-operator diversity on the network performance. Using
multiple cellular networks enables UAV connectivity at higher altitudes.

Fig. 4(b) shows the $OUT_{max}$ as function of the altitude. Using a single
operator results in relatively large outage zones, especially at altitudes
above $100~{}m$. In a multi-operator scenario the size of the outage zones
stays relatively small even at very large altitudes. A single operator network
experiences much larger continuous outage zones than a two or three operator
network. This can be explained by the fact that many of the antenna sites are
different for the selection operators and even if they utilize the same
antenna site they often deploy their sectors with different infrastructure,
resulting in different azimuths, tilts, etc.

###  IV-B Handovers

The assignment map indicates which sector has the highest received power at
each point. In Fig. 5 the assignment map of two operators at different
altitudes is shown. Each color represents a sector, meaning that the color of
a pixel determines which sector a user is assigned to at this location. One
can observe that at $120~{}m$ height the assignment pattern is much more
fragmented than the one at $20~{}m$ height. Due to the antenna sidelobes and
nulls, UAVs at high altitudes will connect to sectors that are not necessarily
the closest or they will alternate rapidly between multiple sectors. This will
lead to higher handover frequency. These handovers result in throughput drops,
where at these altitudes throughput is already a scarce resource. To build a
reliable UAV communications network, reducing the number of handovers and
Radio-Link-Failures is in the best of interest.

(a) Operator 1, $20~{}m$

(b) Operator 1, $120~{}m$

(c) Operator 2, $20~{}m$

(d) Operator 2, $120~{}m$

Figure 5: Sector assignment map for two network operators at $20~{}m$ and
$120~{}m$ height. Each color represents a different sector. The color of a
pixel indicates which sector a user at this location is assigned to.

Fig. 5 also shows the difference in assignment patterns between two different
operators. We can utilize this diversity to create a multi-operator network
where when a handover needs to happen or when a RLF occurs, the user can
always fall back on the other operator’s network.

The results can be neatly observed in the Fig. 6. Fig. 6(a) shows the handover
frequency experienced by a drone flying at certain altitudes. When looking at
a single operator system, we can see that a drone flying at an altitude of
$160~{}m$ experiences a median of five handovers, this is a handover every 12
seconds. It is clear that this will cause significant problems for the command
and control link. In contrast, the multi operator systems experience
approximately zero handovers at the same altitude, due to the network
diversity.

The same positive effect can be seen when monitoring the RLF duration. Fig.
6(b) shows the duration a drone will not be connected to the network when it
encounters a RLF. At an altitude of $160~{}m$ a drone using a single network
will have a median disconnection time of $4~{}s$ when encountering a no-
coverage zone. This is clearly not a desired situation. However, a drone
connecting to multiple networks simultaneously can bring down this duration to
a median of $1~{}s$. This shows the clear benefits for drones of connecting to
multiple cellular networks at the same time.

204080160$0$$5$$10$$15$Height [$m$]Handovers/min.op1op1 + op2op1 + op2 + op3
(a) Number of handovers per minute

204080160$0$$5$$10$$15$Height [$m$]RLF duration [$s$]op1op1 + op2op1 + op2 +
op3 (b) Radio Link Failure duration

Figure 6: Impact of multi-operator diversity on the number of handovers and on
the time spent disconnected due to RLF.

##  V Conclusion

In this article an application of multi-operator network diversity has been
explored to improve the network conditions for UAVs in deployed mobile
networks. The results indicate that the increase in performance is indeed
significant: the coverage can be improved by 20% even in the worst case
scenario of full network load, while the size of the outage zones is up to ten
times smaller, particularly for high altitudes, than for the separate
networks, meaning that the drone will have less chance to be unreachable for a
long time due to bad network conditions. The mobility characteristics can also
be improved, the possible disconnection events can be reduced significantly,
resulting in a better overall latency for the combined connection.

## References

  * [1] A. Jonas, R. Shanker, K. Liwag, M. Sharpe, B. Kovanis, E. Silverberg, E. Zaghi, and M. Pasterchick, “eVTOL/Urban Air Mobility TAM Update: A Slow Take-Off, But Sky’s the Limit,” _Morgan Stanley Research_ , 2021.
  * [2] E. Vinogradov, H. Sallouha, S. De Bast, M. Azari, and S. Pollin, “Tutorial on UAVs: A Blue Sky View on Wireless Communication,” _Journal of Mobile Multimedia_ , vol. 14, no. 4, 2018.
  * [3] M. M. Azari, S. Solanki, S. Chatzinotas, O. Kodheli, H. Sallouha, A. Colpaert, J. F. M. Montoya, S. Pollin, A. Haqiqatnejad, A. Mostaani, E. Lagunas, and B. Ottersten, “Evolution of Non-Terrestrial Networks From 5G to 6G: A Survey,” _arXiv preprint arXiv:2107.06881_ , 2021.
  * [4] G. Geraci, A. Garcia-Rodriguez, M. Azari, A. Lozano, M. Mezzavilla, S. Chatzinotas, Y. Chen, S. Rangan, and M. D. Renzo, “What Will the Future of UAV Cellular Communications Be? A Flight from 5G to 6G,” _arXiv preprint arXiv:2105.04842v1_ , 2021.
  * [5] “Drone market outlook in 2021: Industry growth trends, market stats and forecast,” Business Insider, 2021.
  * [6] Federal Aviation Administration, “FAA Aerospace Forecast, Fiscal Years 2019-2039,” 2019.
  * [7] E. Vinogradov and S. Pollin, “Drone technology: interdisciplinary systematic assessment of knowledge gaps and potential solutions,” _arXiv preprint arXiv:2110.07532_ , 2021.
  * [8] International Civil Avation Organization, “Unmanned Aircraft Systems Traffic Management (UTM) – A Common Framework with Core Principles for Global Harmonization - Edition 3,” Tech. Rep., 2021.
  * [9] A. Bauranov and J. Rakas, “Designing airspace for urban air mobility: A review of concepts and approaches,” _Progress in Aerospace Sciences_ , vol. 125, 2021.
  * [10] 3rd Generation Partnership Project (3GPP), “Enhanced LTE support for aerial vehicles,” Technical report (TR) 36.777, 2018.
  * [11] M. M. Azari, F. Rosas, and S. Pollin, “Cellular Connectivity for UAVs: Network Modeling, Performance Analysis, and Design Guidelines,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 7, 2019.
  * [12] J. Rodríguez-Piñeiro, T. Domínguez-Bolaño, X. Cai, Z. Huang, and X. Yin, “Air-to-Ground Channel Characterization for Low-Height UAVs in Realistic Network Deployments,” _IEEE Transactions on Antennas and Propagation_ , vol. 69, no. 2, 2021.
  * [13] B. Van Der Bergh, A. Chiumento, and S. Pollin, “LTE in the sky: trading off propagation benefits with interference costs for aerial nodes,” _IEEE Communications Magazine_ , vol. 54, no. 5, 2016.
  * [14] A. Colpaert, E. Vinogradov, and S. Pollin, “Aerial Coverage Analysis of Cellular Systems at LTE and mmWave Frequencies Using 3D City Models,” _MDPI Sensors_ , vol. 18, no. 12, 2018.
  * [15] ——, “3D beamforming and handover analysis for UAV networks,” in _IEEE Globecom Workshops (GC Wkshps)_ , 2020.
  * [16] M. Gharib, S. Nandadapu, and F. Afghah, “An Exhaustive Study of Using Commercial LTE Network for UAV Communication in Rural Areas,” in _IEEE International Conference on Communications Workshops (ICC Workshops)_ , 2021.
  * [17] A. Fakhreddine, C. Bettstetter, S. Hayat, R. Muzaffar, and D. Emini, “Handover challenges for cellular-connected drones,” _Proceedings of the 5th Workshop on Micro Aerial Vehicle Networks, Systems, and Applications_ , 2019.
  * [18] R. Amorim, I. Z. Kovacs, J. Wigard, G. Pocovi, T. B. Sorensen, and P. Mogensen, “Improving drone’s command and control link reliability through dual-network connectivity,” in _IEEE 89th Vehicular Technology Conference (VTC2019-Spring)_ , 2019.
  * [19] J. Säe, R. Wirén, J. Kauppi, J. Torsner, S. Andreev, and M. Valkama, “Reliability of uav connectivity in dual-mno networks: A performance measurement campaign,” in _IEEE International Conference on Communications Workshops (ICC Workshops)_ , 2020.
  * [20] M. Bacco, P. Cassarà, and A. Gotta, “Air-to-ground real-time multimedia delivery: A multipath testbed,” _Vehicular Communications_ , vol. 33, 2022\.
  * [21] Y. Zeng and R. Zhang, “Energy-Efficient UAV Communication With Trajectory Optimization,” _IEEE Transactions on Wireless Communications_ , vol. 16, no. 6, 2017.
  * [22] S. De Bast, E. Vinogradov, and S. Pollin, “Cellular Coverage-Aware Path Planning for UAVs,” in _IEEE 20th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)_ , 2019.
  * [23] BIPT: Sitebeheer. [Online]. Available: https://sites.bipt.be/
  * [24] Huawei, “A704516R0 Antenna Specifications.” [Online]. Available: https://www.huawei.com/ucmf/groups/public/documents/attachments/a704516r0-pdf.pdf
  * [25] H. C. Nguyen, R. Amorim, J. Wigard, I. Z. Kovacs, and P. Mogensen, “Using LTE Networks for UAV Command and Control Link: A Rural-Area Coverage Analysis,” in _IEEE 86th Vehicular Technology Conference (VTC-Fall)_ , 2017.
  * [26] 3rd Generation Partnership Project (3GPP), “Radio Frequency (RF) system scenarios,” Technical report (TR) 36.942, 2018.
  * [27] ——, “Technical Specification Group Radio Access Network; NR; Radio Resource Control (RRC) protocol specification Version 15.13.0,” 2021.


# Hardware Acceleration of Fully Quantized BERT for Efficient Natural Language
Processing

Zejian Liu1,2, Gang Li1, Jian Cheng1,2 1National Laboratory of Pattern
Recognition, Institute of Automation, Chinese Academy of Sciences
2School of Future Technology, University of Chinese Academy of Sciences
Beijing, China
<EMAIL_ADDRESS>{gang.li<EMAIL_ADDRESS>
###### Abstract

BERT is the most recent Transformer-based model that achieves state-of-the-art
performance in various NLP tasks. In this paper, we investigate the hardware
acceleration of BERT on FPGA for edge computing. To tackle the issue of huge
computational complexity and memory footprint, we propose to fully quantize
the BERT (FQ-BERT), including weights, activations, softmax, layer
normalization, and all the intermediate results. Experiments demonstrate that
the FQ-BERT can achieve 7.94$\times$compression for weights with negligible
performance loss. We then propose an accelerator tailored for the FQ-BERT and
evaluate on Xilinx ZCU102 and ZCU111 FPGA. It can achieve a performance-per-
watt of 3.18 fps/W, which is 28.91$\times$ and 12.72$\times$ over Intel(R)
Core(TM) i7-8700 CPU and NVIDIA K80 GPU, respectively.

##  I Introduction

In recent years, transformer-based models [1] have been ubiquitous in various
natural language processing (NLP) applications, including machine
translation[2], question answering and document analysis[3]. Compared with
traditional recurrent neural network (RNN) and long short-term memory
(LSTM)[4], transformer-based model uses multi-head self-attention mechanism to
encode and decode the information of input sentence to achieve better
representative ability, and the Bidirectional Encoder Representations from
Transformers (BERT)[5] is the state-of-the-art transformer-based model.

The architecture of BERT is depicted in Figure 1. Different from the original
transformer, BERT only consists of stacked encoder layers as it focuses on
learning language representations, which is similar to convolutional neural
network for computer vision. The building block of self-attention is
essentially matrix-matrix multiplication, which is suitable for massive
parallel computing, therefore high-performance GPUs are preferred to
accelerate the training and inference in the cloud. However, the vast
computational complexity ($>$20GFLOPs) and memory requirement ($>$320MB
floating-point parameters) of BERT poses a significant challenge for resource-
constraint platforms, which hinders the applications on edge devices.

In this paper, we investigate the acceleration of BERT on resource-constraint
FPGAs through algorithm/hardware co-design. Specifically, to reduce memory
footprint, we adopt quantization to compress the original model. Different
from Q8BERT[6] and Q-BERT[7], which only quantize part of parameters, we
propose a fully quantized BERT (FQ-BERT). FQ-BERT represents weights,
activations, scale factors, softmax, layer normalization, and all the other
intermediate results with integer or fixed-point format. Then we present an
accelerator tailored for the proposed FQ-BERT and evaluate on two separate
FPGAs. Experiments demonstrate that the proposed accelerator outperforms CPU
and GPU by a large margin in terms of performance-per-watt with less than 1%
accuracy degradation on SST-2 dataset. The main contributions of this paper
can be summarized as follows:

  * •

We propose an efficient FPGA-based BERT accelerator via algorithm/hardware co-
design. On the algorithm side, we propose a hardware-friendly FQ-BERT that
represents all parameters and intermediate results with integer or fixed-point
data type. On the hardware side, to support different operations during
inference, we present an accelerator with dot-product based PE and bit-level
reconfigurable multiplier for 8/4-bit and 8/8-bit multiplication at run-time.

Figure 1: Left: the overall BERT architecture; middle: the architecture of
multi-head self-attention module; right: the architecture of scaled dot-
product attention layer.

  * •

We evaluate 8/4-bit FQ-BERT on SST-2 and MNLI dataset, which achieve a
compression ratio of 7.94$\times$, with 0.7% and 3.5% accuracy drop
respectively. Then we implement accelerator on Xilinx ZCU102 and ZCU111 MPSoC.
Experiments demonstrate that our accelerator can achieve a performance-per-
watt of 3.18 fps/W, which is 28.91$\times$ and 12.72$\times$ over Intel(R)
Core(TM) i7-8700 CPU and NVIDIA K80 GPU, respectively.

Figure 2: The overall architecture of the proposed accelerator for fully
quantized BERT.

##  II Fully Quantized BERT

In this section, we introduce the methods and tricks that we used for
quantizing parameters (weights, activations, biases and others) of the
original BERT to obtain FQ-BERT.

###  II-A Weights and Activations

To reduce the computational complexity and memory footprint, we adopt
quantization to compress the BERT. In this work, we use symmetric linear
quantization strategy for both weights and activations, because symmetric
quantization is more hardware friendly for the lack of zero-point.
Specifically, for each element $x$ of an input tensor, the $k$-bit
quantization function is:

$\begin{split}x_{c}&=clamp(x,MIN,MAX)\\\ s&=scale(x_{c},k)\\\ x_{I}&=\lfloor
x_{c}\times s\rceil\\\ x_{q}&=x_{I}/s\\\
clamp(x,a,b)&=min(max(x,a),b)\end{split}$ (1)

Figure 3 shows the quantization bitwidth of weights and corresponding accuracy
on two document clssification datasets. It can be seen that when the bitwidth
of weights is lower than 4, the classification accuracy drops dramatically,
which indicates that 4-bit quantization for weights is a reasonable trade-off
to maintain accuracy and considerable compression ratio. $MIN$ and $MAX$ are
clip thresholds, which need to be carefully tuned during training. Its impact
on performance is also depicted in Figure 3. Note that for symmetric
quantization, $MIN=-MAX$. $s$ is a scaling factor, which is determined by the
value of weights or activations and the quantization bitwidth $k$. For
weights, the scaling factor is calculated according to:

$\begin{split}s_{w}=scale(W,k)=\frac{2^{k-1}-1}{max(|W|)}\end{split}$ (2)

We use exponential moving average (EMA) to collect the statistics of
activations to determine scaling factor during inference:

$\begin{split}s_{a}=scale(A,k)=\frac{2^{k-1}-1}{EMA(max(|A|))}\end{split}$ (3)

(a) SST-2

(b) MNLI

Figure 3: Impact of different quantization bitwidth of weights on accuracy.

###  II-B Biases and Others

Different from previous work that only weights and activations are quantized,
our goal is to obtain a hardware-friendly BERT that can facilitate the
deployment on hardware and improve the utilization of computing units. To this
end, we further quantize all the biases to 32-bit integers with the help of
weights’ and activations’ scaling factors:

$\begin{split}bias_{q}&=\lfloor bias\times s_{bias}\rceil/s_{bias}\\\
s_{bias}&=s_{a}\times s_{w}\end{split}$ (4)

As weights, activations and biases are all quantized, the calculation of the
output can be rewritten as:

$\begin{split}y_{I}=\lfloor y\times s_{y}\rceil&=(\sum{a_{I}\times
w_{I}}+b_{I})\times s_{f}\\\ s_{f}&=\frac{s_{y}}{s_{a}\times
s_{w}}\end{split}$ (5)

where $s_{a}$, $s_{w}$ and $s_{y}$ are quantized to 8-bit values, and $s_{f}$
is a 32-bit integer. In addition, we also quantize the numerator and output of
softmax module, and parameters of layer normalization to 8-bit fixed-point
values.

(a) Type A

(b) Type B

Figure 4: Two type of BIM. $M=4$.

##  III Accelerator

###  III-A System Overview

The overall architecture of our proposed accelerator is shown in Figure 2. It
can be divided into two parts: the software program on CPU and the hardware
accelerator on FPGA.

As shown in Figure 1, the whole model consists of embedding layers, encoder
layers and task-specific layers. Different from encoder layers, the
computation cost of other layers is small, but the memory cost is large, so
these computations are completed on CPU, and the results are sent to FPGA. The
weights are stored in off-chip memory and sent to FPGA through AXI4 interface.

On the FPGA side, the accelerator consists of Processing Units (PUs), Layer
Normalization (LN) Core, Softmax Core and On-chip Buffers. Each PU contains
multiple Processing Elements (PEs), which is used for variable bitwidth
matrix-vector multiplication. The On-chip Buffers include input/output buffer,
weight buffer, parameter buffer and intermediate buffer. Specifically, the
weight buffer is double buffered to overlap off-chip data transfer. The
intermediate buffer is used for storing all the intermediate data, including
$Q$, $K$, $V$ and attention matrix. The parameter buffer is used for caching
data for special computation, such as scaling factors and softmax lookup table
values. These data are loaded in the initialization procedure.

###  III-B Computing Unit

PE. As different part of FQ-BERT has different bitwidth, the input bitwidth of
multiplication has different combinations, such as 8-bit $\times$ 4-bit for
$XW^{Q}$ and 8-bit $\times$ 8-bit for $QK^{T}$. In order to reuse the same
module to support different operations, we design a Bit-split Inner-product
Module (BIM), which is similar to Bit-fusion[8], as depicted in Figure 4. Each
BIM contains $M=2m$ 8-bit $\times$ 4-bit multipliers, two $m$-adder tree and
several shift-add logic. Each multiplier has a sign signal to indicate the
input number is signed or unsigned. The shift-add logic is used for shifting
the partial sum when the bitwidth of input is larger than 4 bits. The position
of shift-add logic has two choices, as shown in Fig.4, and using shift logic
at adder tree’s output (Type A) can save more resources, though this need to
rearrange the input data.

The output of BIM is a signed partial sum, which will be sent to an
accumulator, as shown in Figure 2. The accumulator sums this input and
previous data and send the results to the Psum Buf. When the calculation is
completed, the results will be sent to quantization module to get final value
incorporating with biases and scaling factors. Since the quantization process
spends more than one cycle to complete, the Psum Buf is double buffered to
ensure the calculation can be pipelined.

Softmax Core. BERT utilizes the softmax function to convert $QK^{T}$ to
attention matrix, the expression of softmax is:

$Softmax(\boldsymbol{x})_{i}=\frac{exp(x_{i})}{\sum_{j}{exp(x_{j})}}$ (6)

whele $\boldsymbol{x}$ is a row vector of $QK^{T}$ matrix. As the exponential
function $exp(x_{i})$ is very expensive in terms of resource consumption, we
use a lookup table instead. However, it is difficult to determine the number
of entries of lookup table as the range of exponential function is very large
and sensitive to input value. Fortunately, softmax is invariant to
subtraction, i.e. subtract a constant for all the elements in the vector will
not change the final results. If all the elements subtract the maximum value
of the input vector, the output of exponential function will always be limited
between 0 and 1. What’s more, as we quantize $exp(x_{i})$ to 8-bit, only 256
sampling points are needed.

LN Core. As most operations in layer normalization are element-wise
multiplications and additions, with some non-linear operations, PE arrays are
unsuitable for this computation pattern. We design a coarse-grained 3-stage
pipelined SIMD unit to fully utilize its parallelism. The first stage consumes
two input vectors with two scaling factors and generates one vector and a mean
value. The second stage subtracts the mean value from each element in the
input and calculates variance value. The third stage finish the element-wise
multiplication.

Figure 5: The dataflow of proposed accelerator.

###  III-C Scheduling

The dataflow of the accelerator is shown in Figure 5. According to different
computation units and required weights, the computing process can be divide
into several stages. Although the weights have been quantized to 4-bit,
loading all the weights that needed per stage is still unpractical. So we
further divide each stage into several sub-stages, and only load part of
weights that will be used in next sub-stage. Through task-level scheduling,
the off-chip transfer can be completely overlapped by computing.

##  IV Evaluation

In this section, we evaluate the performance of proposed accelerator. First,
we introduce the software and hardware environment setup of our experiments.
Then, the performance of FQ-BERT is given. Finally, hardware resource
utilization, latency and power consumption are provided.

###  IV-A Experimental Setup

We implement the FQ-BERT using PyTorch and evaluate on two tasks, SST-2 and
MNLI, of GLUE benchmark[9]. To get the quantized model, we first train the
original model for 3 epochs with default hyper-parameters. Then we fine-tune
the model with quantization function.

The proposed hardware accelerator is built on Xilinx ZCU102 FPGA and ZCU111
MPSoC, and the frequency of FPGA part is set to 214 MHz. We implement the
accelerator using High Level Synthesis. Vivado HLS 2019.1 is used for
compiling the C++ code to generate synthesizable RTL.

The baseline program runs on Intel(R) Core(TM) i7-8700 CPU @ 3.2GHz and NVIDIA
K80 GPU with CUDA 10.1. As we only consider the latency, all the experiments
are running with batch size of 1, and the length of sentence is set to 128.

###  IV-B Accuracy

The accuracy of FQ-BERT is shown in Table I. For SST-2 task, the accuracy drop
is 0.81%, and for MNLI task, the accuracy drop is 3.08% and 3.61%,
respectively. The accuracy loss on MNLI is larger than SST-2 as it is a more
difficult task. After quantization, the model size is compressed 7.94$\times$.
We also investigate the impact of different parts of BERT when they are
quantized, the results are shown in Table II. We observe that the accuracy
drop is not always increasing when more parameters are quantized, as
quantizing softmax can improve the accuracy from 91.28% to 91.86%.

TABLE I: Accuracy of FQ-BERT and baseline BERT. | w/a | SST-2 | MNLI | MNLI-m | Comp. Ratio
---|---|---|---|---|---
BERT | 32/32 | 92.32 | 84.19 | 83.97 | 1$\times$
FQ-BERT | 4/8 | 91.51 | 81.11 | 80.36 | 7.94$\times$
TABLE II: Ablation study of quantization for different parts of BERT. w/a | scale | softmax | layer norm | accuracy
---|---|---|---|---
- | - | - | - | 92.32
✓ | - | - | - | 91.63
✓ | ✓ | - | - | 91.28
✓ | ✓ | ✓ | - | 91.86
✓ | ✓ | ✓ | ✓ | 91.51

###  IV-C Resource Consumption

Table III shows the resource consumption of the proposed accelerator for
different configurations. Specifically, the number of PEs $N$ and the number
of multipliers in BIM $M$ are examined. We can see that the DSP usage is very
high for the targeted FPGA. To demonstrate the scalability of our accelerator,
we double the number of total multipliers and implement it on ZCU111, and we
get nearly twice the performance improvement.

###  IV-D Performance and Energy Efficiency

We further compare the performance and energy efficiency of our accelerator
with CPU and GPU, results are shown in Table IV. Compared with CPU, our
accelerator achieves 6.10$\times$ and 28.91$\times$ improvement in latency and
energy efficiency. For GPU, our accelerator achieves 1.17$\times$ and
12.72$\times$ improvement.

TABLE III: Resource consumption and latency for different number of PEs and Multipliers in BIM. The number of PUs is 12. (N, M) | BRAM18K | DSP48E | FF | LUT | Latency(ms)
---|---|---|---|---|---
ZCU102 | 1824 | 2520 | 548160 | 274080 | -
(8, 16) | 838 | 1751 | 124433 | 123157 | 43.89
(16, 8) | 877 | 1671 | 151010 | 154192 | 45.35
ZCU111 | 2160 | 4272 | 850560 | 425280 | -
(16, 16) | 679∗ | 3287 | 201469 | 189724 | 23.79
∗Some memory are implemented using URAM, which are not reported
TABLE IV: Performance comparison on CPU, GPU and FPGA. | CPU | GPU | ZCU102 | ZCU111
---|---|---|---|---
Latency(ms) | 145.06 | 27.84 | 43.89 | 23.79
Power(W) | 65 | 143 | 9.8 | 13.2
fps/W | 0.11 | 0.25 | 2.32 | 3.18

##  V Conclusion

In this paper, we investigate the acceleration of BERT on FPGAs. To reduce
memory footprint, we propose to fully quantize all parameters of BERT,
including weights, activations, scale factors, softmax, layer normalization
and other intermediate results. Then we present an energy-efficient
accelerator and evaluate it on two separate FPGAs. Experiments show that our
accelerator outperforms CPU and GPU by factors of 28.91$\times$ and
12.72$\times$ on performance-per-watt respectively.

##  VI Acknowledgment

This work was supported in part by National Natural Science Foundation of
China (No.61972396, 61876182, 61906193), National Key Research and Development
Program of China (No. 2020AAA0103402), the Strategic Priority Research Program
of Chinese Academy of Sciences (No. XDB32050200).

## References

  * [1] A. Vaswani et al., “Attention is all you need,” in NeurIPS, 2017.
  * [2] Q. Wang et al., “Learning deep transformer models for machine translation,” in ACL, 2019.
  * [3] B. van Aken et al., “How does bert answer questions? a layer-wise analysis of transformer representations,” in CIKM, 2019.
  * [4] S. Hochreiter et al., “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
  * [5] J. Devlin et al., “BERT: Pre-training of deep bidirectional transformers for language understanding,” in NAACL, 2019.
  * [6] O. Zafrir et al., “Q8BERT: Quantized 8Bit BERT,” in NIPS workshop, 2019.
  * [7] S. Shen et al., “Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT,” in AAAI, 2019.
  * [8] H. Sharma et al., “Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural networks,” in ISCA, 2018.
  * [9] A. Wang et al., “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in ICLR, 2019.