# Improving Question Answering Performance
Using Knowledge Distillation and Active Learning

Yasaman Boreshban<EMAIL_ADDRESS>
Computer Engineering Department,
Sharif University of Technology, Tehran, Iran
Seyed Morteza Mirbostani<EMAIL_ADDRESS>
Department of Computer Engineering,
University of Guilan, Rasht, Iran
Gholamreza Ghassem-Sani<EMAIL_ADDRESS>
Computer Engineering Department,
Sharif University of Technology, Tehran, Iran
Seyed Abolghasem Mirroshandel<EMAIL_ADDRESS>
Department of Computer Engineering,
University of Guilan, Rasht, Iran
Shahin Amiriparian<EMAIL_ADDRESS>
Chair of Embedded Intelligence for Health Care & Wellbeing,
University of Augsburg, Augsburg, Germany

###### Abstract

Contemporary question answering (QA) systems, including transformer-based
architectures, suffer from increasing computational and model complexity which
render them inefficient for real-world applications with limited resources.
Further, training or even fine-tuning such models requires a vast amount of
labeled data which is often not available for the task at hand. In this
manuscript, we conduct a comprehensive analysis of the mentioned challenges
and introduce suitable countermeasures. We propose a novel knowledge
distillation (KD) approach to reduce the parameter and model complexity of a
pre-trained BERT system and utilize multiple active learning (AL) strategies
for immense reduction in annotation efforts. In particular, we demonstrate
that our model achieves the performance of a 6-layer TinyBERT and DistilBERT,
whilst using only 2% of their total parameters. Finally, by the integration of
our AL approaches into the BERT framework, we show that state-of-the-art
results on the SQuAD dataset can be achieved when we only use 20% of the
training data.

##  1 Introduction

The development of question answering (QA) systems is a relatively new
challenge in the field of natural language processing (NLP) (?). The ultimate
goal of creating such systems is to enable machines to comprehend text as well
as, or even better than, human beings (?). Extensive progress has been made in
this area over the last few years. In QA models, context paragraphs and their
corresponding questions are represented as a series of tokens (?). The
objective of a QA system is to predict the correct span within a paragraph in
which the answer to a given question resides. It is often the case that an
attention mechanism is also used to keep the dependency relations between
questions and paragraphs. Furthermore, two probability values are computed for
each token, which represents the likelihood of the token being the start and
end of an answer span. For each query, the system identifies the span with the
highest probability value, as the answer to the query.

With the insurgence of interest in deep neural networks (DNNs), recent QA
models have achieved excellent results. On some corpora, they have even
reached an accuracy level higher than humans. Nevertheless, these achievements
have been made possible with the cost of building very large and expensive NLP
models. Despite all the progress made, there are still several remaining
challenges and issues that need to be addressed. For instance, these models
often suffer from high complexity and low robustness issues. Moreover, they
normally require a massive amount of labeled data for training. These models
usually have too many parameters, leading to a considerable training time. In
addition, they are subject to extensive resource consumption for performant
operation and reasonable inference time, which makes them unfit for real-world
applications running on devices with limited resources such as mobile and
embedded devices (?). Highly effective deep learning-based approaches can
immensely enhance the performance of distributed systems, embedded devices,
and FPGAs. The use of machine learning technology in virtual and augmented
reality on hardware such as smart wearable devices has brought distinct
accomplishments in terms of features and capabilities. However, due to the
excessive computational complexity imposed by this technology, its
implementation on most portable devices is challenging and bounded by their
hardware limitations. Accordingly, to address this issue, different model
compression techniques have been introduced as a practical solution, which has
absorbed a lot of attention over the recent years.

Current compression techniques can be divided into four general groups of
parameter pruning and quantization, low-rank factorization,
transferred/compact convolutional filters, and knowledge distillation (KD)
(?). It has been suggested that among these methods, using KD can result in a
more significant improvement in terms of accuracy and performance.
Accordingly, we have decided to study the impact of KD on the QA task.

Another concerning issue entangled with DNNs is the robustness deficiency.
Although employing DNNs in NLP models has led to impressive results on
multiple downstream tasks, these models are not robust enough and are
extremely vulnerable to adversarial examples. For QA tasks, it has been
demonstrated that an intentional perturbation of a paragraph through including
adversarial sentences confuses even the best available QA models, causing a
severe reduction of their accuracy. This vulnerability against adversarial
examples also makes these models unsuitable for real-world scenarios.
Consequently, numerous studies addressing this issue have been conducted to
increase the robustness of the proposed models (?).

Recent accomplishments in DNN have been heavily dependent on the use of large
training datasets; conversely, DNNs are inefficient when trained on small
datasets; however, the number of available annotated corpora is inadequate,
and manual annotation is a costly procedure. Moreover, for some languages, the
required amount of annotated datasets is unavailable. In recent years, there
has been a limited number of studies conducted on unsupervised, semi-
supervised, and active learning (AL) for QA systems. In this study, we
introduce a novel combination of a parameter reduction technique and AL for QA
systems. We show that the results of this combination are comparable to that
of state-of-the-art models for this task.

For parameter reduction, we utilize KD to transfer the knowledge of a large
(complex) model to a condensed neural network. In other words, we train a
small model in such a way that its accuracy would be close to that of the
initial large model. In this study, we have used a pre-trained model as our
initial model and transferred its knowledge to a small QA model. It has been
demonstrated that employing KD significantly improves the robustness and
generalization of the models (?). In this paper, we have specifically
investigated the impact of KD on the robustness of QA systems. We also utilize
AL to minimize the cost of data labeling. To the best of our knowledge, AL has
not so far been applied to the task of QA. Since data annotation is an
expensive task, we can employ AL strategies to obtain reasonable results with
a small training dataset. Generally, the primary goal of AL is to reach high
accuracy with low-cost data labeling (?). During the AL process, we use
several strategies to select informative unlabeled data samples with the
ability to transfer more information to the model. Hence, we are able to
drastically reduce the required number of samples and their labeling costs for
training the model. By combining KD and AL methods, we build a model with a
significantly reduced number of parameters and required labeled samples. The
resultant model is capable of achieving comparable results to that of state-
of-the-art models.

The structure of this paper is as follows: we define the theoretical
background of QA systems in Section 2 and introduce our related works in
Section 3. We describe our proposed approaches in detail in Section 4. We give
a brief description of the datasets used in this study and present our
experimental results in Section 5. Finally, Section 6 includes our conclusions
and future works.

##  2 Theoretical Background

In this section, we first introduce domains of QA systems in Section 2.1.
Afterwards, question types and architecture of QA systems are described in
Sections 2.2 and 2.3 respectively. In Section 2.4, we review the concept of KD
as a model compression technique. Ultimately, we describe AL method in Section
2.5 that aims at reducing the annotation costs.

###  2.1 Domains of QA Systems

QA systems fall into two general categories of open domain and closed domain
systems (?). Open (or unrestricted) domains aim at answering various types of
questions about a diverse set of subjects such as sports, politics, religions,
etc. (?). In contrast, closed (or restricted) domains are bound to answer the
questions associated with a specific subject. The task of these systems is in
general simpler than that of open domain cases because NLP models can extract
information from a specific domain and utilize its features to predict a
suitable answer to a given question (?). Typically, the model covers the
answers to a limited number of questions that are frequently used in a
restricted domain (?).

###  2.2 Question Types of QA Systems

Different types of classifications of questions are available; however, in a
particular semantic category, which has absorbed more attention, questions
have been divided into categories of factoid, list, definition (or
description), hypothetical, causal, relationship, procedural, and confirmation
(?). In English, a factoid question normally starts with a Wh-interrogative
word such as “Who,” “What,” “When,” or “Where” (?). The answer to such a
question is usually an embedded fact within the body of the text that can be
either a numerical or a named entity. On the other hand, a list question is a
type of question with an answer as a list of text entities. Alternatively, an
answer to a definition question can be a full sentence about a term used in
the body of the question. Furthermore, answering a hypothetical question
requires information about a hypothetical event. To answer a causal question,
however, the system looks for information or an explanation about an event and
the question typically starts with “Why.” On the other hand, to answer a
relationship question, the system searches for a relationship established
between two entities. A procedural question is a type of question with an
answer including all the instructions required to fulfill the task mentioned
in the question. Lastly, a confirmation question requires a yes or no answer
for the event mentioned in the body of the question. Alternatively, questions
can be divided into two general categories of factoid and non-factoid. In this
case, non-factoid questions, which are more complex to answer, include all
question types except for the factoid one mentioned above.

###  2.3 Architectures of QA Systems

The architectures of QA systems can be divided into information retrieval
(IR)-based or reading comprehension (RC)-based systems (?). In Figures 1 and
2, a general overview of these architectures is depicted. The IR-based systems
includes four modules of question processing, document retrieval, passage
retrieval, and answer extraction.

[mode=buildnew,width=0.6]./assets/ir_based_qa_systems_architecture

Figure 1: The architecture of IR-based QA systems consists of question
processing, document retrieval, passage retrieval, and answer extraction
modules.

In the question processing module, the required processes are performed on the
question body. Semantic and structural relations between the question words
are extracted. Then, in the document retrieval module, the documents are
ranked. Next, in the passage retrieval module, the most fitting segments of
highly ranked documents that are related to the input question are chosen as
the candidate passage. Finally, in the answer extraction module, the candidate
passages are used to return the most probable answer (?).

[mode=buildnew,width=0.6]./assets/rc_based_qa_systems_architecture

Figure 2: The architecture of RC-based QA systems consists of question
processing and machine reading comprehension modules.

In RC-based systems, context paragraphs and their corresponding questions are
represented as a series of tokens $P=\\{p_{1},p_{2},p_{3},...,p_{n}\\}$ and
$Q=\\{q_{1},q_{2},q_{3},...,q_{n}\\}$ (?). The goal here is to predict the
answer in the form of a span within one of the context paragraphs,
$A=\\{p_{j},...,p_{j+k}\\}$. In such cases, the system is expected to analyze
questions and context paragraphs comprehensively to find the best (i.e., the
most relevant) answer. Although several different learning methods have been
employed in RC-based systems, deep learning methods, in particular, have
achieved a higher accuracy (?).

###  2.4 Knowledge Distillation

An effective technique for model compression, known as knowledge distillation,
has recently gained much popularity among researchers. Using KD, a compact
neural network can be trained in such a way that we achieve the same high
accuracy of a much larger network (?).

The KD architecture is composed of two components, i.e., a student model and a
teacher model. The teacher component is a large model with high accuracy but
with heavy computational costs and a large number of parameters. On the other
hand, the student component is a compact model with a smaller number of
parameters. The student model mimics the teacher’s behavior. However, it is
more suitable for deployment due to much lower computational costs. To imitate
the behavior of the teacher, the student, along with its own actual labels
(hard target), also employs the teacher’s output logits (soft target). As it
follows, the loss function consists of adding hard and soft terms:

$L=(1-\rho)C_{hard}({\bf\it x},{\bf\it y})+\rho C_{soft}({\bf\it x},{\bf\it
q}),$ (1)

$C_{hard}({\bf\it x},{\bf\it y})=-\sum\limits_{i=1}^{K}y_{i}\log p_{i}({\bf\it
x}),$ (2)

$C_{soft}({\bf\it x},{\bf\it q})=-\sum\limits_{i=1}^{K}q_{i}\log p_{i}({\bf\it
x}),$ (3)

where $C_{hard}$ is the cross-entropy (CE) loss function of the student model
and $C_{soft}$ is applied to the softmax of the output of both models. $\rho$
is the weight of the hard and soft cross-entropy losses. $K$ is the number of
output classes of $x$. $p_{i}(x)$ is the softmax output probability of the
$i$-th class of the student. The hard target $y$ is a one-hot $K$-dimensional
vector. $q$ is a soft target, which is a $K$-dimensional vector. $q_{i}$ is
the tempered softmax probability for $i$-th class of the teacher model, which
is computed as follows (?):

$q_{i}=\frac{exp(z_{i}({\bf\it x})/T)}{\sum_{j=1}^{K}{exp(z_{j}({\bf\it
x})/T)}},$ (4)

where $z_{i}({\bf\it x})$ is the pre-softmax output of the teacher model for
the $i$-th class. $T$ is the temperature. When $T$ is large, the class
probability distribution will be uniform. In other words, $q$ is a smooth
probability distribution containing the correct class information and between-
class similarity. Learning these correlations has a massive effect on the
performance of the student model. Temperature $T$ controls the importance of
the class similarity information during training. When $T$ is greater than 1,
small probabilities of non-target classes are emphasized; in that case, the
student learns class similarity information more accurately (?).

###  2.5 Active Learning

AL is a learning method that aims at minimizing the annotation costs without
sacrificing the accuracy (?). The main purpose of this approach is that if the
training algorithm is able to choose more informative data during the learning
process, the model can reach almost the same accuracy as a supervised method
with a much less amount of data. AL approaches are classified into three major
categories of membership query synthesis, stream-based selective sampling, and
pool-based sampling.

In membership query synthesis, new instances are generated for which an
omniscient expert is expected to provide the ground-truth labels. However,
those instances may not have a natural distribution, making the annotation
difficult even for a human (?). Selective sampling is an alternative approach
for synthesizing queries. This approach is also called stream-based (or
sequential) AL. Here, unlabeled instances are firstly sampled by the actual
distribution. Then it is decided if the samples should be labeled based on
their value (?). The pool-based sampling approach is based on the assumption
that we have a small set of labeled and an enormous pool of unlabeled data.
The best candidates (i.e., the most informative ones) are selected from the
pool by different selection criteria, annotated by an oracle, and added to the
labeled dataset. The training process is repeated every time that some labeled
samples are added to the training set (?, ?). All AL strategies must measure
the usefulness of unlabeled data based on some specified criteria, among which
the most popular one is the uncertainty measure.

##  3 Related Works

In this section, we first review the conventional and contemporary machine
learning methods for QA systems in Sections 3.1 and 3.2. Then, we compare
various KD and AL approaches in Sections 3.3 and 3.4, respectively.

###  3.1 Deep Learning-Based Models

In 2016, dynamic chunk reader was presented (?). It was able to extract varied
length answers; whereas, its predecessor models returned one word or a named
entity as the answer for each question. One of the widely used models is
called BiDAF (?), which employs long short-term memory (LSTM) and
bidirectional attention flow networks. To implement the character level
embedding, they have applied convolutional neural networks (CNNs), and to
obtain the vector representation of each word, they have used GloVe (?), a
pre-trained word embedding. In 2017, the DrQA model (?) was introduced. It
consists of two modules of document retriever, which extracts five related
documents for each question, and document reader composed of a bidirectional
LSTM network. Transformer was introduced in 2017 (?). Instead of using
recurrent neural networks (RNNs) or CNNs, a self-attention mechanism has been
used to increase parallelism. Transformers are encoder-decoder-based models
that heavily rely on the self-attention mechanism. Despite their overall high
accuracy, these models are extremely vulnerable when facing adversarial
samples, which results in low accuracy. In 2018, a model with a structure
consisting of a sentence selector connected to a QA model was proposed (?).
The sentence selector computes a selection score for each sentence based on
its word-level relevance and semantic similarity to the question. Sentences
with the highest scores are selected to be fed to the QA model. Additionally,
an encoder with a similar structure to DrQA has been used in this model.

QANet is a model which uses CNN instead of a recurrent architecture (?). It
was proposed in 2018. The encoder structure in QANet consists of a
convolution, a self-attention, and a feedforward layer. After encoding the
question and the answer, a standard self-attention mechanism is used to learn
the relations between the question and its corresponding answer. The
improvement of the learning speed has made QANet a suitable candidate for
applying data augmentation. Accordingly, using neural machine translation
(NMT) (?), the back-translation method has also been employed in QANet for the
data augmentation purpose. BERT is an extremely popular model, initially
released in late 2018 (?). Using bidirectional Transformer encoders, BERT was
unsupervised pre-trained on the tasks of masked language modeling (MLM) and
next sentence prediction (NSP). It has the capability of being fine-tuned on a
wide array of downstream tasks. BERT-like models managed to outperform
previous solutions on several NLP tasks, especially QA tasks. XLNet is another
successful architecture, which is based on autoregressive language models. It
has been fine-tuned for QA by some other models (?).

###  3.2 Semi-Supervised Learning

In the past few years, limited researches have been conducted on semi-
supervised QA systems. A model called GDAN (?) was proposed in 2017. This
model uses a generator to make fake questions using a set of unlabeled
documents, in addition to the real questions made by a human expert using the
same dataset. These generated questions are then fed to a discriminator that
tries to distinguish real questions from fake ones. The learning procedure of
both generator and discriminator networks continues until that the
discriminator will be unable to recognize the fake questions. There is another
semi-supervised QA research introduced in 2018, in which a series of questions
corresponding to a specific document structure is generated (?). The main idea
of this research is that the introduction section includes some questions that
are elaborately answered in the body of the article. Accordingly, all
sentences in the introduction, which is assumed to be the initial 20% of the
document, are regarded as questions $\\{q_{1},q_{2},q_{3},...,q_{n}\\}$, and
the remaining 80% is supposed to include the paragraphs
$\\{p_{1},p_{2},p_{3},...,p_{m}\\}$ that contain the answers. Then, the
matching $match(p_{i},q_{i})$ is computed for each given question-paragraph
pair. Whenever there is an exact match between the tokens of a question and a
paragraph, the matched span is identified as the answer to the question.

In another study, both supervised and unsupervised transfer learning has been
used. The focus of the study was on multiple-choice question answering (?).
Additionally, in another research conducted in 2018 (?), the transfer learning
method was employed for improving the learning ability of the network. In this
approach, SQuAD was used as the source dataset for pre-training the model;
both WikiQA and SemEval 2016 were used as the target datasets. In (?),
unsupervised learning was used for QA systems. To generate context, question,
and answer triples, some noun phrases and named entity mentions are selected
as the candidate answers. Then, these answers are converted to the form of
“fill-in-the-blank" cloze questions and finally translated into natural
questions.

In other areas of artificial intelligence, semi-supervised learning is deemed
an attractive technique. Many studies have been conducted on semi-supervised
learning in word sense disambiguation (?), temporal relation extraction (?),
and image classification (?, ?, ?, ?).

###  3.3 Knowledge Distillation

It was shown that KD can improve the model generalization and robustness. For
instance, using this technique in a QA system, the knowledge was transferred
from an ensemble teacher to a single student model (?). The reinforced
mnemonic reader (RMR) is a base model in which attention and reinforcement
learning have been integrated (?). This model was evaluated on SQuAD,
Adversarial SQuAD, and NarrativeQA datasets. In this work, the student was
made of a single RMR and the teacher was an ensemble model composed of 12
copies of the base model (i.e., RMR), each having different initial
parameters. A two-stage KD strategy with multiple teachers was used for web QA
systems (?). These two stages are pre-training and fine-tuning. The results of
this study showed that this method is performant in generalization. Self-
knowledge distillation (SKD) was used in (?). As it was mentioned before, in
KD, the knowledge is normally transferred from a large (teacher) model to a
small (student) model. However, in SKD, the source of the knowledge is the
student model itself. The results of applying KD methods in a study conducted
on dialog systems (?) with a dataset named Holl-E demonstrate that imitating
the behavior of the teacher model has a significant impact on the student’s
performance.

Recently, some studies have focused on KD using the BERT model as the teacher.
The main objective is to create a compact pre-trained (student) model with
much fewer parameters and much less inference time than that of the BERT
model, but at the same time with competitive accuracy. DistilBERT was
presented in 2019 (?). Unlike all previous models, in DistilBERT the KD method
is performed during the model pre-training stage. In this structure, the
BERTBASE model is the teacher. The total number of encoder layers of the
student is half of the layers of the teacher though their structures are
identical. It was demonstrated that using the DistilBERT model, the BERT’s
size can be reduced by 40% while preserving 97% of its language comprehension
capabilities (?). TinyBERT is another BERTBASE model created by KD (?). The KD
method used in this model is called Transformer distillation that is performed
in two stages of general distillation and task-specific distillation. At the
general distillation stage, the BERT model without fine-tuning is used as the
teacher, and TinyBERT is taken as the student that imitates the teacher’s
behavior through the application of Transformer distillation to a general-
domain corpus. At the task-specific distillation stage, however, this general
TinyBERT model is used for distilling the knowledge. For this purpose, at
first, the data augmentation process is performed. Then with the fine-tuned
BERT model used as the teacher, KD is applied to the resultant dataset. Both
stages are necessary for the TinyBERT model to achieve effective performance
and generalization. TinyBERT with four layers is 7.5 times smaller than the
BERTBASE model. Also, in terms of the inference time, it is 9.4 times faster
than BERTBASE. Nonetheless, it has gained 96.8% performance of BERTBASE
applied to the GLUE benchmark. In another study (?), KD was used to transfer
knowledge from the BERT model as the teacher to a student model. In this work,
intermediate layers along with the output of the last layer were used as the
medium of transferring knowledge, which showed satisfactory results in several
NLP downstream tasks. Furthermore, KD has also achieved promising results in
some other concepts such as multi-task learning (?, ?).

###  3.4 Active Learning

AL has been widely used in different subtasks of NLP. As an example, in a
research study focused on named entity recognition (NER), AL was applied to a
deep learning structure (?). The model used two CNNs for encoding characters
and words, in addition to an LSTM network as a decoder. The results showed,
with the aid of AL and merely one-fourth of the training dataset, the model
achieved 99% accuracy of the best deep learning models trained on the whole
dataset. In (?), using the BERT-CRF model, an uncertainty-based AL strategy
was applied to NER and achieved satisfactory results.

Although the combination of AL and deep learning has been recently applied to
other text processing fields such as coreference resolution (?), entity
resolution (?), machine translation (?), and dependency parsing (?), it has
not been used in QA tasks. Moreover, AL has been used in applications such as
imbalanced datasets (?) and black box attack (?).

##  4 Proposed Approaches

We propose an interpolated KD method to transfer knowledge to the model and
reduce its complexity, and AL strategies to minimize the labeled data
requirement. We combine these two approaches to building a small model that
gains the high accuracy of a complex model trained on a large corpus, using
only a small training dataset. Our approaches are explained in detail in
Sections 4.1 and 4.2.

###  4.1 Knowledge Distillation for QA

Pre-trained models such as BERT have achieved outstanding results in several
NLP tasks. However, as it was mentioned before, available DNNs are extremely
complex computational-wise, which makes them unfit for practical applications.
Our proposed approach to tackle this issue in QA systems is to apply KD
methods to such networks.

The proposed model structure is depicted in Figure 3. In this structure,
BERTLARGE (?) and QANet (?) are used as the teacher and the student model,
respectively. Instead of using RNN, QANet has been designed to use CNN in
addition to several self-attention layers. As a result of this change, QANet
has been shown to have a lower training and inference time in comparison with
earlier QA models. The formulations used in Figure 3 are as follows:

$\displaystyle L$ $\displaystyle=(1-\rho)C_{hard}+\rho C_{soft}$ (5)

$\displaystyle C_{hard}$
$\displaystyle=\sum_{i=1,2}\text{CE}(\text{softmax}({\bf\it\beta^{i}}),{\bf\it
y^{i}})$ (6)

$\displaystyle C_{soft}$ $\displaystyle=T^{2}\sum_{i=1,2}\text{KL}({\bf\it
p^{i}},{\bf\it q^{i}})$ (7)

$\displaystyle{\bf\it q^{i}}$
$\displaystyle=\text{softmax}({\bf\it\alpha^{i}}/T)$ (8)

$\displaystyle{\bf\it p^{i}}$
$\displaystyle=\text{softmax}({\bf\it\beta^{i}}/T)$ (9)

[mode=buildnew, width=0.55]./assets/approach_1_knowledge_distillation

Figure 3: Our proposed model structure employs an interpolated KD with
BERTLARGE as the teacher model and QANet as the student model. $C_{soft}$ is
the KL divergence error function that transfer knowledge from BERTLARGE to
QANet. $C_{hard}$ is the CE loss function of the QANet model. $L$ is the
weighted combination of $C_{soft}$ and $C_{hard}$.

The employment of KD in this work has been done as follows:

In standard QA models, the cross-entropy loss function is based on Equation
(10). This term is shown as $C_{hard}$ in Figure 3.

$\displaystyle L_{CE}=-\sum_{k=1}^{m}\sum_{l=1}^{m}{\bf\it y_{k}^{1}}\log
p^{1}(k)+{\bf\it y_{l}^{2}}\log p^{2}(l\lvert k)$ (10)

$y^{1}$ and $y^{2}$ are one-hots for the start and end answer tokens. $m$ is
the length of the paragraph. To apply KD, Kullback-Leibler (KL) divergence
error function is added to the cross-entropy error function, according to
Equation (11). This term is shown as $C_{soft}$ in Figure 3.

$\displaystyle
L_{KD}=\text{KL}(p\|q)=-\sum_{k=1}^{m}\sum_{l=1}^{m}p^{1}(k)\log[p^{1}(k)/q^{1}(k)]+p^{2}(l|k)\log[p^{2}(l|k)/q^{2}(l|k)]$
(11)

$q$ is the probability distribution of the start and end of the answer, which
is extracted from the teacher model. Additionally, log-of-softmax is used to
compute $p$ and $q$. Below, we briefly describe the architecture of both
teacher and student models used in this study.

####  4.1.1 The BERT Model

The Bidirectional Encoder Representations from Transformer (BERT) (?) is a
language model capable of being used as a pre-trained model. The BERT’s
architecture is based on the encoder structure of Transformers. Instead of CNN
and RNN components, the Transformer architecture comprises a number of
attention and self-attention layers, with the aim of increasing parallelism
(?). BERT is trained on a masked language modeling task, which allows
bidirectional training (i.e., simultaneous consideration of both left and
right contexts) in the model. It has been shown that in many NLP downstream
tasks, we can achieve much improved results by just adding a few layers to the
pre-trained BERT model.

####  4.1.2 The QANet Model

The architecture of QANet (?), which is shown in Figure 4, includes five main
layers: embedding, embedding encoder, context-query attention, model encoder,
and output layer. The convolutional and self-attention networks in the
embedding encoder and model encoder layers process the input tokens in
parallel, which leads to a significant increase in the performance of QANet in
contrast with other models.

Self-attention in this model is similar to that of Transformer. The embedding
layer takes advantage of GLoVe (?) for word embedding and CNNs for character
embedding, both of which are connected by their outputs. The embedding encoder
layer consists of a stack of encoders. An encoder block with its internal
components is shown on the right side of Figure 4. These components include a
convolutional, a self-attention, and a feed-forward layer. The size of the
kernel is set to 7. The number of filters and convolution layers are set to
128 and 4, respectively. Similar to Transformers, the self-attention structure
uses a multi-head attention module. All these layers (i.e., convolution, self-
attention, and feed-forward) are inside a separate residual block. The
structure of each model encoder layer is similar to the block on the right-
hand side of Figure 4. However, there are two convolution layers in each
block, and there are a total of seven encoder blocks in each layer.

There are three stacks of model encoders in QANet, and the parameters are
shared between all these encoders. In the output layer, the result of the
three stacks of model encoders is used to compute the start and end
probabilities of the answer span. For each token of the paragraph, the
probability values of the start and end of the span are computed using a
linear network and softmax.

[mode=buildnew, width=0.6]./assets/qanet_model_architecture

Figure 4: The QANet model architecture (left) adopted from (?) consists of
multiple encoder blocks. A single encoder block (right) includes a
convolution, a self-attention, and a feed-forward layer.

####  4.1.3 Tokenizers Alignment

An issue with the KD method in the proposed architecture is that the
tokenization algorithms used by the student and teacher models are not the
same. Spacy and WordPiece are the two different tokenizers used by QANet and
BERT, respectively. As a result of this issue, the size of the output of these
models is different. It should be noted that some words are divided into
multiple tokens using WordPiece. In such cases, a (##) sign will be attached
to all the sub-words except for the first one, indicating that those sub-words
should be appended to their preceding token. To apply the KD loss function,
the output of the models must have the same dimension. To tackle this issue,
we propose the following two approaches:

Rule-Based Approach. The token alignment algorithm that we have used consists
of two main steps of (1) finding exactly matched tokens and (2) finding
partially matched tokens. Before performing any comparison, all tokens are
converted into lower-case characters of ASCII encoding. For example, the word
Accommodation is tokenized as [acc, ##ommo, ##dation] by the BERT tokenizer.
After undergoing the mentioned conversion, these tokens are updated to [acc,
##ommo, ##dation]. In such a case, the same conversion is carried out by the
QANet tokenizer but results in [accommodation]. Then, the two mentioned steps
are performed as follows:

(1) If a QANet token is exactly matched by one of the BERT’s tokens, the
start-end logits computed by the BERT model are retained.

(2) When there are no exact matched tokens, the algorithm skips the first step
in favor of the second one. In this case, one of the following two conditions
holds: Either, (a) the initial character set of the BERT token is a subset of
characters in the QANet token, as shown in Figure 5; or (b) this subset
relation is reversed, as shown in Figure 6.

(2-a) In the majority of cases, when two tokens are not the same, the smaller
one is the token extracted by the BERT tokenizer. For instance, the string
Nuclear Astrophysics is tokenized and processed as follows:

QANet Tokens BERT Tokens 1 nuclear nuclear 2 astrophysics astro 3 . ##physics
4 .

Figure 5: An example where there are no exact matched tokens, and the BERT
token (i.e., astro) is a subset of the QANet token (i.e., astrophysics).

The BERT token astro is matched by the first part of the QANet token
astrophysics. The algorithm continues over the list of the BERT tokens and
finds the token physics, which is a sub-word and should be concatenated with
its previous token, astro. This procedure continues until the concatenated
string of tokens matches the QANet token, i.e., astrophysics. In this example,
with one forward step in the BERT token list, the concatenated string matches
the QANet token. In this case, the start and end logits of the token astro are
retained to be used later for computing the loss function of KD; and that of
the token physics is ignored. The next comparison is between the two (.)
characters, which results in an exact match and retaining the BERT’s start and
end logits for this token. This operation continues until the last member of
the BERT token list is processed.

(2-b) If the QANet token is the smaller one (for example in the tokenization
of the string cannot understand) the procedure is performed as follows:

QANet Tokens BERT Tokens 1 can cannot 2 not understand 3 understand

Figure 6: An example where there are no exact matched tokens, and the QANet
token (i.e., can) is a subset of the BERT token (i.e., cannot).

Comparing the QANet token can and token cannot of BERT determines how much we
should move forward in the token list of QANet. In this example, the algorithm
proceeds to the next token, and a new string is created by concatenation of
that token and its preceding token, i.e., can. The resultant string is then
compared against the current BERT token. In this case, moving just one token
ahead results in an exact match. Similar to (2-a), here the start and end
logits of the token can are retained to be used later for calculating the loss
function of KD; and that of the token not is ignored. In this example, the
algorithm will next find another exact match for token understand.

Interpolation Approach. As it was explained in step (2) of the rule-based
approach, in cases where we do not initially have an exact match, we retain
the start and end logits of the first sub-token and ignore those of other sub-
tokens. This results in losing some useful information. To overcome this
issue, we have designed another approach based on the interpolation of the
student logits. The goal is to make the dimension of the student context
logits to become equal to that of the teacher. This approach retains the
information content of all sub-token logits and results in an extensive
knowledge transfer between the teacher and student models. In this work, we
perform both linear and cubic interpolation methods on the original context
vector of the student to generate new logits required for resizing the
student’s vector. For transferring the knowledge, a mean squared error (MSE)
loss function is added to the main loss function to minimize the distance
between the interpolated student vector and that of the teacher. The new loss
function is as follows:

$\begin{multlined}L=(1-\rho)C_{hard}+\rho
C_{soft}+MSE(stdt_{intrpl},tchr)\end{multlined}L=(1-\rho)C_{hard}+\rho
C_{soft}+MSE(stdt_{intrpl},tchr)$ (12)

###  4.2 Active Learning for QA

AL is an efficient way to reduce the required time for creating a training
dataset. In this research, we have used the pool-based AL method. The
experiments have been performed on the SQuAD v1.1 dataset.

As it is shown in Algorithm 1, at first, all samples of the training dataset
are considered to be unannotated. Then, one percent of the dataset is selected
to be used for training the model. In this experiment, the chosen model is
BERTBASE which is trained for two epochs. Then, 10% of the rest of the
unlabeled dataset is selected to be added to the current training dataset
using the following strategies. This procedure continues until all unlabeled
samples are exhausted.

Input: Unlabeled data pool $\mathcal{U}$, labeled data set $\mathcal{L}$, most
informative unlabeled samples $\mathbf{x^{*}}$, AL sampling
$\phi(\cdot,\cdot)$

1 $\mathbf{x^{*}}\longleftarrow\text{arg}\max_{{\bf\it
x}\in\mathcal{U}}\phi(\mathbf{x},1\%)$;

2 $\mathcal{L}\longleftarrow label(\mathbf{x^{*}})$;

3 $\mathcal{U}\longleftarrow\mathcal{U}\setminus\mathbf{x^{*}}$;

4 repeat

5 $train\\_model(\mathcal{L})$;

6 $\mathbf{x^{*}}\longleftarrow\text{arg}\max_{{\bf\it
x}\in\mathcal{U}}\phi(\mathbf{x},10\%)$;

7 $\mathcal{L}\longleftarrow\mathcal{L}\cup label(\mathbf{x^{*}})$;

8 $\mathcal{U}\longleftarrow\mathcal{U}\setminus\mathbf{x^{*}}$;

9

10until  _$\lvert\mathcal{U}\rvert=0$_ ;

Algorithm 1 Pool-based AL approach

Most data sampling strategies are based on some uncertainty criteria. Next, we
describe the strategies that we have used in this work.

####  4.2.1 Least Confidence

The most widely used strategy of AL is the least confidence sampling (?). The
algorithm selects those instances that have the least confidence (i.e., based
on our model) for labeling. This method can be simply employed in
probabilistic models. For example, in a probabilistic binary classification
model, instances with a probability value around 0.5 are the ones in which the
model has the least confidence.

The output of the QA systems that we are interested in is a span extending
from the start to the end of the answer tokens. For each question, the model
returns multiple answer spans, among which the span with the highest
probability value will be selected. In each cycle, a fixed number (e.g., 10%)
of questions whose selected answer has the least probability value are
selected. The calculations are performed using Equations (13) and (14).

$\displaystyle{\bf\it x^{*}}=\text{arg}\max_{{\bf\it
x}}\big{[}1-p(A^{^}|{\bf\it x})\big{]}$ (13)

$\displaystyle A^{^}$ $\displaystyle=\text{arg}\max_{A}p(A|{\bf\it x})$ (14)

$A$ is the answer set returned by the model for a question. For each instance,
${\bf\it x}$, $A^{^}$ is the answer with the highest probability value given
by the model. In this approach, the selected answer with the least probability
value is chosen as the least confident instance, denoted by ${\bf\it x^{*}}$.
This instance is presumed to contain the highest information content of all.

####  4.2.2 Margin

Another option that can be used for data sampling is the margin criterion. In
this method, the probability difference between the two most probable labels
is calculated. This difference shows that samples with a larger margin are
easier to be classified by the model. That is because the classifier is more
confident about those labels. Conversely, the classifier is less confident
about those labels that have a smaller margin; therefore, knowing the actual
label of such instances helps the model discriminate them more effectively.
For applying this criterion to QA systems, the difference between the two most
probable answers returned for each question is taken as the margin. This
margin is calculated by Equation (15), in which $A_{1}^{^}$ and $A_{2}^{^}$
respectively denote the first two most probable answer to question ${\bf\it
x}$. Here, in each AL cycle, a subset of questions with the highest margin,
denoted by ${\bf\it x^{*}}$, are selected to be added to the training dataset.

${\bf\it x^{*}}=\text{arg}\max_{{\bf\it x}}\big{[}p(A^{^}_{1}|{\bf\it
x})-p(A^{^}_{2}|{\bf\it x})\big{]}$ (15)

####  4.2.3 Entropy

When there exist a large number of labels, the margin sampling method
practically ignores many labels. In such cases, it only considers the first
two labels. In this situation, the sampling method based on entropy, which is
calculated by Equation (16), is more suitable for detecting uncertainty.
$A_{i}^{^}$ denotes the $i$-th most probable answer returned for question
${\bf\it x}$.

${\bf\it x^{*}}=\text{arg}\max_{{\bf\it x}}\big{[}-\sum_{i}p(A^{^}_{i}|{\bf\it
x})\log{(A^{^}_{i}|{\bf\it x})}\big{]}$ (16)

For applying this method to QA systems, the first five most probable answers
for each question are selected as the candidate answers by the BERT model.
After calculating the entropy for these candidates, the samples with the
highest entropy are selected to be added to the training dataset.

####  4.2.4 Clustering Method

Clustering is another approach used in our study for data sampling. For this
purpose, first, some samples are selected from the unlabeled dataset pool by
the least confidence approach. If $k$ instances are to be selected for
labeling, we initially choose $3\times k$ instances based on the least
confidence criterion as our candidates. Then, for clustering, questions are
encoded with the universal sentence encoder (USE) (?), and using the $k$-means
algorithm and based on the Euclidean distance measure, those candidates will
be grouped into 10 clusters. To select final $k$ samples, each cluster is
sampled proportional to the number of its members. Selected instances are
annotated and added to the current labeled dataset. Then the model is re-
trained on the resulting dataset. This procedure continues until our
unlabelled data are exhausted.

##  5 Experimental Results

In this section, to assess the performance of our proposed approaches, we
explain the experiments111Our source code is publicly available at:
https://github.com/mirbostani/QA-KD-AL we have conducted and analyze their
results in detail.

###  5.1 Datasets

Over the past decades, many datasets have been proposed for QA tasks. In this
research, for the evaluation purpose, we have used two datasets, SQuAD v1.1
and Adversarial SQuAD, which are depicted in Table 1 and discussed next.

Dataset | Documents | Paragraphs | Questions
---|---|---|---
SQuAD v1.1: Train | 442 | 18,896 | 87,599
SQuAD v1.1: Development | 48 | 2,067 | 10,570
Adversarial SQuAD: ADDSENT | 48 | 3,358 | 3,560
Adversarial SQuAD: ADDONESENT | 48 | 1,585 | 1,787
Table 1: Statistics of the SQuAD v1.1 and Adversarial SQuAD datasets.

SQuAD. The SQuAD v1.1, released in 2016 (?), contains 107,785 question-answer
pairs on 536 articles extracted from Wikipedia. In SQuAD v1.1, the answer to
each question is a span of the text from the corresponding reading passage.
This dataset has provided the ground for significant progress in building more
accurate QA systems in recent years.

Adversarial SQuAD. In 2017, an adversarial dataset was built on top of SQuAD
v1.1 (?). Its training set has remained unchanged; however, some paragraphs of
its validation set have been extended by some adversarial sentences. This
dataset provides three different methods for generating adversarial sentences.
The first method is called ADDSENT, in which a few adversarial sentences
analogous to the given question are generated and appended to the paragraph
that contains the answer to that question. In the second method, called
ADDONESENT, a fixed sentence is added to all paragraphs. In the last approach,
named ADDANY, adversarial sentences are generated and appended to all
paragraphs, regardless of grammatical considerations. Therefore, using this
dataset, the robustness of QA models can be evaluated.

###  5.2 Evaluation Metrics

Two common measures used for the evaluation of QA systems are the F1-score and
exact match (EM). To evaluate the performance of the system, its predictions
and the ground truth answers are treated as a bag of tokens. The F1-score
measures the average overlap between the predicted and the ground truth
answers. It is computed for each given question and in the end, is averaged
over all the questions. EM measures the percentage of those predictions that
exactly match the ground truth answers. Here, when a prediction matches its
ground truth answer, EM is set to 1; otherwise, it is set to 0 (?).

###  5.3 Applying Knowledge Distillation

To apply KD, we used QANet as our student model because it does not use RNNs
in its architecture; therefore, it has a much lower training and inference
time as opposed to other earlier proposed models. Furthermore, BERTLARGE
(uncased version) was used as our teacher model. It has been suggested that
for training the BERT model, the hyperparameters of the model can be set to
one of the following learning rates: $2\text{\times}{10}^{-4}$,
$3\text{\times}{10}^{-4}$, and $5\text{\times}{10}^{-4}$. In our experiment,
we set the rate to $5\text{\times}{10}^{-4}$. The maximum tokens length, which
is the maximum length of the input to the model after tokenization, was set to
384. We also utilized the Pytorch framework for implementation. $\rho$
parameter, the coefficient of the soft loss function, was set to 0.7, and the
temperature $T$ was set to 10. The model was trained for 30 epochs in a batch
size of 14 samples.

As it is demonstrated in Figure 7, distilling the knowledge from BERT to QANet
increases by around 3.00 F1-score and EM.

(a) F1-score of QANet versus QANet+KD.

(b) EM of QANet versus QANet+KD.

Figure 7: Performance comparison between QANet before and after applying KD.

Table 2 shows the performance of various combinations of our proposed model in
comparison with other related models, using F1-score and EM measure. QANet is
the base model used in our study, and QANet+KD is the model on which KD has
been applied by adding the KL loss function to the model and using the rule-
based alignment technique. The QANet+KD+Linear model has a similar model
implementation as the QANet+KD with the addition of linear interpolation.
Furthermore, the QANet+KD+Cubic model is similar to QANet+KD+Linear except
that it utilizes cubic interpolation. The results of our experiments on the
SQuAD v1.1 dataset show an improvement of 3.50 and 4.00 percentage points in
F1-score and EM of the model, respectively, resulted from KD over the base
model.

Model | F1 | EM
---|---|---
BERTLARGE (Teacher) (?) | 93.15 | 86.91
BERTBASE (?) | 88.34 | 81.00
DistilBERT6 (?) | 86.90 | 79.10
DistilBERT4 (?) | 81.20 | 71.80
TinyBERT6 (?) | 87.50 | 79.70
TinyBERT4 (?) | 82.10 | 72.70
QANet (Base) (?) | 80.09 | 71.16
QANet+KD (Proposed) | 83.01 | 73.94
QANet+KD+Linear (Proposed) | 83.02 | 74.14
QANet+KD+Cubic (Proposed) | 83.51 | 75.20
Table 2: Performance of various models on SQuAD v1.1 dataset. BERTLARGE and
QANet are the teacher and the student model, respectively. QANet+KD is our
proposed model after applying KD, and QANet+KD+Linear/Cubic are the models
with interpolated KD. The best performance results, specified in bold, against
QANet (Base) belong to our QANet+KD+Cubic model, which outperforms both
DistilBERT4 and TinyBERT4 and is comparable with these models with six layers.

One of the problems with large pre-trained language models is their intrinsic
computational complexity. To further investigate this issue, we compared the
number of parameters and the inference time of our models with other related
models. As it is shown in Table 3, our approach does not change the parameters
and inference time of the base model; however, it is capable of improving the
performance of the model. Accordingly, choosing the base model directly
affects the total number of parameters and inference time of the proposed
model.

Model | #Params | Speedup
---|---|---
(Millions) | (batches/second)
BERTBASE (?) | 110.0 | $1.0\times$
DistilBERT6 (?) | 66.0 | $2.0\times$
DistilBERT4 (?) | 52.2 | $3.0\times$
TinyBERT6 (?) | 67.0 | $2.0\times$
TinyBERT4 (?) | 14.5 | $9.4\times$
QANet (Base) (?) | 1.3 | $2.0\times$
QANet+KD (Proposed) | 1.3 | $2.0\times$
Table 3: Number of parameters and speed comparison between our proposed model
and other distilled models on SQuAD v1.1 dataset.

Note that DistilBERT and TinyBERT are pre-trained models that cannot be
trained on a system with limited resources; however, due to the small number
of parameters of our proposed model, it can be fully trained on such systems.
Although the total number of parameters of the resulted model is about 9% of
the 4-Layer TinyBERT parameters, its F1-score and EM are about 1.40 and 2.50
higher, respectively. Additionally, this model has outperformed the 4-Layer
DistilBERT by 2.30 and 3.40 in F1 and EM, respectively, while using 2.5% of
the total number of parameters in DistilBERT. Our model has also achieved
around 95% performance of the 6-Layer TinyBERT and DistilBERT models, using
only 2% of their total number of parameters.

We have validated our results using the bootstrap resampling technique, a
statistical hypothesis testing method, to determine whether there is a
significant difference between the means of the two models’ predictions.
Firstly, as our sample set, 10% of the evaluation dataset, represented as $X$,
was randomly selected and fed to both models. Considering EM as our evaluation
metric, the difference between the performance of the initial model before and
after applying KD on $X$ was calculated as $\delta(X)$. To determine whether
the null hypothesis, $H_{0}:\delta(X)\leq 0$, should be rejected, we must
check whether or not $\text{{p}-value}<\alpha$, where $\alpha$ is the
significance level, and p-value is a conditional probability, based on the
null hypothesis $H_{0}$. For calculating p-value, $\delta(X)$ should be
resampled with replacement $B$ times to create numerous k-sized sets, where
$k$ is the size of $\delta(X)$. Assigning $\alpha$ to $0.05$ and $B$ to
$100000$, our calculated p-value is $0.035$ which rejects the null hypothesis
and shows the models’ performance is statistically significant.

###  5.4 Applying Active Learning

We have also applied AL to the BERTBASE model to evaluate the impact of this
technique on the volume of required labeled data and the performance of this
model. The chosen values for hyperparameters of the model are as follows. The
base model of our study was BERTBASE (uncased version), the learning rate was
set to $5\text{\times}{10}^{-4}$, and the maximum token length was set to 384.
The BERTBASE model was initially fine-tuned for only two epochs. That is
because increasing the number of epochs reduces the accuracy of the model on
the validation dataset (?). In this experiment, the Pytorch framework was used
for implementation. Initially, 1% of the training dataset was randomly chosen
for fine-tuning the BERTBASE model; the remaining 99% of the training data was
assumed to be unlabeled. Then, in each step, according to the sampling
strategies proposed in Section 4.2, in each cycle, 10% of the remaining
samples was added to the current labeled samples used for training. In each
cycle, the model was again fine-tuned on the newly compiled dataset. This
process was repeated until the model was fully trained on the whole dataset.

In Table 4, the impact of various selection strategies on the EM measure is
demonstrated. RAND denotes the random sampling strategy, LC stands for the
least confidence, EN is entropy, M denotes the margin sampling, and LC-CL is
our proposed clustering method. The results of our experiments indicate that
the performance of all the sampling methods that we have used outperform the
random sampling strategy. Moreover, among these sampling methods, the least
confidence strategy has achieved the best results. Using the least confidence
strategy and only 20% the training dataset, the model can achieve 93.83% EM of
the case in which we employ the supervised method and the whole dataset.
Additionally, the model can achieve 98.08% EM with only 40% of the training
dataset.

Dataset | RAND | LC | M | EN | LC-CL
---|---|---|---|---|---
1% | 39.83 | 39.83 | 39.83 | 39.83 | 39.83
10% | 71.98 | 72.04 | 71.32 | 71.65 | 71.74
20% | 73.83 | 76.01 | 75.61 | 75.87 | 75.43
30% | 76.07 | 77.95 | 77.58 | 77.86 | 77.87
40% | 78.42 | 79.45 | 79.08 | 79.69 | 79.50
50% | 79.16 | 79.82 | 79.04 | 80.02 | 79.70
60% | 80.00 | 80.39 | 80.01 | 80.29 | 79.91
70% | 80.27 | 81.50 | 80.55 | 81.09 | 80.77
80% | 80.34 | 81.10 | 80.95 | 81.11 | 80.95
90% | 80.93 | 81.40 | 81.07 | 81.02 | 81.02
100% | 81.00 | 81.00 | 81.00 | 81.00 | 81.00
Table 4: EM measure of different AL strategies on SQuAD v1.1 dataset. The best performance result, specified in bold, belongs to the LC strategy on 70% of the dataset, which outperforms the supervised method. RAND: random sampling stretegy. LC: least confidence. EN: entropy. M: margin sampling. LC-CL: our proposed clustering method. Dataset | RAND | LC | M | EN | LC-CL
---|---|---|---|---|---
1% | 50.64 | 50.64 | 50.64 | 50.64 | 50.64
10% | 79.26 | 81.66 | 81.01 | 80.94 | 80.43
20% | 82.81 | 84.83 | 84.31 | 84.67 | 84.40
30% | 84.73 | 86.36 | 85.83 | 85.97 | 86.01
40% | 86.51 | 87.50 | 86.79 | 87.44 | 87.21
50% | 86.97 | 87.56 | 86.96 | 87.77 | 87.23
60% | 87.69 | 88.05 | 87.75 | 87.89 | 87.88
70% | 87.92 | 88.60 | 88.00 | 88.39 | 88.12
80% | 87.96 | 88.38 | 88.13 | 88.23 | 88.20
90% | 88.12 | 88.56 | 88.25 | 88.27 | 88.24
100% | 88.34 | 88.34 | 88.34 | 88.34 | 88.34
Table 5: F1-score measure of different AL strategies on SQuAD v1.1 dataset.
The LC strategy on 70% of the dataset, specified in bold, performs better than
the supervised method. RAND: random sampling stretegy. LC: least confidence.
EN: entropy. M: margin sampling. LC-CL: our proposed clustering method.

As it is shown in Table 5, using the least confidence strategy and only 20%
and 40% of the training dataset, the model can respectively achieve 96.02% and
99.04% F1-score of the case in which we employ the supervised method and the
whole dataset. As it can be seen in Tables 4 and 5, using 70% of the training
dataset and the least confidence strategy, the model can even outperform the
supervised method by 0.50 and 0.26 in terms of the EM measure and F1-score,
respectively. We think this is because AL is effectively using more
informative samples for training and ignoring some noisy data. To the best of
our knowledge, our work is the first application of AL to the QA task.

###  5.5 Joint Application of Knowledge Distillation and Active Learning

In this section, to examine the joint application of KD and AL to a single
model, at first, 40% of the training dataset was selected by the least
confidence sampling method. Then, BERTLARGE, as the teacher model, was fine-
tuned on this training set. Next, QANet was trained on the same dataset while
its knowledge was being distilled using the teacher model.

Model | Dataset | F1 | EM
---|---|---|---
QANet (Base) (?) | 100% | 80.09 | 71.19
BERTLARGE (Teacher) (?) | 40% | 89.57 | 82.36
QANet (Base) (?) | 40% | 74.77 | 63.68
QANet+KD (Proposed) | 40% | 78.83 | 68.31
QANet+KD+Linear (Proposed) | 40% | 79.51 | 69.92
QANet+KD+Cubic (Proposed) | 40% | 79.10 | 69.47
Table 6: Applying both KD and AL on a single model trained on SQuAD v1.1
dataset. The results in bold show that our proposed model trained on 40% of
the dataset performs almost the same as the base model trained on 100% of the
dataset.

The results of this experiment demonstrated in Table 6 show the
QANet+KD+Linear model has outperformed the QANet (Base) model by 4.74 and 6.24
percentage points in F1 and EM, respectively, while trained on 40% of the
dataset. Besides, our model has achieved 99.20% F1 and 98.20% EM of the QANet
(Base) model trained on 100% of the dataset.

###  5.6 Robustness Against Adversarial Datasets

For analyzing the impact of KD on the robustness of QA models, QANet was
trained and assessed on the Adversarial SQuAD dataset before and after
applying KD.

Model | Dataset | F1 | EM
---|---|---|---
BERTLARGE (?) | 100% | 67.81 | 63.10
BERTBASE (?) | 100% | 53.79 | 48.20
QANet (Base) (?) | 100% | 41.91 | 36.20
QANet+KD (Proposed) | 100% | 44.07 | 37.90
QANet+KD+Linear (Proposed) | 100% | 45.91 | 40.10
QANet+KD+Cubic (Proposed) | 100% | 45.84 | 40.30
QANet (Base) (?) | 40% | 37.29 | 31.00
QANet+KD (Proposed) | 40% | 41.07 | 34.60
QANet+KD+Linear (Proposed) | 40% | 41.02 | 34.30
QANet+KD+Cubic (Proposed) | 40% | 41.24 | 35.00
Table 7: Performance of our proposed models trained on SQuAD v1.1 dataset and evaluated on AddSent adversarial dataset. The best results, in bold, belongs to our proposed models trained on 100% and 40% of the dataset, which demonstrates the substantial impact of KD on the robustness of the models. Model | Dataset | F1 | EM
---|---|---|---
BERTLARGE (?) | 100% | 76.92 | 71.70
BERTBASE (?) | 100% | 64.80 | 58.00
QANet (Base) (?) | 100% | 50.74 | 43.50
QANet+KD (Proposed) | 100% | 53.64 | 46.20
QANet+KD+Linear (Proposed) | 100% | 55.68 | 49.10
QANet+KD+Cubic (Proposed) | 100% | 55.90 | 49.40
QANet (Base) (?) | 40% | 46.58 | 38.30
QANet+KD (Proposed) | 40% | 50.50 | 42.40
QANet+KD+Linear (Proposed) | 40% | 50.47 | 42.40
QANet+KD+Cubic (Proposed) | 40% | 50.72 | 43.00
Table 8: Performance of our proposed models trained on SQuAD v1.1 dataset and
evaluated on AddOneSent adversarial dataset. The best experiment results, in
bold, of our models on this type of adversarial dataset exhibit the strong
impact of KD and AL least confidence strategy on the robustness of the models.

The results of our experiments in Tables 7 and 8 show that using KD increases
both F1-score and EM of the base model that is trained on 100% of SQuAD v1.1
by around 4.00 and 5.00 percentage points when it is tested on the AddSent and
AddOneSent datasets, respectively. We also evaluated the performance of the
model on the adversarial datasets when the model is equipped with both KD and
AL. The QANet+KD+Cubic model has been trained on 40% of SQuAD v1.1 and sampled
by the least confidence strategy. On the AddSent adversarial dataset, our
model has outperformed the QANet (Base) model, trained on 40% of SQuAD v1.1,
by around 4.00 percentage points in F1-score and EM. It has also achieved
98.40% F1-score of the base model that is trained on 100% of the training
dataset. The evaluation of this model on the AddOneSent adversarial dataset
shows that using only 40% of SQuAD v1.1, it can almost reach the same F1-score
and EM as the base model that is trained on the whole training dataset.

##  6 Conclusions

In this paper, we have proposed a novel combination of an interpolated KD and
AL for QA systems, which is comparable to state-of-the-art models in this
task. Our experiments showed that our model while having a fewer number of
parameters, outperformed both DistilBERT and TinyBERT with four layers and was
comparable with these models with six layers. With AL and using only 40% of
the training data, we achieved a 99.04% F1-score of the supervised model
trained on the whole dataset. Furthermore, we showed that our proposed
approach further boosts the performance of QA models by reducing both the
complexity of the model and required training data at the same time.
Additionally, by testing the model on adversarial datasets, we showed that
using KD can also increase the robustness of the model.

As our future work, one interesting direction would be to further improve the
effectiveness of KD by connecting the intermediate layers of the teacher and
student models to transfer the knowledge between those layers. Recently, pre-
trained models such as ALBERT (?), XLNet (?), and RoBERTa (?) have been
introduced that have managed to improve the performance in some downstream
tasks. It is interesting to investigate the usage of these models as the
teacher model to improve the performance in the QA task, too. Also, it may be
beneficial if a combination of multiple teacher models would be used as an
ensemble model.

## References

  * Aggarwal, Popescu, & Hudelot Aggarwal, U., Popescu, A., & Hudelot, C. (2020).  Active learning for imbalanced datasets.  In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1417–1426.
  * Amiriparian, Pugachevskiy, Cummins, Hantke, Pohjalainen, Keren, & Schuller Amiriparian, S., Pugachevskiy, S., Cummins, N., Hantke, S., Pohjalainen, J., Keren, G., & Schuller, B. (2017).  Cast a database: Rapid targeted large-scale big data acquisition via small-world modelling of social media platforms.  In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 340–345. IEEE.
  * Arora, Khapra, & Ramaswamy Arora, S., Khapra, M. M., & Ramaswamy, H. G. (2019).  On knowledge distillation from complex networks for response prediction.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3813–3822, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Asami, Masumura, Yamaguchi, Masataki, & Aono Asami, T., Masumura, R., Yamaguchi, Y., Masataki, H., & Aono, Y. (2017).  Domain adaptation of dnn acoustic models using knowledge distillation.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5185–5189.
  * Başkaya & Jurgens Başkaya, O., & Jurgens, D. (2016).  Semi-supervised learning with induced word senses for state of the art word sense disambiguation.  Journal of Artificial Intelligence Research, 55, 1025–1058.
  * Cer, Yang, Kong, Hua, Limtiaco, St. John, Constant, Guajardo-Cespedes, Yuan, Tar, Strope, & Kurzweil Cer, D., Yang, Y., Kong, S.-y., Hua, N., Limtiaco, N., St. John, R., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., Strope, B., & Kurzweil, R. (2018).  Universal sentence encoder for English.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 169–174, Brussels, Belgium. Association for Computational Linguistics.
  * Chen, Fisch, Weston, & Bordes Chen, D., Fisch, A., Weston, J., & Bordes, A. (2017).  Reading Wikipedia to answer open-domain questions.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * Cheng, Wang, Zhou, & Zhang Cheng, Y., Wang, D., Zhou, P., & Zhang, T. (2020).  A survey of model compression and acceleration for deep neural networks.  ArXiv, abs/1710.09282.
  * Chung, Lee, & Glass Chung, Y.-A., Lee, H.-Y., & Glass, J. (2018).  Supervised and unsupervised transfer learning for question answering.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1585–1594, New Orleans, Louisiana. Association for Computational Linguistics.
  * Clark, Luong, Khandelwal, Manning, & Le Clark, K., Luong, M.-T., Khandelwal, U., Manning, C. D., & Le, Q. V. (2019).  BAM! born-again multi-task networks for natural language understanding.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5931–5937, Florence, Italy. Association for Computational Linguistics.
  * Devlin, Chang, Lee, & Toutanova Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019).  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dhingra, Danish, & Rajagopal Dhingra, B., Danish, D., & Rajagopal, D. (2018).  Simple and effective semi-supervised question answering.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 582–587, New Orleans, Louisiana. Association for Computational Linguistics.
  * Fu, Zhu, & Li Fu, Y., Zhu, X., & Li, B. (2013).  A survey on instance selection for active learning.  Knowledge and Information Systems, 35(2), 249–283.
  * Hahn & Choi Hahn, S., & Choi, H. (2019).  Self-knowledge distillation in natural language processing.  In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pp. 423–430, Varna, Bulgaria. INCOMA Ltd.
  * Hinton, Vinyals, & Dean Hinton, G. E., Vinyals, O., & Dean, J. (2015).  Distilling the knowledge in a neural network.  ArXiv, abs/1503.02531.
  * Hu, Peng, Huang, Qiu, Wei, & Zhou Hu, M., Peng, Y., Huang, Z., Qiu, X., Wei, F., & Zhou, M. (2018a).  Reinforced mnemonic reader for machine reading comprehension.  In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI’18, p. 4099–4106. AAAI Press.
  * Hu, Peng, Wei, Huang, Li, Yang, & Zhou Hu, M., Peng, Y., Wei, F., Huang, Z., Li, D., Yang, N., & Zhou, M. (2018b).  Attention-guided answer distillation for machine reading comprehension.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2077–2086, Brussels, Belgium. Association for Computational Linguistics.
  * Jia & Liang Jia, R., & Liang, P. (2017).  Adversarial examples for evaluating reading comprehension systems.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics.
  * Jiao, Yin, Shang, Jiang, Chen, Li, Wang, & Liu Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020).  TinyBERT: Distilling BERT for natural language understanding.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4163–4174, Online. Association for Computational Linguistics.
  * K & K.M. K, L., & K.M., A. (2019).  A survey on datasets for machine reading comprehension.  In Proceedings of International Conference on Recent Trends in Computing, Communication & Networking Technologies (ICRTCCNT).
  * Kasai, Qian, Gurajada, Li, & Popa Kasai, J., Qian, K., Gurajada, S., Li, Y., & Popa, L. (2019).  Low-resource deep entity resolution with transfer and active learning.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5851–5861, Florence, Italy. Association for Computational Linguistics.
  * Kolomiyets & Moens Kolomiyets, O., & Moens, M.-F. (2011).  A survey on question answering technology from an information retrieval perspective.  Information Sciences, 181(24), 5412–5434.
  * Laine & Aila Laine, S., & Aila, T. (2017).  Temporal ensembling for semi-supervised learning.  ArXiv, abs/1610.02242.
  * Lan, Chen, Goodman, Gimpel, Sharma, & Soricut Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2020).  Albert: A lite bert for self-supervised learning of language representations.  In International Conference on Learning Representations.
  * Lewis, Denoyer, & Riedel Lewis, P., Denoyer, L., & Riedel, S. (2019).  Unsupervised question answering by cloze translation.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4896–4910, Florence, Italy. Association for Computational Linguistics.
  * Li, Stanovsky, & Zettlemoyer Li, B. Z., Stanovsky, G., & Zettlemoyer, L. (2020).  Active learning for coreference resolution using discrete annotation.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8320–8331, Online. Association for Computational Linguistics.
  * Li, Yi, & Zhang Li, P., Yi, J., & Zhang, L. (2018).  Query-efficient black-box attack by active learning.  ArXiv, abs/1809.04913.
  * Liu, Buntine, & Haffari Liu, M., Buntine, W., & Haffari, G. (2018).  Learning to actively learn neural machine translation.  In Proceedings of the 22nd Conference on Computational Natural Language Learning, pp. 334–344, Brussels, Belgium. Association for Computational Linguistics.
  * Liu, Tu, Wang, & Xu Liu, M., Tu, Z., Wang, Z., & Xu, X. (2020).  Ltp: A new active learning strategy for bert-crf based named entity recognition.  ArXiv, abs/2001.02524.
  * Liu, He, Chen, & Gao Liu, X., He, P., Chen, W., & Gao, J. (2019a).  Improving multi-task deep neural networks via knowledge distillation for natural language understanding.  ArXiv, abs/1904.09482.
  * Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer, & Stoyanov Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019b).  Roberta: A robustly optimized bert pretraining approach.  ArXiv, abs/1907.11692.
  * Luong, Pham, & Manning Luong, T., Pham, H., & Manning, C. D. (2015).  Effective approaches to attention-based neural machine translation.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412–1421, Lisbon, Portugal. Association for Computational Linguistics.
  * Min, Seo, & Hajishirzi Min, S., Seo, M., & Hajishirzi, H. (2017).  Question answering through transfer learning from large fine-grained supervision data.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 510–517, Vancouver, Canada. Association for Computational Linguistics.
  * Min, Zhong, Socher, & Xiong Min, S., Zhong, V., Socher, R., & Xiong, C. (2018).  Efficient and robust question answering from minimal context over documents.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1725–1735, Melbourne, Australia. Association for Computational Linguistics.
  * Mirroshandel & Ghassem-Sani Mirroshandel, S. A., & Ghassem-Sani, G. (2012).  Towards unsupervised learning of temporal relations between events.  Journal of Artificial Intelligence Research, 45, 125–163.
  * Mirroshandel & Nasr Mirroshandel, S. A., & Nasr, A. (2011).  Active learning for dependency parsing using partially annotated sentences.  In Proceedings of the 12th International Conference on Parsing Technologies, pp. 140–149, Dublin, Ireland. Association for Computational Linguistics.
  * Miyato, Maeda, Koyama, & Ishii Miyato, T., Maeda, S., Koyama, M., & Ishii, S. (2019).  Virtual adversarial training: A regularization method for supervised and semi-supervised learning.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 41, 1979–1993.
  * Molla & Vicedo Molla, D., & Vicedo, J. L. (2006).  Question answering in restricted domains: An overview.  Computational Linguistics, 33(1), 41–61.
  * Oguntola, Olubeko, & Sweeney Oguntola, I., Olubeko, S., & Sweeney, C. (2018).  Slimnets: An exploration of deep model compression and acceleration.  In 2018 IEEE High Performance extreme Computing Conference (HPEC), pp. 1–6. IEEE.
  * Papernot, McDaniel, Wu, Jha, & Swami Papernot, N., McDaniel, P., Wu, X., Jha, S., & Swami, A. (2016).  Distillation as a defense to adversarial perturbations against deep neural networks.  In 2016 IEEE Symposium on Security and Privacy (SP), pp. 582–597.
  * Pennington, Socher, & Manning Pennington, J., Socher, R., & Manning, C. (2014).  GloVe: Global vectors for word representation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Rajpurkar, Zhang, Lopyrev, & Liang Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016).  SQuAD: 100,000+ questions for machine comprehension of text.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Rasmus, Valpola, Honkala, Berglund, & Raiko Rasmus, A., Valpola, H., Honkala, M., Berglund, M., & Raiko, T. (2015).  Semi-supervised learning with ladder network.  ArXiv, abs/1507.02672.
  * Sanh, Debut, Chaumond, & Wolf Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019).  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  ArXiv, abs/1910.01108.
  * Seo, Kembhavi, Farhadi, & Hajishirzi Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2017).  Bidirectional attention flow for machine comprehension.  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
  * Settles Settles, B. (2009).  Active learning literature survey.  Computer sciences technical report 1648, University of Wisconsin–Madison.
  * Shen, Yun, Lipton, Kronrod, & Anandkumar Shen, Y., Yun, H., Lipton, Z., Kronrod, Y., & Anandkumar, A. (2017).  Deep active learning for named entity recognition.  In Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 252–256, Vancouver, Canada. Association for Computational Linguistics.
  * Sun, Cheng, Gan, & Liu Sun, S., Cheng, Y., Gan, Z., & Liu, J. (2019).  Patient knowledge distillation for BERT model compression.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 4322–4331. Association for Computational Linguistics.
  * Tarvainen & Valpola Tarvainen, A., & Valpola, H. (2017).  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In NIPS.
  * Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, & Polosukhin Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017).  Attention is all you need.  In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., & Garnett, R. (Eds.), Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc.
  * Yang, Shou, Gong, Lin, & Jiang Yang, Z., Shou, L., Gong, M., Lin, W., & Jiang, D. (2019a).  Model compression with multi-task knowledge distillation for web-scale question answering system.  ArXiv, abs/1904.09636.
  * Yang, Dai, Yang, Carbonell, Salakhutdinov, & Le Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019b).  Xlnet: Generalized autoregressive pretraining for language understanding.  In NeurIPS.
  * Yang, Hu, Salakhutdinov, & Cohen Yang, Z., Hu, J., Salakhutdinov, R., & Cohen, W. (2017).  Semi-supervised QA with generative domain-adaptive nets.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1040–1050, Vancouver, Canada. Association for Computational Linguistics.
  * Yu, Dohan, Luong, Zhao, Chen, Norouzi, & Le Yu, A. W., Dohan, D., Luong, M.-T., Zhao, R., Chen, K., Norouzi, M., & Le, Q. V. (2018).  Qanet: Combining local convolution with global self-attention for reading comprehension.  ArXiv, abs/1804.09541.
  * Yu, Zhang, Hasan, Yu, Xiang, & Zhou Yu, Y., Zhang, W., Hasan, K., Yu, M., Xiang, B., & Zhou, B. (2016).  End-to-end answer chunk extraction and ranking for reading comprehension.  ArXiv, abs/1610.09996.
  * Zhang, Yang, Li, & Wang Zhang, X., Yang, A., Li, S., & Wang, Y. (2019).  Machine reading comprehension: a literature review.  ArXiv, abs/1907.01686.