# FedQAS: Privacy-aware machine reading comprehension with federated learning

Addi Ait-Mlouk
Department of Information Technology
Division of Scientific Computing
Uppsala University, Sweden
<EMAIL_ADDRESS>&Sadi Alawadi
Department of Information Technology
Division of Scientific Computing
Uppsala University, Sweden
<EMAIL_ADDRESS>&Salman Toor
Department of Information Technology
Division of Scientific Computing
Uppsala University, Sweden
Scaleout Systems, Sweden
<EMAIL_ADDRESS>&Andreas Hellander
Department of Information Technology
Division of Scientific Computing
Uppsala University, Sweden
Scaleout Systems, Sweden
<EMAIL_ADDRESS>Corresponding author.

###### Abstract

Machine reading comprehension (MRC) of text data is one important task in
Natural Language Understanding. It is a complex NLP problem with a lot of
ongoing research fueled by the release of the Stanford Question Answering
Dataset (SQuAD) and Conversational Question Answering (CoQA). It is considered
to be an effort to teach computers how to "understand" a text, and then to be
able to answer questions about it using deep learning. However, until now
large-scale training on private text data and knowledge sharing has been
missing for this NLP task. Hence, we present FedQAS, a privacy-preserving
machine reading system capable of leveraging large-scale private data without
the need to pool those datasets in a central location. The proposed approach
combines transformer models and federated learning technologies. The system is
developed using the FEDn framework and deployed as a proof-of-concept alliance
initiative. FedQAS is flexible, language-agnostic, and allows intuitive
participation and execution of local model training. In addition, we present
the architecture and implementation of the system, as well as provide a
reference evaluation based on the SQUAD dataset, to showcase how it overcomes
data privacy issues and enables knowledge sharing between alliance members in
a Federated learning setting.

_Keywords_ Machine reading comprehension $\cdot$ Natural Language
Understanding $\cdot$ Question answering $\cdot$ Data privacy $\cdot$
Federated learning $\cdot$ Transformer

##  1 Introduction

Machine reading comprehension (MRC) is a sub-field of natural language
understanding (NLU), it aims to teach machines to read and understand human
languages (text). A user can ask the machine to answer questions based on a
given paragraph or text document. Generally, MRC requires modeling complex
interactions between the context and the query in a specific domain. It could
be used in many NLP applications such as dialogue systems and search engines
as shown in Figure 1 – a Google search engine with MRC techniques can directly
return the correct answers to questions rather than a list of content and web
pages. These kinds of techniques have been based on hand-crafted rules that
need substantial human effort and resources. However, recently there has been
an explosion of various MRC benchmark datasets that leads to a variety of
models such as BiDAF (Seo et al., 2018) and other models based on BERT (Devlin
et al., 2019), RoBERTa (Zhuang et al., 2021), XLNet (Yang et al., 2019), ELMo
(Peters et al., 2018) and transformer (Vaswani et al., 2017).

Despite this rapid progress on MRC datasets and models, most of the existing
work has focused on algorithms for improving model performance. At present,
several MRC models have already surpassed human performance on many of the MRC
datasets (Rajpurkar et al., 2016a), but there is still a limit in terms of
data privacy, collaborative training, resource-consuming, and security. Hence,
there is a need of extending existing MRC models to the context of federated
learning and decentralized AI for system performance, collaborative privacy-
preserving training, and knowledge sharing by participating in collaborative
training. To address these gaps, we proposed a privacy-aware approach based on
federated learning technology to learn new global models in a geographically
distributed manner using FEDn framework (Ekmefjord et al., 2021), build more
challenging MRC models by integrating with private data generation and
labeling for local accurate training as well as incremental learning approach
to strengthening the model performances during collaborative training without
compromising the data.

The remainder of this paper is organized as follows. Section 2 surveys related
work. Section 3 details the proposed approach and architecture of FedQAS, with
an emphasis on its privacy and scalability properties. In Section 4, we
demonstrate the frameworks potential in an evaluation based on the SQUAD
dataset. Finally, section 4 concludes the work and outlines future work.

Figure 1: An example of Google search engine with machine reading
comprehension techniques.

##  2 Related work

Machine reading comprehension was proposed for the first time in 1977 by
Lehnert, who built a question answering program called the QUALM (Lehnert,
1977). In 1999, Hirschman et al. (Hirschman et al., 1999) built a reading
comprehension system using a corpus of 60 development and 60 test stories of
3rd to 6th-grade material. Because of the lack of benchmark datasets in that
period, most MRC systems were rule-based or statistical models (Riloff and
Thelen, 2000; Charniak et al., 2000). In recent years, many benchmark datasets
have been released that focus on MRC by answering questions, see Table 1.
Since these datasets were made available, there has been considerable progress
on MRC tasks. The Stanford Question Answering Dataset (SQuAD) is one of the
most well-known reading comprehension datasets, consisting of 100.000
questions posed by crowd-workers on Wikipedia articles, where the answer to
every question is a segment of text, or span, from the corresponding reading
context. Important progress based on SQUAD concerns include the attention
method (Wang et al., 2017) and Bi-Directional Attention Flow
(BiDAF)(Richardson et al., 2013) which improved considerably the question
answering performance. These two methods compute Context to Question attention
and Question to Context attention using a similarity matrix computed directly
from context and question. Authors in (Wang et al., 2018) describe a novel
hierarchical attention network for reading comprehension style question
answering, which aims to answer questions for a given narrative paragraph. In
their work, attention and fusion are conducted horizontally and vertically
across layers at different levels of granularity between question and
paragraph. In recent work in language modeling, authors in (Zhang et al.,
2020) incorporate explicit contextual semantics from pre-trained semantic role
labeling and introduce an improved language representation model, Semantics-
aware BERT (SemBERT), which is capable of explicitly absorbing contextual
semantics over a BERT backbone. Moreover, Zhuosheng et al.(Zhang et al., 2019)
propose using syntax to guide the text modeling by incorporating explicit
syntactic constraints into the attention mechanism for better linguistically
motivated word representations. Other relevant works have been proposed
including (Yamada et al., 2020; Lan et al., 2020; Zhang et al., 2021).

All these proposed approaches required a very large amount of data for
training, which is not always available in some cases, in particular when the
text data is sensitive, private (medical text, business, social media), and
very big. In this context, we here propose the use of federated learning as a
method for distributed and collaborative machine learning. Organizations
maintain and govern their data locally and participate in learning a new
global, federated model by sending only their model updates (model weights) to
a server for aggregation into the global model. Hence, all participants
(clients) can benefit from a newly trained model without exposing their data
publicly.

Our contributions in this paper can be summarized as follows: (1) We propose
federated learning models for MRC using a transformer architecture, (2) We
design and develop the FedQAS system for collaborative training, (3) we
preserve data privacy (4) we improve the local training with incremental
learning scheme and private data generation (5) Our analyses of the models
respect data privacy regulations and outperforms the baseline model on SQuAD.

Table 1: List of some existing MRC datasets Dataset | Answer Type | Domain
---|---|---
MCTest (Richardson et al., 2013) | Multiple choice | Children’s
|  | stories
CNN/Daily Mail (Hermann et al., 2015) | Spans | News
Children’s book (Hill et al., 2016) | Spans | Children’s
|  | stories
MS MARCO (Bajaj et al., 2018) | Free-form text | Web Search
NewsQA (Trischler et al., 2017) | Spans | News
SearchQA (Dunn et al., 2017) | Spans | Jeopardy
TriviaQA (Joshi et al., 2017) | Spans | Trivia
SQuAD (Rajpurkar et al., 2016b) | Spans | Wikipedia
SQuAD 2.0 (Rajpurkar et al., 2018a) | Spans, Unanswerable | Wikipedia
CoQA (Reddy et al., 2019) | Free-form text, | News, Reddit
|  | Wikipedia

##  3 Proposed approach

The overall architecture of our proposed FedQAS system is shown in Figure 2.
The main modules are private data pipeline, federated learning settings,
question answering, and incremental learning. The private data pipeline module
allows local users (clients) to process and prepare their data locally to be
used by federated learning algorithms. The federated learning module enables
multiple private clients to form an alliance to collaboratively train machine
learning/deep learning models and send parameters to the server for global
model generation (aggregation of local models). Afterward, the system allows
the client to add new data locally and train the model incrementally through a
defined number of rounds to improve the performance using incremental learning
techniques. Finally, participating clients can use the global model as a
question answering system. The system is implemented using the FEDn federated
learning framework (Ekmefjord et al., 2021) and Flask
111https://flask.palletsprojects.com/.FEDn provides a highly scalable
federated learning run-time, and Flask is used to develop interactive and
user-friendly interfaces for the different processes in the workflow. The list
of available datasets related to question answering used for the demo is
placed in the local data sources. Moreover, the developed system is scalable,
flexible, and can be expanded with new clients/data sets on-demand (without
the need to re-train the federated model).

Figure 2: Overview of the FedQAS architecture. The FedQAS approach is
organized in three main logical layers, the first one is for collaborative
privacy-preserving training, the second one is a federated question answering
manager, and the third is for incremental learning and private data
generation.

###  3.1 Data processing pipeline

Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2018b) is a
machine reading comprehension dataset, consisting of questions posed by crowd-
workers on a set of Wikipedia articles, where the answer to every question is
a segment of text, or span, from the corresponding reading passage,
alternatively the question might be unanswerable. SQuAD2.0 combines the
100.000 questions in SQuAD1.1 with over 50,000 unanswerable questions written
adversarially by crowd-workers to look similar to answerable ones. To do well
on SQuAD2.0, systems must not only answer questions when possible but also
determine when no answer is supported by the paragraph and abstain from
answering 222https://rajpurkar.github.io/SQuAD-explorer/, see Figure 3 for an
example of text, questions, and answers.

Figure 3: A paragraph from Wikipedia and three associated questions together
with their answers, taken from the SQuAD dataset.

###  3.2 Data privacy

To train a model with a high level of accuracy, machine reading comprehension
models require large datasets to ‘learn’ from, however, data might be
sensitive and private. To preserve data privacy, different anonymization
techniques have been used. The most relevant are k-anonymity (Sweeney, 2002),
l-diversity (Machanavajjhala et al., 2007), and t-closeness (Li et al., 2007).
In k-anonymity, specific columns (e.g., name, religion, sex) are removed or
altered (e.g., replacing a specific age with an age span). L-diversity and
t-closeness are extensions of k-anonymity, which are used to protect attribute
disclosure, these anonymization techniques are applied before data is shared
for training. However, with the rise of AI, this form of anonymizing personal
data is not enough to protect privacy because the data can often be reverse-
engineered using machine learning to re-identify individuals (Rocher L. and de
Montjoye YA., 2019). In question answering systems, there might be sensitive
documents, personal data that needs to be processed for MRC task, without
exposing data. To handle this issue, we propose a question answering system
based on federated learning methodology to protect data leakage and ensure
secure collaborative training. The proposed system follows a federated
learning paradigm in which participating clients are required to train their
local models and then share the gradient (model parameters) for an eventual
aggregation strategy in a central server. This approach ensures input data
privacy, enables collaborative training, low-cost training by distributing the
workload across clients instead of training a large model individually,
sharing the local learning model within the alliance (training clients)
without compromising private data, and improving local learning by using
incremental learning and local data generation pipeline.

###  3.3 Federated learning

Federated learning is an emerging technology enabling multiple parties to
jointly train machine learning models on private data. These parties could be
mobile and IoT devices (cross-device FL), or organizations (cross-silo). Data
remain locally at each party, only the parameter updates are communicated with
a server and other parties. In our system, we use FL to develop FedQAS based
on transformer architectures for Question Answering. FedQAS trains a global
model on large amounts of data from multiple geographically distributed
parties. Each party trains a local transformer model on its data (Algorithm 2)
and sends parameters to the central server for aggregation (FedAVG (McMahan et
al., 2017)) instead of the whole model. In the aggregation part, the
aggregator (running in the _combiner_ in FEDn (Ekmefjord et al., 2021))
combines parameters (Algorithm 1) and generates a single global model for each
round using federated incremental averaging (McMahan et al., 2017).

###  3.4 Incremental Learning

Incremental learning is a machine learning case in which input data is
continuously used to extend the existing model’s knowledge i.e. to further
train the model. It attempts to improve a model’s performance while adding the
fewest samples possible. In the proposed system, adding data locally by
clients is an important task to improve the local model performance first,
then propagating these improvements into the global model after new training
rounds in a privacy-preserving manner. We have engineered an intuitive process
for each local client to contribute to the adding of new samples on top of
their local data (Figure 4). The first step is to add a new data point that
will remain on the local site, this allows the user to add their private data,
questions, and correct answers. The incremental learning module will process
and transform the private data locally and generate training points to be used
in the local training. This process enables collaborative data generation
between organizations in a private way in order to strengthen data protection
and avoid unnecessary sharing within the alliance. In addition, a database
layer is used to store user queries and global model predictions as feedback
to enhance and improve the performance for further usage.

Figure 4: Federated incremental learning process. Clients can add data
continuously to extend the existing model’s knowledge locally while training a
global model, the generated data can be stored locally on the client-side.

input : $D_{i}$

output : $M(W_{t})$

1 _$W_{0}\leftarrow\emptyset$ _;

2 for  _$t\leftarrow 1$ to $r$_ do

3 _$S_{t}\leftarrow\Gamma(k,B)$_ ;

4 for  _client $k\in S_{t}$_ do

5 _In parallel_ ;

6 $W_{t+1}^{k}\leftarrow ClientUpdate(k,W_{t})$;

7 $W_{t+1}\leftarrow\sum_{k=1}^{k}\frac{n_{k}}{n}W_{t+1}^{k}$

8 $W_{t}\leftarrow(W_{t-1}+(W_{t}-W_{t-1})/t)$

return $M(W_{t})$

Algorithm 1 Incremental FedAvg algorithm. $\Gamma$: Participants, k: number of
clients, r: number of rounds, $D_{i}$: Datasets, $W_{i}$: model weights and
$M$: Global models

Output: $W_{t}$

1

2 // Run on client k

3 Function  _ClientUpdate(_$k,W_{t}$_)_:

4 $B\leftarrow(split~{}P_{k}~{}into~{}batches~{}of~{}size~{}B$)

5 for  _$local~{}epoch~{}e_{i}\in 1,\dots e$_ do

6 for  _batch b $\in$ B_ do

7 $W_{t}\leftarrow W_{t}-\eta\nabla l(W_{t},b)$

8

9 end for

10

11 end for

12 return _$W_{t}$_

13

14

Algorithm 2 Local client update, k: number of clients, B: sample size, e:
number of local epochs, and $\eta$ is the learning rate

##  4 Experiment and results

We conduct a validation experiment on the SQUAD 1.0 dataset, which contains
100.000+ question-answer pairs on 500+ articles. To ensure collaborative
training, we randomly select and split data over 5 clients with 20% for
validation dataset for all clients. We used the BERT base as an encoder to
build our model and the implementations are based on the public TensorFlow
implementation from Keras. For the fine-tuning in our task, we set the initial
learning rate to 5e-5. The batch size is set to 8. The maximum number of
epochs is set to 1. Texts are tokenized using Wordpieces (Wu et al., 2016)
with a maximum length of 384. More configuration is shown in Table 2.

Table 2: Federated training configuration Rounds | Nbr of clients | Update size | Nbr of parameters
---|---|---|---
5 | 5 | 400 MB | 109.483.776

###  4.1 Evaluation

For the evaluation, we used exact match (EM) and F1 score, the main metrics
commonly used for question answering systems. These metrics are computed on
individual (question, answer) pairs. In case of multiple correct answers for a
given question, the maximum score over all possible correct answers is
computed. In the EM metric, For each pair (question, answer), if the
characters of the model’s prediction exactly match the characters of (one of)
the True Answer(s), EM = 1, otherwise EM = 0.

The Accuracy represents the percentage of the questions that an MRC system
accurately answers. Each question corresponds to one correct answer. For the
span prediction task, the accuracy is the same as Exact Match and can be
computed by the formula 1 as follows:

$Accuracy=EM=\frac{Number~{}of~{}correct~{}answers}{Number~{}of~{}questions}$
(1)

The precision represents the percentage of token overlap between the tokens in
the correct answer and the tokens in the predicted answer, while the recall is
the percentage of tokens in a correct answer that have been correctly
predicted in a question. The True Positive (TP) denotes the same tokens
between the predicted answer and the correct answer, the False Positive (FP)
denotes the tokens which are not in the correct answer but the predicted
answer while the False Negative (FN) presents the tokens which are not in the
predicted answer but the correct answer. Precision and Recall can be computed
by the formulas 2 and 3 as follows:

$Precision=\frac{N(TP)}{N(TP)+N(FP)}$ (2)

$Recall=\frac{N(TP)}{N(TP)+N(FN)}$ (3)

The F1 score is a measure of a test’s accuracy. It is the weighted average
between precision and recall. The formula for this score is given in 4. In our
case, it’s computed over the individual words in the prediction against those
in the True Answer. The number of shared words between the prediction and the
truth is the basis of the F1 score.

$F1~{}score=2\times\frac{(Precision\times Recall)}{(Precision+Recall)}$ (4)

To demonstrate the benefit of FedQAS, we partitioned the SQUAD dataset into 5
equal chunks, so that each client has "20%" of the total dataset. We then
compare the federated scenario to centralized model training. Table 3 lists
the available metrics for different training rounds of the global model. Our
implemented model baselines show similar EM and F1 scores with the global
model during the first rounds and slightly outperform the baseline with
respect to data privacy and enabling knowledge sharing across participants.
Overall, the result shows that question-answering in federated learning
settings performs well compared to centralized settings, see Figure 5 for the
convergence of accuracy (EM) and Figure 6 for the convergence of F1.

Table 3: Comparisons with equivalent parameters on the validation set of SQuAD1.0 Model | F1 score | Accuracy (EM)
---|---|---
Baseline | 0.31 | 0.75
FedQAS | 0.33 | 0.81
Figure 5: Convergence of accuracy (Exact Match) on the SQUAD dataset with 1
combiner, 5 clients and 5 rounds Figure 6: Convergence of F1 score on SQUAD
dataset with 1 combiner, 5 clients and 5 rounds

In terms of resources, the result proves the fact that model architecture
affects client training time and combiner round time. Hence, training a large
model (±400MB) in a centralized way requires more resources than the federated
setting. For demonstration, we consider a FEDn network consisting of a single,
high-powered combiner (8 VCPU, 32GB RAM) with connected clients (8 VCPU, 32GB
RAM) instances in SSC (SNIC Science Cloud (Toor et al., 2017)) and measure the
average round time over 5 global rounds. Figure 7 shows round time for global
model training.

Figure 7: Round times for global model training (FEDn)

To gain an intuitive observation of the predictions, we give a prediction
example on SQuAD1.0 from both the baseline and federated model in Table 4,
which shows that FedQAS works better at answering the question on a given
passage. Hence, the proposed approach has contributed overall to a better
understanding of QA, preserving data privacy, and contributed to low-cost
training as well as a collaborative question answering system task using very
large models.

Table 4: Comparison of answer prediction on test data Title: Project Apollo
---
Passage: The Apollo program, also known as Project Apollo, was the third
United States human spaceflight program
carried out by the National Aeronautics and Space Administration(NASA), which
accomplished landing the
first humans on the Moon from 1969 to 1972. First conceived during Dwight D.
Eisenhower’s administration
as a three-man spacecraft to follow the one-man Project Mercury which put the
first Americans in space,
Apollo was later dedicated to President John F. Kennedy’s national goal of
landing a man on the Moon and
returning him safely to the Earth by the end of the 1960s, which he proposed
in a May 25, 1961,
address to Congress. Project Mercury was followed by the two-man Project
Gemini. The first manned flight
of Apollo was in 1968. Apollo ran from 1961 to 1972, and was supported by the
two man Gemini program
which ran concurrently with it from 1962 to 1966…
Question 1: How long did Project Apollo run?
Gold answer (human): 1961 to 1972
Google search engine answer: see Figure 1
Baseline model answer: 1961 to 1972
FedQAS answer: 1961 to 1972
Question 2: What program was created to carry out these projects and missions?
Gold answer (human): Apollo program
Baseline model answer: National Aeronautics and Space Administration
FedQAS answer: Apollo program
Question 3: What year did the first manned Apollo flight occur?
Gold answer (human): 1968
Baseline model answer: 1968
FedQAS answer: 1968
Question 4: What President is credited with the original notion of putting
Americans in space?
Gold answer (human): John F. Kennedy
Baseline model answer: John F. Kennedy
FedQAS answer: John F. Kennedy
Question 5: Who did the U.S. collaborate with on an Earth orbit mission in
1975?
Gold answer (human): Soviet Union
Baseline model answer: Soviet Union
FedQAS answer: Soviet Union
Question 6: How long did Project Apollo run?
Gold answer (human): 1962 to 1966
Baseline model answer: 1961 to 1972, and was supported by the two man Gemini
program which ran 1966
FedQAS answer: 1962 to 1966
Question 7: What program helped develop space travel techniques that Project
Apollo used?
Gold answer (human): Gemini
Baseline model answer: Gemini
FedQAS answer: Gemini
Question 8: What space station supported three manned missions in 1973-1974?
Gold answer (human): Skylab
Baseline model answer: Skylab
FedQAS answer: Skylab

###  4.2 Implementation and demo environment

Designing and developing Question Answering in a privacy-aware manner is not a
trivial task. It requires design strategies to comply with data governance and
privacy regulations. Several third-party frameworks have been proposed for
federated learning; providing open source building blocks that help to
collaborate in training machine learning models. In the present QAS
application such a framework need to provide scalability, large models
training, and production-grade features such as robustness to failure. Based
on these requirements, we chose to design and develop our proposed FedQAS
system on top of private data using the FEDn framework (Ekmefjord et al.,
2021). FEDn is an open-source, modular, and model agnostic framework for
federated machine learning. We developed interactive and user-friendly
interfaces using Flask framework 333https://flask.palletsprojects.com which
make it easy for a third party to contribute to data annotation and then
participate in training global models directly from their location site. The
proposed FedQAS has the following features:

  * •

Privacy-preserving: sharing only model parameters with a central server
(cloud) and keeping data private on client side,

  * •

Incremental learning: improving the global models by attaching more clients
and adding new data points,

  * •

Robust: robust enough to deal with natural language tasks (e.g., question
answering, chatbot, etc.) and large models in a geographically distributed
manner,

  * •

Multilingual: language agnostic, can be trained on any language,

  * •

Standalone: multiple platforms (i.e., guarantee for low disk and memory
footprint). It can be run production-grade on a standard laptop having two
cores and 2GB of RAM,

  * •

Accuracy and F1 score: achieve competitive performance compared with
centralized training and the used baseline model (see experiment and
evaluation section).

The proposed FedQAS is composed of three main components: FEDn for
collaborative training, MongoDB (MongoDB, 2021) as a NoSQL (Pokorny, 2011)
database and question answering UI for prediction and local incremental
learning. The system is interactive, scalable, suitable for secure
collaborative training and data privacy-preserving, and can be used both in
the cloud, on edge nodes and in a standalone mode. It is accessible from
different platforms to engage a wide range of users, and it is also optimized
for both desktop and mobile. The source code is publicly available on Github
via this link https://github.com/aitmlouk/FEDn-client-FedQAS-tf.git.

To summarise, FedQAS is, to the best of our knowledge, the only approach for
question answering that supports data privacy and knowledge sharing. Its main
value is to provide an environment to quickly ensure data privacy and low-cost
training by collaborative training. Nearly every deep learning application can
benefit from data privacy and knowledge sharing across a different client in a
federated learning setting. The transformer model used in FedQAS can be
improved by tuning parameters and using transfer learning for new pre-trained
models (GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), etc.)

##  5 Conclusion

In this paper, we have proposed FedQAS, a high-quality question answering
approach, to address the data-sharing issue in machine learning. Validation
experiments for FedQAS was implemented based on 5 rounds of training with a
transformer neural network. The system consists of several components
including the private data pipeline, collaborative training and private
incremental learning. Experiments on the SQUAD dataset using the transformer
architecture demonstrate that our FedQAS significantly outperforms the
baseline model performances, protecting data privacy and sharing knowledge
within an alliance. The proposed FedQAS allows collaborators (collaborative
training participants) to have overall control of their sensitive and private
data while collaboratively training question answering models. The integration
of federated learning within machine reading comprehension provides a
sustainable solution by preserving data privacy and ensuring low-cost
training. We conclude that the application of FL to NLP tasks such as question
answering can contribute to solving the problem that arises when using machine
learning in the context of data protection and privacy. The system actively
supports end-users in joining training and improving the performance through
incremental learning on a various range of local clients. In future work, we
aim to extend the FedQAS to cover more question answering datasets and allow
federated learning for large private documents.

## Availability

FedQAS is publicly available under the Apache2 license at
https://github.com/aitmlouk/FEDn-client-FedQAS-tf.

## Acknowledgement

This research is funded by the Swedish eSSENCE strategic collaboraion on
eScience. Authors also would like to acknowledge SNIC for cloud resources.

## References

  * Seo et al. [2018] Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi.  Bidirectional attention flow for machine comprehension, 2018.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi:10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Zhuang et al. [2021] Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun.  A robustly optimized BERT pre-training approach with post-training.  In _Proceedings of the 20th Chinese National Conference on Computational Linguistics_ , pages 1218–1227, Huhhot, China, August 2021. Chinese Information Processing Society of China.  URL https://aclanthology.org/2021.ccl-1.108.
  * Yang et al. [2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf.
  * Peters et al. [2018] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi:10.18653/v1/N18-1202.  URL https://aclanthology.org/N18-1202.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2017.
  * Rajpurkar et al. [2016a] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  Squad: 100,000+ questions for machine comprehension of text, 2016a.
  * Ekmefjord et al. [2021] Morgan Ekmefjord, Addi Ait-Mlouk, Sadi Alawadi, Mattias Åkesson, Desislava Stoyanova, Ola Spjuth, Salman Toor, and Andreas Hellander.  Scalable federated machine learning with fedn, 2021.
  * Lehnert [1977] W.G. Lehnert.  _The Process of Question Answering._ PhD thesis, USA, 1977.
  * Hirschman et al. [1999] Lynette Hirschman, Marc Light, Eric Breck, and John D. Burger.  Deep read: A reading comprehension system.  In _Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics on Computational Linguistics_ , ACL ’99, page 325–332, USA, 1999. Association for Computational Linguistics.  ISBN 1558606093.  doi:10.3115/1034678.1034731.  URL https://doi.org/10.3115/1034678.1034731.
  * Riloff and Thelen [2000] Ellen Riloff and Michael Thelen.  A rule-based question answering system for reading comprehension tests.  In _Proceedings of the 2000 ANLP/NAACL Workshop on Reading Comprehension Tests as Evaluation for Computer-Based Language Understanding Sytems - Volume 6_ , ANLP/NAACL-ReadingComp ’00, page 13–19, USA, 2000. Association for Computational Linguistics.  doi:10.3115/1117595.1117598.  URL https://doi.org/10.3115/1117595.1117598.
  * Charniak et al. [2000] Eugene Charniak, Yasemin Altun, Rodrigo de Salvo Braz, Benjamin Garrett, Margaret Kosmala, Tomer Moscovich, Lixin Pang, Changhee Pyo, Ye Sun, Wei Wy, Zhongfa Yang, Shawn Zeiler, and Lisa Zorn.  Reading comprehension programs in a statistical-language-processing class.  In _ANLP-NAACL 2000 Workshop: Reading Comprehension Tests as Evaluation for Computer-Based Language Understanding Systems_ , 2000.  URL https://aclanthology.org/W00-0601.
  * Wang et al. [2017] Zhiguo Wang, Wael Hamza, and Radu Florian.  Bilateral multi-perspective matching for natural language sentences.  In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17_ , pages 4144–4150, 2017.  doi:10.24963/ijcai.2017/579.  URL https://doi.org/10.24963/ijcai.2017/579.
  * Richardson et al. [2013] Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw.  MCTest: A challenge dataset for the open-domain machine comprehension of text.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 193–203, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/D13-1020.
  * Wang et al. [2018] Wei Wang, Ming Yan, and Chen Wu.  Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1705–1714, Melbourne, Australia, jul 2018. Association for Computational Linguistics.  doi:10.18653/v1/P18-1158.  URL https://www.aclweb.org/anthology/P18-1158.
  * Zhang et al. [2020] Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou.  Semantics-aware bert for language understanding, 2020.
  * Zhang et al. [2019] Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang.  Sg-net: Syntax-guided machine reading comprehension, 2019.
  * Yamada et al. [2020] Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto.  Luke: Deep contextualized entity representations with entity-aware self-attention, 2020.
  * Lan et al. [2020] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations, 2020.
  * Zhang et al. [2021] Zhuosheng Zhang, J. Yang, and Hai Zhao.  Retrospective reader for machine reading comprehension.  In _AAAI_ , 2021.
  * Hermann et al. [2015] Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom.  Teaching machines to read and comprehend, 2015.
  * Hill et al. [2016] Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston.  The goldilocks principle: Reading children’s books with explicit memory representations, 2016.
  * Bajaj et al. [2018] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang.  Ms marco: A human generated machine reading comprehension dataset, 2018\.
  * Trischler et al. [2017] Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman.  NewsQA: A machine comprehension dataset.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_ , pages 191–200, Vancouver, Canada, August 2017. Association for Computational Linguistics.  doi:10.18653/v1/W17-2623.  URL https://www.aclweb.org/anthology/W17-2623.
  * Dunn et al. [2017] Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Guney, Volkan Cirik, and Kyunghyun Cho.  Searchqa: A new qa dataset augmented with context from a search engine, 2017.
  * Joshi et al. [2017] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017.
  * Rajpurkar et al. [2016b] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas, November 2016b. Association for Computational Linguistics.  doi:10.18653/v1/D16-1264.  URL https://www.aclweb.org/anthology/D16-1264.
  * Rajpurkar et al. [2018a] Pranav Rajpurkar, Robin Jia, and Percy Liang.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia, July 2018a. Association for Computational Linguistics.  doi:10.18653/v1/P18-2124.  URL https://www.aclweb.org/anthology/P18-2124.
  * Reddy et al. [2019] Siva Reddy, Danqi Chen, and Christopher D. Manning.  CoQA: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266, March 2019.  doi:10.1162/tacl_a_00266.  URL https://www.aclweb.org/anthology/Q19-1016.
  * Rajpurkar et al. [2018b] Pranav Rajpurkar, Robin Jia, and Percy Liang.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia, July 2018b. Association for Computational Linguistics.  doi:10.18653/v1/P18-2124.  URL https://www.aclweb.org/anthology/P18-2124.
  * Sweeney [2002] Latanya Sweeney.  K-anonymity: A model for protecting privacy.  _Int. J. Uncertain. Fuzziness Knowl.-Based Syst._ , 10(5):557–570, oct 2002.  ISSN 0218-4885.  doi:10.1142/S0218488502001648.  URL https://doi.org/10.1142/S0218488502001648.
  * Machanavajjhala et al. [2007] Ashwin Machanavajjhala, Daniel Kifer, Johannes Gehrke, and Muthuramakrishnan Venkitasubramaniam.  L-diversity: Privacy beyond k-anonymity.  _ACM Trans. Knowl. Discov. Data_ , 1(1):3–es, mar 2007.  ISSN 1556-4681.  doi:10.1145/1217299.1217302.  URL https://doi.org/10.1145/1217299.1217302.
  * Li et al. [2007] Ninghui Li, Tiancheng Li, and Suresh Venkatasubramanian.  t-closeness: Privacy beyond k-anonymity and l-diversity.  In _2007 IEEE 23rd International Conference on Data Engineering_ , pages 106–115, 2007.  doi:10.1109/ICDE.2007.367856.
  * Rocher L. and de Montjoye YA. [2019] Hendrickx JM Rocher L. and de Montjoye YA.  Estimating the success of re-identifications in incomplete datasets using generative models.  _Nature Communications_ , 10, 2019.  ISSN 2041-1723.  doi:10.1038/s41467-019-10933-3.
  * McMahan et al. [2017] H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas.  Communication-efficient learning of deep networks from decentralized data, 2017.
  * Wu et al. [2016] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean.  Google’s neural machine translation system: Bridging the gap between human and machine translation, 2016.
  * Toor et al. [2017] Salman Toor, Mathias Lindberg, Ingemar Falman, Andreas Vallin, Olof Mohill, Pontus Freyhult, Linus Nilsson, Martin Agback, Lars Viklund, Henric Zazzik, Ola Spjuth, Marco Capuccini, Joakim Möller, Donal Murtagh, and Andreas Hellander.  Snic science cloud (ssc): A national-scale cloud infrastructure for swedish academia.  In _2017 IEEE 13th International Conference on e-Science (e-Science)_ , pages 219–227, 2017.  doi:10.1109/eScience.2017.35.
  * MongoDB [2021] MongoDB.  Mongodb, 2021.  URL https://www.mongodb.com/.
  * Pokorny [2011] Jaroslav Pokorny.  Nosql databases: A step to database scalability in web environment.  iiWAS ’11, page 278–283, New York, NY, USA, 2011. Association for Computing Machinery.  ISBN 9781450307840.  doi:10.1145/2095536.2095583.  URL https://doi.org/10.1145/2095536.2095583.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.