# “A Passage to India”: Pre-trained Word Embeddings for Indian Languages

###### Abstract

Dense word vectors or ‘word embeddings’ which encode semantic properties of
words, have now become integral to NLP tasks like Machine Translation (MT),
Question Answering (QA), Word Sense Disambiguation (WSD), and Information
Retrieval (IR). In this paper, we use various existing approaches to create
multiple word embeddings for 14 Indian languages. We place these embeddings
for all these languages, viz., Assamese, Bengali, Gujarati, Hindi, Kannada,
Konkani, Malayalam, Marathi, Nepali, Odiya, Punjabi, Sanskrit, Tamil, and
Telugu in a single repository. Relatively newer approaches that emphasize
catering to context (BERT, ELMo, etc.) have shown significant improvements,
but require a large amount of resources to generate usable models. We release
pre-trained embeddings generated using both contextual and non-contextual
approaches. We also use MUSE and XLM to train cross-lingual embeddings for all
pairs of the aforementioned languages. To show the efficacy of our embeddings,
we evaluate our embedding models on XPOS, UPOS and NER tasks for all these
languages. We release a total of 436 models using 8 different approaches. We
hope they are useful for the resource-constrained Indian language NLP. The
title of this paper refers to the famous novel “A Passage to India” by E.M.
Forster, published initially in 1924.

Keywords: word embeddings, Indian languages, monolingual embeddings, cross-
lingual embeddings, contextual embeddings

“A Passage to India”: Pre-trained Word Embeddings for Indian Languages

Kumar Saurav$\dagger$, Kumar Saunack$\dagger$, Diptesh
Kanojia$\dagger$,$\clubsuit$,$\star$, and Pushpak Bhattacharyya$\dagger$
---
$\dagger$Indian Institute of Technology Bombay, India
$\clubsuit$IITB-Monash Research Academy, India
$\star$Monash University, Australia
{krsrv, krsaunack, diptesh<EMAIL_ADDRESS>

Abstract content

##  1\. Introduction

India has a total of 22 scheduled languages with a combined total of more than
a billion speakers. Indian language content on the web is accessed by
approximately 234 million speakers across the world111Source Link. Despite the
enormous user base, Indian languages are known to be low-resource or resource-
constrained languages for NLP. Word embeddings have proven to be important
resources, as they provide a dense set of features for downstream NLP tasks
like MT, QA, IR, WSD, etc. Unlike in classical Machine Learning wherein
features have at times to be extracted in a supervised manner, embeddings can
be obtained in a completely unsupervised fashion. For Indian languages, there
are little corpora and few datasets of appreciable size available for
computational tasks. The wikimedia dumps which are used for generating pre-
trained models are insufficient. Without sufficient data, it becomes difficult
to train embeddings.

NLP tasks that benefit from these pre-trained embeddings are very diverse.
Tasks ranging from word analogy and spelling correction to more complex ones
like Question Answering [Bordes et al., 2014], Machine Translation [Artetxe et
al., 2019], and Information Retrieval [Diaz et al., 2016] have reported
improvements with the use of well-trained embeddings models. The recent trend
of transformer architecture based neural networks has inspired various
language models that help train contextualized embeddings [Devlin et al.,
2018, Peters et al., 2018, Melamud et al., 2016, Lample and Conneau, 2019].
They report significant improvements over various NLP tasks and release pre-
trained embeddings models for many languages. One of the shortcomings of the
currently available pre-trained models is the corpora size used for their
training. Almost all of these models use Wikimedia corpus to train models
which is insufficient for Indian languages as Wikipedia itself lacks
significant number of articles or text in these languages. Although there is
no cap or minimum number of documents/lines which define a usable size of a
corpus for training such models, it is generally considered that the more
input training data, the better the embedding models.

Acquiring raw corpora to be used as input training data has been a perennial
problem for NLP researchers who work with low resource languages. Given a raw
corpus, monolingual word embeddings can be trained for a given language.
Additionally, NLP tasks that rely on utilizing common linguistic properties of
more than one language need cross-lingual word embeddings, i.e., embeddings
for multiple languages projected into a common vector space. These cross-
lingual word embeddings have shown to help the task of cross-lingual
information extraction [Levy et al., 2017], False Friends and Cognate
detection [Merlo and Rodriguez, 2019], and Unsupervised Neural Machine
Translation [Artetxe et al., 2018b]. With the recent advent of contextualized
embeddings, a significant increase has been observed in the types of word
embedding models. It would be convenient if a single repository existed for
all such embedding models, especially for low-resource languages. Our work
creates such a repository for fourteen Indian languages, keeping this in mind,
by training and deploying 436 models with different training algorithms (like
word2vec, BERT, etc.) and hyperparameters as detailed further in the paper.
Our key contributions are:
(1) We acquire raw monolingual corpora for fourteen languages, including
Wikimedia dumps. (2) We train various embedding models and evaluate them. (3)
We release these embedding models and evaluation data in a single
repository222Repository Link.

The roadmap of the paper is as follows: in section 2, we discuss previous
work; section 3 discusses the corpora and our evaluation datasets; section 4
briefs on the approaches used for training our models, section 5 discusses the
resultant models and their evaluation; section 6 concludes the paper.

##  2\. Literature Survey

Word embeddings were first introduced in [Y. Bengio, 2003] when it was
realised that learning the joint probability of sequences was not feasible due
to the ‘curse of dimensionality’, i.e., at that time, the value added by an
additional dimension seemed much smaller than the overhead it added in terms
of computational time, and space. Since then, several developments have
occurred in this field. Word2Vec [Mikolov et al., 2013a] showed the way to
train word vectors. The models introduced by them established new state-of-
the-art on tasks such as Word Sense Disambiguation (WSD). GloVE [Pennington et
al., 2014] and FastText [Bojanowski et al., 2017] further improved on results
shown by ?), where GloVE used a co-occurrence matrix and FastText utilized the
sub-word information to generate word vectors. Sent2Vec [Pagliardini et al.,
2017] generates sentence vectors inspired by the same idea. Universal Sentence
Embeddings [Cer et al., 2018], on the other hand, creates sentence vectors
using two variants: transformers and DANs. Doc2Vec [Le and Mikolov, 2014]
computes a feature vector for every document in the corpus. Similarly,
Context2vec [Melamud et al., 2016] learns embedding for variable length
sentential context for target words.

The drawback of earlier models was that the representation for each word was
fixed regardless of the context in which it appeared. To alleviate this
problem, contextual word embedding models were created. ELMo [Peters et al.,
2018] used bidirectional LSTMs to improve on the previous works. Later,
BERT[Devlin et al., 2018] used the transformer architecture to establish new a
state-of-the-art across different tasks. It was able to learn deep
bidirectional context instead of just two unidirectional contexts, which
helped it outperform previous models. XLNet [Yang et al., 2019] was a further
improvement over BERT. It addressed the issues in BERT by introducing
permutation language modelling, which allowed it to surpass BERT on several
tasks.

Cross-lingual word embeddings, in contrast with monolingual word embeddings,
learn a common projection between two monolingual vector spaces. MUSE [Conneau
et al., 2017] was introduced to get cross-lingual embeddings across different
languages. VecMap[Artetxe et al., 2018a] introduced unsupervised learning for
these embeddings. BERT, which is generally used for monolingual embeddings,
can also be trained in a multilingual fashion. XLM[Lample and Conneau, 2019]
was introduced as an improvement over BERT in the cross-lingual setting.

The official repository for FastText has several pretrained word embedding for
multiple languages, including some Indian languages. The French, Hindi and
Polish word embeddings, in particular, have been evaluated on Word Analogy
datasets, which were released along with the paper. ?) release word embeddings
for the Urdu language, which is one of the Indian languages we do not cover
with this work. To evaluate the quality of embeddings, they were tested on
Urdu translations of English similarity datasets.

##  3\. Dataset and Experiment Setup

We collect pre-training data for over 14 Indian languages (from a total of 22
scheduled languages in India), including Assamese (as), Bengali (bn), Gujarati
(gu), Hindi (hi), Kannada (kn), Konkani (ko), Malayalam (ml), Marathi (mr),
Nepali (ne), Odiya (or), Punjabi (pa), Sanskrit (sa), Tamil (ta) and Telugu
(te). These languages account for more than 95% of the entire Indian
population, with the most widely spoken language, Hindi, alone contributing
43% to the
figure333https://en.wikipedia.org/wiki/List_of_languages_by_number_of_native_speakers_in_India.
Nonetheless, data that is readily available for computational purposes has
been excruciatingly limited, even for these 14 languages.

One of the major contributions of this paper is the accumulation of data in a
single repository. This dataset has been collected from various sources,
including ILCI corpora [Choudhary and Jha, 2011, Bansal et al., 2013], which
contains parallel aligned corpora (including English) with Hindi as the source
language in tourism and health domains. As a baseline dataset, we first
extract text from Wikipedia dumps444As on 15th August, 2019, and then append
the data from other sources onto it. We added the aforementioned ILCI corpus,
and then for Hindi, we add the monolingual corpus from HinMonoCorp 0.5 [Bojar
et al., 2014], increasing the corpus size by 44 million sentences. For Hindi,
Marathi, Nepali, Bengali, Tamil, and Gujarati, we add crawled corpus of film
reviews and news websites555https://github.com/goru001. For Sanskrit, we
download a raw corpus of
prose666http://sanskrit.jnu.ac.in/currentSanskritProse/ and add it to our
corpus. Further, we describe the preprocessing and tokenization of our data.

###  3.1. Preprocessing

The corpora collected is intended to be set in general domain instead of being
domain-specific, and hence we start by collecting general domain corpora via
Wikimedia dumps. We also add corpora from various crawl sources to respective
individual language corpus. All the corpora is then cleaned, with the first
step being the removal of HTML tags and links which can occur due to the
presence of crawled data. Then, foreign language sentences (including English)
are removed from each corpus, so that the final pre-training corpus contains
words from only its language. Along with foreign languages, numerals written
in any language are also removed. Once these steps are completed, paragraphs
in the corpus are split into sentences using sentence end markers such as full
stop and question mark. Following this, we also remove any special characters
which may have included punctuation marks (example - hyphens, commas etc.).

The statistics for the resulting corpus are listed in Table 1.

Language | Abbr. | Sentences | Words
---|---|---|---
Hindi | hin | 48,115,256 | 3,419,909
Bengali | ben | 1,563,137 | 707,473
Telugu | tel | 1,019,430 | 1,255,086
Tamil | tam | 881,429 | 1,407,646
Nepali | nep | 705,503 | 314,408
Sanskrit | san | 553,103 | 448,784
Marathi | mar | 519,506 | 498,475
Punjabi | pan | 503,330 | 247,835
Malayalam | mal | 493,234 | 1,325,212
Gujarati | guj | 468,024 | 182,566
Konkani | knn | 246,722 | 76,899
Oriya | ori | 112,472 | 55,312
Kannada | kan | 51,949 | 30,031
Assamese | asm | 50,470 | 29,827
Table 1: Corpus statistics for each Indic language with their ISO 639-3
abbreviations (total number of sentences and words)

(a) Performance on UPOS tagged dataset

(b) Performance on XPOS tagged dataset

Figure 1: Performance of skip-gram, CBOW, and fasttext models on POS tagging
task. Plotted graph is Accuracy vs Dimension. Legend is ”language”-”model”.
Note that FastText is the best performer in each case, and learning saturates
around 200 dimensions Figure 2: Performance of skip-gram, CBOW, and fasttext
models on NER tagged dataset

###  3.2. Experiment Setup

There is a prevailing scarcity of standardised benchmarks for testing the
efficacy of various word embedding models for resource-poor languages. We
conducted experiments across some rare standardised datasets that we could
find and created new evaluation tasks as well to test the quality of non-
contextual word embeddings. The Named Entity Recognition task, collected from
[Murthy et al., 2018], and FIRE 2014 workshop for NER, contains NER tagged
data for 5 Indian languages, namely Hindi, Tamil, Bengali, Malayalam, and
Marathi. We also use a Universal POS (UPOS), as well as an XPOS (language-
specific PoS tags) tagged dataset, available from the Universal Dependency
(UD) treebank [Nivre et al., 2016], which contains POS tagged data for 4
Indian languages, Hindi, Tamil, Telugu, and Marathi.

For the tasks of NER, UPOS tagging, XPOS tagging, we use the Flair library
[Akbik et al., 2018], which embeds our pre-trained embeddings as inputs for
training the corresponding tagging models. The tagging models provided by
Flair are vanilla BiLSTM-CRF sequence labellers. For the task of word analogy
dataset, we simply use the vector addition and subtraction operators to check
accuracy (i.e., $v(\text{France})-v(\text{Paris})+v(\text{Berlin})$ should be
close to $v(\text{Germany})$).

For contextual word embeddings, we collect the statistics provided at the end
of the pre-training phase to gauge the quality of the embeddings - perplexity
scores for ELMo, masked language model accuracy for BERT, and so on. We report
these values in Table 2.

##  4\. Models and Evaluation

Language | as | bn | gu | hi | ml | mr | kn | ko | ne | or | pa | sa | ta | te
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Perplexity | 455 | 354 | 183 | 518 | 1689 | 522 | 155368 | 325 | 253 | 975 | 145 | 399 | 781 | 82
Table 2: ELMo prerplexity scores

In this section, we briefly describe the models created using the approaches
mentioned above in the paper.

###  4.1. Word2Vec (skip-gram and CBOW)

Word2Vec embeddings [Mikolov et al., 2013b] of dimensions {50, 100, 200, 300}
for both skip-gram and CBOW architectures are created using the gensim library
[Řehůřek and Sojka, 2010] implementation of Word2Vec. Words with a frequency
less than 2 in the entire corpus are treated as unknown (out-of-vocabulary)
words. For other parameters, default settings of gensim are used. There are no
pre-trained Word2Vec word embeddings for any of the 14 languages available
publicly.

###  4.2. FastText

FastText embeddings [Bojanowski et al., 2017] of dimensions {50, 100, 200,
300} (skip-gram architecture) were created using the gensim library [Řehůřek
and Sojka, 2010] implementation of FastText. Words with a frequency less than
2 in the entire corpus are treated as unknown (out-of-vocabulary) words. For
other parameters, default settings of gensim are used. Except for Konkani and
Punjabi, the official repository for FastText provides pre-trained word
embeddings for the Indian languages. However, we have trained our word
embeddings on a much larger corpus than those used by FastText.

###  4.3. GloVe

We create GloVe embeddings [Pennington et al., 2014] of dimensions {50, 100,
200, and 300}. Words with occurrence frequency less than 2 are not included in
the library. The co-occurrence matrix is created using a symmetric window of
size 15. There are no pre-trained word embeddings for any of the 14 languages
available with the GloVE embeddings
repository777https://nlp.stanford.edu/projects/glove/. We create these models
and provide them with our repository.

###  4.4. MUSE

MUSE embeddings are cross-lingual embeddings that can be trained using the
fastText embeddings, which we had created previously. Due to resource
constraints and the fact that cross-lingual representations require a large
amount of data, we choose to train 50-dimensional embeddings for each language
pair. We train for all the language pairs (14*14) and thus produce 196 models
using this approach and provide them in our repository. The training for these
models took 2 days over 1 x 2080Ti GPU (12 GB).

###  4.5. ELMo

We train ELMo embeddings [Peters et al., 2018] of 512 dimensions. These
vectors are learned functions of the internal states of a deep bidirectional
language model (biLM). The training time for each language corpus was
approximately 1 day on a 12 GB Nvidia GeForce GTX TitanX GPU. The batch size
is reduced to 64, and the embedding model was trained on a single GPU. The
number of training tokens was set to tokens multiplied by 5. We choose this
parameter based on the assumption that each sentence contains an average of 4
tokens. There are no pre-trained word embeddings for any of the 14 languages
available on the official repository. We provide these models in our
repository.

###  4.6. BERT

We train BERT (Bidirectional Encoder Representations from Transformers)
embeddings [Devlin et al., 2018] of 300 dimensions. Since BERT can be used to
train a single multilingual model, we combine and shuffle corpora of all
languages into a single corpus and used this as the pre-training data. We use
sentence piece embeddings [Google, 2018] that we trained on the corpus with a
vocabulary size of 25000. Pre-training this model was completed in less than 1
day using 3 * 12 GB Tesla K80 GPUs. The official repository for BERT provides
a multilingual model of 102 languages, which includes all but 4 (Oriya,
Assamese, Sanskrit, Konkani) of the 14 languages. We provide a single
multilingual BERT model for all the 14 languages, including these 4 languages.

###  4.7. XLM

We train cross-lingual contextual BERT representation language model using the
XLM git repository888https://github.com/facebookresearch/XLM. We train this
model for 300-dimensional embeddings and over the standard hyperparameters as
described with their work. The corpus vocabulary size of 25000 was chosen. We
use a combined corpus of all 14 Indian languages and shuffle the sentences for
data preparation of this model. We use the monolingual model (MLM) method to
prepare data as described on their Git repository. This model also required
the Byte-pair encoding representations as input and we train them using the
standard fastBPE implementation as recommended over their Github. The training
for this model took 6 days and 23 hours over 3 x V100 GPUs (16 GB each).

###  4.8. Evaluation

We evaluate and compare the performance of FastText, Word2Vec, and GloVE
embedding models on UPOS and XPOS datasets. The results are shown in the image
1(a) and in 1(b), respectively. The performance of non-contextual word
embedding models on NER dataset is shown in image 2. The perplexity scores for
ELMo training are listed in table 2. We observe that FastText outperforms both
GloVE and Word2Vec models. For Indian languages, the performance of FastText
is also an indication of the fact that morphologically rich languages require
embedding models with sub-word enriched information. This is clearly depicted
in our evaluation.

The overall size of all the aforementioned models was very large to be hosted
on a Git repository. We host all of these embeddings in a downloadable ZIP
format each on our server, which can be accessed via the link provided above.

##  5\. Results and Discussion

We have created a comprehensive set of standard word embeddings for multiple
Indian languages. We release a total of 422 embedding models for 14 Indic
languages. The models contain 4 varying dimensions (50, 100, 200, and 300)
each of GloVE, Skipgram, CBOW, and FastText; 1 each of ELMo for every
language; a single model each of BERT and XLM of all languages. They also
consist of 182 cross-lingual word embedding models for each pair. However, due
to the differences in language properties as well as corpora sizes, the
quality of the models vary. Table 1 shows the language wise corpus statistics.
Evaluation of the models has already been presented in Section 4.8.. An
interesting point is that even though Tamil and Telugu have comparable corpora
sizes, the evaluations of their word embeddings show different results. Telugu
models consistently outperform Tamil models on all common tasks. Note that the
NER tagged dataset was not available for Telugu, so they could not be compared
on this task.

This also serves to highlight the difference between the properties of these
two languages. Even though they belong to the same language family, Dravidian,
and their dataset size is the same, their evaluations show a marked
difference. Each language has 3 non-contextual embeddings (word2vec-skipgram,
word2vec-cbow and fasttext-skipgram), and a contextual embedding (ElMo). Along
with this, we have created multilingual embeddings via BERT. For BERT pre-
training, the masked language model accuracy is 31.8% and next sentence
prediction accuracy is 67.9%. Cross-lingual embeddings, on the other hand,
have been created using XLM and MUSE.

##  6\. Conclusion and Future Work

The recent past has seen tremendous growth in NLP with ElMo, BERT and XLNet
being released in quick succession. All such advances have improved the state-
of-the-art in various tasks like NER, Question Answering, Machine Translation,
etc. However, most of these results have been presented predominantly for a
single language- English. With the potential that Indian languages computing
has, it becomes pertinent to perform research in word embeddings for local,
low-resource languages as well. In this paper, we present the work done on
creating a single repository of corpora for 14 Indian languages. We also
discuss the creation of different embedding models in detail. As for our
primary contribution, these word embedding models are being publicly released.

In the future, we aim to refine these embeddings and do a more exhaustive
evaluation over various tasks such as POS tagging for all these languages, NER
for all Indian languages, including a word analogy task. Presently evaluations
have been carried out on only a few of these tasks. Also, with newer embedding
techniques being released in quick successions, we hope to include them in our
repository. The model’s parameters can be trained further for specific tasks
or improving their performance in general. We hope that our work serves as a
stepping stone to better embeddings for low-resource Indian languages.

##  7\. Bibliographical References

## References

  * Akbik et al., 2018 Akbik, A., Blythe, D., and Vollgraf, R.  (2018).  Contextual string embeddings for sequence labeling.  In COLING 2018, 27th International Conference on Computational Linguistics, pages 1638–1649.
  * Artetxe et al., 2018a Artetxe, M., Labaka, G., and Agirre, E.  (2018a).  A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789–798.
  * Artetxe et al., 2018b Artetxe, M., Labaka, G., and Agirre, E.  (2018b).  Unsupervised statistical machine translation.  arXiv preprint arXiv:1809.01272.
  * Artetxe et al., 2019 Artetxe, M., Labaka, G., and Agirre, E.  (2019).  An effective approach to unsupervised machine translation.  arXiv preprint arXiv:1902.01313.
  * Bansal et al., 2013 Bansal, A., Banerjee, E., and Jha, G. N.  (2013).  Corpora creation for indian language technologies–the ilci project.  In the sixth Proceedings of Language Technology Conference (LTC ‘13).
  * Bojanowski et al., 2017 Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T.  (2017).  Enriching word vectors with subword information.  Transactions of the Association for Computational Linguistics, 5:135–146.
  * Bojar et al., 2014 Bojar, O., Diatka, V., Rychlỳ, P., Stranák, P., Suchomel, V., Tamchyna, A., and Zeman, D.  (2014).  Hindencorp-hindi-english and hindi-only corpus for machine translation.  In LREC, pages 3550–3555.
  * Bordes et al., 2014 Bordes, A., Chopra, S., and Weston, J.  (2014).  Question answering with subgraph embeddings.  arXiv preprint arXiv:1406.3676.
  * Cer et al., 2018 Cer, D., Yang, Y., Kong, S.-y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al.  (2018).  Universal sentence encoder.  arXiv preprint arXiv:1803.11175.
  * Choudhary and Jha, 2011 Choudhary, N. and Jha, G. N.  (2011).  Creating multilingual parallel corpora in indian languages.  In Language and Technology Conference, pages 527–537. Springer.
  * Conneau et al., 2017 Conneau, A., Lample, G., Ranzato, M., Denoyer, L., and Jégou, H.  (2017).  Word translation without parallel data.  arXiv preprint arXiv:1710.04087.
  * Devlin et al., 2018 Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  (2018).  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * Diaz et al., 2016 Diaz, F., Mitra, B., and Craswell, N.  (2016).  Query expansion with locally-trained word embeddings.  arXiv preprint arXiv:1605.07891.
  * Google, 2018 Google.  (2018).  Sentence piece embeddings.  https://github.com/google/sentencepiece.
  * Haider, 2018 Haider, S.  (2018).  Urdu word embeddings.  In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA).
  * Lample and Conneau, 2019 Lample, G. and Conneau, A.  (2019).  Cross-lingual language model pretraining.  arXiv preprint arXiv:1901.07291.
  * Le and Mikolov, 2014 Le, Q. and Mikolov, T.  (2014).  Distributed representations of sentences and documents.  In International conference on machine learning, pages 1188–1196.
  * Levy et al., 2017 Levy, O., Seo, M., Choi, E., and Zettlemoyer, L.  (2017).  Zero-shot relation extraction via reading comprehension.  arXiv preprint arXiv:1706.04115.
  * Melamud et al., 2016 Melamud, O., Goldberger, J., and Dagan, I.  (2016).  context2vec: Learning generic context embedding with bidirectional lstm.  In Proceedings of the 20th SIGNLL conference on computational natural language learning, pages 51–61.
  * Merlo and Rodriguez, 2019 Merlo, P. and Rodriguez, M. A.  (2019).  Cross-lingual word embeddings and the structure of the human bilingual lexicon.  In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 110–120.
  * Mikolov et al., 2013a Mikolov, T., Chen, K., Corrado, G., and Dean, J.  (2013a).  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781.
  * Mikolov et al., 2013b Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J.  (2013b).  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119.
  * Murthy et al., 2018 Murthy, R., Khapra, M. M., and Bhattacharyya, P.  (2018).  Improving ner tagging performance in low-resource languages via multilingual learning.  ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 18(2):9.
  * Nivre et al., 2016 Nivre, J., De Marneffe, M.-C., Ginter, F., Goldberg, Y., Hajic, J., Manning, C. D., McDonald, R., Petrov, S., Pyysalo, S., Silveira, N., et al.  (2016).  Universal dependencies v1: A multilingual treebank collection.  In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1659–1666.
  * Pagliardini et al., 2017 Pagliardini, M., Gupta, P., and Jaggi, M.  (2017).  Unsupervised learning of sentence embeddings using compositional n-gram features.  arXiv preprint arXiv:1703.02507.
  * Pennington et al., 2014 Pennington, J., Socher, R., and Manning, C.  (2014).  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  * Peters et al., 2018 Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L.  (2018).  Deep contextualized word representations.  arXiv preprint arXiv:1802.05365.
  * Řehůřek and Sojka, 2010 Řehůřek, R. and Sojka, P.  (2010).  Software Framework for Topic Modelling with Large Corpora.  In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45–50, Valletta, Malta, May. ELRA.  http://is.muni.cz/publication/884893/en.
  * Y. Bengio, 2003 Y. Bengio, R. Ducharme, P. V.  (2003).  A neural probabilistic language model.  In Journal of Machine Learning Research, pages 3:1137–1155.
  * Yang et al., 2019 Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q. V.  (2019).  Xlnet: Generalized autoregressive pretraining for language understanding.  arXiv preprint arXiv:1906.08237.


11institutetext:  University of Florence, Dept. of Mathematics and Computer
Science, Florence, Italy 22institutetext: University of Perugia, Dept. of
Mathematics and Computer Science, Perugia, Italy 33institutetext: University
of Perugia, Information Systems Division, Perugia, Italy

# Implementing a scalable and elastic computing environment based on Cloud
Containers

Damiano Perri ORCID:0000-0001-6815-6659
Marco Simonetti ORCID:0000-0003-2923-5519
Sergio Tasso ORCID:0000-0001-9174-9065
Federico Ragni and
Osvaldo Gervasi ORCID:0000-0003-4327-520X 11221122223322

###### Abstract

In this article we look at the potential of cloud containers and we provide
some guidelines for companies and organisations that are starting to look at
how to migrate their legacy infrastructure to something modern, reliable and
scalable. We propose an architecture that has an excellent relationship
between the cost of implementation and the benefits it can bring, based on the
”Pilot Light” topology. The services are reconfigured inside small docker
containers and the workload is balanced using load balancers that allow
horizontal autoscaling techniques to be exploited in the future. By generating
additional containers and utilizing the possibilities given by load balancers,
companies and network systems experts may model and calibrate infrastructures
based on the projected number of users. Containers offer the opportunity to
expand the infrastructure and increase processing capacity in a very short
time. The proposed approach results in an easily maintainable and fault-
tolerant system that could help and simplify the work in particular of small
and medium-sized organisations.

###### Keywords:

High Availability, Docker, Load Balancing, Elastic Computing, Disaster
Recovery, High Performance Computing, Public Cloud, Private Cloud, Hybrid
Cloud

##  1 Introduction

The cloud is one of the biggest technological innovations of recent years [1,
2]. In fact, it enables the creation of complex, reliable and available
technological infrastructures that provide services of various kinds: from
calculation services to storage ones, to servers for the contents’
distribution via web pages. There are various forms of cloud: for example we
can use a public cloud or set up a private cloud [3, 4, 5]. In the case of a
public cloud, hardware provided by third-party companies is used, such as
Amazon AWS, Microsoft Azure, Google Cloud, and so on. Through small initial
investments, only what is actually used is paid for, and the hardware which is
made available to the users, will be maintained by the companies.

In contrast to the public cloud, there is the private cloud. This term refers
to infrastructures that are typically corporate, providing virtualised and
reduntant hardware and highly reliable services for employees, associated
staff, or a external users who consume the content produced by the company. In
this case, the company will have the responsibility of maintaining and
configuring the hardware needed to set up and build the infrastructure. This
initial disadvantage, however, is counterbalanced by a huge strategic
advantage: the data (either sensitive either ultra-sensitive data) is
completely entrusted to the management of the company itself and does not have
to transit to third-party servers, so this solution may be preferable for
certain applications [6]. The private cloud is a type of development
architecture in which the computational resources are reserved and dedicated
for the organisation managing the system.

In the public cloud, on the other hand, the services offered by a provider use
pools of machines that also accommodate other users. There is a third
alternative, called hybrid cloud, which represents a combination of the two
architectural strategies [7]. In the hybrid cloud, part of the services are
given by a third-party provider, and some subsystems are allocated within the
corporate platform. For example, it is reasonable to imagine a situation where
the provider is demanded to maintain a certain number of virtual machines and
a certain number of disks for data storage while a second pool of hard disks
containing encrypted sensitive data or databases for permanent information
storage, is located inside the private organisation.

The Covid-19 pandemic has highlighted a feature common to many countries
around the world: many organisations and companies have inadequate
infrastructure to meet the pressing needs arising as a result of the
digitisation processes that have become extremely urgent. Our intent is to
provide a model that can speed up the technological evolution of companies and
organisations or improve their current computing infrastructure. The solution
we propose, supported by practical experimentation, allows a transition from
an obsolete infrastructure of an SME (small and medium-sized enterprise),
representing a typical case, to a system based on a cluster whose services are
deployed by docker containers, and delivered through a cluster of nodes
configured according to the best practices of an High Availability (HA)
approach. The idea sprang from a report of the Italian National Institute of
Statistics (STAT), which examined active companies with at least 10 employees,
finding out that the Information Technology (IT) infrastructure are still
inadequate in many cases to their
needs.111http://dati.istat.it/Index.aspx?DataSetCode=DCSP_ICT

##  2 Related works

The cloud is being studied and analysed by many researchers around the world
because of its strong capabilities. In the course of this decade, it is
expected that the cost-benefit ratio will continue to increase and there will
be new patterns of developments focused on the Internet of Things (IOT) [8],
and other emerging technologies. The impacts that this type of architecture
have on the environment are also studied. The data centres which are required
by the Cloud, consume large amounts of energy and it is necessary to make
accurate estimates of the pollution that will be produced in the next few
years [9]. The cloud can also be used to make sites and content available for
use in the world of education [10, 11]. A number of researchers are tackling
extremely topical and interesting subjects, e.g. techniques for implementing
and exploiting Function as a Service (FaaS) [12]. FaaS are serverless systems,
where programmers can insert their own snippets of code (Python, PHP, etc.)
and call them up via APIs. The result is that virtual machines automatically
execute the code, without the necessity of setting up a server. The cloud can
also be used to do complex computational calculations that require expensive
GPUs and CPUs to perform machine learning calculations [13, 14, 15, 16].

Docker containers are lightweight cloud technologies that are dominating among
IT solutions because they allow applications to be released faster and more
efficiently than in the case of virtual machines. The adoption of Docker
containers in dynamic, heterogenous environments and the ability to deploy and
effectively manage containers across multiple clouds and data centers has made
these technologies dominant and fundamental [17]. The improvements in terms of
increased performance and reduced overhead have made the cloud container
approach indispensable for building cloud environments that keep pace with the
demands emerging from various application domains [18, 19].

##  3 Towards scalable and reliable services

Legacy service delivery architectures are based on a single, centralised
server. The main disadvantage of this solution is its low maintainability and
the lack of fault tolerance. If a hardware problem or a hacker attack occur,
service delivery may be compromised. Monolithic architectures also suffer from
another disadvantage: they cannot effectively scale in case of peak demands.

Other types of architectures support scaling. Scaling can be of two types:
vertical or horizontal [20, 21, 22]. Vertical scaling is defined as an
operation that increases the machine hardware resources, for example, rising
the number of vCPUs or the amount of GiB of RAM. Horizontal scaling is defined
as the operation that creates replicas of the server (node) that provides the
service. The new node is identical to the original one and will help it to
respond to client requests. In the case of monolithic architectures,
horizontal scaling is not possible.

Vertical scaling, on the other hand, involves shutting down the machine and
upgrading the hardware. This is unsuitable and not acceptable by modern
standards. The problem of fault tolerance must also be taken into account.

###  3.1 Disaster recovery

Legacy architectures often do not have particularly complex plans for fault
management or data loss [23, 24]. Disaster recovery (DR) is the procedure
implemented to restore the functionality of a system, suffering a disaster, a
damage: for example, the loss of system data, the loss of user data, the
compromise of security following a hacker attack or hardware damage due to a
natural disaster. When planning internal disaster recovery policies, three
objectives must be defined: the Service Level Agreement (SLA), the Recovery
Time Objective (RTO) and the Recovery Point Objective (RPO) [25, 26].

The SLA is a percentage value that indicates the minimum uptime that the
system guarantees during the year: for example, if a system has an SLA value
of 99%, it means that in one year it could be unavailable for 3d 15h 39m 29s.
An SLA of 99.99% instead could accept an annual downtime of 8h 45m 56s. The
higher the desired SLA value, the higher the costs to build an architecture to
meet our demand.

The RTO is the maximum acceptable time between the interruption of the service
and the moment when it is restored. In the case of legacy architectures the
RTO is generally about 48 hours; for example, in case of hardware problems,
spare parts must be found, repairs or replacements must be made, and the
system must be restored.

The RPO indicates the maximum acceptable time between the last data backup and
time data loss because of a disaster. This indicates how much data time, in
terms of hours, we can accept to lose. Legacy architectures often rely only on
RAID 1 storage systems (mirroring) and do not perform daily incremental and
offsite backups. Unlike the SLA, we have that the lesser RTO and RPO time is
required by the system, the higher it is the cost to achieve this requirement.

Figure 1: Disaster recovery plans

The types of disaster recovery plans[27] that can be implemented are
summarised in Figure 1.

Generally, legacy architectures implement a ”Backup and Restore” type of
architecture. This means that in case of critical problems, the only thing
that can be done is to shut down the machine (if it is still working), restore
the backup on the repaired machine or on a new machine, and restart the system
with the data updated to the last backup. The ”backup and restore” mode has an
RPO in hours and an RTO in the 24 to 48 hour range.

”Pilot Light” mode creates hourly or daily backups and maintains a complete
copy of the architecture in a separate place, away from the system. In the
event of a disaster, IT technicians will initiate backup and services will be
restored. The ”pilot light” mode guarantees an RPO in minutes and an RTO in
hours.

Alternatively, there is the ”Warm Standby” mode. In this mode, a backup system
is replicated on a different location with respect to the main system and is
synchronised in near real-time. This mode provides for a synchronisation in
seconds (RPO) and an RTO time in minutes.

On the other hand, the ”Active / Active” mode is the last possible
architecture according to the diagram shown, and it is the safest and the most
expensive from the point of view of service availability. It requires that two
or more service delivery systems are always synchronised and active at the
same time. A load balancing service is required to sort requests between the
two systems. For example, an active / active system of the type 70 / 30 could
be configured. A 70% of the requests will go to the main system and a 30% of
the requests will go to the secondary backup system.

If the main system experiences a problem, the secondary backup system will be
active and the user will not experience any particular problem. This type of
architecture provides for an RPO around milliseconds’ time, or even 0, and an
RTO potentially equal to 0. It should be noted that, as we move towards the
right-hand side of the Figure 1, the protection and ability of the system to
resist faults increases, so too the costs of implementing the architecture.

There are several problems with Legacy architectures: first, we can be exposed
to ransomware hacking attacks that can irretrievably destroy our data store;
second, there could be a software problem or a badly crafted query by a
software developer that would irrevocably wipe out the database; therefore,
other kinds of software problems can exist: minor portability issues among
operating systems.

An example of legacy architecture with no virtual machines is shown in the
Figure 2A.

In past years, it was in fact common practice to install a Linux distribution
and directly configure the application servers at the operating system level.
A further problem is related to system updates. If, for example, it is
necessary to update the PHP server in order to be able to use the new version
and the latest security improvements, this might be difficult and problematic
in the legacy case. Furthermore, it would be necessary to assess how much
downtime we can accept in order to carry out the update; but there would be no
guarantee that a rollback could be carried out quickly in the event of
incompatibility problems.

###  3.2 Legacy with Virtual Machines

A slightly better solution, falling however under legacy configurations, is to
use several Virtual Machines (VMs), one for each service-providing
application. An example is shown in the Figure 2B. In the second scenario,
using Virtual Machines, programs may be moved from one system to another,
because virtual machines are kept in one or more files readily transferred
across devices. The side effect is that they have a significant hardware
impact.

Figure 2: Legacy Architecture

Since a complete system needs to be executed, a lot of hardware (especially
RAM) resources are used even when not required by the applications. In
addition, further clock cycles are used to perform and maintain the secondary
operating system activities, which also waste energy. The installation of the
operating system also requires that virtual machines occupy a considerable
amount of disk space, each time that a new one is generated. There are
consequently two benefits of this architecture: the ability to make backups
quickly, for example by duplicating the Virtual Machine-related files and the
independence from the hardware subsystem and operating system.

##  4 The proposed architecture

In this section we describe the main techniques we used to implement a modern,
reliable and highly available system (HA). The technology behind the proposed
architecture is based on the use of Docker containers, which have the enormous
advantage of being much leaner and more efficient than a virtual machine.
Containers also make it possible to isolate an application at the highest
level, making it a completely separate entity.

Figure 3: Docker architecture

The first step is to design and configure the services and applications using
Docker containers, as described in Subsection 4.1 [28, 29]. Docker also
guarantees high security thanks to its container architecture with separate
storage spaces and access permissions [30]. In the Subsection 4.3 we explain
the way we can set up the databases in a Master/Slave configuration.

Once the services are up and running inside the containers, a distributed,
redundant network environment must be prepared using load balancers, as
described in Section 4.2. A networked, redundant and available file system
must then be set up as described in Section 4.4. The proposed architecture is
shown in the Figure 4. As it is shown, two availability zones are configured
according to the Pilot Light scheme. The two zones are connected to each other
via a VPN connection. The primary zone allows horizontal scaling thanks to the
use of the load balancer and small servers inside docker containers. The
secondary zone is on IDLE state and is kept at minimum power and CPU
consumption; services are configured but not active. Data is copied between
the two zones in an automated way. In case there is a problem and zone A
fails, the backup zone will take its place.

Figure 4: Pilot light architecture

###  4.1 Docker

Docker allows to build an architecture as shown in the Figure 3. Each service,
such as the Web Server or the Mail Server, is encapsulated within its own
container. Containers are defined by ”yml” files, and an example of them is
given in the code shown in Listing 1, where a Web Server Apache container is
defined, exposing the HTTP and HTTPS ports. Each ”yml” file may contain the
definition of one or more containers. Containers are extremely light from a
computational point of view and do not instantiate a real operating system;
the applications that run inside them only need to allocate the libraries and
binaries necessary for the application to work. Containers are stateless by
definition, which means they have no true running state. To ensure data
consistency, we should mount the folders of the filesystems where we want to
execute input/output operations in a permanent way within them.

⬇

1version ’3’

2services

3 web

4 image apache

5 container_name apache_web

6 restart always

7 ports

8 - "80:80"

9 - "443:443"

10 volumes

11 - "/home/user/myWebsite:/var/www/html/"

12 deploy

13 resources

14 limits

15 cpus ’4.0’

16 memory 2048M

17 reservations

18 memory 1768M

Listing 1: Web Server without Load Balancer

###  4.2 The load balancing service

Load balancers are fundamental to the implementation of an HA architecture.
They are customizable and can be adapted to a wide range of applications. The
standard task of a load balancer is to distribute incoming requests to a pool
of worker nodes. The nodes will process the requested information and if
necessary provide the output to the users. The requests load can be managed in
two different ways: balanced and unbalanced.

In the balanced mode each node receives a quantity of requests equal to 1/N,
with N equal to the number of nodes. This type of balancing can be implemented
in the case of an Active/Active architecture.

In unbalanced mode, percentage values can be defined to indicate the amount of
requests each node will receive. For example, a 60/40 configuration allows 60%
of requests to be sent to Node1 and 40% of requests to Node2.

Moreover, there are various request scheduling algorithms [31, 32]. The first
algorithm is called round-robin. Requests are sorted cyclically across nodes
using the round-robin algorithm. This method ensures that nodes receive an
equal amount of requests regardless of their CPU use or complexity. A second
algorithm is Least Outstanding Requests (LOR) which sorts requests across
nodes trying to balance the number of ”unprocessed requests”. In our
architecture, two load balancers are configured. The first is HAproxy placed
in a docker container. HAproxy only exposes ports 80 HTTP and 443 HTTPS [31].
Its role is to obtain requests from clients and sort them within the primary
zone, evenly distributing the workload across nodes. So, The file haporxy.cfg
must be configured.

⬇

1version ’3’

2services

3 haproxy

4 image haproxy2.3.5

5 hostname haproxy

6 ports

7 - 8080

8 - 443443

9 volumes

10 - /myHaproxy.cfg/usr/local/etc/haproxy/haproxy.cfg

11 deploy

12 placement

13 constraints [node.role == worker]

Listing 2: ”Definition of the haproxy”

The Listing 2 defines a new docker container with the image haproxy. The
container will expose the correct ports and the custom configuration file
”myHaproxy.cfg” will be mounted into the file system.

###  4.3 The Database service

A MariaDB RDBMS cluster is configured by defining ’yml’ files in the
Master/Slave configuration [33, 34]. In a ”yml” file it is in fact possible to
define more containers by adding more elements in the ”services” branch.

We have provided a single master database whose data is directly saved to the
system disk. As this is a master slave mode, it is important to enable logging
on the file system so that debugging can be carried out and configuration
errors can be analysed. The Slave database containers have a ”yml” definition
similar to that of the Master: differences only concern the names and IP
addresses. We will now describe how the load balancer for Databases is
configured. The docker image used is mariadb/maxscale [35].

Figure 5: Database: Master and Slave

First we need to publish the MariaDB MaxScale REST API, an HTTP interface,
which generates data in JSON format, offering visual management tools. MariaDB
MaxScale splits requests in such a way that write instructions are sent to the
Master container and read instructions are balanced among the Master and Slave
containers. A specific user for maxscale, with GRANT ALL privileges, must then
be defined, acting in the Master database. MariaDB MaxScale is distributed
with a BSL (Business Source License) and is capable of doing much more than
just load balancing: it also has the ability to perform failover and
switchover. The failover mode allows to monitor and operate even if one of the
nodes in the database is in an unhelpful state. If, for example, the Master
database were to crash, the Max Scale load balancer would be able to promote
the Slave database to the role of new Master. The configuration involves a few
steps: definition of the servers, creation of the monitoring service,
definition of the traffic routing using readwritesplit mode and finally the
configuration of the listner using the mariadbclient protocol and its TCP
port. The resulting configuration is shown in Figure 5.

###  4.4 Network file system

The file system must be highly available, reliable and distributed. Nodes must
be able to access the file system where they will safely store data,
regardless of where it is located within the infrastructure. To do this, we
have chosen to implement the GlusterFS network file system [36, 37]. GlusterFS
is distributed, open source and highly scalable.

The file system of the individual nodes has been configured to use XFS [38].
GlusterFS includes commands for defining the list of trustworthy servers that
comprise the trusted pool for sharing disk space for replication. They are
defined by executing the command gluster peer probe <hostname> command,
specifying the various nodes. In order to create the gluster volume, it is
necessary to specify in sequence: the volume name, the type (e.g. Replica),
and the nodes involved with their brick paths.

You need to authorise the four nodes running GlusterFS to connect to the
created volumes. To do this, we need to specify the IP addresses for each node
that we want to connect to the gluster volume. The parameter to use is
auth.allow and then start the volume. Finally, the common folder where the
data will be stored must be created, for example /var/gvolume. This folder
must be created in each node that will use it. To mount it, simply use the
specific script mount.glusterfs. In order to speed up the start-up time of the
glusterd daemon and ensure that the reboot process is automated, we recommend
automount. GlusterFS is one of the easiest persistent storage solutions to
implement, combined with the use of SSD disks, which are now widely used.

###  4.5 Scaling

Scaling is simple to implement adopting our recommended design. Our
architecture enables to expand the computational power available in the system
without requiring substantial structural modifications or shutting it down.
Assume, for example, that we expect to have five times as many users in
November as we do the rest of the year. In this situation, we might install
more nodes, for example, by adding containers containing a Web Server
instance. The HAproxy load balancer will take care of sorting requests across
nodes as described in Section4.2. Because our design involves operating in a
private cloud environment, it is possible that we may need to employ more
hardware and processors. The docker setup, on the other hand, comes to our
rescue. Since all services are described by ”yml” files, it will not be
necessary to reconfigure the machines from scratch. All is needed is to launch
new containers within the Linux distribution and tell HAproxy which new IPs
are to be used in the pool of web servers. These procedures may be carried out
without ever shutting down or disrupting the infrastructure.

##  5 Conclusion and future developments

Until recently, the term ”cluster” was associated with huge corporations and
data centres. Thanks to the Open Source software available, everyone has the
opportunity to deploy a Docker Cluster. Containers allow for the expansion of
infrastructure and the increase of computing capability that may be provided
in a relatively short period of time. Companies and network system engineers
may model and calibrate infrastructures based on the estimated number of users
by introducing additional containers and utilising the capabilities of load
balancers. Autoscaling refers to more sophisticated approaches that can
generate or delete containers based on the number of users currently present.
We have achieved long-term dependability and a low RPO and RTO time using the
Pilot Light model, which will ensure that we do not lose data and keep our
services available to consumers in the case of various problems occur (such as
hacker attacks or natural catastrophes). In the future, we want to offer pre-
configured Docker images that consumers may freely utilise. Furthermore, we
want to establish a pre-configured architecture utilising the Infrastructure
as a Code (IAAC) paradigm, which allows the entire virtual structure to be
described in a text file and then automated reproduced in the many
organisations where it may be required.

## References

  * [1] Brian Hayes  “Cloud Computing”  In _Commun. ACM_ 51.7 New York, NY, USA: Association for Computing Machinery, 2008, pp. 9–11  DOI: 10.1145/1364782.1364786
  * [2] Nick Antonopoulos and Lee Gillam  “Cloud computing”  Springer, 2010  DOI: 10.1007/978-1-84996-241-4
  * [3] Robert Basmadjian, Hermann De Meer, Ricardo Lent and Giovanni Giuliani  “Cloud computing and its interest in saving energy: the use case of a private cloud”  In _Journal of Cloud Computing: Advances, Systems and Applications_ 1.1 Springer, 2012, pp. 1–25
  * [4] Frank Doelitzscher et al.  “Private cloud for collaboration and e-Learning services: from IaaS to SaaS”  In _Computing_ 91.1 Springer, 2011, pp. 23–42
  * [5] Ang Li, Xiaowei Yang, Srikanth Kandula and Ming Zhang  “Comparing public-cloud providers”  In _IEEE Internet Computing_ 15.2 IEEE, 2011, pp. 50–53
  * [6] Kui Ren, Cong Wang and Qian Wang  “Security challenges for the public cloud”  In _IEEE Internet computing_ 16.1 IEEE, 2012, pp. 69–73
  * [7] Jin Li et al.  “A hybrid cloud approach for secure authorized deduplication”  In _IEEE Transactions on Parallel and Distributed Systems_ 26.5 IEEE, 2014, pp. 1206–1216
  * [8] Rajkumar Buyya et al.  “A Manifesto for Future Generation Cloud Computing: Research Directions for the Next Decade”  In _ACM Comput. Surv._ 51.5 New York, NY, USA: Association for Computing Machinery, 2018  DOI: 10.1145/3241737
  * [9] Sukhpal Singh Gill and Rajkumar Buyya  “A Taxonomy and Future Directions for Sustainable Cloud Computing: 360 Degree View”  In _ACM Comput. Surv._ 51.5 New York, NY, USA: Association for Computing Machinery, 2018  DOI: 10.1145/3241038
  * [10] Francesca Santucci et al.  “An Immersive Open Source Environment Using Godot”  In _Computational Science and Its Applications – ICCSA 2020_ Cham: Springer International Publishing, 2020, pp. 784–798
  * [11] Marco Simonetti, Damiano Perri, Natale Amato and Osvaldo Gervasi  “Teaching Math with the Help of Virtual Reality”  In _Computational Science and Its Applications – ICCSA 2020_ Cham: Springer International Publishing, 2020, pp. 799–809
  * [12] Vikram Sreekanti et al.  “Cloudburst: Stateful Functions-as-a-Service”  In _Proc. VLDB Endow._ 13.12 VLDB Endowment, 2020, pp. 2438–2452  DOI: 10.14778/3407790.3407836
  * [13] Giulio Biondi, Valentina Franzoni, Osvaldo Gervasi and Damiano Perri  “An Approach for Improving Automatic Mouth Emotion Recognition”  In _Computational Science and Its Applications – ICCSA 2019_ Cham: Springer International Publishing, 2019, pp. 649–664
  * [14] Damiano Perri et al.  “Binary Classification of Proteins by a Machine Learning Approach”  In _International Conference on Computational Science and Its Applications_ , 2020, pp. 549–558  Springer
  * [15] Priscilla Benedetti et al.  “Skin Cancer Classification Using Inception Network and Transfer Learning”  In _Computational Science and Its Applications – ICCSA 2020_ Cham: Springer International Publishing, 2020, pp. 536–545
  * [16] Damiano Perri et al.  “Towards a Learning-Based Performance Modeling for Accelerating Deep Neural Networks”  In _Computational Science and Its Applications – ICCSA 2019_ Cham: Springer International Publishing, 2019, pp. 665–676
  * [17] Moustafa Abdelbaky et al.  “Docker Containers across Multiple Clouds and Data Centers”  In _2015 IEEE/ACM 8th International Conference on Utility and Cloud Computing (UCC)_ , 2015, pp. 368–371  DOI: 10.1109/UCC.2015.58
  * [18] Anderson M. Maliszewski et al.  “Minimizing Communication Overheads in Container-based Clouds for HPC Applications”  In _2019 IEEE Symposium on Computers and Communications (ISCC)_ , 2019, pp. 1–6  DOI: 10.1109/ISCC47284.2019.8969716
  * [19] Weining Zhang Zhang and David Holland Holland  “Using Containers to Execute SQL Queries in a Cloud”  In _2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion)_ , 2018, pp. 26–27  DOI: 10.1109/UCC-Companion.2018.00028
  * [20] Zhenhuan Gong, Xiaohui Gu and John Wilkes  “Press: Predictive elastic resource scaling for cloud systems”  In _2010 International Conference on Network and Service Management_ , 2010, pp. 9–16  Ieee
  * [21] Luis M Vaquero, Luis Rodero-Merino and Rajkumar Buyya  “Dynamically scaling applications in the cloud”  In _ACM SIGCOMM Computer Communication Review_ 41.1 ACM New York, NY, USA, 2011, pp. 45–52
  * [22] Ming Mao, Jie Li and Marty Humphrey  “Cloud auto-scaling with deadline and budget constraints”  In _2010 11th IEEE/ACM International Conference on Grid Computing_ , 2010, pp. 41–48  IEEE
  * [23] Omar H Alhazmi and Yashwant K Malaiya  “Assessing disaster recovery alternatives: On-site, colocation or cloud”  In _2012 IEEE 23rd International Symposium on Software Reliability Engineering Workshops_ , 2012, pp. 19–20  IEEE
  * [24] Omar H Alhazmi and Yashwant K Malaiya  “Evaluating disaster recovery plans using the cloud”  In _2013 proceedings annual reliability and maintainability symposium (rams)_ , 2013, pp. 1–6  IEEE
  * [25] Victor Chang  “Towards a big data system disaster recovery in a private cloud”  In _Ad Hoc Networks_ 35 Elsevier, 2015, pp. 65–82
  * [26] Mohammad Ali Khoshkholghi, Azizol Abdullah, Rohaya Latip and Shamala Subramaniam  “Disaster recovery in cloud computing: A survey”  Citeseer, 2014
  * [27] Siham Hamadah  “Cloud-based disaster recovery and planning models: An overview”  In _ICIC Express Letters_ 13.7, 2019, pp. 593–599
  * [28] Carl Boettiger  “An introduction to Docker for reproducible research”  In _ACM SIGOPS Operating Systems Review_ 49.1 ACM New York, NY, USA, 2015, pp. 71–79
  * [29] James Turnbull  “The Docker Book: Containerization is the new virtualization”  James Turnbull, 2014
  * [30] Theo Combe, Antony Martin and Roberto Di Pietro  “To docker or not to docker: A security perspective”  In _IEEE Cloud Computing_ 3.5 IEEE, 2016, pp. 54–62
  * [31] Agung B Prasetijo, Eko D Widianto and Ersya T Hidayatullah  “Performance comparisons of web server load balancing algorithms on HAProxy and Heartbeat”  In _2016 3rd International Conference on Information Technology, Computer, and Electrical Engineering (ICITACEE)_ , 2016, pp. 393–396  IEEE
  * [32] Luthfan Hadi Pramono, Robby Cokro Buwono and Yanuar Galih Waskito  “Round-robin Algorithm in HAProxy and Nginx Load Balancing Performance Evaluation: a Review”  In _2018 International Seminar on Research of Information Technology and Intelligent Systems (ISRITI)_ , 2018, pp. 367–372  IEEE
  * [33] Daniel Bartholomew  “Getting started with MariaDB”  Packt Publishing Ltd, 2013
  * [34] William Wood  “MariaDB Solution”  In _Migrating to MariaDB_ Springer, 2019, pp. 59–71
  * [35] Mark Zaslavskiy et al.  “Full automated continuous integration and testing infrastructure for MaxScale and MariaDB”  In _2016 19th Conference of Open Innovations Association (FRUCT)_ , 2016, pp. 273–278  IEEE
  * [36] Eric B Boyer, Matthew C Broomfield and Terrell A Perrotti  “Glusterfs one storage server to rule them all”, 2012
  * [37] Manikandan Selvaganesan and Mohamed Ashiq Liazudeen  “An insight about GlusterFS and its enforcement techniques”  In _2016 International Conference on Cloud Computing Research and Innovations (ICCCRI)_ , 2016, pp. 120–127  IEEE
  * [38] Brian Pawlowski et al.  “NFS Version 3: Design and Implementation.”  In _USENIX Summer_ , 1994, pp. 137–152  Boston, MA