# Models In a Spelling Bee:
Language Models Implicitly Learn the Character Composition of Tokens

Itay Itzhak Omer Levy
The Blavatnik School of Computer Science
Tel Aviv University
<EMAIL_ADDRESS>
###### Abstract

Standard pretrained language models operate on sequences of subword tokens
without direct access to the characters that compose each token’s string
representation. We probe the embedding layer of pretrained language models and
show that models learn the internal character composition of whole word and
subword tokens to a surprising extent, without ever seeing the characters
coupled with the tokens. Our results show that the embedding layers of RoBERTa
and GPT2 each hold enough information to accurately spell up to a third of the
vocabulary and reach high character $n$gram overlap across all token types. We
further test whether enriching subword models with character information can
improve language modeling, and observe that this method has a near-identical
learning curve as training without spelling-based enrichment. Overall, our
results suggest that language modeling objectives incentivize the model to
implicitly learn some notion of spelling, and that explicitly teaching the
model how to spell does not appear to enhance its performance on such
tasks.111Our code is available at: https://github.com/itay1itzhak/SpellingBee

##  1 Introduction

Contemporary subword tokenization algorithms such as BPE (Sennrich et al.,
2016) partition a string into contiguous spans of characters. Each span
represents a frequent character $n$gram, from individual characters (a),
through prefixes (uni) and suffixes (tion), and even complete words (cats).
The tokenizer then converts each such span into a discrete symbol (a token)
with no internal structure, effectively discarding the token’s orthographic
information. Therefore, a model operating over sequences of subword tokens
should be oblivious to the spelling of each token. In this work, we show that
despite having no direct access to the subwords’ internal character
composition, pretrained language models do learn some notion of spelling.

To examine what pretrained language models learn about spelling, we present
the SpellingBee probe. SpellingBee is a generative language model that
predicts the character composition of a token given only its
(uncontextualized) vector representation from the pretrained model’s
embeddings matrix. SpellingBee is trained on part of the model’s vocabulary,
and then tested by spelling unseen token types. If the probe can successfully
reconstruct the correct character sequence from an unseen token’s embedding,
then there must be significant orthographic information encoded in the vector.

We find that the embedding layers of several pretrained language models
contain surprising amounts of character information. SpellingBee accurately
spells 31.8% of the held-out vocabulary for RoBERTa-Large (Liu et al., 2019),
32.9% for GPT2-Medium (Radford et al., 2019), and 40.9% for the Arabic
language model AraBERT-Large (Antoun et al., 2020). A softer metric that is
sensitive to partially-correct spellings (chrF) Popović (2015) shows a similar
trend, with 48.7 for RoBERTa-Large and 62.3 for AraBERT-Large. These results
are much higher than the baseline of applying SpellingBee to randomly-
initialized vectors, which fails to spell a single token.

Given that subword models learn some notion of character composition to
fulfill language modeling objectives, could they perhaps benefit from knowing
the exact spelling of each token a priori? To that end, we reverse
SpellingBee’s role and use it to pretrain the embedding layer of a randomly-
initialized model, thus imbuing each token representation with its
orthographic information before training the whole model on the masked
language modeling objective. We compare the pretraining process of the
character-infused model to that of an identical model whose embedding layer is
randomly initialized (and not pretrained), and find that both learning curves
converge to virtually identical values within the first 1,000 gradient
updates, a fraction of the total optimization process. This experiment
suggests that while language models may need to learn some notion of spelling
to optimize their objectives, they might also be able to quickly acquire most
of the character-level information they need from plain token sequences
without directly observing the composition of each token.

##  2 Spelling Bee

To measure how much a model knows the character composition of its tokens, we
introduce SpellingBee, a generative probe that tries to spell out a token
character-by-character. Specifically, SpellingBee probes the original model’s
embedding matrix, since spelling is a property of token types, invariant to
context. For example, given the embedding of the token cats, SpellingBee will
try to generate the sequence [c, a, t, s]. We do so by modeling SpellingBee as
a character-based language model,222Implemented using the transformer decoder
architecture, following standard practice in language modeling. where the
first token is a vector representation of the vocabulary item.333Some
vocabularies have symbols for indicating preceding whitespaces (_) or that the
next token is part of the same word (##). SpellingBee learns to predict these
symbols too.

### Training

We split the vocabulary to train and test sets,444We test various train/test
splits to ensure the robustness of our findings. See Section 3 for more
detail. and use teacher forcing to train SpellingBee. In the example of cats,
SpellingBee will compute the following probabilities:

$\displaystyle P(x_{1}=\text{{c}}$ $\displaystyle\mid x_{0}=\text{{cats}})$
$\displaystyle P(x_{2}=\text{{a}}$ $\displaystyle\mid
x_{0}=\text{{cats}},x_{1}=\text{{c}})$ $\displaystyle P(x_{3}=\text{{t}}$
$\displaystyle\mid x_{0}=\text{{cats}},x_{1}=\text{{c}},x_{2}=\text{{a}})$
$\displaystyle\vdots$

All of SpellingBee’s parameters are randomly initialized. The only parameters
that are pretrained are the token embeddings (e.g. the representation of cats
or a), which are taken from the original pretrained language model we intend
to probe, and treated as constants; i.e. kept frozen during SpellingBee’s
training.

### Inference & Evaluation

Once SpellingBee is trained, we apply it to the test set using greedy
decoding. For each vocabulary item $w$ in the test set, SpellingBee is given
only the corresponding embedding vector $e_{w}$, and is expected to generate
the character sequence $w_{1},\ldots,w_{n}$ that defines $w$. We measure
success on the test set using two metrics: exact match (EM), and character
$n$gram overlap score using chrF Popović (2015). While EM is strict, chrF
allows us to measure partial success. We also report edit distance using
Levenshtein distance ratio in Appendix A.

### SpellingBee for Pretraining Embeddings

While we mainly use SpellingBee as a probe, a variation of our method could
potentially imbue the embedding layer with character information before
training a language model. We could train a probe with randomly-initialized
embeddings (instead of pretrained embeddings from another model) to predict
the spelling of all vocabulary items, and use these trained probe embeddings
to initialize any target model’s embedding layer (instead of random
initialization). We experiment with this method in Section 5, but find that it
does not have any significant impact on the convergence of language models.

##  3 Experiment Setup

We begin with a series of probing experiments, where we apply SpellingBee to
the embedding layer of various pretrained models.555Hyperparameters are
detailed in Appendix E.

### Pretrained Models

We probe four pretrained models: RoBERTa-Base and Large Liu et al. (2019),
GPT2-Medium Radford et al. (2019), and AraBERT-Large Antoun et al. (2020).
This set introduces some diversity in vocabulary, objective, and scale: the
first three models are trained on English corpora, while AraBERT is trained on
text in Arabic; GPT2 is an autoregressive language model, while the rest are
masked language models; RoBERTa-Base consists of 125M parameters (with 768
dimensions per embedding), while the other models have approximately 350M
parameters (with 1024 dimensions per embedding).

### Control

Since SpellingBee is a trained probe, we wish to establish the probe’s
baseline performance when provided with inputs with no orthographic
information. As an empirical control, we train and test SpellingBee on
randomly-initialized vectors, in addition to the main experiments where we
utilize the pretrained embedding layers.

### Training & Testing Data

We split the vocabulary into training and testing data using the following
protocol. First, we randomly sample 1000 token types as test. We then filter
the remaining vocabulary to eliminate tokens that may be too similar to the
test tokens, and randomly sample 32000 training examples.We experiment with
three filters: none, which do not remove tokens beyond the test-set tokens;
similarity, which removes the top 20 most similar tokens for every token in
test, according to the cosine similarity induced by the embedding vectors;
lemma, which removes any token type that shares a lemma with a test-set token
(e.g. if diving is in the test set, then diver cannot be in the training
set).666We lemmatize using NLTK’s WordNet lemmatizer Loper and Bird (2002) for
English and Farasa’s Stemmer Darwish and Mubarak (2016) for Arabic. The lemma
filter always applies the similarity filter first, providing an even more
adversarial approach for splitting the data. To control for variance, we
create 10 such splits for each model and filter, and report the averaged
evaluation metrics over all 10 test sets.

| Filter | RoBERTa | GPT2 | AraBERT
---|---|---|---|---
| Base | Large | Medium | Large
EM | None | 27.3 | 31.8 | 32.9 | 40.9
Similarity | 15.7 | 18.2 | 17.9 | 21.9
Lemma | 15.7 | 17.7 | 16.5 | 19.5
Control |  0.0 |  0.0 |  0.0 |  0.0
chrF | None | 44.7 | 48.7 | 51.6 | 62.3
Similarity | 32.7 | 35.1 | 36.4 | 46.0
Lemma | 32.6 | 34.8 | 35.2 | 43.9
Control |  7.0 |  7.0 |  7.0 |  7.0
Table 1: The percent of token types that can be spelled out exactly (EM) from
their embeddings by SpellingBee, and the $n$gram overlap between SpellingBee’s
reproductions and the token types’ true spellings (chrF). The first three rows
reflect different methods for filtering the training data, and the fourth
represents the control experiment, which uses randomly initialized embeddings.
All SpellingBee instances in this table are trained on 32,000 examples.

##  4 Results

### Main Result

Table 1 shows how well SpellingBee can spell a vocabulary token using only its
frozen pretrained embedding. We observe that SpellingBee is able to accurately
recover the spelling of up to 40.9% of the test set, while the control is
unable to spell even a single word correctly. A similar trend can be seen when
considering the finer character $n$gram metric (chrF). Manually analyzing the
predictions of the control baselines (see Appendix D) indicate that it
primarily generates combinations of frequent character sequences, which mildly
contributes to the chrF score, but does not affect EM. These results are
persistent across different models and filters, strongly indicating that the
embedding layer of pretrained models contains significant amounts of
information about each token’s character composition.

One may suggest that training SpellingBee over 32,000 examples may leak
information from the test set; for example, if dog was seen during training,
then spelling out dogs might be easy. We thus consider the similarity and
lemma filters, which remove such near-neighbors from the training set. While
results are indeed lower (and probably do account for some level of
information leakage), they are still considerably higher than the control,
both in terms of EM and chrF. Results using the similarity and lemma filters
are rather similar, suggesting that embedding-space similarity captures some
information about each token’s lemma.

Finally, we find that the properties of pretrained models also seem to have a
significant effect on the amount of spelling information SpellingBee can
extract. Larger models tend to score higher in the probe, and the model
trained on text in Arabic appears to have substantially higher EM and chrF
scores than those trained on English corpora. One possibility is that Arabic’s
rich morphology incentivizes the model to store more information about each
token’s character composition; however, it is also possible that AraBERT’s
different vocabulary, which allocates shorter character sequences to each
token type, might explain this difference (we discuss the link between
sequence length and accuracy in Appendix C).

Overall, our probing experiments show that even though subword-based language
models do not have direct access to spelling, they can and do learn a
surprising amount of information about the character composition of each
vocabulary token.

| Filter | RoBERTa | CharacterBERT | GloVe
---|---|---|---|---
| Base | Base | 300D
EM | None | 43.0 | 28.2 | 2.0
Similarity | 9.6 | 12.9 | 1.6
Lemma | 9.9 | 12.9 | 1.6
Control | 0.0 | 0.0 | 0.0
chrF | None | 58.8 | 53.3 | 13.6
Similarity | 27.0 | 37.5 | 13.2
Lemma | 27.3 | 37.5 | 13.0
Control | 7.9 | 8.0 | 8.0
Table 2: The percent of whole words that can be spelled out exactly (EM) from
their embeddings by SpellingBee, and the $n$gram overlap between SpellingBee’s
reproductions and the token types’ true spellings (chrF). All SpellingBee
instances in this table are trained on 32,000 examples of whole words.

### Character-Aware Models

Some models are provided with the raw character sequence of each token. To
test whether the embedding layers of such models are indeed more informed
about each token’s spelling, we apply SpellingBee to CharacterBERT El
Boukkouri et al. (2020), a BERT-style model whose layer-zero word embeddings
are derived from a character CNN, following ELMo Peters et al. (2018).

Table 2 shows that the spelling-aware embeddings of CharacterBERT score higher
on the SpellingBee probe when the similarity and lemma filters are applied.
However, when no filter is applied, RoBERTa’s character-oblivious but highly-
tuned training process produces embeddings that score higher on SpellingBee,
presumably by leveraging implicit similarity functions in the embedding space.

Although CharacterBERT’s embedding layer is better at reconstructing original
words (when similarity filters are applied), this does not mean that
character-aware models are necessarily better downstream. El Boukkouri et al.
(2020) report performance increases only on the medical domain. In Section 5,
we demonstrate that initializing a masked language model’s embedding layer
with character information has a negligible effect on its perplexity.

### Context-Oblivious Models

The first generation of neural word representations Mikolov et al. (2013a, b)
contained only embedding layers, without any contextualization mechanism. We
thus use GloVe Pennington et al. (2014) to estimate a lower bound on character
information that can be obtained by simple context-oblivious models. We probe
the first 50K words in GloVe’s vocabulary with SpellingBee. Table 2 shows that
GloVe embeddings do contain a weak orthographic signal, better than random
embeddings, but substantially weaker than the information stored in the
embedding layer of large transformer-based language models.

### Probing with Less Training Data

We further examine whether SpellingBee can extract information when trained on
less examples. Figure 1 shows how well SpellingBee can spell RoBERTa-Large’s
vocabulary when trained on varying amounts of data, across all filters. We
find that more data makes for a better probe, but that even a few thousand
examples are enough to train SpellingBee to extract significant character
information from the embeddings, which cannot be extracted from randomized
vectors (the control).777We provide additional analysis on spelling accuracy
by subword frequency and length in Appendices B and C.

Figure 1: The amount of character information SpellingBee is able to extract
from RoBERTa-Large, as measured by EM (top) and chrF (bottom), given different
quantities of training examples.

Figure 2: The overall training loss (left), first steps of training loss
(center), and validation loss (right) of RoBERTa-Large, when training on the
masked language modeling objective with embeddings pretrained by SpellingBee
(pretrained) and randomly-initialized embeddings (control).

##  5 Pretraining Language Models to Spell

Our probing experiments reveal that language models learn some partial notion
of spelling, despite the lack of direct access to characters. Therefore, we
hypothesize that learning to spell is beneficial for language models, and
propose pretraining the embedding layer using a variant of the SpellingBee
probe described in Section 2. Here, the goal is to imbue each embedding with
enough information for SpellingBee to accurately generate its surface form,
and then initialize the language model with the pretrained embeddings before
it starts training on the language modeling objective.

We apply this process to RoBERTa-Large, training the model’s embedding layer
with SpellingBee using the same hyperparameter settings from Appendix E, with
the key difference being that the embeddings are now tunable parameters (not
frozen).888To verify that this process does indeed encode the tokens’
spellings into the embeddings, we apply a SpellingBee probe (using a different
random initialization) to the learned embeddings, which yields 93.5% EM on
held-out token types. We train RoBERTa-Large on English Wikipedia using the
hyperparameter configuration of 24hBERT Izsak et al. (2021), and cease
training after 24 hours (approximately 16,000 steps). For comparison, we train
exactly the same model with a randomly-initialized embedding layer.

Figure 2 shows the masked language modeling loss with and without pretrained
embeddings. We see that the curves quickly converge into one. After only 1000
training steps, the difference between the validation losses never exceeds
0.01. This result indicates that in this scenario, the model does not utilize
the character information injected into the tokens’ embeddings.

Although there are many possible ways to explicitly add orthographic
information to tokens embeddings, our method is relatively straightforward as
it gives the model a chance to utilize pre-stored character information. Along
with the results from Section 4, we hypothesize that the implicit notion of
spelling that the model learns during pretraining might be sufficient for
masked language modeling.

##  6 Conclusion

This work reveals that pretrained language models learn, to some extent, the
character composition of subword tokens. We show that our SpellingBee probe
can spell many vocabulary items using their uncontextualized embedding-layer
representations alone. Trying to explicitly infuse character information into
the model appears to have a minimal effect on the model’s ability to optimize
its language modeling objective, suggesting that the model can independently
learn all the character-level information it needs for the task.

## Acknowledgements

This work was supported by the Tel Aviv University Data Science Center, Len
Blavatnik and the Blavatnik Family foundation, the Alon Scholarship, Intel
Corporation, and the Yandex Initiative for Machine Learning. We thank Avia
Efrat for his valuable feedback.

## References

  * Antoun et al. (2020) Wissam Antoun, Fady Baly, and Hazem Hajj. 2020.  AraBERT: Transformer-based model for Arabic language understanding.  In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , pages 9–15, Marseille, France. European Language Resource Association.
  * Darwish and Mubarak (2016) Kareem Darwish and Hamdy Mubarak. 2016.  Farasa: A new fast and accurate Arabic word segmenter.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 1070–1074, Portorož, Slovenia. European Language Resources Association (ELRA).
  * El Boukkouri et al. (2020) Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun’ichi Tsujii. 2020.  CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6903–6915, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Izsak et al. (2021) Peter Izsak, Moshe Berchansky, and Omer Levy. 2021.  How to train bert with an academic budget.  _arXiv preprint arXiv:2104.07705_.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980.
  * Levenshtein et al. (1966) Vladimir I Levenshtein et al. 1966.  Binary codes capable of correcting deletions, insertions, and reversals.  In _Soviet physics doklady_ , volume 10, pages 707–710. Soviet Union.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loper and Bird (2002) Edward Loper and Steven Bird. 2002.  NLTK: The natural language toolkit.  In _Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics_ , pages 63–70, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b.  Distributed representations of words and phrases and their compositionality.  In _NIPS_.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.

##  Appendix A Levenshtein Distance

Levenshtein distance Levenshtein et al. (1966) is an edit distance metric
that, given two strings, calculates the minimal number of changes needed to be
done in order to make the two strings identical. Levenshtein distance ratio is
the length-normalized version, which is computed by adding the sum of lengths
of both strings to the edit distance and dividing by the same sum of lengths.
We report the main experiment’s results using this ratio in Table 3.

Filter | RoBERTa | GPT2 | AraBERT
---|---|---|---
Base | Large | Medium | Large
None | 69.7 | 72.7 | 74.4 | 83.6
Similarity | 61.5 | 63.7 | 64.5 | 75.8
Lemma | 61.4 | 63.3 | 63.7 | 74.8
Control |  25.6 |  26.4 |  27.0 |  25.7
Table 3: Levenshtein distance ratio. The first three rows reflect different
methods for filtering the training data, and the fourth represents the control
experiment, which uses randomly initialized embeddings. All SpellingBee
instances in this table are trained on 32000 examples.

##  Appendix B Spelling Accuracy by Frequency

We test whether pretrained models tend to store more spelling-related
information in higher-frequency token types. We focus on RoBERTa-Large, and
assign each token in the test set to its frequency quintile according to the
number of times it appeared in the pretraining corpus – from the 10000 most
frequent token types (top 20%) to those ranked 40000-50000 in the vocabulary
(bottom 20%) – and measure the average performance of SpellingBee within each
quintile. Figures 3 and 4 shows the results with and without the similarity
filter. We observe that SpellingBee is indeed able to extract more information
from higher-frequency token types, suggesting that the pretrained model has
more information about their character composition.

Figure 3: The EM scores of SpellingBee on RoBERTa-Large for each frequency
quintile with the none filter (top) and the similarity filter (bottom).
Figure 4: The chrF scores of SpellingBee on RoBERTa-Large for each frequency
quintile with the none filter (top) and the similarity filter (bottom).

##  Appendix C Spelling Accuracy by Length

We analyze the effect of token length on the probe’s ability to spell. A
priori, it is reasonable to assume that it is easier for the probe to spell
shorter tokens, since less information needs to be extracted from the
embedding and there are less discrete decisions to be made while decoding.
Indeed, Figure 5 shows that with the none filter most vocabulary tokens with
2-4 characters can be accurately reproduced from their vector representations,
while longer tokens are harder to replicate. This trend is particularly sharp
when the similarity filter is applied, as the probe is hardly able to spell
tokens with 6 or more characters accurately; having said that, the probe is
able to generate many partially correct spellings, as measured by chrF (Figure
6). Perhaps a less intuitive result is the probe’s failure to spell single-
character tokens. A closer look reveals that many of these examples are rare
or non-alphanumeric characters (e.g. ç and $), which are probably very
difficult for the probe to generate if it had not seen them during training.
While these results show strong trends with respect to length, token length is
also highly correlated with frequency, and it is not necessarily clear which
of the two factors has a stronger impact on the amount and resolution of
character-level information stored in the embedding layer of pretrained
models.

Figure 5: The EM scores of SpellingBee on RoBERTa-Large for each token length
with the none filter (top) and the similarity filter (bottom). The rightmost
column groups together tokens with length of 11 or above. Figure 6: The chrF
scores of SpellingBee on RoBERTa-Large for each token length with the none
filter (top) and the similarity filter (bottom). The rightmost column groups
together tokens with length of 11 or above.

##  Appendix D Manual Error Analysis

We manually analyze 100 random tokens that SpellingBee spelled incorrectly
with the lemma filter to understand the nature of the spelling mistakes. Out
of those 100 we display 20 mistakes in Table 4 alongside the spelling
prediction of the control baseline. SpellingBee’s mistakes vary from single-
character typos to completely different words. Having said that, the vast
majority of mistakes have significant overlap with the correct spelling, such
as shared prefixes and capitalization.

Token | SpellingBee | Control
---|---|---
_Issa | _Asey | _kinston
_Rhod | _Rob | _hoedn
Memory | Mathinge | _entically
_metals | _metrys | _leaved
_Reed | _Redd | _fomparing
_break | _breach | _promoters
_summit | _mosump | _seasons
Catholic | Cravital | _tonversal
_cleanup | _lamed | _paclus
_Winner | _Womer | _purden
_LIM | _LUM | _Send
Copy | Cople | _providers
_voicing | _relicing | _walking
_Stab | _Stamb | _hoviders
_356 | _353 | _budiance
find | wive | _malding
_Psychic | _Syptanc | _joacter
_Looking | _Lowing | parging
CLOSE | DEFIC | _tuldence
_prolific | _promistic | _complexement
Table 4: Sampled SpellingBee errors with the lemma filter alongside the
control baseline’s spelling for the same tokens. The underscore (_) represents
a preceding whitespace.

##  Appendix E Hyperparameters

We implement SpellingBee with a 6-layer encoder-decoder model, with 512 model
dimensions. The model parameters are optimized with Adam Kingma and Ba (2015)
for 1000 steps with up to 1024 tokens per batch, a learning rate of 5e-4, and
a dropout rate of 0.1. These are the default hyperparameters for training a
transformer language model in Fairseq Ott et al. (2019).


# Performance of Distributed File Systems on Cloud Computing Environment: An
Evaluation for Small-File Problem

Thanh Duong  John von Neumann Institute
Vietnam National University at HCMC
E-mail<EMAIL_ADDRESS>Quoc Luu  John von Neumann Institute
Vietnam National University at HCMC
E-mail<EMAIL_ADDRESS>Hung Nguyen  Faculty of Economics and
Management
Thai Binh Duong University
E-mail<EMAIL_ADDRESS>
###### Abstract

Various performance characteristics of distributed file systems have been well
studied. However, the performance efficiency of distributed file systems on
small-file problems with complex machine learning algorithms scenarios is not
well addressed. In addition, demands for unified storage of big data
processing and high-performance computing have been crucial. Hence, developing
a solution combining high-performance computing and big data with shared
storage is very important. This paper focuses on the performance efficiency of
distributed file systems with small-file datasets. We propose an architecture
combining both high-performance computing and big data with shared storage and
perform a series of experiments to investigate the performance of these
distributed file systems. The result of the experiments confirms the
applicability of the proposed architecture in terms of complex machine
learning algorithms.

###### Index Terms:

Hadoop, HDFS, distributed file system, Lustre, Spark

††publicationid: pubid: 978-1-5090-4134-3/16/$31.00 © 2016 IEEE

##  I Introduction

From creating big models in machine learning to answering big questions in
data processing, big data has been widely adopted and growing as a promising
technology for enterprises to develop and improve their businesses
efficiently. In research disciplines, along with high-performance computing,
big data also plays a critical role in data processing [1, 2, 3]. However,
high-performance computing mainly uses parallel distributed file systems for
I/O and very limited local storage, whereas big data processing relies on
Hadoop distributed file system (HDFS) and local storage [4]. In addition to
storage issues, the output of scientific applications is often very large and
would take a long time to transfer to external big data processing clusters
(i.e. Hadoop system). Furthermore, the principle “write-once, read-many” of
HDFS design does not meet high-performance computing requirements [5]. Thus,
it requires an approach to provide a unified storage solution.

Hadoop distributed file system (HDFS) is an open-source distributed storage
platform that implements a data processing framework called MapReduce on top
of it. Many companies (e.g., AOL, Yahoo!, Facebook) use HDFS to manage and
store petabyte or exabyte-scale enterprise data on clusters due to its
reliability and scalability [4, 6]. A fundamental philosophy of Hadoop system
design is that “moving computation is cheaper than moving data” to leverage
local storage (direct attached storage - DAS) and reduce network transfer [7].
In other words, it means that Hadoop achieves more efficiency locally
executing computations on nodes storing the data involved rather than moving
it to other compute nodes across the network [8]. Hadoop can run on various
hardware, especially commodity hardware. For instance, instead of using RAID
to achieve fault tolerance, Hadoop replicates data across machines in a large
cluster [7]. In addition to Hadoop system, Spark is a data processing
framework running on top of HDFS and YARN to leverage data locality like
MapReduce of Hadoop system [9]. As an alternative to MapReduce framework,
Spark has been widely adopted for big data processing due to its speed,
easiness, and complement to MapReduce framework limitations [10]. Moreover,
beyond HDFS, Spark can also interface with a wide variety of distributed file
systems (e.g., MapR File System, Cassandra, OpenStack Swift, Amazon S3,
Chukwa, and Lustre) and local file system [11, 12, 9].

Lustre is a massively global and parallel distributed file system for high-
performance computing [13]. It is among the most commonly deployed on
supercomputing clusters [14]. Lustre provides POSIX-compliant with more than a
terabyte per second (TB/s) of aggregate I/O throughput [15]. With InfiniBand,
it will be able to directly transfer data from virtual memory on one node to
virtual memory on another node for improvement of throughput and CPU usage
[16]. However, for data-intensive jobs with large datasets, researchers argue
that network transfers should be a potential bottleneck on non-HDFS systems
[11, 17, 18]. In contrast, Ananthanarayanan et al. suggest that data locality
becomes irrelevant when networking technology is improving faster than the
disk speeds [19]. Furthermore, Xuan et al. suggest that the design of high-
performance computing architecture provides high storage capacity with low-
cost data fault tolerance [5].

Since there are many challenges of high-performance computing and big data
processing combination, it is useful to re-examine Spark with both data-
intensive and compute-intensive under HDFS and non-HDFS. Furthermore, the
small-file problem is an important criterion for examining the efficiency of
distributed file systems because the storage of numerous small files reduces
the efficiency of clusters. As a result, the paper aims to study the
applicability of a unified architecture combining high-performance computing
and big data processing. In this paper, we expect that the performance of both
Lustre and HDFS file systems will be able to achieve equivalently for big data
analytics without leveraging data locality.

##  II Related Work

Performance evaluation of distributed file systems with big data processing
frameworks has been the subject of numerous recent studies [2, 8, 20, 5, 21].
In MapReduce applications with a local disk for the shuffle phase, Zhao et al.
found that the performance of the I/O throughput of Hadoop-Lustre is nearly
equal to Hadoop-HDFS [21]. Especially, TestDFSIO benchmark shows that the read
operation of Hadoop-Lustre always achieves higher performance than Hadoop-
HDFS. Rutman [8] suggests that using Lustre as an HDFS file system replacement
on an “expensive” Hadoop compute clusters in HPC environments can
significantly improve overall “system performance and cluster efficiency or
potentially offer a lower overall system cost”. Furthermore, researchers use
Tachyon as an in-memory file system with OrangeFS distributed filesystem to
deploy data-intensive frameworks in HPC clusters [5]. The result of the
theoretical model indicates an improvement in read-throughput. Also, it will
be able to scale up with the number of compute nodes for Hadoop.

Leveraging RDMA of Infiniband on HPC clusters to increase horizontal data
movement has been the target of many studies [4, 22, 23]. Bhat et al. show
that RDMA-based HDFS plug-in significantly outperforms original Hadoop
distributions (i.e., Cloudera, Hortonworks). The approach, which accesses the
native communication library via JNI, can be deployed on many Hadoop
distributions with many different versions to leverage the high-speed
interconnect of HPC clusters (i.e., InfiniBand). Moreover, many attempts
(e.g., Hadoop Adapter for Lustre from Intel and Apache Hadoop on Lustre
Connector from Seagate) to optimize data movements in the shuffle phase of
Hadoop have been carried out [24, 25, 26]. In contrast to RDMA-based HDFS
plugins, these Hadoop plug-ins for Lustre propose an approach called “shuffle
with zero-copy”. In other words, this approach, which allows any data to be
available to all compute nodes at any time, uses Lustre as a centralized
storage. Hence, it eliminates transfers during the MapReduce shuffle phase to
achieve higher job throughput [26].

##  III Statement of Problems

Figure 1: Proposed Architecture for high-performance Computing and Big Data
Processing

###  III-A The Movement of Data in Spark

Data movement is one of the key factors affecting the performance of a Spark
application in any distributed system. Essentially, there are two types of
data movement. They are vertical data movement and horizontal data movement
[11] as shown in our proposed architecture in Figure 1.

Source: Adopted from [27]

Figure 2: Iteration jobs in MapReduce

The first type refers to “the movement through the entire memory hierarchy,
including persistent storage”. Block of data from various input sources will
be moving into memory to create resilient distributed datasets (RDDs) for
processing [28]. By default, RDDs are lazy and ephemeral; they are recomputed
after every action is run. However, with the LRU (Least Recently Used)
eviction rules, Spark allows developers to persist (i.e., save and cache) data
in a number of different places (e.g., in memory and on disk) for reuse to
minimize vertical movement for RDDs [11, 28, 9]. Persisting data in Spark is
more useful for iterative algorithms, especially in machine learning (e.g.,
Gradient-Descent for parameter optimization) [10]. Conversely, instead of
utilizing memory for storing intermediate working sets, Hadoop MapReduce
framework loads data from disk on each iteration [29]. As Figure 2 suggests,
due to the requirement of many I/O activities, Hadoop MapReduce framework
leads to the increment of bandwidth usage. In other words, with iteration
algorithms, data locality is not as important in Spark as in MapReduce.
Despite the advantage of persisting data, the capacity of memory is limited.
The block manager of Spark runtime can drop or move cached data across the
memory hierarchy. Additional vertical data movement for re-computation will be
needed. Spark is designed with a smart algorithm enough to be able to deal
with the trade-off between storing RDDs, speed of accessing it, probability of
losing it, and cost of re-computing [10]. In addition to reducing vertical
data movement for big data processing frameworks, project Tachyon helps these
frameworks decrease the re-computation and transfer time by using data lineage
and fast memory [30].

Source: Adopted from [11]

Figure 3: Data movement in Spark and the interaction with the memory hierarchy

The next data movement refers to “the shuffle communication phase between
compute nodes” [11]. The shuffle manager is responsible for managing this data
movement, along with the block manager. For scratch storage during shuffle
operations, Spark uses a local disk to handle the whole process [28]. There
are many ways to optimize the performance of horizontal data movement. The
first approach uses built-in compression techniques in Spark to compress
shuffle outputs, and data is spilled during shuffles. The next approach is
leveraging RDMA interconnection in high-performance computing clusters to
accelerate shuffle data [22]. However, the level of parallelism is highly
dependent on every practical problem. Hence, these techniques do not always
intensely impact the overall performance of a Spark application due to the
dominance of vertical data movement [11, 31].

###  III-B The Movement of Data in System Architectures

From the beginning, Hadoop system, which is based on Google File System (GFS)
and Google MapReduce framework, is designed to work with commodity hardware
for data-intensive applications with local storage to reduce the communication
cost and replication data to achieve fault tolerance [32, 17, 29]. The Hadoop
system provides a linear scale by adding commodity servers to increase
computation capacity and storage capacity [33]. The system will likely achieve
petascale [34, 29]. By default, HDFS splits a dataset into 128-MB blocks. The
block size can be changed for the cluster or any specified file when it is
created [29]. Typically, datasets stored in HDFS can be varied from gigabytes
to terabytes in size [35]. Like other distributed file systems, HDFS stores
file system metadata and application data separately [33, 36]. HDFS assigns
NameNode servers to store metadata and DataNodes servers to store application
data. The HDFS Federation feature allows multiple NameNode servers to operate
simultaneously and actively to avoid bottlenecks. Every communication protocol
of HDFS is layered on top of the TCP/IP protocol [7]. In the client node, the
HDFS client determines the location of blocks using remote procedure calls
(RPC). However, HDFS is unable to be mounted and does not fully support POSIX-
compliant because the file system is optimized for specific workloads to help
targeted applications of HDFS achieve higher performance by eliminating many
hard requirements of POSIX-compliant file system [7, 37]. In other words, this
traded feature helps HDFS to increase data throughput rates, especially the
cost of support for non-POSIX operations such as Append [7]. Moreover, as a
complement to the principle “write-once, read-many”, HDFS is designed for
immutable files with “single-writer, multiple-reader” model [33]. More
importantly, in the Hadoop system, one of the most important components is the
resource-management platform YARN (Yet Another Resource Negotiator) [38].
Before YARN, Hadoop restricted the processing model of batch-oriented
MapReduce jobs with resource management infrastructure. Vavilapalli at el.
argue that this inadequacy forces developers to abuse the MapReduce
programming model [38]. In the next generation of Hadoop, MapReduce is just
one of the frameworks running on top of YARN. Regarding data locality
optimization, YARN assigns JobTracker to place every task close to their input
data in HDFS. Hence, as mentioned before, in the world of Hadoop, a compute
node should be the same as a data node to leverage this design. Despite many
optimizations in architecture design, there are some drawbacks of Hadoop HDFS.
Firstly, Li et al. state that the I/O throughput of HDFS is limited due to
data replication for fault-tolerance [30]. To overcome this limitation, using
RAID configurations is still useful [39]. Furthermore, with a RAID 6, the
capacity and performance of storage devices are more efficiently utilized.
Secondly, in terms of data storage capacity and deployment cost, using a
separate sub-cluster for storage (high-performance computing architecture) is
more efficient than deploying DAS on compute nodes [5]. Finally, as suggested
by Ananthanarayanan et al., the “moving computation is cheaper than moving
data” principle does not reflect the recent change of technology when
bandwidth growth and cost declines [40, 41, 42, 43]. Also, by leveraging fast
memory, project Tachyon indicates that for MapReduce applications, locality
data should be irrelevant on non-HDFS systems [5, 30].

Lustre is a storage architecture for clusters only supported on the Linux
operating system [44]. In contrast to HDFS of Hadoop system, Lustre uses its
own file system (an improved version of the ext4 journaling file system) as a
back-end to store data and metadata and supports POSIX-compliant file system
interface allowing mounting from clients [45, 36]. Moreover, Lustre also
supports ZFS as a back-end to use many advantages of this file system [46].
For instance, ZFS file system allows storing over 1.5 million files in a
single directory, online data integrity checking, and error handling, high
data compression ratio without reducing performance [47]. By aggregating the
storage capacity and throughput of many servers, Lustre storage cluster
permits clients to nearly fully utilize the I/O throughput of interconnect
devices with low latency. Hence, Lustre storage architecture is adopted for
various purposes, from home users to high-performance computing clusters [48].
It is best known for serving many of the largest supercomputers around the
globe, with thousands of client systems, petabytes (PB) of storage, and over a
terabyte per second (TB/sec) of I/O throughput [14, 15, 44]. In terms of
scalability, like HDFS of Hadoop system, Luster allows adding servers
dynamically to increase both throughput and capacity of storage. Lustre also
supports RDMA for InfiniBand to achieve higher I/O throughput, network
transport efficiency and reduced CPU usage. Basically, a Lustre file system is
made up of three components: MDS, OSS, and client [44]. This first component
is one or more Metadata Server (MDSs - server nodes) with one or more Metadata
Target (MDTs - persistent storage like SSD, HDD) to store metadata (such as
filenames, directories, permissions, and file layout). The MDS does not
involve a file I/O process once the client obtains stripped information from
the file. Instead, the client directly interacts with OSSs of the Luster file
system to perform I/O operations. The second component is one or more Object
Storage Server (OSSs - server nodes) with one or more Object Storage Target
(OSTs - persistent storage like SSD, HDD) to store one or more objects (depend
upon stripe configuration and size of file), each object on a separate OST
across the Lustre file system. Along with other high-performance computing
applications, in Spark applications, the throughput of vertical data movement
on the Lustre file system is the aggregated throughput provided by each OST.
Likewise, the capacity of the Lustre file system is the total capacity
provided by each OST. The final component is one or more clients to mount the
Lustre file system as an accessible file system. In contrast to HDFS, Lustre
file system is carefully optimized for “many-write-many-read” principle, which
is a critical requirement for scientific applications [49]. In other words,
Luster fully supports concurrent and coherent read and write access to the
files in the file system. Lustre does not support data redundancy. To achieve
failover, it must rely on backup techniques like RAID. Like Hadoop HDFS,
despite many optimizations in architecture design, there are some drawbacks of
Lustre file system, which will be covered in the next section.

Source: Adopted from [44]

Figure 4: Example of Lustre cluster layout

###  III-C The Small-File Problem of Distributed File System

Every distributed file system is designed with the assumption that the file
size is very large. Thus, its metadata management becomes bottle-necked
because of handling large numbers of small-file requests from clients [50].

Due to the nature of both scientific applications and many Internet-oriented
applications, dealing with a large number of small files is unavoidable [51,
49]. Accordingly, “the percentage of files smaller than 1 KB is 49.3%, and
only 0.8% files are larger than 10 MB” [51]. In HDFS, namenode keeps metadata
of the filesystem in memory, and small files dramatically increase the amount
of memory for the metadata on the namenode [29]. The number of map tasks is
equal to the amount of files. Hence, it leads to an overhead parallelism
level, which is tens or hundreds of times slower than the equivalent job with
a single input file [52]. Moreover, it takes about 7.7 hours to store 550000
small files, whose sizes range from 1KB to 10KB, into HDFS while in ext3 local
file system, it is only 660 seconds [53]. Likewise, many researchers argue
that small files significantly impact the overall performance of Lustre file
system [45, 54]. The number of operational MDS is restricted to one; the other
MDS becomes operational if the first MDS crashes. Hence, it remains a
potential bottleneck as the number of clients and/or files increases
significantly. However, Lustre Distributed Namespace (DNE) projects allow
multiple pairs of active/active metadata servers to improve Lustre metadata
performance and scalability [55]. In addition, for every client request, two
RPCs are generated at least [45]. It is very costly to request many small
files concurrently. Moreover, like HDFS, in Lustre, the number of map tasks
equals the number of files.

Hence, dealing with small files is a serious problem. It requires many
advanced techniques, including a back-end modified layer of Lustre and/or
powerful hardware combination to overcome the issue [5, 56, 30, 45, 2, 55]. As
mentioned in this paper, we focus on the performance efficiency of distributed
file systems on small files issues with complex machine learning algorithms
scenarios. Hence, we do not include these suggestions in the proposed
architecture.

##  IV Empirical Results

###  IV-A Experimental Setup

The experiment was performed on two clusters, which ran on the top of KVM
(Kernel-based Virtual Machine). In Lustre cluster, we deployed Lustre 2.7 on a
2-core, 2GB of RAM with a 100-GB MDT for the MDS and 2-core, 2GB of RAM with a
100-GB OST for each OSS. However, the allocated resource for Lustre cluster
storage does not fully utilize the read-only caching of data feature in the
OSS to enhance performance. In Hadoop cluster, we deployed Cloudera
Distribution Hadoop 5.7.1 (including Spark 1.6). The Cloudera master node had
a 6-core with 24GB of RAM and a 6-core with 12GB of RAM for each worker. Also,
the Hadoop cluster could mount Lustre file system from the Lustre cluster.
Each physical node uses a 1-Gigabit TCP/IP Ethernet.

On our benchmark, we use Kaggle Display Advertising Challenge Dataset from
Criteo with functions (i.e., LogisticRegressionWithSGD, LabeledPoint and
SparseVector) from Spark MLlib [57]. We split the training dataset from Criteo
into multiparts, and each part contains 460 lines. We use 10000-file (1.1 GB
in total), 20000-file (2.2GB in total), and 30000-file (3.3GB in total) for
the benchmark. Moreover, we use Spark Standalone for the data source of jobs
from Lustre, and we use Spark with YARN scheduler for the data source of jobs
from HDFS.

###  IV-B Result Analysis

Figure 5: Execution time of jobs on Lustre and HDFS file system with different
amounts of files

As Figure 5 suggests, the execution time of jobs on both Lustre and HDFS file
systems increases quickly as the amount of data for processing becomes larger.
Especially when dealing with 30000 files, the running time on Lustre and HDFS
file systems dramatically increases.

The execution time of jobs in Spark with Lustre slightly outperforms Spark
with HDFS. However, it is substantial. Moreover, the gap increases when the
number of small files is larger. In other words, regarding the small file
problem, Lustre is appropriate for big data storage.

As a result, as Figure 5 indicates, a unified storage solution for both high-
performance computing and big data processing is achievable by using Luster
distributed file system for both jobs.

##  V Conclusion

In this paper, we design and propose a unified platform for high-performance
computing and big data processing by leveraging contemporary literature. The
experimental benchmark verifies the applicability of the architecture. In our
benchmark, we determine the performance of the small file problem on the
distributed file system with a complex machine learning algorithm. The
benchmark indicates that Lustre can substantially achieve better performance
than HDFS, even though Lustre hardware resources, which do not leverage fast
memory for cache, are very limited.

However, there are some limitations in our work. First, in terms of
scalability, the architecture is only verified in a small-scale environment in
which hardware resources do not meet the criteria of a production environment.
Second, the amount of small files used for the benchmark is not big enough to
absolutely reflect the consistent benchmark result of distributed file systems
in extreme scenarios. Third, we did not include Tachyon in this proposed
architecture to increase overall performance for both HDFS and Lustre file
systems. Finally, we did not cover a write benchmark, a crucial factor for
scientific computing and big data processing.

In future work, we will verify the applicability of the proposed architecture
in a larger-scale environment with a larger number of small files to be able
to determine hidden problems that are impossible to find in a small-scale
environment. In addition, we will also further investigate the impact of
Tachyon on overall performance. Furthermore, by using libhdfs (JNI based C API
for Hadoop HDFS), we will perform a write benchmark with small files for both
Luster and HDFS.

## References

  * [1] G. C. Fox, J. Qiu, S. Kamburugamuve, S. Jha, and A. Luckow, “HPC-ABDS High Performance Computing Enhanced Apache Big Data Stack,” in _2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid)_. IEEE, pp. 1057–1066.
  * [2] D. Moise, “Experiences with Performing MapReduce Analysis of Scientific Data on HPC Platforms,” in _DIDC ’16: Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing_ , Cray Inc. New York, New York, USA: ACM, Jun. 2016, pp. 11–18.
  * [3] D. Zhao, Z. Zhang, X. Zhou, T. Li, K. Wang, D. Kimpe, P. Carns, R. Ross, and I. Raicu, “FusionFS: Toward supporting data-intensive scientific applications on extreme-scale high-performance computing systems,” in _2014 IEEE International Conference on Big Data (Big Data)_. IEEE, pp. 61–70.
  * [4] A. Bhat, N. S. Islam, X. Lu, M. Wasi-ur Rahman, D. Shankar, and D. K. DK Panda, “A Plugin-Based Approach to Exploit RDMA Benefits for Apache and Enterprise HDFS,” in _Big Data Benchmarks, Performance Optimization, and Emerging Hardware_. Cham: Springer International Publishing, Jan. 2016, pp. 119–132.
  * [5] P. Xuan, J. Denton, P. K. Srimani, R. Ge, and F. Luo, “Big data analytics on traditional HPC infrastructure using two-level storage,” in _the 2015 International Workshop_. New York, New York, USA: ACM Press, 2015, pp. 1–8.
  * [6] B. T. Rao and L. S. S. Reddy, “Survey on Improved Scheduling in Hadoop MapReduce in Cloud Environments,” _arXiv.org_ , Jul. 2012.
  * [7] D. Borthakur, “The hadoop distributed file system: Architecture and design,” _Hadoop Project Website_ , 2007.
  * [8] N. Rutman, _Map/reduce on lustre-hadoop performance in hpc environments_. Langstone Road, 2011.
  * [9] M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, “Spark: cluster computing with working sets,” in _HotCloud’10: Proceedings of the 2nd USENIX conference on Hot topics in cloud computing_ , University of California, Berkeley. USENIX Association, Jun. 2010, pp. 10–10.
  * [10] K. G. Srinivasa and A. K. Muppalla, _Guide to High Performance Distributed Computing: Case Studies with Hadoop, Scalding and Spark_. Springer Publishing Company, Incorporated, Feb. 2015.
  * [11] N. Chaimov, A. Malony, S. Canon, C. Iancu, K. Z. Ibrahim, and J. Srinivasan, “Scaling Spark on HPC Systems,” in _HPDC ’16: Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing_ , Lawrence Berkeley National Laboratory. New York, New York, USA: ACM, May 2016, pp. 97–110.
  * [12] M. Daz, C. Martn, and B. Rubio, “State-of-the-art, challenges, and open issues in the integration of Internet of things and cloud computing,” _Journal of Network and Computer Applications_ , vol. 67, no. C, pp. 99–117, May 2016.
  * [13] P. Raj, A. Raman, D. Nagaraj, and S. Duggirala, _High-Performance Big-Data Analytics: Computing Systems and Approaches, 1st edition_. Springer Publishing Company, Incorporated, Oct. 2015.
  * [14] Lustre to DAOS: Machine Learning on Intel’s Platform.
  * [15] Spider – the Center-Wide Lustre File System.
  * [16] W. Yu, R. Noronha, S. Liang, and D. K. Panda, “Benefits of high speed interconnects to cluster file systems: a case study with lustre,” in _IPDPS’06: Proceedings of the 20th international conference on Parallel and distributed processing_ , Ohio State University. IEEE Computer Society, Apr. 2006, pp. 273–273.
  * [17] J. Dean and S. Ghemawat, “MapReduce: simplified data processing on large clusters,” _Communications of the ACM_ , vol. 51, no. 1, pp. 107–113, Jan. 2008.
  * [18] M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, “Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling,” in _EuroSys ’10: Proceedings of the 5th European conference on Computer systems_ , University of California, Berkeley. New York, New York, USA: ACM, Apr. 2010, pp. 265–278.
  * [19] _Disk-locality in datacenter computing considered irrelevant_ , University of California, Berkeley. USENIX Association, May 2011.
  * [20] M. Wasi-ur Rahman, X. Lu, N. S. Islam, R. Rajachandrasekar, and D. K. Panda, “High-Performance Design of YARN MapReduce on Modern HPC Clusters with Lustre and RDMA,” in _IPDPS ’15: Proceedings of the 2015 IEEE International Parallel and Distributed Processing Symposium_. IEEE Computer Society, May 2015, pp. 291–300.
  * [21] T. Zhao, Z. Zhang, and X. Ao, “Application Performance Analysis of Distributed File Systems under Cloud Computing Environment,” _Information Science and Control_ , pp. 152–155, 2015.
  * [22] X. Lu, M. W. U. Rahman, N. Islam, D. Shankar, and D. K. Panda, “Accelerating Spark with RDMA for Big Data Processing: Early Experiences,” in _HOTI ’14: Proceedings of the 2014 IEEE 22nd Annual Symposium on High-Performance Interconnects_. IEEE Computer Society, Aug. 2014, pp. 9–16.
  * [23] D. Shankar, X. Lu, M. Wasi-ur Rahman, N. Islam, and D. K. Panda, “Characterizing and benchmarking stand-alone Hadoop MapReduce on modern HPC clusters,” _The Journal of Supercomputing_ , Jun. 2016.
  * [24] Intel Corporation. (2015) Intel Rolls Out Enhanced Lustre File System.
  * [25] ——. (2015) Lustre at the Core of HPC and Big Data Convergence.
  * [26] (2015) Seagate Apache Hadoop on Lustre Connector.
  * [27] C. McDonald. (2015) Parallel and Iterative Processing for Machine Learning Recommendations with Spark.
  * [28] M. Hamstra, H. Karau, M. Zaharia, A. Knwinski, and P. Wendell, _Learning Spark: Lightning-Fast Big Analytics_. O’Really Media, 2015.
  * [29] T. White, _Hadoop: The Definitive Guide_. O’Reilly Media, Inc., May 2012.
  * [30] H. Li, A. Ghodsi, M. Zaharia, and E. Baldeschwieler, “Tachyon: Memory throughput i/o for cluster computing frameworks,” _memory_ , 2013.
  * [31] J. Sparks, H. Pritchard, and M. Dumler, “The Cray Framework for Hadoop for the Cray XC30,” _Cray User Group Conference (CUG’14)_ , 2014.
  * [32] _The Google file system_ , 2003.
  * [33] _The Hadoop Distributed File System_ , 2010.
  * [34] A. Fasale and N. Kumar, _YARN Essentials_. Packt Publishing Ltd, Feb. 2015.
  * [35] S. U. Khan and A. Y. Zomaya, _Handbook on Data Centers_. Springer, May 2015.
  * [36] F. Wang, S. Oral, G. Shipman, and O. Drokin, “Understanding lustre filesystem internals,” 2009.
  * [37] D. Moise, G. Antoniu, and L. Bougé, “Improving the Hadoop map/reduce framework to support concurrent appends through the BlobSeer BLOB management system,” 2010.
  * [38] V. K. Vavilapalli, S. Seth, B. Saha, C. Curino, O. O’Malley, S. Radia, B. Reed, E. Baldeschwieler, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, and H. Shah, “Apache Hadoop YARN,” in _the 4th annual Symposium_. New York, New York, USA: ACM Press, 2013, pp. 1–16.
  * [39] W. Xu, W. Luo, and N. Woodward, “Analysis and optimization of data import with hadoop,” _Parallel and Distributed_ , 2012.
  * [40] S. Kipp. (2012) Exponential bandwidth growth and cost declines.
  * [41] D. Lago, E. Madeira, and D. Medhi, _High speed network impacts and power consumption estimation for cloud data centers_. New York, New York, USA: ACM, Apr. 2015.
  * [42] D. J. Law, W. W. Diab, A. Healey, and S. B. Carlson, “IEEE 802.3 industry connections Ethernet bandwidth assessment,” 2012.
  * [43] T. P. Morgan, “InfiniBand Too Quick For Ethernet To Kill,” Apr. 2015.
  * [44] Intel Corporation, _Lustre Software Release 2.x Operations Manual_ , 2013.
  * [45] V. Meshram, X. Ouyang, and D. K. Panda, “Minimizing Lookup RPCs in Lustre File System using Metadata Delegation at Client Side,” _Tech Rep OSU-CISRC-7/11_ , 2011.
  * [46] L. Grandinetti, G. R. Joubert, and M. Kunze, _Big Data and High Performance Computing_. IOS Press, Oct. 2015.
  * [47] D. M. Stearman, “ZFS on RBODs Leveraging RAID Controllers for Metrics and Enclosure Management,” 2015.
  * [48] R. Brueckner, “Building a CIFS/NFS Gateway to Lustre - insideHPC,” Oct. 2014.
  * [49] D. Zhao, X. Yang, I. Sadooghi, G. Garzoglio, S. Timm, and I. Raicu, _High-Performance Storage Support for Scientific Applications on the Cloud_. New York, New York, USA: ACM, Jun. 2015.
  * [50] K. V. Shvachko, “HDFS scalability: the limits to growth,” _; login:: the magazine of USENIX SAGE_ , vol. 35, no. 2, pp. 6–16, 2010.
  * [51] P. Wang, L. Cao, C. Lai, L. Zou, G. Sun, and J. Cong, _InterFS: An Interplanted Distributed File System to Improve Storage Utilization_ , ser. An Interplanted Distributed File System to Improve Storage Utilization. New York, New York, USA: ACM, Jul. 2015.
  * [52] T. White. (2009) The Small Files Problem. [Online]. Available: http://blog.cloudera.com/blog/2009/02/the-small-files-problem/
  * [53] X. Liu, J. Han, Y. Zhong, C. Han, and X. He, “Implementing WebGIS on Hadoop: A case study of improving small file I/O performance on HDFS,” in _2009 IEEE International Conference on Cluster Computing and Workshops_. IEEE, 2009, pp. 1–8.
  * [54] M. Pershin. Intel Lustre Data on MDT/Small File I/O.
  * [55] J. Simmons, D. Leverman, J. Hanley, and S. Oral. (2016, Jun.) Lustre Distributed Name Space (DNE) Evaluation at the Oak Ridge Leadership Computing Facility (OLCF).
  * [56] S. Ihara. Lustre Metadata Fundamental Benchmark and Performance.
  * [57] Criteo. (2014) Kaggle display advertising challenge. [Online]. Available: http://labs.criteo.com/downloads/2014-kaggle-display-advertising-challenge-dataset