# Native Language Identification with Big Bird Embeddings

Sergey Kramp, Giovanni Cassani, Chris Emmery
CSAI, Tilburg University
<EMAIL_ADDRESS>
###### Abstract

Native Language Identification (NLI) intends to classify an author’s native
language based on their writing in another language. Historically, the task
has heavily relied on time-consuming linguistic feature engineering, and
transformer-based NLI models have thus far failed to offer effective,
practical alternatives. The current work investigates if input size is a
limiting factor, and shows that classifiers trained using Big Bird embeddings
outperform linguistic feature engineering models by a large margin on the
Reddit-L2 dataset. Additionally, we provide further insight into input length
dependencies, show consistent out-of-sample performance, and qualitatively
analyze the embedding space. Given the effectiveness and computational
efficiency of this method, we believe it offers a promising avenue for future
NLI work.

##  1 Introduction

Native Language Identification (NLI) operates under the assumption that an
author’s first language (L1) produces discoverable patterns in a second
language (L2) Odlin (1989); MacDonald (2013). Classifying one’s native
language proves highly useful in various applications, such as in language
teaching, where customized feedback could be provided based on the learner
native language; in fraud detection, where identifying an unknown author’s
native language can aid in detecting plagiarism and web fraud; and in consumer
analytics. NLI models historically relied on handcrafted linguistic patterns
as input features Koppel et al. (2005); Chen et al. (2017); Tetreault et al.
(2013); Cimino et al. (2013); however, such representations are unlikely to
capture all required nuances and complexities of this task Moschitti and
Basili (2004), in particular on noisier sources of data.

Current transformer models Vaswani et al. (2017) have shown success in such
challenges Brown et al. (2020) but are often limited by input size. This is
particularly problematic for NLI which often deals with long texts, such as
essays, documents or social media posts. Our work is the first to employ long-
form transformer models to overcome these limitations. We train a simple
logistic regression classifier using embeddings from a fine-tuned Big Bird
model, and demonstrate it significantly outperforms a similar classifier
trained using costly handcrafted feature representations.111Code and models
available at github.com/SergeyKramp/mthesis-bigbird-embeddings.

##  2 Related Work

Seminal NLI work by Koppel et al. (2005) used function words, character
$n$-grams, and handcrafted error types as features—restricted to 1000 articles
in 5 languages. The TOEFL-11 dataset Blanchard et al. (2013) proved a fruitful
resource for two NLI shared tasks Tetreault et al. (2013); Malmasi et al.
(2017). However, its controlled collection environment and limited range of
topics affected generalization of traditional linguistic features to noisy
Internet data Baldwin et al. (2013). An example of such noisy data is the
Reddit-L2 dataset Rabinovich et al. (2018); the current de facto benchmark for
NLI, which we employ in the current study as well.

Despite various attempts using neural architectures Ircing et al. (2017);
Bjerva et al. (2017); Franco-Salvador et al. (2017), the current best
performance on the Reddit-L2 dataset was obtained by Goldin et al. (2018)
using a logistic regression classifier trained on a combination of linguistic
features. We will implement (and thereby directly compare to) their work in
our experiments.

Most related are two studies using transformers for NLI. Steinbakken and
Gambäck (2020) fine-tuned BERT on a less challenging part of the Reddit-L2
dataset Devlin et al. (2018); stand-alone, and in an ensemble of classifiers.
Lotfi et al. (2020) fine-tuned GPT-2 Radford et al. (2019) per language in the
TOEFL-11 dataset (i.e., 11 in total), using the lowest loss among them to
classify an instance. Our method offers a stand-alone transformer model
approach with a much lower computational footprint. We will evaluate
performance on the Reddit-L2 split with little to no information related to
(linguistic) geography.

Rabinovich et al. (2018) have used hierarchical clustering to investigate the
relationship between an author’s native language and their lexical choice in
English. Using word frequency and word embeddings of English words, they
measured distances between 31 L1s. Languages from the same family appear
closest in this space. They further suggested that authors with a similar L1
have similar idiosyncrasies in their English writing. Hence, given an accurate
model, we expect to find similar representations in our embedding spaces.

##  3 Methodology

We test if Big Bird embeddings are a suitable application of the transformer
architecture for NLI, thereby mostly replicating the experimental design of
Goldin et al. (2018).

###  3.1 Data

We used a derivative of the Reddit-L2 dataset, first introduced as L2-Reddit
by Rabinovich et al. (2018), and used in Goldin et al. (2018). The raw
data222Via: http://cl.haifa.ac.il/projects/L2/ consists of $200\text{M}$
sentences ($\sim 3\text{B}$ tokens), and spans the years 2005-2017 and used
the old (i.e., free) Reddit API. Data collection used flairs that report
country of origin on subreddits discussing European politics, yielding a total
of $45K$ labeled native and non-native English-speaking users and their entire
post history. Between-group language proficiency was accounted for through
several syntactic and lexical metrics, and languages with fewer than 100
authors were removed. Each author profile was split per 100 sentences, and
these “chunks” were subsequently divided in two splits: one partition with
subreddits discussing European politics (referred to as the europe partition),
and a second partion from all other subreddits (the non_europe partition).

#### Sampling

For L1 identification, we regrouped the Reddit-L2 dataset on native language
rather than nationality. After filtering predominantly multi-lingual
countries, this resulted in 23 labels. We found that the majority are native
English speakers, with Dutch native speakers constituting the second largest
part, and that there is a stronger label imbalance in the non_europe partition
than in the europe partition.

In accordance with Goldin et al. (2018), the data was balanced through
downsampling by randomly selecting 273 and 104 authors respectively for each
language in our two partitions. These author proportions are based on the
least represented language in each partition: Slovenian and Lithuanian.
Similarly, to reduce the skew that highly active authors for a given language
add to the data, the amount of chunks per author was capped. These were
randomly sampled until the median value per author; 17 for the non_europe
partition, and 3 for the europe one.

#### Preprocessing

For this, we removed redundant blank spaces and replaced all URLs with a
special token. While minimal, these changes improved classification
performance across the board.

#### Splitting

We split the non-europe partition on chunk level333Splitting by authors had
negligible effects. into equal fine-tuning ($D_{\text{tune}}$), and training
and testing ($D_{\text{exp}}$) parts. We hypothesized that due to the size and
variety of the non_europe partition, it is a more realistic, challenging part
of the data. Unlike the europe partition used by Steinbakken and Gambäck
(2020), it covers a variety of topics and contains fewer context words (e.g.,
countries and nationalities) that might pollute classification. Instead, we
dedicated the entire europe partition to conduct an out-of-sample evaluation.
We refer to this data as $D_{\text{oos}}$. As this part of the data contains
texts on topics not seen in $D_{\text{tune}}$ and $D_{\text{exp}}$, this
allows us to gauge the context specificity of our representations.

###  3.2 Feature Engineering Baseline

Here we describe how the linguistic features444For comparison sake, we did not
optimize these further. (5186 total) were constructed. We followed Goldin et
al. (2018) or found equivalents to the features used in their work. These were
extracted for each chunk.

####  $n$-Grams

To create word unigram and character trigram features, we used scikit-learn
Pedregosa et al. (2011). Both vectorizers were fit on the text chunks of
$D_{\text{exp}}$ and cut off at the 1000 most common $n$-grams.

#### Edit Distance and Substitution

To collect the spelling errors in the data, we used the
symspellpy555github.com/mammothb/symspellpy package. For each misspelled word
in $D_{\text{exp}}$, we obtained its closest correction with a maximal edit
distance of 2. Words for which no correction was found were ignored. Next, we
tracked which characters were inserted, deleted or replaced to arrive at the
correction. This resulted in a substitution frequency list, of which the top
400 were used. The number of occurrences of each substitution type in the
chunk was used as features. Subsequently, for each chunk we aggregated the
Levenshtein distance between all words and their corrections, and divided this
by the total number of words, giving the average edit distance.

#### Grammar, POS, Function Words, Length

For the other features, each chunk in $D_{\text{exp}}$ was split into
individual sentences (by \n). The grammar error features were extracted using
the LanguageTool Python wrapper666github.com/jxmorris12/language_tool_py to
produce a list of errors for all sentences in $D_{\text{exp}}$. In total, we
found 2017 error types in the data and used all of them as binary features
(i.e., the presence or absence of a grammar error in that chunk). POS trigrams
were created through nltk777We used the pre-trained Averaged Perceptron Tagger
in combination with the Punkt Tokenizer. Bird et al. (2009), and their top 300
used as features. For the function word frequency features, we used a list of
467 function words taken from Volansky et al. (2015). For average sentence
length, we removed all non-alphanumeric symbols of length 1 to exclude
punctuation and special symbols, then divided sentence length (on word level)
by the total number of sentences in a chunk (i.e., 100).

###  3.3 Transformer Model

The main focus of this study was to find an efficient method to use
transformers for NLI. To this end, we chose Big Bird (google/bigbird-roberta-
base) from the Hugging Face Model Hub Wolf et al. (2019), as it provides a
relatively large context length of 4096 tokens while fitting on one GPU.888We
used an Nvidia Titan X with 12 GB of VRAM.

#### Fine-tuning

We fine-tuned all layers of Big Bird on $D_{\text{tune}}$ using the
hyperparameters specified in the original paper: Adam Kingma and Ba (2015) to
optimize with the learning rate set to $10^{-5}$ and epsilon to $10^{-8}$.
Warm-up on 10% of all training inputs ran during the first epoch. Fine-tuning
ran for 3 epochs totaling 15 hours. Due to memory constraints, we used an
input size of 2048, with a batch size of 2. Chunks that were shorter were
padded to match the input length; longer inputs were split into sub-chunks
(padded to full length).

#### Embedding Representation

In order to compare Big Bird to lingustic features, we do not train Big Bird
end-to-end. Rather, we extract its embeddings (either pre-trained from the
Model Hub or our own fine-tuned version) and use them as input for a
downstream classifer. For tokenization, we used the matching pre-trained
tokenizers from transformers,999github.com/huggingface/transformers which were
fine-tuned during our experiments. We added [CLS] at the beginning of the
first sentence of each chunk, and manually inserted a separator token between
each sentence in the chunk and at the end of the chunk.

Following Devlin et al. (2018), we used the last hidden states for [CLS] as
768-dimensional embedding features per chunk. We experimented with 3 token
input sizes: 512 (BERT’s input size), 2048 (size also used when fine-tuning),
and 4096 (Big Bird’s maximum input size).

model | dur | acva | oosa
---|---|---|---
$FeatureEngineering$ | 13.00 | .475 | .637
$BigBird_{512}$ | 0.27 | .364 | -
$BigBird_{512\\_tuned}$ | 0.27 | .432 | -
$BigBird_{2048}$ | 2.50 | .493 | .774
$BigBird_{2048\\_tuned}$ | 2.50 | .654 | .855
$BigBird_{4096}$ | 3.00 | .500 | -
$BigBird_{4096\\_tuned}$ | 3.00 | .635 | -
Table 1: The models (name) annotated with their input dimensions and if they
were fine-tuned, how long feature extraction took on $D_{\text{exp}}$ (dur, in
hours), their average cross-validation accuracy scores on $D_{\text{exp}}$
(acva) and accuracy scores on $D_{\text{oos}}$ (oosa).

##  4 Experimental Setup

For our main experiment, we followed the experimental design in Goldin et al.
(2018):

###  4.1 Main Experiment

We trained a logistic regression classifier on the output of each feature
extractor. To further establish an equal ground for comparison, we did not
tune the hyperparameters of these classifiers. Hence, we adopted scikit-
learn’s default parameters: $\ell_{2}$ normalization, $C=1$, L-BFGS Liu and
Nocedal (1989) for optimization, and maximum iterations set to 1000. To gauge
the robustness of each classifier’s performance, we used 10-fold cross-
validation (CV); in particular, we looked at the average CV accuracy score of
each classifier. Given that we adhere to prior work and accordingly balanced
the labels, we found that additional metrics provided little added insight.

###  4.2 Embedding Space Analysis

Following Rabinovich et al. (2018), we used hierarchical clustering to analyze
how each native language is represented in the 768-dimensional embedding
space. We used the best performing pre-trained and fine-tuned Big Bird models
from our main experiment to compute the centroids (23 in total) on
$D_{\text{exp}}$. Subsequently, we used scipy’s Virtanen et al. (2020)
implementation of Ward’s linkage function Ward Jr (1963) to create a cluster
dendrogram, and scikit-learn’s default implementation of Principal Component
Analysis (Hotelling, 1933; Tipping and Bishop, 1999, PCA) to visualize the
centroids in a 2-dimensional space.

[width=0.47trim=0.5cm 0.34cm 0 0.12cm,clip]figures/text_length

Figure 1: Baseline and embedding model accuracy scores by percentage
increments of total input length.

###  4.3 Error Analysis

We conducted two additional error analyses to test the robustness of the
embeddings:

#### Out-of-sample Analysis

To assess generalization, we trained 3 classifiers on $D_{\text{exp}}$ and
tested on $D_{\text{oos}}$. As mentioned, text in $D_{\text{oos}}$ only
concerns European politics, which is close to absent in the training data. In
particular, we trained three classifiers using different features: our
baseline using the linguistic features, and two classifiers using Big Bird
embeddings, using the best performing pre-trained and fine-tuned feature
extractors (see Table 1). We considered both versions of the feature extractor
to control for any data leakage that occurred during fine-tuning.

#### Sensitivity to Text Length

To gauge the effect of text length on performance, we randomly sampled 1000
chunks from $D_{\text{exp}}$ and created slices101010Sliced on \n. We also
experimented with sentence, clause, and character-level, but observed similar
results in all cases. of 10%, 20%, 40%, and 80% of the total length of the
chunk, following a similar baseline and embedding extraction method as the
out-of-sample analysis. Next, we trained a logistic regression classifier,
similar to those described in Section 4.1, on all of $D_{\text{exp}}$ except
the 1000 randomly sampled chunks. Then, we obtained predictions for all
slices, and computed the accuracy for each slice group; i.e., accuracy for all
10% slices, 20% slices, etc.

##  5 Results

###  5.1 Main Experiment

Table 1 shows the average CV scores of each classifier.
$BigBird_{2048\\_tuned}$ yielded the highest average CV accuracy with 65.38%;
a 17 point increase over the baseline trained on linguistic features (47.55%).
The classifiers trained on fine-tuned embeddings outperformed their pre-
trained versions across all three model variants. However, differences are
smallest for $BigBird_{512}$, suggesting that the short input size limits
fine-tuning’s efficiency. Increasing input size seems to have a small effect,
though we note that the average chunk length in $D_{\text{exp}}$ is 1726
tokens; i.e., with an input size of 2048 tokens, most are captured already.

###  5.2 Embedding Space Analysis

[scale=0.78,trim=0cm 0.05cm 0.1cm 0.12cm,clip]figures/clustering

Figure 2: Hierarchical clustering dendograms of native language centroids in
the 768-dimensional Big Bird embedding space before and after fine-tuning.

Although our clustering shows some overlap with the results of Rabinovich et
al. (2018), there are some deviations. Languages from the same language family
are not always close (see Figure 2, fine-tuned or not). For example, Russian
is clustered with Turkish (pre-trained) and Italian with the former
Yugoslavian languages (fine-tuned). Furthermore, fine-tuning shifts the
embedding space more toward separating individual languages, rather than
separating native-English from non-native English (as indicated by English
having it own cluster). This effect is most apparent in the low-dimensional
PCA space (see Figure 3). In the fine-tuned space, an interesting artifact can
be observed, where the space roughly mimics the languages’ geographical
orientation to each other.

###  5.3 Error Analysis

#### Out-of-sample Analysis

Here we see the same pattern as in our main experiment (see Table 1), with the
fine-tuned embedding approach yielding the most accurate classifier,
outperforming the feature engineering baseline by 22 percentage points,
whereas the pre-trained model gains 13.7.

#### Sensitivity to Text Length

In Figure 1, it can be observed that the performance of both embedding and
feature engineering classifiers deteriorates as text length decreases.
However, the deterioration is not linear, which suggests there is increased
redundancy in the information used for classification the longer the input
becomes. The embeddings are more affected, with a 12 point drop when reducing
from 80% to 40% and a 14 point drop when reducing from 40% to 20%, compared to
5 points and 7 points for the feature engineering model.

[scale=0.8,trim=0.65cm 0.2cm 0cm 0.12cm,clip]figures/pca

Figure 3: 2-dimensional PCA space showing the language centroids before and
after fine-tuning.

##  6 Discussion & Conclusion

Our experiments demonstrate how fairly straightforward featurization using
embeddings from transformers that account for long enough input sequences is
faster, and substantially outperforms prior best performing models. Some
limitations should be mentioned, such as the restricted domain (Reddit only),
the dataset containing mostly highly fluent English speakers, and English
being the only L2. Moreover, while out-of-sample, $D_{\text{oos}}$ was likely
not completely new; Big Bird might have been trained on Reddit prior, and,
therefore, other social platforms are worth evaluating on as well (although
label collection will likely be significantly more challenging). We expect
even better results if other classifiers are used and tuned, and a comparison
with similar transformers such as Longformer Beltagy et al. (2020) and
Transformer-XL Dai et al. (2019) is certainly worthwhile Bulatov et al.
(2023). As is commonly observed Devlin et al. (2018); Sun et al. (2019);
Howard and Ruder (2018), fine-tuning Big Bird on our data improved
performance, and our observations proved robust both throughout cross-
validation and on out-of-sample data. Given these results, we believe our
works offers a promising avenue for future NLI work.

##  7 Acknowledgments

Our research strongly relied on openly available resources. We thank all whose
work we could use.

## References

  * Baldwin et al. (2013) Timothy Baldwin, Paul Cook, Marco Lui, Andrew MacKinlay, and Li Wang. 2013.  How noisy social media text, how diffrnt social media sources?  In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_ , pages 356–364.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _CoRR_ , abs/2004.05150.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  ” O’Reilly Media, Inc.”.
  * Bjerva et al. (2017) Johannes Bjerva, Gintarė Grigonytė, Robert Östling, and Barbara Plank. 2017.  Neural networks and spelling features for native language identification.  In _Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 235–239, Copenhagen, Denmark. Association for Computational Linguistics.
  * Blanchard et al. (2013) Daniel Blanchard, Joel Tetreault, Derrick Higgins, Aoife Cahill, and Martin Chodorow. 2013.  Toefl11: A corpus of non-native english.  _ETS Research Report Series_ , 2013(2):i–15.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Bulatov et al. (2023) Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev. 2023.  Scaling transformer to 1m tokens and beyond with rmt.  _arXiv preprint arXiv:2304.11062_.
  * Chen et al. (2017) Lingzhen Chen, Carlo Strapparava, and Vivi Nastase. 2017.  Improving native language identification by using spelling errors.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 542–546.
  * Cimino et al. (2013) Andrea Cimino, Felice Dell’Orletta, Giulia Venturi, and Simonetta Montemagni. 2013\.  Linguistic profiling based on general–purpose features and native language identification.  In _Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 207–215.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019.  Transformer-xl: Attentive language models beyond a fixed-length context.  _arXiv preprint arXiv:1901.02860_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * Franco-Salvador et al. (2017) Marc Franco-Salvador, Greg Kondrak, and Paolo Rosso. 2017.  Bridging the native language and language variety identification tasks.  _Procedia Computer Science_ , 112:1554–1561.
  * Goldin et al. (2018) Gili Goldin, Ella Rabinovich, and Shuly Wintner. 2018.  Native language identification with user generated content.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3591–3601, Brussels, Belgium. Association for Computational Linguistics.
  * Hotelling (1933) Harold Hotelling. 1933.  Analysis of a complex of statistical variables into principal components.  _Journal of educational psychology_ , 24(6):417.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  _arXiv preprint arXiv:1801.06146_.
  * Ircing et al. (2017) Pavel Ircing, Jan Švec, Zbyněk Zajíc, Barbora Hladká, and Martin Holub. 2017.  Combining textual and speech features in the NLI task using state-of-the-art machine learning techniques.  In _Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 198–209, Copenhagen, Denmark. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Koppel et al. (2005) Moshe Koppel, Jonathan Schler, and Kfir Zigdon. 2005.  Automatically determining an anonymous author’s native language.  In _Intelligence and Security Informatics_ , pages 209–217, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Liu and Nocedal (1989) Dong C. Liu and Jorge Nocedal. 1989.  On the limited memory BFGS method for large scale optimization.  _Math. Program._ , 45(1-3):503–528.
  * Lotfi et al. (2020) Ehsan Lotfi, Ilia Markov, and Walter Daelemans. 2020.  A deep generative approach to native language identification.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1778–1783, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * MacDonald (2013) Maryellen C MacDonald. 2013.  How language production shapes language form and comprehension.  _Frontiers in psychology_ , 4:226.
  * Malmasi et al. (2017) Shervin Malmasi, Keelan Evanini, Aoife Cahill, Joel Tetreault, Robert Pugh, Christopher Hamill, Diane Napolitano, and Yao Qian. 2017.  A report on the 2017 native language identification shared task.  In _Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 62–75.
  * Moschitti and Basili (2004) Alessandro Moschitti and Roberto Basili. 2004.  Complex linguistic features for text classification: A comprehensive study.  In _Advances in Information Retrieval: 26th European Conference on IR Research, ECIR 2004, Sunderland, UK, April 5-7, 2004. Proceedings 26_ , pages 181–196. Springer.
  * Odlin (1989) Terence Odlin. 1989.  _Language transfer_ , volume 27.  Cambridge University Press Cambridge.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Rabinovich et al. (2018) Ella Rabinovich, Yulia Tsvetkov, and Shuly Wintner. 2018.  Native language cognate effects on second language lexical choice.  _CoRR_ , abs/1805.09590.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Steinbakken and Gambäck (2020) Stian Steinbakken and Björn Gambäck. 2020.  Native-language identification with attention.  In _Proceedings of the 17th International Conference on Natural Language Processing (ICON)_ , pages 261–271, Indian Institute of Technology Patna, Patna, India. NLP Association of India (NLPAI).
  * Sun et al. (2019) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019.  How to fine-tune BERT for text classification? _CoRR_ , abs/1905.05583.
  * Tetreault et al. (2013) Joel Tetreault, Daniel Blanchard, and Aoife Cahill. 2013.  A report on the first native language identification shared task.  In _Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 48–57, Atlanta, Georgia. Association for Computational Linguistics.
  * Tipping and Bishop (1999) Michael E Tipping and Christopher M Bishop. 1999.  Probabilistic principal component analysis.  _Journal of the Royal Statistical Society Series B: Statistical Methodology_ , 61(3):611–622.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Virtanen et al. (2020) Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. 2020.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ , 17:261–272.
  * Volansky et al. (2015) Vered Volansky, Noam Ordan, and Shuly Wintner. 2015.  On the features of translationese.  _Digit. Scholarsh. Humanit._ , 30:98–118.
  * Ward Jr (1963) Joe H Ward Jr. 1963.  Hierarchical grouping to optimize an objective function.  _Journal of the American statistical association_ , 58(301):236–244.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _CoRR_ , abs/1910.03771.