# Fostering the Ecosystem of Open Neural Encoders for Portuguese with
Albertina PT* Family

###### Abstract

To foster the neural encoding of Portuguese, this paper contributes foundation
encoder models that represent an expansion of the still very scarce ecosystem
of large language models specifically developed for this language that are
fully open, in the sense that they are open source and openly distributed for
free under an open license for any purpose, thus including research and
commercial usages. Like most languages other than English, Portuguese is low-
resourced in terms of these foundational language resources, there being the
inaugural 900 million parameter Albertina and 335 million Bertimbau. Taking
this couple of models as an inaugural set, we present the extension of the
ecosystem of state-of-the-art open encoders for Portuguese with a larger, top
performance-driven model with 1.5 billion parameters, and a smaller,
efficiency-driven model with 100 million parameters. While achieving this
primary goal, further results that are relevant for this ecosystem were
obtained as well, namely new datasets for Portuguese based on the SuperGLUE
benchmark, which we also distribute openly.

Keywords: Large language model, foundation model, encoder, Portuguese, open-
source

Fostering the Ecosystem of Open Neural Encoders for Portuguese with Albertina
PT* Family

Rodrigo Santos†, João Rodrigues†, Luís Gomes†, João Silva†, António Branco†,
---
Henrique Lopes Cardoso‡, Tomás Freitas Osório‡, Bernardo Leite‡
†University of Lisbon
NLX - Natural Language and Speech Group, Department of Informatics
Faculdade de Ciências, Campo Grande, 1749-016 Lisboa, Portugal
{rsdsantos, jarodrigues, luis.gomes<EMAIL_ADDRESS>
‡University of Porto
Faculty of Engineering, Department of Informatics Engineering
Rua Dr. Roberto Frias, 4200-465 Porto, Portugal
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

The present paper contributes foundation models that represent the development
and the populating of the still very scarce ecosystem of fully open large
language models of the encoder family of Transformers specifically developed
for the Portuguese language, that is models that are open source and openly
distributed with for free with an open license.

Since their appearance in (Vaswani et al., 2017) and given their superior
performance vis a vis their viable alternatives, neural language models based
on the Transformer architecture became the mainstream approach for virtually
any natural language processing task (Brown et al., 2020; Raffel et al., 2020;
He et al., 2021). Transformers were proposed in an encoder-decoder setup
(Raffel et al., 2020), but encoder-only and decoder-only setups have also been
shown highly competitive by subsequent research (Devlin et al., 2019; He et
al., 2021; Brown et al., 2020).

Despite the outstanding visibility that the Transformer-based decoder models
have deservedly garnered, especially with the availability of ChatGPT for the
general public, the models of the encoder family have not lost their traction
as they have maintained a competitive performance in non-generative tasks,
especially in those tasks primarily related to classification (He et al.,
2021; Zhong et al., 2022).111At the time of writing, as a way of confirmation
of this remark, the top performing model in the SuperGLUE benchmark
(https://super.gluebenchmark.com/leaderboard) is an encoder, namely the Vega
v2 model (Zhong et al., 2022).

The largest and more powerful foundation models have been developed for
English — (He et al., 2021; Touvron et al., 2023) among many others —, which
is the language that, among the more than 7 000 idioms on the planet, is by a
very large margin the one whose research is better funded, better
technologically prepared for the digital age and for which more language
resources have been developed (Rehm and Way, 2023).

Additionally, multilingual models have also been developed, whose training is
done over datasets that extend its majority of English data with
proportionally much smaller data portions from a few other languages (Devlin
et al., 2019; Chowdhery et al., 2022; Scao et al., 2022). Leveraged by the
sheer volume of data thus made available, these models have shown competitive
performance in handling tasks in the languages, other than English, whose data
portions are a minority in their training set (Wu and Dredze, 2019).

On par with these results and their relevance for some multilingual natural
language tasks, especially machine translation, other approaches have been
explored, namely with the continuation of the pre-training of multilingual or
plain English models with data from a specific language. Reported results seem
to converge in indicating that when their continued training is appropriately
setup, the performance of the resulting models on language-specific tasks
shows important improvements over a possible baseline model whose training was
performed from scratch with the same (comparatively small) amount of language-
specific data (Kim et al., 2021; Pires et al., 2023; Rodrigues et al., 2023).

Adopting this latter approach and adding to the previous work on the neural
encoding of Portuguese (Rodrigues et al., 2023; Souza et al., 2020), the
present paper puts forward further models for this language that expand its
ecosystem of open encoders. These encoders cumulatively comply with all the
features of being open source, publicly available for free, and distributed
under a most permissive license (including for research and for commercial
purposes). Furthermore, they are available for two variants of Portuguese:
European Portuguese, spoken in Portugal (PTPT), and American Portuguese,
spoken in Brazil (PTBR).

Taking as reference the existing state-of-the-art 900 million parameter
encoder Albertina (Rodrigues et al., 2023), which complies with all the above
requirements, in this paper we present the extension of the ecosystem of open
encoders for Portuguese with a larger, top performance-driven encoder model
with 1.5 billion parameters, Albertina 1.5B PT, and a smaller, efficiency-
driven encoder model with 100 million parameters, Albertina 100M PT. These
models are distributed from https://huggingface.co/PORTULAN.

While achieving these central goal, further results that are relevant for this
ecosystem were obtained as well: new datasets for Portuguese based on the
trusted GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) benchmarks,
which are distributed openly; and state-of-the-art performance for Portuguese
in various natural language processing tasks in these benchmarks.

The remainder of this paper is structured as follows: the next Section 2
discusses related work. In Section 3 the data used in the creation of the
various models is presented; the encoder models created in this study are
described in Section 4; Section 5 presents the evaluation results; and Section
6 closes the paper with concluding remarks.

## 2\. Related work

The advent of the Transformer architecture (Vaswani et al., 2017) represents a
revolutionary milestone in the field of Natural Language Processing. With its
attention mechanisms, the Transformer enabled the efficient modeling of
contextual information in text, paving the way for the development of powerful
models.

The success of this architecture led to the emergence of various encoder
models, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and
DeBERTa (He et al., 2021), which set new standards for language comprehension
tasks. Nevertheless, they cater exclusively for the English language.

To address linguistic diversity, multilingual encoder models emerged as a
promising solution. Notable examples include mBERT (Devlin et al., 2019), XLM
(Conneau and Lample, 2019) and XLM-R (Conneau et al., 2020), among others,
which support multiple languages and seek to bridge language barriers.

In contrast, a few encoder models that cater for specific languages have also
been introduced. For instance, ERNIE (Sun et al., 2021) for Chinese, CamemBERT
(Martin et al., 2020) for French, and MarIA (Gutiérrez-Fandiño et al., 2022)
for Spanish, among others. These have demonstrated the importance of language-
tailored models in capturing language-specific nuances, which multilingual
models cannot so easily ensure (Papadimitriou et al., 2023).

Concerning Portuguese, previous encoder models such as the 900 million
parameter Albertina (Rodrigues et al., 2023) and the 335 million parameter
BERTimbau (Souza et al., 2020) have made significant contributions. With
BERTimbau covering PTBR, and Albertina covering both PTPT and PTBR variants,
these models have not only bolstered the Portuguese NLP ecosystem but have
also set the path for the development of more advanced language models
tailored to the Portuguese language.

In this paper, we aim at adding to this existing work by contributing further
encoder models with further dimensions, also covering both the European PTPT
and the American PTBR variants of Portuguese.

## 3\. Data

In this section, we present the data used for the training and testing of our
encoder models.

In both their variants, PTBR and PTPT, for our smaller, 100 million parameter
model, we resort to the Portuguese subset of the OSCAR dataset (Abadji et al.,
2022). And for our larger, 1.5 billion parameter model, we resort to the
Portuguese subset of the CulturaX dataset (Nguyen et al., 2023). Additionally,
for the models handling the PTPT variants, the dataset we used included also
the monolingual corpora DCEP, ParlamentoPT and Europarl (Hajlaoui et al.,
2014; Koehn, 2005; Rodrigues et al., 2023).

These corpora and their curation are described in detail below in the next
Subsection, and their sizes are summarized in Table 1.

dataset | exs (M) | words (B)
---|---|---
Albertina 100M PTPT | 10.2 | 2.4
Albertina 100M PTBR | 4.1 | 2.7
Albertina 1.5B PTPT | 16.1 | 4.3
Albertina 1.5B PTBR | 87.9 | 36.2
Table 1: Size of datasets used for training, in millions of examples (exs) and
in billions of words.

### 3.1. Training Data

While both multilingual datasets, OSCAR and CulturaX, distribute their
Portuguese subsets separately, they do not provide further separation between
European Portuguese and American Portuguese within these subsets. To separate
the texts in one variant from the texts in the other, we use the source URLs
provided with every data entry and filter by top-level domain. We only keep
entries with the “.br” top-level domain, and add them to the PTBR subset, and
with the “.pt” top-level domain, for the PTPT subset.

From these datasets, data entries of domains whose content should not be
redistributed were removed, in order to limit the possibility of content
reproduction by the models or by future derivatives that will resort to these
datasets.

#### OSCAR Corpus

The project promoting the OSCAR corpus is an open source project which
distributes multilingual datasets for machine learning and artificial
intelligence applications Abadji et al. (2022).

The OSCAR subset for Portuguese we use is based on November/December 2022
version of Common Crawl, which is an automatic crawl from the web. Despite
being a crawl, the final dataset is of relatively good quality due the
filtering performed on the corpus by its authors. As can be seen in Table 2,
we end up with subsets of OSCAR for the two Portuguese variants that have a
not too distinct number of examples and words.

#### CulturaX Corpus

CulturaX is a multilingual corpus, freely available for research and AI
development Nguyen et al. (2023), created by combining and extensively
cleaning two other large datasets, mC4 (Xue et al., 2021) and OSCAR.

The CulturaX subset for PTBR is an order of magnitude larger than for PTPT, as
depicted in Table 2, both in examples and words. This does not present itself
as a problem since we aim to develop the best model possible for each variant.

#### Other Corpora

In addition to the above language resources, for the European Portuguese
versions we also include in our training set: (i) the Portuguese portion of
DCEP (Hajlaoui et al., 2014), a Digital Corpus of the European Parliament;
(ii) the Portuguese portion of Europarl (Koehn, 2005), the European Parliament
Proceedings Parallel Corpus; and (iii) ParlamentoPT (Rodrigues et al., 2023),
a corpus of transcriptions of the debates in the Portuguese Parliament.

These corpora are based on human transcriptions of parliamentary debates and
can be assumed to be of very high quality, despite their limited domain. They
provide a good complement to OSCAR and CulturaX.

####

Finally, we apply further quality filtering to all corpora—except to CulturaX,
since it already has a good quality filtering step—, through the use of the
Bloom pre-processing pipeline (Laurençon et al., 2022).

Table 2 presents statistics for all the corpora used in this work; all these
numbers are calculated right before training the model, i.e. after splitting
between variants and applying all types of additional content filtering.

dataset | examples (M) | words (M)
---|---|---
OSCAR ptbr | 4.1 | 2,728
OSCAR ptpt | 3.0 | 1,976
CulturaX ptbr | 87.9 | 36,201
CulturaX ptpt | 8.9 | 3,896
DCEP | 2.5 | 76
ParlamentoPT | 2.9 | 289
Europarl | 1.8 | 49
Table 2: Number of examples and words for each dataset for training

### 3.2. Testing Data

The performance of encoder models are typically undertaken by testing them in
downstream tasks. For the Portuguese language, both variants, there is however
a lack of such datasets, either in quality or in quantity, to appropriately
evaluate an encoder models. The only dataset created from scratch in
(American) Portuguese, that we could find, is the ASSIN 2 dataset (Real et
al., 2020) that was used to evaluate BERTimbau.

To cope with this hindrance, we contribute new test datasets for Portuguese
based on the GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019)
benchmarks.

We obtain these datasets through machine translation from English using
DeepL,222https://www.deepl.com/ which allows translation either to PTPT or to
PTBR, and is regarded as one of the best machine translation services
available online.333The construction is thoroughly presented in (Osório et
al., submited)

The exception to this translation process, concerns the PTBR portion of GLUE,
which we took from PLUE (Gomes, 2020), to avoid redoing valid work already
present in the literature and openly distributed.

#### ASSIN 2 tasks

The ASSIN 2 dataset contains two tasks: (i) RTE, for recognizing textual
entailment, and (ii) STS, for semanting textual similarity.

#### GLUE tasks

From GLUE we chose four tasks: two similarity tasks, (i) MRPC, for detecting
whether two sentences are paraphrases of each other, and (ii) STS-B, for
semantic textual similarity; and two inference tasks, (iii) RTE, for
recognizing textual entailment, and (iv) WNLI, for coreference and natural
language inference.

#### SuperGLUE tasks

As for SuperGlue, we also chose four tasks: two QA tasks, (i) MultiRC, for
detecting whether an answer to a question about a paragraph is correct or not,
and (ii) BoolQ, for answering yes or no to a question about a passage; one
reasoning task, (ii) COPA, given a premise sentence and two possible choices,
the system must determine either the cause or effect of the premise from two
possible choices; and one inference task with three labels, (iv) CB, for
predicting how much the text commits to the clause.

## 4\. Models

This section describes the training of the models contributed in this paper.

### 4.1. The starting models

We use DeBERTa (He et al., 2021) as a starting point from which to continue
the pre-training of our models over Portuguese data. This is an encoder that
incorporates a new attention mechanism, making it particularly effective for a
wide range of natural language processing tasks. DeBERTa’s architecture
disentangles attention patterns, improving its ability to capture
relationships between words and phrases in a text.

With its different model sizes, including the compact DeBERTa-Base with 100
million parameters, the DeBERTa-XLarge with 900 million parameters, and the
high-capacity DeBERTa-XXLarge with 1.5 billion parameters, it caters for
various NLP requirements.

The only encoder for both variants PTP and PTBR variants of Portuguese, the
existing 900 million parameter model Albertina, was obtained by continuing the
pre-training of DeBERTa-XLarge with Portuguese (Rodrigues et al., 2023).

With the same goal in mind, we start from the DeBERTa-Base to construct our
Albertina 100M PT models, and from the DeBERTa-XXLarge, for our Albertina 1.5B
PT models.

### 4.2. The Albertina 100M PT foundation model

The two smaller models, Albertina 100M PTPT and Albertina 100M PTBR, are
constructed upon the DeBERTa Base V1 model, comprising 100 million parameters.

The models were trained on a a2-megagpu-16gb Google Cloud A2 node equipped
with 16 GPUs, 96 vCPUs, and 1.360 GB of RAM, and their training took
approximately one day of compute. This configuration resulted in a batch size
of 3072 samples, with 192 samples allocated per GPU, when trying to fill the
whole memory available.

We used the original DeBERTa tokenizer for both models, implementing a
128-token sequence truncation and dynamic padding. The training was performed
under a learning rate of 1e-5, with linear decay and 10k warm-up steps,
determined after a few exploratory trials. The PTPT model underwent 200
training epochs, while the PTBR model underwent 150, accumulating roughly 180k
training steps in each case.

### 4.3. The Albertina 1.5B PT foundation model

As for the larger models, Albertina 1.5B PTPT and PTBR, we developed them upon
the DeBERTa XXLarge V2 encoder, comprising 1.5 billion parameters.

Similarly to the smaller models, the two Albertina 1.5B PTmodels were trained
on a a2-megagpu-16gb Google Cloud A2 node.

We resorted to the original DeBERTa V2 tokenizer for both models, implementing
a 128-token sequence truncation and dynamic padding for 250k steps, a
256-token sequence-truncation for 80k steps and finally a 512-token sequence-
truncation for 60k steps. These steps correspond to the equivalent setup of 48
hours on a2-megagpu-16gb Google Cloud A2 node for the 128-token input
sequences, 24 hours of computation for the 256-token input sequences and 24
hours of computation for the 512-token input sequences.

We applied a learning rate of 1e-5, with linear decay and 10k warm-up steps,
determined after a few exploratory trials

## 5\. Evaluation and discussion

This section presents and discusses the evaluation of our models, introduced
just above in Section 4, with respect to the downstream tasks, introduced in
Section 3.2, after their fine-tuning on these tasks.

Additionally, for the sake of a thorough comparative evaluation of these
models, this section also presents the results of fine-tuning and evaluating
in the same downstream tasks, the pre-existing models in the ecosystem of
encoders for Portuguese, namely the 900 million parameter Albertina and the
335 million parameter BERTimbau We also evaluate with the two DeBERTa baseline
models, with 100 million and 1.5 billion parameter, trained mostly with
English data, which we did not continue the training on further Portuguese
data.

The compilation of all these results are in Table 4, for the model versions
concerning the PTBR variant, and Table 5, for the PTPT variant.

| ASSIN2
---|---
model | RTE | STS
Albertina 1.5B PTBR L | 0.9153 | 0.8647
Albertina 1.5B PTBR S | 0.9109 | 0.8688
Albertina 900M PTBR | 0.9130 | 0.8676
BERTimbau (335M) | 0.8913 | 0.8531
Albertina 100M PTBR | 0.8747 | 0.8269
DeBERTa 1.5B EN | 0.8803 | 0.8356
DeBERTa 100M EN | 0.8369 | 0.7760
Table 3: Evaluation scores for PTBR on the ASSIN2 native American Portuguese dataset. Performance on RTE is measured with accuracy and on STS with Pearson. | GLUE | SuperGLUE
---|---|---
model | RTE | WNLI | MRPC | STS-B | COPA | CB | MultiRC | BoolQ
Albertina 1.5B PTBR L | 0.8676 | 0.4742 | 0.8622 | 0.9007 | 0.7767 | 0.6372 | 0.7667 | 0.8654
Albertina 1.5B PTBR S | 0.8123 | 0.4225 | 0.8638 | 0.8968 | 0.8533 | 0.6884 | 0.6799 | 0.8509
Albertina 900M PTBR | 0.7545 | 0.4601 | 0.9071 | 0.8910 | 0.7767 | 0.5799 | 0.6731 | 0.8385
BERTimbau (335M) | 0.6446 | 0.5634 | 0.8873 | 0.8842 | 0.6933 | 0.5438 | 0.6787 | 0.7783
Albertina 100M PTBR | 0.6582 | 0.5634 | 0.8149 | 0.8489 | n.a. | 0.4771 | 0.6469 | 0.7537
DeBERTa 1.5B EN | 0.7810 | 0.4789 | 0.8555 | 0.8600 | 0.4733 | 0.4648 | 0.6738 | 0.8315
DeBERTa 100M EN | 0.5716 | 0.5587 | 0.8060 | 0.8266 | n.a. | 0.4739 | 0.6391 | 0.6838
Table 4: Evaluation scores for PTBR. Performance on RTE, WNLI, BoolQ and COPA is measured with accuracy, on MRPC, MultiRC and CB with F1, and on STS-B with Pearson. | GLUE | SuperGLUE
---|---|---
model | RTE | WNLI | MRPC | STS-B | COPA | CB | MultiRC | BoolQ
Albertina 1.5B PTPT L | 0.8809 | 0.4742 | 0.8457 | 0.9034 | 0.8433 | 0.7840 | 0.7688 | 0.8602
Albertina 1.5B PTPT S | 0.8809 | 0.5493 | 0.8752 | 0.8795 | 0.8400 | 0.5832 | 0.6791 | 0.8496
Albertina 900M PTBR | 0.8339 | 0.4225 | 0.9171 | 0.8801 | 0.7033 | 0.6018 | 0.6728 | 0.8224
Albertina 100M PTPT | 0.6919 | 0.4742 | 0.8047 | 0.8590 | n.a. | 0.4529 | 0.6481 | 0.7578
DeBERTa 1.5B EN | 0.8147 | 0.4554 | 0.8696 | 0.8557 | 0.5167 | 0.4901 | 0.6687 | 0.8347
DeBERTa 100M EN | 0.6029 | 0.5634 | 0.7802 | 0.8320 | n.a. | 0.4698 | 0.6368 | 0.6829
Table 5: Evaluation scores for PTPT. Performance on RTE, WNLI, BoolQ and COPA
is measured with accuracy, on MRPC, MultiRC and CB with F1, and on STS-B with
Pearson.

### 5.1. Fine-tuning

Each model under evaluation was fine-tuned on each of the eight downstream
tasks obtained from GLUE and SuperGLUE and introduced in Section 3.2.444The
exception were the 100M DeBERTa models (DeBERTa-base and both versions of
Albertina 100M PT), which were not evaluated on the COPA task because the
Hugging Face head for multiple choice does not support DeBERTa v1 models. In
order to proceed with hyper-parameter optimization, the following hyper-
parameter values were chosen for our grid-search:

  * •

Epochs: $5$

  * •

Batch size: $4$

  * •

Learning rate: $\\{1\times 10^{-5},5\times 10^{-5},1\times 10^{-6}\\}$

  * •

Learning rate scheduler type: linear

  * •

Warm up ratio: $0.1$

  * •

Adam epsilon: $1\times 10^{-6}$

  * •

Weight decay: $0.01$

  * •

Dropout: $\\{0,0.1\\}$

  * •

BF16: $\\{0,1\\}$

A hyper-parameter grid search was performed for each pre-trained model/task
combination, resulting in a total of 4104 fine-tuned and evaluated models.
This number results from 12 combinations of hyper-parameter values (3 learning
rates $\times$ 2 dropout values $\times$ 2 BF16 values), times the number of
tasks (10 for PT-BR and 8 for PT-PT), times the number of evaluated pre-
trained models555The 100M parameter models could not be evaluated in the COPA
task for lack of support for these models in the HuggingFace head
implementation for this task. (7 for PT-BR and 6 for PT-PT), times 3 random
seeds.

As presented in Section 3, the GLUE and SuperGLUE evaluation datasets were
translated into both Portuguese variants from their English originals.

It is noteworthy that the test sets from from the GLUE and SuperGLUE datasets
are not distributed with ground labels, as evaluation is setup to proceed by
submitting online the data to be evaluated. Given that the number of such
online submissions per month for each user is highly limited and very small,
and given the very large number of models and tasks and thus of evaluation
runs we needed to cope with, it was not practically viable to resort to such
online evaluation service. As a consequence, to proceed with our very large
experimental space, we adopted the same methodology as we did for the 900
million parameter Albertina Rodrigues et al. (2023): we used the validation
partitions of the downstream datasets for testing; and for training, we
randomly split the partition that is originally distributed for training into
90% that we used for actual training and into the remaining 10% that we used
for development and validation purposes.

After acquiring the best hyper-parameter values on the data that were set
aside for development purposes and by using such hyper-parameters, the
performance scores were obtained by testing on the subsets that were left for
evaluation, which are displayed in Tables 3, 4 and 5. The values presented are
the average scores of 3 runs with different random seeds.

### 5.2. Albertina 1.5B PT fine-tuned

Since most tasks have input sizes closer to 256 than to 512, we evaluated two
variants of the Albertina 1.5B PT model: the models with suffix S (short) in
Tables 4 and 5 are fine-tuned from checkpoints after pre-training with
sequences of 256 tokens; while the models with suffix L (long) are fine-tuned
from the final checkpoints, i.e. after pre-training with sequences of 512
tokens.

In almost all tasks and for both language variants, our largest model, with
1.5 billion parameters, shows the best performance scores, and in the few
cases where that is not the case, it competitively come close to the best
scoring model.

It is of note that among the downstream tasks, WNLI appears somehow as an
outlier as the performance level of the different models on it is not aligned
with their performance level in the other tasks. This has been already
observed also with Albertina 900 M Rodrigues et al. (2023), which attributed
this to the very small size of the WNLI dataset.

In its overall performance, this largest model surpasses the previously best
model Albertina 900M in this ecosystem, and offers thus the state-of-the-art
performance in most tasks for Portuguese by an open encoder.

### 5.3. Albertina 100M PT fine-tuned

With 100 million parameters, our Albertina 100M PT model is the smallest in
this ecosystem of open encoders for Portuguese. Yet, it has very good
performance taking into account its reduced size.

Taking WNLI aside, Albertina 100M PT matches or surpasses its base model
(DeBERTa 100M) in all 16 tasks, except in CB for PTPT.

On the other hand, our Albertina 100M PTBR is very competitive with respect to
the BERTimbau model, whose 335 million parameters are more than the triple of
its size. It surpasses BERTimbau’s performance in GLUE’s RTE, and supports a
very competitive second position in most of the other tasks. Likely, this is
the consequence of BERTimbau having BERT Devlin et al. (2019) as its base
model, while Albertina 100M PTis based in the more advanced DeBERTa He et al.
(2021).

### 5.4. Discussion

#### The larger the better

Taking a broad view of the results in Tables 4 and 5, overall and as expected,
the larger the Albertina model the better is its performance in downstream
tasks.

In this respect, and taking aside WNLI, already commented on above, the
exception to this trend is MRPC. In this task, the 1.5B Albertina models are
outperformed by the smaller 900M Albertinas. Although we don’t have a
compelling explanation for this, it appears that the 900M parameter network
may provide the optimal expressive power for learning this particular task and
dataset, across the various model sizes under evaluation.

#### The more monolingual the better

When compared to their respective DeBERTa baseline counterparts, our newly
contributed models, Albertina 1.5B PTand Albertina 100M PT, present superior
performance in general.

This adds to the empirical evidence in the literature, commented in Section 2,
for the importance of continuing the pre-training of models with monolingual
data for the language of interest, even if they started multilingual or were
initially developed for another language. If appropriately prepared, the
resulting models typically represent a better solution for that language.

Concerning the largest model Abertina 1.5B, and taking aside the WNLI outlier,
it always improves over its baseline model.

As for our smaller model Albertina 100M, the exception to this trend appears
once again in WNLI, for PTPT, and CB, by a small margin, also for PTPT.

#### The more advanced the base model the better

Comparing the new Albertina 100M PT and Albertina 1.5B PT models to the
previously existing models, it is clear that the larger models offer
improvements over smaller models as noted above.

However, it is important also to note that the difference between the
performance scores of Albertina 100M PTBR and of the 335M BERTimbau is rather
small, which seems to suggest that the improvements in DeBERTa, on which our
Albertina 100M PTis based, over BERT, which used as a base model by BERTimbau,
have allowed for more efficient parameter utilization and improved performance
in general.

#### The more language variants the better

For the same task and the same model dimension, the models for the European
PTPT and American PTBR variants of Portuguese show different performance
scores. While in general not representing a wide gap, these differences exist,
as expected.

These differences should be attributed, for instance, to the possible
different quality of the translations produced for the English datasets,
depending on the Portuguese variant, and also attributed in some cases to the
different sizes of the training corpora, etc. For instance, the training of
the 1.5 billion model for PTBR was based on a 36.2 billion token dataset,
while the same size model for PTPT resorted to a much smaller, 4.3 billion
token corpus, as indicated in Table 1.

From the three models with two versions, i.e. one version per variant, namely,
the Albertina 100M, 900M and 1.5B models, it is the 900M one than may permit a
more insightful comparison among its two variants given the conditions of
their training were closer to each other, with a 2.7M and a 2.2M token
training dataset for PTBR and PTPT, respectively Rodrigues et al. (2023).

Thus looking to the experimental results we obtained for the two Albertina
900M versions, PTBR and PTPT, across the Tables 4 and 5, one finds deltas, for
instance, of 0.079 (accuracy) in RTE, 0.073 (F1) in COPA, or 0.022 (accuracy)
in CB. This is in line with the same lessons drawn in Rodrigues et al. (2023),
and it is confirming its results. It is thus relevant to keep the two variants
of Portuguese addressed by different model versions if possible.

## 6\. Conclusions

The results reported in the present paper demonstrate that the models hereby
contributed represent valuable advances for the ecosystem of fully open large
language models of Portuguese.

With its 1.5 billion parameters, Albertina 1.5B PT becomes the largest open
encoder specifically developed for this language, and the one that better
support state of the art performance in downstream tasks.

With its 100 million parameter, Albertina 100M PT becomes, in turn, the
smallest, appropriately curated and documented, open encoder of this
ecosystem, and thus the one that ensures an encoding solution for this
language that favours efficiency and is available to run in limited hardware.

It is also worth noting that the advancements contributed in this paper for
both American and European variants of Portuguese cater for the linguistic
diversity in this language, ensuring their relevance and applicability to a
broad user base.

In conclusion, this paper presents a significant contribution to the field of
language technology for Portuguese by introducing state-of-the-art large
language models that serve the technological preparation of this language. The
models are not only technically robust but also fully open, in the sense that
are open source, openly distributed for free under an open license for both
research and commercial purposes. They are adaptable for various applications,
thus facilitating innovation and progress in the field.

These models can be obtained from https://huggingface.co/PORTULAN.

Future work will include further expanding and updating this ecosystem of
fully open encoders for Portuguese with other model dimensions, other language
variants and other design features.

## Acknowledgements

This research was partially supported by: PORTULAN CLARIN — Research
Infrastructure for the Science and Technology of Language, funded by Lisboa
2020, Alentejo 2020 and FCT (PINFRA/22117/2016); ACCELERAT.AI - Multilingual
Intelligent Contact Centers, funded by IAPMEI (C625734525-00462629); ALBERTINA
- Foundation Encoder Model for Portuguese and AI, funded by FCT (CPCA-
IAC/AV/478394/2022); and LIACC - Artificial Intelligence and Computer Science
Laboratory (FCT/UID/CEC/0027/2020).

*

## 7\. Bibliographical References

##

  * Abadji et al. (2022) Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022.  Towards a cleaner document-oriented multilingual crawled corpus.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC)_ , pages 4344–4355.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in Neural Information Processing Systems_ , 33:1877–1901.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Gomes (2020) J. R. S. Gomes. 2020.  Plue: Portuguese language understanding evaluation.  https://github.com/ju-resplande/PLUE.
  * Gutiérrez-Fandiño et al. (2022) Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Marc Pàmies, Joan Llop-Palao, Joaquin Silveira-Ocampo, Casimiro Pio Carrino, Carme Armentano-Oller, Carlos Rodriguez-Penagos, Aitor Gonzalez-Agirre, and Marta Villegas. 2022.  MarIA: Spanish language models.  _Procesamiento del Lenguaje Natural_ , pages 39–60.
  * Hajlaoui et al. (2014) Najeh Hajlaoui, David Kolovratnik, Jaakko Väyrynen, Ralf Steinberger, and Daniel Varga. 2014.  DCEP—Digital corpus of the European Parliament.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021.  DeBERTa: Decoding-enhanced BERT with disentangled attention.  In _International Conference on Learning Representations_.
  * Kim et al. (2021) Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Jeon Dong Hyeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, Suk Hyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang, Na-Hyeon Ryu, Kang Min Yoo, Minsuk Chang, Soobin Suh, Sookyo In, Jinseong Park, Kyungduk Kim, Hiun Kim, Jisu Jeong, Yong Goo Yeo, Donghoon Ham, Dongju Park, Min Young Lee, Jaewook Kang, Inho Kang, Jung-Woo Ha, Woomyoung Park, and Nako Sung. 2021.  What changes can large-scale language models bring? Intensive study on HyperCLOVA: Billions-scale Korean generative pretrained transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3405–3424, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A parallel corpus for statistical machine translation.  In _Proceedings of Machine Translation Summit X: papers_ , pages 79–86.
  * Laurençon et al. (2022) Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. 2022.  The BigScience ROOTS corpus: A 1.6 TB composite multilingual dataset.  _Advances in Neural Information Processing Systems_ , 35:31809–31826.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Martin et al. (2020) Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020.  CamemBERT: a tasty French language model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7203–7219.
  * Nguyen et al. (2023) Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. 2023.  CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages.  _arXiv preprint arXiv:2309.09400_.
  * Osório et al. (submited) Tomás Freitas Osório, Bernardo Leite, Henrique Lopes Cardoso, Luís Gomes, João Rodrigues, Rodrigo Santos, and António Branco. submited.  Extraglue datasets and models: Kick-starting a benchmark for the neural processing of portuguese.
  * Papadimitriou et al. (2023) Isabel Papadimitriou, Kezia Lopez, and Dan Jurafsky. 2023.  Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models.  In _Proceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP_ , pages 143–146. Association for Computational Linguistics.
  * Pires et al. (2023) Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira. 2023.  Sabiá: Portuguese large language models.  _arXiv preprint arXiv:2304.07880_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21:5485–5551.
  * Real et al. (2020) Livy Real, Erick Fonseca, and Hugo Gonçalo Oliveira. 2020.  The assin 2 shared task: a quick overview.  In _Computational Processing of the Portuguese Language: 14th International Conference, PROPOR 2020, Evora, Portugal, March 2–4, 2020, Proceedings 14_ , pages 406–412. Springer.
  * Rehm and Way (2023) Georg Rehm and Andy Way, editors. 2023.  _European Language Equality: A Strategic Agenda for Digital Language Equality_.  Cognitive Technologies. Springer.
  * Rodrigues et al. (2023) João Rodrigues, Luís Gomes, João Silva, António Branco, Rodrigo Santos, Henrique Lopes Cardoso, and Tomás Osório. 2023.  Advancing neural encoding of Portuguese with transformer Albertina PT-*.  In _Progress in Artificial Intelligence (EPIA 2023)_.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Souza et al. (2020) Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020.  BERTimbau: Pretrained BERT Models for Brazilian Portuguese.  In _Intelligent Systems_ , pages 403–417.
  * Sun et al. (2021) Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. 2021.  Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation.  _arXiv preprint arXiv:2107.02137_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _Advances in neural information processing systems_ , 32.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355.
  * Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019.  Beto, Bentz, Becas: The surprising cross-lingual effectiveness of BERT.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 833–844, Hong Kong, China. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498.
  * Zhong et al. (2022) Qihuang Zhong, Liang Ding, Yibing Zhan, Yu Qiao, Yonggang Wen, Li Shen, Juhua Liu, Baosheng Yu, Bo Du, Yixin Chen, et al. 2022.  Toward efficient language model pretraining and downstream adaptation via self-evolution: A case study on SuperGLUE.  _arXiv preprint arXiv:2212.01853_.

## 8\. Language Resource References

##

  * Fábio Souza and Rodrigo Nogueira and Roberto Lotufo (2020) Fábio Souza and Rodrigo Nogueira and Roberto Lotufo. 2020.  _BERTimbau Large_.  Hugging Face.
  * Gomes (2020) J. R. S. Gomes. 2020.  _PLUE: Portuguese Language Understanding Evaluation_.  Hugging Face.
  * Hajlaoui Najeh, Kolovratnik David, Vaeyrynen Jaakko, Steinberger Ralf, and Varga Dániel (2012) Hajlaoui Najeh, Kolovratnik David, Vaeyrynen Jaakko, Steinberger Ralf, and Varga Dániel. 2012.  _DCEP: Digital Corpus of the European Parliament_.  European Parliament - DG TRAD. European Parliament - DG TRAD, ISLRN 823-807-024-162-2.
  * João Rodrigues and Luís Gomes and João Silva and António Branco and Rodrigo Santos and Henrique Lopes Cardoso and Tomás Osório (2023a) João Rodrigues and Luís Gomes and João Silva and António Branco and Rodrigo Santos and Henrique Lopes Cardoso and Tomás Osório. 2023a.  _Albertina PT-BR_.  PORTULAN CLARIN. distributed via PORTULAN CLARIN.  PID https://hdl.handle.net/21.11129/0000-000F-FF43-7.
  * João Rodrigues and Luís Gomes and João Silva and António Branco and Rodrigo Santos and Henrique Lopes Cardoso and Tomás Osório (2023b) João Rodrigues and Luís Gomes and João Silva and António Branco and Rodrigo Santos and Henrique Lopes Cardoso and Tomás Osório. 2023b.  _Albertina PT-PT_.  PORTULAN CLARIN. distributed via PORTULAN CLARIN.  PID https://hdl.handle.net/21.11129/0000-000F-FF42-8.
  * Julien Abadji and Pedro Ortiz Suarez and Laurent Romary and Benoît Sagot (2023) Julien Abadji and Pedro Ortiz Suarez and Laurent Romary and Benoît Sagot. 2023.  _OSCAR 23.01 – Open Source Project on Multilingual Resources for Machine Learning_.  the OSCAR project.
  * Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen (2023) Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen. 2023.  _DeBERTa: Decoding-enhanced BERT with Disentangled Attention_.  Microsoft.
  * Philipp Koehn (2012) Philipp Koehn. 2012.  _European Parliament Proceedings Parallel Corpus (v7)_.  EuroMatrixPlus project.
  * Real et al. (2020) Real, Livy and Fonseca, Erick and Gonçalo Oliveira, Hugo. 2020.  _ASSIN 2 (The ASSIN 2 Shared Task: A Quick Overview)_.  Hugging Face.
  * Thuat Nguyen and Chien Van Nguyen and Viet Dac Lai and Hieu Man and Nghia Trung Ngo and Franck Dernoncourt and Ryan A. Rossi and Thien Huu Nguyen (2023) Thuat Nguyen and Chien Van Nguyen and Viet Dac Lai and Hieu Man and Nghia Trung Ngo and Franck Dernoncourt and Ryan A. Rossi and Thien Huu Nguyen. 2023.  _CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages_.  Hugging Face.
  * Wang et al. (2019) Wang, Alex and Pruksachatkun, Yada and Nangia, Nikita and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel. 2019.  _Superglue: A stickier benchmark for general-purpose language understanding systems_.  Hugging Face.
  * Wang et al. (2018) Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel. 2018.  _GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding_.  Hugging Face.


# Foil Diffuser Investigation with GEANT4

Joseph M. Fabritius II, Konstantin Borozdin, Peter Walstrom

###### Abstract

An investigation into the appropriate materials for use as a diffuser foil in
electron radiography was undertaken in GEANT4. Simulations were run using
various refractory materials to determine a material of appropriate Z number
such that energy loss is minimal. The plotted results of angular spread and
energy spread are shown. It is concluded that higher Z number materials such
as tungsten, tantalum, platinum or uranium could be used as diffuser
materials. Also, an investigation into the handling of bremsstrahlung,
multiple coulomb scattering, and ionization in GEANT4 was performed.

##  1 Motivation

In deciding on the best material for a diffuser foil for increasing the
angular spread of the accelerator beam in electron radiography, one must take
into account four physical phenomena:

  1. 1.

Multiple Coulomb scattering (the desired effect, which increases the angular
spread of the beam). This is mostly due to elastic scattering from nuclei and
is approximately described by the Particle Data Group (PDG) formula 27.14.

  2. 2.

Ionization energy loss and straggling (the part of the energy loss not due to
bremsstrahlung)

  3. 3.

Energy loss due to bremsstrahlung

  4. 4.

Melting temperature

Choosing a diffuser foil for electron radiography experiments requires the
material must be both refractory, that is resistant to melting or deformation
under high temperatures, and to also have a low energy spread so that the
effects of chromatic blur are lessened. Chromatic blurring effects can be
directly seen in the energy loss of the beam through the material. By choosing
a material with less energy loss an appropriate diffuser material can be
found. To accomplish this task several elements were chosen from the Particle
Data Group table on atomic properties of materials. Those materials that were
found to have a melting point of over 1400 K were chosen for investigation:
carbon (graphite), silicon, iron, tantalum, tungsten, platinum, and uranium.

The optimal material Z was found by first choosing the foil thickness of each
refractory material so that a specified angular spread, $\theta_{rms}$, was
achieved. A desired angular spread of 0.2 mRad was chosen as the reference of
comparison between diffuser materials. The diffuser thickness was calculated
using the multiple scattering distribution equation:

$\theta_{rms}=\frac{13.6\textnormal{MeV}}{\beta
cp}z\sqrt{\frac{x}{X_{0}}}\left[1+0.038\frac{x}{X_{0}}\right]$ (1)

where $\beta c$ is the electron speed, $z$ is the particle charge number, $x$
is the thickness of the material, and $X_{0}$ is the radiation length of the
material. The main material dependence is in the factor
$\sqrt{\frac{x}{X_{0}}}$, where $x$ is the thickness in g/cm2 and $X_{0}$ is
the radiation length, also in g/cm2. The above equation was taken from the PDG
journal(2010 pg 290, 27.14) on Multiple scattering through small angles.

The simulations for investigating the foil materials were run in GEANT4 using
the same code for previous electron radiography investigations. A pencil beam
of 12 GeV electrons was fired at a slab of material with a detector situated
just beyond the object for capturing deflected electrons. Secondary particles
were ignored in the detector for these simulations. The geometry is shown in
the figure below.

Figure 1: The test simulation geometry. Figure is not to scale.

Using Equation (1), the initial material thicknesses were used for preliminary
simulations. With these initial simulations the angular distribution of the
beam through the diffuser was plotted in ROOT to verify the spread was 0.2
mRad. The thickness of the material slab was then incrementally adjusted until
the resulting angular spread was 0.2 $\pm$ 0.09 mRad. The calculated and
adjusted thicknesses are presented in the table below.

Material | Z | Melting | Calculated | Adjusted
---|---|---|---|---
| number | Point (K) | Thickness (m) | Thickness (m)
Graphite | 6 | 3600** | 0.007736 | 0.006900
Silicon | 14 | 1687 | 0.003847 | 0.003100
Titanium | 22 | 1941 | 0.001470 | 0.001150
Iron | 26 | 1811 | 0.000722 | 0.000600
Tantalum | 73 | 3293.15 | 0.000168 | 0.000120
Tungsten | 74 | 3695 | 0.000144 | 0.000110
Platinum | 78 | 4098 | 0.000125 | 0.000090
Uranium | 92 | 1408 | 0.000129 | 0.000095
|  | **Sublimation |  |
|  | Temperature |  |

##  2 Plots and Analysis

For each diffuser material, a histogram of the energy loss of the electron
beam through the material was created. When compared along the same energy
range it can be seen that for the lower Z materials there is a much larger
energy loss, evident in the RMS values shown in the plots below. The peak of
the histogram can be seen to decrease as Z number decreases.

Figure 2: Energy loss histograms plotted for the lower Z materials. The energy
scale was focused on the range of 11994 MeV to 12000 MeV, with 500 bins.
Figure 3: Energy loss histograms plotted for the higher Z materials. The
energy scale was focused on the range of 11994 MeV to 12000 MeV, with 500
bins.

Higher Z materials evidently have less energy loss, and will thus make for
better diffusers in electron radiography as the chromatic blur effects will be
lessened than with lower Z. To examine a better comparison of the higher Z
materials the energy histograms were replotted in a smaller interval to focus
on the peak area. From the newer energy plots it is apparent that the peaks
and RMS values are close enough that there is no appreciable difference and
the choice of diffuser material will depend on other criteria, such as
availability of material in foil form, or secondary particle creation. Further
investigative studies will be required.

Figure 4: Energy loss histograms plotted for the higher Z materials. The
energy scale was focused on the range of 11999 MeV to 12000 MeV, with 500
bins.

##  3 GEANT4 Physics Investigation

Apart from the refractory nature of the material, the three other processes
are important in our investigation. The Multiple Coulomb scattering effect is
described by Equation (1) above.

The deterministic part of the ionization energy loss
$\frac{dE}{dx}_{\textit{ion.}}$ (in units of MeV-cm2/g) has a somewhat more
complicated material dependence, including some dependence on the mean
ionization potential of the material, but the main factor is trend is that
$\frac{dE}{dx}_{\textit{ion.}}$ increases as $\frac{Z}{A}$ increases. This
dependence is illustrated by Fig. 5, which is a plot of the minimum
$\frac{dE}{dx}$ for various elements $vs.$ $\frac{Z}{A}$. Random ionization
energy straggling, which is added to the deterministic energy loss, giving a
Landau distribution for thin objects, also increases as $\frac{Z}{A}$
increases.

Bremsstrahlung energy loss is small compared to ionization energy loss for
thin foils. In the thick limit, where an electron emits a substantial number
of “hard” photons, total bremsstrahlung energy loss is proportional to beam
energy, i.e. $\frac{dE}{dx}\approx\frac{E_{0}}{X_{0}}$, where $E_{0}$ is the
incident energy. However, for typical diffuser foils, we are in the “thin”
limit, where the probability of a “hard” bremsstrahlung event is low (the
definition of a hard event is somewhat arbitrary, but it can be taken to be
emission of a photon with an energy of 0.1% of the incident electron energy).

Figure 5: Miniumum ionization $\frac{dE}{dx}$ in MeV-cm2/g vs. $\frac{Z}{A}$
for various materials. The materials in order of increasing $\frac{Z}{A}$ are
U, W, Be, Cu, Al, and C. The outlier is Be.

Using the MCS mean-angle formula and ignoring the log factor, we can write for
the foil thicknesss $x_{\textit{foil}}$ required to get a certain mean MCS
angle $\theta_{0}$, $x_{\textit{foil}}=C(E)X_{0}\theta_{0}^{2}$, where $C(E)$
is approximately material-independent and contains the dependence on the
electron energy. On the other hand, the ionization energy loss distribution
for a particular foil thickness $x$ scales roughly as $\frac{Z}{A}$, so the
ionization energy loss in a foil of a particular material with a thickness
that gives a specified mean scattering angle $\theta_{0}$ is $\Delta
E_{\textit{ion}}\sim\frac{Z}{A}\hskip 3.61371ptX_{0}\theta_{0}^{2}$. Since
both $\frac{Z}{A}$ and $X_{0}$ decrease with increasing atomic weight, this
favors high-Z diffuser materials, provided that their melting temperature is
high.

To investigate the dominant effect in electron deflection within the GEANT4
code a simple scheme was developed. Using the same simulation set-up as the
diffuser investigation, a simplified Physics List was written that only
included the processes G4eBremsstrahlung and G4eMultipleScattering.
Simulations consisted of firing 1 million electrons at a 168 $\mu$m slab of
tantalum. Three separate simulations were run with only bremsstrahlung, only
multiple Coulomb Scattering, and both processes active. Histograms of the
angular spread were plotted and are presented below. It is obvious from the
plots that angular deflection is dominated by multiple Coulomb scattering,
with electron bremsstrahlung only contributing a small amount to the
deflection of the electron as it travels through the tantalum.

Figure 6: Histograms of angular distribution. TOP: Both the G4eBremsstrahlung
and G4eMultipleScattering processes were active for this simulation. BOTTOM
LEFT: Only the G4eMultipleScattering process was active for this simulation.
BOTTOM RIGHT: Only the G4eBremsstrahlung process was active for this
simulation.

After investigating the angular spread effects of the physical processes in
the GEANT4 code we also wanted to confirm the energy loss effects of those
processes. The prior manufactured physics list was modified to include the
G4eIonisation process and simulations were run with all three processes, and
with only G4eIonisation active. The results, shown in the figure below,
confirm that the energy loss of the electrons through the tantalum sample is
dominated by the ionization process.

Figure 7: Histograms of energy loss. TOP: Only G4eBremsstrahlung active, the
majority of electrons( 67.5$\%$) did not lose energy and passed right through
the foil. BOTTOM: Only G4eIonisation active. This is the dominant effect on
energy loss, as seen when compared to the energy loss diagrams using a full
physics list.

We were also curious about the angular dependence of the energy loss from
bremsstrahlung in GEANT4. Histograms were created by plotting logarithmic
angle versus logarithm of total energy and subtracting the electrons final
energy at the detector. The final plot shows there is a correlation between
energy loss and angle, so another simulation was run using 12 MeV electrons
instead of 12 GeV electrons to see how the correlation would change or if the
relation was a static product of a random distribution. Both plots are
presented below, and it can be seen that the relation becomes steeper for
higher energy particles.

Figure 8: Histogram of angle distribution versus energy loss for 12 GeV
electron beam incident on 168 $\mu$m tantalum slab. Figure 9: Histogram of
angle distribution versus energy loss for 12 MeV electron beam incident on 168
$\mu$m tantalum slab.


# WordRep: A Benchmark for Research on Learning Word Representations

Bin Gao  Jiang Bian  Tie-Yan Liu  Microsoft Research

###### Abstract

WordRep is a benchmark collection for the research on learning distributed
word representations (or word embeddings), released by Microsoft Research. In
this paper, we describe the details of the WordRep collection and show how to
use it in different types of machine learning research related to word
embedding. Specifically, we describe how the evaluation tasks in WordRep are
selected, how the data are sampled, and how the evaluation tool is built. We
then compare several state-of-the-art word representations on WordRep, report
their evaluation performance, and make discussions on the results. After that,
we discuss new potential research topics that can be supported by WordRep, in
addition to algorithm comparison. We hope that this paper can help people gain
deeper understanding of WordRep, and enable more interesting research on
learning distributed word representations and related topics.

##  1 Introduction

The success of machine learning methods depends much on data representation,
since different representations may encode different explanatory factors of
variation behind the data. Conventional natural language processing (NLP)
tasks often take the 1-of-$v$ word representation, where $v$ is the size of
the entire vocabulary, and each word in the vocabulary is represented as a
long vector with only one non-zero element. However, such simple form of word
representation meets several challenges. The most critical one is that
1-of-$v$ word representation cannot indicate any relationship between
different words even though they yield high semantic or syntactic correlation.
For example, while _elegant_ and _elegantly_ have quite similar semantics,
their corresponding 1-of-$v$ representation vectors trigger different indexes
to be the hot value, and it is not explicit that _elegant_ is much closer to
_elegantly_ than other words like _rough_ via 1-of-$v$ representations. To
deal with this problem, Latent Semantic Analysis (LSA) (Dumais, ) and Latent
Dirichlet Allocation (LDA) (Blei et al., 2003) were proposed to learn
continuous word representations. Unfortunately, it is quite difficult to train
LSA or LDA model efficiently on large-scale text data.

Recently, with the rapid development of deep learning techniques, researchers
have started to train complex and deep models on large amounts of text corpus,
to learn distributed representations of words (also known as word embeddings)
in the form of continuous vectors (Collobert & Weston, 2008; Bengio et al.,
2003; Glorot et al., 2011; Mikolov, 2012; Socher et al., 2011; Tur et al.,
2012). While conventional NLP techniques usually represent words as indices in
a vocabulary causing no notion of relationship between words, word embeddings
learned by deep learning approaches aim at explicitly encoding many semantic
relationships as well as linguistic regularities and patterns into the new
word embedding space. For example, a previous study (Bengio et al., 2003)
proposed a widely used model architecture for estimating neural network
language model. Collobert et al. (Collobert & Weston, 2008; Collobert et al.,
2011) introduced a unified neural network architecture that learns word
representations based on large amounts of unlabeled training data, to deal
with several different natural language processing tasks. Mikolov et al.
(Mikolov et al., 2013a, b) proposed the continuous bag-of-words model (CBOW)
and the continuous skip-gram model (Skip-gram) for learning distributed
representations of words also from large amount of unlabeled text data; these
two models can map the semantically or syntactically similar words to close
positions in the word embedding space, based on the intuition that similar
words are likely to yield similar context.

Although the study in learning distributed word representations has become
very hot recently, there are very few large and public datasets for the
evaluation of word representations. In this paper, we introduce a new large
benchmark collection named WordRep, which is built from several different data
source. We describe which evaluation tasks are selected in WordRep, how the
data are sampled, and how the evaluation tool is built. We also compare the
performance of several state-of-the-art word representations on WordRep.
Moreover, we take further discussions on new potential research topics that
can be supported by WordRep.

The rest of the paper is organized as follows. Section 2 gives a detailed
description about how WordRep is created. In Section 3, we report the
performance of several state-of-the-art word representations on WordRep. We
then show how to leverage WordRep to study other research topics in Section 4.
Finally, the paper is concluded in Section 5.

##  2 Creating WordRep Collection

In this section, we introduce the process of creating WordRep collection,
consisting of three main steps: selecting evaluation tasks, generating
evaluation samples, and finalizing datasets.

###  2.1 Selecting Evaluation Tasks

The main task for WordRep is the _analogical reasoning task_ introduced by
Mikolov et al. (Mikolov et al., 2013a). The task consists of 19,544 questions,
each of which is a tuple composed by two word pairs $(a,b)$ and $(c,d)$. From
the tuple, the question is of the form “$a$ is to $b$ is as $c$ is to  ”,
denoted as $a$ : $b$ $\rightarrow$ $c$ : ?. Suppose $\overrightarrow{w}$ is
the learned representation vector of word $w$ and is normalized to unit norm.
Following (Mikolov et al., 2013a), we answer this question by finding the word
$d^{*}$ whose representation vector is the closest to vector
$\overrightarrow{b}-\overrightarrow{a}+\overrightarrow{c}$ according to cosine
similarity excluding $b$ and $c$, i.e.,

$d^{*}=\arg\max_{x\in V,x\neq b,x\neq
c}(\overrightarrow{b}-\overrightarrow{a}+\overrightarrow{c})^{T}\overrightarrow{x}.$
(1)

The question is judged as correctly-answered only if $d^{*}$ is exactly the
answer word in the evaluation set. There are two categories of analogical
tasks, including 8,869 semantic analogies in five subtasks (e.g., _England_ :
_London_ $\rightarrow$ _China_ : _Beijing_) and 10,675 syntactic analogies in
nine subtasks (e.g., _amazing_ : _amazingly_ $\rightarrow$ _unfortunate_ :
_unfortunately_).

Table 1 gives a summary of Mikolov et al.’s evaluation set. The first five
subtasks are semantic questions and the rest nine subtasks are syntactic
questions. The table gives one example tuple for every subtask. It also shows
the number of unique word pairs and the number of tuples combined from these
pairs in each subtask. Regarding to the numbers of unique word pairs, we can
see that there are actually very small number of meaningful pairs in this
dataset. After further checking the number of tuples, we find that not all
possible combinations of word pairs are used as the tuple questions. For
example, in the subtask of _City-in-state_ , there should be as many as 4,556
tuples combined from 68 word pairs, but only 2,467 tuples are used in the
published dataset. It is not clear about the reason or how the 2,467 tuples
were sampled.

Table 1: Summary of Mikolov et al.’s evaluation set on analogical reasoning task. Subtask | Word pair 1 ($a$, $b$) | Word pair 2 ($c$, $d$) | # word pairs | # tuples
---|---|---|---|---
Common capital city | Athens | Greece | Oslo | Norway | 23 | 506
All capital cities | Astana | Kazakhstan | Harare | Zimbabwe | 116 | 4,524
Currency | Angola | kwanza | Iran | rial | 30 | 866
City-in-state | Chicago | Illinois | Stockton | California | 68 | 2,467
Man-Woman | brother | sister | grandson | granddaughter | 23 | 506
Adjective to adverb | apparent | apparently | rapid | rapidly | 32 | 992
Opposite | possibly | impossibly | ethical | unethical | 29 | 812
Comparative | great | greater | tough | tougher | 37 | 1,332
Superlative | easy | easiest | lucky | luckiest | 34 | 1,122
Present Participle | think | thinking | read | reading | 33 | 1,056
Nationality adjective | Switzerland | Swiss | Cambodia | Cambodian | 41 | 1,599
Past tense | walking | walked | swimming | swam | 40 | 1,560
Plural nouns | mouse | mice | dollar | dollars | 37 | 1,332
Plural verbs | work | works | speak | speaks | 30 | 870
# Total | / | / | / | / | 573 | 19,544

In the WordRep collection, we merged the above 14 subtasks into 12 subtasks,
and expanded the set of word pairs in each subtask by extracting new word
pairs from the Wikipedia and an English dictionary; we also added 13 new
subtasks by deriving pairwise word relationship from WordNet (WordNet, 2010).

###  2.2 Generating Evaluation Samples

Before we describe how we generate the evaluation samples, we first introduce
the scope of the vocabulary in WordRep. Since the words extracted from the
Wikipedia, the dictionary, and the Web knowledge bases (WordNet) may contain
many rare words that are not commonly used, we filter out those words that are
not covered by the vocabulary of _wiki2010_ (Shaoul & Westbury, 2010). This
corpus is a snapshot of the Wikipedia corpus in April 2010, which contains
about two million articles and 990 million tokens. The vocabulary size of
_wiki2010_ is 731,155, we regard which is large enough for common NLP tasks.
Furthermore, note that the evaluation set proposed by Mikolov et al. (Mikolov
et al., 2013a) contains a question set of phrase pairs besides the word pairs.
To deal with phrases like _New York_ , they simply connect the tokens by an
underline and write it as _New_York_. In this release of WordRep, we will only
generate question sets with word pairs and leave phrase pairs for the future
versions.

####  2.2.1 Wikipedia Knowledge

We leverage Wikipedia knowledge to enlarge the semantic analogical tasks. For
the first two subtasks in Table 1, we merged _Common capital city_ into _All
capital cities_ for simplicity. Then, we extracted the Wikipedia pages for all
the counties and areas to get a full list of countries, capitals, currencies,
and nationality adjectives, so that we can enlarge the question sets in the
subtasks of _All capital cities_ , _Currency_ , and _Nationality adjective_.
For _City-in-state_ , we found the top cities in population in the 50 states
in the U.S. from Wikipedia, and built the city-state pairs accordingly. Note
that we only kept the word based name entities, and removed the phrase based
names.

####  2.2.2 Dictionary Knowledge

We take advantage of dictionary knowledge to enlarge both semantic and
syntactic analogical tasks. For the subtask _Opposite_ , we merged it into a
new subtask _Antonym_ extracted from WordNet, which will be described in the
following. For the rest subtasks (_Man-Woman_ , _Adjective to adverb_ ,
_Comparative_ , _Superlative_ , _Present Participle_ , _Past tense_ , _Plural
nouns_ , _Plural verbs_) in Table 1, we enlarged their corresponding word
pairs by extracting new candidates from the Longman
Dictionaries111http://www.longmandictionariesonline.com/. For _Man-Woman_ , we
also added some word pairs of male and female animals like _cock_ and _hen_.
Note that the newly extracted words are filtered by the vocabulary of
_wiki2010_. The statistics about the enlarged subtasks by Wikipedia knowledge
and Dictionary knowledge is shown in Table 2.

####  2.2.3 WordNet Knowledge

WordNet (WordNet, 2010) is a large lexical database of English. It contains
both the words and the relations among the words, which is a gold mine to
enlarge both semantic and syntactic analogical tasks. We extracted 13 new
subtasks based on the WordNet relations including _Antonym_ , _MemberOf_ ,
_MadeOf_ , _IsA_ , _SimilarTo_ , _PartOf_ , _InstanceOf_ , _DerivedFrom_ ,
_HasContext_ , _RelatedTo_ , _Attribute_ , _Causes_ , and _Entails_. Note that
we merged the original subtask _Opposite_ in Table 1 into the new subtask
_Antonym_ , and we also filtered the words in the 13 new subtasks by the
vocabulary of _wiki2010_. The statistics and examples about the 13 WordNet
subtasks are shown in Table 3.

###  2.3 Finalizing Datasets

From Table 2 and Table 3, we can see that WordRep has much larger number of
word pairs as well as word tuples. The word pairs and word tuples can be
downloaded (http://research.microsoft.com/en-
us/um/beijing/events/kpdltm2014/WordRep-1.0.zip). The size of the compressed
package is 1.61 GB.

##  3 Evaluation on The Benchmark Collection

In this section, we report the performance of several state-of-the-art
distributed word representations on WordRep, including CW08 (Collobert &
Weston, 2008), RNNLM (Mikolov, 2012), and CBOW (Mikolov et al., 2013a). In
particular, we download the public word representations of
CW08222http://ml.nec-labs.com/senna/ (Collobert & Weston, 2008) whose
dimension is 50; the public word representations of RNNLM (Mikolov, 2012) are
obtained from its public site333http://www.fit.vutbr.cz/~imikolov/rnnlm/,
which includes three word representation models with the dimension of 80, 640,
and 1600, respectively; we obtain the CBOW (Mikolov et al., 2013a) models by
using its online tools444https://code.google.com/p/word2vec/ to train word
representations directly on the wiki2010 dataset, where we set the dimension
as 100, 200, and 300, respectively.

Table 4 demonstrates the accuracy of each of word representations on the
enlarged evaluation set of analogical reasoning tasks. From this table, we can
find that different word representations yield quite various accuracy on the
analogical reasoning tasks. In particular, CW08 with 50 dimension can only
achieve relatively low accuracy compared with RNNLM and CBOW, which may be due
to the difference in terms of the dimension of word representations, the
training data, or the training algorithms. We can also find that, with respect
to the same training method, larger dimension of word representations is more
likely to result in better performance.

Table 5 demonstrates the accuracy of each of word representations on the
WordNet evaluation set of analogical reasoning tasks. From this table, we can
see the analogous observations with the Table 4. From these two tables, we can
also find that different subtasks could yield quite diverse accuracy by
evaluating on the same word representation model.

Note that the above evaluation experiments can be done using the evaluation
tool provided by Word2Vec555https://code.google.com/p/word2vec/. The only
difference is that we treat all tuple questions as _seen_ questions. Thus, if
a tuple question is _unseen_ by the word embedding being evaluated (i.e., at
least one word in the tuple question is not in the vocabulary of the word
embedding), we will regard it to be answered incorrectly.

##  4 Supporting New Research Topics

Besides measuring the quality of word representations, we can also use WordRep
for, but not limited to, the following tasks.

  * •

WordRep can be used to evaluate the embedding for relations. Recently some
researchers have attempted to do word embedding and relation embedding
simultaneously. WordRep contains 25 subtasks in total, i.e., there are 25
types of relations that can be used for evaluating relation embeddings.

  * •

WordRep can be used to evaluate relation prediction. The 25 types of relations
can be regarded as labels of word pairs. Researchers can test their elation
prediction methods using these labels as ground-truth.

  * •

WordRep provides several good word lists for general NLP tasks. For example,
there are lists for different syntactic forms of nouns, verbs, adjectives, and
adverbs, and there are lists for commonly used relations.

##  5 Conclusions

In this paper, we have introduced a new data collection called WordRep, which
can be used for the evaluation of distributed word representations. We
described how we built the data collection and reported the evaluation
performance of several state-of-the-art word representations on it. We also
discussed the possible research topics that WordRep may support.

For the future work, we plan to further expand the evaluation set to phrase
pairs, and we also plan to enrich the collection by considering other Web
knowledge bases like Freebase (Bollacker et al., 2008).

##  6 Acknowledgements

We would like to thank Siyu Qiu, Chang Xu, Yalong Bai, Hongfei Xue, and Rui
Zhang for their contributions in the preparation of the collection and the
evaluation of the state-of-the-art word embeddings.

Table 2: Summary of enlarged evaluation set on analogical reasoning task. Subtask | Word pair 1 ($a$, $b$) | Word pair 2 ($c$, $d$) | # word pairs | # tuples
---|---|---|---|---
All capital cities | Astana | Kazakhstan | Harare | Zimbabwe | 131 | 17,030
Currency | Angola | kwanza | Iran | rial | 119 | 14,042
City-in-state | Chicago | Illinois | Stockton | California | 3,296 | 10,860,320
Man-Woman | brother | sister | grandson | granddaughter | 90 | 8,010
Adjective to adverb | apparent | apparently | rapid | rapidly | 689 | 474,032
Comparative | great | greater | tough | tougher | 97 | 9,312
Superlative | easy | easiest | lucky | luckiest | 87 | 7,482
Present Participle | think | thinking | read | reading | 2,951 | 8,705,450
Nationality adjective | Switzerland | Swiss | Cambodia | Cambodian | 139 | 19,182
Past tense | walking | walked | swimming | swam | 2,827 | 7,989,102
Plural nouns | mouse | mice | dollar | dollars | 5,868 | 34,427,556
Plural verbs | work | works | speak | speaks | 2,708 | 7,330,556
# Total | / | / | / | / | 19,002 | 69,862,074
Table 3: Summary of WordNet evaluation set on analogical reasoning task. Subtask | Word pair 1 ($a$, $b$) | Word pair 2 ($c$, $d$) | # word pairs | # tuples
---|---|---|---|---
Antonym | succeed | fail | major | minor | 973 | 945,756
MemberOf | Zealander | Zealand | Malawian | Malawi | 406 | 164,430
MadeOf | tear | water | glassware | glass | 63 | 3,906
IsA | yoga | exercise | yogurt | food | 10,615 | 112,667,610
SimilarTo | worthy | value | employ | work | 3,489 | 12,169,632
PartOf | eyebrow | face | column | temple | 1,029 | 1,057,812
InstanceOf | Dallas | city | Somerset | county | 1,314 | 1,725,282
DerivedFrom | king | kingdom | liberate | liberty | 6,119 | 37,436,042
HasContext | underhand | sport | rhyme | poetry | 1,149 | 1,319,052
RelatedTo | look | watch | use | consume | 102 | 10,302
Attribute | size | small | depth | shallow | 184 | 33,672
Causes | ring | sound | give | have | 26 | 650
Entails | buy | pay | beat | win | 114 | 12,882
# Total | / | / | / | / | 25,583 | 167,547,028
Table 4: Accuracy of various word representations on the enlarged evaluation set of analogical reasoning tasks. | CW08 | RNNLM | CBOW
---|---|---|---
Subtask | dim=50 | dim=80 | dim=640 | dim=1600 | dim=100 | dim=200 | dim=300
All capital cities | 0.62% | 0.76% | 1.23% | 1.81% | 6.62% | 9.04% | 11.28%
Currency | 0.25% | 0.41% | 0.66% | 0.87% | 3.13% | 4.06% | 4.32%
City-in-state | 0.67% | 1.36% | 3.14% | 3.38% | 1.55% | 1.86% | 2.25%
Man-Woman | 4.83% | 6.67% | 18.46% | 20.82% | 25.89% | 29.06% | 28.60%
Adjective to adverb | 1.40% | 0.93% | 1.17% | 2.01% | 3.45% | 3.44% | 3.23%
Comparative | 1.55% | 16.61% | 34.92% | 40.28% | 33.41% | 41.70% | 42.53%
Superlative | 1.94% | 9.18% | 25.33% | 26.21% | 23.56% | 24.99% | 29.07%
Present participle | 1.53% | 9.67% | 20.03% | 23.26% | 8.20% | 11.25% | 11.75%
Nationality adjective | 3.07% | 1.62% | 3.15% | 3.76% | 23.66% | 40.19% | 47.44%
Past tense | 1.84% | 10.43% | 19.51% | 22.77% | 15.51% | 21.60% | 24.15%
Plural nouns | 3.21% | 3.32% | 14.42% | 18.28% | 23.95% | 34.64% | 38.82%
Plural verbs | 2.44% | 7.89% | 22.41% | 26.62% | 17.28% | 27.47% | 31.82%
Total | 2.36% | 5.08% | 14.69% | 17.85% | 16.70% | 24.16% | 27.10%
Table 5: Accuracy of various word representations on the WordNet evaluation set of analogical reasoning tasks. | CW08 | RNNLM | CBOW
---|---|---|---
Subtask | dim=50 | dim=80 | dim=640 | dim=1600 | dim=100 | dim=200 | dim=300
Antonym | 0.28% | 1.21% | 2.88% | 3.12% | 2.74% | 4.08% | 4.57%
Attribute | 0.22% | 0.11% | 0.24% | 0.42% | 0.68% | 1.09% | 1.18%
Causes | 0.00% | 0.31% | 0.00% | 0.00% | 0.15% | 0.31% | 1.08%
DerivedFrom | 0.05% | 0.06% | 0.16% | 0.18% | 0.33% | 0.53% | 0.63%
Entails | 0.05% | 0.05% | 0.05% | 0.07% | 0.26% | 0.44% | 0.38%
HasContext | 0.12% | 0.06% | 0.16% | 0.19% | 0.28% | 0.38% | 0.35%
InstanceOf | 0.08% | 0.24% | 0.81% | 0.64% | 0.48% | 0.60% | 0.58%
IsA | 0.07% | 0.18% | 0.42% | 0.47% | 0.42% | 0.64% | 0.67%
MadeOf | 0.03% | 0.15% | 0.10% | 0.13% | 0.33% | 1.02% | 0.72%
MemberOf | 0.08% | 0.07% | 0.11% | 0.13% | 0.58% | 0.84% | 1.06%
PartOf | 0.31% | 0.29% | 0.55% | 0.60% | 1.17% | 1.24% | 1.27%
RelatedTo | 0.00% | 0.04% | 0.02% | 0.00% | 0.20% | 0.09% | 0.05%
SimilarTo | 0.02% | 0.07% | 0.14% | 0.18% | 0.14% | 0.23% | 0.29%
Total | 0.06% | 0.15% | 0.35% | 0.40% | 0.40% | 0.61% | 0.66%

## References

  * Bengio et al. (2003) Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C.  A neural probabilistic language model.  In _The Journal of Machine Learning Research_ , pp. 3:1137–1155, 2003.
  * Blei et al. (2003) Blei, David M., Ng, Andrew Y., and Jordan, Michael I.  Latent dirichlet allocation.  _J. Mach. Learn. Res._ , 3:993–1022, March 2003.  ISSN 1532-4435.
  * Bollacker et al. (2008) Bollacker, K., Evans, C., Paritosh, P., Sturge, T., and Taylor, J.  Freebase: a collaboratively created graph database for structuring human knowledge.  In _Proceedings of the 2008 ACM SIGMOD international conference on Management of data_ , pp. 1247–1250. ACM, 2008.
  * Collobert & Weston (2008) Collobert, R. and Weston, J.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In _Proceedings of the 25th International Conference on Machine Learning_ , ICML ’08, pp. 160–167, New York, NY, USA, 2008. ACM.
  * Collobert et al. (2011) Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P.  Natural language processing (almost) from scratch.  _Journal of Machine Learning Research_ , 12:2493–2537, November 2011.  ISSN 1532-4435.
  * (6) Dumais, Susan T.  Latent semantic analysis.  _Annual Review of Information Science and Technology_ , 38(1).  ISSN 1550-8382.
  * Glorot et al. (2011) Glorot, X., Bordes, A., and Bengio, Y.  Domain adaptation for large-scale sentiment classification: A deep learning approach.  In _In Proceedings of the Twenty-eight International Conference on Machine Learning, ICML_ , 2011.
  * Mikolov (2012) Mikolov, T.  _Statistical Language Models Based on Neural Networks_.  PhD thesis, Brno University of Technology, 2012.
  * Mikolov et al. (2013a) Mikolov, T., Chen, K., Corrado, G., and Dean, J.  Efficient estimation of word representations in vector space.  _CoRR_ , abs/1301.3781, 2013a.
  * Mikolov et al. (2013b) Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J.  Distributed representations of words and phrases and their compositionality.  In Burges, Christopher J. C., Bottou, Léon, Ghahramani, Zoubin, and Weinberger, Kilian Q. (eds.), _NIPS_ , pp. 3111–3119, 2013b.
  * Shaoul & Westbury (2010) Shaoul, C. and Westbury, C.  The westbury lab wikipedia corpus, edmonton, ab: University of alberta, 2010.
  * Socher et al. (2011) Socher, R., Lin, C. C., Ng, A. Y., and Manning, C. D.  Parsing natural scenes and natural language with recursive neural networks.  In _Proceedings of the 26th International Conference on Machine Learning (ICML)_ , 2011.
  * Tur et al. (2012) Tur, G., Deng, L., Hakkani-Tur, D., and He, X.  Towards deeper understanding: Deep convex networks for semantic utterance classification.  In _ICASSP_ , pp. 5045–5048, 2012.
  * WordNet (2010) WordNet.  “about wordnet”, princeton university, 2010.  URL http://wordnet.princeton.edu.