# PhayaThaiBERT: Enhancing a Pretrained Thai Language Model with Unassimilated
Loanwords

Panyut Sriwirote
Department of Linguistics
Chulalongkorn University
<EMAIL_ADDRESS>Jalinee Thapiang
ClickNext
<EMAIL_ADDRESS>Vasan Timtong
ClickNext
<EMAIL_ADDRESS>Attapol T. Rutherford
Department of Linguistics
Chulalongkorn University
<EMAIL_ADDRESS>
###### Abstract

Although WangchanBERTa has become the de facto standard in transformer-based
Thai language modeling, it still has shortcomings in regard to the
understanding of foreign words, most notably English words, which are often
borrowed without orthographic assimilation into Thai in many contexts. We
identify the lack of foreign vocabulary in WangchanBERTa’s tokenizer as the
main source of these shortcomings. We then expand WangchanBERTa’s vocabulary
via vocabulary transfer from XLM-R’s pretrained tokenizer and pretrain a new
model using the expanded tokenizer, starting from WangchanBERTa’s checkpoint,
on a new dataset that is larger than the one used to train WangchanBERTa. Our
results show that our new pretrained model, PhayaThaiBERT, outperforms
WangchanBERTa in many downstream tasks and datasets.

##  1 Introduction

The use of pretrained transformer-based masked language models has led to
state-of-the-art performance in many NLP tasks, both in English Devlin et al.
(2019) Liu et al. (2019) and other languages Conneau et al. (2020) Martin et
al. (2020) Nguyen and Tuan Nguyen (2020). The transformer-based architecture
and its word embedding vocabulary are pretrained on a large amount of data on
certain objective functions. And then we can perform supervised fine-tuning on
the pretrained model for specific tasks. For Thai, WangchanBERTa, the state-
of-the-art transformer-based language model, resulted in superior performance
in many Thai NLP tasks Lowphansirikul et al. (2021a), such as misspelling
correction Pankam et al. (2023), text classification Chotirat et al. (2022),
essay quality check Noiyoo and Thutkawkornpin (2023), and sentiment analysis
Khamphakdee and Seresangtakul (2023).

Borrowing the architecture from RoBERTa Liu et al. (2019), WangchanBERTa
employs some special techniques aimed to address some characteristics of the
Thai language, such as forcing its tokenizer to preserve spaces by replacing
them with a special token that will never be included with other tokens, and
normalizing sequences of more than 3 repeated characters, often used for
emphasis in casual texts, to just 1 character. However, a characteristic of
the Thai language underemphasized by this model is the code-switching nature
of the language. Thai natural user-generated content usually contains
orthographically unassimilated loanwords, foreign words that are written using
non-Thai characters exactly like they would be in the original language. This
phenomenon is most evident for English words, especially in discussions
involving foreign culture or in academic context, where many important
concepts simply do not have a widely accepted Thai equivalent.

The previous model does not contain enough foreign vocabulary to address this
nature. We hypothesize that the vocabulary of the transformer-based model
should be expanded and pretrained to improve the performance of the downstream
supervised fine-tuning process.

The main aim of this study is, therefore, to train a new transformer-based
masked Thai language model that has been enhanced to better understand
instances of code-swtiched language and unassimilated loanwords. A secondary
aim is to pretrain the model using even larger training data than the one used
to train WangchanBERTa. The new pretrained model should then be able to
achieve better performance. Indeed, our results in Section 5.2 have shown this
to be the case.

Our new pretrained model, which we call PhayaThaiBERT, is publicly available
on Hugging Face Model Hub.111https://huggingface.co/clicknext/phayathaibert
The source code used in our experiment is available on our GitHub
repository.222https://github.com/clicknext-ai/phayathaibert

##  2 Related Work

Modern language modeling has largely been focused on transformer-based models
as pioneered by the BERT architecture Devlin et al. (2019). The original BERT
is implemented using 2 objectives, masked token prediction and next sentence
prediction. Its tokenizer uses the WordPiece algorithm Wu et al. (2016). Later
studies further improve upon this architecture and method, most notably the
RoBERTa architecture Liu et al. (2019), which drops the next sentence
prediction objective and optimizes many aspect of BERT pretraining. New
tokenization algorithms have since been proposed, most notably the unigram
algorithm Kudo (2018), which is reported to result in better performance
compared to previously used tokenization algorithms. In addition to single-
language modeling, multilingual models have also been implemented, most
notably XLM-R Conneau et al. (2020), which combines the method proposed by
RoBERTa with the unigram tokenization algorithm, along with other techniques
to scale the model to 100 languages. It takes advantage of the SentencePiece
library Kudo and Richardson (2018), which implements BPE and unigram
tokenization algorithm in a way that is highly portable and language-agnostic.
For Thai, a model has also been implemented. As stated in the introduction,
WangchanBERTa, a RoBERTa-based model that uses SentencePiece’s unigram
tokenization, has been the de facto standard in Thai langauge modeling since
its release in 2021. It is precisely this model that we wish to improve
further.

##  3 New Dataset and Vocabulary Expansion

In order to achieve our goal of creating a language model capable of
understanding unassimilated loanwords, we employ 2 strategies, 1) further
pretraining on a new, larger dataset, and 2) expansion of WangchanBERTa’s
vocabulary to include foreign words.

##### New Dataset

We create a large raw text dataset using various sources, including a large
portion of the data used to train WangchanBERTa, most notably wisesight-large
and pantip-large, which account for over 90% of WangchanBERTa’s training set.
The inclusion of WangchanBERTa’s data is in the hope that our model, which
uses WangchanBERTa’s weights as the starting point (see Section 4.2), could
have a chance to recalibrate its parameters to account for new tokenization
resulting from added vocabulary. The total size of our preprocessed and
tokenized training data is 156.5GB, 2 times larger than the one used to train
WangchanBERTa. The list of sources we use to create our dataset is shown in
the appendix.

##### Vocabulary Expansion

One source of the shortcomings of WangchanBERTa when it comes to understanding
unassimilated loanwords is its tokenizer, whose vocabulary contains only a
small number of foreign words, making it unable to tokenize unassimilated
words in a way that makes sense. In our experiment, we try improving it by
adding new foreign vocabulary to WangchanBERTa’s SentencePiece unigram
tokenizer. This is done through a simple transfer of vocabulary, along with
each word’s unigram score, from XLM-R’s tokenizer. Sine both tokenizers use
SentencePiece’s implementation of the unigram algorithm and both store their
vocabulary in the Protobuf format333https://protobuf.dev/, it is trivially
easy to transfer vocabulary between them. We only transfer vocabulary that
does not already exist in WangchanBERTa’s vocabulary and we intentionally do
not transfer XLM-R’s vocabulary that consists of at least 1 character from the
Thai Unicode block to avoid interfering with the tokenization of existing
vocabulary. In addition to this, we also add all emojis as defined by the
Unicode Consortium to the vocabulary. These emojis have no associated unigram
scores and will be tokenized using simple pattern matching. We believe this is
justifiable since we do not expect emojis to form any morphological or
syntactic unit larger than 1 character.

After vocabulary expansion, the model’s vocabulary size increases from 25,005
to 249,262, significantly increasing the size of the model from 106M
parameters to 278M parameters.

##  4 Experimental Setup

###  4.1 Model Architecture

We implement the same model architecture as that of WangchanBERTa, namely the
RoBERTa architecture, with some minor modification. Our architecture has 2
separate embedding lookup tables. One contains the embeddings of the existing
vocabulary while the other contains the embeddings of the added vocabulary.
During forward pass, the embedding of each token will be looked up from their
respective tables. The result will then be combined in such a way as to be
identical to the case with only 1 embedding lookup table. This ensures that
there is no need to modify any of the other layers of the RoBERTa
architecture. The reason for this modification will be discussed in section
4.6.

The final model that we publish on Hugging Face Model Hub, however, does not
use the architecture described above. It has been converted back to the
RoBERTa architecture for compatibility and ease-of-use.

###  4.2 Starting Model Checkpoint

We use the best checkpoint of WangchanBERTa (wangchanberta-base-att-spm-
uncased) as the model’s starting point. Every parameter weight of the model is
transfered directly from said checkpoint, with the exception of the added
vocabulary’s embeddings, which are randomly initialized.

###  4.3 Preprocessing and Tokenization

Since most of the model’s parameters are transfered directly from
WangchanBERTa, it is imperative that we also preprocess and tokenize our
training data in the same manner. Accordingly, we preprocess and tokenize all
of our training data following the procedure described in the Methodology
section of the WangchanBERTa paper Lowphansirikul et al. (2021a), with the
exception of the tokenization section, where we will use our expanded
tokenizer instead.

The resulting preprocessed and tokenized texts are then grouped into chunks,
each no longer than 416 tokens. Pieces of texts that are by itself longer than
416 tokens are simply discarded. The final preprocessed and tokenized data is
stored in the Arrow format444https://arrow.apache.org/ using Hugging Face’s
datasets library Lhoest et al. (2021).

###  4.4 Train-Validation-Test Splits

After preprocessing and tokenization, our training data has a size of
91,130,926 examples (156.5GB). We then randomly split the data into 91,030,926
examples (156.3GB) for Train set, 50,000 examples (87.9MB) for Validation set,
and 50,000 (88MB) examples for Test set.

###  4.5 Pretraining Task

The objective of the model’s pretraining is masked language modeling. In the
same manner as WangchanBERTa, we masked tokens at the subword level rather
than the whole word. We sample 15% of the tokens in a sequence as tokens to be
masked. Masking is done by replacing the token with the special token <mask>
80% of the time, replacing them with a random token from the vocabulary 10% of
the time, and leaving the token unchanged 10% of the time. This is done on the
fly as the training progresses, making the training samples different from
epoch to epoch. The goal of the model is to predict what token is being masked
by <mask>. Loss is computed using cross entropy over the entire vocabulary.

###  4.6 Training Procedure

We discover that mere further pretraining from WangchanBERTa’s checkpoint
without any modification to the training procedure invariably results in a
spike in validation loss after only a few hundred steps. We solve this problem
by following some of the techniques proposed in Howard and Ruder (2018), which
are:

##### Discriminative fine-tuning

We use different learning rates for each layer in the model. We group all the
model parameters into 15 layers, 12 of which correspond to the 12 transformer
blocks while the remaining 3 correspond to 1) the embeddings of the added
vocabulary, 2) the embeddings of the existing vocabulary, and 3) the masked
language modeling head. The learning rate for the added vocabulary’s
embeddings are set to be the largest, as determined by peak learning rate (see
appendix). All other layers’ learning rates are set to be progressively lower
the further they are from the last layer in the forward pass. Each layer’s
learning rate is set to be 2.6 times smaller than its next layer.
($lr^{l-1}=lr^{l}/2.6$). Following this scheme, the layer with the smallest
learning rate is the embeddings of the existing vocabulary.

##### Gradual unfreezing

At the start of the training, we freeze every model parameter except those
belonging to the embeddings of the added vocabulary. Other layers will then be
gradually unfrozen every 1,000 steps. Among the yet-unfrozen layers, the one
closest to the last layer in the forward pass will be chosen to be unfrozen at
each unfreezing step. The grouping of parameters into layers is identical to
the one used for discriminative fine-tuning, meaning the last layer to be
unfrozen at step 14,000 is the embeddings of the existing vocabulary.

#####

During the course of the training, we save a checkpoint of the model every 100
steps, and evaluate the model at that checkpoint with the Validation set. The
computed loss is then used to decide whether the model has already converged
or not. Training is done on the LANTA HPC cluster555https://thaisc.io/thaisc-
resorces/lanta using the Accelerator API of Hugging Face’s accelerate library
Gugger et al. (2022).

Due to resource and time limitation, we are able to train only one model in a
single run, forcing us to train the model in a somewhat irregular manner. The
irregular actions we are forced to perform are as follows:

##### Disabling discriminative fine-tuning

We discover that the model fails to improve further at around step 4,200. We
hypothesize that this is caused by our implementation of discriminative fine-
tuning, which causes higher layers to overfit more quickly while lower layers
still fail to converge due to their learning rate being too small. We then
decide to disable discriminative fine-tuning, after which the model starts to
improve again.

##### Periodically resetting the learning rate scheduler

Since we cannot predict how many steps it would take for the model to
converge, we cannot decide exactly what value we should set for max steps. We
decide to set the learning rate scheduler’s max steps to 500,000, which, due
to the way Hugging Face’s accelerate library handles multi-GPU training, will
be reached within 31,250 actual update steps when trained on 16 GPUs. The
scheduler will then be periodically reset when we subjectively decide that
there is still potential for the model to improve using a larger learning
rate.

##### Switching self-attention layers from FP16 to FP32 training

At around step 6,200, the training becomes numerically unstable, with NaN
consistently appearing in both training and evaluation to the point where the
model cannot learn anything further. We determine that NaN always appears for
the first time in the computation graph in self-attention layers. We then
decide to switch those layers from FP16 to FP32 training, after which the
problem is fixed for good.

###  4.7 Evaluation

In order to assess its capability, we fine-tune the pretrained model on
several downstream tasks and datasets. We also fine-tune other similar models
for comparison.

####  4.7.1 Datasets

The datasets are chosen as to be mostly similar to those in the WangchanBERTa
paper Lowphansirikul et al. (2021a). We use the train-validation-test split
exactly as provided by the datasets hosted on Hugging Face Dataset
Hub.666https://huggingface.co/datasets In cases where certain splits are
unavailable, we randomly split the provided train split using the
train_test_split method from Hugging Face’s datasets library Lhoest et al.
(2021) with seed = 2020, the same seed used in the WangchanBERTa paper. A
summary of all the datasets used in the final evaluation is shown in Table 1.

Name | Classification Task | Labels | Train | Validation | Test
---|---|---|---|---|---
1\. wisesight_sentiment | single-label, sequence | 4 | 21,628 | 2,404 | 2,671
2\. wongnai_reviews | single-label, sequence | 5 | 36,000* | 4,000* | 6,203
3\. yelp_review_full** | single-label, sequence | 5 | 36,000* | 4,000* | 6,200
4\. generated_reviews_enth | single-label, sequence | 5 | 141,369 | 15,708 | 17,453
5\. prachathai67k | multi-label, sequence | 12 | 54,379 | 6,721 | 6,789
6\. thainer (ner) | single-label, token | 13*** | 5,079* | 635* | 634*
7\. lst20 (pos) | single-label, token | 16 | 63,310 | 5,620 | 5,250
8\. lst20 (ner) | single-label, token | 10 | 63,310 | 5,620 | 5,250
9\. thai_nner (layer 1) | single-label, token | 104 | 2,934* | 980* | 980
* Randomly split from the provided train split with seed = 2020
** Downsampled from the full dataset
*** The unconfirmed tag is removed
Table 1: Description of the datasets used in the final evaluation.

##### Wisesight Sentiment Suriyawongkul et al. (2019)

A dataset of 26,703 Thai messages from Thai social media. Each message is
labeled as either positive, neutral, negative, or question.

##### Wongnai Reviews Wongnai.com (2018)

A dataset of 46,203 Thai reviews from Wongnai, a Thai reviewing platform. Each
review is labeled with a rating from 1 to 5 stars.

##### Yelp Review Full Zhang et al. (2015)

A dataset of 700,000 English reviews from Yelp, an American reviewing
platform. Each review is labeled with a rating from 1 to 5 stars. We do not
use the entire dataset for our fine-tuning. Instead, we split the provided
train split into 2 new splits, train and validation, each with 600,000 and
50,000 reviews respectively. We then downsample all splits by taking only the
front of each split. The exact number taken in each split is shown in Table 1.

##### Generated Reviews EN-TH Lowphansirikul et al. (2021b)

A dataset of 174,530 reviews generated by CTRL Keskar et al. (2019) originally
in English and then translated into Thai using Google Translate API. Each
review is labeled with a rating from 1 to 5 stars. We only use the translated
Thai reviews as a feature to predict the labels.

##### Prachathai67k cstorm125 and lukkiddd (2019)

A dataset of 67,889 Thai news articles from Prachathai, a Thai online news
site. Each article is labeled with a set of tags representing the themes of
its content. Each article can have more than 1 tag. Possible tags are
politics, human_rights, quality_of_life, international, social, environment,
economics, culture, labor, national_security, ict, and education. We only use
each article’s headline as a feature to predict the tags.

##### ThaiNER Phatthiyaphaibun (2019)

A dataset of 6,348 Thai sentences that have been tokenized and annotated with
NER (IOB format) and POS tags. Possible NER tags are DATE, EMAIL, LAW, LEN,
LOCATION, MONEY, ORGANIZATION, PERCENT, PERSON, PHONE, TIME, URL, and ZIP.
Originally, the dataset also has the unconfirmed tag that represents tokens
recognized by annotators as part of a named entity but whose precise tags are
undecided. We thus replace every occurrence with O, reducing the number of
possible labels by 1.

##### LST20 Boonkwan et al. (2020)

A dataset of 74,180 Thai sentences that have been tokenized and annotated with
NER (IOBE format) and POS tags. Possible NER tags are BRN, DES, DTM, LOC, MEA,
NUM, ORG, PER, TRM, and TTL while possible POS tags are AJ, AV, AX, CC, CL,
FX, IJ, NG, NN, NU, PA, PR, PS, PU, VV, and XX.

##### Thai-NNER Buaphet et al. (2022)

A dataset of 4,894 Thai documents that have been tokenized and annotated with
NER tags in the IOBES format. The annotation is done in a nested manner, with
NER tags separated across 8 layers. The span of each named entity in a deeper
layer will always be contained within the span of a named entity in shallower
layers. We only fine-tune the models to predict NER tags on the shallowest
layer. There are 104 possible NER tags, which are listed at the end of the
dataset’s associated paper.

####  4.7.2 Benchmarking Models

In addition to our model, PhayaThaiBERT, we also fine-tune other similar
models for comparison, namely mBERT (bert-base-multilingual-cased) Devlin et
al. (2019), XLM-R (xlm-roberta-base) Conneau et al. (2020), and WangchanBERTa
(wangchanberta-base-att-spm-uncased) Lowphansirikul et al. (2021a), all of
which are pretrained transformer-based masked language models.

####  4.7.3 Fine-tuning Procedure

Our fine-tuning procedure is designed to be mostly similar to the one
described in the Downstream Tasks section of the WangchanBERTa paper
Lowphansirikul et al. (2021a).

For WangchanBERTa and PhayaThaiBERT, the datasets are preprocessed in the same
manner as described in the WangchanBERTa paper. For mBERT and XLM-R, the
datasets are simply tokenized using their respective tokenizers. Although the
datasets for token classification task already consist of pre-tokenized
sentences, they are not tokenized in a way that can be directly used as input
to the models. Each token is still required to be tokenized by each model’s
specific tokenizer, after which each token in the original datasets could be
separated into a sequence of more than 1 token, in which case we decide to
label only the first token, meaning the models will be trained to predict only
the first token of each sequence.

For mBERT and XLM-R, we limit the length of the input to 512 tokens, the
maximum allowed by the models. For WangchanBERTa and PhayaThaiBERT, we fine-
tune each model 2 times, once with 416 as the length limit, the same value
used during training, and once with 510 as the length limit, the maximum
allowed by the models. The models with better performance are then selected
and their performance is reported for each model. For token classification
task, where every token needs to be accounted for, we simply split each
example into chunks, each no longer than the length limit.

Table 2 shows the hyperparameters we use to fine-tune the models for each
task. All fine-tuning is done using the Trainer API of Hugging Face’s
transformers library Wolf et al. (2020).

Hyperparameter |  | Single-label
---
sequence
classification
| Multi-label
---
sequence
classification
| Single-label
---
token
classification
Peak learning rate | 3e-5
Warmup step ratio | 0.1
Weight decay | 0.01
Adam $\epsilon$ | 1e-8
Adam $\beta_{1}$ | 0.9
Adam $\beta_{2}$ | 0.999
FP16 | True
Batch size | 16 | 32
Number of training epochs | 3 | 6*
Checkpointing steps | 100**
Metric for best checkpoint | micro-average F1 | macro-average F1 | loss
| * Except for Thai-NNER, which we train the models for 20 epochs to
compensate for
---
the fact that Thai-NNER has an extremely large tag set of over 400 unique NER
tags
| ** Except for ThaiNER and Thai-NNER, which we save a checkpoint of the
models
---
every 20 steps since these datasets have a very small training set
Table 2: Hyperparameters used in the fine-tuning of each task.

####  4.7.4 Metrics

The metrics we use to evaluate the fine-tuned models are micro-average F1
score and macro-average F1 score. In case of NER task, the metrics are
computed using seqeval’s classification_report function Nakayama (2018). In
case of other tasks, they are computed using scikit-learn’s
classification_report function Pedregosa et al. (2011).

##  5 Results and Discussion

###  5.1 Pretraining

The Validation loss computed during training reaches an apparent minimum at
step 51,500 (epoch 2.3), after which it fails to decrease further even after
another 3,800 update steps. We thus take the model’s checkpoint at step 51,500
as our final pretrained model. The Validation loss at step 51,500 is 1.6524.

###  5.2 Fine-tuning on downstream tasks

Dataset | mBERT | XLM-R | WangchanBERTa | PhayaThaiBERT
---|---|---|---|---
1\. wisesight_sentiment | 70.57 / 55.62 | 71.77 / 58.29 | 74.35 / 65.23 |  76.15 / 66.80
2\. wongnai_reviews | 58.08 / 38.67 | 62.73 / 51.18 | 63.86 / 53.26 |  64.02 / 52.74
3\. yelp_review_full | 63.52 / 63.23 |  65.19 / 64.80 | 54.97 / 54.40 | 61.69 / 61.26
4\. generated_reviews_enth | 61.79 / 56.04 |  65.06 / 60.28 | 64.75 / 59.91 | 64.85 / 59.44
5\. prachathai67k | 63.90 / 52.95 | 66.63 / 58.01 | 67.51 / 59.05 |  69.11 / 61.10
6\. thainer (ner) | 79.58 / 69.87 | 84.95 / 72.20 | 84.64 / 68.19 |  86.42 / 74.77
7\. lst20 (pos) | 95.80 / 83.95 | 95.99 / 85.09 | 96.74 / 86.59 |  96.79 / 86.26
8\. lst20 (ner) | 76.48 / 70.27 |  78.37 / 72.82 | 77.99 / 72.86 | 78.11 / 72.69
9\. thai_nner (layer 1) | 61.19 / 23.45 | 63.88 / 23.28 | 59.31 / 22.85 |  64.26 / 25.70
Table 3: Fine-tuning results. The reported metrics are micro-average and macro-average F1 score respectively. Dataset | mBERT | XLM-R | WangchanBERTa
---|---|---|---
1\. wisesight_sentiment | 70.05 / 57.81 | 73.57 / 62.21 |  76.19 / 67.05
2\. wongnai_reviews | 47.99 / 12.97 | 62.57 / 52.75 |  63.05 / 52.19
3\. yelp_review_full | - | - | -
4\. generated_reviews_enth | 62.14 / 57.20 |  64.91 / 60.29 | 64.66 / 59.54
5\. prachathai67k | 66.47 / 60.11 | 68.18 / 63.14 |  69.78 / 64.90
6\. thainer (ner) | 81.48 / 73.97 | 83.25 / 67.23 |  86.49 / 79.29
7\. lst20 (pos) | 96.44 / 85.86 | 96.57 / 85.00 |  96.62 / 85.44
8\. lst20 (ner) | 75.05 / 68.25 | 73.61 / 68.67 |  78.01 / 72.25
9\. thai_nner (layer 1) | - | - | -
Table 4: Original fine-tuning results as reported in the WangchanBERTa paper
Lowphansirikul et al. (2021a).

The evaluation results of all fine-tuned models on test splits can be seen in
Table 3. The results originally reported in the WangchanBERTa paper is also
given in Table 4 for comparison. While we try to fine-tune our models using
procedures as close as possible to those in the original paper, we fail to
reproduce the same level of performance for many models. Nevertheless, our
results should give a valid comparison since all our fine-tuned models are
guaranteed to come from the exact same procedures and environments.

Our model outperforms all other models in both micro-average and macro-average
F1 scores in 4 out of the 9 datasets (1, 5, 6, 9). Our model outperforms all
other models only in micro-average F1 scores in 2 out of the remaining 5
datasets (2, 7), where our model is defeated in both cases by WangchanBERTa in
macro-average F1 scores. Among the remaining 3 datasets, XLM-R achieves both
the highest micro-average and macro-average F1 scores in 2 of them (3, 4). For
the last remaining dataset (8) the highest micro-average F1 score is achieved
by XLM-R while the highest macro-average F1 score is achieved by
WangchanBERTa.

However, if we only compare between WangchanBERTa and our model, our model
defeats WangchanBERTa in both metrics in 5 out of the 9 datasets (1, 3, 5, 6,
9), with our model defeating WangchanBERTa in 1 out of the 2 metrics in the
remaining 4 datasets (2, 4, 7, 8). In no case does WangchanBERTa defeat our
model in both metrics. These results demonstrate that our model has indeed
been improved beyond WangchanBERTa’s capability, although at a cost of a
significantly larger model size.

The outlying results in Yelp Review Full (3) and Generated Reviews EN-TH (4),
where XLM-R achieved the highest score in both metrics, could be explained by
the non-Thai-ness of their samples. Yelp Review Full consists entirely of
English reviews while Generated Reviews EN-TH’s examples are originally
English texts that have been translated into Thai using a machine translation
model. It could thus be reasonably expected that XLM-R, which has been trained
extensively on English language pattern, would give better performance in such
cases. Our results confirm the finding in the WangchanBERTa paper
Lowphansirikul et al. (2021a), in which XLM-R is also reported to give better
performance for Generated Reviews EN-TH.

###  5.3 Effect of Our Enhancement on OOV Rate

Out-Of-Vocabulary (OOV) rate is the proportion of input texts that cannot be
tokenized because there are no appropriate words in the tokenizer’s
vocabulary. In case of RoBERTa-based models, this is represented by <unk>
tokens. Since every English alphabet is already part of WangchanBERTa’s
vocabulary, the OOV rate is already small even without our enhancement, with
<unk> tokens appearing only in cases where unusual characters are used, such
as emojis or words written in scripts other than Thai and basic Latin.
Nevertheless, our enhancement manages to reduce this rate even further, as can
be seen in Table 5.

Dataset | WangchanBERTa | PhayaThaiBERT
---|---|---
<unk> count | Percentage | <unk> count | Percentage
1\. wisesight_sentiment | 2,900 | 0.32% | 34 |  $\sim$0%
2\. wongnai_reviews | 3,855 | 0.05% | 90 |  $\sim$0%
3\. yelp_review_full | 15 |  $\sim$0% | 0 | 0%
4\. generated_reviews_enth | 99 |  $\sim$0% | 10 |  $\sim$0%
5\. prachathai67k | 241 | 0.02% | 61 |  $\sim$0%
6\. thainer | 8 |  $\sim$0% | 1 |  $\sim$0%
7\. lst20 | 19 |  $\sim$0% | 15 |  $\sim$0%
8\. thai_nner | 267 | 0.01% | 23 |  $\sim$0%
Table 5: OOV rates of WangchanBERTa’s tokenizer and our expanded tokenizer for
each dataset.

###  5.4 Do Unassimilated Loanwords Affect Models’ Performance?

Table 6 shows the proportion of samples that contain unassimilated English
words in each dataset along with our model’s performance gain compared to
WangchanBERTa. We only show the statistics for English because English words
are by far the most common loanwords in Thai texts.

For sequence classification datasets (1, 2, 3, 4, 5), there does not seem to
be any relation between prevalence of unassimilated loanwords and performance
except for Yelp Review Full (3). This is likely because the important pieces
of text used to determine the labels are mostly in Thai, making any
improvement to the understanding of foreign words unlikely to result in any
significant performance gain. This of course does not apply to Yelp Review
Full, which is entirely English, and it is here that we see a large gain due
to our enhancement.

In constrast, for token classification datasets (6, 7, 8, 9), there seems to
be a correlation between prevalence of unassimilated loanwords and performance
gain. That is, the higher the prevalence of unassimilated English words, the
higher the performance gain. The difference from previous datasets could be
explained by the fact that token classification requires the models to
understand every token in order to be able to assign their labels correctly.
This means that any improvement to the understanding of foreign words should
contribute directly to the accuracy of the models.

Dataset |  Proportion of Samples with Unassimilated Loanwords |  Performance Gain Compared to WangchanBERTa
---|---|---
1\. wisesight_sentiment |  27.59% |  +1.8 / +1.57
2\. wongnai_reviews |  37.20% |  +0.16 / -0.52
3\. yelp_review_full |  Entirely English |  +6.72 / +9.86
4\. generated_reviews_enth |  36.68% |  +0.1 / -0.47
5\. prachathai67k |  8.54% |  +1.6 / +2.05
6\. thainer |  10.90% |  +1.78 / +6.58
7\. lst20 (pos) |  2.58% |  +0.05 / -0.33
8\. lst20 (ner) |  2.58% |  +0.12 / -0.17
9\. thai_nner |  38.54% |  +4.95 / +2.85
Table 6: Proportion of samples with unassimilated English words compared with
performance gain for each dataset.

##  6 Conclusion

In this study, we introduce PhayaThaiBERT, a pretrained Thai masked language
model based on WangchanBERTa that has been enhanced to improve its
understanding of orthographically unassimilated loanwords. With WangchanBERTa
as foundation, we improve upon it by expanding its vocabulary and pretraining
it further on new and larger training data. Some modifications and techniques
are applied to prevent loss of performance due to these procedures. We fine-
tune the pretrained model on several tasks and datasets. The results show that
our model has improved beyond WangchanBERTa’s capability, albeit not decisive
in some context. As with WangchanBERTa, our model also fails to achieve better
performance than XLM-R for Generated Reviews EN-TH, a dataset whose texts are
translated from English using a machine translation model.

## Limitations

Our model, while achieving better performance, is significantly larger than
WangchanBERTa and thus will require significantly more computing resources to
use. The sole source of the increase in model size is the added vocabulary,
which contains a large number of foreign words that would not all be needed
for every downstream application. Steps that could be taken to alleviate this
problem might include model distillation Sanh et al. (2020), and exclusion of
unnecessary vocabulary from the model.

Moreover, our training procedure is far from ideal, with many aspects based on
arbitrary and subjective decision. While we are successful in defeating
WangchanBERTa, performing more experiments to identify the optimal procedure
and hyperparameters could potentially improve our model further. This has,
however, been impossible given the time and resources available to us.

## Acknowledgements

We would like to thank Lalita Lowphansirikul, one of the authors of the
WangchanBERTa paper, for providing us with various pieces of advice and
insights about the training of large masked language models. We thank the
WangchanBERTa team for providing us with the original data used in the
training of WangchanBERTa. We thank NSTDA Supercomputer Center for giving us
free of charge access to the LANTA HPC cluster (project lt200063). Finally, we
would also like to thank our colleagues at ClickNext who contribute in various
ways to this study, especially Jakkrit Diloksrisakul, Pontakorn Trakuekul,
Jiratchaya Sutthirod, and Tanhaporn Tosirikul.

## References

  * Boonkwan et al. (2020) Prachya Boonkwan, Vorapon Luantangsrisuk, Sitthaa Phaholphinyo, Kanyanat Kriengket, Dhanon Leenoi, Charun Phrombut, Monthika Boriboon, Krit Kosawat, and Thepchai Supnithi. 2020.  The annotation guideline of lst20 corpus.
  * Buaphet et al. (2022) Weerayut Buaphet, Can Udomcharoenchaikit, Peerat Limkonchotiwat, Attapol Rutherford, and Sarana Nutanong. 2022.  Thai nested named entity recognition corpus.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1473–1486, Dublin, Ireland. Association for Computational Linguistics.
  * Chotirat et al. (2022) Saranlita Chotirat, Phayung Meesad, and Herwig Unger. 2022.  Question classification from thai sentences by considering word context to question generation.  In _2022 Research, Invention, and Innovation Congress: Innovative Electricals and Electronics (RI2C)_ , pages 9–14.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * cstorm125 and lukkiddd (2019) cstorm125 and lukkiddd. 2019.  prachathai67k.  https://github.com/PyThaiNLP/prachathai-67k.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Gugger et al. (2022) Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. 2022.  Accelerate: Training and inference at scale made simple, efficient and adaptable.  https://github.com/huggingface/accelerate.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 328–339, Melbourne, Australia. Association for Computational Linguistics.
  * Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019.  Ctrl: A conditional transformer language model for controllable generation.
  * Khamphakdee and Seresangtakul (2023) Nattawat Khamphakdee and Pusadee Seresangtakul. 2023.  An efficient deep learning for thai sentiment analysis.  _Data_ , 8(5).
  * Kudo (2018) Taku Kudo. 2018.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 66–75, Melbourne, Australia. Association for Computational Linguistics.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Lhoest et al. (2021) Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021.  Datasets: A community library for natural language processing.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 175–184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Lowphansirikul et al. (2021a) Lalita Lowphansirikul, Charin Polpanumas, Nawat Jantrakulchai, and Sarana Nutanong. 2021a.  Wangchanberta: Pretraining transformer-based thai language models.
  * Lowphansirikul et al. (2021b) Lalita Lowphansirikul, Charin Polpanumas, Attapol T. Rutherford, and Sarana Nutanong. 2021b.  A large english–thai parallel corpus from the web and machine-generated text.  _Language Resources and Evaluation_ , 56(2):477–499.
  * Martin et al. (2020) Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020.  CamemBERT: a tasty French language model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7203–7219, Online. Association for Computational Linguistics.
  * Nakayama (2018) Hiroki Nakayama. 2018.  seqeval: A python framework for sequence labeling evaluation.  Software available from https://github.com/chakki-works/seqeval.
  * Nguyen and Tuan Nguyen (2020) Dat Quoc Nguyen and Anh Tuan Nguyen. 2020.  PhoBERT: Pre-trained language models for Vietnamese.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1037–1042, Online. Association for Computational Linguistics.
  * Noiyoo and Thutkawkornpin (2023) Nichaphan Noiyoo and Jessada Thutkawkornpin. 2023.  A comparison of machine learning and neural network algorithms for an automated thai essay quality checking.  In _2023 20th International Joint Conference on Computer Science and Software Engineering (JCSSE)_ , pages 482–487.
  * Pankam et al. (2023) Idhibhat Pankam, Peerat Limkonchotiwat, and Ekapol Chuangsuwanich. 2023.  Two-stage thai misspelling correction based on pre-trained language models.  In _2023 20th International Joint Conference on Computer Science and Software Engineering (JCSSE)_ , pages 7–12.
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011.  Scikit-learn: Machine learning in python.  _Journal of Machine Learning Research_ , 12(85):2825–2830.
  * Phatthiyaphaibun (2019) Wannaphong Phatthiyaphaibun. 2019.  wannaphongcom/thai-ner: Thainer 1.3.
  * Sanh et al. (2020) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.
  * Suriyawongkul et al. (2019) Arthit Suriyawongkul, Ekapol Chuangsuwanich, Pattarawat Chormai, and Charin Polpanumas. 2019.  Pythainlp/wisesight-sentiment: First release.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wongnai.com (2018) Wongnai.com. 2018.  Wongnai-corpus.  https://github.com/wongnai/wongnai-corpus.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.

##  Appendix A Sources of Training Data

Below is the list of sources of raw texts that we use to create our training
data. For datasets that contain multiple languages, we only take their subsets
that contain Thai. The listed data sizes are measured after preprocessing but
before tokenization.

Dataset | Data size |  Source
---|---|---
40-Thai-Children-Stories | 416KB |  https://github.com/dsmlr/40-Thai-Children-Stories
Thai QA | 3.2MB |  https://aiforthai.in.th/corpus.php (registration required)
BeneficialTweetCOVID-19 | 1.9MB |  https://github.com/ENEmyr/BeneficialTweetCOVID-19
mr-tydi | 660KB |  https://huggingface.co/datasets/castorini/mr-tydi
CC100-Thai | 73GB |  https://autonlp.ai/datasets/cc100-thai
miracl-th-corpus-22-12 | 480MB |  https://huggingface.co/datasets/Cohere/miracl-th-corpus-22-12
miracl-th-queries-22-12 | 5.6MB |  https://huggingface.co/datasets/Cohere/miracl-th-queries-22-12
common_voice | 1.4MB |  https://huggingface.co/datasets/common_voice
xlsum | 9.2MB |  https://huggingface.co/datasets/csebuetnlp/xlsum
Thaisong-ML | 63MB |  https://github.com/aofiee/Dataset-Thaisong-ML
flores_101 | 716KB |  https://huggingface.co/datasets/gsarti/flores_101
wiki_lingua | 20MB |  https://huggingface.co/datasets/GEM/wiki_lingua
generated_reviews_enth | 123MB |  https://huggingface.co/datasets/generated_reviews_enth
han-corf-dataset-v1.0 | 1.1MB |  https://huggingface.co/datasets/pythainlp/han-corf-dataset-v1.0
xnli2.0_thai | 1.1MB |  https://huggingface.co/datasets/Harsit/xnli2.0_thai
hc3-24k-th | 88MB |  https://huggingface.co/datasets/Thaweewat/hc3-24k-th
helltaker_thai_localization | 72KB |  https://github.com/nekopaldee/helltaker_thai_localization
iapp_wiki_qa_squad | 5.3MB |  https://huggingface.co/datasets/iapp_wiki_qa_squad
tydiqa-goldp | 5MB |  https://huggingface.co/datasets/khalidalt/tydiqa-goldp
tydiqa-primary | 158MB |  https://huggingface.co/datasets/khalidalt/tydiqa-primary
krathu-500 | 55MB |  https://github.com/Pittawat2542/krathu-500
LimeSoda | 4.2MB |  https://github.com/byinth/LimeSoda
Bactrian-X | 107MB |  https://huggingface.co/datasets/MBZUAI/Bactrian-X
mc4 | 41GB |  https://huggingface.co/datasets/mc4
blognone-20230430 | 47MB |  https://huggingface.co/datasets/Noxturnix/blognone-20230430
xsum_eng2thai | 71MB |  https://huggingface.co/datasets/potsawee/xsum_eng2thai
Prime Minister 29 | 96KB |  https://github.com/PyThaiNLP/lexicon-thai/tree/master/thai-corpus/Prime%20Minister%2029
traditional-thai-songs | 836KB |  https://github.com/jaypeerachai/python-search-engine-for-thai-songs
sentiment_analysis_thai | 528KB |  https://github.com/JagerV3/sentiment_analysis_thai
Thai-Clickbait | 13MB |  https://github.com/9meo/Thai-Clickbait
thai-covid-19-situation | 13MB |  https://github.com/PyThaiNLP/thai-covid-19-situation
thaigov-v2-corpus | 267MB |  https://github.com/PyThaiNLP/thaigov-v2-corpus
thai-joke-corpus | 556KB |  https://github.com/iapp-technology/thai-joke-corpus
Thai-Lao-Parallel-Corpus | 628KB |  https://github.com/PyThaiNLP/Thai-Lao-Parallel-Corpus
thailaw | 349MB |  https://huggingface.co/datasets/pythainlp/thailaw
thai-mscoco-2014-captions | 71MB |  https://huggingface.co/datasets/patomp/thai-mscoco-2014-captions
TNBT | 2.6MB |  https://github.com/pepa65/TNBT
Thai Plagiarism | 882MB |  https://aiforthai.in.th/corpus.php (registration required)

Dataset | Data size |  Source
---|---|---
ThaiQA_LST20 | 13MB |  https://huggingface.co/datasets/SuperAI2-Machima/ThaiQA_LST20
thaiqa_squad | 684KB |  https://huggingface.co/datasets/thaiqa_squad
Thai-Semantic-Textual-Similarity-Benchmark | 280KB |  https://huggingface.co/datasets/mrp/Thai-Semantic-Textual-Similarity-Benchmark
Thai-sentence | 68KB |  https://github.com/PyThaiNLP/Thai-sentence
Thai_sentiment | 2.3MB |  https://huggingface.co/datasets/sepidmnorozy/Thai_sentiment
thai_toxicity_tweet | 496KB |  https://huggingface.co/datasets/thai_toxicity_tweet
Thai-UCC | 24MB |  https://huggingface.co/datasets/nakcnx/Thai-UCC
thai_wikipedia (20230101) | 666MB |  https://github.com/PyThaiNLP/ThaiWiki-clean
Thai Wiki QA | 667MB |  https://copycatch.in.th/corpus/thai-wikiqa-nsc2020.html
alpaca-cleaned-52k-th | 89MB |  https://huggingface.co/datasets/Thaweewat/alpaca-cleaned-52k-th
alpaca-finance-43k-th | 64MB |  https://huggingface.co/datasets/Thaweewat/alpaca-finance-43k-th
chain-of-thought-74k-th | 56MB |  https://huggingface.co/datasets/Thaweewat/chain-of-thought-74k-th
chatmed-5k-th | 4.3MB |  https://huggingface.co/datasets/Thaweewat/chatmed-5k-th
codegen-th | 1.1MB |  https://huggingface.co/datasets/Thaweewat/codegen-th
databricks-dolly-15k-th | 27MB |  https://huggingface.co/datasets/Thaweewat/databricks-dolly-15k-th
gpteacher-20k-th | 24MB |  https://huggingface.co/datasets/Thaweewat/gpteacher-20k-th
hc3-24k-th | 108MB |  https://huggingface.co/datasets/Thaweewat/hc3-24k-th
HealthCareMagic-100k-th | 285MB |  https://huggingface.co/datasets/Thaweewat/HealthCareMagic-100k-th
hh-rlhf-th | 252MB |  https://huggingface.co/datasets/Thaweewat/hh-rlhf-th
instruction-wild-52k-th | 96MB |  https://huggingface.co/datasets/Thaweewat/instruction-wild-52k-th
LaMini-instruction-th | 2.5GB |  https://huggingface.co/datasets/Thaweewat/LaMini-instruction-th
onet-m6-social | 180KB |  https://huggingface.co/datasets/Thaweewat/onet-m6-social
pobpad | 191MB |  https://huggingface.co/datasets/Thaweewat/pobpad
universal_dependencies | 292KB |  https://huggingface.co/datasets/universal_dependencies
Sanook-news | 2.3MB |  https://github.com/sorayutmild/Unsupervised-Thai-Document-Clustering-with-Sanook-news
VISTEC-TP-TH-21 | 38MB |  https://github.com/mrpeerat/OSKut/tree/main/VISTEC-TP-TH-2021
tatoeba.tha | 52KB |  https://huggingface.co/datasets/xtreme/viewer/tatoeba.tha
thai-instructions-rallio | 8.6MB |  https://huggingface.co/datasets/yadapruk/thai-instructions-rallio
OSCAR-2301 | 69GB |  https://huggingface.co/datasets/oscar-corpus/OSCAR-2301
Thai News Articles | 255MB |  https://thematter.co/https://themomentum.co/https://thestandard.co/
pantip-large | 34.2GB |  Raw texts provided by the WangchanBERTa team. Originally packaged by Chaos Theory.
wisesight-large | 62.5GB |  Raw texts provided by the WangchanBERTa team. Originally packaged by Wisesight.

##  Appendix B Hyperparameters

Below is the specifications of the best-performing model hyperparameters used
in the training process of PhayaThaiBERT. * indicates hyperparameters that are
involved in some irregularities during training.

Hyperparameter | Value
---|---
Number of layers | 12
Hidden size | 768
FFN hidden size | 3,072
Attention heads | 12
Dropout | 0.1
Attention dropout | 0.1
Max sequence length | 416
Batch size per GPU | 8
Number of GPUs | 16
Gradient accumulation steps | 32
Effective batch size | 4,096
Warmup steps | 24,000
Peak learning rate | 3e-4
Learning rate decay | Linear*
Max steps | 500,000
Weight decay | 0.01
Adam $\epsilon$ | 1e-6
Adam $\beta_{1}$ | 0.9
Adam $\beta_{2}$ | 0.999
FP16 | True, False*