# MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics
Education

Jia Tracy Shen  Penn State UniversityUniversity ParkPAUSA16802<EMAIL_ADDRESS>,
Michiharu Yamashita  Penn State UniversityUniversity ParkPAUSA ,  Ethan Prihar
Worcester Polytechnic InstituteWorcester, MAUSA ,  Neil Heffernan  Creator of
ASSISTMentsWorcesterMAUSA ,  Xintao Wu  University of Arkansas30 Shuangqing
RdFayettevilleARUSA and  Dongwon Lee  Penn State UniversityUniversity
ParkPAUSA

###### Abstract.

Due to the transfer learning nature of BERT model, researchers have achieved
better performance than base BERT by further pre-training the original BERT on
a huge domain-specific corpus. Due to the special nature of mathematical texts
which often contain math equations and symbols, the original BERT model pre-
trained on general English context will not fit Natural Language Processing
(NLP) tasks in math education well. Therefore, we propose MathBERT, a BERT
pre-trained on large mathematical corpus including pre-k to graduate level
mathematical content to tackle math-specific tasks. In addition, We generate a
customized mathematical vocabulary to pre-train with MathBERT and compare the
performance to the MathBERT pre-trained with the original BERT vocabulary. We
select three important tasks in mathematical education such as knowledge
component, auto-grading, and knowledge tracing prediction to evaluate the
performance of MathBERT. Our experiments show that MathBERT outperforms the
base BERT by 2-9% margin. In some cases, MathBERT pre-trained with
mathematical vocabulary is better than MathBERT trained with original
vocabulary.To our best knowledge, MathBERT is the first pre-trained model for
general purpose mathematics education tasks.

AI, NLP, BERT, Math, Text Classification

††ccs: Applied computing Education††ccs: Computing methodologies Natural
language processing

##  1\. Introduction

In mathematical education, there are three common tasks that cause
researchers/educators headache: (i) large-scale knowledge component (KC)
prediction, noted as $T_{kc}$, (ii) open-ended question answer scoring (Auto-
grading), noted as $T_{ag}$, (iii) knowledge tracing (KT) correctness
prediction, noted as $T_{kt}$. The struggle with KC problem (e.g. a
mathematical skill) is attributed to its tediousness and labor-intensive work
for teachers/tutors to label all the knowledge components when they need to
organize mathematical problems, instructional videos, etc. The traditional way
is utilizing machine learning methodology to classify them via feature
extraction (Karlovčec et al., 2012; Pardos, 2017; Patikorn et al., 2019) with
some decent results. Shen et al. (Shen et al., 2021) is the first work we know
to predict KC via NLP model BERT where they trained BERT on the task-specific
corpus. Open-ended essay or mathematical problem questions are facing a
drastic decline in students’ assignments due the difficulty of developing a
universal automated support in assessing the response quality, causing
favoritism of selecting multiple choice questions to test. According to
Erikson et al. (Erickson et al., 2020)’s study, below 15% of the assigned open
response problems on ASSISTMents (Heffernan and Heffernan, 2014) platform are
ever graded by the teacher from 2010 to 2020. However, open-ended questions
provides critical evaluation in testing students’ true critical thinking and
understanding. Matinez (Martinez, 2010) ’s study showed that constructed
response questions (open response questions) can elicit a larger range of
cognition than that of just multiple choices. Currently, researchers use
feature engineering (Graesser et al., 2000; Sordoni et al., 2015) and deep
learning methods (Mikolov et al., 2013; Pennington et al., 2014; Zhao et al.,
2017; Riordan et al., 2017; Lan et al., 2015; Erickson et al., 2020) to seek a
solution. However, there seems to be a lack of utilizing NLP model to solve
the auto-grading problem. Knowledge tracing, a very important task in the
education domain, is defined as the task of tracing students’ knowledge state,
which represents their mastery of knowledge based on their past learning
activities. Predicting the next question correctness as one of the KT tasks is
well studied (Corbetr and Anderson, 1995; Thai-Nghe et al., 2010; Liu et al.,
2019a; Lee et al., 2019; Pandey and Karypis, 2019) but relies on high-
dimensional sequential data. The current solution is still not able to capture
the complex nature of students’ learning activities throughout the time.

The solutions to the aforementioned three tasks typically framed the problem
as a text classification problem with $T_{kc}$ and $T_{ag}$ being multinominal
classification and $T_{kt}$ as binary classification –i.e., given the input
text, predicts one most relevant KC or auto-grading score out of two or many
other labels: $I(nput)\mapsto text$ and $O(utput)\mapsto
KC/Score/Correctness$.

In light of the huge success from transfer learning models such as ELMo
(Peters et al., 2018), ULMFiT (Howard and Ruder, 2018) and BERT (Devlin et
al., 2019), we propose to use a BERT-like model to predict the aforementioned
three tasks in one shot as BERT has been proved to have outstanding
performances in various NLP tasks. However, directly applying BERT to
mathematical related tasks has limitations. First, the original BERT was
trained mainly on general domain texts (e.g. news article and Wikipedia), it
is difficult to estimate their performance on data sets containing
mathematical texts. Second, the word distributions of general and mathematical
corpora are quite different, which can often be a problem for mathematical
task related models. Therefore, we hypothesize that a special BERT needs to be
trained on mathematical domain corpora to be effective in mathematical related
tasks. That is, we further pre-train on the original BERT (BASE BERT) on huge
mathematical corpora to build a MathBERT. Then we use MathBERT to fine-tune on
the mathematical task-specific text data for classification. Therefore, we
make the following contributions:

  1. (1)

We build and publicly release MathBERT as a resource which is pre-trained on
huge mathematical domain text ranging from pre-k to graduate level
mathematical curriculum, books and paper abstracts.

  2. (2)

We pre-train MathBERT with a custom vocabulary `mathVocab` to reflect the
different nature of mathematical corpora (e.g.mathematical equations and
symbols) and compare its performance to MathBERT pre-trained with original
BERT vocabulary.

  3. (3)

We evaluate the performance of MathBERT for $T_{kc}$, $T_{ag}$ and $T_{kt}$,
and compare its performance to five baseline models.

##  2\. Related Work

The state-of-the-art language model BERT (Bidirectional Encoder
Representations From Transformer) (Devlin et al., 2019) is a pre-trained
language representation model that was trained on 16 GB of unlabeled texts
including Books Corpus and Wikipedia with a total of 3.3 billion words and a
vocabulary size of 30,522. Its advantage over other pre-trained language
models such as ELMo (Peters et al., 2018) and ULMFiT (Howard and Ruder, 2018)
is its bidirectional structure by using the masked language model (MLM) pre-
training objective. The MLM randomly masks 15% of the tokens from the input to
predict the original vocabulary id of the masked word based on its context
from both directions (Devlin et al., 2019). The pre-trained model can be
either directly used to fine-tune on new data for NLP understanding and
inference tasks or further pre-trained to get a new set of pre-trained weights
for transfer learning.

The further pre-training process has become popular in the past two years as
it can usually achieve better results than fine-tuning only strategy.
According to Gururangan et al. (Gururangan et al., 2020), there are two styles
of further pre-training on the original BERT (Devlin et al., 2019): (i)
further pre-train the original BERT on a task-specific data set with tasks
being text classification, question and answering inference, paraphrasing,
etc. Gururangan et al. (Gururangan et al., 2020) call this kind of model Task-
adaptive Pre-trained Model (TAPT). (ii) further pre-train the original BERT on
a domain-specific data set with domains being finance, bio-science, clincal
fields, etc. Gururangan et al. (Gururangan et al., 2020) call this kind of
model Domain-adaptive Pre-trained Model (DAPT). These DAPT Models were often
pre-trained on huge domain corpora (e.g. billions of news articles, clinical
texts or PMC full-text and abstracts) and achieved state-of-art (SOTA)
performance in their domains such as FinBERT (Liu et al., 2020), ClinicalBERT
(Alsentzer et al., 2019), BioBERT (Lee et al., 2020), SCIBERT (Beltagy et al.,
2019). Other DAPT models such as E-BERT (Zhang et al., 2020), MelBERT (Choi et
al., 2021) (Methaphor Detection BERT) not only further pre-train but also
modified the transformer architecture a bit to achieve a better performance
for the domain related tasks. There are also a few works that focus on TAPT
models. Sun et al. (Sun et al., 2019) proposed a detailed process on how to
further a TAPT BERT model and fine-tune for classification task, achieving a
new record accuracy. Shen et al. (Shen et al., 2021) pre-trained a TAPT BERT
model to predict knowledge components surpassing the original BERT performance
by about 2% in accuracy. To compare the performance between DAPT and TAPT,
DAPT models usually achieve better performance (1-8% higher) than TAPT models
(Gururangan et al., 2020). However, TAPT requires less time and resource to
train. In light of this finding, MathBERT is a DAPT model that is further pre-
trained from the original BERT model on a huge mathematical corpus.

##  3\. MathBERT Training

###  3.1. Data set

The Data set we use to pre-train the MathBERT is referred as ‘Math Corpus’
below. The data set we use to fine-tune on MathBERT for classification or pre-
train to create a Task-adaptive Pre-trained (TAPT) BERT is referred as ‘Task
Data’ below.

Math Corpus: MathBERT is pretrained on mathematics related corpus that
comprises mathematic curriculum from pre-k to high school, mathematic text
books written for high school and college students, mathematic course syllabus
from Massive Online Open Courses (MOOC) as well as mathematic paper abstracts
(See in Table 1). We crawled these data from popular mathematic curriculum
websites such as illustrativemathematics.org, utahmiddleschoolmath.org,
engageny.org, free text book website such as openculture.com, MOOC platform
classcentral.com as well as ArXiv.org with a total data size of 3GB and 100
Million tokens. Our data not only contains text but also math symbols and
equations. Among all these data, the text book data is in the PDF format and
we hence converted them into pure text data but still observing the mathematic
symbols and equations (see sample text in Fig. 1).

Table 1. Math Corpus Details

Source | Type | Tokens
---|---|---
arxiv.org | Paper abstract | 64M
classcentral.com | College MOOC math syllabus | 111K
ck12.org | K-12 Math Curriculum | 910K
illustrativemathematics.org | K-12 Math Curriculum | 4M
utahmiddleschoolmath.org | G6-8 Math Curriculum | 2M
engageny.org | Pre-k to HS Math Curriculum | 18M
openculture.com | pre-k to College Math Textbook | 11M

(a) Curriculum

(b) Book Content

(c) arXiv Abstract

Figure 1. Sample mathematical corpora text from curriculum, books and arXiv
paper

Task Data: Based on the three downstream tasks mentioned in Section 1, we note
the task data for KC as $D_{kc}$, the task data for Auto-grading as $D_{ag}$,
and the task data for KT as $D_{ag}$ to evaluate the performance of MathBERT.
The data sets are also used to pre-train for TAPT BERT for performance
comparison in Section 4. All of the three data sets are from from ASSISTMents
(Heffernan and Heffernan, 2014). We use the same mathematical problem data set
as in Shen et al. (Shen et al., 2021)’s work with 13,722 texts and 213 labels
for KC prediction. The Auto-grading task data is the same as in Erikon et al
(Erickson et al., 2020)’s work with 141,186 texts and predicting scores 1 to
5. The KT data is the text data (269,230 texts and 2 labels) of the
ASSISTMents2009 data
111https://sites.google.com/site/assistmentsdata/home/assistment-2009-2010-data/skill-
builder-data-2009-2010, the common benchmark data that researchers in KT field
usually only have access to the numerical data. Among the three tasks, KC task
has the smallest data set (13,722 rows) but biggest number of labels (213
labels) whereas KT task has the largest data set (269,230 rows) but smallest
set of labels (2 labels) (see in Table 2). The choice on the three data sets
are random and we don’t expect the results will be significant better or worse
than if choosing other data sets. When fine-tuning, both the labels and texts
are used (see Column 3 and 4) with split ratio of 72% training, 8% validating,
and 20% testing. When pre-training for TAPT models, only the unlabeled texts
are used (see Column 4) without splitting.

Table 2. Task Data Details. KC: Knowledge Component, KT: Knowledge Tracing.
All data from ASSISTMents platform(Heffernan and Heffernan, 2014)

Task | #Labels | #Texts | #Fine-tune Split
---|---|---|---
Train (72%) | Validate (8%) | Test (20%)
$D_{kc}$ | 213 | 13,722 | 9,879 | 1,098 | 2,745
$D_{ag}$ | 5 | 141,186 | 101,653 | 11,295 | 28,238
$D_{kt}$ | 2 | 269,230 | 193,845 | 21,539 | 53,846

Table 3. Example texts of the three tasks with labels

Task Data | Label | Text
---|---|---
$D_{kc}$ | 8.EE.A.1 | Simplify the expression: (z2)2
Put parentheses around the power
if next to coefficient, for example:
3x2=3($x^{2}$),x5=$x^{5}$
$D_{ag}$ | 5 | Q: Explain your answer
on the box below.
A: because it is the same shape,
just larger, making it similar
$D_{kt}$ | 1 | Q: What is 2.6 + -10.9?
A: -8.3

###  3.2. Training Details

To pre-train our MathBERT more efficiently, we adopt the similar data
processing strategy as ROBERTa model which threads all the sentences together
and split them into a maximum length of 512-token sequence sections (Liu et
al., 2019b). Inspired by SciBERT (Beltagy et al., 2019), we create a custom
mathematical vocabulary `mathVocab` using Huggingface
`BertWordPieceTokenizer`222https://huggingface.co/docs/tokenizers/python/latest/quicktour.html
with the same size of 30,522 from the original BERT. We selected 50 words from
the same rank tier of #2050 to #2100 and discover `mathVocab` has more
mathematical jargon and numbers than the original vocabulary (see in Figure
4).

Table 4. Vocabulary Comparison: origVocab vs. mathVocab

Vocab Type | 50 Selected Vocab (from #2050-#2100)
---|---
origVocab | its, ##a, time, would, no, what,
about, said, we, over, then, other,
so, more, ##e, can, if, like,
back, them, only, some, could, ##i,
where, just, ##ing, during, before, ##n,
| do, ##o, made, school, through, than,
| now, years, most, world, may, between,
| down, well, three, ##d, year, while,
| will, ##ed, ##r
mathVocab | sequ, used, ##very, wr, vect, fl,
describ, properties, sing, deg, symm, hom,
high, mathcal, nys, spaces, answ, ##22,
##als, sin, into, ##ras, three, ##ertain,
##utes, discus, associ, id, length, correspond,
| ##ression, ##ould, ##ics, second,##fficient, invari,
| ##fy, end, map, about, am, ##cul,
| ##00, infin, ##ech, local, structure, right,
| conditions, ##ature, pos

As to training strategy, we pre-train our models in three different ways. 1.
pre-train on the mathematic corpus from Table 1 for a DAPT MathBERT with both
the original and mathematical vocabularies. 2. pre-train on the task texts
from Table 2 for a TAPT Model using original vocabulary only. 3. pre-train on
task texts from MathBERT weights (instead of original BERT weights) with both
original and mathematical vocabulary (see in Fig. 2).

Figure 2. An illustration of training and fine-tuning process of BASE vs. TAPT
vs. DAPT BERT

###  3.3. Training Outcome

We use 8-core TPU machine from Google Colab to pre-train all these models. We
find the largest batch size (bs) we can fit into the TPU memory for our data
set is 128 and the best training learning rate (lr) is $5e-5$ with maximum
sequence length (max-seq) of 512 for all of the models. We measure the
effectiveness of training via Mask Language Modeling (MLM) accuracy where the
model predicts the vocabulary ID of the masked words in a sentence (Devlin et
al., 2019). For training steps, we find both versions of MathBERT reaches its
best result at 600K with MLM accuracy of above 99.8% after training time of 5
days each. The KC and Auto-grading TAPT models reach their best results at
100K steps and the KT TAPT model reaches its best result at 120K steps with
the accuracy of above 99% within 1 day. The MathBERT+TAPT models reach their
best results at steps of 100K for both original and math vocabulary versions
with accuracy of above 99.6% within 1 day (see in Table 5). We release
MathBERT model artifacts trained with `origVocab` and `mathVocab` in both
Tensorflow and Pytorch versions. You can use S3 bucket URL 333http://tracy-
nlp-models.s3.amazonaws.com/mathbert-basevocab
http://tracy-nlp-models.s3.amazonaws.com/mathbert-mathvocab to download the
Tensorflow version of model artifact. The Pytorch version can be downloaded
from the Huggingface Repo `https://huggingface.co/tbs17/MathBERT` or directly
installed within the Huggingface’s framework under the namespace `tbs17` using
the code below.

⬇

1from transformers import AutoTokenizer

2from transformers import AutoModelForMaskedLM

3# Download the MathBERT-basevocab

4tokenizer = AutoTokenizer.from_pretrained("tbs17/MathBERT")

5model = AutoModelForMaskedLM.from_pretrained("tbs17/MathBERT")

6# Download the MathBERT-mathvocab

7tokenizer = AutoTokenizer.from_pretrained("tbs17/MathBERT-custom")

8model = AutoModelForMaskedLM.from_pretrained("tbs17/MathBERT-custom")

Table 5. Training Steps and Accuracy

Model | Task | Steps | MLM ACCU (%)
---|---|---|---
origVocab | mathVocab
MathBERT | / | 600K | 99.85 | 99.95
TAPT | $T_{kc}$ | 100K | 100 | /
$T_{ag}$ | 100K | 99.10 | /
$T_{kt}$ | 120K | 99.04 | /
MathBERT+TAPT | $T_{kc}$ | 100K | 100 | 99.99
$T_{ag}$ | 100K | 99.95 | 99.96
$T_{kt}$ | 100K | 99.67 | 99.68

##  4\. Performance Evaluation

To evaluate MathBERT’s effectiveness on $T_{kc}$, $T_{ag}$ and $T_{kt}$, we
fine-tune on $D_{kc}$, $D_{ag}$ and $D_{kt}$ and compare the performance to
the ones from baseline models. We group the baseline models into four
categories: (1) Prior methods with the best performance (Shen et al., 2021;
Erickson et al., 2020; Lee et al., 2019) (2) Base BERT without any further
pre-training; (3) TAPT BERT models pre-trained on the task specific texts from
original BERT weights; (4) MathBERT+TAPT models pre-trained on the task-
specific texts from MathBERT weights in both `origVocab` and `mathVocab`
versions. We use both ‘F1’ and ‘Accuracy’ (ACCU) to measure $T_{kc}$
prediction results because traditionally KC problems are evaluated by ACCU
(Karlovčec et al., 2012; Pardos, 2017; Patikorn et al., 2019; Rosé et al.,
2005). However, due to the imbalance between the KC labels in $D_{kc}$, we
provide an additional measure (F1) to count in the imbalances. We use ‘Area
Under the Curve’ (AUC) to measure $T_{ag}$ as AUC is the typical measure for
auto-grading method. ‘AUC’ and ‘ACCU’ are used to measure $T_{kt}$ because
historically they are evaluated by both metrics (Lee et al., 2019; Zhang et
al., 2017; Pandey and Karypis, 2019; Piech et al., 2015).

We obtain the best MathBERT-orig fine-tuning result for $T_{kc}$ with lr=5e-5,
bs=64, max-seq=512 and epochs (ep)=25 and lr=5e-5, bs=64, max-seq=512 and ep=5
for $T_{kc}$, and lr=5e-5, bs=128, max-seq=512 and ep=5 for $T_{kt}$.The same
hyperparameters coincides with MathBERT-custom version. All the results are
averaged over five random seeds with significance.

Overall, we see MathBERT pre-trained with `origVocab` (MathBERT-orig) is about
1.28% to 16.97% better and MathBERT pre-trained with `mathVocab` (MathBERT-
custom) is about 1.06% to 16.90% better than the best prior methods for
$T_{kc}$, $T_{ag}$, $T_{kt}$. MathBERT-orig out-performs BASE BERT by about
1.90 % to 7.20% and MathBERT-custom outperforms it by about 1.78% to 7.13%
across all three metrics. We see both versions of MathBERT underperforms the
MathBERT+TAPT models by 0.01 % to 1.66% across all the metrics except for F1
score for $T_{kc}$. This is expected as MathBERT+TAPT was further pre-trained
by adapting it to task-specific data on top of the MathBERT weights (see in
Table 6). Between the two versions of MathBERT, we discover that MathBERT-
custom doesn’t outperform MathBERT-orig when they are directly fine-tuned for
tasks whereas MathBERT-custom outperforms MathBERT-orig when they are used to
further pre-train and fine-tune for tasks (see Rows of MathBERT and
MathBERT+TAPT). In particular, we see the best and second best performance
located between MathBERT and MathBERT+TAPT. For example, for $T_{kc}$, the
best F1 performance is from MathBERT-orig followed by the second best in
MathBERT+TAPT and the best and second-best ACCU are from both of the
MathBERT+TAPT versions (orig and custom). For $T_{ag}$, we find the best AUC
is from MathBERT+TAPT-orig followed by MathBERT-orig. For $T_{kt}$, both the
best AUC and ACCU are from MathBERT+TAPT-custom followed by MathBERT+TAPT-
orig.

Table 6. Performance Comparison: MathBERT vs. Baseline Methods across Five Random Seeds. Bold font indicates best performance and underlined values are the second best. Method | Vocab |  $T_{kc}$ (%) |  $T_{ag}$ (%) |  $T_{kt}$ (%)
---|---|---|---|---
F1 | ACCU | AUC | AUC | ACCU
Prior Best (p) | / | 88.69 | 92.51 | 85.00 | 81.82 | 77.11
BASE-BERT (b) | orig | 90.14 | 91.78 | 88.67 | 88.90 | 86.88
TAPT | orig | 91.77 | 92.96 | 90.34 | 95.88 | 93.49
MathBERT (m) | origVocab | 92.67 | 93.79 | 90.57 | 96.03 | 94.08
mathVocab | 92.42 | 93.57 | 90.45 | 95.95 | 94.01
MathBERT+TAPT (mt) | origVocab | 92.54 | 93.82 | 90.73 | 97.25 | 95.52
mathVocab | 92.65 | 93.92 | 90.46 | 97.57 | 95.67
$\Delta_{m-p}$ | origVocab | 3.98 | 1.28 | 5.57 | 14.21 | 16.97
mathVocab | 3.73 | 1.06 | 5.45 | 14.13 | 16.90
$\Delta_{m-b}$ | origVocab | 2.53 | 2.01 | 1.90 | 7.13 | 7.20
mathVocab | 2.28 | 1.79 | 1.78 | 7.05 | 7.13
$\Delta_{m-t}$ | origVocab | 0.90 | 0.83 | 0.23 | 0.15 | 0.59
mathVocab | 0.65 | 0.61 | 0.11 | 0.07 | 0.52
$\Delta_{m-mp}$ | origVocab | 0.13 | -0.03 | -0.16 | -1.22 | -1.44
mathVocab | -0.23 | -0.35 | -0.01 | -1.62 | -1.66

Below is an example shell script of how users can use MathBERT-orig to fine-
tune their custom data sets for text classification using TPU processor.

⬇

1os.environ[’TFHUB_CACHE_DIR’] = OUTPUT_DIR

2python bert/run_classifier.py \

3 \--data_dir=$dataset \

4 \--bert_config_file=uncased_L-12_H-768_A-12_original/bert_config.json \

5 \--vocab_file=uncased_L-12_H-768_A-12_original/vocab.txt \

6 \--task_name=$TASK \

7 \--output_dir=$OUTPUT_DIR \

8 \--init_checkpoint=$MathBERT-orig_checkpoint \

9 \--do_lower_case=True \

10 \--do_train=True \

11 \--do_eval=True \

12 \--do_predict=True \

13 \--max_seq_length=512 \

14 \--warmup_step=200 \

15 \--learning_rate=5e-5 \

16 \--num_train_epochs=5 \

17 \--save_checkpoints_steps=5000 \

18 \--train_batch_size=64 \

19 \--eval_batch_size=32 \

20 \--predict_batch_size=16 \

21 \--tpu_name=$TPU_ADDRESS \

22 \--use_tpu=True

##  5\. Use Cases

##  6\. Discussion and Limitation

Although we have proven that a BERT specifically pre-trained on mathematical
texts is more effective than the original BERT for mathematics related tasks
with a margin of [1.78%,7.20%], the effect from an in-domain vocabulary
(`mathVocab`) is not what we expect as to MathBERT-custom has worse
performance than MathBERT-orig when directly fine-tuned on but better
performance when further pre-train on task specific data. As SciBERT (Beltagy
et al., 2019) pointed out, the in-domain vocabulary is helpful but the out-
performance over BASE BERT could be mainly from the domain corpus pre-
training. In addition, we want to point out that MathBERT is not only
applicable in text prediction tasks but also for other NLP understanding tasks
such as paraphrasing, question and answering or sentence entailment tasks. The
reason we evaluate MathBERT for $T_{kc}$, $T_{ag}$ and $T_{kt}$ as their data
is relatively easy to get and have been heavily studied on. Users are more
than welcome to use MathBERT for any task tha BERT is suited for.

As to future directions, we are considering deploying MathBERT to two real-
world applications: (1) question generation for mathematics exercises from
math passages, (2) comment generation for open-ended response grading which is
a step beyond auto-grading. We are also considering pre-training another
MathBERT on informal mathematics related texts as opposed to the formal
mathematical content (e.g. math curriculum, book and paper) which the current
MathBERT is pre-trained on. We could potentially use the informal MathBERT to
generate answers/conversations for mathematics tutoring chat bots.

##  7\. Conclusion

In this article, we released MathBERT, a pre-traiend language model with both
original and mathematical vocabulary for mathematical tasks. We showed that
the pre-trained BERT on mathematical corpora is more effective than BASE BERT
with [1.78%, 7.20%] margin in predicting mathematical related tasks. We also
confirmed the performance advantage of DAPT over TAPT BERT by [0.07%, 0.90%].
Users can download both the tensorflow and pytorch version from our github
link444https://github.com/tbs17/MathBERT.

## References

  * (1)
  * Alsentzer et al. (2019) Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew B A Mcdermott. 2019.  Publicly Available Clinical BERT Embeddings. In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_. 72–78.  https://www.aclweb.org/anthology/W19-1909.pdf
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SCIBERT: A pretrained language model for scientific text. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing_. 3615–3620.
  * Choi et al. (2021) Minjin Choi, Sunkyung Lee, Eunseong Choi, Heesoo Park, Junhyuk Lee, Dongwon Lee, and Jongwuk Lee. 2021.  MelBERT : Metaphor Detection via Contextualized Late Interaction using Metaphorical Identification Theories. In _Proceedings of NAACL_.
  * Corbetr and Anderson (1995) Albert T Corbetr and John R Anderson. 1995.  Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge.  _User Modeling and User-Adapted Interaction_ 4 (1995), 253–278.
  * Devlin et al. (2019) Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding. In _2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , Vol. 1. 4171–4186.
  * Erickson et al. (2020) John A Erickson, Anthony F Botelho, Steven Mcateer, Ashvini Varatharaj, and Neil T Heffernan. 2020\.  The Automated Grading of Student Open Responses in Mathematics ACM Reference Format. In _Proceedings of the 10th Learning Analytics and Knowledge Conference_.  https://doi.org/10.1145/3375462.3375523
  * Graesser et al. (2000) Arthur C. Graesser, Peter Wiemer-Hastings, Katja Wiemer-Hastings, Derek Harter, Tutoring Research Group Tutoring Research Group, and Natalie Person. 2000\.  Using Latent Semantic Analysis to Evaluate the Contributions of Students in AutoTutor.  _Interactive Learning Environments_ 8, 2 (2000), 129–147.  https://doi.org/10.1076/1049-4820(200008)8:2;1-b;ft129
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasovi´cmarasovi´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A Smith, and Allen. 2020.  Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Heffernan and Heffernan (2014) Neil T. Heffernan and Cristina Lindquist Heffernan. 2014\.  The ASSISTments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching.  _International Journal of Artificial Intelligence in Education_ 24, 4 (2014), 470–497.  https://doi.org/10.1007/s40593-014-0024-x
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal Language Model Fine-tuning for Text Classification. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_. 328–339.
  * Karlovčec et al. (2012) Mario Karlovčec, Mariheida Córdova-Sánchez, and Zachary A. Pardos. 2012.  Knowledge component suggestion for untagged content in an intelligent tutoring system.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ 7315 LNCS (2012), 195–200.
  * Lan et al. (2015) Andrew S Lan, Divyanshu Vats, Andrew E Waters, and Richard G Baraniuk. 2015.  Mathematical Language Processing: Automatic Grading and Feedback for Open Response Mathematical Questions. In _Proceedings of the Second ACM Conference on Learning @Scale_. 167–176.  https://doi.org/10.1145/2724660.2724664
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Data and text mining BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ (2020), 1234–1240.  https://doi.org/10.1093/bioinformatics/btz682
  * Lee et al. (2019) Youngnam Lee, Youngduck Choi, Junghyun Cho, Alexander R Fabbri, Hyunbin Loh, Chanyou Hwang, Yongku Lee, Sang-Wook Kim, and Dragomir Radev. 2019.  Creating A Neural Pedagogical Agent by Jointly Learning to Review and Assess.  9 (2019).  https://doi.org/10.1145/nnnnnnn.nnnnnnn
  * Liu et al. (2019a) Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. 2019a.  EKT: Exercise-aware knowledge tracing for student performance prediction.  _IEEE Transactions on Knowledge and Data Engineering_ 33, 1 (2019), 100–115.  https://doi.org/10.1109/TKDE.2019.2924374
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov, and Paul G Allen. 2019b.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  https://github.com/pytorch/fairseq
  * Liu et al. (2020) Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao. 2020.  FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence Special Track on AI in FinTech_.
  * Martinez (2010) Michael E Martinez. 2010\.  Educational Psychologist Cognition and the question of test item format.  _Educational Psychologist_ (2010), 207–218.  https://doi.org/10.1207/s15326985ep3404{_}2
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013\.  Efficient estimation of word representations in vector space.  _Workshop Track Proceedings of 1st International Conference on Learning Representations(ICLR)_ January 2013 (2013).
  * Pandey and Karypis (2019) Shalini Pandey and George Karypis. 2019.  A Self-Attentive model for Knowledge Tracing. In _Proceedings of The 12th International Conference on Educational Data Mining_.  https://github.com/chrispiech/DeepKnowledgeTracing/tree/
  * Pardos (2017) Zachary A Pardos. 2017\.  Imputing KCs with Representations of Problem Content and Context. In _Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization_. 148–155.  https://doi.org/10.1145/3079628.3079689
  * Patikorn et al. (2019) Thanaporn Patikorn, David Deisadze, Leo Grande, Ziyang Yu, and Neil Heffernan. 2019.  Generalizability of methods for imputing mathematical skills needed to solve problems from texts.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ 11625 LNAI (2019), 396–405.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  GloVe: Global Vectors for Word Representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 1532–1543.  http://nlp.
  * Peters et al. (2018) Matthew E Peters, Mark Neumann, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations. In _Proceedings of NAACL-HLT_. 2227–2237.
  * Piech et al. (2015) Chris Piech, Jonathan Spencer, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas Guibas, Jascha Sohl-Dickstein, Stanford University, and Khan Academy. 2015\.  Deep Knowledge Tracing. In _Advances in Neural Information Processing Systems_.
  * Riordan et al. (2017) Brian Riordan, Andrea Horbach, Aoife Cahill, Torsten Zesch, and Chong Min Lee. 2017.  Investigating neural architectures for short answer scoring. In _Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications_. 159–168.  https://www.kaggle.com/c/asap-sas
  * Rosé et al. (2005) Carolyn Rosé, Pinar Donmez, Gahgene Gweon, Andrea Knight, Brian Junker, William Cohen, Kenneth Koedinger, and Neil Heffernan. 2005\.  Automatic and Semi-Automatic Skill Coding With a View Towards Supporting On-Line Assessment. In _Proceedings of the conference on Artificial Intelligence in Education_. 571–578.
  * Shen et al. (2021) Jia Tracy Shen, Michiharu Yamashita, Ethan Prihar, Neil Heffernan, Xintao Wu, Sean Mcgrew, and Dongwon Lee. 2021.  Classifying Math Knowledge Components via Task-Adaptive Pre-Trained BERT. In _Proceedings of the Conference on Artificial Intelligence in Education_.  https://github.com/tbs17/TAPT-BERT
  * Sordoni et al. (2015) Alessandro Sordoni, Michel Galley, † Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015.  A Neural Network Approach to Context-Sensitive Generation of Conversational Responses. In _Proceedings of the Annual Conference of the North American Chapter of the ACL_. 196–205.
  * Sun et al. (2019) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019\.  How to Fine-Tune BERT for Text Classification?  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ 11856 LNAI, 2 (2019), 194–206.
  * Thai-Nghe et al. (2010) Nguyen Thai-Nghe, Lucas Drumond, Artus Krohn-Grimberghe, and Lars Schmidt-Thieme. 2010.  Recommender System for Predicting Student Performance.  _Procedia Computer Science_ 1 (2010), 2811–2819.  https://doi.org/10.1016/j.procs.2010.08.006
  * Zhang et al. (2020) Denghui Zhang, Zixuan Yuan, Yanchi Liu, Zuohui Fu, Fuzhen Zhuang, Pengyang Wang, and Hui Xiong. 2020.  E-BERT: Adapting BERT to E-commerce with Adaptive Hybrid Masking and Neighbor Product Reconstruction. (2020).
  * Zhang et al. (2017) Jiani Zhang, Xingjian Shi, Irwin King, and Dit-Yan Yeung. 2017\.  Dynamic Key-Value Memory Networks for Knowledge Tracing.  (2017).  https://doi.org/10.1145/3038912.3052580
  * Zhao et al. (2017) Siyuan Zhao, Yaqiong Zhang, Xiaolu Xiong, Anthony Botelho, and Neil Heffernan. 2017.  A Memory-Augmented Neural Model for Automated Grading. In _Proceedings of ACM Conference on Learning @ Scale_.  https://doi.org/10.1145/3051457.3053982


# An active learning model to classify animal species in Hong Kong

Gareth Lamb, Ching Hei Lo, Jin Wu, Calvin K. F. Lee*
School of Biological Sciences
University of Hong Kong
Hong Kong

###### Abstract

Camera traps are used by ecologists globally as an efficient and non-invasive
method to monitor animals. While it is time-consuming to manually label the
collected images, recent advances in deep learning and computer vision has
made it possible to automating this process [1]. A major obstacle to this is
the generalisability of these models when applying these images to
independently collected data from other parts of the world [2]. Here, we use a
deep active learning workflow [3], and train a model that is applicable to
camera trap images collected in Hong Kong.

_K_ eywords biodiversity monitoring, camera traps, Hong Kong, species
classification

††*Corresponding author

##  1 Introduction

Wildlife management, conservation, and understanding biodiversity requires
accurate monitoring of animals through observation. Camera trap surveys are
popular tools used for observing animals due to their non-invasive nature,
capacity to sample a broad spectrum of species, and continuous operation
throughout the day after they are deployed [4, 5]. While this continuous
nature of camera traps offers amazing opportunities for researchers, they also
generate vast amounts of data that needs to be processed and analysed,
requiring huge investments in time and human efforts.

Advances in deep learning methods have offered promising alternatives to
manual processing of camera trap data, massively reducing time required [6, 7,
3, 8]. Recent advances and developments have led to models that were able to
achieve high accuracies when classifying animals to the species level from
camera trap images at speeds that are impossible when manually labelling by
hand [6, 9].

While the highest accuracies have been achieved when applying these models to
data originating from the same region as the training data, accuracies are
lower when applied to out-of-sample, or “unseen”, images gathered from other
sources [6, 9], as these networks do not generalise well. This means that
despite the relatively low-cost and effort to set up camera trap projects,
these often cannot take advantage of the deep learning models, thus increasing
the cost and effort required to fully make use of the collected data [3].

One method of overcoming having too few labelled images to train a deep
learning model is through via using transfer learning [3]. Transfer learning
involves using information from another similar but different task to reduce
training costs [10], improving model accuracies while reducing training time
and the amount of labelled data required [11, 8, 12]. Transfer learning is
particularly effective for species classification from camera trap images as
existing, openly-available models can provide the information required for
general-purpose image classification tasks such as edge detection, while the
addition of a smaller, specific target dataset can be used to fine-tune the
model [11, 3].

Here, we present a model that was trained through transfer learning
specifically for camera trap photos collected from Hong Kong. In addition to
the model and associated accuracy assessments, we present a pipeline that we
aim to provide an easier method for ecologists to adapt to their own study
sites. Our pipeline used MegaDetector [2, 13] to first generate cropped images
of any animals detected. We then created a model via transfer learning, using
a ResNet50 pre-trained from classifying images from the ImageNet database
[14].

##  2 Animal classification workflow

Prior to classifying the species of detected animals from camera traps, the
images needed to first be sorted, removing any empty images which did not have
any animals present within. To achieve this, we used MegaDetector v5a [13],
which sorted through all the available images and created bounding boxes for
any animals that are present.

After running all the images through MegaDetector, image crops containing only
the animals based on MegaDetector’s outputs are used as input into a species
classification model that was trained specifically using training data from
Hong Kong.

The data used for this study was gathered by different organisations and
groups across Hong Kong. As each group collecting this data had varying
objectives, and conducted their data collection independently, the dataset had
varying camera placement strategies, camera settings, and camera models. To
train the model, we used an initial dataset that included fifteen groupings of
animals/species (Table 1).

Table 1: Number of images within each grouping used to classify animals within the dataset. Grouping | Count
---|---
Birds | 185
Canis Lupis familaris | 396
Lutra lutra | 1445
Felis catus | 80
Herpestes javanicus | 71
Hystrix brachyura | 3911
Macaca mulatta | 1274
Melogale spp. | 74
Muntiacus spp. | 2733
Other animal | 9
Paguma larvata | 165
Prionailurus bengaliensis | 1614
Rodent | 185
Sus scrofa | 2192
Viverricula indica | 2084

##  3 Model development and results

To create a model that is trained specifically to identify animals in Hong
Kong, we first used a transfer learning approach [15]. Taking a pre-trained
ResNet50 model that has learned to classify images from the ImageNet database
[16], we finetuned the model using the camera trap images collected in Hong
Kong. For this initial model, we were able to achieve an overall accuracy of
89% using the test set. However, the generalisability of the model is limited,
this accuracy drops to 50% when the model is applied to an independent dataset
collected in Hong Kong.

A subsequent model was built using an active learning method, following
Norouzzadeh et al., [3]. The model achieved incremental improvements to
accuracy as training images were added (Fig. 1). Using active learning
minimises the time needed to improve the model, as the algorithm is able to
specify which images needs labels as the model is being trained. Our pipeline
makes use of Timelapse [17] to streamline the labelling workflow.

Figure 1: Accuracy, precision and recall of the model as labels are added to
training images.

To test the generalisability of this active learning model, we subsequently
tested the model using an independent dataset using images collected from
other sites in Hong Kong, and show the results using a confusion matrix (Fig.
2).

Figure 2: Confusion matrix generated by applying the active learning model
onto an independent dataset using images collected from other sites in Hong
Kong.

Based on the independent dataset, the model achieved an overall accuracy of
94.1%, along with precision, recall, and F1-score of 93.7%, 81.9%, and 87.4%,
respectively. Specific classes, such as “other animal”, where the total number
of samples available, had much lower accuracies, while more common species
achieved F1-scores higher than 90%. Specifically, “Melogale spp.”, “other
animals”, and “Paguma larvata” had F1-scores lower than 80%. The model was
unable to accurately classify animals into the “other animal” group, as there
was only a total of 18 samples within this class. This class was created as a
group that included animals there were different from each other (e.g. snakes,
tortoises), so no real patterns was detectable within the group.

##  4 Conclusion

We applied a workflow using MegaDetector and an active learning algorithm to
train a deep learning classifier for animals in Hong Kong efficiently and
requiring a relatively small amount of labelled images.

##  5 Acknowledgements

The authors acknowledge Green Power, Kadoorie Farm and Botanical Garden, World
Wide Fund Hong Kong, Ecosystems Limited, Aurecon, and Anthony Yim for
contributing their camera trap data to the project.

## References

  * [1] Guobin Chen, Tony X. Han, Zhihai He, Roland Kays, and Tavis Forrester.  Deep convolutional neural network based species recognition for wild animal monitoring.  In 2014 IEEE International Conference on Image Processing (ICIP), pages 858–862, Paris, France, October 2014. IEEE.
  * [2] Sara Beery, Dan Morris, and Siyu Yang.  Efficient Pipeline for Camera Trap Image Review.  arXiv preprint, 2019.  Publisher: arXiv Version Number: 1.
  * [3] Mohammad Sadegh Norouzzadeh, Dan Morris, Sara Beery, Neel Joshi, Nebojsa Jojic, and Jeff Clune.  A deep active learning system for species identification and counting in camera trap images.  Methods in Ecology and Evolution, 12(1):150–161, January 2021.
  * [4] A. Cole Burton, Eric Neilson, Dario Moreira, Andrew Ladle, Robin Steenweg, Jason T. Fisher, Erin Bayne, and Stan Boutin.  Wildlife camera trapping: a review and recommendations for linking surveys to ecological processes.  Journal of Applied Ecology, 52(3):675–685, June 2015.
  * [5] Oliver R. Wearn, J. Marcus Rowcliffe, Chris Carbone, Marion Pfeifer, Henry Bernard, and Robert M. Ewers.  Mammalian species abundance across a gradient of tropical land-use intensity: A hierarchical multi-species modelling approach.  Biological Conservation, 212:162–171, August 2017.
  * [6] Danielle L. Norman, Philipp H. Bischoff, Oliver R. Wearn, Robert M. Ewers, J. Marcus Rowcliffe, Benjamin Evans, Sarab Sethi, Philip M. Chapman, and Robin Freeman.  Can CNN-based species classification generalise across variation in habitat within a camera trap survey?  Methods in Ecology and Evolution, 14(1):242–251, January 2023.
  * [7] Mohammad Sadegh Norouzzadeh, Anh Nguyen, Margaret Kosmala, Alexandra Swanson, Meredith S. Palmer, Craig Packer, and Jeff Clune.  Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning.  Proceedings of the National Academy of Sciences, 115(25), June 2018.
  * [8] Marco Willi, Ross T. Pitman, Anabelle W. Cardoso, Christina Locke, Alexandra Swanson, Amy Boyer, Marten Veldthuis, and Lucy Fortson.  Identifying animal species in camera trap images using deep learning and citizen science.  Methods in Ecology and Evolution, 10(1):80–91, January 2019.
  * [9] Michael A. Tabak, Mohammad S. Norouzzadeh, David W. Wolfson, Steven J. Sweeney, Kurt C. Vercauteren, Nathan P. Snow, Joseph M. Halseth, Paul A. Di Salvo, Jesse S. Lewis, Michael D. White, Ben Teton, James C. Beasley, Peter E. Schlichting, Raoul K. Boughton, Bethany Wight, Eric S. Newkirk, Jacob S. Ivan, Eric A. Odell, Ryan K. Brook, Paul M. Lukacs, Anna K. Moeller, Elizabeth G. Mandeville, Jeff Clune, and Ryan S. Miller.  Machine learning to classify animal species in camera trap images: Applications in ecology.  Methods in Ecology and Evolution, 10(4):585–590, April 2019.
  * [10] Mohammadreza Iman, Hamid Reza Arabnia, and Khaled Rasheed.  A Review of Deep Transfer Learning and Recent Advancements.  Technologies, 11(2):40, March 2023.
  * [11] Mahbub Hussain, Jordan J. Bird, and Diego R. Faria.  A Study on CNN Transfer Learning for Image Classification.  In Ahmad Lotfi, Hamid Bouchachia, Alexander Gegov, Caroline Langensiepen, and Martin McGinnity, editors, Advances in Computational Intelligence Systems, volume 840, pages 191–202. Springer International Publishing, Cham, 2019.  Series Title: Advances in Intelligent Systems and Computing.
  * [12] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  arXiv preprint, 2014.  Publisher: arXiv Version Number: 1.
  * [13] Microsoft.  MegaDetector, 2023.
  * [14] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  ImageNet Classification with Deep Convolutional Neural Network.  Advances in neural information processing systems, 25, 2012.
  * [15] ME Taylor and P Stone.  Transfer Learning for Reinforcement Learning Domains: A Survey.  Journal of Machine Learning Research, 10:1633–1685, 2009.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  arXiv:1512.03385 [cs], December 2015.  arXiv: 1512.03385.
  * [17] S Greenberg.  Timelapse: An Image Analyser for Camera Traps, 2023.


Measurement of Intermittency for Charged Particles in Au + Au Collisions at
$\sqrt{s_{\mathrm{NN}}}$ = 7.7-200 GeV from STAR

Jin Wu1$\star$ (for the STAR Collaboration)

1 Central China Normal University

*<EMAIL_ADDRESS>
50th International Symposium on Multiparticle Dynamics
(ISMD2021)
12-16 July 2021 10.21468/SciPostPhysProc.?

## Abstract

Local density fluctuations near the QCD critical point can be probed by
intermittency analysis of scaled factorial moments in relativistic heavy-ion
collisions. We report the first measurement of intermittency for charged
particles in Au + Au collisions at $\sqrt{s_{\mathrm{NN}}}$ = 7.7-200 GeV from
the STAR experiment at RHIC. We observe scaling behaviors in central Au + Au
collisions, with the extracted scaling exponent decreasing from mid-central to
the most central Au + Au collisions. Furthermore, the scaling exponent
exhibits a non-monotonic energy dependence with a minimum around
$\sqrt{s_{\mathrm{NN}}}$ = 20-30 GeV in central Au + Au collisions.

###### Contents

  1. 1 Introduction
  2. 2 Analysis Details
  3. 3 Results and Discussion
  4. 4 Summary

##  1 Introduction

The major goal of the Beam Energy Scan (BES) at the Relativistic Heavy Ion
Collider (RHIC) is to explore the phase diagram of quantum chromodynamics
(QCD) [1, 2]. An important landmark of the QCD phase structure is the critical
point (CP), which is the end point of first-order phase boundary between
quark-gluon and hadronic phases [3]. In the thermodynamic limit, the
correlation length diverges at the CP and the system becomes scale invariant
and fractal [4]. It is shown that the density fluctuations near the QCD
critical point form a distinct pattern of power-law or intermittent behavior
in the matter produced in high energy heavy-ion collisions [5].

In analogy to the critical opalescence observed in conventional matter near
the critical point, the related fractal and self-similar geometry of QCD
matter will lead to local density fluctuations that obey intermittent behavior
[5]. Based on the effective action belonging to three-dimensional Ising
universality class, the intermittency of QCD matter is revealed in transverse
momentum spectra as a power-law (scaling) behavior of scaled factorial moment
(SFM) in heavy-ion collisions [5]. An intermittent behavior has observed in Si
+ Si collisions at 158A GeV from the NA49 experiment [6]. Meanwhile, studies
based on a critical Monte Carlo with self-similar property [7] and transport
model with hadronic potentials [8] demonstrate that the intermittency could be
visible in Au + Au collisions at RHIC energies.

##  2 Analysis Details

In high-energy experiments, local power-law fluctuations can be detectable
through the measurements of scaled factorial moment (SFM) which is defined as:

$F_{q}(M)=\frac{\langle\frac{1}{M^{D}}\sum_{i=1}^{M^{D}}n_{i}(n_{i}-1)\cdots(n_{i}-q+1)\rangle}{\langle\frac{1}{M^{D}}\sum_{i=1}^{M^{D}}n_{i}\rangle^{q}},$
(1)

where $M^{D}$ is the number of cells in D-dimensional momentum space, $n_{i}$
is the measured multiplicity in the $i$-th cell, and $q$ is the order of
moment.

Another expected power-law behavior that describes relationship between
$F_{q}(M)$ and $F_{2}(M)$ is defined as [9, 10]:

$F_{q}(M)\propto F_{2}(M)^{\beta q}.$ (2)

Moreover, the scaling exponent$\nu$ quantitatively describes the values of
$\beta_{q}$:

$\beta_{q}\propto(q-1)^{\nu}.$ (3)

Here $\nu$ specifies scaling (power-law) behavior of $F_{q}(M)$. According to
Ginzburg-Landau (GL) theory, the critical $\nu$ is equal to 1.304 in entire
space phase [9], while it is equal to 1.0 from the two-dimensional Ising model
[10].

Figure 1:  $F_{q}(M)$ (up to sixth order) of charged particles in transverse
momentum space for the most central (0-5%) Au + Au collisions at
$\sqrt{s_{\mathrm{NN}}}$ = 7.7-200 GeV in double-logarithmic scale.

The data reported here were obtained from Au + Au collisions at
$\sqrt{s_{\mathrm{NN}}}$ = 7.7, 11.5, 14.5, 19.6, 27, 39, 54.4, 62.4 and 200
GeV, which were recorded by the STAR experiment at RHIC from 2010 to 2017.
Protons ($p$), antiprotons ($\bar{p}$), kaons ($K^{\pm}$) and pions
($\pi^{\pm}$) are analyzed as charged particles, and their identifications are
carried out using the Time Projection Chamber (TPC) and the Time-of-Flight
(TOF) detectors. To avoid the self-correlation, the centrality was determined
from uncorrected charged particles within a pseudo-rapidity window of
$0.5<\mid\eta\mid<1$, which was chosen to be beyond the analysis window of
$\mid\eta\mid<0.5$.

To subtract the background at the level of SFM, a correlator $\Delta F_{q}(M)$
is defined in terms of original and mixed events, i.e., $\Delta
F_{q}(M)=F_{q}(M)^{data}-F_{q}(M)^{mix}$ [6]. In addition, a cell-by-cell
method is proposed for efficiency correction on SFM [11]. The statistical
uncertainties are estimated by Bootstrap method, and the systematic
uncertainties are estimated by varying the experimental requirements for
tracks in the TPC and TOF.

##  3 Results and Discussion

Figure 1 shows $F_{q}(M)^{data}$ and $F_{q}(M)^{mix}$, from the second order
to the sixth order in the most central (0-5%) collisions for various
$\sqrt{s_{\mathrm{NN}}}$. Based on the statistics of BES-I data, $F_{q}(M)$
can be calculated in the range of $M^{2}$ from 1 to $100^{2}$ and up to the
sixth order (q=6). It is observed that $F_{q}(M)^{data}$ is larger than
$F_{q}(M)^{mix}$ at large $M^{2}$ region for various $\sqrt{s_{\mathrm{NN}}}$,
thus a deviation of $\Delta F_{q}(M)$ from zero is present in central Au + Au
collisions.

Figure 2: $\Delta F_{q}(M)$ (q=3-6) as a function of $\Delta F_{2}(M)$ in the
most central (0-5%) Au + Au collisions at $\sqrt{s_{\mathrm{NN}}}$ = 7.7-200
GeV in double-logarithmic scale.

Figure 2 shows $\Delta F_{q}(M)$ (q=3-6), as a function of $\Delta F_{2}(M)$
in the most central (0-5%) collisions for various $\sqrt{s_{\mathrm{NN}}}$. We
clearly observe that the correlators $\Delta F_{q}(M)$ (q=3-6) exhibit scaling
behavior with $\Delta F_{2}(M)$.

The value of $\beta_{q}$ is obtained through a power-law fit of Eq. (2) as
shown in Figure 2, and its statistical error is determined by the fit. Figure
3(a) shows $\beta_{q}$ as a function of $q-1$ in the most central Au + Au
collisions for $\sqrt{s_{\mathrm{NN}}}$ = 7.7-200 GeV. Consistent with
theoretical expectation, $\beta_{q}$ also obeys a good scaling behavior with
q, thus $\nu$ can be obtained through a power-law fit of Eq. (3). Figure 3(b)
shows the extracted $\nu$ as a function of $\langle N_{part}\rangle$ in
central Au + Au collisions at various $\sqrt{s_{\mathrm{NN}}}$. We find that
$\nu$ decreases from mid-central (30-40%) to the most central (0-5%) Au + Au
collisions.

Figure 4 shows the energy dependence of $\nu$ of charged particles in central
Au + Au collisions at $\sqrt{s_{\mathrm{NN}}}$ = 7.7-200 GeV. It is observed
that the $\nu$ exhibits a non-monotonic behavior on collision energy and seems
to reach a minimum around $\sqrt{s_{\mathrm{NN}}}$ = 20-30 GeV. Higher
statistics data from BES-II will help to confirm the trend of energy
dependence of $\nu$.

Figure 3: (a) $\beta_{q}$ (q=3-6) as a function of q-1 in most central Au + Au
collisions at $\sqrt{s_{\mathrm{NN}}}$ = 7.7-200 GeV. (b) $\nu$ as a function
of $\langle N_{part}\rangle$ in central Au + Au collisions. Figure 4: Energy
dependence of $\nu$ for charged particles in Au + Au collisions at
$\sqrt{s_{\mathrm{NN}}}$ = 7.7- 200 GeV. The statistical and systematic errors
are shown in bars and brackets, respectively.

##  4 Summary

In summary, we report the first measurements of intermittency for charged
particles in Au + Au collisions at $\sqrt{s_{\mathrm{NN}}}$ = 7.7-200 GeV from
the STAR experiment. Scaled factorial moments (up to the sixth order) for $p$,
$\bar{p}$, $K^{\pm}$ and $\pi^{\pm}$ within $|\eta|<0.5$, have been measured
in available transverse momentum space. Scaling behavior is clearly visible in
Au + Au collisions which is consistent with theoretical predictions. The
scaling exponent is related to the critical component, and we observe that it
shows a non-monotonic behavior on $\sqrt{s_{\mathrm{NN}}}$ with a dip around
20-30 GeV in the most central (0-5%) Au + Au collisions. This non-monotonic
behavior needs to be understood with more theoretical inputs. With
significantly improved statistics, the RHIC BES Phase-II program will allow
for a more precise measurement of intermittency in heavy-ion collisions.

## Acknowledgements

This work is supported by the National Key Research and Development Program of
China (Grants No. 2020YFE0202002), the National Natural Science Foundation of
China (Grants No. 12122505). And the Ministry of Science and Technology (MoST)
under grant No. 2016YFE0104800 are also acknowledged.

## References

  * [1] A. Bzdak _et al._ ,  _Mapping the Phases of Quantum Chromodynamics with Beam Energy Scan_ ,  Phys. Rept. 853, 1 (2020),  10.1016/j.physrep.2020.01.005.
  * [2] J. Adam et al. (STAR Collaboration),  _Nonmonotonic Energy Dependence of Net-Proton Number Fluctuations_ ,  Phys. Rev. Lett. 126(9), 092301 (2021),  10.1103/PhysRevLett.126.092301.
  * [3] Y. Hatta and M. A. Stephanov,  _Proton number fluctuation as a signal of the QCD critical endpoint_ ,  Phys. Rev. Lett. 91, 102003 (2003),  10.1103/PhysRevLett.91.102003.
  * [4] E. A. De Wolf _et al._ ,  _Scaling laws for density correlations and fluctuations in multiparticle dynamics_ ,  Phys. Rept. 270, 1 (1996),  10.1016/0370-1573(95)00069-0.
  * [5] N. G. Antoniou _et al._ ,  _Critical opalescence in baryonic QCD matter_ ,  Phys. Rev. Lett. 97, 032002 (2006),  10.1103/PhysRevLett.97.032002.
  * [6] T. Anticic et al. (NA49 Collaboration),  _Critical fluctuations of the proton density in A+A collisions at 158 $A$ GeV_,  Eur. Phys. J. C 75(12), 587 (2015),  10.1140/epjc/s10052-015-3738-5.
  * [7] J. Wu _et al._ ,  _Probing QCD critical fluctuations from intermittency analysis in relativistic heavy-ion collisions_ ,  Phys. Lett. B 801, 135186 (2020),  10.1016/j.physletb.2019.135186.
  * [8] P. Li _et al._ ,  _Proton correlations and apparent intermittency in the UrQMD model with hadronic potentials_ ,  Phys. Lett. B 818, 136393 (2021),  10.1016/j.physletb.2021.136393.
  * [9] R. C. Hwa and M. T. Nazirov,  _Intermittency in second order phase transition_ ,  Phys. Rev. Lett. 69, 741 (1992),  10.1103/PhysRevLett.69.741.
  * [10] R. C. Hwa,  _Scaling exponent of multiplicity fluctuation in phase transition_ ,  Phys. Rev. D 47, 2773 (1993),  10.1103/PhysRevD.47.2773.
  * [11] J. Wu _et al._ ,  _Intermittency analysis of proton numbers in heavy-ion collisions at energies available at the BNL Relativistic Heavy Ion Collider_ ,  Phys. Rev. C 104(3), 034902 (2021),  10.1103/PhysRevC.104.034902.


# Modelling the Daily Electricity Demand of Electric Vessels in Plymouth

Lauren Ansell  School of Engineering, Computing and Mathematics, Faculty of
Science and Engineering, Plymouth University, Drake Circus, Plymouth, PL4 8AA,
UK

##  1 Abstract

The international Maritime Organization (IMO) has set the target of reducing
the emissions from the shipping sector to at least 50% of the 2008 levels. One
potential method to cut emissions is to convert vessels to battery powered
propulsion in a similar manner to that which has been adopted for motor
vehicles. Although, battery powered propulsion will not be suitable for all
vessels, the conversion of those that are will lead to an increase in the
energy demand from the national grid. This study uses historic port call data
is used to model the timings of arrivals and the number of vessels in the port
of Plymouth to predict the increase in additional energy demand required for
battery powered vessels through a period of 24 hours as a greater proportion
of the fleet move to battery powered propulsion.
Keywords: electric vessels, energy demand, big data, clean maritime, real-
world vessel data

##  2 Introduction

As we move towards the decarbonisation of travel and the reduction in the use
of fossil fuels, much of the focus of policy previously has been on the motor
vehicle sector [2, 4], however this focus is now shifting towards transport in
the maritime sector. In 2019, the Department of Transport published the Clean
Maritime report [3] stating the aim to reach zero emissions shipping by 2050
to align with the International Maritime Organization (IMO) target of reducing
the emissions in international shipping by at least 50% when compared to the
levels of 2008 [10]. To address the reduction in emissions from vessels,
similar technology employed in the electrification of motor transport can be
employed in marine transport. In April 2021, the city of Plymouth became the
home of the UK’s first sea-going electric ferry [12] and the first UK city to
install a network of electric charge points for battery propulsion vessels
[13].

As potentially more vessels more towards battery powered propulsion, this
implies that a greater draw of energy will be required from the National grid.
Understanding when and how much additional energy could be necessary is key to
making sure the country has robust energy resources in place to handle the
extra demand. Another issue which could also arise, along with ensuring the
supply exists to meet the demand, is verify that the required energy can be
delivered form the nearest substation at the times it will be required without
detriment to the existing users.

Currently we have little to no data regarding the charging demand from
electric vessel, with the technolnogy being new to the sector. However the
demand of energy from the charging of electric vehicles has been extensively
modelled using a variety of methods. The demand for electricity in the Greater
Stuttgart area of Germany was studied by [11] by using the mobiTopp model to
simulate the demand and journeys of electric vehicles and hence derive the
power demand for this on a weekly basis. The study by [1] took a different
approach and combined the use of historical traffic and weather data to
identify overarching patterns of travel through the application of machine
learning techniques. The time at which electric vehicle charging began was
treated as a random variable to identify charging demand patterns during the
day through probabilistic methods. Van Viliey et. al.[16] investigated the
current and future impact electric cars has on electricity demand and the
infrastructure required for generation and distribution by way of efficiency,
cost and emissions of greenhouse gases. Studies have carried out to assess
both the impact on energy demand and the reduction in emissions, for example
the study by [5], which used a bottom-up approach to the modelling, uses 2005
as the base year to predict the emissions reduction and energy demand in 2035
in São Paulo and into the comparison of different modes of electric
transportation to assess the contrast in demand, for example [8] in
Christchurch, New Zealand.

In this paper, the approach we will take to model the daily electricity demand
will be a simplification of that used in [1], where the arrival patterns for
vessels that commonly frequent the port of Plymouth are derived from previous
port calls and the charge demand is calculated using the average number of
arrivals.

The remainder of the paper is organised as follows. Section 3 describes the
data sources used in the analysis; Section 4 reports the results of the
analysis; finally, concluding remarks are presented in Section 5.

##  3 Data and Methodology

Plymouth is an international sea port on the south west coast of the UK which
covers an area of approximately 4620 square kilometres, shown in figure 1. The
city has recently seen the installation of 3 charge points in-situ and fully
operational with charging powers of 25kW, 75kW and 150kW [13], the last of
these being a world first.

Figure 1: Satellite image of the port of Plymouth [7]

To construct the predictions of the energy demand from charging electric
vessels, historical data of the port calls for Plymouth covering the time
period from the 1st January 2019 to the 31st December 2019 were used. This
time period was selected as it was not affected by any restrictions put in
place in response to the Covid-19 pandemic in March 2020 or changes in
regulations following the UK exit from the European Union following the
transition period on 1st January 2021. The data was supplied by Fleetmon [6],
where a port call is defined as a vessel travelling into the polygons of the
Plymouth port area. The original dataset contained information on 88,379
individual calls from June 2009 to November 2021.

The vessels that call at the port of Plymouth range from 2 meters in size, for
vessels usch as pleasure crafts, to 587 meters in size for the Naval vessels.
For this analysis, only vessels under 25 meters in length were considered as
this length of vessel is more suitable for conversion to battery powered
propulsion in regards to the power required from the motor to generate motion.
The dataset was initially cleaned to remove any port calls which fell outside
the time period of interest, after which vessels of length 25 meters or more
where removed. After the removal of these vessels, the dataset was reduced to
9,469 individual port calls and identified 31 different types of vessel. Many
of these vessel types had a small frequency of calls and therefore vessel
types with a frequency of port calls less than 500 were then removed so to
capture the demand which would occur from the most common vessels accessing
the port. This final stage of filtering reduced the types of vessels to five,
shown in table 1 below, and are the primary focus in this analysis:

Vessel type | Frequency
---|---
Sailing ship | 2056
Fishing vessel | 1655
Pusher/Tug | 1546
Yacht | 1176
Trawler | 553
Table 1: Types of vessels under 25m recorded in Plymouth

The resulting dataset shows an acceptable balance between commercial vessels
and leisure vessels, with the former comprising 53% of the dataset, which
allows for predictions from both leisure and commercial usage to be made. Once
the vessel types have been selected for analysis,the lubridate [9] package
within R [14] was employed to extract the day of the week and the hour of
arrival for each of the different vessel types to analyse the distribution
throughout the 24 hour period of a day and are shown in figure 2.

(a) Sailing ship

(b) Fishing vessels

(c) Pushers/Tugs

(d) Yachts

(e) Trawlers

Figure 2: Number of arrivals by vessel type

###  3.1 Commerical vessels

We can see from the figure above that the fishing vessels, figure 2(b), and
trawlers, figure 2(e), display an approximately similar pattern of behaviour
over a period of 24 hours. We see small fluctuations in arrivals before a
clear peak of arrivals beginning at approaching 4pm through until 6 pm. This
pattern of behaviour is replicated on each day of the week, however with a
reduction of almost half in the number of arrivals over the weekend period.
Due to this pattern in the behaviour, we will assume that a trawler or fishing
vessel will leave the port with the battery charged at 100% which provides
enough power until the vessel returns to port at the end of the working day to
recharge.

Similar to the trawlers and fishing vessels, the pushers/tugs record lower
numbers of arrivals over the weekend period. However, during the working week,
we can see that arrivals in the port begin at 6am and continue on throughout
the day, with the number of arrivals fluctuating until approximately 6pm. For
these vessels, we will assume that each vessel will have a fully charged
battery at the commencement of the working period, however, these vessels will
“top up” their charge throughout the day upon returning to port by rapid
charging.

###  3.2 Leisure vessels

As we saw a similar pattern across two of the types of commercial vessels, we
find that the yachts and sailing ships present similar patterns in arrivals
for leisure vessels. Both of these types of vessels see small fluctuations in
arrivals throughout the day with the number of arrivals increasing from midday
for yachts and 2pm for the sailing ships until 4pm. We note that the leisure
vessels display the opposite behaviour to the work vessels with the highest
number of port calls for these types of vessels being recorded on the weekend.
For these vessels, we will assume that the vessel leaves the port with the
battery at 100% and will commence recharging the battery when the vessel
returns to the port to recharge fully.

###  3.3 Calculating the demand

Once the patterns in behaviour have been identified, the potential power
demand per hour will be calculated. To do this, a piecewise linear charging
profile has been adopted of the form:

$P(t)=\begin{cases}P_{r}&\quad 0<t\leq t_{1}\\\
P_{r}\left(\frac{t_{2}-t}{t_{2}-t_{1}}\right)&\quad t_{1}<t\leq
t_{2}\end{cases}$ (1)

This is where $P_{r}$ is the charging power, this is determined by the
charging mode and $t_{1}$ and $t_{2}$ are the times that determine the
variation in the charging power magnitude as the battery nears full charge
[15]. The total power demand is calculated for all the vessels by evaluating
the following summation [1],

$P_{t}=\sum_{i=1}^{n}P_{d}(t).$ (2)

This results in the total hourly power demand being given by the sum of all
the charging power being demanded by vessels at the particular time $t$. For
the modelling it has been assumed that the $P_{r}$ value for the slow charging
scenario is 75kW and the for the rapid charging scenario, we have $P_{r}=150$
kW. We have made the assumption for the slow charging scenario, the total time
taken to achieve full charge is hours and the vessel will begin charging in
the hour at which it arrives back into port. In the rapid charging scenario,
once again the assumption is made that charging begins within the hour the
vessel arrives back into port and it takes one hour to achieve full charge.

(a) Slow charging

(b) Rapid charging

Figure 3:  Battery charge profiles for slow and rapid charging

As the distribution of arrivals remains consistent across a 24 hour period,
the average of the arrivals in the hour interval has been calculated to model
the number of arrivals on any given day and in turn used to calculate the
power demand from charging.

##  4 Result Analysis

As seen with electric vehicles, there will be a gradual increase in the number
of electric vessels and therefore to reflect this the total amount of charging
power demand has been calculated on the condition that 10%, 25%, 50% and 100%
of the vessels under analysis here switched to battery propulsion
respectively, as shown in figure 4.

(a) Sailing ship

(b) Fishing vessels

(c) Pushers/Tugs

(d) Yachts

(e) Trawlers

Figure 4: Potential electricty demand by vessel type

We that if at first only a small percent from each vessel type transfer to
battery powered propulsion, initially there will be a modest amount of
additional demand on the national grid. However this demand for maritime
charging will potential occur where the demand for charging increases
coincides with demand from other sources such as the charging of electric
vehicles following the evening commute. The additive effect of the additional
demand resulting from electrification of maritime vessel could potentially
lead to creating a surge in demand in a short space of time.

As we move towards the 2050 deadline for clean maritime [3], with the entire
vessel fleet moving to battery power propulsion, we see the potential demand
increase to a point where there is no point during a 24 hour period where
energy is not being demanded due to the volatile nature of vessel arrivals. We
also see a potential peak demand of power over 18 MW in the evening period
should all the vessels convert to electric.

##  5 Discussion

The approach taken in this paper has been very simplistic, only taking into
account the average frequency of the different types of vessel arrivals in the
port of Plymouth in each hour and using set proportions of these averages to
predict the potential electricity demand from the charging of the vessels. To
create a more accurate model, further analysis should be done to investigate
how vessel movements could possibly be tied to the weather conditions and
seasonality, for example adopting a similar approach such as the one taken in
[1]. Here machine learning techniques such as clustering could be employed to
identify patterns and seasonal variations in the vessel movement which would
allow for a greater selection of vessel types to be included in the analysis
and therefore producing a more robust prediction of the energy demand.

Although it has been assumed here that there will not be a sudden switch to
electric vessels in the main, the numbers converting to battery power will
gradually increase over time and the analysis here has not taken into account
any growth in the number of vessels at Plymouth that would potentially also be
electric. Additional assumptions used in this analysis include the assumption
that charging will begin once the vessel returns to port and that all the
marine vessels in the port will be using batteries of the same power. Due to
the tiny number of vessels using this technology we currently have no evidence
to support such assumptions and further work should include treating the
charging start time as a random variable, particularly in the case of the
pushers and tugs where it is unlikely that charging will be required every
time they return to the port but is not fully known at this point in time.

Due to the small number of vessel currently using this technology, there is
limited data regarding the run distance and time of a single charge, which may
also be affected by the weather conditions in which the vessel is sailing.
This could result in an increase in the number of port calls as vessels will
be required to return to port and recharge more frequently then the current
distances and times recorded from traditional combustion engines. In addition
to this, due to the technology being in it’s infancy, rapid charging for
marine vessels is not available yet, although discussed here as the preferred
method of charging for vessels which would be required to make multiple
journeys in a 24 hour period, akin to electric buses.

In the future, as more vessels adopt electric motors, the data gathered can be
used to test the assumptions made here more robustly and updated to reflect
the realisation of fully a electric maritime system.

##  6 Acknowledgements

This work was supported through the Clean Maritime Call, a Maritime Research
and Innovation UK (MarRI-UK) initiative supported by the Department for
Transport, grant number 10008313.

## References

  * [1] M. B. Arias and S. Bae.  Electric vehicle charging demand forecasting model based on big data technologies.  Applied Energy, 183:327–339, 2016.
  * [2] Department of Transport.  The road to zero, 2018.
  * [3] Department of Transport.  Clean maritime plan, 2019.
  * [4] Department of Transport.  Taking charge: the electric vehicle infrastructure strategy, 2022.
  * [5] M. V. X. Dias, J. Haddad, L. Horta Nogueira, E. da Costa Bortoni, R. A. Passos da Cruz, R. Akira Yamachita, and J. L. Goncalves.  The impact on electricity demand and emissions due to the introduction of electric cars in the são paulo power system.  Energy Policy, 65:298–304, 2014.
  * [6] J. C. S. GmbH.  www.fleetmon.com.  Accessed on 2022-01-10.
  * [7] Google Maps.  Google map.  https://www.google.co.uk/maps?hl=en, 2021.  Accessed on 2021-01-27.
  * [8] A. Grenier and S. Page.  The impact of electrified transport on local grid infrastructure: A comparison between electric cars and light rail.  Energy Policy, 49:355–364, 2012.  Special Section: Fuel Poverty Comes of Age: Commemorating 21 Years of Research and Policy.
  * [9] G. Grolemund and H. Wickham.  Dates and times made easy with lubridate.  Journal of Statistical Software, 40(3):1–25, 2011.
  * [10] International Maritime Organization.  Initial imo strategy on reduction of ghg emissions from ships, 2018.
  * [11] N. Mallig, M. Heilig, C. Weiss, B. Chlond, and P. Vortisch.  Modelling the weekly electricity demand caused by electric cars.  Procedia Computer Science, 52:444–451, 2015.  The 6th International Conference on Ambient Systems, Networks and Technologies (ANT-2015), the 5th International Conference on Sustainable Energy Information Technology (SEIT-2015).
  * [12] Pymouth Boat Trips.  Plymouth Boat Trips and Voyager Marine launch UK’s first sea-going electric ferry.  https://www.plymouthboattrips.co.uk/plymouth-boat-trips-voyager-marine-launch-uks-first-sea-going-electric-ferry/, 2020\.
  * [13] Pymouth Live.  Plymouth to become first UK city to install electric charging points for boats.  https://www.business-live.co.uk/technology/plymouth-become-first-uk-city-21584083, 2021\.
  * [14] R Core Team.  R: A Language and Environment for Statistical Computing.  R Foundation for Statistical Computing, Vienna, Austria, 2021.
  * [15] Society of Automotive Engineers.  www.standards.sae.org/j1772_201001/.  Accessed on 2022-04-10.
  * [16] O. van Vliet, A. S. Brouwer, T. Kuramochi, M. van den Broek, and A. Faaij.  Energy use, cost and co2 emissions of electric cars.  Journal of Power Sources, 196(4):2298–2310, 2011.