# Removing Non-Stationary Knowledge From Pre-Trained Language Models for
Entity-Level Sentiment Classification in Finance

Guijin Son,1 Hanwool Lee, 2 Nahyeon Kang, 3 Moonjeong Hahm 4

###### Abstract

Extraction of sentiment signals for stock movement prediction from news text,
stock message boards, and business reports have been a rising field of
interest in finance. Building upon past literature, the most recent work
attempt to better capture sentiment from sentences with complex syntactic
structures by introducing aspect-level sentiment classification (ASC). Despite
the growing interest, however, fine-grained sentiment analysis has not been
fully explored in non-English literature due to the shortage of annotated
finance-specific data. Accordingly, it is necessary for non-English languages
to leverage datasets and pre-trained language models (PLM) of different
domains, languages, and tasks to improve their performance. To facilitate
finance-specific ASC research in the Korean language, we build $KorFinASC$, a
Korean aspect-level sentiment classification dataset for finance consisting of
12,613 human-annotated samples, and explore methods of intermediate transfer
learning. Our experiments indicate that past research has been ignorant
towards the potentially wrong knowledge of financial entities encoded during
the training phase, which has overestimated the predictive power of PLMs. In
our work, we use the term “non-stationary knowledge” to refer to information
that was previously correct but is likely to change, and present “TGT-
Masking”, a novel masking pattern to restrict PLMs from speculating knowledge
of the kind. Finally, through a series of transfer learning with TGT-Masking
applied we improve 22.63% of classification accuracy compared to standalone
models on $KorFinASC$.

Over the last decade, sentiment analysis has been widely adopted in the field
of finance to extract sentiment signals from news text, stock message boards,
and business reports (Ding et al. 2014; von Beschwitz, Keim, and Massa 2019).
While it has been expected to automate the extraction of nuances from
sentiment-bearing expressions fully, traditional coarse-grained sentiment
analysis techniques fail to disambiguate sentences including conflicting
sentiments. For example, given the following news headline: “Energy stocks
rallied while S&P 500 falls into a bear market”, FinBERT (Araci 2019) fails to
differentiate between the two contradictory sentiments present . To resolve
such problems, recent studies introduce aspect-level sentiment classification
(ASC), a task that identifies the polarity of each sentence with specific
entities in interest (Consoli, Barbaglia, and Manzan 2022). Compared to
traditional coarse-grained sentiment analysis, however, training a model to
perform ASC requires a dataset with richer annotations.

Despite the growing number of finance-specific ASC datasets made public (e.g.,
SemEval 2017 Task 5, FiQA, SEntiFiN 1.0), the majority of literature is built
upon English corpora, excluding low-resource languages from the scope of
research (Cortis et al. 2017; Maia et al. 2018; Sinha et al. 2022). Aiming to
facilitate finance-specific ASC research using the Korean language we present
$KorFinASC$, a Korean aspect-level sentiment classification dataset for
finance consisting of 12,613 human-annotated samples. $KorFinASC$ is, to the
best of our knowledge, the largest publicly available human-generated finance-
specific dataset and aspect-level classification dataset, written in Korean,
each irrespective of task and domain.

Figure 1: Experimental pipeline with variants of pre-trained language models,
and intermediate tasks for intermediate transfer learning.

Further in this work, we use the pipeline in figure 1 over $KorFinASC$ to
offer a comprehensive investigation into whether datasets of different
domains, languages, and tasks can be leveraged to improve classification
performance. Also, referred to as intermediate transfer learning, multiple
researches has already reported that cross-lingual transfer from English
improves performance in general domain tasks (Pruksachatkun et al. 2020;
Sattar et al. 2021). We find that the nature of finance-specific ASC problems
differs significantly from that of general domain tasks requiring additional
measures to be properly transferred. We postulate that such behavior origins
from “non-stationary knowledge”, once right facts that shift with time.
Modern-day natural language processing (NLP) has been built upon the
assumption that the likelihood of a word’s appearance is constant over time.
However, financial markets are extremely non-stationary systems, where
exploiting past knowledge may lead to a lack of robustness (Schmitt et al.
2013). Our work provides empirical evidence that repeated appearances of
financial entities in pre-training, intermediate training, and fine-tuning
deeply encode non-stationary knowledge regarding financial entities in pre-
trained language models (PLM), hence overestimating its predictive ability on
test datasets collected from the same period of time. To mitigate such biases
from distorting predictions we introduce “TGT-Masking”, a novel masking
pattern that restricts PLMs from speculating non-stationary knowledge. We
discover that combined with a series of transfer learning, “TGT-Masking”
improves 22.63% of classification accuracy compared to standalone models on
$KorFinASC$.

In summary, our contributions are as follows:

  * •

We create and make public $KorFinASC$, the first finance-specific ASC dataset
in Korean.

  * •

We report the existence of “non-stationary knowledge” that has overestimated
the classification accuracy of PLMs and introduce “TGT-Masking” a novel
masking pattern to restrict PLMs from speculating “non-stationary knowledge.”

  * •

We discover that datasets of different domains, languages, and tasks can be
leveraged to improve classification performance for low-resource languages on
finance-specific ASC tasks.

##  1 Background and Related Work

###  1.1 Aspect-Level Sentiment Classification in Finance

Ever since it has been proven that textual data is correlated with stock
price, a growing number of researchers have devised methods to enhance the
predictive power via sophisticated NLP algorithms (Wysocki 1998). Among the
various attempts, the largest branch of research employs sentiment analysis to
translate financial corpora into sentiment signals that enhance investment
decision-making (Das and Chen 2007; Ding et al. 2014). Building upon past
literature, the most recent work attempt to better capture sentiment from
sentences with complex syntactic structures by introducing aspect-level
sentiment classification (ASC) (Consoli, Barbaglia, and Manzan 2022). In
contrast to traditional course-grained sentiment analysis, which is incapable
of disambiguation of sentences with multiple sentiments, ASC involves the
classification of sentiment with regard to each aspect present in the input.
ASC, due to its relatively short history, has not been fully explored in
finance, nonetheless, it is anticipated by academia to improve sentiment
signal extraction by filtering out noises irrelevant to the target in interest
(Sinha et al. 2022).

###  1.2 Intermediate Transfer-Learning

Unlike traditional learning, in which datasets and training are strictly
isolated for each task, transfer learning involves using datasets from
different domains, languages, and tasks to improve performance in a target
task. For instance, given source and target domains $D_{s}$ and $D_{t}$,
source and target tasks $T_{s}$ and $T_{t}$, and source and target languages
$L_{s}$ and $L_{t}$, learning the target conditional probability for target
task from a dataset where $D_{s}\neq D_{t}$, $T_{s}\neq T_{t}$, or $L_{s}\neq
L_{t}$ can be referred to as transfer learning (Zhuang et al. 2019). The
majority of previous research on cross-lingual transfer for general-domain ASC
involves parallel annotated data for both the source and target languages;
ironically such data is equally difficult to acquire (Balahur and Turchi 2012;
Lambert 2015). Only very recently unsupervised cross-lingual transfer was
proposed, and surprisingly it has reported 20 to 23% improvement on state-of-
the-art approaches (Sattar et al. 2021). However, whether such behavior is
equally applicable to the finance domain has not yet been explored.

###  1.3 Financial Biases in Pre-trained Language Models

Leveraging the vast amount of text corpora available on the internet, PLMs
have achieved unprecedented levels of performance, with some even surpassing
human baselines (Raffel et al. 2019; Brown et al. 2020). Recently, however, an
increasing number of research have pointed out that stereotypical biases in
the text corpora have been propagated to PLMs raising questions over the
fairness of these models (Nadeem, Bethke, and Reddy 2021). The issue of
prejudiced knowledge is equally prevalent in the finance domain, as FinBERT,
the most widely used finance-specific PLM model, has been reported to prefer
stocks from the Materials and Industrials sector amongst others (Chuang and
Yang 2022).

##  2 Non-Stationarity in Financial Corpora

Financial markets are well known to be non-stationary, implying that
historical data may not always lead to accurate forecasts (Schmitt et al.
2013). In this paper, we question whether contemporary PLMs, which assume that
the likelihood of a word’s appearance is static, is suitable for financial
decision-making. We hypothesize that the non-stationarity of financial data is
likewise represented in financial corpora, making PLMs trained on such corpora
equally non-stationary. To demonstrate whether PLMs contain outdated knowledge
regarding financial entities we select two PLMs trained on a corpus collected
from a different period of time, BERT and FinBERT. For financial entity C, we
construct a template sentence “C is a [MASK] company” and through a masked
token prediction task probe the two models.

Figure 2 is the 10 most probable words and their conditional probability
predicted by BERT and FinBERT using the above-mentioned prompt for companies
Tesla, and Ford. Surprisingly, with the exception of a small number of
phrases, the two models predict distinct outputs. We presume that this
behavior is due to the difference in the corpora used to train each model.
Moreover, we observe that the generated words can be sorted into the following
four categories.

Figure 2: The 10 most probable words and their conditional probability
predicted by BERT and FinBERT for “Tesla is a [MASK] company.” Figure 3: The
10 most probable words and their conditional probability predicted by BERT and
FinBERT for “Ford is a [MASK] company.”

  1. 1.

Correct Generations: Words such as “technology”, and “motor” are correct
generations that well explain the target entities.

  2. 2.

Wrong Generations: Words such as “chinese”, “british”, “canadian” or “gold”
are factually wrong generations that do not fit the target entities.

  3. 3.

Sentiment-Bearing Generations: FinBERT generates “great” for Tesla and
“dangerous” for Ford, both of which are evaluative terms.

  4. 4.

Once Correct Generations: Interestingly, we also observe generations that were
once correct but outdated. For instance, both models predict the term
“private” for Tesla, which suggests that Tesla is a private firm. However,
Tesla became public in 2010 and has remained so since.

All cases corresponding to categories 2, 3, and 4 are critical if propagated
to downstream tasks; however, factually wrong generations have been dealt with
often in recent years, so our work focuses on sentiment-bearing and once-
correct generations (Weidinger et al. 2021). The two errors are similar in
that they incorrectly use historical distributions to forecast future values,
i.e., they mistake textual data to be stationary.

The existence of sentiment-bearing generations reaffirms that language models
have been trained to prefer a specific entity over the other, even in the
absence of context. It is a dangerous assumption for PLMs to associate a
financial institution with a certain sentiment based on historical data, given
a great quantity of research in finance has already shown that past returns
have little or no bearing on future returns (Fama 1965; Huang et al. 2020).
Once-correct generations, to the best of our knowledge, have never been
mentioned in previous financial NLP studies. However, the existence of once-
correct generations indicates that current PLMs, in which training is ceased
once served, will become obsolete as soon as they do so, meaning that they
cannot keep up with the non-stationary behavior of financial data.
Accordingly, to generically refer to sentiment-bearing and once-correct
generations, two errors that mistake text to be non-stationary, we coincide
the term “non-stationary knowledge” and demonstrate its presence and toxicity
further in our work.

##  3 Dataset Creation

The goal of this research is to explore whether datasets and PLMs of different
languages, tasks, and domains, can have their non-stationary knowledge removed
and then transferred to improve the performance of finance-specific ASC in
low-resource languages. However, finance-specific ASC datasets in languages
other than English, are not publicly available. Accordingly, we create
$KorFinASC$, the first finance-specific fine-grained sentiment analysis
dataset in Korean.

|  |  | Entity-Sentiment Annotations
---|---|---|---
| Num Sample | Average Word Num | Positive | Negative | Neutral | Total
SINGLE_ENT | 5,964 | 15.6 | 2,491 (41.7%) | 2,082 (34.9%) | 1,391 (23.3%) | 5,964
MULTIPE_ENT | 2,779 | 18.4 | 2,079 (31.3%) | 2,532 (30.7%) | 2,532 (38.1%) | 6,649
Total | 8,743 | 16.5 | 4,570 (36.2%) | 3,923 (32.6%) | 3,923 (31.1%) | 12,613
Table 1: KorFin-ASC statistics.

###  3.1 Data Collection

Neural models trained to perform sentiment classification on financial
statements are expected to behave analogously with a human investor; thus we
collect the training data from Naver Finance111https://finance.naver.com, an
analyst report aggregator commonly used in the finance industry. The collected
reports were segmented into sentences and then we applied the following pre-
processing methods to remove undesirable context.

  * •

Remove sentences with less than 40 characters.

  * •

Remove sentences including phrases such as:
All Rights Reserved,  (Click For Contents),
@domain.com,  (Redistribution Prohibited)

  * •

Remove sentences with less than 2 organization entities included.

###  3.2 Annotation process

For the annotation process, three native Korean speakers with degrees in
finance and economics were employed. Each annotator was provided with 10,000
partially overlapping samples and was instructed to identify existing entities
and annotate the subjected sentiment. The annotators were required to think as
real-life investors and assign sentiment between “positive” or “negative”
based on how the sample will contribute to the market value of the entity.
Once they encounter sentences with insufficient information for judgment they
were guided to either label “neutral” or remove the sentence. However,
difficulties exist to reach a perfect inter-annotator agreement, particularly
when more than one annotator is involved in the production of the dataset. To
minimize disagreement, the annotators were provided table 2 from one of the
primary authors for sentiment labeling and were advised to refrain from
speculating previous knowledge beyond the given conditions.

To measure the consistency of annotations from different annotators, an
agreement study has been conducted using a subset of 611 overlapping samples.
Surprisingly only 12 samples, approximately 1.9%, failed to meet a consensus.
The final decision for the above-mentioned samples was provided by the primary
authors.

Sentiment |  Example Cases
---|---
Positive |  Overperformed market expectations/market/sector/competitor, Raised Funding, Analyst buy rating, Entry to new market, Alleviation of risk.
Negative |  Underperformed market expectations/market/sector/competitor, Failed fund raising, Analyst sell rating, Default filing, Employee layoff, Owner risk.
Table 2: Pre-definded cases of Positive/Negative samples provided by the
primary authors.

###  3.3 Dataset Statistics

Resultingly, we build and release $KorFinASC$, a Korean aspect-level sentiment
classification dataset for finance consisting of 12,613 human annotated
samples. To the best of our knowledge, $KorFinASC$ achieves to be the largest
publicly available human-generated finance-specific dataset and aspect-level
sentiment classification dataset, written in Korean, each irrespective of task
and domain.

$KorFinASC$ contains 8,743 unique samples annotated with entities and
corresponding financial nuances. Each sample is tagged either MULTIPLE_ENT or
SINGLE_ENT depending on the number of financial entities included.
MULTIPLE_ENT samples make up 31.8%, contributing 6,649 entity-sentiment
annotations to the dataset. The distribution of sentiments was monitored
throughout the annotation process resulting in a fairly low-class imbalance as
shown in table 1.

However, one feature that concerns the authors is that 4,354 unique entities
appear throughout the dataset, indicating an average of 2.9 appearances per
each. Accordingly, per-entity sentiment distribution is highly imbalanced for
most of the entities which might mislead a neural model trained on this corpus
to directly link an entity to a specific sentiment instead of analyzing its
semantics.

##  4 TGT-Masking

Earlier in this work, we have criticized that PLMs grounding predictions on
non-stationary knowledge, are toxic behaviors likely to overestimate the
model’s predictive power. To demonstrate that this tendency is a language-
agonstic behavior resulting from the nature of PLMs, we conduct a perturbation
sensitivity analysis (PSA) on RoBERTa-Large and KLUE-RoBERTa-Large models
trained using SentFiN and KorFin-ASC (Prabhakaran, Hutchinson, and Mitchell
2019; Liu et al. 2019; Park et al. 2021). PSA measures the sensitivity of a
classification model by calculating the mean, and standard deviation of scores
with entity-perturbed inputs. For this process, we use a template sentence
“[TARGET_ENTITY_1] rallied while [TARGET_ENTITY_2] falls into a bear market.”,
to probe the models. Sentiment scores for each polarity were derived by
applying a softmax function on the model’s output layer. To determine the
perturbation sensitivity for the positive polarity, perturbations were applied
to [TARGET ENTITY 1], and for negative polarity, perturbations were applied to
[TARGET ENTITY 2].

Figure 4: A plot assuming a normal distribution for polarities positive(POS)
and negative(NEG), over datasets KorFinASC and SentFiN.

Formally, $X_{\eta}$ denotes the perturbed sentence with the target entity
replaced with an alternative entity $\eta\in N$. To compare the model’s
behavior to perturbations from in-sample and out-of-sample entities two types
of $\eta$ are introduced. $X^{i}_{\eta}$ denotes a sentence perturbed with an
entity already seen from the training phase while $X^{o}_{\eta}$ denotes a
perturbation with an entity never seen by the model before. Finally,
$f^{P}(X)$ is used to denote the positive polarity score and $f^{N}(X)$ for
the negative polarity score. The calculation of scores is repeated $N$ times
each for English and Korean.

Ideally, a text classification model must produce scores independent from the
financial entities discussed in the context. However, sentences subjected to
perturbations resulted in an output with a standard deviation ranging from
0.016 to 0.2. In figure 4, we observe that models exhibit lower confidence
levels and more uncertainty in response to perturbations involving out-of-
sample entities (distributions in red). For instance, the average mean score
of $F^{P}(X)$ and $F^{N}(X)$ for out-of-sample entities was 0.06 points lower
for both models when compared to sentences with in-sample entity
perturbations. Consequently, we discover that PLMs referencing non-stationary
information result in unstable prediction ability that varies depending on the
financial entity present in the context.

To mitigate the issue we propose “TGT-Masking”, a novel masking pattern that
restricts PLMs from speculating non-stationary knowledge. TGT-Masking applies
to both the training and test phases by substituting the financial entity of
interest with a special token [TGT], a short for “target”. Doing so, not only
prevents the PLM from learning non-stationary traits about the entity during
the training phase but also, by replacing the target entity with an identical
token for each prediction, keeps the output polarity score for homogeneous
contexts constant. We postulate that the proposed method is a better option
for debiasing compared to past approaches which mostly concentrated on de-
biasing augmentations or time-aware embedding (Lee et al. 2021b; Guo, Yang,
and Abbasi 2022; Dhingra et al. 2022). First, debiasing-augmentation
approaches involve using a fixed prompt to generate unseen samples to align
the model’s predicted label distribution, inevitably training the model on an
identical template with varying entities multiple times. However, recent work
has proven that duplication of data harms the model from achieving a higher
accuracy (Lee et al. 2021a). Second, augmentation methods require a generative
language model to generate data samples requiring large amount of computation
resources. Moreover, time-aware embeddings are also a data-heavy solution
since they require timestamped data to address the temporal change of
knowledge. Compared to previous solutions, TGT-Masking is implemented by
simple replace functions in the data loader without requiring additional
generative models, computing resources, or datasets which makes it the
cheapest solution available.

##  5 Experimental Setup

We conduct our experiments, using the pipeline mentioned in figure 1, on
$KorFinASC_{test}$ and $KorFinASC$-$Perturb_{test}$.
$KorFinASC$-$Perturb_{test}$ is a maliciously modified version of the original
test set in which the original financial institutions have been substituted
with out-of-distribution entities, similar to control tasks described in past
literature(Hewitt and Liang 2019). As larger models are expected to be more
robust towards biases, or non-stationary knowledge in this research, we
compare mT5-Large(1.2B params), XLM-RoBERTa-Large(550M params), and KLUE-
RoBERTa-Large (337M params). The size of the three models is based on the
original BERT recipe, however, mT5 is an encoder-decoder model having twice as
many parameters as its encoder-only counterparts. Note that, the difference in
parameter count between encoder-only models comes from the larger vocabulary
size used on XLM-RoBERTa-Large.

###  5.1 Intermediate Transfer Learning

In our research, we investigate the following types of intermediate transfer
learning. When the source and target domains are denoted as $D_{s}$ and
$D_{t}$, source and target tasks as $T_{s}$ and $T_{t}$, and source and target
language as $L_{s}$ and $L_{t}$, and dataset as $DS$:

  * •

Language Transfer (Language.T): Uses $DS=\\{Ds=Dt,Ts=Tt,Ls\neq Lt\\}$, to
inform the model about the task and domain beforehand. In our paper, we use
SentFiN.

  * •

Domain Transfer (Domain.T): Uses $DS=\\{Ds\neq Dt,Ts=Tt,Ls=Lt\\}$, to inform
the model about the task and language beforehand. In our paper, we use Ko-
ABSA.

  * •

Task Transfer (Task.T): Uses $DS=\\{Ds=Dt,Ts\neq Tt,Ls=Lt\\}$, to inform the
model about the domain and language beforehand. In our paper, we use Financial
PhraseBank, and Ko-FinSA.

Additionally, we test variations with two successive transfers, including
Task.T + Language.T, Domain.T + Language.T, and Task.T + Domain.T. Details for
the datasets used for intermediate transfer learning are listed in table 4. In
our work, PLMs are trained for only 3 epochs in each phase unlike Domain
Adaptive Pretraining (DAPT) or Task Adaptive Pretraining (TAPT) described in
past publication (Gururangan et al. 2020). This reduces the models’ repetitive
exposure to identical texts.

|  | $KorFinASC_{test}$ |  $KorFinASC$-$Perturb_{test}$
---|---|---|---
TGT-Masking | Transfer Learning | SINGLE_ENT | MULTIPLE_ENT | SINGLE_ENT | MULTIPLE_ENT
|  | ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1
Model: XLM-RoBERTa-Large
TRUE | N/A | 77.85 | 73.24 | 79.23 | 79.27 | 77.85 | 73.24 | 79.23 | 79.27
| Language.T | 79.24 | 75.74 | 81.79 | 81.77 | 79.24 | 75.74 | 81.79 | 81.77
| Task.T + Language.T | 79.35 | 76.54 | 82.91 | 82.58 | 79.35 | 76.54 | 82.91 | 82.58
| Language.T + Domain.T | 79.12 | 75.66 | 82.21 | 82.24 | 79.12 | 75.66 | 82.21 | 82.24
FALSE | N/A | 79.18 | 75.35 | 81.42 | 81.49 | 65.16 | 60.62 | 60.28 | 58.05
| Language.T | 79.41 | 75.3 | 83.82 | 83.67 | 61.1 | 58.24 | 61.79 | 59.69
| Task.T + Language.T | 82.87 | 79.13 | 83.67 | 83.69 | 72.82 | 68.33 | 58.47 | 57.94
| Language.T + Domain.T | 79 | 73.99 | 82.15 | 82.11 | 62.17 | 61.74 | 64.16 | 61.09
Model : mT5-Large
TRUE | N/A | 70.76 | 64.6 | 57.86 | 55.41 | 70.76 | 64.6 | 57.86 | 55.41
| Language.T | 68.75 | 75.72 | 60.69 | 58.11 | 68.75 | 75.72 | 60.69 | 58.11
| Task.T + Language.T | 76.06 | 70.59 | 79.74 | 77.41 | 76.06 | 70.59 | 79.74 | 77.41
| Language.T + Domain.T | 72.99 | 67.77 | 74.45 | 71.31 | 72.99 | 67.77 | 74.45 | 71.31
FALSE | N/A | 77.06 | 71.55 | 64.11 | 62.31 | 76.12 | 69.77 | 60.64 | 58.76
| Language.T | 78.24 | 73.1 | 79.54 | 77.22 | 73.38 | 65 | 71.07 | 68.59
| Task.T + Language.T | 72.77 | 68.83 | 79.23 | 76.32 | 76.51 | 71.76 | 79.39 | 77.77
| Language.T + Domain.T | 53.64 | 48.89 | 47.43 | 44.03 | 40.29 | 36.42 | 36.04 | 32.98
Model: KLUE-RoBERTa-Large
TRUE | N/A | 81.08 | 76.7 | 79.94 | 79.72 | 81.08 | 76.7 | 79.94 | 79.72
| Task.T | 80.25 | 77.07 | 83.07 | 83.73 | 80.25 | 77.07 | 83.07 | 83.73
| Domain.T | 80.47 | 76.77 | 82.26 | 82.23 | 80.47 | 76.77 | 82.26 | 82.23
| Task.T + Domain.T | 79.3 | 76.72 | 83.01 | 82.41 | 79.3 | 76.72 | 83.01 | 82.41
| Domain.T + Task.T | 79.41 | 76.94 | 82.51 | 82.03 | 79.41 | 76.94 | 82.51 | 82.03
FALSE | N/A | 81.98 | 79.06 | 83.72 | 83.52 | 76.62 | 70.92 | 77.17 | 76.59
| Task.T | 82.09 | 79.30 | 85.43 | 85.18 | 78.4 | 72.77 | 58.71 | 57.88
| Domain.T | 80.41 | 76.31 | 85.13 | 85.05 | 63.72 | 61.26 | 74.13 | 72.22
| Task.T + Domain.T | 81.08 | 78.93 | 83.62 | 83.39 | 71.37 | 69.6 | 70.26 | 68.39
| Domain.T + Task.T | 80 | 75.63 | 80.42 | 80.5 | 52.68 | 44.81 | 41.84 | 43.36
Table 3: Results for XLM-RoBERTa-Large, mT5-Large, and KLUE-RoBERTa-Large over
$KorFinASC_{test}$ and $KorFinASC$-$Perturb_{test}$. All models are trained
for 3 epochs in each training phase. F1(MAX) and accuracy are reported for the
tests.

Dataset | $L_{s}$ | $D_{s}$ | $T_{s}$ | N
---|---|---|---|---
SentFiN | En | Finance | ASC | 14k
Financial PhraseBank | En | Finance | SA | 5k
Ko-ABSA | Ko | General | ASC | 5k
Ko-FinSA | Ko | Finance | SA | 10k

Table 4: Datasets used for intermediate transfer learning. N denotes the
number of samples in the dataset.

###  5.2 Task Formulation

Unlike conventional sentiment analysis tasks which receive raw sentences as
input, target aspects must be specified for ASC tasks. For models which use
TGT-Masking during its training phase, we anticipate the existence of [TGT]
mask to guide the model to analyze the input in the interest of the token
while preventing the encoding of non-stationary information. On the other
hand, for models without TGT-Masking, we follow previous research and present
the target aspect with a special token delimiting it from the original
sequence. Following are example inputs for each condition.

  * •

With TGT-Masking:
Energy stocks rallied while [TGT] falls into a bear market.

  * •

Without TGT-Masking:
Energy stocks rallied while S&P 500 falls into a bear market. [SEP] S&P 500

###  5.3 Optimization

In our work, AdamW with the following parameters was used to optimize the
models: r=3e-4(for mT5), 3e-6(for else), betas=(0.9, 0.999), eps=1e-08, weight
decay=0.01. Models are trained up to 3 epochs for each training phase, with a
maximum length of 300 tokens.

##  6 Results

This section explores and measures the impact of TGT-Masking and intermediate
transfer learning on PLMs. F1(MAX) and accuracy are reported separately for
SINGLE_ENT and MULTIPLE_ENT. SINGLE_ENT scores represent classification
performance for sentences containing a single sentiment. MULTIPLE_ENT denotes
scores for classification performance over inputs with two or more sentiments
included.

###  6.1 TGT-Masking and Intermediate Transfer Learning

Table 3 compares XLM-RoBERTa-Large, mT5-Large, and KLUE-RoBERTa-Large over
$KorFinASC_{test}$ and $KorFinASC$-$Perturb_{test}$. Surprisingly, for XLM-
RoBERTa-Large, mT5-Large, and KLUE-RoBERTa-Large, models trained with TGT-
Masking and intermediate transfer learning overperform their standalone
versions on MULTIPLE_ENT accuracy by 22.63, 19.1, and 5.9 percentage points,
correspondingly. Furthermore, through our experiments, we report 3 notable
findings.

First, we show that TGT-Masking is an effective way for stopping PLMs from
speculating non-stationary knowledge regarding financial entities. Earlier in
our work, we discussed that the existence of non-stationary knowledge disturbs
PLMs from making reliable predictions independent of the entities given in the
input context. In our experiments, models trained with TGT-Masking demonstrate
stable performance across both test sets, but models without TGT-Masking
underperform themselves on $KorFinASC$-$Perturb_{test}$. Interestingly,
mT5-Large with Task.T + Language.T applied results in greater performance on
$KorFinASC$-$Perturb_{test}$; nevertheless, this may also be seen as a failure
of PLMs, since ideally, scores should not change and moreover, the improvement
cannot be explained.

Second, we provide evidence that intermediate transfer learning improves the
performance of PLMs on KorFin-ASC. The performance of the transfer-learned
multilingual models is comparable to that of KLUE-RoBERTa-Large, a PLM native
to the Korean language, recommending the use of intermediate transfer learning
with multilingual models in future research on languages without a native PLM.

Finally, we find that for multilingual models XLM-RoBERTa-Large and mT5-Large,
Task.T is preferable to Domain.T. We assume that this is because acquiring
knowledge in the finance domain is substantially more challenging than
acquiring task-related knowledge. Consequently, it is unlikely for such
behavior to be universal over different tasks and domains.

###  6.2 Transferring in Lower-Resource Environments

In the previous section, models were trained using $KorFinASC$. However, it is
difficult to find datasets with $n>10K$ samples for finance-specific ASC for
the majority of languages. Accordingly, we constructed $KorFinASC(0.3)$, a
variation of the original dataset that is 30% in size. Using $KorFinASC(0.3)$
we investigate the effectiveness of intermediate transfer learning and TGT-
Masking in even lower-resource environments. The scope of experiments in this
section was downsized according to the findings from the preceding section.
First, because we have shown that TGT-Masking effectively eliminates non-
stationary information from distorting the predictions, we no longer test
models without TGT-Masking. Second, for each model, just one transfer learning
setting, the one with the highest performance in previous tests, was
investigated.

| $KorFinASC(0.3)_{test}$
---|---
Transfer Learning | SINGLE_ENT | MULTIPLE_ENT
| ACC | F1 | ACC | F1
Model: XLM-RoBERTa-Large
N/A | 47.18 | 36.13 | 33.58 | 31.25
Task.T + Language.T | 78.47 | 75.54 | 79.21 | 79.26
Model: mT5-Large
N/A | 55.52 | 50.33 | 52.67 | 45.92
Task.T + Language.T | 76.9 | 72.69 | 78.02 | 75.63
Model: KLUE-RoBERTa-Large
N/A | 79.02 | 76.23 | 82.36 | 82.31
Task.T | 77.06 | 73.09 | 79.74 | 79.91

Table 5: Results for XLM-RoBERTa-Large, mT5-Large, and KLUE-RoBERTa-Large over
$KorFinASC(0.3)_{test}$. All models are trained for 3 epochs in each training
phase. F1(MAX) and accuracy are reported for the tests.

We observe that test results, table 5, reaffirm our findings. Despite the
limited data, intermediate transfer learning allows multilingual models XLM-
RoBERTa-Large and mT5-Large, to reach performance that does not differ
significantly from previous experiments using full-scale data.

##  7 Conclusion

Our work explores non-English aspect-level sentiment classification in the
finance domain. We discuss “non-stationary knowledge” existent in financial
corpora that have overestimated the performance of past models and present
“TGT-Masking”, a novel masking pattern, that allows PLMs to output reliable
predictions independent from the aforementioned biases. Finally, we
investigate a series of intermediate transfer learning and conclude that PLM
and datasets from different languages, domains, or tasks can be used to
improve classification performance in low-resource languages. We benchmark our
findings on KorFin-ASC, the first finance-specific ASC dataset in Korean, and
achieve 22.63, 19.1, and 5.9 percentage points improvements with XLM-RoBERTa-
Large, mT5-Large, and KLUE-RoBERTa-Large against their standalone
counterparts. We release our models, dataset, and code 222Our models and code
are available at:
https://github.com/guijinSON/KorFin_ASC.

## References

  * Araci (2019) Araci, D. 2019.  FinBERT: Financial Sentiment Analysis with Pre-trained Language Models.  arXiv:1908.10063.
  * Balahur and Turchi (2012) Balahur, A.; and Turchi, M. 2012.  Multilingual Sentiment Analysis using Machine Translation?  In _Proceedings of the 3rd Workshop in Computational Approaches to Subjectivity and Sentiment Analysis_ , 52–60. Jeju, Korea: Association for Computational Linguistics.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  In _Advances in neural information processing systems_ , volume 33, 1877–1901.
  * Chuang and Yang (2022) Chuang, C.; and Yang, Y. 2022.  Buy Tesla, Sell Ford: Assessing Implicit Stock Market Preference in Pre-trained Language Models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 100–105. Dublin, Ireland: Association for Computational Linguistics.
  * Consoli, Barbaglia, and Manzan (2022) Consoli, S.; Barbaglia, L.; and Manzan, S. 2022.  Fine-grained, aspect-based sentiment analysis on economic and financial lexicon.  In _Knowledge-Based Systems_ , volume 247, 108781.
  * Cortis et al. (2017) Cortis, K.; Freitas, A.; Daudert, T.; Huerlimann, M.; Zarrouk, M.; Handschuh, S.; and Davis, B. 2017.  SemEval-2017 Task 5: Fine-Grained Sentiment Analysis on Financial Microblogs and News.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , 519–535. Vancouver, Canada: Association for Computational Linguistics.
  * Das and Chen (2007) Das, S. R.; and Chen, M. Y. 2007.  Yahoo! For Amazon: Sentiment Extraction from Small Talk on the Web.  In _Management Science_ , volume 53, 1375–1388. INFORMS.
  * Dhingra et al. (2022) Dhingra, B.; Cole, J. R.; Eisenschlos, J. M.; Gillick, D.; Eisenstein, J.; and Cohen, W. W. 2022.  Time-aware language models as temporal knowledge bases.  In _Transactions of the Association for Computational Linguistics_ , volume 10, 257–273. MIT Press.
  * Ding et al. (2014) Ding, X.; Zhang, Y.; Liu, T.; and Duan, J. 2014.  Using Structured Events to Predict Stock Price Movement: An Empirical Investigation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 1415–1425. Doha, Qatar: Association for Computational Linguistics.
  * Fama (1965) Fama, E. F. 1965.  The behavior of stock-market prices.  In _The journal of Business_ , volume 38, 34–105. JSTOR.
  * Guo, Yang, and Abbasi (2022) Guo, Y.; Yang, Y.; and Abbasi, A. 2022.  Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 1012–1023. Dublin, Ireland: Association for Computational Linguistics.
  * Gururangan et al. (2020) Gururangan, S.; Marasović, A.; Swayamdipta, S.; Lo, K.; Beltagy, I.; Downey, D.; and Smith, N. A. 2020.  Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 8342–8360.
  * Hewitt and Liang (2019) Hewitt, J.; and Liang, P. 2019.  Designing and Interpreting Probes with Control Tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2733–2743. Hong Kong, China: Association for Computational Linguistics.
  * Huang et al. (2020) Huang, D.; Li, J.; Wang, L.; and Zhou, G. 2020.  Time series momentum: Is it there?  In _Journal of Financial Economics_ , volume 135, 774–794. Elsevier.
  * Lambert (2015) Lambert, P. 2015.  Aspect-Level Cross-lingual Sentiment Classification with Constrained SMT.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , 781–787. Beijing, China: Association for Computational Linguistics.
  * Lee et al. (2021a) Lee, K.; Ippolito, D.; Nystrom, A.; Zhang, C.; Eck, D.; Callison-Burch, C.; and Carlini, N. 2021a.  Deduplicating training data makes language models better.  arXiv:2107.06499.
  * Lee et al. (2021b) Lee, M.; Won, S.; Kim, J.; Lee, H.; Park, C.; and Jung, K. 2021b.  CrossAug: A Contrastive Data Augmentation Method for Debiasing Fact Verification Models.  In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 3181–3185.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  Roberta: A robustly optimized bert pretraining approach.  arXiv:1907.11692.
  * Maia et al. (2018) Maia, M.; Handschuh, S.; Freitas, A.; Davis, B.; McDermott, R.; Zarrouk, M.; and Balahur, A. 2018.  WWW’18 Open Challenge: Financial Opinion Mining and Question Answering.  In _Companion Proceedings of the The Web Conference 2018_ , WWW ’18, 1941–1942. Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steering Committee.  ISBN 9781450356404.
  * Nadeem, Bethke, and Reddy (2021) Nadeem, M.; Bethke, A.; and Reddy, S. 2021.  StereoSet: Measuring stereotypical bias in pretrained language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 5356–5371. Online: Association for Computational Linguistics.
  * Park et al. (2021) Park, S.; Moon, J.; Kim, S.; Cho, W. I.; Han, J.; Park, J.; Song, C.; Kim, J.; Song, Y.; Oh, T.; et al. 2021.  Klue: Korean language understanding evaluation.  arXiv:2105.09680.
  * Prabhakaran, Hutchinson, and Mitchell (2019) Prabhakaran, V.; Hutchinson, B.; and Mitchell, M. 2019.  Perturbation sensitivity analysis to detect unintended model biases.  arXiv:1910.04210.
  * Pruksachatkun et al. (2020) Pruksachatkun, Y.; Phang, J.; Liu, H.; Htut, P. M.; Zhang, X.; Pang, R. Y.; Vania, C.; Kann, K.; and Bowman, S. R. 2020.  Intermediate-task transfer learning with pretrained models for natural language understanding: When and why does it work?  arXiv:2005.00628.
  * Raffel et al. (2019) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2019.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  arXiv:1910.10683.
  * Sattar et al. (2021) Sattar, K.; Umer, Q.; Vasbieva, D. G.; Chung, S.; Latif, Z.; and Lee, C. 2021.  A Multi-Layer Network for Aspect-Based Cross-Lingual Sentiment Classification.  In _IEEE Access_ , volume 9, 133961–133973. IEEE.
  * Schmitt et al. (2013) Schmitt, T. A.; Chetalova, D.; Schäfer, R.; and Guhr, T. 2013.  Non-stationarity in financial time series: Generic features and tail behavior.  In _EPL (Europhysics Letters)_ , volume 103, 58003. IOP Publishing.
  * Sinha et al. (2022) Sinha, A.; Kedas, S.; Kumar, R.; and Malo, P. 2022.  SEntFiN 1.0: Entity‐aware sentiment analysis for financial news.  In _Journal of the Association for Information Science & Technology_, volume 73.
  * von Beschwitz, Keim, and Massa (2019) von Beschwitz, B.; Keim, D. B.; and Massa, M. 2019.  First to “Read” the News: News Analytics and Algorithmic Trading.  In _The Review of Asset Pricing Studies_ , volume 10, 122–178.
  * Weidinger et al. (2021) Weidinger, L.; Mellor, J.; Rauh, M.; Griffin, C.; Uesato, J.; Huang, P.-S.; Cheng, M.; Glaese, M.; Balle, B.; Kasirzadeh, A.; et al. 2021.  Ethical and social risks of harm from language models.  arXiv:2112.04359.
  * Wysocki (1998) Wysocki, P. D. 1998.  Cheap talk on the web: The determinants of postings on stock message boards.  Technical Report 98025, University of Michigan Business School.
  * Zhuang et al. (2019) Zhuang, F.; Qi, Z.; Duan, K.; Xi, D.; Zhu, Y.; Zhu, H.; Xiong, H.; and He, Q. 2019\.  A Comprehensive Survey on Transfer Learning.  arXiv:1911.02685.


# PERMUTOOLS: A MATLAB Package for
Multivariate Permutation Testing

Michael J. Crosse1,2,3∗ John J. Foxe3,4 Sophie Molholm3,4
1Segotia, Galway, Ireland 2Trinity College Dublin, Ireland
3Albert Einstein College of Medicine, NY 4University of Rochester, NY
<EMAIL_ADDRESS>
###### Abstract

Statistical hypothesis testing and effect size measurement are routine parts
of quantitative research. Advancements in computer processing power have
greatly improved the capability of statistical inference through the
availability of resampling methods. However, many of the statistical practices
used today are based on traditional, parametric methods that rely on
assumptions about the underlying population. These assumptions may not always
be valid, leading to inaccurate results and misleading interpretations.
Permutation testing, on the other hand, generates the sampling distribution
empirically by permuting the observed data, providing distribution-free
hypothesis testing. Furthermore, this approach lends itself to a powerful
method for multiple comparison correction — known as max correction — which is
less prone to type II errors than conventional correction methods. Parametric
methods have also traditionally been utilized for estimating the confidence
interval of various test statistics and effect size measures. However, these
too can be estimated empirically using permutation or bootstrapping
techniques. Whilst resampling methods are generally considered preferable,
many popular programming languages and statistical software packages lack
efficient implementations. Here, we introduce PERMUTOOLS, a MATLAB package for
multivariate permutation testing and effect size measurement.

## Background

### Hypothesis testing

For over a century, researchers have relied on parametric statistical
procedures for conducting hypothesis testing, such as the famous Student’s
t-test (Student, 1908). However, parametric testing was developed out of the
necessity to make inferences about the null distribution, as it was
impractical to generate it empirically. Today, it is possible to do so using
permutation tests. Permutation tests work by permuting the observed data in an
appropriate manner to compute the empirical distribution of the test statistic
of interest — known as the permutation distribution — which approaches the
null distribution (Fisher, 1935). From the permutation distribution, we can
estimate the confidence interval (CI) of the test statistic by computing the
corresponding percentiles (e.g. 2.5% and 97.5% percentiles for 95% CI). We can
also estimate the probability of observing such a result by chance (i.e. the
p-value) by calculating the proportion of the permutation distribution that is
greater than or equal to the magnitude of the test statistic (Fig. 1). The
more permutations generated, the more accurate the estimate. Typically,
permutations in the order of several thousand are required to obtain reliable
p-values, which is computationally trivial for modern computers (Ernst, 2004).

Permutation testing can be applied to any statistical test. As no assumptions
are made about the shape of the underlying distribution, permutation tests
provide distribution-free, nonparametric hypothesis testing without the need
for rank-transformations (Holt and Sullivan, 2023). For independent samples,
permutation tests have been shown to be relatively insensitive to differences
in population variance when samples of equal size are used (Murphy, 1967;
Groppe et al., 2011b). Moreover, permutation tests have been shown to be more
accurate than parametric tests in fields such as biomedical research (Ludbrook
and Dudley, 1998), and yield more reliable p-values and thus are more likely
to produce replicable results (Noguchi et al., 2021). Despite the widespread
use of significance thresholding, it is recommended to treat p-values as a
continuous measure (McShane et al., 2019). However, their continuous value
does not indicate the strength of an effect, only its likelihood. PERMUTOOLS
offers permutation testing and confidence interval estimation for a range of
statistical tests, including the ANOVA (one-way, two-way), t-test (one-sample,
paired-sample, two-sample), F-test (two-sample), Z-test (one-sample), and
correlation test (Pearson, Spearman, rankit). It does not output dichotomous
test results based on an arbitrary significance threshold to discourage this
practice.

### Correcting for multiple comparisons

When conducting multiple hypothesis tests simultaneously, there is an
increased risk of false discoveries or type I errors. However, many of the
traditional correction methods used to control family-wise error rate (FWER)
tend to be overly conservative, resulting in increased type II errors (e.g.
Bonferroni correction, Holm–Bonferroni method). Researchers are often faced
with the delicate task of controlling the trade-off between type I and type II
errors. In the biomedical field, researchers tend to preference controlling
for type I errors, because the consequences of false positives can be
detrimental, e.g. the introduction of an ineffective new therapy or treatment
(Ludbrook and Dudley, 1998). This inherent bias, along with the use of overly
conservative correction methods, can be limiting in the pursuit of scientific
progress.

Another advantage of permutation tests is that they can utilize a powerful
technique for correcting for multiple comparisons — known as max correction —
which is less prone to type II errors than conventional correction methods
(Blair and Karniski, 1993; Westfall and Young, 1993). Max correction (also
known as tmax correction (Blair et al., 1994) or joint correction (Boca et
al., 2014)) works as follows: on each permutation of the data, a separate test
statistic is computed for each variable and the maximum absolute value (or
most extreme positive or negative value) is taken across all variables.
Repeating this procedure thousands of times produces a single permutation
distribution against which the actual test statistic is compared (Fig. 1).
Thus, taking the maximum across more variables naturally produces a more
conservative permutation distribution. This highly intuitive approach provides
strong control of FWER, even for small sample sizes (Gondan, 2010; Groppe et
al., 2011a; Rousselet, 2023). Max correction has been used in various
scientific disciplines, including the study of electrophysiological data
(Blair and Karniski, 1993; Groppe et al., 2011a, b) and human behavioural data
(Gondan, 2010; Shaw et al., 2020; Crosse et al., 2022). PERMUTOOLS
automatically applies max correction to multivariate data, unless specified
otherwise.

### Measuring effect size

Effect size measurement is an equally important part of inferential statistics
(Hentschke and Stüttgen, 2011). Today, most scientific journals require that
authors report the size of an effect, and not just its dichotomous existence.
A common effect size measure used in research is the standardised mean
difference, known as Cohen’s d (Cohen, 1969). Standardised effect sizes have
the advantage of being metric-free, meaning that they can be directly compared
across different studies. For independent samples, Cohen’s d uses the pooled
standard deviation when they are assumed to have equal variances, and an
unpooled estimate when this cannot be assumed. For independent samples with
significantly different variances, an estimate based on the control sample’s
variance can be used, known as Glass’ delta (Glass, 1976). In the case of
ordinal data, an alternative formulation known as Cliff’s d should be used
(Cliff, 1993). PERMUTOOLS gives the option to implement any of the above
standardised effect size measures, as well as several unstandardised measures.

It is also typical to report the confidence interval of an effect size. Whilst
they have traditionally been calculated using parametric methods, these too
can be estimated empirically by generating the sampling distribution using a
resampling procedure known as bootstrapping. Bootstrapping is fundamentally
different to permutation testing in that it resamples with replacement,
whereas permutation testing resamples without replacement. PERMUTOOLS uses an
efficient bootstrapping algorithm that is optimised for multivariate data
analysis.

### Correcting for sample size

As with all statistical tests, the larger the sample size, the more accurately
it can describe the population. However, in certain fields such as cognitive
science, researchers are often required to work with relatively small ($n<50$)
sample sizes due to the limited availability of test subjects or other
logistical study constraints. This can hinder the researcher’s ability to
measure the true, unbiased size of an effect. Despite the advantages of
standardised effect size measures, metrics such as Cohen’s d have been shown
to have an upwards bias of up to about 4% for sample sizes of less than 50.
This bias is somewhat reduced by using the pooled weighted standard deviation
of the samples (Hedges, 1981). Additionally, a bias correction factor can be
applied to the effect size measure and CI, which is approximately equal to
$1-3/(4n-9)$ (Hedges and Olkin, 1985). When this correction factor is applied,
it is usual to refer to the effect size as Hedges’ g. PERMUTOOLS automatically
applies bias correction when calculating Cohen’s d and Glass’ delta (and their
CIs), unless specified otherwise.

Figure 1: Permutation distributions for two-tailed tests based on the
t-statistic (Blue: uncorrected, Red: max-corrected). Multivariate data were
randomly generated to simulate two independent samples, both consisting of 20
variables (i.e. adjusted for 20 comparisons), each with 30 observations (see
example code below). The uncorrected permutation distribution shown is that
generated for the third test (i.e. variable 3).

## Statement of need

Whilst resampling methods have gained widespread acceptance, many popular
programming languages, including MATLAB, lack efficient implementations or
have not yet fully integrated them into their core statistical packages. This
hinders the adoption of these robust and versatile techniques by researchers,
therefore limiting the quality and reliability of quantitative research. To
address this need, PERMUTOOLS provides a comprehensive set of functions in the
MATLAB programming language for conducting resampling-based inferential
statistics that are easy to use and computationally efficient. Moreover, it is
optimised for dealing with large, multivariate datasets and offers powerful
methods for correcting for multiple comparisons and sample size, making it an
invaluable tool for researchers across various fields of quantitative
research. PERMUTOOLS offers a range of new features that distinguish it from
existing statistical software packages. Some of the key features are described
below.

### Key features

  * •

Optimised Resampling Algorithms: PERMUTOOLS utilizes efficient implementations
of resampling algorithms that are optimised for multivariate data, ensuring
efficient processing of even large datasets with negligible compute times.

  * •

Multiple Comparison Correction: PERMUTOOLS implements the powerful max
correction method to adjust p-values and CIs for multiple comparisons when
dealing with multivariate data, reducing the risk of both type I and type II
errors.

  * •

Sample Size Correction: PERMUTOOLS applies a bias correction factor to the
relevant standardised effect size measures and their CIs to adjust for any
inherent inflation due to sample size, ensuring less biased estimates for
small ($n<50$) samples.

  * •

Multivariate Processing: PERMUTOOLS handles multivariate datasets with ease,
allowing for multiple tests to be performed simultaneously, as well as the
option to perform pairwise comparisons between every combination of variables
in a matrix (e.g. correlation matrix).

  * •

Continuous Framework: PERMUTOOLS does not output test results under the
traditional dichotomous decision-based framework (i.e. $H=0,1$). Instead, it
outputs various quantitative measures, encouraging researchers to interpret
their results in a continuous and holistic manner.

## Using PERMUTOOLS

The PERMUTOOLS functions are designed to mimic the API of the equivalent
parametric functions in MATLAB, with the addition of the prefix ‘permu’ at the
beginning of the function name. For example, to conduct a permutation test
based on the t-statistic, the usual function `ttest()` becomes `permuttest()`
etc. The input and output arguments are the same as before with the exception
that the first output variable is always the test statistic (and not a
dichotomous test result). An additional output variable containing the
sampling distribution is included, as well as additional resampling-related
input arguments that are described in the help documentation of each function.
Unlike many existing statistical software packages, PERMUTOOLS uses a
consistent input/output argument framework across all its functions to
optimise usability.

### Example

The following example illustrates typical usage of the PERMUTOOLS toolbox.
Here, we compare the means of two independent multivariate samples using
permutation tests based on the t-statistic with max correction, and contrast
the results with the equivalent parametric tests in MATLAB (i.e. two-sample
t-tests). We then measure the associated effect sizes based on the bias-
corrected standardised mean difference (i.e. Hedges’ g), as well as the
corresponding 95% CIs using both a bootstrapping and parametric approach.

First, we generate random multivariate data for two independent samples X and
Y. Both samples have 20 variables, each with a mean value of approximately
$0$, except for the first 10 variables of Y which have a mean value of
approximately $-1$. Each variable contains 30 observations.

⬇

% Generate random data

rng(42);

x = randn(30,20);

y = randn(30,20);

y(:,1:10) = y(:,1:10)-1;

Because we generated the random multivariate data from the same (normal)
distribution, we can assume that their variances are approximately equal. If
we could not assume this, we would first conduct a two-tailed test of variance
based on the F-statistic using PERMUTOOLS’ `permuvartest2()` function. Thus,
we can proceed using the standard Student’s t-statistic (as opposed to Welch’s
t-statistic). The two-sample permutation tests are implemented using
PERMUTOOLS’ `permuttest2()` function and the equivalent parametric tests are
implemented using MATLAB’s `ttest2()` function. By default, max correction is
applied to the permutation tests.

⬇

% Run MATLAB's two-sample parametric t-test

[h1,p1,ci1,stats1] = ttest2(x,y);

% Run PERMUTOOLS' two-sample permutation t-test

[t2,p2,ci2,stats2] = permuttest2(x,y);

To measure the corresponding effect sizes, we compute Hedges’ g and the 95%
CIs using an efficient bootstrapping procedure. Effect size measures and
bootstrapped CIs are calculated using PERMUTOOLS’ `booteffectsize()` function.
The equivalent parametric measures are calculated using MATLAB’s
`meanEffectSize()` function. By default, bias-correction is applied in both
cases.

⬇

% Run MATLAB's parametric effect size analysis

d3 = zeros(1,20);

ci3 = zeros(2,20);

for j = 1:20

stats3 = meanEffectSize(x(:,j),y(:,j),'effect','cohen','paired',0);

d3(j) = stats3.Effect;

ci3(:,j) = stats3.ConfidenceIntervals';

end

% Run PERMUTOOLS' bootstrapped effect size analysis

[d4,ci4,stats4] = booteffectsize(x,y,'effect','cohen','paired',0);

To compare the results of our parametric and resampling analyses, we next plot
some of the statistics as a function of variable (Fig. 2). The effect of max
correction is clearly visible on the test statistic CIs and the resulting
p-values, which are consistently more conservative than those of the
uncorrected parametric tests (Fig. 2, left, middle). Importantly, spurious
effects observed in the parametric case (e.g. variable 20) did not survive max
correction in the permutation tests. In the effect size analysis, the 95% CIs
estimated via bootstrapping appear to approximate the parametric CIs
reasonably well (Fig. 2, right).

⬇

% Set up figure

figure('Name','Permutation Tests & Effect Size Analysis')

set(gcf,'color','w')

xaxis = 1:20;

% Plot test statistic & CI

subplot(1,3,1), hold on

plot(xaxis,stats2.mu,'LineWidth',2.5)

plot(xaxis,ci1,'k',xaxis,ci2,'--r','LineWidth',1)

xlim([0,21]), ylim([-3,3]), box on, grid on

title('Test Statistic'), xlabel('variable'), ylabel('X-Y')

legend('mean difference','95% CI (param.)','','95% CI (perm.)')

% Plot p-value

subplot(1,3,2), hold on

plot(xaxis,p1,'k',xaxis,p2,'--r','LineWidth',1.5)

ylim([0,1]), xlim([0,21]), box on, grid on

title('{\itP}-value'), xlabel('variable'), ylabel('probability')

legend('{\itp}-value (param.)','{\itp}-value (perm.)')

% Plot effect size & CI

subplot(1,3,3), hold on

plot(xaxis,d4,'LineWidth',2.5)

plot(xaxis,ci3,'k',xaxis,ci4,'--r','LineWidth',1)

ylim([-2,6]), xlim([0,21]), box on, grid on

title('Effect Size'), xlabel('variable'), ylabel('Hedges'' {\itg}')

legend('effect size','95% CI (param.)','','95% CI (boot.)')

From the above statistical analysis, we can examine the results of each
individual pairwise comparison between X and Y, which are contained in the
variables output by the PERMUTOOLS functions. Taking the first variables of X
and Y as an example, we see that the mean of X1 ($M=-0.06$, $SD=0.91$) was
significantly greater than the mean of Y1 ($M=-1.09$, $SD=0.86$), even after
adjusting for multiple comparisons ($t(58)=4.49$, $p=0.0008$, Hedges’
$g=1.14$, 95CI $[0.68,1.72]$). We recommend always reporting the absolute
values of the tests as shown here, in particular, with the effect size and CI
included.

## Future work

PERMUTOOLS is under active development, and new functions and features are
constantly being added to it as needed. Future work aims to expand the toolbox
to provide new permutation-based tests, including repeated measures ANOVAs
(one-way, two-way), and multi-way ANOVAs (n-way), as well as expanding the
functionality of existing ANOVA tests to be able to deal with unbalanced and
multivariate samples. Whilst permutation tests have been shown to outperform
nonparametric tests based on rank transformations (Holt and Sullivan, 2023),
there are certain situations where a rank-transformation approach is
desirable; for example, when dealing with ordinal data and outliers. To
accommodate a wider range of data types, future work aims to develop
permutation-based solutions for nonparametric tests based on rank
transformations, e.g. Sign test, Wilcoxon signed rank test, Mann-Whitney
U-test, Kruskal-Wallis test, Friedman test, etc.

PERMUTOOLS is maintained by the corresponding author but accepts contributions
from the research community at large. If you would like to contribute to
PERMUTOOLS, or request that a specific statistical test or feature be added to
it, please email the corresponding author at the email address provided above,
or use the formal channels on GitHub, such as creating a pull request or
opening an issue.

Figure 2: Results of the permutation tests and effect size analysis. Left:
Test statistic (unstandardised) and 95% CI (parametric and permutation) for
each test. Middle: P-value (parametric and permutation) for each test. Right:
Effect size (Hedges’ g) and 95% CI (parametric and bootstrapped) for each
test.

## Availability

PERMUTOOLS is freely available to download from GitHub
(https://github.com/mickcrosse/PERMUTOOLS) and is compatible with Windows,
Linux, and macOS. Example code demonstrating its usage is included in the
examples folder and on the GitHub page.

## Acknowledgments

We would like to thank David M. Groppe for his helpful correspondences during
the development of the toolbox, as well as his open-source code which served
as a valuable benchmark. This work was supported by the National Institute of
Mental Health of the National Institutes of Health (NIH) under award number
R01MH085322 (S.M. and J.J.F.), and the Eunice Kennedy Shriver National
Institute of Child Health and Human Development of the NIH under the award
numbers P50HD105352, previously U54HD090260 (Rose F. Kennedy Intellectual and
Developmental Disabilities Research Center), and P50HD103536 (University of
Rochester Intellectual and Developmental Disabilities Research Center).

## Author contributions

M.J.C. conceived of the toolbox and manuscript, developed the toolbox and
example code, and wrote the first draft of the manuscript. S.M. and J.J.F.
supervised and edited the manuscript.

## References

  * Blair and Karniski [1993] R Clifford Blair and Walt Karniski.  An alternative method for significance testing of waveform difference potentials.  _Psychophysiology_ , 30(5):518–524, 1993.
  * Blair et al. [1994] R Clifford Blair, James J Higgins, Walt Karniski, and Jeffrey D Kromrey.  A study of multivariate permutation tests which may replace hotelling’s t2 test in prescribed circumstances.  _Multivariate Behavioral Research_ , 29(2):141–163, 1994.  ISSN 0027-3171.
  * Boca et al. [2014] Simina M Boca, Rashmi Sinha, Amanda J Cross, Steven C Moore, and Joshua N Sampson.  Testing multiple biological mediators simultaneously.  _Bioinformatics_ , 30(2):214–220, 2014.
  * Cliff [1993] Norman Cliff.  Dominance statistics: Ordinal analyses to answer ordinal questions.  _Psychological bulletin_ , 114(3):494, 1993.
  * Cohen [1969] Jacob Cohen.  _Statistical power analysis for the behavioral sciences_.  Academic press, 1969.
  * Crosse et al. [2022] Michael J Crosse, John J Foxe, Katy Tarrit, Edward G Freedman, and Sophie Molholm.  Resolution of impaired multisensory processing in autism and the cost of switching sensory modality.  _Communications biology_ , 5(1):601, 2022.
  * Ernst [2004] Michael D Ernst.  Permutation methods: a basis for exact inference.  _Statistical Science_ , pages 676–685, 2004.
  * Fisher [1935] Ronald A Fisher.  _The Design of Experiments_.  Hafner, New York, 1935.
  * Glass [1976] Gene V Glass.  Primary, secondary, and meta-analysis of research.  _Educational researcher_ , 5(10):3–8, 1976.
  * Gondan [2010] Matthias Gondan.  A permutation test for the race model inequality.  _Behavior Research Methods_ , 42(1):23–28, 2010.  ISSN 1554-351X.
  * Groppe et al. [2011a] David M Groppe, Thomas P Urbach, and Marta Kutas.  Mass univariate analysis of event‐related brain potentials/fields i: A critical tutorial review.  _Psychophysiology_ , 48(12):1711–1725, 2011a.  ISSN 0048-5772.
  * Groppe et al. [2011b] David M Groppe, Thomas P Urbach, and Marta Kutas.  Mass univariate analysis of event-related brain potentials/fields ii: Simulation studies.  _Psychophysiology_ , 48(12):1726–1737, 2011b.
  * Hedges [1981] Larry V Hedges.  Distribution theory for glass’s estimator of effect size and related estimators.  _journal of Educational Statistics_ , 6(2):107–128, 1981.
  * Hedges and Olkin [1985] Larry V Hedges and Ingram Olkin.  _Statistical methods for meta-analysis_.  Academic press, 1985.
  * Hentschke and Stüttgen [2011] Harald Hentschke and Maik C Stüttgen.  Computation of measures of effect size for neuroscience data sets.  _European Journal of Neuroscience_ , 34(12):1887–1894, 2011.
  * Holt and Sullivan [2023] Charles A Holt and Sean P Sullivan.  Permutation tests for experimental data.  _Experimental Economics_ , pages 1–38, 2023.
  * Ludbrook and Dudley [1998] John Ludbrook and Hugh Dudley.  Why permutation tests are superior to t and f tests in biomedical research.  _The American Statistician_ , 52(2):127–132, 1998.
  * McShane et al. [2019] Blakeley B McShane, David Gal, Andrew Gelman, Christian Robert, and Jennifer L Tackett.  Abandon statistical significance.  _The American Statistician_ , 73(sup1):235–245, 2019.
  * Murphy [1967] Brian P Murphy.  Some two-sample tests when the variances are unequal: a simulation study.  _Biometrika_ , 54(3-4):679–683, 1967.
  * Noguchi et al. [2021] Kimihiro Noguchi, Frank Konietschke, Fernando Marmolejo-Ramos, and Markus Pauly.  Permutation tests are robust and powerful at 0.5% and 5% significance levels.  _Behavior Research Methods_ , 53(6):2712–2724, 2021.
  * Rousselet [2023] Guillaume A Rousselet.  Using cluster-based permutation tests to estimate meg/eeg onsets: how bad is it?  _bioRxiv_ , pages 2023–11, 2023.
  * Shaw et al. [2020] Luke H Shaw, Edward G Freedman, Michael J Crosse, Eric Nicholas, Allen M Chen, Matthew S Braiman, Sophie Molholm, and John J Foxe.  Operating in a multisensory context: Assessing the interplay between multisensory reaction time facilitation and inter-sensory task-switching effects.  _Neuroscience_ , 436:122–135, 2020.
  * Student [1908] Student.  The probable error of a mean.  _Biometrika_ , 6(1):1–25, 1908.
  * Westfall and Young [1993] Peter H Westfall and S Stanley Young.  _Resampling-based multiple testing: Examples and methods for p-value adjustment_ , volume 279.  John Wiley & Sons, 1993.