1

Training a HyperDimensional Computing Classifier using a Threshold on its
Confidence

Laura Smets1, Werner Van Leekwijck1, Ing Jyh Tsang1, Steven Latré1
1University of Antwerp - imec, IDLab - Department of Computer Science, Sint-
Pietersvliet 7, 2000 Antwerp, Belgium, E-mail<EMAIL_ADDRESS>

Keywords: Hyperdimensional Computing, Vector Symbolic Architectures

Abstract

Hyperdimensional computing (HDC) has become popular for light-weight and
energy-efficient machine learning, suitable for wearable Internet-of-Things
(IoT) devices and near-sensor or on-device processing. HDC is computationally
less complex than traditional deep learning algorithms and achieves moderate
to good classification performance. This article proposes to extend the
training procedure in HDC by taking into account not only wrongly classified
samples, but also samples that are correctly classified by the HDC model but
with low confidence. As such, a confidence threshold is introduced that can be
tuned for each dataset to achieve the best classification accuracy. The
proposed training procedure is tested on UCIHAR, CTG, ISOLET and HAND dataset
for which the performance consistently improves compared to the baseline
across a range of confidence threshold values. The extended training procedure
also results in a shift towards higher confidence values of the correctly
classified samples making the classifier not only more accurate but also more
confident about its predictions.

##  1 Introduction

Hyperdimensional computing (HDC) has gained a lot of interest in the field of
low-power, brain-inspired artificial intelligence (AI). It tries to mimic the
human brain by distributing the information across thousands of vector
elements in analogy to the large number of neurons present in our brains. HDC
is a light-weight and energy-efficient algorithm that has already been used in
several applications which can be divided in three categories according to
[7]: (i) text classification ([23]; [26]), (ii) signals such as speech
recognition ([11]), human activity recognition ([15]), handgesture recognition
([24, 21, 31]) and time series classification ([27]), and (iii) images such as
classification of medical images ([16, 28]), character recognition ([20]) and
robotics ([22]). It has been shown that HDC is suitable for wearable Internet-
of-Things (IoT) devices, near-sensor AI applications and on-device processing
due to few data requirement ([25]), robustness to noise ([14, 30, 25]), low
latency ([25]) and fast processing ([25]). This avoids the limitations of IoT
architectures in which data is sent to the cloud and consequently processed
causing high latencies, large communication energy and privacy concerns ([2]).

Although HDC has the advantage of being computationally less complex than
traditional deep learning algorithms, it is only able to achieve moderate to
good performance in classification tasks. Hence, research is ongoing to adjust
and improve HDC to boost its performance. This article aims to contribute to
this research by proposing a simple, yet effective extended training procedure
in the binary HDC framework to improve its performance on signal applications,
suitable for wearable IoT devices. The next section gives a detailed
introduction to HDC after which HDC adjustments that are already proposed in
literature are discussed in Section 3. Thereafter, the proposed extended
training procedure is introduced in Section 4. In the fifth section, an
overview of the performed experiments is given of which the results are
presented and discussed in the sixth section. Finally, the conclusions of the
article are presented.

##  2 Hyperdimensional computing

HDC is a mathematical framework using hyperdimensional (HD) vectors (i.e.,
vectors with very high dimension typically up to ten thousands, also called
hypervectors (HVs)) and simple HD arithmetic operations to represent data. The
focus of this article is on dense binary HVs (i.e., the elements are 0 or 1
with an equal probability of occurrence of both values). The analysis of data
relies on the similarity between HVs which is calculated using the normalized
Hamming distance between two binary HVs $\textbf{v}_{1}$ and
$\textbf{v}_{2}$111A list of used symbols can be found in the appendix (Table
A1).:

$s(\textbf{v}_{1},\textbf{v}_{2})=1-\frac{h(\textbf{v}_{1},\textbf{v}_{2})}{D}$
(1)

with $s$ the similarity between $\textbf{v}_{1}$ and $\textbf{v}_{2}$, $D$ the
dimensionality (e.g., $D=10,000$) and $h$ the Hamming distance between
$\textbf{v}_{1}$ and $\textbf{v}_{2}$ which counts the number of elements for
which the coordinates differ (i.e., the sum of elements of the exclusive
disjunction (XOR)):

$h(\textbf{v}_{1},\textbf{v}_{2})=\sum_{d=1}^{D}(\textbf{v}_{1}[d]\textrm{ XOR
}\textbf{v}_{2}[d]).$ (2)

The HD arithmetic operations include:
(a) bundling $\oplus$: $\mathcal{B}\times\mathcal{H}\to\mathcal{B}$:
$(\textbf{B},\textbf{v})\to\textbf{B}+\textbf{v}$ where
$\mathcal{B}=\mathbb{N}^{D}$ and $\mathcal{H}=\\{0,1\\}^{D}$ (i.e., element-
wise addition) after which the bundle B is binarized into the HV v with the
majority rule $[.]:\mathcal{B}\to\mathcal{H}$: $\textbf{B}\to\textbf{v}$
according to:

$\textbf{v}[d]=[\textbf{B}[d]]=\begin{cases}1&\text{if}\,\textbf{B}[d]>\frac{n}{2}\\\
0&\text{if}\,\textbf{B}[d]<\frac{n}{2}\\\
rand(0,1)&\text{if}\,\textbf{B}[d]=\frac{n}{2}\end{cases}$ (3)

with $n$ the number of HVs bundled in B and $rand(0,1)$ means that the
component $\textbf{v}[d]$ is randomly assigned to 0 or 1 in the presence of
ties (which can only occur when the number of bundled vectors is even);
(b) binding $\otimes$: $\mathcal{H}\times\mathcal{H}\to\mathcal{H}$:
$(\textbf{v}_{1},\textbf{v}_{2})\to\textbf{v}_{1}\textrm{ XOR }\textbf{v}_{2}$
(i.e., XOR in binary HDC); and
(c) permutation $\rho$ (i.e., cyclic shift in binary HDC).

Figure 1 gives a schematic overview of the framework of HDC in which five main
steps are distinguished: (1) mapping, (2) encoding, (3) initial prototype
construction, (4) training and (5) inference, which will be explained in more
detail in the following section.

Figure 1: Schematic overview of the HDC framework in which five main steps are
distinguished: (1) mapping, (2) encoding, (3) initial prototype construction,
(4) training and (5) inference.

(1) Mapping. The way of mapping depends on the type of data:
(a) For nominal data, each possible category is mapped to a randomly chosen
atomic HV and stored in an Item Memory (IM). These random HVs are pseudo-
orthogonal in high dimensional spaces which converges to exact orthogonality
with increasing dimensionality ([17]).
(b) In the case of ordinal or discrete data, there is a natural ordering of
levels of categories or integer values such that closer levels should be
mapped to more similar HVs than levels further apart. This is typically
achieved by a Continuous Item Memory (CIM) applying linear mapping of levels
to atomic HVs ([24, 18]). Figure 2 illustrates the similarity of values to the
lowest level (feature value = $-100$) that decreases linearly up until
orthogonality (similarity = 0.5) and the similarity of values to the feature
value equal to $-30$ that decreases linearly for smaller and larger feature
values.

Figure 2: Example of linear mapping for a feature with discrete values ranging
from -100 to 100 with steps of 10. The similarity of each feature value’s
level hypervector to the lowest level hypervector (feature value $=-100$) and
to the hypervector for the feature value of $-30$ is shown.

(c) Continuous data is quantized with a quantization step into a predefined
number of discrete levels to which linear mapping can be applied.

(2) Encoding. Input data is encoded in HVs using the atomic vectors made in
the previous step, and HD arithmetic operations. An input sample $x$ having
$n$ features is encoded as (Figure 3):
For each feature ($j=1...n$), a CIM translates the feature value to an HV
$\textbf{v}_{x[j]}$. Next, all $\textbf{v}_{x[j]}$’s are bundled together to
form the sample bundle S by initializing

$\textbf{B}_{0}=\\{0\\}^{D}$ (4)

and bundling each $\textbf{v}_{x[j]}$ one at a time:

$\textbf{B}_{j}=\textbf{B}_{j-1}\oplus\textbf{v}_{x[j]}.$ (5)

The sample bundle S is then simply:

$\textbf{S}=\textbf{B}_{n}.$ (6)

For notation purposes, this iterative bundling (Equation 4-6) will be written
in short as:

$\textbf{S}=\bigoplus_{j=1}^{n}\textbf{v}_{x[j]}$ (7)

which is binarized into the HV $\textbf{s}=[\textbf{S}]$ with the majority
rule (Equation 3).

Figure 3: Schematic overview of the encoding with a separate CIM for the
values of each feature.

Additionally, when the input data is encoded as $n$-grams, permutations can be
used for the encoding ([24]):

$\textbf{B}_{0}=\\{0\\}^{D}$
$\textbf{B}_{j}=\rho(\textbf{B}_{j-1})\oplus\textbf{s}_{j}$ (8)
$\textbf{s}_{n-gram}=[\textbf{S}_{n-gram}]=[\textbf{B}_{n}]$

where $j=1...n$, $\textbf{s}_{n-gram}$ is the encoded, binarized HV of the
$n$-gram of samples, and $\textbf{s}_{j}$ is a sample HV calculated with
Equation 7 followed by binarization (Equation 3).

(3) Initial prototype construction. Sample HVs $\textbf{s}_{i}$ belonging to
the same class $l$ are bundled to form a class bundle $\textbf{C}_{l}$
representing the considered class:

$\textbf{C}_{l}=\bigoplus_{i=1}^{m}\\{\textbf{s}_{i}|y_{i}=l\\}$ (9)

with $y_{i}$ the $i$th sample’s class. After binarization of the class bundle
$\textbf{C}_{l}$, the $l$th class prototype $\textbf{c}_{l}=[\textbf{C}_{l}]$
is obtained and stored in the Associative Memory (AM). This is repeated for
each class present in the dataset.

(4) Training. The HDC classifier predicts the class for all training samples
by calculating the similarity between the training sample’s HV
$\textbf{s}_{i}$ and each class prototype $\textbf{c}_{k}$ stored in the AM.
The predicted class $\hat{y}_{i}$ of the input sample is the class with the
highest similarity to the input sample’s HV:

$\hat{y}_{i}=\operatorname*{arg\,max}_{k}s(\textbf{s}_{i},\textbf{c}_{k})$
(10)

If the predicted class is correct (i.e., $\hat{y}_{i}=y_{i}=l$), nothing
happens. However, if the sample HV $\textbf{s}_{i}$ is wrongly classified
(i.e., $\hat{y}_{i}\neq y_{i}$), it is bundled again in the class bundle of
the correct class $\textbf{C}_{l}$ and bundled out of the class bundle of the
wrong class $\textbf{C}_{\hat{l}}$:

$\textbf{C}_{l}=\textbf{C}_{l}\oplus\textbf{s}_{i}$ (11)
$\textbf{C}_{\hat{l}}=\textbf{C}_{\hat{l}}\ominus\textbf{s}_{i}$ (12)

with $\ominus$: $\mathcal{B}\times\mathcal{H}\to\mathcal{B}$:
$(\textbf{B},\textbf{v})\to\textbf{B}-\textbf{v}$ the bundling out operation
which is element-wise subtraction. As such, the class prototypes are adjusted
to better classify wrongly classified samples.
The training procedure is performed iteratively until either a predefined
accuracy on the training set is reached or either a predefined number of
iterations is performed. After each iteration, the updated class bundles are
binarized into updated class prototypes to be used in the next iteration or
finally, in the inference phase.

(5) Inference. The $i$th test sample is encoded in a query HV $\textbf{q}_{i}$
following the same encoding procedure as for the training samples. The
predicted class is obtained similarly as during the training procedure with
Equation 10 where $\textbf{s}_{i}=\textbf{q}_{i}$, i.e., the predicted class
label of the test sample is the class with the highest similarity to the test
samples’s HV $\textbf{q}_{i}$.

##  3 Previously proposed adjustments to HDC

Already several suggestions have been made to improve the initial prototype
construction and the training procedure in the HDC framework. For instance,
[24] only add a sample vector to the class bundle if the similarity between
the sample vector and class vector is smaller than 0.9 such that a sample
vector is not added to the class bundle if the class vector is already highly
similar to the sample vector. Consequently, no redundant information is added
and the initial prototypes are assumed to be better, reducing the training
time afterwards. [12] propose to perform the training procedure with an
adaptive learning rate that depends (1) on the average error rate over the
last few training iterations (= iteration-dependent learning), (2) on the
difference in similarity between query and wrong class on the one hand and
query and right class on the other hand (= data-dependent learning) or (3) a
combination of both (= hybrid learning). During the training process explained
by [8], sample vectors are added to or subtracted from class bundles with a
weight depending on the similarity of that sample vector to the considered
class vector. As such, a sample with high similarity will be added with
smaller weight than a sample with low similarity, since the sample is already
highly represented by the class vector in case of high similarity and thus
redundant information in the class prototype is limited.

In addition, more complex methods to improve HDC have been introduced in
literature such as applying manifold learning for unsupervised non-linear
dimensionality reduction to project the original data to a smaller dimension
before applying HD encoding ([32]). [10] use a learnable projection to train
the HDC in a similar way as a binary neural network. The trained binary
weights are then transformed into learned IM and CIM. A different way of
linear mapping is proposed by [2] where a variable number of bits are flipped
between levels to emphasize the distinct levels, instead of flipping a uniform
number of bits. [4] use a confidence metric to decide whether binary HDC is
suitable for a specific sample, i.e., samples predicted with low confidence in
binary HDC will be predicted with non-binary HDC to improve the classification
performance. On the other hand, [6] map the HDC framework to an equivalent
binary neural network (BNN) which is trained to minimize the training loss to
increase the confidence in predictions. These methods in general achieve
higher accuracies, but as they are more complex, they also have a large
computational overhead.

In contrast, the focus of this article is on a simple, yet effective extension
of the binary HDC training procedure that consistently improves the baseline
performance, but without additional complexity. Its basic idea is to not only
take into account wrong predictions when updating the class bundles, but also
samples that are correctly classified but for which the class with the second
highest similarity is only slightly less similar to the sample than the
correct class. This simple extension of the HDC training procedure is
explained in the next section.

##  4 Our proposal: confidence-based training procedure

As indicated by [6], for a correctly classified sample the similarity to the
class with the second highest similarity could only be slightly lower than the
similarity to the class with the highest similarity. In such cases, the HDC
classifier is less confident about its prediction for that specific sample.
Knowing this, a confidence metric $c_{i}$ is introduced by [4] as the
difference between similarity of the sample vector $\textbf{s}_{i}$ to the
class vector with the highest similarity and the similarity to the class
vector with the second highest similarity:

$c_{i}=\max_{k}s(\textbf{s}_{i},\textbf{c}_{k})-\max_{k\neq
l}s(\textbf{s}_{i},\textbf{c}_{k}).$ (13)

The confidence metric reflects with how much certainty the HDC model
classifies a specific sample, i.e., if the confidence is low, the HDC model is
less certain about its prediction of a specific sample.

While [4] use the confidence metric only to measure certainty of
classification on the test set, our proposal is to use this confidence metric
in a simple way to extend the training procedure in HDC resulting in a more
accurate binary HDC model. In all the basic HDC training frameworks, class
bundles are only updated in case of a wrong prediction. However, in our
proposal a training sample’s HV $\textbf{s}_{i}$ that is correctly classified
(i.e., $\hat{y}_{i}=y_{i}=l$) but with a confidence smaller than a threshold
$\alpha$ (i.e., $c_{i}<\alpha$), is also added again to the class bundle of
the correct class $\textbf{C}_{l}$ and bundled out of the class bundle of the
class with second highest similarity $\textbf{C}_{\hat{l}^{\prime}}$ with
$\hat{l}^{\prime}=\operatorname*{arg\,max}_{k\neq
l}s(\textbf{s}_{i},\textbf{c}_{k})$:

$\textbf{C}_{l}=\textbf{C}_{l}\oplus\textbf{s}_{i}$ (14)
$\textbf{C}_{\hat{l}^{\prime}}=\textbf{C}_{\hat{l}^{\prime}}\ominus\textbf{s}_{i}$
(15)

Note that if $\alpha=0$, no correctly classified samples are used in the
training procedure since the confidence is non-negative. As a consequence,
this threshold setting corresponds to the baseline HDC classifier. Updating
the class bundles with samples correctly classified with low confidence could
be seen as pulling the wrong class further away from the considered sample and
pushing the right class closer to the sample. As such, the main idea of this
updating procedure has some analogies to prototype learning ([3, 13]),
prototype alignment ([9]), distance metric learning ([29, 19]), linear
discriminant analysis ([29]) and support vector machines ([29]) where the goal
is to minimize the distance between samples from the same class while
maximizing the distance between samples from different classes.

With this proposal of a simple extended training procedure, an improved
classification performance is expected since the class prototypes are not only
adjusted to better classify wrongly classified samples (Equation 11-12), but
also samples that are correctly classified with low confidence (Equation
14-15).

##  5 Experiments

Four datasets, that are publicly available and commonly or previously used in
other HDC-related research, are selected to test the proposed extended
training procedure (a more detailed summary of the datasets can be found in
Table 1):
(1) UCIHAR dataset ([1, 5]). To obtain this dataset, 30 subjects performed six
activities (walking, walking upstairs, walking downstairs, sitting, standing
and laying) during which the acceleration and velocity were recorded with the
accelerometer and gyroscope of a smartphone attached to the chest of the
subjects.
(2) Cardiotocography (CTG) dataset ([5]). This dataset consists of features
for 2,126 fetal cardiotocograms that are classified with respect to one of
three fetal states (normal, suspect or pathologic).
(3) ISOLET dataset ([5]). This dataset includes features extracted from speech
signals that were collected for 150 subjects speaking each letter of the
alphabet twice.
(4) Hand gesture (HAND) dataset ([24]). EMG signals with four channels of five
subjects are included in this dataset. During the recording, the subjects
performed four hand gestures (closed hand, open hand, 2-finger pinch and point
index) ten times for three seconds each. Between each hand gesture
contraction, a period of three seconds in the rest position is included which
serves as the fifth class.

To quantize the feature / signal values of all datasets, a step size is chosen
such that they are quantized into 21 quantization levels. As such, a step size
of 0.1 is chosen for UCIHAR because values are between -1 and 1, a step size
of 5 for CTG’s values ranging from 0 to 100, a step size of 10 for ISOLET
since the values range from -100 to 100 and a step size of 1 for HAND because
the signals can take amplitudes from 0 to 20. The samples of UCIHAR, CTG and
ISOLET are encoded following Equation 7 followed by binarization (Equation 3),
whereas $4-grams$ (Equation 8) are created for the encoding of HAND samples
that are also encoded according to Equation 7 with binarization (Equation 3)
where the four channels are treated as four features.

Table 1: Summary of the four datasets used to test the proposed extended training procedure. The table includes the number of training samples, the number of test samples, the number of classes, the number of features or channels, the range of values of the features, the step size that is used to quantize the feature values and the number of vectors (i.e., quantization levels) that are stored in the CIM. | UCIHAR | CTG | ISOLET | HAND
---|---|---|---|---
# training samples | 7,352 | 1,701 | 6,238 | 526,396
# test samples | 2,947 | 425 | 1,559 | 131,608
# classes | 6 | 3 | 26 | 5
# features / channels | 561 | 21 | 617 | 4
range of feature values | [-1,1] | [0,100] | [-100,100] | [0,20]
quantization step size | 0.1 | 5 | 10 | 1
# vectors in CIM | 21 | 21 | 21 | 21

The training procedure is performed iteratively for a maximum of 2500
iterations while saving the classifier with the best accuracy. After every 100
iterations, it is evaluated whether this best training accuracy exceeds 99%.
If this is the case, the training procedure is terminated and the classifier
with the best accuracy is used in the inference phase. The HDC classifier is
performed for 50 independent runs for UCIHAR, CTG and ISOLET, and for 10
independent runs for each subject of HAND (thus, also 50 independent runs in
total) since it starts from random vectors to form the atomic vectors.

For each dataset, the distribution of confidence values of all correctly
classified training samples after initial prototype construction is
investigated to decide the range of confidence thresholds to be tested (Figure
4). This figure illustrates that the confidence values are rather low for all
datasets (i.e., $<$ 8%). Hence, the choice for the thresholds to be tested are
$\alpha=\\{0.00;$ $0.25;$ $0.50;$ $0.75;$ $1.00;$ $1.25;$ $1.50\\}$ for
UCIHAR, $\alpha=\\{0.00;$ $1.00;$ $2.00;$ $3.00;$ $4.00;$ $5.00;$ $6.00\\}$
for CTG, $\alpha=\\{0.00;$ $0.25;$ $0.50;$ $0.75;$ $1.00;$ $1.25;$ $1.50\\}$
for ISOLET and $\alpha=\\{0.00;$ $1.00;$ $2.00;$ $3.00;$ $4.00;$ $5.00\\}$ for
HAND, since these values include most correctly classified samples for the
considered datasets. The performance of the HDC classifier for each confidence
threshold setting is documented as the classification accuracy and error rate
on the test set averaged over all performed independent runs.

(a) UCIHAR.

(b) CTG.

(c) ISOLET.

(d) HAND.

Figure 4: Distribution of the confidence values of all correctly classified
training samples of (a) UCIHAR, (b) CTG, (c) ISOLET and (d) HAND after initial
prototype construction.

##  6 Results and discussion

The accuracies on the train and test set and the error rate on the test set
averaged over all independent runs for each of the chosen confidence threshold
settings and each of the four datasets are given in Table 2. The introduction
of the confidence metric in the training procedure improves the baseline
classification accuracy ($\alpha=0.00$) for each dataset for all tested
confidence thresholds. The mean test accuracy for all datasets increases with
increasing $\alpha$ up to the point where maximal performance is reached after
which the mean test accuracy decreases for larger $\alpha$.
(1) UCIHAR. The obtained baseline accuracy of 92.75% is improved by 1.58% with
$\alpha=0.75$ reaching an accuracy of 94.33%. (A 21.79% relative decrease in
error rate from 7.25% for the baseline to 5.67%.)
(2) CTG. The obtained baseline accuracy of 73.65% is improved by 13.24% with
$\alpha=4.00$ reaching an accuracy of 86.89%. (A 50.25% relative decrease in
error rate from 26.35% for the baseline to 13.11%.)
(3) ISOLET. The obtained baseline accuracy of 92.12% is improved by 2.18% with
$\alpha=1.00$ reaching an accuracy of 94.30%. (A 27.66% relative decrease in
error rate from 7.88% for the baseline to 5.70%.)
(4) HAND. The obtained baseline accuracy of 95.38% is improved by 0.93% with
$\alpha=5.00$ reaching an accuracy of 96.31%. (A 20.13% relative decrease in
error rate from 4.62% for the baseline to 3.69%.)

Table 2: Averaged accuracy (%) on the train and test set and averaged error
rate (%) on test set over 50 independent runs (for UCIHAR, CTG and ISOLET) and
10 independent runs (for each subject of HAND, thus 50 in total) of the four
datasets for the tested settings of the confidence threshold $\alpha$. Data
are mean ($\pm$ standard deviation), in bold are the test accuracies (and
error rates) that are higher (and lower) than the baseline ($\alpha=0.00$) and
underlined is the best test accuracy and error rate.

(a) UCIHAR. $\alpha$ | Train accuracy | Test accuracy | Test error rate
---|---|---|---
0.00 | 99.43 ($\pm$ 0.24) | 92.75 ($\pm$ 0.51) | 7.25
0.25 | 99.11 ($\pm$ 0.20) |  93.94 ($\pm$ 0.50) | 6.06
0.50 | 98.96 ($\pm$ 0.51) |  94.03 ($\pm$ 0.53) | 5.97
0.75 | 98.62 ($\pm$ 0.61) |  94.33 ($\pm$ 0.49) | 5.67
1.00 | 98.32 ($\pm$ 0.99) |  94.25 ($\pm$ 0.70) | 5.75
1.25 | 97.13 ($\pm$ 1.15) |  94.16 ($\pm$ 0.62) | 5.84
1.50 | 95.99 ($\pm$ 1.19) |  93.45 ($\pm$ 0.90) | 6.55

(b) CTG. $\alpha$ | Train accuracy | Test accuracy | Test error rate
---|---|---|---
0.00 | 99.57 ($\pm$ 0.21) | 73.65 ($\pm$ 2.48) | 26.35
1.00 | 98.91 ($\pm$ 0.17) |  77.52 ($\pm$ 2.53) | 22.48
2.00 | 98.05 ($\pm$ 0.13) |  80.44 ($\pm$ 1.79) | 19.56
3.00 | 96.39 ($\pm$ 0.17) |  85.14 ($\pm$ 1.28) | 14.86
4.00 | 94.47 ($\pm$ 0.29) |  86.89 ($\pm$ 1.04) | 13.11
5.00 | 93.19 ($\pm$ 0.27) |  86.19 ($\pm$ 0.94) | 13.81
6.00 | 92.47 ($\pm$ 0.14) |  84.97 ($\pm$ 1.01) | 15.03

(c) ISOLET. $\alpha$ | Train accuracy | Test accuracy | Test error rate
---|---|---|---
0.00 | 100.00 ($\pm$ 0.00) | 92.12 ($\pm$ 0.47) | 7.88
0.25 | 100.00 ($\pm$ 0.00) |  93.19 ($\pm$ 0.42) | 6.81
0.50 | 99.99 ($\pm$ 0.03) |  93.80 ($\pm$ 0.45) | 6.20
0.75 | 99.83 ($\pm$ 0.27) |  94.13 ($\pm$ 0.55) | 5.87
1.00 | 99.81 ($\pm$ 0.22) |  94.30 ($\pm$ 0.73) | 5.70
1.25 | 99.70 ($\pm$ 0.28) |  93.98 ($\pm$ 0.75) | 6.02
1.50 | 99.47 ($\pm$ 0.25) |  93.68 ($\pm$ 0.76) | 6.32

(d) HAND. $\alpha$ | Train accuracy | Test accuracy | Test error rate
---|---|---|---
0.00 | 97.05 ($\pm$ 1.84) | 95.38 ($\pm$ 1.82) | 4.62
1.00 | 96.94 ($\pm$ 1.92) |  95.77 ($\pm$ 1.83) | 4.23
2.00 | 96.73 ($\pm$ 1.85) |  95.60 ($\pm$ 2.25) | 4.40
3.00 | 96.45 ($\pm$ 2.01) |  95.81 ($\pm$ 2.03) | 4.19
4.00 | 96.07 ($\pm$ 2.20) |  96.11 ($\pm$ 1.65) | 3.89
5.00 | 95.83 ($\pm$ 2.26) |  96.31 ($\pm$ 1.34) | 3.69

The effect of introducing the confidence threshold in the training procedure
of HDC is visualized in Figure 5. This figure gives for each of the four
datasets (one column for each) the distribution of the confidence values of
all correctly classified training samples after training with $\alpha=0.00$
(baseline, top row) and after training with the setting of $\alpha$ resulting
in the best performance for the considered dataset (bottom row). When
comparing with Figure 4, already larger confidence values are seen after
training with $\alpha=0.00$ for all four datasets. However, there is an even
more clear shift in the distribution towards higher confidence values for the
best settings of $\alpha$ for each dataset illustrating nicely the effect of
the proposed simple extended training procedure in the HDC framework.

The abovementioned results show the impact of the parameter $\alpha$ on the
test set. To determine the optimal $\alpha$ for a particular dataset, the
standard procedure of using a validation set should be used. Table A2 in the
appendix illustrates the procedure for the ISOLET dataset when performing
10-fold cross validation, resulting in an optimal $\alpha$ of 1.25.

The obtained highest accuracy of 94.30% for ISOLET is an improvement compared
to the accuracy of 92.4% obtained by [12] using an adaptive learning rate
during training procedure and by [10] who project original HDC to learnable
HDC that is trained similarly as a BNN. Also for the CTG dataset, the highest
accuracy of 86.89% is higher compared to the accuracy of 82% obtained by [2]
who flip a variable number of bits between levels when applying linear
mapping.

While our method consistently improves the accuracy compared to baseline, some
other studies in literature report slightly better results on the considered
datasets but with often more complex methods. For example, [12] report an
accuracy of 96% on the UCIHAR dataset with their adaptive learning rate, [8]
use weighted bundling in and out of class bundles and obtain an accuracy of
96.5% for UCIHAR and 94.6% for ISOLET, learnable HDC of [10] results in an
accuracy of 95.54% for UCIHAR, [32] obtain an accuracy of 98% for UCIHAR and
95% for ISOLET applying manifold learning and [6] map HDC into a BNN achieving
95.23% and 94.89% accuracy for UCIHAR and ISOLET, respectively.

As future work, the proposed training procedure could possibly be extended
with an adaptive confidence threshold. Namely, the distribution of confidence
values shift towards higher confidence while correcting for low-confident
correctly classified training samples (Figure 5) such that the threshold could
increase along with this shift in distribution. Moreover, it might be
interesting to combine the proposed extended training procedure with
previously proposed adjustments to the training procedure ([12, 8]) and
investigate whether this would further improve the HDC performance.

(a) UCIHAR
$\alpha=0.00$.

(b) CTG
$\alpha=0.00$.

(c) ISOLET
$\alpha=0.00$.

(d) HAND
$\alpha=0.00$.

(e) UCIHAR
$\alpha=0.75$.

(f) CTG
$\alpha=4.00$.

(g) ISOLET
$\alpha=1.00$.

(h) HAND
$\alpha=5.00$.

Figure 5: Distribution of the confidence values of all correctly classified
training samples of UCIHAR (first column), CTG (second column), ISOLET (third
column) and HAND (last column) after training with $\alpha=0.00$ (first row)
and with $\alpha$ yielding the best accuracy for the considered dataset
(second row).

## Conclusion

A simple, yet effective extension to the training procedure in binary HDC is
introduced which takes into account not only wrongly classified samples, but
also samples that are correctly classified by HDC but with a low confidence. A
threshold on the confidence value is introduced and tested on four datasets
for which the performance consistently improves compared to the baseline
across a range of confidence threshold values. The extended training procedure
also results in a shift towards higher confidence values of the correctly
classified samples making the classifier not only more accurate but also more
confident about its predictions.

## Appendix

##  A.1 Notation

A summary of notation can be found in Table A1.

Table A1: List of symbols. (HD = hyperdimensional, CIM = Continuous Item Memory) Symbol | Definition | Symbol | Definition
---|---|---|---
$x$ | vector in input space |  |
$m$ | number of samples | $i$ | $1...m$
$n$ | number of features | $j$ | $1...n$
$K$ | number of classes | $k$ | $1...K$
$D$ | HD vector dimension | $d$ | $1...D$
$s$ | similarity | $y_{i}$ | true class of $i$th sample
$h$ | Hamming distance | $\hat{y}_{i}$ | predicted class of $i$th sample
$c$ | confidence value | $l$ | true class label
$\alpha$ | confidence threshold | $\hat{l}$ | predicted class label
|  | $\hat{l}^{\prime}$ | class label with second highest similarity
v | vector in HD space $\mathcal{H}$ | B | bundle in HD space $\mathcal{B}$
s | sample vector | S | sample bundle
c | class vector/prototype | C | class bundle
q | query vector |  |
$\mathcal{H}$ | vector HD space, $\\{0,1\\}^{D}$ | $\mathcal{B}$ | bundle HD space, $\mathbb{N}^{D}$
$CIM_{j}$ | CIM of $j$th feature | $\oplus$ | bundling operator
$\otimes$ | binding operator | $\ominus$ | bundling out operator
$\rho$ | permutation operator | $[.]$ | majority rule

##  A.2 Cross-Validation

Table A2 contains the results of 10-fold cross validation (CV) on the training
set of ISOLET for the different settings of $\alpha$. The table includes the
training accuracy, validation accuracy and validation error rate, averaged
over the ten folds of 10-fold CV. This shows that the optimal $\alpha$ value
with 10-fold CV is $\alpha=1.25$, resulting in a test accuracy of 93.98%
(Table 2(c)) which is very near to the best test accuracy achieved in Table
2(c), i.e., 94.30% for $\alpha=1.00$.

Table A2: Averaged accuracy (%) on the train and validation folds and averaged error rate (%) on the validation folds of 10-fold cross validation for ISOLET for the tested settings of the confidence threshold $\alpha$. Data are mean ($\pm$ standard deviation), in bold are the validation accuracies (and error rates) that are higher (and lower) than the baseline ($\alpha=0.00$) and underlined is the best validation accuracy and error rate. $\alpha$ | Train accuracy | Validation accuracy | Validation error rate
---|---|---|---
0.00 | 100.00 ($\pm$ 0.00) | 90.56 ($\pm$ 1.97) | 9.44
0.25 | 100.00 ($\pm$ 0.00) |  92.43 ($\pm$ 1.96) | 7.57
0.50 | 99.99 ($\pm$ 0.01) |  93.14 ($\pm$ 1.85) | 6.86
0.75 | 99.92 ($\pm$ 0.13) |  93.64 ($\pm$ 1.30) | 6.36
1.00 | 99.62 ($\pm$ 0.29) |  93.41 ($\pm$ 2.06) | 6.59
1.25 | 99.78 ($\pm$ 0.21) |  93.83 ($\pm$ 1.93) | 6.17
1.50 | 99.61 ($\pm$ 0.24) |  93.68 ($\pm$ 1.55) | 6.32

## References

  * [1] Davide Anguita et al.  “A Public Domain Dataset for Human Activity Recognition Using Smartphones”  In _21th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2013_ , 2013
  * [2] Toygun Basaklar et al.  “Hypervector Design for Efficient Hyperdimensional Computing on Edge Devices”  In _TinyML Research Symposium_ , 2021  URL: http://arxiv.org/abs/2103.06709
  * [3] Fu Chang, Chin-Chin Lin and Chi-Jen Lu  “Adaptive Prototype Learning Algorithms: Theoretical and Experimental Studies Chin-Chin Lin”  In _Journal of Machine Learning Research_ 7, 2006, pp. 2125–2148
  * [4] Yu Chuan Chuang, Cheng Yang Chang and An Yeu Andy Wu  “Dynamic Hyperdimensional Computing for Improving Accuracy-Energy Efficiency Trade-Offs”  In _IEEE Workshop on Signal Processing Systems, SiPS: Design and Implementation_ 2020-October Institute of ElectricalElectronics Engineers Inc., 2020  DOI: 10.1109/SiPS50750.2020.9195216
  * [5] Dheeru Dua and Casey Graff  “UCI Machine Learning Repository”  University of California, Irvine, School of InformationComputer Sciences, 2019  URL: http://archive.ics.uci.edu/ml
  * [6] Shijin Duan, Yejia Liu, Shaolei Ren and Xiaolin Xu  “LeHDC: Learning-Based Hyperdimensional Computing Classifier”  In _Design Automation Conference_ , 2022  URL: http://arxiv.org/abs/2203.09680
  * [7] Lulu Ge and Keshab K Parhi  “Classification using Hyperdimensional Computing: A Review”  In _IEEE Circuits and Systems Magazine_ 20, 2020, pp. 30–47
  * [8] Alejandro Hernández-Cano, Namiko Matsumoto, Eric Ping and Mohsen Imani  “OnlineHD: Robust, Efficient, and Single-Pass Online Learning Using Hyperdimensional System”  In _Design, Automation & Test in Europe Conference & Exhibition (DATE)_, 2021  URL: https://gitlab.com/biaslab/onlinehd
  * [9] Michael Hersche et al.  “Constrained Few-shot Class-incremental Learning”  In _Conference on Computer Vision and Pattern Recognition_ , 2022  URL: https://github.com/
  * [10] Yu-Ren Hsiao, Yu-Chuan Chuang, Cheng-Yang Chang and An-Yeu Wu  “Hyperdimensional computing with learnable projection for user adaptation framework”  In _IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI)_ , 2021, pp. 436–447  DOI: 10.1007/978-3-030-79150-6˙62
  * [11] Mohsen Imani, Deqian Kong, Abbas Rahimi and Tajana Rosing  “VoiceHD: Hyperdimensional Computing for Efficient Speech Recognition”  In _IEEE International Conference on Rebooting Computing (ICRC)_ , 2017, pp. 1–8
  * [12] Mohsen Imani et al.  “AdaptHD: Adaptive Efficient Training for Brain-Inspired Hyperdimensional Computing”  In _IEEE Biomedical Circuits and Systems Conference (BioCAS)_ , 2019, pp. 1–4
  * [13] Zhong Ji et al.  “Zero-shot classification with unseen prototype learning”  In _Neural Computing and Applications_ Springer ScienceBusiness Media Deutschland GmbH, 2021  DOI: 10.1007/s00521-021-05746-9
  * [14] Pentti Kanerva  “Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors”  In _Cognitive Computation_ 1, 2009, pp. 139–159  DOI: 10.1007/s12559-009-9009-8
  * [15] Yeseong Kim, Mohsen Imani and Tajana S. Rosing  “Efficient human activity recognition using hyperdimensional computing”  In _Proceedings of the 8th International Conference on the Internet of Things_ Association for Computing Machinery, 2018  DOI: 10.1145/3277593.3277617
  * [16] Denis Kleyko, Sumeer Khan, Evgeny Osipov and Suet Peng Yong  “Modality classification of medical images with distributed representations based on cellular automata reservoir computing”  In _Proceedings - International Symposium on Biomedical Imaging_ IEEE Computer Society, 2017, pp. 1053–1056  DOI: 10.1109/ISBI.2017.7950697
  * [17] Denis Kleyko, Dmitri A. Rachkovskij, Evgeny Osipov and Abbas Rahimi  “A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations”  In _ACM Computing Surveys_ , 2022  URL: http://arxiv.org/abs/2111.06077
  * [18] Denis Kleyko et al.  “Classification and Recall With Binary Hyperdimensional Computing: Tradeoffs in Choice of Density and Mapping Characteristics”  In _IEEE Transactions on Neural Networks and Learning Systems_ 29 Institute of ElectricalElectronics Engineers Inc., 2018, pp. 5880–5898  DOI: 10.1109/TNNLS.2018.2814400
  * [19] Brian Kulis  “Metric learning: A survey”  In _Foundations and Trends in Machine Learning_ 5, 2012, pp. 287–364  DOI: 10.1561/2200000019
  * [20] Alec Xavier Manabat, Celine Rose Marcelo, Alfonso Louis Quinquito and Anastacia Alvarez  “Performance Analysis of Hyperdimensional Computing for Character Recognition”  In _International Symposium on Multimedia and Communication Technology (ISMAC)_ , 2019
  * [21] Ali Moin et al.  “A wearable biosensing system with in-sensor adaptive machine learning for hand gesture recognition”  In _Nature Electronics_ 4 Nature Research, 2021, pp. 54–63  DOI: 10.1038/s41928-020-00510-8
  * [22] Peer Neubert, Stefan Schubert and Peter Protzel  “An Introduction to Hyperdimensional Computing for Robotics”  In _KI - Kunstliche Intelligenz_ 33 Springer ScienceBusiness Media Deutschland GmbH, 2019, pp. 319–330  DOI: 10.1007/s13218-019-00623-z
  * [23] D.A. Rachkovskij  “Linear Classifiers based on Binary Distributed Representations”  In _Information Theories and Applications_ 14, 2007, pp. 270–274
  * [24] Abbas Rahimi et al.  “Hyperdimensional Biosignal Processing: A Case Study for EMG-based Hand Gesture Recognition”  In _IEEE International Conference of Rebooting Computing (ICRC)_ , 2016
  * [25] Abbas Rahimi, Pentti Kanerva, Luca Benini and Jan M. Rabaey  “Efficient Biosignal Processing Using Hyperdimensional Computing: Network Templates for Combined Learning and Classification of ExG Signals”  In _Proceedings of the IEEE_ 107 Institute of ElectricalElectronics Engineers Inc., 2019, pp. 123–143  DOI: 10.1109/JPROC.2018.2871163
  * [26] Abbas Rahimi, Pentti Kanerva and Jan M. Rabaey  “A Robust and Energy-Efficient Classifier Using Brain-Inspired Hyperdimensional Computing”  In _Proceedings of the International Symposium on Low Power Electronics and Design_ Institute of ElectricalElectronics Engineers Inc., 2016, pp. 64–69  DOI: 10.1145/2934583.2934624
  * [27] Kenny Schlegel, Peer Neubert and Peter Protzel  “HDC-MiniROCKET: Explicit Time Encoding in Time Series Classification with Hyperdimensional Computing”  In _International Joint Conference on Neural Network_ , 2022  URL: http://arxiv.org/abs/2202.08055
  * [28] Neftali Watkinson et al.  “Detecting COVID-19 Related Pneumonia on CT Scans Using Hyperdimensional Computing”  In _Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS_ Institute of ElectricalElectronics Engineers Inc., 2021, pp. 3970–3973  DOI: 10.1109/EMBC46164.2021.9630898
  * [29] Kilian Q Weinberger and Lawrence K Saul  “Distance Metric Learning for Large Margin Nearest Neighbor Classification”  In _Journal of Machine Learning Research_ 10, 2009, pp. 207–244
  * [30] Dominic Widdows and Trevor Cohen  “Reasoning with vectors: A continuous model for fast robust inference”  In _Logic Journal of the IGPL_ 23 Oxford University Press, 2015, pp. 141–173  DOI: 10.1093/jigpal/jzu028
  * [31] Andy Zhou, Rikky Muller and Jan Rabaey  “Memory-Efficient, Limb Position-Aware Hand Gesture Recognition using Hyperdimensional Computing”  In _TinyML Research Symposium_ , 2021  URL: http://arxiv.org/abs/2103.05267
  * [32] Zhuowen Zou, Yeseong Kim, M Hassan Najafi and Mohsen Imani  “ManiHD: Efficient Hyper-Dimensional Learning Using Manifold Trainable Encoder”  In _Design, Automation Test in Europe Conference Exhibition (DATE)_ , 2021, pp. 850–855


Gopalswamy et al. Cycle-25 halo CMEs

17 2024 10.1017/xxxxx 388 2024 Solar and Stellar Coronal Mass Ejections

Proceedings of the IAU Symposium

# Implications of the abundance of halo coronal mass ejections for the
strength of solar cycle 25

Nat Gopalswamy1 Grzegorz Michałek2 Seiji Yashiro3,1 Pertti Mäkelä3,1 Sachiko
Akiyama3,1 and Hong Xie3,1<EMAIL_ADDRESS>1NASA Goddard Space Flight
Center, Greenbelt, MD, USA
email:  2Jagiellonian University, Kraków, Poland  3The Catholic University of
America, Washington, DC, USA

###### Abstract

We assess the relative strength of solar cycle (SC) 25 with respect to SCs 23
and 24 based on the abundance of halo coronal mass ejections (CMEs). We make
use of the halo CME database
(https://cdaw.gsfc.nasa.gov/CME_list/halo/halo.html) to compare the halo CME
abundance during the first four years in each of SCs 23–25. The main result is
that in several aspects such as the abundance, occurrence rate, source
locations, and halo heights, halo CMEs are similar between SCs 24 and 25 but
different from SC 23. This result follows from the fact that weaker cycles
have low heliospheric total pressure, whose backreaction on CMEs allows them
to expand more and hence enhancing the chance of becoming a halo. The solar
cycle variation of halo CME properties is consistent with the precursor-based
cycle prediction methods that indicate SC 25 is similar to or only slightly
stronger than SC 24.

###### keywords:

coronal mass ejections, solar cycle, heliospheric state

††editors: N. Gopalswamy, O. Malandraki, A. Vidotto & W. Manchester, eds.

##  1 Introduction

The occurrence rate of halo coronal mass ejections (CMEs) is correlated with
the sunspot number (SSN), so it is likely that this correlation has a bearing
on the cycle strength. During the maximum phase of solar cycle (SC) 24 it was
recognized that the halo CME abundance had a discordant behavior with SSN: the
halo CME abundance normalized to SSN was higher in the weaker cycle 24 than in
SC 23 (Gopalswamy et al., 2015; Dagnew et al., 2022). This was explained to be
a consequence of the anomalous expansion of CMEs due to the weak heliospheric
state (Gopalswamy et al., 2014). Comparing occurrence rates of halo CMEs
during the rise phases of SCs 23–25, it was recently reported that the halo
CME abundance is higher in SC 25 as well. Furthermore, the halo CME abundance
indicates that the strength of SC 25 is intermediate between SCs 23 and 24,
but closer to SC 24 (Gopalswamy et al., 2023). Thus, the halo CME abundance
provides clues to the relative strength of solar cycles. Given the widely
distributed predictions of the strength of SC 25, the halo CME abundance is
able to discriminate among the predictions that vary from less than half to
almost twice the strength of SC 24 (Nandy, 2021). In this paper, we extend the
results of Gopalswamy et al. (2023) to the maximum phase thereby including a
bigger sample of CMEs over the first four years of cycles 23–25 to assess
their relative strengths.

##  2 Observations

We use the halo CME catalog
(https://cdaw.gsfc.nasa.gov/CME_list/halo/halo.html, Gopalswamy et al., 2010)
based on the observations from the Large Angle and Spectrometric Coronagraph
(LASCO; Brueckner et al., 1995) on board the Solar and Heliospheric
Observatory (SOHO). The catalog includes the solar sources of the halo CMEs
(heliographic coordinates). We use the v2 SSN from the Sunspot Index and Long-
term Solar Observations (SILSO, World Data Center,
https://sidc.be/SILSO/datafiles). Figure 1 shows the semiannual number of halo
CMEs as a function of time from 1996 to the end of 2023. We see that the
number of halos during the first 49 months of SC 23 is only 100 compared to
137 in SC 24 and 144 in SC 25. Thus, the halo CME abundance did not decrease
in SCs 24 and 25 although the SSN decreased significantly.

Figure 1: Semiannual variation of the number of halo CMEs. During the first 49
months (indicated by arrows), 100, 137, and 144 halos were observed in SCs 23,
24 and 25, respectively. SC 23: 1996 May 10 to 2000 June 9; SC 24: 2008
December 1 to 2012 December 31; SC 25: 2019 December 1 to 2023 December 31.
The error bars are based on the data gaps in the LASCO data. SSN (Source: WDC-
SILSO, Royal Observatory of Belgium, Brussels) is shown for reference.

##  3 Analysis and Results

###  3.1 Halo CME abundance

Table 1: Comparison of the number of halos in SCs 23–25

| SC 23 | SC 24 | SC 25
---|---|---|---
Total # Halos | 100 (109) | 137 | 144
Change |  | +26% | +32%
Avg SSN | 82.7 | 47.8 | 60.5
Change |  | -42% | -17%
#Halo/$<$SSN$>$ | 1.32 | 2.87 | 2.38
Change |  | +117% | +80%

Table 1 compares the number of halos during the first 49 months of each cycle
and the relative changes with respect to cycle 23 as absolute values and
relative to the average SSN. In cycle 23, there was a 4-month data gap. If we
assume that halo CMEs occurred during the data gaps at the same rate as the
average rate over the whole interval, the estimated number is 109 CMEs instead
of 100. We use the corrected value (109) in comparisons. In terms of absolute
numbers, we see that the number of halos increased by 26% and 32% in SCs 24
and 25, respectively. On the other hand, the average SSN declined by 42% in SC
24 and 27% in SC 25. When we normalize the halo number to SSN, the relative
numbers are 1.32, 2.87, and 2.38 per SSN. Clearly, the normalized halo CME
abundance approximately doubled in SCs 24 and 25. Furthermore, the SC 24
abundance is the highest. The SC 25 abundance is much closer to that of SC 24.

Figure 2: (left) Scatter plot between SSN and the occurrence rate of halo CMEs
per Carrington rotation (CR) period. The number of data points during the
first 49 months is not the same in the three cycles due to data gaps. The
beginning and end dates of the compared intervals, the correlation
coefficients and equations of the regression lines are shown on the plot.
(right) Schematic showing the size CMEs in each cycle for a given CME speed.

###  3.2 Correlation between SSN and halo CME occurrence rate

Figure 2 shows the scatter plot between SSN and the number of halo CMEs in
each Carrington rotation (CR) period. The SSN used in the plot is an average
over CR periods. We see that the halo CME rate is positively correlated with
SSN in each of the three SCs. All correlations are significant because the
Pearson’s critical coefficients for 47, 55, and 54 samples are 0.372, 0.363,
and 0.360, respectively for a p value of 1% (the probability that the obtained
correlation is by chance). The regression lines have similar slopes in SCs 24
and 25 but significantly higher than that in SC 23. These plots indicate that
there are more halo CMEs for a given SSN in SCs 24 and 25, confirming the
result shown in Table 1 for the whole interval. The effect of the weak
heliosphere on the CME size is shown schematically in Fig. 2. For a given
speed, CMEs expand more in SCs 24 and 25 than in SC 23 resulting in the higher
abundance of halo CMEs in SCs 24 and 25.

Figure 3: Solar source locations (heliographic coordinates) of halo CMEs
during the first 49 months in SCs 23–25, color coded in red, blue, and green,
respectively. Only frontside halos are included.

###  3.3 Distribution of solar source locations

The distribution of halo CME sources on the solar disk is shown in Fig. 3. The
source locations are generally concentrated in the latitude range 15∘–30∘,
which corresponds to the active region belt. Most halo CMEs originate from
active regions, where large amount of free energy can be stored and released.
Most of the source locations of SC 23 CMEs are concentrated near the disk
center (within the longitude range of $\pm$45∘). On the other hand, the halos
from SCs 24 and 25 originate from a broader range of longitudes. In SC 23,
only one halo source is located beyond a central meridian distance (CMD) of
45∘ in the eastern hemisphere and 6 in the western hemisphere out of the 60
frontside halos (or $\sim$12%). On the other hand, there are 25 (out of 68)
and 24 (out of 70) halos beyond a CMD of 45∘ in SCs 24 and 25, respectively
amounting to 37% and 34% of the frontside halos in these cycles. Clearly, the
large-CMD halos in SCs 24 and 25 outnumber those in SC 23 by a factor of
$\sim$3\. Once again, we see that the longitudinal distribution of halo CME
sources in SCs 24 and 25 are close to each other, but much wider than that in
SC 23. Halos originating beyond a CMD of 60∘ have a similar pattern: 4 out of
60 (or 7%) in SC 23 compared to 17 out of 68 (or 25%) in SC 24 and 15 out of
70 (or 21%) in SC 25. We also note that the large-CMD halo sources in SC 25
are intermediate between SCs 23 and 24 but closer to SC 24.

###  3.4 Halo heights of limb halo CMEs

Gopalswamy et al. (2020) introduced a new parameter known as ”halo height”
defined as the height of the leading edge of a limb CME when it appears as a
halo for the first time in the LASCO FOV. In such limb halos, a CME
originating close to one of the solar limbs ends up with features appearing on
the opposite limb. Figure 4 shows the distributions of halo heights in the
three cycles. The halo heights in SCs 24 and 25 are smaller than that in SC
23. Furthermore, the average speeds of the limb halos are lower in SCs 24 and
25. In other words, limb halos formed earlier in SCs 24 and 25 and at lower
speeds. The limb halo sample size is small, so the results are not very firm.
However, the trend is similar to what was seen between SCs 23 and 24 when the
full-cycle data were used (Gopalswamy et al., 2020).

Figure 4: (top) Halo heights of limb CMEs in SCs 23–25 in panels a–c. The
average of the distributions and their standard deviations are shown on the
plots. The number of limb halos is higher than in Fig. 3 because we have
included cases in which the sources are behind the limb but within 30∘.
(bottom) Halo heights when the LASCO image cadence in SCs 24 and 25 reduced to
match that in SC 23. Panels a–c display the average speeds of limb halos in
three cycles. The speeds are obtained from
https://cdaw.gsfc.nasa.gov/CME_list/halo/halo.html.

##  4 Discussion and Summary

We investigated the properties of halo CMEs in SCs 23–25 that occurred during
the first 49 months of each cycle, focusing on the abundance, occurrence rate,
source locations, and halo heights. The results are consistent with the
expected backreaction of the weaker heliospheric states in SCs 24 and 25 than
that in SC 23. In all the aspects studied, we find that the halo CME
properties are similar between SCs 24 and 25 and different from SC 23.
Furthermore, the halo CME properties in SC 25 are intermediate between those
in SCs 23 and 24, but closer to SC 24. These results strongly support the
predictions of the strength of SC 25 based on precursor methods: the strength
of SC 25 is similar to or slightly greater than that of SC 24 but much lower
than that of SC 23. The trend that the space weather is milder in SC 24 than
in SC 23 continues to SC 25 as well as indicated by the relatively smaller
number of intense geomagnetic storms and high-energy particle events. The
state of the heliosphere assessed from in-situ solar wind observations confirm
the weaker state of the heliosphere in SC 25. These aspects of the three
cycles will be reported elsewhere. The main results of this paper can be
summarized as follows:

  1. 1.

The number of halo CMEs in SCs 23–24 is 109, 137, and 144 respectively during
the first 49 months in each cycle. When normalized to SSN, the abundances
become 1.32, 2.87, and 2.38 per SSN, respectively in SCs 23, 24, and 25.

  2. 2.

The number of halos binned by Carrington rotation periods are highly
correlated with the corresponding CR-averaged SSN. However, the slopes of the
regression lines are steeper in SCs 24 and 25 than in SC 23, reflecting the
higher abundance in the later cycles.

  3. 3.

The longitudinal distribution of halo CME sources is much wider in SCs 24 and
25 than in SC 23, e.g., the fraction of halos originating beyond a CMD of 60∘
is only 7% in SC 23 compared to 25% in SC 24 and 21% in SC 25.

  4. 4.

The halo heights in SCs 24 and 25 are smaller than that in SC 23 but the
average speeds of the limb halos are smaller than that in SC 23. In other
words, limb halos form closer to the Sun at a smaller CME speed.

We thank SOHO, STEREO, and SDO teams for making their data available online.
The sunspot number is from WDC–SILSO, the Royal Observatory of Belgium,
Brussels. SOHO is a joint project of ESA and NASA. STEREO is a mission in
NASA’s Solar Terrestrial Probes program. Work supported by NASA’s LWS and
STEREO programs. H.X. was partially supported by NSF grant AGS-2228967. P.M.
was partially supported by NSF grant AGS-2043131.

## References

  * Brueckner et al. (1995) Brueckner, G. E., Howard, R. A., Koomen, M. J., et al. 1995, Sol. Phys., 162, 357, doi: 10.1007/BF00733434
  * Dagnew et al. (2022) Dagnew, F. K., Gopalswamy, N., Tessema, S. B., Akiyama, S., & Yashiro, S. 2022, ApJ, 936, 122, doi: 10.3847/1538-4357/ac8744
  * Gopalswamy et al. (2020) Gopalswamy, N., Akiyama, S., & Yashiro, S. 2020, ApJ, 897, L1, doi: 10.3847/2041-8213/ab9b7b
  * Gopalswamy et al. (2014) Gopalswamy, N., Akiyama, S., Yashiro, S., et al. 2014, Geophys. Res. Lett., 41, 2673, doi: 10.1002/2014GL059858
  * Gopalswamy et al. (2023) Gopalswamy, N., Michalek, G., Yashiro, S., et al. 2023, ApJ, 952, L13, doi: 10.3847/2041-8213/acdde2
  * Gopalswamy et al. (2015) Gopalswamy, N., Xie, H., Akiyama, S., et al. 2015, ApJ, 804, L23, doi: 10.1088/2041-8205/804/1/L23
  * Gopalswamy et al. (2010) Gopalswamy, N., Yashiro, S., Michalek, G., et al. 2010, SunGe, 5, 7
  * Nandy (2021) Nandy, D. 2021, Sol. Phys., 296, 54, doi: 10.1007/s11207-021-01797-2


#  Transformer Encoder for Social Science ††thanks: This project is funded by
Niehaus Center for Globalization and Governance. We are grateful for the
advice from David Turner, Marc Ratkovic, Walter Mebane, Sirus Bouchat, Brian
Klobucher, and Pradhan Abhijeet. ††thanks:
GitHub:https://github.com/haosenge/TESS, Hugging Face:
https://huggingface.co/hsge/TESS_768_v1

Haosen Ge
Princeton University
<EMAIL_ADDRESS>
In Young Park
Boston University
<EMAIL_ADDRESS>
Xuancheng Qian
Princeton University
<EMAIL_ADDRESS>
Grace Zeng
Princeton University
<EMAIL_ADDRESS>

###### Abstract

High-quality text data has become an important data source for social
scientists. We have witnessed the success of pretrained deep neural network
models, such as BERT and RoBERTa, in recent social science research. In this
paper, we propose a compact pretrained deep neural network, Transformer
Encoder for Social Science (TESS), explicitly designed to tackle text
processing tasks in social science research. Using two validation tests, we
demonstrate that TESS outperforms BERT and RoBERTa by 16.7% on average when
the number of training samples is limited (<1,000 training instances). The
results display the superiority of TESS over BERT and RoBERTa on social
science text processing tasks. Lastly, we discuss the limitation of our model
and present advice for future researchers.

##  1 Introduction

Text data has become a central piece of social science research. In
particular, political scientists make use of text data such as legislative or
legal documents [24, 5], news articles [15, 12, 4, 27, 14, 11, 1], speeches
[21, 7, 3, 9], and open-ended survey questions [23, 22, 20].

We have witnessed the success of applying pretrained DNNs, such as BERT and
RoBERTa [6, 18], on a wide array of text-related supervised learning tasks
[13, 26]. However, while DNNs can achieve near-human performance, training
DNNs well usually requires a significant amount of data, and annotating a
large training set is often costly in social science research settings.

To reduce the cost of using DNNs in social science, we propose a pretrained
DNN model designed for social science text processing tasks, Transformer
Encoder for Social Science (TESS). Our model offers four advantages over the
popular models used by political scientists (e.g., BERT and RoBERTa) on social
science-related tasks:

  * •

Less Data Hungry: TESS demonstrates superior performance in small training
sample settings compared to other popular models.

  * •

Smaller Size: TESS is about half the size of BERT and RoBERTa, which makes it
easy to use on machines with limited memory.

  * •

Longer Input Sequence: TESS can read text sequences up to 768 tokens at a
time, while most models on the market take text sequences up to 512 tokens as
input.

  * •

Better Performance: TESS shows a modest increase in performance than other
models on social science related tasks.

Therefore, we believe TESS can be a good alternative to many pretrained DNNs
when conducting social science research. Like BERT and RoBERTa, TESS adopts
the structure of a transformer encoder [25]. The encoder structure uses
parameters more efficiently than the decoder (e.g., GPT-2) but cannot be used
for language generation purposes. Since most language processing tasks
encountered by social scientists do not include natural language generation,
we believe that the encoder structure fits researchers’ needs better than the
decoder structure.

However, we improve upon BERT and RoBERTa by introducing the following
changes:

  * •

Embedding Dimensions and Weight Sharing: Following the findings by [17], we
reduce the wording embedding dimension from 768 to 128 and enable weight-
sharing. The model shares the attention weights across all layers but uses
individual feed-forward networks. This architecture significantly reduces the
number of required parameters (by around 50%) while still preserving the
performance.

  * •

Pretraining on a Social Science Corpus: We conducted masked language modeling
on a considerable social science corpus that includes U.S. congressional
bills, annual reports of U.S. public firms, Preferential Trade Agreements,
UNGA Resolutions and U.S. Court Opinions.

  * •

Increasing Input Sequence Length: we set the input sequence length to be 768
instead of 512, as most text processing tasks in social science require
dealing with long documents.

To evaluate the model’s performance, we conducted two validation studies with
tasks closely related to political science research. First, we test the
model’s performance using the dataset published by [13]. In their project,
they propose a multi-class classification task in which human annotators are
tasked with comparing the similarity of two subsections of U.S. congressional
bills. Later, they test the performance of a series of models, including
RoBERTa and BERT. In the second validation test, we utilize the original data
proposed by [8]. He proposed a binary sequence classification task to
distinguish whether a text sequence contains information about regulatory
barriers in certain countries. Our model, TESS, achieves a 6.6% increase in
full sample performance compared with BERT and RoBERTa. Furthermore, we test
the model’s performance under the small training sample scenario. On average,
TESS outperforms BERT and RoBERTa by 16.7% when the number of training samples
is below 1,000.

In the next section, we will introduce the model, corpus, and pretraining
tasks. The final section will present results from validation analyses.

##  2 TESS: Transformer Encoder for Social Science

###  2.1 Training Corpora

We collected over 35GB of compressed text from multiple sources. Table 1
offers a brief view of the text corpora.

Table 1: Corpora TEXT | SOURCE
---|---
Preferential Trade Agreements | Texts of Trade Agreements
Congressional Bills | [16]
UNGA Resolutions | UN
Firms’ Annual Reports | [19]
U.S. Court Opinions | Caselaw Access Project

The corpora differ significantly from the Wikipedia corpus, which is commonly
used in pretrained language models. They are, however, close to the text that
many political scientists encounter in research. Recent studies show that
pretraining on different corpora will significantly increase the model’s
performance in the corpora [2, 10]. We cannot collect comprehensive data from
news outlets or twitter due to copyright issues.

###  2.2 Model Pretraining

Similar to BERT and RoBERTa, TESS is a 12-layer transformer encoder. However,
all 12 attention layers share the same weights, but the feed-forward layer has
its individual weights. This architectural design is based on the results from
[17], which shows that sharing attention weights do not negatively affect
model performance while sharing the feed-forward network weights does. By
sharing attention weights, we reduce the model size by 50% but still preserve
the model performance.

We use the classic masked language modeling to pretrain the encoder. Following
the literature, the masking probability is set to be 15%. However, we allow
n-gram masking, which is shown to lead to superior performance. In the
original BERT training, the authors also implement next sentence prediction.
We exclude the next sentence prediction task because our text is usually
comprised of multiple sections that are not closely connected. Our experiment
also found that training with next sentence prediction leads to worse
performance than only training with masked language modeling.

We train the model for 120K steps in batches of 2,000 to facilitate model
convergence. We use the AdamW optimizer with a learning rate of 1e-4. The
model is trained on 4 NVIDIA A100 GPUs hosted by Lambda Labs GPU Cloud.

##  3 Validation

###  3.1 Bill Similarity

[13] propose a five-class classification task to compare bill similarity. The
similarity between two subsections of bills ranges from 0 (unrelated) to 4
(identical). The authors manually annotated 4,721 pairs of bill subsections.
Among all pairs of subsections, the unrelated bills account for more than 50%
of total examples, which shows considerable data imbalance. In the paper, the
authors find that the best performance is achieved by RoBERTa (F1 = 76.8).

We fine-tuned TESS on the training set with 3,305 human-annotated samples and
tested the model performance on the authors’ validation set. We experimented
with learning rates = {1e-5, 3e-5, 5e-5} and epochs = {1, 3, 5} and report
best outcomes for each model.

In Figure 1(a), we present the comparison of TESS, RoBERTa, and BERT. To test
model performance under small sample scenarios, we fine-tune each model using
the sample sizes {50, 200, 500, 800, 1000, 3305 (full sample)}. It can be
observed that TESS outperforms BERT and RoBERTa across almost all sample sizes
except for the sample size of 50. When trained on the full sample, TESS
achieved an F1 of 0.79, higher than BERT, RoBERTa, and the authors’ best
results in the paper. With only $500$ training instances ($1/6$ of the total
training samples), TESS can achieve an F1 above 0.7, which showcases the
model’s ability to reduce researchers’ annotation costs.

(a) Bill Similarity

(b) Regulatory Barrier

Figure 1: Validation Results

###  3.2 Regulatory Barrier

In the second validation study, we utilize the original data set compiled by
[8]. They manually annotate sentences and paragraphs from U.S. public
companies’ annual reports (i.e., 10-Ks). By reading the sentences, they aim to
classify whether a sentence or section contains information about the
regulatory barrier encountered by the firm. We use three instances from the
training set to illustrate the task:

  * •

Positive: We have difficulty gaining market share in countries such as Japan
because of regulatory restrictions and customer preferences.

  * •

Positive: Burdens of complying with a variety of foreign laws, including more
protective employment laws affecting our sizable workforce in Germany.

  * •

Negative: We are obligated to comply with various foreign, federal, state, and
other environmental laws and regulations, including the environmental laws and
regulations of the United States, Iceland, China, and the EU.

We again experiment with learning rates = {1e-5, 3e-5, 5e-5} and epochs = {1,
3, 5}. When trained on the full training sample ($n$ = 2,401), TESS has the
best performance among the three models (F1 = 0.69). We further investigate
the small sample scenarios, $n=\\{100,400,700,1000\\}$. Similarly, TESS
outperforms BERT and RoBERTa in small training samples: the gap between TESS
and the other two models shrinks when the sample size increases.

The two validation results prove TESS’ effectiveness in saving annotation
costs without sacrificing the large sample performance.

##  4 Conclusions and Limitations

This paper proposes a transformer encoder designed for the social science
research context. By validating on political science text processing tasks, we
demonstrate the superiority of TESS over BERT and RoBERTa, especially in small
training sample settings.

We also prove the value of building social science-specific language models,
as many texts encountered by social scientists are quite different from the
generic text corpora used by computer scientists. This project shows that the
path to studying pre-trained DNNs in the social science context may be
fruitful. We encourage future researchers to explore whether certain
pretraining tasks may serve social scientists better than the standard masked
language modeling or next word prediction.

Since we do not include text from news outlets and Twitter data, we overlook a
significant portion of the text that political scientists often use. It is
unclear whether TESS can outperform BERT and RoBERTa on the news or Twitter
data, as TESS may specialize in government documents at the expense of
understanding other text types. For that reason, we recommend using TESS for
processing text similar to our pretraining corpora while remaining cautious if
adapting it to a new domain.

## References

  * [1] Pablo Barberá et al.  “Automated Text Classification of News Articles: A Practical Guide”  In _Political Analysis_ 29.1 Cambridge University Press, 2021, pp. 19–42
  * [2] Iz Beltagy, Kyle Lo and Arman Cohan  “SciBERT: A pretrained language model for scientific text”  In _arXiv preprint arXiv:1903.10676_ , 2019
  * [3] Constantine Boussalis, Travis G. Coan, Mirya R. Holman and Stefan Müller  “Gender, Candidate Emotional Expression, and Voter Reactions During Televised Debates”  In _American Political Science Review_ 115.4 Cambridge University Press, 2021, pp. 1242–1257
  * [4] Charles Chang and Michael Masterson  “Using Word Order in Political Text Classification with Long Short-term Memory Models”  In _Political Analysis_ 28.3 Cambridge University Press, 2019, pp. 395–411
  * [5] Frederick Chen and Jian Xu  “Settling Disputes in Authoritarian Courts: Multinational Corporations and State Capture”  In _Available at SSRN 3597182_ , 2021
  * [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova  “Bert: Pre-training of deep bidirectional transformers for language understanding”  In _arXiv preprint arXiv:1810.04805_ , 2018
  * [7] Ted Enamorado, Gabriel López-Moctezuma and Marc Ratkovic  “Scaling Data from Multiple Sources”  In _Political Analysis_ 29.2 Cambridge University Press, 2021, pp. 212–235
  * [8] Haosen Ge  “Measuring Regulatory Barriers Using Annual Reports of Firms”  In _working paper_ , 2021
  * [9] Max Goplerud and Daniel M. Smith  “Who Answers for the Government? Bureaucrats,Ministers, and Responsible Parties”  In _American Journal of Political Science_ 00.0, 2021, pp. 1–16
  * [10] Suchin Gururangan et al.  “Don’t stop pretraining: adapt language models to domains and tasks”  In _arXiv preprint arXiv:2004.10964_ , 2020
  * [11] Andrew Halterman, Katherine A Keith, Sheikh Muhammad Sarwar and Brendan O’Connor  “Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence”  In _Findings of ACL_ , 2021
  * [12] Connor Huff and Joshua D. Kertzer  “How the Public Defines Terrorism”  In _American Journal of Political Science_ 62.1, 2018, pp. 55–71
  * [13] Jiseon Kim, Elden Griggs, In Song Kim and Alice Oh  “Learning Bill Similarity with Annotated and Augmented Corpora of Bills”  In _arXiv preprint arXiv:2109.06527_ , 2021
  * [14] Nam Kyu Kim and Jun Koga Sudduth  “Political institutions and coups in dictatorships”  In _Comparative Political Studies_ 54.9 SAGE Publications Sage CA: Los Angeles, CA, 2021, pp. 1597–1628
  * [15] Gary King and Will Lowe  “An automated information extraction tool for international conflict data with performance as good as human coders: A rare events evaluation design”  In _International Organization_ 57.3 Cambridge University Press, 2003, pp. 617–642
  * [16] Anastassia Kornilova and Vlad Eidelman  “BillSum: A corpus for automatic summarization of US legislation”  In _arXiv preprint arXiv:1910.00523_ , 2019
  * [17] Zhenzhong Lan et al.  “Albert: A lite bert for self-supervised learning of language representations”  In _arXiv preprint arXiv:1909.11942_ , 2019
  * [18] Yinhan Liu et al.  “Roberta: A robustly optimized bert pretraining approach”  In _arXiv preprint arXiv:1907.11692_ , 2019
  * [19] Tim Loughran and Bill McDonald  “Textual analysis in accounting and finance: A survey”  In _Journal of Accounting Research_ 54.4 Wiley Online Library, 2016, pp. 1187–1230
  * [20] Devorah Manekin and Tamar Mitts  “Effective for Whom? Ethnic Identity and Nonviolent Resistance”  In _American Political Science Review_ 116.1 Cambridge University Press, 2022, pp. 161–180
  * [21] Burt L Monroe, Michael P Colaresi and Kevin M Quinn  “Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict”  In _Political Analysis_ 16.4 Cambridge University Press, 2008, pp. 372–403
  * [22] Lisa Mueller  “Crowd Cohesion and Protest Outcomes”  In _American Journal of Political Science_ 00.0, 2022, pp. 1–16
  * [23] Margaret E Roberts et al.  “Structural topic models for open-ended survey responses”  In _American journal of political science_ 58.4 Wiley Online Library, 2014, pp. 1064–1082
  * [24] Rory Truex  “Authoritarian gridlock? Understanding delay in the Chinese legislative system”  In _Comparative Political Studies_ 53.9 SAGE Publications Sage CA: Los Angeles, CA, 2020, pp. 1455–1492
  * [25] Ashish Vaswani et al.  “Attention is all you need”  In _Advances in neural information processing systems_ 30, 2017
  * [26] Sandra Wankmüller  “Neural transfer learning with transformers for social science text analysis”  In _arXiv preprint arXiv:2102.02111_ , 2021
  * [27] Omar Wasow  “Agenda Seeding: How 1960s Black Protests Moved Elites, Public Opinion and Voting”  In _American Political Science Review_ 114.3 Cambridge University Press, 2020, pp. 638–659