# Multicomponent Activity Cycles using Hilbert-Huang Analysis

E. N. Velloso Universidade Federal do Rio Grande do Norte (UFRN), 59078-970,
Natal, RN, Brazil  F. Anthony Universidade Federal do Rio Grande do Norte
(UFRN), 59078-970, Natal, RN, Brazil  J.-D. do Nascimento, Jr. Universidade
Federal do Rio Grande do Norte (UFRN), 59078-970, Natal, RN, Brazil  Center
for Astrophysics $|$ Harvard $\&$ Smithsonian, 60 Garden Street, Cambridge, MA
02138, USA  L. F. Q. Silveira Universidade Federal do Rio Grande do Norte
(UFRN), 59078-970, Natal, RN, Brazil  J. Hall  Lowell Observatory, 1400 West
Mars Hill Road, Flagstaff, AZ 86001, USA  S. H. Saar Center for Astrophysics
$|$ Harvard $\&$ Smithsonian, 60 Garden Street, Cambridge, MA 02138, USA

###### Abstract

The temporal analysis of stellar activity evolution is usually dominated by a
complex trade-off between model complexity and interpretability, often by
neglecting the non-stationary nature of the process. Recent studies appear to
indicate that the presence of multiple coexisting cycles in a single star is
more common than previously thought. The correct identification of physically
meaningful cyclic components in spectroscopic time series is therefore a
crucial task, which cannot overlook local behaviors. Here we propose a
decomposition technique which adaptively recovers amplitude- and frequency-
varying components. We present our results for the solar activity as measured
both by the sunspot number and the $K$-line emission index, and we
consistently recover the Schwabe and Gleissberg cycles as well as the
Gnevyshev-Ohl pattern probably related to the Hale cycle. We also recover the
known 8-year cycle for 61 Cygni A, in addition to evidence of a three-cycles
long pattern reminiscent of the Gnevyshev-Ohl rule. This is particularly
interesting as we cannot discard the possibility of a relationship between the
measured field polarity reversals and this Hale-like periodicity.

Computational methods (1965), Solar cycle (1487), Solar activity (1475),
Stellar activity (1580)

##  1 Introduction

The number of spots that emerge on the solar surface is known to vary in a
regular 11-year cycle (Schwabe, 1844). Polarimetric observations of the Sun
have revealed the relationship between these structures and the solar magnetic
field (Hale, 1908), with the interesting observation that the magnetic
polarity of sunspots switches sign every consecutive cycle. This polarity
reversal means that a complete magnetic cycle spans 22-years, twice the length
of the Schwabe (amplitude) cycle.

A stellar monitoring survey was conducted at Mount Wilson Observatory for over
30 years and led the way towards detecting similar starspot cycles in the
chromospheric activity of solar-type stars, as measured by the Ca II H+K flux
(Wilson, 1978). For the many main-sequence stars in which such cyclic behavior
was observed, Brandenburg et al. (1998) identified two relationships between
cycle length, rotation period and mean activity level, known as the active and
inactive branches. However, many stars since then have been found to present a
coexistence of long and short activity cycles, some of these falling on both
branches simultaneously, and some between them (e.g., Boro Saikia et al.,
2018).

For a few stars, the large-scale magnetic field has been reconstructed using
Zeeman Doppler imaging (ZDI) spanning several years, allowing the detection of
field topology variations. On $\xi$ Boo A, $\epsilon$ Eri and HN Peg, such
variations were found to be rapid and localized in time (Morgenthaler et al.,
2012; Jeffers et al., 2014; Boro Saikia et al., 2015). Although $\tau$ Boo was
originally thought to reverse its polarity yearly, significantly longer than
its 120-day $S$-index cycle, it was found later that this discrepancy had
arisen from poor time sampling. Jeffers et al. (2018) discovered $\tau$ Boo’s
240-day magnetic cycle, the fastest ever observed, which coincided with two
complete consecutive chromospheric cycles. 61 Cyg A was the first cool star
beyond the Sun where a magnetic cycle was detected in phase with its
chromospheric cycle (Boro Saikia et al., 2016); similar results were recently
also confirmed in HD 75332 (Brown et al., 2021) and 18 Sco.

In order to better trace the magnetic evolution of solar-type stars, the Mount
Wilson dataset has been recently revisited on several occasions, either by
extending the data using multiple other surveys while sticking to traditional
periodogram methods (e.g., Boro Saikia et al., 2018), or by introducing new
methods to take into account the non-sinusoidal and locally harmonic nature of
the cycles (e.g., Olspert et al., 2018). Indeed, as is well known for the Sun,
stellar activity cycles are not perfectly harmonic, or even strictly periodic;
variations in both cycle length and amplitude are common and can make it
challenging to interpret the results of methods in the Fourier domain.

As shown by Oláh et al. (2016), time-frequency analysis can be used to reveal
multiple and changing cycles in the Mount Wilson survey dataset. In this
letter, we propose to extend this idea by using fully adaptive time-frequency
methods with the specific purpose of disentangling multiple physically
meaningful components from the data.

The Hilbert-Huang Transform (HHT), based on the Empirical Mode Decomposition
(EMD; Huang et al., 1998), is a data-driven technique that addresses this
exact issue. The goal of Hilbert-Huang analysis is to decompose a signal into
a finite sum of amplitude-modulated frequency-modulated (AM-FM) oscillations
plus some monotonic trend. Such oscillations, known as Intrinsic Mode
Functions (IMFs), present well-defined instantaneous values of amplitude and
frequency. These can be derived from their analytic signal using the Hilbert
transform.

This method of obtaining a time-frequency representation heavily contrasts
with other traditional methods from the literature. Linear methods based on
inner products with a pre-determined family of signals, such as the Short-Term
Fourier Transform (STFT) or the Wavelet Transform, have their resolution
limited by the Heisenberg Uncertainty Principle. On the other hand, the HHT is
able to precisely track single values of instantaneous frequencies for each
component, while also being agnostic to the specific signal shape.

Many improvements to the original EMD algorithm have been proposed to address
different issues arising from the original formulation. For a more in-depth
review of the HHT and EMD, see Huang & Shen (2005) and Huang & Attoh-Okine
(2005). In this letter, we use a combination of the Complete Ensemble EMD with
Adaptive Noise (CEEMDAN; Colominas et al., 2014) and the post-processing steps
proposed by Wu & Huang (2009). The instantaneous frequency is then calculated
on the resulting modes using the Normalized Hilbert Transform (NHT; Huang et
al., 2009).

Hilbert-Huang analysis has been able to extract and characterize periodicities
in the solar cycles from many activity indicators, such as the sunspot number
(Barnhart & Eichinger, 2011; Gao, 2017), the 10.7 cm radio flux, and
helioseismic frequency shifts (Kolotkov et al., 2015).

The remaining parts of this letter are organized as follows: in Section 2 we
define our activity time series datasets both for the Sun and for 61 Cyg A;
next, Section 3 details our data-driven multicomponent analysis method as well
as the preprocessing steps needed; Section 4 presents the results of our
experiments, and we conclude our discussion in Section 5.

##  2 Observations

For the Sun, the first observational dataset used in our analysis as a proxy
for magnetic activity is the daily total sunspot number series obtained from
the World Data Center for Solar Index and Long-term Solar Observations (WDC-
SILSO) at the Royal Observatory of Belgium
(http://www.sidc.be/silso/datafiles). The dataset spans January 1818 to the
present. The second dataset is the monthly-averaged Ca II K emission index
composite; derived from the Kodaikanal, Sacramento Peak, and SOLIS/ISS
observations (Bertello et al., 2016, 2017; Egeland et al., 2017).

The $S$-index time series data for K dwarf 61 Cyg A is composed of the long-
term surveys from the NSO Mount Wilson project (MW; Duncan et al., 1991;
Baliunas et al., 1995) and the solar and stellar activity program from the
Lowell observatory (SSS; Hall et al., 2007; Lockwood et al., 2007). We also
used data published by Boro Saikia et al. (2016) using the NARVAL high-
resolution spectropolarimeter and the data presented in the radial velocity
catalog from the California Planet Search (CPS; Howard et al., 2010; Rosenthal
et al., 2021). We additionally determined the $S$-index from the ESPaDOnS and
NARVAL spectropolarimeter data from 2015 onwards following the methodology
described in Marsden et al. (2014). These measurements span more than 50 years
of observations, extending from mid-1967 through early 2018.

##  3 Hilbert-Huang Analysis

The goal of our method is to find a finite set of oscillatory components that
add up to the original data, while also describing each individual component
by its instantaneous values of amplitude and frequency. To this end, our
pipeline consists of a preprocessing step to correct sampling issues, followed
by the actual decomposition of the signal, and a post-processing step to
ensure the components meet certain conditions needed for the Hilbert analysis
to, in the final step, extract their time-frequency representations.

###  3.1 Preprocessing

Although the EMD is a fully data-driven algorithm (meaning that it makes no
assumptions a priori on specific signal models), it still performs poorly if
the input signal is unevenly sampled (see e.g., Barnhart et al., 2011).
Therefore, we follow a similar preprocessing approach to the one used by
Kolláth & Oláh (2009), in which the data is first downsampled to uniform bins,
and later upsampled with a cubic smoothing spline.

We bin each time series at 90-day intervals using a triangular window to
compute the averages and to estimate the variances. The exact value of the
binsize has no strong impact on the resulting signal, as long as it’s within a
reasonable distance from the periods of interest; for stars with wildly
different rotation rates, this value should vary accordingly.

The binned data is then interpolated at 10-day intervals by a regularized
cubic spline with weights inversely proportional to the bin variances. The
resulting smoothed series preserves the local time-frequency content of the
original data to a high extent, as demonstrated by Kolláth & Oláh (2009).
Since there are no significant gaps in the solar observations, the amount of
added information to fill in missing data is negligible. Even for the 61 Cyg A
data, which has a few minor gaps, the added information is limited.

###  3.2 Decomposition

The EMD consists of an iterative algorithm in which each mode is successively
extracted through a sifting process. This sifting iteratively updates the
current mode by subtracting an estimate of the local mean, corresponding to
the average of the upper and lower envelopes, which are taken as the spline
interpolations of the local maxima and minima. Convergence is assessed through
a stopping criterion that checks if the resulting IMFs have the following
properties:

  1. 1.

the number of local extrema and the number of zero crossings are equal (or
differ by one);

  2. 2.

the local mean is (close to) zero (almost) everywhere.

In other words, all local maxima must be positive, all local minima must be
negative, and their envelopes must be symmetrical. After enough IMFs have been
subtracted from the original signal $x(t)$ for the remaining data to be
completely monotonic, we can write the resulting decomposition as the finite
sum:

$\displaystyle x(t)=\sum_{k=1}^{K}C_{k}(t)+r(t),$ (1)

where each $C_{k}(t)$ is one IMF and $r(t)$ is the monotonic trend.

The local nature of EMD may result in mode-mixing, i.e., similar oscillations
being separated in different modes or vice-versa. The improved algorithm used
in this work is the Complete Ensemble EMD with Adaptive Noise (CEEMDAN;
Colominas et al., 2014). Its main idea is to apply the EMD to an ensemble of
degraded copies of the original signal, each arising from a different
realization of additive white Gaussian noise. The EMD then behaves more
closely to a filter bank and the resulting modes are more regular; the
decomposition of spurious modes is also avoided.

Specifically, let $\mathcal{E}_{k}(\cdot)$ be an operator that returns the
$k$-th IMF via EMD, then define $\mathcal{M}(\cdot)$ as the local mean
operator:

$\displaystyle\mathcal{M}(x)=x-\mathcal{E}_{1}(x)$ (2)

Also, let $w^{(i)}$ be the $i$-th white noise realization ($i=1,\dots,I$).
Then, in each CEEMDAN iteration, we calculate the ensemble average of the
local mean:

$\displaystyle
q_{k}=\dfrac{1}{I}\sum_{i=1}^{I}\mathcal{M}(q_{k-1}+\beta_{k-1}\mathcal{E}_{k}(w^{(i)}))$
(3)

with $q_{0}=x$ and where $\beta_{k}$ are coefficients controlling the relative
energy of the introduced noise. The $k$-th pseudo-IMF $D_{k}$ is then found by
the difference between consecutive averages:

$\displaystyle D_{k}=q_{k-1}-q_{k}$ (4)

###  3.3 Post-processing

In the above analysis, we referred to the output of the CEEMDAN algorithm as
“pseudo-IMFs”. This is due to the fact that a sum of two different IMFs does
not in general preserve the necessary properties that define an IMF; in
particular, the ensemble averages computed during CEEMDAN cause the resulting
$D_{k}$ to lose such features.

Since these properties are important guarantees for the assumptions that each
component is a single AM-FM oscillation with well-defined amplitudes and
frequencies, a post-processing step is needed to transform the pseudo-IMFs
into proper IMFs $C_{k}$.

In this letter, we formalize the generic steps proposed by Wu & Huang (2009)
in the following post-processing algorithm:

$\displaystyle C_{k}$ $\displaystyle=\mathcal{E}_{1}(D_{k}+r_{k-1})$ (5)
$\displaystyle r_{k}$
$\displaystyle=r_{k-1}+D_{k}-C_{k}=\mathcal{M}(D_{k}+r_{k-1})$ (6)

with $r_{0}=0$. Since consecutive modes present oscillations on similar
timescales, this algorithm extracts a proper IMF from each consecutive pair of
pseudo-IMFs, resulting in the same number $K$ of components. The final
monotonic trend is given by $r(t)=r_{K}(t)+q_{K}(t)$, the sum of the final
step’s trends.

###  3.4 The marginal Hilbert spectrum

The main purpose of our analysis is not only to separate each individual
component of the time-series signal, but also to identify their underlying
periodicities. Since the extracted IMFs are not constrained to harmonic tones,
but are, by construction, single-mode signals, the instantaneous frequency is
a better measure for their time-varying periodicities.

We start by assuming each IMF can be written as an AM-FM oscillation with an
arbitrary, slow-varying amplitude and phase functions $a_{k}(t)$ and
$\phi_{k}(t)$:

$\displaystyle C_{k}(t)=a_{k}(t)\cos(2\pi\phi_{k}(t))$ (7)

The demodulation of the AM and FM subcomponents can be done with stable
results by using the Local Mean Decomposition (LMD; Smith, 2005). This method
is similar in nature to the EMD, but it decomposes a signal into a sum of
Product Functions (PFs):

$\displaystyle x(t)=\sum_{j=1}^{J}A_{j}(t)F_{j}(t)+m(t)$ (8)

where $A_{j}$ and $F_{j}$ correspond to the AM and FM factors of each PF and
$m$ is the local mean residue. Due to the nature of the stopping criteria for
the EMD, equation 7 is only true as an approximation, while equation 8 holds
whenever the LMD converges; however, the LMD can diverge when processing
arbitrarily complex signals, which is why it is only applied here to the
already simplified IMFs. Other examples of this joint EMD-LMD technique can be
found in Yue et al. (2020).

A single PF can thus very closely approximate each IMF, and we can write:

$\displaystyle C_{k}(t)=A_{k}(t)F_{k}(t)+\mu_{k}(t)$ (9)

where $\mu_{k}(t)$ is very close to zero everywhere, and can locally account
for some edge effects of the EMD. We therefore associate $a_{k}=A_{k}$ and
$\cos(2\pi\phi_{k})=F_{k}$ and derive the time-varying instantaneous frequency
$f_{k}=\phi_{k}^{\prime}$ by Hilbert-transforming the FM component (Huang et
al., 2009). The Hilbert spectra of each mode can hence be defined as the
distribution of the instantaneous frequencies weighted by the instantaneous
squared amplitudes.

##  4 Results and Discussion

###  4.1 Solar Cycles

Figure 1: Result of CEEMDAN for the Sun. The sunspot number is shown in blue
and the $K$-index in orange. The top-left panel shows the preprocessed input
data. The right-side panels show the Hilbert spectra corresponding to their
respective time-domain components.

To validate our methods, we tested the extent of our ability to recover
multiple, distinct cycles on solar data using two different observational
proxies, as detailed in Section 2: the daily total sunspot number series from
1818 to 2021, and the composite K-line emission index from 1907 to 2017
(Bertello et al., 2016; Egeland et al., 2017), hereafter referred to as the
$K$-index.

Figure 1 presents the result of our decomposition of solar activity data. The
$K$-index is shown to correlate strongly with the sunspot number counts, and
for both series our analysis identifies three main modes: the first one at
$\approx$11 yr, another one at twice that period, and a longer
$\approx$90-year cycle. Although the $K$-index series is too short to resolve
what appears to be the Gleissberg cycle, its trend component can be seen to
correlate very well with the latter half of the final sunspot IMF, which
indicates that this method is able to identify long-term trends and potential
analogues of the solar Gleissberg cycle within the relatively short
spectroscopic time series currently available for other stars.

It is reassuring to notice that the amplitude of the primary cycle seems to be
modulated in a shape very similar to the Gleissberg component, which is to be
expected if these components truly represent the underlying physical
processes. Also, the secondary cycle appears to capture the empirical
Gnevyshev-Ohl rule, which dictates that odd solar cycles tend to be stronger
than even ones (Gnevyshev & Ohl, 1948; Hathaway, 2015). The underlying cause
of this observed behavior is thought to be related to the $\approx$22-year
Hale magnetic cycle, but the connection is not yet entirely understood.

###  4.2 61 Cygni A

We also applied our decomposition technique to the much shorter time series of
61 Cyg A $S$-index measurements. The composition of observations from
different instruments is shown in the top left panel of Figure 2 together with
the smooth interpolation resulting from our preprocessing steps. The remaining
panels of Figure 2 present the three best defined cyclic components found by
the application of CEEMDAN and their corresponding Hilbert spectra.
Interestingly, besides the well-known $\approx$8-years long chromospheric
activity cycle (e.g., Baliunas et al., 1995, ; cycle 1), a component with a
period about three times as long is also well separated (cycle 2). This
component can be visually identified in the original data as a variation in
activity maxima repeating itself in a pattern every three consecutive cycles.
Such behavior is reminiscent of the solar Gnevyshvev-Ohl rule, except with a
period $\approx$3 times rather than twice that of the activity cycle.

This star had its large-scale magnetic field detected via ZDI in six different
epochs ranging from 2007 to 2015 (Boro Saikia et al., 2016) and six more
epochs from 2015 to 2018 (Boro Saikia et al., 2018). Its magnetic field
polarity was found to flip twice between the first and last observation,
indicating the presence of a Hale-like magnetic polarity cycle in this star.
Those polarity measurements are shown in Figure 2 as vertical lines, colored
according approximately to the average field polarity and intensity at these
epochs. An inversion can be seen to occur between the timeframes of two
chromospheric activity minima, in agreement with the hypothesis of a $\approx
15$-years long magnetic cycle. However, it is also important to note that a
very short baseline was used to draw this inference, and the few datapoints we
have cannot exclude the possibility that the Hale-like polarity cycle is
actually linked to the observed three-cycle Gnevyshev-Ohl pattern, with a
somewhat longer inversion period of $\approx 11$ years. The predictions of
both these hypotheses are shown in Figure 2 as the colored red and blue
backgrounds in the two cyclic components.

Figure 2: Result of our analysis for 61 Cyg A. The colored vertical lines
indicate the epochs for which its large scale magnetic geometry was
reconstructed using ZDI (Boro Saikia et al., 2016, 2018). The background
colors indicate predicted polarities for a hypothesis of a complete magnetic
reversal taking three times cycle 1 (shown in the Cycle 2 panel) or two times
cycle 1 (as in the Sun, shown in the Cycle 1 panel).

##  5 Conclusions

We introduced a fully data-driven approach to separate different coexisting
activity cycle components using the Hilbert-Huang transform that is immune to
the non-stationary variations in cycle lengths and amplitudes known to exist.

This methodology was shown to be capable of not only recovering the mean
periods of known solar cycles at different timescales, but also the time-
domain characteristics of these components, such as amplitude variations, in
agreement with expectations.

For the K-dwarf 61 Cygni A, our method confirms its Schwabe-like 8-year cycle
and also presents evidence of a potential three-cycle long Gnevyshev-Ohl-like
behavior, shown to be possibly tied to a polarity reversal. A confirmation of
this result would be of some importance in the sense that (i) it strengthens
the relationship between the solar Gnevyshev-Ohl rule and the Hale cycle, and
(ii) it shows that we are able to detect the activity signature of magnetic
polarity changes in stellar spectroscopic calcium data.

##  Appendix A On the Dependence of the Results on the Choice of Smoothing

While describing our proposed pipeline and its main advantages, we highlighted
the fact that it was a fully adaptive and data-driven method and that it
doesn’t rely on previous knowledge surrounding the dataset. We also claimed
that, even though there apparently is need of a choice surrounding the
smoothing procedures during the preprocessing steps, there was no strong
effect on the results as long as the bin sizes were chosen within some
reasonable interval. This short appendix intends to justify such a claim.

The main parameter that controls how much the input data is smoothed out is
the size of the bins used for averaging. Figure 3 shows an example when the
bins are almost twice as spaced out as what was used in the paper (see Figure
2). The spikes get smoothed out and the high-frequency residuals found by the
decomposition tend to disappear, but the overall structure of the two main
cycles stays closely similar. The same qualitative conclusions can be taken
out from this, thanks to the adaptive filtering nature of the method.

Figure 3: Same as Figure 2, however for a smoothed out version of the signal
using 150-day bins during downsampling.

The HK_Project_v1995_NSO data derives from the Mount Wilson Observatory HK
Project, which was supported by both public and private funds through the
Carnegie Observatories, the Mount Wilson Institute, and the Harvard-
Smithsonian Center for Astrophysics starting in 1966 and continuing for over
36 years. These data are the result of the dedicated work of O. Wilson, A.
Vaughan, G. Preston, D. Duncan, S. Baliunas, and many others. SHS is grateful
for support from NASA Heliophysics LWS grant NNX16AB79G, NASA XRP grant
80NSSC21K0607, and NASA EPRV grant 80NSSC21K1037. JDNJr acknowledges the
financial support by CNPq funding through the grant PQ 310847/2019-2.

## References

  * Baliunas et al. (1995) Baliunas, S. L., Donahue, R. A., Soon, W. H., et al. 1995, ApJ, 438, 269, doi: 10.1086/175072
  * Barnhart & Eichinger (2011) Barnhart, B. L., & Eichinger, W. E. 2011, Sol. Phys., 269, 439, doi: 10.1007/s11207-010-9701-6
  * Barnhart et al. (2011) Barnhart, B. L., Nandage, H. K. W., & Eichinger, W. 2011, Adv. Data Sci. Adapt. Anal., 3, 483
  * Bertello et al. (2017) Bertello, L., Marble, A. R., & Pevtsov, A. A. 2017, arXiv e-prints, arXiv:1702.00838.  https://arxiv.org/abs/1702.00838
  * Bertello et al. (2016) Bertello, L., Pevtsov, A., Tlatov, A., & Singh, J. 2016, Sol. Phys., 291, 2967, doi: 10.1007/s11207-016-0927-9
  * Boro Saikia et al. (2015) Boro Saikia, S., Jeffers, S. V., Petit, P., et al. 2015, A&A, 573, A17, doi: 10.1051/0004-6361/201424096
  * Boro Saikia et al. (2016) Boro Saikia, S., Jeffers, S. V., Morin, J., et al. 2016, A&A, 594, A29, doi: 10.1051/0004-6361/201628262
  * Boro Saikia et al. (2018) Boro Saikia, S., Marvin, C. J., Jeffers, S. V., et al. 2018, A&A, 616, A108, doi: 10.1051/0004-6361/201629518
  * Brandenburg et al. (1998) Brandenburg, A., Saar, S. H., & Turpin, C. R. 1998, ApJ, 498, L51, doi: 10.1086/311297
  * Brown et al. (2021) Brown, E. L., Marsden, S. C., Mengel, M. W., et al. 2021, MNRAS, 501, 3981, doi: 10.1093/mnras/staa3878
  * Colominas et al. (2014) Colominas, M. A., Schlotthauer, G., & Torres, M. E. 2014, Biomedical Signal Processing and Control, 14, 19, doi: 10.1016/j.bspc.2014.06.009
  * Duncan et al. (1991) Duncan, D. K., Vaughan, A. H., Wilson, O. C., et al. 1991, ApJS, 76, 383, doi: 10.1086/191572
  * Egeland et al. (2017) Egeland, R., Soon, W., Baliunas, S., et al. 2017, ApJ, 835, 25, doi: 10.3847/1538-4357/835/1/25
  * Gao (2017) Gao, P. X. 2017, MNRAS, 472, 2913, doi: 10.1093/mnras/stx2206
  * Gnevyshev & Ohl (1948) Gnevyshev, M. N., & Ohl, A. I. 1948, Astronomicheskii Zhurnal, 25, 18
  * Hale (1908) Hale, G. E. 1908, ApJ, 28, 315, doi: 10.1086/141602
  * Hall et al. (2007) Hall, J. C., Lockwood, G. W., & Skiff, B. A. 2007, AJ, 133, 862, doi: 10.1086/510356
  * Hathaway (2015) Hathaway, D. H. 2015, Living Reviews in Solar Physics, 12, 4, doi: 10.1007/lrsp-2015-4
  * Howard et al. (2010) Howard, A. W., Johnson, J. A., Marcy, G. W., et al. 2010, ApJ, 721, 1467, doi: 10.1088/0004-637X/721/2/1467
  * Huang & Attoh-Okine (2005) Huang, N., & Attoh-Okine, N. 2005, The Hilbert-Huang Transform in Engineering (CRC Press), doi: 10.1201/9781420027532
  * Huang & Shen (2005) Huang, N., & Shen, S. 2005, Hilbert-Huang Transform and Its Applications (World Scientific)
  * Huang et al. (2009) Huang, N. E., Wu, Z., Long, S. R., et al. 2009, Advances in Adaptive Data Analysis, 01, 177, doi: 10.1142/s1793536909000096
  * Huang et al. (1998) Huang, N. E., Shen, Z., Long, S. R., et al. 1998, Proceedings of the Royal Society of London Series A, 454, 903, doi: 10.1098/rspa.1998.0193
  * Jeffers et al. (2014) Jeffers, S. V., Petit, P., Marsden, S. C., et al. 2014, A&A, 569, A79, doi: 10.1051/0004-6361/201423725
  * Jeffers et al. (2018) Jeffers, S. V., Mengel, M., Moutou, C., et al. 2018, MNRAS, 479, 5266, doi: 10.1093/mnras/sty1717
  * Kolláth & Oláh (2009) Kolláth, Z., & Oláh, K. 2009, A&A, 501, 695, doi: 10.1051/0004-6361/200811303
  * Kolotkov et al. (2015) Kolotkov, D. Y., Broomhall, A. M., & Nakariakov, V. M. 2015, MNRAS, 451, 4360, doi: 10.1093/mnras/stv1253
  * Lockwood et al. (2007) Lockwood, G. W., Skiff, B. A., Henry, G. W., et al. 2007, ApJS, 171, 260, doi: 10.1086/516752
  * Marsden et al. (2014) Marsden, S. C., Petit, P., Jeffers, S. V., et al. 2014, MNRAS, 444, 3517, doi: 10.1093/mnras/stu1663
  * Morgenthaler et al. (2012) Morgenthaler, A., Petit, P., Saar, S., et al. 2012, A&A, 540, A138, doi: 10.1051/0004-6361/201118139
  * Oláh et al. (2016) Oláh, K., Kővári, Z., Petrovay, K., et al. 2016, A&A, 590, A133, doi: 10.1051/0004-6361/201628479
  * Olspert et al. (2018) Olspert, N., Lehtinen, J. J., Käpylä, M. J., Pelt, J., & Grigorievskiy, A. 2018, A&A, 619, A6, doi: 10.1051/0004-6361/201732525
  * Rosenthal et al. (2021) Rosenthal, L. J., Fulton, B. J., Hirsch, L. A., et al. 2021, ApJS, 255, 8, doi: 10.3847/1538-4365/abe23c
  * Schwabe (1844) Schwabe, H. 1844, Astronomische Nachrichten, 21, 233, doi: 10.1002/asna.18440211505
  * Smith (2005) Smith, J. R. 2005, Journal of The Royal Society Interface, 2, 443
  * Wilson (1978) Wilson, O. C. 1978, ApJ, 226, 379, doi: 10.1086/156618
  * Wu & Huang (2009) Wu, Z., & Huang, N. E. 2009, Advances in Adaptive Data Analysis, 01, 1, doi: 10.1142/s1793536909000047
  * Yue et al. (2020) Yue, S., Wang, Y., Wei, L., Zhang, Z., & Wang, H. 2020, Aerospace Science and Technology, 105, 105969


Implementing and Evaluating BERTopic on Open-Question Responses in Course
Evaluations

submitted in partial fulfillment for the degree of master of science

Muriël de Groot
12870331

Master Information Studies
Data Science
Faculty of Science
University of Amsterdam

2022-06-30

| First Supervisor | Second Supervisor
---|---|---
Title, Name | Dr. Mohammad Aliannejadi | Dr. Marcel Haas
Affiliation | Universiteit van Amsterdam, IR Lab | Universiteit van Amsterdam
Email |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>

###### Abstract.

Topic modeling is a widely used analytical tool for evaluating large
collections of textual data. Currently, the most popular topic modeling
technique is Latent Dirichlet Allocation, which, although flexible and
adaptive, is not optimal for less conventional texts such as short documents
or documents from various domains. This research explores whether the state-
of-the-art BERTopic algorithm performs better on short documents from various
domains, by applying the model to a corpus of open-question responses from
course evaluations of different faculties within the University of Amsterdam.
Besides the clustering algorithm HDBSCAN that BERTopic by default uses,
k-Means is investigated, which has not been researched before in combination
with BERTopic. It is found that BERTopic HDBSCAN is generalizable to a
university-wide topic model with minimal tuning, outperforming LDA in terms of
topic coherence (0.091) and diversity (0.880). Although its superior
performance, BERTopic HDBSCAN classified 74% of the student responses as
outliers, questioning the model’s suitability for the use case of university
course evaluations.

Topic modeling, LDA, BERTopic, HDBSCAN, k-Means, UMAP, topic coherence, topic
diversity

##  1\. Introduction

Due to the fast-rising availability of textual data, there is a growing demand
for tools to analyze these massive amounts of text. A popular technique that
aids in the organization, summarization, and comprehension of these large
collections of textual data is topic modeling. Topic modeling is an
unsupervised machine learning technique that is used for extracting abstract
topics from documents. Topic modeling techniques have gotten attention from
academics recently across various fields, including natural language
processing (NLP) and information retrieval (IR). Currently, the most widely
used topic model in NLP and IR is Latent Dirichlet Allocation (LDA)
(blei2003latent). Although LDA is effective at modeling conventional text
collections such as news articles, it performs poorly on less conventional
data such as short documents (zuo2016word; hong2010empirical;
zhao2011comparing; qiang2017topic). Furthermore, LDA models perform less well
when trained on documents from different domains, as different domains might
use different words to describe the same concept (zhao2014supervised;
zhuang2010collaborative). LDA models cannot understand these synonyms and the
context in which these words are used, as the models are based on the Bag-of-
Words (BoW) assumption.

Topic modeling is increasingly implemented by organizations and instances that
deal with large collections of textual data. One of these instances is the
University of Amsterdam (UvA). Each year, the UvA receives around 30.000
course evaluations from students, including responses to open questions.
Currently, the UvA applies LDA topic modeling to identify the most often
discussed topics among student responses. This requires considerable manual
work, where for each isolated batch a new LDA topic model is fitted. This lack
of generalizability is a problem that is often seen with topic models in
practice. Moreover, it is a time-consuming task to tune the hyperparameters of
LDA models, such as the number of topics that the model should generate.

BERTopic is a state-of-the-art topic modeling technique that could potentially
overcome the mentioned limitations of LDA (grootendorst2022bertopic). BERTopic
leverages pre-trained language embeddings and a class-based TF-IDF to create
dense clusters allowing for easily interpretable topics. BERTopic models are
said to require less manual tuning and to be easier generalizable to various
domains. This paper explores whether BERTopic indeed has a better performance
when dealing with short documents from different domains (faculties). Using a
corpus of open-question responses on course evaluations, BERTopic is compared
against LDA to investigate how well BERTopic performs when scaled to a
university-wide level. Besides the default HDBSCAN clustering algorithm that
BERTopic uses, it is investigated whether k-Means clustering results in better
topic modeling performance.

_Research Objectives_ The goal of this research is to create a topic model
architecture that is scalable to any faculty and requires minimal manual
tuning.

  * •

RQ I: To what extent are BERTopic HDBSCAN and k-Means superior to LDA in
extracting meaningful topics from open-question responses in course
evaluations?

  * •

RQ II: What effect does different vocabulary across faculties (a) and the
short length of documents (b) have on topic model performance?

  * •

RQ III: How can the quality of generated topics from different unsupervised
topic models be compared?

The rest of this paper is structured as follows. The next section presents
related work, followed by a background section in which the concepts behind
BERTopic are explained. Then, the methodology is described in section 4,
followed by the experimental setup of this research in section 5. The results
of the experiments are presented in section 6, and their implications are
discussed in section 7. Finally, the paper ends with a conclusion in section
8, in which the research questions are answered.

##  2\. Related Work

###  2.1. Classic Topic Modeling

Classic topic models are built on the same premise: each document consists of
a variety of topics, and each topic comprises a set of words
(liu2016overview). In other words, topic models assume that the semantics of
documents are determined by some hidden latent variables, which we refer to as
topics. The most widely used topic modeling method is LDA, which is an
extension of Probabilistic Latent Semantic Analysis (PLSA)
(hofmann2001unsupervised) that uses Dirichlet priors for the document-topic
and word-topic distributions. LDA approaches topics as a multinomial
distribution of words, which is presented to users via its top-N highest
probability words (blei2003latent).

Although LDA is a widely used topic modeling technique, it has several
limitations, two of which specifically impact topic modeling performance on
university course evaluation responses. Firstly, LDA follows the BoW
assumption, which means that the sentence structure is not modeled, with the
result that the model cannot understand the context of words in a sentence
(blei2003latent). To illustrate, LDA does not understand that the word ”apple”
in the sentence _”An apple fell from the tree”_ means something different than
in the sentence _”Steve Jobs is the founder of Apple”_. As LDA cannot capture
the difference between these synonyms, it performs less well on corpora that
contain multiple words for the same concept. This also means that LDA models
might perform worse on documents from different domains, because documents
among various domains can be different in terms of word choices and writing
styles (zhao2014supervised).

Secondly, it is challenging for LDA models to reveal topics from short
documents (hong2010empirical; zhao2011comparing; tang2014understanding). This
is because LDA groups semantically related words into a single topic using
document-level word co-occurrence information. The consequence is that the
model is sensitive to document length: the shorter the document, the less word
co-occurrence information is available (zuo2016word). Besides that, the
frequency of words plays a less discriminative role in short texts, and the
limited context makes determining the meanings of ambiguous words more
difficult (qiang2017topic).

###  2.2. Topics as Clusters of Pretrained Language Embeddings

An alternative way to look at topic modeling, which is popular among
researchers nowadays, is to generate topics as clusters of pre-trained word
and document embeddings. This way of topic modeling overcomes the BoW
limitation that LDA has. Learning word and document embeddings is based on the
distributional hypothesis, which can be described by the famous words of the
English linguist John Rupert Firth: _”You shall know a word by the company it
keeps”_ (widdowson2007jr). In other words, words with similar meanings are
used in similar contexts, making them semantically similar. The semantic
properties of these contextual word- and document embeddings can be clustered
in a high-dimensional space based on their distance. The actual generation of
topic representations differs between algorithms and will be elaborated on in
the Background section.

This new way of looking at topic modeling serves as the basis for BERTopic.
Another well-known algorithm that regards topics as clusters of embeddings is
Top2Vec (angelov2020top2vec). Currently, Top2Vec and BERTopic achieve top
performance on well-known NLP data sets such as 20 Newsgroups and BBC News
(Newsgroups20; greene06icml), with coherence scores between 0.166-0.192 as
compared to 0.014-0.058 for LDA models (grootendorst2022bertopic). Coherence
and other measures to evaluate topic model performance are discussed in the
next section.

###  2.3. Topic Model Evaluation

Evaluating and comparing topic models has always been a challenging task in
research, due to the unsupervised nature of the models. The most widely used
evaluation metric in recent topic modeling literature is topic coherence,
which computes how the top-n words of a topic are related to each other
(newman2010automatic). A topic is ’coherent’ when it displays words that tend
to occur in the same documents. There are multiple different coherence
measures (newman2010automatic; mimno2011optimizing; aletras2013evaluating),
from which the normalized pointwise mutual information (NPMI) measure
(bouma2009normalized) has the largest correlation to human judgment of how
coherent a topic is (lau2014machine). NPMI ranges from [-1, 1], where a higher
positive NPMI means that words within a topic are more related, which is
desirable. Topic coherence metrics are frequently combined with diversity
metrics (dieng2020topic; grootendorst2020bertopic). Topic diversity is defined
as the percentage of unique words in the top-n words of all topics. The
measure ranges from [0,1], where a topic diversity close to 0 indicates
redundant topics and diversity close to 1 indicates more varied topics, which
is desirable. However, since there is no ’ground truth’ in the evaluation of
unsupervised models, these measures should only be used as proxies for a topic
model’s performance.

Another method to evaluate topic model performance is by means of human
judgment (chang2009reading). However, a problem with human judgment is that it
is subjective: one annotator might judge the quality of a topic differently
from another annotator. Moreover, human judgment is unreliable when comparing
different topic modeling algorithms, as topic interpretability by humans does
not necessarily mean that the actual topic modeling task was performed well.
To illustrate, centroid-based topic representations, such as Top2Vec
(angelov2020top2vec), could result in better human evaluations than other
algorithms. This is not due to the actual topic model performance but simply
because the words in a topic that are shown are synonyms, as these are the
words close to a cluster’s centroid. Showing a topic that contains several
synonyms could result in bias as people think the topic is of higher quality.

##  3\. Background

BERTopic is a state-of-the-art topic modeling algorithm that is based on the
approach of generating topics as clusters of pretrained language embeddings.
It combines and leverages four components: pre-trained language embeddings
(1), uniform manifold approximation and projection (UMAP) dimensionality
reduction (2), hierarchical density-based spatial clustering of applications
with noise (HDBSCAN) (3), and class-based term frequency-inverse document
frequency (c-TF-IDF) (4) (grootendorst2022bertopic). The first three
components of BERTopic are inspired by other studies that approach topic
modeling as a clustering task, such as Top2Vec (angelov2020top2vec), whereas
the fourth component, the class-based TF-IDF procedure, is novel.

###  3.1. Document Embeddings

BERTopic assumes that documents containing the same topic are semantically
similar. To embed documents, sentences are converted to dense vector
representations employing the Sentence-BERT (SBERT) framework
(reimers2019sentence). To convert sentences, various pre-trained BERT language
models can be used (devlin2018bert). BERT language models are transformer-
based, which allows for the generation of context-dependent embeddings. The
language models can understand the meaning of ambiguous language because the
surrounding text around a particular word is used to establish context
(devlin2018bert). Besides BERT, various other language models can be
implemented as embeddings in BERTopic (lan2019albert; yang2019xlnet;
sanh2019distilbert; liu2019roberta).

###  3.2. UMAP

After extracting document embeddings, BERTopic uses UMAP by default, which is
an algorithm to lower the dimensionality of the embeddings (mcinnes2018umap).
Dimensionality needs to be reduced because most clustering algorithms like
HBDSCAN cannot handle high dimensionality well. Besides UMAP, scholars have
implemented other algorithms to lower the dimensionality of word and document
embeddings, such as Principal Component Analysis (PCA) (sia2020tired) and
t-distributed Stochastic Neighbor Embedding (t-SNE) (choo2013utopian).
However, an advantage of UMAP is that it keeps a significant portion of the
high-dimensional local structure in lower dimensionality, leading to a higher
clustering performance (allaoui2020considerably). Another advantage of UMAP
over t-SNE is that it scales better to very large datasets
(angelov2020top2vec).

###  3.3. Embedding Clustering

After the dimensionality of the embeddings is reduced, these vectors are
clustered into semantically similar clusters, which are regarded as topics. To
perform the clustering, BERTopic uses by default HDBSCAN (mcinnes2017hdbscan),
a density-based algorithm known to work well with UMAP (malzer2020hybrid).
HDBSCAN assigns a label to each dense cluster of document vectors and a noise
label to all document vectors that are not in a dense cluster. This means that
the algorithm is robust to outliers, as it does not force all documents
towards a certain cluster. A document is regarded as an outlier if no cluster
can be found. Creating clusters of document embeddings with HDBSCAN is known
to work well for topic modeling (angelov2020top2vec;
grootendorst2022bertopic).

Another well-known clustering algorithm is k-Means. As opposed to HDBSCAN,
k-Means is a centroid-based algorithm, which forces all data points into a
certain cluster. The result is that no outliers are generated. K specifies the
number of clusters the algorithm needs to yield, which must be specified
beforehand. The top words of a certain cluster are naturally those closest to
the cluster center, the ”centroid” (hartigan1979algorithm). Creating k-Means
clusters of document embeddings for topic modeling has been done before by
(sia2020tired). They found that density-based clustering algorithms resulted
in lower performance and more hyperparameters to tune. However, academics have
not yet evaluated how BERTopic performs when k-Means clustering is used
instead of HDBSCAN.

###  3.4. c-TF-IDF

After the embeddings have been grouped into semantically similar clusters,
BERTopic derives topics from these clustered documents using c-TF-IDF, a
class-based variant of TF-IDF (ramos2003using). c-TF-IDF considers all
documents in a given cluster to be a single document and then applies TF-IDF
to it. Then, the top-n words are chosen based on their c-TF-IDF score,
representing a specific topic. The higher the c-TF-IDF score, the more
representative a word is within that topic.

The c-TF-IDF method that BERTopic applies is new compared to other studies
that approach topic modeling as a clustering task, that generate topic
representations from a centroid-based perspective instead (sia2020tired;
angelov2020top2vec). These studies assume that words in close proximity to a
cluster’s centroid are most representative of that cluster and, thereby, a
topic. However, (grootendorst2022bertopic) states that the assumption that a
cluster always lies within a sphere around a cluster centroid is not always
valid in reality, which means that this way of creating topic representations
might be misleading.

##  4\. Methodology

###  4.1. Data Sets

The Course Evaluation Responses data set (CER) is retrieved from UvA Q, which
is the instrument used by the UvA to collect student feedback on education.
The data set consists of open-question responses to course evaluations from
five faculties at the UvA over 2020-2021 and 2021-2022. For details on the
faculties, please refer to Table 8 in the Appendix. The raw data set contains
62.522 unique responses. Of these responses, 70% are labeled as English, and
30% are labeled as Dutch.

Besides the CER data set, the 20 Newsgroups data set (20NG) is used
(Newsgroups20). This external data set comprises 11.096 news articles across
20 categories. This additional data set is chosen because the 20NG data set is
a reliable and widely used data set in NLP research that consists of clearly
identifiable topics. It serves as a benchmark data set in this research to
amplify external validity.

###  4.2. Data Cleaning

The CER data set is cleaned as follows. First, responses that do not contain
any useful information are removed from the data, such as null values or
entries like ”-” or ”none”. Moreover, by applying the _langdetect_ module
(nakatani2010langdetect) to the course evaluation responses, it was found that
certain responses were mistakenly classified to the wrong language in the data
set. As a result, the detected Dutch responses in the English labeled data and
the detected English responses in the Dutch labeled data are switched to the
correct language. Subsequently, the CER data is filtered on English responses.
The result of the data cleaning is a data set that contains 36.502 open-
question responses.

###  4.3. Document Preprocessing

The CER and 20NG data are preprocessed in the same way. See Table 1 for an
example of the preprocessing that is performed.

_Preprocessed_ : the documents are lowercased, and non-alpha numerical
characters, digits, and double spaces are removed. Moreover, words are
lemmatized, which means that words are converted to their base form, depending
on the context. The preprocessing function consists of regular expressions and
a SpaCy lemmatizer (spacy2). The lemmatizer of SpaCy is chosen because it
creates very accurate lemmas and is easy to implement as it determines the
part-of-speech tag by default, as opposed to the lemmatizer of NLTK
(bird2009natural).

_Without stopwords_ : stopwords are removed from the preprocessed responses.
The stopwords list of SpaCy (spacy2) is used as this list of stopwords is very
extensive: it includes 326 stopwords, while NLTK has a list of ’only’ 179
stopwords (bird2009natural). An extensive stopword list is desirable for LDA
models as we do not want any stopwords to appear in the topic representations.
For BERTopic, stopword removal is not necessary in general; however, it is
advised when dealing with short documents.

Table 1. Example of preprocessing steps.

Column | Response
---|---
Original response | All tutors & lecturers are very friendly.
Preprocessed | all tutor lecturer are very friendly
Without stopwords | tutor lecturer friendly

###  4.4. Data Description

The analyses in this section are based on the cleaned and fully preprocessed
course evaluation responses.

####  4.4.1. Word Cloud

To have a general overview of what words are present in the CER corpus, a word
cloud is created. The word cloud shows frequently used words, where the
biggest and boldest letters show the most frequent words in the corpus. As can
be seen in Figure 1, the most frequent words in the responses are ’course’,
’lecture’, ’time’, ’student’ and ’appreciate’. The word cloud is built on the
corpus of responses of all faculties combined.

Figure 1. Word cloud of most frequently used words across all faculties.

####  4.4.2. Detection of Different Vocabulary across Faculties

To investigate whether there are vocabulary differences between faculties, it
is counted how many times a particular word is present in the corpus of
responses from each faculty. After the frequency of words is counted, these
words are ranked, with the most commonly used word ranked first. Now that the
words are ranked per faculty, the differences in ranks can be compared through
a heat map, see Figure 2. The words among the vertical axis are words that are
present in at least 1 of the faculties’ top 10 ranked words. For ease of
readability, the words among the vertical axis are sorted on the leftmost
column, which is the FDR faculty. The brighter the color in the heat map, the
higher the word is ranked in terms of usage within a faculty.

Figure 2. Most commonly used words in responses per faculty, in ranks.

The heat map shows vocabulary differences between faculties, as different
synonyms are used for the same word. To illustrate, whereas faculties FdR and
FEB tend to use ’tutorial’, this word is rarely used at the faculty FGW.
Instead, students from FGW use the word ’class’ or ’seminar’ to describe the
same concept.

####  4.4.3. Response Length

Besides exploring differences in vocabulary, it is interesting to see the
differences in the length of responses between faculties. Figure 3 visualizes
these differences by means of box plots, where the response length is measured
with the number of words. The box plots show that the median response length
is highest for the faculty FDR (20 words), and lowest for FMG (14 words).

Figure 3. Response length in words per faculty.

###  4.5. Topic Modeling

####  4.5.1. Package

To run experiments and validate results, OCTIS (Optimizing and Comparing Topic
models is Simple) is used. OCTIS is an open-source Python package that
includes all features needed in this research: the topic modeling algorithms,
the 20NG benchmark data set, and a wide range of evaluation metrics (terragni-
etal-2021-octis).

####  4.5.2. Models

The LDA model that OCTIS has implemented is from Gensim (rehurek2011gensim;
blei2003latent). This model is used as a benchmark in this research. Besides
the benchmark model, two BERTopic models are evaluated: the default BERTopic
model that uses HDBSCAN as clustering algorithm, and an adapted BERTopic model
that uses k-Means for clustering instead (grootendorst2022bertopic). To create
embeddings for the documents, responses are converted to dense vector
representations by means of the Sentence-BERT (SBERT) framework
(reimers2019sentence). Usually, documents are split up into sentences before
feeding them into Sentence-BERT. However, as can be seen in Figure 3, the
median document length in the corpus of responses in this research is
comparable to a standard sentence length, which is 15-20 words
(cutts2020oxford). As such, the documents are kept in their original format.
For fair comparisons between the topic models, parameters are set to default.
Moreover, for consistency, a random seed is fixed for UMAP; see Table 9 in the
Appendix. The language model that is used in the BERTopic models is the _all-
mpnet-base-v2_ SBERT model. It is trained on a large and diverse data set of
over 1 billion training pairs, making it an all-round model that shows high
performance across a broad spectrum of use cases (reimers2019sentence;
grootendorst2022bertopic).

####  4.5.3. Evaluation metrics

As topic coherence is the most widely used way to validate topic models, this
measure is chosen for performance evaluation. From the multiple coherence
measures that are out there (newman2010automatic; mimno2011optimizing;
aletras2013evaluating), the normalized pointwise mutual information (NPMI)
measure is picked (bouma2009normalized), as this coherence measure has the
largest correlation to human topic coherence judgments (lau2014machine). As
discussed in section 2.4, coherence computes how the top-n words of a topic
are related to each other (newman2010automatic). For the top-n words, an n of
5 is chosen, as the top 5 words per topic allow for a fast and clear
interpretation to the user and can be easily displayed in the course
evaluation dashboard that the UvA will implement in the future. Topic
diversity is used as a second measure to evaluate topic model performance. As
discussed, topic diversity is the percentage of unique words in the top-n
words of all topics (dieng2020topic). Again, 5 is chosen for the top-n words.
It is important to note that there is no ’ground truth’ in evaluating
unsupervised models. Nevertheless, by combining the two measures mentioned
above, an indication of a topic model’s performance can be established.

##  5\. Experimental setup

This section describes the setup of the experiments that answer the research
questions. The main experiment compares the performance of the LDA, BERTopic
HDBSCAN, and BERTopic k-Means models across corpora of individual faculties,
as well as a corpus of all faculties combined (RQ I). The experiment is also
conducted on the 20NG data set, which serves as a benchmark for validation.
The two experiments that follow investigate whether vocabulary usage and
document length impacted the performance differences between models in
experiment 1 (RQ II). The last two experiments assess the impact of outlier
generation on topic model performance (RQ III).

###  5.1. Topic Model Comparison

The performance of LDA, BERTopic HDBSCAN, and BERTopic k-Means are compared in
terms of topic coherence and topic diversity. The experiment starts with five
topics, and this number is gradually increased with steps of five to a maximum
of 30, as more topics will get computationally expensive. For the parameter
settings, please refer to Table 9 in the Appendix. For each topic size, the
topic coherence and diversity are calculated. These performance measures are
calculated on the corpora of each faculty separately, as well as from all
faculties combined. In this way, it is investigated what the impact on
performance is when incorporating documents from different faculties in the
same topic model. In other words, it is evaluated how scalable the LDA and
BERTopic models are.

The following two sections test two potential reasons why the performance is
as is.

###  5.2. Impact of Different Vocabulary across Faculties

It is stated that the more difference in vocabulary between documents in a
corpus, the worse LDA models perform (zhao2014supervised). One reason is that
LDA models do not understand that one word might have multiple synonyms. This
might have consequences on the topic models that include all faculties. In
contrast, one of BERTopic’s stated strengths is that its performance is not
reliant on this vocabulary difference, as it can understand the meaning of
ambiguous language because it uses the surrounding text around a certain word
to establish context (devlin2018bert). Therefore, this experiment investigates
whether the claim of BERTopic is valid.

Figure 2 showed vocabulary differences between faculties. To explore whether
these detected vocabulary differences between faculties impact the topic
quality of a university-wide BERTopic HDBSCAN model, two dendrograms are
created, which are evaluated in the Results section. The dendrograms show the
hierarchical structure between the created topics. A ward linkage function is
used to perform the hierarchical clustering based on the cosine similarity
between topic embeddings. Long horizontal lines on the left of the dendrogram
indicate that topics are farther from each other in terms of cosine distance.
In other words, the longer the line, the stronger the topics the model has
created. Likewise, short horizontal lines on the left of the dendrogram show
weak topics, potentially indicating a difference in vocabulary rather than a
topic in itself (brown-etal-1992-class; ushioda1996hierarchical).

If the assumption holds that BERTopic can handle vocabulary differences well,
there should not be a visible difference between a dendrogram of one faculty
and a dendrogram that includes all faculties. If vocabulary influences topic
creation, we expect to see shorter lines on the left of the university-wide
BERTopic dendrogram compared to a faculty-level topic model.

###  5.3. Impact of Document Length

Another potential reason for performance differences between topic models is
document length. As discussed, LDA models use document-level word co-
occurrence information, which makes it challenging for the models to reveal
topics from short texts (hong2010empirical; zhao2011comparing;
tang2014understanding). Compared to LDA models, BERTopic has shown better
performance on short documents, such as tweets (grootendorst2022bertopic). In
fact, as BERTopic uses Sentence Transformers to create embeddings, short
documents are preferred as they result in better embeddings
(reimers2019sentence). To investigate the impact of document length on LDA and
BERTopic performance, an experiment will be conducted.

For this experiment, the 20NG data set is used. Two samples are taken from the
data set: one consists of documents with a length between 10 and 25 words, and
the other includes documents between 60 and 100 words. The sample that
consists of short documents has a median document length of 16, which is equal
to the median document length of the CER data set. In other words, this sample
of short documents from the 20NG data set represents the same document length
as the CER data set. The short- and long document samples both consist of 1680
documents.

The goal of the experiment is to test the assumption that LDA performance is
negatively impacted by short documents, whereas BERTopic’s performance is not
impacted. If the experiment confirms this, we can keep the assumption in mind
when comparing LDA and BERTopic on the CER data set. We could argue that a
potential better performance of BERTopic compared to LDA is partly because
BERTopic is less susceptible to the short nature of documents in this
research. However, it should be noted that other factors within the BERTopic
algorithm might influence a superior performance compared to LDA.

###  5.4. Impact of Outliers on Topic Model Performance

####  5.4.1. Topic Model Performance on Non-Outlier Documents

As mentioned, an important characteristic of HDBSCAN is that it generates
outliers. This means that certain documents in the corpora will not be
considered when generating topics. Moreover, it implies that for BERTopic
HDBSCAN, topic coherence and diversity will only be measured on the non-
outlier documents. This can be regarded as unfair, as LDA and k-Means do not
generate outliers, which means that all documents are taken into account when
evaluating the topic model by coherence and diversity. To establish a fairer
comparison between the topic models, new corpora are created for the CER and
20NG data set in this experiment. These two corpora only include documents
that BERTopic HDBSCAN initially regarded as non-outliers. Consequently, all
three models are applied to these non-outlier corpora, and the performance of
the topic models will be compared.

####  5.4.2. The Effect of Outliers on HDBSCAN Performance

It is possible to tune certain parameters of HDBSCAN such that fewer documents
are classified as outliers. The parameter _min cluster size_ controls the
minimum size of a cluster and thereby the number of clusters that are
generated. Increasing this value results in fewer clusters but of larger size,
whereas decreasing this value results in more micro clusters being generated.
Another parameter of HDBSCAN is _min samples_. Setting this parameter
significantly lower than _min cluster size_ can reduce the number of outliers
(grootendorst2020bertopic). In this experiment, the tuning of the parameters
as mentioned above will be used to influence the fraction of outliers that
BERTopic HDBSCAN generates on the university-wide CER corpus, see Table 10 in
the Appendix for the parameter settings. Then, for each BERTopic HDBSCAN model
with a corresponding outlier fraction, topic coherence and diversity are
calculated. The goal of this experiment is to assess to what extent the
fraction of outliers influences topic coherence and diversity scores of
BERTopic HDBSCAN.

Table 2. Results. Ranging from 5 to 30 topics with steps of 5, topic coherence (TC) and topic diversity (TD) are calculated at each step for each topic model. All results are averaged across 3 runs for each step. Hence, each score is the average of 18 separate runs. The performance difference between the baseline LDA model and the best performing BERTopic model, HDBSCAN, is denoted with δ. |  | FDR | FEB | FGW | FMG | FNWI | All faculties | Average | 20NG
---|---|---|---|---|---|---|---|---|---
LDA | TC | -0.022 | -0.020 | -0.013 | 0.025 | -0.015 | 0.031 | -0.002 | 0.057
TD | 0.358 | 0.528 | 0.414 | 0.621 | 0.500 | 0.718 | 0.523 | 0.752
BERTopic HDBSCAN | TC | 0.046 | 0.057 | 0.030 | 0.118 | 0.145 | 0.091 | 0.081 | 0.166
TD | 0.735 | 0.837 | 0.855 | 0.829 | 0.903 | 0.880 | 0.840 | 0.902
BERTopic k-Means | TC | 0.028 | 0.033 | 0.028 | 0.050 | 0.029 | 0.033 | 0.032 | 0.113
TD | 0.531 | 0.626 | 0.598 | 0.653 | 0.618 | 0.571 | 0.600 | 0.853
δ | TC | 0.068 | 0.077 | 0.043 | 0.093 | 0.160 | 0.060 | 0.083 | 0.109
TD | 0.377 | 0.309 | 0.441 | 0.208 | 0.403 | 0.162 | 0.317 | 0.150

##  6\. Results

###  6.1. Topic Model Comparison (RQ I)

As can be seen in Table 2 on the next page, the best performing topic model
across all corpora is BERTopic HDBSCAN, with an average topic coherence and
diversity score of 0.081 and 0.840, respectively. LDA has the lowest overall
performance: a topic coherence and diversity of -0.002 and 0.523,
respectively. BERTopic k-Means has an average coherence score of 0.032 and a
diversity score of 0.600.

The highest score on topic coherence and diversity on the CER corpus is
obtained with BERTopic HDBSCAN on the faculty FNWI: 0.145 and 0.903,
respectively. Conversely, the lowest score is obtained with LDA on the faculty
FdR: -0.022 and 0.358, respectively.

Another finding from Table 2 is that on the corpora of responses from all
faculties combined, BERTopic HDBSCAN scores highest: 0.091 on topic coherence
and 0.880 on topic diversity. These scores are higher than the average
performance of BERTopic HDBSCAN across all corpora. As the goal of this
project is to implement a topic model that is generalizable to multiple
faculties with minimal tuning, BERTopic HDBSCAN could be scaled to a
university-wide level, as it results in an above-average topic model
performance.

Table 2 reported the average performance ranging from 5 to 30 topics. To find
the optimal number of topics, Figure 4 shows the impact of topic count on
topic coherence and diversity scores. The figures show topic coherence and
diversity scores on the y-axis and the corresponding number of topics on the
x-axis. The lines show the mean of 3 iterations, and the standard deviation
among the iterations is shown with a spread around the line.

(a) Coherence (b) Diversity

Figure 4. Topic coherence and diversity of LDA and BERTopic models, applied on
a university-wide corpus, with a topic count ranging from [5,30].

BERTopic HDBSCAN with 25 topics results in the highest performance with
regards to topic coherence. Moreover, the highest performance with regards to
topic diversity is obtained at 5 topics; however, the spread around this point
indicates high variance among the iterations. Keeping reliability in mind, a
topic count of 15 instead results in the best performance regarding the topic
diversity of BERTopic HDBSCAN. Hence, it can be concluded that if the UvA
implements a university-wide BERTopic model based on HDBSCAN, it should
specify the number of topics between 15 and 25.

Another thing that stands out in Figure 4 b) is that BERTopic with k-Means
scores significantly lower in terms of topic diversity compared to the other
two models. A potential reason for this might be that whereas HDBSCAN can deal
with outliers, k-Means cannot, resulting in outlier documents being forced to
a particular cluster, thus topic. This might result in topic representations
that more often include certain words that are very common in the corpus.

To assess whether the actual topic representations correspond with the
mentioned coherence and diversity scores, tables 3, 4 and 5 depict the
representations for each topic model. The ”topic words” column denotes the 4
most representative words for each topic, and the column ”topic
interpretation” tries to summarize these 4 words into one theme.

Table 3. Top 10 LDA topics. Rank | Topic words | Topic interpretation
---|---|---
1 | clear, good, lecture, assignment | ?
2 | teacher, feedback, essay, thesis | Feedback
3 | exam, feedback, grade, time | Exam
4 | course, assignment, time, student | Assignment
5 | lecture, course, student, class | ?
6 | lecture, text, interesting, nice | ?
7 | student, question, course, time | ?
8 | week, course, time, work | Time
9 | course, seminar, learn, theory | Seminar
10 | course, topic, material, reading | Reading material

As can be seen in Table 3, the LDA model creates topics that are hard to
interpret. In other words, the topics are not coherent. This is in line with
the coherence scores in Figure 4 a) and Table 2.

Table 4. Top 10 BERTopic HDBSCAN topics. Rank | Topic words | Topic interpretation
---|---|---
1 | group, work, assignment, room | Group assignment
2 | online, course, lecture, student | Online lecture
3 | law, legal, case, course | Law
4 | lecture, clip, slide, powerpoint | Lecture
5 | feedback, assignment, received, grade | Feedback
6 | exam, question, time expect | Exam
7 | workload, deadline, week, time | Workload
8 | lecturer, professor, enthusiasm, great | Teacher
9 | reading, literature, read, book | Reading material
10 | course, european, political , perspective | European politics
Table 5. Top 10 BERTopic k-Means topics. Rank | Topic words | Topic interpretation
---|---|---
1 | course, topic, theory, interesting | Theory
2 | tutorial, group, online, student | Online tutorial
3 | lecture, slide, question, clear | Lecture
4 | course, structure, uva, structured | Course structure
5 | assignment, feedback, essay, grade | Assessment
6 | lecturer, teacher, subject, teaching | Teacher
7 | exam, question, exercise, test | Exam
8 | reading, literature, read, book | Reading material
9 | week, workload, deadline, assignment | Workload
10 | student, time, course, class | Time

Table 4 shows the output of BERTopic HDBSCAN. Compared to LDA, these topics
are easier to interpret, thus indicating a higher coherence score. Moreover,
the topic words are more specific than the other two models, as topics 3 and
10 describe a particular field of study. Furthermore, it should be noted that
BERTopic HDBSCAN’s output resulted in a large number of responses that were
classified as outliers, which are not visible in Table 4.

Table 5 displays the top 10 topics of BERTopic k-Means. These topics are more
general than the top 10 topics of HDBSCAN. Moreover, the topics are easily
interpretable. Another thing that stands out in Table 5 is that certain topic
words, such as ’course’, appear in many topic representations. This indicates
a low topic diversity. This finding is in line with the conclusions from
Figure 4 b) and Table 2.

###  6.2. Impact of Vocabulary Differences between Faculties (RQ IIa)

As shown in Figure 5, there is no visible difference between the two
dendrograms regarding the variation in ward linkage on the left of the
diagrams. The dendrogram in Figure 5 a) is based on the responses of the FEB
faculty, and the dendrogram in Figure 5 b) is based on a university-wide
level.

The same experiment was conducted with the other four faculties, which all
showed no visible difference compared to the dendrogram in Figure 5, see
Figure 7 in the Appendix. Hence, the dendrograms confirm the statement that
BERTopic can handle vocabulary differences across different domains.

(a) FEB faculty (b) All faculties

Figure 5. Hierarchical structure of topics generated from BERTopic HDBSCAN.

###  6.3. Impact of Document Length (RQ IIb)

Table 6 shows the results of the document length experiment. The performance
difference of the models between long and short documents is denoted with
$\delta$ in the last column. As can be seen, BERTopic k-Means is the least
susceptible to short documents. Moreover, we see an expected decline in
coherence for LDA on short documents. Surprisingly, we also see a comparable
decline in coherence with BERTopic HDBSCAN. Hence, the claim that BERTopic
HDBSCAN is better able to handle short documents compared to LDA is not
confirmed in terms of topic coherence. Another finding is that short documents
do not negatively impact topic diversity. In fact, Table 6 shows a slight
performance increase in topic diversity on the short document corpus.

Table 6. Results document length experiment on the 20NG corpus. |  | Long documents | Short documents | δ
---|---|---|---|---
LDA | TC | -0.020 | -0.180 | -0.160
TD | 0.614 | 0.656 | 0.042
BERTopic HDBSCAN | TC | 0.080 | -0.065 | -0.145
TD | 0.910 | 0.972 | 0.062
BERTopic k-Means | TC | 0.058 | -0.023 | -0.081
TD | 0.851 | 0.850 | -0.001

###  6.4. Impact of Outliers on Topic Model Performance (RQ III)

Table 7 shows the performance of the topic models on the corpora of documents
that BERTopic HDBSCAN initially did not regard as outliers. The CER corpus
consists of 9.446 non-outlier documents, and the 20NG corpus consists of 5.316
documents classified as non-outlier. The numbers between brackets denote the
performance difference compared to the original CER and 20NG corpora,
including 36.502 and 11.096 documents, respectively. As shown in the table,
LDA shows the largest decline in performance. Moreover, it is shown that the
performance of BERTopic HDBSCAN is relatively the same, while the performance
of BERTopic k-Means is improved.

Table 7. Topic model performance on the non-outlier corpora. |  | CER | 20NG
---|---|---|---
LDA | TC | 0.019 (-0.018) |  0.020 (-0.037)
TD |  0.565 (-0.153) | 0.622 (-0.130)
BERTopic HDBSCAN | TC | 0.093 (0.002) | 0.152 (-0.014)
TD | 0.864 (-0.016) | 0.875 (-0.027)
BERTopic k-Means | TC |  0.083 (0.056) | 0.138 (0.025)
TD |  0.736 (0.165) | 0.881 (0.028)

Figure 6 shows the impact of outliers on the coherence and diversity scores of
BERTopic HDBSCAN. On the x-axes, we see the fraction of documents in the
corpus classified as outliers. The rightmost point on the x-axis in the
figures denotes the default BERTopic HDBSCAN model without any tuning. The
other points along the x-axes are fractions of outliers as a result of HDBSCAN
parameter tuning. On the y-axes, the corresponding coherence and diversity
scores of the models are shown. Both figures show a positive trend, implying
that the more documents BERTopic HDBSCAN classifies as outliers, the higher
the topic models score on coherence and diversity.

(a) Coherence (b) Diversity

Figure 6. The impact of outliers on topic coherence and diversity on the
university-wide CER corpus with topics ranging from [5,30].

##  7\. Discussion

###  7.1. Topic Model Performance

The results show that combining course evaluation responses from all faculties
in one BERTopic HDBSCAN model results in above-average scores compared to
topic models based on separate faculties, see Table 2. As the goal of this
project is to implement a topic model that is generalizable to a university-
wide level, the results confirm that this is obtainable with BERTopic HDBSCAN
(RQ I).

The results of the vocabulary usage experiment (RQ IIa) reinforce the claim
that BERTopic HDBSCAN is generalizable to a university-wide level. Comparing
the hierarchical structures of a single faculty topic model with a university-
wide topic model, no visible differences in terms of ward linkage could be
found, see Figure 5. This implies that BERTopic HDBSCAN can handle vocabulary
differences between faculties well.

In contrast, the results of the document length experiment (RQ IIb) do not
confirm the claim that BERTopic HDBSCAN is better able to handle short
documents, see Table 6. Hence, the short nature of documents, which
characterizes the CER corpus, did not play a role in the superior performance
of BERTopic HDBSCAN compared to LDA. This is in contrast with
(grootendorst2022bertopic), which showed superior performance of BERTopic on
short documents compared to LDA. A potential reason for the disparity in
findings might be the use of different corpora. While
(grootendorst2022bertopic) used a corpus of tweets, this research used a
subset of short documents from the 20NG data set. Moreover, the findings in
this research are also in contrast with (qiang2017topic), which states that
word-embedding-based topic models show superior performance in extracting
topics from short texts compared to LDA. However, the structure of their
embedding-based topic model algorithm is different from BERTopic, which may
explain the inconsistency in findings.

###  7.2. How About Outliers?

Although the results confirm that it is possible with BERTopic HDBSCAN to
create a topic model that is generalizable to a university-wide level, this
finding is questionable in practice. From the corpus of all course evaluation
responses, HDBSCAN classified 74% of documents as outliers. The consequence is
that the majority of student responses are not taken into account when
creating topics. Only taking into account one-quarter of the opinions from
students might not be ethical in the case of course evaluations. Moreover, the
fact that outliers are not considered in evaluating BERTopic HDBSCAN in terms
of coherence and diversity makes it hard to compare the model to other topic
models.

In Table 7 the topic models are compared in a fairer way by creating new
corpora of non-outlier documents. What stands out is that LDA performed worse
on the corpus of non-outlier documents. This is interesting, as one could
imagine that removing noisy documents would benefit any topic model.
Nevertheless, the results show that this is not the case: what HDBSCAN regards
as noisy documents do not negatively impact LDA topic generation. Another
finding from Table 7 is that BERTopic HDBSCAN shows a comparable performance
on the non-outlier documents compared to the original corpora. This finding is
less surprising as removing outlier documents from a corpus should not
severely impact BERTopic HDBSCAN performance, as outliers are not considered
when calculating topic coherence and diversity. Lastly, what stands out from
Table 7 is that k-Means performance improved significantly on the non-outlier
corpora. This makes intuitive sense, as the inclusion or exclusion of outliers
is the main difference between BERTopic HDBSCAN and k-Means. Now, when we
exclude the outliers beforehand for k-Means, we ’simulate’ the behavior of
HDBSCAN.

The statement that HDBSCAN creates a high number of outliers, no matter what
documents are included in the corpus, can be validated with the number of
outlier documents on the carefully preprocessed and widely-used 20NG data set.
From this data set, BERTopic HDBSCAN flagged 52% of the documents as outliers.
Hence, it can be concluded that the large number of outliers is not entirely
reliant on corpus characteristics but mainly on the behavior of HDBSCAN.

###  7.3. HDBSCAN vs. k-Means

Although the robustness of HDBSCAN to outliers might be beneficial in some use
cases, we can conclude that HDBSCAN results in more outliers than desired in
the case of course evaluation responses. As was shown in Figure 6, tuning the
parameters of HDBSCAN such that fewer outliers are generated is possible;
however, the quality of the corresponding topic model decreases. Moreover, as
the goal of this project is to create a topic model architecture that needs
minimal manual tuning, HDBSCAN is also not desirable in this sense. Lastly, as
shown in Table 6, another downside of HDBSCAN is that it performs worse on
short documents, which is an essential characteristic of the CER corpus.

The above-mentioned downsides of BERTopic HDBSCAN do not hold for BERTopic
k-Means. In fact, BERTopic k-Means does not generate any outliers, does not
need manual tuning, and is more robust to short documents, see Table 6.
Moreover, as seen in Table 5, k-Means creates well interpretable topics.
However, in terms of generalizability to a university-wide corpus, BERTopic
k-Means performs less well: the model has a similar coherence and a lower
diversity score than LDA, see Table 2.

To summarize, it can be stated that there is not one superior topic model, as
each model has its strengths and weaknesses. When choosing between topic
models, one should make a trade-off between quantitative evaluation measures,
human interpretability, and the implications of outliers.

###  7.4. Limitations

A limitation of BERTopic and other models that regard topics as clusters of
pretrained language embeddings, such as Top2Vec (angelov2020top2vec), is that
no topic distributions can be generated within a single document. Instead,
each document is assigned to one single topic, like hard clustering. This is
in contrast to LDA, where a document can belong to multiple topics, which is
more of a soft clustering-like approach. In the case of course evaluation
responses, we often see that students talk about multiple topics in the same
response. For example, to the question _”What suggestions do you have for
improving this subject?”_ a certain student discusses the lecturer, the course
structure, and the reading materials in his or her response. LDA models create
a topic distribution from this response, whereas BERTopic would only assign a
single topic to the response. As can be seen in the example, the assignment of
a single topic to a document is not always realistic.

Another limitation in this research are the performance measures chosen to
evaluate topic models. As discussed, evaluating and comparing topic models is
a challenging task in research, due to the unsupervised nature of the models.
Although topic coherence and diversity give an indication, this is only one
way to look at the quality of topic models. To illustrate, the large number of
outliers that BERTopic HDBSCAN generated is not taken into account with topic
coherence and diversity, while in reality, this might be an essential factor
influencing the usability of such a model in practice. Moreover, although the
topic coherence measure NPMI has been shown to correlate with human judgment
(lau2014machine), recent research states that this might only be the case for
classical models like LDA and that this relationship might not exist with
neural topic models like BERTopic (hoyle2021automated). As such, the validity
of the measures used in this research to evaluate BERTopic is questioned.

###  7.5. Future Research

As discussed, topic model evaluation suffers from a validation gap since
coherence, originally developed for classical models like LDA, has not been
validated for neural models like BERTopic. A future research direction would
be to address this validation gap and to explore other automated metrics for
neural models that do correlate with human judgment.

With regards to the exploration of the BERTopic algorithm, future work could
further explore the potential of incorporating k-Means instead of HDBSCAN as
clustering algorithm. Alternatively, other clustering algorithms can be
investigated that are more flexible than k-Means, but result in fewer outliers
than HDBSCAN when incorporated in BERTopic. Lastly, future research could
explore the impact on topic generation of dimensionality reduction algorithms
other than UMAP, such as PCA and t-SNE.

##  8\. Conclusions

This paper explored the state-of-the-art BERTopic algorithm on open-question
responses from course evaluations, with the goal to create a topic model
architecture that is scalable to a university-wide level with minimal tuning.
Besides the default clustering algorithm HDBSCAN that is incorporated in
BERTopic, k-Means was investigated. The models were compared to a baseline LDA
model.

It was found that BERTopic HDBSCAN can successfully create a university-wide
topic model, with a topic coherence of 0.091 and a topic diversity of 0.880.
With regards to these metrics, it outperforms LDA, which has coherence and
diversity scores of 0.031 and 0.718, respectively. Moreover, differences in
vocabulary among faculties were detected; however, these vocabulary
differences did not impact the topic generation of BERTopic. Furthermore, it
was observed that the topic coherence of BERTopic declines when applied to
short documents, making the model as susceptible to document length as LDA.

A limitation of BERTopic HDBSCAN is the outlier generation that characterizes
the model. 74% of the student responses were classified as outliers, which
makes BERTopic HDBSCAN not suitable for the use case of university course
evaluations, in which each student response should count equally. In contrast,
BERTopic k-Means did not generate outliers, created well interpretable topics,
and showed to be more robust to short documents than BERTopic HDBSCAN and LDA.
However, in terms of generalizability to a university-wide corpus, BERTopic
k-Means performed less well, with a coherence of 0.032 and a diversity of
0.571.

This research has two main contributions. Firstly, it is the first time that
BERTopic is applied to course evaluation responses, characterized as short
documents originating from different domains. Secondly, it is the first time
that it is investigated what the impact is on the performance of BERTopic when
k-Means clustering is applied, rather than its default density clustering
approach, HDBSCAN.

In conclusion, it can be stated that the topic model evaluation measures used
in this research do not capture the full scope of what a topic model user
might deem as important. Besides the quantitative evaluation measures, one
should keep the consequences of outlier generation in mind. The absence of an
all-encompassing topic model performance measure and the lack of validation of
the used evaluation measures is a limitation in this research and should be
considered in future work.

## Acknowledgements

I would like to express my gratitude to my supervisors, Dr. Mohammad
Aliannejadi and Dr. Marcel Haas. They have provided me with a great deal of
support, valuable advice, and helpful suggestions during this research.

## Appendix

Table 8. UvA faculties. Faculty abbreviation | Faculty | Translation
---|---|---
FDR | Faculteit der Rechtsgeleerdheid | Amsterdam Law School
FEB | Faculteit der Economie en Bedrijfskunde | Faculty of Economics and Business
FGW | Faculteit der Geesteswetenschappen | Faculty of Humanities
FMG | Faculteit der Maatschappij en Gedrag | Faculty of Social and Behavioural Sciences
FNWI | Faculteit der Natuurwetenschappen, Wiskunde en Informatica | Faculty of Science
Table 9. Parameter settings. Parameters not mentioned in this table are set to default. Parameter | Value | Applicable model
---|---|---
nr topics | [($i+$1)$*$5 for $i$ in range(6)] | LDA, BERTopic HDBSCAN & BERTopic k-Means
embedding model | all-mpnet-base-v2 | BERTopic HDBSCAN & BERTopic k-Means
min topic size | 15 | BERTopic HDBSCAN & BERTopic k-Means
umap random state | 33 | BERTopic HDBSCAN & BERTopic k-Means
n clusters | 50 | BERTopic k-Means
Table 10. HDBSCAN parameter settings to control outlier generation. Min cluster size | Min samples | Fraction of outliers
---|---|---
5 | 5 | 0.74
10 | 2 | 0.54
40 | 2 | 0.37
80 | 2 | 0.19
120 | 1 | 0.01

(a) FDR faculty (b) FGW faculty

(c) FMG faculty (d) FNWI faculty

Figure 7. Hierarchical structure of topics generated from BERTopic HDBSCAN on
faculty level.