# A Collection of Quality Diversity Optimization Problems Derived from
Hyperparameter Optimization of Machine Learning Models

Lennart Schneider  LMU MunichLudwigstraße 33MunichGermany80337
<EMAIL_ADDRESS>,  Florian Pfisterer  LMU
MunichLudwigstraße 33MunichGermany80337<EMAIL_ADDRESS>,  Janek Thomas  LMU MunichLudwigstraße 33MunichGermany80337
<EMAIL_ADDRESS>and  Bernd Bischl  LMU MunichLudwigstraße
33MunichGermany80337<EMAIL_ADDRESS>
(2022)

###### Abstract.

The goal of Quality Diversity Optimization is to generate a collection of
diverse yet high-performing solutions to a given problem at hand. Typical
benchmark problems are, for example, finding a repertoire of robot arm
configurations or a collection of game playing strategies. In this paper, we
propose a set of Quality Diversity Optimization problems that tackle
hyperparameter optimization of machine learning models - a so far
underexplored application of Quality Diversity Optimization. Our benchmark
problems involve novel feature functions, such as interpretability or resource
usage of models. To allow for fast and efficient benchmarking, we build upon
YAHPO Gym, a recently proposed open source benchmarking suite for
hyperparameter optimization that makes use of high performing surrogate models
and returns these surrogate model predictions instead of evaluating the true
expensive black box function. We present results of an initial experimental
study comparing different Quality Diversity optimizers on our benchmark
problems. Furthermore, we discuss future directions and challenges of Quality
Diversity Optimization in the context of hyperparameter optimization.

Machine Learning, Hyperparameter Optimization, Quality Diversity Optimization,
Benchmarking

††journalyear: 2022††copyright: rightsretained††conference: Genetic and
Evolutionary Computation Conference Companion; July 9–13, 2022; Boston, MA,
USA††booktitle: Genetic and Evolutionary Computation Conference Companion
(GECCO ’22 Companion), July 9–13, 2022, Boston, MA, USA††doi:
10.1145/3520304.3534003††isbn: 978-1-4503-9268-6/22/07††ccs: Computing
methodologies Search methodologies

##  1\. Introduction

Quality Diversity Optimization (QDO) aims to generate a collection of diverse,
high-performing solutions. Classical algorithms are Novelty Search with Local
Competition (Lehman and Stanley, 2011) and MAP-Elites (Mouret and Clune,
2015), which rely on the concepts of evolutionary computation. While the
development of new QDO algorithms has seen significant progress (Fontaine et
al., 2020; Cully, 2021; Fontaine and Nikolaidis, 2021), the community lacks
larger testbeds of benchmark problems. In this paper, we propose a set of QDO
problems derived from hyperparameter optimization (HPO) of machine learning
(ML) models. So far, HPO has been an underexplored area for the application of
QDO. Our benchmark problems involve feature functions of high practical
importance, such as interpretability or resource usage of models, making the
application of QDO algorithms particularly relevant due to their illuminating
properties. As an example, consider a random forest (Breiman, 2001) in a
binary classification setting. Depending on the choice of hyperparameters, a
random forest can potentially contain many deep trees. On the one hand, we
expect such a random forest to yield good performance, on the other hand, a
random forests with thousands of deep trees is not interpretable. We could for
example be interested in relating the performance of a random forest to
interpretability measures such as the number of features the model uses or the
interaction strength of features. In general, a random forest that uses few
features and has low interaction strength is expected to perform worse than a
large and deep forest, but this strongly depends on the concrete data set at
hand. It is therefore of high interest to illuminate the relationship of
performance and interpretability and QDO algorithms are very well suited for
this task.

###  1.1. Hyperparameter Optimization

An ML learner or inducer $\mathcal{I}$ configured by hyperparameters
$\bm{\lambda}\in\Lambda$ maps a data set $\mathcal{D}\in\mathds{D}$ to a model
$\hat{f}$, i.e.,

$\mathcal{I}:\mathds{D}\times\Lambda\to\mathcal{H},(\mathcal{D},\bm{\lambda})\mapsto\hat{f}.$

HPO methods aim to identify a high-performing hyperparameter configuration
(HPC) $\bm{\lambda}\in\tilde{\Lambda}$ for $\mathcal{I}_{\bm{\lambda}}$
(Bischl et al., 2021). The so-called search space
$\tilde{\Lambda}\subset\Lambda$ is typically a subspace of the set of all
possible HPCs:
$\tilde{\Lambda}=\tilde{\Lambda}_{1}\times\tilde{\Lambda}_{2}\times\dots\times\tilde{\Lambda}_{d},$
where $\tilde{\Lambda}_{i}$ is a bounded subset of the domain of the $i$-th
hyperparameter $\Lambda_{i}$. $\tilde{\Lambda}_{i}$ can be either continuous,
discrete, or categorical. It can also dependent on other hyperparameters,
meaning that $\tilde{\Lambda}_{i}$ is only active when $\tilde{\Lambda}_{j}$
takes certain values, resulting in a possibly hierarchical search space. The
classical (single-objective) HPO problem is defined as:

(1)
$\displaystyle\bm{\lambda}^{*}\in\operatorname*{arg\,min}_{\bm{\lambda}\in\tilde{\Lambda}}\widehat{\mathrm{GE}}(\bm{\lambda}),$

i.e., the goal is to minimize the estimated generalization error. This
typically involves a costly resampling procedure that can take a significant
amount of time (see (Bischl et al., 2021) for further details).
$\widehat{\mathrm{GE}}(\bm{\lambda})$ is a black-box function, as it generally
has no closed-form mathematical representation, and analytic gradient
information is generally not available. Therefore, the minimization of
$\widehat{\mathrm{GE}}(\bm{\lambda})$ forms an _expensive black-box_
optimization problem. Furthermore, it is typically multimodal and noisy, as
$\widehat{\mathrm{GE}}(\bm{\lambda})$ is only a stochastic estimate of the
true unknown generalization error.

###  1.2. Quality Diversity Optimization

The goal of QDO is to generate a collection of diverse yet high-performing
solutions. Diversity is defined via behavioral niches on so-called feature
functions (behavior functions), whereas performance is defined on an objective
function (fitness function). In the following, we assume that a single multi-
objective function $f:\Lambda\rightarrow\mathds{R}^{m},m\geq 2$ returns both
the objective function as well as the feature functions that span the behavior
space. Without loss of generality, we assume that the first dimension returned
by $f$ is the objective function, whereas the dimensions $2,\ldots,m$ are the
feature functions. We denote by $y_{i}$ the i-th output value of
$f(\bm{\lambda})$. We make few assumptions regarding the nature of niches but
note that a uniform grid of $k$ niches is classically constructed by dividing
the behavior space into equally sized hyperrectangles (Mouret and Clune,
2015). In this case, a niche $N_{j}\subseteq\mathds{R}^{m-1}$ is simply
defined by lower ($l_{ij}$) and upper ($u_{ij}$) boundaries on each feature
function $i\in\\{2,\ldots,m\\}$, with $l_{ij}=u_{ij-1}$ for
$j\in\\{2,\ldots,k\\}$ . A point $\bm{\lambda}$ “belongs” to niche $N_{j}$ if
$\forall i\in\\{2,\ldots,m\\}:y_{i}\in[l_{ij},u_{ij})$. The best-performing
point of a niche is typically referred to as a so-called elite. In this paper,
we propose several benchmark instances for QDO problems, which we formally
introduce below.

###### Definition 1 (QDO Benchmark Problem).

A QDO benchmark problem consists of a function
$f:\Lambda\rightarrow\mathds{R}^{m},m\geq 2$ returning both the objective
function and feature function values, a bounded search space
$\tilde{\Lambda}$, and a set of $k$ behavioral niches $\\{N_{1},\ldots
N_{k}\\}$. $f(\bm{\lambda})=\bm{y}$ returns $m$ values
$\bm{y}=(y_{1},...,y_{m})^{\prime}$, where $y_{1}$ is the objective function
value and $y_{2},\ldots,y_{m}$ are feature function values. Based on the
niches and their properties (e.g., boundaries), solutions are assigned to
these niches and collected in an archive.

##  2\. Proposed Benchmarks

We propose a collection of twelve QDO benchmark problems belonging to two
different contexts: First, we are interested in illuminating interpretability
(measured by the number of features used in a model and their interaction
strength) and performance trade-offs of ML models. Second, we examine
performance trade-offs with respect to model size (memory requirement) and
inference time (i.e., time required to make a prediction). As ML models, we
select random forest (Breiman, 2001) and extreme gradient boosting (XGBoost)
models (Chen and Guestrin, 2016) in binary classification settings. As an
objective function, we choose the classification error in the case of
formulating a minimization problem or the accuracy (1 - classification error)
in the case of formulating a maximization problem. In the following, we assume
that the accuracy is to be maximized. Our benchmark problems rely on YAHPO Gym
(Pfisterer et al., 2021), a recently proposed open source benchmarking suite
for HPO. YAHPO Gym provides so-called surrogate-based benchmarks: For a given
learner and a given data set, a large amount of HPCs (typically sampled
uniformly at random from the search space) were evaluated, and surrogate
models have been fitted on these data. When optimizing a benchmark problem
(instance) in YAHPO Gym, the costly real black-box evaluation of an HPC is
skipped, and instead, the surrogate model’s prediction is returned. This
allows for efficient benchmarking of HPO problems, as YAHPO Gym typically
requires as little as one millisecond for predicting the performance metric of
an HPC and induces minimal memory overhead (Pfisterer et al., 2021). In
general, performance estimation of ML models is noisy. YAHPO Gym can handle
both scenarios of providing a deterministic or noisy HPO benchmark. In the
following, we employ YAHPO Gym in the deterministic setting and leave the
construction of noisy QDO problems to future research.

We build upon YAHPO Gym’s iaml_ranger and iaml_xgboost benchmark scenarios. A
scenario is a collection of benchmark instances that share the same learner
and search space but contain multiple data sets. Both iaml_ranger and
iaml_xgboost allow for HPO on four different data sets with the following
OpenML (Vanschoren et al., 2013) IDs: 41146, 40981, 1489, 1067111These IDs
correspond to OpenML data set ids through which data set properties can be
queried via https://www.openml.org/d/<id>..

###  2.1. Accuracy and Interpretability

Eight of our proposed QDO benchmark problems belong to the context of
performance and interpretability. We suggest benchmarking all instances of the
iaml_ranger and iaml_xgboost scenarios because previous benchmarks on YAHPO
Gym have shown that these are interesting problems with relevant search
spaces. The search spaces (genotype spaces) are given in Table 1 and Table 2.
These search spaces are specifically tailored for existing QDO algorithms and
implementations, which often do not support categorical parameters or
hierarchical dependencies but only numeric parameters. We discuss this
limitation in Section 3. The objective function is given by the accuracy that
is to be maximized. The behavior space is spanned by two feature functions:
the number of features (NF) used by a model and their interaction strength
(IAS). These measures were proposed to quantify interpretability for any ML
model in a model-agnostic way (Molnar et al., 2020). A feature is regarded as
being used by a model if changing the feature changes the prediction of the
model. NF is then estimated via a sampling procedure as described in Algorithm
1 of (Molnar et al., 2020). The IAS is based on accumulated local effects
(ALE) (Apley and Zhu, 2020) of a model and is given by the scaled
approximation error between the ALE main effect model (the sum of first-order
ALE effects) and the prediction function of the model. For more details, see
Section 3.2 of (Molnar et al., 2020). A model with low NF and IAS is more
interpretable than a model with high NF or high IAS, but optimal performance
is often reflected in either a high NF or IAS or both. However, this strongly
depends on the data set at hand; as ML models can always overfit, resulting in
bad generalization performance, simply opting for models having a high NF and
IAS is prone to poor performance. It is therefore a priori unknown whether low
or high NF / IAS will result in good performance, making the application of
QDO algorithms particularly attractive due to the illuminating aspect of QDO.
In line with most QDO algorithms and their implementations, we propose to
define niches, using a discrete uniform grid archive with ranges $[0,p]$ for
NF and $[0,1]$ for IAS. Here, $p$ is the number of features present in a data
set (e.g., $14$ for data set 40981). We propose to use a bin size of $p+1$ for
NF (i.e., models in each bin use exactly $0,1,\dots$ or $p$ features) and
$100$ for IAS, resulting in $2100$, $1500$, $600$, and $2200$ niches overall
for the data sets above. We summarize all benchmark problems in Table 3. In
Figure 1, we visualize the exemplary iaml_ranger_40981 problem via a heatmap
of the accuracy (color) of different elites, which allows for visually
inspecting the alignment of the feature functions and objective function. We
present similar heatmaps for all remaining benchmark problems in Appendix A.

###  2.2. Accuracy and Resource Usage

Four of our proposed QDO problems belong to the context of performance and
resource usage. We suggest benchmarking all instances of the iaml_ranger
scenario. The search space is again given in Table 1. The objective function
is given by the accuracy that is to be maximized. The behavior space is
spanned by two feature functions: the size of the model if stored on disk (in
MB) and the time required to make a prediction on the same data set the model
has been trained on (in seconds). These measures are typically of interest in
the context of deploying ML models in production environments on different
hardware with varying restrictions regarding RAM and latency of predictions
(Nan et al., 2016). A larger random forest typically consists of plenty and
deep trees, and prediction time is both affected by the depth of the
individual trees and the number of trees in the forest. In general, a large
random forest with deep trees should result in a better performance, but this
again depends on the data set at hand. As before, we propose to define niches
via a discrete uniform grid archive (for more details, see Table 3). In Figure
1, we visualize the exemplary iaml_ranger_41146 problem via a heatmap of the
accuracy (color) of different elites, which allows for visually inspecting the
alignment of the feature functions and objective function. We present similar
heatmaps for all remaining benchmark problems in Appendix A.

Table 1. Search space of iaml_ranger scenarios. Hyperparameter | Type | Range | Trafo
---|---|---|---
num.trees | integer | [1, 2000] |
mtry.ratio | continuous | [0, 1] |
min.node.size | integer | [1, 100] |
sample.fraction | continuous | [0.1, 1] |
Table 2. Search space of iaml_xgboost scenarios.

Hyperparameter | Type | Range | Trafo
---|---|---|---
alpha | continuous | [1e-04, 1000] | log
lambda | continuous | [1e-04, 1000] | log
nrounds | integer | [3, 2000] | log
subsample | continuous | [0.1, 1] |
colsample_bylevel | continuous | [0.01, 1] |
colsample_bytree | continuous | [0.01, 1] |
eta | continuous | [1e-04, 1] | log
gamma | continuous | [1e-04, 7] | log
max_depth | integer | [1, 15] |
min_child_weight | continuous |  [$\exp(1)$, 150] | log

  * •

”log” in the Trafo column indicates that this parameter is optimized on a
logarithmic scale, i.e., the range is given by
$[\log(\mathrm{lower}),\log(\mathrm{upper})]$, and values are re-transformed
via the exponential function prior to evaluating the HPC.

Table 3. Summary of our QDO benchmark problems.

Problem | Learner | Data set | Search space | Objective | Features | Niches
---|---|---|---|---|---|---
Illuminating interpretability |  |  |  |  |  |
iaml_ranger_41146 | rf | 41146 | Table 1 | acc | NF + IAS |  uniform on $[0,20]\times[0,1]$ $21,100$ bins
iaml_ranger_40981 | rf | 40981 | Table 1 | acc | NF + IAS |  uniform on $[0,14]\times[0,1]$ $15,100$ bins
iaml_ranger_1489 | rf | 1489 | Table 1 | acc | NF + IAS |  uniform on $[0,5]\times[0,1]$ $6,100$ bins
iaml_ranger_1067 | rf | 1067 | Table 1 | acc | NF + IAS |  uniform on $[0,21]\times[0,1]$ $22,100$ bins
iaml_xgboost_41146 | XGBoost | 41146 | Table 2 | acc | NF + IAS |  uniform on $[0,20]\times[0,1]$ $21,100$ bins
iaml_xgboost_40981 | XGBoost | 40981 | Table 2 | acc | NF + IAS |  uniform on $[0,14]\times[0,1]$ $15,100$ bins
iaml_xgboost_1489 | XGBoost | 1489 | Table 2 | acc | NF + IAS |  uniform on $[0,5]\times[0,1]$ $6,100$ bins
iaml_xgboost_1067 | XGBoost | 1067 | Table 2 | acc | NF + IAS |  uniform on $[0,21]\times[0,1]$ $22,100$ bins
Illuminating resource usage |  |  |  |  |  |
iaml_ranger_41146 | rf | 41146 | Table 1 | acc | rm + tp |  uniform on $[1,200]\times[0.19,4.5]$ $33,33$ bins
iaml_ranger_40981 | rf | 40981 | Table 1 | acc | rm + tp |  uniform on $[1,40]\times[0.10,0.65]$ $33,33$ bins
iaml_ranger_1489 | rf | 1489 | Table 1 | acc | rm + tp |  uniform on $[1,200]\times[0.19,4.5]$ $33,33$ bins
iaml_ranger_1067 | rf | 1067 | Table 1 | acc | rm + tp |  uniform on $[1,78]\times[0.13,1.55]$ $33,33$ bins

  * •

”rf” = random forest. ”acc” = accuracy. ”rm” = rammodel. ”tp” = timepredict.
Niches of the uniform grid archive are constructed by dividing each dimension
of the feature space as indicated in the Niches column into equally spaced
bins, forming hyperrectangles.

Figure 1. Heatmaps of the iaml_ranger_40981 interpretability (left) and
iaml_ranger_41146 resource usage (right) benchmark problems. The color
gradient represents the accuracy (objective function), with black indicating
lowest accuracy and bright yellow/white indicating highest accuracy.

###  2.3. Benchmark Experiments and Results

We benchmark a basic variant of MAP-Elites (Mouret and Clune, 2015), a basic
variant of CMA-ME (Fontaine et al., 2020), and Random Search (i.e., sampling
HPCs uniformly at random). To illustrate the potential of fast prototyping of
new methods, we also include an optimizer that is a mix between MAP-Elites and
CMA-ME, where half of the evaluations of a batch are proposed using a Gaussian
emitter and the other half using an improvement emitter, which we abbreviate
as “Gauss.+Imp.”. We implement all optimizers in pyribs (Tjanaka et al.,
2021). All optimizer use a batch size of $100$ and run for $1000$ iterations,
resulting in $100000$ evaluations in total. For MAP-Elites, we use
$\sigma_{0}=0.1$ as the standard deviation of the Gaussian distribution. For
CMA-ME, we set $\sigma_{0}=0.1$ as the initial step size and use “filter“ as a
selection rule. All optimizers use a single emitter. We want to note that we
did not perform any tuning of hyperparameters of the optimizers themselves.
Therefore, results are preliminary and better performance of an optimizer
could be obtained by more carefully chosen design choices. All runs are
replicated ten times with different random seeds. We always normalize all
parameters of the search space to the unit cube and optimize within these
normalized bounds (during evaluation of an HPC, parameters are then re-
transformed to their original scale). Note that integer-valued parameters are
treated as continuous. However, prior to evaluating an HPC, those parameter
values are then rounded to the nearest integer. As performance metrics, we
report the coverage (percent of niches occupied with a solution), the QD-Score
(Pugh et al., 2016) (sum of best accuracies of occupied niches), and the
overall best accuracy found. Results for the interpretability context are
given in Table 4 for the iaml_ranger scenarios and in Table 5 for the
iaml_xgboost scenarios. Results for the resource usage context are given in
Table 6. We also visualize the anytime QD-Score in Figure 2, Figure 3, and
Figure 4. In general, MAP-Elites shows the strongest performance but is
sometimes outperformed by the Gauss.+Imp. optimizer. The performance of CMA-ME
varies strongly between benchmark problems, whereas MAP-Elites and Gauss.+Imp.
appear to be more consistent in their performance. Random Search performs
overall poorly, indicating that structural information of the problems can be
efficiently leveraged by more sophisticated optimizers. We release all our
code for running the benchmarks and analyzing results via the following GitHub
repository: https://github.com/slds-lmu/qdo_yahpo.

Table 4. Benchmark results on iaml_ranger (interpretability).

| data set: 41146 |  |  | data set: 40981 |  |
---|---|---|---|---|---|---
Algorithm | Coverage % | QD-Score | Max Acc | Coverage % | QD-Score | Max Acc
MAP-Elites | 58.95 | 1124.69 | 0.9444 | 80.91 | 1026.84 | 0.8747
CMA-ME | 31.43 | 604.66 | 0.9444 | 79.03 | 1007.62 | 0.8745
Gauss.+Imp. | 64.61 | 1235.72 | 0.9444 | 80.63 | 1026.35 | 0.8746
Random | 9.69 | 188.52 | 0.9443 | 29.39 | 378.26 | 0.8746
| data set: 1489 |  |  | data set: 1067 |  |
Algorithm | Coverage % | QD-Score | Max Acc | Coverage % | QD-Score | Max Acc
MAP-Elites | 34.73 | 175.24 | 0.9112 | 58.10 | 1083.08 | 0.8706
CMA-ME | 14.57 | 77.41 | 0.9109 | 43.87 | 823.86 | 0.8700
Gauss.+Imp. | 30.83 | 155.62 | 0.9111 | 57.57 | 1080.00 | 0.8705
Random | 17.63 | 92.51 | 0.9109 | 10.22 | 191.19 | 0.8705

  * •

Averaged over 10 replications. ”Max Acc” = maximum accuracy. Best performance
is highlighted in bold.

Table 5. Benchmark results on iaml_xgboost (interpretability).

| data set: 41146 |  |  | data set: 40981 |  |
---|---|---|---|---|---|---
Algorithm | Coverage % | QD-Score | Max Acc | Coverage % | QD-Score | Max Acc
MAP-Elites | 47.94 | 847.67 | 0.9446 | 26.00 | 333.70 | 0.8870
CMA-ME | 45.38 | 802.77 | 0.9444 | 24.61 | 314.66 | 0.8849
Gauss.+Imp. | 47.03 | 831.91 | 0.9446 | 30.76 | 391.40 | 0.8868
Random | 20.10 | 364.31 | 0.9432 | 14.07 | 179.34 | 0.8822
| data set: 1489 |  |  | data set: 1067 |  |
Algorithm | Coverage % | QD-Score | Max Acc | Coverage % | QD-Score | Max Acc
MAP-Elites | 38.85 | 200.69 | 0.9147 | 42.90 | 810.19 | 0.8824
CMA-ME | 37.92 | 194.44 | 0.9132 | 38.47 | 728.03 | 0.8814
Gauss.+Imp. | 43.52 | 223.11 | 0.9147 | 41.55 | 785.12 | 0.8821
Random | 22.55 | 114.04 | 0.9052 | 18.19 | 342.40 | 0.8736

  * •

Averaged over 10 replications. ”Max Acc” = maximum accuracy. Best performance
is highlighted in bold.

Table 6. Benchmark results on iaml_ranger (resource usage).

| data set: 41146 |  |  | data set: 40981 |  |
---|---|---|---|---|---|---
Algorithm | Coverage % | QD-Score | Max Acc | Coverage % | QD-Score | Max Acc
MAP-Elites | 41.50 | 422.99 | 0.9444 | 29.44 | 279.65 | 0.8749
CMA-ME | 38.82 | 395.85 | 0.9444 | 27.29 | 259.23 | 0.8749
Gauss.+Imp. | 41.36 | 421.55 | 0.9444 | 29.30 | 278.35 | 0.8749
Random | 31.25 | 318.20 | 0.9443 | 24.92 | 236.56 | 0.8746
| data set: 1489 |  |  | data set: 1067 |  |
Algorithm | Coverage % | QD-Score | Max Acc | Coverage % | QD-Score | Max Acc
MAP-Elites | 44.52 | 435.75 | 0.9112 | 41.04 | 387.44 | 0.8709
CMA-ME | 42.84 | 419.21 | 0.9112 | 39.53 | 373.20 | 0.8709
Gauss.+Imp. | 44.34 | 434.06 | 0.9112 | 40.77 | 384.92 | 0.8709
Random | 36.74 | 358.33 | 0.9109 | 38.06 | 358.90 | 0.8705

  * •

Averaged over 10 replications. ”Max Acc” = maximum accuracy. Best performance
is highlighted in bold.

Figure 2. Anytime QD-Score for the iaml_ranger benchmarks (interpretability).
Averaged over 10 replications. Ribbons represent standard errors. Figure 3.
Anytime QD-Score for the iaml_xgboost benchmarks (interpretability). Averaged
over 10 replications. Ribbons represent standard errors. Figure 4. Anytime QD-
Score for the iaml_ranger benchmarks (resource usage). Averaged over 10
replications. Ribbons represent standard errors.

##  3\. Outlook

The benchmark problems proposed in this paper pose novel applications for QDO
methods. QDO is interesting for HPO problems in contexts where users are
interested in criteria that go beyond traditional performance metrics. This
can be highly relevant in the context of ML models that should be
interpretable to a certain degree or ML models that must be deployed across a
diverse set of systems, ranging from FPGAs to compute clusters. We propose
several such benchmarks but hope that HPO problems for QDO can go far beyond
what is proposed in the context of this work. Furthermore, we identify two
properties of existing HPO algorithms and implementation that might help to
drive greater adaptation of QDO methods.

###  3.1. Mixed Search Spaces

Available implementations of QDO methods, such as pyribs, consider only
continuous search spaces. In many practical applications, search spaces
include categorical and conditionally active hyperparameters – so-called
hierarchical, mixed search spaces (Thornton et al., 2013). While existing
methods can theoretically be extended to such spaces, a lack of available
implementations might inhibit adoption of QDO methods. Simultaneously, this
might also open up a richer set of readily available problem instances, and
our proposed benchmarks can be trivially extended to include such search
spaces 222for the full search spaces of our benchmark problems, see (Pfisterer
et al., 2021).

###  3.2. Overlapping Niches

Similarly, while not formally required, niches in the QDO literature have
traditionally been defined as being disjoint, i.e., a point can only belong to
a single niche. Real-world applications in the context of HPO provide an
interesting field of study for methods that go beyond those assumptions. As an
example, consider again the setting where an ML model should be deployed
across diverse hardware. In this context, niches could overlap, as a smaller
model (e.g., fitting on an FPGA or microcontroller) is always also a candidate
for larger devices (e.g., fitting on a laptop or workstation).

The field of hardware-aware neural architecture search (Benmeziane et al.,
2021) seeks to find neural network architectures that are optimal for specific
hardware. If such models are now required for a variety of different hardware
architectures, QDO could be applied to find models for each of those niches
while simultaneously exploiting similarities between architectures. In this
setting, niches cannot be defined on simple feature functions, as a solution
can fall into multiple niches that defy definition based on simple rules.

Such applications form an interesting new playground for QDO methods,
especially when considering that HPO problems are usually considered to be
_expensive_. This might provide a valuable new avenue for future method
development towards approaches that require fewer function evaluations (Kent
and Branke, 2020) or that can efficiently make use of evaluations at multiple
fidelity levels (Li et al., 2017).

###### Acknowledgements.

The authors of this work take full responsibilities for its content. Lennart
Schneider is supported by the Bavarian Ministry of Economic Affairs, Regional
Development and Energy through the Center for Analytics - Data – Applications
(ADACenter) within the framework of BAYERN DIGITAL II (20-3410-2-9-8). This
work was supported by the German Federal Ministry of Education and Research
(BMBF) under Grant No. 01IS18036A.

## References

  * (1)
  * Apley and Zhu (2020) Daniel W. Apley and Jingyu Zhu. 2020.  Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ 82, 4 (2020), 1059–1086.
  * Benmeziane et al. (2021) Hadjer Benmeziane, Kaoutar El Maghraoui, Hamza Ouarnoughi, Smail Niar, Martin Wistuba, and Naigang Wang. 2021.  Hardware-Aware Neural Architecture Search: Survey and Taxonomy. In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_ , Zhi-Hua Zhou (Ed.). 4322–4329.
  * Bischl et al. (2021) Bernd Bischl, Martin Binder, Michel Lang, Tobias Pielok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Marc Becker, Anne-Laure Boulesteix, Difan Deng, and Marius Lindauer. 2021\.  Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges.  _arXiv:2107.05847 [cs, stat]_ (2021).
  * Breiman (2001) Leo Breiman. 2001\.  Random Forests.  _Machine Learning_ 45, 1 (2001), 5–32.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  XGBoost: A Scalable Tree Boosting System. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. 785–794.
  * Cully (2021) Antoine Cully. 2021\.  Multi-Emitter MAP-Elites: Improving Quality, Diversity and Data Efficiency with Heterogeneous Sets of Emitters. In _Proceedings of the 2021 Genetic and Evolutionary Computation Conference_. 84–92.
  * Fontaine and Nikolaidis (2021) Matthew C. Fontaine and Stefanos Nikolaidis. 2021.  Differentiable Quality Diversity. In _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. 10040–10052.
  * Fontaine et al. (2020) Matthew C. Fontaine, Julian Togelius, Stefanos Nikolaidis, and Amy K. Hoover. 2020.  Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space. In _Proceedings of the 2020 Genetic and Evolutionary Computation Conference_. 94–102.
  * Kent and Branke (2020) Paul Kent and Juergen Branke. 2020.  BOP-Elites, a Bayesian Optimisation Algorithm for Quality-Diversity Search.  _arXiv:2005.04320 [cs, math, stat]_ (2020).
  * Lehman and Stanley (2011) Joel Lehman and Kenneth O. Stanley. 2011.  Evolving a Diversity of Virtual Creatures Through Novelty Search and Local Competition. In _Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation_. 211–218.
  * Li et al. (2017) Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2017.  Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization.  _The Journal of Machine Learning Research_ 18, 1 (2017), 6765–6816.
  * Molnar et al. (2020) Christoph Molnar, Giuseppe Casalicchio, and Bernd Bischl. 2020\.  Quantifying Model Complexity via Functional Decomposition for Better Post-Hoc Interpretability. In _Machine Learning and Knowledge Discovery in Databases_ , Peggy Cellier and Kurt Driessens (Eds.). 193–204.
  * Mouret and Clune (2015) Jean-Baptiste Mouret and Jeff Clune. 2015.  Illuminating Search Spaces by Mapping Elites.  _arXiv:1504.04909 [cs, q-bio]_ (2015).
  * Nan et al. (2016) Feng Nan, Joseph Wang, and Venkatesh Saligrama. 2016\.  Pruning Random Forests for Prediction on a Budget. In _Proceedings of the 30th International Conference on Neural Information Processing Systems_. 2342–2350.
  * Pfisterer et al. (2021) Florian Pfisterer, Lennart Schneider, Julia Moosbauer, Martin Binder, and Bernd Bischl. 2021\.  YAHPO Gym – An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization.  _arXiv:2109.03670 [cs, stat]_ (2021).
  * Pugh et al. (2016) Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. 2016\.  An Extended Study of Quality Diversity Algorithms. In _Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion_. 19–20.
  * Thornton et al. (2013) Chris Thornton, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. 2013.  Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms. In _Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. 847–855.
  * Tjanaka et al. (2021) Bryon Tjanaka, Matthew C. Fontaine, Yulun Zhang, Sam Sommerer, Nathan Dennler, and Stefanos Nikolaidis. 2021.  pyribs: A Bare-Bones Python Library for Quality Diversity Optimization.  https://github.com/icaros-usc/pyribs.
  * Vanschoren et al. (2013) Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luís Torgo. 2013.  OpenML: Networked Science in Machine Learning.  _SIGKDD Explor._ 15, 2 (2013), 49–60.

##  Appendix A Heatmaps for all Benchmark Problems

In this section, we provide heatmaps for all our benchmark problems. Three
Gaussian emitters with a standard deviation of $0.1$, $0.2$ or $0.3$ and one
random emitter were run each with a batch size of $25$ for $10000$ iterations
resulting in $1000000$ evaluations in total. The color gradient represents the
accuracy (objective function), with black indicating lowest accuracy and
bright yellow/white indicating highest accuracy. Figure 5 shows heatmaps for
the iaml_ranger interpretability benchmark problems. Figure 6 shows heatmaps
for the iaml_xgboost interpretability benchmark problems. Figure 7 shows
heatmaps for the iaml_ranger resource usage benchmark problems.

Figure 5. Heatmaps of the iaml_ranger interpretability benchmark problems.

Figure 6. Heatmaps of the iaml_xgboost interpretability benchmark problems.

Figure 7. Heatmaps of the iaml_ranger resource usage benchmark problems.

##  Appendix B Technical Details

Benchmark experiments were conducted on a single Intel Core i7-10510U CPU and
took around 7 hours in total. YAHPO Gym v1.0 was used. For more details on how
the iaml_ranger and iaml_xgboost benchmark scenarios of YAHPO Gym were
constructed, please see (Pfisterer et al., 2021).


# AI-Cybersecurity Education Through Designing
AI-based Cyberharassment Detection Lab

Ebuka Okpala1, Nishant Vishwamitra2, Keyan Guo3, Song Liao1, Long Cheng1,
Hongxin Hu3, Xiaohong Yuan4, Jeannette Wade5 and Sajad Khorsandroo4

###### Abstract

Cyberharassment is a critical, socially relevant cybersecurity problem because
of the adverse effects it can have on targeted groups or individuals. While
progress has been made in understanding cyber-harassment, its detection,
attacks on artificial intelligence (AI) based cyberharassment systems, and the
social problems in cyberharassment detectors, little has been done in
designing experiential learning educational materials that engage students in
this emerging social cybersecurity in the era of AI. Experiential learning
opportunities are usually provided through capstone projects and engineering
design courses in STEM programs such as computer science. While capstone
projects are an excellent example of experiential learning, given the
interdisciplinary nature of this emerging social cybersecurity problem, it can
be challenging to use them to engage non-computing students without prior
knowledge of AI. Because of this, we were motivated to develop a hands-on lab
platform that provided experiential learning experiences to non-computing
students with little or no background knowledge in AI and discussed the
lessons learned in developing this lab. In this lab used by social science
students at North Carolina A&T State University across two semesters (spring
and fall) in 2022, students are given a detailed lab manual and are to
complete a set of well-detailed tasks. Through this process, students learn AI
concepts and the application of AI for cyberharassment detection. Using pre-
and post-surveys, we asked students to rate their knowledge or skills in AI
and their understanding of the concepts learned. The results revealed that the
students moderately understood the concepts of AI and cyberharassment.
Comparing the learning experiences of students in the Spring and Fall
semesters using the post-surveys, in the Spring semester, while students
understood the purpose of detecting cyberharassment, their knowledge of how AI
works, state-of-the-art (SOTA) cyber-harassment detectors and automated
cyberharassment detection did not improve. After refining the lab using the
student feedback from the Spring semester, students’ knowledge of AI and
automated cyberharassment detection significantly improved. We learn that
supplementing the hands-on lab with a theoretical background lecture improves
understanding of the lab contents, indicating that supplementing experiential
learning with theoretical constructs improves the student learning experience.
These findings confirm that the developed lab is viable for teaching AI-driven
socially relevant cybersecurity to non-computing students and can be used by
other institutions.

##  1 Introduction

With the rise of social media, cyberharassment (_e.g._ , cyberbullying and
cyberhate) has become more prevalent in daily interactions (Thomas et al.
2021). It often involves inappropriate online behavior and deliberate cyber
threats against individuals (such as teenagers (Swearer 2013)), or specific
social groups on the grounds of characteristics such as race, sexual
orientation, gender, or religious affiliation (Wachs, Wright, and Vazsonyi
2019). Cyberharassment is identified as a critical socially-relevant
cybersecurity problem (Social-Cybersecurity 2019a, b), since it can have
significant negative impacts on the safety and emotional well-being of
targeted groups, especially teens and minority communities. The Cyberbullying
Research Center’s research reported that 37% of middle and high school
students have been cyberbullied during their lifetime (Patchin 2019), and this
number is expected to further increase as teens continue to have an increased
online presence. Cyberharassment can even result in catastrophic consequences
of increased suicide among the affected teens who are unable to appropriately
get away from the harassment (Ducharme 2017). The shift from traditional text-
based cyberharassment to multimodal (_i.e._ , both texts and images)
(Challenge 2020) cyberharassment poses a challenge to effective
cyberharassment detection.

Artificial Intelligence (AI)/Machine Learning (ML) has immense potential to
solve this critical problem. Automatic detection methods of both text-based
and image-based cyberbullying using AI techniques have emerged (Zhong et al.
2016). Internet companies such as Facebook and Google have also deployed AI
algorithms to detect toxic content on social media (Facebook 2023; Google-
Jigsaw 2019). Meanwhile, adversaries may exploit vulnerabilities of AI-based
classifiers to evade existing cyberharassment detectors (Goodfellow, Shlens,
and Szegedy 2015; Li et al. 2019; Zhang et al. 2020). There exist social
problems, such as fairness and ethics, in AI models for cyberharassment
detection. For example, some particular demographic groups are unfairly
treated by AI-based detectors (Sap et al. 2019; Okpala et al. 2022). Concerns
have been raised that the vulnerabilities of AI models as well as the
robustness against attacks are biased towards underrepresented groups (Nanda
et al. 2020). As such, an unfair AI-based cyberharassment detection system may
perpetuate and aggravate existing prejudices and inequalities in society.

As cyberharassment grows online, particularly on social media, there is a need
to equip computing students with the AI skills and knowledge required to
design and develop AI-based systems to detect and remove cyberharassment. As
the field of cyberharassment is interdisciplinary, to develop better detection
systems, non-computing students, especially social science students, need to
have a general understanding of AI and how it is being used in detecting
cyberharassment. A major concern with University training is how siloed it can
be and how challenging it is for young adults to truly explore their options.
Interventions such as the one outlined here would enable effective
collaboration between Computer science and social science students and
researchers to create better cyberharassment detection systems. It is,
therefore, imperative to teach AI-based cyberharassment in universities to
both computing and non-computing students. This intervention also benefits
social science students by providing insights into how they can address social
problems using computer science, a discipline they may have otherwise had no
exposure. This practical application of their social scientific passions may
influence the degree minors or graduate programs students pursue moving
forward as more researchers are needed in this intersectional area since this
is a growing area of concern.

To teach and engage students in learning cybersecurity and AI-related topics
such as data science, instructors have adopted a wide range of pedagogical
methods such as flipped classroom (Nwokeji, Stachel, and Holmes 2019; Eybers
and Hattingh 2016), project-based learning (Nwokeji and Frezza 2017; Velaj et
al. 2022), gamification (Matovu et al. 2022), among others. Experiential
learning has been regarded as one of the best ways to train future engineers
by engineering educators (Samavedham and Ragupathi 2012). Towards this end,
experiential learning could be used in teaching AI socially relevant
cybersecurity to non-computing students.

Experiential learning, simply put, is learning from experience or learning by
doing. More formally, experiential learning is a type of active learning where
students learn through experience (Kolb and Kolb 2005; Kolb 2014).
Experiential learning is active rather than passive. Instructors have
recognized how instrumental experiential learning is in providing students
with valuable hands-on experience in an AI/ML-related field such as data
science (Barman et al. 2022; Allen 2021; Rosenthal and Chung 2020; Anderson et
al. 2014). In our study, we teach AI-based socially relevant cybersecurity for
cyberharassment detection for two semesters using a hands-on experiential lab.
Before the introduction of the lab, a questionnaire was used to rate the AI
skills and knowledge of the students. After the end of the lab, another
questionnaire was used to ask the students to rate their skills and knowledge
of AI and AI-based cyberharassment detection covered in the lab.

Our analysis and statistical results (using sample t-test) showed that
experiential learning engages students in learning AI-based cyberharassment,
and it is viable for teaching AI skills to non-computing students. Also, our
findings show that having a theoretical lecture before the experiential lab
improves understanding of the lab contents. These findings confirm that the
developed lab is viable for teaching AI-driven socially relevant cybersecurity
to non-computing students and can be used in other institutions.

##  2 Related Work

Instructors have mainly employed active learning paradigms such as
experiential learning to teach AI/ML-related courses, mostly in engineering
and Computer Science. The shortcoming of standard pedagogical methods in data
science in online courses and data science specializations are detailed in
(Serrano et al. 2017). Noting that experiential learning mitigates the
shortcomings as it focuses on problems to be solved instead of on specific
methods being used. Using the experiential learning style theory introduced in
(Kolb 2014), they developed a framework to create experiences in a deep
learning course. Finally, they review dataset repositories used in data
science and propose requirements for an experiential learning platform to
offer experiences.

Understanding the importance of capstone projects in data science courses,
(Allen 2021) observed that more attention needs to be paid to how these
projects or curriculum are structured. In their work, they develop an
interdisciplinary, client-sponsored capstone program in data science and
machine learning. In the program, students from different undergraduate and
graduate degree programs engage in experiential learning by completing a
large-scale data science or machine learning capstone project toward the
program’s end— the projects were framed to be challenging and encompass all
aspects of data science. The curriculum was split into modules focusing on the
data science pipeline, ethics, and communication. It was developed using
evidence-based approaches from capstone and design programs in engineering,
practicums, and other project-based courses.

In (Barman et al. 2022), the challenges in using capstone projects as
experiential learning opportunities in data science courses due to resource
constraints and data legalities involved in students working with clients on
clients’ real-world data sets are emphasized. A novel client-facing consulting
data science course that provides experiential learning to undergraduate and
graduate students is developed to tackle this issue.

In an ethics and data analytics course where engineering students developed
solutions for smart grid, smart health, and smart mobility, (Martin and
Bombaerts 2022) explored how professional responsibility is understood by
engineering students working on a solution to a real-world problem proposed by
a client. The authors acknowledge that as technology such as AI/ML advances,
it presents complex challenges that require an interdisciplinary approach. In
line with this, they introduce students to real-life problems presented as
socio-technical challenges embedded in a learning context using Challenge
Based Learning (CBL), an experiential learning method.

The most recent works more closely related to our work are the works of
(Salazar-Gomez et al. 2022) and (Hashim, Tatarniuk, and Harasymchuk 2022). AI
education is a challenging task because it is not well studied, making it one
of the challenges in engineering education (Salazar-Gomez et al. 2022). With
this knowledge, through a series of directives, acts, and laws, the United
States of America government highlighted the importance of the Department of
Defence to embrace AI at speed and scale (Salazar-Gomez et al. 2022). In
(Salazar-Gomez et al. 2022), the Department of Defence (DoD) and the United
States Air Force (USAF) partnered with MIT to design and develop educational
research activities that will provide AI training for DoD and USAF personnel
with diverse professional and educational backgrounds from high school to
graduate degrees, and to the general public. The educational research
activities explored various ways to teach AI education, online and in-person,
to deliver experiential learning experiences to learners of diverse
backgrounds. In the program’s first iteration, they developed and studied the
learning journeys of three different types of USAF employees (leaders,
developers, and users). The findings will be used in future iterations of the
program.

The authors in (Velaj et al. 2022) design a data science course for non-
computer science students. The course goal was to develop a new method for
designing a data science course suitable for teaching students with different
backgrounds in data science. The goal of the new method was to ensure that
students gained skills on how to set up, manage, and conduct data science
projects. The data science course was taught in Business Analytics and Data
Science and the Digital Humanities master’s programs. Students were introduced
to KNIME, an open-source analytics platform for creating data science
workflows without coding. Students were also encouraged to use Python and R
programming languages if they were already familiar with them. The results
reveal that the course is well designed and structured for students with
different backgrounds, that the students gained skills to carry out a data
science project, and that students liked the analytics platform used in the
project work.

The popularity of AI/ML has led to the proliferation of research studies on
designing better data science education materials, as shown in some of the
works reviewed. These works mainly focused on designing data science courses
for engineering and computing students, with a few focusing on non-computing
students. Most importantly, these works do not focus on experiential, hands-on
labs that will provide students with the experience to bridge the gap between
theoretical knowledge and practice. We fill this gap by developing
experiential, hands-on labs for non-computing students and discuss the lessons
learned in developing these hands-on labs.

##  3 Design & Development of AI Socially Relevant Cyberharassment Lab

###  3.1 Lab Structure

The experiential learning laboratory consists of two labs where students
become familiar with the basics of AI and the AI/ML pipeline for applying ML
to a problem.

#### Objectives

We designed the experiential learning laboratories with specific learning
objectives. Essential for guiding student learning, the learning objectives
guided and helped develop our laboratories. This ensured that our study
covered the fundamental elements of AI and different dimensions of AI-driven
social cybersecurity. It also demonstrated the interplay between AI and
cybersecurity and how AI is used for cyberharassment detection. Specifically,
our learning objective is to develop hands-on experiential labs that will
increase general awareness of socially relevant cybersecurity and AI, which is
suitable for teaching AI socially relevant cybersecurity to non-computing
students.

#### Lab

Our lab adopts a phased design approach. Initially, AI and cybersecurity
experts in our team designed the preliminary lab and implemented and
integrated the cyberharassment detection code in the lab. The lab was designed
considering the student profiles, AI and cyberharassment learning objectives,
and desired skills. Subsequently, social scientists in our team engaged in
dialogues with the AI and cybersecurity scholars to enhance the lab’s reach
and inclusivity. After the concerns of the social scientists have been
addressed in the next iteration of the lab, the researchers collaboratively
designed the lecture sessions and lab assignments. The collaborative and
interdisciplinary approach ensured that the lab was accessible to a broad
range of learners.

Our lab, Cyberbullying Detection Using AI, is designed to guide students
through a series of learning objectives and the AI development process. The
learning objectives include understanding AI, understanding the concept and
severity of cyberharassment, the importance of using AI in addressing this
widespread online social issue, and introduction to the development and the
use of AI systems for cyberharassment detection. For the AI development
process, the AI experts ensured that the design followed the AI development
pipeline (data collection, verification and preprocessing, feature extraction,
AI system training, and testing, and use of trained AI systems on a task) to
provide students with the fundamentals of the processes followed to develop an
AI system. In parallel, the social scientists offered profound insights into
the peculiarities of cyberharassment, highlighting its significance and the
pressing need for AI intervention. All researchers from diverse academic
backgrounds involved in this work cross-verified the lab content to ensure the
lab is easily accessible to non-computing students. This process ensures that
those new to AI and cyberharassment can easily grasp and engage with our lab
materials. Figure 2 shows a comprehensive instruction manual prepared for the
lab to facilitate independent learning and ensure students accomplish the
learning objectives outside of the classroom.

Figure 1: A screenshot illustrating the lab interface Figure 2: A screenshot
illustrating Lab 1 instruction manual

#### Lab Delivery

To facilitate a comprehensive understanding of our lab, we implement a three-
fold approach that includes a lecture phase, an experiential hands-on
experience phase, and a phase dedicated to independent work.

Before the students can work on the hands-on lab, a background lecture
designed and developed by the team of researchers is given to the students. In
the lecture, the students get acquainted with the nature and nuances of
cyberharassment, AI, the AI development process, and the need for utilizing AI
for cyberharassment detection.

After introducing the students to the fundamentals needed to complete the lab
through the lecture, students are introduced to the hands-on lab and guided
through the hands-on experience platform depicted in Figure 1. The lab is
developed on the Google Colab platform. The hands-on experience allows
students to apply theory in practice. It facilitates a deeper understanding of
how AI solutions are developed and, most importantly, how AI can be utilized
to mitigate cyberharassment. We have chosen to utilize the Google Colab
platform for several compelling reasons. First and foremost, Google Colab
provides an interactive environment that integrates text and code cells. This
not only enables us to write and execute Python code for deploying AI models
and detecting cyberharassment content, but it also allows us to provide clear,
step-by-step explanations alongside the code. Moreover, these text cells can
be utilized to embed visual aids and explanations such as Figure 3,
facilitating a more comprehensive understanding of the concepts and processes
involved. Secondly, Google Colab offers access to free GPU resources. This is
a significant advantage for our participants as the computational power of
GPUs can greatly expedite the execution of AI models, ensuring that
experiments are completed within a reasonable time frame. Furthermore, Google
Colab’s cloud-based nature eliminates the need for complex setups on personal
machines, lowering the entry barrier for participants. This easy access,
combined with the platform’s robust functionality, makes Google Colab an ideal
tool for our hands-on experiments in AI education.

Figure 3: The visual aid for model explanation in Lab 1 Figure 4: One example
of a lab activity

In our labs, we have meticulously curated post-lab assignments that not only
enhance the engagement factor but also deepen students’ comprehension of the
material. For example, as depicted in Figure 4, students are tasked with using
arbitrary input statements to test their AI’s capability to recognize
cyberharassment content. In the Google Colab platform, the correct execution
of a cell could depend on the execution of the previous cell, and students are
made aware of this, which is also in the lab manual. In the example activity
shown in Figure 4, students must complete all prior steps to access the
developed AI model for cyberharassment detection.

Figure 5: One example of Lab 1’s discussion questions

Additionally, we have formulated post-lesson discussion questions. Figure 5
depicts an example question. We aim to stimulate students’ critical thinking
by encouraging them to consider other real-world problems that could be
alleviated through AI.

###  3.2 Course Structure

The Sociology program at North Carolina A&T State University includes courses
on Social Statistics, parts 1 and 2. Students from Social Statistics 1 were
included in this intervention. Students in Social Statistics 1 learn how to
interpret and describe data. Students are exposed to topics such as
distributions; descriptive statistics (e.g., measures of central tendency and
dispersion); statistical null-hypothesis testing, and independent and
dependent samples t-tests. They also learn the basic operation of the
statistical computing software program SPSS. Throughout the course, students
learn the value statistical analysis offers to our attempts to address real-
world social problems. This course runs 15 weeks and includes lecture and lab
time for analysis. Students attend two 75-minute classes each week. Lectures
involve definitions and the demonstration of practice problems. Lab time
involves hands-on computing using SPSS. In Table 1, we report some basic
demographics of the students in the Social Statistics 1 class in the Spring
and Fall semesters.

| Spring 2022 | Fall 2022
---|---|---
Demographic | Percent | Percent
Male | 23.81% | 14.29%
Female | 71.43% | 85.71%
Freshman | 23.81% | 4.76%
Sophomore | 33.33% | 66.67%
Junior | 38.10% | 9.52%
Senior | 4.76% | 14.29%
Graduate | 0.00% | 4.76%
Table 1: Demographics of students who participated in the Social Statistics 1
course in the Spring and Fall 2022 semesters.

##  4 Methods

###  4.1 Survey Design

To evaluate the impact of our lab on students, we designed two surveys (pre-
survey and post-survey) to collect quantitative and qualitative data from the
students. Students rate their knowledge about AI and machine learning methods
for cyberharassment detection. For both surveys, students rated their
knowledge using a 5-point scale, with 1 representing ”Proficient or strongly
agree” and 5 representing ”None or strongly disagree”. In the post-survey,
students also rate the lab from different perspectives. The post-survey also
includes three open questions to understand what helped the students the most
in understanding the concepts taught, the difficulties faced in using the lab,
and any suggestions for improving the learning experience of the lab.

###  4.2 Data Collection

Before our data collection, we obtained our institution’s Institution Review
Board (IRB) approval. All the students were notified that the survey would
remain confidential and only the research team could view the data. Students
were also informed that their participation was optional. Before the class,
students filled out the pre-survey to evaluate their knowledge level, and
after the lab, students filled the post-survey. Our surveys are conducted
using the Qualtrics platform (Qualtrics 2024).

###  4.3 Data Analysis

To analyze the collected data, we first compared the average knowledge score
between pre-survey and post-survey. Then we used the sample t-test to
determine the existence of statistical significance between the observed
differences. We use a significance level of 0.05. If the p-value of the t-test
is less than 0.05, we conclude a significant difference exists. Scipy, an
open-source Python library for scientific computing, was used to analyze the
collected data.

1 = Proficient, 5 = None |  Pre |  Post
---|---|---
Automated cyberharassment detection. |  3.95 |  3.56
State-of-The-Art cyberharassment detectors. |  4.33 |  3.67
How machine learning works. |  4.10 |  3.89
The lab engaged me in learning the topic of AI Driven Socially-Relevant Cybersecurity. |  - |  2.78
I enjoyed the learning experience of this lab(s). |  - |  2.89
I think the learning experience with the lab(s) is effective. |  - |  2.78
I am satisfied with the level of effort the lab requires for learning this topic. |  - |  2.78
After using the lab(s), I have better understanding about the concepts learned. |  - |  3.44

Table 2: Spring 2022 semester survey results.

1 = Proficient, 5 = None |  Pre |  Post
---|---|---
Automated cyberharassment detection. |  4.38 |  3.31
State-of-The-Art cyberharassment detectors. |  4.71 |  3.44
How machine learning works. |  4.43 |  3.13
The lab engaged me in learning the topic of AI Driven Socially-Relevant Cybersecurity. |  - |  2.56
I enjoyed the learning experience of this lab(s). |  - |  2.44
I think the learning experience with the lab(s) is effective. |  - |  2.13
I am satisfied with the level of effort the lab requires for learning this topic. |  - |  2.56
After using the lab(s), I have better understanding about the concepts learned. |  - |  2.31

Table 3: Fall 2022 semester survey results.

##  5 Results AND Discussion

We analyze the data for the Spring 2022 and Fall 2022 semesters and compare
the learning experiences of both semesters because the student feedback from
the Fall 2022 semester, as discussed in the lesson learned section 5.10,
provided data and insights that were used to inform and refine the lab in the
Spring 2022 semester. Before introducing the labs to the students, they were
invited to complete a pre-questionnaire, and after completing the labs, they
were invited to complete a post-questionnaire. The pre and post-questionnaire
contained eleven questions in total. The first three questions were asked in
both the pre and post-questionnaire. The remaining eight questions were only
asked in the post-questionnaire in addition to the first three questions. Of
the eight questions, the last three were open-ended questions. In the initial
data analysis, we focus on the first three questions in the pre-and-post
questionnaires administered to the students. In the final data analysis, we
focus on only the last eight questions in the post-questionnaire that the
students completed after completing the labs to gauge their understanding and
perception of the lab. The data analysis focused on survey participation,
knowledge in automated cyberharassment detection, evaluation of ML
classifiers, current state-of-the-art cyberharassment systems, how ML works,
general learning experience and engagement of the labs, and student
qualitative feedback. Tables 2 and 3 show the Spring 2022 and Fall 2022 survey
results.

###  5.1 Survey Participation

The survey was presented to students in the Spring 2022 and Fall 2022
semesters. Of the 21 students enrolled in the course in Spring 2022, 21 (100%)
students completed the pre-survey, and 9 (43.9%) students completed the post-
survey. In the Fall 2022 semester, 21 students were enrolled in the class, of
which all students completed the pre-survey, and 16 (80%) students completed
the surveys—showing that the Fall 2022 session of the course had more student
participation than when the lab was first introduced in Spring 2022.

###  5.2 Cyberharassment Detection

During the pre-survey of Spring 2022, 19.05% of the students rated their
knowledge or skills in automated cyberharassment detection in the Proficient
and Good category, about 14.29% of the students rated themselves as having
moderate knowledge or skill, and more than half (66.67%) of the students rated
their knowledge or skills in the A little and None category. After the post-
survey, 33.33% of the students rated their knowledge or skills in the
Proficient and Good category, 0% rated themselves as moderate, and 66.66%
rated their knowledge in the A little and None category.

In the pre-survey Fall 2022, 4.76% of the students rated their knowledge in
the Proficient and Good category, 19.05% as having moderate skill or
knowledge, and 76.19% rated their knowledge or skills in the A little and None
category. On the other hand, in the post-survey of Fall 2022, 25% of the
students rated their knowledge in the Proficient and Good categories, 25%
rated themselves as moderate, and 50% rated their knowledge in the A little
and None category.

Most students had little knowledge of automated cyberharassment detection
before enrolling in the class and participating in the lab. After completing
the lab and comparing the means of the pre-survey and post-survey of Spring
2022 as shown in Table 2, there is no significant difference ($p>0.05$) in the
knowledge of automated cyberharassment detection. For the Fall 2022 semester,
there is a significant difference ($p<0.05$) in the knowledge of automated
cyberharassment detection as indicated in Table 3. Indicating that the
improvements made after the Spring 2020 semester helped improve students’
knowledge or skill in the Fall 2022 semester.

###  5.3 State of the Art Cyberharassment Systems

We determined the knowledge or skills of students in the state-of-the-art
cyberharassment systems before and after the lab. During the pre-survey of
Spring 2022, 4.76% of the students rated their knowledge or skills in state-
of-the-art systems in the Proficient and Good categories, about 19.05% of the
students rated themselves as having moderate knowledge or skill, and more than
half (76.19%) of the students rated their knowledge or skills in the A little
and None category. After the post-survey, 33.33% of the students rated their
knowledge or skills in the Proficient and Good category, 0% rated themselves
as moderate, and 66.66% rated their knowledge in the A little and None
category.

In the pre-survey Fall 2022, 4.76% of the students rated their knowledge in
the Proficient and Good categories, 4.76% as having moderate skill or
knowledge, and 90.47% rated their knowledge or skills in the A little and None
category. On the other hand, in the post-survey of Fall 2022, 18.75% of the
students rated their knowledge in the Proficient and Good categories, 25%
rated themselves as moderate, and 56.25% rated their knowledge in the A little
and None category.

Many students had little knowledge of the state-of-the-art cyberharassment
system before the lab. After completing the lab and comparing the means of the
pre-survey and post-survey of Spring 2022, there is no significant difference
($p>0.05$) in the knowledge of state-of-the-art systems. For the Fall 2022
semester, there is a significant difference ($p<0.05$) in the knowledge of
state-of-the-art systems. Similar to the knowledge of cyberharassment
detection, the improvements made after the Spring 2020 semester helped in
improving students’ knowledge or skill in the Fall 2022 semester.

###  5.4 How Machine Learning Works

We also determined if the students knew how machine learning worked before the
lab and if they learned how machine learning worked after the lab. From the
pre-survey of Spring 2022, results show that 9.52% of the students rated their
knowledge in how ML worked in the Proficient and Good categories, about 28.57%
of the students rated themselves as having moderate knowledge, and more than
half (61.9%) of the students rated their knowledge or skills in the A little
and None category. After the post-survey, 22.22% of the students rated their
knowledge in the Proficient and Good category, 0% rated themselves as
moderate, and 77.77% rated their knowledge in the A little and None category.

In the pre-survey Fall 2022, 4.76% of the students rated their knowledge in
the Proficient and Good categories, 14.29% as having moderate skill or
knowledge, and 80.96% rated their knowledge or skills in the A little and None
category. On the other hand, in the post-survey of Fall 2022, 26.66% of the
students rated their knowledge in the Proficient and Good categories, 33.33%
rated themselves as moderate, and 40% rated their knowledge in the A little
and None category.

In both semesters, more students had little knowledge of how ML worked before
the lab. After completing the lab and comparing the means of the pre-survey
and post-survey of Spring 2022, there is no significant difference ($p>0.05$)
in the knowledge of how ML worked. For the Fall 2022 semester, there is a
significant difference ($p<0.05$) in the knowledge of how ML worked. The lab
refinements made after the Spring 2020 semester helped improve students’
knowledge or skill in the Fall 2022 semester.

###  5.5 Lab Engagement

The students responded positively to how the lab engaged them in learning
about AI-driven socially relevant cybersecurity. In the Spring of 2022, 44.44%
of the students rated how engaging the lab was in the Strongly agree and
Somewhat agree categories, 22.22% rated engagement as moderate, and 33.33%
rated engagement in the Somewhat disagree and Strongly disagree category. In
the Fall of 2022, more than half (62.5%) of the students rated how engaging
the lab was in the Strongly agree and Somewhat agree categories, 12.50% rated
engagement as moderate, and 25% rated engagement in the Somewhat disagree and
Strongly disagree category.

The mean response by students in the Spring and Fall semesters was 2.78 and
2.56, respectively. The mean values are between the Somewhat agree and Neither
agree nor disagree with the mean value of Fall closer to Somewhat agree than
the Spring mean, indicating that the students’ lab engagement was positive and
not very negative.

###  5.6 Learning Experience

The student’s response to the overall learning experience and the
effectiveness of the experience was somewhat positive. In the Spring of 2022,
33.33% of the students rated the learning experience in the Strongly agree and
Somewhat agree categories, 44.44% rated the learning experience as moderate,
and 22.22% rated the learning experience in the Somewhat disagree and Strongly
disagree category. For the effectiveness of the learning experience, 44.44% of
the students rated the effectiveness in the Strongly agree and Somewhat agree
category, 33.33% rated effectiveness as moderate, and 22.22% rated learning
experience in the Somewhat disagree and Strongly disagree category.

In the Fall of 2022, more than half (68.75)% of the students rated the
learning experience in the Strongly agree and Somewhat agree categories,
12.50% rated learning experience as moderate, and 18.75% rated learning
experience in the Somewhat disagree and Strongly disagree category. For the
effectiveness of the learning experience, 43.75% of the students rated the
effectiveness in the Strongly agree and Somewhat agree category, 37.50% rated
effectiveness as moderate, and 18.75% rated learning experience in the
Somewhat disagree and Strongly disagree category.

For the overall learning experience, the mean response by students in the
Spring and Fall semesters was 2.89 and 2.44, respectively. The mean value of
the Fall semester is closer to Somewhat agree, indicating that the students
had a more positive learning experience in the Fall semester. For the
effectiveness of the learning experience, the mean response from students in
the Spring and Fall semesters was 2.78 and 2.13, respectively. Similar to the
learning experience, the mean value of the Fall semester is closer to Somewhat
agree, indicating that in the Fall semester, the effectiveness of the learning
experience was more positive for the students.

###  5.7 Lab Difficulty

We gauged the difficulty of the lab for the students, and the response
indicated that the students neither agreed nor disagreed that the labs were
demanding. In the Spring of 2022, 44.44% of the students rated the level of
effort required to learn the topic in the Strongly agree and Somewhat agree
categories, 22.22% rated engagement as moderate, and 33.33% rated learning
effort in the Somewhat disagree and Strongly disagree category. In the Fall of
2022, more than half (62.5%) of the students rated how engaging the lab was in
the Strongly agree and Somewhat agree categories, 25% rated learning effort as
moderate, and 12.50% rated learning effort in the Somewhat disagree and
Strongly disagree category.

The mean response by students in the Spring and Fall semesters was 2.78 and
2.56, respectively. The mean values are between the Somewhat agree and Neither
agree nor disagree with the mean value of Fall closer to Somewhat agree than
the Spring mean, indicating that the level of effort required to learn the
topic by the students was better in the Fall than in Spring even though the
students neither agree nor disagree.

###  5.8 Understanding of Concepts

To understand whether the students better understood the concepts learned, we
determined students’ grasp of the concepts introduced in the lab. In the
Spring of 2022, 22.22% of the students rated understanding of concepts in the
Strongly agree and Somewhat agree categories, 33.33% rated understanding of
concepts as moderate, and 44.44% rated understanding of concepts in the
Somewhat disagree and Strongly disagree category. In the Fall of 2022, 43.75%
of the students rated understanding of concepts in the Strongly agree and
Somewhat agree categories, 25% rated understanding of concepts as moderate,
and 31.25% rated learning effort in the Somewhat disagree and Strongly
disagree category.

The mean response by students in the Spring and Fall semesters was 3.44 and
2.31, respectively. The mean value for the Spring semester is between Neither
agree nor disagree and Somewhat disagree. For the Fall semester, the mean
value is between Somewhat agree and Neither agree nor disagree. These values
indicate that in the Spring semester, students understanding of concepts
learned was closer to negative (Somewhat disagree). However, in the Fall
semester, students better understood the concepts learned.

###  5.9 Qualitative Feedback

Our last three survey questions were open-ended questions about what has been
the most helpful for learning, what has caused the most difficulty using the
lab, and how the lab can be improved. We used these as the qualitative data
source, providing insights into students’ perceptions and preferences. In
general, the students understood the purpose of the lab and cyberharassment,
found the terminology confusing, and wanted the lab to be more fun and
engaging. In the Spring 2022 semester, in response to What has been most
helpful for your learning in using the lab so far. Notable student responses
were: I understand the purpose of cyberbullying and its purpose and how it is
designed to be successful and The guest speakers coming in to help. For the
Fall 2022 semester, the notable student responses were: The most helpful part
for my learning has been the hands-on activity, being able to ask questions
while going through the work and having a guest speaker gave some new
insight., I learned a lot about cyber bullying that I did not know about and
the different forms it can come in., The lab was helpful with detecting cyber
harassment., It was nice knowing the set up and watching the steps be
performed, and The videos and zoom call.

For the question In terms of your learning, what has caused you the most
difficulty in using the lab so far. The notable student responses in the
Spring 2022 semester were: I had a hard time understanding how to actually
complete on my own, The terminology, I could not stay focus and lacked
engagement, and Being online. Notable responses in the Fall semester were: It
was difficult that when there was a troubleshooting error that I could not
walk through it with someone like I could in person., The most difficult part
is not usage of the lab itself; it is remembering certain aspects of what to
do and what to look for when in the lab., The most difficulty experienced in
the labs is facing errors and technical difficulties., I think the directions
were hard to follow because the instructor seemed like he assumed we knew
something about this content., and Fully understanding what was being done in
the lab was the most difficult.

Finally, for the question What suggestion(s) can you make that would enhance
your learning experience with the lab. Notable student responses in the Spring
2022 semester were: Make the lab more engaging/fun, Break down steps on how to
actually complete the activity, I would say, trying using a different online
platform for this lab, to make everything a little bit easier for students to
understand., Being in person, and Better terminology. In the Fall 2022
semester, notable responses were: The instructor was helpful it’s just hard to
learn over the computer such a difficult thing to do., I cannot think of
anything at the moment. I really enjoyed learning about this lab and how it
worked., An in person option, provide a tutorial video., I would say slowing
down the directions., and Teach more about how to face technical difficulties.

###  5.10 Lesson Learned

The authors learned the following lessons in implementing a cloud-based
laboratory experience. We outline tips for developing a cloud-based laboratory
for teaching AI socially relevant cybersecurity.

#### Code Dilution

The lab is implemented on Google Colab, Google’s cloud-based jupyter notebook
platform. The initial implementation of the lab on Google Colab presented the
students with the raw code implementation of cyberharassment detection using
PyTorch, an open-source Python framework for developing ML, especially deep
learning systems. From the Spring 2022 feedback from the students, we observed
that the students were not positive towards the code implementation since they
have very little programming experience. This frustrated students and slowed
interest and learning. With this knowledge, in preparation for the Fall 2022
semester, we improved the learning experience by re-implementing the lab with
the code hidden to enable the students to think about social issues and focus
on understanding how AI can be used to approach social issues such as
cyberharassment.

#### Lab Instructions

Developing the lab instructions in the lab manual is crucial to enhancing the
student experience. If the lab instructions are not very detailed, with step-
by-step instructions on how to complete the lab activities, the students
struggle with understanding and completing the lab independently. From the
Spring 2022 student feedback, students complained that the instruction manual
needed more detail and felt the instructors assumed they were familiar with AI
and AI terminology. In the Fall 2022 semester, we improved the lab manual by
toning down the terminology and providing more step-by-step descriptions so
the students could complete the labs independently and understand the purpose
and rationale behind each step.

#### Pre-lab Lecture

Before allowing the students to complete the lab independently, we prepared
lecture slides about the lab and introduced them to the problem and the
activities they would be completing. The students found this pre-lab lecture
particularly helpful.

Other best practices include recording the pre-lab lecture so that the
students can refer back to it when working on the lab activities, anticipating
possible technical difficulties, and including steps to solve the issues in
the lab manual. Having an in-person option where the students can complete the
labs during class could help with engagement and technical issues they might
encounter.

##  6 Limitations

Our work has limitations. First, our study is focused on one public
institution in the United States, limiting our findings’ generalizability.
Second, the generalizability of our findings is also limited by the focus of
our study on one non-computing program - Social Statistics, excluding other
non-computing programs. Third, the number of participants in our study further
limits our conclusions. Finally, the current lab only focuses on the general
concepts of AI and how it can be utilized to address social issues such as
cyberharassment. It does not cover other areas of AI, such as how adversarial
attacks can fool cyberharassment models, multi-modal cyberharassment
detection, and the issue of fairness and bias in cyberharassment models.

##  7 Conclusion and Future Work

We have developed an AI socially relevant cybersecurity lab for
cyberharassment detection for non-computing students. We introduced a
cyberharassment detection lab’s development, implementation, and assessment.
The development process has been guided by the learning objective of
introducing a hands-on experiential lab that will increase general awareness
of socially relevant cybersecurity and AI and is suitable for teaching AI
socially relevant cybersecurity to non-computing students. Our lab offers
meaningful experiential learning opportunities that allow students to work on
real-world social issues such as cyberharassment. After incorporating student
feedback in the redesign of the lab used in the Fall semester, the knowledge
or skills of most students in automated cyberharassment detection and how ML
works improved significantly compared to the Spring semester. Also, students
found the detection of cyberharassment helpful and understood the purpose of
using AI for social issues. We plan on continuing to refine the lab and use
the knowledge gained in developing four labs currently under development that
cover multi-modal (text and image) cyberharassment detection, adversarial
attacks on cyberharassment systems, bias mitigation in cyberharassment
systems, and the use of generative AI models such as ChatGPT for
cyberharassment detection. Additionally, we plan on developing these labs for
computer science and engineering students in the future.

## Acknowledgment

The work was supported by National Science Foundation (NSF) under the Grant
No. 2239605, 2228616 and 2114920.

## References

  * Allen (2021) Allen, G. I. 2021.  Experiential learning in data science: Developing an interdisciplinary, client-sponsored capstone program.  516–522.
  * Anderson et al. (2014) Anderson, P.; Bowring, J.; McCauley, R.; Pothering, G.; and Starr, C. 2014.  An undergraduate degree in data science: curriculum and a decade of implementation experience.  145–150.
  * Barman et al. (2022) Barman, A.; Chen, S.; Chang, A.; and Allen, G. 2022.  Experiential Learning in Data Science Through a Novel Client-Facing Consulting Course.  1–9.
  * Challenge (2020) Challenge, H. M. 2020.  Hateful Memes Challenge and Data Set.  https://ai.facebook.com/tools/hatefulmemes/.  Accessed: 2024-02-20.
  * Ducharme (2017) Ducharme, D. 2017.  _Machine Learning for the Automated Identification of Cyberbullying and Cyberharassment_.  Ph.D. thesis, University of Rhode Island.
  * Eybers and Hattingh (2016) Eybers, S.; and Hattingh, M. 2016.  Teaching Data Science to Post Graduate Students: A Preliminary Study Using a” FLIP” Class Room Approach.  _International Association for Development of the Information Society_.
  * Facebook (2023) Facebook. 2023.  AI advances to better detect hate speech.  https://ai.facebook.com/blog/ai-advances-to-better-detect-hate-speech/.  Accessed: 2023-11-01.
  * Goodfellow, Shlens, and Szegedy (2015) Goodfellow, I. J.; Shlens, J.; and Szegedy, C. 2015.  Explaining and Harnessing Adversarial Examples.
  * Google-Jigsaw (2019) Google-Jigsaw. 2019.  Google’s Hate Speech Detection A.I. Has a Racial Bias Problem.  https://fortune.com/2019/08/16/google-jigsaw-perspective-racial-bias/.  Accessed: 2023-11-01.
  * Hashim, Tatarniuk, and Harasymchuk (2022) Hashim, H. A.; Tatarniuk, C.; and Harasymchuk, B. 2022.  First Year Engineering Design: Course Design, Projects, Challenges, and Outcomes.  1–9.
  * Kolb and Kolb (2005) Kolb, A. Y.; and Kolb, D. A. 2005.  Learning styles and learning spaces: Enhancing experiential learning in higher education.  _Academy of management learning & education_, 4(2): 193–212.
  * Kolb (2014) Kolb, D. A. 2014.  _Experiential learning: Experience as the source of learning and development_.  FT press.
  * Li et al. (2019) Li, J.; Ji, S.; Du, T.; Li, B.; and Wang, T. 2019.  TextBugger: Generating Adversarial Text Against Real-world Applications.
  * Martin and Bombaerts (2022) Martin, D. A.; and Bombaerts, G. 2022.  Enacting socio-technical responsibility through Challenge Based Learning in an Ethics and Data Analytics course.  1–7.
  * Matovu et al. (2022) Matovu, R.; Nwokeji, J. C.; Holmes, T.; and Rahman, T. 2022.  Teaching and Learning Cybersecurity Awareness with Gamification in Smaller Universities and Colleges.  1–9.
  * Nanda et al. (2020) Nanda, V.; Dooley, S.; Singla, S.; Feizi, S.; and Dickerson, J. P. 2020.  Fairness Through Robustness: Investigating Robustness Disparity in Deep Learning.  _CoRR_ , abs/2006.12621.
  * Nwokeji and Frezza (2017) Nwokeji, J. C.; and Frezza, P. S. T. 2017.  Cross-course project-based learning in requirements engineering: An eight-year retrospective.  1–9.
  * Nwokeji, Stachel, and Holmes (2019) Nwokeji, J. C.; Stachel, R.; and Holmes, T. 2019.  Effect of instructional methods on student performance in flipped classroom.  1–9.
  * Okpala et al. (2022) Okpala, E.; Cheng, L.; Mbwambo, N.; and Luo, F. 2022.  AAEBERT: Debiasing BERT-based Hate Speech Detection Models via Adversarial Learning.  In _2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA)_ , 1606–1612. IEEE.
  * Patchin (2019) Patchin, J. W. 2019.  Cyberbullying Statistics.  https://cyberbullying.org/2019-cyberbullying-data.
  * Qualtrics (2024) Qualtrics. 2024.  Qualtrics.  https://www.qualtrics.com/.  Accessed: 2024-02-20.
  * Rosenthal and Chung (2020) Rosenthal, S.; and Chung, T. 2020.  A data science major: Building skills and confidence.  178–184.
  * Salazar-Gomez et al. (2022) Salazar-Gomez, A. F.; Bagiati, A.; Minicucci, N.; Kennedy, K. D.; Du, X.; and Breazeal, C. 2022.  Designing and implementing an AI education program for learners with diverse background at scale.  1–8.
  * Samavedham and Ragupathi (2012) Samavedham, L.; and Ragupathi, K. 2012.  Facilitating 21st century skills in engineering students.  _The Journal of Engineering Education_ , 26(1): 38–49.
  * Sap et al. (2019) Sap, M.; Card, D.; Gabriel, S.; Choi, Y.; and Smith, N. A. 2019.  The Risk of Racial Bias in Hate Speech Detection.  1668–1678.
  * Serrano et al. (2017) Serrano, E.; Molina, M.; Manrique, D.; and Baumela, L. 2017.  Experiential learning in data science: From the dataset repository to the platform of experiences.  122–130.
  * Social-Cybersecurity (2019a) Social-Cybersecurity. 2019a.  Center for Informed Democracy & Social-Cybersecurity.  https://www.cmu.edu/ideas-social-cybersecurity/.  Accessed: 2023-11-01.
  * Social-Cybersecurity (2019b) Social-Cybersecurity, C. 2019b.  CASOS.  http://www.casos.cs.cmu.edu/projects/projects/social_cyber_security.php.  Accessed: 2023-11-01.
  * Swearer (2013) Swearer, D. E. S. 2013.  Research on school bullying and victimization: What have we learned and where do we go from here?  _School Psychology Review_ , 365–383.
  * Thomas et al. (2021) Thomas, K.; Akhawe, D.; Bailey, M.; Boneh, D.; Bursztein, E.; Consolvo, S.; Dell, N.; Durumeric, Z.; Kelley, P.; Kumar, D.; McCoy, D.; Meiklejohn, S.; Ristenpart, T.; and Stringhini, G. 2021.  SoK: Hate, Harassment, and the Changing Landscape of Online Abuse.  473–493.
  * Velaj et al. (2022) Velaj, Y.; Dolezal, D.; Ambros, R.; Plant, C.; and Motschnig, R. 2022.  Designing a Data Science Course for Non-Computer Science Students: Practical Considerations and Findings.  1–9.
  * Wachs, Wright, and Vazsonyi (2019) Wachs, S.; Wright, M. F.; and Vazsonyi, A. T. 2019.  Understanding the overlap between cyberbullying and cyberhate perpetration: Moderating effects of toxic online disinhibition.  _Criminal Behaviour and Mental Health_ , 29(3): 179–188.
  * Zhang et al. (2020) Zhang, W. E.; Sheng, Q. Z.; Alhazmi, A. A. F.; and Li, C. 2020.  Adversarial Attacks on Deep-learning Models in Natural Language Processing: A Survey.  _ACM Trans. Intell. Syst. Technol._ , 11(3): 1–41.
  * Zhong et al. (2016) Zhong, H.; Li, H.; Squicciarini, A.; Rajtmajer, S.; Griffin, C.; Miller, D.; and Caragea, C. 2016.  Content-Driven Detection of Cyberbullying on the Instagram Social Network.  3952–3958.