figurec [firstpage,color=gray!60,angle=90,scale=0.32, xpos=-4.05in,ypos=0]
[firstpage,color=gray!60,angle=90,scale=0.32, xpos=3.9in,ypos=0]Preprint doi
[firstpage,color=gray!90,angle=0,scale=0.28,
xpos=0in,ypos=-5in]*correspondence<EMAIL_ADDRESS>
# Binary-level Software Compatibility Tool Agreement

Vanessa Sochat ID Lawrence Livermore National Lab, Livermore CA, USA  Tim
Haines ID University of Wisconsin Madison, Madison WI, USA

###### Abstract

Application Binary Interface (ABI) compatibility is essential for system or
software updates to ensure that libraries continue to function. Tools that can
assess a binary or library ABI can thus be used to make predictions about
compatibility, and predict downstream bugs by informing developers and users
about issues. In this work, we are interested in describing a set of well-
known tools for assessing ABI, and testing them in a controlled set
experiments to assess tool agreement. We run 7660 smaller experiments across
tools (N=30,640 total results) to evaluate not only predictions, but also each
tool’s ability to provide detail about underlying issues. In this work, along
with highlighting the problem of assessing ABI compatibility and critiquing
the pros and cons of currently available tools, we provide guidance to
developers interested to test ABI based on our empirical results and
suggestions for future work.

_K_ eywords application binary interface $\cdot$ ABI $\cdot$ software
compatibility

##  1 Introduction

In the same way that an Application Programming Interface (API) provides an
interface to access the functionality and data provided by a library and a
calling client must understand the details of these interfaces to make
successful requests, the Application Binary Interface (ABI) of any software or
operating system is interested in similar compatibility, but on the level of
machine code. This means that as new software versions are released to update
a system, the functions, parameter types, and many other attributes of the new
binaries must be compatible with other libraries on the system. Akin to
calling a RESTful API endpoint that expects a particular set of parameters
from a client, if you change or otherwise provide an incorrect set, your
client will no longer work with the API. The same is true for the binaries on
a system, and careful work must be taken to ensure enduring compatibility. Any
new library installed to the system must function without requiring
recompilation of other libraries already in the ecosystem.

Ensuring compatibility is a non-trivial task, as different systems provide
different compilers, operating systems, and supporting libraries, and these
varying tool chains are expected to build and run the software. Even a simple
case of a function changing argument types can mean a break in functionality.
Thus, introducing multiple compiler tool-chains, library versions, languages,
and even architectures for connected systems [1], presents the community with
an enduring challenge: How do we assess ABI compatibility, and intelligently
update tools with some assurance that the software will continue to work?

###  1.1 The Origins of ABI Compatibility

When we install software on a computer and run it, we often take it for
granted that it just works. Software is created by way of the compilation
process - starting with source code, and compiling it with a language-specific
compiler into byte-code that a particular machine can execute. The files
containing these byte code can either be referred to as executables or binary
libraries (often just called libraries). Installation of these binaries then
comes down to either retrieving a previously built artifact (e.g., from a
package manager) or building from source. The ultimate successful outcome of
this process is the software running successfully on your system.

The question about ABI compatibility dates back to the origins of software.
Although we cannot pin an exact date on when people started to think about
compatibility, we can go as far back to 1951 and find that engineers were
thinking about a documenting “Specifications of Library Subroutines” [2] that
detailed the same concerns that we have today – namely types, addresses, and
functionality. It was not until the 1980’s that discussion started about
application binary standards, and how ABI was important for microprocessors
and architecture compatibility [3, 4] across computer magazines [5, 6],
academic publications [7, 8, 9], and even user group newsletters alike [10].
More specifically, the Unix System V version I was first released in 1983
111https://en.wikipedia.org/wiki/UNIX_System_V, and was a prominent standard
under discussion for that decade [5, 11]. At the time of this writing
(October, 2022) there are thousands of mentions [12], and we can speculate
that this growth reflects the term becoming more understood and used across
the software development community. For the interested reader, we include a
brief list of notable ABI initiatives in Table 1.

Table 1: Early Application Binary Interface Efforts Name | Published | Year
---|---|---
UUABI "Unix to Unix“ ABI [13] | Compcon | 1989
PAX "Parallel Architecture Extended for parallel computing“ [14] | Penn State | 1989
Unix System V 22footnotemark: 2 |  | 1983
Intel Binary Compatibility 222https://en.wikipedia.org/wiki/Intel_Binary_Compatibility_Standard | Intel / AT&T, SCO | 1988

This need for compatibility, and discussing it, has led to the creation of
communities of people that are concerned with software’s ABI, and standards
documents [15] that define strict rules for how software in a particular
language is expected to interact with a specific architecture. More
abstractly, these standards define a contract between programs that promises
consistent interoperability between applications even as components are
updated [16]. ABI compatibility also makes a promise that, given that two
binaries follow an ABI specification, we can move one binary to another system
that conforms to the same system without any modifications [17]. Any given
library on a system is considered ABI compatible if the interface that it
provides to access its functions and data is compatible with other libraries
on that system that need it. However, in practice, the idea of ABI
compatibility goes well beyond what is written down in a standards document,
and includes any detail about the structure or functionality of a system that
might render it incompatible with another system’s interface.

###  1.2 Problems that warrant understanding ABI

While there is no universally agreed upon definition for what features
constitute an ABI, there is much discussion around what people think an ABI is
[17, 18] and these ideas are generally based around real problems that
developers face when building software. Questions and topics include (but are
not limited to) parameter passing in registers, data types, exceptions, global
variables, configuration, optimization, dynamic dispatching, and virtual
tables in C++. Common compatibility problems faced by developers include
library versions and change over time, and implementation and compatibility
problems in compilers. We direct the interested reader to the provided
references for a more comprehensive listing of ABI problems. An aggregation of
these ideas might be considered as a larger interface for ABI. Many of these
problems (e.g., configuration- and optimization-related problems), while
important to be worked on, have a level of complexity that deems them out of
scope for this work.

The simplest, and most consistent challenge for a library ABI is with respect
to the interfaces that libraries provide. As an example, if a math library
version 1.0.0 is compiled with a function to do addition that expects two
integer arguments, if the library changes those integer arguments to floats
for version 2.0.0, a previous library that worked with 1.0.0 and expects
integers will not work with version 2.0.0. Running such a library relies on
dynamic linking [19], a process by which the function signatures "symbols" are
loaded into memory and shared. While outside the scope of this paper, we have
provided a comprehensive description of the dynamic linking process for the
interested reader [20]. In the case of our C++ example here, this means that
changing the function name or parameter names or types changes a symbol name.
It follows that removing a function deletes a symbol, and adding a function
adds a new symbol. An example of function signatures and the C++ symbols
called "mangled strings" they produce are shown below, before and after
changing a parameter type:

⬇

int Add(int a, int b); // _ZN11MathLibrary10Arithmetic3AddEii

int Add(double a, double b); // _ZN11MathLibrary10Arithmetic3AddEdd

Notice that the mangled strings end with two different sets of letters that
indicate parameter types, "ii" to indicate two integer parameters, and "dd" to
indicate two double parameters, respectively. A program that uses this math
library that is expecting the first symbol (integer parameters to "Add") would
fail if provided the second library where they have been changed to doubles
due to the missing symbol. This reflects a simple example of a set of much
more complex issues around dynamic linking. While it allows for patching
individual libraries to address security issues or add functionality without
needing to rebuild an entire system, any change in a library could have dire
consequences if the new library is not compatible with other software on the
system. This particular problem is exacerbated for other languages that do not
use parameter type information in the mangled strings. As an example, the same
library compiled with C would produce the symbol "Add" in both cases, meaning
that the incompatibility could not be detected at link time. A more complete
example of this math library [21] with releases is provided.

Having these problems has led to the development of tools that try to solve
them. At the highest level, much of ABI compatibility is the responsibility of
a compiler implementation, and thus falls under the responsibility of compiler
authors. While compilers can catch many of these issues at compile time, third
party tooling can assist with detection after this step. Any such detection
tool inherently has an underlying model about what constitutes compatibility.
For the purposes of this work, we can define a model of ABI compatibility as
an approach a prediction tool makes about ABI compatibility. When two binaries
are determined to not be compatible, we call this a breakage or ABI break. The
model may operate in different contexts or environments, such as doing a
pairwise comparison of two libraries, versus a global assessment of
compatibility of an embedded system with a host architecture. Having such
tools, or more generally checking for ABI compatibility, is important when
performing code instrumentation to ensure memory safety [22] or when improving
compilers for ABI-sensitive regions of code generation [23].

Thus, a reasonable lens to look through to understand our current ability to
assess ABI is through the tooling. While we cannot assess the accuracy of tool
predictions without a ground truth for whether two binaries are compatible, we
can compare consistency across tools and uncover interesting cases of
disagreement. We can highlight these areas as important for future, more in-
depth work, and to provide suggestions to developers for good practices for
developing their software. For this work, we will outline existing models and
tools for assessing ABI 2, design an experiment that tests tools across
several versions of an operating system 3, discuss results, pros and cons of
each tool, and provide suggestions for utilization 5, and conclude by
discussing future work in this space.

##  2 Methods

###  2.1 Tools

The tools discussed are scoped to those that work with the Executable and
Linking Format (ELF) [24] and debugging information (DWARF) that is supported
for it [25]. These binary formats are the most widely used (Libabigail) and
popular [26] across Unix compatible operating systems. To the knowledge of
these authors, there are no well-established ABI checking tools for other
platforms. These tools generally use language level models, meaning they look
at variable types, sizes, and other associated metadata. In these sections we
will describe three tools that can be use to make an assessment, including
libraries Libabigail and the ABI Compliance Checker, and a general method to
assess symbols. In this paper we refer to these tools and stratgegy as
libagigail, abi-compliance-checker, and symbols in italics, respectively.

###  2.2 Libabigail

Libabigail is a project maintained by RedHat that was first publicly released
in 2015 [27]. The C++ library tackles the problem of assessing ABI
compatibility by reducing a library to a set of ABI artifacts called a
“corpus” that can include types, functions and declarations, variables, and
other metadata parsed from ELF and DWARF [28]. The library provides command
line executables to generate these artifacts in XML format to work with tools
to assess compatibility (abicompat), differences (abidiff), and output of the
XML structures (abidw). The strongest use case for Libabigail is for Red Hat
Enterprise Linux (RHEL), Fedora, and the Android operating system. Using
Libabigail, developers ensure that they don’t unintentionally break ABI when
doing updates. While Libabigail uses DWARF to improve comparison, a strength
of the tool is that it does not require it for doing a basic comparison using
only symbols [29].

###  2.3 ABI Compliance Checker

The ABI Compliance Checker [18, 17, 30], akin to Libabigail, is concerned with
assessing ABI compatibility of C and C++ libraries. It does this by way of
defining a set of rules that define ABI compatibility [31] and also outputting
XML to then compare against the rules. The authors make results of their
checks available in a public web interface [32]. The comparisons are driven by
a tool to dump out a representation of the binary (akin to a corpus) using the
abi-dumper [33] and then comparing two dumps with the abi-compliance-checker
[34]. Unlike Libabigail, the ABI compliance checker requires DWARF information
to be present. These tools are appealing to developers as they come with
visualization tools [35] to generate entire web interfaces for results.

###  2.4 Symbols

While there are many tools for extracting symbols (e.g., dpkg-gensymbols,
chkshlib, cmpdylib, cmpshlib), our choice of Python for running experiments
led us to use the pyelfutils [36] library to extract lists of symbols from
binaries. For our compatibility assessment, we start with the ELF file’s
.symtab and .dynsym sections, filter down symbols to with size greater than 0,
not belonging to an undefined section, not missing a type, and not local [37].
An exported symbol is one that is provided by a library and be consumed by
some other library or executable binary via linking. If we let $A$ be an
initial version of a library and $B$ a second, more recent, version of the
same library, then symbols calculates the simple set difference:

⬇

missing-previously-found-exports(A, B):

{exported A} \ {exported B}

In the above, an empty set would indicate that there is no change (and thus no
ABI break) between the sets. This simple symbol comparison "symbols“ serves as
a meaningful base case test. For the interested reader, the above
functionality is implemented in the library "elfcall“ as the symbols predictor
[20].

##  3 Experiment

Our experiment is concerned with tool agreement. Given the three predictors
(Libabigail, ABI Laboratory, symbols) described in Section 2.1 run across a
large set of pairwise comparisons of matched libraries (discussed below)
between Fedora releases 34, 35, 36, and 37 we asked the question what is the
agreement between the tools that an ABI break is detected. We used abidiff for
Libabigail, the abi-dumper and abi-compliance-tester for the ABI Compliance
checker, and the set comparison previously described for symbols. With results
from the predictors, we are interested in the agreement between predictors.
Given the console output from each predictor, we will be able to classify
predicted breakages into the categories of removals, changes, and additions
for each of function parameters, virtual tables, enumerators and global
variables.

###  3.1 Reproducibility Statement

We approach tackling this problem with a goal of reproducibility - it would
not be enough to run experiments that rely on our proprietary systems or
software, so we do the entire analysis using publicly available continuous
integration (CI), unit tests, and automated builds of containers with open
source software. In addition, given the complexity of needing to provide
several builds of prediction software that fold into common software to
perform the experiments, the authors provide automated builds of all container
bases required for the experiments. The containers are provided as automated
builds alongside the main repository that outlines the experiment and how to
run it [38], and the automated runs were done on a separate repository [39]
that saves results as artifacts that are retrieved with the final analysis
repository [40]. All of our tooling, unit tests, and testing infrastructure
are publicly available for others to reproduce or extend our work.

###  3.2 Design

We start with pre-built containers across four Fedora versions with all
predictors installed. Since we need to compare across versions, this requires
saving an artifact from each Fedora version that includes all binaries and
debug information directories, which are installed separately [41]. We can
then create a second job in our continuous integration environment 3.1 that
creates a matrix of pairwise Fedora versions (not comparing a version to
itself) to run the analysis described previously. In layman’s terms, we are
asking each tool if there are ABI breaks between "equivalent“ libraries
provided in different Fedora operating systems. We emphasize "equivalent”
because our best effort to match libraries means checking for equivalent
directory and file name prefixes (e.g., /lib/dirx/libx.so.1 would be matched
to /lib/dirx/libx.so.2 by way of the prefix "libx" and a matching absolute
path parent directory name "/lib/dirx"). This means that it is possible to
have name collisions, or cases of finding the same prefix for two different
underlying libraries (e.g., as is common with symbolic links). However, we did
not find any of these cases in our analysis. For each match, the comparison
procedure is performed with the respective tools, essentially asking if the
two versions are compatible.

To mimic an upgrade process, for the directionality of our comparison we
choose to compare the lower Fedora version (e.g., 34) with a newer one (e.g.,
35). This asks the question if the newer library is ABI compatible with the
older one. In addition to recording complete terminal output to later derive
rationale for a tool’s decision, we also record the library size (in bytes)
along with the time to reach a decision to make an assessment about
bytes/second and relative tool speed.

We developed a Python library, "spliced" [42] that runs the experiments within
container bases matched to our Fedora versions, and with all predictors
installed. Spliced serves as a wrapper to provide a set of binaries, A and B,
to each of the predictors, to capture the output, and save to a common format
(JSON) for further analysis. For each result (a comparison between A and B),
we capture high level metadata about the binaries, sizes in bytes, and a set
of predictions that are namespaced by the predictor. As these experiments are
run in GitHub actions, the results themselves are saved as GitHub artifacts
that can be programmatically retrieved for further analysis [43].

##  4 Results

The goal of our experiment is to do qualitative comparisons between the ABI
compatibility assessment tools Libabigail, the ABI Laboratory, and symbols
(see Section 2.1). We ran pairwise comparisons of matching binaries between
four versions of the Fedora operating system (34, 35, 36, 37), and use the
output (JSON) from these runs to drive our exploratory analysis.

During our experiments, we often found that the abi-compliance-checker
predictor was terminated prematurely by the Linux stack-smashing detector
[44]. Attempting to re-run those analyses yielded inconsistent results such
that we could not feasibly execute this predictor on all of the libraries
under consideration. To account for this, we split the most recent set of
results into two categories: those for which there are results for all three
predictors and those for which there are results only for Libabigail and
symbols. The subset having all three predictors has $59.5^{71.0}_{43.8}\%$
fewer binaries 333Throughout, we use the notation $average^{max}_{min}$ to
represent the sample distribution of a quantity measured across all OS
comparisons. compared to the equivalent experiment having two predictors.

Additionally, we consider a technique often used by library authors to
explicitly signal a break in ABI- a change in the file name reflecting the
corresponding change in the library version number as defined in the libtools
guidelines [45]. As such, we further decompose our analysis into the set of
libraries that changed their file name (after complete symlink resolution)
between OS versions and those that did not. We find between $6$ and $20\%$ of
binaries have a different file name across OS versions in the two-predictor
case and between $2$ and $4\%$ in the three-predictor case. Finally, 72 linker
scripts [46] mistaken for libraries were excluded.

###  4.1 Predictor Agreement

With the datasets in place, we turn our attention to the central question of
agreement between predictors. For the cases with two predictors, at least one
of them reports an ABI incompatibility in $63.3^{70.9}_{50.5}\%$ of cases when
the file name changes and in $12.3^{16.0}_{9.0}\%$ of cases when it doesn’t,
and both predictors agree on the question of compatibility
$58.9^{61.7}_{55.4}\%$ and $82.7^{86.2}_{78.5}\%$ of the time, respectively.
For the three-predictor cases, the number of reported incompatibilities
increases to $75.1^{92.3}_{60.8}\%$ for changed file names and
$16.6^{14.0}_{19.5}\%$ for unchanged ones. Three-way agreement between the
predictors is $56.1^{76.9}_{35.3}\%$ and $79.8^{83.4}_{76.5}\%$, respectively.
To better understand these levels of disagreement, we dissect each case in
detail.

In Table 2, we see $40.2\%$ (N=392) of libraries across all OS comparisons
with changed file names are labelled as compatible by symbols but as
incompatible by Libabigail. The presence of such cases is expected as there
are many other types of ABI incompatibilities beyond just changes to exported
symbols. If we assume that a change in file name is used by the library’s
author to signal a break in ABI, then this behavior is substantiated by the
tools in $86.1\%$ of cases. The remaining 135 cases where no incompatibility
is detected represent either incompleteness in the tools or that a change in
file name indicates something other than an ABI break (e.g., semantic
versioning). When the file name does not change, Table 3 shows that the
fraction of incompatibilities detected only by Libabigail drops to $17.2\%$
but skyrockets to $83\%$ of all detected incompatibilities. Finally, we see
$79.3\%$ (N=5274) of libraries have no detected incompatibilities which is
consistent with our expectation given our assumption of the meaning of a
change in file name.

Table 2: Prediction counts for the two-predictor case when file names changed. | symbols
---|---
Libabigail | compatible | incompatible
compatible | 135 | 0
incompatible | 392 | 447
Table 3: Prediction counts for the two-predictor case when file names did not change. | symbols
---|---
Libabigail | compatible | incompatible
compatible | 5274 | 0
incompatible | 1143 | 233

Looking now at the corresponding three-predictor cases, Table 4 shows that
Libabigail and ABI Laboratory are in agreement $73.4\%$ of the time for
libraries with changed file names have have only a single case where ABI
Laboratory reports a detected incompatibility that Libabigail does not.
Comparing against the symbols predictor, the two are in agreement $74.3\%$ of
the time. Quite surprisingly, symbols reports 7 incompatibilities that ABI
Laboratory does not. For the cases of libraries with no change in file name,
shown in Table 5, the agreement percentages increase to $95.6\%$ and $82.6\%$,
respectively. This time, ABI Laboratory reports 63 incompatible libraries not
labelled as such by Libabigail, and symbols reports one such case.

As noted above, we encountered many difficulties while executing ABI
Laboratory, and the technical issues continued as we attempted to retrieve its
reports for the cases where it disagrees with the other two predictors. The
tool merely reported that it was unable to find tens (sometimes hundreds) of
unnamed symbols. We carefully examined the inputs and found no issues. In
particular, the same libraries were readily processed by the other tools. At
this time, we are unable to say anything further about the types of checks ABI
Laboratory uses and how they differ from the other tools presented here.

Table 4: Prediction counts for the three-predictor case when file names changed. | symbols | ABI Laboratory
---|---|---
Libabigail | compatible | not comp. | compatible | not comp.
compatible | 5 | 0 | 4 | 1
not comp. | 41 | 63 | 28 | 76
ABI Laboratory |
compatible | 25 | 7 |
not comp. | 21 | 56 |
Table 5: Prediction counts for the three-predictor case when file names did not change. | symbols | ABI Laboratory
---|---|---
Libabigail | compatible | not comp. | compatible | not comp.
compatible | 3075 | 0 | 3012 | 63
not comp. | 747 | 181 | 114 | 814
ABI Laboratory |
compatible | 3125 | 1 |
not comp. | 697 | 180 |

###  4.2 Filename and SONAME Changes

Returning back to the two-predictor cases, there are clearly instances when
the file name change (or lack thereof) does not reflect our assumed meaning.
The libtools guidelines provide a second indicator for this purpose: the
SONAME. This is a simple string containing the semantic version of the library
based on a major, minor, and patch level (e.g., 3.2.1). For library authors
that use libtools to build their projects, the SONAME is intended as an
automatic means of changing the generated file name of the library. Of the
tools we explore, only Libabigail uses the SONAME as a criterion for ABI
compatibility- where a change in SONAME is considered an ABI break. We explore
this for the two-predictor case in Table 6. We see that a change in the file
name and a change in SONAME are in agreement $57.5\%$ of the time. However,
the SONAME is changed if and only if the file name is changed in just $27.4\%$
of cases. There are only a few libraries labelled as incompatible solely
because of an SONAME change: 45 ($4.6\%$) when the file name changed and 48
($0.7\%$) when it didn’t.

Table 6: Change in SONAME for libraries with two predictors. | SONAME
---|---
File name | changed | unchanged
changed | 230 | 609
unchanged | 52 | 1324

We are able to correlate the many other ABI incompatibility checks Libabigail
performs with the change in SONAME. Table 7 lays out these results. We note
that the percentages represent the fraction of reported breakages containing
at least one of the features. Because a library may have many detected
incompatibilities, these fractions do not add to $100\%$. For example,
$52.6\%$ of all reported incompatibilities in libraries having both the file
name and SONAME changed have at least one function removed, but could also
have functions with changed signatures. From the summary column, we see that,
unsurprisingly, function signature changes dominate the space of reported ABI
incompatibilities. Unexpectedly, modifications of enumerator values is the
second-largest category followed by global variables and then virtual tables
(vtable). The dearth of results for libraries without a change in file name
but with a change in SONAME is due to $92.3\%$ (N=48) of them being labelled
as incompatible solely because of a change in SONAME.

Table 7: Frequency of ABI breakage types detected by Libabigail for two-
predictor cases.

| Filename Change | Filename No Change |
---|---|---|---
Feature |  | SONAME Change | No Change | SONAME Change | No Change | Total
functions | removed | 52.6% | 32.0% | 7.7% | 48.9% | 43.7%
changed | 48.7% | 80.5% | 0 | 28.0% | 43.9%
$\Delta$subtype | 39.6% | 64.2% | 0 | 6.6% | 25.7%
$\Delta$return type | 30.0% | 51.9% | 0 | 15.0% | 26.3%
vtable | added | 18.3% | 25.0% | 0 | 0.3% | 8.9%
removed | 16.5% | 12.2% | 0 | 0.3% | 5.2%
enumerator | added | 27.8% | 7.9% | 0 | 11.3% | 11.8%
removed | 25.7% | 4.3% | 0 | 11.0% | 10.4%
changed | 26.5% | 8.0% | 0 | 11.6% | 11.9%
global variables | removed | 19.6% | 6.7% | 7.7% | 12.6% | 11.6%
changed | 7.8% | 3.9% | 0 | 5.4% | 5.1%

Note: We use the shorthand $\Delta x$ to represent “a change in x” to save
space.

Table 8: Comparison with ABI laboratory Kind of ABI Change Detected | ABI Lab | This Work
---|---|---
Removing functions from the library | 55.67% | 12.7%
Changing virtual table structure | 1.38% | 2.6%
Changing number or order of parameters | 2.07% | 12.8%
Changing type of parameter | destructive | 9.55%
non-destructive | 15.89%
Changing type of return value | destructive | 0.12% | 7.6%
non-destructive | 3.27%
Changing parameter subtype |  | 7.5%
Adding/removing “static” specifier | 0.15% |
Changing values in enumeration types or macros | 7.47% | 3.6%
Global variables | removed |  | 4.0%
changed type |  | 1.5%

###  4.3 Breakage Types

While it was out of scope for our study to do a detailed categorization of
types of breakages, we can classify them into the categories based on whether
features (e.g., functions, vtable, enumerator, global variables) were removed,
changed, or added 7. For each of these cases, we can show example cases for
illustrative purposes.

####  4.3.1 Function parameters

An added or removed function speaks for itself as it means a symbol "mangled
string" in C++ is either missing after a library change or newly present. As
previously shown , a simple change As shown previously 1.2, a function change
that would lead to a breakage can be as simple as changing a parameter type.

##### Structure layout change

This kind of change might not be obvious on the level of the function, but
could lead to a breakage. As an example, imagine we have a function that takes
a structure S with members of an int and a double, and used by function foo:

⬇

struct S { int x; double d; }

void foo(S s);

If we were to then change the structure S to instead have a char p (as shown
below), although the signature of foo does not change, the structure itself
would be a different size with an expectation of different parameter types.
Passing the wrong structure type to either the new or old function would come
down to passing a pointer when a double is expected, or vice versa. In both
cases the application would not function as expected.

⬇

struct S { int x; char *p; }

void foo(S s);

##### Subtype change

This example is similar to the previous, however there is a layer of
indirection through the inheritance of the Base class.

⬇

struct Base { int x; };

struct Derived : Base {};

void foo(Derived *d) {}

In this example, if we were to change the Base class member from an int to a
double, it would also not show up in the function signature for foo, nor would
it show up in the Derived structure. This would lead to undefined behavior.

⬇

struct Base { double x; };

struct Derived : Base {};

void foo(Derived *d) {}

##### Return type change

We start with a structure S that is returned by a function foo.

⬇

struct S { int x; double d; }

S foo();

We later change that return type to be an int. Since return types are not
encoded in mangled strings, we would not see this change on the level of the
symbol. It would also lead to undefined behavior.

⬇

int foo();

####  4.3.2 Virtual tables

##### Adding a virtual function entry

We start with a structure Base that has one entry in its virtual table, a
function named bar. Another function, foo takes a pointer to a Base as the
only parameter.

⬇

struct Base { virtual void bar(); };

void foo(Base *b) {}

Now let’s imagine that base has another entry added to its virtual table – a
function named baz. Notice that the function foo that uses the Base has
superficially not changed.

⬇

struct Base {

virtual void bar();

virtual void baz();

};

void foo(Base *b) {}

This is a breaking change because virtual tables are part of the layout of the
structure Base, and the function is now expecting a different layout that
might not be obvious. The order in which virtual functions are added to the
table is not specified, meaning the compiler can choose to do what it wants.
This means that the original entry for function bar might result in a
different location.

####  4.3.3 Enumerator

##### Adding an enumerator

We start with an enum, Foo, that has two enumerators, and that is used by a
function foo as the only parameter.

⬇

enum class Foo {ONE,TWO};

void foo(Foo f);

If we then add a third enumerator to Foo, the function signature does not
change, however the underlying data that is expected has. This would lead to
undefined behavior. This is another kind of layout change, however distinct
because enumerations do not have sizes. An enumerator is a label that contains
a value, which is an implementation detail.

⬇

enum class Foo {ONE,TWO,THREE};

void foo(Foo f);

This means that we can tell the enumerations to take on certain values, as we
do below.

⬇

enum class Foo {ONE=17,TWO,THREE};

void foo(Foo f);

While this might not be an ABI error in the traditional sense, this would be a
behavioral change, akin to an integer parameter taking on different values. It
would be ultimately important to signal these kinds of behavioral changes,
regardless of how they are classified.

####  4.3.4 Global Variables

##### Changing a global variable

While the classification of global for a variable indicates visibility, the
linkage of the variable means that it is part of the ABI surface. In layman’s
terms, a library might expect to use a global variable provided by another
one. In the example below, both x and y have global visibility. However, in
adding static to y, this means it has internal linkage and can no longer be
seen by the ABI surface.

⬇

int x=1;

static int y=2;

Both of these would be called global variables, however the second would not
be seen in the context of an ABI. In that int x is seen, a change to its type
would be breaking. The kinds of changes previously discussed in the previous
sections could also apply here.

##  5 Discussion

We conducted an experiment to compare tool predictions across several versions
of the Fedora operating system, focusing on comparing the strengths and
weaknesses of the tools. Although we’ve done a quantitative analysis, we are
notably still doing a qualitative comparison of these tools. Our results are
limited based on the dearth of tools in this space, and as a result, we
recognize that much of our qualitative analysis ultimately focused on
Libabigail, which is arguably the most mature tool in the space.

Our decision to focus on the two-predictor case is not only driven by having
49.5% more results, but also a desire to focus on the tool that is most widely
used in developer communities (e.g., Google’s Android team and Red Hat Linux).
In stark contrast, at the time of this writing, the ABI Laboratory abi-
compliance-checker [30] has not been updated on the GitHub repository for over
a year. We believe there is valuable knowledge embedded in the tool, and hope
that either the maintainer can continue working on it, another set of
maintainer(s) can contribute, or the knowledge can be extracted and moved
elsewhere. It is not clear the extent to which the community or original
developers are interested in such an effort.

We found that, on average, about 12$\%$ of libraries have a different file
name between OS versions and account for $37.9\%$ of all libraries with a
least one ABI break reported. Interestingly, there is nearly uniform N-way
agreement between the predictors when looking across the subsets of changed
and unchanged ($\sim 75\%$) file names. Recall that the symbols predictor only
analyzes the sets of exported symbols (i.e., those in .symtab and .dynsym ELF
sections). This means that the cases where the file name does not change, the
simple symbol name check discovers $\sim 75\%$ of libraries that would be
labelled as incompatible by the more in-depth analyses performed by Libabigal
and ABI Laboratory. Additionally, we found the symbols predictor to be about a
thousand times faster than the other two with its execution on the order of a
few milliseconds compared to the other two on the order of a few seconds. This
suggests that a simple symbols comparison could both be an effective (but not
complete) and fast means of examining a very large body of libraries like a
complete Linux distribution in near real-time.

Given an approximately 80% of agreement of the symbols predictor with
Libabigail when the file name changes, our results suggest that the majority
of cases are related to symbol names. Our results suggest that SONAME and
filename changes are not reliable to indicate change, as it can go either way
to have a library with a changed name that has not actually changed, or a
library with an unchanged name that has. We suspect that the underlying SONAME
is perhaps overlooked by developers, the actual practice is being deprecated
in favor of using a file name, or that developers are treating the SONAME
differently than as defined in the libtool guidelines. Given this result, it
appears that software today does not reliably use the SONAME to indicate an
ABI breaking-change, and thus we suggest that the SONAME is not a good
indicator of such a break. Between the two, SONAME is slightly less reliable
than filename. A better alternative, given no tool to do an assessment, might
be to look at symbols. We suggest to the interested developer that needs to
assess ABI breaks across a large set of pairs (where computational complexity
and running time is an issue) that a simple comparison of symbol sets can get
them most of the way there. Given the large speed difference (three orders of
magnitude faster) between the simple symbols predictor and the others, the
developer can detect a large number of problems substantially faster.

In comparing our results to previous work (see Table 8) this previous work
[18] found that approximately 55% of breakages were due to removing functions
in a library, whereas we found approximately 12%. We believe this discrepancy
could result from a difference in the size of datasets used for the analysis.
The previous work used a dataset of approximately 250, and our dataset size is
a factor of 27 times that. We are interested in comparing the result that
inspects destructive vs. non-destructive types of changes, however while our
primary tool Libabigail has that information, it is not presented in a
machine-readable way. A substantially useful addition to the Libabigail
software would be to better expose results in a machine-readable form. Both
our analyses and previous work detect enumerator breakages, however our work
detects approximately half the quantity, percentage wise. The also could be
attributed to the size differences of the datasets. As both results suggest
this is the second most abundant type of change, our findings suggest that
enumerators are an important place where ABI checks should be performed.

The third most common type of breakage, as reported by Libabigail, are from
global variables (approximately 5.5%). We find it interesting that this
breakage is not detected by ABI Laboratory. This case represents an example of
how tools choose to create an internal model of ABI that is likely to differ
with other tools. Work to better standardize the underlying model that tool
implementations use would be a suitable contribution to this space. Finally,
the fourth most common breakage for both analyses is with respect to virtual
tables. We find a higher fraction of libraries that exhibit changes in the
structure of these tables ( 7.5% of all libraries) than the previous analysis,
and we are also able to detect changes in derived types in a type hierarchy
(e.g., changing parameter subtype), which was not stated in the previous
analysis. Because the previous work was based on source-level analysis, they
were able to directly detect the addition or removal of the static specifier
functions. This type of change would manifest in a binary analysis either as a
added/removed function symbol or a add/removed function parameter (via the
hidden this parameter in C++).

###  5.1 Limitations

Although we have control in the compiler and general environment when using a
container for our experiments, we don’t have control over the
microarchitecture of the public GitHub runners. In general, although there are
known ABI issues when changing underlying microarchitecture (Section 3.2.3 of
[47]), to our knowledge, the current tooling available is not able to check at
this level of detail, so it would not impact our superficial analysis. For
provenance, the runners that we encountered included broadwell, cascade lake,
haswell, and generic x86_64_v4 (unspecified).

We recognize that our results might be biased based on using a production-
ready set of binaries provided by a single package manager for the Fedora
operating system. We suspect that looking at binaries from a different package
manager or Linux distribution could result in different results. Finally, our
symbols analyses assume binaries compiled on a Unix type system. The tools
that we chose are also stated to be primarily intended for ELF binaries.
Extending this work to other types of binaries or a Windows system would
require implementing this process for a different binary format or algorithm,
in the case of Windows.

##  6 Conclusion

In this paper we have reviewed well-known tools for checking ABI compatibility
of software. As the number of architectures, compilers, and languages
increases, having an ability to model ABI compatibility is becoming
increasingly important, along with having automated means to assess it. We are
pleased to share this early work that compares agreement between ABI
prediction tools, and hope that it inspires other developers to contribute to
work in space.

Although ABI is a niche topic of interest in the developer space, we strongly
recommend that developers working on C/C++ packages add basic ABI checks to
their continuous integration as a good practice. To support this, we have
created a GitHub action [48] that easily runs Libabigail, and allows the
developer to check some number of libraries in development (in a pull request)
against a previous latest release, and the current main branch. While there
aren’t many tools in the space, we champion using a tool like Libabigail, and
specifically for this tool, hope to see continued development toward having
machine-readable output to allow for easier interpretation of the results.
While ABI breaks are not avoidable, we believe these checks to be important so
developers have awareness about updates to their software that might break
ABI.

We are concerned about the incredibly small number of projects working in this
space given how fundamental this problem is in modern software. Application
binary interfaces changing and resulting in an error that can hinder usage,
performance, or portability is an inescapable problem. Having these kinds of
tools embedded in build tools would be incredibly beneficial to the process.
If this kind of work is valuable or essential for development (and we suspect
it is the case that there are far more wanting to use this kind of tooling
than develop it) there is a huge opportunity to better form a community
focused around shared goals for development. Based on our observation of
likely different underlying models between the tools that we tested, we
suggest moving forward that work is needed in this space to better categorize
and communicate more complex ABI breakages, both for understanding and mapping
of the space.

#### Contributions

Author VS designed the original experiment, implemented the spliced software
framework and automation (containers and workflows along with software) to
interact with all predictors, ran the experiments, and wrote the first and
second drafts of the manuscript. Author TH contributed significantly to the
experiment design, outline for the paper, final draft, and analysis and
interpretation of results.

#### Acknowledgements

We are thankful to Ben Woodard, Jim Kupsch, and Bart Miller for feedback on an
initial draft, and specifically to Ben Woodard and Dodji Seketeli for their
work on Libabigail. Thank you also to Matt Legendre and Todd Gamblin for
providing a funding structure to support this work via the BUILD SI project.
Finally, thank you to "Old Man Cat," "Tiny Sir", "Enzo" who was consistently
present for pair programming and work sessions, and also was consistently no
use at all.

This work was performed under the auspices of the U.S. Department of Energy by
Lawrence Livermore National Laboratory, LLNL-JRNL-842973-DRAFT.

## References

  * Tobias Becker, Robert Haas, Johannes Schemmel, Steve Furber, Sagar Dolas [2022] Tobias Becker, Robert Haas, Johannes Schemmel, Steve Furber, Sagar Dolas.  Unconventional HPC architectures, April 2022.
  * Wilkes et al. [1951] M.V. Wilkes, D.J. Wheeler, and S. Gill.  _The Preparation of Programs for an Electronic Digital Computer: With Special Reference to the EDSAC and the Use of a Library of Subroutines_ , page 72.  Addison-Wesley mathematics series. Addison-Wesley Press, 1951.  URL https://books.google.com/books?id=HwkuAAAAIAAJ.
  * 198 [1988a] Infoworld, 1988a.  URL https://books.google.com/books?id=r4k_AQAAIAAJ.
  * IDG Network World Inc [1987] IDG Network World Inc.  _Network World_.  IDG Network World Inc, November 1987.
  * IDG Enterprise [1989] IDG Enterprise.  _Computerworld_.  IDG Enterprise, February 1989.
  * InfoWorld Media Group and Inc. [1989] InfoWorld Media Group and Inc.  _InfoWorld_.  InfoWorld Media Group, Inc., October 1989.
  * Conference and Exposition [1989] National Computer Graphics Association (U.S.). Conference and Exposition.  _NCGA ’89 Conference Proceedings: 10th Annual Conference and Exposition Dedicated to Computer Graphics : April 17-20, 1989, Philadelphia, Pennsylvania_.  Number v. 2 in NCGA ’89 Conference Proceedings: 10th Annual Conference and Exposition Dedicated to Computer Graphics : April 17-20, 1989, Philadelphia, Pennsylvania. National Computer Graphics Association, 1989.  URL https://books.google.com/books?id=5FmvAAAAIAAJ.
  * 198 [1988b] Computer systems science and engineering, 1988b.  URL https://books.google.com/books?id=YG9VAAAAMAAJ.
  * 198 [1989] Edp in-depth reports, 1989.  URL https://books.google.com/books?id=BeQcAQAAMAAJ.
  * AUUG and Inc. [1988] AUUG and Inc.  _AUUGN_.  AUUG, Inc., October 1988.
  * 198 [1988c] Mini-micro systems, 1988c.  URL https://books.google.com/books?id=wFNVAAAAMAAJ.
  * sea [2022] 2022\.  URL {https://www.google.com/search?q=%22application+binary+interface%22&biw=1540&bih=807&source=lnt&tbs=cdr%3A1%2Ccd_min%3A1%2F1%2F2000%2Ccd_max%3A12%2F31%2F2022&tbm=bks}.
  * Society et al. [1989] IEEE Computer Society, Institute of Electrical, and Electronics Engineers.  Digest of papers, 1989.  URL https://books.google.com/books?id=WPpVAAAAMAAJ.
  * Predicasts [1989] inc Predicasts.  Predicasts technology update, 1989.  URL https://books.google.com/books?id=2GkUAQAAMAAJ.
  * noa [2021a] x86-64-psabi pdf, 2021a.
  * Atkinson [2011] Kevin Jay Atkinson.  Application binary interface compatibility through a customizable language, 2011.
  * Ditu [2015] Bogdan Ditu.  Model-based function call code generation and stack management in retargetable compilers: Application binary interface modeling of stack layout and function call sequence.  In _2015 20th International Conference on Control Systems and Computer Science_ , pages 883–888, 2015.  doi: 10.1109/CSCS.2015.38.
  * Ponomarenko and Rubanov [2011] Andrey Ponomarenko and Vladimir Rubanov.  Automatic backward compatibility analysis of software component binary interfaces.  In _2011 IEEE International Conference on Computer Science and Automation Engineering_ , volume 3, pages 167–173, 2011.  doi: 10.1109/CSAE.2011.5952657.
  * Agrawal et al. [2015] Varun Agrawal, Abhiroop Dabral, Tapti Palit, Yongming Shen, and Michael Ferdman.  Architectural support for dynamic linking.  _SIGARCH Comput. Archit. News_ , 43(1):691–702, mar 2015.  ISSN 0163-5964.  doi: 10.1145/2786763.2694392.  URL https://doi.org/10.1145/2786763.2694392.
  * noa [2021b] elfcall - generate call graph data for an elf binary.  https://vsoch.github.io/elfcall/getting_started/background.html, 2021b.  Accessed: 2022-10-11.
  * noa [2021c] build-abi-test-mathclient: Testing case for abi break with a math client.  https://github.com/buildsi/build-abi-test-mathclient, 2021c.
  * Burow et al. [2018] Nathan Burow, Derrick McKee, Scott A. Carr, and Mathias Payer.  CUP.  In _Proceedings of the 2018 on Asia Conference on Computer and Communications Security_. ACM, may 2018.  doi: 10.1145/3196494.3196540.  URL https://doi.org/10.1145%2F3196494.3196540.
  * Krause [2021] Philipp K. Krause.  Efficient calling conventions for irregular architectures, 2021.  URL https://arxiv.org/abs/2112.01397.
  * elf [1995] Tool interface standard (tis) executable and linking format (elf) specification, 1995.  URL https://refspecs.linuxfoundation.org/elf/elf.pdf.
  * noa [2022a] Dwarf home.  https://dwarfstd.org/, 2022a.  Accessed: 2022-10-11.
  * Hemel [2013] Armijn Hemel.  Research into dynamically linked elf executables/libraries.  https://github.com/armijnhemel/conference-talks, 2013.
  * noa [2015a] Index of /sourceware/libabigail/.  https://mirrors.kernel.org/sourceware/libabigail/, 2015a.  Accessed: 2022-10-11.
  * noa [2015b] My project: The ABI generic analysis and instrumentation library.  https://sourceware.org/libabigail/, 2015b.  Accessed: 2022-10-11.
  * abi [2015] abidiff — libabigail documentation.  https://sourceware.org/libabigail/manual/abidiff.html, 2015.  Accessed: 2022-10-11.
  * noa [2012a] ABI compliance checker.  https://lvc.github.io/abi-compliance-checker/, 2012a.  Accessed: 2022-10-11.
  * Ponomarenko [2012a] Andrey Ponomarenko.  Rulesbin.xml abi compliance checker.  https://github.com/lvc/abi-compliance-checker/blob/master/modules/RulesBin.xml, 2012a.  Accessed: 2022-10-11.
  * noa [2012b] ABI tracker: Tested libraries.  https://abi-laboratory.pro/tracker/, 2012b.  Accessed: 2022-10-11.
  * Ponomarenko [2012b] Andrey Ponomarenko.  abi-dumper: Dump ABI of an ELF object containing DWARF debug info.  https://github.com/lvc/abi-dumper, 2012b.
  * Ponomarenko [2012c] Andrey Ponomarenko.  abi-compliance-checker: A tool for checking backward API/ABI compatibility of a C/C++ library.  https://github.com/lvc/abi-compliance-checker, 2012c.
  * Ponomarenko [2012d] Andrey Ponomarenko.  abi-tracker: A tool to visualize ABI changes timeline of a C/C++ software library.  https://github.com/lvc/abi-tracker, 2012d.
  * Bendersky [2012] Eli Bendersky.  pyelftools: Parsing ELF and DWARF in python.  https://github.com/eliben/pyelftools, 2012.
  * sym [2010] Symbol table (linker and libraries guide).  https://docs.oracle.com/cd/E19683-01/816-1386/chapter6-79797/index.html, 2010\.  Accessed: 2022-10-11.
  * noa [2021d] splice-experiment: Preparing for the splice experiment.  https://github.com/buildsi/splice-experiment, 2021d.
  * noa [2021e] splice-experiment-runs: Results for buildsi splice experiment.  https://github.com/buildsi/splice-experiment-runs, 2021e.
  * noa [2021f] splice-experiment-results: Run results for the splice experiment.  https://github.com/buildsi/splice-experiment-results, 2021f.
  * noa [2014] DNF debuginfo-install plugin — dnf-plugins-core 4.3.1-1 documentation.  https://dnf-plugins-core.readthedocs.io/en/latest/debuginfo-install.html, 2014\.  Accessed: 2022-10-11.
  * noa [2021g] User guide — spliced 0.0.18 documentation.  https://buildsi.github.io/spliced/getting_started/user-guide.html, 2021g.  Accessed: 2022-10-11.
  * noa [2020] Storing workflow data as artifacts.  https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts, 2020\.
  * hul [2017] Stack smashing protector.  https://wiki.osdev.org/Stack_Smashing_Protector, 2017.  Accessed: 2022-10-12.
  * lib [1997] Updating version info (libtool).  https://www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html, 1997\.  Accessed: 2022-10-12.
  * noa [2007] Scripts (LD).  https://sourceware.org/binutils/docs/ld/Scripts.html, 2007.  Accessed: 2022-10-12.
  * noa [2022b] x86-64 psABI.  https://gitlab.com/x86-psABIs/x86-64-ABI, 2022b.  Accessed: 2022-10-14.
  * lib [2021] libabigail-action: Ensure changes to your code don’t break ABI (via a GitHub action!).  https://github.com/buildsi/libabigail-action, 2021.


# With ChatGPT, do we have to rewrite our learning objectives - CASE study in
Cybersecurity

Peter Jamieson  Department of ECE
Miami University, Ohio
Email<EMAIL_ADDRESS>Suman Bhunia, Dhananjai M. Rao  Department of CSE
Miami University, Ohio
Email: {bhunias<EMAIL_ADDRESS>
###### Abstract

With the emergence of Artificial Intelligent chatbot tools such as ChatGPT and
code writing AI tools such as GitHub Copilot, educators need to question what
and how we should teach our courses and curricula in the future. In reality,
automated tools may result in certain academic fields being deeply reduced in
the number of employable people. In this work, we make a case study of
cybersecurity undergrad education by using the lens of “Understanding by
Design” (UbD). First, we provide a broad understanding of learning objectives
(LOs) in cybersecurity from a computer science perspective. Next, we dig a
little deeper into a curriculum with an undergraduate emphasis on
cybersecurity and examine the major courses and their LOs for our
cybersecurity program at Miami University. With these details, we perform a
thought experiment on how attainable the LOs are with the above-described
tools, asking the key question “what needs to be enduring concepts?” learned
in this process. If an LO becomes something that the existence of automation
tools might be able to do, we then ask “what level is attainable for the LO
that is not a simple query to the tools?”. With this exercise, we hope to
establish an example of how to prompt ChatGPT to accelerate students in their
achievements of LOs given the existence of these new AI tools, and our goal is
to push all of us to leverage and teach these tools as powerful allies in our
quest to improve human existence and knowledge.

Our general hypothesis is that the LOs of early first- and second-year courses
will need to remain as they are in the revised Bloom’s Taxonomy verbiage, but
the rate at which we can go deeper in the LO can be aided with ChatGPT tools.
Additionally, in our use of UbD to create courses, we believe the exercise
will need to be done at all levels of the undergraduate curriculum such that
from the LOs and the respective levels we can then devise valuable activities
and assessments that will help students develop into valuable thinkers as they
navigate their future careers. We will argue that similar to mathematics where
the 3 steps in mathematics (1. Devise 2. Compute 3. Evaluate Correctness of
Result) that just like computers have severely reduced the needs of step 2,
the same model can be thought of for computing education where ideas and
skills such as syntax and algorithmic implementation might be shifted more to
the AI-based assistive tools, and the learner/user needs to be able to create
the design (analogous to step 1 in mathematics) and evaluate (analogous to
step 2 in mathematics) the results produced by the tools.

##  I Introduction

Technological advancements can have an influence on how we approach education,
and the emergence of Artificially Intelligent (AI) chatbot tools such as
ChatGPT will result in the next wave of debates and questions on how the
existence of these tools will change our educational approaches. ChatGPT is a
natural language chatbot tool (where a chatbot is an autonomous robot that
will respond to user queries) that has been trained on large language models
such that the tool can respond to questions and generate user-requested
solutions. For example, you can ask ChatGPT to write a story in the voice of
your favorite author and define a simple plot and characters. ChatGPT will
create that story, and in many cases, the story will be reasonably written.
Additionally, ChatGPT and other tools such as Github’s Copilot provide users
with the ability to help them design and code in a variety of programming and
design languages to build various user-requested functionality. For example,
we have used ChatGPT to design Verilog HDL modules of a Radix-2 SRT divider
with reasonable success (noting that the tool is not perfect and expertise is
needed for a well-formed design).

These later capabilities have caused those of us who sit in the computing
education space into debates about how the existence of these tools changes
what and how we teach. In this work, we propose to approach this debate from
the perspective of student learning outcomes (LOs) as defined using verbiage
from Bloom’s Taxonomy [1]. With LOs for a particular curriculum (our example
case will focus on cybersecurity), we can pose the simple question of how
these LOs might change, and what additional LOs should be added to the
curriculum. To aid in this case study, we also will guide our discussion with
the help of the useful concept filter described by Wiggins and McTyghe in
“Understanding by Design” to classify the importance of LOs. By approaching a
curriculum and the courses it comprises with these perspectives, our goal is
to demonstrate how we might approach integrating AI tools into our curricula
and education.

For this case study, we use existing cybersecurity organizational schemes to
extract the “enduring understanding” concepts. This organizational scheme will
then look at some of the LOs asking if the LO can be “accelerated” given the
available AI tools. Next, we will look at our existing cybersecurity curricula
with courses and look at each LO mapping and where it sits in Bloom’s
Taxonomy. This will allow us to see what are the LOs for cybersecurity
education and ask the question, “Should they be leveled up?” We will provide
ChatGPT example usages on how students can be taught to use the tool to deepen
their understanding of a knowledge unit. These examples show that much of what
we do as teachers may be offset to ChatGPT given that they learn to use the
tool.

##  II Background and Discussion

Our approach in this paper is to use the lenses of Bloom’s Taxonomy [1] and
Understanding by Design’s [2] filter of concepts to understand the Learning
Outcomes (LOs) in a particular curriculum and its respective courses. We have
used a similar approach in our previous work [3] to help educators evaluate if
they should spend time creating a game to teach a concept.

Prescribing the approach that student LOs should be written using Bloom’s
Taxonomy verbs allows us to rank an LO using Bloom’s Taxonomy. This then
allows us to question if the existence of an AI tool, which for the remainder
of this paper we assume is ChatGPT as of April 2023, changes either the need
for the LO or can we accelerate a student learning of the LO and raise the
Bloom’s Taxonomy verb-level. Note, this approach is simplistic, and first, we
need to have a deeper understanding of Bloom’s Taxonomy and a little knowledge
of how curricula are created.

###  II-A Bloom’s Taxonomy and Learning Outcomes

We define learning in this work, using ideas from [4], [5]: “Learning is the
stabilization of neural networks based on a desired response”. This definition
allows us to relate the idea of learning things to the necessity of trying and
doing things, since if the neurons are not firing (by thinking and doing) then
your neurons are not wiring (Donald Hebb in 1949 stated, “neurons that fire
together wire together”).

A _Learning Outcomes_ (LO) is an educational goal for a learner such that they
will be able to cognitively perform the objective once they have learned it.
Typically, an LO is described by a cognitive process taken from Bloom’s
Taxonomy [1] and applying the process to some content as related to the field
of study. Bloom’s taxonomy provides a hierarchy of cognitive processes as
defined from “lower-order thinking skills”, such as recall and classification,
to “higher-order thinking skills” such as creating or planning. The “higher-
order” thinking skills tend to be what society wants of our workers, but these
higher-order skills are much more challenging to develop and take
significantly more time than the lower-order thinking skills to teach and
learn because of the profession nature of learning. Figure 1 shows two
dimensions of Bloom’s taxonomy in terms of cognitive and knowledge dimensions
[6].

Figure 1: The two dimensions of Bloom’s taxonomy [6]

This distinction between “higher” and “lower” can, however, confuses educators
to somehow evaluate the quality of an LO based on where it sits in the
taxonomy. As illuminated by Lang [7] professors can fall into the trap of
simply wanting to teach “higher-order thinking skills” and assume that the
lower skills can simply be looked up using technology such as search engines
and the like (including tools like ChatGPT). The reality is to achieve skills
such as “create” and “plan” in Bloom’s taxonomy requires that a human can keep
and think about ideas and concepts within a field of study that requires the
“remember” and “understanding”. In this work, we are asking this question in
terms of accelerating an LO to a higher Bloom’s level, but we are aware of
this pitfall, and will hopefully provide insight into how the new technology
modifies the space.

A university-level course in a particular subject matter will include a number
of LOs. Wiggins and McTighe’s book “Understanding by Design” [2] examines
course design by using LOs as the starting point for course design, and then
designing a course in reverse from these starting LO goals.

###  II-B Enduring Concepts and Learning Outcomes

The LOs need to be defined in a sentence where a verb is taken from Bloom’s
taxonomy [1] and the object of the sentence describes some knowledge or skill.
For example, a sample LO might be, “A student should be able to _construct_ a
program that will then generate a desired output with a proper sequence of
program statements”. This objective is _apply_ in Bloom’s taxonomy cognitive
process dimension [1] and is _procedural_ in the knowledge dimension [6]. From
the taxonomy, this learning objective would be considered a “middle-order
thinking skill”.

To further determine how important the LO is, we use Wiggins and McTighe’s
filter [2]:

  1. 1.

worth being familiar with

  2. 2.

important to know and do

  3. 3.

enduring understanding

where the importance of a LO is more important to a student’s future as the
above number increases above. This filter allows a teacher to consider their
LO and qualify how important it is. The reason this is useful is that just
because an objective ranks as a “low-order thinking skill” if it is an
enduring concept for that student in the respective field, then it is
important to emphasize and should be achieved by the student. Similarly,
depending on the time in the curriculum of a field, different ideas will need
to be learned to holistically progress in a field.

###  II-C Curriculum Design and Cybersecurity Curricula

Curriculum design is not a simple process by any means, and for most
educators, we can understand this by realizing that a single course is hard to
design, and a curriculum is a series of courses; hence it is of greater
complexity. For this reason, we will leave much of the previous research on
curriculum design with the exception of noting Sheenan’s work [8] that looked
at the two conceptual models of the curriculum being either a product model
(resulting in a student objective ending) versus the process model (resulting
in a student with capabilities). We argue that the majority of our technical
curricula are mostly process models.

For cybersecurity curricula, a recent paper by Mouheb et. al. [9] broadly
reviews the range of attempts to design cybersecurity curricula. This broad
range of designs sits both in and outside of cybersecurity curricula for
computer science. For the sake of this work, we will focus on undergraduate
cybersecurity in computer science to limit the breadth of this space. Of
interest, in Mouheb’s work is the three main cybersecurity activities of
Prepare, Defend, and Act [10] where these activities can be grouped into three
distinct phases _Prevention_ , _Detection_ , and _Response_ [11].

Harrison et. al. [12] have looked at Bloom’s approach to the design of an
information technology curriculum as mapped to Bloom’s taxonomy. Their
approach to designing the curriculum provides a nice visual representation of
how we could approach asking our technological change questions about the
existence of ChatGPT.

##  III Learning Outcomes and their link to a Cybersecurity Education

Using the classification of _P̱revention_ , _Ḏetection_ , and _Ṟesponse_ we
will start by looking at what LOs we believe are part of a modern
cybersecurity computer science curriculum. Here, we note the work on the
Cybersecurity Concept Inventory (CCI) which is currently being validated in
work by Poulsen et. al. [13] and by Herman et. al. [14]. Their concept
approach identifies concepts that can be validated, and with the validation of
this work, an important next step would be having learners progress through
our curriculum (ChatGPT enhanced and unenhanced) and evaluate our learners.

Table I: List of courses at Miami University to cover the Cybersecurity ACM guideline Course Number | Course Name
---|---
CYB 134 | Introduction to Cybersecurity
CYB 234 | Sysadmin & scripting
CYB 235 | Computer networks
CYB 236 | Data Security
CYB 331 | Software Security
CYB 332 | Organizational and Human Security
CYB 334 | Network security
CYB 335 | Defensive Security
CYB 435 | Offensive Security
Table II: List of recommended Cybersecurity topics and knowledge mapped to Miami’s curriculum and related Learning Outcome Bloom’s Taxonomy Level Knowledge Areas | Knowledge Units | Course coverage
---|---|---
Course Label | LO level
Data Security | Cryptography | CYB 236 | Apply
Digital Forensics | CYB 236 | Remember
Data Integrity and Authentication | CYB 334 | Apply
CYB 236 | Analyze
Access Control | CYB 334 | Apply
CYB 236 | Understand
Secure Communication Protocols | CYB 334 | Analyze
Cryptanalysis | CYB 236 | Understand
Data Privacy | CYB 236 | Understand
Information Storage Security | CYB 236 | Understand
Software Security | Fundamental Principles | CYB 331 | Understand
Design | CYB 331 | Understand
Implementation | CYB 331 | Apply
Analysis and Testing | CYB 331 | Apply
Deployment and Maintenance | CYB 331 | Understand
Documentation | CYB 331 | Understand
Ethics | CYB 331 | Understand
Component Security | Component Design | CYB 331 | Understand
Component Fabrication | CYB 331 | Apply
Component Procurement | CYB 331 | Remember
Component Testing | CYB 331 | Apply
Component Reverse Engineering | CYB 331 | Remember
Connection Security | Physical Media | CYB 235 | Remember
Physical Interfaces and Connectors | CYB 235 | Remember
Hardware Architecture | CYB 235 | Remember
Distributed Systems Architecture | CYB 235 | Remember
Network Architecture | CYB 235 | Analyze
Network Implementations | CYB 235 | Apply
Network Services | CYB 235 | Remember
Network Defense | CYB 334 | Apply
System Security | System Thinking | CYB 335 | Remember
System Management | CYB 335 | Understand
System Access | CYB 334 | Apply
CYB 234 | Understand
System Control | CYB 234 | Understand
System Retirement | CYB 335 | Remember
System Testing | CYB 435 | Apply
Example System Architectures | CYB 435 | Understand
Human Security | Identity Management | CYB 332 | Understand
Social Engineering | CYB 332 | Understand
Personal Compliance with Cybersecurity Rules/Policy/Ethical Norms | CYB 332 | Remember
Awareness and Understanding | CYB 332 | Remember
Social and Behavioral Privacy | CYB 332 | Understand
Personal Data Privacy and Security | CYB 332 | Understand
Usable Security and Privacy | CYB 332 | Understand
Organizational Security | Risk Management | CYB 335 | Understand
CYB 332 | Understand
Security Governance & Policy | CYB 435 | Understand
CYB 332 | Remember
Analytical Tools | CYB 335 | Understand
Systems Administration | CYB 335 | Apply
CYB 234 | Apply
Cybersecurity Planning | CYB 335 | Understand
CYB 332 | Remember
Business Continuity, Disaster Recovery, and Incident Management | CYB 335 | Remember
Security Program Management | CYB 335 | Understand
Personnel Security | CYB 335 | Remember
Security Operations | CYB 335 | Understand
Societal Security | Cybercrime | CYB 435 | Understand
CYB 134 | Remember
Cyber Law | CYB 134 | Remember
Cyber Ethics | CYB 435 | Understand
CYB 134 | Remember
Cyber Policy | CYB 134 | Remember
Privacy | CYB 234 | Apply
CYB 134 | Remember

Table I shows our Miami University’s cybersecurity courses [15] and the course
name. The topics covered for ACM’s cybersecurity curriculum [16] are presented
in Table II. In Columns 1 and 2, we show the knowledge area (a larger
grouping) and knowledge units, where there are several knowledge units per
area. Column 3 shows which course covers a particular knowledge unit, and
Column 4 lists the Bloom taxonomy level for the learning outcome for each
knowledge unit. We do not list the exact LO for each knowledge unit, and
instead, we write out the LO when we talk about how the verbiage is
potentially increased for a particular example.

The interesting information from this table is a summary view of each of all
the knowledge units and noting how “low-order” the majority of knowledge units
are with respect to Bloom’s Taxonomy. For the first three levels of “1 -
Remember”, “2 - Understand”, and “3 - Apply” in Table II, there are 22, 28,
and 14 instances of these, totaling 65 of the 66 listed course LOs. The
highest Bloom’s level for our curriculum includes 1 instance of “4 - Analyze”.

The main reason for the above result is the number of LOs listed for covering
the cybersecurity curriculum over nine courses. The reality is there is a lot
of ideas to learn about in the space of cybersecurity as a system, and its
complexities (beyond hardware and software) make technological and
psychological knowledge and skills a tremendous burden on this type of
education. As we’ve alluded to in this paper, the question is, will tools such
as ChatGPT allow us to push our learners and teachers into an accelerated
space?

##  IV GPT Examples where the Tool may allow us to Raise and Accelerate
Learning Outcomes

As noted in the last section’s analysis, the demands of remembering,
understanding, and applying new knowledge and skills are overwhelming for the
cybersecurity curriculum. This section provides a few examples of how an LO
can be linked with ChatGPT to accelerate and potentially raise the skill and
knowledge to a higher Bloom’s taxonomy level. As stated earlier, the goal is
not to simply move up in terms of Bloom’s Taxonomy levels as this is not a
feasible approach to learning and becoming an effective practitioner. Instead,
our goal is to provide examples of how ChatGPT can be used to extend what the
learner achieves in terms of the knowledge unit. In each example, we will show
the knowledge unit in the title, show the prompt and response and then discuss
how this allows the learner to push their knowledge further.

###  IV-A Data Security - Access Control

The original LO for this knowledge unit is “Identify and explain the
importance of basic system administration tasks” in the course CYB 234
111https://www.miamioh.edu/cec/academics/departments/cse/academics/course-
descriptions/cyb234/index.html.

Prompt: What is the Linux command to add execute permission to just group
users Response: The command to add execute permission to just group users in
Linux is ⬇ chmod g+x <filename> Here, “chmod” is the command used to change
file permissions, “g” stands for a group, and “x” adds execute permission to
the file for the group. Replace $<$filename$>$ with the name of the file for
which you want to add execute permission to the group.

Figure 2: Prompt and response for using Linux file permissions

Figure 2 shows an example of how to move from Understand to Apply in data
security access control. Instead of just describing how the system permissions
allow users to perform certain actions in Linux, we now have details of what
is the exact Linux syntax for the command line and can now test it in the
tool.

With ChatGPT acting as a tutorial agent, we can imagine a student being able
to test out on a Linux system how the command works and observe what the
system does to manage access control for a shared system.

###  IV-B Software Security Analysis and Testing

The original LO for this knowledge unit is “Describe and apply static and
dynamic software testing methods” in CYB 331 - Software Security.

Prompt: Does the following program have a memory leak problem? ⬇ void f() {
int *ptr = (int *) malloc(sizeof(int)); /* Do some work */ return; /* Return
without freeing ptr*/ } Response: Yes, the program has a memory leak problem.
The “malloc()” function allocates memory dynamically, but the “ptr” pointer is
never freed, which means that the memory remains allocated even after the
function returns. This can lead to memory leaks and other issues if the
program continues to allocate memory without freeing it. To fix this problem,
the memory allocated by malloc() should be freed before the function returns
using the free() function. Here’s an example of how to modify the code to fix
the memory leak: ⬇ void f() { int *ptr = (int ) malloc(sizeof(int)); /* Do
some work */ free(ptr); /* Free the memory before returning */ return; }

Figure 3: Prompt and response for analyzing some C-code

Figure 3 shows how we can use ChatGPT to accelerate a student’s understanding
of software security analysis from Apply to Analysis. With the ChatGPT, the
user can prompt the tool to analyze their code and make it more secure.

We imagine providing code to the student that shows a number of cases such as
memory leaks and buffer overflows that ChatGPT can help them analyze and
understand. Next, given a larger code base, either their own or a manufactured
case, could the learner use the tool to analyze the code, and then help the
student create a safer version of it.

###  IV-C Component reverse engineering

The original LO for this knowledge unit is “Explain how failures in
fundamental security design principles can lead to system vulnerabilities that
can be exploited as part of an offensive cyber operation” in CYB 331 -
Software Security.

Prompt: Reverse engineer the following code to C code: ⬇ fact(int): movl $1,
%eax cmpl $1 %edi jle .L1 .L2: movl %edi, %edx subl $1, %edi imull %edx, %eax
cmpl $1 %edi jne .L2 .L1: ret"" Response: Sure, here’s the equivalent C code
for the given x86 assembly code: ⬇ int fact(int x) { int result = 1; if (x <=
1) { return result; } else { do { result *= x; x\--; } while (x > 1); return
result; } } The fact function takes an integer argument x and returns the
factorial of x. Here’s how the x86 code translates to C code: • movl $1, %eax
sets the eax register to 1, which initializes the result variable to 1. • cmpl
$1, %edi compares the value of x with 1. • jle .L1 jumps to the .L1 label if x
is less than or equal to 1. This is the base case of the factorial function. •
.L2: is a label that marks the start of the loop. • movl %edi, %edx moves the
value of x into the edx register, which is used to calculate the next
iteration of the result. • subl $1, %edi decrements x by 1. • imull %edx, %eax
multiplies the current value of result (stored in the eax register) with the
current value of x (stored in the edx register), and stores the result back
into eax. • cmpl $1, %edi compares the new value of x with 1. • jne .L2 jumps
back to .L2 label if x is not equal to 1, which means that we need to continue
the loop to calculate the next iteration of the result. • .L1: is the label
that we jump to if x is less than or equal to 1. This is the base case of the
factorial function. • ret returns the result of the function to the caller.

Figure 4: Prompt and response for analyzing some C-code

Figure 4 shows what ChatGPT provides when prompted to reverse engineer some
assembly code. This allows students to go beyond remembering what reverse
engineering is and into the space of understanding what potentially is
possible with using reverse engineering.

In this space, the tool provides rudimentary capabilities in reverse
engineering. The question is, can students reverse engineer other more complex
pieces of code, and similarly, obfuscate their own code examples such that
they can fool this simple tool?

###  IV-D Connection Security - Distributed System Architecture

The original LO for this knowledge unit is “Describe application layer
security protocols” in CYB
334222https://www.miamioh.edu/cec/academics/departments/cse/academics/course-
descriptions/cyb334/index.html.

Prompt: Write a command that will find all computers in a network running
MySQL Response: Assuming you have nmap installed on your system, you can run
the following command: ⬇ sudo nmap -p 3306 \--open -sV <network-address>
Replace ¡network-address¿ with the IP address or the network address of the
network that you want to scan for devices running MySQL. For example, if the
network address is 192.168.0.0/24, you would run: ⬇ sudo nmap -p 3306 \--open
-sV 192.168.0.0/24 This command will scan the IP addresses in the specified
network range and check if port 3306 (default MySQL port) is open on each of
them. It will also attempt to determine the version of MySQL running on each
device. Note that you may need to modify the IP address range according to
your network configuration. Additionally, the sudo command is used to run nmap
with elevated privileges, which may be necessary depending on your system
configuration.

Figure 5: Prompt and response for looking at a network to find out what
servers are accepting

Prompt: What linux commands can help a sys admin to evaluate their network
security? Response: Here are some Linux commands that can help a system
administrator evaluate their network security: 1. nmap: nmap is a powerful
network scanner that can be used to identify open ports, running services, and
detect vulnerabilities in a network. It can be used to scan specific hosts or
entire networks. 2. netstat: netstat is a command-line tool used for
monitoring network connections. It can show open network sockets, their
associated processes, and network interface statistics. 3. tcpdump: tcpdump is
a packet analyzer that can capture and display network traffic in real-time.
It can be used to inspect packets for potential security issues and can also
be used to identify suspicious network activity. 4. iptables: iptables is a
command-line utility used for configuring firewall rules in Linux. It can be
used to create rules that filter or block network traffic based on specific
criteria, such as IP address or port number. 5. fail2ban: fail2ban is a log
analysis and intrusion prevention tool. It can be used to monitor log files
for suspicious activity and automatically block IP addresses that show signs
of malicious activity. 6. chkrootkit: chkrootkit is a tool that can be used to
scan for rootkits on a Linux system. It checks for common signs of a rootkit
infection, such as hidden files and processes. 7. lynis: lynis is a security
auditing tool that can be used to scan Linux systems for security
vulnerabilities. It can perform a variety of checks, including system and file
permissions, firewall configuration, and malware scanning. These commands can
be combined with other security tools and practices to help ensure the
security of a Linux network. However, it’s important to note that evaluating
network security is a complex and ongoing process that requires a
comprehensive approach.

Figure 6: Prompt and response for a query on network security tools in Linux

Figure 5 shows what ChatGPT can describe in terms of using a Linux system to
evaluate the state of the network. Using a simple ChatGPT query based on
Figure 6 allows a learner even to understand what are the possible system
administrative tools for network security. Again, using ChatGPT as a
personalized tutor allows the learner to get direct feedback on how to proceed
in the space.

##  V Conclusion

In this work, we looked at our existing cybersecurity curriculum as a case
study of how well AI-driven natural language processing tools such as ChatGPT
change the LOs of a curriculum. To do this we used the lense of LOs and their
respective Bloom’s taxonomy verbiage and mapped out our cybersecurity
curriculum. We noticed that there are a large number of LOs in Cybersecurity
and most of those LOs sit in the “low-order” thinking skills. Based on this,
we hypothesize that ChatGPT will allow us to accelerate some of our existing
LOs, given the tool’s capabilities. To show this we provide five examples of
how an existing LO can be accelerated with ChatGPT, given the capabilities of
the tool.

From this exercise, we have learned two things in particular that we believe
we will need to be further examined by all educators. First, our experiences
with ChatGPT suggest that the tool can provide a powerful means to allow
learners to generate pieces of their work quickly. However, learners will need
to be taught how to create prompts, and how to validate the responses and
designs provided by the tool, and just like using the browser to find things,
the learner needs to be aware of the vocabulary used to get acceptable
responses. Second, we will need to consider how to teach concepts that need to
be experienced from “first-principle” learning approaches and learn how to
motivate students to perform some rudimentary exercises that “the tool” can
easily do for me. We will argue, that some basic experiments need to be done
that provide us with evidence of why certain concepts still need to be done to
help us prove to our students the importance of early learning exercises.

The experiment we suggest going forward is to use the cybersecurity concept
inventory for a control class that is taught various cyber concepts in a
traditional method and compares a group that is taught similar ideas but is
also prompted to use ChatGPT to explore the ideas more deeply. This depth
could be facilitated by designing assignments that ask for more detailed
challenges within the cybersecurity space.

## References

  * [1] L. W. Anderson, D. R. Krathwohl, and B. S. Bloom, _A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives_. Allyn & Bacon, 2001.
  * [2] G. P. Wiggins, J. McTighe, L. J. Kiernan, and F. Frost, _Understanding by design_. Association for Supervision and Curriculum Development Alexandria, VA, 1998.
  * [3] P. Jamieson and L. Grace, “A framework to help analyze if creating a game to teach a learning objective is worth the work,” in _2016 IEEE Frontiers in Education Conference_ , 2016, pp. 1–7.
  * [4] J. Zull, “The Art of Changing the Brain: Enriching the Practice of Teaching by Exploring the Biology of Learning,” 2002.
  * [5] D. A. Kolb, _Experiential Learning_. Englewoods Cliff, NJ: Prentice Hall Inc., 1984.
  * [6] D. R. Krathwohl, “A revision of bloom’s taxonomy: An overview,” _Theory into practice_ , vol. 41, no. 4, pp. 212–218, 2002.
  * [7] J. M. Lang, _Small teaching: Everyday lessons from the science of learning_. John Wiley & Sons, 2021.
  * [8] J. Sheehan, “Curriculum models: product versus process,” _Journal of advanced nursing_ , vol. 11, no. 6, pp. 671–678, 1986.
  * [9] D. Mouheb, S. Abbas, and M. Merabti, “Cybersecurity curriculum design: A survey,” _Transactions on Edutainment XV_ , pp. 93–107, 2019.
  * [10] D. C. Rowe, B. M. Lunt, and J. J. Ekstrom, “The role of cyber-security in information technology education,” in _Proceedings of the 2011 conference on Information technology education_ , 2011, pp. 113–122.
  * [11] M. Riaz, S. Elder, and L. Williams, “Systematically developing prevention, detection, and response patterns for security requirements,” in _2016 IEEE 24th International Requirements Engineering Conference Workshops (REW)_. IEEE, 2016, pp. 62–67.
  * [12] M. A. Harris _et al._ , “Using bloom’s and webb’s taxonomies to integrate emerging cybersecurity topics into a computic curriculum,” _Journal of Information Systems Education_ , vol. 26, no. 3, pp. 219–234, 2015.
  * [13] S. Poulsen, G. L. Herman, P. A. Peterson, E. Golaszewski, A. Gorti, L. Oliva, T. Scheponik, and A. T. Sherman, “Psychometric evaluation of the cybersecurity concept inventory,” _ACM Transactions on Computing Education (TOCE)_ , vol. 22, no. 1, pp. 1–18, 2021.
  * [14] G. L. Herman, S. Huang, P. A. Peterson, L. Oliva, E. Golaszewski, and A. T. Sherman, “Psychometric evaluation of the cybersecurity curriculum assessment,” in _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_ , 2023, pp. 228–234.
  * [15] Miami University of Ohio, “Course Descriptions for Department of Computer Science and Software Engineering,” https://www.miamioh.edu/cec/academics/departments/cse/academics/course-descriptions/, 2023\.
  * [16] ACM, “Curriculum Guidelines for Post-Secondary Degree Programs in Cybersecurity,” chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://cybered.hosting.acm.org/wp-content/uploads/2018/02/newcover_csec2017.pdf, 31 December 2017.