# A Reverse Engineering Education Needs Analysis Survey

Charles R. Barone IV  University of New HavenUnited States of America
<EMAIL_ADDRESS>,  Robert Serafin  University of New HavenUnited
States of America<EMAIL_ADDRESS>,  Ilya Shavrov  University of New
HavenUnited States of America<EMAIL_ADDRESS>,  Ibrahim Baggili
Louisiana State University
Center for Computation & TechnologyUnited States of America<EMAIL_ADDRESS>,
Aisha Ali-Gombe  Louisiana State University
Center for Computation & TechnologyUnited States of America<EMAIL_ADDRESS>,  Golden G. Richard III  Louisiana State University
Center for Computation & TechnologyUnited States of America<EMAIL_ADDRESS>and  Andrew Case  Louisiana State University
Center for Computation & TechnologyUnited States of America<EMAIL_ADDRESS>
(2018)

###### Abstract.

This paper presents the results of a needs analysis survey for Reverse
Engineering (RE). The need for reverse engineers in digital forensics,
continues to grow as malware analysis becomes more complicated. The survey was
created to investigate tools used in the cybersecurity industry, the methods
for teaching RE and educational resources related to RE. Ninety-three (n=93)
people responded to our 58 question survey. Participants did not respond to
all survey questions as they were optional. The data showed that the majority
of $\nicefrac{{24}}{{71}}$ (33.8%) responses either strongly agreed and
$\nicefrac{{22}}{{71}}$ (30.99%) of responses somewhat agreed that there is a
shortage in RE resources. Furthermore, a majority of $\nicefrac{{17}}{{72}}$
(23.61%) responses indicated that they strongly disagree and that
$\nicefrac{{27}}{{72}}$ (37.5%) somewhat disagree to the statement that
graduates are leaving college with adequate RE knowledge. When asked if there
is a shortage of adequate RE candidates, the majority of
$\nicefrac{{33}}{{71}}$ (46.48%) responses strongly agreed and
$\nicefrac{{20}}{{71}}$ (28.17%) somewhat agreed. In order to determine if
this was a result of the tools at their disposal, a series of questions in
regards to the two most popular RE tools were also asked. In response to a
question about the efficiency of the Interactive Disassembler (IDA) Graphical
User Interface (GUI), majority of $\nicefrac{{16}}{{50}}$ (32%) respondents
strongly agreed and $\nicefrac{{27}}{{50}}$ (54%) somewhat agreed. Similarly,
in response to the same question, but in regards to the Ghidra GUI, majority
of $\nicefrac{{14}}{{32}}$ (43.75%) respondents somewhat agreed. The results
show that a significant number of respondents were self-taught and weren’t
educated via a university, college, or employer in RE. It was concluded that
RE is still a growing field, with few experts and a lack of proficiency, but
not necessarily impeded by a lack of adequate tools. Our paper provides an
overview of the current state of the RE community to be used for the future
development of the RE domain.

Reverse Engineering, Needs Analysis Survey, Malware Analysis, Reverse
Engineering Learning, Reverse Engineering Education

††copyright: acmcopyright††journalyear: 2022††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation email; August 23–26, 2022; Vienna,
Austria††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06

CISA

    Cybersecurity and Infrastructure Security Agency
NSA

    National Security Agency
RE

    Reverse Engineering
IDA

    Interactive Disassembler
GDB

    GNU Debugger
UI

    User Interface
OSS

    Open Source Software
CTF

    Capture The Flag
DMCA

    Digital Millennium Copyright Act
GUI

    Graphical User Interface
EULA

    End-User License Agreement
IRB

    Institutional Review Board
CPU

    Central Processing Unit
API

    Application Programming Interface
ARM

    Advanced RISC Machines
WinDbg

    Windows Debugger
CAE-CO

    Centers of Academic Excellence in Cyber Operations

##  1\. Introduction

The origins of RE can be traced back to hardware, where physical objects were
deconstructed in order to determine design logic as well as functionalities.
While hardware RE is still prevalent today, there is a significant shift in
focus to RE software, especially within the security community. A formal
definition of software RE was presented by (Chikofsky & Cross, 1990) in 1990,
as “…the process of analyzing a subject system to identify the system’s
components and their relationships and create a representation of the system
in another form or at a higher level of abstraction.” Over the years, software
RE has gained significant momentum, particularly in the areas of program
analysis and re-engineering and transformation (Conti et al., 2008; Canfora &
Di Penta, 2007). Various RE toolkits were created that support a number of
different architectures, assist with design recovery, and allow visualization
of extracted data.

Although these advancements are significant, with the increasing complexity of
cyberattacks and proliferation of highly stealthy and obfuscated malware, the
need for software RE in the digital forensics and incident response continues
to grow. Today, RE plays a critical role in the deep understanding and
attempted mitigation of malicious software (Richard III, 2009), which (Luciano
et al., 2018) stated is imperative to digital forensics. Major companies, such
as Target, Equifax, and Apple, have been compromised and experienced the
leakage of user data for millions of users. Additionally, cities in the United
States have had to pay a significant amount of money to regain access to their
systems after sophisticated ransomware attacks, as seen in Florida towns
(Norwood, 2019). The Cybersecurity and Infrastructure Security Agency
Cybersecurity and Infrastructure Security Agency (CISA) director Christopher
Krebs has gone on to state that ransomware attacks are a national crisis
(Seals, 2020). Finally, the United States government’s response to combating
cyber attacks further demonstrates this urgency. The U.S. Fiscal Year Budget
for 2020 has allocated 18.4 billion dollars for cybersecurity purposes to a
number of agencies within the federal government (Executive Office of the
President et al., 2019; Cybsecurity Funding, 2019).

While it is important to highlight technical advancements in RE and to
continue to build powerful tools for RE, it is also critical that we try to
understand outstanding issues that plague this field so that we can focus our
energy on effective solutions. (Selfridge et al., 1993) points out some
fundamental problems of RE, such as using example programs, understanding
economic impact, and inter-researcher communication. It’s also clear that
there are significant barriers to entry for students. These include the
availability of appropriate training, the need to use complex or expensive
software packages, and the lack of prerequisite knowledge. The latter is a
serious issue, as the skills and knowledge most helpful in RE, including
advanced systems programming, operating systems internals, assembler language,
compilers, and more, are increasingly marginalized in academic programs
(Richard III, 2009).

In this paper, we present the results of a detailed needs survey for RE. Our
survey seeks to explore the general need for RE in the cybersecurity and
digital forensics communities and how RE is taught in universities and learned
via on-the-job training. Similar needs analysis surveys have been conducted in
the literature for other fields of cybersecurity, such as cyber forensics
(Rogers & Seigfried, 2004; Harichandran et al., 2016). We undertook this
effort because it is crucial for the growth of RE as a whole to reflect on the
current standing of the field and its challenges. Our contributions are as
follows:

  * •

We present the primary account of a needs analysis survey for RE. Our 58
question survey gained the feedback from (n=93) respondents, for the analysis
of RE needs.

  * •

We present primary data on how people learn, teach and work in RE while also
explaining what challenges they face during that process.

  * •

Our analysis provides insight for advancing the RE in education and tools.

The paper is divided into several sections. First, we discuss background
information on RE in Section 2. In Section 3, we share the methodology and
survey design. The results are presented in Section 4. Section 5 describes the
limitations of our work, and Section 6 shares related work. Lastly, Section 7
discusses and concludes our work.

##  2\. Background

In recent years, software RE has played a pivotal role in cybersecurity and
digital forensics (Akbanov et al., 2018; Burji et al., 2010; Rahimian et al.,
2013). There are basically two major static analysis techniques for software
RE \- disassembly and decompilation. Disassemblers are tools that generate
low-level assembly code from a compiled binary for further scrutiny. On the
other hand, decompilation is the process of translating an executable into
high-level source code. These RE techniques are widely used in various
computing fields ranging from software development to malware and
vulnerability analysis. In software development, RE provides a mechanism for
recovering and recreating a higher-level abstraction of legacy software for
maintenance and patching even where the original source code is not available.
In addition, these RE techniques can be applied for the recovery of source
code, architecture and design discovery (Canfora & Di Penta, 2007), and the
integrity validation of cryptographic algorithms (Eilam, 2011). In the malware
analysis domain, RE is applied to identify the malicious payload, obfuscation,
and other functionalities.

###  2.1. Current Tools

A variety of tools are used to reverse engineer applications. Generally, these
tools can be separated into two different categories: static analysis and
dynamic analysis. Static analysis tools examine the application without
executing it while dynamic analysis tools examine the application during
execution. These categories are not mutually exclusive–some tools provide both
static and dynamic analysis functionality. While open source tool development
for RE is growing, reverse engineers typically use a mixture of open and
closed source tools.

####  2.1.1. Static Analysis Tools

IDA by Hex Rays is a disassembler and decompiler that is interactive,
programmable and extensible. IDA is hosted on Windows, Linux, or Mac OS X
(Interactive Disassembler, n.d.). It was created by Ilfak Guilfanov, who later
founded Hex Rays. Ghidra is a suite of tools developed for software RE created
by the National Security Agency (NSA)’s Research Directorate in support of the
cybersecurity mission (Ghidra, n.d.). It was publicly released by the NSA on
March 2019, with the source code being released on Github (National Security
Agency, 2020). Radare2 is an open source RE tool suite that initially provided
only static analysis capabilities, but now also provides an integrated low-
level debugger (Radare Portable Reversing Framework, n.d.).

####  2.1.2. Dynamic Analysis Tools

There are a variety of publicly available tools that can be used to perform
dynamic analysis of an executable. The application will typically execute in a
sandbox or virtual environment (e.g., Virtualbox, VMWare Workstation,
Microsoft Hyper-V) to avoid contamination of the analysis machine as the user
monitors application behavior. There are also web-based tools that allow users
to upload files to be scanned in order to determine if they are malicious.
Some examples are Virustotal and Hybrid Analysis. x64dbg is an open-source
binary debugger for Windows, aimed at malware analysis and RE of executables
for which source code is not available (x64dbg, n.d.). Immunity Debugger
builds on a solid User Interface (UI) with function graphing, the industry’s
first heap analysis tool built specifically for heap creation, and a large and
well supported Python Application Programming Interface (API) for easy
extensibility (Immunity Debugger, n.d.).

The Windows Debugger (WinDbg) can be used to debug kernel-mode and user-mode
code, analyze crash dumps, and examine the Central Processing Unit (CPU)
registers while the code executes (Download Debugging Tools for Windows,
2020). OllyDbg is a debugger that operates at the assembler level on Windows
and focuses on binary code analysis. It is free and open source, which makes
it a very popular debugging solution (OllyDbg, n.d.). Cheat Engine is a tool
created for use primarily with single player video games and allows a user to
scan for and change values of variables that control game behavior. It also
includes a debugger, disassembler, assembler, and system inspection tools
which can be of use to someone performing analysis of other types of
applications (Cheat Engine, n.d.).

##  3\. Methodology

Given the different flavors of RE tools and the different ways in which
practitioners use the tools for analysis and training, it is clear that there
is no harmonized approach to RE within the security and forensics communities.
Thus, to understand the driving factors and motivation for adopting specific
tools or collection of tools by practitioners and educators, we conduct a need
analysis survey. The survey was created and taken on Qualtrics. We began by
obtaining a category two exemption from the Institutional Review Board (IRB)
at the BLINDED FOR REVIEW (this meant that the survey did not record
participant identification information or behavior, and posed no risk or harm
to subjects not encountered in every day life). The recorded responses from
the survey were then analyzed using various exploratory analysis methods.

###  3.1. Survey Design & Data Analysis

The survey was mainly geared towards individuals who were professionals or
educators in the cybersecurity and digital forensics fields. It consisted of
58 questions with the following composition:

  * •

10 Likert Scale

  * •

21 Multiple Choice

  * •

15 Multiple Choice (Checkbox)

  * •

9 Free Response

  * •

1 Slider

The survey questions covered the following: demographics, teaching RE,
learning RE, RE Tools and platforms, IDA versus Ghidra, potential for web
based RE platforms, and legal challenges in adopting and usage of RE.
Collected responses were analyzed using exploratory data analysis and
visualization techniques.

##  4\. Results

Ninety-three participants were recorded to have accessed the survey. All
(n=93) participants consented to take the survey, three did not continue the
survey beyond the definitions and seven did not record any responses beyond
the first question. Not all participants completed every question in the
survey. This is discussed further in the Limitations Section 5 of the paper.

###  4.1. Demographics

The demographic results as shown in Appendix A, Table A.1 shows that the
majority of the responses $\nicefrac{{56}}{{85}}$ (65.88%) were white,
$\nicefrac{{69}}{{85}}$ (81.18%) were males and $\nicefrac{{30}}{{85}}$
(35.29%) within the age range of 30-39. The majority of responses
$\nicefrac{{24}}{{85}}$ (28.24%) indicates a Masters degree as participants
highest level of education. Most of the responses indicate degrees related to
the field of technology with a majority of in Computer Science
$\nicefrac{{26}}{{83}}$ (31.33%).

Interest in RE (Figure 3), was answered by 72 of 93 participants. The majority
indicated that they either agree or strongly agree that they were interested
in RE. Results indicate that $\nicefrac{{41}}{{72}}$ (56.94%) of responses
strongly agree and $\nicefrac{{22}}{{72}}$ (30.56%) of responses somewhat
agree.

The mean percentage of time spent on RE during work, shown in Appendix A,
Table A.2 was found to be 26.56% with a maximum of 90%. The mean percentage of
time spent RE outside of work was found to be 27.02% with a maximum of 80%.
Additionally when asked about how many years work has involved RE,
$\nicefrac{{26}}{{83}}$ (31.33%) responses indicated that their career has not
involved RE, followed by $\nicefrac{{13}}{{83}}$ (15.66%) responses indicating
that their career has involved RE for 5 to 6 years as shown in Appendix A,
Table A.3.

Participants then went into detail explaining how their work requires RE. Many
of the participants have worked on reversing malware either as part of an
incidence response team or as RE instructors. When explaining what RE
activities are done outside of work, most like to compete in Capture The Flag
(CTF) competitions. Some are researches and study malware using RE.

The professional roles participants have worked in are shown in Appendix A;
Tables A.4, A.5. The responses indicated that $\nicefrac{{29}}{{344}}$ (8.43%)
worked as engineers in the cybersecurity industry, $\nicefrac{{27}}{{344}}$
(7.85%) worked as professors or educators while $\nicefrac{{27}}{{344}}$
(7.85%) worked as software developers. It is important to note that the
responses to this question varied indicating that participants held a diverse
set of professional roles in the field of cybersecurity, and further
exemplifying that RE is a field that broadly applies to a number of job
functions. It is important to note that the high volume of responses to this
question (n=344), is due to participants selecting multiple options.

###  4.2. User Education

The proficiency for first RE related roles (Appendix B; Table B.1), shows
$\nicefrac{{28}}{{75}}$ (37.33%) responses indicating that they were not
proficient, and only $\nicefrac{{9}}{{75}}$ (12.00%) responses indicate they
were very proficient followed by $\nicefrac{{2}}{{75}}$ (2.67%) responses
indicate they were an expert.

For where participants education in RE took place (Appendix B; Table B.2),
$\nicefrac{{57}}{{92}}$ (61.96%) indicated that they were self-taught,
$\nicefrac{{23}}{{92}}$ (25.00%) responses indicated they were taught from
university or college, and $\nicefrac{{12}}{{92}}$ (13.04%) responses indicate
that they were taught by their employer. It is important to note that a
significant amount of these participants indicated that they were not educated
in RE via a university, college, or employer.

Participants were then asked which resources were beneficial when learning RE
(Appendix B; Table B.3). The majority of the responses indicated that they
utilized online instructional guides, online challenges or “crackme”, and
textbooks. $\nicefrac{{45}}{{213}}$ (21.13%) responses indicates that they
utilized online instructional guides, $\nicefrac{{40}}{{213}}$ (18.78%)
responses indicates that they utilized online challenges or “crackmes”, and
$\nicefrac{{36}}{{213}}$ (16.90%) responses indicated that they employed
textbooks. It is important to note that the high volume of responses to this
question (n=213), is due to some participants selecting multiple options.

Most valuable books for learning RE had (Appendix B; Table B.5) majority of
participants indicating that Practical Malware Analysis : The Hands-On Guide
to Dissecting Malicious Software and The IDA/IDA Pro Book were the most
valuable. $\nicefrac{{34}}{{143}}$ (23.78%) responses indicated Practical
Malware Analysis : The Hands-ON Guide to Dissecting Malicious Software and
$\nicefrac{{23}}{{143}}$ (16.08%) responses indicated The IDA/IDA Pro Book. It
is important to note that the high volume of responses to this question
(n=143), is due to some participants selecting multiple options.

Most valuable websites for learning RE resulted with (Appendix B; Table B.4)
majority of participants indicating that youtube.com, sans.com, and
crackmes.one were most valuable. $\nicefrac{{36}}{{114}}$ (31.58%) responses
indicated youtube.com, $\nicefrac{{16}}{{114}}$ (14.04%) responses indicated
sans.com, and $\nicefrac{{16}}{{114}}$ (14.04%) responses indicated
crackmes.one. It is important to note that the high volume of responses to
this question (n=114), is due to participants selecting multiple options.

With the agreement if there is a shortage of effective RE educational
resources (Figure 3), majority of participants indicated that they either
strongly agree or somewhat agree. $\nicefrac{{24}}{{71}}$ (33.80%) responses
indicated strongly agree and $\nicefrac{{22}}{{71}}$ (30.99%) responses
indicated somewhat agree. Additionally, when asked if participants agree that
cybersecurity graduates are leaving school with adequate proficiency in RE
(Figure 3), majority of participants indicated that they somewhat disagree or
strongly disagree. $\nicefrac{{27}}{{72}}$ (37.50%) responses indicate
somewhat disagree and $\nicefrac{{17}}{{72}}$ (23.61%) responses indicated
strongly disagree. It should also be noted that only $\nicefrac{{4}}{{72}}$
(5.56%) responses indicated strongly agree. Furthermore, when asked if
participants agree that there is a shortage of qualified RE candidates in the
workforce (Figure 3), majority of participants indicated that they either
strongly agree or somewhat agree. $\nicefrac{{33}}{{71}}$ (46.48%) responses
indicated strongly agree and $\nicefrac{{20}}{{71}}$ (28.17%) responses
indicated somewhat agree. It is also important to note that none of the
participants indicated that they strongly disagree.

###  4.3. Reverse Engineering Tools

From the RE tools participants had heard of (Figure 4), majority heard of
IDA/IDA Pro, Ghidra, and GNU Debugger (GDB). $\nicefrac{{61}}{{804}}$ (7.59%)
responses indicated IDA/IDA Pro, $\nicefrac{{60}}{{804}}$ (7.46%) responses
indicated Ghidra, and $\nicefrac{{49}}{{804}}$ (6.09%) responses indicating
GDB. It is important to note that per the high volume of responses to this
question (n=804), some participants selected multiple options. When asked
which tools participants use in RE (Figure 4), the majority indicated that
they made use of IDA/IDA Pro, GDB, and Ghidra. $\nicefrac{{51}}{{389}}$
(13.11%) responses indicated IDA/IDA Pro, $\nicefrac{{34}}{{389}}$ (8.74%)
indicated Ghidra, and $\nicefrac{{31}}{{389}}$ (7.97%) indicated GDB. It is
important to note that the high volume of responses to this question (n=389),
is due to participants selecting multiple options.

Participants were then asked if the tools that they use for RE are adequate
for the work they need to accomplish (Figure 3). The majority either somewhat
agree or strongly agree. $\nicefrac{{27}}{{65}}$ (41.54%) responses somewhat
agree and $\nicefrac{{21}}{{65}}$ (32.31%) responses strongly agree. When
asked what some weaknesses in regards to current RE tools are (Figure 1),
based upon the mean responses most participants agreed that the primary pain
points are the steep learning curve to use tools, open-source software (OSS)
need more funding, tools need to cost less, and poor documentation of tools.

$3$$3.5$$4$More OSS fundingSteep learning curvePoor documentationTools need to
cost lessPoorly designed UIOSS not updated enoughLow community engagementTools
are too nicheTools are too generalTools high fail
rate$4.16$$4.11$$3.64$$3.51$$3.4$$3.37$$3.25$$3.05$$2.9$$2.81$AveragePain
PointsWeaknesses in RE Tools Figure 1. Pain points in regards to RE tools.
Note: The average is calculated on a scale from 1 - 5. Strongly Disagree = 1
and Strongly Agree = 5

The operating systems most utilized by the participants for RE (Appendix C;
Table C.1), were either Unix/Linux and Windows. $\nicefrac{{52}}{{124}}$
(41.94%) responses indicated utilize Unix/Linux and $\nicefrac{{47}}{{124}}$
(37.90%) indicated Windows. It is important to note that the high volume of
responses to this question (n=124), is due to participants selecting multiple
options.

The majority of architectures encountered in RE (Appendix C; Table C.2), were
x86/i386, x86-64/AMD64, and Advanced RISC Machines (ARM).
$\nicefrac{{50}}{{131}}$ (38.17%) responses indicated x86/AMD64,
$\nicefrac{{45}}{{131}}$ (34.35%) indicated x86/i386 , and
$\nicefrac{{24}}{{131}}$ (18.32%) indicated ARM. It is important to note that
the high volume of responses to this question (n=131), is due to participants
selecting multiple options. When asked which platforms participants have
encountered while RE (Appendix C; Table C.3), the majority indicated that they
had encountered Windows, *nix, and Android. $\nicefrac{{53}}{{131}}$ (40.46%)
responses indicated Windows, $\nicefrac{{39}}{{131}}$ (29.77%) indicated *nix,
and $\nicefrac{{29}}{{131}}$ (22.14%) indicated Android. It is important to
note that the high volume of responses to this question (n=131), is due to
participants selecting multiple options.

Languages that participants have encountered while RE (Appendix C; Table C.4),
indicated that majority had encountered C/C++, Java, Compiled Python, and C#.
$\nicefrac{{58}}{{193}}$ (30.05%) responses indicated C/C++,
$\nicefrac{{39}}{{193}}$ (20.21%) indicated Java, $\nicefrac{{33}}{{193}}$
(17.10%) indicated C# and $\nicefrac{{30}}{{193}}$ (15.54%) indicated Compiled
Python. It is important to note that the high volume of responses to this
question (n=193), is due to participants selecting multiple options. When
asked which software/systems participants have reverse engineered or
encountered whilst RE (Appendix C; Table C.5), the majority indicated malware
and desktop applications. $\nicefrac{{50}}{{176}}$ (28.41%) responses
indicated malware and $\nicefrac{{40}}{{176}}$ (22.73%) indicated desktop
applications. It is important to note that the high volume of responses to
this question (n=176), is due to participants selecting multiple options.

Participants elaborated on their experience on learning RE. Many learned
though academic courses, CTF challenges, online tutorials and a lot of
practice on their own. Reading was said to only help understand general
concepts. Anything other than hands on experience people don’t find helpful.
Not everyone started with the assembly languages when learning RE. There were
people who started by understanding Python and how C compiles to assembly.
Many found that it was much harder if you didn’t have experience in
programming before starting to learn RE.

###  4.4. Ghidra vs IDA

$3.5$$4$$4.5$DisassemblerGraph viewVariable renamingFunction call
graphReference tableFunction signature mod.Hex viewLibrary func. sig.
recogDebuggerThird-party
plugins$4.39$$4.22$$3.94$$3.94$$3.88$$3.85$$3.69$$3.67$$3.56$$3.55$$4.41$$4.33$$4.31$$4.06$$4$$3.94$$3.88$$3.81$$3.69$$3.63$AverageFeaturesEffectiveness
of Features Between ToolsGhidraIDA Figure 2. Effectiveness of features between
IDA/IDA Pro and Ghidra for RE tasks. Note: The average is calculated on a
scale from 1 - 5. Strongly Disagree = 1 and Strongly Agree = 5
$0\%$$50\%$$100\%$Interested in RERE resource shortageGrads fair RE
knowledgeFit candidate shortageAdequate toolsIDA GUI efficacyGhidra GUI
efficacyPercentageStatementsAgreement AnalysisSAANDSD Figure 3. Results from
agreement questions. Note: Strongly Agree (SA), Somewhat Agree (A), Neither
(N), Somewhat Disagree (D), Strongly Disagree (SD)
$0\%$$5\%$$10\%$IDA/IDAProGhidraGDBApkToolYARARadare2BinaryNinjaCuckooSandboxBinwalkObjdumpDex2jarRemnuxHopperFridaCFFExplorerAngrCheatEngineJadxPEDAJebTritonAndroguardBarfUncompyleKrakatauBoomerangObjectionApk2GoldQiraOtherPercentageToolsKnowledge
and Usage of RE ToolsKnowledgeUsage Figure 4. participants knowledge in RE
tools compared to their usage of RE tools.

Whether participants use the free or commercial edition of IDA (Appendix D;
Table D.1), there was an even split of 25 responses for both. The majority of
responses agreed that disassembler, variable renaming, graph view, function
call graph, and reference table were the most effective features of IDA/IDA
Pro for RE (Figure 2). When asked which IDA/IDA Pro GUI features participants
believe could be improved (Appendix D; Table D.2), majority of responses
indicated debugger, third-party plugins, and disassembler.
$\nicefrac{{18}}{{60}}$ (30%) responses indicated debugger,
$\nicefrac{{11}}{{60}}$ (18.33%) indicated third-party plugins, and
$\nicefrac{{9}}{{60}}$ (15%) indicated disassembler. When asked if
participants agree that IDA/IDA Pro’s GUI is effective for RE (Figure 3),
majority of $\nicefrac{{27}}{{50}}$ (54%) responses indicated somewhat agree.
Lastly, when asked if participants believe that IDA/IDA Pro has any features
which are advantageous over those in Ghidra (Appendix D; Table D.3), majority
of $\nicefrac{{14}}{{27}}$ (51.85%) responses indicated no.

For features of Ghidra that are effective for RE (Figure 2), the majority
agreed on: disassembler, variable renaming, reference table, function
signature modification, and data type viewer. When asked which features of
Ghidra could be improved (Appendix D; Table D.4), majority of responses
indicated debugger and graph view. $\nicefrac{{9}}{{43}}$ (20.93%) responses
indicated debugger and $\nicefrac{{8}}{{43}}$ (18.60%) indicated graph view.
When asked if participants agree that Ghidra’s GUI is effective in
representing information (Figure 3), majority of $\nicefrac{{14}}{{32}}$
(43.75&) responses indicated somewhat agree. It is important to note that
participants could select multiple options.

Lastly, when asked if participants believe that Ghidra has any features which
are better than those in IDA/IDA Pro (Appendix D; Table D.5),
$\nicefrac{{13}}{{25}}$ (52%) of responses indicated yes and
$\nicefrac{{12}}{{25}}$ (48%) of responses indicated no.

###  4.5. Teaching Reverse Engineering

The question if participants teach or have ever taught RE in a classroom
setting (Appendix E; Table E.1), majority of $\nicefrac{{48}}{{72}}$ (66.67%)
responses indicated that they have not, whereas $\nicefrac{{24}}{{72}}$
(33.33%) indicated that they have.

The resources participants used to teach RE (Appendix E; Table E.2), had a
majority indicating that they utilized demonstrations and lab activities.
$\nicefrac{{20}}{{72}}$ (27.28%) responses indicated that they utilized lab
activities and $\nicefrac{{19}}{{72}}$ (26.39%) indicated they utilized
demonstrations. Additionally, when asked what types of assignments they gave
to their students to teach RE (Appendix E; Table E.3), majority of responses
indicated that they utilized projects and CTF style assignments.
$\nicefrac{{15}}{{40}}$ (37.50%) responses indicated that they utilized
projects and $\nicefrac{{15}}{{40}}$ (37.50%) indicated that they utilized CTF
style assignments. Furthermore, when asked what methods participants use to
assess their students’ knowledge of RE (Appendix E; Table E.4), majority of
responses indicated in-class presentations, exams and projects/labs.
$\nicefrac{{8}}{{32}}$ (25.00%) responses indicated that they utilize in-class
presentations, $\nicefrac{{7}}{{32}}$ (21.88%) indicated that they utilize
exams, and $\nicefrac{{7}}{{32}}$ (21.88%) indicated that they utilize other
methods, most of which were either labs or projects. It is important to note
that participants could select multiple options.

In the agreement that certain circumstances are challenges in teaching RE
(Appendix E; Table E.5), majority of participants agreed on three
circumstances. $\nicefrac{{18}}{{61}}$ (29.51%) responses agreed on students’
lack of prerequisite knowledge, $\nicefrac{{13}}{{61}}$ (21.31%) agreed on
students’ understanding of assembly language, and $\nicefrac{{11}}{{61}}$
(18.03%) agreed on the learning curve of RE.

Lastly, when asked what tools are used in classes to teach RE (Appendix E;
Table E.6), majority of participants used IDA/IDA Pro, GDB, Ghidra and
objdump. $\nicefrac{{14}}{{98}}$ (14.29%) responses indicated IDA/IDA Pro,
$\nicefrac{{10}}{{98}}$ (10.20%) indicated GDB, $\nicefrac{{10}}{{98}}$
(10.2%) indicated Ghidra, and $\nicefrac{{10}}{{98}}$ (10.2%) indicated
Objdump. It is important to note that the high volume of responses to this
question (n=98), is due to participants selecting multiple options.

###  4.6. Web-Based Reverse Engineering

Participants were asked if they knew of an existing web-based RE platform
(Appendix F; Table F.1)and the majority of $\nicefrac{{49}}{{63}}$ (77.78%)
responses indicated no, but $\nicefrac{{14}}{{63}}$ (22.22%) indicated yes.
The web-based RE platforms participants knew of include the following:
Acunetix, Binary Ninja Cloud, IDA, and radare2.

The interest of a web-based RE platform (Appendix F; Table F.2), were the
majority of maybe and yes. $\nicefrac{{31}}{{65}}$ (47.69%) responses
indicated maybe and $\nicefrac{{24}}{{65}}$ (36.92%) indicated yes. Lastly,
when asked what characteristics or aspects of such a web-based RE platform
participants would like to see (Appendix F; Table F.3), majority of responses
indicated access to various tools/features and allows for collaboration with
other users. $\nicefrac{{46}}{{216}}$ (21.30%) of responses indicated access
to various tools/features and $\nicefrac{{40}}{{216}}$ (18.52%) responses
indicated allowing for collaboration with other users. It is important to note
that the high volume of responses to this question (n=216), is due to
participants selecting multiple options.

###  4.7. Challenges

Whether there are legal challenges when it comes to RE the majority said no.
Some participants mention that they get permission from the authors of the
software before reversing it. Participants also mentioned how the Digital
Millennium Copyright Act (DMCA) and End-User License Agreement (EULA) limits
what can be used as an RE task.

With future challenges , the majority see malware and software becoming more
complex. In particular, encryption and other software protections will be
getting more advanced. Some see this rapid growth as a negative as their are
already a lack of experts. It will also be harder for beginners who don’t have
much practical knowledge.

##  5\. Limitations

Our target audience included students, professionals, and educators that have
RE experience. When trying to find participants, it was found that this area
of cybersecurity isn’t as popular and the understanding is still relatively
new. This resulted in fewer number of participants, even though overall, we
received an acceptable sample size. Throughout the survey each question was
optional allowing participants to skip and leave questions blank. It became
clear that some questions were favored over others which resulted in
inconsistent response counts for each question. There were also three
participants that dropped out after the definitions portion and seven
participates that stopped after the first question. This could be the result
of loss of interest from the length of the survey with fifty-eight questions.

##  6\. Related Work

Extensive work has been conducted on how to best engage computer science
students and on the best methods for teaching cybersecurity concepts and
methodologies. Work has been done in identifying the learning styles of
computer science and engineering students in order to determine the best
teaching methods (Felder et al., 1988; Thomas et al., 2002).

CTF style challenges are created for the learning benefit of students. They
consist of categories like web exploitation, cryptography and binary RE. The
creators of CTF competitions create these to teach old and new techniques. The
website https://tryhackme.com/ has many modules we’re you can learn and
practice RE concepts as well as other cybersecurity subjects in a CTF like
fashion. Other works have tried to incorporate CTF style labs into education
for methods of teaching cybersecurity principals (Cheung et al., 2011; Chothia
& Novakovic, 2015; Olano et al., 2014), which has also been used to teach RE
concepts (Feng, 2015).

A number of educators have documented their approach to creating and teaching
RE courses (Barr et al., 2000; Richard III, 2009).

Despite the fact that 21 higher education institutions (National Security
Agency, 2021) have been designated as Centers of Academic Excellence in Cyber
Operations (CAE-CO) institutions by the NSA, which requires a course in RE,
there is still a very large knowledge gap in individuals that can effectively
reverse engineer.

Another way in which students have approached learning cybersecurity concepts
is through self-taught methods such as certifications. The problem with this
is that there is a severe lack of resources for individuals who decide to take
this approach. Given the prerequisite knowledge necessary for RE, it is very
difficult for beginners to get started. While there are a number of avenues
available to individuals interested in learning, e.g., programming, such as
online interactive teaching tools, among the few “beginner” resources in RE
that are available, most are very intimidating (Yurichev, 2020).

##  7\. Discussion and Conclusions

The results from our work illustrates that this domain is at its infancy with
few experts and a lack of proficiency. The results also illustrate a lack of
diversity amongst RE professionals. Among the participants that had an
understanding of RE, a focus in cybersecurity or RE, could explain why most
participants have either a masters or doctorate degree. Further development of
courses at a basic level for undergraduate students and activities for high
school students could prove to be beneficial. An early introduction to RE to
students during middle school and high school may open up new interests that
could turn into future academic focuses. The data suggests that computer
science is the most popular field of study, perhaps due to it being the oldest
major, or due to the advanced knowledge of programming needed when performing
RE. Being self-taught turned out to be how most learned RE. This could also be
why many respondents enjoy RE as a hobby just as much when working at their
job.

Furthermore, another important conclusion made from the results of this survey
is that the number of tools individuals use in RE are limited: the two most
popular being IDA/IDA Pro and Ghidra, which were followed by GDB and Objdump
as the next most popular. This popularity could be the result of the age of
these tools, although Ghidra was only made public recently, which may hint
that the maturity and ease of use of the tools may be a more important factor
in their adoption in RE. One interesting item to note is that more participant
claimed to use IDA/IDA Pro, than actually considering themselves knowledgeable
in IDA/IDA Pro. In fact the trend as seen in Figure 4 is that the popular
tools for RE are generally used more than they are understood which may
indicate an underlying issue in RE educational approaches. This could be the
result of the lack of schools with RE courses, resulting in the only option
being to teach yourself and find other resources through either books or
online resources. It may be helpful to add lectures or a playlist of video
tutorials to already existing RE courses on the usage and functionality of the
more popular tools.

Additionally, on the front of IDA versus Ghidra, it is important to note, as
shown in Figure 2, that IDA ranks higher than Ghidra across the board, in some
features more than others. The feature preferred in IDA over Ghidra the most
was variable renaming, this was the only feature that did not have minimal
differences in the preference between the two RE tools which makes it clear
that there should probably be work done to improve variable renaming in
Ghidra. We also found it interesting that exactly half of respondents
indicated their use of the free edition of IDA where as the other half used
the commercial edition. This may represent an even split in the demand for RE
tools in a commercial setting v.s. everyday, or minimal functionality usage.
Although it should also be noted that when asked about if IDA/IDA Pro was
advantageous over Ghidra based upon their feature sets, a slim majority
indicated no.

The results showed that most who have taught RE prefer to use hands on
activities and labs. This is a step in the positive direction as it tells us
that instructors may have found an optimal teaching approach. The lack of a
student’s prerequisite knowledge may be a result of RE classes being an
elective, rather than a requirement. RE is a more advanced class that requires
knowledge in multiple areas. The steep learning curve may discourage new
students away, which could explain the lack of experts found in the results.
RE is a field in the workforce that is very much needed for incident response
teams.

The researchers acknowledge that a follow-up survey or additions to the
current survey would be beneficial to gain more insight on specifics. Those
that indicated they disagreed with certain question statements, could have
also been asked to provide an explanation. With all surveys, there needs to be
an understanding of what is wrong and what solutions must be made. There were
also questions that participants could select tools to indicate their usage or
efficacy. Future work could allow participants to rank the chosen tools.
Lastly, a longitudinal study that includes both surveys and interviews with
experts may shed light on how the domain and its educational capacity is
changing over time. This is imperative as RE is becoming a cornerstone for
improving the cybersecurity and incidence response stature of public and
private organizations.

## References

  * (1)
  * Akbanov et al. (2018) Akbanov, M., Vassilakis, V. G., Moscholios, I. D. & Logothetis, M. D. (2018), Static and dynamic analysis of wannacry ransomware, in ‘Proc. IEICE Inform. and Commun. Technol. Forum ICTF’, Vol. 2018.
  * Barr et al. (2000) Barr, R. E., Schmidt, P. S., Krueger, T. J. & Twu, C.-Y. (2000), ‘An introduction to engineering through an integrated reverse engineering and design graphics project’, Journal of Engineering Education 89(4), 413–418.
  * Burji et al. (2010) Burji, S., Liszka, K. J. & Chan, C.-C. (2010), Malware analysis using reverse engineering and data mining tools, in ‘2010 International Conference on System Science and Engineering’, IEEE, pp. 619–624.
  * Canfora & Di Penta (2007) Canfora, G. & Di Penta, M. (2007), New frontiers of reverse engineering, in ‘Future of Software Engineering (FOSE’07)’, IEEE, pp. 326–341.
  * Cheat Engine (n.d.) Cheat Engine (n.d.).
https://www.cheatengine.org/aboutce.php

  * Cheung et al. (2011) Cheung, R. S., Cohen, J. P., Lo, H. Z. & Elia, F. (2011), Challenge-based learning in cybersecurity education, in ‘Proceedings of the International Conference on Security and Management (SAM)’.
  * Chikofsky & Cross (1990) Chikofsky, E. J. & Cross, J. H. (1990), ‘Reverse engineering and design recovery: A taxonomy’, IEEE software 7(1), 13–17.
  * Chothia & Novakovic (2015) Chothia, T. & Novakovic, C. (2015), An offline capture the flag-style virtual machine and an assessment of its value for cybersecurity education, in ‘2015 USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE 15)’.
  * Conti et al. (2008) Conti, G., Dean, E., Sinda, M. & Sangster, B. (2008), Visual reverse engineering of binary and data files, in ‘International Workshop on Visualization for Computer Security’, Springer, pp. 1–17.
  * Cybsecurity Funding (2019) Cybsecurity Funding (2019).
https://www.whitehouse.gov/wp-content/uploads/2019/03/ap_24_cyber_security-
fy2020.pdf

  * Download Debugging Tools for Windows (2020) Download Debugging Tools for Windows (2020).
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-
download-tools

  * Eilam (2011) Eilam, E. (2011), Reversing: secrets of reverse engineering, John Wiley & Sons.
  * Executive Office of the President et al. (2019) Executive Office of the President, Office of Management & Budget (2019), ‘A budget for a better America: Promises kept. taxpayers first: Fiscal year 2020 budget of the U.S. government’.
  * Felder et al. (1988) Felder, R. M., Silverman, L. K. et al. (1988), ‘Learning and teaching styles in engineering education’, Engineering Education 78(7), 674–681.
  * Feng (2015) Feng, W.-c. (2015), A scaffolded, metamorphic CTF for reverse engineering, in ‘2015 USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE 15)’.
  * Ghidra (n.d.) Ghidra (n.d.).
https://ghidra-sre.org/

  * Harichandran et al. (2016) Harichandran, V. S., Breitinger, F., Baggili, I. & Marrington, A. (2016), ‘A cyber forensics needs analysis survey: Revisiting the domain’s needs a decade later’, Computers & Security 57, 1–13.
  * Immunity Debugger (n.d.) Immunity Debugger (n.d.).
https://www.immunityinc.com/products/debugger/

  * Interactive Disassembler (n.d.) Interactive Disassembler (n.d.).
https://www.hex-rays.com/products/ida/

  * Luciano et al. (2018) Luciano, L., Baggili, I., Topor, M., Casey, P. & Breitinger, F. (2018), Digital forensics in the next five years, in ‘Proceedings of the 13th International Conference on Availability, Reliability and Security’, pp. 1–14.
  * National Security Agency (2020) National Security Agency (2020), ‘Ghidra’.
https://github.com/NationalSecurityAgency/ghidra

  * National Security Agency (2021) National Security Agency (2021), ‘Centers of academic excellence in cyber operations’.
https://www.nsa.gov/resources/students-educators/centers-academic-
excellence/cae-co-centers/

  * Norwood (2019) Norwood, C. (2019), ‘Another Florida city hacked: This time for $490,000 ransom’.
https://www.governing.com/topics/public-justice-safety/tns-another-florida-
city-hacked.html

  * Olano et al. (2014) Olano, M., Sherman, A., Oliva, L., Cox, R., Firestone, D., Kubik, O., Patil, M., Seymour, J., Sohn, I. & Thomas, D. (2014), SecurityEmpire: Development and evaluation of a digital game to promote cybersecurity education, in ‘2014 USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE 14)’.
  * OllyDbg (n.d.) OllyDbg (n.d.).
http://www.ollydbg.de/

  * Radare Portable Reversing Framework (n.d.) Radare Portable Reversing Framework (n.d.).
https://https://www.radare.org

  * Rahimian et al. (2013) Rahimian, A., Ziarati, R., Preda, S. & Debbabi, M. (2013), On the reverse engineering of the citadel botnet, in ‘International Symposium on Foundations and Practice of Security’, Springer, pp. 408–425.
  * Richard III (2009) Richard III, G. G. (2009), A Highly Immersive Approach to Teaching Reverse Engineering, in ‘2nd USENIX Workshop on Cybersecurity Instrumentation and Test CSET ’09’.
  * Rogers & Seigfried (2004) Rogers, M. K. & Seigfried, K. (2004), ‘The future of computer forensics: a needs analysis survey’, Computers & Security 23(1), 12–16.
  * Seals (2020) Seals, T. (2020), ‘Rsac 2020: Ransomware a ’national crisis,’ cisa says, ramps ics focus’.
https://threatpost.com/ransomware-national-crisis-cisa-ics/153322/

  * Selfridge et al. (1993) Selfridge, P. G., Waters, R. C. & Chikofsky, E. J. (1993), Challenges to the field of reverse engineering, in ‘Proceedings of the Working Conference on Reverse Engineering’, IEEE, pp. 144–150.
  * Thomas et al. (2002) Thomas, L., Ratcliffe, M., Woodbury, J. & Jarman, E. (2002), ‘Learning styles and performance in the introductory programming sequence’, ACM SIGCSE Bulletin 34(1), 33–37.
  * x64dbg (n.d.) x64dbg (n.d.).
https://github.com/x64dbg/x64dbg

  * Yurichev (2020) Yurichev, D. (2020), Reverse Engineering for Beginners.

##  Appendix A. Demographics

Table A.1. Demographics.

|  Count |  Percentage
---|---|---
Race |  |
White |  56 |  65.88%
Do not wish to specify |  13 |  15.29%
Asian |  11 |  12.94%
Black or African American |  1 |  1.18%
Other |  4 |  4.71%
\- Israeli |  |
\- White, African American, and American Indian |  |
Sex |  |
Male |  69 |  81.18%
Female |  10 |  11.76%
Do not wish to specify |  6 |  7.06%
Age |  |
30-39 |  30 |  35.29%
20-29 |  24 |  28.24%
40-49 |  12 |  14.12%
Younger than 20 |  6 |  7.06%
50-59 |  5 |  5.88%
60-69 |  4 |  4.71%
Do not wish to specify |  3 |  3.53%
Older than 60 |  1 |  1.18%
Level of Education |  |
Master’s degree |  24 |  28.24%
Doctoral degree |  18 |  21.18%
Bachelor’s degree |  17 |  20.00%
Some university or college |  10 |  11.76%
High school diploma or GED |  6 |  7.06%
Associate’s degree |  5 |  5.88%
Some high school |  5 |  5.88%
Field of Study |  |
Computer Science |  26 |  31.33%
Cybersecurity |  24 |  28.92%
Computer Engineering |  9 |  10.84%
Information Technology |  7 |  8.43%
Other |  5 |  6.02%
Electrical Engineering |  3 |  3.61%
Criminal Justice |  2 |  2.41%
Mechanical Engineering |  1 |  1.20%
Mathematics |  1 |  1.20%
Computer Networks |  1 |  1.20%
Civil Engineering |  1 |  1.20%
Business Administration |  1 |  1.20%
Biology |  1 |  1.20%

Table A.2. Percentage of time related to reverse engineering.

Location |  Average Percentage
---|---
During work |  26.56%
Outside work |  27.02%

Table A.3. Years of reverse engineering during career.

Years |  Count |  Percentage
---|---|---
My career has not involved reverse engineering |  26 |  31.33%
5-6 |  13 |  15.66%
3-4 |  12 |  14.46%
Less than 1 |  10 |  12.05%
1-2 |  10 |  12.05%
More than 10 |  6 |  7.23%
7-8 |  4 |  4.82%
9-10 |  2 |  2.41%

Table A.4. Professional Roles.

Role |  Count |  Percentage
---|---|---
Engineer in Cybersecurity Industry |  29 |  8.43%
Professor or Educator |  27 |  7.85%
Software Developer |  27 |  7.85%
Forensic Analyst |  22 |  6.40%
System Administrator |  19 |  5.52%
Cyber Instructor |  16 |  4.65%
Cyber Instructional Curriculum Developer |  12 |  3.49%
Director or Management |  11 |  3.20%
Research & Development Specialist |  11 |  3.20%
Cyber Defense Analyst |  11 |  3.20%
Cyber Defense Incident Responder |  11 |  3.20%
Vulnerability Analyst |  11 |  3.20%
Cyber Defense Forensics Analyst |  11 |  3.20%
Security Architect |  9 |  2.62%
Testing and Evaluation Specialist |  9 |  2.62%
Systems Developer |  9 |  2.62%
Cyber Crime Investigator |  9 |  2.62%
Network Operations Specialist |  8 |  2.33%
Database Administrator |  7 |  2.03%
System Security Analyst |  7 |  2.03%
Information Systems Security Developer |  6 |  1.74%
Technical Support Specialist |  6 |  1.74%
Secure Software Assessor |  4 |  1.16%
IT Project Manager |  4 |  1.16%
Exploitation Analyst |  4 |  1.16%

Table A.4. Professional Roles (Continued). Role |  Count |  Percentage
---|---|---
Cyber Intel Planner |  4 |  1.16%
Cyber Operator |  4 |  1.16%
Executive |  3 |  0.87%
Enterprise Architect |  3 |  0.87%
Warnings Analyst |  3 |  0.87%
All-Source Analyst |  3 |  0.87%
Cyber Workforce Developer and Manager |  2 |  0.58%
Program Manager |  2 |  0.58%
Cyber Defense Infrastructure Support Specialist |  2 |  0.58%
Security Control Assessor |  1 |  0.29%
Requirements Planner |  1 |  0.29%
Knowledge Manager |  1 |  0.29%
Information Systems Security Manager |  1 |  0.29%
Mission Assessment Specialist |  1 |  0.29%
Target Developer |  1 |  0.29%
Target Analyst |  1 |  0.29%
Language Analyst |  1 |  0.29%
Language Analyst |  1 |  0.29%
All Source-Collection Manager |  1 |  0.29%
All Source-Collection Requirements Evaluation Manager |  1 |  0.29%
Cyber Operations Planner |  1 |  0.29%
Other |  5 |  6.02%
\- Outdoor education |  |
\- Telecommunications |  |

##  Appendix B. User Education

Table B.1. Proficiency in reverse engineering .

Proficiency |  Count |  Percentage
---|---|---
Not Proficient |  28 |  37.33%
Little Proficiency |  16 |  21.33%
Somewhat Proficient |  20 |  26.67%
Very Proficient |  9 |  12.00%
Expert |  2 |  2.67%

Table B.2. Source of reverse engineering education.

Education Source |  Count |  Percentage
---|---|---
Self-taught |  57 |  61.96%
University of College |  23 |  25.00%
Employer |  12 |  13.04%

Table B.3. Beneficial resources for learning reverse engineering.

Resource |  Count |  Percentage
---|---|---
Online instructional guides |  45 |  21.13%
Online challenges or ”crackmes” |  40 |  18.78%
Textbooks |  36 |  16.90%
Online video tutorials |  33 |  15.40%
College or University lectures |  25 |  11.74%
On-the-job training |  22 |  10.33%
Employer-sponsored training |  12 |  5.63%

Table B.4. Most valuable websites for learning reverse engineering. Website |  Count |  Percentage
---|---|---
youtube.com |  36 |  31.58%
crackmes.one |  16 |  14.04%
sans.com |  16 |  14.04%
codeacademy.com |  8 |  7.02%
begin.re |  6 |  5.26%
compilerexplorer.com |  3 |  2.63%
udemy.com |  3 |  2.63%
datacamp.com |  3 |  2.63%
leetcode.com |  1 |  0.88%
hackerrank.com |  1 |  0.88%
khanacademy.com |  0 |  0.00%
Other |  9 |  14.52%
\- pwnable.kr |  |
\- pluralsight.com |  |
\- hackthebox.eu |  |
\- tuts4you.com |  |
\- rtn-team.cc |  |
\- sharplab.io |  |
\- google.com |  |
\- stackoverflow.com |  |
\- tuts4you.com |  |
\- sans.org |  |
\- overthewire.org |  |

Table B.5. Most valuable books for learning reverse engineering. Book | Count | Percentage
---|---|---
Practical Malware Analysis: The Hands-on Guide to Dissecting Malicious Software | 34 | 23.78%
The IDA/IDA Pro Book | 23 | 16.08%
The Art of Memory Forensics: Detecting Malware and Threats in Windows, Linux, and Mac Memory | 18 | 12.59%
Hacking: The Art of Exploitation | 15 | 10.49%
Reversing: Secrets of Reverse Engineering | 13 | 9.09%
Practical Binary Analysis | 13 | 9.09%
Modern X86 Assembly Language Programming: 32-bit, 64-bit, SSE, and AVX | 8 | 5.59%
Mastering Reverse Engineering | 6 | 4.20%
Reverse Engineering: Mechanisms, Structures, Systems & Materials | 3 | 2.10%
Mastering Reverse Engineering: Re-engineer Your Ethical Hacking Skills | 3 | 2.10%
Other | 7 | 4.90%
Windows Internals |  |

##  Appendix C. Reverse Engineering Tools

Table C.1. Operating systems utilized in reverse engineering environment.

Operating System |  Count |  Percentage
---|---|---
Unix/Linux |  52 |  41.94%
Windows |  47 |  37.90%
macOS |  21 |  16.94%
Other |  4 |  3.25%

Table C.2. Architectures encountered when reverse engineering.

Architecture |  Count |  Percentage
---|---|---
x86-64/AMD64 |  50 |  38.17%
x86/i386 |  45 |  34.35%
ARM |  24 |  18.32%
MIPS |  10 |  7.63%
RISC-V |  2 |  1.53%

Table C.3. Platforms encountered when reverse engineering.

Platform |  Count |  Percentage
---|---|---
Windows |  53 |  40.46%
nix |  39 |  29.77%
Android |  29 |  22.14%
Apple iOS |  10 |  4.63%

Table C.4. Languages encountered when reverse engineering.

Language |  Count |  Percentage
---|---|---
C/C++ |  58 |  30.05%
Java |  39 |  20.21%
C# |  33 |  17.10%
Compiled Python |  30 |  15.54%
Objective-C |  15 |  7.77%
Other |  11 |  5.70%
Swift |  7 |  3.63%

Table C.5. Software/Systems encountered when reverse engineering.

Software/System |  Count |  Percentage
---|---|---
Malware |  50 |  28.41%
Desktop applications |  40 |  22.73%
Mobile applications |  25 |  14.20%
Kernel |  22 |  12.50%
Embedded systems including IoT |  17 |  9.66%
Firmware |  16 |  9.09%
Other |  6 |  3.41%

##  Appendix D. Ghidra v.s. IDA

Table D.1. Free or commercial edition of IDA.

IDA Edition |  Count |  Percentage
---|---|---
Free |  25 |  50%
Commercial |  25 |  50%

Table D.2. IDA/IDA Pro GUI features that should be improved.

Feature |  Count |  Percentage
---|---|---
Debugger |  18 |  30%
Third-party plugins |  11 |  18.33%
Disassembler |  9 |  15%
Graph view |  6 |  10%
Function call graph |  4 |  6.67%
Library function signature recognition |  4 |  6.67%
Reference table |  3 |  5%
Hex view |  2 |  3.33%
Variable renaming |  0 |  0.00%
Function signature modification |  0 |  0.00%
Other |  3 |  5%

Table D.3. Does IDA/IDA Pro have any features that are advantageous over those
in Ghidra?

|  Count |  Percentage
---|---|---
Yes |  13 |  48.15%
No |  14 |  51.85%

Table D.4. Ghidra GUI features that should be improved.

Feature |  Count |  Percentage
---|---|---
Debugger |  9 |  20.93%
Graph view |  8 |  18.60%
Library function signature recognition |  8 |  18.60%
Function call graph |  6 |  13.95%
Hex view |  6 |  13.95%
Disassembler |  3 |  6.98%
Function signature modification |  1 |  2.33%
Third-party plugins |  1 |  2.33%
Data type viewer |  1 |  2.33%
Reference table |  0 |  0.00%

Table D.5. Does Ghidra have any features which are better than those in
IDA/IDA Pro?

|  Count |  Percentage
---|---|---
Yes |  13 |  52.00%
No |  12 |  48.00%

##  Appendix E. Teaching Reverse Engineering

Table E.1. Do you teach or have you ever taught reverse engineering. |  Count |  Percentage
---|---|---
No |  48 |  66.67%
Yes |  24 |  33.33%
Table E.2. Resources used to teach reverse engineering. Resource |  Count |  Percentage
---|---|---
Lab activities |  20 |  27.28%
Demonstration |  19 |  26.39%
Textbooks |  11 |  15.28%
Websites |  11 |  15.28%
Online videos |  10 |  13.89%
Other |  1 |  1.39%
Table E.3. Types of assignments used to teach students reverse engineering. Assignment Type |  Count |  Percentage
---|---|---
Projects |  15 |  37.50%
CTF-style Assignments |  15 |  37.50%
Exams |  5 |  12.50%
Written assignments |  3 |  7.50%
Other |  2 |  5.00%
\- Real Malware |  |
\- Labs |  |
Table E.4. Methods used to assess student’s knowledge of reverse engineering. Method |  Count |  Percentage
---|---|---
In-class Presentations |  8 |  25.00%
Exams |  7 |  21.88%
Timed assignments |  6 |  18.75%
Informal Verbal Assessment |  4 |  12.50%
Other |  7 |  21.88%
\- Projects |  |
\- Labs |  |
Table E.5. Challenges in teaching reverse engineering. Challenge |  Count |  Percentage
---|---|---
Students’ lack of prerequisite knowledge |  18 |  29.51%
Students’ understanding of assembly language |  13 |  21.31%
Learning curve of reverse engineering software |  11 |  18.03%
Lack of reverse engineering educational resources |  8 |  13.11%
Quality of existing reverse engineering educational resources |  7 |  11.48%
Usability of reverse engineering software |  4 |  6.56%
Table E.6. Tools used in classes to teach reverse engineering. Tool |  Count |  Percentage
---|---|---
IDA/IDA Pro |  14 |  14.29%
Ghidra |  10 |  10.20%
GDB |  10 |  10.20%
Objdump |  10 |  10.20%
Radare2 |  7 |  7.14%
Other |  6 |  6.12%
CFF Explorer |  6 |  6.12%
YARA |  4 |  4.08%
Cuckoo Sandbox |  4 |  4.08%
Binwalk |  4 |  4.08%
ApkTool |  4 |  4.08%
Remnux |  3 |  3.06%
Peda |  3 |  3.06%
Hopper |  3 |  3.06%
Dex2jar |  3 |  3.06%
Frida |  2 |  2.04%
Jeb |  1 |  1.02%
Jadx |  1 |  1.02%
Binary Ninja |  1 |  1.02%
Angr |  1 |  1.02%
Androguard |  1 |  1.02%
Uncompyle |  0 |  0.00%
Triton |  0 |  0.00%
Qira |  0 |  0.00%
Objection |  0 |  0.00%
Karakatau |  0 |  0.00%
Cheat Engine |  0 |  0.00%
Boomerang |  0 |  0.00%
Barf |  0 |  0.00%
Apk2Gold |  0 |  0.00%

##  Appendix F. Web-Based Reverse Engineering

Table F.1. Do you know of a web-based reverse engineering platform.

|  Count |  Percentage
---|---|---
No |  49 |  77.78%
Yes |  14 |  22.22%

Table F.2. Interested in a web-based reverse engineering platform?

|  Count |  Percentage
---|---|---
Maybe |  31 |  47.69%
Yes |  24 |  36.92%
No |  10 |  15.38%

Table F.3. Desired characteristics or aspects of a web-based reverse
engineering platform.

Characteristic |  Count |  Percentage
---|---|---
Access to various tools/features |  46 |  21.30%
Allows for collaboration with other users |  40 |  18.52%
Ability to create educational lessons |  31 |  14.35%
CTF style challenges |  30 |  13.89%
Community forums |  20 |  9.26%
Annual competition |  15 |  6.94%
Ability to generate groups |  13 |  6.02%
Online job fair opportunities |  13 |  6.02%
User ranking system |  8 |  3.70%
  *[RE]: Reverse Engineering
  *[IDA]: Interactive Disassembler
  *[GUI]: Graphical User Interface
  *[CISA]: Cybersecurity and Infrastructure Security Agency
  *[NSA]: National Security Agency
  *[UI]: User Interface
  *[API]: Application Programming Interface
  *[WinDbg]: Windows Debugger
  *[CPU]: Central Processing Unit
  *[IRB]: Institutional Review Board
  *[CTF]: Capture The Flag
  *[GDB]: GNU Debugger
  *[ARM]: Advanced RISC Machines
  *[DMCA]: Digital Millennium Copyright Act
  *[EULA]: End-User License Agreement
  *[CAE-CO]: Centers of Academic Excellence in Cyber Operations