necessary concepts and methods have been introduced previously, so it is now
time to present those main program in details.

####  4.1.1 Mem2Graph

The present report has already presented the Mem2Graph program throughout the
heap dump memory parsing algorithm, graph construction and embeddings. This
program has been developed in Rust and is an active collaboration between the
author and Clément Lahoche, another PhDTrack student at Passau. The program is
available on GitHub at https://github.com/passau-masterarbeit-2023/mem2graph.

The program is composed of several layers that build on top of each other. The
first layer is dedicated into loading a RAW heap dump file with its annotation
JSON file, performs some checks and prepare the data for further analysis. The
next one performs the graph construction following the algorithms introduced
in the Methods section. Another layer performs some annotations of the nodes.
The final layer is more versatile and dependent on the input program
parameters. For the need of this report, several pipelines of memgraph with
and without embeddings have been added, namely graph and graph-with-embedding-
comments. The first pipeline doesn’t perform any embedding and export the
memory graph to a text file following the DOT format [5]. The second pipeline
performs the same operations, but also exports the memory graph with the
embeddings as comments in the DOT file.

Since the prediction effort is focused on memory chunks, this embedding is
generally called with the \--no-value-node parameter, which transform the
memory graph from a graph of blocks of 8 bytes, into a memory graph of memory
chunks, connected by their pointers inside their respective user data space.
Several chunk node embeddings have been implemented, and the user can choose
which one to use. The chunk node embeddings are the following: chunk-semantic-
embedding, chunk-statistic-embedding, and chunk-start-bytes-embedding.

####  4.1.2 Machine Learning Pipelines Runner

When working with machine learning, Python is a dominant language, benefiting
from a rich ecosystem of libraries and frameworks.

The project leverages a wide range of Python libraries to build comprehensive
machine learning pipelines:

  * •

NetworkX for graph-based data structures and algorithms.

  * •

PyGraphviz for graph visualization.

  * •

Torch for tensor computations and building neural networks.

  * •

Scikit-learn for classical machine learning algorithms.

  * •

Pandas for data manipulation and analysis.

  * •

NumPy for numerical computations.

  * •

Matplotlib for data visualization.

  * •

Torch-Geometric for Geometric Deep Learning extensions for PyTorch.

The program encompasses a variety of machine learning models to compare how
different models react to the embeddings and representations developed
earlier. It includes classical machine learning models from the Scikit-learn
library, such as Random Forest, Stochastic Gradient Descent (SGD), and
Logistic Regression. These models serve a point of comparison since they don’t
rely on graph-based data structures and algorithms.

The program also includes Graph Convolutional Network (GCN) models built on
top of PyTorch and Torch-Geometric. These models are more complex and
powerful, leveraging the graph structure and embeddings to achieve better
results. Those models specifically leverage graph-based embeddings and input
data, generated using the Node2Vec algorithm to create dense and continuous
node features that can be used for subsequent analysis or machine learning
tasks.

Main Pipelines: The program is organized around three main pipelines:

  1. 1.

GCN Pipeline: For Graph Convolutional Network models, built using libraries
such as Torch and Torch-Geometric.

  2. 2.

Classical ML Pipeline: Leveraging algorithms like Random Forest, SGD, and
Logistic Regression from Scikit-learn.

  3. 3.

Feature Evaluation Pipeline: Primarily aimed at evaluating the quality and
importance of generated features or graph embeddings.

####  4.1.3 Other programs

A lot of other programs have been developed for the need of this thesis, but
they are not as important as the ones presented above. Those scripts and short
program have been developed for several purposes:

  * •

Data exploration: Several scripts have been developed to explore the data, and
to understand the structure of the heap dump files, the annotations, and the
memory graphs.

  * •

Heap Dump parsing algorithm testers: Several scripts have been developed to
test the heap dump parsing algorithms, and to ensure that the algorithms are
working as intended on all possible situations.

  * •

Graph and embedding generation testers: Others have been developed to test the
graph and embedding generation algorithms, and to ensure that the algorithms
are working as intended on all possible situations.

  * •

Result visualization, analysis and latex generators: Later in the report, the
results will be presented and discussed. Several scripts have been developed
to generate the latex tables and plots, and to analyze the results.

All those programs represent a consequent amount of work, and have been
invaluable to the success of this thesis. Using CLOC (Count Lines of Code), a
command-line utility that can count lines of code in various languages, the
following statistics have been obtained:

⬇

1 cloc mem2graph research-base predicting_ssh_key_masterarbeit_report
phdtrack_server_scripts phdtrack_project_3 memory_graph_gcn
data_processing_masterarbeit \--exclude-dir=.venv

Listing 27: Command used to count the number of lines of code in the phdtrack
directory, containing the reposiroties of the present thesis.

The following table shows the number of lines of code for each programming
language used in the present thesis:

Table 4.1: Code Statistics for Masterarbeit Language | Files | Blank Lines | Comments | Code Lines
---|---|---|---|---
CSV | 867 | 0 | 0 | 158305697
Text | 25 | 272 | 0 | 50073
Python | 132 | 2326 | 2682 | 8050
Rust | 50 | 1343 | 1149 | 6823
TeX | 28 | 963 | 141 | 6636
Markdown | 33 | 1016 | 0 | 2375
JSON | 1345 | 1 | 0 | 1677
Jupyter Notebook | 2 | 0 | 1811 | 381
Nix | 12 | 50 | 31 | 290
TOML | 1 | 2 | 1 | 22
Bourne Shell | 3 | 8 | 8 | 18
make | 1 | 8 | 3 | 18
Dockerfile | 1 | 1 | 0 | 2

In the context of this thesis, three programming languages stand out for their
specialized roles: Python, Rust, and Nix. Python is predominantly used for the
machine learning pipeline, offering ease of use and a rich ecosystem for data
science tasks. Rust serves as the backbone for the Mem2Graph program,
providing the efficiency required for graph construction and manipulation. Nix
is employed for package management and building, ensuring reproducibility
across different computing environments. These languages complement each other
well, with Python offering high-level abstractions for machine learning, and
Rust providing low-level control for performance-critical tasks. Additionally,
CSV files are utilized to store model results. TeX is used for generating this
report, highlighting the diverse yet complementary set of tools and languages
employed in this research.

###  4.2 Experimental Setup

The experimental setup serves as the backbone of this research, providing a
structured framework for conducting large-scale experiments on the server.
This section delves into the intricacies of the setup, detailing the steps
involved and the tools used. It also includes selected program output logs to
offer a granular view of the program parameters, environment, and usage.

The elements discussed in this section are not merely illustrative; they offer
invaluable insights into the challenges encountered during the experiments.
These details are particularly crucial when discussing the large-scale
experiments, as they provide a comprehensive understanding of the various
facets involved.

The final experiments were conducted in a systematic manner, following these
steps:

  1. 1.

Data Cleaning: The original Zenodo dataset was cleaned to produce a RAW heap
dump dataset, serving as the foundational data for the experiments.

  2. 2.

Graph and Embedding Generation: The Mem2Graph Rust program was employed to
generate graphs along with their embeddings. A Python launcher script
facilitated the generation of multiple memgraph datasets, each with varying
combinations of program parameters.

  3. 3.

Data Preloading and Validation: A sanity check Python program was used for
data preloading and validation, ensuring the integrity and consistency of the
data before proceeding to the experimental phase.

  4. 4.

ML and DL GCN Pipelines: The main Python program was responsible for the
seamless launching of data science tasks, machine learning training, and model
evaluation. It was designed to cover a predefined range of parameters and
model hyperparameters.

  5. 5.

Result Collection and Evaluation: The final step involved the collection,
evaluation, and visualization of the results. Error handling mechanisms were
in place to make necessary corrections and prepare for the next iteration of
experiments. This step also facilitated the confrontation of hypotheses and
research questions.

Initial small-scale tests were conducted on a laptop to validate the programs
and their results. The final, large-scale experiments were carried out on the
Drogon server, equipped with 80 threads and 256 GB of RAM. The computational
resources provided by the University of Passau have been invaluable for
conducting these experiments, sometimes running for several days straight.

####  4.2.1 Generation of the memgraph datasets

Using Mem2Graph powerful features, it is possible to generate several memgraph
datasets with different parameters. The following command has been used to
generate 6 memgraph datasets, each containing 26202 graphs, for a total of
157212 graphs. Those datasets account for different chunk node embeddings and
a potential additional filtering feature. The command is run on the Drogon
server, with 80 threads.

⬇

1 [2023-10-24T20:42:44 UTC][INFO mem_to_graph::exe_pipeline::pipeline] OK [t:
worker-63] [N*202 / 26202 files] [fid: 8683-1650977906] (Nb samples: 0)

2 [2023-10-24T20:42:44 UTC][INFO mem_to_graph::graph_data::heap_dump_data]
FILE heap dump raw file path:
"/root/phdtrack/phdtrack_data_clean/Performance_Test/V_8_1_P1/32/8794-1650977906-heap.raw"

3 [2023-10-24T20:42:44 UTC][INFO mem_to_graph::exe_pipeline::pipeline] OK [t:
worker-63] [N*203 / 26202 files] [fid: 8794-1650977906] (Nb samples: 0)

4 [2023-10-24T20:42:44 UTC][INFO mem_to_graph::exe_pipeline::pipeline] TIME
[total pipeline time: 114.84s]

5 100%|##############################| 6/6 [1:07:15<00:00, 672.62s/it]

6 Finished! Total time: hours: 1, minutes: 7, seconds: 15 (Drogon Server)

Listing 28: Sample of final logs of the Mem2Graph program, after generating 6
memgraph datasets from the cleaned heap dump dataset.

It took a little more than an hour to generate 6 memgraph datasets, each
containing 26202 graphs. The total number of graphs generated is 157212. The
generated memgraph datasets are stored in the data/ directory, as follows:

Figure 4.1: Illustration of the Data Directory Structure

.1 /data. .2 0_graph_with_embedding_comments_-v_-a_chunk-header-node_-c_chunk-
semantic-embedding_-e_none_-s_none. .2 1_graph_with_embedding_comments_-v_-
a_chunk-header-node_-c_chunk-statistic-embedding_-e_none_-s_none. .2
2_graph_with_embedding_comments_-v_-a_chunk-header-node_-c_chunk-start-bytes-
embedding_-e_none_-s_none. .2 3_graph_with_embedding_comments_-v_-a_chunk-
header-node_-c_chunk-semantic-embedding_-e_only-max-entropy_-s_activate. .2
4_graph_with_embedding_comments_-v_-a_chunk-header-node_-c_chunk-statistic-
embedding_-e_only-max-entropy_-s_activate. .2
5_graph_with_embedding_comments_-v_-a_chunk-header-node_-c_chunk-start-bytes-
embedding_-e_only-max-entropy_-s_activate.

As one can see, the dataset directory names are composed with the most
important Mem2Graph program parameters, responsible for some feature and
embedding generations. The -e flag, short for \--entropy-filter is responsible
for the filtering using the Shannon entropy, the -s for \--chunk-byte-size-
filter for chunk byte size filtering. The -c, \--graph-comment-embedding-type
controls the custom embedding being save alongside each node in the generated
.GV memgraph files.

####  4.2.2 Sanity checking and preloading the generated memgraph datasets

Loading the graph from DOT files into NetworkX graph in Python is a resource
intensive operation that consumes all the available computing power on all
tested platforms (laptop, servers). It takes several dozens of seconds up to a
minute to load a memory graph containing only around 1000 chunk nodes. It has
been experimented that saving those loaded graphs using the pickle python
library allows to perform checks while loading the graph, add more information
about each graph. The subsequent retrieval of the graph is much faster, and
allows to perform the sanity checks before any further processing. So to
verify the memory graph dataset generation as well as transforming DOT files
into pre-saved NetworkX graph, a sanity checking script has been developed.

Below is a sample of the logs generated by the sanity check script:

⬇

1 * Running program...

2 Passed program params:

3 param[0]: src/sanity_check_gv_files.py

4 param[1]: -k

5 Parsed program params:

6 keep_old_output: True

7 skip_dir_starting_with_number: None

8 dry_run: False

9 * Now, performing data loading and sanity checks...

10 * Looking for Mem2Graph dataset directories in
/root/phdtrack/mem2graph/data...

11 * Skipping .gitignore...

12 * Found 6 Mem2Graph dataset directories.

13 [...]

14 Loading graphs: 100%|##########| 26201/26202 [3:22:17<00:00, 1.17it/s]

15 Loading graphs: 100%|##########| 26202/26202 [3:22:25<00:00, 3.19s/it]

16 Loading graphs: 100%|##########| 26202/26202 [3:22:25<00:00, 2.16it/s]

17 * Checking embedding length of graphs in
/root/phdtrack/mem2graph/data/5_graph_with_embedding_comments_-v_-a_chunk-
header-node_-c_chunk-start-bytes-embedding_-e_only-max-entropy_-s_activate...

18 -> [x] 26202 graphs in
/root/phdtrack/mem2graph/data/5_graph_with_embedding_comments_-v_-a_chunk-
header-node_-c_chunk-start-bytes-embedding_-e_only-max-entropy_-s_activate
have been loaded and checked.

19 -> [_] 0 graphs in
/root/phdtrack/mem2graph/data/5_graph_with_embedding_comments_-v_-a_chunk-
header-node_-c_chunk-start-bytes-embedding_-e_only-max-entropy_-s_activate
have been skipped (deleted).

20 [x] 157212 total graphs in the input mem2graph dataset dir paths have been
loaded and checked.

21 [_] 0 total graphs in the input mem2graph dataset dir paths have been
skipped (deleted).

22 <END> Program took: 42626.062309 total sec (11h 50m 26s)

Listing 29: Result logs of the memory_graph_gcn/src/sanity_check_gv_files.py
program.

The sanity checking file can be considered very long to run, having taken
almost 12 hours straight, but it is a necessary step to ensure the quality of
the generated memgraph datasets. It is also a good way to check the validity
of the generated embeddings, and to ensure that the graphs are re-exported as
NetworkX graphs that can be loaded much faster than the original DOT files.

####  4.2.3 Launching the experiments

Two pass of experiments have been conducted, with the exact same parameters
and input memgraph dataset. Contrary to expectations, the experiments were
much faster on the laptop that on the Drogon server.

  * •

Time take for the experiments on the laptop: 12h 31m 53s

  * •

Time take for the experiments on the Drogon server: 29h 17m 57s

The experiments have been launched using the following command (here, on
Drogon server):

⬇

1 nohup python src/main_gcn.py -i /root/phdtrack/phdtrack\_data\_clean/ -p
gcn-pipeline classic-ml-pipeline feature-evaluation-pipeline -b 6 -a -q -n 16
> output\_ml\_2023\_10\_27\_16h\_35.log 2>&1 &

Listing 30: Command used to launch final experiments, on Drogon server.

Using nohup and redirecting the output to a log file allows the experiments to
run in the background and enables the retrieval of the output at a later time.
The command specifies the input directory containing the annotated DOT (.gv)
graph directory with the -i flag. All three pipelines are chosen: gcn-
pipeline, classic-ml-pipeline, and feature-evaluation-pipeline, as indicated
by the -p flag. The batch size for parallel processing is set to 6 using the
-b flag, and the -a flag indicates the use of all available Mem2Graph
datasets. The -q flag enables quiet mode for Node2Vec, and the -n flag
specifies the use of 16 input graphs.

####  4.2.4 Dealing with hyperparameter tuning

In the quest to optimize the performance of both ML and DL models,
hyperparameter tuning plays a crucial role. This section elaborates on the
various strategies and tools employed for hyperparameter tuning in this
research.

  * •

Precise Command Lines for Tuning: The compute instances and experiment
parameters can be finely tuned using precise command-line options. This
flexibility allows for a more targeted approach to model optimization,
enabling the user to specify various hyperparameters and settings right from
the terminal.

  * •

Python Program for Experiment Launch: A dedicated Python program has been
developed to automate the launching of ML, GCN and FE pipelines. This program
takes different hyperparameters as input and initiates the corresponding
experiments, thereby streamlining the entire process.

  * •

Automatic Logging in CSV: All the results from each experiment, along with the
hyperparameters used, are automatically logged into a CSV file. This
facilitates easy tracking and comparison of different model configurations and
their corresponding performances.

  * •

Timestamps and Duration Steps: Each experiment is meticulously logged with
precise timestamps and duration steps. This includes the time taken for
generating embeddings, as well as the time required for the training and
testing phases. Such detailed logging aids in identifying bottlenecks and
optimizing the pipeline further.

  * •

Extensive Experimentation: Over the course of this research, thousands of
experiments have been conducted. These experiments span a wide range of
parameters, models, and embeddings, providing a comprehensive understanding of
the model behaviors and their sensitivities to different hyperparameters.

  * •

Automated Result Analysis and Visualization: To aid in the interpretation of
the extensive experimental results, automated scripts have been developed for
result analysis and visualization. These scripts generate various plots and
metrics that provide insights into the performance and reliability of the
different models and configurations.

Of all the parameters, the model types, their respective hyperparameters, the
combinations of possible embeddings with their own parameters, and the number
of input graphs are the most important. All those parameters explains the
large number of experiments that have been conducted, and the need for a
precise and automated way to launch them.

#####  4.2.4.1 Limited number of input graphs due to compute time and memory
usage

It also explains why the input number of memgraphs has been limited to 16, as
it is already a very large number of experiments to run. Each file taking
several dozens of seconds to be transformed into an embedding, this represents
around 10 minutes just for the embedding generation phase, for each pipeline.
With more than 600 pipelines to run, this already represents dozens of hours
of compute time. Depending on the context and model, the training and
evaluation phases are also time-consuming, and the more input graphs there
are, the longer it takes to train and test the model.

The compute time is not the only issue with dealing with a large number of
input graphs. The memory usage is also a problem, as the memory usage
increases linearly with the number of input graphs. The parallel processing of
only 6 pipelines having 16 memgraphs as inputs generally represents between 16
and 50 GB of RAM usage, depending on the model and the embedding used. Due to
this, all tests consisting of trying to run this already limited number of
input graphs on the GPU have failed, as the GPU memory is not sufficient to
handle the memory usage of the program. The slowness of the CPU alongside
memory bandwidth limitations are the main bottlenecks of the program, and the
main reasons why the number of input graphs has been limited to 16.

#####  4.2.4.2 Parallel launch of experiments for hyperparameters tuning

To maximize efficiency and expedite the research process, a Python program was
developed to launch multiple experiments in parallel. This approach allows for
simultaneous testing of various input graphs, machine learning and deep
learning models, embedding techniques, and hyperparameters. By leveraging
parallel computing, the program significantly reduces the time required for
extensive experimentation, thereby accelerating the overall research and
development cycle.

Below are the hyperparameters used during the main experiments, as stored in
the hyperparams.json file:

⬇

1 {

2 "node2vec_dimensions_range": [128],

3 "node2vec_walk_length_range": [16],

4 "node2vec_num_walks_range": [50],

5 "node2vec_p_range": [0.5, 1.0, 1.5],

6 "node2vec_q_range": [0.5, 1.0, 1.5],

7 "node2vec_window_range": [10],

8 "node2vec_batch_words_range": [8],

9 "node2vec_workers_range": [16],

10 "randomforest_trees_range": [100, 500, 1000],

11 "gcn_training_epochs_range": [20]

12 }

Listing 31: JSON hyperparameters used during experiments

Even though the above JSON file shows the hyperparameters selected for the
large scale final experiments, pre-experiments have been previously conducted
to find some usable values. Due to compute time limitations, the ranges of
hyperparameters have been limited to a few values, and the number of input
graphs has been limited to 16. The following JSON file shows the
hyperparameters used during the pre-experiments:

⬇

1 {

2 "node2vec_dimensions_range": [16, 32, 128],

3 "node2vec_walk_length_range": [16, 32],

4 "node2vec_num_walks_range": [50, 100],

5 "node2vec_p_range": [0.5, 1.0, 1.5],

6 "node2vec_q_range": [0.5, 1.0, 1.5],

7 "node2vec_window_range": [10, 20],

8 "node2vec_batch_words_range": [4, 8, 16],

9 "node2vec_workers_range": [16, 32],

10 "randomforest_trees_range": [100, 500, 1000],

11 "gcn_training_epochs_range": [5, 10, 20]

12 }

Listing 32: JSON hyperparameters used during experiments

Several observations were made during the early experiments concerning the
impact of different hyperparameters on the model’s performance, especially
concerning the Node2Vec parameters. The following observations were made:

  * •

Walk Length and Number of Walks: Increasing the number of walks and the walk
length generally improved the model’s performance. However, this came at the
cost of significantly increased computational time. The benefits plateaued
after reaching a certain threshold.

  * •

Number of Dimensions: A higher number of dimensions generally led to better
results. Lower values were found to be detrimental to the model’s performance,
indicating the importance of this parameter.

  * •

Impact of $p$ and $q$: The parameters $p$ and $q$ had an unpredictable impact
on the model’s performance. While some combinations seemed to yield better
results, no clear pattern was observed, making these parameters challenging to
optimize.

  * •

Batch Word Range: The range of batch words had a minimal impact on the model’s
performance. As a result, an intermediate value was selected for this
parameter to balance computational efficiency and performance.

These observations provide valuable insights into the behavior of the models
under different hyperparameter settings and serve as a guide for future
experiments.

#####  4.2.4.3 Description of the results.csv File

The *results.csv files are used to store the results of machine learning and
deep learning classifiers, including Graph Convolutional Networks (GCNs) and
classical classifiers like Random Forest. The files are regular CSVs organized
with the following headers:

  * •

system: The operating system on which the experiment was run. Here, Linux.

  * •

node_name: The name of the node in the cluster. Here, either nixos (local
machine) or rascoussie (lab server).

  * •

release, version: OS release and version information.

  * •

machine, processor: Hardware details.

  * •

physical_cores, total_cores: Number of physical and total cores.

  * •

total_memory, available_memory: Total and available memory in bytes.

  * •

start_time: The start time of the experiment.

  * •

nb_input_graphs: Number of input graphs.

  * •

duration_embedding: Time taken for embedding.

  * •

subpipeline_name, index, pipeline_name: Information about the pipeline model
used, with values like sgd-classifier or gcn-with-dropout.

  * •

input_mem2graph_dataset_dir_path: Path to the dataset directory.

  * •

node_embedding: Type of node embedding used. Several values are possible, like
solo embeddings like chunk-header-node or node2vec, or combined embeddings
like node2vec-chunk-semantic-embedding.

  * •

node2vec_*: Parameters for the Node2Vec algorithm, like the number of walks,
the walk length, the window size, the number of dimensions, the number of
epochs, and the p and q parameters.

  * •

random_forest_*: Parameters for the Random Forest algorithm, like the number
of estimators (trees) or the number of parallel jobs.

  * •

imbalance_ratio: Ratio of the classes in the dataset. For instance, a value of
496.33 means that the dataset contains 496.33 times more negative samples than
positive samples. Since no filtering is applied, the ratio is always greater
than 1, and generally very high.

  * •

precision_class_*, recall_class_*, f1_score_class_*, support_class_*: Metrics
for each class.

  * •

true_positives, true_negatives, false_positives, false_negatives: Confusion
matrix elements.

  * •

AUC: Area under the ROC curve.

  * •

duration_train_test: Time taken for training and testing.

  * •

nb_node_features: Number of node features. This value directly depends on the
node embedding used, and the input memory graph dataset parameters used during
generation.

  * •

first_gcn_training_epochs: Number of epochs in GCN training phase.

Each row in the *results.csv files represents a single experiment run with a
specific configuration and its corresponding results. Keeping a precise track
of the parameters used for each experiment is crucial for reproducibility and
traceability. They also form the basis for the analysis and visualization of
the results.

###  4.3 Obtained Results

This section presents a comprehensive overview of the results obtained with
the final experimentation. The outcomes are detailed in multiple formats,
including correlation matrices, performance metrics tables, and
visualizations, to provide a precise understanding of the model performances.
The aim is to elucidate the effectiveness of different features and embeddings
in the context of our machine learning pipelines. In-depth discussions on
these results will be reserved for the following "Discussions" part.

####  4.3.1 Feature Engineering results

This section delves into the results obtained from the feature engineering
efforts carried out during the experiments. The results are presented in
multiple correlation matrices.

Figure 4.2: Feature correlation matrices on the different Mem2Graph-generated
datasets. Used algorithm: Kendall. Figure 4.3: Feature correlation matrices on
the different Mem2Graph-generated datasets. Used algorithm: Pearson. Figure
4.4: Feature correlation matrices on the different Mem2Graph-generated
datasets. Used algorithm: Spearman.

####  4.3.2 Classic Model results

Tables are provided to summarize the performance of different pipelines and
models. These tables include four classical machine learning metrics:
precision, recall, F1 score, and the Area Under the Curve (AUC). Each table
offers a snapshot of how well each model performs on the key chunk prediction
task.

Table 4.2: Best instances of model: logistic-regression. Best at | Precision | Recall | F1 Score | AUC
---|---|---|---|---
precision | 1.0000 | 0.0417 | 0.0800 | 0.5208
recall | 0.3333 | 0.5000 | 0.4000 | 0.7471
f1 score | 0.3333 | 0.5000 | 0.4000 | 0.7471
AUC | 0.2449 | 0.5000 | 0.3288 | 0.7486
Table 4.3: Best instances of model: random-forest. Best at | Precision | Recall | F1 Score | AUC
---|---|---|---|---
precision | 1.0000 | 0.0417 | 0.0800 | 0.5208
recall | 1.0000 | 0.0833 | 0.1538 | 0.5417
f1 score | 1.0000 | 0.0833 | 0.1538 | 0.5417
AUC | 1.0000 | 0.0833 | 0.1538 | 0.5417
Table 4.4: Best instances of model: sgd-classifier. Best at | Precision | Recall | F1 Score | AUC
---|---|---|---|---
precision | 1.0000 | 0.0417 | 0.0800 | 0.5208
recall | 0.4615 | 1.0000 | 0.6316 | 0.9962
f1 score | 0.4615 | 1.0000 | 0.6316 | 0.9962
AUC | 0.4615 | 1.0000 | 0.6316 | 0.9962

####  4.3.3 Deep Learning GCN Model results

Best models obtained after the hyperparameter search, on a range of embeddings
and models, this time focusing on the GCN models.

Table 4.5: Best instances of model: very-simple-gcn. Best at | Precision | Recall | F1 Score | AUC
---|---|---|---|---
precision | 0.6000 | 0.5000 | 0.5455 | 0.7489
recall | 0.2609 | 1.0000 | 0.4138 | 0.9907
f1 score | 0.6000 | 0.5000 | 0.5455 | 0.7489
AUC | 0.2609 | 1.0000 | 0.4138 | 0.9907
Table 4.6: Best instances of model: simple-gcn. Best at | Precision | Recall | F1 Score | AUC
---|---|---|---|---
precision | 0.5000 | 0.5000 | 0.5000 | 0.7484
recall | 0.2308 | 1.0000 | 0.3750 | 0.9891
f1 score | 0.5000 | 0.5000 | 0.5000 | 0.7484
AUC | 0.2609 | 1.0000 | 0.4138 | 0.9907
Table 4.7: Best instances of model: first-gcn. Best at | Precision | Recall | F1 Score | AUC
---|---|---|---|---
precision | 0.5000 | 0.5000 | 0.5000 | 0.7484
recall | 0.2727 | 1.0000 | 0.4286 | 0.9913
f1 score | 0.5000 | 0.5000 | 0.5000 | 0.7484
AUC | 0.2727 | 1.0000 | 0.4286 | 0.9913
Table 4.8: Best instances of model: gcn-with-dropout. Best at | Precision | Recall | F1 Score | AUC
---|---|---|---|---
precision | 0.3500 | 0.2333 | 0.2800 | 0.6152
recall | 0.0863 | 0.8000 | 0.1558 | 0.8810
f1 score | 0.2110 | 0.7667 | 0.3309 | 0.8767
AUC | 0.0863 | 0.8000 | 0.1558 | 0.8810
Table 4.9: Best instances of model: advanced-gcn. Best at | Precision | Recall | F1 Score | AUC
---|---|---|---|---
precision | 0.2097 | 0.4333 | 0.2826 | 0.7129
recall | 0.0533 | 0.9000 | 0.1006 | 0.8943
f1 score | 0.2097 | 0.4333 | 0.2826 | 0.7129
AUC | 0.1552 | 0.9000 | 0.2647 | 0.9390

###  4.4 Compared Performances of models and embeddings

In our experiments, we limited the analysis to a maximum of 16 input graphs,
which may appear to be a relatively low number. However, this limitation was
necessary due to the extensive range of hyperparameters, embeddings, and
models that we aimed to evaluate. Despite this constraint, we were able to run
a total of 7976 machine learning pipelines, accumulating over 100 hours of
compute time. This extensive computational effort underscores the complexity
and depth of the feature engineering and model evaluation processes undertaken
in this study. The following tables and visualizations provide a comprehensive
overview of the results obtained from these experiments.

Table 4.10: Results for the model very-simple-gcn Model | Best Precision | Best Recall | Best F1 Score | Best AUC
---|---|---|---|---
advanced-gcn | 0.2097 | 0.9000 | 0.2826 | 0.9390
first-gcn | 0.5000 | 1.0000 | 0.5000 | 0.9913
gcn-with-dropout | 0.3500 | 0.8000 | 0.3309 | 0.8810
logistic-regression | 1.0000 | 0.5000 | 0.4000 | 0.7486
random-forest | 1.0000 | 0.0833 | 0.1538 | 0.5417
sgd-classifier | 1.0000 | 1.0000 | 0.6316 | 0.9962
simple-gcn | 0.5000 | 1.0000 | 0.5000 | 0.9907
very-simple-gcn | 0.6000 | 1.0000 | 0.5455 | 0.9907

In addition to the tabular data, we also offer visualizations to facilitate a
more intuitive comparison between models and embeddings. These graphical
representations aim to make the complex data more digestible and provide
insights that may not be immediately obvious from the tables alone.

Figure 4.5: Visualization of the result metrics use to compare model
performance on memory graphs, for different embeddings and hyperparameters.
Figure 4.6: Visualization of the result metrics use to compare model
performance per memory graph node embedding strategies.

It’s worth noting that while this section provides a detailed account of the
results, an in-depth discussion about these findings, their implications, and
potential future work will be covered in the following "Discussions" section.

## 5 Discussion

In the previous chapter, the results of the experiments were presented. This
chapter aims to provide an in-depth discussion of those results, as well as to
identify the limitations of the experiments and to propose avenues for future
research.

###  5.1 Discussion of the results

The following subsections will discuss the results obtained in the
experiments, and will provide insights into the performance of the different
models, as well as the impact of the different features and embeddings on the
performance of the models.

####  5.1.1 Objectives of the experiments

The primary objectives of the experiments conducted in this study are multi-
faceted. First and foremost, we aim to demonstrate the feasibility of
utilizing machine learning and deep learning models to predict chunks with
keys in the OpenSSH program based on a graph-like representation of the heap
dump files provided in the original dataset. To achieve this, we employ a
range of algorithms to extract features from memory graphs, or ’memgraphs’.
These algorithms include not only custom solutions tailored to our specific
needs but also well-established, powerful algorithms like Node2Vec.
Furthermore, we seek to evaluate the impact of these diverse features on the
performance metrics of the predictive models. Lastly, we compare the
performances of various models to identify the most effective approaches for
our specific use-case.

####  5.1.2 Discussing features and embeddings

In this section, we delve into the intricacies of the features and embeddings
used in our experiments, focusing particularly on their interrelationships as
revealed by correlation matrices. Correlation matrices provide a quantitative
measure of how different features of custom embeddings relate to each other.
Each cell in the matrix represents the correlation coefficient between two
features, which ranges from -1 to 1. A high positive value indicates a strong
positive correlation, meaning that as one feature increases, the other tends
to also increase. A negative value would indicate the opposite.

It’s worth noting that performing this analysis on Node2Vec embeddings is
generally considered irrelevant. Node2Vec embeddings are designed to capture
the neighborhood structure of nodes in a way that is optimized for machine
learning tasks, and their dimensions do not have an easily interpretable
meaning. Therefore, analyzing the correlation between different dimensions of
a Node2Vec embedding is unlikely to provide insights that are useful for
feature engineering.

To interpret the correlation matrices, we use a color-coded system where red
signifies high correlation and blue signifies low or no correlation. In the
context of machine learning, understanding feature correlation is crucial for
several reasons:

  * •

Feature Selection: Highly correlated features carry redundant information,
which may not only be unnecessary but can also lead to overfitting and poor
generalization.

  * •

Interpretability: Understanding how features are correlated can provide
insights into the underlying structure of the data and the problem being
solved.

  * •

Computational Efficiency: Eliminating correlated features can reduce the
dimensionality of the problem, making the model simpler and faster to train.

Therefore, the correlation matrices serve as a valuable tool for both feature
selection and model interpretation. In that context, and looking at the
correlation matrices provided for the different algorithms like Pearson,
Spearman, and Kendall correlation, we can see that the correlation between the
different features is generally very low, meaning that the features are not
correlated to each other. This is a good thing, as it means that the features
are not redundant, and that they are all bringing new information to the
model. No matter the correlation algorithm used, the matrices look very
similar, and the correlation between the features is very low. The only
features that stands a bit from the rest are the features corresponding to the
filtering and entropy. This is actually just a sign that the entropy was
indeed used in the filtering algorithm, since key chunks are generally more
entropic than non-key chunks. In practice, that’s also why the experiments
have been run with and without this filtering feature.

####  5.1.3 Classic ML models performances

In this subsection, we discuss the performance of three tested classical
binary classification machine learning models, namely Logistic Regression,
Random Forest, and SGD Classifier, in the context of key chunk prediction. The
models were evaluated based on four key metrics: Precision, Recall, F1 Score,
and AUC (Area Under the Curve).

#####  5.1.3.1 Logistic Regression

The Logistic Regression model excels in precision with a perfect score of
1.0000 but falls short in recall, F1 score, and AUC. The model is highly
precise but fails to capture the majority of the positive instances, as
indicated by the low recall of 0.0417. This suggests that while the model
makes very few false-positive errors, it misses a large number of true
positives.

#####  5.1.3.2 Random Forest

Random Forest shows excellent precision at the expense of recall. It has a
high precision of 1.0000 but a very low recall of 0.0833, indicating that it
is precise but not sensitive. The AUC of 0.5417 suggests that the model’s
ability to distinguish between the classes is slightly better than random
guessing.

#####  5.1.3.3 SGD Classifier

The SGD Classifier stands out in terms of recall and AUC, both scoring close
to a perfect score This indicates that the model is excellent at identifying
all the positive instances and distinguishing between the two classes.
However, its precision isn’t always perfect, suggesting a higher rate of false
positives.

#####  5.1.3.4 Comparison

Upon comparing the three models, it’s evident that each has its own strengths
and weaknesses. Logistic Regression has a much better overall recall than
Random Forest which is very precise but fails to find a lot of cases. However,
SGD Classifier is clearly the best overall model here, since it is highly
sensitive and excellent in class separation but lacks in precision.

In summary, the SGD Classifier would be more appropriate in the binary
classification task of predicting is a chunk is a key chunk or not. Random
Forest offers a balanced but mediocre performance and could serve as a
baseline model. Those models merely serve as a comparison point for the deep
learning models.

####  5.1.4 GCN models performances

In this subsection, we delve into the performance metrics of five different
Graph Convolutional Networks (GCN)models: Very Simple GCN, Simple GCN, First
GCN, GCN with Dropout, and Advanced GCN. These models were evaluated on the
same four key metrics as the classical models: Precision, Recall, F1 Score,
and AUC.

#####  5.1.4.1 Very Simple GCN

The Very Simple GCN model shows generally balanced performances. It’s worth
noting that the best instance of the model reached a perfect recall of 1.0000,
indicating excellent sensitivity. However, the precision of the model is at
best only of 0.6000, suggesting a higher rate of false positives. Its best
instance having an AUC score of 0.9907 suggests excellent class separation
capabilities.

#####  5.1.4.2 Simple GCN

This model has similar or slightly lower performance metrics to the Very
Simple GCN, with a precision and recall of 0.5000. The AUC score is slightly
lower at 0.9891 but still indicates excellent class separation.

#####  5.1.4.3 First GCN

The First GCN model also looks very similar to the two previous models. Its
best AUC instance has a high AUC indicating excellent sensitivity and class
separation.

#####  5.1.4.4 GCN with Dropout

This model has the lowest precision among the GCN models at 0.3500 but shows a
decent AUC score of 0.8810. The model overall seems to be more sensitive, but
less precise compare to the simpler models introduced before. This was indeed
expected due to the use of dropout, which is known to increase sensitivity at
the expense of precision.

#####  5.1.4.5 Advanced GCN

The Advanced GCN model, despite its complexity, does not outperform the
simpler models in this limited dataset. It displays the worst performances in
precision, f1 score and AUC, but has the best recall of all the GCN models, at
0.9000. This suggests that the model is very sensitive but not very precise,
which is not surprising considering the complexity of the model, and the
limited number of input memgraphs. This trend is likely to change if tested
with a larger number of training graphs.

#####  5.1.4.6 Layer Complexity

It’s worth noting that the simpler models (Very Simple and Simple GCN) with
only 2 layers tend to perform better in this limited training dataset. This
could be due to the low number of example graphs, which might not be
sufficient to train the more complex, 4-layer Advanced GCN model effectively.
Yet, the Advanced GCN model has the best recall of all the GCN models,
suggesting that it is more sensitive than the simpler models.

In summary, each GCN model has its own set of strengths and weaknesses. While
some excel in precision, others are more sensitive or better at class
separation. The choice of model would depend on the specific requirements of
the task at hand.

####  5.1.5 Comparing the embeddings impact on the models

Based on the results of the experiments, it’s evident that the choice of
embedding has a significant impact on the performance of the models. As this
can be seen in 4.6, and surprisingly, it seems that the custom embeddings
developed specifically for this research are actually not the best performing
embeddings. As such, adding the features from those custom embeddings to the
Node2Vec embeddings does not improve the performances of the models, but
actually degrade them. This is a very surprising result. The Node2Vec
embedding is actually the best performing embedding, no matter the model used,
which is even more surprising, since the GCN models were actively using the
graph structure, which is not the case for the classic ML models. This means
that the Node2Vec embedding, which is purely based on the graph structure, is
actually the best performing embedding, and that the custom embeddings are not
bringing any additional information to the models. Thus, this demonstrates
that the memgraph structure is in and on itself sufficiently meaningful to
perform the classification task.

It’s worth mentioning the master thesis of Clément Lahoche that actually dives
deeper in the question of embedding quality and impacts. His work displays
significantly different results than the ones obtained in this research about
embeddings. It shows that the question of embeddings is indeed a complex
topic, and that the results can vary a lot depending on the dataset, the
model, and the approach used.

The big difference between this present work and his thesis is that he used a
much larger number of input memgraphs, but also that he actually perform a
real rebalancing on the dataset before training. This means that whereas the
experiments perform in this work have been dealing with a very high imbalance
rate, his classifiers have been trained on a much better balanced dataset.
This is a huge difference, and it’s not surprising that the results are so
different. The reason why this thesis did not use active rebalancing is that
we wanted to explore the impact of learning on a full memgraph dataset,
without alteration of those memgraphs. This is due to the fact that this work
has been focused around those graphs and their equivalent in the
classification phase with GCNs that rely on the graph structure. The goal was
to see if the graph structure was enough to perform the classification, and
the results are very promising. Results show that the imbalance ratio is not a
problem for the models, as they are able to perform well despite the very high
imbalance ratio.

####  5.1.6 Comparing GCN and classical ML models

In this subsection, we aim to compare the performance of classical machine
learning models with Graph Convolutional Networks (GCNs) for the task of key
chunk prediction. The comparison is based the results of the experiments, the
best instances of each model on each metric, but also the distribution of the
metrics as illustrated in 4.5.

#####  5.1.6.1 Overall Performance

For overall balanced performance, the SGD Classifier from the classical models
and the Very Simple GCN from the GCN models stand out. The SGD Classifier
excels in recall and AUC, making it highly sensitive and excellent in class
separation, although it lacks in precision. On the other hand, the Very Simple
GCN shows balanced metrics across the board. It has a decent recall and AUC,
but its precision is not perfect. However, it is the best performing GCN model
in terms of precision.

#####  5.1.6.2 Precision

If precision is the primary concern, then Logistic Regression and Random
Forest from the classical models and the Very Simple GCN from the GCN models
are the best choices. Logistic Regression and Random Forest both have a
perfect precision score of 1.0000, while the Very Simple GCN has a precision
of 0.6000, which is the highest among the GCN models.

#####  5.1.6.3 Recall

For applications where high recall is crucial, the SGD Classifier from the
classical models and the Advanced GCN from the GCN models are the most
suitable. The SGD Classifier has a near-perfect recall, while the Advanced GCN
has the highest recall among the GCN models at 0.9000.

#####  5.1.6.4 Class Separation (AUC)

If the ability to distinguish between classes is of utmost importance, then
the SGD Classifier from the classical models and the Very Simple GCN from the
GCN models are the best options. Both models have AUC scores close to 1,
indicating excellent class separation capabilities, between key and non-key
chunks.

#####  5.1.6.5 Considerations for Small Datasets

It’s worth noting that the simpler GCN models (Very Simple and Simple GCN)
tend to perform better on the limited dataset used for training. This suggests
that for small datasets, simpler models may be more effective. The Advanced
GCN model, despite its complexity, does not perform as well but shows promise
in terms of high recall, which could potentially improve with more training
data.

#####  5.1.6.6 Summary

In summary, the choice of model would depend on the specific metric that is
most important for the task. For balanced performance, the SGD Classifier and
Very Simple GCN are recommended. For high precision, Logistic Regression or
Very Simple GCN should be considered. For high recall, the SGD Classifier or
Advanced GCN would be the most appropriate. Finally, for excellent class
separation, the SGD Classifier and Very Simple GCN are the best choices
according to the experiments.

###  5.2 Limitations of the Experiments

While the experiments conducted offer valuable insights into the performance
and capabilities of the machine learning and deep learning classifiers, it is
crucial to acknowledge the limitations that were inherent in the experimental
setup. These limitations range from computational resources to the scale and
duration of the experiments. Understanding these constraints is essential for
interpreting the results accurately and for identifying avenues for future
research. This section aims to discuss the following limitations in detail:

  * •

Number of Compute Instances: The experiments were constrained by the available
number of compute instances, affecting the scale at which they could be
conducted.

  * •

Number of Input Graphs: The quantity of input graphs used in the experiments
was rather limited, which could impact the generalizability of the results.

  * •

Duration of the Experiments: The time allocated for each experiment was
finite, potentially affecting the depth of the analysis.

  * •

CPU-only Environment: The experiments were conducted in a CPU-only setting,
without the acceleration benefits that a GPU could offer, due to problems of
memory consumption being too high for the GPU. Additional work on this aspect
could significantly improve the performance of the experiments, especially the
Node2Vec embedding generation.

  * •

High Memory Bandwidth Usage: The experiments were characterized by high memory
bandwidth usage, which could have implications for performance.

The subsequent subsections will delve into each of these limitations,
providing a comprehensive understanding of their impact on the experiments.

####  5.2.1 Limited number of input graphs

The number of input graphs is a parameter that can be easily changed, and the
experiments can be run again with a higher number of input graphs, but it
would take a very long time to run, and the results would be similar although
improved to the ones obtained with 16 input graphs. Improving the performances
could be done essentially by recoding the program Node2Vec embedding part and
adding those results directly inside the Mem2Graph program. Leveraging the
Rust zero-abstraction costs philosophy, it would be possible to improve the
performances by a probable factor of 100 to 1000 times, and to run the
experiments with a higher number of input graphs. For ease of development and
handling of the results, I would still recommend to perform the machine
learning related experiments using the powerful Python dedicated libraries.

That being said, it’s remarkable that the models can perform so well
considering the very small number of training memgraph and the very high
imbalance ratio of the dataset. The imbalance ratio is the ratio of the number
of negative samples over the number of positive samples. In the case of the
dataset used in this research, the imbalance ratio is very high, ranging
generally at several hundred times more negative samples than positive
samples. No rebalancing has been performed on the dataset since we wanted to
explore the impact of learning on a full memgraph dataset, without alteration
of those memgraphs.

The results obtained in this research are already very promising, and the
imbalance ratio is not a problem for the models, as they are able to perform
very well despite the very high imbalance ratio.

####  5.2.2 Duration of the experiments

Although a lot of efforts have been put to deal both with dataset reduction,
for instance transforming the initial block address prediction into a chunk
address prediction problem, then optimizing a lot of computing through the use
of a dedicated Rust parallel and optimized program, then using techniques like
file preloading, the sheer number of hyperparameters and the number of
experiments to run, as well as the compute time for the Node2Vec embedding
generation, make the experiments very long to run.

Below are the duration times for the different steps of the experiments.

Table 5.1: Duration times for duration of embedding generation in ML/DL/FE pipeline (in seconds). Model | Min duration | Max duration | Min duration
---|---|---|---
advanced-gcn | 506.5721079074733 | 1548.909129 | 0.129933
first-gcn | 503.49931116140345 | 1548.909129 | 0.129933
gcn-with-dropout | 506.5721079074733 | 1548.909129 | 0.129933
logistic-regression | 505.3690870955711 | 1565.660571 | 0.06828
random-forest | 505.3690870955711 | 1565.660571 | 0.06828
sgd-classifier | 505.3690870955711 | 1565.660571 | 0.06828
simple-gcn | 506.5721079074733 | 1548.909129 | 0.129933
very-simple-gcn | 506.5721079074733 | 1548.909129 | 0.129933

Considering the tested models are not especially complex, and since the number
of input memgraph stays limited, the duration of the training and testing
steps is actually quite small:

Table 5.2: Duration times for duration of training and testing in ML/DL/FE pipeline (in seconds). Model | Min duration | Max duration | Min duration
---|---|---|---
advanced-gcn | 10.843279690391459 | 496.664757 | 2.247229
first-gcn | 5.5063934491228075 | 279.007307 | 0.583914
gcn-with-dropout | 8.272014035587189 | 418.445809 | 0.905496
logistic-regression | 1.3495811305361307 | 4.165722 | 0.362695
random-forest | 11.72722453030303 | 48.723031 | 0.315739
sgd-classifier | 0.6751382750582751 | 6.405859 | 0.020952
simple-gcn | 4.337255024911032 | 163.587265 | 0.509536
very-simple-gcn | 8.188304871886121 | 58.307836 | 0.242535

All those values have been produced only by the python pipeline program. The
embedding time is actually mostly accounting for the Node2Vec generation,
since the other embeddings are already loaded in memory as they are produced
and included in the outputs of Mem2Graph. The Node2Vec generation is the most
time-consuming part of the pipeline, and it is the reason why the experiments
take so long to run. Transferring this algorithm into Mem2Graph would be a
huge improvement, and would allow to run the experiments with a much higher
number of input memgraphs.

Throughout this report, we have introduced a lot of concepts, diverse
algorithms, and different models. The experiments conducted in this research
were limited in scope due to the focus around a large set of models,
embeddings and hyperparameters which already represented a consequent amount
of work and computational resources. However, there are many more avenues for
future research, which will be introduced with the conclusion of this report.

## 6 Conclusion

The evolving landscape of cybersecurity necessitates robust techniques for
safeguarding digital communications. OpenSSH, a pivotal element in this
landscape, is a popular implementation of the Secure Shell (SSH) protocol,
which enables secure communication between two networked devices. The protocol
is widely used in the industry, particularly in the context of remote access
to servers. Using digital forensic techniques, it is possible to extract the
SSH keys from memory dumps, which can then be used to decode encrypted
communications thus allowing the monitoring of controlled systems. At the crux
of this Masterarbeit is the development of algorithms and machine learning
models to predict SSH keys within these heap dumps, focusing on using graph-
like-structures and vectorization for custom embeddings. With an
interdisciplinary approach that fuses traditional feature engineering with
graph-based methods as well as memory modelization for inductive reasoning and
learning inspired by recent developments in Knowledge Graph (KG)s, this
research not only leverages existing machine learning paradigms but also
explores new avenues, such as Graph Convolutional Networks (GCN)applied to
memory forensics. The present work also introduces a new memory forensics
tool, mem2graph, which is designed to be modular and extensible, and which can
be used to generate memory graphs from memory dumps.

###  6.1 Summary of Results

Below is a summary of the results achieved in the present work.

####  6.1.1 Dataset Exploration

A careful exploration of the dataset, and deep understanding of the original
heap dumps have been invaluable in discovering patterns in the raw data. This
exploration has allowed the development of a range of parsing algorithm able
to extract information like structure and content of a given heap dump.

It has been discovered that the problem of finding the address of keys in the
heap dump can be reduced to identifying the chunks that contain those keys.
This allows to reduce the size of the problems from around 100 000 of blocks
per heap dump, to around 1000 chunks per file. This also allows to concentrate
the heap dump memory graph representation around the chunks.

It has also been demonstrated that two powerful chunk filtering techniques can
drastically reduce the number of chunks to consider. The first filter
criterion consists in the Shannon’s entropy value of a chunk user data start
bytes. This is because the keys are expected to have a high entropy compared
to other raw data types. The second important criterion is the chunk byte
size. It has been shown that key chunks actually have a small size in the
range of possible key size. If filtering is not possible, as it is the case
with GCN models, those filters can actually be converted in powerful float and
boolean features.

However, its worth noting that instead of doing active node filtering and data
rebalancing, we have run the experiments and model training and evaluation on
full memory graphs with high imbalance ratio. This is because we wanted to
test GCN that are able to handle imbalanced data with graphs of varying size,
and because we wanted to test the ability of the GCN to learn from the
imbalanced data.

####  6.1.2 Memory Graph Generation

This Masterarbeit has introduced a range of algorithms able to generate memory
graphs from memory dumps. The algorithms are designed to be as generic as
possible, and can be applied to any memory dump dataset. The algorithms are
mostly implemented in the mem2graph program, and many exploration and sanity
checking scripts are also available in Python.

With those algorithms, it is possible to parse a RAW heap dump file, and
transform it into a memory graph, or memgraph. This graph is a data structure,
where each node represents a memory block with a precise address in the heap.
Each edge represents either a pointer pointing to another block, or
materialize the fact that a block belongs to a specific chunk. In order to
reduce the size of the graph, it is possible to compact the block graph into a
chunk graph, where each node represents a chunk, and chunks are connected
through their pointers. Those kinds of graphs are only composed of Chunk
Header Nodes whose address is considered the address of the related chunk.
This allows to reduce the size of the graph by a factor of 10 to 100.

####  6.1.3 Feature Engineering and Embeddings

The memory graph can be used to extract features from the memory dump, and to
apply machine learning algorithms to the memory dump. It can also be used for
direct graph visualization. The memory graph serves as a direct source of
embedding whether they are made manually or using readily available and tested
techniques like RandomWalks or Node2Vec.

All those embeddings can be combined. The feature evaluation has shown that
those features are very lowly correlated, meaning that their quality is high.
However, all those different embeddings doesn’t have the same results on the
ML and GCN models, depending on the strengths and weaknesses of the different
models.

However, it’s worth noting that the best results are always obtained when
using the Node2Vec embedding, no matter the type of model used. These
observations are likely to be due to the fact that Node2Vec is able to capture
the structure of the graph, and that the structure of the graph is the most
important feature for the models, given a relatively small number of input
memgraph and considering that those memgraphs are highly imbalanced.

####  6.1.4 Conclusion on Model Performances

In this study, we compared the efficiency of classical machine learning models
and Graph Convolutional Networks (GCN)in the task of key chunk prediction. Our
findings indicate that the choice of model largely depends on the specific
metric of interest. For a balanced performance encompassing recall, precision,
and AUC, the SGD Classifier from the classical models and the Very Simple GCN
from the GCN models are the most promising.

If precision is the primary metric of concern, Logistic Regression and Random
Forest from the classical models excel with perfect scores, while the Very
Simple GCN leads among the GCNs. For scenarios where high recall is crucial,
the SGD Classifier and the Advanced GCN model stand out. Both models also
perform exceptionally well in class separation, as indicated by their high AUC
scores.

It’s also worth noting that simpler GCN models like the Very Simple and Simple
GCN tend to perform better on limited datasets, suggesting their suitability
for tasks with constrained data availability. In contrast, more complex models
like the Advanced GCN show promise in terms of high recall but require more
extensive training data for optimal performance.

In summary, the optimal model selection is contingent upon the specific
requirements of the task, whether it be balanced performance, high precision,
high recall, or excellent class separation. The choice of model also depends
on the availability of training data. For instance, if the dataset is limited,
a simpler GCN model like the Very Simple GCN is preferable. However, if the
dataset is extensive, a more complex model like the Advanced GCN is more
suitable.

###  6.2 Outlook on Future Work

The current report, in conjunction with the associated Masterarbeit, has
introduced numerous novel algorithms and implementations. These have been
instrumental in addressing the initial research questions. However, as with
most research endeavors, new queries and potential avenues for enhancement
have emerged, paving the way towards further exploration.

The methodologies and algorithms introduced for the OpenSSH memory dump
dataset are versatile and can be extended to other memory dump datasets
utilizing the GLIBC library. Given that this library is the default for Linux,
adapting the methods from this Masterarbeit to other applications requires
minimal effort. The mem2graph program is inherently modular and built for
extensibility. Furthermore, this tool can be employed to produce memory graphs
for diverse datasets. Thanks to the universal character of the generated
embeddings and memory graphs, new datasets can be readily integrated into the
ML and DL pipelines crafted in Python. While an extensive array of features
and embedding techniques have been explored in this report, there remains
ample opportunity for innovative experimentation.

For a seamless fusion of machine learning into the mem2graph program, further
effort is required. Embedding machine learning immediately post-memory graph
creation can substantially boost efficiency, particularly when aiming to craft
a real-time OpenSSH memory forensics utility. However, this integration is
challenging due to the current limited ML support within Rust.

Another avenue for enhancement involves analyzing the effects of different C
libraries on allocated chunks and the layout of heap dump memory.
Investigating various languages could also be insightful. Depending on the
level of variation encountered, modifications to the algorithms might be
required, especially concerning the architecture involved in generating or
extracting heap dump configurations. Pursuing this direction could
significantly advance the development of a universal machine learning-assisted
memory forensics tool for key extraction.

While the background section underscores the vast array of ML architectures
available, it’s clear that not all can be thoroughly explored. This research
has primarily addressed the most common and promising ones, yet numerous
others await investigation. The tools crafted to bolster ML pipelines present
a solid foundation for such endeavors. Another dimension to consider is
hyperparameter optimization. Given the constraints of time and resources, only
certain parameter ranges were tested. Expanding these tests, incorporating
larger datasets, and harnessing increased computational capacity can directly
enhance performance.

## Appendix

###  .3 Code and files

⬇

1 strict digraph "17016-1643962152" {

2 "CHN(0x558343d21d40)" [label="CHN(1)" color="cyan" style=filled
shape=square];

3 "CHN(0x558343d1a448)" [label="CHN(2)" color="cyan" style=filled
shape=square];

4 "VN(0x558343d1a450)" [label="VN" color="grey" style=filled];

5 "VN(0x558343d1a458)" [label="VN" color="grey" style=filled];

6 "PN(0x558343d24ae8)" [label="PN" color="orange" style=filled shape=hexagon];

7 "KN_KEY_A(0x558343d29460)" [label="KN(A)" color="green" style=filled];

8 "KN_KEY_B(0x558343d2b960)" [label="KN(B)" color="green" style=filled];

9 "CHN(0x558343d21d40)" -> "KN_KEY_A(0x558343d29460)" [label="dts" weight=1]

10 "PN(0x558343d204e8)" -> "KN_KEY_A(0x558343d29460)" [label="ptr" weight=1]

11 "CHN(0x558343d21d40)" -> "KN_KEY_B(0x558343d2b960)" [label="dts" weight=1]

12 "PN(0x558343d2deb8)" -> "KN_KEY_B(0x558343d2b960)" [label="ptr" weight=1]

13 "CHN(0x558343d21d40)" -> "KN_KEY_C(0x558343d29080)" [label="dts" weight=1]

14 "PN(0x558343d204e0)" -> "KN_KEY_C(0x558343d29080)" [label="ptr" weight=1]

15 "PN(0x558343d24ae8)" -> "VN(0x558343d1a010)" [label="ptr" weight=1]

16 "PN(0x558343d1a240)" -> "VN(0x558343d20680)" [label="ptr" weight=1]

17 }

Listing 33: The DOT file of uncompressed block memgraph, here
Training/basic/V_7_1_P1/24/17016-1643962152-heap.raw, with real addresses.
Output is cropped.

###  .4 Memory Graphs

Figure 1: Visualization of the full memory graph generated from
Training/scp/V_7_8_P1/16/302-1644391327-heap.raw. Figure 2: Visualization of
the full memory graph generated from
Training/basic/V_7_1_P1/24/17016-1643962152-heap.raw. Figure 3: Visualization
of a truncated memory graph generated from
Training/basic/V_7_1_P1/24/17016-1643962152-heap.raw. Here with real
addresses.

Generated using a slightly different command, for better layout of the nodes:

⬇

1 sfdp -Gsize=30! -Goverlap=ortho -Tpng 17016-1643962152_truncated.gv >
17016-1643962152_truncated.png

Listing 34: Command used to generate the memory graph visualization of
Training/basic/V_7_1_P1/24/17016-1643962152-heap.raw here using real
addresses. Figure 4: Visualization of a chunk memory graph generated from
Validation/Validation/basic/V_7_8_P1/24/8708-1643979488-heap.raw. Figure 5:
Visualization of a chunk memory graph generated from
Training/Training/basic/V_6_8_P1/24/28621-1643890740-heap.raw.

###  .5 Latest ML Experiment results

The following are the latest results for ML experiments. Those tables were
generated after the submission of the thesis.

Table 1: Best instance for each model, with respect to accuracy. Model | Accuracy | Precision | Recall | F1 Score | Embedding
---|---|---|---|---|---
random-forest | 0.9984 | 1.0000 | 0.0833 | 0.1538 | chunk-start-bytes
sgd-classifier | 0.9984 | 0.6250 | 0.2083 | 0.3125 | node2vec
logistic-regression | 0.9983 | 1.0000 | 0.0417 | 0.0800 | node2vec
advanced-gcn | 0.9969 | 0.0000 | 0.0000 | 0.0000 | node2vec
gcn-with-dropout | 0.9969 | 0.0000 | 0.0000 | 0.0000 | node2vec
first-gcn | 0.9963 | 0.0000 | 0.0000 | 0.0000 | node2vec-chunk-statistic
simple-gcn | 0.9962 | 0.0000 | 0.0000 | 0.0000 | chunk-statistic
very-simple-gcn | 0.9959 | 0.0000 | 0.0000 | 0.0000 | node2vec
Table 2: Best instance for each model, with respect to precision. Model | Accuracy | Precision | Recall | F1 Score | Embedding
---|---|---|---|---|---
logistic-regression | 0.9944 | 1.0000 | 0.0417 | 0.0800 | node2vec
random-forest | 0.9983 | 1.0000 | 0.0417 | 0.0800 | chunk-start-bytes
sgd-classifier | 0.9983 | 1.0000 | 0.0417 | 0.0800 | node2vec
very-simple-gcn | 0.9946 | 0.6000 | 0.5000 | 0.5455 | node2vec
first-gcn | 0.9935 | 0.5000 | 0.5000 | 0.5000 | node2vec
simple-gcn | 0.9935 | 0.5000 | 0.5000 | 0.5000 | node2vec
gcn-with-dropout | 0.9920 | 0.3500 | 0.2333 | 0.2800 | node2vec
advanced-gcn | 0.9898 | 0.2097 | 0.4333 | 0.2826 | node2vec
Table 3: Best instance for each model, with respect to recall. Model | Accuracy | Precision | Recall | F1 Score | Embedding
---|---|---|---|---|---
first-gcn | 0.9826 | 0.2727 | 1.0000 | 0.4286 | node2vec
sgd-classifier | 0.9924 | 0.4615 | 1.0000 | 0.6316 | node2vec
simple-gcn | 0.9783 | 0.2308 | 1.0000 | 0.3750 | node2vec
very-simple-gcn | 0.9815 | 0.2609 | 1.0000 | 0.4138 | node2vec
advanced-gcn | 0.8887 | 0.0533 | 0.9000 | 0.1006 | node2vec
gcn-with-dropout | 0.9613 | 0.0863 | 0.8000 | 0.1558 | node2vec
logistic-regression | 0.9912 | 0.3333 | 0.5000 | 0.4000 | node2vec
random-forest | 0.9984 | 1.0000 | 0.0833 | 0.1538 | chunk-start-bytes
Table 4: Best instance for each model, with respect to f1 score. Model | Accuracy | Precision | Recall | F1 Score | Embedding
---|---|---|---|---|---
sgd-classifier | 0.9924 | 0.4615 | 1.0000 | 0.6316 | node2vec
very-simple-gcn | 0.9946 | 0.6000 | 0.5000 | 0.5455 | node2vec
first-gcn | 0.9935 | 0.5000 | 0.5000 | 0.5000 | node2vec
simple-gcn | 0.9935 | 0.5000 | 0.5000 | 0.5000 | node2vec
logistic-regression | 0.9912 | 0.3333 | 0.5000 | 0.4000 | node2vec
gcn-with-dropout | 0.9858 | 0.2110 | 0.7667 | 0.3309 | node2vec
advanced-gcn | 0.9898 | 0.2097 | 0.4333 | 0.2826 | node2vec
random-forest | 0.9984 | 1.0000 | 0.0833 | 0.1538 | chunk-start-bytes
Table 5: Best model instance for each metric. Metric | Model | Accuracy | Precision | Recall | F1 Score | Embedding
---|---|---|---|---|---|---
accuracy | random-forest | 0.9984 | 1.0000 | 0.0833 | 0.1538 | chunk-start-bytes
precision | logistic-regression | 0.9944 | 1.0000 | 0.0417 | 0.0800 | node2vec
recall | first-gcn | 0.9826 | 0.2727 | 1.0000 | 0.4286 | node2vec
f1 score | sgd-classifier | 0.9924 | 0.4615 | 1.0000 | 0.6316 | node2vec

## Acronyms

AI

    Artificial Intelligence
DEL

    Directed Edge-labelled Graphs
DL

    Deep Learning
ESS

    Estimated Security Strength
FE

    Feature Evaluation
GCN

    Graph Convolutional Networks
GNN

    Graph Neural Network
GRU

    Gated Recurrent Units
KG

    Knowledge Graph
KNN

    K-Nearest Neighbors
LDA

    Linear Discriminant Analysis
LLM

    Large Language Model
LSB

    Least Significant Bit
ML

    Machine Learning
NLP

    Natural Language Processing
OWL

    Web Ontology Language
PCA

    Principal Component Analysis
RDF

    Resource Description Framework
RNN

    Recurrent Neural Networks
SMOTE

    Synthetic Minority Over-sampling Technique
SSH

    Secure Shell Protocol
SVM

    Support Vector Machine
t-SNE

    t-distributed Stochastic Neighbor Embedding
VMI

    Virtual Machine Introspection

## Glossary

CHN

    Chunk Header Node. This is a node whose bytes have been identified as a data structure header. In the graph, this node is the root node of an malloc-allocated memory chunk.
KN

    Key Node. This is a node whose bytes have been identified as a key. This identification relies both on the annotations and some verification checks.
memory graph

    A memory graph, or memgraph is a graph representation of a memory dump. This graph can be a graph of blocks, where each node in the graph corresponds to a block of 8 bytes in the heap dump and each edge corresponds to a pointer from one block to another, or describes which blocks are part of a chunk whose root note is a Chunk Header Node. It can also be a graph of chunks (only CHNs), where each node in the graph corresponds to a chunk in heap dump and each edge corresponds to a pointer from one object to another.
PN

    Pointer Node. This is a node whose bytes have been identified as a pointer.
VN

    Value Node. These are all blocks that have not been identified. It is the default node type.

## References

  * [1] Stewart Sentanoe and Hans P. Reiser  “SSHkex: Leveraging virtual machine introspection for extracting SSH keys and decrypting SSH network traffic”  In _Forensic Science International: Digital Investigation_ 40, 2022, pp. 301337  DOI: 10.1016/j.fsidi.2022.301337
  * [2] Clement Lahoche  “Structure embeddings for OpenSSH heap dump analysis” Available online, 2023  URL: https://github.com/passau-masterarbeit-2023/masterarbeit-report-clement
  * [3] Florian O. Rascoussier and C. Lahoche  “Machine Learning Assisted SSH Keys Extraction From The Heap Dump | Cleaned for Embedding”  Zenodo, 2022  DOI: 10.5281/zenodo.10514199
  * [4] Matthias Fey and Jan Eric Lenssen  “Fast Graph Representation Learning with PyTorch Geometric”, 2019  arXiv:1903.02428 [cs.LG]
  * [5] Emden R. Gansner, Eleftherios Koutsofios and Stephen North  “Drawing graphs with dot” Accessed: 2023-10-27, 2015  URL: https://graphviz.org/pdf/dotguide.pdf
  * [6] Wolfram Gloger and Doug Lea  “Malloc implementation for multiple threads without lock contention” Accessed: 2023-09-22, 2001  Free Software Foundation, Inc.  URL: https://elixir.bootlin.com/glibc/glibc-2.28/source/malloc/malloc.c
  * [7] DJ Delorie, Florian Weimer, Carlos O’Donell and Andreas Schwab  “Malloc Internals” Accessed: 2023-09-22, 2023  Sourceware  URL: https://sourceware.org/glibc/wiki/MallocInternals
  * [8] Unknown  “How does glibc malloc work?” Asked 6 years, 6 months ago; Modified 3 years, 6 months ago; Viewed 11k times; Accessed: 2023-09-25, 2023  URL: https://reverseengineering.stackexchange.com/questions/15033/how-does-glibc-malloc-work/15038#15038
  * [9] Joep Meindertma  “Ordered data in RDF: About Arrays, Lists, Collections, Sequences and Pagination” Accessed: 2023-09-22, 2020  URL: https://ontola.io/blog/ordered-data-in-rdf
  * [10] Christofer Fellicious, Stewart Sentanoe, Michael Granitzer and Hans P. Reiser  “SmartKex: Machine Learning Assisted SSH Keys Extraction From The Heap Dump” arXiv:2209.05243 [cs]  arXiv, 2022  DOI: 10.48550/arXiv.2209.05243
  * [11] M. Baushke  “Key Exchange (KEX) Method Updates and Recommendations for Secure Shell (SSH)” Updates: 4250, 4253, 4432, 4462; Errata exist, 2022
  * [12] Nicole Perlroth, Jeff Larson and Scott Shane  “N.S.A. Able to Foil Basic Safeguards of Privacy on Web”  In _The New York Times_ , 2013  URL: https://www.nytimes.com/2013/09/06/us/nsa-foils-much-internet-encryption.html
  * [13] James Ball, Julian Borger and Glenn Greenwald  “Revealed: how US and UK spy agencies defeat internet privacy and security” Published at 11.24 BST  In _The Guardian_ , 2013  URL: https://www.theguardian.com/world/2013/sep/05/nsa-gchq-encryption-codes-security
  * [14] Aris Adamantiadis  “OpenSSH introduces<EMAIL_ADDRESS>key exchange !” Retrieved 2023-09-05  libssh.org, 2013  URL: https://www.libssh.org/2013/11/03/openssh-introduces-curve25519-sha256libssh-org-key-exchange/
  * [15] OpenSSH  “OpenSSH 5.7 release notes” Retrieved 2022-11-13  OpenSSH, 2011  URL: https://www.openssh.com/txt/release-5.7
  * [16] Stewart Sentanoe, Benjamin Taubmann and Hans P. Reiser  “Sarracenia: Enhancing the Performance and Stealthiness of SSH Honeypots Using Virtual Machine Introspection” Conference paper  In _Lecture Notes in Computer Science_ 11252, 2018  URL: https://link.springer.com/chapter/10.1007/978-3-030-03638-6_16
  * [17] OpenSSH  “OpenSSH 6.5 release notes” Retrieved 2022-11-13  OpenSSH, 2014  URL: https://www.openssh.com/txt/release-6.5
  * [18] OpenSSH  “OpenSSH 7.0 release notes” Retrieved 2022-11-13  OpenSSH, 2015  URL: https://www.openssh.com/txt/release-7.0
  * [19] Christine Solnon  “Théorie des graphes et optimisation dans les graphes”  In _INSA de Lyon_
  * [20] Douglas Brent West  “Introduction to graph theory”  Prentice hall Upper Saddle River, 2001
  * [21] OpenSSH  “OpenSSH 7.2 release notes” Retrieved 2022-11-13  OpenSSH, 2016  URL: https://www.openssh.com/txt/release-7.2
  * [22] OpenSSH  “OpenSSH 8.2 release notes” Retrieved 2022-11-13  OpenSSH, 2020  URL: https://www.openssh.com/txt/release-8.2
  * [23] OpenSSH  “OpenSSH 8.8 release notes” Retrieved 2022-11-13  OpenSSH, 2021  URL: https://www.openssh.com/txt/release-8.2
  * [24] Samina Khalid, Tehmina Khalil and Shamila Nasreen  “A survey of feature selection and feature extraction techniques in machine learning”  In _2014 Science and Information Conference_ , 2014, pp. 372–378  DOI: 10.1109/SAI.2014.6918213
  * [25] Sinan Ozdemir and Divya Susarla  “Feature Engineering Made Easy: Identify unique features from your dataset in order to build powerful machine learning systems”  Packt Publishing Ltd, 2018
  * [26] C E Shannon  “A Mathematical Theory of Communication”  In _The Bell System Technical Journal_ 27, 1948, pp. 379–423
  * [27] F. Pedregosa et al.  “Scikit-learn: Machine Learning in Python” Software available at https://scikit-learn.org/stable/index.html  In _Journal of Machine Learning Research_ 12, 2011, pp. 2825–2830
  * [28] Richard Boddy and Gordon Smith  “Statistical methods in practice: for scientists and technologists”  John Wiley & Sons, 2009
  * [29] Michael I Jordan and Tom M Mitchell  “Machine learning: Trends, perspectives, and prospects”  In _Science_ 349.6245 American Association for the Advancement of Science, 2015, pp. 255–260  URL: https://www.science.org/doi/full/10.1126/science.aaa8415
  * [30] Todd G Nick and Kathleen M Campbell  “Logistic regression” Publisher: Springer  In _Topics in biostatistics_ , 2007, pp. 273–301
  * [31] S.. Kotsiantis  “Decision trees: a recent overview”  In _Artificial Intelligence Review_ 39.4, 2013, pp. 261–283  DOI: 10.1007/s10462-011-9272-4
  * [32] Philipp Probst, Marvin Wright and Anne-Laure Boulesteix  “Hyperparameters and Tuning Strategies for Random Forest”  In _WIREs Data Mining and Knowledge Discovery_ 9.3, 2019, pp. e1301  DOI: 10.1002/widm.1301
  * [33] Yann LeCun, Leon Bottou, Yoshua Bengio and Patrick Ha  “Gradient-Based Learning Applied to Document Recognition”  In _proc of the IEEE_ , 1998
  * [34] Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen and Dinh Phung  “A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network”  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ Association for Computational Linguistics, 2018  DOI: 10.18653/v1/n18-2053
  * [35] Daixin Wang, Peng Cui and Wenwu Zhu  “Structural Deep Network Embedding”  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16  San Francisco, California, USA: Association for Computing Machinery, 2016, pp. 1225–1234  DOI: 10.1145/2939672.2939753
  * [36] Michael Schlichtkrull et al.  “Modeling relational data with graph convolutional networks”  In _The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15_ , 2018, pp. 593–607  Springer  URL: https://arxiv.org/pdf/1703.06103.pdf
  * [37] Liang Yao, Chengsheng Mao and Yuan Luo  “KG-BERT: BERT for knowledge graph completion”  In _arXiv preprint arXiv:1909.03193_ , 2019  URL: https://arxiv.org/pdf/1909.03193.pdf
  * [38] Ralph C. Merkle  “Secure Communications over Insecure Channels”  In _Commun. ACM_ 21.4 New York, NY, USA: Association for Computing Machinery, 1978, pp. 294–299  DOI: 10.1145/359460.359473
  * [39] Ye Liu et al.  “Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 35.7, 2021, pp. 6418–6425  URL: file:///home/onyr/Downloads/16796-Article%20Text-20290-1-2-20210518.pdf
  * [40] Aditya Grover and Jure Leskovec  “node2vec: Scalable feature learning for networks”  In _Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2016, pp. 855–864  URL: https://dl.acm.org/doi/pdf/10.1145/2939672.2939754
  * [41] Jian Tang et al.  “Line: Large-scale information network embedding”  In _Proceedings of the 24th international conference on world wide web_ , 2015, pp. 1067–1077  URL: https://dl.acm.org/doi/pdf/10.1145/2736277.2741093
  * [42] Petar Velickovic et al.  “Graph attention networks”  In _stat_ 1050.20, 2017, pp. 10–48550  URL: https://personal.utdallas.edu/~fxc190007/courses/20S-7301/GAT-questions.pdf
  * [43] Keiron O’Shea and Ryan Nash  “An Introduction to Convolutional Neural Networks”  In _CoRR_ abs/1511.08458, 2015  arXiv: http://arxiv.org/abs/1511.08458
  * [44] Eelco Dolstra  “The purely functional software deployment model”  Utrecht University, 2006  URL: https://edolstra.github.io/pubs/phd-thesis.pdf
  * [45] OpenScience ASAP  “Was ist Open Science?” Accessed: 2023/09/19, 2023  URL: http://openscienceasap.org/open-science/
  * [46] DGE Gomes et al.  “Why don’t we share data and code? Perceived barriers and benefits to public archiving practices”  In _Proc Biol Sci_ 289.1987, 2022, pp. 20221113  DOI: 10.1098/rspb.2022.1113
  * [47] Eelco Dolstra and Andres Löh  “NixOS: A Purely Functional Linux Distribution”  In _SIGPLAN Not._ 43.9 New York, NY, USA: Association for Computing Machinery, 2008, pp. 367–378  DOI: 10.1145/1411203.1411255
  * [48] Jamie Ludwig  “Image convolution”  In _Portland State University_ , 2013  URL: https://web.pdx.edu/~jduh/courses/Archive/geog481w07/Students/Ludwig_ImageConvolution.pdf
  * [49] Vic Degraeve, Gilles Vandewiele, Femke Ongenae and Sofie Van Hoecke  “R-GCN: the R could stand for random”  In _arXiv:2203.02424 preprint_ , 2022  URL: https://arxiv.org/pdf/2203.02424.pdf
  * [50] Bryan Perozzi, Rami Al-Rfou and Steven Skiena  “Deepwalk: Online learning of social representations”  In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2014, pp. 701–710  URL: https://dl.acm.org/doi/pdf/10.1145/2623330.2623732
  * [51] Will Hamilton, Zhitao Ying and Jure Leskovec  “Inductive representation learning on large graphs”  In _Advances in neural information processing systems_ 30, 2017  URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/5dd9db5e033da9c6fb5ba83c7a7ebea9-Paper.pdf
  * [52] Zonghan Wu et al.  “A comprehensive survey on graph neural networks”  In _IEEE transactions on neural networks and learning systems_ 32.1 IEEE, 2020, pp. 4–24  URL: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9046288
  * [53] Zonghan Wu et al.  “Beyond low-pass filtering: Graph convolutional networks with automatic filtering”  In _IEEE Transactions on Knowledge and Data Engineering_ IEEE, 2022  URL: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9806316
  * [54] Federico Monti et al.  “Geometric deep learning on graphs and manifolds using mixture model cnns”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5115–5124  URL: https://openaccess.thecvf.com/content_cvpr_2017/papers/Monti_Geometric_Deep_Learning_CVPR_2017_paper.pdf
  * [55] Franco Scarselli et al.  “The graph neural network model”  In _IEEE transactions on neural networks_ 20.1 IEEE, 2008, pp. 61–80  URL: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4700287
  * [56] Joan Bruna, Wojciech Zaremba, Arthur Szlam and Yann LeCun  “Spectral networks and locally connected networks on graphs”  In _arXiv preprint arXiv:1312.6203_ , 2013  URL: https://arxiv.org/pdf/1312.6203.pdf
  * [57] Quoc V Le  “A tutorial on deep learning part 2: Autoencoders, convolutional neural networks and recurrent neural networks”  In _Google Brain_ 20, 2015, pp. 1–20  URL: https://ai.stanford.edu/~quocle/tutorial2.pdf
  * [58] Thomas N Kipf and Max Welling  “Semi-supervised classification with graph convolutional networks”  In _arXiv preprint arXiv:1609.02907_ , 2016  URL: https://arxiv.org/pdf/1609.02907.pdf
  * [59] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho and Yoshua Bengio  “Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling”  arXiv, 2014  arXiv: http://arxiv.org/abs/1412.3555
  * [60] J. Laaksonen and E. Oja  “Classification with learning k-nearest neighbors”  In _Proceedings of International Conference on Neural Networks (ICNN’96)_ 3 Washington, DC, USA: IEEE, 1996, pp. 1480–1483  DOI: 10.1109/ICNN.1996.549118
  * [61] Sepp Hochreiter and Jürgen Schmidhuber  “Long short-term memory” Publisher: MIT Press  In _Neural computation_ 9.8, 1997, pp. 1735–1780
  * [62] Ian Goodfellow, Yoshua Bengio and Aaron Courville  “Deep learning”  MIT press, 2016  URL: https://books.google.de/books?hl=en&lr=&id=omivDQAAQBAJ&oi=fnd&pg=PR5&dq=deep+learning&ots=MNV2eosBRS&sig=jN2QwFikq3g_YqU3hJVPEP0XIJ4&redir_esc=y#v=onepage&q=deep%20learning&f=false
  * [63] Qiang Wu and Ding-Xuan Zhou  “Analysis of Support Vector Machine Classification”  In _Journal of Computational Analysis & Applications_ 8.2, 2006
  * [64] Jianlong Zhou, Amir H. Gandomi, Fang Chen and Andreas Holzinger  “Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics”  In _Electronics_ 10.5, 2021, pp. 593  DOI: 10.3390/electronics10050593
  * [65] Jose Manuel Gomez-Perez, Ronald Denaux and Andres Garcia-Silva  “Understanding Word Embeddings and Language Models”  In _A Practical Guide to Hybrid Natural Language Processing: Combining Neural Models and Knowledge Graphs for NLP_ Cham: Springer International Publishing, 2020, pp. 17–31  DOI: 10.1007/978-3-030-44830-1_3
  * [66] OpenSSH  “OpenSSH 9.0 release notes” Retrieved 2022-11-13  OpenSSH, 2022  URL: https://www.openssh.com/txt/release-9.0
  * [67] Unknown  “How does SSH use both RSA and Diffie-Hellman?” Asked 8 years, 9 months ago; Modified 8 years, 9 months ago; Viewed 22k times; Accessed: 2023-09-21, 2023  URL: https://security.stackexchange.com/questions/76894/how-does-ssh-use-both-rsa-and-diffie-hellman
  * [68] T. Ylonen and C. Lonvick  “The Secure Shell (SSH) Protocol Architecture” Updated by: 8308, 9141  SSH Communications Security Corp; Cisco Systems, Inc., 2006
  * [69] T. Ylonen and C. Lonvick  “The Secure Shell (SSH) Transport Layer Protocol” Updated by: 6668, 8268, 8308, 8332, 8709, 8758, 9142; Errata Exist  SSH Communications Security Corp; Cisco Systems, Inc., 2006
  * [70] Girish Venkatachalam  “The OpenSSH Protocol under the Hood”  In _Linux Journal_ , 2007  URL: https://www.ecb.torontomu.ca/~courses/coe518/LinuxJournal/elj2007-156-OpenSSH.pdf
  * [71] Oscar M. Guillen et al.  “Towards post-quantum security for IoT endpoints with NTRU”  In _Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017_, 2017, pp. 698–703  DOI: 10.23919/DATE.2017.7927079
  * [72] Paulo Nunes Souza and Pavel Gladyshev  “Inference of Endianness and Wordsize From Memory Dumps”  In _European Conference on Cyber Warfare and Security_ , 2017, pp. 619–627  Academic Conferences International Limited
  * [73] P. McLaren, G. Russell, W.J. Buchanan and Z. Tan  “Decrypting live SSH traffic in virtual environments”  In _Digital Investigation_ 29, 2019, pp. 109–117  URL: https://www.sciencedirect.com/science/article/abs/pii/S1742287619300647
  * [74] Tatu Ylonen  “Portable OpenSSH” Github repositor. Accessed on 25.08.2023, 1995  URL: https://github.com/openssh/openssh-portable/
  * [75] Auguste Kerckhoffs  “La cryptographic militaire”  In _Journal des sciences militaires_ , 1883, pp. 5–38
  * [76] José Tomás Martínez Garre, Manuel Gil Pérez and Antonio Ruiz-Martínez  “A novel Machine Learning-based approach for the detection of SSH botnet infection”  In _Future Generation Computer Systems_ 115, 2021, pp. 387–396  DOI: 10.1016/j.future.2020.09.004
  * [77] SSH Communications Security  “SSH Annual Report 2018” Accessed: 2023-08-30, 2018  URL: https://info.ssh.com/hubfs/2021%20Investor%20documents/SSH_Annual_Report_2018_final.pdf
  * [78] W. Yurcik and Chao Liu  “A first step toward detecting SSH identity theft in HPC cluster environments: discriminating masqueraders based on command behavior”  In _CCGrid 2005. IEEE International Symposium on Cluster Computing and the Grid, 2005._ 1, 2005, pp. 111–120 Vol. 1  DOI: 10.1109/CCGRID.2005.1558542
  * [79] “Weak CRC allows packet injection into SSH sessions encrypted with block ciphers” Accessed: 2023-08-30, 2001  URL: https://www.kb.cert.org
  * [80] Core Security Technologies  “SSH Insertion Attack” Archived from the original on 2011-07-08, https://www.coresecurity.com/core-labs/advisories/ssh-insertion-attack, 2023
  * [81] US CERT  “SSH CBC vulnerability” Archived from the original on 2011-06-22, https://www.kb.cert.org/vuls/id/958563, 2011
  * [82] Spiegel Online  “Prying Eyes: Inside the NSA’s War on Internet Security” Archived from the original on January 24, 2015, Spiegel Online, 2014  URL: https://www.spiegel.de/international/germany/inside-the-nsa-s-war-on-internet-security-a-1010361.html
  * [83] Olivier Bilodeau et al.  “Operation WINDIGO”, 2014, pp. 69  URL: https://web-assets.esetstatic.com/wls/2014/03/operation_windigo.pdf
  * [84] Pooneh Nikkhah Bahrami* et al.  “Cyber Kill Chain-Based Taxonomy of Advanced Persistent Threat Actors: Analogy of Tactics, Techniques, and Procedures”  In _Journal of Information Processing Systems_ 15.4, 2019, pp. 865–889  DOI: 10.3745/JIPS.03.0126
  * [85] Sanjay Madan and Monika Singh  “Classification of IOT-Malware using Machine Learning”  In _2021 International Conference on Technological Advancements and Innovations (ICTAI)_ , 2021, pp. 599–605  DOI: 10.1109/ICTAI53825.2021.9673185
  * [86] Connor Hetzler, Zachary Chen and Tahir M. Khan  “Analysis of SSH Honeypot Effectiveness”  In _Advances in Information and Communication_ , Lecture Notes in Networks and Systems  Cham: Springer Nature Switzerland, 2023, pp. 759–782  DOI: 10.1007/978-3-031-28073-3_51
  * [87] ppacher  “honeyssh” GitHub repository, https://github.com/ppacher/honeyssh, 2017
  * [88] Dimitrios Georgoulias, Jens Myrup Pedersen, Morten Falch and Emmanouil Vasilomanolakis  “Botnet Business Models, Takedown Attempts, and the Darkweb Market: A Survey”  In _ACM Computing Surveys_ 55.11, 2023, pp. 1–39  DOI: 10.1145/3575808
  * [89] Aidan Hogan et al.  “Knowledge Graphs”  In _ACM Comput. Surv._ 54.4 New York, NY, USA: Association for Computing Machinery, 2021  DOI: 10.1145/3447772
  * [90] Aidan Hogan et al.  “Knowledge Graphs (Extended)” arXiv:2003.02320 [cs]  In _ACM Computing Surveys_ 54.4, 2022, pp. 1–37  DOI: 10.1145/3447772
  * [91] Paul Groth, Elena Simperl, Marieke Erp and Denny Vrandečić  “Knowledge Graphs and their Role in the Knowledge Engineering of the 21st Century” Report from Dagstuhl Seminar 22372. Specific usage: pp. 60-72, Subsection "3.2 A Brief History of Knowledge Engineering: A Practitioner’s Perspective"  In _Dagstuhl Reports_ 12.9 Dagstuhl Seminar, 2022, pp. 60–120  DOI: 10.4230/DagRep.12.9.60
  * [92] Marvin Hofer et al.  “Construction of Knowledge Graphs: State and Challenges”  In _arXiv preprint arXiv:2302.11509_ , 2023  URL: https://doi.org/10.48550/arXiv.2302.11509
  * [93] Lisa Ehrlinger and Wolfram Wöß  “Towards a Definition of Knowledge Graphs”  In _SEMANTiCS (Posters, Demos, SuCCESS)_ , 2016, pp. 1–4
  * [94] Frederick Edward Hulme  “Proverb Lore: Many Sayings, Wise Or Otherwise, on Many Subjects, Gleaned from Many Sources”  E. Stock, 1902, pp. 188
  * [95] Google  “Introducing the Knowledge Graph: Things, not strings” Accessed: 2023-06-16  In _Google Blog_ , 2012  URL: https://blog.google/products/search/introducing-knowledge-graph-things-not/
  * [96] Michelle Venables  “An Introduction to Graph Theory” Accessed: 2023-06-12  Towards Data Science, 2019  URL: https://towardsdatascience.com/an-introduction-to-graph-theory-24b41746fabe
  * [97] Gianluca Fiorelli  “Best of 2013: No 13 - Search in the Knowledge Graph era” Accessed: 2023-06-12  State of Digital, 2013  URL: https://www.stateofdigital.com/search-in-the-knowledge-graph-era/
  * [98] Jackson Gilkey  “Graph Theory and Data Science” Accessed: 2023-05-25  Towards Data Science, 2019  URL: https://towardsdatascience.com/graph-theory-and-data-science-ec95fe2f31d8
  * [99] M.S. Jawad, Chitra Dhawale, Azizul Azhar Bin Ramli and Hairulnizam Mahdin  “Adoption of knowledge-graph best development practices for scalable and optimized manufacturing processes”  In _MethodsX_ 10, 2023, pp. 102124  DOI: https://doi.org/10.1016/j.mex.2023.102124

### Eidesstattliche Erklärung

Hiermit versichere ich, dass ich diese Masterarbreit selbstständig und ohne
Benutzung anderer als der angegebenen Quellen und Hilfsmittel angefertigt habe
und alle Ausführungen, die wörtlich oder sinngemäß übernommen wurden, als
solche gekennzeichnet sind, sowie, dass ich die Masterarbreit in gleicher oder
ähnlicher Form noch keiner anderen Prüfungsbehörde vorgelegt habe.

Passau, August 28, 2024

Rascoussier, Florian Guillaume Pierre