# BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer

Lucas Georges Gabriel Charpentier,* Sondre Wold,*
David Samuel and Egil Rønningstad
University of Oslo, Language Technology Group
<EMAIL_ADDRESS>

###### Abstract

Retrieval-based language models are increasingly employed in question-
answering tasks. These models search in a corpus of documents for relevant
information instead of having all factual knowledge stored in its parameters,
thereby enhancing efficiency, transparency, and adaptability. We develop the
first Norwegian retrieval-based model by adapting the REALM framework and
evaluate it on various tasks. After training, we also separate the language
model, which we call the reader, from the retriever components, and show that
this can be fine-tuned on a range of downstream tasks. Results show that
retrieval augmented language modeling improves the reader’s performance on
extractive question-answering, suggesting that this type of training improves
language models’ general ability to use context and that this does not happen
at the expense of other abilities such as part-of-speech tagging, dependency
parsing, named entity recognition, and lemmatization. Code, trained models,
and data are made publicly available.111https://github.com/ltgoslo/brent

**footnotetext: The authors contributed equally to this work

##  1 Introduction

Figure 1: The proposed architecture, based on the REALM method from Guu et al.
(2020).

Retrieval-based language models meet some important shortcomings associated
with pre-trained language models (PLMs): they are more dynamic, allowing for
updating of knowledge without having to re-train the model from scratch; they
are more transparent, allowing backtracking the source of returned statements;
and they are more efficient, as retrieval provides a non-parametric memory.
The accentuated benefit of these models has been the OpenQA task – where they
have established new state-of-the-art results on datasets like
NaturalQuestions (Kwiatkowski et al., 2019) and WebQuestions Berant et al.
(2013). There, models first fetch a relevant passage from a data source in
order to be able to answer a question — as compared to extractive QA, where a
passage with the correct answer is provided explicitly as additional input to
the model, also referred to as machine reading comprehension.

In this work, we develop the first Norwegian retrieval-based model, BRENT:
Bidirectional Retrieval Enhanced Norwegian Transformer, based on the general
approach proposed by Guu et al. (2020). Our model consists of two encoders
that respectively learn to embed documents and queries into dense vector
representations, and a reader module that learns to utilize the retrieved
context for prediction, as shown in Figure 1. These are trained jointly and
end-to-end, and we start their training from an already pre-trained Norwegian
LM. Compared to previous work, we use a relatively small retrieval corpus
consisting of 730k Wikipedia documents.

The learning objective is masked language modeling (MLM), and the top $k$ most
relevant documents are retrieved from the retrieval corpus through a maximum
inner product search (MIPS).

The size of our retrieval corpus allows us to update the search index
synchronously during training and do exact matching, as opposed to the
asynchronous updates and approximations done in Guu et al. (2020).
Furthermore, we do not consider OpenQA as an evaluation task, but instead, we
study how retrieval-augmented language modeling can be used as a continued
pre-training step in order to improve context utilization in the reader model.
That is, we evaluate the reader as a stand-alone — extracting it from the
overall pipeline so that it can be distributed and used as a normal LM.

In order to analyze the effect of this continued pre-training, we also
benchmark the reader against other NLP tasks that by intuition should not
benefit from this type of training, such as part-of-speech-tagging, named
entity recognition, dependency parsing, and lemmatization. We find that the
retrieval-augmented training procedure increases the reader’s performance on
extractive QA without decreasing performance on these tasks. However, we find
that it decreases performance on both targeted and sentence-level sentiment
analysis. To summarize, our contributions are:

  * •

We develop and release the first Norwegian retrieval-based language model.

  * •

We study how retrieval improves the reader’s ability to use context on
extractive QA while still performing on par with comparable baselines on
morpho-syntactic tasks.

  * •

We analyze the different components of a retrieval-based system through a
series of ablations, addressing problems associated with common design
choices.

##  2 Related work

The basic setup for most retrieval-based approaches to NLP is that for a query
$q$, be it a question for QA or a premise in natural language inference, the
model must retrieve a set of passages relevant to $q$. Relevant candidates are
then typically appended to $q$ before being passed to a classification layer.

While earlier work approached this using heuristics and sparse retrieval
methods like BM25 (Robertson et al., 2009), recent work has focused on
learning this retrieval step. Most of these use an architecture with an
encoder and a reader: the encoder learns to represent $q$ and the retrieved
passages in a representation space that makes it possible to match documents
using the inner product operation, while the reader learns how to utilize the
retrieved passage for downstream prediction. Recent work by Jiang et al.
(2022) shows how it is also possible to model this interaction using a single
transformer model (Vaswani et al., 2017) with a retrieval as attention
technique, as compared to having separate encoders and readers.

Lee et al. (2019) note that it is computationally impractical to learn to make
predictions conditioned on a retrieval corpus from scratch and thus proposed
to pre-train the encoders with an inverse cloze task (ICT) in order to “prime”
the model for retrieval. This is also done in Sachan et al. (2021). We outline
more details on this and how we use ICT in the following section.

The most direct application of retrieval is to use a supervision signal such
as OpenQA to train the context and passage encoders, such as in Khattab et al.
(2021). However, Guu et al. (2020) show how this setup can also be used for
language modeling. Using English Wikipedia as the retrieval corpus, they
perform MLM conditioned on retrieved passages. Passages are retrieved using
MIPS over an index that is asynchronously updated during training. They also
use a masking technique that prioritizes named entities in order to
incentivize the usage of world knowledge from the retrieved passages. A
similar approach to language modeling is also done in Borgeaud et al. (2022),
but over a corpus consisting of trillions of tokens. For both works, the LMs
are trained for a number of steps with retrieval before being fine-tuned on a
downstream task, as is the typical workflow with PLMs.

Lewis et al. (2020b) demonstrate how the encoder-reader architecture can be
used for language generation as well. They propose both a sequence model where
the generation is conditioned on the same set of retrieved documents for the
entire sequence and a token model where a different document is used per
target token. The retriever is based on Dense Passage Retrieval (DPR)
(Karpukhin et al., 2020), which uses the same general approach to retrieval as
Guu et al. (2020), where a PLM like BERT (Devlin et al., 2019) is used as the
encoder. The reader model is swapped with a generator based on BART (Lewis et
al., 2020a).

##  3 Method

As in Guu et al. (2020), the architecture of BRENT can be separated into two
parts: a retriever and a reader. Our architecture is modified to improve the
speed of training, to ensure that the retrieved documents affect the
predictions, and to incentivize the retrieval of world knowledge from the
retrieval corpus instead of the reader memorizing it. This section puts forth
the architecture and these modifications.

###  3.1 Architecture

##### Retriever

The first part of BRENT is the retriever, which consists of two components:
the Query Encoder ($\mathrm{Enc_{query}}$) and the Document Encoder
($\mathrm{Enc_{doc}}$). Both have their own sets of weights and in our case
have a BERT-style architecture and tokenizer. However, these can be
initialized with other types of dense representation learners and could
potentially also share weights for faster training.

The retriever receives as input the query, $q$, which is the masked sentence
from the pre-training corpus, and passes it to $\mathrm{Enc_{query}}$ to get a
dense representation. $\mathrm{Enc_{doc}}$ encodes all the documents in the
retrieval corpus. Once all the documents and the query are encoded, a
similarity score is calculated between each document $d$ and the query $q$:

$\mathrm{sim}(q,d)=\frac{\mathrm{Enc_{query}}(q)^{T}\mathrm{Enc_{doc}}(d)}{\sqrt{h_{\text{dim}}}},$

where $h_{\textit{dim}}$ represents the encoding dimension of the retriever
encoders. Since the query and doc vectors are not normalized, the inner
product can be very large. In order to stabilize the training, we scale the
inner product by dividing it by the square root of the hidden dimension.

Once all the similarity scores are calculated, we use softmax to create a
probability distribution over all the documents for a given query:

$p(d|q)=\frac{\exp(\mathrm{sim}(q,d))}{\sum_{d^{\prime}\in
D}\exp(\mathrm{sim}(q,d^{\prime}))}.$

Finally, we create the inputs to the reader by appending the representations
of each $d$ to $q$, i.e. $[q;d]$. In other words, if we have a retrieval
corpus $D$ with $N$ documents, then a single query generates $N$ inputs to the
reader — effectively multiplying by $N$ the batch size passed to the reader.
However, it is unfeasible to do this for the whole corpus, therefore we only
retrieve the top-k documents based on the similarity scores.

##### Reader

The reader is a single pre-trained language model taking as input the document
$d$ appended to query $q$ ($[q;d]$). During continued pre-training, the reader
is optimized for MLM. For each input to the model, we get predictions on what
the masked words in the query are — given the context provided by document
$d$. Formally, each input generates the following probability for the correct
masked words $y$:

$p(y|d,q)=\prod_{y_{i}\in M_{q}}p(y_{i}|d,q),$

where $y_{i}$ is the $i$-th masked word in query $q$ and $M_{q}$ is the set of
all masked words in $q$. However, we want $p(y|q)$. Therefore, for a query
$q$, we need to do $k$ forward passes to get all the $p(y|d,q)$. Finally, to
obtain $p(y|q)$ we marginalize:

$p(y|q)=\sum_{d\in D}p(y|d,q)\,p(d|q).$

##### Loss

With $p(y|q)$ we can calculate the loss. During the loss function (cross-
entropy) derivation, the error backpropagation is spread between the reader
and the retriever. For the reader, this is the same as for any transformer-
based model trained on the MLM task except that it averages over the batch
size, number of retrieved documents, and the number of masked tokens. For the
retriever, it is updated based on whether $p(y|d,q)$ was better or worse than
$p(y|q)$. Specifically, if $p(y|d,q)$ is higher than $p(y|q)$, then the
similarity score between $q$ and $d$ should increase. This can be seen with
the following equation:

$\displaystyle\nabla_{\theta}\log p(y|q)$ $\displaystyle=\sum_{d\in
D}u(d)p(d|q)\nabla_{\theta}\mathrm{sim}(d,q)$ $\displaystyle u(d)$
$\displaystyle=\biggl{(}\frac{p(y|d,q)}{p(y|q)}-1\biggl{)},$

where $\theta$ represents the parameters of $\mathrm{Enc_{doc}}$.222The full
derivation can be found in the appendix of Guu et al. (2020), where $z=d$,
$x=q$ and $f$ represents the function $\mathrm{sim}$ The same derivation
applies to the parameters of $\mathrm{Enc_{query}}$.

###  3.2 Null Document

As pointed out in Guu et al. (2020), not all masked words need world knowledge
to be predicted correctly. Therefore, we also add a null document appended to
the query $q$. There are two ways to encode the null document. The first, and
most obvious, is to pass the empty string to $\mathrm{Enc_{doc}}$ and use the
resulting encoding as the null document. However, we use a parameter tensor
initialized with all zeros instead. This saves us one forward and backward
pass of $\mathrm{Enc_{doc}}$ per step, without affecting performance.

###  3.3 Corpus

We use a snapshot of the Norwegian Wikipedia from October 2022 as our corpus,
limited to the Bokmål written standard. We pre-process the articles into
chunks of token length $128$, padding the last chunk of each article so that
no chunk contains text from two different sources. After processing, the
corpus consists of $735\,000$ documents, with an average number of words per
chunk being about $100$ ($\mu=102,\sigma=25$). We use this corpus for sampling
sentences to mask for MLM and as a retrieval corpus during continued pre-
training.

###  3.4 Search index

As described in our architecture, we use the documents $d$ in the retrieval
corpus $D$ to improve the model’s predictions. Ideally, we would use all the
documents of the retrieval corpus to make the prediction. Then, the model
would assign close to zero probabilities to most documents, while
simultaneously having access to all documents, and therefore identifying the
most relevant ones. However, this is not feasible, as it would require a very
high amount of resources (which would keep increasing as we increase our
retrieval corpus) and be unreasonably time-consuming. Therefore, we instead
only retrieve the top-k documents in terms of similarity score. To be able to
efficiently retrieve and find these documents, we use a search index. We build
this index using the encoding of the documents produced by
$\mathrm{Enc_{doc}}$. Since we update $\mathrm{Enc_{doc}}$ at every backward
pass, it follows that we should re-index the documents at each backward pass.
However, this is too time-consuming and we therefore only re-index each $s$
steps. We want to note here that there has been recent work on how to more
efficiently retrieve from such an index (Alon et al., 2022; He et al., 2021).
Since we use the same corpus for both MLM training and retrieval, the first
retrieved document is often the same document from which the query comes from,
as this will naturally have a high similarity score. To avoid directly giving
our model the answer with the unmasked token in it, we make sure to remove
this document.

###  3.5 Inverse cloze task

We warm up the encoders for both the query and the document with the ICT task
from Lee et al. (2019) on $68k$ Wikipedia article introductions limited to 128
tokens, from a snapshot from October 2020. For each pass, the model must
predict the relevant pseudo-document for a pseudo-question from a set of
distractors. The question is a random sentence and the document is the text
surrounding it, the distractors are sampled from the same batch.

###  3.6 Span Masking

For the MLM task, we combine both salient masking (Guu et al., 2020), where
only named entities and dates that require world knowledge are masked, and
random masking. We identify entities using an off-the-shelf named entity
recognizer and dates with a simple parsing algorithm.333spaCy:
https://spacy.io/ We use 15% salient masking, making sure to mask at least one
salient span for each sample, and 3.75% random span masking (which is 25% of
15%). By doing this, we encourage the network to learn to retrieve spans
requiring world knowledge while ensuring that the model is still able to model
linguistic features.

##  4 Experiments

We evaluate BRENT on a wide range of Norwegian NLP tasks. We do this both
without retrieval using the extracted reader, and with retrieval turned on
using the full model. By doing this, we highlight both the improved capacity
of the reader to use context and show how retrieval in general affects
performance on NLP tasks other than QA. This section describes the specific
datasets and models we use during experimentation.

###  4.1 Models

##### NorBERT2

A baseline Norwegian LM, originating from Kutuzov et al. (2021).

##### NorBERT250k

A NorBERT2 model trained for 50k additional steps on Wikipedia using MLM as
described in Section 3.6, with a batch size of 1024. We show the performance
of this model in order to get a more fair comparison, showcasing the
improvements gained from the actual retrieval-augmented pre-training as
compared to just doing regular pre-training for 50k more steps on the same
corpora.

##### BRENT

The entire model with retrieval turned on during fine-tuning. This is akin to
a Norwegian version of REALM Guu et al. (2020), but with our modifications.
When subscripted, this indicates the source of the retrieval corpus, which
could be either from Wikipedia or a task-specific dataset.

##### BRENTreader

The reader model extracted after continued pre-training, used without any
retrieval during fine-tuning on the downstream tasks.

Model | Wiki | News | All
---|---|---|---
EM | F1 | EM | F1 | EM | F1
Human* | 72.65 | 88.84 | 83.61 | 93.43 | 78.13 | 91.14
NorBERT2 | 57.76±1.15 | 71.89±0.89 | 64.05±1.27 | 76.93±1.15 | 64.64±1.40 | 77.86±0.65
NorBERT250k | 59.14±0.55 | 73.98±1.05 | 64.89±1.44 | 77.22±0.57 | 63.88±0.49 | 77.05±0.55
BRENTreader | 62.57±1.77 | 76.45±1.40 | 68.10±2.87 | 80.40±1.71 | 66.56±1.36 | 80.01±1.16
Table 1: Results on different domains of the NorQuAD dataset. Results are reported as the mean and standard deviation over five random seeds. *Human performance is the mean performance of two annotators as reported in Ivanova et al. (2023)) Model | UPOS | UFeats | Lemma | LAS | NER
---|---|---|---|---|---
NorBERT2 | 98.65±0.04 | 97.58±0.06 | 98.18±0.03 | 93.15±0.05 | 88.13±0.34
NorBERT250k | 98.64±0.04 | 97.54±0.04 | 98.12±0.06 | 93.10±0.22 | 88.41±0.45
BRENTreader | 98.62±0.06 | 97.55±0.02 | 98.09±0.04 | 92.96±0.15 | 87.70±0.49
Table 2: Results on the morpho-syntactic tasks: accuracy of UPOS and UFeats
tagging, the accuracy of lemmatization, the labeled attachment scores of
dependency parsing, F1 scores of named entity recognition, where the
evaluation requires an exact match on both span and label. Results are
reported as the mean and standard deviation over five random seeds.

###  4.2 Datasets

##### NorQuAD

A Norwegian question answering dataset for machine reading comprehension
(Ivanova et al., 2023) based on the SQuAD format (Rajpurkar et al., 2016). For
a given question, the model must predict the correct span in a provided
passage that answers the question. NorQuAD includes three domain splits: one
sourced from the Norwegian Wikipedia ($N=2351$), one from Norwegian news
articles ($N=2398$), and one split that combines both of them ($N=4749$). We
use an $80-10-10$ split on all three domains for training, validation, and
testing.

##### NoReC${}_{\textit{fine}}$

A fine-grained sentiment analysis dataset for Norwegian (Øvrelid et al.,
2020). The texts are a subset of the NoReC dataset (Velldal et al., 2018), a
multi-domain dataset of full-text professional reviews published in Norwegian
online news sources. Each sentence in NoReC${}_{\textit{fine}}$ is annotated
for sentiment holders, targets, polar expressions, expression polarities, and
polar intensities. A version for targeted sentiment analysis (TSA) is released
on GitHub where only the sentiment targets are
labeled.444https://github.com/ltgoslo/norec_tsa

##### NoReC${}_{\textit{sent}}$

A sentence-level sentiment analysis dataset for Norwegian derived from
NoReC${}_{\textit{fine}}$ (Øvrelid et al., 2020; Kutuzov et al., 2021). This
dataset is generated by aggregating the entity sentiments in each sentence.
The sentences are then labeled as either positive, negative, or neutral. We
use the version only containing positive and negative sentiments. Both
versions of the dataset (with and without neutral sentiment sentences) are
available on GitHub.555https://github.com/ltgoslo/norec_sentence/

##### Morpho-syntactic tasks

This group of tasks is based on annotations from the Norwegian Dependency
Treebank (Solberg et al., 2014), which were converted to the Universal
Dependencies (UD) format by Øvrelid and Hohle (2016) and later enriched with
named-entity types by Jørgensen et al. (2020). The resulting dataset is called
NorNE and we use its latest version.666https://github.com/ltgoslo/norne The
source of NorNE is mostly news texts, but also government reports, parliament
transcripts, and blogs. We evaluate the models on all available UD tasks for
Norwegian Bokmål (UPOS and UFeats tagging, lemmatization, and dependency
parsing; Nivre et al., 2016),777We use the official evaluation script from
CoNLL 2018 shared task (Zeman et al., 2018,
https://universaldependencies.org/conll18/evaluation.html). as well as on
named entity recognition (NER).888We employ the evaluation method from SemEval
2013 task 9.1 (Segura-Bedmar et al., 2013), re-implementated in
https://github.com/davidsbatista/NER-Evaluation.

###  4.3 Implementation details

Since running these models is resource intensive, we do not do a
hyperparameter search. Instead, we base our hyperparameters on previous
research where available. The following paragraphs outline the details of our
experiments.

##### Search Index

We use the FlatIndexIP from the FAISS (Johnson et al., 2019) library to
construct our index. This allows us to get the most relevant documents rather
than an approximation of the best documents. We can do this since our corpus
of documents is relatively small. We retrieve the top-7 documents and append
the null document, in essence retrieving 8 documents in total. We re-index
every 100 steps.

##### ICT

We use NorBERT2 as the initialization for the ICT warmup. For this, we use a
learning rate of $1*10^{-4}$ and batch size of $128$ for 10 epochs with early
stopping on a single NVIDIA A100 GPU. After the warmup, these weights are then
used as the starting point for $\mathrm{Enc_{query}}$ and $\mathrm{Enc_{doc}}$
in the retriever, while the reader uses NorBERT2 without any warmup.

##### Pre-training

We then train BRENT for 50k steps with a batch size of $1024$ divided over 128
AMD MI250X GPUs,999These resources were made available to us through the
EuroHPC JU project: https://www.lumi-supercomputer.eu/ a learning rate of
$2*10^{-5}$, using the AdamW optimizer, and a Cosine scheduler with a warmup,
on the chunked Wikipedia corpus. A full description of the model and the
hyperparameters can be found in Section A.2.

##### Fine-tuning

We run all experiments using five different seeds and report the average
result and standard deviation. For the fine-tuning of the retrieval-enhanced
models, we test both with and without re-indexing, i.e., fine-tuning
$\mathrm{Enc_{doc}}$. In both cases, we continue to fine-tune
$\mathrm{Enc_{query}}$. When fine-tuning, we use a higher learning rate for
the retriever as compared to the reader, since we saw experimentally that this
obtained better results. When re-indexing, we do it every 100 steps and at the
end of each epoch. Hyperparameters for all evaluation tasks can be found in
Section A.3. We fine-tune all models on a single GPU.

###  4.4 Results

Model | BSA F1 % | TSA F1 %
---|---|---
NorBERT2 | 85.52±0.74 | 47.58±0.49
NorBERT250k | 84.62±0.50 | 46.70±0.65
BRENTreader | 83.33±0.47 | 46.48±0.26
BRENTWiki | 84.21±0.37 | 44.06±0.73
BRENTWiki;nri | 84.18±0.53 | 43.38±1.45
BRENTNoReC | 84.35±0.41 | 43.55±0.26
BRENTNoReC;nri | 83.90±0.56 | 44.22±0.42
Table 3: Results of the binary sentiment analysis task (BSA) on the
NoReC${}_{\textit{sent}}$ dataset and targeted sentiment analysis (TSA) on the
NoReC${}_{\textit{fine}}$ dataset. Evaluation is on the test set and is based
on the best model found during training. Results are reported as the mean and
standard deviation over five random seeds. $\mathrm{nri}$ stands for no re-
indexing. The $\mathrm{NoReC}$ subscript represents the training dataset being
used as the retrieval corpus.

####  4.4.1 Extractive QA

Table 1 shows the exact match (EM) and token-level F1 scores of different
approaches on the NorQuAD dataset. BRENTreader outperforms all other
approaches on the three domain splits, especially with respect to the EM
metric, which we explain by the salient masking technique. Although
BRENTreader was only trained on Wikipedia, the improvement in performance is
significant also for questions in the news category. Naturally, NorBERT250k
also improves a bit compared to NorBERT2 on the Wikipedia split, but not by
the same margin, and not at all on the news category. This indicates that
BRENTreader actually learns to use context better, that this generalizes
beyond the style of Wikipedia, and that this could not be achieved by simply
training the same underlying LM for 50k additional steps on the same corpus
with the same MLM setup.

####  4.4.2 Sentiment analysis

Figure 2: Training perplexity during the first 10 000 steps. The values are
smoothed with an exponential moving average, using $\alpha=0.99$.

As for sentiment analysis, Table 3 shows that BRENTreader performs worse
compared to the baseline of NorBERT2 on the binary sequence classification
task, indicating that the continued pre-training with retrieval does not
actually help for this task, but rather impedes performance. This is also the
case for the NorBERT250k model, albeit with a smaller impediment to
performance, suggesting that it might be the continued training on Wikipedia
reducing the performance of the models on this task. When retrieval is used,
as can be seen in the bottom half of Table 3, the performance is better, but
still short of the baseline. For TSA, the reader performs on par with the
baselines but turning retrieval on substantially decreases performance.

When retrieving from a corpus during fine-tuning, our model retrieves reviews
that are related with respect to inner product similarity, not necessarily
sentiment. If classifying a negative review of a TV, our model could end up
retrieving another review about some other electronic apparatus — which might
be positive. This is clearly not helpful for the task at hand. In order to
teach the retrievers to retrieve based on sentiment, we would need a bigger
dataset to fine-tune on. Despite this, manual inspection shows that the
retrieved contexts are sometimes very relevant for the query when the
retrieval corpus is NoReC. When the retrieval corpus is Wikipedia, however,
the contexts are of low relevance. Examples of queries and retrieved contexts
for BRENTWiki and BRENTNoReC on binary sentiment analysis (BSA) can be found
in Section A.1.1 and Section A.1.2.

We also note that not re-computing the search index decreases performance.
However, as performance is relatively similar, it might not be worth it as re-
indexing is a lot more resource-demanding. With Wikipedia as the retrieval
corpus on our computing setup, TSA fine-tuning takes about $7$ hours with re-
indexing, compared to $2.5$ hours without.

####  4.4.3 Morpho-syntactic

Table 2 shows the results of the reader compared to the baselines on a series
of Norwegian token-level labeling tasks. BRENTreader performs on par with the
baseline models, which strengthens our hypothesis that the continued pre-
training with retrieval does not impede the model’s ability to perform morpho-
syntactic tasks while simultaneously increasing performance on extractive QA.
This claim is further supported by the fact that the same happens with
NorBERT250k, which indicates that adding the retrieval is no worse than just
continuing to do MLM over the same corpus for additional steps.

###  4.5 Analysis of the pre-training

Figure 2 shows the perplexity values of BRENT and NorBERT250k during the first
10k steps of continued pre-training on the Wikipedia corpus. After the initial
convergence phase, NorBERT250k establishes itself on values around $40$, while
BRENT sits at around $20$. As we do mainly salient masking, perplexity is a
proxy for how well the models predict the correct named entities and dates.
The difference between the two shows how retrieval is helpful for predicting
masked entities.

##  5 Ablations

As with other retrieval-augmented LMs, BRENT is a pipeline model — consisting
of multiple parts that interact according to a series of design choices that
impact the outcome. Due to the computational cost of pre-training, it is not
feasible to quantitatively determine the effect of all these choices,
resulting in a poor understanding of some aspects of these models. To mitigate
this, we study the effect of some of these choices with respect to the overall
loss during pre-training with a series of ablations. We do this for a reduced
number of steps, but with the same retrieval corpus and with the same GPU
setup as described in Section 3.3 and Section 4.3.

###  5.1 ICT

Figure 3 shows the effect of the ICT warmup task with respect to the loss for
$2000$ steps. When ICT is turned off, $\mathrm{Enc_{doc}}$ and
$\mathrm{Enc_{query}}$ are initialized with the same weights as the reader. As
can be seen from the figure, the loss converges slower when the ICT task is
not used, but it is quickly matching the setting when it is used. Guu et al.
(2020) claims that without ICT one would encounter a cold-start problem where
the retrieved documents will likely be unrelated to the query at the beginning
of training, causing a cycle where the encoders do not receive meaningful
gradients. We find that this is not the case and that the effect of ICT warmup
is minimal.

###  5.2 The effect of the null document

As mentioned in Section 3.2, we use a parameter tensor initialized with all
zeros for representing the null document. This is optimized jointly with the
rest of the weights. Figure 3 shows how the model behaves when the null
document is removed, which is done by making the probability of the null
document zero, as compared to the test model which has it included. Contrary
to Guu et al. (2020), we find that the effect of the null document is
questionable. It makes sense to have a ”sink” to use when no retrieval is
necessary, but we do not find the null document to fulfill this need.

Figure 3: Loss curves of having no null document or no ICT warmup compared to
the tested model. The values of all three runs are smoothed with an
exponential moving average, using $\alpha=0.99$.

###  5.3 Varying the number of documents to retrieve

Figure 4 shows how the number of retrieved documents influences training with
respect to loss. For the first 2 000 training steps, $k=16$ converges a bit
quicker than the $k=8$. However, we see that the result is minimal after that
point, which is also the conclusion in Guu et al. (2020). Given that it is
more computationally expensive to train with a higher $k$ and that the gain of
going from $8$ to $16$ is negligible, we keep $k$ at $8$.

Figure 4: Loss curves of the first 2 000 training steps with varying number of
retrieved documents $k$. The values are smoothed with an exponential moving
average, using $\alpha=0.99$.

##  6 Conclusion

We develop the first Norwegian retrieval augmented language model, BRENT,
based on the REALM method proposed by Guu et al. (2020). The model uses an
encoder-reader architecture, and we train it on a relatively small corpus
consisting of 735k Wikipedia documents. In addition to the model itself, our
contribution has been to demonstrate how the use of continued pre-training
with retrieval benefits the context utilization of the reader, which we
extract from the pipeline. The reader performs better than comparable
baselines on the extractive QA task without losing performance on morpho-
syntactic tasks. We also evaluate our full retriever model on sentiment
analysis with two different corpora as the retrieval corpus, but here we
observe a decrease in performance overall. Contrary to some previous work, our
ablation studies find that the effect of having a null document and using ICT
as a warmup task is minimal.

##  7 Future work

A future direction for our work is to study in greater detail how retrieval
influences the language modeling task. In particular, we would like to train a
retrieval model from scratch. Another direction, which has also been pointed
out in related work, is to experiment with cross-lingual retrieval, especially
in the case where the retrieval corpus is from a high-resource language. This
would be useful in scenarios where a large knowledge source like English
Wikipedia could be used to augment a lower resource language, like Norwegian,
which does not have such an extensive source available.

## Acknowledgements

Parts of the work documented in this publication have been carried out within
the NorwAI Centre for Research-based Innovation, funded by the Research
Council of Norway (RCN), with grant number 309834.

## References

  * Alon et al. (2022) Uri Alon, Frank Xu, Junxian He, Sudipta Sengupta, Dan Roth, and Graham Neubig. 2022\.  Neuro-symbolic language modeling with automaton-augmented retrieval.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 468–485. PMLR.
  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on Freebase from question-answer pairs.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1533–1544, Seattle, Washington, USA. Association for Computational Linguistics.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022.  Improving language models by retrieving from trillions of tokens.  In _International conference on machine learning_ , pages 2206–2240. PMLR.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 3929–3938. PMLR.
  * He et al. (2021) Junxian He, Graham Neubig, and Taylor Berg-Kirkpatrick. 2021.  Efficient nearest neighbor language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5703–5714, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ivanova et al. (2023) Sardana Ivanova, Fredrik Aas Andreassen, Matias Jentoft, Sondre Wold, and Lilja Øvrelid. 2023.  NorQuAD: Norwegian Question Answering Dataset.  In _Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)_ , Torshavn, Faroe Islands.
  * Jiang et al. (2022) Zhengbao Jiang, Luyu Gao, Jun Araki, Haibo Ding, Zhiruo Wang, Jamie Callan, and Graham Neubig. 2022.  Retrieval as attention: End-to-end learning of retrieval and reading within a single transformer.  _arXiv preprint arXiv:2212.02027_.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547.
  * Jørgensen et al. (2020) Fredrik Jørgensen, Tobias Aasmoe, Anne-Stine Ruud Husevåg, Lilja Øvrelid, and Erik Velldal. 2020.  NorNE: Annotating Named Entities for Norwegian.  In _Proceedings of the 12th Edition of the Language Resources and Evaluation Conference_ , Marseille, France, 2020.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Khattab et al. (2021) Omar Khattab, Christopher Potts, and Matei Zaharia. 2021.  Relevance-guided supervision for OpenQA with ColBERT.  _Transactions of the Association for Computational Linguistics_ , 9:929–944.
  * Kutuzov et al. (2021) Andrey Kutuzov, Jeremy Barnes, Erik Velldal, Lilja Øvrelid, and Stephan Oepen. 2021.  Large-scale contextualised language modelling for Norwegian.  In _Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pages 30–40, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
  * Lewis et al. (2020a) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lewis et al. (2020b) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 9459–9474. Curran Associates, Inc.
  * Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016.  Universal Dependencies v1: A multilingual treebank collection.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 1659–1666, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Øvrelid and Hohle (2016) Lilja Øvrelid and Petter Hohle. 2016.  Universal Dependencies for Norwegian.  In _Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC’16)_ , pages 1579–1585, Portorož, Slovenia.
  * Øvrelid et al. (2020) Lilja Øvrelid, Petter Mæhlum, Jeremy Barnes, and Erik Velldal. 2020.  A fine-grained sentiment dataset for Norwegian.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 5025–5033, Marseille, France. European Language Resources Association.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends® in Information Retrieval_ , 3(4):333–389.
  * Sachan et al. (2021) Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton, and Bryan Catanzaro. 2021.  End-to-end training of neural retrievers for open-domain question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6648–6662, Online. Association for Computational Linguistics.
  * Segura-Bedmar et al. (2013) Isabel Segura-Bedmar, Paloma Martínez, and María Herrero-Zazo. 2013.  SemEval-2013 task 9 : Extraction of drug-drug interactions from biomedical texts (DDIExtraction 2013).  In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_ , pages 341–350, Atlanta, Georgia, USA. Association for Computational Linguistics.
  * Solberg et al. (2014) Per Erik Solberg, Arne Skjærholt, Lilja Øvrelid, Kristin Hagen, and Janne Bondi Johannessen. 2014.  The Norwegian Dependency Treebank.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation_ , Reykjavik, Iceland.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in Neural Information Processing Systems_ , 30.
  * Velldal et al. (2018) Erik Velldal, Lilja Øvrelid, Eivind Alexander Bergem, Cathrine Stadsnes, Samia Touileb, and Fredrik Jørgensen. 2018.  NoReC: The Norwegian review corpus.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Zeman et al. (2018) Daniel Zeman, Jan Hajič, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018.  CoNLL 2018 shared task: Multilingual parsing from raw text to Universal Dependencies.  In _Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_ , pages 1–21, Brussels, Belgium. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 BSA retrieval examples

####  A.1.1 Wikipedia examples

Examples of retrieved contexts from the BRENTWiki model fine-tuned on the BSA
task with Wikipedia as the retrieval corpus.

  * •

Query: Men så kommer de skjærende lydene ‘But then the squeaky sounds appear’

    * –

Retrieved context: Øynene ‘The eyes’.

  * •

Query: Broen går seg vill i sitt eget ønske om å vært artsy ‘”The Bridge” is
lost in its own wish to be ”artsy”’

    * –

Retreived context: Sverige ‘Sweden’

####  A.1.2 NoReC examples

Examples of retrieved contexts from the BRENTNoReC model fine-tuned on the BSA
task with the $NoReC$ dataset as the retrieval corpus.

  * •

Query: Men så kommer de skjærende lydene ‘But then the squeaky sounds appear’

    * –

Retrieved context: Allerede under første låt får vi slengt alle klisjeene i
trynet. ‘Already during the first song we are hit in the face with all the
clichés’.

  * •

Query: Begeistringen var uvanlig stor og applausen deretter da det 70
minutters lange verket var fullført ‘The enthusiasm was unusually great and so
was the applause that followed when the 70-minute long piece was over’.

    * –

Retreived context: En helt utrolig konsertopplevelse ‘A wonderful concert
experience’.

  * •

Query: Å være eksperimentell er ikke positivt i seg selv; de mange sjangrene
og retningene i musikken gjør helehetsinntrykket rotete og meningsløst ‘Being
experimental is not positive in and of itself; the many genres and directions
makes the music seem messy and meaningless’.

    * –

Retrieved context: Automatisk to-soners klimaanlegg ‘Automatic two-zone
aircondition’.

###  A.2 Model

The hyperparameters used for the continued pre-training can be found in Table
4.

Hyperparameter | Value
---|---
Number of parameters | 125M
Number of attention heads | 12
Number of layers | 12
Hidden dimension ($h_{\textit{dim}}$) | 768
Activation function | GeLU
Vocabulary size | 50104
Seed | 42
Dropout | 0.1
lr | $2*10^{-5}$
Weight decay | 0.1
Batch size | 1024
$k$ | 8
re-indexing frequency | 100 steps
Steps | 50k
Scheduler | Cosine with warmup
Warmup | 800 steps
Final lr | $2*10^{-6}$
Optimizer | AdamW
Index | FlatIndexIP
Table 4: Hyperparameters for the continued pre-training of both BRENT and
NorBERT2. $k$ represents the number of documents retrieved including the null
document.

###  A.3 Hyperparameters

####  A.3.1 NorQuAD

For comparison, we use the same set of hyperparameters as in Ivanova et al.
(2023), visible in Table 5.

Hyperparameter | Value
---|---
Batch size | 16
Epochs | 3
lr | $5*10^{-5}$
Scheduler | Linear
Optimizer | AdamW
Seeds | $[42,437,4088,3092,9720]$
Table 5: Hyperparemeters for fine-tuning on the NorQuAD dataset

####  A.3.2 Sequence labeling

For the task of targeted sentiment analysis, we fine-tune and report the
average test results over five runs, from the epoch providing the best results
on the development set. Hyperparameters can be found in Table 6.

Hyperparameter | Value
---|---
Batch size | 32
Epochs | 8
lrreader | $5*10^{-5}$
lrretriever | $1.5*10^{-4}$
$k$ | 4
re-indexing frequency | 100 steps + end of epoch
Scheduler | Linear
Optimizer | AdamW
Seeds | $[101,202,303,404,505]$
Table 6: Hyperparemeters for fine-tuning on the NoREC${}_{\textit{fine}}$
dataset. $k$ represents the number of documents retrieved including the null
document. The lrreader is for both the retrieval and non-retrieval models.

###  A.4 Binary Sentiment Analysis

For the task of binary sentiment analysis, we fine-tune for three epochs and
select the best model based on the development set’s f1 score. We average our
test results over five runs. Hyperparameters can be found in table Table 7.

Hyperparameter | Value
---|---
Batch size | 32
Epochs | 3
lrreader | $1*10^{-5}$
lrretriever | $3*10^{-5}$
$k$ | 4
re-indexing frequency | 100 steps + end of epoch
Scheduler | Cosine
Optimizer | AdamW
Seeds | $[42,456,78463,27485,34586]$
Table 7: Hyperparemeters for fine-tuning on the NoREC${}_{\textit{sent}}$
dataset. $k$ represents the number of documents retrieved including the null
document. The lrreader is for both the retrieval and non-retrieval models.

###  A.5 Morpho-syntactic

For morpho-syntactic tasks, we fine-tune for 10 epochs and select the best
model based on the average performance on the development split. We average
our test results over five runs. Hyperparameters can be found in Table 8.

Hyperparameter | Value
---|---
Batch size | 32
Epochs | 10
LRreader | $1*10^{-4}$
LRheads | $1*10^{-3}$
Scheduler | Cosine
Optimizer | AdamW
Seeds | $[1234,2345,3456,4567,5678]$
Table 8: Hyperparemeters for fine-tuning on the morpho-syntactic tasks. $k$
represents the number of documents retrieved including the null document. The
learning rate is different for the fine-tuned language model and for the
classification heads.


# Mining Function Homology of Bot Loaders from Honeypot Logs

Yuhui Zhu, Zhenxiang Chen, Qiben Yan, Shanshan Wang, Enlong Li, Lizhi Peng,
Chuan Zhao

###### Abstract

Self-contained loaders are widely adopted in botnets for injecting loading
commands and spawning new bots. While researchers can dissect bot clients to
get various information of botnets, the cloud-based and self-contained design
of loaders effectively hinders researchers from understanding the loaders’
evolution and variation using classic methods. The decoupled nature of bot
loaders also dramatically reduces the feasibility of investigating
relationships among clients and infrastructures. In this paper, we propose a
text-based method to investigate and analyze details of bot loaders using
honeypots. We leverage high interaction honeypots to collect request logs and
define eight families of bot loaders based on the result of agglomerative
clustering. At the function level, we push our study further to explore their
homological relationship based on similarity analysis of request logs using
sequence aligning techniques. This further exploration discloses that the
released code of Mirai keeps spawning new generations of botnets both on the
client and the server side. This paper uncovers the homology of active botnet
infrastructures, providing a new prospect on finding covert relationships
among cybercrimes. Bot loaders are precisely investigated at the function
level to yield a new insight for researchers to identify the botnet’s
infrastructures and track their evolution over time.

###### Index Terms:

IoT botnet, loader, honeypot, classification, homology

##  I Introduction

Accompanying the growth of the IoT market, botnets targeting IoT devices have
become a major threat to cyber security. The IoT botnet is a worm-like malware
system targeting monopolizing IoT devices to execute malicious actions.
Nowadays, IoT Botnets have become the most popular solution for attackers
conducting DDoS attacks[1], exploiting other hosts[2] and even
cryptomining[3]. Mirai[4, 5], the most famous descendant of classic botnet
Bashlite, was recorded launching a 1.1Tbps DDoS attack using 148,000 IoT
devices, which broke the record and makes it the top botnet in the following
years. This large family has been constantly active since its inception. Anna-
senpai’s code release[6] in September 2016 further stimulated botmasters to
create variants based on its idea. New botnets like Hajime[7] are also under
active development to equip themselves with new technologies.

Security industries leverage honeypots[8] to collect comprehensive information
on botnet threats, including malware samples, interaction logs, and attackers’
IP addresses, then publish Indicators of Compromise (IoC) as their product or
service. To better understand botnets, some studies[9, 10, 11, 12] introduced
malware lineage inference into this field for a systematic insight into the
relationships of botnets. However, due to the decoupled design of popular IoT
botnets, honeypots cannot collect every component of botnets, especially
executables of bot loaders. Most botnets use a self-contained loader server to
drop bot executables during the propagating period, which executable can never
be captured by honeypots. The limited dataset forces systematic studies
focusing on dissecting bot executables to investigate the relationship among
botnets. Bot loaders are less investigated, which dramatically reduces the
feasibility of inferring the infrastructures’ evolution and transfers
conducted by botmasters.

Despite it is impractical to dissect bot loaders’ executables, some studies
[13, 14, 15, 4, 16, 17] made elementary comparative analyses on specified
properties in telnet interaction, including initialize command lists,
credential dictionaries, and query tokens. These studies reached a preliminary
conclusion that the relationships among bot loaders can be inferred from their
shared features, but did not make any comparison on their intrusion behaviors.
Though the homology of bot loaders’ functions is critical for inferencing
their covert relationships, the lack of methodology hinders researchers from
understanding bot loaders and their codebases on a systematic aspect.

The main purpose of this paper is to reveal the homology of active botnets’
infrastructures, precisely speaking, investigating bot loaders on the function
level to find their undercovered homology relationship. To achieve this goal,
we apply NLP-based methods to classify interaction logs captured by telnet
honeypots and extract stable strings from interaction logs to represent the
behavior feature of every bot loader family. We provide a new homology view on
bot loaders, helping researchers obtain deeper knowledge about IoT botnets.
The main contributions of our work are summarized as follows:

  * •

Our work provides a server-side perspective for cyber forensics to investigate
cybercrimes beside bots. We try to describe loaders’ functions using request
logs captured by the telnet honeypot, which is an alternative method to
understand the inner details of bot loaders.

  * •

We propose a semantic-aware yet an elastic method for classifying bot loaders
based on interaction logs from honeypots. We define 8 families out of 19
clusters and describe their behavior similarity based on the result of
agglomerative clustering and several empirical criteria. We also extracted
representative strings for each cluster to show their functional characters,
which provides an explainable result for our family definition.

  * •

We made an in-depth investigation of loaders’ behaviors and their
relationships. To examine the component homologies of loaders, we analyze
extracted representative strings associated with the result of agglomerative
clustering. The common skeleton and shared components across different
families are well described based on these results. This prospect can provide
solid evidence for investigating underground industries to uncover their
connections.

The rest of the paper is organized as follows: In Section II, we introduce the
basics of IoT botnets and the status of botnet studies. We also discuss the
limitations of current studies on botnet infrastructures. In Section III, we
introduce our honeypot system and propose our classification method. Based on
self-supervised learning, we classify request logs and provide a primary
classification of active loaders. In Section IV, we evaluate the result in
Section III to introduce our definition of loader families. We describe their
functions and depict their homology based on their common behaviors. Finally,
we discuss the limitation of our work in Section V and conclude our work in
Section VI.

##  II Related Works

###  II-A Basics on IoT Botnets

On the dark side, since the industrial community introduced “Internet of
Things”, cybercrimes had targeted them and developed various botnet software
to exploit vulnerabilities on these platforms and devices.

In the Anna-senpai’s release[6], the botnet was designed as a distributed
system that consists of 3 major components, _bot_ , _loader server_ , and _C
&C server_. The overall architecture is dipicted in Fig. 1. In the attack
phase (①), the C&C server sends attack commands to bots like traditional
botnets. But in the propagate phase, bots only scan for vulnerable devices but
never load bot executables (②). They merely report collected IP addresses and
credentials to the loader server. The self-contained loader server executes
loading scripts to launch bot malware on victim devices (③). The decoupled
loader and distributed architecture simplified its design and robustified its
infrastructure to survive taking-downs.

Figure 1: Common botnet architecture.

###  II-B Investigating and Tracing Botnets

On the light side, the dynamics of botnets are traced closely to investigate
their background information. Firstly, researchers collect everything about
how botnets work and propagate, which is the fundamental identity of malicious
activities. Secondly, they make a comprehensive inspection of threats to
investigate the criminal on the dark side. Lastly, they catch up with the
dynamic of these botnets to keep a long-term track of these botnets.

Figure 2: Schematic depiction of the analysis steps in our work.

Honeypot is one of the most commonly used tools for collecting threat
information, especially botnets’ dynamics. As defined by Spitzner[18],
Honeypot is “an information system resource whose value lies in the
unauthorized or illicit use of that resource which acts like a vulnerable
device to interact with malicious adversaries”. Honeypots record intrusion
conversations from attackers, including the IP address and network
interactions. Sometimes it also stores loaded executables for further
analyses. Pa et al.[19] first introduced honeypot into the IoT domain to
capture botnet malwares.

To monitor botnets and observe their evolution, researchers continuously
gather information and track dynamics to follow up on botnets’ entire
lifecycle. Emerging botnets and infected devices are continuously measured to
summarize the dynamics of threats. The first research about Mirai made by
Antonakakis et al.[4] tried to observe Mirai’s spread and evolution from many
aspects. This work utilized a network telescope, scanner, honeypot, DNS probe,
and C&C milker to collect information, then analyzed DDoS records provided by
CDN providers. In this paper, the authors recorded the growth and evolution
ever since Mirai first appears, then summarized details about infections,
ownerships, and attacks. By relating C&C servers, C&C domain names, and
credential dictionaries, this work examined several individual botnets and
observed the expansion of their variants. The Hajime botnet was analyzed in
detail by [7]. This work actively tracks bot nodes of Hajime on the DHT
network. This work also investigated injection requests based on the DNS
backscatter dataset. Griffioen’s work[16] took a wider view of all descendants
of Mirai. This work tracked how these botnets infect, retain control, and
hijack devices from other botnets using virtual high-interaction honeypots,
then demonstrated the mutation and evolution of Mirai in these years. Wang et
al. [20] applied an NLP-based method to correlate the information published by
researchers and constructed a lineage graph for 72 IoT malware families to
reflect their evolutionary relationships.

To keep up with the evolving network environment, botnets also evolved to
enhance themselves in exploitation by integrating new technologies. Tracking
emerging exploitations is a fundamental issue for tracking botnets’ evolution
and helping IDS systems protect devices. Pa’s work[19] made an elementary
analysis based on credential dictionaries, intrusion commands, and hash values
of executable samples. Dang et al.[2] focused on file-less attacks conducted
by IoT botnets and classified common types using a hybrid honeypot system.
Safaei et al.[21] proposed a method aiming at classifying and inferring
compromised IoT devices by solely observing one-way network traffic. This
helps uncover, report, and thoroughly analyze “in the wild” IoT botnets on the
Internet scale. Abbas et al.[22] made a comparative analysis of Mirai and
Bashlite variants by analyzing collected attack vectors and malware
executables.

###  II-C Comparative Study on Bot Loaders

The released source code inspired botmasters to engage new techniques in the
existing version. This also boosted the diversity of bot loaders. Comparing
behavior features of loaders become a significant topic for understanding the
evolution of botnets. Lingenfelter et al.[15] analyzed interaction logs from
medium interaction honeypots, made a comparison of initial commands and query
tokens to demonstrate the variation among IoT botnets. Torabi et al.[13] tried
mining unique strings from logs to build associations among active botnets.
Tabari et al.[14] made a statical analysis of the most commonly exploited
vulnerabilities, credentials, and intrusion commands.

Although these works introduced a comparison aspect on active bot loaders,
their conclusions are still less solid to reach a systematic conclusion on how
the loaders variated and derived predecessors’ codebase. Some studies[4, 16]
adopted password dictionaries to track botnets’ variation and analyze their
lineage, but these works didn’t focus on the implementation of intrusion
commands, resulting in insufficient evidence for homology analysis of loaders.

###  II-D Summary

The existing research mostly focused on revealing botnets’ behavior in
specified fields. Lineage analyses are commonly applied on executables of bot
clients, but infrastructures, especially loaders, are less investigated due to
their self-contained design and cloud-side deployment. Analyzing these
fundamental components of botnets needs better mining and representing methods
for monitored exploits. This hinders researchers from extracting their
implementation details, and also makes it difficult to analyze their evolution
and homology.

##  III Methodology

In this section, we propose a novel methodology for investigating the
relationships of bot loaders by analyzing their homology. Based on the
agglomerated clusters of the dataset, we extract the command-based features of
loader families and try to figure out the lineage among different loaders.

###  III-A Dataset Collecting

Figure 3: Honeypot system architecture.

Weak credentials on telnet servers are the most common vulnerability among IoT
devices, thus telnet-based intrusion is widely adopted as the primary attack
vector by most botnets. Analyzing and comparing the behavior of telnet loaders
can be the most efficient way to make comparative and systematic studies on
bot loaders. In this study, we implemented a simple high interaction honeypot
system to capture telnet interactions on the transport layer. High interaction
honeypot (HIH)[23, 24, 25] handles a bunch of actual devices or services to
show vulnerabilities and interact with attackers, which is opposite to _low
interaction honeypot_ whose decoy is totally virtual. Like many other
honeypots in past studies, our honeypot consists of two major parts, the
frontend, and the backend. Frontends are deployed on cloud servers to interact
with scanners and loaders directly, while the backend is deployed on a central
server to dispatch requests to backing devices on a local network bridge. The
honeypot captures all interaction messages on the transport layer and
generates conversation logs for the following analysis. The overall
architecture is depicted in Fig. 3.

The bridge module is a transparent proxy working on TCP protocol between the
attacker and backing IoT devices. It simply forwards bytes in telnet
conversations to backing devices listed in Table I, but substitutes
credentials to allow logging into the device. The dispatcher will
automatically pick a device for every telnet conversation.

TABLE I: Deployed backing devices during the experiment Type | Device name | Software version
---|---|---
Smart router | Lenovo Y1S | PandoraBox git-6fcbaa5
Netgear R7800 | OpenWRT 21.02
Netgear R6300v2 | KoolShare Merlin
IP Camera | Hikvision | (Stock)
ONU | CMCC I-120EM | (Stock)
Other | Raspberry Pi 3B | Raspberry Pi OS Lite Jan 2021

In the final dataset, we expect a control group that all the samples generated
by loaders or scanners from certain families, to help us evaluate the
effectiveness and accuracy of our work. Here we run a Hajime bot and a Mirai
loader in a QEMU ARM sandbox and redirect requests on TCP/23 to a local
honeypot. The Hajime bot sample is provided by
MalwareBazaar111https://bazaar.abuse.ch/. These conversation logs collected
from the local sandbox are distinctly marked as the control group in the
database.

Telnet is a text-based protocol commonly used for accessing a remote shell.
Telnet loaders leverage a set of command-line toolkits to start an automatic
intrusion and set up a botnet client on victim devices. Botmasters can take a
codebase off-the-shelf (e.g. Mirai’s loader) or implement new ones by
themselves. Also, they can modify the code to improve an existing loader or
take some modules from others to enhance their own toolkit. All these
differences and relationships in their codebases can be reflected by the
behavior fingerprints of loaders. In the following sections, we leverage
several common NLP methods to analyze the collected dataset to investigate
these characters.

###  III-B Request Filtering and Concatenating

Telnet, and its underlying TCP protocol, transfer messages in duplex mode.
Conversation logs from the honeypot consist of both request and response
messages sorted by the time sequence. However, in this work, we only focus on
investigating loaders, so all the response messages in conversation logs are
dropped in this step. We also filtered out messages for credential prompt and
protocol setup. All remained requests are concatenated into a single “request
log” to represent all the behavior of a loader in the conversation, which is
depicted in Fig. 4.

###  III-C Tokenize and N-gram vectorize

In this step, we split request logs into tokens and generate n-gram feature
vectors. The process is depicted in Fig. 4.

To tokenize messages from text-based protocols, PRISMA[26] uses a set of
characters as delimiters. However, in telnet messages, delimiters vary in
different contexts. Diverse delimiters also reflect the syntax of a message,
which could not simply drop. The tokenize algorithm should preserve all bytes
to retain features of message contents. To satisfy the above needs, we
tokenize request logs based on the type of bytes. According to our early
experiments, plain text is the major form of messages in a shell-based telnet
conversation, while symbols and spaces compose semantic structures of shell
commands. Unprintable bytes are also valid payloads for telnet messages, which
could be used for controlling the terminal and transferring files. Based on
this observation, we distinguish bytes into 3 types: alphanumeric, symbolic
(including punctuations and spaces), and unprintable bytes. Our method scans
the type of every byte in a request log. When finding two contiguous bytes of
different types, our method cut the request log at this position. Our method
iterate on every byte of a request log to get the token sequence. This keeps a
token consisting of only one type of bytes and reserves all features from both
printable and unprintable parts.

Our vectorize method uses a joint vector of BoW, 2-gram, and 3-gram to
represent the semantic context of every token. In shell command lines, the
context of a token implies semantic information. The bag-of-words (BoW) model
only describes the counts of tokens, but the feature vector should reflect the
previous and next tokens of them, which could be better described by the
n-gram method. The N-gram method slides an n-sized window on the sample from
the beginning to the ending, generates an n-gram token, and finally generates
the feature vector based on the count of these contiguous subsequences.
Therefore we represent the combinations of nearby tokens by 2-gram tokens.
Botmasters often make minor changes to intrusion toolkits. In common
situations, these variable tokens are surrounded by stable tokens constructing
a context skeleton. Thus we use 3-gram vectors to represent these minor
modifications.

Figure 4: Byte-based tokenization and n-gram vectorization of request logs.

###  III-D Agglomerative Clustering

To help define loader families based on their behaviors and further
investigate their functions, we agglomerate request logs into clusters based
on their similarities to partition the dataset.

Due to the deficiency of labeled request logs, unsupervised learning,
especially clustering, is the optimal choice for this task. However, most of
the clustering methods rely on pre-defined amount of clusters to initialize
themselves, which is ambiguous indeed for the task of family definition.
Moreover, our method should retain the pairwise similarity of clusters for
further homological analysis. In this case, we leverage the agglomerative
clustering algorithm[27] to generate an elastic cluster definition on the
dataset.

Agglomerative clustering is a bottom-top approach to building hierarchy
clusterings. Each observation, which is the feature vector of a request log in
our method, starts in its own cluster. A pairs of clusters $\\{C_{x},C_{y}\\}$
are merged as one in every iteration, which is selected by finding the minumun
inter-cluster distance $\mathcal{D}\left(C_{x},C_{y}\right)$ based on two
hyperparameters: linkage criterion $crit$ and distance metric $d$.

$\begin{split}\mathcal{D}\left(C_{x},C_{y}\right)=crit\left(d\left(x,y\right)\mid
x\in C_{x},y\in C_{y}\right)\end{split}$ (1)

These two hyperparameters can dramatically impact the shape of clusters. Our
method should select appropriate hyperparameters to evaluate the similarity of
two request logs or clusters.

In agglomerative clustering, the distance metric $d$ calculates the distance
between two observations. The most common metrics are the Euclidean distance
and the cosine distance. In this task, the Euclidean distance is the optimal
choice for calculating the dissimilarity of request logs, which robustifies
our method. According to our previous experiments, the Euclidean distance
performs better than the cosine distance in classifying request logs when
there are shared components.

When two clusters to be evaluated consist of multiple observations, the
linkage criterion determines the inter-cluster distance based on pairwise
distances of observations. While the average, complete and minimum linkage are
common ones, we choose the ward as the linkage criterion, which minimizes the
variance of the clusters being merged.

Finally, all the request logs can be hierarchically organized into an
agglomerative binary tree, on which leaves correspond to request logs, and
trunk nodes are agglomerative clusters.

The agglomerative clusters on the tree can be denoted as $C\in\mathbb{N}$,
where $\mathbb{N}$ denotes the full set of them. Every cluster
$C\in\mathbb{N}$ can be further splitted into two sub-clusters
$\\{C_{1},C_{2}\\}$ or merged into a super-cluster $C^{S}$. Cutting the tree
at a given height $\mathcal{T}$ by evaluating the condition $F\left(C\right)$
on every $C\in\mathbb{N}$ will produce a preliminary partitioning
$\mathbb{P}\subset\mathbb{N}$ at a selected precision:

$\begin{split}\mathbb{P}=&\\{C\ |\ C\in\mathbb{N}\wedge F\left(C\right)\\}\\\
F\left(C\right)=&\left[\mathcal{D}_{sub}\left(C^{S}\right)\geqslant\mathcal{T}\right]\wedge\left[\mathcal{D}_{sub}\left(C\right)<\mathcal{T}\right]\end{split}$
(2)

Note that we define the intra-cluster distance
$\mathcal{D}_{sub}\left(C\right)$ as the inter-cluster distance between its
sub-clusters $\mathcal{D}\left(C_{1},C_{2}\right)$, which is the height of
corresponding node on the agglomerative tree. We leverage the implementation
from Scikit-learn[28] for the experiment.

###  III-E Template Extraction

To identify the shared codebase of cousin loaders out of their request logs,
we should make a comparative analysis to generate “log templates” for
agglomerative clusters.

The agglomerative tree reveals the pairwise similarities of request logs and
clusters. Similar ones are coherent on the tree as the heterogeneous ones are
far from each other. So it is feasible to find a representative subsequence
for every cluster based on the request logs by scanning the agglomerative tree
from the bottom to the top. We apply the Smith-Waterman algorithm[29] on every
cluster to generate “templates” for them.

In our method, we use the tokenized sequence in the section III-C to align two
request logs. As depicted in Fig. 5, the $align(A_{C1},A_{C2})$ function
leverages the Smith-Waterman algorithm to scan both two tokens sequences from
head to tail. It performs sequence alignment on a local scope to determine
similar regions between two strings. Identical tokens at the same position
will be aligned in the right position. Our method adds placeholders (shadow
cells in Fig. 5) to replace different tokens and help identical ones align,
which finally generates a template. For any cluster $C\in\mathbb{N}$, the
corresponding template $A_{C}$ is generated recursively based on the templates
of its sub-clusters $A_{C1}$ and $A_{C2}$. Finally, every node on the
agglomerative tree will get a representative “template” which could be easily
interpreted.

Figure 5: Smith-Waterman algorithm on the agglomerative tree.

###  III-F Clustering Refinement

However, although we empirically cut the agglomerative tree to roughly
classify the request logs, this is far from being taken as the final class
definition. The unique $\mathcal{T}$ value may not fit all branches on the
tree, so we have to interpret the templates of clusters in $\mathbb{P}$ and
adjust the partitioning to calibrate the family definition. Here we
empirically set some criteria for accepting or denying a merged cluster:

  * •

Commands is the major evidence for the judgment, their arguments and calling
order should also be taken into consideration.

  * •

When there are complex statements in command lines, the structure and syntax
are more important than its component commands.

  * •

Tokens used for manifesting botmasters should be ignored unless it appears
with new commands or different arguments.

We start from nodes in $\mathbb{P}$ and check corresponding templates to
decide if a cluster should be kept, merged, or further split. Finally, for
every host we observed, we collect all related samples and their class labels,
then pick the label with the maximum count as the label of this loader host.

##  IV Evaluation

In this section, we discuss functions and behaviors of active loaders based on
the aforementioned methods and make a conclusion about their homologies.

Figure 6: Geographical distribution of loader hosts.

###  IV-A Captured Dataset

The following analyses is based on conversation logs we captured from November
14 to December 31 in 2021. We observed that 527 hosts had interacted with our
honeypot system. These hosts distributed across 35 countries/regions and 121
autonomy systems as listed in Fig. 6 and Table II. To reduce the scale of the
dataset, we took less than 20 conversation logs for each host and selected
4,855 out of over 3 million items. This produced 481 unique request logs and
generated feature vectors of 8,083 dimensions.

TABLE II: Distribution of loader hosts in autonomy systems Country | Name | Hosts | Logs
---|---|---|---
China | CHINA UNICOM China169 Backbone | 184 | 2145
India | Hathway IP Over Cable Internet | 29 | 361
Russia | MTS PJSC | 29 | 360
China | Chinanet | 28 | 286
India | National Internet Backbone | 20 | 243
Others |  | 237 | 1454

We made a simple statistical analysis of loader hosts’ geographical
distribution. It shows that China, Russia, India, and the United States are
preferred countries for botmasters to setup loader servers. Over 75% loaders
are located in these 4 countries. While it is common sense for Mirai-based
botnets to build infrastructures on public cloud servers, we can still see
there are lots of servers deployed in consumer networks like ASN4837 and
ASN17488. This means that some botnets might have a different architecture
from Mirai, and loaders are also distributed across multiple networks and
regions.

###  IV-B Family Definition

In this step, we leverage agglomerative clustering and try to define several
families for bot loaders based on the collected dataset.

####  IV-B1 Clustering overview

The agglomerative clustering algorithm generated a tree with a height of
1193.07, which dendrogram is depicted in 7. While the tree is relatively tall,
most of the branches are at a height below 200. Minority branches at a higher
height may manifest significant differences in corresponding bot loaders’
behaviors. Based on the method in Section III-E, we recursively generated
templates for 480 non-singleton clusters to describe their common behaviors.

Figure 7: Dendrogram of clustering result and family definitions. The colored
bars indicate the family definition of the agglomerative tree. The counts of
member clusters of each family are listed in the legend on the right side. Red
horizontal line for $\mathcal{T}=60$.

####  IV-B2 Threshold selection

Although the agglomerative clustering algorithm hierarchically clustered all
the samples we captured into a tree, we should cut the tree at a threshold
$\mathcal{T}$ to obtain an elementary partition on the dataset for further
manual analyses. Griffioen and Doerr [16] examined the battle among IoT
botnets. Although this study listed 39 families according to their
observation, most of the behavior analyses only focused on 14 botnets which
are the most active ones. In Wang’s text-based analysis on botnet reports[20],
the experiment shows that despite there is a large number of observed botnets,
most of them are actually minor revisions or just aliases of some popular
botnets. Combining conclusions from these studies, we expect a $\mathcal{T}$
value producing around 20 clusters, which is a reasonable amount for covering
active bot loaders. The limited number of clustering also makes it convenient
to examine their templates and further refine the family definition. Based on
the clustering result, we empirically set $\mathcal{T}=60$, which is displayed
as a red horizontal line in the Fig. 7. This produced 19 clusters from the
previous clustering result.

####  IV-B3 Interpreting the agglomerative tree

TABLE III: Index Table of Loader Functions Discussed in Section IV-C

Based on the extracted templates, we evaluated these clusters based on the
aforementioned criteria in section III-F, and finally figured out several
families out of the dataset. We traverse their sub-clusters and find some
representative token as their name, which may not follow the common rule of
naming a botnet. We listed these families and indexed their functions in Table
III.

The yellow branch in the Fig. 7 consists of 3 families. These families share
the original Mirai codebase and made minor revisions to fit their botmasters’
needs:

  * •

_Nippon-kami._ This is the family which directly derived from Anna-Senpai’s
release[6]. “Query token” which queries the status of command are frequently
modified to declare the botmaster’s identity, and the initialize command list
to enable shell interface is the most volatile part all over the class.

  * •

_SEFA._ This loader manifests itself with the token “SEFA” in initial commands
and the query token, but obviously share the same codebase with the _“nippon-
kami”_ family.

  * •

_Port._ This loader calls openssl for some unknown reason and we have not seen
it dropping any malware yet. We observed this loader only in late December
2021, which means it might have retired in January 2022.

We also found several families that were located relatively further to the
yellow branch, so we color the branch red in Fig. 7. These 5 families are
implemented significantly different and modified some crucial components to
satisfy the needs of their botmasters:

  * •

_No-path-check._ This is a tailored variant of the _“nippon-kami”_ loader. All
the commands prior to the step of checking architecture are removed and it
simply uses the default working directory of the logged-in user.

  * •

_SwitchBlades._ This loader derived the framework of _nippon-kami_ , but has a
different implementation to detect writable directory, which actually acts
like _Sofia_. We took the name from the first sample we saw in this family,
but the identification token is volatile and we also see others including
Layer1 and skull.

  * •

_Sofia._ This loader extremely extended the initial command list. It
implemented a different method to detect writable directories and use a hard-
coded list to scan the file system.

  * •

_6-chars._ The most significant feature of it is the query string. Unlike
other loaders, loaders of this family generate 6 random escaped characters as
a query string for every session. We noticed that some scanners also share the
same pattern, but here we distinguish them as an independent _6-chars-scanner_
family and won’t do functional analysis on them.

  * •

_“whattttttlol”._ This loader doesn’t have any shared parts with the
aforementioned loaders, which makes its behavior significantly different from
others. It runs a fixed command list and downloads multiple shell scripts
named “whattttttlol*.sh” to load the malware.

####  IV-B4 Clustering of scanners

During the experiment, we notice that some samples in particular clusters do
not carry any downloading commands. These data all distributed on the green
branch in Fig. 7 but can be divided into 3 clusters. The data of previously
mentioned _6-chars-scanner_ also distributes on this branch. Based on this
phenomenon, we regard them as scanners (named “others”) which simply probe
exploitable hosts but never load an executable.

####  IV-B5 The control group

Our method successfully classified the control group dataset we mentioned in
Section I. The request logs generated by the local Hajime scanner are located
closely with other scanners and Anna-senpai’s Mirai loader is located along
with its variants in the yellow branch. This result substantiated the
effectiveness of our method in clustering similar loaders and distinguishing
different malicious adversaries.

###  IV-C Behavior Investigation

Figure 8: Homology dendrogram of captured loaders. Horizon lines depict
identified families, while the yellow one highlights the _nippon-kami_
released by Anna-Senpai. In the description of a connecting line between two
families, green “+” indicates adding functions, red “-” indicates removing
functions, and blue “±” indicates modifications. All the functions are denoted
by indexes in Table III.

_B 1-2_

⬇

busybox echo -e ’\\\x6b\\\x61\\\x6d\\\x69/proc’ > /proc/.nippon;

busybox cat /proc/.nippon;

busybox rm /proc/.nippon

_B 2 and B3_

⬇

>/var/tmp/.file && cd /var/tmp/

Figure 9: Sample codes of the “Get working directory” function denoted by
indexes in Table III.

_D 1-1_

⬇

/bin/busybox cp /bin/echo sefaexecbi; >sefaexecbi; /bin/busybox chmod 777
sefaexecbi;

_D 1-2_

⬇

/bin/busybox cat /bin/echo

_D 2_

⬇

/bin/busybox cat /bin/busybox || while read i; do echo $i; done < /bin/busybox

Figure 10: Sample codes of the “Test environment” function denoted by indexes
in Table III.

In this section, we interpret the Table III vertically to make a comprehensive
comparison of functions based on extracted templates and make a conclusion
about their shared codebases. In this section, we denote all the discussed
components by alphanumeric indexes in Table III. By discussing all the
relationships of their components, we draw a dendrogram (Fig. 8) to show the
homology of functions among all classes we defined. The yellow line represents
nippon-kami which is directly derived from Anna-Senpai’s release. The
intrusion process can be distinguished into 6 major functions:

####  IV-C1 Initialize

To enable the full shell on an IoT device in the telnet conversation, loaders
execute an initialization command list to cover all the possible commands.
Then, the loader checks the environment to scan existing bot instances or
security services.

As Nippon-kami, SEFA, and Port share the same codebase, their initial command
list varies slightly depending on their botmasters. Then they run ps command
to check suspicious processes in the environment (A1). The SEFA loader added a
command for modifying the victim’s hostname to SEFA_ID:<4-digit numbers> (A2)
which could be an identification of bots in the botnet. Sofia has an extended
list for initializing commands, but the commands checking environments have
been removed. While the whattttttlol holds a fixed command list, it runs ls
/home to scan files in the directory (A3). 6-chars only checks the wget
command here, which is commonly executed later in other loaders (A4).

####  IV-C2 Get working directory

Most of the loaders require a writable directory to temporarily drop the
executable. The directory scanner is an important component of loaders which
has many different implementations. Nippon-kami, SEFA, and Port get mounted
directories (B1-1) and run a composed statement (Bto check its write privilege
in this step1-2). It prints an escaped token kami<path> using echo to a file
named .nippon, reads the file and test if the file is successfully written.
Port uses a different escaped token and file name in this step.

SwitchBlades, Sofia, 6-chars and whattttttlol all use a hard-coded list
instead of scanning the mounted filesystems at the run-time. Both SwitchBlades
(B2) and 6-chars (B3) run an “&&” (and) statement to test writable privilege
using a hard-coded list, while the list is slightly different in these two
families. A variant of 6-chars runs this step twice, which shows a difference
in the request logs. SwitchBlades uses a return to join these statements. The
last usable directory will be used in the following steps.

Whattttttlol uses a simple “||” (or) statement to join multiple cd <directory>
(B4). This changes the working directory to the first available one in the
hard-coded list, regardless of its writable privilege.

Some of the implementations are displayed in Fig. 9 to show their difference.

####  IV-C3 Monopolize

It is extremely normal for vulnerable devices to be infected repeatedly. The
newcomer should kill the existing bot client to acquire control of the device.
Most loaders will try eliminating competitors by deleting certain files in a
built-in list. Nippon-kami and its variants use .sh .t .human (C1), Sofia use
.file .cowbot.bin retrieve cowffxxna (C3), and 6-chars removes .i only (C4).
SwitchBlades tries to delete two files that may be related to its old
executables. The list is unstable among different variations (C2).

####  IV-C4 Test environment

In this step, the loaders should probe the CPU architecture and available
download commands to decide which executable will be loaded and how it will be
loaded. The classic method (D1-1, D1-2, D1-3) tests cp command, prints
/bin/echo, then test wget and tftp commands. The CPU architecture can be
obtained by parsing an existing executable on the device. D1-2 implement this
by running cat /bin/echo. For D2, no-path-check and Sofia prints /bin/busybox
to obtain the same information. In case of cat command is unavailable, they
also use a shell-based while read statement to print the file. For cross-
platform botnets, this will decide which cross-compiled executable will be
dropped in the next step.

Some of the implementations are displayed in Fig. 10 to show their difference.

####  IV-C5 Drop & run malware

Loader download the executable, setup the environment, execute it and clean
all its trails in the final stage. In this step, loaders cd to the probed
working directory leverage the available download tool detected in the
previous step to drop the malware.

Specifically for nippon-kami (E1-1), if both wget and tftp are unavailable, it
will run the fallback command and load the whole file with echo command and a
stdout redirect statement. SEFA, no-path-check and SwitchBlades all reused
this implementation (E1-*).

6-chars leverages an “||” (or) statement to call multiple commands
sequentially (E3-*) until a command succeeds. Whattttttlol called multiple
commands sequentially to download and run 4 scripts and deleted them all after
execution (E4-*).

In our collected dataset, we did not observe Port and Sofia downloading
executables in this step. Instead, Port called openssl for unknown reason
(E2), while Sofia only checked writable privilege in the current directory
(D1-1).

####  IV-C6 Query token

This command is designed to run after every command to identify this step has
finished. The loader would wait for a specified token in response before it
goes on.

Figure 11: Geological distribution of observed loader families.

###  IV-D Geological Distribution of Loader Families

We draw a circos graph to further demonstrate the geological distribution of
different loaders. Scanners except 6-char-scanner are dropped. As shown in
Fig. 11, 6-char loaders are distributed diversely across China, India, Russia,
and Pakistan, while nippon-kami and all its variants are only observed to have
a few loader hosts distributed in a small number of countries. This inspired
us to reach the following two conclusions.

  * •

While Mirai is designed to have a centralized loader system, many other
botnets also try integrating loader programs into bot clients, including the
botnet using 6-chars as the loader. A distributed loading system makes the
botnet more robust when coping with taking-downs on single loader servers.
Hajime had already implemented this leveraging its modularized design.

  * •

For small cybercrime teams, developing on Mirai’s codebase is still a good
choice. The existence of SEFA and Port is a piece of important evidence on
this. Some other teams also implement their loader for some reason. These
small botnets implemented some new features but do not have the redundant
resources to build multiple loader servers. We conclude that there is still a
long way to go before Mirai disappears if ever possible.

##  V Discussion

For this study, we focus on investigating loaders based on the interaction
logs. However, this limited methodology is far from ultimate for mining all
covert information of botnets.

#### Associated malware

Bot client is the most volatile part of botnets. Some existing research made
lineage analyses on executables captured by honeypots. However, our work lacks
aspects on bot clients. Combining these two analyses to analyze relationships
among loaders and clients may extend the conclusion of homology analysis,
which is a feasible and promising field for botnet studies.

#### Family definition

Our work draws a conclusion based on the result of agglomerative clustering.
The empirical value of $\mathcal{T}$ determined the family definition of bot
loaders. However, due to the heterogeneity of loaders’ behavior, a homogeneous
$\mathcal{T}$ cannot fit all situations. For example, We found that
whattttttlol was misclassified along with 6-chars, which reflects our method
is still deficient to make a clear definition of classes. The cutting
condition of the agglomerative tree should be further discussed in the
following studies. Note that the query tokens and credentials are not fully
investigated, which could miss critical information of loaders.

#### Early stage detection

For IDS systems, the loading stage is one of the best chances to intercept
emerging attacks. Our methodology for analyzing network messages push the
identification further to recognize the identity of attackers, which is of
great importance in reinforcing the security of the Internet.

##  VI Conclusion

In recent years, the open-source botnets spawned a mass of variants, which is
a major threat to IoT devices. While understanding the behaviors of botnet
malware is crucial, their infrastructures, especially loader servers, are not
systematically investigated yet. In this work, we proposed a novel aspect of
homology analysis on bot loaders. We analyze loaders’ implementation using
honeypot logs and evaluate their relationship by shared components. Our
experiment consists of 3 components. First, we collect interaction logs using
a high interaction honeypot system to represent the functions and behaviors of
loader hosts. Second, we use agglomerative clustering to cluster these logs,
fit all relationships onto a tree, and cut at a threshold to classify the
loaders. Finally, we evaluate the tree bottom-top and leverage the align
algorithm to evaluate shared components in each loader family. We find that
the ecosystem of Mirai is still active despite numerous takedowns. New
variants are created to satisfy botmasters’ needs in many aspects. We also
discover that some small-scale botnets come and fade away in a very short
timespan. The security community should further develop the methodology of
botnet investigation to enhance the knowledge of botnets’ relationships, which
could help eliminate cybercrimes more efficiently.

## References

  * [1] C. Kolias, G. Kambourakis, A. Stavrou, and J. Voas, “DDoS in the IoT: Mirai and Other Botnets,” _Computer_ , vol. 50, no. 7, pp. 80–84, 2017.
  * [2] F. Dang, Z. Li, Y. Liu, E. Zhai, Q. A. Chen, T. Xu, Y. Chen, and J. Yang, “Understanding Fileless Attacks on Linux-based IoT Devices with HoneyCloud,” in _Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services_. Seoul Republic of Korea: ACM, Jun. 2019, pp. 482–493.
  * [3] H. L. J. Bijmans, T. M. Booij, and C. Doerr, “Just the Tip of the Iceberg: Internet-Scale Exploitation of Routers for Cryptojacking,” in _Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security_ , ser. CCS ’19. New York, NY, USA: Association for Computing Machinery, Nov. 2019, pp. 449–464.
  * [4] M. Antonakakis, T. April, M. Bailey, M. Bernhard, E. Bursztein, J. Cochran, Z. Durumeric, J. A. Halderman, L. Invernizzi, M. Kallitsis, D. Kumar, C. Lever, Z. Ma, J. Mason, D. Menscher, C. Seaman, N. Sullivan, K. Thomas, and Y. Zhou, “Understanding the Mirai Botnet,” in _26th USENIX Security Symposium (USENIX Security 17)_ , 2017, pp. 1093–1110.
  * [5] Malware Must Die!, “MMD-0056-2016 - Linux/Mirai, how an old ELF malcode is recycled..” Sep. 2016.
  * [6] Anna-senpai, “[FREE] World’s Largest Net:Mirai Botnet, Client, Echo Loader, CNC source code release,” https://hackforums.net/showthread.php?tid=5420472, Sep. 2016.
  * [7] S. Herwig, K. Harvey, G. Hughey, R. Roberts, and D. Levin, “Measurement and Analysis of Hajime, a Peer-to-peer IoT Botnet,” in _Proceedings 2019 Network and Distributed System Security Symposium_. San Diego, CA: Internet Society, 2019.
  * [8] W. Fan, Z. Du, D. Fernandez, and V. A. Villagra, “Enabling an Anatomic View to Investigate Honeypot Systems: A Survey,” _IEEE Systems Journal_ , vol. 12, no. 4, pp. 3906–3919, Dec. 2018.
  * [9] E. Cozzi, P.-A. Vervier, M. Dell’Amico, Y. Shen, L. Bilge, and D. Balzarotti, “The Tangled Genealogy of IoT Malware,” in _Annual Computer Security Applications Conference_ , ser. ACSAC ’20. New York, NY, USA: Association for Computing Machinery, Dec. 2020, pp. 1–16.
  * [10] J. Jang, M. Woo, and D. Brumley, “Towards Automatic Software Lineage Inference,” in _22nd USENIX Security Symposium (USENIX Security 13)_ , 2013, pp. 81–96.
  * [11] M. Dib, S. Torabi, E. Bou-Harb, and C. Assi, “A Multi-Dimensional Deep Learning Framework for IoT Malware Classification and Family Attribution,” _IEEE Transactions on Network and Service Management_ , vol. 18, no. 2, pp. 1165–1177, Jun. 2021.
  * [12] V. Ghiette, H. Griffioen, and C. Doerr, “Fingerprinting Tooling used for {}SSH{} Compromisation Attempts,” in _22nd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2019)_ , 2019, pp. 61–71.
  * [13] S. Torabi, M. Dib, E. Bou-Harb, C. Assi, and M. Debbabi, “A Strings-Based Similarity Analysis Approach for Characterizing IoT Malware and Inferring Their Underlying Relationships,” _IEEE Networking Letters_ , vol. 3, no. 3, pp. 161–165, Sep. 2021.
  * [14] A. Z. Tabari, X. Ou, and A. Singhal, “What are Attackers after on IoT Devices? An approach based on a multi-phased multi-faceted IoT honeypot ecosystem and data clustering,” _arXiv:2112.10974 [cs]_ , Dec. 2021\.
  * [15] B. Lingenfelter, I. Vakilinia, and S. Sengupta, “Analyzing Variation Among IoT Botnets Using Medium Interaction Honeypots,” in _2020 10th Annual Computing and Communication Workshop and Conference (CCWC)_ , Jan. 2020, pp. 0761–0767.
  * [16] H. Griffioen and C. Doerr, “Examining Mirai’s Battle over the Internet of Things,” in _Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security_ , ser. CCS ’20. New York, NY, USA: Association for Computing Machinery, Oct. 2020, pp. 743–756.
  * [17] B. Stephens, A. Shaghaghi, R. Doss, and S. S. Kanhere, “Detecting Internet of Things Bots: A Comparative Study,” _IEEE Access_ , vol. 9, pp. 160 391–160 401, 2021.
  * [18] L. Spitzner, “Honeypots: Catching the insider threat,” in _19th Annual Computer Security Applications Conference, 2003. Proceedings._ , Dec. 2003, pp. 170–179.
  * [19] Y. M. P. Pa, S. Suzuki, K. Yoshioka, T. Matsumoto, T. Kasama, and C. Rossow, “IoTPOT: Analysing the Rise of IoT Compromises,” in _WOOT ’15 - 9th USENIX Workshop on Offensive Technologies_ , Washington D.C., USA, Aug. 2015, p. 9.
  * [20] H. Wang, W. Zhang, H. He, P. Liu, D. X. Luo, Y. Liu, J. Jiang, Y. Li, X. Zhang, W. Liu, R. Zhang, and X. Lan, “An Evolutionary Study of IoT Malware,” _IEEE Internet of Things Journal_ , vol. 8, no. 20, pp. 15 422–15 440, Oct. 2021.
  * [21] M. Safaei Pour, A. Mangino, K. Friday, M. Rathbun, E. Bou-Harb, F. Iqbal, S. Samtani, J. Crichigno, and N. Ghani, “On data-driven curation, learning, and analysis for inferring evolving internet-of-Things (IoT) botnets in the wild,” _Computers & Security_, vol. 91, p. 101707, Apr. 2020.
  * [22] S. G. Abbas, F. Hashmat, G. A. Shah, and K. Zafar, “Generic signature development for IoT Botnet families,” _Forensic Science International: Digital Investigation_ , vol. 38, p. 301224, Sep. 2021.
  * [23] J. D. Guarnizo, A. Tambe, S. S. Bhunia, M. Ochoa, N. O. Tippenhauer, A. Shabtai, and Y. Elovici, “SIPHON: Towards Scalable High-Interaction Physical Honeypots,” in _Proceedings of the 3rd ACM Workshop on Cyber-Physical System Security - CPSS ’17_. Abu Dhabi, United Arab Emirates: ACM Press, 2017, pp. 57–68.
  * [24] G. Wagener, R. State, A. Dulaunoy, and T. Engel, “Heliza: Talking dirty to the attackers,” _Journal in Computer Virology_ , vol. 7, no. 3, pp. 221–232, Aug. 2011.
  * [25] A. Vetterl and R. Clayton, “Honware: A Virtual Honeypot Framework for Capturing CPE and IoT Zero Days,” in _2019 APWG Symposium on Electronic Crime Research (eCrime)_. Pittsburgh, PA, USA: IEEE, Nov. 2019, pp. 1–13.
  * [26] T. Krueger, H. Gascon, N. Krämer, and K. Rieck, “Learning stateful models for network honeypots,” in _Proceedings of the 5th ACM Workshop on Security and Artificial Intelligence - AISec ’12_. Raleigh, North Carolina, USA: ACM Press, 2012, p. 37.
  * [27] K. Chidananda Gowda and G. Krishna, “Agglomerative clustering using the concept of mutual nearest neighbourhood,” _Pattern Recognition_ , vol. 10, no. 2, pp. 105–112, Jan. 1978.
  * [28] “Scikit-learn: Machine learning in Python,” https://scikit-learn.org/stable/.
  * [29] T. F. Smith and M. S. Waterman, “Identification of common molecular subsequences,” _Journal of Molecular Biology_ , vol. 147, no. 1, pp. 195–197, Mar. 1981.