11institutetext: Ahsanullah University of Science and Technology, Bangladesh
11email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# Effectiveness of Transformer Models on IoT Security Detection in
StackOverflow Discussions

Nibir Chandra Mandal  G. M. Shahariar Both authors contributed equally to this
research. Md. Tanvir Rouf Shawon††footnotemark:

###### Abstract

The Internet of Things (IoT) is an emerging concept that directly links to the
billions of physical items, or “things", that are connected to the Internet
and are all gathering and exchanging information between devices and systems.
However, IoT devices were not built with security in mind, which might lead to
security vulnerabilities in a multi-device system. Traditionally, we
investigated IoT issues by polling IoT developers and specialists. This
technique, however, is not scalable since surveying all IoT developers is not
feasible. Another way to look into IoT issues is to look at IoT developer
discussions on major online development forums like Stack Overflow (SO).
However, finding discussions that are relevant to IoT issues is challenging
since they are frequently not categorized with IoT-related terms. In this
paper, we present the "IoT Security Dataset", a domain-specific dataset of
7147 samples focused solely on IoT security discussions. As there are no
automated tools to label these samples, we manually labeled them. We further
employed multiple transformer models to automatically detect security
discussions. Through rigorous investigations, we found that IoT security
discussions are different and more complex than traditional security
discussions. We demonstrated a considerable performance loss (up to 44%) of
transformer models on cross-domain datasets when we transferred knowledge from
a general-purpose dataset "Opiner", supporting our claim. Thus, we built a
domain-specific IoT security detector with an F1-Score of 0.69. We have made
the dataset public in the hope that developers would learn more about the
security discussion and vendors would enhance their concerns about product
security . The dataset can be found at -
https://anonymous.4open.science/r/IoT-Security-Dataset-8E35

###### Keywords:

IoT IoT Security Transformers StackOverflow Machine Learning

##  1 Introduction

The IoT (Internet of Things) is a widespread connection of intelligent devices
that are bridged through the internet and are rapidly expanding to every
corner of the globe. The number of IoT devices will touch a $36.8$ billion
margin by 2025, which is $107\%$ higher than in 2020 [7]. Because IoT devices
carry a lot of information about their users, the growing number of IoT
devices presents a lot of security concerns for developers. An unintentional
intrusion into the device might cause significant harm to the users. As a
result, today’s developers are paying close attention to this scenario and
attempting to identify the best solution for protecting devices from
attackers. In numerous developer communities, there is a lot of discussion
about the security of IoT devices. One of the communities that developers use
to address security solutions is Stack Overflow (SO). The SO community is
quite active and has a lot of IoT security discussions. A great number of
works have been done in this research field [9][10][12]. In this work, we have
tried to demonstrate a comprehensive analysis of determining whether an aspect
of a discussion is an IoT security aspect or not. The purpose of this study is
to assist developers in learning more about security issues in the IoT space,
where there is a lot of discussion about various aspects. Furthermore, with
the aid of our efforts, vendors may increase the security of their products.
We have basically employed some of the popular transformer models like
BERT[2], RoBERTa[3], XLNet[5] and BERTOverflow[6] in $3$ of our different
experiments. For our initial approach, we used a benchmark dataset called
Opiner [8], which recorded a variety of aspects of discussions such as
performance, usability, security, community, and so on. The field of
discussion was also diverse. Instead of a general-purpose dataset, we worked
to create a domain-specific dataset focused solely on IoT security issues,
which we named the "IoT Security Dataset". Following the creation of the
dataset, we attempted to transfer the knowledge gained from the Opiner dataset
to our own IoT dataset, but the results were unsatisfactory. Then we decided
to fine-tune the pretrained tansformer models using our own dataset, and we
achieved a decent result. From these experiments, we discovered that knowledge
transfer is not a good technique to get a decent outcome. The best way to
identify security issues in developer discussions is to conduct domain-
specific experiments using domain-specific data. In summary, we have made the
following main contributions in this paper:

  * •

We have created a domain-specific dataset called "IoT Security Dataset" that
focuses on security aspects of IoT related textual developer discussions.

  * •

We further employed multiple transformer models to automatically detect
security discussions. Through rigorous investigations, we found that IoT
security discussions are different and more complex than traditional security
discussions.

  * •

We demonstrated a considerable performance loss (up to 44%) of transformer
models on cross-domain datasets when we transferred knowledge from a general-
purpose dataset "Opiner", supporting our claim.

##  2 Background Study

###  2.1 Transformers

A neural network that learns factors in sequential data is referred to as a
transformer model. A transformer model provide us with an word embedding which
is used as the input of a fully connected neural network for further
classification task. In natural language processing transformer models are
used frequently as it tracks the meaning of consecutive data by tracking the
relationships among them like the words in a sentence. Transformer models were
first introduced by Google [1] and are one of the most powerful models till
date in the history of computer science. We incorporated four different kinds
of transformer models in our work which are RoBERTa, BERT, XLNET and
BERTOverflow. Bidirectional Encoder Representations from Transformers or BERT
[2] was first established by Jacob Devlin et al. which is a transformer model
to represent language. It was modelled in such way that anyone can fine tune
it with a supplementary layer to create new models to solve a wide range of
tasks. Yinhan Liu et al. introduced RoBERTa [3] as a replication study of BERT
[2]. Authors showed that their model was highly trained than BERT overcoming
the limitations and showed a good performance over basic BERT model. Zhilin
Yang et al. introduced XLNet [5] which overcomes the constraints of BERT [2]
using a universal autoregressive pre-training technique. Maximum expected
likelihood was taken into consideration over all arrangements of the
factorization order. Authors showed that their model beat BERT by a huge
margin on a variety of tasks like sentiment analysis, language inference or
document ranking etc. BERTOverflow [6] by Jeniya Tabassum et al. is a kind of
transformer model trained for identifying code tokens or software oriented
entities inside the area of natural language sentences. Authors introduced a
NER corpus of total $15,372$ sentences for the domain of computer programming.

###  2.2 Evaluation matrices

Any kind of model requires some measurement criteria to understand its’
efficiency. In this project $4$ different kinds of performance matrices were
used which are Precision, Recall, F1-Score and Area Under Curve (AUC). The
percentage of accurately predicted labels that are literally correct is
calculated by a model’s precision score. It also indicates the calibre of a
model to predict the labels correctly. The model’s ability to reliably
discover positives among existing positives is measured by its recall score.
We can also state it as a model’s capacity to locate all relevant cases within
a data set. The precision and recall score are used to generate the F1-Score.
The F1-score is a metric in which Precision and Recall play an equal role in
evaluating the model’s performance. A graphical illustration of how
effectively binary classifiers work is the Receiver Operating Characteristic
(ROC) curve. (AUC) score is calculated using ROC and is a very useful tool for
calculating a model’s efficiency in an uneven dataset.

###  2.3 Cross validation

A reshuffling process for assess various algorithms and models on a little
amount of data is known as Cross-validation. It comprises an attribute, k,
denoting the quantity of sets a provided sample of data should be partitioned.
That is why, the technique is widely known as k-fold cross-validation. When we
give a value to the parameter k, it means we want to partition the whole data
into k sets.

##  3 The IoT Security Dataset

In this section, we will present details of the IoT security dataset creation
process. The whole process is divided into three subprocesses: (1) Collect IoT
posts (2) Extract sentences from IoT posts and (3) Label extracted sentences.
Each process is discussed in the following subsections.

###  3.1 IoT Posts Collection

As IoT tags do not cover all IoT related posts, we applied three steps to
collect IoT related posts.

  1. Step 1)

Download the SO dataset: We used the September 2021 Stack Overflow (SO) data
dump, which was the most recent dump accessible at the time of research. The
following meta data may be found in each post: (1) text data and code samples;
(2) creation and edition times; (3) score, favorite, and view counts; (4) user
information of the post creator; and (5) if the post is a question, tags given
by the user. If the user who posted the question marked the response as
approved, the answer is flagged as accepted. A question might include anything
from one to five tags. All postings from 2008 to September 2021 are included
in the SO dataset.

  2. Step 2)

Identify IoT Tagset: Because not all of the posts in the SO dataset are linked
to the IoT, we had to figure out which ones had IoT-related topics. We
established the set of tags that may be used to label a discussion as IoT-
related using the user-defined tags attached to the questions. We followed the
two-step method described in [14]: (1) In SO, we discovered three prominent
IoT-related tags. (2) We gathered postings with those three initial tags and
studied the tags that were applied to them. There are 78 tags in the final tag
collection. We have discussed each step in detail below.

    1. (a)

Identify initial tags: Using the Stack Overflow (SO) search engine, the three
initial tags were chosen. We began our search by looking for queries in SO
that were labeled with "iot", and the SO search engine returned posts that
were tagged with "iot" as well as a set of 25 additional tags that were
related to these topics, such as "raspberry-pi", "arduino", "windows-10-iot-
core", "python," and so on. In our initial set, we thus considered the
following three tags: (a) "iot" or any tag including the term "iot", such as
"windows-10-iot-core"; (b) "arduino", and (c) "raspberry-pi".

    2. (b)

Determine final tagset: To begin, we looked for IoT-related posts using the
three initial tags. Second, we retrieved all questions from the SO dataset
that were tagged with one of the initial three tags. Finally, we collected all
of the tags from the question set. It is possible that not all of the tags in
the questions set match to IoT subjects (for example, "python"). As a result,
the significance and relevance specified in [14] for each tag were computed to
pick filtered and finalized IoT-related tags in the questions set. If a tag’s
significance and relevance values exceed certain criteria, it is considered
significantly relevant to the IoT. After comprehensive testing, we discovered
78 IoT-related tags. Please see [14] for a more complete discussion of the
tagset identification technique.

  3. Step 3)

Collect IoT Posts: All posts labeled with at least one of the selected 78 tags
make up our final dataset. If a SO question is labeled with one or more tags
from the final tagset, it is an IoT question. In our SO dump, we discovered a
total of 40K posts based on the 78 tags.

###  3.2 Sentence Level Dataset Creation

As security discussions can be buried inside a post, So we followed sentence-
level analysis to get a better understanding of the security discussion. We
got the sentences as follows: As our main focus is textual analysis, we
avoided security discussion inside code snippets and urls. We used the
BeautifulSoup library to extract all that extra information such as titles,
codes, and urls. We also used the NLTK sentence tokenizer to get all the
sentences from those 40K posts. Finally, we found around 200K sentences.

###  3.3 Benchmark Dataset Creation

As there are not any automated tools to label this large data set, we chose to
label them manually. Due to the data size and sparsity, we labeled a
statistically significant portion of the data. For our experiment, we randomly
selected 7147 (99% confidence with a confidence interval of 1.5) sentences and
labeled them. Without knowing each other, two developers labeled this data
first. In addition, a third developer was introduced to resolve the conflict.
We labeled each sentence by majority voting. For example, if both developers
agreed, we labeled them with the same label they agreed on. Otherwise, we took
into account the third developer’s decision. All these developers have
industrial experience of more than 2 years and have enough knowledge of the
domain. The summary of our dataset can be seen here in Table 1.

Table 1: Security Data Distribution for Opiner and IoT security datasets. Dataset | Size | Security | Kappa |  | Agreement
---
Score
Opiner | 4522 | 163 (3.6%) | - | -
IoT | 7147 | 250 (3.5%) | 0.92 | 98.5%

##  4 Opiner Dataset

Opiner datatset was created by Uddin et al. [19] to study developers opinion
in SO for different APIs. The dataset includes developer discussions in form
of text and related aspects of that discussions. Each opinion may have
multiple aspects such as "Performance", "Usability", etc. One of those aspects
is "Security". In this research, we only took consideration of this aspect. We
discarded other aspects and labeled all security aspect related samples as 1
and other samples as 0. We found a total of 163 samples are security related.

##  5 Methodology

The proposed system for aspect categorization is presented in this section.
The key phases of the proposed method for binary aspect classification
(considering usability aspect as an example) are summarized in Figure 1 and
are further detailed below.

  1. Step 1)

Input Sentence: Each raw sentence from the dataset is presented to the
proposed model one by one for further processing. Before that each sentence
goes through some pre-processing steps: all urls and codes are removed.

  2. Step 2)

Tokenization: Each processed sentence is tokenized using the BERT Tokenizer.
Each tokenized sentence gets a length of 100 tokens and zero padded when
required. In the event of length more than 100, we cut off after 100. The
output of this step is a tokenized sentence of size 100.

  3. Step 3)

Embedding: For word embedding, BERT [2] is used to turn each token in a
sentence into a numeric value representation. Each token is embedded by 768
real values via BERT. The input to this step is a tokenized sentence of size
100 and output of this step is an embedded sentence of size 100*768.

  4. Step 4)

Pooling: To reduce the dimension of the feature map (100*768) for each
tokenized sentence in step 3, max pooling is used which provides a real valued
vector representation of size 768 per sentence.

Figure 1: Aspects Classification Process

  5. Step 5)

Classification: Aspect classification is accomplished by the use of transfer
learning [22]. To classify security and non security aspects, a basic neural
network with two dense layers (with 128 and 2 neurons) is utilized to fine-
tune the four pretrained transformer models one by one such as RoBERTa [3],
BERT [2], DistilBERT [4], and XLNET [5]. The pretrained weights were
initialized using glorot_uniform in the simple neural network. A dropout rate
of 0.25 is employed between the dense layers. Finally, we apply a ‘Softmax’
activation to get the prediction.

##  6 Experimental Results

###  6.1 Experiments

At first, we found that Uddin el.’s Logistic Regression (Logits) model [19]
outperformed other rule-based models. Thus, we took this Logits as our
benchmark model. In recent times, deep models, specifically pretrained models,
have shown a huge improvement over both shallow machine learning models such
as SVM and Logistics Regression and classic deep models such as Long Short
Term Memory (LSTM), Convolutional Neural Network (CNN), and Bidirectional Long
Short Term Memory (Bi-LSTM). Therefore, we decided to use the Transformer
models. We applied these already trained models on Opiner dataset to test the
IoT dataset which we have prepared for this study to check whether a general
purpose dataset can give a good generalization performance for a domain
specific dataset. We did not get a satisfactory result from this experiment as
the generalization is not a good idea. Finally, we added domain-specific IoT
security data during the tuning of the pre-trained models and found acceptable
results from the experiment. The performances of all the models on opiner
dataset are shown in Table 2. The result of logistic regression is taken from
the experiments [8] of Gias Uddin et al on the benchmark Opiner dataset. A
comparison of performances among all the $3$ experiments on Cross-validation
on Opiner dataset, Cross-Domain dataset (Trained on Opiner and tested on IoT
dataset) and Cross-validation on IoT Dataset for all the transformer models we
used are shown in figure 2.

Table 2: Performance of $4$ different transformer model on Opiner dataset  Type | Model | Precision | Recall | F1-Score | AUC
---|---|---|---|---|---
Proposed Models | BERT | 0.72 | 0.79 | 0.74 | 0.89
RoBERTa | 0.86 | 0.76 | 0.79 | 0.92
XLNet | 0.72 | 0.84 | 0.77 | 0.91
BERTOverflow | 0.79 | 0.76 | 0.76 | 0.88
Previous Work | Logits [19] | 0.77 | 0.57 | 0.60 | 0.69

Figure 2: A comparison of performance among all the $3$ experiments for BERT
(top left), RoBERTa (top right), XLNET (lower left) and BERTOverflow (lower
right) models.

###  6.2 Result Analysis

First, We experimented with three generic transformers, BERT, RoBERTa, and
XLNet, as well as one domain-specific model, BERTOverflow. As these robust
pretrained models can capture underlying implicit contexts, the performance of
these models significantly improved over shallow models, although the size of
the dataset is comparatively low. We found that the lowest performing
transformer model has 23% better F1-Score than the baseline models.

We found that RoBERTa model has the best performance in terms of F1-Score
among other transformers. Despite having a low recall, RoBERTa achieves the
highest F1-Score of 0.79, followed by XLNet (0.77), BERTOverflow (0.76), and
BERT (0.74). This indicates that RoBERTa is the most precise model, but it has
lower discoverability. Thus, the generic purpose transformer RoBERTa
outperforms both the domain-specific BERTOverflow and other generic purpose
models. This happens for two reasons. First, RoBERTa is the most optimized
transformer model. Due to this optimization, RoBERTa gets an extra edge to
outperform its base model, BERT, for our task. Second, security discussions in
the Opiner dataset are more generic than domain-specific. Although the
traditional rule-based approach fails to guess the security discussions as
correctly as deep models like the transformer model, SO domain knowledge makes
the prediction more intriguing. XLNet has enough insights to identify security
discussions but it suffers from precision. As example, we found the following
sentences in Opiner dataset: ‘For instance, messages sent by the client, may
be digitally signed by the applet, with the keys being managed on per-user USB
drives (containing the private keys).’ It is clearly visible that the word
‘signed’ is a pivotal security-related word in this sentence. As our SO
domain-knowledge specific model BERTOverflow is trained on both programming
and non-programming data, the model finds ambiguity in such a scenario and
thus makes erroneous predictions. However, RoBERTa handles the scenario in a
better way.
We found that transformers improved the performance of baseline models by a
long way for security aspect detection. This motivated us to dive deeper into
security aspects. However, we found that the performance significantly dropped
when the transformers trained on Opiner security data were tested on the IoT
dataset. We investigated this drop in performance and found that security
discussions in the IoT dataset are more implicit compared to the generic
security discussion in Opiner. Moreover, the domain knowledge that we need to
discern those security discussions is missing in the generic Opiner dataset.
For instance, the following sentence includes the IoT domain specific keyword
RFID which is an identification key: ‘Is there any way to restart the screen
after using the RFID reader?’. As such discussions are only available in IoT
discussions, transformer models that are created and trained on only for
general purposes fail to perceive the security context. As a result, the model
underperforms in the IoT domain. The performance drop in F1-Score for all the
transformers can be seen in Figure 3. We can clearly see from the charts that
the performance of BERT, RoBERTa, XLNET and BERTOverflow has dropped by 28%,
32%, 43% and 44% respectively after testing our own IoT dataset with the
transformers trained on Opiner dataset.
As our previous experiment result analysis denotes that the model is missing
IoT contextual insights, we included IoT discussions in our dataset to check
how these discussions influence the performance. We observed that adding
domain-specific data during the tuning of the pretrained model resulted in
more robust performance. We found that the performance improved over $50\%$ in
terms of F1-score. Like our previous findings, the RoBERTa model has the best
F1-Score of 0.69, followed by BERTOverflow (0.66), BERT (0.64), and XLNet
(0.54). In addition, RoBERTa also shows the best precision of 0.73 and recall
of 0.66. This again shows that RoBERTa is the most effective model for
security aspect detection in any domain.

Figure 3: A comparison of F1-Score between the experiments on Cross-validation
on Opiner dataset and Cross-Domain dataset

We compared this result with the results of previous experiments. We have
showed bar-charts in Figure 2 for all these four models. We found that all
models have lower precision than previous experimental results. However, the
models had improved their recall compared to experiment 2 but still lagged
behind experiment 1. As a result, the models have a higher F1-Score than
experiment 2 but lower than experiment 1. For example, RoBERTa has a 46%
better F1-Score than experiment 2, but 13% lower than experiment 1. Based on
this observation, we came to two conclusions regarding the detection of IoT
security aspects. First, IoT security is different from the general type of
security we discuss more often. We conclude this based on the evidence that
the same transformers that can detect security aspects reliably (Experiment-1
results) fail to perform similarly in the IoT domain (Experiment-2 results),
but the performance improves while domain knowledge is added to the model
(Experiment-3 results). Second, IoT security aspects are more complex, sparse,
and implicit than normal security aspects. The basis of this conclusion is the
evidence that even if the transformer models are adopted to IoT domain
knowledge, the models fail to perform as well as they do for general security
aspect detection (Experiment-3 results). Thus, we believe by incorporating
more IoT knowledge (i.e., increasing training samples) the performance of the
models can be further improved, which we left as our future work.

##  7 Related Work

Several recent papers have used SO posts to investigate various aspects of
software development using topic modeling, such as what developers are talking
about in general [9], or about a specific problem like big data [11],
concurrency [10], security [12][24], and trends in block chain reference
architectures [13]. All of these projects demonstrate that interest in the
Internet of Things is growing, and conversations about it are becoming more
common in online developer communities like Stack Overflow (SO). Analyzing the
presence, popularity, and complexity of certain IoT issues may be gained
through understanding these discussions. Uddin et al. [14] investigated at
nearly 53,000 IoT-related posts on SO and used topic modeling [15] to figure
out what people were talking about. On SO, Aly et al. [16] addressed at
questions of IoT and Industry 4.0. They utilized topic modeling to identify
the themes mentioned in the investigated questions, similar to the work given
by Uddin et al. [14]. Their study concentrated on the industrial issues of IoT
technology, whereas Uddin et al. [14] intended to learn about the practical
difficulties that developers experience when building real IoT systems. Recent
studies [17, 18] explored the connection between API usage and Stack Overflow
discussions. Both research discovered a relationship between API class use and
the number of Stack Overflow questions answered. But Gias Uddin [19] utilized
their constructed benchmark dataset named "OPINER" [19] to carry out the study
and noticed that developers frequently provided opinions about vastly
different API aspects in those discussions which was the first step towards
filling the gap of investigating the susceptibility and influence of
sentiments and API aspects in the API reviews of online forum discussions.
Uddin and Khomh [19] introduced OPINER, a method for mining API-related
opinions and providing users with a rapid summary of the benefits and
drawbacks of APIs when deciding which API to employ to implement a certain
feature. Uddin and Khomh [20] used an SVM-based aspect classifier and a
modified Sentiment Orientation algorithm [23] to comply with API opinion
mining. Based on the positive and negative results emphasized in earlier
attempts to automatically mine API opinions, as well as the seminal work in
this field by Uddin and Khomh [20], Lin et al. [21] introduced a new approach
called Pattern-based Opinion MinEr (POME), which utilizes linguistic patterns
preserved in Stack Overflow sentences referring to APIs to classify whether a
sentence is referring to a specific API aspect (functional, community,
performance, usability, documentation, compatibility or reliability), and has
a positive or negative polarity.

##  8 Conclusion and Future works

We found identifying security aspects in IoT-related discussions to be a
difficult task since domain-specific datasets on security-related discussions
are not commonly available. In our study, we attempted to create a one-of-a-
kind dataset in this respect, and we presented a brief comparison between our
IoT security dataset experiments and the benchmark dataset on aspect
identification. We have come to the conclusion that generalization is not
really the best method for identifying security discussions on sites like
StackOverflow. Domain-specific knowledge transfer via various transformer
models might be a superior alternative to security aspect detection. In the
future, we can incorporate other transfer learning methods to improve our
performance. The results we obtained are not quite satisfactory as our dataset
is not very large. Increasing the number of samples in the dataset is another
effort we may undertake in the future to enhance the outcome.

## References

  * [1] Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." Advances in neural information processing systems 30 (2017).
  * [2] Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805 (2018).
  * [3] Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. "Roberta: A robustly optimized bert pretraining approach." arXiv preprint arXiv:1907.11692 (2019).
  * [4] Sanh, Victor, Lysandre Debut, Julien Chaumond, and Thomas Wolf. "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter." arXiv preprint arXiv:1910.01108 (2019).
  * [5] Yang, Zhilin, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. "Xlnet: Generalized autoregressive pretraining for language understanding." Advances in neural information processing systems 32 (2019).
  * [6] Tabassum, Jeniya, Mounica Maddela, Wei Xu, and Alan Ritter. "Code and named entity recognition in stackoverflow." arXiv preprint arXiv:2005.01634 (2020).
  * [7] "Industrial IOT Connections to Reach 37 Billion Globally by 2025, as ’Smart Factory’ Concept Realised." Accessed April 9, 2022.
  * [8] Uddin, Gias, and Foutse Khomh. "Automatic summarization of API reviews." In 2017 $32^{n}d$ IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 159-170. IEEE, 2017.
  * [9] A. Barua, S. W. Thomas, and A. E. Hassan. "What are developers talking about? an analysis of topics and trends in stack overflow." Empirical Software Engineering, pages 1-31, 2012.
  * [10] S. Ahmed and M. Bagherzadeh. "What do concurrency developers ask about?: A largescale study using stack overflow." In Proceedings of the $12^{th}$ ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 30, 2018.
  * [11] M. Bagherzadeh and R. Khatchadourian. "Going big: A large-scale study on what big data developers ask." In Proceedings of the 2019 $27^{t}h$ ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, pages 432-442, New York, NY, USA, 2019.
  * [12] X.-L. Yang, D. Lo, X. Xia, Z.-Y. Wan, and J.-L. Sun. "What security questions do developers ask? a large-scale study of stack overflow posts." Journal of Computer Science and Technology, 31(5):910-924, 2016.
  * [13] Z. Wan, X. Xia, and A. E. Hassan. "What is discussed about blockchain? a case study on the use of balanced lda and the reference architecture of a domain to capture online discussions about blockchain platforms across the stack exchange communities." IEEE Transactions on Software Engineering, 2019.
  * [14] Uddin, Gias, Fatima Sabir, Yann-Gaël Guéhéneuc, Omar Alam, and Foutse Khomh. "An empirical study of IoT topics in IoT developer discussions on Stack Overflow." Empirical Software Engineering 26, no. 6 (2021): 1-45.
  * [15] D. M. Blei, A. Y. Ng, and M. I. Jordan. "Latent dirichlet allocation." Journal of Machine Learning Research, 3(4-5):993-1022, 2003.
  * [16] M. Aly, F. Khomh, and S. Yacout. "What do practitioners discuss about iot and industry 4.0 related technologies? characterization and identification of iot and industry 4.0 categories in stack overflow discussions." Internet of Things, 14:100364, 2021.
  * [17] D. Kavaler, D. Posnett, C. Gibler, H. Chen, P. Devanbu, and V. Filkov. "Using and asking: Apis used in the android market and asked about in stackoverflow." In Proceedings of the International Conference on Social Informatics, pages 405–418, 2013.
  * [18] C. Parnin, C. Treude, L. Grammel, and M.-A. Storey. "Crowd documentation: Exploring the coverage and dynamics of api discussions on stack overflow." Technical report, Technical Report GIT-CS-12-05, Georgia Tech, 2012.
  * [19] Gias Uddin and Foutse Khomh. "Mining API aspects in API reviews." Technical report, 2017.
  * [20] Gias Uddin and Foutse Khomh. Opiner: an opinion search and Gias Uddin and Foutse Khomh. "Opiner: an opinion search and summarization engine for apis." In Proceedings of the $32^{n}d$ IEEE/ACM International Conference on Automated Software Engineering (ASE 2017), pages 978–983. IEEE Computer Society, 2017.
  * [21] Lin, Bin, Fiorella Zampetti, Gabriele Bavota, Massimiliano Di Penta, and Michele Lanza. "Pattern-based mining of opinions in Q&A websites." In 2019 IEEE/ACM $41^{s}t$ International Conference on Software Engineering (ICSE), pp. 548-559. IEEE, 2019.
  * [22] Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. "Exploring the limits of transfer learning with a unified text-to-text transformer."(2019).
  * [23] Minqing Hu and Bing Liu. "Mining and summarizing customer reviews." In Proceedings of the $10^{t}h$ ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD 2004), pages 168–177. ACM, 2004.
  * [24] Nibir Mandal and Gias Uddin. (2022) "An empirical study of IoT security aspects at sentence-level in developer textual discussions."  Information and Software Technology, 150, 106970. https://doi.org/10.1016/j.infsof.2022.106970.