# Zero-Shot Learning for Requirements Classification:
An Exploratory Study

Waad Alhoshan  Alessio Ferrari  Liping Zhao

###### Abstract

Context and motivation: Requirements engineering (RE) researchers have been
experimenting machine learning (ML) and deep learning (DL) approaches for a
range of RE tasks, such as requirements classification, requirements tracing,
ambiguity detection, and modelling. Question/problem: Most of today’s ML/DL
approaches are based on supervised learning techniques, meaning that they need
to be trained using annotated datasets to learn how to assign a class label to
sample items from an application domain. This constraint poses an enormous
challenge to RE researchers, as the lack of annotated datasets makes it
difficult for them to fully exploit the benefit of advanced ML/DL
technologies. Principal ideas/results: To address this challenge, this paper
proposes a approach that employs the embedding-based unsupervised Zero-Shot
Learning (ZSL) technique to perform requirements classification. We focus on
the classification task because many RE tasks can be framed as classification
problems. In this study, we demonstrate our approach for three tasks. (1)
FR/NFR: classification functional requirements vs non-functional requirements;
(2) NFR: identification of NFR classes; (3) Security: classification of
security vs non-security requirements. The study shows that the ZSL approach
achieves an F1 score of 0.66 for the FR/NFR task. For the NFR task, the
approach yields F1$\sim 0.72-0.80$, considering the most frequent classes. For
the Security task, F1 $\sim 0.66$. All of the aforementioned F1 scores are
achieved with zero-training efforts. Contribution: This study demonstrates the
potential of ZSL for requirements classification. An important implication is
that it is possible to have very little or no training data to perform
multiple tasks. The proposed approach thus contributes to the solution of the
long-standing problem of data shortage in RE.

###### keywords:

Zero-shot learning , language models , contextual word-embeddings ,
requirements classification , zero-shot text classification , unsupervised
learning , multi-label classification , transfer learning , deep learning ,
requirements engineering , AI for requirements engineering , AI for software
engineering

###### PACS:

0000 , 1111

###### MSC:

0000 , 1111

††journal: Information and Software Technology

[inst1]organization=Imam Mohammad Ibn Saud Islamic University (IMSIU), College
of Computer and Information Sciences,city=Riyadh, postcode = 11564,
country=Saudi Arabia [inst2]organization=Istituto di Scienza e Tecnologie
dell’Informazione “A. Faedo”, Consiglio Nazionale delle Ricerche (ISTI-
CNR),addressline=Via G. Moruzzi 1, city=Pisa, postcode=56126, country=Italy
[inst3]organization=Department of Computer Science,addressline=University of
Manchester, city=Manchester, postcode=M13 9PL, country=UK

Supervised machine learning (ML) methods have some major limitations and the
most notable one is that they need to be trained using a large amount of task-
specific labelled training data.

This problem poses an enormous challenge to requirements engineering (RE)
researchers, as the lack of labelled datasets makes it difficult for them to
fully exploit the benefit of advanced learning technologies.

Transfer learning deals with this problem by transferring the knowledge gained
in solving upstream tasks to downstream tasks, but it still requires a large
amount of labelled training data to train downstream tasks, thus not
addressing the problem facing RE researchers.

Zero-shot learning (ZSL), an emerging learning paradigm, can overcome the
problem of supervised learning as it does not require any labelled training
data, thus called zero-data learning.

This study demonstrates the potential of ZSL for requirements classification.
An important implication is that it is possible to have very little or no
training data to perform multiple tasks. The proposed approach thus
contributes to the solution of the longstanding problem of data shortage in
RE.

##  1 Introduction

In requirements engineering (RE), system and software requirements
specifications are typically written in natural language (NL) [1, 2]. In the
last few years, natural language processing (NLP) techniques based on
supervised machine learning (ML), and, more recently, deep learning (DL), have
been applied to address several RE tasks, driven by the success of these
techniques in a range of domains, including medical diagnosis, credit card
fraud detection, and sentiment analysis [3, 4, 5]. The majority of RE tasks
can be framed as NLP classification tasks, to be solved with supervised ML
techniques [6]. Relevant examples are: classifying requirements into different
categories [7, 8, 9, 10]; identifying requirements from software contracts
[11]; discerning requirements and non-requirements [12]; detecting ambiguity
from regulatory requirements [13]; establishing relationships between
requirements [14]; and discovering requirements-relevant content from app
reviews [15, 16]. To date, research ML-based RE has been primarily focused on
supervised classification approaches [17]. However, supervised ML has some
major limitations. The most notable one is that supervised ML methods need to
be trained on a large amount of labelled data before they can be ready for
predicting the outcomes on new data [4, 18, 19, 20]. This problem is
exacerbated in domains like RE where collecting sufficient training data, and
labelling them, is often expensive, time-consuming, error-prone [9, 11], and
requires sufficient domain- and even project-specific knowledge [20].
Furthermore, labelled data used in previous studies are frequently
unavailable. This happens even for the lively area of app review analysis, in
which most studies have not released their labelled dataset, according to a
recent survey (cf. [15], p. 34).

Another limitation of supervised learning methods is that they can only
classify the data belonging to seen classes (i.e., classes covered by the
training data), but they cannot classify the data into previously unseen
classes (i.e., classes not covered by the training data) [18, 21]. Although
this limitation is inherent in supervised learning, the ability to deal with
previously unseen classes can bring huge benefit to many real-world
applications where classes are artificially defined, with no common consensus,
or where classes may evolve over time, with new classes emerging and old ones
becoming obsolete. One such example is requirements classification where
several classification schemes exist for non-functional requirements (NFRs)
[22, 23]. As software applications, their requirements, and the theory of NFRs
itself evolve over time, so do the classification schemes. Consequently, a
dataset labelled using one set of classes (e.g., the PROMISE NFR Dataset [24])
cannot be reused to train a method that intends to predict a different set of
classes (e.g., based on the latest ISO/IEC/IEEE 29148 standard [25]). Each
time a new classification scheme is used, the datasets must be relabelled
accordingly, incurring expensive data-labelling costs.

To address these problems, different learning paradigms have been proposed in
recent years [4]. One such paradigm is transfer learning. Based on the idea of
knowledge transfer and domain adaptation [26, 27], transfer learning aims at
improving the performance of target learners or models on target domains by
transferring the knowledge contained in different but related source domains
[28]. In doing so, transfer learning intends to alleviate the problems of data
shortages and expensive data labelling efforts.

Another paradigm is zero-shot learning (ZSL) (also known as zero-data learning
[29]). ZSL intends to overcome the aforementioned two limitations of
supervised learning, by predicting both seen and unseen classes, to classify
instances belonging to the classes that have no labeled instances [29, 30,
31].

Expanding on our preliminary study [6], this paper aims to study how Zero-Shot
Learning can be used for requirements classification and to gain insight into
this new paradigm in the context of RE. Accordingly, we intend to make the
following contributions:

  * 1.

We introduce the concept of ZSL and its relationship with transfer learning.

  * 2.

We demonstrate how ZSL can be applied to requirements classification through a
series of experiments.

  * 3.

We discuss the potential and challenges of using this new learning approach in
RE.

The preliminary study [6] only assessed ZSL on the classification of security
and usability requirements selected from a portion of the PROMISE NFR dataset.
This paper substantially extends the previous contribution by evaluating ZSL
on different tasks, namely classification of functional vs non-functional
requirements, identification of non-functional requirements classes, and
classification of security vs non-security requirements. To this end, the full
PROMISE NFR dataset and the SecReq dataset [32] are used. Furthermore, four
different language models are compared, two domain-generic ones, i.e.,
Sentence-BERT and All-MiniLM, and two domain-specific ones based on documents
from the software engineering domain, i.e., Bert4RE and BERTOverflow.

The remaining paper is structured as follows: Section 2 briefly reviews the
current ML approaches for requirements classification. Section 3 introduces
zero-shot learning. Section 4 states our research questions and describes our
experimental design. Section 5 analyzes the experimental results, while
Section 6 answers our research questions based on these results. Section 7
examines the validity threats to our experiments and our mitigation
strategies. Finally, Section 8 concludes the paper.

##  2 Related Work

Most studies in ML-based requirements classification focus on categorisation
between functional (FR) and non-functional (NFR, or “quality” [33])
requirements, and on further categorisation of different NFR classes, such as
security, performance, usability, etc. However, the distinction between FR and
NFR has been a matter of debate in the RE community [34, 9], and the empirical
study by Eckhardt et al. [23] shows that NFRs can include functional aspects.
Furthermore, there is a more fine-grained representation of FRs and NFRs given
by the ISO/IEC/IEEE 29148:2018(E) Standard [25], which distinguishes between
functional/performance, quality, usability, interface, and other classes, thus
refining the conceptualisation already elaborated by the NFR classification
from Glinz [22]. Yet, despite the lack of consensus what NFRs are, and how we
should classify and represent them, the differentiation between FRs and NFRs
is a common categorisation in RE, and in the following we will use this
distinction, keeping in mind that it is an artificial construct [23].

ML-based approaches for requirements classification were examined in a
systematic literature review by Binkhonain and Zhao [17]. One of the findings
of that review was that while the majority of proposed solutions focuses on
classification of FRs and different NFRs categories, there is a group of
solutions targets specifically at security requirements. Accordingly, our
discussion of related work will also differentiate these two groups of
studies.

###  2.1 Classification of FRs and NFRs

One of the earliest adoptions of ML to RE was due to Cleland-Huang et al. [7,
35], who proposed to use a set of indicator terms to identify different
classes of NFR. The approach was supervised, in that it first identified a set
of indicator terms on a set of manually annotated requirements, and then used
this set to classify unseen cases. The approach achieved a recall up to 0.80,
but suffered from low precision, up to 0.21. This study also introduced the
PROMISE NFR dataset [24], which has been widely used by the research
community, and it is also one of the benchmarks of our work.

To mitigate the problem of dataset annotation, Casamayor et al.[36] proposed a
semi-supervised method, based on an iterative process similar to active
learning, in which the user provided feedback to the classifier. Their
approach used Naive Bayes (NB) as a classification algorithm and the PROMISE
NFR dataset as the training set. After multiple iterations in which an
increasing number of training examples were used, they obtained a maximum
precision of above 0.80 and a maximum recall of above 0.70 on most classes
except underrepresented one. A semi-supervised learning approach was also
proposed by Younas et al. [37]. Similar to Cleland-Huang et al., indicator
terms/keywords for NFR classes were selected from the literature and the
semantic similarity was calculated between the requirement statements and
indicator keywords, using the word2vec language model [38]. The method
achieved 0.75 precision and 0.59 recall. These results were lower than those
of Casamayor et al. [36], but considerable less effort was required, as the
method did not require labelled requirements as input.

A better-known ML approach is provided by Kurtanović and Maalej [8], who
applied Support Vector Machine (SVM) for requirements classification. They
selected relevant features with an ensemble of different supervised
classifiers and achieved a precision and recall up to 0.92 for identifying FRs
and NFRs. For the identification of specific NFRs classes, they achieved the
highest precision and recall for security and performance classes with 0.92
precision and 0.90 recall. Dalpiaz et al. [9] reconstructed the study of the
work by Kurtanović and Maalej and used the results obtained as a baseline to
evaluate their proposed approach using interpretable linguistic features.

To overcome the problem of labour intensive feature engineering, Navarro et
al. [39] proposed one of the first approaches using a deep learning (DL)
model. They used a Convolutional Neural Network (CNN) model on the PROMISE
dataset, and obtained precision and recall of 0.80 and 0.79 respectively, thus
addressing the problem of limited precision observed by Cleland-Huang et al.
Similarly, Dekhtyar and Fong [40] also used CNN, together with language
representations based on pre-trained word2vec embeddings of the words found in
the requirements. They used the PROMISE dataset and obtained precision and
recall 0.93 and 0.92 respectively. However, their classification was only
binary and only identified between FRs and NFRs. More recently, Aldhafer et
al. [41] used a Bidirectional Gated Recurrent Neural Network (BiGRU) model to
classify requirements. On the PROMISE dataset they reached 0.93 and 0.95
precision precision and recall respectively for FRs vs NFRs. On the different
NFR classes, they obtain 0.78 and 0.76 precision and recall.

A more closely related work to ours is due to Hey et al. [10], who proposed
NoRBERT, a transfer learning approach for requirements classification. Their
approach is based on fine-tuning the BERT model (Bidirectional Encoder
Representations from Transformers) [42]. They achieved similar or better
results with respect to previous works, with F1-scores of up to 0.94 on the
PROMISE dataset for FR vs NFR requirements classification. NoRBERT also
outperformed recent approaches at classifying NFRs classes. The most frequent
classes were classified with an average F1-score of 0.87. The proposed
solution was also applied for the classification of different types of
functional requirements concerns in the PROMISE dataset, achieving F1-score up
to 0.92.

###  2.2 Classification of Security Requirements

One of the early works on security requirements classification was by Knauss
et al. [43], who used a Bayesian classifier to identify security-relevant
requirements on three industrial datasets. These datasets are also used in our
paper (aggregated into the SeqReq dataset). They achieved precision $>0.8$ and
recall $>0.9$. In another work, Riaz et al. [44] proposed an approach to
extract security-relevant sentences from requirements documents. They used a
dataset of 10,963 sentences belonging to six different documents from the
healthcare domain. The proposed approach was semi-automatic and based on KNN
(K-nearest Neighbours) classification. The authors achieved a precision of
$0.82$, and a recall of $0.79$.

Addressing the lack of domain-specific data sets, Munaiah et al. [45] proposed
a domain-independent classification model for identifying domain-specific
security requirements. The proposed approach, a one-class SVM classifier, was
used to identify general descriptions related to software security weaknesses,
but not the actual security requirements per se. The authors showed that the
one-class classifier achieved an average precision, recall and F1-score of
0.67, 0.70 and 0.68, respectively. Varenov et al. [46] compared the
performance of different LMs, namely BERT, XLNET, and DistilBERT, for security
requirements classification. They identified $1,086$ security requirements of
seven different classes collected from multiple existing datasets, such as
PURE [47], SecReq [43] and Riaz’s dataset [44]. Unlike previous studies, this
work aimed to classify security requirements into more fine-grained classes,
i.e., Confidentiality, Integrity, Availability, Accountability, Operational,
Access Control, and Other. DistilBERT achieved the best results, with F1-score
of 0.78.

###  2.3 Contribution

In comparison with the related work, our study aims to present a comparative
analysis of different ZSL configurations for the classification of
requirements. Similarly to the proposal of Hey et al. [10], we explore the
potential of a DL solution on the widely used PROMISE dataset. Differently
from Hey et al. [10], this is the first work in RE that proposes to use ZSL
for the classification task. While Hey et al. focus on addressing
generalisability of the classifier by means of transfer learning, our
proposal: (1) avoids the need of a tagged dataset, therefore addressing the
well known problem of the scarcity of annotated datasets in RE [1, 20, 48,
47]; (2) is inherently generalisable to different projects, thus addressing
the problem of decreasing performance with unseen projects, which typically
affects requirements classifiers [10, 9]. Concerning security requirements
classification, our proposal overcomes the problem of dataset annotation as
Munaiah et al. [45]. However, their approach is specific to security
requirements, while our proposal is more generalisable and adaptable to
different classification tasks.

##  3 Zero-Shot Learning

In this section, we introduce ZSL in the context of transfer learning and
language models (LMs). Furthermore, we show that ZSL is in fact an extreme
version of transfer learning that takes advantage of LMs.

###  3.1 Transfer Learning

The traditional supervised ML paradigm is based on learning in isolation,
where a single predictive model is trained for a task using a single dataset
[49]. This paradigm requires a large number of labelled training data and it
performs best for well-defined and narrow tasks [49]. However, this paradigm
breaks down when we do not have sufficient labelled data for the desired task
to train a reliable model. Transfer learning allows us to deal with this
problem by leveraging the knowledge gained in solving the source task in the
source domain and then transfer this knowledge to the target task and target
domain [50, 49, 26, 27].

Transfer learning can be broadly classified into two types, based on the
relationship between the source and target tasks or domains: inductive
transfer learning refers to the setting where the target task is different
from the source task while the source and target domains can be the same or
different; transductive transfer learning refers to the setting where the
source and target tasks are the same, but the source and target domains are
different [26].

Currently, the most promising transfer learning approach is sequential
inductive transfer learning [49], where a task is learned in sequence,
consisting of a pretraining and a adaptation phase. In the pretraining phase,
a source model is normally trained on a general representation using large
quantities of data that can be adapted to a wide range of target tasks in the
adaptation phase [50]. Normally, model pretraining uses unsupervised learning
on unlabeled data whereas model adaptation uses supervised learning on task-
specific labelled data [50, 49, 51].

Ideally, pretraining should aim to train a source model with general-purpose
abilities and knowledge that can then be transferred to a wide range of target
tasks [52]. The pretraining phase is therefore generally (much) more expensive
than the adaptation phase [50, 51].

###  3.2 Language Models

Transfer learning is a central concept in language models (LMs) [53, 52]. A LM
is a language representation developed for deep learning based NLP tasks. LMs
are typically pretrained unsupervised on unlabeled text data available en
masse through the Internet [52]. They can then be applied — i.e., transferred
— to different downstream tasks [54]. Today, transfer learning with LMs have
achieved state-of-the-art results across a wide range of NLP tasks (e.g.,
sentence and token prediction [42], named entity recognition [55], machine
translation [56] and text classification [57]).

There are two general approaches for transferring pretrained LMs to downstream
tasks: feature-based and fine-tuning [42]. The feature-based approach, such as
ELMo [55], uses a pretrained model as the input features for the downstream
task without touching the pretrained model; the representations resulted from
such a LM are used to enrich the features used in the downstream supervised
learning. On the other hand, the fine-tuning approach aims to modify the
weights and parameters in certain layers of the LM in order to train the LM to
perform a specific NLP downstream task such as classification, sentiments
analysis, and question-answering; for example, LMs such as OpenAI GPT [58] and
BERT [42], can be trained to perform downstream tasks by fine-tuning all or
part of the parameters on these pretrained models.

In the last few years, word embeddings — continuous vector representations of
words and phrases based on word co-occurrences in large corpora — became a
popular method for language representations [38]. Word embeddings capture
syntactic and semantic characteristics of the words and have shown to be very
effective on word similarity tasks [59]. The early word embedding models
include Skip-gram [38], Word2Vec [60], GloVe [61], and FastText [62]. More
recently, LMs such as BERT, ELMo, and GPT-2, offer an advantage over the early
embedding models. In contrast to fixed representations produced by the early
models, LMs produce word embeddings that are dynamically informed by the words
around them, known as contextual word embeddings [63]. Contextual word
embeddings can tell if the word “mouse” refers to a computer device or a small
rodent from the surrounding context where this word is mentioned.

###  3.3 Zero-Shot Learning: Concepts and Approaches

ZSL has been originally used in image processing to predict unseen images
[64], and has recently been adapted to text classification to predict unseen
classes [65]. ZSL has shown great potential in various NLP tasks, such as
entity recognition [66], relation extraction [67], document classification
[68], and text classification [65]. Unlike supervised classification models
that are trained to classify a text as one of the possible classes, the ZSL
models predict whether a text is related to a label or not. Thus, ZSL treats a
classification task (binary, or multi-class) as a problem of finding
relatedness between texts (e.g., requirements) and labels (classes).
Furthermore, unlike the aforementioned transfer learning with LMs, ZSL simply
applies LMs without fine-tuning or feature extraction, thus taking transfer
learning to the extreme [54].

According to the literature, there are two approaches to ZSL: entailment-based
and embedding-based. The entailment-based approach was initially proposed by
[69], which treats classification task as a natural language inference (NLI)
task. In the context of ZSL text classification, the entailment-based approach
teats an input text sequence as a premise and the labels as a hypothesis, and
then infers if the input text is an entailment of any of the labels or not
[69]. For example, given the sentence as a premise “the system must be
deployed on Azure” and the label as hypothesis string “this is about software
maintenance”, the entailment-based classifier provides a score which is then
translated into one of the following outputs: entailment (yes), contradiction
(no), or undecided. This ZSL approach requires a large inference-based LM that
can interpret the entailment relation between an input sequence and a label.

The embedding-based ZSL approach was introduced by Veeranna et al. [59]. Under
this approach, both class labels (e.g., Usability, Security) and input text
are represented as word sequences using word embeddings. Text classification
then involves computing the semantic similarity between each label sequence
and the text sequence. If the similarity score is greater than a certain
threshold, then the document can be classified into a specific category
represented by the label; otherwise, the text document does not belong to that
category. Note that as a label is treated as a sequence of words, it can
contain any number of words or their combinations.

Due to the simplicity of the embedding-based approach to ZSL, we have applied
it in our study. However, unlike the original proposal by Veeranna et al.
[59], who used the static word-embedding technique (Skip-gram) for word
representation, we take advantage of state-of-the-art LMs and use them to
produce contextual word-embeddings for both labels and input text. In so
doing, our embeddings not only capture syntactic and semantic characteristics
of the words, but also their context.

Another difference between our approach and the original one is that we do not
use similarity thresholds to determine the predicted label; instead, we treat
all text classification as a multi-label classification task and rank-order
all the labels with their similarity scores. For a binary or multi-class
classification task, we select the label with the highest similarity score as
the predicted label. For a multi-label task, we check the top-$n$ labels to
ensure we do not miss the predicted labels.

The contextual word embedding-based approach adopted in our study is depicted
in Figure 1. We illustrate this approach with a simple example. Suppose we
have a requirement: “CNG shall support mechanisms for secure authentication
and communication with the remote management system”. We want to classify this
requirement according to these two labels: “Usability” and “Security”. We feed
these three word sequences (a requirement plus two labels) into a LM to
produce three contextual word embeddings. We then compute the $Cosine$
similarity111The $Cosine$ similarity function is the standard way to compute
semantic similarity between a label and a text [60, 61, 62, 63]. between each
label and the requirement and obtained two similarity scores: $0.86$ for
Security and $0.25$ for Usability. Based on these scores, we deduce that the
given requirement is a security requirement.

Figure 1: The Contextual Word Embedding-Based ZSL Approach.

Clearly, the accuracy of this approach highly depends on the choice of 1) the
labels and 2) the LMs. The above example only shows single word labels, but to
fully exploit the benefit and potential of ZSL and contextual word-embeddings
for requirements classification, our study will investigate different label
configurations. For example, by composing the Usability label as “instructive,
easy, helpful, useful,learnable, explainable, affordable, intuitive, or
understandable”, the LM can produce a more meaningful embedding that can
capture a range of connotations of the Usability requirement, rather than just
its face value. Our study will explore different LMs, to find out which one is
most effective for requirements classification.

##  4 Experimental Design

###  4.1 Research Questions

The purpose of this paper is to provide insights into how embedding-based ZSL
can be used for requirements classification. In particular, we want to
understand what are the most suitable LMs to use, in which way the naming of
the labels used for requirements classification influences the results, and
what are the performance of ZSL compared to previous approaches. Accordingly,
this paper aims to answer the following research questions (RQs):

RQ1 Which language model achieves the best performance for zero-shot
requirements classification?
In embedding-based zero-shot classification, the LM influences the
representation of the requirements. To answer this RQ, we want to compare how
generic and domain-specific LMs perform when used in zero-shot requirements
classification. The answer to this question will indicate what are the most
suitable LMs.

RQ2: To what extent can different label configurations affect the performance
of zero-shot requirements classification?
As the embedding-based ZSL approach aims to find semantic similarity between a
requirement sentence and a label, the choice of words for labels influences
the performance of the zero-shot classifier. The answer to this RQ will
suggest which label configuration can improve the performance of the
classifier.

RQ3: Is zero-shot learning effective for requirements classification?
This involves a comparison of the best results obtained in our study, against
the best results obtained with the performance of previous proposals available
in the literature. With this RQ, we also aim to reflect on the practical
implications and lessons learned throughout the experiments.

To answer these RQs, we conduct a series of experiments based on the ZSL
approach described in Section 3 Each experiment is designed with a specific LM
to perform different requirements classification tasks using relevant datasets
and label configurations. Our experimental design involves five steps,
detailed in the following sections: selection of datasets and tasks; LM
selection; label configuration; performance measure selection; experiments.

These steps are detailed in the subsections below.

###  4.2 Dataset and Task Selection

We select the following two datasets for our experiments:

  * 1.

PROMISE NFR dataset [24], introduced by Cleland Huang et al. [7]: This dataset
contains 625 requirements, partitioned into 255 FRs, and 370 NFRs. The NFRs
are further partitioned into 11 different classes, namely: A = Availability
(21 requirements), L = Legal (13), LF = Look and feel (38), MN =
Maintainability (17), O = Operational (62), PE = Performance (54), SC =
Scalability (21), SE = Security (66), US = Usability (67), FT = Fault
tolerance (10), and PO = Portability (1). These classes are unevenly
distributed, ranging from 67 requirements for Usability to one for
Portability. Each of the large classes - Usability, Security, Operational, and
Performance classes - has more than 50 examples, while the small classes -
Fault Tolerance, Legal, Maintainability and Portability - have from one to 17
requirements each. The dataset has been widely used in the literature, e.g.,
by Kurtanović and Maalej [8], and by Hey et al. [10].

  * 2.

SecReq dataset [32], introduced by Knauss et al. [43]: This dataset contains
510 requirements, made of security-related requirements (187) and non-security
related requirements (323). The requirements were collected from three
projects: Common Electronic Purse (ePurse), Customer Premises Network (CPN),
and Global Platform Spec (GPS). The dataset has been used, e.g., by Varenov et
al. [46].

We select the following representative classification tasks for our study:

  * 1.

Task FR/NFR—Binary Classification of FRs vs. NFRs. With this task we aim to
distinguish FRs from NFRs, assuming that a requirement belongs to either a FR
or a NFR class. We use the PROMISE NFR dataset for this task.

  * 2.

Task NFR — Binary, Multi-class and Multi-label Classification of NFRs. This
task aims to classify different types of NFRs based on the 10 different
classes of the PROMISE NFR dataset (we excluded the Portability class as it
only has one single sample in PROMISE dataset). We perform three sub-tasks to
understand how ZSL reacts to different ways of classifying NFRs: 1) binary
classification which discerns if a NFR belongs to a particular class or not;
2) multi-class single-label classification (simply, multi-class
classification) which assigns a NFR to one of the top or all NFR classes; 3)
multi-class multi-label classification (simply, multi-label classification),
which allocates a NFR to one or more NFR classes. The purpose of the third
sub-task is to check if the top-n NFR classes returned by the ZSL classifier
correlate with the assigned NFR label in the datatset.

  * 3.

Task Security —Binary Classification of security related vs. non-security
related requirements. This task assumes that a requirement belongs only to one
of these two classes: security related and non-security related. We use the
SeqReq dataset for this task.

These datasets and tasks are selected for our experiments as they are
frequently considered in the literature (cf. Sect. 2) and will enable us to
compare our results directly with those obtained in previous work. Table 1
summarizes the above tasks and their associated datasets.

Table 1: Classification Tasks Performed in our Study and Corresponding
Datasets

| Task
---
Aim |  | Type
---
Dataset (size)
Task FR/NFR |  | To detect FRs from NFRs.
---
Binary | \- PROMISE (625)
Task NFR |  | To detect a specific NFR class (e.g., usability) other NFRs.
---
Binary |  \- Top-NFR PROMISE (249) \- PROMISE (369-370)
| To classify NFRs to one NFR class.
---
Multi-class
| To classify NFRs into one or more NFR classes.
---
Multi-label
Task Security | To detect security requirements. | Binary |  | \- SeqReq (510) - GPS (176)
---
\- CPN (210) - ePurse (124)

###  4.3 Language Model Selection

We select the following four BERT-based LMs for our study: two generic and two
SE domain-specific. We focus on BERT-based models, due to the popularity of
BERT models and the suitability of these models for requirements
classification [10]. Other state-of-the-art LMs, such as GPT-2 and GPT-3 by
OpenAI, and XLNet by Carnegie Mellon University [70], have not been included
in our study, as they are not suitable for requirements classification. BERT
was considered the most suitable solution for a requirements classification
task as ours. For example, GPT-2 and GPT-3 are mainly oriented to generation-
related tasks, such as language translation and text summarization [71]. XLN-
Net is for tasks that involve long contexts, e.g., paragraphs [70], while
requirements are typically sentences.

The first two LMs are generic, made available by
HuggingFace222https://huggingface.co, the well-known NLP community sharing LMs
and other resources. The remaining two are domain-specific for requirements
and software engineering. These models are:

  * 1.

Sentence-BERT (Sbert): This generic LM is proposed by Reimers and Gurevych
[72] as a fine-tuned version of BERT LM to enrich the semantic embedding
representation, i.e., to aid in deriving semantically meaningful sentence
embeddings that can be compared with the unseen labels using the cosine-
similarity measure. The solution overcomes the drawbacks of using BERT, for
which a sentence embedding is typically computed as averaging BERT output
layer for the token in the given sentence. This approach was observed to lead
to poor embeddings, and thus semantic representations of
sentences/requirements [72]. The implementation of Sbert LM is provided at
HuggingFace models hub by deepset.ai
contributors333huggingface.co/deepset/sentence_bert.

  * 2.

All-MiniLM-L12 (AllMini): This generic LM is introduced by Wang et al. [73],
from Microsoft Research. It overcomes the complexity of BERT, which requires
substantial resources and it is often not applicable for real-life
applications in which reduced latency is required. The LM uses an approach
that reduces (distils) the BERT LM, while preserving its performance. It is
specifically targeted to sentence embeddings, and therefore suitable for
requirements encoding to provide contextual embeddings. In this experiment, we
use a fine-tuned version which is pretrained on 1B sentence pairs dataset
(All-MiniLM v2), which used as a part of HuggingFace’s project for encoding
sentences and short paragraphs444huggingface.co/sentence-transformers/all-
MiniLM-L12-v2. The LM proves its efficiency for semantic search and sentence
clustering tasks.

  * 3.

BERT4RE: Developed by Ajagbe and Zhao [74], this RE domain-specific LM was
trained on BERTbase model with more than seven million words from different RE
related datasets, including the PROMISE NFR dataset, the PURE dataset [47],
and app reviews from Google Playstore and App Store. Although BERT4RE aims to
support a wide range of RE tasks, it has only been tested on the task of
identifying semantic roles from requirements documents. As this is the only
publicly available RE-specific LM, we include it in our study. The Bert4RE LM
is provided by the authors at Zenodo repository 555zenodo.org/record/6354280.

  * 4.

BERTOverflow (SObert): This SE domain-specific LM was developed by Tabassum et
al.[75] and was trained on 152 million sentences from StackOverflow.
BERTOverflow shares the same vision as BERT4RE, aiming to overcome the problem
of generic LMs such as BERT, which may not be able to fully capture the
semantics of the SE terminology. Although BERTOverflow has been trained to
perform domain-specific named entity recognition (NER) tasks in SE, our
research shows that it is among the few software engineering-specific LMs that
can be potentially adopted requirements classification666Note that another LM
for SE is CodeBERT [76], but this is trained on both natural language and
source code, and it is oriented to tasks that involve these two
representations, such as code retrieval based on natural language queries..
The implementation of the SObert can be found at HuggingfFace models hub
777huggingface.co/jeniya/BERTOverflow.

Table 2: Technical Specifications of the Selected LMs.

LM | Domain | Layers |  | Hidden
---
size
| Attention
---
heads
Vocab size |  | Embedding
---
size
Sbert |  | 12 | 768 | 12 | 30,522 | 128
AllMini |  generic | 12 | 384 | 12 | 30,522 | 512
Bert4RE |  | 12 | 768 | 6 | 32,000 | 512
SObert |  RE/SE specific | 12 | 768 | 12 | 82,000 | 512

##### Technical Specifications

The technical specifications of the selected LMs are listed in Table 2. As
shown in the table, all the LMs share the same underlying LM architecture,
namely BERT, with an embedding size of 512 tokens, except for Sbert which only
embeds sentence with only 128 tokens as a maximum length. The embedding size
is not a major concern in our experimentation settings since the average
length of requirements in the selected datasets is lower than the specified LM
embedding sizes. For example, the longest requirement is found in SeqReq
dataset with a total of 94 words which is less than the listed embedding
sizes. For the vocabulary size, all of the LMs share approximated number of
vocab ($\sim$ 30k words), except for the domain specific SObert which holds
more than 80k words in its context files. The vocab size does not affect the
ability of these LMs in detecting single words, as any word that does not
occur in the vocabulary is splitted into sub-words, for example in Sbert the
word usability is processed as: us`#``#` and `#``#`ability . This processing
enables the contextual embedding within these LMs to perform well even with
out-vocabulary words, unlike static word embedding such as word2vec. The other
specifications in the table are related to the attention-mechanism in BERT-
based architecture. Typically in BERT-based related LMs, the number of
transformers layers are 12, and each layer contains 12 attention heads. These
attention heads aim to compute the interrelationships between input tokens at
each layer, where each layer processes the tokens representation using the
attention heads and the output of one layer is an input of the next layer. For
instance, with 12 attention heads and 12 layers, each token in a given input
is analysed by these heads on 12 distinct interrelationships of other tokens,
i.e., how likely that token is related to the other tokens, and this process
is repeated at each one of the 12 layers. Therefore, the attention head
computes the probabilities of different kind of constituent combinations
between tokens. This extensive computational process enriches the contextual
understanding of words and sentences in these LMs. As shown in the LMs
specification table, all the selected LMs share similar attributes to process
the tokens, with 12 attention heads and layers. The dissimilarity is only
encountered with AllMini with a hidden size of 384 (i.e., output
representation size has 384 dimensions, also known as the width of layers
output) less than the hidden size of other selected LMs (768 as hidden size).
The other variation can be observed with Bert4RE, with only 6 attention heads
compared to 12 heads in the other selected LMs. These variations might have an
impact on the contextual representations which could lead to variant
representations of requirement inputs across these selected LMs. [77].

It is worth remarking that the vector-based representations of sentences,
requirements, and labels that are derived from these transformer-based LM are
very different from more traditional LM, such as word2vec and gloVe. In
particular, transformer-based LM are able to identify the deep meaning of
sentences, while traditional static LMs, such as Word2Vec word embedding,
simply expand words in a sentence with related ones. In practice, while with
traditional word embeddings a sentence like “this is not about usability” is
mapped into a vector that is very similar to “this is about usability” (they
differ solely by one word), with transformer-based, contextual, LMs the
vectors are different, since the meaning is actually the opposite. This
characteristics naturally plays a crucial role for requirements analysis, as
requirements sentences often use a very restricted vocabulary [47], but convey
different meanings.

###  4.4 Label Configuration

Different labelling strategies, and combinations thereof, are used to define
the strings that represent the labels of each class.

  * 1.

Original labels are the labels derived from the original class names used in
the datasets. For example for the task FR/NFR, the original label for the
class FR is “functional”, while for NFR we use two types of labelling: 1) the
expression “not about functional”888Preliminary experiments have shown that
this term is more effective for ZSL with respect to “non functional”. or 2) a
string including all the NFR class names (“usability, security, availability,
…”). Though the resulting labels are different, the strategy is common, as no
additional knowledge is used to derive the labels, except for the information
coming from the names used in the datasets.

  * 2.

Expert curated labels are the labels selected by the three authors of this
paper via brainstorming, and based on their understanding of the requirements
classes. The goal of the brainstorming activity was to produce a limited set
of relevant terms that represent the fundamental meaning of the class, and
which that could be combined in a single string representing the label. For
example, for the task FR/NFR, the label for the class FR is “functional,
system, behavior, shall or must”.

  * 3.

Word-embedding generated labels are the terms extracted and selected by
considering terms that are similar to the class name, according to a word-
embedding LM learned from the text of Wikipedia pages belonging to the
Computer Science (CS) portal. The idea is that the LM learned from the CS
portal represents the meaning of words in the CS domain, and it is therefore
more suitable than a generic LM in providing similar terms for our CS context.
The model was learned based on the approach and code by Ferrari and Esuli
[78]. To produce a label for a certain class, we apply the following
procedure. Given the LM, and given a class name (e.g., security), we perform
the following steps: (1) identify the top-n most similar terms according to
the LM; (2) each of the three author independently tags the terms with yes =
the term is representative for the class, no = the term is not representative
for the class, or maybe = the term could be representative for the class; (3)
each author revised their maybe as yes or no, also considering the annotations
of other authors. (4) the final tags are decided through majority voting
(e.g., yes, yes, no $\rightarrow$ yes; no, yes, no $\rightarrow$ no). After
this procedure, the terms that were tagged with yes were included in the
string that represents the label, e.g., for security, the label is
“vulnerability, securing, protecting, protection, cybersecurity, assurance,
cyber, countermeasure, threat, privacy, authentication, prevention, or
confidentiality”. To assess the consistency of annotations between the three
authors at step (2), we evaluate the overall percentages of agreement (i.e.,
all annotators tagged the term with yes or no), partial agreement (i.e., two
out of the three annotators tagged the term with yes or no), and disagreement
(i.e., the three annotators selected three different tags: yes, no, and
maybe). In addition, after resolving the tags maybe at step (3), we consider
the inter-rater agreement (IRR) based on Krippendorff’s alpha test [79] as
well as Fleiss’s Kappa [80]. These statistical tests are used to measure the
level of agreement among multiple annotators and categories. The
interpretation of test results follows the guidelines reported in the Koch
Kappa benchmark [81].

It should be noticed that these are general strategies adopted, which are then
combined and instantiated for each specific task. The different combinations
of labels produced, and variants of the strategies are reported in Sect. 5 for
each task. Not all possible combinations are tested and reported, but only
those that were incrementally defined based on the results obtained from the
application of single strategies.

###  4.5 Performance Measures

The results from each task, with each LM and label configuration, are measured
using the standard metrics of precision (P), recall (R), and F1-score (F1).
Particularly, for the binary classification task, we apply the weighted
measures of P, R and F1 (respectively represented as wP, wR and wF1) to take
into account for unbalanced classes, and enable comparison with other studies
in requirements classification [10, 8]. We also report the same unweighted
measures for each class, to study the classifier with best performance in
order to discuss possible issues that may not emerge from the weighted
measures. For the multi-class and multi-label classification tasks, we report
the standard P, R and F1 by class for the best performing classifiers, then we
report the overall weighted F1 scores across all classes based on the
distributions in the test dataset.

###  4.6 Experimental Setup

Each experiment is combination of one LM and one label configuration (i.e.,
the settings of ZSL classifier), applied on a dataset to address a task. We
use the Transformer 999huggingface.co/docs/transformers API Python package to
import and prepare the selected LMs with their transformer-like tokenizers,
and we use torch.nn module
101010pytorch.org/docs/stable/generated/torch.nn.CosineSimilarity.html in
PyTorch to compute the cosine similarity score between two tensors (i.e., the
PyTorch tensor objects obtained from the contextual representation by the
selected LMs). For reporting the results in terms of P, R, and F1 rates we use
SciKit package, classification report. In more details, the implementation
steps of the embedding-based ZSL text classification approach are:

Preparation. We import the pretrained transformer-like model with its
tokenizer–-keeping the model configuration without modification. Then, we
import the test dataset, and prepare the sequence X (which is the requirement
to be labelled) and the label names Y (as a list of string labels) to be
processed by the LM.

Encoding. We tokenise the input sequence (X) and and each label in the list
(Y) using the model’s tokenizer to prepare the input tokens to be processed
later by the LM. If length of the sequence and the length of the labels is not
the same, a zero-padding will be set to fill in the gaps and to make both
lists equal in length. The output of the encoding step is a PyTorch tensor
object for each input — that is, the output of the model’s tokenizer has two
parts: 1) input IDs, and 2) attention mask. Input IDs are obvious: these are
simply mappings between tokens and their respective IDs. The attention mask is
to prevent the model from looking at padding tokens. Then, we pass the input
IDs with their corresponding attention mask to the model to get the
contextualized embeddings. This achieved by retrieving the first layer result,
i.e., ‘the contextualised word embedding’, to get the sequence and labels
representations.

Classification. We apply the cosine similarity measure from PyTorch between
the sequence embedding and each label embedding. The output is a list of
similarity scores to each label as in multi-label classification. Then, we
sort the calculated similarity scores from highest to lowest, and select the
highest one as the predicted label in all both binary and multi-class
classification tasks, and for the multi-label classification task, we retrieve
top-n labels as selected labels.

##  5 Experimental Results

In this section, we report the results by task. For each task, we discuss the
labels configurations, and the results.

###  5.1 Task FR/NFR

For the FR/NFR task we perform a binary classification, which aims to classify
a requirement as either FR or NFR, where “or” is considered exclusive.

####  5.1.1 Label Configuration

Table 3: Label configurations for Task FR/NFR.

Label Abbr. | Label Configuration | FR Label | NFR Label
---|---|---|---
FR_A | Original 1 | “functional” | “not about functional”
FR_B | Expert curated |  | “functional, system, behavior,
---
shall, or must”
| ”not about functional, system, behavior,
---
shall, or must”
FR_C |  | Word embedding
---
(selected from top 20 words) +
Expert curated
| “functional, system, behavior,
---
shall, must, procedural, structural,
or characterize”
| “not about functional, system, behavior,
---
shall, must, procedural, structural,
or characterize”
FR_D | Original 2 | “functional” |  | “usability, security, availability, legal, look & feel,
---
scalability, fault tolerance, performance,
operational, maintainability, or portability”
FR_E |  | Expert curated +
---
Original 2
| “functional, system, behavior,
---
shall, or must”
| “usability, security, availability, legal, look & feel,
---
scalability, fault tolerance, performance,
operational, maintainability, or portability”
FR_F |  | Word embedding
---
(selected from top 20 words) +
Expert curated + Original 2
| “functional, system, behavior,
---
shall, must, procedural, structural,
or characterize”
| “usability, security, availability, legal, look & feel,
---
scalability, fault tolerance, performance,
operational, maintainability, or portability”

The label configurations for the FR/NFR Task are reported in Table 3. The
labels consist of two groups, one group to represent the FR class, and the
other the NFR class. Six configurations are used, which combine the different
strategies discussed in Sect. 4.4. We did not apply all the possible
configurations derived from the strategies, and in particular we did not apply
the word embedding strategy alone. Indeed, the word-embedding LM did not lead
to representative words related to the term term “functional”, with the
exception of the terms “procedural”, “structural” and “characterize”. This is
due to the abstract nature of the the word “functional”, which is hard to
associate to concrete words to be used to enrich the labels. For this reason,
the word embedding strategy, considering the top-20 terms, is used solely in
combination with other strategies. In the selection of the word-embedding
generated terms, the annotation procedure produced the following statistics.
75% perfect agreement, 25 % partial agreement, and 0% disagreement. We also
computed the IRR, and we obtained 0.41 as Krippendorff’s alpha and a Fleiss’
kappa score of 0.40, and both test results indicate a moderate agreement
between the three annotators.

Concerning the other strategies, it is worth remarking the usage of
“functional” vs “not about functional” (strategy, FR_1 Original 1). This type
of strategy, in which the orginal label is negated with the prefix “not about”
is also applied for the NFR class label of FR_B and FR_C, and will be applied
also later on in this paper to represent the negation of a class in a binary
classification. The choice of this expression was made after preliminary
experiments using different forms of negation, e.g., “no”, “not”, which led to
lower performance.

####  5.1.2 FR vs NFR Binary Classification

Table 4: Overall classification results for Task FR/NFR for each ZSL Classifier, which is a combination of a specific LM and a specific label configuration. Bold values indicate the best results for a specific ZSL classifier; underlined values indicate the overall best performance across all the classifiers; values in italics indicate a possibly misleading performance, related to the imbalance of F1 between classes. ZSL Classifier | wP | wR | wF1
---|---|---|---
Sbert + FR_A | 0.52 | 0.43 | 0.35
Sbert + FR_B | 0.60 | 0.46 | 0.37
Sbert + FR_C | 0.64 | 0.48 | 0.40
Sbert + FR_D | 0.54 | 0.55 | 0.54
Sbert + FR_E | 0.71 | 0.66 | 0.66
Sbert + FR_F | 0.69 | 0.64 | 0.64
AllMini + FR_A | 0.56 | 0.55 | 0.55
AllMini + FR_B | 0.53 | 0.48 | 0.46
AllMini + FR_C | 0.54 | 0.45 | 0.39
AllMini + FR_D | 0.63 | 0.59 | 0.59
AllMini + FR_E | 0.60 | 0.53 | 0.51
AllMini + FR_F | 0.66 | 0.56 | 0.53
Bert4RE + FR_A | 0.35 | 0.59 | 0.44
Bert4RE + FR_B | 0.52 | 0.56 | 0.51
Bert4RE + FR_C | 0.58 | 0.56 | 0.57
Bert4RE + FR_D | 0.59 | 0.56 | 0.56
Bert4RE + FR_E | 0.59 | 0.52 | 0.51
Bert4RE + FR_F | 0.57 | 0.47 | 0.42
SObert + FR_A | 0.35 | 0.59 | 0.44
SObert + FR_B | 0.55 | 0.59 | 0.51
SObert + FR_C | 0.58 | 0.59 | 0.58
SObert + FR_D | 0.53 | 0.58 | 0.48
SObert + FR_E | 0.60 | 0.56 | 0.56
SObert + FR_F | 0.60 | 0.55 | 0.54

Table 4 reports the overall classification results for all LMs and labelling
strategy combinations. In bold, we highlight the best combination for each LM.

The overall best combination is Sbert + FR_E, achieving a wF1 score of 0.66,
with wP = 0.71 and wR = 0.66. This indicates that the domain agnostic Sbert
model, designed to provide a semantic-laden representation for generic
sentences, substantially outperforms the other models for this task.
Furthermore, the best labelling strategy for Sbert is FR_E, i.e., the one that
uses the Expert curated labels + Original labels, which identifies the NFRs
using the names of the NFR classes (Usability, Security, Availability, etc.).

Table 5: Classification results for FR and NFR classes on Task FR/NFR, obtained from the best combination for each LM and label configuration. Bold values indicate the overall best classification results for each performance measure. ZSL Classifier | FR (255) | NFR (370)
---|---|---
P | R | F1 | P | R | F1
Sbert + FR_E | 0.55 | 0.82 | 0.66 | 0.82 | 0.54 | 0.65
AllMini + FR_D | 0.50 | 0.71 | 0.58 | 0.72 | 0.50 | 0.59
Bert4RE + FR_D | 0.47 | 0.65 | 0.54 | 0.67 | 0.50 | 0.57
SObert + FR_C | 0.50 | 0.41 | 0.45 | 0.64 | 0.72 | 0.68

Looking at Table 5, we can see how the performance is divided between FR and
NFR classification. We see that the model tends to have higher precision on
NFR (P = 0.82), and higher recall on FR (R = 0.82). This is an interesting
results, as FRs are less frequent in the dataset (255 FR, 370 NFR), and one
would expect to have the opposite result. Indeed, the most frequent class is
typically returned more frequently in ML approaches, as it happens, e.g., for
NorBERT (cf. Hey et al. [10], Table III of their paper). This phenomenon
occurs also for the other best configurations of LMs. This highlights a
characterising element of ZSL: the performance does not depend on the size of
the dataset for each class, because no actual learning is performed on the
tagged data.

The second-best model is still a domain-agnostic one, AllMini + FR_D, with wF1
= 0.59, i.e., 0.7 points lower than the first-best. The labelling scheme is
only slightly different, as for FR we use solely the label “functional”,
whereas for NFR we use the same set of labels as for the fist-best case.
Overall, with the exception of SObert—third-best model—all models have their
best performance when using this labelling strategy for NFR.

In Table 4, we also highlighted in italic style those classifiers whose
performance could be misleading. These classifiers are: BERT4RE and SObert
with FR_A as a label configuration. Both classifiers achieved a modest F1
score, which is not the lowest, and obtained wF1=0.44, wR=0.59 and wP=0.35.
The performance rates do not reflect the suitability of these two classifiers,
as they tend to classify all the input requirements into one class which is
NFR (the most frequent class in the dataset). This has no relation to the
over-fitting learning issue, since these classifiers are not trained-on or
learned-from any features of the given dataset. However, we can attribute
these poor results to the fact that FRs might have several terms and
expressions intersecting with NFRs, and domain-specific LMs, which are trained
on smaller dataset with respect to generic LM, can encounter difficulties in
distinguishing them. Consider these two example requirements from the PROMISE
dataset: Any disputes cases that have been closed for over 6 months must be
purged from the online disputes database and All actions that modify an
existing dispute case must be recorded in the case history which they are
tagged with FR and NFR labels, respectively, but share several terms. Using
original labels, as in FR_A, the distinction is hard to tackle by a domain-
specific LM compared to the generic LMs (i.e., Sbert and AllMini) which
managed to correctly classify the above-mentioned requirement examples. This
issue could be potentially solved with a more project-specific labelling
strategy for functional requirements (i.e., choosing terms that characterise
functional requirements in the specific project).

###  5.2 Task NFR

In this task, we performed three classification sub-tasks: A binary
classification to detect a specific NFR category (e.g., “usability” vs
“other”), a multi-class classification to classify a requirement into one
class out of a set of NFR classes and a multi-label classification, in which
each requirement is associated with a ranked list of NFR classes, and we want
to see if the correct label is in the top-k classes. This last approach can be
applied in a semi-automatic classification context, in which the requirement
analyst is proposed the two top-k classes, and they are asked to select the
correct one. For all the sub-tasks, we evaluate the results: 1) considering
only requirements in the largest classes, namely security, usability,
performance and operational, which include the majority of the requirements;
2) considering all the classes, except the portability class, which includes
one requirement only. For the multi-label classification case, we consider
$k=2$ when only the largest classes are considered, and $k=3$ when all the
classes are considered.

####  5.2.1 Label Configuration

Table 6: Label configurations for Usability and Security classes for Task NFR,
binary classification case.

Label Abbr. |  Label Configuration |  NFR Label |  “Other” Label
---|---|---|---
|  |  |
Usability |  |  |
US_A |  Original 1 |  “usability” |  “not about usability”
|  |  |
US_B |  Expert curated |  | “instructive, easy, helpful, useful,
---
learnable, explainable, affordable,
intuitive, or understandable”
| “not about instructive, easy,
---
helpful, useful, learnable,
explainable, affordable,
intuitive, or understandable”
|  |  |
US_C |  | Word embedding
---
(selected from top 20 words )
| “accessibility, aesthetic, contextual,
---
experience, satisfaction, HCI, UX,
questionnaire, ease, or ergonomics”
| ”not about accessibility, aesthetic,
---
contextual, experience, satisfaction,
HCI, UX, questionnaire,
ease, or ergonomics”
|  |  |
US_D |  | Word embedding
---
(selected from top 20 words) +
Original 2
| “accessibility, aesthetic, contextual,
---
experience, satisfaction, HCI, UX,
questionnaire, ease, or ergonomics”
| “security, performance, operational,
---
look feel, legal, fault tolerance,
maintainability, scalability, availability,
or portability”
|  |  |
US_E |  | Word embedding
---
(selected from top 50 words)
\+ Original 2
| “accessibility, aesthetic, contextual,
---
experience, satisfaction, HCI, UX,
questionnaire, ease, ergonomics,
designer, evaluate, multimodal,
practitioner, prototyping, preference,
personalization, suitability, focus,
clarity, responsiveness, judgement,
feel, or helpful”
| “security, performance, operational,
---
look feel, legal, fault tolerance,
maintainability, scalability, availability,
or portability”
|  |  |
Security |  |  |
SE_A | Original 1 | ”security” | “not about security”
SE_B | Expert curated | “security, authorization, or protection” | “not about security, authorization, or protection”
SE_C |  | Word embedding
---
(selected from top 20 words)
| “vulnerability, securing, protecting,
---
protection, cybersecurity, assurance,
cyber, countermeasure, threat, privacy,
authentication, prevention, or confidentiality”
| “not about vulnerability, securing, protecting,
---
protection, cybersecurity, assurance,
cyber, countermeasure, threat, privacy,
authentication, prevention, or confidentiality”
SE_D |  | Word embedding
---
(selected from top 20 words) +
Original 2
| “vulnerability, securing, protecting,
---
protection, cybersecurity, assurance,
cyber, countermeasure, threat, privacy,
authentication, prevention, or confidentiality”
| “usability, performance, operational,
---
look & feel, legal, fault & tolerance,
maintainability, scalability, availability, or portability”
SE_E |  | Word embedding
---
(selected from top 50 words) +
Original 2
| “vulnerability, security, protection, cybersecurity,
---
assurance, countermeasure, threat, privacy,
authentication, prevention, confidentiality, trusted,
intrusion, compromise, safety, insecure, defensive,
breach, proactive, tampering, penetration, policy, phishing,
vulnerable, authorization, dependability, or certification”
| “usability, performance, operational,
---
look & feel, legal, fault & tolerance,
maintainability, scalability, availability, or portability”

Table 7: Label configurations for the top 4 largest NFR classes (US, SE, O,
and PE) for the multi-class and multi-label classification sub-tasks in Task
NFR.

Label Abbr. |  Label Configuration |  List of Labels
---|---|---
|  |
MultiNFR_A |  Original label |  [“usability”, “security”, “performance”, “operational”]
|  |
MultiNFR_B |  Expert curated |  | [“instructive, easy, helpful, useful, learnable, explainable, affordable, intuitive, or understandable”,
---
“security, authorization, or protection”,
“periodic execution or efficacy performance”,
“working, running, connecting, interfacing, or operative environment”]
|  |
MultiNFR_C |  | Word embedding
---
(selected from top 20 words)
| [“accessibility, aesthetic, contextual, experience, satisfaction, HCI, UX,
---
questionnaire, ease, or ergonomics” ,
“vulnerability, securing. protecting, protection, cybersecurity, assurance,
cyber, countermeasure, threat, privacy, authentication, prevention, or
confidentiality”,
“throughput, reliability, scalability, responsiveness, efficiency, workload,
benchmark,
latency, speed, improvement, or accuracy”,
“environmental, organizational, coordination, systemic, or logistics”]
|  |
MultiNFR_D |  | Word embedding
---
(selected from top 50 words)
| [“accessibility, aesthetic, contextual, experience, satisfaction, HCI, UX,
questionnaire, ease,
---
ergonomics, designer, evaluate, multimodal, practitioner, prototyping,
preference, personalization,
suitability, focus, clarity, responsiveness, judgement, feel, or helpful”,
“vulnerability, security, protection, cybersecurity, assurance,
countermeasure, threat,
privacy, authentication, prevention, confidentiality, trusted, intrusion,
compromise, safety,
insecure, defensive, breach, proactive, tampering, penetration, policy,
phishing, vulnerable,
authorization, dependability, or certification”,
“throughput, reliability, scalability, responsiveness, efficiency, workload,
benchmark, latency, speed, improvement, accuracy, achieve, tuning, bottleneck,
better, high, optimize, effectiveness, low, enhances, reducing, increased,
quality, faster, or degrades",
“environmental, organizational, coordination, systemic, logistics,
coordination, or automation”]
|  |

Two labelling configurations are used for the three sub-tasks, one for the
binary case (Table 6), and the other for the both multi-class and multi-label
classification cases (Table 7).

##### Binary classification

For the binary case, we have 5 configurations for each NFR class considered,
i.e., each binary ZSL classifier. In Table 6 we report only the labels for the
usability and security classes, while the other labels are reported in the
Appendix Table a1. The strategies are analogous to those already discussed for
the FR/NFR Task. The only main difference is the usage of the top-50 words
from the word embeddings, besides the top-20. We performed some preliminary
experiments and saw that the list of similar words for the NFR class names
included relevant words also beyond the top-20, and therefore we considered it
reasonable to extend the list of terms to be included in the labels.

Concerning the agreement in the word selection (top-50) we have the following
statistics: 52% perfect agreement, 44 % partial agreement, and 2%
disagreement. We computed the IRR using Krippendorff’s alpha test at micro
(IRR rate of all NFR labels) and macro (an average of the IRR rate per NFR
label). We obtained an alpha rate of 0.45 and 0.53 at macro and micro level,
respectively. To confirm the IRR results, we ran Fleiss’s Kappa test, and we
obtained a Fleiss’s kappa of 0.42 and 0.52 at macro and micro level,
respectively. The overall IRR results indicate a moderate agreement between
the three annotators.

##### Multi-class and multi-label classification.

For these tasks, we have a list of labels for each configuration, cf. Table 7.
The list is represented with squared brackets, the elements in the list are
separated by commas, and each elements is a label, expressed between quotes,
as in Python code. In the table, each label in the list is associated to one
of the top-4 largest classes, namely usability, security, performance and
operational NFR. The label configurations considering all classes are reported
in the Appendix Table a2. We did not use combinations of labeling strategies
for these cases, given the extensive number of experiments, and the
exploratory nature of the study.

####  5.2.2 NFR Binary Classification

Table 8: Binary classification results for top four NFR classes in Task NFR.
Bold values indicate the best performance for the specific ZSL classifier;
underlined values indicate the overall best performance across all the
classifiers; values in italics indicate a possibly misleading performance,
related to the imbalance of F1 between classes.

ZSL Classifier |  US |  SE |  O |  PE
---|---|---|---|---
|  |  |
|  wP |  wR |  wF1 |  wP |  wR |  wF1 |  wP |  wR |  wF1 |  wP |  wR |  wF1
|  |  |  |  |  |  |  |  |  |  |  |
Sbert + (NFR)_A |  0.74 |  0.40 |  0.37 |  0.49 |  0.28 |  0.18 |  0.79 |  0.45 |  0.44 |  0.76 |  0.39 |  0.39
|  |  |  |  |  |  |  |  |  |  |  |
Sbert + (NFR)_B |  0.71 |  0.43 |  0.42 |  0.35 |  0.24 |  0.17 |  0.78 |  0.43 |  0.42 |  0.75 |  0.36 |  0.35
|  |  |  |  |  |  |  |  |  |  |  |
Sbert + (NFR)_C |  0.74 |  0.43 |  0.41 |  0.34 |  0.22 |  0.19 |  0.80 |  0.45 |  0.44 |  0.76 |  0.38 |  0.38
|  |  |  |  |  |  |  |  |  |  |  |
Sbert + (NFR)_D |  0.77 |  0.78 |  0.78 |  0.72 |  0.71 |  0.72 |  0.68 |  0.72 |  0.70 |  0.80 |  0.67 |  0.70
|  |  |  |  |  |  |  |  |  |  |  |
Sbert + (NFR)_E |  0.81 |  0.82 |  0.80 |  0.76 |  0.78 |  0.75 |  0.68 |  0.63 |  0.65 |  0.78 |  0.78 |  0.78
|  |  |  |  |  |  |  |  |  |  |  |
AllMini + (NFR)_A |  0.73 |  0.62 |  0.64 |  0.72 |  0.50 |  0.51 |  0.54 |  0.43 |  0.47 |  0.73 |  0.59 |  0.63
|  |  |  |  |  |  |  |  |  |  |  |
AllMini + (NFR)_B |  0.54 |  0.35 |  0.35 |  0.73 |  0.73 |  0.73 |  0.71 |  0.65 |  0.67 |  0.80 |  0.70 |  0.78
|  |  |  |  |  |  |  |  |  |  |  |
AllMini + (NFR)_C |  0.31 |  0.23 |  0.17 |  0.59 |  0.65 |  0.61 |  0.67 |  0.58 |  0.61 |  0.68 |  0.57 |  0.60
|  |  |  |  |  |  |  |  |  |  |  |
AllMini + (NFR)_D |  0.75 |  0.75 |  0.75 |  0.84 |  0.84 |  0.84 |  0.65 |  0.55 |  0.58 |  0.81 |  0.69 |  0.71
|  |  |  |  |  |  |  |  |  |  |  |
AllMini + (NFR)_E |  0.69 |  0.71 |  0.69 |  0.73 |  0.72 |  0.73 |  0.60 |  0.54 |  0.57 |  0.75 |  0.73 |  0.74
|  |  |  |  |  |  |  |  |  |  |  |
Bert4RE + (NFR)_A |  0.53 |  0.73 |  0.62 |  0.59 |  0.70 |  0.62 |  0.56 |  0.75 |  0.64 |  0.61 |  0.58 |  0.59
|  |  |  |  |  |  |  |  |  |  |  |
Bert4RE + (NFR)_B |  0.51 |  0.30 |  0.24 |  0.81 |  0.74 |  0.63 |  0.67 |  0.70 |  0.68 |  0.61 |  0.75 |  0.67*
|  |  |  |  |  |  |  |  |  |  |  |
Bert4RE + (NFR)_C |  0.52 |  0.43 |  0.46 |  0.62 |  0.65 |  0.63 |  0.72 |  0.73 |  0.72 |  0.68 |  0.41 |  0.43
|  |  |  |  |  |  |  |  |  |  |  |
Bert4RE + (NFR)_D |  0.54 |  0.42 |  0.45 |  0.56 |  0.70 |  0.61 |  0.63 |  0.41 |  0.42 |  0.60 |  0.46 |  0.51
|  |  |  |  |  |  |  |  |  |  |  |
Bert4RE + (NFR)_E |  0.53 |  0.73 |  0.62 |  0.54 |  0.73 |  0.62 |  0.63 |  0.40 |  0.41 |  0.61 |  0.78 |  0.69
|  |  |  |  |  |  |  |  |  |  |  |
SObert + (NFR)_A |  0.53 |  0.73 |  0.62 |  0.54 |  0.73 |  0.62 |  0.56 |  0.75 |  0.64 |  0.61 |  0.78 |  0.69
|  |  |  |  |  |  |  |  |  |  |  |
SObert + (NFR)_B |  0.55 |  0.51 |  0.53 |  0.81 |  0.74 |  0.63 |  0.65 |  0.73 |  0.67 |  0.61 |  0.78 |  0.68
|  |  |  |  |  |  |  |  |  |  |  |
SObert + (NFR)_C |  0.51 |  0.32 |  0.30 |  0.66 |  0.53 |  0.56 |  0.71 |  0.74 |  0.71 |  0.61 |  0.55 |  0.58
|  |  |  |  |  |  |  |  |  |  |  |
SObert + (NFR)_E |  0.54 |  0.45 |  0.48 |  0.58 |  0.69 |  0.61 |  0.68 |  0.58 |  0.61 |  0.61 |  0.76 |  0.67
|  |  |  |  |  |  |  |  |  |  |  |
SObert + (NFR)_D |  0.53 |  0.70 |  0.60 |  0.63 |  0.73 |  0.63 |  0.68 |  0.57 |  0.60 |  0.66 |  0.77 |  0.69
|  |  |  |  |  |  |  |  |  |  |  |

Table 8 reports the classification results for the 4 largest NFR classes. The
overall results indicate an acceptable performance rates, nearly above 0.7 as
a wF1, across all the best rates reported in that table.

The highest F1 score of 0.84 is achieved for the security class, with AllMini
+ SE_D which uses the word embedding selected labels (top-20) for the security
class, and the original NFR labels for the “other” class. Following that, is
the usability class with wF1 = 0.80, using Sbert + US_E which includes the
word embedding selected labels (top-50) for the usability class, and the
original labels for the “other” class. This suggests that, for this task and
for highly represented classes such as security and usability, generic LMs
combined with selected terms from word-embedding as labels appears to be the
most effective configuration. However, it should be noted that this also
depends on the specific requirement class considered. Indeed, for performance
requirements we still see that generic models still outperform the others (F1
= 0.78 for both Sbert and AllMini), but the labelling strategy B (Expert
curated) appears to be the most effective for AllMini. Furthermore, for
Operational requirements, the best performance are achieved with BERT4RE (F1 =
0.72), though comparable with SBert (F1 = 0.70). While, in general, generic
models outperforms the specific ones, the performance may depend on the class
considered, i.e., the specific task addressed.

Table 9: Binary classification results of the top 4 NFR classes in Task NFR.
Bold values indicate the overall best classification results for a particular
measure.

NFR class (249) | ZSL classifier | NFR class | “Other” class
---|---|---|---
P | R | F1 | P | R | F1
US (67) | Sbert + US_E | 0.73 | 0.49 | 0.59 | 0.83 | 0.93 | 0.88
SE (66) | AllMini + SE_D | 0.67 | 0.73 | 0.70 | 0.90 | 0.87 | 0.89
O (62) | Bert4RE + O_C | 0.46 | 0.37 | 0.41 | 0.80 | 0.86 | 0.83
PE (54) | Sbert + PE_E | 0.50 | 0.46 | 0.48 | 0.85 | 0.87 | 0.86
PE (54) | AllMini + PE_B | 0.47 | 0.63 | 0.54 | 0.89 | 0.81 | 0.84

Table 9 reports the detailed classification report of the best ZSL classifiers
from Table 8, considering P, R, and F1 for each class. We see that all the
best classifiers tend to achieve higher performance on the “other” class (best
F1 for NFR class 0.70, vs 0.89 for the “other” class). This suggests that the
ZSL binary classifier encounters some difficulty in associating the
requirements to the specific labels, despite the extensive set of terms used.
A more accurate selection of terms, or the usage of terms directly coming from
the requirements themselves111111We did not consider this option, as it would
have biased the classification. However, it is a viable choice in practical
contexts, could overcome this issue.

Table 10: Best performance results of binary classification of all 10 NFR
classes in Task NFR. The results are different with respect to the binary
classification using solely four classes because more requirements are
involved in the classification, thus leading to lower performance.

NFR class (369) | ZSL classifier | wP | wR | wF1 |  NFR class |  “Other” class
---|---|---|---|---|---|---
|
|  |  |  |  | P | R | F1 | P | R | F1
US (67) | Sbert + US_E | 0.80 | 0.79 | 0.80 | 0.44 | 0.49 | 0.46 | 0.88 | 0.86 | 0.87
SE (66) | AllMini + SE_D | 0.87 | 0.85 | 0.85 | 0.61 | 0.33 | 0.43 | 0.87 | 0.95 | 0.91
O (62) | Bert4RE + O_C | 0.78 | 0.77 | 0.77 | 0.32 | 0.37 | 0.35 | 0.87 | 0.85 | 0.86
PE (54) | Sbert + PE_E | 0.82 | 0.78 | 0.80 | 0.33 | 0.46 | 0.38 | 0.90 | 0.84 | 0.87
LF (38) | Sbert + LF_D | 0.85 | 0.76 | 0.80 | 0.18 | 0.21 | 0.20 | 0.91 | 0.89 | 0.90
A (21) | SObert + A_D | 0.90 | 0.70 | 0.78 | 0.10 | 0.43 | 0.14 | 0.95 | 0.72 | 0.82
SC (21) | AllMini + SC_E | 0.92 | 0.74 | 0.81 | 0.13 | 0.62 | 0.21 | 0.97 | 0.75 | 0.85
MN (17) | AllMini + MN_E | 0.93 | 0.86 | 0.89 | 0.16 | 0.47 | 0.24 | 0.97 | 0.88 | 0.92
L (13) | AllMini + L_E | 0.95 | 0.84 | 0.88 | 0.11 | 0.54 | 0.19 | 0.98 | 0.85 | 0.91
FT (10) | AllMini + FT_E | 0.96 | 0.86 | 0.91 | 0.10 | 0.50 | 0.17 | 0.98 | 0.88 | 0.93

Finally, Table 10 lists the top performance rates considering all classes, and
the entire set of requirements. Comparing these results with Table 9, we see
that there is no substantial decrease in terms of performance for the largest
classes, e.g., US still achieves wF1 = 0.80, while SE achieves wF1 = 0.85,
which is even higher than wF1 = 0.84 in Table 9. However, the results are all
biased towards the “other” class, since, even in the best case, F1 for the NFR
class is lower than 0.50. This problem was not so evident in Table 10, at
least for the US and SE classes, where F1 is still acceptable also for the NFR
class. We can therefore conclude that, in case of requirements belonging to
many different classes, a binary ZSL classification leads to poor
classification results, with the selected labelling strategy.

####  5.2.3 NFR Multi-class Classification

Table 11: Multi-class classification results for top 4 NFR classes. Bold
values indicate the best performance of a specific ZSL classifier; underlined
values indicate the overall best performance across all the classifiers;
values in italics indicate poor performance of the classifier.

ZSL Classifier | US | SE | O | PE
---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Sbert + MultiNFR_A | 0.44 | 0.52 | 0.48 | 0.57 | 0.70 | 0.63 | 0.47 | 0.45 | 0.46 | 0.43 | 0.22 | 0.29
Sbert + MultiNFR_B | 0.73 | 0.43 | 0.54 | 0.69 | 0.64 | 0.66 | 0.53 | 0.71 | 0.61 | 0.48 | 0.57 | 0.52
Sbert + MultiNFR_C | 0.62 | 0.31 | 0.42 | 0.54 | 0.62 | 0.58 | 0.59 | 0.21 | 0.31 | 0.34 | 0.74 | 0.47
Sbert + MultiNFR_D | 0.69 | 0.40 | 0.51 | 0.65 | 0.53 | 0.58 | 0.38 | 0.23 | 0.47 | 0.46 | 0.53 | 0.58
AllMini + MultiNFR_A | 0.67 | 0.36 | 0.47 | 0.66 | 0.92 | 0.77 | 0.33 | 0.32 | 0.33 | 0.45 | 0.50 | 0.47
AllMini + MultiNFR_B | 0.77 | 0.36 | 0.49 | 0.63 | 0.94 | 0.76 | 0.64 | 0.64 | 0.64 | 0.59 | 0.70 | 0.64
AllMini + MultiNFR_C | 0.54 | 0.45 | 0.49 | 0.76 | 0.67 | 0.71 | 0.36 | 0.37 | 0.37 | 0.41 | 0.54 | 0.47
AllMini + MultiNFR_D | 0.34 | 0.43 | 0.38 | 0.88 | 0.33 | 0.48 | 0.25 | 0.42 | 0.31 | 0.47 | 0.28 | 0.35
Bert4RE + MultiNFR_A | 0.32 | 0.09 | 0.14 | 0.18 | 0.11 | 0.13 | 0.33 | 0.02 | 0.03 | 0.19 | 0.67 | 0.30
Bert4RE + MultiNFR_B | 0.33 | 0.60 | 0.42 | 0.00 | 0.00 | 0.00 | 0.28 | 0.53 | 0.37 | 0.13 | 0.02 | 0.03
Bert4RE + MultiNFR_C | 0.33 | 0.06 | 0.10 | 0.18 | 0.08 | 0.11 | 0.24 | 0.68 | 0.36 | 0.25 | 0.17 | 0.20
Bert4RE + MultiNFR_D | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.24 | 0.34 | 0.28 | 0.25 | 0.04 | 0.06
SObert + MultiNFR_A | 0.00 | 0.00 | 0.00 | 0.60 | 0.05 | 0.08 | 0.22 | 0.71 | 0.33 | 0.12 | 0.09 | 0.10
SObert + MultiNFR_B | 0.30 | 0.81 | 0.44 | 0.00 | 0.00 | 0.00 | 0.29 | 0.16 | 0.21 | 0.08 | 0.06 | 0.07
SObert + MultiNFR_C | 0.31 | 0.36 | 0.33 | 0.24 | 0.21 | 0.22 | 0.31 | 0.56 | 0.40 | 0.00 | 0.00 | 0.00
SObert + MultiNFR_D | 0.20 | 0.04 | 0.07 | 0.00 | 0.00 | 0.00 | 0.24 | 0.90 | 0.38 | 0.00 | 0.00 | 0.00

Table 11 reports the multi-class classification results for the 4 largest NFR
classes. We see that, compared to the binary classification, results are
substantially lower, although still acceptable for SE (F1 = 0.76), O (F1 =
0.64) and PE (F1 = 0.64). In the majority of the cases, the best results are
obtained again with the domain-generic LMs, and using MultiNFR_A or MultiNFR_B
as labels. These are the shortest labels, which do not use the word embedding
strategies. This result is the opposite of what was observed for binary
classification for NFR. We argue therefore that, in a multi-class
classification setting, longer and more informative labels can lead to some
possible overlapping between the represented meaning of each class. Instead,
in a binary classification setting, more informative labels, i.e., using word
embeddings, appear to be more effective.

Table 12: Overall performance results of multi-class classification of all 10
NFR classes in PROMISE dataset (369 requirements) for Task NFR. Bold values
indicate the best performance for the specific ZSL classifier; underlined
values indicate the overall best performance across all the classifiers;
values in italics indicate poor performance of the classifier.

ZSL Classifier | US | SE | O | PE | LF
---|---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Sbert + MultiNFR_A | 0.50 | 0.21 | 0.29 | 0.56 | 0.44 | 0.49 | 0.29 | 0.24 | 0.26 | 0.29 | 0.07 | 0.12 | 0.33 | 0.13 | 0.19
Sbert + MultiNFR_B | 0.64 | 0.31 | 0.42 | 0.76 | 0.38 | 0.51 | 0.44 | 0.32 | 0.37 | 0.30 | 0.26 | 0.28 | 0.34 | 0.58 | 0.43
Sbert + MultiNFR_C | 0.28 | 0.07 | 0.12 | 0.57 | 0.44 | 0.50 | 0.52 | 0.18 | 0.27 | 0.15 | 0.31 | 0.20 | 0.28 | 0.18 | 0.22
Sbert + MultiNFR_D | 0.39 | 0.10 | 0.16 | 0.74 | 0.30 | 0.43 | 3.32 | 0.32 | 0.32 | 0.61 | 0.20 | 0.31 | 0.20 | 0.29 | 0.23
AllMini + MultiNFR_A | 0.48 | 0.16 | 0.24 | 0.79 | 0.62 | 0.69 | 0.20 | 0.06 | 0.10 | 0.56 | 0.28 | 0.37 | 1.00 | 0.13 | 0.23
AllMini + MultiNFR_B | 1.00 | 0.13 | 0.24 | 0.70 | 0.67 | 0.68 | 0.60 | 0.29 | 0.39 | 0.13 | 0.06 | 0.08 | 0.51 | 0.47 | 0.49
AllMini + MultiNFR_C | 0.28 | 0.07 | 0.12 | 0.88 | 0.21 | 0.34 | 0.18 | 0.15 | 0.16 | 0.29 | 0.28 | 0.28 | 0.22 | 0.05 | 0.09
AllMini + MultiNFR_D | 1.00 | 0.01 | 0.03 | 1.00 | 0.02 | 0.03 | 0.16 | 0.19 | 0.18 | 1.00 | 0.02 | 0.04 | 0.00 | 0.00 | 0.00
Bert4RE + MultiNFR_A | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.11 | 0.18 | 0.14
Bert4RE + MultiNFR_B | 0.23 | 0.60 | 0.33 | 0.00 | 0.00 | 0.00 | 0.17 | 0.39 | 0.24 | 0.50 | 0.02 | 0.04 | 0.05 | 0.03 | 0.04
Bert4RE + MultiNFR_C | 0.00 | 0.00 | 0.00 | 0.16 | 0.06 | 0.09 | 0.16 | 0.47 | 0.23 | 0.00 | 0.00 | 0.00 | 0.19 | 0.21 | 0.20
Bert4RE + MultiNFR_D | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.16 | 0.73 | 0.26 | 0.19 | 0.13 | 0.16 | 0.00 | 0.00 | 0.00
SObert + MultiNFR_A | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.10 | 0.95 | 0.18
SObert + MultiNFR_B | 0.21 | 0.81 | 0.33 | 0.00 | 0.00 | 0.00 | 0.16 | 0.11 | 0.13 | 0.14 | 0.06 | 0.08 | 0.50 | 0.05 | 0.10
SObert + MultiNFR_C | 0.23 | 0.18 | 0.20 | 0.16 | 0.05 | 0.07 | 0.29 | 0.21 | 0.24 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
SObert + MultiNFR_E | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.21 | 0.34 | 0.26 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
ZSL Classifier | SC | A | MN | L | FT
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Sbert + MultiNFR_A | 0.24 | 0.48 | 0.32 | 0.11 | 0.10 | 0.10 | 0.09 | 0.47 | 0.15 | 0.38 | 0.23 | 0.29 | 0.04 | 0.20 | 0.07
Sbert + MultiNFR_B | 0.20 | 0.14 | 0.17 | 0.00 | 0.00 | 0.00 | 0.22 | 0.71 | 0.33 | 0.67 | 0.31 | 0.42 | 0.07 | 0.50 | 0.13
Sbert + MultiNFR_C | 0.05 | 0.14 | 0.08 | 0.00 | 0.00 | 0.00 | 0.09 | 0.18 | 0.12 | 0.14 | 0.23 | 0.18 | 0.09 | 0.20 | 0.12
Sbert + MultiNFR_D | 0.15 | 0.76 | 0.25 | 0.10 | 0.24 | 0.14 | 0.08 | 0.06 | 0.07 | 0.18 | 0.15 | 0.17 | 0.00 | 0.00 | 0.00
AllMini + MultiNFR_A | 0.27 | 0.62 | 0.38 | 0.18 | 0.86 | 0.30 | 0.24 | 0.41 | 0.30 | 0.05 | 0.08 | 0.06 | 0.07 | 0.30 | 0.11
AllMini + MultiNFR_B | 0.25 | 0.33 | 0.29 | 0.13 | 0.43 | 0.21 | 0.42 | 0.47 | 0.44 | 0.20 | 0.69 | 0.32 | 0.11 | 0.60 | 0.19
AllMini + MultiNFR_C | 0.12 | 0.29 | 0.17 | 0.09 | 0.62 | 0.16 | 0.07 | 0.06 | 0.06 | 0.28 | 0.38 | 0.32 | 0.50 | 0.20 | 0.29
AllMini + MultiNFR_D | 0.02 | 0.05 | 0.03 | 0.12 | 0.52 | 0.19 | 0.05 | 0.05 | 0.03 | 0.00 | 0.00 | 0.00 | 1.00 | 0.20 | 0.33
Bert4RE + MultiNFR_A | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.06 | 0.94 | 0.10 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Bert4RE + MultiNFR_B | 0.00 | 0.00 | 0.00 | 0.07 | 0.10 | 0.08 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Bert4RE + MultiNFR_C | 0.13 | 0.05 | 0.07 | 0.07 | 0.19 | 0.10 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Bert4RE + MultiNFR_D | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.03 | 0.06 | 0.04 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
SObert + MultiNFR_A | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
SObert + MultiNFR_B | 0.08 | 0.14 | 0.10 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.20 | 0.08 | 0.11 | 0.00 | 0.00 | 0.00
SObert + MultiNFR_C | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.05 | 0.41 | 0.08 | 0.02 | 0.08 | 0.03 | 0.04 | 0.10 | 0.06
SObert + MultiNFR_D | 0.00 | 0.00 | 0.00 | 0.05 | 0.60 | 0.10 | 0.08 | 0.18 | 0.11 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00

Table 12 reports the multi-class classification results for all the NFR
classes, except for the portability class. The performance in terms of F1
remains acceptable only for SE class (F1= 0.69), while for the other classes
poor results are obtained. Similarly to what was observed for the binary
classification, when requirements belonging to many classes are considered,
ZSL does not appear to be sufficiently effective.

####  5.2.4 NFR Multi-label Classification

Table 13: Multi-label classification results for 4 largest NFR classes—Top 2
results—249 requirements from the PROMISE dataset. Bold values indicate the
best performance for the specific ZSL classifier; underlined values indicate
the overall best performance across all the classifiers; values in italics
indicate poor performance of the classifier.

ZSL Classifier | US | SE | O | PE
---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Sbert + MultiNFR_A | 0.60 | 0.67 | 0.60 | 0.74 | 0.84 | 0.79 | 0.72 | 0.71 | 0.72 | 0.68 | 0.46 | 0.55
Sbert + MultiNFR_B | 0.93 | 0.60 | 0.73 | 0.86 | 0.77 | 0.82 | 0.71 | 0.92 | 0.80 | 0.73 | 0.91 | 0.81
Sbert + MultiNFR_C | 0.84 | 0.73 | 0.78 | 0.76 | 0.79 | 0.78 | 0.81 | 0.48 | 0.60 | 0.57 | 0.91 | 0.70
Sbert + MultiNFR_D | 0.92 | 0.70 | 0.80 | 0.83 | 0.76 | 0.79 | 0.62 | 0.85 | 0.72 | 0.81 | 0.80 | 0.80
AllMini + MultiNFR_A | 0.82 | 0.60 | 0.69 | 0.79 | 1.00 | 0.88 | 0.62 | 0.58 | 0.60 | 0.69 | 0.74 | 0.71
AllMini + MultiNFR_B | 0.93 | 0.58 | 0.72 | 0.81 | 0.98 | 0.89 | 0.80 | 0.84 | 0.82 | 0.82 | 0.94 | 0.88
AllMini + MultiNFR_C | 0.63 | 0.70 | 0.66 | 0.94 | 0.73 | 0.82 | 0.63 | 0.63 | 0.63 | 0.64 | 0.72 | 0.68
AllMini + MultiNFR_D | 0.63 | 0.88 | 0.74 | 0.96 | 0.74 | 0.84 | 0.61 | 0.58 | 0.60 | 0.74 | 0.63 | 0.68
Bert4RE + MultiNFR_A | 0.84 | 0.70 | 0.76 | 0.39 | 0.20 | 0.26 | 0.96 | 0.44 | 0.60 | 0.32 | 0.78 | 0.45
Bert4RE + MultiNFR_B | 0.41 | 0.67 | 0.51 | 1.00 | 0.08 | 0.14 | 0.45 | 0.87 | 0.60 | 0.93 | 0.24 | 0.38
Bert4RE + MultiNFR_C | 0.98 | 0.76 | 0.86 | 0.28 | 0.11 | 0.15 | 0.30 | 0.73 | 0.45 | 0.53 | 0.31 | 0.40
Bert4RE + MultiNFR_D | 1.00 | 0.12 | 0.21 | 0.00 | 0.00 | 0.00 | 0.31 | 0.95 | 0.47 | 0.90 | 0.87 | 0.89
SObert + MultiNFR_A | 0.00 | 0.00 | 0.00 | 1.00 | 0.44 | 0.61 | 0.36 | 0.98 | 0.53 | 0.57 | 0.54 | 0.55
SObert + MultiNFR_B | 0.39 | 0.85 | 0.54 | 1.00 | 0.05 | 0.09 | 0.75 | 0.92 | 0.83 | 0.29 | 0.13 | 0.18
SObert + MultiNFR_C | 0.71 | 0.10 | 0.83 | 0.46 | 0.38 | 0.42 | 0.43 | 0.66 | 0.52 | 1.00 | 0.09 | 0.17
SObert + MultiNFR_E | 0.89 | 1.00 | 0.94 | 0.00 | 0.00 | 0.00 | 0.34 | 0.92 | 0.49 | 1.00 | 0.06 | 0.11

Table 13 reports the multi-label classification results for the 4 largest NFR
classes, considering the top-2 labels returned by the classifier. In other
terms, when the right label is returned by the classifier in the top-2 labels,
we consider it a true positive. We see that in this case performance
substantially increases with respect to the multi-class classification case in
Table 11, e.g., reaching $F1=0.94$ for US requirements, $0.89$ for SE, $0.83$
for O, and $0.89$ for PE. This suggests that the multi-label classification
strategy may be the most effective when dealing with NFR classification.

Looking at the results based on the LMs, we do not have a clear pattern, and
each LM appears to be suitable for a certain requirement type. Concerning
labels, simple configurations as MultiNFR_A and MultiNFR_B appears to be the
most effective for all classes, except US, for which the embedding-based
labels are more effective. This could be due to a better, and more clear-cut
characterisation of US requirements with respect to other types.

Table 14: Multi-label classification results for all NFR classes—Top 3
results—369 requirements from PROMISE dataset. Bold values indicate the best
performance for the specific ZSL classifier; underlined values indicate the
overall best performance across all the classifiers; values in italics
indicate poor performance of the classifier.

ZSL Classifier | US | SE | O | PE | LF
---|---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Sbert + MultiNFR_A | 0.74 | 0.55 | 0.63 | 0.78 | 0.71 | 0.75 | 0.56 | 0.53 | 0.55 | 0.73 | 0.35 | 0.48 | 0.64 | 0.37 | 0.47
Sbert + MultiNFR_B | 0.87 | 0.51 | 0.64 | 0.87 | 0.62 | 0.73 | 0.71 | 0.74 | 0.72 | 0.62 | 0.69 | 0.65 | 0.55 | 0.76 | 0.64
Sbert + MultiNFR_C | 0.76 | 0.39 | 0.51 | 0.80 | 0.71 | 0.75 | 0.80 | 0.39 | 0.52 | 0.41 | 0.81 | 0.54 | 0.68 | 0.55 | 0.61
Sbert + MultiNFR_D | 0.82 | 0.40 | 0.54 | 0.90 | 0.58 | 0.70 | 0.63 | 0.73 | 0.68 | 0.81 | 0.46 | 0.59 | 0.46 | 0.58 | 0.51
AllMini + MultiNFR_A | 0.73 | 0.45 | 0.56 | 0.92 | 0.88 | 0.90 | 0.52 | 0.90 | 0.28 | 0.76 | 0.46 | 0.57 | 1.00 | 0.32 | 0.48
AllMini + MultiNFR_B | 1.00 | 0.40 | 0.57 | 0.80 | 0.92 | 0.86 | 0.78 | 0.52 | 0.62 | 0.70 | 0.65 | 0.67 | 0.71 | 0.63 | 0.67
AllMini + MultiNFR_C | 0.79 | 0.49 | 0.61 | 0.96 | 0.39 | 0.56 | 0.49 | 0.32 | 0.39 | 0.56 | 0.52 | 0.54 | 0.79 | 0.29 | 0.42
AllMini + MultiNFR_D | 1.00 | 0.36 | 0.53 | 1.00 | 0.22 | 0.37 | 0.32 | 0.35 | 0.34 | 1.00 | 0.33 | 0.50 | 0.75 | 0.32 | 0.28
Bert4RE + MultiNFR_A | 1.00 | 0.16 | 0.28 | 1.00 | 0.09 | 0.17 | 1.00 | 0.10 | 0.18 | 1.00 | 0.52 | 0.68 | 0.60 | 0.39 | 0.48
Bert4RE + MultiNFR_B | 0.29 | 0.66 | 0.40 | 0.00 | 0.00 | 0.00 | 0.35 | 0.81 | 0.49 | 0.83 | 0.09 | 0.17 | 0.63 | 0.58 | 0.60
Bert4RE + MultiNFR_C | 1.00 | 0.06 | 0.11 | 0.50 | 0.27 | 0.35 | 0.25 | 0.71 | 0.37 | 1.00 | 0.13 | 0.23 | 0.33 | 0.37 | 0.35
Bert4RE + MultiNFR_D | 1.00 | 0.03 | 0.06 | 0.00 | 0.00 | 0.00 | 0.22 | 0.89 | 0.35 | 0.30 | 0.19 | 0.23 | 1.00 | 0.45 | 0.62
SObert + MultiNFR_A | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.06 | 0.12 | 1.00 | 0.04 | 0.07 | 0.00 | 0.00 | 0.00
SObert + MultiNFR_B | 0.28 | 0.85 | 0.43 | 0.00 | 0.00 | 0.00 | 0.62 | 0.76 | 0.68 | 0.40 | 0.19 | 0.25 | 0.93 | 0.34 | 0.50
SObert + MultiNFR_C | 0.54 | 0.63 | 0.58 | 0.64 | 0.32 | 0.42 | 0.60 | 0.65 | 0.62 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
SObert + MultiNFR_E | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.42 | 0.77 | 0.55 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
ZSL classifier | SC | A | MN | L | FT
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Sbert + MultiNFR_A | 0.53 | 0.76 | 0.63 | 0.41 | 0.33 | 0.37 | 0.16 | 0.59 | 0.25 | 0.50 | 0.31 | 0.38 | 0.22 | 0.70 | 0.33
Sbert + MultiNFR_B | 0.55 | 0.29 | 0.38 | 0.83 | 0.24 | 0.37 | 0.42 | 0.94 | 0.58 | 1.00 | 0.31 | 0.47 | 0.20 | 0.90 | 0.32
Sbert + MultiNFR_C | 0.36 | 0.81 | 0.50 | 0.50 | 0.14 | 0.22 | 0.36 | 0.47 | 0.41 | 0.38 | 0.38 | 0.38 | 0.21 | 0.40 | 0.28
Sbert + MultiNFR_D | 0.24 | 0.81 | 0.38 | 0.33 | 0.76 | 0.46 | 0.57 | 0.47 | 0.52 | 0.42 | 0.23 | 0.30 | 0.67 | 0.40 | 0.50
AllMini + MultiNFR_A | 0.45 | 0.81 | 0.58 | 0.31 | 1.00 | 0.48 | 0.58 | 0.65 | 0.61 | 0.43 | 0.77 | 0.56 | 0.14 | 0.70 | 0.23
AllMini + MultiNFR_B | 0.57 | 0.57 | 0.57 | 0.54 | 0.95 | 0.69 | 0.62 | 0.76 | 0.68 | 0.35 | 1.00 | 0.52 | 0.24 | 0.60 | 0.34
AllMini + MultiNFR_C | 0.35 | 0.81 | 0.49 | 0.15 | 0.81 | 0.26 | 0.33 | 0.35 | 0.34 | 0.50 | 0.34 | 0.52 | 1.00 | 0.50 | 0.67
AllMini + MultiNFR_D | 0.22 | 0.38 | 0.28 | 0.30 | 1.00 | 0.46 | 0.09 | 0.59 | 0.18 | 1.00 | 0.31 | 0.47 | 1.00 | 0.60 | 0.75
Bert4RE + MultiNFR_A | 1.00 | 0.62 | 0.76 | 0.00 | 0.00 | 0.00 | 0.06 | 1.00 | 0.11 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Bert4RE + MultiNFR_B | 1.00 | 0.05 | 0.09 | 0.40 | 0.57 | 0.47 | 1.00 | 0.12 | 0.21 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Bert4RE + MultiNFR_C | 0.14 | 0.05 | 0.07 | 0.24 | 0.62 | 0.35 | 0.67 | 0.47 | 0.55 | 0.00 | 0.00 | 0.00 | 0.05 | 0.10 | 0.07
Bert4RE + MultiNFR_D | 0.24 | 0.19 | 0.21 | 1.00 | 0.76 | 0.86 | 0.37 | 0.65 | 0.47 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
SObert + MultiNFR_A | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.94 | 0.97 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00
SObert + MultiNFR_B | 0.33 | 0.71 | 0.45 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.50 | 0.31 | 0.38 | 0.00 | 0.00 | 0.00
SObert + MultiNFR_C | 1.00 | 0.14 | 0.25 | 0.20 | 0.14 | 0.17 | 0.06 | 0.41 | 0.11 | 0.02 | 0.08 | 0.03 | 0.13 | 0.20 | 0.15
SObert + MultiNFR_E | 1.00 | 0.95 | 0.98 | 0.10 | 0.95 | 0.19 | 0.24 | 0.53 | 0.33 | 0.00 | 0.00 | 0.00 | 0.25 | 0.10 | 0.14

Table 14 reports the multi-label classification results for all the NFR
classes, considering the top-3 results, i.e., if the right label is returned
among the top-3 labels, we consider it a true positive. Also in this case, the
performance remain rather high, frequently with F1 above $0.90$ for the best
configurations. In terms of LMs, a clear pattern cannot be identified,
although we see that, for more populated classes (top), the highest F1 is
always achieved by generic LM, while for less populated classes (bottom, e.g.,
SC, A, MN) the best results are obtained with specific models. Overall, we can
say that ZSL in the multi-label classification context appears to be effective
also in case of NFRs belonging to many classes.

###  5.3 Task Security

This task is based on SeqReq dataset. We performed a series of binary
classification tasks on all or parts of the above-mentioned dataset. In the
following sub-sections, we report the results of the labels selection and
experiment results.

####  5.3.1 Security Label Configuration

Table 15: Label configurations for Task Security.

Label Abbr. | Label Configuration | Security | Non-Security
---|---|---|---
Sec_A | Original label | “Security” | “not about security”
Sec_B | Expert curated | “Security, authorization, or protection” | “not about security, authorization, or protection”
Sec_C |  | Word embedding
---
(selected from top 20 words)
| “vulnerability, securing. protecting,
---
protection, cybersecurity, assurance,
cyber, countermeasure,
threat, privacy, authentication,
prevention, or confidentiality”
| “not about vulnerability, securing. protecting,
---
protection, cybersecurity, assurance, cyber,
countermeasure, threat, privacy, authentication,
prevention, or confidentiality”
Sec_D |  | Word embedding
---
(selected from top 50 words)
| “vulnerability, security, protection,
---
cybersecurity, assurance, countermeasure,
threat, privacy, authentication,
prevention, confidentiality, trusted,
intrusion, compromise, safety, insecure,
defensive, breach, proactive, tampering,
penetration, policy, phishing,
vulnerable, authorization, dependability,
or certification”
| “not about vulnerability, security, protection,
---
cybersecurity, assurance, countermeasure,
threat, privacy, authentication,
prevention, confidentiality,
trusted, intrusion, compromise, safety,
insecure, defensive, breach,
proactive, tampering, penetration,
policy, phishing, vulnerable,
authorization, dependability, or certification”

The labelling of the security class (Table 15) is very much similar to the
labels groups related to security as an NFR class in the binary classification
task (cf. 5.2.1). The specific results obtained for the word-embedding of the
term “security” are the following: 50% perfect agreement, 48 % partial
agreement, and 2% disagreement. For IRR we obtained 0.46 as Krippendorff’s
alpha and a Fleiss’ kappa score of 0.45, and both test results indicate a
moderate agreement between the three annotators.

####  5.3.2 Security Binary Classification

Table 16: Overall classification results for Task Security, considering all the requirements in SecReq as a single dataset. Bold values indicate the best configuration for each LM. ZSL Classifier | wP | wR | wF1
---|---|---|---
Sbert + Sec_A | 0.62 | 0.40 | 0.30
Sbert + Sec_B | 0.58 | 0.39 | 0.28
Sbert + Sec_C | 0.58 | 0.41 | 0.31
Sbert + Sec_D | 0.56 | 0.40 | 0.31
AllMini + Sec_A | 0.69 | 0.58 | 0.58
AllMini + Sec_B | 0.68 | 0.65 | 0.66
AllMini + Sec_C | 0.65 | 0.45 | 0.38
AllMini + Sec_D | 0.52 | 0.45 | 0.45
Bert4RE + Sec_A | 0.52 | 0.53 | 0.52
Bert4RE + Sec_B | 0.40 | 0.63 | 0.49
Bert4RE + Sec_C | 0.48 | 0.39 | 0.36
Bert4RE + Sec_D | 0.50 | 0.50 | 0.50
SObert + Sec_A | 0.40 | 0.63 | 0.49
SObert + Sec_B | 0.40 | 0.63 | 0.49
SObert + Sec_C | 0.56 | 0.54 | 0.55
SObert + Sec_D | 0.70 | 0.46 | 0.39

Table 16 reports the results for the Security task, considering all the
requirements in the three datasets. The best performance is achieved by
AllMini + Sec_B, with a wF1 score of 0.66, with wP = 0.68 and wR = 0.65. The
generic LM, AllMini, thus achieves best results. On the other hand, the other
generic model, Sbert, achieves the worst results (wF1 = 0.31), thus suggesting
that generic models are not necessarily better for this specific task. The
best set of labels, Sec_B, is the expert’s curated one, which includes a
limited set of three security-related words. This suggests that a limited
number of well-selected terms is sufficient to identify security requirements
in this dataset.

Table 17: Binary classification results for Task Security. All the requirements are considered together in one single SeqReq dataset. Bold values indicate the best performance relating to a particular measure. ZSL classifier | Security (187) | Non-Security (323)
---|---|---
P | R | F1 | P | R | F1
Sbert + Sec_C | 0.37 | 0.92 | 0.53 | 0.70 | 0.11 | 0.19
AllMini + Sec_B | 0.52 | 0.67 | 0.58 | 0.77 | 0.63 | 0.70
Bert4RE + Sec_A | 0.34 | 0.30 | 0.32 | 0.62 | 0.66 | 0.64
SObert + Sec_C | 0.39 | 0.51 | 0.45 | 0.66 | 0.55 | 0.60

Table 17 shows the performance for the two classes. We see that better
performance in terms of F1 is achieved for Non-Security requirements (F1 =
0.70 vs 0.58), using AllMini + Sec_B, i.e., the best configuration. Looking in
more detail, the best recall (R = 0.92) is obtained by Sbert + Sec_D.
Therefore, if one seeks for a better ability to identify security
requirements, i.e., high recall on this set, this configuration—though having
the worst overall performance—should be preferred. This is an important
observation, since for many requirement tasks, including this one, high recall
is more important than high precision, as remarked by Berry [82]—if one
searches for security requirements, then one wants as less false negatives as
possible.

Table 18: Binary classification results for Task Security. The results are obtained from each of the three projects in SecReq - CPN, GPS and ePurse. Bold values indicate the best configurations. ZSL Classifier | wP | wR | wF1
---|---|---|---
CPN
Sbert + Sec_C | 0.85 | 0.30 | 0.25
AllMini + Sec_B | 0.79 | 0.77 | 0.78
Bert4RE + Sec_B | 0.65 | 0.80 | 0.72
SObert + Sec_A | 0.65 | 0.80 | 0.72
SObert \+ Sec_B | 0.65 | 0.80 | 0.72
GPS
Sbert+ Sec_D | 0.67 | 0.40 | 0.30
AllMini + Sec_B | 0.62 | 0.61 | 0.61
Bert4RE + Sec_C | 0.59 | 0.53 | 0.54
SObert + Sec_C | 0.63 | 0.63 | 0.63
ePurse
Sbert + Sec_D | 0.59 | 0.64 | 0.60
AllMini + Sec_A | 0.69 | 0.70 | 0.69
Bert4RE + Sec_A | 0.67 | 0.44 | 0.40
SObert \+ Sec_D | 0.66 | 0.69 | 0.62

Table 18 reports the results for the Security task, divided by each dataset
included in SecReq. Best results are achieved for CPN (wF1 = 0.78), while
worst results are for GPS (wF1 = 0.63). This could be due to the specific
characteristics of the datasets. In some cases, security and non-security
requirements in GPS are expressed with very similar sentences are likely to be
classified similarly though they belong to different classes (e.g., class
Security: The Load File Data Block Hash is used in the computation of the Load
File Data Block Signature vs Non-Security: The Load File Data Block Hash is
used in the computation of The Load Token). Misclassifications can also be due
to the quality of the tagging of the requirements in the annotated datasets.
For example, in CPN we have the following requirement marked as non-security
related: CNG shall support mechanisms for secure authentication and
communication with the remote management system. In GPS, we have another
example, also marked as non-security: The security level of the communication
with an off-card entity does not necessarily apply to each individual message
being transmitted but can only apply to the environment and/or context in
which messages are transmitted.

##  6 Research Findings

This section derives the answer to the three RQs from the reported results and
provides some general observations obtained from our study.

###  6.1 Best Language Model (RQ1)

To answer this RQ, we focus on the performance of LMs and ignore their label
configurations.

  * 1.

For Task FR/NFR, the best overall performer is Sbert, achieving a wF1 score of
0.66, with wP = 0.71 and wR = 0.66. This indicates that the generic Sbert
model, designed to provide a semantic representation for generic sentences,
substantially outperforms the other LMs for this task.

  * 2.

For Task NFR, we show the performance of LMs in each sub-task as follows:

    * (a)

For binary classification of NFR, the generic AllMini model outperforms the
other three LMs, particularly on the SE class.

    * (b)

For multi-class classification of NFR, in the majority of the cases, the best
results are obtained by the generic LMs (Sbert and AllMini).

    * (c)

For multi-label classification of NFR, there is no clear winner as each LM
appears to be suitable for a certain requirement class.

  * 3.

For Task Security, the best overall performer is the generic AllMini model;
however, on the other hand, the generic Sbert model achieves the worst results
(wF1 = 0.31). This suggests that generic models are not necessarily better for
this specific task and a careful selection of the best generic LM is key to
the success of ZSL.

Based on the above findings, we can state that:  In the majority of the cases,
generic LMs perform better than domain-specific LMs on requirements
classification tasks.

Our findings thus contrast the claims that generic LMs do not perform
particularly well on domain-specific tasks, as they cannot recognize highly
domain-specific vocabulary [83, 84, 85, 11, 74].

From this we can conclude that generic LMs, being trained on generic data, are
more generalizable and adaptable,—the very sense of being generic; by
contrast, domain-specific LMs, being trained on domain-specific data, are less
generalizable and adaptable,—the very sense of being specific. Future
developments of LMs, we posit, should not differentiate between generic vs.
specific, but rather, should focus on continual learning on new tasks and new
data [49]. As LMs retain and accumulate knowledge across many tasks, they will
become more adaptable to new tasks, domain-specific or otherwise.

###  6.2 Best Label Configuration (RQ2)

From Section 5 we found:

  1. 1.

For Task FR/NFR, the best label configuration is FR_E for Sbert. This
configuration is composed of the Expert curated and the Original labels, which
identifies the NFRs using the names of the NFR classes (Usability, Security,
Availability, etc.). This shows that the knowledge of NFR characteristics
plays an important role on label configuration for this task.

  2. 2.

For Task NFR, we show the performance of label configurations in each sub-task
as follows:

     * (a)

For binary classification of NFR, the best label configuration is SE_D for
AllMini, which uses the word embedding with top-20 words for the SE class, and
the original NFR labels for the “Other” class.

     * (b)

For multi-class classification of NFR, in the majority of the cases, the best
label configurations for individual NFR classes are MultiNFR_A (Original
label) and MultiNFR_B (Expert curated label) for Sbert and AllMini.

     * (c)

For multi-label classification of NFR, simple label configurations based on
either original label (MultiNFR_A) and expert curated label (MultiNFR_B)
appear to be most effective for all classes, except US, for which the
embedding-based labels (MultiNFR_D and MultiNFR_E) are more effective. This
exception could be due to, in comparison with other NFR classes, that US
requirements require more contextual information to identify.

  3. 3.

For Task Security, the best label configuration is Sec_B. Curated by expert,
this label only contains a limited set of three security-related words.
However, the results show that it is sufficient to use this simple label
configuration to identify security requirements in the given dataset.

Based on the above findings, we can conclude that:  In general, simple label
configurations with the original class names or with a combination of original
and expert curated labels appear to be more effective than, more complex word-
embedding generated labels.

Selecting the best—more precisely, the most effective—label configuration is a
difficult task and requires testing many different labels by trial and error.
Our study shows how we have handcrafted each label using one of the three
strategies (namely, using original label, expert curated label and word-
embedding generated label). More work is needed in search for a more
systematical way of label configuration. We argue that expert knowledge of RE,
but also domain-, and possibly project-specific knowledge plays an important
part in choosing correct terms for the labels.

###  6.3 Effectiveness of ZSL for RE (RQ3)

Here we address the effective of ZSL by first comparing our best ZSL results
to the state-of-the-art results achieved by Kurtanovic̀ and Maleej (K&M) [8],
Hey et al. (NoRBERT) [10] and Knauss et al. (Knauss) [43], with respect to the
exactly same classification tasks (i.e., binary and multi-class
classification). Second, we discuss our best ZSL results obtained from multi-
label classification with the state-of-the-art results and provide our insight
into ZSL classification.

Table 19: Binary classification results obtained from Task FR/NFR compared to
the results obtained by Kurtanović & Maalej (2017) and Hey et al. (2020). Bold
values indicate the best performance results. Underlined values refer to the
best performance rates with the ZSL classifier. The last column “A” represents
the overall weighted F1 score across classes.

Reference | Approach (para. , train/test) | FR (255) | NFR (370) | A
---|---|---|---|---
P | R | F1 | P | R | F1
Kurtanović Maalej (2017) | SVM(word features, 10-fold) | 0.92 | 0.93 | 0.93 | 0.93 | 0.92 | 0.92 | 0.92
Kurtanović Maalej (2017) | SVM(best 100 features, 10-fold) | 0.86 | 0.51 | 0.63 | 0.65 | 0.92 | 0.76 | 0.71
Kurtanović Maalej (2017) | SVM(best 500 features, 10-fold) | 0.92 | 0.79 | 0.85 | 0.82 | 0.93 | 0.87 | 0.86
Hey el al. (2020) | NoRBERT (base+ep.16 121212ep. refers to the number of passes of the training dataset during LM learning process., 10-fold) | 0.89 | 0.88 | 0.89 | 0.92 | 0.93 | 0.92 | 0.91
Hey el al. (2020) | NoRBERT (large+ep.10+OS, 10-fold) | 0.92 | 0.88 | 0.90 | 0.92 | 0.95 | 0.93 | 0.92
Task FR/NFR | ZSL(Sbert + FR_E, all) | 0.55 | 0.82 | 0.66 | 0.82 | 0.54 | 0.65 | 0.65
ZSL(AllMini + FR_D, all) | 0.50 | 0.71 | 0.58 | 0.72 | 0.50 | 0.59 | 0.59
ZSL(Bert4RE + FR_D, all) | 0.47 | 0.65 | 0.54 | 0.67 | 0.50 | 0.57 | 0.56
ZSL(SObert + FR_C, all) | 0.50 | 0.41 | 0.45 | 0.64 | 0.72 | 0.68 | 0.59

Table 20: Binary classification results obtained from Task NFR compared to the
results obtained by K&M and NoRBERT. Bold values indicate the best performance
results. Underlined values refer to the best performance rates with the ZSL
classifier. The last column A represents the overall weighted F1 score across
classes. Only top 4 NFR classes are considered.

Approach(parameters) | US (67) | SE (66) | O (62) | PE (54) |  wF1
---|---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
10-fold cross val. on Top-4 NFR (249) | K&M(w/o features selection) | 0.81 | 0.85 | 0.82 | 0.91 | 0.90 | 0.88 | 0.72 | 0.75 | 0.73 | 0.93 | 0.90 | 0.90 | 0.83
K&M(best 50 features) | 0.70 | 0.57 | 0.61 | 0.81 | 0.77 | 0.74 | 0.78 | 0.50 | 0.57 | 0.87 | 0.57 | 0.67 | 0.65
K&M(best 500 features) | 0.80 | 0.71 | 0.74 | 0.74 | 0.81 | 0.74 | 0.72 | 0.73 | 0.71 | 0.87 | 0.81 | 0.82 | 0.75
NoRBERT(base + ep.10) | 0.81 | 0.69 | 0.74 | 0.93 | 0.82 | 0.87 | 0.80 | 0.53 | 0.64 | 0.88 | 0.80 | 0.83 | 0.77
NoRBERT(base + ep.10 - OS131313OS refers to Oversampling technique for randomly selecting data from the minority class by adding them to the training dataset Sampling ) | 0.78 | 0.70 | 0.74 | 0.90 | 0.86 | 0.88 | 0.88 | 0.71 | 0.79 | 0.88 | 0.80 | 0.83 | 0.81
NoRBERT(large+OS+ES141414ES refers Early Stopping, a feature that enables the model training to be automatically stopped when a selected metric (e.g., F1) has stopped improving) | 0.89 | 0.70 | 0.78 | 0.89 | 0.89 | 0.89 | 0.90 | 0.71 | 0.79 | 0.88 | 0.81 | 0.85 | 0.83
Test Top-4 NFR (249) w/o training | ZSL(Sbert + (NFR)_D) | 0.77 | 0.78 | 0.78 | 0.72 | 0.71 | 0.72 | 0.68 | 0.72 | 0.70 | 0.80 | 0.67 | 0.70 | 0.72
ZSL(Sbert + (NFR)_E) | 0.81 | 0.82 | 0.80 | 0.76 | 0.78 | 0.75 | 0.68 | 0.63 | 0.65 | 0.78 | 0.78 | 0.78 | 0.74
ZSL(AllMini + (NFR)_B) | 0.54 | 0.35 | 0.35 | 0.73 | 0.73 | 0.73 | 0.71 | 0.65 | 0.67 | 0.80 | 0.70 | 0.78 | 0.62
ZSL(AllMini + (NFR)_D) | 0.75 | 0.75 | 0.75 | 0.84 | 0.84 | 0.84 | 0.65 | 0.55 | 0.58 | 0.81 | 0.69 | 0.71 | 0.72
| ZSL(BERT4RE + (NFR)_C)
---
0.52 | 0.43 | 0.46 | 0.62 | 0.65 | 0.63 | 0.72 | 0.73 | 0.72 | 0.68 | 0.41 | 0.43 | 0.56
ZSL(BERT4RE + (NFR)_D) | 0.54 | 0.42 | 0.45 | 0.56 | 0.70 | 0.61 | 0.63 | 0.41 | 0.42 | 0.60 | 0.46 | 0.51 | 0.50
ZSL(SObert + (NFR)_B) | 0.55 | 0.51 | 0.53 | 0.81 | 0.74 | 0.63 | 0.65 | 0.73 | 0.67 | 0.61 | 0.78 | 0.68 | 0.62
ZSL(SObert + (NFR)_C) | 0.51 | 0.32 | 0.30 | 0.66 | 0.53 | 0.56 | 0.71 | 0.74 | 0.71 | 0.61 | 0.55 | 0.58 | 0.53

Table 21: Multi-class classification results obtained from Task NFR compared
to the results obtained by Kurtanović Maalej (2017) and Hey et al. (2020).
Bold values indicate the best performance results. Underlined values refer to
the best performance rates with the ZSL classifier. The last column A
represents the overall weighted F1 score across classes. Only top 4 NFR
classes are considered.

Approach (parameters) | US (67) | SE (66) | O (62) | PE (54) | A
---|---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
10-fold cross val. on Top-4 NFR (249) | K&M(word features) | 0.65 | 0.82 | 0.70 | 0.81 | 0.77 | 0.75 | 0.81 | 0.86 | 0.82 | 0.86 | 0.81 | 0.80 | 0.76
K&M(best 50 features) | 0.49 | 0.68 | 0.55 | 0.60 | 0.50 | 0.39 | 0.42 | 0.47 | 0.33 | 0.85 | 0.53 | 0.63 | 0.47
K&M(best 500 features) | 0.70 | 0.66 | 0.64 | 0.64 | 0.53 | 0.56 | 0.47 | 0.62 | 0.51 | 0.81 | 0.74 | 0.76 | 0.61
NoRBERT(base+ep.32) | 0.78 | 0.84 | 0.81 | 0.89 | 0.85 | 0.87 | 0.79 | 0.73 | 0.76 | 0.88 | 0.78 | 0.82 | 0.82
NoRBERT(large+ep.32) | 0.86 | 0.82 | 0.84 | 0.91 | 0.91 | 0.91 | 0.83 | 0.71 | 0.77 | 0.90 | 0.81 | 0.85 | 0.84
Test Top-4 NFR (249) w/o training | ZSL(Sbert + MultiNFR_B) | 0.73 | 0.43 | 0.54 | 0.69 | 0.64 | 0.66 | 0.53 | 0.71 | 0.61 | 0.48 | 0.57 | 0.52 | 0.58
ZSL(Sbert + MultiNFR_D) | 0.69 | 0.40 | 0.51 | 0.65 | 0.53 | 0.58 | 0.38 | 0.23 | 0.47 | 0.46 | 0.53 | 0.58 | 0.53
ZSL(AllMini + MultiNFR_A) | 0.67 | 0.36 | 0.47 | 0.66 | 0.92 | 0.77 | 0.33 | 0.32 | 0.33 | 0.45 | 0.50 | 0.47 | 0.51
ZSL(AllMini + MultiNFR_B) | 0.77 | 0.36 | 0.49 | 0.63 | 0.94 | 0.76 | 0.64 | 0.64 | 0.64 | 0.59 | 0.70 | 0.64 | 0.63
ZSL(AllMini + MultiNFR_C) | 0.54 | 0.45 | 0.49 | 0.76 | 0.67 | 0.71 | 0.36 | 0.37 | 0.37 | 0.41 | 0.54 | 0.47 | 0.51
ZSL(BERT4RE + MultiNFR_A) | 0.32 | 0.09 | 0.14 | 0.18 | 0.11 | 0.13 | 0.33 | 0.02 | 0.03 | 0.19 | 0.67 | 0.30 | 0.14
ZSL(BERT4RE + MultiNFR_B) | 0.33 | 0.60 | 0.42 | 0.00 | 0.00 | 0.00 | 0.28 | 0.53 | 0.37 | 0.13 | 0.02 | 0.03 | 0.21
ZSL(BERT4RE + MultiNFR_D) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.24 | 0.34 | 0.38 | 0.25 | 0.04 | 0.06 | 0.11
, ZSL(SObert + MultiNFR_A) | 0.00 | 0.00 | 0.00 | 0.60 | 0.05 | 0.08 | 0.22 | 0.71 | 0.33 | 0.12 | 0.09 | 0.10 | 0.13
ZSL(SObert + MultiNFR_B) | 0.30 | 0.81 | 0.44 | 0.00 | 0.00 | 0.00 | 0.29 | 0.16 | 0.21 | 0.08 | 0.06 | 0.07 | 0.19
ZSL(SObert + MultiNFR_C) | 0.31 | 0.36 | 0.33 | 0.24 | 0.21 | 0.22 | 0.31 | 0.56 | 0.40 | 0.00 | 0.00 | 0.00 | 0.25

Table 22: Comparison between the classification results obtained by Knauss et
al. (2011) and Task Security divided by the subset of SeqReq dataset: CPN,
GPS, and ePurse. Bold values indicate the best performance results. Underlined
values refer to the best performance rates with the ZSL classifier.

Approach (parameters) | wP | wR | wF1
---|---|---|---
SeqReq (510)
| 10-fold cross val.
---
on SeqReq (510)
Knauss et al. (Bayesian classifier) | 0.79 | 0.91 | 0.84
Test SeqReq (510) w/o training | ZSL(Sbert + Labels_Sec_C) | 0.58 | 0.41 | 0.31
ZSL(AllMini + Labels_Sec_B) | 0.68 | 0.65 | 0.66
ZSL(BERT4RE + Labels_Sec_A) | 0.52 | 0.53 | 0.52
ZSL(SObert + Labels_Sec_C) | 0.56 | 0.54 | 0.55
CPN (210)
Train on ePurse (124) | Knauss et al. (Bayesian classifier) | 0.23 | 0.54 | 0.33
Train on GPS (176) | Knauss et al. (Bayesian classifier) | 0.29 | 0.65 | 0.40
Train on ePurse + GPS (300) | Knauss et al. (Bayesian classifier) | 0.26 | 0.85 | 0.40
Test CPN (210) w/o training | ZSL(AllMini + Labels_Sec_B) | 0.79 | 0.77 | 0.78
GPS (176)
Train on ePurse (124) | Knauss et al. (Bayesian classifier) | 0.43 | 0.85 | 0.57
Train on CPN (210) | Knauss et al. (Bayesian classifier) | 0.29 | 0.19 | 0.23
Train on ePurse + CPN (334) | Knauss et al. (Bayesian classifier) | 0.51 | 0.56 | 0.53
| Test GPS (176)
---
w/o training
ZSL(SObert + Labels_Sec_C) | 0.63 | 0.63 | 0.63
ePurse (124)
Train on CPN (210) | Knauss et al. (Bayesian classifier) | 0.99 | 0.33 | 0.47
Train on GPS (176) | Knauss et al. (Bayesian classifier) | 0.72 | 0.48 | 0.58
Train on ePurse + CPN (386) | Knauss et al. (Bayesian classifier) | 0.84 | 0.31 | 0.46
Test ePurse (124) w/o training | ZSL(AllMini + Labels_Sec_A) | 0.64 | 0.70 | 0.69

  1. 1.

Binary Classification of FR vs. NFR: Table 19 shows that both K&M and NoRBERT
outperform all our ZSL classifiers. In particular, on FR, K&M produces the
best results with a SVM model that applies all the word features in the
PROMISE dataset (i.e., without feature selection), achieving F1 = 0.93. On
NFR, NoRBERT produces the best results with the fine-tuned BERTlarge model,
with F1 = 0.93. By contrast, the best ZSL classifier (with Sbert LM) has only
managed to achieve F1 = 0.66 on FR and F1 = 0.65 on NFR. On average, the
performance of the best ZSL classifier is 0.27 lower than that of K&M and
NoRBERT. Clearly, these results show that the ZSL approach is (much) less
effective than K&M and NoRBERT with respect to this particular task.

  2. 2.

Binary Classification of NFRs: Table 20 shows that overall, both K&M and
NoRBERT outperform our best ZSL classifier, with wF1 = 0.83 achieved by their
best model; on the other hand, the performance of our best classifier (ZSL
with Sbert) is only slighly worse, with wF1 = 0.73. By examining the results
obtained for each class, on US, our ZSL classifier (with Sbert) performs
slightly worse than K&M, but outperforms NoRBERT. On SE, although both K&M and
NoRBERT outperform our best ZSL classifier (with AllMini), the difference is
not large. A similar observation can be made to classes O and PE.

  3. 3.

Multi-class Classification of NFRs: For this task, we notice in Table 21 large
gaps between the results of K&M and NoRBERT and our results on every class. As
the purpose of this task is basically the same as the binary classification of
NFR task, the inconsistent results achieved by ZSL in these two tasks indicate
that when requirements belonging to many classes are considered, ZSL does not
appear to be sufficiently effective.

  4. 4.

Binary Classification of Security vs Non-Security Requirements: Table 22
reveals very interesting results: When treating all the security requirements
as a whole (i.e., without separating them into different project), Knauss
outperforms the best ZSL classifier by 0.18 points on wF1. However, when the
requirements are divided by three projects (i.e., CPN, GPS and ePurse), ZSL
outperforms Knauss on all individual projects. In particular, ZSL (AllMini +
Sec_B) achieved a high wF1 = 0.78, compared to Knauss’s wF1 = 0.40 on CPN.
This again seems to suggest that ZSL performs well with binary classification
of security requirements when opposite labels are clearly defined.

Based on the above findings, we can conclude that:

Unsupervised learning with ZSL achieves acceptable performance for binary and
multi-class classification tasks. However, it does not outperform supervised
classification models, as RE tasks are narrowly defined, and often require
well-trained, specifically fine-tuned models on specifically labelled dataset.
Nevertheless, without training or fine-tuning, ZSL is more flexible, open to
less data-rich tasks, and easily adaptable to the evolution of classification
schemes.

In fact, we suggest that ZSL is better to be viewed as a universal multi-label
classification approach. Intuitively, the more the labels, the better the
chance to find a match between a requirement and a label. In our study, we
have demonstrated how we fit conventional binary and multi-class
classification as one-label or single label classification in ZSL and we still
achieved encouraging results.

In relation to multi-label classification, the following results are obtained:

  * 1.

From Table 13, concerning the 4 largest NFR classes, best performance for each
class are F1 $\sim 0.83-0.94$, which are comparable with the average results
of NoRBERT (large + ep.32) for multi-class classification (average F1 = 0.84,
cf. Table 21), and higher than those of K&M.

  * 2.

From Table 14, concerning all NFR classes, we observe that the best F1 $\sim
0.64-0.98$, with variations that depend on the requirements class.

To achieve state-of-the-art performance of ZSL for multi-class classification,
a multi-label strategy is recommended. In practice, this implies that a semi-
automated classification approach should be followed, in which a human
operator is asked to select the most suitable class among the top ones
returned by the classifier.

##  7 Threats to Validity

##### Construct Validity

The main threat to construct validity in our study are the adopted concepts of
FR and NFRs. As previously discussed, this is an artificial distinction that
doe not often apply in practice [23], as non-functional requirements are
better identified as qualities [25], and the classification is frequently non-
binary, i.e., a multi-label classification. However, FR/NFR is a traditional
distinction, still common in the industrial practice, and in research.
Furthermore, ZSL is inherently designed for single-label classification, and
its usage for multi-label classification—in case binary classification— would
introduce the need for threshold values in the classification (i.e., when both
classes has a correlation score above a certain threshold, then classify the
requirement as both FR and NFR). To have rigorous results, the threshold value
would need to be assessed with sensitivity analysis which are not realistic
with the limited datasets available, thus leading to non-representative
thresholds. For this reason, we excluded the annotation of PROMISE by Dalpiaz
et al. [9] from our evaluation. For security vs non-security requirements, the
same observations as for FR/NFR hold. Finally, the adopted metrics for
evaluation (precision, recall, weighted F1, accuracy) are those typically used
for ML systems, so we do not foresee any major construct validity issue on
this aspect.

##### Internal Validity

This is an experiment with software subjects, and the intervention of the
researchers is limited, thus granting minimal researcher’s bias. In the
evaluation, we have used established and widely used annotated datasets from
the literature. Therefore, the internal validity threats are somewhat
inherited from the labelling performed by previous work. While the labelling
of the PROMISE dataset has been questioned by previous work [9,
10]—requirements have been labelled by students, classification is binary, and
some classes are not sufficiently represented—it represents a classical
benchmark, which can be used to compare our results with previous state-of-
the-art proposals. Concerning internal threats due to implementation issues,
we have adopted widely used LMs, made available by Hugging Face (domain-
generic models), or developed by authors of the software engineering community
(domain-specific models). These models have been tested in other environments,
thus increasing the confidence on their reliability. Concerning the
implementation of the ZSL algorithm, we have used the Transformers package in
Python to retrieve the LMs from HuggingFace hub and apply encoding for the
labels and requirements representations. However, this package is also widely
used, and we have made our code available for inspection in a google Colab
Notebook, so that results can be replicated.

##### External Validity

The PROMISE dataset include requirements written by students, which may not be
representative of industrial requirements. Instead the SecReq dataset includes
requirements from three industrial projects. Our evaluation of ZSL is limited
to the three tasks of FR/NR, NFR, and Security, and therefore our conclusion
apply only to these tasks. Different results may be obtained when other
classification schemes are used, or other types of requirements-related
information (e.g., user stories, or app reviews) is used. Another limitation
is the experimentation with the embedding-based ZSL approach only, and not
considering other ZSL strategies, as, e.g., the entailment approach. This is
due to the exploratory nature of the study, which is focused on the comparison
of different types of LM, and different labelling strategies. Future work will
provide a thorough comparison with other ZSL approaches, stemming from our
results. Concerning the coverage of possible embedding-based ZSL
configurations, we have considered both state-of-the-art domain-generic LMs
(SentenceBERT, and All Mini LM) and domain-specific ones (BERT4RE and
BERTOverflow), all based on the BERT LM, which is widely used and has been
shown to achieve high performance also for RE classification tasks [10].
Furthermore, we have used multiple labelling strategies, designed for the
specific tasks addressed. Therefore, we argue that our analysis can be
considered representative of the usage of embedding-based ZSL for requirements
classification, using LMs derived from BERT.

##  8 Conclusion

This study presents an investigation of the performance of zero-shot learning
(ZSL) for the classification of requirements. We consider three main
classification tasks, namely: 1) FR/NFR—binary classification into functional
and non-functional requirements, 2) NFR—binary and multi-class classification
into different non-functional classes, e.g., performance, usability, etc.,
also considering the multi-label case and 3) Security—classification of
security vs non-security requirements. For each task, we identify the most
suitable language model (LM) to be used for requirements representation and
the most suitable label selection strategy, as the encoding of requirements as
well as the terms used as labels to be predicted influence the performance of
the approach. Our results show that generic LM, and simple labelling
strategies perform better than software engineering/RE-specific LMs, and
complex labelling strategies. Best performance are: F1=0.66 for FR/NFR;
F1=$\sim 0.72-0.80$ for binary NFR; up to F1=76 for multi-class single label;
up to F1=0.98 for multi-class multi-label NFR; F1=0.66 for Security, and up to
F1=0.78 for part of the Security dataset. Though performance are lower with
respect to previously explored supervised techniques, it is worth highlighting
that ZSL is fully unsupervised. Therefore, it does not require a pre-labelled
dataset, except for its initial evaluation, to be carried out to check its
performance. Furthermore, it is inherently flexible to changes in the types of
requirements classes to consider, and associated labels. As classification
schemes change, the approach naturally adapts to the new labelling. As future
work, we will consider the following directions: 1) assess ZSL for the
classification of app reviews, using existing datasets made available by
previous studies (cf., Dabrowski for a complete list [15]); 2) consider other
RE tasks, and corresponding strategies to approaches to frame them in terms of
classification problems suitable for ZSL; 3) replicate current experiments
with the entailed-based ZSL approach, to explore whether better performance
can be achieved; 4) consider the few-shot learning approach, and assess to
what extent the shortcomings of ZSL can be addressed by including a limited
set of labelled examples.

## Replication

We shared our experimentation settings including Colab notebook and the
results we obtained from all the ZSL classifiers at
https://github.com/waadalhoshan/ZSL4REQ.

## References

  * [1] L. Zhao, W. Alhoshan, A. Ferrari, K. J. Letsholo, M. A. Ajagbe, E.-V. Chioasca, R. T. Batista-Navarro, Natural language processing for requirements engineering: A systematic mapping study, ACM Computing Surveys (CSUR) 54 (3) (2021) 1–41.
  * [2] M. Kassab, C. Neill, P. Laplante, State of practice in requirements engineering: contemporary data, Innovations in Systems and Software Engineering 10 (4) (2014) 235–241.
  * [3] J. A. Sidey-Gibbons, C. J. Sidey-Gibbons, Machine learning in medicine: a practical introduction, BMC medical research methodology 19 (1) (2019) 1–18.
  * [4] I. H. Sarker, Machine learning: Algorithms, real-world applications and research directions, SN Computer Science 2 (3) (2021) 1–21.
  * [5] S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, J. Gao, Deep learning–based text classification: a comprehensive review, ACM Computing Surveys (CSUR) 54 (3) (2021) 1–40.
  * [6] W. Alhoshan, L. Zhao, A. Ferrari, K. J. Letsholo, A zero-shot learning approach to classifying requirements: A preliminary study, in: International Working Conference on Requirements Engineering: Foundation for Software Quality, Springer, 2022, pp. 52–59.
  * [7] J. Cleland-Huang, R. Settimi, X. Zou, P. Solc, Automated classification of non-functional requirements, Requirements engineering 12 (2) (2007) 103–120.
  * [8] Z. Kurtanović, W. Maalej, Automatically classifying functional and non-functional requirements using supervised machine learning, in: 2017 IEEE 25th International Requirements Engineering Conference (RE), Ieee, 2017, pp. 490–495.
  * [9] F. Dalpiaz, D. Dell’Anna, F. B. Aydemir, S. Çevikol, Requirements classification with interpretable machine learning and dependency parsing, in: 2019 IEEE 27th International Requirements Engineering Conference (RE), IEEE, 2019, pp. 142–152.
  * [10] T. Hey, J. Keim, A. Koziolek, W. F. Tichy, Norbert: Transfer learning for requirements classification, in: 2020 IEEE 28th International Requirements Engineering Conference (RE), IEEE, 2020, pp. 169–179.