11institutetext:  Blekinge Institute of Technology, Valhallavägen 1, 371 41
Karlskrona, Sweden 11email<EMAIL_ADDRESS>22institutetext:
fortiss GmbH, Guerickestraße 25, 80805 Munich, Germany 22email:
<EMAIL_ADDRESS>
# NLP4RE Tools: Classification, Overview, and Management

Julian Frattini 11 0000-0003-3995-6125 Michael Unterkalmsteiner 11
0000-0003-4118-0952 Davide Fucci 11 0000-0002-0679-4361 Daniel Mendez 1122
0000-0003-0619-6027

###### Abstract

Tools constitute an essential contribution to natural language processing for
requirements engineering (NLP4RE) research. They are executable instruments
that make research usable and applicable in practice. In this chapter, we
first introduce a systematic classification of NLP4RE tools to improve the
understanding of their types and properties. Then, we extend an existing
overview with a systematic summary of 126 NLP4RE tools published between April
2019 and June 2023 to ease reuse and evolution of existing tools. Finally, we
provide instructions on how to create, maintain, and disseminate NLP4RE tools
to support a more rigorous management and dissemination.

###### Keywords:

Natural language Processing Requirements Engineering Tool Open science

##  1 Introduction

The prevalence of unrestricted natural language (NL) to specify requirements
in the requirements engineering (RE) phase of a software development project
[25] complicates approaches for automation due to the inherent complexity and
ambiguity of NL. Natural language processing (NLP) techniques meet this
challenge through the application of machine and deep learning techniques to
NL [15]. An essential contribution to this natural language processing for
requirements engineering (NLP4RE) research domain are NLP4RE tools. They
constitute a primary research artifact to transfer technology into practice
[34] by automating a variety of tasks relevant to RE or by providing decision
support to managers in practice. These tasks include requirements elicitation,
e.g., by mining new requirements from a large body of user reviews [56] or by
extracting relevant clauses from regulatory text [3], but also requirements
management, i.e., retrieving dependencies between requirements among each
other [14] or between requirements and other software artifacts [54],
requirements modeling, e.g., the transformation of NL requirements into cause-
effect graphs [27] or state models [42], and validation and verification, like
automatically or semi-automatically generating test cases from NL requirements
[26]. The degree of support ranges from attempts at full automation [23, 26]
to recommendations aiming to ease manual tasks [54].

However, developing, deploying, maintaining and disseminating NLP4RE tools is
challenging and sometimes even unnecessary, as similar tools may already
exist, yet overlooked due to the lack of a comprehensive overview. To meet
these challenges and equip the reader with the knowledge to contribute to the
landscape of NLP4RE tools, this chapter treats the following three topics:

  * •

Classification (Section 2): a systematic classification of NLP4RE tools to
categorize them by relevant attributes.

  * •

Overview (Section 3): a comprehensive overview of existing NLP4RE tools to
show which tools are already available and can be reused instead of re-
implementing already existing solutions.

  * •

Management Guidelines (Section 4): instructions on developing, documenting,
and disseminating NLP4RE tools to make contributing to the landscape of
existing tools more approachable.

##  2 Classification

This section introduces a classification of NLP4RE tools. This classification
allows for a systematic understanding of the types of tools and the attributes
in which they differ, but also aids in reporting new NLP4RE tools. The
classification is general enough to be applicable to any RE tool, not just
those processing NL requirements, which allows future extensions to a larger
scope. Adherence to this general classification scheme is a first and
accessible step towards a more consistent reporting of NLP4RE tools, and it
paves the way for future, more comprehensive approaches [1].

The classification is derived from a previous NLP4RE tool classification
scheme [60], and complemented by an ontology of requirements quality factors
[28]. The tool classification scheme stems from a seminal, comprehensive
secondary study on NLP for RE [60]. The ontology extends several aspects of
this scheme, especially regarding the type and availability of the tool [28],
which improves the overview of the tools’ accessibility.

We focus on the attributes listed in Table 1. Section 2.1 presents the
attributes describing the identification of the tool, i.e., their name and
description. Section 2.2 details attributes of the artifact itself, i.e., the
link to the source, its release type, availability, and license. Section 2.3
specifies the task which the tool performs in terms of supported RE activity
and task type. This selection of attributes suffices, in our experience, to
categorize NLP4RE tools at a high level and it serves the purpose of providing
an overview of NLP4RE tools. For a more detailed classification, the
interested reader may refer to the NLP4RE ID card by Abualhaija et al. [1].
Table 2 lists the classification of two tools [22, 54] which will be used as
running examples in the following subsections.

Table 1: Attributes of NLP4RE tools Attribute | Type |  Values
---|---|---
Identification
Name | text |
Description | text |
Artifact
Source | text |  URL
Release | categorical |  {source code, executable}
Availability | categorical |  {archived, open source, reachable, upon request, broken, unavailable, private, proprietary}
License | categorical |  {MIT, Apache 2.0, GPL, … }
Task
RE Activity | categorical |  {elicitation, analysis, modeling, validation & verification, management, other}
Task type | categorical |  {modeling, detection, extraction, classification, tracing & relating, search & retrieval}
Table 2: Characterization of the CiRA [22] and TT-RecS tools [54]
Identification
---
Name |  CiRA |  TT-RecS
Description |  A tool that classifies a natural language requirements sentence as either causal or non-causal by leveraging a fine-tuned BERT model |  The tool supports the annotation of requirements with concepts from a domain-specific taxonomy, providing suggestions which concepts fit to the requirement content. The goal is to enable traceability through the use of a taxonomy, assuming that downstream artifacts are also classified.
Artifact
Source |  https://zenodo.org/record/8033347 |  https://zenodo.org/records/3827169
Release |  Source code |  Source code
Availability |  Archived |  Archived
License |  Apache 2.0 |  Apache 2.0
Task
RE Activity |  Validation & Verification |  Management
Task type |  Classification |  Tracing & Relating

###  2.1 Identification

Every NLP4RE tool should be characterized by a descriptive name and a
comprehensive description, though many tools do not bear a distinct name.
There are no restrictions to tool names, but one popular pattern is to name a
tool after the abbreviation of a concise description of its objective, e.g.,
the CiRA classifier, a causality in requirements artifacts classifier [22], or
TT-RecS, a taxonomic trace recommender system [54].

The description of a tool should fulfill two purposes: (1) it should describe
the main objective of the tool, and (2) it should explain the key technologies
used. The first purpose is tailored towards users of the tool who are only
interested in a black-box view of its functionality. The second purpose
pertains towards researchers and developers who want to assess the tool’s
technological components and assess whether and how they could improve it. For
example, the descriptions of the two tools provided in Table 2 contain black-
box input and output combination as well as information about the tools’ core
technology. While scientific manuscripts—other than technical reports—often
constrain these descriptions to a very high level, this purpose can be
achieved to a much greater degree when properly documenting the tool by
describing the processing pipeline, adding architectural views, and specifying
the core algorithms using pseudo code. The repository hosting the tool
described in a manuscript is the appropriate location for extending these
technological descriptions, as further explained in Section 4.1.2.

###  2.2 Artifact

The source link is a URL pointing to the location of the tool, like
https://zenodo.org/record/8033347 for the CiRA classifier [22]. The type of
release denotes what kind of artifact has been made available. For tools, the
type of release is usually either source code or a pre-compiled executable,
but in rare cases authors release both. The degree to which the tool can be
accessed constitutes the availability of a tool. Table 3 shows an eight-tier
ranking of availability levels proposed in prior research [29].

Table 3: Levels of availability [29] Status |  Explanation
---|---
Archived |  The tool is hosted in a service that satisfies the following criteria: (1) immutable URL (cannot be altered by the author or someone else), (2) permanent (the hosting organization has a mission to maintain artifacts for the foreseeable future), and (3) accessible (there is a DOI pointing to the real data source URL)
Open Source |  The source code is disclosed and contains an opensource license which grants access and re-use
Reachable |  The tool is reachable now but missing some of the archived or open source aspects
Upon request |  Authors claim the tool is available upon request
Broken |  A link to the tool is contained in the paper, but it does not resolve
Unavailable |  A tool is presented, but no indication on how to access it is provided
Private |  The authors state that a tool exists, but it is private for some reasons (such as industry collaboration with private data, etc.)
Proprietary |  The tool is proprietary, and access is granted upon payment

In the best case, tools are archived properly, which ensures their long-term
availability. In most cases, however, tools are simply unavailable without
explanation. This significantly inhibits adoption, replication, and evolution
of contributions to the NLP4RE research domain [29]. Section 3 presents the
current distribution of tool availability and Section 4 guidelines to ensure
availability.

The fourth attribute of the artifact category, the license, has a major
influence on its availability. A license specifies if and how a third party
can use the artifact and is, hence, imperative for the reuse and evolution of
NLP4RE tools. Section 4.2.2 describes the types and properties of licenses in
more detail.

###  2.3 Task

Finally, Zhao et al. [60] classify NLP4RE tools based on the RE activity,
which the tool intends to support, e.g., elicitation or analysis, and the task
type, which the tool employs, e.g., classification or extraction. These two
attributes are orthogonal to each other, i.e., different tasks can be used to
facilitate different RE activities. The RE activity attribute denotes the
high-level requirements engineering activity that the tool supports. This
attribute contains the following categories adapted from Zhao et al. [60]:

  * •

Elicitation: Emergence of new requirements

  * •

Analysis: Assessment of requirements specifications through attribution and
evaluation

  * •

Modeling: Translation of requirements into different representations (e.g.,
UML models or code)

  * •

Validation & Verification: Assertion of whether the behavior of a system under
test matches the expected behavior as specified in requirements

  * •

Management: Coordination and evolution of requirements and their
interrelations

  * •

Multiple: Several of the above

  * •

Other: Any RE-activity not covered by the aforementioned

As specified in Table 2, the CiRA classifier supports the RE activity
validation & verification, as the classification serves to identify causal
requirements which can be automatically translated into test cases [23]. The
TT-RecS tool supports the RE activity management, as it recovers trace links
between requirements and other software requirements [54].

The task type denotes the NLP task which the tool is fulfilling. Where the
previously explained RE activity determines what a tool is supposed to do, the
task type defines how it achieves this goal. Note that an NLP4RE tool can
employ different task types to achieve a similar goal. The task type attribute
contains the following categories adapted from Zhao et al. [60]:

  * •

Generation: Formation of new, requirements-relevant text

  * •

Detection: Detecting linguistic issues

  * •

Extraction: Identifying key domain abstractions and concepts

  * •

Classification: Classifying NL text into different categories

  * •

Transformation: Translation of text into a semantically similar but
syntactically different specification

  * •

Tracing & Relating: Establishing a relationship between requirements and/or
other software artifacts

  * •

Search & Retrieval: Searching and retrieving requirements-relevant information
from existing corpora

For example, the CiRA tool fulfills a classification task to distinguish
causal from non-causal requirements sentences. The TT-RecS tool fulfills a
tracing & relating task to recover trace links between requirements and other
software artifacts. The two attributes presented in this section, RE activity
and task type, constitute the essence of an NLP4RE tool: The task type
represents the NLP part and the RE activity the RE part of the tool. The
previously introduced description attribute can easily be constructed from
these two attributes.

##  3 Overview

The availability of NLP technology has given rise to a large number of NLP4RE
tools [60]. The abundance of available tools makes it difficult to maintain an
overview, causing too often overlapping and competing research endeavors [28].
Consequently, systematic studies updating an overview and categorization of
existing tools are necessary. This chapter continues the overview over NLP4RE
tools initiated by Zhao et al. [60]. Section 3.1 explains the method of
extraction, Section 3.2 presents the obtained results, and Section 3.3
discusses them.

###  3.1 Method

Zhao et al. contributed a first, systematic overview of the NLP4RE landscape
in their seminal mapping study [60]. This study also included an investigation
of NLP4RE tools up until April 2019. We continued this systematic study of
NLP4RE tools. All study material can be found in our replication
package.111https://zenodo.org/doi/10.5281/zenodo.8341439

####  3.1.1 Article Selection

As data sources, we selected the twelve leading publication venues for NLP4RE
research as identified by Zhao et al. [60] listed in the left-most column of
Table 4. This list included RE- and NLP-specific venues like the Requirements
Engineering Conference222https://requirements-engineering.org/ and the NLP4RE
workshop333https://nlp4re.github.io/2023/ but also general software
engineering venues like the Journal of Systems and
Software444https://www.sciencedirect.com/journal/journal-of-systems-and-
software and the Transactions in Software Engineering
journal.555https://www.computer.org/csdl/journal/ts

Table 4: Number of included articles (journals, conferences, and workshops) Venue | Acro. | 2019 | 2020 | 2021 | 2022 | 2023 | Total
---|---|---|---|---|---|---|---
Transactions on Software Engineering | TSE | 0 | 1 | 0 | 1 | 2 | 4
Journal of Systems and Software | JSS | 0 | 2 | 0 | 1 | 1 | 4
Information and Software Technology | IST | 3 | 1 | 3 | 2 | 2 | 11
Requirements Engineering Journal | REJ | 1 | 1 | 2 | 2 | 0 | 6
Data & Knowledge Engineering Journal | DKE | 0 | 2 | 0 | 0 | 0 | 2
International Conference on Software Engineering | ICSE | 0 | 0 | 1 | 4 | 5 | 10
Automated Software Engineering | ASE | 0 | 3 | 0 | 2 | 0 | 5
Requirements Engineering Conference | RE | 13 | 12 | 13 | 7 | 0 | 45
Requirements Engineering: Foundations of Software Quality | REFSQ | 2 | 0 | 1 | 3 | 2 | 8
Symposium on Applied Computing | SAC | 0 | 1 | 3 | 0 | 0 | 4
Artificial Intelligence for Requirements Engineering | AIRE | 3 | 6 | 6 | 2 | 0 | 17
Natural Language Processing for Requirements Engineering | NLP4RE | 1 | 1 | 5 | 2 | 1 | 10
Total |  | 23 | 30 | 34 | 26 | 13 | 126

The first three authors reviewed all publications in these twelve venues from
April 2019 until June 2023, as far as available up to the time of conducting
this study, by reading the title, abstract, and introduction. For each
article, we determined its potential relevance by deciding the two initial
inclusion criteria PIn1 and PIn2 listed in Table 5. Initial inclusion
criterion PIn1 was necessary to filter the generic venues for RE-relevant
articles. Initial inclusion criterion PIn2 was necessary to filter for
articles potentially presenting tools. The initial inclusion produced 202
eligible articles. After the selection of candidate articles, we conducted a
proper inclusion and exclusion phase with the criteria In1-In3 and Ex1-Ex3
listed in Table 5. After the inclusion and exclusion phase, 126 relevant
articles remained. Table 4 lists the number of included articles per venue per
year.

Table 5: Preliminary inclusion (PIn), final Inclusion (In) and exclusion (Ex) criteria for candidate articles Index |  Criterion
---|---
PIn1 |  The article is either published in a RE-centric venue or the title, abstract, or introduction explicitly mention the area of RE.
PIn2 |  The title, abstract, or introduction mention any sort of software or automation hinting at a tool.
In1 |  The article explicitly declares the relevance of the presented results to the area of requirements engineering by making clear which RE activity the tool supports.
In2 |  The article presents an automatic or semi-automatic algorithm and is an original contribution of the paper.
In3 |  The algorithm explicitly reports to involve at least one technology processing natural language, either as input or output, which is not limited to statistical NLP.
Ex1 |  The article is not accessible freely or through institutional access programs.
Ex2 |  The article is not written in English.
Ex3 |  The article is not extended by or a duplicate of an already included article.

####  3.1.2 Data Extraction and Classification

We prepared guidelines for the data extraction and classification phase, which
can also be found in our replication package. The extraction guidelines
contained one section for each category mentioned in Section 2. Each section
of the extraction guidelines consisted of a description of the attribute, the
purpose of its extraction, the extraction rules to apply, and examples. We
distributed the extraction and classification of candidate articles among the
first three authors based on their availability. Each author performed the
extraction and classification individually but took note of any difficulty
encountered during the process. These difficulties were discussed and resolved
jointly among the three authors in a final meeting.

####  3.1.3 Analysis and Dissemination

Finally, we analyzed the extracted data by generating descriptive statistics
of the distribution of the extracted attributes. Additionally, we disseminated
the results by creating a curated list on
GitHub.666https://github.com/JulianFrattini/nlp4re-
tools/blob/main/tools/nlp4re-tools.md With this list, we provide an up-to-date
overview of the NLP4RE tools landscape. The overview shall support researchers
in finding and accessing available NLP4RE tools. Furthermore, we invite fellow
researchers to contribute to the list using GitHub’s collaboration features in
order to keep it up-to-date.

###  3.2 Results

####  3.2.1 NLP4RE Tools

126 of the 202 candidate articles contained an NLP4RE tool according to our
inclusion and exclusion criteria. The full list of tools with all extracted
attributes can be found online.6 Figure 1 shows the distribution the RE
activity and tool task codes among the extracted tools.

Figure 1: Distribution of RE activity and task type among tools

The distribution highlights several focal points of research presenting NLP4RE
tools. The requirements elicitation activity is most prominently supported by
extraction and classification tools. Classification tools for requirements
elicitation focus on identifying information that is potentially relevant to
the elicitation of new requirements from textual corpora, e.g., user reviews
on app stores. Techniques like deep learning [44] and transfer learning [36]
are applied to classify reviews into relevant (e.g., feature request) and
irrelevant (e.g., noise) groups. On the other hand, extraction tools for
requirements elicitation fully automate the elicitation process by also
consolidating the resulting set of relevant data into requirement proposals.
Tools like OpenReq Analytics [53] or RE-BERT [4] extract new requirements from
app reviews, but there are also examples of tools considering different
sources, like MaRK-II [41], which processes domain documents.

The requirements analysis activity is mostly supported by detection,
extraction, and classification tools. Detection tools for requirements
analysis identify potential requirements quality defects and include tools
like the automatic ambiguity detector [17] and MAANA [18]. Extraction tools
for requirements analysis support information retrieval tasks like REVV-light
for near-synonymy detection [12] or Co-AI for retrieving abstractions [47],
i.e., significant domain terms. Most prominently, classification tools for
requirements analysis enrich requirements with additional information like a
categorization of an NL requirements as functional or non-functional [37] or
the distinction of requirements from prose in large specifications [19].

The requirements modeling activity is mainly supported by extraction and
transformation tools. Extraction tools for requirements modeling focus on
retrieval of relevant concepts from natural requirements like i-Star elements
[58] or system states [48]. Transformation tools further automate the process
by proposing actual models from NL requirements, for example full i-Star
models [8] or cause-effect diagrams [27].

The validation and verification task is mostly supported by extraction tools.
These usually attempt to generate test cases from NL requirements by
extracting causal diagrams [24] and inferring test configurations from them
[38]. Classification tools to identify NL requirements containing causal
information, like the CiRA tool [22] mentioned in Section 2, further aid this
process.

The management activity is predominantly aided by tracing and relating tools.
These tools recover trace links between requirements themselves [16] or
between requirements and other SE artifacts [54]. Only very few tools support
multiple RE activities. Among these are, for example, very general algorithms
like zero-shot learning for a variety of requirements classification tasks
[2].

####  3.2.2 Artifact Availability

Figure 2 visualizes the status of availability of the extracted tools. Figure
2(a) shows the distribution of availability codes as per the ranking
introduced in Table 3. Figure 2(b) summarizes the licenses used by the
extracted tools as well as their frequency in our sample. Note that the 61
unknown licenses are caused by 52 tools being unavailable in addition to the 7
tools where the source link was broken and the 2 tools that are only available
upon request. The 29 tools that were categorized as reachable consist of the
28 tools with license type none plus one tool published under the Microsoft
Reference Source License (MS-RSL) [35], which does not count as open source.

(a) Distribution of availability codes

(b) Distribution of used licenses

Figure 2: Visualization of tool availability

###  3.3 Discussion

####  3.3.1 Interpretation

The number of NLP4RE tools contained in publications appearing in the last
five years of the 12 most prominent venues featuring NLP4RE research speaks
for an active field of research. Note, however, that this number (126 tools)
cannot be directly compared to the 130 tools extracted by Zhao et al. [60] in
the time frame of 1983-2019, since the sampling and inclusion strategy
differed.

The distribution of implemented task types to support various RE activities
shows clear focal points in NLP4RE research. Extraction and classification
tools for requirements elicitation, classification tools for requirements
analysis, and tracing and relating tools for requirements management are
favored combinations. On the other hand, fundamental algorithms that apply NLP
for the particular use in RE but can be used to support multiple activities
like zero-shot learning [2] are rare. Additionally, tools that perform a
search and retrieval or generation tasks barely appear in our sample. The
recent rise of large-language models (LLMs) in the broader scope of generative
artificial intelligence bear the opportunity to change this. First
applications of LLMs to requirements elicitation [50] and inconsistency
detection [20] fall outside of the sampled primary studies due to recency, but
show first promising results.

The availability of tools as visualized in Figure 2(a) shows a more positive
trend towards open source and scientific archival than previous studies [29].
The commendable availability of a large portion of the tools ($29+21+15=65$
tools (51.6%) have an availability status reachable or above) correlates with
their recency, as the likelihood of authors publishing their artifacts has
increased over time [45] and the availability of non-archived artifacts
deteriorates with age [57]. However, the overview shows two remaining
shortcomings regarding the availability of NLP4RE tools. Firstly, almost half
of the available tools (availability level reachable, $29/65\approx 44.6\%$)
have no license specified, which impedes their reuse [51]. Secondly, only
$15/65\approx 23.1\%$ of the available tools are properly archived, which
risks their persistence [57]. The guidelines concluding this chapter, Section
4, aim to provide practical guidance to overcome these shortcomings.

####  3.3.2 Challenges and Limitations

The extracted tools exhibit different levels of maturity. The maturity of
NLP4RE tools ranges from repositories simply containing the necessary scripts
and data for replicating a study, like the NFR classification
tool777https://github.com/rgnana1/NFR_Classification_RNN_LSTM from
Gnanasekaran et al. [33], up to well-maintained and -documented tools like the
dependency detection tool888https://github.com/OpenReqEU/dependency-detection
from the OpenReq initiative by Motger et al. [16] or the ConDec Jira
Plugin999https://github.com/cures-hub/cures-condec-jira by Kleebaum et al.
[40]. The maturity of tools has a significant impact on their usability but
was not analyzed during this study. Assessing the technology readiness level
[43] of the extracted tools may provide further insights on reusability.

The extraction of the RE activity and tool’s task type categories posed
challenges for several tools. In some cases, the terms used by the authors of
a publication did not match the definitions of the types of these categories.
For example, Sainani et al. labeled the task type of their tool as extraction
even though the actual NLP task performed by the tool is a classification
[52]. In these cases, we opted for the most appropriate term according to our
classification scheme to maintain consistency within our sample. In other
cases, tools employed multiple task types. For example, Kifetew et al. present
a tool which both associates user feedback with system features (task type
tracing & relating) and then evaluates the sentiment of the user feedback
(task type extraction) [39]. In these cases, we recorded only the most
dominant task type. However, we reported all of the encountered challenges in
a separate sheet contained in our replication package to support the potential
extension of our extraction.

##  4 Management

The final section of the chapter provides guidelines aiding the development
and management of NLP4RE tools, mainly focusing on two stages of the life
cycle of tool development. Section 4.1 discusses the development and
documentation of new and the contribution to existing tools. Section 4.2
provides guidance on disseminating tools, including licensing, version
control, and reporting.

###  4.1 Development and Documentation

Developing and documenting NLP4RE tools in a way that supports maintenance,
reuse, and improvement is challenging, but essential to ensure a tool provides
value to the NLP4RE community.

####  4.1.1 Tool Development

Prior to developing a new NLP4RE tool, an overview of existing tools—for
example, the initial list of NLP4RE tools provided by Zhao et al. [60] and our
updated list6—should be consulted to determine if a similar tool has already
been proposed and disclosed. For this, classify the intended tool according to
the categories presented in Section 2.3 and identify the subset of tools in
the same class. Review the publications presenting these tools as well as
their source—if available—to identify potential candidates that either fully
cover the intended goal of the tool, that could be tailored towards it, or
that contribute one step of the necessary pipeline. In case an intended tool
does not yet exist, selecting the right technology for implementing a tool
becomes relevant. Table 6 lists potent libraries and frameworks for different
programming languages to consider. Even more NLP resources for different
programming languages can be found at https://github.com/keon/awesome-nlp.

Table 6: NLP resources for different programming languages Language | Resource |  Link
---|---|---
Java | Apache OpenNLP |  https://opennlp.apache.org/)
| Apache UIMA |  https://uima.apache.org/
| Stanford CoreNLP |  https://stanfordnlp.github.io/CoreNLP/
| DKPro |  https://dkpro.github.io/
| GATE |  https://gate.ac.uk/
Python | NLTK |  https://www.nltk.org/
| spaCy |  https://spacy.io/
| HuggingFace |  https://huggingface.co/models
| LangChain |  https://python.langchain.com/
Javascript | LangChain |  https://js.langchain.com/

For some tool types, reference architectures guiding the development of new
tools exist. For example, Dabrowksi et al. have analyzed user feedback miners
for app store reviews, a popular NLP4RE tool type, and distilled a reference
architecture from the elicited problem domain [10]. Empirically derived
reference architectures provide great guidance during tool development and
should be carefully considered when designing a new tool.

Furthermore, training data needs to be prepared as the quality of every NLP
tool depends on the quality of its data. In case it is impossible to procure
data from industry partners, sources like
Kaggle,101010https://www.kaggle.com/datasets HuggingFace
datasets,111111https://huggingface.co/datasets or the requirements quality
factor ontology121212www.reqfactoront.com [28] are valuable starting points.
Another valuable resource is the component-oriented synthetic textural
requirements generator CORG [59], a tool that “automatically generate[s]
comprehensive combinations of structurally-diverse synthesised textual
requirements” [59] from a dictionary of domain words and verb
frames.131313https://github.com/ABC-7/CORG

####  4.1.2 Documentation

An NLP4RE tool—as any piece of software—ought to be well documented to allow
its user to understand, use, and evolve it. At the minimum, this requires a
README.md file located in the top-most directory covering the following
sections [46]:

  1. 1.

Summary: a one-paragraph summary of the purpose and objective of the tool

  2. 2.

Contributor details: explanation of the ownership and contribution, and
information on how to reach out to the contributors

  3. 3.

Description: a detailed description of the components of the tool

  4. 4.

Setup: a concise and self-contained description on how to set the tool up and
use it

  5. 5.

Licenses: chosen license for the work (more in Section 4.2)

Visualizations aid communicating the developed NLP4RE tool. Platforms like
draw.io141414https://draw.io or yEd151515https://www.yworks.com/products/yed
make it easy to produce diagrams and visualizations summarizing high-level
overviews of the tool.

The setup section of the README.md file requires special attention for NLP4RE
tools. As most NLP4RE tools rely on third-party technology like the libraries
mentioned in Table 6 [60], the setup instruction should contain (1) which
resources were used, (2) which versions of these resources are required, and
(3) how to install them. Additional files like a requirements.txt file listing
required Python packages and their versions or a
Dockerfile161616https://www.docker.com/ containerizing the setup greatly
benefit the long-term usability of any NLP4RE tool.

###  4.2 Dissemination

Finally, the successful development of viable NLP4RE tools warrants an equally
successful dissemination to share the contribution with the NLP4RE research
community and relevant industry sectors. This includes hosting the tool in an
accessible manner, proper licensing, and thorough reporting.

####  4.2.1 Hosting

NLP4RE tools presented in publications should be made available online to
allow replication and evolution [30]. While disclosing pre-compiled
executables makes a tool (re-) usable, disclosing its source code allows
further investigation and evolution [51]. The choice of platform affects the
availability of a tool [30]. Hosting tools on private or institutional
platforms has shown to negatively affect their longevity, as these websites
are often abandoned when researchers change affiliation [30, 31]. Hence, we
recommend using platforms like GitHub171717https://github.com/ for version-
controlled hosting of NLP4RE tools. The collaborative features implemented in
GitHub allow streamlined and effective contribution to repositories of code
[9]. However, hosting an NLP4RE tool in a GitHub repository still has the
following three shortcomings:

  1. 1.

If the owner of the repository renames it, links to the repository (e.g., from
the manuscript presenting the tool) will not resolve anymore.

  2. 2.

GitHub is currently owned by Microsoft, a for-profit organization not
dedicated to the permanence of its content. The company can chose to disable
GitHub for the public at any time, making hosted tools unavailable.

  3. 3.

Repositories are not located through a digital object identifier (DOI).

Consequently, authors should aim to host their tools on platforms like
Zenodo181818https://zenodo.org/ to ensure permanent, open hosting referenced
with an unchanging DOI. Since Zenodo does not offer the same collaborative
features as GitHub, the following synergy ensures utilizing the best of both
platforms: A Zenodo account can be connected to a GitHub account.191919See a
detailed guidelines at https://docs.github.com/en/repositories/archiving-a-
github-repository/referencing-and-citing-content Then, every Release created
in the GitHub repository will automatically be archived on Zenodo. This way,
code is hosted in GitHub, making use of its collaborative and version control
features, while every milestone is additionally archived on Zenodo, ensuring
its permanent accessibility without any additional effort. For example, the
CiRA tool [22, 26] is hosted on GitHub
(https://github.com/JulianFrattini/cira) and archived on Zenodo
(https://zenodo.org/record/8033347). Links between the two (a badge in the
Readme linking from the GitHub repository to the Zenodo archive and the
automatic backlink from the Zenodo archive to the GitHub repository) allow a
seamless transition between the two.

####  4.2.2 Licensing

A tool requires a license as a legal mechanism to make explicit which rights
the licensor (the owner of the tool according to the license) grants the
licensee (a user of the tool) [32]. Without a license, an NLP4RE tool falls
under intellectual property laws and becomes unusable for anyone outside of
its owner, even if the code is accessible [51]. Other forms of legal
protection of the tool—like trademarks, trade secrets, or patents [32]—are
also possible, but significantly inhibit the evolution and use of the tool.

##### Choosing a license

One important type of license for academic software is the open source
license. A license is considered open source if it complies to the criteria
defined by the Open Source Initiative202020https://opensource.org/ listed at
https://opensource.org/osd/. In short, open source licenses circumvent the
restrictions of intellectual property law by explicitly granting a licensee to
access, use, and evolve the tool. A repository containing an open source
license is considered open source software [32]. The property of open source
is imperative for academic software to allow other scholars to learn from it,
replicate results, and continue its development [51].

Hence, we strongly recommend choosing an open source license when disclosing
an NLP4RE tool. Popular open source licenses include the MIT
license212121https://opensource.org/license/mit/ and the GNU General Public
License.222222https://opensource.org/license/gpl-3-0/ A list of open source
licenses can be found at https://opensource.org/licenses/. Once you have
chosen an appropriate license for your tool, add (1) the raw license text to
the top-level directory of the repository and (2) a copyright notice to the
bottom of the repository’s Readme file. The copyright notice typically takes
the following form: “Copyright ©{year} {owner}. This work is licensed under
{chosen license}.” See https://github.com/JulianFrattini/cira for an example.

##### Licensing other resources

Integrating external resources, i.e., code, models, data, etc. that are not
owned by the developer of a tool, into an NLP4RE tool requires specific care
when hosting the tool. When using external resources and including them in the
repository, the licensee need to make sure that the copyrights of each
integrated component allow this reuse and re-publishing [32]. If the licenses
of external resources allow this, additionally:

  1. 1.

include the raw license text in the same directory as the licensed resource;

  2. 2.

include a copyright notice stating who the owner of this resource is next to
the licensed resource as a text file;

  3. 3.

adjust the copyright notice at the bottom of your own Readme file to list all
external resources which are excluded from your tool’s license and instead
governed by their own license.

A simpler alternative is to not include external resources in a repository,
only self-owned material, and instead provide to a user detailed descriptions
on how to obtain and integrate these external resources in the Readme file.
This shifts the responsibility of using the external resources from the
repository owner to the repository user. However, this significantly reduces
the usability of the tool as it adds additional steps to its setup.

####  4.2.3 Reporting

Finally, reporting an NLP4RE tool in a research article should follow certain
guidelines. To describe the tool, we recommend reporting at least the
attributes presented in Section 2. Further attributes to report can be derived
from more detailed classification schemes, like the NLP4RE ID card [1]. To
reference the tool, we recommend providing a link to both the GitHub
repository containing the source code as well as to the Zenodo archive. In
case you need to prioritize, chose the latter, as the link to the Zenodo
archive is permanent and contains a reference back to the GitHub repository.

Reporting an NLP4RE tool also includes reporting its evaluation. Using
benchmark tests [5] and adhering to evaluation guidelines like the ECSER
pipeline for classifiers [13] ensures that the reported results are both
complete and comparable, which eases replication and synthesis with other
results [1].

During evaluation, probabilistic components of NLP4RE tools (like classifiers)
require to make a trade-off between precision (minimizing the amount of false
positives) and recall (minimizing the amount of false negatives). Deciding
which to optimize depends on the context, needs, and expectations of the
addressed problem, as well as the cost and likelihood of false positives and
false negatives. A tool with less than 100% recall will always require a
second round of manual reviews complementing the output of the tool [6]. Tools
with low precision burden the reader of its output with many irrelevant
results (false positives) which introduces skepticism and ultimately rejection
of the tool [21]. A conscious choice about which of the two to prefer should
be made and documented in the manuscript presenting the evaluation of an
NLP4RE tool [7].

##  5 Conclusion

This chapter introduces NLP4RE tools by defining a classification scheme
(Section 2), providing an overview over existing tools (Section 3), and
establishing tool management guidelines throughout its life cycle (Section 4).
We hope to incentivize not only the reuse of existing, but also the
contribution of new and properly developed, documented, hosted, and maintained
tools to the NLP4RE landscape. The great potential of NLP4RE tools for both
academia and industry warrants increasing attention [11, 34].

Future research avenues do not only include developing new and evolving
existing tools, but also establishing a benchmark infrastructure specifically
for the NLP4RE research domain. While NLP benchmarks exist [5] and contain
popular resources like SQuAD232323https://rajpurkar.github.io/SQuAD-explorer/
[49] and GLUE242424https://super.gluebenchmark.com/ [55], the NLP4RE domain
lacks any specific benchmark infrastructure. Investing effort into NLP4RE-
specific benchmark data sets as well as an infrastructure to track and compare
tool performances will improve the synthesis of results [1] and advance
progress in the NLP4RE tool development field further [11].

## References

  * [1] Abualhaija, S., Aydemir, F.B., Dalpiaz, F., Dell’Anna, D., Ferrari, A., Franch, X., Fucci, D.: Replication and verifiability in requirements engineering: the NLP for RE case (2023), https://arxiv.org/abs/2304.10265
  * [2] Alhoshan, W., Ferrari, A., Zhao, L.: Zero-shot learning for requirements classification: An exploratory study. Information and Software Technology 159, 107202 (2023). https://doi.org/10.1016/j.infsof.2023.107202
  * [3] Anish, P.R., Sonar, P., Lawhatre, P., Ghaisas, S.: Automated identification and deconstruction of penalty clauses in regulation. In: 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW). pp. 96–105. IEEE (2021). https://doi.org/10.1109/REW53955.2021.00021
  * [4] de Araújo, A.F., Marcacini, R.M.: RE-BERT: automatic extraction of software requirements from app reviews using BERT language model. In: Proceedings of the 36th Annual ACM Symposium on Applied Computing. pp. 1321–1327 (2021). https://doi.org/10.1145/3412841.3442006
  * [5] Becker, C., Hahn, N., He, B., Jabbar, H., Plesiak, M., Szabo, V., To, X., Yang, R., Wagner, J.: Modern approaches in natural language processing (2020), https://slds-lmu.github.io/seminar_nlp_ss20/
  * [6] Berry, D.M.: Evaluation of tools for hairy requirements and software engineering tasks. In: 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW). pp. 284–291. IEEE (2017). https://doi.org/10.1109/REW.2017.25
  * [7] Berry, D.M.: Empirical evaluation of tools for hairy requirements engineering tasks. Empirical Software Engineering 26(6), 111 (2021). https://doi.org/10.1007/s10664-021-09986-0
  * [8] Chen, Y., Wang, Y., Hou, Y., Wang, Y.: T-star: a text-based istar modeling tool. In: 2019 IEEE 27th international requirements engineering conference (RE). pp. 490–491. IEEE (2019). https://doi.org/10.1109/RE.2019.00069
  * [9] Dabbish, L., Stuart, C., Tsay, J., Herbsleb, J.: Social coding in GitHub: transparency and collaboration in an open software repository. In: Proceedings of the ACM 2012 conference on computer supported cooperative work. pp. 1277–1286 (2012). https://doi.org/10.1145/2145204.2145396
  * [10] Dabrowski, J., Letier, E., Perini, A., Susi, A.: Mining user feedback for software engineering: Use cases and reference architecture. In: 2022 IEEE 30th International Requirements Engineering Conference (RE). pp. 114–126. IEEE (2022). https://doi.org/10.1109/RE54965.2022.00017
  * [11] Dalpiaz, F., Ferrari, A., Franch, X., Palomares, C.: Natural language processing for requirements engineering: The best is yet to come. IEEE software 35(5), 115–119 (2018). https://doi.org/10.1109/MS.2018.3571242
  * [12] Dalpiaz, F., Van Der Schalk, I., Brinkkemper, S., Aydemir, F.B., Lucassen, G.: Detecting terminological ambiguity in user stories: Tool and experimentation. Information and Software Technology 110, 3–16 (2019). https://doi.org/10.1016/j.infsof.2018.12.007
  * [13] Dell’Anna, D., Aydemir, F.B., Dalpiaz, F.: Evaluating classifiers in SE research: the ECSER pipeline and two replication studies. Empirical Software Engineering 28(1), 3 (2023). https://doi.org/10.1007/s10664-022-10243-1
  * [14] Deshpande, G., Motger, Q., Palomares, C., Kamra, I., Biesialska, K., Franch, X., Ruhe, G., Ho, J.: Requirements dependency extraction by integrating active learning with ontology-based retrieval. In: 2020 IEEE 28th International Requirements Engineering Conference (RE). pp. 78–89. IEEE (2020). https://doi.org/10.1109/RE48521.2020.00020
  * [15] Eisenstein, J.: Introduction to natural language processing. MIT press (2019)
  * [16] Motger de la Encarnación, J., Borrull Baraut, R., Palomares Bonache, C., Marco Gómez, J.: OpenReq-DD: A requirements dependency detection tool. In: Joint Proceedings of REFSQ-2019 workshops, doctoral symposium, live studies track, and poster track: co-located with the 25th International Conference on Requirements Engineering: Foundation for Software Quality (REFSQ 2019): Essen, Germany, March 18th, 2019. pp. 1–5. CEUR-WS. org (2019), https://ceur-ws.org/Vol-2376/NLP4RE19_paper01.pdf
  * [17] Ezzini, S., Abualhaija, S., Arora, C., Sabetzadeh, M.: Automated handling of anaphoric ambiguity in requirements: a multi-solution study. In: Proceedings of the 44th International Conference on Software Engineering. pp. 187–199 (2022). https://doi.org/0.1145/3510003.3510157
  * [18] Ezzini, S., Abualhaija, S., Arora, C., Sabetzadeh, M., Briand, L.C.: Using domain-specific corpora for improved handling of ambiguity in requirements. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). pp. 1485–1497. IEEE (2021). https://doi.org/10.1109/ICSE43902.2021.00133
  * [19] Falkner, A., Palomares, C., Franch, X., Schenner, G., Aznar, P., Schoerghuber, A.: Identifying requirements in requests for proposal: A research preview. In: Requirements Engineering: Foundation for Software Quality: 25th International Working Conference, REFSQ 2019, Essen, Germany, March 18–21, 2019, Proceedings 25. pp. 176–182. Springer (2019). https://doi.org/10.1007/978-3-030-15538-4_13
  * [20] Fantechi, A., Gnesi, S., Passaro, L., Semini, L.: Inconsistency detection in natural language requirements using ChatGPT: a preliminary evaluation. In: 2023 IEEE 31st International Requirements Engineering Conference (RE). pp. 335–340. IEEE (2023). https://doi.org/10.1109/RE57278.2023.00045
  * [21] Femmer, H.: Requirements quality defect detection with the qualicen requirements Scout. In: REFSQ Workshops (2018), https://ceur-ws.org/Vol-2075/NLP4RE_paper2.pdf
  * [22] Fischbach, J., Frattini, J., Vogelsang, A.: CiRA: A tool for the automatic detection of causal relationships in requirements artifacts. In: Joint Workshops of the 27th International Conference on Requirements Engineering, REFSQ 2021-OpenRE, Posters and Tools Track, and Doctoral Symposium, Essen, Germany, 12 April 2021. vol. 2857. CEUR-WS (2021)
  * [23] Fischbach, J., Frattini, J., Vogelsang, A., Mendez, D., Unterkalmsteiner, M., Wehrle, A., Henao, P.R., Yousefi, P., Juricic, T., Radduenz, J., et al.: Automatic creation of acceptance tests by extracting conditionals from requirements: NLP approach and case study. Journal of Systems and Software 197, 111549 (2023). https://doi.org/10.1016/j.jss.2022.111549
  * [24] Fischbach, J., Junker, M., Vogelsang, A., Freudenstein, D.: Automated generation of test models from semi-structured requirements. In: 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW). pp. 263–269. IEEE (2019). https://doi.org/10.1109/REW.2019.00053
  * [25] Franch, X., Palomares, C., Quer, C., Chatzipetrou, P., Gorschek, T.: The state-of-practice in requirements specification: an extended interview study at 12 companies. Requirements Engineering pp. 1–33 (2023). https://doi.org/10.1007/s00766-023-00399-7
  * [26] Frattini, J., Fischbach, J., Bauer, A.: CiRA: An open-source Python package for automated generation of test case descriptions from natural language requirements. In: 2023 IEEE Tenth International Workshop on Artificial Intelligence for Requirements Engineering (AIRE) (2023), https://ceur-ws.org/Vol-2857/nlp4re4.pdf
  * [27] Frattini, J., Junker, M., Unterkalmsteiner, M., Mendez, D.: Automatic extraction of cause-effect-relations from requirements artifacts. In: Proceedings of the 35th IEEE/ACM International conference on automated software engineering. pp. 561–572 (2020). https://doi.org/10.1145/3324884.3416549
  * [28] Frattini, J., Montgomery, L., Fischbach, J., Unterkalmsteiner, M., Mendez, D., Fucci, D.: A live extensible ontology of quality factors for textual requirements. In: 30th International Requirements Engineering Conference (RE). pp. 274–280. IEEE (2022). https://doi.org/10.1109/RE54965.2022.00041
  * [29] Frattini, J., Montgomery, L., Fucci, D., Fischbach, J., Unterkalmsteiner, M., Mendez, D.: Let’s stop building at the feet of giants: Recovering unavailable requirements quality artifacts. In: Joint Proceedings of REFSQ-2023 Workshops, Doctoral Symposium, Posters and Tools Track and Journal Early Feedback, Barcelona, REFSQ-JP 2023, 17 April 2023 through 20 April 2023. vol. 3378. CEUR-WS (2023), https://arxiv.org/abs/2304.04670
  * [30] Frattini, J., Montgomery, L., Fucci, D., Unterkalmsteiner, M., Mendez, D., Fischbach, J.: Requirements quality research artifacts: Recovery, analysis, and management guideline (2023), submitted to the Journal of Systems and Software
  * [31] Gabelica, M., Bojčić, R., Puljak, L.: Many researchers were not compliant with their published data sharing statement: a mixed-methods study. Journal of Clinical Epidemiology 150, 33–41 (2022). https://doi.org/10.1016/j.jclinepi.2022.05.019
  * [32] German, D.M., Hassan, A.E.: License integration patterns: Addressing license mismatches in component-based development. In: 2009 IEEE 31st international conference on software engineering. pp. 188–198. IEEE (2009). https://doi.org/10.1109/ICSE.2009.5070520
  * [33] Gnanasekaran, R.K., Chakraborty, S., Dehlinger, J., Deng, L.: Using recurrent neural networks for classification of natural language-based non-functional requirements. In: REFSQ Workshops (2021), https://ceur-ws.org/Vol-2857/nlp4re7.pdf
  * [34] Gorschek, T., Garre, P., Larsson, S., Wohlin, C.: A model for technology transfer in practice. IEEE software 23(6), 88–95 (2006). https://doi.org/10.1109/MS.2006.147
  * [35] Gröpler, R., Kutty, L., Sudhi, V., Smalley, D.: Automated requirement formalization using product design specifications. In: REFSQ Workshops (2022), https://ceur-ws.org/Vol-3122/NLP4RE-paper-1.pdf
  * [36] Henao, P.R., Fischbach, J., Spies, D., Frattini, J., Vogelsang, A.: Transfer learning for mining feature requests and bug reports from tweets and app store reviews. In: 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW). pp. 80–86. IEEE (2021). https://doi.org/10.1109/REW53955.2021.00019
  * [37] Hey, T., Keim, J., Koziolek, A., Tichy, W.F.: NoRBERT: Transfer learning for requirements classification. In: 2020 IEEE 28th International Requirements Engineering Conference (RE). pp. 169–179. IEEE (2020). https://doi.org/10.1109/RE48521.2020.00028
  * [38] Jadallah, N., Fischbach, J., Frattini, J., Vogelsang, A.: CATE: Causality tree extractor from natural language requirements. In: 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW). pp. 77–79. IEEE (2021). https://doi.org/10.1109/REW53955.2021.00018
  * [39] Kifetew, F.M., Perini, A., Susi, A., Siena, A., Muñante, D., Morales-Ramirez, I.: Automating user-feedback driven requirements prioritization. Information and Software Technology 138, 106635 (2021). https://doi.org/10.1016/j.infsof.2021.106635
  * [40] Kleebaum, A., Paech, B., Johanssen, J.O., Bruegge, B.: Continuous rationale identification in issue tracking and version control systems. In: Joint Proceedings of REFSQ-2021 workshops, doctoral symposium, live studies track, and poster track: co-located with the 27th International Conference on Requirements Engineering: Foundation for Software Quality (REFSQ 2021): Essen, Germany, April 12th, 2021. CEUR-WS. org (2021), https://ceur-ws.org/Vol-2857/nlp4re1.pdf
  * [41] Lian, X., Liu, W., Zhang, L.: Assisting engineers extracting requirements on components from domain documents. Information and Software Technology 118, 106196 (2020). https://doi.org/10.1016/j.infsof.2019.106196
  * [42] Madala, K., Piparia, S., Blanco, E., Do, H., Bryce, R.: Model elements identification using neural networks: a comprehensive study. Requirements Engineering 26, 67–96 (2021). https://doi.org/10.1007/s00766-020-00332-2
  * [43] Mankins, J.C.: Technology readiness levels. Tech. rep., Advanced Concepts Office, Office of Space Access and Technology, NASA (1995)
  * [44] Mekala, R.R., Irfan, A., Groen, E.C., Porter, A., Lindvall, M.: Classifying user requirements from online feedback in small dataset environments using deep learning. In: 2021 IEEE 29th International requirements engineering conference (RE). pp. 139–149. IEEE (2021). https://doi.org/10.1109/RE51729.2021.00020
  * [45] Minocher, R., Atmaca, S., Bavero, C., McElreath, R., Beheim, B.: Reproducibility improves exponentially over 63 years of social learning research (2020), https://doi.org/10.31234/osf.io/4nzc7
  * [46] Montgomery, L., Frattini, J., Mendez, D., Fucci, D., Unterkalmsteiner, M., Fischbach, J.: Open science – artefact management guideline (2023). https://doi.org/10.5281/zenodo.8134403
  * [47] Peng, Z., Niu, N.: Co-AI: a colab-based tool for abstraction identification. In: 2021 IEEE 29th International Requirements Engineering Conference (RE). pp. 420–421. IEEE (2021). https://doi.org/10.1109/RE51729.2021.00050
  * [48] Pudlitz, F., Brokhausen, F., Vogelsang, A.: Extraction of system states from natural language requirements. In: 2019 IEEE 27th International Requirements Engineering Conference (RE). pp. 211–222. IEEE (2019). https://doi.org/10.1109/RE.2019.00031
  * [49] Rajpurkar, P., Jia, R., Liang, P.: Know what you don’t know: Unanswerable questions for SQuAD. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 784–789 (2018). https://doi.org/10.18653/v1/P18-2124
  * [50] Ronanki, K., Berger, C., Horkoff, J.: Investigating ChatGPT’s potential to assist in requirements elicitation processes. In: 2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). pp. 354–361. IEEE (2023). https://doi.org/10.1109/SEAA60479.2023.00061
  * [51] Rosen, L.: Open source licensing - Software Freedom and Intellectual Property Law. Prentice Hall (2004), https://www.rosenlaw.com/oslbook.htm
  * [52] Sainani, A., Anish, P.R., Joshi, V., Ghaisas, S.: Extracting and classifying requirements from software engineering contracts. In: 2020 IEEE 28th international requirements engineering conference (RE). pp. 147–157. IEEE (2020). https://doi.org/10.1109/RE48521.2020.00026
  * [53] Stanik, C., Maalej, W.: Requirements intelligence with OpenReq Analytics. In: 2019 IEEE 27th International Requirements Engineering Conference (RE). pp. 482–483. IEEE (2019). https://doi.org/10.1109/RE.2019.00066
  * [54] Unterkalmsteiner, M.: TT-RecS: The taxonomic trace recommender system. In: 2020 IEEE Seventh International Workshop on Artificial Intelligence for Requirements Engineering (AIRE). pp. 18–21. IEEE (2020). https://doi.org/10.1109/AIRE51212.2020.00009
  * [55] Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.: Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems 32 (2019), https://arxiv.org/abs/1905.00537
  * [56] Wang, L., Nakagawa, H., Tsuchiya, T.: Opinion analysis and organization of mobile application user reviews. In: REFSQ Workshops (2020), https://ceur-ws.org/Vol-2584/NLP4RE-paper4.pdf
  * [57] Winter, S., Timperley, C.S., Hermann, B., Cito, J., Bell, J., Hilton, M., Beyer, D.: A retrospective study of one decade of artifact evaluations. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 145–156 (2022). https://doi.org/10.1145/3540250.3549172
  * [58] Xiong, H., Wang, Y., Li, T.: BiStar: A template-based iStar modeling tool combining graphical and textual modeling. In: 2022 IEEE 30th International Requirements Engineering Conference (RE). pp. 260–261. IEEE (2022). https://doi.org/10.1109/RE54965.2022.00034
  * [59] Zaki-Ismail, A., Osama, M., Abdelrazek, M., Grundy, J., Ibrahim, A.: CORG: A component-oriented synthetic textual requirements generator. In: International Working Conference on Requirements Engineering: Foundation for Software Quality. pp. 54–70. Springer (2021). https://doi.org/10.1007/978-3-030-73128-1_4
  * [60] Zhao, L., Alhoshan, W., Ferrari, A., Letsholo, K.J., Ajagbe, M.A., Chioasca, E.V., Batista-Navarro, R.T.: Natural language processing for requirements engineering: A systematic mapping study. ACM Computing Surveys (CSUR) 54(3), 1–41 (2021). https://doi.org/10.1145/3444689