# Learning Speaker Representations with Mutual Information

###### Abstract

Learning good representations is of crucial importance in deep learning.
Mutual Information (MI) or similar measures of statistical dependence are
promising tools for learning these representations in an unsupervised way.
Even though the mutual information between two random variables is hard to
measure directly in high dimensional spaces, some recent studies have shown
that an implicit optimization of MI can be achieved with an encoder-
discriminator architecture similar to that of Generative Adversarial Networks
(GANs).

In this work, we learn representations that capture speaker identities by
maximizing the mutual information between the encoded representations of
chunks of speech randomly sampled from the same sentence. The proposed encoder
relies on the SincNet architecture and transforms raw speech waveform into a
compact feature vector. The discriminator is fed by either positive samples
(of the joint distribution of encoded chunks) or negative samples (from the
product of the marginals) and is trained to separate them.

We report experiments showing that this approach effectively learns useful
speaker representations, leading to promising results on speaker
identification and verification tasks. Our experiments consider both
unsupervised and semi-supervised settings and compare the performance achieved
with different objective functions.

Index Terms: Deep Learning, Speaker Recognition, Mutual Information,
Unsupervised Learning, SincNet.

##  1 Introduction

Deep learning has shown remarkable success in numerous speech tasks, including
speech recognition [1, 2, 3, 4] and speaker recognition [5, 6]. The deep
learning paradigm aims to describe data by means of a hierarchy of
representations, that are progressively combined to model higher level
abstractions [7]. Most commonly, deep neural networks are trained in a
supervised way, while learning meaningful representations in an unsupervised
fashion is more challenging but could be useful especially in semi-supervised
settings.

Several approaches have been proposed for deep unsupervised learning in the
last decade. Notable examples are deep autoencoders [8], Restricted Boltzmann
Machines (RBMs) [9], variational autoencoders [10] and, more recently,
Generative Adversarial Networks (GANs) [11]. GANs are often used in the
context of generative modeling, where they attempt to minimize a measure of
discrepancy between a distribution generated by a neural network and the data
distribution. Beyond generative modeling, some works have extended this
framework to learn features that are invariant to different domains [12] or to
noise conditions [13]. Moreover, we recently witnessed some remarkable
attempts to learn unsupervised representations by minimizing or maximizing
Mutual Information (MI) [14, 15, 16, 17]. This measure is a fundamental
quantity for estimating the statistical dependence between random variables
and is defined as the Kullback-Leibler (KL) divergence between the joint
distribution over these random variables and the product of their marginal
distributions [18]. As opposed to other metrics, such as correlation, MI can
capture complex non-linear relationships between random variables [19]. MI,
however, is difficult to compute directly, especially in high dimensional
spaces [20]. The aforementioned works found that it is possible to maximize or
minimize the MI within a framework that closely resembles that of GANs.
Additionally, [15] has further proved that it is even possible to explicitly
compute it by exploiting its Donsker-Varadhan bound.

Here we attempt to learn good speaker representations by maximizing the mutual
information between two encoded random chunks of speech sampled from the same
sentence. Our architecture employs both an encoder, that transforms raw speech
samples into a compact feature vector, and a discriminator. The latter is
alternatively fed by samples from the joint distribution (i.e. two local
encoded vectors randomly drawn from the same speech sentence) and from the
product of marginal distributions (i.e, two local encoder vectors coming
different utterances). The discriminator is jointly trained with the encoder
to maximize the separability of the two distributions. We called our approach
Local Info Max (LIM) to highlight the fact that it is simply based on randomly
sampled local speech chunks. Our encoder is based on SincNet [21, 22], which
efficiently processes the raw input waveforms with learnable band-pass filters
based on sinc functions.

The experimental results show that our approach learns useful speaker
features, leading to promising results on speaker identification and
verification tasks. Our experiments are conducted in both unsupervised and
semi-supervised settings and compare different objective functions for the
discriminator. We release the code of LIM within the PyTorch-Kaldi toolkit
[23].

##  2 Speaker Representations based on MI

The mutual information between two random variables $z_{1}$ and $z_{2}$ is
defined as follows:

$\begin{split}MI(z_{1},z_{2})&=\int_{z_{1}}\int_{z_{2}}p(z_{1},z_{2})log\bigg{(}\frac{p(z_{1},z_{2})}{p(z_{1})p(z_{2})}\bigg{)}dz_{1}dz_{2}\\\
&=D_{KL}\big{(}p(z_{1},z_{2})||p(z_{1})p(z_{2})\big{)},\end{split}$ (1)

where $D_{KL}$ is the Kullback-Leibler (KL) divergence between the joint
distribution $p(z_{1},z_{2})$ and the product of marginals $p(z_{1})p(z_{2})$.
The MI is minimized when the random variables $z_{1}$ and $z_{2}$ are
statistically independent (i.e., the joint distribution is equal to the
product of marginals) and is maximized when the two variables contain the same
information (in which case the MI is simply the entropy of any one of the
variables).

Figure 1: Architecture of the proposed system for unsupervised learning of
speaker representations. The speech chunks $c_{1}$ and $c_{2}$ are sampled
from the same sentence, while $c_{rand}$ is sampled from a different
utterance.

Our LIM system, depicted in Fig.1, aims to derive a compact representation
$z$. The encoder $f_{\Theta}$, with
$f:\mathbb{R}^{N}\rightarrow\mathbb{R}^{M}$, is fed by N speech samples and
outputs a vector composed of M real values, while the discriminator
$g_{\Phi}$, with $g:\mathbb{R}^{2M}\rightarrow\mathbb{R}$, is fed by two
speaker representations and outputs a real scalar. We learn the parameters
$\Theta$ and $\Phi$ of the encoder and the discriminator such that we maximize
the mutual information $MI(z_{1},z_{2})$:

$(\hat{\Theta},\hat{\Phi})=\underset{\Theta,\Phi}{\operatorname{arg}\,\operatorname{max}}\;{MI(z_{1},z_{2})},$
(2)

where the two representations $z_{1}$ and $z_{2}$ are obtained by encoding the
speech chunks $c_{1}$ and $c_{2}$ that are randomly sampled from the same
sentence. Note that one reliable factor that is shared across chunks within
each utterance is the speaker identity. The maximization of $MI(z_{1},z_{2})$
should thus be able to properly disentangle this constant factor from the
other variables (e.g., phonemes) that characterize the speech signal but are
not shared across chunks of the same utterance.

As shown in Alg. 1, the maximization of MI relies on a sampling strategy that
draws positive and negative samples from the joint and the product of marginal
distributions, respectively. As discussed so far, the positive samples
$(z_{1},z_{2})$ are simply derived by randomly sampling speech chunks from the
same sentence. Negative samples $(z_{1},z_{rnd})$, instead, are obtained by
randomly sampling from another utterance. The underlying assumptions
considered here are the following: (1) two random utterances likely belong to
different speakers, (2) each sentence contains a single speaker only. Under
these assumptions, that naturally hold in most of the available speech
datasets, our method can be regarded as unsupervised (or, more precisely,
self-supervised) because no speaker labels are explicitly used.

A set of $N_{samp}$ positive and negative examples is sampled to form a
minibatch $X=\\{X_{p},X_{n}\\}$. The minibatch $X$ feeds the discriminator
$g_{\Phi}$, that is jointly trained with the encoder. Given $z_{1}$, the
discriminator $g_{\Phi}$ has to decide whether its other input ($z_{2}$ or
$z_{rnd}$) comes from the same sentence or from a different one (and generally
a different speaker). Differently to the GAN framework, the encoder and the
discriminator are not adversarial here but must cooperate to maximize the
discrepancy between the joint and the product of marginal distributions. In
other words, we play a max-max game rather than a min-max one, making it
easier to monitor the progress of training (compared to GAN training), simply
as the average loss of the discriminator.

Algorithm 1 Learning speaker representation with MI

1:while Not Converged do

2: for i=1 to $N_{samp}$ do

3: Sample a chunk $c_{1}$ from a random utterance.

4: Sample another chunk $c_{2}$ from the same utterance.

5: Sample a chunk $c_{rnd}$ from another utterance.

6: Process the chunks with the encoder:

7: $z_{1}=f_{\Theta}(c_{1})$, $z_{2}=f_{\Theta}(c_{2})$,
$z_{rnd}=f_{\Theta}(c_{rnd})$.

8: Create positive and negative samples:

9: $X_{p}[i]$=($z_{1}$,$z_{2}$), $X_{n}[i]$=($z_{1}$,$z_{rnd}$).

10: Compute discriminator outputs: $g(X_{p})$, $g(X_{n})$.

11: Compute Loss $L(\Theta,\Phi)$.

12: Compute Gradients $\frac{\partial L}{\partial\Theta}$, $\frac{\partial
L}{\partial\Phi}$.

13: Update $\Theta$ and $\Phi$ to maximize L.

Different objectives functions can be used for the discriminator. The simplest
solution, adopted in [14],[17] and [24], consists in using the standard binary
cross-entropy (BCE) loss111The output layer must be based on a sigmoid when
using BCE.:

$L(\Theta,\Phi)=\mathbb{E}_{X_{p}}[\log(g(z_{1},z_{2}))]+\mathbb{E}_{X_{n}}[\log(1-g(z_{1},z_{rnd}))],$
(3)

where $\mathbb{E}_{X_{p}}$ and $\mathbb{E}_{X_{n}}$ denote the expectation
over positive and negative samples, respectively. Such a metric estimates the
Jensen-Shannon divergence between two distributions rather than the KL
divergence. Consequently, this loss does not optimize the exact KL-based
definition of MI, but a similar divergence between two distributions.
Differently from standard MI, this metric is bounded (i.e., its maximum is
zero), making the convergence of the architecture more numerically stable.

As an alternative, it is possible to directly optimize the MI with the MINE
objective [15]:

$L(\Theta,\Phi)=\mathbb{E}_{X_{p}}[g(z_{1},z_{2})]-log\Big{(}\mathbb{E}_{X_{n}}[e^{g(z_{1},z_{rnd})}]\Big{)}.$
(4)

MINE explicitly computes MI by exploiting a lower-bound based on the Donsker-
Varadhan representation of the KL divergence. The third alternative explored
in this work is the Noise Contrastive Estimation (NCE) loss proposed in [16],
that is defined as follows:

$L(\Theta,\Phi)=\mathbb{E}_{x}\bigg{[}g(z_{1},z_{2})-\log\bigg{(}g(z_{1},z_{2})+\sum_{x_{n}}e^{g(z_{1},z_{rnd})}\bigg{)}\bigg{]},$
(5)

where the minibatch $X$ is composed of a single positive sample and $N-1$
negative samples. In [16] it is proved that maximizing this loss maximizes a
lower bound on MI.

All the aforementioned objectives are based on the idea of maximizing a
discrepancy between the joint and product of marginal distributions.
Nevertheless, such losses might be more or less easy to optimize within the
proposed framework.

The unsupervised representations $z$ are then used to train a speaker-id
classifier in a standard supervised way. Beyond unsupervised learning, this
paper explores two semi-supervised variations for learning speaker
representations. The first one is based on pre-training the encoder with the
unsupervised parameters and fine-tuning it together with the speaker-id
classifier. As an alternative, we jointly train encoder, discriminator, and
speaker-id networks from scratch. This way, the gradient computed within the
encoder not only depends on the supervised loss but also on the unsupervised
objective. The latter approach turned out to be very effective, since the
unsupervised gradient acts as a powerful regularizer.

Similarly to [25, 26, 27, 28, 29], we propose to directly process raw
waveforms rather than using standard MFCC, or FBANK features. The latter hand-
crafted features are originally designed from perceptual evidence and there
are no guarantees that such inputs are optimal for all speech-related tasks.
Standard features, in fact, smooth the speech spectrum, possibly hindering the
extraction of crucial narrow-band speaker characteristics such as pitch and
formants, that are important clues on the speaker identity. To better process
raw audio, the encoder is based on SincNet [21, 22], a novel Convolutional
Neural Network (CNN) that encourages the first layer to discover more
meaningful filters. In contrast to standard CNNs, which learn all the elements
of each filter, only low and high cutoff frequencies of band-pass sinc-based
filters are directly learned from data, making SincNet suitable to process the
high-dimensional audio.

##  3 Related Work

Similarly to this work, other attempts have recently been made to learn
unsupervised representations with mutual information. In [14], a GAN that
minimizes MI using positive and negative samples has been proposed for
Independent Component Analysis (ICA). A similar approach can be used to
maximize MI. In [16] authors proposed a method called Contrastive Predicting
Coding (CPC), that learns representations by predicting the future in a latent
space. It uses an autoregressive model optimized with a probabilistic
contrastive loss. In [17] authors introduced DeepInfoMax (DIM), an
architecture that learns representations based on both local and high-level
global information.

The proposed LIM differs from the aforementioned works in the following way:
DIM performs a maximization of MI between local and global representations,
CPC relies on future predictions, while our method is simply based on random
local sampling. Note that training using local embeddings only is very
efficient since it does not require the expensive computation of a global
representation as in GIM. LIM is also related with the recently-proposed
methods based on triplet loss [30, 31]. Most of the previous works on triplet
loss (with the exception of [32]) rely on the speaker labels [31]. Moreover,
they simply maximize the Euclidean or cosine distance between speaker
embeddings. LIM, instead, is based on maximizing the mutual information, thus
considering a more meaningful divergence that can also capture complex non-
linear relationships between the variables. Maximum Mutual Information (MMI)
is often used in HMM-DNN speech recognition as a loss function [33]. This loss
maximizes the MI between the acoustic probabilities and the targeted word
sequence in a standard supervised framework, while LIM is used in a totally
different unsupervised context that relies on local speech embeddings. Our
work also uses SincNet [21, 22] (that is here used for the first time in an
unsupervised framework), and extends the previous works by also addressing
semi-supervised learning where encoder, discriminator, and speaker-id
classifier are jointly trained from scratch. Moreover, to the best of our
knowledge, this paper is the first that compares several objective functions
for MI optimization in a speech task.

##  4 Experimental Setup

The proposed method has been evaluated using different corpora. In the
following, an overview of the experimental setting is provided.

###  4.1 Corpora

This paper considered the TIMIT (462 spks, train chunk) [34], Librispeech
(2484 spks), and VoxCeleb1 (1251 spks) [35] corpora. To make TIMIT and
Librispeech speaker recognition tasks more challenging, we only employed 12-15
seconds of randomly selected training material for each speaker. Moreover, a
set of TIMIT and Librispeech experiments have also been performed in distant-
talking reverberant conditions. In this case, all the clean signals were
convoluted with a different impulse response, that was sampled from the DIRHA
dataset [36, 37]. The DIRHA corpus contains high-quality multi-room and multi-
microphone impulse responses, that were measured in a domestic environment
with a considerable reverberation time of $T_{60}=0.7s$. This way, we are able
to provide experimental evidence in a much more challenging acoustic scenario
and we can introduce a channel effect that is not natively present in the
clean TIMIT and Librispeech corpora. To study our approach using a more
standard speaker recognition dataset, we also employed the VoxCeleb1 corpus
(using the provided lists).

###  4.2 DNN Setup

The waveform of each speech sentence was split into chunks of 200 ms (with 10
ms overlap), which were fed into the SincNet encoder. The first layer of the
encoder performs sinc-based convolutions, using 80 filters of length $L=251$
samples. The architecture then employs two standard convolutional layers, both
using 60 filters of length 5. Layer normalization [38] was used for both the
input samples and for all convolutional layers. Next, two fully-connected
leaky-ReLU layers [39] composed of 2048 and 1024 neurons (normalized with
batch normalization [40, 41]) were applied. Both the discriminator and the
speaker-id classifier are fed by the encoder output and consist of MLPs based
on a single ReLU layer. Frame-level speaker classification was obtained from
the speaker-id network by applying a softmax output layer, that provides a set
of posterior probabilities over the targeted speakers. A sentence-level
classification was derived by averaging the frame predictions and voting for
the speaker which maximizes the average posterior. Training used the RMSprop
optimizer, with a learning rate $lr=0.001$, $\alpha=0.95$, $\epsilon=10^{-}7$,
and minibatches of size 128. All the hyper-parameters of the architecture were
tuned on TIMIT, then inherited for Librispeech and VoxCeleb as well.

The speaker verification system was derived from the speaker-id neural network
using the d-vector technique. The d-vector [42, 35] was extracted from the
last hidden layer of the speaker-id network. A speaker-dependent d-vector was
computed and stored for each enrollment speaker by performing an L2
normalization and averaging all the d-vectors of the different speech chunks.
The cosine distance between enrolment and test d-vectors was then calculated,
and a threshold was then applied on it to reject or accept the speaker. Note
that to assess our approach on a standard open-set speaker verification task,
all the enrolment and test utterances were taken from a speaker pool different
from that used for training the speaker-id DNN.

##  5 Results

This section summarizes our experimental activity on speaker identification
and verification.

###  5.1 Speaker Identification

Tab. 1 reports the sentence-level classification error rates achieved with
binary cross-entropy (BCE), MINE, Noise Constructive Estimation (NCE), and the
triplet loss used in [31].

| TIMIT | Librispeech
---|---|---
| CNN | SincNet | CNN | SincNet
Unsupervised-Trip. Loss | 2.84 | 2.22 | 1.46 | 1.33
Unsupervised-MINE | 2.15 | 1.36 | 1.43 | 0.94
Unsupervised-NCE | 2.05 | 1.29 | 1.14 | 0.82
Unsupervised-BCE | 1.98 | 1.21 | 1.12 | 0.75
Table 1: Classification Error Rate (CER%) obtained on TIMIT (462 spks) and
Librispeech (2484 spks) speaker-id tasks using LIM embeddings learned with
various objective functions.

The table highlights that our LIM embeddings contain information on the
speaker identity, leading to a CER(%) ranging from 2.84% to 1.21% in all the
considered settings. It is worth noting that mutual information losses (i.e.,
MINE, NCE, BCE) outperform the triplet loss. This result suggests that better
embeddings can be derived with a divergence measure more meaningful than the
simple cosine distance. The best performance is achieved with the standard
binary cross-entropy. Similar to [17], we have observed that this bounded
metric is more stable and more easy to optimize. Both MINE and NCE objective
are unbounded and their value can grow indefinitely during training,
eventually causing numerical issues. The performance achieved with Librispeech
is better than that observed for TIMIT. Even though the former is based on
more speakers, its utterances are on average longer than the TIMIT ones. The
table also shows that SincNet outperforms a standard CNN. This confirms the
promising achievements obtained in [21, 22] in a standard supervised setting.
SincNet, in fact, converges faster and to a better solution, thanks to the
compact sinc filters that make learning from high-dimensional raw samples
easier.

Tab. 2 extends previous speaker-id results to other training modalities,
including supervised and semi-supervised learning in both clean and
reverberant acoustic conditions.

| TIMIT | Librispeech
---|---|---
| Clean | Rev | Clean | Rev
Supervised | 0.85 | 34.8 | 0.80 | 17.1
Unsupervised-BCE | 1.21 | 28.2 | 0.75 | 15.2
Semi-supervised-pretr. | 0.69 | 25.4 | 0.56 | 9.6
Semi-supervised-joint | 0.65 | 24.6 | 0.52 | 9.3
Table 2: Classification Error Rate (CER%) obtained on speaker-id with
supervised, unsupervised and semi-supervised modalities in clean and
reverberat conditions.

From the table, it emerges that the results achieved when feeding the
classifier with our speaker embeddings (unsupervised-BCE) are often better
than those obtained with the standard supervised training (supervised). The
gap becomes more evident when we pass from unsupervised to semi-supervised
learning. In particular, the joint semi-supervised framework (i.e., the
approach that jointly trains encoder, discriminator, and speaker
classification for scratch) yields the best performance, surpassing the
performance obtained when pretraining the encoder and then fine-tuning it with
the supervised task (Semi-supervised-pretr.). The internal representations
discovered in this way are influenced by both the supervised and the
unsupervised loss. The latter one acts as a powerful regularizer, that allows
the neural network to find robust features. The results also show a
significant performance degradation in distant-talking acoustic conditions.
The presence of considerable reverberation and the introduction of
channel/microphone variabilities, in fact, make speaker-id particularly
challenging.

###  5.2 Speaker Verification

We finally extend our validation to speaker verification on the VoxCeleb
corpus. Table 3 compares the Equal Error Rate (EER%) achieved using our best
system (Semi-supervised-pretr.) with some previous works on the same dataset.

| EER (%)
---|---
GMM-UBM [35] | 15.0
I-vectors + PLDA [35] | 8.8
CNN [35] | 7.8
CNN + intra-class + triplet loss[43] | 7.9
SincNet [21] | 7.2
SincNet+LIM (proposed) | 5.8
Table 3: Equal Error Rate (EER%) obtained on speaker verification (using the
VoxCeleb corpus).

The proposed model reaches an EER(%) of 5.8% and outperforms other systems
such as an I-vector baseline [35, 44], a standard CNN [35], and a CNN based on
combination of intra-class and triples loss [43]. Finally, LIM outperforms a
standard SincNet model trained in a fully supervised way [21]. This result
confirms the effectiveness of the proposed approach even in an open-set text-
independent speaker verification setting.

##  6 Conclusion

This paper proposed a method for learning speaker embeddings by maximizing
mutual information. The experiments have shown promising performance on
speaker recognition and have highlighted better results when adopting the
standard binary cross-entropy loss, that turned out to be more stable and
easier to optimize than other metrics. It also highlighted the importance of
using SincNet, confirming its effectiveness when processing raw audio
waveforms. The best results are obtained with end-to-end semi-supervised
learning, where an ecosystem of neural networks composed of an encoder, a
discriminator, and a speaker-id must cooperate to derive good speaker
embeddings. Our achievement can be easily combined with other recent findings
in speaker recognition. For instance, it is possible to use LIM to extract
semi-supervised x-vectors. We can also improve it by employing an attention
mechanism that weights the contribution of each time frame, or by combing our
semi-supervised costs with other losses, such as the center loss.

##  7 Acknowledgment

We would like to thank D. Hjelm, T. Parcollet, and M. Omologo, for helpful
comments. This research was enabled by support provided by Calcul Québec and
Compute Canada.

## References

  * [1] D. Yu and L. Deng, _Automatic Speech Recognition - A Deep Learning Approach_. Springer, 2015.
  * [2] G. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent pre-trained deep neural networks for large vocabulary speech recognition,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 20, no. 1, pp. 30–42, 2012.
  * [3] M. Ravanelli, _Deep learning for Distant Speech Recognition_. PhD Thesis, Unitn, 2017.
  * [4] M. Ravanelli, P. Brakel, M. Omologo, and Y. Bengio, “A network of deep neural networks for distant speech recognition,” in _Proc. of ICASSP_ , 2017, pp. 4880–4884.
  * [5] M. McLaren, Y. Lei, and L. Ferrer, “Advances in deep neural network approaches to speaker recognition,” in _Proc. of ICASSP_ , 2015, pp. 4814–4818.
  * [6] F. Richardson, D. Reynolds, and N. Dehak, “Deep neural network approaches to speaker and language recognition,” _IEEE Signal Processing Letters_ , vol. 22, no. 10, pp. 1671–1675, 2015.
  * [7] I. Goodfellow, Y. Bengio, and A. Courville, _Deep Learning_. MIT Press, 2016.
  * [8] Y. Bengio, P. L., D. Popovici, and H. Larochelle, “Greedy layer-wise training of deep networks,” in _Proc. of NIPS_ , 2007, pp. 153–160.
  * [9] G. Hinton, S. Osindero, and Y. Teh, “A fast learning algorithm for deep belief nets,” vol. 18, 2006, pp. 1527–1554.
  * [10] D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” _CoRR_ , vol. abs/1312.6114, 2013.
  * [11] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Proc. of NIPS_ , 2014, pp. 2672–2680.
  * [12] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” _J. Mach. Learn. Res._ , vol. 17, no. 1, pp. 2096–2030, Jan. 2016\.
  * [13] D. Serdyuk, P. Brakel, B. Ramabhadran, S. Thomas, Y. Bengio, and K. Audhkhasi, “Invariant representations for noisy speech recognition,” _arXiv e-prints_ , vol. abs/1612.01928, 2016.
  * [14] P. Brakel and Y. Bengio, “Learning independent features with adversarial nets for non-linear ica,” _arXiv e-prints_ , vol. 1710.05050, 2017.
  * [15] M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and R. D. Hjelm, “Mutual information neural estimation,” in _Proc. of ICML_ , 2018, pp. 531–540.
  * [16] A. van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _CoRR_ , vol. abs/1807.03748, 2018.
  * [17] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, A. Trischler, and Y. Bengio, “Learning deep representations by mutual information estimation and maximization,” _arXiv e-prints_ , vol. 1808.06670, 2018.
  * [18] D. Applebaum, _Probability and Information: An Integrated Approach_ , 2nd ed. Cambridge University Press, 2008\.
  * [19] J. B. Kinney and G. S. Atwal, “Equitability, mutual information, and the maximal information coefficient,” _Proceedings of the National Academy of Sciences_ , vol. 111, no. 9, pp. 3354–3359, 2014.
  * [20] L. Paninski, “Estimation of entropy and mutual information,” _Neural Comput._ , vol. 15, no. 6, pp. 1191–1253, Jun. 2003.
  * [21] M. Ravanelli and Y. Bengio, “Speaker Recognition from raw waveform with SincNet,” in _Proc. of SLT_ , 2018.
  * [22] M. Ravanelli and Y.Bengio, “Interpretable Convolutional Filters with SincNet,” in _Proc. of NIPS@IRASL_ , 2018.
  * [23] M. Ravanelli, T. Parcollet, and Y. Bengio, “The PyTorch-Kaldi Speech Recognition Toolkit,” in _Submitted to ICASSP_ , 2019.
  * [24] P. Velickovic, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, and R. D. Hjelm, “Deep graph infomax,” _CoRR_ , vol. abs/1809.10341, 2018.
  * [25] D. Palaz, M. Magimai-Doss, and R. Collobert, “Analysis of CNN-based speech recognition system using raw speech as input,” in _Proc. of Interspeech_ , 2015.
  * [26] T. N. Sainath, R. J. Weiss, A. W. Senior, K. W. Wilson, and O. Vinyals, “Learning the speech front-end with raw waveform CLDNNs,” in _Proc. of Interspeech_ , 2015.
  * [27] Z. Tüske, P. Golik, R. Schlüter, and H. Ney, “Acoustic modeling with deep neural networks using raw time signal for LVCSR,” in _Proc. of Interspeech_ , 2014.
  * [28] G. Trigeorgis, F. Ringeval, R. Brueckner, E. Marchi, M. A. Nicolaou, B. Schuller, and S. Zafeiriou, “Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network,” in _Proc. of ICASSP_ , 2016, pp. 5200–5204.
  * [29] H. Muckenhirn, M. Magimai-Doss, and S. Marcel, “Towards directly modeling raw speech signal for speaker verification using CNNs,” in _Proc. of ICASSP_ , 2018.
  * [30] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering.” _CoRR_ , vol. abs/1503.03832, 2015.
  * [31] C. Li, X. Ma, B. Jiang, X. Li, X. Zhang, X. Liu, Y. Cao, A. Kannan, and Z. Zhu, “Deep speaker: an end-to-end neural speaker embedding system,” _CoRR_ , vol. abs/1705.02304, 2017.
  * [32] A. Jati and P. G. Georgiou, “Neural predictive coding using convolutional neural networks towards unsupervised learning of speaker characteristics,” _CoRR_ , vol. abs/1802.07860, 2018.
  * [33] L. Bahl, P. Brown, P. de Souza, and R. Mercer, “Maximum mutual information estimation of hidden markov model parameters for speech recognition,” in _Proc. of ICASSP_ , vol. 11, 1986, pp. 49–52.
  * [34] J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren, “DARPA TIMIT Acoustic Phonetic Continuous Speech Corpus CDROM,” 1993.
  * [35] A. Nagrani, J. S. Chung, and A. Zisserman, “Voxceleb: a large-scale speaker identification dataset,” in _Proc. of Interspech_ , 2017.
  * [36] M. Ravanelli, L. Cristoforetti, R. Gretter, M. Pellin, A. Sosi, and M. Omologo, “The DIRHA-ENGLISH corpus and related tasks for distant-speech recognition in domestic environments,” in _Proc. of ASRU 2015_ , pp. 275–282.
  * [37] M. Ravanelli, A. Sosi, P. Svaizer, and M. Omologo, “Impulse response estimation for robust speech recognition in a reverberant environment,” in _Proc. of EUSIPCO_ , 2012, pp. 1668–1672.
  * [38] J. Ba, R. Kiros, and G. E. Hinton, “Layer normalization,” _CoRR_ , vol. abs/1607.06450, 2016.
  * [39] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities improve neural network acoustic models,” in _Proc. of ICML_ , 2013.
  * [40] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _Proc. of ICML_ , 2015, pp. 448–456.
  * [41] M. Ravanelli, P. Brakel, M. Omologo, and Y. Bengio, “Batch-normalized joint training for DNN-based distant speech recognition,” in _Proc. of SLT_ , 2016\.
  * [42] E. Variani, X. Lei, E. McDermott, I. L. Moreno, and J. Gonzalez-Dominguez, “Deep neural networks for small footprint text-dependent speaker verification,” in _Proc. of ICASSP_ , 2014, pp. 4052–4056.
  * [43] N. Le and J. Odobez, “Robust and discriminative speaker embedding via intra-class distance variance regularization,” in _Proc. of Interspeech_ , 2018, pp. 2257–2261.
  * [44] A. K. Sarkar, D. Matrouf, P. Bousquet, and J. Bonastre, “Study of the effect of i-vector modeling on short and mismatch utterance duration for speaker verification,” in _Proc. of Interspeech_ , 2012, pp. 2662–2665.


11institutetext: Innopolis University, Innopolis, Russia
http://university.innopolis.ru 22institutetext: Softeam, France
http://www.softeam.com

# Extracting Software Requirements from Unstructured Documents

Vladimir Ivanov 11 Andrey Sadovykh 1122 Alexandr Naumchev 11 Alessandra
Bagnato 22 Kirill Yakovlev 11

###### Abstract

Requirements identification in textual documents or extraction is a tedious
and error prone task that many researchers suggest automating. We manually
annotated the PURE dataset and thus created a new one containing both
requirements and non-requirements. Using this dataset, we fine-tuned the BERT
model and compare the results with several baselines such as fastText and
ELMo. In order to evaluate the model on semantically more complex documents we
compare the PURE dataset results with experiments on Request For Information
(RFI) documents. The RFIs often include software requirements, but in a less
standardized way. The fine-tuned BERT showed promising results on PURE dataset
on the binary sentence classification task. Comparing with previous and recent
studies dealing with constrained inputs, our approach demonstrates high
performance in terms of precision and recall metrics, while being agnostic to
the unstructured textual input.

###### Keywords:

software requirements requirements elicitation BERT fastText ELMo sentence
classification

##  1 Introduction

Identifying and understanding technical requirements is a challenging task for
many reasons. Requirements come from various sources and in various forms.
These different forms may serve different purposes and levels, from initial
project proposals down to detailed contracts of individual methods. The forms
that appear earlier in the software process are consumed by more business-
oriented stakeholders. Technical people, such as testers, require detailed
specifications produced later in the software process to convert them into
tests. The evolving form of describing the initial problem is natural and
reflects the process of understanding the problem better with time. The
available requirement datasets reflect those variations.

One of such is the PURE dataset [1] presented by A. Ferrari et al., which
includes 79 Software Requirements Specification (SRS) documents. Mainly this
set is based on the Web search and includes documents with different
peculiarities as well as a lexicon with the widespread writing style of
requirements. Authors argue that this dataset fairly fits for various Natural
Language Processing (NLP) tasks such as requirements categorisation, ambiguity
detection, and equivalent requirements identification, etc. But at the same
time, additional expertise, as well as an annotation process, are primarily
required. However, some other datasets might be challenging. The Request for
Information (RFI) documents in itself specify an initial set of “best wishes”
for a required solution as well as a variety of constraints in terms of
deployment, compatibility or guarantees for maintenance and support. Those
documents are meant for engineers though the language used, format, styles and
structure are far from being standard. Those parameters largely depend on the
organization that issued those documents, personnel involved in designing
specifications and traditional practices of the domain the organization
belongs to. On the other side, the provider of the solution must quickly
qualify the RFI and prepare a well-thought offer that would maximize the value
for the customer at minimum costs for the provider. Such analysis is mostly
done manually that is costly, time-consuming and error-prone.

In this paper, we present an approach for extraction of requirements coming
from unstructured text sources. We fine-tuned the Bidirectional Encoder
Representations from Transformers (BERT) [2] model with PURE-based manually
annotated dataset of 7,745 sentences to identify technical requirements in
natural text. The ReqExp showed 92% precision and 80% recall on the PURE
dataset. In order to evaluate BERT, we compared the results with previous
state-or-art methods such as fastText and Embeddings from Language Model
(ELMo) with an SVM classifier on the same PURE-based dataset, where BERT
showed superior results in the precision, but worse results in the recall.
Afterwards we carried-out additional experiments with more semantically
difficult texts from RFIs. We applied all the candidate models trained on
PURE-based dataset to a private dataset of 380 sentences extracted from RFI
documents in order to evaluate the possibility to transfer the result to a new
domain. BERT again showed superior results in precision (90%), but inferior in
recall (71%). With a superior recall metrics fastText (82%) and ELMo (72%)
baseline methods showed less false negatives in detecting requirements, i.e.
passed through less of undetected requirements. The results obtained in the
limited set of experiments showed optimistic outcomes. We ran an additional
experiment with a larger training set, consisted of all the text samples from
the PURE and obtained precision of 86% and recall of 84% that suggests further
fine-tuning of ReqExp on a larger dataset. Moreover, the analysis of related
work indicated that the ReqExp approach has certain advantages, since, for
example, it is agnostic to the type and the structure of input documents.

Unfortunately, the datasets of comparable studies are not available for a
thorough analysis. In order to avoid the same problem with our study, we
provide our dataset with manually annotated sentences from the PURE corpus to
the community for comparison studies. The initial intention for this study
came from to Softeam company from France. They kindly provided the dataset of
RFI documents and endorsed our work. Softeam colleagues confirmed that the
results are promising for the industry with a high potential to be used in the
company’s products and processes.

To summarize this paper, the contributions of this work are: (1) adapted PURE-
based dataset for classification tasks with manually annotated sentences
containing both requirements and non-requirements; (2) the description of
BERT-based tool chain applying NLP for requirements extraction; (3) the
results of evaluation on SRS sentences from the PURE dataset in comparison
with baseline methods – fastText and ELMo; (4) the results of evaluation on a
provide dataset of RFI documents; (5) analysis of comparable studies.
Therefore, the current paper starts with the outline of the ReqExp approach in
Section 2 describing the conceptual method and the dataset construction for
profiling BERT neural network, followed by classifier models details in
Section 3 , continued by validation of the prototype on PURE-based dataset and
real RFI documents in Section 4, to be concluded by a comparison with the
related work in Section 5.

##  2 Methods and Datasets

In this section we describe the process of building a binary sentence
classifier. This process has several steps presented in Figure 1. The ultimate
goal was to efficiently build a training dataset that can be used to fine-tune
the BERT model and compare it with other advanced NLP methods. Bidirectional
Encoder Representations from Transformers (BERT) is a model by Google pre-
trained on a huge text corpus[2]. BERT was tested on multiple NLP tasks and
achieved state-of-the-art performance. A common way to use BERT for a domain-
specific task, such as software requirements extraction, is to fine-tune the
model i.e., to optimize model’s weights on a domain-specific dataset. To solve
a binary classification problem, such dataset should contain two subsets of
sentences: (i) a subset of sentences that contain requirements, and (ii) a
subset of sentences that does not contain requirements. However, there were no
such dataset readily available for requirements extraction.

Thus, the first step of our approach is creating the domain-specific training
dataset for fine-tuning. To speed up the process of corpus creation, we
decided to process an existing corpus of software specification documents. In
[1], Ferrari et al. presented a text corpus of public requirements documents
(PURE)111Data of the corpus can be found here:
https://zenodo.org/record/1414117. The dataset contains 79 publicly available
natural language requirements documents. We applied PURE corpus as our main
source of both requirements and non-requirements for designing our own
dataset. Initially, documents were presented in the form of raw text with
different format, structure and writing style.

At the beginning we made an attempt to extract sentences with requirements
using some parsing and extraction engines. Unfortunately, this idea was
rejected due to unsatisfactory quality of extracted text samples. Despite that
authors did provide a set of text requirements in the form of .json files to
simplify the extraction process, this was not enough to design a
representative subset for each class. It was decided to process documents
manually preserving initial syntax and semantics in the extracted text
samples. Still this process was associated with some challenges, which
required us to apply the following conventions:

  * •

Due to different length and structure of requirements, it was decided to
assume one sentence as one data unit. It means that a requirement, which
consists of several sentences, was divided into several units.

  * •

Sometimes requirements were written in the form of a list with many elements.
In such case, this list was transformed into one sentence separating parts by
commas. In some rare cases, where elements were individual sentences, such
requirement was divided into several sentences accordingly.

Overall, we extracted 7,745 sentences, where 4,145 were requirements and 3,600
were non-requirements from 30 documents. The extraction was done from the
appropriate sections of the documents. Usually each document provides
structural elements like table of elements, requirements-focused sections with
appropriate names or contextual footnotes. The latter one might have some
specific numbering used in a document and initially explained by the authors,
or lexical elements that are usually inherent for requirements such as modal
verbs like ’must’, ’shall’, ’should’, etc. When identifying requirements and
non-requirements we followed principle described in [3].

Sentences with requirements were manually extracted by one specialist in
Software Engineering area and then independently validated by another expert
from our research group. Some special cases were separately discussed in the
group to make annotation more consistent. In the case when there was no
agreement concerning which label should be utilized to those text samples,
they were removed to maintain consistency of the set.

Figure 1: The training process to obtain a model for binary sentence
classification for requirements extraction.

##  3 Models for Sentence Classification

For this study we chose two baseline approaches, which preceded the BERT
architecture, for binary sentence classification. Specifically we considered
fastText model and ELMo embeddings with a classical machine learning
classifier. The latter one was chosen through an experiment stage to choose
the most promising model based on the most explicit and informative metrics
during comparison.

###  3.1 fastText baseline model

fastText is a method for constructing word embeddings based on the Word2Vec
principle, but with some extension. An embedding for each word is learnt from
embedding vectors for character n-grams that constitute the word. Thus, each
word vector is represented as the sum of the n-gram vectors. This extends the
Word2Vec type models with subword information. Once a word is represented
using character n-grams, a skipgram model is trained to learn the embeddings.
This approach provides a fast and scalable solution, allowing to train models
on large corpora [4]. For the down-stream task (binary classification of
sentences) we use a linear classifier that is default (a multinomial logistic
regression) in the fastText library.

An important part in designing predictive models is associated with finding
the best hyper-parameters. Usually, it is a cumbersome and time-consuming
task, especially when doing it manually. For this purpose, we applied an
autotune feature located inside the fastText functionality by using the
validation set presented earlier 222https://fasttext.cc/docs/en/autotune.html.

###  3.2 ELMo with SVM classifier

Embeddings from Language Models, or ELMo is an advanced method for
constructing a contextualized word representation. The core ability is to
model different characteristics of word use together with linguistic context.
Specifically, these representations are internal states of a deep
bidirectional language model that is pre-trained on a large corpus. ELMo
maximizes the log-likelihood of the forward and backward directions in a
sequence of words [5]. These produced vectors can be applied as features for
various NLP purposes.

To make a decision for distinguishing among sentences with requirement and
without, one needs to use a classifier on top of the features extracted from
text. To find the best classifier, we compared several popular machine
learning models such as Logistic Regression, Random Forest classifier, Support
Vector Machines and Multi-Layer Perceptron. To this end the Grid Search was
applied with specific set of initially predefined hyperparameters for each
model. The process of assessment was based entirely on the training set. As a
result we chose with Support Vector Machines with polynomial kernel that
showed the highest result in terms of F1-score. The results related to this
baseline will be marked as “ELMo+SVM” in the following sections.

###  3.3 BERT-based approach

Recently using a large pre-trained language model has shown a breakthrough in
many tasks of natural language processing including text classification. The
baseline approach to transfer learning typically implies pre-training a neural
network on a large corpus of texts with further fine-tuning the model on a
specific task. In this study, we use pre-trained BERT [2]
model333Specifically, we use the BERT-base uncased model from
https://github.com/google-research/bert as a baseline. In all experiments with
BERT, we adopted the implementation of fine-tuning process for binary text
classification presented in [6]. Hyper-parameters used in fine-tuning are the
following: batch size is 64, learning rate is $2\cdot 10^{-5}$, max sequence
length is 100, number of epochs is 2. Overall schema is presented in Fig. 1.

##  4 Experimental Setup and Validation

###  4.1 Experimental setup

Empirical evaluation of model performance was carried out in two phases.
First, we used a classical approach with splitting dataset into disjoint sets
of sentences (namely, ‘train’, ‘test’ and ‘validation’ sets). All models were
trained on the same ‘train’ subset and tested on the same ‘test’ part from the
PURE dataset. We did not apply cross-validation due to the special rule of
splitting collection into train and test sets, i.e. sentences from some
document should appear in either the train set or in the test set. The dataset
as well as the split is available for the sake of reproducibility and
comparison of results.

Second, we used an out-of-sample data for evaluation of the classifiers. To
this end we tested the performance on sentences manually extracted from the
RFI documents. All models were trained on the same ‘train’ part and tested on
the same group of sentences extracted from RFI. We argue that such
experimental conditions are more relevant in practice. Indeed, we expected
decreasing of the performance on the RFI documents. This shows that fine-
tuning of large pre-trained language models is not a panacea in practice,
however it gives a viable alternative to rule-based tools. Finally, we show
that adding more data on the fine-tuning phase leads to better performance on
the out-of-sample data.

###  4.2 Train, test and validation sets

As for the data split, we had 7,745 sentences extracted from the PURE dataset
to split them up into train, test and validation subsets by the parts of
approximately 70%, 20 % and 10% accordingly. In order to consistently separate
our dataset, sentences from every document were fully added only to one
specific subset. We found a particular combination of documents for each
subset that preserve the stated separation and at the same time preserves the
balance of class labels. Statistics of the split is in sentences is the
following: train set (5306); test set (1534); validation set (905). The
dataset is available at https://zenodo.org/record/4630687.

###  4.3 Experiments with PURE documents

The main purpose of this study was about assessing three major classification
metrics: Precision, Recall, and F1-score. Those metrics are perceived as the
golden standard in Machine Learning and Deep Learning areas.

Result of the first phase of evaluation is presented in Table 1. As it was
expected, more advanced model (BERT) showed better results in terms of
F1-score. BERT-based model showed high precision (0.92) and lower recall
(0.8). Results of the BERT model is available at https://bit.ly/3oPElMm.
However, the values of precision and recall metrics behave differently for the
fastText and ELMo-based baselines. fastText-based classifier showed better
Recall (0.93) comparing with other architectures. This property might be
useful in some cases when it is necessary to extract more relevant sentences
and text patterns associated with requirements.

Table 1: Results of experiments with PURE Model | F1 | P | R | TP | TN | FP | FN
---|---|---|---|---|---|---|---
fastText | .81 | .72 | .93 | 763 | 419 | 295 | 57
ELMo+SVM | .83 | .78 | .88 | 827 | 364 | 231 | 112
BERT | .86 | .92 | .80 | 841 | 407 | 69 | 217

###  4.4 Experiments with RFI documents

In order to further evaluate our approach on less standardized requirements
documents, we applied our prototypes to five anonymized documents provided by
Softeam, France, as issued by their customers. We parsed the obtained
documents to extract the paragraphs and then triggered the automatic
requirements extraction from those paragraphs. After that we manually
annotated all the sentences and thus assessed the four major metrics - the
number of the true positives, the true negatives, the false positives and the
false negatives. Then the other metrics such as precision, recall and F1-score
were derived. The results can be found in Table 2.

To the validity of the results, it should be noted that the annotation was
done by one single specialist who was aware of the context of the Request for
Information (RFI) documents. This choice is justified to ensure the uniformity
of the manual annotation, since we noticed that the understanding of whether a
sentence is a requirement depends on context very much. In addition, it should
be mentioned that our attempts of manual annotation by junior researchers were
unfruitful since the results were completely unsatisfactory.

Table 2: Results of experiments with RFI Model | F1 | P | R | TP | TN | FP | FN
---|---|---|---|---|---|---|---
fastText | .65 | .54 | .82 | 146 | 83 | 120 | 31
ELMo+SVM | .69 | .66 | .73 | 177 | 48 | 89 | 66
BERT | .80 | .90 | .71 | 190 | 93 | 21 | 76

We also faced several challenges when were manually annotating the sentences
from RFIs. First, RFI documents often specify technical requirements in the
form of a question: “Will your solution provide a particular feature?” These
questions are difficult to qualify as technical requirements in traditional
sense. Second, a large part of RFI documents concern the constraints for the
submission of the response such as delay, length, and language of the
response. Should that be qualified as a requirement to documentation? All
these issues are postponed to the follow up studies where we intend to refine
the results. In particular, the study triggered more reflection about the
nature of the requirements, since the specification practice is far from the
form specified in the traditional requirements engineering standards.

Finally, we ran an additional experiment when we fine-tuned the best model on
the sentences from train, validation and test sets altogether with further
evaluation on the sentences from RFI documents. All parameters of fine-tuning
were the same as in the first set of experiments. The performance has grown
significantly (P=0.86, R=0.84, F1=0.85) compared to the initial setup with the
fine-tuning on the train set only (Table 2). Thus, this indicates a
possibility that the quality of the result can be improved further by
improving and augmenting train dataset.

##  5 Related Work

The related work analysis in the present section relies on a recent mapping
study of natural language processing (NLP) techniques in requirements
engineering [7]. The research group analyzed 404 primary studies relevant to
the NLP for requirements engineering (NLP4RE) domain. They mapped these
studies into several dimensions and identified the key categories along each
dimension. Here, we focus on the _Research Facet_ that enumerates categories
of research and of evaluation methods. Our contribution falls into the
_solution proposal_ category: we develop a new solution (a tool) for
identifying likely requirements in software-related texts. We used the
following methods for evaluating our proposed solution:

  * •

_Experience Report_ : “the result has been used on real examples, but not in
the form of case studies or controlled experiments, the evidence of its use is
collected informally or formally.”

  * •

_Laboratory Experiment with Software Subjects (LESS)_ : “a laboratory
experiment to compare the performance of newly proposed system with other
existing systems.”

Below we list studies from [7] that fall into the same categories as the
ReqExp approach does. Some of the studies make strong assumptions about the
respective input and/or output artifacts. We do not compare ReqExp with these
studies because we do not make such assumptions: ReqExp takes as input
arbitrary technical documents and identifies in them statements that look like
potential system requirements. The approaches that work with documents from
later stages of the requirements process are:

  * •

ARSENAL [8] deals with detailed behavior specifications taking the form of
“shall” statements and automatically converts them into formal models.

  * •

AutoAnnotator [9] assumes to receive requirement documents as input and
assigns semantic tags to these documents.

  * •

CHOReOS [10] works with well-formed requirements specified by domain experts.

  * •

Doc2Spec [11] works with JavaDoc and other in-code documentation.

  * •

ELICA [12] extracts and classifies requirements-relevant information based on
domain repositories and either existing requirement documents or elicitation
transcripts.

  * •

FENL [13] extracts software features from online software reviews.

  * •

GUEST [14] works with well-formed requirement documents to extract goal and
use case models.

  * •

GaiusT [15] applies to regulatory documents and extracts statements of
specific types with predefined structures.

  * •

NLP-KAOS [16] analyzes research abstracts and produces KAOS goal diagrams from
them.

  * •

SAFE [17] analyzes descriptions and user reviews of mobile apps in application
stores and extracts features from them.

  * •

SNACC [18] automatically checks design information for compliance with
regulatory documents.

  * •

Text2Policy [19] processes requirements in natural language to search
statements of a predefined type (access control policies).

  * •

UCTD [20] takes use cases as input and produces use cases enriched with
transaction trees.

  * •

Guidance Tool [21] works with texts that already look much like use cases and
transforms them into properly structured use cases.

  * •

Text2UseCase [22] works with texts that already describe scenarios and
transforms them into well-formed use cases.

The recent study [3] has explored the very related topic of demarcating
requirements in textual specifications with ML-based approaches. The authors
applied similar methods on a private dataset of 30 specification from
industrial companies with presumably 2,145 sentences containing 663
requirements and 1,482 non-requirements candidates. They obtained comparable
average precision of 81.2% and an average recall of 95.7%. Unfortunately, it
impossible to compare our results on the same dataset. However, we should
underline the major differences such as (1) our publicly available dataset is
much larger and more balanced; (2) we trained and validated the model on
completely different documents; (3) we applied much simpler end-to-end
training method without any costly feature engineering; (4) that way our
method is arguably more suitable for documents with complex semantics such as
RFIs.

Another similar study [23] has presented an application of BERT for multiclass
classification of requirements. Authors were focused on the following
subtasks:

  * •

Binary classification of functional (F) and non-functional (NFR) requirements
on the original dataset.

  * •

Binary and multiclass classification of the four most frequent NFR subclasses
on all NFR in the original dataset.

  * •

Multiclass classification of all NFR subclasses on NFR in the original NFR
dataset.

  * •

Binary classification of requirements based on functional and quality aspects
using the relabeled NFR dataset by another study.

For each subtask they provided classification results in terms of precision,
recall and F1-score. As it was expected, BERT showed the most promising
results comparing with other baseline approaches.

##  6 Conclusions

Addressing the customers’ needs involves a huge number of documents expressing
requirements in various forms. In the initial steps of those interactions with
customer the requirements are provided in RFI documents. Those documents often
combine technical and commercial requirements as well as provide other
constraints. The formats and styles of RFI vary a lot, which makes it
difficult to technically qualify the needs and timely prepare an offer.

In this paper we propose a requirements extraction approach based on Machine
Learning. We used BERT as the basis of the ML model, which we fine-tuned with
a manually annotated dataset constructed from an existing requirements
specification repository. We have compared the model to two competitive
baselines (fastText-based and ELMo-based). Indeed, the study is preliminary as
we just compare baselines models and do not propose any specific improvements
for the architectures. However, the results of the baselines are quite
promising and can be used in industry.

We validated our approach on five documents coming from Softeam company. The
approach and validation results were compared to our previous studies. The
five documents were annotated automatically at the sentence level (in total,
380 sentences with 6,676 tokens were extracted) and then evaluated by an
expert to calculate the performance scores. This analysis showed that our
approach superior, since it supports unconstrained textual specifications - it
is agnostic to the format of the input textual document. The performance
characteristics of our approach are on the comparable level even without
costly feature engineering though a progress can be made. We may further
evolve the prototype if, for example, we continue the fine-tuning. In
addition, by combining train, test and validation sets into one training set
has shown a dramatic increase of BERT performance on more semantically complex
RFI documents (P=0.86, R=0.84, F1=0.85). This suggests usefulness of further
work on manual annotation of the remaining documents from the PURE dataset and
augmenting the train set.

The collaboration with Softeam made us realize the industrial usefulness of
the approach in real cases as it can improve current tools and service by that
company. Overall, the obtained results motivate us to continue further evolve
our approach. In fact, the improvement will include (i) using a corpus of
software-related texts for further pre-training, (ii) fine-tuning models on a
multi-task datasets and (iii) annotating more data with requirements.

## Acknowledgments

This research has been financially supported by The Analytical Center for the
Government of the Russian Federation (Agreement No. 70-2021-00143 01.11.2021).

## References

  * [1] A. Ferrari, G. O. Spagnolo, and S. Gnesi, “Pure: A dataset of public requirements documents,” in _2017 IEEE 25th International Requirements Engineering Conference (RE)_ , Sep. 2017, pp. 502–505.
  * [2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [3] S. Abualhaija, C. Arora, M. Sabetzadeh, L. C. Briand, and E. Vaz, “A machine learning-based approach for demarcating requirements in textual specifications,” in _2019 IEEE 27th International Requirements Engineering Conference (RE)_ , 2019, pp. 51–62.
  * [4] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,” _Transactions of the Association for Computational Linguistics_ , vol. 5, pp. 135–146, 2017.
  * [5] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” _arXiv preprint arXiv:1802.05365_ , 2018.
  * [6] C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to fine-tune bert for text classification?” 2020.
  * [7] L. Zhao, W. Alhoshan, A. Ferrari, K. J. Letsholo, M. A. Ajagbe, E.-V. Chioasca, and R. T. Batista-Navarro, “Natural language processing (nlp) for requirements engineering: A systematic mapping study,” 2020. [Online]. Available: https://arxiv.org/abs/2004.01099
  * [8] S. Ghosh, D. Elenius, W. Li, P. Lincoln, N. Shankar, and W. Steiner, “Arsenal: Automatic requirements specification extraction from natural language,” in _NASA Formal Methods_ , S. Rayadurgam and O. Tkachuk, Eds. Cham: Springer International Publishing, 2016, pp. 41–46.
  * [9] S. J. Körner and M. Landhäußer, “Semantic enriching of natural language texts with automatic thematic role annotation,” in _Natural Language Processing and Information Systems_ , C. J. Hopfe, Y. Rezgui, E. Métais, A. Preece, and H. Li, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 92–99.
  * [10] N. Maiden, J. Lockerbie, K. Zachos, A. Bertolino, G. De Angelis, and F. Lonetti, “A requirements-led approach for specifying qos-aware service choreographies: An experience report,” in _Requirements Engineering: Foundation for Software Quality_ , C. Salinesi and I. van de Weerd, Eds. Cham: Springer International Publishing, 2014, pp. 239–253.
  * [11] H. Zhong, L. Zhang, T. Xie, and H. Mei, “Inferring resource specifications from natural language api documentation,” in _2009 IEEE/ACM International Conference on Automated Software Engineering_ , 2009, pp. 307–318.
  * [12] Z. Shakeri Hossein Abad, V. Gervasi, D. Zowghi, and K. Barker, “Elica: An automated tool for dynamic extraction of requirements relevant information,” in _2018 5th International Workshop on Artificial Intelligence for Requirements Engineering (AIRE)_ , 2018, pp. 8–14.
  * [13] N. H. Bakar, Z. M. Kasirun, N. Salleh, and H. A. Jalab, “Extracting features from online software reviews to aid requirements reuse,” _Applied Soft Computing_ , vol. 49, pp. 1297 – 1315, 2016. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S1568494616303830
  * [14] T. H. Nguyen, J. Grundy, and M. Almorsy, “Rule-based extraction of goal-use case models from text,” in _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering_ , ser. ESEC/FSE 2015. New York, NY, USA: Association for Computing Machinery, 2015, p. 591–601. [Online]. Available: https://doi.org/10.1145/2786805.2786876
  * [15] N. Zeni, N. Kiyavitskaya, L. Mich, J. R. Cordy, and J. Mylopoulos, “Gaiust: supporting the extraction of rights and obligations for regulatory compliance,” _Requir. Eng._ , vol. 20, no. 1, pp. 1–22, 2015.
  * [16] E. Casagrande, S. Woldeamlak, W. L. Woon, H. H. Zeineldin, and D. Svetinovic, “Nlp-kaos for systems goal elicitation: Smart metering system case study,” _IEEE Transactions on Software Engineering_ , vol. 40, no. 10, pp. 941–956, 2014.
  * [17] T. Johann, C. Stanik, A. M. Alizadeh B., and W. Maalej, “Safe: A simple approach for feature extraction from app descriptions and app reviews,” in _2017 IEEE 25th International Requirements Engineering Conference (RE)_ , 2017, pp. 21–30.
  * [18] J. Zhang and N. M. El-Gohary, “Integrating semantic nlp and logic reasoning into a unified system for fully-automated code checking,” _Automation in Construction_ , vol. 73, pp. 45 – 57, 2017. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0926580516301819
  * [19] X. Xiao, A. Paradkar, S. Thummalapenta, and T. Xie, “Automated extraction of security policies from natural-language software documents,” in _Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering_ , ser. FSE ’12. New York, NY, USA: Association for Computing Machinery, 2012\. [Online]. Available: https://doi.org/10.1145/2393596.2393608
  * [20] M. Ochodek and J. Nawrocki, “Automatic transactions identification in use cases,” in _Balancing Agility and Formalism in Software Engineering_ , B. Meyer, J. R. Nawrocki, and B. Walter, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 55–68.
  * [21] C. Rolland and C. B. Achour, “Guiding the construction of textual use case specifications,” _Data & Knowledge Engineering_, vol. 25, no. 1, pp. 125 – 160, 1998. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0169023X97862234
  * [22] S. Tiwari, D. Ameta, and A. Banerjee, “An approach to identify use case scenarios from textual requirements specification,” in _Proceedings of the 12th Innovations on Software Engineering Conference (Formerly Known as India Software Engineering Conference)_ , ser. ISEC’19. New York, NY, USA: Association for Computing Machinery, 2019\. [Online]. Available: https://doi.org/10.1145/3299771.3299774
  * [23] T. Hey, J. Keim, A. Koziolek, and W. F. Tichy, “Norbert: Transfer learning for requirements classification,” in _2020 IEEE 28th International Requirements Engineering Conference (RE)_ , 2020, pp. 169–179.