∎

11institutetext:  Renaud Rwemalika 22institutetext: University of Luxembourg
22email<EMAIL_ADDRESS>33institutetext: Sarra Habchi
44institutetext: University of Luxembourg 44email<EMAIL_ADDRESS>55institutetext: Mike Papadakis 66institutetext: University of Luxembourg
66email<EMAIL_ADDRESS>77institutetext: Yves Le Traon
88institutetext: University of Luxembourg 88email<EMAIL_ADDRESS>99institutetext: Marie-Claude Brasseur 1010institutetext: BGL BNP Paribas
1010email<EMAIL_ADDRESS>
# Smells in System User Interactive Tests

Renaud Rwemalika  Sarra Habchi  Mike Papadakis  Yves Le Traon  Marie-Claude
Brasseur

## Abstract

Test smells are known as bad development practices that reflect poor design
and implementation choices in software tests. Over the last decade, test
smells were heavily studied to measure their prevalence and impacts on test
maintainability. However, these studies focused mainly on the unit level and
to this day, the work on system tests that interact with the System Under Test
through a Graphical User Interface remains limited. To fill the gap, we
conduct an exploratory analysis of test smells occurring in System User
Interactive Tests (SUIT). First, based on a multi-vocal literature review, we
propose a catalog of 35 SUIT-specific smells. Then, we conduct an empirical
analysis to assess the prevalence and refactoring of these smells in 48
industrial test suites and 12 open-source projects. We show that the same type
of smells tends to appear in industrial and open-source projects, but the
symptoms are not addressed in the same way. Smells such as Obscure Test,
Sneaky Checking, and Over Checking show symptoms in more than 70% of the
tests. Yet refactoring actions are much less frequent with less than 50% of
the affected tests ever undergoing refactoring. Interestingly, while
refactoring actions are rare, some smells, such as Narcissistic, disappear
through the removal of old symptomatic tests and the introduction of new tests
not presenting such symptoms.

##  1 Introduction

Today, Graphical User Interface (GUI) are becoming ubiquitous as a means for
users to interact with a software application (Myers and Rosson, 1992; Myers,
1995; Brooks et al., 2009; Memon and Nguyen, 2010). As a consequence of the
predominance of GUI-based applications, a lot of effort has been geared
towards GUI test automation, notably focusing on desktop applications (Nguyen
et al., 2014; Advolodkin, 2018; Pezzè et al., 2018), web applications (Mesbah
and van Deursen, 2009; Biagiola et al., 2019), mobile applications (Machiry et
al., 2013; Gomez et al., 2013; Mao et al., 2016; Salihu et al., 2019; Yu et
al., 2019) or cross-platform applications (Canny et al., 2020).

Thanks to the maturity of test automation frameworks, both commercial and
open-source projects are increasingly adopting automated test suites to ensure
software quality. However, testing interactive applications remains a major
challenge for software projects. Indeed, similarly to production code, quality
issues might emerge in the test code. For example, prior work showed the
existence of sub-optimal design choices that lower the quality of the test
code (van Deursen et al., 2001; Meszaros, 2007; Reichhart et al., 2007; Van
Rompaey et al., 2007; Bavota et al., 2015; Tufano et al., 2016; Bowes et al.,
2017; Kim, 2020; Peruma et al., 2020) in the form of test smells.

The concept of smells was first introduced by Fowler et al. (1999), who
defined code smells as poor design and implementation choices that hinder the
system maintainability. Later the concept of smells was extended to cover
maintainability issues in test code (van Deursen et al., 2001). Test smells
differ from classical code smells by covering the organization,
implementation, and interactions of software tests (Moonen et al., 2008).

In this paper, we focus on smells that appear in tests residing at the higher-
levels of the test pyramid, namely System User Interactive Tests (SUITs). More
specifically, a SUIT can be defined as an automatic test exercising a software
application with a graphical front-end to guarantee that it meets its
specification by performing a sequence of events against the GUI elements
(Cunha et al., 2010; Banerjee et al., 2013; Issa et al., 2012). As such, these
tests present unique characteristics in their (1) structure, (2) interaction
with the System Under Test (SUT), and (3) the actors involved in their
maintenance.

Indeed, interacting through the user interface, SUITs present unique
characteristics that originate from their mode of interaction and
synchronization with the SUT. SUITs consider the SUT as a black box, ignoring
its implementation details to test higher behavioral components. Thus, SUITs
present fundamental differences in the way they are crafted when compared to
unit tests. For example, the smell _Mystery Guest_ , which is well studied in
test smell literature (van Deursen et al., 2001; Bavota et al., 2015; Tufano
et al., 2016; De Bleser et al., 2019; Peruma et al., 2020; Virgínio et al.,
2020) appears when a test class relies on an external resource file. However,
in the case of SUITs, extracting away data is considered a good practice and
is highly used in data-driven testing (Baker et al., 2008) in order to
decouple the data from the behavior specifications. Indeed, one of the
functions of a SUIT being communication between different roles, namely
business analysts, testers and developers, technical details are hidden away
to ease communication between the stakeholders. Our study aims to understand
the effects of test smells when accounting for the unique characteristics
present in SUITs. Specifically, we aim to answer the following research
questions:

  * •

RQ1: What are the SUIT smells studied in academic and grey literature?
Goal: This question aims to explore and identify test smells that are specific
to SUITs. Identifying and categorizing test smells that are mentioned and
discussed in the grey literature bridges the gap between research community
and practitioners since the majority of grey literature is based on the
practitioners views. In view of this, our intent is to produce an almost
exhaustive catalog of known SUIT smells along with their definition and
impact.

  * •

RQ2: How widespread are SUIT smell symptoms in SUITs?
Goal: This question aims at assessing the prevalence of SUIT smells in
industrial and open-source projects and how different smells appear in the
test codebases. An additional aim in analysing this question is the analysis
of how smells are viewed and managed by practitioners. Empirically answering
this question provides a sanity check on the importance of the SUIT smells,
i.e., very rare smells are probably not interesting as they are not part of
the current practice.

  * •

RQ3: How often do we observe refactoring actions removing SUIT smell symptoms?
Goal: This question aims to analyse the refactoring actions performed by
maintainers to remove SUIT smells. While there may be a large amount of smells
present in the test codebase, practitioners may be unaware of their presence.
To explore this, we identify and detect fine-grained smell refactoring
operations that reflect an interest by the maintainers of the test suite.
Answering this question provides evidence in the importance of the smells,
i.e., they are associated with development time and effort.

To answer these questions, we combine a multi-vocal literature review and an
empirical study on a large industrial project and 12 open-source repositories.
Relying on the multivocal literature review, we answer RQ1 and build a catalog
of 35 SUIT smells. For 16 out of the 35 SUIT smells of this catalog we propose
an automated approach for detecting their diffusion and refactoring.
Leveraging this approach, we evaluate the prevalence of SUIT smells and their
removal in more than two million tests from our dataset composed of industrial
and open-source projects. The main contributions of this study are:

  * •

We introduce the first catalog of SUIT-specific smells building on the
knowledge of both researchers and practitioners. Alongside with it, we propose
an open-source automated tool111Available at https://github.com/UL-SnT-
Serval/ikora-evolution to detect symptoms and refactorings for 16 SUIT smells.

  * •

We show that the same type of smells tend to appear in industrial and open-
source projects, but the symptoms are not addressed in the same way. The
smells _Hard Coded Values_ (85% to 90% of tests), _Over Checking_ (75% to 80%
of tests), and _Sneaky Checking_ (around 70% of tests) are the most prevalent
SUIT smells in both types of projects.

  * •

Our empirical results suggest that less than half of SUITs ever experience
smell refactorings. _Missing Assertion_ is a unique exception with up to 90%
of the symptomatic tests refactored. Interestingly, while refactoring actions
are rare, SUIT smells like _Narcissistic_ and _Middle Man_ still disappear
from the codebase as a side effect of unintended maintenance and refactoring
operations.

##  2 Background

This section presents Keyword-Driven Testing (KDT), a popular paradigm to
create SUITs, along with an illustrative example and a description of its
supporting framework, Robot Framework222https://robotframework.org/.

###  2.1 Keyword-Driven Testing

KDT is a software testing technique using _Keywords_ where each _Keyword_
describes a set of actions that are required to perform a specific step.

Through the use of _Keywords_ , KDT aims at separating test design from the
technical implementation of tests, thus, limiting exposure to unnecessary
details and avoiding duplication. KDT advocates that this separation of
concerns makes tests easier to write and more maintainable. On top of that,
the separation enables experts from different fields and backgrounds to work
together, at different levels of abstraction (Jingfan Tang et al., 2008). For
these reasons, KDT is an ideal candidate to write SUITs.

Humble and Farley (2010) formalize a hierarchy under three distinctive layers:
(1) the acceptance criteria which describes the functional behavior, usually
written in a form close to natural language; (2) the application driver layer
that contains the underlying implementation of test using the vocabulary from
the application domain; and (3) the application driver layer that understands
how to interact with the SUT and is expressed in the domain language of the
driver that is used to communicate with the SUT.

Figure 1 shows an example of a KDT test adapted from the official Robot
Framework documentation. The test A user logs in with his username and
password (line 6) is responsible for validating the correct behavior of the
login form in an imaginary SUT by interacting with its user interface.

As can be seen from the listing, most parts of this fully automated test are
written in plain English. This enables the unobstructed collaboration in the
creation and analysis of the tests between different experts. To further
extend the reuse of _Keywords_ , they can have _Arguments_. For instance, the
_Library Keyword_ Open browser (line 21) takes two arguments, ${LOGIN URL} and
${BROWSER}.

Lines 6–8 present the test steps, which represent the test acceptance
criteria. In KDT, each step is implemented as a _Keyword_. In turn, these
_Keywords_ are defined in their respective definition blocks between lines 12
and 41. The actions composing the body of a _Keyword_ are subsequent _Keyword_
calls.

A the end of the call tree, we find _Library Keywords_ performing concrete
actions such as interacting with the SUT or managing the test execution. For
example, in Figure 1, line 2 shows that the script is using the web driver
automation library _Selenium_ to interact with the SUT. This library allows
the test script to perform concrete actions such as navigating through the
website using the Go To _Library Keywords_ at line 26 and interact with
elements present on a web page with library keywords such as Input Text (lines
31 and 35) or Click Button (line 38). Furthermore, _Library Keywords_ can be
used to perform assertions such as Title Should Be at line 41 ensure that the
title (title element in the header of the DOM) is of a specific value.

Thus, KDT makes it easy to follow the three-layer architecture described by
Humble and Farley (2010). In this model, the _Test Case_ and its steps are the
acceptance criteria, the intermediate _User Keywords_ are the implementation
layer and finally, the _Library Keywords_ interacting with the system under
test represent the application driver layer.

Figure 1: Example of Robot Framework test

⬇

1 *** Settings ***

2 Test Teardown Close All Browsers

3 Library Selenium2Library

4

5 *** Test Cases ***

6 A user logs in with his username and password

7 Given browser is opened to login page

8 When user "demo" logs in with password "mode"

9 Then welcome page should be open

10

11 *** Keywords ***

12 Browser is opened to login page

13 Open browser to login page

14

15 User "${username}" logs in with password "${password}"

16 Input username ${username}

17 Input password ${password}

18 Submit credentials

19

20 Open Browser To Login Page

21 Open Browser ${LOGIN URL} ${BROWSER}

22 Maximize Browser Window

23 Title Should Be Login Page

24

25 Go To Login Page

26 Go To ${LOGIN URL}

27 Login Page Should Be Open

28

29 Input Username

30 [Arguments] ${username}

31 Input Text username_id ${username}

32

33 Input Password

34 [Arguments] ${password}

35 Input Text password_id ${password}

36

37 Submit Credentials

38 Click Button validate_id

39

40 Welcome Page Should Be Open

41 Title Should Be Welcome Page

42

43 *** Variables ***

44 ${SERVER} localhost:7272

45 ${BROWSER} Chrome

46 ${LOGIN URL} http://${SERVER}

###  2.2 Robot Framework

Robot Framework is an open-source KDT framework that was originally developed
by Nokia Networks and is widely adopted by companies. Robot framework relies
on human-readable keywords that make the code easy to understand by non-
technical people. Indeed, as shown in Section 2.1, the acceptance criteria of
the test A user logs in with his username and password in Figure 1 (lines
6-9), written using the Robot Framework syntax, is entirely expressed using
natural language.

One of the main advantages of Robot Framework is its high modularity. The
framework is platform-agnostic and thanks to its driver plugin architecture,
the core framework does not require any knowledge of the SUT. Indeed,
libraries accomplishing the concrete actions are created either by the Robot
Framework core team and integrated in the language, developed by third parties
and distributed as library plugins, or developed directly by the test team in
which case the library code can be maintained alongside the test code. Thus,
the concrete implementation of an action in a Robot Framework test script is
hidden from the test. On the down side, while good practices promote such
isolation and separation of concerns, the introduction of suboptimal decision
can hinder the readability of the test code, alter interactions with the SUT,
or just make it harder to adapt the test codebase to SUT evolution.

Figure 2: Tree representation of a Robot Framework test (Rwemalika et al.,
2019a).

A Robot Framework script can be represented using a tree structure, _i.e._ a
rooted, ordered, directed, acyclic graph. Figure 2 shows the tree
representation of the test presented in Listing 1. The nodes of the tree
represent an entity from the Robot Framework language (_Test Case_ , _Keyword_
or _Variable_) and the edges represent _Keyword_ calls. At the root of the
tree, we find the _Test Case_ (purple in the figure). Each test sep composing
the acceptance criteria is represented by a call from the _Test Case_ to a
_Keyword_. The intermediate nodes (white in the figure) represent _User
Keywords_. _User Keywords_ are calling subsequent _Keywords_ which can be
either other _User Keywords_ or _Library Keywords_. Finally, the leaves nodes
are the _Library Keywords_ (in green in the figure). Because they perform
concrete actions that are hidden from the test script, they do appear in the
graph as leaf nodes. When building the graph representation of the tests, we
further annotate the _Library Keyword_ with a category defining which type of
action they perform (Rwemalika et al., 2019a), namely: INTERACTION, ASSERTION,
CONTROLFLOW, GETTER, LOGGING and SYNC.

Building on this model we rely on graph theory algorithms to traverse all the
nodes of a test and extract properties of the tests and the symptoms presented
in Section 4.2.

##  3 Experimental Design

###  3.1 RQ1: Identification of SUIT Smells

To perform a study on the impact of smells in SUITs, we first need to build a
reliable and representative catalog of test smells. To this end, we start our
investigation by collecting SUIT smells presented in both academic and grey
literature. We classify in the academic or white literature papers that are
published in peer reviewed conferences or journals. On the other hand, we
classify as grey literature white-papers, magazines, online blog-post,
question-answers sites, survey results and technical reports following the
methodology presented in Ricca and Stocco (2021). Indeed, the grey literature
constitutes a rich source of documents where practitioners share their
experiences, propose guidelines or even ask and answer questions. Thus, we
conduct a multivocal literature review following the steps depicted by Garousi
and Küçük (2018). Figure 3 summarizes our adoption of these steps in the
process of building a catalog of SUIT smells.

Figure 3: Overview of the process to establish the SUIT smell catalog.

##### Initial Search.

To mine the academic sources we rely on the Google Scholar Search engine,
whereas for the grey literature we rely on the Google Search engine. These two
search engines are known to subsume other databases and repositories hence we
limit our investigation to them. We compile a list of search terms such as
“acceptance test”, “test automation”, “end-to-end test”, “system test” and
“behavior test” to define the type of tests that we are targeting and combine
those strings with each of the following: “smell”, “symptom”, “anti pattern”
and “bad practice”. This allows to generate the following search string for
the engines: _software and ("acceptance test" OR "test automation" OR "end-to-
end test" OR "system test" OR "functional test") AND ("smell" OR "symptom" OR
"anti pattern" OR "bad practice")_. To validate the search string we run it
and make sure that relevant documents known _a priori_ appear in the results
(Kitchenham and Charters, 2007; Ricca and Stocco, 2021). Then, we proceed to a
systematic review of all the sources in order to identify test smells.
However, because the search string returns about 1,370,000 results in the
Google Search Engine and about 12,100 results in the Google Scholar Search
Engine, we restrict our analysis to an initial pool composed of the first 200
results in each engine. From those we read the title and abstract available to
decide whether or not to further analyze the articles. This leads to a initial
list composed of 47 sources in the grey literature and 32 sources in the
academic literature for a total of 79 sources.

##### Inclusion/Exclusion Criteria.

To be eligible for our study, a smell must address the code of a SUITs.
However, the terms adopted in both industry and academia might not refer to
how the test interact with the application, but rather to what is the intent
of the test (_e.g._ acceptance testing) or its scope (_e.g._ system test).
This leads to a series of sources discussing tests that are not SUITs and are
not compatible with them, typically, white-box tests. Furthermore, some
sources address testing issues that are not related to the codebase such as
organizational smells, which are outside the scope of this study. Indeed, some
smells do not target the test code but testers behavior _e.g._ , _Making
Intermittent Bugs Low Priority_ (StackExchange, 2017) for which bugs making
the functional test suite fail intermittently at low intervals are ignored. By
excluding these irrelevant sources, we reduce our list to 32 sources from the
grey literature and 6 sources in the academic literature for a total of 38
sources. The drastic drop in the academic literature is due to the large
number of studies targeting unit tests instead of SUITs. The selection process
is depicted in the summary sheet333Available as a spreadsheet.

##### Smell Identification.

Relying on this final pool of sources, where each source contains at least one
SUIT smell, we proceed to the smell identification where we filter out all the
smells that are not suitable for our analysis. Indeed, some smells are not
applicable to the framework under study _e.g._ , _Dependence on Record and
Playback_ (StackExchange, 2017) in which tests are created using Record and
Playback, a feature that is not supported by Robot Framework. Other smells are
too technology specific _e.g._ , not using the page object pattern in Selenium
tests (Advolodkin, 2018). The outcome of this process is an initial smell
catalog of 84 unique SUIT smells for which we can derive a metric by analyzing
the code representing the symptom observed.

##### Smell Generalization.

Some SUIT smells gathered in the previous step exhibit large overlaps; thus,
we proceed to a smell generalization step. For instance, the smell _Enter
Enter Click Click_ (Buwalda, 2015) is grouped with _Comments and documentation
instead of abstraction_ (Klarck, 2014), since both smells target the presence
of low level actions in the acceptance criteria but in the latter a specific
emphasis is put on the documentation aspect. Hence, we consider those two
smells to present similar symptoms and effects and we group them in one smell
named _Lack of encapsulation_ (Chen and Wang, 2012; Klarck, 2014; Buwalda,
2015; England, 2016; Renaudin, 2016). Finally, we observe that some test
smells are subsuming others. This is the case for the smells _Hardcoded
Values_ and _Using condition in test logic_ that are both subsumed by the
definition of _Obscure test_ (Hauptmann et al., 2013; Gawinecki, 2016;
Siminiuc, 2019). In this example we only keep the latter. Finally, from this
list, we filter out any test smell that would require the test to be executed
in order to be observed. As a result, the outcome of this step is a list of 35
SUIT smells444The complete catalog is available at https://github.com/UL-SnT-
Serval/suit-smell-catalog that can be detected statically.

Finally, conducting our study using Robot Framework, we exclude four test
smells that cannot be observed in this specific language and its associated
framework. Furthermore, we omit 15 test smells because, despite our best
efforts, no automated metric avoiding false positive could be constructed by
analyzing the test code. Hence, in this work, we present a list of 16 SUIT
smells. A comprehensive description of each smell is presented in Section 4.1.

###  3.2 Dataset

To answer RQ2 and RQ3, we conduct a case study on 13 test suites written in
Robot Framework. We establish two sets of projects: the first set is composed
of 48 repositories from our partner site, BGL BNP Paribas and the second set
of projects is composed of 12 open-source projects mined from Github. Table 1
summarizes the overall properties of these projects. In the following we
describe the projects and present the collection process.

####  3.2.1 Industrial project:

We leverage the codebase of our industrial partner, BGL BNP Paribas. The test
suite is maintained by a dedicated quality assurance team which role is to
ensure the compliance to the requirements of the products deployed to
production. Thus, the team tests desktop, web and mobile applications that are
depending on services developed following a service-oriented architecture
(SOA). The goal of the team is to assessing compliance to functional
requirements, applications are tested in a black box fashion through their
user interface.

Historically, the team relied on manual testing to perform its tasks. However,
with the release cycles becoming shorter and the number of tests increasing,
the execution burden on the team became unmaintainable. Thus, in the end of
2016, the team started migrating from an manual execution of the tests to the
execution of automated tests written in Robot Framework. Robot Framework was
chosen because of the ease it provides to develop tests targeting application
written in different technologies requiring different mode of interactions.

Today, the test suite consists of 559 tests stored across 48 repositories on
an on premise GitLab instance. While some repositories are defining _Test
Cases_ , others are used as resources of common _Keywords_ where a series of
generic _Keywords_ specific to the BGL BNP Paribas architecture have been
created to help with the development effort as well as to avoid code
duplication. A typical example is the login to the ecosystem which is common
to a large amount of services, and consequently can be mutualized. Hence, in
this study, we merge all the repositories to count them as one project, which
explains the larger number of _User Keywords_ observed in Table 1 for the
project _bgl_.

####  3.2.2 Open-source projects:

To collect the open-source test suites, we use the Github Search API to mine
repositories containing suitable test suites, _i.e_ Robot Framework test
suites. From the results of this first step, we filter out toy projects and
tutorials, projects using Robot Framework for its robot process automation
capabilities in production and libraries extending the capabilities of Robot
Framework. Following this approach, we gather 23 repositories from Github.
Finally, to ensure that maintenance was performed on the test suite itself, we
analyze the number of commit involving the Robot Framework test suite for each
project, _i.e._ SUIT modifying commit, and discard any project with less than
100 SUIT modifying commits. This process yields a total of 12 repositories
presented in Table 1.

The data collection process leads to a total of 2,884,383 SUITs analyzed
across all the versions of all the projects where 2,742,271 originate from the
open-source projects and 142,112 from the industrial projects gathered at BGL
BNP Paribas.

Name |  LoC |  #Commits |  #TestCases |  #Keywords
---|---|---|---|---
bgl |  57030 |  309 |  559 |  5000
apinf |  1068 |  345 |  74 |  138
bikalims |  4446 |  1279 |  76 |  172
collective-cover |  1317 |  943 |  24 |  31
cumulus-ci |  1599 |  1429 |  132 |  34
harbor |  7270 |  3131 |  246 |  472
ifs |  8466 |  5606 |  494 |  605
mms-alfresco |  1797 |  504 |  156 |  8
mystamps |  2361 |  353 |  212 |  111
ozone |  2636 |  271 |  230 |  90
plone |  1046 |  211 |  59 |  163
plone-intranet |  2225 |  1759 |  213 |  134
rspamd |  3397 |  685 |  429 |  151
Table 1: Metrics for the 13 projects under study. _LoC_ is the number of lines
of Robot Framework Code, _#Commits_ is the number of commits implicating Robot
Framework code, and _#TestCases_ and _#Keywords_ are the number of test cases
and user keywords respectively in the last commit of the projects.

These 12 projects cover various technology stacks. For instance, BGL BNP
Paribas developed internal services using Java Swing which rely on either the
API exposed by the Swing development or rely on computer vision, through the
use of the Sikuli555http://sikulix.com/ library. BGL BNP Paribas also use the
Robot Framework to test its mobile client application relying on the
Appium666https://appium.io/ Library. Finally, the majority of the test suites
in both industrial and open-source projects from this dataset target web
applications. The SUITs interact with such applications (1) through their
webpages by interacting with the DOM, (2) through their APIs or (3) directly
accessing the state of the database after an operation. Note that some
operations generate emails or reports, thus, some tests rely on operating
system interaction to check that an email was properly sent through a mail
client or assess the existence and parses the content of a generated PDF
documents.

To conclude this section, we observe that the dataset is composed of projects
covering various domains, technology stacks, sizes and modes of development.
We believe that this diversity allows to avoid biases observed in one type of
projects and improves the generalization of the observations.

###  3.3 RQ2: SUIT Smells Distribution

For each of the SUIT smells that we gathered, we compute a metric to
automatically measure a smelliness score for the affected test. We rely on
heuristics to construct these metrics. To this end, we apply the high-level
investigation mechanism framed by Marinescu (2004) called “detection
strategy”. As defined by the authors, a “detection strategy” is _a generic
mechanism for analyzing a source code model using metrics_. The detection
strategies are formulated in a series of steps: (1) Break-down informal rules
into symptoms that can be captured by a single metric; (2) Select a proper set
of metrics to quantify each of the symptoms; (3) Define thresholds that
classify a test as smelly or not; (4) Use operators to correlate the symptoms
leading to the final rule for detecting the smells.

Note that the step 3 of this approach describes the definition of a threshold.
Even though different approaches have been proposed, _e.g._ using semantical
properties and statistical distributions (Marinescu, 2004) or using Bayesian
belief networks (Khomh et al., 2009), defining a good threshold remains a hard
and error prone task. Thus, more recently, researchers have been trying to
avoid this limitation by using machine learning to directly learn what a smell
looks like avoiding altogether the use a threshold (Arcelli Fontana et al.,
2016). In this work, in order to avoid any bias generated by a binary
classification, we do not apply any empirical threshold (with the exception of
one smell, _Long Test Steps_) but focus our analysis on the observation of the
symptoms associated with SUIT smells. Hence, for each test we attribute a
metric that represents the number of symptoms that are observed in a test.
Furthermore, we propose a density metric, which normalizes the number of
symptoms observed over the worst case scenario for a given test. Indeed, our
goal is to analyze how the symptoms are treated by the maintainers of the test
codebase and not to classify tests as smelly or not.

To extract code metrics, we rely on a parsing engine presented in Rwemalika et
al. (2019a). The parser generates for each test a call tree, where the root is
the entry point of the test and the leaf nodes are the actions generated on
the system under test, or the application driver layer if we refer to the
three-layer architecture described by Humble and Farley (2010). Each leaf node
is annotated with information such as the type of action (assertion, getter,
event, etc.) and each variable can be inferred a type based on the _Library
Keywords_ calling it. To create the metrics, we rely on heuristics exhibiting
the symptom associated with the smell. While building the heuristics, we focus
on a high precision, even though this might lead to low recall. For instance,
for the _Over-checking_ , we simply get the number of assertions over the
total number of nodes in the application driver layer, therefore, missing
information about the position and of the assertions. Hence, we might end up
missing tests exhibiting the smell when the assertions are performed
prematurely, checking every action setting up the environment. However, when
the ratio is high, it is a clear indication that too many assertions are made,
making the intent of the test less clear.

Finally, note that for one of the SUIT smells, namely _Using Personal Pronoun_
, relying on code metrics alone does not provide sufficient information.
Hence, we rely on textual metrics to detect the symptom. More specifically, we
use natural language processing to extract information about the name of some
_Keywords_ and use text tagging to define whether or not the subject of the
_Keywords_ is a first person pronoun or not. All the results for this research
question are presented in Section 4.3.

When computing the metrics, we introduce two types of metrics: a count metric
$S$ and a density metric $D$. The count metric $S$ counts the number of
instances of a symptom observed in a test. On the other hand, the density
metric $D$ provides an indication of the number of instances over a maximum
value that could have appeared in the test. Thus, the density metrics vary
between 0 and 1, 0 indicating the absence of any of the symptoms and 1
indicates that for each of the possible location for the symptom to be
present, it appeared in the test. Note that we focus on metrics with a high
precision at the detriment of the recall. Indeed, some of the metrics only
cover one form of symptoms leading to a smell but lead to low rate of false
positive.

###  3.4 RQ3: SUIT Smells Refactoring

Refactoring can be defined as a technique to improve the design of a system
and enable its evolution (Fowler et al., 1999). Relying on this definition, to
answer our third research question, we conduct an analysis of the refactoring
actions during the maintenance of the test suites. To do so, we collect every
pair of subsequent versions and identify refactoring changes occurring in the
test codebase. A key component in this type of analysis is to properly
identify what constitutes a valid refactoring action. Indeed, just observing a
decreases in smell metrics does not imply that a smell has been specifically
addressed. Other actions such as changing the scope of a test can, as a side
effect, remove symptoms of a SUIT smells without specifically targeting it.
Thus, following the line of work present in the literature to identify
refactoring activity (Tsantalis et al., 2013; Silva and Valente, 2017), we
address this limitation by using heuristics based on static test code
analysis. More specifically, we use the fine-grained change algorithm
presented in Rwemalika et al. (2019a) to extract instances of refactoring
patterns. These patterns are derived from the definition of the symptoms
present in the literature.

Consequently, for each SUIT smell $s$ we store the set of nodes $N$ that are
exhibiting a symptom of the SUIT smell and the refactoring actions $A_{s,N}$
that when performed on one or more elements of $N$ removes the symptom. We
obtain a series of tuples $<s,N,a_{s,N}>$ for each pair of subsequent versions
which describes the fine-grained refactoring actions that were performed on
one or more nodes to fix a specific SUIT smell symptom, _i.e._ the refactoring
actions.

To generate this list, first we compile the set of potential actions $A_{s,N}$
that can fix a symptom of a SUIT smell $s$ given a set of nodes $N$ (the
complete list is presented in Section 4.2). Then, for each tuple $<N,a_{N}>$
provided by the fine-grained change extraction tool and the knowledge of which
nodes present a symptom in the previous version, we check if $a_{N}\in
A_{s,N}$. If it is the case, then the fine-grained change $<N,a_{N}>$ is
considered to ba a refactoring action and added to the list of tuples
$<s,N,a_{s,N}>$.

To assess to which extent practitioners refactor SUIT smell symptoms, we count
the number of refactoring actions across all the versions,
$\mathinner{\\!\left\lvert a_{s,N}\right\rvert}$. Thus, the higher this
number, the more time developers spend in removing the symptoms. However,
because some symptoms might appear much more often than others, this number
might be skewed towards symptoms that are frequent. To alleviate this
limitation, we also present the number of refactoring actions over the total
number of symptoms that could have been refactored by this action:
$\mathinner{\\!\left\lvert a_{s,N}\right\rvert}/\mathinner{\\!\left\lvert
n_{s}\right\rvert}$ where $\mathinner{\\!\left\lvert n_{s}\right\rvert}$ is
the number of nodes $n$ exhibiting a symptom $s$ across all versions. Thus, if
a symptom is not removed during a multitude of commits, the metric decreases
with each version where the symptom is not addressed increasing the value on
the denominator ($\mathinner{\\!\left\lvert n_{s}\right\rvert}$) and the
numerator ($\mathinner{\\!\left\lvert a_{s,N}\right\rvert}$) remaining
unchanged. The results of this analysis are presented in Section 4.4.

Name |  Notation |  Description
---|---|---
# Symptoms |  $S_{s}$ |  The number of symptoms for a smell $s$ counted in a single test.
% Symptoms |  $D_{s}$ |  The number of symptoms for a smell $s$ counted in a single test divided by the number of location they could have appeared in that test.
# Refactoring |  $\mathinner{\\!\left\lvert R_{s}\right\rvert}$ |  The total number of action refactoring nodes exhibiting a symptom $R_{s}$ for a smell $s$ counted across all tests in all versions.
% Refactoring |  - |  The proportion of symptomatic tests undergoing at least one refactoring action through their lifespan.
Table 2: Definitions of the metrics used in the study.

##  4 Results

###  4.1 RQ1: SUIT Smells Identification

Figure 4a shows the number of test smells that were found in academic and grey
literature. We show both the 35 initial SUIT smells extracted from the
literature and the subset of 16 SUIT smells for which we could extract a
metric in the test code.

A SUIT smell is considered as covered if at least one of the smells grouped in
the generalization step (Section 3.1) is covered by the literature. If at
least one source is covering a smell in the academic literature, we consider
it as being covered by academia. Thus, smells labeled as originating from the
grey literature are never mentioned in academic literature. The figure shows
that while there exists an interest from practitioners with 35 unique smells
discussed, the number of smells discussed in peer reviewed literature remains
rather limited with only 10 smells identified. We also see that smells
identified from the literature could generate a metric in six out of ten
cases. The ones for which no metric could be derived are _Inconsistent
Wording_ (Hauptmann et al., 2013), _Unsuitable Naming_ (Chen and Wang, 2012),
_Inconsistent Hierarchy_ (Hauptmann et al., 2013) and _Data Creep_ (Alegroth
et al., 2016). While Hauptmann et al. (2013) propose metrics for the smells
they introduce, during our evaluation we observed a high rate of false
positive for _Inconsistent Wording_ and _Inconsistent Hierarchy_ thus we
excluded them for the final list. Chen and Wang (2012) and Alegroth et al.
(2016) on the other hand do not propose a metric to automatically measure the
smells they present.

To conclude this analysis of Figure 4a, we see that there exists a gap between
the grey literature and the academic literature. We believe that more work
need to be conducted to better understand and automatically detect and
refactor SUIT smells in the academia. The present work is an attempt to fill
this gap with the introduction of 15 new smells not previously studied in
academia.

Figure 4b presents the effects associated to each SUIT smell. Note that a
smell can lead to issues from different categories. Thus, the total number of
issues is greater than the number of smells. From the figure, we observe that
although readability issues appear the most often with 18 instances,
maintenance issues with 12 instances and execution issues with 14 instances do
not fall far behind. Furthermore, to our surprise, SUIT smells affecting
readability where the ones for which the most metrics could be computed. One
reason explaining this phenomenon is that in the case of execution issues,
information about the SUT and its execution are required. The same observation
can be made for maintenance where the danger is coming from a divergence of
the test from the SUT as it is the case for instance with the _Lifeless_ smell
(Buwalda, 2015; Renaudin, 2016; Buwalda, 2019) where the test is not following
the same lifecycle as the application. Thus, in both cases, the extraction of
the symptoms requires information about the structure and lifecycle of the
SUT.

(a) Source

(b) Issues

Figure 4: Properties of the identified SUIT smells. Figure 4a displays the
number of smells found in the academic and in the grey literatures. Figure 4b
displays the number of smells exhibiting a specific issue. Note that a smell
can lead to several issues, hence the total is greater than the total number
of SUIT smells.

There is a gap between the academic and grey literatures in terms of SUIT
smells, with 10 and 35 covered smells respectively. SUIT smells are reported
to affect readability, maintenance, and test execution.

###  4.2 SUIT Smells Catalog

The process described in Section 3.1 leads to a final list of 35 SUIT smells
presented in Table 3. Out of this list, we find that 16 smells can be
automatically detected using metrics of the Robot Framework. Hence, we focus
on these smells in our following questions.

Name |  Sources
---|---
Army of Clones |  (Chen and Wang, 2012; Hauptmann et al., 2013, 2015; Knight, 2019)
Complicated Setup Scenarios |  (Scott, 2015)
Conditional Assertions |  (Gawinecki, 2016)
Conspiracy Of Silence |  (Gawinecki, 2016; Sheth, 2020)
Data Creep |  (Alegroth et al., 2016; Siminiuc, 2019; Shay, 2019)
Dependencies between tests |  (Klarck, 2014; Advolodkin, 2018; Cripsin, 2018; Bushnev, 2019; Goldberg, 2019)
Directly Executing UI Scripts |  (Scott, 2015)
Duplicate Check |  (Buwalda, 2019)
Eager Test |  (England, 2016; Renaudin, 2016; Cripsin, 2018; Sciamanna, 2019; Temov, 2020)
Hardcoded Environment |  (Gawinecki, 2016; Sheth, 2020)
Hiding Test Data |  (Jain, 2007)
Implementation Dependent |  (Jain, 2007; Kapelonis, 2018; Goldberg, 2019)
Inconsistent Hierarchy |  (Clayton, 2014; Gawinecki, 2016; Buwalda, 2019)
Inconsistent Wording |  (Hauptmann et al., 2013)
Lack of Encapsulation |  (Chen and Wang, 2012; Evangelisti, 2012; Klarck, 2014; Buwalda, 2015; England, 2016; Renaudin, 2016; Knight, 2017a; Goldberg, 2019; Knight, 2019; Shay, 2019)
Lack Of Early Feedback |  (Dharmender, 2017)
Lifeless |  (Buwalda, 2015; Renaudin, 2016; Buwalda, 2019)
Long Test Steps |  (Chen and Wang, 2012; Hauptmann et al., 2013; Buwalda, 2019)
Middle Man |  (Chen and Wang, 2012)
Missing Assertion |  (Klarck, 2014)
Narcissistic |  (England, 2016; Knight, 2017b).
Noisy Logging |  (Jain, 2007).
Obscure Test |  (Hauptmann et al., 2013; Gawinecki, 2016; Siminiuc, 2019).
On the Fly |  (Archer, 2010)
Over-Checking |  (Buwalda, 2015; Renaudin, 2016)
Pointless Scenario Descriptions |  (England, 2016)
Sensitive Locators |  (Scott, 2015; Knight, 2019; Battat, 2020; Sheth, 2020)
Sneaky Checking |  (Kirkbride, 2014; Buwalda, 2015; Renaudin, 2016)
Stinky Synchronization |  (Gawinecki, 2016; Renaudin, 2016; Bushnev, 2019; Knight, 2019; Sheth, 2020)
Test Data Loss |  (Siminiuc, 2019)
Testing Data Not Code |  (Dharmender, 2017)
Unnecessary Navigation |  (Archer, 2010)
Unrealistic Data |  (Goldberg, 2019)
Unsecured Test Data |  (Morlion, 2019)
Unsuitable Naming |  (Chen and Wang, 2012; Goldberg, 2019; Shay, 2019; Sheth, 2020)

Table 3: Catalog of SUIT Smells and their origin. Smell in bold are associated
with a metric and a complete description is presented in Section 4.2

.

Building on the tree model introduced in Section 2.2 we represent each test
$t$ as a rooted, ordered, directed, acyclic graph. The nodes, $N_{t}$, of the
tree represent the set of calls to _Keywords_ , $C_{t}$ and the set of of
arguments $A$ passed to the calls. A call, $c_{t}$ can be associated to a type
namely: USER, INTERACTION, ASSERTION, CONTROLFLOW, GETTER, LOGGING and SYNC.
Thus, the set of calls in a test performing an assertion is noted
$C_{t,assertion}$. Finally, the definition of the _User Kyeword_ called by a
test $t$ is noted $K_{t}$. Following this notation, we formally describe each
smell for which a metric can be extracted with respect to its symptoms, its
impact, and the metrics chosen to measure the prevalence of the symptoms
following the protocol presented in Section 3.3. Finally, we present the
refactoring actions removing the symptoms from the test codebase.

####  4.2.1 Army of Clones (AoC)

##### Description:

Different tests perform and implement similar actions, leading to duplicated
pieces of test code.

##### Impact on Readability:

Test sequences which are similar but not identical are not easy to
distinguish. It is not easy to grasp the intention of the test in comparison
with its clone.

##### Impact on Maintenance:

The effort to maintain duplicated parts of tests increases. Furthermore, it is
difficult to determine where maintenance has to be performed.

##### Detection Method:

Code duplication can be observed at different levels. Here, for the body of a
_User Keyword_ , we detect if there exists a clone of type 1 (code duplication
at the token level) or type 2 (code duplication at syntax level allowing for
minor syntactic changes such as variables name) in the test suite. Thus, we
express the count metric, $S_{AoC}(t)$, as the number of calls to _User
Keywords_ that have a clone and the density metric, $D_{AoC}(t)$, as the
number of calls to _User Keywords_ that have a clone over the total number of
unique _User Keywords_ called by the test. More formally:

$S_{AoC}(t)=\mathinner{\\!\left\lvert K_{t}\cap K_{clone}\right\rvert}$

$D_{AoC}(t)=\frac{\mathinner{\\!\left\lvert K_{t}\cap
K_{clone}\right\rvert}}{\mathinner{\\!\left\lvert K_{t}\right\rvert}}$

where $K_{t}$ is the set of unique _User Keywords_ called by test $t$ and
$K_{clone}$ is the set of unique _User Keywords_ that have at least one clone
in the test suite.

##### Refactoring Actions:

The symptom is considered refactored if the _User Keyword_ that is called by a
test and have at least one clone in the test suite, $k_{t,clone}$, is removed.
Thus, we propose the refactoring pattern, $R_{AoC}(k)$, as follow:

$R_{AoC}(k)=k_{t,clone}\xrightarrow{action}\emptyset$

####  4.2.2 Conditional Assertions (CA)

##### Description:

The test verifies different properties depending on the environment when the
environment state may change from one execution to the next.

##### Impact on Readability:

With more complex logic in the assertions, it becomes harder to capture their
meaning.

##### Impact on Execution:

More complex code might introduce bugs in the test code.

##### Detection Method:

We consider assertions nodes, $C_{assertion}$, to be symptomatic if they have
a parent node which is a conditional node and have no sibling nodes in the
call graph, $C_{condition}$. Thus, we express the count metric, $S_{CA}(t)$,
as the number of conditional assertion calls and the density metric,
$D_{CA}(t)$, as the number of conditional assertion calls over the total
number of assertion calls.

$S_{CA}(t)=\mathinner{\\!\left\lvert C_{t}\cap C_{assertion}\cap
C_{condition}\right\rvert}$

$D_{CA}(t)=\frac{\mathinner{\\!\left\lvert C_{t}\cap C_{assertion}\cap
C_{condition}\right\rvert}}{\mathinner{\\!\left\lvert C_{t}\cap
C_{assertion}\right\rvert}}$

where $\mathinner{\\!\left\lvert C_{t}\cap C_{assertion}\right\rvert}$ is the
size of the set of calls to _Library Keyword_ annotated as “assertion” for a
test $t$ and $\mathinner{\\!\left\lvert C_{t}\cap C_{assertion}\cap
C_{condition}\right\rvert}$ is the size of the set of calls to _Library
Keyword_ annotated as “assertion” for which the caller is a conditional node
that has only one child (logging nodes excluded).

##### Refactoring Action:

The symptom is considered refactored if the conditional assertion node is
removed from the call graph. Thus, we accept the following refactoring
pattern, $R_{CA}(c)$, as removing a symptom in a node $c$:

$R_{CA}(c)=c_{t,assertion,condition}\xrightarrow{action}c_{t,assertion}$

The assertion $c_{t,assertion}$ replaces its former parent node
$c_{t,assertion,condition}$. Note that removing a parent of
$c_{t,assertion,condition}$ or adding a sibling to the child assertion node
are not considered as fixing the symptom.

####  4.2.3 Hardcoded Environment (HE)

##### Description:

The test contains hardcoded references to the environment when the same
requirement must be run against different test environments instead of having
an environment-agnostic test.

##### Impact on Maintenance:

Updating the configuration requires modifying multiple locations in different
tests.

##### Detection Method:

The metric we propose covers the case of multi-browser configuration. Here,
when a browser is loaded, the metric ensure that the web-driver is not
instantiated with a hardcoded configuration for the browser. Thus, we express
the count metric, $S_{HE}(t$, as the number of configuration arguments that
are hardcoded and the density metric, $N_{HE}(t)$, as he number of
configuration arguments that are hardcoded over the total number of
configuration arguments. More formally:

$S_{HE}(t)=\mathinner{\\!\left\lvert A_{t}\cap A_{config}\cap
A_{hardcoded}\right\rvert}$

$D_{HE}(t)=\frac{\mathinner{\\!\left\lvert A_{t}\cap A_{config}\cap
A_{hardcoded}\right\rvert}}{\mathinner{\\!\left\lvert A_{t}\cap
A_{config}\right\rvert}}$

where $\mathinner{\\!\left\lvert A_{t}\cap A_{config}\right\rvert}$ is the
size of the set of arguments in calls to _Library Keywords_ annotated as
“configuration” in a test $t$ and $\mathinner{\\!\left\lvert A_{t}\cap
A_{config}\cap A_{hardcoded}\right\rvert}$ is the size of the set of arguments
in calls to _Library Keywords_ annotated as “configuration” for which the
value is hardcoded.

##### Refactoring Action:

The symptom is considered refactored if the hardcoded argument in a call to a
_Library Keyword_ annotated as “configuration”, $a_{t,config,hardcoded}$, is
replaced with a variable, $a_{t,config,variable}$. Thus, we propose the
following refactoring pattern, $R_{HE}(a)$, for an argument $a$:

$R_{HE}(a)=a_{t,config,hardcoded}\xrightarrow{action}a_{t,config,variable}$

####  4.2.4 Hidden Test Data (HTD)

##### Description:

The data are not directly visible and understandable in the test but are
hidden in the fixture code.

##### Impact on Readability:

The data is completely obscure to the future reader making the intent of the
test difficult to understand.

##### Detection Method:

In this work, we associate the fixture code to the setup of a test. We
consider data access as reading input from external resources through a
_Library Keyword_ annotated as “getter”. Thus, we express the count metric,
$S_{HTD}(t)$, as the number of calls to getter in the setup of a test and
$D_{HTD}(t)$ as the number of calls to getter in the setup of a test over the
total number of calls in the setup of that test. More formally:

$S_{HTD}(t)=\mathinner{\\!\left\lvert C_{t}\cap C_{setup}\cap
C_{getter}\right\rvert}$

$D_{HTD}(t)=\frac{\mathinner{\\!\left\lvert C_{t}\cap C_{setup}\cap
C_{getter}\right\rvert}}{\mathinner{\\!\left\lvert C_{t}\cap
C_{setup}\right\rvert}}$

where $\mathinner{\\!\left\lvert C_{t}\cap C_{setup}\right\rvert}$ is the size
of the set of calls to _Library Keywords_ in the setup of test $t$ and
$\mathinner{\\!\left\lvert C_{t}\cap C_{setup}\cap C_{getter}\right\rvert}$ is
the size of the set of calls to _Library Keywords_ annotated as “getter” in
the setup of test $t$.

##### Refactoring Action:

The symptom is considered refactored if the call to the _Library Keywords_
annotated as “getter” in the setup of test $t$, $c_{t,setup,getter}$, is
removed. Thus, we propose the following refactoring pattern, $R_{HTD}(c)$
performed on a _Library Keyword_ call $c$ as follow:

$R_{HTD}(c)=c_{t,setup,getter}\xrightarrow{action}\emptyset$

Note that removing a parent node of $c_{t,setup,getter}$ is not considered as
a fix.

####  4.2.5 Lack of Encapsulation (LoE)

##### Description:

The implementation details of a test are not properly hidden in the
implementation layer and start appearing in its acceptance criteria.

##### Impact on Readability:

The acceptance criteria is meant to convey intention over implementation.
Focusing on implementation in the acceptance criteria makes the intent harder
to grasp.

##### Detection Method:

Typically the acceptance criteria makes call to the implementation layer which
subsequently relies on the application driver layer. The metric detects the
direct calls from the acceptance criteria (test steps) to _Library Keywords_.
Thus, we express the count metric, $S_{LoE}(t)$, as the number of direct calls
to a driver from the acceptance criteria of a test and the density metric,
$D_{LoE}(t)$, as the number the number of direct calls to a driver from the
acceptance criteria of a test over the total number of steps of the acceptance
criteria of a test. More formally:

$S_{LoE}(t)=\mathinner{\\!\left\lvert C_{t}\cap C_{step}\cap
C_{driver}\right\rvert}$

$D_{LoE}(t)=\frac{\mathinner{\\!\left\lvert C_{t}\cap C_{step}\cap
C_{driver}\right\rvert}}{\mathinner{\\!\left\lvert C_{t}\cap
C_{step}\right\rvert}}$

where $\mathinner{\\!\left\lvert C_{t}\cap C_{step}\right\rvert}$ is the size
of the set of steps in the acceptance criteria of the test $t$ and
$\mathinner{\\!\left\lvert C_{t}\cap C_{step}\cap C_{driver}\right\rvert}$ is
the size of the set of steps in the acceptance criteria of the test $t$
directly calling the application driver, _i.e._ a (_Library Keyword_).

##### Refactoring Action:

The symptom is considered refactored if the direct call to a _Library Keyword_
is removed from the acceptance criteria. Thus, we propose the refactoring
patterns, $R_{LoE,1}(c)$ and $R_{LoE,2}(c)$, performed on a _Library Keyword_
call $c$ as follow:

$R_{LoE,1}(c)=c_{t,step,driver}\xrightarrow{action_{1}}c_{t,step,\neg driver}$

$R_{LoE,2}(c)=c_{t,step,driver}\xrightarrow{action_{2}}\emptyset$

In the first equation, $R_{LoE,1}(c)$, the direct call to a _Library Keyword_
in the acceptance criteria is replaced by a call to a _User Keyword_ where as
in the second equation, $R_{LoE,2}(c)$, the call is removed.

####  4.2.6 Long Test Steps (LTS)

##### Description:

One or many test steps are very long, performing a lot of actions on the
system under test.

##### Impact on Readability:

The intention of the step is difficult to grasp.

##### Impact on Execution:

With each action on the system under test, there is a chance of something
going wrong. The higher this number, the more fragile the test becomes.

##### Detection Method:

For a test step, $K_{t,step}$, of a test $t$, the metric counts the number of
_Library Keyword_ annotated as “action” (triggering an event on the SUT)
called directly or indirectly by $K_{t,step}$. If the value is greater than a
threshold $L$, then $K_{t,step}$ is considered symptomatic. Thus, we express
the count metric, $S_{LTS}(t)$, as the number of steps that are performing a
number of actions greater than a threshold and the density metric,
$D_{LTS}(t)$, s the number of steps that are performing a number of actions
greater than a threshold over the total number of steps. More formally:

$S_{LTS}(t)=\mathinner{\\!\left\lvert C_{t}\cap C_{step\geq L}\right\rvert}$

$D_{LTS}(t)=\frac{\mathinner{\\!\left\lvert C_{t}\cap C_{step\geq
L}\right\rvert}}{\mathinner{\\!\left\lvert C_{t}\cap C_{step}\right\rvert}}$

where $\mathinner{\\!\left\lvert C_{t}\cap C_{step}\right\rvert}$ is the
number of steps in a test $t$ and $\mathinner{\\!\left\lvert C_{t}\cap
C_{step\geq L}\right\rvert}$ is the number of steps calling more than $L$
_Library Keyword_ annotated as “action”.

Because the parameter $L$ needs to be set empirically, we compute a deviation
threshold based on the distribution of our dataset. Using the analysis of the
evolution of the quantiles, we compute at which point the values start to
rapidly deviates by computing the knee curve of the quantiles distribution
function proposed by Satopaa et al. (2011). Following this approach we find
the knee point at 13 actions on the SUT for a step (quantile = 0.986, see
Figure 5). Therefore, we set $L=13$ and consider any step performing a
sequence of actions on the SUT greater than 13 to be too long.

Figure 5: The blue curve represent the evolution of the length of the
sequences of the step as the quantiles increase. The intersection of the black
doted lines displays the knee point (0.986, 13) at which the values sequence
lengths values start to increase dramatically.

##### Refactoring Action:

The symptom is considered refactored if the number of actions performed on the
SUT by a long step sees its value pass under the threshold $L$. We do not
specify how the calls on the SUT have to be transformed, as long as the test
step call is left unchanged. Thus, we propose the refactoring pattern,
$R_{LTS}(c)$, where $c$ is a long test step:

$R_{LTS}(c)=c_{t,step\geq L}\xrightarrow{action}c_{t,step<L}$

where $c_{t,step\geq L}$ is a step yielding at least $L$ actions on the SUT
while $c_{t,step<L}$ is a step yielding less than $L$ actions on the SUT.

####  4.2.7 Middle Man (MM)

##### Description:

A test component (keyword, macro, function) delegates all its tasks to another
test component.

##### Impact on Readability:

The levels of indirection make the test harder to follow by future readers.

##### Detection Method:

The principle of delegating is doing nothing and simply calling another
function. Thus, we express the count metric, $S_{MM}(t)$, as the number of
_User Keywords_ called from a test that are composed of a single call to
another _Keyword_ and the density metric, $D_{MM}(t)$, as the he number of
_User Keywords_ called from a test that are composed of a single call to
another _User Keyword_ and the density metric over the total number of
_Keyword_ called by the test. More formally:

$S_{MM}(t)=\mathinner{\\!\left\lvert K_{t}\cap K_{delegate}\right\rvert}$

$D_{MM}(t)=\frac{\mathinner{\\!\left\lvert K_{t}\cap
K_{delegate}\right\rvert}}{\mathinner{\\!\left\lvert K_{t}\right\rvert}}$

where $\mathinner{\\!\left\lvert K_{t}\right\rvert}$ is the number of _User
Keywords_ in a test $t$ and $\mathinner{\\!\left\lvert K_{t}\cap
K_{delegate}\right\rvert}$ is the number of _User Keywords_ performing a
single call to one other _User Keyword_ (we ignore logging action) without
performing any subsequent action on their own, _i.e._ _Delegate Keyword_.

##### Refactoring Actions:

The symptom is considered refactored if the _Delegate Keyword_ call is
replaced with another call which is not simply delegating its actions. Thus,
we propose the refactoring pattern, $R_{MM}(c)$ where $c$ is a call to a
_Delegate Keyword_ as:

$R_{MM}(c)=c_{K_{t,delegate}}\xrightarrow{action}c_{K_{t,\neg delegate}}$

where $c_{K_{t,delegate}}$ is a call to a _Delegate Keyword_ and $c_{K_{t,\neg
delegate}}$ is a call to any _Keyword_ not performing delegation.

####  4.2.8 Missing Assertion (MA)

##### Description:

The test lacks any explicit assertions.

##### Impact on Readability:

Future readers are left in the potentially frustrating position of puzzling
over the intention of the test.

##### Detection Method:

The metric detects the absence of call to _Library Keyword_ annotated as
“assertion” within the test. Because the symptom can at most appear once in
the test, both the count metric, $S_{MA}(t)$ and the density metric
$D_{MA}(t)$ are express as the absence of an assertion call in the test. More
formally:

$S_{MA}(t)=D_{MA}(t)=\begin{cases}1,&\text{if }C_{t}\cap
C_{assert}=\emptyset\\\ 0,&\text{otherwise}\end{cases}$

##### Refactoring Actions:

The symptom is considered refactored if _Library Keyword_ annotated as
“assertion” is introduced in test that is missing any assertion. Thus, we
propose the refactoring pattern, $R_{MA}(t)$, where $t$ is the test lacking
any assertion as:

$R_{MA}(t)=\emptyset\xrightarrow{action}c_{t,assertion}$

where $c_{t,assertion}$ is a call to a _Library Keyword_ annotated as
“assertion” in a test $t$.

####  4.2.9 Narcissistic (N)

##### Description:

The test uses the first person pronoun “I” to refer to its actors and does not
uniquely qualify those actors.

##### Impact on Readability:

The test is harder to read because it is not clear who “I” is and what are the
roles that “I” has.

##### Detection Method:

Using text tagging, we identify calls from the acceptance criteria, _i.e._
“test steps”, using a personal pronouns as the subject in there name as
symptomatic. Furthermore, the implementation used in our experiments supports
the different languages present in our dataset (namely, French and English)..
Thus, we express the count metric, $S_{N}(t)$, as the number of steps in the
acceptance criteria of a test that are using a personal pronoun and the
density metric, $D_{N}(t)$, as the number of steps in the acceptance criteria
of a test that are using a personal pronoun over the total number of steps in
the acceptance criteria of the test:

$S_{N}(t)=\mathinner{\\!\left\lvert C_{t}\cap C_{step}\cap C_{I}\right\rvert}$

$D_{N}(t)=\frac{\mathinner{\\!\left\lvert C_{t}\cap C_{step}\cap
C_{I}\right\rvert}}{\mathinner{\\!\left\lvert C_{t}\cap
C_{step}\right\rvert}}$

where $\mathinner{\\!\left\lvert C_{t}\cap C_{step}\right\rvert}$ is the
number of “test steps” for a test $t$ and $\mathinner{\\!\left\lvert C_{t}\cap
C_{step}\cap C_{I}\right\rvert}$ is the number of “test steps” using a
personal pronouns as the subject in there name.

##### Refactoring Actions:

The symptom is considered refactored if the name of a symptomatic “test steps”
is changed so that it does not contain a personal pronoun anymore. Thus, we
propose the refactoring pattern, $R_{N}(c)$, where $c$ is a step of the
acceptance criteria of a test $t$:

$R_{N}(c)=c_{t,step,I}\xrightarrow{action}c_{t,step,\neg I}$

where $c_{t,step,I}$ is a _User Keyword_ called by a “test steps” using the
personal pronoun “I” as the subject and $c_{t,step,\neg I}$ is the _User
Keyword_ with its new name not using the personal pronoun “I” as the subject.
Therefore, a fix is detected only when the name of a _User Keyword_ called by
a “test steps” is changed.

####  4.2.10 Noisy Logging (NL)

##### Description:

The test logs the state of the fixtures.

##### Impact on Execution:

There is too much noise in the output from the tests, making its analysis more
cumbersome.

##### Detection Method:

In this work, we associate the fixture code to the setup of a test. Thus, we
express the count metric, $S_{NL}(t)$, as the number of calls to _Library
Keywords_ annotated as “logging” from the setup of a test and the density
metric, $D_{NL}(t)$, as the number of calls to _Library Keywords_ annotated as
“logging” from the setup of a test over the total number of calls from the
setup of the test. More formally:

$S_{NL}(t)=\mathinner{\\!\left\lvert C_{t}\cap C_{setup}\cap
C_{logging}\right\rvert}$

$D_{NL}(t)=\frac{\mathinner{\\!\left\lvert C_{t}\cap C_{setup}\cap
C_{logging}\right\rvert}}{C_{t}\cap C_{setup}}$

where $\mathinner{\\!\left\lvert C_{t}\cap C_{setup}\right\rvert}$ is the size
of the set of _Library Keyword_ called from the setup of a test $t$ and
$\mathinner{\\!\left\lvert C_{t}\cap C_{setup}\cap C_{logging}\right\rvert}$
is the size of the set of _Library Keyword_ annotated as “logging” called from
the setup of test $t$.

##### Refactoring Action:

The symptom is considered refactored if the call to the _Library Keyword_
annotated as “logging” called from the setup of test $t$, $c_{t,setup,log}$,
is removed. Thus we propose the refactoring pattern, $R_{NL}(c)$, performed on
a _Keyword_ call $c$ as follow:

$R_{NL}(c)=c_{t,setup,log}\xrightarrow{action}\emptyset$

Note that removing a parent of $c_{t,setup,log}$ is not considered as fixing
the smell.

####  4.2.11 Obscure Test (OT)

##### Description:

The test behavior is difficult to understand because the test does not clearly
state what it is verifying. Typical symptoms are hardcoded values, high
cyclomatic complexity and/or function or procedure calls with high number of
parameters.

##### Impact on Readability:

Future reader might not understand the meaning of a hardcoded value, hence,
missing the intention of the test. As with a high cyclomatic complexity it
becomes hard for the future reader to follow the execution flow of the test
and grasp what it is doing.

##### Impact on Maintenance:

It is difficult to determine where to perform changes if hardcoded values are
scattered all over the test code. Furthermore, test with high cyclomatic
complexity might have side effect overseen during maintenance which might lead
to future problems.

##### Detection Method:

In this work, we focus on one of the expression of an obscure test: the
overuse of hardcoded values. The code starts to smell when hardcoded values
are used directly in calls to both _User Keyword_ and _Library Keywords_ ,
instead of relying on variables. Thus, we express the count metric,
$S_{OT}(t)$, as the number of hardcoded arguments present in a test and the
density metric $D_{OT}(t)$ as the number of hardcoded arguments present in the
test over the total number of arguments from that test. More formally:

$S_{OT}(t)=\mathinner{\\!\left\lvert A_{t}\cap A_{hardcoded}\right\rvert}$

$D_{OT}(t)=\frac{\mathinner{\\!\left\lvert A_{t}\cap
A_{hardcoded}\right\rvert}}{\mathinner{\\!\left\lvert A_{t}\right\rvert}}$

where $\mathinner{\\!\left\lvert A_{t}\right\rvert}$ is the size of the set of
arguments passed to _Keyword_ calls in a test $t$ and
$\mathinner{\\!\left\lvert A_{t}\cap A_{hardcoded}\right\rvert}$ is the size
of the set of arguments passed to _Keyword_ calls which are hardcoded.

##### Refactoring Actions:

Focusing on hardcoded values, the symptom is considered refactored if an
argument passed to _Keyword_ call as a hardcoded value is replaced by a
variable. Thus, we propose the refactoring pattern, $R_{OT}(a)$, where $a$ is
a _Keyword_ call arguments as:

$R_{OT}(v)=a_{t,hardcoded}\xrightarrow{action}a_{t,variable}$

where $a_{t,hardcoded}$ is a hardcoded _Keyword_ call argument and
$a_{t,variable}$ is a variable _Keyword_ call argument.

####  4.2.12 On the Fly (OtF)

##### Description:

The test calculates an expected results during its execution instead of
relying on pre-computed values.

##### Impact on Readability:

By embedding the business rule in the assertion, the code for the automated
test can become as complicated as the system under test.

##### Detection Method:

The expected value should be a constant or a reference to a constant and not
computed during the test. Thus, we express the count metric, $S_{OtF}(t)$, as
the number of expected values that are computed in the test and the density
metric, $D_{OtF}(t)$, as the number of expected arguments from assertion calls
that are computed in the test over the total number of expected arguments from
assertion calls:

$S_{OtF}(t)=\mathinner{\\!\left\lvert A_{t}\cap A_{expected}\cap
A_{computed}\right\rvert}$

$D_{OtF}(t)=\frac{\mathinner{\\!\left\lvert A_{t}\cap A_{expected}\cap
A_{computed}\right\rvert}}{\mathinner{\\!\left\lvert A_{t}\cap
A_{expected}\right\rvert}}$

where $\mathinner{\\!\left\lvert A_{t}\cap A_{expected}\right\rvert}$ is the
number of “expected” arguments in calls to _Library Keywords_ annotated as
“assertion” for a test $t$ and $\mathinner{\\!\left\lvert A_{t}\cap
A_{expected}\cap A_{computed}\right\rvert}$ is the number of “expected”
arguments in calls to _Library Keywords_ annotated as “assertion” resolved
during the execution of the test $t$. Note that the identification of the
“expected” argument is based on the definition of the _Library Keyword_. When
a _Library Keyword_ annotated as “assertion” contains a field called
“expected”, it is considered by the engine as the placeholder for the expected
value. For instance, the library keyword _Should be equal_ from the Builtin
library takes six arguments: _value_ , _expected_ , _message_ , _values_ ,
_ignore_case_ and _formatter_. Hence, in this case the expected argument is
the second one.

##### Refactoring Action:

The symptom is considered refactored when the assertion is preserved but the
expected value is not computed on the fly. This leads to the following
equation for a refactoring action, $R_{OtF}$, addressing a symptom in an
argument $a$:

$R_{OtF}(a)=a_{t,expected,computed}\xrightarrow{action}a_{t,expected,\neg
computed}$

where $a_{t,expected,computed}$ is an expected value that is computed on the
fly and $a_{t,expected,\neg computed}$ is an expected value that is not
computed on the fly, being either hardcoded or provided through a variable
pointing to a static value. Thus removing the assertion would not be
considered as removing the symptom since $a_{t,expected,\neg computed}$ would
not be present.

####  4.2.13 Over-Checking (OC)

##### Description:

The test performs some assertions that are not relevant for its scope.

##### Impact on Readability:

It becomes harder to understand what is the main intent of the test. Many
assertions suggests the test is checking many different properties.

##### Impact on Maintenance:

The test may be too sensitive to the evolution of the SUT, verifying
implementation properties instead of behavioral ones.

##### Detection Method:

As the ratio assertions to actions on SUT increases, the chances that all the
assertions are relevant decreases. Thus we express the count metric,
$S_{OC}(t)$, as the number of assertions in a test and the density metric,
$N_{OC}(t)$, as the number of assertions in a test over the total number of
calls in the test. More formally:

$S_{OC}(t)=\mathinner{\\!\left\lvert C_{t}\cap C_{assertion}\right\rvert}$

$N_{OC}(t)=\frac{\mathinner{\\!\left\lvert C_{t}\cap
C_{assertion}\right\rvert}}{\mathinner{\\!\left\lvert C_{t}\right\rvert}}$

where $\mathinner{\\!\left\lvert C_{t}\right\rvert}$ is the size of the set
_Library Keywords_ calls and $\mathinner{\\!\left\lvert C_{t}\cap
C_{assertion}\right\rvert}$ is the size of the set of calls to _Library
Keywords_ annotated as “assertion”.

##### Refactoring Actions::

The symptom is considered refactored if the call to a _Library Keyword_
annotated with “assertion”, $c_{t,assertion}$ , is removed from a test $t$.
Thus, we propose the refactoring pattern, $R_{OC}(c)$ where $c$ is a call to
an _Library Keyword_ annotated with “assertion” as:

$R_{OC}(c)=c_{t,assertion}\xrightarrow{action}\emptyset$

####  4.2.14 Sensitive Locators (SL)

##### Description:

The test uses element identification selectors that have long chains to
identify an element in the user interface. e.g. complex x-pass or CSS selector
for web application.

##### Impact on Maintenance:

This leads to fragile tests, as any change in that chain from the user
interface representation will break the tests.

##### Detection Method:

The complexity of a locator can be expressed by how deep the locator needs to
go in the hierarchy of the UI, be it an x-pass, a CSS selector or any UI
representation based on a hierarchy. A locator, $A_{locator}$, can be
expressed as the number of GUI element, $\mathinner{\\!\left\lvert
E\right\rvert}$, that have to be traversed to uniquely locate the target GUI
element. Thus, we express the count metric, $S_{SL}(t)$, as the number of
locator arguments that visit more than one GUI elements and the density
metric, $D_{SL}(t)$, the number of of locator arguments that visit more than
one GUI elements in a test over the total number of locator arguments present
in the test. More formally:

$S_{SL}(t)=\mathinner{\\!\left\lvert A_{t}\cap A_{locator}\cap
A_{\mathinner{\\!\left\lvert E\right\rvert}>1}\right\rvert}$

$D_{SL}(t)=\frac{\mathinner{\\!\left\lvert A_{t}\cap A_{locator}\cap
A_{\mathinner{\\!\left\lvert
E\right\rvert}>1}\right\rvert}}{\mathinner{\\!\left\lvert A_{t}\cap
A_{locator}\right\rvert}}$

where $\mathinner{\\!\left\lvert A_{t}\cap A_{locator}\cap
A_{\mathinner{\\!\left\lvert E\right\rvert}>1}\right\rvert}$ is the number of
locators that require to visit more than one element $E$ of the GUI to be
uniquely identified (_e.g._ the XPath “/html/body/div[4]/button” visits 4
elements to quality the button where the XPath “//button[@id ="unique-id"]”
only needs to visit one element). Note that Robot Framework using dynamic
types, only _Library Keyword_ calls explicitly specify a type for their
parameters. Therefore, for each _Library Keyword_ call requiring a locator as
an argument, the engine resolve all the values possible for the argument
within the test to populate the set $A_{locator}$.

##### Refactoring Actions:

The symptom is considered refactored if the value of a node $l$ flagged as
complex locator sees its length go down to one. Thus, we propose the
refactoring pattern, $R_{SL}(l)$, as follow:

$R_{SL}(c)=l_{\mathinner{\\!\left\lvert
E\right\rvert}>1}\xrightarrow{action}l_{\mathinner{\\!\left\lvert
E\right\rvert}=1}$

where $l_{\mathinner{\\!\left\lvert E\right\rvert}>1}$ is a node defining the
value of a sensitive locator and $l_{\mathinner{\\!\left\lvert
E\right\rvert}=1}$ is the same node but with a simple locator expression. Note
that a change is only accounted for when the value of the locator is modified.

####  4.2.15 Sneaky Checking (SC)

##### Description:

The test hides its assertions in actions that are at the wrong level of
details.

##### Impact on Readability:

The future reader is not able to understand what is being tested by just
looking at the main steps of the acceptance criteria without a need to inspect
how low level details are implemented.

##### Detection Method:

A _User Keyword_ only calling an _Library Keyword_ annotated as “assertion”
can be seen as hiding the assertion to the callers of that _User Keyword_.
Thus, we express the count metric, $S_{SC}(t)$, as the number of unique _User
Keywords_ called by a test and only calling an assertion and the density
metric, $D_{SC}(t)$, as the number of unique _User Keywords_ called by the
test and only calling an assertion over the number of unique _User Keywords_
called by that test. More formally:

$S_{SC}(t)=\mathinner{\\!\left\lvert K_{t}\cap K_{assert}\right\rvert}$

$D_{SC}(t)=\frac{\mathinner{\\!\left\lvert K_{t}\cap
K_{assert}\right\rvert}}{\mathinner{\\!\left\lvert K_{t}\right\rvert}}$

where $\mathinner{\\!\left\lvert K_{t}\right\rvert}$ is the total number of
unique _User Keywords_ and $\mathinner{\\!\left\lvert K_{t}\cap
K_{assert}\right\rvert}$ is the number of _User Keywords_ only calling an
_Library Keyword_ annotated as “assertion” (logging actions are ignored).

##### Refactoring Actions:

The symptom is considered refactored if a _User Keywords_ only calling a
_Library Keyword_ annotated as “assertion”, $k_{t,assert}$, is removed from
the test $t$. Thus, we propose the refactoring pattern, $R_{SC}(k)$, where $k$
is a _User Keywords_ as:

$R_{SC}(k)=k_{t,assert}\xrightarrow{action}\emptyset$

####  4.2.16 Stinky Synchronization (SS)

##### Description:

The test fails to use proper synchronization points with the system under
test.

##### Impact on Execution:

The test becomes oversensitive to the response time, leading to flaky tests,
or very slow tests when choosing very conservative wait points.

##### Detection Method:

This symptom is associated with the use of explicit and fixed synchronization,
independent from the SUT such as a pausing the test for a specific amount of
time. Thus, we express the count metric, $S_{SS}(t)$, as the number of calls
to explicit pause from a test and the density metric, $D_{SS}(t)$, as the
number of calls to explicit pause from a test over the total number of
synchronization calls. More formally:

$S_{SS}(t)=\mathinner{\\!\left\lvert C_{t}\cap C_{sync}\cap
C_{sleep}\right\rvert}$

$D_{SS}(t)=\frac{\mathinner{\\!\left\lvert C_{t}\cap C_{sync}\cap
C_{sleep}\right\rvert}}{\mathinner{\\!\left\lvert C_{t}\cap
C_{sync}\right\rvert}}$

where $\mathinner{\\!\left\lvert C_{t}\cap C_{sync}\right\rvert}$ is the size
of the set of calls to _Library Keyword_ annotated as “synchronization” in a
test $t$ and $\mathinner{\\!\left\lvert C_{t}\cap C_{sync}\cap
C_{sleep}\right\rvert}$ is the size of the set of calls to _Library Keyword_
annotated as “synchronization” by pausing the test execution for a specified
amount of time. In the case of Robot Framework, it is instantiated by calls to
the _Library Keyword_ “Sleep”.

##### Refactoring Actions:

The symptom is considered refactored if a _Library Keyword_ call annotated as
“synchronization” which pausing the test execution of the test for a specified
amount of time, $c_{t,sync,sleep}$, is removed or replaced by another _Library
Keyword_ call annotated as “synchronization”, $c_{t,sync,\neg sleep}$. Thus,
we propose two refactoring patterns, $R_{SS,1}(c)$ and $R_{SS,2}(c)$, as
follow:

$R_{SS,1}(c)=c_{t,sync,sleep}\xrightarrow{action_{1}}\emptyset$

$R_{SS,2}(c)=c_{t,sync,sleep}\xrightarrow{action_{2}}c_{t,sync,\neg sleep}$

###  4.3 RQ2: SUIT Smells Distribution

Figure 6: Percentage of tests exhibiting at least one symptom in open-source
and industrial projects.

Building on the metrics derived in Section 4.2, we show in Figure 6 the number
of tests exhibiting at least one smell symptom. The smell _Hidden Test Data_
does not in any of the studied projects. Furthermore, the smells _Noisy
Logging_ , _Sensitive Locator_ , _Narcissistic_ and _HardCoded Environment_
manifest in less than 10% of the tests in both open-source and industrial test
suites.

On the other end of the spectrum, three smells appear in the majority of the
projects: the symptom _Hard Coded Values_ appears in more than 90% of the
tests, _Over Checking_ between 75% (industrial) and 79.5% (open-source), and
finally _Sneaky Checking_ appears in 70% of the tests.

Finally, an interesting observation concerns the presence of significant
difference between industrial and open-source projects. Notably, in industrial
projects three symptoms appear much more often than in open-source projects,
namely, _Middle Man_ (93.5%), _Army of Clones_ (91.2%) and _Long Test Steps_
(58.8%). The results for the _Army of Clones_ and _Middle Man_ can be
explained by the structure of the projects at BGL BNP Paribas. Symptoms for
the SUIT smell _Army of Clones_ manifest because many repositories are
interconnected, leading to code duplication across projects. Hence, testers
working on different applications or different parts of the system test the
same functionality over and over. As for the symptoms of _Middle Man_ , their
presence is explained by the use of _Keywords_ that act as translation layers.
While two actions can perform the same concrete actions on the SUT, in
different contexts they might operate on a different business logic. Thus,
developers created translation layers to unify the vocabulary used in the
acceptance layer of each test. These translation layers are implemented by the
use of a _Keyword_ only calling one other _Keyword_ with a different name.
Finally, _Long Test Step_ originates from the complex business logic that is
expressed by the step in a test at BGL BNP Paribas and is more related to the
SUT and the vocabulary employed by the business analysts than the test suite
itself.

Figure 7: Distribution of the number of symptoms in symptomatic tests for all
projects across all commits.

We continue our analysis with the evaluation of the number of symptoms
appearing per tests (Figure 7). Note that only tests presenting at least one
symptom are considered in our analysis, _i.e._ symptomatic tests. Looking at
_Hard Coded Values_ , we see that even both industrial and open-source
projects exhibit the same proportion of tests affected by _Hard Coded Values_
, the number varies significantly from a median of 27 in the case of
industrial projects to 59 for open-source ones. A similar observation can be
made for the case of _Over Checking_ where the median varies from 5 (open-
source) to 70 (industrial). Finally, the last significant difference regards
the symptom _Army of Clones_. Not only the number of tests containing
duplicated code (Figure 6) is higher, but the average number of duplicated
_Keywords_ in symptomatic tests is higher in industrial projects (3) than in
open-source (2).

Figure 8: Distribution of the density of symptoms in each test for all
projects across all commits where the density is the percentage of locations
that a symptom could appear at which actually exhibit the symptom.

To put the results of Figure 7 into perspective, we show the density metrics,
$D_{s}(t)$, in Figure 8. As a reminder, the density value presents the number
of symptoms appearing in a SUIT divided by the number of times it could have
appeared. One interesting finding to put in contrast with the previous figure
regards the smell _Hardcoded Environment_. While the symptoms do not appear
often (only outliers present a value greater than 0 in Figure 7), whenever an
environment variable is used, it will generally be through the use of a
hardcoded value which is shown by a median value for its density metric at 1
for both open-source and industrial projects. That is, with regard to its
potential occurrences, the SUIT smell _Hardcoded Environment_ is actually very
frequent.

On the other hand, the smells _Noisy Logging_ , _Hidden Test Data_ ,
_Sensitive Locator_ , _Narcissistic_ and _Conditional Assertion_ present low
scores in the density value. This means that even when the conditions for
their occurrence are present, the symptoms of these smells do not manifest.

To assess its potential to appear in a test suite, we consider a symptom as
frequent if the median value of the density metric is greater than 0. Thus,
_Hardcoded Values_ with median values of 0.67 and 0.33, _Over Checking_ with
median values of 0.17 and 0.08, and _Sneaky Checking_ with median values of
0.33 and 0.14, for open-source and industrial projects respectively, are
frequent in both industrial and open-source projects. This aligns with the
observations from the count metric and confirms the prevalence of these
smells. Similarly, following the same trend from Figure 7 _Army of Clones_
(median value of 0.40), _Long Test Step_ (median value of 0.5) and _Middle
Man_ (median value of 0.29) appear often in industrial projects.

Nonetheless, in the case of the industrial project, we observe that _On the
Fly_ with a median value of 0.67 shows a deviation from the count metric,
$S_{s}(t)$. As for open-source projects, we observe that _Sticky
Synchronization_ is frequent, with a median value of 0.2. The divergence with
the count metric can be explained by the fact that many tests do not use
explicit synchronization mechanism, thus, limiting the number of tests where
the symptom could appear. Furthermore, the difference from the industrial
project can explained by the hard policy in the QA team at BGL BNP Paribas,
where explicit timeouts are considered as a major source of flakiness. Indeed,
explicit timeouts are strongly discouraged by the QA team and are usually
removed during code reviews.

Both industrial and open-source projects present the some similar trends
despite their difference in terms of design and architecture. We observe that
7 out of the 16 smells symptoms do not typically appear in SUITs, whereas the
smells _Hardcoded Values_ , _Over Checking_ , and _Sneaky Checking_ are
frequent. As for the difference observed between industrial and open-source
projects, we note that the smells _Army of Clones_ , _Long Test Step_ ,
_Middle Man_ and _On the Fly_ often exhibit symptoms in the industrial project
but not in the open-source projects and inversely symptoms of _Sticky
Synchronization_ tend to appear more often in open-source projects. Finally,
the symptoms of _Hardcoded Environment_ appearing systematically would suggest
that the developers do not consider it as a bad practice.

Finally, we perform a ranking analysis between industrial projects and open-
source projects. The goal of this analysis is to determine if the symptoms
appear in the same order in both project. Thus, we use the normalized
Levenshtein similarity where the order of the symptoms is determined by the
mean number of symptoms and percentage of symptom. This metric provide an
indication as the number of permutations that are necessary in order to go
from one list to the other, in other words their similarity. A value of $1$
indicates a perfect similarity and a value of $0$ that the two list are total
dissimilar. We obtain a value of $0.1819$ when comparing the number of
symptoms and a value of $0.3125$ in the case of the number of symptoms. These
low scores suggest that the relative importance of the symptoms present in
SUITs differs from industrial project and open-source projects.

The smells _Hard Coded Values_ , _Over Checking_ , and _Sneaky Checking_ are
prevalent in both industrial and open-source projects. These smells appear in
large numbers and exhibit a high density compared to their potential number of
occurrences. Due to project and team structure, the industrial project is more
prone to complexity smells such as _Army of Clones_ and _Middle Man_. On the
other hand, their code reviewing policies seem to prevent the symptoms of
_Stinky synchronization_.

###  4.4 RQ3: SUIT Smells Refactoring

| Industrial | Open-source
---|---|---
Symptom |  Count |  Percent |  Count |  Percent
Army of Clones |  738 |  36.64 |  10,139 |  24.42
Conditional Assertions |  9 |  4.95 |  270 |  0.32
Hardcoded Environment |  0 |  0.00 |  882 |  9.95
Hardcoded Values |  226 |  18.44 |  28,863 |  17.09
Hidden Test Data |  0 |  0.00 |  0 |  0.00
Lack of Encapsulation |  8 |  0.00 |  10,944 |  11.52
Long Test Steps |  0 |  0.00 |  34 |  0.19
Middle Man |  1,037 |  39.57 |  55,509 |  27.62
Missing Assertion |  6,647 |  90.45 |  137,707 |  72.86
Narcissistic |  0 |  0.00 |  0 |  0.00
Noisy Logging |  0 |  0.00 |  0 |  0.00
On the Fly |  27 |  13.85 |  516 |  7.93
Over-Checking |  35 |  3.46 |  21,586 |  15.50
Sensitive Locators |  2 |  4.55 |  0 |  0.00
Sneaky Checking |  0 |  0.00 |  0 |  0.00
Stinky Synchronization |  38 |  4.92 |  23,163 |  23.28
Table 4: Total Number of refactoring actions (_Count_) and percentage of
symptomatic tests where at least one refactoring action was performed during
their lifetime (_Percent_) for industrial and open-source projects.

While Section 4.3 focuses on the prevalence of smell symptoms across SUITs,
this section tackles the question of how often the symptoms are refactored by
practitioners. We use the methodology described in Section 3.4 to extract the
fine-grained changes removing a symptom from the test, _i.e._ refactoring
actions.

Column _Count_ in Table 4 shows the number of refactoring actions across all
SUIT-modifying commits. Adding assertions to tests presenting the symptom
_Missing Assertion_ is the most common type of refactoring action as it occurs
6,647 times in the industrial project and 137,707 times in open-source
projects. This result is explained by the fact that in SUITs, creating a
scenario and being able to run it from beginning to end already provide a
signal. However, once the test is ready and behaves as expected, more specific
checks, _i.e._ assertions, are added to improve its readability and its fault
detection capabilities. Thus, we observe that some SUITs are missing assertion
when created but assertions are added in later commits.

In the industrial project, three other types of refactoring actions see high
values compared to the rest, namely, _Army of Clones_ with 738 refactoring
actions, _Hardcoded Value_ with 226 refactoring actions, and _Middle Man_ with
1,037 refactoring actions. Indeed, with the introduction of the tool presented
in Rwemalika et al. (2019b), the team at BGL BNP Paribas became aware of the
existence of a large amount of code duplication and actively started to work
on reducing it, which explains the number of refactorings for _Army of
Clones_. As for _Middle Man_ , during the year 2019, the team performed a
normalization in the naming of _Keywords_ , in an effort to improve
readability in the codebase. Consequently, names such as “Fill Form Next Page”
were changed to more expressive forms such as “Fill Login Form and Validate”.
The goal was to increase the expressiveness of the test codebase and
consequently, it reduced the need for a translation layer. Note that in this
case, the team was targeting another SUIT smell, _Unsuitable Naming_ , where
the name of the _Keyword_ does not provide indication as what it is doing, but
ended up also addressing another smell, _Middle Man_. Finally, observing the
number of refactoring actions addressing _Hardcoded Value_ is mainly due to
the fact that the symptom appears often. Indeed, when observing the column
_Percent_ from Table 4, we see only 18% of the test exhibiting the symptom are
refactored through their lifetime.

Still focusing on the results for the industrial project, for some symptoms,
we never observe refactoring actions. This is the case for: _Hidden Test Data_
, _Noisy Logging_ , _Narcissistic_ , _Sensitive Locator_ and _Sneaky
Checking_. With the exception of _Sneaky Checking_ , these symptoms only
rarely occur in the test codebase. Hence, in the absence of symptoms, there is
nothing to refactor. _Sneaky Checking_ , on the other hand, is considered
frequent according to our metric, but is never addressed. One reason can be
that to increase readability, assertions in the acceptance criteria are often
encapsulated in a _User Keyword_ with more meaningful names. When looking at
the metric _Narcissistic_ for the industrial project, we observe no fine-
grained refactoring.

In Listing 1 (Section 2.1) we present an example of a KDT extracted from the
official document of Robot Framework. We observe at line 9 a call to “welcome
page should be open” which exhibits the symptom for _Sneaky Checking_.
However, calling directly the _Library Keyword_ “Title Should be” would have
decreased the readability of the test and introduced another symptom, _Lack of
Encapsulation_.

Putting these results into perspective, Figure 9 shows the evolution of the
symptom _Narcissistic_ over time. We observe an abrupt decrease of the number
of symptoms until they completely disappear. Where this observation seem to
contrast with previous results, this is explained by old tests presenting this
pattern are deprecated by the team and replaced by new ones not presenting the
smell. Thus, while there is no specific refactoring action happening, the
symptoms are removed from the test suite as new test are introduced and old
tests deprecated.

Figure 9: Evolution of the number of symptoms for the smell _Narcissistic_
over time for the industrial project (BGL BNP Paribas).

The results for open-source projects depict a similar picture. Symptoms for
_Missing Assertion_ , _Middle Man_ , _Army of Clones_ , and _HardCoded Values_
show also relatively high values compared to other symptoms. Similarly, we
also notice a relatively large number of refactoring actions for the symptoms
of _Stinky Synchronization_ , _Over Checking_ , and _Lack of Encapsulation_.
The difference between the results from _Stinky Synchronization_ can explain
by the active policy in the industrial team preventing such symptoms to appear
altogether.

Finally, similarly to Section 4.3, we perform a rank analysis using the
normalized Levenshtein similarity. The value obtained when comparing the
number of fixes performed in open-source and industrial projects is $0.4375$
and $0.2667$ when accounting for the percentage of refactoring. These low
values show that the majority of refactoring operations are not ranked in the
same way in both industrial and open-source projects. Taking into account the
results of Section 4.3 this comparison suggests that when generalizing results
obtained from open-source projects, researchers should remain careful to their
applicability in industrial contexts.

In conclusion, the results presented in this section show that the refactoring
operations performed in industry and open-source projects are different in
type and frequency. However, in both cases, the proportion of symptomatic
tests that are refactored remain low with the exception of the symptoms for
_Missing Assertion_.

For half of the smells less than 50% of the symptomatic tests are ever
subjects of refactoring. _Missing assertion_ is a unique exception with
between 70% (open-source) and 90% (industrial) of the symptomatic tests being
refactored. Interestingly, while refactoring actions are rare, test smells
like _Narcissistic_ and _Middle Man_ still disappear from the test codebase as
a side effect of the replacement of symptomatic tests by new tests not
exhibiting the symptom.

##  5 Threats to Validity

Threat to construct validity result from the non suitability of the metrics
used to evaluate the results. To detect test smells we rely on heuristics
based on code metrics. While we focus on metric that offer good precision, we
cannot ignore the fact that some of the smells are not detected by the metrics
devised in this work.

Threats to the internal validity are due to the design of the study,
potentially impacting our conclusions. Such threats typically do not affect
exploratory studies like the one in this work. A caveat can be raised on how
the changes are extracted. Indeed, changes are recorded when developers check-
in their changes to the control version system. Thus, refactoring actions
might be lost if developers do not check-in often their changes or on the
contrary, we might flag artifacts of the development process (_e.g._
Assertions added only at the end of the test) as refactoring actions. To
account for this phenomenon, we analyze manually a subset of the results to
ensure the soundness of the process.

Finally, the threats to external validity, regarding the generalization of the
results, concerns mainly the choice of the projects analyzed. Indeed,
conducting our analysis on a limited sample of projects, our results might not
generalize to other projects. However, we try to control for this limitation
by selecting projects of different sizes, from different domains and in
different development cultures. Furthermore, working with Robot Framework,
there is no guarantee that the results presented in this work are transferable
to other languages or technologies.

##  6 Related work

Considering the uniqueness and impacts of test smells, the research community
entailed studies to extend the catalogs of known test smells, empirical
studies have been conducted to analyze their introduction, diffusion and
suppression, and finally, tools have been proposed to automatically detect
them.

van Deursen et al. (2001) are among the first to introduce the concept of test
smells. They describe a catalog of 11 test smells and refactoring operation to
address them. Following their work, different studies were conducted to
further extend the catalog of test smells. Departing from the analysis of the
code to identify test smells, Bowes et al. (2017) and Tufano et al. (2016)
conducted human studies to isolate test smells and report the perceived
impacts on both test code and production code.

With good test smell catalogs established, the community investigated the
diffusion and evolution of smells in test code. Bavota et al. (2015) conducted
an empirical studies to analyze the diffusion and the impact of test smells.
In their work, Tufano et al. (2016) and later Kim (2020) showed that test
smells are introduced when the test is written and are long lived.

To help practitioners identify potential sub-optimal patterns in their test
codebase, researchers introduced tooling to automatically detect them.
Typically, detection tools analyze test code metrics using rule based
heuristics (Van Rompaey et al., 2007; Reichhart et al., 2007; Peruma et al.,
2020) to isolate test smells.

Nonetheless, all of the work mentioned above focuses on test smells present in
unit testing. Hauptmann et al. (2013) conducted a study on test smells present
in tests expressed in natural language in industrial systems. However, their
experiments show that the metrics extracted cannot be used for assessment of
the quality of a test suite. Femmer et al. (2014, 2017) introduce the concept
of requirement smells and conducted an empirical evaluation of their approach
using industrial projects. In their work, the author present an automated
static analysis technique relying on natural language processing (NLP) to
detect smells in the requirements. The main goal at the requirement level is
to detect whether or not a requirement contains ambiguities, therefore, the
techniques developed rely on concrete instances of ambiguities. Finally Chen
and Wang (2012), propose a catalog of 11 test smell present in KDT along with
16 refactoring methods.

##  7 Conclusion and Implications

The goal of this paper is to shed light on the smells occurring in SUITs. To
do so, we conducted a multivocal literature review and identified SUIT-
specific smells from both formal and grey literature. This process lead to a
catalog of 35 SUIT smells. For 16 of these smells we derive metrics to
characterize their presence and refactoring in the test code. Our results show
that three smells _Hardcoded Values_ (90% of the tests), _Over Checking_
(between 75% and 80% of the tests), and _Sneaky Checking_ (70% of the tests)
are prevalent in both industrial and open-source projects. While these
symptoms are largely present in the test codebase, their refactoring on the
other hand remains low with only less than half the tests ever observing any
targeted refactoring action with the exception of _Missing Assertion_.

Contrastingly, one symptom from the catalog never appears in the subjects of
our study, namely, _Hidden Test Data_. Similarly, other symptoms tend to only
appear in rare occasions such as _Sensitive Locator_ , _Noisy Logging_ and
_Conditional Assertion_. Finally, the smell _Hardcoded Environment_ while
remaining rare, appears systematically when an environment variable is
required.

Though we observe general trends common to both industrial and open-source
projects, when performing a more rigorous comparison between the two sets of
projects, we observe significant differences. Indeed, the projects differ both
in terms of prevalence of the symptoms and refactoring operations. This
results can be explain by the difference in scope, actors, and lifecycle
present in each context. Consequently, these results suggest that when
conducting studies researchers should be aware of these fundamental
differences which might limit the generalization of their results.

In light of the results from this exploratory analysis, we believe that
extending the catalog of known SUIT smells will have an impact on our
partner’s development process. Indeed, they already started using our tooling
to address some of the bad design choices which we observed in their test
codebase. Moreover, with our new catalog and these first observations, we open
perspective for future research on awareness of bad testing practices and the
pitfalls to avoid when evolving SUITs.

This study alongside with the tooling developed and the available
dataset777https://github.com/kabinja/suit-smells-replication-package lays the
ground for future research on the impact of smells on SUIT suites. Our future
agenda is focused on devising automated refactoring approaches. We also plan
to extend the list of 16 metrics to cover more smells from our catalog.

## References

  * Advolodkin (2018) Advolodkin N (2018) Top 17 automated testing best practices (supported by data). URL https://ultimateqa.com/automation-patterns-antipatterns/
  * Alegroth et al. (2016) Alegroth E, Steiner M, Martini A (2016) Exploring the Presence of Technical Debt in Industrial GUI-Based Testware: A Case Study. In: Proceedings of the 9th International Conference on Software Testing, Verification and Validation Workshops, IEEE, pp 257–262, DOI 10.1109/ICSTW.2016.47
  * Arcelli Fontana et al. (2016) Arcelli Fontana F, Mäntylä MV, Zanoni M, Marino A (2016) Comparing and experimenting machine learning techniques for code smell detection. Empirical Software Engineering 21(3):1143–1191, DOI 10.1007/s10664-015-9378-4
  * Archer (2010) Archer M (2010) How test automation with selenium can fail. URL https://mattarcherblog.wordpress.com/2010/11/29/how-test-automation-with-selenium-or-watir-can-fail/
  * Baker et al. (2008) Baker P, Dai ZR, Grabowski J, Haugen, Oystein Schieferdecker I, Williams C (2008) Data-driven testing. In: Model-Driven Testing, Springer, pp 87–95
  * Banerjee et al. (2013) Banerjee I, Nguyen B, Garousi V, Memon A (2013) Graphical user interface (GUI) testing: Systematic mapping and repository. Information and Software Technology 55(10):1679–1694, DOI 10.1016/j.infsof.2013.03.004
  * Battat (2020) Battat M (2020) How do you simplify end-to-end test maintenance? URL https://dzone.com/articles/how-do-you-simplify-end-to-end-test-maintenance-au
  * Bavota et al. (2015) Bavota G, Qusef A, Oliveto R, De Lucia A, Binkley D (2015) Are test smells really harmful? An empirical study. Empirical Software Engineering 20(4):1052–1094, DOI 10.1007/s10664-014-9313-0
  * Biagiola et al. (2019) Biagiola M, Stocco A, Ricca F, Tonella P (2019) Diversity-based web test generation. In: Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ACM Press, Tallinn, Estonia, 1, pp 142–153, DOI 10.1145/3338906.3338970
  * Bowes et al. (2017) Bowes D, Hall T, Petric J, Shippey T, Turhan B (2017) How Good Are My Tests? In: 2017 IEEE/ACM 8th Workshop on Emerging Trends in Software Metrics (WETSoM), IEEE, pp 9–14, DOI 10.1109/WETSoM.2017.2
  * Brooks et al. (2009) Brooks P, Robinson B, Memon AM (2009) An initial characterization of industrial graphical user interface systems. Proceedings of the 2nd International Conference on Software Testing, Verification, and Validation pp 11–20, DOI 10.1109/ICST.2009.11
  * Bushnev (2019) Bushnev Y (2019) Top 15 ui test automation best practices. URL https://www.blazemeter.com/blog/top-15-ui-test-automation-best-practices-you-should-follow
  * Buwalda (2015) Buwalda H (2015) Test design for automation: Anti-patterns. URL https://www.techwell.com/techwell-insights/2015/09/test-design-automation-anti-patterns
  * Buwalda (2019) Buwalda H (2019) 8 test automation anti-patterns (and how to avoid them). URL https://dzone.com/articles/8-test-automation-anti-patterns-and-how-to-avoid-t
  * Canny et al. (2020) Canny A, Palanque P, Navarre D (2020) Model-Based Testing of GUI Applications Featuring Dynamic Instanciation of Widgets. In: Proceedings of the International Conference on Software Testing, Verification and Validation Workshops, IEEE, pp 95–104, DOI 10.1109/ICSTW50294.2020.00029
  * Chen and Wang (2012) Chen WK, Wang JC (2012) Bad smells and refactoring methods for GUI test scripts. Proceedings of the 13th International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing pp 289–294, DOI 10.1109/SNPD.2012.10
  * Clayton (2014) Clayton J (2014) Acceptance tests at a single level of abstraction. URL https://thoughtbot.com/blog/acceptance-tests-at-a-single-level-of-abstraction
  * Cripsin (2018) Cripsin L (2018) Keep your automated testing simple and avoid anti-patterns. URL https://www.mabl.com/blog/keep-your-automated-testing-simple
  * Cunha et al. (2010) Cunha M, Paiva ACR, Ferreira HS, Abreu R (2010) PETTool: A pattern-based GUI testing tool. In: Proceedings of the 2nd International Conference on Software Technology and Engineering, IEEE, vol 1, pp 202–206, DOI 10.1109/ICSTE.2010.5608882
  * De Bleser et al. (2019) De Bleser J, Di Nucci D, De Roover C (2019) Assessing Diffusion and Perception of Test Smells in Scala Projects. In: Proceedings of the 16th International Conference on Mining Software Repositories, IEEE, pp 457–467, DOI 10.1109/MSR.2019.00072
  * van Deursen et al. (2001) van Deursen A, Moonen L, van den Bergh A, Kok G (2001) Simulation of corrugated horns and exciters for circular corrugated waveguides. In: Proceedings of the 2nd International Conference on Extreme Programming and Flexible Processes in Software Engineering, pp 92–95, DOI 10.1134/S106422690609004X
  * Dharmender (2017) Dharmender K (2017) Automation testing: Anti-patterns. URL https://alisterbscott.com/2015/01/20/five-automated-acceptance-test-anti-patterns/
  * England (2016) England T (2016) Cucumber anti-patterns (part one). URL https://cucumber.io/blog/bdd/cucumber-antipatterns-part-one/
  * Evangelisti (2012) Evangelisti A (2012) How to transform bad acceptance tests into awesome ones. URL https://mysoftwarequality.wordpress.com/2012/12/14/how-to-transform-bad-acceptance-tests-into-awesome-ones/
  * Femmer et al. (2014) Femmer H, Fernández DM, Juergens E, Klose M, Zimmer I, Zimmer J (2014) Rapid requirements checks with requirements smells: two case studies. In: Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering, ACM Press, New York, New York, USA, pp 10–19, DOI 10.1145/2593812.2593817
  * Femmer et al. (2017) Femmer H, Méndez Fernández D, Wagner S, Eder S (2017) Rapid quality assurance with Requirements Smells. Journal of Systems and Software 123:190–213, DOI 10.1016/j.jss.2016.02.047, 1611.08847
  * Fowler et al. (1999) Fowler M, Beck K, Brant J, Opdyke W, Roberts D (1999) Refactoring: Improving the Design of Existing Code. Addison-Wesley Longman Publishing Co., Inc., USA
  * Garousi and Küçük (2018) Garousi V, Küçük B (2018) Smells in software test code: A survey of knowledge in industry and academia. Journal of Systems and Software 138:52–81, DOI 10.1016/j.jss.2017.12.013
  * Gawinecki (2016) Gawinecki M (2016) Anti-patterns in test automation. URL http://nomoretesting.com/blog/2016/05/23/anti-patterns/
  * Goldberg (2019) Goldberg Y (2019) Node.js and javascript testing best practices (2020). URL https://yonigoldberg.medium.com/yoni-goldberg-javascript-nodejs-testing-best-practices-2b98924c9347
  * Gomez et al. (2013) Gomez L, Neamtiu I, Azim T, Millstein T (2013) RERAN: Timing- and touch-sensitive record and replay for Android. In: Proceedings of the 35th International Conference on Software Engineering, IEEE, pp 72–81, DOI 10.1109/ICSE.2013.6606553
  * Hauptmann et al. (2013) Hauptmann B, Heinemann L, Vaas R, Braun P (2013) Hunting for smells in natural language tests. In: Proceedings of the 35th International Conference on Software Engineering, IEEE, San Francisco, CA, USA, pp 1217–1220, DOI 10.1109/ICSE.2013.6606682
  * Hauptmann et al. (2015) Hauptmann B, Eder S, Junker M, Juergens E, Woinke V (2015) Generating Refactoring Proposals to Remove Clones from Automated System Tests. In: Proceedings of the 23rd International Conference on Program Comprehension, IEEE, vol 2015-August, pp 115–124, DOI 10.1109/ICPC.2015.20
  * Humble and Farley (2010) Humble J, Farley DG (2010) Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley, Upper Saddle River, NJ
  * Issa et al. (2012) Issa A, Sillito J, Garousi V (2012) Visual testing of Graphical User Interfaces: An exploratory study towards systematic definitions and approaches. In: Proceedings of the International Symposium on Web Systems Evolution, IEEE, pp 11–15, DOI 10.1109/WSE.2012.6320526
  * Jain (2007) Jain N (2007) Patterns and anti-patterns: Acceptance testing with fitnesse. URL https://blogs.agilefaqs.com/2007/08/25/patterns-and-anti-patterns-acceptance-testing-with-fitnesse/
  * Jingfan Tang et al. (2008) Jingfan Tang, Xiaohua Cao, Ma A (2008) Towards adaptive framework of keyword driven automation testing. In: Proceedings of the IEEE International Conference on Automation and Logistics, IEEE, September, pp 1631–1636, DOI 10.1109/ICAL.2008.4636415
  * Kapelonis (2018) Kapelonis K (2018) Software testing anti-patterns. URL http://blog.codepipes.com/testing/software-testing-antipatterns.html
  * Khomh et al. (2009) Khomh F, Vaucher S, Guéhéneuc YG, Sahraoui H (2009) A bayesian approach for the detection of code and design smells. In: Proceedings of the 9th International Conference on Quality Software, IEEE Computer Society, USA, p 305–314, DOI 10.1109/QSIC.2009.47
  * Kim (2020) Kim DJ (2020) An empirical study on the evolution of test smell. In: Proceedings of the 42nd International Conference on Software Engineering: Companion Proceedings, ACM, New York, NY, USA, i, pp 149–151, DOI 10.1145/3377812.3382176
  * Kirkbride (2014) Kirkbride J (2014) Testing anti-patterns. URL https://medium.com/jameskbride/testing-anti-patterns-b5ffc1612b8b
  * Kitchenham and Charters (2007) Kitchenham BA, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Tech. Rep. EBSE 2007-001, Keele University and Durham University Joint Report
  * Klarck (2014) Klarck P (2014) Robot framework dos and don’ts. URL https://slideshare.net/pekkaklarck/robot-framework-dos-and-donts
  * Knight (2017a) Knight A (2017a) Bdd 101: Writing good gherkin. URL https://automationpanda.com/2017/01/30/bdd-101-writing-good-gherkin/
  * Knight (2017b) Knight A (2017b) Should gherkin steps use first-person or third-person? URL https://automationpanda.com/2017/01/18/should-gherkin-steps-use-first-person-or-third-person/
  * Knight (2019) Knight A (2019) Bdd 101: Writing good gherkin. URL https://techbeacon.com/app-dev-testing/7-ways-tidy-your-test-code
  * Machiry et al. (2013) Machiry A, Tahiliani R, Naik M (2013) Dynodroid: an input generation system for Android apps. In: Proceedings of the 9th Joint Meeting on Foundations of Software Engineering, ACM Press, New York, New York, USA, p 224, DOI 10.1145/2491411.2491450
  * Mao et al. (2016) Mao K, Harman M, Jia Y (2016) Sapienz: multi-objective automated testing for Android applications. In: Proceedings of the 25th International Symposium on Software Testing and Analysis, ACM, New York, NY, USA, pp 94–105, DOI 10.1145/2931037.2931054
  * Marinescu (2004) Marinescu R (2004) Detection strategies: metrics-based rules for detecting design flaws. In: Proceedings of the 20th IEEE International Conference on Software Maintenance, IEEE, pp 350–359, DOI 10.1109/ICSM.2004.1357820
  * Memon and Nguyen (2010) Memon AM, Nguyen BN (2010) Advances in Automated Model-Based System Testing of Software Applications with a GUI Front-End. In: Advances in Computers, vol 80, 1st edn, Elsevier Inc., pp 121–162, DOI 10.1016/S0065-2458(10)80003-8
  * Mesbah and van Deursen (2009) Mesbah A, van Deursen A (2009) Invariant-based automatic testing of AJAX user interfaces. In: Proceedings of the 31st International Conference on Software Engineering, IEEE, pp 210–220, DOI 10.1109/ICSE.2009.5070522
  * Meszaros (2007) Meszaros G (2007) xUnit Test Patterns: Refactoring Test Code, 1st edn. Addison-Wesley
  * Moonen et al. (2008) Moonen L, van Deursen A, Zaidman A, Bruntink M (2008) On the Interplay Between Software Testing and Evolution and its Effect on Program Comprehension, Springer Berlin Heidelberg, Berlin, Heidelberg, pp 173–202. DOI 10.1007/978-3-540-76440-3_8
  * Morlion (2019) Morlion P (2019) Software testing anti patterns. URL https://www.enov8.com/blog/software-testing-anti-patterns/
  * Myers (1995) Myers BA (1995) User Interface Software Tools. ACM Transactions on Computer-Human Interaction 2(1):64–103, DOI 10.1145/200968.200971
  * Myers and Rosson (1992) Myers BA, Rosson MB (1992) Survey on user interface programming. In: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press, New York, New York, USA, pp 195–202, DOI 10.1145/142750.142789
  * Nguyen et al. (2014) Nguyen BN, Robbins B, Banerjee I, Memon A (2014) GUITAR: an innovative tool for automated testing of GUI-driven software. Automated Software Engineering 21(1):65–105, DOI 10.1007/s10515-013-0128-9
  * Peruma et al. (2020) Peruma A, Almalki K, Newman CD, Mkaouer MW, Ouni A, Palomba F (2020) tsDetect: an open-source test smells detection tool. In: Proceedings of the 28th Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ACM, New York, NY, USA, pp 1650–1654, DOI 10.1145/3368089.3417921
  * Pezzè et al. (2018) Pezzè M, Rondena P, Zuddas D (2018) Automatic GUI testing of desktop applications. In: Companion Proceedings for the ISSTA/ECOOP 2018 Workshops, ACM, New York, NY, USA, i, pp 54–62, DOI 10.1145/3236454.3236489
  * Reichhart et al. (2007) Reichhart S, Gîrba T, Ducasse S (2007) Rule-based Assessment of Test Quality. The Journal of Object Technology 6(9):231, DOI 10.5381/jot.2007.6.9.a12
  * Renaudin (2016) Renaudin J (2016) Software testing anti patterns. URL https://www.slideshare.net/JosiahRenaudin/antipatterns-for-automated-testing
  * Ricca and Stocco (2021) Ricca F, Stocco A (2021) Web Test Automation: Insights from the Grey Literature. In: Proceedings of the 47th International Conference on Current Trends in Theory and Practice of Computer Science, Bolzano, Italy, pp 472–485, DOI 10.1007/978-3-030-67731-2_35
  * Rwemalika et al. (2019a) Rwemalika R, Kintis M, Papadakis M, Le Traon Y, Lorrach P (2019a) On the evolution of keyword-driven test suites. In: Proceedings of the 12th International Conference on Software Testing, Verification and Validation, New York, NY, USA, pp 335–345
  * Rwemalika et al. (2019b) Rwemalika R, Kintis M, Papadakis M, Le Traon Y, Lorrach P (2019b) Ukwikora: Continuous inspection for keyword-driven testing. In: Proceedings of the 28th International Symposium on Software Testing and Analysis, Association for Computing Machinery, New York, NY, USA, p 402–405, DOI 10.1145/3293882.3339003
  * Salihu et al. (2019) Salihu Ia, Ibrahim R, Ahmed BS, Zamli KZ, Usman A (2019) AMOGA: A Static-Dynamic Model Generation Strategy for Mobile Apps Testing. IEEE Access 7(c):17158–17173, DOI 10.1109/ACCESS.2019.2895504
  * Satopaa et al. (2011) Satopaa V, Albrecht J, Irwin D, Raghavan B (2011) Finding a "Kneedle" in a Haystack: Detecting Knee Points in System Behavior. In: Proceedings of the 31st International Conference on Distributed Computing Systems Workshops, IEEE, pp 166–171, DOI 10.1109/ICDCSW.2011.20
  * Sciamanna (2019) Sciamanna A (2019) What are the anti patterns of automation with selenium? URL https://anthonysciamanna.com/2019/10/20/avoiding-automated-testing-pitfalls.html
  * Scott (2015) Scott A (2015) Five automated acceptance test anti-patterns. URL https://alisterbscott.com/2015/01/20/five-automated-acceptance-test-anti-patterns/
  * Shay (2019) Shay L (2019) Bdd cucumber features best practices. URL https://www.linkedin.com/pulse/bdd-cucumber-features-best-practices-liraz-shay/
  * Sheth (2020) Sheth H (2020) 16 selenium best practices for efficient test automation. URL https://www.lambdatest.com/blog/selenium-best-practices-for-web-testing/
  * Silva and Valente (2017) Silva D, Valente MT (2017) RefDiff: Detecting Refactorings in Version Histories. In: Proceedings of the 14th International Conference on Mining Software Repositories, IEEE, pp 269–279, DOI 10.1109/MSR.2017.14, 1704.01544
  * Siminiuc (2019) Siminiuc A (2019) What are the anti patterns of automation with selenium? URL https://www.quora.com/What-are-the-anti-patterns-of-automation-with-selenium
  * StackExchange (2017) StackExchange (2017) What are anti-patterns in test automation? URL https://sqa.stackexchange.com/questions/8508/what-are-anti-patterns-in-test-automation
  * Temov (2020) Temov J (2020) Want to speed end-to-end testing? don’t send in the clones. URL https://techbeacon.com/app-dev-testing/want-speed-end-end-testing-dont-send-clones
  * Tsantalis et al. (2013) Tsantalis N, Guana V, Stroulia E, Hindle A (2013) A Multidimensional Empirical Study on Refactoring Activity. In: Proceedings of the 2013 Conference of the Center for Advanced Studies on Collaborative Research, IBM Corp., Ontario, Canada, pp 132–146
  * Tufano et al. (2016) Tufano M, Palomba F, Bavota G, Di Penta M, Oliveto R, De Lucia A, Poshyvanyk D (2016) An empirical investigation into the nature of test smells. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, ACM Press, Singapore, Singapore, pp 4–15, DOI 10.1145/2970276.2970340
  * Van Rompaey et al. (2007) Van Rompaey B, Du Bois B, Demeyer S, Rieger M (2007) On The Detection of Test Smells: A Metrics-Based Approach for General Fixture and Eager Test. IEEE Transactions on Software Engineering 33(12):800–817, DOI 10.1109/TSE.2007.70745
  * Virgínio et al. (2020) Virgínio T, Martins LA, Soares LR, Santana R, Costa H, Machado I (2020) An empirical study of automatically-generated tests from the perspective of test smells. In: Proceedings of the 34th Brazilian Symposium on Software Engineering, ACM, New York, NY, USA, pp 92–96, DOI 10.1145/3422392.3422412
  * Yu et al. (2019) Yu S, Fang C, Feng Y, Zhao W, Chen Z (2019) LIRAT: Layout and Image Recognition Driving Automated Mobile Testing of Cross-Platform. In: Proceedings of the 34th International Conference on Automated Software Engineering, IEEE, pp 1066–1069, DOI 10.1109/ASE.2019.00103