# On the Distribution of “Simple Stupid Bugs” in Unit Test Files: An
Exploratory Study

Anthony Peruma  Rochester Institute of Technology
Rochester, New York, USA
<EMAIL_ADDRESS>Christian D. Newman  Rochester Institute of Technology
Rochester, New York, USA
<EMAIL_ADDRESS>
###### Abstract

A key aspect of ensuring the quality of a software system is the practice of
unit testing. Through unit tests, developers verify the correctness of
production source code, thereby verifying the system’s intended behavior under
test. However, unit test code is subject to issues, ranging from bugs in the
code to poor test case design (i.e., test smells). In this study, we compare
and contrast the occurrences of a type of single-statement-bug-fix known as
“simple stupid bugs” (SStuBs) in test and non-test (i.e., production) files in
popular open-source Java Maven projects. Our results show that SStuBs occur
more frequently in non-test files than in test files, with most fix-related
code associated with assertion statements in test files. Further, most test
files exhibiting SStuBs also exhibit test smells. We envision our findings
enabling tool vendors to better support developers in improving the
maintenance of test suites.

##  I Introduction

Unit testing is an essential strategy employed by developers to ensure the
quality of their software system. Under this strategy, developers write code
(i.e., test cases) that verifies the behavior of individual units of work of
the system under test [1]. However, the test code written by developers is
vulnerable to issues such as bugs (i.e., functional defects) and smells (i.e.,
bad programming practices), which impact not only the quality of the system
but also the system’s maintenance; specifically, the maintenance of test cases
[2].

However, correcting these software issues might not always be a complicated
task, with some bugs being much easier to correct than others; often requiring
a change to a single statement. These are sometimes referred to as “simple
stupid bugs” (SStuBs) [3]. While there exist studies that investigate defects
in source code, these studies on program repair [4] do not differentiate
defects occurring in test and non-test files, nor do these studies focus
exclusively on SStuB-like defects. These studies mostly focus on a test case’s
ability to identify defects in the system under test [5, 6, 7]. Furthermore,
studies that focus on the quality of test cases focus on test smells [8]
exhibited by test files such as investigating the flakiness (i.e., non-
deterministic outcome) of test cases [9, 10]. In this paper, we focus on
SStuBs in test files. We explore, compare, and contrast the existence of
SStuBs in test files against non-test files. Furthermore, our study also looks
at the co-occurrence between SStuBs and test smells.

###  I-A Goal & Research Questions

The goal of this study is to explore the quality of test suites from a
functional and non-functional perspective, and the relationship between these
viewpoints. Hence, we utilize SStuBs to represent the functional quality
aspects of the test suite, and test smells for the non-functional quality
aspects. As an exploratory study, we aim to have a better understanding of
this problem and determine the feasibility of further research in this area.
Our findings provide developers with insight to better design, implement, and
maintain test suites. Additionally, our findings can improve the support
provided by refactoring and other code quality tools. Hence, our study aims at
answering the following research questions (RQs):

  * •

RQ1: To what extent do SStuBs occur in test files as they do in non-test
files? This RQ compares the occurrence of SStuBs in test files and non-test
files. We examine the volume and types of SStuBs occurring in files along with
the developers responsible for fixing SStuBs in the two groups of file types.

  * •

RQ2: To what extent do test files containing SStuB fixes also exhibit test
smells? This RQ explores the existence of test smells in files containing
fixes for SStuBs. This RQ aims to understand if SStuBs can be an indicator for
the presence of test smells in the code.

##  II Data Extraction and Collection

Figure 1: Overview of our data extraction and collection process. TABLE I:
Summary of the test smell detection rules utilized by tsDetect [11].

Test Smell | Detection Rule
---|---
Assertion Roulette | A test method contains more than one assertion statement without an explanation/message (parameter in the assertion method)
Conditional Test Logic | A test method that contains one or more control statements (i.e., if, switch, conditional expression, for, foreach and while statement)
Constructor Initialization | A test class that contains a constructor declaration
Duplicate Assert | A test method that contains more than one assertion statement with the same parameters
Empty Test | A test method that does not contain a single executable statement
Exception Handling | A test method that contains either a throw statement or a catch clause
General Fixture | Not all fields instantiated within the setUp method of a test class are utilized by all test methods in the same test class
Ignored Test | A test method or class that contains the @Ignore annotation
Magic Number Test | An assertion method that contains a numeric literal as an argument
Mystery Guest | A test method containing object instances of files and databases classes
Redundant Print | A test method that invokes either the print or println or printf or write method of the System class
Redundant Assertion | A test method that contains an assertion statement in which the expected and actual parameters are the same
Resource Optimism | A test method utilizes an instance of a File class without calling the method exists(), isFile() or notExists() methods of the object
Sensitive Equality | A test method invokes the toString() method of an object
Sleepy Test | A test method that invokes the Thread.sleep() method
Unknown Test | A test method that does not contain a single assertion statement and @Test(expected) annotation parameter

Figure 1 shows a general overview of our data collection and extraction
process. In the below subsections, we elaborate on the activities involved in
the process. The dataset we utilize/generate is available at [12] for
replication/extension.

###  II-A Source Dataset

Our study utilizes the ManySStuBs4J dataset, which contains bug-fixing details
of SStuBs extracted from popular open-source Java Maven projects [3].
Contained within this dataset is the relative path of the file containing the
fix, the commit id associated with the fix, and the type of SStuB fixed by the
developer, among other details.

###  II-B Repository Cloning & File Extraction

In this stage of the process, we clone the Maven repositories that contain
bug-fixing commits for SStuBs. The purpose of this activity is twofold. First,
we extract metadata associated with the commit, such as the author (i.e.,
developer) of the fix. Second, we extract the source code files associated
with each SStuB-fixing commit. The original Maven dataset contains 10,231
SStuB-fixing instances distributed over 84 projects. However, at the time of
cloning these repositories, only 83 of the repositories were publicly
available. Hence, in this study, we analyze 10,225 SStuB-fixing instances.

###  II-C Test File Detection

In this study, we only consider projects that utilize JUnit [13] as the
testing framework since prior unit testing based research has frequently
focused on JUnit (e.g., test smells [8]). JUnit recommends that developers
either prepend or append the term ‘Test’ to the name of the production file to
be tested (i.e., Test*.java or *Test.java). However, this mechanism can lead
to false positives. For our study, we utilize the approach by [14] and [11] to
detect test files. In this approach, we utilize JavaParser [15] to build an
abstract syntax tree for each source file. We mark a file as a unit test file
if the file contains JUnit import statements (i.e., org.junit.* or junit.*)
and a test method. To be a test method, the method should have an annotation
called @Test (JUnit 4), or the method name should start with ‘test’ (JUnit 3).

###  II-D Test Smell Detection

For the detection of test smells we utilize tsDetect [11], an open-source test
smells detection tool. tsDetect detects 19 test smell types and has been
utilized in multiple studies [14, 16, 17, 18, 19, 20]. Provided in Table I is
a summary of the detection rules for each smell type. However, for this study,
we ignore the Default Test smell as this smell is exclusive to Android
applications. We also ignore the smells Eager Test and Lazy Test as these
smells require the class under test to be known; which is not known for this
dataset. Hence, our analysis is limited to 16 smell types. Details about the
smell types, along with examples, are available in [14].

###  II-E Research Question Analysis

To answer our research questions, we execute a series of database queries and
custom scripts on the mined data. Additionally, where applicable, we provide
examples from our dataset to supplement our findings. When reporting on our
research questions, we first provide details about the experiment’s
methodology before presenting the findings.

##  III Experimental Results

###  III-A RQ1: To what extent do SStuBs occur in test files as they do in
non-test files?

Methodology. In this RQ, we group the files containing fixes for SStuBs into
two groups– test files and non-test files. The test files represent JUnit
based unit testing files, while the non-test files represent the set of
production files. We examine and compare details such as the distribution of
SStuBs between the two groups, the developers responsible for applying the
fix, and the clustering of file types in a bug-fixing commit.

Results. First, we look at the projects containing SStuBs fixing commits. From
the set of 83 projects in the dataset, 57 ($\approx$ 68.67%) projects contain
SStuBs occurring in both test and non-test files, while 25 ($\approx$ 30.12%)
projects contain SStuBs occurring in only non-test files, and only one project
contains SStuBs occurring in only test files. Our next examination shows that
developers apply SStuB fixes to 5,587 Java files. From this set, 1,066
($\approx$ 19.08%) are test files and 4,521 ($\approx$ 80.92%) are non-test
files. Moreover, from a total of 10,225 instances of SStuB fixes, 1,946
($\approx$ 19.03%) of these instances were in test files, while 8,279
($\approx$ 80.97%) were in non-test files. Furthermore, we observe, on
average, 2.20 SStuBs occur in test files, and 2.37 occur in non-test files.

Our next analysis focuses on SStuB categories. In total, there are 16 SStuB
categories, the details of which are available at [3]. In Table II, we provide
the top five frequently occurring SStuB categories in test and non-test files.
From this table, we observe that even though the categories are the same for
both file types, the ratio of occurrences of these categories differs between
file types. Further, we also observe that the categories Change Numeric
Literal and Change Modifier occur at different positions relative to the other
categories. These top five categories contribute to 88.23% and 81.83% of the
complete set of occurrences in test and non-test file types, respectively.
Additionally, we also observe that while the non-test files contain instances
of all 16 categories, the test files do not exhibit the Change Operand
category. In general, we observe 314 ($\approx$ 16.13%) SStuB instances in
test files are associated with assertion statements. Examining the code
statement containing the fix for Change Numeric Literal in test files, we
observe that 112 ($\approx$ 20.4%) of the instances occur with an assertion
statement (e.g.,
$\texttt{assertEquals(2,map.size())}\rightarrow\texttt{assertEquals(3,map.size())}$
[21]). Furthermore, 116 ($\approx$ 21.13%) instances are related to numeric
values associated with time-related identifiers (e.g.,
$\texttt{timeout=2000}\rightarrow\texttt{timeout=1000}$ [22]). In contrast,
non-test files contain five instances of SStuBs in assertion statements and 31
time-related SStuBs. For cases under the Change Modifier category, we observe
that developers either remove or include the static keyword with a class or
method in test files. We observe 14 cases falling under the Same Function More
Args category are related to assertion methods in test files. In most of these
cases, developers include a textual message related to the assertion
condition’s failure (e.g., [23]). Additionally, we also encounter instances,
in test files, where the methods that are updated are related to mocking;
either API’s related to Mockito or custom methods that accept mocked objects
as arguments (e.g.,
$\texttt{Mockito.any()}\rightarrow\texttt{Mockito.any(ProducerRecord.class)}$
[24]).

Our subsequent analysis is on the grouping of test and non-test files in a
single commit. Since SStuBs can occur in both test and non-test files of a
project, we are interested in knowing whether developers make fixes (i.e.,
commit) for both file types in unison or prioritize fixing one file type over
another. Hence, for this analysis, we only focus on the 57 projects that
contain SStuBs in test and non-test files. Our analysis shows that only one
project meets this criterion. Furthermore, this project had only one commit
operation that contained a test and non-test file. This phenomenon shows that
developers prefer to treat SStuBs in isolation, most likely so that they can
keep track of the changes and revert the change if necessary.

Finally, we examine the distribution of developers that fix SStuBs. Hence, we
perform this analysis on the set of 57 projects containing test and non-test
files. To detect unique developers, we utilize the same approach as [25]. The
approach utilizes the SStuB fixing commit author’s email to identify unique
developers for a project. In total, we encounter 1,116 unique bug fixing
developers. From this set, 84 ($\approx$ 7.53%) developers are responsible for
fixing SStuBs occurring only in test files, while 789 ($\approx$ 70.70%) fix
SStuBs occurring only in non-test files. Developers that fix SStuBs in both
test and non-test files account for 243 ($\approx$ 21.77%).

TABLE II: Top five SStuB categories in (non-)test files. SStuB Category | Count | Percentage
---|---|---
Test Files (Total Bug Type Instances: 1,946)
Change Identifier Used | 554 | 28.47%
Change Numeric Literal | 549 | 28.21%
Wrong Function Name | 310 | 15.93%
Same Function More Args | 158 | 8.12%
Change Modifier | 146 | 7.50%
Non-Test Files (Total Bug Type Instances: 8,279)
Change Identifier Used | 2,708 | 32.71%
Change Modifier | 1,705 | 20.59%
Wrong Function Name | 1,175 | 14.19%
Same Function More Args | 600 | 7.25%
Change Numeric Literal | 587 | 7.09%

Summary. This RQ shows that non-test files contain more SStuBs than test
files. Additionally, we observe that the volume of SStuBs occurring in
individual test and non-test files are very similar ($\approx$ 2 instances).
We also observe that the majority of developers work on fixing SStuBs in non-
test files than test files. Additionally, The top five popular SStuB
categories for test and non-test files are the same. However, the ratio of
occurrences of these categories differs. Furthermore, the code associated with
SStuB fixes differs between test and non-test files; assertion statements in
test files are frequently updated due to SStuB fixes.

###  III-B RQ2: To what extent do test files containing SStuB fixes also
exhibit test smells?

Methodology. SStuBs represent the functional quality of a system. In contrast,
test smells represent the non-functional quality of a system, both of which
are important to the maintenance of the system’s test suite. In this RQ, we
look at the existence of test smells in test files exhibiting SStuBs.
Additionally, we also investigate if the fixing of SStuBs causes a change in
the number of test smell types exhibited by the file. To this extent, we
utilize tsDetect to analyze the test files containing SStuB fixes and the
prior version (i.e., commit) of the file for the existence of test smells.

TABLE III: Co-occurrence between the top five frequently occurring SStuB bug
categories and test smell types.

| SStuB
---
Category
| Assertion
---
Roulette
| Conditional
---
Test Logic
| Constructor
---
Initialization
| Empty
---
Test
| Exception
---
Handling
| General
---
Fixture
| Mystery
---
Guest
| Redundant
---
Print
| Redundant
---
Assertion
| Sensitive
---
Equality
| Sleepy
---
Test
| Duplicate
---
Assert
| Unknown
---
Test
| Ignored
---
Test
| Resource
---
Optimism
| Magic
---
Number Test
Change Numeric Literal | 375 | 373 | 354 | 326 | 375 | 366 | 344 | 346 | 304 | 355 | 364 | 372 | 372 | 368 | 355 | 373
Change Identifier Used | 356 | 352 | 346 | 327 | 356 | 341 | 326 | 336 | 319 | 347 | 339 | 351 | 353 | 346 | 328 | 356
Wrong Function Name | 192 | 189 | 183 | 170 | 192 | 177 | 170 | 177 | 160 | 184 | 178 | 188 | 190 | 183 | 172 | 192
Same Function More Args | 110 | 110 | 109 | 97 | 110 | 106 | 104 | 102 | 100 | 109 | 105 | 110 | 110 | 107 | 104 | 110
Change Modifier | 95 | 94 | 94 | 91 | 95 | 93 | 93 | 92 | 78 | 94 | 93 | 94 | 95 | 95 | 95 | 95
Other Categories | 164 | 163 | 162 | 155 | 164 | 161 | 158 | 157 | 153 | 162 | 159 | 163 | 164 | 163 | 159 | 164

Results. From the set of 1,066 test files containing SStuB fixes, 1,064 files
show the presence of test smells, with each file exhibiting, on average, 15.41
smell types. Examining each smell type, we observe that each of the 16 smell
types occur in over 80% of the test files exhibiting SStuB fixes. Furthermore,
the smell types Assertion Roulette and Exception Handling occur in all the
test files. In terms of SStuB categories, we observe that Change Numeric
Literal and Change Identifier Used are the top two categories occurring in 375
and 356 smelly test file instances, respectively. This is in contrast to the
popularity results shown in Table II.

Next, we look at the co-occurrence of each SStuB category with each smell
type. In Table III we present the number of instances (i.e., test files) where
a SStuB bug category co-occurs with a test smell. Due to space constraints, we
only present the top five occurring SStuB bug categories. Our analysis shows
that the smells Assertion Roulette, Exception Handling, and Magic Number Test
are the three smell types that most frequently co-occur with each of the SStuB
categories. In contrast, the least co-occurring smell type is Redundant
Assertion.

Finally, we look at the version of the test file the developer modifies just
before the commit containing the SStuB fix. Our objective is to determine if
the smell count decreases when the developer fixes a SStuB. Our findings show
that most SStuB fixes do not result in a decrease in smell count. More
specifically, 853 ($\approx$ 80.17%) instances do not show a change in smell
count, while 182 ($\approx$ 17.11%) instances show a decrease and 29
($\approx$ 2.73%) instances show an increase. Looking at the instances that
show a decrease, we observe that, on average, 3.22, smell types are removed
between the two versions of the file. We observe that the smells Assertion
Roulette and Exception Handling do not show a reduction, while the smell
Redundant Assertion is a popular smell that reduces.

Summary. Test files exhibiting SStuBs are very likely to contain test smells,
with the Assertion Roulette and Exception Handling smell types frequently
occurring in such files. Additionally, test files that exhibit certain types
of SStuBs, such as Change Numeric Literal, also exhibit test smells. However,
developers rarely fix these smells when addressing SStuBs.

##  IV Discussion

As an exploratory study, our research aims to understand the extent to which
SStuBs occur in test files and their relationship to test smell so as to
provide direction to research areas that support developers in designing and
maintaining test suites.

From RQ1, we observe that SStuBs tend to occur more frequently in non-test
files than test files. However, this should not signify that test files are of
better quality than non-test files; this might be a situation where developers
might not always be addressing these types of defects in test suites. It is
interesting to note that developers usually address test and non-test SStuBs
in separate commits. This is interesting since a change in code in a non-test
file would require an appropriate change to the test code to ensure that the
test case passes [26]. Committing a non-test change independently of a test
change would most usually cause the test case to fail in an automated
build/test environment. Hence, it is most likely that such systems either do
not use an automated build system or lack sufficient code coverage.
Additionally, our observation of most developers working on fixing non-test
SStuBs over test SStuBs can be a further indicator that developers prioritize
non-test files over test files; and might need to adhere to a test-driven
development approach [27]. However, further research into this area is
warranted to understand the rationale as to why developers treat test files
differently from non-test files. Additionally, there needs to be research into
tools that support developers with automatically finding/recommending changes
to test files based on SStuB fixes applied to non-test files.

RQ1 also shows that the code related to SStuB fixes tend to differ between
test and non-test files. We observe that the code usually associated with
SStuB fixes in test files are frequently associated with assertion statements,
such as in the case of the Change Numeric Literal SStuB category. The frequent
occurrence of issues related to assertion statements should not be surprising
as asserts are one of the most fundamental parts of a test case. Furthermore,
in RQ2, we observe the frequent occurrence of the smell Assertion Roulette;
which corroborates with findings from prior test smell studies [2, 9]. These
findings show that the occurrence of a specific SStuB category in the test
suite indicates the occurrence of specific test smells in the same file, and
thereby help developers in addressing multiple issues in the code that would
otherwise be missed. A similar finding reported by Spadini et al. [9] shows an
association between smelly tests and defect-proneness of the production code
under test. For example, in our dataset, we observe instances where the Change
Numeric Literal SStuB category occurs due to changes made to the
Thread.sleep() value, which is also an indicator of the Sleepy Test smell
(e.g., [28]). This specific smell can lead to unexpected results as the
processing time for a task differs when executed in various environments and
configurations; most likely, the developer experienced this situation due to
the sleep duration change. Lastly, our findings on the non-removal of smells
also corroborate with research by Tufano et al. [29]. Once more, all our
findings can help tool/IDE vendors better equip their devices to better
support developers to improve the functional and non-functional aspects of
their test suites.

##  V Threats to Validity

Even though the projects in our dataset are some of the most popular open-
source Maven-based Java systems, the results may not generalize to systems
written in other languages. Furthermore, we confine our analysis to the JUnit
testing framework. However, prior unit testing-based research has frequently
focused on JUnit [8]. Our selection of tsDetect is due to its ability to
detect multiple smell types and better detection performance [20]. Finally,
even though non-test files have more SStuBs than test files, the per-file rate
of SStuBs of test and non-test files are very similar. This aspect is
interesting and requires more in-depth analysis, such as the possibility that
SStuBs are not found at the same rate in test and non-test files.

##  VI Conclusion & Future Work

In this study, we explore the occurrence of SStuBs in test files and their
relation to test smells. We observe that SStuBs occur more frequently in non-
test files than test files and that most of the fix related code differs
between test and non-test files. Finally, we show that test files exhibiting
SStuBs also exhibit test smells and tend to co-occur with specific types of
SStuBs. These findings show that there is indeed scope for future research in
this area, especially around the maintenance of test suites. Future work in
this area includes investigating the flakiness of test suites brought about by
SStuBs.

## References

  * [1] R. Pressman and D. Bruce R. Maxim, Software Engineering: A Practitioner’s Approach.  McGraw-Hill Education, 2014.
  * [2] G. Bavota, A. Qusef, R. Oliveto, A. De Lucia, and D. Binkley, “An empirical analysis of the distribution of unit test smells and their impact on software maintenance,” in 2012 28th IEEE International Conference on Software Maintenance (ICSM), pp. 56–65, 2012.
  * [3] R.-M. Karampatsis and C. Sutton, “How often do single-statement bugs occur? the manysstubs4j dataset,” in Proceedings of the International Conference on Mining Software Repositories (MSR 2020), 2020.
  * [4] L. Gazzola, D. Micucci, and L. Mariani, “Automatic software repair: A survey,” IEEE Transactions on Software Engineering, vol. 45, no. 1, pp. 34–67, 2019.
  * [5] D. Kim, J. Nam, J. Song, and S. Kim, “Automatic patch generation learned from human-written patches,” in 2013 35th International Conference on Software Engineering (ICSE), pp. 802–811, 2013.
  * [6] S. Shamshiri, R. Just, J. M. Rojas, G. Fraser, P. McMinn, and A. Arcuri, “Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges (t),” in 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 201–211, 2015.
  * [7] M. M. Almasi, H. Hemmati, G. Fraser, A. Arcuri, and J. Benefelds, “An industrial evaluation of unit test generation: Finding real faults in a financial application,” in 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), pp. 263–272, 2017.
  * [8] V. Garousi and B. Küçük, “Smells in software test code: A survey of knowledge in industry and academia,” Journal of Systems and Software, vol. 138, pp. 52 – 81, 2018.
  * [9] D. Spadini, F. Palomba, A. Zaidman, M. Bruntink, and A. Bacchelli, “On the relation of test smells to software code quality,” in 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 1–12, 2018.
  * [10] G. Grano, F. Palomba, D. Di Nucci, A. De Lucia, and H. C. Gall, “Scented since the beginning: On the diffuseness of test smells in automatically generated test code,” Journal of Systems and Software, vol. 156, 2019.
  * [11] A. Peruma, K. Almalki, C. D. Newman, M. W. Mkaouer, A. Ouni, and F. Palomba, “Tsdetect: An open source test smells detection tool,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, (New York, NY, USA), p. 1650–1654, Association for Computing Machinery, 2020.
  * [12] “Dataset.” https://doi.org/10.5281/zenodo.4608719.
  * [13] “Junit.” https://junit.org/.
  * [14] A. Peruma, K. Almalki, C. D. Newman, M. W. Mkaouer, A. Ouni, and F. Palomba, “On the distribution of test smells in open source android applications: An exploratory study,” in Proceedings of the 29th Annual International Conference on Computer Science and Software Engineering, CASCON ’19, (USA), p. 193–202, IBM Corp., 2019.
  * [15] “Javaparser.” http://javaparser.org/.
  * [16] M. Schvarcbacher, D. Spadini, M. Bruntink, and A. Oprescu, “Investigating developer perception on test smells using better code hub-work in progress,” in 2019 Seminar Series on Advanced Techniques and Tools for Software Evolution, SATTOSE 2019, 2019.
  * [17] D. Spadini, M. Schvarcbacher, A.-M. Oprescu, M. Bruntink, and A. Bacchelli, “Investigating severity thresholds for test smells,” in Proceedings of the 17th International Conference on Mining Software Repositories, MSR ’20, 2020\.
  * [18] D. J. Kim, “An empirical study on the evolution of test smell,” in Proceedings of the 42nd International Conference on Software Engineering: Companion Proceedings, ICSE ’20, 2020.
  * [19] A. Peruma, C. D. Newman, M. W. Mkaouer, A. Ouni, and F. Palomba, “An exploratory study on the refactoring of unit test files in android applications,” in Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW’20, (New York, NY, USA), p. 350–357, Association for Computing Machinery, 2020.
  * [20] A. Panichella, S. Panichella, G. Fraser, A. A. Sawant, and V. J. Hellendoorn, “Revisiting test smells in automatically generated tests: Limitations, pitfalls, and opportunities,” in 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 523–533, 2020\.
  * [21] “Configurationpropertiesreportendpointserializationtests.java.” https://github.com/spring-projects/spring-boot/commit/0757d24.
  * [22] “Beanrecipientlisttimeouttest.java.” https://github.com/apache/camel/commit/cb6f6e2.
  * [23] “Evictiontest.java.” https://github.com/hazelcast/hazelcast/commit/7fa8196.
  * [24] “Kafkaproducertest.java.” https://github.com/apache/camel/commit/82bd0ba.
  * [25] A. Peruma, M. W. Mkaouer, M. J. Decker, and C. D. Newman, “Contextualizing rename decisions using refactorings, commit messages, and data types,” Journal of Systems and Software, vol. 169, p. 110704, 2020.
  * [26] E. Burns and W. Prakash, Hudson Continuous Integration in Practice.  McGraw-Hill Education, 2013.
  * [27] K. Beck, Test-driven Development: By Example.  Addison-Wesley signature series, Addison-Wesley, 2003.
  * [28] “Exponentialbackoffmsgretrymanagertest.java.” https://github.com/apache/storm/commit/9134320.
  * [29] M. Tufano, F. Palomba, G. Bavota, R. Oliveto, M. D. Penta, A. De Lucia, and D. Poshyvanyk, “When and why your code starts to smell bad (and whether the smells go away),” IEEE Transactions on Software Engineering, vol. 43, no. 11, pp. 1063–1088, 2017.


# Interpret Your Care: Predicting the Evolution of Symptoms for Cancer
Patients

Rupali Bhati 1, Jennifer Jones 2, Audrey Durand 1,3

###### Abstract

Cancer treatment is an arduous process for patients and causes many side-
effects during and post-treatment. The treatment can affect almost all body
systems and result in pain, fatigue, sleep disturbances, cognitive
impairments, etc. These conditions are often under-diagnosed or under-treated.
In this paper, we use patient data to predict the evolution of their symptoms
such that treatment-related impairments can be prevented or effects
meaningfully ameliorated. The focus of this study is on predicting the pain
and tiredness level of a patient post their diagnosis. We implement an
interpretable decision tree based model called LightGBM on real-world patient
data consisting of 20163 patients. There exists a class imbalance problem in
the dataset which we resolve using the oversampling technique of SMOTE. Our
empirical results show that the value of the previous level of a symptom is a
key indicator for prediction and the weighted average deviation in prediction
of pain level is 3.52 and of tiredness level is 2.27.

## Introduction

Nearly half of Canadians are expected to receive a diagnosis of cancer in
their lifetime (Brenner et al. 2020). Unlike other chronic diseases where
disability is commonly caused by the disease process, disability associated
with cancer is often caused more by treatment than the disease itself (Short
PF 2008). Individual cancer patients may undergo different treatments, for
different amounts of time and may even suffer from multiple diagnoses. Such
experiences render the patients with distinct symptoms along their treatment,
for example pain, fatigue, sleep disturbances, or cognitive impairments. There
is a need for tailored models of rehabilitation that account for the fact that
“one size is not likely to fit all” needs of different cancer survivors (Chan
et al. 2021). To this aim, in many cancer centres, patient symptom data is
collected via surveys which are filled by the patient when they come for their
cancer clinic visits. However, asking patients to fill in surveys about their
health conditions too often can be very burdening. Therefore, there is a need
to systematically identify the adverse effects of cancer such that they can be
treated efficiently.

Our aim is to leverage previously collected patient reported data to predict
the evolution of a patient’s symptoms. Predicting when a patient encounters
high levels of symptom burden may help clinicians take precautionary measures
to avoid such situations or be able to effectively manage the patient’s
conditions. For a predictive model to be used within such an application, it
should be interpretable by clinicians (Zhu et al. 2014). The clinician not
only needs to see the predicted value of the symptom but also needs to
understand the criteria based on which the prediction is made.

In this paper we, therefore, propose an interpretable pipeline for predicting
the level of two different symptoms, i.e. pain and tiredness levels, using
LightGBM (Ke et al. 2017), a decision tree (DT) based model which is widely
used for its efficiency and interpretability. DT based methods have a wide
application in healthcare (Podgorelec, Kokol, and et al. 2002; Rufo et al.
2021) and cancer-related applications, for example to predict breast cancer
survivability (Khan et al. 2008). Although other, possibly more complex,
predictive models have displayed good performances at predicting future
symptoms of oncology patients (N et al. 2018), there has been resistance to
adopting these machine learning based methods in clinical practice due to a
perception that these are “black-box” techniques and incompatible with
decision-making based on evidence and clinical experience (Zhu et al. 2014).
Therefore, the proposed pipeline rather relies on an interpretable approach
based on DTs, combined with a training procedure based on oversampling (Chawla
et al. 2002) to compensate for the important data imbalance observed across
the different values of symptom level. By evaluating this approach against a
naive predictor based solely on prior distributions of symptoms, we highlight
the necessity of using evaluation metrics that compensate for data imbalance
in order to evaluate fairness in model performances. Finally, although a
feature importance analysis of our models reveals that the previous symptom
value plays an important role in predicting the current symptom value, our
results show a clear benefit in leveraging clinical variables in the
predictions and, in some cases, the correlation between symptoms.

## Problem Statement

Post diagnosis, a patient visits the doctor or clinician either when they are
going through a treatment or require some assistance. These visits can take
place at irregular intervals. During such visits, the patient is asked to fill
a survey documenting their symptoms. Pain and tiredness are two symptoms which
acutely affect the overall quality of life of a patient undergoing cancer
treatment. Both symptoms are documented by the patient as a discrete value
between 0 and 10, where 0 is the lowest level of pain/tiredness a patient can
feel and 10 is the highest based on the Edmonton Symptom Assessment Scale
(ESAS) (Watanabe, Nekolaichuk, and Beaumont 2012). To demonstrate how the
symptoms of a patient can evolve over a period of time, Figure 1 shows an
example of the evolution of a patient’s symptoms along a timeline.

Figure 1: Example of symptom evolution for one patient

We have access to a dataset of real cases collected for years 2013-2019 on
20163 patients. On average, there are 5.95 observations per patient. Four
cancer types are included in this analysis, namely, breast, colorectal,
lymphoma and head and neck. The dataset includes patient information, i.e.,
sex, age, cancer type, and surgery data, in addition to the survey data, i.e.,
information about the patient’s symptoms, including ESAS values and Patient
Reported Functional Status (PRFS) measures (Popovic et al. 2018) which monitor
overall nutritional status in patients. For the purpose of this study, we
focus on predicting some important symptoms using clinical and demographic
information of the patient and their previous symptom values which are
summarised in Table 1 along with the values each feature can take.

Figure 2: Distribution of number of samples over classes for pain and
tiredness levels

To avoid a situation where the future observations of a patient are in the
training set while the older observations are in the test set, we split the
dataset based on a date. The surveys before the date ’2017-10-04’ are in the
training set and the ones after this date are in the test set. This leads to a
75%-25% split between the training and test set, respectively. The
distribution of number of samples in each class can been seen in Figure 2 for
each symptom. We observe that there are class imbalance problems, where $\sim
44\%$ and $\sim 22\%$ of the outputs belong to one class (level 0) for pain
and tiredness, respectively. It is also noted that the number of samples in a
class are inversely proportional to the symptom level. This occurs because
high levels of pain and tiredness levels are relatively rarer events. This
bias in the data can lead to unfairness in the model’s predictions and
therefore inequity in a patient’s treatment and may exacerbate health care
disparities (Rajkomar et al. 2018).

Variable |  Possible values
---|---
Sex |  male, female
Age |  18 to 93
Cancer type |  breast, head and neck, lymphoma, colorectal
Number of days between the diagnosis and the survey date |  5 to 23091 ($\sim 63$ years)
Number of days between the previous and the current survey date |  1 to 2002 ($\sim 5$ years)
Previous pain level |  {0,1,2,3,4,5,6,7,8,9,10}
Previous tiredness level |  {0,1,2,3,4,5,6,7,8,9,10}
Table 1: Variables available for prediction

## Methodology

In this section, we propose a pipeline to predict the symptom levels of a
patient while trying to maintain equal performance as much as possible over
the different classes. As the values to predict are discrete (they belong to
the finite, discrete set of symptom levels), we formulate this problem as a
classification problem. We propose to evaluate candidate strategies using an
interpretable metric that accounts for class imbalance, ensuring that the
performance over all groups of patients (based on their symptoms) is equally
represented in the global evaluation score.

### Predictive model

Clinical providers and other decision makers in healthcare note
interpretability of model predictions as a priority for implementation and
utilization (Ahmad, Teredesai, and Eckert 2018). To this end, we use DTs for
prediction as each decision point can be visualised individually by
clinicians. Among several DT algorithms, we select LightGBM for several
reasons. Firstly, LightGBM is a gradient boosting DT algorithm. Gradient
boosting is an approach where new models are created that predict the
residuals or errors of prior models and then added together to make the final
prediction. It is called gradient boosting because it uses a gradient descent
algorithm to minimize the loss when adding new models which is more beneficial
as compared to other DT algorithms that either train the different models
separately before aggregating them or do not use multiple models at all.
Secondly, instead of building trees that have a uniform depth across all
branches, LightGBM expands the tree leaf-wise (up to a maximum tree depth
given as a parameter), resulting in much lighter trees and making it suitable
for high-speed and low memory consumption.

For predicting each symptom (pain and tiredness), we consider two variants of
LightGBM models. The first variant relies on clinical variables (first five
rows of Table 1) and previous level of the predicted symptom (row 6 or 7 of
Table 1) in the prediction, while the second variant uses all clinical
variables in addition to previous levels of both symptoms (all rows of Table
1). We refer to the first variants as LP1 and LT1, and to the second variants
as LP2 and LT2, for pain and tiredness respectively.

#### Fighting data imbalance

Inspired by previous works (Yang Zhao and Tsui 2018), we rely on Synthetic
Minority Oversampling TEchnique (SMOTE) (Chawla et al. 2002) to address class
imbalance by generating synthetic samples. At training time, SMOTE over-
samples the minority classes so that the trained model sees the same number of
data points of these classes as for the majority class. A minority class is
over-sampled by taking each of its samples and introducing synthetic examples
along the line segments joining any/all of the $k$ minority class nearest
neighbours. For our experiments, we use $k=5$.

#### Training

To determine the maximum tree depth parameter for LightGBM, we perform a
5-fold cross-validation on the training set (90046 samples from 16674
patients) for tree depths values between 1 and 25 (as the performance
saturates after 25). The maximum tree depth value minimizing the WMAE metric
(described below using Equations 1, 2, 3) averaged over the 5 folds is then
used for retraining a LightGBM model using on the whole training dataset.

### Baselines

We compare the performance of the LightGBM models with two baselines models.
We refer to these baselines as NP and PV for naive prior and previous value,
respectively.

#### Naive prior

This baseline is a naive strategy that always predicts the dominant (most
frequent) symptom level (i.e., 0 for both pain and tiredness).

#### Previous value

This baseline is slightly smarter than NP. It predicts the current symptom
level to be the same as the previous symptom level, which could have been
recorded at any time in the past because surveys are collected at irregular
intervals. Therefore, the previous value may or may not be a good indicator of
the current value of the symptom.

### Metrics

In case of imbalanced classes, there can often be misleading evaluation
metrics where mistakes on a minority class can be hidden by good performance
on a dominant class. This can lead to unfairness in the treatment of patients,
for example, due to the high frequency of low pain level in the data, we might
underestimate the level of pain a patient might be in. To this end, we use
weighted metrics such that the model can focus on achieving good performance
on individual classes as well as overall. Let $\mathcal{C}$ denote the set of
classes. We measure the performance of a given strategy on each class
$c\in\mathcal{C}$ using the mean absolute error (MAE) :

$\text{MAE}_{c}=\frac{1}{N_{c}}\sum_{i=1}^{N_{c}}\left|y_{i}-\hat{y}_{i}\right|,$
(1)

where $y_{i}$ is the true value of the $i^{th}$ sample in class $c$,
$\hat{y_{i}}$ is the value predicted by the strategy for the $i^{th}$ sample
in class $c$, and $N_{c}$ is the number of samples belonging to class $c$. The
MAE corresponds to the absolute deviation in the symptom level making it
interpretable for clinicians to infer and analyse. Therefore, a good model
would have a low MAE so that for the correct predictions, it would be zero and
for the incorrect predictions, the model would predict a symptom level closer
to the true symptom level. We combine these values using the weighted-MAE
(WMAE), which essentially gives value to the MAE of each class inversely
proportional to the number of samples in that class. Therefore, mistakes on a
class for which there is little data account for more in the WMAE. To
calculate the WMAE, we first calculate the weight of each class $c$ as

$w_{c}=\frac{\max_{c^{\prime}\in\mathcal{C}}N_{c^{\prime}}}{N_{c}}.$ (2)

The weight will therefore be of value 1 for the class with the largest amount
of data, and would be 10 times larger for a class with 10 times less data. We
can then compute the global WMAE as

$\text{WMAE}=\frac{\sum_{c\in\mathcal{C}}w_{c}\text{MAE}_{c}}{\sum_{c\in\mathcal{C}}w_{c}}.$
(3)

## Results

|  NP |  PV |  LP1 |  LP2
---|---|---|---|---
$\text{MAE}_{0}$ |  0 |  0.52 |  0.8 |  0.8
$\text{MAE}_{1}$ |  1 |  0.91 |  1.34 |  1.35
$\text{MAE}_{2}$ |  2 |  1.3 |  1.89 |  1.88
$\text{MAE}_{3}$ |  3 |  1.5 |  2.11 |  2.09
$\text{MAE}_{4}$ |  4 |  1.82 |  2.3 |  2.26
$\text{MAE}_{5}$ |  5 |  2.04 |  2.45 |  2.46
$\text{MAE}_{6}$ |  6 |  2.39 |  2.54 |  2.55
$\text{MAE}_{7}$ |  7 |  2.76 |  2.67 |  2.72
$\text{MAE}_{8}$ |  8 |  2.89 |  2.65 |  2.62
$\text{MAE}_{9}$ |  9 |  2.98 |  2.59 |  2.64
$\text{MAE}_{10}$ |  10 |  5.03 |  4.41 |  4.53
WMAE |  8.72 |  3.88 |  3.52 |  3.57
Table 2: Prediction performance per pain level ($\text{MAE}_{c}$) and global performance (WMAE) for each strategy |  NP |  PV |  LT1 |  LT2
---|---|---|---|---
$\text{MAE}_{0}$ |  0 |  0.7 |  0.84 |  0.83
$\text{MAE}_{1}$ |  1 |  0.99 |  1.21 |  1.21
$\text{MAE}_{2}$ |  2 |  1.22 |  1.6 |  1.59
$\text{MAE}_{3}$ |  3 |  1.41 |  1.86 |  1.86
$\text{MAE}_{4}$ |  4 |  1.55 |  2.02 |  2.01
$\text{MAE}_{5}$ |  5 |  1.71 |  2.23 |  2.22
$\text{MAE}_{6}$ |  6 |  1.91 |  2.29 |  2.27
$\text{MAE}_{7}$ |  7 |  2.06 |  2.17 |  2.18
$\text{MAE}_{8}$ |  8 |  2.2 |  2.17 |  2.15
$\text{MAE}_{9}$ |  9 |  2.4 |  2 |  2.03
$\text{MAE}_{10}$ |  10 |  3.25 |  2.63 |  2.52
WMAE |  8.43 |  2.65 |  2.33 |  2.27
Table 3: Prediction performance per tiredness level ($\text{MAE}_{c}$) and
global performance (WMAE) for each strategy

For each symptom, we evaluate the performance of the four considered
strategies: the two baselines (NP and PV) and the two variants of LightGBM
models (LP1/LT1 and LP2/LT2). We consider both the performance per class
($\text{MAE}_{c}$) and the global performance (WMAE).

Table 2 shows the results obtained for the task of pain level prediction.
Recall that the dominant class for pain level is class 0, therefore, the NP
model predicts 0 everywhere. If we did not account for class imbalance, the
average of the $\text{MAE}_{c}$’s would be 55/11 = 5, which is lower than the
WMAE. Even worse, the global MAE would be 1.65. This shows how low MAE values
for high frequency classes can compensate for the high MAE values of low
frequency classes, ultimately leading to an unfair metric to evaluate the
performance of a model. In that case, the model would predict high frequency
classes more often and still be able to maintain a good average MAE. In
application, the model would predict high levels of pain less often which
would be detrimental to the patients who might need immediate care. The LP1
model performs the best overall suggesting that using the previous tiredness
level is not useful to predict the pain level. Although the discrepancy
between class performance is slightly attenuated with the LP1 model compared
with other strategies, we should still note that, its performance
($\text{MAE}_{c}$) is still inversely proportional to the class density. This
indicates that the LP1 model is still unfair towards individuals with stronger
pain and further work would probably be needed to make it clinically
acceptable.

Table 3 shows the results obtained for the 4 models used to predict the
tiredness level. The dominant class here is once again class 0.
Unsurprisingly, we observe lower WMAE when predicting tiredness than pain
since the former suffers less class imbalance. Moreover, it can be observed
that LT2 performs better than LT1 indicated by the lower WMAE. This suggests
that adding the previous pain level is a good indicator to predict the
tiredness level.

It is interesting to note that the advantage of LightGBM-based models is
observed for higher symptom levels, i.e., $\text{MAE}_{c}$ for $c>6$ is higher
with PV than with LP1/LP2 and $\text{MAE}_{c}$ for $c>7$ is higher with PV
than with LT1/LT2. This suggests that patients with higher symptom levels
would benefit the most from more advanced models.

Figures 3 and 4 show the importance of each feature in terms of their SHAP
(Lundberg and Lee 2017) values for the LP1 and LT2 models, respectively. The
SHAP value of a feature indicates the impact of the feature on the magnitude
of the output, helping to understand the contribution of each feature to each
class. For both symptoms, we observe that previous symptom levels strongly
contribute to the prediction for current symptom levels.

Figure 3: Feature Importance obtained from LightGBM model to predict pain
(LP1)  Figure 4: Feature Importance obtained from LightGBM model to predict
tiredness using previous pain level (LT2)

## Conclusion

In this study, we predict the pain and tiredness level of a patient using
their clinical and demographic data and previous observations of the symptom
levels. The results demonstrate the need for weighted metrics and the
importance of previous symptom levels. In the future, we want to formulate
this problem as a a reinforcement learning problem and model the context-
reward relationship using the DTs as discussed in (Elmachtoub et al. 2018). We
also plan to predict more qualitative measures like the indicator of quality
of life.

## Acknowledgements

We thank Mathieu Godbout for his feedback on the paper and the CanRehab team
for their generous support.

## References

  * Ahmad, Teredesai, and Eckert (2018) Ahmad, M. A.; Teredesai, A.; and Eckert, C. 2018.  Interpretable Machine Learning in Healthcare.  In _2018 IEEE International Conference on Healthcare Informatics (ICHI)_ , 447–447.
  * Brenner et al. (2020) Brenner, D. R.; Weir, H. K.; Demers, A. A.; Ellison, L. F.; Louzado, C.; Shaw, A.; Turner, D.; Woods, R. R.; and Smith, L. M. 2020.  Projected estimates of cancer in Canada in 2020.  _CMAJ_ , 192(9): E199–E205.
  * Chan et al. (2021) Chan, R.; Nekhlyudov, L.; Duijts, S.; and et al. 2021.  Future research in cancer survivorship.  _J Cancer Surviv (2021) https://doi.org/10.1007/s11764-021-01102-x_.
  * Chawla et al. (2002) Chawla, N.; Bowyer, K.; Hall, L.; and Kegelmeyer, W. 2002.  SMOTE: Synthetic Minority Over-Sampling Technique.  _J. Artificial Intelligence Research, vol. 16, pp. 321-357, 2002._
  * Elmachtoub et al. (2018) Elmachtoub, A. N.; McNellis, R.; Oh, S.; and Petrik, M. 2018.  A Practical Method for Solving Contextual Bandit Problems Using Decision Trees.  arXiv:1706.04687.
  * Ke et al. (2017) Ke, G.; Meng, Q.; Finley, T.; Wang, T.; Chen, W.; Ma, W.; Ye, Q.; and Liu, T.-Y. 2017.  LightGBM: A Highly Efficient Gradient Boosting Decision Tree.  In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Khan et al. (2008) Khan, M. U.; Choi, J. P.; Shin, H.; and Kim, M. 2008.  Predicting breast cancer survivability using fuzzy decision trees for personalized healthcare.  In _2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society_ , 5148–5151.
  * Lundberg and Lee (2017) Lundberg, S. M.; and Lee, S.-I. 2017.  A unified approach to interpreting model predictions.  In _Proceedings of the 31st international conference on neural information processing systems_ , 4768–4777.
  * N et al. (2018) N, P.; Puschmann D, B. P.; B, C.; X, H.; and R, M. 2018.  Learning from data to predict future symptoms of oncology patients.  _arXiv preprint arXiv:1804.09028_.
  * Podgorelec, Kokol, and et al. (2002) Podgorelec, V.; Kokol, P.; and et al., S. B. 2002.  Decision Trees: An Overview and Their Use in Medicine.  _In: Journal of Medical Systems 26, 445–463 (2002). https://doi.org/10.1023/A:1016409317640_.
  * Popovic et al. (2018) Popovic, G.; Harhara, T.; Pope, A.; Al-Awamer, A.; Banerjee, S.; Bryson, J.; Mak, E.; Lau, J.; Hannon, B.; Swami, N.; et al. 2018.  Patient-reported functional status in outpatients with advanced cancer: correlation with physician-reported scores and survival.  _Journal of pain and symptom management_ , 55(6): 1500–1508.
  * Rajkomar et al. (2018) Rajkomar, A.; Hardt, M.; Howell, M. D.; Corrado, G.; and Chin, M. H. 2018.  Ensuring Fairness in Machine Learning to Advance Health Equity.  _Annals of internal medicine, 169(12), 866–872. https://doi.org/10.7326/M18-1990_.
  * Rufo et al. (2021) Rufo, D. D.; Debelee, T. G.; Ibenthal, A.; and Negera, W. G. 2021.  Diagnosis of Diabetes Mellitus Using Gradient Boosting Machine (LightGBM).  _Diagnostics_ , 11(9).
  * Short PF (2008) Short PF, B. R., Vasey JJ. 2008.  Work disability associated with cancer survivorship and other chronic conditions.  _Psycho-oncology, 17(1), 91–97._
  * Watanabe, Nekolaichuk, and Beaumont (2012) Watanabe, S. M.; Nekolaichuk, C. L.; and Beaumont, C. 2012.  The Edmonton Symptom Assessment System, a proposed tool for distress screening in cancer patients: development and refinement.  _Psycho-Oncology_ , 21(9): 977–985.
  * Yang Zhao and Tsui (2018) Yang Zhao, Z. S.-Y. W.; and Tsui, K. L. 2018.  A Framework of Rebalancing Imbalanced Healthcare Data for Rare Events’ Classification: A Case of Look-Alike Sound-Alike Mix-Up Incident Detection.  _In: Journal of Healthcare Engineering https://doi.org/10.1155/2018/6275435_.
  * Zhu et al. (2014) Zhu, M.; Cheng, L.; Armstrong, J. J.; Poss, J. W.; Hirdes, J. P.; and Stolee, P. 2014.  Using machine learning to plan rehabilitation for home care clients: Beyond ”black-box” predictions.  In _Machine learning in healthcare informatics_ , 181–207. Springer.