# Thermal Quantum Correlations in Two Gravitational Cat States

Moises Rojas<EMAIL_ADDRESS>Physics Department, Federal University of
Lavras, Caixa Postal 3037, 37200-000 Lavras-MG, Brazil.  Iarley P. Lobo
<EMAIL_ADDRESS>Department of Chemistry and Physics, Federal University
of Paraíba, Rodovia BR 079 - Km 12, 58397-000 Areia-PB, Brazil.  Physics
Department, Federal University of Lavras, Caixa Postal 3037, 37200-000 Lavras-
MG, Brazil.

###### Abstract

We consider the effect of a thermal bath on quantum correlations induced by
the gravitational interaction in the weak field limit between two massive cat
states, called gravitational cat (gravcat) states. The main goal of this paper
is to provide a good understanding of the effects of temperature and several
parameters in the entanglement (measured by the concurrence) and quantum
coherence (measured by the $l_{1}$-norm that is defined from the minimal
distance between the quantum state and the set of incoherent states) which are
derived from the thermal quantum density operator. Our results show that the
thermal concurrence and $l_{1}$-norm can be significantly optimized by
increasing the masses or decreasing the distance between them. We investigate
and discuss the behavior of these quantities under temperature variations in
different regimes, including some that are expected to be experimentally
feasible in the future. In particular, we observe that thermal fluctuations
raise non-entangled quantum correlations when entanglement suddenly drops.

The quantization of the gravitational interaction has occupied a prominent
space in the research of gravity and quantum field theory in the past several
decades. This quest has been mainly focused on efforts to coherently quantize
the gravitational field or the spacetime geometry Kiefer:2007ria ;
Rovelli:1997yv ; Mukhi:2011zz ; Ambjorn:2010rx . Since the beginning of the
21st century, the research in quantum gravity has entered a new era in which
phenomenological approaches to this problem have arisen due to technological
advances that have allowed experimental bounds on predictions with Planck
scale sensitivity AmelinoCamelia:2008qg ; Addazi:2021xuf . Besides constraints
from observations of astrophysical events Acciari:2020kpi ; Addazi:2018uhd ,
underground and laboratory-based measurements have been considered in recent
investigations (see, for instance, Arzano:2019toz ; Lobo:2020qoa ;
Carney:2018ofe ; Addazi:2017bbg .)

Recently, a more fundamental question regarding this problem has been matter
of debate: how can one verify if the gravitational interaction indeed needs to
be quantized Marletto:1 ? Besides that, is there a model-independent
experimental protocol that can assure whether gravity is an interaction of
quantum mechanical nature Marletto:2017kzi ; Bose:2017nin ? In these
approaches, based on the theory of quantum information, it is argued that an
interaction that is responsible for creating entanglement between two systems
must be of quantum nature. Therefore, sufficient evidence of quantum gravity
would be the detection of entanglement between massive states induced by the
gravitational interaction Rovelli:2021thx ; Bose:2022uxe .

Despite the ongoing debate on the matter Christodoulou:2018cmk ;
Christodoulou:2018xiy ; Hall:2017nzl ; Reginatto:2018ksa ;
Anastopoulos:2020cdp , the existence of entanglement (besides other
correlations) between massive states constitutes an exciting topic to be
analysed by itself. In this paper, we do not aim to discuss the issue of
whether the detection of gravity-induced entanglement represents a test of the
quantum nature of the gravitational interaction; instead, we pose a very
simple question that we believe has not been sufficiently addressed in the
literature of this specific topic: how are the entanglement and other quantum
correlations in this scenario affected by a thermal bath? The different
subject of some environmental effects in this setup has been analyzed
previously for the case of the decoherence time necessary to observe
entanglement Rijavec:2020qxd ; DiBartolomeo:2021jpc , and its robustness
against stochastic fluctuations was verified Nguyen:2019huk , besides other
recent investigations on this matter Miki:2020hvg ; Qvarfort:2018uag .
However, here we aim to analyze how the influence of temperature (through a
thermal bath) might destroy some correlations presented in this context.

The quantum entanglement plays an important role in quantum computation and
quantum information Bennett:1992tv ; Bennett:1995tk ; Amico:2007ag , which is
a tool that has been recently explored for simulating properties of a quantum
spacetime Li:2017gvt ; Czelusta:2020ryq . In a similar manner, quantum
coherence l1norm1 ; Streltsov:2016iow  is also a central concept in quantum
mechanics and has been extensively explored in the context of quantum
information processing Streltsov:2016iow , quantum metrology frowis ;
giovannetti , thermodynamics santos ; lostaglio , effects of a minimum length
scale through the use of a Generalized Uncertainty Principle
Petruzziello:2020wkd  or the assumption of modified spacetime symmetries in
the Schrödinger equation Arzano:2022nlo , among others.

Moreover, the quantum coherence is a fundamental measure of the quantum nature
of a given phenomenon; for instance, it governs effects such as interference
of states and provides important tools for quantum optics Glauber:1963tx ;
Sudarshan:1963ts . Besides that, it has been recently shown a close relation
between quantifiers of coherence and entanglement Streltsov:2015xia . In
recent years, employing quantum coherence as a resource, several interesting
results have been obtained in different quantum systems, among them, in
quantum thermodynamics Kammerlander ; conditionalentropy , quantum dots q-dots
, Heisenberg spin chains spin-chains , spin waves steady , quantum batteries
batteries , quantum computation shor ; envir , among others.

Several quantities have been proposed as candidate measures of quantum
coherence. In particular, Baumgratz et al. l1norm1  introduced the
$l_{1}$-norm of coherence and the relative entropy of coherence as measures of
this property. In the present work, we confine ourselves to utilize the
$l_{1}$-norm of coherence. In this paper, we aim to contribute to the research
on the use of these quantum correlations in gravity, by investigating the way
temperature (induced by a thermal bath) impacts quantifiers of entanglement
and quantum coherence. We stress that we will consider a closed system, but in
thermal contact with a thermal bath. Further analyses regarding the assumption
of an open system and indeed environmental effects environment1 ; environment2
go beyond the scope of this paper.

The present paper is structured as follows: In Section I, we present the model
describing the gravitational interaction that generates two qubits (which we
call gravcat states), and we introduce the thermal density operator due to the
thermal bath. Subsequently, in Section II, we present thermal concurrence as a
quantifier of entanglement and the $l_{1}$-norm as a quantifier of quantum
coherence (both depend on the temperature of the thermal bath). In Section
III, we present the evolution of these quantum correlations as a function of
the temperature and the different parameters of the model, in which the
temperature range for which the system remains entangled is shown, and the
appearance of thermal fluctuations is discussed. Finally, in Section IV, our
conclusions and further discussions are presented.

##  I The Model

The model consists of a set of two particles of mass $m$, each one in a one-
dimensional double-well potential, with local minima at $x=\pm L/2$, which we
will call gravitational cats (gravcats). The potential is even and is such
that we can associate two eigenstates for each particle describing its
localization in each of the minima $|\pm\rangle$, where
$\hat{x}|\pm\rangle=\pm\frac{L}{2}|\pm\rangle$. From the Landau–Lifschitz
approximation Anastopoulos:2020cdp , they can be written in terms of the
ground $|0\rangle$ and first excited states $|1\rangle$ as

$|\pm\rangle=\frac{1}{\sqrt{2}}(|0\rangle\pm|1\rangle).$ (1)

The use of just two states is justified by the fact that, in this model
(described in details in Anastopoulos:2020cdp ), the energy difference between
the ground and first excited states (calculated using the WKB method in
section 3 of Anastopoulos:2020cdp ) is much smaller than the energy of the
second excited state. This allows such two-dimensional description.

The experimental setup that could be modeled as above can be found in some
references in this subject, for instance Marletto:2017kzi ; Bose:2017nin ;
Carlesso:2019cuh , such as massive molecules, nanomechanical oscillators,
diamond microspheres, optomechanical systems, etc.

The Hamiltonian $\mathcal{H}$ for this model has been analyzed in
Anastopoulos:2020cdp  and can be written as

$\displaystyle\mathcal{H}=$
$\displaystyle\frac{w}{2}\left(\sigma_{z}\otimes\mathbb{I}+\mathbb{I}\otimes\sigma_{z}\right)-\Delta(\sigma_{x}\otimes\sigma_{x})\,,$
(2)

where $\sigma_{x,z}$ is usual Pauli matrices; $w$ is energy difference between
the ground and first excited states and furnishes the energy scale of this
setup.111We assume $\hbar=1$. We also have a quantity $\Delta$ that measures
the intensity of the gravitational interaction between the states with masses
$m$, and is given by
$\Delta=\frac{\alpha}{2}\left(\frac{1}{d}-\frac{1}{d^{{}^{\prime}}}\right)$,
where $\alpha=Gm^{2}$, $G$ is a universal gravitational constant; besides
that, $d$ and $d^{\prime}$ are the relative distances between the two masses,
as depicted in Figure 1.

Figure 1: (Color online) Schematic representation of our model. Two particles
are located in an even double well potential, whose minima are separated by a
distance $L$. The quantity $d$ ($d^{\prime}$) describes the distance between
the particles when each of them are at the same (different) relative minima.

After the diagonalization of the Hamiltonian, we obtain the following
eigenvalues:

$\displaystyle\varepsilon_{1}=$ $\displaystyle-\Delta,$
$\displaystyle\varepsilon_{2}=$ $\displaystyle\Omega,$
$\displaystyle\varepsilon_{3}=$ $\displaystyle-\Omega,$
$\displaystyle\varepsilon_{4}=$ $\displaystyle\Delta\,,$ (3)

where $\Omega^{2}=\Delta^{2}+w^{2}$. Their corresponding eigenstates are
described in terms of the standard basis, respectively, by

$\displaystyle|\varphi_{1}\rangle=$
$\displaystyle\frac{1}{\sqrt{2}}\left(|10\rangle+|01\rangle\right),$ (4)
$\displaystyle|\varphi_{2}\rangle=$
$\displaystyle\sin(\theta_{-})|11\rangle+\cos(\theta_{-})|00\rangle,$ (5)
$\displaystyle|\varphi_{3}\rangle=$
$\displaystyle\sin(\theta_{+})|11\rangle+\cos(\theta_{+})|00\rangle,$ (6)
$\displaystyle|\varphi_{4}\rangle=$
$\displaystyle\frac{1}{\sqrt{2}}\left(-|10\rangle+|01\rangle\right),$ (7)

where $\theta_{\pm}=\arctan\left(\frac{\Delta}{w\pm\Omega}\right)$.

### Thermal Density Operator

In order to study the thermal effects on the quantum correlations of our
model, we consider the system in thermal bath, the system state in the thermal
equilibrium is described by
$\rho(T)=\frac{\exp\left(-\beta\mathcal{H}\right)}{Z}$, where
$\beta=\frac{1}{k_{B}T}$, with $k_{B}$ being the Boltzmann’s constant, $T$ is
absolute temperature, and the partition function of the system is defined by
$Z=Tr\left[\exp\left(-\beta\mathcal{H}\right)\right]$. The thermal state of
this system can be described by the quantum density operator

$\rho=\left[\begin{array}[]{cccc}\rho_{1,1}&0&0&\rho_{1,4}\\\
0&\rho_{2,2}&\rho_{2,3}&0\\\ 0&\rho_{2,3}&\rho_{2,2}&0\\\
\rho_{1,4}&0&0&\rho_{4,4}\end{array}\right],$ (8)

whose elements are expressed by

$\displaystyle\rho_{1,1}=$
$\displaystyle\frac{1}{Z}\left(\mathrm{e}^{-\beta\varepsilon_{2}}\sin^{2}{(\theta_{-})}+\mathrm{e}^{-\beta\varepsilon_{3}}\sin^{2}{(\theta_{+})}\right),$
$\displaystyle\rho_{1,4}=$
$\displaystyle\frac{1}{Z}\left(\frac{\mathrm{e}^{-\beta\varepsilon_{2}}\sin{(2\theta_{-})}+\mathrm{e}^{-\beta\varepsilon_{3}}\sin{(2\theta_{+})}}{2}\right),$
$\displaystyle\rho_{2,2}=$
$\displaystyle\frac{1}{Z}\left(\frac{\mathrm{e}^{-\beta\varepsilon_{1}}+\mathrm{e}^{-\beta\varepsilon_{4}}}{2}\right),$
$\displaystyle\rho_{2,3}=$
$\displaystyle\frac{1}{Z}\left(\frac{\mathrm{e}^{-\beta\varepsilon_{1}}-\mathrm{e}^{-\beta\varepsilon_{4}}}{2}\right),$
$\displaystyle\rho_{4,4}=$
$\displaystyle\frac{1}{Z}\left(\mathrm{e}^{-\beta\varepsilon_{2}}\cos^{2}{(\theta_{-})}+\mathrm{e}^{-\beta\varepsilon_{3}}\cos^{2}{(\theta_{+})}\right),$
(9)

where $\varepsilon_{k}$ are the eigenenergies of the Hamiltonian ${\cal H}$,
given by the energies of the eigenstates $\varphi_{k}$, and
$Z=Tr\left[\exp\left(-\beta\mathcal{H}\right)\right]=\sum_{i}\exp(-\beta\varepsilon_{i})$
assumes this form for the diagonalized Hamiltonian. The thermal density
operator assumes such $X$-shape form due to the symmetry of the Hamiltonian,
in addition to the thermal density operator must be Hermitian, also
$Tr(\rho)=1$, $\rho^{2}\neq\rho$ and $Tr(\rho^{2})\leq 1$, due to the fact
that we are dealing with a mixed density operator. Since $\rho(T)$ describes a
thermal state, correlations of this system are called thermal quantum
correlations.

##  II Thermal Quantum Correlations

In this section, we discuss the thermal entanglement and quantum coherence for
two gravcats in contact with a thermal bath.

###  II.1 Thermal Entanglement

To describe the thermal entanglement of the state $\rho$, we use the
concurrence defined by Wootters wootters ; hill , which is given by

$\mathcal{C}=\mathrm{max}\\{\sqrt{\lambda_{1}}-\sqrt{\lambda_{2}}-\sqrt{\lambda_{3}}-\sqrt{\lambda_{4}},0\\},$
(10)

assuming $\lambda_{i}$ are the eigenvalues in decreasing order of the matrix

$R=\rho\left(\sigma_{y}\otimes\sigma_{y}\right)\rho^{*}\left(\sigma_{y}\otimes\sigma_{y}\right),$
(11)

where $\rho^{*}$ denotes the complex conjugation of $\rho$ and $\sigma_{y}$ is
a Pauli matrix.

In our model, the concurrence of the state can be obtained from Equation (10),
which results in

$\mathcal{C}=2\mathrm{max}\left\\{\mid\rho_{2,3}\mid-\sqrt{\rho_{1,1}\rho_{4,4}},\mid\rho_{1,4}\mid-\mid\rho_{2,2}\mid,0\right\\}.$
(12)

###  II.2 Quantum Coherence

It is well known that quantum coherence is an important physical resource in
different quantum information processing tasks. An intuitive and computable
measure of quantum coherence is the $l_{1}$-norm of coherence l1norm1 . This
measure quantifies coherence by using the minimal distance between the quantum
state and the set of incoherent states. The $l_{1}$-norm coherence of the
quantum state $\rho=\sum_{i,j}\rho_{i,j}|i\rangle\langle j|$ is the sum of the
absolute values of all off-diagonal elements entries

$\mathcal{C}_{l_{1}}(\rho)=\sum_{i\neq j}|\rho_{i,j}|.$ (13)

Calculating the $l_{1}$-norm for the two gravitational cat states is
straightforward and results in

$\mathcal{C}_{l_{1}}(\rho)=2|\rho_{2,3}|+2|\rho_{1,4}|.$ (14)

In the next section, we review and discuss the main results.

##  III Results and Discussion

In what follows, we focus on the thermal quantum correlations of two
gravitational cat states with gravitational interaction between the states. In
what follows, unless the opposite is stated, we shall consider $k_{B}=1$.

In Figure 2, the concurrence $\mathcal{C}$ is shown as a function of
temperature $T$ in the logarithmic scale, at a fixed value of $\Delta=0.01$
and $w=0.1$ (green curve), $\Delta=0.3$ and $w=1.0$ (red curve), $\Delta=1.2$
and $w=2.0$ (blue curve), $\Delta=3.0$ and $w=3.0$ (black curve). Initially,
one can observe that, when $\Delta<w$, the concurrence is weak
$\mathcal{C}\approx 0.1$, (green curve), $\mathcal{C}\approx 0.28$ (red curve)
and $\mathcal{C}\approx 0.51$ (blue curve). In addition, it can be observed
that the concurrence decreases with the increasing of temperature $T$. On the
other hand, when we consider $w=\Delta$, the entanglement is more robust at
low temperatures $\mathcal{C}=\frac{\sqrt{2}}{2}$ (black curve). By analyzing
the matrix elements of the density operator, it is observed that the only
single non-zero state vector in this case is given by
$|\varphi_{3}\rangle=\frac{1}{\sqrt{2}\sqrt{2+\sqrt{2}}}\left(|11\rangle+(1+\sqrt{2})|00\rangle\right)$,
at $T=0$. These results are in accordance with expectations, since it is
showing that the larger the masses of the state for fixed distances
(determined by $\Delta$) and energy scale of the model (determined by $w$),
the more intense is the gravity-mediated entanglement, and a similar behavior
is found for smaller distances between the particles. However, most
interestingly, it shows that the threshold temperature, from which the
entanglement vanishes also increases with $\Delta$ and $w$, shall contribute
to describe the balance between the needed temperature, mass, distance and
energy scales in the type of experiment that aims to measure this effect,
which is fundamental for the study of the interplay between gravity and
quantum mechanics. However, we further discuss this issue in more realistic
scenarios that we address later on in the paper.

Figure 2: (Color online) Concurrence $\mathcal{C}$ as a function of
temperature $T$ in the logarithmic scale, for $w=0.1$ (green curve), $w=1.0$
(red curve), $w=2.0$ (blue curve), $w=3.0$ (black curve) and different values
of $\Delta$. It is assumed that $\Delta<w$ in each case.

In Figure 3, assuming $\Delta>w$, we observe a significant increase in the
concurrence. In fact, the concurrence reaches its maximum value when the
Newton potential parameter $\Delta$ significantly increases in comparison to
the energy parameter $w$. However, as the temperature rises, the concurrence
decreases until it finally disappears. This condition for maximizing the
concurrence can be useful for the planning of experiments, which although not
following exactly this kind of model, presents some similarities, from which
this analysis may be proved useful.

The $l_{1}$-norm of coherence $\mathcal{C}_{l_{1}}$ is an important quantity
to study quantum coherence. In order to investigate the relation of
entanglement and quantum coherence, here we consider the concurrence and
$l_{1}$-norm. In Figure 4, we illustrate the concurrence $\mathcal{C}$ and
$\mathcal{C}_{l_{1}}$ as a function of $T$ in the logarithmic scale for
$\Delta<w$. In particular here, we consider $w=1$ and
$\Delta=\\{0.01,0.1,0.2\\}$. We note that, for low temperatures, there is an
overlap between the $l_{1}$-norm of coherence $\mathcal{C}_{l_{1}}$ (dashed
curve) and the thermal entanglement (solid curve). This is due to the fact
that the quantum coherence and the thermal entanglement have the same order of
magnitude in the low temperature regime, i.e., the quantum coherence is
composed solely of non-local correlations, which is consistent with the
previous result tan . As the temperature increases, thermal fluctuations
generate an increasing of the quantum coherence, while the thermal
entanglement decreases until it vanishes at a threshold temperature; these
fluctuations raise non-entangled quantum correlations, i.e., quantum discord
tan . This jump is working as a siren that marks the threshold temperature
from which entanglement is no longer observable. On the other hand, it is
clear from Figure 4 that, when we increase the Newton potential $\Delta$, the
quantum correlations are more robust.

Figure 3: (Color online) Concurrence $\mathcal{C}$ as a function of $T$ in the
logarithmic scale, for $w=0.01$ (red curve), $w=0.1$ (blue curve), $w=1.0$
(green curve) and different values of $\Delta$. It is assumed that $\Delta>w$
in each case. Figure 4: (Color online) Concurrence $\mathcal{C}$ (solid curve)
and the quantum coherence $\mathcal{C}_{l_{1}}$ (dashed curve) as a function
of $T$ in the logarithmic scale for parameter set $\Delta=0.01$ (red curve),
$\Delta=0.1$ (blue curve) and $\Delta=0.2$ (green curve) and fixed value
$w=1.0$. In each case, it is assumed that $\Delta<w$.

In order to understand the behavior of the $l_{1}$-norm, we rewrite Equation
(14) as $\mathcal{C}_{l_{1}}=g_{1}+g_{2}$, where $g_{1}=2|\rho_{1,4}|$ and
$g_{2}=2|\rho_{2,3}|$. Note that the only term of $l_{1}$-norm that depends on
the ground state $\varepsilon_{3}$ is $g_{1}$. In Figure 5, we observed that
$\mathcal{C}_{l_{1}}\approx g_{1}$ for very low temperature, at $T\approx
0.03$, the term $g_{1}$ starts to decay, whereas the term $g_{2}$ is initially
null from $T=0$ to $T\approx 0.03$, so, by thermal fluctuations, it increases
at a rate greater than decay of the term $g_{1}$. This way, the quantum
coherence reaches the maximum ($\approx$0.233) at $T\approx 0.433$, and then
decreases monotonically. This shows that temperature can generate non-
entangled quantum correlations.

Figure 5:  (Color online) Quantum coherence $\mathcal{C}_{l_{1}}$ (green
curve), $g_{1}$ (blue curve), $g_{2}$ (red curve) as a function of $T$ in the
logarithmic scale for parameter set $w=1.0$ and $\Delta=0.2$. The magenta
vertical solid line indicates the threshold temperature from which thermal
fluctuations occur.

In Figure 6, another plot of $\mathcal{C}_{l_{1}}$ norm of coherence is shown
as a function of the temperature $T$ in the logarithmic scale. In this figure,
we compare the thermal quantum correlations in three cases, namely
$\Delta=\\{0.05,0.1,0.5\\}$. For all cases, we consider $\Delta=w$. This
condition means that the difference in the energy scale due to the
gravitational interaction is equal to the energy difference of each individual
two-level system. As can be seen, the thermal quantum coherence for low
temperatures is $\mathcal{C}=\mathcal{C}_{l_{1}}=\frac{\sqrt{2}}{2}$. As the
temperature increases, thermal fluctuations are introduced into the system;
this should generate an increment in non-entangled quantum correlations,
signaled by a growth in quantum coherence, which then shortly afterwards
decreases monotonically as the temperature increases.

Figure 6: (Color online) Concurrence $\mathcal{C}$ (solid curve) and the
quantum coherence $\mathcal{C}_{l_{1}}$ (dashed curve) as a function of $T$ in
the logarithmic scale for parameter set $w=0.05$ and $\Delta=0.05$ (red
curve), $w=0.1$ and $\Delta=0.1$ (blue curve), $w=0.5$ and $\Delta=0.5$ (green
curve). Here, it is assumed that $\Delta=w$.

In Figure 7, we compare the behavior of the concurrence and quantum coherence
versus the temperature in the logarithmic scale for different values of the
Newton potential $\Delta$ and assuming a fixed value of the energy parameter
$w=3.0$; here, we consider $\Delta>w$. It is easy to see a significant
increase in the thermal quantum correlations; the entanglement is
$\mathcal{C}\approx 1$ for $\Delta>w$ ($\mathcal{C}=1$ when $\Delta\gg w$) for
low temperatures. On the other hand, the $C_{l_{1}}$ coherence norm shows a
significant increase in the non-entangled quantum correlations regarding low
temperatures. However, we notice that, for the conditions assumed on the
determination of $w$ and hierarchy between the parameters $(\Delta,w)$, we see
that, although the concurrence starts fading at lower temperatures as $\Delta$
increases, the opposite behavior occurs for the coherence. We stress the
importance of achieving this kind of condition (energy scale of the
gravitational interaction higher than the energy scale of the individual two-
level system) in order to enhance the capability of detecting this phenomenon.

Figure 7: (Color online) Concurrence $\mathcal{C}$ (solid curve) and the
quantum coherence $\mathcal{C}_{l_{1}}$ (dashed curve) as a function of $T$ in
the logarithmic scale with fixed values $w=3.0$, $\Delta=100$, $\Delta=300$
and $\Delta=600$. It is assumed that $\Delta>w$ in each case.

In the following, we will study the thermal quantum correlations for the
parameters suggested in Marletto:2017kzi ; kris . First, we analyze the
thermal entanglement and quantum coherence, namely the $\mathcal{C}_{l_{1}}$
norm, with two mesoscopic test masses $m_{1}\approx m_{2}\approx
10^{-12}\,\text{kg}$ separated by $d=1.0\,\mu\text{m}$ (see Marletto:2017kzi )
and $L=\frac{d}{2}$. In what follows, we shall recover the Boltzmann constant
$k_{B}\approx 1.4\times 10^{-23}\,\text{J}\cdot\text{K}^{-1}$. In Figure 8, we
illustrate the thermal entanglement and quantum coherence as a function of the
temperature in the logarithmic scale with $\frac{w}{k_{B}}=0.015$ and
$\frac{\Delta}{k_{B}}=0.5101\times 10^{-6}$. From the figure, one can see that
the concurrence (red curve) is significantly weak $(\mathcal{C}\approx
3.4\times 10^{-5})$ at $T=0\,\text{K}$. Then, the thermal entanglement
monotonically decreases with increasing temperature until it rapidly drops to
the zero value at the threshold temperature $T_{th}\approx
0.0013658\,\text{K}$. On the other hand, we can see in the figure that the
quantum coherence (blue curve) is equal to the concurrence at the low
temperature region, that is, the quantum coherence captures the total thermal
entanglement. Soon afterwards, as the temperature increases, the thermal
fluctuations generate an increased quantum coherence, while the thermal
entanglement decreases until it vanishes at the threshold temperature.
Finally, the quantum coherence monotonously fades out. This scenario was also
shown in Figure 4, and the qualitative discussion that took place above also
applies here. However, quantitatively, we present the threshold temperature
for which the entanglement survives in this scenario, which might contribute
to the design of experiments to test this potential quantum aspect of gravity.

The second setup we consider was recently proposed by Krisnanda et al. kris .
Here, we consider the masses $m_{1}\approx m_{2}\approx 10^{-7}\,\text{kg}$
separated by $d=0.3\,\text{mm}$ and $L=\frac{d}{2}$. The configuration is
schematically depicted in Figure 9 with $\frac{\Delta}{k_{B}}=17.0072$,
$\frac{w}{k_{B}}=0.015$. Our results show that the curves for the concurrence
$\mathcal{C}$ and quantum coherence $\mathcal{C}_{l_{1}}$ behave similarly to
the Figure 8, but the concurrence in this case is $\mathcal{C}\approx
0.9999996$ at $T=0\,\text{K}$. However, with an increase of the temperature,
suddenly the concurrence decreases until it disappears close to the threshold
temperature at $T_{th}\approx 2.485053\,\text{K}$ (see Figure 9a).

Figure 8: (Color online) The plot shows the concurrence $\mathcal{C}$ (red
curve) and the quantum coherence $\mathcal{C}_{l_{1}}$ (blue curve) as a
function of $T$ in the logarithmic scale, assuming $\Delta/k_{B}=0.5101\times
10^{-6}$. This case lies within the $\Delta<w$ class.

Figure 9: (Color online) The plot shows the concurrence $\mathcal{C}$ (red
curve) and the quantum coherence $\mathcal{C}_{l_{1}}$ (blue curve) as a
function of $T$, assuming $\Delta/k_{B}=17.0072$. This case lies within the
$\Delta>w$ class.

To better understand the behavior of quantum coherence after the thermal
entanglement begins to decrease, we have plotted the entanglement and quantum
coherence vs. temperature in Figure 9b. We can see clearly from the figure
that the $l_{1}$-norm slightly increases due to the thermal fluctuations and
then decreases monotonically as soon as temperature $T$ increases. Here, we
verify a strong increment in initial quantum correlations due to the the
increase in the intensity of the gravitational interaction in comparison to
the previous case. This condition also allows one to have a higher threshold
temperature, from which entanglement drops. This also shows the difficulties
for experimentally verifying this phenomenon, since the temperature range in
which the systems remaining entangled depends significantly on their
geometric/gravitational parameters, whose increment constitutes a challenge by
themselves.

##  IV Conclusions

In this article, we have studied the effect of temperature produced by a
thermal bath in the entanglement and quantum coherence induced by the
gravitational interaction of a system of two massive particles confined in two
distinct double-well potentials.

We verified the decay of these quantum correlations with the growth of the
thermal bath’s temperature. For low temperatures, the quantum correlations
described by the quantum coherence coincide with the entanglement, i.e., all
quantum correlations are entangled in this regime; this is due to the form of
the thermal density operator of our model and is consistent with results found
in tan . Following the decay of the concurrence, the $l_{1}$-norm slightly
grows due to thermal fluctuations, which afterwards monotonically decays with
the growth of the temperature.

We have also analyzed the thermal effects on these quantifiers for the
parameters suggested in the setups of Marletto et al. Marletto:2017kzi  and
Krisnanda et al. kris . In the first setup, our results show that the
concurrence and $l_{1}$-norm of quantum coherence is particularly weak
assuming such parameters, but non-null, thus demonstrating the existence of
thermal entanglement between massive particles mediated by the Newtonian
interaction in a thermal bath model. In the second setup, the thermal
concurrence and $l_{1}$-norm of quantum coherence turn out to be robust
against decoherence for low temperatures. This is due to fact that the latter
case proposes states with masses that are five orders of magnitude larger than
the former. It is also interesting to verify that entanglement reaches almost
maximal values when $\Delta>w$ at low temperatures, which was also observed in
previous cases analyzed in this paper.

This analysis can impact modern searches for traces of the quantum nature of
the gravitational field using underground or tabletop experiments, since we
showed the presence of a characteristic threshold temperature from which the
system loses its entangled nature and could no longer be a laboratory for
witnessing the quantum nature of gravitational interaction. Besides that, we
verified conditions that enhance quantum correlations’ quantifiers: the
gravitational energy difference between the two cat states, $\Delta$, being
larger than the energy level difference of the particles individually, $w$.

Future explorations of this work could include further thermodynamic
applications of this system and the investigation of whether this behavior
persists in more complicated set ups.

## Acknowledgements

This work was partially supported by CNPq, CAPES, and FAPEMIG. M.R. would like
to thank the National Council for Scientific and Technological Development-
CNPq grant 317324/2021-7. I.P.L. was partially supported by the National
Council for Scientific and Technological Development–CNPq grant 306414/2020-1
and by the grant 3197/2021, Paraíba State Research Foundation (FAPESQ). I.P.L.
would like to acknowledge the contribution of the COST Action CA18108.

## References

  * (1) Kiefer, C. Quantum Gravity, 2nd ed.; Oxford University Press: Oxford, UK, 2007.
  * (2) Rovelli, C. Loop quantum gravity. Living Rev. Rel. 1998, 1, 1–75.
  * (3) Mukhi, S. String theory: A perspective over the last 25 years. Class. Quant. Grav. 2011, 28, 153001–153033.
  * (4) Ambjorn, J.; Jurkiewicz, J.; Loll, R. Causal Dynamical Triangulations and the Quest for Quantum Gravity. arXiv 2010, arXiv:1004.0352.
  * (5) Amelino-Camelia, G. Quantum-Spacetime Phenomenology. Living Rev. Rel. 2013, 16, 5–141
  * (6) Addazi, A.; Alvarez-Muniz, J.; Batista, R.A.; Amelino-Camelia, G.; Antonelli, V.; Arzano, M.; Asorey, M.; Atteia, J.L.; Bahamonde, S.; Bajardi, F.; et al. Quantum gravity phenomenology at the dawn of the multi-messenger era—A review. Prog. Part. Nucl. Phys. 2022, 125, 103948–104066.
  * (7) Acciari, V.A. [MAGIC, Armenian Consortium: ICRANet-Armenia at NAS RA, A. Alikhanyan National Laboratory and Finnish MAGIC Consortium: Finnish Centre of Astronomy with ESO], Bounds on Lorentz invariance violation from MAGIC observation of GRB 190114C. Phys. Rev. Lett. 2020, 125, 021301–021307.
  * (8) Addazi, A.; Marciano, A.; Yunes, N. Can we probe Planckian corrections at the horizon scale with gravitational waves? Phys. Rev. Lett. 2019, 122, 081301–081306.
  * (9) Arzano, M.; Kowalski-Glikman, J.; Wislicki, W. A bound on Planck-scale deformations of CPT from muon lifetime. Phys. Lett. B 2019, 794, 41–44.
  * (10) Lobo, I.P.; Pfeifer, C. Reaching the Planck scale with muon lifetime measurements. Phys. Rev. D 2021, 103, 106025–106032.
  * (11) Carney, D.; Stamp, P.C.E.; Taylor, J.M. Tabletop experiments for quantum gravity: A user’s manual. Class. Quant. Grav. 2019, 36, 034001–034034.
  * (12) Addazi, A.; Belli, P.; Bernabei, R.; Marciano, A. Testing noncommutative spacetimes and violations of the Pauli Exclusion Principle through underground experiments. Chin. Phys. C 2018, 42, 094001–094008.
  * (13) Marletto, C.; Vedral, V. Why we need to quantise everything, including gravity. NPJ Quantum Inf. 2017, 3, 29–33.
  * (14) Marletto, C.; Vedral, V. Gravitationally-induced entanglement between two massive particles is sufficient evidence of quantum effects in gravity. Phys. Rev. Lett. 2017, 119, 240402–240406.
  * (15) Bose, S.; Mazumdar, A.; Morley, G.W.; Ulbricht, H.; Toroš, M.; Paternostro, M.; Geraci, A.; Barker, P.; Kim, M.S.; Milburn, G. Spin Entanglement Witness for Quantum Gravity. Phys. Rev. Lett. 2017, 119, 240401–240406.
  * (16) Rovelli, C. Considerations on Quantum Gravity Phenomenology. Universe 2021, 7, 439-443.
  * (17) Bose, S.; Mazumdar, A.; Schut, M.; Toroš, M. Two mechanisms for quantum natured gravitons to entangle masses. Phys. Rev. D 2022, 105, 106028–106040.
  * (18) Christodoulou, M.; Rovelli, C. On the possibility of laboratory evidence for quantum superposition of geometries. Phys. Lett. B 2019, 792, 64–68.
  * (19) Christodoulou, M.; Rovelli, C. On the possibility of experimental detection of the discreteness of time. Front. Phys. 2020, 8, 207–211.
  * (20) Hall, M.J.W.; Reginatto, M. On two recent proposals for witnessing nonclassical gravity. J. Phys. A 2018, 51, 085303–085317.
  * (21) Reginatto, M.; Hall, M.J.W. Entanglement of quantum fields via classical gravity. arXiv 2018, arXiv:1809.04989.
  * (22) Anastopoulos, C.; Hu, B.L. Quantum Superposition of Two Gravitational Cat States. Class. Quant. Grav. 2020, 37, 235012–235035.
  * (23) Rijavec, S.; Carlesso, M.; Bassi, A.; Vedral, V.; Marletto, C. Decoherence effects in non-classicality tests of gravity. New J. Phys. 2021, 23, 043040–043054.
  * (24) Di Bartolomeo, G.; Carlesso, M.; Bassi, A. Gravity as a classical channel and its dissipative generalization. Phys. Rev. D 2021, 104, 104027–104037.
  * (25) Nguyen, H.C.; Bernards, F. Entanglement dynamics of two mesoscopic objects with gravitational interaction. Eur. Phys. J. D 2020, 74, 69-73.
  * (26) Miki, D.; Matsumura, A.; Yamamoto, K. Entanglement and decoherence of massive particles due to gravity. Phys. Rev. D 2021, 103, 026017–026028.
  * (27) S. Qvarfort, S. Bose and A. Serafini, Mesoscopic entanglement through central–potential interactions. J. Phys. B 2020, 53, 235501–235508.
  * (28) Bennett, C.H.; Brassard, G.; Crepeau, C.; Jozsa, R.; Peres, A.; Wootters, W.K. Teleporting an unknown quantum state via dual classical and Einstein-Podolsky-Rosen channels. Phys. Rev. Lett. 1993, 70, 1895–1899.
  * (29) Bennett, C.H.; Bernstein, H.J.; Popescu, S.; Schumacher, B. Concentrating partial entanglement by local operations. Phys. Rev. A 1996, 53, 2046–2052.
  * (30) Amico, L.; Fazio, R.; Osterloh, A.; Vedral, V. Entanglement in many-body systems. Rev. Mod. Phys. 2008, 80, 517–576.
  * (31) Li, K.; Li, Y.; Han, M.; Lu, S.; Zhou, J.; Ruan, D.; Long, G.; Wan, Y.; Lu, D.; Zeng, B.; et al. Quantum Spacetime on a Quantum Simulator. Comm. Phys. 2019, 2, 122–127.
  * (32) Czelusta, G.; Mielczarek, J. Quantum simulations of a qubit of space. Phys. Rev. D 2021, 103, 046001–046017.
  * (33) Streltsov, A.; Adesso, G.; Plenio, M.B. Quantum Coherence as a Resource. Rev. Mod. Phys. 2017, 89, 041003–041036.
  * (34) Baumgratz, T.; Cramer, M.; Plenio, M.B. Quantifying Coherence. Phys. Rev. Lett. 2014, 113, 140401–140405.
  * (35) Fröwis, F.; Dür, W. Stable macroscopic quantum superpositions. Phys. Rev. Lett. 2011, 106, 110402–110405.
  * (36) Giovannetti, V.; Lloyd, S.; Maccone, L. Quantum-enhanced measurements: Beating the standard quantum limit. Science 2004, 306, 1330–1336.
  * (37) Santos, J.P.; Céleri, L.C.; Landi, G.T.; Paternostro, M. The role of quantum coherence in non-equilibrium entropy production. Nat. Quant. Inf. 2019, 5, 23–29.
  * (38) Lostaglio, M.; Jennings, D.; Rudolph, T. Description of quantum coherence in thermodynamic processes requires constraints beyond free energy. Nat. Commun. 2015, 6, 6383–6391.
  * (39) Petruzziello, L.; Illuminati, F. Quantum gravitational decoherence from fluctuating minimal length and deformation parameter at the Planck scale. Nat. Commun. 2021, 12, 4449–4450.
  * (40) Arzano, M.; D’Esposito, V.; Gubitosi, G. Fundamental decoherence from quantum spacetime. arXiv 2022, arXiv:2208.14119.
  * (41) Glauber, R.J. Coherent and incoherent states of the radiation field. Phys. Rev. 1963, 131, 2766–2788.
  * (42) Sudarshan, E.C.G. Equivalence of semiclassical and quantum mechanical descriptions of statistical light beams. Phys. Rev. Lett. 1963, 10, 277–279.
  * (43) Streltsov, A.; Singh, U.; Dhar, H.S.; Bera, M.N.; Adesso, G. Measuring Quantum Coherence with Entanglement. Phys. Rev. Lett. 2015, 115, 020403–020408.
  * (44) Kammerlander, P.; Anders, J. Coherence and measurement in quantum thermodynamics. Sci. Rep. 2016, 6, 22174–22180.
  * (45) Zhang, K.; Wang, X.; Zeng, Q.; Wang, J. Conditional Entropy Production and Quantum Fluctuation Theorem of Dissipative Information: Theory and Experiments. PRX Quantum 2022, 3, 030315–22131.
  * (46) Filgueiras, C.; Rojas, O.; Rojas, M. Thermal Entanglement and Correlated Coherence in Two Coupled Double Quantum Dots Systems. Ann. Der Phys. 2020, 532, 2000207–2000213. Berrada, K. Quantum coherence in quantum dot systems. Phys. E 2020, 116, 113784–113788.
  * (47) Sha, Y.-T.; Wang, Y.; Sun, Z.-H.; Hou, X.-W. Thermal quantum coherence and correlation in extended $XY$ spin chain. Ann. Phys. 2018, 392, 229–241. Freitas, M.; Filgueiras, C.; Rojas, M.; The effects of an impurity in an Ising-$XXZ$ diamond chain on thermal entanglement, on quantum coherence and on quantum teleportation. Ann. Der Phys. 2019, 531, 1900261–1900270. Arian zad, H.; Rojas, M. Quantum coherence, quantum Fisher information and teleportation in the Ising-Heisenberg spin chain model of a heterotrimetallic Fe-Mn-Cu coordination polymer with magnetic impurity. Phys. -Low-Dimens. Syst. Nanostruct. 2021, 126, 114455.
  * (48) Ullah, K.; Köse, E.; Yagan, R.; Onbasli, M. C.; Müstecaplioglu O. E. Steady state entanglement of distant nitrogen-vacancy centers in a coherent thermal magnon bath. Phys. Rev. Res. 2022, 4, 023221–023243.
  * (49) Shi, H.-L.; Ding, S.; Wan, Q.-K.; Wang, X.-H.; Yang, W.-L. Entanglement, Coherence, and Extractable Work in Quantum Batteries. Phys. Rev. Lett. 2022, 129, 130602–130607.
  * (50) Ahnefeld, F.; Theurer, T.; Egloff, D.; Matera, J.M.; Plenio, M.B. Coherence as a Resource for Shor’s Algorithm. Phys. Rev. Lett. 2022, 129, 120501–120507.
  * (51) Sk, R.; Panigrahi, P.K. Protecting quantum coherence and entanglement in a correlated environment. Phys. A 2022, 596, 127129–140441.
  * (52) Schlosshauer, M. Decoherence and the Quantum to Classical Transition, 1st ed.; Springer: Berlin, Germany, 2007.
  * (53) Breuer, H.-P.; Petruccione, F. The Theory of Open Quantum Systems, 1ST ed.; Oxford University Press: New York, NY, USA, 2003.
  * (54) Carlesso, M.; Bassi, A.; Paternostro, M.; Ulbricht, H. Testing the gravitational field generated by a quantum superposition. New J. Phys. 2019, 21, 093052–093060.
  * (55) Wootters, W.K. Entanglement of Formation of an Arbitrary State of Two Qubits. Phys. Rev. Lett. 1998, 80, 2245–2248.
  * (56) Hill, S.; Wootters, W.K. Entanglement of a Pair of Quantum Bits. Phys. Rev. Lett. 1997, 78, 5022–5025.
  * (57) Tan, K.C.; Kwon, H.; Park, C.-Y.; Jeong, H. Unified view of quantum correlations and quantum coherence. Phys. Rev. A 2016, 94, 022329–022336.
  * (58) Krisnanda, T.; Tham, G.Y.; Paternostro, M.; Paterek, P. Observable quantum entanglement due to gravity. npj Quantum Inf. 2020, 6, 12–17.


# The Prevalence of Code Smells in Machine Learning projects

Bart van Oort1,2, Luís Cruz2, Maurício Aniche2, Arie van Deursen2 Delft
University of Technology
1 AI for Fintech Research, ING
2 Delft, Netherlands
<EMAIL_ADDRESS>{l.cruz, m.f.aniche<EMAIL_ADDRESS>
###### Abstract

Artificial Intelligence (AI) and Machine Learning (ML) are pervasive in the
current computer science landscape. Yet, there still exists a lack of software
engineering experience and best practices in this field. One such best
practice, static code analysis, can be used to find code smells, i.e.,
(potential) defects in the source code, refactoring opportunities, and
violations of common coding standards. Our research set out to discover the
most prevalent code smells in ML projects. We gathered a dataset of 74 open-
source ML projects, installed their dependencies and ran Pylint on them. This
resulted in a top 20 of all detected code smells, per category. Manual
analysis of these smells mainly showed that code duplication is widespread and
that the PEP8 convention for identifier naming style may not always be
applicable to ML code due to its resemblance with mathematical notation. More
interestingly, however, we found several major obstructions to the
maintainability and reproducibility of ML projects, primarily related to the
dependency management of Python projects. We also found that Pylint cannot
reliably check for correct usage of imported dependencies, including prominent
ML libraries such as PyTorch.

###### Index Terms:

Artificial Intelligence, Machine Learning, static code analysis, code smells,
Python, dependency management.

##  I Introduction

Artificial Intelligence (AI) and Machine Learning (ML) are pervasive in the
current landscape of computer science. Companies such as Facebook, Google,
Nvidia and ING are making use of AI and ML for a plethora of tasks that are
difficult (if not impossible) to describe using traditional Software
Engineering (SE) [1, 2, 3, 4]. Examples include facial recognition &
recomposition, natural language processing, real-time video transformation,
detection of medical anomalies and intercepting fraudulent financial
transactions.

Yet, as Sculley et al. [2] wrote in their 2015 paper on the hidden technical
debt in ML systems at Google, “only a small fraction of real-world ML systems
is composed of the ML code. (…) The required surrounding infrastructure is
vast and complex.” This is also in part what leads Menzies [5] to predict that
the future of software will be a rich and powerful mix of ideas from both SE
and AI. Menzies also advocates for more SE experience in the field of AI and
ML, stating that poor SE leads to poor AI while better SE leads to better AI
[5]. The data scientists that write AI / ML code often come from non-SE
backgrounds where SE best practices are unknown [6].

One such SE best practice is the practice of static code analysis to find
(potential) defects in the source code, refactoring opportunities and
violations of common coding standards, which we amalgamate into ‘code smells’
for the rest of this paper. Research has shown that the attributes of quality
most affected by code smells are maintainability, understandability and
complexity, and that early detection of code smells reduces the cost of
maintenance [7].

With a focus on the maintainability and reproducibility of ML projects, the
goal of our research is therefore to apply static code analysis to
applications of ML, in an attempt to uncover the frequency of code smells in
these projects and list the most prevalent code smells. Thus, we formulate the
following research question: What are the most prevalent code smells in
Machine Learning code?

The main contributions of this paper are:

  * •

An empirical study on the prevalence of code smells in 74 Python ML projects.

  * •

A dataset of 74 ML projects and an open-source tool to perform simultaneous
static code analysis on all of these projects.

##  II Related Work

Several studies have investigated linting and static code analysis of non-ML
projects [8, 9, 10, 11]. Tómasdóttir et al. [8] researched why JavaScript (JS)
developers use linters and how they tend to configure them. They found that
maintaining code consistency, preventing errors, saving discussion time and
avoiding complex code were among the top reasons why JS developers use
linters. They also found that JS developers commonly stick with already
existing preset linting configurations. Vassallo et al. [12] found a similar
result; among other results, they found that developers are often unwilling to
configure automatic static analysis tools (ASATs) and emphasise “the necessity
to improve existing strategies for the selection and prioritisation of ASATs
warnings that are shown to developers.”

Within the Python ecosystem, Chen et al. [11] investigated the detection and
prevalence of code smells in 106 Python projects with the most stars on
GitHub. They found that long parameter lists and long methods were more
prevalent than other code smells. Omari and Martinez [9] used Pylint to
analyse the code quality of a dataset of large Python projects. Furthermore,
Bafatakis et al. [10] used Pylint to investigate the Python coding style
compliance of StackOverflow answers.

Within the Machine Learning ecosystem, we only found one paper by Simmons et
al. [6] that performed static code analysis on a large dataset of Data Science
(DS) projects. They also analysed non-DS projects with the goal of comparing
the code quality and coding standard conformance of (open-source) DS projects
versus non-DS projects, using Pylint in its default configuration as a metric.
They sourced their DS projects from Biswas et al. [13], who in 2019 published
a dataset of 1558 “mature Github projects that develop Python software for
Data Science tasks.”. Aside from applications of ML, it also includes ML
libraries and tools.

Our study differs from [6] in that we do not compare against non-DS projects
and in that we do not solely focus on the adherence to coding standards as [6]
does. Our primary focus lies more on investigating obstructions to the
maintainability and reproducibility of ML projects, which includes coding
standards violations, but also entails recognising refactoring opportunities
and other code smells [7]. Moreover, we solely focus on applications of ML,
and leave ML libraries and tools out of scope. We argue that the underlying
nature of ML libraries and tools is very different from ML applications, and
thus different results are expected when studied separately.

Furthermore, Simmons et al. [6] simplified the installation of the projects’
dependencies by using https://pypi.org/project/findimports/ to resolve all
imports used in the projects, instead of relying on what projects’ authors
defined in their repositories, noting that “it was impractical to reliably
determine and install dependencies for the projects analysed.” However, if
there is an inherent difficulty in resolving these dependencies within Python
projects, then that is in itself an obstruction to the reproducibility and
maintainability of these projects. Hence, we investigate this in our study.

##  III Methodology

For this paper, we performed an empirical study on the prevalence of code
smells in ML code. We collected a dataset of 74 ML projects and implemented a
tool to set these projects up with their dependencies in order to replicate
their execution environment. It then runs Pylint with its default
configuration on all projects in the dataset, collecting and counting the
detected code smells. The tool and dataset are both open-source and can be
found on GitLab111https://gitlab.com/bvobart/python-ml-analysis. Our empirical
study follows the methodology illustrated in Figure 1. It comprises three main
steps, namely: A) project selection, B) setting up the codebases, and C)
static analysis.

Figure 1: Methodology Diagram.

###  III-A Project Selection

In total, our collected dataset comprises 74 ML projects; 32 projects come
from finished Kaggle competitions, 38 from paperswithcode.com (of which 25
projects were from the Google-affiliated DeepMind), and 4 from
reproducedpapers.org. It includes projects from academic papers, (student)
reproductions, prize money awarding Kaggle competitions, as well as industry
players such as Facebook, Nvidia and DeepMind. The dataset defines a list of
Git repository URLs and allows for customising the dependencies of particular
projects, when they have not been properly defined in their respective
repository. We elaborate on a number of characteristics of our dataset in
Section III-A, but first, we explain how we collected the projects in the
dataset and what guidelines were used for doing so. We aim for this dataset to
be a systematically gathered set of projects, representative of the current,
real-world state of ML and AI projects. To this end, we have created a set of
guidelines for the inclusion of projects in the dataset, which can be found
below. Each project included in the dataset…

  1. 1.

…must be hosted in an open-source Git repository.

  2. 2.

…must be written in Python 3.

  3. 3.

…must contain pure Python code and does not consist purely of Jupyter
Notebooks. More specifically, a project should contain either a) at least 200
lines of pure Python code, even if the rest of the code is embedded in Jupyter
Notebooks, or b) more lines of pure Python code than there are lines of Python
code in all Jupyter notebooks of that project.

  4. 4.

…must implement an ML or AI model and may not be a library or tool for use in
ML projects.

  5. 5.

…must be considered ‘deliverable’, i.e., either a) the project is part of or
accompanies a published academic paper, or b) the project has been submitted
to paperswithcode.com, reproducedpapers.org or a Kaggle competition (which has
finished and declared the winners at the time of considering the project).

The first guideline limits our scope to open-source projects, as these are
openly available to download and analyse. The second and third guideline stem
from a technical limitations, as Pylint only supports Python 3 and is only
able to analyse pure Python files. Jupyter Notebooks are essentially JSON
files, containing ‘cells’ with code in Markdown, Python, Julia, or a small
selection of other different languages. While it is technically possible to
convert the Python code embedded in these notebooks to pure Python files using
a tool such as pointlesss-statement, wrong-import-statement produce false
positives in notebook-style code. Due to the lack of direct Pylint support for
Jupyter Notebooks and since we do not want to selectively disable Pylint rules
for notebook-extracted code as opposed to pure Python code, we decided to
exclude projects that purely contain Jupyter Notebooks from our dataset. The
minimum of 200 lines of pure Python code in the presence of larger Jupyter
Notebooks was chosen such that this code is likely not to be purely utility
code, but also contain part of the ML code. The fourth guideline embodies that
we are interested in analysing applications of ML rather than libraries used
in their development, such as pandas, or Dataset characteristics We measured
several general characteristics of every project, which can be found in Table
I. The 74 projects in the dataset contain a total of 3156 pure Python files,
amounting to 511.018 lines of Python code, including empty lines. The median
project has 17 pure Python files with 2848.5 lines of code, resulting in an
average of 157.3 lines of Python code per file. The smallest project contained
a single file with 58 lines of Python, while the largest project had 78572
lines of Python code across 229 files. This project was found to be embedding
the code of several dependencies in its repository, multiple times.

TABLE I: Characteristics of our dataset of 74 ML projects Characteristic | Min | Q1 | Median | Q3 | Max | Mean | Std Dev.
---|---|---|---|---|---|---|---
Number of pure Python files | 1 | 8 | 17 | 36 | 730 | 43 | 95
Number of Jupyter Notebook files | 0 | 0 | 0 | 1 | 52 | 3 | 8
Lines of Python code | 58 | 1449 | 2849 | 5243 | 78572 | 6906 | 13568
Lines of Jupyter Notebook Python | 0 | 0 | 0 | 197 | 43387 | 1008 | 5115
Avg. lines of Python code per Python file | 58 | 108 | 157 | 214 | 1151 | 193 | 155
Avg. lines of Jupyter Notebook Python per Jupyter Notebook file | 12 | 73 | 129 | 223 | 1610 | 256 | 335

###  III-B Setting up the codebases

In this step, performed by our analysis tool, for each project, we clone the
latest version of the project’s Git repository, create a virtual environment
in it, ensure that there exists a file that specifies any necessary
dependencies and then install those dependencies into the virtual environment.
These need to be installed, such that our static analysis tool of choice is
able to check whether imports resolve correctly and whether the imported
libraries are used correctly. This is particularly interesting in the case of
ML, as Sculley et al. [2] noted that ML projects have a high degree of glue
code and so make extensive use of libraries. In total, the folder containing
all of the 74 cloned projects from our dataset along with the accompanying
virtual environments with their installed dependencies, amounts to 131 GiB.
Python projects can specify and install their dependencies in a variety of
ways. The most common way to install a Python dependency is to use
requirements.txt, which is conventionally placed at the root of the project’s
source code repository. It is also possible to specify a pip’s default package
index. setup.py. However, there are also other package managers / dependency
management solutions such as Conda, Poetry, python-packaging-guide. These
tools each have their own way of specifying dependencies and – especially in
Conda’s case – may use additional package indexes to PyPI, which makes
resolving these dependencies difficult. It is possible to use requirements.txt
file. This approach is flawed though, as we explain in Section V-B. Our
analysis tool currently only supports installing dependencies with
requirements.txt or requirements.txt file, or alternatively, the contents of a
custom after installing the dependencies from the requirements file. This is
necessary for, e.g., Nvidia’s Apex library, which depends on PyTorch; when
trying to run requirements.txt file, nor a manually defined one, our analysis
tool uses https://github.com/bndr/pipreqs to generate a environment.yml file,
but no setup.py. No projects that we came across were using Pipenv.

###  III-C Static Analysis

This step is also performed by our analysis tool and concerns running the
static code analysis tool Pylint (version 2.6.0) in its default configuration
on all pure Python files in each project (but not on any of the dependencies).
We choose Pylint for static code analysis as it is widely used and widely
accepted in the Python community, as well as being highly configurable [6,
10]. It is also well integrated into IDEs such as PyCharm and VS Code.
Furthermore, Bafatakis et al. [10] used it to measure coding style compliance
in StackOverflow answers, Omari and Martinez [9] used it as a metric for the
code quality of open-source Python projects, and Simmons et al. [6] used it in
their code quality comparison between DS and non-DS Python projects. Pylint
provides an extensive set of messages, not only for stylistic issues, but also
for issues regarding programming conventions, possible refactorings and other
logical code smells. While Pylint is very configurable, we chose to use
Pylint’s default configuration as it reflects the community standards, similar
to Simmons et al. [6]. The code smells that Pylint reports are each identified
by a symbol, such as import-error, which is also how we refer to specific
Pylint messages in this paper. Furthermore, these messages are divided into
five categories (message types), which we describe below. The italic text is
how Pylint describes the category.

  * •

Convention – for programming standard violation – Messages in this category
show violations of primarily code style conventions, as well as documentation
conventions and Pythonic programming conventions.

  * •

Refactor – for bad code smell – Messages in this category indicate that the
smelly code should be refactored.

  * •

Warning – for Python specific problems – This category includes many generic
and Python-specific linting messages.

  * •

Error – for probable bugs in the code – Messages in this category indicate
problems in the code that are very likely to cause run-time problems.

  * •

Fatal – if an error occurred which prevented Pylint from doing further
processing.

Cloning and installing all projects, even though this is performed
automatically by the tool, is the most time-consuming part of the analysis –
it takes roughly three hours. With all projects already cloned and their
dependencies already installed, the analysis of all 74 projects took 8m 53s
using 12 threads on an Intel® Core™ i7-8750H processor. Eight projects
contained code that caused Pylint to crash during analysis, so we excluded
these from the 82 projects we originally had in the dataset, bringing the
total to 74. Several issues have been filed about this, including one by this
paper’s first author222https://github.com/PyCQA/pylint/issues/3986. This bug
has since been fixed.

##  IV Results

Applying our methodology, we collected, installed, and analysed 74 ML
projects. In this section, we present our results and answer to the research
question posed in the introduction:

  * •

RQ – What are the most prevalent code smells in Machine Learning code?

To answer this, we first analysed the distribution of the amount of code
smells per Pylint category per project, of which the characteristics can be
found in Table II. The table shows the minimum, maximum, mean and median
number of messages reported by Pylint for each category, as well as the 25th
percentile (Q1), 75th percentile (Q3), and standard deviation (Std. Dev.). We
use the median as the main measure of central tendency. Our results show that
Pylint messages in the Warning category are the most prevalent – the median
project has 356 warnings – closely followed by messages in the Convention
category with 226 messages for the median project. Messages in the Refactor
and Error categories are less prevalent; respectively 49 and 56 such messages
for the median project. However, especially given the Error category is meant
for messages that show “probable bugs”, this is an interesting observation.
Even more interesting, there was no project for which Pylint reported no error
messages.

TABLE II: Distribution of Pylint messages per category per project.

Category | Min | Q1 | Median | Q3 | Max | Mean | Std Dev.
---|---|---|---|---|---|---|---
Convention | 2 | 57 | 226 | 799 | 9501 | 708 | 1361
Refactor | 0 | 26 | 49 | 140 | 2437 | 183 | 433
Warning | 11 | 74 | 356 | 824 | 14263 | 814 | 1826
Error | 1 | 18 | 56 | 125 | 1696 | 129 | 234
Fatal | 0 | 0 | 0 | 0 | 0 | 0 | 0

As a more direct answer to this research question, we measured across all
projects in our dataset what the top 20 code smells per category are that
Pylint reported, see Table IV. The top 10 messages that Pylint reported,
disregarding category, are in Table III.

TABLE III: Top 10 code smells overall as detected by Pylint. # | Smell | Frequency |
---|---|---|---
1 | unused-wildcard-import | 26307 |
2 | bad-indentation | 19921 |
3 | invalid-name | 19905 |
4 | line-too-long | 10321 |
5 | missing-function-docstring | 6444 |
6 | no-member | 5860 |
7 | duplicate-code | 4649 |
8 | trailing-whitespace | 4477 |
9 | redefined-outer-name | 2548 |
10 | missing-module-docstring | 2504 |
TABLE IV: Top 20 messages per category reported by Pylint along with how often
they were counted across all projects.

| Convention | Refactor | Warning | Error
---|---|---|---|---
| Symbol | Count | Symbol | Count | Symbol | Count | Symbol | Count
1 | invalid-name | 19905 | duplicate-code | 4649 | unused-wildcard-import | 26307 | no-member | 5860
2 | line-too-long | 10321 | too-many-arguments | 2158 | bad-indentation | 19921 | import-error | 1750
3 | missing-function-docstring | 6444 | super-with-arguments | 1802 | redefined-outer-name | 2548 | undefined-variable | 471
4 | trailing-whitespace | 4477 | too-many-locals | 1456 | unused-import | 2321 | not-callable | 397
5 | missing-module-docstring | 2504 | too-many-instance-attributes | 658 | arguments-differ | 1678 | no-name-in-module | 326
6 | wrong-import-position | 2286 | no-else-return | 509 | unused-variable | 986 | no-value-for-parameter | 168
7 | missing-class-docstring | 2060 | too-few-public-methods | 438 | attribute-defined-outside-init | 962 | function-redefined | 100
8 | wrong-import-order | 1750 | no-self-use | 422 | unused-argument | 902 | unsubscriptable-object | 100
9 | ungrouped-imports | 367 | useless-object-inheritance | 351 | abstract-method | 841 | bad-option-value | 94
10 | import-outside-toplevel | 285 | too-many-statements | 265 | redefined-builtin | 536 | unexpected-keyword-arg | 84
11 | consider-using-enumerate | 256 | too-many-branches | 218 | dangerous-default-value | 447 | relative-beyond-top-level | 68
12 | missing-docstring | 246 | cyclic-import | 108 | reimported | 322 | assignment-from-no-return | 27
13 | superfluous-parens | 244 | inconsistent-return-statements | 71 | wildcard-import | 297 | bad-super-call | 21
14 | missing-final-newline | 236 | unnecessary-comprehension | 48 | logging-format-interpolation | 288 | redundant-keyword-arg | 19
15 | multiple-statements | 218 | chained-comparison | 46 | pointless-statement | 253 | too-many-function-args | 18
16 | trailing-newlines | 176 | consider-using-in | 45 | fixme | 247 | invalid-unary-operand-type | 17
17 | bad-whitespace | 166 | simplifiable-if-expression | 34 | protected-access | 223 | no-self-argument | 9
18 | unidiomatic-typecheck | 78 | literal-comparison | 30 | logging-fstring-interpolation | 110 | misplaced-bare-raise | 9
19 | singleton-comparison | 69 | too-many-nested-blocks | 26 | f-string-without-interpolation | 105 | no-method-argument | 6
20 | multiple-imports | 53 | no-else-raise | 24 | pointless-string-statement | 101 | access-member-before-definition | 6

Convention – In this category we found that invalid naming, missing
documentation (missing-module-docstring, missing-docstring) and improper
organisation of imports (wrong-import-order, import-outside-toplevel and
Refactor – The most commonly recognised opportunities for refactoring
pertained to duplicate code (4649), using too many arguments when defining a
function or method (2158, super in the constructor of an inheriting class
(1802, super are necessary. It also shows that functions and classes are often
too complex; Pylint reports 1456 functions that use too many local variables
(too-many-statements) and 218 that have too many branches, as well as 658
classes that have too many attributes on them (Warning – The most reported
Warning messages, by far, are bad-indentation (19921). Code smells relating to
import management, as already indicated in the Convention category, are also
reflected in the Warning category with 26307 counts of unused wildcard
imports, 2321 counts of unused imports, 322 counts of libraries that were
imported multiple times in the same file (redefined-outer-scope) is also
common with 2548 recognised cases, as is redefining Python’s built-in global
names (536, Error – Finally, in the Error category, with 5860 counts, the no-
name-in-module messages are also related to these import problems, as they are
emitted upon using a X is resolved (so no import error is emitted), but None
of the projects have 0 errors. We need to dig into this, otherwise reviewers
will say that our dataset is unlikely to be representative. I wrote this
without reading the discussion, but we must have a compelling explanation for
this result.

##  V Implications

In this section, we discuss the implications of our results for ML developers.
We start by elaborating upon the code smells that we have found to be most
prevalent and continue with a discussion of problems regarding dependency
management that we encountered while performing this research. We also argue
how these problems affect the maintainability and reproducibility of the
analysed ML projects.

###  V-A Explaining the most prevalent code smells

This section aims at providing an explanation for the prevalence of the most
common code smells in our dataset of ML projects by investigating their
occurrences. Error – Most interestingly, we found that there were zero
projects that had zero messages in the Error category. Only the
https://github.com/deepmind/deepmind-research had one error, namely a true
positive no-member and itemize[leftmargin=*] Bad specification of requirements
– Using the Pylint producing false positives on local imports – Taking the
https://github.com/yandex-research/navigan project as an example, even though
we manually fixed the import errors relating to badly specified requirements,
there were 28 errors remaining. These errors come from unresolved imports from
local modules, i.e., Python files in the repository. In Python, a local module
import utils and it is recommended (but not necessary) to add an python-docs-
initpy. However, as a GitHub issue
reports333https://github.com/PyCQA/pylint/issues/3984, Pylint produces false
positive import errors on local imports, but strangely not when the not
present. As for kaggle-kuzushiji-recognition-2019 project – which has 327 of
them – these were primarily caused by false positives from Pylint on the
majority of – if not all – usages of the torch.as_tensor, torch.max. The
project with the most RSNA-STR-Pulmonary-Embolism-Detection (1549), showed the
same trend, as did https://github.com/PyCQA/pylint/issues/{3510, 2708, 2067}.
of which the essence goes back as far as 2013 with a similar problem in the
use of NumPy444https://github.com/PyCQA/pylint/issues/58.. The reason that is
stated in these issues, is that Pylint has trouble extracting the members and
type information of libraries that are backed by bindings with the C
programming language.

_Pylint cannot reliably check for correct uses of import statements; both
local imports, as well as imports from C-backed libraries such as PyTorch,
suffer from a high rate of false positives._

This is especially concerning in the context of ML, as the majority of ML
libraries are backed by C (to make them performant). The fix that the Pylint
developers propose in the relevant GitHub issues all entail (partially)
disabling the sculley2015hidden. Additionally, the fact that Pylint fails to
reliably analyse the usage of prominent ML libraries, provides a major
obstacle to the adoption of Continuous Integration (CI) in the development
environment of ML systems. If a static code analysis tool produces too many
false positives, it will be noisy and counterproductive [12]. Thus, other
important true positives may be overlooked.

_Additionally, the fact that Pylint fails to reliably analyse whether
prominent ML libraries are used correctly, provides a major obstacle to the
adoption of Continuous Integration (CI) in the development environment of ML
systems._

Warning – In this category, we found that one project (unused-wildcard-import
messages, with 53 unused-wildcard-import are emitted per unused function
imported with a wildcard import, this means that there were on average 263
unused imports per wildcard import in this project. Notably, most of these
messages were also (contained in) instances of duplicate code. Such unused
wildcard imports pollute a module’s namespace with the names of all imported
functions, meaning there is a greater chance of (accidentally) redefining an
outer name. Additionally, wildcard imports may also have unintended side-
effects that can be very difficult to debug. The tendency towards using
wildcard imports may stem from the prototypical and experimental nature of ML
projects, combined with the fact that it is simply easier for the developer to
import everything from a library and use whatever they need, rather than
import functions individually. Dead experimental codepaths as found by Sculley
et al. [2] of which the imports still remain, can also be a cause of bad
import management. As for the many https://www.python.org/dev/peps/pep-0008/
prescribes four spaces, but others such as Google’s TensorFlow style
guide555https://www.tensorflow.org/community/contribute/code_style prescribe
two spaces. Refactor – We found that Code duplication is common in ML, but
calls for more extensive research to truly understand to what extent and for
what reasons this occurs, and how it can be avoided. Regarding the high
prevalence of the too-many-locals messages, it is congruent with previous work
which shows that data science projects contain significantly more instances of
these than traditional software projects [6]. Simmons et al. [6] also notes
that these messages are related: function arguments namely also count as local
function variables. By default, a too-many-locals is emitted when a function
or method contains more than 15 local variables. A possible cause for their
prevalence, as Simmons et al. [6] note, are “models with multiple
hyperparameters that are either hard-coded as variables in the function
definition or passed to the function as individual parameters rather than
being stored in a configuration object.” Convention – While line length
violations stem primarily from developer preference, invalid and improper
naming is a problem not exclusive to Python [7]. Pylint by default emits a
hofmeister2019shortids, but Simmons et al. [6] aptly notes that this may not
necessarily be the case in DS and ML code due to its heavily mathematical
background. Thus, ML practitioners may find it easier to comprehend the
details of a piece of ML code written so that it resembles the notation of the
underlying mathematical model, including the names of the identifiers. Future
research on this subject will have to show how this affects the readability of
ML code.

_The PEP8 convention for identifier naming style may not always be applicable
in ML code due to its resemblance with mathematical notation. Future research
is required to investigate how this affects the readability of ML code._

###  V-B Problems installing project dependencies

Setting up the projects’ codebases as detailed in Section III-B was not a
trivial task. While 42 out of 74 projects did have a requirements file in
their repository that installed without a hitch out of the box, there were 32
projects where a yolact_edge project. Secondly, some projects had simply made
a small mistake in their manual maintenance of the requirements file, as was
the case with the https://github.com/yandex-research/navigan/issues/1 about
it. Thirdly, some projects were relying on custom Docker containers for their
runtime environment. These projects, e.g. requirements.txt file that was
likely the result of running itemize[leftmargin=*] Difficult to maintain –
Since all direct, indirect, runtime and development dependencies, without
distinction, in alphabetical order, we conjecture that it is difficult for
maintainers to assess whether a certain dependency can safely be upgraded
without breaking their code or breaking any of their dependencies. May result
in unresolvable dependencies – The resulting requirements file may contain
dependencies sourced from different dependency management tools and package
indexes. These dependencies may have slightly different package names across
package indexes or have only published certain versions to e.g. Conda’s
package index, but not to PyPI. There were also projects that depended on pre-
release versions of certain libraries that are no longer available on PyPI
(e.g., older nightly versions of Tensorflow packages). May include unrelated
dependencies – Especially if the user is not installing their dependencies
into a virtual environment, then the resulting requirements file may also
include unnecessary, unrelated (and potentially unresolvable) Python
dependencies, such as those used by their operating system or those used in
other projects. For example, the youtube-dl (even though the project has
nothing to do with videos), as well as some dependencies from the operating
system level such as python-debian and sudo pip install to install all of
their Python dependencies.

_We have found serious issues with the specification of dependencies that
present a major threat to the reproducibility and maintainability of Python ML
projects. Further research needs to be undertaken to help ML practitioners
avoid issues in the dependency management of their projects._

##  VI Threats to Validity

###  VI-A Validity of the dataset

Our dataset may not yet be fully representative of the real-world state of ML
code, as it currently only contains open-source ML projects, Therefore, in
future research, we want to collect a dataset of closed-source ML projects
from the industry, such as ING’s AI-driven FinTech industry. We will use this
both to compare the prevalence of code smells in these closed-source industry
projects with that of open-source projects as presented in this paper, as well
as to make our dataset more representative of the real-world state of ML and
AI projects. We will also explore adding projects from the dataset published
by Biswas et al. [13]. Furthermore, we currently do not perform any analysis
on the code quality of Jupyter Notebooks, even though they are very popular
and have emerged as a de facto standard for data scientists [17]. This was
deliberate, as Pylint currently does not support directly analysing the Python
code in Jupyter Notebook files and we wanted to avoid applying double
standards to pure Python code and notebook Python code by extracting the
notebook code into pure Python files. However, given their popularity, we do
intend to perform future research on the code quality and linting of Jupyter
Notebook code.

###  VI-B Validity of Pylint

Due to its dynamically typed nature, linting Python code is notoriously
difficult [6, 11]. It is therefore no surprise that Pylint contains bugs and
limitations that cause false positives and false negatives. Pylint’s issue
tracker on GitHub also reports 165 open and 501 closed issues regarding false
positives666See
https://github.com/PyCQA/pylint/issues?q=is%3Aissue+false+positive as of
January 19th 2021. We have also noticed some of these shortcomings for
ourselves during this research, as we have discussed in Section V-A. We
mitigate this threat by manually checking a subset of projects to analyse
potential false positives.

##  VII Conclusion

In this study we investigated the prevalence of code smells in ML projects. We
gathered a dataset of 74 ML projects, ran the static analysis tool Pylint on
them and collected the distribution of Pylint messages per category per
project (Table II), the top 10 code smells in these projects overall (Table
III), and the top 20 code smells per category (Table IV). Moreover, by
performing a manual analysis of a subset of the detected smells, we have found
that code duplication is common in ML, but does require further research to
understand to what extent this occurs and how it can be avoided. We also found
that the PEP8 convention for identifier naming style may not always be
applicable in ML code due to its resemblance with mathematical notation. This
calls for additional research on how it affects the readability of ML code.
Most importantly, however, we have found serious issues with the specification
of dependencies that present a major threat to the reproducibility and
maintainability of Python ML projects. Furthermore, we found that Pylint
produces a high rate of false positives on import statements and thus cannot
reliably check for correct usage of imported dependencies, including prominent
ML libraries such as PyTorch. Both of these problems also provide a major
obstacle to the adoption of CI in ML systems. Further research needs to be
undertaken to help ML practitioners avoid issues in the dependency management
of their projects.

## References

  * Haakman et al. [2020] M. Haakman, L. Cruz, H. Huijgens, and A. van Deursen, “AI lifecycle models need to be revised. An exploratory study in fintech,” _arXiv preprint arXiv:2010.02716_ , 2020.
  * Sculley et al. [2015] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison, “Hidden technical debt in machine learning systems,” in _Advances in neural information processing systems_ , 2015, pp. 2503–2511.
  * Amershi et al. [2019] S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagappan, B. Nushi, and T. Zimmermann, “Software engineering for machine learning: A case study,” in _2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_. IEEE, 2019, pp. 291–300.
  * Gonzalez et al. [2020] D. Gonzalez, T. Zimmermann, and N. Nagappan, “The State of the ML-Universe: 10 Years of Artificial Intelligence & Machine Learning Software Development on GitHub,” in _Proceedings of the 17th International Conference on Mining Software Repositories_ , ser. MSR ’20, 2020, p. 431–442.
  * Menzies [2020] T. Menzies, “The Five Laws of SE for AI,” _IEEE Software_ , vol. 37, no. 1, pp. 81–85, 2020.
  * Simmons et al. [2020] A. J. Simmons, S. Barnett, J. Rivera-Villicana, A. Bajaj, and R. Vasa, “A Large-Scale Comparative Analysis of Coding Standard Conformance in Open-Source Data Science Projects,” in _Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)_ , ser. ESEM ’20, 2020.
  * Lacerda et al. [2020] G. Lacerda, F. Petrillo, M. Pimenta, and Y. G. Guéhéneuc, “Code smells and refactoring: A tertiary systematic review of challenges and observations,” _Journal of Systems and Software_ , vol. 167, p. 110610, 2020.
  * Tómasdóttir et al. [2020] K. Tómasdóttir, M. Aniche, and A. van Deursen, “The Adoption of JavaScript Linters in Practice: A Case Study on ESLint,” _IEEE Transactions on Software Engineering_ , vol. 46, no. 8, pp. 863–891, 2020.
  * Omari and Martinez [2019] S. Omari and G. Martinez, _Enabling Empirical Research: A Corpus of Large-Scale Python Systems_ , 10 2019, pp. 661–669.
  * Bafatakis et al. [2019] N. Bafatakis, N. Boecker, W. Boon, M. Cabello Salazar, J. Krinke, G. Oznacar, and R. White, “Python Coding Style Compliance on Stack Overflow,” in _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_ , 2019, pp. 210–214.
  * Chen et al. [2018] Z. Chen, L. Chen, W. Ma, X. Zhou, Y. Zhou, and B. Xu, “Understanding Metric-Based Detectable Smells in Python Software,” _Inf. Softw. Technol._ , vol. 94, no. C, p. 14–29, Feb. 2018.
  * Vassallo et al. [2020] C. Vassallo, S. Panichella, F. Palomba, S. Proksch, H. C. Gall, and A. Zaidman, “How developers engage with static analysis tools in different contexts,” _Empirical Software Engineering_ , vol. 25, no. 2, pp. 1419–1457, Mar 2020\.
  * Biswas et al. [2019] S. Biswas, M. J. Islam, Y. Huang, and H. Rajan, “Boa Meets Python: A Boa Dataset of Data Science Software in Python Language,” in _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_ , 2019, pp. 577–581.
  * Python Software Foundation [2021a] Python Software Foundation, “Managing Application Dependencies — Python Packaging User Guide,” 2021, (accessed: 19 Jan. 2021). [Online]. Available: https://packaging.python.org/tutorials/managing-dependencies/#managing-dependencies
  * Python Software Foundation [2021b] ——, “6. Modules — Python 3.7.9 documentation,” 2021, (accessed: 19 Jan. 2021). [Online]. Available: https://docs.python.org/3.7/tutorial/modules.html#packages
  * Hofmeister et al. [2019] J. C. Hofmeister, J. Siegmund, and D. V. Holt, “Shorter identifier names take longer to comprehend,” _Empirical Softw. Engg._ , vol. 24, no. 1, p. 417–443, Feb. 2019.
  * Perkel [2018] J. M. Perkel, “Why Jupyter is data scientists’ computational notebook of choice,” _Nature_ , vol. 563, no. 7732, pp. 145–147, 2018.