In order to extract the leading terms of $\chi$ from this summation, we
examine the pairs of $\mathcal{A}$ and $\mathcal{B}$. The pairs
$\mathcal{A}\mathcal{A}$ and $\mathcal{B}\mathcal{B}$ each form two loops that
contribute to the value, $\chi^{2}$. On the other hand, the pairs
$\mathcal{A}\mathcal{B}$ and $\mathcal{B}\mathcal{A}$ produce a single loop
representing $\chi$. As a result, the leading terms have fewer “domain walls”
for $\mathcal{A}$ and $\mathcal{B}$, and the most leading terms for $\chi$
include only one “domain” for $\mathcal{B}$. By summing up the terms with
single “domain” $\mathcal{B}^{l}$, we derive

$\displaystyle\mathrm{Var}\left(\braket{\psi_{\beta}}{\psi_{\beta}}\right)$
$\displaystyle=\chi^{N}\sum_{l=1}^{N}(N-l+1)z(\beta)^{2(N-l)}z(2\beta)^{l}+\mathcal{O}\left(\chi^{N-2}\right).$
(28)

To compute the NFPF, we normalize the variance of the norm as

$\displaystyle\delta z^{2}$
$\displaystyle=\frac{1}{\chi^{2}}\sum_{l=1}^{N}(N-l+1)\left\\{\frac{z(2\beta)}{z(\beta)^{2}}\right\\}^{l}+\mathcal{O}\left(\frac{1}{\chi^{4}}\right).$
(29)

This equation represents a series expansion that can be mathematically
evaluated and can be simplified when we consider
$z(2\beta)/z(\beta)^{2}=e^{-s_{2}(\beta)}$.

$\displaystyle\delta z^{2}$
$\displaystyle=\frac{1}{\chi^{2}}\sum_{l=1}^{N}(N-l+1)e^{-ls_{2}(\beta)}+\mathcal{O}\left(\frac{1}{\chi^{4}}\right)$
$\displaystyle=\frac{1}{\chi^{2}}\left\\{\frac{N}{e^{s_{2}(\beta)}-1}-\frac{1-e^{-Ns_{2}(\beta)}}{(e^{s_{2}(\beta)}-1)^{2}}\right\\}+\mathcal{O}\left(\frac{1}{\chi^{4}}\right)$
(30)

###  III.2 Correction factor for interactions

In the previous subsection, an exact formula for the NFPF of the non-
interacting Hamiltonian was derived. Now, based on this, we derive the formula
of the NFPF for general interacting Hamiltonians, which can be done by
introducing a single correlation factor as a numerically determined parameter.

We first analytically investigate the NFPF for the 1D classical Ising model to
grab an idea of correlation factors without bias or approximations. The
Hamiltonian is given as

$H=-J\sum_{i=1}^{N-1}\sigma^{z}_{i}\sigma^{z}_{i+1},$ (31)

which is the $g=0$ limit of the transverse-field Ising model introduced
shortly. The partition function is represented by the transfer matrix as

$Z(\beta)=\bm{v}^{\top}X^{N-1}\bm{v},$ (32)

where

$X(\beta)=\begin{bmatrix}e^{\beta J}&e^{-\beta J}\\\ e^{-\beta J}&e^{\beta
J}\end{bmatrix},\quad\bm{v}=\begin{bmatrix}1\\\ 1\end{bmatrix}.$ (33)

The free energy in the thermodynamic limit can be computed thorough the
maximal eigenvalue of the transfer matrix.

$\displaystyle f(\beta)$
$\displaystyle=-\frac{1}{\beta}\lim_{N\to\infty}\frac{1}{N}\log Z(\beta)$
$\displaystyle=-\frac{1}{\beta}\log(2\cosh\beta J)$ (34)

The thermal Rényi-2 entropy is obtained as

$\displaystyle s_{2}(\beta)$ $\displaystyle=2\beta\\{f(2\beta)-f(\beta)\\}$
$\displaystyle=\log\left(1+\frac{1}{\cosh(2\beta J)}\right).$ (35)

Next, we calculate a $\chi$-leading term that appeared in the NFPF. The
leading term of the present interacting case can be discussed in the same
manner as the one given previously for the non-interacting case in the
previous subsection. First, we separate the whole system into $A$ and $B$; as
the subsystem $A$, the first $m$ sites and the last $n$ sites are taken, and
for subsystem $B$, we have the center $l$ sites with $m+n+l=N$. Then, the
leading term corresponding to this bipartition is represented as

$\frac{1}{\chi^{2}}\frac{\operatorname{Tr}_{B}\left[\left(\operatorname{Tr}_{A}e^{-\beta
H}\right)^{2}\right]}{\left(\operatorname{Tr}e^{-\beta H}\right)^{2}}.$ (36)

By defining a tensor $Y$ as

${}_{s_{1}}Y(\beta)^{s_{2}}_{s_{3}}=\exp[\beta Js_{1}(s_{2}+s_{3})],$ (37)

the numerator can be expressed as

$\displaystyle\operatorname{Tr}_{B}\left[\left(\operatorname{Tr}_{A}e^{-\beta
H}\right)^{2}\right]$ $\displaystyle=\begin{matrix}\bm{v}^{T}X(\beta)^{m-1}\\\
\bm{v}^{T}X(\beta)^{m-1}\end{matrix}Y(\beta)^{T}X(2\beta)^{l-1}Y(\beta)\begin{matrix}X(\beta)^{n-1}\bm{v}\\\
X(\beta)^{n-1}\bm{v}\end{matrix}.$ (38)

Consequently, the leading term is calculated as

$\displaystyle\frac{\operatorname{Tr}_{B}\left[\left(\operatorname{Tr}_{A}e^{-\beta
H}\right)^{2}\right]}{\left(\operatorname{Tr}e^{-\beta
H}\right)^{2}}=\frac{e^{-(l-1)s_{2}(\beta)}}{2}.$ (39)

The NFPF of the classical Ising chain is obtained by replacing
$e^{-ls_{2}(\beta)}$ with $e^{-(l-1)s_{2}(\beta)}/2$ in Eq. (30).

$\displaystyle\delta
z^{2}=\frac{1}{\chi^{2}}\sum_{l=1}^{N}(N-l+1)\frac{e^{-(l-1)s_{2}(\beta)}}{2}+\mathcal{O}\left(\frac{1}{\chi^{4}}\right)$
$\displaystyle=\frac{1}{\chi^{2}}\frac{e^{s_{2}(\beta)}}{d}\left\\{\frac{N}{e^{s_{2}(\beta)}-1}-\frac{1-e^{-Ns_{2}(\beta)}}{(e^{s_{2}(\beta)}-1)^{2}}\right\\}+\mathcal{O}\left(\frac{1}{\chi^{4}}\right)$
(40)

Here, $d$ is local degrees of freedom; in the case of spin-half chains, $d=2$.

The two cases we obtained analytically without bias, Eqs. (30) and (40), have
most parts of their forms in common. It is then natural to introduce a real
parameter $\alpha$ to bride them as

$\delta
z^{2}\simeq\frac{1}{\chi^{2}}\left(\frac{e^{s_{2}(\beta)}}{d}\right)^{\alpha}\left\\{\frac{N}{e^{s_{2}(\beta)}-1}-\frac{1-e^{-Ns_{2}(\beta)}}{(e^{s_{2}(\beta)}-1)^{2}}\right\\}.$
(41)

We expect this form to be valid for other quantum many-body systems
represented by TPQ-MPS. Indeed, this form is justified by the following
numerical demonstrations where we treat $\alpha$ as a fitting parameter.

Figure 2:  NFPF $\delta z^{2}$ obtained by the TPQ-MPS method as a function of
inverse temperature $\beta$ for the transverse-field Ising chain at (a)
$\theta=\pi/8$ (gapped paramagnetic phase) and (b) $\theta=\pi/4$ (gapless
critical point). We use $N=16,32,64$ and $\chi=20$ and take $M=500$ samples
for each data. The error bars are calculated by a jackknife analysis. The
solid lines are the analytical results in Eq. (41) with fitting parameter (a)
$\alpha\simeq 0.28$ and (b) $\alpha\simeq 0.33$. (c) The system size
dependence of $\chi^{2}\delta z^{2}$ for $\theta=\pi/8$ and $\chi=10,15,20$ at
$\beta=4$. (d) The fitting parameter $\alpha$ for different system parameters
$\theta$. Dashed lines show analytical conjectures for the two limiting cases,
$\theta=0$ and $1$, which are the non-interacting and classical Ising models,
respectively.

###  III.3 Crossover of sample complexity

It is important to note that the NFPF in Eq. (41) exhibits different scaling
forms about the system size at high- and low-temperature regions. When
$Ns_{2}(\beta)\gg 1$, we can discard the term $e^{-Ns_{2}(\beta)}$ and Eq.
(41) becomes

$\delta
z^{2}\simeq\frac{N}{\chi^{2}}\left(\frac{e^{s_{2}(\beta)}}{d}\right)^{\alpha}\frac{1}{e^{s_{2}(\beta)}-1}$
(42)

in the large-$N$ limit. The NFPF scales linearly with the system size $N$ at
high temperature. On the other hand, when $Ns_{2}(\beta)\ll 1$, the
exponential functions can be expanded as

$\displaystyle e^{-Ns_{2}(\beta)}\simeq
1-Ns_{2}(\beta)+\frac{N^{2}s_{2}(\beta)^{2}}{2}$ (43) $\displaystyle
e^{s_{2}(\beta)}\simeq 1+s_{2}(\beta)+\frac{s_{2}(\beta)^{2}}{2}.$ (44)

Therefore, Eq. (41) becomes

$\delta z^{2}\simeq\frac{N^{2}}{\chi^{2}}\frac{1}{2d^{\alpha}}$ (45)

in the large-$N$ limit. The NFPF is proportional to the square of the system
size $N$ at low temperature.

The crossover inverse temperature $\beta_{c}$ is characterized by
$Ns_{2}(\beta_{c})\sim 1$. Because the thermal Rényi-2 entropy scales as
$s_{2}(\beta)\sim e^{-\beta\Delta E}$ in the low-temperature limit where
$\Delta E$ is the spectral gap, the crossover inverse temperature can be
estimated as

$\beta_{c}\sim\frac{1}{\Delta E}\log N.$ (46)

The crossover of the NFPF appears as a finite-size effect.

##  IV Numerical demonstrations

In this section, we demonstrate the validity of Eq. (41) in two quantum many-
body models. At each TEBD step, the bond dimension $\chi$ is truncated to have
the truncation error is smaller than $10^{-8}$. We used the ITensor library
for MPS calculations [77].

###  IV.1 Transverse-field Ising chain

We first consider the transverse-field Ising chain,

$H=-J\sum_{i=1}^{N-1}\sigma_{i}^{z}\sigma_{i+1}^{z}-g\sum_{i}^{N}\sigma_{i}^{x},$
(47)

where $\sigma^{\alpha}_{i}~{}(\alpha=x,y,z)$ is the Pauli operator at site
$i$. We parameterize the coupling interaction $J$ and the transverse field $g$
as $J=\sin\theta$ and $g=\cos\theta$ with a single parameter $\theta$ that
varies in the range $[0,\pi/2]$. It is analytically solved in the
thermodynamic limit [78]. When $J=0$ and $g>0$, the Hamiltonian is represented
as a sum of single-site operators treated in §.III.1, and when $J>0$ and
$g=0$, we find a 1D classical Ising model we considered in §.III.2. The non-
interacting and classical Ising cases correspond to $\theta=0$ and
$\theta=\pi/2$, respectively. When $0\leq\theta<\pi/4$, we have a gapped
paramagnetic phase due to large $g$, and $\pi/4<\theta\leq\pi/2$, the system
in a gapped ferromagnetic Ising phase, and the gap closes at $\theta=\pi/4$.
Whether the system is gapped or gapless is related to the nature of the NFPF.

Figures 2(a) and 2(b) show the temperature dependence of the NFPF $\delta
z^{2}$ for $\theta=\pi/8,\pi/4$, respectively. The solid lines represent the
formula in Eq. (41). The thermal Rényi-2 entropy $s_{2}(\beta)$ of the
transverse-field Ising chain can be analytically calculated from an exact
result, and using its value, we can fit the mean value of the numerically
obtained NFPF data by a single parameter $\alpha$. Here, the error bars are
calculated based on the associated errors in the NFPF. In the gapped
paramagnetic case of Fig. 2(a), using $\alpha\simeq 0.28$ common to three
choices of $N$, the formula and the data agree well throughout the whole
temperature range. All NFPF increase at high temperature and reach a plateau
at low temperature. This behavior indicates the fact that low- and high-
temperature regions follow different scaling laws, as expected from Eqs. (42)
and (45). As discussed in §.III.3, the crossover temperature $\beta_{c}$ in
Eq. (46) grows logarithmically with $N$, consistently with Fig. 2(a).

Figure. 2(b), the gapless case, also shows good agreement with the analytical
result, but there are no plateau regions in contrast to the above example.
This is because $\beta_{c}$ diverges in gapless systems with $\Delta E=0$.

Figure. 2(c) shows the system size dependence of $\chi^{2}\delta z^{2}$.
Because $N$ appears in the numerators of the scaling formula Eqs. (42) and
(45), Eq. (41) is considered to be an expansion for large bond dimensions and
small system size.

In Fig. 2(d), we show the evolution of the fitting parameter $\alpha$ in Eq.
(41) as a function of $\theta$. Here, $\alpha$ is calculated for $N=16$ and
$\chi=20$. In the non-interacting and classical Ising model, numerical results
and analytical estimation agree well. The fitting parameter $\alpha$ varies
smoothly and interpolates the two limits, fairly well within the error bars,
supporting Eq. (41).

###  IV.2 Heisenberg chain

Figure 3:  NFPF $\delta z^{2}$ of $S=1/2$ and $S=1$ (inset) Heisenberg chain
for $N=16,\chi=20$. The solid lines represent the analytical result in Eq.
(41) for fitting parameters $\alpha\simeq 0.66$ and $\alpha\simeq 1.25$
(inset).

We next apply the Heisenberg chain defined as

$H=J\sum_{i=1}^{N-1}\bm{S}_{i}\cdot\bm{S}_{i+1},$ (48)

where $S^{\alpha}_{i}~{}(\alpha=x,y,z)$ is the spin operator at site $i$. The
$S=1/2$ Heisenberg model is exactly solvable [79] and the ground state is
gapless. The $S=1$ case has the Haldane gap of $\Delta E\simeq 0.4105$ [80,
81, 82]. We treat both cases to compare gapless and gapped systems. Figure 3
shows the results of $N=16$ and $\chi=20$, both being consistent with the
analytical formula. The thermal Rényi-2 entropy used in the fitting function
is evaluated by the TPQ-MPS method at $N=100$. We can find a plateau at low
temperature in the inset of Fig. 3, which indicates the crossover of the
system size scaling. The analytical formula in Eq. (41) is thus expected to
hold.

##  V Summary and discussions

As one of the measures of difficulty in computing the finite-temperature
quantum states, we investigated the normalized fluctuation of the partition
function (NFPF). This quantity is well-defined and measurable for a general
random sampling method that generates a thermal state using the imaginary time
evolution from the high-temperature random states. The NFPF is proportional to
the required number of samples, and reflects the sample complexity of the
calculation. Focusing on the matrix-product-state-based (MPS-based) random
sampling method, we analytically derived an exact form of the NFPF in Eq.
(41), and demonstrated its validity numerically for several representative
models. Our formula shows a crossover behavior in varying the temperature; at
high temperatures, the NFPF scales linearly with the system size while at low
temperatures, it is proportional to the square of the system size. This result
gives the first practical and quantitative demonstration of the computational
complexity of 1D systems shown in Fig. 1(a).

The computational cost of the MPS-based sampling relies on two quantities, the
required number of samples and the memory cost per sample. Our result and the
previous conformal field theory (CFT) calculations indicate respectively that
both the former and the latter scale polynomially with the system size at any
finite temperatures. Because the ground state is known to be QMA-complete,
there arises a question; how can such computationally feasible finite-
temperature states be bounded by polynomials can continue to the zero-
temperature limit? As we discussed in the introduction, QMA-completeness may
rely on several non-typical states in 1D models that deviate from CFT
predictions at low temperatures such as the Motzkin chain [83, 84]. In these
models, the bond dimension of the thermal state diverges as the temperature
approaches zero. However, explicitly presenting such models as evidence would
imply proving that P$\neq$PSPACE, a claim we cannot substantiate.

Our analysis in §.III.1 can be straightforwardly extended to higher
dimensions. There, the $\mathcal{B}$ domain can no longer exist in the leading
terms, in the same manner as the classical Ising model requires finite energy
to form domains in higher dimensions. Consequently, the NFPF becomes
proportional to the system size, and the no crossover observed in 1D no longer
exists. However, it is questionable whether this extension has practical
meaning. The entanglement in the ground state of higher dimensions is
extremely complex compared to that in one dimension, and it remains a topic of
ongoing debate, and a similar situation can be expected in thermal states. If
this is the case, our calculation in §.III.1, which assumes a certain type of
cluster property, likely fails. As the high-dimensional and low-temperature
regime is associated with the unsolved quantum PCP conjecture in Fig. 1(a), it
warrants careful discussion.

In this paper, we have focused on the random sampling methods using classical
computers, whilst the conversion of the related classical algorithms to the
quantum algorithms is now ongoing [85, 86, 87, 88, 89, 90, 91, 92, 93].
Because they are demonstrated virtually on classical computers or on real
quantum computers with very small sizes, it is so far difficult to evaluate
their effectiveness. In reality, in near-term quantum computers, the initial
random states are prepared on random circuits of shallow depth, and there, the
NFPF or the number of samples is expected to follow the same scaling as our
results.

We finally note that our calculations in §.III.1 are similar to those treating
the Rényi-2 entropy of an evaporating black hole in Ref. [94], but the
physical implications of these relationships are not yet well understood.
Following the approach in Ref. [94], we can compute cumulants of any order for
the norm of thermal states in MPS-based random sampling. This analysis
pertains to a more detailed efficiency of the method.

###### Acknowledgements.

We thank Kouichi Okunishi and Hosho Katsura for fruitful discussions. A. I.
was supported by a Grant-in-Aid for JSPS Research Fellow (Grant No. 21J21992).
This work was supported by a Grant-in-Aid for Transformative Research Areas
“The Natural Laws of Extreme Universe—A New Paradigm for Spacetime and Matter
from Quantum Information” (Grant No. 21H05191) and other JSPS KAKENHI (No.
21K03440) of Japan, and JSPS KAKENHI Grants No. JP21K03440.

## References

  * Deutsch [2018] J. M. Deutsch, Eigenstate thermalization hypothesis, Reports on Progress in Physics 81, 082001 (2018).
  * Mori  _et al._ [2018] T. Mori, T. N. Ikeda, E. Kaminishi, and M. Ueda, Thermalization and prethermalization in isolated quantum systems: a theoretical overview, Journal of Physics B: Atomic, Molecular and Optical Physics 51, 112001 (2018).
  * Nandkishore and Huse [2015] R. Nandkishore and D. A. Huse, Many-body localization and thermalization in quantum statistical mechanics, Annual Review of Condensed Matter Physics 6, 15â38 (2015).
  * Abanin  _et al._ [2019] D. A. Abanin, E. Altman, I. Bloch, and M. Serbyn, Colloquium: Many-body localization, thermalization, and entanglement, Rev. Mod. Phys. 91, 021001 (2019).
  * Moudgalya  _et al._ [2022] S. Moudgalya, B. A. Bernevig, and N. Regnault, Quantum many-body scars and hilbert space fragmentation: a review of exact results, Reports on Progress in Physics 85, 086501 (2022).
  * Chandran  _et al._ [2023] A. Chandran, T. Iadecola, V. Khemani, and R. Moessner, Quantum many-body scars: A quasiparticle perspective, Annual Review of Condensed Matter Physics 14, 443â469 (2023).
  * Aidelsburger  _et al._ [2013] M. Aidelsburger, M. Atala, M. Lohse, J. T. Barreiro, B. Paredes, and I. Bloch, Realization of the hofstadter hamiltonian with ultracold atoms in optical lattices, Phys. Rev. Lett. 111, 185301 (2013).
  * Miyake  _et al._ [2013] H. Miyake, G. A. Siviloglou, C. J. Kennedy, W. C. Burton, and W. Ketterle, Realizing the harper hamiltonian with laser-assisted tunneling in optical lattices, Phys. Rev. Lett. 111, 185302 (2013).
  * Bernien  _et al._ [2017] H. Bernien, S. Schwartz, A. Keesling, H. Levine, A. Omran, H. Pichler, S. Choi, A. S. Zibrov, M. Endres, M. Greiner, V. VuletiÄ, and M. D. Lukin, Probing many-body dynamics on a 51-atom quantum simulator, Nature 551, 579â584 (2017).
  * Zhang  _et al._ [2017] J. Zhang, G. Pagano, P. W. Hess, A. Kyprianidis, P. Becker, H. Kaplan, A. V. Gorshkov, Z.-X. Gong, and C. Monroe, Observation of a many-body dynamical phase transition with a 53-qubit quantum simulator, Nature 551, 601â604 (2017).
  * King  _et al._ [2018] A. D. King, J. Carrasquilla, J. Raymond, I. Ozfidan, E. Andriyash, A. Berkley, M. Reis, T. Lanting, R. Harris, F. Altomare, K. Boothby, P. I. Bunyk, C. Enderud, A. FrÃ©chette, E. Hoskinson, N. Ladizinsky, T. Oh, G. Poulin-Lamarre, C. Rich, Y. Sato, A. Y. Smirnov, L. J. Swenson, M. H. Volkmann, J. Whittaker, J. Yao, E. Ladizinsky, M. W. Johnson, J. Hilton, and M. H. Amin, Observation of topological phenomena in a programmable lattice of 1, 800 qubits, Nature 560, 456â460 (2018).
  * Harris  _et al._ [2018] R. Harris, Y. Sato, A. J. Berkley, M. Reis, F. Altomare, M. H. Amin, K. Boothby, P. Bunyk, C. Deng, C. Enderud, S. Huang, E. Hoskinson, M. W. Johnson, E. Ladizinsky, N. Ladizinsky, T. Lanting, R. Li, T. Medina, R. Molavi, R. Neufeld, T. Oh, I. Pavlov, I. Perminov, G. Poulin-Lamarre, C. Rich, A. Smirnov, L. Swenson, N. Tsai, M. Volkmann, J. Whittaker, and J. Yao, Phase transitions in a programmable quantum spin glass simulator, Science 361, 162â165 (2018).
  * Kitaev  _et al._ [2002] A. Kitaev, A. Shen, and M. Vyalyi,  _Classical and Quantum Computation_ , Graduate studies in mathematics (American Mathematical Society, 2002).
  * Oliveira and Terhal [2008] R. Oliveira and B. Terhal, The complexity of quantum spin systems on a two-dimensional square lattice, Quantum Information and Computation 8, 900â924 (2008).
  * Aharonov  _et al._ [2009] D. Aharonov, D. Gottesman, S. Irani, and J. Kempe, The power of quantum systems on a line, Communications in Mathematical Physics 287, 41 (2009).
  * Nagaj [2008] D. Nagaj, Local hamiltonians in quantum computation, arXiv:0808.2117 (2008).
  * Hallgren  _et al._ [2013] S. Hallgren, D. Nagaj, and S. Narayanaswami, The local hamiltonian problem on a line with eight staes is qma-complete, Quantum Information and Computation 13, 721â750 (2013).
  * White [1992] S. R. White, Density matrix formulation for quantum renormalization groups, Phys. Rev. Lett. 69, 2863 (1992).
  * White [1993] S. R. White, Density-matrix algorithms for quantum renormalization groups, Phys. Rev. B 48, 10345 (1993).
  * SchollwÃ¶ck [2011] U. SchollwÃ¶ck, The density-matrix renormalization group in the age of matrix product states, Annals of Physics 326, 96 (2011), january 2011 Special Issue.
  * Landau  _et al._ [2015] Z. Landau, U. Vazirani, and T. Vidick, A polynomial time algorithm for the ground state of one-dimensional gapped local hamiltonians, Nature Physics 11, 566 (2015).
  * Chubb and Flammia [2016] C. T. Chubb and S. T. Flammia, Computing the degenerate ground space of gapped spin chains in polynomial time, Chicago Journal of Theoretical Computer Science 2016 (2016).
  * Arad  _et al._ [2017] I. Arad, Z. Landau, U. Vazirani, and T. Vidick, Rigorous rg algorithms and area laws for low energy eigenstates in 1d, Communications in Mathematical Physics 356, 65 (2017).
  * Roberts  _et al._ [2017] B. Roberts, T. Vidick, and O. I. Motrunich, Implementation of rigorous renormalization group method for ground space and low-energy states of local hamiltonians, Phys. Rev. B 96, 214203 (2017).
  * Hastings [2007] M. B. Hastings, An area law for one-dimensional quantum systems, Journal of Statistical Mechanics: Theory and Experiment 2007, P08024 (2007).
  * Eisert  _et al._ [2010] J. Eisert, M. Cramer, and M. B. Plenio, Colloquium: Area laws for the entanglement entropy, Rev. Mod. Phys. 82, 277 (2010).
  * Cubitt  _et al._ [2015] T. S. Cubitt, D. Perez-Garcia, and M. M. Wolf, Undecidability of the spectral gap, Nature 528, 207 (2015).
  * Bausch  _et al._ [2020] J. Bausch, T. S. Cubitt, A. Lucia, and D. Perez-Garcia, Undecidability of the spectral gap in one dimension, Phys. Rev. X 10, 031038 (2020).
  * Kuwahara  _et al._ [2021] T. Kuwahara, A. M. Alhambra, and A. Anshu, Improved thermal area law and quasilinear time algorithm for quantum gibbs states, Phys. Rev. X 11, 011047 (2021).
  * Alhambra and Cirac [2021] A. M. Alhambra and J. I. Cirac, Locally accurate tensor networks for thermal states and time evolution, PRX Quantum 2, 040331 (2021).
  * Kuwahara  _et al._ [2020] T. Kuwahara, K. Kato, and F. G. S. L. Brandão, Clustering of conditional mutual information for quantum gibbs states above a threshold temperature, Phys. Rev. Lett. 124, 220601 (2020).
  * Harrow  _et al._ [2020] A. W. Harrow, S. Mehraban, and M. Soleimanifar, Classical algorithms, correlation decay, and complex zeros of partition functions of quantum many-body systems, in  _Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing_, STOC ’20 (ACM, 2020).
  * Aharonov  _et al._ [2013] D. Aharonov, I. Arad, and T. Vidick, Guest column: the quantum pcp conjecture, ACM SIGACT News 44, 47â79 (2013).
  * Foulkes  _et al._ [2001] W. M. C. Foulkes, L. Mitas, R. J. Needs, and G. Rajagopal, Quantum monte carlo simulations of solids, Rev. Mod. Phys. 73, 33 (2001).
  * Sandvik [2010] A. W. Sandvik, Computational Studies of Quantum Spin Systems, AIP Conference Proceedings 1297, 135 (2010).
  * Domb and Green [1974] C. Domb and M. S. Green,  _Phase Transitions and Critical Phenomena_ (Academic Press, New York, 1974).
  * Oitmaa  _et al._ [2006] J. Oitmaa, C. Hamer, and W.-H. Zheng,  _Series Expansion Methods for Strongly Interacting Lattice Models_ (Cambridge University Press, Cambridge, UK, 2006).
  * Rigol  _et al._ [2006] M. Rigol, T. Bryant, and R. R. P. Singh, Numerical linked-cluster approach to quantum lattice models, Phys. Rev. Lett. 97, 187202 (2006).
  * Rigol  _et al._ [2007a] M. Rigol, T. Bryant, and R. R. P. Singh, Numerical linked-cluster algorithms. i. spin systems on square, triangular, and kagomé lattices, Phys. Rev. E 75, 061118 (2007a).
  * Rigol  _et al._ [2007b] M. Rigol, T. Bryant, and R. R. P. Singh, Numerical linked-cluster algorithms. ii. $t\text{$-$}j$ models on the square lattice, Phys. Rev. E 75, 061119 (2007b).
  * Imada and Takahashi [1986] M. Imada and M. Takahashi, Quantum transfer monte carlo method for finite temperature properties and quantum molecular dynamics method for dynamical correlation functions, Journal of the Physical Society of Japan 55, 3354 (1986).
  * Jaklič and Prelovšek [1994] J. Jaklič and P. Prelovšek, Lanczos method for the calculation of finite-temperature quantities in correlated systems, Phys. Rev. B 49, 5065 (1994).
  * Hams and De Raedt [2000] A. Hams and H. De Raedt, Fast algorithm for finding the eigenvalue distribution of very large matrices, Phys. Rev. E 62, 4365 (2000).
  * Sugiura and Shimizu [2012] S. Sugiura and A. Shimizu, Thermal pure quantum states at finite temperature, Phys. Rev. Lett. 108, 240401 (2012).
  * Sugiura and Shimizu [2013] S. Sugiura and A. Shimizu, Canonical thermal pure quantum state, Phys. Rev. Lett. 111, 010401 (2013).
  * Popescu  _et al._ [2006] S. Popescu, A. J. Short, and A. Winter, Entanglement and the foundations of statistical mechanics, Nature Phys. 2, 754 (2006).
  * Goldstein  _et al._ [2006] S. Goldstein, J. L. Lebowitz, R. Tumulka, and N. Zanghì, Canonical typicality, Phys. Rev. Lett. 96, 050403 (2006).
  * Reimann [2007] P. Reimann, Typicality for generalized microcanonical ensembles, Phys. Rev. Lett. 99, 160404 (2007).
  * Sugita [2007] A. Sugita, On the basis of quantum statistical mechanics, Nonlinear Phenom. Complex Syst 10, 192 (2007).
  * Fannes  _et al._ [1992] M. Fannes, B. Nachtergaele, and R. F. Werner, Finitely correlated states on quantum spin chains, Commun. Math. Phys. 144, 443 (1992).
  * Vidal [2004] G. Vidal, Efficient simulation of one-dimensional quantum many-body systems, Phys. Rev. Lett. 93, 040502 (2004).
  * White and Feiguin [2004] S. R. White and A. E. Feiguin, Real-time evolution using the density matrix renormalization group, Phys. Rev. Lett. 93, 076401 (2004).
  * Daley  _et al._ [2004] A. J. Daley, C. Kollath, U. SchollwÃ¶ck, and G. Vidal, Time-dependent density-matrix renormalization-group using adaptive effective hilbert spaces, Journal of Statistical Mechanics: Theory and Experiment 2004, P04005 (2004).
  * Verstraete  _et al._ [2004] F. Verstraete, J. J. García-Ripoll, and J. I. Cirac, Matrix product density operators: Simulation of finite-temperature and dissipative systems, Phys. Rev. Lett. 93, 207204 (2004).
  * Zwolak and Vidal [2004] M. Zwolak and G. Vidal, Mixed-state dynamics in one-dimensional quantum lattice systems: A time-dependent superoperator renormalization algorithm, Phys. Rev. Lett. 93, 207205 (2004).
  * Feiguin and White [2005] A. E. Feiguin and S. R. White, Finite-temperature density matrix renormalization using an enlarged hilbert space, Phys. Rev. B 72, 220401 (2005).
  * Chen  _et al._ [2018] B.-B. Chen, L. Chen, Z. Chen, W. Li, and A. Weichselbaum, Exponential thermal tensor network approach for quantum lattice models, Phys. Rev. X 8, 031082 (2018).
  * Hauschild  _et al._ [2018] J. Hauschild, E. Leviatan, J. H. Bardarson, E. Altman, M. P. Zaletel, and F. Pollmann, Finding purifications with minimal entanglement, Phys. Rev. B 98, 235163 (2018).
  * Li  _et al._ [2023] Q. Li, Y. Gao, Y.-Y. He, Y. Qi, B.-B. Chen, and W. Li, Tangent space approach for thermal tensor network simulations of the 2d hubbard model, Phys. Rev. Lett. 130, 226502 (2023).
  * Garnerone and de Oliveira [2013] S. Garnerone and T. R. de Oliveira, Generalized quantum microcanonical ensemble from random matrix product states, Phys. Rev. B 87, 214426 (2013).
  * Garnerone [2013] S. Garnerone, Pure state thermodynamics with matrix product states, Phys. Rev. B 88, 165140 (2013).
  * Goto  _et al._ [2021] S. Goto, R. Kaneko, and I. Danshita, Matrix product state approach for a quantum system at finite temperatures using random phases and trotter gates, Phys. Rev. B 104, 045133 (2021).
  * Gao  _et al._ [2023] J. Gao, Y. Gao, Q. Li, and W. Li, Finite-temperature simulations of quantum lattice models with stochastic matrix product states, arXiv:2312.04420 (2023).
  * White [2009] S. R. White, Minimally entangled typical quantum states at finite temperature, Phys. Rev. Lett. 102, 190601 (2009).
  * Stoudenmire and White [2010] E. M. Stoudenmire and S. R. White, Minimally entangled typical thermal state algorithms, New Journal of Physics 12, 055026 (2010).
  * Bruognolo  _et al._ [2015] B. Bruognolo, J. von Delft, and A. Weichselbaum, Symmetric minimally entangled typical thermal states, Phys. Rev. B 92, 115105 (2015).
  * Binder and Barthel [2015] M. Binder and T. Barthel, Minimally entangled typical thermal states versus matrix product purifications for the simulation of equilibrium states and time evolution, Phys. Rev. B 92, 125119 (2015).
  * Binder and Barthel [2017] M. Binder and T. Barthel, Symmetric minimally entangled typical thermal states for canonical and grand-canonical ensembles, Phys. Rev. B 95, 195148 (2017).
  * Goto and Danshita [2020] S. Goto and I. Danshita, Minimally entangled typical thermal states algorithm with trotter gates, Phys. Rev. Res. 2, 043236 (2020).
  * Chen and Stoudenmire [2020] J. Chen and E. M. Stoudenmire, Hybrid purification and sampling approach for thermal quantum systems, Phys. Rev. B 101, 195119 (2020).
  * Chung and Schollwöck [2019] C.-M. Chung and U. Schollwöck, Minimally entangled typical thermal states with auxiliary matrix-product-state bases, arXiv:1910.03329 (2019).
  * Iwaki and Hotta [2022] A. Iwaki and C. Hotta, Purity of thermal mixed quantum states, Phys. Rev. B 106, 094409 (2022).
  * Jin  _et al._ [2021] F. Jin, D. Willsch, M. Willsch, H. Lagemann, K. Michielsen, and H. De Raedt, Random state technology, Journal of the Physical Society of Japan 90, 012001 (2021).
  * Iwaki  _et al._ [2021] A. Iwaki, A. Shimizu, and C. Hotta, Thermal pure quantum matrix product states recovering a volume law entanglement, Phys. Rev. Research 3, L022015 (2021).
  * Gohlke  _et al._ [2023] M. Gohlke, A. Iwaki, and C. Hotta, Thermal pure matrix product state in two dimensions: Tracking thermal equilibrium from paramagnet down to the Kitaev honeycomb spin liquid state, SciPost Phys. 15, 206 (2023).
  * Temperley and Lieb [1971] H. N. Temperley and E. H. Lieb, Relations between the ‘percolation’ and ‘colouring’ problem and other graph-theoretical problems associated with regular planar lattices: some exact results for the ‘percolation’ problem, Proceedings of the Royal Society of London. A. Mathematical and Physical Sciences 322, 251â280 (1971).
  * Fishman  _et al._ [2022] M. Fishman, S. R. White, and E. M. Stoudenmire, The ITensor Software Library for Tensor Network Calculations, SciPost Phys. Codebases , 4 (2022).
  * Pfeuty [1970] P. Pfeuty, The one-dimensional ising model with a transverse field, Annals of Physics 57, 79 (1970).
  * Bethe [1931] H. Bethe, Zur theorie der metalle, Z. Phys. 71, 205 (1931).
  * Haldane [1981] F. D. M. Haldane, Ground state properties of antiferromagnetic chains with unrestricted spin: Integer spin chains as realisations of the o(3) non-linear sigma model, arXiv:1612.00076 (1981).
  * Haldane [1983] F. Haldane, Continuum dynamics of the 1-d heisenberg antiferromagnet: Identification with the o(3) nonlinear sigma model, Physics Letters A 93, 464â468 (1983).
  * White and Huse [1993] S. R. White and D. A. Huse, Numerical renormalization-group study of low-lying eigenstates of the antiferromagnetic s=1 heisenberg chain, Phys. Rev. B 48, 3844 (1993).
  * Bravyi  _et al._ [2012] S. Bravyi, L. Caha, R. Movassagh, D. Nagaj, and P. W. Shor, Criticality without frustration for quantum spin-1 chains, Phys. Rev. Lett. 109, 207202 (2012).
  * Movassagh and Shor [2016] R. Movassagh and P. W. Shor, Supercritical entanglement in local systems: Counterexample to the area law for quantum matter, Proceedings of the National Academy of Sciences 113, 13278â13282 (2016).
  * Motta  _et al._ [2019] M. Motta, C. Sun, A. T. K. Tan, M. J. O’Rourke, E. Ye, A. J. Minnich, F. G. S. L. BrandÃ£o, and G. K.-L. Chan, Determining eigenstates and thermal states on a quantum computer using quantum imaginary time evolution, Nature Physics 16, 205â210 (2019).
  * Sun  _et al._ [2021] S.-N. Sun, M. Motta, R. N. Tazhigulov, A. T. Tan, G. K.-L. Chan, and A. J. Minnich, Quantum computation of finite-temperature static and dynamical properties of spin systems using quantum imaginary time evolution, PRX Quantum 2, 010317 (2021).
  * Seki and Yunoki [2022] K. Seki and S. Yunoki, Energy-filtered random-phase states as microcanonical thermal pure quantum states, Phys. Rev. B 106, 155111 (2022).
  * Coopmans  _et al._ [2023] L. Coopmans, Y. Kikuchi, and M. Benedetti, Predicting gibbs-state expectation values with pure thermal shadows, PRX Quantum 4, 010305 (2023).
  * Goto  _et al._ [2023] S. Goto, R. Kaneko, and I. Danshita, Evaluating thermal expectation values by almost ideal sampling with trotter gates, Phys. Rev. B 107, 024307 (2023).
  * Mizukami and Koga [2023] K. Mizukami and A. Koga, Quantum algorithm for the microcanonical thermal pure quantum state method, Phys. Rev. A 108, 012404 (2023).
  * Davoudi  _et al._ [2023] Z. Davoudi, N. Mueller, and C. Powers, Towards quantum computing phase diagrams of gauge theories with thermal pure quantum states, Phys. Rev. Lett. 131, 081901 (2023).
  * Getelina  _et al._ [2023] J. C. Getelina, N. Gomes, T. Iadecola, P. P. Orth, and Y.-X. Yao, Adaptive variational quantum minimally entangled typical thermal states for finite temperature simulations, SciPost Phys. 15, 102 (2023).
  * Pedersen  _et al._ [2023] J. Pedersen, E. Itou, R.-Y. Sun, and S. Yunoki, Quantum simulation of finite temperature schwinger model via quantum imaginary time evolution, in  _Proceedings of The 40th International Symposium on Lattice Field Theory ― PoS(LATTICE2023)_, LATTICE2023 (Sissa Medialab, 2023).
  * Penington  _et al._ [2022] G. Penington, S. H. Shenker, D. Stanford, and Z. Yang, Replica wormholes and the black hole interior, Journal of High Energy Physics 2022, 205 (2022).


11institutetext: Blekinge Institute of Technology, Valhallavägen 1, 371 41
Karlskrona, Sweden 11email<EMAIL_ADDRESS>
# Identifying relevant Factors of Requirements Quality: an industrial Case
Study

Julian Frattini 11 0000-0003-3995-6125

###### Abstract

[Context and Motivation]: The quality of requirements specifications impacts
subsequent, dependent software engineering activities. Requirements quality
defects like ambiguous statements can result in incomplete or wrong features
and even lead to budget overrun or project failure. [Problem]: Attempts at
measuring the impact of requirements quality have been held back by the vast
amount of interacting factors. Requirements quality research lacks an
understanding of which factors are relevant in practice. [Principal Ideas and
Results]: We conduct a case study considering data from both interview
transcripts and issue reports to identify relevant factors of requirements
quality. The results include 17 factors and 11 interaction effects relevant to
the case company. [Contribution]: The results contribute empirical evidence
that (1) strengthens existing requirements engineering theories and (2)
advances industry-relevant requirements quality research.

###### Keywords:

Requirements quality Case study Interview

##  1 Introduction

Requirements specifications constitute the input to many subsequent software
engineering activities and artifacts. Requirements specifications are used to
design architecture, develop code, or derive test cases. Hence, the quality of
requirements specifications impacts the software engineering process [16, 24].

The requirements quality research domain aims to aid practitioners in
understanding and managing the quality of their requirements specifications by
detecting and removing requirements quality defects [18]. However, empirical
evidence in this research domain remains scarce [14]. This hampers the
adoption of research results in practice [6, 11]. Recent systematic studies of
the requirements quality research domain have identified a lack of industrial
relevance as a main factor holding the field back [18].

The aim of this study is to contribute empirical evidence about the impact of
requirements quality defects by identifying factors of requirements quality
that are relevant in practice. To this end, an industrial case study at
Ericsson has been conducted. The gathered evidence both strengthens existing
requirements engineering theories and steers future research efforts toward
solving practically relevant problems. While the results cannot be generalized
due to the employed research method, we encourage replication in different
industrial contexts with the disclosure of our replication package111Available
at https://doi.org/10.5281/zenodo.10149475.

The remainder of this article is structured as follows: Section 2 introduces
related work on requirements quality research. Section 3 describes the applied
method and Section 4 reports the obtained results. These results are discussed
in Section 5 before concluding in Section 6.

##  2 Background

###  2.1 Requirements Quality

Organizations use requirements specifications in several subsequent software
engineering activities. Non-functional requirements influence a system’s
architecture, functional requirements determine the input and expected output
of the system’s features, and all requirements are ultimately translated into
test cases to assert whether the developed system meets the customers’
expectations. However, quality defects in requirements specifications like
missing or ambiguous requirements impede this reuse [18, 24].

Two factors aggravate the impact of requirements quality defects on subsequent
activities. Firstly, requirements specifications are predominantly written in
natural language [12] (NL). The inherent complexity and ambiguity of NL
benefits quality defects. Secondly, the cost to remove a quality defect scales
the longer it remains undetected [2]. Clarifying and rewriting an ambiguous
requirement takes significantly less time than re-implementing a wrong feature
built based on a misunderstood requirement.

Consequently, organizations aim to detect and remove relevant requirements
quality defects as early as possible [18]. A popular frame for this is the
requirements quality factor. A quality factor is a normative metric that maps
a requirements specification to a level of quality [14]. For example, the
quality factor passive voice associates the use of passive voice in a
requirements sentence with bad quality due to the potential omission of the
subject of the sentence [21]. Requirements quality research abounds with
quality factors and automatic tools to detect violations against them [14].

###  2.2 Requirements Quality Theory

Despite their usability, requirements quality factors suffer from significant
shortcomings. Most significantly, the majority of them lack empirical evidence
for their relevance [18, 14, 13], i.e., they are purely normative. Most
publications empirically investigate the performance of a tool automatically
detecting a violation against a quality factor, but only the fewest
empirically investigate whether this violation does have an actual impact and
is, therefore, even worth detecting and mitigating. This undermines the
practical relevance of requirements quality research, and research results are
rarely adopted in practice [6, 11].

In response, previous research proposed a harmonized requirements quality
theory (RQT). This theory frames requirements quality as the impact that
properties of requirements specifications have on the properties of dependent
activities in a given context [8]. Consequently, the RQT does not consider a
violation against a quality factor as harmful in itself, but only if this
violation has an impact on the activities that use the requirement as an
input.

Figure 1 visualizes a reduced version of the RQT [13]. The RQT consists of
three groups of concepts: artifact concepts, the context concept, and activity
concepts. The concept of an entity represents all types of requirements
artifacts like use cases or sentences. Quality factors are properties of these
entities [14], like the length of a specification, the completeness of a use
case, or the voice (active or passive) of a single sentence. Activities
represent every software engineering process that uses a requirements entity
as an input [8]. This includes activities like implementing or generating test
cases but also more implicit processes like understanding an entity and
estimating its effort. Attributes are properties of these activities and
include metrics like the duration or correctness of an activity. Context
factors represent properties of the process and involved stakeholders [19],
like the domain knowledge of involved engineers, the used process model, or
the distribution of the organization. Finally, the impact represents the
relationship between the quality factors, context factors, and attributes.

Figure 1: Groups and concepts of the reduced Requirements Quality Theory [13]

Not only does the RQT guide the framing of requirements quality, but it also
enables operationalization in practice [13]. By measuring quality factors,
context factors, and attributes, all input and output variables to the impact
become quantified. Once all variables are quantified, a statistical model
trained on historical data can estimate the probability that a certain quality
factor in a given context will affect the attribute of an activity [13]. This
quantification was not attempted yet [7] but advocated for in requirements
quality research roadmaps [6, 13] since it allows to (1) empirically assess
and compare the criticality of quality factors, and (2) predict how a
requirements specification will impact dependent activities. This prediction
model would meet the initially mentioned need of organizations to reliably
detect quality defects that impact the software engineering process.

###  2.3 Gap

Requirements quality research faces two major gaps. Firstly, requirements
quality research lacks empirical evidence for the relevance of requirements
quality factors [18, 14, 13]. This entails the risk that requirements quality
research does not focus on problems relevant to practice.

Secondly, and by extension, the RQT is difficult to operationalize without
empirical evidence about the relevance of quality factors. Previous research
has already identified 206 mostly normative requirements quality factors [14].
Measuring all of them is not feasible, given their amount and complexity.
Empirical evidence about the relevance of quality factors will aid in
prioritizing and selecting factors to consider in statistical models for
impact estimation.

##  3 Method

This study aims to address the gaps outlined in Section 2.3 by contributing
empirical evidence to relevant factors of requirements quality. The study
addresses the following research questions:

  * •

RQ1: Which factors of requirements quality do engineers that process
requirements perceive to be relevant?

  * •

RQ2: Which factors of requirements quality are reported in issues?

The study contrasts two perspectives: relevant factors of requirements quality
as perceived by engineers using requirements (RQ1) and as reported in issues
(RQ2). The study employs case study research to obtain insights on the
necessary level of detail at the expense of generalizability [22]. The
contemporary software engineering phenomenon [25] that is subject of the study
is the impact of requirements quality defects as described through its
factors. The case study research method lends itself to the investigation
since the boundary between the phenomenon and the context is unclear [25].

The method follows Runeson et al. [22] and is reported according to the
guidelines by Höst et al. [15]. Figure 2 visualizes the steps of the process.
Verbatim examples shown in the figure and throughout this section (in
quotation marks) are artificial as the raw data is confidential.

Figure 2: Overview of the data collection and analysis method

###  3.1 Data Collection

Ericsson, the case company providing the data, is a large, globally
distributed software development organization. Ericsson follows an agile
development approach but completes the requirements specification for each new
feature or change request before committing to subsequent phases like design,
implementation, and testing. These requirements specifications use
unconstrained natural language to specify one or more requirements related to
a specific feature or change request. The study evaluates two different types
of data to triangulate and strengthen the results [25]. The upper section of
Figure 2 visualizes the two data collection approaches.

####  3.1.1 Interview Data

To understand the factors that engineers processing requirements perceive to
be relevant, interviews were conducted (RQ1). A contact at the case company
provided a sample of eight software engineers responsible for processing the
requirements specification and developing a solution specification. The
interview participants had an average of 3.5 years of experience in their
role, 7.5 years with the organization, and 15.3 years as software engineers.

A protocol aided in conducting the semi-structured interviews based on
previous requirements quality research. In particular, the eleven themes of
requirements quality by Montgomery et al. [18] served as sections of the
interview. These themes include, for example, ambiguity, completeness, and
correctness. For each theme, the interview participants were asked whether
they experienced any issues of this type when processing requirements. If yes,
they were prompted to elaborate on the issue using the RQT as a frame [13].
Otherwise, the interview moved to the next theme. A slide deck introduced and
visualized the RQT to the interview participants to aid the conversation. The
author of this paper conducted all eight interviews, each taking up to one
hour.

The recorded interview sessions were automatically transcribed using
Descript222https://www.descript.com/. Afterwards, the author manually checked
all transcripts and ensured that the automatic transcription matched the
recording. The replication package contains all supplementary material,
including the interview guidelines. The transcripts contain confidential
information and cannot be shared.

####  3.1.2 Issue Data

To understand which factors of requirements quality cause a reported impact,
issues from an issue tracker were analyzed (RQ2). The contact at the case
company provided access to Ericsson’s database of issue reports. This database
contains issues raised both during the internal development process and from
external customers. Every issue denotes the development phases in which it was
detected and in which the root cause of this issue has been introduced. Domain
experts procure and document the latter information. For this study, the
available issues were filtered to obtain only those that have been introduced
during the requirements engineering phase, resulting in 100 issue reports from
January 2021 until September 2023.

###  3.2 Data Analysis

To analyze the large body of textual data, thematic synthesis as proposed by
Cruzes and Dybå [5] was employed and reported according to their guideline.
The lower section of Figure 2 visualizes the data analysis steps.

####  3.2.1 Data extraction

Extracting relevant data from the textual corpus comprised the first step.
Each defect perceived by an interview participant constitutes one statement
about requirements quality. The eight interview transcripts produced 84
statements about perceived requirements quality. The 100 issues contained 101
statements about reported requirements quality. The three groups of the RQT
(artifact, context, and activity) provided a frame for the data extraction.
For each of the three groups, all relevant mentions from a statement were
extracted. For example, the statement “If a specification rather proposes a
solution, but the feature is completely new, then the development requires
fewer changes.” contains one mention of each group: “a specification rather
proposes a solution” describes a property of the artifact, “the feature is
completely new” describes the context, and “the development requires fewer
changes” represents the activity impacted by the quality and context factor.

####  3.2.2 Data coding

Each mention received codes for all the concepts contained within the
respective RQT group. An artifact mention can contain a number of quality
factors with a value associated with each of them. A context mention can
contain a number of context factors. An activity mention can contain a number
of activities, each associated with one attribute and an impact value. All
codes and coding instructions were documented in extensive coding guidelines.

Coding the artifact and context mentions followed a deductive approach [17]:
quality factors [14, 18] and context factors [19] identified during previous
research constituted the available codes. Coding the activity mentions
followed an inductive approach [23] since literature is still lacking in this
regard [6, 13]. Figure 3 visualizes the activity and attribute codes generated
inductively from the data. Descriptions of all activities and attributes can
be found in the coding guideline contained in the replication package.

Figure 3: Model of activities (as classes) and attributes (as their
attributes)

The artifact mention “if a specification rather proposes a solution” received
the quality factor code orientation and the value solution, since it describes
a solution-oriented requirements specification. The context factor mention
“but the feature is completely new” was coded novelty. Finally, the activity
mention “then the development requires less changes” received the following
codes: the stability attribute of the implementing activity experiences a
positive impact ( +2). We coded the strength of the impact on a discrete
seven-point scale. Positive ( +2), negative ( -2), and no impact ( $\pm 0$)
were used as the default codes depending on the direction of the impact.
Particularly strong ( +3 and -3) or weak ( +1 and -1) codes were only used
when explicitly mentioned by the interview participant, e.g., if an impact was
“very critical” or just “slight.”

Independent coders were involved to ensure the reliability of the subjective
coding task. For the coding of the interview transcripts, a senior researcher
from the same department as the author, who is also familiar with the case
company, independently coded ten randomly selected statements using the coding
guideline. The inter-rater agreement of codes achieved a percentage score of
$83.8\%$ and a Cohen’s Kappa of $71.8\%$. However, Cohen’s Kappa is known to
be unreliable for uneven marginal distributions [9]. Still, the more robust
S-Score [1] resulted in an agreement of $82.3\%$. All scores represent a
substantial agreement and support the reliability of the coding process.

For the coding of the issue reports, a senior engineer from the case company
was involved. In a session between the author and the involved engineer,
questionable codes were reviewed and adjusted. This process confirmed the
applicability of the chosen codes as well as that 21 of the selected issues
were not caused by requirements quality defects but unrelated circumstances.
These 21 statements were not considered in future phases of the data analysis.

####  3.2.3 Theme identification

In this step, individual codes are aggregated into “more meaningful and
parsimonious units” [5]. The theme identification imposes two further
conditions on the coded statements: (C1) A statement must contain at least one
quality factor and at least one activity, and (C2) a statement must not
contain any unspecific activity (activity code: processing, root of Figure 3)
or an unspecific attribute (attribute code: unspecific). This rules out 44
vague interview statements of the form “quality factor is bad” (C1, missing
activity) or “quality factor is bad for implementing” (C2, unspecific
attribute).

The final analysis considered 40 interview and 80 issue statements. The amount
of information per mention differed between the two data sources. While the
interview encouraged participants to elaborate on all concepts of the RQT, the
issue data lacked the same level of control [22]. Issue statements always
contained the activity concepts since they explicitly report the effect of an
issue, but the level of information on the root cause in requirements
engineering was often limited, and the issues did not contain any information
about the context.

The analysis of the more granular and complete interview statements splits the
data into three groups: statements containing a single quality factor and no
context factors, statements containing multiple quality factors but no context
factors, and statements containing context factors. Within each group,
statements about the same quality factors are aggregated to collect all
impacted activities and attributes. Since the issue statements contained codes
on higher levels and no context factors, they were aggregated into one matrix
showing the distribution of quality factors impacting activity attributes.

####  3.2.4 Model creation

The final inferential step of the guideline by Cruzes et al. is the
description of higher-order themes, a taxonomy, model, or a theory [5].
Because this study is already grounded in the theoretical foundation provided
by the RQT [13], but none of the encountered data challenged this theory,
developing a new model or theory was not deemed constructive. Instead, this
study evolves the existing RQT by deriving propositions from the identified
themes. These propositions enrich the theory with empirical insights and
contribute falsifiable hypotheses for further research.

####  3.2.5 Trustworthiness Assessment

The final overall step of the guideline by Cruzes et al. is to assess the
trustworthiness of the synthesis [5]. As these concerns align with threats to
validity, they are addressed in Section 5.2.

##  4 Results

###  4.1 Interview data

####  4.1.1 Impact of single Quality Factors

The interview data contains information about 17 unique quality factors with
an impact on at least one subsequent activity. For brevity, Table 1 lists only
the four quality factors that were contained in at least two statements. Each
cell of the impact column in Table 1 lists the perceived direction and
strength of the impact that the quality factor has on the attribute of the
activity in this cell. For example, two statements of the interview data
described a negative impact of solution-oriented requirements on a unique
understanding of that requirement. One statement stressed that this impact is
major ( -3), the other one did not ( -2). The replication package contains the
remaining quality factors and their impact333Available at
https://github.com/JulianFrattini/rqi-relf/blob/main/src/analytics/results.md.

Table 1: Perceived impact of single quality factors on subsequent activities Quality Factor | Activity | Attribute | Impact
---|---|---|---
Solution-orientation | Understanding | Uniqueness | -3 -2
Verifying | Completeness | -2 -2
Effort Estimation | Traceability | +2
Translating | Stability | +2
Feasibility Assessment | Precision | +2
Planning | Stability | +2
Non-atomic | Translating | Duration | -2 -2
Planning | Stability | -2
Non-concise | Understanding | Uniqueness | -1
Duration | -2
Too dense | Understanding | Duration | -2
Interpreting | Uniqueness | -2
Verifying | Duration | -2

The table shows that the interview participants perceived the four most often
mentioned quality factors to impact a variety of activities and their
attributes. The most frequently mentioned quality factor is solution-
orientation, i.e., requirements that impose on the solution space rather than
elaborating on the problem space [10]. This quality factor is perceived to
cause misunderstandings (i.e., causing the understanding activity to be not
unique) and lack of coverage when deriving test cases (i.e., causing the
verifying activity to be not complete). The table also shows that some quality
factors have a mixed impact on different activities. For example, a solution-
oriented requirements specification is also perceived to aid effort
estimation, translating, and planning.

####  4.1.2 Interaction of multiple Quality Factors

The interview data contains information about four unique interactions between
quality factors. Table 2 lists the four interaction effects. The interview
participants reported that redundant requirements that were also not connected
through horizontal trace links (i.e., links between requirements) caused
incoherent implementations. Furthermore, non-functional requirements were
reported to be susceptible to ambiguous understanding when providing to little
details. Additionally, requirements specifications that were yet immature but
also already committed to were quicker to implement due to the applied time
pressure ( +2), but implementation became much less feasible ( -3). Finally,
the precision of feasibility assessment suffered from jargonic and dense
requirements.

Table 2: Interaction between two quality factors Quality Factor 1 |  Quality Factor 2 | Activity | Attribute | Impact
---|---|---|---|---
Redundancy |  Missing horizontal trace links | Implementing | Coherence | -2
Too little details |  Non-functional requirement | Understanding | Uniqueness | -2
Immature |  Committed | Implementing | Duration | +2
Feasibility | -3
Jargonic |  Too dense | Assessing Feasibility | Precision | -2

####  4.1.3 Interaction with Context Factors

The interview data contains information about seven unique interactions
between quality factors and context factors. The two most prominent
prominently perceived interaction effects involve the quality factor solution-
orientation and density. Figure 4 visualizes one statement describing the
interaction between the quality factor solution-orientation and context factor
involvement on the uniqueness attribute of the understanding activity. If the
stakeholder responsible for processing the requirement was also involved in
writing it, the impact on the understandability is mitigated. The remaining
interaction effects are detailed in our replication package but follow a
similar pattern: Context factors like involvement, experience, and
supplementary communication can mitigate the negative impact of quality
factors. Additionally, solution-oriented requirements exhibit an even stronger
positive impact on several activities like feasibility assessment and effort
estimation when the context of the requirement is new.

Figure 4: Interaction effect between solution-orientation and involvement on
the understanding activity

###  4.2 Issue reports

The issue data only allows to infer general relationships between quality
factors and activities. Table 3 lists the number of statements per
constellation of quality factor, activity, and attribute. The most prominent
impact of requirements quality that results in a reported issue is
completeness, i.e., caused by a missing requirement. This results mostly in
incorrect (i.e., bugs) or incomplete (i.e., missing features) implementations.
In rare cases, the understanding, interpreting, and verifying activity are
reported to be impacted. Behind completeness, the most often reported impact
is consistency and ambiguity.

Table 3: Requirements impact as recorded in the issue reports (Comp. = Completeness, Corr. = Correctness, Cons. = Consistency) Activity | Understanding | Interpreting | Implementing | Verifying
---|---|---|---|---
Attribute | Unique | Unique | Comp. | Corr. | Cons. | Coverage | Feasible
Quality Factor |  |  |  |  |  |  |
Completeness | 1 | 0 | 13 | 34 | 3 | 1 | 1
Consistency | 1 | 0 | 0 | 6 | 2 | 2 | 0
Ambiguity | 1 | 2 | 2 | 1 | 2 | 0 | 0
Correctness | 1 | 0 | 0 | 2 | 1 | 0 | 0
Feasibility | 0 | 0 | 0 | 1 | 0 | 1 | 0
Relevance | 0 | 0 | 1 | 0 | 0 | 0 | 0

###  4.3 Propositions

The triangulation of interview and issue analysis results allows to derive the
following propositions enhancing the RQT.

####  4.3.1 Relevant quality factors

The set of quality factors perceived and reported to be relevant in the case
company is very limited. Among the perceived quality factors, only solution-
orientation, lack of atomicity, lack of conciseness, and density received
support in at least two statements. Among the reported quality factors, lack
of completeness (i.e., missing requirements) stands out as the primary cause
of issues in the down-stream development process.

####  4.3.2 Mixed Impact

The analysis of the interview data shows that quality factors can have mixed
impact on different activities. For example, while a solution-oriented
requirement might negatively impact the activities of understanding and
verifying it, planning and translating activities become more stable.

####  4.3.3 Interactions matter

The analysis of the interview data shows that interactions between quality
factors but also between quality and context factors have a significant
effect. In particular, quality factors like the novelty of a feature, the
experience of the involved engineers, and supplementary communication mediate
the effect of quality factors as shown in Figure 4.

##  5 Discussion

###  5.1 Implications

####  5.1.1 Contribution to Research

The empirical evidence both strengthens existing requirements engineering
theories and guides further advances in requirements quality research. The
results strengthen the RQT [13, 8] in that artifact properties impacting
activity properties constitutes requirements quality. Moreover, the results
confirm that one quality factor may have different impacts on different
activities [8]. Incautiously removing a quality factor from an entity due to
the negative impact on one activity may, therefore, also mitigate its positive
impact on other activities. The results further strengthen the Naming the Pain
in Requirements Engineering (NaPiRE) initiative [16] by contributing more
granular evidence to the problems of RE relevant to practice. The results
agree with the conclusion of NaPiRE that missing requirements are among the
most impactful quality defects. Similarly, the results of both data sources of
the study agree with previous studies that the effect of ambiguity is less
relevant in practice than often assumed [20, 4]. Finally, the results support
the advocacy for context-sensitive research in empirical software engineering
[3] by emphasizing context factors as mediators of the impact of artifacts on
activities. The results of this study guide further research advances through
the approach of identifying relevant factors of requirements quality. The
study reduces the vast space of several hundred potential quality factors [14]
down to about 30 that are relevant to the specific context of an organization.

####  5.1.2 Impact on Practice

The operationalization of the RQT in practice becomes feasible due to the
reduction of variables to measure. The results steer the next step of research
with the case company toward detecting solution-oriented, non-atomic, non-
concise, dense, and incomplete requirements. Additionally, effort will be
focused on measuring the relevant context factors of involvement, experience,
novelty, and supplementary communication. These measurements enable the impact
estimation of requirements quality on dependent activities and advance
requirements quality research roadmaps [6, 13].

####  5.1.3 Limitations

One gap to overcome is the lack of an overview of requirements-dependent
activities and their measurable attributes, as also outlined in previous
research roadmaps [6, 13]. These attributes constitute the dependent variable
in the impact estimation implied by the RQT [13]. The interview data contains
44 statements with either an unclear activity or attribute impacted by a
quality factor. Developing a model of requirements-dependent activities and
their attributes is a necessary next step to achieve operationalization of the
RQT [13].

###  5.2 Threats to Validity

This section discusses threats to validity according to Runesson et al. [22]
and Wohlin et al. [26] and, additionally, addresses concerns of
trustworthiness of the thematic synthesis [5].

Regarding conclusion validity, the reliability of measures is a prevalent
threat given the subjective coding process but has been minimized by involving
independent raters and calculating inter-rater agreement where applicable.
Similarly, the lack of control over the issue data questions their
reliability. Involving a senior engineer from the case company mitigated this
issue, providing an adequate confirmability of the data.

Regarding internal validity, the interview data suffers from selection bias.
The interview participants were sampled by the industry contact. However, the
participants show a wide variation of background and experience, which leads
to assume that this threat is minimal.

Regarding construct validity, the study suffers from inadequate preoperational
explication of constructs, i.e., immaturity of some of the concepts of the
theoretical framework (the RQT). In particular, the activity group within the
RQT (activities and attributes) is insufficiently explored in requirements
quality research [6, 13]. As a consequence, several interview statements
failed to specify the activity and attribute impacted by requirements quality.
The threat was minimized by excluding this data from the analysis.

Regarding external validity, the inference of this study is not generalizable
or transferable [5] by design of the case study method. Additional research
replicating this study in other companies are necessary to generalize the
results.

##  6 Conclusion

This case study demonstrates the application of the requirements quality
theory (RQT) to identify relevant factors of requirements quality. By
analyzing both interview and issue report data, we identified 17 relevant
quality as well as 11 interaction effects among them and with context factors.
The study contributes empirical evidence to the relevance of these factors and
their effects in the case company. The study emphasizes that (1) some
requirements quality factors are more relevant in practice than others, (2)
they may have a simultaneous negative and positive impact on different
activities, and (3) context factors mediate their impact. This research
advances requirements quality research by advancing existing research roadmaps
[6] toward a quantified impact estimation of requirements quality in practice
[13].

## Acknowledgements

This work was supported by the KKS foundation through the S.E.R.T. Research
Profile project at Blekinge Institute of Technology. I thank Michael
Unterkalmsteiner for independently performing the coding task. Furthermore, I
owe great thanks to Parisa Yousefi, Charlotte Ljungman, and Fabiano Sato from
Ericsson for their continuous support that made this work possible in the
first place.

## References

  * [1] Bennett, E.M., Alpert, R., Goldstein, A.: Communications through limited-response questioning. Public Opinion Quarterly 18(3), 303–308 (1954)
  * [2] Boehm, B.W., Papaccio, P.N.: Understanding and controlling software costs. IEEE transactions on software engineering 14(10), 1462–1477 (1988)
  * [3] Briand, L., Bianculli, D., Nejati, S., Pastore, F., Sabetzadeh, M.: The case for context-driven software engineering research: generalizability is overrated. IEEE Software 34(5), 72–75 (2017)
  * [4] de Bruijn, F., Dekkers, H.L.: Ambiguity in natural language software requirements: A case study. In: Requirements Engineering: Foundation for Software Quality: 16th International Working Conference, REFSQ 2010, Essen, Germany, June 30–July 2, 2010. Proceedings 16. pp. 233–247. Springer (2010)
  * [5] Cruzes, D.S., Dyba, T.: Recommended steps for thematic synthesis in software engineering. In: 2011 international symposium on empirical software engineering and measurement. pp. 275–284. IEEE (2011)
  * [6] Femmer, H.: Requirements quality defect detection with the qualicen requirements scout. In: REFSQ Workshops (2018)
  * [7] Femmer, H., Fernández, D.M., Wagner, S., Eder, S.: Rapid quality assurance with requirements smells. Journal of Systems and Software 123, 190–213 (2017)
  * [8] Femmer, H., Mund, J., Fernández, D.M.: It’s the activities, stupid! a new perspective on re quality. In: 2015 IEEE/ACM 2nd International Workshop on Requirements Engineering and Testing. pp. 13–19. IEEE (2015)
  * [9] Feng, G.C.: Mistakes and how to avoid mistakes in using intercoder reliability indices. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences 11(1), 13 (2015)
  * [10] Fernandez, D.M., Wagner, S., Lochmann, K., Baumann, A., de Carne, H.: Field study on requirements engineering: Investigation of artefacts, project parameters, and execution strategies. Information and Software Technology 54(2), 162–178 (2012)
  * [11] Franch, X., Mendez, D., Vogelsang, A., Heldal, R., Knauss, E., Oriol, M., Travassos, G., Carver, J.C., Zimmermann, T.: How do practitioners perceive the relevance of requirements engineering research? IEEE Transactions on Software Engineering (2020)
  * [12] Franch, X., Palomares, C., Quer, C., Chatzipetrou, P., Gorschek, T.: The state-of-practice in requirements specification: an extended interview study at 12 companies. Requirements Engineering pp. 1–33 (2023)
  * [13] Frattini, J., Montgomery, L., Fischbach, J., Mendez, D., Fucci, D., Unterkalmsteiner, M.: Requirements quality research: a harmonized theory, evaluation, and roadmap. Requirements engineering (2023)
  * [14] Frattini, J., Montgomery, L., Fischbach, J., Unterkalmsteiner, M., Mendez, D., Fucci, D.: A live extensible ontology of quality factors for textual requirements. In: 2022 IEEE 30th International Requirements Engineering Conference (RE). pp. 274–280. IEEE (2022)
  * [15] Host, M., Runeson, P.: Checklists for software engineering case study research. In: First international symposium on empirical software engineering and measurement (ESEM 2007). pp. 479–481. IEEE (2007)
  * [16] Méndez, D., Wagner, S., Kalinowski, M., Felderer, M., Mafra, P., Vetrò, A., Conte, T., Christiansson, M.T., Greer, D., Lassenius, C., et al.: Naming the pain in requirements engineering: Contemporary problems, causes, and effects in practice. Empirical software engineering 22, 2298–2338 (2017)
  * [17] Miles, M.B., Huberman, A.M.: Qualitative data analysis: An expanded sourcebook. sage (1994)
  * [18] Montgomery, L., Fucci, D., Bouraffa, A., Scholz, L., Maalej, W.: Empirical research on requirements quality: a systematic mapping study. Requirements Engineering 27(2), 183–209 (2022)
  * [19] Petersen, K., Wohlin, C.: Context in industrial software engineering research. In: 2009 3rd International Symposium on Empirical Software Engineering and Measurement. pp. 401–404. IEEE (2009)
  * [20] Philippo, E.J., Heijstek, W., Kruiswijk, B., Chaudron, M.R., Berry, D.M.: Requirement ambiguity not as important as expected—results of an empirical evaluation. In: Requirements Engineering: Foundation for Software Quality: 19th International Working Conference, REFSQ 2013, Essen, Germany, April 8-11, 2013. Proceedings 19. pp. 65–79. Springer (2013)
  * [21] Rosadini, B., Ferrari, A., Gori, G., Fantechi, A., Gnesi, S., Trotta, I., Bacherini, S.: Using nlp to detect requirements defects: An industrial experience in the railway domain. In: Requirements Engineering: Foundation for Software Quality: 23rd International Working Conference, REFSQ 2017, Essen, Germany, February 27–March 2, 2017, Proceedings 23. pp. 344–360. Springer (2017)
  * [22] Runeson, P., Host, M., Rainer, A., Regnell, B.: Case study research in software engineering: Guidelines and examples. John Wiley & Sons (2012)
  * [23] Strauss, A., Corbin, J.: Basics of qualitative research. Sage publications (1990)
  * [24] Wagner, S., Fernández, D.M., Felderer, M., Vetrò, A., Kalinowski, M., Wieringa, R., Pfahl, D., Conte, T., Christiansson, M.T., Greer, D., et al.: Status quo in requirements engineering: A theory and a global family of surveys. ACM Transactions on Software Engineering and Methodology (TOSEM) 28(2), 1–48 (2019)
  * [25] Wohlin, C.: Case study research in software engineering—it is a case, and it is a study, but is it a case study? Information and Software Technology 133, 106514 (2021)
  * [26] Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in software engineering. Springer Science & Business Media (2012)