# Quantum fluctuations of energy in subsystems of a hot relativistic gas

Arpan Das
Wojciech Florkowski
Radoslaw Ryblewski
Rajeev Singh  Institute of Nuclear Physics Polish Academy of Sciences,
PL-31-342 Krakow, Poland  Institute of Theoretical Physics, Jagiellonian
University, PL-30-348 Krakow, Poland  Institute of Nuclear Physics Polish
Academy of Sciences, PL-31-342 Krakow, Poland  Institute of Nuclear Physics
Polish Academy of Sciences, PL-31-342 Krakow, Poland

(Received: date / Accepted: date)

###### Abstract

We derive a formula that defines quantum fluctuations of energy in subsystems
of a hot relativistic gas. For small subsystem sizes we find substantial
increase of fluctuations compared to those known from standard thermodynamic
considerations. However, if the size of the subsystem is sufficiently large,
we reproduce the result for energy fluctuations in the canonical ensemble. Our
results are subsequently used in the context of relativistic heavy-ion
collisions to introduce limitations of the concepts such as classical energy
density or fluid element. In the straightforward way, our formula can be
applied in other fields of physics, wherever one deals with hot and
relativistic matter.

##  1 Introduction

Fluctuations of various physical quantities play a very important role in all
fields of physics, as they reveal the information about possible phase
transitions [1, 2], formation of structures in the Early Universe [3, 4], and
dissipative phenomena [5]. Most common fluctuations we deal with are those
arising from quantum uncertainty relation or those present in thermodynamic
systems [6]. Our present study combines these two physics aspects — we study
quantum fluctuations of energy in subsystems of a hot relativistic gas and
demonstrate that they agree with thermodynamic fluctuations in canonical
ensemble if subsystems are sufficiently large. On the other hand, we find a
very substantial increase of the fluctuations for small subsystems.

The results of our numerical calculations may serve as a guide that shows, for
a given temperature and particle mass, how large the size of the subsystem is
for which the quantum fluctuations of energy become classical and eventually
may be neglected. This, in turn, may be used to validate the concept of energy
density used in classical description of fluids.

The physical system we are particularly interested in is hot matter produced
in relativistic heavy-ion collisions. The concept of a hot gas is dominantly
used here in various aspects: from perturbative quark-gluon-plasma possibly
produced at the top beam energies to hadron resonance gas produced at hadronic
freeze-out. Combined with the concept of hot gas is the use of relativistic
hydrodynamics which uses equation of state that is very often of the ideal-gas
form.

In fact, relativistic hydrodynamics has become now the main theoretical tool
used to interpret heavy-ion collisions. Its overwhelming successes paved the
road to its wider and wider applications — in the regions where its
applicability range might be questioned [7, 8, 9]. The basic concepts of
energy density and pressure are used now within hydrodynamics to characterize
very small portions of matter at extreme conditions. This leads us back to the
question of how well the energy of such small systems is well defined and how
seriously we can take hydrodynamic structures of the energy density profiles
which vary on a space scale that is a fraction of one Fermi.

The results presented below have two aspects: We first derive a compact
formula that defines quantum fluctuations in a subsystem of a hot relativistic
gas, then we apply this formula in the physical situations expected in
relativistic heavy-ion collisions. Below we use the West–Coast metric
$g_{\mu\nu}=\hbox{diag}(+1,-1,-1,-1)$. Three-vectors are shown in bold font
and a dot is used to denote the scalar product of both four- and three-
vectors, i.e., $a^{\mu}b_{\mu}=a\cdot b=a^{0}b^{0}-\bm{a}\cdot\bm{b}$.

##  2 Basic concepts and definitions

In this work we consider a subsystem $S_{a}$ of the thermodynamic system
$S_{V}$ described by the canonical ensemble characterized by the temperature
$T$ (or its inverse $\beta=1/T$). For sake of simplicity, we assume that the
system $S_{V}$ consists of spinless boson particles with mass $m$. The
characteristic volume of the system $S_{a}$ is always smaller than the volume
$V$ of the system $S_{V}$, and $V$ is sufficiently large to allow for doing
integrals over particle momenta (instead of sums imposed by otherwise commonly
used box periodic conditions).

With these assumptions in mind, we describe our system by a quantum scalar
field in thermal equilibrium. The field operator has the standard form [10]

$\phi(t,{\bm{x}})=\int\frac{d^{3}k}{\sqrt{(2\pi)^{3}~{}2\omega_{\bm{k}}}}\left(a_{\bm{k}}^{{\phantom{\dagger}}}e^{-ik\cdot
x}+a_{\bm{k}}^{\dagger}e^{ik\cdot x}\right),$ (1)

where $a_{\bm{k}}^{{\phantom{\dagger}}}$ and $a_{\bm{k}}^{\dagger}$ are
annihilation and creation operators, respectively, satisfying the canonical
commutation relations
$[a_{\bm{k}}^{{\phantom{\dagger}}},a_{\bm{k}^{\prime}}^{\dagger}]=\delta^{(3)}(\bm{k}-\bm{k}^{\prime})$,
whereas $\omega_{\bm{k}}=\sqrt{{\bm{k}}^{2}+m^{2}}$ is the energy of a
particle. To perform thermal averaging, it is sufficient to know the
expectation values of the products of two and four creation and/or
annihilation operators [11, 12, 13]

$\displaystyle\langle
a^{\dagger}_{{\bm{k}}}a_{{\bm{k}}^{\prime}}^{{\phantom{\dagger}}}\rangle$
$\displaystyle=\delta^{(3)}({\bm{k}}-{\bm{k}}^{\prime})f(\omega_{{\bm{k}}}),$
(2) $\displaystyle\langle
a^{\dagger}_{{\bm{k}}}a^{\dagger}_{{\bm{k}}^{\prime}}a_{{\bm{p}}}^{{\phantom{\dagger}}}a_{{\bm{p}}^{\prime}}^{{\phantom{\dagger}}}\rangle$
$\displaystyle=\bigg{(}\delta^{(3)}({\bm{k}}-{\bm{p}})~{}\delta^{(3)}({\bm{k}}^{\prime}-{\bm{p}}^{\prime})$
(3)
$\displaystyle+\delta^{(3)}({\bm{k}}-{\bm{p^{\prime}}})~{}\delta^{(3)}({\bm{k}}^{\prime}-{\bm{p}})\bigg{)}f(\omega_{{\bm{k}}})f(\omega_{{\bm{k}}^{\prime}}).$

Here $f(\omega_{{\bm{k}}})$ is the Bose–Einstein distribution function,
$f(\omega_{{\bm{k}}})=1/(\exp[\beta~{}\omega_{{\bm{k}}}]-1)$. Any other
combinations of two and four creation and/or annihilation operators can be
obtained from Eqs. (2) and (3) through the commutation relation between
$a_{{\bm{k}}}^{{\phantom{\dagger}}}$ and $a_{{\bm{k}}}^{\dagger}$.

Following [10], we define an operator $\mathcal{H}_{a}$ that describes the
energy density of a finite subsystem $S_{a}$ placed at the origin of the
coordinate system

$\displaystyle\mathcal{H}_{a}=\frac{1}{(a\sqrt{\pi})^{3}}\int
d^{3}{\bm{x}}~{}\mathcal{H}(x)~{}\exp\left(-\frac{{\bm{x}}^{2}}{a^{2}}\right).$
(4)

Here
$\mathcal{H}(x)\equiv(\dot{\phi}^{2}+({\bf{\nabla}\phi)}^{2}+m^{2}\phi^{2})/2$
is the Hamiltonian density of the free real scalar field. The Gaussian profile
used in Eq. (4) is in fact our definition of the subsystem $S_{a}$ – the
smooth profile with a length scale $a$ has been introduced instead of a cube
to remove possible boundary effects coming from sharp boundaries.

The thermal expectation value of the operator $\mathcal{H}_{a}$ is

$\displaystyle\langle:\mathcal{H}_{a}:\rangle=\int\frac{d^{3}{{k}}}{(2\pi)^{3}}~{}\omega_{{\bm{k}}}~{}f\left(\omega_{{\bm{k}}}\right)\equiv\varepsilon(T).$
(5)

This is certainly an expected result known from elementary kinetic-theory
considerations [6]. To remove an infinite vacuum part coming from zero-point
energy contributions, we have applied standard normal ordering procedure to
$\mathcal{H}_{a}$ on the left-hand side of Eq. (5). We note that the energy
density $\varepsilon$ defined by Eq. (5) is independent of $a$, which reflects
the spatial uniformity of the system $S_{V}$. Our results are also independent
of time.

To determine the fluctuation of energy of the subsystem $S_{a}$, in agreement
with general quantum-mechanics rules we consider the variation

$\sigma^{2}(a,m,T)=\langle:\mathcal{H}_{a}::\mathcal{H}_{a}:\rangle-\langle:\mathcal{H}_{a}:\rangle^{2}\,$
(6)

or the normalized standard deviation

$\sigma_{n}(a,m,T)=\frac{(\langle:\mathcal{H}_{a}::\mathcal{H}_{a}:\rangle-\langle:\mathcal{H}_{a}:\rangle^{2})^{1/2}}{\langle:\mathcal{H}_{a}:\rangle}.$
(7)

Using the thermal expectation values of the products of two and four creation
and annihilation operators, as given by Eqs. (2) and (3), we find

$\displaystyle\sigma^{2}(a,m,T)$ $\displaystyle=\int
dK~{}dK^{\prime}f(\omega_{{\bm{k}}})(1+f(\omega_{{\bm{k}}^{\prime}}))$
$\displaystyle\times\bigg{[}(\omega_{{\bm{k}}}\omega_{{\bm{k}}^{\prime}}+{\bm{k}}\cdot{\bm{k}}^{\prime}+m^{2})^{2}e^{-\frac{a^{2}}{2}({\bm{k}}-{\bm{k}}^{\prime})^{2}}$
$\displaystyle+(\omega_{{\bm{k}}}\omega_{{\bm{k}}^{\prime}}+{\bm{k}}\cdot{\bm{k}}^{\prime}-m^{2})^{2}e^{-\frac{a^{2}}{2}({\bm{k}}+{\bm{k}}^{\prime})^{2}}\bigg{]}.$
(8)

where $dK=d^{3}{{k}}/((2\pi)^{3}2\omega_{{\bm{k}}})$. In Eq. (8) we have
discarded a divergent term that is temperature independent and may be
attributed to the pure vacuum energy fluctuation. As a matter of fact, the
latter was studied in Ref. [14], where a technique of smeared and displaced
operators was used. The variation of the energy density divided by the mean
energy density squared was found in this case to be 2/3. As this result
corresponds to the limit $a\to 0$ for $m=0$, it is negligible compared to our
value of $\sigma^{2}_{n}$ that diverges as $a\to 0$. On the other hand, using
dimensional analysis one expects that the variation of the vacuum energy
density for massless particles in a volume of the size $a^{3}$ should decrease
as $1/a^{8}$, which makes it again smaller compared to our effects that
decrease as $T^{5}/a^{3}$, which will be demonstrated below. Altogether, we
expect that the inclusion of the properly regularized vacuum fluctuations into
Eq. (8) cannot quantitatively alter our conclusions.

Equation (8) represents our main result that allows us to determine the energy
fluctuations of “Gaussian” subsystems $S_{a}$ of the system $S_{V}$. By
numerical integration we may obtain the results for any subsystem size $a$,
temperature $T$, and for particle mass $m$. Nevertheless, before we present
our numerical results, it is instructive to consider the limiting case of very
large system size $a$ and to analyze special cases for which analytic results
can be obtained. It is also useful to generalize our formulas to the case
where we deal with several particle types.

##  3 Degeneracy factor

Studying thermodynamic properties of particles one usually introduces
degeneracy factors connected with so-called internal degrees of freedom such
as spin, isospin or color charge. In our approach, we have so far considered
only one type of boson particle without any internal quantum numbers. To take
into account $g$ copies of such particles, one has to include $g$ copies of
the scalar field expressed by the creation and annihilation operators that
commute for different particle species. Altogether this procedure results in
the simple replacements

$\varepsilon\rightarrow g\varepsilon,\quad\sigma^{2}\rightarrow g\sigma^{2}.$
(9)

As the multiplication of the energy density by $g$ is rather straightforward,
the scaling of $\sigma^{2}$ with $g$ is less obvious — it is a consequence of
the fact that the mixed terms describing different types of particles in Eq.
(6) vanish.

Figure 1: Variation of the normalized energy density fluctuation $\sigma_{n}$
in the subsystem $S_{a}$ with the length scale $a$ for different values of the
temperature $T$ and fixed particle mass $m=0.3$ GeV. Figure 2: Same as Fig. 1
but for fixed temperature $T~{}=~{}0.15$ GeV and different particles masses.

##  4 Thermodynamic limit

As we have mentioned earlier, $S_{a}$ is a subsystem of the system $S_{V}$. We
expect that in the limit $a\rightarrow\infty$ (still with $a^{3}\ll V$) our
formula for quantum fluctuation is reduced to that known from classical
statistical mechanics [6]. To check this property we use in Eq. (8) the
Gaussian representation of the three dimensional Dirac delta function

$\displaystyle\delta^{(3)}({\bm{k}}-{\bm{p}})=\lim_{a\to\infty}\frac{a^{3}}{(2\pi)^{3/2}}e^{-\frac{a^{2}}{2}({\bm{k}}-{\bm{p}})^{2}}.$
(10)

This leads us to the formula valid in the large $a$ limit

$\displaystyle\sigma^{2}$
$\displaystyle\sim\frac{g}{(2\pi)^{3/2}a^{3}}\int\frac{d^{3}{{k}}}{(2\pi)^{3}}~{}\omega_{{\bm{k}}}^{2}~{}f(\omega_{{\bm{k}}})(1+f(\omega_{{\bm{k}}})).$

We note that for massless particles $\sigma^{2}\sim T^{5}/a^{3}$, which is the
result mentioned above. The right-hand side of the last equation can be
expressed in terms of the specific heat at constant volume

$c_{V}=\frac{d\varepsilon}{dT}=\frac{g}{T^{2}}\int\frac{d^{3}{{k}}}{(2\pi)^{3}}~{}\omega_{{\bm{k}}}^{2}~{}f(\omega_{{\bm{k}}})(1+f(\omega_{{\bm{k}}})).$
(11)

Therefore, we find (again in the large $a$ limit)

$\displaystyle
V_{a}\sigma_{n}^{2}=\frac{T^{2}c_{V}}{\varepsilon^{2}}=V\frac{\langle
H^{2}\rangle-\langle H\rangle^{2}}{\langle H\rangle^{2}}\equiv
V\sigma^{2}_{H},$ (12)

where $V_{a}=a^{3}(2\pi)^{3/2}$ and $H$ is the Hamiltonian of $S_{V}$. The
right-hand side of Eq. (12) may be identified as the normalized energy
fluctuation in the system $S_{V}$ [6]. One may identify
$V_{a}=a^{3}(2\pi)^{3/2}$ as the volume of the subsystem $S_{a}$ — a
nontrivial factor of $(2\pi)^{3/2}$ is an artifact of using the “Gaussian”
box. We note that Eq. (12) is also consistent with the result obtained in Ref.
[15], where purely classical concepts have been used.

##  5 Massless case with Boltzmann statistics

In principle, the last result can be alternatively shown by first finding an
analytic expression for $V_{a}\sigma_{n}^{2}$ and then finding its asymptotic
form for $a\rightarrow\infty$ limit. Unfortunately, in the general case of
finite particle mass and Bose-Einstein statistics, the integral (8) is not
analytic and such procedure cannot be easily accomplished. However, the
integrals appearing in Eq. (8) become analytic for the Boltzmann statistics
and $m=0$. In this case we obtain

$\langle:\mathcal{H}_{a}:\rangle^{2}=\frac{9g^{2}}{\pi^{4}}T^{8}\,,$ (13)

and

$\displaystyle\sigma_{n}^{2}$
$\displaystyle=\frac{1}{4320\,g}\bigg{[}2970~{}\zeta^{4}-540~{}\zeta^{6}-96~{}\zeta^{8}-28~{}\zeta^{10}$
$\displaystyle-2~{}\zeta^{12}+\sqrt{2\pi}~{}e^{\frac{\zeta^{2}}{2}}\bigg{(}1485~{}\zeta^{3}-765~{}\zeta^{5}+300~{}\zeta^{7}$
$\displaystyle+60~{}\zeta^{9}+15~{}\zeta^{11}+\zeta^{13}\bigg{)}\text{erfc}\bigg{(}\frac{\zeta}{\sqrt{2}}\bigg{)}\bigg{]},$
(14)

where $\zeta=1/(aT)$. Using this result we find

$\displaystyle\lim_{a\to\infty}V_{a}\sigma_{n}^{2}=\frac{11\pi^{2}}{8g}\frac{1}{T^{3}},$
(15)

which is the normalized energy fluctuation $V\sigma^{2}_{H}$ of the system
$S_{V}$ as can be obtained from Eq. (12) for Boltzmann distribution functions
with $m=0$. We emphasize that for $a\rightarrow\infty$ the analytic result for
the case $m=0$ and Boltzmann statistics agrees with the result obtained
earlier with the Gaussian approximation for the Dirac delta function.

##  6 Numerical results

Having checked that our expression for quantum fluctuation correctly
reproduces the thermodynamic limit for $a\rightarrow\infty$ and with the
massless-Boltzmann case known analytically, we can present the results of our
numerical calculations. In Figs. (1) and (2) we show the variation of the
normalized energy density fluctuation $\sigma_{n}$ with the subsystem size $a$
for different values of temperature and mass, respectively. With possible
interpretation of heavy-ion data in mind, we consider temperatures in the
range 100 MeV $<T<$ 250 MeV, and particle masses: $m~{}=~{}0,~{}300$ and 1000
MeV. The effective number of degrees of freedom in a perturbative quark-gluon-
plasma varies between 37 (for two quark flavors) and 47.5 (for three quark
flavors). In a hot hadron gas used to describe the properties of matter at
freeze-out one includes about 400 hadronic states [16] but these are usually
very heavy and equivalent to a smaller number of lighter particles (in terms
of energy density or pressure). In order to get some rough estimate of $g$ we
use the value $g=40$.

Figures (1) and (2) show that $\sigma_{n}$ decreases with increasing size of
the system $a$, which is the expected general behavior of fluctuations. By the
way, we note that Eq. (8) implies that $\sigma_{n}$ diverges in the limit
$a\to 0$ (which is a consequence of uncertainty relation), so our plots start
only at $a=0.3$ fm. For fixed particle mass, the normalized fluctuations
decrease with growing temperature. On the other hand, at fixed $T$ the
fluctuations grow with $m$. Interestingly, the standard deviations alone
exhibit an opposite behavior (not shown here).

Figure 3: Variation of the normalized energy fluctuation in the subsystem
$S_{a}$ with the length scale $a$ for different values of the temperature at
$m=0.3$ GeV.  Figure 4: Variation of the normalized energy fluctuation in the
subsystem $S_{a}$ with the length scale $a$ for different values of mass at
$T=0.15$ GeV.

We have analytically demonstrated above that the volume scaled normalized
fluctuation $V_{a}\sigma_{n}^{2}$ approaches thermodynamic limit in the large
volume limit for massless particles. Figs. (3) and (4) present the variation
of $V_{a}\sigma_{n}^{2}/V\sigma^{2}_{H}$ with the size of the subsystem
$S_{a}$ in the case where particles have a non vanishing mass and they obey
Bose-Einstein statistics. The values of the parameters are the same as in
Figs. (1) and (2). From Eq. (12) one expects that in the thermodynamic limit
$V_{a}\sigma_{n}^{2}/V\sigma^{2}_{H}$ should approach unity. This property is
nicely seen in Figs. (3) and (4), where we observe that the quantum
fluctuations agree with the thermodynamic ones already for $a>$ 1 fm. On the
other hand, the quantum fluctuations become very important at the scale of 0.1
fm, which excludes classical treatment of such small subsystems. It also
suggests that quantum fluctuation should be combined in future works with
hydrodynamic fluctuations [17].

For given values of $T$, $m$, and $g$, our results may be used to quickly
check if the corresponding fluctuations (quantum and thermodynamic ones) in a
finite subsystem of the size $a^{3}$ are sufficiently small, let us say,
smaller than 1. If this condition is not satisfied, the classical picture of a
well defined energy density in the gas/fluid cells of the size $a^{3}$ is not
well defined. For example, for massless particles at the temperature 150 MeV
(with $g=40$), the normalized fluctuations become larger than unity for
$a<0.4$ fm. This suggests that for smaller sizes the effects of fluctuations
become relevant for the system’s description. The classical description with
“well defined energy density” makes sense only after coarse graining over the
scales larger that 0.4 fm.

##  7 Conclusions

In this work we have derived the formula characterizing the quantum
fluctuation of energy in subsystems of a hot relativistic gas. We have shown
that it agrees with the expression for thermodynamic fluctuations, if the size
of the subsystem is sufficiently large. We have found exact analytic
expression for the energy fluctuation in the case of massless particles
described by Boltzmann statistics. The consequences one can draw from our
formula for description of hot and relativistic systems have been discussed in
the case of heavy-ion physics. Applications of our approach to other systems
are straightforward.

We would like to thank Krzysztof Golec-Biernat and Mariusz Sadzikowski for
very useful and illuminating discussions. This research was supported in part
by the Polish National Science Centre Grants No. 2016/23/B/ST2/00717 and No.
2018/30/E/ST2/00432.

## References

  * [1] M. Smoluchowski, Bulletin international de l’Académie des sciences de Cracovie pp. 493–502 (1911)
  * [2] S. Jeon, V. Koch, Phys. Rev. Lett. 85, 2076 (2000).  10.1103/PhysRevLett.85.2076
  * [3] E. Lifshitz, I. Khalatnikov, Adv. Phys. 12, 185 (1963).  10.1080/00018736300101283
  * [4] A.H. Guth, S.Y. Pi, Phys. Rev. Lett. 49, 1110 (1982).  10.1103/PhysRevLett.49.1110
  * [5] R. Kubo, Rep. Prog. Phys. 29, 255 (1966)
  * [6] K. Huang, _Statistical Mechanics_ , 2nd edn. (Wiley New York, 1987)
  * [7] A. Jaiswal, V. Roy, Adv. High Energy Phys. 2016, 9623034 (2016).  10.1155/2016/9623034
  * [8] W. Florkowski, M.P. Heller, M. Spalinski, Rept. Prog. Phys. 81(4), 046001 (2018).  10.1088/1361-6633/aaa091
  * [9] P. Romatschke, U. Romatschke, _Relativistic Fluid Dynamics In and Out of Equilibrium_.  Cambridge Monographs on Mathematical Physics (Cambridge University Press, 2019).  10.1017/9781108651998
  * [10] S. Coleman, _Lectures of Sidney Coleman on Quantum Field Theory_ (WSP, Hackensack, 2018).  10.1142/9371
  * [11] C. Cohen-Tannoudji, B. Diu, F. Laloë, S.R. Hemley, _Quantum mechanics: Vol. 3: fermions, bosons, photons, correlations and entanglement_.  A Wiley-Interscience publication (Wiley, New York, 1977)
  * [12] C. Itzykson, J. Zuber, _Quantum Field Theory_.  International Series In Pure and Applied Physics (McGraw-Hill, New York, 1980)
  * [13] T. Evans, D.A. Steer, Nucl. Phys. B 474, 481 (1996).  10.1016/0550-3213(96)00286-6
  * [14] N. Phillips, B. Hu, Phys. Rev. D 62, 084017 (2000).  10.1103/PhysRevD.62.084017
  * [15] S. Mrowczynski, Phys. Rev. C 57, 1518 (1998).  10.1103/PhysRevC.57.1518
  * [16] A. Kisiel, T. Taluc, W. Broniowski, W. Florkowski, Comput. Phys. Commun. 174, 669 (2006).  10.1016/j.cpc.2005.11.010
  * [17] J. Kapusta, B. Muller, M. Stephanov, Phys. Rev. C 85, 054906 (2012).  10.1103/PhysRevC.85.054906


Conceptual reasoning, the ability to reason in abstract and high-level perspectives, is key to generalization in human cognition. However, limited study has been done on large language models' capability to perform conceptual reasoning. In this work, we bridge this gap and propose a novel conceptualization framework that forces models to perform conceptual reasoning on abstract questions and generate solutions in a verifiable symbolic space. Using this framework as an analytical tool, we show that existing large language models fall short on conceptual reasoning, dropping 9% to 28% on various benchmarks compared to direct inference methods. We then discuss how models can improve since high-level abstract reasoning is key to unbiased and generalizable decision-making. We propose two techniques to add trustworthy induction signals by generating familiar questions with similar underlying reasoning paths and asking models to perform self-refinement. Experiments show that our proposed techniques improve models' conceptual reasoning performance by 8% to 11%, achieving a more robust reasoning system that relies less on inductive biases.

§ INTRODUCTION

Language models have progressed rapidly in recent years after community research into the emergent abilities in models with billions of parameters and trillions of pre-training tokens. While such models achieve admirable performances on many reasoning benchmarks, people have been looking into their decision-making process and trying to understand why large language models (LLM) perform well, especially compared with smaller models. One of the popular theories regarding why LLMs work is that they can memorize a huge amount of facts from pre-training data and generalize to new queries through induction on semantics [Tang et al., 2023, Li et al., 2024] (i.e., associate new inputs with memorized facts and guess the most likely outputs). Induction-based inference in LLMs works well for most daily tasks since these tasks are usually familiar to the model and contain strong induction signals. However, heavily relying on induction-based inference limits the model from performing well on non-induction tasks that require complex reasoning and planning, such as temporal reasoning [Zhou et al., 2021] and travel planning [Xie et al., 2024]. Moreover, relying on induction signals will inevitably lead to biased reasoning, where models perform much worse on queries containing unfamiliar semantics [Li et al., 2024].

A significant gap between LLM's induction-based reasoning and human reasoning, which does not have the issues above, is the ability to perform conceptual and abstract reasoning. Humans develop the ability to reason on a conceptual level from a very young age [Markovits et al., 2015, Walker, 2015], which enables them to overcome the limitations of induction through analogy and planning. For example, when humans travel to a new city, they can devise a plan to find good restaurants based on analogical experiences from the places they are familiar with because they can associate the underlying problem-solving processes on a conceptual level. However, there have been limited attempts to study LLMs' abstract reasoning because 1) it is difficult to force LLMs to perform abstract reasoning, and 2) it is challenging to verify the correctness of models' abstract reasoning process.

This work proposes a novel conceptualization framework to investigate and improve LLM's abstract reasoning capability. The conceptualization framework is composed of two parts, which address the challenges mentioned above respectively: 1) a question abstraction process that removes as much induction signal from the original question by replacing specific special nouns (e.g., named entities, numerical values) with their fine-grained semantic types; 2) a symbolic program space where LLMs will generate their abstract reasoning solutions in so that we can verify their correctness by executing the program with the necessary information from the original question and check for denotation accuracy. We use a state-of-the-art language model to perform the question abstraction step so that all downstream models work with a consistent set of high-quality abstract questions. Then, all downstream models are given the exact instructions regarding the design of the symbolic program space and try to generate solutions based on only the conceptualized question. In <ref>, we show that existing LLMs' performances are much lower in the abstract reasoning setup, dropping 9% to 28% across different models compared to chain-of-thought (CoT) [Wei et al., 2022] that entirely rely on induction, indicating that existing LLMs lack the core reasoning capability to generalize on a conceptual level. This also suggests that while some models have seemingly close performances on various datasets, their underlying reasoning capability varies greatly as some gaps are dramatically larger than others.

We then propose improvement techniques for LLM conceptual reasoning. Our main motivation is finding high-quality induction signals and using them as a reference to the conceptual reasoning process. This approximates the human analogy processes that symbolically associate familiar problem-solving processes from past experiences. Specifically, we find concrete questions that can be reduced to the same conceptualized question but with familiar nouns. These questions are often directly answerable because the supporting evidence commonly appears in the pre-training data, and the LLM answer is more likely to be correct. We propose two ways to use such similar questions. They can be used as a run-time exam to identify high-performing candidate solutions, and we can infer final predictions based only on them. In addition, we can ask LLMs to self-refine the candidate programs based on CoT solutions to the mistakes they make. Experiments show the effectiveness of the proposed methods, which improve 8% to 11% on a wide range of reasoning benchmarks. Our proposed conceptual reasoning method achieves relatively comparable performances with CoT in some scenarios while being more robust and less biased because it produces symbolic solutions without heavily relying on induction signals and makes predictions based on deterministic executions of such symbolic solutions.

§ RELATED WORKS

Many works have identified that large language models

cannot handle tasks that require intensive reasoning [Tang et al., 2023, Saparov & He, 2023, Feng et al., 2023, Li et al., 2024, Xu et al., 2024]. Our work builds on these works and identifies high-level and conceptual reasoning as a major bottleneck shared among these observations, limiting models from system-2 reasoning  [Kahneman, 2011, Weston & Sukhbaatar, 2023, Lin et al., 2024]. This work is also related to decomposition-based reasoning [Min et al., 2019, Zhou et al., 2022], and program-based inference methods [Chen et al., 2023, Gao et al., 2023, Lu et al., 2024], where these methods aim to shift from model performing probabilistic induction to symbolic deduction, similar to our motivations. This work is marginally relevant to other LLM self-revision works [Chen et al., 2024, Madaan et al., 2024], but we consider different signals.

§ CONCEPTUALIZATION FRAMEWORK

An overview of the conceptualization process. From an original question $Q$, GPT-4 generates an abstract question $Q_{abs}$ and corresponding parameters through a conceptualization process, and a downstream LLM will try to generate a symbolic solution based on only $Q_{abs}$, which is later executed with the actual parameters.

This section introduces how the conceptualization framework is defined and implemented. For a natural question $Q$, we first employ a question conceptualization step that transforms $Q$ to $Q_{abs}$, an abstract version of the question. We then define a symbolic solution space using Python programs. At inference time, the downstream model generates a symbolic solution in our defined space based on only $Q_{abs}$, which is later deterministically executed with the concrete input parameters to generate a final prediction to the original question $Q$. This process is overviewed in Fig. <ref>.

§.§ Question Conceptualization

The goal of the question conceptualization step is to replace certain special nouns with their fine-grained semantic types so that humans and systems can develop a general solution that works for any specific nouns that fall under the semantic types. The primary challenge in this step is the tradeoff between solvability and generalizability. Even though all conceptualized questions are solvable with an unlimited program length (i.e., assuming a solution that considers all possible corner cases), it is often impractical and less solvable if we conceptualize too many concrete nouns in the question. On the other hand, if we remove too few nouns, models can still take the semantic shortcuts [Li et al., 2024] based on inductive biases. However, we emphasize that a balanced solution between these two tradeoffs is ideal but not required for our benchmark to be valid since models should, in theory, work on any level of conceptualization as long as the fine-grained semantic types are correct and the question is readable. We verify this later in <ref>.

To balance this tradeoff, we use one of the most capable large language models, GPT-4, and prompt it with carefully designed few-shot examples that teach GPT-4 which nouns to replace and what fine-grained semantic types they should be replaced with. We use six demonstrations, selected manually from the StrategyQA [Geva et al., 2021] dev set. We list the prompt in <ref>. Given the original question $Q$, GPT-4 generates the abstract question $Q_{abs}$, a list of replaced noun phrases with their semantic types and corresponding Python types, and their concrete parameters representing the actual values from the original question $Q$.

§.§ Symbolic Program Space

We design a symbolic space based on Python programs to accommodate solutions for general binary (yes/no) questions. It includes several pre-defined helper functions to facilitate information retrieval and relational operations.

§.§.§ Information Retrieval

We introduce a pre-defined helper function called $\mathrm{ask\_llm(query, type)}$ to handle all cases that require external knowledge. As the function name suggests, it accepts a string query that is often a short question, asks that question to the same downstream LLM that generates the program, and forces it to return the answer in the expected type. To ensure that the returned results are in the expected type, we employ a few-shot prompt that maps natural language queries to JSON structures, followed by a type casting. If the type casting fails, the retriever will retry ten times or raise an exception. We list this helper function with the few-shot examples in <ref>. Because this design separates reasoning and knowledge, future works can adopt other retrieval methods with minimal effort.

§.§.§ Relational Operation Replacement

We implement a special post-processing step to handle mismatches in relational operators. For example, consider a program that needs to find the relationship between two people and check if they are parent-child. It will likely generate a query to the $\mathrm{ask\_llm(query, type)}$ helper function in the form of “What is the relationship between person-1 and person-2?”, and the helper function may return multiple ways of describing the same relation, such as “father-son” or “mother-daughter”, or “descendent.” Python built-in equal operator $==$ cannot handle such fuzzy matches, so the execution may not lead to the correct results.

Operator     Query for Soft Operator

a == b     Consider the implied value, is a roughly the same as b?

a $>$ b     Consider the implied value, is a roughly larger than b?

a $<$ b     Consider the implied value, is a roughly smaller than b?

a in b     Considered the implied value, is a included or mentioned by the list b?

Queries to replace relational operators with soft operators. Other operations are compositions with the listed operations.

To address this, we propose soft relational operators to replace built-in operators, including $>, >=, <, <=, ==, !=, in$, and $not\ in$. Each soft operator translates a compiler-level relational operation to a soft query to large language models. The specific queries that are used to replace the built-in operators are listed in Table <ref>.

We implement actual helper functions that correspond to each operation. For example, we implement a helper function named $\mathrm{gt}(a, b)$ that calls $\mathrm{ask\_llm(\cdot)}$ with the queries for $>$ operations in Table <ref>. Then, we use Python's abstract syntax trees[<https://docs.python.org/3/library/ast.html>] and recursively find BoolOps that fall under our interested relational operators and replace them with calls to our implemented helper functions.

§.§ Symbolic Program Execution

As mentioned above, each question $Q$ is transformed to $Q_{abs}$ and a set of parameters with concrete values, as shown in Fig. <ref>. In addition, each generated symbolic solution will have a function named $\mathrm{answer}(\cdot)$, which accepts the parameters as input values and returns a prediction string. At inference time, we prepend the actual implementations of our helper functions, including $\mathrm{ask\_llm(\cdot)}$ and all relational helper functions to the generated symbolic solution. We also append a function call to the $\mathrm{answer}(\cdot)$ function with the parameters from the original question and print the returned value from this function call. Finally, we execute this generated Python file and use the printed string as the final prediction of the model.

§.§ Applications of the Conceptualization Framework

We discuss two applications of the framework: evaluation and unbiased reasoning.

§.§.§ Analytical Evaluation

Our proposed conceptualization framework can be an analytical tool to check if existing LLMs understand the problem-solving process or are simply taking semantic shortcuts [Li et al., 2024]. At inference time, a downstream model is provided with $Q_{abs}$, and few-shot demonstrations defining the available helper functions and program space. The model then generates K candidate programs, which are later executed with the corresponding input parameters and then used to majority vote a final answer based on the executed results. All downstream models are provided with the same $Q_{abs}$ and few-shot demonstrations and executed with the same input parameters.[$Q_{abs}$ and the input parameters are generated by GPT-4 as shown in Fig. <ref>, and an expert writes the few-shot examples.] We detail the prompts used for generating candidate program solutions in <ref>.

This inference process forces the model not to reason with memorized knowledge from particular nouns in the original question but to devise a solution based on the question's underlying reasoning path on an abstract level. Ideal reasoners should show relatively small gaps between performing conceptual reasoning and direct inferences (e.g., CoT).

§.§.§ Unbiased Reasoning

Comparison of a direct inference method (left) and conceptual inference (right). Results are taken from gpt-3-turbo.

Taking one step further with the analytical settings, we can use the conceptualization framework to perform unbiased reasoning. This is based on observations that LLMs often make stubborn mistakes if certain word combinations strongly hint toward an incorrect answer. For example, as Fig. <ref> shows, GPT-3.5 makes a stubborn mistake with direct inference on the left. This is because the model strongly memorizes prime numbers as a mathematical concept, so it cannot correctly associate that with sports. As a result, even though the model correctly identifies the need to find numbers in the sports' rules and scoring systems, it still cannot derive the correct answer.

On the other hand, if the model follows our proposed conceptual inference process as shown on the right of Fig. <ref>, it can correctly generate an unbiased solution since it cannot see the specific names of the sport and the numerical property, which can later be executed to find the correct answer. That being said, our proposed conceptualization framework can be applied for unbiased and neutral model reasoning, less affected by induction signals that might be irrelevant to problem-solving.

§ CONCEPTUAL REASONING WITH ANALOGY

As mentioned in <ref>, the conceptualization framework can perform unbiased reasoning by removing most induction signals from the input question and forcing the model to perform abduction and deduction. This motivates us to discuss how we can improve the performance of conceptual reasoning and ideally reach the performances of direct inference methods, which have full access to induction signals such as entity names.

This section proposes a method to inject trustworthy induction signals back to the reasoning process. Here, we emphasize that compared to a direct inference method that uses inductions from the original question, which are unreliable when the question differs much from pre-training distribution, our key motivation is to add induction signals familiar to the language model itself and, hence, more trustworthy. Specifically, for $Q_{abs}$, we generate a list of concrete questions with familiar nouns that can be conceptualized into the same abstract question. We use CoT to answer these questions and keep only those that can be confidently answered through self-consistency.

This approximates the human analogy process: when we encounter a new situation, we often resort to familiar situations from personal experiences and use what we did back then to solve the current task.

In <ref>, we introduce how we find these trustworthy similar questions that language models can accurately answer. In <ref>, we describe the first method of injecting such induction signals into the conceptual inference process, namely using them to select better candidate programs. In <ref>, we introduce a self-refinement method using these questions as feedback.

§.§ Similar Question Acquisition

§.§.§ Question Generation

To acquire the maximum diversity and correctness, we design a novel pipeline process to generate similar questions based on an abstract question $Q_{abs}$, with the goal of generating questions with similar underlying reasoning paths and solvable by LLMs directly. This similar question generation pipeline is performed with the same downstream LLM that generates the programs so that we ensure that the reasoning process does not involve a more capable model such as GPT-4.

We first generate a list of concrete nouns/entities for each semantic type. For example, consider Is City X on Coast Y?, the abstract version of Is Miami on the American West Coast? we ask the downstream LLM to generate possible City X's. It will generate a list of city names familiar to the LLMs, such as Hong Kong, New York, and London. Similarly, it will generate a list of Coast Y's similar to American West Coast, such as the Persian Gulf Coast, Baltic Coast, and the Black Sea Coast. During this process, we provide it with the nouns in the original question to ensure that the generated nouns are roughly in the same scope as the original entity. This way, the model will not generate a small coast such as Rockport Coast, which is too small to accommodate any cities.

For each of the generated concrete nouns, we prompt the language model to generate a statement corresponding to either a positive answer or a negative answer to the original question.[This is because we only consider binary questions in this work.] For example, given the newly generated entity “Hong Kong” for City X, we design a few-shot prompt for the language model to generate positive statements with concrete Coast Y, such as Hong Kong is on China's southern coast, as well as negative statements such as Hong Kong is not on the French Riviera. We then transform these statements back to a question in the form of the original question, which is Is Hong Kong on China's southern coast? and Is Hong Kong on the French Riviera?, respectively. At the same time, the model will also generate corresponding input parameters, such as city_x=“Hong Kong”, coast_y=“China's southern coast”. Prompts used in this step are listed in <ref>.

§.§.§ Question Validation

Agreement     Accuracy     % Remaining

None     81.1     100

8/10     89.7     73

10/10     97.2     47

Sanity-check experiment on StrategyQA dev set. This compares the performance and remaining instances if we only evaluate instances where a certain number of predictions out of 10 samples agree. Experiments shown use Llama-70B-instruct and CoT.

We employ a verification process to select those that can be confidently answered and are likely to be more trustworthy by finding silver labels for each generated question with CoT. We run the chain-of-thought inference ten times for each question and keep the questions where at least 8 out of the 10 samples agree with each other.

To motivate this, we conduct a sanity-check experiment on StrategyQA with Llama-70B-instruct.[<https://huggingface.co/upstage/Llama-2-70b-instruct>] As Fig. <ref> shows, there is a significant improvement in performance if we only evaluate instances where CoT predictions agree with each other over a certain amount of times. This suggests that the labels we acquire after enforcing such an agreement are more reliable. On the other hand, an 8/10 agreement provides a good tradeoff between accuracy and the number of remaining instances, which will save computational costs.

§.§ Program Selection

This section describes a straightforward method to apply these generated questions and their silver labels to improve model conceptual reasoning, namely a selection method that finds candidate programs that are likely to be better than the others. Specifically, within each question, we execute the K candidate programs generated in <ref> on all generated similar questions and evaluate each program's accuracy on the corresponding silver labels. We then rank the programs based on their accuracies and predict the original question based on the top programs' execution results. We employ a majority-vote process if multiple programs have the same accuracy. If there is a tie in the majority vote, we perform a weighted vote with all programs using their accuracies as the weight.

§.§ Program Self-refinement

An overview of the self-refinement process with a concrete example. The initial program does not consider that some animals can change their color for camouflage purposes, as pointed out in the generate similar question's CoT solution. Based on this information, the model can refine the initial program to include such consideration.

Using similar questions as a program selection method does not fully utilize their potential. This is because the reasoning processes from chain-of-thought are often correct on the generated similar questions because of the strong induction signal. As a result, if a program fails on certain similar questions, the CoT outputs serve as great feedback for LLMs to self-refine the candidate program. As overviewed in Fig. <ref>, we propose a self-refinement process to ask LLMs to generate a better program based on an existing program and natural language feedback from CoT on why the existing program fails on a similar question. Because of computational limitations, we only conduct this process on programs with top accuracies within each question. We join the new programs with the original programs and conduct the same inference process described in <ref>.

§ EXPERIMENTS

System     BoolQ     CsQA     Hotpot     Strat.dev     Strat.test     All     Exe.%

GPT-3.5 CoT     72.6     60.3     85.3     75.0     63.3     71.9     -

GPT-3.5 Prog.     58.3     37.6     78.3     68.0     61.6     63.4     89

Llama CoT     72.1     80.8     85.1     81.1     69.5     77.6     -

Llama Prog.     58.9     53.4     65.6     61.0     56.8     59.7     81

Mixtral CoT     72.6     67.9     84.9     74.1     67.2     74.0     -

Mixtral Prog.     41.1     10.7     55.3     53.9     52.9     46.1     73

Performance comparisons between CoT (uses original questions) and Prog(ram) (uses conceptualized questions and the proposed program space in <ref>). Exe.% is the program execution success rate. Unsuccessful executions are similar to CoT outputs that do not contain an answer, so they do not affect the fairness of comparison.

System     BoolQ     CsQA     Hotpot     Strat.dev     Strat.test     Avg.$\Delta$

Llama Program     58.9     53.4     65.6     61.0     56.8     -

+ Selection     65.0     76.1     72.1     68.4     60.0     +9.2

+ Self-refine     68.0     78.6     73.9     71.1     61.0     +11.4

Mixtral Program     41.1     10.7     55.3     53.9     52.9     -

+ Selection     53.3     30.8     60.1     56.1     52.1     +7.7

+ Self-refine     55.3     38.0     64.0     58.8     54.8     +11.4

System performances on reasoning datasets with proposed improvements. Selection refers to the selection method proposed in <ref>, and Self-refine refers to the self-refinement technique proposed in <ref>. Program is the same as in Table <ref>.

In this section, we conduct experiments on both the analytical setting as described in <ref> and the reasoning setting as described in <ref> with improvements described in <ref>.

§.§ Datasets, Models and Settings

We consider four reasoning benchmarks that will benefit from symbolic solutions: BoolQ [Clark et al., 2019], CommonsenseQA [Talmor et al., 2019], HotpotQA [Yang et al., 2018], and StrategyQA [Geva et al., 2021]. We only use yes/no questions from HotpotQA. We convert multiple-choice questions in CommonsenseQA to binary questions by prompting Llama to transform the original question with its correct answer to a binary question with a gold answer of `yes', as detailed in <ref>. We randomly sample 197, 234, and 456 questions from BoolQ, CommonsenseQA, and HotpotQA. StrategyQA has 228 questions in the dev set and 482 in the test set that can be successfully conceptualized.

For models, we consider three language models that have a relatively close performance on many popular benchmarks, namely GPT-3-Turbo [Brown et al., 2020], Llama-70B-instruct [Touvron et al., 2023] , and Mixtral-8x7B [Jiang et al., 2024].[We use gpt-3.5-turbo-0301 for GPT-3, upstage/Llama-2-70b-instruct for llama, and mistralai/Mixtral-8x7B-Instruct-v0.1 for Mixtral. The last two models are taken from Huggingface and inferred with vLLM [Kwon et al., 2023].]

All CoT inferences use the same set of few-shot prompts selected from StrategyQA. All experiments use a self-consistency K=10. We count draw cases (i.e., #yes=#no) as `unknown'. Temperature is 0.7, except for similar question generation, which uses 1.0 for diversity. All manual prompts and other parameters are tuned on samples from the StrategyQA dev set.

§.§ Analytical Experiments

Table <ref> shows our experiment results on three large language models' conceptual reasoning performance on four benchmarks. We see that all models drop performances using our conceptualization framework for analytical evaluation. Among these models, GPT-3.5 is the most consistent model, dropping 8.5% on average, while Mixtral has a dramatic drop of 28%, partially due to the lower execution success rate from poorly generated programs. These results show that although some language models claim to have on-par performances on many existing benchmarks, their core reasoning capabilities to not rely on induction are very different. This suggests that some models may show artificial high performances through imitation and memorization rather than actual generalization to reasoning tasks.

§.§ Improvement Experiments

Table <ref> shows the effectiveness of our proposed techniques in improving LLMs' conceptual reasoning capability. Our proposed analogical inference from similar questions improves 9.2% with Llama and 7.7% with Mixtral. Self-refinement further improves both models by over 11%. Such improvements suggest that existing LLMs cannot efficiently make implicit analogies and find these trustworthy induction signals by themselves out of the box. We also observe that Llama achieves comparable performances on BoolQ and CommonsenseQA as CoT after self-refinement, indicating the potential to achieve high-performing, controllable, and unbiased reasoning. We include additional analysis in <ref>.

§.§ Human Analysis

Correctness     Solvability

StrategyQA / CommonsenseQA     96 / 92     88 / 80

Human analysis on correctness (the percentage of questions that are correctly conceptualized by GPT-4), and solvability (the percentage of questions humans can solve with the first few interpretations of the abstract question).

We use two human analysis metrics to demonstrate that our conceptualized questions are reasonable for models to solve. Correctness is the percentage of questions where the semantic abstraction (i.e., the fine-grained semantic types) is correct. Solvability, on the other hand, measures the percentage of solvable questions with only the abstract form of the question. In <ref>, we argued that as long as the correctness of semantic abstraction is guaranteed, models should, in theory, solve all questions, making solvability equal to correctness.

However, in our analysis, a question is considered solvable if the human expert can generate a solution without considering too many corner cases.

One of the authors conducted such analysis with 50 questions sampled from StrategyQA and CommonsenseQA, representing two distinct domains: factual and commonsense. Table <ref> shows the results of our human analysis. Our conceptualization correctness is high, ranging from 92% to 96%. Solvability is slightly lower, but it only represents the percentage of questions that can be solved within the initial thoughts of our human expert.

Both metrics are lower on CommonsenseQA because these questions contain fewer named entities.

§ CONCLUSION

We propose a novel conceptualization framework to investigate LLMs' ability to perform conceptual and high-level reasoning without the help of inductive biases from special nouns in the original question. Experiments show that existing LLMs perform much worse in conceptual reasoning than direct inference methods, dropping 9% to 28% across various models and benchmarks. To improve models' conceptual reasoning capability for unbiased inference, we propose two techniques that add trustworthy induction signals to the conceptual reasoning process, improving over 11%. From an analytical perspective, our work suggests that models cannot sufficiently perform conceptual reasoning and efficiently make analogies to similar situations out of the box. To this end, our proposed improvement techniques that explicitly add back trustworthy analogical signals are shown to be effective, motivating future research on generalizable and unbiased reasoning and planning with minimum reliance on induction.

[Brown et al., 2020]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
In NeurIPS, 2020.

[Chen et al., 2023]
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen.
Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.
In TMLR, 2023.

[Chen et al., 2024]
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou.
Teaching large language models to self-debug.
In ICLR, 2024.

[Clark et al., 2019]
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.
Boolq: Exploring the surprising difficulty of natural yes/no questions.
In NAACL, 2019.

[Feng et al., 2023]
Yu Feng, Ben Zhou, Haoyu Wang, Helen Jin, and Dan Roth.
Generic temporal reasoning with differential analysis and explanation.
In ACL, 2023.

[Gao et al., 2023]
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.
Pal: Program-aided language models.
In ICML, 2023.

[Geva et al., 2021]
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant.
Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.
TACL, 2021.

[Jiang et al., 2024]
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.
Mixtral of experts.
arXiv preprint arXiv:2401.04088, 2024.

[Kahneman, 2011]
Daniel Kahneman.
Thinking, fast and slow.
macmillan, 2011.

[Kwon et al., 2023]
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In Proceedings of the 29th Symposium on Operating Systems Principles, pp.  611–626, 2023.

[Li et al., 2024]
Bangzheng Li, Ben Zhou, Fei Wang, Xingyu Fu, Dan Roth, and Muhao Chen.
Deceptive semantic shortcuts on reasoning chains: How far can models go without hallucination?
NAACL, 2024.

[Lin et al., 2024]
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren.
Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks.
Advances in Neural Information Processing Systems, 36, 2024.

[Lu et al., 2024]
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao.
Chameleon: Plug-and-play compositional reasoning with large language models.
Advances in Neural Information Processing Systems, 36, 2024.

[Madaan et al., 2024]
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al.
Self-refine: Iterative refinement with self-feedback.
Advances in Neural Information Processing Systems, 36, 2024.

[Markovits et al., 2015]
Henry Markovits, Valerie A Thompson, and Janie Brisson.
Metacognition and abstract reasoning.
Memory & cognition, 2015.

[Min et al., 2019]
Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi.
Multi-hop reading comprehension through question decomposition and rescoring.
In ACL, 2019.

[Saparov & He, 2023]
Abulhair Saparov and He He.
Language models are greedy reasoners: A systematic formal analysis of chain-of-thought.
In ICLR, 2023.

[Talmor et al., 2019]
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.
Commonsenseqa: A question answering challenge targeting commonsense knowledge.
In NAACL, 2019.

[Tang et al., 2023]
Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng, Song-Chun Zhu, Yitao Liang, and Muhan Zhang.
Large language models are in-context semantic reasoners rather than symbolic reasoners.
arXiv preprint arXiv:2305.14825, 2023.

[Touvron et al., 2023]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
arXiv preprint arXiv:2307.09288, 2023.

[Walker, 2015]
Caren Michelle Walker.
Learning by thinking and the development of abstract reasoning.
University of California, Berkeley, 2015.

[Wei et al., 2022]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
In NeurIPS, 2022.

[Weston & Sukhbaatar, 2023]
Jason Weston and Sainbayar Sukhbaatar.
System 2 attention (is something you might need too).
arXiv preprint arXiv:2311.11829, 2023.

[Xie et al., 2024]
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su.
Travelplanner: A benchmark for real-world planning with language agents.
arXiv preprint arXiv:2402.01622, 2024.

[Xu et al., 2024]
Nan Xu, Fei Wang, Ben Zhou, Bang Zheng Li, Chaowei Xiao, and Muhao Chen.
Cognitive overload: Jailbreaking large language models with overloaded logical thinking.
In NAACL, 2024.

[Yang et al., 2018]
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning.
Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
In EMNLP, 2018.

[Zhou et al., 2021]
Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth.
Temporal reasoning on implicit events from distant supervision.
In NAACL, 2021.

[Zhou et al., 2022]
Ben Zhou, Kyle Richardson, Xiaodong Yu, and Dan Roth.
Learning to decompose: Hypothetical question decomposition based on comparable texts.
In EMNLP, 2022.


# Exclusive top production at a Linear Collider at and off the threshold

Jürgen Reutera, Fabian Bachb, Bijan Chokoufé Nejada, Andre Hoangc,d, Wolfgang
Kiliane, Jonas Lindertf, Stefano Pozzorinig, Maximilian Stahlhofen, Thomas
Teubneri, Christian Weissa
juergen.reuter@desy.defabian.bach@desy.debijan.chokoufe@desy.deandre.hoang<EMAIL_ADDRESS>siegen.dejonas.m.lindert@durham.ac.ukpozzorin@physik.uzh.chmastahlh@uni-
mainz.dehthomas.teubner<EMAIL_ADDRESS>
###### ABSTRACT

We review exclusive top pair production including decays at a future high-
energy lepton collider, both in the threshold region and for higher energies.
For the continuum process, we take complete QCD next-to-leading order matrix
elements for the $2\to 6$ process with leptonic $W$ decays into account. At
threshold, we match the fixed-order relativistic QCD-NLO cross section to a
nonrelativistic cross section with next-to-leading logarithmic (NLL) threshold
resummation implemented via a form factor.

##  1 Introduction

Top physics is one of the standard pillars of the physics program of any
future high-energy lepton collider. The rationale is to determine the
properties of the heaviest Standard Model (SM) quark, its mass, its width and
its couplings to a level of accuracy that is not possible at hadron colliders
like Tevatron and the LHC, and to use the top quark as a handle to search for
physics beyond the Standard Model. Here, we review recent work on the
precision calculation of the QCD next-to-leading order (QCD-NLO) for off-shell
top quark production, including (leptonic) $W$-decays, in the continuum [1].
The details of this calculation and the results will be discussed in Sec. 2.
As shown in this section, the perturbative fixed-order calculation yields
naive K factors of three and more in the kinematic region close to the top-
antitop production threshold for top-pair production, or better $WbWb$ or
$\ell\nu\ell\nu bb$ production. In the threshold region, fixed-order
perturbation theory in the strong coupling $\alpha_{s}$ is not a good
approximation anymore, but the top velocity $v$ is an additional expansion
parameter and Coulomb-singular terms $\sim(\alpha_{s}/v)^{n}$ and (ideally
also) large logarithms $\sim(\alpha_{s}\log v)^{n}$ have to be resummed. In
recent work [2], we used a previously known effective field theory setup based
on non-relativistic QCD (NRQCD) to compute a form factor accounting for the
resummation of the threshold-singular terms at NLL accuracy, implemented it in
the fixed-order calculation and matched the result to the QCD-NLO cross
section in the transition region between threshold and continuum. We thus
obtained a fully-differential cross section, which gives reliable predictions
for all center-of-mass energies. Depending on how inclusive the process is, we
achieve LL + QCD-NLO (for very exclusive processes) or NLL + QCD-NLO precision
(for inclusive processes) in the threshold region. This will be discussed in
Sec. 3. Finally, we summarize and give an outlook in Sec. 4.

##  2 Continuum calculation

We first discuss the relativistic QCD-NLO corrections to the off-shell top
pair production in the continuum, i.e. away from the production threshold.
This means, investigating either the process $e^{+}e^{-}\to
W^{+}bW^{-}\bar{b}$ or
$e^{+}e^{-}\to\ell^{+}e^{-}\bar{\nu}_{e}\mu^{+}\nu_{\mu}b\bar{b}$ including
leptonic $W$ decays. Looking at the full four- or six-particle final state,
there are double-resonant diagrams included (involving a top and an anti-top
propagator), single-resonant diagrams and non-resonant irreducible background
processes.

In order to study the QCD-NLO corrections for the top production processes
discussed here, we take the WHIZARD framework for (QCD-)NLO processes. WHIZARD
[3] is a multi-purpose event generator. It comes with its own matrix-element
generator for tree-level amplitudes, O’Mega [4, 5] which supports almost
arbitrary models like e.g. supersymmetry [6] and many more. External models
can be used inside WHIZARD by its interface to FeynRules [7]. For the QCD
color decomposition, WHIZARD uses the color-flow formalism [8], and it comes
with its own parton

Figure 1: Total cross section for the process $e^{+}e^{-}\to
W^{+}W^{-}b\bar{b}$ as a function of the center-of-mass energy (left); the
upper ratio plot in the bottom shows the K factor, the lower one the ratio of
the four-body process to on-shell $t\bar{t}$ production. The inset shows the K
factor enhancement in the threshold region. Fully off-shell leptonic
$t\bar{t}$ production ($e^{+}e^{-}\to
e^{-}\bar{\nu}_{\ell}\mu^{+}\nu_{\mu}b\bar{b}$) differential distribution
(right). Blue is LO, red is QCD-NLO including scale variations. Schemes and
input parameters as described in the text.

shower implementation [9]. The work on QCD-NLO within WHIZARD started with a
hard-coded implementation for the production of $b$ jets at LHC [10, 11],
while matching between resummed terms and fixed-order calculations have been
tackled by combining fixed-order electroweak corrections to chargino
production at the ILC with an all-order QED initial-state structure function
[12, 13]. Since quite recently, WHIZARD is able to do automatic POWHEG
matching for $e^{+}e^{-}$ processes [14].

For (QCD-)NLO processes, WHIZARD automatically sets up FKS subtraction [15]
and generates the corresponding phase space for all the singular emission
regions. Here we take virtual matrix elements, which contain up to pentagon
integrals for the processes considered here, as well as the color-correlated
and spin-correlated matrix elements for the collinear and soft splittings,
respectively, from the OpenLoops one-loop provider (OLP) [16, 17, 18]. For the
amplitudes, we take the complex mass scheme using complexified masses
$\mu_{i}^{2}=m_{i}^{2}+im_{i}\Gamma_{i}$, which leads e.g. to a complex weak
mixing angle. The input values are as follows: $m_{W}=80.385$ GeV,
$m_{Z}=91.1876$ GeV, $m_{t}=173.2$ GeV, $m_{H}=125$ GeV, and we perform the
calculation here and for the threshold with massive $b$-quarks of $m_{b}=4.2$
GeV. It is important that the widths used in the calculation are at the same
order and in the same scheme than the scattering process in order to guarantee
properly normalized branching ratios: $\Gamma_{Z}^{\text{LO}}=2.4409$ GeV,
$\Gamma_{Z}^{\text{NLO}}=2.5060$ GeV, $\Gamma_{W}^{\text{LO}}=2.0454$ GeV,
$\Gamma_{W}^{\text{NLO}}=2.0978$ GeV, $\Gamma_{t\to Wb}^{\text{LO}}=1.4986$
GeV, $\Gamma_{t\to Wb}^{\text{LO}}=1.3681$ GeV, $\Gamma_{t\to
f\bar{f}b}^{\text{LO}}=1.4757$ GeV, $\Gamma_{t\to
f\bar{f}b}^{\text{NLO}}=1.3475$ GeV. Note that for the process with stable
$W$s, $e^{+}e^{-}\to W^{+}W^{-}b\bar{b}$, one has to take the total top width
for two-body decays, while for the full process, $e^{+}e^{-}\to
e^{-}\bar{\nu}_{\ell}\mu^{+}\nu_{\nu}b\bar{b}$ the total top width for three-
body decays. For the Higgs boson we take the value $\Gamma_{H}=4.31$ MeV. As
the matrix elements for the full off-shell processes contain narrow
resonances, particularly the $H\to bb$ resonance, we use a resonance-aware
version of the FKS subtraction formalism to make sure that cancellations
between real emissions and subtraction terms do cancel though the real
emission could shift the kinematics on or off the resonance compared to Born
kinematics. This resonance-aware treatment is automatically done in WHIZARD.
As we are using massive $b$-quarks, no cuts are necessary for the process
$e^{+}e^{-}\to W^{+}W^{-}b\bar{b}$, but the process $e^{+}e^{-}\to
e^{-}\bar{\nu}_{\ell}\mu^{+}\nu_{\mu}b\bar{b}$ exhibits a collinear
singularity from photon emission off the electron line going from the initial
to the final state. The integrations for the full QCD-NLO are very stable. We
did two independent own integrations with the serial and the non-blocking MPI-
parallelizable version of VAMP [19] inside WHIZARD, and we confirmed the
result within Sherpa [20] and Munich.

For the QCD-NLO corrections, we take the top mass as renormalization scale.
The scale variations for the process $e^{+}e^{-}\to W^{+}bW^{-}\bar{b}$ is
very small, at the level of two per cent. After one has replaced the top width
in the matrix elements by a running top width $\Gamma_{t}(\mu_{R})$ , the
scale variations for the on-shell process $e^{+}e^{-}\to t\bar{t}$ behave the
same way as for the off-shell process. In Fig. 1 we show in the left panel the
total cross section for $e^{+}e^{-}\to W^{+}bW^{-}\bar{b}$ as a function of
the center-of-mass energy $\sqrt{s}$ over the whole kinematic range from well
below the threshold up to full energy stage of CLIC at 3 TeV. Below the main
plot there are two ratio plots, the first showing the K factor
$\sigma(NLO)/\sigma(LO)$, the second showing the ratio of the off-shell to the
on-shell process, $\sigma(e^{+}e^{-}\to
W^{+}bW^{-}\bar{b})/\sigma(e^{+}e^{-}\to t\bar{t})$. In the upper ratio plot,
the green curve is the K factor for the on-shell process, while the red one is
the K factor for the off-shell process. For the off-shell process, the K
factor tends to almost unity at a TeV and beyond. The second ratio plot shows
that without restricting to single- and double resonant phase-space regions,
the background processes start to more and more dominate over the signal top-
pair production. The colors in the second ratio plot correspond to the legend,
i.e. blue for LO and red for NLO. The inset in the left plot of Fig. 1 shows
the enhancement in the K factor around the top threshold where fixed-order
perturbation theory is not a valid approximation any more, cf. Sec. 3. The K
factor for the on-shell process even becomes infinite here. In the right panel
of Fig. 1 we show as an example for a differential distribution the invariant
mass for the negatively charged $W$ (reconstructed at Monte Carlo truth level)
and the $b$ jet for the full process $e^{+}e^{-}\to
e^{-}\bar{\nu}_{\ell}\mu^{+}\nu_{\mu}b\bar{b}$ at the center-of-mass energy of
$\sqrt{s}=800$ GeV where the cross section peaks. The LO distribution is shown
in blue and the QCD-NLO distribution including scale variations in red. One
clearly sees that the K factor is not constant over the phase space.
Particularly for low invariant masses below the top mass peak, there is a
large enhancement as this part of the phase space gets populated massively by
gluon radiation off the top peak region.

Note that the setup inside WHIZARD allows to immediately do QCD-NLO
calculations and simulations for polarized cross sections, or include QED
initial-state photon radiation as well as effects from beamspectra. In Ref.
[1], we also calculated the processes $e^{+}e^{-}\to t\bar{t}H$,
$e^{+}e^{-}\to W^{+}bW^{-}\bar{b}H$ and $e^{+}e^{-}\to
e^{-}\bar{\nu}_{e}\mu^{+}\nu_{\mu}b\bar{b}H$ at QCD-LO and QCD-NLO, which we
omitted in this proceedings article here.

##  3 Threshold matching

A kinematic fit to the shape of the rising of the cross section at the top
threshold is believed to be the most precise method to measure the top quark
mass with an ultimate precision of 30-80 MeV. For this the systematic
uncertainties of the experimental measurement – especially the details of the
beam spectrum – as well as the theoretical uncertainties have to be well under
control. As shown Sec. 2, close to the kinematical threshold for the on-shell
production of a $t\bar{t}$

Figure 2: Matched NRQCD-NLL + QCD-NLO calculation without (left) and with
(right) QED ISR. The dashed vertical line is the value of twice
$M^{1\text{S}}$. Blue is the fixed QCD-NLO calculation, red is the fully
matched calculation. The matched calculation has a full envelope over
(symmetrized) scale uncertainties as well as variations over switch-off
functions.

pair, fixed-order perturbation theory is not a good approximation. Very close
to threshold, the effective field theory of (v/p)NRQCD separates the hard
scale $m_{t}$, the soft scale given by the top momentum of the non-
relativistic top quark with velocity $v$, $m_{t}v$ and the ultrasoft scale,
given by the kinetic energy of the top quark, $m_{t}v^{2}$ and allows to resum
large logarithms of $v$ with $\alpha_{s}\sim v\sim 0.1$ close to threshold.
”Fixed-order” calculations resumming only Coulomb singularities, but no
velocity logarithms, for the totally inclusive $t\bar{t}$ production have been
carried out in NRQCD to NNNLO [21]. The large velocity logarithms have been
resummed to next-to-next-leading logarithmic (NNLL) [24] order (cf. also [22,
23] for predictions which did not contain the full set of NNLL ultrasoft
logarithms). These NRQCD calculations are based on the optical theorem and
hold only for the total inclusive cross section and in a narrow window around
the $t\bar{t}$ threshold. We report here about work where we combined and
matched the NLL NRQCD-resummed process close to the top threshold with the
fixed-order (relativistic) QCD-NLO process in the continuum. By a carefully
performed matching procedure, our approach smoothly interpolates between the
threshold region and the continuum, and it allows to study all kinds of
differential distributions.

The matching is embedded into the WHIZARD-OpenLoops QCD-NLO fixed-order
framework discussed in Sec. 2. The NLL resummed NRQCD contributions are
included in terms of (S-/P-wave) form factors to the (vector/axial vector)
$\gamma/Z-t-\bar{t}$ vertex. These form factors are obtained from the
numerical solution of Schrödinger-type equations for the NLL Green functions
computed by the Toppik [25, 26, 27] code, which is included in WHIZARD. The
technical details of their implementation and the matching setup can be found
in [2]. In order to avoid double-counting between the fixed-order QCD-NLO part
and the resummed NLL-NRQCD part, one has to expand the form factors to first
order in $\alpha_{s}$ and subtract those pieces. As the NRQCD resummed
calculations are not available for the 5-point functions $\gamma^{*}/Z^{*}\to
W^{+}bW^{-}\bar{b}$, but only for the top-vector and axial-vector currents,
this removal of double-counting has to be done in a factorized approach within
a double-pole approximation. There is no trivial gauge-invariant subset for
the process $e^{+}e^{-}\to W^{+}bW^{-}\bar{b}$. In order to maintain gauge-
invariance of the factorized amplitudes, an on-shell projection of the
exclusive final states to the top mass shell is performed. The technical
details, especially concerning the direction of the three-momenta and the
definition of the on-shell projection below the kinematical threshold can be
found in [2]. The implementation inside WHIZARD has been

Figure 3: Left panel: Matched NRQCD-NLL + QCD-NLO total cross section as on
the left of Fig. 2, but for a single choice of switch off-function. $h$ and
$f$ are renormalization scale parameters as defined in [2, 24]. The grey bands
display the corresponding scale variations with and without symmetrization.
Right panel: $Wb$ invariant mass distribution at threshold ($\sqrt{s}=344$
GeV) as obtained with WHIZARD. The red line represents the full NRQCD-NLL \+
QCD-NLO matched, and the blue line the pure QCD-NLO result. The associated
bands are generated by the same scale variations as in the left panel, here
without symmetrization.

validated with analytical calculations for different invariant mass cuts on
the reconstructed top quarks from Ref. [28].

As for larger values of the top velocity ($v\gtrsim 0.4$) only the
relativistic QCD-NLO result is valid, we define a switch-off function that
smoothly interpolates between the two regions. This function is in principle
arbitrary, and the possibility to vary this function and its parameters adds
another theory uncertainty on top of the different scale variations in the
different kinematic regimes. For technical details again, we refer to [2]. The
results of our matching procedure are displayed in Fig. 2. These plots show
the total inclusive cross section for the process $e^{+}e^{-}\to
W^{+}bW^{-}\bar{b}$, in the left panel without and in the right panel with QED
initial-state radiation (ISR). The dashed vertical line gives the value for
$2M^{1\text{S}}$. The 1S mass $M^{1\text{S}}$ is defined as half of the
perturbative mass of a would-be 1S toponium state and represents a renormalon
free short-distance mass, which we treat as an input parameter in WHIZARD. The
blue line shows the QCD-NLO cross section including scale variations in the
blue shaded areas. The only difference to the results in Sec. 2 is a different
renormalization, see [2]. The red curve shows the NRQCD-NLL + QCD-NLO result,
while the shaded band contains all (symmetrized) scale variations of the hard,
soft and ultrasoft factorization/renormalization scales according to [24] as
well as variations of the switch-off function to a reasonable extent [2]. The
dotted black line shows the matched results without applying a switch-off
function to the factorized NRQCD terms which deviates above threshold from the
relativistic QCD-NLO result. In Fig. 3, left panel, we see the matched result
in the threshold region for a single choice of switch-off parameters, but
scale variations over the full two-dimensional renormalization parameter range
defined in [24]. This shows that the scale variation bands for the resummed
NLL result in the threshold region are highly asymmetric with respect to the
central value which motivates to apply a symmetrization of the error bands
around the central value. This symmetrization is also shown in Fig. 2. In the
right panel of Fig. 3 we show as an example for a differential distribution
the invariant mass of the $W-b$ jet system. Blue is the fixed-order QCD-NLO
distribution, while red is the fully matched distribution including scale
variations, here un-symmetrized. The ratio plot in the bottom does not show a
K factor, but the ratio of the matched result to the QCD-NLO fixed order
result. It shows an enhancement in the top mass peak due to threshold
resummation by a factor of ten to twelve.

##  4 Summary and Outlook

We presented work on the QCD-NLO corrections for exclusive top-quark pair
production including top and (leptonic) $W$ decays. Kinematic regions in the
continuum up to the highest available energies of CLIC were covered as well as
the threshold region. Both calculations have been performed in the QCD-NLO
framework of the WHIZARD event generator which allows to include all important
physics of a lepton collider like polarization, QED ISR radiation and non-
trivial beam spectra. The continuum calculations represent the first massive
$2\to 6$ and $2\to 7$ QCD-NLO calculations for lepton colliders. The matched
threshold calculation smoothly interpolates the threshold region described by
non-relativistic QCD to the relativistic QCD-NLO calculation and constitutes
the highest available precision available at the level of the completely
exclusive final state. While the work presented here is more a proof-of-
principle of the matching procedure between threshold and continuum, it is
obvious that the various differential distributions (which are accessible now
and of which we only showed one here) offer plenty of possibilities for top
mass measurements. This is part of ongoing and future work. Other directions
for future work are the matching at even higher order in QCD, the inclusion of
electroweak corrections to the relativistic continuum process, and the
inclusion of $W$ decays also in the matched calculations.

## Acknowledgments

JRR wants to thank the organizers of LCWS 2017 in Strasbourg for a great
conference at a lovely venue. JRR, BCN and CW acknowledge support from the
Collaborative Research Unit (SFB) 676 of the DFG, projects B1 and B11. We like
to thank Stefan Kallweit for his help with the cross-checks by the Munich
code.

## References

  * [1] B. Chokoufé Nejad, W. Kilian, J. M. Lindert, S. Pozzorini, J. Reuter and C. Weiss, JHEP 1612, 075 (2016) doi:10.1007/JHEP12(2016)075 [arXiv:1609.03390 [hep-ph]].
  * [2] F. Bach, B. C. Nejad, A. Hoang, W. Kilian, J. Reuter, M. Stahlhofen, T. Teubner and C. Weiss, arXiv:1712.02220 [hep-ph].
  * [3] W. Kilian, T. Ohl and J. Reuter, Eur. Phys. J. C 71, 1742 (2011) doi:10.1140/epjc/s10052-011-1742-y [arXiv:0708.4233 [hep-ph]].
  * [4] M. Moretti, T. Ohl and J. Reuter, hep-ph/0102195.
  * [5] B. Chokoufe Nejad, T. Ohl and J. Reuter, Comput. Phys. Commun. 196, 58 (2015) doi:10.1016/j.cpc.2015.05.015 [arXiv:1411.3834 [physics.comp-ph]].
  * [6] T. Ohl and J. Reuter, Eur. Phys. J. C 30, 525 (2003) doi:10.1140/epjc/s2003-01301-7 [hep-th/0212224].
  * [7] N. D. Christensen, C. Duhr, B. Fuks, J. Reuter and C. Speckner, Eur. Phys. J. C 72, 1990 (2012) doi:10.1140/epjc/s10052-012-1990-5 [arXiv:1010.3251 [hep-ph]].
  * [8] W. Kilian, T. Ohl, J. Reuter and C. Speckner, JHEP 1210, 022 (2012) doi:10.1007/JHEP10(2012)022 [arXiv:1206.3700 [hep-ph]].
  * [9] W. Kilian, J. Reuter, S. Schmidt and D. Wiesler, JHEP 1204, 013 (2012) doi:10.1007/JHEP04(2012)013 [arXiv:1112.1039 [hep-ph]].
  * [10] T. Binoth, N. Greiner, A. Guffanti, J. Reuter, J.-P. Guillet and T. Reiter, Phys. Lett. B 685, 293 (2010) doi:10.1016/j.physletb.2010.02.010 [arXiv:0910.4379 [hep-ph]].
  * [11] N. Greiner, A. Guffanti, T. Reiter and J. Reuter, Phys. Rev. Lett. 107, 102002 (2011) doi:10.1103/PhysRevLett.107.102002 [arXiv:1105.3624 [hep-ph]].
  * [12] W. Kilian, J. Reuter and T. Robens, Eur. Phys. J. C 48, 389 (2006) doi:10.1140/epjc/s10052-006-0048-y [hep-ph/0607127].
  * [13] T. Robens, J. Kalinowski, K. Rolbiecki, W. Kilian and J. Reuter, Acta Phys. Polon. B 39, 1705 (2008) [arXiv:0803.4161 [hep-ph]].
  * [14] J. Reuter, B. Chokoufe, A. Hoang, W. Kilian, M. Stahlhofen, T. Teubner and C. Weiss, J. Phys. Conf. Ser. 762, no. 1, 012059 (2016) doi:10.1088/1742-6596/762/1/012059 [arXiv:1602.06270 [hep-ph]].
  * [15] S. Frixione, Z. Kunszt and A. Signer, Nucl. Phys. B 467, 399 (1996) doi:10.1016/0550-3213(96)00110-1 [hep-ph/9512328].
  * [16] F. Cascioli, P. Maierhofer and S. Pozzorini, Phys. Rev. Lett. 108, 111601 (2012) doi:10.1103/PhysRevLett.108.111601 [arXiv:1111.5206 [hep-ph]].
  * [17] F. Buccioni, S. Pozzorini and M. Zoller, arXiv:1710.11452 [hep-ph].
  * [18] F. Buccioni, S. Pozzorini and M. Zoller, arXiv:1801.03772 [hep-ph].
  * [19] T. Ohl, Comput. Phys. Commun. 120, 13 (1999) doi:10.1016/S0010-4655(99)00209-X [hep-ph/9806432].
  * [20] T. Gleisberg, S. Hoeche, F. Krauss, M. Schonherr, S. Schumann, F. Siegert and J. Winter, JHEP 0902, 007 (2009) doi:10.1088/1126-6708/2009/02/007 [arXiv:0811.4622 [hep-ph]].
  * [21] M. Beneke, Y. Kiyo, P. Marquard, A. Penin, J. Piclum and M. Steinhauser, Phys. Rev. Lett. 115, no. 19, 192001 (2015) doi:10.1103/PhysRevLett.115.192001 [arXiv:1506.06864 [hep-ph]].
  * [22] A. H. Hoang, A. V. Manohar, I. W. Stewart and T. Teubner, Phys. Rev. D 65, 014014 (2002) doi:10.1103/PhysRevD.65.014014 [hep-ph/0107144].
  * [23] A. Pineda and A. Signer, Nucl. Phys. B 762, 67 (2007) doi:10.1016/j.nuclphysb.2006.09.025 [hep-ph/0607239].
  * [24] A. H. Hoang and M. Stahlhofen, JHEP 1405, 121 (2014) doi:10.1007/JHEP05(2014)121 [arXiv:1309.6323 [hep-ph]].
  * [25] M. Jezabek, J. H. Kuhn and T. Teubner, Z. Phys. C 56, 653 (1992). doi:10.1007/BF01474740
  * [26] R. Harlander, M. Jezabek, J. H. Kuhn and T. Teubner, Phys. Lett. B 346, 137 (1995) doi:10.1016/0370-2693(94)01668-3 [hep-ph/9411395].
  * [27] A. H. Hoang and T. Teubner, Phys. Rev. D 60, 114027 (1999) doi:10.1103/PhysRevD.60.114027 [hep-ph/9904468].
  * [28] A. H. Hoang, C. J. Reisser and P. Ruiz-Femenia, Phys. Rev. D 82, 014005 (2010) doi:10.1103/PhysRevD.82.014005 [arXiv:1002.3223 [hep-ph]].