# Extending the definition of atomic basis sets to atoms with fractional
nuclear charge

Giorgio Domenichini<EMAIL_ADDRESS>University of Vienna
University of Vienna.

###### Abstract

Alchemical transformations showed that perturbation theory can be applied also
to changes in the atomic nuclear charges of a molecule. The alchemical path
that connects two different chemical species involves the conceptualization of
a non-physical system in which atom possess a non-integer nuclear charge. A
correct quantum mechanical treatment of these systems is limited by the fact
that finite size atomic basis sets do not define exponents and contraction
coefficients for fractional charge atoms. This paper is proposes a solution to
this problem, and shows that a smooth interpolation of the atomic orbital
coefficients and exponents across the periodic table is a convenient way to
produce accurate alchemical predictions, even using small size basis sets.

The following article will be submitted to AIP - The Journal of Chemical
Physics

##  I Introduction

A molecular compound can be defined by its composition, its molecular
geometry, and its molecular charge; in a computational simulation, these
properties are fully described by: nuclear positions $\mathbf{R}$, nuclear
charges $\mathbf{Z}$, and number of electrons $\mathrm{N}_{e}$.
Considering only compounds in their electronic ground state, and assuming
valid the Born-Oppenheimer adiabatic approximationBorn and Oppenheimer (1927),
$\mathbf{R}$, $\mathbf{Z}$, and $\mathrm{N}_{e}$ are the variables which
define a continuous P.E.S. (Potential Energy Surface):
$E(\mathbf{R},\mathbf{Z},\mathrm{N}_{e})$. The derivatives of the energy with
respect to these coordinates give many insights about chemical properties and
chemical reactivity.Cárdenas  _et al._ (2011); Laplaza  _et al._ (2021);
Cárdenas, Heidar-Zadeh, and Ayers (2016); Gómez  _et al._ (2021); Cohen,
Ganduglia-Pirovano, and Kudrnovský (1994, 1995); Balawender and Geerlings
(2001); Flores-Moreno  _et al._ (2021); Balawender and Komorowski (1998);
Baekelandt (1996); Robles  _et al._ (2018); Geerlings  _et al._ (2019)

Perturbations along the nuclear charge coordinates, meaning changing the
chemical composition of one reference molecule to one or more target
molecules, are known as alchemical transmutations.Muñoz  _et al._ (2020);
Muñoz and Cárdenas (2017); Balawender and Geerlings (2023); Griego, Saravanan,
and Keith (2019); Saravanan  _et al._ (2017); Griego, Kitchin, and Keith
(2021); Griego  _et al._ (2021); Keith  _et al._ (2021); von Rudorff and von
Lilienfeld (2021); Von Lilienfeld and Tuckerman (2007); von Rudorff and von
Lilienfeld (2020a); von Lilienfeld (2009); von Rudorff and von Lilienfeld
(2019, 2020b); Von Lilienfeld and Tuckerman (2006); Marcon, von Lilienfeld,
and Andrienko (2007); Krug, von Rudorff, and von Lilienfeld (2022); Krug and
von Lilienfeld (2023); von Lilienfeld and Domenichini (2023) Perturbation
theory can be applied to alchemical transmutations, but it requires the
abstraction of calculating derivatives along a non-physical path. The concept
of nuclear charge has to be extended to non-integer numbers: if the basis set
is kept fixed, a molecule with fractional nuclear charge can be modelled by
scaling the nuclear-electron attraction operator by a real number.

Contracted Gaussian type orbitals (CGTO)Clementi (1965); Clementi and Davis
(1966) are standard atomic basis for quantum mechanical calculations on
molecular systems, their contraction coefficients and exponents are optimized
and tabulated for every chemical element.

Previous worksBalawender  _et al._ (2019); Domenichini, von Rudorff, and von
Lilienfeld (2020) showed that the alchemical perturbation (AP) series
calculated using the atomic orbital (AO) basis set of the reference, converges
to the energy of the target molecule with the basis set of the reference
($E^{T[R]}$). The difference between the target’s energy calculated with
reference basis set, and the target’s energy with its own basis
set($E^{T[T]}$), was definedDomenichini, von Rudorff, and von Lilienfeld
(2020) as "alchemical basis set error" $\Delta E_{BS}=E^{T[R]}-E^{T[T]}$.
Using small basis sets, $\Delta E_{BS}$ can be as big as 1. Ha for the
transmutation of a just one atom. Such huge error makes meaningless any
alchemical energy prediction for basis sets such as STO-3G,STO-6GHehre,
Stewart, and Pople (1969), Pople spilt valence basis sets 3-21G, 6-31GBinkley,
Pople, and Hehre (1980); Francl  _et al._ (1982), Dunning’s cc-pVDZ and cc-
pVTZThorn H. Dunning (1989); Woon and Dunning (1993), and Ahlrichs’s
def2-TZVPSchäfer, Huber, and Ahlrichs (1994).

A way to reduce this error is to use of large augmented basis sets Eikey  _et
al._ (2022a, b); Chang  _et al._ (2016) (e.g. the Dunning’s aug-cc-pVQZ, aug-
cc-pV5Z ). Also uncontracted basis set are more flexible than their respective
contracted versions (e.g. for uncontracted cc-pVDZ Fias, Chang, and von
Lilienfeld (2019) uncontracted Cartesian Def2TZVP Chang  _et al._ (2016)),
therefore more suitable to applications in quantum alchemy. According to a
previous investigationDomenichini and von Lilienfeld (2022) we found that the
smallest alchemical basis set errors are produced using polarized core and
valence basis functions, such as Dunning’s cc-pCVnZWoon and Dunning (1995), or
Jensen’s pC-n Jensen (2001, 2002, 2007) basis sets, similar conclusions were
obtained also by Geerlings et al.Geerlings  _et al._ (2019). Particular good
results can be achieved using the Jensen’s pcX-n Ambroise and Jensen (2019)
basis sets. Some basis set do not differ from reference to target, this is the
case of the universal Gaussian basis setde Castro and Jorge (1998), as well as
the plane waves basis sets.Kresse and Furthmüller (1996); Blöchl (1994)

The most elegant way to solve the basis set issue, suggested in a 2012 paper
from M.Lesiuk, R.Balawender and J.Zachara Lesiuk, Balawender, and Zachara
(2012), involve a smooth transformation of the atomic orbitals as the nuclear
charges of the reference approaches the nuclear charges of the target
molecules. In this paper I show that, as long as target and reference
molecules share the same number of primitive and contracted Gaussian basis
function, it is straightforward to interpolate through Splines basis set
exponents and coefficients, extending the definition of basis sets to atoms
with a non integer nuclear charge. The extended basis set that change
consistently with the nuclear charge will be called alchemically consistent
basis sets (ACBS), a term which is not referred to a particular basis set,
rather to the interpolating procedure on which can be derived from an existing
basis set.

Figure 1: For the simple alchemical transmutation CO$\rightarrow$BF
(HF/6-31G), using reference basis set, the AP series converges $E^{BF[CO]}$,
while using AC-6-31G basis set the series approaches $E^{BF[BF]}$

In section II are explained the interpolation procedure through alchemical
consistent basis sets are defined. For the Restricted Hartree Fock (RHF) level
of theory is proposed a formula to calculate the first alchemical derivative
including the basis set derivatives. In section III are given some example on
how including basis set derivatives in the alchemical expansion is crucial to
produce accurate results while using a small basis set. Numerical evidence of
this are deduced from the two test cases of diatomic molecules (exemplified in
Figure 1) and the boron-nitrogen (BN) doping of benzene.

The results in this paper were obtained from a locally modified version of the
PySCF programSun  _et al._ (2017, 2020), accessible through the GitHub
repository "Supplementary code for Quantum Alchemy"Domenichini (2023). In this
work were used PythonVan Rossum and Drake Jr (1995); Pérez and Granger (2007)
3.8 scripts, using NumpyHarris  _et al._ (2020), ScipyVirtanen  _et al._
(2020) and MatplotlibHunter (2007) libraries.

##  II  Methods

###  II.0.1 Interpolation of orbitals exponent and coefficients

The radial part of a CGTO is:

$R^{CGTO}(r)=N\sum_{i}c_{i}\left({\frac{2\zeta_{i}}{\pi}}\right)^{{3/4}}e^{-\zeta_{i}r^{2}}$
(1)

N is a normalization constant chosen such that the atomic orbitals are
normalized, this constant is calculated automatically by PySCF.

The contraction coefficients $c_{i}$ and the Gaussian exponents $\zeta_{i}$,
are tabulated in the basis set for different elements. In many basis sets,
elements belonging to the same period share the same number of atomic orbitals
and Gaussian primitives. This allows a smooth transformation of orbital
coefficients and exponents from one element to another. Coefficients
$c_{i}(Z)$, and exponents $\zeta_{i}(Z)$ have to be defined as continuous and
differentiable functions of the atomic nuclear charge $Z$, and for integer
nuclear charge they should be equal to the tabulated value.

Figure 2:  Radial density function of Pople’s 6-31G $1s$ and $2s$ atomic
orbitals for the elements of the second period. The solid lines represent the
basis function of the elements, which have an integer nuclear charge $Z$, the
dashed lines represent the basis functions of some non-physical atoms with
fractional nuclear charge $Z+\frac{1}{3}$ and $Z+\frac{2}{3}$

A simple choice that fulfills these requirements is to define
$c_{i}(Z),\zeta_{i}(Z)$, is to use splines interpolations. In this paper will
be used a fifth order spline with the ‘not-a-knot’ boundary conditions. In
Figure 2 are plotted, for Pople’s 6-31G basis set, the tabulated a, with
integer $Z$, and interpolated a with fractional $Z$.

Figure 3:  RHF/6-31G alchemical energy predictions: N${}_{2}\rightarrow$CN- ,
N${}_{2}\rightarrow$NO+ . Derivatives are calculated numerically from a 7
point central finite difference stencil, with spacing $h=0.05e$. ACBS are
defined using either a cubic spline or a 5th order spline.

A $n^{th}$ order spline is a polynomial function that is continuous up to its
$n-1^{th}$ derivative. Discontinuity of higher order derivatives affects the
alchemical perturbations. As an example, Figure 3 shows that if AO
coefficients and exponents are transformed with a cubic spline, the alchemical
series diverges for a perturbation order greater than 4, while if the AO
coefficients and exponents transformed with the fifth order spline diverges
for an alchemical perturbation order greater than 6.

As a rule of thumb, the AO interpolation spline should have an order which is
higher than the alchemical perturbation.

###  II.0.2 First order analytical alchemical derivative

The first order alchemical derivative of the RHF energy can be derived from
the gradient expressionPople  _et al._ (1979); Moccia (1970); Bishop and
Randić (2004).

$\frac{\partial E}{\partial Z}=\sum_{\mu\nu}P_{\mu\nu}\frac{\partial
h^{(1)}_{\mu\nu}}{\partial
Z}+\frac{1}{2}\sum_{\mu\nu\lambda\sigma}P_{\mu\nu}P_{\lambda\sigma}\frac{\partial}{\partial
Z}(\mu\lambda||\nu\sigma)+\frac{\partial V_{nuc}}{\partial
Z}-\sum_{\mu\nu}W_{\mu\nu}\frac{\partial S_{\mu\nu}}{\partial Z}$ (2)

$W$ is the energy weighted density matrix:

$W_{\mu\nu}=\sum_{i}^{mo.occ.}\epsilon_{i}c_{\mu i}c_{\nu i}^{\dagger}$ (3)

If the alchemical derivatives are calculated using the reference basis set,
the derivatives of the AO integrals w.r.t. the nuclear charge are zero
$\left(\frac{\partial}{\partial Z}=0,\frac{\partial}{\partial
Z}(\mu\lambda||\nu\sigma)=0\right)$, and Equation 2 is reduced to the
Hellmann-Feynman derivative:

$\frac{\partial E}{\partial Z}=\sum_{\mu\nu}P_{\mu\nu}\frac{\partial
h^{(1)}_{\mu\nu}}{\partial Z}+\frac{\partial V_{nuc}}{\partial
Z}=\int\rho(\vec{r})\Delta v(\vec{r})d\vec{r}+\frac{\partial V_{nuc}}{\partial
Z}$ (4)

Higher order alchemical derivatives can be evaluated analytically from the
solution of the Coupled Perturbed Hartree Fock (CPHF), or the Couple Perturbed
Kohn Sham (CPKS) equations Caves and Karplus (1969); Dalgarno (1962); Cammi,
Cossi, and Tomasi (1996). These analytical derivatives have been implemented
and used in alchemical perturbations Balawender  _et al._ (2019, 2018a, 2013,
2018b); Domenichini and von Lilienfeld (2022), but only in the case where
basis sets do not depend on nuclear charge (reference basis set are kept
fixed), and will be used in the following section. An implementation of
analytical CPHF second and third derivatives which includes the basis set
derivative (ACBS) is possible, but not yet implemented. Second and third
alchemical derivatives, using ACBS, were calculated from numerical
differentiation (through a three point central finite difference scheme with
spacing $h=0.05e$) of the first alchemical derivative (Equation 2).

##  III  Numerical examples

The basis sets 3-21G, 6-31G, cc-pVDZ have a constant number of CGTO and
Gaussian primitives for all elements of the second period, this allows to
define ACBS which connect all atoms from nuclear charge 3 (Lithium), up to
nuclear charge 10 (Neon). The RHF energy alchemical prediction calculated
using the ACBS will be compared to the AP calculated on the basis set of the
reference molecule.

Basis set | MAE AP3 ACBS | MAE AP3 Ref.BS
---|---|---
3-21G | 0.0077 | 1.2653
6-31G | 0.0039 | 1.2701
cc-pvDZ | 0.0055 | 1.2642
Table 1: MAE of the third order alchemical perturbation of the 14 electrons
diatomic molecules (BF,CO,N2), all target within a $\Delta Z=\pm 1$
perturbation are reached, and the interatomic distance is kept fixed at a
length of 2.05 Bohr

In table 1 are listed the mean absolute errors (MAE) of the third order
alchemical prediction (AP3) of the 14 electrons diatomic molecules BF,CO,N2.
From the reference molecule are reached all possible target with a $\Delta
Z=\pm 1$ perturbation:

  * •

N2 $\rightarrow$ NO+ ,CN- ,CO

  * •

CO $\rightarrow$ CF+, NO+, BO-, CN- , N2, BF

  * •

BF $\rightarrow$ BNe+, CF+, BeF-, BO-,CO,BeNe

The error made using the ACBS, is on the milli Hartree scale, and it comes
from the truncation of the alchemical series, while the MAE from the reference
basis set AP it is constantly off by the huge amount of $\approx 1.2$ Hartree,
mainly due to $\Delta E_{\text{BS}}$.

Figure 4: Alchemical B-N doping of benzene, the plot shows the true versus the second order predictions for AC-6-31G basis set, against the predictions made with the 6-31G basis set of benzene.  B-N substitutions | M.A.E. ACBS | M.A.E. Ref BS
---|---|---
1 | 0.0035 | 1.8064
2 | 0.0273 | 3.6195
3 | 0.0740 | 5.4268
Table 2: MAE for the alchemical energy predictions of the BN doped benzene’s
mutants.

Another common application of alchemical perturbations is the BN doping of
carbon aromatic systems, previous papersBalawender  _et al._ (2013, 2018a);
Shiraogawa and Hasegawa (2023, 2022); Fias, Chang, and von Lilienfeld (2019);
Al-Hamdani, Michaelides, and von Lilienfeld (2017) showed how quantum alchemy
can predict a combinatorially large number of doped mutants from the explicit
calculation of one single reference molecule. In figure 4 and in table 2 are
presented the results for the second order alchemical prediction (AP2) of all
17 B-N doped mutants of benzene, calculated at the RHF level of theory, using
Pople’s 6-31G basis sets, the atomic coordinates are the one of the benzene’s
energy minimum. Figure 4 shows that the alchemical predictions with the ACBS
approach the true energy of the target, while the predictions made from the
reference basis set derivatives approach the energy of the mutants with the
basis set of benzene ($E^{T[R]}$). The mean absolute errors for the second
order predictions, listed in Table 2 show that the error for the AC-BS can be
as little as 4 mHa for a single B-N doping, while grows up to 27 mHa for
double substitutions, and 74 mHa for triple substitutions, even if this is
quite a large error on total energy, however constitutes less than 1% of the
total perturbation energy, and is expected that adding more terms in the
series can eventually reduce this error to the desired accuracy.

##  IV Conclusions

CGTO basis sets are defined only for real elements with integer nuclear
charge. In this paper I present a way to extend the basis set definition, to
non-physical atoms, with non-integer nuclear charge. This is possible as long
as exist a series of contiguous elements which share the same basis set
contraction scheme (they possess the same number of contracted Gaussians and
the same number of Gaussian primitives).

Because of the high $\Delta E_{\mathrm{BS}}$ some basis sets struggle to give
accurate alchemical predictions of the total energy if the derivatives are
calculated on the reference basis set. However if the derivatives include the
alchemical consistent transformation of the atomic orbitals, as is the case of
the first alchemical derivative in Equation 2, than the alchemical
perturbation series converge to the true target energy. I showed that in
section III that for alchemical perturbation with $\Delta Z=\pm 1$, such as
the diatomics in Table 1, or a single B-N doping of benzene in Table 2, the
error is on the milli Hartree scale.

The Gaussian basis set interpolation reported in this paper, is analogous to
the interpolation of pseudo-potential along the periodic table, showed in
various works on quantum alchemy. Sheppard, Henkelman, and von Lilienfeld
(2010); Solovyeva and von Lilienfeld (2016); von Lilienfeld (2013); Chang and
von Lilienfeld (2018); Sahre, von Rudorff, and von Lilienfeld (2023); Sahre
_et al._ (2023); Griego  _et al._ (2020); Caracas and Banigan (2009); Marzari,
de Gironcoli, and Baroni (1994); Saitta, De Gironcoli, and Baroni (1998);
Bellaiche, García, and Vanderbilt (2002) Stretching basis sets across the
periodic table may help to smooth, not only alchemical, but also Machine
Learning predictions Von Lilienfeld (2018); Rupp (2015); von Lilienfeld,
Müller, and Tkatchenko (2020); Rupp  _et al._ (2012); Faber  _et al._ (2017);
Christensen, Faber, and von Lilienfeld (2019); Weinreich, Browning, and von
Lilienfeld (2021); Pilania  _et al._ (2013); Ceriotti, Clementi, and Anatole
von Lilienfeld (2021), and in general may be useful to present a more
continuous and differentiable description of the chemical compound space.

##  V Supplementary Material

The supplementary material includes two tables: in Table LABEL:Tab1_SM are
listed all the AP3 predictions obtained from the $\Delta Z=\pm 1$ perturbation
of BF,CO, and N2, in Table 4 are listed the AP2 predicted energies and errors
of the B-N perturbations of benzene.

##  VI Acknowledgments

I would like to thank professor Anatole von Lilienfeld for the nice
suggestions and helpful feedback received.

##  VII Data availability statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * Born and Oppenheimer (1927) M. Born and R. Oppenheimer, “Zur quantentheorie der molekeln,” Annalen der Physik 389, 457–484 (1927), https://onlinelibrary.wiley.com/doi/pdf/10.1002/andp.19273892002 .
  * Cárdenas  _et al._ (2011) C. Cárdenas, W. Tiznado, P. W. Ayers, and P. Fuentealba, “The fukui potential and the capacity of charge and the global hardness of atoms,” The Journal of Physical Chemistry A 115, 2325–2331 (2011), pMID: 21351729, https://doi.org/10.1021/jp109955q .
  * Laplaza  _et al._ (2021) R. Laplaza, C. Cárdenas, P. Chaquin, J. Contreras-García, and P. W. Ayers, “Orbital energies and nuclear forces in dft: Interpretation and validation,” Journal of Computational Chemistry 42, 334–343 (2021), https://onlinelibrary.wiley.com/doi/pdf/10.1002/jcc.26459 .
  * Cárdenas, Heidar-Zadeh, and Ayers (2016) C. Cárdenas, F. Heidar-Zadeh, and P. W. Ayers, “Benchmark values of chemical potential and chemical hardness for atoms and atomic ions (including unstable anions) from the energies of isoelectronic series,” Phys. Chem. Chem. Phys. 18, 25721–25734 (2016).
  * Gómez  _et al._ (2021) T. Gómez, P. Fuentealba, A. Robles-Navarro, and C. Cárdenas, “Links among the fukui potential, the alchemical hardness and the local hardness of an atom in a molecule,” Journal of Computational Chemistry 42, 1681–1688 (2021), https://onlinelibrary.wiley.com/doi/pdf/10.1002/jcc.26705 .
  * Cohen, Ganduglia-Pirovano, and Kudrnovský (1994) M. H. Cohen, M. V. Ganduglia-Pirovano, and J. Kudrnovský, “Electronic and nuclear chemical reactivity,” The Journal of Chemical Physics 101, 8988–8997 (1994), https://doi.org/10.1063/1.468026 .
  * Cohen, Ganduglia-Pirovano, and Kudrnovský (1995) M. H. Cohen, M. V. Ganduglia-Pirovano, and J. Kudrnovský, “Reactivity kernels, the normal modes of chemical reactivity, and the hardness and softness spectra,” The Journal of Chemical Physics 103, 3543–3551 (1995), https://doi.org/10.1063/1.470238 .
  * Balawender and Geerlings (2001) R. Balawender and P. Geerlings, “Nuclear fukui function from coupled perturbed hartree–fock equations,” The Journal of Chemical Physics 114, 682–691 (2001), https://aip.scitation.org/doi/pdf/10.1063/1.1331359 .
  * Flores-Moreno  _et al._ (2021) R. Flores-Moreno, S. A. Cortes-Llamas, K. Pineda-Urbina, V. M. Medel, and G. K. Jayaprakash, “Analytic alchemical derivatives for the analysis of differential acidity assisted by the h function,” The Journal of Physical Chemistry A 125, 10463–10474 (2021), pMID: 34812636, https://doi.org/10.1021/acs.jpca.1c07364 .
  * Balawender and Komorowski (1998) R. Balawender and L. Komorowski, “Atomic fukui function indices and local softness ab initio,” The Journal of Chemical Physics 109, 5203–5211 (1998), https://doi.org/10.1063/1.477137 .
  * Baekelandt (1996) B. G. Baekelandt, “The nuclear fukui function and berlin’s binding function in density functional theory,” The Journal of chemical physics 105, 4664–4667 (1996).
  * Robles  _et al._ (2018) A. Robles, M. Franco-Pérez, J. L. Gázquez, C. Cárdenas, and P. Fuentealba, “Local electrophilicity,” Journal of Molecular Modeling 24, 245 (2018).
  * Geerlings  _et al._ (2019) P. Geerlings, S. Fias, T. Stuyver, P. Ayers, R. Balawender, and F. De Proft, “New insights and horizons from the linear response function in conceptual dft,” Density Functional Theory , 3–29 (2019).
  * Muñoz  _et al._ (2020) M. Muñoz, A. Robles-Navarro, P. Fuentealba, and C. Cárdenas, “Predicting deprotonation sites using alchemical derivatives,” J. Phys. Chem. A 124, 3754–3760 (2020).
  * Muñoz and Cárdenas (2017) M. Muñoz and C. Cárdenas, “How predictive could alchemical derivatives be?” Phys. Chem. Chem. Phys. 19, 16003–16012 (2017).
  * Balawender and Geerlings (2023) R. Balawender and P. Geerlings, “Chapter 2 - exploring chemical space with alchemical derivatives,” in  _Chemical Reactivity_, edited by S. Kaya, L. von Szentpály, G. Serdaroğlu, and L. Guo (Elsevier, 2023) pp. 15–57.
  * Griego, Saravanan, and Keith (2019) C. D. Griego, K. Saravanan, and J. A. Keith, “Benchmarking computational alchemy for carbide, nitride, and oxide catalysts,” Advanced Theory and Simulations 2, 1800142 (2019).
  * Saravanan  _et al._ (2017) K. Saravanan, J. R. Kitchin, O. A. Von Lilienfeld, and J. A. Keith, “Alchemical predictions for computational catalysis: potential and limitations,” J. Phys. Chem. Lett. 8, 5002–5007 (2017).
  * Griego, Kitchin, and Keith (2021) C. D. Griego, J. R. Kitchin, and J. A. Keith, “Acceleration of catalyst discovery with easy, fast, and reproducible computational alchemy,” International Journal of Quantum Chemistry 121, e26380 (2021).
  * Griego  _et al._ (2021) C. D. Griego, A. M. Maldonado, L. Zhao, B. Zulueta, B. M. Gentry, E. Lipsman, T. H. Choi, and J. A. Keith, “Computationally guided searches for efficient catalysts through chemical/materials space: Progress and outlook,” The Journal of Physical Chemistry C 125, 6495–6507 (2021), https://doi.org/10.1021/acs.jpcc.0c11345 .
  * Keith  _et al._ (2021) J. A. Keith, V. Vassilev-Galindo, B. Cheng, S. Chmiela, M. Gastegger, K.-R. Müller, and A. Tkatchenko, “Combining machine learning and computational chemistry for predictive insights into chemical systems,” Chemical Reviews 121, 9816–9872 (2021), pMID: 34232033, https://doi.org/10.1021/acs.chemrev.1c00107 .
  * von Rudorff and von Lilienfeld (2021) G. F. von Rudorff and O. A. von Lilienfeld, “Simplifying inverse materials design problems for fixed lattices with alchemical chirality,” Science Advances 7, eabf1173 (2021).
  * Von Lilienfeld and Tuckerman (2007) O. A. Von Lilienfeld and M. Tuckerman, “Alchemical variations of intermolecular energies according to molecular grand-canonical ensemble density functional theory,” Journal of chemical theory and computation 3, 1083–1090 (2007).
  * von Rudorff and von Lilienfeld (2020a) G. F. von Rudorff and O. A. von Lilienfeld, “Alchemical perturbation density functional theory,” Phys. Rev. Research 2 (2020a), 10.1103/physrevresearch.2.023220.
  * von Lilienfeld (2009) O. A. von Lilienfeld, “Accurate ab initio energy gradients in chemical compound space,” J. Chem. Phys. 131, 164102 (2009).
  * von Rudorff and von Lilienfeld (2019) G. F. von Rudorff and O. A. von Lilienfeld, “Atoms in molecules from alchemical perturbation density functional theory,” J. Phys. Chem. B (2019), 10.1021/acs.jpcb.9b07799.
  * von Rudorff and von Lilienfeld (2020b) G. F. von Rudorff and O. A. von Lilienfeld, “Rapid and accurate molecular deprotonation energies from quantum alchemy,” Phys. Chem. Chem. Phys. (2020b), 10.1039/c9cp06471k.
  * Von Lilienfeld and Tuckerman (2006) O. A. Von Lilienfeld and M. E. Tuckerman, “Molecular grand-canonical ensemble density functional theory and exploration of chemical space,” J.Chem.Phys. 125, 154104 (2006).
  * Marcon, von Lilienfeld, and Andrienko (2007) V. Marcon, O. A. von Lilienfeld, and D. Andrienko, “Tuning electronic eigenvalues of benzene via doping,” J.Chem.Phys. 127, 064305 (2007).
  * Krug, von Rudorff, and von Lilienfeld (2022) S. L. Krug, G. F. von Rudorff, and O. A. von Lilienfeld, “Relative energies without electronic perturbations via alchemical integral transform,” The Journal of Chemical Physics 157 (2022).
  * Krug and von Lilienfeld (2023) S. L. Krug and O. A. von Lilienfeld, “Generalized alchemical integral transform and the multi-electron atom energy,” arXiv preprint arXiv:2312.04458 (2023).
  * von Lilienfeld and Domenichini (2023) O. A. von Lilienfeld and G. Domenichini, “Even order contributions to relative energies vanish for antisymmetric perturbations,” arXiv preprint arXiv:2306.16409 (2023).
  * Clementi (1965) E. Clementi, “Tables of atomic functions,” IBM Journal of Research and Development 9, 87–89 (1965).
  * Clementi and Davis (1966) E. Clementi and D. Davis, “Electronic structure of large molecular systems,” Journal of Computational Physics 1, 223–244 (1966).
  * Balawender  _et al._ (2019) R. Balawender, M. Lesiuk, F. De Proft, C. Van Alsenoy, and P. Geerlings, “Exploring chemical space with alchemical derivatives: alchemical transformations of h through ar and their ions as a proof of concept,” Phys. Chem. Chem. Phys. 21, 23865–23879 (2019).
  * Domenichini, von Rudorff, and von Lilienfeld (2020) G. Domenichini, G. F. von Rudorff, and O. A. von Lilienfeld, “Effects of perturbation order and basis set on alchemical predictions,” The Journal of Chemical Physics 153, 144118 (2020), https://doi.org/10.1063/5.0023590 .
  * Hehre, Stewart, and Pople (1969) W. J. Hehre, R. F. Stewart, and J. A. Pople, “Self consistent molecular orbital methods. I. Use of Gaussian expansions of slater type atomic orbitals,” J. Chem. Phys. 51, 2657–2664 (1969).
  * Binkley, Pople, and Hehre (1980) J. S. Binkley, J. A. Pople, and W. J. Hehre, “Self-consistent molecular orbital methods. 21 small split-valence basis sets for first-row elements,” Journal of the American Chemical Society 102, 939–947 (1980).
  * Francl  _et al._ (1982) M. M. Francl, W. J. Pietro, W. J. Hehre, J. S. Binkley, M. S. Gordon, D. J.DeFrees, and J. A. Pople, “Self-consistent molecular orbital methods. XXIII. A polarization-type basis set for second-row elements,” J. Chem. Phys. 77, 3654–3665 (1982).
  * Thorn H. Dunning (1989) J. Thorn H. Dunning, “Gaussian basis sets for use in correlated molecular calculations I. The atoms boron through neon and hydrogen,” J. Chem. Phys. 90, 1007–1023 (1989).
  * Woon and Dunning (1993) D. E. Woon and T. H. Dunning, “Gaussian basis sets for use in correlated molecular calculations. III. the atoms aluminum through argon,” J. Chem. Phys. 98, 1358–1371 (1993).
  * Schäfer, Huber, and Ahlrichs (1994) A. Schäfer, C. Huber, and R. Ahlrichs, “Fully optimized contracted gaussian basis sets of triple zeta valence quality for atoms Li to Kr,” J. Chem. Phys. 100, 5829–5835 (1994).
  * Eikey  _et al._ (2022a) E. A. Eikey, A. M. Maldonado, C. D. Griego, G. F. von Rudorff, and J. A. Keith, “Evaluating quantum alchemy of atoms with thermodynamic cycles: Beyond ground electronic states,” The Journal of Chemical Physics 156, 064106 (2022a), https://pubs.aip.org/aip/jcp/article-pdf/doi/10.1063/5.0079483/16535355/064106_1_online.pdf .
  * Eikey  _et al._ (2022b) E. A. Eikey, A. M. Maldonado, C. D. Griego, G. F. von Rudorff, and J. A. Keith, “Quantum alchemy beyond singlets: Bonding in diatomic molecules with hydrogen,” The Journal of Chemical Physics 156, 204111 (2022b), https://pubs.aip.org/aip/jcp/article-pdf/doi/10.1063/5.0079487/16542483/204111_1_online.pdf .
  * Chang  _et al._ (2016) K. Y. S. Chang, S. Fias, R. Ramakrishnan, and O. A. von Lilienfeld, “Fast and accurate predictions of covalent bonds in chemical space,” J. Chem. Phys. 144, 174110 (2016), https://doi.org/10.1063/1.4947217 .
  * Fias, Chang, and von Lilienfeld (2019) S. Fias, K. Y. S. Chang, and O. A. von Lilienfeld, “Alchemical normal modes unify chemical space,” J. Phys. Chem. Lett. 10, 30–39 (2019).
  * Domenichini and von Lilienfeld (2022) G. Domenichini and O. A. von Lilienfeld, “Alchemical geometry relaxation,” The Journal of Chemical Physics 156 (2022).
  * Woon and Dunning (1995) D. E. Woon and J. Dunning, Thom H., “Gaussian basis sets for use in correlated molecular calculations. V. Core-valence basis sets for boron through neon,” The Journal of Chemical Physics 103, 4572–4585 (1995), https://pubs.aip.org/aip/jcp/article-pdf/103/11/4572/10777979/4572_1_online.pdf .
  * Jensen (2001) F. Jensen, “Polarization consistent basis sets: Principles,” J. Chem. Phys. , 9113–9125 (2001).
  * Jensen (2002) F. Jensen, “Polarization consistent basis sets. ii. estimating the kohn-sham basis set limit,” J. Chem. Phys. , 7372–7379 (2002).
  * Jensen (2007) F. Jensen, “Polarization consistent basis sets. 4: The elements he, li, be, b, ne, na, mg, al, and ar,” J. Phys. Chem. A , 11198–11204 (2007).
  * Ambroise and Jensen (2019) M. A. Ambroise and F. Jensen, “Probing basis set requirements for calculating core ionization and core excitation spectroscopy by the ${\Delta}$ self-consistent-field approach,” Journal of Chemical Theory and Computation 15, 325–337 (2019), https://doi.org/10.1021/acs.jctc.8b01071 .
  * de Castro and Jorge (1998) E. V. R. de Castro and F. E. Jorge, “Accurate universal gaussian basis set for all atoms of the periodic table,” J. Chem. Phys. 108, 5225–5229 (1998), https://doi.org/10.1063/1.475959 .
  * Kresse and Furthmüller (1996) G. Kresse and J. Furthmüller, “Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set,” Phys. Rev. B 54, 11169–11186 (1996).
  * Blöchl (1994) P. E. Blöchl, “Projector augmented-wave method,” Phys. rev. B 50, 17953 (1994).
  * Lesiuk, Balawender, and Zachara (2012) M. Lesiuk, R. Balawender, and J. Zachara, “Higher order alchemical derivatives from coupled perturbed self-consistent field theory,” J. Chem. Phys. 136, 034104 (2012), https://doi.org/10.1063/1.3674163 .
  * Sun  _et al._ (2017) Q. Sun, T. C. Berkelbach, N. S. Blunt, G. H. Booth, S. Guo, Z. Li, J. Liu, J. D. McClain, E. R. Sayfutyarova, S. Sharma, S. Wouters, and G. K. Chan, “Pyscf: the python-based simulations of chemistry framework,”  (2017), https://onlinelibrary.wiley.com/doi/pdf/10.1002/wcms.1340 .
  * Sun  _et al._ (2020) Q. Sun, X. Zhang, S. Banerjee, P. Bao, M. Barbry, N. S. Blunt, N. A. Bogdanov, G. H. Booth, J. Chen, Z.-H. Cui, _et al._ , “Recent developments in the pyscf program package,” The Journal of chemical physics 153 (2020).
  * Domenichini (2023) G. Domenichini, “Supplementary code for quantum alchemy,” (accessed in December 2023), Github project: https://github.com/giorgiodomen/Supplementary_code_for_Quantum_Alchemy .
  * Van Rossum and Drake Jr (1995) G. Van Rossum and F. L. Drake Jr,  _Python tutorial_ , Vol. 620 (Centrum voor Wiskunde en Informatica Amsterdam, The Netherlands, 1995).
  * Pérez and Granger (2007) F. Pérez and B. E. Granger, “IPython: a system for interactive scientific computing,” Computing in Science and Engineering 9, 21–29 (2007).
  * Harris  _et al._ (2020) C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant, “Array programming with NumPy,” Nature 585, 357–362 (2020).
  * Virtanen  _et al._ (2020) P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors, “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,” Nature Methods 17, 261–272 (2020).
  * Hunter (2007) J. D. Hunter, “Matplotlib: A 2d graphics environment,” Computing in science & engineering 9, 90–95 (2007).
  * Pople  _et al._ (1979) J. Pople, R. Krishnan, H. Schlegel, and J. S. Binkley, “Derivative studies in hartree-fock and møller-plesset theories,” International Journal of Quantum Chemistry 16, 225–241 (1979).
  * Moccia (1970) R. Moccia, “Variable bases in scf mo calculations,” Chemical Physics Letters 5, 260–264 (1970).
  * Bishop and Randić (2004) D. M. Bishop and M. Randić, “Ab Initio Calculation of Harmonic Force Constants,” The Journal of Chemical Physics 44, 2480–2487 (2004), https://pubs.aip.org/aip/jcp/article-pdf/44/6/2480/11152702/2480_1_online.pdf .
  * Caves and Karplus (1969) T. C. Caves and M. Karplus, “Perturbed hartree–fock theory. i. diagrammatic double-perturbation analysis,” The Journal of Chemical Physics 50, 3649–3661 (1969), https://doi.org/10.1063/1.1671609 .
  * Dalgarno (1962) A. Dalgarno, “Atomic polarizabilities and shielding factors,” Advances in Physics 11, 281–315 (1962), https://doi.org/10.1080/00018736200101302 .
  * Cammi, Cossi, and Tomasi (1996) R. Cammi, M. Cossi, and J. Tomasi, “Analytical derivatives for molecular solutes. iii. hartree–fock static polarizability and hyperpolarizabilities in the polarizable continuum model,” The Journal of Chemical Physics 104, 4611–4620 (1996), https://doi.org/10.1063/1.471208 .
  * Balawender  _et al._ (2018a) R. Balawender, M. Lesiuk, F. De Proft, and P. Geerlings, “Exploring chemical space with alchemical derivatives: Bn-simultaneous substitution patterns in c60,” J. Chem. Theory Comput. 14, 1154–1168 (2018a), pMID: 29300479, https://doi.org/10.1021/acs.jctc.7b01114 .
  * Balawender  _et al._ (2013) R. Balawender, M. A. Welearegay, M. Lesiuk, F. De Proft, and P. Geerlings, “Exploring chemical space with the alchemical derivatives,” J. Chem. Theory Comput. 9, 5327–5340 (2013), pMID: 26592270, https://doi.org/10.1021/ct400706g .
  * Balawender  _et al._ (2018b) R. Balawender, A. Holas, F. De Proft, C. Van Alsenoy, and P. Geerlings, “Alchemical derivatives of atoms: A walk through the periodic table,” in  _Many-body Approaches at Different Scales: A Tribute to Norman H. March on the Occasion of his 90th Birthday_, edited by G. Angilella and C. Amovilli (Springer International Publishing, Cham, 2018) pp. 227–251.
  * Shiraogawa and Hasegawa (2023) T. Shiraogawa and J.-y. Hasegawa, “Optimization of general molecular properties in the equilibrium geometry using quantum alchemy: An inverse molecular design approach,” chemarchive.org (2023).
  * Shiraogawa and Hasegawa (2022) T. Shiraogawa and J.-y. Hasegawa, “Exploration of chemical space for designing functional molecules accounting for geometric stability,” The Journal of Physical Chemistry Letters 13, 8620–8627 (2022).
  * Al-Hamdani, Michaelides, and von Lilienfeld (2017) Y. S. Al-Hamdani, A. Michaelides, and O. A. von Lilienfeld, “Exploring dissociative water adsorption on isoelectronically bn doped graphene using alchemical derivatives,” J. Chem. Phys. 147, 164113 (2017), https://doi.org/10.1063/1.4986314 .
  * Sheppard, Henkelman, and von Lilienfeld (2010) D. Sheppard, G. Henkelman, and O. A. von Lilienfeld, “Alchemical derivatives of reaction energetics,” J. Chem. Phys. 133, 084104 (2010), https://doi.org/10.1063/1.3474502 .
  * Solovyeva and von Lilienfeld (2016) A. Solovyeva and O. A. von Lilienfeld, “Alchemical screening of ionic crystals,” Phys. Chem. Chem. Phys. 18, 31078–31091 (2016).
  * von Lilienfeld (2013) O. A. von Lilienfeld, “First principles view on chemical compound space: Gaining rigorous atomistic control of molecular properties,” International Journal of Quantum Chemistry 113, 1676–1689 (2013), https://onlinelibrary.wiley.com/doi/pdf/10.1002/qua.24375 .
  * Chang and von Lilienfeld (2018) K. Y. S. Chang and O. A. von Lilienfeld, “$\text{Al}_{x}\text{Ga}_{1-x}\text{As}$ crystals with direct 2 ev band gaps from computational alchemy,” Phys. Rev. Materials 2 (2018), 10.1103/PhysRevMaterials.2.073802.
  * Sahre, von Rudorff, and von Lilienfeld (2023) M. J. Sahre, G. F. von Rudorff, and O. A. von Lilienfeld, “Quantum alchemy based bonding trends and their link to hammett’s equation and pauling’s electronegativity model,” Journal of the American Chemical Society 145, 5899–5908 (2023).
  * Sahre  _et al._ (2023) M. J. Sahre, G. F. von Rudorff, P. Marquetand, and O. A. von Lilienfeld, “Transferability of atomic energies from alchemical decomposition,” (2023), arXiv:2311.04784 [physics.chem-ph] .
  * Griego  _et al._ (2020) C. D. Griego, L. Zhao, K. Saravanan, and J. A. Keith, “Machine learning corrected alchemical perturbation density functional theory for catalysis applications,” AIChE Journal 66, e17041 (2020), https://aiche.onlinelibrary.wiley.com/doi/pdf/10.1002/aic.17041 .
  * Caracas and Banigan (2009) R. Caracas and E. J. Banigan, “Elasticity and raman and infrared spectra of mgal2o4 spinel from density functional perturbation theory,” Physics of the Earth and Planetary Interiors 174, 113–121 (2009).
  * Marzari, de Gironcoli, and Baroni (1994) N. Marzari, S. de Gironcoli, and S. Baroni, “Structure and phase stability of ${\mathrm{Ga}}_{\mathit{x}}$ ${\mathrm{In}}_{1\mathrm{-}\mathit{x}}$ P solid solutions from computational alchemy,” Phys. Rev. Lett. 72, 4001–4004 (1994).
  * Saitta, De Gironcoli, and Baroni (1998) A. Saitta, S. De Gironcoli, and S. Baroni, “Structural and electronic properties of a wide-gap quaternary solid solution:$\backslash$(zn, mg$\backslash$)$\backslash$(s, se$\backslash$),” Physical review letters 80, 4939 (1998).
  * Bellaiche, García, and Vanderbilt (2002) L. Bellaiche, A. García, and D. Vanderbilt, “Low-temperature properties of pb (zr 1- x ti x) o 3 solid solutions near the morphotropic phase boundary,” Ferroelectrics 266, 41–56 (2002).
  * Von Lilienfeld (2018) O. A. Von Lilienfeld, “Quantum machine learning in chemical compound space,” Angewandte Chemie International Edition 57, 4164–4169 (2018).
  * Rupp (2015) M. Rupp, “Machine learning for quantum mechanics in a nutshell,” International Journal of Quantum Chemistry 115, 1058–1073 (2015).
  * von Lilienfeld, Müller, and Tkatchenko (2020) O. A. von Lilienfeld, K.-R. Müller, and A. Tkatchenko, “Exploring chemical compound space with quantum-based machine learning,” Nature Reviews Chemistry 4, 347–358 (2020).
  * Rupp  _et al._ (2012) M. Rupp, A. Tkatchenko, K.-R. Müller, and O. A. Von Lilienfeld, “Fast and accurate modeling of molecular atomization energies with machine learning,” Phys. Rev. Lett. 108, 058301 (2012).
  * Faber  _et al._ (2017) F. A. Faber, L. Hutchison, B. Huang, J. Gilmer, S. S. Schoenholz, G. E. Dahl, O. Vinyals, S. Kearnes, P. F. Riley, and O. A. Von Lilienfeld, “Prediction errors of molecular machine learning models lower than hybrid dft error,” J. Chem. Theory Comput. 13, 5255–5264 (2017).
  * Christensen, Faber, and von Lilienfeld (2019) A. S. Christensen, F. A. Faber, and O. A. von Lilienfeld, “Operators in quantum machine learning: Response properties in chemical space,” The Journal of chemical physics 150, 064105 (2019).
  * Weinreich, Browning, and von Lilienfeld (2021) J. Weinreich, N. J. Browning, and O. A. von Lilienfeld, “Machine learning of free energies in chemical compound space using ensemble representations: Reaching experimental uncertainty for solvation,” The Journal of Chemical Physics 154, 134113 (2021).
  * Pilania  _et al._ (2013) G. Pilania, C. Wang, X. Jiang, S. Rajasekaran, and R. Ramprasad, “Accelerating materials property predictions using machine learning,” Scientific Reports 3, 1–6 (2013).
  * Ceriotti, Clementi, and Anatole von Lilienfeld (2021) M. Ceriotti, C. Clementi, and O. Anatole von Lilienfeld, “Introduction: Machine learning at the atomic scale,” (2021).

[Supplementary materials]Supplementary materials to the paper "Extending the
definition of atomic basis sets to atoms with fractional nuclear charge"
Giorgio Domenichini

Table 3: Third order alchemical perturbation (AP) energies of diatomic molecules using basis set of the reference, or the alchemical consistent basis set (ACBS). Basis set | Ref. | Targ. | $E^{T}$ | AP3 ACBS | AP3 ref.BS |  $\Delta E$ ACBS |  $\Delta E$ ref.BS
---|---|---|---|---|---|---|---
3-21G | N2 | NO+ | -128.1332 | -128.1310 | -127.0931 | 0.0022 | 1.0401
3-21G | N2 | CN- | -91.7316 | -91.7354 | -90.8583 | -0.0038 | 0.8733
3-21G | N2 | CO | -112.0879 | -112.1286 | -110.2232 | -0.0406 | 1.8647
3-21G | CO | CF+ | -136.0534 | -136.0556 | -135.0153 | -0.0021 | 1.0381
3-21G | CO | NO+ | -128.1332 | -128.1252 | -127.0740 | 0.0080 | 1.0592
3-21G | CO | BO- | -98.9287 | -98.9233 | -98.0602 | 0.0053 | 0.8685
3-21G | CO | CN- | -91.7316 | -91.7309 | -90.9029 | 0.0006 | 0.8287
3-21G | CO | N2 | -108.3009 | -108.2912 | -106.4623 | 0.0097 | 1.8387
3-21G | CO | BF | -123.3550 | -123.3551 | -121.5055 | -0.0001 | 1.8495
3-21G | BF | BNe+ | -151.6377 | -151.6466 | -150.5707 | -0.0089 | 1.0670
3-21G | BF | CF+ | -136.0534 | -136.0536 | -134.9252 | -0.0001 | 1.1283
3-21G | BF | BeF- | -113.3302 | -113.3359 | -112.4327 | -0.0057 | 0.8975
3-21G | BF | BO- | -98.9287 | -98.9363 | -98.1199 | -0.0077 | 0.8088
3-21G | BF | CO | -112.0879 | -112.0907 | -110.2067 | -0.0027 | 1.8813
3-21G | BF | BeNe | -142.0281 | -142.0463 | -140.0919 | -0.0182 | 1.9363
6-31G | N2 | NO+ | -128.7977 | -128.7971 | -127.7504 | 0.0006 | 1.0473
6-31G | N2 | CN- | -92.2174 | -92.2133 | -91.3502 | 0.0041 | 0.8672
6-31G | N2 | CO | -112.6616 | -112.6696 | -110.7722 | -0.0080 | 1.8893
6-31G | CO | CF+ | -136.7430 | -136.7460 | -135.7128 | -0.0030 | 1.0302
6-31G | CO | NO+ | -128.7977 | -128.7923 | -127.7359 | 0.0054 | 1.0618
6-31G | CO | BO- | -99.4492 | -99.4464 | -98.5853 | 0.0028 | 0.8639
6-31G | CO | CN- | -92.2174 | -92.2201 | -91.3409 | -0.0027 | 0.8765
6-31G | CO | N2 | -108.8679 | -108.8713 | -106.9614 | -0.0034 | 1.9065
6-31G | CO | BF | -123.9888 | -123.9932 | -122.1201 | -0.0044 | 1.8687
6-31G | BF | BNe+ | -152.4083 | -152.4127 | -151.4068 | -0.0044 | 1.0016
6-31G | BF | CF+ | -136.7430 | -136.7411 | -135.6697 | 0.0019 | 1.0733
6-31G | BF | BeF- | -113.9089 | -113.9039 | -113.0392 | 0.0049 | 0.8697
6-31G | BF | BO- | -99.4492 | -99.4523 | -98.5550 | -0.0031 | 0.8942
6-31G | BF | CO | -112.6616 | -112.6566 | -110.7279 | 0.0049 | 1.9336
6-31G | BF | BeNe | -142.7390 | -142.7342 | -140.8718 | 0.0048 | 1.8671
cc-pvDZ | N2 | NO+ | -128.9207 | -128.9180 | -127.8830 | 0.0027 | 1.0377
cc-pvDZ | N2 | CN- | -92.2820 | -92.2802 | -91.4117 | 0.0018 | 0.8703
cc-pvDZ | N2 | CO | -112.7483 | -112.7589 | -110.8809 | -0.0106 | 1.8674
cc-pvDZ | CO | CF+ | -136.8473 | -136.8490 | -135.8296 | -0.0017 | 1.0177
cc-pvDZ | CO | NO+ | -128.9207 | -128.9163 | -127.8615 | 0.0044 | 1.0592
cc-pvDZ | CO | BO- | -99.5217 | -99.5175 | -98.6531 | 0.0041 | 0.8685
cc-pvDZ | CO | CN- | -92.2820 | -92.2823 | -91.4156 | -0.0004 | 0.8663
cc-pvDZ | CO | N2 | -108.9554 | -108.9605 | -107.0683 | -0.0051 | 1.8871
cc-pvDZ | CO | BF | -124.0575 | -124.0605 | -122.2099 | -0.0031 | 1.8475
cc-pvDZ | BF | BNe+ | -152.4629 | -152.4681 | -151.4674 | -0.0052 | 0.9955
cc-pvDZ | BF | CF+ | -136.8473 | -136.8434 | -135.7591 | 0.0038 | 1.0882
cc-pvDZ | BF | BeF- | -113.9656 | -113.9620 | -113.1076 | 0.0036 | 0.8580
cc-pvDZ | BF | BO- | -99.5217 | -99.5250 | -98.6259 | -0.0034 | 0.8958
cc-pvDZ | BF | CO | -112.7483 | -112.7363 | -110.8019 | 0.0120 | 1.9464
cc-pvDZ | BF | BeNe | -142.7729 | -142.7520 | -140.9158 | 0.0210 | 1.8571
Table 4:  Second order alchemical perturbation energies of B-N doped benzene’s mutants, using basis set of the reference, or the alchemical consistent basis set. Substitution | AP2 ref.BS | $E^{T[R]}$ | AP2 ACBS | $E^{T}$
---|---|---|---|---
1N 2B | -232.2207 | -232.2638 | -234.0194 | -234.0179
1N 3B | -232.1337 | -232.1833 | -233.9415 | -233.9453
1N 4B | -232.1521 | -232.2011 | -233.9575 | -233.9625
1,2 N 3,4 B | -233.5743 | -233.7186 | -237.1966 | -237.2612
1,2 N 3,5 B | -233.6614 | -233.7664 | -237.2745 | -237.2895
1,2 N 3,6 B | -233.7116 | -233.8600 | -237.3205 | -237.3774
1,2 N 4,5 B | -233.5057 | -233.6205 | -237.1347 | -237.1658
1,3 N 2,4 B | -233.9224 | -233.9920 | -237.5080 | -237.4881
1,3 N 2,5 B | -233.7986 | -233.8982 | -237.3983 | -237.4021
1,3 N 4,5 B | -233.6614 | -233.7425 | -237.2745 | -237.2776
1,3 N 4,6 B | -233.8538 | -233.8843 | -237.4461 | -237.3925
1,4 N 2,3 B | -233.7116 | -233.8326 | -237.3205 | -237.3640
1,4 N 2,5 B | -233.7802 | -233.8728 | -237.3824 | -237.3844
1,4 N 2,6 B | -233.7986 | -233.8799 | -237.3983 | -237.3915
1,2,3 N 4,5,6 B | -235.0334 | -235.2137 | -240.4677 | -240.5377
1,2,4 N 3,5,6 B | -235.3078 | -235.5129 | -240.7153 | -240.7952
1,3,5 N 2,4,6 B | -235.7295 | -235.7904 | -241.0905 | -241.0183


#  Struc-Bench: Are Large Language Models Really Good at Generating Complex
Structured Data?

Xiangru Tang♠ Yiming Zong♡ Jason Phang♢ Yilun Zhao♠ Wangchunshu Zhou♣
Arman Cohan♠ Mark Gerstein♠11footnotemark: 1
♠ Yale University ♡ Zhejiang University ♢ New York University ♣ ETH Zurich
{xiangru.tang, arman.cohan<EMAIL_ADDRESS>Contributed equally

Despite the power of Large Language Models (LLMs) like GPT-4, they still
struggle with tasks that require generating complex, structured outputs. In
this study, we assess the capability of Current LLMs in generating complex
structured data and propose a structure-aware fine-tuning approach as a
solution to improve this ability. To perform a comprehensive evaluation, we
propose Struc-Bench, include five representative LLMs (i.e., GPT-NeoX-20B,
GPT-3.5, GPT-4, and Vicuna) and evaluate them on our carefully constructed
datasets spanning raw text, HTML, and LaTeX tables. Based on our analysis of
current model performance, we identify specific common formatting errors and
areas of potential improvement. To address complex formatting requirements, we
utilize a FormatCoT (Chain-of-Thought) to generate format instructions from
target outputs. Our experiments show that our structure-aware fine-tuning
method, when applied to LLaMA-7B, significantly improves adherence to natural
language constraints, outperforming other evaluated LLMs. Based on these
results, we present an ability map of model capabilities from six dimensions
(i.e., coverage, formatting, reasoning, comprehension, pragmatics, and
hallucination). This map highlights the weaknesses of LLMs in handling complex
structured outputs and suggests promising directions for future work. Our code
and models can be found at https://github.com/gersteinlab/Struc-Bench.

##  1 Introduction

Significant advancements have been made in various natural language processing
tasks by Large Language Models (LLMs) Brown et al. (2020); Scao et al. (2022);
Ouyang et al. (2022); Muennighoff et al. (2022); OpenAI (2023); Zhao et al.
(2023a), especially in text generation tasks Qin et al. (2023). The ability to
output structured data, one of the key aspects of generative capability, has
also attracted great interest in previous studies Wu et al. (2022); Zhao et
al. (2023c, b).

Figure 1: A system for describing complex structured formats and learning to
follow this format in human language. We use zero-shot for inference.

However, LLMs still underperform in generating complex structured outputs–a
critical ability for various applications ranging from coding assistance to
automated report writing. Furthermore, most evaluation of LLMs has been on
natural text or code generation, and relatively less research has been
conducted to evaluate LLMs on their ability to generate structured output.
This leaves it unclear whether LLMs can generate complex structured data
effectively. We aim to address these unanswered questions and deliver an in-
depth examination in our research.

First, there is a lack of systematic analysis of the ability of LLMs to output
complex structured data. Previous efforts on evaluating LLMs Qin et al.
(2023); Ma et al. (2023) on structured data primarily centered around simple
Information Extraction (IE) tasks: recogniting named entities, extracting
relations, and detecting events. Here the goal of IE tasks is to gathered the
extracted data in a highly structured form Zhong and Chen (2020). Much earlier
work was considerably more task-centric as opposed to LLM-centric. The focus
was predominantly on generating structured data from text (text-to-data) tasks
with pre-trained models He et al. (2023); Rossiello et al. (2022); Whitehouse
et al. (2023); Pietruszka et al. (2022) like BART Lewis et al. (2019) and T5
Raffel et al. (2020).

Second, there is a lack of fine-grained evaluation and comprehensive
benchmarks of LLMs performance. Existing benchmarks often rely on rudimentary
objective metrics such as word overlap to measure the accuracy of the content
generated by the model Li et al. (2023); Wu et al. (2022); Pietruszka et al.
(2022). This may be insufficient for evaluating whether LLMs can generate
structured output, as an ideal evaluation metric ought to also consider the
format of generated content.

Third, is there potential for enhancing the performance of current LLMs to
better follow human natural language inputs, thereby generating outputs with
the accurate format and error-free content?

This work aims to fill in these gaps in the literature and expand on both the
evaluation metrics and training datasets for LLMs generating structured
output. Our contributions are summarized as:

(1) We develop a benchmark, called Struc-Bench focusing on generating
structured texts in raw text, HTML, and LaTeX formats, and thoroughly examine
the capabilities of popular LLMs, uncovering key issues in content accuracy,
formatting, numerical reasoning, and handling long tables. (2) Incorporating
prominent datasets and expanding to diverse domains, we conduct empirical
evaluations of popular LLMs on our structured text generation benchmark,
providing a deeper understanding of the prevalent error types and dimensions
of shortcomings. Our findings suggest that both GPT-3.5 and GPT-4 struggle to
produce outputs that are exactly correct, with issues primarily stemming from
erroneous content, inaccurate formatting, inadequate numerical reasoning
abilities, and their inability to handle long tables. (3) To address these
issues, we introduce structure-aware instruction tuning, using ChatGPT to
generate format instructions and then training the LLaMA model to follow these
formats. The promising results on both seen and unseen data indicate that it
could greatly enhance the ability of LLMs to generate structured outputs.

##  2 Problem Analysis and Benchmark

###  2.1 Preliminary

The task of generating complex structured data presents a notable challenge
that tests the capabilities of LLMs in producing intricate, format-specific
outputs. This task moves beyond conventional text generation. The complexity
lies not only in the need to generate accurate and coherent content but also
in maintaining a strict and specific data structure or format. For example,
text-to-table is a task that aims to convert unstructured textual data into
structured tabular data, by extracting necessary contents from text and
following the required structure or format.

###  2.2 Problem Analysis

In our study, we have identified a significant limitation of GPT-3.5 and GPT-4
in handling complex structured output. Despite being state-of-the-art LLMs
developed by OpenAI, these models both have demonstrated certain limitations
in generating output in more intricate formats, examples could be found in
Appendix A.

This shortcoming becomes evident when the model is tasked with producing data
that adhere to specific structural formats or templates, such as tables. We
find that only 3% of the output of GPT-3.5 111In all our scenarios we are
using Azure OpenAI Service models. GPT-3.5 means gpt-35-turbo. We noticed that
the results of the Azure deployed gpt-35-turbo-v0301 model diverge
substantially from OpenAI gpt-3.5-turbo-0301. is completely correct, while
GPT-4 is only 9%. This could be attributed to the inherent design of the GPT
family, which, while excelling at capturing the statistical patterns of human
language, does not specifically account for structured outputs that require
maintaining a state across a longer span of tokens. Here, we select Rotowire
as an investigation, as shown in Appendix B. We utilized the crowdsourcing
approach on MTurk (See Appendix C) to examine the error types in 100 example
instances. Figure 2 presents the proportions of errors and each error type:
Element Errors, Element Format Errors, Structure Error, Structure Naming
Errors.

Figure 2: Error analysis by human annotation. Some error types are explained
in Appendix A.

###  2.3 Benchmark

In our investigation, we incorporate four prominent data-to-text datasets:
Rotowire (Wiseman et al., 2017), E2E (Novikova et al., 2017), WikiTableText
(Bao et al., 2018), and WikiBio (Lebret et al., 2016), we specifically
selected tables with dimensions greater than 3x3 to ensure a sufficient level
of complexity. Concurrently, we construct more diverse datasets drawn from
broader domains, encompassing tables from LaTeX and HTML data sourced from
GitHub. Each of these table types comes with its unique nuances, complexities,
and levels of structuration, providing extensive coverage for our experiments.
Table 1 gives statistics for the Rotowire dataset and our constructed
datasets. Through empirical testing, we evaluate the capacity of popular LLMs,
including GPT-NeoX-20B Black et al. (2022), GPT-3.5 Ouyang et al. (2022),
GPT-4 OpenAI (2023) and Vicuna-13B Chiang et al. (2023), on our Struc-Bench,
see Section 4.2. For LaTex and HTML without paired text, we use GPT-3.5 to
construct synthetic descriptions as input for our benchmark.

Dataset | # Train | # Test | Format | Rows & Columns
---|---|---|---|---
Rotowire Wiseman et al. (2017) | 3.4k | 728 | Raw tex | 7.26 & 8.75
Struc-Bench LaTeX | 5.3k | 500 | LaTeX | 2.75 & 4.47
Struc-Bench HTML | 5.4k | 499 | HTML | 5.50 & 3.54

Table 1: Struc-Bench data statistics. The number of Rows & Columns has been
averaged.

Raw text tables are more informal, unstandardized, and often need manual
interpretation. In contrast, LaTeX tables are used for scientific documents
and demand high precision in their structure and syntax. HTML tables, widely
used on the web, carry their own tags and structure, aligning with the rules
of HTML language.

Model | SacreBLEU | ROUGE-L | BERTScore | BARTScore | BLEURT | Content GPTscore | Format GPTscore | Content H-Score | Format H-Score
---|---|---|---|---|---|---|---|---|---
_Tables from Raw Text_
GPT-NeoX-20B | 35.24 | 55.78 | 68.91 | -2.34 | 33.51 | 3.86 | 6.10 | 0.50 | -1.32
GPT-3.5 | 56.92 | 70.97 | 91.35 | -1.68 | 36.85 | 6.19 | 8.16 | 0.52 | -1.27
GPT-4 | 68.13 | 75.44 | 94.89 | -0.99 | 55.24 | 6.88 | 8.30 | 0.85 | 0.53
Vicuna-13B | 40.12 | 50.77 | 75.21 | -2.05 | 40.02 | 4.07 | 6.33 | 0.55 | -1.38
Ours-7B | 90.6 | 88.98 | 98.54 | -0.69 | 66.07 | 7.69 | 8.60 | 1.65 | 3.61
$w.o.finetune$ | 9.9 | 36.56 | 81.63 | -2.50 | 70.24 | 4.58 | 6.00 | 0.51 | -1.01
_LaTeX_
GPT-NeoX-20B | 45.92 | 65.10 | 76.09 | -2.05 | 40.87 | 7.23 | 7.02 | 0.56 | 0.72
GPT-3.5 | 56.94 | 75.99 | 86.25 | -1.30 | 42.89 | 8.22 | 8.41 | 0.99 | 1.27
GPT-4 | 78.15 | 85.34 | 88.07 | -1.09 | 67.11 | 8.78 | 8.81 | 1.10 | 1.35
Vicuna-13B | 50.80 | 69.48 | 80.44 | -1.07 | 36.74 | 7.70 | 8.10 | 0.78 | 1.06
Ours-7B | 89.13 | 88.99 | 98.55 | -0.69 | 66.07 | 8.94 | 9.05 | 1.14 | 1.52
$w.o.finetune$ | 47.24 | 70.89 | 73.27 | -2.13 | 38.13 | 7.10 | 6.98 | 0.51 | 0.69
_HTML_
GPT-NeoX-20B | 60.36 | 72.13 | 86.88 | -1.59 | 30.06 | 8.42 | 8.94 | 0.81 | 0.92
GPT-3.5 | 73.80 | 85.19 | 96.76 | -1.46 | 34.81 | 9.11 | 9.35 | 1.10 | 2.15
GPT-4 | 79.25 | 85.95 | 97.22 | -1.31 | 41.59 | 9.17 | 9.62 | 1.15 | 2.29
Vicuna-13B | 58.75 | 70.37 | 88.65 | -1.58 | 31.11 | 8.55 | 8.88 | 0.79 | 0.93
Ours-7B | 77.50 | 86.08 | 96.25 | -1.30 | 42.89 | 9.20 | 9.70 | 1.18 | 2.49
$w.o.finetune$ | 65.30 | 78.24 | 88.12 | -1.57 | 32.78 | 8.22 | 8.81 | 0.92 | 0.96

Table 2: Automated evaluation results on the test set, involving five types of
previous metrics and four proposed ones. $w.o.finetune$ means that we also
compared the performance of our model without structure-aware finetuning as an
ablation study.

##  3 Methodology

###  3.1 Data Generation

As shown in Figure 1, we propose FormatCoT and self-instruct with GPT-3.5 to
generate data, instruction pairs. Inspired by Gorilla Patil et al. (2023), We
provide three demos with in-context learning and task the model with
generating instructions that describe the format of the given structure. We
specifically instruct the model to use natural language. We have structured 6
demos for each of the three data formats, all of which are hand-written or
modified data.

###  3.2 Finetuning LLaMA-7B

Here we propose a structure-aware instruction tuning method to bolster the
capability of LLMs in generating structured text. We employ the standard
instruction tuning method to fine-tune LLaMA-7B (Touvron et al., 2023). Our
ultimate goal is to enable LLaMA to comprehend the task at hand and deliver
the output in a conversational mode. This is akin to engaging in a dialogue
with the user, culminating in the successful completion of our defined task.
The entire pipeline can be found in Figure 1.

###  3.3 Evaluation Metrics

Evaluating the similarity of generated tables to the ground-truth tables is
non-trivial: for instance, the same table can be formatted in many different
ways in HTML or LaTeX. Hence, our evaluation metric should ideally capture
meaningful differences in the data presented, while being invariant to
insignificant differences in formatting.

We propose to break down the similarity of two tables into two coarse
components: content and structure. In scoring content similarity, we attempt
to parse content out the data within the table cells, and compute the
similarity. This similarity is computed between the generated and ground-truth
table cells by commonly used similarity metrics. In scoring structure
similarity, we place higher emphasis on components such as the number of
columns and rows, cell alignment, and the table caption. Both similarity
scores do overlap (e.g. a table with the wrong number of rows/columns would
likely score poorly on content), but we find that these two scoring categories
allow us to perform more involved analysis on where predicted and ground-truth
tables differ.

####  3.3.1 GPTscore

We further take two approaches to score each metric. First, we perform model-
based evaluation, querying GPT-3.5 with both tables and having it score the
similarity of content and structure separately. Following Wang et al. (2023),
we prompt the model to perform Chain-of-Thought Wei et al. (2023) reasoning
before outputting its scores, and we query the model with the predicted and
ground-truth tables in both orders and average the scores. We report these as
the GPTscore. The prompt of GPTscore can be found in Appendix D.

####  3.3.2 H-Score

In addition to model-based evaluation, we also implement hand-crafted scoring
functions to score the similarity of the tables. Because of the many ways, the
tables can be presented in the different data formats, we implement several
heuristics to normalize the tables and to compute their similarity. The
specific implementation of scoring functions for different formats can be
found in Appendix D. Where similarities between strings or data structures are
computed, we use an average of Levenshtein distance and the Ratcliff/Obershelp
similarity metric. We report these heuristically normalized metrics as the
H-Score.

##  4 Experiments

###  4.1 Basic Settings

For metrics, we use SacreBLEU, ROUGE-L, BERTScore, BARTScore and BLEURT
metrics as they are all classical metrics to evaluate text similarity, which
is also useful in this task. Besides, we use our two proposed metrics: GPT
score and H-score. We evaluate the following models: GPT-NeoX-20B, GPT-3.5,
GPT-4, Vicuna-13B, our structure-aware finetuning LLaMa-7B and original
LLaMa-7B. GPT-NeoX-20B, GPT-3.5 and GPT-4 represent the state-of-art
performance of current LLMs and Vicuna-13B is another version finetuned by
LLaMa, which can reach 90% of the capacity of ChatGPT. We think these models
are strong enough to be persuasive. For the first 4 models, we simply call
their APIs from OpenAI or HuggingFace to generate results without further
finetuning. In our dataset, each item consists of three parts: instruction,
input, and output. When generating results, we put each item’s instruction and
input together as the final input to models.

During the inference process, we will provide the model with a natural
language prompt to describe the form and content of our task, as well as the
expected response (e.g., “please generate a table given by the following
information and format”).

###  4.2 Results

Table 2 provides a comparative analysis of different language models based on
several performance metrics. For ‘Tables from Raw Text’, the Ours-7B
outperforms the other models in every metric. Interestingly, without fine-
tuning, the performance drops significantly, particularly in SacreBLEU,
ROUGE-L, and BERTScore. The results for ‘LaTeX’ reveal a similar trend where
we again achieve the best results across all metrics, except for the BLEURT
metric, where GPT-4 takes the lead. In the ‘HTML’ category, GPT-4 scores the
highest in SacreBLEU and BERTScore. However, ours comes out on top for the
rest of the metrics.

Considering the inconsistency observed by different metrics, we also conducted
a human evaluation. We also carried out a human evaluation on 100 examples
using MTurk. Evaluators rated each example on a scale from 0 to 10, assessing
both format consistency and content consistency. Although we cannot enumerate
the details due to space constraints, we discovered that the Content GPTscore
and Content H-Score are more closely aligned with existing metrics. However,
our proposed Format GPTscore and Format H-Score significantly surpass other
metrics, particularly in terms of instance-level Spearman correlation for
format accuracy. These human evaluations underscore the efficacy of our
proposed metrics. However, larger-scale human evaluations are needed to
further explore and substantiate these findings.

Moreover, we delve into an in-depth analysis, attributing observed
shortcomings to several error types, spanning two key dimensions: Content
Selection and Format Planning, as well as the Reasoning Process, see details
in Appendix G. Based on these, we present an ability map of model capabilities
from six dimensions.

Figure 3: Visualization of LLM capability with human evaluation over Struc-
Bench.

##  5 Conclusion

In conclusion, this research offers a comprehensive exploration of the
structured text generation limitations inherent in Large Language Models
(LLMs) like ChatGPT and GPT-4. Through developing a benchmark specifically
designed for structured text generation and integrating a wide range of
datasets, we have been able to thoroughly assess the capabilities of prevalent
LLMs. Our analysis has identified several areas of concern, particularly in
regard to content accuracy, formatting, numerical reasoning, and the handling
of long tables.

##  6 Limitations

Although we present an in-depth and comprehensive analysis, the exploration of
LLMs in structured text generation presented in this paper has several
limitations:

##### Domain-Specific Benchmark Development

While we’ve made strides in constructing benchmarks for structured text
generation, it may be beneficial to develop benchmarks that cater to specific
domains. Different fields might have unique structural requirements and
understanding these nuances can significantly improve the models’
applicability across diverse contexts.

##### Expand the Range of Datasets

There are endless data types and sources that can be explored. Incorporating a
broader variety of datasets could expose the models to an even wider range of
structural formats, ultimately enhancing their overall performance.

##### Enhancing Numerical Reasoning Capabilities

Our study identified inadequate numerical reasoning as one of the challenges
faced by LLMs. Investigating techniques to bolster numerical reasoning in
these models could lead to significant improvements in their performance.

##### Developing Advanced Methods

While our structure-aware instruction tuning method showed promising results,
more sophisticated techniques could be developed. For instance, future work
could explore ways of incorporating more explicit structural information into
the model or developing methods that allow the model to learn structural
patterns more effectively.

##### Exploring Multimodal LLMs

As LLMs continue to evolve, there are opportunities to explore multimodal
models that can process and generate both text and other forms of data, such
as sound or images Kamigaito et al. (2023), in a structured manner.

## References

  * Bao et al. (2018) Junwei Bao, Duyu Tang, Nan Duan, Zhao Yan, Yuanhua Lv, Ming Zhou, and Tiejun Zhao. 2018.  Table-to-text: Describing table region with natural language.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32.
  * Black et al. (2022) Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022.  Gpt-neox-20b: An open-source autoregressive language model.  _arXiv preprint arXiv:2204.06745_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org (accessed 14 April 2023)_.
  * He et al. (2023) Yuxin He, Jingyue Hu, and Buzhou Tang. 2023.  Revisiting event argument extraction: Can eae models learn better when being aware of event co-occurrences?  _arXiv preprint arXiv:2306.00502_.
  * Kamigaito et al. (2023) Hidetaka Kamigaito, Katsuhiko Hayashi, and Taro Watanabe. 2023.  Table and image generation for investigating knowledge of entities in pre-trained vision and language models.  _arXiv preprint arXiv:2306.02115_.
  * Lebret et al. (2016) Rémi Lebret, David Grangier, and Michael Auli. 2016.  Neural text generation from structured data with application to the biography domain.  _arXiv preprint arXiv:1603.07771_.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Li et al. (2023) Tong Li, Zhihao Wang, Liangying Shao, Xuling Zheng, Xiaoli Wang, and Jinsong Su. 2023.  A sequence-to-sequence&set model for text-to-table generation.  _arXiv preprint arXiv:2306.00137_.
  * Ma et al. (2023) Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. 2023.  Large language model is not a good few-shot information extractor, but a good reranker for hard samples!  _arXiv preprint arXiv:2303.08559_.
  * Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022.  Crosslingual generalization through multitask finetuning.  _arXiv preprint arXiv:2211.01786_.
  * Novikova et al. (2017) Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017.  The e2e dataset: New challenges for end-to-end generation.  _arXiv preprint arXiv:1706.09254_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Patil et al. (2023) Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023.  Gorilla: Large language model connected with massive apis.  _arXiv preprint arXiv:2305.15334_.
  * Pietruszka et al. (2022) Michał Pietruszka, Michał Turski, Łukasz Borchmann, Tomasz Dwojak, Gabriela Pałka, Karolina Szyndler, Dawid Jurkiewicz, and Łukasz Garncarek. 2022.  Stable: Table generation framework for encoder-decoder models.  _arXiv preprint arXiv:2206.04045_.
  * Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023.  Is chatgpt a general-purpose natural language processing task solver?  _arXiv preprint arXiv:2302.06476_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Rossiello et al. (2022) Gaetano Rossiello, Faisal Chowdhury, Nandana Mihindukulasooriya, Owen Cornec, and Alfio Gliozzo. 2022.  Knowgl: Knowledge generation and linking from text.  _arXiv preprint arXiv:2210.13952_.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.
  * Wang et al. (2023) Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023.  Large language models are not fair evaluators.
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023.  Chain-of-thought prompting elicits reasoning in large language models.
  * Whitehouse et al. (2023) Chenxi Whitehouse, Clara Vania, Alham Fikri Aji, Christos Christodoulopoulos, and Andrea Pierleoni. 2023.  Webie: Faithful and robust information extraction on the web.  _arXiv preprint arXiv:2305.14293_.
  * Wiseman et al. (2017) Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017.  Challenges in data-to-document generation.  _arXiv preprint arXiv:1707.08052_.
  * Wu et al. (2022) Xueqing Wu, Jiacheng Zhang, and Hang Li. 2022.  Text-to-table: A new way of information extraction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2518–2533, Dublin, Ireland. Association for Computational Linguistics.
  * Zhao et al. (2023a) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023a.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.
  * Zhao et al. (2023b) Yilun Zhao, Haowei Zhang, Shengyun Si, Linyong Nan, Xiangru Tang, and Arman Cohan. 2023b.  Large language models are effective table-to-text generators, evaluators, and feedback providers.  _arXiv preprint arXiv:2305.14987_.
  * Zhao et al. (2023c) Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir Radev. 2023c.  Robut: A systematic study of table qa robustness against human-annotated adversarial perturbations.  _arXiv preprint arXiv:2306.14321_.
  * Zhong and Chen (2020) Zexuan Zhong and Danqi Chen. 2020.  A frustratingly easy approach for entity and relation extraction.  _arXiv preprint arXiv:2010.12812_.

##  Appendix A Analysis with Examples

###  A.1 Example Table A

The main difference between the reference tables and the tables generated by
GPT-3.5 and GPT4 is in the completeness and precision of the data provided.

In the reference tables, all relevant data is fully represented: For the teams
(Table 1), each team has a precise number or percentage for every statistic.
Similarly, for the players (Table 2), each player has a definite number for
every statistic, including minutes played in the format “mm:ss”.

Figure 4: Using GPT-3.5 and GPT-4 to generate a table based on the input text,
the generated results contain a large number of errors, including format
errors and content errors.

In contrast, the generated tables show data that is incomplete and imprecise.
For GPT-3.5 generated one, the team statistics table has some statistics
missing, as represented by empty cells, and some are not presented as
percentages. The player statistics table also has missing data in a similar
fashion, and it lacks the "minutes played" statistics entirely. For instance,
in the ’team’ table, the "Percentage of field goals" column for the Suns is
missing. Similarly, in the ‘player’ table, many key statistics such as
"3-pointers attempted", "3-pointers made", "Field goals attempted", "Field
goals made", and "Minutes played" are missing for various players. Regarding
the format, we observe a lot of format errors. For example, the ‘Percentage of
field goals’ column for Grizzlies is represented as "50" instead of "50.0%".
Moreover, the ‘Wins’ column for the Suns is represented as "3" instead of "0".
This misrepresentation can lead to significant misunderstanding of the data.
The ‘Player’ table also has format errors. For instance, the ‘Minutes played’
column is missing the time format (i.e., “00:00”). On the other hand, the
reference tables adhere to a standard format. Percentage data is represented
with a ‘%’ sign, time data uses the ‘00:00’ format, and numeric data correctly
represents each statistic.

###  A.2 Error Type

##### Structure Errors:

These errors pertain to the structural integrity of the generated tables.
Specifically, they include instances where there are excess or missing rows or
columns in comparison to the correct table structure.

##### Structure Naming Errors:

This category captures errors related to the naming conventions used for rows
or columns. Any discrepancies in a row or column names between the generated
and correct table are flagged as structure naming errors.

##### Element Errors:

These are inaccuracies observed at the element level within the generated
table. Element errors encompass incorrect numbers, values, or inappropriately
empty cells, reflecting discrepancies in individual table entries relative to
the correct table.

##  Appendix B Rationale for Selecting the RotoWire Dataset

Traditional data-to-text datasets include Rotowire Wiseman et al. (2017), E2E
(Novikova et al., 2017), WikiTableText (Bao et al., 2018), and WikiBio (Lebret
et al., 2016). Given that only the RotoWire dataset contains tables with more
than 2 columns, we specifically opted to utilize this dataset. Furthermore, to
maintain a certain level of complexity in our study, we filtered out tables
with dimensions smaller than 3x3 in Rotowire.

Dataset | Train | Valid | Test | # of tokens | # of rows | # of columns
---|---|---|---|---|---|---
E2E | 42.1k | 4.7k | 4.7k | 24.90 | 4.58 | 2.00
WikiTableText | 10.0k | 1.3k | 2.0k | 19.59 | 4.26 | 2.00
WikiBio | 582.7k | 72.8k | 72.7k | 122.30 | 4.20 | 2.00
Table 3: Statistics of E2E, WikiTableText, and WikiBio datasets, including the
number of instances in training, validation, and test sets, number of BPE
tokens per instance, and number of rows per instance.

##  Appendix C MTurk

About the qualifications of Amazon Mechanical Turk (MTurk) workers, we use the
following qualifications to recruit in total of 10 MTurk workers with good
track records: HIT approval rate greater than or equal to 98%, number of HITs
approved greater than or equal to 500, and located in one of the following
English native-speaking countries: Australia, Canada, New Zealand, United
Kingdom, United States. Each annotator is limited to annotating 10 examples,
including both the output of GPT-3.5 and GPT-4.

Annotators workers were compensated $7, calibrated to equal a $42/hour pay
rate. We first annotated examples in-house to determine the required
annotation speed. A summary block usually takes around 10 minutes.

To demonstrate our annotation template and facilitate future research, we show
the interface for annotations.

Figure 5: Interface of Mturk. Figure 6: Interface of Mturk.

##  Appendix D Scoring

###  D.1 GPTscore

Here we display our prompt for GPTscore. First, tell GPT how to score the
"content similarity" and "structural similarity" and then tell GPT how to
output the answer:

“Based on the above, we wanted to determine if the above tables are similar.
Ideally, they should have identical content and structure. Score the "content
similarity" and "structural similarity" between 0 and 10.

\- Content similarity: 10 if the contents of the table cells are identical, 0
if they are entirely different. If about 50% of the cells have the same data,
the score should be 5.

\- Structural similarity: 10 if the tables have the same structure (e.g. same
column and rows with identical ordering, same alignment, etc.) although text
formatting differences can be ignored (e.g. colors, font).

Output a JSON object such as the following:

"""json

{{

"content_similarity": …

"structural_similarity": …

}}

"""

Think carefully, and then output the scores.”

###  D.2 H-Score

##### LaTeX

We use the pylatexenc library to parse a given LaTeX table, and walk through
the parse-tree structure in the tabular environment to identify the table
“cells”. We score the content similarity based on strings within the cells,
and score structural similarity based on having the matching number of rows
and columns, the same caption, and the same cell alignment.

##### HTML

We use the beautifulsoup4 library to parse a given LaTeX HTML snippet and walk
through the parse-tree structure in <table>, <ul> or <ol> tags to identify
data cells. We separately build a tree of white-listed HTML tags to score the
structural similarity, traversing an HTML document tree structure,
disregarding the actual content within the tags and simplifying it by focusing
only on specific HTML tags (defined in RECOGNIZED_HTML_TAGS). We score the
content similarity based on strings within the cells and score structural
similarity based on the similarity of the structure tree and the total number
of cells matching.

White-listed HTML tags:

⬇

RECOGNIZED_HTML_TAGS = [

"table", "tr", "th", "td",

"ul", "ol", "li",

"div", "span", "p",

"a", "img", "embed", "pre",

"h1", "h2", "h3", "h4", "h5", "h6",

"input", "button",

]

##### Raw Text Tables

In our evaluated dataset, each example consists of two tables (Team and
Player). We do a string search for "Team" and "Player" headers to identify the
two tables. We then parse the tables according to Markdown formatting, with
newlines and pipes as row and column dividers respectively, to identify the
table cells. We score the content similarity based on strings within the
cells, and score structural similarity based on the similarity of column names
and the number of rows and columns matching.

##### String Similarity Measurement:

Our script includes methods to calculate the similarity between two strings.
These methods can be used to compare the structure or content of HTML, latex
documents or any other pair of strings. The similarity is evaluated using
well-established algorithms in text analysis: the Levenshtein distance and the
SequenceMatcher from Python’s difflib module.

##  Appendix E Prompt for Description Generation and Inference

##### Raw Text Table Description Prompt

Traditional data-to-text datasets only have raw text for each table. However,
it is not enough for Chatgpt or other LLMs to generate correct tables. As a
result, we added some format descriptions to help them generate the correct
tables. We use GPT-3.5 to achieve this. We want to get detailed format
information without concrete contents in cells, so we explicitly include these
requirements in the prompt. Here is our prompt: Describe details about the
given text. First, give the number of tables, and then for each table,
describe its format such as the number of columns and rows, column names, and
row names.

##### HTML Table Description Prompt

Unlike data-to-text datasets, HTML datasets only have final outputs, so we are
required to generate a detailed description of their format and content. For
content descriptions, we can simply ask GPT-3.5 to output raw text without
HTML tags. For format descriptions, however, we need to ask GPT-3.5 to
describe each tag, otherwise, it will leave out some tags and describe the
table in general rather than detailed information. Moreover, it is necessary
to ask it to use specific numbers instead of ’several’ or ’multiple’. Here is
our prompt for HTML format descriptions: Describe the format of this HTML in
detail according to each HTML tag of the following HTML code. Be careful and
make sure don’t miss any HTML tags. Please use more than 300 words to explain
the format. Use specific numbers rather than being vague about several.

##### Latex Table Description Prompt

Similar to HTML prompt generation, it is necessary to ask GPT-3.5 to generate
both format descriptions and content descriptions as latex datasets only have
final outputs. For content descriptions, we can simply ask GPT-3.5 to describe
the given latex table as detailed as it can and include all cells. For format
description, since the latex format is too complex, we need to give it a small
example to learn. Then we ask GPT-3.5 to describe the detailed format of a
given latex table, including specific questions to help it generate format
descriptions. Here is our prompt for latex format descriptions: Describe the
detailed format of a given latex table according to the commands and tags with
more than 500 words. Include: Whether there is table border lines? How is text
alignment? What are table attributes? Whether to bold? Whether to add \ref?
Please clearly explain whether there are horizontal and vertical lines
bordering each row and column. Say anything about a special "\" format token
in latex if there is. Don’t display latex code directly. Use natural language.
And provide enough format information for me to recreate this table based on
your output description.

##### Prompt for Inference

When inferencing raw text tables, LLMs tend to output tabular results rather
than raw text tables. As a result, we need to give it an example output first,
then tell the model that the input consists of two parts, text and format
descriptions, and ask the model to generate the output based on them. For HTML
and Latex inference, we can simply ask models to infer from the input and
specify the format and content sections in the input, since models can
generate correct syntax.

##  Appendix F Prompt for GPT scores

We prompt the model to perform Chain-of-Thought reasoning before outputting
its scores, and we query the model with the predicted and ground truth tables
in both orders and average the scores. Here we use the GPT scores prompt for
raw text tables as an example:

We want to evaluate how similar the following tables/data structures are.

Table 1:

“‘ input1 “‘

Table 2:

“‘ input2 “‘

Based on the above, we wanted to determine if the above tables are similar.
Ideally, they should have identical content and structure. Score the "content
similarity" and "structural similarity" between 0 and 10.

\- Content similarity: 10 if the contents of the table cells are identical, 0
if they are entirely different. If about 50% of the cells have the same data,
the score should be 5.

\- Structural similarity: 10 if the tables have the same structure (e.g. same
column and rows with identical ordering, same alignment, etc) although text
formatting differences can be ignored (e.g. colors, font).

Output a JSON object such as the following:

“‘json "content_similarity": … "structural_similarity": … “‘

Think carefully, and then output the scores.

##  Appendix G Ability Map

Based on our automated evaluation, we selected Vicuna, ChatGPT, GPT-4, and
Ours as representative models and conducted an in-depth analysis of the causes
of model errors.

We identified content accuracy, formatting, numerical reasoning, and handling
of long tables as the main sources of these errors.

At the fundamental level, we decompose the process of model-generated complex
structured outputs into two parts: Content Selection and Format Planning.
Initially, the model needs to identify key information from a given vast
amount of unstructured input, extract this information, understand it, and
organize it. Subsequently, it needs to plan how to summarize these extracted
details, devise the format of the table to be generated, and then fill in the
information.

Accordingly, we can break down the model’s capabilities into Coverage,
Formatting Reasoning, Comprehension, Pragmatics, and Hallucination Control.

Coverage entails the model’s ability to accurately cover the content in the
input. Formatting Reasoning pertains to judgment about the output format,
assessing if the model can find the most appropriate and reasonable structured
format.

Comprehension reflects whether the model can understand the content of the
input, as there are times when it is necessary to infer from a large amount of
data (including performing addition or subtraction or comparing multiple
elements).

Pragmatics involves the ability to utilize special formats, such as HTML tags
and specific syntax in LaTeX.

Finally, Hallucination Control signifies the model’s ability to refrain from
generating content not present in the input.

We carried out manual annotations and obtained visualized results to
demonstrate these aspects.