# Incremental Treatments of the Full Configuration Interaction Problem

Janus J. Eriksen<EMAIL_ADDRESS>School of Chemistry, University
of Bristol, Cantock’s Close, Bristol BS8 1TS, United Kingdom  Jürgen Gauss
<EMAIL_ADDRESS>Department Chemie, Johannes Gutenberg-Universität Mainz,
Duesbergweg 10-14, 55128 Mainz, Germany

###### Abstract

The recent many-body expanded full configuration interaction (MBE-FCI) method
is reviewed by critically assessing its advantages and drawbacks in the
context of contemporary near-exact electronic structure theory. Besides
providing a succinct summary of the history of MBE-FCI to date within a
generalized and unified theoretical setting, its finer algorithmic details are
discussed alongside our optimized computational implementation of the theory.
A selected few of the most recent applications of MBE-FCI are revisited,
before we close by outlining its future research directions as well as its
place among modern near-exact wave function-based methods.

##  1 Introduction

In a standard basis of Slater determinants, the exact solution to the time-
independent electronic Schrödinger equation in a finite one-electron basis
will correspond to a linear, weighted sum over all possible electron
configurations. This particular wave function Ansatz, known conventionally as
full configuration interaction 1, 2, 3, 4, 5, 6 (FCI), has traditionally held
a prominent and pivotal role in the field of quantum chemistry as the ultimate
benchmark against which to compare and calibrate most of electronic structure
theory. Crucial, though, is the fact that its formal exactness necessarily
comes at the price of factorial computational complexity due to the
combinatorial exercise associated with distributing $N$ electrons among $M$
molecular orbitals (MOs). In extended (or even non-minimal) basis sets,
however, the overwhelming majority of the FCI wave function will be vastly
redundant 7, 8. This empirical observation has thus served to drive a number
of so-called selected CI (SCI) approximations over the years, which all seek
to lift any of such configurational sparsity by focussing on energetically
important determinants only 9. Depending on which particular measure of
importance is being used, as well as whether the selection of determinants
gets based on deterministic, stochastic, or even regressive protocols, the
manifold of resulting methods will ultimately come to differ from one another
in their subtleties and efficacies. For a comprehensive account of such modern
takes on FCI, the reader is referred to a recent perspective on the matter by
one of us 10, which covers early, present, and near-future state-of-the-art
approaches in greater detail than we will possibly be able to afford in the
present review.

As a radically different approach to FCI, we will herein be concerned with its
realizations by means of many-body expansions (MBEs). On that note, we will
first and foremost discuss so-called MBE-FCI theory alongside its various
incarnations. The initial work on modern MBE-FCI appeared in the year 2017 11
and its generalized methods have been in active development by us ever since
12, 13, 14, 15. Coincidentally, yet entirely unrelated, Zimmerman proposed a
comparable theory around the same time, known as incremental FCI (iFCI) 16,
which too continues to be extended to this date 17, 18, 19. As part of the
present work, we will reiterate what common traits and discrepancies exist in-
between these two theoretical frames. At an overall level, though, both MBE-
FCI and iFCI constitute incremental approximations to FCI and thus
fundamentally draw on Nesbet’s original work on so-called Bethe-Goldstone
theory from the 1960s 20, 21, 22. However, generalized MBE-FCI may
conveniently be viewed as the unifying umbrella-like setting, within which all
of the discussed orbital-based incremental methods can be derived and
expressed. The methods themselves are intended as pragmatic ways of
circumventing the prohibitively exponential scaling wall of FCI by simulating
properties without recourse to the exact, $N$-dimensional wave function It is
worth noting that Stoll have also touched upon the idea of using orbital-based
MBEs for calculating correlation energies of solid-state systems, see, e.g.,
Refs. 24, 25, 26. Conventionally, ground-state energies have been the desired
property, but recent work has illustrated how MBEs may also facilitate the
incremental calculation of other properties, not only for ground states, but
rather for any state of arbitrary spin. Given this versatility of incremental
approaches to FCI, it is our belief that methods based on orbital-based MBEs
will continue to make a lasting impact in the electronic structure community,
despite the fact that they are possibly more costly than their SCI
counterparts. We will close our review by conveying a number of outstanding
challenges in MBE-FCI as well as possible near-future solutions.

##  2 Theory

Provided with a complete set of MOs, e.g., resulting from a preceding mean-
field calculation—such as, a restricted Hartree-Fock (RHF) or a complete
active space self-consistent field 27, 28 (CASSCF) treatment—an MBE-FCI
calculation commences by deciding upon a division of these orbitals into two
distinct and non-overlapping spaces, namely, a reference and an expansion
space. The former of these, which, in turn, determines the latter in full, may
be chosen to have any composition; if set to span the entirety of the space of
virtual orbitals, MBE-FCI will coincide with the Nesbet’s original theory
and—subject to further truncation approximations of the reference space—also
the iFCI method, while if the reference space holds all occupied orbitals of a
given system, the method will be that of our initial work on virtual orbital
MBEs 11. However, for any choice of reference space different from these
limiting cases, the basic theory is entirely generalizable to all conceivable
correlation domains.

An MBE-FCI calculation now proceeds by performing an orbital-based MBE in the
resulting expansion space, which will serve to recover the residual
correlation missing from performing an initial complete active space
configuration interaction (CASCI) calculation in the reference space alone.
Denoting this (truncated FCI) reference correlation energy as
$E_{\text{ref}}$, an MBE-FCI decomposition of the FCI correlation energy will
formally read as

$\displaystyle E_{\text{FCI}}$
$\displaystyle=E_{\text{ref}}+\sum_{p}\epsilon_{p}+\sum_{p<q}\Delta\epsilon_{pq}+\sum_{p<q<r}\Delta\epsilon_{pqr}+\ldots$
$\displaystyle\equiv
E_{\text{ref}}+E^{(1)}+E^{(2)}+E^{(3)}+\ldots+E^{(M_{\text{exp}})}$ (1)

where the MOs of the expansion space (of size $M_{\text{exp}}$) of unspecified
occupancy are labelled by indices $\\{p,q,r,s,\ldots\\}$, and $\epsilon_{p}$
designates the correlation energy of a CASCI calculation in the composite
space of orbital $p$ and all of the MOs of the assigned reference space, with
the reference correlation energy, $E_{\text{ref}}$, subtracted. At an
arbitrary order $k$, the orbital increments, $\Delta\epsilon_{[\Omega]_{k}}$,
are recursively defined for a general tuple of $k$ MOs, $[\Omega]_{k}$, via
the following relation

$\displaystyle\Delta\epsilon_{[\Omega]_{k}}=\epsilon_{[\Omega]_{k}}-\big{(}\sum_{p\in
S_{1}[\Omega]_{k}}\epsilon_{p}+\sum_{pq\in
S_{2}[\Omega]_{k}}\Delta\epsilon_{pq}+\ldots+\sum_{pqrs\cdots\in
S_{k-1}[\Omega]_{k}}\Delta\epsilon_{pqrs\cdots}\big{)}\ .$ (2)

In Eq. 2, the action of the operator $S_{l}$ onto $[\Omega]_{k}$ is to yield
all possible unique subtuples of order $l$ ($1\leq l<k$), and
$\epsilon_{[\Omega]_{k}}$ is defined on par with $\epsilon_{p}$ above in Eq.
1.

The most general treatment of electron correlation is now achieved by choosing
upon an empty reference space, in which case the total expansion space will
span all of the MOs of the system at hand. As for when the reference space
encompasses all virtual or occupied MOs, $E_{\text{ref}}$ vanishes in this
case, but the expansion in Eq. 1 will now commence at second rather than first
order with all possible unique pairs of correlating occupied and virtual
orbitals The reason for the absence of first-order contributions in the case
of an empty reference space is simply that a CASCI calculation in the space of
either a single occupied or virtual orbital will miss a corresponding
correlation space to excite electrons into or miss any electrons to excite,
respectively.. However, despite the fact that MBE-FCI will be free of any
constraints in this case, it will importantly not represent a true tabula rasa
approach to FCI as an implicit bias still exists in the choice of MO basis. To
that end, Stoll has recently presented work on the optimization of localized
orbitals at a partially correlated level, rather than for the traditional
uncorrelated HF starting point 30. In our experience hitherto, spatially
localized orbitals often represent an excellent choice of MO basis within the
scope of MBE-FCI as they allow for more compressed expansions (fewer
significant contributions altogether), in turn, leading to faster convergence
profiles. Another typical choice is to use a set of orbitals tailored to a
specific correlated method 31, 32. We have previously proposed the use of
these MOs in combination with a so-called base model in the expansion 12. In
this case, rather than the total correlation energy, one expands the gap
between FCI and an intermediate correlated model, for which a preliminary
calculation on the whole system is feasible, in part, also to compute the one-
electron reduced density matrix needed to derive its natural orbitals (NOs).
As the quantity that need be recovered by the MBE will be significantly
reduced, the use of base models can lead to faster convergences. However,
additional constraints are tied into the MBE in this case as the base model is
fundamentally required to yield a reasonable approximation to FCI for it to
accelerate the underlying MBE-FCI algorithm. In cases where this is not true,
e.g., in the presence of static and strong correlation, typical choices of
base models, e.g., the acclaimed CCSD(T) method of coupled cluster theory 33,
will generally be limited in their performance as they are themselves based on
a single determinant.

Now, in the case where a given system is indeed dominated by a single
reference determinant (say, the HF determinant), this determinant will have
the largest weight in the linear FCI expansion of the wave function and the
system is said to be dominated by dynamical correlation alone. However, in
cases where more than one determinant have significant weights, one will need
to be able to describe the important (and common) constituents of these from
the beginning and they will then need be included in the reference space. More
generally, capturing the integral part of the total electron correlation in
the reference space will lead to a faster convergence of the expansion. This
observation will hold true even when the HF determinant dominates the FCI wave
function, and reducing the size of the expansion space will additionally lead
to fewer possible orbital tuples throughout the MBE. For linear molecules, the
$\pi$-pruning technique of Refs. 13 and 15 has further been introduced in
order to deal with the $D_{\infty\text{h}}$ and $C_{\infty\text{v}}$ point
groups within their $D_{2\text{h}}$ and $C_{2\text{v}}$ subgroups,
respectively. Essentially, $\pi$-pruning is a sort of prescreening filter that
works to prune away all increment calculations that fail to simultaneously
include the $x$\- and $y$-components of a given pair of degenerate
$\pi$-orbitals. The use of $\pi$-pruning generally results in much shorter
(faster) expansions for molecules belonging to non-Abelian point groups, while
at the same time warranting convergence onto states spanned by the correct
irreducible representation. While $\pi$-pruning is a specific filter designed
for the treatment of a specific type of systems, it also serves as an example
of how it is generally possible to add alternative filters in a top-down
manner in order to accelerate or assure convergence onto a target FCI
property.

##  3 Implementation

As a platform for our theoretical work on MBE-FCI over the past couple of
years, we have developed our own Python-based, open-source PyMBE code 34. All
electronic structure kernels within PyMBE are formulated upon the PySCF
program 35, 36, and the MPI4Python module handles parallel communication over
the message passing interface (MPI) standard 37, 38, 39.

Ever since its conception, the PyMBE code has seen several rounds of heavy
optimizations. In particular, the memory handling and footprint of the
involved 1- and 2-electron integrals as well as all involved intermediates and
results have been overhauled. The recursive nature of Eq. 2 implies that we
need to be able to look up a vast number of subset contributions when
calculating a given increment. In PyMBE, this is achieved by representing
every tuple of orbitals by its hash and using binary searches in finding all
subset occurrences. However, this setup implies that a sorted array of hashes
must be stored in addition to the actual increments which they correspond to,
adding to the memory requirements in the code. For that reason, a hybrid
MPI+MPI approach to shared-memory allocation and access has been implemented
throughout our code, in which the MPI$\\_$Win$\\_$allocate$\\_$shared function
is used as a departure from the standard abstract and distributed memory model
of MPI 40. As previously detailed in Ref. 14, the underlying memory
organisation on a given computer node gets exposed to MPI, allowing us, in
turn, to bypass the expensive and convoluted MPI-3 one-sided operations by
instead using shared memory directly between the processes on the node. The
effect of this model is that MPI is employed in markedly different manners
across and within individual nodes. For an MBE-FCI calculation on $A$ number
of nodes—each equipped with $B$ cores—our parallel model comprises a single
global master and $AB-1$ global slaves. Among these, $A-1$ slaves are
additionally promoted to local masters that pass messages onto the global
master via traditional distributed MPI, while each sharing a window to the
physical memory address space on their respective node with their own group of
$B-1$ local slaves. The latter communication proceeds over dedicated
communicators.

At any given order in the MBE, all possible orbital tuples are being yielded
by a generator function, which takes into account the composition of the
reference space. In case this is empty, only tuples that make reference to
both occupied and virtual orbitals are allowed, as touched upon in Section 2.
The task scheduling has been implemented in a round-robin fashion and the
input generator has been designed in such a way that those tuples that
correspond to large (determinant-wise) CASCI calculations will precede the
smaller ones. Once a given process has been assigned a tuple (i.e., a unique
work task), it proceeds by computing the specific core and active space
indices needed for the CASCI calculation. Next, the corresponding 1- and
2-electron integrals are extracted alongside the core energy. In PyMBE, these
are stored in the transformed MO basis in shared memory with the electron
repulsion integrals compressed into a matrix form with 4-fold symmetry.
Finally, the CASCI energy is calculated, before the increment is computed and
stored alongside the corresponding hash. If enabled, a full suite of restart
functionalities guarantee that MBE-FCI is trivially protected against hardware
failures, strict time limits, etc., at only a minimum of associated penalty.

The most recent version of our screening protocol has been implemented such
that MOs get gradually screened away from the full expansion space according
to the absolute magnitude of the tuples which they take part in. Specifically,
a certain percentage of the MOs that contribute the least may get removed
(governed by a dedicated input parameter). As shrinking the expansion space is
accompanied by a corresponding reduction in the number of increment
calculations at the orders to follow, only those specific MOs of the expansion
space (at any given order) that give rise to the numerically largest
increments will be retained among the tuples at the following order. In the
case one or more orbitals get screened away, the code thus enters a recently
implemented purging module, which is designed to retain only those
contributions at lower orders that are needed going forward. The rationale
behind this step in an MBE-FCI calculation is that all subsequent tuples will
not make reference to the screened orbitals and their increments at lower
orders are hence not required anymore. This purging procedure generally works
to lower our memory requirements significantly.

Figure 1: Internode strong scaling of frozen-core MBE-FCI calculations on H2O
13.

The parallel scaling potential of MBE-FCI was assessed in Ref. 13. Being
compute- rather than memory-bound, the resource utilization of the theory and
its implementation within the PyMBE code is best measured in terms of its
strong scalability. In Figure 1, the relative speed-up gained by moving from a
single node (Intel Xeon E$5$–$2680$ $\text{v}3$ hardware with $24$ cores @
$2.5$ GHz and $128$ GB of global memory), on which MPI is employed across all
of its 24 cores, to a total of 512/1,024 nodes (i.e., 12,288/24,576 individual
MPI processes) is shown. The scalability in Figure 1 has been assessed for
MBE-FCI calculations on the standard H2O molecule in medium-sized (cc-pVTZ)
and extended (cc-pV5Z) basis sets 41. At scale, the efficiencies at 512
(12,288 cores) and 1,024 nodes (24,576 cores) amount to $79\%$ and $91\%$ for
the expansions in the cc-pVTZ and cc-pV5Z basis sets, respectively, and the
difference in performance between the two basis sets can be ascribed to the
significantly larger number of individual CASCI calculations in the latter of
the two expansions. MBE-FCI is thus seen to offer a highly scalable treatment
of the electron correlation problem with a massive parallelism that is ideally
suitable for modern distributed supercomputers.

##  4 Applications

Figure 2: MBE-FCI convergence for the frozen-core C6H6/cc-pVDZ problem with
localized MOs and an ($6e$,$6o$) reference space 42. The dashed line indicates
$\Delta E=-863$ m$E_{\text{H}}$. (Reproduced with permission from Ref. 42.
Copyright 2020, American Chemical Society)

Having covered its theoretical basis, we will now review a selected few of the
molecular systems for which MBE-FCI has been applied to date. Most recently,
MBE-FCI took part in a broad, international blind challenge devoted to
computing the best possible estimate of the FCI correlation energy for the
ubiquitous benzene system in a standard correlation consistent double-$\zeta$
basis set 42. The repertoire of high-accuracy methods that entered the
challenge was collectively chosen to constitute a truly diverse view of
leading, contemporary approaches from all around the world, and, as such, the
work was not just testament to what is currently achievable by means of near-
exact quantum chemistry, but also to what the future beholds. In the specific
case of benzene, and with respect to how to optimally apply MBE-FCI to the
problem of estimating its correlation energy, it is imperative to initially
note that one does not necessarily need to include any orbitals in the
reference space at all, as was previously demonstrated in Ref. 14. However,
doing so may potentially lead to faster convergence. For this reason, the MBE-
FCI entry in Ref. 42 was computed using a reference space that included the
six frontier bonding and anti-bonding valence $\pi$-orbitals in a localized
Pipek-Mezey (PM) basis 43. The convergence of the final result is presented in
Fig. 2 ($\Delta E=-863.03$ m$E_{\text{H}}$), in essentially perfect agreement
with what is nowadays regarded as the best estimate of the correlation energy
on the basis of the blind challenge.

The MBE-FCI calculation shown in Fig. 2 involved in excess of 1.55 billion
individual CASCI calculations of variable size and composition and consumed a
staggering 1.5 million core-hours in the process (on Intel Xeon E5-2697v4
(Broadwell) hardware with 36 cores $@$ 2.3 GHz and 3.56 GB/core of physical
memory, Galileo supercomputer at CINECA (Italy)). Although improvements have
subsequently been made to the code base—to the extent where the total compute
time can be reduced by close to a factor 2 (cf. Section 3)—the theory behind
MBE-FCI is arguably somewhat more expensive than its alternatives. However,
with these exhaustive resource requirements follows rigour, as evidenced by
how well converged the final energy is. The change in energy across the final
two orders in the expansion amounts to a mere $-0.04$ m$E_{\text{H}}$, or
$-0.1$ kJ/mol, that is, well within thermochemical tolerance. However, and
this is important to emphasize, no methodical measure of the final uncertainty
against FCI currently exists, which we will elaborate further on in Section 5.
That being said, given ample computational resources, even larger systems (of
a similar nature) will also be amenable to a treatment by MBE-FCI as the
dimension of the largest CASCI calculations in Fig. 2 is well within the
capabilities of even today’s optimized FCI kernels.

Figure 3: RHF, MBE-FCI, and DMRG results for the dissociation curve of an H10
chain in a cc-pVDZ basis set 13, 44. The dashed line indicates the FCI
complete dissociation limit.

As an example of how to deal with static correlation, MBE-FCI was applied in
Ref. 13 to the popular problem of simultaneous stretching all bonds of a
tenfold linear chain of hydrogen atoms. For this model system, the usual
transition from weak to strong correlation observed along practically all bond
dissociations takes place, albeit on a significantly more extended scale as
the involved bonds are all broken at once. While MBE-FCI based on a
CASSCF(10,10) reference was shown to successfully reproduce reference results
obtained using density matrix renormalization group (DMRG) theory 44, MBE-FCI
will hold limited promise of providing detailed information in the
thermodynamic limit whenever it has to rely on CASSCF($N$,$N$) expansion
references. To that end, as was discussed in detail in Ref. 13, canonical
CASSCF orbitals are bound to remain delocalized over large sections of the
chain, a fact which in turn inhibits the orbital screening. For this reason,
additional results, using a reference space comprising only the RHF
determinant, but localized PM rather than canonical virtual orbitals, were
furthermore presented. Displayed in Fig. 3, these results are in excellent
agreement with DMRG for all but the shortest bond distances in the repulsive
region where the concept of locality is anyways somewhat ill-defined.
Importantly, due to the formulation on a standard RHF rather than a CASSCF
reference, MBE-FCI is potentially transferrable to larger chains and basis
sets (and even other topologies, e.g., rings and sheets), thus offering a
viable approach for the treatment of the thermodynamic limit.

While its original formulation was focussed solely on the calculation of
correlation energies for closed- and open-shell systems, MBE-FCI has recently
been extended to the treatment of excitation energies and dipole moments for
ground and excited states 15. In analogy with Eq. 1, excitation energies may
be computed by an expansion of the energetic gap between the ground and an
excited state, $E^{0n}$, rather than the correlation energy

$\displaystyle
E^{0n}_{\text{FCI}}=E^{0n}_{\text{ref}}+\sum_{p}\epsilon^{0n}_{p}+\sum_{p<q}\Delta\epsilon^{0n}_{pq}+\sum_{p<q<r}\Delta\epsilon^{0n}_{pqr}+\ldots$
(3)

As a CASCI calculation in an active space absent of any form of electron
correlation will yield no correlation energy (comprising only the HF solution)
and hence no excited states, $\epsilon^{0n}_{p}$ in Eq. 3 will be defined on
par with $\epsilon^{0}_{p}$ in Eq. 1. With respect to MBE-FCI for static
properties, the calculation of these may be exemplified by focussing on
electronic dipole and transition dipole moments. From the wave function
coefficients of an individual CASCI calculation, the corresponding 1-electron
reduced density matrix (RDM), $\bm{\gamma}^{n}$, for state $n$ may be readily
computed, from which an electronic dipole moment is given as

$\displaystyle\bm{\mu}^{n}_{r}=-\operatorname{Tr}[\bar{\bm{\mu}}_{r}\bm{\gamma}^{n}]$
(4)

in terms of dipole integrals, $\bar{\bm{\mu}}_{r}$, in the MO basis for each
of the three Cartesian components, $r$. Letting these quantities take up the
role of correlation or excitation energies in Eqs. 1 and 3, respectively,
results in the following decomposition of the FCI electronic dipole moment

$\displaystyle\bm{\mu}^{n}_{\text{FCI}}=\bm{\mu}^{n}_{\text{ref}}+\sum_{p}\bm{\mu}^{n}_{p}+\sum_{p<q}\Delta\bm{\mu}^{n}_{pq}+\sum_{p<q<r}\Delta\bm{\mu}^{n}_{pqr}+\ldots$
(5)

Adding the nuclear component, $\bm{\mu}_{\text{nuc}}=\sum_{K}Z_{K}\bm{r}_{K}$,
returns the molecular dipole moment. Finally, transition dipole moments,
$\bm{t}^{0n}$, may be evaluated on par with Eq. 5, except for the fact that
the individual increments are computed on the basis of transition RDMs,
$\bm{\bm{\gamma}}^{0n}$, which may be arrived at using the wave functions of
both states involved in a given CASCI calculation. Being a vector rather than
a scalar quantity, the screening procedure proceeds along all three Cartesian
components ($x,y,z$) in the case of (transition) dipole moments and must be
simultaneously fulfilled for all if a given MO is to be screened away from the
expansion space.

Figure 4: Correlation energies ($\Delta E^{n}$, upper panel) and dipole
moments ($\mu^{n}$, lower panel) for the ground and first two excited states
(${{}^{1}}\Sigma^{+}$ symmetry) of MgO in the aug-cc-pVDZ basis set. Solid and
dashed lines denote MBE-FCI and $i$-FCIQMC results, respectively 15, 45.
(Reproduced with permission from Ref. 15. Copyright 2020, AIP Publishing)

In Figure 4, results are shown for the energies and dipole moments of the
three lowest states of MgO in an aug-cc-pVDZ basis set, for which the frozen-
core FCI correlation problem is described by the distribution of 16 electrons
in 48 orbitals. All of the MBE-FCI results have been obtained using a
reference space spanned by (state-averaged) CASSCF(8,8) calculations, and
comparisons have been made with state-of-the-art FCI quantum Monte Carlo
($i$-FCIQMC) results from Ref. 45. From the MBE-FCI results in Figure 4, it is
observed how the method correctly converges onto the individual states of
interest. Despite the fact that the three states lie in close proximity of
each other, MBE-FCI succeeds in distinguishing between them. In general, the
convergence profiles in the individual plots of Fig. 4 are all different, as
are the MO manifolds being screened away in the calculations, attesting to the
fact that the different properties in question are inherently unrelated and
that MBE-FCI is flexible enough to cope with this within an orbital-based
expansion framework.

Figure 5: Experimental and simulated results for the Cr2 dissociation curve
46, 47, 48.

What are then the practical limitations of current-generation incremental
approaches like MBE-FCI? Consider the potential energy curve of the chromium
dimer in Fig. 5, which has nowadays developed into an appraised stress test
for high-accuracy electronic structure theory. As per valence-bond theory, Cr2
has a formal hextuple bond and the ${{}^{1}}\Sigma^{+}_{g}$ ground state will
eventually dissociate into two equivalent atoms, each in a configuration of
high spin with a total of 6 unpaired electrons in the Cr $3d$ and $4s$ atomic
orbitals (AOs). Unlike for the H10 chain in Fig. 3, where the occupied (and to
some extent even the virtual) MOs localize optimally onto the involved atomic
centers, this is not necessarily the case in general, more complex system
built from atoms of arbitrary covalency. For instance, in the present case of
Cr2, the 12 electrons in question, alongside the 12 MOs that map to the
corresponding AOs, will demand special consideration along the bond
dissociation coordinate. In the language of MBE-FCI, the smallest possible
reference space, which would remain unaltered as the bonds are elongated, will
thus coincide with this ($12$,$12$) active space. As CASSCF in this valence
space only yields a very shallow minimum (at an unreasonable large bond
length) for this particular system, the actual MBE in the expansion space,
which is accountable for the general treatment of dynamical correlation, will
soon come to involve excessively large CASCI calculations. In the vicinity of
the equilibrium bond length, however, MBE-FCI has previously in Ref. 14 been
shown to perform on par with the best possible reference methods 49, even when
based on an empty reference space. However, upon elongating the bonds, the
individual CASCI calculations of such an unbiased MBE start to become
increasingly ill-conditioned and eventually fail to properly converge, as they
cannot possibly accommodate the important (open-shell) determinants of the
aforementioned minimal active space. For the specific case of the Cr2
dissociation, alternative approaches like DMRG or even state-of-the-art
semistochastic heat-bath CI (SHCI) are needed for a qualitatively correct
description of the electronic structure in all correlation domains 48. The
SHCI results from Ref. 48 are reproduced in Fig. 5, but it is important to
note that even this method is having its capabilities stretched in the
shoulder region from $1.8$ to $2.7$ Å, despite the use of a modest basis set
of double-$\zeta$ quality and a scalar-relativistic Hamiltonian Private
correspondence with Cyrus Umrigar (Cornell University).. Further to that, the
basis set dependence exhibited by, e.g., CCSD(T) or SHCI is strong at all
considered bond lengths, which explains the pronounced differences with
respect to experiment data that are still visible in Fig. 5. As impressive as
the SHCI results are on their own, the prospects of rationalizing the odd
profile of the Cr2 dissociation curve by means of near-exact quantum chemistry
thus remain somewhat elusive for now.

##  5 Outlook

In comparison with most of the alternative methods in existence today, many of
these are bound to allow for faster and perhaps more affordable routes towards
simulating FCI properties than MBE-FCI, cf. Ref. 10. However, MBE-FCI arguably
sets itself apart from the rest by offering an incremental, robust, and widely
applicable approach which is principally not restricted by the exponential
scaling wall encountered in, e.g., the various SCI approaches. In addition,
the flexibility of orbital-based MBEs admits a number of further knobs to turn
over traditional approaches centred around individual determinants or
configuration state functions; namely, besides variances with respect to the
employed orbital basis, generalized MBE-FCI allows for the use of different
reference spaces. In the asymptotic limit of an untruncated expansion, the
choice of reference space will be irrelevant as the expansion trivially yields
the exact FCI results, but upon introducing effective protocols for screening
(incrementally) negligible orbitals away from the corresponding expansion
spaces, some choices of reference spaces will yield noticeable more compact
expansions than others. However, how to decide upon an optimal choice in as
black-box a manner as possible still remains mostly unsolved. One feasible
option may be to leverage information on independent orbital correlations at
low orders in the MBE, but additional work on more automated selection schemes
will be the topic of future work on extensions and refinements of MBE-FCI.

Moving forward, it will furthermore be interesting to see if regression
techniques or related statistical processes may be implemented to learn
certain components of MBE-FCI in lieu of a brute-force account of the
underlying electronic structure all the way up throughout the MBE. For
instance, despite the fact that efficient protocols have been implemented to
screen away incremental contributions to the MBE that are deemed energetically
redundant, these remain rather ad hoc in the sense that they lack rigour and
rely on simplified estimates of the correlation between MOs. A promising idea
is now to use modern machine learning 51, 52, particularly models capable of
disentangling the correlation patterns present among the individual MOs on the
basis of the corresponding increments Related ideas have previously also been
proposed by Jason D. Goodpaster (University of Minnesota) in a lecture at the
workshop New Frontiers in Electron Correlation, Telluride, CO, US, in June
2019. Given machine models for different systems, these will collectively aid
in the design of transferable descriptors for use in all future MBE-FCI
calculations. Not only will these enhancements of MBE-FCI see its runtime
execution accelerated, but refined models may further be used to correct final
results by terms that account for the most important of the screened
increments. In turn, this will enable proxies for assessing the inherent
uncertainty of an MBE-FCI run, something which is currently missing. While the
use of regression for this purpose will necessarily result in a departure from
the otherwise rigorous grounds of MBE-FCI, the importance of available error
estimates cannot be underestimated, a point which is equally true in the case
of other near-exact methods. This pertinent issue was also recently discussed
in Ref. 42.

In extending MBE-FCI further, it is worth noting that the underlying theory is
by no means restricted to FCI targets; for instance, orbital-based expansions
may easily well be employed within coupled cluster theory. In addition, the
use of alternative and approximative FCI solvers as a means to allow for
larger (and faster) CASCI calculations remains to be explored within MBE-FCI.
Finally, in the spirit of recent work seeking to revitalize the idea of
transcorrelation 54, 55, 56, 57, 58, 59, a natural, albeit non-trivial
extension of the current generation of MBE-FCI will be to allow for expansions
to spawn from a similarity-transformed Hamiltonian, akin to what is found in
equation-of-motion coupled cluster theory 60. Its appealing traits as a
correlated zeroth-order formulation of electron correlation aside, the non-
Hermiticity of the theory will pose entirely new conceptual as well as
technical challenges. Incorporating correlation directly into the very Ansatz
of MBE-FCI, however, is bound to result in even faster convergent MBEs and may
further prompt the development of a wealth of alternative methods. These may
not necessarily be aimed at a rigorously defined target property (e.g., FCI
ground- or excited-state energies or dipole moments), but rather seeking to
profit from the fact that even low-order truncations of MBE-FCI will yield
qualitatively accurate properties, as an alternative to contemporary
correlated wave function theory.

## Acknowledgments

The authors wish to thank Cyrus Umrigar of Cornell University for fruitful
discussions and for sharing part of the data behind Fig. 5.

## Funding Information

J.J.E. is grateful to the Alexander von Humboldt Foundation as well as The
Independent Research Fund Denmark for financial support throughout the past
couple of years.

## Research Resources

The authors acknowledge access awarded to the Galileo supercomputer at CINECA
(Italy) through the $18^{\text{th}}$ PRACE Project Access Call as well as the
Johannes Gutenberg-Universität Mainz for computing time granted on the MogonII
supercomputer.

## Data Availability Statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * Knowles and Handy 1984 Knowles, P. J.; Handy, N. C. A New Determinant-Based Full Configuration Interaction Method. _Chem. Phys. Lett._ 1984, _111_ , 315
  * Knowles and Handy 1989 Knowles, P. J.; Handy, N. C. A Determinant Based Full Configuration Interaction Program. _Comput. Phys. Commun._ 1989, _54_ , 75
  * Knowles 1989 Knowles, P. J. Very Large Full Configuration Interaction Calculations. _Chem. Phys. Lett._ 1989, _155_ , 513
  * Knowles and Handy 1989 Knowles, P. J.; Handy, N. C. Unlimited Full Configuration Interaction Calculations. _J. Chem. Phys._ 1989, _91_ , 2396
  * Olsen et al. 1988 Olsen, J.; Roos, B. O.; Jørgensen, P.; Jensen, H. J. Aa. Determinant Based Configuration Interaction Algorithms for Complete and Restricted Configuration Interaction Spaces. _J. Chem. Phys._ 1988, _89_ , 2185
  * Olsen et al. 1990 Olsen, J.; Jørgensen, P.; Simons, J. Passing the One-Billion Limit in Full Configuration-Interaction (FCI) Calculations. _Chem. Phys. Lett._ 1990, _169_ , 463
  * Ivanic and Ruedenberg 2001 Ivanic, J.; Ruedenberg, K. Identification of Deadwood in Configuration Spaces through General Direct Configuration Interaction. _Theor. Chem. Acc._ 2001, _106_ , 339
  * Bytautas and Ruedenberg 2009 Bytautas, L.; Ruedenberg, K. A Priori Identification of Configurational Deadwood. _Chem. Phys._ 2009, _356_ , 64
  * Huron et al. 1973 Huron, B.; Malrieu, J.-P.; Rancurel, P. Iterative Perturbation Calculations of Ground and Excited State Energies from Multiconfigurational Zeroth-Order Wavefunctions. _J. Chem. Phys._ 1973, _58_ , 5745
  * Eriksen 2021 Eriksen, J. J. The Shape of Full Configuration Interaction to Come. _J. Phys. Chem. Lett._ 2021, _12_ , 418
  * Eriksen et al. 2017 Eriksen, J. J.; Lipparini, F.; Gauss, J. Virtual Orbital Many-Body Expansions: A Possible Route towards the Full Configuration Interaction Limit. _J. Phys. Chem. Lett._ 2017, _8_ , 4633
  * Eriksen and Gauss 2018 Eriksen, J. J.; Gauss, J. Many-Body Expanded Full Configuration Interaction. I. Weakly Correlated Regime. _J. Chem. Theory Comput._ 2018, _14_ , 5180
  * Eriksen and Gauss 2019 Eriksen, J. J.; Gauss, J. Many-Body Expanded Full Configuration Interaction. II. Strongly Correlated Regime. _J. Chem. Theory Comput._ 2019, _15_ , 4873
  * Eriksen and Gauss 2019 Eriksen, J. J.; Gauss, J. Generalized Many-Body Expanded Full Configuration Interaction Theory. _J. Phys. Chem. Lett._ 2019, _10_ , 7910
  * Eriksen and Gauss 2020 Eriksen, J. J.; Gauss, J. Ground and Excited State First-Order Properties in Many-Body Expanded Full Configuration Interaction Theory. _J. Chem. Phys._ 2020, _153_ , 154107
  * Zimmerman 2017 Zimmerman, P. M. Incremental Full Configuration Interaction. _J. Chem. Phys._ 2017, _146_ , 104102
  * Zimmerman 2017 Zimmerman, P. M. Strong Correlation in Incremental Full Configuration Interaction. _J. Chem. Phys._ 2017, _146_ , 224104
  * Zimmerman 2017 Zimmerman, P. M. Singlet-Triplet Gaps through Incremental Full Configuration Interaction. _J. Phys. Chem. A_ 2017, _121_ , 4712
  * Zimmerman and Rask 2019 Zimmerman, P. M.; Rask, A. E. Evaluation of Full Valence Correlation Energies and Gradients. _J. Chem. Phys._ 2019, _150_ , 244117
  * Nesbet 1967 Nesbet, R. K. Atomic Bethe-Goldstone Equations. I. The Be Atom. _Phys. Rev._ 1967, _155_ , 51
  * Nesbet 1967 Nesbet, R. K. Atomic Bethe-Goldstone Equations. II. The Ne Atom. _Phys. Rev._ 1967, _155_ , 56
  * Nesbet 1968 Nesbet, R. K. Atomic Bethe-Goldstone Equations. III. Correlation Energies of Ground States of Be, B, C, N, O, F, and Ne. _Phys. Rev._ 1968, _175_ , 2
  * 23 It is worth noting that Stoll have also touched upon the idea of using orbital-based MBEs for calculating correlation energies of solid-state systems, see, e.g., Refs. 24, 25, 26
  * Stoll 1992 Stoll, H. The Correlation Energy of Crystalline Silicon. _Chem. Phys. Lett._ 1992, _191_ , 548
  * Stoll 1992 Stoll, H. Correlation Energy of Diamond. _Phys. Rev. B_ 1992, _46_ , 6700
  * Stoll 1992 Stoll, H. On the Correlation Energy of Graphite. _J. Chem. Phys._ 1992, _97_ , 8449
  * Roos 1987 Roos, B. O. The Complete Active Space Self-Consistent Field Method and its Applications in Electronic Structure Calculations. _Adv. Chem. Phys._ 1987, _69_ , 399
  * Olsen 2011 Olsen, J. The CASSCF Method: A Perspective and Commentary. _Int. J. Quant. Chem._ 2011, _111_ , 3267
  * 29 The reason for the absence of first-order contributions in the case of an empty reference space is simply that a CASCI calculation in the space of either a single occupied or virtual orbital will miss a corresponding correlation space to excite electrons into or miss any electrons to excite, respectively
  * Stoll 2019 Stoll, H. Toward a Wavefunction-Based Treatment of Strong Electron Correlation in Extended Systems by Means of Incremental Methods. _J. Chem. Phys._ 2019, _151_ , 044104
  * Löwdin 1955 Löwdin, P.-O. Quantum Theory of Many-Particle Systems. I. Physical Interpretations by Means of Density Matrices, Natural Spin-Orbitals, and Convergence Problems in the Method of Configurational Interaction. _Phys. Rev._ 1955, _97_ , 1474
  * Löwdin and Shull 1956 Löwdin, P.-O.; Shull, H. Natural Orbitals in the Quantum Theory of Two-Electron Systems. _Phys. Rev._ 1956, _101_ , 1730
  * Raghavachari et al. 1989 Raghavachari, K.; Trucks, G. W.; Pople, J. A.; Head-Gordon, M. A Fifth-Order Perturbation Comparison of Electron Correlation Theories. _Chem. Phys. Lett._ 1989, _157_ , 479
  * 34 PyMBE: A Many-Body Expanded Correlation Code, See https://gitlab.com/januseriksen/pymbe
  * Sun et al. 2018 Sun, Q.; Berkelbach, T. C.; Blunt, N. S.; Booth, G. H.; Guo, S.; Li, Z.; Liu, J.; McClain, J. D.; Sayfutyarova, E. R.; Sharma, S.; Wouters, S.; Chan, G. K.-L. PySCF: The Python-Based Simulations of Chemistry Framework. _WIREs Comput. Mol. Sci._ 2018, _8_ , e1340
  * Sun et al. 2020 Sun, Q.; Zhang, X.; Banerjee, S.; Bao, P.; Barbry, M.; Blunt, N. S.; Bogdanov, N. A.; Booth, G. H.; Chen, J.; Cui, Z.-H.; Eriksen, J. J.; Gao, Y.; Guo, S.; Hermann, J.; Hermes, M. R.; Koh, K.; Koval, P.; Lehtola, S.; Li, Z.; Liu, J.; Mardirossian, N.; McClain, J. D.; Motta, M.; Mussard, B.; Pham, H. Q.; Pulkin, A.; Purwanto, W.; Robinson, P. J.; Ronca, E.; Sayfutyarova, E. R.; Scheurer, M.; Schurkus, H. F.; Smith, J. E. T.; Sun, C.; Sun, S.-N.; Upadhyay, S.; Wagner, L. K.; Wang, X.; White, A.; Whitfield, J. D.; Williamson, M. J.; Wouters, S.; Yang, J.; Yu, J. M.; Zhu, T.; Berkelbach, T. C.; Sharma, S.; Sokolov, A. Y.; Chan, G. K.-L. Recent Developments in the PySCF Program Package. _J. Chem. Phys._ 2020, _153_ , 024109
  * Dalcín et al. 2005 Dalcín, L.; Paz, R.; Storti, M. MPI for Python. _J. Parallel Distrib. Comput._ 2005, _65_ , 1108
  * Dalcín et al. 2008 Dalcín, L.; Paz, R.; Storti, M.; D’Elia, J. MPI for Python: Performance Improvements and MPI-2 Extensions. _J. Parallel Distrib. Comput._ 2008, _68_ , 655
  * Dalcín et al. 2011 Dalcín, L.; Kler, P.; Paz, R.; Cosimo, A. Parallel Distributed Computing using Python. _Adv. Water Resour._ 2011, _34_ , 1124
  * Hoefler et al. 2013 Hoefler, T.; Dinan, J.; Buntinas, D.; Balaji, P.; Barrett, B.; Brightwell, R.; Gropp, W.; Kale, V.; Thakur, R. MPI+MPI: A New Hybrid Approach to Parallel Programming with MPI Plus Shared Memory. _Computing_ 2013, _95_ , 1121
  * Dunning Jr. 1989 Dunning Jr., T. H. Gaussian Basis Sets for use in Correlated Molecular Calculations. I. The Atoms Boron through Neon and Hydrogen. _J. Chem. Phys._ 1989, _90_ , 1007
  * Eriksen et al. 2020 Eriksen, J. J.; Anderson, T. A.; Deustua, J. E.; Ghanem, K.; Hait, D.; Hoffmann, M. R.; Lee, S.; Levine, D. S.; Magoulas, I.; Shen, J.; Tubman, N. M.; Whaley, K. B.; Xu, E.; Yao, Y.; Zhang, N.; Alavi, A.; Chan, G. K.-L.; Head-Gordon, M.; Liu, W.; Piecuch, P.; Sharma, S.; Ten-no, S. L.; Umrigar, C. J.; Gauss, J. The Ground State Electronic Energy of Benzene. _J. Phys. Chem. Lett._ 2020, _11_ , 8922
  * Pipek and Mezey 1989 Pipek, J.; Mezey, P. G. A Fast Intrinsic Localization Procedure Applicable for Ab Initio and Semiempirical Linear Combination of Atomic Orbital Wave Functions. _J. Chem. Phys._ 1989, _90_ , 4916
  * Motta et al. 2017 Motta, M.; Ceperley, D. M.; Chan, G. K.-L.; Gomez, J. A.; Gull, E.; Guo, S.; Jiménez-Hoyos, C. A.; Lan, T. N.; Li, J.; Ma, F.; Millis, A. J.; Prokof’ev, N. V.; Ray, U.; Scuseria, G. E.; Sorella, E. M., S. Stoudenmire; Sun, Q.; Tupitsyn, I. S.; White, S. R.; Zgid, D.; Zhang, S. Towards the Solution of the Many-Electron Problem in Real Materials: Equation of State of the Hydrogen Chain with State-of-the-Art Many-Body Methods. _Phys. Rev. X_ 2017, _7_ , 031059
  * Blunt et al. 2017 Blunt, N. S.; Booth, G. H.; Alavi, A. Density Matrices in Full Configuration Interaction Quantum Monte Carlo: Excited States, Transition Dipole Moments, and Parallel Distribution. _J. Chem. Phys._ 2017, _146_ , 244105
  * Casey and Leopold 1993 Casey, S. M.; Leopold, D. G. Negative Ion Photoelectron Spectroscopy of Chromium Dimer. _J. Phys. Chem._ 1993, _97_ , 816
  * Ma et al. 2016 Ma, D.; Li Manni, G.; Olsen, J.; Gagliardi, L. Second-Order Perturbation Theory for Generalized Active Space Self-Consistent-Field Wave Functions. _J. Chem. Theory Comput._ 2016, _12_ , 3208
  * Li et al. 2020 Li, J.; Yao, Y.; Holmes, A. A.; Otten, M.; Sun, Q.; Sharma, S.; Umrigar, C. J. Accurate Many-Body Electronic Structure Near the Basis Set Limit: Application to the Chromium Dimer. _Phys. Rev. Research_ 2020, _2_ , 012015(R)
  * Olivares-Amaya et al. 2015 Olivares-Amaya, R.; Hu, W.; Nakatani, N.; Sharma, S.; Yang, J.; Chan, G. K.-L. The Ab-Initio Density Matrix Renormalization Group in Practice. _J. Chem. Phys._ 2015, _142_ , 034102
  * 50 Private correspondence with Cyrus Umrigar (Cornell University).
  * Dral 2020 Dral, P. O. Quantum Chemistry in the Age of Machine Learning. _J. Phys. Chem. Lett._ 2020, _11_ , 2336
  * von Lilienfeld and Burke 2020 von Lilienfeld, O. A.; Burke, K. Retrospective on a Decade of Machine Learning for Chemical Discovery. _Nat. Commun._ 2020, _11_ , 4895
  * 53 Related ideas have previously also been proposed by Jason D. Goodpaster (University of Minnesota) in a lecture at the workshop New Frontiers in Electron Correlation, Telluride, CO, US, in June 2019
  * Boys and Handy 1969 Boys, S. F.; Handy, N. C. The Determination of Energies and Wavefunctions with Full Electronic Correlation. _Proc. R. Soc. A Math. Phys. Eng. Sci._ 1969, _310_ , 43
  * Handy 1973 Handy, N. C. Towards an Understanding of the Form of Correlated Wavefunctions for Atoms. _J. Chem. Phys._ 1973, _58_ , 279
  * Luo and Alavi 2018 Luo, H.; Alavi, A. Combining the Transcorrelated Method with Full Configuration Interaction Quantum Monte Carlo: Application to the Homogeneous Electron Gas. _J. Chem. Theory Comput._ 2018, _14_ , 1403
  * Dobrautz et al. 2019 Dobrautz, W.; Luo, H.; Alavi, A. Compact Numerical Solutions to the Two-Dimensional Repulsive Hubbard Model Obtained via Nonunitary Similarity Transformations. _Phys. Rev. B_ 2019, _99_ , 075119
  * Cohen et al. 2019 Cohen, A. J.; Luo, H.; Guther, K.; Dobrautz, W.; Tew, D. P.; Alavi, A. Similarity Transformation of the Electronic Schrödinger Equation via Jastrow Factorization. _J. Chem. Phys._ 2019, _151_ , 061101
  * Baiardi and Reiher 2020 Baiardi, A.; Reiher, M. Transcorrelated Density Matrix Renormalization Group. _J. Chem. Phys._ 2020, _153_ , 164115
  * Krylov 2008 Krylov, A. I. Equation-of-Motion Coupled-Cluster Methods for Open-Shell and Electronically Excited Species: The Hitchhiker’s Guide to Fock Space. _Annu. Rev. Phys. Chem._ 2008, _59_ , 433