# A language-inspired machine learning approach for solving strongly
correlated problems with dynamical mean-field theory

Zelong Zhao  King’s College London, Theory and Simulation of Condensed Matter
(TSCM), The Strand, London WC2R 2LS, UK  Hovan Lee  King’s College London,
Theory and Simulation of Condensed Matter (TSCM), The Strand, London WC2R 2LS,
UK  George Booth  King’s College London, Theory and Simulation of Condensed
Matter (TSCM), The Strand, London WC2R 2LS, UK  Weifeng Ge<EMAIL_ADDRESS>Nebula AI Group, School of Computer Science, Fudan University  Cedric Weber
<EMAIL_ADDRESS>King’s College London, Theory and Simulation of
Condensed Matter (TSCM), The Strand, London WC2R 2LS, UK

###### Abstract

We present SCALINN — Strongly Correlated Approach with Language Inspired
Neural Network — as a method for solving the Anderson impurity model and
reducing the computational cost of dynamical mean-field theory calculations.
In analogy to generative Transformer networks and their use in natural
language processing, SCALINN utilises an in-house modified Transformer network
to learn information between low computational cost Green’s function
sequences, in order to generate high accuracy Green’s functions that were
previously restricted behind higher computational costs. Across different
values of temperature and interaction strengths, the performance of SCALINN is
demonstrated in both physical observables (spectral function, Matsubara
Green’s functions, quasi-particle weight), and the mean squared error cost
function values of the neural network.

##  I Introduction

In condensed matter physics, the energy scales of electron itinerancy and
electron-electron Coulomb interactions often define the level of theory
required to adequately describe the phenomena of a system. These two energy
scales overlap for strongly correlated systems, and therefore, the problem
cannot be simplified as the study of one property against the static backdrop
of another. Such as in quasi-particle based models, where electron itinerancy
is considered against renormalised values of effective mass — the
manifestation of electron-electron interactions, a comparatively less
significant effect in such a scenario. As such, the complexity of the problem
increases in systems where it is important to consider both of these
properties on equal footing.

Concomitant with this rise of complexity, comes the emergence of a rich
collection of physics: the Hubbard model Hubbard (1963) is one of the simplest
articulations of this energy overlap, and has maintained its importance to
this day, particularly in the research fields of high temperature
superconductivity Jiang and Devereaux (2019); Kheirkhah  _et al._ (2020);
Foley  _et al._ (2019), cold-atom optical trapping Koepsell  _et al._ (2019);
Brown  _et al._ (2019); Yang  _et al._ (2020), and topologically ordered
phases Shao  _et al._ (2021); Fraxanet  _et al._ (2022); González-Cuadra  _et
al._ (2019) among others.

This Hubbard model, although simple in formulation, has so far only been
analytically solved for the one-dimensional case. Therefore, either a
numerical approach or a truncation of interaction terms is required to solve
its two- and three-dimensional variants. Dynamical mean-field theory Georges
_et al._ (1996) (DMFT) is one such numerical approach that is designed to
solve the Hubbard model through an exact mapping Georges and Kotliar (1992)
onto the Anderson impurity model Anderson (1961). In this scheme, the object
to be converged is the Green’s function: the descriptor of electron creation,
propagation, and subsequent annihilation in a system of other interacting
electrons.

Various methods for carrying out DMFT calculations exist. However, each of
these solvers are plagued with their own limitations, examples of these
include: The Hubbard-I (HI) solver Hubbard (1963), which assumes no electron
itinerancy and is therefore an approximation that is only reasonable for
highly localised systems. The iterative perturbation theory (IPT) solver
Georges  _et al._ (1996) and its third order extension encapsulates all first,
second and third order irreducible processes in the proper self energy. Its
solutions however, do not generalise well beyond systems that are half-filled.
Continuous time quantum Monte Carlo (CTQMC) methods Gull  _et al._ (2007);
Werner and Millis (2006); Werner  _et al._ (2006); Rubtsov  _et al._ (2005)
split the Hamiltonian of the system into two parts — one which is easier to
compute than the other — such that the partition function can be expanded in
orders of the more difficult component. Nonetheless, this method is burdened
with the sign problem. The exact diagonalisation (ED) solver Caffarel and
Krauth (1994); Capone  _et al._ (2007); Liebsch and Ishida (2011) which
calculates the eigenvalues of the Hamiltonian directly, but becomes
computationally prohibitive as an increasing number of impurity and bath
orbitals are taken into consideration. That is, all DMFT solvers to-date are
constrained in one aspect or another, and research on solver improvements are
still ongoing.

A potential solution to this issue could well emerge from another discipline;
due to advancements in hardware accelerators, improvements on memory capacity,
and access to ever-increasingly large databases, deep learning has dominated
the field of machine learning in the last decade Kingma and Ba (2014); LeCun
_et al._ (2015); Vaswani  _et al._ (2017). These developments in deep learning
have enticed materials scientists with possible methods Dunjko and Briegel
(2018); Carleo  _et al._ (2019) to overcome long standing bottlenecks in
condensed matter. A far-from-complete list of these efforts include: The
predicting protein structures with Transformer Jumper  _et al._ (2021). The
learning of functional coefficients in density functional theory Kirkpatrick
_et al._ (2021), the solving analytic continuation with artificial neural
network Fournier  _et al._ (2020), and a previous efforts that aimed to tackle
the same problem of DMFT solvers Arsenault  _et al._ (2014); Sheridan  _et
al._ (2021).

The underlying motivation for these efforts can be elucidated by exploring the
fundamental differences between shallow and deep neural networks. A deep
neural network inherently possesses hierarchical abstraction capabilities,
which enables automated learning of input features without necessitating the
design of specific feature extractors, a property that is absent in shallow
neural networks. As a result, deep neural networks can exhibit simultaneous
sensitivity to minuscule changes and insensitivity to irrelevant variations
LeCun  _et al._ (2015). This aspect is particularly important when dealing
with physical problems, a robust model should be both responsive to phase
transition inducing changes in input variables, whilst remaining insensitive
to changes in symmetry invariants. In contrast, shallow neural networks,
despite their capacity to fully represent any bounded real function Hornik
(1991), typically necessitate a larger number of processing units to reproduce
the representational capabilities of a deep neural network fully. Furthermore,
there exist extreme cases Delalleau and Bengio (2011) where a deep neural
network, with n representing the number of hidden units, would only require
$\mathcal{O}(\text{n})$ to fully represent, while the shallow network would
require $\mathcal{O}(2^{\sqrt{\text{n}}})$. This discrepancy underscores the
superior efficiency and versatility of deep neural networks. This potential
$\mathcal{O}(2^{\sqrt{\text{n}}})$ to $\mathcal{O}(\text{n})$ transformation
is particularly attractive in the field of strongly correlated materials, in
which the dimensionality of the problem scales as $\mathcal{O}(4^{N})$ for $N$
number of orbitals.

In this work, we demonstrate the abilities of SCALINN — Strongly Correlated
Approach with Language Inspired Neural Network — which is based on the
Transformer framework Vaswani  _et al._ (2017). SCALINN predicts the Green’s
functions of strongly correlated systems as sequences in the context of the
single impurity Anderson model (SIAM) and DMFT calculations. The Transformer
framework was chosen due to: 1) the long memory span — Green’s functions as
sequences in frequency exhibit information between frequency points, and
therefore requires the potential unlimited memory provided by the Transformer
network 2) the parallelism enabled by Transformer calculations, and 3) in the
absence of word embedding, the query, key and value method provides a richer
representation between Green’s function frequency sequence entries. To this
end, comparatively cheaper calculations of a system’s Green’s function act as
the encoder input, together with or without other characteristic system
parameters, in order to generate a more accurate estimate of the Green’s
function from the decoder. Through this method, we find that the Transformer
network is able to learn the laws of physics of correlated materials, using
only the curvatures of the input Green’s functions, in order to iteratively
predict the system’s Green’s function in a non-perturbative manner.

The performance of SCALINN at different values of temperature and interaction
strengths is showcased in the main text as predictions of spectral functions,
quasi-particle weights and the imaginary component of the Matsubara Green’s
function (unless otherwise specified, all Matsubara Green’s functions in this
manuscript refer to the imaginary component $Im[G(i\omega_{n})]$ only).
Furthermore, the mean squared error of the network across different network
hyperparameters are listed in the appendix.

##  II Theory

###  II.1 DMFT Primer

In this project, the Hubbard model is mapped onto the SIAM Anderson (1961):

$\displaystyle\hat{H}_{SIAM}=$
$\displaystyle\sum_{\sigma}\varepsilon_{d}\hat{d}_{\sigma}^{\dagger}\hat{d}_{\sigma}+U\hat{d}_{\uparrow}^{\dagger}\hat{d}_{\uparrow}\hat{d}_{\downarrow}^{\dagger}\hat{d}_{\downarrow}+\sum_{\mathbf{k},\sigma}\varepsilon_{\mathbf{k}}\hat{c}_{\mathbf{k}\sigma}^{\dagger}\hat{c}_{\mathbf{k}\sigma}$
(1)
$\displaystyle+\sum_{\mathbf{k},\sigma}V_{\mathbf{k}}\left(\hat{d}_{\sigma}^{\dagger}\hat{c}_{\mathbf{k}\sigma}+\hat{c}_{\mathbf{k}\sigma}^{\dagger}\hat{d}_{\sigma}\right)\,,$

where $\varepsilon_{d}$ is the energy level of the impurity site,
$\hat{d}^{(\dagger)}_{\sigma}$ annihilates (creates) a spin $\sigma$ electron
at the impurity, $U$ determines the energy of the Coulomb interaction between
opposite spin electrons at the impurity. The energy levels of the momentum
$\mathbf{k}$ dependent bath sites are represented as $\epsilon_{\mathbf{k}}$,
and their corresponding annihilation (creation) operators are
$\hat{c}^{(\dagger)}_{\mathbf{k}\sigma}$. Lastly, electrons can hop between
the impurity site and the bath sites with hybridisation strength
$V_{\mathbf{k}}$.

From this SIAM, the creation, propagation, and annihilation of a single
electron (or hole), within the context of the other interacting electrons in
the system is defined by the sum of momentum dependent Green’s functions
$G=\sum_{\mathbf{k}}G_{\mathbf{k}}$. In DMFT, this Green’s function is assumed
to be equal to the impurity Green’s function:

$G_{\mathrm{imp}}(i\omega_{n})=[i\omega_{n}+\mu-\varepsilon_{d}-\Sigma(i\omega_{n})-\Delta(i\omega_{n})]^{-1}\,,$
(2)

where $i\omega_{n}$ are the Matsubara frequency points which are measures of
electron energy and system temperature, $\mu$ is the chemical potential which
controls the number of electrons within the system. The self energy
$\Sigma(i\omega_{n})$ is a measure of how the propagation of the electron
created by the Green’s function affects all other electrons in the system, and
how the system acts on the propagating electron, which in turn influences the
propagation of the created electron. Due to this perpetual relationship
between the propagating electron and all other electrons in the system,
$\Sigma(i\omega_{n})$ has to be solved iteratively towards convergence.
Lastly, the hybridisation function $\Delta(i\omega_{n})$ in principle contains
the dynamics of a bath that spans all values of momentum $\mathbf{k}$, but
within the ED scheme, it is approximated with a discrete number of bath sites:

$\Delta^{ED}(i\omega_{n})=\sum^{N_{b}}_{p=1}V^{2}_{p}/(i\omega_{n}-\epsilon_{p})\,,$
(3)

with number of bath sites $N_{b}$, the hybridisation parameter between the
impurity and each bath site are determined by $V_{p}$, and the bath energies
are denoted as $\epsilon_{p}$.

Since the only term in eq. 2 that describes electron-electron interactions is
$\Sigma(i\omega_{n})$, the Green’s function can be represented in terms of a
free Green’s function $G^{0}$ that does not contain electron-electron
interactions through the Dyson equation:

$(G^{ED}_{\mathrm{imp}}(i\omega_{n}))^{-1}=\underbrace{[i\omega_{n}+\mu-\varepsilon_{d}-\Delta(i\omega_{n})]^{-1}}_{G^{0}}-\Sigma(i\omega_{n})\,,$
(4)

such that the DMFT loop can be written in terms of the ED impurity Green’s
function $G^{ED}(i\omega_{n})$ as:

$\begin{split}G^{ED}(i\omega_{n})&=\sum_{\mathbf{k}}G_{\mathbf{k}}(i\omega_{n})\\\
&=\sum_{\mathbf{k}}[i\omega_{n}+\mu-\varepsilon_{\mathbf{k}}-\Sigma_{\mathbf{k}}(i\omega_{n})]^{-1}\\\
&=\sum_{\mathbf{k}}[i\omega_{n}+\mu-\varepsilon_{\mathbf{k}}-\Sigma(i\omega_{n})]^{-1}\\\
&=\sum_{\mathbf{k}}[\varepsilon_{d}+\Delta^{ED}(\omega)-\varepsilon_{\mathbf{k}}+G^{ED}(i\omega_{n})^{-1}]^{-1}\,,\end{split}$
(5)

where $G^{ED}(i\omega_{n})$ is present on both sides of the equation, and is
updated self-consistently at each iteration.

###  II.2 SCALINN: Inputs and Outputs

Figure 1: Illustrations of the target, inputs and structure of the Transformer
model. In a), a depiction of the ED scheme is shown: on the impurity site and
the bath sites spin up and spin down electrons are drawn as straight green and
orange arrows respectively, the permitted hybridisations between impurity and
bath are shown as curved arrows for each spin. Green’s functions
$G^{ED}(i\omega_{n})$ calculated from this ED scheme are shown in red, and act
as targets of the Transformer model. The truncated scheme is illustrated in
b), where a number of bath sites are discarded from the ED scheme. In this
respect, multiple Green’s functions are calculated per one target
$G^{ED}(i\omega_{n})$, each with a different combination of truncated bath
sites. The HI+IPT hybrid scheme is presented in c). Here, one Green’s function
is calculated with the HI solver and one is calculated with the IPT solver,
these Green’s functions are then concatenated to form
$G^{input}(i\omega_{n})$. The Transformer model is presented in d), where
neural network blocks that are identical to the original network in Vaswani
_et al._ (2017) are presented as the Encoder Block and hashed blocks.

SCALINN is a modified version of the original Transformer model by Vaswani et
al. Vaswani  _et al._ (2017), these changes were made such that the model
could more appropriately tackle the problem of predicting DMFT solutions.
First, the target of the model is defined as the ED solution to a single DMFT
iteration.

The cost function $J$ is therefore defined as the mean squared error:

$J(\hat{G}(i\omega_{n}),G^{ED}(i\omega_{n}))=\frac{1}{N}\sum^{N}_{n=0}(\hat{G}(i\omega_{n})-G^{ED}(i\omega_{n}))^{2}\,,$
(6)

where $\hat{G}(i\omega_{n})$ is the predicted Green’s function,
$G^{ED}(i\omega_{n})$ is the ED solution (as illustrated in fig. 1.a), and the
sum is performed over all Matsubara frequency points.

For the Green’s function input, the goal is to supply the network with Green’s
function sequences that require less computational resources to calculate. In
order to test the validity of this concept, two different sets of models were
trained, one that requires a truncated ED solution as input (fig. 1.b), and
one that requires the HI and IPT solutions as input (fig. 1.c).

Although the ED solver can calculate Green’s functions directly in frequency
$\omega$, the choice of working in Matsubara frequency $i\omega_{n}$ is
motivated by the similarities between $G(i\omega_{n})$ and word sequences:
Firstly, Matsubara points are the poles of the Fermi-Dirac function and hence
the representation is discrete, akin to words of a sentence. Moreover,
$G(i\omega_{n})$ is well defined for all Matsubara points at equilibrium, and
can be calculated independently of one another. Furthermore, entries within
sequences of $G(i\omega_{n})$ are ordered in terms of energy, as such these
entries are not lone, randomly distributed points, there is information to be
inferred between entries within the sequence. These properties, which
$G(i\omega_{n})$ shares with natural language processing, allows the problem
of correlated materials to be transcribed and tackled with the Transformer
network. Lastly, the condition of causality of $G(i\omega_{n})$ can be tested
in straightforward manner, this avoids non-physical solutions, and provides
one more advantage by working in $G(i\omega_{n})$.

The truncated ED inputs were constructed by removing a number of discretised
bath sites from the ED formalism, such that a smaller number of terms is
required in the construction of the hybridisation function
$\Delta(i\omega_{n})$. A set of three of these calculations with different
bath sites truncated were performed for each data sample, and the resultant
Green’s functions were concatenated as the input $G(i\omega_{n})$ of the
Transformer model in fig. 1.d.

Since the Hilbert space of the solution scales exponentially with the number
of sites in the Hamiltonian, this method aims to decrease the degree of
exponentiality of the calculation, and substitute it with a linear increase of
requiring three truncated Green’s functions and the subsequent forward
propagation through the network, in order to predict the solution Green’s
function.

With the HI + IPT hybrid method, the HI and IPT calculated Green’s functions
are concatenated and given as the input $G(i\omega_{n})$ of the Transformer
model. The goal is to combine the results of two methods with complementary
limits: HI, which represents the large $U$ limit where there are no
hybridisations between the impurity and the bath sites, and IPT which is a
perturbation expansion in orders of $U$. The network then aims to predict
$G^{ED}(i\omega_{n})$ with these complementary Green’s functions.

###  II.3 SCALINN Overview

A brief description of the various neural network blocks of SCALINN is as
follows:

Starting at the encoder input, top left of fig. 1.d,
$G^{input}(i\omega_{n})\in\mathbb{R}^{N_{\omega}+1\crossproduct l}$ is
concatenated with the Matsubara encoding, as denoted with the C operation. In
natural language processing, traditional recurrent neural networks have a
short memory, where the context gained from words at the start of a sentence
could be lost when the model reaches the end of the sentence. The original
Transformer network overcomes this difficulty by positional encoding, whereby
the position of words in the sentence are encoded onto the input word entries.

This method of positional encoding however, requires the input words to be
cast into a vector format via word2vec; corresponding to the meanings of
various words encoded in vectors. Since $G(i\omega_{n})$ are sequences of
continuous functions, instead of sequences of words, the original method of
positional encoding would not be a viable solution to deal with the
aforementioned memory issue. Therefore, the fermionic Matsubara frequencies
are concatenated onto $G^{input}(i\omega_{n})$ (Eq. 7), such that if
$G^{input}(i\omega_{n})\in\mathbb{R}^{N_{\omega}+1\crossproduct 3}$, the
concatenation would increase it to $\mathbb{R}^{N_{\omega}+1\crossproduct 4}$.

$PE_{\mathrm{Matsu}}\leavevmode\hbox to9.97pt{\vbox
to9.97pt{\pgfpicture\makeatletter\hbox{\hskip 4.98735pt\lower-4.98735pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{4.78735pt}{0.0pt}\pgfsys@curveto{4.78735pt}{2.64401pt}{2.64401pt}{4.78735pt}{0.0pt}{4.78735pt}\pgfsys@curveto{-2.64401pt}{4.78735pt}{-4.78735pt}{2.64401pt}{-4.78735pt}{0.0pt}\pgfsys@curveto{-4.78735pt}{-2.64401pt}{-2.64401pt}{-4.78735pt}{0.0pt}{-4.78735pt}\pgfsys@curveto{2.64401pt}{-4.78735pt}{4.78735pt}{-2.64401pt}{4.78735pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.22221pt}{-2.15277pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{c}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}G^{input}(i\omega_{n}):=G^{input}(i\omega_{n})\mathbin{\|}\frac{(2n+1)\pi}{\beta}$
(7)

Moreover, to reintroduce the representability of information afforded by
word2vec in the original Transformer network, a multilayer perceptron (MLP)
block is employed. This then leads to the encoder block, which is identical to
the encoder block of the original network. This encoder block allows the
network to perform self-attention, a method of computing the relevance and
context between different entries within the input sequence. In total,
information passes through this encoder block $n_{Encoder}$ number of times.
The performances of the models with difference choice of $n_{Encoder}$ are
shown in table A1.

Another crucial difference between the outputs of the original Transformer
model, and the network presented here, is that the outputs of the original
model are the probabilities that informs which next word would best fit the
output sequence. Whereas, the output to our network is a prediction of the
Green’s function — a continuous function instead of a discrete word — at the
next Matsubara frequency point. As such, the softmax layer of the original
model, which outputs probabilities of a set of discretised solutions (such as
the case of word selection in the original work), is discarded. The network
therefore terminates with a linear layer.

Moving onto the decoder portion of the network, the input to the decoder is
subject to the same Matsubara encoding, but is passed to a another MLP block.
In an effort to enhance the overall performance of the model, we have
integrated a masked multi-head self-attention mechanism. The operation of each
head is defined as follows:

$\displaystyle\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{\mathbf{M}\mathbbm{1}+QK^{T}}{\sqrt{d_{k}}}\right)\cdot
V$ (8) $\displaystyle\mathbf{M}^{\mathrm{RF-TGT}}_{ij}=\begin{cases}0&\text{
if }i-\text{LB}<j<i+1\\\ -\infty&\text{else}\end{cases}$
$\displaystyle\mathbf{M}^{\mathrm{SRC}}_{ij}=\begin{cases}0&\text{ if
}i-\text{LB}<j<i+1+\text{LF}\\\ -\infty&\text{else}\end{cases}$

The query, keys, and values - represented as $Q,K,V$, respectively - are the
learned vector components in this masked multi-head self-attention mechanism
(MHSA). Here, $d_{k}$ is defined as $d_{\text{model}}/h$, where
$d_{\text{model}}$ refers to the dimension of the model and $h$ denotes the
number of attention heads. Moreover, LF and LB, which signify lookforward and
lookbackward respectively, are positive integers. The matrices
$\mathbf{M}^{\mathrm{RF-TGT}}$ and $\mathbf{M}^{\mathrm{SRC}}$ are
incorporated into the ‘Right-Shifted Target Masked Multi-Head Self-Attention’
(RF-TGT Masked MHSA) block and the ‘Source Masked Multi-Head Self-Attention’
(Source Masked MHSA) block respectively. Each attention head is then
concatenated and projected to a dimension of $d_{\text{model}}$, following the
methodology described in Vaswani  _et al._ (2017).

Different Green’s functions are inputted to the Transformer decoder during
training and run-time. During training, the entire
$G^{ED}(i\omega_{n})\in\mathbb{R}^{N_{\omega}},n\in[1,N_{\omega}]$ is already
known. Following projection, the dimensions of the query and keys are
represented as $Q_{\text{RF-TGT}}\in\mathbb{R}^{N_{\omega}\crossproduct
d_{k}}$ and $K_{\text{RF-TGT}}\in\mathbb{R}^{N_{\omega}\crossproduct d_{k}}$
respectively. Accordingly, a square-shaped subsequent mask,
$\mathbf{M}^{\mathrm{RF-TGT}}\in\mathbb{R}^{N_{\omega}\crossproduct
N_{\omega}}$, is applied to make the model predictions of the next Matsubara
point ignorant to the target solution.

In contrast, the inputs to the Transformer encoder remain the same during both
training and run-time, thus ensuring the output dimension of the Transformer
encoder consistently remains $\mathbb{R}^{N_{\omega}+1\crossproduct
d_{\text{model}}}$. Within the source masked MHSA layer, K and V are derived
from the Transformer Encoder, resulting in
$K_{\text{source}}\in\mathbb{R}^{N_{\omega}+1\crossproduct d_{k}}$ and
$V_{\text{source}}\in\mathbb{R}^{N_{\omega}+1\crossproduct d_{k}}$.
Concurrently, Q is sourced from the right-shifted target, leading to
$Q_{\text{source}}\in\mathbb{R}^{N_{\omega}\crossproduct d_{k}}$. As a result,
the source masked MHSA possesses the dimensions of
$\mathbf{M}^{\mathrm{src}}\in\mathbb{R}^{N_{\omega}\crossproduct
N_{\omega}+1}$. This configuration effectively adjusts the attention scores
within a scope that ranges between lookforward and lookbackward, enabling the
model to adapt its focus according to the complexities of the input data.

The RF-TGT Masked MHSA blocks are responsible for performing self-attention
with the decoder input, whereas the source MHSA block calculates self-
attention between the encoder input and the decoder input. Generally, with a
higher number of heads, the richness of information that can be learned in
these multi-head self-attention blocks increases. This hyperparameter is
varied, and the performances of models with different number of heads, and
different number of decoder blocks ($n_{Decoder}$) are listed in table A1.

Analogous to approximating Matsubara Green’s function with a series truncation
in powers of $(i\omega_{n})^{-1}$, where Green’s function points at large
values of $i\omega_{n}$ are well approximated, this Transformer model begins
its sequence prediction at the last Matsubara point
$i\omega_{n=N_{\omega}-1}$. During prediction, the network decoder is ignorant
to the $G^{ED}(i\omega_{n})$ solution, as such, an auxiliary Green’s function
$G^{aux}(i\omega_{n})\in\mathbb{R}^{N_{\omega}},n\in[1,N_{\omega}]$ is
iteratively constructed to fill the role of $G^{ED}(i\omega_{n})$. We make the
assumption that the highest frequency tail points of $G^{aux}$ is close to the
input Green’s function, so
$G^{aux}(i\omega_{n=N_{\omega}})=G^{input}(i\omega_{n=N_{\omega}})$. To begin,
$G^{aux}(i\omega_{n=N_{\omega}})$ are fed into the decoder, together with the
full $G^{input}(i\omega_{n})$ sequence at the encoder, the network generates
the next highest frequency point of $G^{aux}(i\omega_{n=N_{\omega}-1})$. Those
high frequency $G^{aux}$ point are then kept and reintroduced to the decoder
to approach low frequency points in $G^{aux}$, such that
$G^{aux}(i\omega_{n=m+1})$ generates $G^{aux}(i\omega_{n=m})$ until the lowest
frequency $G^{aux}$ point is determined, resulting in the full
$G^{aux}(i\omega_{n})$. The final Green’s function prediction
$\hat{G}(i\omega_{n})\in\mathbb{R}^{N_{\omega}},n\in[0,N_{\omega}-1]$ is
determined by feeding $G^{aux}(i\omega_{n})$ to the decoder.

##  III Results

#### Computational Details

To setup the training targets of the model, three $G^{ED}(i\omega_{n})$ data-
sets at different inverse temperatures $\beta=10,50,100$ were generated, each
consisting of 14,000 entries of 7-bath SIAM samples. The ED samples were
generated utilising the ED-KCL package Weber  _et al._ (2012) and Post-
processing of output Green’s function are conducted by TRIQS package Parcollet
_et al._ (2015). Implementation of Transformer model was facilitated by the
PyTorch package Paszke  _et al._ (2017). The training of our base model,
detailed in Table A1, was conducted on a machine equipped with a single NVIDIA
A100 GPU. Each training step was completed in a span of 0.1 seconds, and we
performed 350,000 training steps in total, which took approximately 9 hours.

The hybridisation functions of these data-sets were calculated by fitting
$\Delta^{ED}(i\omega_{n})$ to a hybridisation function
$\Delta^{SC}(i\omega_{n})$ that is derived from a system with a semi-circular
density of states:

$\Delta^{SC}\left(i\omega_{n}\right)=\frac{1}{2\pi^{2}}\int_{-\infty}^{\infty}d\epsilon\frac{\sqrt{W^{2}-\epsilon^{2}}}{i\omega_{n}-\epsilon}\Theta(W-|\epsilon|)\,,$
(9)

where $W$ is the bandwidth of the semi-circular density of states. This
fitting was performed such that the mean squared difference:

$\sum_{i\omega_{n}}\frac{1}{i\omega_{n}}(\Delta^{ED}(i\omega_{n})-\Delta^{SC}(i\omega_{n}))^{2}$
(10)

falls randomly within the exclusive range of $(0,1eV)$.

For any DMFT calculation, the self-energy is updated, potentially prompting
metal-insulator phase changes as the self-consistent loop iterates. To reflect
this, a range of samples with different values of $U$ were utilised in
training: $U$ is uniformly generated between 0 and 10 eV. Related to this, the
impurity level $\epsilon_{d}$ is kept at $-U/2$, and the chemical potential
$\mu$ is set to $-\epsilon_{d}-U/2$.

###  III.1 Truncated Hamiltonian

Figure 2: Results from a single SCALINN iteration with the truncated scheme.
In a), the ED calculated solution $G^{ED}(i\omega_{n})$ is plotted as blue
lines, and the three truncated $G^{Trunc}(i\omega_{n})$ are plotted in dashed
grey. These $G^{Trunc}(i\omega_{n})$ act as inputs to the Transformer model to
produce the SCALINN predictions $\hat{G}(i\omega_{n})$, which are plotted as
blue dots. In b), the Padé approximation was applied to the predictions
$\hat{G}(i\omega_{n})$ to obtain the predicted spectral function $A(\omega)$,
shown here for three different values of $\beta$ and $U$. Once again, the ED
solution is plotted as blue lines, and the SCALINN predictions are plotted as
blue dots.

As mentioned previously, the calculation complexity of ED scales
exponentially. In order to decrease this computational cost, the 7-bath SIAM
target samples $G^{ED}(i\omega_{n})$ are supplemented with three 3-bath SIAM
input samples $\mathbin{\|}_{l=1}^{3}G^{Trunc}_{l}(i\omega_{n})$, where the
bath sites were chosen such that all three $G^{Trunc}(i\omega_{n})$ preserve
particle-hole symmetry. An example of a set of these $G^{Trunc}(i\omega_{n})$
is shown in fig. 2.a, where each $G^{Trunc}(i\omega_{n})$ differs
significantly from the target $G^{ED}(i\omega_{n})$. However, the three models
at different inverse temperatures (which were all trained to achieve costs of
$J(\hat{G}(i\omega_{n}),G^{ED}(i\omega_{n}))<1e^{-5}$) are able to recover
$G^{ED}(i\omega_{n})$.

It is important to note that the results shown here are obtained from a single
iteration the the DMFT loop, instead of via the self-consistent scheme of
DMFT. The reason for this lack of self-consistency is due to how the truncated
scheme is formulated: in order to perform DMFT with this scheme, after each
iteration, the predicted Green’s function $\hat{G}(i\omega_{n})$ would need to
be recast as a new 7-bath ED system and three new 3-bath truncated systems
with updated $\Delta(i\omega_{n})$ values to be refitted. As such, we only
present the results from a single iteration of SCALINN.

The Padé approximation was applied to these predicted $\hat{G}(i\omega_{n})$
to obtain predictions of the spectral function $\hat{A}(\omega)$, as it is
natural to showcase ED results in real frequency. These predictions, along
with their ED counterparts are illustrated in fig. 2.b for three values of
inverse temperature $\beta$ and three values of interaction $U$.

In this figure, the classical property of correlated materials — the Mott
insulator transition — was reproduced: At low interaction strength $U=1$, the
systems manifest as metals with singular quasi-particle peaks at zero energy.
As temperature increases (or as $\beta$ decreases), so does the rate of
scattering between electrons, leading to a broadening of these peaks. At
intermediate interaction strength $U=3$, lower and upper Hubbard bands at $\pm
U/2$ emerge around the quasi-particle peaks at $\omega=0$. Once again, due to
increasing rates of scattering, these three sets of peaks are broadened as
temperature increases. Lastly, at higher interaction strength still $U=6$, the
magnitude of the quasi-particle peaks are greatly reduced in favour of the
Hubbard bands. Due to the lack of self-consistency, the fully insulating
solution was not recovered.

###  III.2 Hybrid Approximation Solver

Figure 3: Single DMFT iteration HI+IPT hybrid scheme results. The HI solver
solution $G^{HI}(i\omega_{n})$ (dashed red), the IPT solver solution
$G^{IPT}(i\omega_{n})$ (dashed yellow), and the ED target
$G^{ED}(i\omega_{n})$ (solid blue lines) are plotted alongside the SCALINN
predictions (blue dots) at $\beta=10,50,100$, and at $U=1,3,6\,eV$.

In addition to the truncated method, an H1+IPT hybrid scheme was also
investigated. In this hybrid method, models were trained from H1+IPT
concatenated inputs and ED targets, where once again the three separate
inverse temperature models were trained to reach errors of
$J(\hat{G}(i\omega_{n}),G^{ED}(i\omega_{n}))<1e^{-5}$.

In fig. 3, the various Green’s functions are presented in $i\omega_{n}$ to
avoid influencing the H1 and IPT results with Padé or another method of
analytical continuation. As can be expected, the HI solutions
$G^{HI}(i\omega_{n})$ tends towards zero as $i\omega_{n}\rightarrow 0$. This
tendency indicates that the HI solver favours insulator solutions. The IPT
solver on the other hand tends to delocalize the system: As
$i\omega_{n}\rightarrow 0$, $G^{IPT}(i\omega_{n})$ deviate away from zero.

At low interaction strengths $U=1$, the trend of $G^{ED}(i\omega_{n})$ is
similar to that of $G^{IPT}$. This is because the IPT solver is based on
perturbations in powers of $U$, and therefore is accurate at low values of
$U$. At higher interaction strengths $U=3,6$ the ED target
$G^{ED}(i\omega_{n})$ falls between $G^{HI}(i\omega_{n})$ and
$G^{IPT}(i\omega_{n})$.

For all combinations of $\beta$ and $U$, the models are able to predict
$G^{ED}(i\omega_{n})$ accurately, given only $G^{HI}(i\omega_{n})$ and
$G^{IPT}(i\omega_{n})$ as inputs.

Figure 4: Bethe lattice DMFT-converged results at $\mathbf{\beta=10}$. a)
Quasi-particle weight as calculated with HI (dashed red), IPT (dashed yellow)
and CTQMC (solid blue line) solvers, along with the SCALINN predictions. And
b) the imaginary part of the corresponding Green’s functions.

In contrast to the truncated scheme, where $\hat{G}(i\omega_{n})$ has to be
recast into a new 7-site ED problem between DMFT iterations, the HI and IPT
hybrid scheme does not encounter such a problem. Restricting the calculations
to the Bethe lattice, the IPT hybridisation can be updated via:

$\hat{\Delta}(i\omega_{n})=\frac{W^{2}}{4}\hat{G}(i\omega_{n})\,,$ (11)

where $W$ is the full bandwidth of the semi-circular density of states. As
such, the free Green’s function $G^{0}$ in the next iteration of HI and IPT
calculations can be constructed according to eq. 4, and the self-consistent
DMFT loop can be carried out. These DMFT calculations are performed in order
to obtain predicted Green’s functions that exhibits similar levels of accuracy
compared to the exact solutions of DMFT in terms of Matsubara frequency, as
calculated via CTQMC.

The results of the $\beta=10$ case are illustrated in fig. 4. Predictions of
quasi-particle weight $Z$:

$Z=\left.\frac{\operatorname{Im}\left\\{\Sigma\left(i\omega_{n}\right)\right\\}}{\omega_{n}}\right|_{\omega_{n}\rightarrow
0}\,,$ (12)

are shown in fig. 4.a. Here, are as interaction strength increases, $Z$
decreases. This trend suggests that the electron effective mass increases.

Moreover, the Green’s functions are plotted in fig. 4.b, where as interaction
strength increases, the solutions become increasingly insulating, as can be
observed from the $i\omega_{n}\rightarrow 0$ trend of $G(i\omega_{n})$.

It should be noted that these insulating solutions differ from the single
iteration solutions of the left column of fig. 3. This is due to the
difference between the converged self-consistent DMFT solutions, and the
corresponding single iteration calculations.

##  IV Conclusion

With the insights gained from drawing comparisons to natural language
processing problems, we developed a novel and less computationally intensive
approach that predicts Green’s function sequences. These predicted sequences
exhibit levels of accuracy that were previously restricted to comparatively
high computational costs. Our method is based on modifications of the widely
successful Transformer network.

We evaluated the performance of our network on solving single iteration
dynamical mean-field theory calculations with two different schemes: a
truncated exact diagonalisation scheme, in which four of the bath-sites of a
7-bath DMFT exact diagonalisation solver are discarded, together with a
Hubbard I + iterative perturbation theory hybrid scheme. Both of these methods
were found to predict Green’s function, as calculated with the full 7-bath
exact diagonalisation solver, with remarkable accuracy.

Moreover, we showcased a set of DMFT-converged predictions using out network.
These predictions exhibit a level of accuracy comparable to state-of-the-art
methods such as continuous time quantum Monte Carlo. Overall, our findings
highlight the power and adaptability of the Transformer model within the field
of correlated materials and its potential for pushing the frontiers of
computational problem-solving in this domain.

## Acknowledgment

We are grateful to the King’s Computational Research, Engineering and
Technology Environment (CREATE) and UK Materials and Molecular Modelling Hub
for computational resources, which is partially funded by EPSRC (EP/T022213/1,
EP/W032260/1 and EP/P020194/1).

## Data availability

An exemplary dataset can be found at
https://dx.doi.org/10.6084/m9.figshare.23144474.

## Code availability

The code for SCALINN is available at https://github.com/zelong-zhao/SCALINN.

## References

  * Hubbard (1963) J. Hubbard, Proc. Math. Phys. Eng. Sci. P ROY SOC A-MATH PHY 276, 238 (1963).
  * Jiang and Devereaux (2019) H.-C. Jiang and T. P. Devereaux, Science 365, 1424 (2019).
  * Kheirkhah  _et al._ (2020) M. Kheirkhah, Z. Yan, Y. Nagai, and F. Marsiglio, Phys. Rev. Lett. 125, 017001 (2020).
  * Foley  _et al._ (2019) A. Foley, S. Verret, A.-M. S. Tremblay, and D. Sénéchal, Phys. Rev. B 99, 184510 (2019).
  * Koepsell  _et al._ (2019) J. Koepsell, J. Vijayan, P. Sompet, F. Grusdt, T. A. Hilker, E. Demler, G. Salomon, I. Bloch, and C. Gross, Nature 572, 358 (2019).
  * Brown  _et al._ (2019) P. T. Brown, D. Mitra, E. Guardado-Sanchez, R. Nourafkan, A. Reymbaut, C.-D. Hébert, S. Bergeron, A.-M. Tremblay, J. Kokalj, D. A. Huse, _et al._ , Science 363, 379 (2019).
  * Yang  _et al._ (2020) B. Yang, H. Sun, R. Ott, H.-Y. Wang, T. V. Zache, J. C. Halimeh, Z.-S. Yuan, P. Hauke, and J.-W. Pan, Nature 587, 392 (2020).
  * Shao  _et al._ (2021) C. Shao, E. V. Castro, S. Hu, and R. Mondaini, Phys. Rev. B 103, 035125 (2021).
  * Fraxanet  _et al._ (2022) J. Fraxanet, D. González-Cuadra, T. Pfau, M. Lewenstein, T. Langen, and L. Barbiero, Phys. Rev. Lett. 128, 043402 (2022).
  * González-Cuadra  _et al._ (2019) D. González-Cuadra, A. Dauphin, P. R. Grzybowski, P. Wójcik, M. Lewenstein, and A. Bermudez, Phys. Rev. B 99, 045139 (2019).
  * Georges  _et al._ (1996) A. Georges, G. Kotliar, W. Krauth, and M. J. Rozenberg, Rev. Mod. Phys. 68, 13 (1996).
  * Georges and Kotliar (1992) A. Georges and G. Kotliar, Phys. Rev. B 45, 6479 (1992).
  * Anderson (1961) P. W. Anderson, Phys. Rev. 124, 41 (1961).
  * Gull  _et al._ (2007) E. Gull, P. Werner, A. Millis, and M. Troyer, Phys. Rev. B 76, 235123 (2007).
  * Werner and Millis (2006) P. Werner and A. J. Millis, Phys. Rev. B 74, 155107 (2006).
  * Werner  _et al._ (2006) P. Werner, A. Comanac, L. De’Medici, M. Troyer, and A. J. Millis, Phys. Rev. Lett. 97, 076405 (2006).
  * Rubtsov  _et al._ (2005) A. N. Rubtsov, V. V. Savkin, and A. I. Lichtenstein, Phys. Rev. B 72, 035122 (2005).
  * Caffarel and Krauth (1994) M. Caffarel and W. Krauth, Phys. Rev. Lett. 72, 1545 (1994).
  * Capone  _et al._ (2007) M. Capone, L. de’Medici, and A. Georges, Phys. Rev. B 76, 245116 (2007).
  * Liebsch and Ishida (2011) A. Liebsch and H. Ishida, J. Phys. Condens. Matter 24, 053201 (2011).
  * Kingma and Ba (2014) D. P. Kingma and J. Ba, arXiv 10.48550/arXiv.1412.6980 (2014).
  * LeCun  _et al._ (2015) Y. LeCun, Y. Bengio, and G. Hinton, nature 521, 436 (2015).
  * Vaswani  _et al._ (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, NeurIPS 30, 10.48550/arXiv.1706.03762 (2017).
  * Dunjko and Briegel (2018) V. Dunjko and H. J. Briegel, Rep. Prog. Phys. 81, 074001 (2018).
  * Carleo  _et al._ (2019) G. Carleo, I. Cirac, K. Cranmer, L. Daudet, M. Schuld, N. Tishby, L. Vogt-Maranto, and L. Zdeborová, Rev. Mod. Phys. 91, 045002 (2019).
  * Jumper  _et al._ (2021) J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, _et al._ , Nature 596, 583 (2021).
  * Kirkpatrick  _et al._ (2021) J. Kirkpatrick, B. McMorrow, D. H. Turban, A. L. Gaunt, J. S. Spencer, A. G. Matthews, A. Obika, L. Thiry, M. Fortunato, D. Pfau, _et al._ , Science 374, 1385 (2021).
  * Fournier  _et al._ (2020) R. Fournier, L. Wang, O. V. Yazyev, and Q. Wu, Phys. Rev. Lett. 124, 056401 (2020).
  * Arsenault  _et al._ (2014) L.-F. Arsenault, A. Lopez-Bezanilla, O. A. von Lilienfeld, and A. J. Millis, Phys. Rev. B 90, 155136 (2014).
  * Sheridan  _et al._ (2021) E. Sheridan, C. Rhodes, F. Jamet, I. Rungger, and C. Weber, Phys. Rev. B 104, 205120 (2021).
  * Hornik (1991) K. Hornik, Neural networks 4, 251 (1991).
  * Delalleau and Bengio (2011) O. Delalleau and Y. Bengio, NeurIPS 24 (2011).
  * Weber  _et al._ (2012) C. Weber, A. Amaricci, M. Capone, and P. Littlewood, Phys. Rev. B 86, 115136 (2012).
  * Parcollet  _et al._ (2015) O. Parcollet, M. Ferrero, T. Ayral, H. Hafermann, I. Krivenko, L. Messio, and P. Seth, Comput. Phys. Commun. 196, 398 (2015).
  * Paszke  _et al._ (2017) A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, openreview (2017).

Supplemental Materials: A language-inspired machine learning approach for
solving strongly correlated problems with dynamical mean-field theory

##  Appendix A Hyperparameter optimisation

###  A.1 Transformer Hyperparameter Tuning

Task:

The purpose of the Transformer model is to recover the imaginary component of
the Green’s function
$\mathrm{Im}[G(i\omega_{n})]\in\mathbb{R}^{N_{\omega}},n\in[0,N_{\omega}-1],N_{\omega}=31$,
as calculated with a 7-bath exact diagonalisation (ED) solver, by learning the
information from truncated 5-bath ED solutions. This is carried out though
fine-tuning the Transformer model parameters as listed below. Unless stated
otherwise, the models were trained with the full database as shown in fig. A4.
The samples of the database were split such that 80% of the data was allocated
for training, and 20% for validation and the many-body problem were solved at
default values of inverse temperature $\beta$ of 10, 50, and 100.

Model Default Settings:

Following the details of the main text, the positional encoding of the
Transformer network is replaced by the concatenation of Matsubara frequency.
The application of the Gaussian Error Linear Unit (GELU) activation function
has been incorporated into the model. In the following sections, we also
analyse other extensions of this idea of positional encoding via Matsubara
frequency. Moreover, in relation to the masks of the decoder, the look-forward
(LF) and look-backward (LB) masks are limited to five time steps before and
after a particular sequence entry respectively. We utilised a batch size of 8
and a learning rate of 1e-7, with the Adam optimiser employed for training.
Each task was performed three times, from which the metrics corresponding to
the best performing model are reported.

Table A1: Transformer model performance test. Unlisted value are identical to those of base value. N is the number of encoder and decoder blocks within the Transformer. dmodel denotes the dimensions of the multilayer perceptron blocks (MLP) and self-attention layers in the model. h specifies the number of heads. dk/v is the dimensions of the heads in the multi-head self-attention block. The term dff refers to the dimensions of the feedforward layers in the Transformer. In the mask column, LF and LB are abbreviations for look-forward and look-backward, respectively. In the input transformation column, N denotes normalisation, and Z represents standardisation. | N | dmodel | dff | h | dk/v | mask | PE |  | Inputs
---
Transform
dropout | Train-Step |  | MSEβ100
---
| MSEβ50
---
| MSEβ10
---
(Val) ($10^{-6}$)
Base | 4 | 256 | 1024 | 32 | 8 | LF5 LB5 |  | Matsubara
---
Frequency
N/A | 0. | 350K | 22.21 | 7.57 | 4.74
(a.1) |  |  |  | 4 | 64 |  |  |  |  |  | 37.31 | 8.13 | 5.03
(a.2) |  | 128 | 512 | 16 |  |  |  |  |  |  | 31.18 | 14.37 | 6.02
(a.2) | 2 | 64 | 128 | 8 |  |  |  |  |  |  | 46.30 | 21.39 | 12.99
|  |  |  |  |  |  | Learned PE (add) |  |  |  | 2,095.49 | 660.43 | 756.03
(b) |  |  |  |  |  |  | Learned PE (cat) |  |  |  | 57,474.60 | 36,406.33 | 13,774.82
|  |  |  |  |  |  | Cosine |  |  |  | 5,597.37 | 7,055.71 | 15,538.61
|  |  |  |  |  | LF1 LB1 |  |  |  |  | 34.33 | 8.10 | 5.25
(c) |  |  |  |  |  | LF3 LB3 |  |  |  |  | 28.95 | 8.62 | 5.49
|  |  |  |  |  | LF10 LB10 |  |  |  |  | 27.62 | 8.36 | 6.06
|  |  |  |  |  |  |  |  | 0.01 |  | 53.29 | N/A | N/A
(d) |  |  |  |  |  |  |  |  | 0.05 |  | 214.91 | N/A | N/A
|  |  |  |  |  |  |  |  | 0.1 |  | 581.48 | N/A | N/A
(e) |  |  |  |  |  |  |  | N |  |  | 6.30 | 2.70 | 3.98
|  |  |  |  |  |  |  | Z |  |  | 300.61 | 189.57 | 242.42
(a.1) | 6 | 512 | 2048 | 64 |  |  |  |  |  |  | 19.77 | 7.93 | 5.21
Big | 6 | 512 | 2048 | 8 | 64 |  |  |  |  |  | 21.63 | 7.37 | 4.78

Tab. A1 (a): Model Size Test

We conducted a model size test experiment using the model default settings and
task mentioned above. In addition to the base model, alternative settings of
the Transformer size were tested. Each test was performed three times, and the
results with the lowest loss are presented.

Across different temperatures, the shallower and narrower Transformer model
(a.2) converges more slowly compared to models with larger dimensions. Models
with additional Transformer encoder and decoder layers, or increased
dimensions, tend to deliver superior performance. The dimension of the model’s
attention heads (a.1) has a more complex influence on model performance: For
the base model, larger head dimensions degrade the model’s performance.
However, for larger models, increased head dimensions reduce performance at
low temperatures but improve performance at high temperatures.

Tab. A1 (b): Positional Encoding Methods Test

We conducted a positional encoding experiment using the model default settings
and task mentioned above. In addition to the base model, alternative
positional encoding (PE) methods were explored. Four distinct positional
encoding approaches were evaluated: learned PE (cat), learned PE (add) and
concatenation of Matsubara frequencies. These methods involve either trained
or untrained parameters being concatenated or added to the truncated inputs
$\mathrm{Im}[G(i\omega_{n})^{trunc}_{l}]$. In contrast, cosine positional
encoding was concatenated to the encoded
$\mathrm{Im}[G(i\omega_{n})^{trunc}_{l}]$. Each test was performed three
times, and the results with lowest loss are presented. Overall, the
concatenation of Matsubara frequency was found to be the optimal method across
all values of beta.

Tab. A1 (c): Mask Shape Test

We conducted a mask shape test experiment using the model default settings and
task mentioned above. In addition to the base model, alternative settings of
mask were tested. Four sets of values are investigated: 1, 3, 5, and 10
corresponding to the quantity of unmasked (zero) diagonal matrix lines, with
the rest of the matrix entries as negative infinity. Equal steps are taken for
both look forward and look backward. In the Transformer’s decoder-encoder
self-attention layer mask, both look-forward and look-backward are employed.
However, only look-backward is applied in the decoder input self-attention
layer. Each initialised model is then trained using these distinct mask shapes
and the results with lowest loss are presented.

Across different temperatures, neither a small (LF1LB1) nor a large (LF10LB10)
scope of unmasked inputs achieved optimal performance. At a beta of 100,
increasing the unmasked area from LF1LB1 to LF5LB5 improved model performance,
but with LF10LB10, the performance declined. At higher temperatures,
increasing the unmasked area initially decreased model performance. However, a
performance maximum appears around a masked size of (LF5LB5) for all
temperatures. This suggests that overall, a moderate size mask of LF5LB5
should be adopted for the Transformer model.

Tab. A1 (d): Dropout Test

We conducted a dropout test using the model default settings and task
mentioned above. In addition to the base model, three dropout values, 0.01,
0.05 and 0.1 are tested in input layer and the Transformer encoder and decoder
layers. In this test, only the case of $\beta$=100 is considered, as lower
temperatures tend to be more challenging to fit for such tasks. Three separate
models are randomly initialised to minimise noise arising from the initial
configuration. Each initialised model is subsequently trained using varying
dropout rate. Overall, across all dropout values tested in various positions
within the model, excluding dropout leads to a significant performance
improvement compared to including dropout.

Tab. A1 (e): Green’s function transform

We conducted a inputs transform tests using the model default settings and
task mentioned above. In addition to the base model, two choices of inputs
transform were tested. There are several choices of transformation factors for
the target since it is a 7-bath SIAM solved by ED, while the inputs are
truncated solutions with 5 bath sites. We use transformation factors derived
from the target, which is the ED of the 7-bath system. It should be noted that
the input Green’s function is not transformed as the name suggests. However,
the output of the Transformer model is either normalised or standardised. Each
test was performed three times, and the results with lowest loss are
presented.

In summary, across different temperatures, the performance of the model is
negatively impacted by standardizing the Green’s function. As the temperature
increases, models using normalized Green’s functions show better performance
compared to those using the raw physical values of the Green’s functions.

#### Batch LR

We conducted a batch size and learning rate test using the base model and task
mentioned above. In addition to the model default settings, we tested three
learning rates, 1e-3, 1e-5, and 1e-7, along with four batch sizes, 4, 8, 16,
and 32. In this test, only the case of $\beta$=100 is considered, as lower
temperatures tend to be more challenging to fit for such tasks. Three separate
models are randomly initialised to minimise noise arising from the initial
configuration. Each initialised model is subsequently trained using varying
batch sizes and learning rates, and the average results are illustrated in
Figure A1.

Figure A1: Performance of the Transformer network as a function of batch size
and learning rate at an inverse temperature $\beta$ value of 100. Dashed lines
represent training loss, while solid lines represent validation loss.

With a high learning rate of 1e-3, the loss function converges to a relatively
high, yet similar value across all batch sizes. When the learning rate is
decreased to 1e-5, a lower loss is achieved across all batch sizes, however,
the impact of varying batch size on performance is not distinctly clear. At a
learning rate of 1e-7, the model can be trained effectively. However, it is
noteworthy that training the model becomes more difficult with larger batch
sizes, although they eventually converge to similar values. While larger batch
sizes expedite the training process on a per iteration basis, they cause a
slower decrease in the loss function. Consequently, a mid-range batch size is
recommended for achieving an optimal balance between training efficiency and
model performance.

###  A.2 Database Size

We conducted a database size test using the model default settings and base
model. In addition to the task mentioned above, three size of training sets
were tested. In this test, only the case of $\beta$=100 is considered, as
lower temperatures tend to be more challenging to fit for such tasks. Three
separate models are randomly initialised to minimise noise arising from the
initial configuration. Each initialised model is subsequently trained using
varying settings, and the average results are illustrated in Figure A2.

Figure A2: Model performance as a function of database size at an inverse
temperature value of 100. Database size indicates the number of sum of
training and validation samples of the outputs, of which 80 percent was used
for training and 20 percent for validation. The dashed lines represent the
training loss, while the solid lines represent the validation loss.

At a beta of 100, increasing the number of training samples does not appear to
lower the validation loss for this particular task. This could potentially be
due to the saturation of unique features in the larger dataset. Hence, in a
larger database, similar samples would be trained more frequently within a
single epoch compared to a smaller database, which doesn’t necessarily
contribute to a further decrease in validation loss.

###  A.3 Multimodel

In this section, we explore various methods to incorporate Anderson impurity
model (AIM) parameters into the Transformer model, using the tests and model
default settings previously mentioned. We categorise the AIM parameters into
hybridisation parameters and impurity parameters. For the impurity parameters,
a straightforward approach of using a MLP is adopted. However, for the
hybridisation parameters, we implement a more complex configuration: two row-
wise MLPs connected by mean pooling. This approach is designed to maintain the
permutation symmetry properties associated with the bath sites.

In addition to model size, there are several options for incorporating encoded
parameters into the Transformer model. These options revolve around how to
encode these parameters and where to incorporate them. Regarding location, we
tested two scenarios: one where the encoded parameters are concatenated with
the positional encoder, and another where they are concatenated to the output
of the Transformer encoder. Furthermore, in the task of recovering a 7-bath
SIAM Hamiltonian using a 5-bath solution, three Hamiltonians need to be
encoded. Consequently, we also evaluated various pooling methods to enhance
model performance.

Table A2: Parameter encoder model performance test. Unlisted value are identical to those of base value. The base model corresponds to the model in (Tab. A1). ’N base’ stands for Normalised Base, which is identical to the model described in (Tab. A1 (d)). In this test, the Transformer model size is also set to the base size. "ParamConcatPosition" is an abbreviation for the encoded parameter concatenation position in the Transformer. In this column, "PE" represents the position where positional encoding is included in the inputs. "KV" refers to the self-attention layer in the Transformer decoder (base model), where KV is sourced from the Transformer encoder. The "Pool method" refers to the technique employed to facilitate interaction between processed hybridisation inputs and target hybridisation before they are directed into the Transformer decoder. | NHyb | NImp | dmodel | dout |  | Inputs
---
Trasnform
| Param
---
Concat
Position
| Pool
---
method
Train-Step |  | MSEβ100
---
| MSEβ50
---
| MSEβ10
---
(Val) ($10^{-6}$)
Base | N/A | N/A | 256 | N/A | N/A | N/A | N/A | 350K | 22.1 | 7.57 | 4.74
| 1 | 1 |  | 32 |  | PE |  |  | 5,949.96 | 941.17 | 63.57
(a) | 2 | 2 |  | 64 |  | PE |  |  | 5,034.78 | 2386.36 | 581.99
| 4 | 4 |  | 128 |  | PE |  |  | 20,781.07 | 1670.32 | 182.43
| 1 | 1 |  | 32 | N | KV | Mean |  | 2.99 | 2.80 | 2.98
(b) | 2 | 2 |  | 64 | N | KV | Mean |  | 7.64 | 2.93 | 3.14
| 4 | 4 |  | 128 | N | KV | Mean |  | 8.47 | 3.04 | 4.43
| 1 |  |  | 32 | N | KV | Mean |  | 8.35 | 2.63 | 3.99
(c) | 2 |  |  | 64 | N | KV | Mean |  | 7.91 | 3.12 | 4.08
| 4 |  |  | 128 | N | KV | Mean |  | 7.82 | 3.18 | 4.28
|  | 1 |  | 32 | N | KV |  |  | 6.56 | 2.71 | 2.94
(d) |  | 2 |  | 64 | N | KV |  |  | 8.37 | 3.0 | 3.69
|  | 4 |  | 128 | N | KV |  |  | 7.11 | 2.91 | 4.45
| 2 | 2 |  | 64 | N | KV | Cat |  | 7.47 | 3.55 | 3.84
(e) | 2 | 2 |  | 64 | N | KV | Minus |  | 6.58 | 3.44 | 3.78
| 2 | 2 |  | 64 | N | KV | add |  | 9.10 | 3.32 | 3.90
N base |  |  |  |  | N |  |  |  | 6.30 | 2.70 | 3.98

Tab. A2 (a): Integrating Encoded AIM Parameters with Positional Encoder

We conducted a model size test for the parameter encoder using the model
default settings and task mentioned above. In addition to the base model,
encoded parameters were concatenated alongside positional encoding. Three
model sizes were tested. Meanwhile, mean pooling was utilised to handle
multiple hybridisation inputs. Each test was performed three times, and the
results with lowest loss are presented. Overall, concatenating Hamiltonian
parameters with Positional Encoding (PE) to the Transformer led to a reduction
in the Transformer’s performance across all tested temperature.

Tab. A2 (b): Integrating Encoded AIM Parameters in Decoder’s Self-Attention
Layer

We conducted a model size test for the parameter encoder using the model
default settings and task mentioned above. In addition to the base model,
encoded parameters were concatenated the Transformer decoder. Three model
sizes were tested in each case. Meanwhile, mean pooling was utilised to handle
multiple inputs of hybridisation. Each test was performed three times, and the
results with the lowest loss are presented. Overall, one layer of the
parameter encoder achieved the optimal results.

Tab. A2 (c,d): Ablation Study of Multi-Model in Decoder’s Self-Attention Layer

In addition to (Tab. A2 (b)), we investigated how separately encoding the
impurity parameters and hybridisation parameters impacts the performance of
the Transformer across all temperature ranges.

Larger model sizes for the impurity parameter encoder resulted in poorer
performance. The situation for the hybridisation parameter encoder was more
complex. At lower temperatures, larger models tended to have better
performance, while at higher temperatures, larger models performed worse.
Interestingly, when only the impurity or hybridisation parameter encoder was
used, the model performance was sub-optimal (worse than ’N base’). However,
when both were used, the model performance improved at beta of 100.

Tab. A2 (e): Multi-hybridisation Pooling Methods Test

We conducted pooling method test for the parameter encoder using the model
default settings and task mentioned above. In addition to the previous model,
different pooling methods were tested to handle multiple inputs of
hybridisation. Each test was performed three times, and the results with the
lowest loss are presented. Overall, subtracting different hybridisations was
found to be the optimal method for low temperatures, and meaning pooling have
better performance at high temperatures.

###  A.4 Inputs test

In this section we explore changes in model performance when taking different
forms of input at different temperatures. Training settings followed the
default setting mentioned above but the task inputs were changed to different
schemes, where different truncated bath size and different approximation
solutions were tested.

Table A3: NHyb/Imp refers to the number of layers in the parameter encoder, where the dimensions of the parameter encoder equal the number of layers multiplied by 32. For models implementing the parameter encoder, the outputs of the parameter encoder are concatenated with the Transformer encoder output before being fed into the Transformer decoder’s encoder layer, as shown in Table A2 (b). Gin represents the source of the inputs directed into the Transformer model and NG stands for number of source inputs.  | NHyb/Imp | dmodel | Gin | NG |  | Inputs
---
Transform
Train-Step |  | MSEβ100
---
| MSEβ50
---
| MSEβ10
---
(Val) ($10^{-6}$)
Base | N/A | 256 | bath size 5 | 3 | N/A | 350K | 22.1 | 7.57 | 4.74
| 2 |  |  |  | N |  | 7.64 | 2.94 | 3.14
(a) | 2 |  | bath size 4 |  | N |  | 9.98 | 2.81 | 3.32
| 2 |  | bath size 3 |  | N |  | 10.97 | 3.10 | 3.24
|  |  | IPT & HI | 2 |  |  | 18.32 | 6.29 | 4.73
| 2 |  | HI | 1 | N |  | 7.75 | 2.02 | 2.28
(c) | 2 |  | IPT | 1 | N |  | 4.98 | 1.82 | 2.42
| 2 |  | IPT & HI | 2 | N |  | 5.62 | 1.82 | 2.37

Tab. A3 (a): Truncated Bath Size Test

We conducted tests with varying truncated bath sizes, using the model default
settings mentioned above. This refers to the number of baths extracted from
the target bath. Both AIM parameters and the imaginary part of the Green’s
function were used as inputs. Generally, a smaller truncated bath size tends
to contain less information about the entire system. Each test was performed
three times, and the results with the lowest loss are presented. As expected,
the Transformer model tends to perform better with larger bath sizes.

Tab. A3 (b): Transformer with Approximation Solver Test

We conducted inputs test with different inputs using model default settings
mentioned above. We take both AIM parameters and imaginary part of Green’s
function as inputs. Each test was performed three times, and the results with
the lowest loss are presented.

At low temperatures, the model that used IPT as inputs outperformed the model
that used HI inputs. Moreover, the model that incorporated both IPT and HI
(hybrid) solvers demonstrated accuracy comparable to the best-performing model
at moderate temperatures. At other temperatures, its performance was
intermediate between models that used only one of the hybrid solvers.

##  Appendix B Database Sampling

Here we present the distribution of the datasets used for training and
validation. The dataset is collected with random sampling of 1 eV of semi-
circular DOS (Eq. 9) at half-filling. Density and quasi-particle weight
distribution are found by ED of 7 bath sites. $U$ is randomly sampled from 0
to 10 eV. The impurity level $\epsilon_{d}$ is kept at $-U/2$. THe on-site
bath level, $\epsilon_{p}$ is sampled between -5 to 5 eV, and the hopping term
$V_{p}$ changes by temperature. The results of such sampling did not reproduce
perfect half-filled solutions for all the SIAM samples, and as temperature
decrease more samples deviate from being exactly half-filled.

Figure A3: Green’s function found by different solver by given SIAM parameters
at different temperature. n is the Matsubara frequency index. Here, the
distribution of first 8 $G(i\omega_{n})$ points are presented due to their
significance in dtermining the low energy properties of the solution.
Distributions at different frequencies are shown as violin plots which
indicate their density distributions. A white dot is present within each
distribution to denote the median value. While the black boxes illustrate the
interquartille range(IQR), and lastly, black lines represent $1.5\times$ the
interquartile range.

Figure A4: Database of parameters are sampled from a maximum radius of 1 eV
from semicircular Green’s function, Y axis is the count of the samples. The
symbols $U$ and $\epsilon_{d}$ refer to the impurity Hubbard $U$ term and on-
site energy, respectively. On the other hand, $\epsilon_{p}$ and $V_{p}$
denote the on-site energy and hopping amplitude, respectively. a) b) c) are
sampled from different temperatures.