# Pitfalls of Exchange-Correlation Functionals in Descriptions of Magnetism:
Cautionary Tale of the FeRh Alloy

Shishir Kumar Pandey<EMAIL_ADDRESS>Saikat Debnath  Zhanghao
Zhouyin  Qiangqiang Gu<EMAIL_ADDRESS>
###### Abstract

The magnetic ground state of FeRh is highly sensitive towards the lattice
constant. This, in addition to partially filled d-shells of Fe and Rh, posed a
significant challenge for Density Functional Theory (DFT) calculations in the
past. Here, we have investigated the performance of various exchange-
correlation (XC) functionals within the DFT formalism for this challenging
binary alloy. We have employed Local Spin Density Approximation (LSDA),
various Generalized Gradient Approximations (GGAs), and newly developed
Strongly Constrained and Appropriately Normed (SCAN) meta-GGA functional. Our
results show the limitations of any single functional in capturing the
intricate interplay of structural, electronic, and magnetic properties in
FeRh. While SCAN can accurately describe some magnetic features and phonon
dispersion, it significantly overestimates the Fe-Fe magnetic interactions,
leading to an unreasonable magnetic ordering temperature. Conversely, the
Perdew-Burke-Ernzerhof (PBE) GGA exhibits the opposite behavior. These
findings highlight the challenges in simulating materials with partially
filled $d$-shells using DFT, underscoring the crucial need for developing a
versatile XC functional that can effectively account for the multifaceted
nature of such systems.

###### keywords:

Magnetostructural transition , DFT , Phonon , Monte-Carlo simulations , Binary
alloy

[first]organization=AI for Science Institute,city= Beijing, country=China
[second]organization=M. V. College,city=Buxar, state=Bihar, country=India
[third]organization=College of Intelligence and Computing, Tianjin
University,city=Tianjin, postcode=300350, state=Bihar, country=China
[fourth]organization=School of Mathematical Science, Peking University,city=
Beijing, postcode=100871, country=China

##  1 Introduction

Transition metal compounds have become a focal point of intense research due
to their remarkable and exotic properties. Some examples include various forms
of magnetism Middey et al. (2012); Mahana et al. (2020); Gu et al. (2023);
Pandey et al. (2023, 2022); Pandey and Feng (2022); Pandey et al. (2021); Zhou
et al. (2021) and superconductivity Liu et al. (2022); Lee et al. (2006) and
different kinds of phase transitions driven by external perturbations such as
doping Pandey et al. (2017); Lee et al. (2006) and temperature. Magnetic
transitions in these compounds are typically second-order, characterized by a
continuous evolution from one magnetic state to another without any
discontinuous changes in structural parameters. While first-order transitions
can occur, their presence remains relatively rare. In these magnetostructural
first-order phase transitions, a simultaneous change in the magnetic and
structural phase of the material with external stimuli like temperature,
magnetic fields, or strain originates from a strong coupling between lattice
and magnetic degrees of freedom. The purported technological potential of
these first-order magnetic transitions in magnetic recording Thiele et al.
(2003) magnetocalorics Liu et al. (2016), spintronics Wang et al. (2020),
magnetorestrictions O’Handley et al. (2000), heat-assisted magnetic recording
Challener et al. (2009), antiferromagnetic (AFM) spintronics Yan et al.
(2020); Feng et al. (2019), and room-temperature AFM memory resistor Marti et
al. (2014) makes them even more fascinating.

One particularly interesting example of such magnetostructural phase
transition occurs for a transition metal binary alloy, FeRh Fallot and Hocart
(1939); Kouvel and Hartelius (2004). BCC-based (CsCl-type) ordered phase of
FeRh, upon lowering the temperature, first goes through a paramagnetic to
ferromagnetic (FM) second-order phase transition at $\sim$ $T_{\text{C}}$ =
670 K. Further lowering of temperature then leads to an FM to AFM transition
at $\sim$ $T_{\text{N}}$ = 370 K. A sudden increase in resistivity and $\sim$
1% volume collapse with lowering of temperature in AFM phase reveals first-
order nature of the AFM-FM transition Kouvel and Hartelius (2004). The
reversible nature of this phase transition has also been investigated with the
application of an external magnetic field both in bulk and thin films Kouvel
(2004); Maat et al. (2005). What makes this material particularly interesting
in the context of device applications is its AFM-FM transition being close to
the room temperature.

Efforts to investigate phase transitions in FeRh have yielded conflicting
perspectives, with some researchers suggesting simultaneous structural and
magnetic changes Radu et al. (2010); Derlet (2012) while others contend that
the magnetic transition precedes unit cell expansion Gruner et al. (2003a); Ju
et al. (2004); Mryasov (2005); Gu and Antropov (2005a); Quirin et al. (2012).
To gain a deeper understanding of such scenarios, Density Functional Theory
(DFT)-based calculations are often employed for their precision and
reliability. However, in the case of FeRh, prior DFT-based calculations
indicate an AFM-FM transition at a significantly larger volume Yamada et al.
(2006); Gu and Antropov (2005b); Moruzzi and Marcus (1992); Gruner et al.
(2003b); Mryasov (2005); Aschauer et al. (2016) compared to the experimentally
observed 1 % change. Despite the magnetic moments of Fe/Rh ions, obtained in
these calculations, aligning well with experimental data Gruner et al.
(2003a); Gu and Antropov (2005a); Sandratskii and Mavropoulos (2011);
Kudrnovský et al. (2015); Yamada et al. (2006); Moruzzi and Marcus (1992);
Aschauer et al. (2016); Wolloch et al. (2016), it is surprising that DFT
calculations fail to capture the AFM-FM transition. In instances where DFT-
derived parameters are used for modeling, there is a risk of reaching
inaccurate conclusions. Therefore, a reassessment to identify the issues with
DFT calculations in this system is warranted.

Central to these DFT calculations are exchange-correlation (XC) functionals
such as local spin density approximation (LSDA) or various generalized
gradient approximations (GGAs). It has become nearly established that these
functionals inadequately treat the correlated $d$ orbitals of transition
metals, leading to issues in describing materials like strongly correlated
superconductors Pickett (1989) and nickel oxide Sawatzky and Allen (1984).
Such erroneous description arises mainly due to self-interaction error Svane
(1992) often causing inconsistency between theory and experiments. In this
context, alternative approaches like DFT+$U$ Anisimov et al. (1991, 1997) and
semi-local meta-GGA density functionals such as strongly constrained and
appropriately normed (SCAN) functional Sun et al. (2015); Zhang et al. (2018)
have demonstrated notable success in describing the properties of various
materials Sun et al. (2016); Tran et al. (2016a); Isaacs and Wolverton (2018);
Lane et al. (2018). SCAN, in particular, excels in capturing the XC energy of
atoms and is computationally efficient compared to accurate hybrid functionals
Tran et al. (2016b). Therefore, it is worthwhile to investigate whether the
previous discrepancies of DFT calculations for FeRh can be mitigated by
employing different XC functionals. If so, understanding the extent of the
differences in results across various functionals is crucial. It’s noteworthy
that this issue has also been recently highlighted in a comprehensive DFT
study by Bosoni et al. Bosoni et al. (2023).

In this study, we conducted a comparative analysis of results obtained from
the meta-GGA SCAN functional, LDA, and various GGA functionals. We find that
SCAN provides a better description of volume expansion accompanying AFM-FM
transition and magnitude of magnetic moments of Fe/Rh atoms in FeRh than any
other functional. While the phonon dispersions from SCAN and revised-PBE
(RPBE) exhibit close agreement, there are qualitative differences in the spin-
polarized band structures between the two cases. We attribute this disparity
to the larger exchange splitting observed in the band structure calculations
using SCAN. However, magnetic interactions between Fe atoms ($J^{\text{Fe-
Fe}}$) in the AFM ground state, calculated using the Frozen Magnon approach,
are highly overestimated resulting in unreasonably large magnetic ordering
temperature in our Monte-Carlo simulations. In this scenario, the PBE
functional performs reasonably well. Our results emphasize that the
intricately intertwined structural, electronic, and magnetic properties of
FeRh cannot be accurately described by a single functional. This underscores
the need for further development of a versatile exchange-correlation
functional that can effectively capture the multifaceted nature of such
materials.

##  2 Methodology

###  2.1 Total energy and phonon calculations

Density-functional theory calculations are performed using projector-augmented
wave method Kresse and Joubert (1999); Blöchl (1994), implemented within
Vienna $ab$ $initio$ simulation package (VASP) Kresse and Furthmüller (1996).
Total energy calculations are done with LDA, GGA-based functionals namely PBE
Perdew et al. (1996), revised PBE (RPBE) Hammer et al. (1999), revised PBE for
solids (PBEsol) Perdew et al. (2008), and meta-GGA functional namely SCAN Sun
et al. (2015). In GGA pseudopotentials of Fe and Rh, 3$d^{7}$4$s^{1}$ and
4$p^{6}$4$d^{8}$5$s^{1}$ are treated as valence states. We start with a
sixteen-atom conventional magnetic unit cell of cubic crystal structure (space
group: $P\bar{m}3m$, no: 221) of FeRh made from the two-atom primitive cell
with lattice constant 2.99 Å. Given the importance of spin-lattice coupling in
this material, full optimization of crystal structure was performed with each
of the pseudopotentials after imposing the G-type AFM ground state (every Fe-
Fe first neighbors are antiferromagnetically coupled) observed experimentally
at low temperatures. Energy and Hellmann-Feynman force convergence criteria of
10-7 eV and 10-4 eV/Å respectively are considered during the optimization with
the aim of phonon calculations in the next step. Plane-wave cutoff energy 550
eV and 9 $\times$ 9 $\times$ 9 $\Gamma$-centered $k$-mesh, Gaussian smearing
width of 0.05 eV are considered in our calculations. The results obtained are
well converged with respect to these parameters.

Structural optimizations with different XC-functional lead to quite small
changes in the lattice constants except for the LDA as shown in Table 1. The
total energy of the FM state and phonon dispersion for both AFM and FM states
are calculated with PBE, RPBE, and SCAN functionals with respective optimized
structures. We employed the finite displacement method in phonon calculations
and the post-processing is done using Phonopy Togo et al. (2023); Togo (2023)
software.

###  2.2 Estimation of magnetic interactions and Monte Carlo simulations

To quantify the role of XC functionals on the magnetic ground state, exchange
interactions between Fe atoms in the AFM ground state are estimated with the
Frozen-magnon approach (FMA) for both PBE and SCAN functionals. In this
method, the spin spiral state is represented with propagation vector (q)
defined in the reciprocal space of the lattice. Each spin spiral state is
equivalent to a magnetic configuration characterized by,

$\bm{e_{i}}=\left(\sin{\theta}\cos{(\bm{q}\cdot\bm{R}_{i})},\sin{\theta}\sin{(\bm{q}\cdot\bm{R}_{i})},\cos{\theta}\right)$

where $\bm{R}_{i}$ = $\bm{R}+\bm{\tau}_{\nu}$ = $\bm{R}_{\nu+\bm{R}}$, and
$\bm{R}$, $\bm{\tau}_{\nu}$ are the lattice and the basis vectors
respectively. $\theta$ is the polar angle made by the spin with the $z$ axis.
Using noncollinear magnetism formulation implemented within the VASP package,
energy corresponding to each spin-spiral state $E(\bm{q})$ is calculated with
16 atom AFM unit cell. Assuming a localized spin at the Fe site, the results
are then mapped onto an extended classical Heisenberg Hamiltonian,

$H=-\sum_{i\neq j}J_{ij}\bm{e_{i}}\cdot\bm{e_{j}}$ (1)

where $J_{ij}$ is the exchange interaction between the Fe spins at sites $i$
and $j$, $\bm{e_{i}}$ and $\bm{e_{j}}$ are the unit vectors at these two
sites. Within this Heisenberg model, the energy of any frozen-magnon
configurations represented by $\bm{q}$ can be given by,

$\ E(\theta,\bm{q})=E_{0}(\theta)-\sin^{2}(\theta)J(\bm{q})$

For simplicity, in our calculations, we restricted the spins to lie in the
$xy$ plane. Fourier transform of $J_{ij}$ in the above equation can be defined
as,

$J^{\mu\nu}(\bm{q})=\delta_{\mu\nu}J_{\mu\mu}-\sum_{\bm{R}}J_{\mu[\nu+\bm{R}]}e^{-i({\bm{\tau}_{\mu}-\bm{\tau}_{\nu}-\bm{R}})\cdot\bm{q}}$

By taking the inverse Fourier transform, we can obtain, the real space
interaction parameters as,

$J_{ij}=\frac{1}{N}\sum_{\bm{q}}J^{\mu_{i}\nu_{j}}(\bm{q})e^{i({\bm{\mu}_{i}-\bm{\nu}_{j}-\bm{R}})\cdot\bm{q}}$

The magnitude of spins is absorbed in the obtained $J$’s which are then used
in classical Monte Carlo (MC) simulations to estimate the magnetic ordering
temperature. In our in-house MC code, unit vectors on a simple cubic lattice
of 16 $\times$ 16 $\times$ 16 are considered, and starting from a random spin
configuration, the system is brought into a thermal equilibrium within 2
$\times$ 105 MC steps at each temperature. After the thermal equilibrium is
achieved, we calculated the magnetization of each sublattice which was further
used to determine the magnetic ordering temperature. Our results are well-
converged with respect to both the lattice size and the number of MC steps.

##  3 Results

Our analysis begins with the optimization of the crystal structure. For
materials exhibiting magnetostructural transitions, a strong lattice-spin
coupling is expected. In the context of materials undergoing magnetostructural
transitions, a robust lattice-spin coupling is anticipated. In such scenarios,
a comprehensive structural optimization proves invaluable for gaining
insights. Utilizing various exchange-correlation (XC) functionals, we
systematically optimize the crystal structure while considering the
antiferromagnetic (AFM) state, as detailed in the methodology section.

Table 1 presents the optimized lattice constants, their changes observed
during optimization, and the magnetic moments of Fe atoms. Throughout these
calculations, the crystal structure maintains its cubic symmetry. Consistent
with findings from prior Density Functional Theory (DFT)-based studies, the
magnetic moments of Rh atoms persist at zero in the AFM ground state.

Table 1: A comparison of the lattice constants, their variations, and the magnetic moments of Fe after optimizing the crystal structure using different XC functionals for the AFM state. Additionally, the difference in total energy (per formula unit) with the ferromagnetic (FM) state is provided. Notably, the results obtained with the SCAN functional stand out distinctly from those obtained with other XC-functionals. XC-functional |  |  $a$ (Å) |  $\delta a$ (%) | $\mu_{B}^{\text{Fe}}$ | E${}_{\text{FM}}$-E${}_{\text{AFM}}$
---|---|---|---|---|---
|  |  |  |  | (meV)
LDA |  | 5.832 |  -2.3 | 2.810 | 92
PBE |  | 5.979 | +0.2 | 3.078 | 66
RPBE |  | 6.031 | +1.0 | 3.145 | 46
PBEsol |  | 5.984 | +0.1 | 3.074 | 71
SCAN |  | 5.921 |  -0.8 | 3.341 | 12

With the exception of LDA, the variations in lattice constants are relatively
small across different XC functionals. However, the behavior of SCAN stands
out distinctly from other GGA-based XC functionals on three key parameters.
Firstly, it is commonly known that GGAs tend to overestimate lattice constants
due to the underestimation of electron-electron correlations (Hartree and
Hartree-Fock electron-electron correlations), while LDA exhibits the opposite
trend. However, the meta-GGA SCAN demonstrates a unique behavior, leading to a
decrease in lattice constant by approximately 0.8%, contrasting with other GGA
functionals. Secondly, a closer examination of the Fe magnetic moments in
Table 1 reveals that $\mu^{\text{Fe}}_{B}$ is significantly enhanced
(approximately 8%) in the case of SCAN compared to PBE. This value is notably
closer to the experimentally observed 3.3 $\mu_{B}$ from the only neutron
diffraction study in Ref. Shirane et al. (1964). Thirdly, the energy
difference per formula unit with the FM state
(E${}_{\text{FM}}$-E${}_{\text{AFM}}$) in Table 1, calculated using optimized
structures of respective XC functionals, is several times smaller for SCAN
compared to other XC functionals. The AFM-FM transition temperature of 370 K
($\sim$ 32 meV) is slightly higher than the value
E${}_{\text{FM}}$-E${}_{\text{AFM}}$ = 12 meV estimated by SCAN. For other
functionals, this difference is higher than the transition temperature,
suggesting that SCAN might be closer to describing the AFM-FM transition in
FeRh. At this stage, we can assert that SCAN exhibits distinctive behavior for
our current system of interest, FeRh. However, it remains unclear whether its
performance is superior or inferior to other functionals and, in either case,
how the results differ at a quantitative level. Addressing these questions
requires further in-depth analysis, marking the next step in our ongoing
study.

Figure 1: Change in the magnetic ground state with the change of volume
calculated for optimized crystal structure with various XC functionals.
E${}_{\text{FM}}$-E${}_{\text{AFM}}$ infers AFM-FM transition. Notably, except
SCAN, all other functionals anticipate the transition occurring at a
substantially higher volume than the experimentally observed approximately 1 %
change.

As the next step in our analysis, we calculate the energy difference between
the FM and AFM states (E${}_{\text{FM}}$-E${}_{\text{AFM}}$) with the change
in volume using different XC-functionals. The corresponding plot is presented
in Fig. 1. Notably, the AFM-FM transition occurs at approximately 2 % volume
expansion for SCAN, while for PBE/PBEsol, it takes place after a $\sim$ 15 %
expansion. RPBE exhibits relatively better performance among GGAs, although
the transition still occurs after a substantial $\sim$ 10 % volume expansion.
Hence, at first glance, SCAN appears to better perform compared to other
functionals, particularly in describing the magnetostructural transition in
the FeRh system. Consistent with previous DFT-based studies, in the FM state,
Rh magnetic moments at the pristine volume are $\sim$ 1.061 $\mu_{B}$ while no
net Rh moment is observed in the AFM state. The appearance/disappearance of Rh
moments is believed to be attributed to hybridization among Fe and Rh atoms
Sandratskii and Mavropoulos (2011); Kudrnovský et al. (2015). The Fe moment
remains largely unchanged throughout the transition.

Figure 2: Phonon dispersion is computed for both the (a) AFM and (b) FM phases
utilizing the finite displacement method with PBE, RPBE, and SCAN functionals.
For the PBE case in (a), we employ two unit cells of sizes 2 $\times$ 2
$\times$ 2 and 4 $\times$ 4 $\times$ 2 to ensure convergence concerning unit
cell size.

To further assess the reliability of the SCAN functional, we evaluate the
stability of crystal structures by computing phonon dispersion. A similar
analysis is conducted for comparison purposes using PBE and RPBE functionals.
The resulting plots for the AFM and FM phases are presented in Fig. 2(a) and
Fig. 2(b) respectively. Additional details of the calculations are available
in the Methodology section. The phonon dispersion of the two phases $viz$ FM
and AFM, exhibits notable differences, particularly with the softening of
optical modes in the AFM phase. This observation underscores the pronounced
influence of spin-lattice coupling in the FeRh system. In the FM phase, all
three functionals-PBE, RPBE, and SCAN—consistently predict the dynamical
stability of the crystal lattice. Interestingly, soft phonon modes are
observed with the PBE functional at the M point for the AFM phase in Fig.
2(a), a feature absent in the case of RPBE and SCAN functionals. Similar soft
modes have been reported in prior DFT-based alculations Aschauer et al.
(2016); Wolloch et al. (2016). In support of this, Wolloch et. al. Wolloch et
al. (2016) argued that large Fe magnetic moments can dynamically stabilize the
crystal structure in the AFM phase. Both Aschauer et. al. Aschauer et al.
(2016) and Wolloch et. al. Wolloch et al. (2016) manipulated Fe magnetic
moments to reach this conclusion, either through the DFT + U approach or by
artificially increasing them. In our current study, both RPBE and SCAN
functionals (RPBE functional in Ref. Wolloch et al. (2016)) demonstrate
success in predicting the dynamically stable AFM phase of FeRh. However, only
SCAN appears to reasonably predict the AFM-FM transition with volume change.
This discrepancy prompts a natural question about the differences at
microscopic level from these two functionals in describing the properties of
the FeRh alloy.

Figure 3: Spin-polarized band structure plots for the FM phase are presented
with (a) PBE, (b) RPBE, and (c) SCAN functionals. We used two-atom primitive
unit cell in our calculations. Bands in the up and down spin channels are
depicted in red and green, respectively, with the Fermi level set to zero in
the plots. The two differently colored arrows around Fermi level in the plots
illustrate the exchange splitting of the top two bands, measuring 2.064 and
2.603 eV from PBE in (a), 2.105 and 2.699 eV from RPBE in (b), and 2.809 and
3.307 eV from SCAN in (c).

To address the aforementioned point, we conduct a spin-polarized band
structure analysis using PBE, RPBE, and SCAN functionals for the FM phase. In
this investigation, we consider the primitive cell consisting of two atoms to
provide a clear observation of the impact of various functionals on the
electronic structure. The corresponding plots are presented in Fig. 3. Two
significant observations arise from this analysis, shedding light on the
distinctive behavior of SCAN compared to the two GGA functionals. Firstly,
owing to its improved handling of delocalization errors, SCAN pushes the
occupied bands downward and the unoccupied bands upward relative to the Fermi
level (Fig. 3(c)), contrasting with the band structures obtained with PBE and
RPBE (which exhibit more or less the same behavior) as shown in Fig. 3(a) and
(b). This behavior of SCAN suggests a commendable treatment of electronic
correlation, in particular of the Fe/Rh $d$ orbitals contributing dominantly
(not explicitly shown) to the band structure plots in Fig. 3. Secondly, and
intricately connected with the first observation, SCAN reveals an augmented
exchange splitting at the Fermi level. This enhancement is highlighted by
arrows with two different colors for the top two bands in the plots. In the
case of PBE and RPBE functionals, the splitting—depicted by dark and light
blue arrows in Fig. 3(a),(b)-measures 2.064, 2.603 eV and 2.105, 2.699 eV,
respectively. In contrast, the splitting increases to 2.809 and 3.307 eV with
the SCAN functional in Fig. 3(c). The disparity in exchange splitting between
these GGA functionals and SCAN aligns closely with the error by which PBE/RPBE
exaggerates the AFM phase stability over the FM phase in Fig. 1. Consequently,
one can attribute the relatively superior description of the AFM-FM transition
in FeRh by the SCAN functional to its capacity to bring a substantial exchange
splitting in the electronic structure.

The seemingly promising findings regarding the performance of the SCAN
functional for the FeRh system, particularly concerning phonon dispersion and
the AFM-FM transition with volume, require further examination. To validate
the efficacy of the SCAN functional in accurately describing FeRh, magnetic
interactions obtained with it should be able to reproduce the experimentally
observed paramagnetic transition at $\sim$ 670 K. We undertake a thorough
investigation of this aspect in following two steps.

In the first step, the spin-spiral calculations as the starting point of
Frozen Magnon Approximation are performed. As explained in the Methodology
section, each spin spiral state characterized by $\bm{q}$ is equivalent to a
magnetic configuration with energy $E(\bm{q})$. Fig. 4 depicts the
stabilization energy ($E(\bm{q})$ \- $E(\bm{q}$ = 0) dispersions for various
noncollinear magnetic configurations along the $\Gamma-X-M-\Gamma-R-X$ path,
calculated using both PBE and SCAN XC-functionals. In this plot,
$\Gamma$-point ($\bm{q}$ = 0) represents the AFM state observed at low
temperatures. To model the system within a Heisenberg model, linear regression
was employed to extract Fe-Fe magnetic exchange interactions up to the fourth
nearest neighbor. Coefficient of determination ($R^{2}$) are found to be 0.99
and 0.98 for the PBE and SCAN XC-functionals respectively, indicating the
model’s ability to capture the energy dispersion accurately. This can further
be qualitatively verified by looking at the goodness of fitting.

Upon examining Fig. 4, two straightforward observations emerge. Firstly, from
both functionals, the FeRh system attains its lowest energy at the $\Gamma$
point, indicating that the AFM state is the ground state. This again is
consistent with experimental observation and highlight the fact that both
these functional correctly obtain the magnetic ground state. Secondly, while
the nature of the plots from PBE and SCAN functionals remains largely similar,
there is a significant discrepancy in the energy scale. The relative energies
[$E(\bm{q})$ \- $E(\bm{q}=0)$] are notably larger with SCAN compared to PBE.
One way to look at this point is that SCAN predicts the AFM state at low
tenperature to be robustly stabilized against any other competing magnetic
orders, particular against the FM state represented by $R$ point in Fig. 4.
However, from Table 1, one can see that E${}_{\text{FM}}$-E${}_{\text{AFM}}$ =
12 meV meaning FM state is lying in close vicinity to that of the AFM state.
This observation immediately prompts the question of whether their is also
discrepancies among magnetic interactions obtained from the two functionals
and, if so, which one of them aligns with the experimentally observed
paramagnetic transition temperature of $\sim$ 670 K. To address these
questions and assess the validity of SCAN and PBE for this challenging system,
extraction of magnetic interactions and subsequent calculations of the
magnetic ordering temperature are performed in the second step.

Figure 4: Spin-spiral calculations are performed for the low-temperature AFM
phase using PBE (orange-filled circles) and SCAN (green-filled circles) XC
functionals. In this representation, the $\Gamma$ point signifies the AFM
ground state. The obtained data is subjected to fitting with the Heisenberg
model given in Eq. (1), involving up to 4th Fe-Fe neighbors, using linear
regression. The results of the fitting are depicted in the plot with light-
green and blue empty diamond symbols for PBE and SCAN functionals,
respectively.

Hence, to elucidate the impact of XC functionals, we first extract Fe-Fe
magnetic interactions ($J$’s) up to the fourth nearest neighbor using the
inverse Fourier transform method within the Frozen Magnon approach, discussed
in the Methodology section. Subsequently, Monte Carlo simulations are employed
to estimate the magnetic ordering temperature. Fig. 5(a) depicts the extracted
interactions plotted against different nearest neighbor distances. Both PBE
and SCAN agree on the dominant antiferromagnetic first-neighbor interaction
($J_{1}$) at $\sim$ 2.96 $\AA$, consistent with the observed antiferromagnetic
ground state. However, their magnitudes differ significantly. SCAN predicts a
substantially larger $J_{1}$ of -151 meV compared to -30 meV predicted by PBE.
Interestingly, while smaller, the second-neighbor interaction ($J_{2}$) at
4.19 $\AA$ exhibits a contrasting behavior. SCAN predicts a ferromagnetic
$J_{2}$ of 7 meV, whereas PBE estimates it as antiferromagnetic with a value
of -2.6 meV. In the low-temperature AFM configuration, first and second-
neighbor couplings are expected to be AFM and FM, respectively. Therefore,
SCAN’s FM $J_{2}$ at first glance appears more consistent. The estimated
third-neighbor interactions $J_{3}$ at 5.13 $\AA$ are -8 and -9.5 meV, and the
fourth-neighbor interactions $J_{4}$ at 5.92 $\AA$ are 0 and 2.8 meV for SCAN
and PBE, respectively. These discrepancies highlight the significant
qualitative differences in magnetic interactions extracted using the two XC
functionals.

Figure 5: (a) Magnetic interactions between Fe atoms in the AFM ground state
are estimated using the Frozen-magnon approach, as explained in the
methodology section, with both PBE and SCAN functionals. The 1st neighbor
interaction is antiferromagnetic, consistent with the magnetic ground state.
(b) The variation of normalized magnetization with temperature is illustrated
in our Monte Carlo simulations. The interactions obtained with PBE in (a)
yield a magnetic ordering temperature $T_{\text{N}}$ $\sim$ 700 K, closely
matching an experimental estimation of 670 K. Conversely, the magnetic
interactions from the SCAN functional significantly overestimate it.

To check upon the reliability of the two set of interactions obtained above in
reproducing experimentally observed magnetic ordering temperature, we perform
Monte Carlo simulations considering the Heisenberg model on the simple cubic
lattice of Fe atoms (see methodology). Obtained variation in normalized as a
function of temperature is plotted in Fig. 5(b) for the two cases, $viz$ PBE
and SCAN. The estimated magnetic ordering temperature with $J$’s estimated
using PBE and SCAN functionals $\sim$ 710 K and 3240 K respectively. Though
this temperature is overestimated in both cases, the case of SCAN is in
complete disagreement with the experimentally observed value of $\sim$ 670 K.
It an order of magnitude overestimated when compared to PBE-obtained value
which is in a better agreement with experiment. This discrepancy can directly
be attributed to large $J_{1}$ obtained from SCAN in Fig. 5(a). As far the PBE
is concerned, one can tune $J$’s through the application of $U$ on Fe-$d$
states within the PBE+$U$ approach to achieve a closer agreement with
experimentally observed magnetic ordering temperature. However, our presented
results suffice to highlight the behavior of different XC functional in
describing structural, electronic, and magnetic properties of the FeRh system.

##  4 Conclusion

In conclusion, we have investigated the performance of various XC functionals
within the DFT formalism for the challenging binary alloy FeRh. Employing
LSDA, various GGAs, and newly developed SCAN meta-GGA functional, our results
show the limitations of any single functional in capturing the intricate
interplay of structural, electronic, and magnetic properties in FeRh. We have
shown that while SCAN can accurately describe some magnetic features and
phonon dispersion, it significantly overestimates the Fe-Fe magnetic
interactions, leading to an unreasonable magnetic ordering temperature.
Conversely, the PBE GGA exhibits the opposite behavior than SCAN. These
findings highlight the challenges in simulating materials with partially
filled $d$-shells using DFT, underscoring the crucial need for developing a
versatile XC functional that can effectively account for the multifaceted
nature of such systems.

## References

  * Anisimov et al. (1997) Anisimov, V.I., Aryasetiawan, F., Lichtenstein, A.I., 1997.  First-principles calculations of the electronic structure and spectra of strongly correlated systems: the LDA+$U$ method.  Journal of Physics: Condensed Matter 9, 767.  URL: https://dx.doi.org/10.1088/0953-8984/9/4/002, doi:10.1088/0953-8984/9/4/002.
  * Anisimov et al. (1991) Anisimov, V.I., Zaanen, J., Andersen, O.K., 1991.  Band theory and mott insulators: Hubbard u instead of stoner i.  Phys. Rev. B 44, 943–954.  URL: https://link.aps.org/doi/10.1103/PhysRevB.44.943, doi:10.1103/PhysRevB.44.943.
  * Aschauer et al. (2016) Aschauer, U., Braddell, R., Brechbühl, S.A., Derlet, P.M., Spaldin, N.A., 2016.  Strain-induced structural instability in FeRh.  Phys. Rev. B 94, 014109.  URL: https://link.aps.org/doi/10.1103/PhysRevB.94.014109, doi:10.1103/PhysRevB.94.014109.
  * Blöchl (1994) Blöchl, P.E., 1994.  Projector augmented-wave method.  Phys. Rev. B 50, 17953–17979.  doi:10.1103/PhysRevB.50.17953.
  * Bosoni et al. (2023) Bosoni, E., Beal, L., Bercx, M., Blaha, P., Blügel, S., Bröder, J., Callsen, M., Cottenier, S., Degomme, A., Dikan, V., Eimre, K., Flage-Larsen, E., Fornari, M., Garcia, A., Genovese, L., Giantomassi, M., Huber, S.P., Janssen, H., Kastlunger, G., Krack, M., Kresse, G., Kühne, T.D., Lejaeghere, K., Madsen, G.K.H., Marsman, M., Marzari, N., Michalicek, G., Mirhosseini, H., Müller, T.M.A., Petretto, G., Pickard, C.J., Poncé, S., Rignanese, G.M., Rubel, O., Ruh, T., Sluydts, M., Vanpoucke, D.E.P., Vijay, S., Wolloch, M., Wortmann, D., Yakutovich, A.V., Yu, J., Zadoks, A., Zhu, B., Pizzi, G., 2023.  How to verify the precision of density-functional-theory implementations via reproducible and universal workflows.  Nature Reviews Physics URL: http://dx.doi.org/10.1038/s42254-023-00655-3, doi:10.1038/s42254-023-00655-3.
  * Challener et al. (2009) Challener, W.A., Peng, C., Itagi, A.V., Karns, D., Peng, W., Peng, Y., Yang, X., Zhu, X., Gokemeijer, N.J., Hsia, Y.T., Ju, G., Rottmayer, R.E., Seigler, M.A., Gage, E.C., 2009.  Heat-assisted magnetic recording by a near-field transducer with efficient optical energy transfer.  Nature Photonics 3, 220–224.  URL: https://ui.adsabs.harvard.edu/abs/2009NaPho...3..220C, doi:10.1038/nphoton.2009.26.
  * Derlet (2012) Derlet, P.M., 2012.  Landau-Heisenberg hamiltonian model for FeRh.  Phys. Rev. B 85, 174431.  URL: https://link.aps.org/doi/10.1103/PhysRevB.85.174431, doi:10.1103/PhysRevB.85.174431.
  * Fallot and Hocart (1939) Fallot, M., Hocart, R., 1939.  Sur l’apparition du ferromagnétisme par élévation de température dans des alliages de fer et de rhodium.  Rev. Sci 77, 498.
  * Feng et al. (2019) Feng, Z., Yan, H., Liu, Z., 2019.  Electric-field control of magnetic order: From FeRh to topological antiferromagnetic spintronics.  Advanced Electronic Materials 5, 1800466.  URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/aelm.201800466, doi:https://doi.org/10.1002/aelm.201800466.
  * Gruner et al. (2003a) Gruner, M.E., Hoffmann, E., Entel, P., 2003a.  Instability of the rhodium magnetic moment as the origin of the metamagnetic phase transition in $\alpha-\mathrm{FeRh}$.  Phys. Rev. B 67, 064415.  URL: https://link.aps.org/doi/10.1103/PhysRevB.67.064415, doi:10.1103/PhysRevB.67.064415.
  * Gruner et al. (2003b) Gruner, M.E., Hoffmann, E., Entel, P., 2003b.  Instability of the rhodium magnetic moment as the origin of the metamagnetic phase transition in $\alpha-\mathrm{FeRh}$.  Phys. Rev. B 67, 064415.  URL: https://link.aps.org/doi/10.1103/PhysRevB.67.064415, doi:10.1103/PhysRevB.67.064415.
  * Gu et al. (2023) Gu, Q., Pandey, S.K., Lin, Y., 2023.  Computational exploration of a viable route to kitaev-quantum spin liquid phase in oscl3.  arXiv:2304.04257.
  * Gu and Antropov (2005a) Gu, R.Y., Antropov, V.P., 2005a.  Dominance of the spin-wave contribution to the magnetic phase transition in FeRh.  Phys. Rev. B 72, 012403.  URL: https://link.aps.org/doi/10.1103/PhysRevB.72.012403, doi:10.1103/PhysRevB.72.012403.
  * Gu and Antropov (2005b) Gu, R.Y., Antropov, V.P., 2005b.  Dominance of the spin-wave contribution to the magnetic phase transition in ferh.  Phys. Rev. B 72, 012403.  URL: https://link.aps.org/doi/10.1103/PhysRevB.72.012403, doi:10.1103/PhysRevB.72.012403.
  * Hammer et al. (1999) Hammer, B., Hansen, L.B., Nørskov, J.K., 1999.  Improved adsorption energetics within density-functional theory using revised perdew-burke-ernzerhof functionals.  Phys. Rev. B 59, 7413–7421.  URL: https://link.aps.org/doi/10.1103/PhysRevB.59.7413, doi:10.1103/PhysRevB.59.7413.
  * Isaacs and Wolverton (2018) Isaacs, E.B., Wolverton, C., 2018.  Performance of the strongly constrained and appropriately normed density functional for solid-state materials.  Phys. Rev. Mater. 2, 063801.  URL: https://link.aps.org/doi/10.1103/PhysRevMaterials.2.063801, doi:10.1103/PhysRevMaterials.2.063801.
  * Ju et al. (2004) Ju, G., Hohlfeld, J., Bergman, B., van de Veerdonk, R.J.M., Mryasov, O.N., Kim, J.Y., Wu, X., Weller, D., Koopmans, B., 2004.  Ultrafast generation of ferromagnetic order via a laser-induced phase transformation in FeRh thin films.  Phys. Rev. Lett. 93, 197403.  URL: https://link.aps.org/doi/10.1103/PhysRevLett.93.197403, doi:10.1103/PhysRevLett.93.197403.
  * Kouvel (2004) Kouvel, J.S., 2004.  Unusual Nature of the Abrupt Magnetic Transition in FeRh and Its Pseudobinary Variants.  Journal of Applied Physics 37, 1257–1258.  URL: https://doi.org/10.1063/1.1708424, doi:10.1063/1.1708424.
  * Kouvel and Hartelius (2004) Kouvel, J.S., Hartelius, C.C., 2004.  Anomalous Magnetic Moments and Transformations in the Ordered Alloy FeRh.  Journal of Applied Physics 33, 1343–1344.  URL: https://doi.org/10.1063/1.1728721, doi:10.1063/1.1728721.
  * Kresse and Furthmüller (1996) Kresse, G., Furthmüller, J., 1996.  Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set.  Phys. Rev. B 54, 11169–11186.  doi:10.1103/PhysRevB.54.11169.
  * Kresse and Joubert (1999) Kresse, G., Joubert, D., 1999.  From ultrasoft pseudopotentials to the projector augmented-wave method.  Phys. Rev. B 59, 1758–1775.  doi:10.1103/PhysRevB.59.1758.
  * Kudrnovský et al. (2015) Kudrnovský, J., Drchal, V., Turek, I., 2015.  Physical properties of FeRh alloys: The antiferromagnetic to ferromagnetic transition.  Phys. Rev. B 91, 014435.  URL: https://link.aps.org/doi/10.1103/PhysRevB.91.014435, doi:10.1103/PhysRevB.91.014435.
  * Lane et al. (2018) Lane, C., Furness, J.W., Buda, I.G., Zhang, Y., Markiewicz, R.S., Barbiellini, B., Sun, J., Bansil, A., 2018.  Antiferromagnetic ground state of ${\mathrm{la}}_{2}{\mathrm{cuo}}_{4}$: A parameter-free ab initio description.  Phys. Rev. B 98, 125140.  URL: https://link.aps.org/doi/10.1103/PhysRevB.98.125140, doi:10.1103/PhysRevB.98.125140.
  * Lee et al. (2006) Lee, P.A., Nagaosa, N., Wen, X.G., 2006.  Doping a mott insulator: Physics of high-temperature superconductivity.  Rev. Mod. Phys. 78, 17–85.  URL: https://link.aps.org/doi/10.1103/RevModPhys.78.17, doi:10.1103/RevModPhys.78.17.
  * Liu et al. (2022) Liu, X., Pandey, S.K., Feng, J., 2022.  Silver (II) route to unconventional superconductivity.  Phys. Rev. B 105, 134519.  URL: https://link.aps.org/doi/10.1103/PhysRevB.105.134519, doi:10.1103/PhysRevB.105.134519.
  * Liu et al. (2016) Liu, Y., Phillips, L.C., Mattana, R., Bibes, M., Barthélémy, A., Dkhil, B., 2016.  Large reversible caloric effect in ferh thin films via a dual-stimulus multicaloric cycle.  Nature communications 7, 11614.  URL: https://doi.org/10.1038/ncomms11614, doi:10.1038/ncomms11614.
  * Maat et al. (2005) Maat, S., Thiele, J.U., Fullerton, E.E., 2005.  Temperature and field hysteresis of the antiferromagnetic-to-ferromagnetic phase transition in epitaxial FeRh films.  Phys. Rev. B 72, 214432.  URL: https://link.aps.org/doi/10.1103/PhysRevB.72.214432, doi:10.1103/PhysRevB.72.214432.
  * Mahana et al. (2020) Mahana, S., Pandey, S.K., Rakshit, B., Nandi, P., Basu, R., Dhara, S., Turchini, S., Zema, N., Manju, U., Mahanti, S.D., et al., 2020.  Site substitution in GdMnO3: Effects on structural, electronic, and magnetic properties.  Phys. Rev. B 102, 245120.  URL: https://link.aps.org/doi/10.1103/PhysRevB.102.245120, doi:10.1103/PhysRevB.102.245120.
  * Marti et al. (2014) Marti, X., Marti, X., Fina, I., Fina, I., Frontera, C., Liu, J., Wadley, P., He, Q., Paull, R.J., Clarkson, J.D., Kudrnovský, J., Turek, I., Kuneš, J., Yi, D., Chu, J.H., Nelson, C.T., You, L., Arenholz, E.A., Salahuddin, S.S., Fontcuberta, J., Jungwirth, T., Ramesh, R., Ramesh, R., 2014.  Room-temperature antiferromagnetic memory resistor.  Nature materials 13, 367–74.  URL: https://api.semanticscholar.org/CorpusID:33648160.
  * Middey et al. (2012) Middey, S., Nandy, A.K., Pandey, S., Mahadevan, P., Sarma, D., 2012.  Route to high Néel temperatures in 4$d$ and 5$d$ transition metal oxides.  Phys. Rev. B 86, 104406.  URL: https://link.aps.org/doi/10.1103/PhysRevB.86.104406, doi:10.1103/PhysRevB.86.104406.
  * Moruzzi and Marcus (1992) Moruzzi, V.L., Marcus, P.M., 1992.  Antiferromagnetic-ferromagnetic transition in ferh.  Phys. Rev. B 46, 2864–2873.  URL: https://link.aps.org/doi/10.1103/PhysRevB.46.2864, doi:10.1103/PhysRevB.46.2864.
  * Mryasov (2005) Mryasov, O.N., 2005.  Magnetic interactions and phase transformations in FeM, M=(Pt, Rh) ordered alloys.  Phase Transitions 78, 197–208.  URL: https://doi.org/10.1080/01411590412331316591, doi:10.1080/01411590412331316591.
  * O’Handley et al. (2000) O’Handley, R.C., Murray, S., Marioni, M., Nembach, H., Allen, S., 2000.  Phenomenology of giant magnetic-field-induced strain in ferromagnetic shape-memory materials.  Journal of Applied Physics 87, 4712–4717.  URL: https://doi.org/10.1063/1.373136, doi:10.1063/1.373136.
  * Pandey and Feng (2022) Pandey, S.K., Feng, J., 2022.  Spin interaction and magnetism in cobaltate kitaev candidate materials: An ab initio and model hamiltonian approach.  Phys. Rev. B 106, 174411.  URL: https://link.aps.org/doi/10.1103/PhysRevB.106.174411, doi:10.1103/PhysRevB.106.174411.
  * Pandey et al. (2023) Pandey, S.K., Gu, Q., Lin, Y., Tiwari, R., Feng, J., 2023.  Emergence of bond-dependent highly anisotropic magnetic interactions in Sr4RhO6: A theoretical study.  Phys. Rev. B 107, 115119.  URL: https://link.aps.org/doi/10.1103/PhysRevB.107.115119, doi:10.1103/PhysRevB.107.115119.
  * Pandey et al. (2021) Pandey, S.K., Kumar, A., Sarkar, S., Mahadevan, P., 2021.  Understanding the ferromagnetic insulating state in Cr-doped VO2: Density functional and tight binding calculations.  Phys. Rev. B 104, 125110.  URL: https://link.aps.org/doi/10.1103/PhysRevB.104.125110, doi:10.1103/PhysRevB.104.125110.
  * Pandey et al. (2017) Pandey, S.K., Mahadevan, P., Sarma, D.D., 2017.  Doping an antiferromagnetic insulator: A route to an antiferromagnetic metallic phase.  Europhy. Lett. 117, 57003.  URL: https://dx.doi.org/10.1209/0295-5075/117/57003, doi:10.1209/0295-5075/117/57003.
  * Pandey et al. (2022) Pandey, S.K., Nandy, A.K., Kumari, P., Mahadevan, P., 2022.  Microscopic origin of room-temperature ferromagnetism in the double perovskite ${\mathrm{Sr}}_{2}{\mathrm{FeReO}}_{6}$.  Phys. Rev. B 105, 214422.  URL: https://link.aps.org/doi/10.1103/PhysRevB.105.214422, doi:10.1103/PhysRevB.105.214422.
  * Perdew et al. (1996) Perdew, J.P., Burke, K., Ernzerhof, M., 1996.  Generalized gradient approximation made simple.  Phys. Rev. Lett. 77, 3865–3868.  URL: https://link.aps.org/doi/10.1103/PhysRevLett.77.3865, doi:10.1103/PhysRevLett.77.3865.
  * Perdew et al. (2008) Perdew, J.P., Ruzsinszky, A., Csonka, G.I., Vydrov, O.A., Scuseria, G.E., Constantin, L.A., Zhou, X., Burke, K., 2008.  Restoring the density-gradient expansion for exchange in solids and surfaces.  Phys. Rev. Lett. 100, 136406.  URL: https://link.aps.org/doi/10.1103/PhysRevLett.100.136406, doi:10.1103/PhysRevLett.100.136406.
  * Pickett (1989) Pickett, W.E., 1989.  Electronic structure of the high-temperature oxide superconductors.  Rev. Mod. Phys. 61, 433–512.  URL: https://link.aps.org/doi/10.1103/RevModPhys.61.433, doi:10.1103/RevModPhys.61.433.
  * Quirin et al. (2012) Quirin, F., Vattilana, M., Shymanovich, U., El-Kamhawy, A.E., Tarasevitch, A., Hohlfeld, J., von der Linde, D., Sokolowski-Tinten, K., 2012.  Structural dynamics in FeRh during a laser-induced metamagnetic phase transition.  Phys. Rev. B 85, 020103.  URL: https://link.aps.org/doi/10.1103/PhysRevB.85.020103, doi:10.1103/PhysRevB.85.020103.
  * Radu et al. (2010) Radu, I., Stamm, C., Pontius, N., Kachel, T., Ramm, P., Thiele, J.U., Dürr, H.A., Back, C.H., 2010.  Laser-induced generation and quenching of magnetization on FeRh studied with time-resolved x-ray magnetic circular dichroism.  Phys. Rev. B 81, 104415.  URL: https://link.aps.org/doi/10.1103/PhysRevB.81.104415, doi:10.1103/PhysRevB.81.104415.
  * Sandratskii and Mavropoulos (2011) Sandratskii, L.M., Mavropoulos, P., 2011.  Magnetic excitations and femtomagnetism of FeRh: A first-principles study.  Phys. Rev. B 83, 174408.  URL: https://link.aps.org/doi/10.1103/PhysRevB.83.174408, doi:10.1103/PhysRevB.83.174408.
  * Sawatzky and Allen (1984) Sawatzky, G.A., Allen, J.W., 1984.  Magnitude and origin of the band gap in NiO.  Phys. Rev. Lett. 53, 2339–2342.  URL: https://link.aps.org/doi/10.1103/PhysRevLett.53.2339, doi:10.1103/PhysRevLett.53.2339.
  * Shirane et al. (1964) Shirane, G., Nathans, R., Chen, C.W., 1964.  Magnetic moments and unpaired spin densities in the Fe-Rh alloys.  Phys. Rev. 134, A1547–A1553.  URL: https://link.aps.org/doi/10.1103/PhysRev.134.A1547, doi:10.1103/PhysRev.134.A1547.
  * Sun et al. (2016) Sun, J., Remsing, R.C., Zhang, Y., Sun, Z., Ruzsinszky, A., Peng, H., Yang, Z., Paul, A., Waghmare, U., Wu, X., Klein, M.L., Perdew, J.P., 2016.  Accurate first-principles structures and energies of diversely bonded systems from an efficient density functional.  Nature Chemistry 8, 831–836.  URL: http://dx.doi.org/10.1038/nchem.2535, doi:10.1038/nchem.2535.
  * Sun et al. (2015) Sun, J., Ruzsinszky, A., Perdew, J.P., 2015.  Strongly constrained and appropriately normed semilocal density functional.  Phys. Rev. Lett. 115, 036402.  URL: https://link.aps.org/doi/10.1103/PhysRevLett.115.036402, doi:10.1103/PhysRevLett.115.036402.
  * Svane (1992) Svane, A., 1992.  Electronic structure of ${\mathrm{la}}_{2}$${\mathrm{cuo}}_{4}$ in the self-interaction-corrected density-functional formalism.  Phys. Rev. Lett. 68, 1900–1903.  URL: https://link.aps.org/doi/10.1103/PhysRevLett.68.1900, doi:10.1103/PhysRevLett.68.1900.
  * Thiele et al. (2003) Thiele, J.U., Maat, S., Fullerton, E.E., 2003.  FeRh/FePt exchange spring films for thermally assisted magnetic recording media.  Applied Physics Letters 82, 2859–2861.  URL: https://doi.org/10.1063/1.1571232, doi:10.1063/1.1571232.
  * Togo (2023) Togo, A., 2023.  First-principles phonon calculations with Phonopy and Phono3py.  J. Phys. Soc. Jpn. 92, 012001.  doi:10.7566/JPSJ.92.012001.
  * Togo et al. (2023) Togo, A., Chaput, L., Tadano, T., Tanaka, I., 2023.  Implementation strategies in Phonopy and Phono3py.  J. Phys. Condens. Matter 35, 353001.  doi:10.1088/1361-648X/acd831.
  * Tran et al. (2016a) Tran, F., Stelzl, J., Blaha, P., 2016a.  Rungs 1 to 4 of DFT Jacob’s ladder: Extensive test on the lattice constant, bulk modulus, and cohesive energy of solids.  The Journal of Chemical Physics 144, 204120.  URL: https://doi.org/10.1063/1.4948636, doi:10.1063/1.4948636.
  * Tran et al. (2016b) Tran, F., Stelzl, J., Blaha, P., 2016b.  Rungs 1 to 4 of DFT Jacob’s ladder: Extensive test on the lattice constant, bulk modulus, and cohesive energy of solids.  The Journal of Chemical Physics 144, 204120.  URL: https://doi.org/10.1063/1.4948636, doi:10.1063/1.4948636.
  * Wang et al. (2020) Wang, Y., Decker, M.M., Meier, T.N., Chen, X., Song, C., Grünbaum, T., Zhao, W., Zhang, J., Chen, L., Back, C.H., 2020.  Spin pumping during the antiferromagnetic–ferromagnetic phase transition of iron–rhodium.  Nature Communications 11, 275.  URL: https://doi.org/10.1038/s41467-019-14061-w, doi:10.1038/s41467-019-14061-w.
  * Wolloch et al. (2016) Wolloch, M., Gruner, M.E., Keune, W., Mohn, P., Redinger, J., Hofer, F., Suess, D., Podloucky, R., Landers, J., Salamon, S., Scheibel, F., Spoddig, D., Witte, R., Roldan Cuenya, B., Gutfleisch, O., Hu, M.Y., Zhao, J., Toellner, T., Alp, E.E., Siewert, M., Entel, P., Pentcheva, R., Wende, H., 2016.  Impact of lattice dynamics on the phase stability of metamagnetic FeRh: Bulk and thin films.  Phys. Rev. B 94, 174435.  URL: https://link.aps.org/doi/10.1103/PhysRevB.94.174435, doi:10.1103/PhysRevB.94.174435.
  * Yamada et al. (2006) Yamada, H., Shimizu, H., Yamamoto, K., Uebayashi, K., 2006.  Structure and magnetism of 3d and 4d transition-metal alloys TT′ (T=Mn, Fe and T′=Rh, Pd) with CuAu-I type ordered structure.  Journal of Alloys and Compounds 415, 31–37.  URL: https://www.sciencedirect.com/science/article/pii/S0925838805012636, doi:https://doi.org/10.1016/j.jallcom.2005.07.046.
  * Yan et al. (2020) Yan, H., Feng, Z., Qin, P., Zhou, X., Guo, H., Wang, X., Chen, H., Zhang, X., Wu, H., Jiang, C., Liu, Z., 2020.  Electric-field-controlled antiferromagnetic spintronic devices.  Advanced Materials 32, 1905603.  URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/adma.201905603, doi:https://doi.org/10.1002/adma.201905603.
  * Zhang et al. (2018) Zhang, Y., Kitchaev, D.A., Yang, J., Chen, T., Dacek, S.T., Sarmiento-Pérez, R.A., Marques, M.A.L., Peng, H., Ceder, G., Perdew, J.P., Sun, J., 2018.  Efficient first-principles prediction of solid stability: Towards chemical accuracy.  npj Computational Materials 4.  URL: http://dx.doi.org/10.1038/s41524-018-0065-z, doi:10.1038/s41524-018-0065-z.
  * Zhou et al. (2021) Zhou, Z., Pandey, S.K., Feng, J., 2021.  Dynamical correlation enhanced orbital magnetization in VI3.  Phys. Rev. B 103, 035137.  URL: https://link.aps.org/doi/10.1103/PhysRevB.103.035137, doi:10.1103/PhysRevB.103.035137.


[table]capposition=above [table]capposition=top

11institutetext: Computer Vision Center, Catalonia, Spain
11email<EMAIL_ADDRESS>{sbiswas<EMAIL_ADDRESS>22institutetext:
Computer Science Department
Universitat Autònoma de Barcelona, Catalonia, Spain

#  SketchGPT: Autoregressive Modeling for Sketch Generation and Recognition

Adarsh Tiwari 1122 0009-0004-9266-7175 Sanket Biswas Main Corresponding
Author1122 0000-0001-6648-8270 Josep Lladós 1122 0000-0002-4533-4739

###### Abstract

We present SketchGPT, a flexible framework that employs a sequence-to-sequence
autoregressive model for sketch generation, and completion, and an
interpretation case study for sketch recognition. By mapping complex sketches
into simplified sequences of abstract primitives, our approach significantly
streamlines the input for autoregressive modeling. SketchGPT leverages the
next token prediction objective strategy to understand sketch patterns,
facilitating the creation and completion of drawings and also categorizing
them accurately. This proposed sketch representation strategy aids in
overcoming existing challenges of autoregressive modeling for continuous
stroke data, enabling smoother model training and competitive performance. Our
findings exhibit SketchGPT’s capability to generate a diverse variety of
drawings by adding both qualitative and quantitative comparisons with existing
state-of-the-art, along with a comprehensive human evaluation study. The code
and pretrained models will be released on this GitHub†.

$\dagger$$\dagger$footnotetext: https://github.com/eltonjohnfanboy/SketchGPT

###### Keywords:

Sketch Completion Sketch Recognition Next Stroke Prediction Stroke-to-
Primitive Mapping Autoregressive Models

##  1 Introduction

Sketches serve as foundational conceptual elements drawn by humans for a broad
spectrum of applications [20, 39], ranging from technical diagrams in
architecture [17], electronics [32], mechanical engineering [11], to
educational tools and even in entertainment, as seen in the popular games like
Pictionary [5]. In the Graphics Recognition domain, the interpretation of hand
drawn symbols and sketches has gained special attention.

With the growth of Large Language Models (LLMs) [7, 41] for the interpretation
and generation of meaningful textual documents [24], it has also opened doors
to a wide array of parallel applications. Among these applications is the
understanding of sketches, which can be compared to interpreting sentences
generated by a unique grammar or a visual language model [3]. This draws an
interesting parallel, just as the interpretation of a handwritten text
involves recognizing letters or characters as basic building blocks,
understanding sketches demand recognition of the composed objects that can be
associated to bi-dimensional lexical units or strokes. Unlike static images,
sketches are recorded as a sequence of dynamic pen tip movements which follows
a specific order and structure on the canvas, and then finally post-processed
and rendered in the image space. Exploring this sequential nature of sketches,
similar to a language model, can help to reveal the hidden lexicon encoded in
every stroke. Previous attempts like SketchRNN [15] showcased the importance
of sequential stroke information for sketch generation, while Sketch-BERT [22]
extended this approach to sketch recognition and retrieval by adapting BERT’s
language modeling [8] techniques.

Despite these contributions, the field still requires a holistic model capable
of addressing the spectrum of tasks associated with sketches, from generation
to classification and beyond, without the need to rely on a specialized
encoder for bidirectional context understanding. This work introduces
SketchGPT, an autoregressive generative model inspired by the emergent
capabilities of the GPT (Generative Pretrained Transformer) model series [7,
28, 29] in the domain of text. The key advantages of the next element
prediction pretraining principle over existing sketch language modeling
methods [15, 22] are: 1) Such principle is highly suitable for sketch
generation, where the order and sequence of strokes are essential. Each stroke
in a sketch depends on the preceding strokes, which makes it a strong
candidate. 2) Sketches can vary greatly in complexity and style, from simple
line drawings to more complex object compositions. Autoregressive models can
thrive in handling this variability, as they learn to predict a wide range of
possible next steps in the sequence, making them versatile for modeling
different types of sketches. 3) These models can adjust the generation process
at each step based on the previously generated content, to offer a higher
level of precision for capturing the fine-grained information of sketches.

However, autoregressive approaches can sometimes negatively impact the
reconstruction and prediction scores as they are prone to overfitting to the
actual data through teacher forcing, and may also struggle to accurately
complete drawings when they have to rely on their own generated predictions
[1]. Hence, to make the model generalizable, we propose a stroke-to-primitive
mapping strategy inspired by [2] which allows discretizing sketches into a
finite set of abstract primitives, simplifying the learning process and
reducing overfitting. This approach ensures more consistent and structured
predictions, minimizing errors in continuing drawings from inaccurate or
incomplete predictions.

Our main contributions are: 1) We present a GPT-inspired autoregressive model
that learns neural representations of sketches, capturing their dynamic
drawing process. 2) We propose a stroke-to-primitive abstraction strategy to
simplify input data and enhance model generalization across diverse sketches.
3) We present a multi-task model capable of predicting the next stroke,
generating, completing, and recognizing sketches, showcasing its overall
versatility in sketch-related tasks. 4) Empirically, we also propose a
quantitative study for sketch generation to compare with state-of-the-art
models, and a comprehensive human evaluation study to assess the quality of
generated sketches.

##  2 Related Work

Sketch as a Language: Just as language comprises a set of syntactic rules and
semantic structures, sketches can be interpreted as visual sentences [14, 25],
constructed through a series of strokes or ”words” that follow a certain
grammar. This analogy underpins our approach to modeling sketches, where the
sequential nature of drawing parallels the linear construction of sentences in
natural language. SketchRNN [15] employed the first autoregressive model based
on LSTM [18] to understand and generate sketches based on their sequential
stroke data. Following this, SketchBERT [22] leveraged the power of BERT’s
language modeling objective for tasks such as sketch recognition and
retrieval, showcasing the applicability of NLP techniques in the sketch
domain. Recently, SketchKnitter [43] proposed a diffusion model to understand
real human sketch distributions, formulating sketch generation as a denoising
process where the generator adjusts random stroke points over time to create a
recognizable sketch. Our work builds upon learning neural representations of
vectorized sketches in a GPT-like autoregressive way with next stroke
prediction task, so it can be extended to multiple downstream tasks like
sketch recognition, completion and generation.

Sketch Abstraction with Primitives: Simplifying complex shapes into basic
primitives is a key strategy for understanding the environment [6] mimicking
humans, a concept used in various fields like CAD sketches [14], vector-like
bitmap images [30]and 3D shape reconstruction from both sketches [35] and
images [23]. Previous attempts for abstract sketch generation [26, 27]
developed reinforcement learning paradigm of a stroke removal policy that
learns to predict which strokes can be safely removed without affecting
recognizability. While recently Alaniz et. al. [2] proposed a Primitive
Matching Network which learns to map each stroke of a sketch to its most
similar primitive in a given set with a distance transform loss, predicting an
affine transformation which can align the selected primitive to the target
stroke. Inspired by [2] we propose a simple sketch-to-primitive mapping
strategy that helps us to simplify and discretize the representation space of
the input sketch, and improve our autoregressive training strategy.

Sketch Applications: The introduction of the TU-Berlin [13], Sketchy [33] and
QuickDraw [19] datasets has significantly propelled research in sketch
recognition and generation. Initially, the challenge was tackled using maximal
margin classifiers fused with hand crafted features [34, 21]. However, the
introduction of large-scale sketch datasets [19, 33] paved the way for deep
learning models [37, 44], which has achieved performance surpassing that of
humans. Other application domains of interest include sketch-based image
retrieval [9], creative sketch generation [4], sketch-based object
localization [42] where abstract sketches serve as conceptual representations.
Employing transformer-based approaches [4, 22, 31, 40] have been the current
state-of-the-art practice, and innovating with learnt tokenization to enhance
sketch interpretability.

##  3 Method

Figure 1: Overview of SketchGPT. Given an input sketch (complete or
incomplete), the model predicts the next strokes for incomplete sketches or
classifies the complete ones, effectively adapting to different tasks.

Built upon the foundations of autoregressive GPT-like models, SketchGPT is a
task-agnostic generative transformer pre-trained on the QuickDraw dataset [15]
containing multiple object categories of sketches. The model learns neural
representations of sketch data, capturing sequential dependencies among their
compounding strokes. The pre-trained representations learned on multiple
concepts exhibit adaptability to a wide range of downstream sketch application
tasks. Our focus in this work is aimed mainly towards solving sketch
completion, sketch generation and sketch classification tasks. In Fig. 1, we
show an overview of the complete framework, focusing on the aforementioned
downstream tasks. Sketch generation and completion can be seen as two
instances of the same model, thus for the sake of simplicity, we show a single
model in this figure. For both of these tasks, namely classification and
completion, we initiate the process with a sketch abstraction step that
encodes the sketch strokes, in stroke-3 format, as sequences of primitives. A
predefined finite alphabet of straight-line-based tokens has been used as
primitives. The discretization of strokes reduces the complexity of the model
in its learning phase and additionally produces tokenization compatible with
the requirements of the language-based model. The tokenization step yields a
feature vector that serves as an input feed for the fine-tuned models to
conduct the downstream task. In subsequent subsections, detailed explanations
are provided for the aforementioned steps.

###  3.1 Data Preprocessing and Sketch Abstraction

At the basic level, a sketch is a collection of time-stamped coordinate points
obtained by a digital pen device. Some representation models compactly reduce
this sequence of coordinates to polygonal approximations utilizing a
vectorization process. Let us refer as strokes the segments of this
approximation. The QuickDraw dataset [15], used in this work as the benchmark,
represents sketches by the stroke-3 encoding. In this format, each sketch is
stored as a list of points, corresponding to the strokes, i.e. the segments of
the vectorial approximation. The points consist of three values:
($x$,$y$,$p$). The elements $x$ and $y$ are the coordinate offsets, and $p$
represents the status of the pen (on paper or on air). To ensure uniformity in
the data we perform min-max normalization on the coordinate values,
restricting them to the normalized range of 0 to 1 ($x,y\in[0,1]$).

The sketch abstraction phase, inspired by the work of Alaniz et al.[2]
conducts a stroke-to-primitive mapping process that discretizes the strokes of
the normalized sketch with a finite set of predefined primitives (Fig. 2). In
stroke-3 data, a stroke is a straight line defined by the pair of the
coordinates of subsequent tuples $(x_{i},y_{i})$ and $(x_{i+1},y_{i+1})$.
Since the proposed architecture is inspired by a GPT-based language model, we
are required to encode the sketch using a discrete set of tokens. With this
aim, each stroke has been approximated by a primitive out of a dictionary. The
dictionary contains a list of predefined primitives at regular orientation
shifts and fixed length.

Figure 2: Illustration of the stroke-to-primitive mapping and tokenization
process. We observe how raw stroke data is converted into a more structured
representation, involving the interpretation of stroke primitives and then
further tokenizing to be fed to the SketchGPT framework.

Each stroke $s_{i}$ is mapped to its most similar primitive $p_{j}$, in terms
of the cosine similarity between the corresponding orientations, computed as
follows:

$\text{sim}(s_{i},p_{j})=\frac{s_{i}\cdot p_{j}}{\|s_{i}\|\cdot\|p_{j}\|}$ (1)

where $s_{i}\cdot p_{j}$ is the dot product of the vectors $s_{i}$ and
$p_{j}$, and $\|s_{i}\|$ and $\|p_{j}\|$ denote the Euclidean norms of the
vectors $s_{i}$ and $p_{j}$, respectively. The primitive $p_{i}$ assigned to
the stroke $s_{i}$ is the one that maximizes this similarity measure:

$p_{i}=\arg\max_{p_{j}\in P}\text{sim}(s_{i},p_{j})$ (2)

The primitives of the dictionary have a fixed length. Since the strokes of the
input sketch are generated in a vectorization process, their lenght is
variable, depending on the smoothness of the original curve. To better
approximate the final encoding, we introduce a scaling transformation $T$,
that adjusts the length of the selected primitive $p_{i}$ with the length of
the original stroke $s_{i}$:

$T(p_{i},s_{i})=\begin{pmatrix}\left\lceil\frac{m(s_{i})}{m(p_{i})}\right\rceil\end{pmatrix}$
(3)

where $m(s_{i})$ represents the length of the input stroke, and the ceiling
operation ($\left\lceil\frac{m(s_{i})}{m(p_{i})}\right\rceil$) guarantees that
the length of the primitive $p_{i}$ aligns to the nearest integer length of
the original stroke $s_{i}$. After the entire process, the resulting
representation for a sketch $S_{i}$ can be expressed as:

$S_{i}=\\{p_{1}\cdot T(p_{1},s_{1}),p_{2}\cdot
T(p_{2},s_{2}),\ldots,p_{n}\cdot T(p_{n},s_{n})\\}$ (4)

We also make use of this information to tokenize the data. Our tokenization
method establishes a vocabulary $V$ that consists of tokens representing the
primitives ($p_{i}$), enriched with four special tokens: BOS (beginning of
sequence), SEP (separator), EOS (end of sequence), and PAD (padding). Let
$\tau$ denote the vector used for tokenization, constructed by repeating each
primitive $p_{i}$ according to its scaling factor:

$\mathcal{T}=[BOS,p_{1},\ldots,p_{1},p_{2},SEP,p_{2},\ldots,p_{2},\ldots,SEP,p_{n},\ldots,p_{n},EOS]$
(5)

where the repetition of each primitive $p_{i}$ is determined by its scaling
factor $T(p_{i},s_{i})$ defined in Eq. 3. This methodology ensures that the
tokenization incorporates the crucial details of our sketch, creating a
representation that can be fed into the model.

###  3.2 Model Architecture

SketchGPT is an autoregressive decoder-only based transformer model inspired
in the GPT-2 architecture design. The fundamental element of such
architectures, as well as in our proposed model, is the causal masked multi-
head self-attention mechanism. Masked self-attention is a specialized
adaptation of the conventional self-attention mechanism. It adds a constraint
through which each element (a stroke) of our input sequence can only attend to
the previous ones, which enables them to get trained autoregressively, where
the prediction of the next token relies exclusively on the preceding context.
Given a sequence $X$ which represents a sketch, the following equation models
this masking mechanism:

$\text{MaskedAttention}(X)=\text{softmax}\left(\frac{(X\cdot W_{Q}\cdot(X\cdot
W_{K})^{T})\odot\text{Mask}}{\sqrt{d_{k}}}\right)\cdot(X\cdot W_{V})$ (6)

where $W_{Q}$, $W_{K}$, and $W_{V}$ are learnable weight matrices, and $d_{k}$
is the dimension of the query and key vectors. The operation involves
computing query ($Q=X\cdot W_{Q}$), key ($K=X\cdot W_{K}$), and value
($V=X\cdot W_{V}$) projections of the input sequence, followed by a scaled
dot-product attention and a weighted sum using the softmax function. The
$\odot$ denotes element-wise multiplication, and Mask is a binary mask matrix
that prevents attending to future positions, preserving the temporal order of
the sequence.

Given a sequence $X$, the multi-head self-attention operation computes
multiple parallel self-attention blocks (or heads) as shown below:

$\text{MultiHead}(X)=\text{Concat}(\text{head}_{1},\text{head}_{2},\ldots,\text{head}_{h})\cdot
W_{O}$ (7)

where $\text{head}_{i}=\text{MaskedAttention}_{i}(X)$ represents the causal
masked self-attention output for the $i$-th head. The resulting concatenated
representation is linearly transformed by the output projection matrix
$W_{O}$. This enrichment in the architecture through the use of multiple heads
boosts the model’s efficacy in capturing diverse contextual and temporal
relationships within the sequence. After the multi-head attention layer, the
output is directed through a Multi-Layer Perceptron (MLP), giving place to the
fundamental building block of the network, which is the transformer block.
Stacking these transformer blocks shapes the core of the transformer block
used in our work.

###  3.3 Pre-training method for SketchGPT

Equivalently to the original GPT work [28] we use an unsupervised pre-training
approach. In this step, our model is exposed to a large corpus of sketch data,
and it learns to predict the next token in the stroke sequence using a next
token prediction strategy (Fig. 3(a)). Given an unsupervised corpus of stroke
tokens $S$, the model predicts the probability distribution of the next token
$\tau_{n}$ given a finite context window of preceding tokens. Thus, the model
is trained to minimize the negative log-likelihood loss to predict the correct
next token given the previous ones. Formally, the objective function that is
minimized is defined as:

$L_{\text{pretrain}(S)}=\sum_{n=i}-\log
P(\tau_{n}|\tau_{n-k},\ldots,\tau_{n-1};\Theta)$ (8)

where $k$ represents the size of the context window, and $P$ is the
conditional probability modeled in the network with parameters $\Theta$. This
process equips the model with an innate understanding of sequential
dependencies, structural nuances, and contextual relationships within sketch
data.

((a)) Pre-Training for Completion

((b)) Classification as Downstream task

Figure 3: Adaptability of SketchGPT. We illustrate how the sketch completion
pre-training phase could later be adapted towards its application for sketch
recognition downstream task.

###  3.4 Learning tasks through fine-tuning

Once the model has been pre-trained, we explore the ability of this model to
be utilized for downstream tasks, through a fine-tuning process. In particular
we focus in the following tasks:

  * •

Sketch completion and generation: In this first task, the model is trained to
complete and generate sketches autoregressively. The fine-tuning process
consists of feeding the model with partially drawn sketches and training it to
sequentially complete the missing components through a next token prediction
framework within a specific class context (unlike the pre-training step where
the model is trained across the entire corpus without class discrimination).
This task inherently encompasses unconditional sketch generation too, as a
special case, where the model learns to generate a sketch autoregressively
from scratch without any explicit input.

  * •

Sketch classification: This task, illustrated in Fig. 3(b), takes as input a
sketch $S_{i}$ and predicting it’s category label $y_{i}$. We assume a dataset
$D$, where for each instance of that dataset we have the sequence of input
tokens $\tau_{1},\ldots,\tau_{n}$ and its label $y$. To fine-tune the model
for the task, we pass the inputs through our pre-trained model to obtain the
final GPT block’s activation, which is then fed into the classification head
to predict $y$. This leads to the following objective function to minimize in
the fine-tuning process:

$L_{\text{classification}(D)}=\sum_{(S,y)}-\log P(y|\tau_{1},\ldots,\tau_{n})$
(9)

##  4 Experimental Validation

We extensively tested our SketchGPT model to assess its performance in the
proposed downstream tasks. Our goal is to demonstrate the feasibility of using
autoregressive models for sketch-related tasks, employing simplified
representations through sketch abstraction with primitives. We conduct various
experiments to highlight the effectiveness of our approach, analyzing results
both quantitatively and qualitatively.

###  4.1 Datasets

The proposed model is evaluated on the QuickDraw dataset [45], which consists
of over 50 million hand-drawn sketches across 345 different categories in
stroke-3 format. Collected from the Quick, Draw! game by Google, it comprises
sketches drawn by humans within a 20-second limit. The data in available in
two formats: images and stroke-3. For our work we make use of the latter.

###  4.2 Sketch Generation and Completion

In this initial set of experiments, we focus on evaluating the SketchGPT’s
capacity for sketch generation and completion.

Competitor. To evaluate the generation capacity of our model we compare it to
the SketchRNN, a RNN-based seq-to-seq variational autoencoder for vector
drawings. To perform the evaluation a subset form the QuickDraw dataset is
used. Specifically, we chose 7 classes based on the availability of pre-
trained weights provided by the SketchRNN authors. The 7 classes are: bus,
cat, elephant, flamingo, owl, pig and sheep.

Evaluation method. We make use of a CNN-based approach, employing a multi-
class ResNet34 trained to classify images of the 7 selected classes from the
QuickDraw dataset. The idea is to rasterize the generated sketches from
SketchRNN and our method into images, and feed them to the trained CNN. This
way we can discern which method produces sketches that are more readily
recognizable, as reflected by the CNN’s top-1 accuracy (following the common
practice in [36] and [38]) and top-3 accuracy. More concretely, we generate
1000 sketches for each of the 7 classes using both models, resulting in a
total of 7,000 generated samples for each model. ResNet34 classifier is then
used to assess the quality of each approach’s generation. ResNet34 was trained
using an early stopping strategy to prevent overfitting, with the model
reaching a top-1 validation accuracy of 87.92% at the stopping point.

Results and Discussion. Table 1 shows the performance results for the two
compared models in terms of the selected metrics. Notably, SketchGPT achieves
a higher performance for both top-1 and top-3 accuracy, indicating a more
finer generation ability for the selected classes. Interestingly, given that
the CNN used to asses the models was trained on sketches created by humans
(using the QuickDraw dataset), intuitively, this comparison also serves as an
indicator of which method yields sketches more similar to those drawn by
humans. In this case, the performance of the two models suggests that
SketchGPT slightly improves on SketchRNN generating more recognizable and
human-like sketches.

Table 1: Quantitative comparison between SketchRNN and SketchGPT recognition accuracy using a CNN classifier. Method | Top-1 acc. | Top-3 acc.
---|---|---
SketchRNN | 44.6% | 79.1%
SketchGPT | 50.4% | 81.7%
Figure 4: Human User Study results conducted based on the five properties
shown in the legend.

###  4.3 Human Evaluation Study

This experiment aims to evaluate the variability, human plausibility and
overall aesthetic of the sketches generated by our model compared to
SketchRNN. The study involved 100 individuals, showing them five sketches
generated from scratch using both SketchGPT and SketchRNN, for each of the
seven classes of the QuickDraw dataset whose pre-trained weights for SketchRNN
are publicly available (bus, cat, elephant, flamingo, owl, pig and sheep).
Each participant was given 5 different questions regarding the two compared
models to evaluate different qualitative aspects. More concretely, the
questions aimed to assess the two models based on the following criteria: (1)
fine details appreciation, (2) creativity, (3) diversity, (4) human-likeness
and (5) overall preference. As can be observed in Fig. 4, SketchGPT
consistently surpasses SketchRNN in all five examined properties. Notably, it
excels in aspects of diversity and creativity, showcasing a superior
capability to generate a broader and richer spectrum of sketches compared to
SketchRNN, which leans towards more stable but less varied outputs.

Figure 5: Qualitative Analysis for Sketch Generation. We illustrate the
prediction of different possible completion of multiple incomplete sketches.

###  4.4 Qualitative Analysis for Sketch Completion

As discussed earlier, besides generation, SketchGPT also shows promising
capacity in sketch completion tasks. In this section, we explore the
expressive sketch completion capabilities of our model. By inputting an
uncompleted sketch to our model, we generate diverse and plausible completion
variations of the given partial sketch. This unique aspect of SketchGPT
underscores the model’s creativity and adaptability in envisioning different
conceivable outcomes. We show results in Fig. 5, using multiple models trained
on individual classes. With this experimentation study, we highlight the
flexibility and richness of the sketch completion process by our proposed
methodology.

###  4.5 Sketch Recognition

Competitors. We compare SketchGPT to several existing baselines. (1) HOG-SVM
[12]: Classical approach that makes use of HOG features and SVM to perform the
classification task. (2) Ensemble [21]: Makes use of multiple sketch feature
to perform the classification task. (3) Bi-LSTM [18]: A four-layered bi-
directional LSTM with the dimension of the hidden states being 256, used to
perform sketch recognition. (4) Sketch-a-Net [44]: A multi-scale multi-channel
deep CNN framework designed to perform sketch related tasks. (5) DSSA [37]: An
attention module and a high-order energy triplet loss function are introduced
to enhance the original Sketch-A-Net model. (6) ResNet [16]: A popular CNN
architecture highly employed in classification tasks. (7) SketchBERT [22]: A
model that generalizes BERT to sketch domain. (8) ViT [10]: A transformer-
based neural network architecture tailored for computer vision tasks.

Evaluation method. For the evaluation, a subset of classes of the QuickDraw
dataset is used. We compare the results of training our model using 100
classes, following the evaluation procedure proposed in the SketchBERT paper
[22]. We train the SketchGPT for classification task of 100 categories with 5K
train samples, 2.5K validation samples and 2.5K test samples to show a fair
comparison with the proposed baselines results. The performance is evaluated
with Top-1 and Top-5 accuracy metrics. Furthermore, we show the ablation study
of our model with and without pre-training, to showcase the effect of the
process of pre-training in the performance of the model.

Results and Discussion. In Table 2, we can observe the comparative performance
of our classification model against several baselines in terms of Top-1 and
Top-5 accuracy. The 1st/2nd best results are indicated in red/blue. SketchGPT
achieves the second-highest performance for both of the metrics, getting
surpassed only by SketchBERT. It showcases superior recognition capabilities
compared to conventional classification methodologies, prevalent CNN-based
frameworks designed for sketch recognition, and even surpasses state-of-the-
art models such as ViT. However, regarding it’s comparison with SketchBERT, it
is important to note that a fair comparison with the aforementioned baseline
is hard to achieve, principally due to the lack of knowledge regarding the
specific classes used for evaluation. SketchBERT’s paper does not specify the
100 classes utilized, potentially leading to a disparity in complexity between
the classes evaluated. Furthermore, SketchBERT’s reported results are based on
pre-training with 345 classes and 70K samples, followed by fine-tuning for
classification with the 100 classes. In contrast, our pre-training involved
only 50 classes and 5K samples, a considerably smaller dataset, principally
due to computation limits. Notably, SketchBERT’s ablation studies indicate
that using smaller pre-trained datasets can lead to inferior results, as
evidenced by their experiments with 100 classes and 5K samples resulting in
85.82% top-1 accuracy and 97.31% top-5 accuracy. Given all these factors, it
is reasonable to assert that a more equitable comparison, utilizing the same
classes and employing larger pre-training datasets for our model, may yield
even more favorable results for SketchGPT.

Table 2: Top-1 and Top-5 accuracy for Sketch Recogntion when compared to other baselines; w./o., and w indicate the results without and with the pre-training process. Methods |  QuickDraw (%) |
---|---|---
| Top-1 acc. | Top-5 acc.
HOG-SVM [12] | $52.05$ | $74.50$
Ensemble [21] | $60.31$ | $80.22$
Bi-LSTM [18] | $74.68$ | $90.59$
Sketch-a-Net [44] | $70.64$ | $87.93$
DSSA [37] | $79.47$ | $92.41$
ResNet18 [16] | $79.67$ | $91.71$
ResNet34 [16] | $82.02$ | $93.48$
SketchBERT [22] | $88.30$ | $97.82$
ViT [10] | $47.69$ | $68.73$
SketchGPT (w/o PT) | $81.42$ | $91.81$
SketchGPT (w PT) | $83.58$ | $93.65$

###  4.6 Ablation studies

In this section, we conduct multiple ablation studies to analyze how our
models performance is affected by altering different conditions.

Model’s generation ability through temperature (t) modulation. In this
experiment, we systematically vary the temperature parameter of our model for
the generation task from 0.6 to 2.0. The results are shown in Fig. 6 for the
sword class. The temperature parameter controls the degree of randomness in
sampling from the model’s output probability distribution. Upon analyzing the
results, we notice that at lower temperatures (from 0.6 to 0.8), the model
produces abstract and too simplistic sketches with minimal detail, while at
higher temperatures (1.6 to 2.0), it tends to generate more random and
nonsensical compositions. Interestingly, within the range of 1.0 to 1.4, the
model strikes a balance, generating sketches with greater coherence and
complexity. This showcases how temperature plays a major role in influencing
the diversity and quality of generated outputs.

Figure 6: Temperature Scaling impact on our model’s generated sketches.

Impact of total class counts on classification performance. In the context of
the classification task, this experiment analyzes the change in performance of
our model as we increase the number of classes used to train the model. The
study is done training our models with the same settings, but just changing
the class counts from 25 to 200. For each number of categories, we use 5K
samples per class to train the model. In Table 3 we can observe the results. A
slight decrease in performance is observed as the number of categories
increases. Nevertheless, this decline is less pronounced from 25 to 50 to 100
classes, with a more noticeable decrease in performance upon reaching 200
classes. Notably, these observations were made under fixed training size
conditions, augmenting the training dataset size might offer a viable solution
to mitigate the performance disparities observed with higher class counts.

Table 3: Sketch Recognition performance with different class counts/sample sizes/network structures. Class counts. | Top-1 acc. | Top-5 acc.
---|---|---
25 classes | 84.89% | 94.21%
50 classes | $83.68\%$ | $94.12\%$
100 classes | $83.58\%$ | $93.65\%$
200 classes | $79.86\%$ | $90.06\%$

Training size. | Top-1 acc. | Top-5 acc.
---|---|---
5K | $84.89\%$ | $94.21\%$
10K | $85.67\%$ | $95.34\%$
20K | $85.69\%$ | $95.68\%$
75K | 89.95% | 97.51%

Network size. | Top-1 acc. | Top-5 acc.
---|---|---
4-8-256 | $80.75\%$ | $91.56\%$
12-8-256 | $83.65\%$ | $93.32\%$
12-12-768 | $84.47\%$ | $94.16\%$
8-8-512 | 84.89% | 94.21%

Influence of training data size on classification performance. In this study,
we aim to explore the effect of increasing the training sample size on our
model’s performance. Given the practical constraints of the large number of
classes, the study is performed in a reduced group of 25 categories, to
understand if augmenting the training data within the subset actually affects
model performance. The evaluation metrics used are Top-1 and Top-5 accuracy,
and the results are shown in Table 3. We observe that the model benefits from
the increase in the number of samples, yielding a percentage increase of 5.96%
in our Top-1 accuracy when expanding the training samples from 5K to 75K.
Extrapolating from these findings, we anticipate that scaling up the training
size would likely result in improved performance with a greater number of
classes (e.g. 200 classes, 345 classes, etc.), thereby underscoring the
potential benefits of increased sample sizes in enhancing model performance.

Figure 7: Convergence rate of SketchGPT with and without pre-training.

Effects of network’s size on classification performance. We aim to examine how
the variation in the size of our model might impact its performance for the
sketch recognition task. The parameters explored are the number of layers (L),
self-attention heads (A), and the hidden size (H); and four scenarios are
investigated to understand the influence. As we observe in Table 3, the
optimal performance is achieved when L = 8, A = 8 and H = 512, making a good
balance between the model complexity and final performance. It is noteworthy
that augmenting the number of layers yields a substantive performance boost,
elevating top-1 accuracy from 80.75% to 83.65%, even with a smaller hidden
size of 256. On the other hand, opting for a larger network configuration (L =
12, A = 8, and H = 768) approaches optimal performance too, but the training
process duration significantly extends.

Convergence Rate acceleration through the use of pre-training.

Parallel to [22] our model also exhibits a faster convergence rate with pre-
training on the QuickDraw dataset. However, it’s worth noticing that the
observed acceleration in convergence is way less pronounced in our case. As
observed in Fig. 7, which shows the loss curves of our model trained on 100
classes with 5K samples per category, after pre-training the model manages to
converge from about 40 to 22 epochs. This phenomenon further highlights the
importance of pre-training for SketchGPT, demonstrating that besides the boost
in performance, we also achieve a less time-consuming training procedure.

##  5 Conclusion and Future Work

In this work, we have presented a novel approach to employ autoregressive
models in the sketch domain. Through an abstraction process that simplifies
the sketch data, we facilitate the learning process of our model;
demonstrating remarkable proficiency in sketch generation and completion, as
well as strong competitiveness in classification tasks. An inherent limitation
of our model lies in the data representation, where the abstraction process
introduces significant information loss. Future research efforts should
prioritize the development of an improved data representation approach to
alleviate this challenge. By doing so, we anticipate significant enhancements
in model performance and its adaptability to diverse and complex datasets.
Future Scopes and Challenges Despite the strengths demonstrated by our model
in sketch generation and completion tasks, as well as competitive performance
in sketch classification, certain limitations are worth noting from our data
representation methodology. Although the simplification of data through sketch
approximation with primitives aids learning, it also results in information
loss which adversely affects performance. Moreover, the absence of techniques
like the Ramer-Douglas-Peucker (RDP) algorithm [15],[22] to shorten the
sequence length of the sketches hinders the model convergence, affecting the
model performance. Since our approximation method already results in some
information loss, we opted against using additional algorithms like RDP, where
further simplification could lead to more information loss. Our methodology
yields competitive performances when applied to the QuickDraw dataset,
characterized by relatively simpler sketches. However, in more complex
datasets such as TU-Berlin, [12], the data approximation strategy through
primitives results in some information loss that impacts the model’s
performance, particularly in tasks like sketch completion and generation. The
road forward calls for a more effective data representation strategy, which
should minimize information loss while decreasing the maximum sequence length
to enhance both model convergence and effectiveness in performance.

## Acknowledgment

This work has been partially supported by the Spanish project
PID2021-126808OB-I00, the Catalan project 2021 SGR 01559, the CVC Internship
Program and PhD Scholarship from AGAUR 2023 FI-3-00223. The Computer Vision
Center is part of the CERCA Program/Generalitat de Catalunya.

## References

  * [1] Aksan, E., Deselaers, T., Tagliasacchi, A., Hilliges, O.: Cose: Compositional stroke embeddings. Advances in Neural Information Processing Systems 33, 10041–10052 (2020)
  * [2] Alaniz, S., Mancini, M., Dutta, A., Marcos, D., Akata, Z.: Abstracting sketches through simple primitives. In: eccv. pp. 396–412. Springer (2022)
  * [3] Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35, 23716–23736 (2022)
  * [4] Bhunia, A.K., Khan, S., Cholakkal, H., Anwer, R.M., Khan, F.S., Laaksonen, J., Felsberg, M.: Doodleformer: Creative sketch drawing with transformers. In: European Conference on Computer Vision. pp. 338–355. Springer (2022)
  * [5] Bhunia, A.K., Das, A., Muhammad, U.R., Yang, Y., Hospedales, T.M., Xiang, T., Gryaditskaya, Y., Song, Y.Z.: Pixelor: A competitive sketching ai agent. so you think you can sketch? ACM Transactions on Graphics (TOG) 39(6), 1–15 (2020)
  * [6] Biederman, I.: Recognition-by-components: a theory of human image understanding. Psychological review 94(2), 115 (1987)
  * [7] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
  * [8] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [9] Dey, S., Riba, P., Dutta, A., Llados, J., Song, Y.Z.: Doodle to search: Practical zero-shot sketch-based image retrieval. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2179–2188 (2019)
  * [10] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [11] Egiazarian, V., Voynov, O., Artemov, A., Volkhonskiy, D., Safin, A., Taktasheva, M., Zorin, D., Burnaev, E.: Deep vectorization of technical drawings. In: European conference on computer vision. pp. 582–598. Springer (2020)
  * [12] Eitz, M., Hays, J., Alexa, M.: How do humans sketch objects? ACM Transactions on graphics (TOG) 31(4), 1–10 (2012)
  * [13] Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: Sketch-based image retrieval: Benchmark and bag-of-features descriptors. IEEE transactions on visualization and computer graphics 17(11), 1624–1636 (2010)
  * [14] Ganin, Y., Bartunov, S., Li, Y., Keller, E., Saliceti, S.: Computer-aided design as language. Advances in Neural Information Processing Systems 34, 5885–5897 (2021)
  * [15] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR 2018 (2018), https://openreview.net/pdf?id=Hy6GHpkCW, 2018
  * [16] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [17] de las Heras, L.P., Terrades, O.R., Robles, S., Sánchez, G.: Cvc-fp and sgt: a new database for structural floor plan analysis and its groundtruthing tool. International Journal on Document Analysis and Recognition (IJDAR) 18, 15–30 (2015)
  * [18] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)
  * [19] Jongejan, J., Rowley, H., Kawashima, T., Kim, J., Fox-Gieg, N.: The quick, draw!-ai experiment. Mount View, CA, accessed Feb 17(2018), 4 (2016)
  * [20] Jonson, B.: Design ideation: the conceptual sketch in the digital age. Design studies 26(6), 613–624 (2005)
  * [21] Li, Y., Song, Y.Z., Gong, S., et al.: Sketch recognition by ensemble matching of structured features. In: BMVC. vol. 1, p. 2 (2013)
  * [22] Lin, H., Fu, Y., Xue, X., Jiang, Y.G.: Sketch-bert: Learning sketch bidirectional encoder representation from transformers by self-supervised learning of sketch gestalt. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6758–6767 (2020)
  * [23] Liu, Z., Freeman, W.T., Tenenbaum, J.B., Wu, J.: Physical primitive decomposition. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 3–19 (2018)
  * [24] Lv, T., Huang, Y., Chen, J., Cui, L., Ma, S., Chang, Y., Huang, S., Wang, W., Dong, L., Luo, W., et al.: Kosmos-2.5: A multimodal literate model. arXiv preprint arXiv:2309.11419 (2023)
  * [25] Mas, J., Lladós, J., Sánchez, G., Jorge, J.A.P.: A syntactic approach based on distortion-tolerant adjacency grammars and a spatial-directed parser to interpret sketched diagrams. Pattern Recognition 43(12), 4148–4164 (2010)
  * [26] Muhammad, U.R., Yang, Y., Hospedales, T.M., Xiang, T., Song, Y.Z.: Goal-driven sequential data abstraction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 71–80 (2019)
  * [27] Muhammad, U.R., Yang, Y., Song, Y.Z., Xiang, T., Hospedales, T.M.: Learning deep sketch abstraction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
  * [28] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018)
  * [29] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
  * [30] Reddy, P., Gharbi, M., Lukac, M., Mitra, N.J.: Im2vec: Synthesizing vector graphics without vector supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7342–7351 (2021)
  * [31] Ribeiro, L.S.F., Bui, T., Collomosse, J., Ponti, M.: Sketchformer: Transformer-based representation for sketched structure. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14153–14162 (2020)
  * [32] Rusiñol, M., Borràs, A., Lladós, J.: Relational indexing of vectorial primitives for symbol spotting in line-drawing images. Pattern Recognition Letters 31(3), 188–201 (2010)
  * [33] Sangkloy, P., Burnell, N., Ham, C., Hays, J.: The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions on Graphics (TOG) 35(4), 1–12 (2016)
  * [34] Schneider, R.G., Tuytelaars, T.: Sketch classification and classification-driven analysis using fisher vectors. ACM Transactions on graphics (TOG) 33(6), 1–9 (2014)
  * [35] Smirnov, D., Bessmeltsev, M., Solomon, J.: Deep sketch-based modeling of man-made shapes. arXiv preprint arXiv:1906.12337 2 (2019)
  * [36] Song, J., Pang, K., Song, Y., Xiang, T., Hospedales, T.M.: Learning to sketch with shortcut cycle consistency. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. pp. 801–810. Computer Vision Foundation / IEEE Computer Society (2018). https://doi.org/10.1109/CVPR.2018.00090, http://openaccess.thecvf.com/content_cvpr_2018/html/Song_Learning_to_Sketch_CVPR_2018_paper.html
  * [37] Song, J., Yu, Q., Song, Y.Z., Xiang, T., Hospedales, T.M.: Deep spatial-semantic attention for fine-grained sketch-based image retrieval. In: Proceedings of the IEEE international conference on computer vision. pp. 5551–5560 (2017)
  * [38] Su, G., Qi, Y., Pang, K., Yang, J., Song, Y.: Sketchhealer: A graph-to-sequence network for recreating partial human sketches. In: 31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event, UK, September 7-10, 2020. BMVA Press (2020), https://www.bmvc2020-conference.com/assets/papers/0711.pdf
  * [39] Suwa, M., Gero, J.S., Purcell, T.A.: The roles of sketches in early conceptual design processes. In: Proceedings of the twentieth annual conference of the Cognitive Science Society. pp. 1043–1048. Routledge (2022)
  * [40] Tiwari, A., Biswas, S., Lladós, J.: Can pre-trained language models help in understanding handwritten symbols? In: International Conference on Document Analysis and Recognition. pp. 199–211. Springer (2023)
  * [41] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
  * [42] Tripathi, A., Dani, R.R., Mishra, A., Chakraborty, A.: Sketch-guided object localization in natural images. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16. pp. 532–547. Springer (2020)
  * [43] Wang, Q., Deng, H., Qi, Y., Li, D., Song, Y.Z.: Sketchknitter: Vectorized sketch generation with diffusion models. In: The Eleventh International Conference on Learning Representations (2022)
  * [44] Yu, Q., Liu, F., Song, Y.Z., Xiang, T., Hospedales, T.M., Loy, C.C.: Sketch me that shoe. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 799–807 (2016)
  * [45] Zhong, X., Tang, J., Yepes, A.J.: Publaynet: largest dataset ever for document layout analysis. In: Proceedings of the International Conference on Document Analysis and Recognition. pp. 1015–1022 (2019)