††thanks<EMAIL_ADDRESS>Corresponding author<EMAIL_ADDRESS>
# Magnetic Properties of NbSi2N4, VSi2N4, and VSi2P4 Monolayers

Md. Rakibul Karim Akanda  Laboratory for Terahertz and Terascale Electronics
(LATTE), Department of Electrical and Computer Engineering, University of
California, Riverside, CA 92521, USA  Roger K. Lake  Laboratory for Terahertz
and Terascale Electronics (LATTE), Department of Electrical and Computer
Engineering, University of California, Riverside, CA 92521, USA

###### Abstract

The recent demonstration of MoSi2N4 and its exceptional stability to air,
water, acid, and heat has generated intense interest in this family of two-
dimensional (2D) materials. Among these materials, monolayers of NbSi2N4,
VSi2N4, and VSi2P4 are semiconducting, easy-plane ferromagnets with negligible
in-plane magnetic anisotropy. They thus satisfy a necessary condition for
exhibiting a dissipationless spin superfluid mode. The Curie temperatures of
monolayer VSi2P4 and VSi2N4 are determined to be above room temperature based
on Monte Carlo and density functional theory calculations. The magnetic
moments of VSi2N4 can be switched from in-plane to out-of-plane by applying
tensile biaxial strain or electron doping.

Two dimensional (2D) layered materials with transition metals have been of
interest for many decades due to the correlated phenomena and multiple
polymorphs and phases that they can exhibit such as charge density waves,[1]
superconductivity,[2] and magnetism.[3] The ability to exfoliate or grow
single monolayers renewed the interest in these materials for possible
electronic and optoelectronic applications[4, 5] by both traditional
transistor type devices[6, 7] and by exploiting external control of their
phase transitions.[8] The relatively recent demonstration of magnetism in
single monolayer of CrI3[9] and bilayers of Cr2Ge2Te6[10] has spurred intense
experimental and theoretical activity to find other 2D magnetic materials with
higher transition temperatures.[11, 12, 13, 14, 15, 16, 17, 18]

The most recent addition to the family of 2D materials are the transition
metal silicon nitrides, phosphides, and arsenides with the chemical formulas
MA2Z4, where M is the transition metal, A $\in$ {Si, Ge}, and Z $\in$ {N, P,
As}.[19] High quality multilayers and monolayers of MoSi2N4 were grown using
chemical vapor deposition, and what was particularly notable was their
stability to air, water, acid, and heat that was unprecedented among
transition metal 2D materials.[19] This rather mundane property is highly
desirable for manufacturing applications. While BN encapsulation is an
effective solution for stabilizing reactive 2D materials for laboratory
experiments,[20] it is less than ideal for manufacturing. Only MoSi2N4 was
experimentally characterized in detail, WSi2N4 was also grown, and 12
materials were simulated with density functional theory (DFT) and found to be
stable. Among these 12, two of the nitrides, VSi2N4 and NbSi2N4, were
identified as magnetic.

Figure 1: Structures and magnetic orientations of $\alpha_{1}$-NbSi2N4,
$\alpha_{1}$-VSi2N4, and $\alpha_{1}$-VSi2P4. Blue atoms are Si, white atoms
are N, and pink atoms are P. The directions of the magnetic moments are shown
on the transition metals Nb and V. All materials are easy-plane,
semiconducting ferromagnets.

This work motivated immediate follow-on theoretical investigations of this
material family both determining properties of the materials and extending the
list of stable materials. [21, 22, 23, 24, 25, 26, 27, 28, 29] The most
extensive theoretical survey found 32 thermodynamically and dynamically stable
compounds of the form MA2Z4. [24] Of these, 6 were magnetic: VSi2N4, VSi2P4,
NbSi2N4, VGe2N4, VGe2P4, and TaGe2N4. Based on the formation enthalpies, a Si
based compound MSi2Z4 is approximately 3 times more stable than its equivalent
Ge based compound MGe2Z4, and a silicon nitride compound MSi2N4 is also
approximately 3 times more stable than its equivalent silicon phosphide
compound MSi2P4.

Other recent works investigated MSi2Z4 bilayers (M = Ti, Cr, Mo; Z = N, P) for
their sensitivity to vertical strain [28, 27], and MoSi2N4 and WSi2N4 for
their sensitivity to biaxial strain and vertical applied electric field.[27]
It was found that vertical strain can cause an insulator to metal transition;
[28] biaxial strain can lead to an indirect to direct bandgap transition; [27]
and that an applied electric field can result in an insulator to metal
transition. [27] A theoretical investigation of 2D-2D contacts to monolayer
MoSi2N4 using NbS2 and graphene found ultralow p-type Schottky barriers with
NbS2 contacts and approximately equal n-type and p-type Schottky barriers with
graphene contacts. [22] The graphene Schottly barriers were shown to be
tunable with an applied vertical electric field.

In this letter, we theoretically investigate the magnetic properties of the
three Si based magnetic materials: VSi2N4, VSi2P4, NbSi2N4. Values of the
exchange constants and magnetic anisotropy energies are determined, and the
Curie temperatures are calculated. The Curie temperatures of VSi2N4 and VSi2P4
are near or at room temperature and above, depending on the model used,
whereas the Curie temperature of NbSi2N4 is low. Therefore, the primarily
focus will be on the two vanadium compounds. The magnetic anisotropy energies
of VSi2N4 and VSi2P4 are calculated as a function of uniaxial and biaxial
strain and electron and hole doping.

These materials can exist in a variety of hexagonal phases. Ref. [24]
calculated the formation energies of 30 different phases and found that the
lowest energy phase of NbSi2N4 and VSi2N4 was the $\alpha_{1}$ phase, which is
the same phase as the lowest energy phase of MoSi2N4. The formation energy of
the next higher energy phase ($\delta_{4}$) of NbSi2N4 was 13 meV per atom
higher than the $\alpha_{1}$ phase, and the formation energy of the next
higher energy phase ($\beta_{2}$) of VSi2N4 was 6 meV per atom higher than the
$\alpha_{1}$ phase. For VSi2P4, the lowest energy phase was $\delta_{4}$ with
a formation energy 0.3 meV below that of $\alpha_{1}$. Since the formation
energy of the $\alpha_{1}$ phase of VSi2P4 is so close to that of the
$\delta_{4}$ phase, and since the $\delta_{4}$ phase has been previously
analyzed [24], we will consider the $\alpha_{1}$ phase of all 3 materials:
NbSi2N4, VSi2N4, and VSi2P4. We note that the piezoelectric and magnetic
properties of the $\alpha_{1}$ phase of VSi2P4 have recently been investigated
with DFT using the generalized gradient approximation (GGA).[21]

DFT combined with Monte Carlo (MC) calculations are applied to determine the
electronic and magnetic properties of these materials. The magnetic anisotropy
energy (MAE) and exchange energy are evaluated from DFT calculations
implemented in the Vienna ab initio simulation package (VASP).[30] The
electron-core interactions are described by the projected augmented wave (PAW)
potentials.[31] Electronic structure is calculated using three different
functionals: GGA as parameterized by Perdew-Burke-Ernzerhof (PBE), PBE plus
the Hubbard U correction (PBE+U), and the Heyd-Scuseria-Ernzerhof hybrid
functional (HSE06).[32, 33, 34, 35] The PBE+U calculation includes a Hubbard U
correction term $U_{\rm eff}=1$ eV for the V atom where $U_{\rm eff}=U-J$.[36,
37] For calculations of the MAE, spin orbit coupling (SOC) must also be
included. All plots shown for the MAE are the results from PBE(SOC)+U
calculations. The cutoff energies for expanding the plane wave basis are 600
eV. Integration over the Brillouin zone uses a Monkhorst–Pack scheme with a
$\Gamma$-centered $16\times 16\times 1$ k-point grid, an energy broadening
parameter of 50 meV, and the total energy is converged to $10^{-6}$ eV.
Structures are relaxed until the forces are less than 0.001 eV/Å. A vacuum
spacing of 20 Å is used in the direction normal to the 2D monolayer to
eliminate the interactions from periodic images. The calculated lattice
constants for NbSi2N4, VSi2N4, and VSi2P4 are 2.96 Å, 2.88 Å, and 3.47 Å,
respectively, and they are very close to previous reported results.[19, 21]

Uniaxial strain is applied along the $x$ axis corresponding to lattice vector
$a_{1}$. The applied strain is evaluated using $\epsilon=(a-a_{0})/a_{0}\times
100\%$, where $a$ and $a_{0}$ are the lattice parameters of the strained and
unstrained monolayer. Biaxial strain is applied by uniformly varying both in-
plane lattice constants. For each strain, the atomic positions are relaxed
using the DFT parameters described above. Spin-polarized self consistent
calculations are performed with the relaxed structure for each strain to
obtain the charge density. Using the charge densities, total energies are
calculated in the presence of spin orbit coupling (SOC) for in-plane
($E_{||}$) and out-of-plane ($E_{\perp}$) magnetization to find the magnetic
anisotropy energy (MAE). The MAE is defined as $E_{\rm MAE}=E_{\perp}-E_{||}$.
Positive magnetic anisotropy indicates that in-plane magnetization is favored.
The values provided in meV are per magnetic atom, which for these 3 materials
under consideration, are also per unit cell.

To investigate the transition temperatures, a nearest neighbor Heisenberg type
Hamiltonian with magnetic anisotropy and long range dipole-dipole interactions
is constructed,

$H=-\tfrac{1}{2}J\sum_{\langle i,j\rangle}{\bf S}_{i}\cdot{\bf
S}_{j}+k_{u}\sum_{i}\left(S^{z}_{i}\right)^{2}+H_{dd},$ (1)

where ${\bf S}_{i}$ is the spin for magnetic atom $i$, $\langle i,j\rangle$
are the indices of nearest neighbor magnetic atoms, $k_{u}>0$ is the easy-
plane magnetic anisotropy energy per magnetic atom, and $H_{dd}$ is the
dipole-dipole interaction. The exchange energies ($J$) of these materials are
calculated from the total energy differences between the antiferromagnetic
($E_{AFM}$) and ferromagnetic ($E_{FM}$) states.[38, 39, 40, 41] With six
nearest neighbours in the monolayer of 2D hexagonal lattice, the nearest-
neighbor exchange energy is given by,[40, 41, 42] $J=(E_{\rm AFM}-E_{\rm
FM})/12$. In these monolayer materials all of the magnetic atoms are in the
same plane. The second neighbor exchange energies were previously determined
for VSi2P4 and found to be negligible compared to the nearest neighbor
energies, [24] and, therefore, they are ignored. The Curie temperatures are
determined from Monte Carlo (MC) calculations as implemented in the VAMPIRE
software package using a $10\times 10$ supercell.[43, 44] The MC calculations
incorporate the magnetic anisotropy energies, the magnetic moments, the
exchange energies, lattice constants, and atomic positions determined from the
DFT calculations, and they also include the long range dipolar interactions.
This MC approach has been used to determine the Curie temperatures in other
monolayer 2D materials such as VSi2P4, VI3, CrSBr, and CrSeBr. [24, 45, 46] We
note that the Curie temperatures in other monolayer 2D materials determined
from the MC approach have been shown to compare well to temperatures
determined from renormalized spin wave theory (RSWT). For example, the MC/RSWT
predicted Curie temperatures for monolayer CrSBr and CrSeBr were 168/150 K and
150/152 K, respectively.[46]

Figure 2: Spin resolved energy bands of VSi2N4 and VSi2P4 calculated with
(a,b) HSE06 and (c,d) PBE+U. Spin up bands are red and spin down bands are
blue. The two spin-split, narrow bands on either side of the Fermi level are
d-orbital bands centered on the transition metal.

The structures and the ground state orientations of the magnetic moments of
NbSi2N4, VSi2N4, and VSi2P4 are shown in Fig. 1, and the spin-resolved band
structure for VSi2N4 and VSi2P4, calculated with the HSE06 and PBE+U
functionals, are shown in Fig. 2. The band structure for all three materials
calculated with PBE, PBE+U, and HSE06 are shown in the Fig. S3 of the
Supplement, and the d-orbital resolved bands of VSi2N4 and VSi2P4 are shown in
Fig. S2. For all 3 materials at $\Gamma$, the two isolated narrow bands near
the Fermi level are spin-polarized, ${\rm d}_{z^{2}}-{\rm orbital}$ bands
centered on the transition metal atoms. Near K, these bands transition to
d${}_{x^{2}-y^{2}}$. The next higher conduction band is primarily
d${}_{x^{2}-y^{2}}$ at $\Gamma$, and it transitions to d${}_{z^{2}}$ at K. The
electronic structures from the HSE06 calculations show the following
properties. In VSi2N4 and VSi2P4, the higher spin-up conduction band at
$\Gamma$ crosses the lower spin-down band near K where the orbital composition
of the two bands switch, d${}_{z^{2}}$ $\Longleftrightarrow$
d${}_{x^{2}-y^{2}}$. VSi2N4 is direct gap (0.78 eV) at K with a
d${}_{x^{2}-y^{2}}$ spin-up valence band and a d${}_{z^{2}}$ spin-up
conduction band. VSi2P4 is indirect gap (0.84 eV) with the d${}_{z^{2}}$ spin-
up valence band edge at $\Gamma$ and a mixed d${}_{z^{2}}$ \+
d${}_{x^{2}-y^{2}}$ spin-down conduction band edge at M (the spin-up band at K
is 40 meV higher) . NbSi2N4 is indirect gap (0.54 eV) with the spin-up
d${}_{z^{2}}$ valence band at $\Gamma$ and a mixed orbital d${}_{z^{2}}$ \+
d${}_{x^{2}-y^{2}}$ spin-down conduction band near M. The lower valence bands
are primarily p-orbital bands which come from the N and P atoms. At the PBE
level of theory, all 3 materials are semi-metals. Adding the Hubbard U
correction creates a small gap at the Fermi level for VSi2N4 and VSi2P4.
Adding a percentage of exact exchange with the HSE06 functional increases the
gap substantially. We note that the HSE06 calculations match those reported in
the Supplement of Ref. [24].

The equilibrium NbSi2N4, VSi2N4, and VSi2P4 monolayers are easy-plane, semi-
conducting ferromagnets. The calculated equilibrium magnetic moments, exchange
energies, magnetic anisotropy energies, and Curie temperatures are shown in
Table 1 for different levels of theory. The magnetic moments are comparable to
those from prior studies.[19, 21] The positive MAE values indicate in-plane
alignment of the magnetic moments. Within the energy resolution of our
calculations (1 $\mu$eV), the total energy is independent of the angle of the
magnetic moment within the plane of the monolayer. Since the monolayers are
insulating, easy-plane FMs with extremely weak in-plane anisotropy, they
satisfy the necessary conditions for exhibiting a dissipationless spin
superfluid mode. [47]

Materials (Theory) |  Magnetic Moment ($\mu_{B}$) |  Exchange energy ($\times 10^{-21}$J) |  MAE ($\frac{\rm MJ}{\rm m^{3}}$) |  $T_{C}$ (K)
---|---|---|---|---
NbSi2N4 (PBE) |  $0.32$ |  $0.064$ |  $0.30$ |  $12$
VSi2N4 (PBE) |  $0.93$ |  $1.50$ |  $0.24$ |  $301$
VSi2P4 (PBE) |  $0.96$ |  $1.11$ |  $0.14$ |  $235$
VSi2N4 (PBE+U) |  $1.05$ |  $2.53$ |  $0.25$ |  $452$
VSi2P4 (PBE+U) |  $1.04$ |  $1.77$ |  $0.11$ |  $350$
VSi2N4 (HSE06) |  $1.19$ |  $2.80$ |  $--$ |  ${}^{*}506$
Table 1: Magnetic moment, exchange energy per link, equilibrium magnetic
anisotropy energy (MAE), and Curie temperature, calculated using different
functionals for NbSi2N4, VSi2N4, and VSi2P4. ${}^{*}T_{C}$ is calculated using
the HSE06 exchange energy and the PBE(SOC)+U MAE.

The normalized magnetizations, determined from Monte Carlo calculations, are
plotted as a function of temperature for VSi2N4 and VSi2P4 in Fig. 3. The
solid lines show the best fits to the analytic expression,

$m(T)=(1-T/T_{C})^{\beta}.$ (2)

The fitted values of $T_{C}$ and $\beta$ are shown on the plots. At the PBE+U
level of theory, both monolayer VSi2P4 and VSi2N4 have Curie temperatures
above room temperature, 350 K and 452 K, respectively. The effect of the
increasing levels of theory, PBE, PBE+U, and HSE06, is to successively
increase the bandgap between the two spin-polarized d${}_{z^{2}}$ bands, so
that at the HSE06 level, the two bands are completely gapped which maximizes
the spin polarization, magnetic moment, and the exchange constant. The
predicted $T_{C}$ increases with the increasing gap, giving a maximum value of
506 K for VSi2N4 using the exchange constant determined from the HSE06
calculation. Since VSi2N4 has the same structure, surface chemistry, and
formation energy[24] as the experimentally characterized MoSi2N4, we expect
VSi2N4 to also be an air/water-stable material with $T_{c}>100^{\circ}$ C,
which is a criterion for operation in a modern integrated circuit environment.

Figure 3: Monte Carlo calculations of the normalized magnetization as a
function of temperature for (a) VSi2N4 and (b) VSi2P4. The solid lines show
the best fits to the analytical expression given by Eq. (2).

For context, we compare to several other 2D materials with similar predicted
Curie or Néel temperatures. The calculated Curie-temperatures for monolayers
of the transition metal dichalcogenides (TMDs) VS2, VSe2, and VTe2 are $292$
K, $472$ K, and $553$ K, respectively.[48] The predicted monolayer Curie
temperatures for the ternary 2D materials CrSeCl, VSeTe, and CrSeI are $320$
K, $350$ K, and $360$ K, respectively.[49, 46, 16] Monolayers of ferromagnetic
semiconductors TcSiTe3 have a predicted Curie temperature of $538$ K.[50] We
note that Tc is radioactive, so that Tc compounds are unlikely to see magnetic
applications. Monolayers of RuI3, MnN, Co2S2, and 3R-MoN2 have Curie
temperatures of $360$ K, $368$ K, $404$ K, and $420$ K, respectively.[51, 52,
53, 54] We note that synthesis of layered 3R-MoN2 requires high pressure,[55]
which is not amenable to thin film growth techniques, and the proposed
graphitization synthesis route to achieve MnN monolayers[56] has not yet been
demonstrated. Within the list above, VSi2N4 stands out for its combination of
air stability and relatively high Curie temperature.

Figure 4: MAE as a function of uniaxial and biaxial strain calculated with
PBE(SOC)+U. MAE versus uniaxial strain: (a) VSi2N4 and (b) VSi2P4. MAE versus
biaxial strain: (c) VSi2N4 and (d) VSi2P4. Values are shown for the strain
coefficients for different regions discussed in the text. The orientation of
the magnetization of VSi2N4 rotates from in-plane to out-of-plane with 3.3%
biaxial strain.

The equilibrium values of the MAE are listed in Table 1. The equilibrium
values of VSi2N4 and VSi2P4 range from 0.11 to 0.25 meV / magnetic atom. For
comparison, monolayers of CrCl3, CrBr3, and CrI3 have equilibrium MAEs of
$0.02$ meV, $0.16$ meV, and $0.8$ meV, respectively.[57] Two-dimensional
FeCl2, NiI2, Fe3P, and Fe3GeTe2 have equilibrium MAE values of $0.07$ meV,
$0.11$ meV, $0.72$ meV, and $1$ meV, respectively in their monolayer
limit.[58, 59, 60, 61] In general, the MAE values of VSi2N4, and VSi2P4 are on
the lower end of values for other 2D magnetic materials.

The calculated values of the MAE of VSi2N4 and VSi2P4 as a function of
uniaxial strain and biaxial strain are shown in Fig. 4. We quantify the
sensitivity by defining a strain coefficient as $\alpha_{\epsilon}=dE_{\rm
MAE}/d\epsilon$. For small strain of both types in both materials, the values
for $\alpha_{\epsilon}$ are well below the value of 32 $\mu$eV/%strain
recently calculated for a 1.1 nm slab of CrSb.[62] For compressive uniaxial
and biaxial strain in VSi2N4, $\alpha_{\epsilon}=-10$ and $-6$
$\mu$eV/%strain, respectively. For VSi2P4, $\alpha_{\epsilon}=-6$
$\mu$eV/%strain for uniaxial tensile strain, -10 $\mu$eV/%strain for biaxial
tensile strain, and 17 $\mu$eV/%strain for biaxial compressive strain. For
large (3-4%) biaxial strain in VSi2N4, the magnitude of the sensitivity
increases. At a strain of 3.3%, the MAE changes sign, and the spins rotate
from in-plane to out-of-plane. This magnitude of strain should be
experimentally accessible, since 2D materials can sustain large strain.[63]

Figure 5: MAE as a function of band filling calculated with PBE(SOC)+U. MAE as
a function of excess electrons per unit cell for (a) VSi2N4 and (b) VSi2P4.
Values are shown for the filling coefficients discussed in the text. The
orientation of the magnetization of VSi2N4 rotates from in-plane to out-of-
plane with electron doping of $2.0\times 10^{14}$ cm-2.

The effects of band filling on the MAE are shown in Fig. 5. The MAE is reduced
in VSi2N4 and VSi2P4 for both electron and hole doing. To quantify the
sensitivity of the MAE to charge filling we define the parameter[62]
$\alpha_{n}=dE_{\rm MAE}/dn$ evaluated at zero filling. For electron filling
in VSi2N4, $\alpha_{n}=-0.29$ meV, and for hole filling in VSi2P4,
$\alpha_{n}=0.31$ meV. What is most interesting is that the sign of the MAE
can be reversed in VSi2N4 at an electron filling of 0.28 electrons per unit
cell corresponding to a sheet carrier concentration of $n_{s}=2.0\times
10^{14}$ cm-2. At this doping density, the orientation of the magnetization of
VSi2N4 rotates from in-plane to out-of-plane. These densities are high,
however since the thickness of the monolayer is under 1 nm, such densities
could be experimentally accessible using electrolytic gating.

In conclusion, NbSi2N4, VSi2N4, and VSi2P4 are easy-plane, semiconducting FMs
with negligible in-plane anisotropy. Exchange constants extracted from PBE+U
DFT used in MC calculations predict Curie temperatures for monolayer VSi2P4
and VSi2N4 of 350 K and 452 K, respectively. The effect of the increasing
levels of theory, PBE, PBE+U, and HSE06, is to successively increase the
energy gap between the two spin-polarized d${}_{z^{2}}$ bands, so that at the
HSE06 level, the two bands are completely gapped which maximizes the spin
polarization, magnetic moment, and the exchange constant. The predicted
$T_{C}$ increases with the increasing gap, giving a maximum value of 506 K for
VSi2N4 using the exchange constant determined from the HSE06 calculation.
Since VSi2N4 has the same structure, surface chemistry, and formation enthalpy
as MoSi2N4, it is expected to also be air and water stable. The magnetic
anisotropy energies of VSi2N4 and VSi2P4, determined by the difference in the
total energies resulting from out-of-plane versus in-plane alignment of the
magnetic moments, are small, ranging from 0.11 to 0.25 meV / magnetic atom.
Tensile biaxial strain of 3.3% in VSi2N4 causes the MAE to change sign such
that the magnetic moments rotate from in-plane to out-of-plane. Band filling
of 0.28 electrons per unit cell ($2.0\times 10^{14}$ cm-2) also causes VSi2N4
to switch from in-plane to perpendicular magnetic anisotropy.

Supplementary Material: The supplementary material contains plots of orbital
projected band diagrams, comparisons of electronic bandstructures calculated
with PBE, PBE+U, and HSE06 functionals, and MC calculations of the normalized
magnetization as a function of temperature with exchange constants extracted
from PBE DFT. It also contains an extended comparison of Curie temperatures of
2D materials.

Acknowledgement: This work used the Extreme Science and Engineering Discovery
Environment (XSEDE),[64] which is supported by National Science Foundation
Grant No. ACI-1548562 and allocation ID TG-DMR130081. MRKA thanks A. Bafekry
from University of Guilan, and San-Dong Guo from Xi’an University of Posts and
Telecommunications for providing initial crystal structures.

Data Availability Statement: The data that support the findings of this study
are available within the article and its supplementary material.

## References

  * Friend and Yoffe [1987] R. H. Friend and A. D. Yoffe, Adv. Phys. 36, 1 (1987).
  * Klemm [2012] R. A. Klemm,  _Layered Superconductors_ , Vol. 1 (Oxford University Press, New York, 2012).
  * de Jongh [1989] L. J. de Jongh, ed.,  _Magnetic Properties of Layered Transition Metal Compounds_ , Physics and Chemistry of Materials with Low-Dimensional Structures, Vol. 9 (Kluwer Academic Publishers, 1989).
  * Radisavljevic  _et al._ [2011] B. Radisavljevic, A. Radenovic, J. Brivio, V. Giacometti, and A. Kis, Nature nanotechnology 6, 147 (2011).
  * Kang  _et al._ [2020] S. Kang, D. Lee, J. Kim, A. Capasso, H. S. Kang, J.-W. Park, C.-H. Lee, and G.-H. Lee, 2D Materials 7, 022003 (2020).
  * Wang  _et al._ [2012] Q. H. Wang, K. Kalantar-Zadeh, A. Kis, J. N. Coleman, and M. S. Strano, Nature nanotechnology 7, 699 (2012).
  * Fiori  _et al._ [2014] G. Fiori, F. Bonaccorso, G. Iannaccone, T. Palacios, D. Neumaier, A. Seabaugh, S. K. Banerjee, and L. Colombo, Nature Nanotechnology 9, 768 (2014).
  * Li  _et al._ [2021] W. Li, X. Qian, and J. Li, Nature Reviews Materials , 1 (2021).
  * Huang  _et al._ [2017a] B. Huang, G. Clark, E. Navarro-Moratalla, D. R. Klein, R. Cheng, K. L. Seyler, D. Zhong, E. Schmidgall, M. A. McGuire, D. H. Cobden, W. Yao, D. Xiao, P. Jarillo-Herrero, and X. Xu, Nature 546, 270 (2017a).
  * Gong  _et al._ [2017] C. Gong, L. Li, Z. Li, H. Ji, A. Stern, Y. Xia, T. Cao, W. Bao, C. Wang, Y. Wang, Z. Q. Qiu, R. J. Cava, S. G. Louie, J. Xia, and X. Zhang, Nature 546, 265 (2017).
  * Deng  _et al._ [2018] Y. Deng, Y. Yu, Y. Song, J. Zhang, N. Z. Wang, Z. Sun, Y. Yi, Y. Z. Wu, S. Wu, J. Zhu, J. Wang, X. H. Chen, and Y. Zhang, Nature 563, 94 (2018).
  * Gibertini  _et al._ [2019] M. Gibertini, M. Koperski, A. F. Morpurgo, and K. S. Novoselov, Nature Nanotechnology 14, 408 (2019).
  * Li  _et al._ [2019] H. Li, S. Ruan, and Y.-J. Zeng, Advanced Materials 31, 1900065 (2019).
  * Chittari  _et al._ [2020] B. L. Chittari, D. Lee, N. Banerjee, A. H. MacDonald, E. Hwang, and J. Jung, Phys. Rev. B 101, 085415 (2020).
  * Kabiraj  _et al._ [2020] A. Kabiraj, M. Kumar, and S. Mahapatra, npj Computational Materials 6, 35 (2020).
  * Han  _et al._ [2020a] R. Han, Z. Jiang, and Y. Yan, The Journal of Physical Chemistry C 124, 7956 (2020a).
  * Jothi  _et al._ [2020] P. R. Jothi, J. P. Scheifers, Y. Zhang, M. Alghamdi, D. Stekovic, M. E. Itkis, J. Shi, and B. P. T. Fokwa, Physica Status Solidi (RRL) - Rapid Research Letters 14, 1900666 (2020).
  * Khan  _et al._ [2020] Y. Khan, S. M. Obaidulla, M. R. Habib, A. Gayen, T. Liang, X. Wang, and M. Xu, Nano Today 34, 100902 (2020).
  * Hong  _et al._ [2020] Y.-L. Hong, Z. Liu, L. Wang, T. Zhou, W. Ma, C. Xu, S. Feng, L. Chen, M.-L. Chen, D.-M. Sun, X.-Q. Chen, H.-M. Cheng, and W. Ren, Science 369, 670 (2020).
  * Gillgren  _et al._ [2014] N. Gillgren, D. Wickramaratne, Y. Shi, T. Espiritu, J. Yang, J. Hu, J. Wei, X. Liu, Z. Mao, K. Watanabe, T. Taniguchi, M. Bockrath, Y. Barlas, R. K. Lake, and C. N. Lau, 2D Materials 2, 011001 (2014).
  * Guo  _et al._ [2020] S.-D. Guo, W.-Q. Mu, Y.-T. Zhu, and X.-Q. Chen, Phys. Chem. Chem. Phys. 22, 28359 (2020).
  * Cao  _et al._ [2021] L. Cao, G. Zhou, Q. Wang, L. K. Ang, and Y. S. Ang, Applied Physics Letters 118, 013106 (2021).
  * Li  _et al._ [2020] S. Li, W. Wu, X. Feng, S. Guan, W. Feng, Y. Yao, and S. A. Yang, Phys. Rev. B 102, 235435 (2020).
  * Wang  _et al._ [2021] L. Wang, Y. Shi, M. Liu, A. Zhang, Y.-L. Hong, R. Li, Q. Gao, M. Chen, W. Ren, H.-M. Cheng, Y. Li, and X.-Q. Chen, Nature Communications 12, 2361 (2021).
  * Bafekry  _et al._ [2020] A. Bafekry, M. Faraji, D. M. Hoat, M. M. Fadlallah, M. Shahrokhi, F. Shojaei, D. Gogova, and M. Ghergherehchi, arXiv:2009.04267v1 , 1 (2020).
  * Yu  _et al._ [2021] J. Yu, J. Zhou, X. Wan, and Q. Li, New Journal of Physics 23, 033005 (2021).
  * Wu  _et al._ [2021] Q. Wu, L. Cao, Y. S. Ang, and L. K. Ang, Applied Physics Letters 118, 113102 (2021).
  * Zhong  _et al._ [2021] H. Zhong, W. Xiong, P. Lv, J. Yu, and S. Yuan, Phys. Rev. B 103, 085124 (2021).
  * Yao  _et al._ [2021] H. Yao, C. Zhang, Q. Wang, J. Li, Y. Yu, F. Xu, B. Wang, and Y. Wei, Nanomaterials 11, 559 (2021).
  * Kresse and Hafner [1993] G. Kresse and J. Hafner, Phys. Rev. B 47, 558 (1993).
  * Blochl [1994] P. E. Blochl, Phys. Rev. B 50, 17953 (1994).
  * Perdew  _et al._ [1996] P. Perdew, K. Burke, and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996).
  * Dudarev  _et al._ [1998] S. L. Dudarev, G. A. Botton, S. Y. Savrasov, C. J. Humphreys, and A. P. Sutton, Phys. Rev. B 57, 1505 (1998).
  * Heyd  _et al._ [2003] J. Heyd, G. E. Scuseria, and M. Ernzerhof, The Journal of Chemical Physics 118, 8207 (2003).
  * Krukau  _et al._ [2006] A. V. Krukau, O. A. Vydrov, A. F. Izmaylov, and G. E. Scuseria, The Journal of Chemical Physics 125, 224106 (2006).
  * Kinaci  _et al._ [2015] A. Kinaci, M. Kado, D. Rosenmann, C. Ling, G. Zhu, D. Banerjee, and M. K. Y. Chan, Applied Physics Letters 107, 262108 (2015).
  * Wang  _et al._ [2019] Y. Wang, D. Puggioni, and J. M. Rondinelli, Phys. Rev. B 100, 115149 (2019).
  * Zimmermann  _et al._ [2019] B. Zimmermann, G. Bihlmayer, M. Böttcher, M. Bouhassoune, S. Lounis, J. Sinova, S. Heinze, S. Blügel, and B. Dupé, Phys. Rev. B 99, 214426 (2019).
  * Schweflinghaus  _et al._ [2016] B. Schweflinghaus, B. Zimmermann, M. Heide, G. Bihlmayer, and S. Blügel, Phys. Rev. B 94, 024403 (2016).
  * Toyoda  _et al._ [2013] M. Toyoda, K. Yamauchi, and T. Oguchi, Phys. Rev. B 87, 224430 (2013).
  * Xiang  _et al._ [2011] H. J. Xiang, E. J. Kan, S.-H. Wei, M.-H. Whangbo, and X. G. Gong, Phys. Rev. B 84, 224429 (2011).
  * Akanda  _et al._ [2020] M. R. K. Akanda, I. J. Park, and R. K. Lake, Phys. Rev. B 102, 224414 (2020).
  * [43] VAMPIRE software package version 5.0 available from https://vampire.york.ac.uk.
  * Evans  _et al._ [2014] R. F. L. Evans, W. J. Fan, P. Chureemart, T. A. Ostler, M. O. A. Ellis, and R. W. Chantrell, J. Phys.: Condens. Matter 26, 103202 (2014).
  * Subhan and Hong [2020] F. Subhan and J. Hong, Journal of Physics: Condensed Matter 32, 245803 (2020).
  * Wang  _et al._ [2020] H. Wang, J. Qi, and X. Qian, Applied Physics Letters 117, 083102 (2020).
  * Takei and Tserkovnyak [2014] S. Takei and Y. Tserkovnyak, Phys. Rev. Lett. 112, 227201 (2014).
  * Fuh  _et al._ [2016] H.-R. Fuh, C.-R. Chang, Y.-K. Wang, R. F. L. Evans, R. W. Chantrell, and H.-T. Jeng, Scientific Reports 6, 32625 (2016).
  * Guan and Ni [2020] Z. Guan and S. Ni, Nanoscale 12, 22735 (2020).
  * You  _et al._ [2020] J.-Y. You, Z. Zhang, X.-J. Dong, B. Gu, and G. Su, Phys. Rev. Research 2, 013002 (2020).
  * Zhang  _et al._ [2017] Y. Zhang, J. Pang, M. Zhang, X. Gu, and L. Huang, Scientific Reports 7, 15993 (2017).
  * Huang  _et al._ [2017b] C. Huang, J. Zhou, H. Wu, K. Deng, P. Jena, and E. Kan, Phys. Rev. B 95, 045113 (2017b).
  * Wu  _et al._ [2015] F. Wu, C. Huang, H. Wu, C. Lee, K. Deng, E. Kan, and P. Jena, Nano Letters 15, 8277 (2015), pMID: 26575002\.
  * Xu and Zhu [2018] Z. Xu and H. Zhu, The Journal of Physical Chemistry C 122, 14918 (2018).
  * Wang  _et al._ [2015] S. Wang, H. Ge, S. Sun, J. Zhang, F. Liu, X. Wen, X. Yu, L. Wang, Y. Zhang, H. Xu, J. C. Neuefeind, Z. Qin, C. Chen, C. Jin, Y. Li, D. He, and Y. Zhao, Journal of the American Chemical Society 137, 4815 (2015), pMID: 25799018.
  * Sorokin  _et al._ [2014] P. B. Sorokin, A. G. Kvashnin, Z. Zhu, and D. Tománek, Nano Letters 14, 7126 (2014), pMID: 25384500\.
  * Webster and Yan [2018] L. Webster and J.-A. Yan, Phys. Rev. B 98, 144411 (2018).
  * Zhuang  _et al._ [2016] H. L. Zhuang, P. R. C. Kent, and R. G. Hennig, Phys. Rev. B 93, 134407 (2016).
  * Torun  _et al._ [2015] E. Torun, H. Sahin, C. Bacaksiz, R. T. Senger, and F. M. Peeters, Phys. Rev. B 92, 104407 (2015).
  * Zheng  _et al._ [2019] S. Zheng, C. Huang, T. Yu, M. Xu, S. Zhang, H. Xu, Y. Liu, E. Kan, Y. Wang, and G. Yang, The Journal of Physical Chemistry Letters 10, 2733 (2019).
  * Han  _et al._ [2020b] H. Han, H. Zheng, Q. Wang, and Y. Yan, Phys. Chem. Chem. Phys. 22, 26917 (2020b).
  * Park  _et al._ [2020] I. J. Park, S. Kwon, and R. K. Lake, Phys. Rev. B 102, 224426 (2020).
  * Peng  _et al._ [2014] X. Peng, Q. Wei, and A. Copple, Phys. Rev. B 90, 085402 (2014).
  * Towns  _et al._ [2014] J. Towns, T. Cockerill, M. Dahan, I. Foster, K. Gaither, A. Grimshaw, V. Hazlewood, S. Lathrop, D. Lifka, G. D. Peterson, _et al._ , Computing in Science & Engineering 16, 62 (2014).


# Meaning without reference in large language models

Steven T. Piantadosi
Department of Psychology
Helen Wills Neuroscience Institute
University of California, Berkeley
<EMAIL_ADDRESS>Felix Hill
DeepMind
<EMAIL_ADDRESS>
###### Abstract

The widespread success of large language models (LLMs) has been met with
skepticism that they possess anything like human concepts or meanings.
Contrary to claims that LLMs possess no meaning whatsoever, we argue that they
likely capture important aspects of meaning, and moreover work in a way that
approximates a compelling account of human cognition in which meaning arises
from _conceptual role_. Because conceptual role is defined by the
relationships between internal representational states, meaning cannot be
determined from a model’s architecture, training data, or objective function,
but only by examination of how its internal states relate to each other. This
approach may clarify why and how LLMs are so successful and suggest how they
can be made more human-like.

Meaning without reference in large language models

Steven T. Piantadosi Department of Psychology Helen Wills Neuroscience
Institute University of California, Berkeley<EMAIL_ADDRESS>Felix Hill
DeepMind<EMAIL_ADDRESS>

Large language models (LLMs) have begun to display an array of competencies
that were long thought to be out of the reach of neural networks. At the same
time, critics have been vocal that existing methods, model structures, and
training paradigms will be insufficient for anything like human language use.
In particular, it is argued that LLMs will never achieve “meaning” or
“understanding” due to either the objective function they optimize, the format
of their internal representations, or the type of training data that they
receive. This short comment aims to contest the claims that LLMs are
necessarily incapable of acquiring meaning, and suggest that LLM meanings may
already be similar to human-like meaning in many (but not all) ways. We argue
that LLMs have likely already achieved some key aspects of meaning which,
while imperfect, mirror the way meanings work in cognitive theories, as well
as approaches to the philosophy of language.

LLMs are trained to predict words from massive datasets of text from the
internet. They typically contain billions of parameters that are jointly
optimized—at great computational, energy, and financial expense (Bender et
al., 2021)—to make predictions about word occurrence from surrounding context.
This setup differs from human language acquisition in data scale and format.
On the other hand, the core objective of word prediction is a central piece of
human language processing (e.g. Altmann and Kamide, 1999; Hale, 2001; Levy,
2008) and has long been shown capable of providing a learning signal from
which linguistic structures and semantic categories can emerge (Elman, 1990).

A key debate in recent literature has been about _what_ models trained in this
manner come to know. A prominent view is that models trained only on text
cannot acquire realistic meanings because they lack reference, or connection
to objects in the real world. Bender and Koller (2020) illustrate this with an
“octopus test.” They imagine an octopus that learns to use words correctly by
eavesdropping on a conversation between two people on land. If the octopus has
no access to the referents of the words then there are gaps in its meaning for
the words. For example, if the octopus must suddenly determine which _object_
is a coconut, then its expertise in using the word “coconut” won’t help. Its
knowledge of co-occurrence statistics between “coconut” and other words won’t
help either since it also knows nothing about the referents of other words.
The octopus simply does not have the required meanings to find the coconut.
Humans learners don’t have this problem because their input—and consequently
representations—are tied to real-world referents. Bender & Koller’s position
is that no amount of predictive linguistic savvy can give models that are
trained on text alone the knowledge of reference they need to acquire
meanings.

### Meaning and reference

The octopus test assumes that reference determines meaning, but in fact
cognitive scientists and philosophers have found a variety of problems with
this view. One is that there are many terms that are meaningful to us but have
no discernible referent at all, such as abstract words like “justice” and
“wit.” People can think of new concepts like “aphid-sized accordion” that
don’t exist (thus no referent), or even terms that have no _possible_ referent
like “perpetual motion machine” or “imaginary cup of tea.” We can think of
concepts like “king of San Francisco” that pick out nobody, but are at least
meaningful enough to reason about (for example: “If there was a King of San
Francisco, he’d live in The Presidio.”). Other examples show reference can be
quite decoupled from meaning. Terms like “treaty” and “contract” are often
thought to have a concrete referent, but what is important is actually an
abstract entity: a treaty is still valid if the piece of paper is destroyed.
Frege’s example is of the “morning star” and the “evening star.” Both are
terms for the planet Venus, but were once conceived of as different entities
without knowing that they are the same object.

This problem is not solely an issue with abstract concepts. Many concrete
terms seem not to get their most important semantic properties from reference
either. Consider an example like a “postage stamp.” Everyone can conjure up an
image of a typical, physical, postage stamp. But with further scrutiny, none
of the concrete features of typical stamps seem necessary. We can imagine a
country whose postage stamps were made of clear glass, for example, or stamps
that were microscopic so that they could not be seen, or stamps that were paid
for and tracked entirely online, or others that were larger than a house (for
mailing very large packages). We could think of postage stamps that were
_RFID_ tags that went inside the envelope. Maybe intelligent ants would use
postage stamps that were pheromones rather than paper; in the future we might
have postage stamps that sprout wings and fly your letter to its intended
location. Already we have stamps that only have barcodes and stamps that you
can draw yourself. You know the term “postage stamp” but there is no way that
you could have considered all of the possible referents yet, so reference
cannot be what determines the concept.

Similar arguments were made by Wittgenstein (1953) when considering the
concept of “water”, leading him to conclude that reference plays little or no
role in determining meaning in the general case. One way to explain these
observations is to assume that our meaning for terms like “postage stamp” (or
“water”) may be primarily determined by the role these concepts play in some
greater mental theory. Very roughly, people call something a “postage stamp”
if you pay for it and then attach it to a letter in order to have the letter
to be delivered. In this view, the meaning of the word is intrinsically
intertwined with other concepts like “payment”, “letter” and “delivery.” The
interrelation is key (see, e.g. Deacon, 1998; Santoro et al., 2021) because
when the associated terms shift in meaning even slightly (e.g. credit cards
were developed as wholly new a form of “payment”) the meaning of “postage
stamp” comes along for the ride (we know right away that they can be paid for
by a credit card). Such relationships between concepts are _the_ essential,
defining, aspects of meaning and, in fact, possessing the appropriate
relationships allows you to determine the reference. This view makes sense of
the puzzling examples above.

### Conceptual role theory

In philosophy of mind, versions of this approach go by the name “conceptual
role theory.” Following Block (1998), consider statements in physics like
$f=m\cdot a$. This equation is not exactly a definition of $f$ (_force_), nor
is it a definition of _mass_ ($m$) or _acceleration_ ($a$), but, is a
statement about the interrelationship of $f$, $m$, and $a$. Most of us can’t
give much more detail about the ultimate physical reference of these
terms—forces have something to do with interacting elementary particles
(whatever those are), masses have something to do with the Higgs field
(whatever that is)—but our thoughts about $f=m\cdot a$ certainly do not seem
meaningless. Many believe that conceptual roles are one of the most promising
ways to characterize human concepts _in general_ (for an overview of this and
competing theories, see, e.g., Margolis and Laurence, 1999). Murphy and Medin
(1985) for example argue that our organization of categories is based on
entire theories of structured conceptual domains (rather than simple features
or similarities), an idea they trace to Quine (1977). Murphy and Medin note
how we might reflexively consider a composite category such as “prime numbers
or apples” to be an unnatural or incoherent set of entities. However, if we
know someone called Wilma who is a number theorist grew up on an apple farm,
then the category “topics of conversation with Wilma”, involving the same
constituent entities, seems perfectly reasonable. What is a natural category
or concept depends on our mental conception of how the underlying pieces
relate, and concepts can even be assembled fluidly, in an ad hoc manner or
context-dependent manner (Barsalou, 1983; Casasanto and Lupyan, 2015).
Theories in cognition have also been explored in learning models (e.g. Goodman
et al., 2011; Ullman et al., 2012) and experimentally shown to shape how
children explore the world (e.g. Gopnik et al., 1999; Gopnik and Schulz, 2004;
Bonawitz et al., 2012).

### Conceptual role in LLMs

If anything like this view is correct, then the search for meaning in learning
models—or brains—should focus on understanding the way that the systems’
internal representational states _relate to each other_. Once a learning model
finds it probable that “postage stamps” are “affixed” to “letters” so they can
be “delivered,” then it has acquired some important pieces of conceptual role
for these terms. It would not be possible to conclude anything about what
meanings a system does and does not possess from its training data or
architecture because these may not be informative about how the internal
states relate to each other.

Relations between internal states have been long emphasized in cognitive
theories (Shepard and Chipman, 1970; Deacon, 1998; Fodor and Pylyshyn, 1988),
for example early attempts to discover the geometry of psychological space
(Shepard, 1980) and more recent analyses of brain data based on
representational similarity (Kriegeskorte et al., 2008). Elman (2004) argues
for a closely related view of the mental lexicon in which a word’s meaning is
the effect it has on other mental states. In deep learning models, the
relational geometry of vector representations have been examined for instance
in analogy problems (Mikolov et al., 2013), match to human similarities (Hill
et al., 2017), and encoding of humanlike gradient distinctions (Vulić et al.,
2017). Grand et al. (2022) show that semantic embeddings from these models
capture gradient scales of multiple features, like from “small” to “big” or
“safe” to “dangerous.” It is even possible to align the word representations
acquired by text-based models across languages to translate between them
effectively with no prior knowledge of which words or phrases should have the
same meaning (Lample et al., 2017). Similarly, Abdou et al. (2021) show that a
model trained on text can recover key geometry of color space, even without
grounding; with a few examples of grounding, LLMs are able to align their
structure with the real grounded one, suggesting that they already possess the
right relations Patel and Pavlick (2021). Importantly, as the performance of
LLMs has improved in recent years the extent to which their relational
geometry reflects human data has also consistently increased (Peters et al.,
2018; Devlin et al., 2018; Brown et al., 2020). Larger models also better
reflect the human tendency for semantic or mental models to influence formal
reasoning behaviour (Wason and Johnson-Laird, 1972), on challenging logic
problems that are not observed in their training data (Dasgupta et al., 2022).
Moreover, this increasing correspondence between LLMs and human data is not
observed only behaviourally. Recent fMRI studies show that the semantic models
that best account for the representational geometry and processing activity of
human brains are precisely the neural network LLMs which are trained on the
largest amount of data (Schrimpf et al., 2021; Goldstein et al., 2022; Kumar
et al., 2022).

Many of the tasks that LLMs succeed on are ones that require maintaining the
right relationships between concepts. Impressively, the largest models can now
devise coherent narratives (Brown et al., 2020), extend stories (Xu et al.,
2020; Li et al., 2021), answer factual questions (Jiang et al., 2021) solve
Winograd Schema (Kocijan et al., 2020) and resolve complex quantitative
reasoning problems (Lewkowycz et al., 2022). Increasingly, such models are
even aware of the likelihood that they can answer a given question correctly;
i.e. they have an explicit sense of the extent of their own knowledge
(Kadavath et al., 2022). Each of these capacities requires, in some way or
another, sensitivity to conceptual roles because the required words and
concepts must be used jointly together in a coherent way that mimics how
humans would.

Despite these empirical successes, there are many places where these models
can still be improved (for detailed analyses, see, Lake and Murphy (2021);
McClelland et al. (2020); Pavlick (2022)). Lake and Murphy (2021) emphasize
the need for reference, inference, better and more robust compositionality,
more structure and more consistent abstract reasoning. Models trained on
multimodal datasets show better match to human judgements than those trained
on text alone (Hill et al., 2016; De Deyne et al., 2021); at the same time,
even multimodal LLMs are missing many aspects of a complete theory of
semantics, including the ability to simulate situations in which their
physical or linguistic behaviour affects their environment (McClelland et al.,
2020) as well as knowledge of the goals and desires that drive how people use
words (Bisk et al., 2020; Lake and Murphy, 2021).

Our claim, then, is not that LLMs perfectly capture human concepts or
perfectly reflect human meaning. Unlike the more radical perspectives
entertained by Wittgenstein or Quine, we also do not consider that reference
should play _no_ role in a principled treatment of meaning. Instead, we find
it productive to consider reference as just one (optional) aspect of a word’s
full conceptual role. It is relevant for some concepts (see Putnam, 1974) and
not others—just like color, valence, or teleology is relevant for some
concepts and not others. Experience of both agency and a perceptual
environment similar to our own may lead to the richest, most human-like
understanding of language in machines (Bisk et al., 2020; McClelland et al.,
2020).

As these improvements are made, the models will come into closer alignment
with humans, and each such improvement will enrich the model’s sense of
meaning. This process of progressive enrichment is also found in human
concepts. When people discovered that $H_{2}O$ was the chemical composition of
water, they grew their conceptual network and even revised their reference for
the term. There was no hard transition from a meaningless concept of “water”
to a meaningful one. Some meaning was there all along because “water” had a
conceptual role even before its chemical composition was known. What changed
was the richness and interconnection of this concept—the way in which it was
related to other concepts like “hydrogen” and “oxygen.” In much the same way,
we see no reason to assume that the world of a system that receives input from
a single sensory modality is meaningless, even if the addition of further
sensors provides clear enrichment. When thinking about improving LLMs it we
should therefore consider ways to enrich the internal conceptual roles of
these systems, including to better reflect the structure, inference and
algorithmic sophistication of humans (Tenenbaum et al., 2011; Lake and Murphy,
2021; Rule et al., 2020).

### Discovering conceptual role

Conceptual role theory also provides a compelling way to understand learning,
including the way in which neural networks may come to represent symbolic
processes. A symbol like $AND$ only means logical conjunction if it interacts
(composes) with others symbols like $TRUE$ and $FALSE$ in the appropriate
way—i.e. when it has the right conceptual role in the broader system of
symbols. The technique of _church-encoding_ in mathematical logic (see Pierce,
2002) provides a way to understand how such roles may be learned within neural
networks or dynamical systems Piantadosi (2021). In church-encoding, a
representation is constructed in one system (e.g. lambda calculus or a neural
network) in order to mimic the behavior of another system (e.g. boolean logic)
in the sense that the representations in the first system interact with each
other in a way that yields the desired conceptual roles of the second.
Piantadosi (2021) shows how a church-encoding learner could acquire structures
like logic, lists, trees, hierarchies, numbers, quantifiers, and recursion
without possessing them to start, and how this metaphor may provide an
“assembly language” that translates from symbolic computational or cognitive
theories into underlying implementations. In this view, neural networks would
train their parameters so that their internal, intrinsic dynamics church-
encode the conceptual roles of a targeted domain.

The key question for LLMs is whether training to predict text could actually
support discovery of conceptual roles. To us the question is empirical, and we
believe has been answered in a promising, partial affirmative by studies
showing success on tasks that require knowledge of relationships between
concepts. Text provides such clues to conceptual role because human conceptual
roles generated the text. Analogously, it is possible to build a theory of
gravity from measurements of the moon’s movement because gravity _generated_
these movements; the goal of essentially all inductive learning techniques is
to invert from observations to likely generating processes or parameters.
Moreover, the entailment relationships between sentences are often
intrinsically related to analogous patterns in thought (e.g. Fodor and
Pylyshyn, 1988), meaning that a model which captures how sentences relate to
each other might indeed capture how thoughts relate to each other. One helpful
analogy is that of _embedding theorems_ in dynamical systems (e.g. Packard et
al., 1980; Takens, 1981) which allow some properties of systems to be
recovered from seemingly impoverished representations of their state. In the
paper “Geometry from a Time Series”, Packard et al. (1980) show, remarkably,
that one can sometimes reconstruct the geometry of a multi-dimensional
dynamical system from a _one_ -dimensional projection of its state. Thus,
information about high-dimensional state (sometimes essentially all of it) can
be decoded from the trajectory of low-dimensional projection. People use
concepts in thinking and reasoning based on their meaning, and text is a low-
dimensional projection of some of these patterns of use, so it is plausible
that some properties of the real meaning could be inferred from text. At the
very least, embedding theorems illustrate that there may not be a simple way
to intuit what a learner can or cannot deduce about the underlying mental
states from text alone.

The protein folding neural network AlphaFold (Jumper et al., 2021) provides
further evidence that transformer-based networks can infer and generalise
complex latent multi-entity structures. AlphaFold is trained to predict the
configuration of single proteins only, but acquires actionable knowledge of
concepts not explicitly present in its training data. Despite never seeing a
zinc ion, AlphaFold often perfectly infers its location and places all the
protein side chains correctly right around it. Some proteins only fold with
multiple copies of themselves (homomers). Again, AlphaFold has never seen more
than one copy of a protein, but often infers both the number of required
copies and their relative placement correctly. This suggests that the process
of predicting the structure of single proteins enabled the AlphaFold network
to infer non-trivial facts about chemistry and biology.

### Communicative intentions

Separable from meaning and reference, many have also rejected the idea that
LLMs produce language with _intention_ (Bender et al., 2021; Bender and
Koller, 2020). Because LLMs are trained only on sequence prediction, they are
argued to be, “stochastic parrots” Bender et al. (2021) or just sophisticated
“auto-complete” algorithms111https://garymarcus.substack.com/p/nonsense-on-
stilts that cannot access the intentions of those who produced their training
data, and themselves produce language without intending anything in
particular. But in our view, a key difference between autocompleting parrots
and LLMs is that the latter have rich, causal, and structured internal states.

One view of intent is semantic, corresponding to whether the language they
produce arises from an internal representation of (intended) meaning. The
conversion of internal states into language and back _is_ the essential
function of LLMs, embedded in their architecture and training. We have argued
that LLM’s internal state has some notions of conceptual role, so LLM’s
utterances have the semantic intent corresponding to these roles.

Another view of intent is pragmatic, asking what might be achieved by
producing a sentence. This corresponds to asking whether they engage in any
goal-directed _planning_ (Russell, 2010) when producing language. We consider
it probable that multi-layer LLMs do, in an emergent, implicit sense, execute
a form of planning as part of the process of repeated (self-attention-based)
analyses of current and past inputs. These computations likely involve
representation of the current situation and at least implicit evaluation of
consequences of utterances. Recent work has argued that LLMs posses model-like
belief structures Hase et al. (2021) and update representations of dynamic
semantics, objects and situations, throughout a discourse Li et al. (2021).
These emergent semantics causally determine LLM output. Of course, differences
between humans and LLMs in their training experience and objectives mean that
the planning process in LLMs is less explicit and sophisticated than in humans
(making errors more likely, for instance, in cases of hypothetical or
counterfactual reasoning (Ortega et al., 2021)).

### Conclusion

Bender & Koller argue that text-based LLMs will never have meaning because
these models lack reference. However, they do not demonstrate that reference
is the key to meaning—instead they assume it. As we have argued, this
assumption is hard to reconcile with theories of cognition and the phenomena
that motivate them. People are happy to think about concepts without referents
and otherwise often don’t know many details of reference. Meaning instead
seems to come from the way concepts relate to _each other_. It is these
interrelations that LLMs know something about since their internal geometries
and trajectories approximate those of humans. Like people who don’t know that
water is $H_{2}O$ and so could not pick it out based on chemical composition,
Bender & Koller’s octopus lacks some aspects of conceptual role like physical
appearance. But, both the octopus and people know other parts of conceptual
role that are sophisticated in their own right. If theories about conceptual
role are the correct account, then LLMs likely already share the foundation of
how our own concepts get their meaning.

## Acknowledgements

We are grateful to Ev Fedorenko, Adam Santoro, MH Tessler, Dileep George, John
Hale, Miguel Figurnov, John Jumper, Dharsh Kumaran, Matt Botvinick, Paul
Smolensky and Chris Dyer for detailed comments.

## References

  * Abdou et al. (2021) Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. 2021.  Can language models encode perceptual structure without grounding? A case study in color.  _arXiv preprint arXiv:2109.06129_.
  * Altmann and Kamide (1999) Gerry TM Altmann and Yuki Kamide. 1999.  Incremental interpretation at verbs: Restricting the domain of subsequent reference.  _Cognition_ , 73(3):247–264.
  * Barsalou (1983) Lawrence W Barsalou. 1983.  Ad hoc categories.  _Memory & cognition_, 11(3):211–227.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pages 610–623.
  * Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020.  Climbing towards NLU: On meaning, form, and understanding in the age of data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5185–5198, Online. Association for Computational Linguistics.
  * Bisk et al. (2020) Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. 2020.  Experience grounds language.  _arXiv preprint arXiv:2004.10151_.
  * Block (1998) Ned Block. 1998.  Semantics, conceptual role.  _Routledge encyclopedia of philosophy_ , 8:652–657.
  * Bonawitz et al. (2012) Elizabeth Baraff Bonawitz, Tessa JP van Schijndel, Daniel Friel, and Laura Schulz. 2012.  Children balance theories and evidence in exploration, explanation, and learning.  _Cognitive psychology_ , 64(4):215–234.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Casasanto and Lupyan (2015) Daniel Casasanto and Gary Lupyan. 2015.  All Concepts Are Ad Hoc Concepts.  In E. Margolis and S. Laurence, editors, _The Conceptual Mind: New directions in the study of concepts_ , pages 543–566. Cambridge: MIT Press.
  * Dasgupta et al. (2022) Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022.  Language models show human-like content effects on reasoning.  _arXiv preprint arXiv:2207.07051_.
  * De Deyne et al. (2021) Simon De Deyne, Danielle J Navarro, Guillem Collell, and Andrew Perfors. 2021.  Visual and affective multimodal models of word meaning in language and mind.  _Cognitive Science_ , 45(1):e12922.
  * Deacon (1998) Terrence William Deacon. 1998.  _The symbolic species: The co-evolution of language and the brain_.  202\. WW Norton & Company.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Elman (1990) Jeffrey L Elman. 1990.  Finding structure in time.  _Cognitive science_ , 14(2):179–211.
  * Elman (2004) Jeffrey L Elman. 2004.  An alternative view of the mental lexicon.  _Trends in cognitive sciences_ , 8(7):301–306.
  * Fodor and Pylyshyn (1988) Jerry A Fodor and Zenon W Pylyshyn. 1988.  Connectionism and cognitive architecture: A critical analysis.  _Cognition_ , 28(1-2):3–71.
  * Goldstein et al. (2022) Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. 2022.  Shared computational principles for language processing in humans and deep language models.  _Nature neuroscience_ , 25(3):369–380.
  * Goodman et al. (2011) Noah D Goodman, Tomer D Ullman, and Joshua B Tenenbaum. 2011.  Learning a theory of causality.  _Psychological review_ , 118(1):110.
  * Gopnik et al. (1999) Alison Gopnik, Andrew N Meltzoff, and Patricia K Kuhl. 1999.  _The scientist in the crib: Minds, brains, and how children learn._ William Morrow & Co.
  * Gopnik and Schulz (2004) Alison Gopnik and Laura Schulz. 2004.  Mechanisms of theory formation in young children.  _Trends in cognitive sciences_ , 8(8):371–377.
  * Grand et al. (2022) Gabriel Grand, Idan Asher Blank, Francisco Pereira, and Evelina Fedorenko. 2022\.  Semantic projection recovers rich human knowledge of multiple object features from word embeddings.  _Nature Human Behaviour_ , pages 1–13.
  * Hale (2001) John Hale. 2001.  A probabilistic earley parser as a psycholinguistic model.  In _Second meeting of the north american chapter of the association for computational linguistics_.
  * Hase et al. (2021) Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. 2021.  Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs.  _arXiv preprint arXiv:2111.13654_.
  * Hill et al. (2017) Felix Hill, Kyunghyun Cho, Sébastien Jean, and Yoshua Bengio. 2017.  The representational geometry of word meanings acquired by neural machine translation models.  _Machine Translation_ , 31(1):3–18.
  * Hill et al. (2016) Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016.  Learning distributed representations of sentences from unlabelled data.  _arXiv preprint arXiv:1602.03483_.
  * Jiang et al. (2021) Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021.  How can we know when language models know? on the calibration of language models for question answering.  _Transactions of the Association for Computational Linguistics_ , 9:962–977.
  * Jumper et al. (2021) John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. 2021.  Highly accurate protein structure prediction with alphafold.  _Nature_ , 596(7873):583–589.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022.  Language models (mostly) know what they know.
  * Kocijan et al. (2020) Vid Kocijan, Thomas Lukasiewicz, Ernest Davis, Gary Marcus, and Leora Morgenstern. 2020.  A review of winograd schema challenge datasets and approaches.  _arXiv preprint arXiv:2004.13831_.
  * Kriegeskorte et al. (2008) Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. 2008.  Representational similarity analysis-connecting the branches of systems neuroscience.  _Frontiers in systems neuroscience_ , page 4.
  * Kumar et al. (2022) Sreejan Kumar, Theodore R Sumers, Takateru Yamakoshi, Ariel Goldstein, Uri Hasson, Kenneth A Norman, Thomas L Griffiths, Robert D Hawkins, and Samuel A Nastase. 2022.  Reconstructing the cascade of language processing in the brain using the internal computations of a transformer-based language model.  _bioRxiv_.
  * Lake and Murphy (2021) Brenden M Lake and Gregory L Murphy. 2021.  Word meaning in minds and machines.  _Psychological review_.
  * Lample et al. (2017) Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017\.  Unsupervised machine translation using monolingual corpora only.  _arXiv preprint arXiv:1711.00043_.
  * Levy (2008) Roger Levy. 2008.  Expectation-based syntactic comprehension.  _Cognition_ , 106(3):1126–1177.
  * Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022.  Solving quantitative reasoning problems with language models.  _arXiv preprint arXiv:2206.14858_.
  * Li et al. (2021) Belinda Z Li, Maxwell Nye, and Jacob Andreas. 2021.  Implicit representations of meaning in neural language models.  _arXiv preprint arXiv:2106.00737_.
  * Margolis and Laurence (1999) Eric Ed Margolis and Stephen Ed Laurence. 1999.  _Concepts: Core Readings._ The MIT Press.
  * McClelland et al. (2020) James L McClelland, Felix Hill, Maja Rudolph, Jason Baldridge, and Hinrich Schütze. 2020.  Placing language in an integrated understanding system: Next steps toward human-level performance in neural language models.  _Proceedings of the National Academy of Sciences_ , 117(42):25966–25974.
  * Mikolov et al. (2013) Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013.  Linguistic regularities in continuous space word representations.  In _Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies_ , pages 746–751.
  * Murphy and Medin (1985) Gregory L Murphy and Douglas L Medin. 1985.  The role of theories in conceptual coherence.  _Psychological review_ , 92(3):289.
  * Ortega et al. (2021) Pedro A Ortega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, et al. 2021.  Shaking the foundations: delusions in sequence models for interaction and control.  _arXiv preprint arXiv:2110.10819_.
  * Packard et al. (1980) Norman H Packard, James P Crutchfield, J Doyne Farmer, and Robert S Shaw. 1980.  Geometry from a time series.  _Physical review letters_ , 45(9):712.
  * Patel and Pavlick (2021) Roma Patel and Ellie Pavlick. 2021.  Mapping language models to grounded conceptual spaces.  In _International Conference on Learning Representations_.
  * Pavlick (2022) Ellie Pavlick. 2022.  Semantic structure in deep learning.  _Annual Review of Linguistics_ , 8:447–471.
  * Peters et al. (2018) ME Peters, M Neumann, M Iyyer, M Gardner, C Clark, K Lee, and L Zettlemoyer. 2018\.  Deep contextualized word representations. arxiv 2018.  _arXiv preprint arXiv:1802.05365_ , 12.
  * Piantadosi (2021) Steven T Piantadosi. 2021.  The computational origin of representation.  _Minds and machines_ , 31(1):1–58.
  * Pierce (2002) Benjamin C Pierce. 2002.  _Types and programming languages_.  MIT press.
  * Putnam (1974) Hilary Putnam. 1974.  Meaning and reference.  _The journal of philosophy_ , 70(19):699–711.
  * Quine (1977) W.V.O. Quine. 1977.  Natural kinds.  In S.P. Schwartz, editor, _Naming, necessity, and natural kinds_ , pages 155–175. Ithaca, NY: Cornell University Press.
  * Rule et al. (2020) Joshua S Rule, Joshua B Tenenbaum, and Steven T Piantadosi. 2020.  The child as hacker.  _Trends in cognitive sciences_ , 24(11):900–915.
  * Russell (2010) Stuart J Russell. 2010.  _Artificial intelligence a modern approach_.  Pearson Education, Inc.
  * Santoro et al. (2021) Adam Santoro, Andrew Lampinen, Kory Mathewson, Timothy Lillicrap, and David Raposo. 2021.  Symbolic behaviour in artificial intelligence.  _arXiv preprint arXiv:2102.03406_.
  * Schrimpf et al. (2021) Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2021.  The neural architecture of language: Integrative modeling converges on predictive processing.  _Proceedings of the National Academy of Sciences_ , 118(45):e2105646118.
  * Shepard (1980) Roger N Shepard. 1980.  Multidimensional scaling, tree-fitting, and clustering.  _Science_ , 210(4468):390–398.
  * Shepard and Chipman (1970) Roger N Shepard and Susan Chipman. 1970.  Second-order isomorphism of internal representations: Shapes of states.  _Cognitive psychology_ , 1(1):1–17.
  * Takens (1981) Floris Takens. 1981.  Detecting strange attractors in turbulence.  In _Dynamical systems and turbulence, Warwick 1980_ , pages 366–381. Springer.
  * Tenenbaum et al. (2011) Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. 2011.  How to grow a mind: Statistics, structure, and abstraction.  _science_ , 331(6022):1279–1285.
  * Ullman et al. (2012) Tomer D Ullman, Noah D Goodman, and Joshua B Tenenbaum. 2012.  Theory learning as stochastic search in the language of thought.  _Cognitive Development_ , 27(4):455–480.
  * Vulić et al. (2017) Ivan Vulić, Daniela Gerz, Douwe Kiela, Felix Hill, and Anna Korhonen. 2017.  Hyperlex: A large-scale evaluation of graded lexical entailment.  _Computational Linguistics_ , 43(4):781–835.
  * Wason and Johnson-Laird (1972) Peter Cathcart Wason and Philip Nicholas Johnson-Laird. 1972.  _Psychology of reasoning: Structure and content_ , volume 86.  Harvard University Press.
  * Wittgenstein (1953) Ludwig Wittgenstein. 1953.  _Philosophical investigations_.  John Wiley & Sons.
  * Xu et al. (2020) Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, and Bryan Catanzaro. 2020.  Megatron-cntrl: Controllable story generation with external knowledge using large-scale language models.  _arXiv preprint arXiv:2010.00840_.