SM

# Origin of Distinct Insulating Domains in the Layered Charge Density Wave
Material 1T-TaS2

Hyungryul Yang  These authors contributed equally to this work  Department of
Physics, Yonsei University, Seoul 03722, Korea  Byeongin Lee  These authors
contributed equally to this work  Department of Physics, Yonsei University,
Seoul 03722, Korea  Junho Bang  Department of Physics, Yonsei University,
Seoul 03722, Korea  Sunghun Kim  Research Institute for Basic Science, Ajou
University, Suwon 16499, Korea  Dirk Wulferding  Center for Correlated
Electron Systems, Institute for Basic Science, Seoul 08826, Korea  Sung-Hoon
Lee<EMAIL_ADDRESS>Department of Applied Physics, Kyung Hee University, Yongin
17104, Korea  Doohee Cho<EMAIL_ADDRESS>Department of Physics, Yonsei
University, Seoul 03722, Korea

Vertical charge order shapes the electronic properties in layered charge
density wave (CDW) materials. Various stacking orders inevitably create
nanoscale domains with distinct electronic structures inaccessible to bulk
probes. Here, the stacking characteristics of bulk 1T-TaS2 are analyzed using
scanning tunneling spectroscopy (STS) and density functional theory (DFT)
calculations. It is observed that Mott-insulating domains undergo a transition
to band-insulating domains restoring vertical dimerization of the CDWs.
Furthermore, STS measurements covering a wide terrace reveal two distinct band
insulating domains differentiated by band edge broadening. These DFT
calculations reveal that the Mott insulating layers preferably reside on the
subsurface, forming broader band edges in the neighboring band insulating
layers. Ultimately, buried Mott insulating layers believed to harbor the
quantum spin liquid phase are identified. These results resolve persistent
issues regarding vertical charge order in 1T-TaS2, providing a new perspective
for investigating emergent quantum phenomena in layered CDW materials.

Introduction

Layered materials exhibit unique electronic properties governed by the atomic
and electronic characteristics of their constituent building blocks, as well
as their interlayer couplings, including Coulomb interactions, orbital
hybridization, charge transfer, and van der Waals forces 1, 2, 3. Given the
susceptibility of these interlayer interactions to various stimuli, such as
atomic defects 4 and hydrostatic pressure 5, distinct domains with various
stacking configurations naturally coexist in layered materials. To understand
the electronic properties of these diverse structures and the emergence of
exotic quantum phenomena at their interfaces, it is crucial to systematically
investigate domain configurations and their electronic structures.

1T-TaS2 is an extensively studied two-dimensional correlated system, renowned
for its rich phase diagram 5, 6, 7, 8, 9, 10, 11, 12, 13. The atomic layer is
composed of a plane of Ta atoms encapsulated by S atoms with an octahedral
coordination. Upon cooling, it undergoes a metal-insulator transition with a
charge density wave (CDW) reconstruction 14. The CDW phase is accompanied by a
$\sqrt{13}\times\sqrt{13}$ periodic lattice distortion, comprising clusters
that resemble the Star of David (SD). Each SD contains 13 Ta atoms, with 12
forming pairs to create a CDW band gap, while the unpaired electron of the
central Ta atom contributes to a narrow metallic band 6. Due to strong on-site
Coulomb interactions, intralayer electron hopping is strongly suppressed at
low temperatures. Since the localized spins of the correlated insulator are
frustrated by the triangular lattice, this material is an ideal platform to
realize the quantum spin liquid phase (QSL)15, 16.

In bulk 1T-TaS2, the electronic structure is significantly influenced by the
vertical stacking order 17, resulting in the ground state with a dimerized
configuration. Interlayer electron hopping promotes a dispersive metallic band
along the stacking direction 18. However, at low temperatures, it exhibits
insulating electronic properties with commensurate CDWs. Theoretical
calculations associate this with the formation of a dimerized insulating layer
where the SD is vertically aligned without a lateral shift 19, 20. This
dimerized configuration is energetically stable because of the energy gain
from bonding-antibonding hybridization of the unpaired electrons; otherwise,
it would be prohibited due to Coulomb repulsion. Dimerized band insulating
layers are stacked with a lateral shift in the bulk. This is similar to the
Peierls’ CDW metal-insulator transition in one-dimensional half-filled
metallic systems 21.

Cleaving this layered material produces two distinct insulating surfaces:
Type-I with a dimerized layer termination and Type-II with an undimerized
layer termination. The two distinct surface terminations can be distinguished
by their energy band gaps 20, 22, 23, with Type-I exhibiting a larger band gap
compared to Type-II. They also exhibit distinct spectral responses to electron
doping, featuring a rigid band shift and a spectral weight transfer for Type-I
and Type-II surfaces 24, respectively. In contrast, subsequent scanning
tunneling microscopy and spectroscopy (STM/STS) studies have shown a larger
gap spectrum on the surface terminated with a single layer rather than a
smaller gap 25. Moreover, the larger gap domain has been suggested to host a
QSL phase, as evidenced by the band shift in a magnetic field 26, 27, 28.
These results strongly imply that subsurface stacking configurations, which
include stacking faults induced by undimerized layers, are crucial for
reconciling the discrepancy in the measured tunneling spectra 23. However,
experimental verification of subsurface stacking faults, their distribution
and impact on the electronic structure of the surface remains limited.

In this work, we employ STM and STS measurements to visualize distinct
insulating domains which form as a result of the change in stacking
configurations across the surface of bulk 1T-TaS2. Our observations reveal the
coexistence of Type-I and Type-II domains near a monolayer step on the topmost
layer 22, 24. Interestingly, in proximity to the monolayer step, both the
upper and lower layers are predominantly occupied by the Type-I termination,
suggesting that it is more energetically favorable for the stacking fault to
reside beneath the surface. Furthermore, spatially resolved STS measurements
over a large area reveal three distinct Type-I domains separated by both
surface and subsurface domain walls. These distinct Type-I domains show a
rigid band shift and band edge broadening, highlighting the interplay between
subsurface stacking configurations and the electronic properties of the
surface state. Finally, by combining our high-resolution STS data with density
functional theory (DFT) calculations, we demonstrate how different subsurface
stacking orders can lead to the observed electronic structures of the surface.

Results and Discussion

Our data are acquired in the commensurate CDW phase, characterized by the
presence of a triangular SD superlattice (Figure 1a). A representative STM
image (Figure 1b) shows the SD clusters as bright protrusions (inset). The
dark spots in the STM image acquired at $-400$ mV are intrinsic defects
related to sulfur (S) vacancies 29, 30 whose features are more enhanced at the
lower bias due to their in-gap states (Figure S1 in Supplementary
Information). Figure 1c shows a representative differential conductance
($\mathrm{d}I/\mathrm{d}V$) spectrum acquired at a region away from intrinsic
defects. The spectrum displays two prominent peaks located near $\pm 200$ meV,
which show upward band bending near intrinsic defects or CDW domain walls.
These insulating spectral features have been commonly observed in STS
measurements 22, 24, 31. Notably, our high-resolution STS spectra enable us to
finely resolve the band edges as spectral peaks (see red arrows in Figure 1c)
with negative differential resistance, which have usually been resolved as
shoulders in the tunneling spectra. We attribute the peculiar characteristics
to resonant tunneling between localized states of the sample and an atomically
sharp STM tip 32. These enhanced spectral features allow us to distinguish
domains that have subtle differences in the edges of their electronic bands.
The origin of the edges will be discussed in the subsequent section.

We first present STM results acquired near a monolayer step to examine the
electronic structure on both Type-I and Type-II surface terminations. The STM
image (Figure 2a) shows bright and dark regions corresponding to the upper and
lower terraces, respectively. A subsequent STM image at a reduced bias voltage
of $+170$ mV (Figure 2b) displays a uniform electronic structure across the
lower terrace, characterized by a consistent superstructure and absence of
domain walls. In contrast, the upper terrace exhibits two types of domains
with different electronic structures, separated by surface domain walls. The
step height is consistent with the thickness of a single layer 1T-TaS2, but is
slightly higher at the step edge (Figure S2 in Supplementary Information).
This implies that a higher conductance domain locally forms close to the step
edge and rearranges the vertical charge order across the domain wall to
maintain a similar electronic structure to the lower terrace 23.

The $\mathrm{d}I/\mathrm{d}V$ map (Figure 2c), simultaneously acquired with
the STM image (Figure 2b), clearly illustrates the presence of two distinct
types of domains with varying contrasts on the upper terrace: a dark
triangular center domain and bright adjacent domains. Notably, the contrast of
the bright domains on the upper terrace matches that of the lower terrace. A
series of $\mathrm{d}I/\mathrm{d}V$ spectra (Figure 2d), taken along the black
arrow in Figure 2c, indicates that the bright domains on both terraces indeed
share identical electronic structures. Furthermore, it reveals that while both
the bright and dark domains are insulating, their spectra exhibit significant
differences in terms of bandgap values and peak intensities (Figure 2e): the
peak-to-peak bandgaps for the two domain types are around $0.39$ eV and $0.2$
eV, respectively. Since the contrast in the $\mathrm{d}I/\mathrm{d}V$ map
highlights the relative difference in the electronic structures, the domain
walls with in-gap states in larger gap domains are more prominent than those
in smaller gap ones (See Experimental Section). A collection of
$\mathrm{d}I/\mathrm{d}V$ maps obtained near the step edge (Figure S3 in
Supplementary Information) demonstrates that the larger gap spectrum is
dominant across the surface, with the smaller gap spectrum confined to small
domains near the step edge.

According to previous DFT calculations 23 and many-body theoretical
calculations 20, the larger bandgap of $0.39$ eV results from interlayer
hybridization in the dimerized layer (Type-I) and the smaller gap of $0.2$ eV
arises from the correlated Mott gap in the undimerized surface layer (Type-
II). The latter, a half-filled layer, exhibits correlated insulating behavior
with strong on-site Coulomb repulsion, otherwise, it shows metallic behavior.
This correlated insulating layer is expected to have a smaller gap than the
band insulating layer with an appropriate Coulomb energy, which stabilizes the
dimerized configuration. From the characteristic $\mathrm{d}I/\mathrm{d}V$
spectra of the two different domains (Figure 2e), we can identify the areas
with higher and lower $\mathrm{d}I/\mathrm{d}V$ intensities in Figure 2c as
Type-I and Type-II domains, respectively (See Experimental Section).

The crossover from Type-I to Type-II domains within a terrace is incompatible
with the dimerization scenario for the insulating ground state in the layered
CDW material. Prior STM studies have demonstrated that the undimerized layer
is a Mott insulator with a smaller gap 22 and exhibits doping-induced spectral
weight transfer instead of a rigid band shift 24. However, subsequent STM
results have shown surfaces with the larger gap spectrum, even when the
terraces are separated by a monatomic step 25, 33. These observations have
attracted renewed attention to the underlying physics of the ground state.
Importantly, it was predicted that the undimerized layer (Type-II) is less
stable than the dimerized layer at the surface, leading to the surface
reconstruction of the CDWs 23. In systems with undimerized layers at the
surface, CDW phase shifts occur via the lateral bond-relaxation of the SD
clusters in the second layers. This reconstruction relocates the undimerized
layer from the top to the third layer, leaving a dimerized layer on the
surface (Type-I with a subsurface fault). We can interpret this unique
reconstruction as the vertical bond relaxation driven by the lateral bond
relaxation. The unstable Type-II surface can only manifest in small domains
near surface steps, where electronic structures at the domain boundaries can
overcome the surface energy instability within the domain. Our observations,
in which most of the surface is populated by the Type-I domains with the Type-
II domains existing primarily near the step edge, fully corroborate these
theoretical predictions (Figure S3 in Supplementary Information). It is worth
noting that our samples were cleaved at room temperature and subsequently
cooled to cryogenic temperatures, ensuring that the vertical configuration of
CDW stacking is stabilized during the commensurate CDW formation around 180 K.
This strongly implies the presence of buried undimerized layers, which have
been identified, within the otherwise fully dimerized ground state
configuration.

Although the topmost layer is likely to be dimerized, there can be various
deviations from the ground state, forming heterogeneous domain configurations
23, 34. To demonstrate the presence of distinct stacking orders, we visualize
the electronic structure of a large flat terrace without a step edge (Figure
3a). By selecting a bias voltage close to the band edge, we highlight the
spatial variation of the band edge that can be influenced by subsurface
stacking orders (Figure S4 in Supplementary Information). Remarkably, the
$\mathrm{d}I/\mathrm{d}V$ map of the same area (Figure 3b) shows three
distinct insulating domains (marked by $\alpha$, $\alpha^{\rm*}$ and $\beta$)
with varying intensities, which up to now have not been thoroughly
investigated. The bright (dark) lines in Figure 3a (Figure 3b) represent
surface domain walls resulting from CDW phase shifts (Figure S5 in
Supplementary Information). Additionally, the consistency of our tunneling
spectra with previous studies 19, 22, 24, which indicate dimerization of the
first and second layers, leads us to attribute the sharp interfaces lacking
CDW phase shifts to subsurface domain walls. These domain walls may exist in
the third layer or below, giving rise to domains with different stacking
orders. Therefore, these domain configurations and their interfaces strongly
support the presence of distinct stacking configurations beneath the surface,
resulting in a heterogeneous surface electronic structure.

Figure 3c shows the representative $\mathrm{d}I/\mathrm{d}V$ spectra of the
three distinct insulating domains indicated in Figure 3b. They exhibit the
characteristic Type-I spectrum, implied by the large band gap of about 0.39 eV
19, 20, 22. Despite each having dimerized layers at the surface, there are
some clear differences: First, domain-$\alpha^{\rm*}$ shows a notable rigid
band shift of about $45$ meV compared to domain-$\alpha$ without a significant
change in the overall spectral shape. Second, domain-$\beta$ exhibits broader
band edges (marked by red arrows in Figure 3c) than domain-$\alpha$, while the
two prominent peaks seem identical. It is worth noting that the variations in
the surface electronic structure are not local features caused by defects or
domain walls (Figure S6 in Supplementary Information), as evidenced by the
homogeneous background $\mathrm{d}I/\mathrm{d}V$ contrast across each domain
in Figure 3b.

Among the three different Type-I domains, we observed that domain-$\alpha$ and
domain-$\alpha^{\rm*}$ appear most frequently on multiple cleaved surfaces.
This implies that domain-$\alpha$ (or domain-$\alpha^{\rm*}$) has a lower
formation energy compared to domain-$\beta$. To identify the stacking
configurations, we adopt the notations from a previous work 19 (Figure 4a).
Our theoretical calculations clearly show that there are distinct Type-I
configurations with energies lower than the Type-II (colored red in Figures 4b
and c). Note that the layered material likely has all-dimerized configurations
with diverse stacking orders (colored black in Figures 4b and c). In addition,
one or more undimerized layers can be found in the bulk 34, leading to a
slightly higher formation energy (colored blue in Figures 4b and c). A
comprehensive set of calculated formation energies for various stacking
configurations is provided in Figure S7 in Supplementary Information. Although
we cannot precisely determine the number of undimerized layers in the bulk
sample, all-dimerized configurations are likely connected to domain-$\alpha$
and domain-$\alpha^{\rm*}$, while the other Type-I configurations are
associated with domain-$\beta$, with subsurface faults within a few layers
close to the surface.

We now focus on the two distinct tunneling spectroscopic features of the
surface density of states (DOS) of domain-$\alpha^{\rm*}$ and domain-$\beta$
with respect to domain-$\alpha$. The most obvious difference is that
domain-$\beta$ has broader band edges compared to domain-$\alpha$ (Figure 3c).
Our calculated projected density of states (PDOS) spectra reveal that the
spectral features are related to the existence of the undimerized layer
underneath the surface (red arrows on blue curves in Figure 4d). The layer-
selective PDOS spectra (See Figure S7 in in Supplementary Information) clearly
show that the minor peaks at the band edges in the top-layer PDOS can be
attributed to the coupling with the principal peaks of the third-layer central
Ta atoms. The band gap for the subsurface dimerized layer is estimated to be
0.18 eV. In contrast, the band gap of the subsurface undimerized layer is
around 0.11 eV. These results agree with the experimentally observed changes
in the gap values of 0.25 eV and 0.16 eV determined by the minor peaks in
domain-$\alpha$ and domain-$\beta$, respectively.

On the other hand, domain-$\alpha^{\rm*}$ exhibits spectral characteristics
identical to domain-$\alpha$ with an upward band shift (Figure 3c). These
features are well consistent with the calculated spectra for all-dimerized
configurations (black curves in Figure 4c). The band shift is attributed to a
higher defect density in domain-$\alpha^{\rm*}$ compared to domain-$\alpha$
(Figure S8 in Supplementary Information). The upward band bending near the
intrinsic defects is a typical screening behavior against acceptor-type
dopants in the insulating material 30. Taken together, we conclude that
domains with distinct all-dimerized configurations exist and while their
defect densities are not crucial in determining their stacking orders, they do
influence their Fermi levels ($E_{\rm F}$).

Very recently, magnetic field-dependent STM experiments 26 revealed suggestive
fingerprints of the QSL phase predicted to exist in 1T-TaS2. Specifically,
certain Type-I spectra exhibited a downward shift in the presence of an
external magnetic field, consistent with the theoretical predictions for QSL
behavior 27, 28. However, two issues must be addressed to establish this
phenomenon as direct evidence for the QSL phase. First, it conflicts with the
conventional understanding that Type-I domains do not host localized spins due
to interlayer hybridization. Second, some Type-I spectra do not exhibit the
magnetic field response. Our data reconcile these disparate observations by
demonstrating the likely presence of Mott insulating layers beneath band
insulating surfaces. Thus, the two different behaviors of the Type-I spectra
under a magnetic field can be attributed to subsurface stacking faults,
emphasizing that Type-I domains cannot be characterized solely by a single
electronic structure.

Conclusion

In conclusion, we have identified multiple distinct insulating domains in a
wide field of view on the surface of bulk 1T-TaS2 with our high-resolution STS
measurements and DFT calculations. We find that the material prefers to
maintain a dimerized surface via surface reconstruction and that the
subsurface stacking configuration can be altered due to surface
reconstruction. The spatial variations in the surface DOS, identified by the
reduction of the CDW gap, broadening of the band edges and a rigid band shift,
are revealed to be determined by the differences in stacking configurations
and the defect concentration, respectively. These findings provide important
insights into reconciling the discrepancies in spectroscopic measurements and
identifying this layered CDW material as a platform for a QSL phase. Since our
results are related to vertical charge orders, they are envisaged to be highly
relevant to many other layered CDW systems, helping to resolve long-standing
open issues about competing phases and ground state formation.

Experimental Section

Synthesis and STM Measurements

1T-TaS2 crystals (HQ graphene) were synthesized by chemical vapor transport
(CVT) with iodine as a transport agent. The 1T-phase was confirmed by low-
temperature micro-Raman spectroscopy (Figure S9 in Supplementary Information).
Measurements of STM and STS were conducted using a commercial low-temperature
STM (UNISOKU) at a temperature of 4.2 K in an ultra-high vacuum environment of
$1\times 10^{\rm-10}$ Torr. Mechanically sharpened Pt / Ir $(90/10)$ wires
were used for STM tips after characterizing them on a gold (Au) or lead (Pb)
(111) surface. Bulk 1T-TaS2 samples were cleaved at room temperature and then
transferred to the STM head, which had been cooled to 4.2 K. A standard lock-
in technique was used for $\mathrm{d}I/\mathrm{d}V$ measurements, with an ac
voltage (a voltage modulation amplitude of 5 mV and a frequency of 613 Hz)
added to the dc sample bias. Occasionally, $\mathrm{d}I/\mathrm{d}V$ maps
display a reverse contrast to a topographic image, which maps the height of
the STM tip with feedback to keep the tunneling current constant. As the STM
tip moves away from the surface due to the higher conductance,
$\mathrm{d}I/\mathrm{d}V$ signal gets weaker when it is acquired
simultaneously with a topographic image 35.

Density Functional Theory Calculations

DFT investigations were carried out using the Vienna Ab Initio Simulation
Package (VASP) 36, 37. The computational approach involved the projector
augmented wave method 38, the generalized gradient approximation for the
exchange-correlation potential 39, and the DFT + $U$ formalism as formulated
by Dudarev et al. 40, setting the Hubbard $U$ parameter to 1.25 eV for Ta $5d$
orbitals. Van der Waals forces were taken into account by applying the
Tkatchenko-Scheffler method 41. Electronic wave functions were described with
a plane-wave basis set, with cutoff energies of 323 eV for bulk and 259 eV for
surface calculations. Reciprocal space integration was conducted using a
$4\times 4\times 8$ k-point mesh within the Brillouin zone of the
$\sqrt{13}\times\sqrt{13}\times 1$ supercell. For surface models, a periodic
slab geometry with seven or eight layers of TaS2 and a vacuum separation of 20
Å was employed. The relaxation of all atomic positions was continued until the
forces on each atom were below 0.01 eV/Å.

Raman Scattering Measurements

Raman spectroscopic experiments were carried out using a $\lambda=561$ nm
laser (Oxxius) with the sample mounted onto the cold finger of a He-flow
cryostat (Oxford MicroStat). The laser was focused onto the sample with a spot
diameter of 2 $\mu$m and a laser power of less than 0.1 mW. Raman-scattered
light was dispersed through a Princeton Instruments Trivista spectrometer onto
a charge-coupled device detector (PyLoN eXcelon).

Figure 1: Insulating commensurate CDW phase of 1T-TaS2. a) A schematic
representation of the atomic structure of 1T-TaS2 with the Star of David (SD)
superstructure. The black and red arrows correspond to the unit vectors of the
$1\times 1$ and the $\sqrt{13}\times\sqrt{13}$ lattices, respectively. b) An
STM image of the commensurate CDW phase in 1T-TaS2 ($V_{\rm set}=-400$ mV,
$I_{\rm set}=50$ pA). Each bright protrusion represents an SD (inset). c) The
representative $\mathrm{d}I/\mathrm{d}V$ spectrum acquired on the surface of a
Type-I domain ($V_{\rm set}=-400$ mV, $I_{\rm set}=50$ pA, $V_{\rm mod}=5$
mV). The band gap, approximately $0.39$ eV, is marked by a black arrow. Our
spectrum exhibits enhanced band edge features marked by red arrows.

Figure 2: Coexistence of dimerized (Type-I) and undimerized (Type-II) layer
terminations on the upper terrace of a step edge. a-b) STM images of the
surface of bulk 1T-TaS2 with the upper and lower terraces separated by a
monolayer step obtained at different bias voltages (a $V_{\rm set}=-400$ mV,
$I_{\rm set}=100$ pA and b $V_{\rm set}=+170$ mV, $I_{\rm set}=100$ pA,
$V_{\rm mod}=5$ mV). c) The $\mathrm{d}I/\mathrm{d}V$ map was acquired
simultaneously with the STM image shown in b (See Methods). d) Line
$\mathrm{d}I/\mathrm{d}V$ spectra acquired along the line marked by the arrow
shown in c. e) Averaged $\mathrm{d}I/\mathrm{d}V$ spectra of the Type-I
(black) and Type-II (red) domains acquired from defect-free areas ($V_{\rm
set}=-400$ mV, $I_{\rm set}=200$ pA, $V_{\rm mod}=5$ mV).

Figure 3: Three different Type-I insulator domains on a flat surface. a-b)
Three different domains shown by both the STM image (a) and the
$\mathrm{d}I/\mathrm{d}V$ map (b) ($V_{\rm set}=+170$ mV, $I_{\rm set}=50$ pA,
$V_{\rm mod}=5$ mV). c) $\mathrm{d}I/\mathrm{d}V$ spectra of the three domains
($\alpha$, $\alpha^{\rm*}$ and $\beta$) indicated in b averaged over areas not
affected by local defects ($V_{\rm set}=-400$ mV, $I_{\rm set}=50$ pA, $V_{\rm
mod}=5$ mV). Each spectrum is vertically shifted with equal spacing for
clarity. The vertical dashed line indicates the energy for the maps shown in
(a) and (b). The higher $\mathrm{d}I/\mathrm{d}V$ intensity in the tunneling
spectra corresponds to the lower contrast in the $\mathrm{d}I/\mathrm{d}V$ map
(b), which is simultaneously acquired with the topographic image (a) (see
Experimental Section) For comparison, the spectrum of domain-$\alpha$ is
superimposed as a faint gray line. There is an upward band shift for
domain-$\alpha^{\rm*}$ and a band edge broadening (marked by red arrows) for
domain-$\beta$.

Figure 4: Formation energies and electronic structures of the various stacking
configurations. a) A schematic representation of the SD supercell with 13 Ta
atoms, each labelled in line with previous studies which have identified L, I
and H (identical by three-fold rotational symmetry, shaded in gray) as the
most stable sliding configurations among the 12 possibilities (B-M) 19. b)
Direct comparison of the formation energies of two different types of stacking
configurations and their variations: fully dimerized (black), subsurface
undimerized (blue) and surface undimerized configurations (red). Formation
energies of other possible stacking configurations, marked by short gray
lines, and their corresponding PDOS spectra are presented in Figure S7. c) A
visual representation of four distinct stacking configurations discussed in
(b). Each object denotes the position of the supercell center on individual
layers, with the shaded regions (gray) indicating dimerization between
adjacent layers. d) The calculated PDOS spectra of the four different Type-I
stacking configurations identified in (b). Each spectrum is vertically shifted
with equal spacing, in order of their formation energies, with the lowest at
the bottom. The red arrows point to the wider band edges caused by undimerized
subsurface layers.

## References

  * Alden  _et al._ [2013] J. S. Alden, A. W. Tsen, P. Y. Huang, R. Hovden, L. Brown, J. Park, D. A. Muller, and P. L. McEuen, Proceedings of the National Academy of Sciences 110, 11256 (2013).
  * Ritschel  _et al._ [2015] T. Ritschel, J. Trinckauf, K. Koepernik, B. Büchner, M. v. Zimmermann, H. Berger, Y. Joe, P. Abbamonte, and J. Geck, Nature Physics 11, 328 (2015).
  * Vaňo  _et al._ [2021] V. Vaňo, M. Amini, S. C. Ganguli, G. Chen, J. L. Lado, S. Kezilebieke, and P. Liljeroth, Nature 599, 582 (2021).
  * Zhao  _et al._ [2020] X. Zhao, P. Song, C. Wang, A. C. Riis-Jensen, W. Fu, Y. Deng, D. Wan, L. Kang, S. Ning, J. Dan, _et al._ , Nature 581, 171 (2020).
  * Ritschel  _et al._ [2013] T. Ritschel, J. Trinckauf, G. Garbarino, M. Hanfland, M. v. Zimmermann, H. Berger, B. Büchner, and J. Geck, Physical Review B 87, 125135 (2013).
  * Fazekas and Tosatti [1979] P. Fazekas and E. Tosatti, Philosophical Magazine B 39, 229 (1979).
  * Sipos  _et al._ [2008] B. Sipos, A. F. Kusmartseva, A. Akrap, H. Berger, L. Forró, and E. Tutiš, Nature Materials 7, 960 (2008).
  * Stojchevska  _et al._ [2014] L. Stojchevska, I. Vaskivskyi, T. Mertelj, P. Kusar, D. Svetin, S. Brazovskii, and D. Mihailovic, Science 344, 177 (2014).
  * Yu  _et al._ [2015] Y. Yu, F. Yang, X. F. Lu, Y. J. Yan, Y.-H. Cho, L. Ma, X. Niu, S. Kim, Y.-W. Son, D. Feng, _et al._ , Nature Nanotechnology 10, 270 (2015).
  * Gerasimenko  _et al._ [2019] Y. A. Gerasimenko, I. Vaskivskyi, M. Litskevich, J. Ravnik, J. Vodeb, M. Diego, V. Kabanov, and D. Mihailovic, Nature Materials 18, 1078 (2019).
  * Ravnik  _et al._ [2021] J. Ravnik, M. Diego, Y. Gerasimenko, Y. Vaskivskyi, I. Vaskivskyi, T. Mertelj, J. Vodeb, and D. Mihailovic, Nature Communications 12, 2323 (2021).
  * Liu  _et al._ [2023] G. Liu, T. Qiu, K. He, Y. Liu, D. Lin, Z. Ma, Z. Huang, W. Tang, J. Xu, K. Watanabe, _et al._ , Nature Nanotechnology , 1 (2023).
  * Jarc  _et al._ [2023] G. Jarc, S. Y. Mathengattil, A. Montanaro, F. Giusti, E. M. Rigoni, R. Sergo, F. Fassioli, S. Winnerl, S. Dal Zilio, D. Mihailovic, _et al._ , Nature 622, 487 (2023).
  * Wilson  _et al._ [1975] J. A. Wilson, F. Di Salvo, and S. Mahajan, Advances in Physics 24, 117 (1975).
  * Anderson [1973] P. W. Anderson, Materials Research Bulletin 8, 153 (1973).
  * Law and Lee [2017] K. T. Law and P. A. Lee, Proceedings of the National Academy of Sciences 114, 6996 (2017).
  * Wang  _et al._ [2020] Y. Wang, W. Yao, Z. Xin, T. Han, Z. Wang, L. Chen, C. Cai, Y. Li, and Y. Zhang, Nature Communications 11, 4215 (2020).
  * Darancet  _et al._ [2014] P. Darancet, A. J. Millis, and C. A. Marianetti, Physical Review B 90, 045134 (2014).
  * Lee  _et al._ [2019] S.-H. Lee, J. S. Goh, and D. Cho, Physical Review Letters 122, 106404 (2019).
  * Petocchi  _et al._ [2022] F. Petocchi, C. W. Nicholson, B. Salzmann, D. Pasquier, O. V. Yazyev, C. Monney, and P. Werner, Physical Review Letters 129, 016402 (2022).
  * Grüner [1988] G. Grüner, Reviews of Modern Physics 60, 1129 (1988).
  * Butler  _et al._ [2020] C. Butler, M. Yoshida, T. Hanaguri, and Y. Iwasa, Nature Communications 11, 2477 (2020).
  * Lee and Cho [2023] S.-H. Lee and D. Cho, Nature Communications 14, 5735 (2023).
  * Lee  _et al._ [2021] J. Lee, K.-H. Jin, and H. W. Yeom, Physical Review Letters 126, 196405 (2021).
  * Wu  _et al._ [2022] Z. Wu, K. Bu, W. Zhang, Y. Fei, Y. Zheng, J. Gao, X. Luo, Z. Liu, Y.-P. Sun, and Y. Yin, Physical Review B 105, 035109 (2022).
  * Butler  _et al._ [2023] C. Butler, M. Yoshida, T. Hanaguri, and Y. Iwasa, Physical Review B 107, L161107 (2023).
  * He and Lee [2023a] W.-Y. He and P. A. Lee, Physical Review B 107, 195155 (2023a).
  * He and Lee [2023b] W.-Y. He and P. A. Lee, Physical Review B 107, 195156 (2023b).
  * Cho  _et al._ [2015] D. Cho, Y.-H. Cho, S.-W. Cheong, K.-S. Kim, and H. W. Yeom, Physical Review B 92, 085132 (2015).
  * Lutsyk  _et al._ [2023] I. Lutsyk, K. Szalowski, P. Krukowski, P. Dabrowski, M. Rogala, W. Kozlowski, M. Le Ster, M. Piskorski, D. A. Kowalczyk, W. Rys, _et al._ , Nano Research 16, 11528 (2023).
  * Cho  _et al._ [2017] D. Cho, G. Gye, J. Lee, S.-H. Lee, L. Wang, S.-W. Cheong, and H. W. Yeom, Nature Communications 8, 392 (2017).
  * Lyo and Avouris [1989] I.-W. Lyo and P. Avouris, Science 245, 1369 (1989).
  * Park  _et al._ [2023] J. W. Park, J. Lee, and H. W. Yeom, Communications Materials 4, 99 (2023).
  * Ishiguro and Sato [1991] T. Ishiguro and H. Sato, Physical Review B 44, 2046 (1991).
  * Wittneven  _et al._ [1998] C. Wittneven, R. Dombrowski, M. Morgenstern, and R. Wiesendanger, Physical Review Letters 81, 5616 (1998).
  * Kresse and Furthmüller [1996] G. Kresse and J. Furthmüller, Physical Review B 54, 11169 (1996).
  * Kresse and Joubert [1999] G. Kresse and D. Joubert, Physical Review B 59, 1758 (1999).
  * Blöchl [1994] P. E. Blöchl, Physical Review B 50, 17953 (1994).
  * Perdew  _et al._ [1996] J. P. Perdew, K. Burke, and M. Ernzerhof, Physical Review Letters 77, 3865 (1996).
  * Dudarev  _et al._ [1998] S. L. Dudarev, G. A. Botton, S. Y. Savrasov, C. J. Humphreys, and A. P. Sutton, Physical Review B 57, 1505 (1998).
  * Tkatchenko and Scheffler [2009] A. Tkatchenko and M. Scheffler, Physical Review Letters 102, 073005 (2009).
  * Lawler  _et al._ [2010] M. Lawler, K. Fujita, J. Lee, A. Schmidt, Y. Kohsaka, C. K. Kim, H. Eisaki, S. Uchida, J. Davis, J. Sethna, _et al._ , Nature 466, 347 (2010).
  * Nakashizu  _et al._ [1984] T. Nakashizu, T. Sekine, K. Uchinokura, and E. Matsuura, Physical Review B 29, 3090 (1984).

Supplementary Information

Supplementary Fig. 1: Defect spectra. a) STM image and
$\mathrm{d}I/\mathrm{d}V$ map containing defects ($V_{\rm set}=+200$ mV,
$I_{\rm set}=50$ pA). b) $\mathrm{d}I/\mathrm{d}V$ spectra taken across the
red arrows in (a). c) A waterfall plot of the $\mathrm{d}I/\mathrm{d}V$
spectra along the black arrow in (b) with vertical shifts applied for clarity.
Near the defects, the spectra show an upward shift in (b-c), which implies
that these are acceptor-type defects. On top of the defects, in-gap states
appear near the Fermi level ($E_{\rm F}$).

Supplementary Fig. 2: Height of the step edge in Figure 2 of the main text. a)
STM image acquired across the step edge shown in Figure 2 of the main
manuscript ($V_{\rm set}=+170$ mV, $I_{\rm set}=100$ pA). b) Averaged height
profile across the area highlighted in green. A height of around 6 Å is
consistent with previous reports of a monolayer step 22, 24. Type-II surface
is slightly higher than the thickness of the monolayer. It undergoes the
charge rearrangement accompanied by the surface domain wall (marked by a red
arrow), termed surface reconstruction, to become Type-I.

Supplementary Fig. 3: Large field of view of the various insulating states
near a monolayer step edge in Figure 2. a-b) Assembled a) STM images and b)
$\mathrm{d}I/\mathrm{d}V$ maps show a larger field of view of the area shown
in Figure 2 of the main manuscript. We identify various domains, including
Type-II and Type-I* ($\beta$), denoted by white dotted circles. However, both
the upper and lower terraces are dominated by Type-I domains, which can be
seen by the overall homogeneous $\mathrm{d}I/\mathrm{d}V$ intensities ($V_{\rm
set}=+170$ mV, $I_{\rm set}=21$ and $30$ pA).

Supplementary Fig. 4: Series of $\mathrm{d}I/\mathrm{d}V$ maps at various bias
voltages. a-d) $\mathrm{d}I/\mathrm{d}V$ maps acquired in the same field of
view as Figure 3 in the main text at various bias voltages ($V_{\rm
set}=-400,-180,+160$ and $+500$ mV, $I_{\rm set}=50$ pA). The boundaries
dividing the domains become more pronounced with lower bias voltages (both
positive and negative).

Supplementary Fig. 5: CDW displacement analysis. a-b) Cropped areas of the STM
image and $\mathrm{d}I/\mathrm{d}V$ map shown in Figure 3 of the main
manuscript ($V_{\rm set}=+170$ mV, $I_{\rm set}=50$ pA). c) CDW displacement
map of the same area. We apply the Lawler-Fujita algorithm to analyze the CDW
phases across the domain walls, with the black box at the center taken as
reference22, 42. We identify the presence of abrupt changes in the CDW phases
in (c) as surface domain walls, which are seen as bright and dark lines in (a)
and (b), respectively.

Supplementary Fig. 6: Multiple $\mathrm{d}I/\mathrm{d}V$ spectra taken across
the surface and subsurface domain walls shown in Figure 3 in the main text. a)
Cropped $\mathrm{d}I/\mathrm{d}V$ map of Figure 3b. b-e)
$\mathrm{d}I/\mathrm{d}V$ spectra taken across b) a subsurface domain wall
with different electronic structures (blue), c) a surface domain wall with the
same electronic structure (yellow), d) a surface domain wall with different
electronic structures (green) and e) a surface domain wall with the same
electronic structure, continued by a subsurface domain wall with different
electronic structures (magenta) ($V_{\rm set}=-400$ mV, $I_{\rm set}=200$ pA).
The $\mathrm{d}I/\mathrm{d}V$ spectra for domains-$\alpha$, $\alpha$* and
$\beta$ in Figure. 3e of the main text are obtained from regions unaffected by
local defects and identified in b) and d) as white dotted lines.

Supplementary Fig. 7: The formation energies of the favorable stacking
configurations and their corresponding surface electronic structures. a)
Relative formation energies of various stacking configurations, including all-
dimerized layers (gray), subsurface single fault (bright blue), subsurface
double fault (dark blue) and surface fault (red) with the 8ALALALA stacking
configuration taken as the reference point. b) The PDOS spectra of distinct
stacking configurations with the layer selective PDOS for the second (black
dashed curves) and third layers (red dashed curves). The solid curves
correspond to the surface PDOS acquired by integrating $d_{z^{2}}$ orbital
contributions from 13 Ta atoms in the SD cluster.

Supplementary Fig. 8: Distribution of defects in each domain. a) STM images
extracted from the same location as Figure 3(a-b) at negative and positive
bias voltages ($V_{\rm set}=-180$ and $+180$ mV). Defects are highlighted in
red and blue for negative and positive bias voltages. At negative bias
voltage, defects are usually shown as protrusions in the STM images, whereas
at positive bias voltage, they are displayed as depressions. Defect sites in
the STM images are identified and marked through image processing techniques.
b) Superposition of (red and blue) defects from (a). Some defects are shown
only at negative or positive, which are identified as isolated blue or red
dots. However, numerous defects are observable at both negative and positive
bias voltages, which are shown as purple dots. We quantify the area covered by
red, blue and purple to identify defect sites. c) In the individual gray
boxes, we note the ratio of defects, while the total averaged values of each
domain are shown in the white boxes. Domain-$\alpha$* exhibits approximately
1.24 times more defects than the other domains.

Temperature- and position-dependent Raman measurements confirm the global
crystalline structure of the sample. Indeed, both the thermal evolution of
Raman-active modes through the different CDW phases, as well as Raman spectra
taken at ten random spots across the mm-sized sample are in full accordance
with the 1T phase and clearly distinct from the 4$H$ one 43. We note that with
a laser spot diameter of 2 $\mu$m each spectrum contains rather mesoscopic
information compared to the truly microscopic nature of the STM study.
Interestingly, as can be seen in the color contour plot of panel b), we
uncover subtle variations in the form of minor phonon shifts depending on the
measurement position, which are greater than the spectral resolution limit of
our experiment. While the origin of these variations is currently not fully
understood, we interpret them as fingerprints of different Type-I : Type-II
ratios, which are expected to slightly modify the energy of CDW-induced zone-
folded phonons as well as CDW amplitude modes. We envision that future studies
focusing on a detailed Raman mapping through various thermal cycling processes
can pinpoint the origin and thereby directly link mesocopic Raman results to
microscopic STM data.

Supplementary Fig. 9: Raman spectroscopy data. a) Raman spectra of 1$T$-TaS2,
recorded in the ICCDW phase (360 K), the NCCDW phase (275 K) and the CCDW
phase (2 K). b) Raman spectra of 1$T$-TaS2 in the CCDW phase at $T=2$ K
measured at ten random spots across the mm-sized sample.


# Cabbage Sweeter than Cake? Analysing the Potential of Large Language Models
for Learning Conceptual Spaces

Usashi Chatterjee, Amit Gajbhiye, Steven Schockaert
CardiffNLP, nCardiff University, UK
<EMAIL_ADDRESS>

###### Abstract

The theory of Conceptual Spaces is an influential cognitive-linguistic
framework for representing the meaning of concepts. Conceptual spaces are
constructed from a set of quality dimensions, which essentially correspond to
primitive perceptual features (e.g. hue or size). These quality dimensions are
usually learned from human judgements, which means that applications of
conceptual spaces tend to be limited to narrow domains (e.g. modelling colour
or taste). Encouraged by recent findings about the ability of Large Language
Models (LLMs) to learn perceptually grounded representations, we explore the
potential of such models for learning conceptual spaces. Our experiments show
that LLMs can indeed be used for learning meaningful representations to some
extent. However, we also find that fine-tuned models of the BERT family are
able to match or even outperform the largest GPT-3 model, despite being 2 to 3
orders of magnitude smaller.111Our datasets and evaluation scripts are
available at
https://github.com/ExperimentsLLM/EMNLP2023_PotentialOfLLM_LearningConceptualSpace.git.

##  1 Introduction

Conceptual spaces Gärdenfors (2000) represent concepts in terms of cognitively
meaningful features, called quality dimensions. For example, a conceptual
space of colour is composed of three quality dimensions, representing hue,
saturation and intensity. Conceptual spaces provide an elegant framework for
explaining various cognitive and linguistic phenomena Gärdenfors (2014).
Within Artificial Intelligence (AI), the role of conceptual spaces is
essentially to act as an intermediate representation layer, in between neural
and symbolic representations Gärdenfors (2004). As such, conceptual spaces
could play a central role in the development of explainable AI systems.
Unfortunately, such representations are difficult to learn from data. Most
applications of conceptual spaces are thus limited to narrow domains, where
meaningful representations can be learned from ratings provided by human
participants Paradis (2015); Zwarts (2015); Chella (2015).

In this paper, we explore whether Large Language Models (LLMs) could be used
for learning conceptual spaces. This research question is closely related to
the ongoing debate about the extent to which Language Models (LMs) can learn
perceptually grounded representations Bender and Koller (2020); Abdou et al.
(2021); Patel and Pavlick (2022); Søgaard (2023). Recent work seems to suggest
this might indeed be possible, at least for the colour domain. For instance,
Abdou et al. (2021) found that LMs are able to learn representations of colour
terms which are isomorphic to perceptual colour spaces. When it comes to
predicting the typical colour of objectes, Paik et al. (2021) found that the
predictions of LMs are heavily skewed by surface co-occurrence statistics,
which are unreliable for colours due to reporting bias Gordon and Durme
(2013), i.e. the fact that obvious colours are rarely mentioned in text.
However, Liu et al. (2022a) found the effects of reporting bias to largely
disappear in recent LLMs. These findings suggest that it may now be possible
to distill meaningful conceptual space representations from LLMs, as long as
sufficiently large models are used. However, existing analyses are limited in
two ways:

  * •

Several works have explored the colour domain, and visual domains more
generally Li et al. (2023), but little is known about the abilities of LLMs in
other perceptual domains.

  * •

Most analyses focus on classifying concepts, e.g. predicting colour terms or
the materials from which objects are made, rather than on evaluating the
underlying quality dimensions.

We address the first limitation by including an evaluation in the taste
domain. To address the second limitation, rather than considering discrete
labels (e.g. _sweet_), we use LLMs to rank concepts according to the degree to
which they have a particular feature (e.g.  _sweetness_).

##  2 Datasets

The primary focus of our experiments is on the taste domain, which has not yet
been considered in this context, despite having a number of important
advantages. For instance, the relevant quality dimensions are well-established
and have a linear structure (unlike hue in the colour domain). This domain
also seems particularly challenging, as the typical terms which are used to
describe taste only apply to extreme cases. For instance, we can assert that
grapefruit is bitter and bananas are sweet, but it is less clear how a
language model would learn whether chicken is sweeter than cheese. As ground
truth, we rely on the ratings that were collected by Martin et al. (2014).
They rated a total of 590 food items along six dimensions: sweet, salty, sour,
bitter, umami and fat. The ratings were obtained by a panel of twelve
assessors who were experienced in sensory profiling. They scored the food
items during an eight month measurement phase, after having received 55 hours
of training in a laboratory. We manually rephrased some of the names of the
items in this dataset, to make them more natural. For instance, _cherry (fresh
fruit)_ was changed to _cherry_ and _hake (grilled with lemon juice)_ was
changed to _grilled hake with lemon juice_.

We complement our analysis in the taste domain with experiments on three basic
physical domains: mass, size and height. These were found to be particularly
challenging by Li et al. (2023), with LLMs often failing to outperform random
guessing. As the ground truth for mass, we use the household dataset from
Standley et al. (2017), which specifies the mass of 56 household objects. The
original dataset includes images of each object. We removed 7 items which were
not meaningful without the image, namely _big elephant_ , _small elephant_ ,
_Ivan’s phone_ , _Ollie the monkey_ , _Marshy the elephant_ , _boy doll_ and
_Dali Clock_ , resulting in a dataset of 49 objects. We treat this problem as
a ranking problem. Li et al. (2023) also created a binary classification
version of this dataset, which involves judging pairwise comparisons (e.g. is
a red lego brick heavier than a hammer?). For size and height, we use the
datasets created by Liu et al. (2022b). These size and height datasets each
consist of 500 pairwise judgements (e.g. an ant is larger than a bird). Note
that unlike for the other datasets, no complete ranking is provided.

##  3 Methods

We experiment with a number of different models.

### Ranking with GPT-3

We use GPT-3 models of four different sizes222The exact model sizes have not
been made public, but were estimated to be 350M parameters for _ada_ , 1.3B
parameters for _babbage_ , 6.7B parameters for _curie_ and 175B parameters for
_davinci_ : https://blog.eleuther.ai/gpt3-model-sizes/.: _ada_ , _babbage_ ,
_curie_ and _davinci_. To rank items according to a given dimension, we use a
prompt that contains the name of that dimension as the final word, e.g. for
sweetness we could use “ _It is known that [food item] tastes sweet”_. We then
use the probability of this final word, conditioned on the rest of the prompt,
to rank the item: the higher the probability of _sweet_ , the more we assume
the item to be sweet.

### Pairwise Comparisons with GPT-3

To predict pairwise judgements, we consider two approaches. First, we again
use conditional probabilities. For instance, to predict whether _an ant is
larger than a bird_ , we would get the conditional probability of _large_ in
the sentences _an ant is large_ and _a bird is large_. If the conditional
probability we get from the first sentence is lower than the probability from
the second sentence, we would predict that the claim that _an ant is larger
than a bird_ is false. Second, we use a prompt that asserts the statement to
be true (e.g. “An ant is larger than a bird”) and a prompt that asserts the
opposite (e.g. “A bird is larger than an ant”). We compute the perplexity of
both statements and predict the version with the lowest perplexity to be the
correct one.

### Ranking with ChatGPT and GPT-4

ChatGPT and GPT-4 are more difficult to use than GPT-3 because the OpenAI API
does not allow us to compute conditional probabilities for these models.
Instead, to use these conversational models, we directly ask them to rank a
set of items, using a prompt such as: _Rank the following items according to
their size, from the largest to the smallest_ , followed by a list of items to
be ranked.

### Baseline: DeBERTa

We consider two baselines. First, we use a DeBERTa-v3-large model He et al.
(2021), which we fine-tuned to predict the commonsense properties of concepts.
To this end, we used the extended McRae dataset McRae et al. (2005) introduced
by Forbes et al. (2019) and the augmented version of
CSLB333https://cslb.psychol.cam.ac.uk/propnorms introduced by Misra et al.
(2022). Together, these two datasets contain 19,410 positive and 31,901
negative examples of (concept,property) pairs. We fine-tune the model on these
examples using the following prompt: _can <concept> be described as
<property>? <MASK>_. The probability that the concept has the property is then
predicted using a linear classifier that takes the final-layer embedding of
the <MASK> token as input. We use the resulting model for the different
evaluations, without any further fine-tuning.

### Baseline: Bi-encoder

As the second baseline, we use two variants of the bi-encoder model from
Gajbhiye et al. (2022). First, we use the original BERT-large model from
Gajbhiye et al. (2022) that was trained on data from Microsoft Concept Graph
Ji et al. (2019) and GenericsKB Bhakthavatsalam et al. (2020). However, as
these training sets are not specifically focused on commonsense knowledge, we
used ChatGPT to construct a dataset of 109K (concept,property) pairs, since no
existing dataset of sufficient size and quantity was available. The key to
obtain high-quality examples was to ask the model to suggest properties that
are shared by several concepts, and to vary the examples that were provided as
part of a few-shot prompting strategy. More details on how we collected this
dataset using ChatGPT are provided in Appendix A. We then trained the BERT-
large bi-encoder on this dataset.

##  4 Experiments

|  |  Sweet |  Salty |  Sour |  Bitter |  Umami |  Fatty
---|---|---|---|---|---|---|---
prompt 1 | Ada | 17.5 | 8.5 | 12.2 | 16.4 | 22.5 | 10.7
Babbage | 19.5 | 51.1 | 20.2 | 22.0 | 22.6 | 16.0
Curie | 36.0 | 46.3 | 32.8 | 23.2 | 22.6 | 31.7
Davinci | 55.0 | 63.2 | 33.3 | 27.2 | 57.0 | 52.0
prompt 2 | Ada | 23.1 | 11.9 | 8.5 | 16.8 | -6.4 | 9.9
Babbage | 27.9 | 55.7 | 19.3 | 23.9 | 29.7 | 34.0
Curie | 35.4 | 47.9 | 30.3 | 22.7 | 25.4 | 37.5
Davinci | 50.2 | 54.4 | 34.6 | 28.3 | 49.8 | 42.1
| DeBERTa | 69.1 | 67.0 | 43.9 | 24.7 | 34.4 | 64.0
| Bi-encMSCG+GKB | 27.4 | -4.4 | 15.4 | 14.4 | -11.8 | 12.6
| Bi-encChatGPT | 60.3 | 47.1 | 40.4 | 9.0 | 40.7 | 40.2
Table 1: Ranking using conditional probability (Spearman $\rho$%). Prompt1: “[food item] tastes [property]”. Prompt 2: “it is known that [food item] tastes [property]”. Item | Gold | Davinci
---|---|---
Cracker with Nutella spread | 5 | 324
Chocolate with nut | 15 | 201
Sweet pancake with maple syrup | 34 | 279
Fruit cake | 41 | 265
Sweet cookies with chocolate | 60 | 233
Cracker with jam | 67 | 268
Cooked bell pepper | 252 | 8
Redcurrant | 282 | 36
Radish | 431 | 97
Mascarpone cheese | 477 | 38
Cooked green cabbage | 512 | 49
Saint-agur Cheese | 584 | 61
Table 2: Qualitative analysis of the predictions by _davinci_ for sweetness, using prompt 1. The table shows the rank positions of several food items, when ranking the items from the sweetest (rank 1) to the least sweet (rank 590), according to the ground truth and the predictions obtained with the Davinci model.  |  | Mass | Mass | Height | Size
---|---|---|---|---|---
|  | $\rho$ | Acc | Acc | Acc
cond. prob. | Ada | 23.0 | 47.8 | 68.7 | 59.1
Babbage | 48.9 | 80.9 | 67.9 | 76.4
Curie | 30.6 | 65.1 | 77.6 | 86.4
Davinci | 36.2 | 76.8 | 76.4 | 80.4
perplexity | Ada | - | 49.0 | 49.7 | 36.5
Babbage | - | 55.0 | 59.1 | 66.7
Curie | - | 56.6 | 43.3 | 45.5
Davinci | - | 70.8 | 54.7 | 51.1
rank | ChatGPT† | 28.6 | 68.3 | 89.9 | 84.3
GPT-4† | 58.6 | 84.9 | 99.1 | 99.1
| DeBERTa | -8.9 | 42.8 | 86.6 | 93.9
| Bi-encMSCG+GKB | 31.1 | 69.2 | 69.7 | 71.9
| Bi-encChatGPT | 11.8 | 67.6 | 77.2 | 60.3
Table 3: Results for physical properties, viewed as a ranking problem (mass) and as a pairwise judgment problem (mass, height and size). Prompt: “In terms of [mass/height/size], it is known that a typical [concept] is [heavy/tall/large]”. Results with † required manual post-processing of predictions.  | ada | babbage | curie | davinci | DeBERTa | Bi-encMSCG+GKB | Bi-encChatGPT
---|---|---|---|---|---|---|---
$p_{\textit{heavy}}$ | 46.6 | 81.7 | 64.7 | 76.8 | 67.7 | 69.2 | 42.9
$1-p_{\textit{light}}$ | 49.8 | 51.3 | 41.5 | 53.4 | 43.4 | 61.9 | 53.4
$p_{\textit{heavy}}\cdot(1-p_{\textit{light}})$ | 46.6 | 81.7 | 64.7 | 76.8 | 47.8 | 69.2 | 48.1
$p_{\textit{heavy}}/p_{\textit{light}}$ | 61.4 | 68.3 | 52.1 | 65.9 | 65.2 | 75.7 | 46.5
Table 4: Analysis of alternative strategies for predicting pairwise judgements
about mass (accuracy).

### Taste Domain

Table 1 summarises the main results on the taste dataset. For this experiment,
each model was used to produce a ranking of the 590 food items, which was then
compared with the ground truth in terms of Spearman’s rank correlation
($\rho$). For the LMs, we experimented with two prompts. Prompt 1 is of the
form “[food item] tastes [sweet]”, e.g. “apple tastes sweet”. Prompt 2 is of
the form “it is known that [food item] tastes [sweet]”. For the DeBERTa model
and the bi-encoders, we verbalised the sweetness property as “tastes sweet”,
and similar for the others. The results in Table 1 show a strong correlation,
for the GPT-3 models, between model size and performance, with the best
results achieved by _davinci_. Similar as was observed for the colour domain
by Liu et al. (2022a), there seems to be a qualitative change in performance
between LLMs such as davinci and smaller models. While there are some
differences between the two prompts, similar patterns are observed for both
choices. ChatGPT and GPT-4 were not able to provide a ranking of the 590
items, and are thus not considered for this experiment. We also tried ChatGPT
on a subset of 50 items, but could not achieve results which were consistently
better than random shuffling.

Surprisingly, we find that the DeBERTa model outperforms _davinci_ in most
cases, and often by a substantial margin. This is despite the fact that the
model is 2 to 3 orders of magnitude smaller. Furthermore, we can see a large
performance gap between the two variants of the bi-encoder model, with the
model trained on ChatGPT examples outperforming _curie_ , and even _davinci_
in two cases.

Table 2 presents some examples of the predictions that were made by _davinci_
for sweetness (using prompt 1), comparing the ranks according to the ground
truth (_gold_) with the ranks according to the _davinci_ predictions. The
table focuses on some of the most egregious mistakes. As can be seen,
_davinci_ fails to identify the sweetness of common foods such as chocolate,
fruit cake and jam. Conversely, the model significantly overestimates the
sweetness of different cheeses and vegetables.

### Physical Properties

Table 3 summarises the results for the physical properties. For mass, we
consider both the problem of ranking all objects, evaluated using Spearman
$\rho\%$, and the problem of evaluating pairwise judgments, evaluated using
accuracy. Height and size can only be evaluated in terms of pairwise
judgments. To obtain conditional probabilities from the GPT-3 models, we used
a prompt of the form “In terms of [mass/height/size], it is known that a
typical [concept] is [heavy/tall/large]”. We also tried a few variants, which
performed worse. To compute perplexity scores, for evaluating pairwise
judgements, we used a prompt of the form “[concept 1] is heavier/taller/larger
than [concept 2]”. For the baselines, we obtained scores for the properties
_heavy_ , _tall_ and _large_.

The correlation between model size and performance is far from obvious here,
except that _ada_ clearly underperforms the three larger models. However,
among the GPT-3 models, _babbage_ actually achieves the best results in
several cases. The results based on conditional probabilities are consistently
better than those based on perplexity. ChatGPT and GPT-4 were difficult to use
with the ranking prompt, as some items were missing, some were duplicated, and
many items were paraphrased in the ranking. The results in Table 3 were
obtained after manually correcting these issues. With this caveat in mind, it
is nonetheless clear that GPT-4 performs exceptionally well in this
experiment. In accordance with our findings in the taste domain, DeBERTa
performs very well on the height and size properties, outperforming all GPT-3
models by a clear margin. For mass, however, DeBERTa failed completely, even
achieving a negative correlation. The bi-encoder models perform well on height
and size, although generally underperforming the largest GPT-3 models. For
mass, the bi-encoder trained on ChatGPT examples performs poorly, while the
model trained on Microsoft Concept Graph and GenericsKB was more robust. It is
notable that the results in Table 3 are considerably higher than those
obtained by Li et al. (2023) using OPT Zhang et al. (2022). For mass, for
instance, even the largest OPT model (175B) was not able to do better than
random guessing.

In Table 3, the pairwise judgments about mass were assessed by predicting the
probability of the word _heavy_ (for the GPT-3 models) or by predicting the
probability that the property _heavy_ was satisfied (for the baselines).
Another possibility is to use the word/property _light_ instead, or to combine
the two probabilities. Let us write $p_{\textit{heavy}}$ to denote the
probability obtained for _heavy_ (i.e. the conditional probability of the
word, as predicted by the language model, or the probability that the property
is satisfied, as predicted by a baseline model), and similar for
$p_{\textit{light}}$. Then we can also predict the relative mass of items
based on the value $p_{\textit{heavy}}\cdot(1-p_{\textit{light}})$ or based on
the value $p_{\textit{heavy}}/p_{\textit{light}}$. These different
possibilities are evaluated in Table 4. As can be seen, there is no variant
that consistently outperforms the others.

| Bitter | Sour | Mass | Height | Size
---|---|---|---|---|---
| $\rho$ | $\rho$ | $\rho$ | Acc | Acc
Full training | 24.7 | 43.9 | -8.9 | 86.6 | 93.9
Filtered training | 24.8 | 35.0 | 30.7 | 82.0 | 90.8
Table 5: Comparison of the DeBERTa model in two settings: the full training
setting, where the McRae and CSLB datasets are used for fine-tuning, and a
filtered setting, where relevant properties are omitted.

### Analysis of Training Data Overlap

For the baselines, we may wonder to what extent their knowledge comes from the
pre-trained language model, and to what extent it has been injected during the
fine-tuning step. For this analysis, we focus in particular on the DeBERTa
model, which was fine-tuned on the McRae and CSLB datasets. These datasets
indeed cover a number of physical properties, as well as some properties from
the taste domain. Table 5 summarises how the performance of the DeBERTa model
is affected when removing the most relevant properties from the McRae and CSLB
training sets, which we refer to as _filtered training_ in the table. For
instance, for the property _bitter_ , in the filtered setting we omit all
training examples involving the properties “bitter" and “can be bitter in
taste”; for _sour_ we remove the properties “sour” and “can be sour in taste”;
for _mass_ we remove the properties “heavy", “light", “light weight" and “can
be lightweight"; for _height_ we remove the properties “short", “can be
short", “tall" and “can be tall"; and for _size_ we remove the properties
“large" and “small". Note that the McRae and CSLB datasets do not cover any
properties that are related to sweetness, saltiness, umami and fattiness. The
results in Table 5 show that filtering the training data indeed has an effect
on results, although the performance of the model overall remains strong.
Interestingly, in the case of _mass_ , the filtered setting leads to clearly
improved results.

##  5 Conclusions

We proposed the use of a dataset from the taste domain for evaluating the
ability of LLMs to learn perceptually grounded representations. We found that
LLMs can indeed make meaningful predictions about taste, but also showed that
a fine-tuned DeBERTa model, and in some cases even a fine-tuned BERT-large bi-
encoder, can outperform GPT-3. The performance of these smaller models
crucially depends on the quality of the available training data. For this
reason, we explored the idea of collecting training data from ChatGPT, using a
new prompting strategy. We complemented our experiments in the taste domain
with an evaluation of physical properties, where we achieved considerably
better results than those reported in the literature Li et al. (2023). Whereas
previous work was essentially aimed at understanding the limitations of
language models, our focus was more practical, asking the question: _can high-
quality conceptual space representations be distilled from LLMs_? Our
experiments suggest that the answer is essentially positive, but that new
approaches may be needed to optimally take advantage of the knowledge that can
be extracted from such models.

## Limitations

It is difficult to draw definitive conclusions about the extent to which
cognitively meaningful representations can be obtained by querying LLMs. Among
others, previous work has found that performance may dramatically differ
depending on the prompt which is used; see e.g. Liu et al. (2022a). We have
attempted to make reasonable choices when deciding on the considered prompts,
through initial experiments with a few variations, but clearly this is not a
guarantee that our prompts are close to being optimal. However, this also
reinforces the conclusion that LLMs are difficult to use directly for learning
conceptual spaces. While we believe that taste represents an interesting and
under-explored domain, it remains to be verified to what extent LLMs are able
to capture perceptual features in other domains.

## Acknowledgments

This work was supported by EPSRC grant EP/V025961/1.

## References

  * Abdou et al. (2021) Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. 2021.  Can language models encode perceptual structure without grounding? a case study in color.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 109–132, Online. Association for Computational Linguistics.
  * Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020.  Climbing towards NLU: On meaning, form, and understanding in the age of data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5185–5198, Online. Association for Computational Linguistics.
  * Bhakthavatsalam et al. (2020) Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter Clark. 2020.  Genericskb: A knowledge base of generic statements.  _CoRR_ , abs/2005.00660.
  * Chella (2015) Antonio Chella. 2015.  _A Cognitive Architecture for Music Perception Exploiting Conceptual Spaces_, pages 187–203. Springer International Publishing, Cham.
  * Forbes et al. (2019) Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019.  Do neural language representations learn physical commonsense? In _Proceedings of the 41th Annual Meeting of the Cognitive Science Society, CogSci 2019: Creativity + Cognition + Computation, Montreal, Canada, July 24-27, 2019_ , pages 1753–1759. cognitivesciencesociety.org.
  * Gajbhiye et al. (2022) Amit Gajbhiye, Luis Espinosa-Anke, and Steven Schockaert. 2022.  Modelling commonsense properties using pre-trained bi-encoders.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3971–3983, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Gärdenfors (2000) Peter Gärdenfors. 2000.  _Conceptual spaces - the geometry of thought_.  MIT Press.
  * Gordon and Durme (2013) Jonathan Gordon and Benjamin Van Durme. 2013.  Reporting bias and knowledge acquisition.  In _Proceedings of the 2013 workshop on Automated knowledge base construction, AKBC@CIKM 13, San Francisco, California, USA, October 27-28, 2013_ , pages 25–30. ACM.
  * Gärdenfors (2004) Peter Gärdenfors. 2004.  Conceptual spaces as a framework for knowledge representation.  _Mind and Matter_ , 2(2):9–27.
  * Gärdenfors (2014) Peter Gärdenfors. 2014.  _The Geometry of Meaning: Semantics Based on Conceptual Spaces_.  The MIT Press.
  * He et al. (2021) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  _CoRR_ , abs/2111.09543.
  * Ji et al. (2019) Lei Ji, Yujing Wang, Botian Shi, Dawei Zhang, Zhongyuan Wang, and Jun Yan. 2019\.  Microsoft concept graph: Mining semantic concepts for short text understanding.  _Data Intell._ , 1(3):238–270.
  * Li et al. (2023) Lei Li, Jingjing Xu, Qingxiu Dong, Ce Zheng, Qi Liu, Lingpeng Kong, and Xu Sun. 2023\.  Can language models understand physical concepts? _CoRR_ , abs/2305.14057.
  * Liu et al. (2022a) Fangyu Liu, Julian Eisenschlos, Jeremy Cole, and Nigel Collier. 2022a.  Do ever larger octopi still amplify reporting biases? evidence from judgments of typical colour.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 210–220, Online only. Association for Computational Linguistics.
  * Liu et al. (2022b) Xiao Liu, Da Yin, Yansong Feng, and Dongyan Zhao. 2022b.  Things not written in text: Exploring spatial commonsense from visual signals.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2365–2376, Dublin, Ireland. Association for Computational Linguistics.
  * Martin et al. (2014) Christophe Martin, Michel Visalli, Christine Lange, Pascal Schlich, and Sylvie Issanchou. 2014.  Creation of a food taste database using an in-home “taste” profile method.  _Food Quality and Preference_ , 36:70–80.
  * McRae et al. (2005) Ken McRae, George S Cree, Mark S Seidenberg, and Chris McNorgan. 2005.  Semantic feature production norms for a large set of living and nonliving things.  _Behavior research methods_ , 37(4):547–559.
  * Misra et al. (2022) Kanishka Misra, Julia Taylor Rayz, and Allyson Ettinger. 2022.  A property induction framework for neural language models.  _CoRR_ , abs/2205.06910.
  * Paik et al. (2021) Cory Paik, Stéphane Aroca-Ouellette, Alessandro Roncone, and Katharina Kann. 2021.  The World of an Octopus: How Reporting Bias Influences a Language Model’s Perception of Color.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 823–835, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Paradis (2015) Carita Paradis. 2015.  _Conceptual Spaces at Work in Sensory Cognition: Domains, Dimensions and Distances_, pages 33–55. Springer International Publishing, Cham.
  * Patel and Pavlick (2022) Roma Patel and Ellie Pavlick. 2022.  Mapping language models to grounded conceptual spaces.  In _International Conference on Learning Representations_.
  * Søgaard (2023) Anders Søgaard. 2023.  Grounding the vector space of an octopus: Word meaning from raw text.  _Minds Mach._ , 33(1):33–54.
  * Standley et al. (2017) Trevor Standley, Ozan Sener, Dawn Chen, and Silvio Savarese. 2017.  image2mass: Estimating the mass of an object from its image.  In _1st Annual Conference on Robot Learning, CoRL 2017, Mountain View, California, USA, November 13-15, 2017, Proceedings_ , volume 78 of _Proceedings of Machine Learning Research_ , pages 324–333. PMLR.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  OPT: open pre-trained transformer language models.  _CoRR_ , abs/2205.01068.
  * Zwarts (2015) Joost Zwarts. 2015.  _Conceptual Spaces, Features, and Word Meanings: The Case of Dutch Shirts_, pages 57–78. Springer International Publishing, Cham.

##  Appendix A Collecting Concept-Property Pairs using ChatGPT

To obtain training data for the bi-encoder model using ChatGPT, we used the
following prompt:

> _I am interested in knowing which properties are satisfied by different
> concepts. I am specifically interested in properties, such as being green,
> being round, being located in the kitchen or being used for preparing food,
> rather than in hypernyms. For instance, some examples of what I’m looking
> for are: 1\. Sunflower, daffodil, banana are yellow 2. Guitar, banjo,
> mandolin are played by strumming or plucking strings 3. Pillow, blanket,
> comforter are soft and provide comfort 4. Car, scooter, train have wheels 5.
> Tree, log, paper are made of wood 6. Study, bathroom, kitchen are located in
> house. Please provide me with a list of 50 such examples._

We repeated this request with the same prompt around 10 times. After this, we
changed the examples that are given (shown in bold above). This process was
repeated until we had 287K concept-property pairs. After removing duplicates,
a total of 109K such pairs remained. We found it was necessary to regularly
change the examples provided to ensure the examples were sufficiently diverse
and to avoid having too many duplicates. These examples were constructed
manually, to ensure their accuracy and diversity. Asking for more than 50
examples with one prompt became sub-optimal, as the model tends to focus on a
narrow set of similar properties when the list becomes too long.

To verify the quality of the generated dataset, we manually inspected 500 of
the generated concept-property pairs. In this sample, we identified 6 errors,
which suggests that this dataset is of sufficient quality for our intended
purpose. Compared to resources such as ConceptNet, the main limitation of the
ChatGPT generated dataset is that it appears to be less diverse, in terms of
the concepts and properties which are covered. We leave a detailed analysis of
this dataset for future work.

##  Appendix B Issues with ChatGPT and GPT-4

For the experiments in Table 3, we used ChatGPT and GPT-4 to rank 49, 25 and
26 unique objects according to their mass, height and size respectively. The
prompt used was as follows: “Rank the following objects based on their typical
[mass/height/size], from [heaviest to the lightest/ tallest to the shortest/
largest to the smallest]", followed by a list of the items. We could not
directly evaluate the responses of ChatGPT and GPT-4 because of the following
issues:

  * •

Missing objects: For instance, GPT-4 ranked 48 out of 49 objects and ChatGPT
ranked 46 out of 49 objects respectively, according to their mass.

  * •

Paraphrasing: While ranking, both GPT-4 and ChatGPT changed some of the the
names of the objects. For instance, “wooden train track" was renamed as
"wooden train track piece", “gardening shears" was renamed as "garden shears".

  * •

Duplicates: GPT-4 and ChatGPT both occasionally introduced duplicates in the
list.

To address these issues, we removed the duplicates and appended the missing
items at the end of the ranking. We manually corrected the modified names.