IP/BBSR/2021-4

# Displaced Neutrino Jets at the LHeC

Giovanna Cottin1<EMAIL_ADDRESS>Oliver Fischer 2
<EMAIL_ADDRESS>Sanjoy Mandal3<EMAIL_ADDRESS>Manimala
Mitra4,5<EMAIL_ADDRESS>Rojalin Padhan4,5<EMAIL_ADDRESS>1
Departamento de Ciencias, Facultad de Artes Liberales,Universidad Adolfo
Ibáñez, Diagonal Las Torres 2640, Santiago, Chile  2 Department of
Mathematical Sciences, University of Liverpool, Liverpool, L69 7ZL, UK 3 AHEP
Group, Institut de Física Corpuscular – CSIC/Universitat de València, Parc
Científic de Paterna.
C/ Catedrático José Beltrán, 2 E-46980 Paterna (Valencia) - SPAIN  4Institute
of Physics, Sachivalaya Marg, Bhubaneswar-751005, India 5Homi Bhabha National
Institute, Training School Complex, Anushakti Nagar, Mumbai 400094, India

###### Abstract

Extending the Standard Model with right-handed neutrinos (RHNs) is well
motivated by the observation of neutrino oscillations. In the type-I seesaw
model, the RHNs interact with the SM particles via tiny mixings with the
active neutrinos, which makes their discovery in the laboratory, and in
particular at collider experiments in general challenging. In this work we
instead consider an extension of the type-I seesaw model with the addition of
a leptoquark (LQ), and employ a non-minimal production mechanism of the RHN
via LQ decay, which is unsuppressed by neutrino mixing. We focus on relatively
light RHN with mass $\mathcal{O}(10)$ GeV and LQ with mass 1.0 TeV, and
explore the discovery prospect of the RHN at the proposed Large Hadron
electron Collider. In the considered mass range and with the given interaction
strength, the RHN is long lived and, due to it stemming from the LQ decay, it
is also heavily boosted, resulting in collimated decay products. The unique
signature under investigation is thus a displaced fat jet. We use kinematic
variables to separate signal from background, and demonstrate that the ratio
variables with respect to energy/number of displaced and prompt tracks are
useful handles in the identification of displaced decays of the RHN. We also
show that employing a positron beam provides order of magnitude enhancement in
the detection prospect of this signature.

Introduction.– The observation of neutrino oscillations is a clear indicator
for new physics beyond the Standard Model (BSM). A plethora of models exist
that aim at explaining the light neutrino masses and mixings, many of which
contain Standard Model (SM) gauge singlet right-handed neutrino (RHN). The
simplest one among them is the type-I seesaw model Minkowski (1977), where the
RHNs are Majorana particles. Being SM gauge singlet RHNs interact with the SM
particles only via their mixing with the active neutrinos, referred to as
active-sterile mixing, and is proportional to $\sqrt{\frac{m_{\nu}}{M_{N}}}$,
where $m_{\nu}$ and $M_{N}$ are the light neutrino and the RHN mass scale.
Since $m_{\nu}<$ eV, this mixing is small, leading to a suppressed production
of RHN at colliders, which makes their observation challenging. This
limitation can be avoided if RHNs are embedded in a theory framework with a
new production mechanism that involves unsuppressed interactions of RHN with
BSM/SM particles.

Motivated by this, we consider a theory framework that includes a
$\tilde{R}_{2}$ leptoquark (LQ) and Majorana RHN. We consider a mass of $1$
TeV for LQ and focus on RHN with low mass $M_{N}\sim\mathcal{O}(10)$ GeV. The
RHN in this framework interacts with the LQ and an up-type quark and can thus
be produced from LQ decay. For the considered mass scales, the RHN is heavily
boosted and its decay products leptons and jets are collimated, which fail to
satisfy standard isolation criteria, thereby enabling a large radius jet
description as the appropriate description to adopt. Additionally, a RHN in
the considered mass range is a long lived particle and its decay is displaced
from the point of its creation. The final state object of interest is thus a
displaced large radius jet, or a fat jet, which is also accompanied by a
prompt jet.

This signature is challenging to probe at the LHC due to its hadronic nature.
Therefore, we investigate this signature in a relatively clean environment,
i.e., in lepton-hadron collisions at the proposed Large Hadron electron
Collider (LHeC) Abelleira Fernandez  _et al._ (2012). The LHeC allows for
resonant $\tilde{R}_{2}$ production with a sizeable cross-section, which has
been shown to have a very good sensitivity to a LQ with first-generation
coupling Zhang  _et al._ (2018), cf also ref. Agostini  _et al._ (2020);
Padhan  _et al._ (2020). Moreover, the possibilities of electron polarisation
and to use a positron beam provide extra handles to test the nature of the new
physics signature Abelleira Fernandez  _et al._ (2012); Padhan  _et al._
(2020).

The paper is organized as follows: First we briefly review the model and the
existing constraints on LQ. Following that we discuss the production of RHN at
an $ep$ collider LHeC. In the subsequent section, we present a detailed
collider analysis and discuss discovery prospect of the unique RHN signature.
Finally, we present the summary of the paper.

Figure 1: Comparison between theory cross-section and the observed limit on
$eejj$ cross-section from the LHC. The black line is the 13 TeV LHC limit on
$\sigma\beta_{ed}^{2}$ Aad  _et al._ (2020). The green, blue, orange lines
represent the variation of theory cross-section $\sigma(pp\to
LQLQ)\times\beta_{ed}^{2}$ w.r.t LQ mass. The star denotes our chosen
benchmark couplings $Y_{11}=0.2,Z_{11}=1$, also consistent with the APV bound.

The Model.– We consider the $\tilde{R}_{2}$ leptoquark model, which contains a
scalar LQ, and three RHN states ($N_{i}$). The LQ has two iso-spin components,
$\tilde{R}_{2}(3,2,1/6)=(\tilde{R}_{2}^{\frac{2}{3}},\tilde{R}_{2}^{-\frac{1}{3}})^{T}$,
where the superscript of the components denotes the electromagnetic charge.
The following renormalizable terms describe interactions of $\tilde{R}_{2}$
with the SM fermions and $N_{i}$ Doršner  _et al._ (2016); Buchmuller  _et
al._ (1987); Buchmuller and Wyler (1986); Padhan  _et al._ (2020); Mandal  _et
al._ (2018):

$\displaystyle\mathcal{L}_{LQ}=-Y_{ij}\bar{d}_{R}^{i}\tilde{R}_{2}^{a}\epsilon^{ab}L_{L}^{j,b}+Z_{ij}\bar{Q}_{L}^{i,a}\tilde{R}_{2}^{a}N_{R}^{j}+\text{H.c.}$
(1)

In the above, $i,j=1,2,3$ are flavor indices, $a,b=1,2$ are $SU(2)_{L}$
indices, and $N_{R}$ indicates the right chiral component of $N$. Upon
expansion, Eq. (1) becomes:

$\displaystyle\mathcal{L}_{LQ}=-Y_{ij}\bar{d}_{R}^{i}e_{L}^{j}\tilde{R}_{2}^{2/3}+(YU_{\text{PMNS}})_{ij}\bar{d}_{R}^{i}\nu_{L}^{j}\tilde{R}_{2}^{-1/3}+$
(2) $\displaystyle
Z_{ij}\bar{u}_{L}^{i}N_{R}^{j}\tilde{R}_{2}^{2/3}+(V_{\text{CKM}}Z)_{ij}\bar{d}_{L}^{i}N_{R}^{j}\tilde{R}_{2}^{-1/3}+\text{H.c.}.$

Here, $Y$ and $Z$ are the $3\times 3$ complex Yukawa coupling matrices,
$V_{\text{CKM}}$ and $U_{\text{PMNS}}$ are the Cabibbo-Kobayashi-Maskawa and
Pontecorvo-Maki-Nakagawa-Sakata matrices. To investigate the model signature,
it is sufficient to assume that only one generation of RHN couples with
leptons and quarks. We consider this to be $N_{1}$ (denoted henceforth as
$N$), with only $Z_{11}\neq 0$, and all other $Z_{ij}$ being 0. Additionally,
we also consider only $Y_{11}\neq 0$. Due to the $Y_{11},Z_{11}$ couplings
$\tilde{R}^{2/3}_{2}$ can decay to both $ed$ and $Nu$ states. We denote the
corresponding branching ratios by $\beta_{ed}$ and $\beta_{Nu}$, respectively.

Constraints.– Both direct and indirect experimental searches give strong
constraints on a LQ state. Very relevant for LQ production at the LHeC is the
precision measurement of atomic parity violation (APV), which tightly
constrains the LQ coupling to a $d$ quark and $e$ state, as
${Y_{11}}<0.34\frac{M_{\text{LQ}}}{1\,\text{TeV}}$ Dorsner  _et al._ (2014),
$M_{LQ}$ is the LQ mass. Evidently, for larger LQ mass the constraint on
coupling ${Y_{11}}$ relaxes. The tree level lepton flavour violating (LFV)
Kaon decay $K_{L}\to\mu^{-}e^{+}$ also gives a tight constraint on the LQ
couplings $|Y_{22}Y_{11}^{*}|\leq 2.1\times
10^{-5}(\frac{M_{\text{LQ}}}{1\,\text{TeV}})^{2}$ Dorsner  _et al._ (2014,
2011). Due to our choice of $Y_{22}$ = 0, this is not relevant for our study.

LHC search.– LQ are pair-produced in proton-proton collisions, which has been
investigated in few different channels from the process
$pp\to\text{LQ}\,\text{LQ}\to\ell j\ell j$ Sirunyan  _et al._ (2019); Aad  _et
al._ (2020). Non-observation of any signal at $\sqrt{s}=13$ TeV constrains LQ
masses to be larger than 1.8 TeV at $95\%$ C.L Aad  _et al._ (2020) for LQ
coupling exclusively to the first generation fermions. Including several decay
channels simultaneously, in particular $LQ\to Nu$, relaxes the constraint on
the LQ mass, as shown in Fig.1. In the following we consider a large branching
ratio $\beta_{Nu}=96\%$ for $\tilde{R}_{2}\to Nu$ production mode, which
leaves a sufficiently small branching ratio $\beta_{ed}$ for the
$\tilde{R}_{2}\to de$ decay to relax the LHC mass-limit to below 1 TeV. These
branching ratios can be obtained for $Y_{11}=0.2,Z_{11}=1.0$. The point marked
in star in Fig.1 represents this benchmark.

It is important to realise that, as $\beta_{ed}$ decreases, the branching
ratio $\beta_{Nu}$ becomes large, and $LQ\to Nu$ process could in principle be
investigated at the LHC. We remind ourselves that the decay $\tilde{R}_{2}\to
Nu$ is followed by the decay of $N$ leading to a displaced fat jet for
$M_{N}={\cal O}(10)$ GeV. We find that for our benchmark point the cross-
sections for $pp\to LQLQ\to NuNu$, and $pp\to LQLQ\to edNu$ are
$\sigma=7.53,0.31$ fb, respectively, which are much smaller than what we
obtain from $ep\to Nu$ at LHeC, as discussed below. Additionally, the final
state being all-hadronic and being hidden in the large QCD background at the
LHC, we expect the sensitivity to be rather small. Hence for the study of the
displaced fat jet, we rather choose to work with LHeC.

Figure 2: Sample Feynman diagram for the model signature at the LHeC, the blue
circle indicates displaced $N$ decay.

LQ decays.– For non-zero $Y_{11},Z_{11}$ couplings $\tilde{R}_{2}^{2/3}$ can
decay to both $ed$ and $Nu$ states. The analytical expression for these two-
body decays are,

$\displaystyle\Gamma(\text{LQ}\to
ed/Nu)=\frac{|Y_{11}|^{2}f_{1}/|Z_{11}|^{2}f_{2}}{16\pi M_{\text{LQ}}^{3}}\,,$
(3)

where
$f_{1}=\lambda^{\frac{1}{2}}(M_{\text{LQ}}^{2},m_{e}^{2},m_{d}^{2})(M_{\text{LQ}}^{2}-m_{e}^{2}-m_{d}^{2}),f_{2}=\lambda^{\frac{1}{2}}(M_{\text{LQ}}^{2},M^{2}_{N},m^{2}_{u})(M_{\text{LQ}}^{2}-M_{N}^{2}-m_{u}^{2})$
and $\lambda(x,y,z)=x^{2}+y^{2}+z^{2}-2xy-2xz-2yz$. As shown by the green line
in Fig. 1, the LHC searches for the decay $LQ\to ed$ constrains LQ mass as
$M_{\textrm{LQ}}>1.80$ TeV, assuming a branching ratio $\beta_{ed}=1$.
Considering smaller branching fractions this lower bound relaxes to, for
instance, $M_{\textrm{LQ}}=1.0$ TeV when $\beta_{ed}=0.22$. For
$\beta_{ed}=0.04$ the constraint is even more relaxed.

RHN decay.– For the chosen values of the couplings, the dominant decays of $N$
are mediated via an off-shell $\tilde{R}_{2}$. The contributions of SM
mediators are suppressed by small active-sterile mixing. For the considered
mass range $N$ decays to $N\to e^{-}u\bar{d},e^{+}\bar{u}d$ via
$\tilde{R}^{2/3}_{2}$ (two-body decays into lepton and pseudoscalar/vector
meson are relevant for $M_{N}\leq{\cal O}(1)$ GeV). $N$ can also decay to
$N\to\nu q\bar{q}^{\prime}$ via  $\tilde{R}^{-1/3}_{2}$. The partial decay
width for $N\to e^{-}u\bar{d}/e^{+}\bar{u}{d}$ is:

$\displaystyle\Gamma(N\to
e^{-}u\bar{d}/e^{+}\bar{u}{d})=N_{c}\frac{|Z_{11}|^{2}|Y_{11}|^{2}M_{N}^{5}}{512\pi^{3}M_{LQ}^{4}}\mathcal{I}$
(4)

Here,
$\mathcal{I}=I(x_{u},x_{d},x_{e})=\int_{(x_{d}+x_{e})^{2}}^{(1-x_{u})^{2}}\frac{dz}{z}(1+x_{u}^{2}-z)(z-x_{d}^{2}-x_{e}^{2})\lambda^{\frac{1}{2}}(1,x_{u}^{2},z)\lambda^{\frac{1}{2}}(z,x_{d}^{2},x_{e}^{2})$
and $x_{u/d/e}=\frac{m_{u/d/e}}{M_{N}}$, and $N_{c}=3$ is the color factor.
$N\to\nu q\bar{q}^{\prime}$ also has similar dependency on masses and
couplings. The branching ratios for $N$ decaying into $eq\bar{q}^{\prime}$ and
$\nu q\bar{q}^{\prime}$ are 50% each. For $M_{N}<50$ GeV, $N$ has a proper
decay length $c\tau_{N}>{\cal O}(1)\,\mu m$ for our chosen benchmark point. In
particular for $M_{N}=10$ GeV (20 GeV), $c\tau_{N}\sim 1\ mm$ ($c\tau_{N}\sim
0.01\ mm$). On the other hand, for $M_{N}\geq 50$ GeV, $c\tau_{N}$ is less
than $1\ \mu m$, which is below the resolution of the LHeC and also LHC
detectors, such that $N$ decay cannot be considered as being displaced.

Signal.– Our stage is the LHeC with its $7$ TeV proton and 60 GeV  electron
beam and without polarisation, where $N$ is produced via $ep\to Nu$ process.
The signal is dominated by resonant $\tilde{R}_{2}^{2/3}$ production with a
cross section of 9.3 fb, followed by the subsequent decay
$\tilde{R}^{2/3}_{2}\to Nu$ with 96% branching ratio, see Fig. 2 for sample
diagram. This production channel strongly depends on the two couplings
$Y_{11}$ and $Z_{11}$. Also the $t$-channel contribution, mediated by
$\tilde{R}_{2}^{2/3}$, contributes sizeably with $\sigma\sim 5.7$ fb. We
remark, that the production of $N$ via leptonic mixing is suppressed by mixing
square $m_{\nu}/M_{N}<{\cal O}(10^{-10})$ and is therefore completely
negligible in our model. The associated parton $u$ hadronises and gives rise
to a prompt light jet.

Figure 3: Production cross-section of $ep\to Nu$ with and without $80\%$ left
(right) polarised $e^{-}(e^{+})$ beam.

l135ptt10pt

Figure 4:  Distribution of $\lambda^{\textrm{lab}}_{N}$, where the inset
figure represents the proper decay width $\Gamma_{N}$ of $N$. The green and
red lines indicate contribution of ($W,Z,h$), and LQ states in three-body
decay of $N$ with the latter dominating $\Gamma_{N}$. For $W,Z,h$ mediation,
which depend on active-sterile mixing, we use $m_{\nu}=0.1$ eV.

We consider both, $e^{-}$ and $e^{+}$ in the initial state, which lead to
different production cross sections Padhan  _et al._ (2020) due to a
difference in the quark PDFs. This could be a useful handle to fingerprint the
signature, if it were to be observed. The production cross sections are shown
in Fig. 3. The cross-sections are fairly large, $\sigma\sim 14,127$ fb for
$e^{-}$ and $e^{+}$ beam, respectively, and increases even further if
polarisation of $e^{-}/e^{+}$ beam is being used.

For the considered LQ mass $M_{LQ}=1$ TeV, and $N$ with mass
$M_{N}\sim\mathcal{O}(10)$ GeV, $N$ is produced with a boost and its proper
decay length in the laboratory system is enhanced to $\lambda^{\rm
lab}_{N}=\beta\gamma c\tau_{N}$, where $\beta\gamma=|p_{N}|/M_{N}>1$ due to
RHN momentum $p_{N}\sim M_{LQ}/2$. We show the distribution of $\lambda^{\rm
lab}_{N}$ in Fig. 4. As can be seen, $N$ with masses 10, 20 GeV undergoes
displaced decays with decay length in the $\mathcal{O}(mm-100mm)$ range, while
for 50 GeV, this is almost a prompt decay.

The boost of $N$ also leads to very small angular separation of its decay
products, as shown in Fig. 5. Of particular interest is the separation between
the charged lepton and the jets. As the figure shows, a sizeable fraction of
the decays fails to satisfy the standard lepton isolation criterion, $\Delta
R(\ell,j)>0.4$ ($\Delta R$ between other decay products of $N$ also display
similar features), which is more pronounced for smaller $M_{N}$ and implies
that the leptons are not sufficiently isolated to be recognized as such.
Instead, the $N$ decay products tend to appear as a single jet with somewhat
large radius, referred as a “fat jet”. Thus, the very specific signature under
investigation is a fat jet that originates from a RHN displaced vertex and is
accompanied by a prompt jet:

$\displaystyle e^{\pm}p\to jN\to j+j_{N}^{\rm{displaced-fat}}$ (5)

where $j_{N}^{\rm{displaced-fat}}$ denotes the displaced and collimated decay
products of the RHN, forming a fat jet, which we refer as a neutrino jet.
Among the $N\to eq\bar{q}^{\prime},\nu q\bar{q}^{\prime}$ decay modes, we
focus on $N\to e^{\pm}q\bar{q}^{\prime}$ and include these states in the fat
jet description. In terms of sensitivity our results are thus conservative.
The chosen decay mode has the added benefit that it allows the reconstruction
of $M_{N}$. Additional channels $N\to\nu q\bar{q}^{\prime}$ can be vetoed by
imposing a missing transverse momentum cut. For a prompt fat jet signature in
inverse seesaw model at LHeC, see Das  _et al._ (2019).

Notice that, $N$ is produced together with a prompt jet with substantial
$p_{T}$ and moderate $\eta$ values. This jet emerges from the primary vertex
and can be used to determine its three-coordinates. The corresponding
uncertainty of this determination is related to the tracking precision, which
is ${\cal O}(10)\ \mu$m Agostini  _et al._ (2020); Cheung  _et al._ (2020). In
the subsequent discussion, while designing the cuts, we take for the
transverse displacement a detection threshold of 50 $\mu m$, to be
conservative.

Figure 5:  $\Delta\text{R}(eq)$ separation of closely spaced lepton-quark
pair. The vertical line is the typical isolation criterion.

Figure 6: Left panel: $R_{T}$ distribution of the tracks. Middle panel:
invariant mass distribution of neutrino jet. Right panel: distribution of the
summed energy ratio of the displaced tracks and all possible tracks associated
to the displaced jet.

Backgrounds.– We consider a number of SM backgrounds, $ep\to ej,eb,\nu j,\nu
jj,ebj,ebbb$, shown in Table. 1. The relevant processes are those involving
$B$-hadrons, which can give rise to displaced vertices. The light jet
background is also important due to its huge cross-section, see Table. 1. This
is especially relevant for the heavier $N$ which are quasi-prompt due to the
short lifetime. The single top production $ep\to\nu\bar{t}$ is already
included. Other background involving tau-lepton production $ep\to\nu
j(W\to\nu\tau)$ ($\sigma_{\tau}=0.01$ pb with our cuts) can be neglected. Our
main focus is thus on the final states: $e/\nu+n_{j}j+n_{b}b$, with $n_{j}\geq
1$ and/or $n_{b}\geq 1$ being the number of light quarks and $b$ quarks,
respectively.

Signal |  Background processes $e^{-}p/e^{+}p\to abc$
---|---
$ep\to j+j_{N}$ | $\sigma_{b}^{i}$ | $ej$ | $eb$ | $\nu j$ | $\nu jj$ | $ebj$ | $ejj$ | $ebb$ | $e+3j_{b}$ | $\nu+3j_{b}$
6881.8 | 89.9 | 9621.0 | 3482.3 | 20.08 | 2559.4 | 32.67 | 1238.8 | 1394.3
|  | (5099.2) | (89.99) | (3764.3) | (1207.0) | (20.07) | (1787.0) | (32.65) | (890.5) | (486.2)
$M_{N}=10$ GeV |  |  |  |  |  |  |  |  |  |
$\sigma_{s}^{i}=7.37$, $\sigma_{s}^{f}=3.5$ ($e^{-}p$) | $\sigma_{b}^{f}$ | 4.67 | 2.86 | 0 | 0.42 | 0.36 | 1.68 | 1.04 | 1.73 | 0.47
$\sigma_{s}^{i}=64.03$, $\sigma_{s}^{f}=42.97$ ($e^{+}p$) |  | (3.46) | (2.87) | (0) | (0.14) | (0.37) | (1.18) | (1.04) | (1.25) | (0.16)
$M_{N}=20$ GeV |  |  |  |  |  |  |  |  |  |
$\sigma_{s}^{i}=7.21$, $\sigma_{s}^{f}=3.37$ ($e^{-}p$) | $\sigma_{b}^{f}$ | 5.09 | 6.69 | 0 | 0.76 | 1.01 | 2.51 | 2.48 | 3.24 | 0.72
$\sigma_{s}^{i}=63.92$, $\sigma_{s}^{f}=44.65$ ($e^{+}p$) |  | (3.77) | (6.69) | (0) | (0.26) | (1.03) | (1.75) | (2.48) | (2.33) | (0.25)
$M_{N}=30$ GeV |  |  |  |  |  |  |  |  |  |
$\sigma_{s}^{i}=7.1$, $\sigma_{s}^{f}=1.59$ ($e^{-}p$) | $\sigma_{b}^{f}$ | 1.65 | 2.83 | 0.19 | 0.42 | 0.64 | 1.64 | 1.14 | 2.0 | 0.69
$\sigma_{s}^{i}=63.7$, $\sigma_{s}^{f}=25.09$ ($e^{+}p$) |  | (1.22) | (2.84) | (0.07) | (0.14) | (0.65) | (1.14) | (1.14) | (1.44) | (0.24)
Table 1: Initial and after cut cross-sections (in fb) for signal and
backgrounds, which are denoted as $\sigma^{i,f}_{s}$ and $\sigma^{i,f}_{b}$
respectively. For background processes $j_{b}$ implies including light jet and
$b$ jet. The numbers without (within) brackets are for $e^{-}(e^{+})$ mode,
respectively.

Simulation and event selection.– We use MadGraph5 aMC@NLO(v2.7) Alwall  _et
al._ (2014) to simulate both signal and background samples. For the generation
of parton level signal events we implement the LQ model in FeynRules(v2.3)
Alloul  _et al._ (2014) and use the UFO files in MadGraph5 aMC@NLO(v2.7). We
implement the following generation level cuts for background event
$p_{T}(b/j)>20$ GeV, $p_{T}(\ell)>10$ GeV, $|\eta(b/j/\ell)|<5$, $\Delta
R(jj/\ell j/bb/bj/\ell b)>0.05$. We also demand transverse momentum $p_{T}$ of
leading parton $>150$ GeV for background events, so that the majority of
events populate the signal region. We use Herwig (7.2) Bellm  _et al._ (2016);
Bahr  _et al._ (2008) to simulate the hadronization and showering of parton
level events. For the signal, we consider the RHN decay in Herwig (7.2). We
use Rivet (v3.0) Bierlich  _et al._ (2020); Buckley  _et al._ (2013) for event
analysis. For jet formation we use FastJet (v3.3.2) Cacciari  _et al._ (2012).
We reconstruct fat jet using a Cambridge-Aachen algorithm Dokshitzer  _et al._
(1997) with radius parameter $R=1.0$. We select events according to the
following criteria:

  * •

$N_{jet}\geq 2$ and $p_{T}\geq 50$ GeV for all jets.

  * •

We select all the charged final state particles (tracks) with $p_{T}>1$ GeV in
an event and calculate the transverse displacement $R_{T}=\sqrt{X^{2}+Y^{2}}$
($X,Y$ being the coordinates of the production vertex) of each of the tracks
from the interaction point (IP). The distribution is shown in the left panel
of Fig. 6. For the signal the peaks occurring at a higher value of $R_{T}$ are
due to tracks originating from RHN decay. The high track multiplicity at a
lower $R_{T}$ is due to tracks associated with the prompt jet. For the light
jet background (also for signal, and $b$ jet background), peak at a higher
$R_{T}~{}(\simeq 200\ mm)$ occurs due to the presence of long-lived hadrons.

  * •

We define the track as a displaced track if the transverse displacement is
above the detection threshold, $R_{T}>50\,\mu$m. Subsequently, we define a
ratio

$\displaystyle r_{N}=\frac{N_{trk}(displaced)}{N_{total-trk}},$ (6)

where ${N_{trk}(displaced)}$ and ${N_{total-trk}}$ are the number of displaced
tracks, and total number of tracks associated to a jet, respectively. Since
$N$ is long-lived this ratio is expected to be closer to 1 for jets
originating from its decay vertex, while we expect a value $\ll 1$ for any
other prompt jet. We also impose a bound $R_{T}<312\ mm$ , so that we consider
only those decay products appearing within the tracker volume. We label the
jet having the largest value of $r_{N}$ as the displaced jet. For $N$ with
$M_{N}\sim 50$ GeV the decays are rather prompt, and $r_{N}$ can not be
reliably used to identify the jet as stemming from $N$ decay.

  * •

We further cross-check if the displaced jet is originating from $N$ and hence
is a neutrino jet by computing the jet-mass. We identify a jet as the neutrino
jet, if its invariant mass $M(j_{N})$ is closest to $M_{N}$. We find that for
$M_{N}\sim 10,20$ GeV, the jet with highest $r_{N}$ is consistent this
invariant mass condition.

  * •

We define a variable $r_{E}$ as the ratio of the sum of energies of the
displaced tracks and sum of energies carried out by all tracks associated to a
jet:

$\displaystyle r_{E}=\frac{\Sigma E(displaced-trk)}{\Sigma E(trk)},$ (7)

We show the distribution of this variable in Fig. 6 for the jet with highest
$r_{N}$. As can be seen, the distribution for displaced signal and light jet
background are complementary in nature, which is instrumental in separating
the two. For small $N$ masses majority of the tracks are displaced, leading to
a higher energy ratio $r_{E}$. For light jet background, reverse is true.

  * •

Finally we select those events where the neutrino jet satisfies the following
cuts: $r_{N}\geq 0.5$, $r_{E}\geq 0.5$, $p_{T}(j_{N})\geq 150$ GeV and
$M({j_{N}})=M_{N}\pm 3$.

$M_{N}$ [GeV] | $n_{\sigma}$ |  $\mathcal{L}$ [$\rm{fb}^{-1}$] | $\mathcal{Y}^{ex}$
---|---|---|---
10 | 6.0 (41.5) | 34.0 (0.7 ) | 0.067 (0.035)
20 | 4.7 (39.7) | 56.8 (0.8) | 0.059 (0.017)
30 | 3.3 (30.4) | 116.6 (1.3) | 0.047 (0.013)
Table 2:  $n_{\sigma}$ is the significance of the proposed signature with only
$50\,\textrm{fb}^{-1}$ luminosity. $\mathcal{L}$ is the required luminosity to
achieve $5\sigma$ significance. Numbers without (within) brackets corresponds
to $e^{-}$ ($e^{+}$) beam, respectively. $\mathcal{Y}^{ex}$ represents
2$\sigma$ exclusion on $Y_{11}$.

Results.\- The signal cross-sections for 10-30 GeV $M_{N}$ varies as
$\sigma^{i}_{s}\sim 7.10-7.37$ fb before applying any cut for $e^{-}$ mode
(see Table. 1). The final cross-sections after all the cuts are
$\sigma^{f}_{s}\sim 1.6-3.5$ fb. For $M_{N}=50$ GeV as the decay is almost a
prompt decay, we do not include the result in the table. We find the
backgrounds $eb,ej$ are the most relevant ones after cuts, while other
backgrounds $ebb/ebbb,ejj/ejjj$ also give sizeable contributions. Applying a
veto on missing transverse momentum will further suppress the $\nu j/\nu
jj/\nu+3j_{b}$ backgrounds. Instead, here we present a conservative estimate.
For the $e^{+}$ mode, the signal cross-section is one order of magnitude
larger as compared to $e^{-}$ mode, while the backgrounds are relatively
smaller. The signal sensitivity is calculated as

$n_{\sigma}=\sqrt{\mathcal{L}}\frac{\sigma^{f}_{s}}{\sqrt{\sigma^{f}_{s}+\sigma^{f}_{b}}}$
(8)

where $\mathcal{L}$ in $\rm{fb}^{-1}$ is the required luminosity to achieve
$n_{\sigma}$ significance, $\sigma^{f}_{s}/\sigma^{f}_{b}$ are the after-cut
signal/background cross sections. Our findings are summarised in Table 2.

For the considered RHN masses, our signature can be detected with $5\sigma$
significance with an integrated luminosity $\mathcal{L}<120$ $\rm{fb}^{-1}$.
With the total integrated luminosity of 1000 $\rm{fb}^{-1}$ in the $e^{-}$
mode, excluding our signal at 2$\sigma$ significance can be used to put an
upper limit on Yukawa $Y_{11}\leq 0.067$ for $M_{N}=10$ GeV. For higher RHN
mass $M_{N}=30$ GeV, the limit becomes stringent due to the increase in
geometric cut-efficiency ($R_{T}<312\ mm$).

The table further shows that the $e^{+}$ mode clearly outperforms the $e^{-}$
mode as signal sensitivity $n_{\sigma}$ increases by almost one order of
magnitude, and $5\sigma$ significance can be achieved with luminosities
$\mathcal{L}<2\,\rm{fb}^{-1}$ only, for all the mass points considered. With
$\mathcal{L}=1000\,\textrm{fb}^{-1}$ the upper limit on Yukawa tightens
$Y_{11}\leq$0.013 for $M_{N}=30$ GeV. The signal sensitivity can be further
improved with polarised $e^{-}$ and $e^{+}$ beams due to increase in signal
cross-section.

Conclusions.– In this article we considered a model that includes an
$\tilde{R}_{2}$ LQ with $M_{LQ}=1$ TeV and RHN with a mass of ${\cal O}(10)$
GeV. The RHN can be produced from resonant LQ decay, with comparatively large
cross sections, and also via $t$ channel processes. We studied the prospects
of discovering the RHN at the LHeC via a displaced fat jet signature, which is
purely hadronic in nature and therefore difficult to detect at the LHC. We
perform a detailed analysis of the signal and the different SM background
processes. We find that the ratio of energy deposits of the displaced and all
possible tracks associated with the displaced jet is instrumental in
separating displaced decays of RHN from background. We find that RHN in the
considered mass range can be detected at the LHeC with only $\mathcal{L}<120\
(2)\,\rm{fb}^{-1}$ luminosity with $e^{-}(e^{+})$ beam. We observe that, the
use of a positron beam at LHeC clearly enhances the detection prospect of this
signature by order of magnitude.

## Acknowledgements

G.C. acknowledges support from ANID FONDECYT-Chile grant No. 3190051. The work
of S.M. is supported by the Spanish grant FPA2017-85216-P (AEI/FEDER, UE),
PROMETEO/2018/165 (Generalitat Valenciana). M.M thanks Indo-French Centre for
the Promotion of Advanced Research for the funding (project no: 6304-2).

## References

  * Minkowski (1977) P. Minkowski, Phys. Lett. B 67, 421 (1977).
  * Abelleira Fernandez  _et al._ (2012) J. Abelleira Fernandez _et al._ (LHeC Study Group), J. Phys. G 39, 075001 (2012), arXiv:1206.2913 [physics.acc-ph] .
  * Zhang  _et al._ (2018) J. Zhang, C.-X. Yue, and Z.-C. Liu, Mod. Phys. Lett. A 33, 1850039 (2018).
  * Agostini  _et al._ (2020) P. Agostini _et al._ (LHeC, FCC-he Study Group), (2020), arXiv:2007.14491 [hep-ex] .
  * Padhan  _et al._ (2020) R. Padhan, S. Mandal, M. Mitra, and N. Sinha, Phys. Rev. D 101, 075037 (2020), arXiv:1912.07236 [hep-ph] .
  * Aad  _et al._ (2020) G. Aad _et al._ (ATLAS), JHEP 10, 112 (2020), arXiv:2006.05872 [hep-ex] .
  * Doršner  _et al._ (2016) I. Doršner, S. Fajfer, A. Greljo, J. Kamenik, and N. Košnik, Phys. Rept. 641, 1 (2016), arXiv:1603.04993 [hep-ph] .
  * Buchmuller  _et al._ (1987) W. Buchmuller, R. Ruckl, and D. Wyler, Phys. Lett. B 191, 442 (1987), [Erratum: Phys.Lett.B 448, 320–320 (1999)].
  * Buchmuller and Wyler (1986) W. Buchmuller and D. Wyler, Phys. Lett. B 177, 377 (1986).
  * Mandal  _et al._ (2018) S. Mandal, M. Mitra, and N. Sinha, Phys. Rev. D 98, 095004 (2018), arXiv:1807.06455 [hep-ph] .
  * Dorsner  _et al._ (2014) I. Dorsner, S. Fajfer, and A. Greljo, JHEP 10, 154 (2014), arXiv:1406.4831 [hep-ph] .
  * Dorsner  _et al._ (2011) I. Dorsner, J. Drobnak, S. Fajfer, J. F. Kamenik, and N. Kosnik, JHEP 11, 002 (2011), arXiv:1107.5393 [hep-ph] .
  * Sirunyan  _et al._ (2019) A. M. Sirunyan _et al._ (CMS), Phys. Rev. D 99, 052002 (2019).
  * Das  _et al._ (2019) A. Das, S. Jana, S. Mandal, and S. Nandi, Phys. Rev. D 99, 055030 (2019), arXiv:1811.04291 [hep-ph] .
  * Cheung  _et al._ (2020) K. Cheung, O. Fischer, Z. S. Wang, and J. Zurita, (2020), arXiv:2008.09614 [hep-ph] .
  * Alwall  _et al._ (2014) J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H. S. Shao, T. Stelzer, P. Torrielli, and M. Zaro, JHEP 07, 079 (2014), arXiv:1405.0301 [hep-ph] .
  * Alloul  _et al._ (2014) A. Alloul, N. D. Christensen, C. Degrande, C. Duhr, and B. Fuks, Comput. Phys. Commun. 185, 2250 (2014), arXiv:1310.1921 [hep-ph] .
  * Bellm  _et al._ (2016) J. Bellm _et al._ , Eur. Phys. J. C 76, 196 (2016), arXiv:1512.01178 [hep-ph] .
  * Bahr  _et al._ (2008) M. Bahr _et al._ , Eur. Phys. J. C 58, 639 (2008), arXiv:0803.0883 [hep-ph] .
  * Bierlich  _et al._ (2020) C. Bierlich _et al._ , SciPost Phys. 8, 026 (2020), arXiv:1912.05451 [hep-ph] .
  * Buckley  _et al._ (2013) A. Buckley, J. Butterworth, L. Lonnblad, D. Grellscheid, H. Hoeth, J. Monk, H. Schulz, and F. Siegert, Comput. Phys. Commun. 184, 2803 (2013), arXiv:1003.0694 [hep-ph] .
  * Cacciari  _et al._ (2012) M. Cacciari, G. P. Salam, and G. Soyez, Eur. Phys. J. C 72, 1896 (2012), arXiv:1111.6097 [hep-ph] .
  * Dokshitzer  _et al._ (1997) Y. L. Dokshitzer, G. D. Leder, S. Moretti, and B. R. Webber, JHEP 08, 001 (1997), arXiv:hep-ph/9707323 .


pnasbriefreport Marjieh All authors contributed to the design of the
experiments and editing the manuscript. Authors RM, NJ, and TLG conceptualized
the project. Author IS conducted the LLM experiments. Authors RM and PVR
conducted the human experiments. Authors RM, IS, PVR, and NJ analyzed the
results. 1To whom correspondence should be addressed. E-mail:
<EMAIL_ADDRESS>The authors declare no competing interests. 2NJ
contributed equally to this work with TLG.

# Large language models predict human sensory judgments across six modalities

Raja Marjieh  Department of Psychology, Princeton University, USA  Ilia
Sucholutsky  Department of Computer Science, Princeton University, USA  Pol
van Rijn  Max Planck Institute for Empirical Aesthetics, Germany  Nori Jacoby
Max Planck Institute for Empirical Aesthetics, Germany  Thomas L. Griffiths
Department of Psychology, Princeton University, USA  Department of Computer
Science, Princeton University, USA

###### Abstract

Determining the extent to which the perceptual world can be recovered from
language is a longstanding problem in philosophy and cognitive science. We
show that state-of-the-art large language models can unlock new insights into
this problem by providing a lower bound on the amount of perceptual
information that can be extracted from language. Specifically, we elicit
pairwise similarity judgments from GPT models across six psychophysical
datasets. We show that the judgments are significantly correlated with human
data across all domains, recovering well-known representations like the color
wheel and pitch spiral. Surprisingly, we find that a model (GPT-4) co-trained
on vision and language does not necessarily lead to improvements specific to
the visual modality. To study the influence of specific languages on
perception, we also apply the models to a multilingual color-naming task. We
find that GPT-4 replicates cross-linguistic variation in English and Russian
illuminating the interaction of language and perception.

###### keywords:

perception $|$ representation $|$ NLP $|$ AI $|$ cognitive science

This manuscript was compiled on

Figure 1: A. Schematic of the LLM-based and human similarity judgment
elicitation paradigms. B. Correlations between models and human data across
six perceptual modalities, namely, pitch, loudness, colors, consonants, taste,
and timbre (Pearson $r$; 95% CIs).

Imagine that you were chosen to be part of an expedition aimed at studying a
newly discovered alien species on a distant planet. Your task is to understand
the perceptual system of that species. You arrive at the planet and, to your
dismay, discover that its inhabitants are long departed and the only thing
they left behind is a huge archive of text. How much of the perceptual world
of that species can you recover based on text alone? Versions of this question
have occupied philosophers for centuries (1, 2), and decades of psychological
research are beginning to provide glimpses into the rich perceptual content of
language and its influence on perception (3, 4, 5, 6, 7, 8, 9, 10, 11).

But how can the amount of information that language provides about perception
be quantified? Here we propose to do so by eliciting psychophysical judgments
from large language models (LLMs) such as GPT-3 and its recent “ChatGPT”
variants GPT-3.5 and GPT-4 (12, 13). These models are trained on massive text
corpora reflecting a substantial chunk of human language and can be queried in
a way that is analogous to humans.
Recent research shows that LLMs can be used to study various aspects of
cognition (14), including, language processing in the brain (15, 16, 17),
perception (18, 19, 20), cross-modal alignment (21), and morality (22, 23).
Here, we use the fact that they are trained on large amounts of language to
gain insight into the classic problem of the relationship between language and
perception: these models provide a lower bound on the amount of information
about perceptual experience that can be extracted from language alone.

We explored this empirically across six modalities, namely, pitch, loudness,
colors, consonants, taste, and timbre. Given a stimulus space (e.g., colors)
and its stimulus specification (e.g., wavelengths or their corresponding hex-
codes) we elicit pairwise similarity judgments in a direct analogy to the
widespread paradigm of similarity in cognitive science (24) using a carefully
crafted prompt that is given to the model to complete (Figure 1A; see
Methods). Importantly, whereas four of the modalities were based on classical
results from the literature (colors (25), loudness (26), timbre (27) and taste
(28)), two human datasets (pitch and vocal consonants) were novel and thus
were not part of the training set of the models.

It is worthwhile to note that, unlike the other models, GPT-4 was trained in a
multi-modal approach, enabling it to access both written text (similar to the
other two variants) and images. This allowed us to examine if the additional
sensory information resulted in enhanced performance in the color modality
relative to the other domains. Moreover, to further interrogate whether the
LLMs’ sensory representation was language-dependent, we tested whether they
would behave differently in the presence of the same sensory information (a
color hex-code), but respond in different languages. To that end, we conducted
a color-naming task using a paradigm similar to that of (29) and the World
Color Survey (30, 31), and constructed human and GPT-3, GPT-3.5, and GPT-4
color-naming maps in both English and Russian.

## Results

### Similarity study

For each dataset, we designed a tailored prompt template that could be filled
in with in-context examples and the pair of target stimuli for which we would
want the LLM to produce a similarity rating (see Methods and SI for full
specification of the prompts and datasets). Across all domains, we elicited 10
ratings per pair of stimuli from each of the GPT models and then constructed
aggregate similarity ratings by averaging. We then evaluated the resulting
scores by correlating them with human data. The Pearson correlation
coefficients between human data and model predictions are shown in Figure 1B
(See SI for details regarding computing the correlations and CIs). We see that
across all domains, the correlations were significant, and were particularly
high for pitch ($r=.92$, 95% CI $[.91,.92]$ for GPT-4), loudness ($r=.89$, 95%
CI $[.87,.91]$ for GPT-3.5), and colors ($r=.89$, 95% CI $[.87,.91]$ for
GPT-4) (and $>.6$ for all models), followed by moderate but highly significant
correlations for consonants ($r=.57$, 95% CI $[.56,.59]$ for GPT-4), taste
($r=.54$, 95% CI $[.48,.61]$ for GPT-3.5) and timbre ($r=.42$, 95% CI
$[.40,.44]$ for GPT-3.5). For the two modalities for which we collected data,
we could compare model performance to the inter-rater split-half reliability
(IRR). The IRRs for pitch and consonants were $r=.90$ (95% CI $[.87,.92]$) and
$r=.46$ (95% CI $[.36,.56]$), respectively, suggesting that the performance of
GPT-4 is on par with human performance.
We also note that in five out of the six domains, GPT-4 was among the top two
models. Interestingly, the improvement relative to the average performance of
the other models happened across all modalities with the exception of timbre
and loudness, and was not restricted or particularly large for the domain of
colors (compare e.g. $\Delta r=.15$ 95% CI $[.13,.16]$ for colors vs. $\Delta
r=.16$ 95% CI $[.15,.17]$ for pitch, $\Delta r=.16$ for taste 95% CI
$[.12,.19]$, and $\Delta r=.14$ 95% CI $[.12,.16]$ for consonants) suggesting
that this improvement is driven by richer textual training in GPT-4 rather
than the possibility of its inclusion of images in its training set as is
currently being debated (18).

Figure 2: A. Human and LLM similarity marginals and an example GPT-3
corresponding similarity matrix and its three-dimensional MDS solution for
pitch. B. MDS solutions for vocal consonants and colors for GPT-4 similarity
matrices. To illustrate the structure of the results, we highlighted
consonants with the same place of articulation in the vocal tract with the
same shape and color, and added a rotated HSV color wheel for the color MDS.
C. Example GPT-4 explanations for similarity judgment scores. D. Word clouds
for GPT-4 explanations in the domain of pitch, vocal consonants, and colors.

Next, to get a finer picture of the LLM-based judgments and see to what extent
they reflect human representations, we performed the following analyses.
Starting from the domain of pitch, we wanted to see to what extent the LLM
data captures a well-known psychological phenomenon that Western listeners
tend to associate particular musical intervals or ratios of frequencies (such
as the octave or 2:1 frequency ratio) with enhanced similarity (32). To test
this we computed the average similarity score over groups of pitch pairs that
are separated by the same fixed interval (i.e., the same frequency ratios).
Figure 2A shows the resulting average similarity per interval for the models
and humans along with an example corresponding smoothed similarity matrix for
GPT-3 (smoothing was done by averaging the raw similarity matrix over its sub-
diagonals). We can see that apart from the decay as a function of separation
(i.e., tones that are far apart in log frequency are perceived as increasingly
dissimilar) there is a clear spike precisely at 12 semitones (octave),
consistent with the aforementioned phenomenon of “octave equivalence” (33).
Moreover, applying multi-dimensional scaling (MDS) (24) to the smoothed
similarity matrix whereby the different stimuli are mapped into points in a
Euclidean space (also known as “psychological space”) such that similar
stimuli are mapped to nearby points reveals a clear helical structure with
twists that correspond to precisely 12 semitone separations (i.e., octaves)
recovering the pitch spiral representation (Figure 2A). Likewise, applying MDS
to the domains of consonants and colors (Figure 2B) reveals highly
interpretable representations, namely, the familiar color wheel and a
production-based representation for consonants.111An interactive visualization
of all human similarity matrices and their corresponding LLM counterparts, as
well as their two-dimensional MDS solutions is available at:
https://computational-audition.github.io/LLM-psychophysics/all-modalities.html
As an additional test, we asked GPT-4 to provide explanations for the
judgments it made (Figure 2C-D), and remarkably, the model resorted to
explanations involving the octave, ratios, and harmonic relations for pitch,
places of articulation in the vocal tract for consonants, and hue, brightness
and color spectra for colors, consistent with the MDS solutions.

### Color naming study

The results so far suggest that LLMs can use textual information to form
perceptual representations. If this is indeed the case, we hypothesized that
representations in LLMs using different languages could be different even in
the presence of identical input. This would be consistent with cross-cultural
differences in language and perception that were observed in humans (30). To
test this, we propose for the first time to test LLMs on an explicit naming
task proposed by the seminal work of (29) and further explored across cultures
around the globe (30, 31).

We thus tested whether LLMs would yield different naming patterns of color hex
codes depending on the language of the prompt used to elicit those names (see
Methods; Figure 3). Specifically, we presented both humans and LLMs with
different colors and asked them to perform a forced-choice naming task by
selecting from a pre-specified list of 15 color names (see Methods). We
specifically focused on English and Russian as test cases, since Russian
speakers are documented to use richer vocabulary to describe what English
speakers would otherwise describe as blue and purple (29, 34, 35). We
collected data from 103 native English speakers and 51 native Russian speakers
and compared them against LLMs performing the same task, and to the in-lab
data of (31) as an additional baseline.222The human color naming data that we
collected can be explored via: https://computational-audition.github.io/LLM-
psychophysics/color.html

The results are shown in Figure 3. Our first finding was that GPT-4 in both
English and Russian were more human-like than the other variants when compared
with an adjusted Rand index (see Figure 3A; English: GPT-4 $0.59$ 95% CI
$[0.56,0.63]$, GPT-3.5, $0.50$ 95% CI $[0.46,0.52]$, GPT-3 $0.39$ 95% CI
$[0.37,0.42]$; Russian: GPT-4 $0.54$ 95% CI $[0.46,0.54]$, GPT-3.5 $0.50$ 95%
CI $[0.45,0.52]$, GPT-3 $0.35$ 95% CI $[0.29,0.35]$; see Methods). It is
evident from our data, however, that LLMs are still not perfect in predicting
human color naming as compared to a separate lab-based experiment conducted by
Lindsey and Brown (31) (dashed line in Figure 3A top, constrained naming task
0.73 95% CI [0.65,0.73], free naming task 0.75 95% CI [0.66,0.75]). Moreover,
the naming of GPT-4 colors differs from human data in some important cases,
including the color turquoise, which was selected as the dominant color for 46
Munsell colors in GPT4 versus only 15 in human data. Note, however, that our
human English results conducted online and with relative less control over
color presentation were highly consistent with (31) that were conducted in the
lab and under controlled environment, even though (31) used slightly different
paradigms: free naming (consistency to our human data: $0.75$ 95% CI
$[0.66,0.75])$ and forced-choice list with a different set of items ($0.73$
95% CI $[0.65,0.74]$).

Importantly, however, GPT-4 appears to replicate cross-lingual differences
(Figure 3B), for example separating Russian blue and purple into distinct
categories for lighter and darker areas (35). Indeed, the color sínij / Синий
(Blue) was the dominant category for 18 and 29 Munsell colors for GPT-4 and
humans, respectively, and the color golubój / Голубой (Light-blue) was
accordingly the dominant category for 33 and 26 colors. Similarly, the color
fiolétovyj / Фиолетовый (Violet) was the dominant category for 27 and 32
Munsell colors for GPT-4 and humans, respectively, and the color lilóvyj /
Лиловый (Lilac) was accordingly the dominant category for 20 and 18 Munsell
colors.

Figure 3: Color naming experiment using 330 Munsell colors from the World
Color Survey (top, color space). A. Adjusted Rand index illustrating the
alignment between human and LLM experiments (95% CIs). The dashed lines for
English represent lab-based free naming and forced-choice naming experiments
collected by Lindsey and Brown (31) (data reproduced with permission). B. Data
comparison between humans and LLMs in Russian and English. Participants and
LLMs were shown colors and were asked to choose from the same 15-color list.
The count of chosen colors for each option is given in parentheses. The color
of a response cluster in the maps represents its average color.

## Discussion

In this work, we showed how recent advances in large language models and, in
particular, their flexible prompt engineering capabilities provide an elegant
way for extracting clear quantitative psychophysical signals from text
corpora. Our results contribute to a variety of thought-provoking issues in
perception and language research. In particular, our findings further support
recent research suggesting that people that lack direct sensory experiences
(e.g., congenitally blind individuals) could still possess a rich
understanding of perceptual concepts through language (e.g., colors (10)).
Likewise, the language-dependence of the color representation of GPT-4 in the
naming task suggests that the physical stimulus alone (in our case, the color
hex code) is insufficient for explaining the behavioral patterns of the model
since the same hex codes triggered different color categorizations in Russian
and English. This supports the idea that language is not only shaped by
perception but can also reshape it (35). Finally, the fact that GPT-4 achieved
IRR-level performance without any fine-tuning in the newly collected datasets
of pitch and vocal consonants, contributes to our understanding of LLMs’
ability to mimic human behavior (13).
We end by discussing some limitations which point towards future research
directions. First, while in our work we experimented with three different
types of behavioral data (similarity, explanations, and naming), there are
other perceptual measures that one could consider (for example, odd-one-out
triplet judgments (36)). Future work could explore these measures and
interrogate to what extent they too yield a consistent representation. Second,
our work is restricted to population-level averages as a leading-order
analysis, a follow-up study could look into the natural variability of the LLM
judgments and see to what extent they capture individual-level differences in
humans. Third, while our work provides some evidence for LLMs’ ability to
capture cross-cultural differences, it remains to be seen how far this holds
for other languages, especially those that are underrepresented (37). Finally,
it is important to point out that the very same massive training pipelines
that make LLMs a powerful proxy of human language also make them particularly
susceptible to inheriting biases (38). Researchers should be particularly
cautious when interpreting the patterns of behavior elicited from LLMs and
should always benchmark them against genuine human data.
To conclude, our work showcases how LLMs can be used to explore the limits of
what information about the world can be recovered from language alone, and
more broadly, it highlights the potential of combining human and machine
experiments in order to understand fundamental questions in cognitive science.

### GPT prompt elicitation

The general structure of the prompt elicitation template for the similarity
experiments was: one sentence describing the dataset (e.g., “people described
pairs of colors using their hex code.”), one sentence describing the
similarity rating task and scale (e.g., “how similar are the colors in each
pair on a scale of 0-1 where 0 is completely dissimilar and 1 is completely
similar?”), three sets of three lines each corresponding to two stimuli and
their actual similarity rating taken from behavioral experiments which serve
as few-shot in-context examples, and an additional set of three lines
corresponding to the pair of target stimuli and an associated empty rating
field for the model to fill in (“Color one: #FF0000. Color two: #A020F0.
Rating:”). These were necessary to ensure that the model provided numerical
values, and in all cases consisted of three fixed and randomly chosen
comparisons so that most of the content was left for the model to produce. For
each pair of target stimuli, we elicited ten ratings from each GPT model.
Across all repetitions of all pairs of stimuli for a given dataset, we used
only the same three in-context examples to ensure that the model is exposed to
only a very small fraction of the similarity judgments against which its
ratings were compared (see SI for full prompts).

For the color-naming experiments, we first elicited 15 basic color names from
GPT-4 using the prompt “Name 15 basic colors.” and a temperature of 0 (to get
the highest probability answers). We then had GPT-4 name the hex code
corresponding to each of the WCS colors using the following prompt: “Here is a
list of 15 basic color names: $<$shuffled basic color list$>$. Which of these
names best describes the following color: $<$hex-code$>$? Respond only using
the name.” We repeated this prompt ten times for each WCS color with the basic
color list shuffled each time and temperature set to the default 0.7 to elicit
ten names per WCS color. We repeated the full procedure with both prompts
translated to Russian (GPT-4 also responded to both of these in Russian; see
SI for full prompt).

### Stimuli

The six human similarity datasets we considered come in two flavors – direct
(dis-)similarity ratings and confusion matrices – and from two sources –
previous psychological studies from the literature and newly collected
datasets. Confusion matrices provide an alternative way to compute similarity
scores between stimuli by counting the number of times a stimulus $x$ is
confused for a stimulus $y$. By normalizing the counts one gets confusion
probabilities $p_{xy}$ which can be converted into similarity scores using the
formula $s_{xy}=\sqrt{p_{xy}p_{yx}/p_{xx}p_{yy}}$ (39, 40).

##### Colors

This dataset was taken from (25) (also reproduced in (24)) and comprised
direct similarity judgments across a set of 14 colors with wavelengths in the
range $434-674$ nanometers. We converted wavelengths into RGB using the script
at https://hasanyavuz.ozderya.net/?p=211 and then we used the webcolors Python
package to convert into hex codes. To get better coverage of the color wheel
in Figure 2B we extended the space to 23 color stimuli by interpolating
between the original colors in the dataset and eliciting an extended
similarity matrix from GPT-4.

##### Pitch

This dataset was collected and made publicly available very recently by a
subset of the authors in (41) (see details below). It contains similarity
judgments over pairs of 25 harmonic complex tones (10 partials and 3dB/octave
roll-off) over a two octave range from C4 (60 MIDI; 261.626 Hz) to C6 (84
MIDI; 1046.502 Hz). The pitch values were separated by 1 semitone steps to
account for the fact that pitch perception is logarithmic (33) where the
mapping between frequencies $f$ in Hertz and pitch $p$ in semitones are given
by $p=12\log_{2}{f/440}+69$.

##### Vocal consonants

This dataset was also collected via an online study (see below) and comprised
similarity judgments over 16 recordings of vocal consonants taken from the
International Phonetic
Association333https://www.internationalphoneticassociation.org/. The vocal
consonants considered were b (bay), p (pay), m (may), n (no), g (go), k
(cake), d (die), t (tie), f (fee), v (vow), s (so), T (thigh), D (they), Z
(Jacques), and S (show). The recordings came from two speakers, one male and
one female.

##### Loudness

We accessed this dataset via (40) which itself takes the data from (26). The
dataset comes in the form of a confusion matrix over 8 pure tones of different
loudness values ranging from 71.1 to 74.6 decibels.

##### Taste

This dataset was also accessed via (40) and is taken from (28). The data comes
in the form of a confusion matrix over 10 flavors described to participants as
salt, salt-substitute, MSG, quinine, acid, sugar, artificial sweetener, salt-
sugar, acid-sugar, and quinine-sugar.

##### Timbre

This dataset was assembled in (27) based on 1217 subject’s judgments from 5
prior publications. It comprises dissimilarity judgments over 12 instrument
timbres: clarinet, saxophone, trumpet, cello, French horn, oboe, flute,
English horn, bassoon, trombone, violin, and piano.

### Behavioral experiments

To collect similarity judgments over pitch and vocal consonants we deployed
two online experiments on Amazon Mechanical Turk (AMT). Overall, 55
participants completed the pitch study and 64 participants completed the vocal
consonants study. In addition, we collected color naming data in Russian and
British English participants using Prolific444https://www.prolific.co.
Overall, we recruited 154 participants of which 103 were UK participants and
51 were Russian participants. Experiments were implemented using
PsyNet555https://psynet.dev/ and
Dallinger666https://dallinger.readthedocs.io/. See additional details in SI.

### Code and Data availability

All data and code used in this work can be accessed via the following link:
https://tinyurl.com/fudaby5p, with the exception of the (31) color naming
dataset as it is only available upon request from the authors. An interactive
visualization of two-dimensional MDS spaces for the 6 modalities is available
at: https://computational-audition.github.io/LLM-psychophysics/all-
modalities.html. The human color naming data can be interactively explored
via: https://computational-audition.github.io/LLM-psychophysics/color.html.

This research project and related results were made possible with the support
of the NOMIS Foundation, and an NSERC fellowship (567554-2022) to IS.

## References

  * (1) D Hume, An Abstract of a Treatise of Human Nature, 1740.  (CUP Archive), (1740).
  * (2) J Locke, An essay concerning human understanding.  (Kay & Troutman), (1847).
  * (3) RL Goldstone, BJ Rogosky, Using relations within conceptual systems to translate across conceptual systems.  Cognition 84, 295–320 (2002).
  * (4) T Regier, P Kay, N Khetarpal, Color naming reflects optimal partitions of color space.  Proceedings of the National Academy of Sciences 104, 1436–1441 (2007).
  * (5) T Regier, P Kay, Language, thought, and color: Whorf was half right.  Trends in cognitive sciences 13, 439–446 (2009).
  * (6) S Dolscheid, S Shayan, A Majid, D Casasanto, The thickness of musical pitch: Psychophysical evidence for linguistic relativity.  Psychological science 24, 613–621 (2013).
  * (7) N Zaslavsky, C Kemp, T Regier, N Tishby, Efficient compression in color naming and its evolution.  Proceedings of the National Academy of Sciences 115, 7937–7942 (2018).
  * (8) JS Kim, GV Elli, M Bedny, Knowledge of animal appearance among sighted and blind adults.  Proceedings of the National Academy of Sciences 116, 11213–11222 (2019).
  * (9) G Lupyan, RA Rahman, L Boroditsky, A Clark, Effects of language on visual perception.  Trends in cognitive sciences 24, 930–944 (2020).
  * (10) JS Kim, B Aheimer, V Montané Manrara, M Bedny, Shared understanding of color among sighted and blind adults.  Proceedings of the National Academy of Sciences 118, e2020192118 (2021).
  * (11) G Kawakita, A Zeleznikow-Johnston, N Tsuchiya, M Oizumi, Is my “red" your “red"?: Unsupervised alignment of qualia structures via optimal transport.  PsyArXiv (2023).
  * (12) T Brown, et al., Language models are few-shot learners.  Advances in neural information processing systems 33, 1877–1901 (2020).
  * (13) OpenAI, Gpt-4 technical report (2023).
  * (14) A Srivastava, et al., Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  arXiv preprint arXiv:2206.04615 (2022).
  * (15) A Goldstein, et al., Shared computational principles for language processing in humans and deep language models.  Nature neuroscience 25, 369–380 (2022).
  * (16) S Kumar, et al., Reconstructing the cascade of language processing in the brain using the internal computations of a transformer-based language model.  BioRxiv pp. 2022–06 (2022).
  * (17) R Tikochinski, A Goldstein, Y Yeshurun, U Hasson, R Reichart, Perspective changes in human listeners are aligned with the contextual transformation of the word embedding space.  Cerebral Cortex p. bhad082 (2023).
  * (18) M Abdou, et al., Can language models encode perceptual structure without grounding? a case study in color.  arXiv preprint arXiv:2109.06129 (2021).
  * (19) K Siedenburg, C Saitis, How does chatgpt rate sound semantics?  arXiv preprint arXiv:2304.07830 (2023).
  * (20) C Zhang, B Van Durme, Z Li, E Stengel-Eskin, Visual commonsense in pretrained unimodal and multimodal models.  arXiv preprint arXiv:2205.01850 (2022).
  * (21) R Marjieh, et al., Words are all you need? capturing human sensory similarity with textual descriptors.  The Eleventh International Conference on Learning Representations (2022).
  * (22) D Dillion, N Tandon, Y Gu, K Gray, Can ai language models replace human participants?  Trends in Cognitive Sciences (2023).
  * (23) D Ganguli, et al., The capacity for moral self-correction in large language models.  arXiv preprint arXiv:2302.07459 (2023).
  * (24) RN Shepard, Multidimensional scaling, tree-fitting, and clustering.  Science 210, 390–398 (1980).
  * (25) G Ekman, Dimensions of color vision.  The Journal of Psychology 38, 467–474 (1954).
  * (26) DE Kornbrot, Theoretical and empirical comparison of luce’s choice model and logistic thurstone model of categorical judgment.  Perception & Psychophysics 24, 193–208 (1978).
  * (27) P Esling, A Bitton, , et al., Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics.  arXiv preprint arXiv:1805.08501 (2018).
  * (28) TP Hettinger, JF Gent, LE Marks, ME Frank, study of taste perception.  Perception & Psychophysics 61, 1510–1521 (1999).
  * (29) B Berlin, P Kay, Basic color terms: Their universality and evolution.  (Univ of California Press), (1991).
  * (30) P Kay, B Berlin, L Maffi, WR Merrifield, R Cook, The world color survey.  (Citeseer), (2009).
  * (31) DT Lindsey, AM Brown, The color lexicon of american english.  Journal of vision 14, 17–17 (2014).
  * (32) RN Shepard, Geometrical approximations to the structure of musical pitch.  Psychological review 89, 305 (1982).
  * (33) N Jacoby, et al., Universal and non-universal features of musical pitch perception revealed by singing.  Current Biology 29, 3229–3243 (2019).
  * (34) GV Paramei, YA Griber, D Mylonas, An online color naming experiment in russian using munsell color samples.  Color Research & Application 43, 358–374 (2018).
  * (35) J Winawer, et al., Russian blues reveal effects of language on color discrimination.  Proceedings of the national academy of sciences 104, 7780–7785 (2007).
  * (36) MN Hebart, CY Zheng, F Pereira, CI Baker, Revealing the multidimensional mental representations of natural objects underlying human similarity judgements.  Nature human behaviour 4, 1173–1185 (2020).
  * (37) DE Blasi, J Henrich, E Adamou, D Kemmerer, A Majid, Over-reliance on english hinders cognitive science.  Trends in cognitive sciences (2022).
  * (38) TY Zhuo, Y Huang, C Chen, Z Xing, Exploring ai ethics of chatgpt: A diagnostic analysis.  arXiv preprint arXiv:2301.12867 (2023).
  * (39) RN Shepard, Toward a universal law of generalization for psychological science.  Science 237, 1317–1323 (1987).
  * (40) CR Sims, Efficient coding explains the universal law of generalization in human perception.  Science 360, 652–656 (2018).
  * (41) R Marjieh, TL Griffiths, N Jacoby, Musical pitch has multiple psychological geometries.  bioRxiv (2023).
  * (42) P Harrison, et al., Gibbs sampling with people in Advances in Neural Information Processing Systems, eds. H Larochelle, M Ranzato, R Hadsell, MF Balcan, H Lin.  (Curran Associates, Inc.), Vol. 33, pp. 10659–10671 (2020).
  * (43) KJ Woods, MH Siegel, J Traer, JH McDermott, Headphone screening to facilitate web-based auditory experiments.  Attention, Perception, & Psychophysics 79, 2064–2072 (2017).
  * (44) N Kriegeskorte, M Mur, PA Bandettini, Representational similarity analysis-connecting the branches of systems neuroscience.  Frontiers in systems neuroscience 2, 4 (2008).
  * (45) J Clark, The ishihara test for color blindness.  American Journal of Physiological Optics (1924).
  * (46) WM Rand, Objective criteria for the evaluation of clustering methods.  Journal of the American Statistical association 66, 846–850 (1971).

## Extended Methods

### Explicit GPT prompts

GPT prompt elicitation experiments were conducted using the OpenAI Text
Completion (for GPT-3) and Chat Completion (for GPT-3.5 and GPT-4) APIs. The
temperature was set to 0.7 for similarity judgments and 0 for color naming
experiments. The exact prompt formats used are shown below.

#### Similarity judgments

##### Color:

> People described pairs of colors using their hex codes. How similar are the
> two colors in each pair on a scale of 0-1 where 0 is completely dissimilar
> and 1 is completely similar? Respond only with the numerical similarity
> rating.
> Color 1: #ff5700 Color 2: #ff9b00 Rating 0.76
> Color 1: #b3ff00 Color 2: #00ff61 Rating: 0.45
> Color 1: #FF0000 Color 2: #00b2ff Rating: 0.02
> Color 1: $<$hex-code1$>$ Color 2: $<$hex-code2$>$
> Rating:

##### Pitch:

> People described pairs of musical notes using their frequencies in hertz.
> How similar are the musical notes in each pair on a scale of 0-1 where 0 is
> completely dissimilar and 1 is completely similar?
>
> Note 1: 587.3295358348151 Hz
> Note 2: 987.7666025122483 Hz
> Rating: 0.46083740655517463
>
> Note 1: 349.2282314330039 Hz
> Note 2: 277.1826309768721 Hz
> Rating: 0.743838237117938
>
> Note 1: 415.3046975799451 Hz
> Note 2: 987.7666025122483 Hz
> Rating: 0.19874605585261726
>
> Note 1: $<$frequency1$>$
> Note 2: $<$frequency2$>$
> Rating:

##### Vocal consonants:

> People described vocal consonants using the international phonetic alphabet
> (IPA).
> How similar do the vocal consonants in each pair sound on a scale of 0-1
> where 0 is completely dissimilar and 1 is completely similar?
> Respond only with the numerical similarity rating.
>
> Vocal Consonant 1: f
> Vocal Consonant 2: m
> Rating: 0.5
>
> Vocal Consonant 1: n
> Vocal Consonant 2: Z
>  Rating: 0.40740740740740744
>
> Vocal Consonant 1: S
> Vocal Consonant 2: S
>  Rating: 1.0
>
> Vocal Consonant 1: $<$consonant1$>$
> Vocal Consonant 2: $<$consonant2$>$
> Rating:

##### Loudness:

> People described the loudness of pure tones in decibels (dB).
> How similar do the pure tones in each pair sound on a scale of 0-1 where 0
> is completely dissimilar and 1 is completely similar?
>
> Pure Tone 1: 72.6 dB
> Pure Tone 2: 74.1 dB
> Rating: 0.3495324720283043
>
> Pure Tone 1: 74.6 dB
> Pure Tone 2: 73.6 dB
> Rating: 0.5055839477695901
>
> Pure Tone 1: 74.1 dB
> Pure Tone 2: 74.1 dB
> Rating: 1.0
>
> Pure Tone 1: $<$loudness1$>$
> Pure Tone 2: $<$loudness2$>$
> Rating:

##### Taste:

> People described flavors they tasted using words.
> How similar are the flavors in each pair on a scale of 0-1 where 0 is
> completely dissimilar and 1 is completely similar?
>
> Flavor 1: quinine
> Flavor 2: artificial sweetener
> Rating: 0.0
>
> Flavor 1: artificial sweetener
> Flavor 2: salt
> Rating: 0.015433904145892428
>
> Flavor 1: quinine-sugar
> Flavor 2: acid-sugar
> Rating: 0.2539115246067999
>
> Flavor 1: $<$flavor1$>$
> Flavor 2: $<$flavor2$>$
> Rating:

##### Timbre:

> People listened to pairs of musical instruments and rated the similarity of
> their timbre.
> How similar is the timbre of the instruments in each pair on a scale of 0-1
> where 0 is completely dissimilar and 1 is completely similar?
>
> Instrument 1: Cello
> Instrument 2: Flute
> Rating: 0.5604846433040316
>
> Instrument 1: Flute
> Instrument 2: Clarinet
> Rating: 0.270932601836378
>
> Instrument 1: Trombone
> Instrument 2: Bassoon
> Rating: 0.2893895067551666
>
> Instrument 1: $<$instrument1$>$
> Instrument 2: $<$instrument2$>$
> Rating:

#### Color naming

##### Basic color free-elicitation:

English:

> Name 15 basic colors.

Russian:

> Перечислите 15 основных цветов.

##### Color naming elicitation:

English:

> Here is a list of 15 basic color names: $<$shuffled basic color list$>$.
> Which of these names best describes the following color: $<$hex-code$>$?
> Respond only using the name.

Russian:

> Вот список из 15 названий основных цветов: $<$shuffled basic color list$>.$
> Какое из названий цветов лучше всего описывает следующий цвет: $<$hex-
> code$>$?
> Отвечайте только названием одного цвета из списка.
>

We repeated this prompt ten times for each WCS color with the basic color list
shuffled each time and temperature set to the default 0.7 to elicit ten names
per WCS color. For each of the ten elicitations per color, if the output was
not one of the 15 basic colors we would keep re-querying GPT until it did give
a valid color output (GPT-4 is slightly non-deterministic even at temperature
0 due to changes in hardware). If after 10 attempts the response was still
invalid, we would return “error” as the color (this response is later
discarded from the analysis).

### Additional details of behavioral experiments

#### Similarity Experiments: Participants

To collect similarity judgments over pitch and vocal consonants we deployed
two online experiments on Amazon Mechanical Turk
(AMT).777https://www.mturk.com/ The recruitment and experimental pipelines
were automated using PsyNet (42), a modern framework for experiment
design888https://psynet.dev/ and deployment which builds on the
Dallinger999https://dallinger.readthedocs.io/ platform for recruitment
automation. Overall, 55 participants completed the pitch study and 64
participants completed the vocal consonants study. Participants were recruited
from the United States, were paid $9-12 USD per hour, and provided informed
consent as approved by the Princeton IRB (#10859) and the Max Planck Ethics
Council (#2021_42).

To enhance data quality, participants had to pass a standardized headphone
check (43) that ensures good listening conditions and task comprehension, and
were required to have successfully completed at least 3000 tasks on AMT. Upon
passing the prescreening stage, participants were randomly assigned to rate
the similarity between different pairs of stimuli and provided numerical
judgments on a 7-Likert scale ranging from 0 (completely dissimilar) to 6
(completely similar). In the pitch experiment, participants provided an
average of 80 judgments, and in the vocal consonants experiment an average of
55 judgments. See SI for explicit instructions.

#### Similarity Experiments: Procedure

Upon providing informed consent and passing the headphone check, participants
received the following instructions. In the case of the pitch experiment: “In
this experiment we are studying how people perceive sounds. In each round you
will be presented with two sounds and your task will be to simply judge how
similar those sounds are. You will have seven response options, ranging from 0
(‘Completely Dissimilar’) to 6 (‘Completely Similar’). Choose the one you
think is most appropriate. You will also have access to a replay button that
will allow you to replay the sounds if needed. Note: no prior expertise is
required to complete this task, just choose what you intuitively think is the
right answer.” Participants were then informed of an additional small quality
bonus “The quality of your responses will be automatically monitored, and you
will receive a bonus at the end of the experiment in proportion to your
quality score. The best way to achieve a high score is to concentrate and give
each round your best attempt”. While the task is subjective in nature, we used
consistency as a proxy for quality by repeating 5 random trials at the end of
the experiment and computing the Spearman correlation $s$ between the original
responses and their repetitions. The final bonus was computed using the
formula $\min(\max(0.0,0.1s),0.1)$ yielding at most 10 cents. In the main
experiment participants were assigned to random stimulus pairs and were
instructed to rate their similarity using the following prompt: “How similar
are the pair of sounds you just heard?” and provided a response on a Likert
scale. The procedure for the vocal consonants similarity experiment was
identical up to the specific instructions. Specifically, participants received
the following instructions: “In this experiment we are studying how people
perceive the sound of vocal consonants. A consonant is a speech sound that is
pronounced by partly blocking air from the vocal tract. For example, the sound
of the letter _c_ in _cat_ is a consonant, and so is _t_ but not _a_.
Similarly, the sound of the combination _sh_ in _sheep_ is a consonant, and so
is _p_ but not _ee_. In general, vowel sounds like those of the letters _a, e,
i, o, u_ are not consonants.” The instructions then proceeded: “In each round
you will be presented with two different recordings each including one
consonant sound and your task will be to simply judge how similar are the
sounds of the two spoken consonants. We are not interested in the vowel sounds
nor in the voice height, just the sound of the consonants. You will have seven
response options, ranging from 0 (‘Completely Dissimilar’) to 6 (‘Completely
Similar’). Choose the one you think is most appropriate. Note: no prior
expertise is required to complete this task, just choose what you intuitively
think is the right answer.” Participants were then informed of the quality
bonus which was identical to the pitch task, and then rated the similarity
between pairs of random consonants based on the following prompt “How similar
is the sound of the consonants pronounced by the two speakers?” and a Likert
scale as before.

#### Similarity Experiments: Model Evaluation

We quantified model performance in predicting human similarity judgments by
computing the Pearson correlation coefficient between the flattened upper
triangle of the LLM-based and human-based similarity matrices (to account for
the fact that these matrices are symmetric). This approach is similar to
representational similarity analysis (44). To compute 95% confidence
intervals, we bootstrapped with replacement over model predictions with 1,000
repetitions and computed for each repetition the average similarity matrix. We
then correlated the upper triangles of each of those matrices with human data
to produce a list of correlation coefficients on which we computed confidence
intervals.

#### Color Naming Experiments: Participants

To collect the color naming data in Russian and British English participants,
we ran online experiments on Prolific101010https://www.prolific.com. Overall,
we recruited 103 UK participants and 51 Russian participants. All texts in the
interface of the experiment (e.g., buttons, instructions, etc) were presented
in the native language of the participant. The Russian texts were first
automatically translated using DeepL111111https://www.deepl.com and then
manually checked and corrected by a native speaker of Russian (author I.S).
Participants had to be raised monolingually and to speak the target language
as their mother tongue. Each participant was paid 9 GBP per hour and provided
informed consent according to an approved protocol (Max Planck Ethics Council
#2021_42). The experiment was implemented using PsyNet (42). Each session
starts with a free-elicitation task where participants are asked to provide
basic colors:

#### Color Naming Experiments: Procedure

##### Basic color free-elicitation:

English:

> Please name at least 8 basic color names.
> Press enter after each color name.
> Only use lower-case letters.
>

Russian:

> Укажите не менее 8 названий основных цветов.
> Нажмите клавишу Enter после каждого названия цвета.
> Используйте только строчные буквы.
>

Participants may only submit color names without spaces, numbers, or special
characters and can only submit the page if they have provided at least eight
names. The list of obtained colors is highly overlapping with the GPT-4 list,
justifying our choice to use GPT-4 as the basis for the word naming task
(colors are sorted by their naming frequency).

Top 15 terms in English:

> “blue”, “green”, “yellow”, “red”, “purple”, “orange”, “black”, “pink”,
> “white”, “brown”, “grey”, “violet”, “indigo”, “turquoise”, “silver”
>

Top 15 terms in Russian:

> “красный”, “синий”, “белый”, “зеленый”, “оранжевый”, “желтый”, “фиолетовый”,
> “черный”, “голубой”, “коричневый”, “розовый”, “серый”, “жёлтый”, “зелёный”,
> “чёрный”
>

From the top 15 color terms, 11 (English) and 12 (Russian) color terms are
overlapping with the list provided by GPT-4. Before the main experiments,
participants received the following instructions:

##### Color naming instructions:

English:

> During this experiment, you will be presented with a square of a particular
> color and will be required to select the most suitable color term from a
> list of options.
> Please be aware that some of the colors may be repeated to verify
> consistency of your choices.
> If we detect any inconsistencies in your answers, we may terminate the
> experiment prematurely.
> The best strategy is to answer each question truthfully, as attempting to
> memorize responses may prove difficult.
>

Russian:

> В ходе этого эксперимента вам будет представлен квадрат определенного цвета,
> и вам нужно будет выбрать наиболее подходящее название цвета из списка
> вариантов.
> Имейте в виду, что некоторые цвета могут повторяться, чтобы убедиться в
> согласованности вашего выбора.
> Если мы обнаружим какие-либо несоответствия в ваших ответах, мы можем
> досрочно прекратить эксперимент.
> Лучшая стратегия - отвечать на каждый вопрос правдиво, так как попытка
> запомнить ответы может оказаться сложной.
>

The participants then went through the main experiment:

##### Color naming task:

English:

> $<$square of a particular color$>$
> You will see below a list of 15 basic color names. Which of these names best
> describes the color above?
> $<$shuffled basic color list presented as buttons$>$
>

Russian:

> $<$square of a particular color$>$
> Ниже Вы увидите список из 15 основных названий цветов.
> Какое из этих названий лучше всего описывает вышеуказанный цвет?
> $<$shuffled basic color list presented as buttons$>$
>

At the end of the experiment, the participant took a color blindness test
(45). Some participants abandoned the experiment prematurely, but we
nevertheless included their responses (42 English participants, 3 Russian
participants). Only a fraction of the participants failed the color blindness
test (5 of 103 English participants, and 2 of 51 Russian participants).
Consistent with the WCS we included all participants including those who
failed the color blindness test. In a control analysis, we excluded all color
blind individuals and all participants that did not complete the entire
session and got nearly identical results (the adjusted Rand index was 0.92 for
English experiments and 0.97 for Russian experiments).

#### Color Naming Experiments: Analysis

For each color, we collected at least 10 responses per LLM variant, and at
least 10 forced-choice human selections per color (English mean 19.30
responses, Russian mean 12.17 responses). Consistent with previous literature
for each Munsell’s color we selected the most frequently reported term. We
then presented the dominant colors in Figure 3B. To aid visualization we
average the RGB values of all colors with the same color term, and presented
them as the legend and clustered color in that figure. We also listed per
color the number of Munsell’s colors that were associated with each dominant
color term. Figure 3B provides additional information on the degree of
agreement for each color. Colors for which less than 50% and 90% of the times
the dominant color term was selected were indicated by “-” and “*”,
respectively. If the dominant color term was selected more than 90% of the
time, no marking was used.

#### Adjusted Rand index

The Rand index (46) is a label-insensitive measure of clustering similarity
that instead of relying on specific labels (e.g. “Blue”) quantifies the
similarity between two clustering partitions by counting pairs of items (in
our case Munsell colors) that are clustered consistently and dividing them by
the overall number of pairs. This allows to compare different clustering
schemes when the vocabulary of labels is not aligned (e.g. English and
Russian). Formally, we computed: $R=(b+c)/a$; where $b$ is the number of pairs
of items that are in the same subset in one clustering and in the same subset
in the other, $c$ is the number of pairs of items that are in different
subsets in one clustering and in different subsets in the other and $a$ is the
total number of pairs. The Rand index provides high values for two random
clusterings, to adjust for this we used the corrected-for-chance version of
the Rand index (46), which normalizes the raw value by the expected value of
the Rand index for random clusterings. Formally, we have $ARI=(RI-
RI_{\text{rand}})/(1-RI_{\text{rand}})$ where $ARI$ is the adjusted Rand
index, $RI$ is the raw Rand index and $RI_{\text{rand}}$ is the expected Rand
index for random clusterings. The adjusted Rand index is thus ensured to have
a value close to 0.0 for random labeling independently of the number of
clusters and samples, exactly 1.0 when the clusterings are identical (up to a
permutation), and reaches -0.5 for “orthogonal clusters” that are less
consistent relative to what is expected by chance. In our case, all values
were strictly positive suggesting consistency across languages and
experiments. To compute confidence intervals, we created bootstrapped datasets
by sampling the responses of each color with replacement and recomputing the
dominant selected color name. We then obtained CIs by computing the adjusted
Rand index for 1,000 pairs of bootstrapped datasets.

#### Lindsey and Brown dataset

We compared our experimental data to a dataset by (31), reproduced with
permission by Delwin Lindsey. The data contains two experimental conditions
conducted in the lab with the same 51 participants. In the first condition,
participants were instructed to provide free naming responses. In the second
condition, participants were instructed to choose from a pre-specified list of
11 color terms: (Green, Blue, Purple, Pink, White, Brown, Orange, Yellow, Red,
Black, and Gray). Despite the fact that the Lindsey & Brown experiment was
conducted in the lab (and not online like our experiments) and that the
constrained list in our experiment was somewhat different, the results of both
experiments were highly consistent with our human English data (Constrained,
$ARI=0.73$ 95% CI $[0.65,0.74]$, free naming $0.75$ 95% CI $[0.66,0.75]$). In
addition to putting an upper bound on the consistency with which the LLM can
predict human data (by comparing it with another human experiment), these
results prove that despite less control over color presentation compared to
the lab, online presentation still provides high-quality color naming data.