# Low-coherence off-axis digital holographic microscopy

Stephane Perrin  Advanced Photonics Laboratory, Ecole Polytechnique Fédérale
de Lausanne, CH-1015 Lausanne, Switzerland  Corresponding author:
<EMAIL_ADDRESS>Jonas Kühn  Advanced Photonics Laboratory, Ecole
Polytechnique Fédérale de Lausanne, CH-1015 Lausanne, Switzerland  Department
of Astronomy, University of Genève, CH-1211 Genève, Switzerland  Christian
Depeursinge  Advanced Photonics Laboratory, Ecole Polytechnique Fédérale de
Lausanne, CH-1015 Lausanne, Switzerland  Laboratory for Cellular Imaging and
Energetics, King Abdullah University of Science and Technology, Thuwal,
Kingdom of Saudi Arabia

###### Abstract

Usually, off-axis digital holographic microscopy requires a coherent light
source in order to record a full-field hologram. Nevertheless, a LASER-based
illumination leads to a non-negligible coherent noise, decreasing then the
imaging quality. We hereby report a simple method to reduce the coherent noise
contribution using a low-temporal-coherence illumination while maintaining a
large interference area. A diffraction grating is hence introduced in the
reference arm of the interferometer, allowing the coherence plane of the
reference beam to be tilted following angular dispersion. The phase planes of
the reference beam and the object beam appears to be coplanar. The principle
and performance of low-coherence off-axis digital holographic microscopy are
demonstrated. The three-dimensional reconstruction of a biological sample is
performed.

††journal: ol

Since the demonstration of recording a modulated interference signal using a
vidicon detector in 1967 by W. Goodman [1], digital holography has been
improved through the enhancement of sensors and computing power [2]. At the
end of the nineties, digital holography was combined to optical microscopy,
leading to the development of a non-invasive three-dimensional imaging
technique, called digital holographic microscopy (DHM) [3, 4, 5, 6]. DHM is
nowadays widely employed for the visualization of biological samples [7, 8, 9,
10, 12, 11] as well as for the characterization of micro-optical elements [13,
14, 15, 16, 17]. DHM provides an interferometric nanometeric accuracy [18] and
a lateral resolution limited by the diffraction of light [19]. Recently,
super-resolution methods has even been implemented in order to perform a sub-
wavelength transversal resolution [20, 21, 22, 23]. Nevertheless, DHM is
subject to coherent noise due to the use of spatially and temporally coherent
light sources [24]. As a matter of fact, the reflections of coherent beams by
multiple optical interfaces generate undesirable interference signals,
contaminating the hologram and thus introducing unwanted artefacts in the
reconstructed image.

In the last decade, many approaches have been developed in order to remove the
coherent noise such as recording wavelength-dependent holograms [25],
displacing or rotating the sample [26, 27], implementing a syntetic
illumination aperture [28] and processing the images [29]. Ref. [30] reports
these best-performing noise reduction approaches. However, these techniques
present some drawbacks, _i.e._ reducing the resolution of the imaging system,
decreasing the contrast of the interference pattern or necessitating multiple
acquisitions. The coherent noise only occurs when the distance between two
nearby interfaces is smaller than the coherence length of the light source.
Thus, decreasing the spatial or the temporal coherence of light has also been
suggested [31, 32, 33]. In addition, using an incoherent illumination tends to
increase the lateral resolution of an imaging system [19]. Nevertheless, this
implies narrowing the width of the interference pattern in off-axis DHM
configuration. At the begining of 1990’s, it has been shown that angular
dispersion from a diffraction grating (or a prism) provides a tilt of the
pulse-front while not affecting the group velocity [34, 35, 36, 37]. Indeed, a
diffraction grating not only separates an incident beam into several beams at
different angles, but also tilts the pulse plane of a broadband incident beam
[38]. In other words, the pulse front of a beam from a low-coherent light
source appears no longer co-planar to the direction of the beam propagation
after passing through a diffraction element, but inclined with respect to the
diffraction order angle. This phenomenon gave the opportunity to make the
coherence planes of two angled incoherent beams parallel themselves and,
applied to holography, made it possible the full-field single-shot hologram
acquisition using two prisms [39], a diffraction grating [40] or multiple
gratings combined with a dual-wavelengths illumination [41, 42].

This letter presents an easy-to-implement method enabling to enhance the
experimental conditions in transmissive off-axis DHM. Indeed, the interference
system requires a low-coherence illumination and a standard diffraction
grating in the reference arm for making coplanar the beam coherence planes.
The concept of low-coherence off-axis DHM is exposed and the increase in
interference pattern width is demonstrated. Furthermore, performance of the
imaging technique is estimated. Then, the volume of a neuron is reconstructed.

The layout of the off-axis DHM setup is shown in Fig. 1 and consists of a
super-luminescent diode having a central wavelength $\lambda_{0}$ of 680 nm
and a bandwidth $\Delta\lambda$ of 8 nm (SLD-26-HP, Superlum Diode). The
incident beam is firstly spatially filtered before being divided in two by a
beam-splitter cube. The reference beam R of the interferometer is transmitted
and passes through a diffraction grating (46-067, Edmund Optics) having a
groove density of 70 lines/mm. According to angular dispersion, the coherence
plane is tilted by an angle $\Phi(\lambda_{0})$ of 2.7° with respect to the
propagating vector [35]. Only the $+1$-order diffracted beam from the grating
is required in the imaging system. Other order beams are removed using an
obstruction which is not represented in Fig. 1.

Figure 1: Experimental configuration of the low-coherence off-axis DHM. SLD,
Super-luminescent diode; SF, spatial filter; M, mirror; BSC, beam-splitter
cube; G, diffraction grating; CP, coherence plane; S, sample; MO, microscope
objective; TL, tube lens; D, detector.

The beam reflected by the cube, i.e. the object beam O, is oriented by a
mirror in order to illuminate the specimen to be measured. The microscope
objective ($\times$5, NA = 0.17) collects the beam scattered by the sample and
the tube lens forms the image on the CCD camera (TXG50, Baumer). The second
beam-splitter cube allows to surimpose the O beam with the R beam in the off-
axis mode. Therefore, the mirror in the reference arm is able to adjust the
incident angle $\theta$ between the two beams. The interferogram distribution
is then recorded by the camera and, finally, an algorithm processes the full-
field hologram and performs a numerical wavefront propagation using 2D Fast
Fourier Transform (FFT) operators [3]. This numerical operation makes allows
to retrieve the phase distribution of the sample.

Without the diffraction element in the reference arm, the width of
interference area $L$ is limited by the coherence length of the light source
$l_{c}$ and the angle $\theta$ between the two beams
($L=l_{c}/\sin\left(\theta\right)$). Assuming a Gaussian-spectrum light
illumination, the number of interference fringes $N$, having an interference
contrast superior to 50%, can be expressed as [43]:

$N=\frac{2\ln(2)}{\pi}\leavevmode\nobreak\ \frac{\lambda_{0}}{\Delta\lambda}$
(1)

Figure 2: Increase in the interference area size. (a) Overlap of two
incoherent beams in conventional off-axis DHM ($\theta$ = 2.7°). The
interference width $L$ equals 533 µm. (b) Overlap of the two incoherent beams
by placing a diffraction grating in the reference arm. The camera records an
interference pattern over the whole sensor surface. The first-order
diffraction angle of the grating $\Phi(\lambda_{0})$ equals the angle
$\theta$. The normalized intensity profiles are plotted according to the white
doted lines. (c) Evolution of both the calculated (in blue) and the measured
(in red) number of fringes $N$ as a function of the angle $\theta$. Within the
green area, $L$ is wider than the sensor size.

In this case, the camera records only 38 interference fringes, providing thus
fringe-less observation regions as shown in Fig. 2(a). The diffraction grating
is used to tilt the Poynting vector with respect to the propagation direction.
And, when $\theta$ equals $\Phi(\lambda_{0})$, the coherence planes are
collinear and an uniform interference contrast is thus recorded over the
entire sensor area. Figure 2(b) shows the resulting 2D interference pattern,
performing a mean interference contrast of 80%. The number of fringe is now
defined as a function of the angles $\theta$ and $\Phi$.

$N=\frac{2\ln(2)}{\pi}\leavevmode\nobreak\
\frac{\lambda_{0}}{\Delta\lambda}\leavevmode\nobreak\
\frac{\sin\left(\theta\right)}{\sin\left(\Phi(\lambda)\leavevmode\nobreak\
-\leavevmode\nobreak\ \theta\right)}$ (2)

Figure 2(c) shows the evolution of the number of fringes $N$ from Eq. 2 (blue
dashed plot) as a function of the incident angle $\theta$. When $\theta$
equals $\Phi(\lambda_{0})$, an infinity number of fringes should be recorded
by the camera. However, in experiment (red solid plot), the 5-mm-size sensor
limits the lateral field of view, yielding a linear evolution of $N$ between
2.2° and 3.2° (green area). For the measurements, the angle $\theta$ was
implemented by rotating the mirror in the reference arm around the optical
axis and the DHM system was free of object. Note that the incident angle
$\theta$ depends also on the wavelength of light due to the bandwidth of the
light source. This involves adjusting the grating-camera distance in order to
record all the wavelength-dependent diffracted beams, _i.e._ the contributions
from each spectral component. Not considering this chromatic effect could lead
to a fringe wash-out and further a loss of the interference contrast. In
addition, at the ideal angular position, the angle $\theta(\lambda)$ equals
$\Phi(\lambda)$ regardless the wavelength.

Performance of the low-coherence interference imaging system was evaluated.
Figure 3(a) shows the hologram of a 1951 USAF target, allowing the lateral
resolution to be determined. It results in 2.19 µm of resolution limit
(Element 6, Group 7). This allows to highlight that the diffraction grating
has not impact on the imaging quality and the DHM system is thus assumed
diffraction limited (the cut-off frequency of the optical transfer function of
an aberration-free imaging system equals 2.00 µm [19]). Note that the
resolving power of the imaging system would be around 4 µm using a 680-nm-
wavelength monochromatic light source.

Figure 3: Performance of low-coherence off-axis DHM system. (a) Hologram of a
1951 USAF target. The DHM system can resolve Element 6, Group 7. (b)
Measurement of the spatial phase deviation using a flat mirror. The RMS of the
phase deviation is 32 mrad within blue area, 30 mrad within black area and 27
mrad within green and red areas. Each square area consists of 100 $\times$ 100
pixels. $\lambda_{0}$ = 680 nm, NA = 0.17. Figure 4: 3D reconstruction of a
mouse’s neuron using low-coherence off-axis DHM. (a) Width-field hologram. (b)
reconstruction of the morphology. The soma, the dendrites and the axons of the
neuron are recognized with the blue, the green and the red arrows,
respectively. $\lambda_{0}$ = 680 nm, NA = 0.17. White scale bar represents
100 µm.

Furthermore, a quantitative analysis of the phase deviation has been measured
(Fig. 3(b)) through the reconstruction of a plane mirror. The root mean square
(RMS) of the phase noise equals 30 mrad, i.e. 3.2 nm of height deviation, over
the entire field of view. Usually, DHM provides an axial sensibility of around
10 nm [3, 42, 44]

Finally, a biological element has been introduced in the low-coherence off-
axis DHM system for tracking the shape and the cellular behaviour. Figure 4(a)
shows the width-field hologram of the fixed mouse neuron recorded by the
camera. The quantitative phase distribution was then calculated, enabling the
morphology of the brain cell to be reconstructed (Fig. 4(b)). Assuming a mean
refractive index of 1.375 along the propagation axis inside the mouse brain
cell [7, 45], the height of at the center of the neuronal cell body (blue
arrow in Fig. 4(a)), i.e. the soma, equals 12.9 µm. And, the height of the
dendrites (green arrow in Fig. 4(a)) is around 3.8 µm.

This work presents the development of a low-coherence off-axis digital
holographic microscope. Illuminated by a broadband light source, the parasitic
coherent noise contributions which originate from multiple reflections between
the optical component interfaces, are reduced. Despite, this type of
illumination narrows the hologram width in off-axis configuration, a
diffraction grating has been introduced in the reference arm of the
transmissive-configuration interferometer in order to make parallel the
reference and the object wave-packets. This method allows the hologram to
cover the camera and, further, the standard deviation of the phase
reconstruction to be lower than 30 mrad. The imaging technique has been
applied for the reconstruction of a neuron cell. It can be mentioned that this
method can be implemented in a reflective configuration. Funding This research
was funded by the Swiss National Science Foundation (SNSF) and has been
supported by King Abdullah University of Science and Technology. This work has
been made in collaboration with the Microvision Microdiagnostic Group of the
Ecole Polytechnique Federale de Lausanne and Lyncee Tec SA. Disclosures The
authors declare no conflicts of interest.

## References

  * [1] Goodman, W. and Lawrence, R.W. Digital image formation from electronically detected holograms. _Appl. Phys. Lett._ 1967, _11_ , 77-79.
  * [2] Schnars, U. and Juptner, W. Direct recording of holograms by a CCD target and numerical reconstruction. _Appl. Opt._ 1994, _33_ , 179-181.
  * [3] Cuche, E., Marquet, P., and Depeursinge, C. Simultaneous amplitude and quantitative phase-contrast microscopy by numerical reconstruction of Fresnel off-axis holograms. _Appl. Opt._ 1999, _38_ , 6994-7001.
  * [4] Ferraro, P., Coppola, G., De Nicola, S., Finizio, A., and Pierattini, G. Digital holographic microscope with automatic focus tracking by detecting sample displacement in real time. _Opt. Lett._ 2003, _28_ , 1257-1259.
  * [5] Kim, M.K. _Digital Holographic Microscopy_ , Springer-Verlag, New York, USA, 2011.
  * [6] Osten, W., Faridian, A., Gao, P., Korner, K., Naik, D., Pedrini, G., Singh, A.K., Takeda, M., and Wilke, M. Recent advances in digital holography. _Appl. Opt._ 2014, _53_ , G44-G63.
  * [7] Marquet, P., Rappaz, B., Magistretti, P.J., Cuche, E., Emery, Y., Colomb, T., and Depeursinge, C. Digital holographic microscopy: a noninvasive contrast imaging technique allowing quantitative visualization of living cells with subwavelength axial accuracy. _Opt. Lett._ 2005, _30_ , 468-470.
  * [8] Kemper, B., Carl, D., Schnekenburger, J., Bredebusch, I., Schafer, M., Domschke, W., and von Bally, G. Investigation of living pancreas tumor cells by digital holographic microscopy. _J. Biomed. Opt._ 2006, _11_ , 033001.
  * [9] Kemper, B. and von Bally, G. Digital holographic microscopy for live cell applications and technical inspection. _Appl. Opt._ 2008, _47_ , A52-A61.
  * [10] Molder, A., Sebesta, M., Gustafsson, M., Gisselson, L., Wingren, A.G., and Alm, K. Non-invasive, label-free cell counting and quantitative analysis of adherent cells using digital holography. _J Microsc._ 2008, _232_ , 240-247.
  * [11] Yi, F., Moon, I., and Javidi, B. Cell morphology-based classification of red blood cells using holographic imaging informatics. _Biomed. Opt. Express_ 2016, _7_ , 2385-2399.
  * [12] Yi, F., Moon, I., and Javidi, B. Automated red blood cells extraction from holographic images using fully convolutional neural networks. _Biomed. Opt. Express_ 2017, _8_ , 4466-4479.
  * [13] Osten, W., Seebacher, S., and Jueptner, W. Application of digital holography for the inspection of microcomponents. _Proc. SPIE_ 2001, _4400_.
  * [14] Charrière, F., Kühn, J., Colomb, T., Montfort, F., Cuche, E., Emery, Y., Weible, K., Marquet, P., and Depeursinge, C. Characterization of microlenses by digital holographic microscopy. _Appl. Opt._ 2006, _45_ , 829-835.
  * [15] Ferraro, P. and Osten, W. Digital holography and its application in MEMS/MOEMS inspection. In _Optical Inspection of Microsystems_ ; Osten, W. Eds.; Taylor Francis, 2006; pp. 351-425.
  * [16] Weijuan, Q., Choo, C.O., Yingjie, Y., and Asundi, A. Microlens characterization by digital holographic microscopy with physical spherical phase compensation._Appl. Opt._ 2010, _49_ , 6448-6454.
  * [17] Kozacki, T., Jozwik, M., and Lizewski, K. High-numerical-aperture microlens shape measurement with digital holographic microscopy. _Opt. Lett._ 2011, _36_ , 4419-4421.
  * [18] Kuhn, J., Charriere, F., Colomb, T., Cuche, E., Montfort, F., Emery, Y., Marquet, P., and Depeursinge, C. Axial sub-nanometer accuracy in digital holographic microscopy. _Meas. Sci. Technol._ 2008, _19_ , 074007.
  * [19] Goodman, J.W. _Introduction To Fourier Optics_ , 3rd Ed., W.H. Freeman and Co Ltd, 2004.
  * [20] Gao, P., Pedrini, G., and Osten, W. Structured illumination for resolution enhancement and autofocusing in digital holographic microscopy. _Opt Lett._ 2013, _38_ , 1328-1330.
  * [21] Cotte, Y., Toy, F., Jourdain, P., Pavillon, N., Boss, D., Magistretti, P.J., Marquet, P., and Depeursinge, C. Marker-free phase nanoscopy. _Nat. Photon._ 2013, _7_ , 113-117.
  * [22] Aakhte, M., Abbasian, V., Akhlaghi, E.A., Moradi, A.-R., Anand, A., and Javidi, B. Microsphere-assisted super-resolved Mirau digital holographic microscopy for cell identification. _Appl. Opt._ 2017, _56_ , D8-D13.
  * [23] Lin, Y.-C., Tu, H.-Y., Wu, X.-R., Lai, X.-J., and Cheng, C.-J. One-shot synthetic aperture digital holographic microscopy with non-coplanar angular-multiplexing and coherence gating. _Opt. Express_ 2018, _26_ , 12620-12631.
  * [24] Chavel, P. Optical noise and temporal coherence. _J. Opt. Soc. Am._ 1980,_70_ , 935-943.
  * [25] Nomura, T., Okamura, M., Nitanai, E., and Numata, T. Image quality improvement of digital holography by superposition of reconstructed images obtained by multiple wavelengths. _Appl. Opt._ 2008, _47_ , D38-D43.
  * [26] Pan, F., Xiao, W., Liu, S., Wang, F., Rong, L., and Li, R. Coherent noise reduction in digital holographic phase contrast microscopy by slightly shifting object. _Opt. Express_ 2011, _19_ , 3862-3869.
  * [27] Zhong, Z., Xiao, W., Pan, F., and Che, L. Coherent noise reduction in digital holographic interferometry by slightly rotating object. _Proc. SPIE_ 2017, _10255_ , 1025543.
  * [28] Feng, P., Wen, X., and Lu, R. Long-working-distance synthetic aperture Fresnel off-axis digital holography. _Opt. Express_ 2009, _17_ , 5473-5480.
  * [29] Liu, Y., Wang, Z., Huang, J., Gao, J., Li, J., Zhang, Y., and Li, X. Coherent noise reduction of reconstruction of digital holographic microscopy using a laterally shifting hologram aperture. _Opt. Eng._ 2016, _55_ , 121725.
  * [30] Bianco, V., Memmolo, P., Leo, M., Montresor, S., Distante, C., Paturzo, M., Picart, P., Javidi, B., and Ferraro, P. Strategies for reducing speckle noise in digital holography. _Light Sci. Appl._ 2018, _7_ , 48.
  * [31] Pedrini, G. and Schedin, S. Short coherence digital holography for 3D microscopy. _Optik_ 2001, _112_ , 427-432.
  * [32] Dubois, F., Callens, N., Yourassowsky, C., Hoyos, M., Kurowski, P., and Monnom, O. Digital holographic microscopy with reduced spatial coherence for three-dimensional particle flow analysis. _Appl. Opt._ 2006, _45_ , 864-871.
  * [33] Langehanenberg, P., Bally, G., and Kemper, B. Application of partially coherent light in live cell imaging with digital holographic microscopy. _J. Mod. Opt._ 2010, _57_ 709-717.
  * [34] Martinez, O.E. Pulse distortions in tilted pulse schemes for ultrashort pulses. _Opt. Commun._ 1986, _59_ , 229-232.
  * [35] Bor, Z., Racz, B., and Szabo, G. Femtosecond pulse front tilt caused by angular dispersion. _Opt. Eng._ 1993, _32_ , 2501-2504.
  * [36] Hebling, J. Derivation of the pulse front tilt caused by angular dispersion. _Opt. Quant. Electron._ 1996, _28_ , 1759-1763.
  * [37] Maznev, A.A., Crimmins, T.F., and Nelson, K.A. How to make femtosecond pulses overlap. _Opt. Lett._ 1998, _23_ , 1378-1380.
  * [38] Torres, J.P., Hendrych, M., and Valencia, A. Angular dispersion: an enabling tool in nonlinear and quantum optics. _Adv. Opt. Photon._ 2010, _2_ , 319-369.
  * [39] Ansari, Z., Gu, Y., Tziraki, M., Jones, R., French, P.M.W., Nolte, D.D., and Melloch, M.R. Elimination of beam walk-off in low-coherence off-axis photorefractive holography. _Opt. Lett._ 2001, _26_ , 334-336.
  * [40] Yaqoob, Z., Yamauchi, T., Choi, W., Fu, D., Dasari, R.R., and Feld, M.S. Single-shot Full-field reflection phase microscopy. _Opt. Express_ 2011, _19_ , 7587-7595.
  * [41] Monemhaghdoust, Z., Montfort, F., Emery, Y., Depeursinge, C., and Moser, C. Dual wavelength full field imaging in low coherence digital holographic microscopy. _Opt. Express_ 2011, _19_ , 24005-24022.
  * [42] Monemhaghdoust, Z., Montfort, F., Cuche, E., Emery, Y., Depeursinge, C., and Moser, C. Full field vertical scanning in short coherence digital holographic microscope. _Opt. Express_ 2013, _21_ , 12643-12650.
  * [43] Saleh, B.E.A. and Teich, M.C. Statistical Optics. In _Fundamentals of Photonics, 2nd Ed._ ; Wiley, 2007; pp. 403-443.
  * [44] Mann, C.J., Bingham, P.R., Paquit, V.C., and Tobin, K.W. Quantitative phase imaging by three-wavelength digital holography. _Opt. Express_ 2008, _16_ , 9753-9764.
  * [45] Marquet, P., Depeursinge, C., and Magistretti, P.J. Exploring neural cell dynamics with digital holographic microscopy. _Annu. Rev. Biomed. Eng._ 2013 _15_ , 407-431.


# Comparing Color Similarity Structures between Humans and LLMs via
Unsupervised Alignment

Genji Kawakita
Imperial College London
<EMAIL_ADDRESS>
&Ariel Zeleznikow-Johnston
Monash University
<EMAIL_ADDRESS>
Naotsugu Tsuchiya
Monash University
<EMAIL_ADDRESS>
&Masafumi Oizumi
The University of Tokyo
<EMAIL_ADDRESS>

###### Abstract

Large language models (LLMs), such as the General Pre-trained Transformer
(GPT), have shown remarkable performance in various cognitive tasks. However,
it remains unclear whether these models have the ability to accurately infer
human perceptual representations. Previous research has addressed this
question by quantifying correlations between similarity response patterns from
humans and LLMs. Although it has been shown that the correlation between
humans and LLMs is reasonably high, simple correlation analysis is inadequate
to reveal the degree of detailed structural correspondence between humans and
LLMs. Here, we use an unsupervised alignment method based on Gromov-
Wasserstein optimal transport to assess the equivalence of similarity
structures between humans and LLMs in a more detailed manner. As a tractable
study, we compare the color similarity structures of humans (color-
neurotypical and color-atypical participants) and two GPT models (GPT-3.5 and
GPT-4) by examining the similarity structures of 93 colors. Our results show
that the similarity structures of color-neurotypical humans can be remarkably
well-aligned to that of GPT-4 and, to a lesser extent, to that of GPT-3.5.
These results contribute to our understanding of the ability of LLMs to
accurately infer human perception, and highlight the potential of unsupervised
alignment methods to reveal detailed structural equivalence or differences
that cannot be detected by simple correlation analysis.

##  1 Introduction

Large Language Models (LLMs) have demonstrated remarkable performance in a
variety of cognitive task [1, 2, 3]. These LLMs, based on the Transformer
architecture, use self-attention mechanisms to effectively process and
generate sequences of data [4]. Among LLMs, the General Pre-trained
Transformer (GPT) series developed by OpenAI has received considerable public
attention with the introduction of the ChatGPT conversational interface [5,
6]. Recently introduced GPT models can generate human-like responses to
prompts, and are reported to excel at tasks assessing Theory of Mind abilities
[7].

These observations raise an intriguing question: To what extent can Large
Language Models accurately infer human perceptual representations? This
question has previously been addressed by systematically comparing human
similarity judgments with those generated by GPT, across different modalities
(e.g., color, pitch, taste, etc.) in both human subjects and the GPT-3 model
[8]. In the context of comparing human and GPT’s responses, the term “stimuli"
refers to physical inputs for human psychological experiments, while for LLMs
it refers to corresponding textual inputs. The study showed that the
correlation between human and LLM similarity judgments could be as high as
$\rho=0.8$, especially when judging similarities between 14-23 colors.
Although this correlation suggests a remarkable degree of internal
representational alignment between humans and LLMs, interpreting its
quantitative and qualitative significance is challenging due to (1) the lack
of a detailed structural assessment and (2) the lack of an appropriate control
for comparing correlation values.

Regarding the first point, simple correlation does not necessarily entail a
robust structural correspondence between two similarity structures. For
example, previous studies using representational similarity analysis (RSA)
[9], a common method for assessing perceptual representational similarity via
simple correlations between similarity matrices, have shown that high
correlation values (e.g., $\rho=0.95$) may indicate only coarse category-level
correspondence, even while fine-item-level alignment is completely absent
(e.g., Fig. S2 in [10]). Thus, the mere presence of a high correlation does
not clarify whether the structures have a fine-item-level alignment or simply
a coarse-category-level correspondence.

Regarding the second point, simple correlation can only be interpreted
relative to other representational models, especially because simple
correlation does not provide an absolute measure of representational
equivalence. For example, even if correlation values in the range of 0.7-0.8
seem impressively high in the context of a color similarity judgment task,
this does not necessarily mean that such values can only be achieved by
sophisticated neural network models such as GPT. If simpler color space
models, such as RGB or LAB, can achieve similar correlation levels with human
judgments, the significance of a high human-to-GPT correlation becomes less
pronounced.

To address the limitations of simple correlation analysis, we use an
alternative computational method that analyzes the equivalence of similarity
structures. This method is based on unsupervised alignment using Gromov-
Wasserstein Optimal Transport [11, 12, 13], which has been used to evaluate
the equivalence of similarity structures for colors and visual objects [10].
It provides a mapping between two similarity structures that allows the
identification of item correspondences across individuals. Furthermore, it
allows assessment of whether two structures are aligned at the fine-item level
or merely at a coarse-categorical level. If there is perfect alignment at the
fine-item level, we can practically assume that the two similarity structures
are essentially “equivalent”. Since there are cases where alignment is minimal
despite high correlation, this unsupervised alignment method allows us to
establish representational equivalence beyond simple correlations.

Using the proposed method, we compared the similarity structures of 93 color
similarity judgments obtained from humans with those generated by LLMs. We
used GPT-3.5 and GPT-4 for our analyses. The main differences between GPT-3.5
and GPT-4 are: (1) GPT-4 is a multimodal model trained on both text and visual
inputs, whereas GPT-3.5 is a unimodal model trained only on text, and (2) the
parameter size and training dataset size of GPT-4 are substantially larger
than those of GPT-3.5. Therefore, contrasting the performance of GPT-4 with
that of GPT-3.5 could provide insight into the influence of integrating visual
inputs and the effects of model and data size.

To establish appropriate baselines for comparison with the GPT results, we
considered not only the color similarity structure of color-neurotypical
participants, but also that of color-atypical individuals (e.g., those with
color vision deficiencies). We also included the similarity structures of
simple color space models, RGB and LAB. The inclusion of RGB and LAB is
important not only to establish a baseline, but also to rule out the
possibility that GPT responses are based solely on these models. In fact, we
found that GPT explicitly uses the RGB or LAB color space model to generate
responses for certain prompts.

Using our novel computational method and appropriate controls, we found that
the similarity structures of color-neurotypical participants could be
remarkably well aligned with those of GPT-4 and, to a lesser extent, with
those of GPT-3.5. We also found that the similarity structures of color-
neurotypical participants could not be aligned with those of color space
models. The results imply a strong fine-item-level structural correspondence
between color-neurotypical human participants and the recent GPT models, but
not between color-neurotypical human participants and color-space models.

##  2 Related Work

###  2.1 Similarity Judgment Tasks Using Large Language Models

Recent studies have used large language models (LLMs) to investigate the
extent to which language contains information about human perception across
sensory modalities by examining the accuracy of LLM’s inference about human
similarity judgments [14, 15, 8]. These studies have successfully applied LLMs
such as GPT-3 to elicit similarity scores between stimuli, which then have
their underlying psychological spaces visualized through application of
multidimensional scaling (MDS) [8]. In addition, the relationship between
human similarity judgments and language has been investigated in large-scale
behavioral studies using different modalities and textual descriptors [14].

These studies have demonstrated the effectiveness of LLMs in predicting human
similarity judgments, revealing strong correlations with human data, as well
as successfully recovering well-known psychophysical structures such as the
color wheel and pitch spiral [8]. However, they have also revealed meaningful
divergences between LLM and human representations, and the degree to which
textual descriptors and models predict human similarity varies across and
within modalities [14]. By leveraging recent advances in language models and
efficient domain-general methods, these studies have demonstrated the value of
integrating machine learning and cognitive science approaches to better
understand the similarities and differences between human and machine
representations [15].

The main differences between this study and previous ones are threefold. (1)
This study uses unsupervised alignment to assess the degree of similarity
between the similarity structures of humans and LLMs. The unsupervised
alignment method allows us to examine more nuanced structural equivalences or
differences that cannot be revealed by the conventional correlation analysis
used in previous studies. (2) This study more readily facilitates analysis of
the higher-dimensional structure of colors, as our dataset contains 93 colors,
which exceeds the number of colors, ranging from 14 to 23, used in previous
studies. Since the number of colors inherently affects the potential
dimensionality of similarity structures (since the maximum dimensionality must
be less than the number of items), this difference in the number of colors
could affect the results. (3) This study considers appropriate baselines that
should be compared to interpret the significance of the results obtained by
LLMs. We included color-atypical participants and RGB and LAB color space
models as baselines, which were not included in previous studies.

Figure 1: Schematic of unsupervised alignment. (a) Unsupervised alignment of
similarity structures without external labels, based only on similarity
relations. (b) Evaluation of unsupervised alignment using external labels. (c)
Schematic of Gromov-Wasserstein optimal transport. The elements of matrices
$D$ and $D^{\prime}$ are the similarities between the items. $\Gamma$ is the
transportation matrix, where each element indicates the probability of an item
in one similarity structure corresponding to another item in the other
similarity structure. Modified from [10].

###  2.2 Supervised and Unsupervised Alignment

To evaluate the degree of similarity between similarity matrices, a supervised
approach known as Representational Similarity Analysis (RSA) has been widely
used in neuroscience to compare different similarity matrices obtained from
behavioral, neural, and neural network model data [9]. The supervised approach
(or supervised alignment method in general [16]) assumes that an element in
one similarity structure (for example, the color ’red’) corresponds to the
same element in another similarity structure. It then quantifies the degree of
similarity between the different structures, assuming a one-to-one
correspondence as defined by the item labels. Previous studies comparing the
similarity structures of humans and LLMs also use this supervised approach
[14, 15, 8]. Although the correlation assessed by the supervised approach can
provide an overall similarity between two representational structures, it has
limitations in assessing more nuanced structural differences.

In this study, we use an unsupervised alignment approach to assess a more
detailed level of structural equivalence (or difference) between the
similarity structures of humans and LLMs. In unsupervised alignment, the
correspondence between items in two similarity structures is not assumed.
Instead, the correspondences need to be discovered through an alignment
procedure (Fig. 1A), since information about external item labels is not used.
After alignment, external labels are used only to evaluate the alignment (Fig.
1B).

For unsupervised alignment, Gromov-Wasserstein optimal transport (GWOT) [11]
(Fig. 1C) has recently emerged as a promising method for comparing and
aligning similarity structures. This approach has been successfully applied in
various contexts, such as aligning word embedding spaces across languages
[17], single-cell multi-omics data [18]. In the application of comparing
psychological similarity structures, the unsupervised alignment method based
on GWOT is used to evaluate the similarity between the similarity structures
of colors and objects without assuming the correspondence between the external
labels across individuals [10]. The unsupervised optimal transport method has
revealed the structural equivalence of the similarity structures of colors and
natural objects across individuals, facilitating a broad structural
exploration of human perceptual representations. These advances in
unsupervised alignment techniques provide new ways to understand the extent to
which LLMs can accurately infer human perceptual structures.

##  3 Methods

In this study, we collected similarity judgment responses for 93 colors from
both human participants and large language models (GPT-3.5 and GPT-4), to
compare their color similarity structures.

###  3.1 Collecting responses from human participants

We used an online crowdsourcing service to recruit a large group of color-
neurotypical participants (n=426). We also included responses from color-
atypical participants (n=207) for comparison purposes. Further details of the
experimental design and procedure can be found in the supplementary material.

To determine an upper bound for the accuracy of alignment between human color-
neurotypical participants and GPTs or color space models, we divided the
color-neurotypical participants into two non-overlapping participant groups.
The accuracy of alignment between one color-neurotypical group and others was
evaluated in relation to the accuracy of alignment between the two color-
neurotypical groups.

###  3.2 Collecting responses from large language models

For each of the 93 colors, we collected similarity judgment responses for all
possible pairs, for a total of 4,278 responses. To obtain responses from
GPT-3.5 (`gpt-3.5-turbo`) and GPT-4 (`gpt-4-0314`), we used a prompt that
represented each color as a HEX code in line with a previous study [8], since
neither model can process image input.

The prompt we used is as follows:

> People described pairs of colors using their hex codes. Rate the
> dissimilarity of the pair of colors: Color 1:[HEX code] and Color 2:[HEX
> code] on a scale of 0-7 with 0-1 being Very Similar, 2-3 being Similar, 4-5
> being Different, and 6-7 being Very Different. Your rating should be any
> real number between 0 and 7. Your answer should be only the rating in the
> form of a number. No explanation is needed.

The temperature parameter, which determines the degree of randomness in GPT
responses, was set to 0.7. We ran 5 trials for each model, collecting a
complete set of similarity responses using the same prompt for each trial. We
then averaged these responses across trials to create a similarity matrix
($D$) for each model.

###  3.3 Color Space Models

To examine the possibility that LLMs rely on established color space models
when judging color similarity, we created similarity matrices for the 93
colors using two representative color space models: RGB and LAB color space
models. For the RGB color space model, we computed the Euclidean distance as a
measure of dissimilarity between each pair of colors within the 3-dimensional
RGB space. For the LAB color space model, we used the CIDE 2000 color
difference formula [19] as implemented in the Python package `colormath
(delta_e_cie2000)` to compute the difference (or "distance") between colors.

###  3.4 Comparing Color Similarity Structures

####  3.4.1 Conventional representational similarity analysis

To compare the color similarity structures between humans and GPTs in a
supervised manner using external color label information, we used the
conventional representational similarity analysis (RSA) approach. We computed
Spearman correlations between all pairs of similarity matrices (considering
only the upper-triangular elements of each matrix). It is important to note
that this analysis inherently assumes a correspondence between the same colors
across different similarity structures.

####  3.4.2 Unsupervised alignment using Gromov-Wasserstein optimal transport

To evaluate the similarity between the color similarity structures in an
unsupervised manner, i.e., without making any assumptions about the
correspondence of colors between different similarity structures, we used the
Gromov-Wasserstein optimal transport (GWOT) algorithm. GWOT is an unsupervised
alignment method that identifies the optimal transport plan between point
clouds in two domains without requiring information about the correspondence
between each item. The algorithm optimizes the Gromov-Wasserstein distance
(GWD),

${{\rm
GWD}}=\min_{\Gamma}\sum_{i,j,k,l}(D_{ij}-D_{kl}\textquoteright)^{2}\Gamma_{ik}\Gamma_{jl},$
(1)

which quantifies the correspondence between the similarity structures in the
two domains (Fig. 1C). In our problem setting, $D_{ij}$ denotes the
dissimilarity between color $i$ and $j$ in one similarity matrix $D$, while
$D^{\prime}_{kl}$ denotes the dissimilarity between color $k$ and $l$ in
another similarity matrix $D^{\prime}$. We normalized each similarity matrix
$D$ so that the values range between 0 and 1. Solving the minimization problem
of GWD yields the optimal transportation plan, represented by the matrix
$\Gamma^{*}$, which effectively aligns the color structures in the two domains
in an unsupervised manner. An element of the matrix $\Gamma_{ik}$ can be
interpreted as the “probability" that the $i$-th color in one domain
corresponds to the $k$-th color in the other domain.

Efficient optimization of GWD can be achieved by adding an entropy-
regularization term, $H(\Gamma)$, as shown in the following equation.

${{\rm
GWD}}{\epsilon}=\min_{\Gamma}\sum_{i,j,k,l}(D_{ij}-D_{kl}\textquoteright)^{2}\Gamma_{ik}\Gamma_{jl}+\epsilon
H(\Gamma).$ (2)

This addition has been proven to enhance optimization efficiency [12].

The optimization problem in Eq. 2 is non-convex, meaning that the optimal
solutions found by the algorithm are local optima, with no guarantee of
achieving the global optimum. To find good local minima, we conducted
hyperparameter tuning on $\epsilon$ and performed random initialization of the
matrix $\Gamma$. We used $\epsilon$ values ranging from $10^{-4}$ to $10^{-1}$
with logarithmic spacing (500 different $\epsilon$ values), in line with
previous works [10]. For each value of $epsilon$, we used a randomly
initialized matrix $\Gamma$. After finding the optimized $\Gamma$ for each
$epsilon$ value, we selected the solution that minimizes the GWD without the
entropy term (Eq. 1) as the optimal transportation plan.

####  3.4.3 Evaluation of unsupervised alignment

To assess the degree of similarity between the two qualia structures, we
calculated matching rates (accuracy) between two dissimilarity matrices using
color labels. An alignment is considered a match when the transportation plan
assigns the highest probability to the alignment between the same colors in
two groups (e.g. Groups 1 and 2). Given a $n\times n$ transportation plan
$\Gamma$, $\Gamma_{ij}$ represents the probability or weight of transporting
$i$-th color in Group 1 to $j$-th color in Group 2. For each color $i$ from
Group 1, if $\Gamma_{ij}$ is the highest among $j\in\\{1,...,n\\}$ and $i$-th
color in Group 1 and $j$-th color in Group 2 are the same, then we consider
the $i$-th color in Group 1 to be a match with the same color, $j$, in Group
2.

We denote the color labels in Groups 1 and 2 as $c_{1}$ and $c_{2}$
respectively. The matching rate or accuracy is calculated by comparing the
transportation plan $\Gamma$ with these labels. For each color $i$ in Group 1,
denoted by $c_{1i}$, the matching condition can be formalized as:

$\text{{Match}}(i)=\begin{cases}1,&\text{if
}\Gamma_{ij}=\max_{j\in\\{1,...,n\\}}(\Gamma_{ij})\text{ and }c_{1i}=c_{2j}\\\
0,&\text{otherwise}\end{cases}$ (3)

This function indicates whether the $i$-th color in Group 1 $c_{1i}$ matches
with the same color in Group 2 $c_{2j}$. The matching rate is then the
percentage of colors in Group 1 that match with the same color in Group 2,
which can be calculated as:

$\text{{Matching Rate}}=\frac{{\sum_{i=1}^{n}\text{{Match}}(i)}}{n}$ (4)

The matching rate defined above is top 1 matching rate. More generally, we
also define top $k$ matching rate. For each color $i$ in Group 1, we can
define a function to determine if the probability of transporting the $i$-th
color to the same color in Group 2 is within the top-$k$ probabilities:

$\text{{Top}}_{k}(i)=\begin{cases}1,&\text{if }\Gamma_{ij}\text{ is in the
top-}k\text{ for }j\in\\{1,...,n\\}\text{ and }c_{1i}=c_{2j}\\\
0,&\text{otherwise}\end{cases}$ (5)

The top-$k$ matching rate can then be calculated as:

$\text{{Top-}}k\text{{ Matching
Rate}}=\frac{{\sum_{i=1}^{n}\text{{Top}}_{k}(i)}}{n}$ (6)

A high matching rate between two color similarity matrices suggests that two
different groups have similar similarity structures of colors.

####  3.4.4 Visualization of unsupervised alignment

To visually assess the degree of similarity between the two color similarity
structures in an unsupervised manner, we obtained 3-dimensional embeddings of
93 colors. To derive the color embeddings, we applied multidimensional scaling
(MDS) to the similarity matrices, yielding 3-dimensional embeddings. We then
aligned a pair of embeddings, denoted $X$ and $Y$, using the orthogonal
rotation matrix $Q$. This matrix was obtained by solving a Procrustes-type
problem using the optimized transportation plan $\Gamma^{*}$ derived from
GWOT.

$\min_{Q}\|X-QY\Gamma^{*}\|_{F}^{2},$ (7)

where $\|\cdot\|_{F}$ is the Frobenius norm
$\|A\|_{F}=\sqrt{\sum_{i,j}a_{ij}^{2}}$. A solution to the problem can be
found through the singular value decomposition of $X(Y\Gamma^{*})^{\top}$.

Figure 2: Similarity matrices Displayed are the similarity matrices of 93
colors from two non-overlapping color-neurotypical participants groups, one
color-atypical participants group, GPT-4, GPT-3.5, and the RGB and LAB color
space models. All matrices are normalized to have values between 0 and 1.

##  4 Results

###  4.1 Correlations between similarity matrices

We created similarity matrices for two color-neurotypical participant groups,
one color-atypical participant group, GPT-4, GPT-3.5, and the RGB and LAB
color spaces, as summarized in Fig. 2. Visual inspection reveals that the
structure of the similarity matrices varies between the groups.

To quantitatively evaluate the similarity between these color similarity
matrices (Fig. 2), we calculated the Spearman correlations between them. This
analysis corresponds to conventional representational similarity analysis [9],
which implicitly assumes correspondence between the same colors across
different similarity structures. Fig. 3A summarizes the correlations between
the similarity matrices of two independent color-neurotypical groups, a color-
atypical group, GPT-4, GPT-3.5, and the RGB and LAB color spaces.

This analysis yields the following three findings:

  1. 1.

The similarity matrices of the human-neurotypical participants are closest to
that of GPT-4 ($\rho=0.77$) among the models considered (GPTs and color space
models). The correlation between the two color-neurotypical groups and GPT-4
($\rho=0.77$) is very close to the correlation between human color-
neurotypical groups 1 and 2 ($\rho=0.78$). In addition, other models such as
GPT-3.5 ($\rho=0.62$), the RGB color space model ($\rho=0.60$), and the LAB
color space model ($\rho=0.71$) also show reasonable similarity to the
similarity structures of human color neurotypical groups.

  2. 2.

GPT-3.5 shows lower correlations with the similarity matrices of color
neurotypical groups than GPT-4. In addition, the similarity matrix of GPT-3.5
shows lower correlations with the color space models than GPT-4.

  3. 3.

The color-atypical group shows relatively low correlations with the other
similarity matrices, suggesting that the similarity structure of the color-
atypical group is significantly different from that of the color-neurotypical
groups, the GPTs, and the color space models.

The scatter plots of similarity ratings of all the pairs of the similarity
matrices are available for in Fig. S1, providing a visual aid for
understanding the correlation trends between each pair of groups.

Figure 3: Evaluating the similarity of similarity structures in a supervised
and unsupervised method. (A) Conventional representational similarity analysis
based on Spearman correlations. Spearman correlations between similarity
matrices obtained from GPT-4, GPT-3.5, two color-neurotypical groups, color-
atypical group, and the RGB and LAB color spaces are shown. (B) Accuracy of
unsupervised alignment based on GWOT. Top-1 matching rates between similarity
matrices of GPT-4, GPT-3.5, two color-neurotypical groups, color-atypical
group, and the RGB and LAB color spaces are shown.

###  4.2 Accuracy of unsupervised alignment

We then evaluated the performance of unsupervised alignment using the Gromov-
Wasserstein optimal transport (GWOT) algorithm. Optimal transportation plans
were computed between all pairs of the similarity matrices shown in Fig. 2. We
have shown in Fig. 4 only the optimal transportation plans between color-
neurotypical group 1 and color-neurotypical group 2, GPT-4, and GPT-3.5 (for
other results, see Fig. S2). Using these optimal transportation plans, we
computed the top-$k$ matching rates to evaluate the unsupervised alignment. We
then aligned the embeddings of the aforementioned groups with those of color-
neurotypical group 1 for visualization.

As shown in Fig. 3B, the top-1 matching rates between GPT-4 and the color-
neurotypical groups are significantly higher ($80\%$, the average matching
rate between GPT-4 and neurotypical groups 1 and 2) than those of GPT-3.5
($31\%$) and other color space models ($3.8\%$ for RGB and $7.0\%$ for LAB ).
The matching rates between GPT-4 and the color-neurotypical groups ($80\%$)
are almost as high as that between two color neurotypical groups ($86\%$).
Moreover, when misalignments do occur, they tend to be between similar colors.

The difference between the alignment accuracy of GPT-4 and GPT-3.5 can be more
clearly seen in the more detailed optimization results (Fig. 5 and Fig.
S4,S5). For unsupervised alignment to achieve high matching accuracy, local
minima with low GWD tend to have high accuracy. If the opposite is true, i.e.,
local minima with low GWD tend to have low accuracy, the unsupervised
alignment cannot find the alignment with high matching accuracy. For the
alignment between GPT4 and human neurotypical groups, we can see that a group
of local minima with lower GWD than other local minima (red circled in Fig.
5A, B) has high accuracy. This means that it is relatively easy to find the
alignment with high accuracy between GPT4 and the human neurotypical group. In
contrast, for the alignment between GPT-3.5 and human neurotypical groups,
local minima with high accuracy and low accuracy have almost the same values
of GWD (Fig. 5C, D). This means that it is more difficult to find the high
accuracy alignment between GPT-3.5 and the human neurotypical group than GPT4.

Importantly, the unsupervised alignment method was able to reveal more nuanced
structural differences that were not observable using conventional
representational similarity analysis. For example, the Spearman correlations
between color-neurotypical groups and color space models ($\rho=0.6-0.71$) are
similar to the correlations between color-neurotypical groups and GPT-4
($\rho=0.77$). However, the matching rate of unsupervised alignment between
color-neurotypical groups and color space models is very low ($3.8\%$ for RGB
and $7.0\%$ for LAB ), while the matching rate of unsupervised alignment
between color-neurotypical groups and GPT-4 is quite high ($80\%$). This
suggests that the human color similarity structure is not adequately captured
by color space models, but is captured by GPT-4. Such nuanced structural
differences between human color similarity structure and color space models
and GPT-4 cannot be detected by conventional representational similarity
analysis, which is based on simple correlation between similarity structures.

Figure 4: Optimal transportation plans. Displayed are the optimal
transportation plans between color-neurotypical group 1 and color-neurotypical
group 2 (left), color-neurotypical group 1 and GPT-4 (middle), and color-
neurotypical group 1 and GPT-3.5. Figure 5: Results of the optimization of
$\epsilon$. Each dot in the figures represents the result of optimization with
one configuration of $\epsilon$, which was sampled 500 times with its value
ranging from 0.0001 to 0.1 in logarithmic spacing. For the sake of
visualization, we only plotted the results with $\epsilon$ smaller than 0.02,
as GWD monotonically increases for $\epsilon$ larger than 0.02. (A) The
accuracy-GWD plot for the alignment between the neurotypical group 1 and
GPT-4. Trials with lowest GWDs have highest accuracy, as emphasized by a red
dotted circle. (B) The epsilon-GWD plot for the alignment between the
neurotypical group 1 and GPT-4. (C) The accuracy-GWD plot for the alignment
between the neurotypical group 1 and GPT-3.5. Trials with high accuracy have
almost the same GWD as trials with low accuracy. (D) The epsilon-GWD plot for
the alignment between the neurotypical group 1 and GPT-3.5.

Finally, we visualized the aligned embeddings between color-neurotypical group
1 and color-neurotypical group 2, GPT-4, GPT-3.5, RGB and LAB color space
models. As depicted in Fig. 6, identical colors from two color-neurotypical
groups are located in close proximity to each other. Similarly, similar colors
from GPT-4, GPT-3.5, and color-neurotypical group 1 align closely. The colors
from the two color space models are not as well-aligned to color-neurotypical
group 1 as GPTs. These results show that GPT-4 has a color similarity
structure that is strikingly similar to that of color neurotypical
participants, allowing for unsupervised alignment. While GPT-3.5’s alignment
is not as good as GPT-4’s, it still demonstrates a remarkable degree of
alignment, outperforming the RGB and LAB color space models.

Figure 6: Aligned embeddings. Embeddings of 93 colors from color-neurotypical
group 2, GPT-4, GPT-3.5, RGB and LAB color space models were aligned to those
of color-neurotypical group 1 in an unsupervised manner. The aligned
embeddings of the two color-neurotypical groups (left), as well as the aligned
embeddings of color-neurotypical group 1, GPT-4, and GPT-3.5 (middle),
demonstrate that similar colors are closely located. The embeddings of the two
color space models do not well-align with the embeddings of similar colors
from color-neurotypical group 1 compared to GPT-4 and GPT-3.5.

##  5 Discussion

In this study, we used an unsupervised alignment method based on Gromov-
Wasserstein Optimal Transport to investigate the detailed structural
similarity between the color similarity structures of human color-neurotypical
participants and those of two large language models (LLMs) and color space
models as controls. Specifically, we found that among the models considered
(GPT-3.5, RGB, and LAB), GPT-4 had a color similarity structure that most
closely resembled that of color-neurotypical participants. Notably, GPT-4 has
a remarkably higher unsupervised alignment accuracy than GPT-3.5, almost as
high as the accuracy between human color-neurotypical groups. In contrast, we
found that the similarity structures of the color space models could not be
aligned with those of human color-neurotypical participants in an unsupervised
manner, despite reasonably high correlation coefficients ($\rho=0.60$ for RGB,
$\rho=0.71$ for LAB). These results show that our unsupervised alignment
method can reveal nuanced structural similarities and differences between the
similarity structures that are not discernible by simple correlation analysis.

While this study focused on comparing the similarity structures of colors as
an initial tractable attempt, future research could explore other sensory
modalities across a broader range of tasks (e.g., visual object similarity
judgment tasks [20, 21]). This could provide a more comprehensive
understanding of the extent to which large language models accurately capture
the similarity structures inherent in human perception. Some studies have
already begun to compare the similarity structures of LLMs and humans in other
domains based on simple correlation [14]. Our unsupervised alignment method
may provide a novel computational tool to explore more detailed structural
differences or similarities between human cognition and LLMs that cannot be
detected by simple correlation analysis.

It is important to note that we only used text inputs (i.e., color hex codes)
rather than visual images of colors, even though GPT-4 is a multimodal model
capable of accepting both image and text inputs to produce text outputs. This
is due to the unavailability of image inputs to GPT-4 at this time (August
2023). Future work could explore how the results of the present study might
change when visual inputs are used in place of text inputs, understanding the
additional information that visual inputs provide over and above text inputs.

Despite our finding that color similarity structure of GPT-4 is remarkably
similar to that of human color-neurotypical participants, it remains unclear
whether GPT-4 maintains similar internal representations of color to humans.
This is because our study only examined GPT-4’s textual output, provided
instructions to judge the similarity of color pairs, and constructed
similarity matrices based on this output. A valuable direction for future
research would be to directly extract the embeddings of colors in GPT-4 and
evaluate the similarity structures computed as distance matrices between these
embeddings. However, it is important to note that the GPT-4 embeddings were
not available at this time (August 2023).

## References

[1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT:
Pre-training of deep bidirectional transformers for language understanding. In
Proceedings of the Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies, pages 4171–4186,
2018.

[2] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric
Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha
Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial
general intelligence: Early experiments with GPT-4. arXiv preprint
arXiv:2303.12712, 2023.

[3] Marcel Binz and Eric Schulz. Using cognitive psychology to understand
GPT-3. Proc. Natl. Acad. Sci, 120(6):e2218523120, 2023.

[4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.
In Advances in Neural Information Processing System, pages 5998–6008, 2017.

[5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan,
Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan,
Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter,
Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray,
Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford,
Ilya Sutskever, and Dario Amodei. Language models are Few-Shot learners. In
Advances in Neural Information Processing Systems, pages 1877–1901, 2020.

[6] OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.

[7] Michal Kosinski. Theory of mind may have spontaneously emerged in large
language models. arXiv preprint arXiv:2302.02083, 2023.

[8] Raja Marjieh, Ilia Sucholutsky, Pol van Rijn, Nori Jacoby, and Thomas L
Griffiths. What language reveals about perception: Distilling psychophysical
knowledge from large language models. arXiv preprint arXiv:2302.01308, 2023.

[9] Nikolaus Kriegeskorte and Rogier A Kievit. Representational geometry:
integrating cognition, computation, and the brain. Trends Cogn. Sci, pages
401–412, 2013.

[10] Genji Kawakita, Ariel Zeleznikow-Johnston, Ken Takeda, Naotsugu Tsuchiya,
and Masafumi Oizumi. Is my “red” your “red”?: Unsupervised alignment of qualia
structures via optimal transport. PsyArXiv preprint,
doi:10.31234/osf.io/h3pqm, 2023.

[11] Facundo Mémoli. Gromov–Wasserstein distances and the metric approach to
object matching. Found Comput Math 11, pages 417–487, 2011.

[12] Gabriel Peyré, Marco Cuturi, and Justin Solomon. Gromov-Wasserstein
averaging of kernel and distance matrices. In International Conference on
Machine Learning, pages 2664–2672, 2016.

[13] Gabriel Peyré and Marco Cuturi. Computational optimal transport. arXiv
preprint arXiv:1803.00567, 2020.

[14] Raja Marjieh, Pol van Rijn, Ilia Sucholutsky, Theodore R Sumers, Harin
Lee, Thomas L Griffiths, and Nori Jacoby. Words are all you need? capturing
human sensory similarity with textual descriptors. arXiv preprint
arXiv:2206.04105, 2022.

[15] Raja Marjieh, Ilia Sucholutsky, Theodore R Sumers, Nori Jacoby, and
Thomas L Griffiths. Predicting human similarity judgments using large language
models. arXiv preprint arXiv:2202.04728, 2022.

[16] Alex Williams, Erin Kunz, Simon Kornblith, and Scott Linderman.
Generalized shape metrics on neural representations. Advances in Neural
Information Processing Systems, 2021.

[17] David Alvarez-Melis and Tommi S Jaakkola. Gromov-Wasserstein alignment of
word embedding spaces. In Proceedings of the 2018 Conference on Empirical
Methods in Natural Language Processing, pages 1881–1890, 2018.

[18] P Demetci, R Santorella, B Sandstede, W S Noble, and others. Gromov-
Wasserstein optimal transport to align single-cell multi-omics data. BioRxiv,
2020.

[19] Gaurav Sharma, Wencheng Wu, and Edul N Dalal. The CIEDE2000 color-
difference formula: Implementation notes, supplementary test data, and
mathematical observations. Color Res. Appl., 30(1):21–30, 2005.

[20] Martin N Hebart, Charles Y Zheng, Francisco Pereira, and Chris I Baker.
Revealing the multidi- mensional mental representations of natural objects
underlying human similarity judgements. Nature Human Behaviour,
4(11):1173–1185, 2020.

[21] M N Hebart, O Contier, L Teichmann, A H Rockter, C Y Zheng, A Kidder, A
Corriveau, M Vaziri-Pashkam, and C I Baker. THINGS-data: A multimodal
collection of large-scale datasets for investigating object representations in
brain and behavior. eLife, 12:e82580, 2023.

[22] Gunnar P Epping, Elizabeth L Fisher, Ariel Zeleznikow-Johnston, Emmanuel
Pothos, and Naotsugu Tsuchiya. A quantum geometric model of color similarity
judgements. Cognitive Science, 47: e13231, 2023.

[23] Ariel Zeleznikow-Johnston, Yasunori Aizawa, Makiko Yamada, and Naotsugu
Tsuchiya. Are color experiences the same across the visual field? J Cogn
Neurosci, 35(4):509–542, 2023.

[24] Jennifer Birch. Efficiency of the ishihara test for identifying red-green
colour deficiency. Ophthalmic and Physiological Optics, 17(5):403–408, 1997.

[25] Andrew Pouw, Rustum Karanjia, and Alfredo Sadun. A method for identifying
color vision deficiency malingering.Graefes Arch Clin Exp Ophthalmol,
255(3):613–618, 2017.

[26] Noburo Saji, Mutsumi Imai, and Michiko Asano. Acquisition of the meaning
of the word orange requires understanding of the meanings of red, pink, and
purple: Constructing a lexicon as a connected system. Cognitive Science,
44(1):e12813, 2020.

## Supplementary Figure S1

Figure S1: Scatter plots of similarity ratings. The scatter plots of
similarity ratings of all the pairs of the similarity matrices: two color-
neurotypical groups, color-atypical group, GPT-4, GPT-3.5, and the RGB and LAB
color spaces.

## Supplementary Figure S2

Figure S2: Optimal transportation plans. Optimal transportation plans
$\Gamma^{*}$ between the similarity matrices of all the pairs: two color-
neurotypical groups, color-atypical group, GPT-4, GPT-3.5, and the RGB and LAB
color spaces.

## Supplementary Figure S3

Figure S3: Top-$k$ matching rates of unsupervised alignment. Top-$k$ matching
rates between similarity matrices of two color-neurotypical groups, color-
atypical group, GPT-4, GPT-3.5, and the RGB and LAB color spaces. Here, we
show the results with $k=1,3,5$.

## Supplementary Figure S4

Figure S4: Results of the optimization of $\epsilon$. The epsilon-GWD plots
for all the pairs of alignment between two color-neurotypical groups, color-
atypical group, GPT-4, GPT-3.5, and the RGB and LAB color spaces.

## Supplementary Figure S5

Figure S5: Results of the optimization of $\epsilon$. The accuracy-GWD plots
for all the pairs of alignment between two color-neurotypical groups, color-
atypical group, GPT-4, GPT-3.5, and the RGB and LAB color spaces.

## Human color similarity judgment experiments

The data is derived from the same experiment as reported in [10].

### Ethics

Experimental procedures were approved by the Monash University Human Research
Ethics Committee (Project ID: 17674). Participants were provided
electronically with written consent forms prior to the commencement of the
experiment and provided electronic consent to participate. Participants were
compensated for their time at a rate of £5.27 for an experimental duration of
approximately 40 minutes.

### Design

##### Participants

Participants were recruited remotely through Prolific, an online participant
recruitment platform. Participants accessed the experiment and provided data
using their own personal computers. Only English native speakers were
recruited. We recruited 488 general-population (neurotypical) and 360 self-
identified color-atypical (atypical) participants prior to data cleaning.

##### Exclusion - General

Participants who failed to meet the inclusion criteria were excluded from the
analysis. Firstly, we removed participants who failed to complete the
experiment. Secondly, we excluded participants with a catch score (see below)
of $<77$%. Catch trials were included to ensure participant attention and
scattered randomly among the main trials. Lastly, the experiment was designed
as a ‘double-pass paradigm’, meaning participants performed each sequence of
main trials twice. Participants whose responses across the two passes were not
strongly correlated $<0.5$ were excluded, as low ‘double-pass’ correlation is
indicative of inattentive or neglectful responding [22, 23]. 62 out of 488
neurotypical participants were excluded, leaving 426 (87%) for the main
analysis.

##### Exclusion - Color Atypical

We collected a cohort of 360 participants who self-identified as color blind.
In addition to the general exclusion criteria, these participants were also
screened using a modified online Ishihara test. Participants viewed a set of
28 Ishihara color plates and were asked to report the number they observed. 16
of the plates were standard and used as a positive control, with participants
excluded if they correctly identified $>80\%$ of the plates (i.e. made fewer
than three errors) [24]. 12 plates consisted of standard Ishihara plates that
were red-shifted or blue-shifted so that the number should be correctly
identifiable by participants with red-green color deficiencies [25]. These
plates were used to detect participants who falsely identified as red-green
color blind, with participants excluded if they correctly identified $<80\%$.
After these additional exclusion criteria, 207 of 360 (58%) participants who
self-identified as color blind were used for the main analysis.

##### Display apparatus

Due to the nature of online experimentation, participants used their own
computer screen to perform the experiment. The stimuli for the current study
were based on the color swatches used by [26]. This 93-color set was selected
by [26] from the Practical color Co-ordinate System (PCCS). All stimuli were
presented as solid colored circles 120 pixels in diameter on a grey (#7F7F7F)
background.

##### Procedure

After recruitment through Prolific, participants were directed to the
experiment hosted on Pavlovia. The first page of the experiment was a consent
form that they could electronically sign by pressing the spacebar.
Participants were informed that the data collection process was anonymous and
that they could quit the experiment at any time. Following consent,
participants were provided written instructions on how to complete the
experiment. This was followed by 9 practice trials, seven of which were color
similarity judgments and the rest were catch trials.

Main trials for neurotypical participants proceeded as follows. First, a
fixation cross was presented in the centre of the screen for 250ms. Following
this, the two stimuli were presented as solid-colored circles for 250 ms.
Considering the centre of the screen as the midpoint, each stimulus was
presented 180° apart and at a radius of 8% of the width of their screen. The
stimuli were randomly assigned to a position within ±30° of horizontal
meridian in order to prevent retinal adaptation between trials. Lastly, the
participants were presented with a response screen and were directed to select
a specified value from 0 (most similar) to 7 (most dissimilar). After
responding, participants were asked to click on the centre of the screen to
initiate the next trial.

Atypical participants were presented with a slightly updated version of the
same task. Instead of stimuli being presented randomly within ±30° of
horizontal meridian, they were presented randomly in two out of four possible
locations equidistant from the centre of the screen and maximally spaced from
each other. Additionally, participants reported using values from $-$4 to $+$4
(with zero excluded) instead of 0 to 7. All other parameters remained the
same. (Later, we transformed these values to 0 to 7 with a uniform distance of
1).

Catch trials involved no presentation of colored stimuli patches. Instead,
participants were shown a response screen where they were prompted to click a
specific number. All other aspects of the response screen were the same.

During practice trials, participants were provided feedback on what selection
they made, consisting of both the value they selected and the text ‘Very
Similar’, ‘Similar’, ‘Different’ or ‘Very Different’ for selections of 0/1,
2/3, 4/5, 6/7 respectively for the neurotypical participants, or $-$4/$-$3,
$-$2/$-$1, 1/2, 3/4 for the atypical participants. At the cessation of these
practice trials they are asked to press the SPACE button to proceed to the
main trial set.

Following the practice trials, participants completed the main task. As with
the practice trials, catch trials were randomly inserted among the main
trials. Each participant was randomly allocated a set of color pairs out of
the total 4371 unique pairs (of 93 colors), which were presented in a random
sequence. Neurotypical participants were allocated 162 unique color pairs.
After providing a response for each color pair once, neurotypical participants
performed a repeat of the first 162 trials, identical in stimuli and sequence
(double-pass). In total, this comprised of 324 main trials and 20 randomly
interspersed catch trials. Atypical participants were allocated 81 unique
color pairs, which were also presented in a double pass manner for a total of
162 main trials and 10 catch trials.