# Semantic features of object concepts generated with GPT-3

Hannes Hansen<EMAIL_ADDRESS>
Vision and Computational Cognition Group, Max Planck Institute for Human
Cognitive and Brain Sciences, Stephanstraße 1a
04103 Leipzig Germany Martin N. Hebart<EMAIL_ADDRESS>
Vision and Computational Cognition Group, Max Planck Institute for Human
Cognitive and Brain Sciences, Stephanstraße 1a
04103 Leipzig Germany

###### Abstract

Semantic features have been playing a central role in investigating the nature
of our conceptual representations. Yet the enormous time and effort required
to empirically sample and norm features from human raters has restricted their
use to a limited set of manually curated concepts. Given recent promising
developments with transformer-based language models, here we asked whether it
was possible to use such models to automatically generate meaningful lists of
properties for arbitrary object concepts and whether these models would
produce features similar to those found in humans. To this end, we probed a
GPT-3 model to generate semantic features for 1,854 objects and compared
automatically-generated features to existing human feature norms. GPT-3
generated many more features than humans, yet showed a similar distribution in
the types of generated features. Generated feature norms rivaled human norms
in predicting similarity, relatedness, and category membership, while variance
partitioning demonstrated that these predictions were driven by similar
variance in humans and GPT-3. Together, these results highlight the potential
of large language models to capture important facets of human knowledge and
yield a new approach for automatically generating interpretable feature sets,
thus drastically expanding the potential use of semantic features in
psychological and linguistic studies.

Keywords: semantic features; conceptual knowledge; natural language
processing; GPT-3

## Introduction

A central aim in the cognitive sciences is to understand the nature of human
conceptual knowledge. This knowledge is often characterized through semantic
features, which form a set of minimal semantic descriptions of concepts and
which have been at the heart of much theorizing about categorization
(nosofsky1986attention; rosch1973internal), semantic memory Murphy (2004), and
semantic processing more generally Cree McRae (2003). For example, the concept
_car_ can, among others, be described by the features _is a vehicle_ and _has
four wheels_. The relationship of this concept to other concepts can then be
quantified by evaluating the similarities and differences to the features of
other concepts.
A common approach for attaining semantic features of concepts is to instruct
humans to list properties for a given concept, for example by asking _what are
the properties of a cow?_. The popularity of such empirically-generated
semantic features has led researchers to create semantic feature production
norms for a larger number of concepts (devereux2014centre; mcrae2005semantic),
which have been invaluable for improving our understanding of semantic
representations. At the same time, the impact of these norms has remained
constrained to the set of concepts and features that have been made publicly
available. Creating new norms requires collecting responses from hundreds of
participants and necessitates extensive manual curation by researchers,
inherently restricting the scope of such approaches. If there was a
computational model that contained the knowledge sufficient for generating
feature norms of similar quality to humans, this would drastically expand the
possible scope of research with semantic features in the study of conceptual
knowledge.
In recent years, there have been strong advances in the field of natural
language processing. So-called transformer models, such as BERT Devlin .
(2019) or GPT-3 Brown . (2020), often approach human-level performance in
diverse language understanding tasks (floridi2020gpt; wang-etal-2018-glue) and
can even be used to produce prose that can be difficult to distinguish from
human-generated text Dale (2021). While the general linguistic understanding
and reasoning ability of these models are still far from perfect Marcus
(2020), they may offer a valuable computational tool for automatically
producing semantic features for an arbitrary number of concepts Derby .
(2019), thus leveraging the statistical structure of knowledge present in
their immense training text corpora for addressing central questions in
semantic cognition research Bhatia Richie (2022).
Here we tested the degree to which recent transformer models can be used for
automatic production of semantic features and whether the produced features
mirror those found in humans. To this end, we used GPT-3 to generate a
semantic feature norm for 1,854 diverse concepts of concrete objects. We chose
concrete objects for two reasons. First, concrete objects have been used in
much research on conceptual representations and are indeed used in several
existing feature production norms, thus providing a valuable human baseline to
relate our results to. Second, similarity ratings for these 1,854 objects have
recently become available Hebart . (2020), providing a broad test case for
validating these features with existing similarity ratings.

Figure 1: Comparison of features from the CSLB norms and features generated using GPT-3, including feature examples (top) and feature label distributions (bottom). Please note that images were not part of the study and are used for illustration only. Bootstrapped confidence intervals for feature label distributions were too small to be displayed. Table 1: Descriptive statistics of human and GPT-3 generated feature norms Feature | CSLB | McRae | GPT-3 (preprocessed) | GPT-3 (preprocessed + filtered)
---|---|---|---|---
Number of concepts | 638 | 541 | 1,854 | 1,854
Total number of features | 22,667 | 7,259 | 189,126 | 124,569
Number of unique features | 5,929 | 2,524 | 63,467 | 11,683
Number of features per concept | 35.52 | 13.42 | 102.06 | 67.35
Share of unique features to all features | 26.16% | 34.77% | 33.55% | 9.37%

## Related research

Several previous studies have investigated the use of corpus-based language
models to produce sets of features mirroring human conceptual knowledge.
Static word embeddings, including _word2vec_ Mikolov . (2013) or _GloVe_
Pennington . (2014) are trained on lexical co-occurrences in large text
corpora and provide decent predictions of human similarity ratings Hill .
(2015). However, the features of these models typically lack interpretability
Subramanian . (2018). rubinstein2015well used word embeddings to directly
predict a small set of semantic features, concluding that distributed language
models may be better at capturing taxonomic than attributive features. psrl
used partial least squares regression to map word embeddings to feature norms,
with a more recent approach using a non-linear mapping based on a multilayer
perceptron Li Summers-Stay (2019). derby2019feature2vec proposed _Feature2Vec_
, a method which combines the information from word embeddings with human
feature norms by projecting the features into the word embedding space.
Despite good overall performance, these methods rely on a fixed feature
vocabulary, making it only possible to generate features for new concepts but
not completely new features.
More recently, bhatia2021transformer investigated the use of transformer
models to model human conceptual knowledge by finetuning a pretrained BERT
model Devlin . (2019) on a broad set of features and probing the model whether
the concept-feature pairs were correct. The model correctly predicted concept
features even outside of the training set with good accuracy, providing an
important step towards general purpose feature generators. However, this
method still requires probing existing feature-concept pairs, rather than
generating new features. Thus, it remains unknown to what degree it is
possible to generate arbitrary features for arbitrary concepts. Finally,
bosselut-etal-2019-comet and petroni-etal-2019-language proposed models of
commonsense knowledge based on GPT and BERT. Models are finetuned on subject-
relation-object triplets, with the task of predicting the object (e.g. _mango-
IsA-fruit_). While these models can partially generate features for concepts,
they are limited in that they require a prori specification of relations.

Figure 2: Representational similarity matrices for 6 concepts in the
categories land animals, fruits, vehicles, fruits, birds and clothing using
the CSLB and GPT-3 feature norms (top) and pairwise cosine similarities per
category (bottom).

## Methods

### Feature collection from GPT-3

We probed GPT-3 (DaVinci version) to generate semantic features for 1,854
object concepts from the THINGS database Hebart . (2019), using a text
completion task. To instruct GPT-3 on this task, we presented it with a
question about an object (e.g. _What are the properties of a chair?_), had it
generate an answer, and replaced this answer with features from the McRae
feature norm (mcrae2005semantic) which served as ground truth (e.g. _It is
furniture, it is made of wood, etc._). When continuing this process, generated
features appeared to no longer improve in quality after three question and
answer sequences, so we chose this number as a trade-off between monetary cost
and performance. Once GPT-3 was primed on the task, this was followed by an
open question for each of the 1,854 object concepts, after which the answer
was collected and both the answer and the question deleted to keep the context
static across all trials. To reduce bias induced by specific concepts and
associated features and to better mirror experimental approaches that merge
data across human participants, we collected 30 runs, each time using a
different set of example concepts. In cases where a concept occurred multiple
times with different meanings (e.g. bat as animal or bat as sports item), we
added a superordinate category to the training example in parantheses.

Figure 3: Hierarchical clustering for 45 land-animal concepts using the CSLB
and GPT-3 feature norms, highlighting similarities and differences for within
category representational structure.

### Preprocessing of features

For better comparability to humans, we preprocessed and normed generated
features. We automated preprocessing by using part of speech tagging with the
Python library _spacy_ 111https://spacy.io/ and applying a set of
preprocessing rules (see below).
The produced answers consisted of lists of features which were split at commas
in order to attain raw features. Next, raw features that were classified as
nonsensical, consisted of a single word, or were tautological (e.g. _a rose is
a rose_) were removed. In addition, features not beginning with a pronoun
(e.g. _green color_ rather than _it has green color_) or features containing
non-ASCII characters (_it has ©_) or question marks were removed. Finally,
qualifier adverbs (e.g. _usually_ or _really_) were removed, in line with
previous approaches (devereux2014centre). This affected 0.75% of all features.
Next, long features with a subordinate clause (_which, that, when, if, but_)
were shortened by removing the subordinate part. For example, a feature like
_it is a car that drives_ was shortened to _it is a car_. This affected 1.83%
of all features.
These clean and concise features were furthermore normed. Nested features,
containing multiple units of information, were extracted. For example, a
feature like _it is a big tree_ was decomposed automatically into _it is a
tree_ and _it is big_. A feature like _it is blue and green_ was decomposed
into _it is blue_ and _it is green_. Next, features containing synonyms, e.g.
_it is a car_ and _it is an automobile_ were collapsed using Wordnet synsets
in the Python library _nltk_ 222https://www.nltk.org/ by choosing the more
frequent synset. However, to avoid merging non-synonymous words, two words
were only considered as synonyms if their most frequent synset was the same.
4.3% of all features were replaced.

### Filtering

The generated feature norm partially consisted of overly sparse features, with
many features that were unique to individual concepts. However, a smaller set
of features is desirable both for reasons of better interpretability and for
reduced computational cost. Therefore, we removed features that occurred very
infrequently within each concept. This also made the feature norm more
comparable in size to human generated norms. To identify a cutoff, we plotted
the number of unique features after removal of infrequent features and chose
the elbow point at k=4. Importantly, while this step strongly reduced the
number of unique features (see Table 1), it did not affect performance in
validating the norm.

## Results

### Comparison with human feature norms

As a first analysis, we compared the GPT-3 feature norms with human feature
norms McRae McRae . (2005) and CSLB Devereux . (2014), using descriptive
statistics. We did not use the Buchanan norm Buchanan . (2019) as it was
composed only of associative features. As seen in Table 1, the preprocessed
GPT-3 feature norm was computed for a larger number of concepts, leading to a
larger number of total features and more unique features. Without filtering, a
similar share of unique features can be seen as in the McRae norm, indicating
a similar level of redundancy of features across concepts as found in humans.
After excluding rare features, only around 9% of all features remained unique.
Of note, GPT-3 produced a much larger number of features per concept,
indicating that human feature production may be limited to a sparser set of
features than those found in GPT-3.

### Label distribution

Figure 1 (top) shows several example concepts with their five most frequent
features. The results indicate many similarities and no obvious errors in the
types of labels assigned by GPT-3. To directly compare the distribution of
semantic features between humans and GPT-3, samples of features from the CSLB
feature norms and GPT-3 were labeled into the categories taxonomic, visual
perceptual, other perceptual, conceptual, functional and encyclopedic. We use
a slightly different naming scheme of feature types to McRae and CSLB to allow
for more fine-grained differences between conceptual, functional, and
encyclopedic features. To estimate the distribution of features in all three
norms, we randomly sampled 500 features from the 317 concepts shared between
CSLB, McRae, and GPT-3 and 500 features from concepts outside of the
intersection. The corresponding feature labels were assigned manually.
The results in Figure 1 (bottom) show that humans and GPT-3 mostly rely on
encyclopedic features and less on perceptual features. GPT-3 contained a
larger number of functional features as compared to the CSLB norm. However,
the differences in the distribution to the McRae norm, which GPT-3 had been
trained on, were less prominent, indicating that differences of GPT-3 to CSLB
may be driven more by differences in populations for the creation of human
norms or norming processes, rather than intrinsic differences in
representations. Overall, this analysis yielded no obvious differences to
human norms in the types of semantic features that were generated.

Table 2: Similarity and relatedness prediction Pearson correlation | McRae | CSLB | GPT-3
---|---|---|---
MEN (n=55 word pairs) | 0.77 | 0.77 | 0.79
Simlex-999 (n=26 word pairs) | 0.60 | 0.63 | 0.62
THINGS (n=317 concepts) | 0.56 | 0.63 | 0.62

### Prediction of category structure based on feature similarity

Next we tested the degree to which GPT-3 generated feature norms produced
reasonable category structure and how they compared to existing norms. For
better comparability, several analyses were conducted in a similar fashion to
those found for the creation of the CSLB norm Devereux . (2014). For a fair
comparison, we restricted our analyses to the 317 concepts shared between the
McRae, CSLB and GPT-3 feature norms. Similarities were based on the cosine
similarity of the concepts across feature vectors composed of the production
frequency per feature, for human norms across participants and for GPT-3
across the 30 runs for which it had been primed with different examples.
First, we inspected the superordinate category structure. Mirroring the
results of Devereux . (2014), we based these analyses on six exemplars of the
six categories _land animals_ , _clothing_ , _birds_ , _vehicles_ , _fruits_ ,
and _musical instruments_. If the produced features prove to be useful for
predicting high-level category structure, we would expect consistently high
within-category similarity and low between-category similarity. A
visualization of the similarity structure of the CSLB norm with the GPT-3
generated norm is shown in Figure 2 (top). As is evident from the figure, both
norms show excellent category structure, clearly distinguishing high-level
categories from each other. To quantify similarities and differences between
norms, we expanded the set of concepts available within each superordinate
category to the 317 concepts and computed the mean within-category similarity
minus the mean between-category similarity of each concept. The results are
shown in Figure 2 (bottom). Overall, GPT-3 showed comparable or clearer
category structure than human norms, with the only exception of _clothing_.
Together, these results demonstrate that high-level category structure can be
extracted from GPT-3 generated feature norms, with similar performance to
humans.
Beyond between-category effects, we explored whether the within-category
similarity structure was meaningful in GPT-3 generated norms. To this end, we
performed hierarchical clustering of the 45 land animal concepts. Overall, we
expected high similarities between all animals but also more fine-grained
differences. The results comparing CSLB with GPT-3 norms are shown in Figure
3. Overall, the similarities between animals were higher in the GPT-3 norm
than CSLB. A clear clustering of highly similar animals is found in both
matrices (e.g. _lamb_ and _sheep_). However, the subordinate category
structure was slightly different between GPT-3 than CSLB. For example, farm
animals clustered in GPT-3 but were more distributed in CSLB, while dangerous
animals clustered more closely in CSLB. In sum, while the overall category
structure was similar, there were fine-grained differences in the
representations derived from semantic features through GPT-3 as compared to
the CSLB norm.

Figure 4: Variance partitioning of human similarity judgements, demonstrating
strong overlap between human generated and GPT-3 generated norms in predicting
human similarity judgments.

### Prediction of similarity and relatedness ratings

Predictions of similarity and relatedness tasks are often seen as a gold
standard for evaluating the relationship between semantic features and our
conceptual representations. To identify the degree to which GPT-3 generated
norms could be used to predict similarity and relatedness ratings, we used the
overlap between McRae, CSLB, and GPT-3 generated norms with two existing
datasets commonly used as natural language processing benchmarks: The MEN
dataset Bruni . (2014) was used for word relatedness and the Simlex-999
dataset Hill . (2015) for word similarity. We intended to include other common
benchmark datasets but did not find sufficient overlap in concepts. Beyond
these datasets, we used human similarity scores from THINGS Hebart . (2020),
which were available for all included objects. Similarities were compared by
using the Pearson correlation of the flattened lower triangular part of the
similarity matrices.
The overlap in the number of concepts as well as the performance for McRae,
CSLB, and GPT-3 generated norms are found in Table 2. While CSLB performed
better than McRae for the prediction of THINGS, across all three datasets, the
performance of CSLB and GPT-3 was comparable. This indicates that the
representations derived from GPT-3 generated norms were of similar quality as
those found in human norms.
What is left open by these correlations is whether the predictions of
similarity were based on similar information in McRae, CSLB, and GPT-3, or
whether GPT-3 had access to other information not reported by humans. To
address this question, we carried out variance partitioning, identifying the
unique and shared variance components explained in the THINGS similarity
dataset. The results of this analysis are shown in Figure 4. Overall, there
was strong overlap in the explained variance between McRae and GPT-3 as well
as CSLB and GPT-3, with GPT-3 subsuming much of the variance explained by
McRae, while explaining very similar portions of variance than CSLB. Overall,
this result demonstrates that, indeed, GPT-3 is relying on similar information
as CSLB for predicting human similarity.

## Discussion

Overall, the results demonstrate that GPT-3 can be used for automatically
generating semantic feature norms for a large number of concrete objects.
Frequently produced features were meaningful and comparable in distribution to
those found in humans. Further, the results showed that superordinate
categories were well identified by the resulting features and that within-
category structure was reasonable. Predictions of similarity and relatedness
were comparable to humans and relied on similar information. Overall, this
demonstrates that GPT-3 can serve as an effective automatic feature generator,
opening up an efficient and rapid approach to generate large numbers of
features for diverse concepts.
Of note, there were some differences to humans. Overall, GPT-3 produced a much
larger number of features, yet reducing this set through filtering led to very
similar results in the prediction of similarity ratings. Further, the fine-
grained similarity structure was slightly different to CSLB. Future work is
needed to investigate these differences and the degree to which they are
related to differences between representations in humans and GPT-3, or whether
they were driven by the norming process itself.
GPT-3 was primed using only three examples, which highlights the simplicity of
our proposed approach but which may also introduce bias. Rather than
reflecting the knowledge available to the model, it may in fact mimic the
process of feature generation produced by humans in the three examples it was
provided. Other, more effective priming procedures may be discovered in the
future that constitute less bias and are better at revealing the knowledge
available in such models. Future work may also investigate the influence of
model complexity on feature generation.
Nevertheless, the fact that GPT-3 and humans relied on similar information for
predicting similarity ratings is relevant in its own respect, indicating that
GPT-3 may indeed contain a lot of information useful for modeling important
aspects of conceptual knowledge Bhatia Richie (2022). As a consequence, it may
be possible to use GPT-3 to generate other types of norms, for example ratings
of animacy, graspability, or size Grand . (2022). We did not test whether
GPT-3 was able to produce meaningful features for more abstract concepts or
verbs, which is an important avenue for future research. While better
computational models for producing semantic features may exist, our work
demonstrates that it is already possible to create semantic features for
concrete concepts with human level performance in predicting similarity
ratings using GPT-3.

## Conclusions

Here, we introduced a new approach for automatically generating semantic
features for diverse concepts using the recent transformer-based model
architecture GPT-3. Our results demonstrate that recent large language models
are indeed able to accurately reflect important aspects of human conceptual
knowledge. The approach opens the door to automatic feature generation for
arbitrary concepts, thus widening the potential scope of semantic features for
research in psychology and linguistics. To promote the general use of this
method and results, the GPT-3 generated raw data as well as the final feature
norm of the 1,854 object, including the code to probe GPT-3 and to preprocess
and filter raw features, are made publicly
available333https://github.com/ViCCo-Group/semantic_features_gpt_3.

## References

  * Bhatia Richie (2022) bhatia2021transformerBhatia, S. Richie, R.  2022\.  Transformer Networks of Human Conceptual Knowledge Transformer networks of human conceptual knowledge.  Psychological Review.
  * Bosselut . (2019) bosselut-etal-2019-cometBosselut, A., Rashkin, H., Sap, M., Malaviya, C., Celikyilmaz, A. Choi, Y.  201907\.  COMET: Commonsense Transformers for Automatic Knowledge Graph Construction COMET: Commonsense transformers for automatic knowledge graph construction.  Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics Proceedings of the 57th annual meeting of the association for computational linguistics ( 4762–4779).  Florence, ItalyAssociation for Computational Linguistics.  https://aclanthology.org/P19-1470  10.18653/v1/P19-1470
  * Brown . (2020) brown2020languageBrown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, JD., Dhariwal, P.Amodei, D.  2020\.  Language Models are Few-Shot Learners Language models are few-shot learners.  H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan H. Lin (), Advances in Neural Information Processing Systems Advances in neural information processing systems ( 33, 1877–1901).  Curran Associates, Inc.
  * Bruni . (2014) bruni2014multimodalBruni, E., Tran, NK. Baroni, M.  2014\.  Multimodal distributional semantics Multimodal distributional semantics.  Journal of artificial intelligence research491–47.
  * Buchanan . (2019) buchanan2019englishBuchanan, EM., Valentine, KD. Maxwell, NP.  2019\.  English semantic feature production norms: An extended database of 4436 concepts English semantic feature production norms: An extended database of 4436 concepts.  Behavior Research Methods5141849–1863.
  * Cree McRae (2003) cree2003analyzingCree, GS. McRae, K.  2003\.  Analyzing the factors underlying the structure and computation of the meaning of chipmunk, cherry, chisel, cheese, and cello (and many other such concrete nouns). Analyzing the factors underlying the structure and computation of the meaning of chipmunk, cherry, chisel, cheese, and cello (and many other such concrete nouns).  Journal of experimental psychology: general1322163.
  * Dale (2021) dale2021gptDale, R.  2021\.  GPT-3: What’s it good for? Gpt-3: What’s it good for?  Natural Language Engineering271113–118.
  * Derby . (2019) derby2019feature2vecDerby, S., Miller, P. Devereux, B.  201911\.  Feature2Vec: Distributional semantic modelling of human property knowledge Feature2Vec: Distributional semantic modelling of human property knowledge.  Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp) ( 5853–5859).  Hong Kong, ChinaAssociation for Computational Linguistics.  https://aclanthology.org/D19-1595  10.18653/v1/D19-1595
  * Devereux . (2014) devereux2014centreDevereux, BJ., Tyler, LK., Geertzen, J. Randall, B.  2014\.  The Centre for Speech, Language and the Brain (CSLB) concept property norms The centre for speech, language and the brain (cslb) concept property norms.  Behavior research methods4641119–1127.
  * Devlin . (2019) devlin2018bertDevlin, J., Chang, MW., Lee, K. Toutanova, K.  201906\.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding BERT: Pre-training of deep bidirectional transformers for language understanding.  Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers) ( 4171–4186).  Minneapolis, MinnesotaAssociation for Computational Linguistics.  https://aclanthology.org/N19-1423  10.18653/v1/N19-1423
  * Floridi Chiriatti (2020) floridi2020gptFloridi, L. Chiriatti, M.  2020\.  GPT-3: Its nature, scope, limits, and consequences Gpt-3: Its nature, scope, limits, and consequences.  Minds and Machines304681–694.
  * Fǎgǎrǎşan . (2015) psrlFǎgǎrǎşan, L., Vecchi, EM. Clark, S.  2015\.  From distributional semantics to feature norms: grounding semantic models in human perceptual data From distributional semantics to feature norms: grounding semantic models in human perceptual data.  Proceedings of the 11th International Conference on Computational Semantics Proceedings of the 11th international conference on computational semantics ( 52–57).
  * Grand . (2022) grand2022semanticGrand, G., Blank, IA., Pereira, F. Fedorenko, E.  2022\.  Semantic projection recovers rich human knowledge of multiple object features from word embeddings Semantic projection recovers rich human knowledge of multiple object features from word embeddings.  Nature Human Behaviour1–13.
  * Hebart . (2019) hebart2019thingsHebart, MN., Dickter, AH., Kidder, A., Kwok, WY., Corriveau, A., Van Wicklin, C. Baker, CI.  2019\.  THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images Things: A database of 1,854 object concepts and more than 26,000 naturalistic object images.  PloS one1410e0223792.
  * Hebart . (2020) hebart2020revealingHebart, MN., Zheng, CY., Pereira, F. Baker, CI.  2020\.  Revealing the multidimensional mental representations of natural objects underlying human similarity judgements Revealing the multidimensional mental representations of natural objects underlying human similarity judgements.  Nature human behaviour4111173–1185.
  * Hill . (2015) hill2015simlexHill, F., Reichart, R. Korhonen, A.  2015\.  Simlex-999: Evaluating semantic models with (genuine) similarity estimation Simlex-999: Evaluating semantic models with (genuine) similarity estimation.  Computational Linguistics414665–695.
  * Li Summers-Stay (2019) li2019mappingLi, D. Summers-Stay, D.  2019\.  Mapping distributional semantics to property norms with deep neural networks Mapping distributional semantics to property norms with deep neural networks.  Big Data and Cognitive Computing3230.
  * Marcus (2020) marcus2020nextMarcus, G.  2020\.  The next decade in ai: four steps towards robust artificial intelligence The next decade in ai: four steps towards robust artificial intelligence.  arXiv preprint arXiv:2002.06177.
  * McRae . (2005) mcrae2005semanticMcRae, K., Cree, GS., Seidenberg, MS. McNorgan, C.  2005\.  Semantic feature production norms for a large set of living and nonliving things Semantic feature production norms for a large set of living and nonliving things.  Behavior research methods374547–559.
  * Mikolov . (2013) mikolov2013efficientMikolov, T., Chen, K., Corrado, G. Dean, J.  2013\.  Efficient estimation of word representations in vector space Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781.
  * Murphy (2004) murphy2004bigMurphy, G.  2004\.  The big book of concepts The big book of concepts.  MIT press.
  * Nosofsky (1986) nosofsky1986attentionNosofsky, RM.  1986\.  Attention, similarity, and the identification–categorization relationship. Attention, similarity, and the identification–categorization relationship.  Journal of experimental psychology: General115139.
  * Pennington . (2014) pennington2014glovePennington, J., Socher, R. Manning, CD.  2014\.  Glove: Global vectors for word representation Glove: Global vectors for word representation.  Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) Proceedings of the 2014 conference on empirical methods in natural language processing (emnlp) ( 1532–1543).
  * Petroni . (2019) petroni-etal-2019-languagePetroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y. Miller, A.  201911\.  Language Models as Knowledge Bases? Language models as knowledge bases?  Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp) ( 2463–2473).  Hong Kong, ChinaAssociation for Computational Linguistics.  https://aclanthology.org/D19-1250  10.18653/v1/D19-1250
  * Rosch (1973) rosch1973internalRosch, EH.  1973\.  On the internal structure of perceptual and semantic categories On the internal structure of perceptual and semantic categories.  Cognitive development and acquisition of language Cognitive development and acquisition of language ( 111–144).  Elsevier.
  * Rubinstein . (2015) rubinstein2015wellRubinstein, D., Levi, E., Schwartz, R. Rappoport, A.  2015\.  How well do distributional models capture different types of semantic knowledge? How well do distributional models capture different types of semantic knowledge?  Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 2: Short papers) ( 726–730).
  * Subramanian . (2018) subramanian2018spineSubramanian, A., Pruthi, D., Jhamtani, H., Berg-Kirkpatrick, T. Hovy, E.  2018\.  Spine: Sparse interpretable neural embeddings Spine: Sparse interpretable neural embeddings.  Thirty-Second AAAI Conference on Artificial Intelligence. Thirty-second aaai conference on artificial intelligence.
  * Wang . (2018) wang-etal-2018-glueWang, A., Singh, A., Michael, J., Hill, F., Levy, O. Bowman, S.  201811\.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding GLUE: A multi-task benchmark and analysis platform for natural language understanding.  Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP ( 353–355).  Brussels, BelgiumAssociation for Computational Linguistics.  https://aclanthology.org/W18-5446  10.18653/v1/W18-5446