# Categorization in the Wild: Generalizing Cognitive Models to Naturalistic
Data across Languages

Lea Frermann  Mirella Lapata  Amazon CoreAI  University of Edinburgh

###### Abstract

Categories such as animal or furniture are acquired at an early age and play
an important role in processing, organizing, and communicating world
knowledge. Categories exist across cultures: they allow to efficiently
represent the complexity of the world, and members of a community strongly
agree on their nature, revealing a shared mental representation. Models of
category learning and representation, however, are typically tested on data
from small-scale experiments involving small sets of concepts with
artificially restricted features; and experiments predominantly involve
participants of selected cultural and socio-economical groups (very often
involving western native speakers of English such as U.S. college students) .
This work investigates whether models of categorization generalize (a) to rich
and noisy data approximating the environment humans live in; and (b) across
languages and cultures. We present a Bayesian cognitive model designed to
jointly learn categories and their structured representation from natural
language text which allows us to (a) evaluate performance on a large scale,
and (b) apply our model to a diverse set of languages. We show that meaningful
categories comprising hundreds of concepts and richly structured featural
representations emerge across languages. Our work illustrates the potential of
recent advances in computational modeling and large scale naturalistic
datasets for cognitive science research.

###### keywords:

Categorization , Bayesian Modeling , Cognitive Modeling , Natural Language
Processing

##  1 Introduction

Categories such as animal or furniture are fundamental cognitive building
blocks allowing humans to efficiently represent and communicate the complex
world around them. Concepts (e.g.,  _dog_ , _table_) are grouped into
categories based on shared properties pertaining, for example, to their
behavior, appearance, or function.111We denote (superordinate level)
categories (such as animal or vehicle) in small caps; (basic level) _concepts_
(such as _cat_ or _car_) in italics, and feature types (such as function or
behavior) in true type. Individual features (such as {eats, sits, barks}) are
represented as {lists}. Categorization underlies other cognitive functions
such as perception [1, 2] or language [3, 4], and there is evidence that
categories are not only shaped by the world they represent, but also by the
language through which they are communicated [5, 6]. Although mental
categories exist across communities and cultures, their exact manifestations
differ [7, 8, 9, 10]. For example, American English speakers prefer taxonomic
categorizations (e.g., _mouse_ ,_squirrel_) while Chinese speakers tend to
prefer to categorize objects relationally (e.g.,  _tree_ , _squirrel_ ; [8]).

Given their prevalent function in human cognition, the acquisition and
representation of categories has attracted considerable attention in cognitive
science, and numerous theories have emerged [11, 12, 13, 14]. Empirical
studies of category acquisition and representation, have been predominantly
based on small-scale laboratory experiments. In a typical experiment, human
subjects are presented with small sets of often artificial concepts, such as
binary strings [15] or colored shapes [16], with strictly controlled features
[17, 18, 19]. Hypotheses and principles of human categorization are
established based on the processes and characteristics of the categorizations
produced by the participants. The distribution of subjects participating in
such studies is often skewed towards members of cultural and socioeconomic
groups which are prevalent in the environment where the research is conducted,
and typically consists to a large proportion of western, educated, wealthy and
English-speaking participants often sampled from the even more specific
population of college students. The demographic and socioeconomic bias has
been long recognized, and the question of how this bias might impact
conclusions about human cognition in general [20] and category learning
specifically is under active debate [10]. Although laboratory studies are
invaluable for understanding categorization phenomena in a controlled
environment, they are also expensive and time-consuming to conduct, and
consequently problematic to scale.

Figure 1: Illustration of model stimuli for five languages. Each stimulus
contains a mention of a concept (e.g.,  _cat_ or _apple_) in its local
linguistic context. Concepts are grouped into categories (e.g., animal or
fruit) based on the similarity of the contexts they occur in.

In this work, we scale the investigation of category learning and
representation along two axes: (1) the complexity of the learning environment,
and consequently the richness of learnable concept and category
representations, and (2) the diversity of languages and cultures considered in
evaluation. We present a novel knowledge-lean, cognitively motivated Bayesian
model which learns categories and their structured features jointly from large
natural language text corpora in five diverse languages: Arabic, Chinese,
English, French, and German. We approximate the learning environment using
large corpora of natural language text. Language has been shown to redundantly
encode much of the non-linguistic information in the natural environment [21],
and to influence the emergence of categories [5, 6]. Besides text corpora can
cover arbitrarily semantically complex domains, and are available across
languages, providing an ideal test environment for studying categorization at
scale.

Figure 1 illustrates example input to our model, and Figure 2 shows example
categories and associated features as induced by our model from the English
Wikipedia. Following prior work [22, 23], we create language-specific sets of
stimuli, each consisting of a mention of target concept222Throughout this
article we user the term concept to refer to Rosch’s [12] basic-level
categories, and the term category to refer to superordinate-level categories.
(Superordinate-level) categories are inferred based on the features observed
with observations of (basic-level) concepts. (e.g.,  _apple_), within its
local linguistic context (e.g., {contains, seeds}; cf., Figure 1). We consider
each stimulus an observation of the concept, i.e., the word referring to the
concept is an instance of the concept itself, and its context words are a
representation of its features. Our model infers categories as groups of
concepts occurring with similar features; and it infers feature types as
groups of features which co-occur with each other. The output of our model
(cf., Figure 2) are categories as clusters of concepts, each associated with a
set of feature types, i.e., thematically coherent groups of features. We train
a separate model on each of our target languages, each time presenting the
model with input stimuli from the relevant language.

Figure 2: Examples of categories (light red) and feature types (light blue)
inferred by BCF from the English Wikipedia. Connecting lines indicate
associations learned between the category and the respective feature type. For
example, category fruit (k3) is associated with feature types trade (g4)
nutrition (g5), and processed food (g6).

Computational models in general, and Bayesian models in particular, allow to
investigate hypotheses about cognitive phenomena by systematically modifying
the learning mechanism or available input while observing the learning
outcome. Bayesian models have been applied to a variety of cognitive phenomena
[24, 25, 26], and category acquisition is no exception. Following from
Anderson’s [15, 27, 28] seminal work, a number of models have been developed,
and tested in their ability to reproduce human behavior in laboratory settings
by exposing the models to small sets of controlled inputs with restricted
features. In this work we draw on the full potential of computational modeling
by exposing our models to (a) more complex data reflecting the diversity of
contexts in which concepts can be observed; and (b) a input data in different
languages, shedding light on the applicability of computational cognitive
models beyond the prevalent English test language.

Categorization tasks in a laboratory environment typically involve stimuli
with a small set of features which are relevant to the categorization target,
eliminating the need to detect features, and discriminate them in their
relevance. In the real world, however, concepts are observed in contexts and a
substantial part of acquiring categorical knowledge involves learning which
features are useful to discriminate among concepts. In fact, research has
shown that humans learn features jointly with categories [29, 30] and that
these features are themselves structured so as to represent the diversity and
complexity of the properties exhibited in the world [31, 32, 33]. Our novel
model of category learning presented in this article, jointly learns
categories and their structured features from large sets of informationally
rich data.

Our work exemplifies the opportunities that arise from computational models
and large data sets for investigating the mechanisms with which conceptual
representations emerge, as well as the representations themselves in a broader
context. We simulate the acquisition of categories comprising hundreds of
concepts by approximating the learning environment with natural language text.
Language has been shown to redundantly encode much of the non-linguistic
information in the natural environment [21], as well as human-like biases
[34], and to influence the emergence of categories [5, 6]. Text corpora are a
prime example of naturally occurring large-scale data sets [35, 36, 37]. In
analogy to real-world situations, they encapsulate rich, diverse, and
potentially noisy, information. The wide availability of corpora allows us to
train and evaluate cognitive models on data from diverse languages and
cultures. We test our model on corpora from five languages, derived from the
online encyclopedia Wikipedia in Arabic, Chinese, French, English, and German.
Wikipedia is a valuable resource for our study because it (a) discusses
concepts and their properties explicitly and can thus serve as a proxy for the
environment speakers of a language are exposed to; and (b) allows us to
construct corpora which are highly comparable in their content across
languages, controlling for effects of genre or style.

We present a series of evaluations investigating the quality of the induced
categories and features. Leveraging a reference comprising hundreds of
concepts and more than 30 categories, we demonstrate that our model learns
meaningful categories in all five target languages. We furthermore show,
through crowd-sourced evaluations involving native speakers of each target
language, that the induced feature types are (a) each thematically coherent
and interpretable; and (b) are associated with categories in comprehensible
ways. We discuss language-specific idiosyncrasies emerging from the induced
representations.

In the remainder of this article, we first review related literature, before
we present a cognitively motivated model for learning categories and their
structured representations from large natural language corpora. We then
evaluate the quality of the emerging representations, as well as the
generalizability of our model across languages. Note that the primary goal of
this work is not to characterize differences in categories and features
arising from different languages (even though this would be an interesting
avenue for future work). Rather, we aim to demonstrate the utility of large-
scale naturalistic datasets for cognitive modeling, and to verify mechanisms
of categorization known from laboratory studies at scale and across
communities.

##  2 Related Work

In this work we leverage large-scale computational simulations to advance our
understanding of categories and features across languages and cultures. Our
research touches on the representation of categories, concepts, and their
features; the mechanisms with which these are learnt; and the use of
computational models and large-scale naturalistic data sets to investigate
these questions.

###  2.1 Feature Representations of Concepts and Categories

Even though much empirical research glosses over this observation, there is
strong evidence that human conceptual representations are structured (see [38]
for a recent critique and overview of cognitive studies of categorization).
Categories mentally represent the complex structure of the environment. They
allow to make inferences about concepts or categories that go beyond their
perceived similarities capturing abstract and potentially complex properties
(for example, the nutritional value of food items, or the emotional benefits
of pets). Much research on human categorization is based on laboratory
experiments where subjects are presented with artificial stimuli represented
by a restricted set of task-relevant features. Observations of natural
concepts, however, are often noisy or incomplete so that a notion of
systematic relations among features might be more important here than under
artificial conditions in the lab [39].

The existence of structured features has received support through behavioral
results from a variety of categorization related tasks, such as typicality
rating [39] or category-based inductive inference [40, 33]. Experimental
evidence suggests that categories which on the surface do not seem to contain
a coherent set of members (e.g., the category pets) are represented by an
underlying set of abstract features which explain the coherence of the
category (e.g., {keeps_company, lives_in_the_house}). Varying the types of
available features (e.g., providing functional information in addition to
objects’ appearance) leads to different categorization behavior both in adults
[40] and children [41, 42], and different feature types vary in their
predictive value across categories. For example, 2-4-year old children
categorize food items based on their color, however, toys are classified based
on their shape [43].

The structured nature of category features manifests itself in feature norms.
Feature norms are verbalized lists of properties that humans associate with a
particular concept [32]. Features collected in norming studies naturally fall
into different types such as behavior, appearance or function. This suggests
that structure also emerges from verbalized representations of concepts and
features such as mentions in natural language corpora, used as stimuli in this
work. McRae et al. [32] collected a large set of feature norms for more than
500 concepts in a multi-year study, and classified these using a variety of
theoretically motivated schemata, including the feature type classification
scheme developed in [44] and [45]. Their work puts forward the hypothesis that
humans perform a “mental simulation” when describing a concept, scanning the
mental image they create as well as situations associated with that image, and
then verbalize it when producing features.

The model we present in this article aims to capture the evidence summarized
above, and represent categories as structured sets of features with varying
degrees of association. Category-specific features are structured into types
which relate to a particular kind of property of a category (e.g., the
behavior of animals). We also capture the observation that features are
defining for _different_ categories to a varying degree [46, 47] in terms of
category-feature type associations (e.g., the feature function is highly
defining for (or associated with) the category artifact, but not for the
category animal).

###  2.2 Joint Learning of Categories and their Features

Although the majority of models of categorization assume a fixed set of
features underlying the category acquisition and categorization process, there
is increasing evidence that “[…] a significant part of learning a category
involves learning the features entering its representations.” [30, p. 681].
Experimental evidence suggests that not only do features underly the
categorization process but features themselves are susceptible to change over
time and can be modified by the categories which emerge. Evidence ranges from
changing featural perception as a result of expert education (e.g., wine
tasters or doctors learning to interpret X-ray images) to neurological
evidence revealing enhanced neural activity in experts when presented with
pictures of their area of expertise (see [48] for an overview).

The influence of category learning on the perception and use of features has
been studied extensively using visual stimuli of varying degrees of
naturalness and familiarity. Experiments with drawings of 2-dimensional line
segments [49] show that participants who were exposed to categorization
training prior to a feature identification task identified the presence of
category-defining features faster than participants without prior training.
When asked to categorize pictures of (systematically manipulated) human faces,
participants showed higher sensitivity to features relevant for the
categorization task [29, 50].

To the best of our knowledge, we present the first computational investigation
in the joint emergence of categories and features from large sets naturalistic
input data.

###  2.3 Computational Models of Category and Feature Induction

The tasks of category formation and feature learning have been considered
largely independently in the context of computational cognitive modeling.
Bayesian categorization models pioneered by Anderson [15] and recently re-
formalized by Sanborn et al. [51] aim to replicate human behavior in small
scale category acquisition studies, where a fixed set of simple (e.g., binary)
features is assumed. Informative features are pre-defined and available to the
model. The BayesCat model [52] is similar in spirit, but was applied to large-
scale corpora, while investigating incremental learning in the context of
child category acquisition (see also [22] for a non-Bayesian approach).
BayesCat associates sets of features (context words) with categories as a by-
product of the learning process, however these feature sets are independent
across categories and are not optimized during learning.

A variety of cognitively motivated Bayesian models have been proposed for the
acquisition of complex domain knowledge. Shafto et al. [53] present a joint
model of category and feature acquisition in the context of cross-
categorization, i.e., the phenomenon that concepts are simultaneously
organized into several categorizations and the particular category (and
features) that are relevant depend on the context (e.g., concepts of the
category food can be organized based on their nutritional or perceptual
properties). However, while [53] present their model with category-specific
data sets tailored towards their learning objective, we are interested in
acquiring categories and structured associated features jointly from
thematically unconstrained corpora of natural text.

Another line of work [54, 55] models the joint learning of relevant features
and domain-specific feature type biases in children. They focus on the
acquisition of domain-specific representational structures (such as
hierarchies or clusters) and discuss results in the context of word learning.
In contrast to our work, their model assumes a priori established categories
(such as food and animals), and learns from task-specific data representations
in the form of objects described by a limited set of relevant features (even
though a weighting of those features is learnt). Perfors et al. [56] present a
Bayesian model which simultaneously learns categories (i.e., groupings of
concepts based on shared features) and learns to learn categories (i.e.,
abstract knowledge about kinds of featural regularities that characterize a
category). They compare their model predictions against behavioral data from
adult participants, which limits the scope of their experiments to small data
sets.

The ability to automatically extract feature-like information for concepts
from text would facilitate the laborious process of feature norming, i.e.,
eliciting features associated with concepts verbally from human annotators
[32], and improve the coverage of concepts and their features. A few
approaches to feature learning from textual corpora exist, and they have
primarily focused on emulating or complementing norming studies by
automatically extracting norm-like properties from corpora (e.g., _elephant_
has-trunk, _scissors_ used-for-cutting). Steyvers [57] uses a flavor of topic
models to augment data sets of human-produced feature norms. While vanilla
topic models [58] represent documents as sets of corpus-induced topics, [57]
additionally use topics derived from feature norms. The learnt topics yield
useful extensions of the original feature norms, with properties that were
previously not covered, suggesting that corpora are an appropriate resource
for augmenting feature norms of concepts.

Another line of research concerns text-based feature extraction. A common
theme in this line of work is the use of pre-defined syntactic patterns [59],
or manually created rules specifying possible connection paths of concepts to
features in dependency trees [60, 61]. While the set of syntactic patterns
pre-defines the relation types the system can capture, the latter approach can
extract features which are a priori unlimited in their relation to the target
concept. Once extracted, the features are typically weighted using statistical
measures of association in order to filter out noisy instances. Similar to our
own work, the motivation underlying these models is large-scale unsupervised
feature extraction from text. These systems are not cognitively motivated
acquisition models, however, due to (a) the assumption of involved prior
knowledge (such as syntactic parses or manually defined patterns), and (b) the
two stage extraction-and-filtering process which they adopt. Humans arguably
do not first learn a large set of potential features for concepts, before they
infer their relevance. The systems discussed above learn features for
individual concepts rather than categories.

To our knowledge, we propose the first Bayesian model that jointly learns
categories and their features from large sets of naturalistic input data. Our
model is knowledge-lean, it learns from raw text in a single process, without
relying on parsing resources, manually crafted rule patterns, or post-
processing steps; it is more plausible from a cognitive point of view, and
language agnostic. We present simulations with the same model on several
languages varying in word order, morphology, and phonology.

##  3 Category and Feature Learning at Scale

Computational models as simulators of cognitive processes have been used
successfully to shed light on a wide variety of phenomena [25], including
language acquisition [28], generalization, and reasoning [62]. Bayesian models
in particular are amenable towards this goal, because they allow the modeler
to formalize hypotheses rigorously through sets of random variables and their
relations. They use the principled rules of Bayesian probability to select
“good” models which explain the observed data well. We present a Bayesian
model to investigate cognitive processes of categorization, in correspondence
to Marr’s [63] computational level of analysis, i.e., abstracting away from
the algorithms and biological substrates in which these processes are
situated. Starting from Anderson’s [15] pioneering work on rational models of
categorization, a variety of models, both Bayesian [51, 53, 23] and non-
Bayesian [19, 22] have been proposed. Our work advances prior research by
investigating for the first time joint category and feature learning from
noisy stimuli, across diverse languages.

We present BCF, a cognitively motivated Bayesian model for learning Categories
and structured Features from large sets of concept mentions and their
linguistic contexts (see Figure 1). Our model induces categories (as groups of
concepts), feature types which are shared across categories (as groups of
features or context words), and category-feature type associations. Figure 2
shows example output of BCF as learnt from the English Wikipedia, and Figure 4
shows example categories and features learnt for five additional languages.

BCF is a statistical Bayesian model. Given a large set of stimuli, it learns
meaningful categories and features from a countably infinite set of all
possible categorizations and representations. The probability (or
‘meaningfulness’) of any hypothetical categorization and representation $h$
under the stimuli data $d$ can be evaluated using Bayes’ rule:

$p(h|d)\propto p(d|h)p(h),$ (1)

where $p(h)$ is the prior probability of $h$ under the specified model and its
assumptions; and $p(d|h)$ is the likelihood to observe data $d$ given that
hypothesis $h$ holds.

###  3.1 The BCF Model

BCF learns from an input corpus which consists of stimuli covering
$\mathcal{L}$ target concepts, where the set of target concepts is specified
by the modeler a priori. The model induces a categorization of these target
concepts into $K$ categories; as well as a characterization of each category
in terms of $G$ different feature types pertaining to different relevant
properties. The number of categories, $K$, and the number of feature types,
$G$, are model parameters.

symbol | explanation
---|---
$d\in\\{1..D\\}$ | stimulus | (e.g., “This dog likes to catch balls.”)
$c\in\\{1..C\\}$ | concept mention in stimulus (e.g., “dog”)
$i\in\\{1..I\\}$ | context word positions in stimulus
$\ell\in\\{1..\mathcal{L}\\}$ | concept types | (e.g.,  _cat, dog, chair, table_)
$f\in\\{1..V\\}$ | features | (e.g., {runs, barks, eats, red, made_of_wood})
$k\in\\{1..K\\}$ | categories | (e.g., animal, furniture)
$g\in\\{1..G\\}$ | feature types | (e.g., behavior, appearance)
$\theta$ |  $K$-dimensional parameter vector of category distribution
$\\{\mu_{k}\\}_{k=1}^{K}$ |  $G$-dimensional parameter vectors of feature type distributions
$\\{\phi_{g}\\}_{g=1}^{G}$ |  $V$-dimensional parameter vectors of word distributions
Table 1: Notational overview of the BCF model (the category and feature type
labels are provided for illustration; BCF is an unsupervised clustering model
which induces unlabeled categories and feature types).

(a) Generative story of BCF.

Generate category distribution, $\hskip 8.61108pt\theta\sim Dir(\alpha)$

for concept type $\ell=1..\mathcal{L}$ do

Generate category, $\hskip 8.61108ptk^{\ell}\sim Mult(\theta)$

for category $k=1..K$ do

Generate feature type distribution, $\hskip 8.61108pt\mu_{k}\sim Dir(\beta)$

for feature type $g=1..G$ do

Generate feature distribution, $\hskip 8.61108pt\phi_{g}\sim Dir(\gamma)$

for stimulus $d=1..D$ do

Observe concept $c^{d}$ and retrieve category $k^{c^{d}}$

Generate a feature type, $\hskip 8.61108ptg^{d}\sim Mult(\mu_{k^{c^{d}}})$

for feature position $i=1..I$ do

Generate a feature $\hskip 8.61108ptf_{d,i}\sim Mult(\phi_{g^{d}})$

(b) Plate diagram of BCF.

gcf$k^{\ell}$$\theta$$\alpha$$\mu_{k}$$\beta$$\phi$$\gamma$$I$$D$$\mathcal{L}$$K$$G$

Figure 3: Top (a): The generative story of the BCF model. Observations $f$ and
latent labels $k$ and $g$ are drawn from Multinomial distributions ($Mult$).
Parameters for the multinomial distributions are drawn from Dirichlet
distributions ($Dir$). Bottom (b): The plate diagram of the BCF model. Shaded
nodes indicate observed variables, clear nodes denote latent variables, and
dashed nodes indicate constant hyperparameters.

A notational overview is provided in Table 1. The generative story of our
model is displayed in Figure 3(a), and Figure 3(b) shows the plate diagram
representation of BCF. The generative story proceeds as follows. We assume a
global multinomial distribution over categories $Mult(\theta)$. Its parameter
vector $\theta$ is drawn from a symmetric Dirichlet distribution with
hyperparameter $\alpha$. For each target concept $\ell=[1...\mathcal{L}]$, we
draw a category $k^{\ell}$ from $Mult(\theta)$. For each category $k$, we draw
an independent set of multinomial parameters over feature types, $\mu_{k}$,
from a symmetric Dirichlet distribution with hyperparameter $\beta$,
reflecting the relative relevance of each feature type towards this category.
Finally, we associate each feature type with representative words from our
feature vocabulary $f\in 1...V$, by drawing a multinomial distribution over
features, $Mult(\phi_{g})$, from a symmetric Dirichlet distribution with
hyperparameter $\gamma$. From this set of global representations, we can
generate sets of stimuli $d=[1...D]$ as follows: we first retrieve the
category $k^{c^{d}}$ of an observed concept $c^{d}$; we then generate a
feature type $g^{d}$ from the category’s feature type distribution
$Mult(\mu_{k^{c^{d}}})$; and finally, for each context position $i=[1...I]$ we
generate feature $f_{d,i}$ from the feature type-specific feature distribution
$Mult(\phi_{g^{d}})$.

According to the generative story outlined above, the joint probability of the
model over latent categories, latent feature types, model parameters, and data
factorizes as:

$\displaystyle p$
$\displaystyle(g,f,\mu,\phi,\theta,k|c,\alpha,\beta,\gamma)=$ (2)
$\displaystyle
p(\theta|\alpha)\prod_{\ell}p(k^{\ell}|\theta)\prod_{k}p(\mu_{k}|\beta)\prod_{g}p(\phi_{g}|\gamma)\prod_{d}p(g^{d}|\mu_{k^{c^{d}}})\prod_{i}p(f^{d,i}|\phi_{g^{d}}).$

Since we use conjugate priors throughout, we can integrate out the model
parameters analytically, and perform inference only over the latent variables,
namely the category and feature type labels associated with the stimuli.

In sum, our model takes as input a text corpus of concept mentions in their
local context, and infers a concept categorization, a global set of feature
types, as well as a distribution over feature types per category. After
integrating out model parameters where possible, we infer two sets of latent
variables:

  * (1)

feature type-assignments to each stimulus $\\{g\\}^{D}$,

  * (2)

category-assignments to each concept type $\\{k\\}^{\mathcal{L}}$.

The next section introduces a learning algorithm in the form of a Gibbs
sampler for approximate estimation of these parameters.

###  3.2 Approximate Inference for BCF

1:Input: model with randomly initialized parameters.

2:Output: posterior estimate of $\boldsymbol{\theta},\boldsymbol{\phi}$, and
$\boldsymbol{\mu}$.

3:repeat

4: for stimulus $d$ do $\triangleright$ Sample stimulus-feature type
assignments

5: \- decrement stimulus $d$-related counts

6: \- Sample $g^{d}\sim
p(g_{k^{c^{d}}}^{d}=i|\mathbf{g}_{k^{c^{d}}}^{-d},\mathbf{f}^{-},k^{c^{d}},\beta,\gamma)$
Equation (5)

7: \- update stimulus $d$-related counts

8: for concept $c$ do $\triangleright$ Sample concept-category assignments

9: \- retrieve category $k^{c}$

10: \- decrement concept $c$-related counts

11: \- Sample $k^{c}\sim
p(k^{\ell}=j|\mathbf{g}_{k^{\ell}},\mathbf{k^{-}},\alpha,\beta)$ Equation (7)

12: \- update concept $c$-related counts

13:until convergence

Algorithm 1 The Gibbs sampling algorithm for the BCF model.

Exact inference in the BCF model is intractable, so we turn to approximate
posterior inference to discover the distribution over value assignments to
latent variables given the observed data. In this section we introduce a Gibbs
sampling algorithm [64, 65] which is a Markov chain Monte Carlo algorithm
which iteratively computes values of individual random variables in the model,
based on the current value assignments of all other random variables. The
sampling procedure for BCF is summarized in Algorithm 1. The Gibbs sampler
repeatedly iterates over the training corpus and resamples values of the
latent variables. One Gibbs iteration for our model consists of two blocks:

#### Resampling stimulus-feature type assignments

In the first block we iterate over input stimuli $d$, and resample each
stimulus-feature type assignment $g^{d}$ from its full conditional posterior
distribution over feature types conditioned on (a) the values assigned to all
other latent variables unrelated to the current variable of interest, i.e, all
features except those in stimulus $d$, $\big{(}\mathbf{f}^{-}\big{)}$, and all
stimulus-feature type assignments except the one to stimulus $d$,
$\big{(}\mathbf{g}_{k^{c^{d}}}^{-d}\big{)}$; (b) the category currently
assigned to $d$’s target concept $c$, $\big{(}k^{c^{d}}\big{)}$; and (c) the
relevant hyperparameters $\big{(}\beta,\gamma\big{)}$:

$\displaystyle p(g_{k^{c^{d}}}^{d}=i$ $\displaystyle|$
$\displaystyle\mathbf{g}_{k^{c^{d}}}^{-d},\ \mathbf{f}^{-},\ k^{c^{d}}=j,\
\beta,\ \gamma)$ (3) $\displaystyle=$ $\displaystyle\ \
p(g_{k^{c^{d}}}^{d}=i|\mathbf{g}_{k^{c^{d}}}^{-d},k^{c^{d}}=j,\beta)\times\hskip
8.61108ptp(f^{d}|\mathbf{f}^{-},g_{k^{c^{d}}}^{d}=i,\gamma)$ (4)
$\displaystyle\propto$ $\displaystyle\ \
\frac{(n_{i}^{j}+\beta)}{(\sum_{i}n^{j}_{i}+\beta)}\times\hskip
8.61108pt\frac{\prod_{v}\prod_{a=1}^{f_{v}}(n_{v}^{i}+\gamma+a)}{\prod_{a=1}^{f_{*}}(\sum_{v}n_{v}^{i}+\gamma+a)}.$
(5)

The factorization of the posterior distribution in (4) follows from the model
structure as described above and shown in the plate diagram in Figure 3(b).
The posterior distribution factorizes into the probability of a particular
feature type $i$ and the probability of the observed features in the stimulus
given that feature type. Because of the Dirichlet-Multinomial conjugacy in our
model, these two distributions can be straightforwardly computed using only
the counts of current value-assignments to all variables in the model except
the ones currently resampled (equation (5)): the probability of a hypothetical
feature type $i$ is proportional to the number of times it has assigned
previously to stimuli with observed category $j$, $n^{j}_{i}$, smoothed by the
Dirichlet parameter $\beta$. Similarly, the probability of the observed
features of stimulus $d$ under hypothetical feature type $i$ is proportional
to the number of times each individual feature $v$ in $d$ has been observed
under feature type $i$, $n^{i}_{v}$ (smoothed by the Dirichlet parameter
$\gamma$). In the second term in (5), $f_{v}$ refers to the count of any
particular feature $v$ in stimulus $d$, and $f_{*}$ refers to the number of
features in $d$ (irrespective of their value).

We compute the (unnormalized) probabilities of individual hypothetical feature
types $i$ as explained above. These values are then normalized and a new
feature type is sampled from the resulting distribution.

#### Resampling concept-category assignments

The second block of our Gibbs sampler performs a sweep over all concept types
$\ell\in\\{1...\mathcal{L}\\}$, and resamples each concept type $\ell$’s
category assignment $k^{\ell}$. Similarly to the process described above, the
new category assignment of concept $\ell$ is resampled from its full
conditional distribution over categories conditioned on (a) all concept-
category assignments except for $k^{\ell}$, $\big{(}\mathbf{k^{-}}\big{)}$;
(b) the feature type assignments relevant to concept $\ell$,
$\big{(}\mathbf{g}_{k^{\ell}}^{-}\big{)}$; and (c) all relevant
hyperparameters $\big{(}\alpha,\beta\big{)}$:

$\displaystyle p(k^{\ell}=j|\mathbf{g}_{k^{\ell}}^{-},\ \mathbf{k^{-}},\
\alpha,\ \beta)$ $\displaystyle=$ $\displaystyle
p(k^{\ell}=j|\mathbf{k}^{-},\alpha)\times\hskip
8.61108ptp(\mathbf{g}_{k^{\ell}}|\mathbf{g}^{-}_{k^{\ell}},k^{\ell}=j,\beta)$
(6) $\displaystyle\propto$ $\displaystyle(n^{j}+\alpha)\times\hskip
8.61108pt\frac{\prod_{g}\prod_{a=1}^{f^{\ell}_{g}}(n^{j}_{g}+\beta+a)}{\prod_{a=1}^{f^{\ell}_{*}}(\sum_{g}n^{j}_{g}+\beta+a)}.$
(7)

Based on the independence assumptions in our model, this probability
factorizes into the prior probability of hypothetical category $j$ and the
probability of feature types observed with concept $\ell$ under the
hypothetical category $j$ (equation (6)). As above, these probabilities can be
computed purely based on counts of variable-assignments in the current sampler
state (equation (7)). In the second term of (7), $f^{\ell}_{g}$ refers to the
number of times feature type $g$ was assigned to a stimulus containing concept
type $\ell$, and $f^{\ell}_{*}$ to the number of stimuli containing $\ell$
(irrespective of the assigned feature type).

Using the procedure described above we compute an (unnormalized) probability
for each hypothetical category, normalize the probabilities and resample
concept $\ell$’s category $k^{\ell}$ from the resulting distribution.

##  4 Experimental Setup

Can we simulate category acquisition from large amounts of textual data using
cognitively motivated computational models, and infer meaningful
representations across languages?

We approach this question by applying BCF to data sets in five languages:
English, French, German, Arabic, and Chinese. We train five models in total,
one per language, each time using stimuli from the respective language alone.
We evaluate induced categories by comparison against a human-created reference
categorization; and collect judgments on the coherence of learnt feature
types, and their relevance to their associated categories from large crowds of
native speakers.

Is the structure and architecture of BCF appropriate and necessary for
category and structured feature learning? We answer this question by comparing
BCF against a variety of related models. First, we report a random baseline
which assigns concepts to categories at random. Secondly, we compare against a
model entirely based on word co-occurrence. Unlike BCF, the co-occurrence
model cannot learn categories and features jointly, and has no notion of
feature structure. It uses $k$-means clustering [66] to group concepts into
categories, and, subsequently, group features into feature types for each
category (see Section 4.2). Finally, we compare BCF against BayesCat, a
cognitively motivated Bayesian model of category acquisition [23]. Like BCF,
it draws inspiration from topic modeling, however, BayesCat does not learn
categories and features jointly, and does not acquire structured feature
representations.

In the following we describe our data set, as well as the set of models we
compare BCF against. Next, we present a series of simulations evaluating the
quality of the induced categories, their features, and their relevance to the
associated categories.

###  4.1 Experimental Stimuli

| en | ar | ch | fr | ge
---|---|---|---|---|---
concepts | 491 | 394 | 450 | 484 | 482
features | 5,898 | 5,870 | 6,516 | 6,416 | 6,981
stimuli | 418,755 | 86,908 | 147,386 | 258,499 | 233,175
Table 2: Datasets obtained from language-specific Wikipedia dumps for Arabic
(ar), Chinese (ch), English, (en), French (fr), and German (ge). Number of
stimuli (concept mentions in linguistic context), concepts, and features
(context word types) are shown.

Our simulations focused on 491 basic-level concepts of living and non-living
things, taken from two previous studies of concept representation [32, 67],
for which we learn (a) a categorization and (b) structured feature
representations. Human-created gold standard categorizations of the concepts
into 33 categories are available [67, 68]. Since the original studies were
conducted in English, we collected translations of the target concepts and
their categories into Arabic, Chinese, French, and German, created by native
speakers of the target language. The final number of concepts differs across
languages, because some English concepts do not exist (or do not have the same
translation) in the target language. Concept sets and categorizations for all
languages were made available as part of this submission.

We created language specific sets of input stimuli (as illustrated in Figure
1). For each target language we created a corpus as follows: We used a subset
of articles from the Linguatools Wikipedia
dump333http://linguatools.org/tools/corpora/wikipedia-monolingual-corpora/; we
tokenized, POS-tagged and lemmatized the corpus, and removed stopwords using
language-specific lists. From this data set we derived a set of input stimuli
as mentions of a concept from the reference set of concepts in sentence
context (cf., Figure 1). In order to obtain balanced data sets, we
automatically filtered words of low importance to a concept from contexts,
using the term-frequency-inverse-document-frequency (tf-idf) metric. After
filtering, we only kept stimuli with $3\leq n\leq 20$ context words and at
most 1,000 stimuli per target concept. Table 2 summarizes the statistics of
the resulting data sets. The number of stimuli varies across languages as a
function of the number of target concepts, and the size of the respective
Wikipedia corpus.

###  4.2 Comparison Models

We compared BCF against various models explained below. All experiments follow
the same experimental protocol, i.e., we train separate instances of the same
model on each language.

#### Strudel

Following a pattern-based approach, Strudel automatically extracts features
for concepts from text collections. It takes as input a part of speech-tagged
corpus, a set of target concepts and a set of 15 hand-crafted rules. Rules
encode general, but quite sophisticated linguistic patterns which plausibly
connect nouns to descriptive attributes (e.g., extract an adjective as a
property of a target concept mention if the adjective follows the mention, and
the set of tokens in between contain some form of the verb ‘to be’. [69]).
Strudel obtains a large set of concept-feature pairs by scanning the context
of every occurrence of a target concept in the input corpus, and extracting
context words that are linked to the target concept by one of the rules. Each
concept-feature pair is subsequently weighted with a log-likelihood ratio
expressing the pair’s strength of association. Baroni et al. [59] show that
the learnt representations can be used as a basis for various tasks such as
typicality rating, categorization, or clustering of features into types. We
obtained Strudel representations from the same Wikipedia corpora used for
extracting the input stimuli for BCF and BayesCat. Note that Strudel, unlike
the two Bayesian models, is not a cognitively motivated acquisition model, but
a system optimized with the aim of obtaining the best possible features from
data.

#### Co-occurrence Baseline

Strudel relies on manually constructed linguistic patterns, and is
consequently not directly applicable across languages. We report a baseline
which is constructed to resemble Strudel, but does not rely on linguistic
features. It allows us to assess whether pure co-occurrence counts provide a
strong enough learning signal for category and feature induction across
languages. This model represents each concept $c$ as a vector with dimensions
corresponding to its co-occurrence counts with features $f$ (i.e., context
words), capped by a minimum number of required observations, approximating the
concept-feature association:

$assoc(c,f)=\mathcal{N}(c,f).$ (8)

We obtained categories by clustering concepts based on their vector
representations using $k$-means clustering [70]. Based on these categories, we
obtained feature types by (1) collecting all features associated with at least
half the concepts in the category; and (2) clustering these features into
feature types using $k$-means clustering.

#### BayesCat

Similar to BCF, BayesCat is a knowledge-lean acquisition model which can be
straightforwardly applied to input from different languages. It induces
categories $z$ which are represented through a distribution over target
concepts $c$, $p(c|z)$, and a distribution over features $f$ (i.e., individual
context words), $p(f|z)$. BayesCat, like BCF, is a Bayesian model and its
parameters are inferred using approximate MCMC inference, in the form of a
Gibbs sampler. Unlike BCF, however, BayesCat does not induce structured
feature representations, and comparing it to BCF allows us to evaluate the
advantage of joint category and feature learning. BayesCat induces categories
represented through unstructured bags-of-features. As such, the model
structure of BayesCat is closely related to topic models such as Latent
Dirichlet Allocation (LDA; [58]). Comparing our proposed model against
BayesCat allows us to shed light on the benefit of more sophisticated model
structure which allows to learn features jointly with categories, compared to
the information that can be captured in vanilla topic models. For our human
evaluation in Section 7 we construct feature types from BayesCat features as
follows. First we represent each feature $f$ as its probability under each
category $p(z|f)$. Based on this representation, we again employ $k$-means
clustering to group features into $G$ global feature types $g$. Finally, we
compute category-featuretype associations as:

$p(g|z)=\sum_{f\in g}p(f|z),$ (9)

where $p(f|z)$ is learnt by BayesCat.

While BCF induces a hard assignment of concepts to categories, BayesCat learns
a soft categorization. Soft assignments can be converted into hard assignments
by assigning each concept $c$ to its most probable category $z$,

$z(c)=\max_{z}p(c|z)p(z|c).$ (10)

#### Model Parameters

Across all simulations we trained BCF to induce $K=40$ categories and $G=50$
feature types which are shared across categories. We ran the Gibbs sampler for
1,000 iterations, and report the final most likely representation. We trained
BayesCat on the same input stimuli as BCF, with the following parameters: the
number of categories was set to $K=40$, and the hyperparameters to
$\alpha=0.7,\beta=0.1$, and $\gamma=0.1$. From the learnt representations, we
induced $G=50$ global feature types as described above. Again results are
reported as averages over 10 runs of 1,000 iterations of the Gibbs sampler.
The co-occurrence model induces $K=40$ categories, and, subsequently, $G=5$
feature types for each category.

##  5 Experiment 1: Category Quality

In this simulation, We evaluate the extent to which model-induced categories
resemble the human created reference categorization. We report results on
cluster quality for BCF, BayesCat, and the frequency baseline for our five
target languages. For English, we additionally report results for Strudel. We
also lower-bound the performance of all models with a random clustering
baseline (random), which randomly assigns all concepts to $K=40$ categories.

###  5.1 Method

The output clusters of an unsupervised learner do not have a natural
interpretation. Cluster evaluation in this case involves mapping the induced
clusters to a gold standard and measuring to what extent the two clusterings
(induced and gold) agree [71]. Purity ($pu$) measures the extent to which each
induced category contains concepts that share the same gold category. Let
$G_{j}$ denote the set of concepts belonging to the $j$-th gold category and
$C_{i}$ the set of concepts belonging to the $i$-th cluster. Purity is
calculated as the member overlap between an induced category and its mapped
gold category. The scores are aggregated across all induced categories $i$,
and normalized by the total number of category members $N$:

$\text{pu}=\frac{1}{N}\sum_{i}\max\limits_{j}|C_{i}\cap G_{j}|$ (11)

Inversely, collocation ($co$) measures the extent to which all members of a
gold category are present in an induced category. For each gold category we
determine the induced category with the highest concept overlap and then
compute the number of shared concepts. Overlap scores are aggregated over all
gold categories $j$, and normalized by the total number of category members
$N$:

$\text{co}=\frac{1}{N}\sum_{j}\max\limits_{i}|C_{i}\cap G_{j}|$ (12)

Finally, the harmonic mean of purity and collocation can be used to report a
single measure of clustering quality. If $\beta$ is greater than 1, purity is
weighted more strongly in the calculation, if $\beta$ is less than 1,
collocation is weighted more strongly:

$\text{F}_{\beta}=\frac{(1+\beta)\cdot pu\cdot co}{(\beta\cdot pu)+co}$ (13)

We additionally report results in terms of V-Measure (VM, [72]) which is an
information-theoretic measure. VM is analogous to F-measure, in that it is
defined as the weighted harmonic mean of two values, homogeneity (VH, the
precision analogue) and completeness (VC, the recall analogue):

VH $\displaystyle=$ $\displaystyle 1-\frac{H(G|C)}{H(G)}$ (14) VC
$\displaystyle=$ $\displaystyle 1-\frac{H(C|G)}{H(C)}$ (15) VM
$\displaystyle=$ $\displaystyle 1-\frac{(1+\beta)\cdot VH\cdot VC}{(\beta\cdot
VH)+VC}$ (16)

where $H(\cdot)$ is the entropy function; $H(C|G)$ denotes the conditional
entropy of the induced class $C$ given the gold standard class $G$ and
quantifies the amount of additional information contained in $C$ with respect
to $G$. The various entropy values involve the estimation of the joint
probability of classes $C$ and $G$:

$\hat{p}(C,G)=\frac{\mu(C\cap G)}{N}$ (17)

###  5.2 Results

| model | PCF1 | V-Measure |
---|---|---|---|---
|  | $pu$ | $co$ | $F_{1}$ | VH | VC | VM |
English | BCF | 0.552 | 0.432 | 0.484 | 0.652 | 0.598 | 0.623 |
BayesCat | 0.551 | 0.429 | 0.482 | 0.646 | 0.577 | 0.609 |
Strudel | 0.572 | 0.442 | 0.499 | 0.662 | 0.590 | 0.624 |
2-8 | co-occ | 0.550 | 0.394 | 0.459 | 0.626 | 0.559 | 0.591 |
| random | 0.193 | 0.135 | 0.159 | 0.317 | 0.282 | 0.298 |
German | BCF | 0.454 | 0.400 | 0.425 | 0.545 | 0.523 | 0.534 |
BayesCat | 0.458 | 0.378 | 0.414 | 0.563 | 0.513 | 0.537 |
2-8 | co-occ | 0.338 | 0.387 | 0.361 | 0.408 | 0.435 | 0.421 |
| random | 0.194 | 0.134 | 0.158 | 0.316 | 0.280 | 0.297 |
French | BCF | 0.534 | 0.441 | 0.483 | 0.632 | 0.585 | 0.608 |
BayesCat | 0.507 | 0.415 | 0.457 | 0.609 | 0.558 | 0.582 |
2-8 | co-occ | 0.459 | 0.365 | 0.407 | 0.544 | 0.509 | 0.526 |
| random | 0.197 | 0.134 | 0.160 | 0.319 | 0.283 | 0.300 |
Chinese | BCF | 0.441 | 0.349 | 0.389 | 0.510 | 0.497 | 0.503 |
BayesCat | 0.430 | 0.320 | 0.367 | 0.532 | 0.493 | 0.512 |
2-8 | co-occ | 0.367 | 0.327 | 0.345 | 0.408 | 0.422 | 0.415 |
| random | 0.208 | 0.135 | 0.164 | 0.325 | 0.291 | 0.307 |
Arabic | BCF | 0.408 | 0.327 | 0.363 | 0.444 | 0.446 | 0.445 |
BayesCat | 0.394 | 0.298 | 0.339 | 0.491 | 0.462 | 0.476 |
2-8 | co-occ | 0.261 | 0.308 | 0.283 | 0.312 | 0.344 | 0.327 |
| random | 0.214 | 0.125 | 0.158 | 0.329 | 0.296 | 0.312 |
Table 3: Quality of induced categories for BCF (this work), BayesCat, a co-
occurrence baseline, Strudel (for English only), and a random baseline.
Results are reported for English, German, French, Chinese and Arabic.

Table 3 displays the results for all five languages. BCF learns categories
which most closely resemble the human gold standard, and both BCF and the co-
occurrence model clearly outperform the random baseline. The Bayesian models,
BCF and BayesCat, outperform the co-occurrence model across metrics and
languages. For English, Strudel slightly outperforms BCF. Note, however, that,
BCF learns the categories from data, whereas for Strudel we construct the
categories post-hoc after a highly informed feature extraction process
(relying on syntactic patterns). It is therefore not surprising that Strudel
performs well, and it is encouraging to see that BCF learns categories of
comparable quality.

We observe a slight drop in performance for languages other than English which
is likely due to smaller stimuli sets (see Table 2). BCF, nevertheless,
achieves purity scores of 0.4 or higher for all languages, meaning that on
average at least 40% of the members of a gold standard category are clustered
together by BCF (purity rises to 58% for English). This indicates that
meaningful categories emerge throughout. Qualitative model output shown in
Figures 2 (English) and 4 (all languages) corroborates this result. The
categories shown are intuitively meaningful; in particular vegetable and
clothing (Figure 4) are interpretable, and thematically consistent across
languages.

A few interesting idiosyncrasies emerge from our cross-lingual experimental
setup, and the ambiguities inherent in language. For example, the English
concepts _tongue_ and _bookcase_ were translated into French words _langue_
and _bibliothèque_ , respectively. The French BCF model induced a category
consisting of only these two concepts with highly associated feature types
{story, author, publish, work, novel} and {meaning, language, Latin, German,
form}. Although this category does not exist in the gold standard, it is
arguably a plausible inference. Another example concerns the concept _barrel_
, which in the English BCF output, is grouped together with concepts _cannon,
bayonet, bomb_ and features like {kill, fire, attack}. In French, on the other
hand, it is grouped with _stove, oven_ and the features {oil, production, ton,
gas}.

We showed that BCF learns meaningful categories across languages which are
quantitatively better than those inferred by a simpler co-occurrence model.
Although generally consistent, categories are sometimes influenced by
characteristics of the respective training and test language. While the
literature confirms an influence of language on categorization [5, 6], this
effect is undoubtedly amplified through our experimental framework.

(a) Category clothing

(b) Category vegetables

Figure 4: Categories clothing (a) and vegetables (b) (light red), and their
five most highly associated feature types (light blue) for English (en),
German (de), French (fr), Arabic (ar), and Chinese (ch). Model output of
languages other than English was translated into English by native speakers.

##  6 Experiment 2: Feature Quality

| model | pr@1 | pr@10 | pr@20 | avg rank
---|---|---|---|---|---
English | BCF | 0.07 | 0.34 | 0.48 | 61.9
BayesCat | 0.05 | 0.31 | 0.44 | 63.6
Strudel | 0.04 | 0.31 | 0.45 | 76.1
2-6 | co-occ baseline | 0.013 | 0.210 | 0.370 | 97.4
| random baseline | 0.002 | 0.020 | 0.040 | –
German | BCF | 0.07 | 0.31 | 0.42 | 90.5
BayesCat | 0.07 | 0.32 | 0.41 | 66.3
2-6 | co-occ baseline | 0.007 | 0.110 | 0.137 | 177.6
| random baseline | 0.002 | 0.021 | 0.041 | –
French | BCF | 0.07 | 0.31 | 0.44 | 73.2
BayesCat | 0.04 | 0.28 | 0.45 | 64.2
2-6 | co-occ baseline | 0.017 | 0.153 | 0.227 | 136.6
| random baseline | 0.002 | 0.021 | 0.041 | –
Chinese | BCF | 0.09 | 0.37 | 0.49 | 66.0
BayesCat | 0.09 | 0.38 | 0.53 | 41.7
2-6 | co-occ baseline | 0.033 | 0.157 | 0.187 | 139.1
| random baseline | 0.002 | 0.022 | 0.044 | –
Arabic | BCF | 0.13 | 0.49 | 0.61 | 54.9
BayesCat | 0.14 | 0.54 | 0.65 | 27.5
2-6 | co-occ baseline | 0.012 | 0.110 | 0.139 | 154.8
| random baseline | 0.003 | 0.025 | 0.050 | –
Table 4: Model performance on the concept prediction task in terms of
precision at rank 1, 10, 20, and average rank assigned. We compare BCF (this
work), BayesCat, a co-occurrence baseline, Strudel (for English only), and a
random baseline. Results are reported for English, German, French, Chinese and
Arabic.

We next investigate the quality of the features our model learns. We do this
by letting the model predict the right concept solely from a set of features.
If the model has acquired informative features, they will be predictive of the
unknown concept. Specifically, the model is presented with a set of previously
unseen test stimuli with the target concept removed. For each stimulus, the
model predicts the missing concept based on the features $\mathbf{f}$ (i.e.,
context words).

###  6.1 Method

Like in the category evaluation above, we compare the ranking performance of
BCF, BayesCat, the co-occurrence based model, and Strudel for English. For the
Bayesian models, we directly exploit the learnt distributions. For BCF, we
compute the score of a target concept $c$ given a set of features as:

$Score(c|\mathbf{f})=\sum_{g}P(g|c)P(\mathbf{f}|g).$ (18)

Similarly, for BayesCat we compute the score of a concept $c$ given a set of
features as follows:

$Score(c|\mathbf{f})=\sum_{k}P(c|k)P(\mathbf{f}|k).$ (19)

For both Strudel and the co-occurrence model, we rank concepts according to
the cumulative association over all observed features for a particular concept
$c$. For Strudel, association corresponds to log-likelihood ratio-based
association scores, while for the co-occurrence model it corresponds to co-
occurrence counts, concept $c$:

$Score(c|\mathbf{f})=\sum_{f\in\mathbf{f}}association(c,f).$ (20)

We also report a baseline which randomly selects target concepts from the full
set of concepts.

We report precision at rank 1, 10, and 20. We also report the average rank
assigned to the correct concept. All results are based on a random test set of
previously unseen stimuli.

clam |  |  saltwater soft naked marine family
---|---|---
BCF |  a |  snail |  clam |  moth |  hare |  salamander
BayesCat |  |  snail |  mackerel |  clam |  squid |  crab
Strudel |  |  house |  apartment |  snail |  moth |  beetle
Co-occ baseline |  |  level |  snail |  otter |  whale |  dolphin
nylon |  |  product parachute stocking silk fibre
BCF |  |  nylon |  crayon |  mat |  birch |  cedar
BayesCat |  |  card |  myg |  bottle |  doll |  bag
Strudel |  |  nylon |  apple |  spider |  carpet |  knee
Co-occ baseline |  |  skirt |  nightgown |  blouse |  necklace |  swimsuit
lamb |  |  faithful consecrate service gift
BCF |  |  candle |  bouquet |  bread |  robe |  lamb
BayesCat |  |  cabinet |  board |  bin |  level |  whip
Strudel |  |  train |  bus |  church |  ship |  chapel
Co-occ baseline |  |  emu |  ambulance |  train |  hook |  trolley
Figure 5: Illustration of the concept prediction task; we show the input
presented to the model (i.e., features) followed by its predictions (target
concept). The top five predictions for BCF, BayesCat, Strudel, and the co-
occurrence model are shown (left to right) and the correct answer is
highlighted in bold face (if included in the top five predictions).

###  6.2 Results

Figure 5 depicts three English stimuli, together with concept predictions from
BCF and the co-occurrence model. Table 4 shows quantitative results of the
three models averaged over a corpus of 300 test stimuli for all languages.
Both BCF and the co-occurrence model outperform the random baseline by a large
margin, and BCF achieves consistently highest scores. Both Bayesian models
(BCF and BayesCat) outperform the co-occurrence model across all metrics and
conditions. We assume that plain concept-feature co-occurrence information
might be too sparse to provide a strong signal of concept relevance given a
set of features. The Bayesian models, on the other hand, learn complex
correspondences between features and all concepts in a category. BayesCat and
BCF perform comparably given that they exploit local co-occurrence relations
in similar ways. BCF learns feature associations which discriminate concepts
more accurately, suggesting that the joint learning objective and structured
feature information is beneficial. The example predictions in Figure 5
corroborate this.

Cross-lingual comparisons reveal that, compared to BCF, the performance of the
co-occurrence model degrades more severely for languages other than English.
This suggests that BCF can leverage information more efficiently from smaller
learning corpora (see Table 2). The number of concepts (i.e., target items to
be ranked) differs across languages so that absolute numbers are not directly
comparable.

Figures 2 and 4 qualitatively support the claim that BCF learns meaningful
features across languages, which are overall coherent and relevant to their
associated category. Some interesting cultural differences emerge, for example
German is the only language for which a measurement feature type is induced
for vegetables (Figure 4(b); de, 4th from left), while for clothing, a fashion
industry feature emerges in French (Figure 4(a); fr, 3rd from left). For the
same category, a feature type pertaining to colour emerges for all five
languages (4(a), bold margins). In addition, some features in other languages
were not straightforwardly translatable into English. For example, the 3rd
feature type for vegetables in Chinese (Figure 4(b)) includes the word 分 which
refers to the extent to which food is cooked 444Roughly equivalent to the
English rare, medium, well-done, but applicable to all kinds of food. and 烂
which is the stage when food starts to fall apart after cooking (stewing). In
addition, the feature types induced for the Chinese clothing category include
two words which both translate to the English word wear, but in Chinese are
specific to wearing small items (e.g., jewelery; 戴), and wearing clothes (穿),
respectively. Language-specific features are meaningful, and at the same time
category-feature associations across languages reflect culture-driven
differences.

##  7 Experiment 3: Feature Relevance and Coherence

‘Select the intruder word.’
---
$\circ$ |  $\circ$ |  $\circ$ |  $\circ$ |  $\bullet$ |  $\circ$
color |  green |  blue |  white |  milk |  red
$\circ$ |  $\bullet$ |  $\circ$ |  $\circ$ |  $\circ$ |  $\circ$
cell |  violin |  study |  protein |  human |  disease
Figure 6: Example task for our feature coherence study, correct answers are
marked with a filled circle. The example task involves categories and features
induced by BCF from the English Wikipedia. ‘Select intruder feature type
(right) wrt category (left).’
---
wasp ant caterpillar hornet moth housefly beetle honeydew grasshopper | $\circ$ |  insect beetle family larva spider
$\circ$ |  tree leaf plant nest grow
| $\bullet$ |  guitar piano clarinet flute trumpet
| $\circ$ |  male female egg length cm
| $\circ$ |  white brown dark tail color
| $\circ$ |  population habitat bird forest water
Figure 7: An example task for our feature relevance study. The example task
involves categories and features induced by BCF from the English Wikipedia.

Given that our aim is to induce cognitive representations of the world, the
ultimate assessment of the model’s representations is their meaningfulness to
humans, i.e., speakers of our target languages. To this end, we elicited
judgments of feature quality from native speakers using the crowd sourcing
platforms CrowdFlower555https://www.crowdflower.com/ and Amazon Mechanical
Turk.666https://www.mturk.com/ Specifically, we are interested in two
questions: (1) Do induced feature types have a single coherent underlying
theme such as color or function (feature coherence); (2) Do feature types
associated with a category relate to that category (feature relevance)?

We compared the feature types learnt by BCF against the co-occurrence model as
well as BayesCat. For English we also include Strudel. We omitted the random
baseline from this evaluation since it was clearly inferior in previous
simulations.

###  7.1 Method

We adopted the topic intrusion experimental paradigm [73] for assessing the
induced features in two ways. Firstly, we examined whether the feature types
our model learns are thematically coherent. Participants were presented
features types (as lists of words), which were augmented with a random
‘intruder’ feature, and their task was to correctly identify the intruder
feature. Figure 6 displays an example task. If the feature types are
internally coherent we expect annotators to identify the intruder with high
accuracy. We evaluated all 50 feature types as induced by BCF and the co-
occurrence model.

Secondly, we assessed the relevance of feature types assigned to any category.
An example task is shown in Figure 7. We presented participants with a
category and five feature types (each as a list of words), one of which was
randomly added and was not associated with the category in the model output.
Again, they needed to select the correct intruder. If category-feature type
associations induced by the model are generally relevant, annotators will be
able to identify the intruder with high accuracy. We evaluated all 40 induced
categories and their associated features for BCF and the co-occurrence model.

For both elicitation studies, we obtained 10 responses per task (see Figures 6
and 7); participants judged a single concept and its features per task. All
participants were required to be native speakers of the language they were
evaluating, and we filtered crowdworkers through their location of residence
and self-reported native language (using the functionality provided by the
crowdsourcing platforms). We additionally included test questions among tasks
for which the true answer was known, and discarded the data from participants
who failed to achieve high accuracy on these test questions. Overall, we
obtained 50$\times$10 responses for the feature coherence study and
40$\times$10 responses for feature relevance.

We report the average accuracy across participants of selecting the correct
intruder feature and intruder feature type, respectively. In addition we
report inter annotator agreement (IAA) using Fleiss Kappa [74]. The extent to
which annotators agree in their judgments allows us to evaluate the difficulty
of the task, as well as the reliability of the results.

###  7.2 Results

| model | accuracy | IAA
---|---|---|---
English | BCF | 0.752 | 0.700
BayesCat | 0.605 | 0.456
strudel | 0.435 | 0.364
2-4 | co-occ baseline | 0.302 | 0.229
German | BCF | 0.530 | 0.361
BayesCat | 0.370 | 0.206
2-4 | co-occ baseline | 0.340 | 0.201
French | BCF | 0.555 | 0.427
BayesCat | 0.468 | 0.294
2-4 | co-occ baseline | 0.278 | 0.216
Chinese | BCF | 0.468 | 0.419
BayesCat | 0.261 | 0.108
2-4 | co-occ baseline | 0.390 | 0.310
Arabic | BCF | 0.385 | 0.278
BayesCat | 0.385 | 0.215
2-4 | co-occ baseline | 0.278 | 0.155
Table 5: Results of our feature relevance study for BCF (this work), BayesCat,
the co-occurrence model (co-occ), and Strudel (for English only) in terms of
accuracy (the proportion of intruders identified correctly) and inter-
annotator agreement (IAA; Fleiss Kappa [74]) for five target languages.

Table 5 displays the results for the feature relevance study and Table 6 the
feature coherence study. Table 5 shows that humans are able to detect intruder
feature types with higher accuracy in the context of BCF-induced
representations, compared with all comparison models. Additionally, inter
annotator agreement (IAA) is consistently higher for BCF, indicating that
participants more frequently agreed on their selections and that selecting
intruders in the BCF output was an easier task for them compared to the
comparison models. Similar to the previous simulations, we observe that both
Bayesian models (BayesCat and BCF) outperform the count-based models. In this
evaluation, however, we also observe a clear advantage of BCF compared to
BayesCat, which does not learn structured feature types inherently. BCF learns
to associate relevant features to categories.

| model | accuracy | IAA
---|---|---|---
English | BCF | 0.814 | 0.710
BayesCat | 0.642 | 0.529
strudel | 0.260 | 0.294
2-4 | co-occ baseline | 0.266 | 0.296
German | BCF | 0.760 | 0.639
BayesCat | 0.465 | 0.341
2-4 | co-occ baseline | 0.220 | 0.210
French | BCF | 0.690 | 0.527
BayesCat | 0.557 | 0.409
2-4 | co-occ baseline | 0.224 | 0.227
Chinese | BCF | 0.574 | 0.723
BayesCat | 0.269 | 0.234
2-4 | co-occ baseline | 0.228 | 0.227
Arabic | BCF | 0.594 | 0.444
BayesCat | 0.416 | 0.423
2-4 | co-occ baseline | 0.236 | 0.195
Table 6: Results from our feature coherence study for BCF (this work) ,
BayesCat, the co-occurrence model (co-occ), and Strudel (for English only) in
terms of accuracy (the proportion of intruders identified correctly) and
inter-annotator agreement (IAA; Fleiss Kappa [74]) for five target languages.

Table 6 shows the results of the feature coherence study, where the overall
pattern of results is similar as above. We can see that participants are able
to detect intruder features from the types learnt by BCF more reliably than
from those learnt by all comparison models. Again, both Bayesian models
outperform the count-based baselines both in terms of accuracy and inter
annotator agreement. The superior performance of BCF compared to BayesCat
indicates that its ability to learn structured features jointly with
categories in a single process leads to higher quality feature
representations. In particular, in addition to associating relevant feature
types with categories, the feature types themselves are internally coherent,
pertaining to different aspects or properties of the reference category.

Comparing results across languages we observe that scores for English exceed
scores for all other languages. At the same time, for almost all models and
languages the IAA scores fall under the category of ‘fair agreement’
($0.20<\kappa<0.40$) indicating that the elicitation task was feasible for
crowdworkers. This applies to both evaluations (Tables 5 and 6). We observed a
similar pattern in the results of Experiment 1 (Table 3). We believe there are
two reasons for this drop. Firstly, in order to perform cross-linguistic
experiments, we translated English categories into other languages. As
mentioned in Sections 5.2 and 6.2, such a direct correspondence may not always
exist. Consequently, annotators for languages other than English are faced
with a noisier (and potentially harder) task. Secondly, while it is
straightforward to recruit English native speakers on crowd sourcing
platforms, it has proven more challenging for the other languages. We suspect
that our effort to recruit native speakers, might not have been entirely fail-
safe for languages other that English, and that the language competence of
those crowdworkers might have impacted the quality of their judgments.

Overall, we conclude that jointly inducing structured features together with
categories from natural language corpora in different languages enables BCF to
learn feature types which are (1) internally coherent, referring to a single
underlying theme; and (2) informative about the categories with which they are
associated.

##  8 General Discussion

We presented the first large-scale, cross-linguistic analysis of
categorization using naturally occurring data. We showed that rational
Bayesian models of categorization can learn meaningful categories and their
features from complex environments resembling the natural world more closely
than limited laboratory settings.

We developed BCF, a cognitively motivated Bayesian model, and investigated its
ability to learn categories (for hundreds of concepts) and their structured
features from corpora in five languages. Like humans ‘in the wild’, our model
learns categories and relevant features jointly [29, 30], and induces
structured  representations of categories [31, 32, 33]. Compared to a simpler
co-occurrence model and a Bayesian model with no access to these mechanisms
BCF learns better categories and features which are rated as more relevant and
coherent by humans. BCF models category acquisition as a general, language-
independent process. It neither utilizes language-specific knowledge nor
requires language-specific tuning, and as such paves the way for future
investigations involving more languages, or different kinds of corpora.

Our study sheds light on the acquisition of concrete concepts and their
features from text, and as such adopts a constrained view of both the learning
environment and the learning target. It suggests a number of interesting
suggestions for future research. First, this article considered natural
language input as an approximation of the environment from which categories
and their representations are learnt. While we showed that the linguistic
environment is a useful approximation of the full multimodal input a learner
has access to, it is clear that language cannot capture this multimodal
environment is not captured in its entirety. Computational models of word
learning have been trained on multimodal input data (albeit on smaller-scale
problems; [75, 76]). Advantageously, Bayesian models are flexible with respect
to the input data they receive, so we expect the application of our model to
multimodal data to be a feasible avenue for future work. Applying our models
to such data sets would allow to compare the category acquisition process as
well as the acquired representations from multimodal input against those
emerging from language data alone.

A second direction for future work concerns the cognitive assumptions
underlying the learning setup. The models discussed in this article learn from
collections of natural language stimuli consisting of a target concept mention
and its surrounding context. This input is based on the rather bold assumption
that the learner already has substantial linguistic prior knowledge prior to
concept and feature learning: she has successfully mapped each target concept
to a word. As supported by an extensive literature [5, 6, 4], word learning,
itself a fundamental challenge for young infants, and concept learning exhibit
a mutual influence. Our work remains agnostic about the fact that the meaning
of words itself needs to be acquired, and that knowledge about concepts and
categories will help tackle the word learning problem. A fully faithful model
would consider the problems of word and concept or category learning jointly.
Extending BCF to account for this joint optimization, and investigating
emerging acquisition patterns across different languages, will be a very
interesting avenue for future research.

Humans not only categorize the physical world around them, but also infer
complex representations of abstract categories and concepts such as political
(e.g., _parliament_ , _socialist_), legal (e.g., _law_ , _trial_), or feelings
(e.g., _mirth_ or _embarrassment_). Lacking any physical realization, and
hence perceivable properties, there is evidence that language plays a
particularly important role in acquiring the meaning of such abstract concepts
[77]. A data-driven study across languages would be particularly interesting
in the context of abstract categories, whose representations are expected to
be more sensitive to the cultural environment.

In conclusion, our investigations in to category and feature learning from
text across languages corroborate prior results [21] that the non-linguistic
learning environment is to some extent encoded in language. They additionally
provide evidence for the stronger statement that the structure of the world
which affords rich mental categorical representations is encoded in language.
We envision scalable testbeds which combine naturally occurring data from
multiple modalities, for example combining text data with images or video. Our
work exemplifies the potential of interpretable statistical models for gaining
insights into the mechanisms which are at play in human cognition. We
demonstrated the potential of large naturalistic datasets for the development
and testing of computational models, and are confident that computational
cognitive models together with large naturally occurring data set will open up
novel opportunities for investigating human cognition at scale.

## Acknowledgments

This research was funded by the European Research Council (award number
681760). The funding body had no involvement in the study design, data
collection, analysis and interpretation. It was also not involved in the
writing of the report and the decision to submit the article for publication.

## References

  * Schyns and Oliva [1999] P. G. Schyns, A. Oliva,  Dr. angry and mr. smile: when categorization flexibly modifies the perception of faces in rapid visual presentations,  Cognition 69 (1999) 243 – 265.
  * Goldstone [2003] R. L. Goldstone,  Learning to perceive while perceiving to learn,  in: Perceptual organization in vision: Behavioral and neural perspectives, pp. 233–278.
  * Waxman and Markov [1998] S. R. Waxman, D. B. Markov,  Object properties and object kind: twenty-one-month-old infants’ extensions of novel adjectives,  Child Development 69 (1998) 1313–1329.
  * Borovsky and Elman [2006] A. Borovsky, J. Elman,  Language input and semantic categories: a relation between cognition and early word learning,  Journal of Child Leanguage 33 (2006) 759–790.
  * Gopnik and Meltzoff [1987] A. Gopnik, A. Meltzoff,  The Development of Categorization in the Second Year and Its Relation to Other Cognitive and Linguistic Developments,  Child Development 58 (1987).
  * Waxman and Markow [1995] S. R. Waxman, D. B. Markow,  Words as invitations to form categories: Evidence from 12- to 13-month-old infants,  Cognitive Psychology 29 (1995) 257–302.
  * Malt [1995] B. Malt,  Category coherence in cross-cultural perspective,  Cognitive Psychology 29 (1995) 85 – 148.
  * Ji et al. [2004] L.-J. Ji, Z. Zhang, R. E. Nisbett,  Is it culture or is it language? examination of language effects in cross-cultural research on categorization,  Journal of Personality and Psychology 87 (2004) 57–65.
  * Unsworth et al. [2005] S. J. Unsworth, C. R. Sears, P. M. Pexman,  Cultural influences on categorization processes,  Journal of Cross-Cultural Psychology 36 (2005) 662–688.
  * Medin et al. [2007] D. L. Medin, S. J. Unsworth, L. Hirschfeld,  Culture, categorization, and reasoning,  in: S. Kitayama, D. Cohen (Eds.), Handbook of cultural psychology, Guilford Press, 2007.
  * Nosofsky [1988] R. M. Nosofsky,  Exemplar-based accounts of relations between classification, recognition, and typicality,  Journal of Experimental Psychology: Learning, Memory, and Cognition 14 (1988) 700–708.
  * Rosch [1973] E. Rosch,  Natural categories,  Cognitive Psychology (1973) 328–350.
  * Corter and Gluck [1992] J. E. Corter, M. A. Gluck,  Explaining basic categories - feature predictability and information,  Psychological Bulletin 111 (1992) 291–303.
  * Murphy and Medin [1985] G. L. Murphy, D. L. Medin,  The role of theories in conceptual coherence,  Psychological Review 92 (1985) 289–316.
  * Anderson [1991] J. R. Anderson,  The adaptive nature of human categorization.,  Psychological Review 98 (1991) 409–429.
  * Lee and Navarro [2002] M. D. Lee, D. J. Navarro,  Extending the ALCOVE model of category learning to featural stimulus domains.,  Psychonomic Bulletin & Review 9 (2002) 43–58.
  * Bornstein and Mash [2010] M. H. Bornstein, C. Mash,  Experience-based and on-line categorization of objects in early infancy,  Child Development 81 (2010) 884–897.
  * Medin and Schaffer [1978] D. L. Medin, M. M. Schaffer,  Context theory of classification learning,  Psychological Review 85 (1978) 207–238.
  * Kruschke [1993] J. K. Kruschke,  Human category learning: Implications for backpropagation models,  Connection Science 5 (1993) 3–36.
  * Henrich et al. [2010] J. Henrich, S. J. Heine, A. Norenzayan,  The weirdest people in the world?,  Behavioral and brain sciences 33 (2010) 61–83.
  * Riordan and Jones [2011] B. Riordan, M. N. Jones,  Redundancy in perceptual and linguistic experience: Comparing feature-based and distributional models of semantic representation,  Topics in Cognitive Science 3 (2011) 303–345.
  * Fountain and Lapata [2011] T. Fountain, M. Lapata,  Incremental models of natural language category acquisition,  in: Proceedings of the 33nd Annual Conference of the Cognitive Science Society, pp. 255–260.
  * Frermann and Lapata [2016] L. Frermann, M. Lapata,  Incremental bayesian category learning from natural language,  Cognitive Science 40 (2016) 1333–1381.
  * Griffiths et al. [2008] T. L. Griffiths, C. Kemp, J. B. Tenenbaum,  Bayesian models of cognition,  Cambridge Handbook of Computational Cognitive Modeling (2008).
  * Chater et al. [2010] N. Chater, M. Oaksford, U. Hahn, E. Heit,  Bayesian models of cognition,  Wiley Interdisciplinary Reviews: Cognitive Science 1 (2010) 811–823.
  * Perfors et al. [2011] A. Perfors, J. B. Tenenbaum, T. L. Griffiths, F. Xu,  A tutorial introduction to bayesian models of cognitive development,  Cognition 120 (2011) 302 – 321.
  * Sanborn et al. [2010] A. N. Sanborn, T. L. Griffiths, D. J. Navarro,  Rational approximations to rational models: Alternative algorithms for category learning.,  Psychological review 117 (2010) 1144–1167.
  * Xu and Tenenbaum [2007] F. Xu, J. B. Tenenbaum,  Word learning as bayesian inference,  Psychological Review 114 (2007) 245–272.
  * Goldstone et al. [2001] R. L. Goldstone, Y. Lippa, R. M. Shiffrin,  Altering Object Representations through Category Learning,  Cognition 78 (2001) 27–43.
  * Schyns and Rodet [1997] P. G. Schyns, L. Rodet,  Categorization creates functional features,  Journal of Experimental Psychology: Learning, Memory, and Cognition 23 (1997) 681–696.
  * Ahn [1998] W.-K. Ahn,  Why are different features central for natural kinds and artifacts?: the role of causal status in determining feature centrality.,  Cognition 69 (1998) 135.
  * McRae et al. [2005] K. McRae, G. S. Cree, M. S. Seidenberg, C. McNorgan,  Semantic feature production norms for a large set of living and nonliving things.,  Behavioral Research Methods 37 (2005) 547–59.
  * Spalding and Ross [2000] T. L. Spalding, B. H. Ross,  Concept learning and feature interpretation,  Memory & Cognition 28 (2000) 439–451.
  * Caliskan et al. [2017] A. Caliskan, J. J. Bryson, A. Narayanan,  Semantics derived automatically from language corpora contain human-like biases,  Science 356 (2017) 183–186.
  * Goldstone and Lupyan [2016] R. L. Goldstone, G. Lupyan,  Discovering psychological principles by mining naturally occurring data sets,  Topics in Cognitive Science 8 (2016) 548–568.
  * Jones [2016] M. Jones, Big Data in Cognitive Science, Frontiers of Cognitive Psychology, Taylor & Francis, 2016\.
  * Paxton and Griffiths [2017] A. Paxton, T. L. Griffiths,  Finding the traces of behavioral and cognitive processes in big data and naturally occurring datasets,  Behavior Research Methods 49 (2017) 1630–1638.
  * Rips et al. [2012] L. J. Rips, E. E. Smith, D. L. Medin,  Concepts and Categories: Memory, Meaning and Metaphysics,  in: K. J. Holyoak, R. G. Morrison (Eds.), The Oxford Handbook of Thinking and Reasoning, Oxford University Press, 2012, pp. 177–209.
  * Malt and Smith [1984] B. C. Malt, E. E. Smith,  Correlated properties in natural categories,  Journal of Verbal Learning and Verbal Behavior 23 (1984) 250 – 269.
  * Heit and Rubinstein [1994] E. Heit, J. Rubinstein,  Similarity and Property Effects in Inductive Reasoning,  Journal of Experimental Psychology: Learning, Memory and Cognition 20 (1994) 411–422.
  * Trauble and Pauen [2007] B. Trauble, S. Pauen,  The role of functional information for infant categorization,  Cognition 105 (2007) 362–379.
  * Jones et al. [1991] S. Jones, L. Smith, B. Landau,  Object properties and knowledge in early lexical learning,  Child development 62 (1991) 499–516.
  * Macario [1991] J. Macario,  Young children’s use of color in classification: Foods and canonically colored objects,  Cognitive Development 6 (1991) 17–46.
  * Wu and Barsalou [2009] L. Wu, L. W. Barsalou,  Perceptual simulation in conceptual combination: Evidence from property generation,  Acta Psychologica 132 (2009) 173 – 189.
  * Barsalou [1999] L. W. Barsalou,  Perceptual symbol systems,  Behavioral and Brain Sciences 22 (1999) 577–609.
  * Keil [1989] F. C. Keil, Concepts, Kinds, and Cognitive Development, MIT Press, 1989.
  * McRae and Cree [2002] K. McRae, G. S. Cree,  Factors underlying category-specific semantic impairments,  in: E. M. E. Forde, G. Humphreys (Eds.), Category-specificity in mind and brain, Psychology Press, 2002, pp. 211–248.
  * Goldstone et al. [2008] R. L. Goldstone, E. Gerganov, D. L, M. E. Roberts, R. L. Goldstone,  Learning to see and conceive,  in: The new cognitive sciences (Part of the Vienna Series in Theoretical Biology), MIT Press, Cambridge, MA, 2008, pp. 163–188.
  * Pevtzow and Goldstone [1994] R. Pevtzow, R. L. Goldstone,  Categorization and the parsing of objects,  in: Proceedings of the 16th Annual Conference of the Cognitive Science Society, Lawrence Erlbaum Associates, 1994, pp. 712–722.
  * Goldstone and Steyvers [2001] R. L. Goldstone, M. Steyvers,  The sensitization and differentiation of dimensions during category learning,  Journal of Experimental Psychology 130 (2001) 116–139.
  * Sanborn et al. [2006] A. N. Sanborn, T. L. Griffiths, D. J. Navarro,  A more rational model of categorization,  in: Proceedings of the 28th Annual Conference of the Cognitive Science Society, pp. 726–731.
  * Frermann and Lapata [2016] L. Frermann, M. Lapata,  Incremental bayesian category learning from natural language,  Cognitive Science 40 (2016) 1333–1381.
  * Shafto et al. [2011] P. Shafto, C. Kemp, V. Mansinghka, J. B. Tenenbaum,  A probabilistic model of cross-categorization,  Cognition 120 (2011) 1 – 25.
  * Kemp et al. [2003] C. Kemp, A. Perfors, J. B. Tenenbaum,  Learning domain structures,  in: Proceedings of the 26th Annual Conference of the Cognitive Science Society, Erlbaum, 2003, pp. 720–725.
  * Perfors et al. [2005] A. Perfors, C. Kemp, J. B. Tenenbaum,  Modeling the acquisition of domain structure and feature understanding,  in: Proceedings of the 27th Annual Conference of the Cognitive Science Society, pp. 1720–1725.
  * Perfors and Tenenbaum [2009] A. Perfors, J. B. Tenenbaum,  Learning to learn categories,  in: Proceedings of the 31st Annual Conference of the Cognitive Science Society, pp. 136–141.
  * Steyvers [2010] M. Steyvers,  Combining feature norms and text data with topic models.,  Acta Psychologica 133 (2010) 234–243.
  * Blei et al. [2003] D. M. Blei, A. Y. Ng, M. I. Jordan,  Latent Dirichlet allocation,  Journal of Machine Learning Research 3 (2003) 993–1022.
  * Baroni et al. [2010] M. Baroni, B. Murphy, E. Barbu, M. Poesio,  Strudel: A corpus-based semantic model based on properties and types,  Cognitive Science 34 (2010) 222–254.
  * Devereux et al. [2009] B. Devereux, N. Pilkington, T. Poibeau, A. Korhonen,  Towards unrestricted, large-scale acquisition of feature-based conceptual representations from corpus data,  Research on Language & Computation 7 (2009) 137–170.
  * Kelly et al. [2014] C. Kelly, B. Devereux, A. Korhonen,  Automatic extraction of property norm-like data from large text corpora,  Cognitive Science 38 (2014) 638–682.
  * Griffiths and Tenenbaum [2006] T. L. Griffiths, J. B. Tenenbaum,  Optimal predictions in everyday cognition,  Psychological Science 17 (2006) 767–773.
  * Marr [1982] D. Marr, Vision: A Computational Investigation into the Human Representation and Processing of Visual Information, Henry Holt and Co., Inc., New York, NY, USA, 1982.
  * Geman and Geman [1984] S. Geman, D. Geman,  Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images,  IEEE Transactions on Pattern Analysis and Machine Intelligence 6 (1984) 721–741.
  * Bishop [2006] C. M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics), Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2006.
  * MacKay [2002] D. J. C. MacKay, Information Theory, Inference & Learning Algorithms, Cambridge University Press, New York, NY, USA, 2002.
  * Vinson and Vigliocco [2008] D. Vinson, G. Vigliocco,  Semantic feature production norms for a large set of objects and events,  Behavior Research Methods 40 (2008) 183–190.
  * Fountain and Lapata [2010] T. Fountain, M. Lapata,  Meaning representation in natural language categorization,  in: Proceedings of the 32nd Annual Conference of the Cognitive Science Society, pp. 1916–1921.
  * Baroni [2010] M. Baroni, Detailed description of the Strudel algorithm, Technical Report, 2010.
  * Lloyd [1982] S. Lloyd,  Least squares quantization in pcm,  IEEE transactions on information theory 28 (1982) 129–137.
  * Lang and Lapata [2011] J. Lang, M. Lapata,  Unsupervised semantic role induction with graph partitioning,  in: Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 1320–1331.
  * Rosenberg and Hirschberg [2007] A. Rosenberg, J. Hirschberg,  V-measure: A conditional entropy-based external cluster evaluation measure,  in: Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 410–420.
  * Chang et al. [2009] J. Chang, S. Gerrish, C. Wang, J. L. Boyd-graber, D. M. Blei,  Reading tea leaves: How humans interpret topic models,  in: Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, A. Culotta (Eds.), Advances in Neural Information Processing Systems 22, Curran Associates, Inc., 2009, pp. 288–296.
  * Fleiss [1981] J. L. Fleiss, Statistical Methods for Rates and Proportions, Wiley, New York, 1981\.
  * Frank et al. [2009] M. C. Frank, N. D. Goodman, J. B. Tenenbaum,  Using speakers’ referential intentions to model early cross-situational word learning,  Psychological Science 20 (2009) 579–575.
  * Yu and Smith [2007] C. Yu, L. B. Smith,  Rapid Word Learning Under Uncertainty via Cross-Situational Statistics,  Psychological Science 18 (2007) 414–420.
  * Wiemer-Hastings and Graesser [2000] K. Wiemer-Hastings, A. C. Graesser,  Contextually Representing Abstract Concepts with Abstract Structures,  in: Proceedings of the 22nd Annual Meeting of the Cognitive Science Society, pp. 983–989.