# Composing and Embedding the
Words-as-Classifiers Model of Grounded Semantics

Daniele Moro
Boise State University
1910 University Dr.
Boise, ID 83725
danielemoro@
u.boisestate.edu
&Stacy Black
Boise State University
1910 University Dr.
Boise, ID 83725
stacyblack@
u.boisestate.edu
&Casey Kennington
Boise State University
1910 University Dr.
Boise, ID 83725
caseykennington@
boisestate.edu

###### Abstract

The words-as-classifiers model of grounded lexical semantics learns a semantic
fitness score between physical entities and the words that are used to denote
those entities. In this paper, we explore how such a model can incrementally
perform composition and how the model can be unified with a distributional
representation. For the latter, we leverage the classifier coefficients as an
embedding. For composition, we leverage the underlying mechanics of three
different classifier types (i.e., logistic regression, decision trees, and
multi-layer perceptrons) to arrive at a several systematic approaches to
composition unique to each classifier including both denotational and
connotational methods of composition. We compare these approaches to each
other and to prior work in a visual reference resolution task using the
refCOCO dataset. Our results demonstrate the need to expand upon existing
composition strategies and bring together grounded and distributional
representations.

##  1 Introduction

The words-as-classifiers (wac) model of lexical semantics has shown promise as
a way to acquire grounded word meanings with limited training data in
interactive settings. Introduced in Kennington and Schlangen (2015) for
grounding words to visual aspects of objects, Schlangen et al. (2016) showed
that the model could be generalized to work with any object representation
(e.g., a layer in a convolutional neural network) with “real” objects depicted
in photographs. The wac model builds on prior work Larsson (2015) treating
formal predicates as classifiers to effectively learn and determine class
membership of entities (e.g., an object $x$ denoted as “red” in an utterance
belongs to the predicate $red(x)$ class). The wac model has been used to
ground into modalities beyond just vision, including simulated robotic hand
muscle activations Moro and Kennington (2018), and wac has been used for
language understanding in a fluid human-robot interaction ask Hough and
Schlangen (2016) . Beyond comprehension tasks, the wac model has also been
used for referring expression generation Zarrieß and Schlangen (2017). The wac
classifiers can use any set of features from any modality and the model is
_interpretable_ because each word has its own classifier. Moreover, the wac
model allows for incremental, word-by-word composition which has implications
for interactive dialogue: human users of incremental spoken dialogue systems
perceive them as being more natural than non-incremental systems (Aist et al.,
2006; Skantze and Schlangen, 2009; Asri et al., 2014).

Though the wac model has pleasing theoretical properties and practical
implications for interactive dialogue and robotic tasks, Li and Boyer (2016)
and Emerson and Copestake (2017) point out that wac treats all words
independently, thereby ignoring distributional relations and meaning
representations, and _composition_ using the wac model has generally resulted
in averaging over applications of wac to objects–a purely intersectional
approach to semantic composition, which has been shown to fail in many cases
Kamp (1975). As is well known, linguistic structures are compositional—simple
elements can be functionally combined into more complex elements Frege
(1892)—and composition, which is an important aspect of grounding into visual
or other modalities, is itself a process that is a function of how a
particular model of lexical semantics is represented.111To illustrate,
following Baroni and Zamparelli (2010), the _formal semantics_ approach to
lexical semantics, as derived from Montague (1970, 1973), treats lexical
meaning as predicates which functionally determine class membership. In
contrast, _distributional_ representations (e.g., vector space models Turney
and Pantel (2010) including embeddings) differ from formal semantics in that
word meanings are represented not as predicates, but as high-dimensional
vectors and as a result approaches to composition revolve around tensor
operations from summation and multiplication Mitchell and Lapata (2010) to
treating adjectives as matrices and nouns as vectors Baroni and Zamparelli
(2010).

The goal of this paper is (1) to explore and understand possible approaches to
composition for wac, and (2) by determining the best wac model, we extend wac
by using classifier coefficients as embedding vectors in a semantic similarity
task. At the outset, following Schlangen et al. (2016), we make an important
distinction by identifying two composition types: in any task, composition can
be handled at the level of _connotation_ where words are composed _then_
applied in the task (e.g., tensor operations of word embeddings are generally
composed by summation before they are used in a task, such as sentiment
classification), and at the level of _denotation_ where words are applied then
composed in a task (e.g., wac classifiers are applied to object
representations, then the results of those applications are composed, such as
reference resolution to visual objects). These differences in composition
process are depicted in Figure 1 which distinguishes between application to a
task and composition.

Figure 1: Left: connotational _compose-then-apply_ ; Right: denotational
_apply-then-compose_.

More specifically, we alter the wac model to leverage several classifier
types, namely logistic regression, multi-layer perceptrons, and decision trees
which allow us to use both connotative and denotative composition strategies
and benefit from the different classifiers, for example by grafting decision
trees together and by making use of the hidden layers in the multi-layer
perceptrons (explained in Section 4). Our evaluations show that the choice of
classifier and the composition strategies that those classifiers afford yield
varied, yet comparable results in a visual reference resolution task (Section
5). Our analyses of the multi-layer perceptron model shows that the
coefficients of the neurons in the hidden layers show properties that are
similar to distributional embedding approaches to lexical semantics (Section
5.3), which we evaluate in Experiment 3. Finally, we conclude and explain our
plans for future work.

##  2 Related Work

Emerson and Copestake (2017) offer a comprehensive review and discussion on
different approaches to compositionality including formal frameworks, tensor-
based composition, and syntactic dependencies, to which we refer the reader.
Their own model’s application of composition uses distributional semantics and
a probabilistic graphical model.

Recursive neural networks can arguably encode compositional processes
directly, but it has been shown recently in Lake and Baroni (2017) that they
fail in a proof-of-concept neural machine translation task. The authors
suggest a lack of systematicity in how these networks learn the composition
process. Moreover, Socher et al. (2014) reported successful application of
composition using recursive neural models, but their approach made use of
syntactic information from dependency parses; i.e., the composition was likely
guided by the syntactic representation rather than the neural network. Yu et
al. (2018) introduce MAttNet, which leverages recursive networks to obtain
high results on a referring expression task using the same data we use for our
experiments. In contrast with this work, our primary purpose is not to achieve
state-of-the-art results, but rather a systematic check of composition
strategies.

Comparable to their work and ours here is Paperno et al. (2014) which
attempted distributional composition using linguistic information similar to
dependency parses. These approaches and tasks generally apply a connotative
strategy of composition, where the final task is applied after combining
meaning representations of the parts to first arrive at a single meaning
representation of a sentence. Our work also compares to Wu et al. (2019) which
proposed a model that incorporates grounded and distributional information,
which we explore in Experiment 3.

We extend prior work to mitigate wac’s shortcomings by modeling several
different denotative and connotative strategies made possible by the machinery
of our chosen classifiers, and we use the resulting classifier coefficients as
a possible way to bring wac into a distributional space.222While we are not
claiming that our approach is cognitively plausible, we take inspiration from
Barsalou (2015) which notes that physical situations play central roles in
establishing and using concepts–the wac model is such a grounded model that
considers physical context. Moreover, the wac model is not an idealized
account of concept representation; indeed, we take the vocabulary as our (very
noisy) ontology and the classifiers learn probabilistic mappings which are far
from idealized formal representations.

##  3 Data

To give context to understanding our model and composition approaches, we
first explain the data that we use in our experiments. We use the same dataset
as described in Schlangen et al. (2016), the “Microsoft Common Objects in
Context” (MSCOCO) collection Lin et al. (2014), which contains over 300k
images with object segmentations, object labels, and image captions, augmented
by Mao et al. (2016) to add English referring expressions to image regions
(i.e., refCOCO). The average length of the referring expressions is 8.3 tokens
and the average number of image regions (which we treat as reference
candidates in our experiments) is 8. We used the data access interface
provided by Kazemzadeh et al. (2014), which included a defined
training/valiation/test split of the
data.333https://github.com/lichengunc/refer An example image, image region,
and three referring expressions to that image region are depicted in Figure 2.

1

Figure 2: Example of refCOCO image with three referring expressions (one on
each line) made to an image region (see Footnote 3 for source).

##  4 Model

The wac approach to lexical semantics is essentially a task-independent
approach to predicting semantic appropriateness of words in physical contexts.
The wac model pairs each word $w$ in its vocabulary $V$ with a classifier that
maps the real-valued features $x$ of an entity $ent$ to a semantic
appropriateness (i.e., class membership) score:

$[\\![w]\\!]_{ent}=\lambda\mathbf{x}.p_{w}(\mathbf{x})$ (1)

For example, to learn the connotative meaning of the word red, the low-level
features (e.g., visual) of all objects referred to with the word red in a
corpus of referring expressions are given as positive instances to a
supervised learning classifier. Negative instances are randomly sampled from
the complementary set of referring expressions (i.e., not containing the word
red). This results in a trained $\lambda\mathbf{x}.p_{red}(\mathbf{x})$, where
$x$ is an object that can be applied to $red$ to determine class membership.

###  4.1 Approaches to Composition with WAC

Traditionally, the wac model has been applied using independent linear
classifiers, such as logistic regression. In this paper, we expand upon the
previous work by conducting experiments with a variety of classifiers, such as
multi-layer perceptrons (mlp) and decision trees (dt), in addition to logistic
regression (lr). We use these classifiers so as to not make assumptions of
linearity (i.e., mlp) and to use the available machinery that mlps and dts
afford us for exploring composition strategies. We explain below each approach
and the methods of composition for each. It is important to note that in our
explanations and discussions of composition, we are focusing on _application_
of the classifiers; all classifiers are trained as explained above (i.e., by
pairing words with positive and negative examples). We leave more specific
explanation of training to Section 5.

####  4.1.1 Logistic Regression

#####  lr summed-predictions

The traditional approach to composition using wac uses a denotative strategy
where each wac classifier in a referring expression is applied to an entity
(e.g., a visually present object in a scene) which yields a probability (i.e.,
a score of fitness) for each word applied to each object. Following Schlangen
et al. (2016), the resulting probabilities can be combined in several ways
including summing, averaging, or multiplication (we opt for summing in our
experiments) to produce a single overall expression-level fitness score for
each object. This operation constitutes the composition of the referring
expression into a single distribution (hence, denotative: the objects are
_applied_ to each word classifier, then the resulting probabilities are
composed into a single distribution over candidate objects). The object with
the highest score is the hypothesized referred target.

####  4.1.2 Multi-layer Perceptrons

We explain in this section how we leverage mlps for three different approaches
to composition. Unlike lr, the mlp does not need to assume linearity, and its
structure allows us greater flexibility for compositional techniques.

#####  mlp summed-predictions

For completeness and direct comparison to lr, we leverage mlps as we did with
the lr denotative approach by simply using a mlp (i.e., using a single hidden
layer of 3 neurons) in place of a lr classifier for each word. Composition is
applied after application by summing the resulting probabilities.

##### Adjectives and Nouns

Following Baroni and Zamparelli (2010), we looked at adjective-noun pairs,
which are two-word compositions that can identify how an adjective qualifies a
noun. For example, instead of treating large, green and tree separately in the
phrase the large green tree next to the lake, we use the adjectives large and
green to modify the noun tree. We explore two approaches to composing adj-noun
pairs using mlps:

_mlp adj-noun extended hidden layers_: if there exist multiple adjectives for
one noun, a separate adj-noun pair is created for each adjective preceding the
noun. When predicting, the mlp classifiers for each word in every adj-noun
pair in the referring expression is merged together into one classifier by
extending the hidden layer to include neurons from both the adjective and the
noun in a single mlp. The coefficients of the top layer (i.e., a binary
sigmoid) of the original adj and noun mlps are averaged together to produce a
single probability. The rest of the phrase is then composed normally using the
traditional wac methodology. This approach effectively leverages a connotative
strategy to compose the adj-noun pairs, then a denotative strategy to compose
the rest of the expression.444This differs from the lr connotative approach
(i.e., where the coefficients were averaged) because in this case the
coefficients in the hidden layer are not averaged, only the top layer
coefficients are averaged.

_mlp adj-noun warm start_: mlp machinery also affords the use of the warm-
start feature for the mlps. In this approach, we take the noun’s classifier in
the adj-noun pair which has already been trained, and then continue to train
the classifier (i.e., using the warm-start functionality) with data that was
used to train the adjective’s classifier. This results in a single classifier
that theoretically represents the entire adj-noun pair in a single classifier.
This approach keeps the classifiers the same size, while leveraging a
transfer-style learning approach to produce a adj-noun pair classifier that is
composed of its constituent words.

#####  mlp extended hidden layers

For this approach, we generalize the mlp adj-noun extended hidden layers
approach and apply it to the entire expression by concatenating the neurons in
each word mlp’s hidden layer and averaging the coefficients of the top layer
to form a single, composed mlp that has a single hidden layer with nodes that
can determine fitness between objects and all words in an expression.

####  4.1.3 Decision Trees

Figure 3: Example of decision tree graft of _brown_ with _dog_ : the root node
of _dog_ is grafted into the leaf nodes that would output a “true”
classification _brown_. This composes the noun-phrase “brown dog” (the dog
classifier augments the brown classifier).

We opted to apply dts because of their readable internal structure and how the
branching system lends itself to intuitive composition strategies.

#####  dt summed-predictions

For completeness and direct comparison, we leverage dts as we did with the lr
and mlp denotative approaches by simply using a dt classifier for each word.
Composition happens after application by summing the resulting predicted
probabilities.

#####  dt grafting

In this approach to composition, we leverage the dt representation by training
the classifier as usual (i.e., independently), but then to compose the
classifiers into a single classifier, we “graft” the trees of each classifier
$w_{i}$ in a referring expression to the two most probable “true” leaf nodes
of the classifier $w_{i-1}$, beginning with $w_{1}$ as the starting tree with
the root node. This grafting process of two words is depicted in Figure 3
where the root node of the decision tree for the word _dog_ is grafted into
the true leaf nodes of the decision tree classifier for the word _brown_ ,
thus allowing each word to make a contribution of the final decision of
expression-level fitness to objects. This grafting process is repeated for
each word in the referring expression, thus allowing us to apply then compose
an entire phrase.

###  4.2 Composing Relational Expressions

For this final model, we follow and extend Kennington and Schlangen (2015)’s
approach to relational expressions (e.g., containing prepositions such as
_next to_ or _above_ , etc.) by expressing those relational phrases ($r$) as a
fitness for pairs of objects (i.e., $R_{1}$ and $R_{2}$; whereas for non-
relational words, wac only learns fitness scores for single objects):

$[\\![r]\\!]^{W}=P(R_{1},R_{2}|r)$ (2)

However, in prior work, the two candidate objects were known. In our model, we
only assume that the final referred target object is known (i.e., $R_{1}$),
but the relative object is not (i.e., $R_{2}$). To handle this, we interpret
relational phrases $r$ containing prepositional words as “transitions” from
one noun phrase ($NP_{1}$) to the relative noun phrase ($NP_{2}$). For
example, in the phrase _the woman to the right of the tree_ , we first
identify the relational phrase(s) (in this case _right of_), and we use this
relational phrase to make the most likely transition from $NP_{1}$ (i.e., _the
woman_) to $NP_{2}$ (i.e., _the tree_). We do this by using a learned wac
model (i.e., trained on single objects using any of the approaches explained
above) and apply $NP_{2}$ to all candidate objects in the scene. This produces
a distribution over all objects (i.e., a partially observable $NP_{2}$); we
use the resulting argmax of that distribution to arrive at the $R_{2}$ that
was referred to by $NP_{2}$. With $R_{1}$ and $R_{2}$, we train $r$ by taking
the difference in features (i.e., simple vector subtraction) between the
feature vector for $R_{1}$ and $R_{2}$ resulting in a feature vector for $r$
that is the same size as for all other wac classifiers. During application, we
find the most likely pair by applying $NP_{1}$ and $NP_{2}$ to all objects,
then $r$ to all pairs of objects, and force identity on the $NP_{1}$
distribution and candidate $R_{1}$, as well as the $NP_{2}$ distribution and
the candidate $R_{2}$, forming a trellis-like structure. The combined (i.e.,
product) of probabilities for $NP_{1}$, $r$, and $NP_{2}$ result in a final
distribution; we take the argmax probability and the resulting $R_{1}$ object
as the target referred object.

##  5 Experiments

###  5.1 Experiment 1: Simple Expressions

In this experiment, we evaluate the performance of our approaches to
composition of simple (i.e., containing no relational words) referring
expression resolution using the refCOCO data.

##### Task & Procedure

The task is reference resolution to objects depicted in static images. For
example, the referring expression the woman in red sitting on the left would
require composition of each word (with the exception of the quantifier _the_
which signals that the referring expression should only identify a single
entity). We task the wac model (and varying compositional approaches, as
explained above) to produce a “distribution” (i.e., ranked scores) of each
object in an image. We only considered referring expressions that did not
contain words included in this list: _below, above, between, not, behind,
under, underneath, front of, right of, left of, ontop of, next to, middle of_
(as was done in Schlangen et al. (2016)) resulting in simpler referring
expressions. This resulted in 106,336 training instances and 9,304 test
instances (i.e., individual referring expressions and the corresponding image
with multiple object regions). In Experiment 2 we consider all referring
expressions, including those with relations.

Training the wac model follows prior work where each word in the refCOCO
training data is trained on all instances where that word was used to refer to
an object and 5 randomly chosen negative examples sampled from the training
data where that particular word was not used in a referring expression (i.e.,
a 5-to-1 negative to positive example ratio). For each object in the images,
following Schlangen et al. (2016), we used the image region information from
the annotated data and extract that region as a separate image that we then
pass through GoogLeNet Szegedy et al. (2015), a convolutional neural network
that was trained on data from the Large Scale Visual Recognition Challenge
2014 (ILSVRC2014) from the ImageNet corpus Jia Deng et al. (2009). That is,
GoogLeNet is optimized to recognize single objects within an image, which it
can do effectively with our extracted image regions. This results in a vector
representation of the object (i.e., region) with 1024 dimensions (i.e., we use
the layer directly below the predictions layer).555We used the development set
to evaluate other existing neural networks trained on the ImageNet data,
including approaches more recent and with better performance on the task than
GoogLeNet, but we found that GoogLeNet worked better for our task. We
concatenate to this vector 7 additional features that give information about
the image region resulting in a vector of 1031 dimensions: the (relative to
the full image from which it was extracted) coordinates of two corners, its
(relative) area, distance to the center, and orientation of the image.

To ensure that our wac models were made up of reliable classifiers, we threw
out words that had 4 or fewer positive training examples, resulting in an
English vocabulary of 2,349. For training the lr wac model, we used $L1$
normalization. For training the dt wac, we used the GINI splitting criteria
with a maximum depth of 2. For training the mlp wac model, we used a multi-
layer perceptron with a single hidden layer of 3 neurons using the tanh
activation function, the top layer is a binary sigmoid, trained using the adam
solver (alpha value of 0.1) for 2000 maximum epochs. In all cases, the best
hyper parameters were found using the development data.

##### Metrics

The metric we use for this task is _accuracy_ that the highest scoring object
in an image of candidate objects as ranked by our model matches the annotated
target object referred to by the referring expression. For example, in Figure
2, there are several annotated objects including the two people, the trees,
car, ground, etc., but the referring expression _woman on the right in white
shirt_ should uniquely identify the target object in the annotated image
region. Our target is 0.64, the result reported in Schlangen et al. (2016).

##### Results

model | Exp. 1 acc | Exp. 2 acc
---|---|---
lr summed-preds | 0.64 | 0.62
mlp summed-preds | 0.63 | 0.61
mlp adj-noun ext. hidden | 0.62 | 0.60
mlp adj-noun warm-start | 0.62 | 0.61
mlp ext. hidden | 0.6 | 0.58
mlp relational | 0.62 | 0.63
dt summed-preds | 0.55 | 0.54
dt graft | 0.39 | 0.27
Table 1: Experiment 1 & 2 results: accuracy scores for the models and
composition approaches using the refCOCO data.

The results for this experiment are in the Exp. 1 acc column of Table 1.666Yu
et al. (2018) yielded far better results than we show here–our goal is to
explore compositionality directly using a more transparent model. We see that
in all cases, the denotative  _summed-predictions_ for all three classifiers
yield respectable performance (the first row verifies results reported in
Schlangen et al. (2016) for this task). When considering connotative
approaches, the results are more nuanced: for the mlp adj-noun approaches, we
see similar results when the hidden layers are extended and when we use warm-
start. This is a positive result in that composing adj-noun pairs together
using warm-start is theoretically appealing because a single classifier can
perform the work of two, though it needs additional training data to perform
that function, whereas extending the hidden layers requires no additional
retraining, and the compose-then-apply nature of it makes it more appealing
than simply summing the predictions of each word applied to the objects, as
has been done in prior work. Unfortunately, when extending the hidden layers
to contain the neurons from the mlp classifiers for all words in the
expression, the results take a hit when compared to the denotative summed
predictions approach. The dt classifiers did not perform well in this
particular task (both the denotative summed predictions and the grafting)
which is somewhat surprising, though as can be seen in Figure 3, by limiting
the depth, the dt classifiers were unable to make use of the nuance in the
individual features.

###  5.2 Experiment 2: Referring Expressions with Relations

For this experiment, we apply our approaches of composition to all of the
refCOCO data, including cases where relations are present.

##### Task & Procedure

The task and procedure for this experiment are similar to that of Experiment 1
with the important difference that we consider all of the training and test
instances (120,266 and 9,914, respectively) for training; we do not ignore
referring expressions that have relations, for example the woman in red
sitting on the left next to the tree.

##### Metrics

The metrics for this experiment are the same as Experiment 1: accuracy that
the highest scoring object as ranked by our model matches the annotated target
object referred to by the referring expression.

##### Results

Table 1, column Exp. 2 acc contains the results for this experiment. As
expected, nearly all approaches took a hit because they were unable to handle
the composition of multiple noun phrases and learn the semantics of relational
words. The best performing model, as expected, is the mlp relational model
which applies the extended hidden layer to individual noun phrases, but
composes them using the relational word using summed predictions (the results
are statistically significant compared to the lr summed-preds row result for
Exp 2). The final relational model, which uses the extended hidden approach
for each noun phrase, applies more principled connotative composition and
works with relational expressions without requiring annotation of the relative
object.

###  5.3 Analyses

We end this experiment by offering some analysis on what the wac classifiers
are learning and compare some of the composition strategies with each other.
We put focus on the mlp classifier since it produced the best results and
allows more possible composition strategies.

##### Individual Classifiers

Similar to Kennington and Schlangen (2015), we looked at classifiers that, we
assume, learned features relating to colors. To check this, we ranged over a
wide range of colors, producing an image for each color, and passed those
through the GoogLeNet to arrive at a representation for each color that our
model could use. Our analyses show that the classifiers learned prototypical
colors for objects as well as color terms. For example, we passed all colors
through the _water_ classifier and plotted the resulting probabilities,
resulting in Figure 4. This shows that individual classifiers can pick up on
color information where an important visual property is its color (i.e.,
_water_), though that information is not apparent in the underlying
representation (i.e., the GoogLeNet vector).

Figure 4: Probabilities returned by passing color images through the _water_
classifier from the red to the violet range.

This is an important result with useful implications: this kind of transfer
learning using GoogLeNet (or any other model) trained on the ImageNet data
extends the original limited ability of those networks by using the entire
vocabulary to identify not only object types, but also attributes (e.g.,
colors, sizes, spatial placements, etc.). Importantly for interactive dialogue
with robots, the wac approach allows new vocabulary to be added word-by-word
without requiring retraining of the entire underlying network.

##### Semantic Clusters

To see if wac could also yield semantic clusters, we applied a t-distributed
Stochastic Neighbor Embedding (TSNE) van der Maaten and Hinton (2008) to the
hidden layers of the mlp classifiers (i.e., 3021 dimensions, as each hidden
layer had 3 neurons, each neuron had coefficients for 1007 features), mapping
the coefficient vectors to 2 dimensions, then applied the Density-based
Spatial Clustering of Applications with Noise (dbscan; $eps=0.7$) Schubert et
al. (2017) to cluster the TSNE result, which is depicted in Figure 5.777We
found that logistic regression coefficients did not result in any meaningful
clusters. The figure shows a large, central cluster with more informative
clusters closer to the edges. The following are several noteworthy clusters:

Figure 5: Cluster proposals of mlp classifier hidden layer node coefficients.

  * •

_yellow, red, green, blue, light, board_

  * •

_area, between, of, above, edge, next to, right_

  * •

_door, oven, stove, dishwasher, knobs_

  * •

_trailer, semi, rig, car, vehicle, van, ups, suv, taxi_

  * •

_cat, dog, horse, cow, sheep, animal_

This analysis informs us that using the mlp classifier is not only more
satisfying for composition approaches, but it can also yield coefficients that
can be used as vectors for word embeddings. Our final experiment tests this
hypothesis.

###  5.4 Experiment 3: WAC Coefficients for Semantic Similarity

In this experiment, we evaluate the vectors that are derived from the
coefficients of the mlp variant of wac in a semantic similarity task. We
follow a similar approach to Kottur et al. (2016) by creating visually
grounded word embeddings, but we make use of the WAC model and we create our
own dataset for this experiment.

##### Task & Procedure

We evaluated the wac coefficient vectors using the semantic similarity tasks
WordSim-353 Finkelstein et al. (2002) and SimLex-999 Hill et al. (2015) by
comparing the wac vectors with GloVe embeddings Pennington et al. (2014) (6B,
200d), and a concatenation of the vectors for each word in the two models. To
arrive at a model for wac with a large enough vocabulary, we retrieved 100
images (i.e., using Google Image Search) for each of 30,000 common English
words and trained the mlp classifiers as described in Experiment 1 above using
100 images for each word.

##### Metrics

For semantic similarity, we report a Spearman Correlation between the cosine
similarities of the wac coefficient vector, the GloVe embeddings, and the
combination of the two models. High numbers denote better scores.

##### Results

The results of the Spearman Correlation is shown in Table 2. The results show
that, when combined with embeddings trained on large amounts of text such as
GloVe, the wac coefficient vectors can contribute useful information derived
from the visual features that the GloVe embeddings don’t have. We conjecture
that wac does not work well on its own because the words are trained
independently, and the wac model assumes that words trained on visual
representations are concrete, whereas many words are abstract, the semantics
of which is learned in lexical context, such as distributional embeddings.
These results show that wac is a potential model for bridging grounded and
distributional approaches to lexical semantics, which we will explore further
in future work.

model | WordSim-353 | SimLex-999
---|---|---
wac | 0.485 | 0.157
GloVe | 0.630 | 0.339
combined | 0.707 | 0.326
Table 2: Experiment 3 results: Spearman Correlation between the cosine
similarities of wac, GloVe, and the two combined.

##  6 Conclusions

In this paper, we explored using lr, mlp, and dt for composing the words-as-
classifiers approach to grounded, lexical semantics. We evaluated several
methods including denotative where the classifiers are applied to objects,
then the resulting probabilities are summed, connotative where the composition
was applied first (i.e., either by extending hidden layers or using warm start
in mlp, or grafting dt). Our results show that classifiers affect results, and
the kinds of composition that can be accomplished varies depending on the
classifier. We concur with Baroni (2019) that more exploration needs to be
done in this area to learn the systematic functional applications of
composition of language; our results are an important step in that direction.

Furthermore, we showed that the wac model has properties that lends itself for
a straightforward embedding by using the classifier coefficients. We showed in
Experiment 3 that the embeddings show promise in a simple word similarity
task. We are in the process of evaluating the wac embeddings on other common
tasks and analyzing the efficacy of wac embeddings by combining them with
other approaches to semantics.

Combined with prior work, the results reported here have implications for
tasks and settings for speech interaction: wac can be trained with minimal
training data, used in dialogue systems where words ground to multiple
modalities (e.g., vision, proprioperception, predicted emotions, etc.), can be
composed incrementally, and the coefficients of the classifiers can be unified
with other distributional representations.

For future work, we will explore how wac can be coupled with formal and
distributional semantic representations to better exploit and integrate
knowledge from multiple modalities for a more holistic representation of
semantics. We are also evaluating wac in an interactive language learning task
with two different robot platforms.

## References

  * Aist et al. (2006) Gregory Aist, James Allen, Ellen Campana, Lucian Galescu, Carlos Gallo, Scott Stoness, Mary Swift, and Michael Tanenhaus. 2006.  Software architectures for incremental understanding of human speech.  In _Proceedings of CSLP_ , pages 1922—-1925.
  * Asri et al. (2014) Layla El Asri, Romain Laroche, Olivier Pietquin, and Hatim Khouzaimi. 2014.  NASTIA: Negotiating Appointment Setting Interface.  In _Proceedings of LREC_ , pages 266–271.
  * Baroni (2019) Marco Baroni. 2019.  Linguistic generalization and compositionality in modern artificial neural networks.  _arXiv_.
  * Baroni and Zamparelli (2010) Marco Baroni and Roberto Zamparelli. 2010.  Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space.  In _Proceedings of the 2010 Conference on …_ , October, pages 1183–1193, Cambridge, MA. Association for Computational Linguistics.
  * Barsalou (2015) Lawrence W. Barsalou. 2015.  Cognitively Plausible Theories of Concept Composition.  In _Compositionality and concepts in linguistics and psychology_ , pages 1–31. Springer, Cham.
  * Emerson and Copestake (2017) Guy Emerson and Ann Copestake. 2017.  Semantic Composition via Probabilistic Model Theory.  _Proceedings of IWCS_.
  * Finkelstein et al. (2002) Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002.  Placing search in context: the concept revisited.  _ACM Transactions on Information Systems_ , 20(1):116–131.
  * Frege (1892) Gottlob Frege. 1892.  Über Sinn und Bedeutung.  _Erkenntnis_ , 100(1):1–15.
  * Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015.  Simlex-999: Evaluating semantic models with (Genuine) similarity estimation.  _Computational Linguistics_.
  * Hough and Schlangen (2016) Julian Hough and David Schlangen. 2016.  Investigating Fluidity for Human-Robot Interaction with Real-time, Real-world Grounding Strategies.  In _Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 288–298, Los Angeles. Association for Computational Linguistics.
  * Jia Deng et al. (2009) Jia Deng, Wei Dong, R. Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  ImageNet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_.
  * Kamp (1975) Hans Kamp. 1975.  Two Theories about Adjectives.  In _Formal Semantics of Natural Language_ , chapter Two theori. Cambridge University Press.
  * Kazemzadeh et al. (2014) Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L Berg. 2014.  ReferItGame: Referring to Objects in Photographs of Natural Scenes.  In _Proceedings of the 2014 Conrerence on Empirical Methods in Natural Language Processing (EMNLP’14)_ , pages 787–798.
  * Kennington and Schlangen (2015) C. Kennington and D. Schlangen. 2015.  Simple learning and compositional application of perceptually groundedword meanings for incremental reference resolution.  In _ACL-IJCNLP 2015 - 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, Proceedings of the Conference_ , volume 1.
  * Kottur et al. (2016) Satwik Kottur, Ramakrishna Vedantam, José MF Moura, and Devi Parikh. 2016.  Visual word2vec (vis-w2v): Learning visually grounded word embeddings using abstract scenes.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4985–4994.
  * Lake and Baroni (2017) Brenden M. Lake and Marco Baroni. 2017.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  _arXiv_.
  * Larsson (2015) Staffan Larsson. 2015.  Formal semantics for perceptual classification.  _Journal of Logic and Computation_ , 25(2):335–369.
  * Li and Boyer (2016) Xiaolong Li and Kristy Elizabeth Boyer. 2016.  Reference Resolution in Situated Dialogue with Learned Semantics.  In _Proceedings of Sigdial_ , August. Association for Computational Linguistics.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft COCO: Common Objects in Context.  In _Computer Vision – ECCV 2014_ , pages 740–755, Cham. Springer International Publishing.
  * van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing Data using t-SNE.  _Journal of Machine Learning Research_.
  * Mao et al. (2016) Junhua Mao, Xu Wei, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan L. Yuille. 2016\.  Learning like a child: Fast novel visual concept learning from sentence descriptions of images.  In _Proceedings of the IEEE International Conference on Computer Vision_ , volume 11-18-Dece.
  * Mitchell and Lapata (2010) Jeff Mitchell and Mirella Lapata. 2010.  Composition in distributional models of semantics. _Cognitive science_ , 34(8):1388–1429.
  * Montague (1970) Richard Montague. 1970.  Universal grammar.  _Theoria_ , 36(3):373–398.
  * Montague (1973) Richard Montague. 1973.  The proper treatment of quantification in ordinary English.  In _Approaches to natural language_ , pages 221–242. Springer.
  * Moro and Kennington (2018) Daniele Moro and Casey Kennington. 2018.  Multimodal Visual and Simulated Muscle Activations for Grounded Semantics of Hand-related Descriptions.  In _Proceedings of the 22nd Workshop onthe Semantics and Pragmatics of Dialogue_.
  * Paperno et al. (2014) Denis Paperno, Nghia The Pham, and Marco Baroni. 2014.  A practical and linguistically-motivated approach to compositional distributional semantics.  In _Proceedings of ACL_ , pages 90–99. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global Vectors for Word Representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Schlangen et al. (2016) David Schlangen, Sina Zarriess, and Casey Kennington. 2016.  Resolving References to Objects in Photographs using the Words-As-Classifiers Model.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_ , pages 1213–1223.
  * Schubert et al. (2017) Erich Schubert, Martin Ester, Xiaowei Xu, Hans Peter Kriegel, and Jörg Sander. 2017.  DBSCAN Revisited, Revisited: Why and How You Should (Still) Use DBSCAN.  _ACM Transactions on Database Systems_.
  * Skantze and Schlangen (2009) Gabriel Skantze and David Schlangen. 2009.  Incremental dialogue processing in a micro-domain.  _Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics on EACL 09_ , (April):745–753.
  * Socher et al. (2014) Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng. 2014.  Grounded Compositional Semantics for Finding and Describing Images with Sentences.  _Tacl_ , 2(April):207–218.
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.  Going deeper with convolutions.  In _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_.
  * Turney and Pantel (2010) Peter D Turney and Patrick Pantel. 2010.  From Frequency to Meaning: Vector Space Models of Semantics.  _Artificial Intelligence_ , 37(1):141–188.
  * Wu et al. (2019) Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, and Wei-Ying Ma. 2019.  Unified Visual-Semantic Embeddings: Bridging Vision and Language with Structured Meaning Representations.  In _Proceedings of the Combined Workshop on Spatial Language Understanding (SpLU) Grounded Communication for Robotics (RoboNLP) Grounded Communication for Robotics (RoboNLP)_.
  * Yu et al. (2018) Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018.  Mattnet: Modular attention network for referring expression comprehension.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1307–1315.
  * Zarrieß and Schlangen (2017) Sina Zarrieß and David Schlangen. 2017.  Is this a child, a girl or a car? Exploring the contribution of distributional similarity to learning referential word meanings.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , volume 2, pages 86–91.


* [JM12] M. Jonsson, M. Mustaţă, Valuations and asymptotic invariants for sequences of ideals, Ann. Inst. Fourier, vol. 62 (2012), no. 6, 2145-2209 (2013).
  * [Kol08] J. Kollár, Hulls and husks, arXiv:0805.0576 (2008).
  * [KK10] J. Kollár, S. Kovács, Log canonical singularities are Du Bois, Journal of the American Mathematical Society, 23(3), (2010), 791-813.
  * [Kol23] J. Kollár, Families of varieties of general type (Vol. 231). Cambridge University Press (2023).
  * [KS06] M. Kontsevich, Y. Soibelman, Affine structures and non-archimedian geometry, The Unity of Mathematics Progress in Mathematics vol. 244, pp 321-385 (2006).
  * [Lan75] S.G.Langton, Valuative criteria for families of vector bundles on algebraic varieties. Annals of Mathematics, 101(1), (1975), 88-110.
  * [Li18] C. Li, Minimizing normalized volumes of valuations, Math. Z., 289 (2018), pp. 491-513.
  * [LLX20] C. Li,Y. Liu, C.Xu, A guided tour to normalized volume, Geometric Analysis: In Honor of Gang Tian’s 60th Birthday, 167-219 (2020).
  * [LWX19] C. Li, X. Wang, C. Xu, On the proper moduli spaces of smoothable Kähler-Einstein Fano varieties, Duke Math J., 168(8) (2019).
  * [LWX21] C. Li, X. Wang, C. Xu, Algebraicity of the metric tangent cones and equivariant K-stability, Journal of the American Mathematical Society, 34(4), (2021), 1175-1214.
  * [LX18] C. Li, C. Xu, Stability of valuations: higher rational Peking Mathematical Journal, 1, (2018), 1-79.
  * [LXZ22] C. Li, C. Xu, Z. Zhuang, Finite generation for valuations computing stability thresholds and applications to K-stability. Annals of Mathematics, 196(2), 507-566 (2022).
  * [Li19] Y. Li, A new complete Calabi-Yau metric on $\mathbb{C}^{3}$, Inventiones mathematicae, 217, 1-34 (2019).
  * [LS13] A. Liendo, H. Süss, Normal singularities with torus actions, Tohoku Math. Journal 65 (2013), no. 1, 105-130.
  * [Liu18] Y. Liu, The volume of singular Kähler-Einstein Fano varieties, Compos. Math. 154 (2018), no. 6, 1131-1158
  * [LMS23] Y. Liu, J.Moraga, H.Süß, On the boundedness of singularities via normalized volume, arXiv:2205.12326v4 (2023).
  * [LX19] Y. Liu, C. Xu, K-stability of cubic threefolds, Duke Mathematical Journal, 168(11), 2029-2073 (2019).
  * [MSY08] D. Martelli, J. Sparks, S. T. Yau, Sasaki-Einstein manifolds and volume minimisation, Comm. Math. Phys. 280 (2008), no.3, pp. 611-673.
  * [Mat89] H. Matsumura, Commutative Ring Theory, Cambridge Studies in Advanced Mathematics Book no. 8, (English Edition) (1989), Cambridge University Press.
  * [MR88] T. Mora, L. Robbiano, The Gröbner fan of an ideal. Journal of Symbolic Computation, 6(2-3), 183-208 (1988).
  * [MS21] J. Moraga, H. Süß, Bounding toric singularities with normalized volume, arXiv:2111.01738 (2021).
  * [Mum65] D. Mumford, Geometric Invariant Theory, Springer-Verlag (1965).
  * [Mus02] M. Mustaţă, On multiplicities of graded sequences of ideals, J. Algebra 256 (2002), no. 1, 229-249.
  * [MN15] M. Mustaţă, J.Nicaise, Weight functions on non-archimedean analytic spaces and the Kontsevich-Soibelman skeleton, Algebraic Geometry, 2(3), 365-404.
  * [Nak88] H. Nakajima, Hausdorff convergence of Einstein $4$-manifolds, J. Fac. Sci. Univ. Tokyo 35 (1988), 411-424.
  * [Ngh24] Tran-Trung Nghiem, K-stable valuations and Calabi-Yau metrics on affine spherical varieties, arXiv:2405.05833
  * [Od10] Y. Odaka, On the GIT stability of polarized varieties: a survey, Proceeding to Kinosaki algebraic geometry symposium, 2010, 77-89.
  * [Od12] Y. Odaka, On the moduli of Kahler-Einstein Fano manifolds, Proceeding of Kinosaki algebraic geometry symposium 2013, arXiv:1211.4833
  * [OSS16] Y. Odaka, C.Spotti, S.Sun, Compact moduli spaces of del Pezzo surfaces and Kähler-Einstein metrics. Journal of Differential Geometry, 102(1), (2016), pp. 127-172.
  * [Od15a] Y. Odaka, Compact Moduli Spaces of Kähler-Einstein Fano Varieties, Publications of the Research Institute for Mathematical Sciences, 51(3), 549-565.
  * [Od15b] Y. Odaka, Invariants of varieties and singularities inspired by Kähler-Einstein problems, Proc. Japan Acad. Ser. A Math. Sci., 91 (2015)
  * [Od21] Y. Odaka, On log minimality of weak K-moduli compactifications of Calabi-Yau varieties, arXiv:2108.03832 (2021).
  * [Od22a] Y. Odaka, Polystable Log Calabi-Yau Varieties and Gravitational Instantons, J. Math. Sci. Univ. Tokyo, 29, (2022) 1-50.
  * [Od24a] Y. Odaka, Compact moduli of Ricci-flat Kähler cones and Sasaki-Einstein spaces, arXiv:2405.07939.
  * [Od24b] Y. Odaka, Stability theories over toroidal or Novikov type base and canonical modification, arXiv:2406.02489.
  * [Od22b] Y. Odaka, Degenerated Calabi-Yau varieties with infinite components, moduli compactifications, and limit toroidal structures. European Journal of Mathematics, 8(3), (2022) 1105-1157.
  * [OX12] Y. Odaka, C. Xu, Log-canonical models of singular pairs and its applications, Mathematical Research Letters, 19(2) (2012), 325-334.
  * [PSS23] D. Phong, J. Song, J. Sturm, Degeneration of Kähler-Ricci solitons on Fano manifolds, Universitatis Iagellonicae Acta Mathematica, (Tom 52), 29-43.
  * [Pin74] H.C.Pinkham, Deformations of algebraic varieties with $\mathbb{G}_{m}$-action, Astérisque, tome 20 (1974).
  * [Pro00] Y. Prokhorov, Blow-ups of canonical singularities, Algebra. Proc. Internat. Conf. on the Occasion of the 90th birthday of A. G. Kurosh, Moscow, Russia, May 25-30, 1998, Yu. Bahturin ed., Walter de Gruyter, Berlin (2000), 301-317
  * [SU81] J. Sacks, K. Uhlenbeck, The existence of minimal immersions of 2-spheres, Ann. of Math. (2) 113 (1981), 1-24.
  * [Sch68] M. Schlessinger, Functors of Artin rings, Transactions of the American Mathematical Society, 130(2), 208-222 (1968).
  * [Sho96] V. V. Shokurov, 3-fold log models, Algebraic geometry, 4. J. Math. Sci. 81 (1996), no. 3, 2667-2699.
  * [SS17] C. Spotti, S. Sun, Explicit Gromov-Hausdorff compactifications of moduli spaces of Kähler-Einstein Fano manifolds. Pure and Applied Mathematics Quarterly, 13(3), 477-515 (2017).
  * [SSY16] C. Spotti, S. Sun, C. Yao, Existence and deformations of Kähler-Einstein metrics on smoothable Q-Fano varieties. Duke Mathematical Journal, 165(16), (2016).
  * [Sun23] S. Sun, Bubbling of Kähler-Einstein metrics, arXiv:2303.11309 (2023).
  * [SRZ23] S. Sun, R. Zhang, Complex structure degenerations and collapsing of Calabi-Yau metrics, Inventiones mathematicae, (2023), no. 1, pp. 461–494.
  * [SJZ23] S. Sun, J. Zhang, No semistability at infinity for Calabi-Yau metrics asymptotic to cones, Inventiones mathematicae, 233(1), 461-494 (2023)
  * [Sze19] G. Szekelyhidi, Degenerations of $\mathbb{C}^{n}$ and Calabi-Yau metrics, Duke Math. J. 168 no.14 (2019).
  * [Sze20] G. Szekelyhidi, Uniqueness of some Calabi-Yau metrics on $\mathbb{C}^{n}$, Geometric and Functional Analysis, 30(4), 1152-1182 (2020).
  * [Tei03] B. Teissier, Valuations, deformations, and toric geometry, Valuation theory and its applications, Vol. II (Saskatoon, 1999), pp. 361-459, Fields Inst. Commun., 33, A. M. S.., Providence, (2003).
  * [Tei14] B. Teissier, Overweight deformations of affine toric varieties and local uniformization, Valuation theory in interaction, Proceedings of the second international conference on valuation theory, Segovia-El Escorial, 2011, Ed. A.Campillo, F-V.Kuhlmann, B. Teissier, European Math. Soc. Publishing House, Congress Reports Series, (2014), 474-565, arXiv:1401.5204.
  * [Tem14] M.Temkin, Metrization of differential pluriforms on Berkovich analytic spaces, arXiv:1410.3079 (2014).
  * [VdC11] C. Van Coevering, Examples of asymptotically conical Ricci-flat Kähler manifolds, Math. Z. 267 (2011), no. 1-2, pp. 465-496.
  * [Xu20] C. Xu, A minimizing valuation is quasi-monomial, Ann. of Math. (2) 191 (3) 1003 - 1030.
  * [XZ21] C. Xu, Z. Zhuang, Uniqueness of the minimizer of the normalized volume function. Cambridge Journal of Mathematics, (2021) 9(1), 149-176.
  * [XZ24] C. Xu, Z. Zhuang, Boundedness of log Fano cone singularities and discreteness of local volumes, arXiv:2404.17134 (2024).
  * [Zhu22] Z. Zhuang, On boundedness of singularities and minimal log discrepancies of Kollár components, arXiv:2202.06455v2 (2022).
  * [Zhu23] Z. Zhuang, On boundedness of singularities and minimal log discrepancies of Kollár components, II, arXiv:2302.03841 (2023).

Contact<EMAIL_ADDRESS>
Department of Mathematics, Kyoto University, Kyoto 606-8285. JAPAN