# Can Language Models Encode Perceptual Structure Without Grounding?
A Case Study in Color

Mostafa Abdou
University of Copenhagen

&Artur Kulmizev
Uppsala University

&Daniel Hershcovich
University of Copenhagen

Stella Frank
University of Trento

&Ellie Pavlick
Brown University

&Anders Søgaard
University of Copenhagen
For correspondence<EMAIL_ADDRESS>
###### Abstract

Pretrained language models have been shown to encode relational information,
such as the relations between entities or concepts in knowledge-bases —
(Paris, Capital, France). However, simple relations of this type can often be
recovered heuristically and the extent to which models implicitly reflect
topological structure that is grounded in world, such as perceptual structure,
is unknown. To explore this question, we conduct a thorough case study on
color. Namely, we employ a dataset of monolexemic color terms and color chips
represented in CIELAB, a color space with a perceptually meaningful distance
metric.

Using two methods of evaluating the structural alignment of colors in this
space with text-derived color term representations, we find significant
correspondence. Analyzing the differences in alignment across the color
spectrum, we find that warmer colors are, on average, better aligned to the
perceptual color space than cooler ones, suggesting an intriguing connection
to findings from recent work on efficient communication in color naming.
Further analysis suggests that differences in alignment are, in part, mediated
by collocationality and differences in syntactic usage, posing questions as to
the relationship between color perception and usage and context.

##  1 Introduction

Figure 1: Right: Color orientation in 3d CIELAB space. Left: linear mapping
from BERT (CC, see §2) color term embeddings to the CIELAB space.

Without grounding or interaction with the world, language models (LMs) learn
representations that encode various aspects of formal linguistic structure
(e.g., morphosyntax Tenney et al. (2019)) and semantic information (e.g.,
lexical similarity Reif et al. (2019a)). Beyond this, it has been suggested
that text-only training data is enough for LMs to also acquire factual and
relational information about the world Davison et al. (2019); Petroni et al.
(2019). This includes, for instance, some features of concrete and abstract
concepts, such as objects’ attributes and affordances Forbes et al. (2019b);
Weir et al. (2020). Furthermore, the representational geometry of LMs has been
found to naturally reflect human lexical similarity and relatedness
judgements, as well as analogy relationships Chronis and Erk (2020). However,
the extent to which these models reflect the structures that exist in humans’
perceptual world—such as the topology of visual perception Chen (1982), the
structure of the color spectrum Ennis and Zaidi (2019); Provenzi (2020), or of
odour spaces Rossiter (1996); Chastrette (1997)—is not well-understood.

If LMs are indeed able to capture such topologies—in some domains, at least—it
would mean that these structures are a) somehow reflected in language and,
thereby, encoded in the textual training data on which models are trained, and
b) learnable using models’ current training objectives and architectural
inductive biases. To the extent they are not, the question becomes whether the
information is not there in the data, or whether model and training objective
limitations are to blame. Certainly, this latter point relates to an ongoing
debate regarding what exactly language models can be expected to learn from
ungrounded form alone Bender and Koller (2020); Bisk et al. (2020); Merrill et
al. (2021). While there have been many interesting theoretical debates around
this topic, few studies have tried to address this question empirically.

Figure 2: Our experimental setup. In the center is a Munsell color chart. Each
chip in the chart is represented in the CIELAB space (right) and has 51 color
term annotations. Color term embeddings are extracted through various methods.
In the Representation Similarity Analysis experiments, a corresponding color
chip centroid is computed in the CIELAB space. In the Linear Mapping
experiments, a color term embedding centroid is computed per chip.

In this paper, we conduct a case study on color. Indeed, color perception in
humans and its relation to speakers’ use of color terms has long been the
subject of studies in cognitive science Kay and McDaniel (1978); Berlin and
Kay (1991); Regier et al. (2007); Kay et al. (2009). To this end, spaces have
been defined in which Euclidean distances between related colors are
correlated with reported perceptual differences.111The differences between
color stimuli which are perceived by human observers. In addition, the
semantics of color terms have long been understood to hold particular
linguistic significance, as they are theorised to be subject to universal
constraints that arise directly from the neurophysiological mechanisms and
properties underlying visual perception and cognition Kay and McDaniel (1978);
Berlin and Kay (1991); Kay et al. (1991).222These theories have been contested
by work arguing for linguistic relativism (cf. the Sapir–Whorf Hypothesis),
which emphasizes the arbitrariness of language and the relativity of semantic
structures and minimizes the role of universals. Such critiques have, however,
been accommodated for in the Berlin & Kay paradigm Berlin and Kay (1991), the
basic assumptions of which, such as the existence of at least some
perceptually-determined universal constraints on color naming, remain widely
accepted. Due to these factors, color offers a useful test-bed for
investigating whether or not structural information about the topology of the
perceptual world might be encoded in linguistic representations.

To explore this in detail, we employ a dataset of English color terms and
their corresponding color chips333Each chip is a unique color sample from the
Munsell chart, which is made up of $330$ such samples which cover the space of
colors perceived by humans. See §2., the latter of which are represented in
CIELAB — a perceptually uniform color space. In addition to the color chip
CIELAB coordinates, we extract linguistic representations for the
corresponding color terms. With these two representations in mind (see Figure
1 for a demonstrative plot from our experiments), we employ two methods of
measuring structural correspondence, with which we evaluate the alignment
between the two spaces. Figure 2 shows an illustration of the experimental
setup. We find that the structures of various language model representations
show alignment with the structure of the CIELAB space, demonstrating that some
approximation of perceptual color space topology can indeed be learned from
text alone. We also show that part of this distributional signal is learnable
by simple models — e.g. models based on pointwise mutual information (PMI)
statistics — although large-scale language model pretraining (e.g., BERT)
encodes the topology markedly better.

Analysis shows that larger language models align better than smaller ones and
that much of the variance in CIELAB space can be explained by low-dimensional
subspaces of LM-induced color term representations. To better understand the
results, we also analyse the differences in alignment across the color
spectrum, observing that warm colors are generally better aligned than cool
ones. Further investigation reveals a connection to findings reported in work
on communication efficiency in color naming, which posits that warmer colors
are communicated more efficiently. Finally, we investigate various corpus
statistics which could influence alignment, finding that a measure of color
term collocationality based on PMI statistics corresponds to lower alignment,
while the entropy of a color term’s dependency relation distribution (i.e.
terms occurring as adjectival modifiers, nominal subjects, etc.) and how often
it occurs as an adjectival modifier correspond to a stronger one.

##  2 Methodology

### Color data

We employ the Color Lexicon of American English, which provides extensive data
on color naming. The lexicon consists of $51$ monolexemic color name
judgements for each of the $330$ Munsell Chart color
chips444http://www1.icsi.berkeley.edu/wcs/images/jrus-20100531/wcs-
chart-4x.png Lindsey and Brown (2014). The color terms are solicited through a
free-naming task, resulting in $122$ terms.

### Perceptual color space

Following previous work Regier et al. (2007); Zaslavsky et al. (2018);
Chaabouni et al. (2021), we map colors to their corresponding points in the
$3$D CIELAB space, where the first dimension $L$ expresses lightness, the
second $A$ expresses position between red and green, and the third $B$
expresses the position between blue and yellow. Distances between colors in
the space correspond to their perceptual difference.

### Language models

Our analysis is conducted on three widely used language models (LMs): BERT
Devlin et al. (2019) and RoBERTa Liu et al. (2019), both of which employ a
masked language modelling objective, and ELECTRA Clark et al. (2020), which is
trained instead with a discriminative token replacement detection
objective.555bert-large-uncased; roberta-large; electra-large-discriminator

### Baselines

In addition to the aforementioned language models, we consider two different
baselines:

  * •

PMI statistics, which are computed666Using Hyperwords:
https://bitbucket.org/omerlevy/hyperwords for the color terms in common crawl,
using window sizes of $1$ (pmi-1), $2$ (pmi-2), and $3$ (pmi-3). The result is
a vocabulary length vector quantifying the likelihood of co-occurrence of the
color term with every other vocabulary item in within that window.

  * •

Word-type FastText embeddings trained on Common Crawl Bojanowski et al.
(2017).

### Representation Extraction

We follow Bommasani et al. (2020) and Vulić et al. (2020) in defining
configurations for the extraction of word-type representations from LM hidden
states. In the first configuration (NC), a color term is encoded without
context, with the appropriate delimiter tokens attached (e.g. [CLS] red [SEP]
for BERT). In the second, $S$ sentential contexts that include the color term
are encoded and the hidden states representing these contexts are mean pooled.
These $S$ contexts are either randomly sampled from common crawl (RC), or
deterministically generated to allow for control over contextual variation
(CC). If a color term is split by an LM’s tokenizer into more than one token,
subword token encodings are averaged over. For each color term and
configuration, an embedding vector of hidden state dimension $d_{LM}$ is
extracted per layer, per model.

### Controlled context

To control for the effect of variation in the sentence contexts used to
construct color term representations, we employ a templative approach to
generate a set of identical contexts for all color terms. When generating
controlled contexts, we create three frames in which the terms can appear:

  * •

Copula: the <obj> is <col>

  * •

Possession: i have a <col> <obj>

  * •

Spatial: the <col> <obj> is there

We use these frames in order to limit the contextual variation across colors
(<col>) and to isolate their representations amidst as little semantic
interference as possible, all while retaining a naturalistic quality to the
input. We also aggregate over numerous object nouns (<obj>), which the color
terms are used to describe. We select objects from the McRae et al. (2005)
data which are labelled in the latter as plausibly occurring in many colors
and which are stratified across $13$ category sets, e.g. fan $\in$ APPLIANCES,
skirt $\in$ CLOTHING, etc. Collapsing over categories, we generate sentences
combinatorially across frames, objects and color terms, resulting in $3\times
122\times 18=6588$ sentences, $366$ per term.

##  3 Evaluation

We employ two complimentary evaluation methods to gauge the correspondence of
the color term text-derived representations to the perceptual color space. The
first, Representation Similarity Analysis (RSA), is non-parametric and uses
pairwise comparisons of stimuli to provide a measure of the global topological
alignment between two spaces. The second employs a learned linear mapping,
evaluating the extent to which two spaces can be aligned via transformation
(rotation, scaling, etc.).

### RSA

(Kriegeskorte et al., 2008) is a method of relating different representational
modalities, which was first employed in neuroscientific studies. RSA abstracts
away from activity patterns themselves (e.g. neuron values in representational
vectors) and instead computes representational (dis)-similarity matrices
(RSMs), which characterize the information carried by a given representation
method through global (dis)-similarity structure. Kendall’s rank correlation
coefficient ($\tau$) is computed between RSMs derived from the two spaces,
providing a summary statistic indicative of the overall representational
alignment between them. RSA is non-parametric and therefore circumvents many
of the various methodological weaknesses associated with the probing paradigm
Belinkov (2021).

For each color term, we compute a centroid in the CIELAB space following the
approach described in Lindsey and Brown (2014). Each centroid is defined as
the average CIELAB coordinate of the samples (i.e. color chips) that were
named with the corresponding term (across the 51 subjects). This results in
$N$ parallel points in the color term embedding and perceptual color spaces,
where $N$ is the number of color terms considered. For our analysis, we
exclude color terms used less frequently than a cutoff $f=100$ in the color
lexicon, leaving us with the $18$ most commonly used color terms.777This
includes all color terms which are considered ”basic” (red, blue, etc.), and
commonly used ”derived” terms (pink, gray, turquoise, maroon, etc.), but
excludes the rest which are only infrequently used as color terms (forest,
puke, dew, seafoam, etc.). See appendix A for full list of colors included. We
then separately construct an $N\times N$ RSM for each of the LM spaces and for
CIELAB . Each cell in the RSM corresponds to the similarity between the
activity patterns associated with pairs of experimental conditions
$n_{i},n_{j}\in N$.

For the color term embedding space, we employ Pearson’s correlation
coefficient ($r$) as a similarity measure between each pair of embeddings
$n_{i},n_{j}\in N$. For the CIELAB space, we elect to use the following
method, per Regier et al.’s (2007) suggestion:
$sim(n_{i},n_{j})=\exp(-c\times[dist(n_{i},n_{j})]^{2})$, where $c$ is a
scaling factor (set to $0.001$ in all experiments reported here) and
$dist(n_{i},n_{j})$ is the CIELAB distance ($\Delta$ E_CMC∗)888We use the
colormath Python package, setting illuminant to C, and assuming 2 degree
standard observer. between chips $n_{i}$ and $n_{j}$. This similarity measure
is derived from the psychological literature on categorization and is meant to
model the assumption that beyond a certain distance colors appear entirely
different, so that increasing the distance has no further effect on
dissimilarity. Finally, we report the mean Kendall’s $\tau$ between the color
term embedding and color space RSMs. We also report $\tau$ per color term
(i.e. per row in the RSM), which corresponds to how well-aligned each
individual color term is.

### Linear mapping

We train regularised linear regression models to map from color term embedding
space $X\in\mathbb{R}^{n\times d_{LM}}$ to CIELAB space
$Y\in\mathbb{R}^{n\times 3}$, minimising $\mathcal{L}(W;\alpha)=\left\lVert
XW-Y\right\rVert_{2}^{2}+\alpha\left\lVert W\right\rVert_{1}$, where
$W\in\mathbb{R}^{3\times d_{LM}}$ is a linear map and $\alpha$ is the lasso
regularization hyper parameter. We vary $\alpha$ across a wide range of
settings to examine the effect of probe complexity, which we measure using the
nuclear norm of the linear projection matrix
$W\in\mathbb{R}^{\phi\times\iota}$;
$||W||_{*}=\sum_{i=1}^{min(\phi,\iota)}\sigma_{i}(W)$, where $\sigma_{i}(W)$
is the $i$th singular value of $W$ Pimentel et al. (2020). The fitness of the
regressors, evaluated using $n$-fold cross-validation ($n=6$) indicates the
alignability of the two spaces, given a linear transformation. Centroids
corresponding to each Munsell color chip are computed in the color term
embedding space via the weighted mean of the embeddings of the 51 terms used
to label it. As in the RSA experiments, terms occurring less frequently than
the cutoff ($f=100$) are excluded. For evaluation, we compute the average
(across splits and datapoints) proportion of explained variance as well as the
ranking of a predicted color term embedding according to the Pearson distance
($1-r$) to gold.

| NC | RC | CC
---|---|---|---
Model | RSA | lin. map | RSA | lin. map | RSA | lin. map
max | mean | max | mean | max | mean | max | mean | max | mean | max | mean
BERT | 0.16∗ | 0.01±0.09 | 0.75 | 0.73±0.01 | 0.26† | 0.20±0.03 | 0.74 | 0.73±0.08 | 0.24† | 0.19±0.03 | 0.76 | 0.75±0.05
RoBERTa | 0.33§ | 0.02±0.11 | 0.75 | 0.73±0.01 | 0.20∗ | 0.14±0.04 | 0.74 | 0.73±0.01 | 0.19∗ | 0.14±0.04 | 0.77 | 0.76±0.09
ELECTRA | 0.13${}^{\phantom{\S}}$ | 0.01±0.08 | 0.75 | 0.64±0.13 | 0.25† | 0.19±0.05 | 0.75 | 0.73±0.01 | 0.23† | 0.16±0.04 | 0.78 | 0.76±0.01

Table 1: Results for the RSA experiments show max and mean (across layers)
Kendall’s $\tau$; correlations that are significantly non-zero are marked with
*, $\dagger$ and $\S$ for $p<0.05$, $<0.01$ and $<0.001$ respectively. Results
for the linear mapping experiments show max and mean selectivity.

### Control task

As proposed by Hewitt and Liang (2019), we construct a random control task for
the linear mapping experiments, wherein we randomly swap each color chip’s
CIELAB code for another. This is meant to break the mapping between the color
chips and their corresponding terms. Control task results are reported as the
mean of 10 different random re-mappings. We report probe selectivity, which is
defined as the difference between proportion of explained variance in the
standard experimental condition and in the control task Hewitt and Liang
(2019). We run similar control for the RSA experiments, where the CIELAB space
centroids are randomly shuffled.

##  4 Results

Figure 3: RSA results (Kendal’s $\tau$) broken down by color term for each of
the LMs under the CC configuration and for the fastText baseline.

Table 1 shows the max, mean, and standard deviation (across layers) of
alignment scores for each of the LMs, per alignment method and setting. For
RSA, we observe significant correlations across all configurations: most LM
layers show a topological alignment with color space. Notably, this is also
true for the static embeddings and for one of the PMI baselines (Table 2).
Although some variance is observed,999In particular, results for NC show large
variances across layers. The mean correlation across layers in this setup is
near zero, even though max correlations for BERT and RoBERTa are significant;
this is unsurprising, however, as the LM has likely never encountered single
color term tokens in isolation (cf. Bommasani et al. (2020)) the presence of
significant correlations is telling, given the small sample size (18).
Furthermore, randomly permuting the color space centroids leads to RSA
correlations that are non-significant for all setups ($p>0.05$), which lends
further credence to models’ alignment with CIELAB structure.

Figure 3 shows the breakdown of correlations per color term for the three LMs
under CC, as well as for fastText. We find that this ranking of color terms is
largely stable across models and layer. Full RSMs for all models and CIELAB
are in appendix 9. The RSMs show evidence of the higher correlations for
colors like violet, orange, and purple, being driven by general clusterings of
similarity/dissimilarity. For instance, for both the CIELAB and CC BERT RSMs,
violet’s top nearest neighbors include purple, lavender, pink, and orange, and
its furthest neighbors include aqua, olive, black, and gray. Correlations do
not, however, appear to be driven by consistently aligned partial orderings
within the clusters. In addition, we compute RSA correlations between the
different models. Results show that NC embeddings have low alignment to all
others (details in appendix B).

For the linear mapping experiments, we observe the highest selectivity scores
for CC (Table 1, right) compared to NC and RC (Table 1, left, middle) and
baselines (Table 2). This validates our intuition that controlling for
variation in sentence context would reveal increased alignment to color space.

Furthermore, we observe that, over the full range of probe complexities for
the experimental condition and the control task (described as in §3), all
models demonstrate high selectivity (see G for full results). It is,
therefore, safe to attribute the fitness of the probes to information encoded
in the color term representations, rather than to memorization. In terms of
individual colors, Figure 4(a) depicts the ranking of predicted CIELAB codes
per Munsell color chip for BERT (CC). We find that these results are largely
stable across models and layers (see appendix F for full set of results and
for reference chart). Also, we observe that clusterings of chips with certain
modal color terms (green, blue) show worse rankings than the rest.

Model | RSA | lin. map
---|---|---
pmi-1 | 0.14${}^{\phantom{*}}$ | 0.72
pmi-2 | 0.11${}^{\phantom{*}}$ | 0.70
pmi-3 | 0.17∗ | 0.71
fastText | 0.23∗ | 0.72
Table 2: Baseline results. RSA results show Kendall’s $\tau$; results with *
are significantly non-zero ($p<0.05$). Linear mapping results show
selectivity.

(a) Each circle on the chart represents the ranking of the predicted color
chip when ranked according to Pearson distance from gold (larger circle
$\cong$ higher/better ranking).

(b) Each circle on the chart represents a color chip’s suprisal score (larger
circle $\cong$ higher score).

Figure 4: (a) shows linear mapping results for BERT, under the CC
configuration, broken down by Munsell color chip; (b) shows suprisal per chip.
Circle colors reflect the modal color term assigned to the chips.

##  5 Analysis and Discussion

Having demonstrated the existence of models’ alignment to CIELAB across
various configurations, we now present an analysis and discussion of these
results.

### Dimensionality of color subspace

Previous work has shown that linguistic information such as part-of-speech
category, dependency relation type, and word sense, is expressed in low-
dimensional subspaces of language model representations Reif et al. (2019b);
Durrani et al. (2020); Hernandez and Andreas (2021). We investigate the
dimensionality of the subspace required to predict the CIELAB chip codes from
the term embeddings, following the methodology of Durrani et al. (2020).
Averaging over the three predicted CIELAB dimensions, we rank the linear
mapping coefficients (from the experiments described in §2), sorting the
weights by their absolute values in descending order. Results (appendix H)
show that across models and layers, ${\sim}0.4$ of the variance in the CIELAB
chip codes can be explained by assigning $95\%$ of the weights to ${\sim}10$
dimensions. $3040$ dimensions are sufficient to explain ${\sim}0.7$ of the
variance, nearly the proportion of variance explained by the full
representations (Table 1).

Model | RSA max | RSA mean | lin. map.. max | lin. map. mean
---|---|---|---|---
BERT-mini | 0.077 | 0.043 $\pm$ 0.340 | 0.729 | 0.582 $\pm$ 0.291
BERT-small | 0.106 | 0.070 $\pm$ 0.191 | 0.734 | 0.598 $\pm$ 0.294
BERT-medium | 0.097 | 0.057 $\pm$ 0.035 | 0.739 | 0.654 $\pm$ 0.221
BERT-base | 0.162∗ | 0.092 $\pm$ 0.058 | 0.740 | 0.677 $\pm$ 0.182

Table 3: Results for the four smaller BERT models. RSA results (left) show max
and mean (across layers) Kendall’s correlation coefficient ($\tau$).
Correlations that are significantly non-zero are indicated with: * : $p<0.05$.
Results for the Linear Mapping experiments (right) show max and mean
selectivity. Standard deviation across layers is included with the mean
results.

### Effect of model size

We also evaluate the effect of model size on alignment by testing four smaller
BERT (CC) models101010for details see appendix I using the same setup
described above. The results (table 3) show that alignment as measured by both
RSA and linear mapping progressively increases with model size, meaning that
that with growing complexity, model representational geometry of color terms
moves towards isomorphism to CIELAB.

### Color temperature

In Figures 3 & 4(a) we observe that on average, warmer colors (yellow, orange,
red, etc.) show a closer alignment than cooler ones (blue, green, etc.). In
recent work, Gibson et al. (2017) reported that the former are on average
communicated more efficiently (see next paragraph) than the latter, across
languages. This is attributed to warmer colors being more prevalent as colors
of behaviorally relevant items in the environment — salient objects — compared
to cooler ones, which occur more often as background colors. To verify this
observation, we partition the space of chips into two (see appendix D for
details) and compute the average explained variance across warm and cool
colors. The results (see appendix D for plots) show that, term embeddings of
warm colors are better aligned to CIELAB than those of cool ones, across
models and configurations. This is consistent with the bias described in
Gibson et al. (2017), which we conjecture might be filtering through into the
distributional statistics of (color terms in) textual corpora, influencing the
representations learned by various methods which leverage these statistics.

### Connection to listener surprisal

Gibson et al. (2017)’s findings are based on the application of an information
theoretic analysis to color naming, framing it as a communication game where a
speaker has a particular color chip $c$ in mind and uses a word $w$ to
indicate it then a listener has to correctly guess $c$, given $w$.
Communication efficiency is measured through surprisal, $S$, which in this
setting corresponds to the average number of guesses an optimal listener takes
to arrive at the correct color chip. We calculate $S(c)$ for each chip in the
color lexicon. Surprisal is defined as
$S(c)=\sum_{w}P(w|c)\cdot\log\left(\frac{1}{P(c|w)}\right)$, where $P(w|c)$ is
the probability that a color c gets labeled as $w$ and $P(c|w)$ is computed
using Bayes Theorem. Here, $P(w)$ represents how often a particular word gets
used across the color space (and participants), and $P(c)$ is a uniform prior.
Figure 4(b) shows surprisal per chip. High surprisal chips correspond to a
lower color naming consensus among speakers, meaning that a more variable
range of terms is used for these (color) contexts. We hypothesize that this
could be reflected in the representations of color terms corresponding to high
surprisal chips. To test this, we compute Spearman’s correlation ($\rho$)
between a chip’s regression score (predicted color chip code ranking) and its
surprisal. We find significant Spearman’s rank correlation between lower
ranking and higher surprisal for all LMs under all configurations
($0.12\leq\rho\leq 0.17$, $p<0.05$).

### What factors predict color space alignment?

Given that LMs are trained exclusively on text corpora, we hypothesize that
alignment between their embeddings and CIELAB is influenced by corpus usage
statistics. To determine which factors could predict alignment score, we
extract color term log frequency, part-of-speech tag (POS), dependency
relation (DREL), and dependency tree head (HEAD) statistics for all color
terms from a dependency-parsed Straka et al. (2016) common crawl corpus. In
addition to this, we compute, per color term, the entropy of its normalised
PMI distribution (pmi-col, see §2) as a measure of collocation.111111Low
entropy reflects frequent co-occurrence with a small subset of the vocabulary
and high entropy the converse. We then fit a Linear Mixed Effects Model
Gałecki and Burzykowski (2013) to the features listed above, with RSA score
(Table 1) as the response variable, and model type as a random effect.

We follow a multi-level step-wise model building sequence, where a baseline
model is first fit with color term log frequency as a single fixed effect. A
model which includes pmi-col as an additional fixed effect is then fit, and
these two terms are included as control predictors in all later models.
Following this, we compute POS, DREL, and HEAD lemma distribution entropies
per color term (pos-ent, deprel-ent, head-ent). Higher entropies indicate that
the term is employed in more diverse contexts with respect to those
categories. Following entropy computation, we separately fit models including
each three entropy statistic features. Finally, we calculate the proportion
of: POS tags that are adjectives, adj-prop; DRELs that are adjectival
modifiers, amod-prop; and those that are copulas, cop-prop. The first two
evaluate the effect of a color term occurring more or less often as an
adjectival modifier, while the latter tests the hypothesis that assertions
such as The banana is yellow could provide indirect grounding Merrill et al.
(2021), thereby leading to higher alignment. Including the entropy term which
led to the best fit (deprel-ent) in the previous level, models are fit
including terms for each of the proportion statistics. Model comparison is
carried out by computing the log likelihood ratio between models that differ
in a single term. See appendix J for model details.

Results show that:

  * •

pmi-col significantly improves fit above log frequency and has a negative
coefficient, meaning that terms that occur in more fixed collocations are less
aligned to the perceptual space. Intuitively, this makes sense as the color
terms in many collocations such as e.g. Red Army or Black Death are employed
in contexts which are largely metaphorical rather than attributive or
descriptive.

  * •

deprel-ent and head-ent (but not pos-ent) lead to a significantly improved fit
compared to the control predictors; we observe positive coefficients for both,
indicating RSA score is higher for terms that occur in more varied syntactic
dependency relations and modify a more diverse set of syntactic heads. This
suggests that occurring in a more diverse set of contexts might be beneficial
for robust representation learning, in correspondence with the idea of sample
diversity in the active learning literature Brinker (2003); Yang et al.
(2015). pos-ent’s lack of significance, on the other hand, indicates that the
degree of specification offered by the POS tagset might be too coarse to
meaningfully differentiate between color terms, e.g. nouns can occur in a
variety of DRELs such as subjects, objects, oblique modifiers (per the
Universal Dependecies Nivre et al. (2020)).

  * •

out of the proportion statistics, only the amod-prop term improves fit; it has
a positive coefficient, thus color terms occurring more frequently as
adjectival modifiers show higher scores. adj-prop is not significant,
providing further evidence for the POS tagset’s level of granularity being too
coarse. Finally, as cop-prop is not significant, it appears that occurring
more frequently in assertion-like copula constructions does not confer an
advantage in terms of alignment to perceptual structure.

### Vision-and-Language models

In a preliminary set of experiments, we evaluated multi-modal Vision-and-
Language models (VisualBERT Li et al. (2019) and VideoBERT Sun et al. (2019)),
finding no major differences in results from the text-only models presented in
this study.

##  6 Related Work

Distributional word representations have long been theorized to capture
various types of information about the world Schütze (1992). Early work in
this regard employed semantic similarity and relatedness datasets to measure
alignment to human judgements Agirre et al. (2009); Bruni et al. (2012); Hill
et al. (2015). Rubinstein et al. (2015), however, question whether the
distributional hypothesis is equally applicable to all types of semantic
information, finding that taxonomic properties (such as animacy) are better
modelled than attributive ones (color, size, etc.). To a similar end, Lucy and
Gauthier (2017) analyze how well distributional representations encode various
aspects of grounded meaning. They investigate whether language models would
“be worse off for not having physically bumped into walls before they hold
discussions on wall-collisions?”, finding that perceptual features are poorly
modelled compared to encyclopedic and taxonomic ones.

More recently, several studies have asked related questions in the context of
language models. For example, Davison et al. (2019) and Petroni et al. (2019)
mine LMs for factual and commonsense knowledge by converting knowledge base
triplets into cloze statements that are used to query the models. In a similar
vein, Forbes et al. (2019a) investigate LM representations’ encoding of object
properties (e.g., oranges are round), and affordances (e.g. oranges can be
eaten), as well as the interplay between the two. Weir et al. (2020)
demonstrate that LMs can capture stereotypic tacit assumptions about generic
concepts, showing that they are adept at retrieving concepts given their
associated properties (e.g., bear given A ___ has fur, is big, and has
claws.). Similar to other work, they find that LMs better model encyclopedic
and functional properties than they do perceptual ones.

In an investigation of whether or not LMs are able to overcome reporting bias,
Shwartz and Choi (2020) extract all sentences in Wikipedia where one of $11$
color terms modifies a noun and test how well predicted the color term is when
it is masked. They find that LMs are able to model this relationship between
concepts and associated colors to a certain extent, but are prone to over-
generalization. Finally, Ilharco et al. (2020) train a probe to map LM
representations of textual captions to paired visual representations of image
patches, in order to evaluate how useful the former are for discerning between
different visual representations. They find that many recent LMs yield
representations that are effective at retrieving semantically-aligned image
patches, but still far under-perform humans.

##  7 Outlook

It is commonly held that the learning of phenomena which rely on sensory
perception is only possible through direct experience. Indeed, the view that
people born blind could not be expected to acquire coherent knowledge about
colors has been prevalent since at least the empiricist philosophers Locke
(1847); Hume (1938) and still holds currency Jackson (1982). Nevertheless,
recent research highlighting the contribution of language and of semantic
associations between concepts towards learning has demonstrated that the
congenitally blind do in fact show a striking understanding of both color
similarity Saysani et al. (2018) and object colors Kim et al. (2020).

This paper investigated whether representations of color terms that are
derived from text only express a degree of isomorphism to the structure of
humans’ perceptual color space.121212Clearly, complete isomorphism is rather
unlikely: language in general, and color terms by extension, are far from
being simply denotational, and language interacts with and is influenced by a
myriad of factors besides perception. Results from our experiments evidenced
that such a topological correspondence exists. Notably, color term
representations based on simple co-occurance statistics already demonstrated
correspondence; those extracted from language models aligned more closely. We
observed that warm colors, on average, show more alignment than cooler ones,
linking to recent findings on communication efficiency in color naming Gibson
et al. (2017).

Further analysis based on surprisal — an information theoretic measure, used
to evaluate how efficiently a color is communicated between a speaker and a
listener — revealed a correlation between lower topological alignment and
higher color chip surprisal, suggesting that the kind of contexts a color
occurs in play a role in determining alignment. Exploring this, we tested a
set of color term corpus-derived statistics for how well they predict
alignment, finding that a measure of a color term’s collocationality
corresponds to lower alignment, while the entropy of its dependency relation
distribution and it occurring more frequently as and adjectival modifier
correspond to closer alignment.

Our results and analyses present empirical evidence of topological alignment
between text-based color term representations and perceptual color spaces.
With respect to the debate started by Bender and Koller (2020), we hope that
this work offers a modest step towards furthering our understanding of the
kinds of “meaning” we expect language models to acquire, with and without
grounded or embodied learning approaches, and that it will provide motivation
for further work in this direction.

## Acknowledgements

We would like to thank Vinit Ravishankar and Mitja Nikolaus for their feedback
and comments. Mostafa Abdou and Anders Søgaard are supported by a Google
Focused Research Award and a Facebook Research Award.

## References

  * Agirre et al. (2009) Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. 2009.  A study on similarity and relatedness using distributional and wordnet-based approaches.
  * Belinkov (2021) Yonatan Belinkov. 2021.  Probing classifiers: Promises, shortcomings, and alternatives.  _arXiv preprint arXiv:2102.12452_.
  * Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020.  Climbing towards NLU: On meaning, form, and understanding in the age of data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5185–5198, Online. Association for Computational Linguistics.
  * Berlin and Kay (1991) Brent Berlin and Paul Kay. 1991.  _Basic color terms: Their universality and evolution_.  Univ of California Press.
  * Bisk et al. (2020) Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. 2020.  Experience grounds language.  _arXiv preprint arXiv:2004.10151_.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ , 5:135–146.
  * Bommasani et al. (2020) Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020.  Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4758–4781, Online. Association for Computational Linguistics.
  * Brinker (2003) Klaus Brinker. 2003.  Incorporating diversity in active learning with support vector machines.  In _Proceedings of the 20th international conference on machine learning (ICML-03)_ , pages 59–66.
  * Bruni et al. (2012) Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012.  Distributional semantics in technicolor.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 136–145.
  * Chaabouni et al. (2021) Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. 2021.  Communicating artificial neural networks develop efficient color-naming systems.  _Proceedings of the National Academy of Sciences_ , 118(12).
  * Chastrette (1997) M Chastrette. 1997.  Trends in structure-odor relationship.  _SAR and QSAR in Environmental Research_ , 6(3-4):215–254.
  * Chen (1982) Lin Chen. 1982.  Topological structure in visual perception.  _Science_ , 218(4573):699–700.
  * Chronis and Erk (2020) Gabriella Chronis and Katrin Erk. 2020.  When is a bishop not like a rook? when it’s like a rabbi! multi-prototype BERT embeddings for estimating semantic relationships.  In _Proceedings of the 24th Conference on Computational Natural Language Learning_ , pages 227–244, Online. Association for Computational Linguistics.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_.
  * Davison et al. (2019) Joe Davison, Joshua Feldman, and Alexander Rush. 2019.  Commonsense knowledge mining from pretrained models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1173–1178, Hong Kong, China. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Durrani et al. (2020) Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. 2020.  Analyzing individual neurons in pre-trained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4865–4880, Online. Association for Computational Linguistics.
  * Ennis and Zaidi (2019) Robert J Ennis and Qasim Zaidi. 2019.  Geometrical structure of perceptual color space: mental representations and adaptation invariance.  _Journal of vision_ , 19(12):1–1.
  * Forbes et al. (2019a) Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019a.  Do neural language representations learn physical commonsense?  _arXiv preprint arXiv:1908.02899_.
  * Forbes et al. (2019b) Maxwell Forbes, Christine Kaeser-Chen, Piyush Sharma, and Serge Belongie. 2019b.  Neural naturalist: Generating fine-grained image comparisons.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 708–717, Hong Kong, China. Association for Computational Linguistics.
  * Gałecki and Burzykowski (2013) Andrzej Gałecki and Tomasz Burzykowski. 2013.  Linear mixed-effects model.  In _Linear Mixed-Effects Models Using R_ , pages 245–273. Springer.
  * Gibson et al. (2017) Edward Gibson, Richard Futrell, Julian Jara-Ettinger, Kyle Mahowald, Leon Bergen, Sivalogeswaran Ratnasingam, Mitchell Gibson, Steven T Piantadosi, and Bevil R Conway. 2017.  Color naming across languages reflects color use.  _Proceedings of the National Academy of Sciences_ , 114(40):10785–10790.
  * Hernandez and Andreas (2021) Evan Hernandez and Jacob Andreas. 2021.  The low-dimensional linear geometry of contextualized word representations.  _arXiv preprint arXiv:2105.07109_.
  * Hewitt and Liang (2019) John Hewitt and Percy Liang. 2019.  Designing and interpreting probes with control tasks.  _arXiv preprint arXiv:1909.03368_.
  * Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015.  Simlex-999: Evaluating semantic models with (genuine) similarity estimation.  _Computational Linguistics_ , 41(4):665–695.
  * Hume (1938) David Hume. 1938.  _An Abstract of a Treatise of Human Nature, 1740_.  CUP Archive.
  * Ilharco et al. (2020) Gabriel Ilharco, Rowan Zellers, Ali Farhadi, and Hannaneh Hajishirzi. 2020.  Probing text models for common ground with visual representations.  _arXiv preprint arXiv:2005.00619_.
  * Jackson (1982) Frank Jackson. 1982.  Epiphenomenal qualia.  _The Philosophical Quarterly (1950-)_ , 32(127):127–136.
  * Kay et al. (2009) Paul Kay, Brent Berlin, Luisa Maffi, William R Merrifield, and Richard Cook. 2009\.  _The world color survey_.  CSLI Publications Stanford, CA.
  * Kay et al. (1991) Paul Kay, Brent Berlin, and William Merrifield. 1991.  Biocultural implications of systems of color naming.  _Journal of Linguistic Anthropology_ , 1(1):12–25.
  * Kay and McDaniel (1978) Paul Kay and Chad K McDaniel. 1978.  The linguistic significance of the meanings of basic color terms.  _Language_ , pages 610–646.
  * Kim et al. (2020) Judy Sein Kim, Brianna Aheimer, Verónica Montané Manrara, and Marina Bedny. 2020.  Shared understanding of color among congenitally blind and sighted adults.
  * Kriegeskorte et al. (2008) Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. 2008.  Representational similarity analysis-connecting the branches of systems neuroscience.  _Frontiers in systems neuroscience_ , 2:4.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019\.  Visualbert: A simple and performant baseline for vision and language.  _arXiv preprint arXiv:1908.03557_.
  * Lindsey and Brown (2014) Delwin T Lindsey and Angela M Brown. 2014.  The color lexicon of american english.  _Journal of vision_ , 14(2):17–17.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Locke (1847) John Locke. 1847.  _An essay concerning human understanding_.  Kay & Troutman.
  * Lucy and Gauthier (2017) Li Lucy and Jon Gauthier. 2017.  Are distributional representations ready for the real world? evaluating word vectors for grounded perceptual meaning.  _arXiv preprint arXiv:1705.11168_.
  * McRae et al. (2005) Ken McRae, George S Cree, Mark S Seidenberg, and Chris McNorgan. 2005.  Semantic feature production norms for a large set of living and nonliving things.  _Behavior research methods_ , 37(4):547–559.
  * Merrill et al. (2021) William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A Smith. 2021.  Provable limitations of acquiring meaning from ungrounded form: What will future language models understand?  _arXiv preprint arXiv:2104.10809_.
  * Nivre et al. (2020) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020.  Universal Dependencies v2: An evergrowing multilingual treebank collection.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 4034–4043, Marseille, France. European Language Resources Association.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Pimentel et al. (2020) Tiago Pimentel, Naomi Saphra, Adina Williams, and Ryan Cotterell. 2020.  Pareto probing: Trading off accuracy for complexity.  _arXiv preprint arXiv:2010.02180_.
  * Provenzi (2020) Edoardo Provenzi. 2020.  Geometry of color perception. part 1: structures and metrics of a homogeneous color space.  _The Journal of Mathematical Neuroscience_ , 10(1):1–19.
  * Regier et al. (2007) Terry Regier, Paul Kay, and Naveen Khetarpal. 2007.  Color naming reflects optimal partitions of color space.  _Proceedings of the National Academy of Sciences_ , 104(4):1436–1441.
  * Reif et al. (2019a) Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019a.  Visualizing and measuring the geometry of bert.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Reif et al. (2019b) Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019b.  Visualizing and measuring the geometry of bert.  _Advances in Neural Information Processing Systems_ , 32:8594–8603.
  * Rossiter (1996) Karen J Rossiter. 1996.  Structure- odor relationships.  _Chemical reviews_ , 96(8):3201–3240.
  * Rubinstein et al. (2015) Dana Rubinstein, Effi Levi, Roy Schwartz, and Ari Rappoport. 2015.  How well do distributional models capture different types of semantic knowledge?  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 726–730.
  * Saysani et al. (2018) Armin Saysani, Michael C Corballis, and Paul M Corballis. 2018.  Colour envisioned: Concepts of colour in the blind and sighted.  _Visual Cognition_ , 26(5):382–392.
  * Schütze (1992) Hinrich Schütze. 1992.  Dimensions of meaning.  In _SC_ , pages 787–796.
  * Shwartz and Choi (2020) Vered Shwartz and Yejin Choi. 2020.  Do neural language models overcome reporting bias? In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6863–6870, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Straka et al. (2016) Milan Straka, Jan Hajic, and Jana Straková. 2016.  Udpipe: trainable pipeline for processing conll-u files performing tokenization, morphological analysis, pos tagging and parsing.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 4290–4297.
  * Sun et al. (2019) Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019.  Videobert: A joint model for video and language representation learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7464–7473.
  * Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.  BERT rediscovers the classical NLP pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4593–4601, Florence, Italy. Association for Computational Linguistics.
  * Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Well-read students learn better: On the importance of pre-training compact models.  _arXiv preprint arXiv:1908.08962_.
  * Vulić et al. (2020) Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020.  Probing pretrained language models for lexical semantics.  _arXiv preprint arXiv:2010.05731_.
  * Weir et al. (2020) Nathaniel Weir, Adam Poliak, and Benjamin Van Durme. 2020.  Probing neural language models for human tacit assumptions.
  * Yang et al. (2015) Yi Yang, Zhigang Ma, Feiping Nie, Xiaojun Chang, and Alexander G Hauptmann. 2015\.  Multi-class active learning by uncertainty sampling with diversity maximization.  _International Journal of Computer Vision_ , 113(2):113–127.
  * Zaslavsky et al. (2018) Noga Zaslavsky, Charles Kemp, Terry Regier, and Naftali Tishby. 2018.  Efficient compression in color naming and its evolution.  _Proceedings of the National Academy of Sciences_ , 115(31):7937–7942.

##  Appendix A List of included color terms

Red, green, maroon, brown, black, blue, purple, orange, pink, yellow, peach,
white, gray, olive, turquoise, violet, lavender, and aqua.

##  Appendix B RSA between models

Figure 5 shows a the result of representation similarity analysis between the
representations derived from all models (and configurations) as well as
CIELAB, showing Kendall’s correlation coefficient between flattened RSMs.

Figure 5: Result of representation similarity analysis between all models (and
configurations), showing Kendall’s correlation coefficient between flattened
RSMs. Results are shown for layers which are maximally correlated with CIELAB,
per model. -rc indicates random-context, -cc indicates controlled-context, and
-nc indicates non-context.

##  Appendix C Representation Similarity Matrices

Figures 6 to 9 show the representation similarity matrices employed for the
RSA analyses, for the layer with the highest RSA score from each of the
controlled-context (CC) models.

Figure 6: CIELAB RSM Figure 7: BERT(CC) RSM Figure 8: RoBERTa(CC) RSM Figure
9: ELECTRA(CC) RSM

##  Appendix D Warm vs. Cool colors

Figures 10 and 11 show Linear Mapping and RSA results broken down by color
temperature. The color space is split according to temperature measured
according to the Hue dimension in the Hue-Value-Saturation
space131313https://psychology.wikia.org/wiki/HSV_color_space.

Figure 10: Linear mapping results (proportion of explained variance) broken
down by color chip temperature for each of the baselines and the LMs. Figure
11: RSA results (Kendall’s $\tau$ ) broken down by color temperature for each
for each of the baselines and the LMs.

##  Appendix E Corpus statistics

Figures 12 and 13 show log frequency and entropy of distributions over part-
of-speech categories, dependency relations, and lemmas of dependency tree
heads of color terms in common crawl.

Figure 12: Log frequency of color terms in common crawl. Figure 13: Entropy of
distributions over part-of-speech categories, dependency relations, and lemmas
of dependency tree heads of color terms in common crawl.

##  Appendix F Linear mapping results by munsell color chip

Figure 14 shows linear mapping results broken down by Munsell chip for all
models and configurations.

Figure 14: Linear mapping results for each of the baselines and language
models, under all extraction configurations, broken down by Munsell color
chip. Each circle on the chart represents the ranking of the predicted color
chip when ranked according to Pearson distance ($1-$ Pearson’s $r$) from gold
– the larger the circle, the higher (better) the ranking. Circle colors
reflect the modal color term assigned to the chips in the lexicon. Reference
plot showing modal color of all chips also included.

##  Appendix G Linear mapping control task and probe complexity

Figure 15 shows the full results over a range of probe complexities for the
standard experimental condition as well the random control task.

Figure 15: Explained variance for the linear probes trained on the normal
experimental condition (blue) and the control task (red) where color terms are
randomly permuted. The means are indicated by the lines and standard deviation
across layers is indicated by the bands.

##  Appendix H Dimensionality of color subspace

Figure 16 shows the proportion of explained variance with respect to the
number of dimensions which are assigned $95\%$ of the linear regression
coefficient weights.

Figure 16: The y-axis shows explained variance for the linear probes. The
means are indicated by the lines and standard deviation across layers is
indicated by the bands. The x-axis shows the number of regression matrix
coefficients assigned $95\%$ of the weight.

##  Appendix I Effect of model size

Our model size experiments are run using four BERT models of different sizes:
BERT-mini (4 layers, hidden size: 256), BERT-small (4 layers, hidden size:
512), BERT-medium (8 layers, hidden size: 512), and BERT-base (12 layers,
hidden size: 768). Further model specification and training details for the
first three can be found in Turc et al. (2019) and for last in Devlin et al.
(2019).

##  Appendix J Linear Mixed Effects Model

To fit Linear Mixed Effects Models, we use the LME4 package. With model type
(BERT-CC, RoBERTa-NC, etc.) as a random effect, we follow a step-wise model
construction sequence which proceeds along four levels of nesting: (i) in the
first level color log-frequency is the only fixed effect, (ii) in the second
pmi-colloc is added to that, (iii) in the third, each of pos-ent, deprel-ent,
head-ent is added separately to the a model with log frequency and pmi-colloc,
(iv) the term that leads to the best fit from the previous level deprel-ent is
included, then each of the proportion terms adj-prop, amod-prop, cop-prop is
added. The reported regression coefficients are extracted from the minimal
model containing each term.


# Using an Ancillary Neural Network to Capture Weekends and Holidays in an
Adjoint Neural Network Architecture for Intelligent Building Management

Zhicheng Ding, Mehmet Kerem Turkcan, and Albert Boulanger

###### Abstract

The US EIA estimated in 2017 about 39% of total U.S. energy consumption was by
the residential and commercial sectors. Therefore, Intelligent Building
Management (IBM) solutions that minimize consumption while maintaining tenant
comfort are an important component in reducing energy consumption. A
forecasting capability for accurate prediction of indoor temperatures in a
planning horizon of 24 hours is essential to IBM. It should predict the indoor
temperature in both short-term (e.g. 15 minutes) and long-term (e.g. 24 hours)
periods accurately including weekends, major holidays, and minor holidays.
Other requirements include the ability to predict the maximum and the minimum
indoor temperatures precisely and provide the confidence for each prediction.
To achieve these requirements, we propose a novel adjoint neural network
architecture for time series prediction that uses an ancillary neural network
to capture weekend and holiday information. We studied four long short-term
memory (LSTM) based time series prediction networks within this architecture.
We observed that the ancillary neural network helps to improve the prediction
accuracy, the maximum and the minimum temperature prediction and model
reliability for all networks tested.

###### Index Terms:

energy consumption, adjoint neural network, intelligent building management,
time series prediction, multiple-steps ahead prediction.

##  I Introduction

Figure 1: The architecture of the proposed neural network that includes three
subnetworks. A) is a main neural network whose inputs are critical features
and its outputs are the prediction of each input. It includes an LSTM and a
multilayer feed-forward neural network. B) is an ancillary neural network in
which extra features are utilized. This module includes a relatively shallow
neural network. C) is the model weighting part. We use the weighted average of
$\hat{y}_{main}$ and $\hat{y}_{anc}$ to forecast the output. The output
includes the predictions of up to 96 steps (24 hours) in the future. In our
setting, we use a 2-layer LSTM unit, a 7-layer multilayer feed-forward neural
network, and a 4-layer shallow neural network. In total, this model contains
9,349,056 trainable parameters.

With a rapid population growth, the total energy consumption in both
residential and commercial buildings has increased and threatens the
environment of the earth [1]. Developing an energy-saving IBM system helps to
mitigate this problem.

Predicting indoor temperature is the key to building such systems. This is a
difficult task because energy consumption is influenced by many different
factors such as occupancy, outside weather, solar load, and the building’s
construction [2]. As a result, IBM has become a popular research topic [3] in
recent years.

Recent literature on IBM focuses on models utilizing artificial neural
networks (ANN) since such approaches are powerful in modeling nonlinear
problems which are hard to model by other machine learning approaches [4].
Recurrent neural networks (RNNs) have shown remarkable performance in
predicting when trained with large time series datasets [5]. RNNs work well
for indoor temperature forecasting because the indoor temperature has certain
time-related patterns that for example include daily, weekly, monthly,
seasonally, and yearly patterns. A number of practical studies prove the
efficacy of ANNs for IBM. For example, an ANN model was used to predict the
air temperature of buildings and was found to exhibit the best performance
[6]. Another ANN model that was built for indoor air temperature forecasting
outperformed competing regression methods [7].

However, most of these studies utilize a single ANN model [3] and only a few
studies that combine two or more networks or models have so far been proposed.
In addition, only a small set of studies provide the confidence of the model,
which is important for evaluating a model’s reliability [8]. In addition, the
model should predict the indoor temperature in both short-term and the long-
term time periods [9]. Short-term predictions provide relatively precise
results to the IBM whereas the long-term predictions offer an overview to the
system, allowing the IBM enough foresight to optimize current actions to take
care of temperature control challenges later in the day.

Inspired by ensemble methods and dropout inference [10, 11], we propose a
novel adjoint neural network architecture that uses an ancillary neural
network to capture weekends and holidays to increase the indoor temperature
prediction accuracy for IBM. This architecture also combines the advantages of
LSTMs and multilayer feed-forward neural networks [12]. Our proposed method
addresses the aforementioned issues and outperforms the prediction of a single
model. Meanwhile, it provides predictions for every 15 minutes of 24 hours
with 68% and 95% confidence intervals (CIs). The main contributions of this
paper are summarized as follows.

  * •

We propose an adjoint neural network architecture that uses an ancillary
neural network to capture weekends and holidays to increase the accuracy of
indoor temperature predictions. This architecture enables the prediction of up
to 96 steps (24 hours) and provides confidence for all the predictions.

  * •

We conducted a comprehensive analysis and comparison by applying our proposed
architecture to four popular time series prediction models. The comparison
includes three metrics (average error of prediction, the error of max/min
temperature prediction, and reliability of the prediction) in both one-step
ahead prediction and multiple-steps ahead prediction.

##  II Related Work

###  II-A Artificial Neural Networks

ANNs have achieved a great success in many different fields [13]. A typical
neural network contains many artificial neurons which are known as units.
There are three types of units [14]: input units, hidden units, and output
units. With more and more hidden layers employed, the neural work is more able
to learn deeper abstractions and learns as an artificial brain [3].

Techniques to effectively train ANNs with more than three layers have become
standard. Deep Neural Networks (DNNs) are neural networks with more than one
hidden layer and it usually works better than shallow neural networks [15] for
a particular task.

DNNs have been used in many different domains. In the image processing domain,
DNNs were used to learn graphics representations [16] and estimate human pose
[17]. In the traffic domain, DNNs were used to classify traffic signs [18] and
predict traffic flow [19]. In our IBM domain, DNNs helped predict indoor
temperature [20] and energy consumption [21].

In recent years, other ANN designs have become commonly used. The LSTM and
Bayesian Neural Network (BNN) designs are relevant and are reviewed below.

###  II-B Long Short-Term Memory

LSTM networks [22] have become popular in the recent years for time series
prediction [8]. LSTM networks are a special kind of RNNs which are designed to
solve the long-term memory problem. LSTMs have been used in many time series
problems. For example, networks using LSTMs have been successfully employed in
a number of important problems like speech recognition [23], machine
translation [24], and energy load forecasting [25].

In this paper, we will use a densely connected multilayer feed-forward network
after an LSTM encoder. The multilayer feed-forward neural network utilizes the
temporal information for a robust prediction. We take the advantage of the
encoder in the LSTM layer for extracting temporal information. We feed the
final states of the LSTM encoder to the multilayer feed-forward network.

Figure 2: Sample of one-step ahead prediction. The 68% and 95% CIs are shown
as gray and light gray color bands respectively. The black solid line
represents the ground truth indoor temperature. The gray dash line denotes the
prediction of LSTM-DNN. The indoor temperature shows in a different pattern as
it is on weekdays. As for the data used in our work, Columbus day was the 13th
of October. Figure 3: Comparison of one-step ahead prediction by the LSTM-DNN
without the ancillary network and the adjoint LSTM-DNN. The light gray dash
line represents the prediction by the LSTM-DNN without the ancillary neural
network, and the gray dash line denotes the prediction by the adjoint LSTM-
DNN. The indoor temperature shows in a different pattern as it is on weekdays.
As for the data used in our work, Columbus day was the 13th of October.

(a) RMSE of all predictions
(b) MAE of all predictions
(c) MAPE of all predictions

Figure 4: One-step ahead prediction’s error of RMSE (a), MAE (b), and MAPE (c)
of all predictions.

(a) RMSE of max/min temperature predictions
(b) MAE of max/min temperature predictions
(c) MAPE of max/min temperature predictions

Figure 5: One-step ahead prediction’s error of RMSE (a), MAE (b), and MAPE (c)
of max/min temperature predictions.

(a) Probability of predictions are not within 68% CI

(b) Probability of predictions are not within 95% CI

Figure 6: The monthly probability of one-step ahead predictions that are not
within 68% CI (a) and 95% CI (b). Figure 7: Sample prediction of 96-steps
ahead (24 hours) scenario. The 68% and 95% CIs are shown as gray and light
gray color bands respectively. The black solid line denotes the ground truth
indoor temperature and gray dash line represents prediction by adjoint model.

(a) 96-steps ahead prediction in 19th June

(b) 96-steps ahead prediction in 2nd July

(c) 96-steps ahead prediction in 17th July

(d) 96-steps ahead prediction in 18th September

Figure 8: Comparison of multi-step ahead prediction results between the LSTM-
DNN without the ancillary network and the adjoint LSTM-DNN. The light gray
dash line represents the prediction by the LSTM-DNN without the ancillary
neural network, and the gray dash line denotes the predictions by the adjoint
LSTM-DNN.

##  III Methodology

The complete architecture proposed is shown in Figure 1. The architecture
contains three parts: A) is the main neural network (LSTM followed by a
multilayer feed-forward network) which aims to learn temporal patterns from
significant features. B) is an ancillary neural network which takes in extra
features regarding weekends and holidays using an LSTM followed by relatively
shallow multilayer feed-forward network). C) is the combiner part whose
outputs are the weighted average of the predictions from part (A) and part
(B). The output includes the forecast of one-step ahead (next 15 minutes) and
multi-step ahead (up to 96 steps, 24 hours in total). After that, we add a
dropout unit to infer the CIs of the result. We will discuss each module and
CI in the details below.

###  III-A Main Neural Network

This module aims to learn from the important features. Those features usually
change along with time, such as outside temperature, occupancy, and date.
Also, these features have daily, weekly, monthly, seasonally, and yearly
patterns. Thus, we construct the data as continuous time series input. Those
data pass to the LSTM units which learn temporal information and construct
internal states. The internal states are further propagated to a 7-layer feed-
forward network that forecasts the one-step ahead and 96-steps ahead indoor
temperature $\hat{y}_{main}$.

Given a dataset which includes $S$ timestamps, the model needs to predict the
next $n$ timestamps and for each timestamp, there are $m$ features. Thus, we
construct the input data as a three-dimensional matrix and the size of the
matrix is $S\times n\times m$,

We use a 2-layer LSTM network to extract temporal information and then feed
the temporal information to a 7-layer feed-forward neural network. Next, the
network forecasts the indoor temperature $\hat{y}_{main}$ which contains
96-steps ahead predictions. Since the duration of each time step is 15
minutes, the 96-steps ahead predictions will cover the prediction of every 15
minutes of next 24 hours.

###  III-B Ancillary Neural Network

In the ancillary neural network, we aim to utilize extra features like
weekends and holidays. These features are ancillary because, with enough data,
this information could be learned by the main neural network itself from
significant features fed into the main neural network. But explicitly
providing these features helps increase the performance of the model
especially if the size of the dataset is small. These values of these extra
features are either 0 or 1 which is used as an indicator of being a weekend or
holiday. Then, to reduce the model complexity, those extra features are
learned by a relatively shallow neural network (a 4-layer feed-forward neural
network). The size and dimension of the output $\hat{y}_{anc}$ is exactly the
same as the output from the main neural network.

###  III-C Model Weighting

After the main neural network and ancillary neural network are fully trained,
we use the weighted average [26] and use rectifier (ReLU) [27] as the
activation function to forecast the output, shown as below:

$\hat{y}=ReLU(w_{1}\hat{y}_{main}+w_{2}\hat{y}_{anc})$ (1)

where $ReLU$ is the activation function which is computationally efficient to
compute and has less likelihood of a vanishing gradient.

The output of the weighted average $\hat{y}$ includes multiple steps
predictions with the same dimension as the output of main neural network
module $\hat{y}_{main}$ and ancillary neural network $\hat{y}_{anc}$. We
minimize the root mean squared error (RMSE) between all these values and
corresponding ground truth value. In this case, we had prepared our ground
truth indoor temperature with multiple timestamps as a three-dimensional
matrix.

The output layer provides one output for each of the 96 timestamps. The closer
timestamp to the forecast time is usually more accurate than later timestamps.
The later timestamp forecasting provides an overview of how indoor temperature
is going to change and allows for taking some actions ahead of time [28] for
better anticipatory HVAC control of the building. This helps provide desired
temperatures with less energy consumed since less radical (more planned)
actions are taken.

###  III-D Deriving Confidence Intervals

After the model is fully trained, we need to derive the CI of the output of
the model. We use MC dropout proposed in [29] and this provides a framework to
estimate uncertainty without any change of the existing model.

Specifically, we add stochastic dropouts to each hidden layer of the neural
network. Then we add sample variance to the output from the model [10].
Finally, we estimate the uncertainty by approximating the sample variance. We
assume that indoor temperature is approximately a Gaussian distribution. In
this paper, we will derive the 68% and 95% CIs and use it to evaluate the
reliability of the model. Given the sample data, we could calculate the mean
$\mu$ and standard deviation $\sigma$. Then the $100\cdot(1-\alpha)\%$ CI [30]
is:

$\Big{[}\mu-
t_{1-\alpha/2}\cdot\frac{\sigma}{\sqrt{n}},\mu+t_{1-\alpha/2}\cdot\frac{\sigma}{\sqrt{n}}\Big{]}$
(2)

where $n$ is the sample number and $t$ denotes the density function.

(a) RMSE of all predictions
(b) MAE of all predictions
(c) MAPE of all predictions

Figure 9: Multi-step ahead prediction’s error of RMSE (a), MAE (b), and MAPE
(c) of all predictions.

(a) RMSE of max/min temperature predictions
(b) MAE of max/min temperature predictions
(c) MAPE of maxi/min temperature predictions

Figure 10: Multi-step ahead prediction’s error of RMSE (a), MAE (b), and MAPE
(c) of max/min temperature predictions.

(a) Probability of predictions are not within 68% CI

(b) Probability of predictions are not within 95% CI

Figure 11: The monthly probability of multi-step ahead predictions that are not within 68% CI (a) and 95% CI (b). TABLE I: One-step ahead prediction comparison between the model Using the adjoint architecture and the model without the ancillary network. Models | Error of All Predictions | Error of Max/Min Predictions | GT not within CI
---|---|---|---
| RMSE | MAE | MAPE | RMSE | MAE | MAPE | 68% | 95%
LSTM-RNN | 2.96 | 1.33 | 1.73 | 4.23 | 1.87 | 1.69 | 24.79% | 15.43%
LSTM-RNN (adjoint) | 2.65 | 1.20 | 1.55 | 2.82 | 1.23 | 1.84 | 20.13% | 11.69%
LSTM-encoder-decoder | 4.22 | 1.77 | 2.32 | 6.97 | 3.11 | 4.05 | 43.70% | 19.19%
LSTM-encoder-decoder (adjoint) | 1.06 | 0.63 | 0.83 | 3.25 | 1.45 | 1.83 | 23.80% | 13.56%
LSTM-encoder w/predictnet | 1.12 | 0.66 | 0.86 | 3.58 | 1.54 | 1.95 | 41.75% | 20.85%
LSTM-encoder w/predictnet (adjoint) | 0.83 | 0.60 | 0.75 | 2.28 | 1.31 | 1.61 | 19.82% | 13.82%
LSTM-DNN | 1.32 | 0.76 | 0.91 | 3.84 | 1.79 | 2.21 | 19.95% | 13.18%
LSTM-DNN (adjoint) | 0.57 | 0.39 | 0.61 | 1.39 | 0.78 | 1.26 | 9.84% | 7.15%
TABLE II: Multi-steps ahead Prediction Comparison between the model Using the adjoint architecture and the model without the ancillary network Models | Error of All Predictions | Error of Max/Min Predictions | GT not within CI
---|---|---|---
| RMSE | MAE | MAPE | RMSE | MAE | MAPE | 68% | 95%
LSTM-RNN | 3.83 | 1.70 | 2.20 | 5.58 | 2.59 | 3.39 | 33.68% | 25.08%
LSTM-RNN (adjoint) | 2.93 | 1.39 | 1.85 | 4.38 | 2.09 | 2.31 | 24.05% | 11.66%
LSTM-encoder-decoder | 6.21 | 2.23 | 2.91 | 8.03 | 3.41 | 3.77 | 47.71% | 21.68%
LSTM-encoder-decoder (adjoint) | 2.02 | 0.99 | 1.29 | 4.98 | 2.25 | 2.92 | 26.31% | 15.70%
LSTM-encoder w/predictnet | 2.07 | 1.02 | 1.32 | 4.86 | 2.26 | 2.94 | 47.78% | 24.22%
LSTM-encoder w/predictnet (adjoint) | 1.72 | 0.99 | 1.28 | 4.24 | 2.04 | 2.65 | 19.94% | 15.94%
LSTM-DNN | 2.38 | 1.19 | 1.43 | 5.37 | 2.40 | 2.92 | 28.79% | 18.87%
LSTM-DNN (adjoint) | 1.40 | 0.76 | 0.88 | 4.19 | 1.88 | 2.28 | 14.15% | 11.18%

##  IV Experiments

In this section, we will first test our proposed architecture with the LSTM-
DNN [31] network in both one-step ahead (15 minutes) prediction and 96-steps
ahead (24 hours) prediction. Later, we will do similar test to the other three
popular time series prediction models: LSTM RNN [22], LSTM encoder-decoder
[32], and LSTM encoder w/predictnet [8]. We will compare not only the error
(RMSE, MAE, and MAPE) of all predictions but also the error of predicting the
maximum and the minimum temperatures. In addition, we will evaluate the
reliability of the models by calculating the probability of ground truth
temperature being within the 68% and 95% CIs respectively.

###  IV-A Setting

The data comes from a multistory building located in New York City (NYC). Data
are collected by the Building Management System (BMS) every 15 minutes. The
temperature predicted is for one sensor on a floor of the building. Total
building occupancy data are also collected. In addition, we collect the
outside weather information, including wind speed and direction, humidity and
dew point, pressure and weather status (fog, rain, snow, hail, thunder, and
tornado), and temperature from the Central Park NOAA weather station. The
analyzed data are from June 9th, 2012 to November 16th, 2014 (84,768
timestamps). We utilized 67,814 of the timestamps for training and 16,954 for
testing. The ratio of training to testing is 8:2. We then split 20% of the
timestamps from the training dataset for validation. After the neural networks
are trained, we added dropout layer and repeatedly sample for 10,000 times.
Finally, we derive 68% and 95% CIs from the sample outputs.

###  IV-B One-Step Ahead Prediction Evaluation

We first compare the capacity of capturing time series pattern between the
LSTM-DNN model without the ancillary model and the LSTN-DNN model using the
adjoint architecture. Later on, we will evaluate the improvement of using our
proposed architecture from three different perspectives (error of the total
prediction, error of predicting max/min temperature, and model reliability
based on the CIs).

Capacity of Capturing the Time Series Pattern: to retrieve the one-step ahead
data from the output consisting of 96-steps ahead predictions, we extract the
first step of the output temperature of each input and splice these data.
Figure 2 demonstrates the sample predictions of using our proposed
architecture. The plot includes one-step ahead prediction with the 68% CI and
95% CI. The 68% and 95% CIs are shown as gray and light gray color band
respectively. The black solid line represents the ground truth indoor
temperature and the gray dash line is the prediction. We noticed that the
model learns the daily and weekly patterns as the temperature pattern during
weekdays and weekends is different. But we also found that the bands of the
CIs are large at the maximum and the minimum temperature.

In order to evaluate the improvement of prediction using the adjoint neural
network, we plot the predictions (without CIs) of the LSTM-DNN without the
ancillary network, the adjoint LSTM-DNN, and ground truth indoor temperature
in the same plot. Figure 3 shows the selected result of the one-step ahead
prediction from 22nd September to 22nd October. We observed that both models
capture the daily, weekly and holiday patterns well. But the adjoint model has
better predictions, especially when the predicting of the maximum and the
minimum temperature. Next, we will compare the model using our proposed
architecture with the model without the ancillary network these three
different perspectives.

####  IV-B1 Error of All Predictions

First of all, we calculated the monthly error of all predictions.
Specifically, we calculated the monthly error of RMSE, MAE, and MAPE using all
timestamps (every 15 minutes) for the months of June to November. Then we
compared the error of both the LSTM-DNN without the ancillary network and the
adjoint LSTM-DNN. Figure 4 illustrates the comparison of error (RMSE, MAE, and
MAPE) of the two models. It is obvious that the model using our proposed
architecture has a much smaller error than the LSTM-DNN without the ancillary
network.

####  IV-B2 Error of Max/Min Predictions

In addition, we evaluated the improvement of using our proposed architecture
to predict the maximum and the minimum temperatures. To begin with, we found
the timestamps of both the maximum and the minimum temperatures with respect
to each day. Next, we calculated the RMSE, MAE, and MAPE from 30 minutes (2
timestamps) before to 30 minutes after the timestamp of the maximum or the
minimum temperatures. Last, we summarized the error in the same month. Figure
5 illustrates that, using our proposed architecture, the model has a much
better capacity for predicting the maximum and the minimum temperatures.

####  IV-B3 Reliability of Predictions

Last, we evaluated the reliability of the model. This is evaluated by
calculating the probability that the ground truth indoor temperatures are not
within 68% and 95% CIs respectively, as shown in Figure 6. We notice that both
models have similar changes during the month. But the probability of using our
proposed architecture is always lower than the LSTM-DNN without the ancillary
network. Therefore, using the adjoint network increases the reliability of the
model.

Then, the same analysis was conducted for the three other base models, LSTM
RNN, LSTM encoder-decoder, and LSTM encoder w/predictnet, within the adjoint
neural network architecture and without the adjoint architecture.

To sum up, Table 1 illustrates the comparison of one-step ahead predictions
between the models with and without the ancillary network. It shows that the
adjoint neural network architecture decreases the error and increases the
reliability of the models. Also, we observed that our proposed model (LSTM-
DNN) which is not the best model without the ancillary network becomes the
best model when used within our adjoint neural network architecture.

###  IV-C Multiple Timestamps Evaluation

In this section, we will evaluate the prediction errors for 96-steps ahead.
Each step represents 15 minutes, so 96 steps denote the predictions for 24
hours. We will first compare the models’ performance using or not using our
proposed architecture to capture the daily pattern. Then, we will compare the
two models from these three different prospectives.

Capacity of Capturing the Time Series Pattern: Figure 7 shows the predictions
of using our proposed architecture for July 1st which was a Tuesday. The
prediction is for 96-steps ahead including the 68% CI and the 95% CI. The 68%
and 95% CIs are shown as the gray and light gray color bands respectively. The
black solid line represents the ground truth indoor temperature and the gray
dash line is the scalar predictions of the adjoint model. We noticed the model
learned the daily temperature and predicts the 24 hours fairly well. At
midnight, the indoor temperature is high since the HVAC system is shut down at
that hour. During the daytime, the HVAC system is operating, and the indoor
temperature is lower. As people leave the office, the building operators ramp
down and later turn off the HVAC system and the temperature raises again.

In order to evaluate the improvement of the prediction using our proposed
architecture, we plot the predictions (without CIs) of the LSTM-DNN without
the ancillary network, the adjoint LSTM-DNN, and the ground truth indoor
temperature in the same plot, as shown in Figure 8. We observed that the
adjoint neural network has better performance for fitting the 24 hour
prediction, especially for predicting the maximum and the minimum indoor
temperature. Next, we compare the model using our proposed architecture with
the model without the ancillary network from three different perspective.

####  IV-C1 Error for all predictions

Firstly, we calculated the monthly error of all predictions. Specifically, we
calculated the monthly error of RMSE, MAE, and MAPE using 96-steps ahead (24
hours) predictions of all the timestamps in that month respectively. Next, we
compared the error of both the LSTM-DNN without the ancillary network and the
LSTM-DNN using our proposed architecture. Figure 9 demonstrates the comparison
of the two models. Though the error is higher than the one-step ahead
predictions, the model that uses our proposed architecture still outperforms
the model without the ancillary network.

####  IV-C2 Error of Max/Min Predictions

Secondly, we evaluated the predictions on the maximum and the minimum
temperature. To begin with, we found the timestamps of both the maximum and
the minimum temperature. Then we calculated the RMSE, MAE, and MAPE from 30
minutes before and after the time of the maximum or the minimum temperature.
Finally, we summarized the error in the same month. Figure 10 illustrates that
the model using our proposed architecture has the better capacity to predict
the maximum and the minimum temperatures.

####  IV-C3 Reliability of Predictions

Thirdly, we evaluated the reliability of the model. We calculated the
probability of the ground truth indoor temperatures are outside of the 68% CI
and the 95% CI respectively. Thus, the lower the probability is, the more
reliable the model is. Figure 11 shows the result of how the probability
changes from June to November. We noticed that the models have fairly similar
change during this time period, but the adjoint neural network architecture is
more reliable than the LSTM-DNN without the ancillary network.

Then, the same analysis was conducted for the three other base models, LSTM
RNN, LSTM encoder-decoder, and LSTM encoder w/predictnet, within the adjoint
neural network architecture and without the adjoint architecture.

To sum up, Table 2 illustrates the comparison of the multi-step ahead
predictions between the model using our proposed model and the corresponding
model without the ancillary network. It shows that the ancillary neural
network can also decrease the error and increase the reliability of the model
in multi-step ahead forecast. The interesting finding, that the model (LSTM-
DNN) which is not the best model becomes the best model by using our adjoint
neural network architecture, also holds true in the multi-step ahead forecast.

##  V Conclusions

In this paper, we propose a novel adjoint neural network architecture for time
series prediction that uses an ancillary neural network to capture weekend and
holiday information for IBM. We used the dataset of a multistory building in
NYC and compared four different base models (LSTM RNN, LSTM encoder-decoder,
LSTM encoder w/predictnet and our proposed LSTM-DNN model) within the adjoint
architecture with the corresponding models without the ancillary network. The
models’ performance was evaluated in both one-step ahead prediction (15
minutes) and 96-steps ahead prediction (24 hours) in three different
perspectives. First, we compared the total error of RMSE, MAE, and MAPE of all
prediction accounted. Second, we compared the error (RMSE, MAE, and MAPE) of
predicting the maximum and the minimum temperatures. Third, we compared the
reliability of the models by calculating the probability of the ground truth
indoor temperatures are not within CIs.

From the result of the one-step ahead prediction (Figure 3), the models using
an ancillary NN successfully capture the daily, the weekly patterns and the
holidays. In addition, it performs better prediction result than the models
without using ancillary network. Then, from the result of the multi-step ahead
prediction (Figure 7), the models using an ancillary network successfully
captures the daily patterns better, especially on predicting the maximum and
the minimum temperatures.

Our adjoint neural network architecture with the LSTM-DNN model could be used
for building a more dependable IBM system. With accountable predicted indoor
temperatures, the system can provide comfortable indoor temperatures with less
energy consumed.

## References

  * [1] L. Pérez-Lombard, J. Ortiz, and C. Pout, “A review on buildings energy consumption information,” _Energy and Buildings_ , 2008.
  * [2] J. Romero, J. Navarro-Esbrí, and J. Belman-Flores, “A simplified black-box model oriented to chilled water temperature control in a variable speed vapour compression system,” _Applied Thermal Engineering_ , 2011.
  * [3] Z. Wang and R. S. Srinivasan, “A review of artificial intelligence based building energy use prediction: Contrasting the capabilities of single and ensemble prediction models,” _Renewable and Sustainable Energy Reviews_ , vol. 75, no. September 2015, pp. 796–808, 2017. [Online]. Available: http://dx.doi.org/10.1016/j.rser.2016.10.079
  * [4] H. Hippert, C. Pedreira, and R. Souza, “Neural networks for short-term load forecasting: a review and evaluation,” _IEEE Transactions on Power Systems_ , 2001.
  * [5] T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, and S. Khudanpur, “Recurrent Neural Network based Language Model,” _Interspeech_ , 2010.
  * [6] L. Mba, P. Meukam, and A. Kemajou, “Application of artificial neural network for predicting hourly indoor air temperature and relative humidity in modern building in humid region,” _Energy and Buildings_ , 2016.
  * [7] A. Ashtiani, P. A. Mirzaei, and F. Haghighat, “Indoor thermal condition in urban heat island: Comparison of the artificial neural network and regression methods prediction,” _Energy and Buildings_ , 2014.
  * [8] L. Zhu and N. Laptev, “Deep and Confident Prediction for Time Series at Uber,” _IEEE International Conference on Data Mining Workshops, ICDMW_ , vol. 2017-November, pp. 103–110, 2017.
  * [9] A. Marvuglia, A. Messineo, and G. Nicolosi, “Coupling a neural network temperature predictor and a fuzzy logic controller to perform thermal comfort regulation in an office building,” _Building and Environment_ , 2014.
  * [10] Y. Li and Y. Gal, “Dropout Inference in Bayesian Neural Networks with Alpha-divergences,” _ICML_ , 2017.
  * [11] Y. Gal, “Uncertainty in Deep Learning,” _PhD Thesis_ , 2016.
  * [12] D. Svozil, V. Kvasnička, and J. Pospíchal, “Introduction to multi-layer feed-forward neural networks,” in _Chemometrics and Intelligent Laboratory Systems_ , 1997.
  * [13] G. E. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition,” _IEEE Transactions on Audio, Speech and Language Processing_ , 2012.
  * [14] Y. A. LeCun, Y. Bengio, and G. E. Hinton, “Deep learning,” _Nature_ , 2015\.
  * [15] A. Goodfellow, Ian, Bengio, Yoshua, Courville, “Deep Learning,” _MIT Press_ , 2016.
  * [16] S. Cao, W. Lu, and Q. Xu, “Deep Neural Networks for Learning Graph Representations,” _Aaai_ , 2016.
  * [17] A. Toshev and C. Szegedy, “DeepPose: Human pose estimation via deep neural networks,” _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2014.
  * [18] D. Cireşan, U. Meier, J. Masci, and J. Schmidhuber, “Multi-column deep neural network for traffic sign classification,” _Neural Networks_ , 2012\.
  * [19] W. Huang, G. Song, H. Hong, and K. Xie, “Deep architecture for traffic flow prediction: Deep belief networks with multitask learning,” _IEEE Transactions on Intelligent Transportation Systems_ , 2014.
  * [20] P. Romeu, F. Zamora-Martínez, P. Botella-Rocamora, and J. Pardo, “Time-series forecasting of indoor temperature using pre-trained deep neural networks,” in _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , 2013.
  * [21] S. Kalogirou, “Artificial neural networks for the prediction of the energy consumption of a passive solar building,” _Energy_ , 2000.
  * [22] S. Hochreiter and J. Urgen Schmidhuber, “LONG SHORT-TERM MEMORY,” _Neural Computation_ , 1997.
  * [23] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in _ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings_ , 2016.
  * [24] Y. Cui, S. Wang, J. Li, and Y. Wang, “LSTM Neural Reordering Feature for Statistical Machine Translation,” in _In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , 2016.
  * [25] D. L. Marino, K. Amarasinghe, and M. Manic, “Building energy load forecasting using Deep Neural Networks,” in _IECON 2016 - 42nd Annual Conference of the IEEE Industrial Electronics Society_ , 2016.
  * [26] X. Qiu, L. Zhang, Y. Ren, P. Suganthan, and G. Amaratunga, “Ensemble deep learning for regression and time series forecasting,” in _IEEE SSCI 2014 - 2014 IEEE Symposium Series on Computational Intelligence - CIEL 2014: 2014 IEEE Symposium on Computational Intelligence in Ensemble Learning, Proceedings_ , 2014.
  * [27] X. Glorot, A. Bordes, and Y. Bengio, “Deep sparse rectifier neural networks,” _AISTATS ’11: Proceedings of the 14th International Conference on Artificial Intelligence and Statistics_ , 2011.
  * [28] F. J. Chang, Y. M. Chiang, and L. C. Chang, “Multi-step-ahead neural networks for flood forecasting,” _Hydrological Sciences Journal_ , 2007.
  * [29] Y. Gal and Z. Ghahramani, “A Theoretically Grounded Application of Dropout in Recurrent Neural Networks,” no. Nips, 2015. [Online]. Available: http://arxiv.org/abs/1512.05287
  * [30] M. J. Penciana and R. B. D’Agostino, “Overall C as a measure of discrimination in survival analysis: Model specific population value and confidence interval estimation,” _Statistics in Medicine_ , 2004.
  * [31] T. N. Sainath, O. Vinyals, A. Senior, and H. Sak, “Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks,” in _ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings_ , 2015.
  * [32] K. Cho, B. V. Merrienboer, D. Bahdanau, and Y. Bengio, “On the Properties of Neural Machine Translation : Encoder – Decoder Approaches,” _Ssst-2014_ , 2014.