# Meta Sense Embedding

###### Abstract

Various meta embedding methods for word embedding have been proposed to create
a more accurate embedding. However, the effect of applying those meta
embedding methods to sense embeddings is an under-explored area. Sense
embedding is a more fine-grained representation than word embedding, which
provides a mapping from words’ senses to vectors. Our experiments show that
some of the word meta embedding methods may produce meta sense with a large
deviation from the original vector space, which leads to significant
deterioration in the performance of Word sense disambiguation (WSD). In this
thesis, a supervised learning method is proposed to resolve this issue by
learning a projection matrix training on SemCor. This thesis attempted to
propose a novel meta sense embedding method called vector space preserving
(VSP) and it achieves a SOTA performance in WSD and WiC among all the sense
embeddings. In addition, I also explore the performance of the meta-sense
embedding with the different proportions and intersection ratio of the source
embedding.

###### Contents

  1. 1 Introduction
  2. 2 Related Work
    1. 2.1 Word Embedding
    2. 2.2 Static Sense Embedding
    3. 2.3 Meta Embedding
    4. 2.4 Word Sense Disambiguation(WSD)
    5. 2.5 Word in Context (WiC)
  3. 3 Design and Implementation
    1. 3.1 Concatenation(CONC)
    2. 3.2 Averaging (AVG)
    3. 3.3 Singular Value Decomposition(SVD)
    4. 3.4 AutoEncoder
    5. 3.5 Supervised Learning Method for Space shift Issue
    6. 3.6 Vector Space Preserving Meta Embedding (VSP)
    7. 3.7 Normalisation in meta sense embedding
  4. 4 Experiments and Evaluation(Testing)
    1. 4.1 Source Embeddings
      1. 4.1.1 LMMS2048
      2. 4.1.2 SensEmBERT
      3. 4.1.3 ARES
    2. 4.2 Training for projection learning
    3. 4.3 Evaluation
    4. 4.4 Intersection ratio
  5. 5 Conclusion
  6. 6 BCS Criteria
    1. 6.1 Ability to apply skills gained during the degree
    2. 6.2 Innovation and/or creativity
    3. 6.3 Synthesis of information
    4. 6.4 Meet a real need in a wider context
    5. 6.5 Ability to self-manage a significant piece of work
    6. 6.6 Critical self-evaluation of the process

##  § 1 Introduction

Constructing accurate representations of semantics of words has received much
attention from natural language processing (NLP) researchers. Following Bengio
et al. (Bengio03) who applied neural networks language model (NNLM) to build
words’ distributed representation, Miklov et al. proposed Word2Vec NNLM method
and released pre-trained word embeddings, which have been widely used by
subsequent work fasttext.

Despite the success of the works in modelling word vector representation,
these embeddings cannot handle the tasks which require clear differentiation
between each meaning (senses) of an ambiguous word such as Word Sense
Disambiguation (WSD) WSD-survey and Word in Context (WiC) wic. This is caused
by the nature of their construction: these pre-trained embeddings are static
and they always present the ambiguous word using the same vector in different
contexts. There are two different approaches to resolving this issue. One
approach is to build contextualised word embedding. ELMo peter2018 used
Bidirectional LSTM to encode context information into a word, which is able to
dynamically represent a word by different vectors under different contexts.

The other approach is to model the words in a finer semantic granularity
level, namely build sense embedding. Sense embedding builds a one-to-one
mapping between each precise meaning of words, denoted by sense key, to one
vector. Although sense embedding itself is static embedding (insensitive to
context), it contains senses representing different meanings of an ambiguous
using different sense vectors. The collection of all the sense keys is known
as sense inventory. Princeton WordNet wordnet is a large-scale English lexical
database and has been recognised as the standard sense inventory, especially
for WSD. As I will prove in § 4.4, the sense key coverage of a sense embedding
will have a significant influence on the performance of WSD. However, to the
best of our knowledge, public available static sense embeddings with full
coverage of WordNet are only LMMS series LMMS and ARES ARES, which is largely
due to scarcity in sense-level annotated corpora. Improving the coverage of
static sense embedding is therefore a non-trivial issue, which is also one of
the motivations of this thesis.

Meta embedding is a method to create new embedding from an existing embedding
ensemble (also known as source embeddings), which can be considered as a way
to increase the coverage of sense embedding. Meta embedding is originally
designed for word embeddings. Yin and Schütze (ensemble) proposed four methods
in their paper to create meta embedding, which are Concatenation (CONC),
Singular Value Decomposition (SVD), 1${TO}$N, and 1${TO}$N+, and their meta
embedding achieved better performance than source embedding in word
similarity, word analogy and POS tagging. Other researchers utilised applied
meta embedding to create new sentence embedding meta-sentence. However, no one
has attempted to create the meta embedding of sense embedding and discuss
their meta embedding strategies. Inspired by the success of meta embedding of
word embedding, I apply meta embedding to sense embedding (meta sense
embedding) with the aim of increasing source embedding coverage and improving
sense embedding quality. I discovered that meta embedding methods which work
for word embedding do not necessarily improve the performance of meta sense
embedding. Instead, some of these methods such as SVD and AutoEncoder even
lead to a significant decrease in the performance in WSD. I attribute this
issue to the shift of vector space: the meta sense embedding does not
guarantee to stay in the same vector space even if the source embeddings are
in the same vector space. As I will explain in § 2.4, the meta sense embedding
must stay in the same vector space of the encoder of the evaluation system. I
resolve this issue by learning a projection matrix to make sure our meta sense
embedding has consistent vector space with the evaluation system encoder’s
vector space. The contribution of this thesis can be concluded as follows:

(1) Applied meta embedding to sense embedding and examined the influence of
normalisation and intersection rate on meta embedding.

(2) Proposed a supervised learning method to resolve the vector shift issue in
meta sense embedding.

(3) Achieved state-of-the-art performance in static sense embedding-based Word
Sense Disambiguation and Word in Context tasks.

##  § 2 Related Work

###  § 2.1 Word Embedding

Static word embeddings represent the semantic meaning of the words in the form
of vectors. These embeddings are typically trained on fixed-length corpora to
capture the semantic information of words mikolov. However, this ability is
hindered due to meaning conflation deficiency, which is the issue that
different meanings of an ambiguous word share the same vector representation
CardiffSense. For example, “bank” means depository financial institution in
”withdraw money from the bank” while it means land beside a body of water in
“river bank”. It is semantically inappropriate to represent the bank in the
two phrases using the same vector.

The other word type of word embedding is called contextualised embedding.
Unlike static word embedding, contextualised embedding is able to represent
the word sensitive to context information. Even for the same word, different
embedding can be derived due to the difference in context. Shortly after ELMo,
transformers based language model such as BERT BERT, assigning attention
weights according to word relationship in context, has been the mainstream
encoders used for generating high-quality contextualised embeddings.

###  § 2.2 Static Sense Embedding

Constructing sense embeddings can resolve the meaning conflation deficiency of
words. Static sense embeddings are finer-grained representations than static
word embeddings. Using the example mentioned above, two different senses of
“bank” will be denoted by “ depository_financial_institution%1:14:00::” and
“bank%1:17:01::” respectively. Peters et al. (peter2018) proposed a supervised
learning method to compute sense embedding by averaging the contextualised
word embedding with the same sense tags from SemCor semcor. However, the sense
embedding they built covers only a small proportion of WordNet’s sense
inventory. Until recently, sense embeddings with full-coverage of WordNet are
released LMMS; ARES and their construction method will be introduced in § 4.1.

###  § 2.3 Meta Embedding

The underlying reason for the success of meta word embedding may be explained
by this assumption: sources embedding may contain complementary semantic
information goikoetxea-etal-2015-random, combing them correctly will lead to
better overall performance. After the study on word embedding by Yin and
Schütze (ensemble), Coates and Danushka (avg) proposed averaging as another
simple yet effective meta embedding approach. AutoEncoding achieves SOTA
performance in multiple word embedding evaluation tasks autoencoder. This
thesis adopts these methods to meta embedding and reports their performance in
§ 4.3.

###  § 2.4 Word Sense Disambiguation(WSD)

Word sense disambiguation is a classical and important task in NLP WSD-survey.
Raganato, Camacho-Collados, and Navigli proposed an evaluation framework for
WSD and it is been widely used as the test standard WSD-framework. The test
set consists of the following five evaluation set: Senseval-2 senseval2,
Senseval-3 senseval3, SemEval-2007 Task 17 semeval2007, SemEval2013 Task 12
semeval2013 and SemEval-2015 Task 13 semeval2015.

Peters et al. (peter2018) proposed an algorithm closely connecting sense
embedding and WSD: compute contextualised word embedding and pick its most
similar sense from pre-trained sense embedding using cosine similarity
criteria as the prediction. This algorithm has been followed by subsequent
researchers. They achieved better performance by only improving the quality of
sense embedding without further modification to the algorithm itself ARES;
LMMS; scarlini-etal-2020-sensembert. In fact, by only changing sense embedding
and sticking to this 1-Nearest Neighbour (1-NN) algorithm, it can be claimed
sense embeddings’ performance on WSD and WiC is a good indicator of their
quality. A desired construction of sense embedding should have clearly
different representations of all the meanings of an ambiguous word. It is also
worth mentioning that this algorithm has an implicit yet important
requirement: the sense embedding should have the same vector space as the
encoder’s vector space. This is due to the algorithm’s prediction requiring
accurate ranking according to their pair-wise cosine similarities. If these
two vectors are not in the same vector space, their cosine similarity value is
not accurate and meaningful, which can lead to poor performance. For the
purpose of fair comparisons with the source embedding, in this thesis I also
adopt this 1-NN algorithm to evaluate the quality of our meta sense embedding.

###  § 2.5 Word in Context (WiC)

Word in Context (WiC) is a binary classification task which is regrading
making predictions on whether the same word in different contexts has the same
meaning wic. The mainstream methods to resolve this challenge can be divided
into supervised or unsupervised approaches. Based on the result of the 1-NN
approach of WSD, the first way to resolve WiC is to compare whether the
disambiguated senses of the two words are equal loureiro-jorge-2019-liaad. The
second approach proposed by Loureiro and Jorge is to train a classifier using
the similarities of the contextualised embedding and closest sense as the
input feature. In this thesis, I followed their choice of classifiers, namely
Logistic Regression Classifier, for the ease and fairness of comparison.

##  § 3 Design and Implementation

In this section, I will discuss the general design of the meta embedding
methods such as Concatenation, Averaging, and their implementation in our
paper. I will also discuss the issue of normalisation during the
implementation stage of the meta embedding.

###  § 3.1 Concatenation(CONC)

Concatenation creates meta embedding by concatenating different sources
embeddings and this method has proved its success in creating meta word
embedding and meta sentence embedding ensemble; autoencoder; meta-sentence. I
extend this method to create sense embedding. One of the defects of CONC is
that the embedding dimension will increase since the meta embedding’s
dimensionality equals to the sum of all source embeddings’ dimensionality:
m_conc(s) $\in$ $\mathbb{R}^{d1+d2+...+dn}$. Since the meta embedding’s sense
inventory is the union of the source embeddings’ sense inventories, the case
that sense key $s$ does not exist in sense embedding $e_{j}$ needs to be
handled. A solution is to perform $pad$ operation:
$pad(s)=\boldsymbol{v^{zeros}}+\boldsymbol{e_{j}(s)}$ (assume e_j(s) is zero
vector if sense key $s$ is missing ).

$\displaystyle\boldsymbol{{m}_{conc}(s)}$
$\displaystyle=\begin{bmatrix}pad(\boldsymbol{e_{1}(s))}\\\
pad(\boldsymbol{e_{2}(s))}\\\ \vdots\\\
pad(\boldsymbol{e_{n}(s))}\end{bmatrix}$

###  § 3.2 Averaging (AVG)

Coates and Bollegala suggested that computing the average of word embedding is
a valid and effective word meta embedding strategy although correspondence
between the dimensions of different embeddings is not guaranteed avg. They
also proved that given word embeddings are approximately orthogonal, this meta
embedding method performs as well as concatenation(CONC). I extend this method
to the meta sense embedding by computing the average of source sense
embeddings. Different source embeddings may have different embedding
dimensionality and some sense keys may be missing in certain source embedding.
I handle these two issues by applying padding operation mentioned in CONC.

$\displaystyle\boldsymbol{{m}_{avg}(s)}=\sum\limits^{N}_{j=1}\frac{pad(\boldsymbol{e_{j}(s)})}{N}$

###  § 3.3 Singular Value Decomposition(SVD)

SVD is a dimension reduction technique and it is been widely in NLP tasks SVD-
LSI; SVD-LSA; SVD-LRA. It has also been used to create meta word embedding to
reduce embedding dimensionality ensemble. When performing SVD to create meta
sense embedding, I firstly use CONC in § 3.1 to build matrix Mconc, whose row
number equals the size of sense inventories
$\left|\left|\mathcal{V}\right|\right|$ and columns equals to the summation of
source embedding dimensionality $dim$. Next, I perform SVD on Mconc = $U\Sigma
V^{T}$, where U and V are unitary matrices and the diagonal matrix $\Sigma$
contains the singular values of Mconc. I then truncate the left unitary matrix
U by keeping its k leftmost columns, namely a matrix with the shape of
$\left|\left|\mathcal{V}\right|\right|*k$. These k columns correspond to the
top k largest singular vectors from $\Sigma$, which contains top k’s most
significant components of the embedding information.

###  § 3.4 AutoEncoder

Bollegala and Bao (autoencoder) proposed AutoEncoding Meta Embedding(AEME),
which modelled meta embedding task as building AutoEncoder. It achieves SOTA
performance such as Semantic Similarity, Word Analogy, and Relation
Classification compared to other meta embedding approaches. Autoencoder is a
neural network that learns underlying patterns of data that first encodes the
input vectors to latent vectors and then reconstructs the input vectors based
on these latent intermediate vectors.

They proposed three AutoEncoding approaches: Decoupled Autoencoded Meta-
Embedding (DAEME) obtained meta embedding by concatenating encoded the source
and decoding each encoded source embedding separately, Concatenated
Autoencoded Meta-Embedding (CAEME) that obtained meta embedding similar to
DAEME but decode the concatenated encoded vectors and Averaged Autoencoded
Meta-Embedding (AAEME) that is similar to DAEME but decode the average of
encoded vectors. In all these three methods, meta embedding is the
intermediate encoded vector. During the experiments, I set the dimensionality
of the intermediate vector equal to 2048 since all the embedding
dimensionalities of source embeddings are 2048. I choose the AAEME in our
subsequent experiments since it performs best among these three versions of
meta embedding methods. It is also worth mentioning that AAEME also performs
best in the word embedding among the three AEME methods. The formula of the
AAEME is defined as follows: Suppose there are two sources $e_{1}$ and
$e_{2}$, which are encoded by two encoders $E_{1}$ and $E_{2}$ respectively.
The AAEME of sense $s$ is computed as the L2 normalised average of the encoded
source embeddings:

$\displaystyle\boldsymbol{m}_{AAEME}=\frac{E_{1}(\boldsymbol{e_{1}}(s))+E_{2}(\boldsymbol{e_{2}}(s))}{\left|\left|E_{1}(\boldsymbol{e_{1}}(s))+E_{2}(\boldsymbol{e_{2}}(s))\right|\right|_{2}}$

The AAEME tries to minimise the weighted sum of reconstruction loss for all
the sense key in the sense inventories $\mathcal{V}$. The coefficients λ1 and
λ2 are used to give different emphasis to reconstructing the two sources.

$\displaystyle\lambda_{2}\left|\left|\boldsymbol{e_{2}}(s)-D_{2}(E_{2}(\boldsymbol{e_{2}}(s)))\right|\right|^{2}$

However, this method may lead to the problem that the output embedding vector
space shift issue. The meta embedding’s vector space has been transformed by
encoders, which implies they are in the latent vector space rather than the
same vector space as the source embeddings.

###  § 3.5 Supervised Learning Method for Space shift Issue

When conducting WSD evaluation tasks, it is required that the meta sense
embedding m(s) has the same vector space as the contextualised embedding in
the context $t$. As I will demonstrate in § 4.1, the source embeddings
utilised in this thesis are all encoded using BERT. This implies that the
vector space of three source embedding is close to or identical BERT vector
space. Therefore, meta embedding obtained using concatenation (CONC) and
average (AVG) will not change the vector space. But this is not the case when
I apply any other meta embedding method. For the autoencoder method, I will
use the latent embedding which is the encoded embedding of source embedding so
the vector space may change. In general, a projection matrix must be learned
Mm∗n such that after the multiplication of the meta embedding $meta(w_{i})$ by
the matrix $A$, $meta(w_{i})A$ are in the same vector space with the
contextualised embedding. The measurement for the vector space difference can
be defined by the cosine distance, which equals to one minus cosine
similarity. If the constant is dropped, the loss function will be exactly the
negative of the cosine similarity. This dissimilarity will be minimised over
all instances in SemCor for its sense tagged words w. Hence the loss function
can be defined as

$\displaystyle\mathcal{L}(\text{\bf
M})=-\displaystyle\sum\limits_{w^{s}\in\mathcal{T}}Sim(\boldsymbol{m}(s)^{T}\text{\bf
M},{BERT}(w^{s},t))$

Here Sim denote the cosine similarity between two embeddings:

$\displaystyle
Sim(s_{1},s_{2})=\displaystyle{cos(\theta(s_{1},s_{2}))}=\displaystyle{\frac{s_{1}\cdot
s_{2}}{\lVert s_{1}\rVert\lVert s_{2}\rVert}}$

###  § 3.6 Vector Space Preserving Meta Embedding (VSP)

In order to resolve the root of the vector space shift problem, I proposed a
novel meta embedding which has the property that the vector space will stay
close to the source embedding. The experiments show that it can surpass the
performance of averaging and the meta embedding remains its original
dimensionality without the issue of increase of dimensionality as CONC may
cause. It achieves a better performance using only two source embeddings
VSP(ARES+SensEmBERT) than any other meta embedding methods using two or three
source embeddings in the WSD task and better performance in WiC task. Suppose
for sense id set $s$ there are $N$ source sense embeddings $e_{j}$ with $j$
from 1, 2,…, to N, a generic formula of VSP meta embedding can be written and
its loss function to be optimised are (1) and (2) respectively

$\displaystyle\boldsymbol{{m}_{vsp}(s)}=\sum_{j=1}^{N}Encode(\boldsymbol{e_{j}(s)})$
(1)
$\displaystyle\mathcal{L}(\boldsymbol{m}_{vsp})=\sum_{j=1}^{N}Dist(\boldsymbol{e}_{j},\boldsymbol{m}_{vsp})$
(2)

In this thesis, matrices are trained $\text{\bf M}_{j}$ for each source
embedding e_j and the bias term $b_{j}$ as the encoding function. Vector space
distance can be defined as follows: firstly for each sense key s in the sense
keys vocabulary $\mathcal{V}$, compute the sum of cosine distance of the meta
embedding $\boldsymbol{m}_{vsp}(s)$ with all the source embedding
$\boldsymbol{e_{j}}(s)$. The final vector space is the summation of these
distances. $\boldsymbol{m}_{vsp}(s)$ and $\mathcal{L}(\boldsymbol{m}_{vsp})$
are specifically defined as follows:

$\displaystyle\boldsymbol{m}_{vsp}(s)=\sum_{j=1}^{N}\text{\bf
M}_{j}\boldsymbol{e}_{j}(s)+b_{j}$ (3)

$\displaystyle\mathcal{L}(\boldsymbol{m}_{vsp})=$
$\displaystyle-\sum_{j=1}^{N}\sum_{s\in\mathcal{V}}\frac{\boldsymbol{e}_{j}(s){}^{\top}\boldsymbol{m}_{vsp}(s)}{\left|\left|\boldsymbol{e}_{j}(s)\right|\right|_{2}\left|\left|\boldsymbol{m}_{vsp}(s)\right|\right|}_{2}$

As suggested by the supervisor, there are some defects in this method. For
example, this method is suitable for the cases when the input source
embeddings have close vector space. Moreover, when the source embeddings have
inconsistent embedding dimensions, further operations are required such as
padding. Fortunately, these conditions are automatically fulfilled using the
three source embedding which will be mentioned in § 4.1. I will try to improve
this method and provide more explanation in our later conference paper.
Despite these issues, this method is still presented here because according to
the COMP 390 final year project dissertation requirements, the thesis should
present all the methods proposed during the project even if the method is not
a perfect and finalised approach. Moreover, this method provides another proof
of the significance of maintaining the same vector space as BERT the statement
mentioned in § 3.5.

The aforementioned method VSP is an unsupervised learning method. Based on the
VSP, I plan to explore the possibility of extending it to a supervised
learning meta embedding learning, which will be our future work. This
supervised meta embedding method is denoted by $VSP^{sup}$ to avoid confusion.
$VSP^{sup}$ can utilise existing corpora to minimise the distance between the
meta embedding vector space and the training instances’ vector space. For
example, suppose the training instances are encoded using BERT and the
contextualised embedding vector space. Here $\alpha$ and $\beta$ allow us to
give different emphasises on the unsupervised term and supervised term
respectively. The regularisation term
$\lambda\sum_{j=1}^{N}\left|\left|\text{\bf M}_{j}{}^{\top}\text{\bf
M}_{j}-\text{\bf 1}\right|\right|_{2}^{2}$ is used to make the training more
stable by forcing the norm of the $\text{\bf M}_{j}$ close to identity matrix.
The coefficient $\lambda$ represents the factor of the regularisation. Using
the same training loss and notation mentioned in § 3.5 , the loss function for
$\boldsymbol{m}_{vsp^{sup}}$ can be formulated as follows:

$\displaystyle\mathcal{L}(\boldsymbol{m}_{vsp^{sup}})=$
$\displaystyle-\alpha\sum_{j=1}^{N}\sum_{s\in\mathcal{V}}\frac{\boldsymbol{e}_{j}(s){}^{\top}\boldsymbol{m}_{vsp^{sup}}(s)}{\left|\left|\boldsymbol{e}_{j}(s)\right|\right|_{2}\left|\left|\boldsymbol{m}_{vsp^{sup}}(s)\right|\right|_{2}}$
$\displaystyle-\beta\sum\limits_{w^{s}\in\mathcal{T}}\frac{\boldsymbol{m}_{vsp^{sup}}(s)^{T}{BERT}(w^{s},t)}{\left|\left|\boldsymbol{m}_{vsp^{sup}}(s)\right|\right|_{2}\left|\left|{BERT}(w^{s},t)\right|\right|_{2}}+\lambda\sum_{j=1}^{N}\left|\left|\text{\bf
M}_{j}{}^{\top}\text{\bf M}_{j}-\text{\bf 1}\right|\right|_{2}^{2}$ (4)

###  § 3.7 Normalisation in meta sense embedding

Whether performing normalisation of the source embedding before is an issue
worth mentioning. In the prior work of meta word embedding, it was stated that
the normalisation is required so that every source embedding can contribute
equally to the meta embedding ensemble. Moreover, it is argued that benchmark
if the vectors are not normalised the vector space will be cancelled when
averaging. Based on the result of our experiments, I found meta embedding
without normalisation still achieves comparable performance. The reason behind
it is that three source embeddings in this thesis, LMMS, ARES, and SensEmBERT,
are in the same vector space.

##  § 4 Experiments and Evaluation(Testing)

I select three pre-trained sense embeddings as the source embedding to conduct
our experiments.

###  § 4.1 Source Embeddings

####  4.1.1 LMMS2048

LMMS is proposed by Loureiro and Jorge (LMMS), which is a sense embedding with
full coverage of WordNet, which contains 206949 senses in total. They firstly
derived LMMS2048 by using BERT and glossary information with the embedding
dimensionality of 2048. They obtained a 2348 dimension sense embedding
LMMS2348 by appending the static word embedding generated from fastText for
robustness. Since the performance of LMMS2348 and LMMS2048 of the WSD task are
the same, I selected LMMS2048 so that the computation cost is lower.

####  4.1.2 SensEmBERT

SensEmBERT scarlini-etal-2020-sensembert is constructed using a knowledge
based approach covering all the 146312 English nominal senses of the WordNet
with the embedding dimensionality of 2048. Each sense embedding consists of
two components: the first one is the average of the word embedding of the
target sense’s relevant words and the second one is the average of the BERT
encoded tokens of the sense gloss.

####  4.1.3 ARES

Scarlini et al. (ARES) also proposed a sense embedding with the full-coverage
of WordNet and the dimensionality of 2048 called ARES (context-AwaRe
Embeddings. They created the embedding by leveraging the representing
abilities of BERT, the glossary information and the information contained in
the SyntagNet maru-etal-2019-syntagnet. It achieves a better performance than
LMMS in WSD task.

The intersection of the LMMS2048 and ARES contains 206949 senses, which is
equivalent to the number of senses of WordNet since both of them covers all
the sense in WordNet. The intersection of the LMMS2048 and SensEmBERT or the
intersection of the ARES and SensEmBERT contains 146312 senses, which is the
number of all the nominal senses in the WordNet.
Here are the reasons why I choose these sense embeddings. LMMS2048 and ARES
are the best two full-coverage sense embedding which SensEmBERT is that I can
examine the cases when some of the sense key is missing, making our conclusion
more general and not limited to the embeddings with complete sense id overlap.

###  § 4.2 Training for projection learning

The training data can be obtained using sense-annotated corpora, namely SemCor
semcor. The input features are the sense keys and the output labels are the
contextualised BERT embeddings of the word in the sentence. The reason why I
use SemCor is that it remains to be the largest sense-annotated corpora
reloadlmms. SemCor is the Brown Corpus with manually sense annotations for
33,362 sense keys. It covers 16.1% of WordNet’s sense key inventory. The
training process iterates all the items in the SemCor using the vanilla
stochastic gradient descent to minimise the cosine distance of the groud truth
BERT contextualised embedding and static sense embedding.

Table 1: WSD and WiC Result With Different Methods. WiC(training) is the classifier-based method mentioned in § 2.5 | senseval2 | senseval3 | semeval2007 | semeval2013 | semeval2015 | ALL | WiC(With training)
---|---|---|---|---|---|---|---
LMMS only | 76.30% | 75.60% | 68.10% | 75.10% | 77.00% | 75.40% | 69.30%
ARES only | 78.00% | 77.10% | 71.00% | 77.30% | 83.20% | 77.90% | 68.50%
SensEmBERT only | - | - | - | 78.70% | - | - | -
CONC(LMMS+ARES) | 78.20% | 77.10% | 71.00% | 77.30% | 83.00% | 77.90% | 67.55%
CONC(LMMS+ARES+SensEmBERT) | 78.22% | 77.14% | 70.99% | 77.37% | 82.97% | 77.97% | 67.71%
AVG(LMMS+ARES+SensEmBERT) | 79.40% | 77.50% | 70.30% | 77.90% | 80.80% | 78.20% | 71.47%
AVG(LMMS+ARES) | 78.10% | 77.00% | 71.20% | 77.40% | 83.10% | 78.00% | 68.50%
AVG(LMMS+SensEmBERT) | 77.70% | 76.20% | 68.80% | 78.00% | 77.70% | 76.80% | 69.91%
AVG(ARES+SensEmBERT) | 78.60% | 77.40% | 71.20% | 78.10% | 81.70% | 78.10% | 71.32%
CONC(ares+sensem) | 78.79% | 77.68% | 71.21% | 78.47% | 81.21% | 78.30% | 71.63%
CONC(lmms+sensem) | 77.39% | 76.27% | 69.23% | 78.10% | 77.79% | 76.81% | 71.16%
VSP(ARES+SensEmBERT) | 78.57% | 77.62% | 71.21% | 78.71% | 81.80% | 78.35% | 72.25%

###  § 4.3 Evaluation

The f1 score is utilised as the metric for WSD task, while accuracy is used
for the evaluation metric of WiC task.
Observations on WSD Result: Due to SensEmBERT only containing nominal senses,
except for semeval2013 its performance on WSD test sets and WiC is a low
performance comparing the other two source embedding. Therefore, the results
are not presented. CONC and AVG surpass the performance of source embedding.
VSP(ARES+SensEmBERT) achieves SOTA performance in this task among all the
sense embedding based approaches.
Observations on WiC Result: I can observe that generally if a sense embedding
achieves a good performance in WSD, it tends to achieve a good performance in
WiC as well. VSP(ARES+SensEmBERT) also achieves SOTA performance in WiC among
all the sense embedding based approaches.
Observations on Projection learning: From Table 2, it can be seen that after
projection learning, the performance of SVD and AAEME improved significantly
from $\sim$ 40% back to $\sim$ 75%. It is also expected that the WiC without
training, which is namely the WSD-based approach mentioned in § 2.5 also
receive a great benefit from projection learning. An interesting finding is
that classifier-based WiC does change too drastically before and after
projection learning. This is because Logistic Regression Classifier utilised
cosine similarities as features to make predictions. Therefore, the classifier
works as a non-linear mapping function so the performance maintains relatively
stable, which also implies the vector space shift issue is covered.

Table 2: Projection Learning Result. “without Proj” and “with Proj” denote the original result and results after projection learning respectively, SVD (2048) and SVD(4096) represent denote the meta embedding using SVD with output dimensionality equals to 2048 and 4096 respectively.WiC(training) and WiC(WSD-based) are the classifier-based and WSD-based method metioned in § 2.5 | senseval2 | senseval3 | semeval2007 | semeval2013 | semeval2015 | ALL | WiC(training) | WiC(WSD-based)
---|---|---|---|---|---|---|---|---
AAEME without Proj | 43.90% | 38.10% | 26.80% | 43.40% | 46.80% | 41.60% | 65.83% | 53.61%
AAEME with Proj | 77.39% | 75.46% | 67.25% | 75.85% | 78.18% | 76.02% | 66.28% | 68.68%
SVD (2048) without Proj | 39.00% | 31.70% | 20.70% | 38.00% | 40.40% | 35.90% | 67.08% | 60.34%
SVD (2048) with Proj | 76.30% | 74.20% | 68.40% | 74.00% | 76.30% | 74.80% | 66.93% | 67.71%
SVD (4096) without Proj | 39.00% | 31.60% | 20.90% | 38.00% | 40.50% | 35.90% | 66.77% | 58.15%
SVD (4096) with Proj | 76.60% | 73.90% | 69.70% | 74.50% | 76.10% | 74.90% | 68.65% | 67.40%

###  § 4.4 Intersection ratio

Table 3: WSD under different intersection ratio without full-coverage setting CONC |  |  |  |  |  |  |
---|---|---|---|---|---|---|---
Partial Ratio(Proportion of Sense Embedding) | Intersection Ratio | senseval2 | senseval3 | semeval2007 | semeval2013 | semeval2015 | ALL
100%(Original File) | 100% | 78.20% | 77.10% | 71.00% | 77.30% | 83.00% | 77.90%
50% | 0% | 71.30% | 71.60% | 63.50% | 72.50% | 73.30% | 71.40%
| 25% | 67.90% | 67.20% | 57.80% | 69.30% | 69.10% | 67.60%
| 50% | 62.80% | 62.70% | 51.40% | 66.10% | 66.40% | 63.30%
| 75% | 60.30% | 60.50% | 50.10% | 63.50% | 66.00% | 61.30%
| 100% | 55.60% | 56.90% | 47.00% | 60.60% | 64.60% | 57.80%
25% | 0% | 53.00% | 55.00% | 41.80% | 56.90% | 61.50% | 54.90%
| 25% | 52.40% | 53.50% | 39.80% | 55.30% | 61.00% | 53.70%
| 50% | 51.70% | 51.20% | 41.50% | 53.50% | 56.80% | 52.10%
| 75% | 51.30% | 49.80% | 39.60% | 51.80% | 55.20% | 50.80%
| 100% | 50.00% | 49.20% | 40.20% | 49.80% | 54.90% | 49.80%
AVG |  |  |  |  |  |  |
Partial Ratio(Proportion of Sense Embedding) | Intersection Ratio | senseval2 | senseval3 | semeval2007 | semeval2013 | semeval2015 | ALL
100%(Original File) | 100% | 78.10% | 77.00% | 71.20% | 77.40% | 83.10% | 78.00%
50% | 0% | 71.30% | 71.60% | 63.50% | 72.50% | 73.30% | 71.40%
| 25% | 68.10% | 67.50% | 58.20% | 69.40% | 69.80% | 67.80%
| 50% | 63.10% | 63.00% | 52.70% | 66.50% | 66.80% | 63.70%
| 75% | 60.50% | 60.40% | 50.50% | 63.90% | 66.30% | 61.40%
| 100% | 55.50% | 56.80% | 47.00% | 60.70% | 64.60% | 57.80%
25% | 0% | 53.00% | 55.00% | 41.80% | 56.90% | 61.50% | 54.90%
| 25% | 52.40% | 53.40% | 39.80% | 55.50% | 61.00% | 53.80%
| 50% | 51.60% | 51.10% | 41.50% | 53.70% | 56.90% | 52.10%
| 75% | 51.30% | 49.70% | 39.80% | 51.90% | 55.10% | 50.80%
| 100% | 50.00% | 49.20% | 40.20% | 49.80% | 54.90% | 49.80%
Table 4: WSD under different intersection ratio with full-coverage setting Full coverage Setting (AVG) | senseval2 | senseval3 | semeval2007 | semeval2013 | semeval2015 | ALL
---|---|---|---|---|---|---
0% | 71.30% | 71.60% | 63.50% | 72.50% | 73.30% | 71.40%
20% | 72.80% | 72.30% | 64.00% | 72.90% | 76.30% | 72.60%
40% | 75.10% | 73.40% | 63.70% | 74.70% | 78.90% | 74.40%
60% | 75.70% | 75.00% | 65.30% | 74.90% | 79.30% | 75.20%
80% | 76.90% | 75.80% | 67.70% | 75.70% | 81.10% | 76.40%
Original LMMS+ARES(100%) | 78.10% | 77.00% | 71.20% | 77.40% | 83.10% | 78.00%
Full coverage Setting (CONC) | senseval2 | senseval3 | semeval2007 | semeval2013 | semeval2015 | ALL
0% | 71.30% | 71.60% | 63.50% | 72.50% | 73.30% | 71.40%
20% | 72.40% | 71.90% | 64.20% | 72.70% | 75.70% | 72.30%
40% | 74.80% | 72.60% | 63.10% | 74.50% | 78.40% | 73.90%
60% | 75.70% | 74.30% | 64.80% | 74.40% | 79.20% | 74.90%
80% | 76.90% | 75.70% | 67.50% | 75.40% | 80.90% | 76.20%
Original LMMS+ARES(100%) | 78.20% | 77.10% | 71.00% | 77.30% | 83.00% | 77.90%

It is a typical issue that the source embedding may not have the same
vocabulary coverage. Therefore, the union of two meta embedding may not
necessarily have identical sense inventories as before. In our experiment, I
selected part of the LMMS and ARES to conduct experiments examining the effect
of different overlap percentages of two source embedding (intersection ratio)
of the meta embedding. Our experiments showed that if the union of source
embeddings do not cover enough words (for example, not covering all the sense
keys in the test set), a smaller intersection ratio will be better since it
will increase the vocabulary coverage. Since the source embeddings LMMS and
ARES are already full-coverage, a proportion of them is selected to conduct
experiments. The proportion is called “partial ratio” in this thesis. As it
can be seen in Table 3, when the partial ratio equals to 50%, the performance
is best if the intersection ratio equals 0. In this case, the meta embedding
covers all the sense embeddings as the original source embedding does, and
they have similar performance. When the partial ratio is 25%, the best
performance is still achieved if the intersection ratio equals to 0% although
it is not comparable to the performance of original source embedding. Under
this circumstance, the meta embedding covers 50% of the sense inventories.
However, if the union of the source embeddings already covers the full sense
inventories of WordNet, a large intersection may lead to an improvement on the
performance. In Table 4, a pattern can be observed that with the increase of
the intersection rate, the performance improves steadily.

##  § 5 Conclusion

This thesis applied multiple different meta word embedding methods to create
meta sense embedding with LMMS, SensEmBERT and ARES as source embeddings. The
experiments showed that meta embedding created via CONC and AVG achieved
better performance in WSD, which implies these two methods are strong baseline
methods for meta sense embedding. Vector space shift is an issue worth
consideration when more complex meta embedding methods such as SVD and AEME
are applied. A new meta embedding method VSP is proposed and this method
creates sense embedding with SOTA performance in WSD and WiC. In addition to
embedding quality, the experiments proved meta embedding can improve sense key
coverage by only using existing sense embedding, suggesting the significance
of meta embedding.

##  § 6 BCS Criteria

This section will discuss how this project demonstrated the British Computer
Society(BCS)’s six criteria.

###  § 6.1 Ability to apply skills gained during the degree

I acquired fundamental skills during my degree programme. This project is
related to some of the core courses such as COMP219 Advanced Artificial
Intelligence and  COMP 218 Decision, Computation and Language, in which I
learned about machine learning fundamentals such as gradient descent and
neural networks, as well as classical Disambiguation, approaches. In addition,
this project requires a good understanding of linear algebra since it is
related to vector space and projection, which I studied relevant concepts in
my MTH 007 Linear Algebra course.

###  § 6.2 Innovation and/or creativity

This project helped to foster my creativity significantly since this project
is a research-based project. Different from engineering projects such as
building a website using existing frameworks, our project aims to explore new
areas and propose novel solutions and new methods to challenges. For example,
as I already mentioned in § 3.5 and § 3.6, these two novel methods are
proposed to tackle the challenge of the vector space shift.

###  § 6.3 Synthesis of information

The term Meta Sense Embedding infers that this research project requires
knowledge in two aspects: meta embedding and sense embedding. Therefore, I
need to synthesise the knowledge of these two areas. I need to get familiar
with the existing meta embedding and know the construction of sense embedding
in order to adapt suitable meta embedding methods for sense embeddings.
Moreover, I gathered the experiments results and gave analysis to the
underlying patterns of the data. More discussion can be seen in the Experiment
and Evaluation section.

###  § 6.4 Meet a real need in a wider context

In terms of the real-world application, our work may benefit machine
translation. This is because word ambiguity is one significant factor of
translation. A higher-quality sense embedding, which is acquired using meta
sense embedding, may reduce the issue of word sense ambiguity. In terms of
this work’s meaning in academia, the methods discussed in meta sense embedding
and observations such as the effect of normalisation are also meaningful for
other meta embedding methods such as meta word embedding and meta sentence
embedding.

###  § 6.5 Ability to self-manage a significant piece of work

The whole project is complex since it is related to multiple meta sense
embedding methods and requires a large number of experiments. Many files are
generated such as meta embeddings and their corresponding projection matrix. I
developed my ability to handle complex projects by using tools such as Github
and self-studied shell scripts to automate the execution of the codes.

There are three significant components of the project: baseline meta embedding
framework, Auto Encoder methods, and evaluation framework. None of the
components is trivial and it requires careful adaptation to ensure the code
works correctly. For example, the code of the AutoEncoder, written in an early
version of TensorFlow, is already a complete project for an NLP paper. I
needed to confirm the version of related packages, the meaning of each
parameter and the underlying mechanism of AutoEncoder. I also developed
advanced code version control techniques. I used to create the new methods by
directly overwriting the parts of the original code. However, when I wanted to
reproduce the results of experiments I did before, it took time to recover the
original code. Therefore, I adopted better engineering practices by
maintaining GitHub for this project constantly. I also managed to simplify the
cumbersome evaluation section of the project. The evaluation of the WSD task
requires running the code on six different test sets. Moreover, some of the
meta embeddings require the loading of projection matrices which leads to
further complexity. It is laborious and time-consuming to execute them
manually, wait for the execution process and take down the results. I
alleviated this issue by studying shell scripts and the evaluation requires
much fewer manual operations than before.

Moreover, this thesis is more than an undergraduate writing assignment. This
thesis is not the end of our work of meta sense embedding. Instead, it is the
initial version of a paper which will ultimately be improved to the EMNLP
conference quality paper under the supervision of Professor Bollegala. When I
write this thesis, I intentionally make it more like an NLP conference paper
and keep improving its quality by learning from other top NLP conference
papers.

###  § 6.6 Critical self-evaluation of the process

Throughout the project, I constantly reflected on myself to see if there is
any possible mistakes and possible improvements. One thing I will stick to in
future work is to be careful about the experiments. At the beginning of the
project, I did not have a very clear understanding of the components of the
WSD framework. In order to make sure I used the evaluation framework
correctly, I compared the result I produced with the paper. If and only if
they are consistent, I will then execute the following steps.

Another good thing about the whole process is that I am willing to try
different new ideas and successfully discovered the cause of the
unsatisfactory experiment result.

The thing that I need to improve is that a more extensive and intensive paper
reading should have been done. At the beginning of the project, I only treated
the source embeddings and meta embedding methods as black-box. Thanks to the
guidance of my supervisor, I started to read the papers more extensively and
thoroughly. After I read the paper which proposed source embeddings and
surveys on meta embeddings and Word Sense Disambiguation, I had a much better
understanding of these tasks and was able to explain the experiments’ result
better. I realised that utilising sense embedding is not the only way to
resolve the word sense embedding problem. Furthermore, I found I generated new
ideas more frequently than before. In summary, what I learned from this
project is that it is important to treat the experiments seriously, be brave
in the face of challenges, absorb knowledge from outside resources (such as
research papers) and discuss with the supervisor in this domain to help your
own work.