# Static Fuzzy Bag-of-Words: a lightweight sentence embedding algorithm

Matteo Muffo${\dagger}$ *, Roberto Tedesco${\dagger}$, Licia
Sbattella${\dagger}$ Vincenzo Scotti${\dagger}$
${\dagger}$DEIB, Politecnico di Milano
Via Golgi 42, 20133, Milano (MI), Italy
*Indigo.ai
Via Torino 61, 20123, Milano (MI), Italy
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

The introduction of embedding techniques has pushed forward significantly the
Natural Language Processing field. Many of the proposed solutions have been
presented for word-level encoding; anyhow, in the last years, new mechanism to
treat information at an higher level of aggregation, like at sentence- and
document-level, have emerged. With this work we address specifically the
sentence embeddings problem, presenting the _Static Fuzzy Bag-of-Word_ model.
Our model is a refinement of the Fuzzy Bag-of-Words approach, providing
sentence embeddings with a predefined dimension. SFBoW provides competitive
performances in Semantic Textual Similarity benchmarks, while requiring low
computational resources.

##  1 Introduction

Natural Language Processing (NLP) has gained a lot of traction in the last
years, especially thanks to the introduction of learnt semantic
representations. Those representations (usually) called embeddings are, in the
textual context, real-valued vectors able to represent the semantic meaning of
words, sentences or even documents, in a Euclidean space. These vectors are
features generated by models trained using a _self-supervised_ approach on a
huge corpus of unlabelled text. According to ?), leveraging features obtained
through a self-supervised approach instead of “hand-selected” features it’s of
crucial importance for NLP.

Textual learnt representations immediately turned out to be significative in
many NLP tasks, from more simple ones, like Part-Of-Speech (POS) tagging,
Named Entity Recognition (NER) and language modelling [Collobert and Weston
(2007, Collobert and Weston (2008, Collobert et al. (2011], to more complex
problems such as Machine Translation [Sutskever et al. (2014], and even
Conversational Systems [Sordoni et al. (2015]. In fact, this representation
significantly moved forward the state of the art.

Results in the aforementioned tasks have been boosted mostly thanks to learnt
word-level semantic vectors, i.e. _word embeddings_ ; however, as pointed out
by ?), for many problems (like web search, question answering and image
captioning) having access to higher level representations is crucial: this is
where _sentence embeddings_ find their usefulness.

Recent outcomes show that _contextual_ representations, learnt through
Transformer Networks [Vaswani et al. (2017], provide better performances in
all those tasks and are slowly substituting “static” embeddings. Even though
the results of these models are remarkable, their usability is strongly
restricted because of their high demand in terms of computational resources,
so we decided to focus on more lightweight solutions.

With this work, we introduce the Static Fuzzy Bag-of-Words (SFBoW) model, an
improvement of the work from ?), for sentence embedding. SFBoW is
characterised by interesting results in Semantic Textual Similarity (STS),
without the necessity of high computational power.

The rest of this document is organised as follows: Section 2 presents the
related works in the field of learnt semantic representations, Section 3
introduces our SFBoW model, Section 4 describes the planned experiments to
assess the quality of our model, Section 5 presents the results of experiments
and, finally, Section 6 sums up the entire work.

##  2 Related works

Our work revolves around the concept of _vector semantics_ : the idea that the
meaning of a word or a sentence can be modelled as a vector [Osgood et al.
(1958].

First steps on this subject were made in Information Retrieval (IR) context
with the vector space model [Salton (1971, Sparck Jones (1986], where
documents and queries were represented as high dimensional (vocabulary size)
sparse embedding vectors. In this model, each dimension is used to represent a
word, so that given a vocabulary $\mathcal{V}$:

  * •

A word $w_{i}\in\mathcal{V}$, with
$i\in\left[1,|\mathcal{V}|\right]\subseteq\mathbb{N}$, is expressed as a so
called “one hot” binary vector
$\mathbf{v}_{w_{i}}\in\mathds{1}^{|\mathcal{V}|}$, where, calling
$v_{w_{i},j}$ the $j$-th element of the word vector, it holds that
$v_{w_{i},j}=1\Longleftrightarrow j=i$;

  * •

A sentence $S$ is expressed as vector
$\bm{\mu}_{S}\in\mathbb{N}^{|\mathcal{V}|}$, where $\mu_{S,i}$, the $i$-th
element of vector $\bm{\mu}_{S}$, represents the number of times, $c_{S,i}$,
word $w_{i}$ appears in sentence $S$.

The sentence representation, used also for text documents, is called Bag-of-
Words (BoW), and can be summarised in the following equation:

$\bm{\mu}_{S}=\sum_{i=1}^{|\mathcal{V}|}c_{S,i}\cdot\mathbf{v}_{w_{i}}$ (1)

As pointed out by ?), these embedding models needed to be replaced because of
their sparsity, which made them time/memory consuming, and the induced
orthogonality among vectors with similar meanings.

###  2.1 Word and sentence embeddings: a brief summary

Word embeddings refer to the dense semantic vector representation of words.
Approaches can be divided into: _prediction-based_ and _count-based_ [Baroni
et al. (2014].

The former group identifies the embeddings obtained through the training of
models for next/missing word prediction given a context, it was started by ?)
and encompasses models like Word2Vec [Mikolov et al. (2013a, Mikolov et al.
(2013b] and FastText ?). The latter group refers to the embeddings obtained
leveraging words co-occurrence counts in a corpus, it has a longer history,
starting from Latent Semantic Indexing (LSI) over term-document matrices
[Deerwester et al. (1989] up to more recent solutions like GloVe from ?).

Among the aforementioned models, there are some (like Word2Vec, GloVe or
FastText) belonging to a class called _shallow_ models, where the embedding of
a word $w_{i}$ can be simply extracted through lookup over the rows of the
embedding matrix $\mathbf{W}\in\mathbb{R}^{|\mathcal{V}|\times d}$, with $d$
being the desired dimensionality of the embedding space. Given the word
(column) vector $\mathbf{v}_{w_{i}}$, the corresponding word embedding
$\mathbf{u}_{w_{i}}\in\mathbb{R}^{d}$ can be computed as (see Section 2.2):

$\mathbf{u}_{w_{i}}=\mathbf{W}^{\top}\cdot\mathbf{v}_{w_{i}}$ (2)

More recently, the introduction of transformer-based Language Models (LMs),
like BERT [Devlin et al. (2018] (and all its variants) or the two versions of
GPT [Radford et al. (2018, Radford et al. (2019], has spread the concept of
_contextual embeddings_ ; such embeddings proved to be particularly helpful
for a wide variety of NLP problems, as shown by the leader-boards of many NLP
benchmarks like SuperGLUE [Wang et al. (2019].

The inherent hierarchical structure of the human language makes it hard to
understand a text from single words; thus, the birth of higher level semantic
representations for sentences, which are the sentence embeddings, was just a
natural consequence. As for the Word embeddings, also sentence embeddings are
organised into two groups: _parametrised_ and _non-parametrised_ , depending
on whether the model requires parameter training or not.

First examples of parametric models come from ?), with the Skip-Thoughts
vectors, later followed by Sent2Vec [Pagliardini et al. (2018], which
generalised Word2Vec. Transformer LMs influenced the state of the art also at
this level, a clear example is given by Sentence-BERT [Reimers and Gurevych
(2019], obtained by fine-tuning on two Natural Language Inference (NLI)
corpora: the Stanford NLI [Bowman et al. (2015] and the Multi-Genre NLI
[Williams et al. (2018].

Non-parametric models, instead, show that simply aggregating the information
from pre-trained word embeddings, for example through averaging [Arora et al.
(2019, Yang et al. (2019], is sufficient to represent represent higher-level
entities like sentences and paragraphs.

All these models rely on the assumption that cosine similarity is the correct
metric to compute “meaning distance” between sentences, and this is why
parametric models are trained specifically to minimise this measure for
similar sentences and maximise it for dissimilar sentences. However this may
not be the only and best measure. In fact ?) proposed to follow a fuzzy set
representation of sentences and to rely on fuzzy Jaccard similarity, instead
of the cosine one. As a result, their DynaMax model outperformed many non-
parametric models and performed comparably to parametric ones under cosine
similarity measurements, even if competitors were trained directly to optimise
that metric, while DynaMax approach was completely unrelated to that
objective.

The use of fuzzy sets to represent documents is not new, in fact it was
already proposed by ?). However, they relied on a different approach to
compute fuzzy membership, with respect to ?), which lead to inferior results.

###  2.2 Sentence embeddings: Fuzzy Bag-of-Words and DynaMax model

The Fuzzy Bag-of-Words (FBoW) model for text representation, proposed by ?),
then generalised and improved by ?) with their DynaMax through the
introduction of a better similarity metric, represent the starting point of
our work, which will be described in Section 3.

The BoW approach, described at the beginning of Section 2, can be seen as a
multi-set representation of text. As such it enables to measure similarity
between two sentences with set similarity measures, like Jaccard, Otsuka and
Dice indexes. These indexes, as ?) pointed out, share all a common pattern to
measure the similarity $\sigma$ between two sets $A$ and $B$:

$\sigma\left(A,B\right)=n_{\textit{shared}}\left(A,B\right)/n_{\textit{total}}\left(A,B\right)$
(3)

Where $n_{\textit{shared}}\left(A,B\right)$ denotes the count of shared
elements and $n_{\textit{total}}\left(A,B\right)$ denotes the count of total
elements. In particular, the Jaccard index is defined as:

$\sigma_{\textit{Jaccard}}\left(A,B\right)=\left|A\cap B\right|/\left|A\cup
B\right|$ (4)

Simple set similarity is however a rigid approach as it allows for some degree
of similarity when the very same words appear in both sentences, but fails in
presence of synonyms. This is where _Fuzzy Sets theory_ [Zadeh (1965] comes
handy: in fact, fuzzy sets enable to interpret each word in $\mathcal{V}$ as a
singleton set and measure the degree of membership of any word to this one-
element set as the similarity between the two considered words [Zhao and Mao
(2018].

In their FBoW model, ?) proposed to work exactly in this way:

  * •

each single word $w_{i}$ is interpreted as a singleton
$\left\\{w_{i}\right\\}$; thus, the membership degree of any word $w_{j}$ in
the vocabulary (with $j\in\left[1,|\mathcal{V}|\right]\subseteq\mathbb{N}$)
with respect to this set is computed as the similarity $\sigma$ between
$w_{i}$ and $w_{j}$. These similarities can be used to fill a
$|\mathcal{V}|$-sized vector $\hat{\textbf{v}}_{w_{i}}$ used to provide the
fuzzy representation of $w_{i}$ (the $j$-th element
$\hat{\textbf{v}}_{w_{i},j}$ being $\sigma\left(w_{i},w_{j}\right)$);

  * •

a sentence $S$ is simply defined through the fuzzy union operator, which is
determined by the $\max$ operator over the membership degrees. In this case
the $S$ is represented by a vector of $|\mathcal{V}|$ elements.

Then, to reduce the dimension of the vector for $S$, the ?) generalised the
FBoW approach, computing the _fuzzy embedding_ of a word singleton as:

$\hat{\textbf{v}}_{w_{i}}=\mathbf{U}\cdot\mathbf{u}_{w_{i}}=\mathbf{U}\cdot\mathbf{W}^{\top}\cdot\mathbf{v}_{w_{i}}$
(5)

where, $\mathbf{W}\in\mathbb{R}^{|\mathcal{V}|\times d}$ is a word embedding
matrix (defined as in Section 2.1), $\mathbf{u}_{w_{i}}$ is defined in
Equation 2 and $\mathbf{U}\in\mathbb{R}^{u\times d}$ (with $u$ being the
desired dimension of the fuzzy embeddings) is the _universe matrix_ , derived
from the _universe set_ $U$, which is defined as “the set of all possible
terms that occur in a certain domain”. The generalised FBoW produces vectors
of $u$ elements, where $u=|U|$.

Finally, given the fuzzy embeddings of the words in a sentence $S$, the
generalised FBoW representation of $S$ is a vector $\hat{\bm{\mu}}_{S}$ whose
$j$-th element $\hat{\mu}_{S,j}$ ($j\in\left[1,u\right]\subseteq\mathbb{N}$)
can be computed as:

$\hat{\mu}_{S,j}=\max_{w_{i}\in S}c_{S,i}\cdot\hat{v}_{w_{i},j}$ (6)

where $c_{S,i}$ and $\hat{v}_{w_{i},j}$ are, respectively, the number of
occurrences of word $w_{i}$ in sentence $S$ and the $j$-th element of the
$\hat{\textbf{v}}_{w_{i}}$ vector.

The universe set can be defined in different ways; in fact, ?) suggested some
alternatives for the choice of the universe set, and hence the universe
matrix. Among the proposed solutions, in their best performing algorithm for
fuzzy sentence embeddings, the DynaMax, they choose to build the universe
matrix from the word embedding matrix, stacking solely the embedding vectors
of the words appearing in the sentences to be compared.

Notice that in this way the resulting universe matrix is not unique, as a
consequence neither are the embeddings. This condition can be noticed from the
description of the algorithm and from the definition of the universe matrix:
when comparing two sentences $S_{a}$ and $S_{b}$, the universe set $U$ used in
their comparison is $U\equiv S_{a}\cup S_{b}$, so the resulting sentence
embeddings have size $u=\left|U\right|=\left|S_{a}\cup S_{b}\right|$. In fact,
the universe matrix is given by

$\mathbf{U}=\begin{bmatrix}\mathbf{u}_{w_{i}}\forall w_{i}\in
U\end{bmatrix}^{\top}$ (7)

This characteristic is unfortunate as, for example in the field of IR, it
requires a complete re-encoding of the entire document achieve for each query.

The real improvement of DynaMax is in the introduction of the fuzzy Jaccard
index to compute the semantic similarity between two sentences $S_{a}$ and
$S_{b}$ (see Equation 8), rather than the generalisation of the FBoW, which
replaced the cosine similarity employed by ?).

$\hat{\sigma}_{\textit{Jaccard}}\left(\hat{\bm{\mu}}_{S_{a}},\hat{\bm{\mu}}_{S_{b}}\right)=\frac{\sum_{i=1}^{u}\min\left(\hat{\mu}_{S_{a},i},\hat{\mu}_{S_{b},i}\right)}{\sum_{i=1}^{u}\max\left(\hat{\mu}_{S_{a},i},\hat{\mu}_{S_{b},i}\right)}$
(8)

##  3 Static Fuzzy Bag-of-Words model

Figure 1: Visualization of the Sentence Embedding computation process using
SFBoW.

Starting from the the DynaMax, evolved from the FBoW model, we developed our
follow up aimed at providing a unique matrix $\mathbf{U}$ and thus embeddings
with a predefined dimension. In Figure 1 is represented the visualization of
our approach.

###  3.1 Concerning the word embeddings

Word embeddings play a central role in our algorithm as they also provide the
start point of the construction of the universe matrix. For the scope of this
work, we leverage pre-trained shallow models (more details in Section 4.1) for
two main reasons:

  * •

the model is encoded in a matrix where each row corresponds to a word;

  * •

we want to provide a sentence embedding that doesn’t require training, easing
its accessibility.

The vocabulary of these models, composed starting from all the tokens in the
training corpora, is usually bigger than the English vocabulary, as it
contains named entities, incorrectly spelled words, non-existing words, URLs,
email addresses, etc. To reduce the computational effort needed for
construction and usage of the universe matrix, we have considered some subsets
of the employed word embedding model’s vocabulary. Depending on the
experiment, we work with either the 100000 most frequently used terms, the
50000 most frequently used terms (terms frequencies are given by the corpora
used to train the word embedding model) or the subset composed of all the
spell-checked terms present in a reference English dictionary (obtained
through the Aspell English spell-checker111http://aspell.net).

In the following sections, the $\check{\mathbf{W}}$ symbol refers to these as
_reduced_ word embedding matrices/models.

###  3.2 Universe matrix

During the experiments, we try three main approaches to build the universe
matrix $\mathbf{U}$: the first two – proposed, but not explored, by the
original authors of DynaMax – consist, respectively, in the usage of clustered
embedding matrix and an identity matrix with the rank equal to the size of the
word embeddings. The last approach, instead, consists in the application of a
multivariate analysis techniques to the word embedding matrix to build the
universe one. In the following formulae we refer to $d$ as the dimensionality
of the word embedding vectors, while the SFBoW embedding of the singleton of
word $w_{i}$ is represented as $\check{\textbf{v}}_{w_{i}}$.

#### Clustering

The idea is to group the embedding vectors into clusters, and use their
centroids; in this way the fuzzy membership will be computed over the clusters
– which are expected to host semantically similar words – instead of all the
word singletons. The universe set is thus built out of abstract entities only,
which are the centroids. Considering $k$ cluster centroids, the universe
matrix $\mathbf{U}=\mathbf{K}^{\top}\in\mathbb{R}^{k\times d}$, and thus SFBoW
$k$-dimensional embedding $\check{\textbf{v}}_{w_{i}}$ of the singleton of
word $w_{i}$ is

$\check{\textbf{v}}_{w_{i}}=\mathbf{K}^{\top}\cdot\mathbf{u}_{w_{i}}=\begin{bmatrix}\mathbf{k}_{1},\ldots,\mathbf{k}_{k}\end{bmatrix}^{\top}\cdot\mathbf{u}_{w_{i}}=\mathbf{K}^{\top}\cdot\mathbf{W}^{\top}\cdot\mathbf{v}_{w_{i}}$
(9)

where $\mathbf{k}_{j}$, the $j$-th (with
$j\in\left[1,k\right]\subseteq\mathbb{N}$) column of $\mathbf{K}$, corresponds
to the centroid of the $j$-th cluster. This approach generates $k$-dimensional
word embeddings and sentence embeddings.

#### Identity

Alternatively, instead of looking for a group of semantically similar words
that may possibly form a significant group, useful for semantic similarity, we
consider the possibility of re-using the word embedding dimensions (features)
to represent the semantic content of a sentence. So, we just use the identity
matrix as the universe: $\mathbf{U}=\mathbf{I}$, with
$\left|\mathbf{I}\right|=d\times d$, so that
$\check{\textbf{v}}_{w_{i}}\in\mathbb{R}^{d}$ is

$\check{\textbf{v}}_{w_{i}}=\mathbf{I}\cdot\mathbf{u}_{w_{i}}=\mathbf{I}\cdot\mathbf{W}^{\top}\cdot\mathbf{v}_{w_{i}}$
(10)

This approach generates $d$-dimensional word embeddings and sentence
embeddings.

#### Multivariate analysis

The same idea moves our multivariate analysis proposal. In fact, judging by
previous results, word embeddings aggregated in the correct way might be
sufficient to provide a semantically valid representation of the sentence they
compose. What can bring better results, might be as simple as roto-translate
the reference system of the embedding representation. In this sense, we
propose to use to compute the fuzzy membership, and hence the fuzzy Jaccard
similarity index, over these dimensions resulting from roto-translation,
expecting that this “new perspective” will expose better the semantic content.
So, defining $\mathbf{U}=\mathbf{M}$, where $\mathbf{M}$ is the transformation
matrix, with $\left|\mathbf{M}\right|=d\times d$, we have that
$\check{\textbf{v}}_{w_{i}}\in\mathbb{R}^{d}$ is

$\check{\textbf{v}}_{w_{i}}=\mathbf{M}\cdot\mathbf{u}_{w_{i}}=\mathbf{M}\cdot\mathbf{W}^{\top}\cdot\mathbf{v}_{w_{i}}$
(11)

This approach generates $d$-dimensional word embeddings and sentence
embeddings.

Clustering and multivariate analysis can be applied to the whole embedding
vocabulary or the subsets of the vocabulary introduced in Section 3.1. A part
from reducing the computational time, we do so also to see if these subsets
are sufficient to provide a useful representation.

##  4 Experiments with the Static Fuzzy Bag-of-Words

In order to find the best solution in terms of word embedding matrix and
universe matrix, we explored various possibilities. Then, to measure the
goodness of our sentence embeddings, we leveraged a series of STS tasks and
compared the results with preceding models.

###  4.1 Word embeddings

For what concerns the word embeddings, we have decided to work with a
selection of four models:

  * •

Word2Vec, with $300$-dimensional embeddings [Mikolov et al. (2013b];

  * •

GloVe, with $300$-dimensional embeddings [Pennington et al. (2014];

  * •

FastText, with $300$-dimensional embeddings [Bojanowski et al. (2017];

  * •

Sent2Vec, with $700$-dimensional embeddings [Pagliardini et al. (2018].

As shown by the word embedding models list, we are employing also a Sent2Vec
sentence embedding model. In fact the embedding matrix of this model can be
used for word embeddings too [Gupta et al. (2019]. During the experiments we
focused on the universe matrix construction, for this reason we relied on pre-
trained models for word embeddings, available on the web.

###  4.2 Universe matrices

The universe matrices we considered, are divided in three buckets, as
described in Section 3.2.

#### Clustering

Universe matrices built using clustering leverage three different algorithms:
k-Means [MacQueen (1967], Spherical k-Means [Hornik et al. (2012] and DBSCAN
[Ester et al. (1996].

We selected k-Means and Spherical k-Means because they usually lead to good
clustering results, the latter was specifically designed for textual purposes,
with a low demand in terms of time and computation resources. For all
algorithms we considered the same values for $k$ (the number of centroids),
which were 100, 1000, 10 000 and 25 000. For all the values of $k$, we
performed clustering on different subsets of the vocabulary: k-Means was
applied on the whole English vocabulary as well as to the top 100 000
frequently used words subset, while Spherical k-Means was applied to the
subset of the first 50 000 frequently used words (in order to reduce
computational time).

We explored also a density based algorithm (DBSCAN), which does not require to
define in advance the number of clusters, using euclidean and cosine distance
between the word embedding. For the former case we varied the radius of the
neighbourhood $\varepsilon$ between $3$ and $8$ and worked over the same two
subsets considered for k-Means, for the latter $\varepsilon$ was between $0.1$
and $0.55$ and it was applied over the subset of the first 50 000 frequently
used words (for computational reasons, as we did for Spherical k-Means).

#### Identity

This approach consists in using the identity matrix as universe, in this way
the singletons we use to compute the fuzzy membership are the dimensions of
the word embeddings, which corresponds to the learnt features. This is the
most lightweight method as is just requires to compute the word embeddings of
a sentence and then the fuzzy membership over the same $d$ dimensions.

#### Multivariate analysis

We adopted the Principal Component Analysis (PCA) to get a rotation matrix
which will serve as universe matrix to the SFBoW. In fact through PCA the
$d$-dimensional word embedding vectors are decomposed along the $d$ orthogonal
directions of their variance. These components are then reordered in
decreasing order of explained variance and used to represent our semantic
fuzzy sets.

The principal component of the reduced word embedding matrix
$\check{\mathbf{W}}$ are described by the matrix
$\mathbf{T}=\mathbf{P}^{\top}\cdot\check{\mathbf{W}}$, where $\mathbf{P}$ is a
$d\times d$ matrix whose columns are the eigenvectors of the matrix
$\check{\mathbf{W}}^{\top}\cdot\check{\mathbf{W}}$. With our approach, the
matrix $\mathbf{P}^{\top}$, sometimes called the _whitening_ or _sphering
transformation matrix_ , serves as universe matrix $\mathbf{U}$. In this way,
the SFBoW embedding of a word singleton becomes

$\check{\textbf{v}}_{w_{i}}=\mathbf{P}^{\top}\cdot\mathbf{u}_{w_{i}}=\mathbf{P}^{\top}\cdot\check{\mathbf{W}}^{\top}\cdot\mathbf{v}_{w_{i}}$
(12)

As for the clustering approach, we experimented with both the whole vocabulary
and the top 100 000 frequently used words.

###  4.3 Data

The evaluation of our SFBoW is done through a series of reference benchmarks;
we selected the STS benchmark series, one of the tasks of the International
Workshop on Semantic Evaluation
(SemEval)222https://aclweb.org/aclwiki/SemEval_Portal.

SemEval is an ongoing series of evaluations on computational semantics; among
these evaluations, the STS benchmark has become a reference for scoring of
sentence embedding algorithms. In fact, all the previous models we are
considering for a comparison have been benched against STS; this is because
the benchmark highlights a model capability to provide a meaningful semantic
representation by scoring the correlation between model’s and human’s
judgements. For this reason, and also to make comparisons possible, we decided
to evaluate SFBoW on STS.

We worked only on the English language, using the editions of STS from 2012 to
2016 [Agirre et al. (2012, Agirre et al. (2013, Agirre et al. (2014, Agirre et
al. (2015, Agirre et al. (2016]. Each year, a collection of corpora coming
from different sources has been created and manually labelled, in Table 1 is
possible to have a reference in terms of support for each edition. Thanks to
the high number of samples, we are confident about the robustness of our
results.

Table 1: Support of the corpora of the STS benchmark series. Edition | STS ’12 | STS ’13 | STS ’14 | STS ’15 | STS ’16 | Total
---|---|---|---|---|---|---
Number of sentence pairs | 5250 | 2250 | 3750 | 3000 | 1186 | 15 436

The samples constituting the corpora are pair of sentences with a human-given
similarity score (the _gold labels_). The provided score is a real valued
index obtained averaging those of multiple crowd-sourced workers, and is
scaled in a $\left[0,1\right]\in\mathbb{R}$ interval. The final goal of our
work is to provide a model able to provide a score as close as possible to
that of the humans.

###  4.4 Evaluation

To assess the quality of our model, we used it to compute the similarity score
between the sentence pairs provided by the five tasks, and we compared the
output with the target labels. The results are computed as the correlation
between the similarity score produced by SFBoW and the human one, using the
Spearman’s $\rho$ measure, as suggested by ?). The index SFBoW employs to
compute word similarity is the fuzzy Jaccard similarity index, as suggested by
?).

To have terms of comparison, we establish a baseline through the simplest
models possible, the average of the word embedding in a sentence, leveraging
three different word embedding models: Word2Vec, GloVe and FastText. We also
provide results from more complex models: SIF weighting (applied to GloVe),
Sent2Vec, DynaMax (built using Word2Vec, GloVe and FastText) and Sentence-
BERT.

All the sentence embedding models, except DynaMax, and the baselines are
scored using the results of cosine similarity; DynaMax scores are instead
obtained using the fuzzy Jaccard similarity index.

##  5 Results

The results of the Spearman’s $\rho$ correlation in the STS benchmark of our
SFBoW are reported in the last three rows of Table 2. The reported values
belong to the FSBoW configurations that achieved a best score, among the
variants we considered for the experiments, in at least one task; if
interested in the complete experimental results reporting all the SFBoW
configurations, please refer to Appendix A.

Table 2: Comparison in the results over the STS benchmark. The last block
shows the best performing SFBoW models we evaluated. All averages are
expressed as: $\textit{average score}\pm\textit{standard deviation}$. Bold
values represent the best score in each column, underlined values are the
second best score in the column.

| Results (Spearman’s $\rho$)
---|---
STS | Average | Weighted average
’12 | ’13 | ’14 | ’15 | ’16
Word2Vec a | $55.46$ | $58.23$ | $64.05$ | $67.97$ | $66.28$ | $62.4\pm 4.78$ | $61.21\pm 5.04$
GloVe a | $53.28$ | $50.76$ | $55.63$ | $59.22$ | $57.88$ | $55.35\pm 3.06$ | $54.99\pm 2.80$
FastText a | $58.82$ | $58.83$ | $63.42$ | $69.05$ | $68.24$ | $63.67\pm 4.40$ | $62.65\pm 4.20$
SIF weighting b | $56.04$ | $\underline{62.74}$ | $64.29$ | $69.89$ | $70.71$ | $64.73\pm 5.33$ | $62.84\pm 5.54$
Sent2Vec | $56.26$ | $57.02$ | $65.82$ | $74.46$ | $69.01$ | $64.51\pm 7.00$ | $63.21\pm 7.13$
DynaMax c | $55.95$ | $60.17$ | $65.32$ | $73.93$ | $71.46$ | $65.37\pm 6.73$ | $63.53\pm 6.92$
DynaMax b | $57.62$ | $55.18$ | $63.56$ | $70.40$ | $71.36$ | $63.62\pm 6.53$ | $62.25\pm 5.85$
DynaMax d | $61.32$ | $61.71$ | $66.87$ | $\underline{76.51}$ | $\underline{74.71}$ | $\underline{68.22}\pm 6.37$ | $\underline{66.71}\pm 6.1$
Sentence-BERT | $\mathbf{72.27}$ | $\mathbf{78.46}$ | $\mathbf{74.90}$ | $\mathbf{80.99}$ | $\mathbf{76.25}$ | $\mathbf{76.57}\pm 2.98$ | $\mathbf{75.81}\pm 3.27$
SFBoW d, e, f | $61.31$ | $51.21$ | $\underline{67.47}$ | $72.90$ | $73.88$ | $65.35\pm 8.37$ | $64.55\pm 7.20$
SFBoW d, g, h | $\underline{61.42}$ | $51.36$ | $66.44$ | $72.74$ | $73.72$ | $65.14\pm 8.21$ | $64.32\pm 7.00$
SFBoW d, g, i | $60.03$ | $51.96$ | $66.36$ | $72.39$ | $73.25$ | $64.80\pm 7.99$ | $63.81\pm 6.93$

  * a

Used as baseline.

  * b

Built upon a GloVe model for word embeddings.

  * c

Built upon a Word2Vec model for word embeddings.

  * d

Built upon a FastText model for word embeddings.

  * e

Best average score.

  * f

Universe matrix is the identity matrix.

  * g

Universe matrix is the PCA projection matrix.

  * h

Universe matrix is built from the English vocabulary.

  * i

Universe matrix is built from the top 100 000 most frequent words.

Among the four word embeddings models we tried, the best performing is
FastText, confirming the results of ?). Concerning the choice of the universe
matrix, instead, the best scores are achieved either with Identity matrix or
with PCA rotation matrix, highlighting how the features described by the word
embeddings provide a better description of the semantic content of sentences.

About the choice of the universe matrix, clustering provided poor results, so
Table 2 reports only the scores from Identity matrix and PCA. Density based
clustering turned out to give meaningless results, for this reason its
analysis is omitted. k-Means clustering, instead, gave more promising results,
but still inadequate if compared to Identity matrix and PCA. The analysis of
Within Clusters Sum-of-Squares (WCSoS) and Between Clusters Sum-of-Squares
(BCSoS) showed that there’s still room for improvement so, as a future work,
we plan to further investigate this topic.

For the sake of readability, in in Figure 2a, we report only the clustering
analysis of the universe matrices extracted from the FastText embeddings, as
they turned out to provide the final best results. The analysis of clustering
results is conducted on the WCSoS and BCSoS for the different values of the
number of clusters $k$. In fact, from these graphs we can see that for all
k-Means approaches the _knee-elbow_ point is positioned almost in the same
spot, however the WCSoS and BCSoS are particularly high for the simple k-Means
clusterings when compared to the Spherical one. This might seem good in terms
of BCSoS, but isn’t for the WCSoS, which is preferable to be low. Similarly we
have the inverse situation when considering Spherical k-Means against the
standard one, where the BCSoS is too low.

This behaviour highlights how none of the proposed clustering approaches
behaves better than another when considered only in terms of clustering
performances; however Spherical k-Means leads to a more promising universe
matrix, among the clustering techniques we analysed (see Appendix A).

Interesting observations about PCA, instead, are reported in Figure 2b. Here
we show the evolution of the explained variance with the number of retained
components (the graph is cut at $300$ components because all the word
embeddings matrices except Sent2Vec have $300$ dimensions and because all of
them reach and pass the “elbow” within that point). Word2Vec and FastText are
the “fastest” to converge, they retain most of the variance by the 50-th
component, while Sent2Vec is the slowest, this is most probably due to the
fact that Sent2Vec is built with more than double the dimensions of other
embedding algorithms, hence the vectors are more scattered in the hyperspace.

(a)

(b)

Figure 2: Visualisation of the different analyses for the construction of the
universe matrices.

As premised we compare our results with three baseline models and other
sentence embedding approaches, all reported in Table 2. The first group of
scores is from the baselines, the second one is from other sentence embedding
models and, finally, the last group is from our SFBoW model. Additionally, the
best values in each column are highlighted in bold, while the second ones are
underlined.

Sentence-BERT [Reimers and Gurevych (2019] outperforms all the other models in
every task. This was predictable as in relies on a much bigger feature
extraction model and was trained on a much bigger corpus. However this model
requires a considerably higher computational effort without an equally
consistent difference in performances. In fact BERT alone requires more than
100 million parameters just for its base version (and above 300 millions for
the large one), hence taking a lot of (memory) space, not to mention the
amount of time necessary for the self-supervised training and the fine-tuning.

Differently, non-parametric models (like those from ?) and ?) as well as
SFBoW) or shallow parametric ones (Sent2Vec from ?)) require way less
parameters – just those for the embedding matrix
$\left|\mathcal{V}\right|\times{d}$ – and little, if any, training.
Nevertheless, given the low computational demand, they’re all able to reach
significant performances.

When compared to other models, SFBoW results are interesting: either
considering majority of tasks with higher Spearman’s $\rho$ rank or higher
average score, it outperforms all the baselines, as well as SIF weighting and
Sent2Vec. Finally we see as our model performs closely to its predecessor,
especially considering the weighted average of the single tasks results. SFBoW
bests out DynaMax in STS 2014 and gets almost the same results in STS 2012
(the difference of the two $\rho$ is $0.01$), which are the first two corpora
in terms of samples; however the significant difference in STS 2013 goes in
favour of DynaMax.

About the comparison against DynaMax, it is worth to underline a few
additional points: first of all, in both cases fuzzy Jaccard similarity
correlates better with human judgment as measure of sentence similarity.
Secondly both models manage to achieve better results when using FastText word
embedding, possibly underling that they lend better than other models at
sentence level combination; this is also shown by the baseline performances.
Finally we remind that DynaMax, differently from SFBoW, generates embeddings
with variable size, which results in a limited applicability.

##  6 Conclusions

With this work we have proposed SFBoW, a refinement of the FBoW model for
sentence embedding. As its predecessor, our model is non-parametric (i.e. it
doesn’t require any training) and relies on fuzzy sets theory. In fact, it
leverages the fuzzy Jaccard index to compute semantic similarity from the
sentence embeddings (differently from most of the other models, which rely on
cosine similarity).

Even if SFBoW doesn’t achieve state-of-the-art performances on the considered
STS benchmark, our solution performs comparably to its predecessor, while
enabling the possibility of re-usable embeddings as their dimension is fixed.
Moreover, as can be seen from the results, it outperforms in the majority of
tasks all the other compared models except Sentence-BERT, without the need of
specific training or fine tuning on sentence similarity corpora and still
being as lightweight as possible.

As a future work, we plan to further investigate clustering techniques and, in
general, other methodologies for computing the universe matrix.

## References

  * [Agirre et al. (2012]  Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre.  2012\.  SemEval-2012 task 6: A pilot on semantic textual similarity.  In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385–393, Montréal, Canada, 7-8 June. Association for Computational Linguistics.
  * [Agirre et al. (2013]  Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo.  2013\.  SEM 2013 shared task: Semantic textual similarity.  In Second Joint Conference on Lexical and Computational Semantics (SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 32–43, Atlanta, Georgia, USA, June. Association for Computational Linguistics.
  * [Agirre et al. (2014]  Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe.  2014\.  SemEval-2014 task 10: Multilingual semantic textual similarity.  In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 81–91, Dublin, Ireland, August. Association for Computational Linguistics.
  * [Agirre et al. (2015]  Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe.  2015\.  SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability.  In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 252–263, Denver, Colorado, June. Association for Computational Linguistics.
  * [Agirre et al. (2016]  Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe.  2016\.  SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation.  In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 497–511, San Diego, California, June. Association for Computational Linguistics.
  * [Arora et al. (2019]  Sanjeev Arora, Yingyu Liang, and Tengyu Ma.  2019\.  A simple but tough-to-beat baseline for sentence embeddings.  In 5th International Conference on Learning Representations, ICLR 2017.
  * [Baroni et al. (2014]  Marco Baroni, Georgiana Dinu, and Germán Kruszewski.  2014\.  Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors.  In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 238–247, Baltimore, Maryland, June. Association for Computational Linguistics.
  * [Bengio et al. (2003]  Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin.  2003\.  A neural probabilistic language model.  Journal of machine learning research, 3(Feb):1137–1155.
  * [Bengio et al. (2013]  Yoshua Bengio, Aaron Courville, and Pascal Vincent.  2013\.  Representation learning: A review and new perspectives.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, Aug.
  * [Bojanowski et al. (2017]  Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov.  2017\.  Enriching word vectors with subword information.  Transactions of the Association for Computational Linguistics, 5:135–146.
  * [Bowman et al. (2015]  Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning.  2015\.  A large annotated corpus for learning natural language inference.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal, September. Association for Computational Linguistics.
  * [Collobert and Weston (2007]  Ronan Collobert and Jason Weston.  2007\.  Fast semantic extraction using a novel neural network architecture.  In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 560–567.
  * [Collobert and Weston (2008]  Ronan Collobert and Jason Weston.  2008\.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In Proceedings of the 25th international conference on Machine learning, pages 160–167.
  * [Collobert et al. (2011]  Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa.  2011\.  Natural language processing (almost) from scratch.  Computing Research Repository - CORR, 12, 03.
  * [Deerwester et al. (1989]  Scott C. Deerwester, Susan T. Dumais, George W. Furnas, Richard A. Harshman, Thomas K. Landauer, Karen E. Lochbaum, and Lynn A. Streeter.  1989\.  Computer information retrieval using latent semantic structure, June 13.
  * [Devlin et al. (2018]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * [Ester et al. (1996]  Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al.  1996\.  A density-based algorithm for discovering clusters in large spatial databases with noise.  In Kdd. AAAI Press.
  * [Gupta et al. (2019]  Prakhar Gupta, Matteo Pagliardini, and Martin Jaggi.  2019\.  Better word embeddings by disentangling contextual n-gram information.  arXiv preprint arXiv:1904.05033.
  * [He and Yih (2014]  Xiaodong He and Scott Wen-tau Yih.  2014\.  Deep learning and continuous representations for language processing (tutorial for slt-2014), December.
  * [Hornik et al. (2012]  Kurt Hornik, Ingo Feinerer, Martin Kober, and Christian Buchta.  2012\.  Spherical k-means clustering.  Journal of Statistical Software, 50:1–22, 09.
  * [Kiros et al. (2015]  Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  2015\.  Skip-thought vectors.  In Advances in neural information processing systems, pages 3294–3302.
  * [MacQueen (1967]  James MacQueen.  1967\.  Some methods for classification and analysis of multivariate observations.  In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics, pages 281–297, Berkeley, Calif. University of California Press.
  * [Mikolov et al. (2013a]  Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  2013a.  Efficient estimation of word representations in vector space.
  * [Mikolov et al. (2013b]  Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  2013b.  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119.
  * [Osgood et al. (1958]  Charles Egerton Osgood, George J Suci, and Percy H. Tannenbaum.  1958\.  The measurement of meaning.  American Journal of Sociology, 63(5):550–551.
  * [Pagliardini et al. (2018]  Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi.  2018\.  Unsupervised learning of sentence embeddings using compositional n-gram features.  Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).
  * [Pennington et al. (2014]  Jeffrey Pennington, Richard Socher, and Christopher D. Manning.  2014\.  Glove: Global vectors for word representation.  In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543.
  * [Radford et al. (2018]  Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  2018\.  Improving language understanding by generative pre-training.  URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf.
  * [Radford et al. (2019]  Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  2019\.  Language models are unsupervised multitask learners.  OpenAI Blog, 1(8):9.
  * [Reimers and Gurevych (2019]  Nils Reimers and Iryna Gurevych.  2019\.  Sentence-bert: Sentence embeddings using siamese bert-networks.
  * [Reimers et al. (2016]  Nils Reimers, Philip Beyer, and Iryna Gurevych.  2016\.  Task-oriented intrinsic evaluation of semantic textual similarity.  In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 87–96, Osaka, Japan, December. The COLING 2016 Organizing Committee.
  * [Salton (1971]  Gerard Salton.  1971\.  The SMART retrieval system: experiments in automatic document processing.  Prentice-Hall series in automatic computation. Prentice-Hall.
  * [Sordoni et al. (2015]  Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan.  2015\.  A neural network approach to context-sensitive generation of conversational responses.  In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 196–205, Denver, Colorado, May–June. Association for Computational Linguistics.
  * [Sparck Jones (1986]  Karen Sparck Jones.  1986\.  Synonymy and Semantic Classification.  Edinburgh University Press, GBR.
  * [Sutskever et al. (2014]  Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  2014\.  Sequence to sequence learning with neural networks.  In Advances in neural information processing systems, pages 3104–3112.
  * [Vaswani et al. (2017]  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008.
  * [Wang et al. (2019]  Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  2019\.  SuperGLUE: A stickier benchmark for general-purpose language understanding systems.  arXiv preprint 1905.00537.
  * [Williams et al. (2018]  Adina Williams, Nikita Nangia, and Samuel Bowman.  2018\.  A broad-coverage challenge corpus for sentence understanding through inference.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana, June. Association for Computational Linguistics.
  * [Yang et al. (2019]  Ziyi Yang, Chenguang Zhu, and Weizhu Chen.  2019\.  Parameter-free sentence embedding via orthogonal basis.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 638–648.
  * [Zadeh (1965]  Lotfi A. Zadeh.  1965\.  Fuzzy sets.  Information and Control, 8(3):338 – 353.
  * [Zhao and Mao (2018]  Rui Zhao and Kezhi Mao.  2018\.  Fuzzy bag-of-words model for document representation.  IEEE Transactions on Fuzzy Systems, 26(2):794–804, April.
  * [Zhelezniak et al. (2019]  Vitalii Zhelezniak, Aleksandar Savkov, April Shen, Francesco Moramarco, Jack Flann, and Nils Y. Hammerla.  2019\.  Don’t settle for average, go for the max: Fuzzy sets and max-pooled word vectors.

## Appendix A. Extended results

In this appendix we report the complete results of our experiments. Tables 3
and 4, contain the average score on each task fixing, respectively, the word
embedding matrix and the universe matrix construction approach. These first
two tables are provided to ease the understanding of numeric results.

Additionally, Tables 5, 6, 7 and 8 contain more detailed results of the
approaches we used to build the universe matrix using, respectively, the
following word embedding matrices: Word2Vec, GloVe, FastText and Sent2Vec. In
these tables is possible to find, for each word embedding used to build the
universe matrix, the following contents:

  * •

complete scores over each benchmark;

  * •

summary scores of different approaches using the same word embedding matrix
(rightmost column);

  * •

the best score on each benchmark and on average of each approach;

  * •

summary of all the approaches using the same word embedding matrix (bottom
row).

In all the following tables, summary statistics are expressed as
$\textit{average score}\pm\textit{standard deviation}$.

The first thing that comes to the eye in these table, is the common behaviour
on tasks shared by all models independently of the word embedding matrix (see
Table 3) or the universe matrix construction approach (see Table 4). In fact,
they all struggle more with the 2013 edition of the STS (in each row it has
always the lowest Spearman index) while they perform the best with the 2015
and 2016 editions.

Another interesting point is in the standard deviation of the summary
statistics: while the value appear to be high for all the models, when
averaging among the different tasks (as shown in the rightmost columns of
Tables 5, 6, 7 and 8), this doesn’t hold when averaging over the same task
with different models (downmost row of the aforementioned Tables). Models
based on Sent2Vec have a much lower standard deviation in this case, as can be
seen in Table 8, while for other word embedding matrices the standard
deviation is similar to that computed among the tasks.

About Sent2Vec it is curious how far it gets from the best results in our
experiments, especially considering that differently from the other embedding
models we used, it was conceived specifically for sentence embedding purposes
(see Table 8). It is also worth to mention that, even though it is a sentence
embedding algorithm, Sent2Vec performs better using a universe matrix obtained
through Spherical k-Means than the Identity matrix (as we were expecting), as
reported in Table 8. This is the only case where a SFBoW model has better
results in single tasks or in average with a clustering built universe matrix
instead of Identity- or PCA-built, as happened with the other embedding
matrices (see Tables 5, 6 and 7).

Finally we underline a surprising behaviour of the FastText based models. Even
though SFBoW built upon FastText reaches the best results, on average,
considering all the approaches for the universe matrix, they perform worse
than other models. In fact, when used with clustering, results are way lower
than Identity and PCA; this can be seen from Table 7.

Table 3: Summarised results (measured as Spearman’s correlation with human judgment) of the SFBoW model on the STS tasks using various word embedding matrices as source. Values are computed averaging the results for different universe matrices. Bold values represent the best score in each column. Word embedding matrix | Results (Spearman’s $\rho$)
---|---
| STS
| ’12 | ’13 | ’14 | ’15 | ’16
Word2Vec | $52.82\pm 3.54$ | $\mathbf{44.18}\pm 4.83$ | $\mathbf{59.47}\pm 4.32$ | $64.97\pm 5.09$ | $\mathbf{64.52}\pm 4.73$
GloVe | $53.77\pm 4.69$ | $43.84\pm 5.61$ | $55.36\pm 7.30$ | $62.38\pm 7.65$ | $63.56\pm 6.20$
FastText | $\mathbf{54.99}\pm 4.88$ | $43.65\pm 4.03$ | $54.36\pm 5.63$ | $63.75\pm 6.65$ | $64.40\pm 6.22$
Sent2Vec | $53.12\pm 1.34$ | $41.58\pm 2.32$ | $59.24\pm 2.78$ | $\mathbf{64.99}\pm 2.86$ | $62.64\pm 2.24$

Table 4: Summarised results (measured as Spearman’s correlation with human judgment) of the SFBoW model on the STS tasks using various universe matrices. Values are computed averaging the results for different word embedding matrices. Bold values represent the best score in each column. | Results (Spearman’s $\rho$)
---|---
Universe matrix | STS
| ’12 | ’13 | ’14 | ’15 | ’16
Clustering
Algorithm | Source | $k$ |
k-Means | English vocabulary | 100 | $47.24\pm 2.85$ | $36.29\pm 1.05$ | $47.88\pm 5.20$ | $53.81\pm 5.17$ | $55.26\pm 2.06$
1000 | $50.10\pm 2.01$ | $39.70\pm 1.00$ | $52.04\pm 4.69$ | $58.49\pm 4.48$ | $59.48\pm 2.05$
10000 | $52.04\pm 2.26$ | $42.66\pm 2.00$ | $55.26\pm 4.98$ | $61.14\pm 5.13$ | $62.16\pm 2.16$
25000 | $52.34\pm 3.29$ | $42.91\pm 2.56$ | $55.75\pm 6.21$ | $61.56\pm 6.34$ | $62.60\pm 3.05$
Frequent words (top 100000) | 100 | $50.47\pm 2.70$ | $38.34\pm 2.70$ | $52.39\pm 2.45$ | $58.16\pm 2.90$ | $58.15\pm 2.60$
1000 | $52.89\pm 3.56$ | $42.67\pm 3.35$ | $56.38\pm 2.97$ | $62.42\pm 3.64$ | $62.14\pm 3.58$
10000 | $54.64\pm 3.03$ | $45.34\pm 3.12$ | $59.48\pm 2.01$ | $65.37\pm 2.72$ | $66.42\pm 3.05$
25000 | $55.33\pm 2.47$ | $46.18\pm 2.83$ | $60.43\pm 1.93$ | $66.41\pm 2.39$ | $66.33\pm 2.81$
Spherical k-Means | Frequent words (top 50000) | 100 | $51.82\pm 0.99$ | $39.23\pm 3.13$ | $53.83\pm 3.93$ | $60.77\pm 2.52$ | $59.92\pm 3.32$
1000 | $54.32\pm 1.48$ | $43.33\pm 3.26$ | $57.60\pm 3.49$ | $64.89\pm 2.56$ | $63.52\pm 3.07$
10000 | $55.95\pm 1.69$ | $46.22\pm 2.60$ | $60.67\pm 2.48$ | $67.88\pm 1.92$ | $66.36\pm 2.18$
25000 | $56.35\pm 1.52$ | $46.01\pm 2.53$ | $60.92\pm 2.37$ | $68.20\pm 1.73$ | $66.48\pm 2.02$
Identity matrix
| $56.91\pm 3.35$ | $\mathbf{49.45}\pm 3.12$ | $\mathbf{64.56}\pm 1.90$ | $\mathbf{70.59}\pm 1.58$ | $69.33\pm 3.63$
Multivariate analysis
Algorithm | Source |
PCA | English vocabulary | $\mathbf{58.06}\pm 3.36$ | $48.83\pm 3.34$ | $64.35\pm 1.69$ | $\mathbf{70.59}\pm 1.77$ | $\mathbf{69.96}\pm 3.99$
Frequent words (top 100000) | $57.00\pm 2.62$ | $48.45\pm 3.08$ | $64.20\pm 1.60$ | $69.81\pm 1.68$ | $69.64\pm 3.52$

Table 5: Complete experimental results (measured as Spearman’s correlation with human judgment) of the SFBoW model on the STS tasks using a Word2Vec matrix as word embedding source. On the rightmost columns are reported an unweighted and a weighted per-model summary over the tasks, while on the bottom row is reported a per-task summary over the models, finally on the bottom-right corner are the unweighted and weighted per-word embedding matrix statistics. Bold values represent the highest score in the corresponding column. | Results (Spearman’s $\rho$)
---|---
Universe matrix | STS | Average | Weighted average
| ’12 | ’13 | ’14 | ’15 | ’16
Clustering
Algorithm |  Source |  $k$ |
k-Means |  English vocabulary |  100 | $48.75$ | $35.10$ | $51.32$ | $56.58$ | $56.01$ | $49.55\pm 7.79$ | $49.46\pm 6.65$
|  |  1000 | $52.10$ | $41.29$ | $57.15$ | $62.78$ | $61.91$ | $55.05\pm 7.86$ | $54.58\pm 6.83$
|  |  10000 | $54.05$ | $45.63$ | $60.90$ | $66.51$ | $65.13$ | $58.44\pm 7.74$ | $57.76\pm 6.90$
|  |  25000 | $54.53$ | $45.79$ | $61.24$ | $66.86$ | $65.14$ | $58.71\pm 7.73$ | $58.10\pm 6.89$
|  Frequent words (top 100000) |  100 | $46.51$ | $35.11$ | $51.47$ | $54.47$ | $55.93$ | $48.70\pm 7.52$ | $48.32\pm 6.38$
|  |  1000 | $47.41$ | $39.14$ | $53.62$ | $57.78$ | $58.24$ | $51.24\pm 7.19$ | $50.56\pm 6.27$
|  |  10000 | $50.17$ | $43.39$ | $58.03$ | $62.47$ | $63.16$ | $55.44\pm 7.60$ | $54.48\pm 6.75$
|  |  25000 | $51.74$ | $44.63$ | $59.83$ | $64.55$ | $64.73$ | $57.10\pm 7.82$ | $56.16\pm 6.97$
Spherical k-Means |  Frequent words (top 50000) |  100 | $51.40$ | $41.11$ | $58.33$ | $64.20$ | $63.65$ | $55.74\pm 8.65$ | $55.01\pm 7.58$
|  |  1000 | $53.50$ | $45.58$ | $61.19$ | $68.08$ | $66.86$ | $59.04\pm 8.48$ | $58.07\pm 7.59$
|  |  10000 | $54.49$ | $47.70$ | $62.84$ | $69.92$ | $67.85$ | $60.56\pm 8.34$ | $59.55\pm 7.62$
|  |  25000 | $55.18$ | $47.61$ | $63.04$ | $69.88$ | $67.77$ | $60.70\pm 8.27$ | $59.81\pm 7.49$
Identity matrix
| $54.69$ | $\mathbf{50.99}$ | $63.81$ | $69.80$ | $68.52$ | $61.56\pm 7.49$ | $60.37\pm 7.00$
Multivariate analysis
Algorithm |  Source |
PCA |  English vocabulary | $\mathbf{59.78}$ | $50.26$ | $64.79$ | $\mathbf{70.61}$ | $\mathbf{71.81}$ | $\mathbf{63.45}\pm 7.88$ | $\mathbf{62.84}\pm 6.71$
|  Frequent words (top 100000) | $57.97$ | $49.42$ | $\mathbf{64.88}$ | $70.10$ | $71.12$ | $62.70\pm 8.11$ | $61.77\pm 7.05$
Average | $52.82\pm 3.54$ | $44.18\pm 4.83$ | $59.47\pm 4.32$ | $64.97\pm 5.09$ | $64.52\pm 4.73$ |  |

Table 6: Complete experimental results (measured as Spearman’s correlation with human judgment) of the SFBoW model on the STS tasks using a GloVe matrix as word embedding source. On the rightmost columns are reported an unweighted and a weighted per-model summary over the tasks, while on the bottom row is reported a per-task summary over the models, finally on the bottom-right corner are the unweighted and weighted per-word embedding matrix statistics. Bold values represent the highest score in the corresponding column. | Results (Spearman’s $\rho$)
---|---
Universe matrix | STS | Average | Weighted average
| ’12 | ’13 | ’14 | ’15 | ’16
Clustering
Algorithm |  Source |  $k$ |
k-Means |  English vocabulary |  100 | $44.94$ | $36.34$ | $43.07$ | $47.75$ | $53.50$ | $45.12\pm 5.63$ | $44.44\pm 4.31$
|  |  1000 | $47.33$ | $38.62$ | $46.72$ | $52.57$ | $57.24$ | $48.50\pm 6.24$ | $47.69\pm 4.89$
|  |  10000 | $48.23$ | $39.99$ | $48.38$ | $53.91$ | $59.17$ | $49.94\pm 6.41$ | $49.00\pm 4.98$
|  |  25000 | $46.71$ | $38.78$ | $46.43$ | $51.52$ | $57.71$ | $48.23\pm 6.25$ | $47.27\pm 4.76$
|  Frequent words (top 100000) |  100 | $53.88$ | $42.42$ | $54.79$ | $61.83$ | $62.14$ | $55.01\pm 7.17$ | $54.61\pm 6.03$
|  |  1000 | $57.34$ | $48.07$ | $60.09$ | $67.34$ | $67.87$ | $60.14\pm 7.28$ | $59.41\pm 6.17$
|  |  10000 | $58.35$ | $50.21$ | $62.50$ | $69.51$ | $70.72$ | $62.26\pm 7.55$ | $61.29\pm 6.41$
|  |  25000 | $58.52$ | $50.88$ | $62.86$ | $70.06$ | $71.20$ | $62.70\pm 7.53$ | $61.67\pm 6.43$
Spherical k-Means |  Frequent words (top 50000) |  100 | $51.57$ | $34.52$ | $47.52$ | $57.28$ | $54.84$ | $49.15\pm 8.01$ | $49.46\pm 7.00$
|  |  1000 | $53.53$ | $39.32$ | $51.85$ | $61.52$ | $59.13$ | $53.07\pm 7.73$ | $53.03\pm 6.69$
|  |  10000 | $55.58$ | $43.74$ | $56.50$ | $65.35$ | $63.61$ | $56.96\pm 7.64$ | $56.59\pm 6.56$
|  |  25000 | $56.10$ | $43.63$ | $56.91$ | $65.92$ | $63.90$ | $57.29\pm 7.83$ | $56.99\pm 6.74$
Identity matrix
| $\mathbf{58.82}$ | $\mathbf{51.55}$ | $\mathbf{64.74}$ | $71.03$ | $70.96$ | $\mathbf{63.42}\pm 7.46$ | $\mathbf{62.50}\pm 6.55$
Multivariate analysis
Algorithm |  Source |
PCA |  English vocabulary | $58.53$ | $50.59$ | $64.46$ | $\mathbf{71.18}$ | $\mathbf{71.04}$ | $63.16\pm 7.84$ | $62.23\pm 6.86$
|  Frequent words (top 100000) | $57.15$ | $48.91$ | $63.51$ | $68.89$ | $70.36$ | $61.76\pm 7.93$ | $60.79\pm 6.83$
Average | $53.77\pm 4.69$ | $43.84\pm 5.61$ | $55.36\pm 7.30$ | $62.38\pm 7.65$ | $63.56\pm 6.20$ |  |

Table 7: Complete experimental results (measured as Spearman’s correlation with human judgment) of the SFBoW model on the STS tasks using a FastText matrix as word embedding source. On the rightmost columns are reported an unweighted and a weighted per-model summary over the tasks, while on the bottom row is reported a per-task summary over the models, finally on the bottom-right corner are the unweighted and weighted per-word embedding matrix statistics. Bold values represent the highest score in the corresponding column. | Results (Spearman’s $\rho$)
---|---
Universe matrix | STS | Average | Weighted average
| ’12 | ’13 | ’14 | ’15 | ’16
Clustering
Algorithm |  Source |  $k$ |
k-Means |  English Vocabulary |  100 | $44.12$ | $35.77$ | $42.55$ | $50.11$ | $53.24$ | $45.16\pm 6.10$ | $44.39\pm 4.90$
|  |  1000 | $49.02$ | $39.74$ | $48.05$ | $55.73$ | $57.66$ | $50.04\pm 6.35$ | $49.40\pm 5.21$
|  |  10000 | $52.62$ | $42.69$ | $52.76$ | $59.73$ | $61.48$ | $53.86\pm 6.63$ | $53.27\pm 5.46$
|  |  25000 | $53.36$ | $43.28$ | $53.80$ | $60.72$ | $62.35$ | $54.70\pm 6.75$ | $54.12\pm 5.60$
|  English words (top 100000) |  100 | $49.84$ | $38.85$ | $48.81$ | $56.37$ | $55.76$ | $49.93\pm 6.32$ | $49.71\pm 5.37$
|  |  1000 | $53.85$ | $42.64$ | $53.31$ | $60.38$ | $60.34$ | $54.10\pm 6.49$ | $53.85\pm 5.50$
|  |  10000 | $56.19$ | $45.77$ | $57.33$ | $63.49$ | $63.96$ | $57.35\pm 6.59$ | $56.96\pm 5.52$
|  |  25000 | $56.31$ | $45.75$ | $57.65$ | $64.02$ | $64.47$ | $57.64\pm 6.79$ | $57.22\pm 5.70$
Spherical k-Means |  Frequent words (top 50000) |  100 | $53.47$ | $42.82$ | $54.53$ | $61.66$ | $61.93$ | $54.88\pm 6.97$ | $54.42\pm 5.87$
|  |  1000 | $56.88$ | $47.37$ | $59.18$ | $66.52$ | $65.89$ | $59.17\pm 6.98$ | $58.62\pm 5.98$
|  |  10000 | $58.80$ | $49.74$ | $62.21$ | $69.55$ | $69.05$ | $61.87\pm 7.31$ | $61.18\pm 6.30$
|  |  25000 | $58.91$ | $49.33$ | $62.19$ | $69.89$ | $69.02$ | $61.87\pm 7.50$ | $61.22\pm 6.48$
Identity matrix
| $61.31$ | $51.21$ | $\mathbf{67.47}$ | $\mathbf{72.90}$ | $\mathbf{73.88}$ | $\mathbf{65.35}\pm 8.37$ | $\mathbf{64.55}\pm 7.20$
Multivariate analysis
Algorithm |  Source |
PCA |  English vocabulary | $\mathbf{61.42}$ | $51.36$ | $66.44$ | $72.74$ | $73.72$ | $65.14\pm 8.21$ | $64.32\pm 7.00$
|  Frequent words (top 100000) | $60.03$ | $\mathbf{51.96}$ | $66.36$ | $72.39$ | $73.25$ | $64.80\pm 7.99$ | $63.81\pm 6.93$
Average | $54.99\pm 4.88$ | $43.65\pm 4.03$ | $54.36\pm 5.63$ | $63.75\pm 6.65$ | $64.40\pm 6.22$ |  |

Table 8: Complete experimental results (measured as Spearman’s correlation with human judgment) of the SFBoW model on the STS tasks using a Sent2Vec matrix as word embedding source. On the rightmost columns are reported an unweighted and a weighted per-model summary over the tasks, while on the bottom row is reported a per-task summary over the models, finally on the bottom-right corner are the unweighted and weighted per-word embedding matrix statistics. Bold values represent the highest score in the corresponding column. | Results (Spearman’s $\rho$)
---|---
Universe matrix | STS | Average | Weighted average
| ’12 | ’13 | ’14 | ’15 | ’16
Clustering
Algorithm |  Source |  $k$ |
k-Means |  English vocabulary |  100 | $51.14$ | $37.95$ | $54.58$ | $60.81$ | $58.30$ | $52.56\pm 8.01$ | $52.48\pm 6.97$
|  |  1000 | $51.94$ | $39.14$ | $56.24$ | $62.88$ | $61.09$ | $54.26\pm 8.47$ | $53.95\pm 7.35$
|  |  10000 | $53.26$ | $42.33$ | $59.00$ | $65.43$ | $62.86$ | $56.58\pm 8.22$ | $56.16\pm 7.27$
|  |  25000 | $54.76$ | $43.77$ | $61.39$ | $67.01$ | $64.93$ | $58.37\pm 8.40$ | $57.93\pm 7.44$
|  Frequent words (top 100000) |  100 | $51.65$ | $37.00$ | $54.51$ | $59.98$ | $58.78$ | $52.38\pm 8.25$ | $52.38\pm 7.08$
|  |  1000 | $52.96$ | $40.84$ | $58.51$ | $64.20$ | $62.09$ | $55.72\pm 8.36$ | $55.43\pm 7.33$
|  |  10000 | $53.85$ | $41.98$ | $60.08$ | $66.01$ | $63.83$ | $57.15\pm 8.63$ | $56.76\pm 7.61$
|  |  25000 | $54.76$ | $43.77$ | $61.39$ | $67.01$ | $64.93$ | $58.37\pm 8.40$ | $57.93\pm 7.44$
Spherical k-Means |  Frequent words (top 50000) |  100 | $50.83$ | $38.45$ | $54.94$ | $59.92$ | $59.25$ | $52.68\pm 7.83$ | $52.44\pm 6.74$
|  |  1000 | $53.38$ | $41.06$ | $58.17$ | $63.46$ | $62.22$ | $55.66\pm 8.11$ | $55.39\pm 7.03$
|  |  10000 | $54.95$ | $43.71$ | $61.12$ | $66.72$ | $64.94$ | $58.29\pm 8.33$ | $57.87\pm 7.33$
|  |  25000 | $\mathbf{55.22}$ | $43.48$ | $61.53$ | $67.13$ | $\mathbf{65.22}$ | $\mathbf{58.52}\pm 8.54$ | $\mathbf{58.12}\pm 7.52$
Identity matrix
| $52.80$ | $\mathbf{44.05}$ | $\mathbf{62.23}$ | $\mathbf{68.63}$ | $63.96$ | $58.33\pm 8.80$ | $57.75\pm 8.16$
Multivariate analysis
Algorithm |  Source |
PCA |  English vocabulary | $52.51$ | $43.09$ | $61.72$ | $67.84$ | $63.26$ | $57.68\pm 8.83$ | $57.18\pm 8.14$
|  Frequent words (top 100000) | $52.84$ | $43.50$ | $62.04$ | $67.88$ | $63.83$ | $58.02\pm 8.77$ | $57.48\pm 8.06$
Average | $53.12\pm 1.34$ | $41.58\pm 2.32$ | $59.24\pm 2.78$ | $64.99\pm 2.86$ | $62.64\pm 2.24$ |  |


# Role of work function distribution on field emission effects

Nandan Pakhira  Department of Physics, Kazi Nazrul University, Asansol, West
Bengal 713340, India  Rajib Mahato  Department of Physics, Kazi Nazrul
University, Asansol, West Bengal 713340, India  Central Electronics
Engineering Research Institute, Pilani, Rajasthan 333031, India

###### Abstract

Field emission effect is the emission of electrons from a cold metallic
surface in the presence of an electric field. The emission current
exponentially depends on the work function of the metallic surface. In this
work we consider the role of work function distribution on the field emission
current. The work function distribution can arise due to nano-scale
inhomogeneities of the surface as well as for collection of nano-particles
with size distribution. We consider both Gaussian distribution as well as log-
normal distribution. For Gaussian distribution, the field emission current,
$J_{\textrm{av}}$, averaged over work function distribution shows Gaussian
dependence, $J_{\textrm{av}}\propto\exp(\alpha\sigma^{2})$, where $\sigma$ is
the width of the work function distribution and $\alpha$ is a fitting
parameter. For log-normal distribution, $J_{\textrm{av}}$ shows compressed
exponential behaviour, $J_{\textrm{av}}\propto\exp(\gamma\sigma^{n})$, where
the exponent $n>1$ is a non-universal parameter. We also study in detail field
emission current for various electric field strength applied to systems with
high density, characterised by Fermi energy, $E_{F}\gg\Phi$, $\Phi$ being the
work function of the system as well as systems with low density characterised
by $E_{F}\ll\Phi$.

##  I Introduction

Field emission is the process in which electrons from cold surfaces are
emitted in the presence of applied strong electric field. This process should
be compared agaist the thermo-ionic process in which electrons are emitted
from hot metal surface. The field emission forms the back bone of modern
semiconductor devices. This effect was first described by Fowler and Nordheim
FNpaper . They considered quantum mechanical tunneling through a triangular
potential energy (PE) barrier, created by the application of constant electric
field. Much later Murphy and Good MGpaper  (MG) introduced a more realistic PE
barrier by taking into account the induced image charge formed in the presence
of emitted electron. MG calculated barrier transmission coefficient under
semi-classical WKB approximation. More recently, essentially an exact solution
of the problem was obtained by Choy et. al. ChoyJPCM2005 . Also, various cases
including finite temperature (thermal emission), tunneling, curvature of the
emission surface have been considered by various authors
CutlerSurfaceScience1993 ; ForbesUltramicro2001 ; EdgcombePRB2005 ;
JensenAPL2006 ; FischerJVacSciTech2013 ; ForbesJVacSciTech2013 ;
KyritsakisProcRoySocA2015 ; HolgatePRAppl2017

To the best of our knowledge in all of those studies authors have considered
constant local work function. The work function of a material depends on the
composition, structure, geometry, local charge distribution etc. of the
emitting surface. Assumption of constant work function is only suitable for an
atomistic smooth homogeneous surface. For surfaces with inhomogeneities over
nano-scale (much smaller than the size of the collectors) assumption of
constant work function is no longer valid. Also it has been shown
GamezJApplCrys2017  that in a system of nano-particles there is a distribution
of the size of the nano-particles. Since the work function is more of a
property of the surface we naturally can expect that the work function of
nano-particles will also have a distribution. The actual microscopic model for
work function distribution for a system of nano-particles is beyond the scope
of this work. Interestingly, Gamez et. al. GamezJApplCrys2017  using scanning
tunneling microscope (STM) have measured the pair distribution function (PDF)
for Pt nano-particles and they found that it follows log-normal distrubution.

In this work, purely as a mathematical model, we choose Gaussian and log-
normal distribution for the work function. We then study the field emision
current averaged over work function distribution. The organization of the rest
of the paper is as follows. In Sec. II we describe the mathematical formalism
used to calculate field emission current. In Sec. III we describe the work
function distribution used to calculate average current. In Sec. IV we present
our results for both the case of Gaussian distribution and log-normal
distribution. Finally in Sec. V we conclude.

##  II Mathematical formalism

We closely follow and summarize the results obtained by Lopes et. al.
LopesCondMat ; LopesPhysLett2020  for field-emission current density, $J$. In
the standard FN-type MG theory the field emission current density is given by
the well known expression BiswasPhysPlasmas2017 ; HaugBook ; ForbesAPL2006 ;
ForbesJVacSciTech2010 ,

$\displaystyle J$ $\displaystyle=$
$\displaystyle\frac{e^{3}\mathcal{E}^{2}}{16\pi^{2}\hbar\Phi}\frac{1}{t^{2}(y_{0})}\exp\left(-\frac{4}{3}\frac{\sqrt{2m}}{\hbar}\frac{\Phi^{3/2}}{e\mathcal{E}}v(y_{0})\right)$
(1)

where $e$ and $m$ are the charge and mass of electron, $\mathcal{E}$ is the
strength of the applied electric field, $\Phi$ is the local work function of
the emitting surface and

$\displaystyle
v(y)=\left[\frac{1+\sqrt{1-y^{2}}}{2}\right]^{1/2}\left[E(\lambda)-\left(1-\sqrt{1-y^{2}}\right)K(\lambda)\right].$
(2)

$K(\lambda)$ and $E(\lambda)$ are the complete elliptic integral of the first
and second kind, with

$\displaystyle\lambda^{2}$ $\displaystyle=$
$\displaystyle\frac{2\sqrt{1-y^{2}}}{1+\sqrt{1-y^{2}}}$ (3)

and

$\displaystyle y^{2}$ $\displaystyle=$
$\displaystyle\frac{e^{3}\mathcal{E}}{4\pi\epsilon_{0}(V_{0}-E)^{2}}<1.$ (4)

It is important to mention that $V_{0}=\Phi+E_{F}$ is the height of the
potential energy barrier ($E_{F}$ is the Fermi energy) and $E$ is the energy
of the electron. Finally,

$\displaystyle t(y_{0})=\left[v(y)-\frac{2}{3}y\frac{dv}{dy}\right]_{y=y_{0}}$
(5)

with

$\displaystyle y_{0}^{2}$ $\displaystyle=$
$\displaystyle\frac{e^{3}\mathcal{E}}{4\pi\epsilon_{0}\Phi^{2}}$ (6)

From the relations above it is quite evident that calculation of transmission
current requires evaluation of numerical integrals for complete elliptic
integrals. Due to the singularities present in complete elliptic integrals it
is very hard to extract meaningful results purely numerically DolanPhysRev1953
. Under this circumstances we can consider series expansion for complete
elliptic integrals GradshteynBook  as follows

$\displaystyle E(q)$ $\displaystyle=$ $\displaystyle
1+\frac{1}{2}\left[\ln\left(\frac{4}{q}\right)-\frac{1}{2}\right]q^{2}+\cdots$
(7) $\displaystyle K(q)$ $\displaystyle=$
$\displaystyle\ln\left(\frac{4}{q}\right)+\left[\ln\left(\frac{4}{q}\right)-1\right]\frac{q}{4}+\cdots$
(8)

where

$\displaystyle q$ $\displaystyle=$ $\displaystyle\sqrt{1-p^{2}}$ (9)

and $p^{2}=\frac{x_{2}-x_{1}}{x_{2}}$. $x_{1}$ and $x_{2}$ are the roots of
the quadratic equation

$\displaystyle V_{0}-E-\frac{Ze^{2}}{4x}-e\mathcal{E}x=0$ (10)

ls A detailed calculation gives the following form for the field emission
current density

$\displaystyle J$ $\displaystyle=$
$\displaystyle\frac{e^{3}\mathcal{E}^{2}}{16\pi^{2}\hbar\Phi}\frac{\left[1-u(y_{0})\right]}{t^{2}(y_{0})}\exp\left[-\frac{4}{3}\sqrt{\frac{2m}{\hbar^{2}}}\frac{\Phi^{3/2}}{e\mathcal{E}}v(y_{0})\right]$
(11)

where

$\displaystyle u(y_{0})$ $\displaystyle=$
$\displaystyle\left[1+\frac{2\sqrt{2m\Phi}}{e\hbar}\frac{E_{F}}{\mathcal{E}}t(y_{0})\right]\exp\left[-\frac{2\sqrt{2m\Phi}}{e\hbar}\frac{E_{F}}{\mathcal{E}}t(y_{0})\right]$
(12)

with

$\displaystyle v(y_{0})$ $\displaystyle=$ $\displaystyle
1-\left[\frac{3}{8}Z^{\prime}\ln\left(\frac{8}{\sqrt{Z^{\prime}}}\right)+\frac{Z^{\prime}}{16}\right]+\frac{3}{8}Z^{\prime}y_{0}^{2}\ln(y_{0})$
(13) $\displaystyle+$ $\displaystyle\frac{Z^{\prime
2}}{32}\left[1-\ln\left(\frac{8}{\sqrt{Z^{\prime}}}\right)\right]y_{0}^{4}+\frac{Z^{\prime
2}}{32}y_{0}^{4}\ln(y_{0})+\cdots$

and

$\displaystyle t(y_{0})$ $\displaystyle=$ $\displaystyle
v(y_{0})-\frac{2}{3}y_{0}\frac{dv}{dy}(y_{0})$ (14)

with $y_{0}$ given by Eq. 6 and $Z^{\prime}=1.179$ is an arbitrary constant
obtained from suitable boundary condition LopesCondMat .

The expression for transmission current in Eq. 11 carries an interesting pre-
exponential factor $[1-u(y_{0})]$ as compared to the standard form in Eq. 1.
This factor exhibits an explicit dependence on the external electric field
($\mathcal{E}$), work function ($\Phi$) and Fermi energy, $E_{F}$.

It is worth to mention that, from Eq. 11, we can immediately investigate two
important particular cases depending on the value of $E_{F}$. For large values
of Fermi energies, i.e.,
$\frac{2\sqrt{2m}}{e\hbar}\frac{\Phi^{1/2}E_{F}}{\mathcal{E}}\gg 1$, or
equivalently $E_{F}\gg\frac{e\hbar\mathcal{E}}{2\sqrt{2m\Phi}}$, it is easy to
see that $u(y_{0})\approx 0$ in Eq. 12 since the exponential dominates. So, in
this limit we recover the standard FN-type MG equation.

On the other hand, for the opposite regime, i.e., for very small Fermi
energies, we have $E_{F}\ll\frac{e\hbar\mathcal{E}}{2\sqrt{2m\Phi}}$. In this
limit we can expand the exponential in Eq. 12 and get

$\displaystyle
J=\frac{me}{2\pi^{2}\hbar^{3}}E_{F}^{2}\exp\left(-\frac{4}{3}\sqrt{\frac{2m}{\hbar^{2}}}\frac{\Phi^{3/2}}{e\mathcal{E}}v(y_{0})\right)$
(15)

It is worth to emphasize that for small values of Fermi energies a very
different expression, compared to the standard FN-type MG formula, was
obtained LopesCondMat  for the current density. Note that in this regime the
pre-exponential factor in Eq. 15 does not depend on either $t(y_{0})$ or the
external electric field ($\mathcal{E}$), but it remains dependent on the Fermi
energy ($E_{F}$).

##  III Work function distribution

In the previous section we have summarized field emission current for a given
local work function $\Phi$. For a system of nano-particles or systems with
inhomogeneity the work function will be different over the length scale of the
size of the collector for emitted electrons. In such a situation we need to
average the field emission current over the distribution of work function as
follows :

$\displaystyle J_{\textrm{av}}$ $\displaystyle=$ $\displaystyle\int
P(\Phi)J(\Phi,E_{F})d\phi$ (16)

where $P(\Phi)$ is the distribution of the work function, $\Phi$. We choose
two widely used distribution functions, namely (i) Gaussian and (ii) log-
normal distribution. It is well known that systems with bulk disorder follows
Gaussian distribution and the work function distribution function, $P(\Phi)$,
is given by

$\displaystyle P_{N}(\Phi)$ $\displaystyle=$
$\displaystyle\frac{1}{\sigma\sqrt{2\pi}}\exp\left[-\frac{(\Phi-\Phi_{0})^{2}}{2\sigma^{2}}\right],$
(17)

where $\sigma$ is the standard deviation of the distribution and $\Phi_{0}$ is
the known bulk value for a given material.

We also use log-normal distribution for the work function :

$\displaystyle P_{LN}(\Phi)$ $\displaystyle=$
$\displaystyle\frac{1}{\Phi\sigma\sqrt{2\pi}}\exp\left[-\frac{(\ln\Phi-\mu)^{2}}{2\sigma^{2}}\right]$
(18)

It is important to mention that we were inspired by the experimental result of
Gamez et. al. GamezJApplCrys2017 . They showed that for a system of Pt nano-
particles the pair distribution function (PDF) for the radius of nano-
particles follows log-normal distribution

$\displaystyle F(r)=\frac{1}{rs\sqrt{2\pi}}\exp\left[-\frac{(\ln
r-\mu)^{2}}{2s^{2}}\right]$ (19)

with

$\displaystyle s^{2}$ $\displaystyle=$
$\displaystyle\ln\left[\left(\frac{\textrm{P}_{\textrm{sig}}}{\textrm{P}_{\textrm{size}}}\right)+1\right]$
(20) $\displaystyle\mu$ $\displaystyle=$
$\displaystyle\ln\left(\textrm{P}_{\textrm{size}}\right)-\frac{s^{2}}{2}$ (21)

where $\textrm{P}_{\textrm{size}}$ and $\textrm{P}_{\textrm{sig}}$ are the
average particle diameter and standard deviation, respectively. Since work
function crucially depends on the surface properties of a system it may follow
log-normal distribution as in the case of surface disorder.

Various statistical properties of this distribution are summarized in the
table 1 :

Table 1: Statistical properties of log-normal distribution Mean | $e^{\mu+\frac{\sigma^{2}}{2}}$
---|---
Variance | $\nu=e^{2\mu+\sigma^{2}}\left[e^{\sigma^{2}}-1\right]$
Standard deviation | $s=\sqrt{\nu}$
Median | $M=e^{\mu}$
Mode | $\Gamma=e^{\mu-\sigma^{2}}$
Skewness | $\Sigma=(e^{\sigma^{2}}+2)\sqrt{e^{\sigma^{2}}-1}$
Kurtosis | $\kappa=e^{4\sigma^{2}}+2e^{3\sigma^{2}}+3e^{2\sigma^{2}}-3$

##  IV Results

In this section we show our results for current density averaged over two
choice of probability distributions as have been discussed in the previous
section.

###  IV.1 Gaussian work function distribution

We first consider the case of Gaussian distribution for the work function. In
Fig. 1 we show the histogram plot of the work function, sampled over Gaussian
distribution for four choices of bulk work function $\Phi_{0}=3.0,3.5,4.0$ and
$4.5$ eV with $\sigma=0.05$. In each case we also fit the histogram plot to
Gaussian distribution. From this fit we can see that our choice of random
variables for $\Phi$ are well sampled over Gaussian distribution.

(a)

(b)

(c)

(d)

Figure 1: (Color online) Histogram plot of work function, $\Phi$, sampled over
Gaussian distribution for various average work function, $\Phi_{0}$. Solid
line shows fitting to Gaussian distribution and we have chosen $\sigma=0.05$
for all the cases.

####  IV.1.1 Case with $\Phi\ll E_{F}$

We now consider the case with $E_{F}=10$ eV and $\Phi_{0}=$ 3.0, 3.5, 4.0 and
4.5 eV. Since $E_{F}\propto n^{2/3}$, $n$ being the density of electrons, this
will correspond to high density limit. In Fig. 2 we show the current density
averaged over various number of random variables, $N$, for $\Phi$. For
$N<10^{4}$ there is significant fluctuations of $J_{\textrm{av}}$ but for
$N>4\times 10^{4}$, $J_{\textrm{av}}$ show small fluctuations over mean value.
We choose $N=10^{5}$ for calculation of $J_{\textrm{av}}$ for all the cases
and $J_{\textrm{av}}$ will be free of statistical errors.

Figure 2: (Color online) Average current, $J_{\textrm{av}}$, as a function of
number of random variables, N chosen from Gaussian distribution for
$\Phi_{0}\ll E_{F}$.

As shown in Fig. 2, $J_{\textrm{av}}$ varries over several orders of magnitude
as we varry $\Phi_{0}$ from 3.0 eV to 4.5 eV. So in order to understand role
of work function distribution on field emission current we need to study the
dimensionless scaled quantity $J_{\textrm{av}}/J_{0}$, where $J_{0}$ is the
field emission current for bulk value of the work function $\Phi_{0}$ and can
be calculated from Eq. 1 with $\Phi=\Phi_{0}$.

In Fig. 3 we show, $\log(J_{\textrm{av}}/J_{0})$, as a function of the width
of the work function distribution, $\sigma$ for a given applied electric
field, $\mathcal{E}=3\times 10^{7}\textrm{V/cm}$. We choose
$0.01\leq\sigma\leq 0.1$. Since the full width at half maxima (FWHM) for a
Gaussian distribution is $2.35\sigma$ and more than $95\%$ integrated weight
is within a width $4\sigma$ (between $-2\sigma$ and $+2\sigma$) our choice of
$\sigma$ ensures that the deviation of $\Phi$ from its bulk value $\Phi_{0}$
is less than $10\%$. The first noticeable feature in Fig. 3 is that the
current density, $J_{\textrm{av}}$, averaged over work function distribution
increases monotonically with the width of the distribution, $\sigma$. This is
because the transport current, $J(\Phi)\propto\exp(-\zeta\Phi^{3/2})$ where
$\zeta=\frac{4}{3}\sqrt{\frac{2m}{\hbar^{2}}}\frac{v(y_{0})}{e\mathcal{E}}$
and the averaging over work function distribution leads to exploration of
regions with lower barrier height hence increase of tunneling current.

Another noticeable feature in Fig. 3 is that the logarithm of the scaled
average current $\log(J_{\textrm{av}}/J_{0})$ can be fitted well with the
functional form $f(\sigma)=\alpha\sigma^{2}$, where $\alpha$ is a fitting
parameter. This clearly shows that $J_{\textrm{av}}$ follows Gaussian
behaviour $J_{\textrm{av}}=J_{0}\exp(\alpha\sigma^{2})$. The fitting parameter
$\alpha$ in general depends on $\Phi_{0},\mathcal{E}$ and $E_{F}$. The
dependence of $\alpha$ with $\Phi_{0}$ is nearly linear as shown in the inset
of Fig. 3.

Figure 3: (Color online) Logarithm of the scaled average current,
$J_{\textrm{av}}/J_{0}$, as a function of the width, $\sigma$, of the work
function distribution for various average work function, $\Phi_{0}\ll E_{F}$.
Solid lines with symbols are the numerical data and the dashed lines
represents fitting function $f(\sigma)=\alpha\sigma^{2}$. Inset : variation of
the fitting parameter $\alpha$ with $\Phi_{0}$. The strength of the electric
field, $\mathcal{E}=3\times 10^{7}\textrm{V/cm}$, and Fermi energy
$E_{F}=10.0$ eV for all the cases.

In Fig. 4 we show the behaviour of $\log(J_{\textrm{av}}/J_{0})$ for various
field strength $\mathcal{E}$ as well as $\Phi_{0}$. For $\mathcal{E}=10^{7}$
V/cm the increase of $J_{\textrm{av}}$ is more than $e$ times over the bulk
value $J_{0}$ for $\sigma=0.1$. This can be explained by the fact that the
tunneling current
$J\propto\frac{\mathcal{E}^{2}}{\Phi}\exp(-\zeta\Phi^{3/2}/\mathcal{E})$. With
increasing field strength the tunneling current dramatically increases mainly
due to its $\exp(-1/\mathcal{E})$ dependence. Averaging over work function
distribution increases tunneling current as explained earlier. However this
effect becomes sub-dominant for larger field strengths as the tunneling
current increses over several orders of magnitude for just one order of
magnitude increase of $\mathcal{E}$. The behaviour is similar for all the four
choices of $\Phi_{0}$. In the inset of each plot in Fig. 4 we have shown the
dependence of fitting parameter $\alpha$ with the field strength
$\mathcal{E}$. As can be clearly seen $\alpha$ very strongly depends on
$\mathcal{E}$.

Figure 4: (Color online)Panel plot : Logarithm of scaled average current,
$J_{\textrm{av}}/J_{0}$, as a fuction of the width, $\sigma$ of the work
function distribution for various field strength, $\mathcal{E}$. Solid lines
with symbols are numerical data and the dashed lines represent fitting
function $f(\sigma)=\alpha\sigma^{2}$. Inset of each plot shows variation of
$\alpha$ with field strength, $\mathcal{E}$. The value of $\Phi_{0}$ is
indicated in each plot and $E_{F}=10.0$ eV for all the plots.

####  IV.1.2 Case with $\Phi\gg E_{F}$

Next we consider the case with $E_{F}=0.05$ eV and $\Phi_{0}=3.0,3.5,4.0$ and
4.5 eV . This will correspond to systems with low density as $E_{F}\propto
n^{2/3}$. In this case the tunneling current does not depend on the prefactor
$\frac{e^{3}\mathcal{E}^{2}}{16\pi^{2}\hbar\Phi}\frac{[1-u(y_{0})]}{t^{2}(y_{0})}$
but still depends on the standard exponential factor
$\exp(-\zeta\Phi^{3/2}/\mathcal{E})$. In Fig. 5 we show $J_{\textrm{av}}$ as a
function of $N$, the number of random variables for averaging over work
function distribution for $\Phi_{0}=3.0,3.5,4.0$ and 4.5 eV. As can be clearly
seen $J_{\textrm{av}}$ is atleast one order of magnitude smaller than the case
with $\Phi\ll E_{F}$. This is mainly due to the fact that the prefactor
$\frac{me}{2\pi^{2}\hbar^{3}}E_{F}^{2}$ for tunneling current $J$ in this case
is much smaller than the prefactor
$\frac{e^{3}\mathcal{E}^{2}}{16\pi^{2}\hbar\Phi}\frac{[1-u(y_{0})]}{t^{2}(y_{0})}$
corresponding to the case with $\Phi_{0}\ll E_{F}$. As in the earlier case
$J_{\textrm{av}}$ shows significant fluctuation for $N<10^{4}$. We choose
$N=10^{5}$ for various $\Phi_{0}$ and $\mathcal{E}$.

Figure 5: (Color online)Average current, $J_{\textrm{av}}$, as a function of
number of random variables, N, chosen from Gaussian distribution for
$\Phi_{0}\gg E_{F}$.

In Fig. 6 we show $\log(J_{\textrm{av}}/J_{0})$ as a function of the width,
$\sigma$, of the Gaussian work function distribution for a given applied
electric field strength $\mathcal{E}=3\times 10^{7}\textrm{V/cm}$. As in the
case with $\Phi_{0}\ll E_{F}$, $J_{\textrm{av}}$ monotonically increases with
$\sigma$. But the increase is slower than the case with $\Phi_{0}\ll E_{F}$.
This is mainly because of the absence of $\Phi$ dependent prefactor in the
expression for $J$. Most intrestingly, as in the earlier case with
$\Phi_{0}\ll E_{F}$, $\log(J_{\textrm{av}}/J_{0})$ can be well fitted with
$f(\sigma)=\alpha\sigma^{2}$. The fitting parameter $\alpha$ linearly depends
on $\Phi_{0}$.

Figure 6: (Color online) Logarithm of scaled average current,
$J_{\textrm{av}}/J_{0}$, as a function of the width, $\sigma$, of the Gaussian
work function distribution for various average work function, $\Phi_{0}\gg
E_{F}$. Solid lines with symbols are numerical data and dashed lines
represents fitting function $f(\sigma)=\alpha\sigma^{2}$. Inset : variation of
the fitting parameter $\alpha$ with $\Phi_{0}$. The strength of the electric
field $\epsilon=3\times 10^{7}\textrm{V/cm}$ and $E_{F}=0.05$ eV.

In Fig. 7 we show the dependence of $\log(J_{\textrm{av}}/J_{0})$ with
$\sigma$ for various field strength $\mathcal{E}$ and $\Phi_{0}$. As in the
case with $\Phi_{0}\ll E_{F}$, effect of work function distribution is
strongest for the weakest field $\mathcal{E}=10^{7}$ V/cm. The tunneling
current for $\mathcal{E}=10^{7}$ V/cm is very small (
$10^{-10}\textrm{A/cm}^{2}$) and very small change (lowering) of potential
barrier can have much larger effect. With increasing field strength by one
order of magnitude changes the tunneling current by several orders of
magnitude which largely cancels the effect of work function distribution.
However for each case $\log(J_{\textrm{av}}/J_{0})$ can be fitted well with
$f(\sigma)=\alpha\sigma^{2}$. The fitting parameter $\alpha$ strongly depends
on $\mathcal{E}$ as shown in the inset of each plot.

Figure 7: (Color online) Panel plot : Logarithm of the scaled average current,
$J_{\textrm{av}}/J_{0}$, as a function of the width, $\sigma$ of the Gaussian
work function distribution for various electric field strength, $\mathcal{E}$.
Solid lines with symbols are numerical data and the dashed lines are fitting
function $f(\sigma)=\alpha\sigma^{2}$. Inset of each plot shows the variation
of $\alpha$ with field strength, $\mathcal{E}$. The value of $\Phi_{0}\gg
E_{F}$ is indicated in each plot and $E_{F}=0.05$ eV for all the plots.

###  IV.2 Log-normal work function distribution

Next we consider log-normal distribution for the work function. In Fig. 8 we
show the histogram plot of the work function, sampled over log-normal
distribution for four choice of median work function $M\equiv
e^{\mu}=3.0,3.5,4.0$ and 4.5 eV with $\sigma=0.05$. From this plot we can see
that our choice of random variables for $\Phi$ are well sampled over the log-
normal distribution.

(a)

(b)

(c)

(d)

Figure 8: (Color online) Histogram plot of work function, $\Phi$, sampled over
log normal distribution for various $\mu$ (see text). We have chosen
$\sigma=0.05$ for all the figures.

####  IV.2.1 Case with $\Phi\ll E_{F}$

First we consider the familiar case with $\Phi\ll E_{F}$. We choose $E_{F}=10$
eV and $\mu=\ln 3.0,\ln 3.5,\ln 4.0$ and $\ln 4.5$. This will correspond to
median value $M=3.0,3.5,4.0$ and 4.5 eV. In Fig. 9 we show $J_{\textrm{av}}$
as a function of number of random variables for $\Phi$, chosen from log-normal
distribution. The current density is several orders of magnitude higher than
the case of Gaussian distribution. Also the fluctuations are higher than the
case of Gaussian distribution. As in the earlier cases, we choose $N=10^{5}$
to get good statistical measure.

Figure 9: (Color online)Average current, $J_{\textrm{av}}$, as a function of
number of random variables, N, chosen from Log-normal distribution for
$\Phi\ll E_{F}$.

In Fig. 10 we show $\log(J_{\textrm{av}}/J_{0})$ as a function of the width
($\sigma$) of the log-normal work function distribution for $M=3.0,3.5,4.0$
and 4.5 eV and a given field strength $\mathcal{E}=3\times 10^{7}$ V/cm,
respectively. As in the case of Gaussian distribution, $J_{\textrm{av}}$
increases monotonically as a function of the width of the distribution,
$\sigma$. However the increase in this case is much stronger - more than 10
times $J_{0}$ for $\sigma=0.1$. This strong increase can be attributed to the
skewness of distribution function. There is significantly higher probability
of lowering of effective potential barrier than the Gaussian distribution with
no skewness. As a result, the current density
$J\propto\exp(-\zeta\Phi^{3/2}/\mathcal{E})$ will show exponential increase
with the width of the distribution. As shown in Fig. 10,
$\log(J_{\textrm{av}}/J_{0})$ can be fitted well with the function
$g(\sigma)=\gamma\sigma^{n}$ with non-universal exponent $n>1$. The fitting
parameter $\gamma$ increases linearly with the mean of the distribution.

Figure 10: (Color online) Logarithm of scaled average current,
$J_{\textrm{av}}/J_{0}$, as a function of the width, $\sigma$, of the log-
normal work function distribution for $\Phi\ll E_{F}$. Solid lines with
symbols are numerical data and the dashed lines are fitting to
$g(\sigma)=\gamma\sigma^{n}$, where $n>1$ is a non-universal exponent
characterizing compressed exponential behaviour of $J_{\textrm{av}}$. Inset :
variation of fitting parameter $\gamma$ with the median of the distribution
$M=\exp(\mu)$. The value of the exponents $n$ for each case is shown on the
figure and the value of the field strength $\epsilon=3\times
10^{7}\textrm{V/cm}$ and $E_{F}=10$ eV.

In Fig. 11 we show $\log(J_{\textrm{av}}/J_{0})$ as a function of the width,
$\sigma$, of the log-normal work function distribution function for various
field strength $\mathcal{E}$ varying over one order of magnitude - from
$10^{7}\textrm{V/cm}$ to $10^{8}\textrm{V/cm}$. For the lowest field
$\mathcal{E}=10^{7}\textrm{V/cm}$, $J_{\textrm{av}}$ increases by several
orders of magnitude over $J_{0}$. This is mainly because of the fact that low
field tunneling current is very small and the effect of averaging over work
function distribution together with the skewness of the distribution increases
the tunneling current drastically. Also the rise is near exponential as $n\sim
1$. With increasing $\mathcal{E}$ the tunneling current increases and the
effect of work function distribution gets largely suppressed. The fitting
parameter $\gamma$ strongly depends on the field strength $\mathcal{E}$.

Figure 11: (Color online) Panel plot : Logarithm of scaled average current,
$J_{\textrm{av}}/J_{0}$, as a function of the width, $\sigma$, of the log-
normal work function distribution for various applied field strength,
$\mathcal{E}$. $\mathcal{E}$ varies from $10^{7}\textrm{V/cm}$ (top most
curve) to $10^{8}\textrm{V/cm}$ (bottom most curve) in each panel. Solid lines
with symbols are numerical data and dashed lines represent fitting
$g(\sigma)=\gamma\sigma^{n}$, where $n>1$ is non-universal exponent. Inset of
each plot shows variation of $\gamma$ with $\mathcal{E}$ and the variation $n$
is indicated in each plot. $\mu$ is shown in each plot and $E_{F}=10.0$ eV for
all the plots.

####  IV.2.2 Case with $\Phi\gg E_{F}$

Finally we consider the case with $E_{F}=0.05$ eV and log-normal work function
distribution. As in the other cases we first study the dependence of
$J_{\textrm{av}}$ on the number of random variables $N$. As shown in Fig. 12
$J_{\textrm{av}}$ shows strong fluctuations for $N<10^{4}$ and settles to a
mean value for $N>6\times 10^{4}$. We choose $N=10^{5}$ for calculations. The
magnitudes of currents are several orders of magnitude higher than the case of
Gaussian distribution. However the currents are much smaller than the cases
with $E_{F}=10.0$ eV and log-normal distribution.

Figure 12: (Color online)Average current ($J_{\textrm{av}}$) as a function of
number of random variables(N) chosen from Log-normal distribution for $\Phi\gg
E_{F}$.

In Fig. 13 we show $\log(J_{\textrm{av}}/J_{0})$ as a function of the width of
the log-normal work function distribution for a given field strength
$\mathcal{E}=3\times 10^{7}$ V/cm for $\mu=\ln 3.0,\ln 3.5,\ln 4.0$ and $\ln
4.5$, respectively. This choice of $\mu$ corresponds to median value of the
distribution $M=3.0,3.5,4.0$ and 4.5 eV, respectively. As in the earlier cases
$J_{\textrm{av}}$ monotonically increases with $\sigma$. As in the earlier
cases $\log(J_{\textrm{av}}/J_{0})$ can be fitted well with
$g(\sigma)=\gamma\sigma^{n}$ with $n>1$. The fitting parameter $\gamma$ and
the exponent $n$ are nearly identical with the case with $\Phi\ll E_{F}$. With
increasing $M$, $\log(J_{\textrm{av}}/J_{0})$ increases and the exponent $n$
decreases. This is because with increasing $M$ ($\Phi$) the tuneling current
decreases and thereby enhances the work function distribution effect and near
exponental increase in tunneling current with $\sigma$. The fitting parameter
$\gamma$ behaves linearly with $M=\exp(\mu)$

Figure 13: (Color online) Logarithm of scaled average current,
$J_{\textrm{av}}/J_{0}$, as a function of the width, $\sigma$, of the log-
normal work function distribution for $\Phi\gg E_{F}$. Solid lines with
symbols are numerical data and the dashed lines represent fitting
$g(\sigma)=\gamma\sigma^{n}$, where $n>1$ is a non-universal exponent
indicating compressed exponential behaviour of $J_{\textrm{av}}$. The value of
$n$ for each case is indicated in the figure. Inset : variation of $\gamma$
with the median of the distribution $\exp(\mu)$. The applied field strength
$\epsilon=3\times 10^{7}\textrm{V/cm}$ and $E_{F}=0.05$ eV for all the cases.

In Fig. 14 we show $\log(J_{\textrm{av}}/J_{0})$ as a function of the width,
$\sigma$, of the log-normal work function distribution for various field
strength, $\mathcal{E}$, as well as the median of the distribution
$M=e^{\mu}$. For the lowest field strength $\mathcal{E}=10^{7}\textrm{V/cm}$
the tunneling current, $J_{\textrm{av}}$, averaged over the work function
distribution is several orders of magnitude larger than $J_{0}$, the tunneling
current calculated with $\Phi=M$. As stated earlier this is due to the fact
that for $\mathcal{E}=10^{7}$ V/cm the tunneling current is very small and the
skewness of the work function distribution effetively reduces the potential
barrier significantly during the averaging process. With increasing field
strength the tunneling current increases over several orders of magnitude and
the effect of the work function distribution gets masked. With increasing $M$
the tunneling current decreases and the work function distribution effect
enhances. For each case $\log(J_{\textrm{av}}/J_{0})$ can be fitted well with
$g(\sigma)=\gamma\sigma^{n}$ with $n>1$. The exponent, $n$, increases with the
increasing field strength. For a given field strength, $\mathcal{E}$, $n$
decreases with increasing median value $M$ of the work function distribution.
In particular, for $\mathcal{E}=10^{7}$ V/cm $n\sim 1$ giving rise to near
exponential behaviour of the work function averaged tunneling current. The
fitting parameter $\gamma$ sharply decreases with increasing field strength.

Figure 14: (Color online) Panel plot : Logarithm of scaled average current
$J_{\textrm{av}}/J_{0}$, as a function of the width, $\sigma$, of the log-
normal work function distribution for various applied field strength,
$\mathcal{E}$. $\mathcal{E}$ varies from $10^{7}\textrm{V/cm}$ (top most
curve) to $10^{8}\textrm{V/cm}$ (bottom most curve) in each plot. Solid lines
with symbols are numerical data and the dashed lines represent the fitting
function $g(\sigma)=\gamma\sigma^{n}$, where $n>1$ is non-universal exponent.
The value of $n$ for each curve is indicated in each plot. Inset of each plot
shows variation of $\gamma$ with the median of distribution $\exp(\mu)$. Value
of $\mu$ is explicitly shown in each plot and $E_{F}=0.05$ eV for each plot.

##  V Conclusions

In conclusion, we have considered the role of work function distribution on
the tunneling current in field emission effect. We have calculated the
tunneling current, $J_{\textrm{av}}$, averaged over work function
distribution. We have considered both Gaussian as well as a log-normal work
function distribution. For each case we have studied dependence of
$J_{\textrm{av}}$ on $N$, the number of random variables sampled over work
function distribution. $J_{\textrm{av}}$ shows significant fluctuations for
$N<10^{4}$ but settles to an average value for $N>4\times 10^{4}$. We choose
$N=10^{5}$ through out the calculation. We first calculate $J_{0}$, the
tunneling current calculated for $\Phi=\Phi_{0}$(Gaussian) or $\mu=\ln M$
(log-norml), where $\Phi_{0}$ or $M$ (median of the distribution) corresponds
to the bulk value of the work function. For each case we have studied the
logarithm of the scaled current distribution $\log(J_{\textrm{av}}/J_{0})$ as
a function of the width, $\sigma$, of the work function distribution for
various $\Phi_{0}$ (Gaussian) or $M$ (log-normal) and the applied field
strength $\mathcal{E}$. We have also considered systems with high density
characterized by $\Phi\ll E_{F}$ ($E_{F}=10$ eV) as well as systems with low
density characterized by $\Phi\gg E_{F}$ ($E_{F}=0.05$ eV). Both for Gaussian
and log-normal distribution $\log(J_{\textrm{av}}/J_{0})$ increases
monotonically with the increasing width, $\sigma$, of the work function
distribution for a given field strength $\mathcal{E}$. This is due to the fact
that averaging over work function distribution leads to exploration of the
regions of lower potential height and hence increased tunneling current. The
dimensionless quantity $\log(J_{\textrm{av}}/J_{0})$ can be fitted with
$f(\sigma)=\alpha\sigma^{2}$ (Gaussian) or with $g(\sigma)=\gamma\sigma^{n}$
(log-normal). The fitting parameters $\alpha$ and $\gamma$ increases linearly
with the bulk value of the work function i. e. $\Phi_{0}$ (Gaussian) or $M$
(log-normal). For log-normal distribution the non-universal exponent $n>1$
shows compressed exponential behaviour.

### Acknowledgements

We would like to thank Nei Lopes, Arghya Taraphder for many valuable
discussions. One of us (N. P) would like to thank IIT, Kharagpur for local
hospitality where part of the work was done. One of us (R. M) would like to
thank Cetral Electronics Engineering Research Institute, Pilani for providing
local hospitality and research support where part of the work was done.

## References

  * (1) R. H. Fowler and L. Nordheim, Proc. Roy. Soc. London A, 119, 173 (1928).
  * (2) E. L. Murphy and R. H. Good, Phys. Rev., 102, 1464 (1956).
  * (3) T. C. Choy, A. H. Harker and A. M. Stoneham, J. Phys. Cond. Matt., 17, 1505 (2005).
  * (4) P. H. Cutler, J. He, J. Miller, N. M. Miskovsky, B. Weiss and T. E. Sullivan, Progress in Surface Science, 42,169185 (1993).
  * (5) R. G. Forbes, K. L. Jensen, Ultramicroscopy, 89, 1722 (2001).
  * (6) C. J. Edgcombbe, Phys. Rev. B, 72, 045420 (2005).
  * (7) K. L. Jensen and M. Cahay, Appl. Phys. Lett., 88, 154105 (2006).
  * (8) A. Fischer, M. S. Mousa, and R. G. Forbes, J. Vac. Sci. Technol. B, 31, 032201 (2013).
  * (9) R. G. Forbes, A. Fischer, and M. S. Mousa, J. Vac. Sci. Technol. B,31, 02B103 (2013).
  * (10) A. Kyritsakis and J. P. Xanthakis, Proc. R. Soc. A, 471, 20140811 (2015)
  * (11) J. T. Holgate and M. Coppins, Phys. Rev. Appl., 7(4), 044019 (2017).
  * (12) XLiana Gamez, Maxwell Terban, Simon Billinge and Maria Martinez-Inesta, 50, 741 (2017).
  * (13) Nei Lopes and A. V. Andrade-Neto, arXiv:1408.3663v3
  * (14) Nei Lopes and A. V. Andrade-Neto, Phys. Lett. A, 384, 126399 (2020).
  * (15) D. Biswas and R. Ramachandran, Phys. Plasmas 24, 073107 (2017).
  * (16) A. Haug, Theoretical Solid State Physics, Volume 1, (Pergamon Press, Oxford, 1975.)
  * (17) R. G. Forbes, Appl. Phys. Lett. 89, 113122 (2006).
  * (18) R. G. Forbes and J. H. B. Deane, J. Vac. Sci. Technol. B, 28, C2A33 (2010).
  * (19) W. W. Dolan, Phys. Rev. 91, 510 (1953).
  * (20) I. S. Gradshteyn and I. M. Ryzhik, Tables of Integrals, Series and Products. Academic, New York (1965).