# KeyGen2Vec: Learning Document Embedding via Multi-label Keyword Generation
in Question-Answering

Iftitahu Ni’mah♣,♠ Samaneh Khoshrou♣ Vlado Menkovski♣ Mykola Pechenizkiy♣
♣ Eindhoven University of Technology ♠ BRIN Indonesia
{i.nimah, v.menkovski<EMAIL_ADDRESS>
###### Abstract

Representing documents into high dimensional embedding space while preserving
the structural similarity between document sources has been an ultimate goal
for many works on text representation learning. Current embedding models,
however, mainly rely on the availability of label supervision to increase the
expressiveness of the resulting embeddings. In contrast, unsupervised
embeddings are cheap, but they often cannot capture implicit structure in
target corpus, particularly for samples that come from different distribution
with the pretraining source.

Our study aims to loosen up the dependency on label supervision by learning
document embeddings via Sequence-to-Sequence (Seq2Seq) text generator.
Specifically, we reformulate keyphrase generation task into multi-label
keyword generation in community-based Question Answering (cQA). Our empirical
results show that KeyGen2Vec in general is superior than multi-label keyword
classifier by up to 14.7% based on Purity, Normalized Mutual Information
(NMI), and F1-Score metrics. Interestingly, although in general the absolute
advantage of learning embeddings through label supervision is highly positive
across evaluation datasets, KeyGen2Vec is shown to be competitive with
classifier that exploits topic label supervision in Yahoo cQA with larger
number of latent topic labels. 111The empirical study was completed in 2020 at
Eindhoven University of Technology.

##  1 Introduction

(a) Doc2Vec clusters

(b) Fitting labels

(c) Supervised clusters

(d) Fitting labels

Figure 1: The expressiveness between unsupervised and supervised document
embeddings3332-D Projection is based on t-SNE. Points are colored according to
the predicted cluster ids $\Omega$ (left) and exact classes $\mathbb{C}$
(right). For the consistency plotting of cluster membership colors, exact
classes was mapped to prediction cluster ids by “Hungarian” algorithm.. Upper:
unsupervised embeddings via Doc2Vec. Lower: supervised embeddings via a multi-
class classifier.

Keywords or _tags_ have been widely used in community-driven Question-Answer
(cQA) systems and many online platforms, such as Twitter, and online
bibliographic databases and search engines, as metadata describing sub-topics
of an article. Obtaining keywords as supervisory information for training any
machine learning models is considered to be cheaper than obtaining topic
labels since there exists automated keyword extraction methods, e.g. TfIdf,
TextRank Mihalcea and Tarau (2004), SingleRank Wan and Xiao (2008), Maui
Medelyan et al. (2009), which in practice is often accompanied by human
validation to control the quality of keyword labels.

Figure 2: KeyGen2Vec. Model is trained in _autoregressive_ mode. In teacher
forcing mode, $t-1$ shifted version of keyword is given to the model as input
for decoder. Circles represent RNN states in encoder (left/source) and decoder
(right/target) network.

Despite many potential benefits of keywords or tags, such as providing
predictable patterns Golder and Huberman (2005, 2006); Nimah et al. (2019,
2021); knowledge organization and resource discovery Macgregor and McCulloch
(2006); Ames and Naaman (2007); improving retrieval performance Hotho et al.
(2006); Nimah et al. (2019), less attention has been paid to incorporate
keywords as a condition to embed documents in high dimensional embedding
space. Existing approaches that incorporate keywords, tags, or phrases as
additional semantic knowledge for document clustering can be divided into two:
(1) approaches that focus on improving the quality of document embeddings
during training Pu et al. (2015); Sato et al. (2017); and (2) approaches that
focus on improving the clustering algorithm or subsequent tasks by providing
additional post-pipelines Ramage et al. (2009); Rosa et al. (2011); Dao et al.
(2018) to intensify the expressiveness of representations in latent space,
such that semantically similar points in that space are close together
compared to dissimilar points. However, these works depend on multiple
pipelines, which consequently hinder their reproducibility and adaptation as
end-to-end system in many real world NLP applications.

stepwise/.style n args=2 edge path= [draw, edge] (!u.parent anchor) —-
+(#1,#2) —- (.child anchor)edge label; , my shading/.style= for tree=
text/.wrap pgfmath arg=black!##1!#110*level(), edge/.wrap pgfmath arg=-¿,
draw=black!##1!#110*level(), , ,  for tree= edge=-¿, grow=east, align=left,
child anchor=west, edge path= [draw, edge] (!u.parent anchor) —- (.child
anchor)edge label; , font= [Approaches [3. Distributed [e. Discrete predictor
(Supervised) [2. Multi-Label Classifier; [ 4, 5 ] in table 1, name=multilbl]
[1. Multi-Class Classifier; [ 3 ] in table 1, name=upperbound] ] [d.
Pretrained Sentence Encoder: S-BERT Reimers and Gurevych (2019); [ 6 ] in
table 1] [c. Paragraph Vector (Doc2Vec) Le and Mikolov (2014) [ 8,9 ] in table
1] [b. Seq2Seq [2. KeyGen2Vec
[ 1 ] in table 1, my shading=vermilion, name=framework] [1. Autoencoder
Sutskever et al. (2014); Cho et al. (2014); [ 2 ] in table 1] ] [a. Bottom-Up
[2. Covariance Matrix Nikolentzos et al. (2017); Torki (2018) [b. Neural Word
Embedding;
[ 17,18,20 ] in table 1 ] [a. PMI; [ 19 ] in table 1] ] [1. Mean Embedding [d.
Pretrained $\rightarrow$ Wiki2Vec Yamada et al. (2020)
and fastText Bojanowski et al. (2017); [ 11, 13 ] in table 1] [c. Pretrained
$\rightarrow$ GloVe
Pennington et al. (2014); [ 10, 12 ] in table 1] [b. Trainable $\rightarrow$
Word2Vec
Mikolov et al. (2013b, a); [ 14, 15 ] in table 1 ] [a. Pointwise Mutual
Information (PMI)
Levy and Goldberg (2014); [ 16 ] in table 1 ] ] ] ] [2. Probabilistic
[Dirichlet-based Topic Model (LDA)
Blei et al. (2010); Blei (2012);
[7] in table 1 ] ] [1. Non-Distributed [TfIdf Salton et al. (1975); [21] in
table 1 ] ] ] [draw=vermilion, solid, line width=1mm, inner sep=.4em,
fit=(framework), label=[align=left]0: Proposed Framework ] ;

Figure 3: Document embedding approaches in this study.

Our work mainly focuses on topical clustering of cQA archives as a subsequent
task to evaluate currently available document embedding approaches, including
the proposed KeyGen2Vec framework. As a motivating example, Figure 1(a)-1(b)
illustrate how unsupervised-based embeddings are likely random, indicating the
model’s incapability to capture semantic aspects such as latent topics
structure inferred in target data. By contrast, supervised approach is more
expressive, producing separable clusters in latent space that are coherent
with topics, as shown in Figure 1(c)-1(d). However, the latter model requires
learning document embeddings with topics as label supervision. So, it is more
costly than the unsupervised embedding approaches.

To negotiate the trade-offs between utilizing unsupervised and supervised
approaches for learning document embeddings, we utilize keywords as sub-latent
structure in corpora to train Seq2Seq networks referred to as KeyGen2Vec. Our
work holds an assumption that learning a conditioned sequence-to-sequence
mapping between documents and their corresponding keywords equals to learning
the structural similarity that hierarchically links contents in document,
keywords as explicit document abstractions, and topics as latent variables
that further group documents based on keywords co-occurrences. For a fair
comparison, we also train Multi-label and Multi-class Neural Network
classifiers as supervised approaches to learn document embeddings on cQA data.
The main difference between our proposal and classifier-based approaches is
that the classifiers view keywords and topics as discrete labels $Y\in R^{d}$,
while the proposed KeyGen2Vec sees keywords as a sequence of discrete
structure $Y\in\Sigma^{*}$.

Summarizing, our main contributions are:

  * •

We introduce KeyGen2Vec, a simple Seq2Seq framework that can be utilized as a
general tool to learn document embeddings conditioned on sub-topics
information, such as keywords.

  * •

We comprehensively investigate currently available approaches for learning
document embeddings

We empirically show that unsupervised approaches often produce clusters that
are incoherent with hidden semantics or latent structure inferred in target
data.

  * •

We empirically show that training Seq2Seq networks on multi-label keyword
generation is analogous to indirectly incorporating label dependency
assumption.

We demonstrate that the proposed KeyGen2Vec is superior than a classifier that
is trained on multi-label classification task with document source as inputs
and keywords as target outputs for the models.

##  2 Background

###  2.1 Community-based Question Answering

Our study focuses on investigating the potential usefulness of state-of-the-
art document embeddings for clustering cQA archives with topics as latent
structural similarity. Most of previous studies on cQA archives are
centralized on the exploration of retrieval issues, such as learning latent
topics for question retrieval Cai et al. (2011), a retrieval framework with
neural network embedding P et al. (2017), hybrid approach of neural network
and latent topic clustering to rank the candidate answers given question Yoon
et al. (2018); and textual similarity problems between questions and their
candidate answers Wang et al. (2010); Tan et al. (2016); Yang et al. (2018).
Whereas, previous works on clustering cQA archives mainly focus on improving
clustering algorithm based on simple feature extractor method (e.g. TfIdf)
Momtazi and Klakow (2009); P (2016). Topical clustering itself is previously
studied by Rosa et al. (2011) to organize large unstructured twitter posts
into topically coherent clusters with hashtags as a means of guidance.

###  2.2 Document Embedding

No | Model | GLO | SUB | SEQ | PRE | TRA | DIM
---|---|---|---|---|---|---|---
1 | KeyGen2Vec | - | ✓ | ✓ | - | ✓ | 200
2 | S2S-AE | - | - | ✓ | - | ✓ | 200
3 | FC-Mult-Cls ∗) | ✓ | - | - | - | ✓ | 100
4 | Sigm-Mult-Lbl | - | ✓ | - | - | ✓ | 100
5 | Softm-Mult-Lbl | - | ✓ | - | - | ✓ | 100
6 | S-BERT | - | - | - | ✓ | - | 768
7 | LDA-Topic | - | - | - | - | ✓ | ∗
8 | D2V-DBOW100 | - | - | - | - | ✓ | 100
9 | D2V-PVDM100 | - | - | - | - | ✓ | 100
10 | Avg-GloVe100 | - | - | - | ✓ | - | 100
11 | Avg-w2v100 | - | - | - | ✓ | - | 100
12 | Avg-GloVe300 | - | - | - | ✓ | - | 300
13 | Avg-w2v300 | - | - | - | ✓ | - | 300
14 | Avg-w2v50-tr-sm | - | - | - | - | ✓ | 50
15 | Avg-w2v50-tr-lg | - | - | - | - | ✓ | 50
16 | Avg-PMI50 | - | - | - | - | ✓ | 50
17 | DC-GloVe100 | - | - | - | ✓ | - | $100^{2}/2$
18 | DC-w2v100 | - | - | - | ✓ | - | $100^{2}/2$
19 | DC-PMI50 | - | - | - | - | ✓ | $50^{2}/2$
20 | DC-w2v50-tr-lg | - | - | - | - | ✓ | $50^{2}/2$
21 | TfIdf | - | - | - | - | ✓ | ∗∗

Table 1: Properties of models compared in this study: GLO: Global semantics
(topic labels) are exposed to the model; SUB: Sub-semantic structures
(keywords) are exposed to the model; SEQ: Model with sequential assumption;
PRE: Use pretrained embedding - no finetuning; TRA: training on observed
corpus; DIM: Dimension of embeddings; ∗ depends on the chosen hyper-parameter
($N$ topics) ; ∗∗ \- depends on the vocabulary size in corpus. ∗) is upper
bound model.

Our study on currently available document embeddings is constrained on
approaches that are domain independent. Since most space is devoted to the
proposed framework and model evaluation, we refer the future readers to the
original papers. Figure 3 shows document embedding approaches that are being
observed in this study, which we broadly divided based on three categories:
(1) Non-distributed (frequency-based) approach; (2) Probabilistic approach;
and (3) Distributed (neural-based) embedding learning. The property of each
embedding model is briefly described in Table 1. For a fair comparison, we
include methods that learn embeddings based on global semantic structure
(GLO), sub-semantic structure (SUB), sequential assumption (SEQ), pretrained
embeddings (PRE), and directly trained embeddings on the target corpus (TRA).

##  3 KeyGen2Vec Framework

Figure 4: Corpus as hierarchical semantic network of documents, keywords, and
topics.

KeyGen2Vec is built based on a hierarchical semantic assumption of a corpus,
as briefly illustrated in Figure 4. The assumption is that documents and their
corresponding keyword labels form sub-structures or _sub-networks_ of latent
topic structure as global semantics. Our work adopts Seq2Seq-based keyphrase
generation introduced by Meng et al. (2017); Chen et al. (2018). While these
preliminary works are motivated by the intuition of Seq2Seq capturing document
semantics, there is currently neither analysis nor empirical evidence to
support the claim that the learnt context representation has encapsulated
latent semantic concept of document source conditioned on its keyword labels.
We hypothesize that Seq2Seq network that has been trained on a keyword
generation task is capable of capturing such latent semantic structure
inferred in data.

(a)

(b)

(c)

Figure 5: Training and Inference stages of the proposed KeyGen2Vec; (a)
Observation set; (b) Training; (c) Prediction in test set; $c$ is latent
topic.

Figure 5 illustrates the reformulation of multi-label keyword generation as
the training objective of KeyGen2Vec. The objective of the task is to
approximate the mapping function $f:X\mapsto Y$ \- where $X$ denotes a
collection of documents and $Y$ denotes the corresponding set of keywords in
observation set. These sets of observations
$\\{(x_{i},\\{y_{i}^{1},y_{i}^{2}\\})\\}$ were transformed into one-to-one
training examples $\\{x_{i},y_{i}^{1}\\},\\{x_{i},y_{i}^{k}\\}$ (fig. 5(b)).
Each training example is represented as sequences, $\mathcal{X}:\Sigma^{*}$
and $\mathcal{Y}:\Sigma^{*}$. In inference stage, to evaluate how well the
trained Seq2Seq capture the semantic structure inferred in $f$, the
parameterized encoder decoder model $g$ was further utilized as a decoder
framework, to generate keywords given unseen documents. Details of
architecture used is further explained in sec.3.1.

###  3.1 Architecture

Our framework is built based on a standard Sequence-to-Sequence (Seq2Seq)
encoder-decoder framework. An encoder first maps a sequence of words to a
vector $c$ – where $c$ serves as the resulting document embedding. Given the
encoded embedding of document source $c$, the decoder then generates target
sequences.

$\displaystyle\texttt{ENC:}x=\\{w_{1},\ldots,w_{T_{x}}\\}\mapsto
c\in\mathbb{R}^{d}$ $\displaystyle\texttt{DEC:}c\in\mathbb{R}^{d}\mapsto
y=\\{w_{1},\ldots,w_{T_{y}}\\}$

#### Encoder

The encoder network is constructed of bidirectional GRU units for mapping
sequence of embedded words $e_{t\cdots T_{x}}$ into a sequence of intermediate
state representation $h_{t\cdots T_{x}}$, which is a concatenation of forward
and backward hidden states $h_{t\cdots
T_{x}}=[\overrightarrow{h},\overleftarrow{h}]$.

$\displaystyle\overrightarrow{h_{t\ldots
T_{x}}}=\overrightarrow{GRU}(e_{t\cdots T_{x}})$
$\displaystyle\overleftarrow{h_{t\ldots T_{x}}}=\overleftarrow{GRU}(e_{t\cdots
T_{x}})$

#### Decoder

The decoder is a neural language model based on forward GRU network that
conditions on context embedding of encoder $c$. $s_{t-1}$ is decoder state at
previous time step. $y_{t-1}$ denotes prediction at $t-1$. Here, $g(.)$
denotes prediction layer (dense network) with softmax activation function.

$\displaystyle s_{t}=\overrightarrow{\rm GRU}(y_{t-1},s_{t-1},c)$
$\displaystyle p(y_{t}|y_{1,\ldots,t-1},x)=g(y_{t-1},s_{t},c)$

#### Attention

We use Bahdanau’s MLP attention scoring function Bahdanau et al. (2014) to
calculate attention score $\alpha$ corresponds to the importance weight of
words in source sequence given embedding of words in target sequence.

$\displaystyle\alpha_{t}=\frac{exp(score(s_{t}^{(j)},h_{t\cdots
T_{x}}^{(i)}))}{\sum_{t}^{T_{x}}{exp(score(s_{t}^{(j)},h_{t\cdots
T_{x}}^{(i)}))}}$

#### Context (Document) Embeddings

The final document embedding $c$ is computed based on weighted sum between a
sequence of encoder states and attention score.

$\displaystyle{c}_{t}=\sum_{t=1}^{T_{x}}{\alpha_{t}h_{t}}$

###  3.2 On Label Dependency Assumption

In our proposed KeyGen2Vec framework, keywords as target variables are
represented as sequences of words. The probability of a particular keyword
chosen in inference stage equals to the joint probability of words in sequence
$p(w_{1:T})$. Softmax activation function is used for projecting decoder
states $s_{t-1}\in R^{d}$ into probabilistic values over $\mathcal{V}$
vocabulary size, $\in R^{\mathcal{V}}$.

$\displaystyle
p(w_{t}|w_{1},\ldots,w_{t-1};\theta)=\texttt{softmax}(Ws_{t-1}+b)$
$\displaystyle p(w_{1:T})=\prod_{t}p(w_{t}|w_{1},\ldots,w_{t-1})$

where softmax function is formally given by:

$\displaystyle\texttt{softmax}(z)_{i}=\frac{e^{z_{i}}}{\sum_{j=1}^{K}e^{z_{j}}}$

By dividing each softmax unit (the probability of each word $w_{t}$ in
vocabulary $\mathcal{V}$) with the sum of all units, the total probability of
words in $\mathcal{V}$ is ensured to be $1$. An increase of one class
probability $p(y_{t}|x,\theta)$ causes the probability of other class
decreases.

We hypothesize that by transforming one-to-many training objective in multi-
label keyword generation task into one-to-one multi-class learning scheme, as
shown in Figure 5), we indirectly incorporate label dependency assumption
during training stage. The trained model treats each sample as mutually
exclusive event via softmax normalization and outputs final prediction
$\hat{y_{t}}=\operatorname*{argmax}p(y_{t}|x,\theta)$. This results in an
indirect dependent assumption between a pair of keyword labels since the
probability of particular pair of keywords given the same document source
$p(y_{1}^{1}|x_{1})$ and $p(y_{1}^{2}|x_{1})$ are dependent each other. By
contrast, standard multi-label learning commonly uses independent Bernoulli
assumption via Sigmoid function, disregarding the dependency between labels.
We further investigate this problem by comparing models with Softmax-based
multi-class classification loss and a standard Sigmoid-based Multi-label
classifier.

##  4 Experiments

###  4.1 Data

We use the following data constructed from cQA archives as gold standard for
learning and evaluation. The three data sets represent data with different
level of difficulties w.r.t. sentence length, noise-level, and number of
unique keywords and topic labels. Toy data is considered to be less noisy and
balance – each sub-class category is composed of sentences and their
paraphrases, forming natural cluster structure. Yahoo! data sets with 5 topic
categories (5-T) and 11 topics (11-T) are considered to be more noisy and
imbalanced due to many non-informative words (e.g. digits, measures, url-
links, query about address or web sources) and domain specific terms (e.g.
medical and automotive terms).

Data set | #Topics | #Keywords | #Train | #Test | Sentence
---|---|---|---|---|---
| (GLO) | (SUB) |  |  | Length
Wikianswer | NA | NA |  700M | NA | 9 $\pm$ 3
Toy data | 12 | 77 | 1158 | 290 | 9 $\pm$ 3
5-T Yahoo! cQA | 5 | 120 | 23824 | 5957 | 36 $\pm$ 28
11-T Yahoo! cQA | 11 | 179 | 70962 | 17741 | 35 $\pm$ 28

Table 2: Data set; Wikianswer (original corpus of Toy data) is used to train
Word2Vec (w2v50-tr-lg) and PMI method.

#### Toy Data

We created a small set of hand-labelled sentence-keywords-topic pairs (1448
sentences) from WikiAnswer
444http://knowitall.cs.washington.edu/oqa/data/wikianswers/. WikiAnswer is a
data set composed of millions of questions asked by humans, where each
sentence example is accompanied by its paraphrased versions, forming a
paraphrase cluster of one particular question. We use the original WikiAnswer
corpus to train large scale Word2Vec and PMI models incrementally, to inspect
how the scale of data affects model performance. Table. 9 shows a training
example in Toy data. The number of keywords and topic assignment per sentence
were made fixed, i.e. two keywords and one topic for each sentence.

Source: “the sporozoan plasmodium carried from host to host by mosquitoes
causes what serious infection?
---
Keywords: malaria; plasmodium parasite
Topic: virus and diseases
Table 3: Sentence examples in Toy Data set Source: “what is diabetes mellitus?
diabetes mellitus is medical disorder characterized by varying or persistent
hyperglycemia elevated blood sugar levels, $\dots$”
---
Keywords: diabetes; diseases and conditions
Topic: health
Table 4: Sentence example (concatenated cQA pair) in Yahoo! Answer cQA

#### Yahoo! Answer Comprehensive cQA

We reproduce and extend our result on real world cQA archives consisting of
question-answers pairs, accompanied by keywords (tags) and the corresponding
topic. Data was obtained from Yahoo! Answer Comprehensive cQA dataset
555https://webscope.sandbox.yahoo.com/catalog.php, originated from the query
log of Yahoo! Answer. We constructed two corpora: corpus with 5 topic
categorization – referred to as 5-T cQA and corpus with 11 topics – referred
to as 11-T cQA. The training and test examples were constructed by
concatenating each question and the corresponding answers. Table 4 shows a
training example obtained from Yahoo! Answer cQA archives. Likewise, each
document corresponds to a fixed membership: two keywords and one topic
describing the document semantic abstraction.

###  4.2 Training and Hyper-parameters

For training KeyGen2Vec, we use negative log-likelihood loss function with an
adaptive learning rate optimization (Adam Kingma and Ba (2014)),
$lr=0.001,betas=(0.9,0.98),eps=1e-9$. Curriculum learning Bengio et al. (2015)
was employed to sampling whether to use a teacher forcing method during
training stage. For the other models, we refer the reader to the provided code
documentation.

For LDA, trainable Word2Vec, Paragraph Vector, we used Gensim implementation
666https://radimrehurek.com/gensim/. BERT pretrained sentence encoder is taken
from a recent sentence similarity task Reimers and Gurevych (2019). Specific
for Toy data experiment, we trained two Word2Vec models: small scale model
Avg-w2v50-tr-sm was trained on the constructed set of Toy data; and large
scale model Avg-w2v50-tr-lg was trained incrementally on WikiAnswer (the
original large scale corpora of Toy data) – to inspect how model performance
differs based on the scale of data. Classifiers (Multi-class and Multi-label)
were constructed from fully-connected network (MLP) since we do not find a
significant performance differences between using different types of networks
(i.e. dense, convolutional, and recurrent). We trained two types of Multi-
label classifiers (Sigm-Cls and Softm-Cls) to inspect the effect of
incorporating label dependency in multi-label learning.

###  4.3 Clustering as Evaluation

We use K-Means clustering 777scikit-learn.org/../sklearn.cluster.KMeans.html
to evaluate the quality (clusterability) of document embeddings in this study
(table 1). The hyper-parameter choices of K-means is kept as minimum as
possible (init=’random’, n_clusters=$K$, n_init=$10$, max_iter=$50$). This is
to make sure that the clustering is not overly parameterized, which can
obscure the actual quality of the learnt embeddings. Given the actual global
semantic classes (topic labels) $\mathbb{C}$ in the current observed corpora
and the predicted classes $\Omega$ from K-Means method, we employ Purity,
Normalized Mutual Information (NMI), and F1-score Manning et al. (2008)
metrics to objectively measure whether the resulting clustering $\Omega$ can
recreate or approximate the exact classes $\mathbb{C}$.

###  4.4 $\chi^{2}$ Feature selection

Dining Out | Health | Travel | Cars
---|---|---|---
hamburger | medicine | trip | jeep
taco | heart | map | vehicle
sandwich | symptom | disney | auto
buffet | virus | vacation | manual
cafe | treatment | ticket | nisan

Table 5: Example of influential words per topic category – selected based on
$\chi^{2}$ feature selection method on 5-T Yahoo! cQA.

We employ feature selection based on $\chi^{2}$ method Manning et al. (2008)
to select $N-$ most influential words per topic category. Each training
example is then represented as Bag-of-Influential words with
$N\in\\{20,50,100,250\\}$ for Toy data and
$N\in\\{20,50,100,250,500,1000,2500\\}$ for Yahoo! cQA data. The larger the
size of influential words per category, the more noises preserved in the
training data. This experiment was conducted to investigate: (1) the effect of
noises on the clusterability of embeddings; (2) the effect of incorporating
label dependency via Softmax-based loss on the clusterability of embeddings.

(a) Upper-bound

(b) KeyGen2Vec

(c) Sigm-MultLbl

(d) Softm-MultLbl

(e) Avg-w2v-tr

Figure 6: Example of Clustering Visualization on 5-T Yahoo! Q&A.

##  5 Results and Discussion

We summarize our empirical findings as follows:

#### KeyGen2Vec outperforms multi-label classifiers

Based on the clustering performance on three data sets, as shown in Table 6-8,
we demonstrate that although the model does not exploit the actual topic
labels during training stage, the proposed KeyGen2Vec has a capability of
preserving topical proximity in latent space, outperforming its counterparts –
models trained on multi-label classifiers (Sigm-Mult-Lbl and Softm-Mult-Lbl).

Approach | Purity | NMI | F1-score
---|---|---|---
| All | Test | All | Test | All | Test
KeyGen2Vec | 0.734 | 0.726 | 0.769 | 0.774 | 0.631 | 0.614
S2S-AE | 0.288 | 0.317 | 0.213 | 0.269 | 0.167 | 0.164
FC-Mult-Cls∗ | 0.961 | 0.938 | 0.968 | 0.940 | 0.947 | 0.910
Sigm-Mult-Lbl | 0.590 | 0.605 | 0.629 | 0.649 | 0.435 | 0.438
Softm-Mult-Lbl | 0.659 | 0.687 | 0.673 | 0.710 | 0.532 | 0.556
BERT | 0.636 | 0.644 | 0.680 | 0.678 | 0.527 | 0.502 2
LDA-Topic | 0.474 | 0.551 | 0.472 | 0.584 | 0.368 | 0.454
D2V-DBOW100 | 0.179 | 0.207 | 0.049 | 0.115 | 0.111 | 0.100
D2V-PVDM100 | 0.171 | 0.206 | 0.035 | 0.104 | 0.102 | 0.094
Avg-GloVe100 | 0.648 | 0.649 | 0.656 | 0.672 | 0.507 | 0.485
Avg-W2V100 | 0.686 | 0.679 | 0.694 | 0.689 | 0.535 | 0.489
Avg-GloVe300 | 0.668 | 0.693 | 0.688 | 0.719 | 0.523 | 0.531
Avg-W2V300 | 0.655 | 0.654 | 0.685 | 0.688 | 0.511 | 0.475
Avg-W2v50-tr-sm | 0.208 | 0.243 | 0.076 | 0.152 | 0.116 | 0.112
Avg-W2v50-tr-lg | 0.643 | 0.651 | 0.671 | 0.673 | 0.538 | 0.506
Avg-PMI50 | 0.305 | 0.325 | 0.212 | 0.290 | 0.180 | 0.176
DC-GloVe100 | 0.374 | 0.326 | 0.351 | 0.321 | 0.222 | 0.185
DC-W2V100 | 0.407 | 0.312 | 0.447 | 0.325 | 0.225 | 0.180
DC-PMI50 | 0.282 | 0.313 | 0.180 | 0.266 | 0.162 | 0.167
DC-W2V50-tr-lg | 0.555 | 0.546 | 0.545 | 0.567 | 0.376 | 0.359
TfIdf | 0.564 | 0.611 | 0.615 | 0.649 | 0.377 | 0.401

Table 6: Clustering Evaluation on Toy Data. FC-Mult-Cls∗ is an upper bound
model. Scores were calculated based on average score in 10 iterations of
K-means clustering. Higher is better.

Approach | Purity | NMI | F1-score
---|---|---|---
| All | Test | All | Test | All | Test
KeyGen2Vec | 0.801 | 0.784 | 0.657 | 0.603 | 0.668 | 0.630
S2S-AE | 0.337 | 0.341 | 0.012 | 0.014 | 0.241 | 0.242
FC-Mult-Cls∗ | 0.970 | 0.853 | 0.903 | 0.721 | 0.957 | 0.794
Sigm-Mult-Lbl | 0.763 | 0.738 | 0.567 | 0.522 | 0.635 | 0.606
Softm-Mult-Lbl | 0.772 | 0.744 | 0.579 | 0.533 | 0.657 | 0.622
BERT | 0.329 | 0.325 | 0.005 | 0.007 | 0.278 | 0.283
LDA-Topic | 0.541 | 0.583 | 0.222 | 0.269 | 0.479 | 0.488
D2V-DBOW100 | 0.335 | 0.335 | 0.018 | 0.019 | 0.245 | 0.246
D2V-PVDM100 | 0.337 | 0.339 | 0.012 | 0.012 | 0.274 | 0.276
Avg-GloVe100 | 0.495 | 0.502 | 0.122 | 0.133 | 0.330 | 0.349
Avg-W2V100 | 0.531 | 0.533 | 0.202 | 0.207 | 0.371 | 0.379
Avg-GloVe300 | 0.483 | 0.498 | 0.117 | 0.132 | 0.334 | 0.352
Avg-W2V300 | 0.463 | 0.462 | 0.117 | 0.117 | 0.342 | 0.347
Avg-W2V50-tr | 0.609 | 0.621 | 0.318 | 0.327 | 0.477 | 0.491
Avg-PMI50 | 0.339 | 0.343 | 0.024 | 0.026 | 0.276 | 0.273
DC-GloVe100 | 0.359 | 0.361 | 0.031 | 0.034 | 0.296 | 0.311
DC-W2V100 | 0.426 | 0.413 | 0.117 | 0.105 | 0.356 | 0.353
DC-PMI50 | 0.326 | 0.329 | 0.018 | 0.019 | 0.295 | 0.292
DC-W2V50-tr | 0.329 | 0.328 | 0.012 | 0.013 | 0.343 | 0.342
TfIdf | 0.357 | 0.383 | 0.047 | 0.077 | 0.304 | 0.308

Table 7: Clustering Evaluation on 5-T Yahoo! cQA.

Approach | Purity | NMI | F1-score
---|---|---|---
| All | Test | All | Test | All | Test
KeyGen2Vec | 0.841 | 0.797 | 0.723 | 0.662 | 0.655 | 0.603
S2S-AE | 0.306 | 0.303 | 0.037 | 0.036 | 0.135 | 0.133
FC-Mult-Cls∗ | 0.862 | 0.768 | 0.774 | 0.643 | 0.717 | 0.564
Sigm-Mult-Lbl | 0.729 | 0.723 | 0.545 | 0.535 | 0.487 | 0.479
Softm-Mult-Lbl | 0.739 | 0.718 | 0.589 | 0.538 | 0.508 | 0.493
BERT | 0.274 | 0.229 | 0.016 | 0.016 | 0.211 | 0.229
LDA-Topic | 0.518 | 0.534 | 0.275 | 0.300 | 0.304 | 0.293
D2V-DBOW100 | 0.279 | 0.278 | 0.022 | 0.024 | 0.133 | 0.133
D2V-PVDM100 | 0.281 | 0.280 | 0.019 | 0.018 | 0.167 | 0.162
Avg-GloVe100 | 0.435 | 0.434 | 0.193 | 0.199 | 0.212 | 0.214
Avg-W2V100 | 0.407 | 0.409 | 0.155 | 0.159 | 0.198 | 0.201
Avg-GloVe300 | 0.427 | 0.425 | 0.188 | 0.190 | 0.213 | 0.205
Avg-W2V300 | 0.448 | 0.449 | 0.216 | 0.221 | 0.216 | 0.218
Avg-W2V50-tr | 0.551 | 0.552 | 0.309 | 0.311 | 0.294 | 0.287
Avg-PMI50 | 0.283 | 0.283 | 0.032 | 0.035 | 0.138 | 0.139
DC-GloVe100 | 0.303 | 0.307 | 0.053 | 0.059 | 0.169 | 0.182
DC-W2V100 | 0.328 | 0.321 | 0.088 | 0.089 | 0.203 | 0.209
DC-PMI50 | 0.273 | 0.271 | 0.018 | 0.019 | 0.161 | 0.160
DC-W2V50-tr | 0.303 | 0.301 | 0.043 | 0.043 | 0.203 | 0.203
TfIdf | 0.306 | 0.303 | 0.037 | 0.037 | 0.135 | 0.133

Table 8: Clustering Evaluation on 11-T Yahoo! cQA.

#### Pretrained unsupervised embeddings are more random on noisy datasets

Specific to unsupervised pretrained models (S-BERT, Wikipedia2Vec Avg-w2v100
and GloVe Avg-GloVe300), the results show that while these models perform well
in Toy data, their performance degrades in the other two data sets. This
indicates that the embeddings generalized from the source domain in which the
model is trained on are not sufficient for the target corpora (Yahoo! data).
Fine tuning the models or expanding the vocabulary, however, is beyond the
scope of this study.

This finding specifically challenges the prior belief stating an _off-the-
shell_ encoder that has been trained on large scale data or multi-tasks
learning (e.g. Skip-Thought vector, Universal Sentence Encoder, Sentence-BERT,
pretrained word embeddings) can produce highly generic embedding that performs
well in practice. We argue that a generic pretrained embedding may best fit
for tasks that are less noisy and complement to the pretrained source domain,
exemplified in our Toy data experiment.

#### Trained unsupervised embeddings rely on large-scale data

We observe that unsupervised neural embeddings that are trained on the
observed corpora (e.g. Word2Vec, Doc2Vec, Seq2Seq Autoencoder) seemingly rely
on the scale of data. See how small scale Word2Vec (Avg-w2v50-tr-sm) results
in a notably low performance on Toy data (similar to Autoencoder S2S-AE and
Doc2Vec), as compared to large scale Word2Vec (Avg-w2v50-tr-lg).

We argue that the low quality of unsupervised embeddings in the current study
is due to the models mainly depend on local information in document contents –
there is no strong assumption on differentiating salient features (words)
w.r.t. global semantic aspects, which may hinder their direct utilization on a
subsequent predictive analytics tasks. Specific to LDA topic model, we argue
that their low performance in the current task is due to no strong assumption
on distinguishing between local (keywords - or more specific document theme)
and global (more general) latent topics.

#### The effects of noises on embedding quality

We argue that the main reason why the current clustering task is challenging
for all observed models, specifically unsupervised ones is mainly due to the
_noisy_ characteristic of cQA archives. For instance, topic ”Health” and
“Dining out” may both contain queries about dietary or source of healthy food.
Topic “Cars”, “Travel”, “Local Business” may all contain queries about car
rental and service. We empirically show that in a clean scenario – where
training examples only contain $N$-most influential words w.r.t. topic
category (Toy data experiment in fig.7(a)) unsupervised methods sufficiently
perform well. The performance, however, degrades in the occurrence of noises
(larger pre-selected feature size). By contrast, KeyGen2Vec can maintain its
considerably high performance (fig.7(a)-7(c)) regardless the presence of
noises. This indicates the exposure of keywords as sub-topical information
benefits the model to obtain high quality embeddings.

#### Problem reformulation improves the expressiveness of embeddings

Redefining one-to-many multi-label learning into one-to-one multi-class
learning scheme via Softmax normalization, which we argue is analogous to
indirectly incorporating label dependency (sec. 3.2), benefits KeyGen2Vec and
Multi-label learning in the current study, resulting in a more accurate
embedding (higher $F_{1}$-score, in table 6-8 and fig.7(a)-7(c)).

(a) Toy Data

(b) 5-T Yahoo! Answer Data

(c) 11-T Yahoo! Answer Data

Figure 7: Effect of noises on F1-Score; _best viewed in color. The larger pre-
selected feature set size (x-axis), the more noisy the documents are._

##  6 Conclusion

We extensively investigate document embedding approaches for topical
clustering of cQA archives. We show current limitations of unsupervised
embeddings on dealing with noisy articles, indicating the need of
incorporating strong assumption either on learning approach or data. Our
empirical results highlight the capability of the proposed KeyGen2Vec in
preserving topical proximity in latent space via multi-label multi-class
learning.

## References

  * Ames and Naaman (2007) Morgan Ames and Mor Naaman. 2007.  Why we tag: motivations for annotation in mobile and online media.  In _Proceedings of the SIGCHI conference on Human factors in computing systems_ , pages 971–980.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.  Neural machine translation by jointly learning to align and translate.  _CoRR_ , abs/1409.0473.
  * Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015.  Scheduled sampling for sequence prediction with recurrent neural networks.  In _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1_ , NIPS’15, page 1171–1179, Cambridge, MA, USA. MIT Press.
  * Blei et al. (2010) D. Blei, L. Carin, and D. Dunson. 2010.  Probabilistic topic models.  _IEEE Signal Processing Magazine_ , 27(6):55–65.
  * Blei (2012) David M Blei. 2012.  Probabilistic topic models.  _Communications of the ACM_ , 55(4):77–84.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ , 5:135–146.
  * Cai et al. (2011) Li Cai, Guangyou Zhou, Kang Liu, and Jun Zhao. 2011.  Learning the latent topics for question retrieval in community QA.  In _Proceedings of 5th International Joint Conference on Natural Language Processing_ , pages 273–281, Chiang Mai, Thailand. Asian Federation of Natural Language Processing.
  * Chen et al. (2018) Jun Chen, Xiaoming Zhang, Yu Wu, Zhao Yan, and Zhoujun Li. 2018.  Keyphrase generation with correlation constraints.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4057–4066, Brussels, Belgium. Association for Computational Linguistics.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_.
  * Dao et al. (2018) Thi-Bich-Hanh Dao, Chia-Tung Kuo, SS Ravi, Christel Vrain, and Ian Davidson. 2018\.  Descriptive clustering: Ilp and cp formulations with applications.  In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , pages 1263–1269.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Golder and Huberman (2005) Scott Golder and Bernardo A Huberman. 2005.  The structure of collaborative tagging systems.  _arXiv preprint cs/0508082_.
  * Golder and Huberman (2006) Scott A Golder and Bernardo A Huberman. 2006.  Usage patterns of collaborative tagging systems.  _Journal of information science_ , 32(2):198–208.
  * Hotho et al. (2006) Andreas Hotho, Robert Jäschke, Christoph Schmitz, and Gerd Stumme. 2006.  Information retrieval in folksonomies: Search and ranking.  In _European semantic web conference_ , pages 411–426. Springer.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents.  In _International conference on machine learning_ , pages 1188–1196.
  * Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014.  Neural word embedding as implicit matrix factorization.  In _Advances in neural information processing systems_ , pages 2177–2185.
  * Macgregor and McCulloch (2006) George Macgregor and Emma McCulloch. 2006.  Collaborative tagging as a knowledge organisation and resource discovery tool.  _Library review_.
  * Manning et al. (2008) Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008.  _Introduction to information retrieval_.  Cambridge university press.
  * Medelyan et al. (2009) Olena Medelyan, Eibe Frank, and Ian H. Witten. 2009.  Human-competitive tagging using automatic keyphrase extraction.  In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_ , pages 1318–1327, Singapore. Association for Computational Linguistics.
  * Meng et al. (2017) Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017.  Deep keyphrase generation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 582–592, Vancouver, Canada. Association for Computational Linguistics.
  * Mihalcea and Tarau (2004) Rada Mihalcea and Paul Tarau. 2004.  TextRank: Bringing order into text.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 404–411, Barcelona, Spain. Association for Computational Linguistics.
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b.  Distributed representations of words and phrases and their compositionality.  In _Advances in neural information processing systems_ , pages 3111–3119.
  * Momtazi and Klakow (2009) Saeedeh Momtazi and Dietrich Klakow. 2009.  A word clustering approach for language model-based sentence retrieval in question answering systems.  In _Proceedings of the 18th ACM conference on Information and knowledge management_ , pages 1911–1914.
  * Nikolentzos et al. (2017) Giannis Nikolentzos, Polykarpos Meladianos, François Rousseau, Yannis Stavrakas, and Michalis Vazirgiannis. 2017.  Multivariate Gaussian document representation from word embeddings for text categorization.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 450–455, Valencia, Spain. Association for Computational Linguistics.
  * Nimah et al. (2021) Iftitahu Nimah, Meng Fang, Vlado Menkovski, and Mykola Pechenizkiy. 2021.  ProtoInfoMax: Prototypical networks with mutual information maximization for out-of-domain detection.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1606–1617, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Nimah et al. (2019) Iftitahu Nimah, Vlado Menkovski, and Mykola Pechenizkiy. 2019.  Bsdar: Beam search decoding with attention reward in neural keyphrase generation.
  * P (2016) Deepak P. 2016.  MixKMeans: Clustering question-answer archives.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1576–1585, Austin, Texas. Association for Computational Linguistics.
  * P et al. (2017) Deepak P, Dinesh Garg, and Shirish Shevade. 2017.  Latent space embedding for retrieval in question-answer archives.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 855–865, Copenhagen, Denmark. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Pu et al. (2015) Xiaojia Pu, Rong Jin, Gangshan Wu, Dingyi Han, and Gui-Rong Xue. 2015.  Topic modeling in semantic space with keywords.  In _Proceedings of the 24th ACM International on Conference on Information and Knowledge Management_ , CIKM ’15, page 1141–1150, New York, NY, USA. Association for Computing Machinery.
  * Ramage et al. (2009) Daniel Ramage, Paul Heymann, Christopher D. Manning, and Hector Garcia-Molina. 2009\.  Clustering the tagged web.  In _Proceedings of the Second ACM International Conference on Web Search and Data Mining_ , WSDM ’09, page 54–63, New York, NY, USA. Association for Computing Machinery.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Rosa et al. (2011) Kevin Dela Rosa, Rushin Shah, Bo Lin, Anatole Gershman, and Robert Frederking. 2011\.  Topical clustering of tweets.  _Proceedings of the ACM SIGIR: SWSM_ , 63.
  * Salton et al. (1975) Gerard Salton, Anita Wong, and Chung-Shu Yang. 1975.  A vector space model for automatic indexing.  _Communications of the ACM_ , 18(11):613–620.
  * Sato et al. (2017) Motoki Sato, Austin J. Brockmeier, Georgios Kontonatsios, Tingting Mu, John Y. Goulermas, Jun’ichi Tsujii, and Sophia Ananiadou. 2017.  Distributed document and phrase co-embeddings for descriptive clustering.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 991–1001, Valencia, Spain. Association for Computational Linguistics.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  In _Advances in neural information processing systems_ , pages 3104–3112.
  * Tan et al. (2016) Ming Tan, Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2016.  Improved representation learning for question answer matching.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 464–473, Berlin, Germany. Association for Computational Linguistics.
  * Torki (2018) Marwan Torki. 2018.  A document descriptor using covariance of word vectors.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 527–532, Melbourne, Australia. Association for Computational Linguistics.
  * Wan and Xiao (2008) Xiaojun Wan and Jianguo Xiao. 2008.  Single document keyphrase extraction using neighborhood knowledge.  In _Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 2_ , AAAI’08, page 855–860. AAAI Press.
  * Wang et al. (2010) Baoxun Wang, Xiaolong Wang, Chengjie Sun, Bingquan Liu, and Lin Sun. 2010.  Modeling semantic relevance for question-answer pairs in web social communities.  In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_ , pages 1230–1238, Uppsala, Sweden. Association for Computational Linguistics.
  * Wiseman and Rush (2016) Sam Wiseman and Alexander M. Rush. 2016.  Sequence-to-sequence learning as beam-search optimization.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1296–1306, Austin, Texas. Association for Computational Linguistics.
  * Xu et al. (2019) Hu Xu, Bing Liu, Lei Shu, and Philip Yu. 2019.  BERT post-training for review reading comprehension and aspect-based sentiment analysis.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2324–2335, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Yamada et al. (2020) Ikuya Yamada, Akari Asai, Jin Sakuma, Hiroyuki Shindo, Hideaki Takeda, Yoshiyasu Takefuji, and Yuji Matsumoto. 2020.  Wikipedia2vec: An efficient toolkit for learning and visualizing the embeddings of words and entities from wikipedia.  _arXiv preprint 1812.06280v3_.
  * Yang et al. (2018) Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018.  Learning semantic textual similarity from conversations.  In _Proceedings of The Third Workshop on Representation Learning for NLP_ , pages 164–174, Melbourne, Australia. Association for Computational Linguistics.
  * Yoon et al. (2018) Seunghyun Yoon, Joongbo Shin, and Kyomin Jung. 2018.  Learning to rank question-answer pairs using hierarchical recurrent encoder with latent topic clustering.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1575–1584, New Orleans, Louisiana. Association for Computational Linguistics.

##  Appendix A Document Embedding Methods

###  A.1 Non-distributed approach

#### Frequency-based (TfIdf)

_TfIdf_ Salton et al. (1975) – also referred to as “bag-of-words” model, is
commonly used as a standard approach to transform text document into numerical
representation. A document is represented as vector of semantics, where each
dimension reflects a degree of importance (based on relative frequency-based
weight) of a particular word $w$ in the corresponding document $d_{i}$ and
across documents in corpus $d_{i}\in D$, $w\in\mathcal{V}$ – where
$\mathcal{V}$ denotes vocabulary size of the corpus.

###  A.2 Probabilistic approach

#### Probabilistic Topic Model

Latent Dirichlet Allocation (LDA) Blei et al. (2010); Blei (2012) – commonly
known as Topic Model, is a probabilistic mixture model that views a document
as a collection of un-ordered words (“bag-of-words”). The document is
represented as a mixture membership of topics - where each dimension of the
document vector corresponds to a probabilistic distribution of topic-$c_{j}$
in the corresponding document, $c_{j}\in\mathbb{C}$. Each topic is represented
as a vector of most probable words defining the topic.

###  A.3 Distributed approach

#### Mean Embedding

– a bottom-up approach. A document is viewed as a collection of bag-of-word
embeddings. The word-level embedding is learnt through non-linear mapping of
neural network architecture Mikolov et al. (2013b, a). The document
representation is computed based on averaging word-level representations occur
in the corresponding document. We acquire word embedding $w\in\mathcal{V}$
through both pretrained models, i.e. Word2Vec pretrained on Wikipedia corpus
Yamada et al. (2020), GloVe vector Pennington et al. (2014), and trainable
model – i.e. by training the Word2Vec model Mikolov et al. (2013b, a) and
Pointwise Mutual Information (PMI) - based embedding on the current corpus. In
table 1, the pretrained models of word embedding are represented by {Avg-
GloVe100, Avg-w2v100, Avg-GloVe300, Avg-w2v300}. The trainable models are
represented by {Avg-w2v50-tr-sm, Avg-w2v50-tr-lg, Avg-PMI50}.

#### Document Covariance Matrix

– a bottom-up approach, represented by {DC-GloVe100, DC-w2v100, DC-PMI50,
DC-w2v50-tr-lg} in table 1. A document is represented as the multivariate
gaussian embedding (covariance matrix) of its word-level representation
Nikolentzos et al. (2017); Torki (2018). To construct a document covariance
matrix, we use two types of word-level representations: (1) Pointwise Mutual
Information (PMI) Levy and Goldberg (2014) that measures the association of
words $w$ and their context $c$ by calculating the cooccurrence of words and
their neighboured words in sentence or document; (2) Word embedding Mikolov et
al. (2013b) that learn distributed representation of words-context words via a
neural network architecture - or log linear mapping function (_here_ , we use
both pretrained and trainable word embedding models).

#### Paragraph Vector

– represented by {D2V-DBOW100, D2V-PVDM100} in table 1. Document and the
corresponding words are mapped into a share vector space – where the objective
is to predicting target words, given document and context words in Distributed
Memory model (PV-DM); and to predict context words, given a document in
Distributed Bag-of-Words model (PV-DBOW) Le and Mikolov (2014).

#### Multi-class classifier

We utilize neural network with dense connections (MLP, denoted as FC-Mult-Cls
in table 1), representing an upper-bound model in this study. Discrete global
class structure in the observed data sets $\mathbb{C}$ is exposed as training
objective to learn and condition document features from this MLP-based model.
The network architecture is composed of an embedding layer, dropout networks,
a pooling layer as a flattening mechanism, and a stack of two fully-connected
(FC) networks. The objective of the study, thus is to find the best feature
extractor that closer to the quality of features based Multi-class classifier.

#### Multi-label classifier

To provide a _fair_ comparison with the proposed Seq2Seq framework, we utilize
a Multi-label classifier based on dense networks (FC-Mult-Lbl in table 1) to
model document features condition on multiple dependent labels. Compared to
Multi-Class classifier (FC-Mult-Cls) that holds independent assumption of
$X\mapsto Y$ mapping tasks, Multi-label classifier sees the tasks as mutually
inclusive, as such one document can correspond to multiple labels (e.g. tags,
keywords). Compared to Seq2Seq that learn to predict a set of sequences based
on tree-based mutually dependent structure $Y\in\Sigma^{*}$ , the Multi-label
classifier estimates the probability of multiple classes for one source
instance independently (_sigmoid_ , instead of normalized _softmax_
probability outputs).

#### Pretrained Model: BERT

BERT Devlin et al. (2019) is the most recent language representation model
surprisingly performed well in diverse language understanding benchmark -
indicating the network has a capacity to capture structural information from
natural language data Xu et al. (2019); Jawahar et al. (2019); Reimers and
Gurevych (2019). Unlike its deep architecture counterparts commonly composed
of recurrent network to hold the main assumption of sequential data (e.g.
Seq2Seq), BERT network is mainly composed of dense connections - referred to
as “self-attention” network. We utilize a pretrained BERT as static universal
sentence encoder – i.e. to transform documents into vectors unsupervisedly
without fine-tuning, assuming the global and sub semantic structure is
_unobserved_. We use the latest implementation of BERT for sentence embedding
(S-BERT) Reimers and Gurevych (2019), which has shown a better generalization
performance than vanilla BERT Devlin et al. (2019).

##  Appendix B Seq2Seq Networks

###  B.1 Attention Network

We use Bahdanau’s concat attention scoring function Bahdanau et al. (2014),
illustrated in fig. 8, to calculate attention weights $\alpha\in R^{T_{x}}$ of
encoder state representation conditioned by decoder output state
representation.

Figure 8: Attention network Bahdanau et al. (2014).

###  B.2 Teacher forcing

A common strategy to train a recurrent-based Seq2Seq model in a generation
task is incorporating teacher forcing, i.e. by exposing the actual or expected
output $Y_{t<T_{y}}$ at the current decoding time step $t<T_{y}$, rather than
the output generated by the network ($s_{t<T_{y}}$). The drawback, however,
during evaluation stage the network only relies on its own prediction from
previous time steps, resulting a performance degradation referred as
“training-evaluation loss mismatch” in Wiseman and Rush (2016). In this study,
we employ curriculum learning Bengio et al. (2015), i.e. approach to sampling
whether to use teacher forcing or not during training stage in the current
sequence prediction problem. Here, the probability of incorporating teacher
forcing (ratio of teacher forcing) is calculated based on inverse sigmoid
function of scheduled sampling (after $xx-$ batch examples seen during
training).

##  Appendix C Evaluation Metrics

#### Clustering Evaluation

A normalized mutual information (NMI) is used to measure whether the
clustering method can recreate the true or exact structure of the observed
data. We use the following metrics to evaluate the structure representation in
a clustering task, in addition to $F_{1}$-score and purity measure.

$NMI(\Omega,\mathbb{C})=\frac{I(\Omega,\mathbb{C})}{0.5(H(\Omega)+H(\mathbb{C}))}$
(1)

Where $I(\Omega,\mathbb{C})$ measures the mutual information between the
formed cluster membership and exact class. And, $H(\Omega),H(\mathbb{C})$
denotes entropy of the formed structure and exact class respectively.

$I(\Omega,\mathbb{C})=\sum_{k}\sum_{j}P(\omega_{k}\cap
c_{j})\log\frac{P(\omega_{k}\cap c_{j})}{P(\omega_{k})P(c_{j})}$ (2)

$H(\mathbb{C})=-\sum_{k}P(\omega_{k})\log P(\omega_{k})$ (3)

$P(\omega_{k}),P(c_{j}),P(\omega_{k}\cap c_{j})$ are the probability of
document feature being in cluster $\omega_{k}$, class $c_{j}$, and both
memberships.

#### Purity

Each cluster is assigned to the class which has the most frequent members in
the cluster, as such the purity of clusters is computed by
$\texttt{purity}(\Omega,\mathbb{C})=\frac{1}{N}\sum_{k}\texttt{max}_{j}|\omega_{k}\cap
c_{j}|$. $\Omega=\\{\omega_{1},\omega_{2},\ldots,\omega_{k}\\}$ denotes a set
of $k$ clusters, while $\mathbb{C}=\\{c_{1},c_{2},\ldots,c_{j}\\}$ is a set of
$j$ actual classes. A perfect clustering has a purity of $1$. Note: the purity
metric disregards the _uniqueness_ of the cluster since it is computed based
only on the majority class and number of members of the majority class.

#### Normalized Mutual Information (NMI)

NMI or a Mutual Information-based metric measures the amount of information
needed to predict the actual class of a cluster, given a knowledge about
documents in that cluster.

$\displaystyle
NMI(\Omega,\mathbb{C})=\frac{I(\Omega,\mathbb{C})}{[H(\Omega)+H(\mathbb{C})]/2}$
$\displaystyle I(\Omega,\mathbb{C})=\sum_{k}\sum_{j}P(\omega_{k}\cap
c_{j})log\frac{P(\omega_{k}\cap c_{j})}{P(\omega_{k})P(c_{j})}$

where $I$ is mutual information between the predicted clusters $\Omega$ and
the actual classes $\mathbb{C}$, which is measured based on the overlapping
document membership between clusters and actual classes. $H$ denotes an
entropy measure $H(P)=-\sum_{x\in X}P(x)log_{2}P(x)$.

#### F1-Score

We measure F1-score (harmonic mean of Precision and Recall) of document
clustering based on the notion of how a pair of documents (points in latent
space) is assigned into clusters Manning et al. (2008). True Positive (TP)
assignment implies that two _similar_ documents are assigned to the same
cluster. True Negative (TN) assignment refers to the assignment of two
_dissimilar_ documents to different clusters. False Positive (FP) assigns two
dissimilar documents into the same cluster, while False Negative (FN) assigns
two similar documents into different clusters. Precision (P) and Recall (R)
are then computed based on $P=TP/(TP+FP)$ and $R=TP/(TP+FN)$; while
$F_{1}$-score is a harmonic mean of both metrics $F_{1}=(2PR)/(P+R)$.

##  Appendix D Data

Sentence |  Keywords |  Topic
---|---|---
|  (SUB) |  (GLO)
“what was so important about the battle of quebec” |  battle of quebec; american revolutionary war |  history
“who were the commanders that died in the battle of quebec in 1759” |  battle of quebec; american revolutionary war |  history
“the sporozoan plasmodium carried from host to host by mosquitoes causes what serious infection” |  malaria; plasmodium parasite |  virus and diseases
“plasmodium is a malaria causing sporozoan which is transmited by mosquito” |  malaria; plasmodium parasite |  virus and diseases

Table 9: Sentence examples in Toy Data set

“diabetes mellitus is medical disorder characterized by varying or persistent
hyperglycemia elevated blood sugar levels, especially after eating. all types
of diabetes mellitus share similar symptoms and complications at advanced
stages. hyperglycemia itself can lead to dehydration and ketoacidosis. longer
term complications include cardiovascular disease doubled risk, chronic renal
failure it is the main cause for dialysis, retinal damage which can lead to
blindness, nerve damage which can lead to erectile dysfunction impotence,
gangrene with risk of amputation of toes, feet, and even legs”
---
keywords: diabetes; diseases and conditions
topic: health

Table 10: Sentence examples in Yahoo! Answer Q&A

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 9: Corpus visualization as graph network (nodes represent documents,
keywords, and topic categorization); $(a-b)$ Toy data; $(c-h)$ 5-T Yahoo! cQA
data.

##  Appendix E Corpus Visualization

Figure 9 shows a corpus visualization as graph network where nodes represent
document sources that are connected by keywords as sub-structure and topic
labels as global structure.

##  Appendix F More Results

###  F.1 KeyGen2Vec as Embedding Models

“Under which condition KeyGen2Vec is better than the other embedding models?”

#### On Scalability Aspect of Model

Compared to unsupervised methods and pretrained models, KeyGen2Vec has shown a
consistent good performance on capturing semantic structure in data,
outperforming the other models, regardless the number of training examples.
_Note_ : This specifically holds on scale-free network (Figure 9 and Figure
LABEL:fig:net_define), i.e. the growth of the network is independent with the
underlying structure of the network. While word-level embedding and Seq2Seq
for autoencoding suffer on small set of training examples (Toy data, where the
evaluation result is shown in Table 6), Seq2Seq for keyword generation (STS-
KG) shows an ability to learn useful features even in small data set,
indicating the model can be utilized as feature extractor for both small and
large scale data.

#### On Semantic Structure Inferred in Data

While sub-structure can promote the learning of _latent_ global structure
inferred in data, shown in our empirical results, an overlapping sub semantic
structures exemplified by the two real world data sets in the current study,
as shown in Figure 9(c)-9(h), may potentially introduce noises in the
learning. This type of loss is also shown on the extracted features from both
models that learn representation from sub semantic structures: Seq2Seq for
keyword generation STS-KG (Figure 10(a)) and multi-label classifier (Figure
10(b)). The overlapping points in different colors (Figure 10) represent
documents in three category topic labels: ’dining out’, ’travel’, ’local
business’ that shares common set of keywords (‘‘London’’, ‘‘UK’’).

(a)

(b)

Figure 10: Loss in approximating global structure is mainly due to overlapping
sub-structures; (a) STS-KG; (b) Multi-Label Classifier

(a)

(b)

(c)

(d)

Figure 11: (a) Average silhouette score on three data sets; (b) Multi-Class
Classifier on 5-T Yahoo!; (c) STS-KG on 5-T Yahoo!; (d) Multi-Label Classifier
on 5-T Yahoo!;

#### KeyGen2Vec vs. Seq2Seq Autoencoding

The clustering evaluation results presented in table 6-8 notably shows that
Seq2Seq autoencoder (STS-AE) cannot adequately capture the implicit global
semantic structure of data. Similar to the small version of word embedding-
based model in the current work (Avg-w2v-tr-sm table 1), we argue that an
autoencoder heavily relies on the “goodness” in the data, indicating the model
may be useful if meaningful sentences that promote a content-based semantic
structure learning are available abundantly as training examples. If such case
is not available, the performance of feature extractor may be improved by
inferring bias to the model architecture or training objective. For instance,
instead of an autoencoding task, the network can be trained to predict the
semantic structure of the document source, exemplified by Seq2Seq framework
for keyword generation in the current study.

#### KeyGen2Vec vs. Multi-Label Classifier

Figure 11 shows the comparison of cluster separability of the three model
based on silhouette measure: (1) Multi-class classifier as an upper bound
model; (2) Seq2Seq for keyword generator; and (3) Multi-label classifier. The
vertical dashed lines in fig 11(b)-11(d) represents the average score of all
points in the resulting clusters, while the size of the bar plot represents
the size of cluster. Silhouette score was calculated based on the average
distances for all points in the same cluster and the average distances for
points in the closest cluster. The score values range from $[-1,1]$ – where
score$=-1$ indicates sample is assigned to the wrong cluster, average score of
$0$ indicates that the inter-cluster distance is small, and average score of
$1$ indicates the inter-cluster distance is large enough to form separable
clusters. Intuitively, the silhouette score is expected to close to $1$ for a
good quality of clusters. A good quality of clusters infers a good quality of
extracted features.

Overall, without having the exact classes to evaluate the quality of document
clustering, Both features from STS-KG and Multi-Label classifier can
approximate the cluster separability of an upper bound model, indicating both
models has a capacity to extract “good” features according to the current
definition of global semantic structure in the current study. Although the
clustered features based on Seq2Seq (STS-KG) has a slightly higher score than
multi -label classifier clustered features, the actual quality of extracted
features by Seq2Seq outperformed multi-label classifier features when exact
classes are projected on the resulting clusters.

#### KeyGen2Vec vs. LDA Topic Model

While KeyGen2Vec has outperformed all models in the current study, this
performance comes at cost of providing sub semantic structure information. As
a comparison, we discuss the performance trade-offs of LDA Topic Model as an
alternative model for unsupervised approach, in addition to the aforementioned
unsupervised and pretrained models (sec. LABEL:sec:unsup).

Figure 12: Deciding hypothesis space (number of topics) Figure 13: LDA
Performance on different preprocessing steps

While LDA topic model in the current study (results in table 6-8) does not
show an impressive performance w.r.t. the semantic quality of the learnt
clusters, the model promotes a consistent quality as document feature
extractor among three data sets, similar to a trainable word embedding-based
model (Avg-w2v-tr). Nevertheless, deciding a hypothesis space of topic model
is non-trivial. Too small $K$ number of topics results in a very broaden
topics (words corresponds to the topic semantic definition is too general).
While, too large $K$ results in a repetitive topics – i.e. different topics
contain an overlapping set of words. While the hypothesis space of topic model
can be evaluated by coherence measure during training, the measure does not
necesarily correlate to the actual quality of learnt features or an
interpretability aspect of the resulting features in topic space , illustrated
in fig. 12. We refer the “interpretability” here as a degree to which the two
different clusters are adequately far or _separable_.

The performance of topic model in capturing document semantic structure also
highly depends on heavy preprocessing steps. Figure 13 shows how the quality
of extracted features varies, depending on (a) whether the stopword and noisy
words have been removed (e.g. removing question words and non meaningful
abbreviation or short character in dat_sw); (b) whether data is lemmatized
(dat_lem); (c) document length (fuldat_lem); or (d) a balance distribution of
topic categorization labels (fuldat_lem_ba). If such knowledge (e.g. POS tag
linguistic structure in lemmatization step) is not available in particular
data or language, the performance can be expected to degrade.


# On the noncommutative fields method in the three-dimensional Yang-Mills
theory

J. R. Nascimento, A. Yu. Petrov, E. O. Silva  Departamento de Física,
Universidade Federal da Paraíba
Caixa Postal 5008, 58051-970, João Pessoa, Paraíba, Brazil
<EMAIL_ADDRESS>
###### Abstract

We apply the noncommutative fields method to the three-dimensional non-Abelian
gauge theory. We find that, first, implementing the noncommutativity between
the canonical momenta implies in generation of the non-Abelian Chern-Simons
term, second, if one introduces the noncommutativity between the field
operators, the higher derivative terms would arise.

The noncommutativity is treated now as a fundamental quantum property of the
space-time geometry. Beside of the known scheme of introducing the
noncommutativity via the Moyal product SW , an alternative one was recently
developed, that is, so-called noncommutative fields method, in which, instead
of the spacetime coordinates, fields themselves are noncommutative, thus, the
canonical commutation relations turn out to be deformed Gamb0 . This method
turned out to be a new method of generating the Lorentz-breaking correction
after it was shown that the known Lorentz-breaking term initially introduced
by Jackiw and Kostelecky JK  naturally emerges within this formalism Gamb1 .
Further, the non-Abelian analog of this term was generated via the
noncommutative fields method Gamb2 , and in our paper ourgra , this method was
applied to generate the Lorentz symmetry breaking in the linearized gravity.

At the same time, the situation in three-dimensional space-time is different.
Indeed, we have shown in NPR  that application of the noncommutative field
method to three-dimensional electrodynamics, instead of the Lorentz-breaking
terms generates a gauge invariant mass term, that is, the Chern-Simons term,
with the mass turns out to be proportional to the noncommutativity parameter
NPR . We would like to notice that unlike of common perturbative approach (see
f.e. Redlich ), the essence of the noncommutative fields method consists in
possibility to generate new terms without coupling to extra matter fields. The
very natural development of this study would consist in generalization of the
noncommutative fields method for the non-Abelian case, where it is natural to
expect that not only quadratic term but also the interaction term for the
gauge field will arise. Different aspects of the Chern-Simons term, both in
Abelian and non-Abelian cases, such as non-trivial topological nature of this
term DJT  and quantization of the Chern-Simons coefficient quCS  were studied.
In other worlds, it is natural to expect that in this case, the three-
dimensional non-Abelian Chern-Simons term

$\displaystyle L_{CS}=\frac{1}{2}m\epsilon^{\mu\nu\lambda}{\rm
tr}(A_{\mu}\partial_{\nu}A_{\lambda}+\frac{2}{3}gA_{\mu}A_{\nu}A_{\lambda})$
(1)

will be generated. From the other side, we are planning to generalize the
noncommutative field method by introducing of a more general deformation of
the canonical algebra which in principle could imply in arising of the
Lorentz-breaking terms. These problems are considered in the paper.

Let us start our study of the three-dimensional Yang-Mills theory, whose
action is

$\displaystyle S=-\frac{1}{4}\int d^{3}x{\rm tr}F_{mn}F^{mn},$ (2)

with the $F_{mn}=F_{mn}^{a}T^{a}$ is a stress tensor constructed on the base
of the Lie-algebra valued gauge field $A_{m}(x)=A_{m}^{a}(x)T^{a}$ (with ${\rm
tr}(T^{a}T^{b})=\delta^{ab}$, and $[T^{a},T^{b}]=f^{abc}T^{c}$):

$\displaystyle
F_{mn}^{a}=\partial_{m}A_{n}^{a}-\partial_{n}A_{m}^{a}+gf^{abc}A^{b}_{m}A^{c}_{n},$
(3)

so, the Lagrangian, after splitting of the indices into time (zero) and space
ones (denoted by $i,j,k$) looks like

$\displaystyle L=-\frac{1}{4}\int
d^{3}xF_{mn}^{a}F^{mn\,a}=-\frac{1}{4}F^{a}_{ij}F^{a}_{ij}+\frac{1}{2}(\dot{A}^{a}_{i}-\partial_{i}A^{a}_{0}+gf^{abc}A^{b}_{i}A^{c}_{0})^{2}.$
(4)

Let the signature be $diag(-++)$. First, we carry out the canonical
quantization of the theory. The canonical momentum of the theory is

$\displaystyle p^{a}_{m}=\frac{\partial
L}{\partial\dot{A}^{a\,m}}=F^{a}_{0m}.$ (5)

It is clear that $p^{a}_{0}=0$, so, we find the primary constraint
$\Phi^{(1)a}=p^{a}_{0}$. The velocities can be expressed as

$\displaystyle\dot{A}^{a}_{i}=p_{i}^{a}-gf^{abc}A^{b}_{0}A^{c}_{i}+\partial_{i}A_{0}^{a}.$
(6)

Thus, the Hamiltonian is

$\displaystyle
H=p^{a}_{i}\dot{A}^{a}_{i}-L=\frac{1}{2}p^{a}_{i}p^{a}_{i}+\frac{1}{4}F^{a}_{ij}F^{a}_{ij}+p_{i}^{a}(-gf^{abc}A^{b}_{0}A^{c}_{i}+\partial_{i}A_{0}^{a}).$
(7)

The secondary constraint looks like

$\displaystyle\Phi^{(2)b}\equiv\Delta^{a}=\\{p^{a}_{0},H\\}=-\frac{\partial
H}{\partial
A_{0}^{a}}=-(\partial_{i}p_{i}^{a}+gf^{abc}A^{b}_{i}p_{i}^{c})\equiv-{\cal
D}^{ab}_{i}p^{b}_{i}.$ (8)

This constraint evidently generates the gauge transformations:

$\displaystyle\delta A_{i}^{a}$ $\displaystyle=$
$\displaystyle\\{A_{i}^{a},\int
d^{2}\vec{x}\xi^{b}(\vec{x})\Delta^{b}(\vec{x})\\}=\partial_{i}\xi^{a}(\vec{x})+gf^{abc}A^{b}_{i}(\vec{x})\xi^{c}(\vec{x})\,(\equiv{\cal
D}^{ac}_{i}\xi^{c}(\vec{x}));$ $\displaystyle\delta p_{i}^{a}$
$\displaystyle=$ $\displaystyle\\{p_{i}^{a},\int
d^{2}\vec{x}\xi^{b}(\vec{x})\Delta^{b}(\vec{x})\\}=-gf^{abc}\xi^{b}(\vec{x})p^{c}_{i}(\vec{x}),$
(9)

which evidently reproduces the known gauge transformation for the connection
and stress tensor. Here the ${\cal D}^{ac}$ is a gauge covariant derivative.

It is easy to check that the primary and secondary constraints mutually
commute, $\\{\Phi^{(1)a},\Phi^{(2)b}\\}=0$. Further, one can find that
$\\{\Phi^{(2)b},H\\}=0$, thus, no new constraints arise (see also Park ; Wo
for discussion of the canonical structure of the theories with the Chern-
Simons term).

The canonical quantization of the theory can be carried out in a standard way,
that is, we define the canonical variables $A^{a}_{i}$ and $p^{a}_{i}$ to be
operators with the commutation relation
$[A^{a}_{i}(\vec{x}),p^{b}_{j}(\vec{y})]=i\delta_{ij}\delta^{ab}\delta(\vec{x}-\vec{y})$,
with all other commutators of the canonical variables be zero.

Now, let us implement the noncommutative fields method. To do it, we deform
the canonical commutation relations to be

$\displaystyle[A^{a}_{i}(\vec{x}),p^{b}_{j}(\vec{y})]=i\delta_{ij}\delta^{ab}\delta(\vec{x}-\vec{y});$
$\displaystyle[p^{a}_{i}(\vec{x}),p^{b}_{j}(\vec{y})]=i\theta_{ij}\delta^{ab}\delta(\vec{x}-\vec{y});$
$\displaystyle[A^{a}_{i}(\vec{x}),A^{b}_{j}(\vec{y})]=0.$ (10)

Our aim is to deform the secondary constraint $\Delta^{b}$ in a manner
preserving the gauge transformations (On the noncommutative fields method in
the three-dimensional Yang-Mills theory). It is easy to see that this can be
achieved if we modify the secondary constraint as

$\displaystyle\tilde{\Delta}^{b}=-(\partial_{i}p_{i}^{b}+gf^{bcd}A^{c}_{i}p_{i}^{d})+\theta_{ij}(\partial_{i}A^{b}_{j}+\frac{1}{2}gf^{bcd}A^{c}_{i}A^{d}_{j}).$
(11)

This modification of the secondary constraint implies in the modification of
the Hamiltonian which acquires the form

$\displaystyle\tilde{H}=\frac{1}{2}p^{a}_{i}p^{a}_{i}+\frac{1}{4}F^{a}_{ij}F^{a}_{ij}+A_{0}^{b}\theta_{ij}(\partial_{i}A^{b}_{j}+\frac{1}{2}gf^{bcd}A^{c}_{i}A^{d}_{j}).$
(12)

Then, we can introduce the canonical momenta

$\displaystyle\pi_{i}^{a}=p_{i}^{a}-\frac{1}{2}\theta_{ij}A_{j}^{a},$ (13)

and they satisfy the commutation relation $[\pi_{i}^{a},\pi_{j}^{b}]=0$.

The new Lagrangian is

$\displaystyle\tilde{L}=\pi_{i}^{a}\dot{A}_{i}^{a}-\tilde{H}.$ (14)

Substituting the canonical momenta (13) and the modfified Hamiltonian (12) to
this expression, we find that the new Lagrangian can be written as

$\displaystyle\tilde{L}=L+\Delta L\equiv
L-\frac{1}{2}\theta_{ij}\dot{A}_{i}^{a}A_{j}^{a}-A_{0}^{b}\theta_{ij}(\partial_{i}A^{b}_{j}+\frac{1}{2}gf^{bcd}A^{c}_{i}A^{d}_{j}).$
(15)

As a result, we find

$\displaystyle\Delta
L=\theta_{ij}(-\frac{1}{2}\dot{A}_{i}^{a}A_{j}^{a}-A_{0}^{a}\partial_{i}A_{j}^{a}+\frac{1}{2}gf^{bcd}A_{0}^{b}A_{i}^{c}A^{d}_{j}).$
(16)

After an appropriate symmetrization, introducing
$\theta_{ij}=\epsilon_{0ij}\theta$, we find

$\displaystyle\Delta
L=\frac{1}{2}\theta\epsilon^{\mu\nu\lambda}(A_{\mu}^{a}\partial_{\nu}A_{\lambda}^{a}+\frac{1}{3}gf^{abc}A_{\mu}^{a}A_{\nu}^{b}A_{\lambda}^{c})=\frac{1}{2}\theta\epsilon^{\mu\nu\lambda}{\rm
tr}(A_{\mu}\partial_{\nu}A_{\lambda}+\frac{2}{3}gA_{\mu}A_{\nu}A_{\lambda}),$
(17)

which reproduces the structure of the well known non-Abelian Chern-Simons
term, with the mass is proportional to the noncommutativity parameter, just as
in NPR .

We can try to implement a more general deformation of the canonical algebra,
that is,

$\displaystyle[A^{a}_{i}(\vec{x}),p^{b}_{j}(\vec{y})]=i\delta_{ij}\delta^{ab}\delta(\vec{x}-\vec{y});$
$\displaystyle[p^{a}_{i}(\vec{x}),p^{b}_{j}(\vec{y})]=i\theta_{ij}\delta^{ab}\delta(\vec{x}-\vec{y});$
$\displaystyle[A^{a}_{i}(\vec{x}),A^{b}_{j}(\vec{y})]=i\tilde{\theta}_{ij}\delta^{ab}\delta(\vec{x}-\vec{y}).$
(18)

Let us impose again a requirement that the gauge transformations should have
the form (On the noncommutative fields method in the three-dimensional Yang-
Mills theory). First of all, since $\theta_{ij}$ and $\tilde{\theta}_{ij}$ are
constants, we suggest from the beginning that
$\theta_{ij}=\theta\epsilon_{ij}$,
$\tilde{\theta}_{ij}=\tilde{\theta}\epsilon_{ij}$.

To do it, let us suggest the following form of the modified secondary
constraint which is the most general expression of no higher than second order
in canonical variables:

$\displaystyle\Phi^{(2)b}$ $\displaystyle=$
$\displaystyle-\partial_{i}p_{i}^{b}+k_{1}gf^{bcd}A^{c}_{i}p_{i}^{d}+k_{2}\epsilon_{ij}\partial_{i}A^{b}_{j}+k_{3}\epsilon_{ij}\partial_{i}p^{b}_{j}+k_{4}\epsilon_{ij}gf^{bcd}A^{c}_{i}p^{d}_{j}+$
(19) $\displaystyle+$ $\displaystyle
k_{5}gf^{bcd}\epsilon_{ij}p^{c}_{i}p^{d}_{j}+k_{6}gf^{bcd}\epsilon_{ij}A^{c}_{i}A^{d}_{j}.$

Here the coefficients $k_{1}\ldots k_{6}$ depend on $\theta,\tilde{\theta}$.

The corresponding variations of the fields look like

$\displaystyle\delta A^{a}_{n}$ $\displaystyle=$
$\displaystyle\\{A^{a}_{n},\Phi^{(2)b}\\}\xi^{b}=\partial_{n}\xi^{a}-k_{1}gf^{abc}\xi^{b}(\tilde{\theta}\epsilon_{ni}p^{c}_{i}-A^{c}_{n})-k_{2}\tilde{\theta}\partial_{n}\xi^{a}+k_{3}\epsilon_{ni}\partial_{i}\xi^{a}+$
$\displaystyle+$ $\displaystyle
k_{4}gf^{abc}\xi^{b}(\tilde{\theta}p^{c}_{n}-\epsilon_{ni}A^{c}_{i})-2k_{5}g\epsilon_{ni}f^{abc}\xi^{b}p^{c}_{i}+2k_{6}gf^{abc}\tilde{\theta}\xi^{b}A^{c}_{n};$
$\displaystyle\delta p^{a}_{n}$ $\displaystyle=$
$\displaystyle\\{p^{a}_{n},\Phi^{(2)b}\\}\xi^{b}=\theta\epsilon_{ni}\partial_{i}\xi^{a}+k_{1}gf^{abc}\xi^{b}p^{c}_{n}+k_{1}\theta
gf^{abc}\epsilon_{ni}\xi^{b}A^{c}_{i}-k_{2}\epsilon_{ni}\partial_{i}\xi^{a}-k_{3}\theta\partial_{n}\xi^{a}+$
(20) $\displaystyle+$ $\displaystyle
k_{4}\epsilon_{ni}gf^{abc}\xi^{b}p^{c}_{i}+k_{4}\theta
gf^{abc}A^{c}_{n}\xi^{b}+2k_{5}gf^{abc}\theta\xi^{b}p^{c}_{n}-2k_{6}gf^{abc}\epsilon_{ni}\xi^{b}A^{c}_{i}.$

We want these transformations to reproduce (On the noncommutative fields
method in the three-dimensional Yang-Mills theory). For the variation of
$A^{a}_{n}$ this requirement yields $k_{3}=0,k_{4}=0$, so, we will not
consider these terms in the equation for $\delta p^{a}_{i}$. Also, we find

$\displaystyle
k_{2}\tilde{\theta}=0;\quad\,k_{1}+2k_{6}\tilde{\theta}=-1,\quad\,k_{1}\tilde{\theta}+2k_{5}=0.$
(21)

For the second equation, after substituting $k_{3}=k_{4}=0$, we get

$\displaystyle
k_{2}=\theta,\quad\,k_{1}+2k_{5}\theta=-1,\quad\,k_{1}\theta-2k_{6}=0.$ (22)

Comparing these equations, we find that the variations of the fields (On the
noncommutative fields method in the three-dimensional Yang-Mills theory)
reproduce the form of variations under the gauge transformations if and only
if $\theta\tilde{\theta}=0$. Hence, we must have either $\tilde{\theta}=0$,
which is exactly the case studied above, or $\theta=0$. Thus, we conclude that
we cannot impose noncommutativity both in field and momentum sectors in a
manner compatible with the gauge symmetry.

It remains only to finish the study in the case when $\theta=0$. In this case,
the modified constraint is

$\displaystyle\Phi^{(2)b}$ $\displaystyle=$
$\displaystyle-\partial_{i}p_{i}^{b}-gf^{bcd}A^{c}_{i}p_{i}^{d}+\frac{\tilde{\theta}}{2}gf^{bcd}\epsilon_{ij}p^{c}_{i}p^{d}_{j},$
(23)

and the modified Hamiltonian is

$\displaystyle\tilde{H}=\frac{1}{2}p^{a}_{i}p^{a}_{i}+\frac{1}{4}F^{a}_{ij}F^{a}_{ij}+A_{0}^{b}[-\partial_{i}p_{i}^{b}-gf^{bcd}A^{c}_{i}p_{i}^{d}+\frac{\tilde{\theta}}{2}gf^{bcd}\epsilon_{ij}p^{c}_{i}p^{d}_{j}].$
(24)

Since commutation relations between momenta are not modified in this case, the
momenta $p^{a}_{i}$ continue to be canonical ones, whereas the coordinates –
do not more. The correct ”new” canonical coordinates, whose commutators are
equal to zero, are

$\displaystyle\tilde{A}^{a}_{i}=A^{a}_{i}-\frac{1}{2}\tilde{\theta}\epsilon_{ij}p^{a}_{j},$
(25)

with the ”old” velocities are related with momenta as

$\displaystyle\dot{A}^{b}_{i}=\frac{\partial\tilde{H}}{\partial
p^{b}_{i}}=p^{b}_{i}+\partial_{i}A^{b}_{0}+gf^{abc}A^{a}_{0}A^{c}_{i}+g\tilde{\theta}f^{abc}A^{a}_{0}\epsilon_{ij}p^{c}_{j},$
(26)

which for $\tilde{\theta}=0$ evidently reduces to the common expression (6).
Unfortunately, this equation, whose equivalent form is

$\displaystyle
p^{c}_{j}(\delta^{bc}\delta_{ij}+g\tilde{\theta}f^{abc}A^{a}_{0}\epsilon_{ij})=\dot{A}^{b}_{i}-\partial_{i}A^{b}_{0}+gf^{bac}A^{a}_{0}A^{c}_{i}\quad\,(=F^{b}_{0i}),$
(27)

cannot be solved exactly, we can use only iterative approach (however, we
would like to point out that this problem does not arises in the Abelian case
where one finds $p^{b}_{i}=F^{b}_{0i}$). As a zeroth approximation (which,
however, is sufficient to find the corrections in the effective Lagrangian up
to the first order in $\tilde{\theta}$), we can use the $\tilde{\theta}=0$
expression for the canonical momentum $p^{a}_{i}$ (5), thus, the Lagrangian
$\tilde{L}=p^{a}_{i}\dot{\tilde{A}}^{a}_{i}-\tilde{H}$ acquires a correction
$\Delta L$ generated by modifications both of the Hamiltonian and
$\dot{A}^{a}_{i}$. This correction, being expressed in terms of the canonical
momenta, looks like:

$\displaystyle\Delta
L=-\frac{1}{2}\tilde{\theta}\epsilon_{ij}p^{a}_{i}\dot{p}^{a}_{j}-\frac{1}{2}gf^{bcd}A^{b}_{0}\tilde{\theta}\epsilon_{ij}p^{c}_{i}p^{d}_{j}.$
(28)

This expresion is exact, without any approximations. After elimination of
momenta, where we must employ the approximate expressions for $p^{a}_{i}$ in
terms of velocities, we find that

$\displaystyle\Delta
L=-\frac{1}{2}\tilde{\theta}\epsilon_{ij}F^{a}_{0i}\dot{F}^{a}_{0j}-\frac{1}{2}gf^{bcd}A^{b}_{0}\tilde{\theta}\epsilon_{ij}F^{c}_{0i}F^{d}_{0j}+O(\tilde{\theta}^{2}).$
(29)

Thus, one can see that, as a result, the modified Lagrangian in the case of
noncommuting field operators involves higher derivatives (since $F^{a}_{0i}$
contain first temporal derivative). The similar conclusion, that is,
generation of higher derivatives in the case of noncommuting fields (which can
be treated as UV limit of the theory, see discussion of scales in the
noncommutative fields method in Gamb0 ), was obtained in Gamb . Also, we note
that, as this correction to the Lagrangian has quite ugly form, we can
conclude that in this case, unlike of the case of noncommuting momenta, we
meet an explicit Lorentz symmetry breaking.

Let us discuss the results. We studied the generalized version of the
noncommutative field method, in which, differently from the most popular
version Gamb1 ; Gamb2 ; ourgra  not only the commutation relations between
canonical momenta are deformed but also the commutation relations between
canonical field coordinates. The most important conclusions are the following
ones. First, one cannot deform these two canonical commutation relations
simultaneously in a manner compatible with the gauge symmetry. This fact can
be treated as a need to choose between study of the low-energy behaviour
(which corresponds to deformation of commutation relation between canonical
momenta) and study of the high-energy behaviour (which corresponds to
deformation of commutation relation between canonical fields) with no
possibility to consider two limits at the same time. Second, in the low-energy
limit the complete, non-linearized Chern-Simons term is generated, which is a
natural non-Abelian generalization of the result obtained in NPR  where the
quadratic Chern-Simons term was generated for the electrodynamics, with no
Lorentz symmetry breaking terms arises in this case, and both the mass term
and cubic interaction term with a correct coefficient are generated. However,
the new term arisen in the high-energy limit turns out to break the Lorentz
symmetry explicitly, and, moreover, it involves higher derivatives as it was
predicted in Gamb . The natural treating of this result is that the breaking
of the Lorentz symmetry at high energies can be related to the GZK effect and
many other studies predicting Lorentz symmetry breaking namely for high energy
scales (see f.e. Mag ).

Acknowledgments. The work by A. Yu. P. has been supported by CNPq-FAPESQ DCR
program, CNPq project No. 350400/2005-9.

## References

  * (1) N. Seiberg, E. Witten, JHEP 09, 032 (1999), hep-th/9908142.
  * (2) J. Carmona, J. Cortes, J. Gamboa and F. Mendez, Phys. Lett. B565, 222 (2003), hep-th/0207158; JHEP 03, 058 (2003), hep-th/0301248.
  * (3) R. Jackiw, V. A. Kostelecky, Phys. Rev. Lett. 82, 3572 (1999), hep-ph/9903158.
  * (4) J. Gamboa, J. Lopez-Sarrion, Phys. Rev. D71, 067702 (2005), hep-th/0501034.
  * (5) H. Falomir, J. Gamboa, J. Lopez-Sarrion, F. Mendez, A. J. da Silva, Phys. Lett. B632, 740 (2005), hep-th/0504032.
  * (6) A. F. Ferrari, M. Gomes, J. R. Nascimento, E. Passos, A. Yu. Petrov, A. J. da Silva, Phys. Lett. B652, 174 (2007), hep-th/0609222.
  * (7) J. R. Nascimento, A. Yu. Petrov, R. F. Ribeiro, Europhys. Lett. 77, 51001 (2007), hep-th/0601077.
  * (8) A. N. Redlich, Phys. Rev. D29, 2366 (1984).
  * (9) S. Deser, R. Jackiw, S. Templeton, Ann. Phys. 140, 372 (1982); Phys. Rev. Lett. 48, 975 (1982).
  * (10) A. P. Polychronakos, Phys. Lett. B241, 37 (1990); L.-S. Chen, G. Dunne, K. Haller, E. Lim-Lombridas, Phys. Lett. B348, 468 (1995).
  * (11) M.-I. Park, Nucl. Phys. B544, 377 (1999), hep-th/9811033.
  * (12) L. S. Grigorio, M. S. Guimaraes, S. Wotzasek, ”Induced deformation of the canonical structure and UV/IR duality in (1+1)D”, arXiv: 0802.1193; J. Gamboa, L. S. Grigorio, M. S. Guimaraes, F. Mendes, S. Wotzasek, ”Radiative processes as a condensation phenomenon and the physical meaning of deformed canonical structures”, arXiv: 0805.0626.
  * (13) J. Gamboa, J. Lopez-Sarrion, A. Polychronakos, Phys. Lett. B634, 471 (2006), hep-ph/0510113.
  * (14) J. Magueijo, L. Smolin, Class. Quant. Grav. 21, 1725 (2003), gr-qc/0305055.