# Gender Prediction from Tweets: Improving Neural Representations with Hand-
Crafted Features

Erhan Sezerer
Dept. of Computer Engineering
İzmir Institute of Technology
İzmir, Turkey
<EMAIL_ADDRESS>
&Ozan Polatbilek
Dept. of Computer Engineering
İzmir, Turkey
<EMAIL_ADDRESS>
Selma Tekir
Dept. of Computer Engineering
İzmir, Turkey
<EMAIL_ADDRESS>

###### Abstract

Author profiling is the characterization of an author through some key
attributes such as gender, age, and language. In this paper, a RNN model with
Attention (RNNwA) is proposed to predict the gender of a twitter user using
their tweets. Both word level and tweet level attentions are utilized to learn
’where to look’. This model111https://github.com/Darg-Iztech/gender-
prediction-from-tweets is improved by concatenating LSA-reduced n-gram
features with the learned neural representation of a user. Both models are
tested on three languages: English, Spanish, Arabic. The improved version of
the proposed model (RNNwA + n-gram) achieves state-of-the-art performance on
English and has competitive results on Spanish and Arabic.

##  1 Introduction

Author profiling is the characterization of an author through some key
attributes such as gender, age, and language. It’s an indispensable task
especially in security, forensics, and marketing. Recently, social media has
become a great data source for the potential learning approaches. Furthermore,
gender prediction has been a popular profiling task.

The traditional approach to gender prediction problem is extracting a useful
set of hand-crafted features and then feeding them into a standard
classification algorithm. In their study, handcraft-2 work with the style-
based features of message length, stop word usage, frequency of smiley etc.
and use different classifiers such as k-nearest neighbor, naive bayes,
covering rules, and backpropagation to predict gender on chat messages.
Similarly, email-hancraft select some hand-crafted features and feed them into
various classifiers.

Most of the work on gender prediction rely on n-gram features (miller-ngram).
daneshvar2018 give Latent Semantic Analysis (LSA)-reduced forms of word and
character n-grams into Support Vector Machine (SVM) and achieve state-of-the-
art performance. Apart from exploiting n-gram frequencies, there are studies
(language-independent), (Language-Independent-2013), (bleaching-text) to
extract cross-lingual features to determine gender from tweets. Some other
work (language-independent), (metadata-used) exploit user metadata besides
using just tweets.

Recently, neural network-based models have been proposed to solve this
problem. Rather than explicitly extracting features, the aim is to develop an
architecture that implicitly learns. In author profiling, both style and
content-based features were proved useful (argamon2009) and neural networks
are able to capture both syntactic and semantic regularities. In general,
syntactic information is drawn from the local context. On the other hand,
semantic information is often captured with larger window sizes. Thus, CNNs
are preferred to obtain style-based features while RNNs are the methods of
choice for addressing content-based features (goldberg2017). In literature,
CNN (selfcite) or RNN (takahashi2O18), (Kodiyan), (selfcite_SIU) is used on
this task. takahashi2O18 obtain state-of-the-art performance among neural
methods by proposing a model architecture where they process text through RNN
with GRU cells. Also, the presence of an attention layer is shown to boost the
performance of neural methods (takahashi2O18), (selfcite).

In this work, we propose a model that relies on RNN with attention mechanism
(RNNwA). A bidirectional RNN with attention mechanism both on word level and
tweet level is trained with word embeddings. The final representation of the
user is fed to a fully connected layer for prediction. Since combining some
hand-crafted features with a learned linear layer has shown to perform well in
complex tasks like Semantic Role Labeling (SRL) (collobert2008), an improved
version of the model (RNNwA + n-gram) is also tested with hand-crafted
features. In the improved version, LSA-reduced n-gram features are
concatenated with the neural representation of the user. Then the result is
fed into a fully-connected layer to make prediction. Models are tested in
three languages; English, Spanish, and Arabic, and the improved version
achieves state-of-the-art accuracy on English, and competitive results on
Spanish and Arabic corpus.

There are many datasets created for this task (2018overview), (selfcite_ACL).
In this work, we have used the dataset and benchmarks provided by the PAN 2018
shared task on author profiling (2018overview). As the dataset contains a
constant number of $100$ tweets per user, accuracy tests are performed both on
user and tweet level (tweet-level predictions are made by removing the user-
level attention). Tweet-level accuracy tests show interesting results during
hyperparameter optimization. When the tweet-level predictions are averaged to
produce user-level predictions, it is seen that the hyperparameters that gave
the best results in terms of tweet-level accuracy, performs worse in user-
level accuracy. The better user-level models, with different hyperparameters,
that gave the highest user-level accuracy are observed to slightly overfit on
tweet-level. It leads us to believe that the overfitting in the tweet-level
predictions in best user-level models acts similar to an attention mechanism
by over-emphasizing some distinctive tweets and ignoring the rest.

##  2 Model architecture

In author profiling, both style-based and content-based features must be
addressed (argamon2009). An appropriate baseline for this task is a CNN-based
model that is able to capture style-based information (selfcite). The proposed
RNN-based model relies on extracting content-based features. In addition, in
order to improve its accuracy, the proposed model is combined with some hand-
crafted features. For all of the models, Adam optimizer (adam) is used with
cross-entropy loss along with the L2 regularization to prevent from
overfitting.

Figure 1: Proposed model.

###  2.1 Baseline CNN model

CNN model (denoted CNNwA on results) is based on selfcite where each character
in the tweet is represented with a character embedding of size $25$, which is
trained along the neural network. All characters are lower-cased. Non-
alphabetical characters such as punctuation are kept with a view to capturing
some information on the profile of the user since they are heavily used in
twitter as emoticons.

Filters of size $3\times 3$, $6\times 6$ and $9\times 9$ are used for each
language, and the number of filters is determined by performing grid search on
validation set. Among the tested range ($50$-$125$ with intervals of $25$),
the number of filters that gives the best accuracy is $100$ (per each filter),
for all languages.

Figure 2: Tweet-level Attention Layer in Detail.

###  2.2 RNN Model

Since the dataset is not big enough to train word embeddings, Glove word
embeddings (glove) of size $200$ are used in the proposed RNN Model (denoted
RNNwA on results) due to their success at various NLP tasks and their multi-
linguality: They encompass all the languages in the test set. In addition, the
Glove embeddings are also trained on Twitter data which make them reflect the
nature of the dataset better than other alternatives.

A bidirectional RNN with GRU (GRU) cells are used in this model where the
number of cells is a hyperparameter. Among the tested range ($50$-$150$ with
intervals of $25$), best accuracy on validation set is obtained by $150$ cells
in English and $100$ cells in Spanish and Arabic. An attention mechanism is
used on word-level in addition to tweet-level to capture the important parts
of each tweet as shown in Figure 1.

A feature vector for each tweet is created by feeding tweets to RNN
separately. In order to discriminate tweets with respect to their information
carrying capacity on its author’s gender, Bahdanau attention mechanism
(attention) is used to combine the tweets rather than concatenating them
before feeding to the network or averaging their predictions later. Figure 2
shows the tweet-level attention layer in detail which is calculated by the
following formulas:

$\begin{split}A_{i}&=tanh(\boldsymbol{W_{\alpha}}t_{i}+b)\\\
v_{i}&=\frac{exp(A_{i}w_{i})}{\sum_{j}{exp(A_{j}w_{j})}}\\\
o_{i}&=v_{i}t_{i}\\\ K&=\sum_{i}{o_{i}}\\\ \end{split}$

where $\boldsymbol{W_{\alpha}}$ is a learnable weight matrix that is used to
multiply each output of the RNN, $t_{i}$ is the feature vector of $i$th tweet,
$b$ is a learnable bias vector, $w_{i}$ is a learnable attention weight,
$A_{i}$ is the attention context vector, $v_{i}$ is the attention value for
$i$th tweet, $o_{i}$ is attention output vector for the corresponding tweet,
$K$ is the output vector for user. Matrix $\boldsymbol{W_{\alpha}}$ and
vectors $w_{i}$ and $b$ are learned parameters.

Attention layer outputs a single feature vector that corresponds to a user,
which is then fed to a fully-connected layer to lower the dimension to the
number of classes.

There are two different attention layers on the model. One is a word level
attention where it amplifies the signal coming from important words, the other
one is on tweet level where it combines the signals coming from each tweet and
creates the final representation of a user.

###  2.3 RNN with N-gram Model

For this model (denoted RNNwA + n-gram on results), n-gram features are
collected with the same method described in daneshvar2018. At the beginning,
word level and character level n-gram features are obtained and concatenated.
Then they are normalized with tf-idf transformation. For reducing the number
of features and sparsity in n-gram vectors, tuples that have frequency less
than $2$ are ignored. For character level n-gram $N$ is selected as $3,4$, and
$5$ and for word level n-gram, $N$ is $1,2$ for Spanish and Arabic; $1,2,3$
for English. The dimension of the vector is reduced by LSA to $300$. Then the
vector is concatenated with neural representation which is produced right
after tweet level attention in RNNwA model. The resultant representation is
fed to a fully- connected layer that produces predictions.

###  2.4 Dataset

Models are tested on the PAN 2018 author profiling dataset (2018overview),
which provides tweets in three languages: English, Spanish and Arabic with
training/test datasets of sizes ($3000$ users, $1900$ users), ($3000$ users,
$2200$ users), and ($1500$ users, $1000$ users) respectively, where each user
has $100$ tweets. Each training set is further partitioned randomly into
training and validation sets with the ratio ($0.8$, $0.2$) respectively for
hyper-parameter optimization.

##  3 Results

In order to measure the effectiveness of the attention mechanism, in addition
to the CNN baseline model (CNNwA) and RNNwA, two new models (denoted as CNN
and RNN) are created by removing the tweet level attention layer (word level
attention stays the same) and generating a prediction for each tweet then just
simply taking an average to give a user level prediction. Tweet level
accuracies for these models are shown in Table 1.

Table 1: Tweet Level Accuracy of the CNN and RNN Models without Attention. Model | English | Spanish | Arabic
---|---|---|---
RNN | 62.600 | 62.163 | 62.170
CNN | 59.675 | 59.700 | 59.267
Table 2: User Level Accuracy of the Proposed Model (RNNwA) along with the Baselines. Model | English | Spanish | Arabic
---|---|---|---
CNN | 74.947 | 71.772 | 72.100
CNNwA | 78.474 | 75.000 | 71.800
RNN | 79.316 | 74.091 | 77.100
RNNwA | 81.789 | 78.227 | 78.500

In Table 2, user level accuracy results for the proposed model (RNNwA) along
with the baseline models are given. As can be seen in the results, tweet level
attention mechanism increases the score of all baseline models with the only
exception of the CNNwA model in Arabic.

Also, compared to the best neural model (takahashi2O18) where max pooling is
used instead of an attention mechanism on the outputs of RNN, the proposed
model (RNNwA) gives better results in terms of accuracy on English and Arabic
datasets, and produces similar accuracy levels on Spanish dataset (Table 3).
These results show that an attention layer is able to learn ”where/how to
look” for features that are helpful in identifying the gender of a user.

On the other hand, the improved model (RNNwA + n-gram), where neural and hand-
crafted features are concatenated, increases the accuracy of the proposed
model by approximately $0,5$% on English and approximately $2$% in Spanish and
Arabic. This also supports our intuition that the performance of neural models
can be improved by hand-crafted features, which is based on the study of
collobert2008. As can be seen in Table 3, the improved model outperforms the
state-of-the-art method of daneshvar2018 in English and produces competitive
results in Spanish and Arabic.

There is an interesting observation concerning the models without tweet level
attention (RNN and CNN) in hyper-parameter optimization. During the
hyperparameter optimization of the models RNN and CNN, we saved both the
models that gave the best tweet-level accuracy and the models that gave the
best user-level accuracy. The expectation is to see that the best setup on
tweet-level also gives the best performance in user-level, but the outcome is
the opposite: Best setups on tweet-level always fall behind best user-level
setups. Performance differences between various setups can be seen in Figure
LABEL:fig:scatterplot where accuracies of the best three models in terms of
tweet-level and best three models in terms of user-level are shown for all
languages. It can be observed that the best tweet-level setups are almost
$4\%$ worse in terms of user-level accuracy. Deeper investigation shows that
the best user-level models exhibit slight overfitting on tweet-level, in
training. Although overfitting normally leads to poor generalization, in this
case we believe that this overfitting acts similar to an attention mechanism
by over-emphasizing some important tweets and ignoring uninformative ones in
the process. Even though this leads to poor tweet-level accuracy, it improves
the user-level accuracy of the models as it can be seen from the Figure
LABEL:fig:scatterplot.

Table 3: Accuracy on PAN 2018 test set. Model | English | Spanish | Arabic
---|---|---|---
daneshvar2018111In their paper, authors report a result of 82.21 in English but we couldn’t verify their accuracy in our repetitions by using their software and the same dataset. | 81.52 | 82.00 | 80.90
takahashi2O18222Since their software is not provided, we directly take the accuracy values from their paper. | 79.68 | 78.64 | 77.10
Proposed Model (RNNwA) | 81.79 | 78.23 | 78.50
Improved Model (RNNwA + n-gram) | 82.31 | 80.22 | 80.50