# Demographic Inference and Representative Population Estimates from
Multilingual Social Media Data

Zijian Wang  Stanford University<EMAIL_ADDRESS>,  Scott A. Hale
University of Oxford & Alan Turing Institute<EMAIL_ADDRESS>,  David
Adelani MPI-SWS & Saarland University<EMAIL_ADDRESS>,  Przemyslaw A.
Grabowicz MPI-SWS & University of Massachusetts<EMAIL_ADDRESS>,  Timo
Hartmann  GESIS Cologne<EMAIL_ADDRESS>,  Fabian Flöck  GESIS Cologne
<EMAIL_ADDRESS>and  David Jurgens  University of Michigan
<EMAIL_ADDRESS>
(2019)

###### Abstract.

Social media provide access to behavioural data at an unprecedented scale and
granularity. However, using these data to understand phenomena in a broader
population is difficult due to their non-representativeness and the bias of
statistical inference tools towards dominant languages and groups. While
demographic attribute inference could be used to mitigate such bias, current
techniques are almost entirely monolingual and fail to work in a global
environment. We address these challenges by combining multilingual demographic
inference with post-stratification to create a more representative population
sample. To learn demographic attributes, we create a new multimodal deep
neural architecture for joint classification of age, gender, and organization-
status of social media users that operates in 32 languages. This method
substantially outperforms current state of the art while also reducing
algorithmic bias. To correct for sampling biases, we propose fully
interpretable multilevel regression methods that estimate inclusion
probabilities from inferred joint population counts and ground-truth
population counts. In a large experiment over multilingual heterogeneous
European regions, we show that our demographic inference and bias correction
together allow for more accurate estimates of populations and make a
significant step towards representative social sensing in downstream
applications with multilingual social media.

Demographics; Post-stratification; Social Media; Latent Attribute Inference;
Inclusion Probabilities; Multilingual; Deep Learning

††copyright: iw3c2w3††journalyear: 2019††conference: Proceedings of the 2019
World Wide Web Conference; May 13–17, 2019; San Francisco, CA, USA††booktitle:
Proceedings of the 2019 World Wide Web Conference (WWW ’19), May 13–17, 2019,
San Francisco, CA, USA††doi: 10.1145/3308558.3313684††isbn:
978-1-4503-6674-8/19/05††ccs: Human-centered computing Social media††ccs:
Computing methodologies Natural language processing††ccs: Computing
methodologies Object identification††ccs: Computing methodologies Neural
networks

##  1\. Introduction

Data representing the attitudes and behaviors of a (national) base population
is of great importance to policy making, social science research and
commercial prediction tasks, but representative surveys are expensive and
infrequent. Social media data has been proposed as a real-time and inexpensive
way to measure social phenomena—this area of research is known as “social
sensing.” Social media data has been used, with various levels of success, in
areas such as infectious disease (Ginsberg et al., 2008; Lazer et al., 2014;
Generous et al., 2014), migration and tourism (Lamanna et al., 2018; Zagheni
et al., 2017; Barchiesi et al., 2015), and box office takings for films
(Mestyán et al., 2013; de Silva and Compton, 2014).

Figure 1. Our two-stage approach debiases non-representative Twitter data by
(i) inferring demographics with state-of-the-art accuracy from multilingual,
profile-only data and (ii) learning inclusion probabilities to create more
representative samples for Europe-wide subregions

However, despite social media data being easily accessed, large in scale, and
detailed, it is generally not representative of indicator variables measured
in any broader (offline) population due to various biases (Ruths and Pfeffer,
2014; Jungherr, 2017). Especially the self-selection of users joining and
participating on any given social media platform is a large source of error.
In the UK for instance, Twitter users are more likely to be male and young
compared to the national population (Sloan, 2017), and in the U.S. men and
residents of densely populated areas are overrepresented (Hecht and Stephens,
2014), while there exists a mix of over- and undersampling on Twitter for
users with specific racial backgrounds (Mislove et al., 2011). In general,
however, the exact inclusion probabilities for a specific demographic group on
any platform in a given country or region are unknown. And while not all
social sensing tasks require representative data (An and Weber, 2015), recent
studies have pointed out the fallacies in predicting phenomena via social
sensing without controlling for sampling biases in social media data (Jungherr
et al., 2012; Gayo-Avello, 2012, 2013), with research applications aiming to
draw inferences for nation-wide target populations often relying on such
attribute data. Consequently, a systematic methodology for estimating them is
needed.

Survey analysis researchers have dealt with non-representative polls and non-
response through sample re-weighting—with post-stratification being a well-
known technique (Bethlehem and Keller, 1987; Holt and Smith, 1979). These
methods make use of basic demographics attributes: most prominently age,
gender and location (Park et al., 2004; Wang et al., 2015). Where demographic
details are provided, post-stratification has been a valuable technique (Wang
et al., 2015; Zagheni et al., 2017). However, most social media platforms do
not provide demographic data about their users, making it difficult to know or
correct for such biases. Although promising techniques for extracting
demographic attributes from social media have been proposed, there exists no
robust multilingual approach for this task. We address these challenges by
inferring basic demographic attributes and correcting for selection biases on
a large sample of multilingual profile data from Twitter (Figure 1), one of
the most used platforms for social sensing given its ease for obtaining data.

This work offers the following three contributions. First, we introduce a new
multilingual, multimodal, multi-attribute deep learning system for inferring
demographics of users (§2). This pipeline is built from the ground up to
enable inference in 32 different languages, addressing the need for such
methods beyond English and outperforming state-of-the-art methods on the tasks
of predicting the age, gender and is-organization state of Twitter users. We
show this system reduces algorithmic bias with respect to skin tone over the
best-performing commercial systems. Second, we formalize a statistical
framework for models that debias non-representative samples (§3). Our models
explicitly learn per-stratum inclusion probabilities from data, whereas
typical post-stratification methods either assume their knowledge or focus on
obtaining a post-stratified estimator of a response variable. Third, in real-
world evaluations, we show that our debiasing models significantly reduce the
error rate in comparison to a model without post-stratification on the
inferred demographic attributes (§4). In these ways, we make a significant
step towards representative social sensing in downstream applications using
multilingual social media.

##  2\. Demographic Inference

We propose a new demographic inference model for three attributes: gender,
age, and a binary organization indicator (“is-organization”). The first two
attributes, gender and age, are widely reported in census data and are core
features to accurately measure demographic biases. The third attribute was
selected based on a known confounder for people-based studies of online
platforms: the presence of accounts belonging to organizations (Alzahrani et
al., 2018; McCorriston et al., 2015), which shall be distinguished from
individual accounts. Next, we describe the model, its data, and its
evaluation.

###  2.1. Classification Task

We consider the task of estimating population count in Twitter given a stream
of user tweets. This scenario is motivated by the common research use case for
social sensing on Twitter, where a researcher consumes one of the Twitter
streams (e.g., the 10% or 1% sample streams) and desires to make some analysis
from the tweets of a broader phenomena that extends beyond Twitter. In such a
scenario, many users are expected to appear rarely–possibly only once–which
hinders the use of existing demographic methods that rely upon large amounts
of text to infer gender (Tannen, 1991, 1993; Kendall et al., 1997; Coates,
1998; Eckert and McConnell-Ginet, 2003; Lakoff and Bucholtz, 2004; Coates,
2015; Chen et al., 2015; Zhang et al., 2016) and age (Rosenthal and McKeown,
2011; Goswami et al., 2009; Rao et al., 2010; Nguyen et al., 2011, 2013; Sap
et al., 2014).

Instead, we avoid making inferences based on tweets and focus on using only
the information associated with a user’s account, which allows our method to
easily scale to large volumes of users without the need for significant
quantities of text. Not stratifying users based on tweets has the added
advantage of preventing downstream demographic biases to any social sensing
tasks that analyze the same language. Ultimately, our method uses four sources
of information, username, screen name, biography, and profile image, and is
designed to operate effectively on multilingual data.

We formalize the three classification tasks as follows. Gender and is-
organization are modeled as binary classification tasks.111We base our notion
of gender as one of performance (DeFrancisco et al., 2013), in which
individuals adapt their style, name, and picture to (de)emphasize certain
aspects of their gender identity (Eckert, 2008). While our training data uses
self-declared binary gender identities, the M3 model reports gender identity
as a continuum using prediction probability along a graded scale. We recognize
that the current approach does not yet support non-binary gender identities,
which we view as an important future task for full representation.  Age is
known to be a difficult inference task in social media (Nguyen et al., 2013;
Zhang et al., 2016). Here, we opt to categorize age in four levels: $\leq 18$,
$(18,30)$, $[30,40)$, $[40,99)$. Our choice is motivated by (i) our ability to
align these age ranges with census data reports and surveys (UK Office for
National Statistics, 2015; United States Department of Education, 2009), and
(ii) the difficulty of the task (even for humans) when finer-grained age
classes are used (Dehon and Brédart, 2001; Van Rijsbergen et al., 2014), while
(iii) still making the resulting classes amenable to downstream tasks. While
other attributes are possible for use in debiasing (e.g., education, income,
etc.), ground truth population statistics for these attributes are not widely
available in censuses, making it difficult to train and evaluate any model.

Figure 2. The M3 model for inferring gender, age, and organization-identity
from image and text data.

###  2.2. Method and Training Procedure

Demographic attributes are often expressed in both image and text. We propose
a new multimodal inference model with a novel training technique that
leverages both of these sources to significantly augment our predictive
accuracy. We first describe the model and each modality and then discuss the
training procedure.

####  2.2.1. Model

The full model comprises two separate pipelines for processing a profile image
and each of the three text sources of information, as shown in Figure 2. Here,
we use a shared pipeline for all three attributes, using multi-task learning
for the output. We refer to this as the M3 model, after its multimodal,
multilingual, and multi-attribute abilities. Following, we describe the
architectures of each modality and how they are combined.

_Image Model_ The image classifier was constructed using DenseNet (Huang et
al., 2017) based on initial performance experiments over current state-of-the-
art vision models on our data. We scale all profile images to 224x224 to meet
the input size of the model.

_Text Model_ Separate character-based neural models were trained for each text
input and then a multi-input model was tuned based on the weights from each
single model. Embeddings for username and biography were limited to the 3000
most frequent characters in our training corpus, with remaining characters
represented with an encoding for their unicode category; screen name are
embedded in the ASCII range. We use 2-stack bidirectional character-level
Long-short Term Memory (LSTM) architecture (Hochreiter and Schmidhuber, 1997)
to capture necessary information. To incorporate potential different meanings
in different languages for the same character, we concatenate a separate
trainable language embedding to each character embedding and add a fully
connected layer before being fed into an LSTM. The fusion of character-based
input with language embeddings provides two critical benefits that let us
scale to the linguistic diversity seen in global platforms like twitter. (1)
Character-based RNNs are able to capture morphological regularity within a
language and substantially reduce the parameter space due to the reduced
vocabulary size of the model (i.e., the set of characters) as compared with
word-based models with orders-of-magnitude larger vocabularies (Kim et al.,
2016; Faruqui et al., 2016; Chung et al., 2016). (2) By fusing language
embeddings with character embeddings, the joint embedding lets us represent
shared linguistic regularity across languages, which lets us better scale to
the multilingual environment.

_Full Multimodal Model_ To construct the full model, separate models are fully
trained for image and text for all attributes. Then, the softmax output layer
of each is removed and the two models are joined into a new model with a
modality drop-out layer (described next), two fully connected layers of size
2048, a Rectified Linear Unit (ReLU) layer (Nair and Hinton, 2010), and
separate output layers for each task. This full model is then fine-tuned using
the pretrained single-modality models’ parameters for initialization.

####  2.2.2. Training Procedure

Rather than training M3 end-to-end initially, we combine three procedures for
the training process: co-training, multilingual data augmentation, and
modality drop out.

_Co-Training_ Both image and text can have clear signal of demographics. Given
the potentially-limited training data for speakers of less common languages,
we take advantage of the multiple views of the same users by incorporating co-
training: a semi-supervised learning technique. Co-training first learns a
separate classifier for each view using any labeled examples. The most
confident predictions of each classifier on the unlabeled data are then used
to iteratively construct additional labeled training data (Blum and Mitchell,
1998). Ardehaly and Culotta (Ardehaly and Culotta, 2017) recently showed co-
training was effective for demographic classification in social media data
using images and text in a graph-based co-training learning procedure.

In comparison to the variance of gender and age expressions in textual data
across sociolinguistic contexts (Schler et al., 2006; Bamman et al., 2014),
image data provides a more universal depiction, acting as an analog to a
universal pivot language in machine translation (Wu and Wang, 2007). Thus, we
train the image-based portion of the M3 model first, which identifies high
confidence labeled data from an unlabeled multilingual dataset. Second, the
text-based portion of the M3 model is trained using the original dataset plus
the text data associated with any high-confidence image-based classifications.
Finally, we train the full M3 model with the training dataset and all high-
confidence predictions from the unlabeled data. All models are trained as
multi-task, where a common input representation and architecture is used to
predict each output tasks. The probabilities for each task were represented
using a top softmax layer.

_Multilingual Data Augmentation_ The M3 model is designed to operate in online
environments on tens of languages. However, few approaches and little labeled
data exists for this task for languages other than English, with some
exceptions (Ciot et al., 2013; Goot et al., 2018; Rangel et al., 2018). Even
with co-training, the relative differences in number of speakers between
languages (e.g., English vs. Slovenian) create a substantial class imbalance
which would prevent the model from effectively learning signals of each
attribute in users’ biographies written in less-frequent languages. Therefore,
we perform data augmentation by using automatic machine translation of the
training data.

Given the scale of our training data (described later in §2.3), the use of
online translation services is prohibitive both in terms of time and cost.
Instead, we opt to use word-based translation using the MUSE bilingual
dictionaries (Conneau et al., 2017). For each instance in the training data,
all words in the bilingual dictionary are replaced with the translation. A
translated instance is only kept if 80% or more of its words could be
translated. While word substitution typically produces translations that are
more incorrect and less fluent, we observed that the remaining biography
translations were often of reasonable quality due to the declarative nature of
biographies, which often consist primarily of lists of readily-translatable
noun descriptions or attributes of the user without using full grammatical
sentences. Such translations also largely capture topical and semantic
regularity, even if the morphology is not precise.

_Modality Drop-out_ The co-training model uses a conjoining architecture with
pre-trained weights from the image model and the multi-input text model. To
facilitate the model using each source of information, we introduce a new
technique, modality dropout, that can fully hide a source of information
(e.g., the biography) during training. We add an extra input dropout with a
probability of 0.25 (i.e., keep 3 out of 4 input sources on average) to better
regularize the model and support use cases where not all 4 input sources are
available. When doing dropout, images were replaced by a random matrix while
text was turned to a specific empty embedding.

###  2.3. Data

Training and evaluating M3 uses five distinct datasets: (1) a large dataset of
Twitter profiles whose gender and age are heuristically identified,
predominantly in English, (2) a curated dataset of Twitter accounts belonging
to organizations, (3) an image dataset of faces from IMDB and Wikipedia, (4) a
massive unlabeled set of users, and (5) a crowdsourced dataset for all three
attributes that spans all 32 languages in our study. We describe each next.

####  2.3.1. Heuristically-identified users

Heuristically-identified users were drawn from a 10% sample of Twitter from
2014 to 2017 on Twitter, where roughly 40% of the users are English speakers
(Hale, 2014). Here, we identify gender and age from fixed expression in users’
biographies that signal age or gender, e.g., “mother of two wonderful kids” or
“26 y/o dude” using a gender- or age-signaling word. Age data was further
augmented by identifying tweets wishing another user a happy birthday (cf. Al
Zamal et al. (2012)). These pattern-based approaches are known to be highly
precise for certain attributes when properly constructed (Beller et al., 2014;
Bergsma and Van Durme, 2013). Patterns were created for five languages:
English, Spanish, French, German, and Swedish.222Languages were selected to
capture linguistic diversity and where high precision demographic-identify
patterns could be created easily. Organization accounts were manually curated
by identifying 676 Twitter lists that primarily contained organizations, such
as non-profits, local clubs, companies, or municipal services. These lists
resulted in 59.92K unique organization accounts. For heuristically-identified
users and organization accounts, we collect their current profile image,
screen name, username, and biography. User age is adjusted to the present day.
For heuristically-identified users, the biography is altered to replace the
gender- or age-indicating word with a special token so that the model is
forced to look for additional cues to recognize each attribute. The IMDB-WIKI
dataset (Rothe et al., 2016) consists of 523,051 images from headshots of
actors in the IMDB and profile pictures in Wikipedia. This dataset provides an
auxiliary source of information for fine-tuning the image-based part of the M3
model.

These three datasets constitute our initial data prior to performing the
three-step annotation procedure. Table 1 shows the detailed statistics of the
data for each step.

Co-training depends on having access to a large set of unlabeled users whose
high-confidence labels from one modality (e.g., profile image) can be used to
augment the training set for another modality. Here, we collect 36.97M
profiles speaking one of the 32 languages commonly spoken in Europe but for
which we have no groud truth label. Languages were identified using CLD2
(McCandless, 2010). Note that users in this unlabeled dataset will only become
training instances if one view’s classifier (text or image) labels them with
high confidence.

Table 1. Sizes of datasets used for training and testing. Incremental numbers are shown for the outputs of data-augmentation from co-training and translation. Dataset | Gender | Age | Is-Org.
---|---|---|---
Initial | 3.98M | 1.20M | 59.92K
After Image Co-Training | +2.26M | +0.62M | +6.31M
After Translation Augmentation | +5.74M | +1.54M | +4.74M
After Text Co-Training | +8.28M | +0.78M | +11.87M
Final | 14.53M | 2.61M | 23.86M
Held-out Evaluation | 0.38M | 0.36M | 6.91K

####  2.3.2. Crowdsourcing Data

Current gender, age, and organization datasets are nearly all produced for
English-speaking users, which prevents us from evaluating the M3 model in the
multilingual environment. Therefore, we constructed a new dataset of up to 200
randomly-selected Twitter users speaking one of each of the 32 languages in
our study. Three annotators on Figure Eight333https://www.figure-eight.com/
were shown the usernames, screen names, short biographies, and profile images
of each Twitter account. We instructed workers to determine gender and age of
a given user from two drop-down menus based only on this information. The
selection options regarding gender also included a category for profiles
belonging to multiple persons, and to a non-person (e.g., organization/bot).
Regarding age, annotators were given seven categories to choose from: $<=18$,
$(18,30)$, $[30,40)$, $[40,50)$, $[50,60)$, $[60,70)$, $>=69$. They could
select “don’t know” for all questions, if they felt not enough information was
available. We employed Figure Eight’s selection mechanism for speakers of the
target language of the set to be annotated, or chose workers by their country
of residence, if one of that country’s main spoken languages was identical
with the target language of the task. Additionally, each job included a set of
30 test questions, annotated as a gold standard by the authors. These were
used to screen for spam but also for any non-speakers of the target language
remaining after our first filter; to this end, at least 10 questions were only
solvable by sufficiently understanding the target language.

Annotation reliability was measured for each Figure Eight job using
Krippendorff’s $\alpha$ (Krippendorff, 2011). For the computations of
$\alpha$, we excluded all English test-questions from the 31 non-English jobs,
profiles with only one annotation, and included “don’t know” answers as an
answer option. To establish a measure of how well workers did on the age
classification in respect to the age brackets classified by M3, we collapsed
age bracket encodings over 40 into one category. Across all languages, the
mean $\alpha$ for the gender annotations lies at 0.81 and at 0.57 regarding
the age brackets, hinting at the high difficulty of the Twitter profile age
classification task for humans. For a separate measure, we recoded all gender
and organization annotations into the two categories “is organization” and “is
not organization” and calculated Krippendorff’s $\alpha$ based on these data,
resulting in a mean of approximately 0.75 across all languages, showing that
workers could identify non-personal accounts reasonably well. The final
dataset is constructed from all instances where at least two annotators agreed
on the label for a particular attribute (majority vote), resulting in 4,732
profiles across 32 languages.

####  2.3.3. Data Partitions

All self-reported labeled data is partitioned into 80% train, 10% development,
and 10% test splits. No official train/test/dev splits are available for the
IMBD-WIKI datasets so we perform our own partition with the same percentages
as for self-reported. During model training, only the development set for
self-reporting users are used for model selection. Owing to its small size,
crowdsourcing data was only used for testing purposes. For the organizational
data, we create a full dataset with non-organizational accounts by randomly
sampling accounts from the heuristically labeled data to attain a 1:9 ratio,
following the report in McCorriston et al. (McCorriston et al., 2015) that
organizations make up 10% of the accounts on Twitter.

####  2.3.4. Co-Training Setup

The training data is balanced for each stage, where we oversampled gender and
organization status, and undersampled the two most-frequent classes (0-18 and
19-29) while oversampling the other two age classes. High confidence
thresholds were set to 0.9 for gender and organization status in both the
image and text modalities. As a four-class classification task, the high-
confidence for age is set to 0.7 in the image classifier, based on early
examination of the prediction quality. We observed that age is more readily
predictable in text and therefore used a threshold of 0.9 to keep quality high
while also providing a sufficient number of new accounts for co-training.

Separate image and text models are each trained for 10 epochs. Due to the
imbalanced number of instances per attribute (Table 1), each epoch contains
10000 steps where each step contains a mini-batch of 128 instances balanced
across the three attributes. This setup ensures the model sees instances of
the smaller datasets (age and organization). The image model is trained first,
and its high-confidence instances are then used to train the text model. The
final co-training model was trained for 5 epochs with input dropout and
another 3 epochs without input dropout for better convergence. Amsgrad
optimizers (Reddi et al., 2018) were used in all training process, with a
learning rate of 0.001 for ground-up models and 0.0005 for fine-tuning models.
Text models were trained on one GPU, while the image and co-training models
were trained in parallel on 4 GPUs. The full pipeline was built in PyTorch
(Paszke et al., 2017).

###  2.4. Evaluations

Accurate inference of representative attitudes from social media relies upon
accurate stratification approaches. Here, we test the M3 model against current
state of the art systems to describe the potential for error to affect
demographic bias.

####  2.4.1. Comparison Systems and Data

The M3 model is compared with state-of-the-art systems from each modality and
attribute. For images, we compare with Face++ (Jung et al., 2017) and
Microsoft Face API (Azure, 2018) on age and gender performance. For text, we
compare with three current state-of-art systems for inferring gender from
usernames, genderperformr (Wang and Jurgens, 2018), demographer (Knowles et
al., 2016) and Jaech et al. (Jaech and Ostendorf, 2015), which could feasibly
operate on names from multiple languages.

No multilingual organization recognition system exists; so, we limit our
evaluation to the only publicly-available dataset of person-vs-organization
which was scored by the Humanizr (McCorriston et al., 2015) and Demographer
(Wood-Doughty et al., 2018) systems. This data consists of a uniform sample
across Twitter accounts, which were approximately 10% organizations. Here, we
recollect the user profiles for their 20,273 accounts, of which 18,587 (91.6%)
were still available as of October 2018, which matched the distribution
originally reported. Their model was evaluated on the dataset using cross-
validation, whereas we treat the entire data as a test set and report
performance.

####  2.4.2. Gender Recognition

M3 produces state-of-the-art gender recognition for each of the three image-
based datasets (IMDB, WIKI, and Twitter), as seen in Figure 3. M3 provides
significant improvements in performance (F1) at each comparison system’s level
of recall.444The coverage reported is the maximum available from the
commercial APIs, which did not find a face or did not provide a gender/age
estimate for each photo.  The one exception to this trend is the performance
of the MSFT classifier on the WIKI dataset, which has highly similar
performance to M3. Given the MSFT’s model large jump in performance, we
hypothesize that their model may have been trained on Wikipedia data, although
this cannot be confirmed. We observed that the increase in coverage of M3 is
due in part to the model learning non-facial attributes associated with more
than one gender, e.g., certain sports with the male gender, which allow M3 to
process profile photos that other models cannot.

On the text-based data, M3 outperforms all systems except one when using just
the username and masking all other information, shown in Table 2. When the
model is allowed to see all other text information, M3 performance improves
substantially, indicating it can effectively fuse several sources of
information. While this latter setup uses more information than comparison
systems, in practice, the Twitter API includes all text information used by M3
by default; so the only additional step is downloading the profile image.

Figure 3. Model performances on gender classification for the three image
datasets, as measured through F1. Coverage equals % of returned results at
varying confidence levels for M3 and is the maximum available for the
commercial APIs.

####  2.4.3. Age Recognition

Age recognition is a significantly more difficult task in social media, as
seen by the performances in Figure 4. The M3 model offers similar performance
to commercial models on the IMDB and WIKI datasets, which primarily feature
face-forward headshots in good lighting. However, in real-world Twitter
profiles (shown in blue), the M3 model substantially outperforms both in F1 at
each of their respective coverage levels, with 0.16 and 0.11 absolute
improvements over Face++ and Microsoft, respectively.

Figure 4. Model performances on age classification for three datasets, as measured through F1 on classified images and Coverage on the % of images classified. Table 2. Performance comparison on gender using text methods. †Demographer only has a coverage of 0.974.  Method | Macro-F1 |
---|---|---
M3: Full Text | 0.907 |
M3: Username Only | 0.828 |
GenderPerformr (Wang and Jurgens, 2018) | 0.835 |
Demographer (Knowles et al., 2016) | 0.781 | †
Jaech and Ostendorf (2015) | 0.763 |

####  2.4.4. Organization Recognition

For any task that draws inferences about human actors, non-human and
specifically organizational accounts are a major source of error. The results
of Table 3 show that our model is able to identify these accounts with 16.3%
higher accuracy than the next closest system with no drop in performance at
recognizing humans. Results on our test set from our organization data (§2.3)
in Table 4 (top right) show similar performance, with an overall F1 of 0.898
that indicates high performance for both classes.

Table 3. Performance at recognizing organizational accounts, measured as accuracy per class, following (McCorriston et al., 2015) Method | Person | Organization
---|---|---
M3 | 0.986 | 0.807
Demographer (Wood-Doughty et al., 2018) | 0.973 | 0.644
Humanizr (McCorriston et al., 2015) | 0.982 | 0.586

####  2.4.5. Multilingual Performance

As a full test of M3, we evaluate it against our crowdsourced account labels
in 32 languages, shown in Table 4 top right. Performance in this multilingual
setting is on par with the performance on the primarily-English heuristically-
labeled data, though age classification remains the most difficult task.
Figure 5 shows the performance per language for each attribute. Performance at
predicting gender and is-organization are similar for most languages, while
age performance varies significantly by language from 0.28 F1 for Bosnian to
0.73 for Slovenian and Welsh. These results together indicate that M3 is
sufficiently accurate in the multilingual European environment.

Figure 5. M3 performance on the multilingual dataset in 32 languages. Bars
show 95% confidence intervals of accuracy.

####  2.4.6. Ablation Study

To examine which parts of the system are contributing to performance, we
perform an ablation study by (i) restricting the model to one modality and
(ii) training a model without using co-training or translation for data
augmentation. We test each model on (i) the heuristically-labeled and
organisation data and (ii) the multilingual data, which is representative of
the performance on the data used in our downstream task.

The results shown in Table 4 reveal two main trends. First, the model benefits
from both modalities. The removal of textual information causes the biggest
performance drop. Second, although co-training and translation hurt
performance in the heuristically-labeled data, they produced a substantially
better model when evaluated on multilingual data. Since the heuristically-
labeled data is primarily in English, translation potentially adds noise and
forces the model to represent a larger space of inputs that are not
represented in the test data. This performance difference indicates that the
two data augmentation techniques are highly beneficial when bootstrapping a
model from mostly monolingual data.

Table 4. Performance (Macro-F1) for (1) full M3 model, (2-3) model with
modality dropouts, (4) model trained without co-training or translation, (5-6)
Random and and Majority-class baselines.

| Heuristically-labeled | Multilingual
---|---|---
M3 Model | Gender | Age | Org-Status | Gender | Age | Org-Status
Full | 0.918 | 0.522 | 0.898 | 0.915 | 0.425 | 0.898
w/ Text Dropout | 0.743 | 0.349 | 0.837 | 0.837 | 0.359 | 0.892
w/ Image Dropout | 0.905 | 0.493 | 0.823 | 0.862 | 0.372 | 0.822
w/o Co-T. & Trans. | 0.918 | 0.449 | 0.917 | 0.875 | 0.325 | 0.660
Random | 0.494 | 0.181 | 0.500 | 0.494 | 0.211 | 0.466
Majority | 0.377 | 0.214 | 0.500 | 0.334 | 0.187 | 0.210

####  2.4.7. Test for Algorithmic Bias

Image-based models for gender recognition are known to suffer from algorithmic
bias in recognizing darker-skinned individuals (Buolamwini and Gebru, 2018).
This bias is thought to be a result of non-representative training data the
underreprentation of darker skin tones in the training data. As our focus is
on European countries, such a bias could be present in the M3 data. However,
because our co-training procedure uses a large unlabeled set of users from
across the globe, these users can potentially provide a more representative
sample and reduce algorithmic disparity. To test for algorithmic bias, we
compare the M3 model’s performance on the Gender Shades dataset (Buolamwini
and Gebru, 2018), which contains 3964 gender-annotated facial images balanced
across light and dark skin tones. In this earlier algorithmic audit, the
Microsoft (MSFT) and Face++ gender classifiers performed substantially worse
on women, especially darker skinned women.

The gender inference performance, shown in Table 5, reveals that M3 has
substantially less algorithmic bias relative to the two best-performing
commercial systems also tested on Gender Shades. Our model significantly
improves performance on dark-skinned women in comparison to the other two
systems which had the largest performance disparity on that demographic.
However, M3 is least accurate on darker-skinned males, which indicates that
additional work is needed to reach performance parity in skin tones.
Nevertheless, this analysis also indicates that M3 is more suitable than
existing systems for operating in the global environment and increases the
robustness of downstream social sensing applications.

Table 5. Performance on the Gender Shades dataset (cf. Table 4 in (Buolamwini
and Gebru, 2018)). Gender classification performance as measured by the
positive predictive value (PPV), true positive rate (TPR), and false positive
rate (FPR).

Classifier | Metric | All | F | M | Darker | Lighter | DF | DM | LF | LM
---|---|---|---|---|---|---|---|---|---|---
MSFT | PPV | 93.7 | 89.3 | 97.4 | 87.1 | 99.3 | 79.2 | 94.0 | 98.3 | 100.0
TPR | 93.7 | 96.5 | 91.7 | 87.1 | 99.3 | 92.1 | 83.7 | 100.0 | 98.7
FPR | 6.3 | 8.3 | 3.5 | 12.9 | 0.7 | 16.3 | 7.9 | 1.3 | 0.0
Face++ | PPV | 90.0 | 78.7 | 99.3 | 83.5 | 95.3 | 65.5 | 99.3 | 94.0 | 99.2
TPR | 90.0 | 98.9 | 85.1 | 83.5 | 95.3 | 98.8 | 76.6 | 98.9 | 92.9
FPR | 10.0 | 14.9 | 1.1 | 16.5 | 4.7 | 23.4 | 1.2 | 7.1 | 1.1
M3 | PPV | 96.7 | 99.4 | 94.5 | 94.0 | 99.3 | 99.6 | 89.3 | 99.3 | 99.2
TPR | 96.5 | 92.8 | 99.6 | 93.4 | 99.3 | 86.0 | 99.7 | 99.0 | 99.5
FPR | 4.2 | 0.4 | 7.2 | 7.7 | 0.8 | 0.3 | 14.0 | 0.5 | 1.0

##  3\. Learning inclusion probabilities

In social sensing studies, the results of measurements performed on a given
platform, e.g., a social media site, are studied to understand the behavior of
a population. Often, such measurements are biased (Ruths and Pfeffer, 2014),
as individuals with certain demographics are more likely to join these
platforms, e.g., young people may be more likely to join Twitter. Obtaining
representative estimates in these scenarios is challenging, as the probability
of an individual with given demographics to be on a given platform, also
referred to as inclusion probability in sampling methodology, is typically
unknown.

Here, we estimate these inclusion probabilities as a function of demographics.
To this end, we learn the debiasing coefficients on the grounds of statistical
survey analysis with missing data (Bethlehem and Keller, 1987; Little and
Rubin, 2002; Särndal and Lundström, 2005). Specifically, we derive estimators
for the numbers of individuals by making an assumption that per-strata
inclusion probabilities are equal for individuals within a group (e.g., a
country) and different between groups (e.g., between countries). This
assumption allows us to get both across-groups global debiasing coefficients
(i.e., the biases of Twitter users) and the group-specific debiasing
coefficients (i.e., per-country demographics on Twitter, e.g., developed
countries may have more and diverse users on Twitter). Our approach learns the
inclusion probabilities, while typical post-stratification methods either
assume these probabilities are known or focus on obtaining a post-stratified
estimator of a response variable.

###  3.1. Formulation of Debiasing Models

Consider a population $U$ of $N=|U|$ individuals with certain demographics.
For simplicity, we focus on the case of only two discrete demographic
variables, say age $a$ and gender $g$, which are distributed following
$P_{\text{N}}(a,g)$, but the following reasoning applies to other demographics
as well. Out of these $N$ individuals, say, $M\leq N$ joined a certain online
platform with a probability depending on their demographics. For instance,
individuals joined Twitter and younger people were more likely to join than
older ones.

In survey analysis, the probability that an individual with certain
demographics joined a certain platform corresponds to the inclusion
probability of the stratum representing these demographics (Bethlehem and
Keller, 1987; Särndal and Lundström, 2005). In the simplest scenario, when the
probability of joining the platform is homogeneous in time and across
individuals, this probability can be expressed as the ratio of the number of
Twitter users with the demographics $a$ and $g$ to all individuals, i.e.,
$\pi(a,g)=\frac{M(a,g)}{N(a,g)}=\frac{MP_{\text{M}}(a,g)}{NP_{\text{N}}(a,g)}$,
where $P_{\text{M}}(a,g)$ and $P_{\text{N}}(a,g)$ are the distributions of
demographics of Twitter users and the overall population, respectively.
However, the inclusion probability may vary between individuals. To account
for this, we discuss the homogeneity of inclusion probability for a given
partition of the population. A partition of the population $U$ is defined
through non-overlapping and non-empty subsets $U_{i}$ of the population that
together constitute $U$, i.e., $\bigcup_{i}U_{i}=U$. Typically such subsets
will have a certain meaning, e.g., a natural partition of a population is a
split by countries, regions, or cities. The total number of individuals in the
subset $i$ is $N^{i}$ and the number of individuals having particular
demographics is $N^{i}(a,g)$.

####  3.1.1. Homogeneous bias

The inclusion probability, which governs the bias, is homogeneous with respect
to a given set of demographics and a given partition of the population, if and
only if $\pi^{i}(a,g)=\pi(a,g)$ for each subset $i$ of the partition, i.e.,
the inclusion probability does not depend on the elements of the partition.

If the inclusion probability is homogeneous, then we can write

(1) $\displaystyle N^{i}$
$\displaystyle=\sum_{a}\sum_{g}N^{i}(a,g)=\sum_{a}\sum_{g}\frac{M^{i}(a,g)}{\pi(a,g)}.$

Thus, to obtain the inclusion probabilities, we regress $N^{i}$ against
$M^{i}(a,g)$. If the condition of homogeneity holds, then the regression
coefficients are equal to $1/\pi(a,g)$.

####  3.1.2. Inhomogeneous bias

If the inclusion probability is inhomogenous for given demographics and
partition, then we shall model each subset of the partition separately or use
a different model that relaxes the homogenity assumption by specifying the
functional form of inhomogeneity. Here, we consider the inhomogeneity of the
form

(2) $\pi^{i}(a,g)=M^{i}(a,g)^{\nu}f_{1}(a)f_{2}(g),$

where $\nu$ is an unknown exponent and $f_{1}$ and $f_{2}$ are unknown
functions.555Note that the aforementioned homogeneity assumption is a special
case of this assumption, i.e., this assumption relaxes the homogeneity
assumption. Under this assumption, $\log
N^{i}(a,g)=\log\frac{M^{i}(a,g)}{\pi^{i}(a,g)}=(1-\nu)\log M^{i}(a,g)-\log
f_{1}(a)-\log f_{2}(g).$ This time, to obtain the debiasing coefficients and
corresponding inclusion probabilities, we regress the ground-truth population,
$\log N^{i}(a,g)$, against the biased measurements, $\log M^{i}(a,g)$, and the
demographic indicator variables, i.e.,

(3) $\displaystyle\log N^{i}(a,g)$ $\displaystyle=\beta_{1}\log
M^{i}(a,g)+\sum_{\tilde{a}}\beta_{\tilde{a}}\delta_{\tilde{a}a}+\sum_{\tilde{g}}\beta_{\tilde{g}}\delta_{\tilde{g}g}$

where $\delta_{ij}$ is Kronecker delta, i.e., $\delta_{ij}=0$ if $i\neq j$ and
$\delta_{ij}=1$ if $i=j$, $\beta_{1}=(1-v)$, $\beta_{a}=-\log\ f_{1}(a)$, and
$\beta_{g}=-\log\ f_{2}(g)$. From these regression coefficients we can obtain
the inclusion probability $\pi^{i}(a,g)$ of a set of samples $i$ with the
demographics $a$ and $g$ via Equation 2. This debiasing model has been
proposed recently by Zagheni et al. to predict the number of migrants based on
Facebook data (Zagheni et al., 2017), but its derivation and explicit formal
interpretation have not been provided until now, to the best of our knowledge.
Note that this estimation method of inclusion probabilities requires the
ground-truth joint counts $N^{i}(a,g)$ at the time of training, whereas the
model based on the homogeneity assumption requires only the total counts
$N^{i}$. The availability of the joint counts is often limited in practice
because of insufficient number of samples per stratum.

###  3.2. Discussion

We derived debiasing models by making homogeneity or inhomogeneity
assumptions. These models predict population size, so they can be evaluated by
measuring the error of predictions in cross-validation settings or via model
selection methods. In this way, we test which assumption is closer to reality
and find the most accurate method for the given dataset. We evaluate these
models in the next section using the population of regions in EU countries.

##  4\. European Population Inference from Twitter data

Here, we use the debiasing models formulated in the previous section to obtain
the debiasing coefficients and the corresponding inclusion probabilities for
Twitter users of different countries. To this end, we regress the country-
level ground-truth number of people living in a certain location against the
number of Twitter users of different demographics.

###  4.1. Debiasing Models for Population Inference

More specifically, we evaluate five models requiring different amounts of
data. The first model is a baseline, the next three models are based on the
assumption of homogeneous inclusion probabilities (Equation 1), whereas the
last one is based on the inhomogeneity assumption (Equation 3):

$\mathbf{N\sim M}$:

is our base model that uses only the total population count from the census
($N$) and Twitter ($M$).

$\mathbf{N\sim\sum_{g}M(g)}$:

uses gender marginal counts only (i.e., the total counts of males and females
not broken down by ages).

$\mathbf{N\sim\sum_{a}M(a)}$:

uses age marginal counts only.

$\mathbf{N\sim\sum_{a,g}M(a,g)}$:

uses the joint histograms inferred from Twitter but only the total population
values from the census.

$\mathbf{\text{{log}}~{}N(a,g)\sim\text{{log}}~{}M(a,g)+a+g}$:

uses the joint histograms inferred from Twitter and the joint histograms from
the census.

Note that Twitter users are biased in various ways: the platform is more
accessible to tech-savvy individuals and citizens of various countries use it
to a different extent (Hale, 2014). To distinguish the global effect of
Twitter on the overall bias from the local effect of country, we use
multilevel models. Namely, all slopes and intercepts in the introduced models
have random effects specific to a country. Thus, from fixed effects of the
model, we obtain global debiasing coefficients (i.e., the biases of Twitter
users) and, from random effects, the country-specific debiasing coefficients
(i.e., a given country has its own bias towards Twitter).

Note that the homogeneity and inhomogeneity assumptions apply within each
country separately. Namely, we group samples by regions of a country,
described next. Then, the homogeneity assumption translates into the same
inclusion probability for a given demographic across all regions of a country,
whereas the inhomogeneity assumption translates into a dependence of the
inclusion probability on regions of a country that follows Equation 2.

###  4.2. Data

We retrieved joint population distributions for age and gender on a regional
level (NUTS3) for 26 countries of the European Union plus four EFTA members as
made available through the CensusHub system of Eurostat, the European
Statistical Office.666https://ec.europa.eu/CensusHub2/: Luxembourg and Belgium
were not available. Additional EFTA countries: Switzerland, Iceland,
Liechtenstein and Norway. All census data is from 2011, the most recent year
for which comprehensive census data was collectively reported to Eurostat.
NUTS3 regions are the finest-grained level of the “Nomenclature of Territorial
Units for Statistics” used as a standard for statistical reporting in EU
member states (European Commission, [n. d.]). They are based on existing local
administrative boundaries and are usually at the level of local districts.
Although sizes differ to a certain extent per country, NUTS3 provide the most
standardized cross-country geographical units to subdivide populations. We use
the 2010 iteration of the NUTS3 regions as these correspond with the 2011
census subdivisons and is the most recent census available.

Our sample of social media users is derived from the random 10% stream of
tweets from Twitter. We recorded all users observed from September 2015 to
January 2016 and downloaded profile photos for all users in spring 2018. The
location of each user is inferred using the method of Compton et al. (Compton
et al., 2014). This model predicts a user’s latitude and longitude and was
shown to be least biased with respect to urban and rural areas (Johnson et
al., 2017), which is important given the diversity in population centers in
our study. The model was trained on a social network of 781M edges constructed
from Twitter data spanning 2012 to 2017. Five-fold cross validation reports a
median inference error of 7.9km, which is sufficiently accurate for the
geographic granularity we use here. Each user is assigned to a NUTS3 region in
Europe if the inferred latitude-longitude pair of that user lies within the
boundaries of the NUTS3 regions; users not in these regions are discarded.
Ultimately, we obtain a dataset of 3,202,964 users within the NUTS3 regions
for our study. In the remainder, we will refer to NUTS3 regions simply as
“regions.”

The age, gender, and is-organization variables for each user in our dataset
are inferred using M3. As an additional experiment to quantify the impact of
including non-human accounts, we ignore any organization classification and
treat all accounts as humans, grouping them according to their inferred
demographics.

###  4.3. Results

We evaluate the debiasing models in the following cross-validation settings:
leave one region out, leave one country out (i.e., leave out all regions from
a given country), and leave one stratum out (e.g., leave out only males aged
30-39). Per case, we measure the mean absolute percentage error of the
population estimates for the left-out samples, i.e.,
$\text{MAPE}(N)=\frac{100\%}{n}\sum_{i}\frac{|\hat{N}_{i}-N_{i}|}{|N_{i}|}$,
where $\hat{N}_{i}$ and $N_{i}$ are the predicted and actual population sizes,
respectively, and the sum is over all regions. Note that the model $\log
N(a,g)\sim\log M(a,g)+a+g$ operates naturally in the log space; hence, before
calculating the error of this model we first exponentiate the predicted
population sizes to make the results more comparable across all models. Its
results should further be compared to the remaining models with care, as the
effects of moving to log space cannot be clearly untangled from gains due to
learning on joint age and gender attributes.

The results of the leave-one-region-out evaluation show a clear benefit to
debiasing based on the inferred demographics (Figure 6). For the $N\sim M$
model without demographics, MAPE is 88%. The inclusion of the inferred gender
or age demographics in the debiasing models, via $N\sim\sum M(a)$ and
$N\sim\sum M(g)$, decreases MAPE to 59% and 61%, respectively. Including
inferred joint distributions, e.g., the number of males of age 30-39, in the
model $N\sim\sum M(a,g)$, decreases MAPE further to 54%, even though we do not
use joint-distribution data from the census for training. In fact, the use of
census joint distributions at the stage of model training along with a move to
log space, via $\log N(a,g)\sim\log M(a,g)+a+g$, further improves the
accuracy, bringing the MAPE down to 33%. To understand the effect of accounts
belonging to organizations on this population prediction task, we compare the
results of the debiasing models with and without removing the organizations
(Figure 6). Removing organizations results in a small reduction in error for
all but the model trained with only marginal gender counts, though these
differences are not statistically significant. A further analysis shows that
the presence of organizations is heavily skewed towards populous metropolitan
cities and most regions have very few organizations (i.e., organizations are
not equally geographically distributed). However, for populous regions,
organizations can create significant error.

Figure 6. Performance on leave-one-region-out population inference across
different debiasing models. Bars show mean MAPE($N$) with 95% confidence
intervals. The results of the last model are exponentiated to compute
MAPE($N$).

To gain further insights into the introduced models, we show scatter plots of
true and predicted population sizes for each model (Figure 7). The model with
joint Twitter and census counts is noticeably closer to the $y=x$ line, likely
because this models is trained in log space. For this model, we also depict
the geographical variation of MAPE across the EU regions (Figure 8).

Figure 7. Comparing the census population and debiased estimates for leave-
one-region-out evaluations shows that the model with joint Twitter and census
distributions correlates best with the real values for large and small
regions.

The above results test predictions when one region is left out and its
population is predicted. However, in different circumstances, more or less
census information could be available. We test two other cross-validation
settings: leave one stratum out (e.g., leave out only females aged 30–39) and
leave one country out (i.e., leave out all regions from a given country). The
latter reflects the generalizability of the model to completely unseen
countries where platform adoption probabilities and country-specific biases
are not known. The evaluation results for all three cross-validation settings
are plotted in Figure 9 for the most accurate model, i.e., $\log
N(a,g)\sim\log M(a,g)+a+g$. From the first additional cross-validation
setting, we learn that hiding the population sizes for a specific stratum in
all regions results in a minimal penalty to the prediction accuracy. For the
second additional cross-validation setting, we see that the error rate nearly
doubles with an average MAPE of 81%. This result suggests that knowledge of
country-specific platform biases is important for accurate estimates and that
at least some regions within the country should be seen during a model’s
training time to reach high performance.

Figure 8. MAPE for $\log N(a,g)\sim\log M(a,g)+a+g$ model across regions shows
that many regions have population estimate errors below 10% of the true
population, with high-error regions found infrequently in all countries. For
greyed regions we either miss NUT3 data or Twitter counts are zero. Figure 9.
Comparisons of the model with joint Twitter and census distributions as
increasing amounts of information are held out from training: (i) leaving out
one stratum, (ii) leaving out all the strata from one region, and (iii)
leaving out all regions in one country.

###  4.4. Discussion of Debiasing Results and Potential Sources of Error

Debiasing social media data samples is a difficult task, but our results show
that automatic post-stratification with respect to inferred age and gender
notably improves population estimation results. Furthermore, our predictive
models are fully interpretable, which allows us to estimate the inclusion
probabilities and share them with the research community for future reuse.

Even when joint distributions are not given by the census, inferring joint
distributions from social media data with the model $N\sim M(a,g)$ provides a
significant increase in accuracy in population prediction tasks compared to
the baseline without debiasing. However, the model $\log N(a,g)\sim\log
M(a,g)+a+g$ is notably more accurate, suggesting that the assumption of
homogeneity does not hold for the partition of citizens of a particular
country, gender, and age into regions. Note, however, that the predictions of
the most accurate model are not perfect. This may be caused by the time
mismatch of about 5 years between the Twitter and NUTS3 datasets. On the other
hand, this point opens the door for developing debiasing models based on other
inhomogeneity assumptions and searching better partitions, e.g., our
multilevel model could have more levels to capture biases shared by smaller
regions within a country.

To learn more about potential confounders, we compare the MAPE of the model
$\log N(a,g)\sim\log M(a,g)+a+g$ to three variables: the area of the region,
its population density, and average income (Figure 10). NUTS3 regions are
defined by each country and vary considerably in land area from city-states in
Germany to large regions in Sweden (up to 98,000 km2). We do not find
correlations of land area with the MAPE error. Hecht and Stephens (Hecht and
Stephens, 2014) found a clear urban–rural bias in social media data: there
tend to be more users, more information, and higher quality information per
capita within metropolitan areas. Despite this bias, we find our estimates not
heavily correlated with population density, suggesting our models do well at
debiasing these differences in inclusion probabilities. Only two countries
(Czech Republic and Norway) show a correlation that is statically different
from zero across our models. Finally, income has been suggested as an
important explanatory variable for who uses Internet-based platforms and hence
could be a potential confound to our estimates (Chinn and Fairlie, 2007;
Jansen, 2010). However, we similarly find that income and MAPE are not
correlated significantly in our models.

Figure 10. Comparing the model with joint Twitter and census distributions
MAPE to region area (km2), population density (people/km2), and income (USD
per capita) shows that the model performance is not biased towards particular
types of regions. Colors represent countries as in Figure 7.

##  5\. Conclusion

The everyday opinions expressed in social media provide promising
opportunities for measuring population-level statistics for health metrics,
political outcomes, or general attitudes. However, social media is a non-
representative sample of the population due to demographic skew in usage
frequencies and access rates. As such, any direct estimate from a platform is
likely biased towards certain demographics. This work provides a holistic
solution to this problem by developing a novel method for assigning users to
demographic strata and exploiting it in a regression framework for debiasing
that allows direct estimation of the probability of an individual with given
demographics to be on the given social media platform. Our work provides three
main contributions. First, we introduce a state-of-the-art neural system for
multi-attribute classification in 32 languages. This contribution also
includes the system release and the creation of a new dataset of gender, age,
and is-organization annotations in 32 languages. Second, we derive a series of
models to debias social media measurements and provide their explicit formal
interpretations. Third, in a massive study of all of Europe, we show that our
two methods are able to infer regional population counts accurately and
provide demographic corrections for all downstream measurements on the grounds
of the estimated inclusion probabilities. These results pave the way for more
accurate social sensing by laying a foundation of representative population
sampling in social media. Code, software, and debiasing coefficients
pertaining to this work are released for public use at
https://github.com/euagendas/.

## Acknowledgements

This research has received funding through the Volkswagen Foundation and was
supported by The Alan Turing Institute under EPSRC grant EP/N510129/1.

## References

  * (1)
  * Al Zamal et al. (2012) Faiyaz Al Zamal, Wendy Liu, and Derek Ruths. 2012.  Homophily and Latent Attribute Inference: Inferring Latent Attributes of Twitter Users from Neighbors. In _Proceedings of ICWSM_.
  * Alzahrani et al. (2018) Sultan Alzahrani, Chinmay Gore, Amin Salehi, and Hasan Davulcu. 2018\.  Finding Organizational Accounts Based on Structural and Behavioral Factors on Twitter. In _International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation_. Springer, 164–175.
  * An and Weber (2015) Jisun An and Ingmar Weber. 2015.  Whom should we sense in “social sensing”—Analyzing which users work best for social media now-casting.  _EPJ Data Science_ 4, 1 (30 Nov 2015), 22\.  https://doi.org/10.1140/epjds/s13688-015-0058-9
  * Ardehaly and Culotta (2017) Ehsan Mohammady Ardehaly and Aron Culotta. 2017.  Co-training for Demographic Classification Using Deep Learning from Label Proportions. In _Data Mining Workshops (ICDMW), 2017 IEEE International Conference on_. IEEE, 1017–1024.
  * Azure (2018) Microsoft Azure. 2018\.  Cognitive Services.  https://azure.microsoft.com/en-us/services/cognitive-services/
  * Bamman et al. (2014) David Bamman, Jacob Eisenstein, and Tyler Schnoebelen. 2014\.  Gender identity and lexical variation in social media.  _Journal of Sociolinguistics_ 18, 2 (2014), 135–160.
  * Barchiesi et al. (2015) Daniele Barchiesi, Helen Susannah Moat, Christian Alis, Steven Bishop, and Tobias Preis. 2015\.  Quantifying International Travel Flows Using Flickr.  _PLOS ONE_ 10, 7 (07 2015), 1–8.  https://doi.org/10.1371/journal.pone.0128470
  * Beller et al. (2014) Charley Beller, Rebecca Knowles, Craig Harman, Shane Bergsma, Margaret Mitchell, and Benjamin Van Durme. 2014.  I’m a belieber: Social roles via self-identification and conceptual attributes. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , Vol. 2. 181–186.
  * Bergsma and Van Durme (2013) Shane Bergsma and Benjamin Van Durme. 2013.  Using Conceptual Class Attributes to Characterize Social Media Users. In _Proc. ACL_.
  * Bethlehem and Keller (1987) Jelke G Bethlehem and Wouter J Keller. 1987.  Linear weighting of sample survey data.  _Journal of Official Statistics_ 3, 2 (1987), 141–153.
  * Blum and Mitchell (1998) Avrim Blum and Tom Mitchell. 1998.  Combining labeled and unlabeled data with co-training. In _Proceedings of the eleventh annual conference on Computational learning theory_. ACM, 92–100.
  * Buolamwini and Gebru (2018) Joy Buolamwini and Timnit Gebru. 2018.  Gender shades: Intersectional accuracy disparities in commercial gender classification. In _Conference on Fairness, Accountability and Transparency_. 77–91.
  * Chen et al. (2015) Xin Chen, Yu Wang, Eugene Agichtein, and Fusheng Wang. 2015\.  A Comparative Study of Demographic Attribute Inference in Twitter.. In _Proceedings of ICWSM_ , Vol. 15. 590–593.
  * Chinn and Fairlie (2007) Menzie D Chinn and Robert W Fairlie. 2007.  The determinants of the global digital divide: a cross-country analysis of computer and internet penetration.  _Oxford Economic Papers_ 59, 1 (2007), 16–44.
  * Chung et al. (2016) Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016\.  A character-level decoder without explicit segmentation for neural machine translation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, 1693–1703.
  * Ciot et al. (2013) Morgane Ciot, Morgan Sonderegger, and Derek Ruths. 2013\.  Gender inference of Twitter users in non-English contexts. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. 1136–1145.
  * Coates (1998) Jennifer Coates. 1998\.  _Language and gender: A reader_.  Wiley-Blackwell.
  * Coates (2015) Jennifer Coates. 2015\.  _Women, men and language: A sociolinguistic account of gender differences in language_.  Routledge.
  * Compton et al. (2014) Ryan Compton, David Jurgens, and David Allen. 2014\.  Geotagging One Hundred Million Twitter Accounts with Total Variation Minimization. In _IEEE Conference on BigData_.
  * Conneau et al. (2017) Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017\.  Word Translation Without Parallel Data.  _arXiv preprint arXiv:1710.04087_ (2017).
  * de Silva and Compton (2014) Brian de Silva and Ryan Compton. 2014.  Prediction of Foreign Box Office Revenues Based on Wikipedia Page Activity.  _CoRR_ abs/1405.5924 (2014).  arXiv:1405.5924 http://arxiv.org/abs/1405.5924
  * DeFrancisco et al. (2013) Victoria Pruin DeFrancisco, Catherine Helen Palczewski, and Danielle D McGeough. 2013.  _Gender in communication: A critical introduction_.  Sage Publications.
  * Dehon and Brédart (2001) Hedwige Dehon and Serge Brédart. 2001.  An ’other-race’ effect in age estimation from faces.  _Perception_ 30, 9 (2001), 1107–1113.
  * Eckert (2008) Penelope Eckert. 2008\.  Variation and the indexical field.  _Journal of sociolinguistics_ 12, 4 (2008), 453–476.
  * Eckert and McConnell-Ginet (2003) Penelope Eckert and Sally McConnell-Ginet. 2003.  _Language and gender_.  Cambridge University Press.
  * European Commission ([n. d.]) European Commission. [n. d.].  NUTS—Nomenclature of Territorial Units for Statistics.  https://ec.europa.eu/eurostat/web/nuts/background.
  * Faruqui et al. (2016) Manaal Faruqui, Yulia Tsvetkov, Graham Neubig, and Chris Dyer. 2016.  Morphological inflection generation using character sequence to sequence learning. In _Proceedings of EMNLP_.
  * Gayo-Avello (2012) Daniel Gayo-Avello. 2012\.  ” I Wanted to Predict Elections with Twitter and all I got was this Lousy Paper”–A Balanced Survey on Election Prediction using Twitter Data.  _arXiv preprint arXiv:1204.6441_ (2012).
  * Gayo-Avello (2013) Daniel Gayo-Avello. 2013\.  A meta-analysis of state-of-the-art electoral prediction from Twitter data.  _Social Science Computer Review_ 31, 6 (2013), 649–679.
  * Generous et al. (2014) Nicholas Generous, Geoffrey Fairchild, Alina Deshpande, Sara Y. Del Valle, and Reid Priedhorsky. 2014.  Global Disease Monitoring and Forecasting with Wikipedia.  _PLOS Computational Biology_ 10, 11 (11 2014), 1–16.  https://doi.org/10.1371/journal.pcbi.1003892
  * Ginsberg et al. (2008) Jeremy Ginsberg, Matthew H. Mohebbi, Rajan S. Patel, Lynnette Brammer, Mark S. Smolinski, and Larry Brilliant. 2008.  Detecting influenza epidemics using search engine query data.  _Nature_ 457 (nov 2008), 1012.  https://doi.org/10.1038/nature0763410.1038/nature07634
  * Goot et al. (2018) Rob Goot, Nikola Ljubešić, Ian Matroos, Malvina Nissim, and Barbara Plank. 2018\.  Bleaching Text: Abstract Features for Cross-lingual Gender Prediction. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_. 383–389.
  * Goswami et al. (2009) Sumit Goswami, Sudeshna Sarkar, and Mayur Rustagi. 2009\.  Stylometric analysis of bloggers’ age and gender. In _Proceedings of ICWSM_.
  * Hale (2014) Scott A. Hale. 2014\.  Global Connectivity and Multilinguals in the Twitter Network. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’14)_. ACM, New York, NY, USA, 833–842.  https://doi.org/10.1145/2556288.2557203
  * Hecht and Stephens (2014) Brent Hecht and Monica Stephens. 2014.  A Tale of Cities: Urban Biases in Volunteered Geographic Information. In _Proceedings of ICWSM_.  https://www.aaai.org/ocs/index.php/ICWSM/ICWSM14/paper/view/8114
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Holt and Smith (1979) D. Holt and T. M. F. Smith. 1979.  Post Stratification.  _Journal of the Royal Statistical Society. Series A (General)_ 142, 1 (1979), 33–46.  http://www.jstor.org/stable/2344652
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017.  Densely Connected Convolutional Networks.. In _CVPR_ , Vol. 1. 3\.
  * Jaech and Ostendorf (2015) Aaron Jaech and Mari Ostendorf. 2015.  What your username says about you.  _arXiv preprint arXiv:1507.02045_ (2015).
  * Jansen (2010) Bernard J Jansen. 2010\.  _Use of the internet in higher-income households_.  Pew Research Center Washington, DC.
  * Johnson et al. (2017) Isaac Johnson, Connor McMahon, Johannes Schöning, and Brent Hecht. 2017.  The Effect of Population and Structural Biases on Social Media-based Algorithms: A Case Study in Geolocation Inference Across the Urban-Rural Spectrum. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_. ACM, 1167–1178.
  * Jung et al. (2017) Soon-Gyo Jung, Jisun An, Haewoon Kwak, Joni Salminen, and Bernard J Jansen. 2017.  Inferring Social Media Users’ Demographics from Profile Pictures: A Face++ Analysis on Twitter Users. In _Proceedings of The 17th International Conference on Electronic Business_. 140–145).
  * Jungherr (2017) Andreas Jungherr. 2017\.  Normalizing digital trace data.  _Digital Discussions: How Big Data Informs Political Communication_ (2017).
  * Jungherr et al. (2012) Andreas Jungherr, Pascal Jürgens, and Harald Schoen. 2012\.  Why the pirate party won the german election of 2009 or the trouble with predictions: Tumasjan, A., Sprenger, T. O., Sander, P. G., & Welpe, I. M. “Predicting elections with twitter: What 140 characters reveal about political sentiment”.  _Social science computer review_ 30, 2 (2012), 229–234.
  * Kendall et al. (1997) Shari Kendall, Deborah Tannen, et al. 1997\.  Gender and language in the workplace.  _Gender and Discourse. London: Sage_ (1997), 81–105.
  * Kim et al. (2016) Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016\.  Character-Aware Neural Language Models.. In _AAAI_. 2741–2749.
  * Knowles et al. (2016) Rebecca Knowles, Josh Carroll, and Mark Dredze. 2016\.  Demographer: Extremely simple name demographics. In _Proceedings of the First Workshop on NLP and Computational Social Science_. 108–113.
  * Krippendorff (2011) Klaus Krippendorff. 2011\.  Computing Krippendorff’s alpha-reliability.  _University of Pennsylvania Departmental papers (ASC)_ (2011).
  * Lakoff and Bucholtz (2004) Robin Tolmach Lakoff and Mary Bucholtz. 2004.  _Language and woman’s place: Text and commentaries_. Vol. 3.  Oxford University Press, USA.
  * Lamanna et al. (2018) Fabio Lamanna, Maxime Lenormand, María Henar Salas-Olmedo, Gustavo Romanillos, Bruno Gonçalves, and José J Ramasco. 2018\.  Immigrant community integration in world cities.  _PloS one_ 13, 3 (2018), e0191612.
  * Lazer et al. (2014) David Lazer, Ryan Kennedy, Gary King, and Alessandro Vespignani. 2014.  The Parable of Google Flu: Traps in Big Data Analysis.  _Science_ 343, 6167 (2014), 1203–1205.
  * Little and Rubin (2002) Roderick Little and Donald Rubin. 2002.  _Statistical analysis with missing data, Second edition_.  408 pages.
  * McCandless (2010) Michael McCandless. 2010\.  Accuracy and performance of Google’s compact language detector.  http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html.
  * McCorriston et al. (2015) James McCorriston, David Jurgens, and Derek Ruths. 2015\.  Organizations Are Users Too: Characterizing and Detecting the Presence of Organizations on Twitter.. In _Proceedings of ICWSM_. 650–653.
  * Mestyán et al. (2013) Márton Mestyán, Taha Yasseri, and János Kertész. 2013\.  Early Prediction of Movie Box Office Success Based on Wikipedia Activity Big Data.  _PLOS ONE_ 8, 8 (08 2013), 1–8.  https://doi.org/10.1371/journal.pone.0071226
  * Mislove et al. (2011) Alan Mislove, Sune Lehmann, Yong-Yeol Ahn, Jukka-Pekka Onnela, and J Niels Rosenquist. 2011\.  Understanding the Demographics of Twitter Users.  _Proceedings of ICWSM_ 11, 5th (2011), 25.
  * Nair and Hinton (2010) Vinod Nair and Geoffrey E Hinton. 2010.  Rectified linear units improve restricted boltzmann machines. In _Proceedings of the 27th international conference on machine learning (ICML-10)_. 807–814.
  * Nguyen et al. (2013) Dong Nguyen, Rilana Gravel, Dolf Trieschnigg, and Theo Meder. 2013.  “How Old Do You Think I Am?” A Study of Language and Age in Twitter.. In _Proceedings of ICWSM_.
  * Nguyen et al. (2011) Dong Nguyen, Noah A Smith, and Carolyn P Rosé. 2011\.  Author age prediction from text using linear regression. In _Proc. of the Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities_. Association for Computational Linguistics, 115–123.
  * Park et al. (2004) David K Park, Andrew Gelman, and Joseph Bafumi. 2004\.  Bayesian multilevel estimation with poststratification: state-level estimates from national polls.  _Political Analysis_ 12, 4 (2004), 375–385.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017\.  Automatic differentiation in PyTorch. In _NIPS-W_.
  * Rangel et al. (2018) Francisco Rangel, Paolo Rosso, Manuel Montes-y Gómez, Martin Potthast, and Benno Stein. 2018\.  Overview of the 6th author profiling task at PAN 2018: Multimodal gender identification in Twitter.  _Working Notes Papers of the CLEF_ (2018).
  * Rao et al. (2010) Delip Rao, David Yarowsky, Abhishek Shreevats, and Manaswi Gupta. 2010.  Classifying latent user attributes in twitter. In _Proc. of the 2nd International Workshop on Search and Mining User-generated Contents_. ACM, 37–44.
  * Reddi et al. (2018) Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. 2018\.  On the convergence of adam and beyond.  (2018).
  * Rosenthal and McKeown (2011) Sara Rosenthal and Kathleen McKeown. 2011.  Age prediction in blogs: A study of style, content, and online behavior in pre-and post-social media generations. In _Proc. ACL_. Association for Computational Linguistics, 763–772.
  * Rothe et al. (2016) Rasmus Rothe, Radu Timofte, and Luc Van Gool. 2016\.  Deep expectation of real and apparent age from a single image without facial landmarks.  _International Journal of Computer Vision (IJCV)_ (July 2016).
  * Ruths and Pfeffer (2014) Derek Ruths and Jürgen Pfeffer. 2014.  Social media for large studies of behavior.  _Science_ 346, 6213 (2014), 1063–1064.
  * Sap et al. (2014) Maarten Sap, Gregory Park, Johannes Eichstaedt, Margaret Kern, David Stillwell, Michal Kosinski, Lyle Ungar, and H. Andrew Schwartz. 2014\.  Developing Age and Gender Predictive Lexica over Social Media. In _Proc. EMNLP_. Association for Computational Linguistics, 1146–1151.
  * Särndal and Lundström (2005) Carl-Erik Särndal and Sixten Lundström. 2005.  _Estimation in Surveys with Nonresponse_.  John Wiley & Sons, Ltd, Chichester, UK. 1–199 pages.  https://doi.org/10.1002/0470011351
  * Schler et al. (2006) Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James W Pennebaker. 2006.  Effects of age and gender on blogging.. In _AAAI spring symposium: Computational approaches to analyzing weblogs_ , Vol. 6. 199–205.
  * Sloan (2017) Luke Sloan. 2017\.  Who Tweets in the United Kingdom? Profiling the Twitter Population Using the British Social Attitudes Survey 2015.  _Social Media + Society_ 3, 1 (2017), 2056305117698981\.
  * Tannen (1991) Deborah Tannen. 1991\.  _You just don’t understand: Women and men in conversation_.  Virago London.
  * Tannen (1993) Deborah Tannen. 1993\.  _Gender and conversational interaction_.  Oxford University Press.
  * UK Office for National Statistics (2015) UK Office for National Statistics 2015.  _Harmonised Concepts and Questions for Social Data Sources - Primary Principles_.  UK Office for National Statistics.  http://www.ons.gov.uk/ons/guide-method/harmonisation/primary-set-of-harmonised-concepts-and-questions/demographic-information--household-composition-and-relationships.pdf
  * United States Department of Education (2009) United States Department of Education 2009.  _Implementation guidelines: Measures and methods for the national reporting system for adult education_.  United States Department of Education.  http://www.air.org/sites/default/files/downloads/report/ImplementationGuidelines_0.pdf
  * Van Rijsbergen et al. (2014) Nicola Van Rijsbergen, Katarzyna Jaworska, Guillaume A Rousselet, and Philippe G Schyns. 2014\.  With age comes representational wisdom in social signals.  _Current Biology_ 24, 23 (2014), 2792–2796.
  * Wang et al. (2015) Wei Wang, David Rothschild, Sharad Goel, and Andrew Gelman. 2015.  Forecasting elections with non-representative polls.  _International Journal of Forecasting_ 31, 3 (2015), 980–991.
  * Wang and Jurgens (2018) Zijian Wang and David Jurgens. 2018.  It’s going to be okay: Measuring Access to Support in Online Communities. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. 33–45.
  * Wood-Doughty et al. (2018) Zach Wood-Doughty, Praateek Mahajan, and Mark Dredze. 2018\.  Johns Hopkins or johnny-hopkins: Classifying Individuals versus Organizations on Twitter. In _Proceedings of the Second Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media_. 56–61.
  * Wu and Wang (2007) Hua Wu and Haifeng Wang. 2007.  Pivot language approach for phrase-based statistical machine translation.  _Machine Translation_ 21, 3 (2007), 165–181.
  * Zagheni et al. (2017) Emilio Zagheni, Ingmar Weber, and Krishna Gummadi. 2017\.  Leveraging Facebook’s Advertising Platform to Monitor Stocks of Migrants.  _Population and Development Review_ 43, 4 (2017), 721–734.
  * Zhang et al. (2016) Jinxue Zhang, Xia Hu, Yanchao Zhang, and Huan Liu. 2016\.  Your Age Is No Secret: Inferring Microbloggers’ Ages via Content and Interaction Analysis.. In _Proceedings of ICWSM_. 476–485.