# Point-of-Interest Type Inference from Social Media Text

Danae Sánchez Villegasα Daniel Preoţiuc-Pietroβ Nikolaos Aletrasα
α Computer Science Department, University of Sheffield, UK
β Bloomberg
{dsanchezvillegas1<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Physical places help shape how we perceive the experiences we have there. We
study the relationship between social media text and the type of the place
from where it was posted, whether a park, restaurant, or someplace else. To
facilitate this, we introduce a novel data set of $\sim$200,000 English tweets
published from 2,761 different points-of-interest in the U.S., enriched with
place type information. We train classifiers to predict the type of the
location a tweet was sent from that reach a macro F1 of 43.67 across eight
classes and uncover the linguistic markers associated with each type of place.
The ability to predict semantic place information from a tweet has
applications in recommendation systems, personalization services and cultural
geography.111Data is available here: https://archive.org/details/poi-data

##  1 Introduction

Social networks such as Twitter allow users to share information about
different aspects of their lives including feelings and experiences from
places that they visit, from local restaurants to sport stadiums and parks.
Feelings and emotions triggered by performing an activity or living an
experience in a Point-of-Interest (POI) can give a glimpse of the atmosphere
in that place (Tanasescu et al., 2013).

In particular, the language used in posts from POIs is an important component
that contributes toward the place’s identity and has been extensively studied
in the context of social and cultural geography (Tuan, 1991; Scollon and
Scollon, 2003; Benwell and Stokoe, 2006). Social media posts from a particular
location are usually focused on the person posting the content, rather than on
providing explicit information about the place. Table 1 displays example
Twitter posts from different POIs. Users express their feelings related to a
certain place (‘this places gives me war flashbacks’), comments and thoughts
associated with the place they are in (‘few of us dressed appropriately’) or
activities they are performing (‘leaving the news station’, ‘on the way to the
APCE Annual’).

In this paper, we aim to study the language that people on Twitter use to
share information about a specific place they are visiting. Thus, we define
the prediction of a POI type given a post (i.e. tweet) as a multi-class
classification task using only information available at posting time. Given
the text from a user’s post, our goal is to predict the correct type of the
location it was posted, e.g. park, bar or shop. Inferring the type of place
from a user’s post using linguistic information, is useful for cultural
geographers to study a place’s identity (Tuan, 1991) and has downstream
geosocial applications such as POI visualisation (McKenzie et al., 2015) and
recommendation (Alazzawi et al., 2012; Yuan et al., 2013; Preoţiuc-Pietro and
Cohn, 2013; Gao et al., 2015).

Predicting the type of a POI is inherently different to predicting the POI
type from comments or reviews. The role of the latter is to provide opinions
or descriptions of the places, rather than the activities and feelings of the
user posting the text (McKenzie et al., 2015), as illustrated in Table 1. This
is also different, albeit related, to the popular task of geolocation
prediction Cheng et al. (2010); Eisenstein et al. (2010); Han et al. (2012);
Roller et al. (2012); Rahimi et al. (2015); Dredze et al. (2016), as this aims
to infer the exact geographical location of a post using language variation
and geographical cues rather than inferring the place’s type. Our task aims to
uncover the geographic agnostic features associated with POIs of different
types.

Category | Sample Tweet | Train | Dev | Test | Tokens
---|---|---|---|---|---
Arts & Entertainment | i’m back in central park . this place gives me war flashbacks now lol | 40,417 | 4,755 | 5,284 | 14.41
College & University | currently visiting my dream school  | 21,275 | 2,418 | 2,884 | 15.52
Food | Some Breakfast, it’s only right! #LA | 6,676 | 869 | 724 | 14.34
Great Outdoors |  | Sorry Southport, Billy is dishing out donuts at #donutfest today. See you
---
next weekend!
27,763 | 4,173 | 3,653 | 13.49
Nightlife Spot |  | Chicago really needs to step up their Aloha shirt game. Only a few of us
---
dressed “appropriately” tonight. :)
5,545 | 876 | 656 | 15.46
Professional & Other Places | Leaving the news station after a long day | 30,640 | 3,381 | 3,762 | 16.46
Shop & Service | Came to get an old fashioned tape measures and a button for my coat | 8,285 | 886 | 812 | 15.31
Travel & Transport |  | Shoutout to anyone currently on the way to the APCE Annual Event in
---
Louisville, KY! #APCE2018
16,428 | 2,201 | 1,872 | 14.88

Table 1: Place categories with sample tweets and data set statistics.

Our contributions are as follows: (1) We provide the first study of POI type
prediction in computational linguistics; (2) A large data set made out of
tweets linked to particular POI categories; (3) Linguistic and temporal
analyses related to the place the text was posted from; (4) Predictive models
using text and temporal information reaching up to 43.67 F1 across eight
different POI types.

##  2 Point-of-Interest Type Data

We define the POI type prediction as a multi-class classification task
performed at the social media post level. Given a post T, defined as a
sequence of tokens $T=\\{t_{1},...,t_{n}\\}$, the goal is to label T as one of
the $M$ POI categories. We create a novel data set for POI type prediction
containing text and the location type it was posted from as, to the best of
our knowledge, no such data set is available. We use Twitter as our data
source because it contains a large variety of linguistic information such as
expression of thoughts, opinions and emotions Java et al. (2007); Kouloumpis
et al. (2011).

###  2.1 Types of POIs

Foursquare is a location data platform that manages ‘Places by Foursquare’, a
database of more than 105 million POIs worldwide. The place information
includes verified metadata such as name, geo-coordinates and categories as
well as other user-sourced metadata such as tags, comments or photos. POIs are
organized into 9 top level primary categories with multiple subcategories. We
only focus on 8 primary top-level POI categories since the category
‘Residence’ has a considerably smaller number of tweets compared to the other
categories (0.78% tweets from the total). We leave finer-grained place
category inference as well as using other metadata for future work since the
scope of this work is to study the language of posts associated with semantic
type places.

###  2.2 Associating Tweets with POI Types

Twitter users can tag their tweets to the locations they are posted from by
linking to Foursquare places.222https://developer.foursquare.com/places In
this way, we collect tweets assigned to the POIs and associated metadata (see
Table 1). We select a broad range of locations for our experiments. There is
no public list of all Foursquare locations that can be used through Twitter
and can be programmatically accessed. Hence, in order to discover Foursquare
places that are actually used in tweets, we start with all places found in a
1% sample of the Twitter feed between 31 July 2016 and 24 January 2017 leading
us to a total of 9,125 different places. Then, we collect all tweets from
these places between 17 August 2016 and 1 March 2018 using the Twitter Search
API333https://developer.twitter.com/en/docs/tweets/search/guides/tweets-by-
place. We collect the place metadata from the public Foursquare Venues API.
This resulted in a total data set of 1,648,963 tweets tagged to a Foursquare
place. In order to extract metadata about each location, we crawled the
Twitter website to identify the corresponding Foursquare Place ID of each
Twitter place. Then, we used the public Foursquare Venues
API444https://developer.foursquare.com/overview/venues.html to download all
the place metadata.

###  2.3 Data Filtering

To limit variation in our data, we filter out all non-English tweets and non-
US places, as these were very limited in number. We keep POIs with at least 20
tweets and randomly subsample 100 tweets from POIs with more tweets to avoid
skewing our data. Our final data set consists of 196,235 tweets from 2,761
POIs.

###  2.4 Data Split

We create our data split at a location-level to ensure that our models are
robust and generalize to locations held-out in training. We split the
locations in train (80%), development (10%) and test (10%) sets and assign
tweets to one of the three splits based on the location they were posted from
(see Table 1 for detailed statistics).

###  2.5 Text Processing

We lower-case text and replace all URLs and mentions of users with
placeholders. We preserve emoticons and punctuation and replace tokens that
appear in less than five tweets with an ‘unknown’ token. We tokenize text
using a Twitter-aware tokenizer Schwartz et al. (2017).

Arts | College | Food | Outdoors | Nightlife | Professional | Shop | Travel
---|---|---|---|---|---|---|---
Feature | $\chi^{2}$ | Feature | $\chi^{2}$ | Feature | $\chi^{2}$ | Feature | $\chi^{2}$ | Feature | $\chi^{2}$ | Feature | $\chi^{2}$ | Feature | $\chi^{2}$ | Feature | $\chi^{2}$
concert | 167.20 | campus | 298.74 | chicken | 375.52 | beach | 591.81 | #craftbeer | 425.97 | school | 87.46 | mall | 462.03 | airport | 394.20
museum | 152.14 | college | 266.63 | #nola | 340.64 |  | 239.00 |  | 311.68 | students | 79.93 | store | 403.00 |  | 343.30
show | 134.39 | university | 155.65 | lunch | 255.98 | hike | 227.91 | beer | 203.57 | grade | 66.05 | shopping | 359.00 | flight | 292.94
night | 104.48 | class | 112.23 | fried | 216.49 | lake | 193.58 | bar | 93.90 | vote | 65.80 | shop | 132.39 | hotel | 168.38
tonight | 80.76 | semester | 103.19 | dinner | 203.65 | park | 165.92 |  | 67.00 | our | 63.12 |  | 126.07 | conference | 141.74
game | 73.56 | football | 59.24 |  | 195.41 | island | 151.45 |  | 56.94 | jv | 60.64 |  | 95.32 | landed | 118.05
art | 69.77 | student | 57.86 | pizza | 190.83 | sunset | 142.44 | dj | 56.56 | church | 52.97 | apple | 88.74 | plane | 88.42
USER | 66.14 | classes | 57.37 | shrimp | 188.77 | hiking | 137.74 | tonight | 53.39 | hs | 50.63 | market | 76.60 | bound | 78.43
zoo | 66.09 | students | 56.98 |  | 179.39 | beautiful | 109.45 | ale | 52.62 | senior | 50.05 | auto | 73.52 | heading | 62.09
baseball | 62.90 | camp | 44.19 |  | 151.00 | bridge | 108.56 | party | 51.14 | ss | 44.46 | stock | 72.31 | headed | 57.12

Table 2: Unigrams associated with each category, sorted by $\chi^{2}$ value
computed between the normalized frequency of each feature and the category
label across all tweets in the training set ($p<0.001$).

##  3 Analysis

We first analyze our data set to understand the relationship between location
type, language and posting time.

###  3.1 Linguistic Analysis

We analyze the linguistic features specific to each category by ranking
unigrams that appear in at least 5 different locations, such that these are
representative of the larger POI category rather than a few specific places.
Features are normalized to sum up to unit for each tweet, then we compute the
(Pearson) $\chi^{2}$ coefficient independently between its distribution across
posts and the binary category label of the post similar to the approach
followed by Maronikolakis et al. (2020) and Preoţiuc-Pietro et al. (2019).
Table 2 presents the top unigram features for each category.

We note that most top unigrams specific of a category naturally refer to types
of places (e.g. ‘campus’, ‘beach’, ‘mall’, ‘airport’) that are part of that
category. All categories also contain words that refer to activities that the
poster of the tweet is performing or observing while at a location (e.g.
‘camp’ and ‘football’ for College, ‘concert’ and ‘show’ for Arts &
Entertainment, ‘party’ for Nightlife Spot, ‘landed’ for Travel & Transport,
‘hike’ for Greater Outdoors). Nightlife Spot and Food categories are
represented by types of food or drinks that are typically consumed at these
locations. Beyond these typical associations, we highlight that usernames are
more likely mentioned in the Arts & Entertainment category, usually indicating
activities involving groups of users, emojis indicative of the user state
(e.g. happy emoji in Food places) and adjectives indicative of the user’s
surroundings (e.g. ‘beautiful’ in Greater Outdoors places). Finally, we also
uncover words indicative of the time the user is at a place, such as ‘tonight’
for Arts & Entertainment, ‘sunset’ for the Greater Outdoors and ‘night’ for
Nightlife Spots and Arts & Entertainment.

###  3.2 Temporal Analysis

We further examine the relationship between the time a tweet was posted and
the POI type it was posted from. Figure 1 shows the percentage of tweets by
day of week (top) and hour of day (bottom).

We observe that tweets posted from the ‘Professional & Other Places’, ‘Travel
& Transport’ and ‘College & University’ categories are more prevalent on
weekdays, peaking on Wednesday, while on weekends more tweets are posted from
the ‘Great Outdoors’, ‘Arts & Entertainment’, ‘Nightlife & Spot’ and ‘Food’
categories when people focus less on professional activities and dedicate more
time to leisure as expected. The hour of day pattern follows the daily human
activity rhythm, but the differences between categories are less prominent,
perhaps with the exception of the ‘Arts & Entertainment’ category peaks around
8PM and ‘Nightlife Spots’ that see a higher percent of tweets in the early
hours of the day (between 1-5am) than other categories.

Figure 1: Percentage of tweets by day of week (top) and by hour of day
(bottom).

##  4 Predicting POI Types of Tweets

###  4.1 Methods

#### Logistic Regression

We first experiment with logistic regression using a standard bag of n-grams
representation of the tweet (LR-W), including unigrams to trigrams weighted
using TF-IDF. We identified in the analysis section that temporal information
about the tweet may be useful for classification. Hence, to add temporal
information extracted from a tweet, we create a 31-dimensional vector encoding
the hour of the day and the day of the week it was sent from. We experiment
with only using the temporal features (LR-T) and in combination with the text
features (LR-W+T). We use L1 regularization Hoerl and Kennard (1970) with
hyperparameter $\alpha=.01$ (selected based on dev set from {.001, .01, .1}).

#### BiLSTM

We train models based on bidirectional Long-Short Term Memory (LSTM) networks
Hochreiter and Schmidhuber (1997), which are popular in text classification
tasks. Tokens in a tweet are mapped to embeddings and passed through the two
LSTM networks, each processing the input in opposite directions. The outputs
are concatenated and passed to the output layer using a softmax activation
function (BiLSTM). We extend the BiLSTM to encode temporal one-hot
representation by: (a) concatenating the temporal vector to the tweet
representation (BiLSTM-TC); and (b) projecting the time vector into a dense
representation using a fully connected layer which is added to the tweet
representation before passing it through the output layer using a softmax
activation function (BiLSTM-TS). We use 200-dimensional GloVe embeddings
Pennington et al. (2014) pre-trained on Twitter data. The maximum sequence
length is set to 26, covering 95% of the tweets in the training set. The LSTM
size is $h$ = 32 where $h\in\\{32,64,100,300\\}$ with dropout $d$ = 0.5 where
$d\in\\{.2,.5\\}$. We use Adam Kingma and Ba (2014) with default learning
rate, minimizing cross-entropy using a batch size of 32 over 10 epochs with
early stopping.

#### BERT

Bidirectional Encoder Representations from Transformers (BERT) is a pre-
trained language model based on transformer networks Vaswani et al. (2017);
Devlin et al. (2019). BERT consists of multiple multi-head attention layers to
learn bidirectional embeddings for input tokens. The model is trained on
masked language modeling, where a fraction of the input tokens in a given
sequence is replaced with a mask token, and the model attempts to predict the
masked tokens based on the context provided by the non-masked tokens in the
sequence. We fine-tune BERT for predicting the POI type of a tweet by adding a
classification layer with softmax activation function on top of the
Transformer output for the ‘classification’ $[CLS]$ token (BERT). Similarly to
the previous models, we extend BERT to make use of the time vector in two
ways, by concatenating (BERT-TC), and by adding it (BERT-TS) to the output of
the Transformer before passing it to through the classification layer with
softmax activation function. We use the base model (12-layer, 110M parameters)
trained on lower-cased English text. We fine-tune it for 2 epochs with a
learning rate $l=2e^{-5}$, $l\in\\{2e^{-5},3e^{-5},5e^{-5}\\}$ and a batch
size of 32.

###  4.2 Results

Table 3 presents the results of POI type prediction measured using accuracy,
macro F1, precision and recall across three runs. In general, we observe that
we can predict POI types of tweets with good accuracy, considering the
classification is across eight relatively well balanced classes.

Best results are obtained using BERT-based models (BERT, BERT-TC and BERT-TS),
with the highest accuracy of 49.17 (compared to 26.89 majority class) and
highest macro-F1 of 43.67 (compared to 12.64 random). We observe that BERT
models outperform both BiLSTM and linear methods across all metrics, with over
4% improvement in accuracy and 5 points F1. The BiLSTM models perform
marginally better than the linear models. Temporal features alone are
marginally useful when models are evaluated using accuracy (+0.28 BERT, +0.34
for BiLSTMs, +0.69 for LR) and perform similarly on F1, with the notable
exception of the BiLSTM models. We find that adding these features is more
beneficial than concatenating them, with concatenation hurting performance on
accuracy for both BiLSTM and BERT.

Figure 2 shows the confusion matrix of our best performing model, BERT,
according to the macro-F1 score. The confusion matrix is normalized over the
actual values (rows). The category ‘Arts & Entertainment‘ has the greatest
percentage (62%) of correctly classified tweets, followed by the ‘Great
Outdoors‘ category with 54%, and the ‘College & University‘ category with 44%.
On the other hand, the categories ‘Nightlife Spot‘ and ‘Shop & Service‘ have
the lowest results, where 30% of the tweets predicted as each of these classes
is correctly classified. Most common error is when the model classifies tweets
from the category ‘College & University’ as ‘Professional & Other Places’, as
tweets from these places contain similar terms such as ‘students’ or ‘class’.

Model | Acc | F1 | P | R
---|---|---|---|---
Major. Class | 26.89 | 5.30 | 3.36 | 12.50
Random | 13.63 | 12.64 | 13.63 | 15.68
LR-T | 27.93 | 14.01 | 15.78 | 16.06
LR-W | 43.04 | 37.33 | 37.06 | 38.03
LR-W+T | 43.73 | 37.83 | 37.68 | 38.37
BiLSTM | 44.38 | 35.77 | 45.29 | 33.78
BiLSTM-TC | 44.01 | 38.07 | 41.51 | 36.46
BiLSTM-TS | 44.72 | 38.26 | 42.91 | 36.30
BERT | 48.89 | 43.67 | 48.44 | 41.33
BERT-TC | 46.13 | 41.19 | 46.81 | 39.03
BERT-TS | 49.17 | 43.47 | 48.40 | 41.26
Table 3: Accuracy (Acc), Macro-F1 Score (F1), Precision macro (P), and Recall
macro (R) for POI type prediction (all std. dev $<$ 0.01). Best results are in
bold. Figure 2: Confusion Matrix of the best performing model (BERT).

##  5 Conclusion

We presented the first study on predicting the POI type a social media message
was posted from and developed a large-scale data set with tweets mapped to
their POI category. We conducted an analysis to uncover features specific to
place type and trained predictive models to infer the POI category using only
tweet text and posting time with accuracy close to 50% across eight
categories. Future work will focus on using other modalities such as network
Aletras and Chamberlain (2018); Tsakalidis et al. (2018) or image information
Vempala and Preoţiuc-Pietro (2019); Alikhani et al. (2019) and prediction at a
more granular level of POI types.

## Acknowledgments

DSV is supported by the Centre for Doctoral Training in Speech and Language
Technologies (SLT) and their Applications funded by the UK Research and
Innovation grant EP/S023062/1. NA is supported by ESRC grant ES/T012714/1.

## References

  * Alazzawi et al. (2012) Ahmed N Alazzawi, Alia I Abdelmoty, and Christopher B Jones. 2012.  What can I do there? Towards the automatic discovery of place-related services and activities.  International Journal of Geographical Information Science 26(2):345–364.
  * Aletras and Chamberlain (2018) Nikolaos Aletras and Benjamin Paul Chamberlain. 2018.  Predicting Twitter user socioeconomic attributes with network and language information.  In Proceedings of the 29th Conference on Hypertext and Social Media, pages 20–24.
  * Alikhani et al. (2019) Malihe Alikhani, Sreyasi Nag Chowdhury, Gerard de Melo, and Matthew Stone. 2019\.  CITE: A corpus of image-text discourse relations.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, pages 570–575.
  * Benwell and Stokoe (2006) Bethan Benwell and Elizabeth Stokoe. 2006.  Discourse and identity.  Edinburgh University Press.
  * Cheng et al. (2010) Zhiyuan Cheng, James Caverlee, and Kyumin Lee. 2010.  You Are Where You Tweet: A Content-Based Approach to Geo-Locating Twitter Users.  In Proceedings of the 19th ACM International Conference on Information and Knowledge Management. CIKM ’10, pages 759–768.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, pages 4171–4186.
  * Dredze et al. (2016) Mark Dredze, Miles Osborne, and Prabhanjan Kambadur. 2016.  Geolocation for twitter: Timing matters.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, pages 1064–1069.
  * Eisenstein et al. (2010) Jacob Eisenstein, Brendan O’Connor, Noah A. Smith, and Eric P. Xing. 2010.  A latent variable model for geographic lexical variation.  In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Cambridge, MA, pages 1277–1287.
  * Gao et al. (2015) Huiji Gao, Jiliang Tang, Xia Hu, and Huan Liu. 2015.  Content-aware point of interest recommendation on location-based social networks.  In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. AAAI, pages 1721–1727.
  * Han et al. (2012) Bo Han, Paul Cook, and Timothy Baldwin. 2012.  Geolocation prediction in social media data by finding location indicative words.  In Proceedings of COLING 2012. The COLING 2012 Organizing Committee, Mumbai, India, pages 1045–1062.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  Neural computation 9(8):1735–1780.
  * Hoerl and Kennard (1970) Arthur E Hoerl and Robert W Kennard. 1970.  Ridge regression: Biased estimation for nonorthogonal problems.  Technometrics 12(1):55–67.
  * Java et al. (2007) Akshay Java, Xiaodan Song, Tim Finin, and Belle Tseng. 2007.  Why we Twitter: understanding microblogging usage and communities.  In Proceedings of the 9th WebKDD and 1st SNA-KDD 2007 Workshop on Web Mining and Social Network Analysis. pages 56–65.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980 .
  * Kouloumpis et al. (2011) Efthymios Kouloumpis, Theresa Wilson, and Johanna D Moore. 2011.  Twitter sentiment analysis: The good the bad and the omg!  In Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media. ICWSM, pages 538–541.
  * Maronikolakis et al. (2020) Antonios Maronikolakis, Danae Sánchez Villegas, Daniel Preotiuc-Pietro, and Nikolaos Aletras. 2020.  Analyzing political parody in social media.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 4373–4384.
  * McKenzie et al. (2015) Grant McKenzie, Krzysztof Janowicz, Song Gao, Jiue-An Yang, and Yingjie Hu. 2015\.  POI pulse: A multi-granular, semantic signature–based information observatory for the interactive visualization of big geosocial data.  Cartographica: The International Journal for Geographic Information and Geovisualization 50(2):71–85.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, pages 1532–1543.
  * Preoţiuc-Pietro and Cohn (2013) Daniel Preoţiuc-Pietro and Trevor Cohn. 2013.  Mining user behaviours: a study of check-in patterns in location based social networks.  In Proceedings of the 5th annual ACM Web Science Conference. Web Science, pages 306–315.
  * Preoţiuc-Pietro et al. (2019) Daniel Preoţiuc-Pietro, Mihaela Gaman, and Nikolaos Aletras. 2019.  Automatically identifying complaints in social media.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 5008–5019.
  * Rahimi et al. (2015) Afshin Rahimi, Duy Vu, Trevor Cohn, and Timothy Baldwin. 2015.  Exploiting text and network context for geolocation of social media users.  In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Denver, Colorado, pages 1362–1367.
  * Roller et al. (2012) Stephen Roller, Michael Speriosu, Sarat Rallapalli, Benjamin Wing, and Jason Baldridge. 2012.  Supervised text-based geolocation using language models on an adaptive grid.  In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguistics, Jeju Island, Korea, pages 1500–1510.
  * Schwartz et al. (2017) H. Andrew Schwartz, Salvatore Giorgi, Maarten Sap, Patrick Crutchley, Lyle Ungar, and Johannes Eichstaedt. 2017.  DLATK: Differential language analysis ToolKit.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Copenhagen, Denmark, pages 55–60.
  * Scollon and Scollon (2003) Ron Scollon and Suzie Wong Scollon. 2003.  Discourses in place: Language in the material world.  Routledge.
  * Tanasescu et al. (2013) Vlad Tanasescu, Christopher B Jones, Gualtiero Colombo, Martin J Chorley, Stuart M Allen, and Roger M Whitaker. 2013.  The personality of venues: Places and the five-factors (’big five’) model of personality.  In Fourth IEEE International Conference on Computing for Geospatial Research and Application. pages 76–81.
  * Tsakalidis et al. (2018) Adam Tsakalidis, Nikolaos Aletras, Alexandra I Cristea, and Maria Liakata. 2018\.  Nowcasting the stance of social media users in a sudden vote: The case of the Greek Referendum.  In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. pages 367–376.
  * Tuan (1991) Yi-Fu Tuan. 1991.  Language and the making of place: A narrative-descriptive approach.  Annals of the Association of American geographers 81(4):684–696.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In Advances in Neural Information Processing Systems. pages 5998–6008.
  * Vempala and Preoţiuc-Pietro (2019) Alakananda Vempala and Daniel Preoţiuc-Pietro. 2019.  Categorizing and inferring the relationship between the text and image of twitter posts.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, pages 2830–2840.
  * Yuan et al. (2013) Quan Yuan, Gao Cong, Zongyang Ma, Aixin Sun, and Nadia Magnenat Thalmann. 2013.  Time-aware point-of-interest recommendation.  In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. pages 363–372.


# Towards Robustness to Label Noise in Text Classification via Noise Modeling

Siddhant Garg  Amazon Alexa AI<EMAIL_ADDRESS>,  Goutham Ramakrishnan
Health at Scale Corporation<EMAIL_ADDRESS>and  Varun Thumbe  KLA
Corporation<EMAIL_ADDRESS>
(2021)

###### Abstract.

Large datasets in NLP tend to suffer from noisy labels due to erroneous
automatic and human annotation procedures. We study the problem of text
classification with label noise, and aim to capture this noise through an
auxiliary noise model over the classifier. We first assign a probability score
to each training sample of having a clean or noisy label, using a two-
component beta mixture model fitted on the training losses at an early epoch.
Using this, we jointly train the classifier and the noise model through a
novel de-noising loss having two components: (i) cross-entropy of the noise
model prediction with the input label, and (ii) cross-entropy of the
classifier prediction with the input label, weighted by the probability of the
sample having a clean label. Our empirical evaluation on two text
classification tasks and two types of label noise: random and input-
conditional, shows that our approach can improve classification accuracy, and
prevent over-fitting to the noise.

Label Noise; Noise Model; Robustness; Text Classification; NLP

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
30th ACM International Conference on Information and Knowledge Management;
November 1–5, 2021; Virtual Event, QLD, Australia††booktitle: Proceedings of
the 30th ACM International Conference on Information and Knowledge Management
(CIKM ’21), November 1–5, 2021, Virtual Event, QLD, Australia††price:
15.00††doi: 10.1145/3459637.3482204††isbn: 978-1-4503-8446-9/21/11††ccs:
Computing methodologies Natural language processing

##  1\. Introduction

Training modern ML models requires access to large accurately labeled
datasets, which are difficult to obtain due to errors in automatic or human
annotation techniques (Wang et al., 2018; Zlateski et al., 2018). Recent
studies (Zhang et al., 2016) have shown that neural models can over-fit on
noisy labels and thereby not generalize well. Human annotations for language
tasks have been popularly obtained from platforms like Amazon Mechanical Turk
(Ipeirotis et al., 2010), resulting in noisy labels due to ambiguity of the
correct label (Zhan et al., 2019), annotation speed, human error, inexperience
of annotator, etc. While learning with noisy labels has been extensively
studied in computer vision (Reed et al., 2015; Zhang et al., 2018;
Thulasidasan et al., 2019), the corresponding progress in NLP has been
limited. With the increasing size of NLP datasets, noisy labels are likely to
affect several practical applications (Agarwal et al., 2007).

Figure 1.  Illustration of our approach, with an auxiliary noise model $N_{M}$
on top of the classifier $M$. We jointly train the models using a de-noising
loss $\mathcal{L}_{DN}$, and use the clean label prediction $\hat{y}^{(c)}$
during inference.

In this paper, we consider the problem of text classification, and capture the
label noise through an auxiliary noise model (See Fig. 1). We leverage the
finding of learning on clean labels being easier than on noisy labels (Arazo
et al., 2019), and first fit a two-component beta-mixture model (BMM) on the
training losses from the classifier at an early epoch. Using this, we assign a
probability score to every training sample of having a clean or noisy label.
We then jointly train the classifier and the noise model by selectively
guiding the former’s prediction for samples with high probability scores of
having clean labels. More specifically, we propose a novel de-noising loss
having two components: (i) cross-entropy of the noise model prediction with
the input label and (ii) cross-entropy of the classifier prediction with the
input label, weighted by the probability of the sample having a clean label.
Our formulation constrains the noise model to learn the label noise, and the
classifier to learn a good representation for the prediction task from the
clean samples. At inference time, we remove the noise model and use the
predictions from the classifier.

(a) Epoch 1

(b) Epoch 9

(c) Epoch 30

(d) Fitting a BMM at Epoch 9

Figure 2.  (a), (b) and (c) show the histogram of the training loss from the
classifier for the train split (with Clean/ Noisy label) at different epochs
(word-LSTM on TREC with 40% random noise). Initially (see (a)), the losses are
high for all data points (both clean and noisy labels). A fully trained model
achieves low losses on both clean and noisy data points, indicating over-
fitting to the noise, as seen in (c). However, at an early epoch of training,
we observe that samples with clean labels have lower losses while those with
noisy labels have high losses, leading to the formation of two clusters as
seen in (b). We fit a beta-mixture model at this intermediate epoch to
estimate the probability of a sample having a clean or noisy label, as shown
in (d).

Most existing works on learning with noisy labels assume that the label noise
is independent of the input and only conditional on the true label. Text
annotation complexity has been shown to depend on the lexical, syntactic and
semantic input features (Joshi et al., 2014) and not solely on the true label.
The noise model in our formulation can capture an arbitrary noise function,
which may depend on both the input and the original label, taking as input a
contextualized input representation from the classifier. While de-noising the
classifier for sophisticated noise functions is a challenging problem, we take
the first step towards capturing a real world setting.

We evaluate our approach on two popular datasets, for two different types of
label noise: random and input-conditional; at different noise levels. Across
two model architectures, our approach results in improved model accuracies
over the baseline, while preventing over-fitting to the label noise.

##  2\. Related Work

There have been several research works that have studied the problem of
combating label noise in computer vision (Frénay and Verleysen, 2014; Jiang et
al., 2018, 2019) through techniques like bootstrapping (Reed et al., 2015),
mixup (Zhang et al., 2018), etc. Applying techniques like mixup (convex
combinations of pairs of samples) for textual inputs is challenging due to the
discrete nature of the input space and retaining overall semantics. In natural
language processing, Agarwal et al. (2007) study the effect of different kinds
of noise on text classification, Ardehaly and Culotta (2018) study social
media text classification using label proportion (LLP) models, and Malik and
Bhardwaj (2011) automatically validate noisy labels using high-quality class
labels. Jindal et al. (2019) capture random label noise via a
$\ell_{2}$-regularized matrix learned on the classifier logits. Our work
differs from this as we i) use a neural network noise model over
contextualized embeddings from the classifier, with (ii) a new de-noising loss
to explicitly guide learning. It is difficult to draw a distinction between
noisy labels, and outliers which are hard to learn from. While several works
perform outlier detection (Goodman et al., 2016; Larson et al., 2019) to
discard these samples while learning the classifier, we utilise the noisy data
in addition to the clean data for improving performance.

##  3\. Methodology

Problem Setting Let
$(X,Y^{(c)}){=}\\{(x_{1},y_{1}^{(c)}),\dots,(x_{N},y_{N}^{(c)})\\}$ denote
clean training samples from a distribution
$\mathcal{D}{=}{\mathcal{X}}{\times}{\mathcal{Y}}$. We assume a function
$\mathcal{F}:{\mathcal{X}}{\times}{\mathcal{Y}\rightarrow\mathcal{Y}}$ that
introduces noise in labels $Y^{(c)}$. We apply $\mathcal{F}$ on $(X,Y^{(c)})$
to obtain the noisy training data
$(X,Y^{(n)})=\\{(x_{1},y_{1}^{(n)}),\dots,(x_{N},y_{N}^{(n)})\\}$.
$(X,Y^{(n)})$ contains a combination of clean samples (whose original label is
retained $y^{(n)}{=}y^{(c)}$) and noisy samples (whose original label is
corrupted $y^{(n)}{\neq}y^{(c)}$). Let $(X_{T},Y_{T})$ be a test set sampled
from the clean distribution $\mathcal{D}$. Our goal is to learn a classifier
model $\mathcal{M}:\mathcal{X}{\rightarrow}\mathcal{Y}$ trained on the noisy
data $(X,Y^{(n)})$, which generalizes well on $(X_{T},Y_{T})$. Note that we do
not have access to the clean labels $Y^{(c)}$ at any point during training.

Modeling Noise Function $\mathcal{F}$ We propose to capture $\mathcal{F}$
using an auxiliary noise model $N_{M}$ on top of the classifier model $M$, as
shown in Fig. 1. For an input $x$, a representation $R_{M}(x)$, derived from
$M$, is fed to $N_{M}$. $R_{M}(x)$ can typically be the contextualized input
embedding from the penultimate layer of $M$. We denote the predictions from
$M$ and $N_{M}$ to be $\hat{y}^{(c)}$(clean prediction) and
$\hat{y}^{(n)}$(noisy prediction) respectively. The clean prediction
$\hat{y}^{(c)}$ is used for inference.

Model |  |  |  TREC (word-LSTM: 93.8, word-CNN: 92.6) |  |  AG-News (word-LSTM: 92.5, word-CNN: 91.5)
---|---|---|---|---|---
Noise % |  | 10 | 20 | 30 | 40 | 50 |  | 10 | 20 | 30 | 40 | 50
word LSTM | Baseline |  | 88.0 (-0.6) | 89.4 (-9.6) | 83.4 (-19.0) | 79.6 (-24.8) | 77.6 (-27.2) |  |  91.9 (-1.7) |  91.3 (-1.5) | 90.5 (-2.5) | 89.3 (-3.7) | 88.6 (-10.5)
$\mathcal{L}_{DN{-}H}$ |  | 92.2 (-0.6) |  90.2 (-0.2) |  88.8 (-0.4) | 83.0 (-3.6) | 82.4 (0.0) |  | 91.5 (-0.1) | 90.6 (-0.1) | 90.8 (-0.1) |  90.3 (0.0) |  89.0 (-0.1)
$\mathcal{L}_{DN{-}S}$ |  |  92.4 (-1.0) | 90.0 (-0.2) | 87.4 (-2) |  83.4 (-1.0) |  82.6 (-8.4) |  | 91.8 (-0.3) | 90.8 (-0.2) |  91.0 (-0.1) | 90.3 (-0.1) | 88.6 (-0.1)
word CNN | Baseline |  | 88.8 (-1.4) | 89.2 (-1.8) | 84.8 (-8.0) |  82.2 (-15.0) | 77.6 (-16.0) |  | 90.9 (-2.7) | 90.6 (-6.2) | 89.3 (-10.2) |  89.2 (-17.9) |  87.4 (-25.2)
$\mathcal{L}_{DN{-}H}$ |  | 91 (-0.2) | 90.8 (-0.2) |  89.4 (-1.0) | 81.4 (0.0) |  81.4 (-4.8) |  |  91.3 (-0.2) |  91.0 (-0.4) |  90.3 (-0.3) | 88.3 (-3.2) | 86.6 (-3.5)
$\mathcal{L}_{DN{-}S}$ |  |  92.2 (-1.4) |  91.8 (-2.0) | 88.8 (-2.8) | 77.0 (-2.4) | 77.2 (-7.0) |  | 90.9 (0.0) | 90.4 (-0.1) | 88.7 (-1.1) | 86.6 (-3.5) | 84.5 (-10.2)

Table 1.  Results from experiments using random noise. Here for A(B): A refers
to the Best model accuracy while B refers to (Last-Best) accuracy. The models
with highest Best accuracies are in bold. For each noise $\%$, the least and
most reductions in Last accuracy are highlighted in green and red. Baseline
($0\%$ noise) reported beside dataset.

###  3.1. Estimating clean/noisy label using BMM

It has been empirically observed that classifiers that capture input semantics
do not fit the noise before significantly learning from the clean samples
(Arazo et al., 2019). For a classifier trained using a cross entropy
loss($\mathcal{L}_{CE}$) on the noisy dataset, this can be exploited to
cluster the input samples as being clean/noisy in an unsupervised manner.
Initially the training loss on both clean and noisy samples is large, and
after a few training epochs, the loss of majority of the clean samples
reduces. Since the loss of the noisy samples is still large, this segregates
the samples into two clusters with different loss values. On further training,
the model over-fits on the noisy samples and the training loss on both samples
reduces. We illustrate this in Fig. 2(a)$-$(c). We fit a 2-component Beta
mixture model (BMM) over the normalized training losses
($\mathcal{L}_{CE}(\hat{y}^{(c)},\cdot)\in[0,1]$) obtained after training the
model for some warmup epochs $T_{0}$. Using a Beta mixture model works better
than using a Gaussian mixture model as it allows for asymmetric distributions
and can capture the short left-tails of the clean sample losses. For a sample
$(x,y)$ with normalized loss $\mathcal{L}_{CE}(\hat{y}^{(c)},y)=\ell$, the BMM
is given by:

$\displaystyle p(\ell)=\lambda_{c}\cdot p(\ell|\text{clean})+\lambda_{n}\cdot
p(\ell|\text{noisy})$ $\displaystyle
p(\ell|\text{clean})=\frac{\Gamma(\alpha_{c}+\beta_{c})}{\Gamma(\alpha_{c})\Gamma(\beta_{c})}\ell^{\alpha_{c}-1}{(1-\ell)}^{\beta_{c}-1}$
$\displaystyle
p(\ell|\text{noisy})=\frac{\Gamma(\alpha_{n}+\beta_{n})}{\Gamma(\alpha_{n})\Gamma(\beta_{n})}\ell^{\alpha_{n}-1}{(1-\ell)}^{\beta_{n}-1}$

where $\Gamma$ denotes the gamma distribution and $\alpha_{c/n},\beta_{c/n}$
are the parameters corresponding to the individual clean/noisy Beta
distributions. The mixture coefficients $\lambda_{c}$ and $\lambda_{n}$, and
parameters ($\alpha_{c/n},\beta_{c/n}$) are learnt using the EM algorithm. On
fitting the BMM $\mathcal{B}$, for a given input $x$ with a normalized loss
$\mathcal{L}_{CE}(\hat{y}^{(c)},y)=\ell$, we denote the posterior probability
of $x$ having a clean label by:

$\mathcal{B}(x)=\frac{\lambda_{c}\cdot p(\ell|\text{clean})}{\lambda_{c}\cdot
p(\ell|\text{clean})+\lambda_{n}\cdot p(\ell|\text{noisy})}$

The BMM $\mathcal{B}$ learnt from Fig. 2b is shown in Fig. 2d.

Algorithm 1 Training using $\mathcal{L}_{DN-H}$

Input: Train data $(x_{i},y_{i}^{(n)})_{i=1}^{N}$, warmup epochs $T_{0}$,
total epochs $T$, parameter $\beta$, classifier $M$, noise model $N_{M}$

for epoch in $\\{1,\dots,T_{0}\\}$ do

$\hat{y_{i}}^{(c)}\leftarrow M(x_{i})\ \forall\ i\in[N]$

Train $M$ with $\sum_{i}\mathcal{L}_{CE}(\hat{y_{i}}^{(c)},y_{i}^{(n)})$

end for

Fit a 2-mixture BMM $\mathcal{B}$ on
$\\{\mathcal{L}_{CE}(\hat{y_{i}}^{(c)},y_{i}^{(n)})\\}_{i=1}^{N}$

for epoch in $\\{T_{0}+1,\dots,T\\}$ do

$\hat{y_{i}}^{(c)}\leftarrow M(x_{i})$, $\hat{y_{i}}^{(n)}\leftarrow
N_{M}(R_{M}(x_{i}))\ \ \forall\ i\in[N]$

Train $M,N_{M}$ with $\mathcal{L}_{DN{-}H}=$
$\qquad\sum_{i}\big{(}\mathcal{L}_{CE}(\hat{y_{i}}^{(n)},y_{i}^{(n)}){+}$
$\beta\cdot\mathbbm{1}[\mathcal{B}(x){>}0.5]\cdot\mathcal{L}_{CE}(\hat{y_{i}}^{(c)},y_{i}^{(n)})\big{)}$

end for

Return: Trained classifier model $M$

###  3.2. Learning $M$ and $N_{M}$

We aim to train $M,N_{M}$ such that when given an input, $M$ predicts the
clean label and $N_{M}$ predicts the noisy label for this input (if
$\mathcal{F}$ retains the original clean label for this input, then both
$M,N_{M}$ predict the clean label). Thus for an input $(x,y)$ having a clean
label, we want $\hat{y}^{(c)}{=}\hat{y}^{(n)}{=}y$; and for an input $(x,y)$
having a noisy label, we want $\hat{y}^{(n)}{=}y$ and $\hat{y}^{(c)}$ to be
the clean label for $x$. We jointly train $M,N_{M}$ using the de-noising loss
proposed below:

(1) $\displaystyle\mathcal{L}_{DN}{=}\ \mathcal{L}_{CE}(\hat{y}^{(n)},y)\
{+}\beta{\cdot}\mathcal{B}(x){\cdot}\mathcal{L}_{CE}({\hat{y}^{(c)}},y)$

The first term trains the $M{-}N_{M}$ cascade jointly using cross entropy
between $\hat{y}^{(n)}$ and $y$. The second term trains $M$ to predict
$\hat{y}^{(c)}$ correctly for samples believed to be clean, weighted by
$\mathcal{B}(x)$. Here $\beta$ is a parameter that controls the trade-off
between the two terms.

By jointly training $M,N_{M}$ with $\mathcal{L}_{DN}$, we implicitly constrain
the label noise in $N_{M}$. We use an alternative loss formulation by
replacing the Bernoulli $\mathcal{B}(x)$ with the indicator
$\mathbbm{1}[\mathcal{B}(x){>}0.5]$. For ease of notation, we refer the former
(using $\mathcal{B}(x)$) as the soft de-noising loss $\mathcal{L}_{DN{-}S}$
and the latter as the hard de-noising loss $\mathcal{L}_{DN{-}H}$.

Thus we use the following 3-step approach to learn $M$ and $N_{M}$:

  1. (1)

Warmup: Train $M$ using $\mathcal{L}_{CE}(\hat{y}^{(c)},y)$.

  2. (2)

Fitting BMM: Fit a 2-component BMM $\mathcal{B}$ on the
$\mathcal{L}_{CE}(\hat{y}^{(c)},y)$ for all $(x,y)\in(X,Y^{(n)})$.

  3. (3)

Training with $\mathcal{L}_{DN}$: Jointly train $M$ and $N_{M}$ end-to-end
using $\mathcal{L}_{DN{-}S/H}$.

We summarize our methodology in Algorithm 1, when using $\mathcal{L}_{DN-H}$.

Dataset | Num. Classes | Train | Validation | Test
---|---|---|---|---
TREC (Li and Roth (2002)) | 6 | 4949 | 503 | 500
AG-News (Gulli (2005)) | 4 | 112000 | 8000 | 7600

Table 2. Summary statistics of the datasets

Model |  |  | Token (How/What) based |  | Length based
---|---|---|---|---|---
Noise % |  | 10 | 20 | 30 | 40 | 50 |  | 10 | 20 | 30 | 40 | 50
word LSTM | Baseline |  | 89.2 (-0.4) | 84.4 (-8.2) | 77.8 (-10.6) | 76 (-17) | 71.8 (-15.8) |  | 91.4 (-1.0) | 87 (0.6) | 82.2 (1.8) | 82.4 (-2.6) | 74.2 (-3.0)
$\mathcal{L}_{DN{-}H}$ |  |  91.8 (0) | 87.4 (-2.2) |  84.2 (0.4) | 79 (1) | 67.8 (1.4) |  | 91.6 (-0.6) | 90.2 (-0.8) |  87.4 (-0.2) |  87.4 (-0.8) |  79 (0)
$\mathcal{L}_{DN{-}S}$ |  | 91.8 (0.2) |  90.6 (-1.2) | 83.8 (-6.8) |  79.2 (-19.2) |  75.6 (-15.8) |  |  92 (0.4) |  90.6 (1) | 85.4 (0.2) | 84 (-2.8) | 75 (-3)
word CNN | Baseline |  | 90.4 (-3.6) | 83.8 (-1.8) | 82.4 (-7.4) | 78.8 (-17.2) | 52 (1.4) |  | 91 (0) | 88 (1.2) | 85.2 (-1.2) | 82 (-2.6) | 73.6 (-1.4)
$\mathcal{L}_{DN{-}H}$ |  | 90 (0.8) | 86.6 (-3) |  84.4 (-0.6) | 80.6 (-4.2) |  74 (-7.6) |  | 90.6 (-0.8) | 89.6 (-1) |  87.2 (-0.2) | 82.6 (-0.4) |  77 (-6.2)
$\mathcal{L}_{DN{-}S}$ |  |  91.2 (-0.4) |  86.8 (-1.8) | 84.2 (-4.2) |  81.8 (-12) | 65.2 (-12.4) |  |  92.8 (-3.6) |  91 (-2.2) | 86.8 (-1.4) |  86 (-4) | 75.4 (1.8)

Table 3.  Results from experiments using input-conditional noise on the TREC
dataset.

##  4\. Evaluation

Datasets We experiment with two popular text classification datasets: (i) TREC
question-type dataset (Li and Roth, 2002), and (ii) AG-News dataset (Gulli,
2005) (Table 2). We inject noise in the training and validation sets, while
retaining the original clean test set for evaluation. Note that collecting
real datasets with known patterns of label noise is a challenging task, and
out of the scope of this work. We artificially inject noise in clean datasets,
which enables easy and extensive experimentation.

Models We conduct experiments on two popular model architectures: word-LSTM
(Hochreiter and Schmidhuber, 1997) and word-CNN (Kim, 2014). For word-LSTM, we
use a 2-layer BiLSTM with hidden dimension of 150. For word-CNN, we use 300
kernel filters each of size 3, 4 and 5. We use the pre-trained GloVe
embeddings (Pennington et al., 2014) for initializing the word embeddings for
both models. We train models on TREC and AG-News for 100 and 30 epochs
respectively. We use an Adam optimizer with a learning rate of $10^{-5}$ and a
dropout of $0.3$ during training. For the noise model $N_{M}$, we use a simple
2-layer feedforward neural network, with the number of hidden units
$n_{hidden}=4{\cdot}n_{input}$. We choose the inputs to the noise model
$R_{M}(x)$ as per the class of label noise, which we describe in Section 4.1
and 4.2. We conduct hyper-parameter tuning for the number of warmup epochs
$T_{0}$ and $\beta$ using grid search over the ranges of {6,10,20} and
{2,4,6,8,10} respectively.

Metrics and Baseline We evaluate the robustness of the model to label noise on
two fronts: (i) how well it performs on clean data, and (ii) how much it over-
fits the noisy data. For the former, we report the test set accuracy (denoted
by Best) corresponding to the model with best validation accuracy . For the
latter, we examine the gap in test accuracies between the Best, and the Last
model (after last training epoch). We evaluate our approach against only
training $M$ (as the baseline), for two types of noise: random and input-
conditional, at different noise levels.

###  4.1. Results: Random Noise

For a specific Noise %, we randomly change the original labels of this
percentage of samples. Since the noise function is independent of the input,
we use logits from $M$ as the input $R(x)$ to $N_{M}$. We report the Best and
(Last \- Best) test accuracies in Table 1. From the experiments, we observe
that:

(i) $L_{DN-S}$ and $L_{DN-H}$ almost always outperforms the baseline across
different noise levels. The performance of $L_{DN-S}$ and $L_{DN-H}$ are
similar. We observe that training with $L_{DN{-}S}$ tends to be better at low
noise %, whereas $L_{DN{-}H}$ tends to be better at higher noise %. Our method
is more effective for TREC than AG-News, since even the baseline can learn
robustly on AG-News.

(ii) Our approach using $L_{DN{-}S}$ and $L_{DN{-}H}$ drastically reduces
over-fitting on noisy samples (visible from small gaps between Best and Last
accuracies). For the baseline, this gap is significantly larger, especially at
high noise levels, indicating over-fitting to the label noise. For example,
consider word-LSTM on TREC at 30% noise: while the baseline suffers a sharp
drop of 24.8 points from 79.6%, the accuracy of the $L_{DN{-}S}$ model drops
just 1.0% from 83.4%.

Figure 3. Test accuracy across training epochs of word-LSTM model on the TREC
dataset with two levels of random noise: $30\%$ and $40\%$. The baseline
heavily over-fits on the noise degrading performance, while $L_{DN{-}S}$ and
$L_{DN{-}H}$ avoid this.

We further demonstrate that our approach avoids over-fitting, thereby
stabilizing the model training by plotting the test accuracies across training
epochs in Fig. 3. We observe that the baseline model over-fits the label noise
with more training epochs, thereby degrading test accuracy. The degree of
over-fitting is greater at higher levels of noise (Fig. 3(b) vs Fig. 3(a)). In
comparison, our de-noising approach using both $L_{DN-S}$ and $L_{DN-H}$ does
not over-fit on the noisy labels as demonstrated by stable test accuracies
across epochs. This is particularly beneficial when operating in a few-shot
setting where one does not have access to a validation split that is
representative of the test split for early stopping.

| Noise | AP (7.8%) | Reuters (10.8%) | Either (18.6%)
---|---|---|---|---
word LSTM | Baseline | 82.8 (-0.5) | 85.6 (-0.8) | 75.7 (-0.4)
$\mathcal{L}_{DN{-}H}$ | 82.7 (0) |  85.7 (-0.1) |  76.6 (-0.4)
$\mathcal{L}_{DN{-}S}$ |  82.8 (0.3) | 85.5 (0.1) | 76 (-0.1)
word CNN | Baseline |  83.1 (-0.2) | 85.7 (0) |  76.6 (-0.9)
$\mathcal{L}_{DN{-}H}$ | 82.4 (0.8) |  86.2 (0) | 76.1 (0.1)
$\mathcal{L}_{DN{-}S}$ | 82.5 (0.5) | 86.1 (0.1) | 76.4 (0)

Table 4.  Results from input-conditional noise on AG-News.

###  4.2. Results: Input-Conditional Noise

We heuristically condition the noise function $\mathcal{F}$ on lexical and
syntactic input features. We are the first to study such label noise for text
inputs, to our knowledge. For both the TREC and AG-News, we condition
$\mathcal{F}$ on syntactic features of the input: (i) The TREC dataset
contains different types of questions. We selectively corrupt the labels of
inputs that contain the question words ‘How’ or ‘What’ (chosen based on
occurrence frequency). For texts starting with ‘How’ or ‘What’, we insert
random label noise (at different levels). We also consider $\mathcal{F}$
conditional on the text length (a lexical feature). More specifically, we
inject random label noise for the longest x% inputs in the dataset. (ii) The
AG-News dataset contains news articles from different news agency sources. We
insert random label noise for inputs containing the token ‘AP’, ‘Reuters’ or
either one of them. We concatenate the contextualised input embedding from the
penultimate layer of $M$ and the logits corresponding to $\hat{y}^{(c)}$ as
the input $R_{M}(x)$ to $N_{M}$. We present the results in Tables 3 and 4.

On TREC, our method outperforms the baseline for both the noise patterns we
consider. For the question-length based noise, we observe the same trend of
$L_{DN{-}H}$ outperforming $L_{DN{-}S}$ at high noise levels, and vice-versa.
On AG-News, the noise % for inputs having the specific tokens ’AP’ and
’Reuters’ are relatively low, and our method performs at par or marginally
improves over the baseline performance. Interestingly, the input–conditional
noise we consider makes effective learning very challenging, as demonstrated
by significantly lower _Best_ accuracies for the baseline model than for
random noise. As the classifier appears to overfit to the noise very early
during training, we observe relatively smaller gaps between Best and Last
accuracies. Compared to random noise, our approach is less efficient at
alleviating the _(Best-Last)_ accuracy gap for input-conditional noise. These
experiments however reveal promising preliminary results on learning with
input-conditional noise.

##  5\. Conclusion

We have presented an approach to improve text classification when learning
from noisy labels by jointly training a classifier and a noise model using a
de-noising loss. We have evaluated our approach on two text classification
tasks. demonstrate its effectiveness through an extensive evaluation. Future
work includes studying more complex $\mathcal{F}$ for other NLP tasks like
language inference and QA.

## References

  * (1)
  * Agarwal et al. (2007) Sumeet Agarwal, Shantanu Godbole, Shourya Roy, and Diwakar Punjani. 2007.  How much noise is too much: A study in automatic text classification. In _In Proc. of ICDM_.
  * Arazo et al. (2019) Eric Arazo, Diego Ortego, Paul Albert, Noel O’Connor, and Kevin Mcguinness. 2019.  Unsupervised Label Noise Modeling and Loss Correction. In _Proceedings of the 36th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 97)_ , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, Long Beach, California, USA, 312–321.  http://proceedings.mlr.press/v97/arazo19a.html
  * Ardehaly and Culotta (2018) Ehsan Ardehaly and Aron Culotta. 2018.  Learning from noisy label proportions for classifying online social data.  _Social Network Analysis and Mining_ 8 (12 2018).  https://doi.org/10.1007/s13278-017-0478-6
  * Frénay and Verleysen (2014) Benoît Frénay and Michel Verleysen. 2014.  Classification in the Presence of Label Noise: A Survey.  _IEEE Transactions on Neural Networks and Learning Systems_ 25 (2014), 845–869.
  * Goodman et al. (2016) James Goodman, Andreas Vlachos, and Jason Naradowsky. 2016\.  Noise reduction and targeted exploration in imitation learning for Abstract Meaning Representation parsing. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, 1–11.  https://doi.org/10.18653/v1/P16-1001
  * Gulli (2005) A. Gulli. 2005.  The Anatomy of a News Search Engine. In _Special Interest Tracks and Posters of the 14th International Conference on World Wide Web_ (Chiba, Japan) _(WWW ’05)_. Association for Computing Machinery, New York, NY, USA, 880–881.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Comput._ 9, 8 (Nov. 1997), 1735–1780.  https://doi.org/10.1162/neco.1997.9.8.1735
  * Ipeirotis et al. (2010) Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang. 2010\.  Quality Management on Amazon Mechanical Turk. In _Proceedings of the ACM SIGKDD Workshop on Human Computation_ (Washington DC) _(HCOMP ’10)_. Association for Computing Machinery, New York, NY, USA, 64–67.  https://doi.org/10.1145/1837885.1837906
  * Jiang et al. (2019) Junjun Jiang, Jiayi Ma, Zheng Wang, Chen Chen, and Xianming Liu. 2019.  Hyperspectral Image Classification in the Presence of Noisy Labels.  _IEEE Transactions on Geoscience and Remote Sensing_ 57 (2019), 851–865.
  * Jiang et al. (2018) Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. 2018.  MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels. In _Proceedings of the 35th International Conference on Machine Learning_ , Jennifer Dy and Andreas Krause (Eds.). PMLR.  http://proceedings.mlr.press/v80/jiang18c.html
  * Jindal et al. (2019) Ishan Jindal, Daniel Pressel, Brian Lester, and Matthew Nokleby. 2019.  An Effective Label Noise Model for DNN Text Classification. In _Proceedings of North American Chapter of the Association of Computational Linguistics 2019_.
  * Joshi et al. (2014) Aditya Joshi, Abhijit Mishra, Nivvedan Senthamilselvan, and Pushpak Bhattacharyya. 2014.  Measuring Sentiment Annotation Complexity of Text. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, Baltimore, Maryland.
  * Kim (2014) Yoon Kim. 2014.  Convolutional Neural Networks for Sentence Classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014_. 1746–1751.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Andrew Lee, Jonathan K. Kummerfeld, Parker Hill, Michael A. Laurenzano, Johann Hauswald, Lingjia Tang, and Jason Mars. 2019.  Outlier Detection for Improved Data Quality and Diversity in Dialog Systems. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 517–527.  https://doi.org/10.18653/v1/N19-1051
  * Li and Roth (2002) Xin Li and Dan Roth. 2002\.  Learning Question Classifiers. In _Proceedings of the 19th International Conference on Computational Linguistics - Volume 1_ (Taipei, Taiwan) _(COLING ’02)_. Association for Computational Linguistics, Stroudsburg, PA, USA, 1–7.
  * Malik and Bhardwaj (2011) H. H. Malik and V. S. Bhardwaj. 2011.  Automatic Training Data Cleaning for Text Classification. In _2011 IEEE 11th International Conference on Data Mining Workshops_. 442–449.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  GloVe: Global Vectors for Word Representation. In _Empirical Methods in Natural Language Processing (EMNLP)_. 1532–1543.  http://www.aclweb.org/anthology/D14-1162
  * Reed et al. (2015) Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. 2015.  Training Deep Neural Networks on Noisy Labels with Bootstrapping.. In _ICLR (Workshop)_ , Yoshua Bengio and Yann LeCun (Eds.).  http://dblp.uni-trier.de/db/conf/iclr/iclr2015w.html#ReedLASER14
  * Thulasidasan et al. (2019) Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd-Yusof. 2019\.  Combating Label Noise in Deep Learning using Abstention. In _Proceedings of the 36th International Conference on Machine Learning_ , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR.  http://proceedings.mlr.press/v97/thulasidasan19a.html
  * Wang et al. (2018) Fei Wang, Liren Chen, Cheng Li, Shiyao Huang, Yanjie Chen, Chen Qian, and Chen Change Loy. 2018.  The Devil of Face Recognition is in the Noise.  _arXiv preprint arXiv:1807.11649_ (2018).
  * Zhan et al. (2019) Xueying Zhan, Yaowei Wang, Yanghui Rao, and Qing Li. 2019\.  Learning from Multi-Annotator Data: A Noise-Aware Classification Framework.  _ACM Trans. Inf. Syst._ 37, 2, Article 26 (Feb. 2019), 28 pages.  https://doi.org/10.1145/3309543
  * Zhang et al. (2016) Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2016.  Understanding deep learning requires rethinking generalization.  _ICLR_ (2016).  http://arxiv.org/abs/1611.03530 cite arxiv:1611.03530Comment: Published in ICLR 2017.
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018.  mixup: Beyond Empirical Risk Minimization. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=r1Ddp1-Rb
  * Zlateski et al. (2018) Aleksandar Zlateski, Ronnachai Jaroensri, Prafull Sharma, and Fredo Durand. 2018.  On the Importance of Label Quality for Semantic Segmentation. In _CVPR_. 1479–1487.  https://doi.org/10.1109/CVPR.2018.00160