# Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks

Paul Röttger University of Oxford  The Alan Turing Institute  Bertie Vidgen
The Alan Turing Institute  Dirk Hovy Bocconi University  Janet B.
Pierrehumbert University of Oxford

###### Abstract

Labelled data is the foundation of most natural language processing tasks.
However, labelling data is difficult and there often are diverse valid beliefs
about what the correct data labels should be. So far, dataset creators have
acknowledged annotator subjectivity, but rarely actively managed it in the
annotation process. This has led to partly-subjective datasets that fail to
serve a clear downstream use. To address this issue, we propose two
contrasting paradigms for data annotation. The descriptive paradigm encourages
annotator subjectivity, whereas the prescriptive paradigm discourages it.
Descriptive annotation allows for the surveying and modelling of different
beliefs, whereas prescriptive annotation enables the training of models that
consistently apply one belief. We discuss benefits and challenges in
implementing both paradigms, and argue that dataset creators should explicitly
aim for one or the other to facilitate the intended use of their dataset.
Lastly, we conduct an annotation experiment using hate speech data that
illustrates the contrast between the two paradigms.

##  1 Introduction

Many natural language processing (NLP) tasks are subjective, in the sense that
there are diverse valid beliefs about what the correct data labels should be.
Some tasks, like hate speech detection, are highly subjective: different
people have very different beliefs about what should or should not be labelled
as hateful (Talat, 2016; Salminen et al., 2019; Davani et al., 2021a), and
while some beliefs are more widely accepted than others, there is no single
objective truth. Other examples include toxicity (Sap et al., 2019, 2021),
harassment (Al Kuwatly et al., 2020), harmful content (Jiang et al., 2021) and
stance detection (Luo et al., 2020; AlDayel and Magdy, 2021) as well as
sentiment analysis (Kenyon-Dean et al., 2018; Poria et al., 2020). But even
for seemingly objective tasks like part-of-speech tagging, there is subjective
disagreement between annotators (Plank et al., 2014b).

Figure 1: Two key questions for dataset creators.

In this article, we argue that dataset creators should consider the role of
annotator subjectivity in the annotation process and either explicitly
encourage it or discourage it. Annotators may subjectively disagree about
labels (e.g., for hate speech) but dataset creators can and should decide,
based on the intended downstream use of their dataset, whether they want to a)
capture different beliefs or b) encode one specific belief in their data.

As a framework, we propose two contrasting data annotation paradigms. Each
paradigm facilitates a clear and distinct downstream use. The descriptive
paradigm encourages annotator subjectivity to create datasets as granular
surveys of individual beliefs. Descriptive data annotation thus allows for the
capturing and modelling of different beliefs. The prescriptive paradigm, on
the other hand, discourages annotator subjectivity and instead tasks
annotators with encoding one specific belief, formulated in the annotation
guidelines. Prescriptive data annotation thus enables the training of models
that seek to consistently apply one belief. A researcher may, for example,
want to model different beliefs about hate speech ($\rightarrow$ descriptive
paradigm), while a content moderation engineer at a social media company may
need models that apply their content policy ($\rightarrow$ prescriptive
paradigm).

Neither paradigm is inherently superior, but explicitly aiming for one or the
other is beneficial because it makes clear what an annotated dataset can and
should be used for. For example, data annotated under the descriptive paradigm
can provide insights into different beliefs (§2.1), but it cannot easily be
used to train models with one pre-specified behaviour (§3.1). By contrast,
leaving annotator subjectivity unaddressed, as has mostly been the case in NLP
so far, leads to datasets that neither capture an interpretable diversity of
beliefs nor consistently encode one specific belief; an undesirable middle
ground without a clear downstream use.111See Appx. A for a selective overview
of existing datasets.

The two paradigms are applicable to all data annotation. They can be used to
compare existing datasets, and to make and communicate decisions about how new
datasets are annotated as well as how annotator disagreement can be
interpreted. We hope that by naming and explaining the two paradigms, and by
discussing key benefits and challenges in their implementation, we can support
more intentional annotation process design, which will result in more useful
NLP datasets.

#### Terminology

Our use of the terms descriptive and prescriptive aligns with their use in
both linguistics and ethics. In linguistics, descriptivism studies how
language is used, whereas prescriptive grammar declares how language should be
used (Justice, 2006). Descriptive ethics studies the moral judgments that
people make, while prescriptive ethics considers how people ought to act
(Thiroux and Krasemann, 2015). Accordingly, descriptive data annotation
surveys annotators’ beliefs, whereas prescriptive data annotation aims to
encode one specific belief, which is formulated in the annotation guidelines.

##  2 The Descriptive Annotation Paradigm: Encouraging Annotator Subjectivity

###  2.1 Key Benefits

#### Insights into Diverse Beliefs

Descriptive data annotation captures a multiplicity of beliefs in data labels,
much like a very granular survey would. The distribution of data labels across
annotators and examples can therefore provide insights into the beliefs of
annotators, or the larger population they may represent. For example,
descriptive data annotation has shown that non-Black annotators are more
likely to rate African American English as toxic (Sap et al., 2019, 2021), and
that people who identify as LGBTQ+ or young adults are more likely to rate
random social media comments as toxic (Kumar et al., 2021). Similar
correlations between sociodemographic characteristics and annotation outcomes
have been found in stance (Luo et al., 2020), sentiment (Diaz et al., 2018)
and hate speech detection (Talat, 2016).

Even very subjective tasks may have clear-cut entries on which most annotators
agree. For example, crowd workers tend to agree more on the extremes of a hate
rating scale (Salminen et al., 2019), and datasets which consist of clear hate
and non-hate can have very high levels of inter-annotator agreement, even with
minimal guidelines (Röttger et al., 2021). Descriptive data annotation can
help to identify which entries are more subjective. Jiang et al. (2021), for
instance, find that perceptions about the harmfulness of sexually explicit
language vary strongly across the eight countries in their sample, whereas
support for mass murder or human trafficking is seen as very harmful across
all countries.

#### Learning from Disagreement

Annotator-level labels from descriptive data annotation have been shown to be
a rich source of information for model training. First, they can be used to
separately model annotators’ beliefs. For less subjective tasks such as
question answering, this has served to mitigate undesirable annotator biases
(Geva et al., 2019). Davani et al. (2021b) reframe and expand on this idea for
more subjective tasks like abuse detection, showing that multi-annotator model
architectures outperform standard single-label approaches on single label
prediction. Second, instead of modelling each annotator separately, other work
has grouped them into clusters based on sociodemographic attributes (Al
Kuwatly et al., 2020) or polarisation measures derived from annotator labels
(Akhtar et al., 2020, 2021), with similar results. Third, models can be
trained directly on soft labels (i.e., distributions of labels given by
annotators), rather than hard one-hot ground truth vectors (Plank et al.,
2014a; Jamison and Gurevych, 2015; Uma et al., 2020; Fornaciari et al., 2021).

#### Evaluating with Disagreement

Descriptive data annotation facilitates model evaluation that accounts for
different beliefs about how a model should behave (Basile et al., 2021b; Uma
et al., 2021). This is particularly relevant when deploying NLP systems for
practical tasks such as content moderation, where user-facing performance
needs to be considered (Gordon et al., 2021). To this end, comparing a model
prediction to a descriptive label distribution, the crowd truth (Aroyo and
Welty, 2015), can help estimate how acceptable the prediction would be to
users (Alm, 2011). Gordon et al. (2022) operationalise this idea by
introducing jury learning, a recommender system approach to predicting how a
group of annotators with specified sociodemographic characteristics would
judge different pieces of content.

###  2.2 Key Challenges

#### Representativeness of Annotators

The survey-like benefits of descriptive data annotation correspond to survey-
like challenges. First, dataset creators must decide who their data aims to
represent, by establishing a clear population of interest. Arora et al.
(2020), for example, ask women journalists to annotate harassment targeted at
them. Talat (2016) recruits feminist activists as well as crowd workers.
Second, dataset creators must consider whether representativeness can
practically be achieved. To capture a representative distribution of beliefs
for each entry requires dozens, if not hundreds of annotators recruited from
the population of interest. Sap et al. (2021), for example, collect toxicity
labels from 641 annotators, but only for 15 examples. Other datasets generally
use much fewer annotators per entry (see Appx. A) and therefore cannot be
considered representative in the sense that large (i.e., many-participant)
surveys are. A potential approach to mitigating this issue in modelling
annotator beliefs is by introducing information sharing across groups of
annotators (e.g. based on sociodemographics), where annotator behaviour
updates group-specific priors rather than being considered in isolation, and
thus fewer annotations are needed from each annotator (Gordon et al., 2022).

#### Interpretation of Disagreement

In the descriptive paradigm, the absence of a (specified) ground truth label
complicates the interpretation of any observed annotator disagreement: it may
be due to a genuine difference in beliefs, which is desirable in this
paradigm, or due to undesirable annotator error (Pavlick and Kwiatkowski,
2019; Basile et al., 2021a; Leonardelli et al., 2021). The same issue applies
to inter-annotator agreement metrics like Fleiss’ Kappa. When subjectivity is
encouraged, such metrics can at best measure task subjectiveness, but not task
difficulty, annotator performance, or dataset quality (Zaenen, 2006; Alm,
2011).

#### Label Aggregation

Descriptive annotation has clear downstream uses (§2.1) but it is
fundamentally misaligned with standard NLP methods that rely on single gold
standard labels. When datasets are constructed to be granular surveys of
beliefs, reducing those beliefs to a single label, through majority voting or
otherwise, goes directly against that purpose. Aggregating labels conceals
informative disagreements (Leonardelli et al., 2021; Basile et al., 2021b) and
risks discarding minority beliefs (Prabhakaran et al., 2021; Basile et al.,
2021a).

##  3 The Prescriptive Annotation Paradigm: Discouraging Annotator
Subjectivity

###  3.1 Key Benefits

#### Specified Model Behaviour

Encoding one specific belief in a dataset through data annotation is difficult
(§3.2) but advantageous for many practical applications. Social media
companies, for example, moderate content on their platforms according to
specific and extensive content policies.222In March 2021, a whistleblower
shared 300-page content guidelines used by Facebook moderators (Hern, 2021).
Therefore, they need data annotated in accordance with those policies to train
their content moderation models. This illustrates that even for highly
subjective tasks, where different model behaviours are plausible and valid,
one specific behaviour may be practically desirable. Prescriptive data
annotation specifies such desired behaviours in datasets for model training
and evaluation.

#### Quality Assurance

In the prescriptive paradigm, annotator disagreements are a call to action
because they indicate that a) the annotation guidelines were not correctly
applied by annotators or b) the guidelines themselves were inadequate.
Annotator errors can be found using noise identification techniques (e.g.,
Hovy et al., 2013; Zhang et al., 2017; Paun et al., 2018; Northcutt et al.,
2021), corrected by expert annotators (Vidgen and Derczynski, 2020; Vidgen et
al., 2021a) or their impact mitigated by label aggregation. Guidelines which
are unclear or incomplete need to be clarified or expanded by dataset
creators, which may require iterative approaches to annotation (Founta et al.,
2018; Zeinert et al., 2021). Therefore, quality assurance under the
prescriptive paradigm is a laborious but structured process, with inter-
annotator agreement as a useful, albeit noisy, measure of dataset quality.

#### Visibility of Encoded Belief

In the prescriptive paradigm, the one belief that annotators are tasked with
applying is made visible and explicit in the annotation guidelines. Well-
formulated guidelines should give clear instructions on how to decide between
different classes, along with explanations and illustrative examples. This
creates accountability, in that people can review, challenge and disagree with
the formulated belief. Like data statements (Bender and Friedman, 2018),
prescriptive annotation guidelines can provide detailed insights into how
datasets were created, which can then inform their downstream use.

###  3.2 Key Challenges

#### Creation of Annotation Guidelines

Creating guidelines for prescriptive data annotation is difficult because it
requires topical knowledge and familiarity with the data that is to be
annotated. Guidelines would ideally provide a clear judgment on every possible
entry, but in practice, such perfectly comprehensive guidelines can only be
approximated. Even extensive legal definitions of hate speech leave some room
for subjective interpretation (Sellars, 2016). Further, creating guidelines
for prescriptive data annotation requires deciding which one belief to encode
in the dataset. This can be a complex process that risks disregarding non-
majority beliefs if marginalised people are not included in it (Raji et al.,
2020).

#### Application of Annotation Guidelines

Annotators need to be familiar with annotation guidelines to apply them
correctly, which may require additional training, especially if guidelines are
long and complex. This is reflected in an increasing shift in the literature
towards using annotators with task-relevant experience over non-trained crowd
workers (e.g. Basile et al., 2019; Röttger et al., 2021; Vidgen et al.,
2021a). During annotation, annotators will need to refer back to the
guidelines, which requires giving them sufficient time per entry and providing
a well-designed annotation interface.

#### Persistent Subjectivity

Annotator subjectivity can be discouraged, but not eliminated. Inevitable gaps
in guidelines leave annotators no choice but to apply their personal judgement
for some entries, and even when there is explicit guidance, implicit biases
may persist. Sap et al. (2019), for example, demonstrate racial biases in hate
speech annotation, and show that targeted annotation prompts can reduce these
biases but not definitively eliminate them. To address this issue, dataset
creators should work with groups of annotators that are diverse in terms of
sociodemographic characteristics and personal experiences, even when annotator
subjectivity is discouraged.

##  4 An Illustrative Annotation Experiment

#### Experimental Design

To illustrate the contrast between the two paradigms, we conducted an
annotation experiment. 60 annotators were randomly assigned to one of three
groups of 20. Each group was given different guidelines to label the same 200
Twitter posts, taken from a corpus annotated for hate speech by Davidson et
al. (2017), as either hateful or non-hateful. G1, the descriptive group,
received a short prompt which directed them to apply their subjective
judgement (‘Do you personally feel this post is hateful?’). G2, the
prescriptive group, received a short prompt which discouraged subjectivity
(‘Does this post meet the criteria for hate speech?’), along with detailed
annotation guidelines. G3, the control group, received the prescriptive prompt
and a short definition of hate speech but no further guidelines. This is to
control for the difference in length and complexity of annotation guidelines
between G1 and G2.

#### Results

We evaluate average percentage agreement and Fleiss’ $\kappa$ to measure
dataset-level inter-annotator agreement in each group (Table 1). To test for
significant differences in agreement between groups, we use confidence
intervals computed with a 1000-sample bootstrap.

Group |  Avg. % Agree. |  Fleiss’ $\kappa$
---|---|---
G1 \- Descriptive |  73.90 |  0.20
G2 \- Prescriptive |  93.72 |  0.78
G3 \- Control |  72.50 |  0.15
Table 1: Inter-annotator agreement metrics for the three groups of 20
annotators on our 200-post binary dataset.

Agreement is very low in the descriptive group G1 ($\kappa=0.20$), which
suggests that annotators hold varied beliefs about which posts are hateful.
However, agreement is significantly higher ($p<0.001$) in G2 ($\kappa=0.78$),
which suggests that a prescriptive approach with detailed annotation
guidelines can successfully induce annotators to apply a specified belief
rather than their subjective view. Further, agreement in the control group G3
($\kappa=0.15$) is as low as in descriptive G1, which suggests that
comprehensive guidelines are instrumental in facilitating high agreement in
the prescriptive paradigm. G1 and G3 also do not differ systematically on
which posts annotators disagree on, which suggests that annotators with little
prescriptive instruction (G3) tend to apply their subjective views (like G1).

#### Reproducibility

For details on our dataset and annotators, see the data statement (Bender and
Friedman, 2018) in Appendix B. Annotation prompts are given in Appendix C.
Full guidelines, annotated data and code are available on GitHub.

##  5 Conclusion

In this article, we named and explained two contrasting paradigms for data
annotation. The descriptive paradigm encourages annotator subjectivity to
create datasets as granular surveys of individual beliefs, which can then be
analysed and modelled. The prescriptive paradigm tasks annotators with
encoding one specific belief formulated in the annotation guidelines, to
enable the training of models that seek to apply that one belief to unseen
data. Dataset creators should explicitly aim for one paradigm or the other to
facilitate the intended downstream use of their dataset, and to document for
the benefit of others how exactly their dataset was annotated. We discussed
benefits and challenges in implementing both paradigms, and conducted an
annotation experiment that illustrates the contrast between them. We hope that
the two paradigms can support more intentional annotation process design and
thus facilitate the creation of more useful NLP datasets.

## Acknowledgments

Paul Röttger was funded by the German Academic Scholarship Foundation. Bertie
Vidgen and Paul Röttger were both supported by The Alan Turing Institute and
Towards Turing 2.0 under the EPSRC Grant EP/W037211/1. Dirk Hovy received
funding from the European Research Council (ERC) under the European Union’s
Horizon 2020 research and innovation program (grant agreement No. 949944). He
is a member of the Data and Marketing Insights (DMI) Unit of the Bocconi
Institute for Data Science and Analysis (BIDSA). Janet B. Pierrehumbert was
supported by EPSRC Grant EP/T023333/1. We thank the Milan NLP Group, the
Groningen Computational Linguistics Group as well as the Pierrehumbert
Language Modelling Group for helpful comments and all reviewers for their
constructive feedback.

## References

  * Akhtar et al. (2020) Sohail Akhtar, Valerio Basile, and Viviana Patti. 2020.  Modeling annotator perspective and polarized opinions to improve hate speech detection.  In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ , volume 8, pages 151–154.
  * Akhtar et al. (2021) Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021.  Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection.  _arXiv preprint arXiv:2106.15896_.
  * Al Kuwatly et al. (2020) Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020.  Identifying and measuring annotator bias based on annotators’ demographic characteristics.  In _Proceedings of the Fourth Workshop on Online Abuse and Harms_ , pages 184–190, Online. Association for Computational Linguistics.
  * AlDayel and Magdy (2021) Abeer AlDayel and Walid Magdy. 2021.  Stance detection on social media: State of the art and trends.  _Information Processing & Management_, 58(4):102597.
  * Alm (2011) Cecilia Ovesdotter Alm. 2011.  Subjective natural language problems: Motivations, applications, characterizations, and implications.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 107–112, Portland, Oregon, USA. Association for Computational Linguistics.
  * Arora et al. (2020) Ishaan Arora, Julia Guo, Sarah Ita Levitan, Susan McGregor, and Julia Hirschberg. 2020.  A novel methodology for developing automatic harassment classifiers for Twitter.  In _Proceedings of the Fourth Workshop on Online Abuse and Harms_ , pages 7–15, Online. Association for Computational Linguistics.
  * Aroyo and Welty (2015) Lora Aroyo and Chris Welty. 2015.  Truth is a lie: Crowd truth and the seven myths of human annotation.  _AI Magazine_ , 36(1):15–24.
  * Basile et al. (2019) Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019\.  Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 54–63.
  * Basile et al. (2021a) Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. 2021a.  Toward a perspectivist turn in ground truthing for predictive computing.  _Conference of the Italian Chapter of the Association for Intelligent Systems (ItAIS 2021)_.
  * Basile et al. (2021b) Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021b.  We need to consider disagreement in evaluation.  In _Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future_ , pages 15–21, Online. Association for Computational Linguistics.
  * Bender and Friedman (2018) Emily M. Bender and Batya Friedman. 2018.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  _Transactions of the Association for Computational Linguistics_ , 6:587–604.
  * Caselli et al. (2020) Tommaso Caselli, Valerio Basile, Jelena Mitrović, Inga Kartoziya, and Michael Granitzer. 2020.  I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 6193–6202, Marseille, France. European Language Resources Association.
  * Cercas Curry et al. (2021) Amanda Cercas Curry, Gavin Abercrombie, and Verena Rieser. 2021.  ConvAbuse: Data, analysis, and benchmarks for nuanced detection in conversational AI.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7388–7403, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Davani et al. (2021a) Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy, and Morteza Dehghani. 2021a.  Hate speech classifiers learn human-like social stereotypes.
  * Davani et al. (2021b) Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2021b.  Dealing with disagreements: Looking beyond the majority vote in subjective annotations.  _arXiv preprint arXiv:2110.05719_.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language.  In _Proceedings of the 11th International AAAI Conference on Web and Social Media_ , pages 512–515. Association for the Advancement of Artificial Intelligence.
  * Diaz et al. (2018) Mark Diaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018\.  _Addressing Age-Related Bias in Sentiment Analysis_, page 1–14. Association for Computing Machinery, New York, NY, USA.
  * Fornaciari et al. (2021) Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021.  Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2591–2597, Online. Association for Computational Linguistics.
  * Founta et al. (2018) Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018.  Large scale crowdsourcing and characterization of Twitter abusive behavior.  In _Proceedings of the 12th International AAAI Conference on Web and Social Media_ , pages 491–500. Association for the Advancement of Artificial Intelligence.
  * Geva et al. (2019) Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019.  Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1161–1166, Hong Kong, China. Association for Computational Linguistics.
  * Gordon et al. (2022) Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeffrey T Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022.  Jury learning: Integrating dissenting voices into machine learning models.  _arXiv preprint arXiv:2202.02950_.
  * Gordon et al. (2021) Mitchell L Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S Bernstein. 2021.  The disagreement deconvolution: Bringing machine learning performance metrics in line with reality.  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , pages 1–14.
  * Hern (2021) Alex Hern. 2021.  Decoding emojis and defining ’support’: Facebook’s rules for content revealed.  _The Guardian_.
  * Hovy et al. (2013) Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013.  Learning whom to trust with MACE.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1120–1130, Atlanta, Georgia. Association for Computational Linguistics.
  * Jamison and Gurevych (2015) Emily Jamison and Iryna Gurevych. 2015.  Noise or additional information? leveraging crowdsource annotation item agreement for natural language tasks. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 291–297, Lisbon, Portugal. Association for Computational Linguistics.
  * Jiang et al. (2021) Jialun Aaron Jiang, Morgan Klaus Scheuerman, Casey Fiesler, and Jed R Brubaker. 2021\.  Understanding international perceptions of the severity of harmful content online.  _PloS one_ , 16(8):e0256762.
  * Justice (2006) Paul Justice. 2006.  _Relevant Linguistics_ , 2nd edition.  Center for the Study of Language and Information, Stanford University.
  * Kenyon-Dean et al. (2018) Kian Kenyon-Dean, Eisha Ahmed, Scott Fujimoto, Jeremy Georges-Filteau, Christopher Glasz, Barleen Kaur, Auguste Lalande, Shruti Bhanderi, Robert Belfer, Nirmal Kanagasabai, Roman Sarrazingendron, Rohit Verma, and Derek Ruths. 2018.  Sentiment analysis: It’s complicated! In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1886–1895, New Orleans, Louisiana. Association for Computational Linguistics.
  * Kumar et al. (2021) Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021.  Designing toxic content classification for a diversity of perspectives.  In _Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021)_ , pages 299–318.
  * Leonardelli et al. (2021) Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021.  Agreeing to disagree: Annotating offensive language datasets with annotators’ disagreement.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10528–10539, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Luo et al. (2020) Yiwei Luo, Dallas Card, and Dan Jurafsky. 2020.  Detecting stance in media on global warming.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3296–3315, Online. Association for Computational Linguistics.
  * Northcutt et al. (2021) Curtis Northcutt, Lu Jiang, and Isaac Chuang. 2021.  Confident learning: Estimating uncertainty in dataset labels.  _Journal of Artificial Intelligence Research_ , 70:1373–1411.
  * Paun et al. (2018) Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018.  Comparing Bayesian models of annotation.  _Transactions of the Association for Computational Linguistics_ , 6:571–585.
  * Pavlick and Kwiatkowski (2019) Ellie Pavlick and Tom Kwiatkowski. 2019.  Inherent disagreements in human textual inferences.  _Transactions of the Association for Computational Linguistics_ , 7:677–694.
  * Plank et al. (2014a) Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014a.  Learning part-of-speech taggers with inter-annotator agreement loss.  In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 742–751, Gothenburg, Sweden. Association for Computational Linguistics.
  * Plank et al. (2014b) Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014b.  Linguistically debatable or just plain wrong? In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 507–511, Baltimore, Maryland. Association for Computational Linguistics.
  * Poria et al. (2020) Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, and Rada Mihalcea. 2020.  Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research.  _IEEE Transactions on Affective Computing_.
  * Prabhakaran et al. (2021) Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021.  On releasing annotator-level labels and information in datasets.  In _Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop_ , pages 133–138, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Raji et al. (2020) Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020.  Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , FAT* ’20, page 33–44, New York, NY, USA. Association for Computing Machinery.
  * Röttger et al. (2021) Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Talat, Helen Margetts, and Janet Pierrehumbert. 2021.  HateCheck: Functional tests for hate speech detection models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 41–58, Online. Association for Computational Linguistics.
  * Salminen et al. (2019) Joni Salminen, Hind Almerekhi, Ahmed Mohamed Kamel, Soon-gyo Jung, and Bernard J. Jansen. 2019.  Online hate ratings vary by extremes: A statistical analysis.  In _Proceedings of the 2019 Conference on Human Information Interaction and Retrieval_ , CHIIR ’19, page 213–217, New York, NY, USA. Association for Computing Machinery.
  * Sap et al. (2019) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019.  The risk of racial bias in hate speech detection.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1668–1678, Florence, Italy. Association for Computational Linguistics.
  * Sap et al. (2021) Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2021.  Annotators with attitudes: How annotator beliefs and identities bias toxic language detection.
  * Sellars (2016) Andrew Sellars. 2016.  Defining hate speech.  _Berkman Klein Center Research Publication_ , 20(2016):16–48.
  * Talat (2016) Zeerak Talat. 2016.  Are you a racist or am I seeing things? Annotator influence on hate speech detection on Twitter.  In _Proceedings of the First Workshop on NLP and Computational Social Science_ , pages 138–142, Austin, Texas. Association for Computational Linguistics.
  * Talat and Hovy (2016) Zeerak Talat and Dirk Hovy. 2016.  Hateful symbols or hateful people? Predictive features for hate speech detection on Twitter.  In _Proceedings of the NAACL Student Research Workshop_ , pages 88–93, San Diego, California. Association for Computational Linguistics.
  * Thiroux and Krasemann (2015) Jacques P Thiroux and Keith W Krasemann. 2015.  _Ethics: Theory and Practice_ , 11th edition.  Pearson.
  * Uma et al. (2021) Alexandra Uma, Tommaso Fornaciari, Anca Dumitrache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, and Massimo Poesio. 2021.  SemEval-2021 task 12: Learning with disagreements.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 338–347, Online. Association for Computational Linguistics.
  * Uma et al. (2020) Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2020.  A case for soft loss functions.  _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ , 8(1):173–177.
  * Vidgen and Derczynski (2020) Bertie Vidgen and Leon Derczynski. 2020.  Directions in abusive language training data, a systematic review: Garbage in, garbage out.  _PLOS ONE_ , 15(12):e0243300.
  * Vidgen et al. (2021a) Bertie Vidgen, Dong Nguyen, Helen Margetts, Patricia Rossini, and Rebekah Tromble. 2021a.  Introducing CAD: the contextual abuse dataset.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2289–2303, Online. Association for Computational Linguistics.
  * Vidgen et al. (2021b) Bertie Vidgen, Tristan Thrush, Zeerak Talat, and Douwe Kiela. 2021b.  Learning from the worst: Dynamically generated datasets to improve online hate detection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1667–1682, Online. Association for Computational Linguistics.
  * Zaenen (2006) Annie Zaenen. 2006.  Mark-up barking up the wrong tree.  _Comput. Linguist._ , 32(4):577–580.
  * Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019.  Predicting the type and target of offensive posts in social media.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1415–1420, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Zeinert et al. (2021) Philine Zeinert, Nanna Inie, and Leon Derczynski. 2021.  Annotating online misogyny.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3181–3197, Online. Association for Computational Linguistics.
  * Zhang et al. (2017) Jing Zhang, Victor S Sheng, Tao Li, and Xindong Wu. 2017.  Improving crowdsourced label quality using noise correction.  _IEEE transactions on neural networks and learning systems_ , 29(5):1675–1688.

##  Appendix A Overview of Subjective Task Datasets

This appendix gives a selective overview of how existing NLP dataset work has
(or has not) engaged with annotator subjectivity. For reasons of scope, we
focus on 11 English-language datasets annotated for hate speech and other
forms of abuse. Entries are sorted from most descriptive to most prescriptive
annotation, based on our assessment of information made available by the
dataset creators.

Sap et al. (2019) and Sap et al. (2021) annotate toxicity. They do not state
explicitly that they encourage annotator subjectivity, but their annotation
prompts clearly do. Each entry is labelled by up to 641 annotators. Overall,
they are very aligned with the descriptive paradigm.

Kumar et al. (2021) annotate toxicity and types of toxicity. They do not state
explicitly that they encourage annotator subjectivity, but their annotation
prompts clearly do. Each entry is labelled by five annotators. Overall, they
are very aligned with the descriptive paradigm.

Cercas Curry et al. (2021) annotate abuse. They gather ‘views of expert
annotators’ based on guidelines that allow for significant subjectivity and do
not attempt to resolve disagreements, but also do not explicitly encourage
annotator subjectivity. On average, each entry is labelled by around three
annotators. Overall, they are moderately aligned with the descriptive
paradigm.

Talat and Hovy (2016) annotate hate speech. They provide annotators with 11
fine-grained criteria for hate speech, but several criteria invite subjective
responses (e.g., ‘uses a problematic hashtag’). Each entry is labelled by up
to three annotators. Overall, they are not clearly aligned with either
paradigm.

Davidson et al. (2017) annotate hate speech. They provide annotators with a
brief definition of hate speech and an explanatory paragraph, but their
definition also includes subjective criteria like perceived ‘intent’. Most
entries are labelled by three annotators. Overall, they are not clearly
aligned with either paradigm.

Zampieri et al. (2019) annotate offensive content. They provide annotators
with some formal criteria for offensiveness (e.g., ‘use of profanity’), but as
a whole their guidelines are very brief. Each entry is labelled by up to three
annotators. Overall, they are moderately aligned with the prescriptive
paradigm.

Founta et al. (2018) annotate abuse. They provide annotators with fine-grained
definitions for each category and iterate on their taxonomy to facilitate more
agreement, but do not share comprehensive guidelines. Each entry is labelled
by five annotators. Overall, they are moderately aligned with the prescriptive
paradigm.

Caselli et al. (2020) annotate abuse. They provide annotators with a brief
fine-grained decision tree with the explicit intent of reducing annotator
subjectivity, and discuss disagreements to resolve them. Each entry is
labelled by up to three annotators. Overall, they are moderately aligned with
the prescriptive paradigm.

Vidgen et al. (2021b) annotate hate speech. They provide annotators with fine-
grained definitions for each category as well as very detailed annotation
guidelines, and disagreements are resolved by an expert. Each entry is
labelled by up to three annotators. Overall, they are very aligned with the
prescriptive paradigm.

Vidgen et al. (2021a) annotate abuse. They provide annotators with fine-
grained definitions for each category as well as very detailed annotation
guidelines, and they use expert-driven group adjudication to resolve
disagreements. Each entry is labelled by up to three annotators. Overall, they
are very aligned with the prescriptive paradigm.

##  Appendix B Data Statement

Following Bender and Friedman (2018), we provide a data statement, which
documents the generation and provenance of the dataset used for our annotation
experiment.

#### A. CURATION RATIONALE

To create our dataset, we sampled 200 Twitter posts from a larger corpus
annotated for hateful content by Davidson et al. (2017). Of the posts we
sampled, 100 were originally annotated as hateful and 100 as non-hateful by
majority vote between three annotators. We sampled only from those posts that
had some disagreement among their annotators (i.e., two out of three rather
than unanimous agreement), to encourage disagreement in our experiment. The
purpose of our 200-post dataset is to enable the annotation experiment
presented in §1, which illustrates the contrast between the descriptive and
prescriptive data annotation paradigms.

#### B. LANGUAGE VARIETY

The dataset contains English-language text posts only.

#### C. SPEAKER DEMOGRAPHICS

All speakers are Twitter users. Davidson et al. (2017) do not share any
information on their demographics.

#### D. ANNOTATOR RECRUITMENT

We recruited three groups of 20 annotators using Amazon’s Mechanical Turk
crowdsourcing marketplace.333https://www.mturk.com/. Annotators were made
aware that the task contained instances of offensive language before starting
their work, and they could withdraw at any point throughout the work.

#### E. ANNOTATOR DEMOGRAPHICS

All annotators were at least 18 years old when they started their work, and we
recruited only annotators that were based in the UK. This was to facilitate
comparability across groups of annotators. For each group, we recruited 10
male and 10 female annotators, based on self-reported gender. This was to
encourage disagreement within groups, based on the assumption that men would
on average disagree more about hateful content with women than with other men,
and vice versa. No further annotator demographics were recorded.

#### F. ANNOTATOR COMPENSATION

All annotators were compensated for their work at a rate of at least £16 per
hour. The rate was set 50% above the London living wage (£10.85), although all
work was completed remotely.

#### G. SPEECH SITUATION

All entries in our dataset were originally posted to Twitter and then
collected by Davidson et al. (2017), who do not share when the posts were
made.

#### H. TEXT CHARACTERISTICS

All entries in our dataset are individual Twitter text posts, with a length of
140 characters or less. We perform only minimal text cleaning, replacing user
mentions (e.g., "@Obama") with "[USER]" and URLs with "[URL]".

#### I. LICENSE

Davidson et al. (2017) make the Twitter data they collected available for
further research use via GitHub under an MIT
license.444https://github.com/t-davidson/hate-speech-and-offensive-language
Our re-annotated subset of the data is made available under CC0-1.0 license at
github.com/paul-rottger/annotation-paradigms, so that the results of our
experiment can be reproduced.

#### J. ETHICS APPROVAL

We received approval for our experiment and the data annotation it entailed
from our institution’s ethics review board.

##  Appendix C Annotation Prompts

The three groups of annotators in our experiment all annotated the same data
in the same order, but each group received different annotation prompts. The
full annotation guidelines for G2 are available at github.com/paul-
rottger/annotation-paradigms.

#### G1 - Descriptive Group

“Imagine you come across the post below on social media. Do you personally
feel this post is hateful? We want to understand your own opinions, so try to
disregard any impressions you might have about whether other people would find
it hateful.”

#### G2 - Prescriptive Group

“Imagine you come across the post below on social media. Does this post meet
the criteria for hate speech? We are trying to collect objective judgments, so
try to disregard any feelings you might have about whether you personally find
it hateful.

Click here to view the criteria: LINK”

#### G3 - Control Group

“Imagine you come across the post below on social media. Does this post meet
the criteria for hate speech? A post is considered hate speech if it is 1)
abusive and 2) targeted against a protected group (e.g., women) or at its
members for being a part of that group.”


# Salient Object Detection via Augmented Hypotheses

Tam V. Nguyen  Jose Sepulveda
Department for Technology, Innovation and Enterprise
Singapore Polytechnic
{nguyen_van_tam<EMAIL_ADDRESS>
###### Abstract

In this paper, we propose using augmented hypotheses which consider
objectness, foreground and compactness for salient object detection. Our
algorithm consists of four basic steps. First, our method generates the
objectness map via objectness hypotheses. Based on the objectness map, we
estimate the foreground margin and compute the corresponding foreground map
which prefers the foreground objects. From the objectness map and the
foreground map, the compactness map is formed to favor the compact objects. We
then derive a saliency measure that produces a pixel-accurate saliency map
which uniformly covers the objects of interest and consistently separates
fore- and background. We finally evaluate the proposed framework on two
challenging datasets, MSRA-1000 and iCoSeg. Our extensive experimental results
show that our method outperforms state-of-the-art approaches.

##  1 Introduction

The ultimate goal of salient object detection is to search for salient objects
which draw human attention on the image. The research has shown that
computational models simulating low-level stimuli-driven attention Koch and
Ullman (1985); Itti et al. (1998) are quite successful and represent useful
tools in many practical scenarios, including image resizing Achanta et al.
(2009), attention retargeting Nguyen et al. (2013a), dynamic captioning Nguyen
et al. (2013b), image classification Chen et al. (2012) and action recognition
Nguyen et al. (2015). The existing methods can be classified into
biologically-inspired and computationally-oriented approaches. On the one
hand, works belonging to the first class Itti et al. (1998); Cheng et al.
(2011) are generally based on the model proposed by Koch and Ullman Koch and
Ullman (1985), in which the low-level stage processes features such as color,
orientation of edges, or direction of movement. One example of this model is
the work by Itti et al. Itti et al. (1998), which use a Difference of
Gaussians approach to evaluate those features. However, the resulting saliency
maps are generally blurry, and often overemphasize small, purely local
features, which renders this approach less useful for applications such as
segmentation, detection, etc Cheng et al. (2011).

Figure 1: From top to bottom: original images, the objectness hypotheses,
results of our saliency computation, and ground truth labeling. For a better
viewing, only 40 object hypotheses are displayed in each image. Figure 2:
Saliency maps computed by our proposed AH method (t) and state-of-the-art
methods (a-r), salient region detection (AC Achanta et al. (2008)), attention
based on information maximization (AIM Bruce and Tsotsos (2005)), context-
aware (CA Goferman et al. (2010)), frequency-tuned (FT Achanta et al. (2009)),
graph based saliency (GB Harel et al. (2006)), global components (GC Cheng et
al. (2013)), global uniqueness (GU Cheng et al. (2013)), global contrast
saliency (HC and RC Cheng et al. (2011)), spatial temporal cues (LC Zhai and
Shah (2006)), visual attention measurement (IT Itti et al. (1998)), maximum
symmetric surround (MSS Achanta and Süsstrunk (2010)), fuzzy growing (MZ Ma
and Zhang (2003)), saliency filters (SF Perazzi et al. (2012)), induction
model (SIM Murray et al. (2011)), spectral residual (SR Hou and Zhang (2007)),
saliency using natural statistics (SUN Zhang et al. (2008)), and the
objectness map (s). Our result (t) focuses on the main salient object as shown
in ground truth (u).

On the other hand, computational methods relate to typical applications in
computer vision and graphics. For example, frequency space methods Hou and
Zhang (2007) determine saliency based on spectral residual of the Fourier
transform of an image. The resulting saliency maps exhibit undesirable
blurriness and tend to highlight object boundaries rather than its entire
area. Since human vision is sensitive to color, different approaches use local
or global analysis of color contrast. Local methods estimate the saliency of a
particular image region based on immediate image neighborhoods, e.g., based on
dissimilarities at the pixel-level Ma and Zhang (2003) or histogram analysis
Cheng et al. (2011). While such approaches are able to produce less blurry
saliency maps, they are agnostic of global relations and structures, and they
may also be more sensitive to high frequency content like image edges and
noise. In a global manner, Achanta et al. (2009) achieves globally consistent
results by computing color dissimilarities to the mean image color. Murray et
al. Murray et al. (2011) introduced an efficient model of color appearance,
which contains a principled selection of parameters as well as an innate
spatial pooling mechanism. There also exist different patch-based methods
which estimate dissimilarity between image patches Goferman et al. (2010);
Perazzi et al. (2012). While these algorithms are more consistent in terms of
global image structures, they suffer from the involved combinatorial
complexity, hence they are applicable only to relatively low resolution
images, or they need to operate in spaces of reduced image dimensionality
Bruce and Tsotsos (2005), resulting in loss of salient details.

(a) Original image

(b) Hypotheses

(c) Objectness

(d) Margin

(e) Foreground

(f) Compactness

(g) Saliency map

Figure 3: Illustration of the main phases of our algorithm. The object
hypotheses are generated from the input image. The objectness map is later
formed by accumulating all hypotheses. The foreground map is then created from
the difference between the pixel’s color and the background color obtained
following the estimated margins. We then oversegment the image into
superpixels and compute the compactness map based on the spatial distribution
of superpixels. Finally, a saliency value is assigned to each pixel.

Despite many recent improvements, the difficult question is still whether “the
salient object is a real object”. That question bridges the problem of salient
object detection into the traditional object detection research. In the latter
object detection problem, the efficient sliding window object detection while
keeping the computational cost feasible is very important. Therefore, there
exist numerous objectness hypothesis generation methods proposing a small
number (e.g. 1,000) of category-independent hypotheses, that are expected to
cover all objects in an image Lampert et al. (2008); Alexe et al. (2012);
Uijlings et al. (2013); Cheng et al. (2014). Objectness hypothesis is usually
represented as a value which reflects how likely an image window covers an
object of any category. Lampert et al. Lampert et al. (2008) introduced a
branch-and-bound scheme for detection. However, it can only be used to speed
up classifiers that users can provide a good bound on highest score. Alexe et
al. Alexe et al. (2012) proposed a cue integration approach to get better
prediction performance more efficiently. Uijlings et al. Uijlings et al.
(2013) proposed a selective search approach to get higher prediction
performance. However, these methods are time-consuming, taking $3$ seconds for
one image. Recently, Cheng et al. Cheng et al. (2014) presented a simple and
fast objectness measure by using binarized normed gradients features which
compute the objectness of each image window at any scale and aspect ratio only
requires a few bit operations. This method can be run 1,000+ times faster than
popular alternatives.

In this work, we investigate applying objectness to the problem of salient
object detection. We utilize the object hypotheses from the objectness
hypothesis generation augmented with foreground and compactness constraint in
order to produce a fast and high quality salient object detector. The
exemplary object hypotheses and our saliency prediction are shown in the
second and the third row of Figure 1, respectively. As we demonstrate in our
experimental evaluation, each of our individual measures already performs
close to or even better than some existing approaches, and our combined method
currently achieves the best ranking results on two public datasets provided by
Achanta et al. (2009); Batra et al. (2010). Figure 2 shows the comparison of
our saliency map to other baselines in literature. The main contributions of
this work can be summarized as follows.

  * •

We conduct the comprehensive study on how the objectness hypotheses affect the
salient object detection.

  * •

We propose the foreground map and compactness map, derived from the objectness
map, which can cover both global and local information of the saliency object.

  * •

Unlike other works in the literature, we evaluate our proposed method on two
challenging datasets in order to know the impact of our work in different
settings.

Figure 4: From left to right: the original image, the object hypotheses and
the corresponding objectness map, the extended object hypotheses and the
corresponding objectness map.

##  2 Methodology

In this section, we describe the details of our augmented hypotheses (AH), and
we show how the objectness measures as well as the saliency assignment can be
efficiently computed. Figure 3 illustrates the overview of our processing
steps.

###  2.1 Objectness Map

In this work, we extract object hypotheses from the input image to form the
objectness map. We assume that the salient objects attract more object
hypotheses than other parts in the image. As aforementioned, the objectness
hypothesis generators propose a small number $n_{p}$ (e.g. 1,000) of category-
independent hypotheses, that are expected to cover all objects in an image.
Each hypothesis $P_{i}$ has coordinate $(l_{i},t_{i},r_{i},b_{i})$, where
$l_{i},t_{i}$ are the coordinate of the top left point, whereas $r_{i},b_{i}$
are the coordinate of the bottom right point. Here, we formulate each
hypothesis $P_{i}$ $\in\mathbb{R}^{H\times W}$, where $H$ and $W$ are the
height and the width of the input image $\bm{I}$, respectively. The value of
each element $P_{i}(x,y)$ is defined as:

$P_{i}(x,y)=\begin{cases}1\quad\phantom{\infty}\text{if}\,\,t_{i}\leq x\leq
b_{i}\,\,\text{and}\,\,l_{i}\leq y\leq r_{i}\\\
0\quad\phantom{\infty}otherwise\end{cases}.$ (1)

The objectness map is constructed by accumulating all object hypotheses:

$OB(x,y)=\sum_{i=1}^{n_{p}}P_{i}(x,y).$ (2)

The objectness map is later rescaled into the range [0..1]. We observe that
the objectness map discourages the object parts locating close to the image
boundary. Thus we extend the original image by embedding an image border with
the size is $10\%$ of the original image’s size. The addition image border is
filled with the mean color of the original image. We perform the hypothesis
extraction and compute the objectness map similar to the aforementioned steps.
The final objectness map is cropped to the size of the original image. Figure
4 demonstrates the effect of our image extension and the shrinkage of the
objectness map.

###  2.2 Foreground Map

The salient object tends to be distinctive from its surrounding context. Thus,
we aim to model the background which can facilitate the object localization.
In particular, the foreground map is computed by finding the difference
between the color of the original image and the background image. In order to
model the background, we first localize the salient object by the margin shown
as the red rectangle in Fig 3d. To this end, we compute the accumulate
objectness level by four directions $n_{r}$, namely, top, bottom, left, and
right. For each direction, the accumulated objectness level is bounded by a
threshold $\theta$. To boost this process, we utilize the integral image Viola
and Jones (2001) computed from the objectness map. Finally, there are $n_{r}$,
4 in this work, corresponding rectangles surrounding the salient object. Each
bounding rectangle $r_{i}$ is represented by its mean color $\mu_{r_{i}}$. The
foreground value computed for each pixel $(x,y)$ is computed as follows,

$FG(x,y)=\prod_{i=1}^{n_{r}}\|\bm{I}(x,y)-\bm{\mu}_{r_{i}}\|,$ (3)

where $\bm{I}(x,y)$ is the color vector of the pixel $(x,y)$.

###  2.3 Compactness Map

The foreground map prefers the color of the salient object of the foreground.
Unfortunately, it also favors the similar color appearing in the background.
We observe that though the colors belonging to the background will be
distributed over the entire image exhibiting a high spatial variance, the
foreground objects are generally more compact Perazzi et al. (2012).
Therefore, we compute the compactness map in order to remove the noise from
the background. First, we compute the centroid of interest
$(x_{c},y_{c})=(\frac{\sum_{(x,y)}x\times
OF(x,y)}{\sum_{(x,y)}OF(x,y)},\frac{\sum_{(x,y)}y\times
OF(x,y)}{\sum_{(x,y)}OF(x,y)})$, where the objectness-foreground value
$OF(x,y)=OB(x,y)\times FG(x,y)$. Intuitively, the pixel close to the centroid
of interest tends to be more salient, whereas the farther pixels tend to be
less salient. In addition, the saliency value of a certain pixel reduces if
the path between the centroid and that pixel contains many low saliency
values. The naive method is to compute the path from the centroid of interest
to other pixels. However, it is time-consuming to perform this task in the
pixel-level. Therefore, we transform it to superpixel-level. The image is
over-segmented into superpixels, and the $OF$ value of a superpixel is
computed as the average $OF$ values of all containing pixels. The over-
segmented image can be formulated as a graph
$\mathbb{G}=(\mathbb{V},\mathbb{E})$, where $\mathbb{V}$ is the list of
vertices (superpixels) and $\mathbb{E}$ is the list of edges connecting the
neighboring superpixels.

Algorithm 1 Superpixel compactness computation

1:$\bm{l}=\\{v_{c}\\}$.

2:$\bm{c}=\bm{0}\in\mathbb{R}^{n_{sp}}$.

3:$\bm{t}=\emptyset$

4:while $l\neq\emptyset$ do

5: for each vertex $\bm{v}_{i}$ in $\bm{l}$ do

6: for each edge $(\bm{v}_{i},\bm{v}_{j})$ do

7: if $c(v_{j})<\sqrt{c(v_{i})\times OF(v_{j})}$ then

8: $c(v_{j})\leftarrow\sqrt{c(v_{i})\times OF(v_{j})}$

9: $\bm{t}\leftarrow\bm{t}\bigcup\bm{v}_{j}$;

10: end if

11: end for

12: end for

13: $\bm{l}\leftarrow\bm{t}$

14: $\bm{t}=\emptyset$

15:end while

16:return compactness values $\bm{c}$ of superpixels.

The procedure to compute the compactness values of superpixels is summarized
in Algorithm 1. Denote $\bm{v}_{c}$ as the superpixel containing the centroid
of interest. The algorithm transfers the $OF$ value from the $\bm{v}_{c}$ to
all other superpixels. The procedure performs a sequence of relaxation steps,
namely assigning the compactness value $\bm{c}(\bm{v}_{j})$ of superpixel
$\bm{v}_{j}$ by the square root of its neighboring superpixel’s compactness
value and its own $OF$ value. Our algorithm only relaxes edges from vertices
$\bm{v}_{j}$ for which $\bm{c}(\bm{v}_{j})$ has recently changed, since other
vertices cannot lead to correct relaxations. Additionally, the algorithm may
be terminated early when no recent changes exist. Finally, the compactness
value $CN$ is computed as:

$CN(x,y)=\bm{c}(sp(x,y)),$ (4)

where $sp(x,y)$ returns the index of the superpixel containing pixel $(x,y)$.

###  2.4 Saliency Assignment

We normalize the objectness map $OB$, foreground map $FG$, and compactness map
$CN$ to the range [0..1]. We assume that all measures are independent, and
hence we combine these terms as follows to compute a saliency value $S$ for
each pixel:

$S(x,y)=OB(x,y)\times FG(x,y)\times CN(x,y).$ (5)

The resulting pixel-level saliency map may have an arbitrary scale. In the
final step, we rescale the saliency values within [0..1] and to contain at
least 10% saliency pixels.

(a) Fixed threshold

(b) Adaptive threshold

(c) Mean absolute error

Figure 5: Statistical comparison with 18 saliency detection methods using all
the 1000 images from MSRA-1000 dataset Achanta et al. (2009) with pixel
accuracy saliency region annotation: (a) the average precision recall curve by
segmenting saliency maps using fixed thresholds, (b) the average precision
recall by adaptive thresholding (using the same method as in FT Achanta et al.
(2009), SF Perazzi et al. (2012), GC Cheng et al. (2013), etc.), (c) the mean
absolute error of the different saliency methods to ground truth mask. Please
check Figure 2 for the references to the publications in which the baseline
methods are presented.  Figure 6: Visual comparison of saliency maps on iCoSeg
dataset. We compare our method (AH) to other 10 alternative methods. Our
results are close to ground truth and focus on the main salient objects.

###  2.5 Implementation Settings

We apply the state-of-the-art objectness detection technique, i.e., binarized
normed gradients (BING) Cheng et al. (2014), to produce a set of candidate
object windows. Our selection of BING is two-fold. First, BING extractor has a
weak training from the simple feature, e.g., binarized normed gradients.
Therefore, it is useful comparing to bottom-up edge extractor. Second, the
BING extractor is able to run 10 times faster than real-time, i.e., 300 frames
per second (fps). BING hypothesis generator is trained with VOC2007 dataset
Everingham et al. (2010) same as in Cheng et al. (2014). In order to compute
the foreground map, $\theta$ is set as $0.1$ and we convert the color channels
from RGB to Lab color space as suggested in Achanta et al. (2009); Perazzi et
al. (2012). Regarding the image over-segmentation, we use SLIC Achanta et al.
(2012) for the superpixel segmentation. We set the number of superpixels as
$100$ as a trade-off between the fine over-segmentation and the processing
time.

##  3 Evaluation

###  3.1 Datasets and Evaluation Metrics

We evaluate and compare the performances of our algorithm against previous
baseline algorithms on two representative benchmark datasets: the MSRA 1000
salient object dataset Achanta et al. (2009) and the Interactive
cosegmentation Dataset (iCoSeg) Batra et al. (2010). The MSRA-1000 dataset
contains 1,000 images with the pixel-wise ground truth provided by Achanta et
al. (2009). Note that each image in this dataset contains a salient object.
Meanwhile, the iCoSeg contains 643 images with single or multiple objects in a
single image.

The first evaluation compares the precision and recall rates. High recall can
be achieved at the expense of reducing the precision and vice-versa so it is
important to evaluate both measures together. In the first setting, we compare
binary masks for every threshold in the range [0..255]. In the second setting,
we use the image dependent adaptive threshold proposed by Achanta et al.
(2009), defined as twice the mean saliency of the image:

$T_{a}=\frac{2}{W\times H}\sum_{(x,y)}S(x,y).$ (6)

In addition to precision and recall we compute their weighted harmonic mean
measure or $F-measure$, which is defined as:

$F_{\beta}=\frac{(1+\beta^{2})\times Precision\times Recall}{\beta^{2}\times
Precision+Recall}.$ (7)

As in previous methods Achanta et al. (2009); Cheng et al. (2013); Perazzi et
al. (2012), we use $\beta^{2}$ = 0.3.

For the second evaluation, we follow Perazzi et al. Perazzi et al. (2012) to
evaluate the mean absolute error (MAE) between a continuous saliency map S and
the binary ground truth G for all image pixels $(x,y)$, defined as:

$MAE=\frac{1}{W\times H}\sum_{(x,y)}{|S(x,y)-G(x,y)|}.$ (8)

(a) Fixed threshold

(b) Adaptive threshold

(c) Mean absolute error

Figure 7: Statistical comparison with 10 saliency detection methods using all
the 643 images from iCoSeg benchmark Batra et al. (2010) with pixel accuracy
saliency region annotation: (a) the average precision recall curve by
segmenting saliency maps using fixed thresholds, (b) the average precision
recall by adaptive thresholding (using the same method as in FT Achanta et al.
(2009), GC Cheng et al. (2013), etc.), (c) the mean absolute error of the
different saliency methods to ground truth mask.

###  3.2 Performance on MSRA1000 dataset

Following Achanta et al. (2009); Perazzi et al. (2012); Cheng et al. (2013),
we first evaluate our methods using a precision/recall curve which is shown in
Figure 5. Our work reaches the highest precision/recall rate over all
baselines. As a result, our method also obtains the best performance in terms
of F-measure. We also evaluate the individual components in our system,
namely, objectness map (OB), foreground map (FG), and compactness map (CN).
They generally achieve the acceptable performance which is comparable to other
baselines. The performance of the objectness map itself is outperformed by our
proposed augmented hypotheses. In this work, our novelty is that we adopt and
augment the conventional hypotheses by adding two key features: foregroundness
and compactness to detect salient objects. When fusing them together, our
unified system achieves the state-of-the-art performance in every single
evaluation metric.

As discussed in the SF Perazzi et al. (2012) and GC Cheng et al. (2013),
neither the precision nor recall measure considers the true negative counts.
These measures favor methods which successfully assign saliency to salient
pixels but fail to detect non-salient regions over methods that successfully
do the opposite. Instead, they suggested that MAE is a better metric than
precision recall analysis for this problem. As shown in Figure 5c, our work
outperforms the state-of-the-art performance Cheng et al. (2013) by 24%. One
may argue that a simple boosting of saliency values similar as in Perazzi et
al. (2012) results would improve it. However, a boosting of saliency values
could easily result in the boosting of low saliency values related to
background that we also aim to avoid.

###  3.3 Performance on iCoSeg dataset

The iCoSeg dataset is “less popular” in the sense that some baselines do not
even release detection results and source-code. We only reproduced 10 methods
on iCoSeg thanks to their existing source-code. The visual comparison of
saliency maps generated from our method and different baselines are
demonstrated in Figure 6. Our results are close to ground truth and focus on
the main salient objects. We first evaluate our methods using a
precision/recall curve which is shown in Figure 7a, b. Our method outperforms
all other baselines in both two settings, namely fixed threshold and adaptive
threshold. As shown in Figure 7c, our method achieves the best performance in
terms of MAE. Our work outperforms other methods by a large margin, 25%.

###  3.4 Computational Efficiency

It is also worth investigating the computational efficiency of different
methods. In Table 1, we compare the average running time of our approach to
the currently best performing methods on the benchmark images. We compare the
performance of our method in terms of speed with methods with most competitive
accuracy (GC Cheng et al. (2013), SF Perazzi et al. (2012)). The average time
of each method is measured on a PC with Intel i7 $3.3$ GHz CPU and $8$GB RAM.
Performance of all the methods compared in this table are based on
implementations in C++ and MATLAB. The CA method the slowest one because it
requires an exhaustive nearest-neighbor search among patches. Meanwhile, our
method is able to run in a real-time manner. Our procedure spends most of the
computation time on generating the objectness map (about 35%) and forming the
compactness map (about 50%). From the experimental results, we find that our
algorithm is effective and computationally efficient.

Table 1: Comparison of running times in the MSRA 1000 benchmark Achanta et al. (2009). Method |  CA |  RC |  SF |  GC |  Ours
---|---|---|---|---|---
Time (s) | 51.2 | 0.14 | 0.15 | 0.09 | 0.07
Code | Matlab | C++ | C++ | C++ | C++

##  4 Conclusion and Future Work

In this paper, we have presented a novel method, augmented hypotheses (AH),
which adopts the object hypotheses in order to rapidly detect salient objects.
To this end, three maps are derived from object hypotheses: superimposed
hypotheses form an objectness map, a foreground map is computed from
deviations in color from the background, and a compactness map emerges from
propagating saliency labels in the oversegmented image. These three maps are
fused together to detect salient objects with sharp boundaries. Experimental
results on two challenging datasets show that our results are 24% - 25% better
than the previous best results (compared against 10+ methods in two different
datasets), in terms of mean absolute error while also being faster.

For future work, we aim to investigate more sophisticated techniques for
objectness measures and integrate more cues, i.e., depth Lang et al. (2012)
and audio Chen et al. (2014) information. Also, we would like to study the
impact of salient object detection into the object hypothesis process.

##  5 Acknowledgments

This work was supported by Singapore Ministry of Education under research
Grants MOE2012-TIF-2-G-016 and MOE2014-TIF-1-G-007.

## References

  * Achanta and Süsstrunk [2010] Radhakrishna Achanta and Sabine Süsstrunk.  Saliency detection using maximum symmetric surround.  In ICIP, pages 2653–2656, 2010.
  * Achanta et al. [2008] Radhakrishna Achanta, Francisco J. Estrada, Patricia Wils, and Sabine Süsstrunk.  Salient region detection and segmentation.  In International Conference of Computer Vision Systems, pages 66–75, 2008.
  * Achanta et al. [2009] Radhakrishna Achanta, Sheila S. Hemami, Francisco J. Estrada, and Sabine Süsstrunk.  Frequency-tuned salient region detection.  In CVPR, pages 1597–1604, 2009.
  * Achanta et al. [2012] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurélien Lucchi, Pascal Fua, and Sabine Süsstrunk.  SLIC superpixels compared to state-of-the-art superpixel methods.  T-PAMI, 34(11):2274–2282, 2012.
  * Alexe et al. [2012] Bogdan Alexe, Thomas Deselaers, and Vittorio Ferrari.  Measuring the objectness of image windows.  T-PAMI, 34(11):2189–2202, 2012.
  * Batra et al. [2010] Dhruv Batra, Adarsh Kowdle, Devi Parikh, Jiebo Luo, and Tsuhan Chen.  icoseg: Interactive co-segmentation with intelligent scribble guidance.  In CVPR, pages 3169–3176, 2010.
  * Bruce and Tsotsos [2005] Neil Bruce and John Tsotsos.  Saliency based on information maximization.  In NIPS, 2005.
  * Chen et al. [2012] Qiang Chen, Zheng Song, Yang Hua, ZhongYang Huang, and Shuicheng Yan.  Hierarchical matching with side information for image classification.  In CVPR, pages 3426–3433, 2012.
  * Chen et al. [2014] Yanxiang Chen, Tam V. Nguyen, Mohan S. Kankanhalli, Jun Yuan, Shuicheng Yan, and Meng Wang.  Audio matters in visual attention.  T-CSVT, 24(11):1992–2003, 2014.
  * Cheng et al. [2011] Ming-Ming Cheng, Guo-Xin Zhang, Niloy J. Mitra, Xiaolei Huang, and Shi-Min Hu.  Global contrast based salient region detection.  In CVPR, pages 409–416, 2011.
  * Cheng et al. [2013] Ming-Ming Cheng, Jonathan Warrell, Wen-Yan Lin, Shuai Zheng, Vibhav Vineet, and Nigel Crook.  Efficient salient region detection with soft image abstraction.  In CVPR, pages 1529–1536, 2013.
  * Cheng et al. [2014] Ming-Ming Cheng, Ziming Zhang, Wen-Yan Lin, and Philip H. S. Torr.  BING: Binarized normed gradients for objectness estimation at 300fps.  In CVPR, 2014.
  * Everingham et al. [2010] Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (VOC) challenge.  IJCV, 88(2):303–338, 2010.
  * Goferman et al. [2010] Stas Goferman, Lihi Zelnik-Manor, and Ayellet Tal.  Context-aware saliency detection.  In CVPR, pages 2376–2383, 2010.
  * Harel et al. [2006] Jonathan Harel, Christof Koch, and Pietro Perona.  Graph-based visual saliency.  In NIPS, pages 545–552, 2006.
  * Hou and Zhang [2007] Xiaodi Hou and Liqing Zhang.  Saliency detection: A spectral residual approach.  In CVPR, 2007.
  * Itti et al. [1998] Laurent Itti, Christof Koch, and Ernst Niebur.  A model of saliency-based visual attention for rapid scene analysis.  T-PAMI, 20(11):1254–1259, 1998.
  * Koch and Ullman [1985] C Koch and S Ullman.  Shifts in selective visual attention: towards the underlying neural circuitry.  Hum Neurobiol, 1985.
  * Lampert et al. [2008] Christoph H. Lampert, Matthew B. Blaschko, and Thomas Hofmann.  Beyond sliding windows: Object localization by efficient subwindow search.  In CVPR, 2008.
  * Lang et al. [2012] Congyan Lang, Tam V. Nguyen, Harish Katti, Karthik Yadati, Mohan S. Kankanhalli, and Shuicheng Yan.  Depth matters: Influence of depth cues on visual saliency.  In ECCV, pages 101–115, 2012.
  * Ma and Zhang [2003] Yu-Fei Ma and HongJiang Zhang.  Contrast-based image attention analysis by using fuzzy growing.  In ACM MM, pages 374–381, 2003.
  * Murray et al. [2011] Naila Murray, Maria Vanrell, Xavier Otazu, and C. Alejandro Párraga.  Saliency estimation using a non-parametric low-level vision model.  In CVPR, pages 433–440, 2011.
  * Nguyen et al. [2013a] Tam V. Nguyen, Bingbing Ni, Hairong Liu, Wei Xia, Jiebo Luo, Mohan Kankanhalli, and Shuicheng Yan.  Image re-attentionizing.  Multimedia, IEEE Transactions on, 15(8):1910–1919, 2013.
  * Nguyen et al. [2013b] Tam V. Nguyen, Mengdi Xu, Guangyu Gao, Mohan Kankanhalli, Qi Tian, and Shuicheng Yan.  Static saliency vs. dynamic saliency: a comparative study.  In ACM MM, pages 987–996, 2013.
  * Nguyen et al. [2015] Tam V. Nguyen, Zheng Song, and Shuicheng Yan.  STAP: Spatial-temporal attention-aware pooling for action recognition.  T-CSVT, 2015.
  * Perazzi et al. [2012] Federico Perazzi, Philipp Krähenbühl, Yael Pritch, and Alexander Hornung.  Saliency filters: Contrast based filtering for salient region detection.  In CVPR, pages 733–740, 2012.
  * Uijlings et al. [2013] Jasper Uijlings, Koen van de Sande, Theo Gevers, and Arnold Smeulders.  Selective search for object recognition.  IJCV, 104(2):154–171, 2013.
  * Viola and Jones [2001] Paul A. Viola and Michael J. Jones.  Robust real-time face detection.  In ICCV, page 747, 2001.
  * Zhai and Shah [2006] Yun Zhai and Mubarak Shah.  Visual attention detection in video sequences using spatiotemporal cues.  In ACM MM, pages 815–824, 2006.
  * Zhang et al. [2008] Lingyun Zhang, Matthew H. Tong, Tim K. Marks, Honghao Shan, and Garrison W. Cottrell.  Sun: A bayesian framework for saliency using natural statistics.  Journal of Vision, 8(7), 2008.