# How toxic is antisemitism? Potentials and limitations of automated toxicity
scoring for antisemitic online content

Helena Mihaljević
HTW Berlin
<EMAIL_ADDRESS>
Elisabeth Steffen
HTW Berlin
<EMAIL_ADDRESS>

###### Abstract

The Perspective API, a popular text toxicity assessment service by Google and
Jigsaw, has found wide adoption in several application areas, notably content
moderation, monitoring, and social media research. We examine its potentials
and limitations for the detection of antisemitic online content that, by
definition, falls under the toxicity umbrella term. Using a manually annotated
German-language dataset comprising around 3,600 posts from Telegram and
Twitter, we explore as how toxic antisemitic texts are rated and how the
toxicity scores differ regarding different subforms of antisemitism and the
stance expressed in the texts. We show that, on a basic level, Perspective API
recognizes antisemitic content as toxic, but shows critical weaknesses with
respect to non-explicit forms of antisemitism and texts taking a critical
stance towards it. Furthermore, using simple text manipulations, we
demonstrate that the use of widespread antisemitic codes can substantially
reduce API scores, making it rather easy to bypass content moderation based on
the service’s results.

##  1 Introduction

The current COVID-19 pandemic has been accompanied by an increase in insults,
hostility, and hate speech, often directed at Jews who (once again) have been
singled out as one of the main culprits in times of crises. A recent large
scale study of multiple online platforms reveals that “almost 35% of all posts
mentioning Jews or Jewishness expressed negativity toward Jews”, with toxic
speech against Jews amounting for two to five percent of the posts in some
forums Cohen et al. (2021). With regard to the current ‘infodemic’,
antisemitism is of special relevance as it shares relevant features and is
often deeply intertwined with conspiracy theories: Both are based on
simplifying forms of personification in combination with a Manichean worldview
and the ontological construction of group identities Haury (2002). The
hostility towards Jews and other targeted groups expressed in these narratives
has a negative impact not only on digital spheres but also reaches out to the
real world, amplifying verbal and physical acts of violence. It is thus of
great importance for a variety of stakeholders such as content moderators,
researchers or NGOs monitoring societal developments to have access to tools
for automated detection of antisemitic online content.

Despite the current rise of antisemitic conspiracy theories, and the hateful,
toxic characteristics of antisemitism, the phenomenon is still under-explored
in large-scale research of online content in general, and hate-speech in
particular Steffen et al. (2022). To the best of our knowledge, there are
currently no services for automated detection of antisemitic content. However,
progress has been made in form of datasets, code packages, and production-
ready web services regarding the recognition of other linguistic phenomena
intersecting with antisemitism, such as hate speech and toxic language.
Perspective API, a free service created by Jigsaw and Google’s Counter Abuse
Technology team, is one such widely used technology. It allows the detection
of abusive content by computing scores for different attributes such as
toxicity, insult or threat. Perspective API could thus provide a low-threshold
approach to detect certain forms of antisemitic speech and include it in
monitoring and moderation efforts. This paper aims to explore the
possibilities of the service with regard to this objective.

Accordingly, we address the following research questions:

  * •

RQ1: As how toxic are antisemitic texts rated?

  * •

RQ2: Are encoded manifestations of antisemitism rated as toxic?

  * •

RQ3: Are critiques of antisemitic statements rated as toxic?

  * •

RQ4: How do modifications of antisemitic statements affect their toxicity
score?

We use a set of 3.642 German-language Telegram and Twitter posts published
during the COVID-19 pandemic, annotated in terms of content and stance with
respect to antisemitism. We evaluate different attributes of the Perspective
API that, by definition, should produce higher scores when confronted with
antisemitic texts. We further analyze the scores depending on the subform of
antisemitism and the stance towards it. Finally, we perform adversary attacks
to assess in how far modifications of antisemitic statements influence their
scores.

#### Content Warning

This article contains examples of hateful content including offensive,
insulting and threatening comments targeting Jewish people but also other
individuals and groups frequently targeted by antisemitic hate speech. It
might therefore cause anxiety among members of various population groups.

##  2 Related Work

Similar to other sociolinguistic phenomena such as offensive or abusive
language, toxicity is not uniquely defined across existing research and rather
used as an umbrella term. Horta Ribeiro et al. (2021) refer to it as “socially
undesirable content” that includes “sexist, racist, homophobic, or transphobic
posts, targeted harassment, and conspiracy theories that target racial or
political groups”. In Cohen et al. (2021) it is understood as “blatantly
aggressive and demeaning messages about a group or person, such as
dehumanization, incitement of hatred or discrimination, or justification of
violence”. The authors note that “toxic language includes but is not limited
to hate speech”, but in fact utilize machine learning models developed for
hate speech detection.

Perspective API defines “rude, disrespectful or unreasonable” content that is
“likely to make people leave a discussion” as toxic Google (2022a); Thain et
al. (2017) and provides scores representing the likelihood that a reader will
perceive a text as e.g. toxic. The service is used in a variety of
applications such as The New York Times website and the social news platform
Reddit Google (2022a), while also being applied by social and online media
scholars. It has been used as a pre-filtering method for analyses of
moderation measures on Reddit Horta Ribeiro et al. (2021), investigations of
political online communities Rajadesingan et al. (2020) such as the QAnon
movement Hoseini et al. (2021), or to identify antisemitic and islamophobic
texts on 4chan that are subsequently used for the detection of hateful images
via contrastive learning González-Pizarro and Zannettou (2022).

Perspective API also measures severe toxicity of a text as, roughly speaking,
an even stronger form of toxicity. A severe toxicity score of 0.8 is chosen as
the lower limit in a number of studies to preselect particularly toxic texts
Horta Ribeiro et al. (2021); Hoseini et al. (2021); Rajadesingan et al.
(2020); Zannettou et al. (2020). The toxicity scores provided by Perspective
API have been validated on random manually labeled text samples in e.g. Horta
Ribeiro et al. (2021) and Gehman et al. (2020). Horta Ribeiro et al. (2021)
compared its toxicity scores with results from HateSonar, a tool developed for
the detection of hate speech and offensive language Davidson et al. (2017),
deducing that Perspective API yields better results (however, the evaluation
is based on a rather small sample of data).

Despite its broadness and ambiguity, in the definition of Perspective API and
beyond, the term toxicity encompasses antisemitic speech with its widely
accepted operational definition as “a certain perception of Jews, which may be
expressed as hatred toward Jews” International Holocaust Remembrance Alliance
(2016). However, the design of the service already indicates the possibility
of certain shortcomings with respect to detecting toxic antisemitic texts. As
the developers of the Perspective API themselves point out, the very
definition of toxic language has a subjective character Borkan et al. (2019).
Some labeled datasets used for training the respective models were published
as part of Kaggle competitions to improve models and reduce unintended model
bias Thain et al. (2017); Wulczyn et al. (2017); Borkan et al. (2019).
Labeling by a larger number of crowdworkers is used as a vehicle to make the
dataset and the models trained on it more robust. However, a look at the
annotated data exposes various examples confirming that this is not sufficient
to label antisemitic content as toxic. For instance, the following text was
annotated by 54 crowdworkers, with an average toxicity score of 0.33 Thain et
al. (2017): “The US has finally cut bait on the occultist blood suckers. Obama
and Trump just drop kicked bibi down to size. This has been a long time coming
and that is why the zionists wanted Hitlery to win and start ww3.” Research
has shown that the annotation of antisemitic content poses considerable
difficulties, even for scholars with respective backgrounds Ozalp et al.
(2020); Steffen et al. (2022), thus it is plausible that annotators assess
antisemitic texts in diverging ways. The task is further complicated by the
fact that antisemitism is often expressed implicitly, using codes which
annotators need to be familiar with Jikeli et al. (2019) or additional context
Jikeli et al. (2022) in order to recognize them as antisemitic language.
Furthermore, toxicity can be significantly lowered by undertaking minor
changes such as single character-level insertions or perturbations in words
associated with toxicity (e.g. ‘stupid’ $\rightarrow$ ‘st.upid’), while the
scores remain relatively high, if the statement is negated Hosseini et al.
(2017).

The service has also been shown to be biased with respect to differences in
dialect, computing a significantly higher score for texts in African American
English Sap et al. (2019). Recent work demonstrated that systems tend to
produce false positive bias by overestimating the level of toxicity if
minorities are mentioned Dixon et al. (2018); Hutchinson et al. (2020).
Röttger et al. (2021) developed functional tests for hate speech detection
models and evaluated the Perspective API and three other models. Their results
indicate that all models have critical weaknesses, namely an over-sensitivity
to certain keywords, a common misclassification of non-hateful content (such
as counterspeech), and statements including reclaimed slurs. Furthermore, the
models were biased across the different target groups included in the test
data (women, trans people, gay people, Black people, disabled people, Muslims,
and immigrants; Jews were not included).

##  3 Antisemitic language

As a basic definition, we apply the working definition by the International
Holocaust Remembrance Alliance of antisemitism as “a certain perception of
Jews, which may be expressed as hatred toward Jews” which can be “directed
toward Jewish or non-Jewish individuals and/or their property, toward Jewish
community institutions and religious facilities” International Holocaust
Remembrance Alliance (2016). Among several other narrative strategies this may
include calls for the killing or harming of Jews as well as false,
dehumanizing, demonizing, or stereotyping accusations against Jews or the
power of Jews as a collective.

We extend the working definition to also include certain subforms of
antisemitism we consider as specifically relevant in the context of the
COVID-19 pandemic, namely encoded antisemitism and post-Holocaust
antisemitism.

Encoded forms of antisemitism are statements which do not mention Jews or the
State of Israel, but instead turn generally against presumed or actual
economic or political elites while deploying antisemitic codes or stereotypes,
e.g. narratives holding Bill Gates or ‘Big Pharma’ accountable for inventing
and/or benefiting from the COVID-19 pandemic and the resulting global crisis
instead of explicitly mentioning and accusing Jews as initiators. Common
stereotypes which are by no means exhaustive are: Actually or allegedly Jewish
persons or dynasties such as Rothschild, Rockefeller, George Soros, Mark
Zuckerberg, or Bill Gates; animal metaphors: e.g. octopus, snake, pig, rat;
disease and cancer metaphors such as virus, germ, parasite, cancer; codes
referring to the ‘lying press’ trope (‘Lügenpresse’, ‘Pinocchio-Presse’,
‘Systemmedien’); and codes referring to a financial (Jewish) elite in control
of global events (’financial elite’, ’high finance’, ’East coast’, ’Wall
street’). Note that the occurrence of a single code is typically not
sufficient to label a text as antisemitic and that antisemitic codes can be
articulated consciously as well as unconsciously.

Manifestations of post-Holocaust antisemitism explicitly name Jews as part of
argumentation strategies which instrumentalize the victims of the Holocaust
for a political agenda and at the same time shift the perpetrator-victim
coordinates by undertaking relativizing Holocaust comparisons. In the context
of the COVID-19 pandemic, we encounter forms of post-Holocaust antisemitism in
comparisons or equations of the state measures against the pandemic with the
Nazi persecution of Jews. Common examples are the use of the term
‘Giftspritze’ for COVID-19 vaccinations as a more or less implicit reference
to the illegal and often lethal experiments performed on human beings by the
Nazis, the use of the yellow star with the imprint ‘Ungeimpft’ (unvaccinated)
by which anti-vaccination protesters compare themselves to Jews under the Nazi
regime, and references to known victims of and/or resistance fighters against
the Nazi regime such as Anne Frank or Sophie Scholl.

##  4 Data and methods

Our data was annotated using a comprehensive annotation scheme developed as
part of a research project on online antisemitism and conspiracy narratives in
the context of the COVID-19 pandemic. The scheme consists of two main
categories, antisemitism and conspiracy theory, and sub-labels to specify the
content and stance of a message. The scheme and the annotated dataset are
described in detail in Steffen et al. (2022).111The cited manuscript has been
submitted for publication, thus the dataset and the annotation scheme are not
yet publicly available. Until the publication, all documents and data can be
made available to researchers upon request.

The annotation was performed by a team of nine researches with scientific
backgrounds in political science, sociology, or data science. We annotated a
corpus consisting of a few thousand messages from Telegram and Twitter. While
most of the messages were labeled by a single individual, the annotation
process was continuously reflected in regular discussions and a joint
workshop. We furthermore evaluated inter-annotator reliability on an
additional sample of 445222Of the 500 texts originally selected at random, 55
were excluded because, for example, they were not in German, were too short,
or were incomprehensible. records, yielding Cohen’s kappa of $\kappa=0.84$ and
thus strong agreement for the category antisemitism.

For the experiments on the Perspective API presented here, we use a dataset
consisting of 3.642 texts, with $\sim\\!\\!3.200$ Telegram messages and
$\sim\\!400$ Twitter tweets. Around $19\%$ of all posts were classified as
articulating and/or addressing antisemitism, with $\sim\\!40.6\%$ labeled as
encoded antisemitism, $\sim\\!29\%$ as Post-Holocaust antisemitism, and
$\sim\\!29\%$ as explicit forms of antisemitism. The stance expressed was
predominantly affirmative ($\sim\\!68\%$), while $\sim\\!24.4\%$ of the texts
expressed a critical stance ($\sim\\!24.4\%$), and the fewest were classified
as neutral or uncertain ($\sim\\!7.5\%$).

The two sources differ not only in terms of size but also regarding the
sampling approaches: While we selected tweets from a dataset about the German
‘Querdenken’ movement based on antisemitism-related keywords, the Telegram
messages were sampled from pre-selected channels disseminating conspiracy
theory content as well as critique of the anti-COVID-19 measures in Germany.
It is thus not surprising that the Telegram dataset contains a greater variety
of topics and thus a smaller proportion of antisemitic content ($\sim\\!14\%$)
than the analyzed tweets ($\sim\\!56\%$)333Against this background, we believe
that our results should not lead to the conclusion that antisemitism is
generally more prevalent on Twitter than on Telegram..

At the same time, almost all antisemitism-related texts from Telegram are
affirmative towards antisemitism, while Twitter users in our dataset talk
about antisemitism, but not necessarily support antisemitic worldviews (cf.
Figure 1).

Figure 1: Stance towards antisemitic content: while only $\sim\\!16\%$ of
tweets are classified as affirmative and over $\sim\\!70\%$ as critical,
almost $95\%$ of all Telegram messages with antisemitic content were
classified as affirmative.

Encoded antisemitism makes up for almost half of the antisemitic content in
our Telegram dataset ($\sim\\!\\!49\%$), followed by explicitly articulated
antisemitism ($\sim\\!\\!37\%$), and post-Holocaust antisemitism
($\sim\\!\\!13\%$). On Twitter, post-Holocaust is clearly the dominant subform
with $60\%$, followed by encoded ($\sim\\!\\!23\%$) and explicit antisemitism
($17\%$). We suppose that the prevalence of post-Holocaust antisemitism in our
Twitter data is due to the fact that users critically addressed the German
‘Querdenken’ movement and its comparisons of anti-COVID-19 regulations with
the Nazi regime.

We have retrieved the scores for the attributes insult, identity attack,
threat, toxicity and severe toxicity from Perspective API, since their
definition Google (2022a) shares relevant features with antisemitic language
as defined above (cf. Table 1 in Appendix). The returned score is a value
between 0 and 1 that “indicates how likely it is that a reader would perceive
the comment provided in the request as containing the given attribute” Google
(2022b).

##  5 Results

Our results indicate that our dataset has a strong toxic bias: A median severe
toxicity score of $\sim\\!\\!0.18$ clearly exceeds not only the baseline
Telegram dataset compiled in Hoseini et al. (2021) with a median severe
toxicity score of 0.03 but also their QAnon Telegram dataset (median: 0.07).
The CDF further reveals that only $\sim\\!20\%$ of the texts are assigned a
severe toxicity score lower than 0.1, while this holds for around $60\%$ of
all texts in the mentioned baseline set. While the Telegram subset is almost
identical to all texts regarding its CDF and its median of 0.18, the Twitter
subset is more toxic with a median of 0.29. Overall, the median scores range
between $\sim\\!\\!0.18$ for severe toxicity, and $\sim\\!\\!0.35$ for insult
and identity attack.

###  5.1 Antisemitic content

Texts classified as antisemitic have higher scores than those not classified
as such with respect to all attributes. As shown in Figure 2, the median
scores are around twice as high for texts containing antisemitism, with the
greatest difference for severe toxicity (0.16 versus 0.35) and identity attack
(0.33 versus 0.7). These findings support our hypothesis that texts with
antisemitic content share relevant features with messages classified as
threatening, toxic, etc. by the Perspective API. Furthermore, these types of
texts often contain other insults and threats, which further contributes to
the increase of their scores.

Figure 2: Distribution of scores in relation to the presence of antisemitic
content.

###  5.2 Subforms of antisemitism

Explicit forms of antisemitism rank highest in all categories with medians
between 0.56 and 0.83 (identity attack), while those for encoded forms of
antisemitism range between 0.29 and 0.53. This supports our hypothesis that
Perspective API generally does recognize antisemitic content as toxic, but
finds it more difficult to recognize rather implicit forms of antisemitism.
Texts communicating (about) narratives related to post-Holocaust antisemitism
are ranked very similar to those classified as encoded antisemitism except for
identity attack and severe toxicity, where they yield significantly higher
values (0.53 vs. 0.63 and 0.29 vs. 0.35). The higher score, in particular for
identity attack, might originate from more frequent mentions of the Holocaust
and Nazis, which might also explain why there are no texts with a score of 0
in this category. The fact that most of these texts criticize the antisemitism
of the ‘Querdenken’ movement indicates that these two endpoints do not perform
well on capturing the stance of the messages.

Figure 3: Distribution of scores in relation to the subform of antisemitism,
with significantly higher values for explicitly antisemitic texts,

###  5.3 Stance towards antisemitism

As indicated in the previous exploration, Perspective API might not be able to
differentiate the stance of texts even at a basic level. We thus performed
experiments with single sentences and phrases extracted from our dataset that
address a similar topic but with different stance. One such example is
presented in Table 1. Note that the critical example does not contain any
other kinds of toxic language or hate speech, thus side effects can be ruled
out. The scores for the critical example clearly outnumber the affirmative
text in all categories. Furthermore, all scores are in a remarkably high range
with $\sim\\!0.7$ for insult and $>0.9$ for identity attack. By contrast, even
the highest score for the affirmative example is lower than $\sim\\!0.2$, and
in most of the categories close to 0.

text |  insult |  identity attack |  threat |  toxicity |  severe toxicity
---|---|---|---|---|---
The Holocaust was unique in its contempt for humanity and its consequences for the world community.444German original: Der Holocaust war einzigartig in seiner Menschenverachtung und den Folgen für die Weltgemeinschaft. |  0.67 |  0.93 |  0.85 |  0.79 |  0.89
The Holocaust did not happen.555German original: Der Holocaust hat nicht stattgefunden. |  0.05 |  0.07 |  0.15 |  0.13 |  0.04
Table 1: Expression of the historical significance of the Holocaust is
assessed as particularly toxic, while its negation is rated with very low
scores.

As shown in Figure 4, our statistical exploration confirms the observed
tendency for the entire dataset, with texts taking a critical stance towards
antisemitism ranked highest in almost all categories. Texts with affirmative
or neutral/uncertain stance towards antisemitism are ranked roughly equally.

Figure 4: Texts taking a critical stance are ranked highest in all categories
except threat, with medians between 0.45 and 0.74 (threat: 0.34).

###  5.4 Adversarial attacks: the impact of text manipulations

We are interested in how the Perspective API reacts to explicit mentions of
Jews or Israel in comparison to antisemitic codes. To analyze this, we
conducted two experiments:

First, we added ‘#Israel’ and ‘#Juden’ (‘#Jews’) to all texts in the corpus
labeled as antisemitic ($n=679$). Since prior work indicates an
oversensitivity of hate speech detection models to certain keywords Röttger et
al. (2021), our aim was to evaluate if the presence of words explicitly
related to Jewishness would affect the API’s assessment.

Our second experiment focussed on antisemitic codes frequently observed in
online content Zannettou et al. (2020); Finkelstein et al. (2020). Such codes
allow to express antisemitic worldview without explicitly expressing hatred
against Jewish individuals or communities, thus avoiding social ostracism as
well as platform bans or criminal prosecution. We believe that they can play a
crucial role for spreading antisemitism, since their subtlety hampers their
detection and facilitates their dissemination. To evaluate the API’s
assessment of encoded antisemitism, we replaced words related to Jews or
Jewishness by frequently observed antisemitic codes, namely ‘zionist’,
‘globalist’ and ‘satanist Freemason’666German words: ‘Zionist’, ‘Globalist’,
‘satanistischer Freimaurer’, in texts labeled as explicitly and affirmatively
antisemitic ($n=89$). Examples of replacements are listed in Table 2 in the
Appendix. In addition, we used triple parentheses, a “widely used antisemitic
symbol that calls attention to supposed secret Jewish involvement and
conspiracy” Zannettou et al. (2020), known from online communication in
English-language context.

We focus on the attributes identity attack, toxicity and severe toxicity since
we expect these to be affected most by the performed manipulations.

Figure 5 shows the difference between the scores achieved when adding either
‘#Israel’ or ‘#Juden’ to the end of a text and a text’s original score, in
relation to the length of a text. Clearly, the effect is negatively correlated
with the text length, with longer texts ($>2,000$ characters) being almost not
affected at all by the performed manipulation. Shorter texts, however, can be
heavily affected. In almost all cases, adding one of the two expressions
yields an increased score, which can grow up to, e.g., 0.7 for identity
attack. For all three attributes, the difference is larger when adding
‘#Juden’ than ‘#Israel’. Moreover, the effect is least pronounced for the
attribute toxicity (mean: 0.04 for ‘#Israel’ and 0.08 for ‘#Juden’), and
strongest for identity attack (mean: 0.13 for ‘#Israel’ and 0.2 for ‘#Juden’).
The latter is not so surprising given the fact that both additions are
strongly related to Jewish identity. Assuming a lower threshold of 0.5 for a
text to be further analyzed by scientists or monitored by content moderators,
this would imply an increase from 72% to 92% resp. 96% for identity attack, a
significantly smaller increase from 49% to 57% resp. 68% for toxicity, and a
growth from 40% to 48% resp. 59% for severe toxicity.

Figure 5: Difference between scores of manipulated and original message, where
manipulations consist of adding one of the two string ‘#Israel’ and ‘#Juden’
to the end of the text ($n=679$).

Our second experiment indicates that using antisemitic codes instead of
directly mentioning Jews decreases the scores in most of the cases. Figure 6
shows that the code ‘Globalist’ has the strongest decreasing effect, making it
an attractive term for users interested in disseminating antisemitic content
without being moderated or banned from a discussion. Using the code ‘Zionist’
or adding triple parentheses has some decreasing effects as well, though not
as strong as ‘Globalist’. In some cases, we observed that using codes actually
had an increasing effect on the respective scores. This was mainly the case
for the code ‘satanistischer Freimauer’. We assume that this is due to the
negative connotations of the adjective ‘satanistisch’. The effects of
manipulations depend on the score of the original message, with those already
assessed with a score near 1 being least effected. Interestingly, toxicity is
least affected by the manipulations.

Figure 6: Score of original message vs. manipulated text ($n=89$): replacing
words related to Jewishness by codes decreases the scores in most cases.

Considering again 0.5 as a threshold for all three attributes, the codes
‘globalist’ and ‘zionist’ significantly reduce the number of detected
messages, while the usage of triple parentheses has a rather small reducing
effect, and satanist Freemason as well as appending #Israel or #Juden yield a
clear increase (cf. Table 2).

|  identity attack |  toxicity |  severe toxicity
---|---|---|---
original text |  85 |  69 |  60
Zionist |  84 |  59 |  50
Globalist |  73 |  49 |  38
satanistischer Freimaurer |  86 |  76 |  67
(((…))) |  85 |  66 |  55
#Israel |  87 |  72 |  64
#Juden |  87 |  73 |  67
Table 2: Number of explicitly antisemitic texts with affirmative stance with
scores above $0.5$ ($n=89$) after applying different manipulations.

##  6 Summary

We analyzed Perspective API’s assessment of the toxicity of antisemitic online
content using a German-language Telegram and Twitter dataset. We conducted two
experiments to examine the sensitivity of the API towards the direct mention
of Jews and Israel, compared to cases in which these terms were replaced by
different codes.

Regarding RQ1, texts with antisemitic content were generally scored as more
toxic than texts without such content, with median scores approximately two
times higher. This indicates that texts with antisemitic content share
relevant features with texts rated as toxic by the API. We observed the
greatest differences between the positive and negative class regarding
identity attack and severe toxicity.

Explicit forms of antisemitism are rated as more toxic, more threatening, etc.
than forms of encoded antisemitism. In combination with our findings for RQ1,
this indicates that Perspective API generally does recognize antisemitic
content as toxic, but finds it more difficult to recognize rather implicit
forms of antisemitism.

Texts taking a critical stance towards antisemitism are rated with higher
scores compared to both texts with a neutral/uncertain and an affirmative
stance, the latter two being similarly rated. As demonstrated with two
qualitative examples, a statement clearly expressing Holocaust denial is not
rated as toxic, while a statement clearly critical of the Holocaust receives
very high scores. With respect to RQ3, our findings demonstrate that
Perspective API is not able to differentiate the stance of texts even at a
basic level.

To assess the effect of modifying antisemitic statements, we performed two
experiments. Adding direct mentions of Jews or Israel to the end of a text
resulted in an increase of scores, particularly for shorter posts, indicating
that Perspective API is sensitive to the use of identity-group words without
necessarily taking into account their textual context. In contrast, replacing
direct mentions of Jews resulted in a decrease of scores in most of the cases.
We observed the strongest decreasing effect for the code ‘Globalist’, followed
by ‘Zionist’ and the use of triple parentheses around words like e.g. ‘Jude’
(Jew) or ‘jüdisch’ (Jewish). In some cases, the use of codes actually resulted
in an increase of scores. This was mostly the case for the code
‘satanistischer Freimaurer’ (satanic Freemason), probably due to the negative
connotation of the adjective ‘satanisch’.

##  7 Discussion

Perspective API is already widely used in moderation and monitoring of
comments but also as a tool in the study of online communication, including
antisemitic speech. Our findings indicate that on a basic level, Perspective
API recognizes antisemitic content as toxic. When taking a closer look,
however, our investigation reveals several limitations and critical weaknesses
of the service, both for research and content moderation tasks. While it
reacts to explicit forms of antisemitism, it will most likely miss rather
subtle and implicit forms. According to our results, a lot of texts classified
as antisemitic would be neglected by research projects with a toxicity
threshold of 0.8 for data collection. Even with a lower threshold of 0.5, more
than half of the texts expressing encoded or post-Holocaust antisemitism, and
around a third of explicitly antisemitic texts in our corpus, would not be
considered. This indicates that Perspective API is able to detect only the
most blatant manifestations of antisemitism. This is a severe limitation of
the API considering the implicitness of antisemitism, its often encoded
character, but also regarding forms such as post-Holocaust antisemitism which
primarily function via self-victimization instead of direct attacks against
Jewish individuals or communities.

Furthermore, the API clearly struggles with correct stance interpretation.
This is a critical weakness, for using the service to build research corpora
and even more for the task of content moderation. Our results indicate that
Perspective API is more sensitive to (potentially harsh) critiques of
antisemitism rather than to affirmative antisemitic statements. This calls for
further critical research and evaluation, also with regard to the impact of
this bias e.g. for content moderation in social media, since such a bias
penalizes counter-speech and critical discourse about antisemitism.

Last but not least, our adversarial attacks against the API have demonstrated
that even simple text manipulations can noticeably influence the scores. On
the one hand, the service showed a bias towards the presence of identity-
related keywords such as ‘Juden’ or ‘Israel’, assigning higher scores to texts
where these words were added (cf. Jigsaw (2021); Röttger et al. (2021). This
bias can negatively affect content moderation processes since it skews the
focus towards identity-related phrases independent of their context.

On the other hand, it takes only simple manipulations in order to noticeably
decrease the assigned scores. This makes it rather easy to bypass content
moderation based on the API’s results by using simple and known antisemitic
codes, e.g. replacing terms like Jew with ‘Globalist’. This facilitates the
inconspicuous expression and dissemination of antisemitism and undermine
monitoring efforts as well as moderation policies - a problem which should not
be underestimated regarding the strategical behaviour of users on online
platforms to circumvent regulation and moderation policies assisted by machine
learning technologies. Weimann and Am (2020) have analyzed the ‘new language’
of Right-wing extremists, a language which has partly emerged as a direct
counter-reaction to the research initiative behind Perspective API. “To
prevent violating the abuse policies of social media platforms and also to
avoid detection by automatic systems like Google’s Conversation AI, Far-right
extremists have begun to use code words (a movement termed Operation Google)
and thus a new type of hateful online language appears to be emerging: The
systematic use of innocuous words to stand in for offensive racial slurs.”
Weimann and Am (2020)

This clearly shows that human language is not only dynamic, it is also used
strategically, especially in contested spaces of social and political
communication. Both aspects make it difficult for automated tools to meet
their claims of making those spaces less toxic, and more inclusive. So when
using these tools, be it for research, monitoring, or moderation, we need to
be fully aware of the limitations these tools bring along.

When using the API for research purposes, whether for sample selection or for
analyzing the toxicity of online narratives, we believe that researchers
should not rely solely on the automated assessment provided by the API.
Rather, we recommend a thorough manual review of the obtained results before
processing them further. The same applies to its use for content moderation.
However, evaluating Perspective’s results itself can be a challenging task. We
believe that this requires continuous training for e.g. content moderators to
enable them to recognize antisemitism in its various shapes. Existing
annotation efforts regarding antisemitic online content consistently show that
classifying antisemitism is a complex and challenging task that depends on
many factors such as the context of a message or the background knowledge of
the annotators Ozalp et al. (2020); Jikeli et al. (2022). The constant
evolvement of new antisemitic codes adds to this complexity. A profound and
constantly updated knowledge of these codes is thus of crucial importance.
Last but not least, content moderation should be aware of the API’s potential
hypersensitivity towards certain keywords, and to expressions of counter-
speech. Relying on API scoring might result in unintended punishment of
counter-speech, providing another reason for continuous manual sample checks
of results. The hypersensitivity towards keywords might result in unintended
regulation of educational content, as incidents on a number of social
platforms have shown Sales (2021). Furthermore, easily accessible feedback
procedures for users should be provided to allow for early error correction
and monitoring. Last but not least, transparency of the algorithms and tools
used for automated detection needs to be increased, to raise awareness of the
potentials and limitations of such tools, and to foster research into their
strengths and weaknesses so that they can be improved.

## References

  * Borkan et al. (2019) Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019\.  Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification.  In _Companion Proceedings of The 2019 World Wide Web Conference_ , pages 491–500. ACM.
  * Cohen et al. (2021) Katie Cohen, Lisa Kaati, and Björn Pelzer. 2021.  Antisemitism in Social Media. Conspiracies, Stereotypes, and Holocaust Denial.  Technical Report FOI-R–5198–SE, Sweden.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated Hate Speech Detection and the Problem of Offensive Language.  _Proceedings of the International AAAI Conference on Web and Social Media_ , 11(1):512–515.
  * Dixon et al. (2018) Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018.  Measuring and Mitigating Unintended Bias in Text Classification.  In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 67–73. ACM.
  * Finkelstein et al. (2020) Joel Finkelstein, Pamela Paresky, Alex Goldenberg, Savvas Zannettou, Lee Jussim, Denver Riggleman, John Farmer, Paul Goldenberg, Jack Donohue, and Malav H. Modi. 2020.  Antisemitic Disinformation: A Study of the Online Dissemination of Anti-Jewish Conspiracy Theories.  Technical report, The Network Contagion Research Institute.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020\.  RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3356–3369. Association for Computational Linguistics.
  * González-Pizarro and Zannettou (2022) Felipe González-Pizarro and Savvas Zannettou. 2022.  Understanding and Detecting Hateful Content using Contrastive Learning.  _Preprint_.  ArXiv: 2201.08387.
  * Google (2022a) Google. 2022a.  About the API - Attributes and Languages.
  * Google (2022b) Google. 2022b.  About the API - Score.
  * Haury (2002) Thomas Haury. 2002.  _Antisemitismus von links: kommunistische Ideologie, Nationalismus und Antizionismus in der frühen DDR_ , 1 edition.  Hamburger Ed, Hamburg.
  * Horta Ribeiro et al. (2021) Manoel Horta Ribeiro, Shagun Jhaver, Savvas Zannettou, Jeremy Blackburn, Gianluca Stringhini, Emiliano De Cristofaro, and Robert West. 2021.  Do Platform Migrations Compromise Content Moderation? Evidence from r/The_donald and r/Incels.  In _Proceedings of the ACM on Human-Computer Interaction (CSCW2)_ , pages 1–24.
  * Hoseini et al. (2021) Mohamad Hoseini, Philipe Melo, Fabricio Benevenuto, Anja Feldmann, and Savvas Zannettou. 2021.  On the Globalization of the QAnon Conspiracy Theory Through Telegram.  _Preprint_.  ArXiv: 2105.13020.
  * Hosseini et al. (2017) Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017.  Deceiving Google’s Perspective API Built for Detecting Toxic Comments.  _Preprint_.  ArXiv: 1702.08138.
  * Hutchinson et al. (2020) Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020.  Social Biases in NLP Models as Barriers for Persons with Disabilities.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5491–5501, Online. Association for Computational Linguistics.
  * International Holocaust Remembrance Alliance (2016) International Holocaust Remembrance Alliance. 2016.  The working definition of antisemitism.
  * Jigsaw (2021) Jigsaw. 2021.  Unintended Bias and Identity Terms.
  * Jikeli et al. (2019) Gunther Jikeli, Damir Cavar, and Daniel Miehling. 2019.  Annotating Antisemitic Online Content. Towards an Applicable Definition of Antisemitism.  _Preprint_.  ArXiv: 1910.01214.
  * Jikeli et al. (2022) Günther Jikeli, Damir Cavar, Weejeong Jeong, Daniel Miehling, Pauravi Wagh, and Denizhan Pak. 2022.  Toward an AI Definition of Antisemitism?  In Monika Hübscher and Sabine Von Mering, editors, _Antisemitism on Social Media_ , 1 edition, pages 193–212. Routledge.
  * Ozalp et al. (2020) Sefa Ozalp, Matthew L. Williams, Pete Burnap, Han Liu, and Mohamed Mostafa. 2020\.  Antisemitism on Twitter: Collective Efficacy and the Role of Community Organisations in Challenging Online Hate Speech.  _Social Media + Society_ , 6(2).
  * Rajadesingan et al. (2020) Ashwin Rajadesingan, Paul Resnick, and Ceren Budak. 2020.  Quick, Community-Specific Learning: How Distinctive Toxicity Norms Are Maintained in Political Subreddits.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 14, pages 557–568.
  * Röttger et al. (2021) Paul Röttger, Bertram Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet B. Pierrehumbert. 2021.  HateCheck: Functional Tests for Hate Speech Detection Models.  _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 41–58.  ArXiv: 2012.15606.
  * Sales (2021) Ben Sales. 2021.  Are social media platforms banning holocaust education along with hate speech? _The Times of Israel_.  (Accessed on 08/22/2022).
  * Sap et al. (2019) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019.  The Risk of Racial Bias in Hate Speech Detection.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1668–1678, Florence, Italy. Association for Computational Linguistics.
  * Steffen et al. (2022) Elisabeth Steffen, Helena Mihaljević, Milena Pustet, Nyco Bischoff, María do Mar Castro Varela, Yener Bayramoğlu, and Bahar Oghalai. 2022.  Codes, patterns and shapes of contemporary online antisemitism and conspiracy narratives - an annotation guide and labeled German-language dataset in the context of COVID-19.  _Preprint_.
  * Thain et al. (2017) Nithum Thain, Lucas Dixon, and Ellery Wulczyn. 2017.  Wikipedia Talk Labels: Toxicity. figshare. Dataset.
  * Weimann and Am (2020) Gabriel Professor Weimann and Ari Ben Am. 2020.  Digital Dog Whistles: The New Online Language of Extremism.  _International Journal of Security Studies (Vol. 2 : Iss. 1 , Article 4)_ , page 24.
  * Wulczyn et al. (2017) Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017.  Ex Machina: Personal Attacks Seen at Scale.  In _Proceedings of the 26th International Conference on World Wide Web_ , pages 1391–1399. International World Wide Web Conferences Steering Committee.
  * Zannettou et al. (2020) Savvas Zannettou, Mai Elsherief, Elizabeth Belding, Shirin Nilizadeh, and Gianluca Stringhini. 2020.  Measuring and Characterizing Hate Speech on News Websites.  In _12th ACM Conference on Web Science_ , pages 125–134, Southampton United Kingdom. ACM.

##  Appendix A Appendix

Score type |  API Definition
---|---
insult |  Insulting, inflammatory, or negative comment towards a person or a group of people.
identity attack |  Negative or hateful comments targeting someone because of their identity.
threat |  Describes an intention to inflict pain, injury, or violence against an individual or group.
toxicity |  A rude, disrespectful, or unreasonable comment that is likely to make people leave a discussion.
severe toxicity |  A very hateful, aggressive, disrespectful comment or otherwise very likely to make a user leave a discussion or give up on sharing their perspective. This attribute is much less sensitive to more mild forms of toxicity, such as comments that include positive uses of curse words.
Table 3: Attributes from Perspective API that, based on the provided definitions, should react with higher scores towards texts with antisemitic content. original word |  triple parentheses |  code: Zionist |  code: Globalist |  code: satanistischer Freimaurer
---|---|---|---|---
Jude |  (((Jude))) |  Zionist |  Globalist |  satanistischer Freimaurer
JÜDISCH! |  (((JÜDISCH!))) |  ZIONISTISCH! |  GLOBALISTISCH! |  SATANISTISCH!
Antifajuden |  (((Antifajuden))) |  Antifazionisten |  Antifaglobalisten |  satanistische Antifafreimaurer
Judenschwein |  (((Judenschwein))) |  Zionistenschwein |  Globalistenschwein |  satanistisches Freimaurerschwein
Table 4: Examples illustrating the modifications performed in texts labeled as
explicitly antisemitic and containing words referring to Jews and Jewishness.