# Language Model Cascades

David Dohan  Winnie Xu  Aitor Lewkowycz  Jacob Austin  David Bieber  Raphael
Gontijo Lopes  Yuhuai Wu  Henryk Michalewski  Rif A. Saurous  Jascha Sohl-
dickstein  Kevin Murphy  Charles Sutton

###### Abstract

Prompted models have demonstrated impressive few-shot learning abilities.
Repeated interactions at test-time with a single model, or the composition of
multiple models together, further expands capabilities. These compositions are
probabilistic models, and may be expressed in the language of graphical models
with random variables whose values are complex data types such as strings.
Cases with control flow and dynamic structure require techniques from
probabilistic programming, which allow implementing disparate model structures
and inference strategies in a unified language. We formalize several existing
techniques from this perspective, including scratchpads / chain of thought,
verifiers, STaR, selection-inference, and tool use. We refer to the resulting
programs as _language model cascades_.

Machine Learning, ICML

##  1 Introduction

Language models (LMs) have demonstrated impressive few-shot learning abilities
(Brown et al., 2020; Chowdhery et al., 2022). This has led to a number of
proposals to use LMs as the basis of informal reasoning, including scratchpads
(Nye et al., 2021), chain of thought prompting (Wei et al., 2022; Wang et al.,
2022), learned verifiers (Cobbe et al., 2021), selection-inference (Creswell
et al., 2022), and bootstrapping (Zelikman et al., 2022). They have also been
applied in formal mathematics settings to guide theorem provers (Polu &
Sutskever, 2020). These methods involve prompting to encourage step-by-step
reasoning, repeated interactions with a single LM, or multiple LMs linked
together, with the models being fine-tuned or prompted in different ways.

In this position paper, we argue that a useful unifying framework for
understanding and extending this disparate body of work is in terms of
probabilistic programming languages (PPL) extended to work with strings,
instead of more atomic data types like integers and floats. That is, we use a
PPL to define a joint probability model on string-valued random variables,
parameterized using LMs, and then condition this model on string-valued
observations in order to compute a posterior over string-valued unknowns,
which we can then infer. We call such a probabilistic program a _language
model cascade_. We show that this framework captures many recent approaches,
and also allows us to tackle more complex multi-step reasoning problems. By
implementing many disparate model structures and inference strategies in a
single framework, we hope that language model cascades will enable the
development of generic procedures to perform inference, tune parameters, and
choose prompts based on end-to-end objectives.111An implementation is
available at model-cascades.github.io

##  2 Related work

There is a rich prior literature on probabilistic programming languages
(PPLs), which extend probabilistic graphical models to support more complex
joint distributions whose size and “shape” can itself be stochastic (e.g., a
graph unrolled for a random number of iterations, until a data-dependent
stopping criterion is met). PPLs extend traditional programming languages with
the ability to sample from distributions and observe values of variables based
on data (i.e. condition the model). The semantics of sample and observe vary
depending on the inference algorithm. For more details, see van de Meent et
al. (2018).

Recently there has been an explosion of interest in large language models,
such as GPT-3 (Brown et al., 2020) and PaLM (Chowdhery et al., 2022). These
can be used for tasks such as “zero-shot” question-answering. In this setting,
we provide the question $Q$ as a prompt to the LM, and then sample answers
from the model, which we denote by $p(A|Q,\theta)$, where $\theta$ are the
pre-trained model parameters. Alternatively, we can compute the MAP answer,
$\hat{A}=\operatorname*{\arg\\!\max}_{A}p(A|Q,\theta)$.

To ensure the model “does the right thing”, we can provide a small training
set of question-answer pairs, $D=\\{(Q^{m},A^{m}):m=1:M\\}$ pairs. This can be
provided as extra context to the model, provided in the text prompt, followed
by sampling from $p(A|Q,D,\theta)$. We refer to this as “few-shot prompting”.
We can also fine-tune the model parameters on $D$ to get $\theta^{\prime}$,
and then sample from $p(A|Q,\theta^{\prime})$.

We can improve performance by introducing an additional auxiliary “thought”
variable, and then extend the model to have the form
$p(A,T|Q)=p(A|T,Q)p(T|Q)$, where each conditional is computed using an LM
which includes its conditioning variables as a part of its input. Work on
scratchpads (Nye et al., 2021) and chain of thought (Wei et al., 2022)
illustrate this, and finetune or prompt the LM to produce this auxiliary
thought before answering.

We typically condition this on a small set $D_{S}$ of $(A^{m},T^{m},Q^{m})$
triples, and optionally a larger set $D_{L}$ of $(A^{m},Q^{m})$ pairs. We then
compute a distribution over answers to a test question using

$\displaystyle\hat{p}(A|Q)=\sum_{T}\hat{p}(A|Q,T)\hat{p}(T|Q)$ (1)

where $\hat{p}(\cdot)=p(\cdot|D_{L},D_{S},\theta)$ is the prior predictive
distribution. (Scratchpad creates its prior predictive by fine-tuning, while
Chain of Thought adds $D_{S}$ to the LM prompt.)

In practice, we cannot sum over all possible strings $T$ in Equation 1. The
most common approach is to compute the MAP estimate
$\hat{T}=\operatorname*{\arg\\!\max}\hat{p}(T)$ using beam search, and then to
approximate the sum over $T$ with this single value. More recently, Self
Consistency (Wang et al., 2022) proposed to sample multiple values for $T$
using forward sampling of $(A,T)$ given $Q$, and then taking the answer $A$
that is most common in this set222This bucketing is practical because most
standard benchmarks have answers that are just a couple words..

PromptChainer (Wu et al., 2022) proposes a visual interface for composing
language models together, specifying control flow and prompting strategies for
each node in a chain. Nodes may query language models or external systems.
Socratic models (Zeng et al., 2022) extends model chaining to the multimodal
setting and demonstrates zero-shot abilities on tasks for which no single
model exists.

The Eliciting Latent Knowledge proposal (ELK, 2022) suggests making latent
variables explicit, modelled using a Bayesian network, to improve
interpretability and safety for advanced AI systems.

Ortega et al. (2021) explains a formalism for LM finetuning with causal
graphical models in order to extend the predictive capabilities of AI agents
towards more adaptive behaviour. They focus on analysing an auto-regressive
action (random variable) prediction scheme in the interactive setting of RL
where a model is simultaneously a generator and predictor of data.

##  3 Cascades

In this section, we show how to create cascades of LMs to tackle various
language-based reasoning problems. A _cascade_ is a probabilistic program that
includes string-valued random variables, sampled from an LM. For example,
Figure 2 is a simple cascade for question answering. Each of the yield
expressions return a string distributed according to the language model
S.333The first argument to S defines a unique name for the random variable,
and the remaining arguments conditions the LM on a string prefix. A variable
may be marked as observed within the program, S(’varname’, obs=’observed
value’), or at inference time This program defines a joint distribution over
the variables question, thought, and answer. Programs with complex control
flow and observations are included in Appendix A.

We implement cascades as a trace-based probabilistic programming language
embedded in Python via effect handlers, inspired by Bingham et al. (2018);
Phan et al. (2019), and via coroutines, inspired by Kochurov et al. (2019). A
pretrained LM is used to parameterize all conditional distributions. A cascade
supports arbitrary control flow and recursion. While the current presentation
is in terms of few-shot prompting of causal language models, we emphasize that
the ideas are immediately applicable to finetuned models, masked LM setting,
and other complex data types including images.

###  3.1 Scratchpads and Chain of thought

As our first example, we show how to represent a chain of thought (Nye et al.,
2021; Wei et al., 2022) as shown in Figure 1 and subsequent graphical model
figures; refer to the corresponding probabilistic programs in Appendix A. We
condition the $A$ node not just on the test question $Q$, but also on previous
$(Q^{m},T^{m},A^{m})$ triples, which constitute the few-shot prompting part of
the model. This is denoted by the shaded nodes inside the plate. Inference can
be implemented by ancestral sampling.

$Q^{m}$$T^{m}$$A^{m}$$M$$Q$$T$$A$ Figure 1: Question-Thought-Answer model.

    def qta():
      q = yield S(’question’)
      t = yield S(’thought’, question=q)
      a = yield S(’answer’, question=q,
                            thought=t)
      return a

Figure 2: Chain of thought cascade in Python. Each yield S(...) statement
samples a string from an LM. The name of the random variable is provided as
the first argument to S.

###  3.2 Semi-supervised learning

In Section 3.1, we provided a manually created set $(Q^{m},T^{m},A^{m})$
triples, where the “thoughts” or “rationalizations” were provided. A more
scalable approach is to define a small set $D_{S}$ of such “supervised”
triples, but then to provide a larger set $D_{L}$ of $(Q^{m},A^{m})$ pairs,
which are easier to gather. We can augment the pairs in $D_{L}$ by adding the
hidden $T^{m}$ variable to get a semi-supervised setup, shown in Figure 3.

$Q$$T$$A$$Q^{m}$$T^{m}$$A^{m}$$M$ Figure 3: QTA model with hidden thoughts.

The Self-Taught Reasoner (STaR) (Zelikman et al., 2022) proposes a procedure
for fine-tuning LMs in the chain-of-thought type setting. We can interpret
their method as a stochastic EM-like procedure in the cascade of Figure 3. In
particular, they first fine-tune on the “fully observed” dataset
$D_{S}=\\{(Q^{m},T^{m},A^{m})\\}$. Then they impute the unknown $T_{i}$ values
in the “partially observed” dataset $D_{L}=\\{(Q^{m},T^{m}=?,A^{m})\\}$ during
the “E” step by doing rejection sampling on $p(T,A|Q^{m})$ until finding a
thought which leads to the known correct answer. If sampling $(T,A)$ given the
question fails to find the correct answer, they sample thoughts from
$p(T|Q^{m},A^{m})$. This uses a recognition network to approximately sample
from the posterior distribution over thoughts given the known correct answer.
They call this approach “rationale generation with rationalization”. They then
update the parameters in the “M” step based on these imputed thoughts. By
interpreting the rationale generation at this higher level of abstraction, we
open up the possibility of applying this tuning method to other types of
cascades.

###  3.3 Selection-Inference

Selection Inference (Creswell et al., 2022) is a recent example of multiple
interacting LM modules. It proposes splitting reasoning into: the selection
module which selects a subset of facts given a question, and the inference
module which infers new facts given this subset.

It may be represented by the model in Figure 4. Here $S$ is the selection of a
subset of “facts” from a pre-specified set of facts, and $I$ is an inference
driven by that fact. The $S$ and $I$ nodes can be iterated to do multistep
reasoning. The model is “trained” by giving it examples,
$D=\\{(Q^{m},\\{F^{mj}\\},S^{m},I^{m},A^{m}):m=1:M\\}$, as part of the prompt.

$Q$$F$FACTS$S$$I$$A$ Figure 4: Selection inference as a cascade. Here $S$ is
the selected subset of facts and $I$ is an inference driven by this subset.

###  3.4 Verifiers

Although adding explicit “thought” variables to a model has been found to
improve performance, models still arrive at incorrect answers, or the correct
answer for an erroneous reason. An intuitive way to improve model performance
is to train it to judge whether an answer and thought are likely to be
“valid”. Cobbe et al. (2021) propose using a separate model as a verifier to
filter solutions to reasoning tasks.

We can create a “labeled” training set of the form
$D=\\{(Q^{m},T^{m},A^{m},V^{m}\\}$, where we add a “verification” label
$V^{m}\in\\{0,1\\}$, representing whether the thought $T^{m}$ is a valid form
of reasoning for deriving $A^{m}$ from $Q^{m}$, and $A^{m}$ is the correct
answer. This can be particularly helpful in settings where there may be more
than one way of deriving the answer. The verifiers may be used to reject
incorrect examples in ancestral sampling, and the thought generator may itself
be conditioned on the verifiers being correct by finetuning or prompting,
reminiscent of RL as inference (Levine, 2018) and goal-conditioned policies
such as decision-transformer (Chen et al., 2021).

$Q$$T_{1}$$T_{2}$$T_{3}$$A$$V_{1}$$V_{2}$$V_{3}$ Figure 5:  Verifier model.
The small double-ringed nodes are deterministic buffer nodes that concatenate
their inputs, accumulating all past strings. All other nodes are stochastic.
The verifiers are observed to take on the “correct” value.

We can extend this to $N$-step reasoning as follows (where we drop
conditioning on $D$ for brevity):

$\displaystyle p(A|Q,V_{1:N}=1)$
$\displaystyle\propto\sum_{T_{1:N}}p(A,T_{1:N},V_{1:N}=1\,|\,Q),$

where

$\displaystyle p(A,T_{1:N},V_{1:N}=1|Q)$
$\displaystyle=\left[\prod_{t=1}^{N}p(T_{t}|T_{1:t-1},Q)p(V_{t}=1|T_{1:t},Q)\right]$
$\displaystyle\times p(A|T_{1:N},Q).$

We can represent this as shown in Figure 5.

To see why such a verification model can be useful, consider (for simplicity)
the case where $N=1$. Suppose we have trained the model to generate valid
thoughts and answers by giving it suitable training examples, and then we
generate $K$ samples $(T^{k},V^{k},A^{k})\sim p(T,V,A|Q,D)$. We can then rank
the samples for validity by computing $r^{k}=p(V^{k}=1|A^{k},Q,D)$, and then
picking the $A^{k}$ with largest score $r^{k}$.

Cobbe et al. (2021) train the verifier to predict a binary correctness label.
Scheurer et al. (2022) incorporates natural language feedback, and finds that
learning is significantly more sample efficient. Preliminary evidence suggests
that LMs are capable of critiquing their own chain of reasoning in language,
in which case the verifier produces natural language and
$p(V_{1:N}=1|Q,A,T_{1:N})$ becomes the likelihood of the verifier taking on a
particular string value, such as $p(V_{1:N}=\text{"The reasoning and solution
are correct."}|...)$. Saunders et al. (2022) study model generated critiques
in the context of summarization.

###  3.5 Tool-use

The applications discussed so far involve iterating a language model, within
some control flow, without external feedback. There are many tasks of interest
in which a model is interacting with external systems. Cobbe et al. (2021) has
an LM use a calculator to solve math tasks, while Nakano et al. (2021) put an
LM in a loop with a web browser to answer questions. Using PPLs to represent
these probabilistic models allows easily representing these cases, by writing
the call to the external tool, such as the calculator, directly into the
program. Then techniques from simulation based inference, for example, can be
applied to do inference in such situations (Cranmer et al., 2020).

###  3.6 Twenty questions

In this section, we discuss experimental results using cascades to solve the
“Twenty Questions” task from BigBench (BIG-bench collaboration, 2021). This
task involves a conversation between two agents, Alice and Bob. Both agents
are presented with the rules of the game, and Alice is additionally presented
with a concept (e.g. ‘apple’) to describe. Bob has to guess the concept by
asking a series of questions $B_{t}$ of the form “Is it X?”, to which Alice
answers $A_{t}\in\\{\text{`Yes.'},\text{`No.'}\\}$. We repeat this process
until Bob guesses correctly, or we hit the limit of $T$ rounds. This can be
thought of as a pair of interacting Markov chains, which exchange strings,
until some final end state is reached, as illustrated in Figure 6.

RULESCONCEPT$B_{1}$$B_{2}$$A_{1}$$A_{2}$. . . Figure 6: Twenty questions.

The goal is to infer what questions Bob should ask to guess the concept as
quickly as possible. This can be cast as a reinforcement learning problem with
string-valued actions, or equivalently as an inference problem where we
condition on the goal state that $A_{T}=\text{`yes'}$ for the soonest possible
$T$ (c.f., planning as inference (Levine, 2018)).

In our current preliminary experiments, we use a forward sampling approach
(aka ancestral sampling), in which we sample 50 conversations per concept with
temperature $1.0$. We consider a trial successful if the target concept
appears in $B_{t}$. (i.e., Bob guesses the right answer). We reject a sampling
chain early if it is “malformed” (e.g., Bob generates a response that is not a
question).

Bob’s turn starts with ‘Is the concept’ which we complete with the LM. Then we
let Alice generate an answer; we post-process Alice’s response by replacing
all mentions of the true concept with the generic word “concept”, to prevent
information leakage. Using the LaMDA 137B large LM (Thoppilan et al., 2022),
we find that the model is able to solve $29\%$ of the tasks. See Appendix B
for more details.

##  4 Discussion

We have shown how probabilistic programming provides a flexible formalism for
composing models together to define complex probabilistic models over strings,
placing many existing algorithms in a unified framework. While this suggests
the possibility of applying a variety of existing inference and train-time
techniques to the resulting models, the present work does not evaluate methods
beyond rejection sampling.

We can also cast many planning and RL tasks in our framework, by using the
perspective of control as inference. While we restrict presentation to the
string setting, the ideas presented here are applicable to multimodal settings
as well, allowing us to combine image and text models into a larger system.

A challenge applying cascades in practice is the difficulty of probabilistic
inference in models with string-valued variables. Previous work in particle
based inference for probabilistic programs provides some hope in this
direction (Tolpin et al., 2016).

The core technical challenge is efficient inference, as is usually the case
with PPLs. A key insight, which we intend to explore in future work, is that
we can emulate posterior inference by training the LM to “fill in the blanks”,
corresponding to the unknown variables. A similar idea is explored in
foundation posteriors (Wu & Goodman, 2022), applied to Stan probabilistic
programs, demonstrating that LMs are applicable to numerical data types as
well. In other words, we can use LMs as proposal distributions, or guide
networks. We also intend to explore fine-tuning methods, going beyond the few-
shot prompting approach described here.

Recent advances in program synthesis suggest the possibility of probabilistic
program induction (Lake et al., 2015; Goodman et al., 2014) to search for
cascades which solve a target task, rather than assuming a fixed probabilistic
program structure.

##  5 Acknowledgements

We thank Alex Gray, Andreas Stuhlmüller, Ben Poole, Du Phan, Ellen Jiang,
Maarten Bosma, Matt Hoffman, Michael Terry, Sharad Vikram, Sherry Tongshuang
Wu,and Tuan Anh Le for helpful discussions.

## References

  * ELK (2022) ARC’s first technical report: Eliciting latent knowledge - AI alignment forum, 2022.  URL https://www.alignmentforum.org/posts/qHCDysDnvhteW7kRd/arc-s-first-technical-report-eliciting-latent-knowledge.
  * BIG-bench collaboration (2021) BIG-bench collaboration.  Beyond the imitation game: Measuring and extrapolating the capabilities of language models.  _In preparation_ , 2021.  URL https://github.com/google/BIG-bench/.
  * Bingham et al. (2018) Bingham, E., Chen, J. P., Jankowiak, M., Obermeyer, F., Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P., Horsfall, P., and Goodman, N. D.  Pyro: Deep universal probabilistic programming.  October 2018.
  * Brown et al. (2020) Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., and Others.  Language models are few-shot learners.  _arXiv preprint arXiv:2005. 14165_ , 2020.
  * Chen et al. (2021) Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I.  Decision transformer: Reinforcement learning via sequence modeling.  June 2021.
  * Chowdhery et al. (2022) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstern, K., Eck, D., Dean, J., Petrov, S., and Fiedel, N.  PaLM: Scaling language modeling with pathways.  April 2022.
  * Cobbe et al. (2021) Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J.  Training verifiers to solve math word problems.  2021\.  doi: 10.48550/ARXIV.2110.14168.  URL https://arxiv.org/abs/2110.14168.
  * Cranmer et al. (2020) Cranmer, K., Brehmer, J., and Louppe, G.  The frontier of simulation-based inference.  _Proc. Natl. Acad. Sci. U. S. A._ , 117(48):30055–30062, December 2020.
  * Creswell et al. (2022) Creswell, A., Shanahan, M., and Higgins, I.  Selection-Inference: Exploiting large language models for interpretable logical reasoning.  May 2022.
  * Goodman et al. (2014) Goodman, N. D., Tenenbaum, J. B., and Gerstenberg, T.  Concepts in a probabilistic language of thought.  (010), 06/2014 2014.
  * Kochurov et al. (2019) Kochurov, M., Carroll, C., Wiecki, T., and Lao, J.  PyMC4: Exploiting coroutines for implementing a probabilistic programming framework.  In _NeurIPS 2019 Workshop Program Transformations_ , 2019.
  * Lake et al. (2015) Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B.  Human-level concept learning through probabilistic program induction.  _Science_ , 350(6266):1332–1338, December 2015\.
  * Levine (2018) Levine, S.  Reinforcement learning and control as probabilistic inference: Tutorial and review.  May 2018.
  * Nakano et al. (2021) Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J.  WebGPT: Browser-assisted question-answering with human feedback.  December 2021.
  * Nye et al. (2021) Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., Sutton, C., and Odena, A.  Show your work: Scratchpads for intermediate computation with language models.  November 2021.
  * Ortega et al. (2021) Ortega, P. A., Kunesch, M., Delétang, G., Genewein, T., Grau-Moya, J., Veness, J., Buchli, J., Degrave, J., Piot, B., Perolat, J., et al.  Shaking the foundations: delusions in sequence models for interaction and control.  _arXiv preprint arXiv:2110.10819_ , 2021.
  * Phan et al. (2019) Phan, D., Pradhan, N., and Jankowiak, M.  Composable effects for flexible and accelerated probabilistic programming in NumPyro.  December 2019.
  * Polu & Sutskever (2020) Polu, S. and Sutskever, I.  Generative language modeling for automated theorem proving.  September 2020.
  * Saunders et al. (2022) Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J.  Self-critiquing models for assisting human evaluators, 2022.  URL https://arxiv.org/abs/2206.05802.
  * Scheurer et al. (2022) Scheurer, J., Campos, J. A., Chan, J. S., Chen, A., Cho, K., and Perez, E.  Training language models with language feedback.  April 2022.
  * Thoppilan et al. (2022) Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H. S., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhao, V., Zhou, Y., Chang, C.-C., Krivokon, I., Rusch, W., Pickett, M., Srinivasan, P., Man, L., Meier-Hellstern, K., Morris, M. R., Doshi, T., Santos, R. D., Duke, T., Soraker, J., Zevenbergen, B., Prabhakaran, V., Diaz, M., Hutchinson, B., Olson, K., Molina, A., Hoffman-John, E., Lee, J., Aroyo, L., Rajakumar, R., Butryna, A., Lamm, M., Kuzmina, V., Fenton, J., Cohen, A., Bernstein, R., Kurzweil, R., Aguera-Arcas, B., Cui, C., Croak, M., Chi, E., and Le, Q.  LaMDA: Language models for dialog applications.  January 2022.  URL https://arxiv.org/abs/2201.08239.
  * Tolpin et al. (2016) Tolpin, D., van de Meent, J., Yang, H., and Wood, F. D.  Design and implementation of probabilistic programming language anglican.  _CoRR_ , abs/1608.05263, 2016.  URL http://arxiv.org/abs/1608.05263.
  * van de Meent et al. (2018) van de Meent, J.-W., Paige, B., Yang, H., and Wood, F.  An introduction to probabilistic programming.  September 2018.
  * Wang et al. (2022) Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D.  Self-Consistency improves chain of thought reasoning in language models.  March 2022.
  * Wei et al. (2022) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D.  Chain of thought prompting elicits reasoning in large language models.  January 2022.
  * Wu & Goodman (2022) Wu, M. and Goodman, N.  Foundation posteriors for approximate probabilistic inference.  May 2022.
  * Wu et al. (2022) Wu, T., Jiang, E., Donsbach, A., Gray, J., Molina, A., Terry, M., and Cai, C. J.  Promptchainer: Chaining large language model prompts through visual programming, 2022.  URL https://arxiv.org/abs/2203.06566.
  * Zelikman et al. (2022) Zelikman, E., Wu, Y., and Goodman, N. D.  Star: Bootstrapping reasoning with reasoning.  _arXiv preprint arXiv:2203.14465_ , 2022.
  * Zeng et al. (2022) Zeng, A., Wong, A., Welker, S., Choromanski, K., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., Lee, J., Vanhoucke, V., and Florence, P.  Socratic models: Composing Zero-Shot multimodal reasoning with language.  April 2022.

##  Appendix A Implementation

###  A.1 Inference

Given a program representing a probabilistic model, inference reifies specific
unobserved values conditioned on observed values. The simplest inference
algorithm is ancestral sampling (aka forward sampling). The basic inference
API is:

    infer(question_thought_answer_critique,
          seed=0,
          # Specify observed variables:
          observe={’question’: ’Alice made 37 dollars selling ...’,
                   ’critique’: ’The reasoning and arithmetic are correct.’},
          # Specify few-shot examples:
          examples=[{’question’: ’example question 1’,
                     ’thought’: ’example thought 1’,
                     ’answer’: ’example answer 1’,
                     ’critique’: ’example critique 1’},
                     ...])

###  A.2 Code examples

In each example below, S is a string distribution. It consists of turning the
input values into a prompt, together with any examples provided as few-shot
examples to the ‘infer’ method, and sampling until some stopping criterion.

The basic question answering graph directly generates the answer given the
question:

    def question_answer():
      q = yield S(’question’)
      a = yield S(’answer’, question=q)
      return a

Chain of thought introduces a latent thought before producing an answer:

    def question_thought_answer():
      q = yield S(’question’)
      t = yield S(’thought’, question=q)
      a = yield S(’answer’, question=q, thought=t)
      return a

Self critique introduces a step in which the model critiques its own reasoning
in natural language:

    def question_thought_answer_critique():
      q = yield S(’question’)
      t = yield S(’thought’, question=q)
      a = yield S(’answer’, question=q, thought=t)
      c = yield S(’critique’, question=q, thought=t, answer=a)
      return a

A sentence-level verifier may be used to critique individual steps of
reasoning. Furthermore, when to halt generation may itself be a random
variable:

    def qta_verifier(max_steps=3):
      q = yield S(’question’)

      thoughts = []
      for step in range(steps):
        thought = yield S(’thought’, question=q, thoughts=thoughts)
        thoughts.append(thought)

        # Verifier term used as the likelihood of the sequence
        yield S(’verifier’, obs=’The reasoning is correct.’,
                question=q, thoughts=thoughts)

        # Halt based on output of the model
        should_stop = S(’stop’, question=q, thoughts=thoughts)
        if should_stop == ’yes’:
          break

      a = yield S(’answer’, question=q, thoughts=thoughts)
      return answer

Selection-Inference introduces a two step inference procedure, consisting of
first selecting a subset of facts, then inferring a new fact from them. Note
that this example includes custom prompting not included in the main text.

    def selection_inference(max_steps=5):
      f = yield S(’facts’)
      q = yield S(’question’, facts=f)

      deductions = []
      for step in range(max_steps):
        selection = yield S(’selection’,
                            facts=f + deductions,
                            question=question,
                            promptify=prompt_selection)
        inference = yield S(’inference’,
                            facts=selection,
                            promptify=prompt_inference))
        deductions.append(inference)

        # Dynamic loop based on output of model.
        should_stop = S(’stop’, question=q, deductions=deductions)
        if should_stop == ’yes’:
          break
      a = yield S(’answer’, question=question, deductions=deductions)
      return a

    # Nodes may have custom prompts:
    def prompt_selection(facts, question, selected=()):
      facts = ’\n- ’.join(facts)
      selected = ’\n- ’.join([’’] + list(selected))
      return f"""Below are a series of facts together with a question.
      Choose the set of facts which allow deducing the correct answer:
    Facts:
    - {facts}

    Question: {question}

    Selected:
    {selected}"""

    def prompt_inference(facts, deduction=’’):
      facts = ’\n- ’.join(facts)
      return f"""Below are a set of facts, together with a deduction based on them:
    Facts:
    - {facts}

    Therefore: {deduction}"""

##  Appendix B More details on Twenty Questions

###  B.1 Problem definition

In this task there are two agents: Alice and Bob. Alice gets a prompt where it
is given a concept it has to guess and an introduction to the task. Bob gets a
prompt where it is instructed on the task. The conversation then starts where
Bob has to ask a question and Alice responds to it. If Alice’s response
includes the key concept, we change it to the word ‘concept‘ (alternatively,
one might reject the trace). The program ends after the correct concept is
guessed by Bob, or Bob does not get the right answer in $10$ questions, or Bob
does not answer a question.

The 40 concepts that we test the model on are: [’apple’, ’television’,
’dinosaur’, ’airplane’, ’house’, ’tree’, ’coat’, ’shoes’, ’car’, ’train’,
’shower’, ’frisbee’, ’cow’, ’cosmic crisp apple’, ’giganotosaurus’, ’siberian
huskey’, ’glass micropipette’, ’jog’, ’catch’, ’defenestrate’, ’eat’,
’apologize’, ’operate’, ’pretend’, ’anger’, ’love’, ’hate’, ’contentment’,
’jealousy’, ’surprise’, ’disgust’, ’hopefulness’, ’global poverty’, ’phase
transition’, ’positive sum game’, ’beauty’, ’representative democracy’,
’tall’, ’mauve’, ’perspicacious’]. The model gets right the 11 of the least
abstract concepts: [’apple’, ’dinosaur’, ’airplane’, ’house’, ’tree’, ’shoes’,
’car’, ’cow’, ’eat’, ’love’, ’beauty’].

###  B.2 Examples

Alice Prompt: X 0 Hello Alice, I am Bob. X 1 Hello Bob X 2 Hello Alice, we are
going to play twenty questions. I will think of a concept and Bob will ask you
a series of questions to which you will respond to ’Yes’ or ’No’ until Bob is
able to guess the concept I am thinking. X 1 Sounds good. What is the concept?
X 2 The concept is ’love’. X 1 The concept is love ? Perfect, I got it. Bob,
what is your first question? Bob Prompt: X 0 Hello Alice, I am Bob. X 1 Hello
Bob, we are going to play twenty questions. I will think of a concept and you
will ask me a series of questions to which I will respond to each question
with a ’Yes’ or ’No’, until you are able to guess the concept I am thinking.
What is your first question? Common Conversation: X 0 Hello Alice, I am Bob. X
1 Hello Bob, we are going to play twenty questions. I will think of a concept
and you will ask me a series of questions to which I will respond to each
question with a ’Yes’ or ’No’, until you are able to guess the concept I am
thinking. What is your first question? X 0 Is the concept a vegetable? X 1 No
X 0 Is the concept a mineral? X 1 No X 0 Is the concept an animal? X 1 No X 0
Is the concept an instrument? X 1 No X 0 Is the concept an abstract concept
such as love or happiness? SUCCESS!

Alice Prompt: X 0 Hello Alice, I am Bob. X 1 Hello Bob X 2 Hello Alice, we are
going to play twenty questions. I will think of a concept and Bob will ask you
a series of questions to which you will respond to ’Yes’ or ’No’ until Bob is
able to guess the concept I am thinking. X 1 Sounds good. What is the concept?
X 2 The concept is ’shoes’. X 1 The concept is shoes ? Perfect, I got it. Bob,
what is your first question? Bob Prompt: X 0 Hello Alice, I am Bob. X 1 Hello
Bob, we are going to play twenty questions. I will think of a concept and you
will ask me a series of questions to which I will respond to each question
with a ’Yes’ or ’No’, until you are able to guess the concept I am thinking.
What is your first question? Common Conversation: X 0 Is the concept something
that is a natural entity? X 1 NO X 0 Is the concept something that was
invented or is man made? X 1 the concept is concept which is something that
was invented so i will answer the question with ’yes’ X 0 Is the concept a
device? X 1 Yes X 0 Is the concept a computer? X 1 The concept is not about a
computer X 0 Is the concept a piece of clothing? X 1 Yes X 0 Is the concept a
shirt? X 1 no :: the concept concept is not a shirt X 0 Is the concept pants?
X 1 No X 0 Is the concept shoes? SUCCESS!

Example of wrong answer. Alice Prompt: X 0 Hello Alice, I am Bob. X 1 Hello
Bob X 2 Hello Alice, we are going to play twenty questions. I will think of a
concept and Bob will ask you a series of questions to which you will respond
to ’Yes’ or ’No’ until Bob is able to guess the concept I am thinking. X 1
Sounds good. What is the concept? X 2 The concept is ’television’. X 1 The
concept is television ? Perfect, I got it. Bob, what is your first question?
Bob Prompt: X 0 Hello Alice, I am Bob. X 1 Hello Bob, we are going to play
twenty questions. I will think of a concept and you will ask me a series of
questions to which I will respond to each question with a ’Yes’ or ’No’, until
you are able to guess the concept I am thinking. What is your first question?
Common Conversation: X 0 Is the concept an animal, vegetable or mineral? X 1
neither, i already know the answer, it’s concept X 0 Is the concept something
used? X 1 Yes X 0 Is the concept a game? X 1 Yes X 0 Is the concept played
with a board? X 1 No X 0 Is the concept from the Monopoly board? X 1 No (to be
alternated) X 0 Is the concept from the Monopoly board? X 1 No X 0 Is the
concept something that is kept in the box? X 1 No X 0 Is the concept a card? X
1 No X 0 Is the concept something that can be held? X 1 no (concept can’t be
held) X 0 Is the concept money? X 1 No REJECT! Bob did not get ”television” in
ten rounds.

###  B.3 Code for twenty questions

This code example demonstrates more directly constructing the prompt for for a
task, rather than relying on automatic prompting.

    def twenty_questions_program(concept, max_questions):
      alice_prompt = get_prompt_from_alice(concept, max_questions)
      bob_prompt = get_prompt_from_bob(concept, max_questions)
      common_conversation = ""
      # iterate over rounds of questions and answers
      for round_number in range(1, max_questions + 1):

        current_turn = "\nX 0 Is the concept"
        # Bob"s generates question. Program will be rejected if it does not generate a question.
        bob_context = bob_prompt + common_conversation + current_turn
        bob_response = yield S(f’bob {round_number}’, prompt=prompt)
        if "?" not in bob_response:
          yield reject(reason=’Bob response is not a question.’)

        current_turn += bob_response + "\nX 1 "

        if concept.lower() in bob_response.replace(’?’,’’).lower().split(’’):
          # Bob figured it out! Score should be equal to round number.
          yield Success(num_rounds)

        # Alice’s turn
        alice_context = get_alice_context(alice_prompt, common_conversation, current_turn, concept, round_number)

        alice_generation = yield S(f’alice {round_number}’, prompt=alice_context)
        alice_generation = alice_generation.split(".")[0].split("\n")[0].split("X")[0]
        # If Alice outputs the key concept, we hide it. An alternative would be to reject.
        if concept.lower() in  alice_generation:
          alice_generation = alice_generation.lower().replace(
                concept.lower(), "concept")

        current_turn += alice_generation
        common_conversation += current_turn

      # Reject if it runs out of time.
      yield reject(reason=’Ran out of turns.’)


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

De-Factify: Workshop on Multimodal Fact Checking and Hate Speech Detection,
co-located with AAAI 2022. 2022 Vancouver, Canada

<EMAIL_ADDRESS>]<EMAIL_ADDRESS>]
<EMAIL_ADDRESS>]<EMAIL_ADDRESS>url=https://www.logically.ai/team/leadership/anil-bandhakavi ]

# LAHM : Large Annotated Dataset for Multi-Domain and Multilingual Hate Speech
Identification

Ankit Yadav  Shubham Chandel  Sushant Chatufale  Anil Bandhakavi
Logically.ai, Brookfoot Mills, Brookfoot Industrial Estate, Brighouse, HD6
2RW, United Kingdom

(2021)

###### Abstract

Current research on hate speech analysis is typically oriented towards
monolingual and single classification tasks. In this paper, we present a new
multilingual hate speech analysis dataset for English, Hindi, Arabic, French,
German and Spanish languages for multiple domains across hate speech - Abuse,
Racism, Sexism, Religious Hate and Extremism. To the best of our knowledge,
this paper is the first to address the problem of identifying various types of
hate speech in these five wide domains in these six languages. In this work,
we describe how we created the dataset, created annotations at high level and
low level for different domains and how we use it to test the current state-
of-the-art multilingual and multitask learning approaches. We evaluate our
dataset in various monolingual, cross-lingual and machine translation
classification settings and compare it against open source English datasets
that we aggregated and merged for this task. Then we discuss how this approach
can be used to create large scale hate-speech datasets and how to leverage our
annotations in order to improve hate speech detection and classification in
general.

###### keywords:

hate speech, multilingual, multi-domain,cross-lingual, racism, religious hate,
sexism, abuse, extremism, few shot learning, zero shot learning,

##  1 Introduction

Abusive language is an important and relevant issue in social media platforms
such as Twitter. Social media is often exploited to propagate toxic content
such as hate speech or other forms of abusive language. The amount of user-
generated content produced every minute is very large, and manually monitoring
abusive behavior in Twitter is not feasible and impractical. Twitter has made
efforts to eliminate abusive content from their platform by providing clear
policies on hateful conduct, user reporting and using moderators to filter
content. Still, these manual efforts are not scalable enough and are not long
term.

Several studies from the Natural Language Processing (NLP) field have been
done to tackle the problem of hate speech detection in social media. Most
studies proposed a supervised approach to detect abusive content automatically
using various models ranging from traditional machine learning approaches to
deep learning based approaches. However, the majority of work focused only on
a single language, i.e., English, and a single abusive domain phenomenon,
e.g., hate speech, sexism, racism, religious hate and so on, rather than
multiple languages and multiple domains. Twitter supports content in 34
languages and user can use any one of them to express views. Thus the problem
of tackling hateful content in real time in multiple languages becomes a
challenge. We need robust models for hateful content detection across multiple
languages and multiple domains.
In this paper we try to tackle two prominent challenges in hate speech
detection-

  1. 1.

Build a multilingual dataset for hate speech detection across 6 languages -:
_English, Hindi, French, Arabic, German, Spanish._

  2. 2.

Build a multi-domain dataset that covers these hate speech domains -:
_Abusive, Racism, Sexism, Religious Hate and Extremism_

We define the different domains as follows-:

  1. 1.

RACISM: Discrimination based on race, ethnicity, caste, nationality, culture,
skin colour, hair texture, physical aspects.

  2. 2.

SEXISM: Discrimination based on gender/sexual orientation.

  3. 3.

RELIGIOUS HATE: Religious discrimination treating a person or group
differently because of the particular faith/belief which they hold about a
religion.

  4. 4.

ABUSE: Speech that causes or likely to cause distress, disrespect or mental
pain, especially from vulgar and profane comments.

  5. 5.

EXTREMISM: Speech that cause or is likely to cause, harm to individuals,
communities or wider society, and where any political, civil issues can lead
to extremist behaviour through violence.

##  2 Related Work

There have been several studies on abusive language detection [1] [2],
offensive language, hate speech identification[3], toxicity [4], hatefulness
[5], aggression [6], attack [7], racism, sexism [8], obscenity, threats, and
insults.

Along with that, there are several shared tasks that have focused on abusive
language and hate speech detection such as HASOC-2019 [9], TRAC shared task on
aggression identification [6], HatEval [10] and GermEval-2018 [11]which
focused on offensive language identification in German tweets.

Waseem[8] proposed the following list to identify hate speech. Their criteria
are partially derived by negating the privileges observed in McIntosh (2003),
where they occur as ways to highlight importance, ensure an audience, and
ensure safety for white people, and partially derived from applying common
sense. A tweet is categorized as offensive if it:

  1. 1.

uses a sexist or racial slur.

  2. 2.

attacks a minority.

  3. 3.

seeks to silence a minority.

  4. 4.

criticizes a minority (without a well founded argument).

  5. 5.

promotes, but does not directly use, hate speech or violent crime.

  6. 6.

criticizes a minority and uses a straw man argument.

  7. 7.

blatantly misrepresents truth or seeks to distort views on a minority with
unfounded claims.

  8. 8.

shows support of problematic hash tags - “#BanIslam”, “#whoriental”,
“#whitegenocide”

  9. 9.

negatively stereotypes a minority.

  10. 10.

defends xenophobia or sexism.

  11. 11.

contains a screen name that is offensive, as per the previous criteria, the
tweet is ambiguous (at best), and the tweet is on a topic that satisfies any
of the above criteria.

Wide variety of machine learning models have been used to deal with multi-
domain hate speech detection task. Some studies use traditional machine
learning approaches such as, logistic regression [12], support vector machine
[13] [14], linear support vector machine classifiers (LSVC) [15] [16] [17].
They used it for better explainability, along with several deep learning based
models, including convolutions neural networks [18] [19], LSTM [20] [18] [17]
[16] [21], bidirectional LSTM [22]. Most recent works focus on transfer
learning and novel architectures involving Transformers based models such as
Bidirectional Encoder Representations from Transformers (BERT) [23] [24] [25]
[26] [27] and its variants like RoBERTa [25] in the cross-domain abusive
language detection task.

For cross-lingual abusive language detection, most studies utilized
transformers based models. Some traditional models used such as logistic
regression [28] [10] [29], linear support vector machines [16] [17], SVM [30],
LSTM [31] [29] and Bi-LSTMs [31] have also been used. Recent work focused on
several transformer based architectures such as multilingual BERT [32] [28]
[25] [17] [33] [34] [29], RoBERTa [35], XLM [31] [34] and XLM-RoBERTa [35]
[25] [36]. Transformers based models with multilingual language
representations can easily deal with language shift in zero-shot cross-lingual
task.

##  3 LAHM Dataset

In this section, we describe the characteristics that we want to include in
our dataset, our approach to collect different types of hate speech while
covering all major hate speech domains and how to annotate data at large
scale. We also give detailed statistics and analysis for the collected data.

Figure 1: Overall Annotation Pipeline Architecture

###  3.1 Essentials of LAHM dataset

Considering no such dataset is available at present that covers these five
domains of hate speech in the six languages, our aim is to create a reliable
multilingual and multifaceted hate speech dataset.

##### Multilingual data:

Our dataset is created as multilingual resource to facilitate cross-lingual
research. It contains hate speech in English, Hindi, French, Arabic, German
and Spanish languages.

##### Multi-Domain data:

Our dataset consists of fine-grained labels for each hate sample per language.
These aspects cover majority of hate domains such as racism, sexism, abuse,
religious hate and extremism.

###  3.2 Dataset Collection

####  3.2.1 Language and Domain Specific Keywords Collection

Considering the cultural differences in the main regions where English, Hindi,
French, Arabic, Spanish and German are spoken, we start by looking for the
hateful keywords that are native to these languages. We use HateBase
vocabulary dataset, which is a valuable lexicon for creating hateful dataset
from public forums, as well as Hatebase’s sightings dataset, which is useful
for trending analysis of keywords.

We targeted 6 languages. Additionally we tried some indic languages mainly
Marathi and Bangla but hatebase has negligible coverage for these languages in
terms of keywords. Additionally, we extracted the following:

  1. 1.

Targeted groups for the keywords

  2. 2.

Offensive levels of keywords - _Extremely, Highly and Mildly offensive._

  3. 3.

Recent sighting counts

Target groups help us categorise keywords in low level classes of ethnicity &
nationality, religious hate, gender & sexual orientation and general abuse.
Details of keywords per domain and language are provided in Table 1 in dataset
statistics. Total keywords collected were 1022 .
To extract data for extremism, we used a set of keywords related to extremism
and terrorism (including terrorist organisation names) to retrieve news
articles from BBC Monitoring111https://monitoring.bbc.co.uk/.

  1. 1.

We further extracted comments from those articles related to extremism and
terrorism. We also extracted tweets and comments which were extremist in
nature from the counterextremism
database222https://www.counterextremism.com/daily-dose-archive.

  2. 2.

The comments extracted were said by members and suspects of terrorist and
extremist organisations on Facebook, Twitter and YouTube.

####  3.2.2 Large Scale Multilingual Dataset of Tweets

We started our dataset collection by using the keywords built per language. We
utilize twint333https://github.com/twintproject/twint/ API to collect
1000-2000 tweets per keyword as Twitter official API can only be utilized for
limited number of requests. We searched through the API for last one year of
data. We also add additional keywords from MLMA [1] for Arabic and French.

The collected raw data contains cross-lingual tweets and therefore language
detection becomes a part of our process. For each keyword in each language we
consider tweets only in that language and drop the rest. This helps us in
training monolingual hate speech detection without any need to worry for code
switching in languages.

We substituted all usernames with @USER and urls with @URL and cleaned any
unnecessary symbols from the tweet. We also discarded short tweets with less
than 4 words. In total we collected 497660 tweets. Details of tweets per
language is given in section 3.3 Dataset Statistics.

For extremism class we carefully handcrafted a total of 88 keywords and
collected the data from BBC monitoring. Additional data was collected from
other websites for extremism.

###  3.3 Dataset Statistics

Table 1 gives us the distribution of keywords across different languages and
different domains. We have merged classes from hate base to arrive at those
domains. Sexism contains keywords that belong to gender and sexual
orientation. Racism includes classes of ethnicity and nationality. Religious
Hate includes words belonging to Religion category on Hatebase
444https://hatebase.org/. Extremism keywords were handcrafted as described in
section 3.2.

English was the dominant language in terms of the total keywords (500)
extracted. Besides Hindi, racism is the major domain for which all languages
have highest number of hate keywords.

Table 1: Keywords language & domain distribution Language | Sexism | Racism | Abuse | Religious Hate | Extremism
---|---|---|---|---|---
English | 69 | 418 | 30 | 33 | 88
Hindi | 75 | 16 | 9 | 8 | -
Arabic | 27 | 35 | 17 | 17 | -
French | 13 | 85 | 13 | 9 | -
German | 21 | 67 | 13 | 11 | -
Spanish | 43 | 77 | 21 | 2 | -
Total | 248 | 698 | 103 | 80 | 88
Table 2: Tweets distribution Languages | Raw | Processed
---|---|---
English | 184339 | 105120
Hindi | 59321 | 32734
Arabic | 27374 | 5394
French | 75126 | 20809
German | 30868 | 8631
Spanish | 120632 | 55148
Total | 497660 | 227836

###  3.4 HSMerge Dataset

####  3.4.1 Dataset Preparation

In order to build the dataset for different tasks with gold labels, we utilize
10 publicly available datasets for different types (tasks) of hate speech
detection. We sampled annotated examples from diverse English datasets: GAO
hatefulness [5], TRAC aggression [6], offensive [37], racism and sexism [8],
MLMA [1], attack [7], CONAN [38], MMHS150k [39].

We map labels from these datasets into different domains: abusive, non-
abusive, sexism, racism and religious hate. For OLID [37] OFFENSIVE maps to
Abuse. GAO and WUL had binary labels, while the original TRAC uses three
labels: non-aggressive, covertly-aggressive, and openly-aggressive. We relabel
the first as non-abusive, and the other two as abusive. For MLMA we create
labels on the basis of target groups: target group RELIGION becomes RELIGIOUS
HATE. For MLMA [1] target group GENDER and SEXUAL ORIENTATION maps to SEXISM,
target group RELIGION becomes RELIGIOUS HATE, target group ORIGIN becomes
RACISM. For HASOC [9] OFFENSIVE maps to ABUSE. For COnan[40] target groups
MUSLIMS, JEWS becomes RELIGIOUS HATE labels, target groups MIGRANTS, POC maps
to RACISM, target group WOMEN, LGBT+ maps to SEXISM and DISABLED maps to
general ABUSE group. For MMHS150K [39] we got RACISM, SEXISM, RELIGIOUS HATE &
OTHERHATE.

For some of these data we had to hydrate tweets from Twitter using official
Twitter API as only the tweet ids had been provided. For Waseem and Founta
[38], majority of the tweets were not available on Twitter which led to less
number of tweets compared to the official dataset. We standardized these
datasets into a single HSMerge dataset. Details of classes and total samples
is available in Table 3.

Table 3: HSMerge open source gold dataset Datasets | Domain | Hate | Non-Hate | Total
---|---|---|---|---
OLID | Offensive / NotHate | 4400 | 8840 | 13240
COnan | Racism/ Sexism/ Religious/ Abuse/ OtherHate | 938/1025/1921/175/179 | - | 4238
TRAC | Hateful / NotHate | 5993 | 4348 | 10341
MLMA | Racism/ Sexism/ Religious/ OtherHate | 2448 / 1152 / 68/1979 | - | 5647
MLA150K | Racism/ Sexism/ Religious/ OtherHate/ NotHate | 11925 / 7365 / 163 / 5811 | 112845 | 138209
HASOC | Hateful / NotHate | 2261 | 3591 | 5852
WASEEM | Racist/Sexist/NotHate | 2753/15 | 7640 | 10408
WUL | Hate/NotHate | 8817 | 62937 | 71754
GAO | Hate / NotHate | 244 | 675 | 909

##  4 Experiments

###  4.1 Overall Methodology and Models

In this section, we describe the various components of our pipeline as shown
in figure 1 and the models used for semi-supervised annotation process in
different settings. Most of the hate speech detection tasks depend on manual
annotation that limits the number of samples that can be labeled. We used a
hierarchical approach to validate and refine our initial keyword based dataset
from Twitter.

  1. 1.

Monolingual hate speech detection models from Hate-ALERT used as zero shot
pipeline to get high level labels of _Hate_ or _NoHate_ in monolingual
settings.

  2. 2.

Multilingual mbert fine-tuned model to get high level labels.

  3. 3.

Translate the raw data of five languages (other than English) to English.

  4. 4.

Use multilingual binary classifier (developed in step 2) on translated data to
classify tweets at high level.

  5. 5.

Utilize Google Perspective API to predict toxicity of tweets for high level
labels.

  6. 6.

Use two way voting to predict final binary labels.

  7. 7.

Fine-tuned "distilbert-base-uncased" model on only the English dataset, and
used it to do annotations for the English translations of the data collected
from Twitter in 5 languages.

  8. 8.

To validate our domain specific labels we trained various models on HSMerge
data shown in Table 3 and predicted on hateful samples obtained from step 6.

###  4.2 Experimental Setting

To evaluate the performance of the models, we used weighted average F1 for
benchmarking on validation set. All the experiments were done on NVIDIA A100
GPU with up to 20 GiB RAM.

###  4.3 Hatefulness (Hate/No-Hate)

We utilised hate speech models to perform initial validation on the raw LAHM
dataset; details can be found in Section 5.

####  4.3.1 Monolingual Experiments

We utilized BERT based language specific hate speech models from HuggingFace.
The models used were trained, validated and tested on the same language.
We utilized the following language models for our experiments:

  1. 1.

Hate-speech-CNERG/dehatebert-mono-english 555https://huggingface.co/Hate-
speech-CNERG/dehatebert-mono-english

  2. 2.

l3cube-pune/hate-multi-roberta-hasoc-hindi666l3cube-pune/hate-roberta-hasoc-
hindi

  3. 3.

Hate-speech-CNERG/dehatebert-mono-french777https://huggingface.co/Hate-speech-
CNERG/dehatebert-mono-french

  4. 4.

Hate-speech-CNERG/dehatebert-mono-german888https://huggingface.co/Hate-speech-
CNERG/dehatebert-mono-german

  5. 5.

Hate-speech-CNERG/dehatebert-mono-spanish999https://huggingface.co/Hate-
speech-CNERG/dehatebert-mono-spanish

Binary hate labels distribution results from monolingual experiments are
presented in table 8

####  4.3.2 Machine Translation Experiments

We adopted the methodology to utilize open source machine translation models
for translation of English data to multilingual data and vice-versa. To select
the best machine translation model for each of our languages, we evaluated a
number of models on a small manually annotated dataset. The translations were
carried out for each of the these languages: Hindi, Arabic, French, German and
Spanish. Two types of translations were carried out using the translation
models:

  1. 1.

Translation of English dataset collected from various open source hate speech
datasets. All the English language samples in this dataset were translated to
other 5 languages.

  2. 2.

Translation of multilingual data collected from Twitter. This data was
collected for the above 5 languages, and each of this language data was
translated into corresponding English language data.

Consideration for choosing and evaluating the translation models was based on
whether they were open source, free/easy to use, and the translation quality.
Translation models used were:

  1. 1.

Google sheets translation

  2. 2.

m2m-100-1.2B101010https://github.com/UKPLab/EasyNMT

  3. 3.

IndicTrans111111https://github.com/AI4Bharat/indicTrans

The evaluation metrics used were bleu, rougeL and semantic similarity. For
semantic similarity, all-mpnet-base-v2 model from sentence-transformers
library was used to calculate the cosine score between input and translated
sentence embeddings. Figure 4 shows comparisons of bleu, rougeL and semantic
similarity scores for the 3 models on different languages. After comparing the
performances and taking other considerations into account, indicTrans model
was selected for all translations of Indic languages (Hindi), and m2m-100-1.2B
was selected for the other languages.

Figure 2: bleu scores for model translations

Figure 3: rougeL scores for model translations

Figure 4: Semantic similarity scores for model translations

####  4.3.3 Binary Multilingual Experiments

We utilized mbert which is a pre-trained model on the top 104 languages with
the largest Wikipedia corpus using a masked language modeling (MLM) objective,
to fine-tune for binary classification task to perform few shot learning
experiments. The aim was to make the classifier learn over samples from a few
languages and test over other languages. The training dataset was predominant
in English, curated from multiple open source resources. In addition to that
we introduced language samples for a few other languages using a neural
machine translation based open source model
($m2m\\_100\\_1.2B$)121212https://github.com/pytorch/fairseq/tree/main/examples/m2m_100.
NMT is an approach to achieve machine translation using artificial neural nets
to predict the best possible sequence of words.

The model was trained, tested and validated on the custom multilingual dataset
curated with the class distribution shown in Table 4 per chosen language. We
utilized this model to make predictions on the curated dataset.

Table 4: Multilingual binary classifier training data Languages | Hate | NoHate | Total
---|---|---|---
English | 20620 | 18171 | 38791
Arabic | 20539 | 18138 | 38680
Hindi | 20541 | 18138 | 38679
French | 20542 | 18138 | 38680
Total | 82242 | 72584 | 154826

####  4.3.4 Perspective API Experiments

Perspective API is a free to use API that uses machine learning to identify
toxicity in the comments. It has API rate limitations of 1 request per second.
Due to this limitation we utilized it on all languages except English. We
utilized Google Cloud to generate the Perspective API key for sending
requests, and specifically CommentAnalyzer API to get the toxicity score for
each data point. The CommentAnalyzer API supports all the 5 languages we have
considered. We used the Twitter cleaned data we had collected for the 5
languages to get the Toxicity score to classify each tweet as hate or not-
hate. Numbers are reported in Table 8.

###  4.4  Multi-Domain Experiments

This section describes the multi-domain experiments to validate labels on the
LAHM dataset.

####  4.4.1 Machine Translation Based Multi-Domain Experiments

HSMerge data present in Table 3 was used for the multi-domain classification
of hate speech. This dataset was pre-processed by removing URLs,
mentions/usernames, null values and duplicates.

The aim was to utilize transfer learning to fine-tune a model on only the
English dataset, and use it to do annotations for the English translations of
the data collected from Twitter in other 5 languages.

The model used for fine-tuning on this dataset was "distilbert-base-
uncased"131313https://huggingface.co/distilbert-base-uncased. DistilBERT is a
distilled version of the BERT base model. It has 40 percent less parameters
than bert-base-uncased, is 60 percent faster and retains almost 97 percent of
BERT’s performances141414https://arxiv.org/abs/1910.01108. A linear layer on
top of the pooled output of the model was added for the multi-class
classification. During training, the maximum sequence length was limited to
256, batch size was set to 32, learning rate 5e-5 and was trained for 4
epochs. The dataset was split into train, validation and test set using
stratified random sampling. Dataset details :

  1. 1.

20,174 training samples

  2. 2.

5,044 validation samples

  3. 3.

6,305 test samples

The model evaluation metrics on the test set are given in Table 5.

Table 5: Translated multiclass metrics Domain | precision | recall | f1-score | support
---|---|---|---|---
racism | 0.94 | 0.95 | 0.94 | 2882
sexism | 0.93 | 0.93 | 0.93 | 2295
religious hate | 0.92 | 0.93 | 0.92 | 409
abuse | 0.94 | 0.91 | 0.93 | 589
extremism | 0.83 | 0.82 | 0.82 | 130
accuracy |  |  | 0.93 | 6305
macro avg | 0.91 | 0.91 | 0.91 | 6305
weighted avg | 0.93 | 0.93 | 0.93 | 6305

####  4.4.2 Cross lingual based Multi-Domain Experiments

To validate our fine-grained labels with keywords from hatebase, we
experimented with cross-lingual models trained on HSmerge dataset as shown in
Table 3. In this experiment setting we fine tuned the mbert model for the
multi-class classification task and trained the model on HSmerge data in Table
3 and do the prediction on the other languages. Empirical results can be found
under Table 7. In this zero-shot setting, no other language samples were given
to the model after fine-tuning on the English dataset and also machine
translation was not involved at all in either the pre-training or fine-tuning.

We couldn’t manage to maintain the class distribution, and the extremism class
was left with comparatively less data points. To counter the issue we
introduced custom class weights to provide equal attention to the minority
class which in our case was extremism. This was done using a weighted class
random sampler WeightedSampler for class imbalance, so that all classes have
equal probability. The empirical scores achieved by the cross-lingual model
can be found in the Table 7.

The open source mbert model we fine-tuned for our use case has previously been
tested for zero shot experimentation151515https://github.com/google-
research/bert/blob/master/multilingual.md##results and did manage to achieve
decent empirical scores shown in Table 6. Dataset details :

  1. 1.

19298 training samples

  2. 2.

6432 validation samples

  3. 3.

6434 test samples

Table 6: Bert zero shot gold F1 scores Model | English | Spanish | German | Arabic
---|---|---|---|---
BERT-Zero Shot | 81.4 | 74.3 | 70.5 | 62.1
Table 7: HSMerge cross-lingual metrics Domains | precision | recall | f1-score | support
---|---|---|---|---
racism | 0.93 | 0.95 | 0.94 | 2161
sexism | 0.93 | 0.91 | 0.92 | 1722
religious hate | 0.93 | 0.93 | 0.93 | 307
abuse | 0.93 | 0.92 | 0.92 | 442
extremism | 0.85 | 0.80 | 0.83 | 97
accuracy |  |  | 0.93 | 4729
macro avg | 0.91 | 0.90 | 0.91 | 4729
weighted avg | 0.93 | 0.93 | 0.93 | 4729
Table 8:  All models prediction on high level label Hate and NoHate

Language | Monolingual | Multilingual | Perspective |
---|---|---|---|---
Hate | NoHate | Hate | NoHate | Hate | NoHate |
English | 7247 | 101345 | 11380 | 99598 | - | - |
Hindi | 210 | 260 | 3970 | 7247 | 1674 | 3720 |
Arabic | 6469 | 26265 | 3914 | 28820 | 1376 | 31358 |
French | 9148 | 24185 | 8893 | 24440 | 7540 | 13269 |
German | 468 | 23910 | 3905 | 20473 | 4467 | 4164 |
Spanish | 2088 | 10087 | 4734 | 7441 | 24952 | 30196 |
Total | 25630 | 186052 | 36796 | 188019 | 40009 | 82707 |

Table 9: Voting across multiple models for hate detection Language | Mono-Lingual | Multi-lingual | Perspective API
---|---|---|---
English | ✓ |  | ✓
Hindi |  | ✓ | ✓
Arabic | ✓ | ✓ |
French | ✓ | ✓ |
German |  | ✓ | ✓
Spanish | ✓ |  | ✓
Table 10: Cross-lingual multi-domain predictions distribution

Language | Abuse | Sexism | Racism | Religious | Extremism
---|---|---|---|---|---
English | 4228 | 2934 | 1566 | 751 | 154
Arabic | 1895 | 994 | 736 | 164 | 5631
Hindi | 1511 | 440 | 282 | 22 | 1715
French | 813 | 779 | 2393 | 750 | 4413
German | 180 | 123 | 30 | 4 | 131
Spanish | 818 | 483 | 139 | 11 | 637
Total | 9445 | 5753 | 5146 | 1702 | 12681

Table 11: Machine Translation multi-domain predictions distribution

Language | Abuse | Sexism | Racism | Religious | Extremism
---|---|---|---|---|---
English | 8691 | 6693 | 2495 | 261 | 524
Arabic | 430 | 1302 | 209 | 259 | 1138
Hindi | 132 | 204 | 67 | 137 | 105
French | 745 | 1004 | 545 | 273 | 538
German | 232 | 608 | 186 | 58 | 98
Spanish | 2082 | 2024 | 1519 | 347 | 1310
Total | 12312 | 11835 | 5021 | 1335 | 3713

##  5 Evaluation

We performed initial level of validation (Hatefulness) experiments on the raw
LAHM dataset using open source monolingual models, Perspective API and the
multilingual binary classifier we trained. We compared the results for
monolingual and multilingual and noticed that our multilingual few-shot
learning based classifier out-performed the open source monolingual BERT based
models from HuggingFace except on Arabic and French languages where the latter
did a better job; details can be found in Table 8. Based on the analysis we
took at least 2 votes from the models which performed better, details shown in
Table 9.

For Hindi, Perspective API performed better by predicting 31 percent as hate
compared to 12 percent for binary classifier. For Arabic, the binary
classifier performed significantly better.

For domain validation we utilised the zero-shot cross-lingual, multi-domain
and machine translation based multi-domain models trained with HSmerge data in
Table 3 and performed predictions on the different languages. Label
distribution from cross-lingual models are shown in Table 10. Label
distribution of predictions for different languages on the LAHM dataset from
multi-class translated model is in Table 11.

For Hindi, 31.6 percent samples were Abuse, while the lowest were 10.3 percent
for Racism. For Arabic, Sexism and Extremism contributed to 73 percent of all
hate samples. For French, 32 percent were Abuse and lowest 8.3 percent were
Religious Hate. For German, 51 percent were Sexism while the lowest were
Religious Hate and Extremism together amounting to 13 percent. For Spanish,
Abuse and Sexism had 27 percent each, while Religious Hate had 5 percent of
samples.

##  6 Conclusion

We have presented the LAHM dataset, a large scale semi-supervised training
dataset for multilingual and multi-domain hate speech identification, we
created by using 3 layer annotation pipeline and combination of monolingual,
multilingual and cross-lingual models. To the best of our knowledge, LAHM is
the largest of its kind, containing close to 300k tweets across 6 languages
and 5 domains.

LAHM enables cross-lingual abusive language detection across five domains and
in-depth interplay between language shift and domain shift. We have profiled
LAHM as a comprehensive resource for evaluating hate speech detection through
a series of cross-domain experiments in monolingual, multilingual and cross-
lingual setups with state of the art transfer learning models.

We hope that LAHM will inspire more efforts in understanding and building
semi-supervised large scale multilingual and multi-domain abusive language
detection datasets.

##  7 Future Work

For future work we explored leetspeak detection and identification on social
networks. A lot of hate content on social media uses leetspeak to evade
moderators and automated systems. We collected hate keywords belonging in this
category for each of the 5 domains, and experimented with a set of leets for
each keyword to extract the leetspeak hate content. We plan to use this for
future work in multilingual and multi-domain settings.

## References

  * Ousidhoum et al. [2019] N. Ousidhoum, Z. Lin, H. Zhang, Y. Song, D.-Y. Yeung,  Multilingual and multi-aspect hate speech analysis,  in: Proceedings of EMNLP, Association for Computational Linguistics, 2019\.
  * Chung et al. [2019] Y.-L. Chung, E. Kuzmenko, S. S. Tekiroglu, M. Guerini,  Conan–counter narratives through nichesourcing: a multilingual dataset of responses to fight online hate speech,  arXiv preprint arXiv:1910.03270 (2019).
  * Davidson et al. [2017] T. Davidson, D. Warmsley, M. Macy, I. Weber,  Automated hate speech detection and the problem of offensive language,  in: Proceedings of the International AAAI Conference on Web and Social Media, volume 11, 2017\.
  * Kolhatkar and Taboada [????] V. Kolhatkar, M. Taboada,  A corpus for the analysis of online news comments (????).
  * Gao and Huang [2017] L. Gao, R. Huang,  Detecting online hate speech using context aware models,  arXiv preprint arXiv:1710.07395 (2017).
  * Kumar et al. [2018] R. Kumar, A. K. Ojha, S. Malmasi, M. Zampieri,  Benchmarking aggression identification in social media,  in: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), 2018, pp. 1–11.
  * Wulczyn et al. [2017] E. Wulczyn, N. Thain, L. Dixon,  Ex machina: Personal attacks seen at scale,  in: Proceedings of the 26th international conference on world wide web, 2017, pp. 1391–1399.
  * Waseem and Hovy [2016] Z. Waseem, D. Hovy,  Hateful symbols or hateful people? predictive features for hate speech detection on twitter,  in: Proceedings of the NAACL Student Research Workshop, Association for Computational Linguistics, San Diego, California, 2016, pp. 88–93. URL: http://www.aclweb.org/anthology/N16-2013.
  * Mandl et al. [2019] T. Mandl, S. Modha, C. Mandlia, D. Patel, A. Patel, M. Dave, Hasoc-hate speech and offensive content identification in indo-european languages, 2019.
  * Basile and Rubagotti [2018] A. Basile, C. Rubagotti,  Crotonemilano for ami at evalita2018. a performant, cross-lingual misogyny detection system.,  EVALITA Evaluation of NLP and Speech Tools for Italian 12 (2018) 206.
  * Wiegand et al. [2018] M. Wiegand, M. Siegel, J. Ruppenhofer,  Overview of the germeval 2018 shared task on the identification of offensive language (2018).
  * Salminen et al. [2020] J. Salminen, M. Hopf, S. A. Chowdhury, S.-g. Jung, H. Almerekhi, B. J. Jansen,  Developing an online hate classifier for multiple social media platforms,  Human-centric Computing and Information Sciences 10 (2020) 1–34.
  * Chowdhury et al. [2020] S. A. Chowdhury, H. Mubarak, A. Abdelali, S.-g. Jung, B. J. Jansen, J. Salminen,  A multi-platform arabic news comment dataset for offensive language detection,  in: Proceedings of the 12th Language Resources and Evaluation Conference, 2020, pp. 6203–6212.
  * Wiegand et al. [2018] M. Wiegand, J. Ruppenhofer, A. Schmidt, C. Greenberg,  Inducing a lexicon of abusive words–a feature-based approach (2018).
  * Karan and Šnajder [2018] M. Karan, J. Šnajder,  Cross-domain detection of abusive language online,  in: Proceedings of the 2nd workshop on abusive language online (ALW2), 2018, pp. 132–137.
  * Pamungkas and Patti [2019] E. W. Pamungkas, V. Patti,  Cross-domain and cross-lingual abusive language detection: A hybrid approach with deep learning and a multilingual lexicon,  in: Proceedings of the 57th annual meeting of the association for computational linguistics: Student research workshop, 2019, pp. 363–370.
  * Pamungkas et al. [2020] E. W. Pamungkas, V. Basile, V. Patti,  Misogyny detection in twitter: a multilingual and cross-domain study,  Information Processing & Management 57 (2020) 102360.
  * Meyer and Gambäck [2019] J. S. Meyer, B. Gambäck,  A platform agnostic dual-strand hate speech detector,  in: ACL 2019 The Third Workshop on Abusive Language Online Proceedings of the Workshop, Association for Computational Linguistics, 2019\.
  * Wang et al. [2020] K. Wang, D. Lu, S. C. Han, S. Long, J. Poon,  Detect all abuse! toward universal abusive language detection models,  arXiv preprint arXiv:2010.03776 (2020).
  * Arango et al. [2019] A. Arango, J. Pérez, B. Poblete,  Hate speech detection is not as easy as you may think: A closer look at model validation,  in: Proceedings of the 42nd international acm sigir conference on research and development in information retrieval, 2019, pp. 45–54.
  * Waseem et al. [2018] Z. Waseem, J. Thorne, J. Bingel,  Bridging the gaps: Multi task learning for domain transfer of hate speech detection,  in: Online harassment, Springer, 2018, pp. 29–55.
  * Rizoiu et al. [2019] M.-A. Rizoiu, T. Wang, G. Ferraro, H. Suominen,  Transfer learning for hate speech detection in social media,  arXiv preprint arXiv:1906.03829 (2019).
  * Caselli et al. [2020] T. Caselli, V. Basile, J. Mitrović, M. Granitzer,  Hatebert: Retraining bert for abusive language detection in english,  arXiv preprint arXiv:2010.12472 (2020).
  * Koufakou et al. [2020] A. Koufakou, E. W. Pamungkas, V. Basile, V. Patti,  Hurtbert: incorporating lexical features with bert for the detection of abusive language,  in: Proceedings of the fourth workshop on online abuse and harms, 2020, pp. 34–43.
  * Glavaš et al. [2020] G. Glavaš, M. Karan, I. Vulic,  Xhate-999: Analyzing and detecting abusive language across domains and languages,  Association for Computational Linguistics, 2020\.
  * Mozafari et al. [2020] M. Mozafari, R. Farahbakhsh, N. Crespi,  Hate speech detection and racial bias mitigation in social media based on bert model,  PloS one 15 (2020) e0237861.
  * Ozler et al. [2020] K. B. Ozler, K. Kenski, S. Rains, Y. Shmargad, K. Coe, S. Bethard,  Fine-tuning bert for multi-domain and multi-label incivil language detection,  in: Proceedings of the Fourth Workshop on Online Abuse and Harms, 2020, pp. 28–33.
  * Aluru et al. [2021] S. S. Aluru, B. Mathew, P. Saha, A. Mukherjee,  A deep dive into multilingual hate speech classification,  in: Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Part V, Springer International Publishing, 2021, pp. 423–439.
  * Vashistha and Zubiaga [2021] N. Vashistha, A. Zubiaga,  Online multilingual hate speech detection: experimenting with hindi and english social media,  Information 12 (2021) 5.
  * Ibrohim and Budi [2019] M. O. Ibrohim, I. Budi,  Translated vs non-translated method for multilingual hate speech identification in twitter,  Int. J. Adv. Sci. Eng. Inf. Technol 9 (2019) 1116–1123.
  * Corazza et al. [2020] M. Corazza, S. Menini, E. Cabrio, S. Tonelli, S. Villata,  Hybrid emoji-based masked language models for zero-shot abusive language detection,  in: Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online, 2020, pp. 943–949. URL: https://aclanthology.org/2020.findings-emnlp.84. doi:10.18653/v1/2020.findings-emnlp.84.
  * Ahn et al. [2020] H. Ahn, J. Sun, C. Y. Park, J. Seo,  NLPDove at SemEval-2020 task 12: Improving offensive language detection with cross-lingual transfer,  in: Proceedings of the Fourteenth Workshop on Semantic Evaluation, International Committee for Computational Linguistics, Barcelona (online), 2020, pp. 1576–1586. URL: https://aclanthology.org/2020.semeval-1.206. doi:10.18653/v1/2020.semeval-1.206.
  * Pérez et al. [2020] J. M. Pérez, A. Arango, F. Luque,  ANDES at SemEval-2020 task 12: A jointly-trained BERT multilingual model for offensive language detection,  in: Proceedings of the Fourteenth Workshop on Semantic Evaluation, International Committee for Computational Linguistics, Barcelona (online), 2020, pp. 1524–1531. URL: https://aclanthology.org/2020.semeval-1.199. doi:10.18653/v1/2020.semeval-1.199.
  * Stappen et al. [2020] L. Stappen, F. Brunn, B. Schuller,  Cross-lingual zero-and few-shot hate speech detection utilising frozen transformer language models and axel,  arXiv preprint arXiv:2004.13850 (2020).
  * Dadu and Pant [2020] T. Dadu, K. Pant,  Team rouges at SemEval-2020 task 12: Cross-lingual inductive transfer to detect offensive language,  in: Proceedings of the Fourteenth Workshop on Semantic Evaluation, International Committee for Computational Linguistics, Barcelona (online), 2020, pp. 2183–2189. URL: https://aclanthology.org/2020.semeval-1.290. doi:10.18653/v1/2020.semeval-1.290.
  * Ranasinghe and Zampieri [2020] T. Ranasinghe, M. Zampieri,  Multilingual offensive language identification with cross-lingual embeddings,  in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online, 2020, pp. 5838–5844. URL: https://aclanthology.org/2020.emnlp-main.470. doi:10.18653/v1/2020.emnlp-main.470.
  * Zampieri et al. [2020] M. Zampieri, P. Nakov, S. Rosenthal, P. Atanasova, G. Karadzhov, H. Mubarak, L. Derczynski, Z. Pitenis, Ç. Çöltekin,  Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020),  arXiv preprint arXiv:2006.07235 (2020).
  * Founta et al. [2018] A. M. Founta, C. Djouvas, D. Chatzakou, I. Leontiadis, J. Blackburn, G. Stringhini, A. Vakali, M. Sirivianos, N. Kourtellis,  Large scale crowdsourcing and characterization of twitter abusive behavior,  in: Twelfth International AAAI Conference on Web and Social Media, 2018.
  * Gomez et al. [2020] R. Gomez, J. Gibert, L. Gomez, D. Karatzas,  Exploring hate speech detection in multimodal publications,  in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 1470–1478.
  * Fanton, Margherita and Bonaldi, Helena and Tekiroğlu, Serra Sinem and Guerini, Marco [2021] Fanton, Margherita and Bonaldi, Helena and Tekiroğlu, Serra Sinem and Guerini, Marco,  Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech,  in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2021.