# Many vertex-disjoint even cycles of fixed length in a graph

Jianfeng Hou Research was supported by National Natural Science Foundation of
China (Grant No. 12071077). Email<EMAIL_ADDRESS>Center for Discrete
Mathematics, Fuzhou University, Fujian, 350003, China  Caiyun Hu Email:
<EMAIL_ADDRESS>Center for Discrete Mathematics, Fuzhou University,
Fujian, 350003, China  Heng Li Email<EMAIL_ADDRESS>School of
Mathematics, Shandong University, Shandong, 250100, China  Xizhi Liu Research
was supported by ERC Advanced Grant 101020255 and Leverhulme Research Project
Grant RPG-2018-424. Email<EMAIL_ADDRESS>Mathematics Institute and
DIMAP, University of Warwick, Coventry, CV4 7AL, UK  Caihong Yang Email:
<EMAIL_ADDRESS>Center for Discrete Mathematics, Fuzhou University,
Fujian, 350003, China  Yixiao Zhang Email<EMAIL_ADDRESS>Center for
Discrete Mathematics, Fuzhou University, Fujian, 350003, China

###### Abstract

For every integer $k\geq 3$, we determine the extremal structure of an
$n$-vertex graph with at most $t$ vertex-disjoint copies of $C_{2k}$ when $n$
is sufficiently large and $t$ lies in the interval
$\left[\frac{\mathrm{ex}(n,C_{2k})}{\varepsilon n},\varepsilon n\right]$,
where $\varepsilon>0$ is a constant depending only on $k$. The question for
$k=2$ and $t=o\left(\frac{\mathrm{ex}(n,C_{2k})}{n}\right)$ was explored in
prior work [20], revealing different extremal structures in these cases. Our
result can be viewed as an extension of the theorems by Egawa [9] and
Verstraëte [36], where the focus was on the existence of many vertex-disjoint
cycles of the same length without any length constraints.

Keywords: even cycles, the Bondy–Simonovits Theorem, Turán problems, the
Corrádi–Hajnal Theorem.

##  1 Intorduction

The classical Corrádi–Hajnal Theorem [8] asserts that for every $t\geq 2$, a
graph with minimum degree at least $2t$ on at least $3t$ vertices contains $t$
pairwise vertex-disjoint cycles. Thomassen [34] extends this to graphs with
minimum degree at least $3t+1$ on at least $t^{2+o(1)}$ vertices, showing they
contain $t$ pairwise vertex-disjoint cycles of the same length. Thomassen’s
result was further extended by Egawa [9] to graphs with minimum degree at
least $2t$ on at least $(17+o(1))t$ vertices when $t\geq 3$. In [36],
Verstraëte provided a concise proof, showing that for every $t\geq 2$, a graph
with minimum degree at least $2t$ on at least $t^{7t}$ vertices also contains
$t$ pairwise vertex-disjoint cycles of the same length. In particular, the
case $t=2$ confirms a conjecture of Häggkvist [34].

There are some other extensions of the Corrádi–Hajnal Theorem from the average
degree perspective or by imposing different constraints on the length of
cycles. Notable instances include the work of Justesen [22], who demonstrated
that graphs with $n\geq 3t$ vertices and at least
$\max\left\\{(2t-1)(n-t),\binom{3t-1}{2}+(n-3t+1)\right\\}$ edges contains $t$
pairwise vertex-disjoint cycles. Chiba et al. [7] showed that every graph with
minimum degree at least $2t$ and with sufficiently large number of vertices
contains $t$ pairwise vertex-disjoint even cycles except in a few exceptional
cases. Wang [37] proved that for $t\geq 2$, every graph with minimum degree at
least $2t$ and at least $4t$ vertices contains $t$ pairwise vertex-disjoint
cycles of length at least four, except in three exceptional cases. Harvey and
Wood [18] extended this to $t\geq 6$ and $\ell\geq 3$, demonstrating that
every graph with average degree at least $4\ell t/3$ contains $t$ pairwise
vertex-disjoint cycles of length at least $r$.

The problem of finding cycles of a fixed length in a graph also attracted
considerable attention and stands as a central topic in Extremal Graph Theory.
An old theorem due to Mantel [27] states that every $n$-vertex graph with at
least $\lfloor n^{2}/4\rfloor+1$ edges contains a triangle. For every $k\geq
2$, using the Stability method, Simonovits [32] showed that every $n$-vertex
graph with at least $\lfloor n^{2}/4\rfloor+1$ edges contains a copy of
$C_{2k+1}$ when $n$ is sufficiently large. The tightness of the bound $\lfloor
n^{2}/4\rfloor+1$ is demonstrated by the balanced complete bipartite graph on
$n$ vertices in both cases. Finding even cycles of fixed length is much more
challenging. Erdős stated (without a proof) in [11] that for every $k\geq 2$,
every $n$-vertex graph with at least $A_{k}n^{1+\frac{1}{k}}$ edges contains a
copy of $C_{2k}$, where $A_{k}$ is a constant depending on $k$. The classical
Bondy-Simonovits Theorem [3] strengthens this by showing the existence of
$C_{2\ell}$ in an $n$-vertex graph with at least $100kn^{1+\frac{1}{k}}$ edges
for all integers $\ell\in[k,kn^{1/k}]$. Subsequent works [35, 31, 5, 6, 19]
have refined this bound. Determining whether the exponent $1+1/k$ is indeed
tight remains a major open problem in Extremal Graph Theory. Currently,
tightness is only known for $k\in\\{2,3,5\\}$, and we refer the reader to [12,
13, 4, 2, 14, 38, 15, 24, 25, 23, 16, 26, 28, 29, 17] for more related
results.

In this note, we explore the following problem, which can be seen as a
collective extension of the results by Egawa, Verstraëte, and
Bondy–Simonovits:

What is the maximum number of edges in an $n$-vertex graph without $t+1$
pairwise vertex-disjoint cycles of a fixed length?

This study is motivated by a remarkable result of Allen et al. [1], where they
determined, for large $n$, both the maximum number of edges and the structures
of four distinct extremal constructions (corresponding to different values of
$t$) in an $n$-vertex graph without $t+1$ pairwise vertex-disjoint triangles
for all integers $t$ in the interval $[0,n/3]$. Their result builds upon old
theorems established by of Erdős [10], Moon [30], and Simonovits [33], which
specifically address this problem in cases where $t$ is relatively small
compared to $n$.

Let us introduce some definitions before presenting the main result. Given a
graph $F$, let $\mathrm{ex}(n,F)$ denote the maximum number of edges in an
$n$-vertex $F$-free graph. We expand this notation to $\mathrm{ex}(n,(t+1)F)$,
representing the maximum number of edges in an $n$-vertex graph containing at
most $t$ pairwise vertex-disjoint copies of $F$.

The results of Mantel and Simonovits can be expressed as
$\mathrm{ex}(n,C_{2k+1})=\lfloor n^{2}/4\rfloor$ for $k\geq 1$ when $n$ is
sufficiently large. Similarly, the results of Erdős and Bondy–Simonovits can
be written as $\mathrm{ex}(n,C_{2k})\leq 100kn^{1+\frac{1}{k}}$ for all $k\geq
2$. The exact value of $\mathrm{ex}(n,(t+1)C_{2k+1})$ was established in [21]
for large $n$ and $t=o(n)$. Additionally, the exact value of
$\mathrm{ex}(n,(t+1)C_{2k})$ was determined in [20] for large $n$, with $t$
satisfying $t=o\left(\frac{\mathrm{ex}(n,C_{2k})}{n}\right)$ or $\sqrt{n}\ll
t\ll n$. The following result builds upon the previous one, determining the
exact value of $\mathrm{ex}(n,(t+1)C_{2k})$ for
$\frac{\mathrm{ex}(n,C_{2k})}{n}\ll t\ll n$.

######  Theorem 1.1.

For every integer $k\geq 3$ there exists $\varepsilon_{k}>0$ such that for
sufficiently large $n$,

$\displaystyle\mathrm{ex}(n,(t+1)C_{2k})=\binom{n}{2}-\binom{n-k(t+1)+1}{2}+1$

holds for all integers
$t\in\left[\frac{\mathrm{ex}(n,C_{2k})}{\varepsilon_{k}n},\varepsilon_{k}n\right]$.

Define $S(m,n)$ as the graph obtained from an $m$ by $n-m$ complete bipartite
graph by adding all pairs in the part of size $m$ as edges. Let $S^{+}(m,n)$
be the graph obtained from $S(m,n)$ by adding an edge into the part of size
$n-m$. Our proof shows that the unique extremal construction for Theorem 1.1
is simply $S^{+}(k(t+1)-1,n)$. In contrast, the extreml construction for
$\mathrm{ex}(n,(t+1)C_{2k})$ when
$t=o\left(\frac{\mathrm{ex}(n,C_{2k})}{n}\right)$ is different, and refer the
reader to [20] for more details.

Given a graph $F$ and an integer $q\leq v(F)$, let

$\displaystyle F[q]:=\left\\{F[S]\colon S\subset V(F)\text{ and }|S|\geq
q\right\\}.$

For an integer $q$ satisfying $v(F)-\alpha(F)\leq q\leq v(F)$, let

$\displaystyle F^{\mathrm{ind}}[q]:=\left\\{F[S]\colon S\subset V(F),\ |S|\geq
q,\text{ and }V(F)\setminus S\text{ is independent}\right\\}.$

In other words, $F[q]$ consists of induced subgraphs of $F$ that can be
obtained by removing at most $v(F)-q$ vertices, while $F^{\mathrm{ind}}[q]$
consists of induced subgraphs of $F$ that can be obtained by removing an
independent set of size at most $v(F)-q$.

Figure 1: Two examples of ways to remove an independent set of size three from
$C_{8}$.

Recall the definition of the covering number $\tau(F)$ of a graph $F$ is

$\displaystyle\tau(F):=\min\left\\{|S|\colon\text{$S\subset V(F)$ and $S\cap
e\neq\emptyset$ for all $e\in F$}\right\\}.$

A minor adjustment to the proof of Theorem 1.1 leads to the following more
general result. For simplicity, we omit its details in this note.

######  Theorem 1.2.

Let $s_{2}\geq s_{1}\geq 2$ be integers and $\mathbb{B}:=B_{s_{1},s_{2}}$ be
an $s_{1}$ by $s_{2}$ connected bipartite graph with $\tau(B)=s_{1}$. Suppose
that some member in $\mathbb{B}^{\mathrm{ind}}[s_{2}+1]$ is the union of a
matching with isolated vertices. Then there exists $\varepsilon>0$ such that
for sufficiently large $n$,

$\displaystyle\mathrm{ex}(n,(t+1)\mathbb{B})=\binom{n}{2}-\binom{n-\tau(t+1)+1}{2}+\mathrm{ex}(n-\tau(t+1)+1,\mathbb{B}[s_{2}+1])$

holds for all integers $t\in\left[\frac{\mathrm{ex}(n,\mathbb{B})}{\varepsilon
n},\varepsilon n\right]$.

The extremal construction for Theorem 1.2 is the graph obtained from
$S(\tau(t+1)-1,n)$ by adding a maximum $\mathbb{B}[s_{2}+1]$-free graph to the
part of size $n-m$.

The proof of Theorem 1.1 is presented in Section 3. In the next section, we
present some definitions and preliminary results.

##  2 Preliminaries

Given a graph $G$ and a set $W\subset V(G)$, we use $G[W]$ to denote the
induced subgraph of $G$ on $W$, and use $G-W$ to denote the induced subgraph
of $G$ on $V(G)\setminus W$. Given two disjoint sets $S,T\subset V(G)$, we use
$G[S,T]$ to denote the collection of edges in $G$ that have nonempty
intersection with both $S$ and $T$. We use $\overline{G}$ to denote the
complement of $G$.

We will use the following lower bound for $\mathrm{ex}(n,C_{2k})$ which
follows from a standard probabilistic deletion argument.

######  Proposition 2.1.

For every integer $k\geq 2$ it holds that
$\mathrm{ex}(n,C_{2k})=\Omega\left(n^{1+\frac{1}{2k-1}}\right)$.

The following results can be found in [20].

######  Fact 2.2.

Suppose that $n,\ell,x\geq 0$ are integers satisfying $\ell+x\leq n$. Then

$\displaystyle\binom{n}{2}-\binom{n-\ell-x}{2}\geq\binom{n}{2}-\binom{n-\ell}{2}+x(n-\ell-x).$

######  Proposition 2.3 ([20, Proposition 2.8]).

Let $F$ be a connected graph. For all integers $n\geq m\geq 1$ we have

$\displaystyle\frac{\mathrm{ex}(n,F)}{n}\geq\left(1-\frac{m}{n}\right)\frac{\mathrm{ex}(m,F)}{m}.$

The following simple but useful lemma gives a rough upper bound for the number
of edges in an $(t+1)F$-free $r$-graph with bounded degree.

######  Lemma 2.4 ([20, Lemma 2.13]).

Suppose that $F$ is a graph on $m$ vertices. Then for every $t\geq 0$, every
$n$-vertex $(t+1)F$-free graph $G$ satisfies

$\displaystyle|G|\leq mt\cdot\Delta(G)+\mathrm{ex}(n-mt,F).$

Given a family $\mathcal{F}$ of graphs and an integer $t\geq 0$ let

$\displaystyle(t+1)\mathcal{F}:=\left\\{F_{1}\sqcup\cdots\sqcup F_{t+1}\colon
F_{i}\in\mathcal{F}\text{ for all }i\in[t+1]\right\\}.$

######  Proposition 2.5 ([20, Proposition 5.8]).

Let $s_{2}\geq s_{1}\geq 2$ be integers and $\mathbb{B}:=B_{s_{1},s_{2}}$ be
an $s_{1}$ by $s_{2}$ bipartite graph. Let
$\mathbb{B}^{\prime}:=\mathbb{B}[s_{2}+1]$. Suppose that
$t\leq\frac{n}{2e(s_{2}+1)}$. Then

$\displaystyle\mathrm{ex}\left(n,(t+1)\mathbb{B}^{\prime}\right)\leq\mathrm{ex}\left(n,\mathbb{B}^{\prime}\sqcup
tK_{1,s_{2}}\right)\leq\binom{n}{2}-\binom{n-t}{2}+\mathrm{ex}\left(n-t,\mathbb{B}^{\prime}\right).$

Suppose that $G=G[V_{1},V_{2}]$ and $B_{s_{1},s_{2}}=B[W_{1},W_{2}]$ are
bipartite graphs. An ordered copy of $B_{s_{1},s_{2}}$ in $G$ is a copy of
$B_{s_{1},s_{2}}$ in $G$ with the condition that $W_{1}$ is contained in
$V_{1}$ and $W_{2}$ is contained in $V_{2}$.

The following three technical lemmas are useful for finding many vertex-
disjoint copies of a bipartite graph.

######  Lemma 2.6 ([20, Lemma 5.3]).

Let $s_{2}\geq s_{1}\geq 1$ be integers and $\alpha\in(0,1)$ be a real number.
The following statement holds for sufficiently large $n$. Suppose that $G$ is
an $m$ by $n$ bipartite graph on $V_{1}$ and $V_{2}$ that satisfies

$\displaystyle m\leq\frac{(1-\alpha s_{1})s_{1}}{s_{2}}n\quad\text{and}\quad
d_{G}(v)\geq(1-\alpha)n\quad\text{for all $v\in V_{1}$}.$

Then $G$ contains $\left\lfloor m/s_{1}\right\rfloor$ pairwise vertex-disjoint
ordered copies of $K_{s_{1},s_{2}}$.

######  Lemma 2.7 ([20, Lemmas 5.1, 5.2]).

Let $s_{2}\geq s_{1}\geq 1$ be integers and $\alpha\in(0,1)$ be a real number.
The following holds for sufficiently large $n$. Suppose that $G$ is an $m$ by
$n$ bipartite graph on $V_{1}$ and $V_{2}$ such that $m\leq\frac{\alpha
n}{8s_{2}}$ and $d_{G}(v)\geq\alpha n$ for every $v\in V_{1}$. Then $G$
contains at least $\left\lfloor\frac{m}{s_{1}}-\frac{4}{\alpha}\right\rfloor$
pairwise vertex-disjoint ordered copies of $K_{s_{1},s_{2}}$.

If, in addition, there are at least
$\min\left\\{\frac{5s_{1}(s_{1}-1)}{\alpha},\ \frac{s_{1}-1}{s_{1}}m\right\\}$
vertices in $V_{1}$ with degree at least
$\left(1-\frac{\alpha}{2s_{1}}\right)n$, then $G$ contains $\left\lfloor
m/s_{1}\right\rfloor$ pairwise vertex-disjoint ordered copies of
$K_{s_{1},s_{2}}$.

######  Lemma 2.8 ([20, Lemma 5.4]).

Let $s_{2}\geq s_{1}\geq 1$ be integers and $\alpha\in(0,1)$ be a real number.
Let $\mathbb{B}:=B_{s_{1},s_{2}}$ be a bipartite graph with part sizes
$s_{1},s_{2}$. For every $\alpha>0$ there exists $N_{0}$ such that the
following statement holds for all $n\geq N_{0}$ and $t\leq\frac{\alpha
n}{18(s_{1}+s_{2})^{2}}$. Every $n$-vertex $(t+1)\mathbb{B}$-free graph $G$
satisfies

$\displaystyle|G|\leq\left(s_{1}t+\frac{34(s_{1}+s_{2})}{\alpha}\right)\Delta(G)+\frac{\alpha
s_{1}}{4}tn+\mathrm{ex}(n-st,\mathbb{B}).$

If, in addition, $\Delta(G)\leq(1-\alpha)n$ and $t\geq\frac{5}{\alpha
s_{1}}\left(\frac{\mathrm{ex}(n,\mathbb{B})}{n-1}+\frac{36(s_{1}+s2)}{\alpha}\right)$,
then

$\displaystyle|G|\leq\binom{n}{2}-\binom{n-s_{1}(t-1)}{2}.$

##  3 Proof of Theorem 1.1

In this section, we prove Theorem 1.1. Our proof strategy begins by
considering the collection $L\subset V(G)$ of all vertices in a maximum
$n$-vertex $(t+1)C_{2k}$-free graph $G$ with degree $\Omega(n)$. We will show
that a significant proportion of vertices in $L$ possess a degree near $n$.
Consequently, leveraging Lemma 2.7, we can show that the size of $L$ is less
than $k(t+1)$. Following this, a more detailed analysis of the structure of
$G-L$ will show that $G-L$ is $\\{K_{1,2},M_{2}\\}$-free, and hence, contains
at most one edge. Here, $M_{2}$ denotes the matching with two edges.

###### Proof of Theorem 1.1.

Fix an integer $k\geq 3$, and let $\mathbb{C}:=C_{2k}$ for simplicity. Let

$\displaystyle\delta_{1}:=\frac{1}{32k^{2}},\quad\delta_{2}:=\frac{1}{4k},\quad\delta_{3}:=\frac{1}{2(k+1)},\quad\varepsilon:=\frac{\delta_{1}}{144k^{2}},\quad\theta:=\frac{20}{\delta_{1}k}\frac{\mathrm{ex}(n,\mathbb{C})}{n}.$

Let $n$ be sufficiently large and $t$ be an integer satisfying

$\displaystyle\frac{72k}{\delta_{3}}\theta\leq t\leq\varepsilon n.$ (1)

Let $G$ be a maximum $n$-vertex $(t+1)\mathbb{C}$-free graph with vertex set
$V$. Let

$\displaystyle L_{1}:=\left\\{v\in V\colon
d_{G}(v)\geq(1-\delta_{1})n\right\\},\quad L_{2}:=\left\\{v\in V\setminus
L_{1}\colon d_{G}(v)\geq\delta_{2}n\right\\},$

and

$\displaystyle L$ $\displaystyle:=L_{1}\cup L_{2},\quad U:=V\setminus
L,\quad\ell:=|L|,\quad\ell_{1}:=|L_{1}|,\quad\ell_{2}:=|L_{2}|.$

In addition, let

$\displaystyle G_{1}:=G-L_{1},\quad G_{2}:=G[U],\quad
n_{1}:=n-\ell_{1},\quad\text{and}\quad
t_{1}:=t-\left\lfloor\ell_{1}/k\right\rfloor.$

######  Claim 3.1.

We have $\ell_{1}<k(t+1)$ and $G_{1}$ is $(t_{1}+1)\mathbb{C}$-free.

###### Proof.

First, suppose to the contrary that $\ell_{1}\geq k(t+1)$. We may assume that
$\ell_{1}=k(t+1)$ since otherwise we can replace $L_{1}$ by a subset of size
$k(t+1)$. Let $H$ be the collection of edges in $G$ that have exactly one
vertex in $L_{1}$. Notice that $H$ is bipartite, and for every $v\in L_{1}$ it
follows from the assumptions on $t$ and $\delta_{1}$ that

$\displaystyle d_{H}(v)\geq
d_{G}(v)-|L_{1}|\geq(1-\delta_{1})n-k(t+1)\geq(1-\delta_{1})n-2\varepsilon
kn>\left(1-2\delta_{1}\right)n.$

Since, in addition, $\frac{\left(1-2k\delta_{1}\right)k}{k}(n-\ell_{1})\geq
n/4\geq\ell_{1}$ (by (1)), it follows from Lemma 2.6 that
$(t+1)\mathbb{C}\subset(t+1)K_{k,k}\subset H$, a contradiction.

Now suppose to the contrary that there exists a collection
$\mathcal{B}=\\{\mathbb{C}_{1},\ldots,\mathbb{C}_{t_{1}+1}\\}$ of $t_{1}+1$
pairwise vertex-disjoint copies of $\mathbb{C}$ in $G_{1}$. Let
$B:=\bigcup_{i\in[t_{1}+1]}V(\mathbb{C}_{i})$ and $V^{\prime}:=V\setminus B$.
Let $H_{1}$ be the collection of edges in $G[V^{\prime}]$ that contain exactly
one vertex in $L_{1}$. Similar to the argument above, we have

$\displaystyle d_{H_{1}}(v)\geq d_{G}(v)-|L_{1}\cup
B_{1}|\geq(1-\delta_{1})n-2k(t+1)>\left(1-2\delta_{1}\right)n.$

Similarly, it follows from (1) and Lemma 2.6 that
$\lfloor\ell_{1}/k\rfloor\mathbb{C}\subset H_{1}$, which together with
$\mathcal{B}$ implies that $(t+1)\mathbb{C}\subset G$, a contradiction.

By Claim 3.1, we obtain

$\displaystyle n_{1}=n-\ell_{1}\geq n-k(t+1)\geq(1-2\varepsilon
k)n\geq\left(1-\frac{\delta_{1}}{2}\right)n.$

Combined with the definition of $L_{1}$, we obtain

$\displaystyle\Delta(G_{1})\leq(1-\delta_{1})n=(1-\delta_{1})\times\frac{n}{n_{1}}\times
n_{1}\leq\left(1-\frac{\delta_{1}}{2}\right)n_{1}.$

Notice from the definition of $\theta$ and Proposition 2.1 that

$\displaystyle\theta=\frac{20}{\delta_{1}k}\frac{\mathrm{ex}(n,\mathbb{C})}{n}\geq\frac{5}{\delta_{1}k/2}\left(\frac{\mathrm{ex}(n_{1},\mathbb{C})}{n_{1}-1}+\frac{72k}{\delta_{1}/2}\right).$

If $t_{1}\geq\theta$, then it follows from $t_{1}\leq t\leq\varepsilon
n\leq\frac{\delta_{1}n_{1}/2}{72k^{2}}$ and Lemma 2.8 that

$\displaystyle|G_{1}|\leq\binom{n_{1}}{2}-\binom{n_{1}-k(t_{1}-1)}{2}=\binom{n-\ell_{1}}{2}-\binom{n-kt}{2}.$

Consequently,

$\displaystyle|G|\leq\binom{n}{2}-\binom{n-\ell_{1}}{2}+|G_{1}|\leq\binom{n}{2}-\binom{n-kt}{2},$

and we are done.

So we may assume that $t_{1}\leq\theta$. It follows from $t\geq
80k\theta/\delta_{3}$ and the definition of $t_{1}$ that

$\displaystyle\ell_{1}\geq k\left(t-t_{1}\right)\geq
k\left(t-\theta\right)\geq k\left(t-\frac{t-k+1}{k}\right)=\frac{k-1}{k}\times
k(t+1).$ (2)

Let

$\displaystyle t_{2}:=t-\left\lfloor\ell/k\right\rfloor\leq t_{1}\leq\theta.$

######  Claim 3.2.

We have $\ell_{1}+\ell_{2}\leq k(t+1)-1$, and $G[U]$ is
$(t_{2}+1)\mathbb{C}$-free. In particular,

$\displaystyle|G[U]|\leq
2kt_{2}\times\delta_{2}n+\mathrm{ex}(n-\ell-2kt_{2},\mathbb{C})\leq\frac{\theta
n}{2}+\mathrm{ex}(n,\mathbb{C})\leq\theta n.$ (3)

###### Proof.

The proof is similar to that of Claim 3.1. Suppose to the contrary that Claim
3.2 is not true. If $\ell<k(t+1)$, then let
$\mathcal{B}^{\prime}:=\left\\{\mathbb{C}_{1}^{\prime},\ldots,\mathbb{C}_{t_{2}+1}^{\prime}\right\\}$
be a collection of pairwise vertex-disjoint copies of $\mathbb{C}$ in $G[U]$
and let $B^{\prime}:=\bigcup_{i\in[t_{2}+1]}V(\mathbb{C}_{i}^{\prime})$. If
$\ell\geq k(t+1)$, then by removing vertices in $L_{2}$, we may assume that
$\ell=k(t+1)$. In addition, we set $B^{\prime}:=\emptyset$ in this case. Let
$H_{2}$ be the collection of all edges in $G[V\setminus B^{\prime}]$ that
contain exactly one vertex in $L$. To build a contradiction, it suffices to
show that $\lfloor\ell/k\rfloor\mathbb{C}\subset H_{2}$. Indeed, observe that
for every $v\in L_{1}$, we have

$\displaystyle d_{H_{2}}(v)\geq d_{G}(v)-|L\cup
B^{\prime}|\geq(1-\delta_{1})n-2k(t+1)\geq(1-\delta_{1})n-4\varepsilon
kn\geq\left(1-\frac{\delta_{2}}{4k}\right)n,$

and for every $u\in L_{2}$, we have

$\displaystyle d_{H_{2}}(u)\geq d_{G}(u)-|L\cup
B^{\prime}|\geq\delta_{2}n-4\varepsilon kn\geq\frac{\delta_{2}n}{2}.$

Since, in addition, $\ell\leq 2kt\leq 2\varepsilon
kn\leq\frac{\delta_{2}n/4}{8k}\leq\frac{\delta_{2}(n-|B^{\prime}|)/2}{8k}$, it
follows from (2) and Lemma 2.7 that $\lfloor\ell/k\rfloor\mathbb{C}\subset
H_{2}$, a contradiction.

We divide $U$ further by letting

$\displaystyle S:=\left\\{u\in U\colon|N_{G}(u)\cap
L|\leq(1-\delta_{3})|L|\right\\}\quad\text{and}\quad W:=U\setminus S.$

In particular, since, by Claim 3.2, $G[S]\subset G[U]$ is
$(t_{2}+1)\mathbb{C}$-free, it follows from Lemma 2.4 that

$\displaystyle|G[S]|\leq 2kt_{2}|S|+\mathrm{ex}\left(|S|,\mathbb{C}\right)\leq
2k\theta|S|+\mathrm{ex}\left(|S|,\mathbb{C}\right).$ (4)

It follows from the definition of $S$ that

$\displaystyle|G[L,U]|\leq|W||L|+|S|\times(1-\delta_{3})|L|=\ell(n-\ell)-\delta_{3}\ell|S|.$
(5)

Combining (3) and (5), we obtain

$\displaystyle|G|=|G[L]|+|G[L,U]|+|G[U]|$
$\displaystyle\leq\binom{\ell}{2}+\ell(n-\ell)-\delta_{3}\ell|S|+\theta n.$

So it follows from
$|G|\geq|S(k(t+1)-1,n)|=\binom{n}{2}-\binom{n-k(t+1)+1}{2}\geq\binom{\ell}{2}+\ell(n-\ell)$
that

$\displaystyle|S|\leq\frac{\theta n}{k(t-\theta)}<\frac{n}{2}.$

which combined with Proposition 2.3 implies that

$\displaystyle\frac{\mathrm{ex}\left(n,\mathbb{C}\right)}{n}\geq\left(1-\frac{|S|}{n}\right)\frac{\mathrm{ex}\left(|S|,\mathbb{C}\right)}{|S|}\geq\frac{1}{2}\frac{\mathrm{ex}\left(|S|,\mathbb{C}\right)}{|S|}.$

Consequently, by (1), we obtain

$\displaystyle\frac{\delta_{3}\ell}{2}|S|-\mathrm{ex}\left(|S|,\mathbb{C}\right)\geq\left(\frac{\delta_{3}}{2}\times
k(t-\theta)-2\frac{\mathrm{ex}\left(n,\mathbb{C}\right)}{n}\right)|S|\geq 0.$
(6)

In addition,

$\displaystyle\frac{\delta_{3}\ell}{2}-2k\theta-16k^{2}\theta\geq\frac{\delta_{3}}{2}\times
k(t-\theta)-2k\theta-16k^{2}\theta>0.$ (7)

Let

$\displaystyle t_{3}:=k(t+1)-1-\ell,\quad S_{1}:=\left\\{v\in
S\colon|N_{G}(v)\cap W|\geq 16k^{2}\theta\right\\},\quad\text{and}\quad
x:=|S_{1}|.$

It follows from (4), (6), (7) and definition of $S_{1}$ that

$\displaystyle|G[S]|+|G[L,S]|+|G[S,W]|$ $\displaystyle\leq
2k\theta|S|+\mathrm{ex}\left(|S|,\mathbb{C}\right)+(1-\delta_{3})\ell|S|+x\Delta+16k^{2}\theta|S|$
$\displaystyle\leq\ell|S|-\left(\delta_{3}\ell-2k\theta-16k^{2}\theta\right)|S|+\mathrm{ex}\left(|S|,\mathbb{C}\right)+\frac{xn}{2}$
$\displaystyle\leq\ell|S|+\frac{xn}{2}-\frac{\delta_{3}\ell}{2}|S|+\mathrm{ex}\left(|S|,\mathbb{C}\right)\leq\ell|S|+\frac{xn}{2}.$

Consequently,

$\displaystyle|G[L,U]|+|G[S]|+|G[S,W]|$
$\displaystyle=|G[L,W]|+|G[S]|+|G[L,S]|+|G[L,W]|$
$\displaystyle\leq\ell|W|+\ell|S|+\frac{xn}{2}=\ell(n-\ell)+\frac{xn}{2}.$ (8)

For every set $T\subset U$ let $N_{T}:=\bigcap_{u\in T}N_{G}(u)\cap L$ denote
the set of common neighbors of $T$ in $L$. It follows from the definition of
$W$ that for every $(k+1)$-set $T\subset U$ we have

$\displaystyle|N_{T}|\geq\ell-\sum_{u\in T}\left(\ell-|N_{G}(u)\cap
L|\right)\geq\left(1-\delta_{3}(k+1)\right)\ell=\frac{\ell}{2}.$ (9)

$L$$W$$S$$S_{1}$ Figure 2: Supplementary diagram for Claim 3.3 when $k=3$.

Let $\mathbb{C}^{\prime}:=\mathbb{C}[k+1]$ and note that every
$\mathbb{C}^{\prime}$-free graph with at least $k+1$ vertices is also
$\\{K_{1,2},M_{2}\\}$-free (see Figure 1).

######  Claim 3.3.

We have $x\leq t_{3}$ and $G[W]$ is
$(t_{3}-x+1)\cdot\\{K_{1,2},M_{2}\\}$-free. Moreover,

$\displaystyle|G[W]|\leq\frac{t_{3}-x}{2}n+1.$ (10)

###### Proof.

First, suppose to the contrary that there exists a $(t_{3}+1)$-set
$\\{v_{1},\ldots,v_{t_{3}+1}\\}\subset S_{1}$ such that $|N_{G}(v_{i})\cap
W|\geq 16k^{2}\theta\geq 4k(t_{3}+1)$ for all $i\in[t_{3}+1]$. Choose an
$k$-set $X_{i}$ from $N_{G}(v_{i})\cap W$ for each $i\in[t_{2}+1]$ such that
$\\{X_{1},\ldots,X_{t_{3}+1}\\}$ are pairwise disjoint (the definition of $S$
ensures that this is possible). For each $N_{i}$ choose an $(k-1)$-set
$Y_{i}\subset N_{X_{i}}$ such that $\\{Y_{1},\ldots,Y_{t_{3}+1}\\}$ are
pairwise disjoint (the existence of such sets are guaranteed by (9)). Notice
that $\mathbb{C}\subset K_{k,k}\subset G[\\{v_{i}\\}\cup X_{i}\cup Y_{i}]$ for
$i\in[t_{3}+1]$.

Let $Z:=\bigcup_{i\in[t_{3}+1]}\left(\\{v_{i}\\}\cup X_{i}\cup Y_{i}\right)$
and $G^{\prime}$ be the induced subgraph of $G$ on $V\setminus Z$. For every
$v\in L_{1}\setminus Z$ we have

$\displaystyle d_{G^{\prime}}(v)\geq
d_{G}(v)-|Z|\geq(1-\delta_{1})n-2k(t_{3}+1)\geq(1-2\delta_{1})n,$

and for every $u\in L_{2}\setminus Z$ we have

$\displaystyle d_{G^{\prime}}(v)\geq
d_{G}(v)-|Z|\geq\delta_{2}n-2k(t_{3}+1)\geq\frac{\delta_{2}}{2}n.$

Since, in addition, $\ell\leq k(t+1)\leq 2\varepsilon
kn\leq\frac{\delta_{2}(n-\ell-|Z|)/2}{8k}$, it follows from Lemma 2.7 that
$G^{\prime}$ contains at least

$\displaystyle\left\lfloor\frac{\ell-(k-1)t_{3}}{k}\right\rfloor=\ell-(k-1)(t+1)$

pairwise vertex-disjoint copies of $\mathbb{C}$. This implies that $G$
contains

$\displaystyle t_{3}+1+\ell-(k-1)(t+1)=k(t+1)-1-\ell+1+\ell-(k-1)(t+1)=t+1$

pairwise vertex-disjoint copies of $\mathbb{C}$, a contradiction. This proves
that $x\leq t_{3}$. The proof for the
$(t_{3}-x+1)\cdot\\{K_{1,2},M_{2}\\}$-freeness of $G[W]$ is similar (see
Figures 1 and 2), so we omit the details.

Recall that $\Delta(G[W])\leq\Delta(G[U])\leq\delta_{3}n$, so it follows from
Lemma 2.4 and $(t_{3}-x+1)\cdot\\{K_{1,2},M_{2}\\}$-freeness of $G[W]$ that

$\displaystyle|G[W]|\leq(t_{3}-x)(k+1)\times\delta_{3}n+\mathrm{ex}(n,\\{K_{1,2},M_{2}\\})\leq\frac{t_{3}-x}{2}n+1,$

completing the proof of Claim 3.3.

It follows from (3), (10), and Fact 2.2 that

$\displaystyle|G|$ $\displaystyle=|G[L]|+|G[L,U]|+|G[S]|+|G[S,W]|+|G[W]|$
$\displaystyle\leq\binom{\ell}{2}+\ell(n-\ell)+\frac{xn}{2}+\frac{t_{3}-x}{2}n+1$
$\displaystyle\leq\binom{n}{2}-\binom{n-k(t+1)+1}{2}-t_{3}\times\frac{2n}{3}+\frac{t_{3}}{2}n+1$
$\displaystyle\leq\binom{n}{2}-\binom{n-k(t+1)+1}{2}+1,$

completing the proof of Theorem 1.1.

## References

  * [1] P. Allen, J. Böttcher, J. Hladký, and D. Piguet.  A density Corrádi-Hajnal theorem.  Canad. J. Math., 67(4):721–758, 2015.
  * [2] C. T. Benson.  Minimal regular graphs of girths eight and twelve.  Canadian J. Math., 18:1091–1094, 1966.
  * [3] J. A. Bondy and M. Simonovits.  Cycles of even length in graphs.  J. Combinatorial Theory Ser. B, 16:97–105, 1974.
  * [4] W. G. Brown.  On graphs that do not contain a Thomsen graph.  Canad. Math. Bull., 9:281–285, 1966.
  * [5] B. Bukh and Z. Jiang.  A bound on the number of edges in graphs without an even cycle.  Combin. Probab. Comput., 26(1):1–15, 2017.
  * [6] B. Bukh and Z. Jiang.  Erratum for ‘A bound on the number of edges in graphs without an even cycle’.  Combin. Probab. Comput., 26(6):952–953, 2017.
  * [7] S. Chiba, S. Fujita, K.-i. Kawarabayashi, and T. Sakuma.  Minimum degree conditions for vertex-disjoint even cycles in large graphs.  Adv. in Appl. Math., 54:105–120, 2014.
  * [8] K. Corradi and A. Hajnal.  On the maximal number of independent circuits in a graph.  Acta Math. Acad. Sci. Hungar., 14:423–439, 1963.
  * [9] Y. Egawa.  Vertex-disjoint cycles of the same length.  J. Combin. Theory Ser. B, 66(2):168–200, 1996.
  * [10] P. Erdős.  Über ein Extremalproblem in der Graphentheorie.  Arch. Math. (Basel), 13:222–227, 1962.
  * [11] P. Erdős.  Extremal problems in graph theory.  In Theory of Graphs and its Applications (Proc. Sympos. Smolenice, 1963), pages 29–36. Publ. House Czech. Acad. Sci., Prague, 1964\.
  * [12] P. Erdős and A. Rényi.  On a problem in the theory of graphs.  Magyar Tud. Akad. Mat. Kutató Int. Közl., 7:623–641, 1962\.
  * [13] P. Erdős, A. Rényi, and V. T. Sós.  On a problem of graph theory.  Studia Sci. Math. Hungar., 1:215–235, 1966.
  * [14] Z. Füredi.  Graphs without quadrilaterals.  J. Combin. Theory Ser. B, 34(2):187–190, 1983.
  * [15] Z. Füredi.  Quadrilateral-free graphs with maximum number of edges.  preprint, 1994.
  * [16] Z. Füredi.  On the number of edges of quadrilateral-free graphs.  J. Combin. Theory Ser. B, 68(1):1–6, 1996.
  * [17] Z. Füredi, A. Naor, and J. Verstraëte.  On the Turán number for the hexagon.  Adv. Math., 203(2):476–496, 2006.
  * [18] D. J. Harvey and D. R. Wood.  Cycles of given size in a dense graph.  SIAM J. Discrete Math., 29(4):2336–2349, 2015.
  * [19] Z. He.  A new upper bound on the Tuŕan number of even cycles.  Electron. J. Combin., 28(2):Paper No. 2.41, 18, 2021.
  * [20] J. Hou, C. Hu, H. Li, X. Liu, C. Yang, and Y. Zhang.  Toward a density Corrádi–Hajnal theorem for degenerate hypergraphs.  preprint.
  * [21] J. Hou, H. Li, X. Liu, L.-T. Yuan, and Y. Zhang.  A step towards a general density Corrádi–Hajnal theorem.  arXiv preprint arXiv:2302.09849, 2023.
  * [22] P. Justesen.  On independent circuits in finite graphs and a conjecture of Erdős and Pósa.  In Graph theory in memory of G. A. Dirac (Sandbjerg, 1985), volume 41 of Ann. Discrete Math., pages 299–305. North-Holland, Amsterdam, 1989.
  * [23] F. Lazebnik and V. A. Ustimenko.  Explicit construction of graphs with an arbitrary large girth and of large size.  volume 60, pages 275–284. 1995.  ARIDAM VI and VII (New Brunswick, NJ, 1991/1992).
  * [24] F. Lazebnik, V. A. Ustimenko, and A. J. Woldar.  Properties of certain families of $2k$-cycle-free graphs.  J. Combin. Theory Ser. B, 60(2):293–298, 1994.
  * [25] F. Lazebnik, V. A. Ustimenko, and A. J. Woldar.  A new series of dense graphs of high girth.  Bull. Amer. Math. Soc. (N.S.), 32(1):73–79, 1995.
  * [26] F. Lazebnik, V. A. Ustimenko, and A. J. Woldar.  Polarities and $2k$-cycle-free graphs.  volume 197/198, pages 503–513. 1999.  16th British Combinatorial Conference (London, 1997).
  * [27] W. Mantel.  Vraagstuk XXVIII.  Wiskundige Opgaven, 10(2):60–61, 1907.
  * [28] K. E. Mellinger.  LDPC codes from triangle-free line sets.  Des. Codes Cryptogr., 32(1-3):341–350, 2004.
  * [29] K. E. Mellinger and D. Mubayi.  Constructions of bipartite graphs from finite geometries.  J. Graph Theory, 49(1):1–10, 2005.
  * [30] J. W. Moon.  On independent complete subgraphs in a graph.  Canadian J. Math., 20:95–102, 1968.
  * [31] O. Pikhurko.  A note on the Turán function of even cycles.  Proceedings of the American Mathematical Society, 140(11):3687–3692, 2012.
  * [32] M. Simonovits.  A method for solving extremal problems in graph theory, stability problems.  In Theory of Graphs (Proc. Colloq., Tihany, 1966), pages 279–319. Academic Press, New York-London, 1968.
  * [33] M. Simonovits.  A method for solving extremal problems in graph theory, stability problems.  In Theory of Graphs (Proc. Colloq., Tihany, 1966), pages 279–319. Academic Press, New York, 1968.
  * [34] C. Thomassen.  Girth in graphs.  J. Combin. Theory Ser. B, 35(2):129–141, 1983.
  * [35] J. Verstraëte.  On arithmetic progressions of cycle lengths in graphs.  Combin. Probab. Comput., 9(4):369–373, 2000.
  * [36] J. Verstraëte.  Vertex-disjoint cycles of the same length.  J. Combin. Theory Ser. B, 88(1):45–52, 2003.
  * [37] H. Wang.  An extension of the Corrádi-Hajnal theorem.  Australas. J. Combin., 54:59–84, 2012.
  * [38] R. Wenger.  Extremal graphs with no $C^{4}$’s, $C^{6}$’s, or $C^{10}$’s.  J. Combin. Theory Ser. B, 52(1):113–116, 1991.


# System Demo: Tool and Infrastructure for
Offensive Language Error Analysis (OLEA) in English

Marie Grace , Xajavion “Jay” Seabrum11footnotemark: 1, Dananjay
Srinivas11footnotemark: 1, Alexis Palmer
University of Colorado Boulder
<EMAIL_ADDRESS><EMAIL_ADDRESS>These authors contributed
equally.Please direct inquiries about the library to this email.

###### Abstract

The automatic detection of offensive language is a pressing societal need.
Many systems perform well on explicit offensive language but struggle to
detect more complex, nuanced, or implicit cases of offensive and hateful
language. OLEA is an open-source Python library that provides easy-to-use
tools for error analysis in the context of detecting offensive language in
English. OLEA also provides an infrastructure for re-distribution of new
datasets and analysis methods requiring very little coding.

##  1 Introduction

Offensive language111We use the term ”offensive language” to encompass
offensive language and hate speech. This paper contains censored offensive
language examples. detection models have become integral to online platforms’
moderation systems. Such systems excel at detecting and filtering out messages
with explicit keywords and mentions; however, these systems are known (1) to
perform poorly on messages that are implicitly offensive and that have
negation Röttger et al. (2020); Palmer et al. (2020); (2) to have annotator
biases affecting the detection of offensive language Sap et al. (2021); (3)
not to be robust to diachronic language and its usage Florio et al. (2020);
and (4) to be insensitive to and overdetect AAE as offensive language Sap et
al. (2021); Blodgett et al. (2016). These issues and gaps are important to
recognize, as failing to address them can cause marginalized groups to be
further dehumanized or attacked Mathew et al. (2021); Kennedy et al. (2020).

All of these complexities underscore the importance of having a detailed and
nuanced understanding of the true capabilities of models for automatic
detection of offensive language. We need to move away from viewing the task
primarily as a binary classification problem Kennedy et al. (2020), and we
need to consider more than a single F1 score when evaluating our models. We
also need a cohesive and common framework for error analysis, a tool to help
researchers more easily understand why their systems fail to detect certain
types of offensive speech Poletto et al. (2021). We need a better
understanding of model performance on out-of-domain data if we are to achieve
robust models that are quickly deployable on new domains. Finally, we need to
address some of the practical hurdles that prevent more widespread adoption of
some datasets, taking advantage of recent improvements in NLP research
infrastructure.

With these issues in mind, we introduce OLEA222
https://pypi.org/project/olea/,333
https://www.youtube.com/watch?v=730MZktD5q4, an open-source Offensive Language
Error Analysis tool and infrastructure, providing:

  1. 1.

easy-to-use methods for error analysis and evaluation of new models on
existing diagnostic datasets, including model comparison;

  2. 2.

interfaces to two diagnostic datasets focused on nuanced linguistic analysis;
and

  3. 3.

scaffolding to support easy distribution of new datasets and associated
analysis methods.

The error analysis and evaluation tools provide insights into where and how
models can be improved. The infrastructure provided by OLEA helps researchers
distribute datasets to the broader research community, together with dataset-
specific analysis methods, all with very low overhead.

##  2 Background and Related Work

Figure 1: Diagram of the OLEA Library pipeline.

Offensive language is complex, and systems for detecting it automatically need
to be able to handle both explicit and implicit cases Schmidt and Wiegand
(2017); Waseem et al. (2017). Detecting explicit offensive language often
relies on keyword detection Wiegand et al. (2019), but keyword-driven systems
can lead to messages being falsely flagged, causing unchecked or unnoticed
racial biases to propagate in the system’s decisions Sap et al. (2021);
Blodgett et al. (2016). Implicit offensive language is generally more
difficult to detect than its explicit counterpart ElSherief et al. (2021);
Caselli et al. (2020). Furthermore, it is more likely to change over time
given real world circumstances and coinages of new phrases and terms to
implicitly refer to minority groups Florio et al. (2020).

Datasets for this task take varied approaches. For example, HateXplain Mathew
et al. (2021) and CAD Vidgen et al. (2021) both provide rationales indicating
where annotators see offensive content. OLID Zampieri et al. (2019a)
identifies offensive text and the specific targeted minority group in a three-
tiered labeling structure. HateCheck Röttger et al. (2020) and COLD Palmer et
al. (2020) are described more in Section 3.1. Because many of these datasets
address different (often overlapping) concerns, direct comparison is
difficult.

Additionally, linguistic explainability of the prediction failures of NLP
models has lagged behind their performance as measured against benchmark
datasets Hovy (2022). To address part of this concern, McMillan-Major et al.
(2022) provide an interactive system mostly for end users of offensive
language detection systems. Their system helps users explore datasets and
understand how individual text inputs are scored and classified by different
models. OLEA has complementary functionality, focusing on fine-grained
analysis of model performance (especially misclassifications) across existing
evaluation datasets. We focus on model developers rather than end users,
providing streamlined error analysis and interpretation of system outputs
relative to linguistically-grounded categorizations.

##  3 Library Tour and Design

Figure 1 shows an overview of OLEA’s core functionalities. Users submit their
model’s predictions (3.2) on OLEA’s preloaded datasets (3.1) and then call
error analysis and evaluation functions (3.3). Users may also extend OLEA with
new datasets and may write new analysis functions, adding to the library’s
capabilities (3.4).

OLEA can be installed using pip (pip install olea). Most of OLEA’s modules
expect a Pandas444https://pandas.pydata.org/docs/ dataframe with the text of
the instance to be classified, one or more labels indicating offensiveness,
and a predicted label for the instance. Dataframes may include columns with
additional information related to the instance and/or its annotation.

###  3.1 Preloaded diagnostic datasets

The primary function of OLEA is to make it easy for users to evaluate the
capabilities of their models in a fine-grained way. We provide interfaces (via
HuggingFace’s datasets library555https://huggingface.co/docs/datasets/index
and the HuggingFace Hub) to two datasets designed specifically for diagnostic
evaluation of offensive language detection systems. Both datasets include
fine-grained annotations and binary offensiveness labels and were curated to
compare model performance with linguistic phenomena. Appendices C and D list
the features available for analysis.

HateCheck Röttger et al. (2020) provides a test suite that includes labels
reflecting some of the specific linguistic constructions often seen in online
hate speech, such as use of spelling changes to obscure hateful language and
presence of threatening language. HateCheck also includes annotations of the
specific identities targeted in each instance of hate speech. COLD Palmer et
al. (2020) provides fine-grained labels, from multiple annotators, of some
linguistic phenomena relevant for implicit hate speech and offensive language.
Some examples are presence/absence of slur terms and presence/absence of
adjectival nominalizations.

###  3.2 Submitting predictions

Before using the analysis functions described below, the user needs to submit
their model’s predictions on the selected dataset, as well as a mapping
between the model’s predicted labels (e.g. 1, 0) and the labels in the
selected dataset (e.g. hateful, non-hateful).666Note that this process applies
both for preloaded datasets and for datasets read in from the user’s own
system. The code snippet below illustrates the process and assumes that the
user’s data has been stored as a Pandas dataframe named user_data. In this
example, the model predictions are found in a column called predictions. The
user has selected three features for potential analysis: Text, is_slur and
text_length.

⬇

1from olea.data import Dataset

2setup = Dataset(

3 data = user_data,

4 features = ["Text","is_slur",

5 "text_length"],

6 gold_column = "gold_labels",

7 text_column = "Text")

8predictions = user_data["predictions"]

9mapping = {"hateful": 1,

10 "non-hateful": 0}

11data_submit = setup.submit(

12 batch = user_data,

13 predictions = predictions,

14 map = mapping)

The submit method passes the relevant parameters to the analysis module.

###  3.3 Error analysis functions

The heart of our library is a collection of functions for detailed evaluation
and error analysis. Throughout, we evaluate the model’s coarse-grained
classification performance (e.g. offensive vs. not offensive) for subsets of
instances grouped according to a particular feature. The features generally
correspond to dataframe columns. For example, we may compare performance for
instances containing a slur term to performance for instances with no slur
term. Plots are produced using Matplotlib Hunter (2007), and we include the
option to save plots to files. Section 4 shows concrete examples of the
analysis outputs, and code examples appear in Appendix A.

#####  analyze_on.

In its most general version, this function evaluates model performance for a
categorical column specified by the user. OLEA includes versions of analyze_on
customized to the two preloaded datasets. The COLD-specific version evaluates
performance for features constructed from combinations of four binary
features: offensiveness, presence of slur term, presence of adjectival
nominalization, and presence of linguistic distancing. The HateCheck-specific
version includes linguistic features (e.g. negation, derogation, or profanity)
and features related to the identity of the targeted individual or class (e.g.
trans people, Muslims, or disabled people).

#####  check_anno_agreement.

This function is intended for datasets which include labels from multiple
annotators, such as COLD. The function compares performance on instances with
full annotator agreement for the label of offensiveness to performance on
instances with partial agreement. Full annotator agreement is taken as a proxy
for instances that are “easy” to classify, and partial agreement indicates
more complex cases.

#####  aave.

This function evaluates performance for instances (likely) written using
African American English. The scores are calculated using the TwitterAAE model
Blodgett et al. (2016). These scores represent an inference of the proportion
of words in the instance that come from a demographically-associated
language/dialect.

#####  check_substring.

Given a user-specified text string, this function compares performance on
instances with the substring to instances without.

#####  str_len_analysis.

This function outputs a histogram showing model performance on instances of
different lengths (character or word count).

###  3.4 Adding datasets and analyses

Extensibility is a key principle guiding the design of OLEA, with the goal of
providing an easy-to-use, uniform platform for error analysis in the context
of offensive language detection. In addition to the two preloaded datasets,
users can submit their own datasets using the process described in 4.2.
Modifying a single Python class enables a built-in suite of analysis
functions.

OLEA has a helper function for preprocessing English text to remove user names
and URLs and convert emoji to their textual descriptions.777Preprocessing
scheme is described in Palmer et al. (2020).

⬇

1from olea.utils.preprocess_text import PreprocessText as pt

2processed_text = pt.execute(user_data["raw_text"])

3user_data["preprocessed_text"] = preprocessed_text

For example, the text preprocessor will convert "@username_1 Have you seen the
video that @another_user made? :eyes: :fire: :hundred-points:
https://fakelink.io" to "USER have you seen the video that USER made? eyes
fire hundred points HTML".

Finally, users can write and share their own analysis functions, focusing on
user-specified dimensions, as in 4.2.2. OLEA’s code is modularized such that
adding a new analysis requires enough Python knowledge to write the function,
but not a detailed understanding of the entire codebase.

##  4 Use Case Demonstrations

The three main use cases envisioned are: a) analysis on preloaded datasets
(4.1), including model comparison (4.1.4), b) analysis on custom data (4.2),
and c) sharing datasets and analysis functions (4.2).

###  4.1 User model performance evaluation using preloaded datasets

This section demonstrates how to use OLEA for detailed analysis of the
strengths and weaknesses of existing offensive language detection models. For
this demo, we use roBERTa-offensive (Barbieri et al., 2020), a pre-trained
generic language model, fine-tuned on the SemEval2019 OffensEval dataset
Zampieri et al. (2019b). We use this model to make top-level predictions
(offensive or not) for both COLD and HateCheck.

Each individual error analysis function shows the model’s performance with
respect to a particular feature (i.e. an existing dataframe column, or a new
one added by the function). Each function returns two dataframes. The metrics
dataframe contains a classification report for the analysis.888Appendix A
provides more code examples for loading in data and starting generic analyses.
This dataframe uses OLEA’s built in Metrics function, which is built upon and
uses Scikit-learn’s Pedregosa et al. (2011) metrics library.

The plot_info dataframe contains details of the analysis for the selected
dimension, plus computed accuracy and the option to show textual examples. If
show_examples = True, the function returns one randomly-selected incorrectly-
classified instance for each value of the dimension being analyzed.999The
variable show_examples defaults to false to avoid accidental viewing of
hateful or offensive language. If the plot option is selected, the plots are
built from the plot_info dataframe.

####  4.1.1 Generic analysis functions

| N | Y | macro avg | weighted avg
---|---|---|---|---
precision | 0.743 | 0.587 | 0.665 | 0.670
recall | 0.502 | 0.803 | 0.652 | 0.643
f1-score | 0.599 | 0.678 | 0.639 | 0.636
support | 1072 | 944 | 2016 | 2016

Table 1: Metrics classification report for roBERTa-offensive on COLD, using
analyze_on on the dimension of offensiveness. N$=$not offensive,
Y$=$offensive.

Annotator Agreement | Full | Partial
---|---|---
Total | 1431 | 585
Total Correct | 1004 | 292
Accuracy | 0.702 | 0.499
| Incorrect Classification
---
Example
| an illegal is not an
---
immigrant and
illegals do take
american jobs
considering they
are not americans.
| USER yooo i was
---
thinking bout
that the other day
lol.. you only really
speaking of one
person my n*ggah
| Example’s
---
Predicted Label
N | Y
Example’s Gold Label | Y | N

Table 2: plot_info report for roBERTa-offensive on COLD, using
check_anno_agreement (full vs. partial), with randomly-selected examples.

Table 1 shows the classification report for roBERTa-offensive on COLD. Here,
the classification report provides F1, precision, and recall for the two
categories of offensive and non-offensive, as well as the macros and weighted
averages. This model performs better overall on offensive instances, with high
recall, but shows much better precision for non-offensive instances. These
reports can be easily modified to analyze subsets of the data.

Table 2 shows plot_info for roBERTa-offensive on COLD, using
check_anno_agreement. The table shows accuracy for each category (full vs.
partial) and one example incorrect prediction. Accuracy is much higher for
instances with full agreement than for those with some disagreement.
Offensiveness can be subjective, so it is useful to examine model performance
on these different cases. Showing examples allows users to review difficult
cases and may provide insights for model improvement.

Figure 2: Results for roBERTa-offensive on COLD, focusing on fine-grained
categories. Percent value above a bar shows percent accuracy for that
category.

####  4.1.2 COLD analysis

Figure 3: Results for roBERTa-offensive on HateCheck, focusing on identity of
the target. Figure 4: Comparison of HateXplain and roBERTa models on the fine-
grained HateCheck categories. The category labels are followed by either (h)
to denote that it is comprised of only hateful instances, (nh) for only non-
hateful instances, or nothing to denote a mix of hateful and non-hateful
instances

(a) Results for roBERTa-offensive on HateCheck, focusing on fine-grained
categories.

(b) Results for HateXplain on HateCheck, focusing on fine-grained categories.

The next analysis (Fig. 2) is on the fine-grained COLD categories. roBERTa
performs well on offensive tweets containing slurs and poorly on the reclaimed
category (non-offensive tweets containing slurs). The model also performs
poorly on offensive tweets containing adjectival nominalization. These
insights suggest that the model relies too much on slurs for identifying
offensive language.

####  4.1.3 HateCheck analysis

HateCheck includes a column identifying the targeted group of the offensive
language. This analysis can help identify which groups a model is more or less
sensitive to. Figure 3 shows that the model does a better job of recognizing
hateful speech when an identity is targeted than when one is not.

Figure 4(a) analyzes the performance of roBERTa-offensive on the fine-grained
HateCheck categories; complete table is in Appendix B. Most instances of
counter-speech (when hate speech is quoted or is referred to with a non-
hateful effect) are incorrectly classified. An example of counter-speech is "I
lost all respect for you when you said women are pathetic". Instances
belonging to the categories of derogation, identity, threats, pronoun
references, and phrasing are mostly correctly classified.

####  4.1.4 Model comparison

So far we have compared one model’s performance on two different datasets.
Next, we compare two different models on the same dataset. The second model is
HateXplain Mathew et al. (2021). Both models of course suffer when classifying
data sourced from outside their training domain.101010roBERTa-offensive
reports an F1 of 0.78 on OLID but drops to 0.62 on HateCheck. HateXplain
reports F1 of 0.69 on the HateXplain dataset, and drops to 0.37 on
HateCheck.,111111We map HateXplain’s “offensive” and “hate speech” labels both
to HateCheck’s “hateful”. But we get a much truer sense of the models’ out-of-
domain capabilities when we look at the side-by-side analysis of the two
models for the fine-grained HateCheck categories (Figures 4(a) and 4(b)).
While roBERTa-offensive does not perform well on counter-speech, HateXplain
correctly classifies most counter-speech instances. And while HateXplain
struggles to recognize hateful expressions with spelling changes, roBERTa does
much better.

###  4.2 OLEA as infrastructure: Extending functionality

OLEA is open-source121212https://github.com/alexispalmer/olea, Licensed under
MIT License and has been designed to be extensible with new datasets and new
analyses.

####  4.2.1 Analysis on custom data

The analysis methods described above can be easily applied to new corpora. The
code below shows the process of loading the OLID dataset Zampieri et al.
(2019a) as a pandas dataframe. The user only needs to specify a path to the
data and the relevant column headings. The Dataset class acts as a wrapper for
the data loaded from disk and allows the user to access class utilities such
as generator(), which in turn is helpful for accessing data in batches.

⬇

1olid = pd.read_csv(’data/olid/olid_levela.csv’)

2olid_dataset = Dataset(data = olid,

3 features = ’Text’,

4 gold_column = ’label’,

5 text_column = ’Text’)

6data_gen = olid_dataset.generator(batch_size=64)

7data = next(data_gen)

We can now submit model predictions, returning a DatasetSubmissionObject which
can be used to conduct the generic analyses previously described; code in
Appendix A.

####  4.2.2 Sharing datasets and analysis patterns

With just a bit of coding, interfaces new datasets can be added to the OLEA
library more permanently, and for the benefit of all users.131313OLEA is not
currently hosting datasets. The preloaded datasets are hosted via
HuggingFace’s datasets library. We demonstrate again using OLID,141414Note
that we only consider OLID’s ”level-A” annotations. establishing the new
OLIDDataset class which inherits from Dataset.

⬇

1class OLIDDataset(Dataset) :

2 text_column = ’Text’

3 gold_column = ’label’

4 features = [’Text’,’label_id’]

5def __init__(self, olid_csv_path:str) :

6 self.olid_csv_path = olid_csv_path

7 self._data = self._load_data()

8def _load_data(self) -> pd.DataFrame:

9 return pd.read_csv(self.olid_csv_path)

To accommodate the properties of the new dataset, we need to override some
attributes of the Dataset class and to modify the method for loading data.

OLEA’s scaffolding minimizes the amount of new code needed to add a new
dataset, as well as automatically handling helper utilities such as mapping
model predictions to the custom Dataset object. The library also runs sanity
checks on submitted predictions before returning a DatasetSubmissionObject.

The advantage of using a native DatasetSubmissionObject is that users may run
Generic analyses on it. However, if authors have a unique analysis that they
wish to couple with their dataset, they may specify a special Analysis class
that can operate on submissions. The class methods can be modified to
accommodate patterns or properties specific to the dataset.

⬇

1class OLIDAnalysis(object) :

2 @classmethod

3 def analyze_on(cls, submission:DatasetSubmissionObject, on:str) :

4 ’’’

5 Unique OLID analysis goes here!

6 ’’’

7 return get_metrics(submission, on)

##  5 Conclusion and Future Directions

This paper introduces OLEA, a tool for easy, in-depth error analysis and an
infrastructure for sharing new datasets and analysis methods. OLEA helps
researchers understand the strengths and weaknesses of their offensive
language detection models. In the near term, we will continue to add new
analysis methods and datasets, including methods for corpus exploration. Mid
term, we plan to extend OLEA to additional languages, and eventually we would
like to expand OLEA into a general error analysis library for a range of
language classification tasks. Because OLEA is a convenient way for authors to
share datasets and analyses, it is our hope that a community will develop
around the library, and that models ultimately will improve as we learn more
about what they can and cannot do.

## Limitations

OLEA is restricted in the languages that it can be used on. Currently, it
assumes for some of the analyses on substrings that the language is delimited
by spaces. Additionally, the library is primarily focused on providing error
analyses for offensive language applications. Its use outside of this domain
is not known or well defined. Though we focus on error analysis for offensive
language identification, the system makes no binding assumptions as to the
proper definitions of offensive language and hate speech, nor does it assert
(or assume) any difference between these two categories. This lack of
distinction complicates cross-model comparison. Furthermore, this tool is just
an analysis tool; it does not address concerns regarding language drift and
other sociolinguistic biases that may be present within a user’s dataset, nor
does it address any annotator biases present in original datasets. OLEA is not
a fully independent package as it relies on the external site HuggingFace for
dataset hosting.

## Ethics Statement

This work provides important insights on hate speech and offensive language
models for users, but we acknowledge the ethical implications of releasing a
tool that encourages accessing hate speech datasets where tweet author
anonymity may not be ensured. We attempt to take steps to minimize user
exposure to hateful or offensive language when possible, by defaulting
show_examples to False during analysis. This tool is intended to help
researchers understand their model performance, and should not be used for any
task that promotes or spreads the usage of hate speech or unnecessarily
exposes people to hate speech.

## References

  * Barbieri et al. (2020) Francesco Barbieri, José Camacho-Collados, Leonardo Neves, and Luis Espinosa Anke. 2020.  Tweeteval: Unified benchmark and comparative evaluation for tweet classification.  _CoRR_ , abs/2010.12421.
  * Blodgett et al. (2016) Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016.  Demographic dialectal variation in social media: A case study of African-American English.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1119–1130, Austin, Texas. Association for Computational Linguistics.
  * Caselli et al. (2020) Tommaso Caselli, Valerio Basile, Jelena Mitrović, Inga Kartoziya, and Michael Granitzer. 2020.  I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 6193–6202, Marseille, France. European Language Resources Association.
  * ElSherief et al. (2021) Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021.  Latent hatred: A benchmark for understanding implicit hate speech.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 345–363, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Florio et al. (2020) Komal Florio, Valerio Basile, Marco Polignano, Pierpaolo Basile, and Viviana Patti. 2020.  Time of your hate: The challenge of time in hate speech detection on social media.  _Applied Sciences_ , 10(12):4180.
  * Hovy (2022) Eduard Hovy. 2022.  Rediscovering the need for representation and knowledge.  ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics.
  * Hunter (2007) J. D. Hunter. 2007.  Matplotlib: A 2d graphics environment.  _Computing in Science & Engineering_, 9(3):90–95.
  * Kennedy et al. (2020) Chris J Kennedy, Geoff Bacon, Alexander Sahn, and Claudia von Vacano. 2020.  Constructing interval variables via faceted rasch measurement and multitask deep learning: a hate speech application.  _arXiv preprint arXiv:2009.10277_.
  * Mathew et al. (2021) Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021.  Hatexplain: A benchmark dataset for explainable hate speech detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 14867–14875.
  * McMillan-Major et al. (2022) Angelina McMillan-Major, Amandalynne Paullada, and Yacine Jernite. 2022.  An interactive exploratory tool for the task of hate speech detection.  In _Proceedings of the Second Workshop on Bridging Human–Computer Interaction and Natural Language Processing_ , pages 11–20, Seattle, Washington. Association for Computational Linguistics.
  * Palmer et al. (2020) Alexis Palmer, Christine Carr, Melissa Robinson, and Jordan Sanders. 2020.  COLD: Annotation scheme and evaluation data set for complex offensive language in English.  _Journal for Language Technology and Computational Linguistics_ , 34(1):1–28.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Poletto et al. (2021) Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2021.  Resources and benchmark corpora for hate speech detection: a systematic review.  _Language Resources and Evaluation_ , 55(2):477–523.
  * Röttger et al. (2020) Paul Röttger, Bertram Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet B Pierrehumbert. 2020.  Hatecheck: Functional tests for hate speech detection models.  _arXiv preprint arXiv:2012.15606_.
  * Sap et al. (2021) Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A Smith. 2021.  Annotators with attitudes: How annotator beliefs and identities bias toxic language detection.  _arXiv preprint arXiv:2111.07997_.
  * Schmidt and Wiegand (2017) Anna Schmidt and Michael Wiegand. 2017.  A survey on hate speech detection using natural language processing.  In _Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media_ , pages 1–10, Valencia, Spain. Association for Computational Linguistics.
  * Vidgen et al. (2021) Bertie Vidgen, Dong Nguyen, Helen Margetts, Patricia Rossini, and Rebekah Tromble. 2021.  Introducing CAD: the contextual abuse dataset.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2289–2303, Online. Association for Computational Linguistics.
  * Waseem et al. (2017) Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017.  Understanding abuse: A typology of abusive language detection subtasks.  In _Proceedings of the First Workshop on Abusive Language Online_ , pages 78–84, Vancouver, BC, Canada. Association for Computational Linguistics.
  * Wiegand et al. (2019) Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019.  Detection of Abusive Language: the Problem of Biased Datasets.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 602–608, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Zampieri et al. (2019a) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019a.  Predicting the Type and Target of Offensive Posts in Social Media.  In _Proceedings of NAACL_.
  * Zampieri et al. (2019b) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019b.  SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval).  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 75–86, Minneapolis, Minnesota, USA. Association for Computational Linguistics.

##  Appendix A Code Examples

In this appendix, we provide code snippets corresponding to Section 4.

###  A.1 Preliminaries

Import Statements

⬇

1import pandas as pd

2from olea.data.cold import COLD, COLDSubmissionObject

3from olea.data.hatecheck import HateCheck

4from olea.analysis.cold import COLDAnalysis

5from olea.analysis.generic import Generic

6from olea.analysis.hatecheck import HateCheckAnalysis

7from olea.utils import preprocess_text

###  A.2 Code examples for Section 4.1

Download and Preprocess COLD text:

⬇

1cold = COLD()

2pt = preprocess_text.PreprocessText()

3processed_text = pt.execute(cold.data()["Text"])

4cold.data()["Text"] =processed_text

Create predictions using roBERTa-offensive

⬇

1from transformers import AutoTokenizer, AutoModelForSequenceClassification

2from transformers import TextClassificationPipeline

3

4 link = "cardiffnlp/twitter-roberta-base-offensive"

5 tokenizer = AutoTokenizer.from_pretrained(link)

6 model = AutoModelForSequenceClassification.from_pretrained(link)

7 #Create Pipeline for Predicting

8 pipe = TextClassificationPipeline(model=model, tokenizer=tokenizer)

9 preds = pd.DataFrame(pipe(list(cold.data()["Text"])))["label"]

Create Submission Objects:

⬇

1cold_so = cold.submit(

2 batch = cold.data(),

3 predictions = preds,

4 map = {"LABEL_0": ’N’, ’LABEL_1’: "Y"})

⬇

1hc_so = hc.submit(

2 batch = hc.data(),

3 predictions = preds,

4 map = {"LABEL_0": ’non-hateful’, ’LABEL_1’: "hateful"})

Generate Table 1:

⬇

1plot_info, metrics = Generic.analyze_on(

2 cold_so,

3 ’Cat’,

4 show_examples = False,

5 plot = False)

Generate Table 1 and Save plot to file:

⬇

1plot_info, metrics = Generic.analyze_on(

2 cold_so,

3 ’Cat’,

4 show_examples = False,

5 plot = False,

6 savePlotToFile= "cold_cats.png")

Generate Table 2:

⬇

1plot_info, metrics = Generic.check_anno_agreement(cold_so,
["Off1","Off2","Off3"],show_examples = True, plot = False)

Generate Figure 2:

⬇

1plot_info, metrics =COLDAnalysis.analyze_on(

2 cold_so,

3 ’Cat’,

4 show_examples = False,

5 plot = True)

Generate Figure 3:

⬇

1plot_info, metrics = Generic.analyze_on(

2 hc_so,

3 ’target_ident’)

Generate Figure 4(a), Figure 4(b), and Table 3:

⬇

1plot_info, metrics = HateCheckAnalysis.analyze_on(

2 hc_so,

3 ’category’)

###  A.3 Code examples for Section 4.2

Run analysis functions on local custom data:

⬇

1predictions = model.predict(data)

2submission = olid_dataset.submit(

3 batch = data,

4 predictions = predictions,

5 map = {1:’OFF’, 0:’NOT’})

6# performance on AAVE

7Generic.aave(submission)

8# performance on texts containing substring ’female’

9Generic.check_substring(submission, "female")

Submit predictions for newly-established dataset class OLIDDataset:

⬇

1olid = OLIDDataset(’data/olid.csv’)

2datagen = olid.generator(64)

3data = next(datagen)

4preds = model.predict(data)

5map = {’OFF’ : 1.0 , ’NOT’ : 0.0}

6submission = olid.submit(batch = data,

7 predictions = preds,

8 map = map)

##  Appendix B Full Results Table for Figure 4(a)

category | Metrics | precision | recall | f1-score | support
---|---|---|---|---|---
counter (nh) | hateful | 0.000 | 0.000 | 0.000 | 0
counter (nh) | non-hateful | 1.000 | 0.038 | 0.074 | 314
counter (nh) | macro avg | 0.500 | 0.019 | 0.037 | 314
counter (nh) | weighted avg | 1.000 | 0.038 | 0.074 | 314
derogation (h) | hateful | 1.000 | 0.805 | 0.892 | 560
derogation (h) | non-hateful | 0.000 | 0.000 | 0.000 | 0
derogation (h) | macro avg | 0.500 | 0.403 | 0.446 | 560
derogation (h) | weighted avg | 1.000 | 0.805 | 0.892 | 560
identity(nh) | hateful | 0.000 | 0.000 | 0.000 | 0
identity(nh) | non-hateful | 1.000 | 0.892 | 0.943 | 315
identity(nh) | macro avg | 0.500 | 0.446 | 0.471 | 315
identity(nh) | weighted avg | 1.000 | 0.892 | 0.943 | 315
negation | hateful | 0.295 | 0.236 | 0.262 | 140
negation | non-hateful | 0.335 | 0.406 | 0.367 | 133
negation | macro avg | 0.315 | 0.321 | 0.315 | 273
negation | weighted avg | 0.315 | 0.319 | 0.313 | 273
nonhateful-abuse (nh) | hateful | 0.000 | 0.000 | 0.000 | 0
nonhateful-abuse (nh) | non-hateful | 1.000 | 0.339 | 0.506 | 192
nonhateful-abuse (nh) | macro avg | 0.500 | 0.169 | 0.253 | 192
nonhateful-abuse (nh) | weighted avg | 1.000 | 0.339 | 0.506 | 192
phrasing (h) | hateful | 1.000 | 0.868 | 0.929 | 273
phrasing (h) | non-hateful | 0.000 | 0.000 | 0.000 | 0
phrasing (h) | macro avg | 0.500 | 0.434 | 0.465 | 273
phrasing (h) | weighted avg | 1.000 | 0.868 | 0.929 | 273
profanity | hateful | 0.601 | 1.000 | 0.751 | 140
profanity | non-hateful | 1.000 | 0.070 | 0.131 | 100
profanity | macro avg | 0.800 | 0.535 | 0.441 | 240
profanity | weighted avg | 0.767 | 0.613 | 0.492 | 240
pronoun-references (h) | hateful | 1.000 | 0.908 | 0.952 | 273
pronoun-references (h) | non-hateful | 0.000 | 0.000 | 0.000 | 0
pronoun-references (h) | macro avg | 0.500 | 0.454 | 0.476 | 273
pronoun-references (h) | weighted avg | 1.000 | 0.908 | 0.952 | 273
slurs | hateful | 0.593 | 0.778 | 0.673 | 144
slurs | non-hateful | 0.515 | 0.306 | 0.384 | 111
slurs | macro avg | 0.554 | 0.542 | 0.528 | 255
slurs | weighted avg | 0.559 | 0.573 | 0.547 | 255
spelling changes (h) | hateful | 1.000 | 0.549 | 0.709 | 760
spelling changes (h) | non-hateful | 0.000 | 0.000 | 0.000 | 0
spelling changes (h) | macro avg | 0.500 | 0.274 | 0.354 | 760
spelling changes (h) | weighted avg | 1.000 | 0.549 | 0.709 | 760
threats (h) | hateful | 1.000 | 0.810 | 0.895 | 273
threats (h) | non-hateful | 0.000 | 0.000 | 0.000 | 0
threats (h) | macro avg | 0.500 | 0.405 | 0.447 | 273
threats (h) | weighted avg | 1.000 | 0.810 | 0.895 | 273

Table 3: The Metrics classification report for roBERTa-offensive on HateCheck

##  Appendix C COLD: Features available for analysis

Feature | Description
---|---
ID | The unique ID for the text
Text |  | The text containing social media messages
---
(some containing offensive language)
Cat | The gold label category of the text
Off |  | Offensive or not? ( Y / N )
---
Majority Vote
Off1, Off2, Off3 | Individual annotator labels for Off ( Y / N )
Slur |  | Contains a slur? ( Y / N )
---
Majority Vote
Slur1, Slur2, Slur3 | Individual annotator labels for Slur ( Y / N )
Nom |  | Contains adjectival nominalization? ( Y / N )
---
Majority Vote
Nom1, Nom2, Nom3 | Individual annotator labels for Slur ( Y / N )
Dist |  | Contains linguistic distancing? ( Y / N )
---
Majority Vote
Dist1, Dist2, Dist3 | Individual annotator labels for Dist ( Y / N )

Table 4: Features available as part of the COLD dataset. The feature names and
descriptions come from Palmer et al. (2020).

##  Appendix D HateCheck: Features available for analysis

Feature | Description
---|---
functionality |  | The shorthand for the functionality tested
---
by the test case.
case_id |  | The unique ID of the test case (assigned to
---
each of the 3,901 cases initially generated)
test_case | The text of the test case.
label_gold |  | The gold standard label (hateful/non-hateful)
---
of the test case. All test cases within a given
functionality have the same gold standard label.
target_ident |  | Where applicable, the protected group
---
targeted or referenced by the test case. We
cover seven protected groups in the test suite:
women, trans people, gay people, black people,
disabled people, Muslims and immigrants.
direction |  | For hateful cases, the binary secondary label
---
indicating whether they are directed at an
individual as part of a protected group or aimed
at the group in general.
focus_words |  | Where applicable, the key word or phrase in a
---
given test case (e.g. "cut their throats").
focus_lemma |  | Where applicable, the corresponding
---
lemma (e.g. "cut sb. throat").
ref_case_id |  | For hateful cases, where applicable, the ID of the
---
simpler hateful case which was perturbed to
generate them. For non-hateful cases, where
applicable, the ID of the hateful case which is
contrasted.
ref_templ_id | The equivalent, but for template IDs.
templ_id |  | The unique ID of the template from which the
---
test case was generated (assigned to each of the
866 cases and templates from which we
generated the 3,901 initial cases).

Table 5: Features available as part of the HateCheck dataset. Feature names
and descriptions come from Röttger et al. (2020).