# COVID-19 confines recreational gatherings in Seoul to familiar, less
crowded, and neighboring urban areas

Jisung Yoon  Kellogg School of Management at Northwestern University,
Evanston, IL 60208, USA.  Northwestern Institute on Complex Systems, Evanston,
IL 60208, USA.  Department of Industrial and Management Engineering, Pohang
University of Science and Technology, Pohang 37673, Republic of Korea  Woo-
Sung Jung  Department of Industrial and Management Engineering, Pohang
University of Science and Technology, Pohang 37673, Republic of Korea
Department of Physics, Pohang University of Science and Technology, Pohang
37673, Republic of Korea  Hyunuk Kim  Department of Administrative Sciences,
Metropolitan College, Boston University, MA 02215, USA  Corresponding author:
<EMAIL_ADDRESS>
## Abstract

Recreational gatherings are sources of the spread of infectious diseases.
Understanding the dynamics of recreational gatherings is essential to building
effective public health policies but challenging as the interaction between
people and recreational places is complex. Recreational activities are
concentrated in a set of urban areas and establish a recreational hierarchy.
In this hierarchy, higher-level regions attract more people than lower-level
regions for recreational purposes. Here, using customers’ motel booking
records which are highly associated with recreational activities in Korea, we
identify that recreational hierarchy, geographical distance, and attachment to
a location are crucial factors of recreational gatherings in Seoul, Republic
of Korea. Our analyses show that after the COVID-19 outbreak, people are more
likely to visit familiar recreational places, avoid the highest level of the
recreational hierarchy, and travel close distances. Interestingly, the
recreational visitations were reduced not only in the highest but also in low-
level regions. Urban areas at low levels of the recreational hierarchy were
more severely affected by COVID-19 than urban areas at high and middle levels
of the recreational hierarchy.

## Introduction

Human urban activities are principal elements of social phenomena, including
the growth of cities [1, 2], economies [3, 4], and epidemics [5]. They tend to
be concentrated in parts of cities and form a hierarchy of geographical areas,
where regions at upper levels attract more people than those at lower levels
[6, 7, 8]. A person may frequently visit a popular region, often referred to
as a hotspot [9, 10, 11], even though it is far from living areas.

Strong urban hierarchy raises various concerns during a pandemic [12, 13, 14].
The spread of infectious diseases would be broad and prevalent if it
originates from a hotspot at the top of the hierarchy [15, 16, 17, 18]. The
economic impact of a pandemic also differs by hierarchy level. The income of
the populations working in the informal economy, which is usually located at
low hierarchy levels, was negatively affected by the COVID-19 pandemic [19].
Despite its importance to human activities, the urban hierarchy has been
rarely considered when analyzing behavioral changes in response to a pandemic
[20, 21, 22, 23, 24].

Here, by using individual-level motel booking records (see Methods) from a
leading Korean accommodation platform, we compare a visitation pattern before
and after the COVID-19 outbreak. According to a market report in 2021 [25],
32.6% of the platform’s mobile application installers were in their 20s, 35.4%
were in their 30s, 23.9% were in their 40s, and 6.4% were above 50s.
Additionally, 37.7% were females, and 63.3% were males. Therefore, low- and
middle-income populations are likely to be the primary users of the platform
and the Korean accommodation market.

Motels are often located near recreational places such as pubs, nightclubs,
restaurants, and cafes in urban hotspots [26]. Especially, in recent years
that our data cover, the Korean motel industry has transformed itself into an
entertainment industry that provides physical spaces for relaxation and
cultural activities [27]. Young Koreans increasingly book motels for intimate
relationships and partying with friends because motels are more affordable and
accessible than hotels [28]. For these reasons, we use our motel reservation
data as a proxy for recreational gatherings in Seoul. The hierarchy of
recreational urban areas which is extracted from our motel booking data is
referred to as the recreational hierarchy. Our analyses show that recreational
hierarchy, geographical distance, and attachment to a location are important
factors of recreational gatherings in Seoul, the largest city of the Republic
of Korea.

## Results

Figure 1: (a) Reservation and check-in time distributions. Reservations and
check-in take place primarily after 12pm. (b) A heatmap for the reservation
and check-in times. Both reservation and check-in times show a similar pattern
and are concentrated after 12pm.

In our data, motel reservation and check-in times are concentrated after 12pm
(Fig. 1a) and there is little time difference between reservation and check-in
times (Fig. 1b). To validate whether our data capture recreational gatherings
in Seoul to some extent, we compare reservation counts at the administrative
division level with the mobility inflows from Seoul mobility data aggregating
GPS locations (See Methods). The rank correlation between the reservation
counts and the mobility inflows is significant ($\rho=0.347$, $p$-value
$<0.001$; Fig. 2a). The correlation becomes stronger if we only consider
nighttime inflows (from 9 pm to 6 am, $\rho=0.400$, $p$-value $<0.001$).

To understand the effect of COVID-19 on recreational gathering behaviors, we
split the data into two periods: pre-COVID-19 (From January 21, 2019 to
November 3, 2019) and post-COVID-19 (From January 20, 2020 to November 1,
2020). January 20, 2020 is the first day that a COVID-19 infection case was
reported in Korea. Both periods start from the fourth week of January and span
286 days. The weekly trend of reservation counts is shown in Fig. 2b. A
significant drop appears near Week 5, the first week of the official social
distancing in the Republic of Korea. The reservation counts were recovered
gradually to the normal state even after several restrictions were imposed.

Figure 2: (a) A comparison of the reservation counts to the mobility inflows
in Seoul, Republic of Korea. We aggregate the mobility inflows at the level of
the administrative division. Each dot represents an administrative division. A
significant correlation supports that the reservation history data can be a
good proxy for urban recreational gatherings in Seoul. (b) Weekly reservation
counts. For a data privacy concern, we normalize the weekly reservation counts
by the maximum weekly reservation count.

### Recreational hierarchy of Seoul

Figure 3:  (a-b) The pre- and post-COVID-19 recreational hierarchy maps. Each
cell represents a level-14 Google S2 cell colored by its hotspot level. The
areas with grey color represent cells with no accommodations. (c) The
complementary cumulative distribution function (CCDF) of individual
reservation counts. On average, individual reservation counts decrease after
the COVID-19 outbreak. (d) The reservation count distribution by the hotspot
level $p_{\ell}$. The inset shows the relative change of
$p_{\ell},(p_{\ell,post}-p_{\ell,pre})/p_{\ell,pre}$. A red up-arrow indicates
an increase of the probability and a blue down-arrow indicates a decrease of
the probability compare to the distribution for the pre-COVID-19 period.

We assign each motel to a Google S2 cell
(https://github.com/google/s2geometry). S2 cells are space tessellations that
divide the Earth into cells of a similar size area. It is known as a robust,
flexible spherical geometry [29, 30, 10]. We used level-14 S2 cells of which
size ranges from 0.19$km^{2}$ to 0.40$km^{2}$ (on average 0.32$km^{2}$). Then,
we aggregate the reservation counts by S2 cell and identify a hierarchy of
cells by assigning a hotspot level, an inverse decile rank of aggregated
reservation counts, to each cell. Level 1 is the highest, and level 10 is the
lowest level. Fig. 3a and Fig. 3b show the recreational hierarchy maps for
both periods. The assigned hotspot levels are almost consistent for both
periods. Cells with high levels correspond to popular recreational areas in
Seoul such as Gangnam, Sinchon, and Yeongdeungpo Time Square (highlighted in
Fig. 3a).

To further analyze behavioral changes induced by COVID-19, we take a subset of
customers who have at least two reservation records in both the pre- and post-
COVID-19 periods as the focus group. This focus group covers 30% of the total
customers in the pre-COVID-19 period and 26% in the post-COVID-19 period. The
distributions of individual reservation counts are similar for both periods,
but the average individual reservation counts decreased after the COVID-19
outbreak (Fig. 3c; $\langle l_{pre}\rangle=9.200>\langle
l_{post}\rangle=8.757$; $\text{paired t-statistic}=8.820,\text{p-value}\ll
0.001$).

As shown in Fig. 3d, the majority of reservations (61.6% for the pre-COVID-19
period, 60.7% for the post-COVID-19 period) is concentrated in the top 10%
cells, while the bottom 10% cells only have a few reservations (0.6% for the
pre-COVID-19 period, 0.3% for the post-COVID-19 period), suggesting the
inequality of recreational visitations on urban areas for both periods.
Interestingly, COVID-19 affects the inequality of urban areas differently by
the hierarchy level. The proportion of the highest level decreases after the
COVID-19 outbreak (Fig. 3d inset). However, this proportion was not equally
distributed across other levels. People visited levels 2, 3, and 4 rather than
low levels ($l\leq 6$). Our findings show that the COVID-19 pandemic worsened
the inequality across urban areas, in line with previous studies on income
levels [31, 32] and costs of shutdown [33]. We explain the worsening
inequality by decomposing individual recreational gathering behaviors in the
next section.

### Factors of recreational gatherings

Individual records can be converted to sequences of cells and hotspot levels.
The arrows in Fig. 4a represent a synthetic journey that consists of urban
areas. The cell trajectory of this example is $A\rightarrow B\rightarrow
A\rightarrow C\rightarrow A\rightarrow D$. Note that the same place can appear
multiple times. Based on the assigned levels of the cells, the level
trajectory is $1\rightarrow 1\rightarrow 1\rightarrow 3\rightarrow
1\rightarrow 2$. For each trajectory constructed from the data, we define the
most frequent cell as the recreational home, so $A$ is the home in the
example.

Figure 4: (a) An illustrative example of the cell and level trajectories. (b)
The flow matrix $T^{data}$ for the pre-COVID-19 period. The trips within the
same cell are excluded. We also provide the flow matrix for the post-COVID-19
period in Supplementary Information (Fig. S1). (c) The distributions of the
hotspot entropy $p_{h}^{data}$ and the radius of recreational activities
$p_{r}^{data}$ for both periods. After the outbreak, people explored less
across the recreational hierarchy. (d) Home ratios by sequence length for both
periods. Attachment to a location appears regardless of sequence length,
implying the share of time in the recreational home remains constant after the
outbreak.

#### Hierarchy: Transition between levels

We construct a flow matrix $T^{data}$ where $T^{data}_{ij}$ is the number of
trips between level $i$ and $j$ normalized by the total flows (Fig. 4b). We
here exclude self-transitions, trips within the same cell, to focus on the
transitions between different cells. The majority of transitions are
concentrated in high levels of the hierarchy, and the flow matrix is almost
symmetric. To check whether the transition from level $i$ to level $j$,
$p(j\mid i)$, depends on level $j$, we build a null model [10] that $p(j\mid
i)$ is proportional to the total inflows to destination’s level $j$ as follow,

$T_{ij}^{null}=\sum_{k=1}^{L}T_{ik}\frac{\sum_{m=1}^{L}T_{mj}}{\sum_{m,k=1}^{L}T_{mk}},$
(1)

where $\sum_{k=1}^{L}T_{ik}$ is the total outflow from level $i$ and
$\sum_{m=1}^{L}T_{mj}/\sum_{m,k=1}^{L}T_{mk}$ is the fraction of the inflows
to level $j$ (Supplementary Fig. S2). Comparing the ratio of $T^{data}$ to
$T^{null}$ (Supplementary Fig. S3), we confirm that $T^{data}$ is close to
$T^{null}$ at high levels of the hierarchy, while the ratios of the
transitions from or to low levels of the hierarchy increase. Most of the
transitions are at high levels of the hierarchy, and inflow and outflow are
symmetric (Supplementary Fig. S4; $R^{2}$=0.99 for both periods). Hence,
transitions between hotspot levels are approximately independent of the
previous place’s hotspot level $p(j\mid i)\simeq p(j)$ which follows the
reservation count distribution by the hotspot level $p_{\ell}$.

#### Hierarchy: hotspot entropy

To measure the extent to which hotspot levels are diverse in individual
records, we introduce the hotspot entropy, $h$ (See Methods). $h$ is zero if
the trajectory consists of cells of the same hotspot level. The maximum value
of $h$ is $\ln{\text{(number of hotspot levels)}}=\ln{10}$. The distributions
of $h$ for both periods are different ($\text{KS-
statistic}=0.046,\text{p-value}\ll 0.001$) and shown in Fig. 4c (left). Before
the COVID-19 outbreak, 30% of people stay only at a single level on average,
while this proportion increases after the outbreak. Also, the mean hotspot
entropy decreases ($\langle h_{pre}\rangle=0.518>\langle
h_{post}\rangle=0.475$), implying people are less likely to visit different
levels.

#### Geographical distance

The radius of recreational activities, $r$ (See Methods), a variance of
geographical distances from the recreational home of a sequence, quantifies
how far the places in a trajectory are. The unit of $r$ is a kilometer (km).
The distributions of $r$ show that the majority of people stays within a
single cell without moving to other cells (Fig. 4c, right). The likelihood of
visiting distant places is inversely proportional to geographical distance,
while the hierarchy leads people to move farther than expected (Supplementary
Fig. S5). Considering the radius of the biggest district in Seoul (Seocho
district) that is about 5.523km, we can say that more than 33% of the platform
users in the pre-COVID-19 period visit places outside the home cell ($\sim$30%
for the post-COVID-19 period). Overall, $r$ decreases after the COVID-19
outbreak ($\langle r_{pre}\rangle=4.094>\langle r_{post}\rangle=3.738$), and
the distributions for both periods are significantly different ($\text{KS-
statistic}=0.056,\text{p-value}\ll 0.001$). This evidence suggests that people
tend to stay close to their recreational homes after the outbreak.

#### Attachment to a location

Attachment to a location is an indicator of customer satisfaction and an
important factor for the accommodation business [34, 35]. In Fig. 4d, we show
the home ratio which is the fraction of the most frequent cell in a sequence.
Interestingly, the home ratio is about 0.6 regardless of sequence length. The
home ratio slightly increases after the COVID-19 outbreak for the light users
who booked motels no more than 20 times, while there is no difference in the
home ratio between the two periods for the heavy users who booked motels more
than 20 times (Top 10% users by sequence length).

### A model for replicating reservation records

Our empirical analysis reveals that recreational hierarchy, geographical
distance, and attachment to a location need to be considered simultaneously to
explain recreational gatherings in Seoul, Republic of Korea. Leveraging our
key findings on the individual movements, we develop a model reproducing their
patterns and detecting behavioral changes during the COVID-19 pandemic (Fig.
5a). Our model is motivated by the literature analyzing human mobility [20,
21, 24]. First, an agent starts from an initial cell randomly picked from the
cell-level reservation count distribution. Then, the agent explores places
with probability $p^{i}$ or chooses a previously visited place in proportion
to the frequency in the reservation history with probability $1-p^{i}$, where
$p\in[0,1]$ controls the likelihood that the agent decides to explore places
and $i$ is the number of iterations starting from one. As the iteration $i$
increases, the agent is more likely to choose previously visited places.

Figure 5: (a) A schematic diagram of our model. $k$, $\gamma$, and $p$ control
the effects of recreational hierarchy, geographical distance, and attachment
to a location, respectively. $i$ is the index of an iteration. (b) The flow
matrix of the best model (c) The distributions of the hotspot entropy
distribution $p_{h}$ (left) and the radius of recreational activities $p_{r}$
(right). Blue lines are the empirical distributions, and orange lines are the
simulation results. (d) Home ratios by sequence length. Here, we show the
results from the best model for the pre-COVID-19 period. The best model for
the post-COVID-19 period is in Supplementary Fig. S6.

If the agent decides to explore places, the agent first chooses the hotspot
level for the next move from the distribution $f(\ell)\propto\ell^{-k}$
($k\in[0,\rotatebox{90.0}{8})$). $k$ can be interpreted as the preference to
high hierarchy levels of the agent. For instance, if $k=0$, the agent randomly
selects the next level without considering recreational hierarchy. On the
other hand, with a large $k$, most reservations are concentrated in high
hierarchy levels. Next, the agent chooses a place of the given level in
proportion to the inverse of geographical distance with an exponent
$\gamma\in[0,\rotatebox{90.0}{8})$ which controls the likelihood of visiting
distant places from the recreational home. If $\gamma=0$, the agent ignores
the geographical distance and randomly picks the place with the given level.
In this step, the agent can pick the recreational home by imputing the
$d_{home,home}=1$, and the recreational home cell can change according to the
current history of the agent as the iteration proceeds. The steps above are
repeated until we have a sequence of which length is equal to the length of
the original trajectory. We keep the sequence length distribution from the
data (Fig. 3c) to control the effect of sequence length.

Through a grid search, we estimate the model parameters that minimize the
Jensen-Shannon divergence (JSD) of the hotspot entropy distribution $p_{h}$,
the reservation count distribution $p_{l}$, and the radius of recreational
activities distribution $p_{r}$ between synthetic sequences and the data. Note
that $p$ and $k$ affect $p_{l}$ and $p_{h}$, while $\gamma$ is independent of
$p_{l}$ and $p_{h}$. Taking advantage of this property, we jointly optimize
the model by searching the best $p$ and $k$ that minimize the product of JSD
of $p_{l}$ and $p_{h}$, namely $JSD_{Entropy}$ and $JSD_{Hierarchy}$. Next,
with the best $p$ and $k$, we fit the best $\gamma$ that minimizes
$JSD_{Radius}$, JSD of $p_{r}$. For the grid search, we explore $p$ with
dividing 0 to 1 into 51 bins (bin width = 0.02), $k$ with dividing 0 to 3 into
121 bins (bin width = 0.025), and $\gamma$ with dividing 0 to 5 into 201 bins
(bin width = 0.025). We repeat the simulation ten times and average the
estimated model parameters.

Our model successfully reconstructs the flow matrix $T_{model}$, all
distributions, and the retention of attachment to a location (Fig. 5b-d). Fig.
5b shows the flow matrix from the model, $T^{model}$. A matrix distance
between $T^{model}$ and $T^{data}$ measured by the Frobenius norm
$d_{T}=||T^{model}-T^{data}||_{F}$ is 0.03, indicating the model reproduces
the flows across the recreational hierarchy. Although there are gaps in the
first bin of the generated distributions and the home ratio, our model
captures the overall patterns of individual movements well. It is difficult to
model outliers who rarely move to other places. Furthermore, we compare the
simulated reservation counts from the model and the actual reservation counts
and find that the model also explains the reservation count well for both the
pre- and post-COVID-19 periods (Supplementary Fig. S7).

To better understand the effects of these factors on recreational gatherings,
we examine the variant models that exclude each factor (Supplementary Fig.
S8). From the estimated $p,k$, and $\gamma$ for the best model, we construct
the variant models by changing the target parameter while keeping other
parameters the same. For the model without recreational hierarchy, we simulate
the model with $k=0$. In this model, an agent does not consider recreational
hierarchy, and this change results in a collapse of the model in the flow
matrix ($d_{T}=0.35$; Supplementary Fig. S8a) and $p_{h}$ (Supplementary Fig.
S8b). Similarly, we try the model without geography with $\gamma=0$ where an
agent does not take into account geographical distance. This model still
produces the comparable result on the hierarchy ($d_{T}=0.06$) and $p_{h}$,
while $p_{r}$ is totally collapsed as expected (Supplementary Fig. S8e).
Lastly, we build the model without attachment to a location with $i=1$. In
this model, the likelihood that an agent explores a place is always $p$ so
that the likelihood does not depend on the iteration. The model without the
attachment reproduces the hierarchy ($d_{T}=0.04$), weakly collapses in
$p_{h}$ and $p_{r}$, but does not reproduce the retention of attachment to a
location (Supplementary Fig. S8i). In short, recreational hierarchy,
geographical distance, and attachment to a location are indispensable factors
of urban recreational gatherings in Seoul.

Figure 6:  (a) $JSD_{Entropy}\times JSD_{Hierarchy}$ varied by the model
parameters $p$ and $k$. Bottom annotated $p^{*}$ and $k^{*}$ are the best
parameters for each period. (b) $JSD_{Radius}$ varied by the model parameter
$\gamma$. We change $\gamma$ while keeping the best $p$ and $k$ from Fig. 6a.

### An external shock influences recreational gatherings

We explore the influence of the COVID-19 outbreak on individual movements for
recreational activities by comparing the best model result for each period. We
show the $JSD_{Entropy}$ and $JSD_{Hierarchy}$ varied by model parameters $p$
and $k$ in Fig. 6a and the $JSD_{Radius}$ varied by model parameter $\gamma$
in Fig. 6b. For both periods, the overall fitness landscape does not change,
while the optimal point does. In response to the COVID-19 pandemic, the
likelihood of finding places $p$ decreases ($p_{pre}=0.820>p_{post}=0.800$),
indicating people prefer to stay in familiar places. In addition, the tendency
to explore a high-level place decreases ($k_{pre}=2.075>k_{post}=2.025$), and
people become reluctant to travel far from their recreational homes
($\gamma_{pre}=1.325<\gamma_{post}=1.375$). To check how much difference the
parameter changes make, we simulated the model with the optimal parameters
from the pre-COVID-19 period ($p=0.820$ and $k=2.075$) for the post-COVID-19
period with an assumption that user behaviors do not change. We checked that
$JSD_{Entropy}\times JSD_{Hierarchy}$ of this model increases by 7% compared
to our optimal model. Similarly, if we simulate the model with the previous
optimal $\gamma$ from the pre-COVID-19 period, $JSD_{Radius}$ increases by
16%.

The differences in the estimated parameters between the two periods are not
subtle, and the model’s goodness of fit is sensitive to the parameter changes.
In Fig. 6a, if $p$ increases or decreases by 0.02 from the estimated optimal
point $p^{*}$, $JSD_{Entropy}\times JSD_{Hierarchy}$ increases by a factor of
1.058 and 1.080, respectively. If $k$ increases or decreases by 0.025 from the
estimated optimal point $k^{*}$, $JSD_{Entropy}\times JSD_{Hierarchy}$
increases by a factor of 1.119 and 1.057, respectively. Similarly, if $\gamma$
increases or decreases by 0.025 from the estimated optimal point $\gamma^{*}$,
$JSD_{Radius}$ increases by a factor of 1.022, 1.024, respectively. Also, we
would like to note that the goodness of fit’s standard deviation for ten
repetitions is an order of magnitude smaller than the average value of
goodness of fit, implying our simulation results are robust to random errors.

Intuitively, before the pandemic, people prefer to visit popular places as
they have fewer restrictions on geographic distance. However, during the
pandemic, people chose familiar (high $p$), relatively less popular (low $k$),
and closer places (large $\gamma$) from their recreational homes. Low $k$
would reflect the behaviors of avoiding dense areas to prevent exposure to
COVID-19, and large $\gamma$ would be associated with the tendency not to take
public transportation. After the outbreak, the public transportation system
usage in Seoul declined sharply: $-26.5$% for the bus, $-27.5$% for the subway
in 2020 compared to the previous year, according to a report from the Korean
Ministry of Land, Infrastructure, and Transport [36]. Furthermore, these
changes imply the effect of COVID-19 on urban inequality. As people avert
crowded places but choose less popular places, the concentration of activities
at the highest level is dissolved. However, the number of visitations at low
hierarchy levels ($l\leq 6$) also decreases (Supplementary Fig. S9) because
the probability of exploring places $p^{i}$ quickly converges to zero by
iterations.

## Discussion

In this paper, we quantify the characteristics of recreational visitations
with three factors: recreational hierarchy, geographical distance, and
attachment to a location. Leveraging our findings, we develop a model that
successfully reconstructs and explains empirical patterns found in Seoul,
Republic of Korea. We show that the COVID-19 pandemic led people to be less
likely to visit different levels in the hierarchy. They prefer familiar, less
popular, and closer locations. Furthermore, we suggest a possible mechanism to
explain the worsening inequality with the model parameters $p$ and $k$
simultaneously.

Our study has several limitations. First, agents start from the empty
reservation history and find a place by the model mechanisms. In reality, each
individual could have past reservation records and find a place depending on
the given history. Second, we use the geographic distance between cells, while
the urban transportation systems can distort the distance. Third, our data are
strongly correlated with mobility inflows. Therefore, changes in motel booking
behaviors would explain changes in recreational activities. Considering
diverse layers and their interactions can deeply enrich the understanding of
urban activity. Fourth, our findings on behavioral changes could be the
combination of voluntary willingness to avoid physical contact and public
health guidelines such as social distancing policies, operating hours
restrictions, and maximum occupancy restrictions. People might gather in less
popular places to avoid waiting because many facilities could handle fewer
people than before due to capacity limits. Additionally, they might prefer to
gather in closer locations to return home without spending much time on public
transportation. It would be interesting to decompose the effects of these
factors with high-resolution data and advanced models. Fifth, our findings on
the behavioral changes would be mainly led by low- and middle-income
populations, the primary users of the platform we studied.

Despite these limitations, our study enhances the understanding of urban human
activities and would help design effective public health policies considering
individual movements around home areas. Practically, our model can be a
simulation tool to prepare for unexpected future events that may affect human
behaviors. With our model, academic and industry researchers can also tackle
inequality issues stemming from behavioral changes across the urban hierarchy.

## Materials and Methods

### Accommodation reservation data

We sourced an accommodation reservation data set from Goodchoice Company LTD,
a Korean platform that occupied 29% of the online market share in 2020
(Wiseapp Report, https://www.dailypop.kr/news/articleView.html?idxno=51946, a
news article written in Korean). The data contain anonymized customer-level
reservation histories, spanning the period from January 2019 to November 2020,
and geographic locations of 1,038 unique motels in Seoul, Republic of Korea.
No demographic information is available.

### Seoul mobility data

Seoul mobility data was downloaded from the Seoul Open Data Plaza
(https://data.seoul.go.kr/dataVisual/seoul/seoulLivingMigration.do, Accessed
on December 3, 2021). The data contains mobility flows between administrative
divisions (425 divisions in total) decomposed by gender, age, time of
departure, and time of arrival, by aggregating the mobile phone signals from
transceiver stations in Seoul. Also, for each individual, the data provide an
estimated daytime residence (denoted as “W”, mostly workplace), nighttime
residence (denoted as “H”, mostly home), and other classes (denoted as “E”).
With the classifications above, we can infer the context of urban mobility.
For instance, a movement from a workplace to another area to enjoy
recreational gathering is classified as the “WE” type. We use the mobility
data spanning the period from January 2020 to October 2020 and focus on the
mobility types with “WE”, “HE”, and “EE” to track non-routine mobility
patterns for recreational gatherings.

### Hotspot entropy and the radius of recreational activities

Based on recreational hierarchy, we characterize location trajectories with
two proposed measures: hotspot entropy and radius of recreational activities.
First, the hotspot entropy is the Shannon entropy of hotspot levels in a
trajectory. It is defined as

$h=-\sum^{L}_{i=1}p_{i}\log{p_{i}},$ (2)

where $L$ is the total number of hotspot levels ($L=10$) and $p_{i}$ is the
frequency of a hotspot level $i$ in the trajectory. For example in Fig. 4a,
$p$ is $[\frac{2}{3},\frac{1}{6},\frac{1}{6},0,0,0,0,0,0,0]$ and hotspot
entropy $h$ is
$-\frac{2}{3}\log{\frac{2}{3}}-\frac{1}{6}\log{\frac{1}{6}}-\frac{1}{6}\log{\frac{1}{6}}=0.867$.

Second, to quantify how far the places are in a trajectory, we define the
radius of recreational activities. It is the variance of geographic distances
from the most frequent cell (home cell) in a trajectory (similar to the radius
of gyration) and is calculated as

$r=\sqrt{\frac{\sum_{i=1}^{N}d_{i,home}^{2}}{N}},$ (3)

where $N$ is the length of a trajectory, $d$ is the Haversine distance between
the centers of the two cells, and $home$ is the recreational home cell that is
the most frequent cell in the trajectory. If there are multiple most frequent
cells, we randomly pick one as the home cell.

## Declarations

### Availability of data and materials

Due to privacy concerns, the accommodation reservation data we used cannot be
shared publicly. The Seoul mobility data is publicly available.

### Competing interests

The authors have no competing interests.

### Ethical approval statements

This article does not contain any studies with human participants performed by
any of the authors.

### Informed consent statements

This article does not contain any studies with human participants performed by
any of the authors.

### Funding

This work was supported by the National Research Foundation of Korea (NRF)
with grant number 2021R1F1A106303011. J.Y. was supported by the National
Science Foundation with grant number 2133863.

### Author’s contribution

All authors contributed to the work presented in this paper. J.Y. was involved
in conceptualization, analysis, and writing, W.-S.J. and H.K. contributed to
conceptualization and writing. All authors discussed the results and commented
on the manuscript at all stages.

### Acknowledgements

We thank Goodchoice Company LTD. for making the accommodation reservation data
available for this research. We thank I. Hong, O.-H. Kwon, D. Lee, and Y.-Y.
Ahn for their helpful discussions.

## References

  * [1] Vincent Verbavatz and Marc Barthélemy  “The growth equation of cities”  In _Nature_ 587.7834 Nature Publishing Group, 2020, pp. 397–401
  * [2] Luı́s MA Bettencourt and Daniel Zünd  “Demography and the emergence of universal patterns in urban systems”  In _Nature Communications_ 11.1 Nature Publishing Group, 2020, pp. 1–9
  * [3] Cornelia Storz, Federico Riboldazzi and Moritz John  “Mobility and innovation: A cross-country comparison in the video games industry”  In _Research Policy_ 44.1 Elsevier, 2015, pp. 121–137
  * [4] Jaehyuk Park et al.  “Global labor flow network reveals the hierarchical organization and dynamics of geo-industrial clusters”  In _Nature Communications_ 10.1 Nature Publishing Group, 2019, pp. 1–10
  * [5] Serina Chang et al.  “Mobility network models of COVID-19 explain inequities and inform reopening”  In _Nature_ 589.7840 Nature Publishing Group, 2021, pp. 82–87
  * [6] Marc Barthélemy  “The structure and dynamics of cities”  Cambridge University Press, 2016
  * [7] Michael Batty  “The new science of cities”  MIT press, 2013
  * [8] Wei Pan et al.  “Urban characteristics attributable to density-driven tie formation”  In _Nature Communications_ 4.1 Nature Publishing Group, 2013, pp. 1–7
  * [9] Camille Roth, Soong Moon Kang, Michael Batty and Marc Barthélemy  “Structure of urban movements: Polycentric activity and entangled hierarchical flows”  In _PLoS One_ 6.1 Public Library of Science, 2011, pp. e15923
  * [10] Aleix Bassolas et al.  “Hierarchical organization of urban mobility and its connection with city livability”  In _Nature Communications_ 10.1 Nature Publishing Group, 2019, pp. 1–10
  * [11] Thomas Louail et al.  “Uncovering the spatial structure of mobility networks”  In _Nature Communications_ 6.1 Nature Publishing Group, 2015, pp. 1–8
  * [12] Faheem Ahmed, Na’eem Ahmed, Christopher Pissarides and Joseph Stiglitz  “Why inequality could spread COVID-19”  In _The Lancet Public Health_ 5.5 Elsevier, 2020, pp. e240
  * [13] Qinggang Yu et al.  “Racial residential segregation and economic disparity jointly exacerbate COVID-19 fatality in large American cities”  In _Annals of the New York Academy of Sciences_ Wiley-Blackwell, 2021
  * [14] Aaron Van Dorn, Rebecca E Cooney and Miriam L Sabin  “COVID-19 exacerbating inequalities in the US”  In _Lancet_ 395.10232 Elsevier, 2020, pp. 1243
  * [15] Réka Albert, Hawoong Jeong and Albert-László Barabási  “Error and attack tolerance of complex networks”  In _Nature_ 406.6794 Nature Publishing Group, 2000, pp. 378–382
  * [16] Romualdo Pastor-Satorras and Alessandro Vespignani  “Epidemics and immunization in scale-free networks”  In _Handbook of Graphs and Networks_ Wiley Online Library, 2003
  * [17] Cho Ryok Kang et al.  “Coronavirus disease exposure and spread from nightclubs, South Korea”  In _Emerging Infectious Diseases_ 26.10 Centers for Disease ControlPrevention, 2020, pp. 2499
  * [18] Peter Gould and Rodrick Wallace  “Spatial structures and scientific paradoxes in the AIDS pandemic”  In _Geografiska Annaler: Series B, Human Geography_ 76.2 Taylor & Francis, 1994, pp. 105–116
  * [19] Rajneesh Narula  “Policy opportunities and challenges from the COVID-19 pandemic for economies with large informal sectors”  In _Journal of International Business Policy_ 3.3 Springer, 2020, pp. 302–310
  * [20] Chaoming Song, Tal Koren, Pu Wang and Albert-László Barabási  “Modelling the scaling properties of human mobility”  In _Nature Physics_ 6.10 Nature Publishing Group, 2010, pp. 818–823
  * [21] Markus Schläpfer et al.  “The universal visitation law of human mobility”  In _Nature_ 593.7860 Nature Publishing Group, 2021, pp. 522–527
  * [22] Michael Batty et al.  “London in lockdown: mobility in the pandemic city”  In _COVID-19 Pandemic, Geospatial Information, and Community Resilience_ CRC Press, 2021, pp. 229–244
  * [23] Pierre Nouvellet et al.  “Reduction in mobility and COVID-19 transmission”  In _Nature Communications_ 12.1 Nature Publishing Group, 2021, pp. 1–9
  * [24] Esteban Moro, Dan Calacci, Xiaowen Dong and Alex Pentland  “Mobility patterns are associated with experienced income segregation in large US cities”  In _Nature Communications_ 12.1 Nature Publishing Group, 2021, pp. 1–10
  * [25] “Domestic & Global Accommodation Platform Analysis” Accessed: 2022-08-01, https://market.dighty.com/trendreport/?q=YToxOntzOjEyOiJrZXl3b3JkX3R5cGUiO3M6MzoiYWxsIjt9&bmode=view&idx=8435265&t=board
  * [26] Conrad Lashley  “The Routledge handbook of hospitality studies”  Taylor & Francis, 2016
  * [27] Hyojin Kim and Byung-Gook Kim  “A qualitative approach to automated motels: a rising issue in South Korea”  In _International Journal of Contemporary Hospitality Management_ Emerald Publishing Limited, 2018
  * [28] Jeongyeon Hu  “Trend of using motels for the 2030 generation”  In _The JoongAng_ , 2016  URL: https://www.joongang.co.kr/article/19826193#home
  * [29] Eric Veach et al.  “S2 geometry”  In _S2Geometry_ , 2017  URL: https://s2geometry.io/
  * [30] Zheng Wu et al.  “Travel time estimation using spatio-temporal index based on Cassandra”  In _ISPRS Annals of Photogrammetry, Remote Sensing & Spatial Information Sciences_ 4.4, 2018
  * [31] Michèle Belot et al.  “Unequal consequences of Covid 19: representative evidence from six countries”  In _Review of Economics of the Household_ 19.3 Springer, 2021, pp. 769–783
  * [32] Qi Zhang et al.  “The unequal effect of the COVID-19 pandemic on the labour market and income inequality in China: A multisectoral CGE model analysis coupled with a micro-simulation approach”  In _International Journal of Environmental Research and Public Health_ 19.3 MDPI, 2022, pp. 1320
  * [33] Constantino Hevia, Manuel Macera and Pablo Andrés Neumeyer  “Covid-19 in unequal societies”  In _Journal of Economic Dynamics and Control_ Elsevier, 2022, pp. 104328
  * [34] John T Bowen and Shiang-Lih Chen  “The relationship between customer loyalty and customer satisfaction”  In _International Journal of Contemporary Hospitality Management_ MCB UP Ltd, 2001
  * [35] Jay Kandampully and Dwi Suhartanto  “Customer loyalty in the hotel industry: The role of customer satisfaction and image”  In _International Journal of Contemporary Hospitality Management_ MCB UP Ltd, 2000
  * [36] “Changes in public transportation usage due to COVID-19” Accessed: 2022-08-05, http://www.molit.go.kr/USR/NEWS/m_71/dtl.jsp?lcmspage=1&id=95085321

Supplementary Information : COVID-19 confines recreational gatherings in Seoul
to familiar, less crowded, and neighboring urban areas

Figure S1: The flow matrix $T^{data}$ for the post-COVID-19 period. As same as
Fig. 1 in the main report, we exclude the trips within the same cell. There
are no transition records for the gray cells. Figure S2: The flow matrix of
the null model, $T^{null}$, for the (a) pre-COVID-19 period and the (b) post-
COVID-19 period. Figure S3:  The ratio matrix $T^{data}/T^{null}$ in the (a)
pre-COVID-19 period and the (b) post-COVID-19 period. In each cell, upper
annotated number is $T_{data}/T_{null}$ and below annotated number is the
total number of transitions in the data. Figure S4: Transition outflows and
inflows in (a) the pre-COVID-19 period and (b) the post-COVID-19 period. The
infow is defined as $f^{in}_{i}=\sum_{k=1}^{L}T_{ki}$ and outflow is defined
as $f^{out}_{i}=\sum_{k=1}^{L}T_{ik}$ where $L$ is the total number of hotspot
level. $f^{in}$ and $f^{out}$ is almost symmetry for both periods
($R^{2}>0.999$).  Figure S5: Urban hierarchy leads people to move farther than
expected. We collect the trajectories of which recreational home cells are
near the Gangnam area (red cross, hotspot level 1), which is one of popular
regions in Seoul. Relative visit frequency decays with the geographic distance
from the home cell, but the geographic distance cannot explain the pattern
near regional hotspots.  Figure S6: The best model result for the post-
COIVD-19 period (a) The flow matrix $T^{model}$ of the best model,
$d_{T}=||T^{model}-T^{data}||_{F}$ is 0.02. (b) The hotspot entropy
distribution $p_{h}$ (left) and the distribution of the radius of recreational
activities $p_{r}$ (right). Blue lines are the empirical distributions, and
orange lines are the simulation results. (c) Home ratios by the length of
trajectory. Figure S7: Cell-level comparisons between actual and simulated
reservation counts for the (a) pre-COVID-19 period and the (b) post-COVID-19
period. Overall, our model simulates the cell-level reservation count well for
both periods. For the data privacy concern, we normalize the reservation count
with the maximum reservation count of the actual data set.  Figure S8: Variant
model results for the pre-COIVD-19 period. (a)-(c) The model without urban
hierarchy. (d)-(f) The model without geography. (g)-(h) The model without
attachment to a location. The figures in the first column are the flow
matrices from each model $T^{model}$. The figures in the second column are the
hotspot entropy distributions $p_{h}^{model}$ (left) and the radius of
recreational activities distributions $p_{r}^{model}$ (right) from each model.
Blue lines are the empirical distributions, and orange lines are the
simulation results. Lastly, the figures in the third column are home ratios by
sequence length. The overall pattern is similar to the pattern in the post-
COVID-19 period.  Figure S9: The simulated reservation count distribution by
the hotspot level $p_{\ell}$. The model successfully explains the
decentralization of human urban activities and the worsening inequality of the
urban areas in Seoul.


# MUDES: Multilingual Detection of Offensive Spans

Tharindu Ranasinghe
University of Wolverhampton
Wolverhampton, UK
<EMAIL_ADDRESS>
Marcos Zampieri
Rochester Institute of Technology
Rochester, NY, USA
<EMAIL_ADDRESS>

###### Abstract

The interest in offensive content identification in social media has grown
substantially in recent years. Previous work has dealt mostly with post level
annotations. However, identifying offensive spans is useful in many ways. To
help coping with this important challenge, we present MUDES, a multilingual
system to detect offensive spans in texts. MUDES features pre-trained models,
a Python API for developers, and a user-friendly web-based interface. A
detailed description of MUDES’ components is presented in this paper.

##  1 Introduction

Offensive and impolite language are widespread in social media posts
motivating a number of studies on automatically detecting the various types of
offensive content (e.g. aggression Kumar et al. (2018, 2020), cyber-bullying
Rosa et al. (2019), hate speech Malmasi and Zampieri (2018), etc.). Most
previous work has focused on classifying full instances (e.g. posts, comments,
documents) (e.g. offensive vs. not offensive) while the identification of the
particular spans that make a text offensive has been mostly neglected.
††WARNING: This paper contains text excerpts and words that are offensive in
nature.

Identifying offensive spans in texts is the goal of the SemEval-2021 Task 5:
Toxic Spans Detection Pavlopoulos et al. (2021). The organisers of this task
argue that highlighting toxic spans in texts helps assisting human moderators
(e.g. news portals moderators) and that this can be a first step in semi-
automated content moderation. Finally, as we demonstrate in this paper,
addressing offensive spans in texts will make the output of offensive language
detection systems more interpretable thus allowing a more detailed linguistics
analysis of predictions and improving the quality of such systems.

With these important points in mind, we developed MUDES: Multilingual
Detection of Offensive Spans. MUDES is a multilingual framework for offensive
language detection focusing on text spans. The main contributions of this
paper are the following:

  1. 1.

We introduce MUDES, a new Python-based framework to identify offensive spans
with state-of-the-art performance.

  2. 2.

We release four pre-trained offensive language identification models: en-base,
en-large models which are capable of identifying offensive spans in English
text. We also release Multilingual-base and Multilingual-large models which
are able to recognise offensive spans in languages other than English.

  3. 3.

We release a Python Application Programming Interface (API) for developers who
are interested in training more models and performing inference in the code
level.

  4. 4.

For general users and non-programmers, we release a user-friendly web-based
User Interface (UI), which provides the functionality to input a text in
multiple languages and to identify the offensive span in that text.

##  2 Related Work

Post |  Offensive Spans
---|---
Stupid hatcheries have completely fucked everything |  [0, 1, 2, 3, 4, 5, 34, 35, 36, 37, 38, 39]
Victimitis: You are such an asshole. |  [28, 29, 30, 31, 32, 33, 34]
So is his mother. They are silver spoon parasites. |  []
You’re just silly. |  [12, 13, 14, 15, 16]
Table 1: Four comments from the dataset, with their annotations. The offensive
words are displayed in red and the spans are indicated by the character
position in the instance.

Early approaches to offensive language identification relied on traditional
machine learning classifiers Dadvar et al. (2013) and later on neural networks
combined with word embeddings Majumder et al. (2018); Hettiarachchi and
Ranasinghe (2019). Transformer-based models like BERT Devlin et al. (2019) and
ELMO Peters et al. (2018) have been recently applied to offensive language
detection achieving competitive scores Wang et al. (2020); Ranasinghe and
Hettiarachchi (2020) in recent SemEval competitions such as HatEval Basile et
al. (2019) OffensEval Zampieri et al. (2020).

In terms of languages, the majority of studies on this topic deal with English
Malmasi and Zampieri (2017); Yao et al. (2019); Ridenhour et al. (2020);
Rosenthal et al. (2020) due to the the wide availability of language resources
such as corpora and pre-trained models. In recent years, several studies have
been published on identifying offensive content in other languages such as
Arabic Mubarak et al. (2020), Dutch Tulkens et al. (2016), French Chiril et
al. (2019), Greek Pitenis et al. (2020), Italian Poletto et al. (2017),
Portuguese Fortuna et al. (2019), and Turkish Çöltekin (2020). Most of these
studies have created new datasets and resources for these languages opening
avenues for multilingual models as those presented in Ranasinghe and Zampieri
(2020). However, all studies presented in this section focused on classifying
full texts, as discussed in the Introduction. MUDES’ objective is to fill this
gap and perform span level offensive language identification.

##  3 Data

The main dataset used to train the machine learning models presented in this
paper is the dataset released within the scope of the aforementioned
SemEval-2021 Task 5: Toxic Spans Detection for English. The dataset contains
posts (comments) from the publicly available Civil Comments dataset Borkan et
al. (2019). The organisers have randomly selected 10,000 posts, out of a total
of 1,2 million posts in the original dataset. The offensive spans have been
annotated using a crowd-annotation platform, employing three crowd-raters per
post. By the time of writing this paper, only the trial set and the training
set have been released and the gold labels for the test set have not yet been
released. Therefore, training of the machine learning models presented in
MUDES was done on the training set which we refer to as TSDTrain and the
evaluation was conducted on the trial set which we refer to as TSDTrial set.
In Table 1 we show four randomly selected examples from the TSDTrain dataset
with their annotations.

The general idea is to learn a robust model from this dataset and generalize
to other English datasets which do not contain span annotation. Another goal
is to investigate the feasibility of annotation projection to other languages.

Figure 1: Model Architecture. Architecture consists of two parts. Part A is
the language modelling and Part B is the token classification.

#### Other Datasets

In order to evaluate our framework in different domains and languages we used
three publicly available offensive language identification datasets. As an
off-domain English dataset, we choose the Offensive Language Identification
Dataset (OLID) Zampieri et al. (2019a), used in OffensEval 2019 (SemEval-2019
Task 6) Zampieri et al. (2019b), containing over 14,000 posts from Twitter. To
evaluate our framework in different languages, we selected a Danish
Sigurbergsson and Derczynski (2020) and a Greek Pitenis et al. (2020) dataset.
These two datasets have been provided by the organisers of OffensEval 2020
(SemEval-2020 Task 12) Zampieri et al. (2020) and were annotated using OLID’s
annotation guidelines. The Danish dataset contains over 3,000 posts from
Facebook and Reddit while the Greek dataset contains over 10,000 Twitter
posts, allowing us to evaluate our dataset in an off-domain, multilingual
setting. As these three datasets have been annotated at the instance level, we
followed an evaluation process explained in Section 5.

##  4 Methodology

The main motivation behind this methodology is the recent success that
transformer models had in various NLP tasks Devlin et al. (2019) including
offensive language identification Ranasinghe and Zampieri (2020); Ranasinghe
et al. (2019); Wiedemann et al. (2020). Most of these transformer-based
approaches take the final hidden state of the first token ([CLS]) from the
transformer as the representation of the whole sequence and a simple softmax
classifier is added to the top of the transformer model to predict the
probability of a class label Sun et al. (2019). However, as previously
mentioned, these models classify whole comments or documents and do not
identify the spans that make a text offensive. Since the objective of this
task is to identify offensive spans rather than classifying the whole comment,
we followed a different architecture.

As shown in Figure 1, the complete architecture contains two main parts;
Language Modeling (LM) and Token Classification (TC). In the LM part, we used
a pre-trained transformer model and retrained it on the TSDTrain dataset using
Masked Language Modeling (MLM). In the second part of the architecture, we
used the saved model from the LM part and we perform a token classification.
We added a token level classifier on top of the transformer model as shown in
Figure 1. The token-level classifier is a linear layer that takes the last
hidden state of the sequence as the input and produce a label for each token
as the output. In this case each token can have two labels; offensive and not
offensive. We have listed the training configurations in the Appendix.

We experimented with several popular transformer models like BERT Devlin et
al. (2019), XLNET Yang et al. (2019), ALBERT Lan et al. (2020), RoBERTa Liu et
al. (2019) etc. From the pre-trained transformer models we selected, we
grouped the large models and base models separately in order to release two
English models. A large model; en-large which is more accurate, but has a low
efficiency regarding space and time. The base model; en-base is efficient, but
has a comparatively low accuracy than the en-large model. All the experiments
have been executed for five times with different random seeds and we took the
mode of the classes predicted by each random seed as the final result
Hettiarachchi and Ranasinghe (2020).

#### Multilingual models

\- The motivation behind the use of multilingual models comes from recent
works Ranasinghe and Zampieri (2020, 2021); Ranasinghe et al. (2020) which
used transfer learning and cross-lingual embeddings. These studies show that
cross-lingual transformers like XLM-R Conneau et al. (2019) can be trained on
an English dataset and have the model weights saved to detect offensive
language in other languages outperforming monolingual models trained on the
target language dataset. We used a similar methodology but for the token
classification architecture instead. We used XLM-R cross-lingual transformer
model Conneau et al. (2019) as the Transformer in Figure 1 on TSDTrain and
carried out evaluations on the Danish and Greek datasets. We release two
multilingual models; multilingual-base based on XLM-R base model and
multilingual-large based on XLM-R large model.

##  5 Evaluation and Results

We followed two different evaluation methods. In Section 5.1 we present the
methods used to evaluate offensive spans on the TSDTrial set. In Section 5.2
we presented the methods used to evaluate the other three datasets which only
contained post level annotations.

###  5.1 Offensive Spans Evaluation

For the Toxic Spans Detection dataset, we followed the same evaluation
procedure of the SemEval Toxic Spans Detection competition. The organisers
have used F1 score mentioned in Da San Martino et al. (2019) to evaluate the
systems. Let system $A_{i}$ return a set $S^{t}_{A_{i}}$ of character offsets,
for parts of the post found to be toxic. Let $G_{t}$ be the character offsets
of the ground truth annotations of t. We compute the F1 score of system
$A_{i}$ with respect to the ground truth G for post t as mentioned in Equation
1, where $|$ ·$|$ denotes set cardinality.

$F_{1}^{t}\left(A_{i},G\right)=\frac{2\cdot P^{t}\left(A_{i},G\right)\cdot
R^{t}\left(A_{i},G\right)}{P^{t}\left(A_{i},G\right)+R^{t}\left(A_{i},G\right)}$
(1)

$P^{t}\left(A_{i},G\right)=\frac{\left|S_{A_{i}}^{t}\cap
S_{G}^{t}\right|}{\left|S_{A_{i}}^{t}\right|}$
$R^{t}\left(A_{i},G\right)=\frac{\left|S_{A_{i}}^{t}\cap
S_{G}^{t}\right|}{\left|S_{G}^{t}\right|}$

We present the results along with the baseline provided by the organisers in
Table 2. The baseline is implemented using a spaCy NER pipeline. The spaCy NER
system contains a word embedding strategy using sub word features and Bloom
embedding Serrà and Karatzoglou (2017), and a deep convolution neural network
with residual connections. Additionally, we compare our results to a lexicon-
based word match approach mentioned in Ranasinghe et al. (2021) where the
lexicon is based on profanity words from online
resources111https://www.cs.cmu.edu/~biglou/resources/bad-
words.txt,222https://github.com/RobertJGabriel/Google-profanity-words.

Model Name | Base Model | F1 score
---|---|---
en-large | roberta-large | 0.6886
en-base | xlnet-base-cased | 0.6734
multilingual-large | XLM-R-large | 0.6338
multilingual-base | XLM-R-base | 0.6160
spaCy baseline | NA | 0.5976
Lexicon word match Ranasinghe et al. (2021) | NA | 0.3378

Table 2: Results ordered by F1 score for TSD Trial.

The results show that all MUDES’ models outperform the spaCy baseline and the
lexicon-based word match. From all of the large transformer models we
experimented roberta-large performed better than others. Therefore, we
released it as en-large model in MUDES. From the base models we experimented,
XLNet-base-cased model outperformed all the other base models so we released
it as en-base model. We also released two multilingual models; multilingual-
base and multilingual-large based on XLM-R-base and XLM-R-large respectively.
All the pre-trained MUDES’ models are available to download from HuggingFace
model hub 333MUDES’ models are available on https://huggingface.co/mudes Wolf
et al. (2020).

###  5.2 Off-Domain and Multilingual Evaluation

For the English off-domain and multilingual datasets we followed a different
evaluation process. We used a pre-trained MUDES’ model trained on TSDTrain to
predict the offensive spans for all texts in the test sets of two non-English
datasets (Danish, and Greek) and English off-domain dataset, OLID, which is
annotated at the document level. If a certain text contains at least one
offensive span we marked the whole text as offensive following the OLID
annotation guidelines described in Zampieri et al. (2019a). We compared our
results to the best systems submitted to OffensEval 2020 in terms of macro F1
reported by the task organisers Zampieri et al. (2020). We present the results
along with the majority class baseline for each dataset in Table 3. For
English off domain dataset (OLID) we only used the MUDES en models while for
Danish and Greek datasets we used the MUDES multilingual models.

Language | Model | M F1 |
---|---|---|---
| Pàmies et al. (2020) | 0.8119 |
Danish | multilingual-large | 0.7623 |
| multilingual-base | 0.7143 |
| Majority Baseline | 0.4668 |
| Wiedemann et al. (2020) | 0.9204 |
English | en-large | 0.9023 |
| en-base | 0.8892 |
| Majority Baseline | 0.4193 |
| Ahn et al. (2020) | 0.8522 |
Greek | multilingual-large | 0.8143 |
| multilingual-base | 0.7820 |
| Majority Baseline | 0.4202 |

Table 3: Results ordered by macro (M) F1 for Danish, English and Greek
datasets

Results show that despite the change of domain and the language, MUDES perform
well in all the datasets and compares favourably to the best systems
submitted. It should be noted that the best systems have been predominantly
trained on offensive languages identification task on post level while MUDES’
objective is different. Yet MUDES come closer to the best systems in all the
datasets.

From the results, it is clear that MUDES english models can perform in a
different domain like Twitter. Also the results show that MUDES multilingual
models are capable of identifying offensive spans in other languages too.
Since XLM-R supports 104 languages, this approach will benefit all those
languages without any training data at all.

##  6 System Demonstration

###  6.1 Application Programming Interface

MUDES is available as a Python package in the Python Package Index
(PyPI)444https://pypi.org/project/mudes/. The package is related to MUDES
GitHub repository555https://github.com/tharindudr/MUDES. Users can install it
easily with the following command after installing PyTorch Paszke et al.
(2019).

⬇

1: $ pip install mudes

The Python package contains the following functionalities.

(a) Example from Civil Comments Dataset

(b) Example from Tamil. (Pussy like this !!!!)

Figure 2: Examples in English and in a low-resource languages. The experiments
were conducted with en-large and the multilingual-large models respectively.

#### Get offensive spans with a pretrained model

The library provides the functionality to load a pretrained model and use it
to identify offensive spans. The following code segment downloads and loads
MUDES’ en-base model in a CPU only environment and identifies offensive spans
in the the text; "This is fucking crazy!!". If the users prefer a GPU, the
argument use_cuda should be set to True.

Listing 1: English Inference Example

⬇

1:from mudes.app.mudes_app

2: import MUDESApp

3:

4:sentence = "This is fucking crazy!!"

5:

6:app = MUDESApp("en-base",

7: use_cuda=False)

8:app.predict_toxic_spans(sentence)

#### Train a MUDES model

The library provides the functionality to train a MUDES model from scratch
using the code segment present next. It takes a Pandas dataframe in the format
of TSDTrain, formats it for the token classification task and train a MUDES
model from scratch. MUDES support popular transformer types as bert, xlnet,
roberta etc. as the MODEL_TYPE and name of the model as appear in Hugging Face
Wolf et al. (2020) model repository. 666https://huggingface.co/models

Listing 2: Training Example

⬇

1:from mudes.algo.mudes_model

2: import MUDESModel

3:from mudes.algo.preprocess

4: import read_datafile,

5: format_data

6:

7:train_df = format_data(train)

8:tags = train_df[’labels’]

9: .unique().tolist()

10:

11:model = MUDESModel(MODEL_TYPE,

12: MODEL_NAME, labels=tags)

13:model.train(train_df)

###  6.2 User Interface

We developed a prototype of the User Interface (UI) to demonstrate the
capabilities of the system. The UI is based on Streamlit777www.streamlit.io
which provides functionalities to easily develop dashboards for machine
learning projects. The code base for the UI is available in GitHub
888https://github.com/tharindudr/MUDES-UI.. This UI is hosted in a Linux
server. 999http://rgcl.wlv.ac.uk/mudes/ We also release a Docker container
image of the UI in Docker Hub101010Docker Hub is a hosted repository service
provided by Docker for finding and sharing container images. for those who are
interested in self hosting the UI. Docker enables developers to easily deploy
and run any application as a lightweight, portable, self-sufficient container,
which can run virtually anywhere. The released Docker container image follows
Continuous Integration/Continuous Deployment (CI/CD) from the GitHub
repository which allows sharing and deploying the code quickly and
efficiently.

Once Docker is installed, one can easily run our UI with this command.

⬇

1: $ docker run tharindudr/mudes

This command will automatically install all the required packages, download
and load the pre-trained models and open the system in the default browser. We
provide the following functionalities from the user interface.

#### Switch through pretrained models

\- The users can switch through the pre-trained models using the radio buttons
available in the left side of the UI under Available Models section. They can
select an option from en-base, en-large, multilingual-base and multilingual-
large. These models have been already downloaded from the HuggingFace model
hub and they are loaded in to the random-access memory of the hosting
computer.

#### Switch through available datasets

\- We have made the four datsets used in this paper available from the UI for
the users to experiment with Borkan et al. (2019); Zampieri et al. (2019a);
Pitenis et al. (2020); Sigurbergsson and Derczynski (2020). Once the user
selects a particular option, the system will automatically load the test set
of the selected dataset. Once it is loaded the user can iterate through the
dataset using the scrollbar. For each text the UI will display the offensive
spans in red.

#### Get offensive spans for a custom text

\- The users can also enter a custom text in the text box, hit ctrl+enter and
see the offensive spans available in the input text. Once processed through
the system, any offensive spans available in the text will be displayed in
red. Figure 2 shows several screenshots from the UI. It illustrates an example
on English for the texts taken from civil comments dataset Borkan et al.
(2019) conducted with en-large model. To show that MUDES framework works on
low resource language too, Figure 2 also displays an example from Tamil.

###  6.3 System Efficiency

The time taken to predict the offensive spans for a text will be critical in
an online system developed for real time use. Therefore, we evaluated the time
MUDES takes to predict the offensive spans in 100 texts for all the released
models in a CPU and GPU environment. The results show that large models take
around 3 seconds for a sentence in a CPU and take around 1 second for a
sentence in a GPU on average while the base models take approximately one
third of that time in both environments. From these results it is clear that
MUDES is capable of predicting toxic spans efficiently in any environment. The
full set of results are reported in the Appendix. We used a batch size of one,
in order to mimic the real world scenario. The full specifications of the CPU
and GPU environments are listed in the Appendix.

##  7 Conclusion

This paper introduced MUDES: Multilingual Detection of Offensive Spans. We
evaluated MUDES on the recently released SemEval-2021 Toxic Spans Detection
dataset. Our results show that MUDES outperforms the strong baselines of the
competition. Furthermore, we show that once MUDES is trained on English data
using state of the art cross-lingual transformer models, it is capable of
detecting offensive spans in other languages. With MUDES, we release a Python
library, four pre-trained models and an user interface. We show that MUDES is
efficient to use in real time scenarios even in a non GPU environment. In
future work, we would like to further evaluate MUDES on other datasets.
Finally, we would like to implement a flexible multitask architecture capable
of detecting offense at both span and post level.

## Acknowledgments

We would like to thank the SemEval-2021 Toxic Spans Detection shared task
organisers for making this interesting dataset available. We further thank the
anonymous reviewers for their insightful feedback.

## References

  * Ahn et al. (2020) Hwijeen Ahn, Jimin Sun, Chan Young Park, and Jungyun Seo. 2020.  NLPDove at SemEval-2020 task 12: Improving offensive language detection with cross-lingual transfer.  In _Proceedings of SemEval_.
  * Basile et al. (2019) Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019\.  Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter.  In _Proceedings of SemEval_.
  * Borkan et al. (2019) Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019\.  Nuanced metrics for measuring unintended bias with real data for text classification.  In _Proceedings of WWW_.
  * Çöltekin (2020) Çağrı Çöltekin. 2020.  A Corpus of Turkish Offensive Language on Social Media.  In _Proceedings of LREC_.
  * Chiril et al. (2019) Patricia Chiril, Farah Benamara Zitoune, Véronique Moriceau, Marlène Coulomb-Gully, and Abhishek Kumar. 2019.  Multilingual and multitarget hate speech detection in tweets.  In _Proceedings of TALN_.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of ACL_.
  * Da San Martino et al. (2019) Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019.  Fine-grained analysis of propaganda in news article.  In _Proceedings of EMNLP-IJCNLP_.
  * Dadvar et al. (2013) Maral Dadvar, Dolf Trieschnigg, Roeland Ordelman, and Franciska de Jong. 2013.  Improving Dyberbullying Detection with User Context.  In _Proceedings of ECIR_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of NAACL_.
  * Fortuna et al. (2019) Paula Fortuna, Joao Rocha da Silva, Leo Wanner, Sérgio Nunes, et al. 2019.  A Hierarchically-labeled Portuguese Hate Speech Dataset.  In _Proceedings of ALW_.
  * Hettiarachchi and Ranasinghe (2019) Hansi Hettiarachchi and Tharindu Ranasinghe. 2019.  Emoji powered capsule network to detect type and target of offensive posts in social media.  In _Proceedings of RANLP_.
  * Hettiarachchi and Ranasinghe (2020) Hansi Hettiarachchi and Tharindu Ranasinghe. 2020.  InfoMiner at WNUT-2020 task 2: Transformer-based covid-19 informative tweet extraction.  In _Proceedings of W-NUT_.
  * Kumar et al. (2018) Ritesh Kumar, Atul Kr Ojha, Shervin Malmasi, and Marcos Zampieri. 2018.  Benchmarking aggression identification in social media.  In _Proceedings of TRAC_.
  * Kumar et al. (2020) Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2020.  Evaluating Aggression Identification in Social Media.  In _Proceedings of TRAC_.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.  In _Proceedings of ICLR_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Majumder et al. (2018) Prasenjit Majumder, Thomas Mandl, et al. 2018.  Filtering Aggression from the Multilingual Social Media Feed.  In _Proceedings TRAC_.
  * Malmasi and Zampieri (2017) Shervin Malmasi and Marcos Zampieri. 2017.  Detecting Hate Speech in Social Media.  In _Proceedings of RANLP_.
  * Malmasi and Zampieri (2018) Shervin Malmasi and Marcos Zampieri. 2018.  Challenges in Discriminating Profanity from Hate Speech.  _Journal of Experimental & Theoretical Artificial Intelligence_, 30:1 – 16.
  * Mubarak et al. (2020) Hamdy Mubarak, Ammar Rashed, Kareem Darwish, Younes Samih, and Ahmed Abdelali. 2020\.  Arabic offensive language on twitter: Analysis and experiments.  _arXiv preprint arXiv:2004.02192_.
  * Pàmies et al. (2020) Marc Pàmies, Emily Öhman, Kaisla Kajava, and Jörg Tiedemann. 2020.  LT@Helsinki at SemEval-2020 task 12: Multilingual or language-specific BERT?  In _Proceedings of SemEval_.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In _Proceedings of NeurIPS_.
  * Pavlopoulos et al. (2021) John Pavlopoulos, Léo Laugier, Jeffrey Sorensen, and Ion Androutsopoulos. 2021\.  Semeval-2021 task 5: Toxic spans detection.  In _Proceedings of SemEval_.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep Contextualized Word Representations.  In _Proceedings of NAACL_.
  * Pitenis et al. (2020) Zeses Pitenis, Marcos Zampieri, and Tharindu Ranasinghe. 2020.  Offensive Language Identification in Greek.  In _Proceedings of LREC_.
  * Poletto et al. (2017) Fabio Poletto, Marco Stranisci, Manuela Sanguinetti, Viviana Patti, and Cristina Bosco. 2017.  Hate Speech Annotation: Analysis of an Italian Twitter Corpus.  In _Proceedings of CLiC-it_.
  * Ranasinghe et al. (2020) Tharindu Ranasinghe, Sarthak Gupte, Marcos Zampieri, and Ifeoma Nwogu. 2020.  WLV-RIT at HASOC-Dravidian-CodeMix-FIRE2020: Offensive Language Identification in Code-switched YouTube Comments.  In _Proceedings of FIRE_.
  * Ranasinghe and Hettiarachchi (2020) Tharindu Ranasinghe and Hansi Hettiarachchi. 2020.  BRUMS at SemEval-2020 task 12: Transformer based multilingual offensive language identification in social media.  In _Proceedings of SemEval_.
  * Ranasinghe et al. (2021) Tharindu Ranasinghe, Diptanu Sarkar, Marcos Zampieri, and Alex Ororbia. 2021.  WLV-RIT at SemEval-2021 Task 5: A Neural Transformer Framework for Detecting Toxic Spans.  In _Proceedings of SemEval_.
  * Ranasinghe and Zampieri (2020) Tharindu Ranasinghe and Marcos Zampieri. 2020.  Multilingual Offensive Language Identification with Cross-lingual Embeddings.  In _Proceedings of EMNLP_.
  * Ranasinghe and Zampieri (2021) Tharindu Ranasinghe and Marcos Zampieri. 2021.  Multilingual Offensive Language Identification for Low-resource Languages.  _ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP)_.
  * Ranasinghe et al. (2019) Tharindu Ranasinghe, Marcos Zampieri, and Hansi Hettiarachchi. 2019.  BRUMS at HASOC 2019: Deep Learning Models for Multilingual Hate Speech and Offensive Language Identification.  In _Proceedings of FIRE_.
  * Ridenhour et al. (2020) Michael Ridenhour, Arunkumar Bagavathi, Elaheh Raisi, and Siddharth Krishnan. 2020\.  Detecting Online Hate Speech: Approaches Using Weak Supervision and Network Embedding Models.  _arXiv preprint arXiv:2007.12724_.
  * Rosa et al. (2019) Hugo Rosa, N Pereira, Ricardo Ribeiro, Paula Costa Ferreira, Joao Paulo Carvalho, S Oliveira, Luísa Coheur, Paula Paulino, AM Veiga Simão, and Isabel Trancoso. 2019.  Automatic cyberbullying detection: A systematic review.  _Computers in Human Behavior_ , 93:333–345.
  * Rosenthal et al. (2020) Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, and Preslav Nakov. 2020.  A Large-Scale Weakly Supervised Dataset for Offensive Language Identification.  In _arXiv preprint arXiv:2004.14454_.
  * Serrà and Karatzoglou (2017) Joan Serrà and Alexandros Karatzoglou. 2017.  Getting deep recommenders fit: Bloom embeddings for sparse binary input/output networks.  In _Proceedings of RecSys_.
  * Sigurbergsson and Derczynski (2020) Gudbjartur Ingi Sigurbergsson and Leon Derczynski. 2020.  Offensive Language and Hate Speech Detection for Danish.  In _Proceedings of LREC_.
  * Sun et al. (2019) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019.  How to fine-tune bert for text classification?  In _Proceedings of CCL_.
  * Tulkens et al. (2016) Stéphan Tulkens, Lisa Hilte, Elise Lodewyckx, Ben Verhoeven, and Walter Daelemans. 2016.  A Dictionary-based Approach to Racism Detection in Dutch Social Media.  In _Proceedings of TA-COS_.
  * Wang et al. (2020) Shuohuan Wang, Jiaxiang Liu, Xuan Ouyang, and Yu Sun. 2020.  Galileo at SemEval-2020 task 12: Multi-lingual learning for offensive language identification using pre-trained language models.  In _Proceedings of SemEval_.
  * Wiedemann et al. (2020) Gregor Wiedemann, Seid Muhie Yimam, and Chris Biemann. 2020.  UHH-LT at SemEval-2020 task 12: Fine-tuning of pre-trained transformer networks for offensive language detection.  In _Proceedings of SemEval_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of EMNLP_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  XLNet: Generalized Autoregressive Pretraining for Language Understanding.  In _Proceedings of NeurIPS_.
  * Yao et al. (2019) Mengfan Yao, Charalampos Chelmis, and Daphney-Stavroula Zois. 2019.  Cyberbullying Ends Here: Towards Robust Detection of Cyberbullying in Social Media.  In _Proceedings of WWW_.
  * Zampieri et al. (2019a) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019a.  Predicting the type and target of offensive posts in social media.  In _Proceedings of NAACL_.
  * Zampieri et al. (2019b) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019b.  SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval).  In _Proceedings of SemEval_.
  * Zampieri et al. (2020) Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020.  SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020).  In _Proceedings of SemEval_.

## Appendix

  1. i

Training Configurations We used an Nvidia Tesla K80 GPU to train the models.
We divided the dataset into a training set and a validation set using 0.8:0.2
split. We fine tuned the learning rate and number of epochs of the model
manually to obtain the best results for the validation set. We obtained
$1e^{-}5$ as the best value for learning rate and 3 as the best value for
number of epochs for all the languages. We performed early stopping if the
validation loss did not improve over 10 evaluation steps. Training large
models took around 30 minutes while training base models took around 10
minutes. In addition to the learning rate and number of epochs we used the
parameter values mentioned in Table 4. We kept these values as constants.

Parameter | Value
---|---
adam epsilon | 1e-8
warmup ratio | 0.1
warmup steps | 0
max grad norm | 1.0
max seq. length | 140
gradient accumulation steps | 1

Table 4: Parameter Specifications.

  2. ii

Hardware Specifications

In Table 5 and in Table 6 we mention the specifications of the GPU and CPU we
used for the experiments of the paper. For the training of the MUDES models,
we mainly used the GPU. For the efficiency experiments mentioned in Section
6.3 we used both GPU and CPU environments.

Parameter | Value
---|---
GPU | Nvidia K80
GPU Memory | 12GB
GPU Memory Clock | 0.82GHz
Performance | 4.1 TFLOPS
No. CPU Cores | 2
RAM | 12GB

Table 5: GPU Specifications.

Parameter | Value
---|---
CPU Model Name | Intel(R) Xeon(R)
CPU Freq. | 2.30GHz
No. CPU Cores | 2
CPU Family | Haswell
RAM | 12GB

Table 6: CPU Specifications.

  3. iii

Run time

As expected base models perform efficiently than the large models in both
environments. Large models take around 3 seconds for a sentence in a CPU and
take around 1 second for a sentence in a GPU while the base models take
approximately one third of that time in both environments. From these results
it is clear that MUDES is capable of predicting toxic spans efficiently in any
environment.

Model | GPU Time | CPU Time |
---|---|---|---
en-base | 35.51 | 100.81 |
en-large | 100.36 | 315.72 |
multilingual-base | 36.23 | 115.98 |
multilingual-large | 120.54 | 335.65 |

Table 7: Time taken to do predictions on 100 sentences in seconds.

## Ethics Statement

MUDES is essentially a web-based visualization tool with predictive models
trained on multiple publicly available datasets. The authors of this paper
used datasets referenced in this paper which were previously collected and
annotated. No new data collection has been carried out as part of this work.
We have not collected or processed writers’/users’ information nor have we
carried out any form of user profiling protecting users’ privacy and identity.

We understand that every dataset is subject to intrinsic bias and that
computational models will inevitably learn biased information from any
dataset. We believe that MUDES will help coping with biases in datasets and
models as it features: (1) a freely available Python library that other
researchers can use to train new models on other datasets; (2) a web-based
visualizing tool that can help efforts in reducing biases in offensive
language identification as they can be used to process and visualize
potentially offensive spans new data. Finally, unlike models trained at the
post level, the projected annotation of spans allows users to understand which
part of the instance is considered offensive by the models.