# MAGPIE: Multi-Task Analysis of Media-Bias Generalization
with Pre-Trained Identification of Expressions

###### Abstract

Media bias detection poses a complex, multifaceted problem traditionally
tackled using single-task models and small in-domain datasets, consequently
lacking generalizability. To address this, we introduce MAGPIE, the first
large-scale multi-task pre-training approach explicitly tailored for media
bias detection. To enable pre-training at scale, we present Large Bias Mixture
(LBM), a compilation of 59 bias-related tasks. MAGPIE outperforms previous
approaches in media bias detection on the Bias Annotation By Experts (BABE)
dataset, with a relative improvement of 3.3% F1-score. MAGPIE also performs
better than previous models on 5 out of 8 tasks in the Media Bias
Identification Benchmark (MBIB). Using a RoBERTa encoder, MAGPIE needs only
15% of finetuning steps compared to single-task approaches. Our evaluation
shows, for instance, that tasks like sentiment and emotionality boost all
learning, all tasks enhance fake news detection, and scaling tasks leads to
the best results. MAGPIE confirms that MTL is a promising approach for
addressing media bias detection, enhancing the accuracy and efficiency of
existing models. Furthermore, LBM is the first available resource collection
focused on media bias MTL.

Keywords: Media bias, Multi-task learning, Text classification

MAGPIE: Multi-Task Analysis of Media-Bias Generalization
with Pre-Trained Identification of Expressions

Tomáš Horych1,3,6, Martin Wessel4,6, Jan Philip Wahle2, Terry Ruas2,
---
Jerome Waßmuth5, André Greiner-Petter2,3, Akiko Aizawa3,
Bela Gipp2, Timo Spinde2,6
1Czech Technical University, Prague, Czech Republic
2University of Göttingen, Göttingen, Germany<EMAIL_ADDRESS>
3National Institute of Informatics, Tokyo, Japan<EMAIL_ADDRESS>
4CDTM, TU Munich, Germany, 5University of Konstanz, Germany, {last}@uni-
konstanz.de
6{t.horych, m.wessel<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Media bias is the skewed portrayal of information favoring certain group
interests Recasens et al. (2013), which manifests in multiple facets,
including political, gender, racial, and linguistic biases. Such subtypes of
bias, which can intersect and coexist in complex combinations, make the
classification of media bias a challenging task Raza et al. (2022). Existing
research on media bias detection primarily involves training classifiers on
small in-domain datasets Krieger et al. (2022), which exhibit limited
generalizability across diverse domains Wessel et al. (2023).

This paper builds upon the work of Wessel et al. (2023), emphasizing that the
multifaceted nature of media bias detection requires a shift from isolated
approaches to multi-task methodologies, considering a broad spectrum of bias
types and datasets. The recent advancements in Multi-Task Learning (MTL)
Aribandi et al. (2021a); Chen et al. (2021); Kirstein et al. (2022) open up
promising opportunities to overcome these challenges by enabling knowledge
transfer across domains and tasks. Despite the potential, a comprehensive MTL
approach for media bias detection is yet to be realized. The only other media
bias MTL method Spinde et al. (2022) underperforms due to its narrow task
focus and does not surpass baseline outcomes.

Figure 1:  MAGPIE has a pre-trained representation of multiple biases
(persuasive, subjective, etc.). This enables it to outperform models based on
single-task learning (STL) paradigms.

In this study, we make five main contributions:

  * •

We present MAGPIE (Multi-Task Media-Bias Analysis Generalization for Pre-
Trained Identification of Expressions) the first large-scale multi-task pre-
training approach for media bias detection. By including diverse biases like
persuasive and subjective, a classifier trained on top of MAGPIE correctly
classifies sentences that state-of-the-art single-task models misidentify (we
show an example in Figure 1).

  * •

We introduce LBM (Large Bias Mixture), a pre-training composition of 59 bias-
related tasks encompassing wide range of biases such as linguistic bias,
gender bias and group bias.

  * •

We provide an analysis of a task selection and demonstrate the effectiveness
of scaling the number of tasks.

  * •

We demonstrate that MAGPIE outperforms the previous state-of-the-art model by
3.3% on the Media Bias Annotation by Experts (BABE) dataset Spinde et al.
(2021c) and achieves state-of-the-art results on the Media Bias Identification
Benchmark (MBIB) collection Wessel et al. (2023).

  * •

We make all resources, including datasets, training framework, documentation,
and models, publicly available on GitHub.

github.com/magpie-multi-task

These contributions highlight the potential of MTL in improving media bias
detection. Our findings show, e.g., that tasks like sentiment and emotionality
enhance overall learning, all tasks boost fake news detection, and scaling
tasks leads to optimal results. Another key insight of our research is the
value of MTL in contexts where the primary dataset is small111For example, the
Media Bias Annotation by Experts (BABE) dataset Spinde et al. (2021c).. By
learning generalized bias knowledge from a range of tasks, we can improve the
accuracy and efficiency of existing models, even in the face of limited data.
Overall, our research offers a novel multi-task learning approach with a
variety of media bias tasks, contributing to the understanding of media bias
detection.

## 2\. Related Work

### 2.1. Media Bias

Media bias is a complex issue Lee et al. (2021a); Recasens et al. (2013); Raza
et al. (2022) composed of varying definitions of bias subtypes such as
linguistic bias, context bias, or group bias Wessel et al. (2023). In their
literature review, Spinde et al. (2023) provide an extensive overview of
research on media bias and related subtypes of bias.

Media bias detection approaches have evolved from hand-crafted features
Recasens et al. (2013); Hube and Fetahu (2018); Spinde et al. (2021d) to
neural models Spinde et al. (2022); Chen et al. (2021); Spinde et al. (2021c);
Huguet Cabot et al. (2021); Sinha and Dasgupta (2021); Raza et al. (2022).
However, existing models, so far, focus only on single tasks and saturate
quickly on smaller datasets Wessel et al. (2023). As most neural approaches
require large quantities of data, those relying on single and small datasets
cannot provide a realistic scenario for their solutions (e.g., Fan et al.
(2019)). We will first provide an overview of existing datasets and then show
how to exploit their diversity within the media bias domain.

Media bias tasks and datasets mainly cover individual, self-contained tasks
such as binary classifications Recasens et al. (2013); Spinde et al. (2021b),
which, so far, are not explored in relation to each other Spinde et al.
(2023). Wessel et al. (2023) systematically form the media bias detection
benchmark MBIB by reviewing over 100 media bias datasets and consolidating 22
of them into eight distinct tasks like linguistic, racial, and political bias.
Their study highlights that methods only focused on one of these tasks exhibit
limitations in their detection capabilities. MAGPIE encompasses all the tasks
identified in the MBIB but also significantly expands its scope by
incorporating an additional 51 media bias-related tasks due to a variety of
limitations in MBIB (see Section 3.1).

### 2.2. Multi-Task Learning

MTL shows significant improvements in various NLP tasks, including sentiment
analysis He et al. (2019), text summarization Kirstein et al. (2022), and
natural language understanding Raffel et al. (2020). In MTL, a model leverages
knowledge gained from one task to improve the performance of others. Aribandi
et al. (2021b) demonstrate that increasing the number of tasks generally leads
to improved performance for downstream NLP applications. Aghajanyan et al.
(2021) show that pre-finetuning, a large-scale multi-task learning phase,
consistently improves the performance and efficiency of pre-trained models
across diverse tasks, with results improving linearly with the number of tasks
beyond a certain threshold.

As described above, media bias can be seen as a composite problem composed of
various interrelated bias types Wessel et al. (2023). In the realm of Natural
Language Understanding (NLU), MTL has proven to be highly effective when
incorporating related tasks Aribandi et al. (2021b). For instance, benchmarks
such as GLUE and SuperGLUE successfully decompose the NLU problem into a suite
of proxy tasks, including paraphrase detection and semantic evaluation,
thereby substantially improving performance across a range of NLU tasks Wang
et al. (2018, 2019). Motivated by this success in NLU, we propose to jointly
learn from different bias types within the media bias domain. With this
approach, we aim to treat media bias not as a singular entity but as many
interconnected issues.

The selection of tasks is pivotal to the efficacy of MTL. There have been
several attempts to automate task selection, including learning the data
selection with Bayesian optimization Bingel and Søgaard (2017) or estimating
task relations Ruder and Plank (2017). The most model-agnostic approach is
GradTS Ma et al. (2021), which is highly scalable due to low resource
requirements, and is therefore implemented within MAGPIE. GradTS accumulates
gradients of attention heads and selects tasks based on their correlation with
the primary task’s attention. The selected tasks are trained jointly and share
representations across tasks.

## 3\. Methodolodgy

We implement MAGPIE using pre-finetuning as introduced in Aghajanyan et al.
(2021) (See also Section 2). As such, MAGPIE is an encoder-only MTL
transformer model pre-finetuned on 59 media bias-related auxiliary tasks
provided by Large Bias Mixture (LBM), an extensive task collection of bias-
related datasets222LBM is also published within this paper, see Section 3.1..
We incorporate a novel approach of a Head-Specific Early Stopping and
Resurrection to effectively handle tasks of varying sizes (Section 3.3.1).

Figure 2: The process of training and evaluating MAGPIE. The purple steps
describe the construction and usage of LBM, the yellow the model training, and
the green the model evaluation.

As outlined in Figure 2, our first step involves constructing the LBM.
Following this, we define the model and multi-task learning (MTL) framework
employed to train MAGPIE, which includes optimization strategies, task
sampling, and auxiliary task selection. Lastly, we evaluate MAGPIE on two
primary resources: the Media Bias Annotation by Experts (BABE) dataset333BABE
provides high-quality labels that capture a broad range of linguistic biases,
thus allowing us to evaluate our model’s generalizability within a single
dataset context. Spinde et al. (2021c), and the Media Bias Identification
Benchmark (MBIB) collection444MBIB provides a variety of bias tasks for
testing and is currently the most extensive media bias benchmark available.
Wessel et al. (2023).

### 3.1. The LBM (Large Bias Mixture) task collection

Currently, MBIB is the only collection of media bias tasks. However, it does
not include tasks that constitute a form of media bias555As per the definition
of directly and indirectly related media bias tasks given by Spinde et al.
(2023). more indirectly, such as sentiment analysis or emotion classification.
MAGPIE aims to integrate tasks both directly linked to media bias and those
peripherally related, such as sentiment analysis, to provide broader coverage
of linguistic features in the media bias context. Therefore, we introduce
Large Bias Mixture (LBM), a more general collection of relevant media bias and
media bias-related tasks, more suitable for our MTL approach. We show our task
selection process in Figure 3.

First, we manually assess a list of 115 media bias-related datasets
categorized into task families composed by Wessel et al. (2023). A task family
is a conceptual grouping of tasks that share similar objectives, such as those
related to gender bias, encompassing pronoun coreference resolution, gender
classification, and sexism detection.

We use this notion of task families to analyze general knowledge transfer
between media bias tasks, such as proposed by Aribandi et al. (2021b) for
general language tasks.

We filter the collection of the 115 datasets based on the following
criteria666We base these on the criteria set by Wessel et al. (2023). However,
we acknowledge that determining the dataset quality remains a manual and
subjective choice.:

  * •

Accessibility: Datasets have to be publicly accessible.

  * •

Text granularity: We only use datasets labeled on a sentence level (not on,
for instance, article or word level)

  * •

Quality of annotations: We exclude datasets with little to no documentation or
unreliable annotation sources.

  * •

Duplicates: We filter out datasets that contain full or partial duplicates of
each other

Of the 115 datasets collected, we discard 11 datasets that are not publicly
available. We discard 52 with article-level annotations and 5 with annotations
on other levels777One dataset has annotations on only words, two on users, and
three on outlets.. We remove 5 datasets due to low quality or unreliable
annotations and discard 4 duplicates. Applying these criteria leaves 38
datasets. Including 8 handpicked datasets not originally listed gives us 46
datasets. These are categorized into task families ensuring no overlap and
more than two datasets per family. Finally, ten datasets with multi-level
annotations, e.g., token and sentence level, are split into tasks, yielding
final number of 59 tasks.

Figure 3: The workflow of collecting datasets from the initial list to the
final LBM collection.

In order to standardize examples from various domains, we apply a unified
text-cleaning procedure to each dataset. The process involves: (1) discarding
sentences with fewer than 20 characters, (2) removing URL links, special
characters, and whitespace, and (3) eliminating duplicates potentially created
with steps (1) and (2).

The final Large Bias Mixture (LBM) includes 59 tasks, categorized into 9
distinct task families, encompassing 1,210,084 labeled sentences. We make the
LBM publicly accessible, to facilitate research in media bias detection and
other computational-social-science tasks. References and short descriptions of
all datasets and corresponding tasks can be found in Table 4.

### 3.2. The Base Model

In terms of the base language model for our procedure, we adopt a pre-trained
RoBERTa Liu et al. (2019) encoder due to its proven state-of-the-art
performances across various media bias applications Spinde et al. (2021c);
Krieger et al. (2022).

### 3.3. The MTL framework

Pre-finetuning. To effectively harness the generalization abilities of MTL for
media bias detection, we adopt a pre-finetuning procedure Aghajanyan et al.
(2021). Pre-finetuning is a complementary approach to pre-training, where a
model, already pre-trained on a typically unsupervised task, is subjected to
another intermediate step of pre-training. While incorporating MTL directly
into the pre-training stage has demonstrated performance gains Aribandi et al.
(2021b), we opt for pre-finetuning as it offers significantly reduced
computational demands while still capitalizing on the benefits of MTL
Aghajanyan et al. (2021).

Sharing representations. We use hard parameter sharing to share the underlying
encoder among all tasks while using separate task-specific heads. For each
task, we attach dense layers, or "heads", to the shared encoder. These heads
are optimized individually per task while the shared encoder learns general
bias representations.

However, multi-task optimization presents challenges due to differing gradient
directions and magnitudes Yu et al. (2020). For instance, two tasks, A and B,
may have opposing gradients with the same magnitude, nullifying their sum. On
the other hand, if Task A’s gradient greatly surpasses that of Task B,
gradient A becomes dominant. We counter the gradient misalignment by using a
variation of the PCGrad de-confliction algorithm and loss scaling Yu et al.
(2020).

Conflicting gradients and loss scaling. In multi-task training involving $n$
tasks, encoder parameters receive $n$ potentially conflicting gradients.
Efficient handling of this conflict, such as PCGrad Yu et al. (2020), requires
storing a set of gradients for each task involved in the update, leading to
infeasible memory requirements among our 59 LBM tasks. Therefore, we propose a
variation of PCGrad we call PCGrad-online which preserves the fundamental idea
of the original algorithm but is more memory efficient, requiring only one set
of gradients instead of $n$ sets per update. Adopting Muppet’s method, we
solve the issue of varying task gradient magnitudes by re-scaling the task
loss with the inverse log size of its output space, ensuring balanced
gradients and preventing task dominance in training steps Aghajanyan et al.
(2021).

#### 3.3.1. Data sampling and early stopping

To prevent large tasks from dominating the optimization, we ensure uniform
data distribution by sampling one fixed-size sub-batch from each task per
training step, a regular approach in MTL Aribandi et al. (2021b); Spinde et
al. (2022). We also employ early stopping as a regularization for each task
individually to prevent over-fitting of tasks that converge faster. However,
these methods often fall short when confronted with tasks of varied complexity
and differing convergence speeds, which both is the case for tasks in LBM.
When task A stops early while task B takes longer to converge, the latent
representation of the shared encoder shifts toward task B. We aim to mitigate
this issue by employing a training strategy that tackles the latent
representation shift using two complementary approaches:

  1. 1.

Head-Specific-Early-Stopping (HSES)

  2. 2.

Resurrection

HSES. When the task stops, we stop updating its specific head parameters while
still backpropagating its language model gradients. This method stems from the
observation that not all tasks benefit from the shared layers’ continuous
learning, especially after they have reached an optimal state.

Resurrection. When the task stops, we allow it to resume the training after
its validation loss starts to increase again. This enables the task to adapt
its head parameters to the latest latent representation.

HSES maintains quality of faster-converging tasks, while Resurrection allows
further adaptation when needed. Their combination aims for balanced, adaptive
learning for tasks with varied complexities and convergence speeds. We perform
a preliminary evaluation of the effectiveness of this method in the in Section
4.4. However, we stress the need for further extensive analysis in
Limitations.

#### 3.3.2. Auxiliary task selection

Multi-Task Learning often involves selecting well-used datasets, leading to
potential selection biases. Furthermore, manually handpicking datasets becomes
challenging due to varying and potentially ambiguous bias annotation schemes.
To automate the process of task selection, we utilize the GradTS algorithm Ma
et al. (2021). We choose GradTS due to its demonstrated efficiency and its
simplicity of implementation, which enhances its usability.

Gradient-based Task selection. In line with GradTS, we construct auxiliary
tasks as follows. We individually train all tasks, accumulate absolute
gradients, then extract and layer-wise normalize these in the attention heads,
forming a 12x12 importance matrix888RoBERTa has 12 attention heads on each of
the 12 layers. for each task. Tasks are sorted by correlation between each
task’s matrix and BABE task’s matrix. We pre-finetune $m-1$ models on the
first $k$ tasks from the sorted list, where $k$ varies from 1 to $m-1$ and $m$
is the size of LBM. The BABE task is then evaluated on these pre-finetuned
models, with the optimal $k$ determined by evaluation loss. For further
details on the GradTS algorithm, please see Ma et al. (2021).

#### 3.3.3. Hyperparameters and training stability

Finetuning transformer models on small datasets often leads to inconsistent
outcomes, such as overfitting, training instabilities, or significant
performance variabilities Dodge et al. (2020). To ensure reproducibility and
consistency, we use a fixed random seed for each run and perform individual
task hyperparameter grid searches. To negate the influence of random weight
initializations, we evaluate and compare models using 30 random seeds,
reporting average performance. We use $321$ as a random seed for all
experiments. For the final evaluation, where we employ 30 random seeds, we use
values from $0..29$. For optimizing the model, we use a per-task batch size of
32, an AdamW optimizer, and a polynomial learning scheduler. We run all
experiments on 1 NVIDIA TESLA A100 GPU with 40 GB of memory.

## 4\. Empirical Results

In this section, we present the results of our MTL approach. First, we report
the set of auxiliary tasks selected by GradTS for pre-finetuning. Next, we
assess how the model pre-finetuned on the GradTS set performs during
subsequent finetuning on the BABE task, compared to a random choice of tasks
and a full set of LBM tasks. We also compare the MTL approach to a single-task
baseline and multiple MTL baselines and evaluate the performance of our best
model, MAGPIE, on the MBIB benchmark. Then, we analyze the LBM taxonomy
through a study on knowledge transfer between families. Lastly, we evaluate
the effects of the proposed methods, HSES and Resurrection, through a
preliminary study.

### 4.1. Auxiliary tasks selection

We select suitable auxiliary tasks by calculating the correlation between
attention-head importance matrices. We use Kendall’s correlation coefficient,
as suggested by Puth et al. (2015). We find a local minimum for the BABE
evaluation loss when pre-finetuning on the first $k=10$ most correlated tasks.
The final set of the ten most correlated tasks referred to as gradts set, is
displayed in Table 1. The tasks in the gradts set demonstrate a strong
semantic connection to media bias, encompassing areas such as lexical bias,
rumor detection, and fake news detection.

Task Type |  $\tau$ (correlation)
---|---
Persuasive techniques | 0.73
Lexical/Informational bias | 0.72
Rumour detection | 0.69
Sentiment analysis | 0.68
Global warming stance detection | 0.68
Subjective bias | 0.67
Subjective bias | 0.67
Veracity classification | 0.64
Gender bias | 0.64
Fake news detection | 0.63
Table 1: Attention importance correlation wrt media bias task.

### 4.2. Evaluation

First, we finetune a model pre-finetuned on three different multi-task sets on
the BABE task and compare it against the single-task RoBERTa baseline. The
multi-task sets are the following:

  * •

MTL:Random \- Random Subset of 10 tasks

  * •

MTL:GradTS \- Subset of 10 tasks selected by GradTS algorithm

  * •

MTL:All \- Set of all tasks

We also evaluate the model pre-finetuned on the set of all tasks on MBIB. We
follow the guidelines set by Wessel et al. (2023) for the evaluation. Given
that MAGPIE’s pre-training data includes portions of the MBIB data, we ensure
that the test set for each task in MBIB is not exposed to the model during its
training or validation phases.

Figure 4: Final F1 score on a test set averaged over 30 random seeds. All
three MTL approaches outperform baseline STL finetuning. Pre-finetuning on all
LBM tasks results in significantly improved performance.

Multi-Task Learning Performance. Table 2 summarizes our performance results on
the BABE dataset. We observe that all of our MTL pre-finetuning schemas lead
to performance improvements. In particular, pre-finetuning on all tasks from
LBM yields a SOTA performance on the BABE dataset, achieving an 84.1% F1 score
and a relative improvement of 3.3% compared to the previous baseline by Spinde
et al. (2021c). While both MTL baselines - Muppet Aghajanyan et al. (2021) and
UnifiedM2 Lee et al. (2021b) outperform single-task baseline, they
underperform all of our MTL models.

Task scaling. GradTS task selection outperforms random tasks on average
performance, yet our experiment suggests task number scaling is more crucial.
This is consistent with Muppet and ExT5 results Aghajanyan et al. (2021);
Aribandi et al. (2021b), indicating MTL can compensate for scarce high-quality
media bias datasets through general bias representation from other tasks. It
also supports Kirstein et al. (2022)’s finding that sufficient related tasks
can substitute the original task.

Model | F1 | Acc | loss
---|---|---|---
Baseline (RoBERTa base) | 80.83 (±0.69) | 81.19 (±0.69) | 43.6 (±3.54)
DA-RoBERTa | 77.83 (±1.4) | 78.56 (±1.3) | 47.84 (±2.97)
MUPPET | 80.56 (±1.3) | 81.18 (±1.16) | 44.19 (±4.65)
UnifiedM2 | 81.91 (±0.91) | 82.41 (±0.88) | 44.86 (±3.99)
MTL:Random | 81.88 (±1.02) | 82.28 (±0.97) | 40.35 (±1.73)
MTL:GradTS | 82.32 (±0.79) | 82.64 (±0.8) | 40.96 (±2.36)
MTL:All |  84.1 (±1.33) |  84.44 (±1.25) |  39.46 (±2.41)
Table 2: Performance of two MTL baseline models (Muppet, UnifiedM2) two
single-task baslines (RoBERTa and DA-RoBERTa) and our three MTL models, on
fine-tuning on BABE dataset and evaluating on the held-out test set. The
results are averaged over 30 random seeds.

On MBIB, our performance surpasses that of the five models evaluated by Wessel
et al. (2023) in 5 out of 8 tasks. Additionally, it improves upon the current
baseline with an average F1-Score of 0.7692 across all tasks. However, this
improvement is only marginal. More importantly, MAGPIE outperforms the single-
task RoBERTa baseline on MBIB, showing that an MTL approach is beneficial in
media bias detection.

Figure 5: Evaluation F1 score during the final finetuning where MTL: All shows
superior performance in training-step efficiency. The values are averaged over
30 random seeds. The bands mark the lowest and highest values.

Step efficiency. In addition to the performance improvements achieved, we also
assess our model training efficiency. In Figure 5, we show the F1 score on the
development set for the BABE task, averaged over all 30 runs. Our findings
show that Multi-Task Learning only requires 1̃5% of the training steps used in
single-task finetuning on BABE. This result demonstrates the high training-
step efficiency of MAGPIE in media bias classification, making MTL
implementations in the media bias domain more viable in the future.

### 4.3. LBM taxonomy analysis

In Section 3.1, following Aribandi et al. (2021b), we introduce data task
families. Aribandi et al. (2021b) uses task families for selection and
knowledge transfer. To assess task families’ significance in LBM taxonomy, we
train each pair of families together, investigating knowledge transfer.

To account for potential negative transfer within families, we first calculate
the average transfer within each family and use it as a baseline for measuring
transfer between families. We train tasks from the same family together and
report the average change in task performance, as depicted in Figure 6.
Negative knowledge transfer is prevalent across most of our task families.
However, we observe two exceptions: the hate-speech and stance detection
families, where multi-task training leads to an average improvement in
performance.

Figure 6: Average performance change per task family. Stance detection and
hate speech are the only families, on average, benefitting from Multi-Task
Learning.

Next, we measure knowledge transfer between families by training each pair
together. We report the average impact of each family on others and the
average benefit each family gets through training with others, summarized in
Table 3.

Our results show that, on average, the Emotionality and Sentiment analysis
families provide positive transfer learning to other families. Conversely, we
observe that the Fake News family benefits from knowledge transfer from every
other family, with an average improvement of 1.7%. On the other hand, the
Emotionality family is significantly impaired by negative transfer from other
task families.

Figure 7: Average performance change when training tasks together. The change
is measured with respect to transfer within each respective family (see
results in section Figure 6.). The values in the horizontal axis represent the
received transfer by the family on the y-axis. E.g., when training
Emotionality and Subjective bias together, Emotionality gets worse by 11%
whereas Subjectivity improves by 1.5%.

The full table of transfers can be found in Figure 7. Considering that only
two families show positive transfer learning, with marginal effects of 0.11%
and 0.34%, we conclude that the task families used in the construction of LBM
are generally unsuitable for effectively utilizing knowledge transfer. We
discuss this again in Section 5.

Task Family | Transfer from | Transfer to
---|---|---
media bias | -2.07% | -0.94%
subjectivity | -1.26% | 0.89%
hate speech | -0.87% | 0.17%
gender bias | -1.01% | -1.07%
sentiment analysis | 0.11% | 0.72%
fake news | -0.13% | 1.79%
group bias | -1.04% | 0.09%
emotionality | 0.34% | -6.56%
stance detection | -0.79% | -1.83%
Table 3: Evaluation of averaged transfer between task families.

### 4.4. Resurrection and HSES evaluation

To evaluate the Resurrection and HSES methods in combination with other
training strategies, we run a grid search on the following training
strategies: HSES, Resurrection, Loss Scaling and Gradient Aggregation. We
calculate the average evaluation loss for both Resurrection and HSES methods
across 20 tasks randomly selected from the LBM collection. The boxplot in
Figure 8 shows that both methods reduce the loss by 5% and decrease the
variance across different training setups by 85%. However, we hypothesize that
a random constellation of tasks999Particularly variance in task sizes and
quality. can have a non-trivial effect on the evaluation of our technique;
thus, we opt for robust examination of the methods in future work.

(a) Resurrection

(b) HSES

Figure 8: An averaged evaluation loss of 20 tasks when trained with different
training strategies. Both Resurrection and HSES approaches, compared to the
vanilla setting, lead to significantly lower variance and overall lower loss.

## 5\. Conclusion

This paper contributes to media bias detection by the development of MAGPIE,
our large-scale multi-task learning (MTL) approach that enhances traditional
models with a multi-task structure.

Additionally, we present Large Bias Mixture (LBM), a compilation of 59 bias-
related tasks. This broad collection serves as a resource for the pre-training
of MAGPIE. To the best of our knowledge, it is the first available MTL
resource tailored specifically towards media bias.

Our study investigates the dynamics of transfer learning across tasks and task
families within media bias. Despite the occurrence of negative transfer among
several tasks, scaling the pre-training setup to all collected tasks in Multi-
Task Learning results in a 3.3% improvement over the previous state-of-the-
art, making it the biggest advancement in neural media bias classification so
far. Furthermore, we report that finetuning MAGPIE on the BABE dataset only
requires 15% of steps compared to RoBERTa single-task approaches. These
findings underscore the effectiveness and potency of Multi-Task Learning in
highly specific classification domains such as media bias.

While results suggest benefits in scaling tasks, we see more promise in novel
tasks rooted in media bias, suggesting deeper exploration over simply
expanding the task spectrum. Understanding families and tasks in datasets
necessitates systematic analysis of label definitions, rater agreement, and
inter-relatedness of dataset creation strategies. As media bias is emerging
globally, incorporating multilingual models is a natural extension.

## Limitations

We acknowledge the necessity for a more comprehensive analysis of the
performance of the HSES and Resurrection methods to ensure robust evaluation.

Given the significant computational resources required for a single multi-task
training of all tasks, we allocated these resources towards robustly
evaluating the model performance rather than conducting an in-depth analysis
of the optimization techniques. Consequently, the presented methods may be
less reliable and have limited applicability.

While there are various models available, previous research suggests that
RoBERTa exhibits strong performance on media bias tasks Wessel et al. (2023);
Spinde et al. (2021c). However, due to resource constraints, we were unable to
explore models with different architectures and further refine our selection.

As the landscape of publicly available high-quality datasets for media bias is
not as extensive as desired, we acknowledge our inability to capture all
manifestations of media bias comprehensively. As mentioned in Section 5, a
systematic and comprehensive analysis of the dataset landscape will be part of
our future work.

Furthermore, although analyzing media bias on a sentence level enables a
detailed examination of occurring biases, certain forms of bias, such as bias
resulting from the omission of information, extend beyond the linguistic
aspects of individual statements. Detecting such biases may require
considering an external context.

We leave it to future work to investigate other levels of granularity, such as
paragraph or article levels. In addition to these technical limitations,
conducting a detailed analysis of agreement and label distribution in all
utilized datasets will be necessary to make stronger claims about which
datasets provide more reliable coverage of the overall concept of media bias.
This is particularly important since media bias is a complex phenomenon that
is not always easily identified during the annotation-gathering process.

## Ethics Statement

Detecting (and as a result also highlighting) media bias instances can
mitigate the negative effects of media bias, e.g., on collective decision-
making Baumer et al. (2015). However, media bias strongly depends on the
context and individuals involved, making it a sensitive issue. Some bias forms
depend on factors other than the content, e.g., a different text perception
due to a reader’s background. Therefore, a public classification of possible
polarization and one-sidedness in the news must be performed transparently and
reliably. The datasets used for training the classifier must be transparent
and, ideally, constantly monitored by independent experts. To address possible
implications of MAGPIE or related media bias applications cautiously, it is
crucial to constantly control the classifications and datasets with recent
findings on the perception of media bias, which is a main part of our ongoing
and future work. To do so, we use resources such as, e.g., standardized
questionnaires on media bias perception Spinde et al. (2021a).

When bias detection is balanced and transparent, it can positively affect
collective opinion formation and decision-making. We see no immediate negative
ethical or societal impacts of our work. However, in addition to system
transparency, we want to highlight that we believe it to be required for
future datasets to report in greater detail about a manipulation protection
strategy when developing, training, and presenting any media bias classifier.
To ensure the validity of media bias detection systems, it is essential to
prevent participants, especially in public studies, from manipulating
algorithms by, for example, flagging neutral content as biased. Therefore,
annotations should be compared among multiple users to ensure a higher level
of trustworthiness. Most of the datasets available report only limitedly about
such strategies. In open or crowdsourcing scenarios, it is important to
collect user characteristics and deliberately include specific content that
aims to provide obvious answers but may be answered differently when users
follow certain patterns. This approach helps in detecting and mitigating
potential biases introduced by participants, thereby maintaining the integrity
and reliability of the media bias detection process. To ensure and propose
stronger standards in the future, we aim to analyze all LBM datasets with
regard to potential inherent bias in future work.

## Acknowledgments

We are grateful for the financial support of this project provided by the
Hanns-Seidel Foundation, the DAAD (German Academic Exchange Service, program
number 57515303 and 57515245), the Lower Saxony Ministry of Science and
Culture, the VW Foundation and the AI Center of Czech Technical University.
Furthermore, the authors would like to express their gratitude towards Jan
Drchal for administrative support.

## 6\. Bibliographical References

##

  * Aghajanyan et al. (2021) Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021.  Muppet: Massive multi-task representations with pre-finetuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5799–5811, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * AI (2019) Jigsaw/Conversation AI. 2019.  Jigsaw unintended bias in toxicity classification.
  * ALDayel and Magdy (2021) Abeer ALDayel and Walid Magdy. 2021.  Stance detection on social media: State of the art and trends.  _Information Processing & Management_, 58(4):102597.
  * Aribandi et al. (2021a) Vamsi Aribandi, Yi Tay, and Donald Metzler. 2021a.  How reliable are model diagnostics? In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1778–1785, Online. Association for Computational Linguistics.
  * Aribandi et al. (2021b) Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Prakash Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. 2021b.  ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning.  _CoRR_ , abs/2111.10952.  ArXiv: 2111.10952.
  * Attree (2019) Sandeep Attree. 2019.  Gendered Ambiguous Pronouns Shared Task: Boosting Model Confidence by Evidence Pooling.
  * Barikeri et al. (2021a) Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021a.  RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models.
  * Barikeri et al. (2021b) Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021b.  RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1941–1955, Online. Association for Computational Linguistics.
  * Baumer et al. (2015) Eric Baumer, Elisha Elovic, Ying Qin, Francesca Polletta, and Geri Gay. 2015.  Testing and Comparing Computational Approaches for Identifying the Language of Framing in Political News.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1472–1482, Denver, Colorado. Association for Computational Linguistics.
  * Bingel and Søgaard (2017) Joachim Bingel and Anders Søgaard. 2017.  Identifying beneficial task relations for multi-task learning in deep neural networks.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 164–169, Valencia, Spain. Association for Computational Linguistics.
  * Bostan et al. (2020) Laura Ana Maria Bostan, Evgeny Kim, and Roman Klinger. 2020.  GoodNewsEveryone: A Corpus of News Headlines Annotated with Emotions, Semantic Roles, and Reader Perception.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 1554–1566, Marseille, France. European Language Resources Association.
  * BSI (1973a) BSI. 1973a.  _Natural Fibre Twines_ , 3rd edition.  British Standards Institution, London.  BS 2570.
  * BSI (1973b) BSI. 1973b.  Natural fibre twines.  BS 2570, British Standards Institution, London.  3rd. edn.
  * Castor and Pollux (1992) A. Castor and L. E. Pollux. 1992.  The use of user modelling to guide inference and learning.  _Applied Intelligence_ , 2(1):37–53.
  * Chen et al. (2021) Shijie Chen, Yu Zhang, and Qiang Yang. 2021.  Multi-Task Learning in Natural Language Processing: An Overview.  _arXiv:2109.09138 [cs]_.
  * Chercheur (1994) J.L. Chercheur. 1994.  _Case-Based Reasoning_ , 2nd edition.  Morgan Kaufman Publishers, San Mateo, CA.
  * Chomsky (1973) N. Chomsky. 1973.  Conditions on transformations.  In _A festschrift for Morris Halle_ , New York. Holt, Rinehart & Winston.
  * Conforti et al. (2020) Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier. 2020.  Will-they-won’t-they: A very large dataset for stance detection on Twitter.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1715–1724, Online. Association for Computational Linguistics.
  * Dacon and Liu (2021) Jamell Dacon and Haochen Liu. 2021.  Does Gender Matter in the News? Detecting and Examining Gender Bias in News Articles.  In _Companion Proceedings of the Web Conference 2021_ , WWW ’21, pages 385–392, New York, NY, USA. Association for Computing Machinery.
  * Darwish et al. (2017) Kareem Darwish, Walid Magdy, and Tahar Zanouda. 2017.  Trump vs. Hillary: What Went Viral During the 2016 US Presidential Election.  In _Social Informatics_ , Lecture Notes in Computer Science, pages 143–161, Cham. Springer International Publishing.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 11.  Issue: 1.
  * Dinan et al. (2020) Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. 2020.  Multi-dimensional gender bias classification.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 314–331, Online. Association for Computational Linguistics.
  * Dinan et al. (2018) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018.  Wizard of Wikipedia: Knowledge-Powered Conversational agents.
  * Dinan et al. (2019) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019.  Wizard of Wikipedia: Knowledge-powered Conversational Agents.  In _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Dixon et al. (2018) Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018.  Measuring and Mitigating Unintended Bias in Text Classification.  In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’18, pages 67–73, New York, NY, USA. Association for Computing Machinery.
  * Dodge et al. (2020) Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020.  Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping.
  * Eco (1990) Umberto Eco. 1990.  _The Limits of Interpretation_.  Indian University Press.
  * Fan et al. (2019) Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Prafulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019.  In plain sight: Media bias through the lens of factual reporting.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6343–6349, Hong Kong, China. Association for Computational Linguistics.
  * Ferreira and Vlachos (2016) W. Ferreira and A. Vlachos. 2016.  Emergent: a novel data-set for stance classification.
  * Founta et al. (2018) Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018.  Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior.  _Proceedings of the International AAAI Conference on Web and Social Media_ , 12(1).  Number: 1.
  * Färber et al. (2020) Michael Färber, Victoria Burkard, Adam Jatowt, and Sora Lim. 2020.  A Multidimensional Dataset Based on Crowdsourcing for Analyzing and Detecting News Bias.  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, CIKM ’20, pages 3007–3014, New York, NY, USA. Association for Computing Machinery.  Event-place: Virtual Event, Ireland.
  * Gautam et al. (2020) Akash Gautam, Puneet Mathur, Rakesh Gosangi, Debanjan Mahata, Ramit Sawhney, and Rajiv Ratn Shah. 2020.  #MeTooMA: Multi-Aspect Annotations of Tweets Related to the MeToo Movement.  _Proceedings of the International AAAI Conference on Web and Social Media_ , 14:209–216.
  * Golbeck et al. (2017) Jennifer Golbeck, Zahra Ashktorab, Rashad O. Banjo, Alexandra Berlinger, Siddharth Bhagwan, Cody Buntain, Paul Cheakalos, Alicia A. Geller, Quint Gergory, Rajesh Kumar Gnanasekaran, Raja Rajan Gunasekaran, Kelly M. Hoffman, Jenny Hottle, Vichita Jienjitlert, Shivika Khare, Ryan Lau, Marianna J. Martindale, Shalmali Naik, Heather L. Nixon, Piyush Ramachandran, Kristine M. Rogers, Lisa Rogers, Meghna Sardana Sarin, Gaurav Shahane, Jayanee Thanki, Priyanka Vengataraman, Zijian Wan, and Derek Michael Wu. 2017.  A Large Labeled Corpus for Online Harassment Research.  In _Proceedings of the 2017 ACM on Web Science Conference_ , pages 229–233, Troy New York USA. ACM.
  * Grosz and Conde-Cespedes (2020) Dylan Grosz and Patricia Conde-Cespedes. 2020.  Automatic Detection of Sexist Statements Commonly Used at the Workplace.  In Wei Lu and Kenny Q. Zhu, editors, _Trends and Applications in Knowledge Discovery and Data Mining_ , volume 12237, pages 104–115. Springer International Publishing, Cham.
  * Hamborg et al. (2019) Felix Hamborg, Karsten Donnay, and Bela Gipp. 2019.  Automated identification of media bias in news articles: an interdisciplinary literature review.  _International Journal on Digital Libraries_ , 20(4):391–415.
  * He et al. (2019) Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2019.  An interactive multi-task learning network for end-to-end aspect-based sentiment analysis.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 504–515, Florence, Italy. Association for Computational Linguistics.
  * He et al. (2022) Zihao He, Negar Mokhberian, and Kristina Lerman. 2022.  Infusing Knowledge from Wikipedia to Enhance Stance Detection.
  * Hoel (1971a) Paul Gerhard Hoel. 1971a.  _Elementary Statistics_ , 3rd edition.  Wiley series in probability and mathematical statistics. Wiley, New York, Chichester.  ISBN 0 471 40300.
  * Hoel (1971b) Paul Gerhard Hoel. 1971b.  _Elementary Statistics_ , 3rd edition, Wiley series in probability and mathematical statistics, pages 19–33. Wiley, New York, Chichester.  ISBN 0 471 40300.
  * Hube and Fetahu (2018) Christoph Hube and Besnik Fetahu. 2018.  Detecting Biased Statements in Wikipedia.  In _Companion of the The Web Conference 2018 on The Web Conference 2018 - WWW ’18_ , pages 1779–1786, Lyon, France. ACM Press.
  * Hube and Fetahu (2019) Christoph Hube and Besnik Fetahu. 2019.  Neural Based Statement Classification for Biased Language.  In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_ , WSDM ’19, pages 195–203, New York, NY, USA. Association for Computing Machinery.  Event-place: Melbourne VIC, Australia.
  * Huguet Cabot et al. (2021) Pere-Lluís Huguet Cabot, David Abadi, Agneta Fischer, and Ekaterina Shutova. 2021.  Us vs. Them: A Dataset of Populist Attitudes, News Bias and Emotions.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1921–1945. Association for Computational Linguistics.  Event-place: Online.
  * Jespersen (1922) Otto Jespersen. 1922.  _Language: Its Nature, Development, and Origin_.  Allen and Unwin.
  * Kirstein et al. (2022) Frederic Thomas Kirstein, Jan Philip Wahle, Terry Ruas, and Bela Gipp. 2022.  Analyzing multi-task learning for abstractive text summarization.  In _Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)_ , pages 54–77, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Kochkina et al. (2018) Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. 2018.  All-in-one: Multi-task learning for rumour verification.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3402–3413, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Krieger et al. (2022) David Krieger, Timo Spinde, Terry Ruas, Juhi Kulshrestha, and Bela Gipp. 2022.  A Domain-adaptive Pre-training Approach for Language Bias Detection in News.  In _2022 ACM/IEEE Joint Conference on Digital Libraries (JCDL)_ , Cologne, Germany.
  * Krommyda et al. (2021a) Maria Krommyda, Anastasios Rigos, Kostas Bouklas, and Angelos Amditis. 2021a.  An Experimental Analysis of Data Annotation Methodologies for Emotion Detection in Short Text Posted on Social Media.  _Informatics_ , 8(1):19.
  * Krommyda et al. (2021b) Maria Krommyda, Anastasios Rigos, Kostas Bouklas, and Angelos Amditis. 2021b.  An Experimental Analysis of Data Annotation Methodologies for Emotion Detection in Short Text Posted on Social Media.  _Informatics_ , 8(1):19.
  * Lee et al. (2021a) Nayeon Lee, Yejin Bang, Andrea Madotto, and Pascale Fung. 2021a.  Mitigating Media Bias through Neutral Article Generation.  _CoRR_ , abs/2104.00336.  _eprint: 2104.00336.
  * Lee et al. (2021b) Nayeon Lee, Belinda Z. Li, Sinong Wang, Pascale Fung, Hao Ma, Wen-tau Yih, and Madian Khabsa. 2021b.  On unifying misinformation detection.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5479–5485, Online. Association for Computational Linguistics.
  * Lim et al. (2020) Sora Lim, Adam Jatowt, Michael Färber, and Masatoshi Yoshikawa. 2020.  Annotating and Analyzing Biased Sentences in News Articles using Crowdsourcing.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 1478–1484, Marseille, France. European Language Resources Association.
  * Liu (2012) Bing Liu. 2012.  _Sentiment Analysis and Opinion Mining_.  Synthesis Lectures on Human Language Technologies. Springer International Publishing, Cham.
  * Liu et al. (2021) Rui Liu, Zheng Lin, Yutong Tan, and Weiping Wang. 2021.  Enhancing Zero-shot and Few-shot Stance Detection with Commonsense Knowledge Graph.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3152–3157, Online. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _CoRR_ , abs/1907.11692.  Eprint: 1907.11692.
  * Luo et al. (2020) Yiwei Luo, Dallas Card, and Dan Jurafsky. 2020.  DeSMOG: Detecting Stance in Media On Global Warming.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3296–3315, Online. Association for Computational Linguistics.
  * Ma et al. (2021) Weicheng Ma, Renze Lou, Kai Zhang, Lili Wang, and Soroush Vosoughi. 2021.  GradTS: A gradient-based automatic auxiliary task selection method based on transformer networks.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5621–5632, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Maas et al. (2011a) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011a.  Learning Word Vectors for Sentiment Analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Maas et al. (2011b) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011b.  Learning Word Vectors for Sentiment Analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Mathew et al. (2021) Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021.  HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(17):14867–14875.
  * Miller et al. (2017) Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017.  ParlAI: A dialog research software platform.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 79–84, Copenhagen, Denmark. Association for Computational Linguistics.
  * Mirzakhmedova et al. (2023) Nailia Mirzakhmedova, Johannes Kiesel, Milad Alshomary, Maximilian Heinrich, Nicolas Handke, Xiaoni Cai, Barriere Valentin, Doratossadat Dastgheib, Omid Ghahroodi, Mohammad Ali Sadraei, Ehsaneddin Asgari, Lea Kawaletz, Henning Wachsmuth, and Benno Stein. 2023.  The touché23-valueeval dataset for identifying human values behind arguments.
  * Mohammad et al. (2016) Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016.  Semeval-2016 task 6: Detecting stance in tweets.  In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_ , pages 31–41.
  * Mohammad et al. (2017) Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017.  Stance and Sentiment in Tweets.  _ACM Transactions on Internet Technology_ , 17(3):26:1–26:23.
  * Nadeem et al. (2021a) Moin Nadeem, Anna Bethke, and Siva Reddy. 2021a.  StereoSet: Measuring stereotypical bias in pretrained language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5356–5371, Online. Association for Computational Linguistics.
  * Nadeem et al. (2021b) Moin Nadeem, Anna Bethke, and Siva Reddy. 2021b.  StereoSet: Measuring stereotypical bias in pretrained language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5356–5371. Association for Computational Linguistics.  Event-place: Online.
  * Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020.  CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1953–1967, Online. Association for Computational Linguistics.
  * Pang and Lee (2004) Bo Pang and Lillian Lee. 2004.  A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts.  In _Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics_ , ACL ’04, pages 271–es, USA. Association for Computational Linguistics.  Event-place: Barcelona, Spain.
  * Piskorski et al. (2023) Jakub Piskorski, Nicolas Stefanovitch, Valerie-Anne Bausier, Nicolo Faggiani, Jens Linge, Sopho Kharazi, Nikolaos Nikolaidis, Giulia Teodori, Bertrand De Longueville, Brian Doherty, Jason Gonin, Camelia Ignat, Bonka Kotseva, Eleonora Mantica, Lorena Marcaletti, Enrico Rossi, Alessio Spadaro, Marco Verile, Giovanni Da San Martino, Firoj Alam, and Preslav Nakov. 2023.  News categorization, framing and persuasion techniques: Annotation guidelines.  Technical report, European Commission Joint Research Centre, Ispra (Italy).
  * Pontiki et al. (2014) Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014.  SemEval-2014 Task 4: Aspect Based Sentiment Analysis.  In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_ , pages 27–35, Dublin, Ireland. Association for Computational Linguistics.
  * Pryzant et al. (2020) Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang. 2020.  Automatically neutralizing subjective bias in text.  In _Proceedings of the aaai conference on artificial intelligence_ , volume 34, pages 480–489.  Issue: 01.
  * Pujari et al. (2022) Rajkumar Pujari, Erik Oveson, Priyanka Kulkarni, and Elnaz Nouri. 2022.  Reinforcement Guided Multi-Task Learning Framework for Low-Resource Stereotype Detection.  Technical report.  ADS Bibcode: 2022arXiv220314349P Type: article.
  * Puth et al. (2015) Marie-Therese Puth, Markus Neuhäuser, and Graeme D. Ruxton. 2015.  Effective use of Spearman’s and Kendall’s correlation coefficients for association between two measured traits.  _Animal Behaviour_ , 102:77–84.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Raza et al. (2022) Shaina Raza, Deepak John Reji, and Chen Ding. 2022.  Dbias: Detecting biases and ensuring fairness in news articles.  _International Journal of Data Science and Analytics_.
  * Reardon et al. (2022) Carley Reardon, Sejin Paik, Ge Gao, Meet Parekh, Yanling Zhao, Lei Guo, Margrit Betke, and Derry Wijaya. 2022.  BU-NEmo: an Affective Dataset of Gun Violence News.  _Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022)_ , pages 2507–2516.
  * Recasens et al. (2013) Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013.  Linguistic Models for Analyzing and Detecting Biased Language.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 1650–1659, Sofia, Bulgaria. Association for Computational Linguistics.
  * Ruder and Plank (2017) Sebastian Ruder and Barbara Plank. 2017.  Learning to select data for transfer learning with Bayesian optimization.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 372–382, Copenhagen, Denmark. Association for Computational Linguistics.
  * Safi Samghabadi et al. (2020) Niloofar Safi Samghabadi, Parth Patwa, Srinivas PYKL, Prerana Mukherjee, Amitava Das, and Thamar Solorio. 2020.  Aggression and Misogyny Detection using BERT: A Multi-Task Approach.  In _Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying_ , pages 126–131, Marseille, France. European Language Resources Association (ELRA).
  * Samory et al. (2020) Mattia Samory, Indira Sen, Julian Kohne, Fabian Floeck, and Claudia Wagner. 2020.  "Call me sexist, but…": Revisiting Sexism Detection Using Psychological Scales and Adversarial Samples.
  * Shu et al. (2020a) Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. 2020a.  FakeNewsNet: A Data Repository with News Content, Social Context, and Spatiotemporal Information for Studying Fake News on Social Media.  _Big Data_ , 8(3):171–188.
  * Shu et al. (2020b) Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. 2020b.  FakeNewsNet: A Data Repository with News Content, Social Context, and Spatiotemporal Information for Studying Fake News on Social Media.  _Big Data_ , 8(3):171–188.
  * Singer et al. (1954–58) Charles Joseph Singer, E. J. Holmyard, and A. R. Hall, editors. 1954–58.  _A history of technology_.  Oxford University Press, London.  5 vol.
  * Sinha and Dasgupta (2021) Manjira Sinha and Tirthankar Dasgupta. 2021.  Determining Subjective Bias in Text through Linguistically Informed Transformer based Multi-Task Network.  In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, pages 3418–3422. ACM.  Event-place: Virtual Event Queensland Australia.
  * Sobhani et al. (2017) Parinaz Sobhani, Diana Inkpen, and Xiaodan Zhu. 2017.  A Dataset for Multi-Target Stance Detection.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 551–557, Valencia, Spain. Association for Computational Linguistics.
  * Socher et al. (2013a) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013a.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Socher et al. (2013b) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013b.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Spinde et al. (2023) Timo Spinde, Smi Hinterreiter, Fabian Haak, Terry Ruas, Helge Giese, Norman Meuschke, and Bela Gipp. 2023.  The media bias taxonomy: A systematic literature review on the forms and automated detection of media bias.  _CSUR_.  [in review].
  * Spinde et al. (2021a) Timo Spinde, Christina Kreuter, Wolfgang Gaissmaier, Felix Hamborg, Bela Gipp, and Helge Giese. 2021a.  Do You Think It’s Biased? How To Ask For The Perception Of Media Bias.  In _Proceedings of the ACM/IEEE Joint Conference on Digital Libraries (JCDL)_ , pages 61–69.
  * Spinde et al. (2021b) Timo Spinde, David Krieger, Manu Plank, and Bela Gipp. 2021b.  Towards A Reliable Ground-Truth For Biased Language Detection.  In _Proceedings of the ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL)_ , Virtual Event.
  * Spinde et al. (2022) Timo Spinde, Jan-David Krieger, Terry Ruas, Jelena Mitrović, Franz Götz-Hahn, Akiko Aizawa, and Bela Gipp. 2022.  Exploiting transformer-based multitask learning for the detection of media bias in news articles.  In _Proceedings of the iConference 2022_ , Virtual event.  Tex.pubstate: published tex.tppubtype: inproceedings.
  * Spinde et al. (2021c) Timo Spinde, Manuel Plank, Jan-David Krieger, Terry Ruas, Bela Gipp, and Akiko Aizawa. 2021c.  Neural Media Bias Detection Using Distant Supervision With BABE - Bias Annotations By Experts.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1166–1177, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Spinde et al. (2021d) Timo Spinde, Lada Rudnitckaia, Jelena Mitrović, Felix Hamborg, Michael Granitzer, Bela Gipp, and Karsten Donnay. 2021d.  Automated identification of bias inducing words in news articles using linguistic and context-oriented features.  _Information Processing & Management_, 58(3):102505.
  * Sridhar and Getoor (2019) Dhanya Sridhar and Lise Getoor. 2019.  Estimating causal effects of tone in online debates.  In _Proceedings of the 28th International Joint Conference on Artificial Intelligence_ , IJCAI’19, page 1872–1878. AAAI Press.
  * Strötgen and Gertz (2012) Jannik Strötgen and Michael Gertz. 2012.  Temporal tagging on different domains: Challenges, strategies, and gold standards.  In _Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12)_ , pages 3746–3753, Istanbul, Turkey. European Language Resource Association (ELRA).
  * Superman et al. (2000) S. Superman, B. Batman, C. Catwoman, and S. Spiderman. 2000.  _Superheroes experiences with books_ , 20th edition.  The Phantom Editors Associates, Gotham City.
  * Voigt et al. (2018a) Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. 2018a.  RtGender: A Corpus for Studying Differential Responses to Gender.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Voigt et al. (2018b) Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. 2018b.  RtGender: A Corpus for Studying Differential Responses to Gender.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 3261–3275.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang (2017a) William Yang Wang. 2017a.  "Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 422–426, Vancouver, Canada. Association for Computational Linguistics.
  * Wang (2017b) William Yang Wang. 2017b.  "Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection.
  * Webster et al. (2018) Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018.  Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns.  _Transactions of the Association for Computational Linguistics_ , 6:605–617.
  * Weinzierl and Harabagiu (2022) Maxwell Weinzierl and Sanda Harabagiu. 2022.  VaccineLies: A Natural Language Resource for Learning to Recognize Misinformation about the COVID-19 and HPV Vaccines.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 6967–6975, Marseille, France. European Language Resources Association.
  * Wessel et al. (2023) Martin Wessel, Tomas Horych, Terry Ruas, Akiko Aizawa, Bela Gipp, and Timo Spinde. 2023.  Introducing MBIB - The First Media Bias Identification Benchmark Task and Dataset Collection.  In _Proceedings of 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’23)_ , New York, NY, USA. ACM.  ISBN 978-1-4503-9408-6/23/07.
  * Wilson (2008) Theresa Ann Wilson. 2008.  _Fine-Grained Subjectivity and Sentiment Analysis: Recognizing the Intensity, Polarity, and Attitudes of Private States_.  Ph.D. thesis, USA.  AAI3322382.
  * Wulczyn et al. (2017a) Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017a.  Ex Machina: Personal Attacks Seen at Scale.  In _Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017_ , pages 1391–1399. ACM.
  * Wulczyn et al. (2017b) Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017b.  Ex Machina: Personal Attacks Seen at Scale.  In _Proceedings of the 26th International Conference on World Wide Web_ , WWW ’17, pages 1391–1399, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Yu et al. (2020) Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020.  Gradient surgery for multi-task learning.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 5824–5836. Curran Associates, Inc.
  * Zhang et al. (2015a) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015a.  Character-level convolutional networks for text classification.  In _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1_ , NIPS’15, pages 649–657, Cambridge, MA, USA. MIT Press.
  * Zhang et al. (2015b) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015b.  Character-level Convolutional Networks for Text Classification.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 649–657.

Task Family Dataset # sentences Task Subjective bias SUBJ Pang and Lee (2004)
10.000 Binary Classification Wiki Neutrality Corpus Pryzant et al. (2020)
52.036 Token-level Classification NewsWCL50 Hamborg et al. (2019) 731
Regression CW_HARD Hube and Fetahu (2019) 6.843 Binary Classification News
bias MultiDimNews Färber et al. (2020) 2.015 Multi-Label Classification BASIL
Fan et al. (2019) 7.987 Multi-Class Classification Token-Level Classification
Starbucks Lim et al. (2020) 866 Regression SemEval2023Task3 Piskorski et al.
(2023) 5.219 Binary Classification BABE Spinde et al. (2021c) 3.672 Binary
Classification Token-Level Classification Hate speech OffensiveLanguage
Davidson et al. (2017) 23.198 Multi-Class Classification
OnlineHarassmentDataset Golbeck et al. (2017) 19.613 Binary Classification
WikiDetoxToxicity Wulczyn et al. (2017a) 138.827 Regression
WikiDetoxAggression Wulczyn et al. (2017a) 101.159 Binary Classification
Regression Jigsaw AI (2019) 101.060 Binary Classification MeTooMA Gautam et
al. (2020) 7.388 Multi-Label Classification WikiMadlibs Dixon et al. (2018)
74.972 Binary Classification HateXplain Mathew et al. (2021) 18.962 Multi-
Class Classification Multi-Label Classification Token-Level Classification
HateSpeechTwitter Founta et al. (2018) 48.572 Multi-Class Classification
Gender bias GAP Webster et al. (2018) 4.373 Multi-Class Classification
RtGender Voigt et al. (2018a) 21.690 Binary Classification MDGender Dinan et
al. (2020) 2.332 Multi-Class Classification TRAC2 Safi Samghabadi et al.
(2020) 3.983 Binary Classification Funpedia Miller et al. (2017) 11.256 Multi-
Class Classification WizardsOfWikipedia Dinan et al. (2019) 29.777 Multi-Class
Classification Sentiment analysis SST2 Socher et al. (2013a) 9.436 Binary
Classification IMDB Maas et al. (2011a) 13.139 Binary Classification MPQA
Wilson (2008) 3.508 Binary Classification SemEval2014 Pontiki et al. (2014)
5.794 Token-Level Classification AmazonReviews Zhang et al. (2015b) 167.396
Binary Classification Fake news LIAR Wang (2017a) 12.742 Regression
FakeNewsNet Shu et al. (2020a) 21.299 Binary Classification PHEME Kochkina et
al. (2018) 5.022 Binary Classification Multi-Class Classification Emotionality
GoodNewsEveryone Bostan et al. (2020) 4.428 Token-Level Classification Token-
Level Classification BU-NEMO Reardon et al. (2022) 12.576 Multi-Class
Classification EmotionTweets Krommyda et al. (2021b) 195.744 Multi-Class
Classification DebateEffects Sridhar and Getoor (2019) 6.941 Regression Group
bias CrowSPairs Nangia et al. (2020) 3.009 Binary Classification Multi-Class
Classification Token-Level Classification StereoSet Nadeem et al. (2021b)
4.170 Binary Classification Multi-Class Classification StereotypeDataset
Pujari et al. (2022) 2.208 Binary Classification Multi-Label Classification
RedditBias Barikeri et al. (2021b) 10.395 Binary Classification Multi-Class
Classification Token-Level Classification Stance detection SemEval2023Task4
Mirzakhmedova et al. (2023) 5.219 Binary Classification VaccineLies Weinzierl
and Harabagiu (2022) 4.497 Multi-Class Classification SemEval2016Task6
Mohammad et al. (2016) 4.849 Multi-Class Classification WTWT Conforti et al.
(2020) 24.681 Multi-Class Classification MultiTargetStance Sobhani et al.
(2017) 4.430 Multi-Class Classification GWSD Luo et al. (2020) 2.010 Multi-
Class Classification $\sum$ 1.210.084

Table 4: References and description to all 59 Tasks (46 datasets) in LBM
collection.


#  Kicktionary-LOME: A Domain-Specific Multilingual
Frame Semantic Parsing Model for Football Language

Gosse Minnema
Center for Language and Cognition
University of Groningen, The Netherlands
<EMAIL_ADDRESS>
###### Abstract

This technical report introduces an adapted version of the LOME frame semantic
parsing model Xia et al. (2021) which is capable of automatically annotating
texts according to the Kicktionary domain-specific framenet resource Schmidt
(2009). Several methods for training a model even with limited available
training data are proposed. While there are some challenges for evaluation
related to the nature of the available annotations, preliminary results are
very promising, with the best model reaching F1-scores of 0.83 (frame
prediction) and 0.81 (semantic role prediction).

##  1 Introduction

Frame semantic parsing Gildea and Jurafsky (2002); Baker et al. (2007) is the
task of automatically assigning frame semantic structures Fillmore (2006),
consisting of semantic frames and their associated semantic roles, to a text.
Frame semantic parsers depend on the availability of language resources,
called framenets, for providing the set of semantic frames and roles to be
annotated, as well as a corpus of annotated examples to learn
from.111Throughout this document, I will use framenet (lower-case) for
referring to any frame semantic language resource, and FrameNet (CamelCased)
for the Berkeley FrameNet database. The best-known framenet is Berkeley
FrameNet Baker et al. (2003), a domain-general resource for English, but other
framenets exist that are multilingual Gilardi and Baker (2018) or for specific
other languages (e.g., FrameNet Brasil [Torrent et al. 2018], ASFALDA French
FrameNet [Candito et al. 2014], and many others) and for specific domains
(e.g. Venturi et al. 2009, Datta et al. 2020).

This technical report will introduce a frame semantic parser for Kicktionary
Schmidt (2009)222See http://kicktionary.de/, a domain-specific FrameNet
covering the semantic domain of football (soccer), which provides semantic
frame structures and lexical units (i.e., predicate words) with annotated
examples in French, German, and English. As a basis for our system, we will
use LOME Xia et al. (2021), a recent multilingual frame semantic parsing
model, as a basis, and experiment with several approaches for extending it to
Kicktionary.

##  2 Methods

###  2.1 Dataset: Kicktionary

Kicktionary is organized around scenes (abstract representations of the
football semantic field), each of which contains a number of frames
(representation of a specific event, situation, or concept, associated with a
set of semantic roles), each of which contains a number of lexical units (LUs;
wordform-sense pairings functioning as predicates) in English, German, and
French.

total |  | by language |  | by frame (top-5) |  | by LU (top-5)
---|---|---|---|---|---|---
8,342 |  | DE | 3,730 |  | Shot | 597 |  | treffen.v${}^{\text{DE}}$ | 25
|  | EN | 2,374 |  | Pass | 492 |  | corner.n${}^{\text{EN/FR}}$ | 22
|  | FR | 2,239 |  | Goal | 448 |  | spielen.v${}^{\text{DE}}$ | 20
|  |  |  |  | Player | 313 |  | passer.v${}^{\text{FR}}$ | 17
|  |  |  |  | Save | 256 |  | win.v${}^{\text{EN}}$ | 16

Table 1: Counts of annotated examples in Kicktionary

In turn, each lexical unit has a number of associated example sentences
(exemplars). Together, these form a corpus of 8,342 lexical units with
semantic frame and role labels, annotated on top of 7,452 unique sentences
(meaning that every sentence has, on average 1.11 annotated lexical units).
Table 1 gives more detailed information about the exemplars, split by relevant
variables.

|  | Berkeley FN | Kicktionary
---|---|---|---
fulltext | unique sentences | 6,220 | 0
annotated LUs | 29,359 | 0
exemplar | unique sentences | 163,801 | 7,452
annotated LUs | 174,551 | 8,342
total | unique sentences | 170,021 | 7,452
annotated LUs | 203,910 | 8,342

Table 2: Comparison of sentences and annotations in Berkeley FrameNet 1.7 and
Kicktionary

For the purposes of using the annotations in Kicktionary for training a frame
semantic parsing model, two properties of the corpus are likely to be
important factors for the success of the training process: (i) the quantity of
the data, but also (ii) the type of the corpus. In framenets, there are two
possible types of corpora: fulltext corpora, where entire documents are fully
annotated (i.e., all possible predicates present in the text are annotated),
and exemplar corpora, which contain sentences that are specifically chosen to
illustrate the semantics of particular predicates. Exemplar corpora typically
have only one or a few annotations per sentence, which, depending on the exact
task and the model that are used, can make learning from this data more
challenging because there are many ‘gaps’ (i.e., predicates that are included
in the sentences but not annotated, which could cause problems especially when
training end-to-end parsers); for this and other reasons, some developers of
frame semantic parsers have chosen to ignore this kind of data altogether
(e.g. Swayamdipta et al. 2017). Table 2333Examples are counted after
preprocessing using the bert-for-framenet toolkit
(https://gitlab.com/gosseminnema/bert-for-framenet/), it is possible that
different ways of counting would yield slightly different numbers (e.g. due to
sentence tokenization differences, discarding sentences with conflicting
annotations, etc.) compares both properties (i) and (ii) between the latest
release of Berkeley FrameNet and Kicktionary. This comparison reveals two
important limitations of Kicktionary: it contains only exemplar sentences, and
much fewer of them than Berkeley FrameNet. On the other hand, given the
limited semantic domain, and the limited number of frames in Kicktionary
(Berkeley FrameNet has 1013 frames with at least one exemplar sentence versus
only 106 in Kicktionary).

###  2.2 Training Kicktionary-LOME

LOME Xia et al. (2021) is a recent end-to-end444In this context, ‘end-to-end’
means that the model performs all three traditional steps Baker et al. (2007)
of the frame semantic parsing process: target/predicate identification, frame
identification, and semantic role identification. Until recently, there has
not been much attention for frame semantic parsing as an end-to-end task; see
Minnema and Nissim (2021) for a recent study of training and evaluating
semantic parsing models end-to-end. frame semantic parsing model, and the only
one (as far as I am aware) that can easily be used in a cross-lingual setting.
While the authors of LOME do not provide a detailed evaluation of the model,
they report a state-of-the art accuracy score on frame identification (on
Berkeley FrameNet 1.7), a crucial component of the frame semantic parsing
process.555At the time of publication; a model that was published later Jiang
and Riloff (2021) reported even stronger performance on the same metric, but
is not capable of performing full, end-to-end frame semantic parsing and is
not discussed any further here

LOME consists of a pre-trained XLM-R encoder Conneau et al. (2020), a BIO-
tagger (for finding frame and role spans), and a typing module (for
classifying frame and role labels). Thanks to the multilingual encoder, a
trained LOME model can produce predictions for input texts in any of the 100
languages included in the XLM-R corpus, even if these languages are not
present in the framenet training data.

To make the most out of the limited training data that is available from
Kicktionary, three different training strategies were implemented:

  * •

Simple: follow standard LOME training procedure, training the decoders from
scratch and fine-tuning the XLM-R encoder in the process;

  * •

Champions: first fine-tune the XLM-R encoder on a masked language modeling
task on a corpus666Collected by Albert Jan Schelhaas, a master student in the
Groningen CL/NLP department, as part of his thesis project. of 11,920
sentences of newspaper reports about UEFA Champions League matches. The
motivation behind this approach is that this could help to ‘pre-adapt’ the
encoder towards the semantic domain of interest: if the language model has
seen more examples of football language before it is exposed to the
Kicktionary task, it might have learned better representations for this
domained compared to the standard pre-trained XLM-R model.

  * •

Berkeley: first train LOME on Berkeley FrameNet 1.7 following standard
procedures; then, discard the decoder parameters but keep the fine-tuned XLM-R
encoder. Finally, train the decoders on the Kicktionary dataset on top of this
encoder. The intuition behind this technique is that encoder representations
that are already ‘made useful’ for doing frame semantic parsing (albeit on a
different semantic domain) by fine-tuning on Berkeley FrameNet would make
learning from the Kicktionary data more efficient.

###  2.3 Evaluation

To be able to evaluate the model, the Kicktionary corpus was randomly
split777Splitting was done on the unique sentence level to avoid having
overlap in unique sentences between the training and evaluation sets.
Splitting did not take into account frame and lexical unit labels; any given
frame or LU can have zero or more instances in each of the splits. into train
(85%), development (5%), and test (10%) sets.

The fact that Kicktionary contains only exemplar (and no fulltext) sentences
poses a challenge for evaluation: the trained LOME model will attempt to
produce outputs for every possible predicate in the evaluation sentences, but
since most sentences in the corpus have annotations for only one lexical unit
per sentence, most of the outputs of the model cannot be evaluated: if the
model produces a frame label for a predicate that was not annotated in the
gold dataset, there is no way of knowing if a frame label should have been
annotated for this lexical unit at all, and if so, what the correct label
would have been. This implies that, given outputs from an (end-to-end) LOME
system, one can reliably predict recall scores, but not precision scores.
While it would be possible to partially circumvent this problem by providing
LOME with the gold predicate(s) for each sentence and perform frame and role
prediction for only these predicates, such a setup would be very unrealistic;
it is very likely that in any real-world usage scenario of Kicktionary-LOME,
the predicates to be annotated would not be known in advance.

Instead, I will report two types of scores, which I hope will together give a
rough estimate of the models’ true precision:

  * •

SCORE${}_{\text{RAW}}$: measure end-to-end accuracy given the model’s
predictions and the gold annotations. Because of the limited annotation
coverage, we would expect very low precision scores, even for a hypothetical
perfect model. However, these scores do say something about how ‘talkative’ a
model is in comparison to other models with similar recall: a lower precision
score implies that the model predicts many ‘extra’ labels beyond the gold
annotations, while a higher score that fewer extra labels are predicted.

  * •

SCORE${}_{\text{GOLD\\_PRED}}$: here, evaluation is restricted to only
consider predicates that are present in the gold data. The precision score
resulting from this will indicate how well a model does on predicting the
correct frame and associated semantic roles, given a correctly identified
predicate. This approach is similar to inputting gold predicates at inference
time, but is more realistic because it only considers gold predicates that
would be predicted by an end-to-end model.

##  3 Experiments

###  3.1 Implementation details

LOME training was done using the same setting as in the original published
model. Masked language model fine-tuning (for the Champions strategy) was done
using the method described in Bartl et al. (2020)888I slightly adapted the
system so that it works with XLM-R. See https://github.com/marionbartl/gender-
bias-BERT for the original code., and was done over three epochs. LOME outputs
confidence scores for each frame and role label that it assigns. Based on an
initial manual inspection of dev-set predictions, I decided to apply a
confidence threshold and only consider predictions with a confidence score of
1.0, in order to improve precision.

All training was done on the University of Groningen’s Peregrine
cluster999https://www.rug.nl/society-business/centre-for-information-
technology/research/services/hpc/facilities/peregrine-hpc-cluster?lang=en,
using a single NVIDIA V100 GPU. Training took between 3 and 8 hours per model,
depending on the strategy.

###  3.2 Results and Discussion

| frames |  | roles
---|---|---|---
| raw | gold_pred |  | raw | gold_pred
| R | P | F | R | P | F |  | R | P | F | R | P | F
Simple | 0.80 | 0.12 | 0.21 | 0.80 | 0.86 | 0.83 |  | 0.80 | 0.25 | 0.38 | 0.76 | 0.87 | 0.81
Champions | 0.01 | -0.03 | -0.04 | 0.01 | -0.14 | -0.06 |  | -0.03 | -0.05 | -0.06 | -0.02 | -0.04 | -0.03
Berkeley | -0.05 | 0.02 | 0.03 | -0.05 | -0.05 | -0.05 |  | -0.06 | 0.04 | 0.04 | -0.05 | 0.03 | -0.02

Table 3: Development set results. Colored cells indicate improvement (green)
or worsening (red) with respect to the Simple model

| frames |  | roles
---|---|---|---
| raw | gold_pred |  | raw | gold_pred
| R | P | F | R | P | F |  | R | P | F | R | P | F
Simple | 0.79 | 0.12 | 0.21 | 0.79 | 0.88 | 0.83 |  | 0.79 | 0.26 | 0.39 | 0.75 | 0.89 | 0.81
Champions | 0.02 | -0.03 | -0.04 | 0.02 | -0.15 | -0.07 |  | 0.00 | -0.05 | -0.06 | 0.01 | -0.02 | 0.00
Berkeley | -0.02 | 0.03 | 0.04 | -0.02 | -0.04 | -0.03 |  | -0.05 | 0.04 | 0.03 | -0.06 | 0.00 | -0.03

Table 4: Test set results. Colored cells indicate improvement (green) or
worsening (red) with respect to the Simple model

Results for the LOME models trained using the strategies specified in the
previous sections are given in Table 3 (development set) and Table 4 (test
set). All scores were obtained using the SeqLabel evaluation method proposed
in Minnema and Nissim (2021).

Given the limited availability of training data, the models’ performance seems
surprisingly good: recall scores on both frame and role prediction are close
to 80%, which is in fact considerably higher than published end-to-end
performance (on the same metric and evaluation method) of previous frame
semantic parsing models on Berkeley FrameNet (up to 70% for frames, up to 40%
for roles; Minnema and Nissim 2021).101010No end-to-end scores on Berkeley
FrameNet have been published yet for LOME; my own experiments indicate that
LOME performs similarly to previous models on frame prediction, and improves
on role prediction, with 56% recall and 62% precision on the test set. As
discussed above, precision is much harder to evaluate, but the results here
still seem promising: $P_{\text{GOLD\\_PRED}}$ is between 86-89% on the Simple
model. This means that, for predicates that the model should annotate, it
generally generates the correct frame and role predictions. What cannot be
properly evaluated with currently available data is how frequently the model
will ‘hallucinate’, i.e., predict frame and role annotations for predicates
that should not be annotated at all.

The two strategies for making better use of (labeled and unlabeled) existing
data resources, Champions and Berkeley, do not seem to yield much benefit.
Champions gives a small improvement on frame recall, but this is offset by a
large decrease in (gold predicate) precision. On the other hand, Berkeley
improves on precision, but loses on recall. Thus, surprisingly, the Simple
model, relying only on the limited training data available from Kicktionary,
seems to be the best one overall.

##  4 Conclusions and Next Steps

This technical report addressed the issue of training a frame semantic parser
for Kicktionary, a domain-specific framenet resource for the domain of
football language, and proposed strategies for adapting the LOME system Xia et
al. (2021) for this purposes. While, given the available data, it is
challenging to evaluate end-to-end performance, preliminary results are very
promising. Future work on human evaluation on this data could be very helpful
in getting a better estimate of the porposed models’ true performance and
usability in real-world applications. There is also still room for improvement
of automatic evaluation; for example, work on checking to what degree LOME’s
predictions are consistent with the Kicktionary ontology. Finally, adapting
the SemEval’2007 method of evaluation Baker et al. (2007) to the Kicktionary
dataset could give a better indication of structure-level (as opposed to
sequence label-level) performance.

## Acknowledgements

The research reported in this technical report was funded by the Dutch
National Science organisation (NWO) through the project Framing situations in
the Dutch language, VC.GW17.083/6215. I would also like to thank Prof. Dr.
Thomas Schmidt for giving me access to the original annotations from the
Kicktionary project.

## References

  * Baker et al. (2007) Collin Baker, Michael Ellsworth, and Katrin Erk. 2007.  SemEval-2007 task 19: Frame semantic structure extraction.  In _Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)_ , pages 99–104, Prague, Czech Republic. Association for Computational Linguistics.
  * Baker et al. (2003) Collin F. Baker, Charles J. Fillmore, and Beau Cronin. 2003.  The structure of the FrameNet database.  _International Journal of Lexicography_ , 16(3):281–296.
  * Bartl et al. (2020) Marion Bartl, Malvina Nissim, and Albert Gatt. 2020.  Unmasking contextual stereotypes: Measuring and mitigating BERT’s gender bias.  In _Proceedings of the Second Workshop on Gender Bias in Natural Language Processing_ , pages 1–16, Barcelona, Spain (Online). Association for Computational Linguistics.
  * Candito et al. (2014) Marie Candito, Pascal Amsili, Lucie Barque, Farah Benamara, Gaël de Chalendar, Marianne Djemaa, Pauline Haas, Richard Huyghe, Yvette Yannick Mathieu, Philippe Muller, Benoît Sagot, and Laure Vieu. 2014.  Developing a French FrameNet: Methodology and first results.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 1372–1379, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Datta et al. (2020) Surabhi Datta, Morgan Ulinski, Jordan Godfrey-Stovall, Shekhar Khanpara, Roy F. Riascos-Castaneda, and Kirk Roberts. 2020.  Rad-SpatialNet: A frame-based resource for fine-grained spatial relations in radiology reports.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 2251–2260, Marseille, France. European Language Resources Association.
  * Fillmore (2006) C. Fillmore. 2006.  Frame semantics.  In D. Geeraerts, editor, _Cognitive Linguistics: Basic Readings_ , pages 373–400. De Gruyter Mouton, Berlin, Boston.  Originally published in 1982.
  * Gilardi and Baker (2018) Luca Gilardi and Collin Baker. 2018.  Learning to align across languages: Toward MultiLingual FrameNet.  In _Proceedings of the International FrameNet Workshop_ , pages 13–22.
  * Gildea and Jurafsky (2002) Daniel Gildea and Daniel Jurafsky. 2002.  Automatic labeling of semantic roles.  _Computational Linguistics_ , 28(3):245–288.
  * Jiang and Riloff (2021) Tianyu Jiang and Ellen Riloff. 2021.  Exploiting definitions for frame identification.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2429–2434, Online. Association for Computational Linguistics.
  * Minnema and Nissim (2021) Gosse Minnema and Malvina Nissim. 2021.  Breeding Fillmore’s chickens and hatching the eggs: Recombining frames and roles in frame-semantic parsing.  In _Proceedings of the 14th International Conference on Computational Semantics_.  https://iwcs2021.github.io/proceedings/iwcs/pdf/2021.iwcs-1.15.pdf.
  * Schmidt (2009) Thomas Schmidt. 2009.  _4\. The Kicktionary – a multilingual lexical resource of football language_, pages 101–134. De Gruyter Mouton.
  * Swayamdipta et al. (2017) Swabha Swayamdipta, Sam Thomson, Chris Dyer, and Noah A. Smith. 2017.  Frame-semantic parsing with softmax-margin segmental rnns and a syntactic scaffold.  _CoRR_ , abs/1706.09528.
  * Torrent et al. (2018) Tiago Timponi Torrent, E Matos, Ludmila Lage, Adrieli Laviola, Tatiane Tavares, VG Almeida, and Natália Sigiliano. 2018.  Towards continuity between the lexicon and the constructicon in framenet brasil.  _Constructicography: Constructicon development across languages_ , 22:107.
  * Venturi et al. (2009) Giulia Venturi, Alessandro Lenci, S. Montemagni, Eva Maria Vecchi, M. Sagri, D. Tiscornia, and T. Agnoloni. 2009.  Towards a framenet resource for the legal domain.  In _Proceedings of the Third Workshop on Legal Ontologies and Artificial Intelligence Techniques_.
  * Xia et al. (2021) Patrick Xia, Guanghui Qin, Siddharth Vashishtha, Yunmo Chen, Tongfei Chen, Chandler May, Craig Harman, Kyle Rawlins, Aaron Steven White, and Benjamin Van Durme. 2021.  LOME: Large ontology multilingual extraction.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_ , pages 149–159, Online. Association for Computational Linguistics.