# CORE: A Few-Shot Company Relation Classification Dataset
for Robust Domain Adaptation.

Philipp Borchert1,2, Jochen De Weerdt2, Kristof Coussement1,
Arno De Caigny1, Marie-Francine Moens3
1IESEG School of Management, Univ. Lille, CNRS, UMR 9221 - LEM -
Lille Economie Management, F-59000 Lille, France
2Research Centre for Information Systems Engineering, KU Leuven, Belgium
3Department of Computer Science, KU Leuven, Belgium

###### Abstract

We introduce CORE, a dataset for few-shot relation classification (RC) focused
on company relations and business entities. CORE includes 4,708 instances of
12 relation types with corresponding textual evidence extracted from company
Wikipedia pages. Company names and business entities pose a challenge for few-
shot RC models due to the rich and diverse information associated with them.
For example, a company name may represent the legal entity, products, people,
or business divisions depending on the context. Therefore, deriving the
relation type between entities is highly dependent on textual context. To
evaluate the performance of state-of-the-art RC models on the CORE dataset, we
conduct experiments in the few-shot domain adaptation setting. Our results
reveal substantial performance gaps, confirming that models trained on
different domains struggle to adapt to CORE. Interestingly, we find that
models trained on CORE showcase improved out-of-domain performance, which
highlights the importance of high-quality data for robust domain adaptation.
Specifically, the information richness embedded in business entities allows
models to focus on contextual nuances, reducing their reliance on superficial
clues such as relation-specific verbs. In addition to the dataset, we provide
relevant code snippets to facilitate reproducibility and encourage further
research in the field.111The CORE dataset and code are publicly available at
https://github.com/pnborchert/CORE.

##  1 Introduction

Relation classification (RC) is a fundamental task in natural language
processing that involves identifying the corresponding relation between a pair
of entities based on the surrounding textual context. Several studies
highlight the need for more realistic dataset configurations that include
None-of-the-Above (NOTA) detection Gao et al. (2019); Sabo et al. (2021), few-
shot evaluation Baldini Soares et al. (2019); Sabo et al. (2021), and out-of-
domain adaptation Gao et al. (2019). However, most existing RC datasets
comprise a mixture of domains, and the sets of entities across different
domains are often easily distinguishable from one another. The main challenge
with these general domain datasets is that few-shot models may take shortcuts
during training, given the limited set of possible relation types associated
with specific entities Liu et al. (2014); Lyu and Chen (2021). For instance,
the relation type “place of birth” requires a pair of entities consisting of
entity types “Person” and “Geolocation”. This can lead to poor generalization
performance when evaluated on out-of-domain data.

Coke’s E1 advertising is pervasive, as one of Woodruff’s stated goals was to
ensure that everyone on Earth drank Coca-Cola as their preferred beverage.
Coke’s E1Legal EntityBrandProduct Figure 1: Example for information richness
embedded in named business entities. Published weekly, the Commercial &
Financial Chronicle E1 was deliberately modeled to be an American take on the
popular business newspaper The Economist E2, which had been founded in England
in 1843. competitor ofacquired bymerged with$E_{1}$$E_{2}$Entity position
annotationRelation type annotation Relation direction annotation Figure 2:
CORE example for an undirected business relation.

To address these challenges, we introduce CORE, a high-quality dataset that
focuses specifically on company relations. By narrowing the scope to company
relations and business entities, CORE presents a more difficult task for
models, as business entities embody a variety of information and may be used
interchangeably to represent the legal entity, products or services, and
brands, depending on the context (see Figure 1). As a result, inferring the
relation type solely based on the entities becomes more challenging.
Additionally, business entities are well-suited to minimize entity type
related clues due to the richness of associated information they possess. By
minimizing these clues, CORE aims to incentivize models to more effectively
leverage textual context and improve predictive performance on out-of-domain
data.

The few-shot settings commonly used in distantly supervised datasets are built
upon unrealistic assumptions (Sabo et al., 2021). These assumptions include
unrealistic relation types, NOTA and entity distributions, as well as a narrow
focus on named entities while disregarding common nouns and pronouns. The
study shows that models trained on more realistic few-shot evaluation settings
exhibit a significant decline in predictive performance. In contrast, CORE
overcomes these limitations as it is a human-annotated dataset, not bound by
the relations or named entity types present in knowledge bases. It encompasses
relation types that are absent in knowledge bases, and includes entity types
beyond named entities, such as common nouns and pronouns. On the contrary,
distant supervision relies on the extraction of textual context based on
subject-relation-object triples in knowledge bases Han et al. (2018), which
restricts the scope to named entities Sabo et al. (2021).

We contribute to the field of relation classification in three significant
ways through our study. First, we introduce CORE, a high-quality dataset that
encompasses company relations and extends domain coverage in RC. By focusing
on company names and business entities, CORE creates a more demanding
benchmark for relation classification models. These entities embody diverse
information and can be used interchangeably to represent legal entities,
products or services, and brands, depending on the context. Secondly, our
benchmark of state-of-the-art models on CORE uncovers substantial performance
gaps in few-shot domain adaptation. RC models trained on different domains
struggle to adapt to CORE, whereas models trained on CORE itself outperform
other domain-adapted models. Lastly, our study highlights the critical role of
high-quality datasets in enabling robust domain adaptation for few-shot RC.
Models evaluated on out-of-domain data commonly rely on heuristic patterns and
face challenges in adapting to contextual information.

In summary, this paper makes a significant contribution by introducing CORE, a
novel and high-quality dataset for relation classification. In the context of
few-shot domain adaptation, our study demonstrates the challenges faced by
state-of-the-art models when dealing with CORE, while highlighting the
remarkable performance of models trained on CORE itself.

##  2 Related work

Recent research on datasets for few-shot RC has been focused on improving data
quality. Several approaches have been proposed, including extensions and
augmentations of existing datasets to incorporate features such as NOTA
detection or domain adaptation, as well as correction of labeling errors Alt
et al. (2020); Gao et al. (2019).

In this section, we present an overview of recent RC datasets that encompass
sentence-level textual context. One of such datasets is TACRED, which is a
human-annotated dataset based on online news, focusing on people and
organizations as entities Zhang et al. (2017). It preserves a realistic
relation type distribution and includes 79.5% NOTA instances. Alt et al.
(2020) re-evaluated the original dataset, analyzing labeling errors, and
investigating sources for incorrect model predictions. Another prominent
dataset is FewRel, introduced by Han et al. (2018), which consists of text
passages extracted from English Wikipedia pages. It is annotated using distant
supervision and the Wikidata knowledge base. The model evaluation is primarily
focused on the K-Way N-Shot setting. Gao et al. (2019) extended the dataset by
incorporating NOTA examples and introducing out-of-domain test sets. The
CrossRE dataset specifically targets the domain adaptation setting in relation
classification Bassignana and Plank (2022). It incorporates text from
Wikipedia and news sources, covering six domains. Notably, the dataset
excludes the business domain and is limited to named entities. In terms of
relation types, CrossRE adopts a shared approach across domains, omitting any
domain-specific relation types. Khaldi et al. (2022) acknowledge the value of
business relations extracted from multilingual web sources and present their
dataset, which is collected using distant supervision. This dataset focuses
primarily on named entities and subsequent human annotation. It contains five
types of undirected business relationships.

Domain adaptation refers to the task of transferring information from a source
domain to a specific target domain. Domain adaptation models learn to extract
domain-invariant information from the source domain, which can be applied to
the target domain. Popular approaches include domain adversarial networks Han
et al. (2021a); Gao et al. (2019) and leveraging external information such as
task descriptions or knowledge bases Zhang et al. (2021); Zhenzhen et al.
(2022). Recent work has also explored prompt tuning approaches Gao et al.
(2021); Schick and Schütze (2021); Wu and Shi (2022). In this study, we
address the problem of domain adaptation by exposing the model to a limited
number of instances from the target domain using few-shot learning. To achieve
effective domain adaptation, models combine domain-invariant information
learned from the source domain with the information available from the target
domain.

##  3 CORE

product or service ofundefinedcollaborationacquired byclient ofshareholder
ofsubsidiary oftraded oncompetitor ofbrand ofmerged withregulated
by$0$$200$$400$$600$$800$$715$$659$$564$$492$$429$$302$$240$$216$$160$$92$$85$$46$Count
Figure 3: Relation types and distribution in the training set.

###  3.1 Data collection

To gather corporate Wikipedia pages in English, we conducted a query on
DBpedia Lehmann et al. (2015) using the ontology class “dbo:Company”, which
corresponds to the Wikidata identifier “Q4830453”. The dataset was constructed
using Wikipedia content corresponding to April 2022.

We extract candidate entity pairs based on the company corresponding to the
Wikipedia page and named entities in the text. We use a named entity
recognition (NER) model based on RoBERTa-Large Liu et al. (2019) that was
fine-tuned on the CoNLL2003 dataset to retain entities tagged as organizations
or miscellaneous. The model achieved state-of-the-art performance on CoNLL2003
with a F1-Score of 97% on the full dataset and 96% on entities labeled as
organizations. This extraction procedure allows the company corresponding to
the Wikipedia page to take any entity type, including named entities,
pronouns, and nouns. We use spaCy to create a dictionary of nouns and pronouns
in the dataset and manually validate entity types (Honnibal and Montani,
2017). For each candidate entity pair, we extract the corresponding paragraph
as supporting context characterizing the relation between the entities. We
remove duplicate entity pairs with overlapping context to reduce the number of
candidate samples and manually filter out miscellaneous entities that do not
belong to the category of business entities, such as “German” or “Dutch”.
Finally, we randomly sampled 14,000 candidate samples for subsequent
annotation.

The annotation platform is created by the authors, offering a similar
interface as commercial services such as MTurk (see Appendix A.5). The dataset
is annotated by 50 students in business and management who are presented with
candidate entity pairs and the corresponding textual context. The annotation
consists of selecting one of the predefined relations (see Appendix A.1) and
the textual context relevant to determine the selected relation (see Figure
2). Since relations such as “acquired by” imply a direction, annotators are
also asked to indicate whether the entity order is reversed:
$(e_{1},r,e_{2})\rightarrow(e_{2},r,e_{1})$. The relation annotation and the
position of $e1$ and $e2$ in the textual context are validated in the
subsequent annotation step. Thereby, the validation process includes the
removal of candidate instances that contain non-business entities, incorrect
relation type selection, insufficient context, or inappropriate data.

The annotators were recruited through an open application process. For their
contribution to the annotation process, the students received a reimbursement
of $11 per hour. The total compensation for data annotation amounted to
approximately $6,600.

###  3.2 Quality control

Data quality is particularly important in low resource settings with models
exposed to only a few supporting instances. Alt et al. (2020) find that
incorrectly labeled instances substantially reduce predictive performance in
RC models. In close consideration of recent literature, we maintain high data
quality in CORE Alt et al. (2020); Gao et al. (2019); Sabo et al. (2021). The
annotation process involves at least three different annotators, with the
annotator agreement computed based on relation type annotation, relation
direction annotation, and entity annotation. The corresponding Fleiss Kappa is
0.636. We only retain instances with at least 66% inter-annotator agreement.

We educate annotators by providing a mandatory step-by-step tutorial along
with guiding documentation on the platform. Further quality control measures
include a control set annotated by the authors. Consequently, we excluded
annotations from 6 annotators who did not pass the quality control check. We
employ consistency checks during the annotation dialog on the platform. For
each example, the annotators select business entity types, such as “Company”
or “Brand”, corresponding to $e1$ and $e2$. The platform validates
corresponding relations and entity types, i.e., annotators are required to
indicate one of the entities as “product or service” when selecting the
“product or service of” relation.

###  3.3 Dataset properties and analysis

The dataset’s key properties are presented in Table 1 and discussed in the
subsequent section. CORE consists of 12 annotated relation types that were
predefined by the authors, as shown in Figure 3. Among these, there is an
“undefined” relation that signifies a relationship between $e1$ and $e2$ that
does not match any predefined relation type. Thus, “undefined” corresponds to
the NOTA category. However, examples containing non-business entities or
missing entities were excluded from the dataset.

Category |  | # Train | # Test
---|---|---|---
Instances |  | 4,000 | 708
Relations | Directed | 8 | 8
| Undirected | 4 | 4
| Total | 12 | 12
Entity types | Named entity | 6,777 | 1,214
| Noun | 721 | 114
| Pronoun | 502 | 88
Avg. tokens |  | 34 | 34
Table 1: Dataset characteristics

To better understand the distinctions between CORE and FewRel, which are both
based on Wikipedia, we examined their similarities in terms of shared
instances and relation types. We utilized cosine similarity based on tf-idf
values to quantify the textual similarity between instances. Subsequently, we
manually analyzed 76 instance pairs that exhibited a similarity greater than
50%. Our analysis revealed nine pairs with matching context, among which one
instance pair shared entities and relation type annotations. The remaining
pairs focused on different entities and served as examples for various
relation types. Examples of these instances are provided in Table 2.

Furthermore, we performed an analysis of the Wikidata properties used in
FewRel, including P449 (original broadcaster), P123 (publisher), P178
(developer), P176 (manufacturer) and P750 (distributed by). We observed that
all of these properties align with CORE’s “product or service of” relation
type. Furthermore, we found that the properties P355 (has subsidiary) and P127
(owned by) partially overlap with “subsidiary of” and “acquired by” in CORE.
However, these properties include non-business entities and have a broader
interpretation of “subsidiary” and “ownership”, encompassing concepts like
“part-of”, physical location, or control. Examples illustrating this overlap
are shown in Appendix A.2.

Text | CORE | FewRel
---|---|---
On 4 September 2014, they changed their business model to coupon store and served Flipkart, Jabong, Myntra, Zovi, Koovs & Zivame as their clients. | client of | has subsidiary
Its games include the Microsoft Solitaire Collection which comes loaded on Windows 10, Windows 8, and mobile devices. | undefined | operating system
Other of the company’s hits were Bonequinha de Seda in 1936, Estudantes in 1935, Ganga Bruta in 1933, and Limite in 1931. | product or service of | followed by

Table 2: Shared instances between CORE and FewRel with corresponding entities
highlighted in blue and orange, respectively.

###  3.4 Dataset splits

The annotated instances were randomly divided into a training set (4,000
instances) and a test set (708 instances), each split containing all available
relation types. We do not include a dedicated development set, but use part of
the training set for model development Mukherjee et al. (2021).

##  4 Experiments

###  4.1 Task formulation

We adopt the N-way K-shot evaluation setting, where the model is exposed to a
limited number of instances $N\times K$ during training. The episodes sampled
from the dataset consist of two parts: (1) a support set
$S=\\{(x_{i},r_{i})\\}^{K\times N}_{i=1}$ containing input $x$ along with $N$
randomly sampled relations $r\in\mathcal{R}$ as well as $K$ instances for each
relation and (2) a query set $Q=\\{x_{i}\\}^{K\times N}_{i=1}$ containing $K$
instances of the $N$ relations not included in the support set Gao et al.
(2019, 2021).

In line with recent literature, we emphasize the importance of NOTA instances
in the few-shot evaluation setting Gao et al. (2019). By including NOTA as an
additional relation type, we effectively sample $N+1$ relations in each
episode. Thereby, the NOTA relation consists of the “undefined” relation type
and any relations not used for the training and test episodes Sabo et al.
(2021).

###  4.2 Episode sampling

The sampling procedure accommodates overlapping relation types in the
dedicated training and test sets. We consider two disjoint sets of relation
types $r\textsubscript{train}\cap r\textsubscript{test}=\varnothing$ for
training and test episodes, respectively. We uniformly sample $N\times K$
support instances from the corresponding set of relation types. This exposes
models to the same number of support instances per relation, disregarding the
relation distribution in the dataset. Considering the relation distribution in
the model evaluation, we sample $N\times K$ query instances randomly Sabo et
al. (2021). All instances included in training episodes are sampled from the
training set. For test episodes we sample support instances from the training
set and query instances from the test set.

###  4.3 Models

To facilitate the comparison of evaluation results, we benchmark SOTA models
in few-shot RC literature, considering performance and prior application in
the domain adaptation setting (Sabo et al., 2021; Gao et al., 2021). We
utilize BERT-Base with 110 million parameters Devlin et al. (2019) as an
encoder for all benchmarked models. This prevents information disparities
between language models, which are commonly trained on Wikipedia data.

Prototypical Networks (Proto) create relation type prototypes using the
support examples Snell et al. (2017). Query instances are matched to relation
prototypes by computing similarity scores.

In the BERT-Pair approach, query instances are concatenated with support
instances computing a similarity probability to determine whether the query
instances share the same relation type as the support instances Gao et al.
(2019).

In the BERT with Entity Markers (BERTEM) approach, the entities are embedded
with entity marker tokens at the respective start and end positions in the
context Baldini Soares et al. (2019). The concatenated representation of the
entity start markers is used as the example representation. Query instances
are classified based on similarity to the support instances.

For the BERT-Prompt approach, we express the relation classification task as a
masked language modeling (MLM) problem Gao et al. (2021); Schick and Schütze
(2021). Given a template $\mathcal{T}$, each input is converted into
$x_{\text{prompt}}=\mathcal{T}(x)$ containing at least one $[\text{MASK}]$
token. Similarly to the MLM task performed during pretraining of a language
model $\mathcal{M}$, the masked token is predicted based on the surrounding
context. To accommodate differences in model vocabularies, the relations are
mapped to individual tokens included in the vocabulary:
$\mathcal{R}\mapsto\mathcal{V}_{\mathcal{M}}$. Given input $x$, the
probability of predicting relation $r$ is denoted as:

$\begin{split}p(r\mid x)=p([\text{MASK}]=w_{v}\mid x_{\text{prompt}})\\\
=\frac{\exp{\left(w_{v}\cdot
h_{\text{[MASK]}}\right)}}{\sum_{v^{\prime}\in\mathcal{V}}{\exp{\left(w_{v^{\prime}}\cdot
h_{\text{[MASK]}}\right)}}},\end{split}$ (1)

where $h_{[\text{MASK}]}$ is the hidden vector of $[\text{MASK}]$ and $w_{v}$
denotes the pre-softmax vector corresponding to $v\in\mathcal{V}$ Gao et al.
(2021); Han et al. (2022).

The Hybrid Contrastive Relation-Prototype (HCRP) approach is designed to learn
from relation types that are challenging to distinguish. It employs a hybrid
approach, creating prototypes and integrating additional data in the form of
relation descriptions. HCRP incorporates a task-adaptive focal loss and
utilizes contrastive learning techniques to enhance the discriminative power
of the prototypes (Han et al., 2021b).

###  4.4 Experimental setup

We evaluate RC models in the few-shot domain adaptation setting for CORE,
FewRel and TACRED. Our evaluation process consists of two steps. First, we
evaluate the in-domain model performance. In the second step, we evaluate the
out-of-domain performance on the respective test sets. We keep the number of
training episodes consistent across datasets to prevent information disparity
across domains. For each dataset, we sampled 22,500 training episodes, which
corresponds to approximately 10 epochs for $K=1$ on CORE. In the case of
FewRel, which has an artificial relation type distribution, we followed the
approach described in Gao et al. (2019). They uniformly sample support and
query instances from relation types, which also obscures the distribution of
NOTA instances in the underlying data source. To address this, we selected a
50% NOTA rate Gao et al. (2019) and applied the same sampling procedure to all
models trained on FewRel, as in Sabo et al. (2021).

##  5 Results and discussion

We evaluate the performance of the models using two evaluation metrics: micro
F1 and macro F1. Micro F1 provides an overall assessment of model performance
by treating instances equally and accounting for the impact of an imbalanced
NOTA ratio. In contrast, macro F1 focuses on individual class performance,
treating each class equally and remaining unaffected by the NOTA ratio. We
report micro F1 and macro F1 scores in percentage format in Tables 3, 4, and
5. The best results in each table are highlighted in bold, while the best
performance measures for each domain are underlined. To ensure robust
evaluation, the models were evaluated on ten sets of training and test
relations for each dataset and N-Way K-Shot setting. The resulting evaluation
metrics were then averaged.

###  5.1 Results CORE

Domain | Model | 5-1 | 5-5
---|---|---|---
In-domain | Proto-BERT | 26.54 / 26.92 | 31.96 / 43.87
| BERT-Pair | 26.77 / 25.06 | 34.78 / 34.18
| BERTEM | 36.27 / 44.82 | 44.53 / 56.04
| BERT-Prompt |  53.27 / 67.30 |  74.16 / 88.07
| HCRP | 34.82 / 38.69 | 47.40 / 50.15
FewRel | Proto-BERT | 19.46 / 21.92 | 16.71 / 24.55
| BERT-Pair | 18.74 / 18.86 | 16.58 / 22.79
| BERTEM |  26.65 / 33.97 | 30.44 / 34.22
| BERT-Prompt | 25.01 / 36.35 |  33.26 / 35.70
| HCRP | 23.54 / 37.49 | 25.41 / 41.67
TACRED | Proto-BERT | 17.93 / 18.66 | 15.13 / 21.14
| BERT-Pair | 17.50 / 17.00 | 16.77 / 16.95
| BERTEM | 26.47 / 30.38 | 27.38 / 28.79
| BERT-Prompt |  29.61 / 44.12 | 37.88 / 42.60
| HCRP | 27.23 / 32.85 |  41.91 / 41.61

Table 3: Micro F1 / Macro F1 for in-domain and out-of-domain models evaluated
on CORE.

The in-domain results presented in Table 3 illustrate the superior performance
of the BERT-Prompt model across all few-shot settings, surpassing other
approaches in terms of both micro F1 and macro F1 scores. Additionally, our
findings reveal that the BERTEM model exhibits enhanced performance compared
to the Proto-BERT and BERT-Pair models, as indicated by significant
improvements in both micro F1 and macro F1 metrics.

###  5.2 Results domain adaptation

Out-of-domain evaluation tests models’ ability to transfer information across
different domains. Following the approach in Gao et al. (2019), models learn
to establish mappings between support and query samples using the source
domain data and corresponding labels ($r\textsubscript{train}$). In-domain
evaluation is performed on $r\textsubscript{test}$, where even within this
context, label spaces do not overlap. To accomplish this, the model is exposed
to $N\times K$ support examples from the target domain. Out-of-domain
evaluation follows the same procedure, but with $r\textsubscript{test}$ and
support samples from a different domain.

Table 3 presents the performance of models trained on FewRel and TACRED,
evaluated on CORE. These results reveal substantial performance gaps between
out-of-domain models and their in-domain counterparts, underscoring the
challenges of generalizing from FewRel and TACRED to CORE. Notably, BERT-
Prompt trained on TACRED, achieves the highest performance among the evaluated
models, though it still falls significantly short of the performance of the
in-domain models. Interestingly, the performance differences among the other
out-of-domain models are minimal, implying that all modeling approaches
struggle to generalize to CORE. Furthermore, Proto-BERT, BERT-Pair, and BERTEM
models trained on TACRED consistently demonstrate lower macro F1 values
compared to the models trained on FewRel.

To investigate the performance of in-domain models on out-of-domain datasets,
we compared them with out-of-domain results for FewRel and TACRED, as
presented in Tables 4 and 5. Our findings show that the Proto-BERT, BERTEM and
HCRP models trained on CORE consistently outperform the in-domain models
trained on FewRel, while the in-domain BERT-Prompt model remains the best
performing model. The best performing out-of-domain models are HCRP trained on
CORE and HCRP trained on TACRED.

It is worth noting that both CORE and FewRel are based on examples extracted
from Wikipedia. While lexical similarity can partially account for the
performance differences between the models trained on CORE and TACRED in Table
4, models trained on FewRel fail to outperform the models trained on TACRED,
as shown in Table 3.

Domain | Model | 5-1 | 5-5
---|---|---|---
In-domain | Proto-BERT | 20.48 / 25.40 | 19.86 / 20.75
| BERT-Pair | 21.66 / 21.32 | 23.87 / 33.78
| BERTEM | 35.25 / 48.04 | 46.43 / 51.50
| BERT-Prompt |  56.02 / 71.71 |  59.80 / 86.81
| HCRP | 36.33 / 51.07 | 52.09 / 63.11
CORE | Proto-BERT | 26.19 / 31.64 | 29.82 / 39.52
| BERT-Pair | 18.55 / 20.35 | 18.99 / 22.98
| BERTEM | 40.27 / 52.42 | 54.03 / 65.45
| BERT-Prompt | 45.08 / 48.01 | 51.82 / 39.05
| HCRP |  51.56 / 57.55 |  61.24 / 67.96
TACRED | Proto-BERT | 20.28 / 23.53 | 20.95 / 27.62
| BERT-Pair | 16.44 / 16.66 | 13.78 / 17.67
| BERTEM | 34.19 / 43.95 | 38.88 / 47.29
| BERT-Prompt | 48.18 / 54.80 | 53.82 / 50.50
| HCRP |  49.15 / 53.89 |  61.99 / 63.33

Table 4: Micro F1 / Macro F1 for in-domain and out-of-domain models evaluated
on FewRel. The models are evaluated on the development set corresponding to
the Wikipedia data.

Regarding domain adaptation results on the TACRED dataset, we observe that the
micro F1 performance of the BERT-Prompt model trained on the CORE dataset is
comparable to that of the in-domain model trained on TACRED. However, the
macro F1 scores of the out-of-domain models are substantially lower.
Furthermore, it is evident that Proto, BERT-Pair, and BERTEM models, including
those trained on FewRel, struggle to adapt to TACRED when considering the
micro F1 results. When examining the macro F1 scores, we find that the BERTEM
and BERT-Prompt models trained on FewRel demonstrate relatively better
performance compared to the models trained on CORE. However, this improvement
is not reflected in the micro F1 scores.

Domain | Model | 5-1 | 5-5
---|---|---|---
In-domain | Proto-BERT | 28.50 / 14.68 | 25.33 / 20.38
| BERT-Pair | 24.25 / 17.71 | 30.57 / 18.72
| BERTEM | 25.19 / 19.16 | 44.13 / 60.39
| BERT-Prompt |  35.70 / 63.00 |  67.81 / 74.02
| HCRP | 29.58 / 26.43 | 57.28 / 35.61
CORE | Proto-BERT | 15.72 / 24.67 | 11.26 / 29.20
| BERT-Pair | 14.47 / 17.48 | 11.44 / 21.89
| BERTEM | 16.32 / 38.65 | 18.80 / 48.39
| BERT-Prompt |  36.08 / 45.07 |  65.46 / 35.53
| HCRP | 34.04 / 37.54 | 47.20 / 44.76
FewRel | Proto-BERT | 16.36 / 22.29 | 10.74 / 26.70
| BERT-Pair | 13.16 / 18.47 | 8.75 / 19.69
| BERTEM | 12.81 / 40.07 | 15.77 / 44.51
| BERT-Prompt |  18.71 / 49.10 |  38.21 / 45.67
| HCRP | 13.72 / 33.76 | 33.17 / 40.03

Table 5: Micro F1 / Macro F1 for in-domain and out-of-domain models evaluated
on TACRED.

To demonstrate our model’s performance on domain-specific datasets, as opposed
to general domain datasets, we included PubMed (Gao et al., 2019) and SCIERC
(Luan et al., 2018) in our evaluation. PubMed contains examples extracted from
biomedical literature, featuring 10 unique relation types. Notably, both
FewRel and PubMed are authored by Gao et al. (2019) and share certain
characteristics, including artificially balanced relation distributions.
SCIERC comprises examples from scientific literature within the AI domain,
including 7 unique relation types. It provides a distinctive set of relation
types and follows a different data extraction procedure, adding diversity to
our benchmarked datasets. Both of these datasets have a limited number of
relation types and do not include an annotated NOTA category. Therefore,
following the procedure outlined in Section 4, the inclusion of in-domain
models is not feasible.

Domain | Model | 5-1 | 5-5
---|---|---|---
CORE | Proto-BERT | 22.25 / 21.38 | 27.22 / 28.48
| BERT-Pair | 17.79 / 19.31 | 15.24 / 19.10
| BERTEM |  38.87 / 43.84 |  56.13 / 62.50
| BERT-Prompt | 24.68 / 25.18 | 35.85 / 41.09
| HCRP | 29.74 / 52.35 | 34.96 / 55.66
FewRel | Proto-BERT | 24.63 / 25.14 | 26.36 / 27.43
| BERT-Pair | 17.39 / 18.91 | 24.53 / 32.00
| BERTEM |  42.01 / 34.11 |  52.80 / 62.81
| BERT-Prompt | 25.88 / 41.81 | 28.27 / 41.56
| HCRP | 32.41 / 40.21 | 37.99 / 36.50
TACRED | Proto-BERT | 19.37 / 20.26 | 19.64 / 21.61
| BERT-Pair | 15.61 / 16.05 | 13.57 / 16.28
| BERTEM |  32.12 / 36.99 |  38.72 / 40.32
| BERT-Prompt | 29.36 / 36.00 | 34.60 / 35.14
| HCRP | 32.00 / 45.13 | 37.14 / 48.63

Table 6: Micro F1 / Macro F1 for out-of-domain models evaluated on PubMed.

Domain | Model | 5-1 | 5-5
---|---|---|---
CORE | Proto-BERT | 17.71 / 17.78 | 19.37 / 19.83
| BERT-Pair | 17.69 / 17.54 | 20.42 / 20.03
| BERTEM | 22.20 / 24.94 | 29.28 / 33.69
| BERT-Prompt |  41.31 / 28.13 |  47.88 / 26.57
| HCRP | 18.54 / 22.31 | 18.41 / 23.43
FewRel | Proto-BERT | 16.10 / 17.27 | 16.21 / 18.00
| BERT-Pair | 16.51 / 17.88 | 17.13 / 19.57
| BERTEM | 20.41 / 21.63 | 21.78 / 25.57
| BERT-Prompt |  32.06 / 23.95 |  36.15 / 23.18
| HCRP | 14.20 / 22.21 | 15.40 / 24.87
TACRED | Proto-BERT | 15.31 / 16.93 | 16.54 / 17.06
| BERT-Pair | 15.46 / 15.57 | 16.04 / 15.77
| BERTEM | 17.74 / 20.14 | 17.31 / 20.97
| BERT-Prompt |  39.59 / 24.67 |  48.08 / 22.56
| HCRP | 21.03 / 23.25 | 33.66 / 24.98

Table 7: Micro F1 / Macro F1 for out-of-domain models evaluated on SCIERC.

The out-of-domain evaluation on PubMed, presented in Table 6, demonstrate
that, on average, the BERT-EM model trained on CORE outperforms BERT-EM models
trained on FewRel and HCRP trained TACRED by 2.40% and 9.61%, respectively.
The out-of-domain evaluation results for models evaluated on SCIERC are
provided in Table 7. The BERT-Prompt model trained on CORE outperforms BERT-
Prompt models trained on FewRel and TACRED by 7.14% and 2.25%, respectively.

| CORE | FewRel | TACRED
---|---|---|---
CORE | - | 11.75 % | 20.72 %
FewRel | 53.41 % | - | 37.90 %
TACRED | 44.85 % | 15.12 % | -

Table 8: Average performance decrease between best performing in-domain models
(columns) and out-of-domain models (rows).

In summary, our findings suggest that out-of-domain models face challenges in
adapting to the CORE dataset. These models consistently exhibit lower
performance, both in terms of micro F1 and macro F1 scores. To provide an
overview of the performance differences, we illustrate the average performance
gaps between the best performing in-domain model and the out-of-domain models
across all evaluation metrics in Table 8. Notably, the models trained on CORE
show the best average out-of-domain performance, indicated by lower average
performance differences in comparison to in-domain models. Furthermore, our
experiments show that models trained on CORE outperform those trained on
FewRel when evaluated on datasets with a narrow domain focus, such as PubMed
and SCIERC. Several factors contribute to these performance differences.
First, the information richness in the training data allows the models to
focus on contextual nuances, reducing their reliance on superficial clues such
as relation-specific verbs. Second, the broader entity coverage, including
named entities, nouns, and pronouns, exposes models trained on CORE to more
linguistically diverse instances for the different relation types. These
observations align with the findings of Mao et al. (2022), who report
significant out-of-domain performance improvements in low-resource settings
when training on high-quality data. Considering the benchmark evaluation, the
prompt tuning approach emerges as the best-performing model.

###  5.3 Error analysis

We conducted a qualitative analysis to identify error patterns in the domain
adaptation setting. We used prediction examples from the BERT-Prompt models
trained on different domains, which achieved the highest average performance
in the previous section. From these models, we randomly sampled 100 erroneous
and 100 correct predictions, focusing on the 5-Way 5-Shot evaluation setting.

For models adapted to CORE (as shown in Table 3), we find that correct
predictions often included (i) verbs closely related to the target relation,
such as “sold”, “bought”, “merge” or (ii) combinations of entities that
revealed the relation type, such as “it is listed on the New York Stock
Exchange” or “the company was cited by the Food and Drug Administration”.
While there were no discernible patterns in the context, the domain-adapted
models performed better on NOTA instances compared to instances belonging to
the $N$ relations of interest that did not exhibit patterns (i) and (ii).
Selected examples are displayed below.

Erroneous predictions for out-of-domain models evaluated on CORE.

  * •

competitor of: In 2003, Toronto – dominion bank held talks to merge Waterhouse
E1 with E*TRADE, which would have created the second - largest discount broker
in the united states after Charles Schwab E2, but the two sides could not come
to an agreement over control of the merged entity.

  * •

product or service of: On July 4, 2011, Cope-Com E2 had converted their famed
and legendary Amiga E1 game called battle squadron to iOS devices titled
battle squadron one and published the game through apple app store.

Correct predictions for out-of-domain models evaluated on CORE.

  * •

listed on: Sopheon E1 is listed on the alternative investment market of the
London Stock Exchange E2.

  * •

acquired by: In 2008 the business was restructured with the creation of a new
Irish resident holding company, Charter International plc. E1 the company was
acquired by Colfax corporation E2, an American company, in January 2012.

Our analysis revealed that models trained on CORE recognize typical patterns
based on verbs and entity types, as well as more complex textual examples that
require contextual understanding. Such examples include instances without
verbs, instances involving multiple possible relations, or ambiguous entity
types, as illustrated in Figure 1.

##  6 Conclusion

In this paper, we introduced CORE, a few-shot RC dataset that specifically
focuses on company relations and business entities. To the best of our
knowledge, there are no other datasets that specifically target company
relations for few-shot RC, making CORE a valuable addition to the existing
benchmark resources. By extending the availability of datasets for few-shot
domain adaptation settings, CORE contributes towards more robust and realistic
RC evaluation results.

Despite the progress made in addressing challenges in RC datasets, such as
incorporating more realistic entity types and relation distributions, there is
still a need for further research and attention in adapting models to out-of-
domain datasets using diverse datasets and modeling approaches. Our findings
illustrate that models trained on other RC datasets often rely on shortcuts
that fail to generalize to CORE. Moreover, models trained on CORE display
superior out-of-domain performance. The information richness in the training
data enables models to focus on contextual nuances, reducing their reliance on
superficial clues such as relation-specific verbs. This underscores the
significance of high-quality datasets that include challenging entity types
and require a deeper contextual understanding. In conclusion, CORE serves as a
valuable resource for the few-shot domain adaptation setting in RC. It
highlights the importance of incorporating diverse and challenging datasets
and encourages future research efforts to focus on improving model adaptation
to out-of-domain data. By addressing these challenges, we can advance the
development of more effective and robust RC models.

## Limitations

CORE instances are solely extracted from company Wikipedia pages, which
introduces a limitation in terms of lexical similarity within the contextual
information. To address this constraint, diversifying the dataset by
incorporating company relations from various data sources, such as business
news, would introduce linguistic variety and establish a more rigorous
benchmark for evaluation.

The dataset’s practical applications are restricted to low-resource settings
due to the limited number of instances and relations it contains. Expanding
the dataset by including more instances and relationships incurs substantial
costs, as it necessitates human annotation efforts. Out of the 14,000
annotated candidate instances, only 30% contain any of the predefined
relations. However, the current inclusion of the “undefined” category fails to
accurately reflect the distribution of invalid candidate instances. This
discrepancy primarily arises from the absence of entities within the context
or entities that fall outside the scope of business entities.

Another limitation lies in the dataset being monolingual, as it is currently
limited to English. The transfer of annotations from the English Wikipedia
corpus to other languages would require additional human supervision and
allocation of resources.

## References

  * Alt et al. (2020) Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020.  TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1558–1569. Association for Computational Linguistics.
  * Baldini Soares et al. (2019) Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019\.  Matching the Blanks: Distributional Similarity for Relation Learning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2895–2905, Florence, Italy. Association for Computational Linguistics.
  * Bassignana and Plank (2022) Elisa Bassignana and Barbara Plank. 2022.  CrossRE: A cross-domain dataset for relation extraction.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 3592–3604, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_ , page 4171–4186.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making Pre-trained Language Models Better Few-shot Learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830, Online. Association for Computational Linguistics.
  * Gao et al. (2019) Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019\.  FewRel 2.0: Towards More Challenging Few-Shot Relation Classification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6250–6255, Hong Kong, China. Association for Computational Linguistics.
  * Han et al. (2021a) Chengcheng Han, Zeqiu Fan, Dongxiang Zhang, Minghui Qiu, Ming Gao, and Aoying Zhou. 2021a.  Meta-learning adversarial domain adaptation network for few-shot text classification.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , page 1664–1673, Online. Association for Computational Linguistics.
  * Han et al. (2021b) Jiale Han, Bo Cheng, and Wei Lu. 2021b.  Exploring task difficulty for few-shot relation extraction.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2605–2616, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Han et al. (2022) Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2022.  Ptr: Prompt tuning with rules for text classification.  _AI Open_ , 3:182–192.
  * Han et al. (2018) Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018.  FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4803–4809, Brussels, Belgium. Association for Computational Linguistics.
  * Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear.
  * Khaldi et al. (2022) Hadjer Khaldi, Farah Benamara, Camille Pradel, Grégoire Sigel, and Nathalie Aussenac-Gilles. 2022.  How’s business going worldwide? a multilingual annotated corpus for business relation extraction.  In _Proceedings of the 13th Conference on Language Resources and Evaluation_ , Marseille.
  * Lehmann et al. (2015) Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. 2015.  Dbpedia - a large-scale, multilingual knowledge base extracted from wikipedia. _Semantic Web_ , 6(2):167–195.
  * Liu et al. (2014) Yang Liu, Kang Liu, Liheng Xu, and Jun Zhao. 2014.  Exploring fine-grained entity type constraints for distantly supervised relation extraction.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 2107–2116, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _arXiv:1907.11692 [cs]_.  ArXiv: 1907.11692.
  * Luan et al. (2018) Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018.  Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3219–3232, Brussels, Belgium. Association for Computational Linguistics.
  * Lyu and Chen (2021) Shengfei Lyu and Huanhuan Chen. 2021.  Relation classification with entity type restriction.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , page 390–395, Online. Association for Computational Linguistics.
  * Mao et al. (2022) Yuning Mao, Ming Zhong, and Jiawei Han. 2022.  CiteSum: Citation text-guided scientific extreme summarization and domain adaptation with limited supervision.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10922–10935, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Mukherjee et al. (2021) Subhabrata Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Christopher Meek, Ahmed Hassan Awadallah, and Jianfeng Gao. 2021.  Clues: Few-shot learning evaluation in natural language understanding.  _35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks_.
  * Sabo et al. (2021) Ofer Sabo, Yanai Elazar, Yoav Goldberg, and Ido Dagan. 2021.  Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes.  _Transactions of the Association for Computational Linguistics_ , 9:691–706.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , page 255–269, Online. Association for Computational Linguistics.
  * Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard Zemel. 2017.  Prototypical Networks for Few-shot Learning.  _Advances in Neural Information Processing Systems_ , page 11.
  * Wu and Shi (2022) Hui Wu and Xiaodong Shi. 2022.  Adversarial soft prompt tuning for cross-domain sentiment analysis.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2438–2447, Dublin, Ireland. Association for Computational Linguistics.
  * Zhang et al. (2021) Jiawen Zhang, Jiaqi Zhu, Yi Yang, Wandong Shi, Congcong Zhang, and Hongan Wang. 2021\.  Knowledge-enhanced domain adaptation in few-shot relation classification.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, KDD ’21, page 2183–2191, New York, NY, USA. Association for Computing Machinery.
  * Zhang et al. (2017) Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017.  Position-aware Attention and Supervised Data Improve Slot Filling.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 35–45, Copenhagen, Denmark. Association for Computational Linguistics.
  * Zhenzhen et al. (2022) Li Zhenzhen, Yuyang Zhang, Jian-Yun Nie, and Dongsheng Li. 2022.  Improving few-shot relation classification by prototypical representation learning with definition text.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 454–464, Seattle, United States. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Relations

In the following, we provide textual descriptions of the relations included in
the dataset. The relations and their descriptions were predefined by the
authors.

  * •

Acquired by: $e2$ purchases controlling stake in $e1$. The relation is
directed. The inverted relation is best described by the same relation type.

  * •

Brand of: $e2$ offers products or services of $e1$ (Brand). The relation is
directed. The inverted relation is best described by the same relation type.

  * •

Client of: $e1$ uses (and presumably pays for) products or services offered by
$e2$. The relation is directed. The inverted relation is best described by
“Supplier of”.

  * •

Collaboration: $e1$ and $e2$ collaborate in (parts of their) business
activities. The relation is undirected.

  * •

Competitor of: $e1$ competes for resources with $e2$. The relation is
undirected.

  * •

Merged with: $e1$ and $e2$ merged (parts of) their business activities. The
relation is undirected.

  * •

Product or service of: $e1$ is offered for commercial distribution by $e2$.
The relation is directed. The inverted relation is best described by the same
relation type.

  * •

Regulated by: $e2$ regulates (parts of) the business activity of $e1$. The
relation is directed. The inverted relation is best described by the same
relation type.

  * •

Shareholder of: $e1$ owns shares in $e2$. The relation is directed. The
inverted relation is best described by the same relation type.

  * •

Subsidiary of: $e2$ legally owns $e1$. The relation is directed. The inverted
relation is best described by “Parent of”.

  * •

Traded on: Shares of $e1$ are listed on $e2$ (Stock exchange). The relation is
directed. The inverted relation is best described by “lists”.

  * •

undefined: $e1$ and $e2$ are business entities. The relation between the
entities cannot be described by any other defined relation. The relation can
be directed or undirected, however directed relations are not annotated as
such.

###  A.2 Dataset Analysis

In contrast to the CORE relation “subsidiary of” and “acquired by”, the FewRel
relation P355 (has subsidiary) encompasses non-business entities and offers a
broader interpretation of the term “subsidiary” beyond the corporate parent or
subsidiary relationship.

  * •

has subsidiary: The 2017–18 UMass E1 Minutewomen basketball team represents
the University of Massachusetts Amherst E2 during the 2017–18 college
basketball season.

  * •

has subsidiary: Davidson County is also served by Davidson County Community
College E2, a comprehensive community college that is a member school of the
North Carolina Community College System E1.

The FewRel relation P127 (owned by) exhibits partial overlap with the CORE
relations “subsidiary of” and “acquired by”. However, it introduces a
different perspective on ownership, encompassing a range of interpretations
that extend beyond conventional notions of mere possession, including the
notion of being a constituent part of an entity, physical location, and even
physical control.

  * •

owned by: The school is located in the city of Mannheim, Baden - Württemberg
E2 in Germany at Mannheim Palace E1, one of the largest baroque castles in
Europe.

  * •

owned by: After they returned to the Polanski E2 residence on Cielo Drive E1,
Patricia Krenwinkel, Susan Atkins and Charles “Tex” Watson entered the home.

###  A.3 Model training

The model parameters are shown in Table 9. We adjust the context length
according to the dataset and model. All models were trained on a NVIDIA RTX
A6000 48 GB GPU. Due to memory limitations when training the model variants,
we train BERT-Pair models with half-precision.

Parameter | Values
---|---
Encoder | BERT-Base
Context length | {256, 512}
Training episodes | 22,500
Learning Rate | 2e-5
Batch Size | 64
Optimizer | Adam
Table 9: Overview training parameters

###  A.4 Prompt tuning

We apply a heuristic approach in creating the template $\mathcal{T}$ and
vocabulary mapping $\mathcal{R}\mapsto\mathcal{V}_{\mathcal{M}}$ used in the
BERT-Prompt model. The template was selected by the authors and was applied
consistently across datasets, as follows: “[TEXT]. [E1] is [MASK] by [E2].”.
Similarly, the tokens mapped to the target relations for each benchmarked
dataset are selected by the authors. Prompt engineering, including automated
search procedures for optimal templates and vocabulary mappings, remains an
active research area. Acknowledging that heuristic approaches are suboptimal,
we encourage further research to improve the prompt tuning approach. Our
template and mapping files are available at
https://github.com/pnborchert/CORE.

###  A.5 Annotation Interface

The annotation platform provided clear instructions and mandatory tutorials to
guide the annotators throughout the annotation process. These instructions
included detailed explanations of the annotation task and interface, which was
designed to be intuitive and efficient. Each component of the interface was
explained in detail to ensure that annotators were able to use it effectively.
The following screenshots illustrate the interface provided to the annotators.
Figures 5 and 6 displays the relation annotation task, which included entity
types, relation directions, and relevant context. Figure 7 shows the entity
position annotation step, which verified the relation annotation completed in
the previous step. Both of these steps were completed by different annotators.

Figure 4: Annotation Interface: Instructions. Figure 5: Annotation Interface:
Relation type annotation. Figure 6: Annotation Interface: Relation type
annotation. Figure 7: Annotation Interface: Position annotation and
validation.


missing

# Prompt-based mental health screening from social media text

Wesley Ramos dos Santos1
Ivandré Paraboni1

###### Abstract

This article presents a method for prompt-based mental health screening from a
large and noisy dataset of social media text. Our method uses GPT 3.5.
prompting to distinguish publications that may be more relevant to the task,
and then uses a straightforward bag-of-words text classifier to predict actual
user labels. Results are found to be on pair with a BERT mixture of experts
classifier, and incurring only a fraction of its computational costs.

##  1 Introduction

Textual representation models used in NLP applications have changed
dramatically in recent years, moving on from simple token counts (e.g., bag-
of-words) to transformer-based and large language models (LLMs) such as
GPT111https://platform.openai.com/docs/models and
Bloom222https://huggingface.co/bigscience/bloom, among many others. In
particular, LLMs have successfully circumvented the need for labelled training
data entirely, in so-called prompt-based methods that are now mainstream in
the field, and have been shown to obtain higher accuracy in a wide range of
tasks.

Despite its potential to reduce training costs, however, LLM prompting still
requires careful consideration in applications involving large or noisy data,
as it is often the case of social media. In particular, we notice that social
media data may be available as a long history of publications, often
stretching over thousands of posts, and yet many or most may be unrelated to
the underlying task. This may be the case, for instance, when screening for
user-level information such as mental health statuses (e.g., related to
depression), a task that may challenge supervised and prompt-based methods
alike.

Based on these observations, this work investigates the use of LLM prompting
as an aid to mental health screening from social media text. Taking the issue
of depression detection from Brazilian Twitter timelines as a case study, we
propose a method that combines GPT prompting and standard bag-of-words
classification for fast, computationally inexpensive results, and which is
evaluated against SOTA results obtained by BERT mixture of experts [dos Santos
et al. 2023b].

##  2 Related work

Depression detection may be seen as an instance of author profiling [Rangel et
al. 2020, Flores et al. 2022, Pavan et al. 2023] from social media text.
Recent studies in the field are summarised in Table 1, categorised by text
genre (Reddit, Twitter), text model (b=bag of words, BERT, e=embeddings,
s=sentiment), and methods.

Table 1: Depression prediction from text data. Study | Genre | Text model | Methods
---|---|---|---
[Cohan et al. 2018] | reddit | b,e | FastText
[Aragón et al. 2019] | reddit | s | SVM
[Burdisso et al. 2020] | reddit | b | SS3
[Lin et al. 2020] | twitter | e | CNN
[Souza et al. 2020] | reddit | e | LSTM
[Souza et al. 2021] | reddit | e | LSTM+CNN
[Ansari and Ji 2022] | reddit,twitter | e,s | LR+LSTM
[dos Santos et al. 2023a] | twitter | BERT | Bi-LSTM
[dos Santos et al. 2023b] | twitter | BERT | mixture of experts

We notice a slight predominance of works based on Reddit. This may be
explained by the greater ease of access and reuse of this type of data for
research purposes, which is far more restricted in the case of Twitter/X.
Reddit publications are taken as the basis for some of the best-known datasets
available for the English language, including the SMHD [Cohan et al. 2018] and
eRisk [Parapar et al. 2022] corpora.

As for the kinds of textual model under consideration, Table 1 reflects the
natural evolution of the NLP and related fields, with an initial prevalence of
bag-of-words models and feature engineering, and its gradual replacement by
models based on word embeddings and, more recently, transformers. With regard
to the computational methods used, a similar trajectory is generally
observable, with the use of traditional (e.g., linear) classifiers being
gradually replaced by sequence classification methods based on deep learning,
including the more recent use of transformer-based architectures. In both
cases - text representation and methods - we notice also that better results
are usually accompanied by an increase in computational costs.

Finally, we notice that, with the exception of the work based on the
SetembroBR [dos Santos et al. 2023a, dos Santos et al. 2020] corpus to be used
in the present work, all of the above studies are devoted to the English
language, and none make use of prompt-based methods to query a LLM for
depression directly.

##  3 Method

As in [Cohan et al. 2018, Parapar et al. 2022] and others, our approach to
depression detection relies on a dataset of social media (in our case,
Twitter/X) texts that have been published by either individuals who self-
reported a depression diagnosis, or by a random (control) group. Thus, the
task at hand represents a binary classification problem intended to
distinguish individuals who will most likely receive a depression diagnosis
(called Diagnosed class) in the future from the general population (called
Control class), and not to distinguish depressed from non-depressed
individuals per se.

###  3.1 Data

We use of the depression portion of the SetembroBR corpus [dos Santos et al.
2023a], a collection of Twitter timelines published by Diagnosed and Control
individuals in which only the data prior to the diagnosis date is kept. As in
other language resources of this kind, the Control (i.e., random) subset is
designed so as to be seven times larger than the Diagnosed group, making a
heavily imbalanced classification task. Table 2 presents data descriptive
statistics.

Table 2: SetembroBR depression corpus descriptive statistics. Statistics | Diagnosed | Control | Overall
---|---|---|---
Users (timelines) | 1,684 | 11,788 | 13,472
Words (million) | 29.32 | 201.94 | 231,26
Publications (million) | 2.43 | 16.99 | 19,42

###  3.2 Approach

Screening for depression from social media gives rise to the questions of how
to handle a large number of publications, most of which unlikely to be
relevant to the task. To this end, we envisaged a prompt-based approach called
Prompt.Bow that relies on GPT 3.5 to enrich an otherwise standard text
classifier.

First, GPT 3.5. prompting is used to assess every tweet in our data according
to relevance for mental health prediction. The (English-translated) prompt in
question, adapted from the clinical description of depression in [American
Psychiatric Association 2013], is shown in Figure 1.

Figure 1: Prompt instruction to GPT.

By submitting this prompt to each of the 19.42 million tweets in the corpus,
the data was categorised as having high (1), medium (2) or low (3) relevance
to mental health. Table shows the label distribution across the training
portion of the corpus.

Table 3: Training data distribution according to relevance for depression. | Diagnosed class | Control class
---|---|---
Relevance | Tweets | % | Tokens | % | Tweets | % | Tokens | %
high | 39k | 2.0% | 674k | 5.4% | 153k | 1.1% | 2,372k | 3.0%
medium | 180k | 9.3% | 3,663k | 29.4% | 1,031k | 7.6% | 20,566k | 25.7%
low | 1,727k | 88.7% | 8,122k | 65.2% | 12,439k | 91.3% | 56,971k | 71.3%

As expected, most publications (65.2% in the Diagnosed class, and 71.3% in the
Control class) are deemed of low relevance for mental health prediction
according to the prompted model. On the other hand, highly relevant
publications are relatively rare (5.4% in the Diagnosed class, and 3.0% in the
Control class).

More importantly, however, when prompting an LLM for mental health we are
largely focusing on semantics, that is, on symptoms and other well-known
clinical signs of depression, but this is not to say that other factors can or
should be overlooked. For instance, our current prompt allows the model to
pinpoint a wide range of publications that may suggest, e.g., eating disorders
or negative language use, both of which known to be related to depression, cf.
[American Psychiatric Association 2013], but it does not explicitly account
more fine-grained linguistic indicators of depression such as the use of first
person pronouns [Trifu et al. 2017], absolute terms [Al-Mosaiwi and Johnstone
2018] and others. These indicators, which may be present in any publication of
low or high relevance to depression alike, are also important predictors of
mental health statuses, and no data point should in principle be discarded
solely based on the LLM output.

As a means to keep the full train data available to the classifier while
distinguishing between more and less relevant messages (which clearly show
different distributions across classes in Table 3), Prompt.Bow uses the
category labels provided by the LLM to split the training data into high,
medium, and low relevance subsets, from which we create three individual bag-
of-words vectors to be concatenated as discussed below.

Each of the three vectors consists of TF-IDF counts of texts labelled as
high/medium/low by the GPT prompt method, and are further reduced using
univariate feature selection using F1 as a score function. The final ‘high’
and ‘medium’ vectors were reduced to $k=$ 6000 features each, and the final
‘low’ vector was reduced to $k=$ 3000 features, all of which concatenated as a
single 15000-word vector.

In addition to distinguishing between high-, medium- and low-relevance
messages, we use the information provided by the LLM ALSO to HELP capture
message order, the underlying assumption being that certain patterns (e.g., a
series of consecutive ‘highly relevant’ messages) may be indicative of
depression. To this end, we created a bigram model of high/medium/low labels
only, reduced to its $k=$ 40000 most relevant bigram features by performing F1
univariate feature selection, and then appended to the previous 15000-word
vector.

The resulting vector is taken as the input to a standard logistic regression
classifier. This choice is motivated by the observation that much of the deep
(e.g., semantics-driven) language processing had already been performed by the
LLM, and that a simple text classifier should suffice for user label
prediction.

##  4 Evaluation

Table 4 presents the results of the present Prompt.Bow model alongside the
results reported in [dos Santos et al. 2023b] for BERT mixture of experts
using BERTabaporu [da Costa et al. 2023], which currently stands as the SOTA
for the present setting. In both cases, results are based on the standard test
portion of the corpus.

Table 4: Classification results. Model | Precision | Recall | F1
---|---|---|---
Prompt.BoW | 0.64 | 0.72 | 0.66
BERT.MoE | 0.64 | 0.67 | 0.65

Table 4 shows that results remain close, with a small advantage for the
Prompt.Bow approach over BERT mixture of experts. However, by relying on an
input provided by the pre-trained LLM, we notice that these results were
obtained by using a computationally inexpensive classifier model, which
represents a stark contrasts to computationally-intensive BERT mixture of
experts supervision.

##  5 Final remarks

This work presented an experiment in prompt-based mental health screening from
a large, noisy corpus of social media publications that relies on LLM
prompting to distinguish publications that may be more relevant to the task,
and then uses a straightforward bag-of-words text classifier to predict actual
user labels. This was shown to obtain competitive results if compared to a
BERT-based classifier architecture that represents the SOTA in the present
setting, but incurring only a fraction of its computational costs.

As future work, we intend to refine the present method by using the formal
definition of depression in [American Psychiatric Association 2013] as a
prompt, and by fully integrating symptoms and linguistic indicators within a
neural architecture for both depression and anxiety disorder prediction.

##  6 Acknowledgements

The present work has been financed by the São Paulo Research Foundation
(FAPESP grant #2021/08213-0). The first author has been supported by the
Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) -
Finance Code 001 - grant # 88887.475847/2020-00. This work was carried out at
the Center for Artificial Intelligence (C4AI-USP), with support by the São
Paulo Research Foundation (FAPESP grant #2019/07665-4) and by the IBM
Corporation.

## References

  * [Al-Mosaiwi and Johnstone 2018] Al-Mosaiwi, M. and Johnstone, T. (2018).  In an absolute state: Elevated use of absolutist words is a marker specific to anxiety, depression, and suicidal ideation.  Clinical Psychological Science, 6(4):529–542.
  * [American Psychiatric Association 2013] American Psychiatric Association (2013).  Diagnostic and Statistical Manual of Mental Disorders 5th edition.  American Psychiatric Association, Arlington, VA.
  * [Ansari and Ji 2022] Ansari, L. and Ji, S. (2022).  Ensemble hybrid learning methods for automated depression detection.  IEEE Transactions on computational Social Systems.
  * [Aragón et al. 2019] Aragón, M. E., López-Monroy, A. P., González-Gurrola, L. C., and y Gómez, M. M. (2019).  Detecting depression in social media using fine-grained emotions.  In NAACL-2019 Proceedings, pages 1481–1486, Minneapolis, USA. Assoc for Comp Ling.
  * [Burdisso et al. 2020] Burdisso, S. G., Errecalde, M., and y Gómez, M. M. (2020).  t-SS3: a text classifier with dynamic n-grams for early risk detection over text streams.  Pattern Recognition Letters, 138:130–137.
  * [Cohan et al. 2018] Cohan, A., Desmet, B., Yates, A., Soldaini, L., MacAvaney, S., and v Goharian (2018).  SMHD: a large-scale resource for exploring online language usage for multiple mental health conditions.  In COLING-2018, pages 1485–1497, Santa Fe, USA.
  * [da Costa et al. 2023] da Costa, P. B., Pavan, M. C., dos Santos, W. R., da Silva, S. C., and Paraboni, I. (2023).  BERTabaporu: assessing a genre-specific language model for Portuguese NLP.  In Recents Advances in Natural Language Processing (RANLP-2023), pages 217–223, Varna, Bulgaria.
  * [dos Santos et al. 2023a] dos Santos, W. R., de Oliveira, R. L., and Paraboni, I. (2023a).  SetembroBR: a social media corpus for depression and anxiety disorder prediction.  Language Resources and Evaluation.
  * [dos Santos et al. 2020] dos Santos, W. R., Funabashi, A. M. M., and Paraboni, I. (2020).  Searching Brazilian Twitter for signs of mental health issues.  In 12th International Conference on Language Resources and Evaluation (LREC-2020), pages 6113–6119, Marseille, France. ELRA.
  * [dos Santos et al. 2023b] dos Santos, W. R., Yoon, S., and Paraboni, I. (2023b).  Mental health prediction from social media text using mixture of experts.  IEEE Latin America Tr., 21(6):723–729.
  * [Flores et al. 2022] Flores, A. M., Pavan, M. C., and Paraboni, I. (2022).  User profiling and satisfaction inference in public information access services.  Journal of Intelligent Information Systems, 58(1):67–89.
  * [Lin et al. 2020] Lin, C., Hu, P., Su, H., Li, S., Mei, J., Zhou, J., and Leung, H. (2020).  SenseMood: Depression Detection on Social Media, pages 407–411.  Association for Computing Machinery, New York, USA.
  * [Parapar et al. 2022] Parapar, J., Martin-Rodilla, P., Losada, D. E., and Crestani, F. (2022).  Overview of eRisk 2022: Early Risk Prediction on the Internet.  In Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, pages 821–850, Bologna, Italy.
  * [Pavan et al. 2023] Pavan, M. C., dos Santos, V. G., Lan, A. G. J., ao Trevisan Martins, J., dos Santos, W. R., Deutsch, C., da Costa, P. B., Hsieh, F. C., and Paraboni, I. (2023).  Morality classification in natural language text.  IEEE transactions on Affective Computing, 14(1):857–863.
  * [Rangel et al. 2020] Rangel, F., Rosso, P., Zaghouani, W., and Charfi, A. (2020).  Fine-grained analysis of language varieties and demographics.  Natural Language Engineering, page 1–21.
  * [Souza et al. 2020] Souza, V., Nobre, J., and Becker, K. (2020).  Characterization of anxiety, depression, and their comorbidity from texts of social networks.  In SBBD-2020, pages 121–132, Porto Alegre, Brazil. SBC.
  * [Souza et al. 2021] Souza, V., Nobre, J., and Becker, K. (2021).  A deep learning ensemble to classify anxiety, depression, and their comorbidity from texts of social networks.  Journal of Information and Data Management, 12(3):306–325.
  * [Trifu et al. 2017] Trifu, R., Nemes, B., Bodea-Hategan, C., and Cozman, D. (2017).  Linguistic indicators of language in major depressive disorder (MDD). An evidence based research.  Journal of Evidence-Based Psychotherapies, 17:105–128.