# An Overview of Distant Supervision for Relation Extraction with a Focus on
Denoising and Pre-training Methods

William P Hogan
Department of Computer Science & Engineering
University of California, San Diego

###### Abstract

Relation Extraction (RE) is a foundational task of natural language
processing. RE seeks to transform raw, unstructured text into structured
knowledge by identifying relational information between entity pairs found in
text. RE has numerous uses, such as knowledge graph completion, text
summarization, question-answering, and search querying. The history of RE
methods can be roughly organized into four phases: pattern-based RE,
statistical-based RE, neural-based RE, and large language model-based RE. This
survey begins with an overview of a few exemplary works in the earlier phases
of RE, highlighting limitations and shortcomings to contextualize progress.
Next, we review popular benchmarks and critically examine metrics used to
assess RE performance. We then discuss distant supervision, a paradigm that
has shaped the development of modern RE methods. Lastly, we review recent RE
works focusing on denoising and pre-training methods.

##  1 Introduction

Relation extraction (RE), a subtask of information extraction, is a
foundational task in natural language processing (NLP). The RE task is to
determine a relationship between two distinct entities from text, producing
fact triples in the form [head, relation, tail] or, as referred to in some
works, [subject, predicate, object]. For example, after reading the Wikipedia
page on Noam Chomsky, we learn that Noam was born in Philadelphia,
Pennsylvania, which corresponds to the fact triple [Noam Chomsky, born in,
Philadelphia]. Fact triples are foundational to human knowledge and play a key
role in many downstream NLP tasks such as question-answering, search queries,
and knowledge-graph completion Xu et al. (2016); Lin et al. (2015); Li et al.
(2014).

Figure 1: A sample of relation labels generated by distant supervision. A
knowledge graph, here a small snippet from WikiData Vrandečić and Krötzsch
(2014), is paired with sentences to label instances of relations linking an
entity pair.

Distant supervision for relation extraction is a method that pairs a knowledge
graph—a graph of entities connected by edges labeled with relation
classes—with an unstructured corpus to generate labeled data automatically
Mintz et al. (2009). First, entities from the knowledge graph are identified
in the text, and then the following assumption is made: all sentences
containing an entity pair express the corresponding relation class, as
determined by the accompanying knowledge graph. Figure 1 provides an example
of automatically generated relation labels pairing a knowledge graph, namely
WikiData Vrandečić and Krötzsch (2014), with raw sentences.

Formally, the problem statement for distantly supervised relation extraction
is as follows: given knowledge graph $\mathcal{G}$ and text corpus
$\mathcal{C}$, sentences $s$ from $\mathcal{C}$ that contain two distinct
entities, $({e^{i}}_{1},{e^{i}}_{2})$, that are linked via relationship
$r^{i}$ as determined by $\mathcal{G}$, where
${e^{i}}_{1},{e^{i}}_{2},r^{i}\in\mathcal{G}$, predict the relationship
$r^{i}$ that is expressed in sentence $s^{i}$, forming a fact triple
$t=\\{({e^{i}}_{1},{r^{i}},{e^{i}}_{2})\\}$.

This survey is organized as follows:

  * •

In Section 2, we conduct a high-level overview of relation extraction. We
trace the history of RE methods and highlight limitations and shortcomings to
contextual progress. We review both how and why distant supervision for RE was
developed.

  * •

In Section 3, we review popular RE datasets and critically examine evaluation
metrics used to assess RE performance.

  * •

In Section 4, we review recent methods used for distantly supervised RE. We
focus on the two primary methods characteristic of distantly supervised RE:
denoising methods and pre-training methods.

##  2 Background

The history of RE methods can be roughly organized into four phases: (1)
pattern-based RE, (2) statistical-based RE, (3) neural-based RE, and (4) large
language model-based RE. We provide an approximate year when a particular
phase begins, but we do not provide an end year as many of the methods from
earlier RE phases continue into subsequent phases, creating a rich blend of
techniques and no clear end to any particular phase.

###  2.1 Pattern-based RE ($\sim$1970+)

As the name suggests, pattern-based RE is the development of algorithms to
learn linguistic patterns to extract relation information from text. Pattern-
based RE methods typically use seed words or phrases combined with knowledge
sources such as a part–of–speech tagger and semantic class information to
identify patterns within text relevant to relation extraction.

In Califf and Mooney (1997), an exemplary work of pattern-based RE, the
authors propose the “Robust Automated Production of Information Extraction
Rules” (RAPIER) algorithm. RAPIER is a form-filling algorithm that uses form
fields as seed words to extract relational information. The algorithm expands
the seed words to include all synonyms via the synsets provided by WordNet
Fellbaum (1998), a large lexical database with extensive synsets for English
words. Words within sentences are tagged with a part-of-speech (POS) tagger
and RAPIER then attempts to learn patterns of POS tags before and after seed
words within a sentence. The authors develop and evaluate RAPIER on a small
dataset of job listings, extracting facts such as job location, required
skills, and salary.

Pattern-based methods typically involve bespoke algorithms developed and
refined within a specific domain, and RAPIER is no different. Because of this,
pattern-based methods often fail to generalize to out-of-domain data.
Moreover, while pattern-based methods typically achieve high precision on
relation extraction benchmarks, they suffer from low recall. These limitations
motivated the development of statistical-based relation extraction methods.

###  2.2 Statistical-based RE ($\sim$2004+)

In the early 2000s, advanced statistical-based RE methods gained popularity.
Statistical-based RE looked to leverage advances in machine learning to
extract relations from text. Hand-crafted textual features, such as dependency
trees and part-of-speech tags, were typically used to train logistic
regression classifiers.

Statistical-based machine learning methods require large datasets for training
and evaluation. However, manually labeling relations in text is both expensive
and time-consuming. Dataset creators must develop annotation guidelines to
ensure consistency and pay and train human annotators (e.g., via Mechanical
Turk Buhrmester et al. (2011)) to annotate data.

To address the data scarcity issue, Mintz et al. (2009) released a pivotal
work titled “Distant supervision for relation extraction without labeled
data.” In it, the authors coin the term distant supervision for relation
extraction and define it as the assumption that any sentence containing an
entity pair may express a relationship between that entity pair as determined
by an accompanying knowledge graph. This simple but powerful assumption
unlocks the ability to automatically generate the large amounts of training
data needed for advanced statistical RE models.

In the original distant supervision for RE work, Mintz et al. pair the
Freebase Bollacker et al. (2008) knowledge graph with text from Wikipedia
(English)111https://en.wikipedia.org/ and, in doing so, generated a dataset
consisting of 1.8 million instances of 102 relation classes connecting 940,000
entities. They trained a multi-class logistic regression classifier with
numerous hand-crafted lexical and semantic features using this data. For
lexical features, they use the sequence of words between two entities, POS
tags, a binary flag indicating which entity in a pair is mentioned first in
the sentence, a window of $k$ words and POS tags to the left of the first
entity as well as words and POS tags to the right of the second entity. For
semantic features, they include word dependencies via the dependency parser
MINIPAR Lin (2003) and named entity tags via the Stanford four-class named
entity tagger Finkel and Manning (2009). With the help of human annotators,
they observed that their logistic regression classifier, leveraging distant
supervision and a combination of hand-crafted features, was able to extract
fact triples with decent precision (0.67 on the highest-ranked 1000 results
per relation).

Notably, most of the early distantly supervised RE work focused on intra-
sentence relations (i.e., “sentence-level” relations). However, Quirk and Poon
(2017) observed that many relations within a document were expressed across
multiple sentences (i.e., cross-sentence, inter-sentence, or “document-level”
relations). To extract document-level relations, the authors proposed a
statistical-based RE method named DIstant Supervision for Cross-sentence
Relation EXtraction (DISCREX). DISCREX first labels words with their
corresponding lemma and part-of-speech. It then forms a document graph of
words by leveraging Stanford’s dependency parsing tool which outputs
dependency paths between words in a sentence. The model searches within the
document graph to find the shortest path between two entity mentions, ignoring
sentence boundaries. The intuition is that a relationship between an entity
pair is more likely to be expressed when entities are closer together,
regardless of sentence boundaries. The authors train DISCREX to detect an
association between an entity pair via binary logistic regression. DISCREX is
notable because it is an early model focusing on document-level RE. As
discussed in Section 3, document-level RE has arguably become the primary
focus for current RE methods Soares et al. (2019); Peng et al. (2020); Qin et
al. (2021).

###  2.3 Neural-based RE ($\sim$2010+)

In the early 2010s, with help from large automatically labeled datasets
enabled by distant supervision for RE and the advance of neural networks, RE
entered its third phase—neural-based RE. Neural-based RE departs from
statistical-based RE in two key ways: (1) by using a neural network instead of
logistic regression, and (2) by replacing hand-crafted textual features with
textual features learned via a neural network (e.g., Word2Vec Mikolov et al.
(2013) and GloVe Pennington et al. (2014)).

Neural RE models come in a variety of architectures, such as the recurrent
neural network (RNN) Zhang and Wang (2015), long-short term memory (LSTM)
network Peng et al. (2017); Miwa and Bansal (2016), and the convolutional
neural network (CNN)Zeng et al. (2015). During this era, RE methods enjoyed a
significant boost in performance.

Relationships in text may be expressed within a long sentence or across
multiple sentences and, while high-performing, many early neural-based models
struggled to learn such long-range dependencies. To address this shortcoming,
Vaswani et al. (2017) introduced the transformer architecture, a seminal work
that transformed RE and NLP in general.

The transformer is similar in structure to other sequence transduction models
in that it consists of two modules: an encoder module and a decoder module.
The encoder takes an input sequence x and produces a dense representation z
that is fed to the decoder. The decoder then uses z to produce an output
sequence y. However, the transformer was unique in its use of stacked multi-
head attention functions in the encoder and decoder modules.

In the original transformer architecture, the encoder and decoder modules
consisted of six identical and serially-connected layers. Each layer contained
a multi-head attention function as well as a feed-forward network. The initial
encoder layer is fed an input sequence x, and, using multi-head attention, it
attends to all positions in that sequence—that is, it learns which parts of
the input sequence to focus on for a given task. Next, the result from the
multi-head attention function is sent through a feed-forward network and,
finally, to the next layer in the module. Subsequent layers repeat this
process using the previous layer’s output as their input. The transformer is
arguably the most impactful neural-based architecture for RE and NLP as it is
the primary architecture used in large pre-trained language models.

###  2.4 Large language model-based RE ($\sim$2019+)

The transformer architecture inspired the development of BERT Devlin et al.
(2019), a wildly effective pre-trained language model that produces word
embeddings richer and more informative than Word2Vec and GloVe. BERT’s success
inspired a slew of other pre-trained models, also known as foundation models
Bommasani et al. (2021), all rooted in the same two-step training paradigm—a
large, self-supervised pre-training followed by a smaller, supervised fine-
tuning. This training methodology brought RE into its fourth and current
phase.

By leveraging pre-trained language models, RE performance improved across all
benchmarks. At the time of writing, the top ten models on the DocRED dataset
Yao et al. (2019), the most challenging general domain RE benchmark, all
leverage BERT or a BERT relative (e.g., RoBERTa Liu et al.
(2019))222https://paperswithcode.com/sota/relation-extraction-on-docred. We
conduct a deeper dive into these leading models in Section 4.

##  3 Datasets & Benchmarks

This section briefly introduces some popular RE datasets and benchmarks used
in many of the works discussed throughout this survey. We also critically
examine the current evaluation metrics used to assess RE performance.

The development of relation extraction datasets mirrors the history of RE
development. First, small, human-labeled datasets were developed for pattern-
based methods, followed by larger manually and automatically labeled datasets
used to train statistical models and neural networks Riedel et al. (2010);
Zhang et al. (2017); Yao et al. (2019).

Dataset | RE Type | Data Creation Method | Instances | Classes
---|---|---|---|---
SemEval-2010 Task 8 | Sentence-level | Manual | 8.8k | 9
NYT10 | Sentence-level | Automatic | 21.8k | 53
TACRED | Sentence-level | Manual | 106k | 41
FewRel | Sentence-level | Manual | 70k | 100
BC5CDR | Doc-level | Manual | 3.1k | 1
DocRED | Doc-level | Auto + Manual | 1.56M | 96
Table 1: A sample of sentence-level and document-level RE datasets
highlighting the dataset creation method, size, and the number of relation
classes. Manually created datasets are created by human annotators, while
automatically generated datasets are created using distant supervision for
relation extraction.

RE datasets typically focus on one of two sub-tasks: (1) sentence-level RE or
(2) document-level RE:

1\. Sentence-level RE seeks to extract relationships between entity pairs
within the bounds of a single sentence (i.e., intra-sentence relationships).
Some popular sentence-level RE datasets include TACRED Zhang et al. (2017),
NTY10 Riedel et al. (2010), SemEval-2010 Task 8 Hendrickx et al. (2010), and
FewRel Han et al. (2018b); Gao et al. (2019).

Sentence-level RE is criticized for being overly simplistic since not all
relationships in text are expressed within a single sentence. Yao et al.
(2019) report that 40.7% of relationships in Wikipedia documents are expressed
across multiple sentences. That is, to identify 40.7% of instances of
relationships in Wikipedia, a human or machine learning model must comprehend
multiple sentences to determine a relationship between an entity pair.

2\. Document-level RE was developed to address the shortcoming of sentence-
level RE by extracting relations between entity pairs contained within a
document (i.e., both inter- and intra-sentence relationships). In the general
domain, the DocRED dataset Yao et al. (2019) is a popular document-level RE
dataset that pairs text from Wikipedia to entities and relations from the
WikiData knowledge graph Vrandečić and Krötzsch (2014). In the biomedical
domain, the BioCreative V CDR task corpus (BC5CDR) Li et al. (2016) is a
popular document-level dataset containing binary associations between chemical
and disease entities using text from PubMed Canese and Weis (2013).

Table 1 lists common RE datasets, noting the RE type (sentence-level versus
document-level), the creation method (manually versus automatically labeled),
dataset size, as well as the number of relation classes they contain, not
including “no relation.”

###  3.1 Evaluation

There are two common ways to evaluate a model’s ability to extract
relationships: corpus-based and instance-based.

Corpus-based evaluation, also known as “hold out” evaluation Mintz et al.
(2009), evaluates a model’s ability to predict an unseen, or “held out,” set
fact triples in a test corpus. It is more commonly used for distantly
supervised data that lacks gold labels. For corpus-based evaluation,
precision-recall (PR) curves, area under the PR curve (AUC), and precision@$k$
is reported. Precision@$k$ is generated by pooling all of the model’s inferred
predictions on a test set, sorting them by their softmax probabilities, and
then reporting the precision for the top $k$ predictions.

Instance-based evaluation assesses a model’s ability to predict each relation
instance in a test set. For this, standard precision, recall, and F1-Micro
score are used. Instance-based evaluation is commonly used on manually
annotated datasets where all instances of relations have gold labels.

####  3.1.1 Evaluation Limitations

Both corpus-based and instance-based evaluations have limitations. Corpus-
based evaluation suffers from false negatives since the held-out set of fact
triples, generated from the paired knowledge graph, may be incomplete.
Furthermore, in Gao et al. (2021), the authors manually examined distantly
supervised relation data and found that as many as 53% of the assigned labels
were incorrect. They argue that, with such a high percentage of incorrectly
labeled data, automatic corpus-based evaluation is not a representative metric
of model performance. They suggest that distantly supervised datasets must be
paired with and evaluated on manual labels.

One shortcoming of instance-based evaluation is that it rewards the
memorization of triples seen in training. Some benchmarks, such as those used
for DocRED, avoid this biasing by reporting an F1 score calculated only on
unseen triples in addition to an overall F1 score.

Another shortcoming of instance-based evaluation is that a model’s performance
is boosted by correctly predicting multiple instances of common relation
classes. This can obscure performance on rare, long-tail instances. In Gao et
al. (2021), the authors note that many RE datasets have long-tail
distributions of relation classes, making it possible for an RE model to
achieve high F1-Micro scores by performing well on common classes and poorly
on rare, long-tail classes. They compare a few leading RE models on the NTY10
data Riedel et al. (2010) and show almost no difference in performance on the
top four relation classes in the dataset. There are, however, significant
differences in performance on less-common relation classes. This is because
less-common classes have fewer supporting sentences, making contextual
learning difficult. In these cases, models that make predictions based on
shallow knowledge graph heuristics perform well. Yet, such models can produce
large amounts of false positives—predicting relationships in sentences without
supporting context. As such, Gao et al. suggest adopting F1-Macro scores, in
addition to the F1-Micro score, for instance-based evaluation to highlight
model performance on long-tail relation classes.

Using corpus-based and instance-based evaluation, neural models have
demonstrated superior performance, yet what they are actually learning remains
under-explored. For instance, it is unclear whether a model leans more on
entity mentions or supporting context to make its predictions. Ideally, a
model extracts relationships by understanding both the entity mentions and the
supporting context, but a recent work shows otherwise Peng et al. (2020).

In Peng et al. (2020), the authors use the TACRED dataset and create five
experimental settings designed to determine whether context or entity mentions
are more informative in leading RE models. In the first setting, “context and
entity mentions,” they train models using unaltered entity mentions and
context. This is the baseline setting used in almost all RE models. In the
second setting, “context and entity types,” they remove entity mentions from
sentences and replace them with the corresponding entity types. For instance,
the entity mention “Angela Davis…” is replaced with its entity type, namely
“[person].” This setting is designed to determine the impact of entity type
information in model performance relative to the default surface forms of
entity mentions. The third setting, “only context,” masks entity mentions
entirely and has the model make predictions based purely on context. The
fourth setting, “only entity mentions,” masks the context and only allows the
model to make predictions based on entity mentions. Finally, in the fifth
setting, “only entity types,” they mask context and replace entity mentions
with the corresponding entity types.

The best performance resulted from the “context and entity types” setting
where entities in a sentence were replaced by the corresponding entity types,
indicating that type information from entity mentions is more informative to
RE models than the entity mentions themselves. “Only context” outperformed
“only mentions” which shows that models learn more from context than entity
mentions. The authors conducted an error analysis on the “only context”
results and found that 43% of incorrectly predicted instances contained
sufficient context to make a correct prediction, highlighting that current
models can improve by better learning context.

Interestingly, while the “only context” setting outperformed “only mentions”
setting, the “only mentions” setting still achieved good overall RE
performance. That is, with no context at all, models could still predict
relations between entities. This highlights how pre-trained language models
contain some latent understanding of entities stored in their parameters. This
can be problematic—pre-trained models may be prone to making predictions based
on shallow heuristics, predicting a relationship between entities despite a
lack of supporting textual context. This leads to many false positives and
inflated performance on few-shot relation instances Peng et al. (2020), which
matches the observation made by Gao et al. (2021).

Considering these findings, the authors suggest that future RE works focus on
training models that prioritize learning relationships from context and reduce
biasing from shallow heuristics. Additionally, RE evaluation metrics may
benefit from reporting performance on instances with and without sufficient
context. This will help reveal which models effectively learn context versus
models that predict relations based on shallow knowledge graph heuristics.

##  4 Current Methods

This section delves into recent influential works on distant supervision for
relation extraction. We divide the discussion into two primary methods
characteristic of current distant supervision for RE works: denoising methods
and pre-training methods.

###  4.1 Denoising Methods

As discussed in Section 1, distant supervision for relation extraction pairs a
knowledge graph with raw textual data and assumes that any sentence that
contains two entities linked in the knowledge graph expresses the
corresponding relationship. However, using this method to automatically label
data produces a noisy training signal in the form of false positives since not
all sentences will express a relationship. For example, consider the sentences
in Figure 2. Using distant supervision for RE, each sentence is labeled as a
positive instance of the relationship “born in,” however, only the first
sentence properly expresses the relationship. As such, a large portion of the
work that leverages distant supervision focuses on developing advanced
denoising methods.

Figure 2: A sample of the wrong-labeling issue by automatically labeled
sentences using distant supervision for relation extraction. Only the first
sentence accurately expresses the labeled relationship “born in.”

Shortly after distantly supervised RE was proposed by Mintz et al. (2009),
Riedel et al. (2010) proposed “multi-instance learning” (MIL) as a more
relaxed version of the original distant supervision assumption designed to
better handle noisy labels. Instead of assuming every sentence containing an
entity pair expresses a relationship, multi-instance learning, as shown in
Figure 3, collects a group, or “bag,” of sentences that contain the same
entity pair and assumes that at least one sentence in the group of sentences
expresses the corresponding relationship. The intuition supporting multi-
instance learning is simple: when considering sentences in a corpus that share
the same entity pair, a group of those sentences are more likely to express a
relationship than any single sentence.

Figure 3: A sample of single-instance learning (left) and multi-instance
learning (right) proposed by Riedel et al. (2010). Multi-instance learning is
an effective denoising method for distantly supervised RE as a group of
sentences will more likely express a relationship than a single sentence. In
this example, head and tail entities are wrapped in special tokens (‘$’ and
‘^’, respectively), following a method popularized by Soares et al. (2019).

In Surdeanu et al. (2012), the authors expand on multi-instance learning. They
note that some entity pairs have multiple relationships; however, there is no
way to determine the best relationship label to apply when generating labels
using distant supervision. Analyzing the NTY10 dataset Riedel et al. (2010)
they report that multiple relationships connect 7.5% of entity pairs.

To address this issue, Surdeanu et al. propose a multi-instance, multi-label
model for RE (miml-re). miml-re models multiple relationships via an
Expectation Maximum (EM) algorithm that learns a single latent representation,
z, which expresses $k$ relation classes. They feed z into a multi-label
classifier that is trained to predict the relationship expressed between an
entity pair. The multi-label classifier can learn which relational classes can
and cannot jointly occur. For example, relations “capital of” and “contained”
can jointly occur within a single entity pair while “born in” and “spouse of”
cannot. Tested on NYT10 and compared to the original multi-instance model
proposed by Riedel et al. (2010), they report their miml-re model increases
precision by 2 to 15 points at the same recall point.

Riedel et al.’s multi-instance learning paradigm has inspired the development
of methods that rank or select sentences from a bag of sentences that best
represent a given relationship. For example, Zeng et al. (2015) propose a
method to score sentences within a bag based on their likelihood of expressing
a relationship. Lin et al. (2016) introduced an attention mechanism that
attends to relevant relational information within a bag of sentences. Advanced
attention mechanisms were featured in numerous subsequent works Luo et al.
(2017); Han et al. (2018a); Alt et al. (2019).

However, not all works reported increased performance from applying an
attention mechanism to a bag of sentences. On a distantly supervised dataset
constructed by pairing PubMed abstracts Canese and Weis (2013) with the UMLS
knowledge graph Bodenreider (2004), Amin et al. (2020) reported a drop in RE
performance when using selective attention compared to a simple average
pooling over a bag of sentences.

This phenomenon of poor results while using attention mechanisms with multi-
instance learning on distantly supervised datasets was recently explored by Hu
et al. (2021). They show experimentally that the performance of attention
mechanisms drops significantly compared to average pooling as the ratio of
noise in a dataset increases.

To conduct this experiment, the authors start with FewRel, a manually labeled
sentence-level RE dataset, and inject noise into the dataset by randomly
replacing entity pairs in a sentence with different entity pairs while keeping
the context unchanged. They use this method to automatically construct nine
datasets with varying ratios of noise.

They report that, using multi-instance learning, simple average pooling
outperforms attention mechanisms in all settings with noise. Without noise,
attention mechanisms perform better than average pooling. They concluded that
attention mechanisms applied to bags of sentences are not robust to noise.
They posit that attention mechanisms learn to focus on the relevant context
within a bag of sentences, but informative context is either rare or missing
entirely within noisy bags of sentences. In such cases, better performance is
obtained by only considering the entity mentions via average pooling and
ignoring noisy context.

Leveraging the knowledge that attention mechanisms and average pooling do well
in different settings within multi-instance learning, Li et al. (2020)
introduced a method that enables a model to learn when either method is most
effective. The authors proposed a trainable selection gate that learns when to
use attention versus average pooling for bags of sentences. The gate selects
an attention mechanism to predict relationships for bags of sentences with
good context (i.e., accurately labeled sentences). Likewise, the gate selects
average pooling for bags of sentences with poor context (i.e., wrongly
labeled, noisy sentences). The gating function is a simple sigmoid activation
function that associates a low gating value for noisy sentences, effectively
preventing noise propagation.

One shortcoming of multi-instance learning, in general, is its requirement of
multiple sentences to fill a bag of sentences. In some domains, such as the
biomedical domain, it can be rare to have particular entity pairs mentioned
together in a sentence. Infrequently co-mentioned entity pairs lead to a long-
tail distribution where a majority of entity pairs are supported by only one
or two sentences Hogan et al. (2021). Creating bags with these entity pairs
requires heavy up-sampling, which diminishes the effectiveness of multi-
instance learning.

For example, using a bag size equal to 16 sentences and an entity pair that is
supported by a single sentence, the single sentence is duplicated 15 times to
fill the multi-instance learning bag. In the NYT10 dataset Riedel et al.
(2010), as many as 80% of the bags of sentences contain a single duplicated
sentence. A commonly mentioned entity pair may produce a bag of unique and
informative sentences, while a rarely mentioned entity pair may produce a bag
consisting of a single duplicated sentence. Furthermore, depending on the
noise ratio in the dataset, a bag of sentences may not contain the context
required to determine a relationship.

###  4.2 Pre-training Methods

Current state-of-the-art relation extraction (RE) models typically leverage a
two-phase training: a self-supervised pre-training followed by a supervised
fine-tuning. This training methodology was popularized by BERT Devlin et al.
(2019) and has since made significant gains on the task of RE. To effectively
extract relations from text, one may start with a pre-trained language model
that comes with some latent semantic understanding (e.g., BERT, SciBERT
Beltagy et al. (2019), PubMedBERT Gu et al. (2022)) and then use those
informed representations to fine-tune a model to classify relationships
between pairs of entities in text.

However, one critique of pre-trained language models such as BERT and SciBERT
is that they feature a generic pre-training objective. This is by design—the
generic pre-training objective, namely masked language modeling, allows the
model to generalize to a wide variety of downstream tasks such as text
summarization, named entity recognition, question-answering, relation
extraction, etc.

Soon after BERT’s release, Soares et al. (2019) proposed “Matching the Blanks”
(MTB), a model that featured a pre-training objective explicitly designed for
the task of relation extraction. First, mentions within sentences are wrapped
by special delimiter tokens marking their start and end spans, then 70% of the
entity mentions within a sentence are randomly masked. The modified input
sequences with masked and delimited entities are sent through a BERT encoder
to get an embedded sequence. Next, relationship representations are formed by
concatenating the embedded start tokens of an entity pair (see Figure 4).
Finally, pairs relationship representations are assigned a similarity score
via a dot product.

Figure 4: An illustration showing how Soares et al. (2019) form relationship
representations. Sentences with entities wrapped by special delimiter tokens
(e.g., ‘$’ and ‘^’) are passed through BERT. Then, embeddings of the start
tokens are concatenated to form the relationship embedding. During pre-
training, a dot product calculates pairwise similarity between relationship
embeddings.

The model learns to “match the blanks” by assigning a high similarity between
sentences that contain the same entity pair or, in other words, sentences that
may express the same relationship. They empirically show that the delimiter
tokens are essential to model performance. Without them, the model cannot
decipher the entities of interest within a sentence.

MTB learns the parameters for their relation statement encoder $f_{\theta}$—a
binary classifier function that determines the similarity of two entity pairs:

$p\left(l=1\mid\mathbf{r},\mathbf{r}^{\prime}\right)=\frac{1}{1+\exp
f_{\theta}(\mathbf{r})^{\top}f_{\theta}\left(\mathbf{r}^{\prime}\right)}$

Where $\mathbf{r}$ is the relationship representation formed using the process
illustrated in Figure 4. The inner product $\exp
f_{\theta}(\mathbf{r})^{\top}f_{\theta}\left(\mathbf{r}^{\prime}\right)$
should be high if $\mathbf{r}$ and $\mathbf{r}^{\prime}$ express semantically
similar relations and vice-versa. This binary classifier is used in the
construction of their pre-training loss objective:

$\displaystyle\mathcal{L}(\mathcal{D})=-$
$\displaystyle\frac{1}{|\mathcal{D}|^{2}}\sum_{\left(\mathbf{r},e_{1},e_{2}\right)\in\mathcal{D}}\sum_{\left(\mathbf{r}^{\prime},e_{1}^{\prime},e_{2}^{\prime}\right)\in\mathcal{D}}$
$\displaystyle\delta_{e_{1},e_{1}^{\prime}}\delta_{e_{2},e_{2}^{\prime}}\cdot\log
p\left(l=1\mid\mathbf{r},\mathbf{r}^{\prime}\right)+$
$\displaystyle\left(1-\delta_{e_{1},e_{1}^{\prime}}\delta_{e_{2},e_{2}^{\prime}}\right)$
$\displaystyle\cdot\log\left(1-p\left(l=1\mid\mathbf{r},\mathbf{r}^{\prime}\right)\right)$

Where $\delta_{e_{1},e_{1}^{\prime}}=1$ iff ${e_{1}=e_{1}^{\prime}}$ and
corpus of relation statements
$\mathcal{D}=\left[\left(\mathbf{r}^{0},e_{1}^{0},e_{2}^{0}\right)\ldots\left(\mathbf{r}^{N},e_{1}^{N},e_{2}^{N}\right)\right]$.
Each relational statement contains two entities $e_{1}^{i}$,
$e_{2}^{i}\in\mathcal{E}$, where $\mathcal{E}$ represents the set of all
entities. The corpus $\mathcal{D}$ used for pre-training consisted of 600
million relation statement pairs and was constructed by using Google Cloud
Natural Language API 333https://cloud.google.com/natural-language to identify
entities within passages from Wikipedia (English).

Interestingly, the MTB model departs from the distantly supervised RE paradigm
in that it does not use relation labels during the pre-training phase. They
argue that this creates a more ontology-independent pre-trained model since
training does not rely on an ontology-specific knowledge graph to define
relational classes.

In the second phase of MTB, the model is fine-tuned on an ontology-specific
downstream task such as DocRED or FewRel. The authors report superior
performance compared to generic pre-training models such as BERT. MTB achieves
significant gains on low-resource training (e.g., using less than 20% of the
fine-tuning training data). Notably, MTB outperformed the best-published
methods on FewRel without fine-tuning on FewRel data. This feat highlights
MTB’s effectiveness in learning high-quality representations for
relationships.

MTB’s success inspired numerous subsequent pre-trained models optimized for
relation extraction. Continuing the thread of pre-trained models optimized for
RE, Xiao et al. (2020) propose a three-part pre-training objective optimized
for document-level RE. They pre-train a model to match masked entities,
identify relations, and align relational facts.

For entity matching, they randomly mask entities within a document and have
the model match the masked entities. This is similar to MTB but, instead of
matching pairs of entities to other pairs of entities, they match single
entities to each other. The authors claim that this helps the model learn
anaphora resolution—resolving words referring to or replacing a word used
earlier in a document.

For relation detection, they use a binary classifier; relations are labeled as
either a positive or a negative instance (e.g., “no relation”). They show via
an ablation experiment that relation detection is critical to the model’s
overall performance as it helps the model reduce the noise introduced by
distant supervision. Lastly, they use a pairwise similarity score to pairs of
entities that express the same relationship to align relational facts.

They show that their three-part pre-trained model does well when evaluated on
DocRED. It outperforms BERT as well as a few varieties of BERT (HIN-BERT Tang
et al. (2020), BERT-TS Wang et al. (2019)), as well as a CNN, LSTM, and a
BiLSTM.

Recent general-domain RE works have shown impressive performance gains using a
contrastive learning pre-training objective designed specifically for relation
extraction Peng et al. (2020); Qin et al. (2021). Peng et al. (2020) propose a
model named “Contrastive Pre-training” (CP), where they randomly mask entity
mentions within a sentence and train the model to align different entity pairs
that express the same relationship via contrastive learning. By masking
entities, they encourage the model to learn from the context in a sentence
instead of relying on entity mentions. They pre-train their model on a
distantly supervised RE dataset constructed by pairing text from Wikipedia
with the WikiData knowledge graph, mirroring the method used to construct the
distantly labeled training set in DocRED. They then fine-tune CP on the TACRED
dataset. The authors report that their model makes performance gains over BERT
and MTB, showing that their contrastive learning framework effectively
produces high-quality representations for relationships.

Figure 5:  The FineCL Hogan et al. (2022) framework has three stages: Stage 1:
distantly supervised data ($\mathcal{T}$) is used to train a PLM via cross-
entropy to collect ordered subsets of learned ($\mathcal{A}$) and not learned
($\mathcal{B}$) instances over $k$ epochs. Stage 2: function $f(k)$ weighs
relation instances ($r_{0},r_{1}$) relative to their learning order in a
contrastive learning pre-training objective that uses cosine similarity to
align similar relations. Stage 3: the model is adapted to a discriminative
task.

The success of Peng et al. (2020) inspired Qin et al. (2021)’s ERICA model.
ERICA uses a three-part loss objective for pre-training: entity
discrimination, relation discrimination, and mask language modeling. For
entity discrimination, they use contrastive learning to maximize the cosine
similarity of a given head entity to a corresponding tail entity in a fact
triple. Notably, they use document-level entity representations instead of
sentence-level entity representations. Document-level entity representations
average all mentions of a single entity within a document to form a
representation. The authors claim this helps the model learn sentence-level as
well as document-level relationships.

For relation discrimination, they again use contrastive learning to group
different instances of the same relationship. Finally, they include the masked
language modeling objective introduced by BERT. To construct the pre-training
dataset, they follow Peng et al. (2020) and pair Wikipedia text with entities
and relationships from the WikiData knowledge graph. The pre-trained ERICA
model is then fine-tuned on numerous downstream tasks such as DocRED, TACRED,
SemEval-2010 Task 8. They report improved performance in all settings compared
to MTB and the method proposed by Peng et al. (2020), offering further
evidence that contrastive learning for RE creates deep and informative
relationship representations.

Conventional contrastive learning for RE treats all instances equally, which
can be problematic when training on distantly supervised data that may contain
up to 53% of incorrectly assigned labels Gao et al. (2021). To diminish the
impact of noise from automatically labeled relation data, Hogan et al. (2022)
propose Fine-grained Contrastive Learning (FineCL). Figure 5 illustrates the
end-to-end data flow for the FineCL method. First, automatically labeled
relation data is used to train an off-the-shelf language model (e.g., RoBERTa,
or another domain-specific BERT relative) via cross-entropy. During training,
the learning order of relation instances is recorded. A training instance is
considered “learned” when the model first correctly predicts the corresponding
instance. They show that the order of learned instances corresponds to label
accuracy: clean, accurately labeled relation instances are, on average,
learned first, followed by noisy, inaccurately labeled relation instances.

They then leverage the learning-order of relation instances to improve the
relationship representations learned during pre-training by linearly
projecting the weights of each relation instance corresponding to the order in
which the instance was learned. Higher weights are applied to relation
instances learned earlier in training relative to those learned later in
training. These weights are used to inform a contrastive learning loss
function that learns to group instances of similar relationships. They call
this “fine-grained” contrastive learning since it leverages additional, fine-
grained information about which instances are and are not noisy to produce
higher-quality relationship representations. The representations learned
during pre-training are then used to fine-tune the model on gold-labeled data.

Size | 1% | 10% | 100%
---|---|---|---
Metrics | F1 | IgF1 | F1 | IgF1 | F1 | IgF1
CNN | - | - | - | - | 42.3 | 40.3
BiLSTM | - | - | - | - | 51.1 | 50.3
HINBERT | - | - | - | - | 55.6 | 53.7
CorefBERT | 32.8 | 31.2 | 46.0 | 43.7 | 57.0 | 54.5
SpanBERT | 32.2 | 30.4 | 46.4 | 44.5 | 57.3 | 55.0
ERNIE | 26.7 | 25.5 | 46.7 | 44.2 | 56.6 | 54.2
MTB | 29.0 | 27.6 | 46.1 | 44.1 | 56.9 | 54.3
CP | 30.3 | 28.7 | 44.8 | 42.6 | 55.2 | 52.7
BERT | 19.9 | 18.8 | 45.2 | 43.1 | 56.6 | 54.4
RoBERTa | 29.6 | 27.9 | 47.6 | 45.7 | 58.2 | 55.9
ERICABERT | 22.9 | 21.7 | 48.5 | 46.4 | 57.4 | 55.2
ERICARoBERTa | 30.0 | 28.2 | 50.1 | 48.1 | 59.1 | 56.9
FineCL | 33.2 | 31.2 | 50.3 | 48.3 | 59.5 | 57.1

Table 2: F1-micro scores on the DocRED test set as reported by Hogan et al.
(2022). IgF1 ignores performance on fact triples in the test set overlapping
with triples in the train/dev sets.

Dataset | TACRED | SemEval
---|---|---
Size | 1% | 10% | 100% | 1% | 10% | 100%
MTB | 35.7 | 58.8 | 68.2 | 44.2 | 79.2 | 88.2
CP | 37.1 | 60.6 | 68.1 | 40.3 | 80.0 | 88.5
BERT | 22.2 | 53.5 | 63.7 | 41.0 | 76.5 | 87.8
RoBERTa | 27.3 | 61.1 | 69.3 | 43.6 | 77.7 | 87.5
ERICABERT | 34.9 | 56.0 | 64.9 | 46.4 | 79.8 | 88.1
ERICARoBERTa | 41.1 | 61.7 | 69.5 | 50.3 | 80.9 | 88.4
FineCL | 43.7 | 62.7 | 70.3 | 51.2 | 81.0 | 88.7

Table 3: F1-micro scores on the TACRED and SemEval test sets as reported by
Hogan et al. (2022).

Tables 2 and 3 compare FineCL’s performance on DocRED, TACRED, and SemEval
using the following aforementioned baselines: (1) CNN Zeng et al. (2014), (2)
BiLSTM Hochreiter and Schmidhuber (1997), (3) BERT Devlin et al. (2019), (4)
RoBERTa Liu et al. (2019), (5) MTB Soares et al. (2019), (6) CP Peng et al.
(2020), (7 & 8) ERICABERT & ERICARoBERTa Qin et al. (2021). For fair
comparison, each baseline is identically pre-trained mirroring the pre-
training steps used in Qin et al. (2021). In these experiments, Hogan et al.
(2022) report performance gains over Qin et al. (2021) in all settings on
DocRED, TACRED, and SemEval-2010 Task 8 datasets, illustrating the
effectiveness of leveraging learning order to improve relation
representations.

##  5 Conclusion

In this survey, we reviewed the relation extraction task with a focus on
distant supervision. We traced the history of RE methods by discussing
exemplary works and highlighting shortcomings to contextualize progress. We
noted the key differences between pattern-based methods, statistical-based
methods, neural-based methods, and finally, large language model-based
methods. We conducted an overview of popular RE datasets and discussed some
limitations of both corpus-based and instance-based RE evaluation.

We reviewed current RE methods, focusing on denoising and pre-training
methods—two of the defining methods of distant supervision for RE. We
highlighted various pre-training methodologies used to attain state-of-the-art
performance on RE benchmarks. Finally, we discussed recent RE works that
leverage a contrastive learning pre-training objective.

## References

  * Alt et al. (2019) Christoph Alt, Marc Hübner, and Leonhard Hennig. 2019.  Fine-tuning pre-trained transformer language models to distantly supervised relation extraction.  _ArXiv_ , abs/1906.08646.
  * Amin et al. (2020) Saadullah Amin, Katherine Dunfield, Anna Vechkaeva, and Günter Neumann. 2020\.  A data-driven approach for noise reduction in distantly supervised biomedical relation extraction.  In _BIONLP_.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  Scibert: A pretrained language model for scientific text.  In _EMNLP_.
  * Bodenreider (2004) Olivier Bodenreider. 2004.  The unified medical language system (umls): integrating biomedical terminology.  _Nucleic acids research_ , 32 Database issue:D267–70.
  * Bollacker et al. (2008) Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. 2008.  Freebase: a collaboratively created graph database for structuring human knowledge.  In _SIGMOD Conference_.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021.  On the opportunities and risks of foundation models.
  * Buhrmester et al. (2011) Michael D. Buhrmester, Tracy Nai Kwang, and Samuel D. Gosling. 2011.  Amazon’s mechanical turk.  _Perspectives on Psychological Science_ , 6:3 – 5.
  * Califf and Mooney (1997) Mary Elaine Califf and Raymond J. Mooney. 1997.  Relational learning of pattern-match rules for information extraction.  In _CoNLL_.
  * Canese and Weis (2013) Kathi Canese and Sarah Weis. 2013.  Pubmed: the bibliographic database.  In _The NCBI Handbook [Internet]. 2nd edition_. National Center for Biotechnology Information (US).
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL_.
  * Fellbaum (1998) Christiane Fellbaum. 1998.  _WordNet: An Electronic Lexical Database_.  Bradford Books.
  * Finkel and Manning (2009) Jenny Rose Finkel and Christopher D. Manning. 2009.  Nested named entity recognition.  In _EMNLP_.
  * Gao et al. (2021) Tianyu Gao, Xu Han, Keyue Qiu, Yuzhuo Bai, Zhiyu Xie, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2021.  Manual evaluation matters: Reviewing test protocols of distantly supervised relation extraction.  In _FINDINGS_.
  * Gao et al. (2019) Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019\.  FewRel 2.0: Towards more challenging few-shot relation classification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6251–6256, Hong Kong, China. Association for Computational Linguistics.
  * Gu et al. (2022) Yuxian Gu, Robert Tinn, Hao Cheng, Michael R. Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2022.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Transactions on Computing for Healthcare (HEALTH)_ , 3:1 – 23\.
  * Han et al. (2018a) Xu Han, Zhiyuan Liu, and Maosong Sun. 2018a.  Neural knowledge acquisition via mutual attention between knowledge graph and text.  In _AAAI_.
  * Han et al. (2018b) Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018b.  FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4803–4809, Brussels, Belgium. Association for Computational Linguistics.
  * Hendrickx et al. (2010) Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010.  SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals.  In _Proceedings of the 5th International Workshop on Semantic Evaluation_ , pages 33–38, Uppsala, Sweden. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ , 9:1735–1780.
  * Hogan et al. (2021) William P Hogan, Molly Huang, Yannis Katsis, Tyler Baldwin, Ho-Cheol Kim, Yoshiki Baeza, Andrew Bartko, and Chun-Nan Hsu. 2021.  Abstractified multi-instance learning (AMIL) for biomedical relation extraction.  In _3rd Conference on Automated Knowledge Base Construction_.
  * Hogan et al. (2022) William P Hogan, Jiacheng Li, and Jingbo Shang. 2022.  Fine-grained contrastive learning for relation extraction.  _ArXiv_ , abs/tbd.
  * Hu et al. (2021) Zikun Hu, Yixin Cao, Lifu Huang, and Tat-Seng Chua. 2021.  How knowledge graph and attention help? a qualitative analysis into bag-level relation extraction.  _ArXiv_ , abs/2107.12064.
  * Li et al. (2016) Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016.  Biocreative v cdr task corpus: a resource for chemical disease relation extraction.  _Database : the journal of biological databases and curation_ , 2016:baw068.
  * Li et al. (2020) Y. Li, Guodong Long, Tao Shen, Tianyi Zhou, Lina Yao, Huan Huo, and Jing Jiang. 2020\.  Self-attention enhanced selective gate with entity-aware embedding for distantly supervised relation extraction.  In _AAAI_.
  * Li et al. (2014) Zhixu Li, Mohamed A. Sharaf, Laurianne Sitbon, Xiaoyong Du, and Xiaofang Zhou. 2014\.  Core: A context-aware relation extraction method for relation completion.  _IEEE Transactions on Knowledge and Data Engineering_ , 26:836–849.
  * Lin (2003) Dekang Lin. 2003.  _Dependency-Based Evaluation of Minipar_, pages 317–329. Springer Netherlands, Dordrecht.
  * Lin et al. (2015) Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015.  Learning entity and relation embeddings for knowledge graph completion.  In _AAAI_.
  * Lin et al. (2016) Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016.  Neural relation extraction with selective attention over instances.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2124–2133, Berlin, Germany. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Luo et al. (2017) Bingfeng Luo, Yansong Feng, Zheng Wang, Zhanxing Zhu, Songfang Huang, Rui Yan, and Dongyan Zhao. 2017.  Learning with noise: Enhance distantly supervised relation extraction with dynamic transition matrix.  _ArXiv_ , abs/1705.03995.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  In _Advances in Neural Information Processing Systems_ , volume 26. Curran Associates, Inc.
  * Mintz et al. (2009) Mike D. Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009.  Distant supervision for relation extraction without labeled data.  In _ACL_.
  * Miwa and Bansal (2016) Makoto Miwa and Mohit Bansal. 2016.  End-to-end relation extraction using lstms on sequences and tree structures.  _ArXiv_ , abs/1601.00770.
  * Peng et al. (2020) Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020.  Learning from context or names? an empirical study on neural relation extraction.  In _EMNLP_.
  * Peng et al. (2017) Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen tau Yih. 2017\.  Cross-sentence n-ary relation extraction with graph lstms.  _Transactions of the Association for Computational Linguistics_ , 5:101–115.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Qin et al. (2021) Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. 2021.  Erica: Improving entity and relation understanding for pre-trained language models via contrastive learning.  In _ACL_.
  * Quirk and Poon (2017) Chris Quirk and Hoifung Poon. 2017.  Distant supervision for relation extraction beyond the sentence boundary.  In _EACL_.
  * Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010.  Modeling relations and their mentions without labeled text.  In _ECML/PKDD_.
  * Soares et al. (2019) Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019\.  Matching the blanks: Distributional similarity for relation learning.  _ArXiv_ , abs/1906.03158.
  * Surdeanu et al. (2012) Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D. Manning. 2012\.  Multi-instance multi-label learning for relation extraction.  In _EMNLP_.
  * Tang et al. (2020) Hengzhu Tang, Yanan Cao, Zhenyu Zhang, Jiangxia Cao, Fang Fang, Shi Wang, and Pengfei Yin. 2020.  Hin: Hierarchical inference network for document-level relation extraction.  _Advances in Knowledge Discovery and Data Mining_ , 12084:197 – 209\.
  * Vaswani et al. (2017) Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _ArXiv_ , abs/1706.03762.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: A free collaborative knowledgebase.  _Commun. ACM_ , 57(10):78–85.
  * Wang et al. (2019) Hong Wang, Christfried Focke, Rob Sylvester, Nilesh Mishra, and William Yang Wang. 2019.  Fine-tune bert for docred with two-step process.  _ArXiv_ , abs/1909.11898.
  * Xiao et al. (2020) Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin, and Leyu Lin. 2020.  Denoising relation extraction from document-level distant supervision.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3683–3688, Online. Association for Computational Linguistics.
  * Xu et al. (2016) Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016.  Question answering on freebase via relation extraction and textual evidence.  _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Yao et al. (2019) Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019.  Docred: A large-scale document-level relation extraction dataset.  _ArXiv_ , abs/1906.06127.
  * Zeng et al. (2015) Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015.  Distant supervision for relation extraction via piecewise convolutional neural networks.  In _EMNLP_.
  * Zeng et al. (2014) Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014.  Relation classification via convolutional deep neural network.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 2335–2344, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
  * Zhang and Wang (2015) Dongxu Zhang and Dong Wang. 2015.  Relation classification via recurrent neural network.  _ArXiv_ , abs/1508.01006.
  * Zhang et al. (2017) Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017.  Position-aware attention and supervised data improve slot filling.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017)_ , pages 35–45.


* [138] Benjamin Ennemoser and Ingrid Mayrhofer-Hufnagl.  Design across multi-scale datasets by developing a novel approach to 3dgans.  International Journal of Architectural Computing, page 14780771231168231, 2023.
  * [139] DONGYUN KIM, LLOYD SUKGYO LEE, and HANJUN KIM.  Elemental sabotage.
  * [140] Hang Zhang and Ye Huang.  Machine learning aided 2d-3d architectural form finding at high resolution.  In Proceedings of the 2020 DigitalFUTURES: The 2nd International Conference on Computational Design and Robotic Fabrication (CDRF 2020), pages 159–168. Springer, 2021.
  * [141] Hang Zhang and E Blasetti.  3d architectural form style transfer through machine learning (full version), 2020.
  * [142] KE Asmar and Harpreet Sareen.  Machinic interpolations: a gan pipeline for integrating lateral thinking in computational tools of architecture.  In Proceedings of the 24th Conference of the Iberoamerican Society of Digital Graphics, Online, pages 18–20, 2020.
  * [143] Hang Zhang.  Text-to-form.  08 2021.
  * [144] Kai-Hung Chang, Chin-Yi Cheng, Jieliang Luo, Shingo Murata, Mehdi Nourbakhsh, and Yoshito Tsuji.  Building-gan: Graph-conditioned architectural volumetric design generation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11956–11965, 2021.
  * [145] Zhenlong Du, Haiyang Shen, Xiaoli Li, and Meng Wang.  3d building fabrication with geometry and texture coordination via hybrid gan.  Journal of Ambient Intelligence and Humanized Computing, pages 1–12, 2020.
  * [146] Qiu Yu, Jamal Malaeb, and Wenjun Ma.  Architectural facade recognition and generation through generative adversarial networks.  In 2020 International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE), pages 310–316. IEEE, 2020.
  * [147] Cheng Lin Chuang, Sheng Fen Chien, et al.  Facilitating architect-client communication in the pre-design phase.  In Projections-Proceedings of the 26th International Conference of the Association for Computer-Aided Architectural Design Research in Asia, CAADRIA 2021, volume 2, pages 71–80. The Association for Computer-Aided Architectural Design Research in Asia …, 2021.
  * [148] Cheng Sun, Yiran Zhou, and Yunsong Han.  Automatic generation of architecture facade for historical urban renovation using generative adversarial network.  Building and Environment, 212:108781, 2022.
  * [149] Lei Zhang, Liang Zheng, Yile Chen, Lei Huang, and Shihui Zhou.  Cgan-assisted renovation of the styles and features of street facades—a case study of the wuyi area in fujian, china.  Sustainability, 14(24):16575, 2022.
  * [150] Hongpan Lin, Linsheng Huang, Yile Chen, Liang Zheng, Minling Huang, and Yashan Chen.  Research on the application of cgan in the design of historic building facades in urban renewal—taking fujian putian historic districts as an example.  Buildings, 13(6):1478, 2023.
  * [151] JIAXIN ZHANG, TOMOHIRO FUKUDA, NOBUYOSHI YABUKI, and YUNQIN LI.  Synthesizing style-similar residential facade from semantic labeling according to the user-provided example.
  * [152] Wenyuan Sun, Ping Zhou, Yangang Wang, Zongpu Yu, Jing Jin, and Guangquan Zhou.  3d face parsing via surface parameterization and 2d semantic segmentation network, 2022.
  * [153] Da Wan, Runqi Zhao, Sheng Zhang, Hui Liu, Lian Guo, Pengbo Li, and Lei Ding.  A deep learning-based approach to generating comprehensive building façades for low-rise housing.  Sustainability, 15(3):1816, 2023.
  * [154] JIAHUA DONG, QINGRUI JIANG, ANQI WANG, and YUANKAI WANG.  Urban cultural inheritance.
  * [155] Shengyu Meng.  Exploring in the latent space of design: A method of plausible building facades images generation, properties control and model explanation base on stylegan2.  In Proceedings of the 2021 DigitalFUTURES: The 3rd International Conference on Computational Design and Robotic Fabrication (CDRF 2021) 3, pages 55–68. Springer Singapore, 2022.
  * [156] Selen Çiçek, Gozde Damla Turhan, and Aybüke Taşer.  Deterioration of pre-war and rehabilitation of post-war urbanscapes using generative adversarial networks.  International Journal of Architectural Computing, page 14780771231181237, 2023.
  * [157] Zhenhuang Cai, Yangbin Lin, Jialian Li, Zongliang Zhang, and Xingwang Huang.  Building facade completion using semantic-synchronized gan.  In 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, pages 6387–6390. IEEE, 2021.
  * [158] Xue Sun, Yue Wang, Ting Zhang, Yin Wang, Haoyue Fu, Xuechun Li, and Zhen Liu.  An application of deep neural network in facade planning of coastal city buildings.  In International Conference on Computer Science and its Applications and the International Conference on Ubiquitous Information Technologies and Applications, pages 517–523. Springer, 2022.
  * [159] Frederick Chando Kim and Jeffrey Huang.  Perspectival gan.
  * [160] J Chen and R Stouffs.  From exploration to interpretation: Adopting deep representation learning models to latent space lnterpretation of architectural design alternatives.  2021\.
  * [161] Wolf dPrix, Karolin Schmidbaur, Daniel Bolojan, and Efilena Baseta.  The legacy sketch machine: from artificial to architectural intelligence.  Architectural Design, 92(3):14–21, 2022.
  * [162] Ruşen Eroğlu and Leman Figen Gül.  Architectural form explorations through generative adversarial networks.  Legal Depot D/2022/14982/02, page 575, 2022.
  * [163] Nervana Osama Hanafy.  Artificial intelligence’s effects on design process creativity:” a study on used ai text-to-image in architecture”.  Journal of Building Engineering, 80:107999, 2023.
  * [164] Ville Paananen, Jonas Oppenlaender, and Aku Visuri.  Using text-to-image generation for architectural design ideation.  arXiv preprint arXiv:2304.10182, 2023.
  * [165] Hanım Gülsüm Karahan, Begüm Aktaş, and Cemal Koray Bingöl.  Use of language to generate architectural scenery with ai-powered tools.  In International Conference on Computer-Aided Architectural Design Futures, pages 83–96. Springer, 2023.
  * [166] Junming Chen, Zichun Shao, and Bin Hu.  Generating interior design from text: A new diffusion model-based method for efficient creative design.  Buildings, 13(7):1861, 2023.
  * [167] Frederick Chando Kim, Mikhael Johanes, and Jeffrey Huang.  Text2form diffusion.
  * [168] Gernot Riether and Taro Narahara.  Ai tools to synthesize characteristics of public spaces.
  * [169] Junming Chen, Duolin Wang, Zichun Shao, Xu Zhang, Mengchao Ruan, Huiting Li, and Jiaqi Li.  Using artificial intelligence to generate master-quality architectural designs from text descriptions.  Buildings, 13(9):2285, 2023.
  * [170] Sachith Seneviratne, Damith Senanayake, Sanka Rasnayaka, Rajith Vidanaarachchi, and Jason Thompson.  Dalle-urban: Capturing the urban design expertise of large text to image transformers.  In 2022 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pages 1–9. IEEE, 2022.
  * [171] Jonathan Dortheimer, Gerhard Schubert, Agata Dalach, Lielle Brenner, and Nikolas Martelaro.  Think ai-side the box!
  * [172] Emmanouil Vermisso.  Semantic ai models for guiding ideation in architectural design courses.  In ICCC, pages 205–209, 2022.
  * [173] Daniel Bolojan, Emmanouil Vermisso, and Shermeen Yousif.  Is language all we need? a query into architectural semantics using a multimodal generative workflow.  In POST-CARBON, Proceedings of the 27th International Conference of the Association for Computer-Aided Architectural Design Research in Asia (CAADRIA), volume 1, pages 353–362, 2022.
  * [174] Dongyun Kim.  Latent morphologies: Encoding architectural features and decoding their structure through artificial intelligence.  International Journal of Architectural Computing, page 14780771231209458, 2022.
  * [175] Kaiyu Cheng, Paulina Neisch, and Tong Cui.  From concept to space: a new perspective on aigc-involved attribute translation.  Digital Creativity, 34(3):211–229, 2023.
  * [176] Jeffrey Huang, Mikhael Johanes, Frederick Chando Kim, Christina Doumpioti, and Georg-Christoph Holz.  On gans, nlp and architecture: combining human and machine intelligences for the generation and evaluation of meaningful designs.  Technology— Architecture+ Design, 5(2):207–224, 2021.
  * [177] DONGYUN KIM, GEORGE GUIDA, JOSE LUIS GARCÍA, and DEL CASTILLO Y LÓPEZ.  Participatory urban design with generative adversarial networks.
  * [178] Yeji Hong, Somin Park, Hongjo Kim, and Hyoungkwan Kim.  Synthetic data generation using building information models.  Automation in Construction, 130:103871, 2021.
  * [179] X. Zhuang.  Rendering sketches - interactive rendering generation from sketches using conditional generative adversarial neural network.  Proceedings of the 40th International Conference on Education and Research in Computer Aided Architectural Design in Europe (eCAADe) [Volume 1], 2022.
  * [180] Yuqian Li and Weiguo Xu.  Using cyclegan to achieve the sketch recognition process of sketch-based modeling.  In Proceedings of the 2021 DigitalFUTURES: The 3rd International Conference on Computational Design and Robotic Fabrication (CDRF 2021) 3, pages 26–34. Springer, 2022.
  * [181] Xinyue Ye, Jiaxin Du, and Yu Ye.  Masterplangan: Facilitating the smart rendering of urban master plans via generative adversarial networks.  Environment and Planning B: Urban Analytics and City Science, 49(3):794–814, 2022.
  * [182] YUQIAN LI1and WEIGUO XU.  Research on architectural sketch to scheme image based on context encoder.
  * [183] Yingbin Gui, Biao Zhou, Xiongyao Xie, Wensheng Li, and Xifang Zhou.  Gan-based method for generative design of visual comfort in underground space.  In IOP Conference Series: Earth and Environmental Science, volume 861, page 072015. IOP Publishing, 2021.
  * [184] Ka Chun Shum, Hong-Wing Pang, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung.  Conditional 360-degree image synthesis for immersive indoor scene decoration.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4478–4488, 2023.
  * [185] Matias del Campo.  Deep house-datasets, estrangement, and the problem of the new.  Architectural Intelligence, 1(1):12, 2022.
  * [186] Matias Del Campo, Sandra Manninger, and Alexandra Carlson.  Hallucinating cities: a posthuman design method based on neural networks.  In Proceedings of the 11th annual symposium on simulation for architecture and urban design, pages 1–8, 2020.
  * [187] Wenliang Qian, Yang Xu, and Hui Li.  A self-sparse generative adversarial network for autonomous early-stage design of architectural sketches.  Computer-Aided Civil and Infrastructure Engineering, 37(5):612–628, 2022.
  * [188] Sisi Han, Yuhan Jiang, Yilei Huang, Mingzhu Wang, Yong Bai, and Andrea Spool-White.  Scan2drawing: Use of deep learning for as-built model landscape architecture.  Journal of Construction Engineering and Management, 149(5):04023027, 2023.
  * [189] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  * [190] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel.  Multidiffusion: Fusing diffusion paths for controlled image generation.  2023\.
  * [191] Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang, Huaping Liu, and Guosheng Lin.  Gaussianeditor: Swift and controllable 3d editing with gaussian splatting.  arXiv preprint arXiv:2311.14521, 2023.