# Instructions for EMNLP 2020 Proceedings

Xiaoyu Chen
Dept. of Computer Science
Stanford University
<EMAIL_ADDRESS>
&Rohan Badlani††footnotemark:
Dept. of Computer Science
Stanford University
<EMAIL_ADDRESS>
equal contribution

# Relation Extraction with Contextualized Relation Embedding (CRE)

Xiaoyu Chen
Dept. of Computer Science
Stanford University
<EMAIL_ADDRESS>
&Rohan Badlani††footnotemark:
Dept. of Computer Science
Stanford University
<EMAIL_ADDRESS>
equal contribution

###### Abstract

Relation extraction (RE) is the task of identifying relation instance(s)
between two entities given a corpus whereas Knowledge base (KB) modeling is
the task of representing a knowledge base, in terms of relations between
entities. This paper proposes an architecture for the relation extraction (RE)
task that integrates semantic information with knowledge base (KB) modeling in
a novel manner. Existing approaches for relation extraction either don’t
utilize knowledge base modelling or use separately trained KB models for the
RE task. We present a model architecture that internalizes KB modeling in
relation extraction. This model applies a novel approach to encode sentences
into contextualized relation embeddings (CRE), which can then be used together
with parameterized entity embeddings to score relation instances. The proposed
CRE model achieves state of the art performance on datasets derived from The
New York Times Annotated Corpus111https://catalog.ldc.upenn.edu/LDC2008T19 and
FreeBase222https://developers.google.com/freebase. The source code has been
made available333https://github.com/codchen/CRE.

##  1 Introduction

Relation extraction (RE) is a sub-task under the broad category of information
extraction (IE) that aims to identify relationship(s) between named entities
based on textual information (corpus). The groundtruth relationship(s) between
an entity pair can be either internal, if each sentence in the corpus is
explicitly labeled, or external, in the form of relation instances in a
standalone knowledge base (KB). Such knowledge bases, like Freebase Bollacker
et al. (2008), are collaboratively edited by human beings and thus offer high
informational fidelity.

In this paper, we focus on the problem of predicting the collection of
relations for a new entity pair based on the usage of that entity pair in its
respective collection of sentences. To this end, we assume the general
construction that given a dataset with a set of named entity pairs and each
entity pair’s respective collection of sentences as input along with entity
pair’s respective collection of relations as output, our goal is to train a
model with the objective of predicting collection of relevant relation(s) for
a new entity pair based on its contextual usage.

Distant supervision Mintz et al. (2009) is the most popular approach for this
problem, that has achieved its success by leveraging knowledge base
information for relation extraction tasks. However, this restricts the usage
of semantic information present in knowledge bases, since the distant
supervision work mainly incorporates the knowledge base information as labels
instead of treating it as a graph and thereby losing the dense relationships
between different entities. Knowledge base modelling is an independent area of
research and there has been some recent work on utilizing knowledge base
modeling and incorporating internal structural information from them to the
relation extraction task. Bordes et al. (2013), Trouillon et al. (2016).

Figure 1: Contextualized Relation Embedding (CRE) Model Architecture

The details of "Sentence Encoder" can be found in figure 2.

The scoring function takes one CRE vector and two entity embedding vectors as
inputs, and outputs a scalar score.

Weston et al. (2013) conducted the first research work that utilized knowledge
bases as a structural graph by training a TransE (Bordes et al. (2013)) KB
model alongside a traditional distant supervision extractor. However, this has
been a simple combination of both ideas but despite its simplicity, the model
was able to beat the then-state-of-the-art models in Relation Extraction
tasks. The work by Han et al. (2018), on the other hand, went a step further
by sharing some model weights between the distant supervision extractor and
the knowledge base model. However, although there is some shared architecture,
but the objective function for the KB modelling and the distant supervision
remain separate. Another notable work by Xu and Barbosa (2019) combines the
two models through an additional objective function that guides the training
but during prediction time there is no any shared architecture and hence
knowledge base information is not really incorporated well for relation
prediction.

The model presented in this paper differs from all previous work in that it is
a single relation extraction model that internalizes knowledge base modeling.
Instead of treating relation embeddings as parameters like in standalone
knowledge embedding models (eg: TransE Weston et al. (2013)), where such
embeddings have no context of any textual information, our model expresses
relation embeddings as context-aware latent states generated by encoding
textual data. Such contextualized embeddings represent a natural link between
the textual input and the knowledge base modeling objective, so that the end-
to-end information transformation is completely internal to the model. The
entity embeddings, on the other hand, are not contextualized and will still be
trained as parameters of the same model, because they need to serve as bridges
across different entity pairs and globalize the knowledge base modeling which
is now based on contextualized relation embeddings localized to each entity
pair. A significant innovation of this work is that each sentence in which a
pair of entities occurs, we represent each possible relation between them as a
function of that entire sentence and score the relation for that entity pair
and sentence using the representation of the relation.

##  2 Contextualized Relation Embedding

Let $\mathcal{R}$ stand for the set of all relations and $\mathcal{E}$ stand
for set of all entities. Then, for the $i$-th entity pair, $H_{i}$ and $T_{i}$
stand for the subject and object respectively, both belonging to
$\mathcal{\mathcal{E}}$. $\mathcal{R}^{\prime}_{i}\subset\mathcal{R}$ stand
for the set of relations existing between the $i$-th entity pair. $S_{i}$
stands for the collection of sentences that contain $H_{i}$ as the subject and
$T_{i}$ as the object; $S_{i}^{jk}$ stands for the $k$-th word of the $j$-th
sentence in $S_{i}$, where each $S_{i}^{jk}$ is delimited by spaces except for
the subject and the object, whose phrasal integrities are preserved (e.g. if
it is the subject/object of a sentence, _the United States_ will be considered
as a single “word”).

Figure 2: Single Sentence Encoding Process

We propose a novel neural network model (called CRE model) that implicitly
combines knowledge base to detect relationships between entities, given a
corpus of sentences containing those entities. The overall architecture of a
CRE model is shown in figure 1, which consists of a sentence encoding step and
a knowledge base modeling step. During the sentence encoding step, the model
transforms each $S_{i}^{jk}$ into $E$-dimensional embeddings. Specifically, we
apply pretrained skip-gram embeddings Mikolov et al. (2013), treating all
named entities as unknown, which results in a $W$-dimensional embedding for
each word. We apply a positional embedding based on Zeng et al. (2014), where
each word’s index in the positional “vocabulary” is defined as its relative
distance to one of the named entities. Consider the sentence “A bear entered
the fridge” and named entities “bear” and “fridge”. The word “A” will have
$-1$ as its positional embedding index with respect to “bear” and $-4$ with
respect to “fridge”, whereas “entered” will have $+1$ with respect to “bear”
and $-2$ with respect to “fridge”. For each word (including the two named
entities), we represent it as its skip-gram word embedding of dimension $W$,
concatenated with its positional embeddings (each of dimension $P$) with
respect to the two named entities, to form a final embedding of dimension $E$,
where $E=W+2*P$. The skip-gram embeddings are fixed during training, whereas
the positional embeddings are learnt. This leads to a sentence embedding of
$L*E$, where $L$ is the maximum length of each sentence. We utilize the above
mentioned sentence embeddings and learn an encoder that encodes each sentence
into a $R*K$ embedding, which can be represented as $|\mathcal{R}|$
$K$-dimensional embeddings, each of which corresponds to a distinct relation
in $\mathcal{R}$. Figure 2 illustrates the sentence encoding process for a
single sentence.

It is worth noting that the sentence encoder is a generic framework where we
can plug in any model architecture (both for positional encoding and sentence
encoding). In our experiments, we found that neural network based encoders
tend to result in better performance. We think this is due to their ability to
extract both semantic and syntactic information. We denote sentence $j$’s
representation of $r$’s relation for entity pair $i$ as $CRE_{i}^{jr}$.

In order to provide the model more context, we explicitly apply a
$K$-dimensional embedding for each part of the entity pair represented by
$H_{i}$ and $T_{i}$ respectively. These entity embeddings are learnt as
parameters during training, similar to the positional embeddings above. We
utilize the relation embeddings (contextualized relation embeddings) from the
sentences and these entity embeddings to form relation triplet embeddings.
Specifically, this will result in $R$ triplets for each sentence, since we
computed $R$ relation embeddings for each sentence. This allows us to apply
different knowledge based models and further enhances the generalization of
our model. Depending on the choice of knowledge based model, we use the
relation triplet embeddings to compute its score based on the scoring function
of the respective knowledge base model. For example, if TransE Bordes et al.
(2013) is chosen as the underlying knowledge base model with $H_{i}$ and
$T_{i}$ as head and tail entity embeddings, then embeddings for relation $r$
derived from sentence $j$ will be scored as:

$\begin{split}Score_{i}^{j,r}=1-tanh(||H_{i}+CRE_{i}^{jr}-T_{i}||);\\\ where\
0\leq Score_{i}^{j,r}\leq 2\end{split}$ (1)

Since most knowledge base models are characterized by their scoring functions,
it is straightforward to swap in any knowledge base model here for scoring
purposes.

Finally, the model aggregates scores for the same relation across all
sentences so that we can obtain a single score for each relation given an
entity pair:

$\begin{split}Score_{i}^{r}=\sum_{j}Score_{i}^{j,r}\end{split}$ (2)

For experiments in this paper, we adopted summation as the aggregation
function since we do not have prior knowledge about our dataset and thus want
to take into account the “opinion” of all sentences equally, but it is by no
means prescriptive. One can certainly use maximum, minimum or mean as the
aggregation functions instead of sum in case the dataset is such that any
single sentence’s outcome can be regarded as reliable.

We train the model with a binary cross entropy loss, after normalizing
relation scores and targets and we use an $L2$ regularization that helps
prevent overfitting.

$\begin{split}NS_{i}^{r}=\frac{Score_{i}^{r}}{\sum_{r^{\prime}}Score_{i}^{r^{\prime}}}\end{split}$
(3)

$\begin{split}m_{i}^{r}=\frac{I[r\in\mathcal{R}^{\prime}_{i}]}{|\mathcal{R}^{\prime}_{i}|}\end{split}$
(4)

$\begin{split}L_{i}^{r}=-m_{i}^{r}*log(NS_{i}^{r}))+\\\
(m_{i}^{r}-\frac{1}{|\mathcal{R}^{\prime}_{i}|})*log(1-NS_{i}^{r})\\\
\end{split}$ (5)

$L_{i}=\sum_{r}L_{ir}+\lambda*\sum||w||^{2}$ (6)

We use the normalized scores to predict the relations for the given entity
pair. Note that normalization is still necessary for quantitative evaluation.
We pool together scores for different entity pairs to form a precision-recall
curve, so scores need to be comparable and normalization brings them to the
same scale. The top-k prediction can formulated as:

$\begin{split}Top_{i}^{k}=argmax_{r}^{k}(NS_{i}^{r})\\\
TopScore_{i}^{k}=NS_{i}^{Top_{i}^{k}}\end{split}$ (7)

##  3 Experiments

###  3.1 Data and Experimental Setup

For our experiments, we use the textual data from The New York Times (NYT)
Annotated Corpus Riedel et al. (2010), and the knowledge base are derived from
the most recent FreeBase Bollacker et al. (2008) dump. Note that entities in
each NYT sentence are already annotated. The NYT and Freebase presents a more
challenging task for relation extraction models than FreeBase and Wikipedia
texts due to their heterogeneous nature, since FreeBase itself is largely
derived from Wikipedia. We construct the dataset through the following
procedure:

  1. 1.

Filter out any relation instance from FreeBase dump if the mapping between the
ID of either of its two entities and its corresponding English phrase is not
available.

  2. 2.

Find the top 500K entities based on the number of relation instances they
participate in, and further filter out any relation instance if either of its
two entities is not in the top 500K list.

  3. 3.

Inner join the NYT dataset with the filtered Freebase by aligning NYT entity
annotations and Freebase entity English phrases. In other words, no example in
the NYT dataset that contains unseen entities in the filter Freebase is
preserved, and vice versa.

  4. 4.

Backfill an N/A relation for any entity pair present in the filtered NYT
dataset that has no relation instance in the filtered Freebase.

After this procedure, we have a dataset that contains 465K sentences, 35K
entities, 233K entity pairs, and 238 relations. As most entities in this world
do not possess direct relations with other entities, the dataset is extremely
unbalanced, where only 5K entity pairs out of the 233K have non-N/A relation.
These 5K entity pairs contribute 20K sentences out of the 465K. Because of
this imbalance, we performed train-test splits for positive pairs and negative
pairs separately, so that test set gets around 1200 positive pairs and 57K
negative pairs. Since this test set is still severely imbalanced and we don’t
want our model to be biased in favor an N/A prediction, we further randomly
sample the negative pairs in the test set so that the number of sentences
consisting of negative pairs is roughly the same as that of any positive
relation. We did not perform similar filtering to the training set from the
outset, but randomly applied such filtering at the beginning of each training
epoch, in order to utilize all training data we have.

Figure 3: Weston et al. vs. Han et al. vs. CRE Model

###  3.2 Evaluation Metrics

In order to quantitatively evaluate the performance of our model, we measure
precision over various recall levels. Specifically, we examine precision-
recall curves for top-1 predictions, top-3 predictions, and top-5 predictions.
Top-1 predictions, which only includes the relation with the highest score
among all relations for a given entity pair, are widely used in related
literature, but we believe it alone does not provide a comprehensive view,
since each entity pair can have multiple relations. Besides the precision-
recall curve, we also measure the mean reciprocal rank (MRR) among top-3
predictions and top-5 predictions.

On the qualitative side, we examine the number of distinct top-1 relation
predictions, in order to check if our model achieves high precision just by
heavily favoring certain relations, given the unbalanced nature of our
dataset.

###  3.3 Baseline Models

As discussed in Section 2, our proposed model provides a deep integration of
knowledge bases in the relation prediction task. We select two baseline models
to compare performance: one from Weston et al. (2013) as a representative of
weak integration between knowledge bases and relation extraction, and the
other from Han et al. (2018) to represent a semi-integrated setup. Both models
have already been briefly described in the Introduction section. Since the
state of art of these models is not publically available, we performed
hyperparameter tuning for both models on our dataset, since they may not be
exactly the same as the datasets used in their respective work.

###  3.4 Model Configurations

As shown in the model details section, there can be a myriad of options for
various components of our proposed CRE model. In this evaluation, we will
focus on the effect of different choices of sentence-to-CRE encoders as well
as underlying knowledge base models. Every other aspect will be kept fixed as
described in the preceding section.

For sentence-to-CRE encoders, we will explore 3 options:

  * •

A single 1D convolutional layer with hidden state dimension of 230 and window
size of 3.

  * •

A single LSTM layer with hidden state dimension of 230.

  * •

A double-layered transformer encoder with hidden state dimension of 100 and
5-head attention Vaswani et al. (2017).

All these encoders are followed by a $tanh$ activation layer and a linear
layer to project the low-dimensional hidden state onto
$|\mathcal{R}|K$-dimensional space. 230 is selected as the hidden dimension
for fair comparison purpose because it is the CNN hidden state dimension used
in Han et al. (2018). The complexity of encoders evaluated here are limited by
computational resources available for this work, so it is possible to find
more sophisticated encoders that can achieve further improvements. For the
transformer based encoder, an additional linear projection and tanh activation
is applied at the front to reduce the number of parameters that need to be
trained to a manageable level.

Figure 4: CRE Models with different configurations

Left: precision-recall curves of the most confident prediction for each entity
pairs

Middle: precision-recall curves of the top 3 most confident predictions for
each entity pairs

Right: precision-recall curves of the top 5 most confident predictions for
each entity pairs

Model | MRR among Top-3 | MRR among Top-5
---|---|---
Weston et al. | 0.6010 | 0.6194
Han et al. | 0.5786 | 0.6005
CRE(transformer+ComplEx) | 0.6413 | 0.6588
Table 1: MRR comparison with baseline models Model | MRR among Top-3 | MRR among Top-5
---|---|---
CNN+TransE | 0.6925 | 0.7059
CNN+ComplEx | 0.6681 | 0.6805
LSTM+TransE | 0.6234 | 0.6506
LSTM+ComplEx | 0.6802 | 0.6938
transformer+TransE | 0.6332 | 0.6549
transformer+ComplEx | 0.6413 | 0.6588
Table 2: MRR comparison between different configurations of CRE

For knowledge base model, we explore 2 options:

  * •

TransE with embedding dimension of 50.

  * •

ComplEx Trouillon et al. (2016) with embedding dimension of 25 for the real
part and 25 for the imaginary part.

The dimensions chosen here are based on reports from each model’s respective
paper.

By combining these options, we will explore 6 different model configurations
in total. We use Adam optimizer for training and consider the model to have
converged when the loss on the current epoch is no less than the average loss
of the last ten epochs. Each model is then used to make predictions on the
same held-out test set.

In order to test for statistical significance, we divided the dataset into 3
different subsets randomly and recorded the results of each model on all three
subsets.

###  3.5 Results

####  3.5.1 The best model configuration

We compare the performance of the CRE models across different configurations,
which is illustrated in figure 4. From the results, we can see that the more
sophisticated knowledge base model (ComplEx) outperforms the simpler
alternative (TransE) at lower recall levels (less than 0.25), but under-
performs at higher recall levels. Similarly, for different choices of
sentence-to-CRE encoders, we observe that the more complex transformer encoder
is almost perfect before recall rises above 0.25.

The most consistent configuration, from a quantitative perspective, is the
combination of CNN and TransE. These results are reinforced by the MRR
comparison among different CRE configurations, as shown in table 2. We can see
that the model that achieved the best precision-recall result (CNN+TransE)
also dominates in terms of MRR, whereas the worst results for both metrics are
attributable to the LSTM+TransE configuration.

####  3.5.2 Comparison with baseline models

Figure 3 shows the precision-recall curve comparisons between baseline models
and a CRE-based model with Transformer+ComplEx configuration. Each model’s
confidence interval was obtained via variance among precisions at the same
recall level of 9 different runs: 3 runs for each of 3 different random subset
of the training/testing dataset. The CRE model was able to outperform both
baseline models.

Interestingly, Weston’s approach achieved top-1 prediction accuracy comparable
with what was reported in its original paper, but saw a particularly sharp
drop in precision as the number of predictions examined increases. This
contrast is unsurprising though, because Weston’s approach only re-scores its
most confident predictions, so models trained this way have no capability of
making multiple predictions for a single entity pair.

In addition, we observed the mean reciprocal rank for CRE model is
significantly higher than baseline models, as shown in table 1, which
corresponds to the improvement we saw in terms of precision-recall curve.

####  3.5.3 Qualitative evaluation

Table 3 shows most frequent relations and the number of times each model
concludes one of them to be the most likely relation given an entity pair, as
well as the ground truth. It can be observed that the CRE models tend to
generate a less skewed distribution of these frequent relations compared to
Han’s, which is the better of the two baseline models in terms of quantitative
performance.

Relation Name | Weston et al. | Han et al. | CRE+CNN | CRE+Transformer | Fact
---|---|---|---|---|---
location.contains | 1085 | 1178 | 761 | 840 | 597
person.nationality | 66 | 19 | 189 | 159 | 173
location.containedBy | 198 | 205 | 435 | 346 | 316
people.placeOfBirth | 7 | 0 | 16 | 17 | 91
people.placeOfDeath | 1 | 0 | 0 | 22 | 47
usRepresentative.state | 0 | 0 | 0 | 5 | 11
tvProgram.programCreator | 0 | 0 | 0 | 1 | 2
Table 3: Top relation prediction counts by each model compared to truth

##  4 Discussion

Our experiments from figure 3 clearly demonstrate that models utilizing
contextualized relation embeddings that internalize both relation extractor
modeling and knowledge base modeling tend to perform much better on relation
extraction task than architectures like Weston et al. (2013) and Han et al.
(2018) that join a relation extractor and a knowledge base model in some
arbitrary way. We believe this is due to the fact that the “internal knowledge
base models” within CRE models are context-aware. As a result, CRE models can
take advantage of the contextual information contained in the corpus more
effectively.

We demonstrate the generality and flexibility of the proposed CRE model that
can work with different encoders and knowledge base models. Moreover, it can
be observed from Table 3 that the CRE-models generate less skewed distribution
of frequent relations compared to the baseline models, thus demonstrating that
the CRE model provides robust predictions and works well even with imbalanced
datasets.

It can be observed from Figure 4 that more complex configurations for our
model tend to achieve stellar results in the low recall arena but lose steam
when recall levels are high. This may be explained by the insufficient
training epochs due to resource constraints. As a result, these complex models
did not get the chance to optimize for the less seen prediction targets.
However, it can been seen from Table 3 that the transformer-based CRE model
was able to correctly uncover some rare relations, like _programCreator_ ,
which was missed by the CNN-based CRE model, despite it having better overall
quantitative results. It is reasonable to expect that given a more balanced
dataset with sufficient training time, the transformer-based CRE model may
obtain a better quantitative result than its CNN counterpart.

Since contextual knowledge plays a big role in the performance of CRE models,
it will be interesting to see how such models may perform over prediction on
input texts that are heterogeneous to the New York Times. For example, an
alternative input text source could be a collection of academic publications,
which is of very different genre compared to the New York Times. If our
hypothesis on why these models work well is correct, then we would expect to
see some degradation in performance, though it should be no worse than
context-free approaches. This can be overcome, of course, by utilizing pre-
trained CRE models and finetuning on the new text body, so that the model can
learn about the new context.

##  5 Conclusion and Future Work

In this paper, we introduced a novel contextualized relation embedding (CRE)
model for relation extraction that incorporates knowledge base modeling in a
comprehensive and efficient manner. We demonstrate both empirically and
qualitatively that the CRE model is able to achieve state of the art results
on relation extraction task on the New York times dataset with Freebase as a
knowledge source. We demonstrate the flexibility of the model by using
different encoders and knowledge based modeling schemes which are a testament
to the modular nature of this model, which can easily be upgraded with
alternative configurations with ease. Finally, we showcase that CRE model tend
to generate a less skewed distributions of predicted relations and the model
is robust to imbalances in the dataset.

Interesting directions of future work may include utilizing CRE pre trained
models and evalauting if these models can be finetuned to work in low-resource
situations when the testing set distribution is different than the training
distribution. Another interesting direction may be utilizing pre-trained BERT-
based Devlin et al. (2018) models as sentence encoders and fine-tune them for
the relation extraction task, since such models are already trained on large
amount of textual data and thus may hold contextual knowledge that might be
useful for this task. This may also reveal some hidden connections between
contextualized word embeddings and contextualized relation embeddings.

## References

  * Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008\.  Freebase: A collaboratively created graph database for structuring human knowledge.  In _Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data_ , SIGMOD ’08, page 1247–1250, New York, NY, USA. Association for Computing Machinery.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013.  Translating embeddings for modeling multi-relational data.  In _Advances in Neural Information Processing Systems 26_ , pages 2787–2795. Curran Associates, Inc.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Han et al. (2018) Xu Han, Zhiyuan Liu, and Maosong Sun. 2018.  Neural knowledge acquisition via mutual attention between knowledge graph and text.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _CoRR_ , abs/1310.4546.
  * Mintz et al. (2009) Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009.  Distant supervision for relation extraction without labeled data.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , pages 1003–1011, Suntec, Singapore. Association for Computational Linguistics.
  * Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010.  Modeling relations and their mentions without labeled text.  In _Machine Learning and Knowledge Discovery in Databases_ , pages 148–163, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Trouillon et al. (2016) Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016.  Complex embeddings for simple link prediction.  _CoRR_ , abs/1606.06357.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _CoRR_ , abs/1706.03762.
  * Weston et al. (2013) Jason Weston, Antoine Bordes, Oksana Yakhnenko, and Nicolas Usunier. 2013.  Connecting language and knowledge bases with embedding models for relation extraction.  _CoRR_ , abs/1307.7973.
  * Xu and Barbosa (2019) Peng Xu and Denilson Barbosa. 2019.  Connecting language and knowledge with heterogeneous representations for neural relation extraction.  _CoRR_ , abs/1903.10126.
  * Zeng et al. (2014) Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014.  Relation classification via convolutional deep neural network.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 2335–2344, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.


# Offshore Wind Plant Instance Segmentation Using Sentinel-1 Time Series, GIS,
and Semantic Segmentation Models

Osmar Luiz Ferreira de Carvalho<EMAIL_ADDRESS>Osmar Abílio de
Carvalho Júnior  Anesmar Olino de Albuquerque<EMAIL_ADDRESS>Daniel
Guerreiro e Silva<EMAIL_ADDRESS>
###### Abstract

Offshore wind farms represent a renewable energy source with a significant
global growth trend, and their monitoring is strategic for territorial and
environmental planning. This study’s primary objective is to detect offshore
wind plants at an instance level using semantic segmentation models and
Sentinel-1 time series. The secondary objectives are: (a) to develop a
database consisting of labeled data and S-1 time series; (b) to compare the
performance of five deep semantic segmentation architectures (U-Net, U-Net++,
Feature Pyramid Network - FPN, DeepLabv3+, and LinkNet); (c) develop a novel
augmentation strategy that shuffles the positions of the images within the
time series; (d) investigate different dimensions of time series intervals (1,
5, 10, and 15 images); and (e) evaluate the semantic-to-instance conversion
procedure. LinkNet was the top-performing model, followed by U-Net++ and
U-Net, while FPN and DeepLabv3+ presented the worst results. The evaluation of
semantic segmentation models reveals enhanced Intersection over Union (IoU)
(25%) and F-score metrics (18%) with the augmentation of time series images.
The study showcases the augmentation strategy’s capability to mitigate biases
and precisely detect invariant targets. Furthermore, the conversion from
semantic to instance segmentation demonstrates its efficacy in accurately
isolating individual instances within classified regions—simplifying training
data and reducing annotation effort and complexity.

###### keywords:

instance segmentation , semantic segmentation , renewable energy , wind energy
, deep learning , time series

[icc] organization=University of Brasilia, department=Department of Electrical
Engineering,, addressline=Campus Universitario Darcy Ribeiro, city=Brasilia,
postcode=70910-900, state=Federal District, country=Brazil [unb]
organization=University of Brasilia, department=Departament of Geography,,
addressline=Campus Universitario Darcy Ribeiro, city=Brasilia,
postcode=70910-900, state=Federal District, country=Brazil

##  1 Introduction

Offshore wind energy (OWE) has emerged as a significant and rapidly expanding
contributor to the global power generation portfolio, driven by a compelling
blend of environmental sustainability, technological advancements, and policy
support. Unlike their onshore counterparts, offshore wind plants,
strategically located in oceanic or sea regions, favor larger-scale
constructions, take advantage of the more consistent and stronger winds
available in these open spaces to generate renewable electricity, and reduce
noise pollution by being further away from populated areas [37, 45].
Furthermore, the advent of floating wind turbines will expand offshore wind
energy beyond shallow waters, not relying on the ocean floor and conventional
gravity-based concrete at depths and substantially increasing the potential of
the world’s oceans [3, 31]. These advances also provide a promising renewable
energy source solution to the inherent spatial constraints of densely
populated regions or those with limited land availability.

Therefore, OWE is at a significant inflection point in its evolution due to
technological advances, economies of scale, and increased political support,
making it a more cost-competitive energy source and reducing greenhouse gas
emissions [16]. According to the World Forum Offshore Wind, global offshore
wind capacity in operation surpassed 57 GW at the end of 2022, representing an
additional 9.4 GW during 2022 [41]. China has the highest installed capacity,
with 26.6 GW, followed by the United Kingdom (UK) in second place, reaching
13.6 GW. In 2022, China led the global offshore wind capacity under
construction, with a total capacity of 3.4 GW in Chinese waters. The UK
follows closely in second place with 2.8GW, which includes notable projects
like Dogger Bank A (1.2GW) and Seagreen (1.1GW), both with a capacity in the
gigawatt range (WFO, 2023). The International Energy Agency (IEA) presented
two scenarios for global offshore wind capacity growth by 2040: the State
Policy Scenario, driven by policy targets and falling technology costs, and
the Sustainable Development Scenario, which includes incentives to accelerate
decarbonization efforts in the electricity sector [21]. Under the Sustainable
Development Scenario, global offshore wind capacity will reach approximately
560 GW by 2040, representing a 65% increase compared to the State Policy
Scenario.

However, exponential growth projections on a global scale for offshore wind
energy infrastructures involve several interests and conflicts with legal,
socio-economic, and ecological implications [14]. Despite OWE’s clear
contributions to addressing the global emergence of carbon-neutral energy,
this technology has limitations and encompasses wide-ranging challenges. As
the OWE intensifies and expands, the tendency is to increase conflicts with
nearby coastal communities with interests contrary to the use of marine
environments, such as aquaculture, tourism, navigation, and commercial fishing
[39]. OWE negatively impacts diverse marine life populations due to collision
mortality, disruption of migration corridors, barrier effects, acoustic and
electromagnetic disturbances, ecological function changes, habitat alteration,
and effects on the benthic ecosystem [1, 4, 22, 24]. Besides, OWE growth
between 2020 and 2040 will require significant quantities of raw materials,
including 8.2-14.6 million tons (Mt) of iron, 129-235 Mt of steel, 3.8-25.9 Mt
of concrete, 0.5-1.0 Mt of copper, 0.3-0.5 Mt of aluminum, as well as
substantial amounts of rare earth elements [25].

Monitoring the growth and analyzing the spatial distribution of offshore wind
farms provide valuable information for formulating policies for the efficient
use of available marine space, minimizing conflicts with other activities,
such as maritime routes, fishing grounds, and environmentally sensitive areas.
This spatiotemporal information helps to alert and plan a balance in
generating renewable energy and the sustainable use of marine resources.
However, the on-site monitoring of these large-scale installations, spread
over vast geographic expanses, is logistically complex and economically
prohibitive [23]. In this context, remote sensing technology presents a huge
potential to provide high-quality data for automatically monitoring offshore
wind plants, overcoming the logistical complexities of manual inspections.

Coupled with remote sensing images, recent advances in deep learning models
achieve state-of-the-art object detection and segmentation in digital image
processing [27, 28]. Therefore, deep-learning methods have promoted a
revolution in computer vision and remote sensing, mainly using Convolutional
Neural Networks (CNNs) architectures, obtaining accuracy results that surpass
other approaches [25]. These sophisticated models can learn to distinguish
between different landscape features, differentiate between wind farms and
other structures, and perform these tasks under varying conditions, making
them a promising solution to the detection challenges of offshore wind farms.
Previous studies aimed at detecting wind farms using remote sensing and deep
learning predominantly focused on onshore areas. These onshore wind plant
studies considered different associations of optical images and CNN
architectures: (a) Gaofen-2 images and U-Net [15]; (b) aerial photographs and
comparative analysis between U-Net and LinkNet [29]; (c) aerial photographs
and multiclass reconnaissance network [30]; (d) aerial photographs and R-CNN
Mask [36]; and (e) China-Brazil Earth Resources Satellite (CBERS) 4A scenes
and CNN architecture comparative analysis (U-Net, U-Net++, LinkNet DeepLabv3+,
Feature Pyramid Network - FPN) [5].

In the context of mapping offshore wind farms, the first studies used
traditional techniques considering different methodological approaches, image
data, and study areas: (a) cloud-based geoprocessing algorithm using synthetic
aperture radar (SAR) and the Google Earth Engine (GEE) to detect offshore
infrastructure, including oil platforms and wind turbines in the Gulf of
Mexico [40]; (b) visual saliency detection (VSD) algorithm based on optical
satellite images from various sources (Landsat-5 Thematic Mapper, Landsat-7
Thematic Mapper Plus, Landsat-8 Operational Land Imager, and Sentinel-2 Multi-
Spectral Instrument) in the North Sea and surrounding waters [42]; (c)
percentile-based reduction algorithm using Sentinel-1 SAR time-series images,
constructing a dataset containing over 6,900 turbines in 14 coastal nations
[44]; and (d) random forest (RF) method using Sentinel-1 data on the GEE in
the Yellow Sea of China and the North Sea of Europe [43].

Researcher Thorsten Hoeser led the published research involving offshore wind
plant detection based on deep learning techniques, all using Sentinel-1 data
and object detection methods, considering the following purposes: (a)
development of the DeepOWT dataset with space-time information on offshore
wind energy infrastructure on a global scale [18]; (b) the proposition of
SyntEO, a system that generates large datasets for deep learning, merging
existing and synthetic data through the incorporation of specialized and
structured knowledge [20]; and (c) estimate of the global capacity of offshore
wind turbines based on the analysis of spatiotemporal patterns of offshore
wind turbines, revealing substantial growth of the sector due to the European
Union, China, and the United Kingdom [19].

Despite significant advances in the study of offshore wind farms using object
detection techniques, there is an open opportunity to employ semantic and
instance segmentation methods. Object detection aims to identify the location
of various objects in an image and establish their bounding boxes. In
opposition, semantic and instance segmentation aim at a pixel-level
classification, assigning a class and instance label. This paper provides
methodological and practical novelties, in which the most significant
contributions are:

  1. 1.

Comparative analysis of semantic segmentation methods: This research performs
and compares semantic segmentation algorithms in the context of offshore wind
farms, unlike using object detection techniques.

  2. 2.

Publicly available dataset: Construction of a robust dataset available for
public use, comprising more than 5,000 patches, each with dimensions of 128 x
128 x time sequences (1, 5, 10, and 15 images). This unique dataset serves as
a valuable resource for future research and development of algorithms in
detecting and segmenting offshore wind farms using time series as training
data.

  3. 3.

Simplified instance segmentation: By employing semantic segmentation, a more
straightforward deep learning method, we streamline the process to achieve
instance-level segmentation. Coupling this approach with Geographic
Information System (GIS) tools allows us to detect and segment wind turbines
more effectively, setting a precedent for future research and applications.

##  2 Materials and methods

###  2.1 Dataset Construction

The study area focused on the United Kingdom (UK) (Figure 1), a global pioneer
in offshore wind technology [17]. With the second largest offshore wind
capacity worldwide, the UK presents a unique and invaluable landscape for
understanding the dynamics and complexities of these large-scale renewable
energy installations, including significant spatial, seasonal, and regional
variations [34]. Therefore, this study carefully selected ten regions
distributed across the UK, where each location has distinct environmental
conditions, wind farm sizes, and technology configurations, offering a
comprehensive and diverse set of scenarios for analysis. These regions host
substantial offshore wind farms, contributing significantly to the UK’s
renewable energy portfolio. Given their geographical, technological, and
environmental attributes, the variety in data from these sites provides a rich
resource for understanding the potential and challenges of automated offshore
wind plant detection using remote sensing data, besides evaluating the
robustness, accuracy, and scalability of semantic segmentation models learning
under a range of conditions. Furthermore, we aim to identify critical factors
and patterns that may influence the effectiveness of our proposed methods.

Figure 1: Location of the sites containing offshore wind plants.

In contrast to the challenge of detecting onshore wind farms using SAR images
– due to the complexities arising from the lack of evident distinction in the
terrestrial environment – offshore wind farms have larger dimensions, and over
the vast ocean produces a high-backscatter pattern that facilitates their
detection. Therefore, this research acquired images from the SAR Sentinel-1
mission developed by the European Space Agency (ESA), which comprises a
constellation of two polar-orbiting satellites, Sentinel-1A and Sentinel-1B.
These sensors operate in the 5.4 GHz C-band frequency and capture synthetic
aperture radar (SAR) images in VV (vertical transmission, vertical reception)
and VH (vertical transmission, horizontal reception) polarizations, with free
access on the Copernicus Open Access Hub. The data specifications were
Interferometric Wide Swath (IW) images with VV polarization in Level-1 Ground
Range Detected (GRD) format. Using VV polarization is due to its lower noise
influence and ability to discern better the target characteristics in the sea
[33, 44]. The analysis encompassed a time series of up to 15 SAR images
captured in 2022 and 2023. The image pre-processing used the Sentinel
Application Platform (SNAP) software, including the following steps [13]: (a)
apply orbit file, (b) thermal noise removal, (c) calibration to convert the
digital pixel values to radiometrically calibrated SAR backscatter, (d)
speckle filtering using the Lee Sigma filter, (e) range-Doppler terrain
correction utilizing the Global Earth Topography and Sea Surface Elevation at
30 arc-second resolution data, and (f) conversion of backscattering values to
decibels (dB).

###  2.2 Ground Truth Generation

In the ground truth annotation, the inconsistent sunlight reflection of the
metallic structure of the wind farms in each image was a significant
challenge, making the delineation process complex (Figure 2). Despite the
noise, many pixels, particularly those in the wind farm core, exhibited
consistently high brightness across time series. Taking advantage of this
consistency, we create a mask by averaging all images in the time series
(composite image) and classifying high-backscatter elements that exceed 0 in
the VV band as positive. This approach resembles the proposal developed by
[44]. The data refinement used vector format (shapefile) in the ArcGIS
software, enabling manual noise elimination. In this annotation task, the
optical images served as a support to differentiate the target from other
elements, such as oil platforms or boats.

Offshore wind farm detection using time-series data can overcome difficulties
in detecting wind farms, increasing the chances of obtaining clear and
discernible images of wind farms and removing floating or temporarily mobile
objects. Therefore, time series data strengthens the learning process of deep
architectures to mitigate problems and ensure reliable results, allowing us to
apprehend the invariant behavior of wind farms face to other moving features
or noise and identify their morphological variations due to backscattering
patterns.

Figure 2: Time series of 15 images showing the offshore wind plants.

###  2.3 Semantic Segmentation Dataset Generation

Sample preparation containing the image and its corresponding ground truth
mask is crucial to train semantic segmentation models. This process begins by
breaking down each full-size image into smaller, fixed-size pieces, known as
patches. The spatial distribution and cutout of the patches used the
methodology developed by [6], which employs user-supplied central points
(shapefile format) to delineate the frame. Strategic choice of patches assures
an efficient data selection rather than systematic clipping through the
sliding window. This study selected a patch size of 128x128 pixels, yielding
4947 patches and encompassing 2910 distinct offshore wind plants. The patch
count outweighs the number of wind farms, aiming to capture wind plants under
various scenarios to bolster the robustness of the dataset. Additionally, the
incorporation of background element patches without the presence of wind farms
enriched the overall data.

The training, validation, and testing split considered distinct sets to avoid
overlap. Therefore, six of the ten chosen areas were allocated for training,
two for validation, and the other two for testing. Table 1 lists the
characteristics of each location, including its assigned data set, the number
of patches, and the count of wind plants. Table 2 lists the synthesized
characteristics of the dataset. This careful distribution ensures balanced
exposure to various conditions and scenarios, enhancing the dataset’s overall
efficacy and utility.

Table 1: Detailed description of the dataset, including the number of instances, patches, location of data collection, data partitioning into training, validation, testing subsets, and the time interval from the first and last image in the time series. Location | # of wind plants | # of patches | Split | Time interval
---|---|---|---|---
3°36’9,645”W 54°3’2,492”N | 374 | 652 | Train | 14/08/22 to 29/01/23
3°26’37,919”W 53°29’28,928”N | 269 | 565 | Train | 14/08/22 to 29/01/23
0°9’3,647”E 53°48’23,561”N | 70 | 131 | Train | 16/08/22 to 31/01/23
0°54’38,874”E 53°16’50,015”N | 474 | 611 | Val | 16/08/22 to 31/01/23
1°46’53,292”E 53°55’11,708”N | 308 | 402 | Test | 16/08/22 to 31/01/23
1°22’3,046”E 51°33’52,405”N | 387 | 903 | Train | 16/08/22 to 31/01/23
1°57’20,788”E 51°51’40,077”N | 220 | 557 | Test | 16/08/22 to 31/01/23
2°31’22,737”E 52°12’26,383”N | 96 | 144 | Train | 16/08/22 to 31/01/23
2°58’19,339”E 51°39’41,574”N | 592 | 708 | Val | 16/08/22 to 31/01/23
0°16’34,541”W 50°40’11,965”N | 120 | 274 | Test | 09/08/22 to 17/02/23
Total | 2910 | 4947 | - | -
Table 2: Synthesized table with the number of unique instances and number of patches for each set of data. Set | # of wind plants | # of patches
---|---|---
Train | 1,196 (41.1%) | 2,395 (48.41%)
Val | 1,066 (36.63%) | 1,319 (26.27%)
Test | 648 (22.27%) | 1,233 (24.92%)

###  2.4 Deep Learning approach

####  2.4.1 Deep learning models

This study compared a broad range of deep learning architectures known for
their proficiency in semantic segmentation tasks. Our selection included
Deeplabv3+ [10], U-Net [35], U-Net++ [46], Feature Pyramid Network (FPN) [26],
and LinkNet [9]:

  * 1.

Deeplabv3+: This architecture introduces an encoder-decoder structure to
refine the segmentation results, especially along object boundaries. At its
core, it uses atrous convolution with atrous spatial pyramid pooling (ASPP) to
robustly capture multi-scale context.

  * 2.

U-Net: This architecture has a U-shaped model. The encoder captures context by
down-sampling the input, while the decoder enables precise localization
through a symmetric up-sampling path. A critical feature of U-Net is the
presence of skip connections between encoder and decoder layers at
corresponding resolutions, facilitating the fusion of low-level feature maps
with high-level ones, thereby preserving fine-grained details during up-
sampling.

  * 3.

U-Net++: This architecture leverages the U-Net by introducing nested and dense
skip architecture. The nested structure facilitates feature reuse among
different paths, while the dense skip connections provide more direct paths
for gradients during back-propagation, improving feature propagation and model
performance. This robust design reduces the semantic gap between the encoder’s
and decoder’s feature maps, leading to better feature fusion and enhanced
segmentation performance.

  * 4.

FPN: This architecture is a robust object detection framework that tackles
multi-scale object detection challenges, which was also adapted for the
semantic segmentation task. It constructs a multi-scale, pyramidal hierarchy
of feature maps by incorporating a top-down architecture and lateral
connections into a conventional CNN. The bottom-up pathway extracts a multi-
scale feature hierarchy from a single-scale input. The top-down pathway
enhances the resolution of features, utilizing both the feed-forward feature
and its corresponding feature from the bottom-up pathway. Lateral connections
merge feature maps from both pathways.

  * 5.

LinkNet: This model uses a simplified encoder-decoder structure for tasks such
as image segmentation. Its power lies in proficiently restoring intricate
image details from high-level features. This balancing act between
computational efficiency and segmentation precision makes it an asset for
various vision tasks. The encoder extracts features, and the decoder,
paralleling the encoder’s architecture, reconstructs spatial information,
preserving both the context and details of the image.

Moreover, the EfficientNet-B7 was the backbone of all models. EfficientNet
[38] is a state-of-the-art CNN offering superior accuracy and computational
efficiency. What distinguishes it is its compound scaling method, which
uniformly scales all dimensions of the network’s depth (number of layers),
width (number of channels), and resolution (input image size) while
maintaining a balanced network structure. The foundational EfficientNet-B0
model uses AutoML and reinforcement learning as the baseline network. The
subsequent EfficientNet variants (B1-B7) are improved versions of this
baseline model, with each version delivering progressively higher accuracy
while adhering to the compound scaling rule.

The hyperparameters were the same for all models to ensure a consistent
comparison: (1) Adam optimizer, (2) learning rate of 0.001, (3) batch size of
20, and (4) 150 epochs. Model selection considered the cross-entropy loss in
the validation set, saving the parameters that led to the smallest validation
loss for each architecture for further statistical analysis.

####  2.4.2 Novel data augmentation strategy

While time series typically depend on the data frames’ sequential order, as
for agriculture and phenology studies, our research focuses on a time-
invariant target. In this context, the image sequence within the time series
is irrelevant and could introduce bias. If a model is invariably trained on an
identical sequence of images, there is a risk of inadvertently learning
specific patterns from that sequence. This arrangement could create problems
concerning its capacity to generalize and perform accurately on diverse
datasets.

We propose a novel data augmentation technique tailored for semantic
segmentation tasks to circumvent this potential pitfall. This innovative
strategy involves randomly shuffling the order of the images in the time
series during the model’s training process. This measure ensures that the
model does not unduly familiarize itself with any specific sequence and
consequently improves its generalization ability. Our approach, thus, paves
the way for a more robust and reliable model that can effectively handle
diverse time-invariant datasets. A similar approach has been made, for
instance segmentation of center pivots using optical imagery, in which the
main objective was to avoid the interference of clouds in the model [2].

####  2.4.3 Experiments

The experiment manipulated the time-series interval, analyzing datasets with
1, 5, 10, and 15 images. Even with the preservation of training, validation,
and testing sampling sites, the analysis established different time-series
periods. The proposed augmentation approach randomly flips the order of images
in each dataset (training, validation, and test). However, to maintain
consistency in comparing models’ performance, we performed a single shuffle in
the validation and test samples, preserving their ordering across all
evaluated models. Furthermore, the efficiency analysis of the new augmentation
strategy compared the best-performing model with and without its
implementation in the time series.

####  2.4.4 Sliding Windows approach

Practical applications of semantic segmentation for remote sensing data often
rely on a sliding window approach. This method facilitates the comprehensive
classification of images, typically much larger than the patches used for
model training. The sliding window approach involves a systematic shift of a
window, equivalent to the size of the training patches, across the image in
both x and y dimensions, using a predetermined stride value. The largest
permissible stride value corresponds to the window’s dimension. Stride values
less than this result in overlapping pixels. These overlaps enhance the
resulting classification by mitigating the discontinuities between adjacent
frames, creating a smoother and more coherent segmentation output [11]. This
technique ensures accurate and detailed semantic segmentation across large
remote-sensing datasets.

####  2.4.5 Metrics

This research considered two types of accuracy assessment metrics: per-pixel
and per-object approaches. Per-pixel metrics are valuable to understand how
many prediction pixels matched the reference pixels in patches from the test
set. In this context, the accuracy analysis used the following metrics:
overall accuracy, precision, recall, f-score, and Intersection over Union
(IoU) (Table 3).

Metric | Formula
---|---
Overall Accuracy | $(TP+TN)/(TP+TN+FP+FN)$
Precision | $TP/(TP+FP)$
Recall | $TP/(TP+FN)$
f-score | $2\times((\text{Precision}\times\text{Recall})/(\text{Precision}+\text{Recall}))$
IoU | $TP/(TP+FP+FN)$
Table 3: Mathematical formulation for overall Accuracy (OA), Precision,
Recall, F-score, and Intersection over Union (IoU), considering True Positives
(TP), True Negative (TN), False Positives (FP), and False Negatives (FN).

The per-polygon metrics quantify the number of objects classified correctly,
establishing a correspondence between a segmentation’s mapped and reference
polygons. This approach minimizes the effects of errors in target contours and
focuses on matching object counts. Per-polygon performance analysis quantified
true positives (TP), false positives (FP), and false negatives (FN) and
calculated the following metrics: Overall Quality (TP/(TP+FP+FN)), Correction
(TP/(TP+FP)), and Completeness (TP/(TP+FN)). Designating a polygon as a true
positive assumed an IoU criterion greater than 50% between predicted and
reference polygons. Therefore, the present study used per-pixel metrics to
evaluate the training and test steps using samples with dimensions of 128x128
pixels and per-polygon analysis to analyze the complete image after its
reconstruction using the sliding window method.

##  3 Results

###  3.1 Semantic Segmentation Metrics

This investigation evaluated the performance of five deep-learning semantic
segmentation architectures in different time-series compositions (Table 4).
The first evident pattern is that the main metric (IoU) consistently decreases
when reducing the number of images in the time series. One of the main reasons
for this behavior is that, as the reflection of wind turbines varies in
different images, using more images establishes a more consistent pattern,
improving understanding of the deep-learning model. Furthermore, the time-
invariant behavior of wind farms allows us to distinguish them from other
mobile features such as ships.

Overall, LinkNet presented the best results, slightly better than U-Net++ and
U-Net. These models not only offer a more straightforward path to restore
images to their original size, but their architectures also ensure the
retention of high-resolution features during upsampling. This characteristic
makes them more suitable for tasks requiring detailed semantic segmentation,
as with consistently small, uniformly scaled objects. Contrarily, the FPN
(Feature Pyramid Network) and DeepLabv3+ models yielded significantly poorer
results, potentially due to the inherent characteristics of the object under
study, which were consistently small. DeepLabv3+ typically works best when
applied to datasets featuring a range of scales and sizes. Likewise, the FPN
model has a mechanism that leverages lateral connections between diverse
feature maps, a less beneficial approach when dealing with uniformly small,
unvarying objects.

Then, an effectiveness analysis of the proposed augmentation strategy
(temporal shuffling) took place by comparing the results with and without its
application for the best-training model (LinkNet) and the time series with 5,
10, and 15 images. Table 5 indicates that the results for the different time
series obtained without applying the proposed augmentation strategy show a
significant loss of accuracy, reaching around 18% of the F-score and 25% of
IoU.

The time series with 15 images exhibited a lower performance, resulting in a
24.45% decrease in IoU compared to the trained model with the proposed
augmentation. Similarly, the time series with ten images showed a decline of
26.2%, while the time series with five images experienced a 23.9% decrease in
performance. These findings highlight the benefits of our augmentation
strategy and its ability to enhance the model’s performance across various
time series configurations.

Figure 3 provides a visual demonstration of the predictive capabilities of the
LinkNet model when subjected to various time-series configurations. The
observed variation within the predictions is notably minimal when utilizing
the proposed augmentation strategy. Despite the significant IoU difference,
the models identify the targeted objects proficiently. The leading cause of
this robust performance is the small size of the objects in consideration.
Even a minuscule shift in a pixel or two can instigate a considerable impact
on the overall performance metrics, demonstrating the fine-tuned sensitivity
of the models.

Figure 3: Original SAR images (images), ground truth (GT), and time series
predictions with the LinkNet model using 15 (TS 15), 10 (TS 10), 5 (TS 5), and
1 (TS 1) images. * Denotes the time series predictions without the proposed
augmentation strategy.

Notably, deploying more images led to remarkably consistent predictions and
demonstrated a reduced propensity for misclassification of other high-
backscatter targets. Conversely, a marked decrease in the model’s ability to
discern certain features occurred when the augmentation strategy was not
applied. This result indicates the model’s tendency to memorize patterns
within the time series that do not find representation within the test set.
Therefore, the proposed augmentation strategy offers an ability to improve the
predictive ability and resilience of the model in maintaining object
recognition, providing perspectives on future reliable predictions of other
targets.

Table 4: Semantic segmentation results considering the overall accuracy (OA), precision, recall, f-score, and intersection over union (IoU) metrics for the time series with 15, 10, 5, and 1 image. Model | OA | Precision | Recall | F-score | IoU
---|---|---|---|---|---
Time Series with 15 images
LinkNet | 99.96 | 87.62 | 92.29 | 89.90 | 81.65
U-Net | 99.96 | 87.75 | 91.59 | 89.63 | 81.21
U-Net++ | 99.96 | 87.43 | 92.10 | 89.71 | 81.33
DeepLabv3+ | 99.93 | 79.82 | 86.72 | 83.13 | 71.13
FPN | 99.92 | 79.81 | 85.01 | 82.33 | 69.96
Time Series with 10 images
LinkNet | 99.95 | 86.65 | 89.11 | 87.86 | 78.35
U-Net | 99.95 | 86.04 | 88.55 | 87.28 | 77.42
U-Net++ | 99.94 | 84.83 | 89.44 | 87.07 | 77.11
DeepLabv3+ | 99.92 | 78.82 | 83.60 | 81.14 | 68.26
FPN | 99.91 | 75.76 | 85.47 | 80.32 | 67.11
Time Series with 5 images
LinkNet | 99.93 | 82.94 | 86.03 | 84.45 | 73.09
U-Net | 99.93 | 83.29 | 85.54 | 84.40 | 73.01
U-Net++ | 99.93 | 83.06 | 86.47 | 84.73 | 73.51
DeepLabv3+ | 99.90 | 73.76 | 82.88 | 78.06 | 64.01
FPN | 99.90 | 72.94 | 83.00 | 77.64 | 63.46
Single Image
LinkNet | 99.90 | 76.63 | 78.42 | 77.52 | 63.29
U-Net | 99.90 | 71.29 | 85.51 | 77.76 | 63.61
U-Net++ | 99.90 | 75.34 | 79.08 | 77.16 | 62.82
DeepLabv3+ | 99.88 | 67.70 | 80.38 | 73.50 | 58.10
FPN | 99.88 | 69.76 | 77.96 | 73.63 | 58.27
Table 5: Semantic segmentation results using the LinkNet model for time series with 15, 10, and 5 images without the proposed augmentation strategy. “OA” is overall accuracy, and “IoU” is Intersection over Union. Time Series | OA | Precision | Recall | F-score | IoU
---|---|---|---|---|---
15 | 99.90 | 90.89 | 60.67 | 72.77 | 57.20
10 | 99.86 | 63.87 | 73.96 | 68.55 | 52.15
5 | 99.89 | 87.18 | 53.39 | 66.22 | 49.50

###  3.2 Full image classification

The full image reconstruction of the semantic segmentation used the sliding
window approach. The analysis of the per-object metrics considered the region
with the highest concentration of offshore wind farms and the different time
series composed of 1, 5, 10, and 15 images (Table 6). The results indicated a
significant initial improvement with increasing images from 1 to 5, decreasing
the number of false positives from 14 to 2 objects, and nullifying the
presence of 2 false negatives. Among the two remaining false positives with
five images, the increase to 10 or 15 images eliminated one. Therefore, the
subsequent transitions from 5 to 10 and 15 images show minor improvements
after reaching a certain threshold (5 images in this case).

| Number of images used in the time series
---|---
1 | 5 | 10 | 15
TP | 296 | 297 | 297 | 297
FP | 14 | 2 | 1 | 1
FN | 1 | 0 | 0 | 0
Overall Quality | $\frac{296}{296+14+1}\approx 0.954$ | $\frac{297}{297+2+0}\approx 0.993$ | $\frac{297}{297+1+0}\approx 0.997$ | $\frac{297}{297+1+0}\approx 0.997$
Correctness | $\frac{296}{296+14}\approx 0.955$ | $\frac{297}{297+2}\approx 0.993$ | $\frac{297}{297+1}\approx 0.997$ | $\frac{297}{297+1}\approx 0.997$
Completeness | $\frac{296}{296+1}\approx 0.997$ | $\frac{297}{297+0}=1$ | $\frac{297}{297+0}=1$ | $\frac{297}{297+0}=1$
Table 6: Metrics and Quality Measures for Time Series Images

The raster-to-polygon conversion of the semantic segmentation results in
Geographic Information Systems (GIS) proved highly efficient in representing
the offshore wind farm at the instance level. Figure 4 shows the results in
polygon vectors of offshore wind plants obtained by the Linknet model
considering a 15-image time series. The isolated behavior of wind farms, as
opposed to other overlay or contact targets such as cars [32], provides a
semantic-to-instance segmentation conversion facility.

Figure 4: Semantic segmentation results for a large image using the sliding
window approach for a region with a high concentration of offshore wind farms.
Red polygons derived through the raster-to-polygon conversion of the LinkNet
prediction superimposed on Sentinel-1 VV polarization image delineated the
offshore wind plant features.

Figure 5 demonstrates the errors from using a single image in predicting
offshore wind plants marked in blue. Although the model demonstrated
competence in recognizing brightness and identifying most targets, it also
generated many false positives regarding the other models using time series.

Figure 5: Results using a single image from the time series and their
corresponding errors. Correct results are in red, and incorrect
classifications are in blue.

##  4 Discussion

This research sought to develop a new approach using time series data with
different CNN models to detect offshore wind farms. The developed strategy
comprised generating a data set with different amounts of images and an
innovative augmentation procedure that promotes the shuffling of the
chronological sequence, which allows learning with different sequential
contexts.

###  4.1 Dataset Importance

Anticipating a significant increase in dependence on renewable energies in the
coming years and offshore wind farm growth worldwide, the development of
automatic inspection methods is critical. Understanding regulatory and
environmental constraints and spatial patterns will be crucial to managing
this projected growth effectively. Considering these challenges, the offshore
wind farm dataset developed innovates by being the first in deep learning
semantic segmentation and having multi-channels referring to time series
images, differentiating itself from other databases for object detection and
using only a channel. The proposed dataset considered ten regions of the
United Kingdom (a global leader in offshore wind energy), containing more than
five thousand patches. The dataset also includes different time-series
components, containing up to fifteen images, ensuring a broad scope of
scenarios, backscatter patterns, mobile targets, and noise. Despite its UK-
centric composition, the dataset can be applied to other locations and, if
necessary, adjusted by entering new data based on our guidelines.

###  4.2 Time Series augmentation effect

This research employed an innovative augmentation strategy explicitly designed
for semantic segmentation models of invariant targets, involving a randomized
rearrange of image positions within a time series and offering a unique and
efficacious way to mitigate biases associated with ordering images within the
temporal sequence. Thus, sequential backscatter variations (at a given
chronological position) in the time series are shuffled in the samples,
allowing the adaptation of the training model for the invariant-target
segmentation in other unknown Sentinel-1 images. The proposed augmentation
strategy also presents a promising approach to detect other invariant targets
using radar imagery, such as buildings and fixed infrastructure. However, this
strategy may not be universally beneficial when the chronological order of the
frames is significant due to the disruption of temporal ordering, such as in
studies based on phenology.

The comparative analysis of the interval dimension of the time series
demonstrated that the data incorporation promoted benefits. Inserting target
backscatter variations helps the algorithm navigate several degrees of
complexity, promoting a deeper understanding of the intricate relationships.
The broader time series eliminates noise and mobile elements and reiterates
the mapped features. For example, a ship can exhibit similar patterns and
confusion when analyzing a single image. However, the model improves
perception and accuracy as the time series increases. Nevertheless, increasing
the time series without limits can also lead to complications. For example,
extending the time series too far into the past may include a period when the
wind farm was not installed, negatively affecting model performance.

###  4.3 Segmentation model analysis

The accuracy comparison among the semantic segmentation models for offshore
wind farm detection showed a group composed of LinkNet, U-Net, and U-Net++
that achieved better performance than a second group, composed of DLv3+ and
FPN, with lower precision metrics. The need for more investigation hampers the
comparison with other deep-segmentation studies for detecting offshore wind
farms. Among the few studies, [43] performed a segmentation based on machine
learning methods for offshore wind farms with disadvantages compared to deep-
learning approaches widely described in the literature, particularly the
enhanced understanding of context and the scalability offered by CNNs.

Although this study focused on offshore wind farms with SAR images, other deep
semantic segmentation investigations have been developed on onshore wind farms
using high-resolution optical images [5, 15, 29, 30, 36]. Despite the
differences, the results between the models for both situations, using optical
and radar images, show apparent similarities. The results by [5] that detect
onshore wind farms using CBERS optical images present the same architecture
results based on IoU and F-score, evidencing the LinkNet architecture with
Eff-B7 backbone as the best model, followed by the U-Net and U-Net++ models,
while DLv3+ and FPN showed more than a 3% difference from the three best
models. [29], comparing LinkNet and U-Net using aerial images, also observed
better performance of LinkNet. The other onshore wind farm studies did not
make comparisons, using only one architecture. A possible explanation for the
difference between the models is that the LinkNet, U-Net, and U-Net++ methods
present better results in capturing fine details and preserving spatial
information, which becomes suitable for small objects with fixed dimensions.
On the other hand, DLv3+ and FPN methods offer better performance for targets
with size and shape variations.

Furthermore, the research demonstrated a simple procedure for semantic-to-
instance segmentation conversion using GIS, facilitated by the absence of
overlapping objects. The occurrence of having only isolated targets avoids the
need to use other procedures when the targets present agglutination, such as
edges to isolate the interior of objects and multitask learning [32] or
multiclass learning [7, 12]. The dataset complies with the COCO annotation
guidelines using a specific program that automatically converts GIS data to
the COCO annotation format, allowing the use of advanced Mask-RCNN models [6,
8].

##  5 Conclusion

This research presented a pioneering application of deep segmentation for
offshore wind farms based on the Sentinel-1 time series, proposed a novel
augmentation strategy, compared five CNN architectures, and used GIS to obtain
an instance-level segmentation of these features. Sentinel-1 radar images
revealed a valuable contrast for detecting offshore wind farms. The offshore
wind plant dataset elaborated in this study is not limited to a single date
(single channel), using a time series (multi-channel) that, with the
application of the augmentation strategy, presents a chronological sequence
scrambling to increase the detection potential of invariant targets in unknown
images. Time series instead of a single image favors eliminating other mobile
high-backscatter features (e.g., ships) and reinforcing differences with other
fixed objects (e.g., oil platforms). In all semantic segmentation models, the
IoU and F-score metrics consistently increase as the number of images in the
time series increases. The LinkNet achieved the best results, slightly
surpassing U-Net++ and U-Net, while the FPN and DeepLabv3+ models yielded the
worst results. Using the novel augmentation strategy represented a significant
improvement in accuracy, increasing by around 18% of the F-score and 25% of
the IoU in the different time series with 5, 10, and 15 images. These results
demonstrate that the proposed augmentation strategy is a promising method to
minimize biases in detecting invariant targets. Per-object accuracy metrics
allow the evaluation of a balance in determining the number of images in the
time series and improving performance to optimize the model, reaching 5 or 10
images in this investigation. The semantic-to-instance segmentation conversion
is highly effective, as the offshore wind plants are isolated without contact,
promoting advantages such as simplified training data and accurate semantic
segmentation methods. Therefore, as the world’s dependence on renewable energy
grows, this research proves the potential of deep learning in detecting
offshore wind farms, which is essential to improve the management and
sustainability of natural resources.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Data Availability Statement

The dataset and code can be obtained by contacting the corresponding author
upon reasonable request.

## Acknowledgments

The authors are grateful for financial support from the CNPq fellowship (Osmar
Abílio de Carvalho Júnior). This study was partly financed by the Coordination
for the Improvement of Higher Education Personnel (CAPES) – Finance Code 001.
Finally, the authors acknowledge the contribution of anonymous reviewers.

## References

  * Abramic et al. [2022] Abramic, A., Cordero-Penin, V., & Haroun, R. (2022).  Environmental impact assessment framework for offshore wind energy developments based on the marine good environmental status.  Environmental Impact Assessment Review, 97, 106862. doi:10.1016/j.eiar.2022.106862.
  * de Albuquerque et al. [2021] de Albuquerque, A. O., de Carvalho, O. L. F., e Silva, C. R., Luiz, A. S., de Bem, P. P., Gomes, R. A. T., Guimarães, R. F., & de Carvalho Júnior, O. A. (2021).  Dealing with clouds and seasonal changes for center pivot irrigation systems detection using instance segmentation in sentinel-2 time series.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14, 8447–8457. doi:10.1109/JSTARS.2021.3104726.
  * Bashetty & Ozcelik [2021] Bashetty, S., & Ozcelik, S. (2021).  Review on dynamics of offshore floating wind turbine platforms.  Energies, 14, 6026. doi:10.3390/en14196026.
  * Bergström et al. [2014] Bergström, L., Kautsky, L., Malm, T., Rosenberg, R., Wahlberg, M., Capetillo, N. Å., & Wilhelmsson, D. (2014).  Effects of offshore wind farms on marine wildlife—a generalized impact assessment.  Environmental Research Letters, 9, 034012. doi:10.1088/1748-9326/9/3/034012.
  * de Carvalho et al. [2023] de Carvalho, O. L. F., de Carvalho Junior, O. A., de Albuquerque, A. O., Orlandi, A. G., Hirata, I., Borges, D. L., Gomes, R. A. T., & Guimarães, R. F. (2023).  A data-centric approach for wind plant instance-level segmentation using semantic segmentation and gis.  Remote Sensing, 15, 1240. doi:10.3390/rs15051240.
  * de Carvalho et al. [2022a] de Carvalho, O. L. F., de Carvalho Júnior, O. A., de Albuquerque, A. O., Santana, N. C., Borges, D. L., Gomes, R. A. T., Guimarães, R. F. et al. (2022a).  Panoptic segmentation meets remote sensing.  Remote Sensing, 14, 965. doi:10.3390/rs14040965.
  * de Carvalho et al. [2022b] de Carvalho, O. L. F., de Carvalho Júnior, O. A., de Albuquerque, A. O., Santana, N. C., Guimarães, R. F., Gomes, R. A. T., & Borges, D. L. (2022b).  Bounding box-free instance segmentation using semi-supervised iterative learning for vehicle detection.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15, 3403–3420. doi:10.1109/JSTARS.2022.3169128.
  * Carvalho et al. [2020] Carvalho, O. L. F. d., de Carvalho Junior, O. A., Albuquerque, A. O. d., Bem, P. P. d., Silva, C. R., Ferreira, P. H. G., Moura, R. d. S. d., Gomes, R. A. T., Guimaraes, R. F., & Borges, D. L. (2020).  Instance segmentation for large, multi-channel remote sensing imagery using mask-rcnn and a mosaicking approach.  Remote Sensing, 13, 39. doi:10.3390/rs13010039.
  * Chaurasia & Culurciello [2017] Chaurasia, A., & Culurciello, E. (2017).  Linknet: Exploiting encoder representations for efficient semantic segmentation.  In 2017 IEEE visual communications and image processing (VCIP) (pp. 1–4).  IEEE.  doi:10.1109/VCIP.2017.8305148.
  * Chen et al. [2018] Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018).  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV) (pp. 801–818).  doi:10.1007/978-3-030-01234-2_49.
  * Costa et al. [2021] Costa, M. V. C. V. d., Carvalho, O. L. F. d., Orlandi, A. G., Hirata, I., Albuquerque, A. O. d., Guimarães, R. F., Gomes, R. A. T., Júnior, O. A. d. C. et al. (2021).  Remote sensing for monitoring photovoltaic solar plants in brazil using deep semantic segmentation.  Energies, 14, 2960. doi:10.3390/en14102960.
  * De Carvalho et al. [2022] De Carvalho, O. L., de Carvalho Júnior, O. A., de Albuquerque, A. O., Santana, N. C., & Borges, D. L. (2022).  Rethinking panoptic segmentation in remote sensing: A hybrid approach using semantic segmentation and non-learning methods.  IEEE Geoscience and Remote Sensing Letters, 19, 1–5. doi:10.1109/LGRS.2022.3172207.
  * Filipponi [2019] Filipponi, F. (2019).  Sentinel-1 grd preprocessing workflow.  In International Electronic Conference on Remote Sensing (p. 11).  MDPI.  doi:10.3390/ecrs-3-06201.
  * Glasson et al. [2022] Glasson, J., Durning, B., Welch, K., & Olorundami, T. (2022).  The local socio-economic impacts of offshore wind farms.  Environmental Impact Assessment Review, 95, 106783. doi:10.1016/j.eiar.2022.106783.
  * Han et al. [2018] Han, M., Wang, H., Wang, G., & Liu, Y. (2018).  Targets mask u-net for wind turbines detection in remote sensing images.  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 42, 475–480.
  * Hasager et al. [2015] Hasager, C. B., Mouche, A., Badger, M., Bingöl, F., Karagali, I., Driesenaar, T., Stoffelen, A., Peña, A., & Longépé, N. (2015).  Offshore wind climatology based on synergetic use of envisat asar, ascat and quikscat.  Remote Sensing of Environment, 156, 247–263. doi:10.1016/j.rse.2014.09.030.
  * Higgins & Foley [2014] Higgins, P., & Foley, A. (2014).  The evolution of offshore wind power in the united kingdom.  Renewable and sustainable energy reviews, 37, 599–612. doi:10.1016/j.rser.2014.05.058.
  * Hoeser et al. [2022] Hoeser, T., Feuerstein, S., & Kuenzer, C. (2022).  Deepowt: A global offshore wind turbine data set derived with deep learning from sentinel-1 data.  Earth System Science Data, 14, 4251–4270. doi:10.5194/essd-14-4251-2022.
  * Hoeser & Kuenzer [2022a] Hoeser, T., & Kuenzer, C. (2022a).  Global dynamics of the offshore wind energy sector monitored with sentinel-1: Turbine count, installed capacity and site specifications.  International Journal of Applied Earth Observation and Geoinformation, 112, 102957. doi:10.1016/j.jag.2022.102957.
  * Hoeser & Kuenzer [2022b] Hoeser, T., & Kuenzer, C. (2022b).  Synteo: Synthetic dataset generation for earth observation and deep learning–demonstrated for offshore wind farm detection.  ISPRS Journal of Photogrammetry and Remote Sensing, 189, 163–184. doi:10.1016/j.isprsjprs.2022.04.029.
  * International Energy Agency (2019) [IEA] International Energy Agency (IEA) (2019).  Offshore Wind Outlook 2019.  Technical Report International Energy Agency Paris, France.  URL: https://iea.blob.core.windows.net/assets/495ab264-4ddf-4b68-b9c0-514295ff40a7/Offshore_Wind_Outlook_2019.pdf.
  * Lane et al. [2020] Lane, J. V., Jeavons, R., Deakin, Z., Sherley, R. B., Pollock, C. J., Wanless, R. J., & Hamer, K. C. (2020).  Vulnerability of northern gannets to offshore wind farms; seasonal and sex-specific collision risk and demographic consequences.  Marine Environmental Research, 162, 105196. doi:10.1016/j.marenvres.2020.105196.
  * Leung & Yang [2012] Leung, D. Y., & Yang, Y. (2012).  Wind energy development and its environmental impact: A review.  Renewable and sustainable energy reviews, 16, 1031–1039. doi:10.1016/j.rser.2011.09.024.
  * Li et al. [2023] Li, C., Coolen, J. W., Scherer, L., Mogollón, J. M., Braeckman, U., Vanaverbeke, J., Tukker, A., & Steubing, B. (2023).  Offshore wind energy and marine biodiversity in the north sea: Life cycle impact assessment for benthic communities.  Environmental Science & Technology, 57, 6455–6464. doi:10.1021/acs.est.2c07797.
  * Li et al. [2022] Li, C., Mogollón, J. M., Tukker, A., Dong, J., von Terzi, D., Zhang, C., & Steubing, B. (2022).  Future material requirements for global sustainable offshore wind energy development.  Renewable and Sustainable Energy Reviews, 164, 112603. doi:10.1016/j.rser.2022.112603.
  * Lin et al. [2017] Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017).  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2117–2125).  doi:10.1109/CVPR.2017.106.
  * Liu et al. [2020] Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2020).  Deep learning for generic object detection: A survey.  International journal of computer vision, 128, 261–318. doi:10.1007/s11263-019-01247-4.
  * Ma et al. [2019] Ma, L., Liu, Y., Zhang, X., Ye, Y., Yin, G., & Johnson, B. A. (2019).  Deep learning in remote sensing applications: A meta-analysis and review.  ISPRS journal of photogrammetry and remote sensing, 152, 166–177. doi:10.1016/j.isprsjprs.2019.04.015.
  * Manso-Callejo et al. [2020] Manso-Callejo, M.-Á., Cira, C.-I., Alcarria, R., & Arranz-Justel, J.-J. (2020).  Optimizing the recognition and feature extraction of wind turbines through hybrid semantic segmentation architectures.  Remote Sensing, 12, 3743. doi:10.3390/rs12223743.
  * Manso-Callejo et al. [2021] Manso-Callejo, M.-Á., Cira, C.-I., Garrido, R. P. A., & Matesanz, F. J. G. (2021).  First dataset of wind turbine data created at national level with deep learning techniques from aerial orthophotographs with a spatial resolution of 0.5 m/pixel.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14, 7968–7980. doi:10.1109/JSTARS.2021.3101934.
  * McMorland et al. [2022] McMorland, J., Collu, M., McMillan, D., & Carroll, J. (2022).  Operation and maintenance for floating wind turbines: A review.  Renewable and Sustainable Energy Reviews, 163, 112499. doi:10.1016/j.rser.2022.112499.
  * Mou & Zhu [2018] Mou, L., & Zhu, X. X. (2018).  Vehicle instance segmentation from aerial image and video using a multitask learning residual fully convolutional network.  IEEE Transactions on Geoscience and Remote Sensing, 56, 6699–6711. doi:10.1109/TGRS.2018.2841808.
  * de Moura et al. [2022] de Moura, N. V. A., de Carvalho, O. L. F., Gomes, R. A. T., Guimarães, R. F., & de Carvalho Júnior, O. A. (2022).  Deep-water oil-spill monitoring and recurrence analysis in the brazilian territory using sentinel-1 time series and deep learning.  International Journal of Applied Earth Observation and Geoinformation, 107, 102695. doi:10.1016/j.jag.2022.102695.
  * Potisomporn & Vogel [2022] Potisomporn, P., & Vogel, C. R. (2022).  Spatial and temporal variability characteristics of offshore wind energy in the united kingdom.  Wind Energy, 25, 537–552. doi:10.1002/we.2685.
  * Ronneberger et al. [2015] Ronneberger, O., Fischer, P., & Brox, T. (2015).  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 (pp. 234–241).  Springer.  doi:10.1007/978-3-319-24574-4_28.
  * Schulz et al. [2021] Schulz, M., Boughattas, B., & Wendel, F. (2021).  Deteektor: Mask r-cnn based neural network for energy plant identification on aerial photographs.  Energy and AI, 5, 100069. doi:10.1016/j.egyai.2021.100069.
  * Sun et al. [2012] Sun, X., Huang, D., & Wu, G. (2012).  The current state of offshore wind energy technology development.  Energy, 41, 298–312. doi:10.1016/j.energy.2012.02.054.
  * Tan & Le [2019] Tan, M., & Le, Q. (2019).  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning (pp. 6105–6114).  PMLR.
  * Virtanen et al. [2022] Virtanen, E. A., Lappalainen, J., Nurmi, M., Viitasalo, M., Tikanmäki, M., Heinonen, J., Atlaskin, E., Kallasvuo, M., Tikkanen, H., & Moilanen, A. (2022).  Balancing profitability of energy production, societal impacts and biodiversity in offshore wind farm design.  Renewable and Sustainable Energy Reviews, 158, 112087. doi:10.1016/j.rser.2022.112087.
  * Wong et al. [2019] Wong, B. A., Thomas, C., & Halpin, P. (2019).  Automating offshore infrastructure extractions using synthetic aperture radar & google earth engine.  Remote Sensing of Environment, 233, 111412. doi:10.1016/j.rse.2019.111412.
  * World Forum Offshore Wind (2023) [WFO] World Forum Offshore Wind (WFO) (2023).  Global Offshore Wind Report 2022.  Technical Report February.  URL: https://wfo-global.org/wp-content/uploads/2023/03/WFO_Global-Offshore-Wind-Report-2022.pdf.
  * Xu et al. [2020] Xu, W., Liu, Y., Wu, W., Dong, Y., Lu, W., Liu, Y., Zhao, B., Li, H., & Yang, R. (2020).  Proliferation of offshore wind farms in the north sea and surrounding waters revealed by satellite image time series.  Renewable and Sustainable Energy Reviews, 133, 110167. doi:10.1016/j.rser.2020.110167.
  * Xu et al. [2022] Xu, Z., Zhang, H., Wang, Y., Wang, X., Xue, S., & Liu, W. (2022).  Dynamic detection of offshore wind turbines by spatial machine learning from spaceborne synthetic aperture radar imagery.  Journal of King Saud University-Computer and Information Sciences, 34, 1674–1686. doi:10.1016/j.jksuci.2022.02.027.
  * Zhang et al. [2021] Zhang, T., Tian, B., Sengupta, D., Zhang, L., & Si, Y. (2021).  Global offshore wind turbine dataset.  Scientific Data, 8, 191. doi:10.1038/s41597-021-00982-z.
  * Zheng et al. [2016] Zheng, C. W., Li, C. Y., Pan, J., Liu, M. Y., & Xia, L. L. (2016).  An overview of global ocean wind energy resource evaluations.  Renewable and Sustainable Energy Reviews, 53, 1240–1251. doi:10.1016/j.rser.2015.09.063.
  * Zhou et al. [2018] Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N., & Liang, J. (2018).  Unet++: A nested u-net architecture for medical image segmentation.  In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4 (pp. 3–11).  Springer.  doi:10.1007/978-3-030-00889-5_1.