# Online-updated High-order Collaborative Networks for Single Image Deraining

Cong Wang1, Jinshan Pan2, Xiao-Ming Wu1 Corresponding author

###### Abstract

Single image deraining is an important and challenging task for some
downstream artificial intelligence applications such as video surveillance and
self-driving systems. Most of the existing deep-learning-based methods
constrain the network to generate derained images but few of them explore
features from intermediate layers, different levels, and different modules
which are beneficial for rain streaks removal. In this paper, we propose a
high-order collaborative network with multi-scale compact constraints and a
bidirectional scale-content similarity mining module to exploit features from
deep networks externally and internally for rain streaks removal. Externally,
we design a deraining framework with three sub-networks trained in a
collaborative manner, where the bottom network transmits intermediate features
to the middle network which also receives shallower rainy features from the
top network and sends back features to the bottom network. Internally, we
enforce multi-scale compact constraints on the intermediate layers of deep
networks to learn useful features via a Laplacian pyramid. Further, we develop
a bidirectional scale-content similarity mining module to explore features at
different scales in a down-to-up and up-to-down manner. To improve the model
performance on real-world images, we propose an online-update learning
approach, which uses real-world rainy images to fine-tune the network and
update the deraining results in a self-supervised manner. Extensive
experiments demonstrate that our proposed method performs favorably against
eleven state-of-the-art methods on five public synthetic datasets and one
real-world dataset. The source code will be available at
https://supercong94.wixsite.com/supercong94.

## Introduction

Outdoor images taken in rainy conditions have limited visibility, which
degrades the performance of various applications such as video surveillance
and self-driving systems. Hence, it is essential to recover the degraded
images to improve scene visibility and the performance of downstream
applications. This paper considers single image deraining.

---
Figure 1: Results on the real-world image (b-d) and the synthetic Rain200H
dataset (e). Our proposed method generates a better deraining result on the
real-world image and achieves the best performance-parameter trade-off on the
synthetic dataset.

A rainy image $O$ can be modeled as a linear combination of a rain-free image
$B$ and a rain streaks image $R$:

$O=B+R.$ (1)

Many deraining approaches have been developed based on this simple rainy image
model, including prior-based methods and deep-learning-based methods. Prior-
based methods usually explore empirical statistical properties of rain streaks
and rain-free images, such as image decomposition (Kang, Lin, and Fu 2012),
sparse coding (Luo, Xu, and Ji 2015; Zhang and Patel 2017), low-rank
representation (Chen and Hsu 2013), and Gaussian mixture model (Li et al.
2016). However, since the priors are based on empirical statistical
observations, they do not hold when real-world complex rainy conditions
deviate from the simplified assumptions.

Recent years have witnessed the successful application of deep learning
methods to image deraining (Fu et al. 2017; Yang et al. 2017; Li et al. 2018b;
Zhang and Patel 2018; Pan et al. 2018; Wang et al. 2019a; Jiang et al. 2020;
Wang et al. 2020b; Zamir et al. 2021; Wang et al. 2021; Pan et al. 2021). Most
of these methods develop joint learning networks (Zhang and Patel 2018; Hu et
al. 2019; Jiang et al. 2020) or explore multi-scale architectures (Wang et al.
2020b; Zhu et al. 2020; Zamir et al. 2021) for image deraining. The methods
based on joint learning networks include density-guided multi-stream networks
(Zhang and Patel 2018) and progressive networks motivated by patch similarity
and guided by multi-scale architectures (Jiang et al. 2020). While these
methods are able to remove rain streaks with the guiding networks, they do not
perform well when the guiding process is not accurately estimated. In
addition, although multi-scale architectures have been demonstrated to be
effective, existing methods usually adopt a straightforward way to fuse
features from different scales and do not explore the properties of multi-
scale features. Last but not least, existing methods commonly use synthetic
datasets to train the deep models due to the lack of well-constructed real-
world datasets. However, the gap between synthetic and real data limits the
performance of these methods in real-world applications.

Due to the difficulty of single image deraining, how to fully utilize
convolutional features from both shallow and deep layers of a deep model and
explore multi-scale features are important for rain streaks removal. In
addition, it is essential to develop an effective algorithm to improve the
deraining performance on real-world images.

To this end, we propose a high-order collaborative network with multi-scale
compact constraints and a bidirectional scale-content mining module to remove
rain streaks. The high-order collaborative design allows exploring features
from the shallower and deeper layers of different sub-networks
collaboratively, while the multi-scale compact constraints are used to
effectively learn features from intermediate convolutional layers and the
bidirectional scale-content mining module is embedded in an encoder-decoder
network to explore features of different scales.

Specifically, the high-order collaborative design contains three sub-networks
(bottom, middle, and up) which are learned in a collaborative manner, where
the bottom network transmits intermediate features to the middle network which
also receives shallower rainy features from the top network and sends back
features to the bottom network. The middle and top networks provide deep
supervision for the bottom network to better learn and transmit shallow rainy
features to higher layers to facilitate deraining. To generate more useful
features from a deep network internally, we propose to enforce multi-scale
compact constraints on the intermediate layers to learn better features with
Laplacian pyramid images. In addition, we develop a bidirectional scale-
content similarity mining module in a down-to-up and up-to-down manner to
capture long-range dependencies between features at different scales, which is
embedded in an encoder-decoder architecture to explore useful features.

Finally, to improve the model performance on real-world images, we propose a
simple yet effective online-update learning approach, to fine-tune the model
trained on synthetic datasets using real-world data in a self-supervised
manner with a KL-Divergence loss function. The proposed network design and
online-update learning approach enable our model to achieve state-of-the-art
deraining performance, especially on real-world images, as illustrated in Fig.
1.

The main contributions of this paper include:

  * •

We propose a collaborative deraining framework with multi-scale compact
constraints to control the learning process in an external and internal
manner, with a new bidirectional scale-content similarity mining module to
adaptively learn richer feature representations.

  * •

We present a simple yet effective online-update learning approach to fine-tune
the model trained on synthetic datasets to adapt to real rainy conditions in a
self-supervised manner for real-world image deraining.

  * •

We conduct extensive experiments and ablation studies to evaluate the proposed
method. The results demonstrate that our method performs favorably against
state-of-the-art methods with fewer parameters on both synthetic and real-
world datasets.

## Related Work

In this section, we briefly review recent works on image deraining, which are
based on deep learning, as well as some image restoration methods based on
similarity mining.

### Single Image Deraining

In recent years, deep-learning-based approaches have dominated the research of
image deraining due to the strong representation learning ability of deep
neural networks. Fu et al. (2017) observe that high-frequency details provide
more rain streaks details and less background interference and design a deep
detail residual network to learn rain streaks. Some deraining methods explore
the properties of multi-scale images. Wang et al. (2020b) propose a cross-
scale framework to fuse features of different scales from sub-networks. Jiang
et al. (2020) design a multi-scale progressive fusion network to transmit and
fuse small-scale features to the original scale based on the similarity of
multi-scale images. Recurrent networks are also used for deraining. Li et al.
(2018b) develop a recurrent squeeze-and-excitation network with dilation
convolution to model channel context relation. Ren et al. (2019) propose a
progressive recurrent network to remove rain streaks stage by stage and
analyze the effect of inputs, outputs, and loss functions for image deraining.
Some non-local methods are based on attention mechanisms. Li et al. (2018a)
propose to embed a non-local module to an encoder-decoder framework to capture
long-range feature dependency for improving representation learning. Wang et
al. (2020a) attempt to combine self-attention and scale-aggregation in a self-
calibrated network. Besides, some semi-supervised approaches have been
proposed for removing rain streaks in real-world images. For example, Wei et
al. (2019) develop a semi-supervised transfer learning approach, and Yasarla,
Sindagi, and Patel (2020) design a Gaussian-process-based model that learns on
both synthetic and real data.

Different from the above works, our proposed method aims to explore inner
structures and useful features of the deraining network in an external and
internal manner to train the network for better rain streaks removal. We also
propose an effective online-update learning approach to fine-tune the model
trained with synthetic data on real data for real-world image deraining.

### Similarity Mining

Similarity mining aims to find the most matched content at the feature level.
Mei et al. (2020b) study cross-scale feature mining by exploring the inherent
properties of images for single image super-resolution. Mei et al. (2020a)
mine the similarity from coarser to finer levels in a multi-scale manner for
image restoration. Different from these works that only explore single
directional mining (from coarser to finer levels), we in this paper design a
bidirectional scale-content similarity mining module to adaptively learn
richer features for better deraining.

---
Figure 2: Proposed high-order collaborative networks with multi-scale compact
constraints. Each network module $\mathcal{N}_{mn}$ has a same encoder-decoder
structure with the bidirectional scale-content similarity minding module shown
in Fig. 3.

## Proposed Method

In this section, we introduce each element of the proposed method including
high-order collaborative network, multi-scale compact constraint, encoder-
decoder structure with a bidirectional scale-content similarity mining module,
supervised loss function, and online-update learning.

### High-order Collaborative Networks

How to exploit features from the intermediate layers and different modules in
a collaborative manner is important for image restoration. Here, we design an
effective high-order collaborative network111We regard two-stream networks as
collaborative networks and three-stream networks as high-order. to assist each
sub-network to learn features in an external manner for better image
deraining. The overall network structure of the high-order collaborative
network is shown in Fig. 2, which consists of three sub-networks with similar
structure. The bottom ($\mathcal{B}$), middle ($\mathcal{M}$), and top
($\mathcal{T}$) sub-networks respectively contain three, two, and one encoder-
decoder component with a bidirectional scale-content similarity mining module
that will be explained in detail in Sec. Encoder and Decoder with BiSCSM.

The three sub-networks can transmit shallower and deeper rainy features to
each other in a collaborative manner to improve the performance of the
deraining network $\mathcal{B}$. Note that $\mathcal{M}$ and $\mathcal{T}$ are
supervised by ground truth, which can be regarded as a sort of deep
supervision in an external manner such that $\mathcal{M}$ and $\mathcal{T}$
can transmit useful features to assist $\mathcal{B}$ to learn better features
for deraining. The three sub-networks are trained by:

$\begin{array}[]{ll}\mathcal{L}_{\text{collaborative}}=\sum_{i=1}^{3}\alpha_{i}\big{(}-SSIM(\hat{B}_{i},B)\big{)},\end{array}$
(2)

where $\\{\hat{B}_{i}\\}$ ($i=1,2,3$) denote the output of networks
$\mathcal{B}$, $\mathcal{M}$, and $\mathcal{T}$ respectively, $B$ is the
ground truth, and $\alpha_{i}$ are weight parameters.

### Multi-Scale Compact Constraints

Although the high-order collaborative design enables the bottom sub-network
$\mathcal{B}$ to exploit deeper and shallower features from the other two sub-
networks, the features of each intermediate convolutional layer are learned
without any constraints, making the solution space too large.

To regularize the solution space of intermediate convolutional layers, we
design multi-scale compact constraints (MSCC) in an internal manner to enforce
the network modules $\mathcal{N}_{11}$, $\mathcal{N}_{12}$, and
$\mathcal{N}_{21}$, which are parts of the sub-networks $\mathcal{B}$ and
$\mathcal{M}$, to learn more useful features to facilitate image deraining.
The multi-scale compact constraints are motivated by the multi-scale Laplacian
images which can better model image structures than the original scale image.
We use them to constrain the intermediate convolutional layers of the network.

We first obtain the Laplacian pyramid images and then use the scaled images to
constrain the modules $\mathcal{N}_{11},\mathcal{N}_{12},\mathcal{N}_{21}$ as
shown in Fig. 2. Specifically, the multi-scale compact constraints are
enforced by:

$\begin{array}[]{ll}\mathcal{L}_{\text{mscc}}=\sum_{j=1}^{2}\beta_{j}\big{(}-SSIM(\hat{B}_{\frac{1}{2^{j}}},B_{\frac{1}{2^{j}}})\big{)},\end{array}$
(3)

where $\hat{B}_{\frac{1}{2^{j}}}$ are the output of intermediate layers as
denoted in Fig. 2, $B_{\frac{1}{2^{j}}}$ are the corresponding
$\frac{1}{2^{j}}$ scale Laplacian pyramid rain-free images, and $\beta_{j}$
are weight parameters.

### Encoder and Decoder with BiSCSM

Further, we develop a bidirectional scale-content similarity mining module
(BiSCSM) to explore similar features from different scales, which is motivated
by Mei et al. (2020c, b). The architecture is shown in Fig. 3.

The proposed BiSCSM contains Down-to-Up mining and Up-to-Down mining modules.
The Up-to-Down mining module is defines as:

$\begin{array}[]{ll}y_{i,j}^{p\times
p}=\frac{1}{\sigma(x,z)}\sum_{g,h}\phi(x^{p\times p}_{i,j},z^{p\times
p}_{u,v})\theta(x_{g,h}^{p\times p}),\end{array}$ (4)

where $\phi(x^{p\times p}_{i,j},z^{p\times p}_{u,v})=e^{(W_{f}x_{i,j}^{p\times
p})^{T}(W_{g}z_{u,v}^{p\times p})}$, $\theta=W_{\theta}x_{g,h}^{p\times p}$,
$\sigma=\sum_{u,v}\phi(x^{p\times p}_{i,j},z^{p\times p}_{u,v})$,
$y_{i,j}^{p\times p}$ is a $p\times p$ feature patch located at ($i$, $j$),
and $W_{f},W_{g},W_{\theta}$ are learnable filters. The Down-to-Up mining
module is defined similarly as shown in Fig. 4.

Our proposed BiSCSM is different from the one in Mei et al. (2020c, b) which
only mines patch similarity in a single direction. Our Down-to-Up mining and
Up-to-Down mining modules enable capturing bidirectional similar content from
large to small scales and from small to large scales to mine rich rainy
features. The BiSCSM is embedded in an encoder-decoder framework as in Fig. 3
to learn useful features for rain streaks removal. We also use positional
embedding to encode relations among rain streak features. Finally, we fuse the
learned features at difference scales.

---
Figure 3: Proposed encoder and decoder with a bidirectional scale-content similarity mining module. The Down-to-Up and Up-to-Down mining modules are shown in Fig. 4.  |
---|---
(a) Down-to-Up Mining | (b) Up-to-Down Mining
Figure 4: The scale-content similarity mining modules.

### Supervised Loss Function

Based on the above network design, the overall loss function for training on
synthetic datasets is:

$\begin{array}[]{ll}\mathcal{L}_{\text{synthetic}}={\mathcal{L}_{\text{collaborative}}}+{\mathcal{L}_{\text{mscc}}}.\end{array}$
(5)

### Online-update Learning

Since a deraining model trained on synthetic data normally does not generalize
well to real-world images, we propose an online-update learning approach to
fine-tune the model on real-world rainy images. The key challenges are two-
fold. The first is that there are no ground-truth images available for
training. The second is how to keep the training stable and improve the model
performance. To address these issues, we use the derained results of real-
world images generated by the model trained on synthetic data as pseudo ground
truth and update the pseudo ground truth at each epoch with the fine-tuned
model. The loss function is based on Kullback-Leibler divergence and defined
as:

$\begin{array}[]{ll}\mathcal{L}_{\text{real}}=\underbrace{||\hat{B}^{k}-\hat{B}^{k-1}||_{1}}_{\text{content
term}}+\underbrace{{\lambda
KL_{Loss}(\hat{R}^{k},\hat{R}^{k-1}_{\text{random}})}}_{\text{regularization
term}},\end{array}$ (6)

where $\hat{B}^{k}$ and $\hat{B}^{k-1}$ are the deraining results of a real-
world rainy image $O$ generated by the model $\mathcal{B}$ at epoch $k$ and
$k-1$ respectively. Note that $\hat{B}^{0}$ is the initial deraining result
produced by $\mathcal{B}$ after training on synthetic data.
$\hat{R}^{k}=O-\hat{B}^{k}$ is the estimated rain streaks of $O$.
$\hat{R}^{k-1}_{\text{random}}=O_{\text{random}}-\hat{B}_{\text{random}}^{k-1}$
is the estimated rain streaks of a randomly selected real-world rainy image
$O_{\text{random}}$ and $\hat{B}_{\text{random}}^{k-1}$ is the deraining
result of $O_{\text{random}}$ at epoch $k-1$. The first term of Eq. (6)
ensures the content of image background consistent between different epochs
and keeps the training stable, and the second term is a regularization term
that enforces similarity in rain streaks and improves the deraining
performance as the training proceeds (see Fig. 11). The fine-tuning process is
illustrated in Fig. 5 and described in Alg. 1.

---
Figure 5: Proposed online-update learning approach.  Algorithm 1 Online-update
Learning on Real-world Data

Preparation: The initial deraining result $\hat{B}^{0}$ of a real-world image
$O$ generated by the model trained on synthetic data.
Input: $\\{O,\hat{B}^{0}\\}$.
Output: Derained image {$\hat{B}$}.

1: While $k$ $\leq$ $\mathrm{Epoch}_{\mathrm{Real}}$ do:

2: Randomly crop training image pairs {$O,\hat{B}^{k-1}$}

3: Randomly select a real-world rainy image $O_{\text{random}}$ and obtain
$\hat{R}_{\text{random}}^{k-1}=O_{\text{random}}-\hat{B}_{\text{random}}^{k-1}$

4: Update the deraining model by Eq. (6)

5: Output the current deraining result: {$\hat{B}^{k}$}

6: Update the pseudo ground truth:{$\hat{B}^{k-1}$}
$\leftarrow${$\hat{B}^{k}$}

7: $k$ $\leftarrow$ $k+1$

8: End while

Methods | Rain200H | Rain200L | Rain1200 | Rain1400 | Rain12 | # Param
---|---|---|---|---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
RESCAN (ECCV’18) | 26.661 | 0.8419 | 36.993 | 0.9788 | 32.127 | 0.9028 | 30.969 | 0.9117 | 32.965 | 0.9545 | 0.15M
NLEDN (MM’18) | 27.315 | 0.8904 | 36.487 | 0.9792 | 32.473 | 0.9198 | 31.014 | 0.9206 | 33.028 | 0.9615 | 1.01M
SSIR (CVPR’19) | 14.420 | 0.4501 | 23.476 | 0.8026 | 24.427 | 0.7713 | 25.772 | 0.8224 | 24.138 | 0.7768 | 0.06M
PreNet (CVPR’19) | 27.525 | 0.8663 | 34.266 | 0.9660 | 30.456 | 0.8702 | 30.984 | 0.9156 | 35.095 | 0.9400 | 0.17M
SpaNet (CVPR’19) | 25.484 | 0.8584 | 36.075 | 0.9774 | 27.099 | 0.8082 | 29.000 | 0.8891 | 33.217 | 0.9546 | 0.28M
DCSFN (MM’20) | 28.469 | 0.9016 | 37.847 | 0.9842 | 32.275 | 0.9228 | 31.493 | 0.9279 | 35.803 | 0.9683 | 6.45M
MSPFN (CVPR’20) | 25.553 | 0.8039 | 30.367 | 0.9219 | 30.382 | 0.8860 | 31.514 | 0.9203 | 34.253 | 0.9469 | 21.00M
DRDNet (CVPR’20) | 15.102 | 0.5028 | 37.465 | 0.9806 | 28.386 | 0.8574 | 28.359 | 0.8574 | 25.435 | 0.7550 | 5.23M
RCDNet (CVPR’20) | 28.698 | 0.8904 | 38.400 | 0.9841 | 32.273 | 0.9111 | 31.016 | 0.9164 | 31.038 | 0.9069 | 3.67M
Syn2Real (CVPR’20) | 14.495 | 0.4021 | 31.035 | 0.9365 | 28.812 | 0.8400 | 28.582 | 0.8586 | 31.038 | 0.9069 | 2.62M
MPRNet (CVPR’21) | 29.949 | 0.9151 | 36.610 | 0.9785 | 33.655 | 0.9310 | 32.257 | 0.9325 | 36.578 | 0.9696 | 3.64M
Ours | 29.985 | 0.9218 | 39.284 | 0.9875 | 33.718 | 0.9327 | 32.617 | 0.9334 | 36.851 | 0.9714 | 2.04M

Table 1: Quantitative results on five synthetic datasets. The best results are marked in bold. $\uparrow$ denotes higher is better.  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
(a) Input | (b) DCSFN | (c) MSPFN | (d) DRDNet | (e) RCDNet | (f) Syn2Real | (g) MPRNet | (h) Ours | (i) GT
Figure 6: Comparison with state-of-the-art methods on synthetic datasets. The
proposed network is able to restore better texture.

## Experiments

We compare the proposed approach with 11 state-of-the-art methods (SOTAs),
including RESCAN (Li et al. 2018b), NLEDN (Li et al. 2018a), SSIR (Wei et al.
2019), PreNet (Ren et al. 2019), SpaNet (Wang et al. 2019b), DCSFN (Wang et
al. 2020b), MSPFN (Jiang et al. 2020), DRDNet (Deng et al. 2020), RCDNet (Wang
et al. 2020c), Syn2Real (Yasarla, Sindagi, and Patel 2020), and MPRNet (Zamir
et al. 2021), on five widely used synthetic datasets and a real-world dataset.

### Datasets and Evaluation Criteria

#### Synthetic dataset

We use Rain200H (Yang et al. 2017), Rain200L (Yang et al. 2017), Rain1200
(Zhang and Patel 2018), Rain1400 (Fu et al. 2017), and Rain12 (Li et al. 2016)
as the synthetic datasets for training and evaluation. Rain200H is the most
challenging dataset, which has 1800 image pairs for training and 200 pairs for
testing. Rain200L is the easiest dataset with the same number of training and
testing samples as Rain200H. Rain1200 has images with heavy, middle, and light
rain, and there are 4000 training images and 400 testing images for each
density. Rain1400 has 12600 training samples and 1400 testing samples. Since
Rain12 only contains 12 testing samples, we use the model trained on Rain200H
to test the deraining results. We use Rain200H as the dataset for ablation
study and analysis.

#### Real-world dataset

Yang et al. (2017), Li et al. (2019), and Wang et al. (2020a) provide a large
body of real-world rainy images. We use them to evaluate the deraining results
on real-world data.

#### Evaluation criteria

We use two widely used metrics, peak signal to noise ratio (PSNR) (Huynh-Thu
and Ghanbari 2008) and structural similarity index measure (SSIM) (Wang et al.
2004) to evaluate the quality of restored images on synthetic datasets. As
there are no ground-truth for real-world rainy images, we only compare the
results visually.

### Implementation Details

We set the number of channels of each convolutional layer except the last one
as $20$, and LeakyReLU with $\alpha=0.2$ is used after each convolutional
layer except for the last one. For the last layer, we use $3\times 3$
convolution without any activation function in $\mathcal{B}$, $\mathcal{M}$,
and $\mathcal{T}$. We randomly crop $128\times 128$ image patches as input,
and the batch size is set as $12$. We use ADAM optimizer (Kingma and Ba 2015)
to train the network. The initial learning rate is 0.0005, which will be
divided by $10$ at the $300$-th and $400$-th epochs, and the model training
terminates after $500$ epochs. We set $\lambda=0.0001$, $\alpha_{1}=1$,
$\alpha_{2}=1$, $\alpha_{3}=1$, $\beta_{1}=0.05$, and $\beta_{2}=0.001$. We
train the model for 30 epochs on the real-world dataset, i.e.,
$\mathrm{Epoch}_{\mathrm{Real}}$ = 30. Our model is trained with four NVIDIA
RTX TITAN GPUs on the Pytorch platform.

### Results and Analysis on Synthetic Datasets

#### Comparisons with SOTAs on Synthetic Datasets

Tab. 1 reports the results of our method and SOTAs on five synthetic datasets.
We can see that our method achieves the best results on all tested datasets in
PSNR and SSIM. We further show some deraining results in Fig. 6. It can be
observed that our method can restore better details and textures and obtain
clearer background images, while other approaches hand down some rain streaks
or lose some details.

#### Analysis on BiSCSM

We analyze the effect of different components of BiSCSM and report the results
in Tab. 2. Note that the method ($M_{1}$) does not generate better results if
we remove BiSCSM from the encoder-decoder framework. Furthermore, we replace
the Down-to-Up and Up-to-Down operations in BiSCSM by element-wise summation
($M_{2}$) and $1\times 1$ convolution ($M_{3}$) to fuse features at different
scales. We find that both operations do not perform well compared to the
proposed ones, demonstrating the effectiveness of the mining modules. Fig. 7
visualizes the feature maps before and after BiSCSM. One can see that the
features are significantly enhanced after BiSCSM.

Experiments | $M_{1}$ | $M_{2}$ | $M_{3}$ | $M_{4}$ | $M_{5}$ | $M_{6}$ |  $M_{7}$ (Ours)
---|---|---|---|---|---|---|---
Sum | ✗ | ✔ | ✗ | ✗ | ✗ | ✗ | ✗
Conv1×1 | ✗ | ✗ | ✔ | ✗ | ✗ | ✗ | ✗
Down-to-Up | ✗ | ✗ | ✗ | ✔ | ✗ | ✗ | ✗
Up-to-Down | ✗ | ✗ | ✗ | ✗ | ✔ | ✗ | ✗
BiSCSM | ✗ | ✗ | ✗ | ✗ | ✗ | ✔ | ✔
Position | ✗ | ✔ | ✔ | ✔ | ✔ | ✗ | ✔
PSNR $\uparrow$ | 29.583 | 29.625 | 29.541 | 29.393 | 29.828 | 29.953 | 29.985
SSIM $\uparrow$ | 0.9168 | 0.9177 | 0.9156 | 0.9172 | 0.9201 | 0.9218 | 0.9218

Table 2: Ablation study of BiSCSM. ✔and ✗denote that the corresponding
component is adopted and not adopted, respectively.

We also test the single-directional mining modules, i.e., using Down-to-Up
($M_{4}$) or Up-to-Down ($M_{5}$) mining module alone, and the results show
the bidirectional manner is better and can mine richer features for better
deraining ($M_{4}$ and $M_{5}$ vs. $M_{7}$). Finally, we observe that position
awareness can improve the deraining performance ($M_{6}$ vs. $M_{7}$).

|  |
---|---|---
(a) Input | (b) Before | (c) After
Figure 7: Visualization of the feature maps before and after BiSCSM.

Experiments | $R_{1}$ | $R_{2}$ | $R_{3}$ | $R_{4}$ |  $R_{5}$ (Ours)
---|---|---|---|---|---
$1/2$ Scale | ✗ | ✔ | ✗ | ✗ | ✔
$1/4$ Scale | ✗ | ✗ | ✔ | ✗ | ✔
Full Scale | ✗ | ✗ | ✗ | ✔ | ✗
PSNR $\uparrow$ | 29.824 | 29.879 | 29.839 | 29.970 | 29.985
SSIM $\uparrow$ | 0.9195 | 0.9209 | 0.9204 | 0.9211 | 0.9218

Table 3: Ablation study of the multi-scale compact constraints.

#### Analysis on MSCC

Compared with deep models that have no constraints on intermediate layers, our
proposed MSCC can make the deep network more compact and learn more useful
features. Tab. 3 shows the results of different types of constraints. We
observe that the model without any contraints ($R_{1}$) performs the worst,
while the results get better as we add different constraints and reach the
best when both $1/2$ ($R_{2}$) and $1/4$ ($R_{3}$) scale constraints are
added.

We also consider replacing the $1/2$ and $1/4$ scale images with full-scale
images to constrain the network. The comparison between $R_{4}$ and $R_{5}$
demonstrates that full-scale images are less effective than the multi-scale
images, which is probably due to the better structures of Laplacian pyramid
images.

#### Ablation Study on Collaborative Learning

We conduct an ablation study of the proposed collaborative network in Tab. 4.
The results show that the model achieves the best performance when the sub-
networks $\mathcal{T}$ and $\mathcal{M}$ are learned with $\mathcal{B}$ in a
collaborative manner.

We also consider a case by cascading the three sub-networks to form a deep
network model. Specifically, we cascade the network modules
$\mathcal{N}_{11}$, $\mathcal{N}_{12}$, $\mathcal{N}_{13}$,
$\mathcal{N}_{21}$, $\mathcal{N}_{22}$, and $\mathcal{N}_{31}$ and ensure the
cascaded network has roughly the same number of parameters as the
collaborative network. We find that the cascaded network does not perform as
well as the collaborative network, which further demonstrates the
effectiveness of the proposed collaborative learning manner.

| w/o $\mathcal{T}$ & w/o $\mathcal{M}$ | w/o $\mathcal{T}$ & w/ $\mathcal{M}$ | Cascaded | w/ $\mathcal{T}$ & w/ $\mathcal{M}$ (Ours)
---|---|---|---|---
PSNR $\uparrow$ | 29.563 | 29.355 | 29.642 | 29.985
SSIM $\uparrow$ | 0.9158 | 0.9146 | 0.9188 | 0.9218

Table 4: Ablation study on collaborative learning.  |
---|---
(a) PSNR | (b) SSIM
Figure 8: Comparison between single-stream learning and collaborative learning.  |
---|---
(a) PSNR | (b) SSIM
Figure 9: Further ablation study on collaborative learning (CL) and multi-scale compact constraints (MSCC).  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
|  |  |  |  |  |  |
(a) Input | (b) DCSFN | (c) MSPFN | (d) DRDNet | (e) RCDNet | (f) Syn2Real | (g) MPRNet | (h) Ours
Figure 10: Comparisons with state-of-the-art methods on real-world images. Our
proposed online-update learning approach is able to deal with various rainy
conditions to better remove rain streaks and even haze and recover clearer
images.

#### Comparison between Single-stream Learning and Collaborative Learning

In Fig. 8, we provide the comparison of the performance of each sub-network
trained in single-stream learning and collaborative learning manners for image
deraining. Single-stream learning means training the sub-networks
$\mathcal{T}$, $\mathcal{M}$, and $\mathcal{B}$ independently. The results
show that the sub-networks trained by collaborative learning preform much
better than by single-stream learning. It can be also observed that the sub-
network $\mathcal{T}$ trained by collaborative learning generates comparable
results as the sub-network $\mathcal{B}$ trained by single-stream learning,
while the size of $\mathcal{T}$ is two-thirds of $\mathcal{B}$.

#### Further Ablation Study on Collaborative Learning and Multi-scale Compact
Constraints

Fig. 9 provides a further ablation study of collaborative learning and multi-
scale compact constraints, where the learning curves of different variants are
plotted. The results show that both collaborative learning and multi-scale
compact constraints are useful for improving the deraining performance. It can
also be observed that without either collaborative learning or multi-scale
compact constraints, the models perform worse than the cascaded network, which
further demonstrates the effectiveness of the proposed external and internal
learning manners for the deraining task.

### Results and Analysis on Real-world Datasets

|  |  |
---|---|---|---
(a) | (b) | (c) | (d)
Figure 11: Ablation study of the proposed online-update learning approach. (a)
Input. (b) Only trained on synthetic data. (c) Directly fine-tuned on real-
world images. (d) Online-update learning on real-world images.

#### Comparisons with SOTAs on Real-world Images

We further demonstrate the effectiveness of our method on the real-world
dataset by comparing with state-of-the-art methods. Fig. 10 presents the
deraining results of several challenging cases. It can be observed that our
method produces cleaner and clearer deraining results than others,
demonstrating its effectiveness in removing rain streaks of real-world rainy
images.

#### Effectiveness of the Online-update Learning Approach

Fig. 11 presents the ablation study of the online-update learning manner.
Compared with only using synthetic data to train the model (Fig. 11(b)) or
directly fine-tuning the model on real-world images without updating the
pseudo ground truth (Fig. 11(c)), our proposed online-update learning manner
(Fig. 11(d)) is able to further improve the deraining performance on real-
world images, demonstrating its effectiveness.

#### Generality of the Online-update Learning Approach

To further demonstrate the effectiveness of the online-update learning
approach for real-world image deraining, we apply it on a state-of-the-art
method, DCSFN (Wang et al. 2020b). Similar to Fig. 11, Fig. 12 presents the
results on a real-world rainy image by different ways of learning. It can be
seen that the proposed online-update learning approach can be successfully
applied to DCSFN to significantly improve its deraining performance on real-
world images. Since the online-update learning approach is generic, we believe
it be applied to many other existing methods to improve their performance in
real-world image deraining.

|  |  |
---|---|---|---
(a) | (b) | (c) | (d)
Figure 12: Applying the proposed online-update learning approach on DCSFN
(Wang et al. 2020b). (a) Input. (b) Only trained on synthetic data. (c)
Directly fine-tuned on real-world images. (d) Online-update learning on real-
world images.

## Conclusion

In this paper, we have proposed a high-order collaborative network with multi-
scale compact constraints to control the learning process in an external and
internal manner for image deraining. We have further developed a bidirectional
scale-content similarity mining module to learn useful features at different
scales in a down-to-up and up-to-down way to facilitate rain streaks removal.
Finally, to improve the deraining performance on real-world images, we have
proposed an effective online-update learning approach to fine-tune the
deraining model on real-world rainy images in a self-supervised manner.
Extensive experiments show that the proposed model outperforms state-of-the-
art methods on five public synthetic datasets and one real-world dataset.

## Acknowledgements

We would like to thank the anonymous reviewers for their helpful comments.
This work was supported by the Fundamental Research Funds for the Central
Universities (No. 30920041109), and the Grant of DaSAIL Project P0030935
funded by PolyU/UGC.

## References

  * Chen and Hsu (2013) Chen, Y.; and Hsu, C. 2013.  A Generalized Low-Rank Appearance Model for Spatio-temporally Correlated Rain Streaks.  In _IEEE ICCV_ , 1968–1975.
  * Deng et al. (2020) Deng, S.; Wei, M.; Wang, J.; Feng, Y.; Liang, L.; Xie, H.; Wang, F. L.; and Wang, M. 2020.  Detail-recovery Image Deraining via Context Aggregation Networks.  In _IEEE CVPR_ , 14548–14557.
  * Fu et al. (2017) Fu, X.; Huang, J.; Zeng, D.; Huang, Y.; Ding, X.; and Paisley, J. 2017.  Removing Rain from Single Images via a Deep Detail Network.  In _IEEE CVPR_ , 1715–1723.
  * Hu et al. (2019) Hu, X.; Fu, C.; Zhu, L.; and Heng, P. 2019.  Depth-Attentional Features for Single-Image Rain Removal.  In _IEEE CVPR_ , 8022–8031.
  * Huynh-Thu and Ghanbari (2008) Huynh-Thu, Q.; and Ghanbari, M. 2008.  Scope of validity of PSNR in image/video quality assessment.  _Electronics Letters_ , 44(13): 800–801.
  * Jiang et al. (2020) Jiang, K.; Wang, Z.; Yi, P.; Chen, C.; Huang, B.; Luo, Y.; Ma, J.; and Jiang, J. 2020.  Multi-Scale Progressive Fusion Network for Single Image Deraining.  In _IEEE CVPR_ , 8343–8352.
  * Kang, Lin, and Fu (2012) Kang, L.; Lin, C.; and Fu, Y. 2012.  Automatic Single-Image-Based Rain Streaks Removal via Image Decomposition.  _IEEE TIP_ , 21(4): 1742–1755.
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015.  Adam: A Method for Stochastic Optimization.  In _ICLR_.
  * Li et al. (2018a) Li, G.; He, X.; Zhang, W.; Chang, H.; Dong, L.; and Lin, L. 2018a.  Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining.  In _ACM MM_ , 1056–1064.
  * Li et al. (2019) Li, S.; Araujo, I. B.; Ren, W.; Wang, Z.; Tokuda, E. K.; Junior, R. H.; Cesar-Junior, R.; Zhang, J.; Guo, X.; and Cao, X. 2019.  Single Image Deraining: A Comprehensive Benchmark Analysis.  In _IEEE CVPR_ , 3838–3847.
  * Li et al. (2018b) Li, X.; Wu, J.; Lin, Z.; Liu, H.; and Zha, H. 2018b.  Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining.  In _ECCV_ , 262–277.
  * Li et al. (2016) Li, Y.; Tan, R. T.; Guo, X.; Lu, J.; and Brown, M. S. 2016.  Rain Streak Removal Using Layer Priors.  In _IEEE CVPR_ , 2736–2744.
  * Luo, Xu, and Ji (2015) Luo, Y.; Xu, Y.; and Ji, H. 2015.  Removing Rain from a Single Image via Discriminative Sparse Coding.  In _IEEE ICCV_ , 3397–3405.
  * Mei et al. (2020a) Mei, Y.; Fan, Y.; Zhang, Y.; Yu, J.; Zhou, Y.; Liu, D.; Fu, Y.; Huang, T. S.; and Shi, H. 2020a.  Pyramid Attention Networks for Image Restoration.  _arXiv preprint arXiv:2004.13824_.
  * Mei et al. (2020b) Mei, Y.; Fan, Y.; Zhou, Y.; Huang, L.; Huang, T. S.; and Shi, H. 2020b.  Image Super-Resolution With Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining.  In _IEEE CVPR_ , 5689–5698.
  * Mei et al. (2020c) Mei, Y.; Fan, Y.; Zhou, Y.; Huang, L.; Huang, T. S.; and Shi, H. 2020c.  Image Super-Resolution With Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining.  In _IEEE CVPR_ , 5689–5698.
  * Pan et al. (2021) Pan, J.; Dong, J.; Liu, Y.; Zhang, J.; Ren, J. S. J.; Tang, J.; Tai, Y.; and Yang, M. 2021.  Physics-Based Generative Adversarial Models for Image Restoration and Beyond.  _IEEE TPAMI_ , 43(7): 2449–2462.
  * Pan et al. (2018) Pan, J.; Liu, S.; Sun, D.; Zhang, J.; Liu, Y.; Ren, J. S. J.; Li, Z.; Tang, J.; Lu, H.; Tai, Y.; and Yang, M. 2018.  Learning Dual Convolutional Neural Networks for Low-Level Vision.  In _IEEE CVPR_ , 3070–3079.
  * Ren et al. (2019) Ren, D.; Zuo, W.; Hu, Q.; Zhu, P.; and Meng, D. 2019.  Progressive Image Deraining Networks: A Better and Simpler Baseline.  In _IEEE CVPR_ , 3937–3946.
  * Wang et al. (2020a) Wang, C.; Wu, Y.; Su, Z.; and Chen, J. 2020a.  Joint Self-Attention and Scale-Aggregation for Self-Calibrated Deraining Network.  In _ACM MM_ , 2517–2525.
  * Wang et al. (2020b) Wang, C.; Xing, X.; Wu, Y.; Su, Z.; and Chen, J. 2020b.  DCSFN: Deep Cross-scale Fusion Network for Single Image Rain Removal.  In _ACM MM_ , 1643–1651.
  * Wang et al. (2019a) Wang, C.; Zhang, M.; Su, Z.; Wu, Y.; Yao, G.; and Wang, H. 2019a.  Learning a multi-level guided residual network for single image deraining.  _Signal Process. Image Commun._ , 78: 206–215.
  * Wang et al. (2020c) Wang, H.; Xie, Q.; Zhao, Q.; and Meng, D. 2020c.  A Model-Driven Deep Neural Network for Single Image Rain Removal.  In _IEEE CVPR_ , 3100–3109.
  * Wang et al. (2019b) Wang, T.; Yang, X.; Xu, K.; Chen, S.; Zhang, Q.; and Lau, R. W. H. 2019b.  Spatial Attentive Single-Image Deraining With a High Quality Real Rain Dataset.  In _IEEE CVPR_ , 12270–12279.
  * Wang et al. (2004) Wang, Z.; Bovik, A. C.; Sheikh, H. R.; and Simoncelli, E. P. 2004.  Image quality assessment: from error visibility to structural similarity.  _IEEE TIP_ , 13(4): 600–612.
  * Wang et al. (2021) Wang, Z.; Wang, C.; Su, Z.; and Chen, J. 2021.  Dense Feature Pyramid Grids Network for Single Image Deraining.  In _IEEE ICASSP_ , 2025–2029.
  * Wei et al. (2019) Wei, W.; Meng, D.; Zhao, Q.; Xu, Z.; and Wu, Y. 2019.  Semi-Supervised Transfer Learning for Image Rain Removal.  In _IEEE CVPR_ , 3877–3886.
  * Yang et al. (2017) Yang, W.; Tan, R. T.; Feng, J.; Liu, J.; Guo, Z.; and Yan, S. 2017.  Deep Joint Rain Detection and Removal from a Single Image.  In _IEEE CVPR_ , 1685–1694.
  * Yasarla, Sindagi, and Patel (2020) Yasarla, R.; Sindagi, V. A.; and Patel, V. M. 2020.  Syn2Real Transfer Learning for Image Deraining Using Gaussian Processes.  In _IEEE CVPR_ , 2723–2733.
  * Zamir et al. (2021) Zamir, S. W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F. S.; Yang, M.; and Shao, L. 2021.  Multi-Stage Progressive Image Restoration.  In _IEEE CVPR_.
  * Zhang and Patel (2017) Zhang, H.; and Patel, V. M. 2017.  Convolutional Sparse and Low-Rank Coding-Based Rain Streak Removal.  In _IEEE WACV_ , 1259–1267.
  * Zhang and Patel (2018) Zhang, H.; and Patel, V. M. 2018.  Density-Aware Single Image De-Raining Using a Multi-Stream Dense Network.  In _IEEE CVPR_ , 695–704.
  * Zhu et al. (2020) Zhu, H.; Wang, C.; Zhang, Y.; Su, Z.; and Zhao, G. 2020.  Physical Model Guided Deep Image Deraining.  In _IEEE ICME_ , 1–6.


#  WikiGoldSK: Annotated Dataset, Baselines and Few-Shot Learning Experiments
for Slovak Named Entity Recognition

Dávid Šuba Marek Šuppa Jozef Kubík Endre Hamerlik Martin Takáč
Comenius University in Bratislava, Slovakia
Contact<EMAIL_ADDRESS>
###### Abstract

Named Entity Recognition (NER) is a fundamental NLP tasks with a wide range of
practical applications. The performance of state-of-the-art NER methods
depends on high quality manually anotated datasets which still do not exist
for some languages. In this work we aim to remedy this situation in Slovak by
introducing WikiGoldSK, the first sizable human labelled Slovak NER dataset.
We benchmark it by evaluating state-of-the-art multilingual Pretrained
Language Models and comparing it to the existing silver-standard Slovak NER
dataset. We also conduct few-shot experiments and show that training on a
sliver-standard dataset yields better results. To enable future work that can
be based on Slovak NER, we release the dataset, code, as well as the trained
models publicly under permissible licensing terms at
https://github.com/NaiveNeuron/WikiGoldSK.

##  1 Introduction

Named Entity Recognition (NER) is a lower-level Natural Language Processing
(NLP) task in which the aim is to both identify and classify named entity
expressions in text into a pre-defined set of semantic types, such as
Location, Organization or Person Goyal et al. (2018). It is a key component of
many downstream NLP tasks, ranging from information extraction, machine
translation, question answering to entity linking and co-reference resolution,
among others. Since its introduction at MUC-6 Grishman and Sundheim (1996),
the task has been studied extensively, usually as a form of token
classification. In recent years, the advent of pre-trained language models
(PLMs) combined with the availability of sufficiently large high quality NER-
annotated datasets has led to the introduction of NER systems with very high
reported performance, sometimes nearing human annotation quality He et al.
(2021).

As the predominant method for adapting PLMs to a specific task of interest is
model fine-tuning using training data, the availability of annotated NER
datasets for both the training as well as the evaluation part of the process
of creating a NER system is critical. Since their creation is expensive, many
works have focused on extracting multilingual silver-standard NER datasets
from publicly available corpora such as Wikipedia, exploiting the link
structure to locate and classify named entities Nothman et al. (2013); Al-Rfou
et al. (2015); Tsai et al. (2016); Pan et al. (2017). While these methods have
yielded NER-annotated datasets of significant size, with the recent follow-up
work reporting quality comparable to that of datasets created via manual
annotation Tedeschi et al. (2021), their application has multiple limitations:
only a limited amount of Wikipedia text is inter-linked, mapping Wikipedia
links to the pre-defined NER classes is non-trivial and their application
often depends on the existence of high quality knowledge bases which may not
be available for some domains and languages.

In this paper we focus on Slovak, a language of the Indo-European family,
spoken by 5 million native speakers, which is still missing a manually
annotated NER dataset of substantial size. To fill this gap, we propose the
following contributions:

  * •

We introduce a novel, manually annotated NER dataset called WikiGoldSK built
by annotating articles sampled from Slovak Wikipedia and labeled with four
entity classes.

  * •

We evaluate a selection of multilingual NER baseline models on the presented
dataset to compare its quality with that of existing silver-standard Slovak
NER datasets.

  * •

We treat Slovak as a low-resource language and also assess the possibility of
using few-shot learning to train a Slovak NER model using a small part of the
introduced dataset.

##  2 Related Work

#### NER datasets

Much of the progress in NER over the past decades can be attributed to and
evidenced by the results reported on standard benchmarks, which in turn
originate from shared tasks. This is because they generally provide high-
quality annotation datasets, which are key both for the evaluation as well as
creation of NER systems. Shared tasks were first introduced for resource-rich
languages, such as English, Spanish, German and Dutch Tjong Kim Sang (2002);
Tjong Kim Sang and De Meulder (2003) and later established for other language
groups, such as Indic Rajeev Sangal and Singh (2008) or Balto-Slavic languages
Piskorski et al. (2017, 2019, 2021). The ”First Cross-Lingual Challenge on
Recognition, Normalization and Matching of Named Entities in Slavic Languages”
(BSNLP 2017) Piskorski et al. (2017) is of particular relevance for our work,
as to the best of our knowledge, it introduced the only publicly available
human annotated Named Entity Recognition dataset based specifically on Slovak
newswire. The dataset, however, consists of less than 50 human-annotated
articles and can at best only be used for evaluation of Slovak NER systems but
not their training.

The over-reliance on newswire text in the shared tasks has been noticed by the
authors of Balasuriya et al. (2009) who introduced the manually annotated
WikiGold dataset based on English Wikipedia articles. Despite its limited
size, it is still used as an evaluation benchmark. As our aim is also to
create a manually annotated (gold-standard) dataset based on Wikipedia
articles, we use WikiGoldSK to refer to the dataset introduced in this work.

To alleviate the need for sizeable datasets at low cost across multiple
languages, various methods of automatically generated NER-annotated datasets
have been introduced. In Nothman et al. (2013) the authors introduce the
WikiNER datasets, which makes use of Wikipedia articles and spans 9 languages
but does not include Slovak. Utilizing a similar approach, Pan et al. (2017)
first classified English Wikipedia pages to specific entity types and then
used the cross-lingual links to transfer the annotations to other languages.
As not all entries are linked, the authors also utilize self-training and
translation methods to match as many entries as possible. This pipeline
generates a dataset that covers 282 languages and includes Slovak as well.
With roughly 50 thousand entities annotated with categories Person, Location
and Organization, it is the largest publicly available Slovak NER dataset to
date.

Another approach of resolving the need for a sizable training dataset is to
utilize few-shot learning, in which only a couple of expertly annotated
training examples are provided. Recently, the methods based on the combination
of cloze-style rephrasing with language models have been shown to perform
comparably to GPT-3 Brown et al. (2020) while having significantly fewer
parameters Schick and Schütze (2020b). We consider a variant of Pattern-
Exploiting Training Schick and Schütze (2020a) called PETER La Gatta et al.
(2021) and to the best of our knowledge for the first time evaluate its
performance for a general-purpose NER system in a specific language.

#### Slovak NER

The prior art in Slovak NER is limited. In Kaššák et al. (2012) the authors
identified potential named entities as words with capital letters and
recognized new entities by finding the entity scope through Wikipedia parsing.
For the purposes of this work they also created a dataset annotated by 60
human experts totalling 1620 entities. The authors of Maruniak (2021) and
Lupták (2021) worked with datasets based on more than 5000 articles extracted
from Slovak Wikipedia, containing more than 15 000 entities and used multiple
well-established NLP toolkits and libraries (such as SpaCy) to train NER
models on this dataset. Utilizing a different datasource, Mičo (2019) have
focused on the Twitter account of one of the biggest Slovak journal and
created a dataset with 10 000 of its NER-annotated tweets and almost 16 000
entities, and used it to train a NER model which utilized both FastText
Bojanowski et al. (2016) vectors and the BiLSTM neural network architecture.
Unfortunately, none of the datasets and models mentioned in the aforementioned
works are publicly available.

Despite having 5 million native speakers and being one of the official
languages of the European Union, there are relatively few readily available
NLP tools tailored specifically for Slovak, which might be to some extent
caused by its linguistic and historical closeness to the much better resourced
Czech. This creates a peculiar dichotomy: Slovak has too many native speakers
to be considered ”low-resource” but at the same time lacks readily available
labelled datasets that are a prerequisite for many standard NLP tools. The
”language richness” taxonomies such as Joshi et al. (2020) consider Slovak
among the ”The Rising Stars” of languages, but it is, to the best of our
knowledge, one of the few in this category that lacks a sizable, manually
labelled NER dataset111The other languages lacking a sizable, manually NER-
annotated datasets are Uzbek, Georgian, Belarusian, Egyptian Arabic and
Cebauano.. The introduction of SlovakBERT in Pikuliak et al. (2021) does
suggest, however, that there is interest in creating Slovak-specific NLP tools
and resources. Our work aims to help push this trend further.

##  3 Dataset

| WikiANN | BSNLP2017 | WikiGoldSK
---|---|---|---
# doc | N/A | 49 | 412
# sent | 30 000 | 741 | 6 696
# tok | 263 516 | 14 400 | 128 944
split | 2:1:1 | 0:0:1 | 7:1:2
LOC | 19 643 | 244 | 4 459
PER | 18 238 | 255 | 2 739
ORG | 15 286 | 273 | 1 929
MISC | N/A | 55 | 1 668
Table 1: The comparison of WikiGoldSK to the other publicly available Slovak
NER datasets. The terms # doc, # sent and # tok refer to the number of
documents, sentences and tokens in the specific datasets, respectively. Note
that WikiANN does not provide document-level split and is not labeled with the
MISC entity.

When creating the WikiGoldSK dataset, our principal aim was to create a high
quality, publicly available human annotated corpora that could be used to both
evaluate and build Slovak NER systems and that would be comparable to well-
established benchmark datasets in other languages. To ensure the resulting
dataset can be used in the future for research as well as commercial use, we
sampled 412 articles from the skwiki-20210701 dump of Slovak Wikipedia,
licensed under the terms of the Creative Commons Attribution-Share-Alike
License 3.0. In order for an article to be included, its last change date
needed to be in 2021 and its length had to fit between 500 and 5 000
characters222This is motivated by the observation that long articles may shift
the dataset towards their domain, whereas short articles often do not contain
any named entity.. The raw text of the articles was tokenized by the generic
English spaCy tokenizer, with a manual pass over the dataset in which the
Slovak-specific tokenization mistakes were remedied.

| train | dev | test
---|---|---|---
# sent | 4 687 | 669 | 1 340
# tok | 90 616 | 12 794 | 25 534
split size | 70% | 10% | 20%
LOC | 3 040 | 461 | 958
PER | 1 892 | 298 | 549
ORG | 1 361 | 190 | 378
MISC | 1 184 | 160 | 324
Table 2: The frequency distribution across the WikiGoldSK’s train/dev/test
splits.

We use the same set of tags as the CoNLL-2003 NER Shared task Tjong Kim Sang
and De Meulder (2003), that is Location (LOC), Person (PER), Organization
(ORG) and Miscellaneous (MISC), and our annotation guidelines are inspired by
the ones introduced by the BSNLP 2017 shared task Piskorski et al. (2017). The
annotation was done using Prodigy333https://prodi.gy/ in which the whole
dataset was pre-loaded with the labels predicted by the SlovakBERT model
finetuned on the training part of the Slovak portion of the WikiANN dataset.
The dataset was annotated by three Slovak native speakers who are also authors
of this paper. Two annotators provided annotations for the full dataset
whereas one annotator corrected half of the dataset. The Cohen’s kappa
coefficient between the first two annotators is 0.90 when compared on the
token level and 0.81 for the tokens where both annotators agreed that they
were not a part of a named entity. As per the benchmark established in Landis
and Koch (1977), the coefficient values in both cases suggest ”almost perfect”
strength of agreement and a high quality of the annotation. To arrive at the
final dataset, the ambiguities were resolved in a discussion between the
annotators.

The summary statistics of the resulting dataset, along with the existing
Slovak NER datasets, can be found in Table 1. As we can see, WikiGoldSK is
larger than the Slovak portion of the BSNLP2017 dataset but smaller than the
Slovak portion of the WikiANN dataset. At the same time, one can see that the
distribution of Named Entities in WikiANN and WikiGoldSK follows the same
pattern, with the order of LOC, PER, ORG holding for both datasets in terms of
entity frequency, which is not the case in BSNLP2017. This is probably caused
by the fact that both WikiANN and WikiGoldSK are based on Wikipedia articles
whereas BSNLP2017 is based on newswire text.

To make the dataset compatible with existing benchmarks, we also introduce a
standard train/dev/test split in the 7:1:2 ratio, described in detail in Table
2. We note that the size of the test portion of WikiGoldSK is on the same
order as that of WikiGold which consists of 1 696 sentences and 39 007 tokens.

##  4 Experiments

We conduct two types of experiments with the newly introduced dataset. First,
we establish a set of baselines based on existing state-of-the-art PLMs that
were pre-trained on Slovak data. Next, we emulate a low-resource setup by only
using a small sample of the training set and use it to evaluate a few-shot
learning approach as well.

###  4.1 Baselines

To evaluate a broad set of baselines on WikiGoldSK, we choose three well-
established NLP toolkits:

  * •

spaCy444https://spacy.io, which provides a pipeline for converting words to
embedding of user’s choice and then models NER as a structured prediction
task,

  * •

Trankit Nguyen et al. (2021), which is based on XLM-RoBERTa Conneau et al.
(2019), provides pre-trained models for 56 languages, including Slovak, along
with the ability to finetune on custom NER datasets, and

  * •

Transformers Wolf et al. (2019), which has become the standard tool for
training, storing and sharing Transformer-based models and also includes
readily available scripts for finetuning PLMs on NER datasets.

When it comes to the models chosen as baselines, we again chose well-
established models relevant to the task of Slovak NER:

  * •

XLM-RoBERTa (XLM-R-base), a multilingual Transformer model pretrained on text
spanning 100 languages, including Slovak,

  * •

SlovakBERT, the only BERT-based model specifically optimized for Slovak, which
was pre-trained on almost 20GB of Slovak text obtained from various sources,
including crawling Slovak web, and

  * •

mDeBERTav3 He et al. (2021), a multilingual Transformer model pretrained on
the same dataset as XLM-RoBERTa using a different training objective which
leads to more efficient training and better performance on various benchmarks.

Our experiments were generally conducted by finetuning a given model using a
specific NLP toolkit on a selected dataset, while utilizing the test set of
the WikiGoldSK for evaluation. We use three datasets for finetuning: WikiANN,
WikiANN combined with WikiGoldSK and just WikiGoldSK. Only the training
portions of the respective datasets were used for finetuning. Additionally, we
also benchmark the models trained on the WikiGoldSK dataset on the Slovak
portion of the BSNLP2017 dataset.

###  4.2 Few-shot learning

To evaluate the possibility of building a Slovak NER system, we chose the
PETER (PET Schick and Schütze (2020a) for NER) method introduced in La Gatta
et al. (2021). At its core, it uses pattern-verbalizer pairs (PVP), in which
the ”pattern” part converts a sentence with a token that corresponds to a
named entity and creates a cloze-style phrase containing exactly one [MASK]
token and the ”verbalizer” maps tokens predicted by a PLM in place of [MASK]
to one of the considered Named Entity classes. Each labeled sentence $s$ is
converted into $|s|$ pairs of training inputs $x=(s,t)$ where $t$ is a
particular token from the sentence we are predicting a label to; the training
set then consist of pairs $(x,y)$ where $y$ is the ground-truth label. A
separate language model $M$ is fine-tuned for each PVP, a soft-label dataset
created from unlabeled data and finally, the resulting classifier is trained
on this dataset.

In our experiments, we use two PVPs below. More details can be found in
Appendix B.

  * •

$P_{1}((s,t))$: "$s$. V predchádzajúcej vete slovo $t$ označuje entitu
[MASK]." (English translation: _” $s$. In the previous sentence, the word $t$
refers to a/an [MASK] entity._)

  * •

$P_{2}((s,t))$: "$s$. $t$ je [MASK]." (English translation: _” $s$. $t$ is a
[MASK]._)

WikiANN WikiANN \+ WikiGoldSK WikiGoldSK BSNLP2017 P R F1 P R F1 P R F1 P R F1
spaCy XLM-RoBERTa 0.5639 0.7413 0.6405 0.8809 0.8973 0.8890 0.9145 0.8955
0.9049 0.8102 0.7722 0.7907 SlovakBERT 0.5509 0.7285 0.6274 0.8754 0.8932
0.8842 0.8889 0.9122 0.9004 0.7186 0.7704 0.7436 mDeBERTaV3 0.5925 0.7572
0.6648 0.8621 0.8855 0.8737 0.9151 0.9167 0.9159 0.8024 0.8122 0.8073 Trankit
XLM-RoBERTa 0.6110 0.7020 0.6533 0.8833 0.8805 0.8819 0.8869 0.9014 0.8941
0.7882 0.8252 0.8063 Transformers XLM-RoBERTa 0.5247 0.7423 0.6148 0.8815
0.9018 0.8915 0.9210 0.9339 0.9274 0.7760 0.8226 0.7986 SlovakBERT 0.5265
0.7428 0.6162 0.9020 0.9208 0.9113 0.9179 0.9262 0.9221 0.7900 0.8278 0.8085
mDeBERTaV3 0.5092 0.7471 0.6056 0.8835 0.9063 0.8948 0.9302 0.9412 0.9357
0.7793 0.8322 0.8049

Table 3: The results of finetuning various baselines using the three selected
NLP toolkits on three dataset combinations and evaluating on the test set of
WikiGoldSK. The P, R and F1 refer to Precision, Recall and the F1 score,
respectively. Best result per metric and dataset is boldfaced.

##  5 Results

The results of the evaluation of baselines can be seen in Table 3. They
suggest that XLM-RoBERTa can still be considered a strong baseline, as its
performance is similar to that of SlovakBERT, despite the latter being
specifically trained and optimized for Slovak. Across the three NLP toolkits,
we observe that the performance of Trankit is generally lower than that of
spaCy and Transformers, given the same dataset. Comparing the three models
finetuned either using spaCy or Transformers, Table 3 suggests that mDeBERTaV3
obtains performance that is either very similar or better than that of XLM-
RoBERTa across all considered datasets. A model based on mDeBERTaV3 also
reported the best performance out of all models evaluated on WikiGoldSK and
performance on par with SlovakBERT on the BSNLP2017 dataset. Finally, we also
note that the choice of the training dataset has significant impact on the
performance of the resulting NER model, as the difference between the F1
scores of the best performing model on the WikiANN dataset and the WikiGoldSK
dataset is over 0.27. Despite the much larger size of the WikiANN dataset, the
results in Table 3 suggest it is best not to combine it with the manually
annotated dataset in order to obtain the best results.

| P | R | F1
---|---|---|---
10 shots |  |  |
PVP 1 | 0.4262 | 0.5290 | 0.4720
PVP 2 | 0.4320 | 0.6163 | 0.5079
PVP 1 & 2 | 0.4834 | 0.5937 | 0.5329
30 shots |  |  |
PVP 1 | 0.4853 | 0.5968 | 0.5353
PVP 2 | 0.4921 | 0.6502 | 0.5602
PVP 1 & 2 | 0.4857 | 0.6072 | 0.5397
50 shots |  |  |
PVP 1 | 0.5198 | 0.6176 | 0.5645
PVP 2 | 0.5041 | 0.6688 | 0.5749
PVP 1 & 2 | 0.5321 | 0.6484 | 0.5845
Table 4: The results of the PETER few-shot experiments for various shots and
combinations of patter-verbalizer pairs (PVP) in terms of Precision (P),
Recall (R) and F1 score. Best results are boldfaced.

When it comes to the few-shot learning experiments, the results can be seen in
Table 4. We note that the combination of PVP 1 and PVP 2 (denoted "PVP 1 & 2"
in Table 4) yields better results than when they are used separately.
Comparing the results with those presented in Table 3, we can see that the
supervised models outperform the few-shot learning approaches, even when
trained on a silver-standard dataset. This suggests that more work is
necessary for few-shot NER approaches to be competitive with supervised
approaches.

##  6 Conclusion

In this work, we introduce WikiGoldSK, the first sizable, manually annotated
NER dataset in Slovak. We have established first baseline benchmarks on the
dataset using state-of-the-art models, including multilingual and Slovak-
specific models. The experiments with few-shot learning suggest that its
performance does not reach that of supervised learning. The WikiGoldSK dataset
is publicly released under permissible licensing terms, enabling training and
evaluation of future models as well as tracking the progress in Slovak NER.

## Limitations

While WikiGoldSK is currently the largest manually annotated Slovak NER
dataset, it is still small in the great scheme of things, especially when its
size (roughly 10 thousand labelled entities) gets compared to that of the
CoNLL-2003 or Czech Named Entity Corpus 2.0 datasets (both with 35 thousand
labelled entities). Our few-shot experiments have only been conducted in the
case of Slovak and the newly introduced dataset, and may not generalize to
other languages and datasets.

## Ethics Statement

The dataset used for annotation was sampled from Slovak Wikipedia, which
allows for reuse of its content under the terms of the Creative Commons
Attribution-Share-Alike License 3.0. The annotated dataset is released under
the same license.

## Acknowledgements

This project was supported by grant APVV-21-0114.

## References

  * Al-Rfou et al. (2015) Rami Al-Rfou, Vivek Kulkarni, Bryan Perozzi, and Steven Skiena. 2015.  Polyglot-ner: Massive multilingual named entity recognition.  In _Proceedings of the 2015 SIAM International Conference on Data Mining_ , pages 586–594. SIAM.
  * Balasuriya et al. (2009) Dominic Balasuriya, Nicky Ringland, Joel Nothman, Tara Murphy, and James R. Curran. 2009.  Named entity recognition in Wikipedia.  In _Proceedings of the 2009 Workshop on The People’s Web Meets NLP: Collaboratively Constructed Semantic Resources (People’s Web)_ , pages 10–18, Suntec, Singapore. Association for Computational Linguistics.
  * Bojanowski et al. (2016) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016.  Enriching word vectors with subword information.  _arXiv preprint arXiv:1607.04606_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.  _CoRR_ , abs/1911.02116.
  * Goyal et al. (2018) Archana Goyal, Vishal Gupta, and Manish Kumar. 2018.  Recent named entity recognition and classification techniques: a systematic review.  _Computer Science Review_ , 29:21–43.
  * Grishman and Sundheim (1996) Ralph Grishman and Beth M Sundheim. 1996.  Message understanding conference-6: A brief history.  In _COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics_.
  * He et al. (2021) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  _arXiv preprint arXiv:2111.09543_.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Kaššák et al. (2012) Ondrej Kaššák, Michal Kompan, and Mária Bieliková. 2012\.  Extrakcia pomenovaných entít pre slovenský jazyk.  In _Krajči, S. Znalosti 2012: Proc. of the 11th Conference, Mikulov_ , pages 52–66.
  * La Gatta et al. (2021) Valerio La Gatta, Vincenzo Moscato, Marco Postiglione, and Giancarlo Sperlì. 2021.  Few-shot named entity recognition with cloze questions.  _arXiv preprint arXiv:2111.12421_.
  * Landis and Koch (1977) J Richard Landis and Gary G Koch. 1977.  The measurement of observer agreement for categorical data.  _biometrics_ , pages 159–174.
  * Lupták (2021) Dávid Lupták. 2021.  Rozpoznávanie pomenovaných entít metódami strojového učenia.  Technická univerzita v Košiciach.
  * Maruniak (2021) Jakub Maruniak. 2021.  Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku.  Technická univerzita v Košiciach.
  * Mičo (2019) Jakub Mičo. 2019.  Rozpoznávanie pomenovaných entít metódami strojového učenia.  Slovenská technická univ. v Bratislave FIIT.
  * Nguyen et al. (2021) Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Veyseh, and Thien Huu Nguyen. 2021\.  Trankit: A light-weight transformer-based toolkit for multilingual natural language processing.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_.
  * Nothman et al. (2013) Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R Curran. 2013\.  Learning multilingual named entity recognition from wikipedia.  _Artificial Intelligence_ , 194:151–175.
  * Pan et al. (2017) Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017.  Cross-lingual name tagging and linking for 282 languages.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.
  * Pikuliak et al. (2021) Matúš Pikuliak, Štefan Grivalský, Martin Konôpka, Miroslav Blšták, Martin Tamajka, Viktor Bachratý, Marián Šimko, Pavol Balážik, Michal Trnka, and Filip Uhlárik. 2021\.  Slovakbert: Slovak masked language model.  _arXiv preprint arXiv:2109.15254_.
  * Piskorski et al. (2021) Jakub Piskorski, Bogdan Babych, Zara Kancheva, Olga Kanishcheva, Maria Lebedeva, Michał Marcinczuk, Preslav Nakov, Petya Osenova, Lidia Pivovarova, Senja Pollak, et al. 2021.  Slav-ner: the 3rd cross-lingual challenge on recognition, normalization, classification, and linking of named entities across slavic languages.
  * Piskorski et al. (2019) Jakub Piskorski, Laska Laskova, Michał Marcińczuk, Lidia Pivovarova, Pavel Přibáň, Josef Steinberger, and Roman Yangarber. 2019.  The second cross-lingual challenge on recognition, normalization, classification, and linking of named entities across slavic languages.  In _Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing_. ACL.
  * Piskorski et al. (2017) Jakub Piskorski, Lidia Pivovarova, Jan Šnajder, Josef Steinberger, and Roman Yangarber. 2017.  The first cross-lingual challenge on recognition, normalization, and matching of named entities in Slavic languages.  In _Proceedings of the 6th Workshop on Balto-Slavic Natural Language Processing_ , pages 76–85, Valencia, Spain. Association for Computational Linguistics.
  * Rajeev Sangal and Singh (2008) Dipti Misra Sharma Rajeev Sangal and Anil Kumar Singh, editors. 2008.  _Proceedings of the IJCNLP-08 Workshop on Named Entity Recognition for South and South East Asian Languages_.  Asian Federation of Natural Language Processing, Hyderabad, India.
  * Schick and Schütze (2020a) Timo Schick and Hinrich Schütze. 2020a.  Exploiting cloze questions for few shot text classification and natural language inference.  _arXiv preprint arXiv:2001.07676_.
  * Schick and Schütze (2020b) Timo Schick and Hinrich Schütze. 2020b.  It’s not just size that matters: Small language models are also few-shot learners.  _arXiv preprint arXiv:2009.07118_.
  * Tedeschi et al. (2021) Simone Tedeschi, Valentino Maiorca, Niccolò Campolungo, Francesco Cecconi, and Roberto Navigli. 2021.  WikiNEuRal: Combined neural and knowledge-based silver data creation for multilingual NER.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2521–2533, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tjong Kim Sang (2002) Erik F. Tjong Kim Sang. 2002.  Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition.  In _COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)_.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Tsai et al. (2016) Chen-Tse Tsai, Stephen Mayhew, and Dan Roth. 2016.  Cross-lingual named entity recognition via wikification.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 219–228.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.

##  Appendix A Annotation Manual

For the purpose of the WikiGoldSK dataset, we define the following classes of
Named Entities:

  * •

PER Names, surnames, nicknames of living beings, without titles. Groups of
people that belong to a nation, city, family…, e.g. Slováci (_Slovaks_ in
English), Bratislavčania (meaning: people who live in the city of Bratislava),
Kováčovci (meaning: family name). General adjectives are not entities e.g.
rímsky vojak (_roman soldier_ in English), slovenský jazyk (_slovak language_
in English), but personal adjectives are PER entity, e.g. in "to je Petrov
kufor" (_"it’s Peter’s suitcase"_ in English), "Petrov" is a PER entity.

  * •

LOC All territorial and and geo-political units, such as countries, cities,
regions… Physical locations as rivers, parks, buildings, bridges, castles,
roads… Streets were also classified as LOC entities, but without building
numbers.

  * •

ORG Political parties, companies, government institutions,
political/sport/educational organizations, music bands. Museums, zoos and
theaters were also annotated as ORG, although they are very close to LOC.
However, in our opinion, their meaning exceed the location aspect. But if the
context makes it clear that the described object is only a building and/or an
area that belongs to an organisation, LOC should be used. Companies were also
labeled with legal suffix, e.g. ESET, spol. s r.o. is all standing for one ORG
entity.

  * •

MISC Names of movies, awards, events, festivals, newspapers, TV or radio
station names. Also sport series, cups and leagues were annotated as MISC.

In case of nested entities, the outer one is recognized as entity, e.g. whole
"Národná Banka Slovenska" (_National Bank of Slovakia_ in English) is ORG
entity. Abbreviations following entity is separate entity, e.g. in "Úrad
verejného zdravotníctva (UVZ)" (_Office of Public Health (OPH)_ in English) we
annotate 2 separate ORG entities.

The main differences between our guidelines and that of the BSNLP 2017 shared
task Piskorski et al. (2017) are as follows:

  * •

For entities such as museums, theathers, zoos we’ve preferred ORG entity and
only if it’s clear from the context, LOC could be used. However, in BSNLP 2017
shared task these entities were always annotated as LOC.

  * •

We’ve used MISC entity for newspapers, TV or radio stations. In BSNLP 2017
shared task guidelines it’s not explicitly stated but in dataset these
entities are mostly annotated as ORG.

##  Appendix B PETER training details

The unlabeled dataset is created by sampling 1000 sentences from the train
split of WikiGoldSK. As a base model for training, we used SlovakBERT. To make
the prediction comparable with that of the baselines, the token-level
predictions were converted to the IOB2 form using a simple heuristic: whenever
there is a sequence of entities of the same type, the tag of the very first
entity is prefixed with B- while the rest is prefixed with I-. Note that this
is a very imperfect heuristic, as it for instance cannot handle the cases
where two entities from the same class are following straight after each
other.