:
LEAVE UNSET 2023 LEAVE UNSET LEAVE UNSET Machine Learning for Health (ML4H)
2023

# Two-stage Joint Transductive and Inductive learning for Nuclei Segmentation

Hesham Ali<EMAIL_ADDRESS>
Nile University These authors contributed equally Egypt  Idriss
Tondji11footnotemark: 1<EMAIL_ADDRESS>
African Institute for Mathematical Sciences (AIMS-AMMI)  Senegal  Mennatullah
Siam<EMAIL_ADDRESS>
Ontario Tech University  Canada

###### Abstract

AI-assisted nuclei segmentation in histopathological images is a crucial task
in the diagnosis and treatment of cancer diseases. It decreases the time
required to manually screen microscopic tissue images and can resolve the
conflict between pathologists during diagnosis. Deep Learning has proven
useful in such a task. However, lack of labeled data is a significant barrier
for deep learning-based approaches. In this study, we propose a novel approach
to nuclei segmentation that leverages the available labelled and unlabelled
data. The proposed method combines the strengths of both transductive and
inductive learning, which have been previously attempted separately, into a
single framework. Inductive learning aims at approximating the general
function and generalizing to unseen test data, while transductive learning has
the potential of leveraging the unlabelled test data to improve the
classification. To the best of our knowledge, this is the first study to
propose such a hybrid approach for medical image segmentation. Moreover, we
propose a novel two-stage transductive inference scheme. We evaluate our
approach on MoNuSeg benchmark to demonstrate the efficacy and potential of our
method.

###### keywords:

Transductive learning, Semi-supervised learning, Image segmentation.

##  1 Introduction

Figure 1: An overview of our joint transductive and inductive learning for
nuclei segmentation. Left: Our architecture with both transduction and
induction branches. Right: Our two-stage transductive inference that proposes
a refinement step in the second stage using the unlabelled test data and their
pseudo-labels from the first stage.

Accurate nuclei segmentation is important for a variety of downstream
applications, such as identifying the type of cells, detecting abnormalities
in cells, and quantifying cell proliferation. However, nuclei segmentation is
a challenging problem due to the variability in size, shape, and staining of
nuclei, as well as the presence of overlapping cells and artifacts in the
image. Over the years, classical computer vision approaches have been
developed to address this problem, such as thresholding, watershed
segmentation, and active contours Sadeghian et al. (2009); Robertson et al.
(2018); Aly et al. (2011) . However, these methods have limited accuracy and
robustness, and often require manual tuning of parameters. With the recent
advances in deep learning, there has been a growing interest in using it for
nuclei segmentation. Some notable works that have utilized deep learning for
nuclei segmentation in digital pathology include Qu et al. (2020); Valanarasu
et al. (2021). Despite the success of deep learning, one major challenge in
developing accurate models is the lack of labeled data, which is often time-
consuming and expensive to obtain. To address this issue, semi-supervised
learning methods have emerged as a promising direction with minimal labeled
data, such as Zhou et al. (2020).

Along the same line, transductive inference can leverage the unlabelled test
data itself and improve the classification decision Vapnik (1999), where
generally semi-supervised learning would utilize extra unlabelled data beyond
the test set. The standard procedure in learning is the inductive approach,
which uses the labelled training data to learn a function that can generalize
to unseen test data. This is highly dependent on the training data quality and
abundance. Yet, it has the advantage of approximating the general function and
is not specific to a certain test set. On the other hand, transductive
learning allows for the use of unlabelled test data during inference. Thus,
overcomes issues emerging from the distribution shift between the training and
test for better generalization. Nonetheless, it only improves predictions on a
specific test set. Combining the power of both can benefit from the advantages
in both learning schemes.

In this work, we present a novel approach of merging transductive and
inductive learning inspired by previous work Mao et al. (2021) that was
developed for video object segmentation on natural images. Unlike previous
work, our approach is carefully designed for medical image segmentation, where
we propose a novel two-stage transductive inference that utilizes the initial
predictions on the test set for a second refinement stage. An overview of our
approach is presented in Fig. 1. Our paper presents two key contributions:

  * •

First, we introduce a novel framework that combines transductive and inductive
learning, which has not been explored in medical image segmentation before.

  * •

Second, we present a novel two-stage inference mechanism for transductive
learning.

These contributions collectively demonstrate promising results on the well
established MoNuSeg benchmark for nuclei segmentation. Moreover, our ablation
study shows the benefit of merging both transduction and induction branches
with the two-stage transductive inference mechanism.

##  2 Methodology

###  2.1 Learning and two-stage inference

As shown in Fig. 1, we pose the data-efficient segmentation task as learning a
general function approximation to segment any unseen target image (i.e.,
inductive branch). Simultaneously a matching operation of that image to a set
of template images with their corresponding pseudo-labels, is conducted (i.e.,
transductive branch). We follow a meta-learning scheme to train such
architecture, where we emulate the inference during training.

During training, we sample pairs of template set, $\mathcal{T}$, and target
image, $X$. The template set, $\mathcal{T}=\\{T_{i},M_{i}\\}_{i=0}^{N}$,
contains a set of $N$ images, $T_{i}$, and their corresponding labels,
$M_{i}$, sampled randomly from the training set. The target image, $X$, is to
be segmented after being matched to the template images. This episodic
training within a meta-learning framework allows for training the joint
inductive and transductive branches. Since the transductive branch is based on
matching the template and target images while propagating the labels to the
target image.

During the inference stage, we propose a two-stage transductive inference
mechanism. During the first stage of inference, we have our target image, $X$,
and we similarly sample a template set, $\mathcal{T}^{(1)}$, from the training
set, where, $(1)$, indicates the first stage. After the first stage, we obtain
initial predictions, $\hat{M}^{(1)}$, for all the test data. For the second
stage we sample the template set, $\mathcal{T}^{(2)}$, from the test set
images and their corresponding first-stage pseudo-labels,
$\mathcal{T}^{(2)}=\\{T_{i},\hat{M}_{i}^{(1)}\\}$. The images in the second
stage are selected based on proximity in the feature space to the target
image, $X$. Hence, we leverage the unlabelled test set relying on the pseudo-
labels obtained from the first stage.

###  2.2 Architecture overview

In this section we detail our architecture as presented in Fig. 1. Our
architecture encompasses a convolutional backbone, $\phi$, a mask encoder,
$\gamma$, and two branches; the transduction and induction branches. The
convolutional backbone, $\phi$, extracts the features from both the template
and target images as, $F_{T}=\phi(T),F=\phi(X)$, resp. A two-head mask
encoder, $\gamma$, computes the encoded template labels,
$\\{E^{\text{tra}}_{T},E^{\text{ind}}_{T}\\}=\gamma(M)$, which are used as
input for both branches resp. Finally, the obtained mask encodings from both
branches, along with the target features from different backbone layers, are
used as input to the segmentation decoder to predict the final result.

Transduction branch. The main building block in the transduction branch is
multi-head attention Vaswani et al. (2017). It takes as input the query
$Q\in\mathbb{R}^{n\times d_{k}}$, key $K\in\mathbb{R}^{n\times d_{k}}$ and
value $V\in\mathbb{R}^{n\times d_{v}}$, where $d_{k},d_{v}$ correspond to the
channels for the keys/queries and the values respectively. Attention is then
performed as follows,

$\mathcal{A}(Q,K,V)=\text{Softmax}\left(\frac{QW^{Q}(KW^{K})^{T}}{\tau}\right)VW^{V},$
(1)

where $\tau$ donates the scaling factor and $W^{Q},W^{K},W^{V}$ denotes the
learnable weight matrices. The transformer encoder performs self attention,
where it takes a template feature $F_{T}\in\mathbb{R}^{N\times H\times W\times
C}$ as its input. The template feature is then flattened into,
$\tilde{F}_{T}\in\mathbb{R}^{NHW\times C}$. Self attention is then
instantiated as follows,
$O_{T}=\mathcal{A}(\tilde{F}_{T},\tilde{F}_{T},\tilde{F}_{T})$. Similarly, the
target features, $F$, go through the transformer encoder to provide the
encoded features, $O$. The transformer encoder learns global contextual
information from the template and target for subsequent feature matching in
the transformer decoder.

The transformer decoder is mainly composed of cross-attention, as it
propagates rich information based on the pixel level correspondence between
the target and template images. It takes the output encoded features for the
template and target images, $O_{T},O$, and the encoded template masks,
$E^{\text{tra}}_{T}\in\mathbb{R}^{NHW\times D}$, then performs cross attention
as follows,
$\widetilde{E}^{\text{tra}}_{T}=\mathcal{A}(O,O_{T},E^{\text{tra}}_{T})$. The
transformer decoder learns to propagate the labels from the encoded template
masks based on the pixel-level correspondence between the target and template
images.

Induction branch. For the induction branch, we use a few-shot learner inspired
by previous work Bhat et al. (2020) that can learn from limited labels and is
trained in an offline manner. This branch takes template feature $F_{T}$ and
mask encoding $E^{\text{ind}}_{T}$ as training sample pairs and optimizes the
kernel of a convolutional layer $\mathcal{C}_{\omega}:\mathbb{R}^{H\times
W\times C}\rightarrow\mathbb{R}^{H\times W\times D}$ using the squared error
loss.

$\mathcal{L}(\omega)=\sum_{i=1}^{N}\left\lVert\mathcal{C}_{\omega}(F_{T})-E^{\text{ind}}_{T,i}\right\rVert^{2}+\lambda\lVert\omega\rVert^{2},$
(2)

The kernel $\omega$ is a parameter that is optimized through training. i is
the index of the image in the template set, and $\lambda$ is a hyperparamter
to control the impact of the regularization term. The output from the
induction branch is then,
$\widetilde{E}^{\text{ind}}_{T}=\mathcal{C}_{\omega}(F_{T})$. It is worth
noting that the entire optimization process is fully differentiable to train
the model end-to-end.

##  3 Experimental results

Dataset description We evaluate our proposed method on the Multi-Organ Nuclei
Segmentation (MoNuSeg) dataset, which is a publicly available dataset that has
been created by labeling nuclei in pathology images. These images were
captured using H&E staining and a 40x magnification and were obtained from The
Cancer Genome Atlas (TCGA) which is a cancer genomics program. The training
data includes annotations for around 22,000 nuclei.

Table 1: Ablation study results on MoNuSeg benchmark. The final row shows our full model with join transductive and inductive branches along with two-stage inference. 2.2exMethods | Dice | F1
---|---|---
2.2exBaseline | 80.6 | 73.3
Induction | 81.6 | 73.6
Transduction | 81.2 | 73.5
Joint | 82.2 | 74.6
0.2exJoint & 2-Stage Inf. | 85.1 | 75.9

Implementation details In this study, the ResNet-50 model He et al. (2016) is
employed as the feature extractor. The scaling factor in Eq. 1, represented by
$\tau$, has been set to a value of $\frac{1}{30}$. For the few-shot learning
aspect of the induction branch, the settings used in a previous study on
Learning What to Learn (LWL) Bhat et al. (2020) were implemented. The label
encoder used in the model has two heads and produces mask representations with
16 channels.

For preparing the data, the training and testing images are cropped into
smaller patches of $256\times 256$ pixels with an overlap of $128$ pixels.
During training, each batch is constructed from five random images from the
training set as template and another five images as target. During testing,
the target images are only sampled from the test set. We use flipping and
rotation for the data augmentation. The network is trained using Adam
optimization (Loshchilov and Hutter, 2019) with learning rate of $10^{-4}$.
The full model is trained using the cross entropy loss, combined with the loss
presented in Eq. 2. We split the training set into a validation set and a
smaller training set, to identify the epoch with the highest F1 score on the
validation set. This is followed by retraining on the full training set and
using the trained weights for that best epoch. To assess the performance of
our method, we utilize the Dice coefficient and F1 score.

Table 2: Comparison of our joint inductive-transductive framework on MoNuSeg dataset with other fully supervised methods. FCN Long et al. (2015), PSP Zhao et al. (2017), DeepLab Chen et al. (2017), U-Net Ronneberger et al. (2015), ResUNet Qu et al. (2020). 2.2exMethods | Dice | F1 | Dice&F1
---|---|---|---
2.2ex FCN | 73.91 | 71.93 | 72.92
PSP | 71.52 | 73.64 | 72.58
DeepLab | 73.31 | 74.45 | 73.88
U-Net | 73.49 | 75.75 | 74.62
ResUNet | 77.13 | 80.23 | 78.68
0.2exOurs | 85.12 | 75.86 | 80.49

Ablation study We validate the effectiveness of each component of our method
on the MoNuSeg dataset. Results are shown in Tab. 1. The baseline model is
without transduction or induction branches compared to using induction branch
only and transduction branch only. The use of transduction or induction show a
marginal improvement. However, once we utilize our joint transductive and
inductive branches we gain the benefits from both. Finally, once we utilize
the two-stage inference, where the second stage uses the unlabelled test set
and its pseudo-labels as template images, a considerable increase in dice
coefficient is shown. These results confirm on the benefits from our method,
further qualitative results are shown in Fig. 2.

Figure 2: Qualitative Results of our variants.

Discussion When comparing our method on the MoNuSeg benchmark as shown in Tab.
2, we outperform previous methods on the dice coefficient and on the mean of
both the dice and F1 scores. It shows potential in our method to leverage the
unlabelled data for better performance. Note that our method is compared to
other methods that use the same convolutional backbone. Other recent
approaches that rely on transformers in their backbone e.g., Medical
Transformer (MedT) Valanarasu et al. (2021) has not been compared to. For our
future work, we plan to compare to MedT through using their transformer based
backbone in ours.

##  4 Conclusion

In this work, we proposed a novel framework that combines transductive and
inductive learning, which is the first attempt to investigate it in medical
image segmentation. We have also demonstrated the benefit from our proposed
two-stage inference mechanism.

## References

  * Aly et al. (2011) Ashraf A Aly, Safaai Bin Deris, and Nazar Zaki.  Research review for digital image segmentation techniques.  _International Journal of Computer Science & Information Technology_, 3(5):99, 2011.
  * Bhat et al. (2020) Goutam Bhat, Sibaji Saha, and Mayank Vatsa.  Learning what to learn for video object segmentation.  In _Computer Vision–ECCV 2020: 16th European Conference_ , page 16. Springer International Publishing, 2020.
  * Chen et al. (2017) Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(4):834–848, 2017.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Long et al. (2015) Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3431–3440, 2015.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2019.
  * Mao et al. (2021) Yunyao Mao, Ning Wang, Wengang Zhou, and Houqiang Li.  Joint inductive and transductive learning for video object segmentation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9670–9679, 2021.
  * Qu et al. (2020) Hui Qu, Pengxiang Wu, Qiaoying Huang, Jingru Yi, Zhennan Yan, Kang Li, Gregory M Riedlinger, Subhajyoti De, Shaoting Zhang, and Dimitris N Metaxas.  Weakly supervised deep nuclei segmentation using partial points annotation in histopathology images.  _IEEE transactions on medical imaging_ , 39(11):3655–3666, 2020.
  * Robertson et al. (2018) Stephanie Robertson, Hossein Azizpour, Kevin Smith, and Johan Hartman.  Digital image analysis in breast pathology—from image processing techniques to artificial intelligence.  _Translational Research_ , 194:19–35, 2018.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Sadeghian et al. (2009) Farnoosh Sadeghian, Zainina Seman, Abdul Rahman Ramli, Badrul Hisham Abdul Kahar, and M-Iqbal Saripan.  A framework for white blood cell segmentation in microscopic blood images using digital image processing.  _Biological procedures online_ , 11:196–206, 2009.
  * Valanarasu et al. (2021) Jeya Maria Jose Valanarasu, Poojan Oza, Ilker Hacihaliloglu, and Vishal M Patel.  Medical transformer: Gated axial-attention for medical image segmentation.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part I 24_ , pages 36–46. Springer, 2021.
  * Vapnik (1999) Vladimir N Vapnik.  An overview of statistical learning theory.  _IEEE transactions on neural networks_ , 10(5):988–999, 1999.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems 30_ , 2017.
  * Zhao et al. (2017) Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2881–2890, 2017.
  * Zhou et al. (2020) Yanning Zhou, Hao Chen, Huangjing Lin, and Pheng-Ann Heng.  Deep semi-supervised knowledge distillation for overlapping cervical cell instance segmentation.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part I 23_ , pages 521–531. Springer, 2020.


# Data Assimilation using ERA5, ASOS, and the U-STN model for Weather
Forecasting over the UK

Wenqi Wang
Department of Earth Science and Engineering
Imperial College London
<EMAIL_ADDRESS>
&Jacob Bieker
Open Climate Fix
<EMAIL_ADDRESS>
&Rossella Arcucci
Department of Earth Science and Engineering
Imperial College London
<EMAIL_ADDRESS>
&César Quilodrán-Casas
Department of Earth Science and Engineering
Imperial College London
<EMAIL_ADDRESS>

###### Abstract

In recent years, the convergence of data-driven machine learning models with
Data Assimilation (DA) offers a promising avenue for enhancing weather
forecasting. This study delves into this emerging trend, presenting our
methodologies and outcomes. We harnessed the UK’s local `ERA5` 850 hPa
temperature data and refined the U-STN12 global weather forecasting model,
tailoring its predictions to the UK’s climate nuances. From the `ASOS`
network, we sourced `T2m` data, representing ground observations across the
UK. We employed the advanced kriging method with a polynomial drift term for
consistent spatial resolution. Furthermore, Gaussian noise was superimposed on
the `ERA5` `T850` data, setting the stage for ensuing multi-time step
synthetic observations. Probing into the assimilation impacts, the `ASOS`
`T2m` data was integrated with the `ERA5` `T850` dataset. Our insights reveal
that while global forecast models can adapt to specific regions, incorporating
atmospheric data in DA significantly bolsters model accuracy. Conversely, the
direct assimilation of surface temperature data tends to mitigate this
enhancement, tempering the model’s predictive prowess.

##  1 Introduction

Numerical Weather Prediction (NWP) has progressed over the decades,
establishing itself as a cornerstone in weather forecasting [22]. Its
efficacy, demonstrated across diverse environmental scenarios, provides
probabilistic forecasts [15]. Among various NWP models, the European Centre
for Medium-Range Weather Forecasting (ECMWF) IFS (Integrated Forecasting
System) is consistently recognised for its accurate representation of
atmospheric conditions in designated geographical areas [18]. While it
maintains a lead over machine learning (ML) models, the rise of ML-driven
models — with their refined designs and increasing resolutions — cannot be
overlooked [17]. These models leverage historical atmospheric data, aspiring
to match or surpass IFS predictions while conserving computational resources
and boosting processing speeds. This includes resolutions from 5.625 degrees
in the WeatherBench Convolution Neural Network (CNN) series [16] to 1.0 degree
in Keisler’s Graph Neural Network (GNN) model [10] and 0.25 degree in
GraphCast [12]. Even though ML model developers do not anticipate replacing
NWP entirely, the strides made in ML suggest its capacity to refine existing
forecasting approaches [12]. In recent years, the integration of data-driven
ML models and Data Assimilation (DA) techniques has attracted substantial
attention due to its potential to enhance model performance [3]. DA serves as
an intermediary, merging observational data with model outputs to hone initial
backgrounds, thereby fostering improved forecast results. Integrating
obervational data in a NWP is not trivial as the model needs to be restarted
from new initial conditions [4]. Integrating new observational data in an ML
model through fine-tuning via DA has been shown to be very effective [6]. In
the domain of atmospheric research, 850 hPa temperature (`T850`) and 500 hPa
geopotential height (`Z500`) are fundamental datasets. Their significance
stems from their representation of large-scale circulation in the troposphere,
which inherently affects near-surface weather conditions and extremities [5].
The objective of weather forecasting is to predict the future state of the
atmosphere for a designated time [16]. However, this forecasting is confined
to the atmosphere’s prediction horizon, estimated to be approximately two
weeks [20].

In this study, we not only used data from ERA5 and the ASOS network but also
incorporated global meteorological data from the U-STN model. ML models excel
at handling heterogeneous data from multiple sources. They can automatically
identify correlations between different datasets and effectively integrate
this information, which is crucial for improving DA outcomes. Weather systems
are highly complex and nonlinear. ML models, especially deep learning models,
are adept at capturing this complexity, providing a finer understanding
compared to traditional numerical weather prediction models. One key advantage
of ML models is their ability to adapt and respond to changes in new data.
This means that as we collect more meteorological data and real-time
observations, the models can continuously self-optimize and adjust, thereby
enhancing long-term prediction accuracy.

##  2 Methods

In this study, we adapted a global weather forecasting model [5] employing a
U-Net [19] enhanced with a deep spatial transformer (U-STN) and focused on a
single data variable. This model was specifically retrained for the UK region.
To explore the optimisation effects of DA on predictive outcomes, we
incorporated the sigma-point Ensemble Kalman Filter (SPEnKF) algorithm [1],
particularly leveraging surface data from ground observation stations. A
primary model was trained using a distinct 12-hour time interval on `ERA5`
dataset: U-STN12. In our study, four different data typologies were utilized
for DA: (1) Employing $\sigma_{\text{obs}}=\sigma_{z}$ and
$\sigma_{\text{obs}}=0.5\sigma_{z}$, the `T850` data was augmented with
Gaussian noise at two levels, simulating observational data with noise
coefficients of 1 and 0.5, respectively. This methodology is aligned with
standard practices in DA and facilitates the examination of DA’s impact [5];
(2) Utilizing $\sigma_{\text{obs}}=\sigma_{z}$ and
$\sigma_{\text{obs}}=0.5\sigma_{z}$, synthetic observations were generated on
`T850` via U-STN12 model, offering a closer approximation to actual
observations than Gaussian noise; (3) Observational data were directly
simulated using data from ASOS observation stations; (4)The `ERA5` `T2m` data
served as a surrogate for simulated observational data. Theoretically, its
data distribution mirrors that of the ASOS data, thereby mitigating potential
alterations in data distribution characteristics during the processing of ASOS
data. Our primary objective is to find the influence of incorporating ground
observational data in DA processes for atmospheric datasets.

###  2.1 Datasets

####  2.1.1 ERA5

The ECMWF’s `ERA5` reanalysis dataset integrates NWP forecast models with
contemporaneous observations using 4D-Var DA [8]. We collected `T850` hourly
data from 1940 to present [2] and `T2m` hourly data from 1940 to present,
encompassing the timeframe from 1979 to 2022. The 2022 annual `T850`
temperature pattern is shown in 1 (b). For more details of the data and the
processing procedures, see Appendix Appendix

####  2.1.2 ASOS Dataset

Recognising the potential correlations with surface observational data, we
incorporated 2-meter temperature (`T2m`) data from `ASOS` via the Iowa
Environmental Mesonet [9], which delivers hourly surface observations spanning
the period from 1979 to 2022 and encapsulating readings from 112 observation
stations within the UK. This system predominantly supports aviation weather
forecasting and further weather forecasting research [13]. The 2022 annual
`T2m` temperature and distribution of observation stations are depicted in 1
(a). Our aim with the `ASOS` data was to align its structure with the `ERA5`
data, as visualised in Figure 1.

Figure 1: (a) Geographical distribution of ASOS surface observation stations
and annual average T2m values between latitudes 50° to 57.75° and longitudes
-6° to 1.875°(2022). (b) Annual average T850 temperature pattern from ERA5
within the same latitudinal and longitudinal bounds (2022).

###  2.2 Model

We employed a U-Net-based ML model enhanced with a deep spatial transformer
[5]. Using the Adam optimiser [11] and a mean squared error (MSE) loss
function, the model’s parameters were aligned with those specified in [5].
Although our primary goal is the exploration of DA from ground observation
stations, promising results may prompt further hyperparameter optimisation
specific to the UK or other regions. Our prediction approach is
autoregressive: a prediction at time $T(t+\Delta t)$ becomes the input for
$T(t+2\Delta t)$. Depending on the forecast’s intended duration, $\Delta t$
might vary. In this work, we set $\Delta t$ at 12h for a model named U-STN12.
The specific model code and implementation are available on GitHub:
https://anonymous.4open.science/r/TacklingClimateChangeAIforNeurIPS2023_2_02DF/src/models.

###  2.3 Data assimilation

We adopted the SPEnKF approach [21]. The SPEnKF, unlike the EnKF, employs an
unscented transformation that utilizes sigma points, which are a predefined
set of points used to estimate the mean and variance of a nonlinear function.
By processing these sigma points through the nonlinear function, it becomes
feasible to gauge the output’s mean and covariance without relying on
perturbation-based linearisation [1]. The specific assimilation code and
implementation are available on GitHub: https://github.com/acse-
ww721/DA_ML_ERA5_ASOS_Weather_Forecasting_UK.

##  3 Results

We employed U-STN12 as our ML model and SPEnKF as the DA algorithm. The
prediction’s initial value is derived from a noisy observation. Every 24
hours, we assimilate this noisy observation to refine the prediction. By
introducing various noise levels, Gaussian noise $N(0,\sigma_{\text{obs}})$ is
superimposed onto the `ERA5` `T850` data. We then analysed the root mean
square error (RMSE) between the predicted mean and the noisy data in the
`T850` domain over the initial 120 hours across 50 random conditions. The
result is shown in Figure 2. Our attention is primarily drawn to the
similarities and disparities in trends before and after the DA juncture. Given
that our model commences training from the 12 lead time, the RMSE for the
initial 12 hours exhibits an upward trend due to lack of training. Between the
12 and 48 hours lead time, there is a consistent reduction in RMSE,
underscoring the model’s effective predictive performance. Notably, a marked
decrease in RMSE is observed at the 24-hour lead time point, corresponding
with the introduction of DA. This decrease is followed by a gradual increase
in RMSE over time, likely a consequence of accumulating losses. A scenario
where $\sigma_{obs}=\sigma_{T}$ results in a larger RMSE, suggesting that
increased noise levels detrimentally impact prediction accuracy. Nonetheless,
the RMSE trends remain consistent between $\sigma_{obs}=\sigma_{T}$ and
$\sigma_{obs}=0.5\sigma_{T}$, attesting to the model’s robustness.

Figure 2:  Root Mean Square Error (RMSE) Comparison for the U-STN12-SPEnKF
Model Across Different DA Sources.

As illustrated in Figure 2 (b), We experimented with using the model’s
training data at a time step $t=12$ hours as simulated observations, as an
alternative to Gaussian noise simulated observations over a 24-hour DA span.
Both $\sigma_{obs}=0.5\sigma_{T}$ and $\sigma_{obs}=\sigma_{T}$ exhibit
consistent performance trends. Notably, with every 12-hour increment when
synthetic observations are introduced, there is a discernible reduction in
RMSE.

In Figure 2 (c) and (d), when contrasting this with the RMSE trend without the
addition of synthetic observations, it is evident that introducing these
synthetic observations results in a more gradual ascent in the model’s RMSE.
When juxtaposed with the RMSE trajectory without synthetic observations, the
early stages, characterised by high model prediction accuracy, did not
manifest any pronounced benefits from the synthetic measurements. However,
past the 48-hour mark, as the model’s predictive efficacy weakens, the RMSE
resulting from the inclusion of synthetic observations was notably lower than
that from Gaussian noise simulation.

As illustrated in Figure 2 (e), the integration of `T2m` data from the `ASOS`
ground observation station into the DA process impairs its efficacy. A
significant RMSE spike at the assimilation juncture manifestly evidences this
degradation. As Figure 2 reveals, the original `ASOS` ground observation
stations exhibit a sparse and non-uniform distribution. When interpolated to
match the `ERA5` dataset resolution, inevitable errors emerge. To validate
that discrepancies between ground and atmospheric data diminish DA
performance, we similarly integrated `ERA5` `T2m` data into the assimilation
process. As shown in Figure2 (f), this integration mirrors the previous trend,
with RMSE elevating at assimilation points, confirming the detrimental impact
on DA. However, it is salient that the RMSE when utilising `ASOS` ground data
is less than that from the `ERA5` `T2m` data. This observation paves the way
for further optimisation of the interpolation technique and exploration of
ground-atmospheric observation correlations.

##  4 Conclusion

Adapting a global ML model to regional predictions is viable. However,
regional-specific re-tuning of hyperparameters or model architecture is
essential. The SPEnKF assimilation method, when applied with model-based noisy
or multi-time step synthetic observations, augments prediction accuracy. Given
the inherent discrepancies and spatial irregularities of `ASOS` observational
data, especially within the UK context, the direct assimilation of surface
observation data with atmospheric `T850` appears inadvisable. Based on our
findings, we propose three refinements: (1) Data Sources and Interpolation:
Acquire denser datasets and implement advanced interpolation and preprocessing
techniques to better align `T2m` values with `T850`; (2) Hyperparameter
Tuning: Optimise hyperparameters in our models, including evaluating the
number of autoregressive time steps, to enhance performance; and (3)
Incorporation of Multi-Layer Pressure Levels: Integrate ground observation
data with models employing multi-layer pressure levels, such as Graph Neural
Networks (GNNs), to reduce interpolation errors and improve prediction
accuracy. Through these enhancements, we aim to uncover patterns in using
ground observation data for DA. Beyond the structural and data-based
refinements previously mentioned, incorporating regional domain knowledge can
dramatically improve performance and adaptability. Establishing a feedback
system where the predictions of the model are continuously compared against
actual observations. Any discrepancies can be used as learning points for the
model, enabling it to self-correct and adapt to the specificities of the
region over time.

## References

  * [1] Sigma-Point Kalman Filter Data Assimilation Methods for Strongly Nonlinear Systems.  Journal of the Atmospheric Sciences, 66(2), 2009.
  * [2] B. Bell, H. Hersbach, A. Simmons, P. Berrisford, P. Dahlgren, A. Horányi, J. Muñoz-Sabater, J. Nicolas, R. Radu, D. Schepers, C. Soci, S. Villaume, J.-R. Bidlot, L. Haimberger, J. Woollen, C. Buontempo, and J.-N. Thépaut.  The ERA5 global reanalysis: Preliminary extension to 1950.  Quarterly Journal of the Royal Meteorological Society, 147(741):4186–4227, 2021.
  * [3] Mark Buehner, Ron McTaggart-Cowan, and Sylvain Heilliette.  An Ensemble Kalman Filter for Numerical Weather Prediction Based on Variational Data Assimilation: VarEnKF.  Monthly Weather Review, 145(2):617–635, February 2017.
  * [4] César Quilodrán Casas, Rossella Arcucci, Pin Wu, Christopher Pain, and Yi-Ke Guo.  A reduced order deep data assimilation model.  Physica D: Nonlinear Phenomena, 412:132615, 2020.
  * [5] Ashesh Chattopadhyay, Mustafa Mustafa, Pedram Hassanzadeh, Eviatar Bach, and Karthik Kashinath.  Towards physics-inspired data-driven weather forecasting: integrating data assimilation with a deep spatial-transformer-based U-NET in a case study with ERA5.  Geoscientific Model Development, 15(5):2221–2237, March 2022.
  * [6] Sibo Cheng, César Quilodrán-Casas, Said Ouala, Alban Farchi, Che Liu, Pierre Tandeo, Ronan Fablet, Didier Lucor, Bertrand Iooss, Julien Brajard, et al.  Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review.  IEEE/CAA Journal of Automatica Sinica, 10(6):1361–1387, 2023.
  * [7] J. Zhuang et al.  pangeo-data/xesmf: v0.8, Sep. 01 2023.
  * [8] Hersbach H., Bell, B., P. Berrisford, G. Biavati, A. Horányi, and J. Muñoz Sabater.  ERA5 hourly data on pressure levels from 1940 to present.  Copernicus Climate Change Service (C3S) Climate Data Store (CDS).
  * [9] Daryl Herzmann.  Iem:: Download asos/awos/metar data.  https://mesonet.agron.iastate.edu/request/download.phtml.  Accessed: Sep. 03, 2023.
  * [10] Ryan Keisler.  Forecasting Global Weather with Graph Neural Networks, February 2022.  arXiv:2202.07575 [physics].
  * [11] Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  3rd International Conference for Learning Representations, Jan 2015.
  * [12] Remi Lam et al.  GraphCast: Learning skillful medium-range global weather forecasting, December 2022.
  * [13] Richie Lengel.  Everything You Need to Know about AWOS and ASOS, June 2018.
  * [14] S. Müller and L. Schüler.  Geostat-framework/gstools: v1.5.0 "nifty neon".  Zenodo, Jun. 15 2023.
  * [15] Tim Palmer.  The ECMWF Ensemble Prediction System: Looking Back (more than) 25 Years and Projecting Forward 25 Years.  Quarterly Journal of the Royal Meteorological Society, 145(S1):12–24, 2019.
  * [16] Stephan Rasp, Peter D. Dueben, Sebastian Scher, Jonathan A. Weyn, Soukayna Mouatadid, and Nils Thuerey.  WeatherBench: A benchmark dataset for data-driven weather forecasting.  Journal of Advances in Modeling Earth Systems, 12(11), November 2020.
  * [17] Stephan Rasp, Stephan Hoyer, Alexander Merose, Ian Langmore, Peter Battaglia, Tyler Russel, Alvaro Sanchez-Gonzalez, Vivian Yang, Rob Carver, Shreya Agrawal, et al.  Weatherbench 2: A benchmark for the next generation of data-driven global weather models.  arXiv preprint arXiv:2308.15560, 2023.
  * [18] Stephan Rasp and Nils Thuerey.  Data-driven medium-range weather prediction with a Resnet pretrained on climate simulations: A new model for WeatherBench.  Journal of Advances in Modeling Earth Systems, 13(2):e2020MS002405, February 2021.
  * [19] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  * [20] S. Scher and G. Messori.  Generalization properties of feed-forward neural networks trained on lorenz systems.  Nonlinear Processes in Geophysics, 26(4):381–399, 2019.
  * [21] Rudolph F. van der Merwe, Eric A. Wan, and Simon J. Julier.  Sigma-Point Kalman Filters for Nonlinear Estimation and Sensor-Fusion–Applications to Integrated Navigation.  In Proceedings of the AIAA Guidance, Navigation & Control Conference, volume 3, Aug 2004.
  * [22] Jonathan Weyn, Dale Durran, and Rich Caruana.  Can machines learn to predict weather? using deep learning to predict gridded 500-hpa geopotential height from historical weather data.  Journal of Advances in Modeling Earth Systems, 11, 08 2019.

## Appendix

### Datasets

#### ERA5

We utilised the `ERA5` dataset spanning the years 1979 to 2020 for training,
which comprises approximately 367,920 samples. The 2021 dataset, containing
around 8,760 samples, was designated for validation, while the 2022 dataset,
with a similar count of roughly 8,760 samples, was allocated for testing. Both
`ERA5` datasets encompass a geographical expanse with latitude ranging from
49°to 58°and longitude from -8°to +2°.

The inception year of 1979 was strategically chosen, reflecting a pronounced
enhancement in data availability and quality post this year. This period
correlates with significant advancements in ancillary research domains, such
as climate change studies [8].

To ensure integrity in the preprocessing of the downloaded raw data, efforts
were directed to retain its intrinsic properties. This strategy aids
subsequent comparisons with prevailing global weather forecast models [5] and
the WeatherBench [16]and WeatherBench2 [17]. The preprocessing steps involved:

(1) Cropping: The focus area was delimited to latitudes between 57.75°and
50.0°, and longitudes between -6° and 1.875°. (2) Remeshing: The tool xsemf
[7]was employed to realize a grid configuration of 32x64. (3) Handling Null
Values: In instances of null data points, a 3-hour sliding window was applied,
imputing the average value. Consequently, the annual datasets were re-gridded,
conforming to the dimensions (time, latitude, longitude) of 8760×32×64, with a
slight adjustment to 8784×32×64 for leap years.

#### ASOS Dataset

We adopted the `ASOS` dataset by the following preprocessing steps: (1) Hourly
Normalisation: Recognizing the varied reporting schedules of `ASOS` stations,
we standardized the hourly reporting times. Specifically: (a)For timestamps
within the first 30 minutes of an hour (e.g., 01:20:00), if a full-hour data
point precedes it, we omit the data. If not, it’s shifted to the preceding
full hour. (b)For timestamps beyond the 30-minute mark (e.g., 01:50:00), if a
full-hour data point follows, the data is omitted; otherwise, it’s shifted to
the subsequent full hour. (2)Region Filtering: We retained data within the
bounds of latitude [50, 58] and longitude [-6, +2]. (3) Kriging Interpolation:
Given the uneven station density, we used kriging interpolation via gstools
[14] to ensure spatial consistency. Polynomial drift terms were incorporated
to capture trends along latitude and longitude, and their interplay. The
formula for drift is:

$\text{drift}=F(1,\text{lat},\text{lon},\text{lat}^{2},\text{lon}^{2},\text{lat}\times\text{lon})$
(1)

where $1$ is the baseline constant term, the lat term represents the linear
variation with latitude, the lon term indicates the linear variation with
longitude, $\text{lat}^{2}$ denotes the quadratic trend with latitude,
$\text{lon}^{2}$ signifies the quadratic trend with longitude, and
$\text{lat}\times\text{lon}$ term captures the interaction of latitude and
longitude.

In meteorology, spatial data often exhibit non-stationarity, meaning that the
statistical characteristics of the data (such as mean and variance) vary
across space. For example, the climate in the UK is influenced by various
factors, including topography and proximity to the ocean, which can lead to
different trends in variables like temperature and rainfall across space. The
Kriging method with polynomial drift effectively handles this non-stationarity
by introducing a polynomial trend term to capture these spatial variations.

The polynomial drift term allows the model to consider trends in variables as
they vary geographically. This is particularly important when dealing with
variables like temperature, which may change with latitude, altitude, and
other geographic factors. By modelling these trends, we can more accurately
estimate meteorological conditions at unobserved points.


# A Confidence-based Partial Label Learning Model for
Crowd-Annotated Named Entity Recognition

Limao Xiong1, Jie Zhou1 , Qunxi Zhu2, Xiao Wang1, Yuanbin Wu3, Qi Zhang1,
Tao Gui4, Xuanjing Huang1, Jin Ma5, Ying Shan5
1 School of Computer Science, Fudan Univerisity
2 Research Institute of Intelligent Complex Systems, Fudan University
3 The Department of Computer Science and Technology, East China Normal
University
4 Institute of Modern Languages and Linguistics, Fudan University
5 Applied Research Center (ARC), Tencent PCG
∗ Corresponding author<EMAIL_ADDRESS>
###### Abstract

Existing models for named entity recognition (NER) are mainly based on large-
scale labeled datasets, which always obtain using crowdsourcing. However, it
is hard to obtain a unified and correct label via majority voting from
multiple annotators for NER due to the large labeling space and complexity of
this task. To address this problem, we aim to utilize the original multi-
annotator labels directly. Particularly, we propose a Confidence-based Partial
Label Learning (CPLL) method to integrate the prior confidence (given by
annotators) and posterior confidences (learned by models) for crowd-annotated
NER. This model learns a token- and content-dependent confidence via an
Expectation–Maximization (EM) algorithm by minimizing empirical risk. The true
posterior estimator and confidence estimator perform iteratively to update the
true posterior and confidence respectively. We conduct extensive experimental
results on both real-world and synthetic datasets, which show that our model
can improve performance effectively compared with strong baselines.

##  1 Introduction

Named entity recognition (NER) plays a fundamental role in many downstream
natural language processing (NLP) tasks, such as relation extraction Bach and
Badaskar (2007), event extraction Wadden et al. (2019); Zhou et al. (2022).
Recently, by leveraging deep learning models, existing NER systems have
witnessed superior performances on NER datasets. However, these models
typically require a massive amount of labeled training data, such as MSRA
Levow (2006), Ontonotes 4.0 Weischedel et al. (2011), and Resume Zhang and
Yang (2018). In real applications, we often need to consider new types of
entities in new domains where we do not have existing annotated. The majority
way to label the data at a lower cost is crowdsourcing Peng and Dredze (2015),
which labels the data using multiple annotators.

The crowd-annotated datasets are always low quality for the following two
reasons. First, as an exchange, crowd annotations are always nonexperts.
Various annotators may have different interpretations of labeling guidelines.
Moreover, they may make mistakes in the labeling process. It is hard to
require a number of annotations to reach an agreement. For example, annotator
1 labels “David and Jack" as a PER entity, while the correct label is “David"
and “Jack" under our guidelines (Table 1). Also we should label the continuous
time and place as one entity (e.g, “tomorrow at 10:00 a.m." and “company (
room 1003 )"). Second, due to the ambiguous word boundaries and complex
composition, the NER task is more challenging compared with the text
classification tasks. Annotator 3 ignores the token “a.m." for the time entity
and adds “the" as part of the place entity falsely. Also, he/she misses the
person entities in the text. In this paper, we focus on building a powerful
NER system based on crowd-annotated data, which is of low quality.

There are two main ways to utilize crowd-annotated data. One simple and
important way to obtain high-quality annotations for each input instance is
majority voting. As shown in Table 1, the majority voting method can not
obtain the correct answers from these three annotations well. The right labels
(e.g., “David", “Jack", “tomorrow at 10:00 a.m.", and “company ( room 1003 )")
are only annotated by annotators 1 and 2 once. Another majority of work models
the differences among annotators by finding the trustworthy annotators
Rodrigues et al. (2014); Nguyen et al. (2017); Yang et al. (2018). From Table
1, we can find that none of the three annotators labels the entities
absolutely right. Thus, these two kinds of methods are a waste of human labor.

3 Annotator 1 |  $\overbrace{\mathrm{\hbox{\pagecolor{green!30}David~{}and~{}Jack}}}^{\mathrm{PER~{}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\usym{2717}}}}$ will hold a meeting $\overbrace{\mathrm{\hbox{\pagecolor{blue!30}tomorrow~{}at~{}10:00~{}a.m.}}}^{\mathrm{TIME~{}{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\usym{2713}}}}$ in the $\overbrace{\mathrm{\hbox{\pagecolor{red!30}company~{}(~{}room~{}1003~{})}}}^{\mathrm{PLACE~{}{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\usym{2713}}}}$.
---|---
Annotator 2 |  $\overbrace{\mathrm{\hbox{\pagecolor{green!30}David}}}^{\mathrm{PER~{}{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\usym{2713}}}}$ and $\overbrace{\mathrm{\hbox{\pagecolor{green!30}Jack}}}^{\mathrm{PER~{}{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\usym{2713}}}}$ will hold a meeting $\overbrace{\mathrm{\hbox{\pagecolor{blue!30}tomorrow}}}^{\mathrm{TIME~{}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\usym{2717}}}}$ at $\overbrace{\mathrm{\hbox{\pagecolor{blue!30}10:00~{}a.m.}}}^{\mathrm{TIME~{}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\usym{2717}}}}$ in the $\overbrace{\mathrm{\hbox{\pagecolor{red!30}company}}}^{\mathrm{PLACE~{}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\usym{2717}}}}$ ( $\overbrace{\mathrm{\hbox{\pagecolor{red!30}room~{}1003}}}^{\mathrm{PLACE~{}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\usym{2717}}}}$ ).
Annotator 3 | David and Jack will hold a meeting $\overbrace{\mathrm{\hbox{\pagecolor{blue!30}tomorrow~{}at~{}10:00}}}^{\mathrm{TIME~{}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\usym{2717}}}}$ a.m. in $\overbrace{\mathrm{\hbox{\pagecolor{red!30}the~{}company~{}(~{}room~{}1003}}}^{\mathrm{PLACE~{}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\usym{2717}}}}$ ).
3 |
Table 1: The spans marked with blue, green, and red are time (TIME), person
(PER), and place (PLACE) entities labeled by three annotators.

To address this problem, we translated this task into a partial label learning
(PLL) problem, which trains the model based on the dataset where each sample
is assigned with a set of candidate labels Cour et al. (2011); Wen et al.
(2021). Thus, it is natural to utilize all human labor via PLL, which can be
divided into two types: 1) average-based methods which consider each candidate
class equally Hüllermeier and Beringer (2006); Zhang and Yu (2015); 2)
identification-based methods which predict the ground-truth label as a latent
variable via a translation matrix to describe the scores of each candidate
label Feng and An (2019); Yan and Guo (2020); Feng et al. (2020). Despite
extensive studies on PLL methods, there are still two challenges in our
condition. One challenge (C1) is that these methods are criticized when the
same candidate label occurs more than once. The general PLL is under the
assumption that each candidate label is only been assigned once, while each
sample may be assigned the same classes multiple times by the different
annotators in our situation. Another challenge (C2) is that most of the
existing studies about PLL focus on image or text classification tasks, while
we focus on a more complex task, sequence labeling, where each token is
asserted with a label. Thus, the token itself and its content should be
considered in this task.

In this paper, we propose a Confidence-based Partial Label Learning (CPLL)
model for crowd-annotated NER. For C1, we treat the classes’ labeled number
for each sample as prior confidence provided by the annotators. Also, we learn
the confidence scores via an Expectation–Maximization (EM) algorithm Dempster
et al. (1977). We estimate the real conditional probability
$P(Y=y|T=t,X=\mathbf{x})$ via a true posterior estimator based on the
confidence that consists of the prior and posterior confidences. For C2, we
learn a token- and content-dependent confidence via a confidence estimator to
consider both the token $t$ and sequence input $\mathbf{x}$, because the
candidate labels are always token-dependent and content-dependent. In fact,
our model can be applied to all the sequence labeling tasks, such as word
segment, part of speech, etc. We conduct a series of experiments on one real-
world dataset and four synthetic datasets. The empirical results show that our
model can make use of the crowd-annotated data effectively. We also explore
the influence of annotation inconsistency and balance of prior and posterior
confidences.

The main contributions of this work are listed as follows.

  * •

To better utilize the crowd-annotated data, we propose a CPLL algorithm to
incorporate the prior and posterior confidences for sequence labeling task
(i.e., NER).

  * •

To take the confidence scores into account, we design a true posterior
estimator and confidence estimator to update the probability distribution of
ground truth and token- and content-dependent confidence iteratively via the
EM algorithm.

  * •

Extensive experiments on both real-world and synthetic datasets show that our
CPLL model outperforms the state-of-the-art baselines, which indicates that
our model disambiguates the noise labels effectively.

Figure 1: The framework of our CPLL model, which consists of a true posterior
estimator and confidence estimator. The true posterior estimator is used to
predict the true posterior $P(Y=y|T=t,X=\mathbf{x})$ based on the confidence
score learned by the confidence estimator. The confidence estimator learns the
confidence based on the prior confidence obtained from annotators and the
posterior confidence learned by the model.

##  2 Our Approach

In this section, we first give the formal definition of our task. Then, we
provide an overview of our proposed CPLL model. Finally, we introduce the main
components contained in our model.

###  2.1 Formal Definition

Given a training corpus
$\mathcal{D}=\\{\mathbf{x}_{i},(\hat{Y}_{i},A_{i})\\}_{i=1}^{|\mathcal{D}|}$,
where $\mathbf{x}=\\{t_{1},t_{2},...,t_{|\mathbf{x}|}\\}$,
($\hat{Y},A)=\\{(\hat{\mathbf{y}}_{1},\mathbf{a}_{1}),(\hat{\mathbf{y}}_{2},\mathbf{a}_{2}),...,(\hat{\mathbf{y}}_{|\mathbf{x}|},\mathbf{a}_{|\mathbf{x}|})\\}$.
Here, $\hat{\mathbf{y}}=\\{y_{1},y_{2},....,y_{|\hat{\mathbf{y}|}}\\}$ is the
candidate label set of the token $t$ and
$\mathbf{a}=[a_{1},a_{2},...,a_{|\hat{\mathbf{y}}|}]$ is labeled times
obtained from annotations. Specifically, $a$ is the labeled times of candidate
label $y$ for token $t$.
$\hat{\mathbf{y}}\in\\{2^{\mathcal{Y}}\setminus\emptyset\setminus\mathcal{Y}\\}$
where $\mathcal{Y}$ is the label space and $2^{\mathcal{Y}}$ means the power
set. For the rest of this paper, $y$ denotes the true label of token $t$ in
text $\mathbf{x}$ unless otherwise specified. The goal of this task is to
predict the truth posterior probability $P(Y=y|T=t,X=\mathbf{x})$ of token $t$
in text $\mathbf{x}$.

###  2.2 Overview

In this paper, we propose a CONfidence-based partial Label Learning (CPLL)
model for crowd-annotated NER (Figure 1). Particularly, we learn the true
posterior $P(Y=y|T=t,X=\mathbf{x})$ via a true posterior estimator $f$ and a
confidence score $g(y;\hat{Y},t,\mathbf{x})$ by minimizing the following risk.

$\displaystyle
R=\mathbb{E}_{p(\mathbf{x},\hat{\mathbf{y}})}\left[\sum_{t\in\mathbf{x}}\sum_{y}\underbrace{g(y;\hat{\mathbf{y}},t,\mathbf{x})}_{\mathrm{Confidence}}*\underbrace{\mathcal{L}(f(y;t,\mathbf{x}),y)}_{\mathrm{True~{}posterior}}\right]$
(1)

where the classifier $f(y;t,\mathbf{x})$ is used to predict
$P(Y=y|T=t,X=\mathbf{x})$ and $\mathcal{L}$ is the loss. Particularly, we rely
on the Expectation-Maximization algorithm Dempster et al. (1977) to find the
maximum likelihood parameters of CPLL by regarding the ground truth as a
latent variable. In the M-step, we train a naive classifier $f$ to predict the
true posterior $P(Y=y|T=t,X=\mathbf{x})$ via a true posterior estimator
(Section 2.3). In the E-step, we update the confidence score via a confidence
estimator (Section 2.4), which consists of the prior confidences (calculated
from annotations) and posterior confidences (learned by model).

###  2.3 True Posterior Estimator

First, we train a naive classifier as our true posterior estimator $f$ to
infer the true posterior $P(Y=y|T=t,X=\mathbf{x})$. To model the sequence, we
adopt a pre-trained language model (BERT Kenton and Toutanova (2019))
$\mathcal{M}$ to learn a content-aware token representation. Specifically, we
input the sequence $\mathbf{x}=\\{t_{1},t_{2},...,t_{|\mathbf{x}|}\\}$ into
$\mathcal{M}$ to obtain the sequence representations,

$H=\mathcal{M}(\mathbf{x},\theta_{\mathcal{M}})$ (2)

where $\theta_{\mathcal{M}}$ is the parameters of $\mathcal{M}$,
$H=[h_{1},h_{2},...,h_{|\mathbf{x}|}]$. $h$ is token $t$’s content-aware
representation.

Then, we utilize a fully connected layer (FC) to predict the probability
distribution,

$f(y;t,\mathbf{x})=\sigma(W*h+b)$ (3)

where $\sigma$ is a sigmoid function, $\theta_{FC}=\\{W,b\\}$ is the learnable
parameters of FC. We regard $\theta=\\{\theta_{\mathcal{M}},\theta_{FC}\\}$ as
a parameter set of true posterior estimator $f$. Negative learning Kim et al.
(2019) is adopted, which not only considers “the token belongs to positive
label (candidate label $y\in\hat{\mathbf{y}}$)" but also “the token does not
belong to negative label (its complementary label $y\notin\hat{\mathbf{y}}$)".
The loss function is computed,

$\mathcal{L}(f(y;t,\mathbf{x}),y)=\left\\{\begin{array}[]{lr}-\log(f(y;t,\mathbf{x})),&y\in\hat{\mathbf{y}}\\\
-\log(1-f(y;t,\mathbf{x})),&y\notin\hat{\mathbf{y}}\\\ \end{array}\right.$ (4)

Finally, we optimize the empirical risk by integrating confidence
$g(y;\hat{\mathbf{y}},t,\mathbf{x})$ with the loss function (Equation 1). We
will introduce the confidence $g(y;\hat{\mathbf{y}},t,\mathbf{x})$ in detail
below.

###  2.4 Confidence Estimator

The confidence estimator is used to learn the confidence scores
$g(y;\hat{\mathbf{y}},t,\mathbf{x})$, which represents the confidence of label
$y$ given the token $t$, text sequence $\mathbf{x}$, and partial label
$\hat{\mathbf{y}}$.

$g(y;\hat{\mathbf{y}},t,\mathbf{x})=\alpha*c^{A}_{y;t,\mathbf{x}}+(1-\alpha)*c^{M}_{y;t,\mathbf{x}}$
(5)

where the confidence score $c^{M}_{y;t,\mathbf{x}}$ is learned by model and
$c^{A}_{y;t,\mathbf{x}}$ is given by annotators. $\alpha$ is a hyper-parameter
used to balance these two terms. The annotators will affect the quality of the
datasets and we can calculate the prior confidence based on the labeled times
of each class. However, prior confidence is biased since the annotators we
selected have biases. To address this problem, we also let the model learn the
posterior confidence to reduce the biases in prior confidence.

#### Posterior Confidence

We update posterior confidence $c^{M}_{y;t,\mathbf{x}}$ based on true
posterior distribution $P(Y=y|T=t,X=\mathbf{x})$ estimated by true posterior
estimator $f(y;t,\mathbf{x})$.

$c^{M}_{y;t,\mathbf{x}}=\left\\{\begin{array}[]{lr}\frac{exp(P(Y=y|T=t,X=\mathbf{x}))}{\sum_{\hat{y}\in\hat{\mathbf{y}}}{exp(P(Y=\hat{y}|T=t,X=\mathbf{x}))}},&y\in\hat{\mathbf{y}}\\\
\frac{exp(P(Y=y|T=t,X=\mathbf{x}))}{\sum_{\hat{y}\notin\hat{\mathbf{y}}}{exp(P(Y=\hat{y}|T=t,X=\mathbf{x}))}},&y\notin\hat{\mathbf{y}}\\\
\end{array}\right.$ (6)

We calculate the confidence score for positive and negative labels
independently.

#### Prior Confidence

We translate the labeled times $\mathbf{a}$ obtained from annotation into
prior confidence $c^{A}_{y;t,\mathbf{x}}$.

$c^{A}_{y;t,\mathbf{x}}=\left\\{\begin{array}[]{lr}\frac{exp(a)}{\sum_{\tilde{a}\in\mathbf{a}}exp(\tilde{a})},&y\in\hat{\mathbf{y}}\\\
0,&y\notin\hat{\mathbf{y}}\\\ \end{array}\right.$ (7)

Note that both $c^{M}_{y;t,\mathbf{x}}$ and $c^{A}_{y;t,\mathbf{x}}$ are token
and content dependent. The annotations are always affected by both the token
self and the content of the token. Thus, we model the confidence by
considering both the token and content. Finally, we compute the final
confidence score $g(y;\hat{\mathbf{y}},t,\mathbf{x})$ via Equation 5, which
considers both biases from annotators and models.

We update the parameters $\theta$ and confidence score in the M step and E
step of the EM algorithm. Specifically, we perform the true posterior
estimator and confidence estimator iteratively. The initialization of
$c^{M}_{y;t,\mathbf{x}}$ is $\frac{1}{|\hat{\mathbf{y}}|}$ for
$y\in\hat{\mathbf{y}}$ and $\frac{1}{|\mathcal{Y}|-|\hat{\mathbf{y}}|}$ for
$y\notin\hat{\mathbf{y}}$.

##  3 Experimental Setups

In this section, we first introduce one real-world and four synthetic datasets
we adopted to evaluate the performance (Section 3.1). Then, we list the
selected popular baselines to investigate the validity of our CPLL model
(Section 3.2). Finally, we present the implementation details and metrics to
replicate the experiment easily (Section 3.3).

###  3.1 Datasets

Real-World Dataset. To build the real-world dataset, we ask the annotators to
label the person, place, and time in the text independently. Each sample is
assigned to three annotators with guidelines and several examples. To be
specific, we ask three students to label 1000 samples as the training set. The
average Kappa value among the annotators is 0.215, indicating that the crowd
annotators have low agreement on identifying entities in this data. In order
to evaluate the system performances, we create a set of the corpus with gold
annotations. Concretely, we randomly select 881 sentences from the raw dataset
and let two experts generate the gold annotations. Among them, we use 440
sentences as the development set and the remaining 441 as the test set. Table
2 shows the statistical information of this dataset.

3 | #Sample | #TIME | #PLACE | #PERSON
---|---|---|---|---
Training | 1000 | 6934 | 958 | 3518
Dev | 440 | 955 | 147 | 351
Test | 441 | 1015 | 171 | 356
3 |  |  |  |
Table 2: The statistical information of real-world dataset. #Sample means the
number of samples in the corresponding dataset. #TIME, #PLACE and #PERSON
represent the number of time, place, and person entities.

Synthetic Datasets. Inspired by Rodrigues et al. (2014), we build synthetic
datasets by adding noise on four typical NER datasets: MSRA Levow (2006),
Weibo Peng and Dredze (2015), Ontonotes 4.0 Weischedel et al. (2011) and
Resume Zhang and Yang (2018). To simulate a real noise situation, we add noise
to the original datasets using four rules: 1) BE (Bound Error) that adds or
deletes some tokens of the entity to destroy the bound (change “room 1003" to
“(room 1003"); 2) ME (Missing Error) that removes the entity from the label
(“David" is not labeled); 3) CE (Category Error) that changes the category of
the entity (change “Location" to “Organization"); 4) SE (Segmentation Error)
that splits the entity into two entities (change “tomorrow at 10:00 am" to
“tomorrow" and “at 10:00 am"). We run each rule randomly with a perturbation
rate $r$, which is set as 10% in the experiments. Additionally, we explore the
influence of annotation inconsistency with different rates. Table 3 shows
statistical information of these datasets based on token-level majority
voting. We can find that a large number of entities are perturbed by our
rules. For example, more than 40% tokens labeled as entities are perturbed
with a perturbation rate $r$ of 20%.

3 | #Original | $r$ | #Error
---|---|---|---
| BI | C | Percent
Weibo | 4951 | 5% | 35 | 134 | 3.4%
10% | 143 | 546 | 13.9%
20% | 494 | 1706 | 44.4%
25% | 615 | 2411 | 61.0%
Resume | 79014 | 5% | 244 | 2011 | 2.8%
10% | 920 | 7361 | 10.4%
20% | 2979 | 25408 | 35.9%
25% | 4145 | 37585 | 52.8%
Ontonotes | 41203 | 5% | 295 | 1246 | 3.7%
10% | 978 | 4368 | 12.9%
20% | 3151 | 14849 | 43.6%
25% | 4420 | 20542 | 60.5%
MSRA | 241809 | 5% | 1439 | 6869 | 3.4%
10% | 5115 | 26343 | 13.0%
20% | 16729 | 86549 | 42.0%
25% | 23163 | 120707 | 59.4%
3 |  |  |  |  |
Table 3: The statistical information of synthetic datasets. #Original means
the number of the tokens labeled as an entity (not O) in the original dataset.
BI/C means the number of tokens that have a wrong BI/Category label but the
right Category/BI label. Percent $=$ (BI+C)/#Original.

###  3.2 Baselines

To verify the effectiveness of our CPLL model, we compare it with several
strong and typical baselines, which can be categorized into three groups:
voting-based models, partial label learning-based models, and annotator-based
models.

  * •

Voting-based models. We select two voting-based models, entity-level and
token-level voting models. The entity-level voting model obtains the ground
truth by voting at the entity level. The token-level voting model calculates
the ground truth by voting at the token level. A BERT-based sequence labeling
model Kenton and Toutanova (2019) is trained based on the ground truth
calculated by voting.

  * •

Partial label learning-based models. We adopt two classic PLL baselines to
utilize the crowd-annotated data with multiple candidate labels. PRODEN-mlp Lv
et al. (2020) adopts a classifier-consistent risk estimator with a progressive
identification method for PLL. Wen et al. (2021) propose a Leveraged Weighted
(LW) loss for PLL to take the partial and non-partial labels into account,
which is proved to be risk consistency. It achieved state-of-the-art results
on various computer version tasks. We implement the models by translating the
official codes to our NER task.

  * •

Annotator-based models. After seeing researchers achieve great success in
fully-supervised learning, we are easily going to think about how to gain
fully-supervised data from crowd-annotated data when we use crowdsourcing.
Seqcrowd Nguyen et al. (2017) uses a crowd component, a Hidden Markov Model
(HMM) learned by the Expectation-Maximization algorithm, to transform crowd-
annotated data into fully-supervised data instead of simply voting at token-
level or entity-level. When we get the ground truth calculated by this crowd
component, we can adopt some efficient fully-supervised learning method to
finish the corresponding task.

3 |  | Real-World | Ontonotes | Weibo | Resume | MSRA
---|---|---|---|---|---|---
|  | Dev | Test | Dev | Test | Dev | Test | Dev | Test | Test
Ours | CPLL | 90.37 | 90.60 | 79.39 | 81.47 | 69.72 | 68.23 | 96.57 | 96.07 | 95.42
Voting | Token-level | 89.45 | 90.40 | 78.17 | 80.12 | 67.79 | 63.81 | 95.81 | 95.39 | 94.68
Entity-level | 89.79 | 90.04 | 78.02 | 79.30 | 65.59 | 59.34 | 95.64 | 94.88 | 94.78
PLL | PRODEN-mlp | 87.39 | 87.90 | 73.04 | 75.36 | 66.37 | 61.85 | 93.90 | 94.90 | 92.46
LW loss | 88.80 | 89.83 | 79.07 | 80.45 | 69.63 | 64.26 | 96.37 | 95.64 | 95.35
Annotator | Seqcrowd | - | - | 62.80 | 65.34 | 47.56 | 41.49 | 92.73 | 93.30 | 91.90
Upper Bound | Clean data | - | - | 79.74 | 81.47 | 70.83 | 68.87 | 96.64 | 96.31 | 95.53
3 |  |  |  |  |  |  |  |  |  |

Table 4: The performance of our model and baselines in terms of F1. For real-
world dataset, we do not report the results on clean data and Seqcrowd since
we do not have ground truth for the training set.

3 | Real-World | Ontonotes | Weibo | Resume | MSRA
---|---|---|---|---|---
| Dev | Test | Dev | Test | Dev | Test | Dev | Test | Test
CPLL | 90.37 | 90.60 | 79.39 | 81.47 | 69.72 | 68.23 | 96.57 | 96.07 | 95.42
w/o Posterior Confidence | 89.51 | 90.08 | 79.11 | 80.42 | 68.83 | 65.84 | 95.74 | 95.38 | 94.79
w/o Prior Confidence | 90.60 | 90.94 | 79.68 | 80.87 | 70.57 | 64.90 | 96.21 | 95.70 | 95.20
w/o Both | 86.73 | 86.32 | 78.66 | 80.22 | 67.33 | 61.59 | 95.72 | 95.23 | 94.61
3 |  |  |  |  |  |  |  |  |

Table 5: The performance of ablation studies.

###  3.3 Implementation Details and Metrics

We adopt a PyTorch Paszke et al. (2019) framework Transformers to implement
our model based on GPU GTX TITAN X. Chinese-roberta-wwm-ext model Cui et al.
(2019) 111https://huggingface.co/hfl/chinese-roberta-wwm-ext/tree/main is used
for our true posterior estimator. We utilize Adam optimizer Kingma and Ba
(2014) to update our model and set different learning rates for the BERT
module (0.00002) and the rest module (0.002). The max sequence length is 512,
the batch size is 8 and the dropout rate is 0.1. We search the best $\alpha$
from 0.1 to 0.9 with step 0.1 using the development set. All the baselines use
the same settings hyper-parameters mentioned in their paper. Our source code
will be available soon after this paper is accepted.

To measure the performance of the models, we adopt Macro-F1 as the metric,
which is widely used for NER Yadav and Bethard (2018). In particular, we
evaluate the performance on the span level, where the answer will be
considered correct only when the entire span is matched.

##  4 Experimental Results

In this section, we conduct a series of experiments to investigate the
effectiveness of the proposed CPLL model. Specifically, we compare our model
with three kinds of strong baselines (Section 4.1) and do ablation studies to
explore the influence of the key parts contained in CPLL (Section 4.2). Also,
we investigate the influence of annotation inconsistency (Section 4.3) and
hyper-parameter $\alpha$, which controls the balance of posterior confidence
and prior confidence (Section 4.4).

(a) Weibo (b) Resume (c) Ontonotes

Figure 2: The influence of annotation inconsistency.

###  4.1 Main Results

To evaluate the performance of our model, we present the results of compared
baselines and our CPLL model (See Table 4). First, we can find that our model
outperforms all the baselines on both the real-world and synthetic datasets.
The labels obtained by voting-based methods (e.g., Token-level voting and
entity-level voting) always contain much noise because of the large labeling
space and the complexity of this task. For PLL-based models (e.g., PRODEN-mlp
and LW loss), they ignore the labeled times by the annotators. Furthermore,
annotator-based methods (e.g., Seqcrowd) aim to find the trustworthy label or
annotator. Note that Seqcrow does not work on Weibo and performs poorly on
Ontonotes. It is because Seqcrow cannot solve the case of small sizes or large
noise of datasets, which is also verified in Section 2. All these methods
cause information loss which affects the performance of the models largely.
Our CPLL model makes use of the crowd-annotated data by translating this task
into a PLL task to integrate confidence. Second, our CPLL model can reduce the
influence of noise effectively. From the results, we observe that CPLL obtains
comparable results with the model trained on the clean data. Our confidence
estimator can learn the bias generated by annotations effectively via the
posterior and prior confidence.

###  4.2 Ablation Studies

To evaluate the effectiveness of each part contained in our model, we do
ablation studies (See Table 5). We remove posterior confidence (w/o Posterior
Confidence), prior confidence (w/o Prior Confidence), and both of them (w/o
Both) from CPLL model. For w/o Both, we remove the confidence estimator by
setting the confidences as $1/|\hat{\mathbf{y}}|$ for partial labels and $0$
for non-partial labels.

From the results, we find the following observations. 1) Confidence estimator
can learn the annotation bias effectively. Removing it (w/o Both) reduces more
than 4 points in terms of F1 on the test sets over real-world and Weibo
datasets. 2) Both posterior confidence and prior confidence are useful for
this task. Obviously, prior confidence is vital to leverage the labeled
confidence given by annotators. However, prior confidence may exist bias since
the annotators are limited. Thus, the posterior confidence learned by the
model is also crucial for partial label learning to rectify the prediction.

###  4.3 Influence of Annotation Inconsistency

We also explore the influence of annotation inconsistency on synthetic
datasets with various perturbation rates. Annotation inconsistency is used to
model the label quality of crowd-sourcing. The bigger the perturbation rate,
the worse the quality of the annotation. We report the results with a rate
from 5% to 25% with step 5% over Weibo, Resume ,and Ontonotes datasets (Figure
2).

First, our CPLL model outperforms all the baselines with different
perturbation rates. Moreover, the higher the annotation inconsistency, the
more our model improves relative to the baselines. Our model can reduce the
influence of annotation inconsistency more effectively. Second, several
baselines almost do not work with a large perturbation rate (e.g., 25%), while
our model can handle it effectively. The F1 score of Seqcrowd is only less
than 20 when the rate $r$ is larger than 20%. Third, it is obvious that the
annotation quality will affect the performance of the model largely. The
higher the inconsistency, the worse the quality of the annotation and the
worse the performance of the model.

(a) Weibo (b) Resume (c) Ontonotes

Figure 3: The influence of hyper-parameter $\alpha$, which is leveraged to
control the balance between the posterior and prior confidence.

###  4.4 Influence of Hyper-parameter $\alpha$

We further investigate the influence of the hyper-parameter $\alpha$ (in
Equation 5), which is used to balance the posterior and prior confidence
(Figure 3). The prior confidence demonstrates the labeled confidence given by
the annotators, which is biased due to the selection of annotators. To reduce
this bias, we enhance our model to estimate the posterior confidence that is
learned by the model.

From the figures, we can observe the following observations. First, when the
noise is high, the smaller the $\alpha$, the better the performance.
Intuitively, the confidence given by annotators is not reliable when the
perturbation rate $r$ is large. Second, when the noise is low, the trend that
the larger the $\alpha$, the better the performance is relatively not as
obvious. The reason is that the model can disambiguate the ground truth from
the candidates easily since the data is clear. Most of the labels are correct
and confidence is not important at this time. All the findings indicate that
our confidence estimator can make use of prior confidence and learn posterior
confidence effectively.

##  5 Related Work

In this section, we mainly review the most related works about named entity
recognition (Section 5.1) and partial label learning (Section 5.2).

###  5.1 Named Entity Recognition

Named Entity Recognition (NER) is a research hotspot since it can be applied
to many downstream Natural language Processing (NLP) tasks. A well-trained NER
model takes language sequence as input and marks out all the entities in the
sequence with the correct entity type. NER is widely treated as a sequence
labeling problem, a token-level tagging task Chiu and Nichols (2015); Akbik et
al. (2018); Yan et al. (2019). Also, some of the researchers regard NER as a
span-level classification task Xue et al. (2020); Fu et al. (2021); Alemi et
al. (2023). In these works, NER is a fully-supervised learning task based on
large-scale labeled data, where each token is asserted with a golden label.

Crowdsourcing platforms (e.g., Amazon Mechanical Turk) are a popular way to
obtain large labeled data. Due to the large label space and complexity of NER,
the quality of labeled data is low. The ground truth obtained by simple
majority voting contains a lot of noise, which limits the performance of the
model largely. There is some literature that trains the model from multiple
annotators directly Simpson and Gurevych (2019); Nguyen et al. (2017). They
mainly focus on modeling the differences among annotators to find a
trustworthy annotator. In fact, a sentence may not be correctly labeled by all
the annotators while they all may label part of the right entities. To address
this problem, we translate this task into a partial label learning problem
with a prior confidence score.

###  5.2 Partial Label Learning

Unlike fully-supervised learning, which uses data with golden label y, Partial
Label Learning (PLL) asserts a candidate set $\mathcal{Y}$ for each input x
Zhang et al. (2016); Wang et al. (2023); Lv et al. (2020). Despite the fact
that we can not ensure golden label y always in the candidate set
$\mathcal{Y}$, most PLL researchers assume one of the candidate labels is the
golden label for simplicity. The existing studies about PLL can be categorized
into two groups, average-based methods Zhang and Yu (2015) and identification-
based methods Jin and Ghahramani (2002); Lyu et al. (2019). Average-based
methods Zhang and Yu (2015); Hüllermeier and Beringer (2006) intuitively treat
the candidate labels with equal importance. The main weakness of these
algorithms is that the false positive may severely distract the model with
wrong label information. Recently, identification-based methods Jin and
Ghahramani (2002); Wang et al. (2023) are proposed to identify the truth label
from the candidates by regarding the ground truth as a latent variable. More
and more literature pays attention to representative methods Lyu et al.
(2019); Nguyen and Caruana (2008), self-training methods Wen et al. (2021),
loss function adjustments Wu and Zhang (2018).

However, most of the current work focuses on image classification or text
classification tasks, while how to model the confidence for NER is not well
studied. The sequence labeling task aims to identify the entities in the
sentence with an entity type in the token level. Thus, how to model the token
self and its content also plays an important role in this task. To address
this problem, we design a confidence estimator to predict the token- and
content-dependent confidence based on the prior confidence given by
annotators.

##  6 Conclusion and Future Work

In this paper, we translate crowd-annotated NER into a PLL problem and propose
a CPLL model based on an EM algorithm. To rectify the model’s prediction, we
design a confidence estimator to predict token- and content-dependent
confidence by incorporating prior confidence with posterior confidence. We
conduct the experiments on one real-world dataset and four synthetic datasets
to evaluate the performance of our proposed CPLL model by comparing it with
several state-of-the-art baselines. Moreover, we do ablation studies to verify
the effectiveness of the key components and explore the influence of
annotation inconsistency. In the future, we would like to investigate the
performance of our model on other sequence labeling tasks.

## Limitations

Although our work shows that our CPLL model can learn from crowd-annotated NER
data well, there are at least two limitations. First, we set the hyper-
parameter $\alpha$ manually. It would be better if we could design a strategy
to learn a $alpha$ adaptive value for each sample atomically. Second, though
we mainly experiment on NER tasks, our model can be applied to all sequence
labeling tasks, such as part-of-speech tagging (POS), Chinese word
segmentation, and so on. We would like to explore it in further work.

## Acknowledgements

The authors wish to thank the anonymous reviewers for their helpful comments.
This work was partially funded by National Natural Science Foundation of China
(No.62206057), Shanghai Rising-Star Program (23QA1400200), Natural Science
Foundation of Shanghai (23ZR1403500), and CCF-Tencent Open Fund.

## References

  * Akbik et al. (2018) Alan Akbik, Duncan A. J. Blythe, and Roland Vollgraf. 2018.  Contextual string embeddings for sequence labeling.  _international conference on computational linguistics_.
  * Alemi et al. (2023) Alexander Alemi, Ian Fischer, Joshua Dillon, Jacob Devlin, Ming-Wei Chang, Kenton Lee, Marco Federici, Anjan Dutta, Patrick Forré, Nate Kush, Robert Geirhos, Jörn-Henrik Jacobsen, Richard Michaelis, and Wieland Zemel. 2023\.  Miner: Improving out-of-vocabulary named entity recognition from an information theoretic perspective.  _meeting of the association for computational linguistics_.
  * Bach and Badaskar (2007) Nguyen Bach and Sameer Badaskar. 2007.  A review of relation extraction.  _Literature review for Language and Statistics II_ , 2:1–15.
  * Chiu and Nichols (2015) Jason P.C. Chiu and Eric Nichols. 2015.  Named entity recognition with bidirectional lstm-cnns.  _Transactions of the Association for Computational Linguistics_.
  * Cour et al. (2011) Timothee Cour, Ben Sapp, and Ben Taskar. 2011.  Learning from partial labels.  _The Journal of Machine Learning Research_ , 12:1501–1536.
  * Cui et al. (2019) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019.  Pre-training with whole word masking for chinese bert.  _arXiv: Computation and Language_.
  * Dempster et al. (1977) Arthur P Dempster, Nan M Laird, and Donald B Rubin. 1977.  Maximum likelihood from incomplete data via the em algorithm.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 39(1):1–22.
  * Feng and An (2019) Lei Feng and Bo An. 2019.  Partial label learning with self-guided retraining.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 3542–3549.
  * Feng et al. (2020) Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. 2020.  Provably consistent partial-label learning.  _Advances in Neural Information Processing Systems_ , 33:10948–10960.
  * Fu et al. (2021) Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021.  Spanner: Named entity re-/recognition as span prediction.  _meeting of the association for computational linguistics_.
  * Hüllermeier and Beringer (2006) Eyke Hüllermeier and Jürgen Beringer. 2006.  Learning from ambiguously labeled examples.  _Intelligent Data Analysis_ , 10(5):419–439.
  * Jin and Ghahramani (2002) Rong Jin and Zoubin Ghahramani. 2002.  Learning with multiple labels.  _Advances in neural information processing systems_ , 15.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * Kim et al. (2019) Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. 2019.  Nlnl: Negative learning for noisy labels.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 101–110.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Levow (2006) Gina-Anne Levow. 2006.  The third international chinese language processing bakeoff: Word segmentation and named entity recognition.  In _Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing_ , pages 108–117.
  * Lv et al. (2020) Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. 2020.  Progressive identification of true labels for partial-label learning.  In _International Conference on Machine Learning_ , pages 6500–6510. PMLR.
  * Lyu et al. (2019) Gengyu Lyu, Songhe Feng, Tao Wang, Congyan Lang, and Yidong Li. 2019.  Gm-pll: graph matching based partial label learning.  _IEEE Transactions on Knowledge and Data Engineering_ , 33(2):521–535.
  * Nguyen et al. (2017) An T Nguyen, Byron C Wallace, Junyi Jessy Li, Ani Nenkova, and Matthew Lease. 2017\.  Aggregating and predicting sequence labels from crowd annotations.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , volume 2017, page 299. NIH Public Access.
  * Nguyen and Caruana (2008) Nam Nguyen and Rich Caruana. 2008.  Classification with partial labels.  In _Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 551–559.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  _neural information processing systems_.
  * Peng and Dredze (2015) Nanyun Peng and Mark Dredze. 2015.  Named entity recognition for chinese social media with jointly trained embeddings.  In _Proceedings of the 2015 conference on empirical methods in natural language processing_ , pages 548–554.
  * Rodrigues et al. (2014) Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. 2014.  Sequence labeling with multiple annotators.  _Machine learning_ , 95(2):165–181.
  * Simpson and Gurevych (2019) Edwin Simpson and Iryna Gurevych. 2019.  A Bayesian approach for sequence tagging with crowds.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1093–1104, Hong Kong, China. Association for Computational Linguistics.
  * Wadden et al. (2019) David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019.  Entity, relation, and event extraction with contextualized span representations.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5784–5789.
  * Wang et al. (2023) Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. 2023.  Pico: Contrastive label disambiguation for partial label learning.  _Learning_.
  * Weischedel et al. (2011) Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Martha Palmer, Nianwen Xue, Mitchell Marcus, Ann Taylor, Craig Greenberg, Eduard Hovy, Robert Belvin, et al. 2011.  Ontonotes release 4.0.  _LDC2011T03, Philadelphia, Penn.: Linguistic Data Consortium_.
  * Wen et al. (2021) Hongwei Wen, Jingyi Cui, Hanyuan Hang, Jiabin Liu, Yisen Wang, and Zhouchen Lin. 2021.  Leveraged weighted loss for partial label learning.  In _International Conference on Machine Learning_ , pages 11091–11100. PMLR.
  * Wu and Zhang (2018) Xuan Wu and Min-Ling Zhang. 2018.  Towards enabling binary decomposition for partial label learning.  In _IJCAI_ , pages 2868–2874.
  * Xue et al. (2020) Mengge Xue, Bowen Yu, Zhenyu Zhang, Tingwen Liu, Yue Zhang, and Bin Wang. 2020.  Coarse-to-fine pre-training for named entity recognition.  _empirical methods in natural language processing_.
  * Yadav and Bethard (2018) Vikas Yadav and Steven Bethard. 2018.  A survey on recent advances in named entity recognition from deep learning models.  In _27th International Conference on Computational Linguistics, COLING 2018_ , pages 2145–2158. Association for Computational Linguistics (ACL).
  * Yan et al. (2019) Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. 2019.  Tener: Adapting transformer encoder for named entity recognition.  _arXiv: Computation and Language_.
  * Yan and Guo (2020) Yan Yan and Yuhong Guo. 2020.  Partial label learning with batch label correction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 6575–6582.
  * Yang et al. (2018) YaoSheng Yang, Meishan Zhang, Wenliang Chen, Wei Zhang, Haofen Wang, and Min Zhang. 2018.  Adversarial learning for chinese ner from crowd annotations.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Zhang and Yu (2015) Min-Ling Zhang and Fei Yu. 2015.  Solving the partial label learning problem: An instance-based approach.  In _Twenty-fourth international joint conference on artificial intelligence_.
  * Zhang et al. (2016) Min-Ling Zhang, Bin-Bin Zhou, and Xu-Ying Liu. 2016.  Partial label learning via feature-aware disambiguation.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1335–1344.
  * Zhang and Yang (2018) Yue Zhang and Jie Yang. 2018.  Chinese ner using lattice lstm.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1554–1564.
  * Zhou et al. (2022) Jie Zhou, Qi Zhang, Qin Chen, Liang He, and Xuan-Jing Huang. 2022.  A multi-format transfer learning model for event argument extraction via variational information bottleneck.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 1990–2000.