# Noisy-Labeled NER with Confidence Estimation

Kun Liu1 Yao Fu2∗ Chuanqi Tan1 Mosha Chen1
Ningyu Zhang3 Songfang Huang1 Sheng Gao4
1Alibaba Group 2University of Edinburgh 3Zhejiang University
4 Guizhou Provincial Key Laboratory of Public Big Data, Guizhou University
{kun.liu624<EMAIL_ADDRESS><EMAIL_ADDRESS>
{chuanqi.tcq; chenmosha.cms<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Equal Contribution. Corresponding author.

###### Abstract

Recent studies in deep learning have shown significant progress in named
entity recognition (NER). Most existing works assume clean data annotation,
yet a fundamental challenge in real-world scenarios is the large amount of
noise from a variety of sources (e.g., pseudo, weak, or distant annotations).
This work studies NER under a noisy labeled setting with calibrated confidence
estimation. Based on empirical observations of different training dynamics of
noisy and clean labels, we propose strategies for estimating confidence scores
based on local and global independence assumptions. We partially marginalize
out labels of low confidence with a CRF model. We further propose a
calibration method for confidence scores based on the structure of entity
labels. We integrate our approach into a self-training framework for boosting
performance. Experiments in general noisy settings with four languages and
distantly labeled settings demonstrate the effectiveness of our method 111Our
code can be found at https://github.com/liukun95/Noisy-NER-Confidence-
Estimation.

##  1 Introduction

Recent progress in deep learning has significantly advanced NER performances
(Lample et al., 2016; Devlin et al., 2018). While most existing works assume
clean data annotation, real-world data inevitably involve different levels of
noise (e.g., distant supervision from the dictionary (Peng et al., 2019), or
weak supervision from the web Vrandečić and Krötzsch, 2014; Cao et al.,
2019a). Figure 1 gives an example of such noisy labels. To train robust models
with high performance, it is fundamentally critical to tackle the challenges
associated with noisy data annotation.

In this work, we propose a confidence estimation approach for NER with noisy
labels. We motivate our approach with important empirical observations of the
training dynamics of clean and noisy labels: usually, clean data are easier to
fit with faster convergence and smaller loss values (Jiang et al., 2018; Han
et al., 2018a; Arazo et al., 2019). Consequently, loss values (probabilities
or scores of labels) can serve as strong indicators for the existence of
noise, which we utilize to build our confidence estimation.

Figure 1:  A noisy label example. Brooklyn and York are noisy positives. New
is noisy negative.

The key contribution of this work is a confidence estimation method with
calibration. We use probabilities of labels as confidence scores and apply two
estimation strategies based on global or local normalization that assume
different dependency structures about how the noisy labels are generated. We
further calibrate the confidence score for positive labels (labels
representing entity parts, e.g., B-LOC) based on the structure of these
labels: we separately estimate scores for the position part (e.g., B in B-LOC)
and the type part (e.g., LOC in B-LOC). Such fine-grained calibration leads to
a more accurate estimation and better performance in our experiments.

We apply our method in a CRF model (Bellare and McCallum, 2007; Yang et al.,
2018), marginalize out labels we do not trust, and maximize the likelihood of
trusted labels. We use a self-training approach (Jie et al., 2019) that
iteratively estimates confidence scores in multiple training iterations and
re-annotates the data at each iteration. Experiments show that our approach
outperforms baselines on a general noisy-labeled setting with datasets in four
languages and shows promising results on a distantly-labeled setting with four
datasets.

##  2 Method

Given a sentence $x=[x_{1},...,x_{n}]$ and its tag sequence
$\hat{y}_{1},...,\hat{y}_{n}$, $n$ is the sentence length. We model the
conditional probability of $y$ with a bi-directional LSTM-CRF (Huang et al.,
2015):

$\displaystyle h=\text{BiLSTM}(x)\quad\Phi_{i}=\text{Linear}(h_{i})$ (1)
$\displaystyle p(y|x)=\Phi(y)/Z\quad\alpha,Z=\text{Forward}(\Phi)$ (2)

Where $h$ denotes LSTM states, Linear$(\cdot)$ denotes a linear layer,
$\Phi(y)$ denotes the potential (weight) evaluated for tag sequence $y$, $Z$
denotes the partition function, $\alpha$ denotes the forward variables, and
Forward$(\cdot)$ denotes the Forward algorithm (Sutton and McCallum, 2006).
The advantage of the CRF model is that it gives us a probabilistically uniform
way to handle labels we do or do not trust by partial marginalization, which
we discuss later.

###  2.1 Confidence Score Estimation

Our confidence estimation model reuses the base LSTM-CRF architecture and
assigns a confidence score $s_{i}$ for each $\hat{y}_{i}$. A natural choice is
to use the CRF marginal probability:

$\displaystyle s_{i}=p(\hat{y}_{i}|x)\quad p(y_{i}|x)=\alpha_{i}\beta_{i}/Z$
(3)

where $\beta$ is the backward variable and can be computed with the Backward
algorithm (Sutton and McCallum, 2006). This strategy infers $s_{i}$ based on
global-normalization and assumes strong dependency between consecutive labels.
The intuition is that annotators are more likely to make mistakes on a label
if they have already made mistakes on previous labels.

Our second strategy makes a stronger local independence assumption and
considers a noisy label at step $i$ only relies on the word context, not the
label context. To this end, we use a simple categorical distribution
parameterized by a Softmax:

$\displaystyle s_{i}=p(\hat{y}_{i}|x)\quad
p(y_{i}|x)=\text{Softmax}(\Phi_{i})$ (4)

Here we reuse the factor $\Phi_{i}$ as the logits of the Softmax because in
the CRF context it also means how likely a label $y_{i}$ may be observed given
the input $h_{i}$. Intuitively, this strategy assumes that annotators make
mistakes solely based on words, no matter whether they have already made
mistakes previously.

Method | General Noise |  | Distant Supervision
---|---|---|---
En |  Sp |  Ge |  Du |  |  CoNLL |  Tweet |  Webpage |  Wikigold
1\. BiLSTM-CRF |  73.3 |  61.9 |  57.7 |  58.3 |  |  59.5 |  21.8 |  43.3 |  42.9
2\. BiLSTM-CRF (clean data upper bound) |  90.3 |  85.2 |  77.3 |  81.1 |  |  91.2 |  52.2 |  52.3 |  54.9
3\. RoBERTa (clean data upper bound) |  - |  - |  - |  - |  |  90.1 |  52.2 |  72.4 |  86.4
Proposed for General Noise Setting |  |
4\. NA Hedderich and Klakow (2018) |  61.5 |  57.3 |  46.1 |  41.5 |  |  - |  - |  - |  -
5\. CBL Mayhew et al. (2019) |  82.6 |  76.1 |  65.6 |  68.5 |  |  75.4 |  18.2 |  31.7 |  42.6
6\. Self-training (Jie et al., 2019) |  84.0 |  71.4 |  66.5 |  59.6 |  |  77.8 |  42.3 |  49.6 |  51.3
Proposed for Distant Supervision Setting |  |
7\. AutoNER Shang et al. (2018) |  - |  - |  - |  - |  |  67.0 |  26.1 |  51.4 |  47.5
8\. LRNT Cao et al. (2019a) |  - |  - |  - |  - |  |  69.7 |  23.8 |  47.7 |  46.2
9\. BOND (RoBERTa Liang et al., 2020) |  - |  - |  - |  - |  |  81.5 |  48.0 |  65.7 |  60.1
Ours, best configurations |  |
10\. Ours (local, $\tau^{*}$) |  87.0 |  78.8 |  68.3 |  69.1 |  |  79.4 |  43.6 |  51.8 |  54.0
11\. Ours (global, $\tau^{*}$) |  86.4 |  79.0 |  69.2 |  71.2 |  |  79.2 |  43.1 |  50.0 |  53.0
Ours, other possible configurations |  |
12\. Ours (local, $\tau^{\star}$) |  86.2 |  79.2 |  68.2 |  67.2 |  |  - |  - |  - |  -
13\. Ours (global, $\tau^{\star}$) |  85.4 |  75.4 |  68.4 |  69.0 |  |  - |  - |  - |  -
14\. Ours (local, $\tau^{*}$, w/o. calibration) |  85.8 |  77.3 |  67.2 |  68.0 |  |  79.9 |  40.8 |  46.9 |  50.0
Ours with pretrained LM |  |
15\. Ours (local, $\tau^{*}$, BERT) |  - |  - |  - |  - |  |  77.2 |  46.7 |  59.3 |  57.3
16\. Ours (global, $\tau^{*}$, BERT) |  - |  - |  - |  - |  |  78.9 |  47.3 |  61.9 |  57.7
Table 1: Results (F1%) on artificially perturbed datasets and distantly
supervised datasets. $\tau^{*}$ = searched, $\tau^{\star}$ = oracle.

###  2.2 Confidence Calibration and Partial Marginalization

We use $s_{i}$ to decide if we want to trust a label $\hat{y}_{i}$ and
marginalize out labels we do not trust. Our marginalization relies on a
threshold to determine the portion of trusted labels and the noise ratio that
we believe the data contain. Given a batch of $(x,\hat{y})$ pairs, after
confidence estimation, we collect all word-label-confidence triples into a set
$\mathcal{D}=\\{x_{j},\hat{y}_{j},s_{j}\\}_{j=1}^{N}$, $N$ denotes total
number of the triples.

We further separate the estimation for positive labels (entities) and negative
labels (i.e., the O label) because we empirically observe that their
probabilities are consistently different. To this end, we divide $\mathcal{D}$
into positive and negative groups
$\mathcal{D}_{\text{p}}=\\{(x_{j},\hat{y}_{j},s_{j}),\hat{y}_{j}\in\mathcal{Y}_{\text{p}}\\}$
and
$\mathcal{D}_{\text{n}}=\\{(x_{j},\hat{y}_{j},s_{j}),\hat{y}_{j}\in\mathcal{Y}_{\text{n}}\\}$,
$\mathcal{Y}_{\text{p}}$ and $\mathcal{Y}_{\text{n}}$ denotes sets of positive
and negative labels. We rank triples in $\mathcal{D}_{l}$ ($l\in$ {p, n})
according to confidence scores and retain the most confident
$r_{l}(e)\cdot|\mathcal{D}_{l}|$ triples at epoch $e$ as clean for which we do
maximum likelihood. We view the remaining triples as noisy and marginalize
them out. We update the keep ratio $r_{l}(e)$ at each epoch following Han et
al. (2018b):

$\displaystyle
r_{l}(e)=1-\min\left\\{\frac{e}{K}\tau_{l},\tau_{l}\right\\},l\in\\{\text{p},\text{n}\\}$
(5)

where $\tau_{l}$ is the ratio of noise that we believe in the training data.
Basically this says we gradually decrease the epoch-wise keep ratio $r_{l}(e)$
to the full ratio $1-\tau_{l}$ after $K$ epochs. We grid-search $\tau_{l}$
heuristically in experiments (results in Figure 3(b)).

Figure 2: A partial marginalization example after confidence estimation. In
this example, we do not trust any labels for New (so we marginalize all labels
out), partially trust labels for Brooklyn (B part) and York (LOC part, so we
sum over labels we trust), and fully trust labels for the rest words (so we
simply evaluate and maximize their weights.).

For positive cases in $\mathcal{D}_{\text{p}}$ viewed as noisy according to
the previous procedure, we do a further confidence calibration. Noting that a
$y_{i}$ always take the form $y_{i}^{{p}}$-$y_{i}^{{t}}$ (position-type) (e.g.
if $y_{i}=\textit{B-LOC}$, then $y_{i}^{{p}}=\textit{B}$ and
$y_{i}^{{t}}=\textit{LOC}$), an important assumption is that annotators are
unlikely to mistake both parts — mistakes usually happen on only one of them.
So we calculate two calibrated confidence scores $s_{i}^{p}$ and $s_{i}^{t}$
for $\hat{y}_{i}^{p}$ and $\hat{y}_{i}^{t}$:

$\displaystyle s_{i}^{p}$
$\displaystyle=\frac{1}{|Y(\hat{y}_{i}^{p})|}\sum_{y_{i}}p(y_{i}|x)\quad\text{
where }y_{i}^{p}=\hat{y}_{i}^{p}$ (6) $\displaystyle s_{i}^{t}$
$\displaystyle=\frac{1}{|Y(\hat{y}_{i}^{t})|}\sum_{y_{i}}p(y_{i}|x)\quad\text{
where }y_{i}^{t}=\hat{y}_{i}^{t}$ (7)

where $Y(\hat{y}_{i}^{t})$ denotes the set of labels sharing the same
$\hat{y}_{i}^{t}$ part, and $Y(\hat{y}_{i}^{p})$ is defined similarily. If
$s_{i}^{p}>s_{i}^{t}$, we trust the $\hat{y}_{i}^{p}$ (position) part of the
label and marginalize out all labels with different positions except for the O
label. For example, in Figure 2, for the word Brooklyn we trust the all labels
with the position B (B-PER and B-LOC) and the O label, sum over the tag
sequences passing these labels, and reject other labels. Similar operation
applies for cases where $s_{i}^{p}<s_{i}^{t}$ (E.g., the word York). For
labels we do not trust in the negative group $\mathcal{D}_{\text{n}}$, we
simply marginalize all labels out (E.g., the word New). We maximize the
partially marginalized probability (Bellare and McCallum, 2007):

$\displaystyle\tilde{p}(\hat{y}|x)=\sum_{y\in\tilde{Y}}\Phi(y)/Z$ (8)

where $\tilde{Y}$ denotes the set of tag sequences compatible with $\hat{y}$
after confidence estimation. A concrete example is given in Figure 2. The
summation in equation 8 can be calculated exactly with Forward-styled dynamic
programming (Sasada et al., 2016).

###  2.3 Self Training

We integrate our approach into a self-training framework proposed by Jie et
al. (2019). At each round, the training set is randomly divided into two parts
for cross-validation. We iteratively re-annotate half of the training set with
a model trained on the other half. After a round, we use the updated training
set to train the next round.

##  3 Experiments

(a) Training curve

(b) Noise rate search curve

(c) Results w.r.t. noise level

Figure 3: Analysis on English CoNLL03 dataset. (a) Dev performance strongly
correlates to loss values (confidence scores) and noise detection performance.
(b) An over-estimate of noise tends to give better performance. (c). Our
approach is particularly effective under larger noise (lower recall = larger
noise). Figure 4: Confidence estimation case study. Red fonts = noisy
positive, blue fonts = noisy negatives. Green shade = correct noise detection,
red shade = wrong noise detection.

###  3.1 Datasets and Baselines

General Noise. Following Mayhew et al. (2019), we first consider general noise
by artificially perturbing the CoNLL dataset (Sang and De Meulder, 2003) on
four languages including English, Spanish, German, and Dutch. Gold annotations
are perturbed by: (a) tagging some entities to O to lower the recall to 0.5;
(b) introducing some random positive tags to lower the precision to 0.9. We
compare our methods with Noise Adaption (NA, Hedderich and Klakow, 2018), Self
Training Jie et al. (2019), and CBL (Mayhew et al., 2019). This setting is for
testing our approach in a controlled environment.

Distant Supervision. We consider four datasets including CoNLL03 (Sang and De
Meulder, 2003), Tweet (Godin et al., 2015), Webpage (Ratinov and Roth, 2009),
and Wikigold (Balasuriya et al., 2009). In this setting, the distantly
supervised tags are generated by the dictionary following BOND (Liang et al.,
2020). We compare our methods with AutoNER (Shang et al., 2018), LRNT (Cao et
al., 2019a), and BOND. This setting aims to test our approach in a more
realistic environment.

###  3.2 Results

Table 1 shows our primary results. We use local and global to denote locally /
globally normalized confidence estimation strategies. We use oracle
(unavailable in real settings) / searched $\tau$ to denote how we obtain the
prior noise ratio $\tau$. We note that the Self-training baseline (Jie et al.,
2019, line 6) is the most comparable baseline since our confidence estimation
is directly integrated into it. We primarily compare this baseline with our
best configurations (line 10 and 11). We focus on theshaded results as they
are the most informative for demonstrating our method.

General Noise. Our methods (both local and global) outperforms the state-of-
the-art method (Jie et al., 2019) by a large margin in three datasets (En, Sp,
Du, line 10 and 11 v.s. 6), showing the effectiveness of our approach. We
observe the oracle $\tau$ does not necessarily give the best performance and
an over estimate of confidence could leave a better performance. Ablation
results without calibration further show the effectiveness of our calibration
methods (line 10 v.s. 14). We note that the CoNLL dataset is an exception
where the calibration slightly hurts performance. Otherwise the improvements
with calibration is clear in the other 7 datasets.

Distant Supervision. Our method outperforms AutoNER and LRNT without pre-
trained language models. Reasons that we are worse than BOND (line 16 v.s. 6)
are: (a) many implementation aspects are different, and it is (currently)
challenging to transplant their settings to ours; (b) they use multiple
tailored techniques for distantly-labeled data (e.g., the adversarial
training), while ours is more general-purpose. Though our method does not
outperform BOND, it still outperforms AutoNER and LRNT (under the setting all
without pretrained model, line 10 and 11 v.s. 7 and 8) and shows promising
gain.

###  3.3 Further Analysis

We conduct more detailed experiments on the general noise setting for more in-
depth understanding.

Training Dynamics (Figure 3(a)). As the model converges, as clean data
converge faster, the confidence gap between the clean and the noisy is larger,
thus the two are more confidently separated, so both noise detection F1 and
dev F1 increase.

Noise Rate Search (Figure 3(b)). Our method consistently outperforms baseline
without confidence estimation. Lines tend to be higher at the right side of
the figure, showing an over-estimate of noise tends to give better
performance.

Level of Noise (Figure 3(c)). In many real-world scenarios, the noise w.r.t.
precision is more constant and it is the recall that varies. So we simulate
the level of noise with different recall (lower recall = larger noise ratio).
Our method outperforms baselines in all ratios and is particularly effective
under a large noise ratio.

Case Studies (Figure 4). The top three cases give examples of how our method
detects: (1) false negative noise when an entity is not annotated, (2)
entities with wrong boundaries and (3) wrong entity types. The last example
(case 4) gives a failure case when the model treats some correct tags as noise
due to our over-estimate of noise (for better end performance).

##  4 Related Works

State-of-the-art NER models Ma and Hovy (2016); Lample et al. (2016); Devlin
et al. (2018) are all under the traditional assumption of clean data
annotation. The key motivation of this work is the intrinsic gap between the
clean data assumption and noisy real-world scenarios. We believe that the
noisy label setting is a fundamentally challenging in NER and all related
supervised learning tasks.

Previous works on NER with noise could be organized into two threads: (a) some
works treat this task as learning with missing labels. Bellare and McCallum
(2007) propose a missing label CRF to deal with partial annotation. Jie et al.
(2019) propose a self-training framework with marginal CRF to re-annotate the
missing labels. (b) other works treat missing labels as noise and try to avoid
them in the training process. For example, Mayhew et al. (2019) train a binary
classifier supervised by entity ratio to classify tokens into entities and
non-entities.

A widely-used way to collect NER annotations is distant supervision, which
consequently becomes an important source of noise. Peng et al. (2019)
formulate this task as the positive-unlabeled (PU) learning to avoid using
noisy negatives. AutoNER Shang et al. (2018) trains the model by assigning
ambiguous tokens with all possible labels and then maximizing the overall
likelihood using a fuzzy LSTM-CRF model. Cao et al. (2019b) and Yang et al.
(2018) try to select high-quality sentences with less annotation errors for
sequential model. Liang et al. (2020) leverage pre-trained language models to
improve the prediction performance of NER models under a self-training
framework.

Our inspiration of confidence estimation comes from the so-called memorization
effect observed in the computer vision Jiang et al. (2018); Han et al.
(2018a); Arazo et al. (2019). It observes that neural networks usually take
precedence over noisy data to fit clean data, which indicates that noisy data
are more likely to have larger loss values in the early training epochs Arpit
et al. (2017). In this work, we leverage it to estimate the confidence scores
of labels.

##  5 Conclusion

In this work, we propose a calibrated confidence estimation approach for
noisy-labeled NER. We integrate our method in an LSTM-CRF model under a self-
training framework. Extensive experiments demonstrate the effectiveness of our
approach. Our method outperforms strong baseline models in a general noise
setting (especially for larger noise ratios), and shows promising results in a
distant supervision setting.

##  6 Acknowledgments

We thank all anonymous reviewers for their helpful comments. This work is
supported by Alibaba Group through Alibaba Research Intern Program and AZFT
Joint Lab for Knowledge Engine.

## References

  * Arazo et al. (2019) Eric Arazo, Diego Ortego, Paul Albert, Noel O’Connor, and Kevin Mcguinness. 2019\.  Unsupervised label noise modeling and loss correction.  In _International Conference on Machine Learning_ , pages 312–321.
  * Arpit et al. (2017) Devansh Arpit, Stanisław Jastrzundefinedbski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. 2017.  A closer look at memorization in deep networks.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, page 233–242. JMLR.org.
  * Balasuriya et al. (2009) Dominic Balasuriya, Nicky Ringland, Joel Nothman, Tara Murphy, and James R Curran. 2009.  Named entity recognition in wikipedia.  In _Proceedings of the 2009 Workshop on The People’s Web Meets NLP: Collaboratively Constructed Semantic Resources (People’s Web)_ , pages 10–18.
  * Bellare and McCallum (2007) Kedar Bellare and Andrew McCallum. 2007.  Learning extractors from unlabeled text using relevant databases.  In _Sixth international workshop on information integration on the web_.
  * Cao et al. (2019a) Yixin Cao, Zikun Hu, Tat-seng Chua, Zhiyuan Liu, and Heng Ji. 2019a.  Low-resource name tagging learned with weakly labeled data.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 261–270, Hong Kong, China. Association for Computational Linguistics.
  * Cao et al. (2019b) Yixin Cao, Zikun Hu, Tat-Seng Chua, Zhiyuan Liu, and Heng Ji. 2019b.  Low-resource name tagging learned with weakly labeled data.  _arXiv preprint arXiv:1908.09659_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _NAACL-HLT_.
  * Godin et al. (2015) Fréderic Godin, Baptist Vandersmissen, Wesley De Neve, and Rik Van de Walle. 2015.  Multimedia lab@ acl wnut ner shared task: Named entity recognition for twitter microposts using distributed word representations.  In _Proceedings of the workshop on noisy user-generated text_ , pages 146–153.
  * Han et al. (2018a) Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. 2018a.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In _NeurIPS_ , pages 8535–8545.
  * Han et al. (2018b) Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018b.  FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4803–4809. Association for Computational Linguistics.
  * Hedderich and Klakow (2018) Michael A. Hedderich and Dietrich Klakow. 2018.  Training a neural network in a low-resource setting on automatically annotated noisy data.  In _Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP_ , pages 12–18, Melbourne. Association for Computational Linguistics.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional lstm-crf models for sequence tagging.  _arXiv preprint arXiv:1508.01991_.
  * Jiang et al. (2018) Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. 2018.  Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.  In _ICML_.
  * Jie et al. (2019) Zhanming Jie, Pengjun Xie, Wei Lu, Ruixue Ding, and Linlin Li. 2019.  Better modeling of incomplete annotations for named entity recognition.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 729–734. ACL.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.  Neural architectures for named entity recognition.  _NAACL_ , pages 260–270.
  * Liang et al. (2020) Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang. 2020.  Bond: Bert-assisted open-domain named entity recognition with distant supervision.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1054–1064.
  * Loper and Bird (2002) Edward Loper and Steven Bird. 2002.  Nltk: The natural language toolkit.  In _Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics_ , pages 63–70.
  * Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016.  End-to-end sequence labeling via bi-directional lstm-cnns-crf.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_ , pages 1064–1074.
  * Mayhew et al. (2019) Stephen Mayhew, Snigdha Chaturvedi, Chen-Tse Tsai, and Dan Roth. 2019.  Named entity recognition with partially annotated training data.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning_ , pages 645–655.
  * Peng et al. (2019) Minlong Peng, Xiaoyu Xing, Qi Zhang, Jinlan Fu, and Xuan-Jing Huang. 2019.  Distantly supervised named entity recognition using positive-unlabeled learning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2409–2419.
  * Ratinov and Roth (2009) Lev Ratinov and Dan Roth. 2009.  Design challenges and misconceptions in named entity recognition.  In _Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009)_ , pages 147–155.
  * Robbins and Monro (1951) Herbert Robbins and Sutton Monro. 1951.  A stochastic approximation method.  _The annals of mathematical statistics_ , pages 400–407.
  * Sang and De Meulder (2003) Erik Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Sasada et al. (2016) Tetsuro Sasada, Shinsuke Mori, Tatsuya Kawahara, and Yoko Yamakata. 2016.  Named entity recognizer trainable from partially annotated data.  _Communications in Computer and Information Science_ , 593:148–160.
  * Shang et al. (2018) Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018\.  Learning named entity tagger using domain-specific dictionary.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2054–2064.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958.
  * Sutton and McCallum (2006) Charles Sutton and Andrew McCallum. 2006.  An introduction to conditional random fields for relational learning.  _Introduction to statistical relational learning_ , 2:93–128.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: a free collaborative knowledgebase.  _Communications of the ACM_ , 57(10):78–85.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yang et al. (2018) Yaosheng Yang, Wenliang Chen, Zhenghua Li, Zhengqiu He, and Min Zhang. 2018.  Distantly Supervised NER with Partial Annotation Learning and Reinforcement Learning.  _COLING_ , pages 2159–2169.

Dataset | Training | Dev | Test
---|---|---|---
#entity | #sent | #entity | #sent | #entity | #sent
English | 23,499 | 14,041 | 5,942 | 3,250 | 5,648 | 3,453
Spanish | 18,796 | 8,322 | 4,338 | 1,914 | 3,559 | 1,516
German | 11,851 | 12,152 | 4,833 | 2,867 | 3,673 | 3,005
Dutch | 13,344 | 15,806 | 2,616 | 2,895 | 3,941 | 5,195
CoNLL | - | 14,041 | - | 3,250 | - | 3,453
Tweet | - | 2,393 | - | 999 | - | 3,844
Webpage | - | 385 | - | 99 | - | 135
Wikigold | - | 1,142 | - | 280 | - | 274
Table 2: Statistics of datasets.

##  Appendix A Dataset Processing

###  A.1 Artificially Perturbed Dataset

The gold annotations of training data are perturbed by lowering the recall and
precision following Mayhew et al. (2019). Firstly, we randomly select an
entity from the whole entity set and tag all of its occurrences to ‘O’. We
repeat this operation until the recall decreases to 0.5. Then, we randomly tag
some tokens/spans to the entity label to decrease the precision to 0.9. The
detailed data statistics are shown in Table 2.

###  A.2 Distarntly Supervised Dataset

All distantly supervised datasets in our experiments are the same as those in
Liang et al. (2020). The distant labels are generated by external knowledge
bases (e.g. Wikidata Vrandečić and Krötzsch, 2014) and gazetteers collected
from multiple online resources. Specifically, the entity candidates are first
detected by POS tagger (NLTK Loper and Bird, 2002). Next, the ambiguous
candidates are filtered out by the Wikidata query service. Then, they match
the entities with words in multi-resources gazetteers to get their entity
types. Additional rules are used to get the entity labels of the unmatched
tokens. The detailed data statistics are shown in Table 2.

##  Appendix B Implementation Details

###  B.1 Model Structure and Implementation

For all the experiments with LSTM, we use the same word embeddings as Lample
et al. (2016). We use the character-level LSTM with hidden size 25 to produce
character-level word embeddings. The concatenation of the two embeddings are
fed into BiLSTM with hidden size 100. We also apply the dropout Srivastava et
al. (2014) between layers, with a rate of 0.5. The model is optimized using
Stochastic Gradient Descent Robbins and Monro (1951) with a learning rate of
0.01.

For experiments with BERT, we use the BERT-base Devlin et al. (2018) as our
encoder. The implementation is based on the codebase HuggingFace Transformers
Wolf et al. (2020). The dropout is set to 0.2. The model is optimized using
Adam Kingma and Ba (2014) with an initial learning rate of 3e-5.

###  B.2 Hyper-Parameters

There are two important hyper-parameters in our model as the positive noise
rate $\tau_{p}$ and the negative noise rate $\tau_{n}$. Based on our
observation, the initial noise rates are various in different datasets.
However, since our model has the ability to handle the noise, the noise rates
are relatively stable after the first iteration of self training. Therefore,
we empirically set $\tau_{p}$ and $\tau_{n}$ to 0.005 and 0.15 for all
experiments from the second iteration. For the first iteration, we report the
results of two strategies as follows:

Oracle. ‘Oracle’ means that we use the gold noise ratio (unavailable in real
settings) of positive noise rate $\tau_{p}$ and negative noise rate
$\tau_{n}$. The strategy is only applicable for artificially perturbed
datasets since the completed annotation is known.

Searched. ‘Searched’ means that we search the two hyper-parameters for best
performance on the development set. We search two parameters separately since
we assume $\tau_{n}$ and $\tau_{p}$ are independent. The search ranges from
0.0 to 0.2 with an interval of 0.01. We determine the two parameters with the
best development result on different datasets.

##  Appendix C Analysis of Self Training

Figure 5: Results of self-training.

The self training is borrowed from Jie et al. (2019) and not our main
contribution. However, to be self-contained, we also report the results of
self training in Figure 5. Our method (both local and global) outperforms the
baseline by a large margin at the first iteration, which indicates we have a
better base model of handling noise. Also, all curves raise in the first
several iterations and maintain stable relatively in the subsequent
iterations.


# Investigating Multi-Feature Selection and Ensembling for Audio
Classification

Muhammad Turab1 Teerath Kumar2,3 Malika Bendechache2,3,4 and Takfarinas
Saber4,5<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>1 Mehran University of
Engineering and Technology, Jamshoro, Pakistan. 2 ADAPT – Science Foundation
Ireland Research Centre 3 CRT AI, School of Computing, Dublin City University,
Dublin, Ireland 4 Lero – the Irish Software Research Centre 5 School of
Computer Science, National University of Ireland, Galway, Ireland

###### Abstract

Deep Learning (DL) algorithms have shown impressive performance in diverse
domains. Among them, audio has attracted many researchers over the last couple
of decades due to some interesting patterns–particularly in classification of
audio data. For better performance of audio classification, feature selection
and combination play a key role as they have the potential to make or break
the performance of any DL model. To investigate this role, we conduct an
extensive evaluation of the performance of several cutting-edge DL models
(i.e., Convolutional Neural Network, EfficientNet, MobileNet, Supper Vector
Machine and Multi-Perceptron) with various state-of-the-art audio features
(i.e., Mel Spectrogram, Mel Frequency Cepstral Coefficients, and Zero Crossing
Rate) either independently or as a combination (i.e., through ensembling) on
three different datasets (i.e., Free Spoken Digits Dataset, Audio Urdu Digits
Dataset, and Audio Gujarati Digits Dataset). Overall, results suggest feature
selection depends on both the dataset and the model. However, feature
combinations should be restricted to the only features that already achieve
good performances when used individually (i.e., mostly Mel Spectrogram, Mel
Frequency Cepstral Coefficients). Such feature combination/ensembling enabled
us to outperform the previous state-of-the-art results irrespective of our
choice of DL model.

###### keywords:

Audio Classification, Audio Features, Deep Learning, Ensembling, Feature
Selection.

##  1 Introduction

Audio data has been around us for a long time and is becoming an integral part
of several cutting-edge computing and multimedia applications in several
fields, e.g., security, healthcare monitoring, and context-aware services. The
success of such applications stands on their capability to effectively store
such data [4] and perform audio related tasks such as classifying or
retrieving audio files/signals (e.g., speech, music, environment sound/noise
and other audio signals) based on their sound properties/content [6].

While it has been, and it still is, a challenge for machines to accurately
perform such audio related tasks, we are continuously devising better content-
based classification and retrieval of audio databases to help machines perform
these tasks [6, 43]–some of which are emerging as commercial products (e.g.,
findsounds.com and midomi.com) or part of larger applications (e.g., Google
Hum to Search or voice recognition in virtual assistants).

Deep learning has been successful in audio classification [6] with a
tremendous amount of applications ranging from speech recognition [22], to
music classification[19], and environmental sound classification [25, 26].
While earlier works have previously attempted to train Neural Networks using
original audio data (e.g., raw audio signals, and standard low-level signal
parameters) [12], more recent works have since observed that they could
achieve significantly better performances by training the neural networks on
extracted features that are tailored to the audio data at hand [23].

There have been many studies on audio content analysis, using different
features and different methods [23, 37, 13]. Despite the significant gains
obtained by using extracted features, there is still a gap in terms of
efficiency, reliability, and accuracy as most of existing methods use a
single-modality along with the feature extraction.

Previous works demonstrated that the features fed to neural networks influence
significantly the accuracy of the classification results. For instance, in the
context of image classification, Wang et al. [42] have shown that combining
both spatial and spectral features improved greatly the classification
accuracy. In our work, we seek to do the same for audio data, i.e., we would
like to identify what combinations of features would enable different types of
neural network models to achieve the best accuracy in audio classification. We
particularly investigate the combination of three audio features (i.e., Mel
Spectrogram [2], Mel Frequency Cepstral Coefficients (MFCC [17]), and Zero
Crossing Rate (ZCR [7])), when used as ensembling with different deep learning
models (i.e., Convolutional Neural Network (CNN), EfficientNet and MobileNet)
on three benchmark speech classification datasets.

In this paper, we make the following contributions:

  * •

We explore different features and their ensembling for audio digit
classification.

  * •

We investigate the best combination of features through a wide range of
experiments using different models, on various datasets.

  * •

Our experiments suggest that our the proposed approach is effective in terms
of both time and accuracy.

  * •

Finally, we release our source code and trained models for the research
community to carry out the future research.

The rest of this paper is organised as follows. First, we present the context
of our work, and in particular, we describe the related work and background of
audio features (Section 2), then, we describe features ensembling approaches
(Section 3), next, present the design of our experiments (Section 4), present
evaluation (Section 5) and finally, we conclude this paper (Section 6).

##  2 Background and Related Work

Audio classification has been a focus of a large number of works [8, 35, 43,
26] each leveraging different features including Mel Spectrogram (MS [2]), Mel
Frequency Cepstral Coefficients (MFCC [17]) and Zero Crossing Rate (ZCR [7])
or a combination of any two features as an ensemble.

###  2.1 Mel Spectrogram (MS)

Audio signals are one dimensional, i.e., a time series of varying amplitudes.
Since neural networks require fix dimensional inputs, it is necessary to
convert/adapt audio signals into better formats which neural networks are able
to process efficiently. One such format could be obtained by transforming
audio signals into Mel Spectrogram [40, 2] which have the advantage of
providing the same information that the humans perceive. Spectrograms also
provide a visual understanding of audio signals. Furthermore, Mel Scale is
used to make the signal linear–matching with the human auditory system.
Mathematically, it is formulated as in Equation 1:

$m=2595\log_{10}\left(1+\frac{f}{700}\right)$ (1)

Where $m$ and $f$ represent Mel Spectrogram and frequency in Hz, respectively.
Previously, many researchers have used Mel Spectrogram as feature for
classification [35, 18, 24]. Sakashita and Aono [35] compute Mel Spectrogram
from different audio channels (i.e. Binaural, Mono, Harmonicpercussive source
separation). Then, they segment the spectrogram into different flavours, and
finally they train and ensemble many neural networks. McKinney and Breebaart
[18] use features that incorporate Low-Level Signal Properties, Mel-Frequency
Spectral Coefficients, and two other sets. Park et al. [24] apply three kinds
of Log Mel Spectrograms including time wrapping, a deformation of the time
series in the time direction, and the frequency masking. In their approach,
the authors proposed a simple data augmentation method for speech recognition
which is applied to listen, attend and spell networks for end-to-end speech
recognition tasks.

###  2.2 Mel Frequency Cepstral Coefficients (MFCC)

The MFCC feature has been popular due to compressed representation of the
signal [14, 17]. The computation of MFCC feature starts by segmenting audio
signals into frames before taking discrete Fourier Transform and logs of
amplitude spectrum. Then, it performs Mel scaling and smoothing. Next, it
takes a discrete cosine transform of the previous step to finally get the MFCC
features. A detailed description of the features is provided by Logan [14].
Like MS, many researchers used MFCC due to its compressed representation for
audio classification (e.g., [10, 26, 11]) in two ways: (i) extract MFCC then
train different neural networks [10, 26] or (ii) extract both MFCC and MS and
use them to train two networks that are later ensembled [11].

###  2.3 Zero Crossing Rate (ZCR)

ZCR measures how signals change from positive to negative via zero or vice-
versa [7]. It helps to distinguish between highly correlated and uncorrelated
features. Due to its correlation property, it is used by many researchers and
it shows massive gain in performance [15, 26]. Lu and Hankinson [15] use zero-
crossing rate (ZCR) and its various combinations for the automatic audio
indexing and retrieval systems. Whereas Piczak [26] provide a baseline
performance using MFCC and ZCR as features. Both of these features drastically
improved accuracy.

###  2.4 Feature Ensembling

Many works also tried ensemble features and networks [20, 11, 19, 25, 44].
Nanni et al. [19] first get three features including spectrograms, a
gammatonegram, and a rhythm from audio input, and segment them into different
windows, before training many SVM is trained and ensembling their predictions.
Moreover, Nanni et al. [19] leverage multiple additional features and perform
data augmentation to increase the data, then ensembled the multiple model
predictions. Similarly, Niranjan et al. [11] use two extra features (i.e.,
MFCC and MS) for ensembling of CNN which showed a massive performance gain on
the ESC50 dataset (i.e., a dataset for environmental sound classification).
Some studies have attempted using multiple features to train their deep
learning algorithms. Piczak [25] devised a convolutional neural network for
classifier training which combines two features (i.e., MFCC and its delta)
whereas Zhang et al. [44] extract three Mel Spectrogram features (i.e.,
static, delta, and delta) for their training.

So far, all the existing work in the literature with feature ensembling for
audio data are only proposing and describing their approach with a unique
configuration of features and model. Instead, in this paper, we explore
different combinations of features with a diverse set of models, on different
datasets. The goal of our work is to identify the best combination of features
and models in terms of type and number.

##  3 Ensembling Approach

In this section, we explain our investigated ensembling approach (publicly
available on github.com/turab45/multi-features-ensembler-for-audio-
classification). The approach starts by taking input audio sample $X_{input}$.
From $X_{input}$, we extract three different features namely, Mel Spectrogram
[2] $X_{mel_{s}pe}$, Mel Frequency Cepstral Coefficients (MFCC [17])
$X_{MFCC}$, Zero Crossing Rate (ZCR [7]) $X_{ZCR}$. After feature extraction,
we propose the approach as described below:

###  3.1 Multi-modality approach

In this approach, we first train each model for each feature like $M_{MS}$,
$M_{MFCC}$ and $M_{ZCR}$ for $X_{MS}$ $X_{MFCC}$ and $X_{ZCR}$, respectively.
Once the models have been trained, we save those models. During test, sample
$X_{test}$ is converted to three features and then passed to each model
accordingly. Probability value of each model are average as ensembler. Then we
predict the class for sample $X_{test}$, as shown in Figure 1.

Figure 1: The investigated multi-modality approach with three models

We describe the whole steps during test once models are trained:

  * •

Get probabilities:

    * –

$P_{MS}$ = $M_{MS}(X_{MS})$

    * –

$P_{MFCC}$ = $M_{MFCC}(X_{MFCC})$

    * –

$P_{ZCR}$ = $M_{ZCR}(X_{ZCR})$

  * •

We then explore combination of two feature probabilities first, then we
combined all these features probability, like
$Average=\frac{P_{MS}+P_{MFCC}+P_{ZCR}}{3}$

  * •

Finally, we predict the class or the label using argmax function:

    * –

$Predicted_{label}=argmax(Average)$

##  4 Experiment Design

In this section, we describe our experimental design in three parts: (i) the
data set on which we are basing our experiments, (ii) the algorithms we are
comparing against, and (iii) the setup of our system and the values defined
for the parameters of our algorithms.

###  4.1 Datasets

We used three different datasets for digit classification in English, Urdu and
Gujrati languages. We describe each of the datasets below.

####  4.1.1 Free Spoken Digits Dataset (FSDD [39])

This dataset is about spoken digits of English pronunciation (0 to 9) and
consists of 1500 recordings. These are recorded from 3 different speakers and
50 of each digit per speaker. Each recording is mono and sampled at 8kHz and
saved in _.wav_ format. It has 10 classes (0 to 9). All participants are male.
Information about audio duration is unknown, but we noticed that its minimum
duration length is 1 second and maximum is 2 second.

####  4.1.2 Audio Urdu Digits Dataset (AUDD [1])

This dataset is an audio spoken digits dataset for Urdu language. This dataset
has 25218 samples collected from 740 participants aged between 5 an 89 for
diversity purpose, but the majority of participants were 5 to 14 years old and
male participants were slightly more numerous than female participants. Each
sample is stereo and sampled at 48 $kHz$ and is mono-channel with a minimum
length of 1 second and maximum length of 2 seconds. Furthermore, the number of
samples in the dataset is almost balanced between all the classes (i.e., digit
per age).

####  4.1.3 Audio Gujarati Digits Dataset (AGDD [5])

This dataset is an audio digits dataset for Gujarati language which has 1940
samples sampled at 44.1 kHz. Recordings are obtained by 20 users, including 14
male and 6 female, from five different regions of Gujarat i.e. Central Zone,
North Zone, South Zone, Saurashtra, Kutch Region. Information about duration
of audio is unknown but we notice that minimum and maximum duration of audio
is 1 and 2 seconds, respectively. Each audio is saved in .wav format.

###  4.2 Algorithms

We used different deep learning models for generalization purpose. We used
3-layers CNN, EfficientNet [38] and mobileNet [9]. For a fair comparison, we
took the same model architectures that were defined in [1, 5]. For the CNN
model, we implemented a 3-layer architecture as described in [1] and shown in
Table 1.

Table 1: CNN Model, architecture taken from [1] Layer type | Dimensions | Other Details
---|---|---
Input | Input layer | $(32,32,1)$
CNN | $(30,30,64)$ |  kernel $3\times 3;$ stride 1 ;
|  | relu activation
Max Pool | $(15,15,64)$ | N/A
BN | $(15,15,64)$ | default value as
|  |  given in Keras [3]
CNN | $(13,13,64)$ |  kernel $3\times 3;$ stride 1;
|  | relu activation
Max Pool | $(6,6,64)$ | N/A
BN | $(6,6,64)$ | default value as
|  |  given in Keras [3]
CNN | $(4,4,64)$ |  kernel $3\times 3;$ stride 1;
|  | relu activation
Max Pool | $(2,2,64)$ | N/A
BN | $(2,2,64)$ | default value as
|  |  given in Keras [3]
Dropout | $(2,2,64)$ | dropout rate=0.1
Flatten | 256 | N/A
Fully Connected | 512 | N/A
Dropout | 512 | dropout rate=0.1
Fully Connected | 128 | N.A
Dropout | 512 | dropout rate=0.1
Fully Connected | 10 | softmax activation

###  4.3 Setup

Once features are extracted by using MS, MFCC, ZCR, we reshape the features to
$32\times 32\times 1$ as 2D images. Each dataset is randomly split into
training and test sets with an 8 to 2 ratio. Furthermore, the training set is
also split into training and validation sets with a 9 to 1 ratio.

We used hyperparameters as epoch 150, learning rate 0.01, batch size of 64 and
set the loss to categorical cross entropy as shown in Equation 2.

$L(\Theta)=-\sum_{i=1}^{k}y_{i}\log\left(\hat{y}_{i}\right)$ (2)

where $y_{i}$ is the ground truth, $\hat{y_{i}}$ is the predicted label, $k$
is the number of samples in the batch.

##  5 Evaluation

We evaluated the performance using accuracy metric (as shown in Equation 3)
and time required for testing (in millisecond).

$A=\frac{P}{T}$ (3)

where $A$, $P$ and $T$ represent accuracy, correct number of predicted samples
and total number of samples, respectively.
We performed each experiment three times and average accuracy is reported. In
Tables 2, 3 and 4, the values MS, MFCC and ZCR represent Mel Spectrogram, Mel
Frequency Cepstral Coefficients, and Zero Crossing Rate, respectively. In
braces, MS means the experiment is performed using a single feature and this
is the same for MFCC and ZCR. For two or three features in braces, the model
prediction is obtained using an ensemble of models trained using two or three
features. EfficientNetB0 to EfficientNetB7 are the different versions of
EfficientNet and similarly we used a single version of MobileNetV1.

Tables 2, 3 and 4, also report the average time taken for testing (in
milliseconds). We evaluated the testing time for each feature individually and
when combined as an ensemble, with each of the considered models and datasets.

Obtained results suggest that testing time is dependent of all dataset, model
and feature. For the same dataset, time for CNN with MS is lower than with
ZCR. For the same model, time of CNN with MFCC is larger than CNN with MS. For
the same feature, time of CNN with MS on AUDD is lower than CNN with MS on
FSSD. Otherwise, it is natural that when increasing the number of features,
time increases as well. So time for two and three features increases. Overall,
obtained results suggest that if we want to achieve a trade-off between
accuracy and speed, we should also select our features and models carefully.

For AUDD dataset [1], among single feature experiments, CNN with MS feature
achieves the highest performance, whereas the same CNN with the ZCR feature
achieves the worst performance. That is due to the loss of sound features
during ZCR extraction.

While we are not able to identify a rule on what number of features is ideal
during emsembling (i.e., either two or three features at a time), we see that
the features showing high performance independently lead to better performance
when combined in a feature ensembling. ZCR which achieved less performance
acts as a handicap as it drops the performance whenever it is combined with
MFCC or MS. MFCC and MS combination with CNN model showed the best performance
over previous SOTA performance with an absolute improvement of 3.00%, as shown
in Table 2.

Similarly for the FSSD dataset, we performed many experiments using diverse DL
models. First we explore single feature based performances. Then, we check
combinations of feature ensemblings. Unlike ZCR for AUDD, ZCR for FSSD a shows
better performance. Therefore, the combination ZCR with MFCC and MS using
either CNN or EfficientNet helps to improve the performance during ensembling
and shows superior performance with a 1% improvement to the previous SOTA
performance, as shown in Table 3.

Moreover, to check the effectiveness of the approach, we perform the
experiment on Audio Gujarati Digits Dataset (AGDD). In AGDD dataset, MS single
feature has been effective with CNN and similarly MFCC is also effective. When
these two features are used during ensembling, it further improved the
performance over SOTA performance with an absolute improvement of 0.2%.
However, these two individual features with EfficientNet and MobileNet have
shown worse performance than CNN. None of the different DL models using three
features ensembling were unable to improve performance due to the presence of
the ZCR feature which acted as a handicap–achieving worse performance compared
to MS and MFCC features.

Overall, MS and MFCC combination showed the best performance for audio digit
classificaiton, nevertheless model selection is very important. MS and MFCC
combination with EfficientNet and MobileNet unable to improve the performance.
To select the good features, we should also consider the model choice. ZCR
with MS and MFCC ensembling is only good choice of selection, whenever it
shows good performance individually as shown in Table 2. In the most cases, MS
and MFCC has shown the best performance and results suggest that we should use
those two features combination for ensembling.

Table 2: Different features accuracy using different models using AUDD [1] dataset Model Name | Accuracy | Time(ms)
---|---|---
Single feature
Support Vector Machine (MS) [1] | 0.65 $\pm$ 0.0 | –
Multilayer Perceptron (MS) [1] | 0.73 $\pm$ 0.02 | –
CNN (MS) [1] | 0.86 $\pm$ 0.02 | 0.264
EfficientNetB0 (MS) [1] | 0.84$\pm$ 0.05 | 1.60
EfficientNetB1 (MS) [1] | 0.82$\pm$ 0.02 | –
EfficientNetB2 (MS) [1] | 0.83$\pm$ 0.04 | –
EfficientNetB3 (MS) [1] | 0.84$\pm$ 0.06 | –
EfficientNetB4 (MS) [1] | 0.82$\pm$ 0.03 | –
EfficientNetB5 (MS)[1] | 0.84$\pm$ 0.04 | –
EfficientNetB6 (MS) [1] | 0.81$\pm$ 0.06 | –
EfficientNetB7 (MS) [1] | 0.56$\pm$ 0.07 | –
MobileNetV1 (MS) | 0.83$\pm$ 0.00 | 0.531
CNN (MFCC) | 0.85 $\pm$ 0.01 | 0.541
CNN (ZCR) | 0.40 $\pm$ 0.03 | 0.360
EfficientNetB0 (MFCC) | 0.81$\pm$ 0.03 | 1.124
EfficientNetB0 (ZCR) | 0.27 $\pm$ 0.02 | 1.681
MobileNetV1 (MFCC) | 0.80$\pm$ 0.02 | 0.628
MobileNetV1 (ZCR) | 0.36 $\pm$ 0.03 | 0.689
Two Features Ensembler
CNN (MS and MFCC) | 0.89 $\pm$ 0.01 | 0.896
CNN (MS and ZCR) | 0.82 $\pm$ 0.03 | 0.805
CNN (MFCC and ZCR) | 0.81 $\pm$ 0.01 | 0.631
EfficientNetB0 (MS and MFCC ) | 0.88$\pm$ 0.02 | 4.431
EfficientNetB0 (MS and ZCR) | 0.82$\pm$ 0.03 | 3.899
EfficientNetB0 (MFCC and ZCR) | 0.80$\pm$ 0.02 | 4.076
MobileNetV1 (MS and MFCC ) | 0.87$\pm$ 0.03 | 1.65
MobileNetV1 (MS and ZCR) | 0.66$\pm$ 0.02 | 1.70
MobileNetV1 (MFCC and ZCR) | 0.72$\pm$ 0.00 | 1.13
Three Features Ensembler
CNN (MS, MFCC and ZCR) | 0.87 $\pm$ 0.03 | 0.8167
EfficientNetB0 (MS, MFCC and ZCR) | 0.86$\pm$ 0.03 | 5.859
MobileNetV1 (MS, MFCC and ZCR) | 0.85$\pm$ 0.05 | 2.540
Table 3: Performance comparison multiple features using different models on FSSD [39] Model Name | Accuracy | Time(ms)
---|---|---
CNNDigitReco-speakerindependent [36] | 0.78 $\pm$ 0.0 | –
Single Feature
Support Vector Machine [41] | 0.90 $\pm$ 0.02 | –
Random Forest [41] | 0.96 $\pm$ 0.06 | –
English Digit Model[21] | 0.97 $\pm$ 0.09 | –
CNN (MS) [1] | 0.973 $\pm$ 0.01 | 0.3441
CNN (MFCC) | 0.978 $\pm$ 0.03 | 0.2895
CNN (ZCR) | 0.572 $\pm$ 0.08 | 0.306
EfficientNetB0 (MS) | 0.947 $\pm$ 0.05 | 1.154
EfficientNetB0 (MFCC) | 0.968 $\pm$ 0.07 | 0.740
EfficientNetB0 (ZCR) | 0.378 $\pm$ 0.06 | 1.148
MobileNetV1 (MS) | 0.877 $\pm$ 0.01 | 4.855
MobileNetV1 (MFCC) | 0.980 $\pm$ 0.02 | 3.367
MobileNetV1 (ZCR) | 0.538 $\pm$ 0.03 | 4.035
Two Feature Ensembler
CNN (MS and MFCC) | 0.987 $\pm$ 0.02 | 0.651
CNN (MS and ZCR) | 0.980 $\pm$ 0.03 | 0.630
CNN (MFCC and ZCR) | 0.977 $\pm$ 0.05 | 0.639
EfficientNetB0 (MS and MFCC) | 0.987 $\pm$ 0.07 | 2.811
EfficientNetB0 (MS and ZCR) | 0.957 $\pm$ 0.08 | 3.583
EfficientNetB0 (MFCC and ZCR) | 0.970 $\pm$ 0.01 | 2.838
MobileNetV1 (MS and MFCC) | 0.985 $\pm$ 0.05 | 4.9
MobileNetV1 (MS and ZCR) | 0.960 $\pm$ 0.06 | 1.682
MobileNetV1 (MFCC and ZCR) | 0.970 $\pm$ 0.09 | 2.308
Three Features Ensembler
CNN (MS, MFCC, ZCR) | 0.99 $\pm$ 0.03 | 0.921
EfficientNetB0 (MS, MFCC, ZCR) | 0.99 $\pm$ 0.04 | 4.977
MobileNetV1 (MS, MFCC, ZCR) | 0.987 $\pm$ 0.05 | 2.903
Table 4: Performance comparison multiple features using different models on Gujarati Digits dataset[5] Model Name | Accuracy | Time(ms)
---|---|---
Single Feature
Gujarati Digits Model[5] | 0.75 $\pm$ 0.02 | –
CNN (MS) [1] | 0.970 $\pm$ 0.03 | 1.770
CNN (MFCC) | 0.959 $\pm$ 0.05 | 1.761
CNN (ZCR) | 0.572 $\pm$ 0.07 | 1.750
EfficientNetB0 (MS) | 0.880 $\pm$ 0.02 | 9.402
EfficientNetB0 (MFCC) | 0.907 $\pm$ 0.01 | 9.781
EfficientNetB0 (ZCR) | 0.321 $\pm$ 0.04 | 9.536
MobileNetV1 (MS) | 0.856 $\pm$ 0.01 | 0.63
MobileNetV1 (MFCC) | 0.89 $\pm$ 0.05 | 0.791
MobileNetV1 (ZCR) | 0.557 $\pm$ 0.03 | 1.35
Two Features Ensembler
CNN (MS and MFCC) | 0.972 $\pm$ 0.04 | 2.996
CNN (MS and ZCR) | 0.936 $\pm$ 0.08 | 1.006
CNN (MFCC and ZCR) | 0.943 $\pm$ 0.07 | 1.723
EfficientNetB0 (MS and MFCC) | 0.970 $\pm$ 0.01 | 3.491
EfficientNetB0 (MS and ZCR) | 0.881 $\pm$ 0.06 | 3.338
EfficientNetB0 (MFCC and ZCR) | 0.94 $\pm$ 0.01 | 3.334
MobileNetV1 (MS and MFCC) | 0.933 $\pm$ 0.03 | 1.84
MobileNetV1 (MS and ZCR) | 0.80 $\pm$ 0.04 | 2.396
MobileNetV1 (MFCC and ZCR) | 0.884 $\pm$ 0.02 | 2.280
Three Features Ensembler
CNN (MS, MFCC and ZCR) | 0.966 $\pm$ 0.03 | 1.254
EfficientNetB0 (MS, MFCC and ZCR) | 0.948 $\pm$ 0.05 | 5.261
MobileNetV1 (MS, MFCC and ZCR) | 0.917 $\pm$ 0.03 | 3.520

We have also analyzed the stability of the models with a single feature at a
time and their behavior during validation on AUDD, FSSD, and AGDD
respectively, as shown in Figures 2.

Interestingly, all Figures 2 have the same pattern as CNN with the MS feature,
i.e., they are very stable. Furthermore, the analysis shows consistent
accuracy improvements from one epoch to another, except for EfficientNet with
ZCR which has shown a worse performance. Feature-wise, MS and MFCC are nearly
equally successful among all cases with the different models, whereas the use
of ZCR drops the performance of all models.

Overall, Figures 2 suggest that to achieve the best performance and stability,
the choice of model is important, whereas MS seems to be an excellent feature
extractor and the feature of choice.

In summary, we have shown that ensembling all features at the same time does
not guarantee achieving the best performance (in the contrary, it acts as a
handicap) and that beyond the selection of features, the choice of the model
is important. Therefore, it will be beneficial to design automatic and
interpretable ensembling techniques, potentially through reinforcement
learning techniques such as grammar-guided genetic programming [16, 34, 31,
30, 32]. Furthermore, while we have only focused on accuracy in this work, it
is possible that a feature/model ensembling does not achieve the best
accuracy, but performs better on other metrics. Therefore, it will be
beneficial to formulate the problem as a multi-objective feature selection
[29, 27, 28, 33].

---
(a) Urdu digits Dataset
(b) FSDD Dataset
(c) Gujarati Dataset
Figure 2: Validation Accuracy over Epochs on the Different Datasets

##  6 Conclusion

This paper investigates the use of multi-feature ensemblers, combining three
state-of-the-art audio features (i.e., Mel Spectrogram, Mel Frequency Cepstral
Coefficients, and Zero Crossing Rate) to alleviate the performance constrained
by the features when dealing with audio classification tasks.

In our work, we sought to explore different combinations of the three state-
of-the-arts features with a diverse set of models, on different datasets with
the goal of identifying the best combination of features and models. To check
the generalization of our results, we used three different audio datasets
including, Free Spoken Digits Dataset, Audio Urdu Digits Dataset and Audio
Gujarati Digits Dataset.

We trained our models with each feature individually, then with a combination
of two and three features. We evaluated the performance of each configuration,
i.e., model and feature(s) in terms of accuracy and testing time. Our thorough
experimental evaluation has shown that it is only better to combine features
that already perform well individually (i.e., mostly Mel Spectrogram, Mel
Frequency Cepstral Coefficients).

Our future research direction is in two folds: (i) to reduce testing time by
stacking multiple features and (ii) to explore these features from data
augmentation perspective to generate novel features.

##  7 Acknowledgment

This research was supported by Science Foundation Ireland (SFI) under grant
numbers 18/CRT/6223, 13/RC/2094$\\_$P2 (Lero SFI Centre for Software) and
13/RC/2106$\\_$P2 (ADAPT SFI Research Centre for AI-Driven Digital Content
Technology).

## References

  * [1] Aisha Aiman, Yao Shen, Malika Bendechache, Irum Inayat, and Teerath Kumar.  Audd: Audio urdu digits dataset for automatic audio urdu digit recognition.  Applied Sciences, 11(19):8842, 2021.
  * [2] David Warren Ball.  Field guide to spectroscopy, volume 8.  Spie Press Bellingham, Washington, 2006.
  * [3] F. & Others Chollet.  Keras.  2015\.
  * [4] Hilmi Egemen Ciritoglu, Takfarinas Saber, Teodora Sandra Buda, John Murphy, and Christina Thorpe.  Towards a better replica management for hadoop distributed file system.  In 2018 IEEE International Congress on Big Data (BigData Congress), pages 104–111. IEEE, 2018.
  * [5] Nikunj Dalsaniya, Sapan H Mankad, Sanjay Garg, and Dhuri Shrivastava.  Development of a novel database in gujarati language for spoken digits classification.  In International Symposium on Signal Processing and Intelligent Recognition Systems, pages 208–219. Springer, 2019.
  * [6] Liang Gao, Kele Xu, Huaimin Wang, and Yuxing Peng.  Multi-representation knowledge distillation for audio classification.  Multimedia Tools and Applications, pages 1–24, 2022.
  * [7] Theodoros Giannakopoulos and Aggelos Pikrakis.  Introduction to audio analysis: a MATLAB® approach.  Academic Press, 2014.
  * [8] Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin Wilson.  Cnn architectures for large-scale audio classification.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 131–135, 2017.
  * [9] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [10] Chadawan Ittichaichareon, Siwat Suksri, and Thaweesak Yingthawornsuk.  Speech recognition using mfcc.  In International conference on computer graphics, simulation and modeling, volume 9, 2012.
  * [11] Niranjan K, Shankar Kumar S, and Vedanth S.  Ensemble and multi model approach to environmental sound classification.  In 2021 Fourth International Conference on Electrical, Computer and Communication Technologies (ICECCT), pages 1–5, 2021.
  * [12] Jongpil Lee, Jiyoung Park, Keunhyoung Luke Kim, and Juhan Nam.  Sample-level deep convolutional neural networks for music auto-tagging using raw waveforms.  arXiv preprint arXiv:1703.01789, 2017.
  * [13] Xinyu Li, Venkata Chebiyyam, and Katrin Kirchhoff.  Multi-stream network with temporal attention for environmental sound classification.  arXiv preprint arXiv:1901.08608, 2019.
  * [14] Beth Logan.  Mel frequency cepstral coefficients for music modeling.  In In International Symposium on Music Information Retrieval. Citeseer, 2000.
  * [15] Guojun Lu and Templar Hankinson.  An investigation of automatic audio classification and segmentation.  In WCC 2000-ICSP 2000. 2000 5th International Conference on Signal Processing Proceedings. 16th World Computer Congress 2000, volume 2, pages 776–781. IEEE, 2000.
  * [16] David Lynch, Takfarinas Saber, Stepán Kucera, Holger Claussen, and Michael O’Neill.  Evolutionary learning of link allocation algorithms for 5g heterogeneous wireless communications networks.  In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1258–1265, 2019.
  * [17] Sayf A Majeed, Hafizah Husain, Salina Abdul Samad, and Tariq F Idbeaa.  Mel frequency cepstral coefficients (mfcc) feature extraction enhancement in the application of speech recognition: a comparison study.  Journal of Theoretical and Applied Information Technology, 79(1):38, 2015.
  * [18] Martin McKinney and Jeroen Breebaart.  Features for audio and music classification.  2003\.
  * [19] Loris Nanni, Yandre MG Costa, Diego Rafael Lucio, Carlos Nascimento Silla Jr, and Sheryl Brahnam.  Combining visual and acoustic features for audio classification tasks.  Pattern Recognition Letters, 88:49–56, 2017.
  * [20] Loris Nanni, Gianluca Maguolo, Sheryl Brahnam, and Michelangelo Paci.  An ensemble of convolutional neural networks for audio classification.  Applied Sciences, 11(13):5796, 2021.
  * [21] Seham Nasr, Muhannad Quwaider, and Rizwan Qureshi.  Text-independent speaker recognition using deep neural networks.  In 2021 International Conference on Information Technology (ICIT), pages 517–521. IEEE, 2021.
  * [22] Jayashree Padmanabhan and Melvin Jose Johnson Premkumar.  Machine learning in automatic speech recognition: A survey.  IETE Technical Review, 32(4):240–251, 2015.
  * [23] Kamalesh Palanisamy, Dipika Singhania, and Angela Yao.  Rethinking cnn models for audio classification.  arXiv preprint arXiv:2007.11154, 2020.
  * [24] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le.  Specaugment: A simple data augmentation method for automatic speech recognition.  arXiv preprint arXiv:1904.08779, 2019.
  * [25] Karol J Piczak.  Environmental sound classification with convolutional neural networks.  In 2015 IEEE 25th international workshop on machine learning for signal processing (MLSP), pages 1–6. IEEE, 2015.
  * [26] Karol J Piczak.  Esc: Dataset for environmental sound classification.  In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015–1018, 2015.
  * [27] Takfarinas Saber, Malika Bendechache, and Anthony Ventresque.  Incorporating user preferences in multi-objective feature selection in software product lines using multi-criteria decision analysis.  In International Conference on Optimization and Learning, pages 361–373. Springer, 2021.
  * [28] Takfarinas Saber, David Brevet, Goetz Botterweck, and Anthony Ventresque.  Is seeding a good strategy in multi-objective feature selection when feature models evolve?  Information and Software Technology, 95:266–280, 2018.
  * [29] Takfarinas Saber, David Brevet, Goetz Botterweck, and Anthony Ventresque.  Reparation in evolutionary algorithms for multi-objective feature selection in large software product lines.  SN Computer Science, 2(3):1–14, 2021.
  * [30] Takfarinas Saber, David Fagan, David Lynch, Stepan Kucera, Holger Claussen, and Michael O’Neill.  A hierarchical approach to grammar-guided genetic programming the case of scheduling in heterogeneous networks.  In TPNC, pages 118–134, 2018.
  * [31] Takfarinas Saber, David Fagan, David Lynch, Stepan Kucera, Holger Claussen, and Michael O’Neill.  Hierarchical grammar-guided genetic programming techniques for scheduling in heterogeneous networks.  In CEC, 2020.
  * [32] Takfarinas Saber, David Fagan, David Lynch, Stepan Kucera, Holger Claussen, and Michael O’Neill.  A multi-level grammar approach to grammar-guided genetic programming: the case of scheduling in heterogeneous networks.  GPEM, pages 1–39, 2019.
  * [33] Takfarinas Saber, James Thorburn, Liam Murphy, and Anthony Ventresque.  VM reassignment in hybrid clouds for large decentralised companies: A multi-objective challenge.  Future Generation Computer Systems, 79:751–764, 2018.
  * [34] Takfarinas Saber and Shen Wang.  Evolving better rerouting surrogate travel costs with grammar-guided genetic programming.  In IEEE CEC, pages 1–8, 2020.
  * [35] Yuma Sakashita and Masaki Aono.  Acoustic scene classification by ensemble of spectrograms based on adaptive temporal divisions.  Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge, 2018.
  * [36] Oscar Saz.  CNNDigitReco-speakerindependent.  https://www.kaggle.com/saztorralba/cnndigitreco-speakerindependent, 2020\.
  * [37] Alexander Schindler, Thomas Lidy, and Andreas Rauber.  Multi-temporal resolution convolutional neural networks for acoustic scene classification.  arXiv preprint arXiv:1811.04419, 2018.
  * [38] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [39] Zohar Jackson; Cesar Souza; Jason Flaks; Yuxin Pan; Hereman Nicolas; Adhish Thite.  A free audio dataset of spoken digits. think mnist for audio.  https://doi.org/10.5281/zenodo.1342401, 2018.  [Online; accessed 1-March-2022].
  * [40] BZJLS Thornton.  Audio recognition using mel spectrograms and convolution neural networks.  2019\.
  * [41] Inam Ur Rehman.  Classification on fsdd using spectograms.  https://www.kaggle.com/iinaam/classification-on-fsdd-using-spectograms, 2021\.
  * [42] Qi Wang, Xiang He, and Xuelong Li.  Locality and structure regularized low rank representation for hyperspectral image classification.  IEEE Transactions on Geoscience and Remote Sensing, 57(2):911–923, 2018.
  * [43] Yong Xu, Qiuqiang Kong, Wenwu Wang, and Mark D Plumbley.  Large-scale weakly supervised audio classification using gated convolutional neural network.  In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 121–125. IEEE, 2018.
  * [44] Shiqing Zhang, Shiliang Zhang, Tiejun Huang, and Wen Gao.  Speech emotion recognition using deep convolutional neural network and discriminant temporal pyramid matching.  IEEE Transactions on Multimedia, 20(6):1576–1590, 2017.

## Authors

|  Muhammad Turab is an undergraduate student at MUET, Jamshoro. He has been
working in the field of Deep Learning and Machine learning for two years. He
has completed more than 70 projects on GitHub. His research interests include
deep learning, computer vision and data augmentation for medical imaging.
---|---
|  Teerath Kumar received his Bachelor’s degree in Computer Science with
distinction from National University of Computer and Emerging Science (NUCES),
Islamabad, Pakistan, in 2018. Currently, he is pursuing PhD from Dublin City
University, Ireland. His research interests include advanced data
augmentation, deep learning for medical imaging, generative adversarial
networks and semi-supervised learning.
|  Malika Bendechache is a Lecturer/Assistant Professor in the School of
Computing at Dublin City University, Ireland, and a Funded Investigator at
both ADAPT and Lero Science Foundation Ireland research centres. Malika holds
a PhD in Computer Science from University College Dublin, Ireland. Her
research interests span the areas of Big data Analytics, Machine Learning,
Data Governance, Parallel and Distributed Systems, Cloud/Edge/Fog Computing,
Blockchain, Security, and Privacy.
|  Takfarinas Saber is a Lecturer/Assistant Professor in the School of
Computer Science at National University of Ireland, Galway, Ireland, and a
Funded-Investigator in Lero, the Science Foundation Ireland Research Centre
for Software. Takfarinas holds a PhD in Computer Science from University
College Dublin, Ireland. His area of expertise is in the optimisation of
Complex Software Systems such as Cloud Computing, Smart Cities, Distributed
Systems, and Wireless Communication Networks.


$\rm{D^{0}}$ cross section in pp collisions at $\sqrt{s}=7$ TeV, measured with
the ALICE experiment

Xianbao Yuan, for the ALICE Collaboration
Institute of Particle Physics, Central China Normal University
University and INFN, Padova, Italy
<EMAIL_ADDRESS>or<EMAIL_ADDRESS>

The measurement of the cross-section for charm production in pp collisions at
the LHC is not only a fundamental reference to investigate medium properties
in heavy-ion collisions, but also a key test of pQCD predictions in a new
energy domain.

The ALICE [1] experiment has measured the D meson production in pp collisions
at $\sqrt{s}=7$ TeV. We present the analysis procedure for $\rm D^{0}\to\rm
K^{-}\rm\pi^{+}$ and for the calculation of efficiency and acceptance
corrections. Finally, we show the preliminary results on $\rm D^{0}$ cross
section in pp collisions at $\sqrt{s}=7$ TeV, measured in the region $2<p_{\rm
t}<12$ GeV/$c$ at central rapidity $|y|<0.5$. These results are compared to
perturbative QCD predictions.

The analysis is based on an invariant mass analysis of opposite-charge pairs
of reconstructed tracks that can represent a $\rm D^{0}$ with a displaced
vertex (the mean proper decay length of the $\rm D^{0}$ is $c\tau\approx
123~{}\mu$m). The selection is based on topological cuts and particle
identification via specific energy deposit and time-of-flight measurement. The
cross section is calculated from the raw signal yield extracted with the
invariant mass analysis, $N^{\rm D^{0}~{}raw}(p_{\rm t})$, using the following
formula:

$\left.\frac{{\rm d}\sigma^{\rm D^{0}}}{{\rm d}p_{\rm
t}}\right|_{|y|<0.5}=\frac{1}{2}\frac{1}{2\,y_{\rm acc}\Delta p_{\rm
t}}\frac{\left.f_{prompt}\cdot N^{\rm D^{0}~{}raw}(p_{\rm
t})\right|_{|y|<y_{\rm acc}}}{\epsilon_{prompt}\cdot{\rm
BR}\cdot\mathit{L}_{int}}\cdot$ (1)

Here, $\epsilon_{prompt}$ means the efficiency of prompt mesons, which
accounts for selection cuts, for track and primary vertex reconstruction
efficiency, and for detector acceptance. The $f_{prompt}$ is the prompt
fraction of raw yield.

Figure 1 (Left) shows the invariant mass distribution for $p_{\rm t}>2$
GeV/$c$ after applying the cuts, which corresponds to $1.1\times 10^{8}$
minimum bias events collected by ALICE in 2010 at $\sqrt{s}=7$ TeV. Figure 1
(Right) shows the efficiencies for $\rm D^{0}\to\rm K^{-}\rm\pi^{+}$ with all
the decay particles in the acceptance $\left|\eta\right|<0.9$. The
efficiencies increase and flatten at about 0.1 at $p_{\rm t}>2$ GeV/$c$. The
efficiency without particle identification selection, shown for comparison, is
the same as with particle identification for $p_{\rm t}>2$ GeV/$c$, indicating
that this selection is essentially fully efficient for the signal. The
efficiencies for $\rm D^{0}$ meson from B meson decay, also shown for
comparison, are larger by a factor about 2, because this feed-down component
is more displaced from the primary vertex, due to the longer B life time. The
$10-15\%$ feed-down from B decays is subtracted based on pQCD prediction [2].

Several sources of systematic uncertainties were considered, namely those
affecting the signal extraction from the invariant mass spectra and all the
correction factors applied to obtain the $p_{\rm t}$-differential cross
sections. A summary of the estimated relative systematic errors is shown in
Fig 2 (Left).

The $p_{\rm t}$-differential cross section for prompt $\rm D^{0}$, obtained
from the yields extracted by fitting the invariant mass spectra and corrected
for efficiency and B feed-down, is shown in Fig 2 (Right). The error bars
represent the statistical errors, while the systematic errors are plotted as
rectangle areas around the data points. The measured $\rm D^{0}$ meson
production cross section is compared to two theoretical predictions, namely
FONLL [2] and GM-VFNS [3]. Our measurement of $\rm D^{0}$ at $\sqrt{s}=7$ TeV
is reproduced by both models within their theoretical uncertainties.

Figure 1: Left: $p_{\rm t}>$2 GeV/$c$ invariant mass distribution. Right:
efficiencies for $\rm D^{0}$ as a function of $p_{\rm t}$.

## References

  * [1] B. Abelev, A. Abrahantes Quintana, $\mathit{et~{}al}$., [ALICE Coll.], __Arxiv: hep-ex/1111.1553v1.
  * [2] M. Cacciari $\mathit{et~{}al}$., __JHEP 0407 (2004) 033; Private communication.
  * [3] B.A. Kniehl $\mathit{et~{}al}$., __Phys. Rev. D77 (2008) 014011; Private communication.

Figure 2: Left: systematic errors summary plot. Right: $p_{\rm
t}$-differential cross section for prompt $\rm D^{0}$ in pp collisions at
$\sqrt{s}=7$ TeV compared with FONLL [2] and GM-VFNS [3] theoretical
predictions.