# On $G(A)_{\mathbb{Q}}$ of rings of finite representation type

Tony J. Puthenpurakal  Department of Mathematics, IIT Bombay, Powai, Mumbai
400 076<EMAIL_ADDRESS>
###### Abstract.

Let $(A,\mathfrak{m})$ be an excellent Henselian Cohen-Macaulay local ring of
finite representation type. If the AR-quiver of $A$ is known then by a result
of Auslander and Reiten one can explicity compute $G(A)$ the Grothendieck
group of finitely generated $A$-modules. If the AR-quiver is not known then in
this paper we give estimates of
$G(A)_{\mathbb{Q}}=G(A)\otimes_{\mathbb{Z}}\mathbb{Q}$ when $k=A/\mathfrak{m}$
is perfect. As an application we prove that if $A$ is an excellent equi-
characteristic Henselian Gornstein local ring of positive even dimension with
$\operatorname{char}A/\mathfrak{m}\neq 2,3,5$ (and $A/\mathfrak{m}$ perfect)
then $G(A)_{\mathbb{Q}}\cong\mathbb{Q}$.

###### Key words and phrases:

Grothendieck group, finite representation type, AR sequence

###### 1991 Mathematics Subject Classification:

Primary 13D15; Secondary 16G50, 16G60, 16G70

##  1\. introduction

Let $(A,\mathfrak{m})$ be a Henselian Noetherian local ring. Then it is well-
known that the category of finitely generated $A$-modules satisfy the Krull-
Schmidt property, i.e., every finitely generated $A$-module is uniquely a
direct sum of indecomposable $A$-modules (with local endomorphism rings). Now
assume that $A$ is Cohen-Macaulay. Then we say $A$ is of finite (Cohen-
Macaulay) representation type if $A$ has only finitely many indecomposable
maximal Cohen-Macaulay $A$-modules upto isomorphism. To study (not necessarily
commutative) Artin algebra’s Auslander and Reiten introduced the theory of
almost-split sequences. These are now called AR-sequences. Later Auslander and
Reiten extended the theory of AR-sequences to the case of commutative
Henselian isolated singularities. Good references for this topic are [Y] and
[LW]. Let $\operatorname{CM}(A)$ denote the full subcategory of maximal Cohen-
Macaulay (= MCM) $A$-modules.

######  Remark 1.1.

Note we can define Grothendieck group of any extension closed subcategory
$\mathcal{S}$ of $\operatorname{mod}(A)$ the category of all finitely
generated $A$-modules, we denote it by $G(\mathcal{S})$. By [Y, 13.2] the
natural map $G(\operatorname{CM}(A))\rightarrow G(\operatorname{mod}(A)$ is an
isomorphism. Throughout this section we work with $G(\operatorname{CM}(A))$
and by abuse of notation denote it by $G(A)$.

Let $A$ be a Henselian Cohen-Macaulay local of finite representation type. Set
$\mathcal{I}_{A}$ to be the set of all indecomposable MCM $A$-modules upto
isomorphism. If $W$ is a subset of $\mathcal{I}_{A}$ then set
$\operatorname{add}(W)$ be the set consisting of finite direct sums of
elements of $W$. Also let $\mathcal{AR}(A)$ denote the set of all AR-sequences
in $A$ upto isomorphism. Let $F(A)$ be the free abelian group generated on
$\operatorname{add}(\mathcal{I}_{A})$. Let $\mathcal{AR}_{0}(A)$ be the
subgroup of $F(A)$ generated by

$\\{X_{1}-X_{2}+X_{3}\mid\text{ there is a sequence}\ 0\rightarrow
X_{1}\rightarrow X_{2}\rightarrow X_{3}\rightarrow 0\ \text{in}\
\mathcal{AR}(A)\\}.$

By a result due to Auslander-Reiten [AR, 2.2] (also see [Y, 13.7]) we have
$G(A)=F(A)/\mathcal{AR}_{0}(A)$.

Computing AR-sequences is usually a tedious task and usually we assume $A$ is
equi-characteristic, complete with algebraically closed residue field. The
main objective of this paper is to give estimates of rank of $G(A)$ when the
residue field is perfect but not necessarily algebraically closed.

In our introduction let us assume $(A,\mathfrak{m})$ is an excellent Henselian
Cohen-Macaulay local of finite representation type and containing a field
isomorphic to $k=A/\mathfrak{m}$ which we also denote by $k$. (In our proofs
we will deal with a more general case). We assume $k$ is perfect. Let
$\overline{k}$ be the algebraic closure of $k$. Let

$\mathcal{C}_{k}=\\{E\mid E\ \text{is a finite extension of }\ k,\ \text{and}\
E\subseteq\overline{k}\\}.$

For each $E$ in $\mathcal{C}_{k}$ set $A^{E}=A\otimes_{k}E$. We have an
obvious directed system of rings $\\{A^{E}\\}_{E\in\mathcal{C}_{k}}$. Set
$T=\lim_{E\in\mathcal{C}_{k}}A^{E}=A\otimes_{k}\overline{k}$. Then $A^{E},T$
are excellent Henselian Cohen-Macaulay local of finite representation type. So
$\widehat{T}$ the completion of $T$ is Cohen-Macaulay of finite representation
type. It is not difficult to show $G(T)\cong G(\widehat{T})$. For $k\subseteq
F\subseteq E$, where $E,F\in\mathcal{C}_{k}$, we have an obvious map
$\eta^{E}_{F}\colon G(A^{F})\rightarrow G(A^{E})$ given by $M\rightarrow
M\otimes_{A^{F}}A^{E}$. It is clear that we have a direct system of abelian
groups $\\{G(A^{E})\\}_{E\in\mathcal{C}_{k}}$. So we have an abelian group
$\lim_{E\in\mathcal{C}_{k}}G(A^{E})$ and natural maps $\eta_{E}\colon
G(A^{E})\rightarrow\lim_{E\in\mathcal{C}_{k}}G(A^{E})$. Let
$E\in\mathcal{C}_{k}$. As $T$ is a flat $A^{E}$-algebra we have an obvious map
$\xi_{E}\colon G(A^{E})\rightarrow G(T)$ given by $M\rightarrow
M\otimes_{A^{E}}T$. The maps $\xi_{E}$ are compatible with $\eta^{E}_{F}$
whenever $k\subseteq F\subseteq E$. So we have a natural map

$\xi\colon\lim_{E\in\mathcal{C}_{k}}G(A^{E})\rightarrow G(T).$

Our main result is

######  Theorem 1.2.

$\xi$ is an isomorphism.

Theorem 1.2 does not give us any estimates on $G(A)$. If $H$ is an abelian
group then we set $H_{\mathbb{Q}}=H\otimes_{\mathbb{Z}}\mathbb{Q}$ and if
$f\colon H\rightarrow L$ is a homomorphism of abelian groups then we set
$f_{\mathbb{Q}}$ to be map from $H_{\mathbb{Q}}\rightarrow L_{\mathbb{Q}}$
induced by $f$.

It is well-known that direct limits commutes with tensor products, see [Mat,
Theorem A1, p. 270]. So we have an isomorphism

$\xi_{\mathbb{Q}}\colon\lim_{E\in\mathcal{C}_{k}}G(A^{E})_{\mathbb{Q}}\rightarrow
G(T)_{\mathbb{Q}}.$

Our next result essentially an observation.

######  Proposition 1.3.

Let $F\in\mathcal{C}_{k}$ be any. Then the map

$(\eta_{F})_{\mathbb{Q}}\colon
G(A^{F})_{\mathbb{Q}}\rightarrow\lim_{E\in\mathcal{C}_{k}}G(A^{E})_{\mathbb{Q}}$

is an injection. In particular we have an injection from $G(A)_{\mathbb{Q}}$
to $G(T)_{\mathbb{Q}}$.

Complete equi-characteristic Gorenstein local rings of finite representation
type (with $\operatorname{char}A/\mathfrak{m}\neq 2,3,5$) are precisely the
ADE-singularities, see [LW, 9.8]. Furthermore in this case their AR-quiver is
known and so their Grothendieck groups have been computed, see [Y, 13.10]. As
an easy consequence to our results we show that

######  Corollary 1.4.

Let $(A,\mathfrak{m})$ be an excellent equi-characteristic Henselian
Gorenstein local ring of finite representation type. Assume $k$ is perfect.
Then

  1. (1)

If $\dim A$ is positive and even then $G(A)_{\mathbb{Q}}\cong\mathbb{Q}$.

  2. (2)

If $\dim A$ is odd then $\dim_{\mathbb{Q}}G(A)_{\mathbb{Q}}\leq 3$.

In fact in (1) we have $G(A)_{\mathbb{Q}}\cong G(\widehat{T})_{\mathbb{Q}}$.
In section five we give an example which shows that in (2);
$G(A)_{\mathbb{Q}}$ can be a proper subspace of $G(\widehat{T})_{\mathbb{Q}}$.
The same example shows that we cannot in general compare torsion of $G(A)$
with torsion of $G(\widehat{T})$.

We now describe in brief the contents of this paper. In section two we discuss
some preliminaries on AR-sequences that we need. In section three we describe
our construction. In section four we prove Theorem 1.2, Proposition 1.3 and
Corollary 1.4. Finally in section five we discuss an example which shows that
torsion does not behave well with our construction.

_Convention_ : Throughout this paper all rings are commutative Noetherian and
all modules (unless stated otherwise) are finitely generated.

##  2\. Some preliminaries on Auslander-Reiten sequences

In this section we discuss some preliminaries on Auslander-Reiten (AR)
sequences that we need. The reference for this section is [Y, Chapter 2]. Let
$(A,\mathfrak{m})$ be a Henselian Cohen-Macaulay local ring.

######  2.1.

Let $M\in\operatorname{CM}(A)$ be indecomposable. We define a set of short
exact sequences in $\operatorname{CM}(A)$ as follows:

$\mathcal{S}(M)=\\{s\colon 0\rightarrow N_{s}\rightarrow E_{s}\rightarrow
M\rightarrow 0\mid N_{s}\ \text{is indecomposable and}\ s\ \text{is non-
split}\\}.$

If $M$ is non-free then $\mathcal{S}(M)$ is non-empty, [Y, 2.2]. Define a
partial order $>$ on $\mathcal{S}(M)$ as follows. Let $s,t\in\mathcal{S}(M)$.
Then we say $s>t$ if there is $f\in\operatorname{Hom}_{A}(N_{s},N_{t})$ such
that $\operatorname{Ext}^{1}_{A}(M,f)(s)=t$. This is equivalent to the
existence of a commutative diagram:

$\textstyle{0\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{N_{s}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{f}$$\textstyle{E_{s}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{M\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{j}$$\textstyle{0}$$\textstyle{0\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{N_{t}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{E_{t}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{M\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{0\
}$

where $j$ is the identity map. We write $s\sim t$ if $f$ is an isomorphism.


[datatype=bibtex, overwrite=true] [fieldsource=booktitle,
match=.*Interspeech.*, replace=Proc. Interspeech] [fieldsource=journal,
match=.*INTERSPEECH.*, replace=Proc. Interspeech] [fieldsource=booktitle,
match=.*ICASSP.*, replace=ICASSP] [fieldsource=booktitle,
match=.*icassp_inpress.*, replace=ICASSP (in press)] [fieldsource=booktitle,
match=.*International.*Conference.*on.*Acoustics,.*Speech.*and.*Signal.*Processing.*,
replace=ICASSP] [fieldsource=booktitle,
match=.*International.*Conference.*on.*Learning.*Representations.*,
replace=ICLR] [fieldsource=booktitle,
match=.*International.*Conference.*on.*Machine.*Learning.*, replace=ICML]
[fieldsource=booktitle,
match=.*Automatic.*Speech.*Recognition.*and.*Understanding.*, replace=Proc.
ASRU] [fieldsource=booktitle, match=.*Spoken.*Language.*Technology.*,
replace=Proc. SLT] [fieldsource=booktitle,
match=.*Speech.*Synthesis.*Workshop.*, replace=Proc. SSW]
[fieldsource=booktitle, match=.*workshop.*on.*speech.*synthesis.*,
replace=Proc. SSW] [fieldsource=booktitle,
match=.*Advances.*in.*neural.*information.*processing.*, replace=Proc. NIPS]
[fieldsource=booktitle,
match=.*Advances.*in.*Neural.*Information.*Processing.*, replace=Proc. NIPS]
[fieldsource=booktitle, match=.*Workshop.*on.* Applications.* of.*
Signal.*Processing.*to.*Audio.*and.*Acoustics.*, replace=Proc. WASPAA]
[fieldsource=booktitle,
match=.*International.*Conference.*on.*Language.*Resources.*and.*Evaluation.*,
replace=Proc. LREC] [fieldsource=journal,
match=.*Spontaneous.*Speech.*Processing.*and.*Recognition, replace=Proc. SSPR]
[fieldsource=publisher, match=.+, replace=] [fieldsource=month, match=.+,
replace=] [fieldsource=location, match=.+, replace=] [fieldsource=address,
match=.+, replace=] [fieldsource=organization, match=.+, replace=]
[fieldsource=doi, match=.+, replace=] [fieldsource=url, match=.+, replace=]
[fieldsource=editor, match=.+, replace=] 1The University of Tokyo, Japan,
2Technical University of Munich, Germany,
3Tokyo Metropolitan University, Japan, 4Carnegie Mellon University, USA

# JTubeSpeech: corpus of Japanese speech collected from YouTube
for speech recognition and speaker verification

###### Abstract

In this paper, we construct a new Japanese speech corpus called “JTubeSpeech.”
Although recent end-to-end learning requires large-size speech corpora, open-
sourced such corpora for languages other than English have not yet been
established. In this paper, we describe the construction of a corpus from
YouTube videos and subtitles for speech recognition and speaker verification.
Our method can automatically filter the videos and subtitles with almost no
language-dependent processes. We consistently employ Connectionist Temporal
Classification (CTC)-based techniques for automatic speech recognition (ASR)
and a speaker variation-based method for automatic speaker verification (ASV).
We build 1) a large-scale Japanese ASR benchmark with more than 1,300 hours of
data and 2) 900 hours of data for Japanese ASV.

Index Terms—  automatic speech recognition, automatic speaker verification,
speech corpus, YouTube

##  1 Introduction

Powered by the development of deep learning, significant progress has been
made on various speech recognition tasks, e.g., automatic speech recognition
(ASR) [3, 4, 5] and automatic speaker verification (ASV) [6, 7]. Due to data
hungriness of deep learning, massive-size speech corpora have been constructed
and published. It is desirable to build and publish speech corpora of all
languages for decentralizing the speech technologies. However, corpora in
languages other than English are very poor. For example, while several
thousands of hours of corpora have been published in English and Chinese [8,
9, 10, 11, 12, 13, 14], similar-size corpora are very limited for other
languages.

Japanese, the target language in this paper, is also this example. The CSJ
corpus [15] is the most frequently used corpus for Japanese ASR, but its size
is relatively small ($600$ hours) compared to modern English corpora such as
Common Voice [10] ($1,100$ hours). Also, the modern size of ASV corpora is
more than $1,000$ hours [16, 14], but there is no open-sourced corpus for
Japanese ASV.

To construct a large-scale corpus, several studies have collected text-audio
pairs from videos [17, 16, 11, 18]. For example, YouTube provides videos in
diverse genres, recording environments, speakers, and language accents. There
is no doubt that such in-the-wild data is useful for wide purpose of modern
speech technology. Chen [19] proposed a strategy to collect English videos for
ASR use. Also, Fan [14] manually selected Chinese celebrities and extracted
videos for ASV use. Unlike these methods that require language-dependent and
manual processes, this paper aims to develop a corpus with almost no language-
dependent and no manual processes. The establishment of this method will be
useful for building corpora of many languages, not limited to English and
Chinese.

In this paper, we propose a mostly language-independent strategy to construct
a speech corpus for both ASR and ASV, and then construct a Japanese speech
corpus “JTubeSpeech” by using the proposed method. We first crawl YouTube in
order to generate candidate audio-text pair data. For ASR, the subtitles are
aligned to the audio using a CTC-based ASR model using CTC segmentation [20].
This method calculates a confidence score to filter the audio-text pair [20,
21, 22]. Unlike the conventional hidden Markov model (HMM) based cleaning [23,
24], this method does not require any language-dependent pre-processing thanks
to the end-to-end framework. Also, for ASV, the audio can be filtered by
calculating variation of speaker representation within a video. This paper
applies the above techniques to a Japanese corpus as a case study.
Experimental evaluation demonstrates 1) we designed a new Japanese ASR
benchmark with more than 1,300 hours of training data and the official test
sets, and 2) we also constructed 900 hours of Japanese ASV corpus. The
contributions of this work are as follows:

  * •

We construct a new modern-size corpus for Japanese ASR and ASV from YouTube
videos. The video list is open-sourced in our project
page111https://github.com/sarulab-speech/jtubespeech.

  * •

Our process is applicable to many languages with high reproducibility. The
repository also contains the script, which has been extended to support
multiple languages for the data collection.

##  2 Corpus construction

Fig. 1: Procedure of corpus construction.

All of our steps are performed with minimal language-dependent processing.
Figure 1 shows the procedure.

###  2.1 Data collection

Creating search terms. The first step is to create search terms to be entered
into the video search engine. We extract the target language’s words with
hyperlinks from HTML files of Wikipedia articles. Unlike Gigaspeech [19],
categories are not specified, i.e., all articles are used. Also, we extract
“sudden-rise search terms” in the past few years from Google Trends.

Obtaining video IDs that have subtitles. Next, we retrieve video IDs that have
subtitles. Entering a search word into the video search engine, we obtain a
list of video ID candidates. Then, for each video, we retrieve whether the
video has subtitles. In this paper, we use only manual subtitles but also make
the list for automatic (i.e., machine-generated) subtitles.

Downloading audio and caption. Finally, we download the audio and manual
subtitles from videos. Since the number of channels and sampling frequency of
the audio file varies from video to video, we reformat the audio to 16 kHz-
sampled monaural WAV format.

###  2.2 Specific cleansing for speech recognition

The obtained audio data were already annotated with subtitles including
timings. This dataset still included many bad samples, e.g., unspoken
subtitles, English audio with Japanese subtitles, and other variations of
audio–text mismatches. To sort-out these bad samples, we calculated a score of
how well the audio segment fits to the subtitle and then filtered the
utterances based on their score. Furthermore, as many subtitle timings were
inaccurate, we fully re-aligned the subtitles to the audio. To calculate a
score and to re-align the subtitles to the audio, we use CTC segmentation [20]
as an alignment tool [25]. CTC segmentation utilizes CTC log-posteriors to
determine utterance timings in the audio given a ground-truth text.

Text pre–processing. We apply minimal text pre–processing so that the ground
truth text obtained from subtitles is composed of characters or tokens in the
model dictionary. Numbers are replaced with their spoken equivalent using the
_num2words_ Python library [26]. UTF-16 characters are mapped to the Japanese
character set. Automated subtitles are detected and filtered out based on the
average relative Levenshtein distance between subtitles.

Alignment. The onset and offset timings of an utterance are then estimated in
three steps: (1) In a forward pass, transition probabilities are mapped into a
trellis diagram of the ground-truth token sequence over all time steps. (2)
Backtracking starts from the most probable timing of the last character and
then determines the most probable path through the trellis. (3) A confidence
score is derived for each utterance from the per-token probabilities in the
trellis. The score is determined by the $L$ consecutive CTC output frames of
the utterance with the lowest token probabilities; we chose $L=30$ that
relates to approximately $1$s of audio. For a more in-depth description,
readers could refer to [20]. We apply one further modification that helps with
the alignment of fragmented subtitle text: In the original publication, the
algorithm was configured to skip preambles by setting transition cost to zero
for the token that marks the start of the first utterance. We extended this in
our setup to all utterances in order to skip any unrelated audio segments.

CTC scoring. We also calculate a CTC score for each audio-text pair from its
YouTube timings instead of fully aligning the subtitles to audio. For this,
the audio segment to each subtitle is cut out to then derive its confidence
score as described above.

Cleaning. Bad samples are then eliminated based on this confidence score. The
confidence score provides an estimated log-space probability of how well the
subtitles fit the audio data; this value is mainly influenced by the quality
of ASR models, input data, and data pre-processing. Note that a score
threshold $\theta$ of $-0.3$, used in one of our experiments, can be
interpreted as a production probability of at least $75\%$ each second.

Inference of long audio files. Alignment requires inference of the audio using
the encoder and the CTC layer of a pre-trained ASR model. We found that many
audio files are longer than three hours and their inference reaches a
practical limitation: The memory complexity using a Transformer-based model is
quadratic with audio length – a device with 64 GB memory is at most able to
infer $500$s of audio at once. Due to their rather linear memory complexity
along longer audio data, RNN-based models require less memory and thus, it is
possible to decode longer files. This can be scaled up to $2.7$h of audio
until the encoding reaches a memory limit of the Pytorch toolkit. To overcome
this limitation, we partition long audio files.

Partitioning is done by splitting the audio into smaller blocks, performing
inference on the parts, and then concatenating CTC posteriors. Maximum block
sizes were chosen depending on the memory consumption of the model and
available memory; the last block may be $25\%$ longer to avoid too short
blocks. As abrupt cuts in the audio cause distortions during inference, we add
an overlap to each side of the block; overlapping posteriors are later omitted
for scoring. By inspection of CTC posteriors, an overlap of at least $600$ms
is required to reduce impact of partitioning on scoring to an acceptable
level. All lengths were chosen as multiples of the samples–to–posteriors ratio
to preserve timing information and ensure the correct shape of the
concatenated CTC posterior tensor.

###  2.3 Specific cleansing for speaker verification

Unlike an ASR corpus, ASV requires high-quality speaker labels. Therefore, we
propose an unsupervised method for only extracting monologue videos (i.e.,
single-speaker videos) from the obtained videos. Furthermore, we remove videos
with text-to-speech (TTS) voices, which have different characteristics from
natural speech.

Fig. 2: Comparison of distributions in speaker space.

Removing non-speech and too-short videos. First, we delete videos without
speech. Unlike the corpus for ASR, there is no need to align subtitle and
speech. Therefore, we simply use voice activity detection (VAD) here. We
applied VAD to sections trimmed out based on the subtitles and used only
sections mainly consisting of speech. Also, too-short videos are deleted. This
is for robustly calculating the intra-video statistics described below.

Evaluating intra-video variation in speaker space. To extract single-speaker
videos, we compute the speaker variability in the video. Figure 2 illustrates
the concept. We use the $d$-vector [6], deep learning-based speaker
representation, which is extracted using pre-trained models. The $d$-vector is
first calculated for each utterance. Then, the variance of the $d$-vectors is
calculated within the video. We expect that the variance of TTS voices becomes
smaller than the single-speaker voices because the TTS voice has no
fluctuation among utterances. Also, if the different speakers’ voices are
contaminated (i.e., multi-speaker video), the variance will become larger than
the single-speaker video. Therefore, we can eliminate TTS videos and multi-
speaker videos by setting the appropriate threshold to the variance. For
implementation, the $d$-vector is reduced to a lower dimension by t-SNE [27],
and the determinant of its covariance matrix is obtained.

Grouping videos by YouTube Channel ID. It is necessary to prevent the same
speaker from being counted as different speakers. We obtain and use the
YouTube Channel ID of each video. Single-speaker videos with the same Channel
ID are grouped together and considered as a unique speaker.

##  3 Experimental evaluation

###  3.1 Evaluation in data collection

The period of data collection was between February and April of 2021. From the
data collection results listed in Table 1, we can describe 1) $5.09$ videos
are found for each search term and 2) $0.92$ % of videos have manual
subtitles, and $41.7$ % have automatic subtitles. In the end, we obtained
approximately $10,000$ hours of speech data from $110,000$ YouTube videos.

Table 1: Results of data collection. Videos with automatic subtitles are not used in this paper, but the video ID is also opensourced. Retrieved entity | Value
---|---
#search-terms | 2.34M terms
#videos found in the search | 11.9M videos
#videos with manual subtitles | 0.11M videos
(#videos with auto subtitles) | 4.96M videos
Table 2: Comparison of speech corpora of Japanese (upper half) and other rich-resource languages (lower half). Lang. | Task | Corpus | Open-source | Duration
---|---|---|---|---
Ja | ASR/ASV |  JNAS [28] | No | 90
Ja | ASR |  CSJ [15] | No | 600
Ja | ASR |  LaboroTVspeech [29] | Yes | 2,000
Ja | ASR |  Common Voice [10] | Yes | 2
Ja | ASV |  Liveness [30] | No | 4
Ja | ASR/ASV | JTubeSpeech (ours) | Yes | 1,300/900
En | ASR |  Librispeech [9] | Yes | 982
En | ASR |  Common Voice [10] | Yes | 1,100
En | ASR |  SPGISpeech [11] | Yes | 5,000
En | ASR |  GigaSpeech [19] | Yes | 10,000
En | ASV |  VoxCeleb [16] | Yes | 2,800
Zh | ASR |  Common Voice [10] | Yes | 12
Zh | ASR |  AISHELL-2 [31] | Yes | 1,000
Zh | ASV |  CN-Celeb [14] | Yes | 1,000

Table 2 shows a comparison with the existing corpus. The duration of our
corpus is a subset used in the ASR or ASV experiments described below. Among
ASR corpora, our corpus is similar in size to LaboroTVspeech (Ja) [29] and
Common Voice (En) [10]. Also, among ASV corpora, it is the first open-source
Japanese corpus and is similar in size to CN-Celeb (Zh) [14].

###  3.2 Evaluation in speech recognition

Data cleaning. The most important data pre-processing is to prune utterances
that have the wrong transcriptions and fix the incorrect timing. We
consistently employed CTC segmentation and CTC scoring, as described in
Section 2.2. We used a pre-trained CTC model based on the ESPnet
LaboroTVspeech [29] recipe [32]. Table 3 summarizes the statistic of the
various training and test sets.

As a pilot study, we used two subsets of our corpus: (1) ”single_speaker” (or
”ss” in short) based on the single speaker subset, as described in Section
2.3, and (2) ”top_15k” (or ”15k” in short) extracted based on top 15,000
videos in terms of the average score for each video. Note that top_15k is
pruned only based on utterance confidence scores and it may contain multi
speaker videos, unlike single_speaker. Also a part of these subsets is
overlapped.

Fig. 3: Histograms of the score for different ASR models over all utterances
of top_15k.

These subsets are further decomposed into the training and test sets, which
will be explained in the next section. We also prepare the various training
sets by changing the threshold value $\theta$ of the confidence score obtained
by CTC segmentation for the single_speaker and top_15k subsets. To determine
the threshold value, we investigate the distribution of the confidence score
over all utterances in the top_15k subset, as shown in Figure 3. This
distribution clearly shows that the peak of the distribution starts around
-3.0 in both RNN and transformer based CTC models. Based on the observation,
we regard the utterances located in the wide-based region as outlier data
points, and use -3.0 as the lowest threshold for pruning in Table 3. The
largest training set is obtained by combining the single_speaker and top_15k
subsets (train_ss_15k).

Test set design. Focusing on the ”single_speaker” videos reduces the cost of
manual process of verifying audio and transcriptions. The rest of the
procedure is as follows: (1) Select 1,621 videos from the ”single_speaker”
videos that include ”easy” utterances scored more than -0.3 threshold value
based on CTC. (2) Randomly pick up 324 videos, around 20%, and use them as a
test video set, which has 3,396 easy utterances in total. (3) Manually listen
utterances and identify 1,614 utterances having correct transcriptions. (4)
Split them into the development dev_easy_jun21 and evaluation eval_easy_jun21
sets, which have 785 and 829 utterances, respectively. We also made an
additional test set with ”normal” utterances222 The definition of ”easy” and
”normal” test sets are determined based on the CTC score objectively. Our
listening process and later ASR experiments confirmed that this categorization
is reasonable.  scored more than $-1.0$ threshold value. We fixed to use the
same test video set as we defined before, and performed steps 3 and 4. We made
dev_normal_jun21 and eval_normal_jun21 sets, which have 1,036 and 834
utterances, respectively.

Table 3: Training and test data statistics for the ASR task. $\theta$ is a threshold used to prune bad utterances based on the CTC score. | $\theta$ | # videos | # utts | hours
---|---|---|---|---
dev_easy_jun21 | -0.3 | 110 | 785 | 0.7
eval_easy_jun21 | -0.3 | 106 | 829 | 0.7
dev_normal_jun21 | -1.0 | 128 | 1,036 | 1.1
eval_normal_jun21 | -1.0 | 129 | 834 | 0.8
train_single_speaker | -0.3 | 1,297 | 14,797 | 12.7
train_single_speaker | -0.5 | 1,792 | 26,209 | 24.2
train_single_speaker | -1.0 | 2,906 | 66,563 | 71.9
train_single_speaker | -3.0 | 4,342 | 285,846 | 362.0
train_top_15k | -3.0 | 14,418 | 1,048,699 | 1087.1
train_ss_15k | -3.0 | 17,761 | 1,270,124 | 1376.9

Experimental results. We used an ESPnet state-of-the-art conformer model [5,
24] based on hybrid CTC/attention architectures [33]. The detailed
configuration can be found in the ESPnet JTubeSpeech recipe.

Figure 4 shows the ASR performance for various amounts of training data by
changing the score threshold $\theta$. We confirmed that the normal test set
was more difficult than the easy test set. This result validates our design of
the test set by controlling the difficulties based on the score. As regards
the amounts of training data, if we increase the amount of training data by
reducing $\theta$, the training data may contain more noisy transcriptions or
distorted audio segments. Nevertheless, the ASR performance was improved, and
the final CERs were $5.2\%$ in eval_easy_jun21 and $10.7\%$ in
eval_normal_jun21. These CER ranges are similar to other Japanese ASR
benchmarks, e.g., $4-6\%$ in CSJ [15] and $13\%$ in LaboroTVspeech [29]. We
also list the largest training data case (ss_15k_1376h) by combining the
single_speaker subset and top_15k subsets. The performance was improved in
most cases except for dev_easy_jun21. This result confirms that more training
data is generally helpful to improve the performance of more challenging data.

Fig. 4: ASR performance for various amounts of training data (12, 24, 71, 362,
and 1376 hours) by changing the score threshold and combining multi-speaker
recordings.

Finally, we evaluate the effectiveness of CTC segmentation, as discussed in
Section 2.2. We used train_single_speaker with $-1.0$ score threshold (71.9
hours) and prepared the corresponding training data based on the original
YouTube timings only with CTC scoring. Table 4 shows that the CTC segmentation
significantly improved the performance and shows the effectiveness of the re-
aligning for the YouTube audio data, as suggested by [19].

Table 4: The effectiveness of CTC segmentation. | dev_easy | dev_normal | dev_easy | dev_normal
---|---|---|---|---
original timing | 11.5 | 16.5 | 9.2 | 15.7
CTC segmentation | 9,2 | 14.3 | 6.9 | 13.6

###  3.3 Evaluation in speaker verification

####  3.3.1 Data cleansing

We used py-webrtcvad333https://github.com/wiseman/py-webrtcvad for VAD and a
pre-trained model444https://github.com/yistLin/dvector for extracting
$d$-vectors. The variation in speaker space was computed from more than 10
utterances for each video. As a pilot study, we used randomly selected 35,000
videos (approx. 30% of total). Figure 5 shows two levels of rapid increase:
near 0 and near 8 to 9 on the x-axis. Following the concept in Figure 2, we
set a threshold around these levels and assign “TTS,” “single speaker,” or
“multi speakers,” class to the videos.

Fig. 5: The intra-video variation in speaker space

We quantitatively evaluate the authenticity of this classification. We
randomly extracted 100 videos from each class and annotated a true class
label. Multi-TTS videos and overdubbed-voice videos were annotated as “multi
speakers.” To accurately find TTS videos, a TTS specialist participated in
this annotation. Table 5 shows that each class has their suitable videos.
Especially, the “single speaker” class does not contain multi-speaker videos.
Therefore, our method works for choosing the single-speaker videos. Some
single-speaker videos leak to the “TTS” class, but this impact is limited
because the “TTS” class has only a few videos.

Table 5: Comparison of classified and annotated video types. Classified \Annotated | TTS | single speaker | multi speakers
---|---|---|---
TTS | 20 | 80 | 0
single speaker | 5 | 95 | 0
multi speakers | 1 | 36 | 63

####  3.3.2 ASV dataset design and models

From the single-speaker videos, 92 unique speakers were selected as the
enrollment and testing speakers. For the training and development datasets,
1,795 unique speakers were selected. In the training, 127,997 and 25,392
utterances were used for training and development datasets, respectively. In
the testing, we designed an enrollment utterance and a testing one were picked
up from the same video but different segments. In the testing, we performed
25,392 trial pairs, included 276 correct pairs and 25,116 incorrect pairs.

We used a speaker embedding network, which was built with four convolutional
neural networks, pooling layers and two full connected layers. As the input
feature, 40-order Mel-Frequency Cepstrum Coefficients with 25-ms frame length
and half overlapped shift were used. The speaker-embedding vector was 512
dimensions. The evaluation metric is equal error rate (EER) was used for our
evaluation by comparing the cosine distance.

####  3.3.3 ASV performance

We performed ASV evaluations, and the EER of our ASV system with JTubeSpeech
was 10.9%. Our system was regarded as the speaker embedding system based on
the CNN-based model. Even though such a simple system, this result had the
similar performance to the CNN-based model (VGG-M) trained with Voxceleb1
[16]. As a pilot study, we designed a simple benchmark. For example, the
result was obtained with the small model and probabilistic linear discriminant
analysis (PLDA) scoring and data augmentation techniques were not performed.
However, from this simple benchmark, we could show the techniques of data
cleaning for selecting a unique speakers and constructing ASV systems
performed well. Consequently, we established the Japanese ASV systems with
large-scale open source media in the first time.

##  4 Conclusion

In this paper, we propose a speech corpus construction strategy and build
$1,300$ and $900$ hours data for Japanese ASR and ASV. One of our future
directions is to extend the corpus to multiple languages.

## References

  * [1] John Towns et al. “Xsede: Accelerating scientific discovery computing in science & engineering, 16 (5): 62–74, sep 2014” In _URL https://doi. org/10.1109/mcse_ 128, 2014
  * [2] Nicholas A Nystrom et al. “Bridges: a uniquely flexible HPC resource for new communities and data analytics” In _Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure_ , 2015, pp. 1–8
  * [3] George E. Dahl et al. “Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition” In _IEEE Transactions on Audio, Speech, and Language Processing_ 20.1, 2012, pp. 30–42
  * [4] Alex Graves and Navdeep Jaitly “Towards end-to-end speech recognition with recurrent neural networks” In _International conference on machine learning_ , 2014, pp. 1764–1772
  * [5] Anmol Gulati et al. “Conformer: Convolution-augmented Transformer for Speech Recognition” In _Proc. Interspeech_ , 2020, pp. 5036–5040
  * [6] E. Variani et al. “Deep neural networks for small footprint text-dependent speaker verification” In _ICASSP_ , 2014, pp. 4080–4084
  * [7] D. Snyder et al. “X-Vectors: robust DNN embeddings for speaker recognition” In _ICASSP_ , 2018, pp. 5329–5333
  * [8] Christopher Cieri, David Miller and Kevin Walker “The Fisher corpus: A resource for the next generations of speech-to-text.” In _LREC_ 4, 2004, pp. 69–71
  * [9] Vassil Panayotov et al. “Librispeech: An ASR corpus based on public domain audio books” In _ICASSP_ , 2015, pp. 5206–5210
  * [10] Rosana Ardila et al. “Common Voice: A Massively-Multilingual Speech Corpus” In _Proceedings of the 12th Language Resources and Evaluation Conference_ , 2020, pp. 4218–4222
  * [11] Patrick K. O’Neill et al. “SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition”, 2021 arXiv:2104.02014 [cs.CL]
  * [12] Yi Liu et al. “HKUST/MTS: A very large scale Mandarin telephone speech corpus” In _International Symposium on Chinese Spoken Language Processing_ , 2006, pp. 724–735
  * [13] Jiayu Du et al. “Aishell-2: Transforming mandarin asr research into industrial scale” In _arXiv preprint arXiv:1808.10583_ , 2018
  * [14] Yue Fan et al. “CN-CELEB: a challenging Chinese speaker recognition dataset”, 2019 arXiv:1911.01799 [eess.AS]
  * [15] Kikuo Maekawa et al. “Spontaneous Speech Corpus of Japanese.” In _Proc. LREC_ , 2000, pp. 947–952
  * [16] Arsha Nagrani et al. “Voxceleb: Large-scale speaker verification in the wild” In _Computer Science and Language_ , 2019
  * [17] Sami Abu-El-Haija et al. “YouTube-8M: A Large-Scale Video Classification Benchmark”, 2016 arXiv:1609.08675 [cs.CV]
  * [18] Daniel Galvez et al. “The People’s Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage” https://openreview.net/forum?id=R8CwidgJ0yT, 2021
  * [19] Guoguo Chen et al. “GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio” In _arXiv preprint arXiv:2106.06909_ , 2021
  * [20] Ludwig Kürzinger et al. “CTC-Segmentation of Large Corpora for German End-to-End Speech Recognition” In _Speech and Computer_ , 2020, pp. 267–278
  * [21] Yang Zhang et al. “NeMo Inverse Text Normalization: From Development To Production” In _arXiv preprint arXiv:2104.05055_ , 2021
  * [22] Evelina Bakhturina et al. “Hi-Fi Multi-Speaker English TTS Dataset”, 2021 arXiv:2104.01497 [eess.AS]
  * [23] Jan Trmal et al. “The Kaldi OpenKWS System: Improving Low Resource Keyword Search.” In _Proc. Interspeech_ , 2017, pp. 3597–3601
  * [24] Pengcheng Guo et al. “Recent developments on espnet toolkit boosted by conformer” In _ICASSP_ , 2021, pp. 5874–5878
  * [25] Ludwig Kürzinger and Dominik Winkelbauer “Ctc-Segmentation: Segment an audio file and obtain utterance alignments. (python package)” https://github.com/lumaku/ctc-segmentation In _GitHub_ , 2021
  * [26] Virgil Dupras, Marius Grigaitis and Taro Ogawa “Num2Words: Modules to convert numbers to words.” https://github.com/savoirfairelinux/num2words In _GitHub_ , 2021
  * [27] L.v.d. Maaten and G. Hinton “Visualizing data using t-SNE” In _Journal of machine learning research_ 9, 2008, pp. 2579–2605
  * [28] K. Itou et al. “JNAS: Japanese speech corpus for large vocabulary continuous speech recognition research” In _Journal of the Acoustical Society of Japan (E)_ 20.3, 1999, pp. 199–206
  * [29] Shintaro Ando and Hiromasa Fujihara “Construction of a Large-Scale Japanese ASR Corpus on TV Recordings” In _ICASSP_ , 2021, pp. 6948–6952
  * [30] Sayaka Shiota et al. “Voice liveness detection algorithms based on pop noise caused by human breath for automatic speaker verification” In _Sixteenth annual conference of the international speech communication association_ , 2015
  * [31] J. Du et al. “AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale” In _ArXiv_ , 2018 eprint: 1808.10583
  * [32] Various authors “Espnet/Egs2/Laborotv/Asr1 · espnet/espnet” https://github.com/espnet/espnet/tree/master/egs2/laborotv/asr1 In _GitHub_ , 2021
  * [33] Shinji Watanabe et al. “The 2020 ESPNet update: New features, broadened applications, performance improvements, and future plans” In _arXiv preprint arXiv:2012.13006_ , 2020


# When the Ground Truth is not True:
Modelling Human Biases in Temporal Annotations

Taku Yamagata1*, Emma L. Tonkin1*, Benjamin Arana Sanchez1, Ian Craddock1,
Miquel Perello Nieto1,
Raul Santos-Rodriguez1, Weisong Yang1, Peter Flach1 1University of Bristol,
United Kingdom
<EMAIL_ADDRESS>

###### Abstract

In supervised learning, low quality annotations lead to poorly performing
classification and detection models, while also rendering evaluation
unreliable. This is particularly apparent on temporal data, where annotation
quality is affected by multiple factors. For example, in the post-hoc self-
reporting of daily activities, cognitive biases are one of the most common
ingredients. In particular, reporting the start and duration of an activity
after its finalisation may incorporate biases introduced by personal time
perceptions, as well as the imprecision and lack of granularity due to time
rounding. Here we propose a method to model human biases on temporal
annotations and argue for the use of soft labels. Experimental results in
synthetic data show that soft labels provide a better approximation of the
ground truth for several metrics. We showcase the method on a real dataset of
daily activities.

11footnotetext: These authors contributed equally to this work.

##  I Introduction

The development of systems designed for detection and monitoring of everyday
human activities (e.g. movement, cooking, hygiene, sleep) benefits from
accurate labelled (annotated) data. Diarising methods (participant self-
reporting of events), often used for self-annotation, display several
limitations. Reliability is limited, e.g. Möller et al [1] found that subjects
in a 6-week study of smartphone use reported at most only 70% of detected
events, perhaps due to forgetfulness. Intentional misreporting may also be a
factor in recording of some activities, e.g. fitness [2]. Since the
contemporaneous annotation of a task usually requires interruption of a
participant’s activity, it is also likely that participants will not record
events contemporaneously, to retain goal focus and avoid excessive task
switching [3] and for practical reasons, e.g. it is impractical to input or
write down annotations during a task such as vacuuming or taking a shower. As
a consequence, participant-contributed annotations may be characterised as
unreliable and incomplete, in terms of event logging, event start/end time and
event duration.

Uncertainty in the estimation of time reflects participant use of coarse
temporal units to describe time and duration (e.g. an event began around a
certain time, estimated with a certain granularity [‘around half-past’] and
preference for ‘prototypical’ times [4, 5] and is estimated with a certain
duration). Self-reporting of event duration has been studied in linguistics
(often via large corpora [6]) and in music [7]. Precision of estimation of
time of day reflects participant time awareness. Time perception is complex,
elastic [8] and varies according to demographics such as age [9] and context,
e.g. time of day [10]. It is also worth noting that disrupted time awareness
is a clinical feature in various diagnoses, such as Alzheimer’s disease [11].
At present, the characteristics of participant temporal annotation are not
widely modelled in analysis of participant-labelled sensor data.

In summary, while a thorough discussion is beyond the scope of this paper,
unless supported by electronic time-stamping, participants’ timings for
activity start, end and duration are often inexact; we give an example of this
in Section 2. This quality issue may be compared with Kwon et al’s ‘label
jitter’ [12]. However, jitter in post-hoc annotation (e.g. of video) is likely
to be of the order of fractions of a second. As we see in Section 2,
participant estimation of activity times is found to have large uncertainty.
Much of the work on both evaluating label quality [13] and learning from weak
labels [14] originates from post-hoc annotation, in which multiple-annotator
performance can be compared (e.g. via statistical measures) and probability
distributions built accordingly.

This paper explores a means of modeling temporal uncertainty when making use
of participant contributed labels describing time-series annotation tasks, and
provides a first step in exploring the method’s suitability for particular
applications (e.g. evaluation of predictions, training models); it is
organised as follows. Section II presents a motivating case study. Section III
describes our approach to estimating the annotation ambiguity and computing
soft labels. Section IV demonstrates the characteristics of the soft labels
with artificial and real-world datasets. Section V presents critical
discussions of our approach. Finally, Section VI concludes with a summary and
possible future work.

##  II Case study: SRM-17 recording of activities of daily living

Figure 1: Histogram of granularity patterns found on SRM-17 shower/bath
annotations for dataset SPHERE 100-Homes, coloured by annotator.

During the SPHERE 100-homes project, a subset of participants were encouraged
to provide daily annotations via a modified Social Rhythm Metric (SRM-17),
designed to measure daily lifestyle regularity [15], self-administered by
participants for up to seven days, resulting in approximately 100 days of
data. Analysis of this small dataset demonstrated many features identified in
the introduction. Taking the example of shower/bath events, as can be seen in
Figure 1, participants’ contributed annotations fall on particular times
within the hour, e.g. on the hour or half past. Data often appears to be
reported to the nearest five minutes. This does not reflect instructions given
to participants, who were given no specific guidance regarding precision of
time reporting; researchers expected participants to report time read from a
watch or clock. That said, in the clinical domain it is common for
instructions to mandate a relatively low precision of time reporting. For
example, the frequently-used Hauser home diary to assess functional status in
Parkinson’s disease asks participants to fill out diaries every half hour
[16]. Therefore, we may envisage several circumstances in which data analysts
working with clinical data may find themselves in one of two conditions:
annotation reporting is constrained by diary frequency (e.g. ‘during the last
half hour’); or by participants’ chosen granularity of time approximation
(e.g. to the nearest five minutes, half hour, etc).

While we would like to evaluate how well predictions made on the basis of
sensor data correlate to the annotations provided by participants, sensor data
itself may give inaccurate timings. Activity and event predictions rely on
signals that vary due to incidental characteristics about a home e.g.
measurement of heat from cooking varies by stove type, placement, response
time and sensor placement[17]. Hence,[17] apply a ‘thermal lag’ parameter to
account for the time it takes for a temperature increase to be detected.
Similarly, humidity sensors used to predict when shower or bath events
occurred exhibit a ‘humidity lag’, due to characteristics of the bathroom
appliances (e.g. shower, bath), configuration (e.g. closed shower stall,
curtain, no enclosure from the bathroom), and architecture (e.g. room size,
window placement and ventilation such as extractor fans).

##  III Method

We propose to tackle the uncertainty in time-series annotations in two stages.
In the first step, we predict the time resolution each annotator uses
(ambiguity) with a Bayesian approach. In the second stage, we produce soft
labels based on the predicted annotation’s time resolution. It is also
important to consider how to evaluate a target model with soft labels (i.e.
how to use the soft labels), but this is outside the current paper’s scope.

Figure 2: Examples of biases added during the annotation process. The bottom
graph shows the time in which the person took a 37 minutes shower. The graph
above shows a possible annotation B estimating that it took about 30 minutes
at 8am, while the graph above that shows a different annotation A estimating a
shower of an hour at 8am. Each annotation indicates the true (and false)
positives (and negatives) that would result from comparison with the ground
truth.

###  III-A Predicting Annotator’s Habit and Annotation’s Uncertainties

First, we introduce predefined categories ($c_{n}$) for annotation time
resolutions. Each category has a subset of possible annotation values
($\mathbb{C}_{n}$) that indicate how the person who belongs to the category
tends to annotate. For example, if we define the 1st category ($c_{1}$) as 30
minutes time resolution, then the possible annotation times would be 0 and 30
minutes ($\mathbb{C}_{1}=\\{0,30\\}$). We use the following five categories to
demonstrate our method throughout this paper.

$c_{1}$ | : 30 mins. res. | $\mathbb{C}_{1}=\\{0,30\\}$
---|---|---
$c_{2}$ | : 15 mins. res. | $\mathbb{C}_{2}=\\{0,15,30,45\\}$
$c_{3}$ | : 10 mins. res. | $\mathbb{C}_{3}=\\{0,10,20,30,40,50\\}$
$c_{4}$ | : 5 mins. res. | $\mathbb{C}_{4}=\\{0,5,10,15,\dots,55\\}$
$c_{5}$ | : 1 mins. res. | $\mathbb{C}_{5}=\\{0,1,2,3,4,\dots,59\\}$

Then, we introduce two random variables. One indicates the annotator’s habit
($H_{j}$), which indicates which time resolution category the annotator tends
to use, and the other is a category ($C_{j,i}$) that is actually used for each
annotation. The index $j$ and $i$ are the annotators and the annotations
index, respectively. $C_{j,i}$ is defined for each annotation, and $H_{j}$ is
defined for each annotator. Now, we introduce our method to derive the
posterior probabilities of the two random variables ($H_{j}$ and $C_{j,i}$).
For simplicity, we drop the annotator index $j$ in the rest of the paper, as
the following process can be repeated for each annotator separately. We
present the notation below.

$\mathcal{D}=\\{d_{i}\\}_{i=1}^{N}$ |  Set of annotations
---|---
$N$ |  Number of annotations
$d^{*}_{i}$ |  True value for an annotation $d_{i}$
$H$ |  Annotator’s habit
$C_{i}$ |  i-th annotation category
$c_{n}$ |  Categories for $H$ and $C_{i}$
$|c_{n}|$ |  Number of annotations in category $c_{n}$
$T_{c_{n}}$ |  Time resolution for category $c_{n}$
$N_{c}$ |  Number of the categories

We assume that the $i$-th annotations ($d_{i}$) are generated from $C_{i}$ and
$d^{*}_{i}$, and $C_{i}$ is generated from $H$. Figure 3 shows the
probabilistic graphical model relating the variables. To infer the annotator’s
$H$ and annotations’ $C_{i}$, we compute their posterior probabilities given
annotations $\mathcal{D}$ i.e. $P(H|\mathcal{D})$ and $P(C_{i}|\mathcal{D})$.

Figure 3: Probabilistic graphical model relating the variables for
annotations. We assume that the annotation $d_{i}$ is generated from the
ground truth $d^{*}_{i}$ and the annotation time resolution $C_{i}$. The
annotation time resolution depends upon the annotator’s habit $H$. The index
$i$ is the annotation’s index.

$\begin{split}P(H|\mathcal{D})=\medmath{\frac{P(H)}{P(\mathcal{D})}\prod_{i=1}^{N}\sum_{C_{i}}P(C_{i}|H)P(d_{i}|C_{i})}\\\
\end{split}$ (1)

Eq. 1 shows how we can compute $P(H|\mathcal{D})$. Its detailed derivations
are in Appendix A. The term $P(d_{i}|C_{i})$ is the probability of getting the
annotation $d_{i}$ with the given annotation category $C_{i}$. Here, we assume
a uniform distribution for $P(d^{*}_{i})$; hence we define it as Eq. 2.

$\medmath{P(d_{i}|C_{i})=\begin{cases}\frac{1}{|C_{i}|},&\text{if
}d_{i}\in\mathbb{C}_{i}\\\ 0,&\text{otherwise}\end{cases}}$ (2)

where $|C_{i}|$ is a number of annotations in the category $C_{i}$. For
example, when $C_{i}=c_{1}$, $|C_{i}|=2$ as $c_{1}$ has two members
$\mathbb{C}_{1}=\\{0,30\\}$. Hence the probability of $d_{i}=0$ or $30$ when
$C_{i}=c1$ is $0.5$ in Eq. 2. The term $P(C_{i}|H)$ is the probability of
using the annotation category $C_{i}$ when the annotator’s habit is $H$. We
use the following $P(C_{i}|H)$ in this work:

$P(C_{i}|H)=\begin{cases}1-\delta,&\text{if }C_{i}=H\\\
\frac{\delta}{N_{c}-1},&\text{otherwise}\end{cases}$ (3)

where $\delta\in[0,1]$ is the probability of taking a different annotation
category to the annotator’s habit $H$. In this paper, we use $\delta=0.1$.
Further discussion regarding the choice of $\delta$ value is in Section V. We
assume the prior of $H$ is the uniform distribution, as we do not assume any
prior knowledge about the annotator’s habit. Now, we compute the posterior of
$C_{i}$, which can be derived as follows (again, the detailed derivations are
in Appendix B):

$\begin{split}P(C_{i}|\mathcal{D})=\sum_{H}\frac{P(d_{i}|C_{i})P(C_{i}|H)}{P(d_{i}|H)}P(H|\mathcal{D}),\\\
\end{split}$ (4)

where we can compute $P(d_{i}|C_{i})$ from Eq. 2, $P(C_{i}|H)$ from Eq. 3,
$P(H|D)$ from Eq. 1 and $P(d_{i}|H)=\sum_{C_{i}}P(d_{i}|C_{i})P(C_{i}|H)$.
Once we have computed the posteriors, we use them to produce the soft labels.
It is possible to prepare the soft labels for all categories and combine them
according to the posteriors (fully Bayesian approach). This work takes the
maximum a posteriori (MAP) estimation for $C_{i}$ and generates the soft label
from it.

###  III-B Generating Soft Labels

The annotations for time-series data have been produced by recording an
event’s start and end time. We introduce a probability distribution over the
timings based on the category $C_{i}$ detected in the previous subsection and
compute soft labels that indicate a probability of the event happening at a
given time. We adopt a uniform distribution for the true start and end timings
$d^{*}_{s}$ and $d^{*}_{e}$ because we assume the annotations are generated by
rounding the actual time. (Again, it is interesting for future work to study
different distributions.) The distributions are uniform across the period of
the inferred annotation interval and placed its centre at the annotated
timing. For example, if a start timing annotation category $C_{s}$ is $c_{3}$,
it becomes uniform distribution bounded by $d_{s}-5$ and $d_{s}+5$ as the
category $c_{3}$ has annotations with 10 minutes intervals, where $d_{s}$ and
$d_{e}$ are annotations for the start and end timing.

$\begin{split}p(d^{*}_{s})&=\mathrm{U}(d_{s}-T_{C_{s}}/2,d_{s}+T_{C_{s}}/2),\\\
p(d^{*}_{e})&=\mathrm{U}(d_{e}-T_{C_{e}}/2,d_{e}+T_{C_{e}}/2),\\\ \end{split}$
(5)

where $T_{C_{s}}$ and $T_{C_{e}}$ are the annotation time intervals for the
inferred annotation categories $C_{s}$ and $C_{e}$.***These probabilities must
be conditioned on $C_{i}$ and $d_{i}$ like $p(d^{*}_{s}|C_{s},d_{s})$. We drop
the conditioning for simplicity here. Then, we compute probabilities of the
event that has started $P(d^{*}_{s}\leq t)$ and has not yet ended
$P(d^{*}_{e}>t)$ at time $t$ by taking the integration of Eq. 5.

$\begin{split}P(d^{*}_{s}\leq
t)&=\int_{-\infty}^{t}p(d^{*}_{s})\leavevmode\nobreak\ dd^{*}_{s},\\\
P(d^{*}_{e}>t)&=\int_{t}^{+\infty}p(d^{*}_{e})\leavevmode\nobreak\
dd^{*}_{e}.\\\ \end{split}$ (6)

Finally, we compute the soft label ($P^{\text{(label)}}(t)$), which is the
probability of the event at time $t$ by multiplying $P(d^{*}_{s}<t)$ and
$P(d^{*}_{e}>t)$. Here, we assume the start and end timings are statistically
independent. It simplifies the following derivations and helps convey the idea
of our approach. It is a strong assumption, and it does not hold in some
cases. For example, if the start and end timings are close relative to the
annotation time resolution, then we need to consider the dependency between
the start and end timings – the end timing must be later than the start
timing. We want to extend our model to support such a scenario in the future.

$\begin{split}P^{\text{(label)}}(t)&=P(d^{*}_{s}\leq t\wedge d^{*}_{e}>t)\\\
&=P(d^{*}_{s}\leq t)P(d^{*}_{e}>t).\\\ \end{split}$ (7)

Figure 4 shows the above soft label computation process with uniform
distributions. It starts with the given start/end time annotations ($d_{s}$
and $d_{e}$) and the estimated time resolutions, we assume the probability
distribution of the actual start/end time ($d^{*}_{s}$ and $d^{*}_{e}$). Then,
compute the probabilities that the event has already started and ended for
each time slot. Finally, we compute the probability that the event has started
but not ended for each time slot, becoming the soft label.

Figure 4: Soft label for time series data. Given start/end time annotations
($d_{s}$ and $d_{e}$) and the estimated time resolutions, we assume the
probability distribution of the actual start/end time ($d^{*}_{s}$ and
$d^{*}_{e}$) – top of the figure. Then, we compute the probabilities that the
event has already started and ended for each time slot. Finally, we compute
the probability that the event has started but not ended for each time slot,
becoming the soft label.

##  IV Experiments

In this section, we compare the hard and soft labels with artificial examples
and real-world datasets. The hard labels have either true or false values and
are set to true for the time slots between the start and end time annotations.
The artificial examples allow us to demonstrate the characteristics of the
soft labels in a controlled environment. We show the actual use case with
real-world datasets.

###  IV-A Simple Task

First, we compare the soft and conventional hard labels in a simple synthetic
example. It consists in a series of tasks with a start and end events (e.g.
sleeping, cooking or showering). Here, we randomly generate the true start and
end times (ground truth), then we make the annotations by rounding the times
based on a given time resolution (e.g. 5, 10, 15 and 30 minutes). We consider
the rounded start and end times as the provided hard labels, and apply the
proposed method to obtain the soft labels as described in Sec. III. Finally,
we evaluate the hard and soft labels by comparing them to the ground truth.
Figure 5 shows the mean squared error (MSE) of the ground truth with respect
to the hard and soft labels, with the time resolution of the annotations in
the X-axis. The MSEs are measured around the ground truth start and end time
$\pm 15$ minutes range. The result suggests that the hard label has a
progressively larger error than the soft label when the resolution interval of
the time annotations increase. This clearly indicates that the proposed soft
labels are less penalizing than the hard labels.

Figure 5: MSE for the hard and soft labels against the ground truth. The
X-axis is the annotation time resolution. The MSE measured around the ground
truth start and end time $\pm 15$ minutes range. The results show the soft
label is better (smaller MSE) than the hard label.

Next, we perform a similar analysis with F1 score, as in most detection
scenarios the positive class is more important than the negative one. The left
plot of Figure 6 shows the results with the annotation time resolution on the
X-axis. This suggests the hard label is better (higher F1 score) than the soft
label. This is the opposite result to the MSE result and counter-intuitive as
the soft label accurately reflects the degree of ambiguity by having values
between zero and one, whereas the hard label has only either zero or one. The
F1 score (or any metric based on the confusion matrix) is penalised by having
a value between zero and one (like soft labels). Also, this is the best-case
scenario for the hard label, as the annotation is produced by using rounding.
Hence, the expected ground truth is matched with the hard label. We are
interested in exploring evaluation metrics that do not penalise the prediction
or ground truth ambiguity. However, this is left for future work.

Figure 6: F1 score for the hard and soft labels against the ground truth with
different annotation time resolutions. The annotations are produced from the
ground truth by applying a bias and rounding. Here, we use zero and half of
the annotation time resolution as the bias (offset). The results suggest that
the soft label performs better than the hard label with the bias.

We also evaluate the annotations produced by applying a bias (offset) and
rounding to the ground truth. This is useful to simulate a case in which the
annotator does not remember accurately the beginning and end of the event,
which adds a possible bias on top of the rounding error. With this bias, the
expected ground truth no longer matches the hard label. We set the bias to
half of the annotation resolution and measure the F1 score for the hard and
soft labels. The result (the right plot of Figure 6) suggests that the soft
label is better than the hard label in this case. We can see that the hard
label results are degraded due to the bias in the annotation, and the soft
label results stay the same as before. This is because we shift the uniform
distribution for producing the soft label (Eq. 5) based on the estimated
annotation resolution.

###  IV-B Humidity Event Detection

We test the soft and hard labels for evaluating a shower event detection
model. The model uses the humidity sensor reading and predicts if someone is
taking a shower at each time slot – binary classification task. The model used
is a hidden Markov model. It takes the humidity level as the observations and
treats the shower status (on/off) as the hidden variable. We assume that the
predictive model is already provided, as the learning process is not the focus
of this paper; but the evaluation.

Table I: Confusion matrix for the shower event detection model with soft and
hard labels.

(a) With hard labels | Prediction |
---|---|---
Label | No | $Yes$
No | 3186 | $63$
Yes | 41 | $70$

(b) With soft labels | Prediction |
---|---|---
Label | No | $Yes$
No | 3185.14 | $69.50$
Yes | 41.86 | $63.50$

We use the SRM-17 dataset [15] with self-reported annotations. We pick the
dataset for ID:4 (Figure 1) as it has a coarse annotation time resolution.
First, we generate the hard and soft labels based on self-report annotations
and then compare them against the prediction model output. Table I shows the
confusion matrices for the evaluation results with the hard and soft labels.
We do not know the ground truth; hence we cannot directly compare the
performance of these labels. We can say that the soft and hard labels give
different results. We discuss approaches to further detailed characterisation
and evaluation in the future work section of this paper. We can also see that
the results with the soft label are slightly worse than those with the hard
labels (slightly fewer true-positives and true-negatives and more false-
positives and false-negatives). It is because these matrices penalise the
ambiguities (soft labels); as in Section 4.1, the suitability of these metrics
for comparative evaluation is left for further discussion.

##  V Discussion

In this section, we first discuss the findings from synthetic data and then
present implications and limitations for real-world examples.

#### Accuracy of the category estimation

Our method estimates the posterior of the random variables $H$ and $C_{i}$. It
is designed to pick the annotation category for coarser time resolution
categories than the finer resolution ones. It is intuitively correct because
if the annotations have only 30 minutes of the resolution, it is natural to
think the annotator uses 30 minutes rather than the finer resolution that also
possible to have 0 and 30 minutes, such as 15 minutes or 10 minutes. Our
method wrongly estimates the annotation categories if all (or most)
annotations landed on a coarser resolution time. The likelihood of such an
error diminishes if it has more annotations.

Figure 7: Error rate of the annotation’s category estimation. The error rates
are measured on synthetically generated annotations. Each line is for the
annotation category. The X-axis is the number of annotations. The error rate
decrease as it has more annotations.

Figure 7 shows the error rate of the annotation category estimation
$\operatorname*{arg\,max}_{C_{i}}{P(C_{i}|\mathcal{D})}$. As the plot
suggests, the error rate decreases quickly with the more annotation it
receives. Here we assume all annotations come from a single time resolution
category. Each line indicates the annotation category for the annotation.
There is no line for resolution=30 minutes as the error rate is always zero.

#### Choice of $\delta$ in Eq. 3

The parameter $\delta$ indicates how likely the annotator uses the different
annotation categories from the habit ($H$). $\delta=0.1$ means that the
annotator uses different annotation categories once in ten times. It affects
both of the two random variable posteriors $P(H|\mathcal{D})$ and
$P(C_{i}|\mathcal{D})$. If $\delta$ is small (close to zero), then $H$ would
be the category compatible with all annotations. For example, if all but one
annotation are 30 minutes (e.g. 9:30), and one has 1-minute time resolution
(e.g. 13:16), then $H$ would be $c_{5}$ (1-minute resolution) when $\delta$ is
too small. On the other hand, if $\delta$ is too large ($\sim 1.0$), then the
posterior of $C_{i}$ might ignore the $H$ and pick the category based only on
the annotation value $d_{i}$. For example, if all annotations are 5 minutes
time resolution, then $H$ would correctly be $c_{4}$ (5-minutes resolution).
However, $C_{i}$ would be wrongly $c_{1}$ if $d_{i}\in\mathbb{C}_{1}$ with too
large $\delta$. We pick $\delta=0.1$ as it seems a good balance for picking
the right $H$ and $C_{i}$.

#### Choice of $p(d^{*}_{i}|d_{i},C_{i})$

In this paper, we use the uniform distribution for $p(d^{*}_{i}|d_{i},C_{i})$
(Eq. 2) as we assume a simple annotator behaviour – just rounding the actual
time. However, it is more plausible to use gradually increasing and decreasing
distributions – e.g. Gaussian or trapezoidal shape distributions. This is left
for future work.

#### Learning from soft labels

Our proposed method can be potentially applied during the learning process of
a classifier/regressor as well. For example, by setting the start and end of
the events with the proposed soft labels, we can interpret the augmented
regions as weak labels [18, 19, 14]. Two possible algorithms that can be
potentially used in this scenario are pseudo-labeling [20] and Optimistic
Superset Learning [21], which are iterative learning methods that consider the
model’s predictions among the candidate labels as correct, and retrain the
model with those labels. It is also possible to add those samples for which
the model is more confident (e.g. exceeding a certain threshold).

We could also consider the proposed soft labels as probabilities coming from a
Bernoulli distribution, or as prior beliefs of belonging to the positive
class. Some algorithms that use similar soft labels are label smoothing [22]
and an Expectation Maximization method proposed by Jin and Ghahramani [23].

#### Evaluation metrics for soft labelling

The commonly used evaluation metrics for classification tasks (such as
accuracy, precision, recall and F1 score) are penalised by the ambiguity in
the labels or the predictions (the ambiguity reduces these scores). In this
paper, we show that MSE does not have such an issue and correctly show the
benefit of the soft labels in our experiment. Also, the MSE is similar to the
Brier score, which is commonly used to evaluate probabilistic predictions’
accuracy. It may suggest that the MSE is the right evaluation metric for soft
labels. However, we need further study to understand which metric is
appropriate in which scenario (objective) with the soft labels.

#### Towards real-world comparison of soft and hard labels

Real-world comparison of soft and hard labels in contemporaneous self-
annotation by participants is complicated by the lack of wholly reliable
ground truth, particularly in environments in which it is not possible to rely
on data collection suitable for post-hoc annotation methods (e.g. video is
unlikely to be appropriate in domestic bathrooms). However, we may look toward
other data sources to resolve or reduce some of the ambiguities identified.
For example, we might look to sensor fusion, referencing other sensors for
associated information such as presence, temperature or power use e.g. to
detect operating times of the appliance, or to get accurate bounds of a
person’s entrance into and departure from the room. This helps to resolve the
confounding question of sensor lag (e.g. time taken before sensor detects
temperature or humidity rises), giving us a greater insight into real-world
timings.

#### Human performance in temporal estimation

Time estimation performance and bias is a complex topic and the underlying
mechanisms and their causes are largely beyond the scope of this paper.
However, we hope that discussion of this mechanism may spark further
exploration of the characterisation and handling of this aspect of self-
reported data.

##  VI Conclusion

We proposed a method to identify the annotator’s approach to the task and the
ambiguity that comes with it. Also, we devised a way to generate soft labels
based on the estimated ambiguity. Our evaluation results suggest that the soft
label is better in mean squared error than the hard label. However, the soft
labels show worse results than the hard labels in terms of F1 score, because
metrics like F1 score inherently penalise the ambiguities. We consider many
avenues for future work, namely, improving the model of human annotations,
designing new evaluation metrics for soft labels and using soft labels for the
learning stage. We also hope to report on variance in granularity of
participant annotation of other types of events in a future publication.

## Acknowledgment

This work was supported by the SPHERE Next Steps Project funded by EPSRC under
Grant EP/R005273/1. RSR is funded by the UKRI Turing AI Fellowship
EP/V024817/1.

$\begin{split}P(H|\mathcal{D})&=P(\mathcal{D}|H)P(H)/P(\mathcal{D})\\\
&=\sum_{C_{1},\dots,C_{N}}\sum_{d^{*}_{1},\dots,d^{*}_{N}}P(d_{1},\cdots,d_{N}|d^{*}_{1},\cdots,d^{*}_{N},C_{1},\cdots,C_{N})P(C_{1},\cdots,C_{N}|H)P(d^{*}_{1},\cdots,d^{*}_{N})\frac{P(H)}{P(\mathcal{D})}\\\
&=\medmath{\sum_{C_{1},\dots,C_{N}}\sum_{d^{*}_{1},\cdots,d^{*}_{N}}\prod_{i=1}^{N}P(d_{i}|d^{*}_{i},C_{i})P(d^{*}_{i})P(C_{i}|H)\frac{P(d^{*}_{1},\cdots,d^{*}_{N})}{P(d^{*}_{1})\cdots
P(d^{*}_{N})}\frac{P(H)}{P(\mathcal{D})}}\\\
&=\medmath{\frac{P(d^{*}_{1},\cdots,d^{*}_{N})}{P(d^{*}_{1})\cdots
P(d^{*}_{N})}\frac{P(H)}{P(\mathcal{D})}\prod_{i=1}^{N}\sum_{C_{i}}P(C_{i}|H)\sum_{d^{*}_{i}}P(d_{i}|d^{*}_{i},C_{i})P(d^{*}_{i})}\\\
&=\medmath{\frac{P(d^{*}_{1},\cdots,d^{*}_{N})}{P(d^{*}_{1})\cdots
P(d^{*}_{N})}\frac{P(H)}{P(\mathcal{D})}\prod_{i=1}^{N}\sum_{C_{i}}P(C_{i}|H)P(d_{i}|C_{i})}\end{split}$
(8)

## References

  * [1] A. Möller, M. Kranz, B. Schmid, L. Roalter, and S. Diewald, “Investigating self-reporting behavior in long-term studies,” in _Proceedings of SIGCHI_ , ser. CHI ’13, 2013, p. 2931–2940.
  * [2] J. Lester, T. Choudhury, and G. Borriello, “A practical approach to recognizing physical activities,” in _International conference on pervasive computing_. Springer, 2006, pp. 1–16.
  * [3] E. L. Tonkin, A. Burrows, P. R. Woznowski, P. Laskowski, K. Y. Yordanova, N. Twomey, and I. J. Craddock, “Talk, text, tag? understanding self-annotation of smart home data from a user’s perspective,” _Sensors_ , vol. 18, no. 7, p. 2365, 2018.
  * [4] G. Labianca, H. Moon, and I. Watt, “When is an hour not 60 minutes? deadlines, temporal schemata, and individual and task group performance,” _Academy of Management Journal_ , vol. 48, no. 4, pp. 677–694, 2005. [Online]. Available: https://doi.org/10.5465/amj.2005.17843945
  * [5] J. E. McGrath, “The place of time in social psychology,” _The social psychology of time: New perspectives_ , pp. 7–17, 1988.
  * [6] F. Pan, R. Mulkar-Mehta, and J. R. Hobbs, “Modeling and learning vague event durations for temporal reasoning,” in _AAAI_ , 2007, pp. 1659–1662.
  * [7] Y. Ni, M. McVicar, R. Santos-Rodríguez, and T. De Bie, “Understanding effects of subjectivity in measuring chord estimation accuracy,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 21, no. 12, pp. 2607–2615, 2013.
  * [8] S. Droit-Volet and W. H. Meck, “How emotions colour our perception of time,” _Trends in Cognitive Sciences_ , vol. 11, no. 12, pp. 504–513, 2007.
  * [9] M. Wittmann and S. Lehnhoff, “Age effects in perception of time,” _Psychological reports_ , vol. 97, no. 3, pp. 921–935, 2005.
  * [10] S. S. Campbell, P. J. Murphy, and C. E. Boothroyd, “Long-term time estimation is influenced by circadian phase,” _Physiology & Behavior_, vol. 72, no. 4, pp. 589–593, 2001.
  * [11] M.-C. Requena-Komuro, C. R. Marshall, R. L. Bond, L. L. Russell, C. Greaves, K. M. Moore, J. L. Agustus, E. Benhamou, H. Sivasathiaseelan, C. J. Hardy _et al._ , “Altered time awareness in dementia,” _Frontiers in neurology_ , vol. 11, p. 291, 2020.
  * [12] H. Kwon, G. D. Abowd, and T. Plötz, “Handling annotation uncertainty in human activity recognition,” in _Proceedings of the 23rd International Symposium on Wearable Computers_ , ser. ISWC ’19, 2019, p. 109–117.
  * [13] R. Poyiadzi, W. Yang, N. Twomey, and R. Santos-Rodríguez, “Statistical hypothesis testing for class-conditional label noise,” in _European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases_ , 2022.
  * [14] R. Poyiadzi, D. Bacaicoa-Barber, J. Cid-Sueiro, M. Perello-Nieto, P. Flach, and R. Santos-Rodriguez, “The weak supervision landscape,” in _2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events_ , 2022, pp. 218–223.
  * [15] T. H. Monk, E. Frank, J. M. Potts, and D. J. Kupfer, “A simple way to measure daily lifestyle regularity,” _Journal of sleep research_ , vol. 11, no. 3, pp. 183–190, 2002.
  * [16] R. A. Hauser, F. Deckers, and P. Lehert, “Parkinson’s disease home diary: further validation and implications for clinical trials,” _Movement Disorders_ , vol. 19, no. 12, pp. 1409–1413, 2004.
  * [17] E. R. Coffey, E. C. Mesenbring, M. Dalaba, D. Agao, R. Alirigia, T. Begay, A. Moro, A. Oduro, Z. Brown, K. L. Dickinson, and M. P. Hannigan, “A glimpse into real-world kitchens: Improving our understanding of cookstove usage through in-field photo-observations and improved cooking event detection (cooked) analytics,” _Development Engineering_ , vol. 6, 2021.
  * [18] J. Hernández-González, I. Inza, and J. A. Lozano, “Weak supervision and other non-standard classification problems: A taxonomy,” _Pattern Recognition Letters_ , vol. 69, pp. 49–55, 1 2016.
  * [19] M. Perelló-Nieto, R. Santos-Rodríguez, and J. Cid-Sueiro, “Adapting supervised classification algorithms to arbitrary weak label scenarios,” in _Advances in Intelligent Data Analysis XVI_ , 2017, pp. 247–259.
  * [20] D.-H. Lee, “Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks,” _ICML 2013 Workshop: Challenges in Representation Learning (WREPL)_ , 2013.
  * [21] E. Hüllermeier and W. Cheng, “Superset learning based on generalized loss minimization,” in _Machine Learning and Knowledge Discovery in Databases_ , 2015, pp. 260–275.
  * [22] R. Müller, S. Kornblith, and G. E. Hinton, “When does label smoothing help?” in _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019.
  * [23] R. Jin and Z. Ghahramani, “Learning with multiple labels,” in _Proceedings of the 15th International Conference on Neural Information Processing Systems_ , ser. NIPS’02, S. Becker, S. Thrun, and K. Obermayer, Eds. Cambridge, MA, USA: MIT Press, 2002, p. 921–928.

##  Appendix A $H$ posterior ($P(H|\mathcal{D})$) derivation

Eq. 8 shows how to derive $P(H|\mathcal{D})$. We insert two sets of latent
variables ($C_{1},\cdots,C_{N}$ and $d^{*}_{1},\cdots,d^{*}_{N}$) and apply
the chain rule to get the second line. Then, factorising $d_{i}$ and $C_{i}$
by using the graphical model shown in Figure 3. Finally, the last line is
derived by pushing the $\sum$ operators into the relevant terms and replacing
the term $\sum_{d^{*}_{i}}P(d_{i}|d^{*}_{i},C_{i})d(d^{*}_{i})$ with
$P(d_{i}|C_{i})$. We also assume
$\frac{P(d^{*}_{1},\cdots,d^{*}_{N})}{P(d^{*}_{1})\cdots P(d^{*}_{N})}=1.0$
(assumed $d^{*}$ are all independent each other) to get Eq. 1.

##  Appendix B $C_{i}$ posterior ($P(C_{i}|\mathcal{D})$) derivation

Eq. 9 shows the derivation of $P(C_{i}|\mathcal{D})$. First, we introduced the
hidden variable $H$, then replace $\mathcal{D}$ with $d_{i}$ as $C_{i}$ only
depends upon $d_{i}$ in $\mathcal{D}$. The next, we apply Bayes rule to
$P(C_{i}|d_{i},H)$ and drop $H$ from $P(d_{i}|C_{i},H)$, as $d_{i}$ does not
depend upon $H$ when $C_{i}$ is given (Figure 3). Finally, it replace
$P(d_{i}|H)$ with $\sum_{C_{i}}P(d_{i}|C_{i})P(C_{i}|H)$.

$\begin{split}P(C_{i}|\mathcal{D})&=\sum_{H}P(C_{i},H|\mathcal{D})\\\
&=\sum_{H}P(C_{i}|d_{i},H)P(H|\mathcal{D})\\\
&=\sum_{H}\frac{P(d_{i}|C_{i},H)P(C_{i}|H)}{P(d_{i}|H)}P(H|\mathcal{D})\\\
&=\sum_{H}\frac{P(d_{i}|C_{i})P(C_{i}|H)}{P(d_{i}|H)}P(H|\mathcal{D})\\\
&=\sum_{H}\frac{P(d_{i}|C_{i})P(C_{i}|H)}{\sum_{C_{i}}P(d_{i}|C_{i})P(C_{i}|H)}P(H|\mathcal{D}),\end{split}$
(9)

  *[MAP]: maximum a posteriori