# A Phoneme-informed Neural Network Model for Note-level Singing Transcription

###### Abstract

Note-level automatic music transcription is one of the most representative
music information retrieval (MIR) tasks and has been studied for various
instruments to understand music. However, due to the lack of high-quality
labeled data, transcription of many instruments is still a challenging task.
In particular, in the case of singing, it is difficult to find accurate notes
due to its expressiveness in pitch, timbre, and dynamics. In this paper, we
propose a method of finding note onsets of singing voice more accurately by
leveraging the linguistic characteristics of singing, which are not seen in
other instruments. The proposed model uses mel-scaled spectrogram and phonetic
posteriorgram (PPG), a frame-wise likelihood of phoneme, as an input of the
onset detection network while PPG is generated by the pre-trained network with
singing and speech data. To verify how linguistic features affect onset
detection, we compare the evaluation results through the dataset with
different languages and divide onset types for detailed analysis. Our approach
substantially improves the performance of singing transcription and therefore
emphasizes the importance of linguistic features in singing analysis.

Index Terms—  singing transcription, onset detection, phoneme classification,
music information retrieval

##  1 Introduction

Note-level singing transcription is an music information retrieval (MIR) task
that predicts attributes of note events (i.e., onset time, offset time, and
pitch value) from audio recordings of singing voice. Although this task has
been studied for a long time, the performance of singing transcription is
generally inferior to those of other musical instruments such as polyphonic
piano music [1, 2]. The lack of large-scale labeled datasets is one of the
major technical barriers. In addition, singing voice has highly diverse
expressiveness in terms of pitch, timbre, dynamics, as well as phonation of
lyrics. For example, singing techniques such as vibrato, bending, and
portamento make it difficult to find note boundaries and note-level pitches.
This variability makes even manual note transcription by human experts
difficult [3]. This in turn has resulted in the lack of high-quality labeled
datasets.

Another important characteristic of singing voice which is well distinguished
from other instruments is that it conveys linguistic information through
lyrics and this influences note segmentation. Given that most singing notes
are syllabic (i.e., one syllable of text is set to one note of music) and
melismatic (i.e., one syllable is sung with multiple notes), the relationship
between the change of syllables and the change of notes is sophisticated. This
makes certain kinds of note patterns of singing voice not seen in any other
instruments. Therefore, we need to consider such linguistic characteristic in
automatic singing transcription models.

Fig. 1: An example of singing voice: mel-spectrogram (top), piano roll with
onsets and pitches of notes (middle), and phonetic posteriorgram (PPG)
(bottom) from singing (phonemes with probability under 0.5 in this example
were omitted).

In this paper, we propose a neural network model that incorporates linguistic
information into the input to improve note-level singing transcription for
singing voice. Similar to earlier research, we use log-scaled mel-spectrogram
as a primary input. In addition to that, we take phonetic posteriorgram (PPG)
from a pre-trained phoneme classifier as the second input. As shown in Figure
1, PPG shows a pattern distinct from the ones of mel-spectrogram, and it can
be noted that the transition pattern of PPG can better describe the onset
event at 1.2 and 2.8 second. We propose a two-branch neural network model
based on a convolutional recurrent neural network (CRNN) backbone to represent
both of the input features effectively. In the experiment, we conduct an
ablation study to examine the effectiveness of model design, mel-spectrogram,
and PPG. Also, we compare the effects of mel-spectrogram and PPG on transition
and re-onset, the two types of challenging onset events in singing
transcription. Finally, we demonstrate that our proposed model outperforms a
few state-of-the-art note-level singing transcription models, especially in
terms of onset detection.

Fig. 2: The proposed model architecture

##  2 Related Works

Traditional studies mainly used various types of spectral difference for onset
detection of audio signals [4]. The spectral difference is particularly
successful at finding percussive onsets but it performs poorly on expressive
instruments that have soft onsets. Deep neural networks have been actively
applied to singing voice as well. Nishikimi _et al_. [5] suggested an
attention-based encoder-decoder network with long short-term memory (LSTM)
modules. Fu _et al_. [6] proposed a hierarchical structure of note change
states to segment singing notes and used multi-channel features to increase
the performance. Hsu et al. [7] suggested a semi-supervised AST framework.
More recently, [8] proposed the object detection-based approach to
significantly improve the performance of singing voice onset/offset detection.
While the majority of them relied on note onset and offset information from
melody labels, one recent attempted to use phoneme information as part of
input features for note segmentation [9]. However, the performance was not
convincing. In this work, we present a neural network architecture to make an
effective use of the phoneme information.

Training dataset |  | SSVD v2.0 | CSD-refined
---|---|---|---
Evaluation dataset |  | ISMIR2014 | SSVD v2.0 | CSD-refined | ISMIR2014 | SSVD v2.0
| Feature | COn | COff | COn | COff | COn | COff | COn | COff | COn | COff
(a) Single CRNN | $X$ | 0.8244 | 0.7751 | 0.8956 | 0.8983 | 0.9797 | 0.9719 | 0.8812 | 0.7524 | 0.8866 | 0.8007
(b) Dual CRNNs + one RNN | $X,X$ | 0.9133 | 0.8513 | 0.9486 | 0.9566 | 0.9888 | 0.9838 | 0.9004 | 0.7636 | 0.8988 | 0.8089
(c) Single CRNN | $\hat{P}$ | 0.8655 | 0.7776 | 0.9223 | 0.9105 | 0.9890 | 0.9660 | 0.9048 | 0.7685 | 0.9063 | 0.8296
(d) Dual CRNNs + one RNN | $\hat{P},\hat{P}$ | 0.9094 | 0.8310 | 0.9342 | 0.9470 | 0.9907 | 0.9638 | 0.9090 | 0.7733 | 0.9142 | 0.8336
(e) Dual CNNs + one RNN | $X,\hat{P}$ | 0.9024 | 0.8349 | 0.9439 | 0.9420 | 0.9877 | 0.9791 | 0.9016 | 0.7852 | 0.9098 | 0.8340
(f) Dual CNNs + two RNNs | $X,\hat{P}$ | 0.9230 | 0.8538 | 0.9496 | 0.9531 | 0.9914 | 0.9839 | 0.9150 | 0.7804 | 0.9199 | 0.8328
(g) Dual CRNNs + one RNN | $X,\hat{P}$ | 0.9305 | 0.8576 | 0.9569 | 0.9692 | 0.9923 | 0.9864 | 0.9145 | 0.7723 | 0.9166 | 0.8257
Table 1: Onset/Offset detection results from various neural network
architectures with two input features. $X$ and $\hat{P}$ denote mel-
spectrogram and PPG, respectively. (g) corresponds to the neural network
architecture in Figure 2.

##  3 Proposed Method

###  3.1 Model Architecture

Our proposed model architecture consists of two branch networks and a single
RNN with a dense layer as illustrated in Figure 2. One branch network takes
log-scaled mel-spectrogram $X$ and the other branch network takes phonetic
posteriorgram (PPG) $\hat{P}$ from a pretrained phoneme classifier. Both of
the branches are CRNN where CNN architectures are a modified version of
_ConvNet_ proposed in [10], which is commonly used in the piano transcription
task [1, 11]. To get the wider time-scale receptive field, we changed the
first convolution layer with a dilated convolution with 2 dilation on the time
frame axis. To predict the note events, we combined the two branch networks by
concatenating the outputs and connecting them to an additional RNN layer and a
dense layer. The output layer is represented with a 3-dimensional sigmoid
vector where each element detects onset, offset, and activation as binary
states. The activation indicates whether the note is on or off at each frame.

###  3.2 Framewise Phoneme Classifier

We extracted the phonetic information using a phoneme classifier which returns
the output as a PPG. We implemented it using a single CRNN network with a
dense layer. We used the original _ConvNet_ architecture for the CNN part. We
tried two loss functions to train the phoneme classifier network. One is the
framewise cross entropy loss, which is possible when we have time-aligned
phoneme labels. Since it is difficult to obtain time-aligned phoneme labels in
frame-level especially for singing voice, we also used the connectionist
temporal classification (CTC) loss function [12] which can handle the
alignment between the predicted phoneme sequence ($\hat{p}$) and the ground
truth phoneme sequence ($S$) which have unequal lengths. The CTC algorithm
predicts phoneme sequences with inserted blank labels along the possible
prediction paths $\mathcal{B}$. Since the CTC loss function is optimized for
predicting the entire sequence, the prediction pattern tends to be spiky and
sparse and thus it does not find the boundaries of phonemes well [12, 13]. To
solve this problem, we used two layers of bidirectional LSTM layers and a
single dense layer that reconstruct the input log-scaled mel-spectrogram
($\hat{X}$). This was proposed to enhance the time alignment when the CTC loss
is used [14]. For the reconstruction loss ($\mathcal{L}_{\text{recon}}$), we
normalized the log-scaled mel-spectrogram from $-1$ to $1$ ($\tilde{X}$) and
applied the $\tanh$ function for the activation and used the $L_{2}$ loss
function. These loss functions are defined as:

$\displaystyle\mathcal{L}_{\text{CTC}}$
$\displaystyle=-\log\sum_{\hat{p},\mathcal{B}(\hat{p})=p}\prod_{t=0}^{T-1}\mathbb{P}(\hat{p}_{t}|X)\,,$
$\displaystyle\vspace{2mm}\mathcal{L}_{\text{recon}}$
$\displaystyle=\|\hat{X}-\tilde{X}\|^{2}\,,$ (1)
$\displaystyle\vspace{2mm}\mathcal{L}_{\text{PPG}}$
$\displaystyle=\mathcal{L}_{\text{CTC}}+\mathcal{L}_{\text{recon}}\,,$

where $T$ is the total number of time steps, $p$ is the ground truth phoneme
sequence and $\mathbb{P}(\hat{p}_{t}|X)$ is the PPG at time $t$.

###  3.3 Label Smoothing

Unlike other instruments, synthesized or auto-aligned onset/offset labels are
hardly available in the case of the singing datasets [15]. In addition, since
singing onsets are temporally soft, has a soft onset, to locate the exact
onset positions of singing by means of with a waveform or mel-spectrogram is
by no means straightforward. Such softness of the onset is one of the factors
that makes the onset of singing voices more challenging to train. Previous
frame-wise onset detection studies [6, 7] extended the duration of the onset
label to solve this problem.

Following these previous studies, we also used a smoothing method to increase
the length of the onset and offset label. Specifically, we smoothed the 1-D
one-hot onset label sequence $y_{\text{on}}:=y_{\text{on}}[n]$ ($n$ denotes
the time index) and the offset label sequence
$y_{\text{off}}:=y_{\text{off}}[n]$ through the linear convolution with a
scaled triangular window function $w_{\text{tri}}[n]$ to improve the precision
simultaneously. The scale factor of the triangular function $N$ stands for the
number of frames with nonzero values. To make the center of the label to $1$
after the smoothing, we only used the odd numbers for the scale factor $N$.
The convolution process is represented as

$\displaystyle w[n]$
$\displaystyle=\begin{cases}1-\left|\frac{n}{(N+1)/2}\right|&\text{if
$|n|\leq\frac{(N+1)}{2}$}\\\ 0&\text{otherwise.}\end{cases}$ $\displaystyle
y_{\text{on\\_s}}[n]$ $\displaystyle=y_{\text{on}}[n]\ast w_{\text{tri}}[n]$
(2) $\displaystyle y_{\text{off\\_s}}[n]$ $\displaystyle=y_{\text{off}}[n]\ast
w_{\text{tri}}[n]$

where the operation $\ast$ represents the linear convolution and $n$ is the
frame index.

###  3.4 Note Decoding

To find the positions of onsets from the prediction output, we set a constant
threshold and set the frame with the maximal value above the threshold as the
position of onset. When finding the offset of a note, we first find the offset
candidates between the current onset time and the next onset time. The offset
candidate is either the highest peak of the offset prediction or the time
frame that the activation prediction goes lower than 0.5. If multiple offset
candidates exist, we set the offset to the latest offset candidate. If no
offset candidate is found, the offset of the note is set to the time frame of
the next onset. The threshold of onset and offset is set to 0.2. In order to
determine the threshold, we evaluated the validation set using a threshold
ranging from 0.1 to 0.9 in increments of 0.1 to identify the optimal
threshold.

For note-level singing transcription, we estimated the note-level pitch from
frame-wise F0s of the note segment to find the pitch of the note, following
[6]. We extracted F0s with the PYIN algorithm [16], which is one of the most
accurate pitch trackers. To compress the F0 contour to the note-level pitch,
we used the weighted median algorithm, which finds the 50% percentile in the
ordered elements with given weights. In this experiment, we use the normalized
Hann window function with the same length of the note segment frames as the
weight of the weighted median to reduce the influence of the F0 near the
boundaries, which are the most expressive part. Since the sum of all weight
values should be one, the Hann window function is normalized by dividing by
the sum of the window elements.

##  4 Experiments

###  4.1 Datasets

We used SSVD v2.0 as the primary dataset [8]. It contains multiple sight-
singing recordings, consisting of 67 singing audio files for the train and
validation set, and 127 audio files for the test set. The human labeled
annotations include onset, offset, and averaged note pitch. To use both
phoneme and note labels given the audio, we also used the 50 songs in Korean
from the CSD dataset [17], which have both note and phoneme labels of a female
professional singer. Since the original note annotations of CSD was targeted
for singing voice synthesis, we found it needs some refinement for the note
transcription task. Thus, we re-annotated 50 songs of CSD for our experiment,
following the rule suggested by [3]. The re-annotated label of CSD can be
found on our GitHub page 111https://github.com/seyong92/CSD_reannotation. The
refined CSD is split 35, 5, and 10 songs for train, validation, and test set
each.

To train the phoneme classifier, we used TIMIT [18] which contains English
speech with time-aligned phoneme labels for the model with SSVD v2.0. TIMIT
contains 5.4 hours of audio of English speech. While training the phoneme
classifier network, we reduced the phoneme types to 39 following the CMU
pronouncing dictionary [19]. For the model with CSD, we used the unaligned
phoneme label in CSD to train.

To compare the transcription performance of the proposed model with previous
work, we also used the ISMIR2014 [3] dataset, which contains 38 songs sung by
both adults and children, as a test set.

###  4.2 Evaluation Metrics

We evaluated the models with the mir_eval library [20] for onset/offset
detection and note-level transcription. We used the metrics proposed in [3]:
F1-measure of COn (correct onset), COff (correct offset), COnOff (correct
onset and offset), COnP (correct onset and pitch), and COnPOff (Correct onset,
offset and pitch). We used the default parameters of mir_eval, which sets the
onset tolerance to 50 ms, the offset tolerance to larger value between 50 ms
and 0.2 of note duration, and the 50 cents for the pitch tolerance. Also, we
report the results when the onset/off thresholds are 100 ms considering the
softness of singing onsets.

###  4.3 Training Details

We computed 80 bin mel-spectrogram $X$ with 320 samples in hop size (20 ms)
and 1024 samples in FFT size after resampling audio files to 16 kHz. For the
modified _ConvNet_ module, we set 48/48/96 nodes to the convolutional layers
and 768 nodes to the dense layer. We used 768 nodes in all bidirectional LSTM
layers and set the last FC layer in the note onset/activation detector to have
two separate nodes for onset and activation detection, respectively. For the
label smoothing, we used a scale factor of 5 to extend the label length to 100
ms, which shows the best results in our experiment.

To train the note onset/offset detection network, we used the AdamW optimizer
[21] with a batch size of 8 and a learning rate of 1e-6. We reduced the
learning rate with a reducing factor of 0.98 for every 1000 steps. While
training, we used the random audio segment with 5 seconds. The validation set
was evaluated for every 500 steps and we stopped training when there is no
advance in the model for 10 validation steps. To train the phoneme classifier,
we used the Adam optimizer with a batch size of 16 and a learning rate of
2e-4. We reduced the learning rate with a reducing factor of 0.98 for every
900 steps. We validated the model with every 500 steps for the phoneme
classifier and trained the model while there is no advance in the model for 5
validation steps.

Fig. 3: Transition and re-onset recall of the models in the ablation study on ISMIR2014. The red triangle is the model with mel-spectrogram, the blue square is the model with PPG, and the green circle is the model with both features. Model | COn (50ms) | COn (100ms) | COff (50ms) | COff (100ms)
---|---|---|---|---
| P | R | F | P | R | F | P | R | F | P | R | F
TONY [22] | 0.7068 | 0.6326 | 0.6645 | 0.8402 | 0.7486 | 0.7877 | 0.7862 | 0.6981 | 0.7358 | 0.8405 | 0.7471 | 0.7870
Omnizart [7, 23] | 0.7797 | 0.8229 | 0.7951 | 0.8667 | 0.9153 | 0.8843 | 0.7698 | 0.8132 | 0.7852 | 0.8394 | 0.8842 | 0.8554
MusicYOLO (retrained) [8] | 0.9427 | 0.8970 | 0.9176 | 0.9711 | 0.9247 | 0.9456 | 0.8924 | 0.8504 | 0.8693 | 0.9476 | 0.9024 | 0.9227
Proposed | 0.9448 | 0.9188 | 0.9305 | 0.9652 | 0.9387 | 0.9506 | 0.8701 | 0.8473 | 0.8576 | 0.9429 | 0.9176 | 0.9290
Table 2: Onset/Offset detection results on ISMIR2014. Both of MusicYOLO and
the proposed model were trained with SSVD v2.0. Omnizart is a pretrained note
transcription model package (not with SSVD v2.0). Tony is a free, open-source
application for pitch and note transcription.

##  5 Results and Discussions

###  5.1 Ablation Study

We conducted an ablation study to see the effect of input features and model
architectures. The proposed model shown in Figure 2 corresponds to "Dual CRNNs
+ one RNN" in (g). We first compare it to a single CRNN model with only one
type of features (either mel spectrogram in (a) or PPG in (c)). Considering
that the model architecture can affect the performance, we also compared the
proposed model to the same "Dual CRNNs + one RNN" but with one type of input
features for both inputs (either mel spectrogram in (b) or PPG in (d)). Given
the proposed model, we also removed the RNN module in each CRNN branch in (e),
and then stacked another RNN module on top of (e) in (f).

Table 1 show the onset/offset detection results of all compared models. Single
CRNNs with only one input features in (a) and (c) have significantly lower
accuracy than the proposed model in (g). The gap is relatively lower when the
model was trained with CSD. Interestingly, the single CRNN model with PPG
consistently outperformed the one with mel spectrogram. The results from the
same model architecture with different input features in (b), (d), and (g)
shows that using both mel-spectrogram and PPG is more effective than using
either one of them. However, the gaps are less significant than those in the
comparison with single CRNN in (a) and (c). This indicates that model
architecture is also important to improve the performance. Likewise, the
results in (e), (f), and (g) show that the design choice of neural network
affects the performance. Since CSD is a small dataset, the proposed model have
a tendency to overfit it. Overall, the propose model in (g) shows the best
performance.

We further investigated the effect of the input features by looking into the
recall accuracy for two special types of onsets: re-onset and transition. They
are note onsets which have 20 ms or less apart from the offset of the previous
note. The difference between the two types is whether the pitch changes
(transition) or not (re-onset). The re-onset usually occurs when the syllable
in lyrics or energy changes while continuing the same pitch. Note that, since
our model does not predict the onset types, only recall accuracy can be
computed. As shown in Figure 3, the models with mel-spectrogram (in red) tend
to detect more transitions, indicating that it is more sensitive to pitch
change. On the other hand, the models with PPG (in blue) tend to detect more
re-onsets, showing that it captures phonetic changes well. Lastly, the models
with both features have more balanced accuracy in both transition and re-
onset. The demo examples, more analysis, and pre-trained models are available
on the companion website. 222https://seyong92.github.io/phoneme-informed-
transcription-blog/

###  5.2 Comparison with Prior Work

Table 2 shows the comparison with prior work on the ISMIR2014 dataset, which
has been widely used for singing voice onset/offset detection (or note
segmentation). For fair comparison, we retrained a recent state-of-the-art
model [8] with the same dataset we used for the proposed model. Our proposed
model outperforms the state-of-the-art model in onset F-score in both
tolerances while it is slightly worse in offset F-score in 50ms tolerance. The
publicly available note transcription software (TONY) and model package
(Omnizart) have significantly lower accuracy than the two models. Finally, to
see the performance for singing note transcription including pitch
information, we measured COnP and COnPOff on ISMIR2014 and SSVD v2.0 in Table
3. The results show that the proposed model achieves consistently better
performances than TONY and Omnizart.

| ISMIR2014 | SSVD v2.0
---|---|---
Model | COnP | COnPOff | COnP | COnPOff
Tony [22] | 0.6009 | 0.4621 | 0.7311 | 0.6794
Omnizart [7, 23] | 0.6174 | 0.4992 | 0.6047 | 0.5151
Proposed | 0.8975 | 0.7728 | 0.8558 | 0.8303
Table 3: Note transcription results on ISMIR2014 and SSVD v2.0. The proposed
model was trained with SSVD v2.0

##  6 Conclusion

We presented a neural network architecture for note-level singing
transcription that takes advantage of PPG on top of mel-spectrogram. Through
the ablation study, we examined various architectures along with the two input
features, showing that the additional phonetic information is effective in
singing onset/offset detection. Also, we showed that the proposed model
outperforms the compared models on ISMIR2014 and SSVD v2.0. For future work,
we plan to explore models that effectively handle weak supervision from noisy
melody and lyrics labels on a large-scaled dataset [24].

## References

  * [1] Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck,  “Onsets and frames: dual-objective piano transcription,”  in Proc. of the 19th Int. Society for Music Information Retrieval Conf., Paris, France, 2018, pp. 50–57.
  * [2] Qiuqiang Kong, Bochen Li, Xuchen Song, Yuan Wan, and Yuxuan Wang,  “High-resolution piano transcription with pedals by regressing onset and offset times,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3707–3717, October 2021.
  * [3] Emilio Molina, Ana M. Barbancho, Lorenzo J. Tardón, and Isabel Barbancho,  “Evaluation framework for automatic singing transcription,”  in Proc. of the 15th Int. Society for Music Information Retrieval Conf., Taipei, Taiwan, 2014, pp. 567–572.
  * [4] Simon Dixon,  “Onset detection revisited,”  in Proc. of the 9th Int. Conf. on Digital Audio Effects, Montréal, Canada, 2006, pp. 133–137.
  * [5] Ryo Nishikimi, Eita Nakamura, Satoru Fukayama, Masataka Goto, and Kazuyoshi Yoshii,  “Automatic singing transcription based on encoder-decoder recurrent neural networks with a weakly-supervised attention mechanism,”  in IEEE Int. Conf. on Acoustics, Speech, and Signal Processing, Brighton, UK, 2019, pp. 161–165.
  * [6] Zih-Sing Fu and Li Su,  “Hierarchical classification networks for singing voice segmentation and transcription,”  in Proc. of the 20th Int. Society for Music Information Retrieval, Delft, The Netherlands, 2019, pp. 900–907.
  * [7] Jui-Yang Hsu and Li Su,  “VOCANO: A note transcription framework for sining voice in polyphonic music,”  in Proc. of the 22nd Int. Society for Music Information Retrieval, Online, 2021, pp. 293–300.
  * [8] Xianke Wang, Wei Xu, Weiming Yang, and Wenqing Cheng,  “Musicyolo: A sight-singing onset/offset detection framework based on object detection instead of spectrum frames,”  in IEEE Int. Conf. on Acoustics, Speech, and Signal Processing, Singapore, Singapore, 2022, pp. 396–400.
  * [9] Yukun Li, Emir Edmirel, Polina Proutskova, and Simon Dixon,  “Phoneme-informed note segmentation of monophonic vocal music,”  in Proc. of the 2nd Workshop on NLP for Music and Spoken Audio (NLP4MusA), Online, 2021, pp. 17–21.
  * [10] Rainer Kelz, Matthias Dorfer, Filip Korzeniowski, Sebastian Böck, Andreas Arzt, and Gerhard Widmer,  “On the potential of simple framewise approaches to piano transcription,”  in Proc. of the 17th Int. Society for Music Information Retrieval Conf., Linz, Austria, 2016, pp. 475–481.
  * [11] Taegyun Kwon, Dasaem Jeong, and Juhan Nam,  “Polyphonic piano transcription using autoregressive multi-state note model,”  in Proc. of the 21st Int. Society for Music Information Retrieval Conf., Montréal, Canada, 2020, pp. 454–461.
  * [12] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber,  “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,”  in Proc. of the 23rd International Conference on Machine Learning, Pittsburgh, PA, 2006, pp. 369–376.
  * [13] Haşim Sak, Andrew Senior, Kanishka Rao, Ozan İrsoy, Alex Graves, Françoise Beaufays, and Johan Schalkwyk,  “Learning acoustic frame labeling for speech recognition with recurrent neural networks,”  in IEEE Int. Conf. on Acoustics, Speech, and Signal Processing, South Brisbane, QLD, Australia, 2015, pp. 4280–4284.
  * [14] Yann Teytaut and Axel Roebel,  “Phoneme-to-audio alignment with recurrent neural networks for speaking and singing voice,”  in Proc. Interspeech 2021, Brno, Czechia, 2021, pp. 61–65.
  * [15] Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck,  “Enabling factorized piano music modeling and generation with the maestro dataset,”  in The Int. Conf. on Learning Representations, New Orleans, LA, USA, 2019.
  * [16] Matthias Mauch and Simon Dixon,  “PYIN: A fundamental frequency estimator using probabilistic threshold distributions,”  in IEEE Int. Conf. on Acoustics, Speech, and Signal Processing, Florence, Italy, 2014, pp. 659–663.
  * [17] Soonbeom Choi, Wonil Kim, Saebyul Park, Sangeon Yong, and Juhan Nam,  “Children’s song dataset for singing voice research,”  in ISMIR Late Breaking and Demo Papers, Montréal, Canada, 2020\.
  * [18] John S. Garofolo, Lori F. Lamel, William M. Fisher, Jonathan G. Fiscus, David S. Pallett, Nancy L. Dahlgren, and Victor Zue,  “TIMIT acoustic-phonetic continuous speech corpus,”  LDC93S1, Philadelphia: Linguistic Data Consortium, 1993.
  * [19] “The cmu pronouncing dictionary,” http://www.speech.cs.cmu.edu/cgi-bin/cmudict, accessed 2022-10-22.
  * [20] Colin Raffel, Brian McFee, Eric J. Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, and Daniel P. W. Ellis,  “mir_eval: A transparent implementation of common MIR metrics,”  in Proc. of the 15th Int. Society for Music Information Retrieval, Taipei, Taiwan, 2014, pp. 367–372.
  * [21] Ilya Loshchilov and Frank Hutter,  “Decoupled weight decay regularization,”  in The Int. Conf. on Learning Representations, New Orleans, LA, USA, 2019.
  * [22] Matthias Mauch, Chris Cannam, Rachel Bittner, Geroge Fazekas, Justin Salamon, Jiajie Dai, Juan Bello, and Simon Dixon,  “Computer-aided melody note transcription using the tony software: Accuracy and efficiency,”  in Proc. of Sound and Music Computing, Maynooth, Ireland, 2015.
  * [23] Yu-Te Wu, Yin-Jyun Luo, Tsung-Ping Chen, I-Chieh Wei, Jui-Yang Hsu, Yi-Chin Chuang, and Li Su,  “Omnizart: A general toolbox for automatic music transcription,”  Journal of Open Source Software, vol. 6, no. 68, pp. 3391, Dec 2021\.
  * [24] Gabriel Meseguer-Brocal, Alice Cohen-Hadria, and Geoffroy Peeters,  “DALI: A large dataset of synchronized audio, lyrics, and notes, automatically created using teacher-student machine learning paradigm,”  in Proc. of the 19th Int. Society for Music Information Retrieval, Paris, France, 2018, pp. 431–437.


#  Learning Constraints and Descriptive Segmentation for Subevent Detection

Haoyu Wang1, Hongming Zhang2, Muhao Chen3 & Dan Roth1
1Department of Computer and Information Science, UPenn
2Department of Computer Science and Engineering, HKUST
3Department of Computer Science, USC
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
This work was done when the author was visiting the University of
Pennsylvania.

###### Abstract

Event mentions in text correspond to real-world events of varying degrees of
granularity. The task of subevent detection aims to resolve this granularity
issue, recognizing the membership of multi-granular events in event complexes.
Since knowing the span of descriptive contexts of event complexes helps infer
the membership of events, we propose the task of _event-based text
segmentation_ (EventSeg) as an auxiliary task to improve the learning for
subevent detection. To bridge the two tasks together, we propose an approach
to learning and enforcing constraints that capture dependencies between
subevent detection and EventSeg prediction, as well as guiding the model to
make globally consistent inference. Specifically, we adopt Rectifier Networks
for constraint learning and then convert the learned constraints to a
regularization term in the loss function of the neural model. Experimental
results show that the proposed method outperforms baseline methods by 2.3% and
2.5% on benchmark datasets for subevent detection, HiEve and IC, respectively,
while achieving a decent performance on EventSeg prediction111Our code is
publicly available at http://cogcomp.org/page/publication_view/950..

##  1 Introduction

Since real-world events are frequently conveyed in human languages,
understanding their linguistic counterparts, i.e. event mentions in text, is
of vital importance to natural language understanding (NLU). One key challenge
to understanding event mentions is that they refer to real-world events with
varied granularity Glavaš et al. (2014) and form _event complexes_ Wang et al.
(2020). For example, when speaking of a coarse-grained event “publishing a
paper”, it can involve a complex of more fine-grained events such as “writing
the paper,” “passing the peer review,” and “presenting at the conference.”
Naturally, understanding events requires to resolve the granularity of events
and infer their memberships, which corresponds to the task of subevent
detection (a.k.a. event hierarchy extraction). Practically, subevent detection
is a key component of event-centric NLU Chen et al. (2021), and is beneficial
to various applications, such as schema induction Zhang et al. (2020); Li et
al. (2020a), task-oriented dialogue agents Andreas et al. (2020),
summarization Chen et al. (2019); Zhao et al. (2020), and risk detection Pohl
et al. (2012).

Figure 1: An example of Parent-Child relations and EventSegs from the HiEve
dataset Glavaš et al. (2014). The blue and yellow segments denote the textual
spans of event complexes “posted” and “scandal” respectively. Curved arrows
denote Parent-Child relations within a text segment, whereas the dotted arrows
denote cross-segment Parent-Child relations.

As a significant step towards inducing event complexes (graphs that recognize
the relationship of multi-granular events) in documents, subevent detection
has started to receive attention recently Wang et al. (2020); Han et al.
(2021). It is natural to perceive that in documents, there might be several
different event complexes and they often span in different descriptive
contexts that form relatively independent text segments. Consider the example
in Figure 1, where the two membership relations in the event complex (graph
consisting of “scandal (e7),” “charges (e6),” “ousting (e8),” and relations)
are both within the segment marked in yellow that describes the event complex.
As can be seen in the paragraph, though we cannot deny the existence of cross-
segment subevent relations (dotted arrows), events belonging to the same
membership are much more often to co-occur in a text segment. This correlation
has been overlooked by existing data-driven methods Zhou et al. (2020); Yao et
al. (2020), which formulate subevent detection as pairwise relation
extraction. On the other hand, while prior studies have demonstrated the
benefits of incorporating logical constraints among event memberships and
other relations (such as co-reference) Glavaš and Šnajder (2014); Wang et al.
(2020), the constraints between the memberships and event co-occurences in
text segments remain uncertain. Hence, how to effectively learn and enforce
hard-to-articulate constraints as in the case of subevent detection and
segmentation of text is another challenge.

Our _first_ contribution is to improve subevent detection based on an
auxiliary task of EventSeg prediction. By EventSeg prediction, we seek to
segment a document into descriptive contexts of different event complexes.
Evidently, with EventSeg information, it would be relatively easy to infer the
memberships of events in the same descriptive context. Using annotations for
subevent detection and EventSeg prediction, we aim to adopt a neural model to
jointly learn these two tasks along with the (soft) logical constraints that
bridge their labels together. In this way, we incorporate linear discourse
structure of segments into membership relation extraction, avoiding
complicated feature engineering in the previous work Aldawsari and Finlayson
(2019). From the learning perspective, adding EventSeg prediction as an
auxiliary task seeks to provide effective incidental supervision signals Roth
(2017) to the subevent detection task. This is especially important in the
current scenario where annotated learning resources for subevents are
typically limited Hovy et al. (2013); Glavaš et al. (2014); O’Gorman et al.
(2016).

To capture the logical dependency between subevent structure and EventSeg, our
_second_ contribution is an approach to automatically learning and enforcing
logical constraints. Motivated by Pan et al. (2020), we use Rectifier Networks
to learn constraints in the form of linear inequalities, and then convert the
constraints to a regularization term that can be incorporated into the loss
function of the neural model. This allows any hard-to-articulate constraints
to be automatically captured for interrelated tasks, and efficiently guides
the model to make globally consistent inference. By learning and enforcing
task-specific constraints for subevent relations, the proposed method achieves
comparable results with SOTA subevent detection methods on the HiEve and IC
dataset. Moreover, by jointly learning with EventSeg prediction, the proposed
method surpasses previous methods on subevent detection by relatively 2.3% and
2.5% in $F_{1}$ on HiEve and IC, while achieving decent results on EventSeg
prediction.

##  2 Related Work

We discuss three lines of relevent research.

Subevent Detection. Several approaches to extracting membership relations have
been proposed, which mainly fall into two categories: statistical learning
methods and data-driven methods. Statistical learning methods Glavaš et al.
(2014); Glavaš and Šnajder (2014); Araki et al. (2014); Aldawsari and
Finlayson (2019) collect a variety of features before feeding into classifiers
for pairwise decision. Nevertheless the features often require costly human
effort to obtain, and are often dataset-specific. Data-driven methods, on the
other hand, automatically characterize events with neural language models like
BERT Devlin et al. (2019), and can simultanously incorporate various signals
such as event time duration Zhou et al. (2020), joint constraints with event
temporal relations Wang et al. (2020) and subevent knowledge Yao et al.
(2020). Among recent methods, only Aldawsari and Finlayson (2019) utilize
discourse features like discourse relations between elementary discourse
units, but still document-level segmentation signals are not incorporated into
the task of subevent detection. Actually, research on event-centric NLU Chen
et al. (2021) has witnessed the usage of document-level discourse relations:
different functional discourse structures around the main event in news
articles have been studied in Choubey et al. (2020). Hence, we attempt to
capture the interdependencies between subevent detection and segmentation of
text, in order to enhance the model performance for event hierarchy
extraction.

Text Segmentation. Early studies in this line have concentrated on
unsupervised text segmentation, quantifying lexical cohesion within small text
segments Choi (2000), and unsupervised Bayesian approaches have also been
successful in this task Eisenstein and Barzilay (2008); Eisenstein (2009);
Newman et al. (2012); Mota et al. (2019). Given that unsupervised algorithms
are difficult to specialize for a particular domain, Koshorek et al. (2018)
formulate the problem as a supervised learning task. Lukasik et al. (2020)
follow this idea by using transformer-based architectures with cross segment
attention to achieve state-of-the-art performance. Focusing on creating
logically coherent sub-document units, these prior work do not cover
segmentation of text regarding descriptive contexts of event complexes, which
is the focus of the auxiliary task in this work.

Learning with Constraints. In terms of enforcing declarative constraints in
neural models, early efforts Roth and Yih (2004); Glavaš and Šnajder (2014)
formulate the inference process as Integer Linear Programming (ILP) problems.
Pan et al. (2020) also employ ILP to enforce constraints learned automatically
from Rectifier Networks with strong expressiveness Pan and Srikumar (2016).
Yet the main drawback of solving an ILP problem is its inefficiency in a large
feasible solution space. Recent work on integrating neural networks with
structured outputs has emphasized the importance of the interaction between
constraints and representations Rocktäschel and Riedel (2017); Niculae et al.
(2018); Li and Srikumar (2019); Li et al. (2019, 2020b). However there has
been no automatic and efficient ways to learn and enforce constraints that are
not limited to first-order logic, e.g., linear inequalities learned via
Rectifier Networks. And this is the research focus of our paper.

##  3 Preliminaries

A document $\mathcal{D}$ consists of a collection of $m$ sentences
$\mathcal{D}=[s_{1},s_{2},\cdots,s_{m}]$, and each sentence, say $s_{k}$,
contains a sequence of tokens $s_{k}=[w_{1},w_{2},\cdots,w_{n}]$. Some tokens
in sentences belong to the set of annotated event triggers, i.e.,
$\mathcal{E}_{\mathcal{D}}=\\{e_{1},e_{2},\cdots,e_{l}\\}$. Following the
notation by Koshorek et al. (2018), a segmentation of document $\mathcal{D}$
is represented as a sequence of binary values:
$\mathcal{Q}_{\mathcal{D}}=\\{q_{1},q_{2},\cdots,q_{m-1}\\}$, where $q_{i}$
indicates whether sentence $s_{i}$ is the end of a segment.

Subevent Detection is to identify membership relations between events, given
event mentions in documents. Particularly, $\mathcal{R}$ denotes the set of
relation labels as defined in Hovy et al. (2013) and Glavaš et al. (2014)
(i.e., Parent-Child, Child-Parent, Coref, and NoRel). For a relation
$r\in\mathcal{R}$, we use a binary indicator $Y_{i,j}^{r}$ to denote whether
an event pair $(e_{i},e_{j})$ has relation $r$, and use $y_{i,j}^{r}$ to
denote the model-predicted possibility of an event pair $(e_{i},e_{j})$ to
have relation $r$.

EventSeg prediction aims at finding an optimal segmentation of text that
breaks the document into several groups of consecutive sentences, and each
sequence is a descriptive context of an event complex Wang et al. (2020).
Being different from the traditional definition of text segmentation, EventSeg
focuses on the change of event complex (which is not necessarily the change of
topic). For a pair of events $(e_{i},e_{j})$, we use a binary indicator
$Z_{i,j}$ to denote whether the two events are within the same descriptive
context of event complexes, and $z_{i,j}$ to denote the model-predicted
possibility of two events to belong to the same segment. Details on how to
obtain EventSeg are described in Table 1.

Connections between Two Tasks. Statistically, through an analysis of the HiEve
and IC corpus, Parent-Child and Child-Parent relations appear within the same
descriptive context of event complex with a probability of 65.13% (see Table
1). On the other hand, the probability for each of the two other non-
membership relations (i.e., Coref and NoRel) to appear within the same segment
approximately equals that of its appearence across segments. This demonstrates
that subevent relations tend to appear within the same EventSeg. Since this is
not an absolute logical constraint, we adopt an automatic way of modeling such
constraints instead of manually inducing them, which is described in the next
section.

##  4 Methods

We now present the framework for learning and enforcing constraints for the
main task of subevent detection and the auxiliary EventSeg prediction. We
start with learning the hard-to-articulate constraints (Section 4.1), followed
by details of joint learning (Section 4.2) and inference (Section 4.3) for the
two tasks.

###  4.1 Learning Constraints

From the example shown in Figure 1 we can construct an event graph $G$ with
all the events, membership relations, and EventSeg information. Figure 2 shows
a three-event subgraph of $G$. The goal of constraint learning is as follows:
given membership relations $Y_{i,j}^{r},Y_{j,k}^{r}$ and segmentation
information $Z_{i,j},Z_{j,k}$ about event pairs $(e_{i},e_{j})$ and
$(e_{j},e_{k})$, we would like to determine whether a certain assignment of
$Y_{i,k}^{r},$ and $Z_{i,k}$ is legitimate.

Feature Space for Constraints. We now define the feature space for constraint
learning. Let $\mathbf{X}_{p}=\\{Y_{p}^{r},r\in\mathcal{R}\\}\cup\\{Z_{p}\\}$
denote the set of features for an event pair $p$. Given features
$\mathbf{X}_{i,j}$ and $\mathbf{X}_{j,k}$, we would like to determine the
value of $\mathbf{X}_{i,k}$, yet the mapping from the labels of
$(e_{i},e_{j}),(e_{j},e_{k})$ to the labels of $(e_{i},e_{k})$ is a one-to-
many relationship. For instance, if $r=$ Parent-Child,
$Y_{i,j}^{r}=Y_{j,k}^{r}=1$, and $Z_{i,j}=Z_{j,k}=0$, then due to the
transitivity of Parent-Child, we should enforce $Y_{i,k}^{r}=1$. Yet we cannot
tell whether $e_{i}$ and $e_{k}$ are in the same EventSeg, i.e., both
$Z_{i,k}=1$ and $Z_{i,k}=0$ could be legitimate. In other words, we actually
want to determine the _set of possible values_ of $\mathbf{X}_{i,k}$ and thus
we need to expand the constraint features to better capture relationship
legitimacy. We employ the _power set_ of $\mathbf{X}_{i,k}$,
$\mathcal{P}(\mathbf{X}_{i,k})$, as our new features for event pair
$(e_{i},e_{k})$. And now a subgraph with three events $e_{i}$, $e_{j}$, and
$e_{k}$ can be featurized as

$\mathbf{X}=\mathbf{X}_{i,j}\cup\mathbf{X}_{j,k}\cup\mathcal{P}(\mathbf{X}_{i,k}).$
(1)

Constraint Learning with Rectifier Network. When we construct three-event
subgraphs from documents, a binary label $t$ for structure legitimacy is
created for each subgraph. Inspired by how constraints are learned for several
structured prediction tasks Pan et al. (2020), we represent constraints for a
given subgraph-label pair $(\mathbf{X},t)$ as $K$ linear inequalities.222Here
we assume $K$ constraints is the upper bound for all the rules to be learned.
Formally, $t=1$ if $\mathbf{X}$ satisfies constraints $c_{k}$ for all
$k=1,\cdots,K$. And the $k^{\text{th}}$ constraint $c_{k}$ is expressed by a
linear inequality

$\displaystyle\begin{split}\mathbf{w}_{k}\cdot\mathbf{X}+b_{k}\geq
0,\end{split}$

whose weights $\mathbf{w}_{k}$ and bias $b_{k}$ are learned. Since a system of
linear inequalities is proved to be equivalent to the Rectifier Network
proposed in Pan et al. (2020), we adopt a two-layer rectifier network for
learning constraints

$p=\sigma\Big{(}1-\sum_{k=1}^{K}\operatorname{ReLU}\big{(}\mathbf{w}_{k}\cdot\mathbf{X}+b_{k}\big{)}\Big{)},$
(2)

where $p$ denotes the possibility of $t=1$ and $\sigma(\cdot)$ denotes the
sigmoid function. We train the parameters $\mathbf{w}_{k}$’s and $b_{k}$’s of
the rectifier network in a supervised setting. The positive examples are
induced from subgraph structures that appear in the training corpus, while the
negative examples are randomly chosen from the rest possibilities that do not
exist in the training corpus.

Figure 2: A legitimate structure for three-event subgraph obtained from the
example shown in Figure 1. The constraint features for the subgraph can be
expressed by
$\mathbf{X}=\mathbf{X}_{7,6}\cup\mathbf{X}_{6,2}\cup\mathcal{P}(\mathbf{X}_{7,2})$,
and the label $t$ for this structure is 1.

###  4.2 Joint Task Learning

After learning the constraints using Rectifier Networks, we introduce how to
jointly model membership relations and EventSeg with neural networks and how
to integrate the learned constraints into the model. The model architecture is
shown in Figure 3.

Figure 3: An overview of our approach. The model takes three pairs of events
at a time in training to enforce constraints over three-event subgraphs (an
example can be found in Figure 2). Event pair representations are obtained
from RoBERTa where the context of two events are taken into consideration.
Soft logical constraints learned in Section 4.1 are converted to a
regularization term in the loss function for subgraph structure legitimacy.

Local Classifier. To characterize event pairs in documents, we employ a neural
encoder, which obtains contextualized representations for event triggers from
the pre-trained transformer-based language model RoBERTa Liu et al. (2019). As
the context of event pairs, the sentences where two event mentions appear are
concatenated using [CLS] and [SEP]. We then calculate the element-wise average
of subword-level contextual representations as the representation for each
event trigger. To obtain event pair representation for $(e_{i},e_{j})$, we
concatenate the two contextual representations, together with their element-
wise Hadamard product and subtraction as in Wang et al. (2020). The event pair
representation is then sent to a multi-layer perceptron (MLP) with
$|\mathcal{R}|$ outputs for estimation of the confidence score $y_{i,j}^{r}$
for each relation $r$. To make EventSeg as an auxiliary task, the model also
predicts whether two events belong to the same segment using another separate
MLP with a single-value output $z_{i,j}$. In accordance with the learned
constraints in Section 4.1, the model takes three pairs of events at a time.
The annotation loss in Figure 3 is a linear combination of a four-class cross-
entropy loss $L_{A,sub}$ for subevent detection and a binary cross-entropy
loss $L_{A,seg}$ for EventSeg.

Incorporating Subgraph Constraints. The $K$ constraints learned in Section 4.1
are encoded into the weights $\mathbf{w}_{k}$ and bias $b_{k}$,
$k=1,\cdots,K$. Now that the input $\mathbf{X}$ is considered valid if it
satisfies all $K$ constraints, we obtain the predicted probability $p$ of
$\mathbf{X}$ being valid from Equation 2. To add the constraints as a
regularization term in the loss function of the neural model, we convert $p$
into the negative log space Li et al. (2019) which is same as the cross-
entropy loss. And thus the loss corresponding to the learned constraints is

$\displaystyle\begin{split}L_{cons}=-log\Big{(}Sigmoid\big{(}1-\sum_{k=1}^{N}ReLU(\mathbf{w}_{k}\cdot\bm{\psi}+b_{k})\big{)}\Big{)}.\end{split}$

And the loss function of the neural model is

$L=\lambda_{1}L_{A,sub}+\lambda_{2}L_{A,seg}+\lambda_{3}L_{cons},$ (3)

where the $\lambda$’s are non-negative coefficients to control the influence
of each loss term. With the loss function in Equation 3, we train the model in
a supervised way to fine-tune RoBERTa.

###  4.3 Inference

At inference time, to extract relations in the subevent detection task, we
input a pair of events into the model and compare the predicted possibility
for each relation, leaving the other two input pairs blank. For EventSeg
prediction, we let the model predict $z_{i,i+1}$ for each pair of adjacent
events $(e_{i},e_{i+1})$ that appear in different sentences. If $z_{i,i+1}=1$,
it means there is a segment break between $e_{i}$ and $e_{i+1}$. When there
are intermediate sentences between the two adjacent event mentions, we treat
the sentence that contains $e_{i}$ as the end of a previous segment. In this
way, we provide an approach to solving two tasks together via automatically
learning and enforcing constraints in the neural model. We provide in-depth
experimentation for the proposed method in the next section.

##  5 Experiments

Here we describe the experiments on subevent detection with EventSeg
prediction as an auxiliary task. We first introduce the corpora used (Section
5.1), followed by evaluation for subevent detection and an ablation study for
illustrating the importance of each model component (Section 5.2-Section 5.4).
We also provide a case study on EventSeg prediction (Section 5.5) and an
analysis of the constraints learned in the model (Section 5.6).

###  5.1 Datasets

Relations | HiEve | IC
---|---|---
Within | Across | Within | Across
Parent-Child | 1,123 | 679 | 1,698 | 550
Child-Parent | 1,067 | 779 | 1,475 | 863
Coref | 322 | 436 | 1,476 | 877
NoRel | 32,029 | 31,726 | 40,072 | 41,815

Table 1: Statistics of the HiEve and IC dataset. Numbers in column “Within”
denote the number of relations appearing within the same descriptive context
of event complex, whereas numbers under “Across” denote those across different
segments.

HiEve The HiEve corpus Glavaš et al. (2014) contains 100 news articles. Within
each article, annotations are given for both subevent membership and
coreference relations. Using the same measurement of inter-annotator agreement
(IAA) as event temporal relations in UzZaman and Allen (2011), the HiEve
dataset has an IAA of 0.69 F1.

Intelligence Community (IC) The IC corpus Hovy et al. (2013) also contains 100
news articles annotated with membership relations. The articles report
violence events such as attack, war, etc. We discard those relations involving
implicit events annotated in IC, and calculate transitive closure for both
subevent relations and co-reference to get annotations for all event pairs in
text order as it is done for HiEve Glavaš et al. (2014).

Labeling EventSeg We explain how to segment the document using annotations for
subevent relations. First, we use the annotated subevent relations (Parent-
Child and Child-Parent only) to construct a directed acyclic event graph for
each document. Due to the property of subevent relations, each connected
component in the graph is actually a tree with one root node, which forms an
event complex. If the graph constructed from document has one connected
component, we remove the root node and separate the event graph into more than
one event complexes. Since each event complex has a textual span in the
document, we obtain several descriptive contexts that may or may not overlap
with each other. For those documents with non-overlapping descriptive
contexts, their segmentations are therefore obtained. In cases where two
descriptive contexts of event complexes overlap with each other, if there
exists such an event whose removal results in non-overlapping contexts, then
we segment the contexts assuming this event is not considered. Otherwise, we
merge the contexts into one segment. Through this event-based text
segmentation, on average we obtain 3.99 and 4.29 EventSegs in the HiEve and IC
corpus, respectively.

We summarize the data statistics in Table 1.

###  5.2 Baselines and Evaluation Protocols

On IC dataset, we compare with two baseline approaches. Araki et al. (2014)
propose a logistic regression model along with a voting algorithm for parent
event detection. Wang et al. (2020) use a data-driven model that incorporates
handcrafted constraints with event temporal attributes to extract event-event
relations. On Hieve333Despite carefully following the details described in
Aldawsari and Finlayson (2019) and communicating with the authors, we were not
able to reproduce their results. Therefore, we choose to compare with other
methods., we compare with a transformer-based language model TacoLM Zhou et
al. (2020) that fine-tunes on a temporal common sense corpora, and the method
proposed by Wang et al. (2020) which also serves as the second baseline for
IC.

We use the same evaluation metric on HiEve as previous methods Zhou et al.
(2020), leaving 20% of the documents out for testing444To make predictions on
event complexes, we keep all negative NoRel instances in our experiments
instead of strictly following Zhou et al. (2020) and Wang et al. (2020) where
negative instances are down-sampled with a probability of 0.4.. The $F_{1}$
scores of Parent-Child and Child-Parent and the micro-average of them are
reported. In accordance with HiEve, the IC dataset is also evaluated with
$F_{1}$ scores of membership relations instead of BLANC Araki et al. (2014),
while the other settings remain the same with previous works.

###  5.3 Experimental Setup

We fine-tune the pre-trained 1024 dimensional RoBERTa Liu et al. (2019) to
obtain contextual representations of event triggers in a supervised way given
labels for membership relations and EventSeg. Additionally, we employ 18
dimensional one-hot vectors for part-of-speech tags for tokens in documents to
include explicit syntactic features in the model. For each MLP we set the
dimension to the average of the input and output neurons, following Chen et
al. (2018). The parameters of the model are optimized using AMSGrad Reddi et
al. (2018), with the learning rate set to $10^{-6}$. The training process is
limited to 40 epochs since it is sufficient for convergence.

###  5.4 Results

We report the results for subevent detection on two benchmark datasets, HiEve
and IC, in Table 2. Among the baseline methods, Wang et al. (2020) has the
best results in terms of $F_{1}$ on both datasets. They integrate event
temporal relation extraction, common sense knowledge and handcrafted logical
constraints into their approach. In contrast, our proposed method does not
require constraints induced by domain experts, but still outperforms their
$F_{1}$ score by 2.3 - 2.5%. We attribute this superiority to the use of
connections between subevent relations and the linear discourse structure of
segments. Thanks to the strong expressiveness of Rectifier Networks, we
utilize these connections via the learning of linear constraints, thus
incorporating incidental supervision signal from EventSeg. Furthermore, the
event pair representation in our model is obtained from broader contexts than
the local sentence-level contexts for events in Wang et al. (2020). The new
representation not only contains more information on events but naturally
provides necessary clues for determining whether there is a break for
EventSeg.

|  |  $F_{1}$ score
---|---|---
Corpus | Model | PC | CP | Avg.
IC | Araki et al. (2014) | - | - | 0.262
Wang et al. (2020) | 0.421 | 0.495 | 0.458
Our model | 0.446 | 0.516 | 0.481
HiEve | Zhou et al. (2020) | 0.485 | 0.494 | 0.489
Wang et al. (2020) | 0.472 | 0.524 | 0.497
Our model | 0.534 | 0.510 | 0.522

Table 2: Experimental results for subevent detection on IC and HiEve corpus.
PC, CP and Avg. denote Parent-Child, Child-Parent and their micro-average,
respectively. $F_{1}$ scores for PC and CP are not reported in Araki et al.
(2014).

We further perform an ablation analysis to aid the understanding of the model
components and report our findings in Table 3. Without any constraints,
integrating EventSeg prediction as an auxiliary task brings along an absolute
gain of 0.2% and 0.6% in $F_{1}$ on HiEve and IC respectively over the vanilla
single-task model with RoBERTa fine-tuning. This indicates that EventSeg
information is beneficial to the extraction of membership relations. When
membership constraints are added via the regularization term into the loss
function, the model’s performance on subevent detection is significantly
improved by 2.1% in $F_{1}$ on HiEve dataset. Incorporating constraints
involving two tasks further enhances the model performance by 0.5% - 1.1%.
This indicates that the global consistency ensured within and across EventSegs
is important for enhancing the comprehension for subevent memberships.

| HiEve | IC
---|---|---
Model | $P$ | $R$ | $F_{1}$ | $P$ | $R$ | $F_{1}$
Single-task Training | 43.9 | 56.6 | 49.4 | 44.5 | 46.9 | 45.8
Joint Training | 45.7 | 54.2 | 49.6 | 39.9 | 56.5 | 46.4
\+ Membership Constraints | 55.6 | 48.5 | 51.7 | 50.1 | 45.8 | 47.0
\+ Membership + EventSeg | 51.9 | 53.6 | 52.2 | 39.6 | 64.0 | 48.1
Table 3: Ablation study results for subevent detection. The results on both
datasets are the micro-average of Parent-Child and Child-Parent in terms of
precision, recall, and $F_{1}$. “+ Membership Constraints” denotes adding
automatically learned constraints for membership relations upon the joint
training model. The row of “+ Membership + EventSeg” shows the results of the
complete model.

###  5.5 Case Study for EventSeg Prediction

Here we provide an analysis of model performance on the task of EventSeg
prediction. Though EventSeg prediction is somewhat different from text
segmentation in concept, we can use methods for text segmentation as baselines
for EventSeg prediction. We train a recent BERT-based model Lukasik et al.
(2020) for text segmentation based on annotations for EventSeg in the HiEve
and IC corpora and compare our method with this baseline. In Table 4 we show
the performances of the baseline model and ours for EventSeg prediction in
terms of $F_{1}$ on HiEve and IC. Since our solution for EventSeg prediction
is essentially similar to the cross-segment BERT model in terms of
representations of segments, our performance is on par with the baseline
model.

Model | HiEve | IC
---|---|---
Cross-segment BERT Lukasik et al. (2020) | 55.2 | 58.3
Our model | 56.8 | 57.4

Table 4: EventSeg prediction performance in terms of $F_{1}$ on the HiEve and
IC corpus.

###  5.6 Analysis on Constraint Learning

We further provide an in-depth qualitative analysis on different types of
logical constraints captured by the constraint learning.

####  5.6.1 Types of Learned Constraints

We expect that both task-specific constraints (membership relations only) in
previous works Glavaš and Šnajder (2014); Wang et al. (2020) and cross-task
constraints can be automatically captured in our framework. Accordingly, we
separately analyze these two constraints.

Task-specific Constraints. Since we are using three-event subgraph for
constraint learning, apparently, transitivity constraints for membership
relations like

$\begin{split}Y_{i,j}^{r}+Y_{j,k}^{r}&-Y_{i,k}^{r}\leq 1,\\\
r\in\\{\textsc{Parent-Child},&\textsc{Child-
Parent},\textsc{Coref}\\},\end{split}$

can be learned; whereas constraints that typically involve two events, e.g.,
symmetry constraints for membership relations like

$\begin{split}Y_{i,j}^{r}=&Y_{j,i}^{\bar{r}},\\\ r\in\\{\textsc{Parent-
Child}&,\textsc{Child-Parent}\\},\end{split}$

can also be learned by assigning the third event $e_{k}$ to the same event as
$e_{i}$ and treating the relation of $(e_{i},e_{k})$ as Coref.

Cross-task Constraints. Here we provide an analysis of cross-task constraints
for both membership relations and EventSeg information learned in the model.
We give an example constraint in the form of linear inequality learned from
HiEve

$\displaystyle\begin{split}&0.13x_{0}+0.19x_{1}+0.27x_{2}+0.08x_{3}-0.18x_{4}\\\
+&0.09x_{5}+0.13x_{6}+0.25x_{7}+0.04x_{8}-0.18x_{9}\\\
+&\cdots+0.02x_{18}+0.07x_{19}+\cdots+0.05\geq 0,\end{split}$

where $x_{1}$ and $x_{6}$ denote the variables for $Y_{i,j}^{r}=1$ and
$Y_{j,k}^{r}=1$ ($r=$ Child-Parent) respectively, and they both have positive
coefficients. If we look at expected labels for
$\mathcal{P}(\mathbf{X}_{i,k})$, we can see that $x_{18}$ and $x_{19}$ which
denote the variables for $Y_{i,k}^{r}=1,Z_{i,k}=0$ and
$Y_{i,k}^{r}=1,Z_{i,k}=1$ have coefficients of 0.02 and 0.07, respectively.
The two positive coefficients for $x_{18}$ and $x_{19}$ indicate that (a)
$(e_{i},e_{k})$ is possible to have Child-Parent relation, and (b) the
possibility of $(e_{i},e_{k})$ being in the same EventSeg is greater than two
events being in different EventSegs.

####  5.6.2 Qualitative Analysis

We set $K$ to 10 since we observe less number of constraints will decrease the
performance of learning accuracy while increasing $K$ does not cause
noticeable influence. We optimize the parameters using Adam with a learning
rate of 0.001 and the training process is limited to 1,000 epochs. We show the
performance of constraint learning in Table 5. Since the constraints for
membership relations should be declarative hard constraints like symmetry and
transitivity constraints in Section 5.6.1, the accuracy of constraint learning
is equal or close to 100%. Yet, those hard-to-articulate constraints that
incorporate EventSeg information are more difficult to learn, and thus the
Rectifier Network has a less satisfying performance in terms of accuracy on
the test set of HiEve and IC (96.44% and 98.01%).

Constraints | HiEve | IC
---|---|---
Membership | 99.13 | 100.00
Membership + EventSeg | 96.44 | 98.01
Table 5: Constraint learning performance in terms of accuracy on test set.
“Membership” denotes the constraints involving membership relations only,
while “Membership + EventSeg” denotes full constraints.

##  6 Conclusion

In this work we propose an automatic and efficient way of learning and
enforcing constraints for subevent detection. By noticing the connections
between subevent dection and EventSeg, we adopt EventSeg prediction as an
auxiliary task which provides effective incidental supervision signals.
Through learning and enforcing constraints that can express hard-to-articulate
constraints, the logical rules for both tasks are captured to regularize the
model towards consistent inference. The proposed approach outperforms SOTA
data-driven methods on benchmark datasets and provides comparable results with
recent text segmentation methods on EventSeg prediction. This demonstrates the
effectiveness of the framework on subevent detection and the potential of
solving other structured predictions tasks in NLP.

## Ethical Considerations

This work does not present any direct societal consequence. The proposed
method aims at supporting high-quality extraction of event complexes from
documents with the awareness of discourse structures and automated constraint
learning. We believe this study leads to intellectual merits of developing
robust event-centric information extraction technologies. It also has broad
impacts, since constraints and dependencies can be broadly investigated for
label structures in various natural language classification tasks. The
acquired eventually knowledge, on the other hand, can potentially benefit
various downstream NLU and NLG tasks.

For any information extraction methods, real-world open source articles to
extract information from may include societal biases. Extracting event
complexes from articles with such biases may potentially propagate the bias
into acquired knowledge representation. While not specifically addressed in
this work, the ability to incorporate logical constraints and discourse
consistency can be a way to mitigate societal biases.

## Acknowledgement

We appreciate the anonymous reviewers for their insightful comments.

This research is supported by the Office of the Director of National
Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA),
via IARPA Contract No. 2019-19051600006 under the BETTER Program, and by
contract FA8750-19-2-1004 with the US Defense Advanced Research Projects
Agency (DARPA). The views expressed are those of the authors and do not
reflect the official policy or position of the Department of Defense or the
U.S. Government.

## References

  * Aldawsari and Finlayson (2019) Mohammed Aldawsari and Mark Finlayson. 2019.  Detecting subevents using discourse and narrative features.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4780–4790, Florence, Italy. Association for Computational Linguistics.
  * Andreas et al. (2020) Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, et al. 2020.  Task-oriented dialogue as dataflow synthesis.  _Transactions of the Association for Computational Linguistics_ , 8:556–571.
  * Araki et al. (2014) Jun Araki, Zhengzhong Liu, Eduard Hovy, and Teruko Mitamura. 2014.  Detecting subevent structure for event coreference resolution.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Chen et al. (2018) Muhao Chen, Changping Meng, Gang Huang, and Carlo Zaniolo. 2018.  Neural article pair modeling for wikipedia sub-article matching.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pages 3–19. Springer.
  * Chen et al. (2021) Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji, Kathleen McKeown, and Dan Roth. 2021.  Event-centric natural language understanding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics – Tutorials_.
  * Chen et al. (2019) Xiuying Chen, Zhangming Chan, Shen Gao, Meng-Hsuan Yu, Dongyan Zhao, and Rui Yan. 2019.  Learning towards abstractive timeline summarization.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ , pages 4939–4945. International Joint Conferences on Artificial Intelligence Organization.
  * Choi (2000) Freddy Y. Y. Choi. 2000.  Advances in domain independent linear text segmentation.  In _1st Meeting of the North American Chapter of the Association for Computational Linguistics_.
  * Choubey et al. (2020) Prafulla Kumar Choubey, Aaron Lee, Ruihong Huang, and Lu Wang. 2020.  Discourse as a function of event: Profiling discourse structure in news articles around the main event.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5374–5386, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Eisenstein (2009) Jacob Eisenstein. 2009.  Hierarchical text segmentation from multi-scale lexical cohesion.  In _Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 353–361, Boulder, Colorado. Association for Computational Linguistics.
  * Eisenstein and Barzilay (2008) Jacob Eisenstein and Regina Barzilay. 2008.  Bayesian unsupervised topic segmentation.  In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_ , pages 334–343, Honolulu, Hawaii. Association for Computational Linguistics.
  * Glavaš and Šnajder (2014) Goran Glavaš and Jan Šnajder. 2014.  Constructing coherent event hierarchies from news stories.  In _Proceedings of TextGraphs-9: the workshop on Graph-based Methods for Natural Language Processing_ , pages 34–38, Doha, Qatar. Association for Computational Linguistics.
  * Glavaš et al. (2014) Goran Glavaš, Jan Šnajder, Marie-Francine Moens, and Parisa Kordjamshidi. 2014.  HiEve: A corpus for extracting event hierarchies from news stories.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 3678–3683, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Han et al. (2021) Rujun Han, I Hsu, Jiao Sun, Julia Baylon, Qiang Ning, Dan Roth, Nanyun Pen, et al. 2021.  Ester: A machine reading comprehension dataset for event semantic relation reasoning.  _arXiv preprint arXiv:2104.08350_.
  * Hovy et al. (2013) Eduard Hovy, Teruko Mitamura, Felisa Verdejo, Jun Araki, and Andrew Philpot. 2013\.  Events are not simple: Identity, non-identity, and quasi-identity.  In _Workshop on Events: Definition, Detection, Coreference, and Representation_ , pages 21–28, Atlanta, Georgia. Association for Computational Linguistics.
  * Koshorek et al. (2018) Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, and Jonathan Berant. 2018.  Text segmentation as a supervised learning task.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 469–473, New Orleans, Louisiana. Association for Computational Linguistics.
  * Li et al. (2020a) Manling Li, Qi Zeng, Ying Lin, Kyunghyun Cho, Heng Ji, Jonathan May, Nathanael Chambers, and Clare Voss. 2020a.  Connecting the dots: Event graph schema induction with path language modeling.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 684–695, Online. Association for Computational Linguistics.
  * Li et al. (2019) Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. 2019.  A logic-driven framework for consistency of neural models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3924–3935, Hong Kong, China. Association for Computational Linguistics.
  * Li et al. (2020b) Tao Li, Parth Anand Jawale, Martha Palmer, and Vivek Srikumar. 2020b.  Structured tuning for semantic role labeling.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8402–8412, Online. Association for Computational Linguistics.
  * Li and Srikumar (2019) Tao Li and Vivek Srikumar. 2019.  Augmenting neural networks with first-order logic.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 292–302, Florence, Italy. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Lukasik et al. (2020) Michal Lukasik, Boris Dadachev, Kishore Papineni, and Gonçalo Simões. 2020\.  Text segmentation by cross segment attention.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4707–4716, Online. Association for Computational Linguistics.
  * Mota et al. (2019) Pedro Mota, Maxine Eskenazi, and Luísa Coheur. 2019.  BeamSeg: A joint model for multi-document segmentation and topic identification.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 582–592, Hong Kong, China. Association for Computational Linguistics.
  * Newman et al. (2012) David Newman, Nagendra Koilada, Jey Han Lau, and Timothy Baldwin. 2012.  Bayesian text segmentation for index term identification and keyphrase extraction.  In _Proceedings of COLING 2012_ , pages 2077–2092, Mumbai, India. The COLING 2012 Organizing Committee.
  * Niculae et al. (2018) Vlad Niculae, Andre Martins, Mathieu Blondel, and Claire Cardie. 2018.  SparseMAP: Differentiable sparse structured inference.  In _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 3799–3808. PMLR.
  * O’Gorman et al. (2016) Tim O’Gorman, Kristin Wright-Bettner, and Martha Palmer. 2016.  Richer event description: Integrating event coreference with temporal, causal and bridging annotation.  In _Proceedings of the 2nd Workshop on Computing News Storylines (CNS 2016)_ , pages 47–56, Austin, Texas. Association for Computational Linguistics.
  * Pan et al. (2020) Xingyuan Pan, Maitrey Mehta, and Vivek Srikumar. 2020.  Learning constraints for structured prediction using rectifier networks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4843–4858, Online. Association for Computational Linguistics.
  * Pan and Srikumar (2016) Xingyuan Pan and Vivek Srikumar. 2016.  Expressiveness of rectifier networks.  In _Proceedings of The 33rd International Conference on Machine Learning_ , volume 48 of _Proceedings of Machine Learning Research_ , pages 2427–2435, New York, New York, USA. PMLR.
  * Pohl et al. (2012) Daniela Pohl, Abdelhamid Bouchachia, and Hermann Hellwagner. 2012.  Automatic sub-event detection in emergency management using social media.  In _Proceedings of the 21st international conference on world wide web_ , pages 683–686.
  * Reddi et al. (2018) Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. 2018.  On the convergence of adam and beyond.  In _International Conference on Learning Representations (ICLR)_.
  * Rocktäschel and Riedel (2017) Tim Rocktäschel and Sebastian Riedel. 2017.  End-to-end differentiable proving.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Roth (2017) Dan Roth. 2017.  Incidental Supervision: Moving beyond Supervised Learning.  In _Proc. of the Conference on Artificial Intelligence (AAAI)_.
  * Roth and Yih (2004) Dan Roth and Scott Yih. 2004.  A Linear Programming Formulation for Global Inference in Natural Language Tasks.  In _Proc. of the Conference on Computational Natural Language Learning (CoNLL)_ , pages 1–8. Association for Computational Linguistics.
  * UzZaman and Allen (2011) Naushad UzZaman and James Allen. 2011.  Temporal evaluation.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 351–356, Portland, Oregon, USA. Association for Computational Linguistics.
  * Wang et al. (2020) Haoyu Wang, Muhao Chen, Hongming Zhang, and Dan Roth. 2020.  Joint constrained learning for event-event relation extraction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 696–706, Online. Association for Computational Linguistics.
  * Yao et al. (2020) Wenlin Yao, Zeyu Dai, Maitreyi Ramaswamy, Bonan Min, and Ruihong Huang. 2020.  Weakly Supervised Subevent Knowledge Acquisition.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5345–5356, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Hongming Zhang, Muhao Chen, Haoyu Wang, Yangqiu Song, and Dan Roth. 2020.  Analogous process structure induction for sub-event sequence prediction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1541–1550, Online. Association for Computational Linguistics.
  * Zhao et al. (2020) Lulu Zhao, Weiran Xu, and Jun Guo. 2020.  Improving abstractive dialogue summarization with graph structures and topic words.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 437–449, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Zhou et al. (2020) Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020.  Temporal common sense acquisition with minimal supervision.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7579–7589, Online. Association for Computational Linguistics.