# Unsupervised Chunking with Hierarchical RNN

Zijun Wu1, Anup Anand Deshmukh∗2, Yongkang Wu3, Jimmy Lin2, Lili Mou†1
1Dept. Computing Science, Alberta Machine Intelligence Institute (Amii),
University of Alberta
2University of Waterloo, 3Huawei Poisson Lab
∗Work partially done during the internship at UofA/Amii, †Canada CIFAR AI
Chair, Amii
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

In Natural Language Processing (NLP), predicting linguistic structures, such
as parsing and chunking, has mostly relied on manual annotations of syntactic
structures. This paper introduces an unsupervised approach to chunking, a
syntactic task that involves grouping words in a non-hierarchical manner. We
present a two-layer Hierarchical Recurrent Neural Network (HRNN) designed to
model word-to-chunk and chunk-to-sentence compositions. Our approach involves
a two-stage training process: pretraining with an unsupervised parser and
finetuning on downstream NLP tasks. Experiments on the CoNLL-2000 dataset
reveal a notable improvement over existing unsupervised methods, enhancing
phrase F1 score by up to 6 percentage points. Further, finetuning with
downstream tasks results in an additional performance improvement.
Interestingly, we observe that the emergence of the chunking structure is
transient during the neural model’s downstream-task training. This study
contributes to the advancement of unsupervised syntactic structure discovery
and opens avenues for further research in linguistic theory.111This paper is a
substantially extended version of an EMNLP Findings paper (Deshmukh et al.,
2021). The main extension is a novel approach that improves unsupervised
chunking performance in a downstream NLP task, which is an interesting result
by itself. Part of the text is reused with permission from our previous work
(Deshmukh et al., 2021), available at https://aclanthology.org/2021.findings-
emnlp.307.pdf, licensed under CC BY 4.0. Our code and results are available at
https://github.com/MANGA-UOFA/UCHRNN

##  1 Introduction

Understanding the linguistic structure of language, such as parsing and
chunking, is an important research topic in NLP. While most previous work
(Kudo and Matsumoto, 2001; Zhang et al., 2002; Pradhan et al., 2004) employs
supervised machine learning methods to predict linguistic structures and can
achieve high performance, they rely heavily on manual annotations of syntactic
structures like Treebanks (Marcus et al., 1993). As a result, there has been a
growing interest in unsupervised linguistic structure discovery in recent
years (Kim et al., 2020; Shen et al., 2018, 2019), which is important to
natural language processing (NLP) research because it sheds light on
linguistic theories and can potentially benefit low-resource languages.
Previous work on unsupervised syntactic structure discovery has mainly focused
on unsupervised constituency parsing (Kim et al., 2020, 2019; Shen et al.,
2018, 2019) and dependency parsing (Klein and Manning, 2004; Gillenwater et
al., 2010; Shen et al., 2021).

In this work, we address unsupervised chunking, another meaningful task for
discovering syntactic structure. Unlike parsing that induces tree structures
from a sentence, chunking aims to group consecutive words of a sentence in a
non-hierarchical fashion; each chunk can be intuitively thought of as a phrase
(Tjong Kim Sang and Buchholz, 2000; Clark, 2001). In fact, unsupervised
chunking has wide applications in real-world scenarios, as understanding text
fundamentally requires finding spans like noun phrases and verb phrases. It
would benefit other NLP tasks, such as keyword extraction (Firoozeh et al.,
2020), named entity recognition (Sato et al., 2017), open information
extraction (Niklaus et al., 2018; Borgeaud et al., 2022; Izacard et al.,
2022), and logical reasoning (Wu et al., 2023).

We propose a two-layer hierarchical recurrent neural network (HRNN) to
accomplish the chunking task. Our HRNN is designed to explicitly model the
word-to-chunk composition by a lower-level RNN, and chunk-to-sentence
composition by an upper-level RNN. We further design a trainable chunking gate
that switches between lower word-level RNN and upper phrase-level RNN, which
is also used for chunk prediction.

We propose a two-stage training framework for HRNN without using linguistic
annotations, namely, pretraining with an unsupervised parser and finetuning
with a downstream NLP task. In the first stage of pretraining, we adopt the
recent advances of unsupervised parsing (Kim et al., 2020, 2019), and propose
a maximal left-branching heuristic to induce sensible (albeit noisy and
imperfect) chunk labels from an unsupervised parser. In the second stage, we
finetune HRNN chunking by feeding its upper-level RNN representations to a
downstream-task network. Our intuition is that a more meaningful chunking
structure is beneficial for the downstream task. Therefore, optimizing
downstream-task performance can, in turn, improve HRNN chunking without using
syntactically annotated labels.

We conducted experiments on the CoNLL-2000 chunking dataset (Tjong Kim Sang
and Buchholz, 2000). Results show that the unsupervised parser-pretrained HRNN
significantly improves the best-performing unsupervised baseline, with a
considerable margin of $5$ percentage points in terms of the phrase F1 score.
We then finetuned the pretrained HRNN model with three downstream text
generation tasks: summarization, paraphrasing, and translation. Compared with
our pretrained model, it achieves a chunking performance improvement of up to
$6$ percentage points in downstream datasets, and $2$ percentage points on
CoNLL-2000. The results suggest that our method not only bridges the gap
between supervised and unsupervised chunking methods but also shows the
generalizability across different downstream tasks and datasets.

In our experiment, we also find an intriguing linguistic phenomenon during the
finetuning stage: the neural model’s emergence of linguistic structure is
transient. That is, although the downstream performance consistently improves,
the chunking performance improves significantly but only in the first few
thousand steps. Then, it starts to decrease and eventually drops to the
initial level of the finetuning stage or even lower, suggesting that
linguistic structures are a convenient vehicle for a downstream task when the
training begins and the model capacity is small. However, the neural network
tends to discard such linguistic structures to achieve higher performance in
the downstream task as the training proceeds.

In summary, our main contributions are as follows:

  * •

We address the task of unsupervised chunking, and propose a hierarchical
recurrent neural network (HRNN) that can explicitly model the word-to-chunk
and chunk-to-sentence compositions.

  * •

We propose a two-stage training framework for HRNN, largely outperforming
previous unsupervised chunking methods.

  * •

We observe the neural model’s emergence of chunking structure is transient,
which may inspire further study of linguistic theory.

##  2 Approach

In this section, we first introduce the proposed Hierarchical RNN (HRNN)
model. Then we discuss the process of pretraining a HRNN with the induced
labels from a state-of-the-art unsupervised parser. Finally, we propose to
finetune the model with a downstream task to improve chunking performance.

###  2.1 Hierarchical RNN

Figure 1: An overview of our chunking induction method. On the left, the
pretraining of HRNN using the induced labels from the Compound PCFG parser is
shown. On the right, the HRNN is fine-tuned with text generation, specifically
a summarization task in this example. A weight matrix is created from the
switching gate’s values and then inserted into the Transformer’s encoder-
decoder attention modules.

We model the chunking task in the BI schema (Ramshaw and Marcus, 1995), where
“B” denotes the beginning of a chunk, and “I” denotes the inside of a chunk.
We observe that encoder-only RNNs or Transformers may not be suitable for the
chunking task because they lack autoregressiveness, which means the prediction
at a time step is unaware of previously predicted chunks. Feeding predicted
chunk labels like a sequence-to-sequence model is also inadequate because a BI
label only provides one-bit information and does not offer meaningful
autoregressive information.

To this end, we design a hierarchical RNN to model the autoregressiveness of
predicted chunks. Our HRNN contains a lower-level RNN operating at the word
level and an upper-level RNN operating at the chunk level. We design a gating
mechanism that switches between the two RNNs in a soft manner, also serving as
the chunk label. The bottom-left part of Figure 1 shows the structure of HRNN.

Given a sentence with words $\mathrm{x}^{(1)},\cdots,\mathrm{x}^{(n)}$, we
first apply a pretrained Transformer encoder (Devlin et al., 2019) to obtain
the contextual representations of the words that help our model to understand
the previous and future context of the sentence. We denote the contextual
embeddings of these words by $\bm{x}^{(1)},\cdots,\bm{x}^{(n)}$. For a step
$t$, we predict a switching gate $m^{(t)}\in(0,1)$ as the chunking
decision.222Here, $m^{(t)}=1$ corresponds to “B,” i.e., a new chunk, and
$m^{(t)}=0$ corresponds to “I,” i.e., inside of a chunk.

$\displaystyle
m_{\text{logit}}^{(t)}=W\bigl{[}\underline{\bm{h}}^{(t-1)};\overline{\bm{h}}^{(t-1)};\bm{x}^{(t)}\bigr{]}$
(1) $\displaystyle m^{(t)}=\sigma(m_{\text{logit}}^{(t)})$ (2)

where $\underline{\bm{h}}^{(t-1)}$ is the hidden state of the lower RNN and
$\overline{\bm{h}}^{(t-1)}$ is that of the upper RNN. The semicolon represents
vector concatenation, and $\sigma$ represents the sigmoid function.

The switching gate is also used to control the information flow between the
lower-level word RNN and the upper-level chunk RNN. In this way, it provides
meaningful autoregressive information that enables HRNN to be aware of
previously detected chunks.

Suppose our model predicts the $t$th word as the beginning of a chunk, which
essentially “cuts” the sequence into two parts at this step. The lower RNN and
upper RNN are updated by

$\displaystyle\underline{\bm{h}}^{(t)}_{\text{cut}}=\underline{f}\bigl{(}\bm{x}^{(t)},\underline{\bm{h}}^{\text{(sos)}}\bigr{)}$
(3)
$\displaystyle\overline{\bm{h}}^{(t)}_{\text{cut}}=\overline{f}\bigl{(}\underline{\bm{h}}^{(t-1)},\overline{\bm{h}}^{(t-1)}\bigr{)}$
(4)

where $\underline{f}$ and $\overline{f}$ are the transition functions of the
lower and upper RNNs, respectively, both employing the hyperbolic tangent
(tanh) as their activation function.

In Equation (3), the lower RNN ignores its previous hidden state and starts
anew from a learnable initial state $\underline{\bm{h}}^{\text{(sos)}}$,
because a new chunk is predicted at this step. On the other hand, in Equation
(4), the upper RNN picks the newly formed chunk $\underline{\bm{h}}^{(t-1)}$
captured by the lower RNN, and fuses it with the previous upper RNN’s state
$\overline{\bm{h}}^{(t-1)}$.

Suppose our model predicts that the $t$th word is not the beginning of a
chunk, i.e., “no cut” occurs at this step. The RNNs are then updated by

$\displaystyle\underline{\bm{h}}^{(t)}_{\text{nocut}}=\underline{f}\bigl{(}\bm{x}^{(t)},\underline{\bm{h}}^{(t-1)}\bigr{)}$
(5)
$\displaystyle\overline{\bm{h}}^{(t)}_{\text{nocut}}=\overline{\bm{h}}^{(t-1)}$
(6)

In this scenario, the lower RNN updates its hidden state with the input
$\bm{x}^{(t)}$ in the same manner as a typical RNN, whereas the upper RNN
remains idle because no chunk is formed.

The “cut” and “no cut” cases can be unified by

$\displaystyle\overline{\bm{h}}^{(t)}=m^{(t)}\overline{\bm{h}}^{(t)}_{\text{cut}}+(1-m^{(t)})\overline{\bm{h}}^{(t)}_{\text{nocut}}$
(7)
$\displaystyle\underline{\bm{h}}^{(t)}=m^{(t)}\underline{\bm{h}}^{(t)}_{\text{cut}}+(1-m^{(t)})\underline{\bm{h}}^{(t)}_{\text{nocut}}$
(8)

In our work, we keep $m^{(t)}$ as a real number and adopt a soft gating
mechanism that fuses “cut” and “no cut” in a soft manner. This is because
chunking, by its nature, may be ambiguous, and our soft gating mechanism can
better handle such ambiguity.

Our proposed HRNN supports end-to-end learning, which can be weakly supervised
by downstream tasks. Nevertheless, a randomly initialized HRNN may have
difficulty modeling the desired chunking structure and suffer from the cold-
start problem. We thus propose to pretrain HRNN by heuristically induced
chunking labels from an unsupervised parser, which will be discussed in
Section 2.2.

###  2.2 Pretraining HRNN by Unsupervised Parsing

We propose to induce chunk labels from an unsupervised parser. The intuition
is that the chunking structure can be considered a flattened parse tree, thus
sharing some commonalities with the parsing structure. Our approach is able to
take advantage of recent advances in unsupervised parsing (Kim et al., 2020,
2019).

Specifically, we adopt a state-of-the-art unsupervised parser, Compound
Probabilistic Context-Free Grammar (PCFG) (Kim et al., 2019), which is a tuple
$\mathcal{G}=(\mathcal{S},\mathcal{N},\mathcal{P},\Sigma,\mathcal{R})$, where
$\mathcal{S}$ is a start symbol; $\mathcal{N}$, $\mathcal{P}$, and $\Sigma$
are finite sets of nonterminal, preterminal, and terminal symbols,
respectively. $\mathcal{R}$ is a finite set of rules taking one of the
following forms:

$\displaystyle\mathrm{S}\rightarrow\mathrm{A}$
$\displaystyle\mathrm{A}\in\mathcal{N}$ (9)
$\displaystyle\mathrm{A}\rightarrow\mathrm{B}\ \mathrm{C}$
$\displaystyle\mathrm{B},\mathrm{C}\in\mathcal{N}\cup\mathcal{P}$ (10)
$\displaystyle\mathrm{T}\rightarrow\mathrm{w}$
$\displaystyle\mathrm{T}\in\mathcal{P},\mathrm{w}\in\Sigma$ (11)

where $\mathrm{S}\rightarrow\mathrm{A}$ is the start of a sentence and
$\mathrm{T}\rightarrow\mathrm{w}$ indicates the generation of a word.
$\mathrm{A}\rightarrow\mathrm{B}\ \mathrm{C}$ models the bifurcations of a
binary constituency tree, where a constituent node is not explicitly
associated with a type (e.g., noun phrase) in our setting.

In addition, the model maintains a continuous random vector at the sentence
level, which serves as the prior of PCFG. To train Compound PCFG, the maximum
likelihood of the text is utilized, and a Viterbi-like algorithm marginalizes
the PCFG. Amortized variational inference is employed to handle the continuous
distribution from the sentence-level vector. We refer readers to Kim et al.
(2019) for details.

We aim to induce chunk labels from the state-of-the-art unsupervised parser,
Compound PCFG. Given a sentence, we obtain its parse tree by applying the
Viterbi-like CYK algorithm (Jurafsky and Martin, 2000) to Compound PCFG. Then,
we propose a simple yet effective heuristic that extracts maximal left-
branching subtrees as chunks. It is known that the English language has a
strong bias toward right-branching structures (Williams et al., 2018a; Li et
al., 2019a). We observe, on the other hand, that a left-branching structure
often indicates words that are closely related. Here, a left-branching subtree
means that the words are grouped in the form of
$((\cdots((\mathrm{x}_{i}\mathrm{x}_{i+1})\mathrm{x}_{i+2})\cdots)\mathrm{x}_{i+n-1})$.
A left-branching subtree for words $\mathrm{x}_{i}\cdots\mathrm{x}_{i+n-1}$ is
maximal if neither $\mathrm{x}_{i-1}\mathrm{x}_{i}\cdots\mathrm{x}_{x+n-1}$
nor $\mathrm{x}_{i}\cdots\mathrm{x}_{x+n-1}\mathrm{x}_{i+n}$ is left-
branching. We extract all maximal left-branching subtrees as chunks. Our
heuristic can provide unambiguous chunk labels for any sentence with any parse
tree, as demonstrated by the following theorem.

######  Theorem 1.

Given any binary parse tree, every word will belong to one and only one chunk
by the maximal left-branching heuristic.

###### Proof.

[Existence] A single word itself is a left-branching subtree, which belongs to
some maximal left-branching subtree.

[Uniqueness] We will show that two different maximal left-branching subtrees
$\textrm{s}_{1}$ and $\textrm{s}_{2}$ cannot overlap. Assume by way of
contradiction that there exists a word $\textrm{x}_{i}$ in both
$\textrm{s}_{1}$ and $\textrm{s}_{2}$. Then, $\textrm{s}_{1}$ must be a
substructure of $\textrm{s}_{2}$ or vice versa; otherwise, the paths,
$\textrm{root}\text{\textendash}\textrm{s}_{1}\text{\textendash}\textrm{x}_{i}$
and
$\textrm{root}\text{\textendash}\textrm{s}_{2}\text{\textendash}\textrm{x}_{i}$,
violate the acyclic nature of a tree. But $\textrm{s}_{1}$ being a subtree of
$\textrm{s}_{2}$ (or vice versa) contradicts the maximality of
$\textrm{s}_{1}$ and $\textrm{s}_{2}$. ∎

###  2.3 Finetuning Hierarchical RNN with Downstream Tasks

We propose to finetune the HRNN in a weakly supervised manner to learn better
chunking structures from a downstream task. Our intuition is that
accomplishing a downstream NLP task requires understanding meaningful semantic
units/chunks of a sentence (Wu et al., 2023), and that optimizing the
downstream task may benefit the chunk prediction.

Specifically, we consider text generation tasks, namely, summarization (Rush
et al., 2015; Liu et al., 2022; Liu, 2019), paraphrasing (Liu et al., 2020a;
Li et al., 2019b), and translation (Bojar et al., 2014; Liu et al., 2020b).
Although the HRNN may be finetuned with classification tasks as well, we found
in our preliminary experiments that classification tasks yield marginal
improvement, probably because the classification training signal is too sparse
(one label per sample) in comparison to generation tasks. Therefore, we only
consider finetuning HRNN with text generation tasks in this work.

We propose a decoder network, whose attention is based on the upper-level RNN
(chunk representations). This is accomplished by performing conventional
attention over the RNN’s states but reweighing it by whether a chunk is
predicted at the step. Since the HRNN’s gate is differentiable, we can train
it end to end with the downstream tasks.

We first argue that the chunk representation is indeed given by the upper-
level RNN where a “cut” is predicted. Suppose we have a chunk
$\mathrm{x}^{(i)}\cdots\mathrm{x}^{(i+k)}$, i.e., HRNN predicts two
consecutive “cuts” (end of a chunk) at the time steps $i-1$ and $i+k$.
According to our design (Section 2.1), the lower-level RNN initializes its
hidden state with $\underline{\bm{h}}^{\text{(sos)}}$ when $t=i-1$, and
updates its hidden state $\underline{\bm{h}}^{(t)}$ with the input words
$\mathrm{x}^{(i)}\cdots\mathrm{x}^{(i+k)}$. In the meantime, the upper-level
RNN is idle for time steps $i$ to $i+k-1$, but picks up the
$\underline{\bm{h}}^{(i+k)}$ at step $i+k$. This follows a conventional RNN,
demonstrating that the chunk representation is given by the upper-level RNN at
a “cut” step.

Now consider the scenarios with “soft” chunks, since we keep the chunking
decision $m^{(t)}$ as a real number in our HRNN. When the “cut” strength is
high ($m^{(t)}$ close to 1), the step is more likely to be a chunk, and we
would like to attend to this step more from the decoder. Conversely, if the
“cut” strength is low, it should be less attended. This can be achieved by
applying conventional encoder–decoder attention to every step, but reweighing
it by the “cut” strength. For one attention head, we form a query vector
$\bm{q}^{(j)}$ for the $j$th decoder step and a key vector $\bm{k}^{(i)}$ for
the $i$th encoder step. Here, the key vector $\bm{k}^{(i)}$ is given by the
upper-level RNN’s state $\overline{\bm{h}}^{(i)}$. We reweigh the attention by
first computing an unnormalized measure.

$\displaystyle\widetilde{\alpha}\Bigl{(}\bm{q}^{(j)},\bm{k}^{(i)},m_{\text{logit}}^{(i)}\Bigr{)}=\operatorname{exp}{\Bigl{(}\frac{(\bm{q}^{(i)})^{\top}\bm{k}^{(j)}}{\sqrt{d}}+\gamma
m_{\text{logit}}^{(i)}\Bigr{)}}$ (12)

where $\gamma$ is the coefficient hyperparameter. Then, we normalize it to
attention probabilities as

$\displaystyle\alpha^{(j,i)}=\frac{\widetilde{\alpha}\Bigl{(}\bm{q}^{(j)},\bm{k}^{(i)},m_{\text{logit}}^{(i)}\Bigr{)}}{\sum_{i^{\prime}}\widetilde{\alpha}\Bigl{(}\bm{q}^{(j)},\bm{k}^{(i^{\prime})},m_{\text{logit}}^{(i^{\prime})}\Bigr{)}}$
(13)

Empirically, we find that an end-to-end reweighted attention head tends to
ignore $m_{\text{logit}}^{(i)}$ and make it close to 0. Consequently, the
gating probability $m^{(i)}$ will be close to 0.5 unaminously, and the HRNN
will be in a situation of neither “cut” nor “no cut”. We alleviate this by
proposing an auxiliary training objective that pushes $m^{(i)}$ to either 0 or
1. We introduce a chunking strength hyperparameter $\kappa\%$ that aims to
control the granularity of chunking. A detailed analysis of this
hyperparameter is presented in Section 3.4.3. The final values for $\kappa\%$
are decided based on validation performance. We then define the loss at the
$i$th encoder step as

$\displaystyle L^{(i)}=\begin{cases}(m^{(i)}-1)^{2},&\text{if $m^{(i)}$ is
among the top-$\kappa$\% gate values,}\\\
(m^{(i)}-0)^{2},&\text{otherwise.}\end{cases}$ (14)

The overall auxiliary loss is $L_{\text{aux}}=\sum_{i}L^{(i)}$, and is
combined with the end-to-end downstream task by $L_{\text{task}}+\eta
L_{\text{aux}}$, where $\eta$ is a hyperparameter balancing the two training
objectives. In this way, we are able to learn a meaningful chunking gate in
the downtream NLP task.

##  3 Experiments

Task | Dataset |  $\\#$Train |  $\\#$Dev |  $\\#$Test | Batch size | Learning rate | $\eta$ | $\gamma$ | top-$\kappa$%
---|---|---|---|---|---|---|---|---|---
Chunking | CoNLL-2000 | 8K | 1K | 2K | 32 | 5e-5 | – | – | –
Downstream | Gigaword | 3.8M | 2K | 2K | 32 | 5e-5 | 0.1 | 0.1 | 50$\%$
MNLI-Gen | 131K | 3.5K | 3.5K | 32 | 4e-5 | 0.1 | 0.1 | 60$\%$
WMT-14 (En-De) | 4.5M | 3K | 2.7K | 32 | 1e-5 | 0.1 | 0.1 | 50$\%$

Table 1: Details on the dataset and implementation. Symbol $\eta$ represents
the balance between two training objectives, while $\gamma$ is the hyper-
parameter for the reweight operation, as defined in Equation 12. The term
top-$\kappa$ denotes the chunking strength heuristics.

###  3.1 Datasets and Metrics

We adopted a widely used chunking dataset, CoNLL-2000 (Tjong Kim Sang and
Buchholz, 2000), to evaluate the general chunking performance of a model.
CoNLL-2000 contains 8K training, 1K validation, and 2K test samples. Each
sample is labeled with the BIO schema, where “B” indicates the beginning of a
chunk, “I” indicates inside a chunk, and “O” indicates outside a chunk (mainly
punctuation). We used the BI schema and ignore the “O” tokens.

In this work, however, we addressed chunking in the unsupervised setting, and
thus we did not use the groundtruth chunk labels in the training set, but only
the training sentences. Groundtruth chunk labels are used for
validation333Certain researchers argue that labeled validation sets cannot be
used for unsupervised grammar induction and resort to the test set for
hyperparamter tuning and model selection (Shi et al., 2020), which violates
the training-validation-test framework of machine learning. We agree that it
is important to investigate unsupervised validation measures for grammar
induction, but without an established one, we use labeled data for validation
purposes in this work. Notice that this fundamentally differs from supervised
training, as we only use labels for collecting statistics of model
performance. and test purposes.

We finetuned the HRNN model with several downstream text generation tasks,
whose datasets are introduced as follows.

  * •

Gigaword (Rush et al., 2015): A widely used article–headline dataset for text
summarization with about 3.8M training, 20K validation, and 2K test samples.
We randomly picked 2K samples (same size as the test set) for more efficient
validation.

  * •

MNLI-Gen: MNLI (Williams et al., 2018b) is a massive dataset for the natural
language inference task. We used the entailment subset, generating the
entailed hypothesis based on the premise. Since the test set of MNLI is
private, we adopted the matched section of the original validation set as our
test set, while the mismatched section served as our validation set. The
resulting entailment generation dataset contains 131K training, 3.5K
validation, and another 3.5K test samples.

  * •

WMT-14 (Bojar et al., 2014): A multilingual dataset used for machine
translation. We used the English-to-German subset, having nearly 4.5M
training, 3K validation, and 2.7K test samples.

Table 1 shows the statistics of the datasets we have introduced. In addition
to the general chunking performance on the CoNLL-2000 corpus, we are
interested in in-domain chunking performance after finetuning with a
downstream task, whose test set, unfortunately, does not have human annotation
of chunking labels. Therefore, we resorted to NLTK (Bird and Loper, 2004),
which has an off-the-shelf chunker that achieves high performance. We used
NLTK to chunk the source sentences of the downstream datasets, and the NLTK’s
outputs (also with the BI schema) served as the in-domain chunking labels.

Regarding the evaluation metrics, we adopted the standard phrase F1 score and
tagging accuracy. In our implementation, they are realized by the CoNLL-2000
evaluation script (Tjong Kim Sang and Buchholz, 2000).

###  3.2 Implementation Details

We utilized an encoder-only BERT model and two encoder-decoder models, BART
and mBART, in our experiments. For the BERT model (Devlin et al., 2019), we
adopted its base version that has $12$ Transformer layers with $768$ hidden
dimensions, and its total parameter number is $110$ million. For the BART
model (Lewis et al., 2020), we also used its base version, which has $6$
encoder layers and another $6$ decoder layers. It has 768 dimensions and 139
million parameters in total. The mBART model (Liu et al., 2020b) has 610
million parameters, both the encoder and decoder having $12$ layers and $1024$
dimensions. We only used the encoder modules of BART and mBART for the
chunking pretraining, which are then combined with their respective pretrained
decoder for finetuning on downstream tasks. Our HRNN has the same dimension as
the underlying Transformer models.

We used the Adam optimizer (Kingma and Ba, 2015) for both pretraining and
finetuning. The optimizer hyperparameters were set as $\beta_{1}=0.9$,
$\beta_{2}=0.999$. We applied a learning rate warm-up for the first $10$
percent of total steps, followed by a linear decay. The detailed
hyperparameters for both stages can be found in Table 1.

The pretraining phase generally concludes within three epochs, whereas the
finetuning process requires less than one. Although one-epoch finetuning may
appear unreasonable at first glance, we observed that Transformers are very
powerful models, which may accomplish a downstream task while bypassing
syntactic information. To let meaningful syntax emerge in the downstream task,
we have to consider early stopping and only very few iterations are needed.
Details will be presented in Section 3.4.3.

###  3.3 Main Results

# | Method | CoNLL-2000 | Gigaword | MNLI-Gen | WMT-14
---|---|---|---|---|---
Phrase F1 | Tag Acc. | Phrase F1 | Tag Acc. | Phrase F1 | Tag Acc. | Phrase F1 | Tag Acc.
Supervised Methods
1 | NLTK-tagger-chunker | 83.71 | 89.51 | – | – | – | – | – | –
2 | Supervised HMM | 87.68 | 93.99 | 87.54 | 94.37 | 86.81 | 93.40 | 86.49 | 93.50
Unsupervised Methods
3 | PMI Chunker | 35.64 | 64.50 | 37.44 | 65.18 | 44.37 | 67.14 | 45.70 | 67.20
4 | Baum–Welch HMM | 25.04 | 58.93 | 38.35 | 54.75 | 29.87 | 32.06 | 34.87 | 45.00
5 | LM Chunker | 42.05 | 68.74 | 29.31 | 51.34 | 25.85 | 49.57 | 32.84 | 50.72
6 | Compound PCFG Chunker | 62.89 | 81.64 | 25.69 | 51.70 | 32.96 | 55.73 | 52.16 | 73.73
Our Pretrain Methods
7 | HRNN w/ BERT | 68.12 | 83.90 | 64.20 | 82.61 | 66.38 | 81.60 | 67.27 | 82.70
8 | HRNN w/ BART | 68.57 | 84.04 | 64.60 | 82.24 | 68.81 | 82.04 | 72.09 | 84.87
9 | HRNN w/ mBART | 68.65 | 84.42 | 67.08 | 83.49 | 67.25 | 81.35 | 71.24 | 84.55
Our Finetune Methods
10 | HRNN w/ BART (Gigaword) | 70.83 | 85.16 | 70.66† | 85.15† | 73.96 | 84.93 | 75.87 | 86.66
11 | HRNN w/ BART (MNLI-Gen) | 69.68 | 84.72 | 67.15 | 83.36 | 70.79† | 83.11† | 73.26 | 85.43
12 | HRNN w/ mBART (WMT-14) | 69.77 | 84.69 | 67.91 | 83.58 | 69.93 | 82.75 | 71.85† | 84.69†

Table 2: Main results. The numbers with † indicate that the model is tested on
the same datasets as the finetuning tasks.

Table 2 presents the main results of our approach. We used CoNLL-2000 as the
main dataset to evaluate the general chunking performance. To measure in-
domain performance on Gigaword, MNLI-Gen, and WMT-14, we used the NLTK
supervised chunker (Bird and Loper, 2004) as pseudo-groundtruth labels. For
comparison of inducing chunks from unsupervised parsing, we include Compound
PCFG (discussed in Section 2.2, Kim et al., 2019) and another recent
unsupervised parser based on the features of a pretrained language model
proposed by Kim et al. (2020). The latter, called an LM Chunker in Table 2,
thresholds the BERT (Devlin et al., 2019) similarity of consecutive words for
chunking. We observe from Lines 5 and 6 that the LM-based unsupervised chunker
is worse than the Compound PCFG in phrase F1 (42.05 vs. 62.89) and tag
accuracy (68.74 vs. 81.64). Therefore, we use the chunking labels induced by
Compound PCFG to pretrain our HRNN model in all experiments.

We also include several traditional unsupervised chunking methods as
baselines: a pointwise mutual information (PMI) chunker (Van de Cruys, 2011)
cuts two consecutive words if their PMI score is below a threshold; Baum–Welch
HMM (Baggenstoss, 2001) applies expectation–maximization to a hidden Markov
model (HMM, Rabiner, 1989). These methods perform significantly worse than
recent advances in unsupervised syntactic structure discovery.

However, we notice that both Compound PCFG and LM chunkers (Lines 5–6 in Table
2) show a significant decrease in performance when transferring to other
datasets. For example, when the two chunkers are transferred from CoNLL-2000
to Gigaword, their tag accuracy is dropped by 25 percent and 37 percent,
respectively. By contrast, the PMI chunker has a 7 percent drop, while the
Baum–Welch HMM even has a 1 percent increase (Lines 3–4). The results indicate
that traditional unsupervised chunking methods are more stable, although it
achieves lower performance on CoNLL-2000. Overall, unsupervised chunkers may
suffer from the problem of poor transferability in addition to less
satisfactory chunking performance in general.

Regarding our method, we pretrained HRNN with several Transformer encoders:

  1. a)

BERT (Devlin et al., 2019), an encoder-only Transformer model used in our
preliminary conference paper (Deshmukh et al., 2021).

  2. b)

BART (Lewis et al., 2020), chosen due to its pretrained decoder, which allows
HRNN to be finetuned on Gigaword and MNLI-Gen datasets for generation tasks.

  3. c)

mBART (Liu et al., 2020b), a multilingual variant of BART, which is needed for
the translation task.

We see from Lines 7–9 that all the pretrained HRNN variants achieve an
improvement of more than $5$ percentage points in phrase F1 based on the
Compound PCFG chunker on the CoNLL-2000 dataset. BART-based and mBART-based
HRNN is better than the BERT-based variant not only on the standard CoNLL-2000
dataset but also in transferring its learned chunking knowledge to three
downstream datasets. The large margins between the Compound PCFG chunker and
our pretrained HRNN model imply that HRNN can indeed smooth out the noise of
heuristics while capturing meaningful chunking patterns. More importantly, our
method drastically improves the transferability of unsupervised chunkers
(Lines 7–9 vs. Lines 3–6).

We then finetuned the HRNN on some downstream task for syntactic structure
discovery. We observed an improved chunking performance across all datasets,
including Gigaword, MNLI-Gen, and WMT-14. As shown in Lines 11–12, finetuning
HRNN with MNLI-Gen and WMT-14 leads to a 1–2 percent increase in phrase F1 on
the in-domain datasets (70.79 vs. 68.81; 71.81 vs. 71.24) as well as the
general CoNLL-2000 corpus (69.68 vs. 68.57; 69.77 vs. 68.65).

Notably, finetuning with the summarization task yields the most significant
improvements, as indicated in Line 10. It achieves nearly an increase of $6$
percentage points in phrase F1 on the in-domain Gigaword dataset (70.66 vs.
64.60), as well as an increase of $5$ and $4$ percentage points on the out-of-
domain MNLI-Gen and WMT-14 datasets (73.96 vs. 68.81; 75.87 vs. 72.09).
Furthermore, it brings another 2.2-point improvement to the CoNLL-2000
dataset. The superiority of chunk induction from the summarization task may be
due to the inherent connection between summarization and chunking. This is in
line with a summarization method known as phrasal extractive summarization
(Muresan et al., 2001; Xu and Zhao, 2022), where the salient information
(often key phrases) is extracted to generate summarized text.

###  3.4 Detailed Analyses

In this subsection, we conduct comprehensive experiments to verify our
proposed approach. We will first show an analysis of the proposed heuristic
for chunk label induction from unsupervised parsers for pretraining HRNN.
Then, we will show an ablation study on the proposed HRNN structure. Finally,
we will analyze the effectiveness of finetuning HRNN on downstream text
generation tasks.

Chunking Heuristics | Phrase F1 | Tag Acc.
---|---|---
1-word & 2-word chunks | 55.72 | 75.14
Maximal right branching | 40.83 | 69.28
Maximal left branching | 62.89 | 81.64

Table 3: Analysis of the heuristics for inducing chunking labels from Compound
PCFG.

# | Method | Phrase F1 | Tag Acc.
---|---|---|---
1 | Compound PCFG chunker | 62.89 | 81.64
2 |  $\longrightarrow$ HRNN only | 65.01 | 82.22
3 |  $\longrightarrow$ BERT+1-layer RNN | 67.19 | 83.86
4 |  $\longrightarrow$ BERT+2-layer RNN | 66.53 | 83.34
5 |  $\longrightarrow$ BERT+HRNN (hard) | 67.90 | 83.80
6 |  $\longrightarrow$ BERT+HRNN | 68.12 | 83.90

Table 4: Ablation study of the HRNN model.

####  3.4.1 Analysis of the Left-Branching Chunking Heuristic

We provide a detailed analysis of our maximal left-branching chunking
heuristic, where we used the CoNLL-2000 dataset as a testbed because we
pretrained all HRNN variants on CoNLL-2000. Without loss of generality, we
conducted this analysis with BERT-based HRNN.

Table 4 shows a comparison between different heuristics for inducing chunks
from parse trees. We observe that our maximal left-branching heuristic
outperforms right branching by 20 points in phrase F1. Additionally, we
introduce a thresholding approach that only extracts one-word and two-word
chunks since most of the groundtruth chunks contain one or two words. The
performance of such a heuristic is better than right-branching but worse than
our left-branching approach. Our findings support our hypothesis that right-
branching is a common structure in English and does not suggest meaningful
chunks. Conversely, left-branching identifies closely related words and is an
effective heuristic for inducing chunks from parse trees.

####  3.4.2 Ablation Study on the HRNN Architecture

Table 4 presents an ablation study on the HRNN model. Here, we only consider
pretraining HRNN from unsupervised parsers, while excluding the phase of
finetuning on downstream tasks. This allows us to eliminate distracting
factors in this ablation study.

We experimented with various neural architectures for the chunker, which
learns from Compound PCFG with the maximal left-branching heuristic. We see
all variants outperform the Compound PCFG chunker (Lines 2–6 vs. Line 1 in
Table 4), demonstrating that machine learning models can effectively smooth
out noise and mitigate the imperfection of chunk heuristics.

When we compare the traditional RNN with HRNN, the latter’s switching gate
provides autoregressive information. Our HRNN performs better with a
controlled number of layers (Lines 4 vs. 6). Furthermore, the soft HRNN
surpasses the hard HRNN (Lines 5 vs. 6), indicating its enhanced capability to
manage chunk ambiguity and provide superior autoregressive information.
Additionally, integrating HRNN with a Transformer encoder such as BERT (Lines
2 vs. 6) is beneficial, as it can capture global contextual information.

####  3.4.3 Analysis of Finetuning

Chunking structure discovery with finetuning HRNN on downstream tasks requires
several treatments: 1) We need to pretrain the HRNN, whose architecture has
been analyzed in Section 3.4.2, and 2) We need to apply an auxiliary loss push
HRNN’s predicted chunking probability away from $0.5$, as presented in
Equation (14). We analyze the effect of pretraining and the auxiliary loss in
this part.

Figure 2: Ablation study of our HRNN finetuning method. We plot the learning
curves in terms of phrasal F1 (left) and tag accuracy (right).

Figure 2 presents the learning curve on the downstream dataset Gigward. As
seen, the HRNN with both pretraining and auxiliary loss is the only
configuration that achieves an improved chunking performance (green curves
with dots), whereas other settings (without pretraining and/or without
auxiliary loss) yield worse performance. This ablation study underscores the
significance of both the pretraining strategy and auxiliary loss for HRNN’s
chunking structure discovery in downstream tasks.

Figure 3: This analysis examines the effect of the chunking strength
hyperparameter $\kappa$ on the actual chunking ratio. The dashed line
represents the groundtruth chunking ratio in the dataset. The purple lines
indicate that the finetuned chunking performance outperforms the pretrained
model. Additionally, the color depth and width of these lines indicate the
ranking of the chunking performance achieved, with deeper and wider lines
indicating better performance.

Moreover, the auxiliary loss provides the opportunity to control the chunking
granularity. This can be shown by varying the chunking strength hyperparameter
$\kappa$ in Equation (14) from $0$ to $1$ in increments of $0.1$. The results,
presented in Figure 3, confirm that $\kappa$ is indeed able to control the
ratio of predicted cuts (i.e., chunking granularity), although the percentages
may not match as there could be other factors contributing to the model’s
behavior. Interestingly, closely aligning the predicted and groundtruth
chunking ratios does not lead to better performance. The optimal $\kappa$
value turns out to be $0.5$ in our experiments, even though the predicted
ratio slightly exceeds the groundtruth ratio.

We also notice an intriguing phenomenon in Figure 2: all models converge to a
phrase F1 score of approximately $40$. By manually checking their predictions,
we realize that the models will eventually predict a cut for almost all steps,
e.g., every word is a chunk by itself. Even with our full approach (green
curves), the chunking performance is only improved within limited training
steps, after which it drops as well. This raises a curious question about the
dynamics of the chunking structure during the process of downstream-task
finetuning.

Figure 4: Learning curves of finetuning HRNN with text generation tasks.
Phrasal F1 and accuracy reflect the chunking performance, while the Rouge1
score measures the text generation performance. The gray dashed lines mark the
steps when HRNN achieves the highest chunking performance.

To further investigate this, we show in Figure 4 the learning curves of both
chunking performance and the downstream-task performance, evaluated by Rouge1
score (Lin, 2004). The learning curves show that the emerging of meaningful
chunking structures is a transient phenomenon only during the early stage of
downstream-task learning. The chunking performance drops after several
thousand iterations, while the downstream-task performance continues to
increase monotonically.

We hypothesize that chunking information (or linguistic information in
general) is a meaningful abstraction of human languages. When our exquisitely
designed model has not mastered the downstream NLP task well (e.g., at an
early stage), it will produce structures akin to linguistically defined
chunks, which serve as a convenient intermediate step for the downstream task.
As training proceeds, however, such linguistically meaningful structures are
abandoned by the model, as deep neural networks are known to achieve high
performance with end-to-end unexplainable features (Wu et al., 2023). Our work
sheds light on future research in linguistics.

####  3.4.4 Case Studies

Figure 5: Examples of chunking structures generated by Compound PCFG,
pretrained HRNN, and finetuned HRNN are provided. Differences are highlighted
in thick blue. Groundtruth chunks are also displayed in green for reference.

In Figure 5, we show examples of chunking structures generated by Compound
PCFG, pretrained HRNN, and finetuned HRNN in the summarization task.

As seen, our pretrained HRNN produces longer noun phrases than Compound PCFG,
such as Carlyle Group and a merchant banking concern in Example (c), and
finetuning HRNN can further detect longer noun phrases—such as certain estate-
tax returns in Example (a)—which are usually considered as a chunk by
groundtruth labels.

Furthermore, our approach is able to correct nonsensical chunks produced by
Compound PCFG. In Example (a), the two-word chunk grants relief is split into
grants and relief as they are not in the same semantic unit. In Example (c),
the chunk In January he is first split into In January and he, which makes
more sense. Then the finetuned HRNN further splits In January into In and
January, which agree more with the groundtruth chunks.

In general, the pretrained HRNN not only effectively learns the chunking
patterns from Compound PCFG, but also can smooth out its noise. The finetuned
HRNN model further improves the pretrained model and achieves better
performance of unsupervised chunking.

##  4 Related Work

This section briefly discusses recent advances in chunking and unsupervised
grammar induction approaches, followed by recent literature on weakly
supervised neuro-symbolic methods.

###  4.1 The Chunking Task

Chunking is a prominent task in natural language processing with various
applications, such as named entity recognition (Wang et al., 2013), syntactic
analysis (Zhou et al., 2012), and fine-grained sentiment analysis (Yang and
Cardie, 2013). Chunking is also referred to as shallow parsing, where the goal
is to separate a text into non-overlapping chunks that are syntactically
connected (Abney, 1991).

The CoNLL-2000 shared task (Tjong Kim Sang and Buchholz, 2000), derived from
the Penn Treebank (Marcus et al., 1993), has provided a platform for system
comparison of English chunking research. In the early stage of research,
various classification models, such as support vector machines (SVMs, Kudo and
Matsumoto, 2001) and the Winnow algorithm (Zhang et al., 2002), are applied to
accomplish the chunking task. Additionally, sequential labeling models are
extensively used for chunking; examples include the hidden Markov model (HMM)
(Molina and Pla, 2002) and conditional random fields (CRFs, Sha and Pereira,
2003; McDonald et al., 2005). Sun et al. (2008) adopt latent-dynamic
conditional random fields to the chunking task, which leads to further
performance improvement.

Zhou et al. (2012) propose to use chunk-level features for the chunking task,
outperforming most previous research based on word-level features (Kudo and
Matsumoto, 2001; Zhang et al., 2002; Molina and Pla, 2002; McDonald et al.,
2005; Sun et al., 2008). Inspired by this, we design a hierarchical RNN that
captures chunk features that are formed dynamically. We further leverage
contextual word representations from a language model and enhance performance
through weak supervision from downstream tasks.

###  4.2 Unsupervised Syntactic Structure Induction

Detecting syntactic structures without supervision has attracted significant
interest due to its applications in scenarios with limited resources (Clark,
2001; Klein, 2005).

Unsupervised parsing is one of the most widely explored areas in syntactic
structure detection. It can be broadly divided into three areas: unsupervised
constituency parsing (Klein and Manning, 2002; Haghighi and Klein, 2006;
Reichart and Rappoport, 2008; Clark, 2001), which aims to derive a
constituency parse tree for a given sentence; unsupervised dependency parsing
(Seginer, 2007; Paskin, 2001), which focuses on identifying the grammatical
relationships between words in a sentence; and unsupervised joint parsing
(Klein and Manning, 2004; Shen et al., 2018), which seeks to combine both
constituency and dependency parsing into a unified framework.

Among these settings, unsupervised constituency parsing has been explored
most. Klein and Manning (2002) propose an approach that utilizes an
expectation–maximization (EM) algorithm to model the constituency of each
span. Haghighi and Klein (2006) propose to employ a probabilistic context-free
grammar (PCFG) based on manually designed features. Kim et al. (2019)
introduce the Compound PCFG for unsupervised parsing. On the other hand,
methods have been developed to obtain tagged parse trees without supervision.
Reichart and Rappoport (2008) perform clustering by syntactic features to
obtain tagged parse trees. Clark (2001) clusters sequences of tags based on
their local mutual information to build tagged parse trees. Overall, these
early studies mostly rely on heuristics, linguistic knowledge, and manually
designed features.

Unsupervised parsing has revived interest in the deep learning era. Socher et
al. (2011) propose a recursive autoencoder that constructs a binary tree by
greedily minimizing the reconstruction loss. Other methods for learning
recursive tree structures in an unsupervised manner include CYK-style
marginalization (Drozdov et al., 2019) and Gumbel-softmax (Choi et al., 2018).
Yogatama et al. (2017) present a shift–reduce parser learned by reinforcement
learning to improve performance on a downstream task such as textual
entailment. However, evidence shows the above approaches do not yield
linguistically plausible trees (Williams et al., 2018a).

Li et al. (2019a) propose to transfer knowledge between multiple unsupervised
parsers to improve their performance. Our pretraining method draws inspiration
from such knowledge transfer, but we introduce insightful heuristics that
generate chunk labels from unsupervised parsers. Additionally, we have
developed a hierarchical recurrent neural network (HRNN) that learns from the
chunk labels generated by our heuristics, and can further be finetuned with
the downstream tasks for performance improvement.

Regarding unsupervised chunking, the speech processing community has addressed
the task as a component of speech detection systems and utilized acoustic
information to determine the chunks (Pate and Goldwater, 2011; Barrett et al.,
2018). Our work only relies on textual information and views unsupervised
chunking as a new task of inducing syntactic structure.

###  4.3 Weakly Supervised Neuro-Symbolic Methods

In recent years, neuro-symbolic approaches have gained significant attention
from the AI and NLP communities for interpreting deep learning models (Liu et
al., 2023). Many of these approaches treat neuro-symbolic methods as combining
deep learning and symbolic representations. For example, Lei et al. (2016)
extract key phrases that rationalize the neural model’s predictions on
classification tasks. Liu et al. (2018) model text classification as a
sequential decision process, and propose a policy network that is able to skip
reading and make decisions. Liang et al. (2017) and Mou et al. (2017) perform
SQL-like execution based on input text for semantic parsing.

However, training a neuro-symbolic model is more complicated than an end-to-
end neural model, because the explainable discrete latent structure lacks
ground truth for direct supervision. As a result, these approaches are
typically trained in a weakly supervised manner, where the training signals
only exist at the end of the entire model. Reinforcement learning or its
relaxation, such as Gumbel-softmax (Jang et al., 2017), is often used in the
training procedure due to the non-differentiable latent structure. Recently,
Wu et al. (2023) propose neural fuzzy logic to explain the predictions of the
natural language inference model, which is trained end-to-end. Our HRNN model
uses switching gates as the intermediary of the weak supervision from text
generation for chunk induction, which allows it to be trained with different
downstream datasets.

##  5 Limitations and Future Work

One limitation of this paper is that we have only applied our approach to the
English language, as our HRNN chunker is pretrained by unsupervised parsers
(which are mainly built for English) and our maximal left-branching heuristic
for chunk induction utilizes the right-branching prior of English. To
generalize unsupervised grammar induction to other languages, we may develop
models that incorporate language-specific prior, e.g., left branching for
Japanese (Martin, 2003). Another viable path is to translate a sentence into
English, perform unsupervised syntactic structure prediction on the English
translation, and map the structure back to the original language. Extending
this, multilingual unsupervised grammar induction is also a promising research
direction.

Our long-term vision is to build an interpretable NLP system, trained in an
end-to-end fashion, that can detect semantic units (chunks), determine their
relationships, perform task-specific reasoning, and ultimately achieve high
performance for the task of interest (Liu et al., 2023). The main obstacle to
this ambitious goal is the difficulty of end-to-end training: this paper
performs chunking during text generation, but the obtained chunking structure
is later abandoned by the model; another study of ours performs chunk
detection and alignment by heuristics, after which a differentiable fuzzy
logic mechanism is designed to perform logical reasoning (Wu et al., 2023). An
important future direction is to put these components together and make
traditional NLP pipelines end-to-end learnable (Sutskever et al., 2014). We
anticipate that, with proper development, such a neuro-pipeline approach will
achieve high performance in end tasks similar to black-box neural networks,
while excelling in terms of explainablility and interoperability.

##  6 Conclusion

In this paper, we propose a framework to induce chunks in an unsupervised
manner as syntactic structure discovery. Specifically, we propose a
hierarchical RNN (HRNN) with switching gates to learn from the chunk labels
induced by a state-of-the-art unsupervised parser. The HRNN is further
finetuned with various downstream text generation tasks to achieve better
chunking results.

The experiments show that our approach largely improves unsupervised chunking.
Additionally, we provide comprehensive analyses of our HRNN, the pretraining
stage using unsupervised parser-induced labels, as well as the finetuning
stage in a downstream task. We also point out limitations and discuss future
directions.

## References

  * Abney (1991) Steven P Abney.  Parsing by chunks.  In _Principle-based Parsing_ , pages 257–278. Springer, 1991.  URL https://link.springer.com/chapter/10.1007/978-94-011-3474-3_10.
  * Baggenstoss (2001) Paul M Baggenstoss.  A modified baum-welch algorithm for hidden markov models with multiple observation spaces.  _IEEE Transactions on Speech and Audio Processing_ , pages 411–416, 2001.  URL https://ieeexplore.ieee.org/document/859060.
  * Barrett et al. (2018) Maria Barrett, Ana Valeria González-Garduño, Lea Frermann, and Anders Søgaard.  Unsupervised induction of linguistic categories with records of reading, speaking, and writing.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2028–2038, 2018.  URL https://aclanthology.org/N18-1184.
  * Bird and Loper (2004) Steven Bird and Edward Loper.  NLTK: The natural language toolkit.  In _Proceedings of the ACL Interactive Poster and Demonstration Sessions_ , pages 214–217, 2004.  URL https://aclanthology.org/P04-3031.
  * Bojar et al. (2014) Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna.  Findings of the 2014 Workshop on Statistical Machine Translation.  In _Proceedings of the Workshop on Statistical Machine Translation_ , 2014.  URL https://aclanthology.org/W14-3302.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre.  Improving language models by retrieving from trillions of tokens.  In _Proceedings of the International Conference on Machine Learning_ , pages 2206–2240, 2022.  URL https://proceedings.mlr.press/v162/borgeaud22a.html.
  * Choi et al. (2018) Jihun Choi, Kang Min Yoo, and Sang-goo Lee.  Learning to compose task-specific tree structures.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 5094–5101, 2018.  URL https://ojs.aaai.org/index.php/AAAI/article/view/11975.
  * Clark (2001) Alexander Clark.  Unsupervised induction of stochastic context-free grammars using distributional clustering.  In _Proceedings of the Workshop on Computational Natural Language Learning_ , 2001.  URL https://aclanthology.org/W01-0713.
  * Deshmukh et al. (2021) Anup Anand Deshmukh, Qianqiu Zhang, Ming Li, Jimmy Lin, and Lili Mou.  Unsupervised chunking as syntactic structure induction with a knowledge-transfer approach.  In _Findings of the Association for Computational Linguistics: EMNLP_ , pages 3626–3634, 2021.  URL https://aclanthology.org/2021.findings-emnlp.307.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4171–4186, 2019.  URL https://aclanthology.org/N19-1423.
  * Drozdov et al. (2019) Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit Iyyer, and Andrew McCallum.  Unsupervised latent tree induction with deep inside-outside recursive auto-encoders.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1129–1141, 2019.  URL https://aclanthology.org/N19-1116.
  * Firoozeh et al. (2020) Nazanin Firoozeh, Adeline Nazarenko, Fabrice Alizon, and Béatrice Daille.  Keyword extraction: Issues and methods.  _Natural Language Engineering_ , 26:259–291, 2020.  URL https://www.cambridge.org/core/journals/natural-language-engineering/article/keyword-extraction-issues-and-methods/84BFD5221E2CA86326E5430D03299711.
  * Gillenwater et al. (2010) Jennifer Gillenwater, Kuzman Ganchev, João Graça, Fernando Pereira, and Ben Taskar.  Sparsity in dependency grammar induction.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pages 194–199, 2010.  URL https://aclanthology.org/P10-2036.
  * Haghighi and Klein (2006) Aria Haghighi and Dan Klein.  Prototype-driven grammar induction.  In _Proceedings of the International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics_ , pages 881–888, 2006.  URL https://aclanthology.org/P06-1111.
  * Izacard et al. (2022) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave.  Few-shot learning with retrieval augmented language models.  _arXiv preprint arXiv:2208.03299_ , 2022.
  * Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  In _International Conference on Learning Representations_ , 2017.  URL https://openreview.net/forum?id=rkE3y85ee.
  * Jurafsky and Martin (2000) Daniel Jurafsky and James H. Martin.  _Speech and Language Processing_.  Prentice Hall, 2000.
  * Kim et al. (2020) Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang goo Lee.  Are pre-trained language models aware of phrases? Simple but strong baselines for grammar induction.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=H1xPR3NtPB.
  * Kim et al. (2019) Yoon Kim, Chris Dyer, and Alexander Rush.  Compound probabilistic context-free grammars for grammar induction.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pages 2369–2385, 2019.  URL https://aclanthology.org/P19-1228.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations_ , 2015.  URL https://arxiv.org/abs/1412.6980.
  * Klein (2005) Dan Klein.  _The Unsupervised Learning of Natural Language Structure_.  Stanford University, 2005.  URL https://nlp.stanford.edu/~manning/dissertations/Klein-Dan-thesis.pdf.
  * Klein and Manning (2004) Dan Klein and Christopher Manning.  Corpus-based induction of syntactic structure: Models of dependency and constituency.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pages 478–485, 2004.  URL https://aclanthology.org/P04-1061.
  * Klein and Manning (2002) Dan Klein and Christopher D. Manning.  A generative constituent-context model for improved grammar induction.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pages 128–135, 2002.  URL https://aclanthology.org/P02-1017.
  * Kudo and Matsumoto (2001) Taku Kudo and Yuji Matsumoto.  Chunking with support vector machines.  In _Proceedings of the North American Chapter of the Association for Computational Linguistics_ , 2001.  URL https://aclanthology.org/N01-1025.
  * Lei et al. (2016) Tao Lei, Regina Barzilay, and Tommi Jaakkola.  Rationalizing neural predictions.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , pages 107–117, 2016.  URL https://aclanthology.org/D16-1011.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, 2020.  URL https://aclanthology.org/2020.acl-main.703.
  * Li et al. (2019a) Bowen Li, Lili Mou, and Frank Keller.  An imitation learning approach to unsupervised parsing.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pages 3485–3492, 2019a.  URL https://aclanthology.org/P19-1338.
  * Li et al. (2019b) Zichao Li, Xin Jiang, Lifeng Shang, and Qun Liu.  Decomposable neural paraphrase generation.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pages 3403–3414, 2019b.  URL https://aclanthology.org/P19-1332.
  * Liang et al. (2017) Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. Forbus, and Ni Lao.  Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pages 23–33, 2017.  URL https://aclanthology.org/P17-1003.
  * Lin (2004) Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, 2004.  URL https://aclanthology.org/W04-1013.
  * Liu et al. (2022) Puyuan Liu, Chenyang Huang, and Lili Mou.  Learning non-autoregressive models from search for unsupervised sentence summarization.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pages 7916–7929, 2022.  URL https://aclanthology.org/2022.acl-long.545.
  * Liu et al. (2018) Xianggen Liu, Lili Mou, Haotian Cui, Zhengdong Lu, and Sen Song.  JUMPER: Learning when to make classification decisions in reading.  In _Proceedings of the International Joint Conference on Artificial Intelligence_ , pages 4237–4243, 2018.  URL https://dl.acm.org/doi/10.5555/3304222.3304359.
  * Liu et al. (2020a) Xianggen Liu, Lili Mou, Fandong Meng, Hao Zhou, Jie Zhou, and Sen Song.  Unsupervised paraphrasing by simulated annealing.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pages 302–312, 2020a.  URL https://aclanthology.org/2020.acl-main.28.
  * Liu et al. (2023) Xianggen Liu, Zhengdong Lu, and Lili Mou.  Weakly supervised reasoning by neuro-symbolic approaches.  In _Compendium of Neurosymbolic Artificial Intelligence_ , pages 665–692. IOS Press, 2023.  URL https://ebooks.iospress.nl/volumearticle/63739.
  * Liu (2019) Yang Liu.  Fine-tune BERT for extractive summarization.  _arXiv preprint arXiv:1903.10318_ , 2019.
  * Liu et al. (2020b) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742, 2020b.  URL https://aclanthology.org/2020.tacl-1.47.
  * Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz.  Building a large annotated corpus of English: The Penn Treebank.  _Computational Linguistics_ , pages 313–330, 1993.  URL https://aclanthology.org/J93-2004.
  * Martin (2003) Samuel E Martin.  _A Reference Grammar of Japanese_.  University of Hawaii Press, 2003.
  * McDonald et al. (2005) Ryan McDonald, Koby Crammer, and Fernando Pereira.  Flexible text segmentation with structured multilabel classification.  In _Proceedings of the Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing_ , pages 987–994, 2005.  URL https://aclanthology.org/H05-1124.
  * Molina and Pla (2002) Antonio Molina and Ferran Pla.  Shallow parsing using specialized HMMs.  _Journal of Machine Learning Research_ , 2:595–613, 2002.  URL https://dl.acm.org/doi/abs/10.5555/944790.944819.
  * Mou et al. (2017) Lili Mou, Zhengdong Lu, Hang Li, and Zhi Jin.  Coupling distributed and symbolic execution for natural language queries.  In _Proceedings of the International Conference on Machine Learning_ , pages 2518–2526, 2017.  URL https://proceedings.mlr.press/v70/mou17a.html.
  * Muresan et al. (2001) Smaranda Muresan, Evelyne Tzoukermann, and Judith L. Klavans.  Combining linguistic and machine learning techniques for email summarization.  In _Proceedings of the ACL Workshop on Computational Natural Language Learning_ , 2001.  URL https://aclanthology.org/W01-0719.
  * Niklaus et al. (2018) Christina Niklaus, Matthias Cetto, André Freitas, and Siegfried Handschuh.  A survey on open information extraction.  In _Proceedings of the International Conference on Computational Linguistics_ , pages 3866–3878, 2018.  URL https://aclanthology.org/C18-1326.
  * Paskin (2001) Mark Paskin.  Grammatical bigrams.  In _Advances in Neural Information Processing Systems_ , 2001.  URL https://proceedings.neurips.cc/paper/2001/file/89885ff2c83a10305ee08bd507c1049c-Paper.pdf.
  * Pate and Goldwater (2011) John Pate and Sharon Goldwater.  Unsupervised syntactic chunking with acoustic cues: Computational models for prosodic bootstrapping.  In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics_ , 2011.  URL https://aclanthology.org/W11-0603.
  * Pradhan et al. (2004) Sameer S. Pradhan, Wayne H. Ward, Kadri Hacioglu, James H. Martin, and Dan Jurafsky.  Shallow semantic parsing using support vector machines.  In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 233–240, 2004.  URL https://aclanthology.org/N04-1030.
  * Rabiner (1989) L.R. Rabiner.  A tutorial on hidden markov models and selected applications in speech recognition.  _Proceedings of the IEEE_ , pages 257–286, 1989.
  * Ramshaw and Marcus (1995) Lance Ramshaw and Mitch Marcus.  Text chunking using transformation-based learning.  In _Workshop on Very Large Corpora_ , 1995.  URL https://aclanthology.org/W95-0107.
  * Reichart and Rappoport (2008) Roi Reichart and Ari Rappoport.  Unsupervised induction of labeled parse trees by clustering with syntactic features.  In _Proceedings of the International Conference on Computational Linguistics_ , pages 721–728, 2008.  URL https://aclanthology.org/C08-1091.
  * Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston.  A neural attention model for abstractive sentence summarization.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , pages 379–389, 2015.  URL https://aclanthology.org/D15-1044.
  * Sato et al. (2017) Motoki Sato, Hiroyuki Shindo, Ikuya Yamada, and Yuji Matsumoto.  Segment-level neural conditional random fields for named entity recognition.  In _Proceedings of the International Joint Conference on Natural Language Processing_ , pages 97–102, 2017.  URL https://aclanthology.org/I17-2017.
  * Seginer (2007) Yoav Seginer.  Fast unsupervised incremental parsing.  In _Proceedings of the Annual Meeting of the Association of Computational Linguistics_ , pages 384–391, 2007.  URL https://aclanthology.org/P07-1049.
  * Sha and Pereira (2003) Fei Sha and Fernando Pereira.  Shallow parsing with conditional random fields.  In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 213–220, 2003.  URL https://aclanthology.org/N03-1028.
  * Shen et al. (2018) Yikang Shen, Zhouhan Lin, Chin wei Huang, and Aaron Courville.  Neural language modeling by jointly learning syntax and lexicon.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=rkgOLb-0W.
  * Shen et al. (2019) Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville.  Ordered neurons: Integrating tree structures into recurrent neural networks.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=B1l6qiR5F7.
  * Shen et al. (2021) Yikang Shen, Yi Tay, Che Zheng, Dara Bahri, Donald Metzler, and Aaron Courville.  Structformer: Joint unsupervised induction of dependency and constituency structure from masked language modeling.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics and International Joint Conference on Natural Language Processing_ , pages 7196–7209, 2021.  URL https://aclanthology.org/2021.acl-long.559.
  * Shi et al. (2020) Haoyue Shi, Karen Livescu, and Kevin Gimpel.  On the role of supervision in unsupervised constituency parsing.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , pages 7611–7621, 2020.  URL https://aclanthology.org/2020.emnlp-main.614.
  * Socher et al. (2011) Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning.  Semi-supervised recursive autoencoders for predicting sentiment distributions.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , pages 151–161, 2011.  URL https://aclanthology.org/D11-1014.
  * Sun et al. (2008) Xu Sun, Louis-Philippe Morency, Daisuke Okanohara, Yoshimasa Tsuruoka, and Jun’ichi Tsujii.  Modeling latent-dynamic in shallow parsing: A latent conditional model with improved inference.  In _Proceedings of the International Conference on Computational Linguistics_ , pages 841–848, 2008.  URL https://aclanthology.org/C08-1106.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  In _Advances in Neural Information Processing Systems_ , 2014.  URL https://proceedings.neurips.cc/paper_files/paper/2014/file/a14ac55a4f27472c5d894ec1c3c743d2-Paper.pdf.
  * Tjong Kim Sang and Buchholz (2000) Erik F. Tjong Kim Sang and Sabine Buchholz.  Introduction to the CoNLL-2000 shared task chunking.  In _Conference on Computational Natural Language Learning and Learning Language in Logic Workshop_ , 2000.  URL https://aclanthology.org/W00-0726.
  * Van de Cruys (2011) Tim Van de Cruys.  Two multivariate generalizations of pointwise mutual information.  In _Proceedings of the Workshop on Distributional Semantics and Compositionality_ , pages 16–20, 2011.  URL https://aclanthology.org/W11-1303.
  * Wang et al. (2013) Mengqiu Wang, Wanxiang Che, and Christopher D Manning.  Joint word alignment and bilingual named entity recognition using dual decomposition.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pages 1073–1082, 2013.  URL https://aclanthology.org/P13-1106/.
  * Williams et al. (2018a) Adina Williams, Andrew Drozdov, and Samuel R. Bowman.  Do latent tree learning models identify meaningful structure in sentences?  _Transactions of the Association for Computational Linguistics_ , 6:253–267, 2018a.  URL https://aclanthology.org/Q18-1019.
  * Williams et al. (2018b) Adina Williams, Nikita Nangia, and Samuel Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1112–1122, 2018b.  URL https://aclanthology.org/N18-1101.
  * Wu et al. (2023) Zijun Wu, Zi Xuan Zhang, Atharva Naik, Zhijian Mei, Mauajama Firdaus, and Lili Mou.  Weakly supervised explainable phrasal reasoning with neural fuzzy logic.  In _International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=Hu4r-dedqR0.
  * Xu and Zhao (2022) Wang Xu and Tiejun Zhao.  Jointly learning guidance induction and faithful summary generation via conditional variational autoencoders.  In _Findings of the Association for Computational Linguistics: NAACL_ , pages 2340–2350, 2022.  URL https://aclanthology.org/2022.findings-naacl.180.
  * Yang and Cardie (2013) Bishan Yang and Claire Cardie.  Joint inference for fine-grained opinion extraction.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pages 1640–1649, 2013.  URL https://aclanthology.org/P13-1161/.
  * Yogatama et al. (2017) Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Wang Ling.  Learning to compose words into sentences with reinforcement learning.  In _International Conference on Learning Representations_ , 2017.  URL https://openreview.net/forum?id=Skvgqgqxe.
  * Zhang et al. (2002) Tong Zhang, Fred Damerau, and David Johnson.  Text chunking based on a generalization of winnow.  _Journal of Machine Learning Research_ , pages 615–637, 2002.  URL https://jmlr.csail.mit.edu/papers/v2/zhang02c.html.
  * Zhou et al. (2012) Junsheng Zhou, Weiguang Qu, and Fen Zhang.  Exploiting chunk-level features to improve phrase chunking.  In _Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pages 557–567, 2012.  URL https://aclanthology.org/D12-1051/.


ArnavKundu∗ PrateethNayak∗ HywelRichards PriyankaPadmanabhan DevangNaik

# RepCNN: Micro-sized, Mighty Models for Wakeword Detection

###### Abstract

Always-on machine learning models require a very low memory and compute
footprint. Their restricted parameter count limits the model’s capacity to
learn, and the effectiveness of the usual training algorithms to find the best
parameters. Here we show that a small convolutional model can be better
trained by first refactoring its computation into a larger redundant multi-
branched architecture. Then, for inference, we algebraically re-parameterize
the trained model into the single-branched form with fewer parameters for a
lower memory footprint and compute cost. Using this technique, we show that
our always-on wake-word detector model, RepCNN, provides a good trade-off
between latency and accuracy during inference. RepCNN re-parameterized models
are 43% more accurate than a uni-branch convolutional model while having the
same runtime. RepCNN also meets the accuracy of complex architectures like BC-
ResNet, while having 2x lesser peak memory usage and 10x faster runtime.

###### keywords:

speech recognition, efficientML

**footnotetext: Equal Contribution

##  1 Introduction

The design and development of efficient machine learning models for always-on
mobile applications have progressed significantly, witnessing an overall trend
of reduced FLOPs (floating-point operations) and memory usage while
maintaining or improving accuracy [1, 2, 3]. This poses a challenge of
achieving higher accuracy within limited memory and compute. Specifically
related to speech and vision applications on mobile devices, novel
architecture optimizations [4, 5, 6] are proposed to achieve high accuracy,
but their components, such as residual connections make the inference graph
complex (although FLOP is low) and lead to increased runtime memory usage and
end-to-end inference latency [7, 8]. This makes it challenging to deploy these
models on resource-constrained devices, ultimately leading to higher latency
or unsupported configurations. Due to these strict memory and compute demands,
existing models developed for always-on keyword spotting have often been
simplistic inference graphs [9, 10, 11].

Although these simpler models are efficient in terms of memory usage, they
often fall short in achieving comparable accuracy to more complex multi-
branched architectures. One reason for this is that multi-branch topology,
like ResNet, create an implicit ensemble of shallower models. This ensemble
effect inherently mitigates the gradient vanishing problem, which can be a
challenge for single-branch models, allowing more accurate predictions [12].
To this end, we propose a re-parameterizable fully convolutional encoder
architecture (RepCNN) that is:

  * •

Multi-branched during training to accommodate easier gradient flow.

  * •

Simple single branched inference graph to ensure less compute and memory usage
during inference execution flow.

Considering that the benefits of multi-branch architectures are predominantly
seen during training and contribute to increased inference costs, we have
proposed a novel approach to decouple the training and inference model graphs
for keyword-spotting. By doing so, we can leverage the advantages of multi-
branch training while reducing its impact on inference. Further, we
hypothesize that using multiple identical kernels in parallel could introduce
new learning paths, potentially enhancing the model's accuracy. During
inference, these redundant kernels can be combined into a single kernel,
without any increase in latency.

##  2 Related Works

Traditional wake-word detection approaches are largely based on Hidden Markov
Models which includes acoustic DNN model that computes phoneme probabilities
followed usually by viterbi-based HMM decoder with output observation
scores[9, 13, 14, 15, 16], but these systems often suffer from loss-metric
mismatches and are difficult to train. With advances in recurrent
architectures, there have been complex networks[17, 18, 19] which are easier
to train and more accurate, however are not suitable for mobile inference due
to high latency concerns. With accuracy gains and decent computational
complexities, several End-to-End CNN-based models[2, 12, 4, 1, 20] developed
for mobile-vision applications provide a great motivation to apply such
architectures to wake-word detection. Additionally, convolutional models have
the advantage of having streaming audio with input audio length being
arbitrary. In this direction, there have been specific single-path CNN-based
hand-crafted architectures[10, 11, 21] that have shown good performance at low
latency and runtime memory usage. However, only with complex architectures[3,
22, 23, 24] with high parameter count, bottlenecks or skip-connections we are
able to see huge gain in accuracy. Novel residual connections[5] in these work
provide further gains, but presence of these residual structures during
inference on mobile-applications not only adds computational overhead, but
also increase runtime memory size required on-device to store these branched
intermediate outputs. Models with such inference graphs become unsuitable when
operating in strict memory and computation budget.

Recently, structural re-parameterization has shown significant promise in the
field of computer vision with [8, 7, 25, 26]. These models use a multi-
branched architecture during training and are converted to a single branched
architecture during inference. Motivated by these methods, we extend the
discriminative training work[6, 11, 14] with the approach of multi-branch
topology for 1D depth-wise convolutions during training to over-parameterize
and increase network capacity, and during inference we linearly combine the
convolutional parameters to a trivial architecture that has lesser parameters
as well as lower latency and memory usage.

##  3 Model Architecture

As discussed earlier, conventional single branch architectures have poor
performance, primarily because of being difficult to train. Multi-branch
architectures on the other hand are easier to train while being slower for on-
device applications. This is true despite having lower theoretical FLOPs,
primarily because of memory access cost [4]. Therefore, we use structural re-
parameterization to have different training and inference graphs. This section
explains our model architecture in detail that incorporates the benefits of
multi-branch training graph and converting to a simple inference graph.

###  3.1 RepConvBlock

Figure 1: RepConvBlock: Re-parameterizable Convolutional Block

Based on the principle of structural re-parameterization, we introduce Re-
parameterizable Convolutional Block (RepConvBlock) as shown in Figure 1. This
acts as the foundational element of our work with the training graph shown on
the left and inference graph shown on the right. Since we are focusing on tiny
ML models for speech processing, the RepConvBlock is designed for 1-D depth-
wise separable convolution to be compute and memory efficient [8]. The
RepConvBlock consists of 3 architectural elements during training as described
below:

  * •

Multiple convolutional kernels of the same size [8], followed by batch
normalization layers in parallel in the training graph. This is done to over
parameterize the training graph which has proven to be beneficial in prior
works [8, 7].

  * •

1x1 convolutional kernel followed by a batch normalization layer parallel to
the kernels above to emulate a skip connection.

  * •

ReLU layers as non-linearity.

During inference the parallel convolutional and batch normalization layers are
replaced by an equivalent fused convolutional layer. The fusion process is
described in Section 3.2. We also employ non-linear layers that are replaced
by a clip layer to keep the activations bounded.

###  3.2 Reparameterization

A 1-D convolutional layer parameterized by $w_{k}$, followed by a batch
normalization layer, during inference can be re-parameterized into another 1-D
convolutional layer parameterized by $(W_{k},W_{b})$ as shown in Equation 1.

$\begin{split}y&=bn(\text{Conv}(x))\\\
&=\frac{\alpha_{bn}}{\sigma_{bn}}\times\text{Conv}(x)-\frac{\mu_{bn}}{\sigma_{bn}}+\beta_{bn}\\\
&=\frac{\alpha_{bn}}{\sigma_{bn}}\times
w_{k}*x-\frac{\mu_{bn}}{\sigma_{bn}}+\beta_{bn}\\\ &=W_{k}*x+W_{b}\end{split}$
(1)

Here $k$ is the size of the kernel, $W_{b}$ is the bias of the re-
parameterized kernel. $\alpha,\beta,\mu,\sigma$ correspond to the weight,
bias, tracked mean and standard deviation for the batch normalization layer,
respectively. We re-parameterize individual branches into equivalent
convolutional kernels parameterized by $W_{k}^{i}$.

Consequently, parallel convolutional kernels of size $k$ can be fused together
into a single kernel by simply adding the corresponding weights. The branch
with $\text{kernel size}=1$ can be represented as a kernel of size $k$ where
all weights except the central element are zeros. The re-parameterization of
the parallel branches $\\{W_{1},W_{k}^{1},W_{k}^{2}...\\}$ is shown in
Equation 2. We get a single convolutional kernel ($\hat{W_{k}}$) to represent
the combination of multiple parallel branches as shown in Figure 1.

$\begin{split}y&=W_{1}*x+\Sigma_{i}(W_{k}^{i}*x)\\\
&=W_{1}*x+\Sigma_{i}(W_{k}^{i})*x\\\
&=\underbrace{[0,\cdots,W_{1},\cdots,0]}_{k-elements}+(\Sigma_{i}W_{k}^{i})*x\\\
&=\hat{W_{k}}*x\end{split}$ (2)

Figure 2: RepCNN training architecture. The RepConvBlocks are parameterized as
(input-channels, output-channels, kernel-size, num-branches). Here p=q=2 i.e.,
block1 and block2 are stacked twice. $block1_{1}$, $block1_{2}$, $block2_{1}$,
$block2_{2}$ use kernel-size as $k=7,9,11,13$ respectively. non linearity =
ReLU.

###  3.3 RepCNN

The design of the RepCNN model during training is illustrated in Figure 2. The
model comprises of a regular 1D-convolutional layer with kernel size 5 and a
stride of 2, followed by a 1D-batch normalization layer and a ReLU layer. The
RepConvBlocks are used in block1 and block2. In both of these modules we stack
2 RepConvBlocks next to one another with the same kernel size followed by a
point-wise convolution, batch normalization and ReLU. We use relatively larger
kernels in these models with modules of block1 type using kernels of $k=5$ &
$k=7$, and modules of block2 type using kernels of $k=11$ & $k=13$
respectively. We can use these larger kernels without significant increase in
runtime latency and memory usage because of absence of branching during
inference, which avoid data movement in between nodes of the model as in [8].

##  4 Experiments and Results

We apply our method for detection of a given wakeword like "Alexa" or "Hey
Google" in streaming speech. The following sections will provide further
details on our dataset, training strategies, results, and ablation studies
that delve into the impact of various architectural choices made in this
paper.

###  4.1 Datasets

We have constructed a robust training dataset of 1M audio utterances, each of
which includes the wake-word followed by a user query. These audio samples
have been meticulously enhanced to mimic real-world acoustic conditions using
room-impulse responses (RIRs) along with echo residuals. To augment the
training data and improve model robustness, we have included ambient noise
samples from various acoustic environments. Furthermore, we applied a diverse
range of training-time gain augmentations, varying from 10dB to -40dB, to
simulate various volume levels in real-world scenarios. For each utterance in
training data, we use an acoustic model to get per-frame forced alignments.
This provides us with the indices of start and end frame phonemes of the wake-
word. We define a positive sample as a window containing the entire wake-word
while ending with indices of the end frame. We also harvest random windows
with partial or no wake word in them as negative samples. For every positive
sample, we pick 20 random negative windows to mimic the real world probability
of wake word being present in streaming speech.

The positive test data encompass both near-field and far-field utterances,
clearly capturing the wake-word at 3ft and 6ft distances, providing an
extensive range of testing scenarios. The negative test data consists of
approximately 2000 hours of dense general speech, ensuring a thorough
evaluation of the model's ability to distinguish between relevant and
irrelevant cues. These datasets have been collected through internal user
studies with informed consent approvals.

###  4.2 Training and Evaluation

In order to conduct a comparative study, we trained three models with
different architectural designs. The first model is BC-ResNet[3], which serves
as an example of a multi-branch architecture and has the same training and
inference graph. The second model is a basic stacked 1D convolutional
model[11] which is used to represent a single branch architecture. The third
model is our proposed model, which utilizes stacked RepConvBlocks in place of
simple 1-D depth-wise convolutional layers during training and is re-
parameterized during evaluation to single-branch architecture. All these
models have roughly 15k parameters and a receptive field of 1.5 seconds to
ensure fair comparison.

For input featurization, we apply 26 Mel-Spectral Filter-Banks and extract 16
Mel-frequency cepstral coefficients (MFCCs) at a 10 ms frame rate for every 25
ms of audio, a commonly used approach in speech processing. We use focal-loss
[27] as the classification loss to handle the skewed distribution of psoitive
to negative samples in every batch. To make the best use of limited model
capacity, for every batch iteration we sort the loss values of the negative
samples and only pick the top-$K$ ($K=50$) hardest negative samples along with
all positive sample losses for back-propagation. We also repeat the training
with 3 different seeds and the results reported are the average of them.

Model | FRR (%) @ 3FA/hr
---|---
BC-ResNet[3] | 1.66
Single branch Arch. | 2.88
RepCNN | 1.66
Table 1: Comparative results of BC-ResNet, single branch architecture, and
RepCNN on wake-word detection at 3 FA/hr operating point

In our evaluation framework, a detected wake-word is considered a true-
positive(TP) if it has any overlap with the ground-truth segment. If there is
no overlap, it is counted as a false accept (FA). On the other hand, all
positive segments containing wake-word where the model fails to detect are
classified as false rejects (FR). We measure the False-Reject Rate (FRR) which
is proportion of samples containing the wake-word that were rejected from the
positive set. And, we measure the number of False-Activation per hour (FA/hr)
using the negative set. To compare between different model performances, we
report the FRR at a chosen operating point of 3 FA/hr. We also compute the
DET-curve characteristic of the different models.

Figure 3: DET curves for model architectures as compared to RepCNN (Orange).
We choose 3FA/hr as an ideal operating point for the detection.

###  4.3 Results

For comparison the results for all the candidate models are reported in Table
1. These results demonstrate that complex multi-branch architecture like BC-
ResNet[3] are approximately 43% more accurate than the single branched stacked
convolutional model. RepCNN is able to meet or beat this performance accuracy
at 3FA/hr operating point or lower, and the DET-curve shown in Figure 3 shows
the differences in accuracy at various operating points. We can see that
single branch architecture is worse than multi-branch architecture approaches.

To compare the overall performance of BC-ResNet vs RepCNN, we compute the ROC-
AUC of the models. Both BC-ResNet and RepCNN have 97.9% AUC score, however BC-
ResNet is better than RepCNN at higher FA/hr operating points. To validate
that our model has a faster inference speed, we have conducted a on-device
latency and memory profile of the BC-ResNet multi-branch architecture against
the RepCNN training graph, and the RepCNN re-parameterized model on the CPU of
an iPhone Xs. The results are presented in Table 2. The latency results show
that the complex inference graph of RepCNN re-parameterized is 10x faster than
BC-ResNet and consumes half the peak runtime memory usage.

Model | Latency (ms) | Peak memory (MB)
---|---|---
BC-ResNet[3] | $4.5$ | $1.2$
RepCNN Training graph (n=2) | $1.8$ | $0.8$
Re-parameterized RepCNN | 0.4 | 0.5
Table 2: Latency and runtime memory usage to produce one output from various
model architectures on iPhone Xs CPU.

This validates our approach that a multi-branch re-parameterizable model can
be designed such that it is as accurate as a complicated BC-ResNet model while
being more efficient to run on a device for always-on power constrained
applications like wake-word triggering. To test its extensibility, we verified
that on Google Speech commands [28], RepCNN achieves the same accuracy as
BCResNet with >50% faster runtime and 40% lesser memory.

###  4.4 Ablation study

To further validate that our approach is easier to train, we plot the training
and validation loss curves of RepCNN in Figure 4. For exactly same training
setup and hyper-parameters, it can be observed that RepCNN models have a
better training and validation loss curve trend compared to the no branch case
(a vanilla stacked 1D-CNN similar to [11]). This is realized across different
seeds of the training as well.

Figure 4: Plot of training and validation loss of RepCNN as compared to a no
branch architecture. Figure 5: Plot of validation loss of RepCNN with various
degrees of over-parameterization. Branches corresponds to the number of
parallel 1D depth-wise convolutional kernels apart from the kernel with size
1.

We also explore the effect of different number of branches in the multi-branch
topology during training and compare it against the single-branch as part of
the ablation study. Primarily, we are interested in comparing the validation
loss curves which is an indicator of the training health and also the final
performance metrics. In Figure 5, we verify that over-parameterization does
helps to train a better model by looking into the validation loss of various
branched versions of RepCNN versus the single-branch architecture. The
variations in the loss is due to randomness in sampling hard negative sub-
windows and varying augmentations across epochs during training. These models
have different degrees of over-parameterization introduced by the number of
parallel convolutional kernels (denoted as n in Figure 1) in RepConvBlock. The
final inference graph and parameter count for all these versions is the same
after training due to re-parameterization. Additionally, on evaluation we find
that using 2 branches gives the best FRR at chosen operating point while using
3,4 or 5 branches is always better than using 1 branch (single-branch) during
training as shown in Table 3. It can be observed that allowing multiple
branched paths helps to converge to a better model but beyond a limit such
redundancy does not tend to improve the metrics.

Num. branches | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
FRR (%) @ 3 FA/hr | 2.16 | 1.66 | 1.93 | 1.98 | 2.03
Table 3: Ablation study on the effect of accuracy with various levels of over-
parameterization

##  5 Conclusion

In this work, we have introduced structural re-parameterization of complex
multi-branch models to detect wake words accurately and fast. We have shown
that we can take advantage of multi-branch architectures during training
without impacting inference latency and runtime memory. We achieve it by
designing RepCNN which is a multi-branch architecture during training and is
re-parameterized into a single branch architecture during inference. We
demonstrate that RepCNN is 10x faster than BC-ResNet. It runs as fast as a
regular stacked 1D-CNN while being 43% better in accuracy. In our further work
in this direction, we plan to improve the gap in accuracy with complex
architectures, like BC-ResNet at higher FA operating points, through additions
like squeeze and excite components while ensuring simplistic inference graph.
Furthermore, we aim to explore compression techniques that help in better re-
parameterization between multi-branch convolutions.

## References

  * [1] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, ``Mobilenets: Efficient convolutional neural networks for mobile vision applications,'' _CoRR_ , vol. abs/1704.04861, 2017.
  * [2] A. Howard, M. Sandler, G. Chu, L. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, Q. V. Le, and H. Adam, ``Searching for mobilenetv3,'' _CoRR_ , vol. abs/1905.02244, 2019.
  * [3] B. Kim, S. Chang, J. Lee, and D. Sung, ``Broadcasted Residual Learning for Efficient Keyword Spotting,'' in _Proc. Interspeech 2021_ , 2021, pp. 4538–4542.
  * [4] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun, ``Shufflenet v2: Practical guidelines for efficient cnn architecture design,'' in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 116–131.
  * [5] J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, ``Squeeze-and-excitation networks,'' 2019.
  * [6] A. Kundu, M. Samragh, M. Cho, P. Padmanabhan, and D. Naik, ``Heimdal: Highly efficient method for detection and localization of wake-words,'' in _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5.
  * [7] X. Ding, X. Zhang, N. Ma, J. Han, G. Ding, and J. Sun, ``Repvgg: Making vgg-style convnets great again,'' in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021, pp. 13 733–13 742.
  * [8] P. K. A. Vasu, J. Gabriel, J. Zhu, O. Tuzel, and A. Ranjan, ``Mobileone: An improved one millisecond mobile backbone,'' in _CVPR_ , 2023. [Online]. Available: https://arxiv.org/abs/2206.04040
  * [9] S. Sigtia, R. Haynes, H. B. Richards, E. Marchi, and J. S. Bridle, ``Efficient voice trigger detection for low resource hardware,'' in _INTERSPEECH_ , 2018.
  * [10] T. Sainath and C. Parada, ``Convolutional neural networks for small-footprint keyword spotting,'' in _Interspeech_ , 2015.
  * [11] T. Higuchi, M. Ghasemzadeh, K. You, and C. Dhir, ``Stacked 1D Convolutional Networks for End-to-End Small Footprint Voice Trigger Detection,'' in _Proc. Interspeech 2020_ , 2020, pp. 2592–2596.
  * [12] A. Veit, M. J. Wilber, and S. Belongie, ``Residual networks behave like ensembles of relatively shallow networks,'' _Advances in neural information processing systems_ , vol. 29, 2016.
  * [13] I.-F. Chen and C.-H. Lee, ``A hybrid hmm/dnn approach to keyword spotting of short words.'' in _INTERSPEECH_ , 2013, pp. 1574–1578.
  * [14] R. Alvarez and H.-J. Park, ``End-to-end streaming keyword spotting,'' in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 6336–6340.
  * [15] R. Prabhavalkar, R. Alvarez, C. Parada, P. Nakkiran, and T. N. Sainath, ``Automatic gain control and multi-style training for robust small-footprint keyword spotting with deep neural networks,'' in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2015, pp. 4704–4708.
  * [16] A. Shrivastava, A. Kundu, C. Dhir, D. Naik, and O. Tuzel, ``Optimize what matters: Training dnn-hmm keyword spotting model using end metric,'' in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 4000–4004.
  * [17] S. Fernández, A. Graves, and J. Schmidhuber, ``An application of recurrent neural networks to discriminative keyword spotting,'' in _International Conference on Artificial Neural Networks_. Springer, 2007, pp. 220–229.
  * [18] C. Lengerich and A. Hannun, ``An end-to-end architecture for keyword spotting and voice activity detection,'' _arXiv preprint arXiv:1611.09405_ , 2016.
  * [19] K. Hwang, M. Lee, and W. Sung, ``Online keyword spotting with a character-level recurrent neural network,'' _arXiv preprint arXiv:1512.08903_ , 2015.
  * [20] M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen, ``Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation,'' _CoRR_ , vol. abs/1801.04381, 2018.
  * [21] R. Tang and J. Lin, ``Deep residual learning for small-footprint keyword spotting,'' in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 5484–5488.
  * [22] S. Choi, S. Seo, B. Shin, H. Byun, M. Kersner, B. Kim, D. Kim, and S. Ha, ``Temporal convolution for real-time keyword spotting on mobile devices,'' _CoRR_ , vol. abs/1904.03814, 2019.
  * [23] X. Li, X. Wei, and X. Qin, ``Small-footprint keyword spotting with multi-scale temporal convolution,'' 2020.
  * [24] S. Majumdar and B. Ginsburg, ``MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognition,'' in _Proc. Interspeech 2020_ , 2020, pp. 3356–3360.
  * [25] C. Chen, Z. Guo, H. Zeng, P. Xiong, and J. Dong, ``Repghost: a hardware-efficient ghost module via re-parameterization,'' _arXiv preprint arXiv:2211.06088_ , 2022.
  * [26] A. Wang, H. Chen, Z. Lin, H. Pu, and G. Ding, ``Repvit: Revisiting mobile cnn from vit perspective,'' _arXiv preprint arXiv:2307.09283_ , 2023.
  * [27] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, ``Focal loss for dense object detection,'' in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [28] P. Warden, ``Speech commands: A dataset for limited-vocabulary speech recognition,'' _ArXiv_ , vol. abs/1804.03209, 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:4719239