# Unsupervised Mismatch Localization in
Cross-Modal Sequential Data
with Application to Mispronunciations Localization

Wei Wei 222These authors contributed equally to this work<EMAIL_ADDRESS>
National University of Singapore Hengguan Huang 222These authors contributed
equally to this work<EMAIL_ADDRESS>
National University of Singapore Xiangming Gu<EMAIL_ADDRESS>
National University of Singapore Hao Wang<EMAIL_ADDRESS>
Rutgers University Ye Wang<EMAIL_ADDRESS>
National University of Singapore

###### Abstract

Content mismatch usually occurs when data from one modality is translated to
another, e.g. language learners producing mispronunciations (errors in speech)
when reading a sentence (target text) aloud. However, most existing alignment
algorithms assume that the content involved in the two modalities is perfectly
matched, thus leading to difficulty in locating such mismatch between speech
and text. In this work, we develop an unsupervised learning algorithm that can
infer the relationship between content-mismatched cross-modal sequential data,
especially for speech-text sequences. More specifically, we propose a
hierarchical Bayesian deep learning model, dubbed mismatch localization
variational autoencoder (ML-VAE), which decomposes the generative process of
the speech into hierarchically structured latent variables, indicating the
relationship between the two modalities. Training such a model is very
challenging due to the discrete latent variables with complex dependencies
involved. To address this challenge, we propose a novel and effective training
procedure that alternates between estimating the hard assignments of the
discrete latent variables over a specifically designed mismatch localization
finite-state acceptor (ML-FSA) and updating the parameters of neural networks.
In this work, we focus on the mismatch localization problem for speech and
text, and our experimental results show that ML-VAE successfully locates the
mismatch between text and speech, without the need for human annotations for
model training 111Codes will be soon available at https://github.com/weiwei-
ww/ML-VAE.

##  1 Introduction

Sequential data is prevalent in daily life and usually comes in multiple
modalities simultaneously, such as video with audio, speech with text, etc.
Research problems about multi-modal sequential data processing have attracted
great attention, especially on the alignment problem, such as aligning a video
footage with actions (Song et al., 2016) or poses (Kundu et al., 2020),
aligning speech with its text scripts (Chung et al., 2018), aligning audio
with tags (Favory et al., 2021), etc.

The content mismatch could come from various aspects. For example,
mispronounced words in speech or incorrect human annotation will cause
mismatch between speech and text; actors not following scripts will cause
mismatch between the video and the pre-scripted action list. Locating such
content mismatch is an important task and has many potential applications. For
example, locating the content mismatch between speech and text can help detect
mispronunciations produced by the speaker, which is crucial for language
learning.

In this work, we focus on the problem of mismatch localization between speech
and text inputs. In other words, i.e., to locate the mispronunciations in the
speech produced by a speaker. Such mispronunciation localization task plays an
important role in real-world applications such as computer-aided language
learning system, where the mispronunciation localization results can be used
to provide multimodal feedback for the users. Most existing cross-modal
alignment algorithms assume that data from a variety of modalities are matched
to each other (Song et al., 2016; Kundu et al., 2020; Chung et al., 2018;
Favory et al., 2021); this strong assumption leads to difficulty in locating
the mismatch between the two modalities. In particular, recent speech-text
alignment approaches (Kürzinger et al., 2020; McAuliffe et al., 2017) mostly
work under the assumption that the speaker has correctly pronounced all the
words and are therefore incapable of detecting mispronunciation. Although
earlier studies Finke & Waibel (1997); Hazen (2006); Braunschweiler et al.
(2010); Bell & Renals (2015) have considered the mismatch between speech and
text, they require human-annotated speech to train an acoustic model. However,
labeling such data is labor intensive and expensive. Similarly, traditional
mispronunciation detection methods (Leung et al., 2019) also need to be
trained on a large number of human-annotated speech samples from second
language (L2) speakers, whose annotation process is even more time-consuming
and requires professional linguists’ support. Furthermore, these studies on
mispronunciation detection can only detect which phonemes/words in the text
input are mispronounced, without locating them in the speech.

To address these issues, we propose a hierarchical Bayesian deep learning
model, dubbed mismatch localization variational autoencoder (ML-VAE), which
aims at performing content mismatch localization between cross-modal
sequential data without requiring any human annotation during the training
stage. Our model is a hierarchically structured variational autoencoder (VAE)
containing several hierarchically structured discrete latent variables. These
latent variables describe the generative process of speech from L2 speakers
and indicate the relationship between the two modalities.

One challenge for ML-VAE is that training such an architecture is very
difficult due to the discrete latent variables with complex dependencies. To
address this challenge, we propose a novel and effective training procedure
that alternates between estimating the hard assignments of the discrete latent
variables over a specifically designed finite-state automaton (FSA) and
updating the parameters of neural networks.

The main contributions of our work include:

  * •

We propose a hierarchical Bayesian deep learning model, ML-VAE, to address the
problem of content mismatch localization from cross-modal sequential data.

  * •

Our ML-VAE is the first method that bridges finite-state automata and
variational autoencoders; this is achieved via our proposed mismatch
localization finite-state acceptor (ML-FSA), which allows the ML-VAE to locate
the mismatch by searching for the best path in ML-FSA.

  * •

To address the challenge of inferring the latent discrete variables with
complex dependencies involved in ML-VAE, we propose a novel and effective
alternating inference and learning procedure.

  * •

We apply ML-VAE to the mispronunciation localization task; experiments on a
non-native English corpus to demonstrate ML-VAE’s effectiveness in terms of
unsupervisedly locating the mispronunciation segments in the speech.

##  2 Related Work

##### Cross-Modal Sequential Data Alignment

There have been several studies on aligning sequential data from different
modalities. For example, for video-action alignment, most existing work
focuses on learning spatio-temporal relations among the frames. Dwibedi et al.
(2019) proposes a self-supervised learning approach named temporal cycle
consistency learning to learn useful features for video action alignment. Liu
et al. (2021) proposes to learn a normalized human pose feature to help
perform the alignment task. Song et al. (2016) adopts an unsupervised way to
align the actions in a video with its text description. In terms of speech-
text alignment, a recent study by Kürzinger et al. (2020) proposes using the
CTC output to perform speech-text alignment. However, these aforementioned
studies assume a perfect match in the content of the cross-modal sequential
data, making it impossible to locate the content mismatch from the data.

Speech-text alignment is usually referred to as the forced alignment (FA) task
(McAuliffe et al., 2017) in the speech processing community. There is a fairly
long history of research related to FA. Traditional method is to run the
Viterbi decoding algorithm (Forney, 1973) on a Hidden-Markov-Model-based (HMM-
based) acoustic model with a given text sequence. Even though several FA
studies have considered the particular problem setting where the text is not
perfectly matched to the speech, they require training an acoustic model using
a large amount of human-annotated speech, which is laborious and costly. For
instance, Finke & Waibel (1997); Moreno et al. (1998); Moreno & Alberti
(2009); Bell & Renals (2015); Stan et al. (2012) require an acoustic model for
text-to-speech alignment with a modified lattice, and Bordel et al. (2012)
requires an acoustic model to obtain the recognized phoneme sequence.
Therefore, these methods fail to handle our problem setting, where the speech
data from L2 speakers is unlabelled.

##### Variational Autoencoders and Bayesian Deep Learning

Variational autoencoder (VAE) (Kingma & Welling, 2013) is proposed to learn
the latent representations of real-world data by introducing latent variables.
It adopts an encoder to approximate the posterior distribution of the latent
variable and and a decoder to model the data distribution. VAEs have a wide
range of applications, such as data generation (Mescheder et al., 2017),
representation learning (Oord et al., 2017), etc. As an extension to VAE to
handle sequential data (e.g., speech), the variational recurrent neural
network (VRNN) (Chung et al., 2015) introduces latent variables into the
hidden states of a recurrent neural network (RNN), allowing the complex
temporal dependency across time steps to be captured. Along a similar line of
research, Johnson et al. (2016) proposes the structured VAE, which integrates
the conditional random field-like structured probability graphical model with
VAEs to capture the latent structures of video data. Factorized hierarchical
variational autoencoder (FHVAE) (Hsu et al., 2017) improves upon VRNN and SVAE
through introducing two dependent latent variables at different time scales,
which enables the learning of disentangled and interpretable representations
from sequential data. However, the hierarchical models mentioned above are
trained by directly optimizing the evidence lower bound (ELBO), which may fail
when discrete latent variables with complex dependencies are involved. To
address this issue, we propose a novel learning procedure to optimize our ML-
VAE.

To deal with data from different modalities, Jo et al. (2019) proposes a
cross-modal VAE to capture both intra-modal and cross-modal associations from
input data. Theodoridis et al. (2020) proposes a VAE-based method to perform
cross-modal alignment of latent spaces. However, existing work on processing
cross-modal data focuses mainly on learning the relationship between
modalities while ignoring the potential mismatch between them. Therefore, we
propose the ML-VAE to address this issue. In general, our work belongs to the
category of Bayesian deep learning (BDL) (Wang & Yeung, 2016; 2020; Wang et
al., 2015; Huang et al., 2020; 2021; 2022), and is the first BDL method that
performs mismatch localization.

##  3 Problem Formulation

Figure 1: Problem formulation.

Here, we propose to generalize the traditional speech-text alignment problem
by considering the content mismatch between the input sequences (see Fig. 1).
Specifically, given two cross-modal sequences: (1) a speech feature sequence
${\mathcal{X}}=(x_{1},...,x_{T})$ ($x_{t}\in\mathbb{R}^{D_{f}}$, where $D_{f}$
is the feature dimension) as the source sequence, and (2) a phoneme sequence
${\mathcal{C}}=(c_{1},...,c_{L})$ ($c_{l}\in{\mathcal{P}}$, where
${\mathcal{P}}$ is the phoneme set) as the target sequence, our goal is to
identify the mismatched target elements of ${\mathcal{C}}$ (i.e.,
mispronounced phonemes) while locating the corresponding elements in the
source sequence (i.e. incorrect speech segments). Concretely, let
${\mathcal{C}}^{\prime}=(c^{\prime}_{1},...,c^{\prime}_{L})$ be the mismatch-
identified target sequence. We use the notation $c_{l}^{*}$ to present
mismatched content; then $c^{\prime}_{l}=c_{l}^{*}$ if $c^{\prime}_{l}$ is
identified as mismatched content (i.e., a mispronounced phoneme) and
$c^{\prime}_{l}=c_{l}$ if $c^{\prime}_{l}$ is identified as matched content
(i.e., a correctly pronounced phoneme). The final localization result
$\hat{\mathcal{C}}=(\hat{c}_{1},...,\hat{c}_{T})$ is therefore a repeated
version of ${\mathcal{C}}^{\prime}$ with each element $c^{\prime}_{l}$
repeating for $d_{l}$ times, indicating that $c^{\prime}_{l}$ lasts for
$d_{l}$ frames. The mismatch localization performance is evaluated by
averaging the intersection-over-union for each successfully detected
mismatched phoneme, with more details to be introduced in Sec. 8.1.

##  4 Model

ML-VAE is a hierarchical Bayesian deep learning model (Wang & Yeung, 2020)
based on VAE (Kingma & Welling, 2013). Our model aims at performing content
mismatch localization when aligning cross-modal sequential data. In this work,
we focus on the speech-text mismatch localization task.

This is made possible by decomposing the generative process of the speech into
hierarchically structured latent variables, indicating the relationship
between the two modalities.

ML-VAE is designed to use a hierarchical Gaussian mixture model (GMM) to model
the match/mismatch between the sequences. Each Gaussian component corresponds
to a type of mismatch, and the selection of the Gaussian component depends on
the discrete latent variables of ML-VAE.

##### Latent Variables and Generative Process

In the context of speech-text mismatch localization, since both the mismatch-
identified phoneme sequence ${\mathcal{C}}^{\prime}$ and the duration of each
phoneme are unobservable, we introduce several latent variables to achieve the
goal of mismatch localization for speech-text sequences. Instead of using a
duration variable to describe the duration of each phoneme, we use a binary
boundary variable sequence ${\mathcal{B}}=(b_{1},...,b_{T})$, where $b_{t}=1$
means the $t$-th frame marks the start of a phoneme segment, and thus
$\sum_{t=1}^{T}b_{t}=L$. Besides, a binary correctness variable sequence
$\Pi=(\pi_{1},...,\pi_{T})$ is introduced to describe the matched/mismatched
content in speech. Each entry $\pi_{t}\in\\{0,1\\}$, with $\pi_{t}=1$ if the
$t$-th speech frame contains mismatched content (i.e. a mispronounced
phoneme), and $\pi_{t}=0$ otherwise.

To model the data generative process, at each time step $t$, we introduce
three more latent variables: 1) $y_{t}$, which denotes the estimated phoneme,
2) $z_{t}$, which represents the Gaussian component indicator, and 3) $h_{t}$,
which is the speech latent variable.

Figure 2: Graphical model for ML-VAE.

As shown in Fig. 2, we draw the estimated phoneme $y_{t}$ from a categorical
distribution, and draw the correctness variable $\pi_{t}$ from a Bernoulli
distribution. Meanwhile, the boundary variable $b_{t}$ is assumed to be drawn
from a Bernoulli distribution $Bernoulli(\eta_{t})$, which is further
parameterized by a Beta distribution $Beta(\alpha,\beta)$.

In the GMM part of ML-VAE, $z_{t}$, indicating the index of the selected
Gaussian distribution, is an one-hot variable drawn from a categorical
distribution, and $h_{t}$, the latent variable of speech, is drawn from the
selected Gaussian distribution. For each phoneme type, we use one component to
model the matched content (i.e., correct pronunciation) and $N_{m}$ components
to model different mismatch types (i.e., mispronunciation variants). Such a
design is motivated by the following observations: all correct pronunciations
of the same phoneme are usually similar to each other, while its
mispronunciations often severely deviate from the correct one and have
multiple variants (e.g., the consonant ‘DH’ may be mispronounced into ‘T’,
‘D’, or ‘TH’). Therefore, for $N$ phoneme types, there are $N$ components
modeling correct pronunciations and $N*N_{m}$ components modeling
mispronunciations, making the GMM part of ML-VAE contain a total number of
$K=N+N*N_{m}$ components.

Given the design introduced above, the generative process of ML-VAE is as
follows:

  * •

For the $t$-th frame ($t=1,2,\dots,T$):

    * –

Draw the estimated phoneme $y_{t}\sim Categorical(\zeta)$.

    * –

Draw the boundary variable $b_{t}\sim Bernoulli(\gamma)$.

    * –

Draw the correctness variable $\pi_{t}\sim Bernoulli(\eta_{t})$ , where
$\eta_{t}\sim Beta(\alpha,\beta)$.

    * –

With $y_{t}$, $b_{t}$, and $\pi_{t}$, draw $z_{t}$ from
$z_{t}|y_{t},\pi_{t},b_{t}\sim Categorical(f_{z}(y_{t},b_{t},\pi_{t}))$, where
$f_{z}(\cdot)$ denotes a learnable neural network.

    * –

Given that $z_{t}[k]=1$, select the $k$-th Gaussian distribution and draw
$h_{t}|z_{t}\sim{\mathcal{N}}(\mu_{k},\sigma^{2}_{k})$.

    * –

Draw $x_{t}|h_{t}\sim{\mathcal{N}}(f_{\mu}(h_{t}),f_{\sigma}(h_{t}))$, where
$f_{\mu}(\cdot)$ and $f_{\sigma}(\cdot)$ denote learnable neural networks.

The parameters of the prior distributions of the latent variables (e.g.,
$\alpha$, $\beta$) in ML-VAE are determined by the training data.

##### An Example

We will take an audio recording of reading the word ‘NOT’ as an example to
explain the generative process. More specifically, the phoneme sequence
contains three phonemes: n, aa, and t. The estimated phoneme $y_{t}$ is
sampled to estimate the phoneme pronounced by the speaker at each time step.
The boundary variable $b_{t}$ is sampled to determine whether the speaker
starts to pronounce the next phoneme at time step $t$ (e.g., from n to aa).
The correctness variable $\pi_{t}$ is then sampled to determine whether this
phoneme (e.g., aa) is correctly pronounced (e.g., whether aa is mispronounced
as another phoneme). Based on $y_{t}$, $b_{t}$, and $\pi_{t}$, if the phoneme
(e.g., aa) is correctly pronounced, the Gaussian component indicator $z_{t}$
will select the Gaussian distribution for the correct pronunciation to
generate the speech latent variable $h_{t}$; otherwise, based on how the
phoneme is mispronounced (e.g, aa mispronounced as ae), $z_{t}$ will select
the corresponding Gaussian distribution to generate the speech latent variable
$h_{t}$.

##### Model Architecture

ML-VAE contains three components: boundary detector $\phi_{b}$, phoneme
estimator $\phi_{p}$, and speech generator $\phi_{h}$, as shown in Fig. 3. The
boundary detector outputs the approximated posterior
$q_{\phi_{b}}(b_{t}|x_{t})$, and the phoneme estimator outputs the
approximated posterior $q_{\phi_{p}}(y_{t}|x_{t})$. The speech generator aims
to reconstruct the input speech feature sequence following the generation
process discussed in Sec. 4. More details are introduced in Sec. 6.2.

Figure 3: Model architecture for ML-VAE.

##  5 Mismatch Localization Finite-State Acceptor

In this section, we describe our proposed mismatch localization finite-state
acceptor (ML-FSA) that bridges finite-state automata and variational
autoencoders, allowing our ML-VAE to locate the mispronounced segment in the
speech by efficiently searching the best path in ML-FSA. Our ML-FSA is a
special type of finite-state acceptor (FSA) that describes possible hypothesis
of mispronunciations and phoneme boundaries. We show ML-FSA for the $l$-th
phoneme, $c_{l}$, in the given phoneme sequence in Fig. 4. Such a phoneme-
level ML-FSA contains a set of states and a set of state-to-state transitions;
the initial state 0 and the accepting state 5 are represented by a bold circle
and concentric circles respectively; each transition includes a source state,
a destination state, a label and a corresponding weight. Specifically, from
the initial state (state 0), it can transit to state 1 or 3 based on
pronunciation correctness ($R$ stands for correct pronunciation and $W$ stands
for mispronunciation). This further leads to two different paths, one for
correct pronunciation (denoted by $c_{l}$) leading to state 2, and the other
one for mispronunciation (denoted by $c_{l}^{*}$) leading to state 4. In each
path, at each time step, since each phoneme may last for several frames, it
either still holds at the current state (denoted by $H$), or moves forward to
the final state 5 (denoted by $S$).

With the help of the phoneme sequence ${\mathcal{C}}$, we can naturally build
a sentence-level ML-FSA by combining the corresponding ML-FSA for each
phoneme. With the sentence-level ML-FSA, a dynamic programming (DP) algorithm
can be applied to search for the optimal path based on the weights on the
transitions. More specifically, the weight of each transition in ML-FSA can be
estimated by the modules in ML-VAE, with more details to be introduced in Sec.
6.1.

Figure 4: Proposed ML-FSA for the $l$-th phoneme, $c_{l}$.

In contrast to weighted finite state transducers (WFSTs) Mohri et al. (2002),
which only model the segmentation of audio into acoustic units (e.g.,
phonemes), our ML-FSA models the correctness of pronunciation ($R$ and $W$),
the phoneme type ($c_{l}$, $c_{l}^{*}$), and the phoneme boundary ($H$ and
$S$), which can jointly detect and segment the mispronunciation in the speech
and thus perform _unsupervised content mismatch localization_. Furthermore,
WFSTs require training a supervised acoustic model for weighting transitions
between states, while our ML-FSA adopts the unsupervised ML-VAE to calculate
all the weights of transitions. The details on locating the mispronunciation
using ML-FSA are described in the next section. Note that ML-FSA takes all
possible mispronunciation variants as a single symbol ($c_{l}^{*}$), which
trades fidelity for efficiency. This is advantageous for addressing our
problem settings: (1) our task focuses on distinguishing between the correct
pronunciation and mispronunciation, rather than identifying the
mispronunciation variants; (2) combining mispronunciation variants for each
phoneme can dramatically reduce the computational cost of the dynamic
programming (DP) algorithm searching for the optimal path.

##  6 Learning

The learning of ML-VAE consists of two main components: latent variables
$\Psi=\\{{\mathcal{Y}},{\mathcal{B}},\Pi\\}$ and neural network parameters
$\Phi=\\{\phi_{p},\phi_{b},\phi_{h}\\}$, where
${\mathcal{Y}}=(y_{1},...,y_{t})$, and $\phi_{p}$, $\phi_{b}$, and $\phi_{h}$
denote parameters of the three modules of ML-VAE: phoneme estimator, boundary
detector, and speech generator, respectively.

Following the traditional variational inference and the training objective for
FHVAE (Hsu et al., 2017), the ELBO for the joint training objective of ML-VAE
can be written as:

$\text{ELBO}=\sum_{t=1}^{T}\Big{(}\mathbb{E}_{q(y_{t},b_{t},\pi_{t}|x_{t})}\big{[}\log
p(x_{t}|y_{t},b_{t},\pi_{t})\big{]}-D_{\text{KL}}(q(y_{t},b_{t},\pi_{t}|x_{t})||p(y_{t},b_{t},\pi_{t})\Big{)},$
(1)

where $D_{\text{KL}}$ is the function to calculate the Kullback–Leibler (KL)
divergence, $q(y_{t},b_{t},\pi_{t}|x_{t})$ is the joint approximate posterior
distribution of the latent variables, $p(y_{t},b_{t},\pi_{t})$ is the joint
prior distribution.

Figure 5: The training framework of ML-VAE.

However, unlike FHVAE, learning ML-VAE with such an objective function is very
difficult due to ML-VAE’s discrete latent variables with complex dependencies,
as pointed out by Rolfe (2016). Furthermore, we empirically tested an
intuitive approach – using a hard expectation-maximization (EM) algorithm
(Moon, 1996) – and found that the system would not reliably converge. As
pointed out by Locatello et al. (2019), lacking inductive bias for
unsupervised training could render the model unidentifiable. Therefore, we
improve this approach by providing pseudo-labels to some latent variables, as
introduced by Khemakhem et al. (2020). Note that our learning algorithm
remains unsupervised since it does not require any external data other than
${\mathcal{X}}$ and ${\mathcal{C}}$. Our new training procedure (see Fig. 5)
can be described as:

  * •

Step 1 (E step). Given the model parameters $\Phi$, estimate the hard
assignments of the latent variables $\hat{\Psi}=\\{\hat{\mathcal{Y}}$,
$\hat{\mathcal{B}}$, $\hat{\Pi}\\}$ by:

    * –

Step 1.1. Given $\phi_{p}$, estimate $\hat{\mathcal{Y}}$.

    * –

Step 1.2. Given $\Phi$, estimate $\hat{\mathcal{B}}$ and $\hat{\Pi}$.

  * •

Step 2 (M step). Given the hard assignments of the latent variables
$\hat{\Psi}$, optimize the model parameters by:

    * –

Step 2.1. Given the phoneme sequence ${\mathcal{C}}$ as the training target,
optimize $\phi_{p}$ .

    * –

Step 2.2. Given a forced alignment result $\bar{{\mathcal{B}}}$, optimize
$\phi_{b}$.

    * –

Step 2.3. Given the estimated hard assignments $\hat{\Psi}$ of latent
variables, optimize $\phi_{h}$.

Our approach differs from the standard hard EM algorithm in several aspects.
First, since the phoneme sequence ${\mathcal{C}}$ is given under our problem
setting, the phoneme estimator $\phi_{p}$ can be trained by directly
optimizing the cross-entropy loss towards ${\mathcal{C}}$ (Step 2.1). Second,
we can directly adopt the predictions of $\phi_{p}$ to obtain
$\hat{\mathcal{Y}}$ (Step 1.1). Third, we design an FSA to assist
approximating a maximum a posteriori (MAP) estimate of $\hat{\mathcal{B}}$ and
$\hat{\Pi}$ (Step 1.2). Fourth, we find that directly using
$\hat{{\mathcal{B}}}$ from Step 1.2 deteriorates the model performance;
therefore in Step 2.2, we adopt a forced alignment result
$\bar{{\mathcal{B}}}$ (Tebelskis, 1995; McAuliffe et al., 2017) to train the
boundary detector $\phi_{b}$. It is worth noting that obtaining such an
alignment in Step 2.2 does not require any external data other than
${\mathcal{X}}$ and ${\mathcal{C}}$.

###  6.1 Step 1: Estimation of the Hard Assignments $\hat{\Psi}$

Next, we discuss the details of estimating the hard assignments.

##### Estimation of $\hat{\mathcal{Y}}$

We obtain the estimated phoneme sequence $\hat{\mathcal{Y}}$ with:

$\hat{{\mathcal{Y}}}=\mathop{\rm
argmax}_{{\mathcal{Y}}}{\mathcal{J}}_{\mathcal{Y}}({\mathcal{Y}};\phi_{p})=\mathop{\rm
argmax}_{{\mathcal{Y}}}q_{\phi_{p}}({\mathcal{Y}}|{\mathcal{X}}),$ (2)

where $q_{\phi_{p}}({\mathcal{Y}}|{\mathcal{X}})$ is the variational
distribution calculated by the phoneme estimator $\phi_{p}$ to approximate the
intractable true posterior $p({\mathcal{Y}}|{\mathcal{X}})$.

##### Estimation of $\hat{\mathcal{B}}$ and $\hat{\Pi}$ via ML-FSA

The estimation of $\hat{\mathcal{B}}$ and $\hat{\Pi}$ is done by finding the
optimal path in the sentence-level ML-FSA. As introduced in Sec. 5, the
weights of transitions of ML-FSA are all estimated by the model ML-VAE. More
specifically, from state 0, the weights of the two transitions, labeled with
$R$ and $W$, are estimated by the approximate posterior
$q_{\phi_{h}}(\pi_{t}=0|x_{t},y_{t},b_{t})$ and
$q_{\phi_{h}}(\pi_{t}=1|x_{t},y_{t},b_{t})$ respectively. Afterwards, the
weights of the next two transitions, labeled with $c_{l}$ and $c_{l}^{*}$, are
estimated by the posterior $q_{\phi_{p}}(y_{t}=c_{l}|x_{t})$ and
$q_{\phi_{p}}(y_{t}\neq c_{l}|x_{t})$ respectively. Finally, the weights of
the last two transitions, marked with $H$ and $S$, are estimated by the
posterior $q_{\phi_{b}}(b_{t}=0|x_{t})$ and $q_{\phi_{b}}(b_{t}=1|x_{t})$
respectively. The details of how these posteriors are estimated will be
introduced in Sec. 6.2.

Given all the weights in the sentence-level ML-FSA available, a typical
dynamic programming (DP) algorithm can be used to find the optimal path which
yields the maximum probability, whereby $\hat{\mathcal{B}}$ and $\hat{\Pi}$
are backtracked along the optimal DP path. Meanwhile, the optimal DP path also
yields the mismatch localization result $\hat{\mathcal{C}}$. More details are
given in Appendix A.

###  6.2 Step 2: Learning Model Parameters $\Phi$

In this section, we present how ML-VAE’s parameters
$\Phi=\\{\phi_{p},\phi_{b},\phi_{h}\\}$ are learned given the estimated
$\hat{\Psi}$ obtained from the previous stage. Implementation and
parameterization details can be found in Appendix E. Different objectives are
proposed for $\phi_{p}$, $\phi_{b}$, and $\phi_{h}$.

####  6.2.1 Boundary Detector $\phi_{b}$

The boundary detector takes the speech feature sequence ${\mathcal{X}}$ as
input and outputs the probability $q_{\phi_{b}}(b_{t}|x_{t})$. The boundary
variable $b_{t}$ is drawn from a Bernoulli distribution parameterized by an
auxiliary latent variable $\eta_{t}$, that is,
$b_{t}\sim\text{Bernoulli}(\eta_{t})$, where we assume $\eta_{t}$ follows a
Beta distribution: $\eta_{t}\sim Beta(\alpha,\beta)$. Therefore here we models
an auxiliary distribution $q_{\phi_{b}}(\eta_{t}|x_{t})$. As introduced, we
use the forced alignment result sequence
$\bar{\mathcal{B}}=(\bar{b}_{1},...,\bar{b}_{T})$ as the pseudo-label to aid
the training process. The training objective is to minimize the loss
${\mathcal{L}}_{b}$:

${\mathcal{L}}_{b}(\phi_{b};\bar{\mathcal{B}})=-\sum_{t=1}^{T}\Big{(}\mathbb{E}_{q_{\phi_{b}}(\eta_{t}|x_{t})}\big{[}\log
p(b_{t}=\bar{b}_{t}|\eta_{t})\big{]}-\lambda_{b}D_{\text{KL}}(q_{\phi_{b}}(\eta_{t}|x_{t})||p(\eta_{t}))\Big{)},$
(3)

where $\lambda_{b}$ is a hyperparameter controlling the weight of the KL term,
$q_{\phi_{b}}(\eta_{t}|x_{t})$ is the approximate posterior distribution, and
$p(\eta_{t})$ is the prior distribution of $\eta_{t}$.

Note that no human annotation is required when obtaining the forced alignment
result sequence $\bar{\mathcal{B}}$, since $\bar{\mathcal{B}}$ is obtained
with ${\mathcal{X}}$ and ${\mathcal{C}}$ as inputs without using any human
annotation of mispronounced phonemes in ${\mathcal{C}}$.

####  6.2.2 Phoneme Estimator $\phi_{p}$

Similar to the boundary detector, the phoneme estimator takes the speech
feature sequence ${\mathcal{X}}$ as input and outputs the probability
$q_{\phi_{p}}(y_{t}|x_{t})$. We use the pseudo label
$\tilde{\mathcal{C}}=(\tilde{c}_{1},...,\tilde{c}_{T})$ as our training
target, which can be obtained by extending the phoneme sequence
${\mathcal{C}}$ according to the estimated duration of each phoneme in
$\bar{\mathcal{B}}$. Therefore, the model is optimized by minimizing the
negative log-likelihood:

${\mathcal{L}}_{p}(\phi_{p};{\mathcal{C}})=-\sum_{t=1}^{T}\log(q_{\phi_{p}}(y_{t}=\tilde{c}_{t}|x_{t})).$
(4)

####  6.2.3 Speech Generator $\phi_{h}$

The speech generator $\phi_{h}$ aims to reconstruct the input speech feature
sequence. It takes the speech feature sequence ${\mathcal{X}}$, along with the
estimated values of the latent variables $\hat{\mathcal{Y}}$ and
$\hat{\mathcal{B}}$ as input and reconstructs the speech feature sequence
following the generation process discussed in Section 4.

We use the variational inference to learn the speech generator $\phi_{h}$ and
thus the ELBO loss is calculated by:

${\mathcal{L}}_{r}(\phi_{h})=-\sum_{t=1}^{T}\Big{(}\mathbb{E}_{q_{\phi_{h}}(h_{t}|x_{t})}\big{[}\log
p_{\phi_{h}}(x_{t}|h_{t})\big{]}-\lambda_{r}D_{\text{KL}}(q_{\phi_{h}}(h_{t}|x_{t})||p(h_{t}))\Big{)}$
(5)

where $\lambda_{r}$ controls the weight of the KL term,
$q_{\phi_{h}}(h_{t}|x_{t})$ is the approximate posterior distribution, and
$p(h_{t})$ is the prior distribution of $h_{t}$.

Besides, we also augment the ELBO using the estimated correctness variable
sequence $\hat{\Pi}$ obtained from Step 1 (E Step) as a supervision signal and
the new loss is written as:

${\mathcal{L}}_{h}(\phi_{h};\hat{\Psi})={\mathcal{L}}_{r}(\phi_{h})+\lambda_{l}{\mathcal{L}}_{l}(\phi_{h};\hat{\Psi}),$
(6)

where $\lambda_{l}$ is an importance weight;
${\mathcal{L}}_{l}(\phi_{h};\hat{\Psi})$ is the negative log-likelihood loss,
which is computed by:

${\mathcal{L}}_{l}(\phi_{h};\hat{\Psi})=-\sum_{t=1}^{T}\log
q_{\phi_{h}}(\pi_{t}=\hat{\pi}_{t}|x_{t},y_{t},b_{t}),$ (7)

where $q_{\phi_{h}}(\pi_{t}|x_{t},y_{t},b_{t})$ is the approximate posterior
distribution.

###  6.3 Overall Learning Algorithm

The overall algorithm to learn ML-VAE is shown in Algorithm 1.

Algorithm 1 Learning ML-VAE

0: Speech feature sequence ${\mathcal{X}}$, phoneme sequence ${\mathcal{C}}$

0: Mismatch localization result $\hat{\mathcal{C}}$

1: Initialize the model parameters $\phi_{p}$, $\phi_{b}$, and $\phi_{h}$.

2: Obtain the forced alignment result $\bar{{\mathcal{B}}}$.

3: while not converged do

4: Estimate $\hat{\mathcal{B}}$ and $\hat{\Pi}$ with ML-FSA.

5: Using Eq. 4, optimize $\phi_{p}$ with the phoneme sequence ${\mathcal{C}}$.

6: Using Eq. 3, with the help of $\bar{{\mathcal{B}}}$, optimize $\phi_{b}$.

7: Given $\hat{\Pi}$, optimize $\phi_{h}$ using Eq. 6.

8: end while

9: Obtain the mismatch localization result $\hat{\mathcal{C}}$ with ML-FSA.

10: return $\hat{\mathcal{C}}$

##  7 ML-VAE with REINFORCE Algorithm

In practice, we found that the training procedure is sometimes not stable due
to large gradient variance. We suspect this is due to the sampling process of
the discrete latent variables (such as $\pi_{t}$). To overcome such an issue
and better reason about the distribution of the correctness variable during
training, we further propose a variant of ML-VAE that uses the REINFORCE
algorithm (Williams, 1992), which we call ML-VAE-RL.

Following the work by Xu et al. (2015), the reward term is defined as
${\mathcal{R}}(\Pi)={\mathcal{L}}_{h}(\phi_{h};\hat{\Psi})$, and to further
reduce the gradient variance, a baseline term $b({\mathcal{X}})$ is introduced
to calculate the calibrated reward term
$\hat{\mathcal{R}}(\Pi)={\mathcal{R}}(\Pi)-b({\mathcal{X}})$.
$b({\mathcal{X}})$ is estimated by a fully-connected neural network and
trained by minimizing the mean squared error (MSE) between $b({\mathcal{X}})$
and the uncalibrated reward ${\mathcal{R}}(\Pi)$:
$\text{MSE}(b({\mathcal{X}}),{\mathcal{R}}(\Pi))=\frac{1}{T}\sum^{T}_{t=1}[b(x_{t})-{\mathcal{R}}(\pi_{t})]^{2}$.

Then we define the new optimization objective
${\mathcal{L}}_{rl}(\phi_{h};\hat{\Psi})$ whose gradient with respect to the
model parameters $\phi_{h}$ can be calculated with the Monte Carlo method:

$\nabla_{\phi_{h}}{\mathcal{L}}_{rl}(\phi_{h};\hat{\Psi})=\sum_{i=1}^{N_{mc}}\bigg{(}\nabla_{\phi_{h}}{\mathcal{L}}_{h}(\phi_{h};\hat{\Psi})+\hat{\mathcal{R}}(\Pi^{(i)})\nabla_{\phi_{h}}\big{[}-\log
q_{\phi_{h}}(\Pi^{(i)}=\hat{\Pi}|{\mathcal{X}},{\mathcal{Y}},{\mathcal{B}})\big{]}\bigg{)}-\nabla_{\phi_{h}}H[\Pi],$
(8)

where $H[\pi]$ is the entropy of the distribution of the correctness variable
whose gradient can be calculated explicitly. $N_{mc}$ is the number of Monte
Carlo samples, and $\Pi^{(i)}$ is the $i$-th sample drawn from the posterior
distribution $\Pi|{\mathcal{X}},{\mathcal{Y}},{\mathcal{B}}$.

With the help of the REINFORCE algorithm, Step 7 in Algorithm 1 is replaced by
optimizing with Eq. 8 so that $\phi_{h}$ is reinforced to sample the
correctness variable sequence $\Pi$ that produces lower
${\mathcal{L}}_{h}(\phi_{h};\hat{\Psi})$.

##  8 Experiments

We evaluate our proposed ML-VAE and ML-VAE-RL on the mispronunciation
localization task to test their mismatch localization ability. We first
conduct experiments on a synthetic dataset (Mismatch-AudioMNIST) and then
further apply our proposed models to a real-world speech-text dataset
(L2-ARCTIC).

###  8.1 Evaluation Metrics

The F1 score is a commonly used metric to evaluate binary classification
tasks, e.g., mispronunciation detection (Leung et al., 2019). However, we find
that F1 score is not suitable to evaluate the mispronunciation localization
task, as it is computed without evaluating if the model successfully locates
the detected mispronunciations in speech.

As such, we improve upon the traditional F1 score by proposing a set of new
evaluation metrics. For each true positive (TP) case, we calculate the
intersection over union (IoU) metric of its corresponding phoneme segment,
which demonstrates the performance of localization. IoU is computed by
$IoU=\frac{I}{U}$, where $I$ and $U$ are the intersection length and the union
length obtained by comparing the detected phoneme segment with the ground
truth phoneme segment. Then such IoU metrics of all TP cases are summed and
denoted as $\text{TP}_{\text{ML}}$. After calculating $\text{TP}_{\text{ML}}$,
the TP in equations to calculate the F1 score is replaced with
$\text{TP}_{\text{ML}}$ to calculate our proposed metrics. For example, if
there are two TP cases detected by the model, then instead of calculating F1
score with $\text{TP}=2$, we calculate the IoU of these two cases’
corresponding phoneme segment, e.g. $0.3$ and $0.6$ respectively. Then we
calculate our proposed metrics with $\text{TP}_{\text{ML}}=0.3+0.6$. Our
proposed metrics are denoted as mismatch localization precision
($\text{PR}_{\textbf{ML}}$), recall ($\text{RE}_{\textbf{ML}}$), and F1 score
($\text{F1}_{\textbf{ML}}$).

To be more specific, we first calculate the the intermediate metrics: true
positive (TP), true negative (TN), false positive (FP), and false negative
(FN). Then for all the TP cases detected by the model, we take the sum of
their corresponding segments’ intersection over union (IoU), which is denoted
as $\text{TP}_{\text{ML}}$. Then the final metrics are computed as:
$\text{PR}_{\textbf{ML}}=\frac{\text{TP}_{\text{ML}}}{\text{TP}+\text{FP}},\;\text{RE}_{\textbf{ML}}=\frac{\text{TP}_{\text{ML}}}{\text{TP}+\text{FN}},\;\text{F1}_{\textbf{ML}}=\frac{2\times\text{PR}_{\textbf{ML}}\times\text{RE}_{\textbf{ML}}}{\text{PR}_{\textbf{ML}}+\text{RE}_{\textbf{ML}}}$

###  8.2 Baselines

To our knowledge, there is no existing work specifically designed for
mispronunciation localization. Therefore, we adapt some existing methods on
related tasks (e.g. ASR) to perform mispronunciation localization, and compare
our proposed two ML-VAE models with them:

  * •

FA (McAuliffe et al., 2017), which performs forced alignment using a deep-
neural-network-HMM-based (DNN-HMM-based) acoustic model.

  * •

Two-Pass-FA (Tebelskis, 1995), which first performs phoneme recognition based
on a DNN-HMM-based acoustic model and then performs forced alignment on the
recognized phoneme sequence and input speech.

  * •

w2v-CTC (Baevski et al., 2020), which is built with wav2vec 2.0 and trained
with CTC loss; CTC-segmentation (Kürzinger et al., 2020) is used to align the
recognized phonemes with speech.

All models above are trained under the same problem setting where no human
annotated data is used, i.e. only the speech feature sequence ${\mathcal{X}}$
and phoneme sequence ${\mathcal{C}}$ which contains mismatch are used during
the training process.

###  8.3 Synthetic Dataset: Mismatch-AudioMNIST

We first test our proposed models on a synthetic dataset, named as Mismatch-
AudioMNIST, which is built based on the AudioMNIST (Becker et al., 2018)
dataset. Our synthetic dataset contains 3000 audio samples, each produced by
concatenating three to seven spoken digits randomly selected from the original
AudioMNIST. To simulate the content mismatch between audio and the
corresponding text annotations, we randomly select 20.1% of the spoken digits
as mismatched content, thereby labeling them as random digits. The total
duration of Mismatch-AudioMNIST is 2.67 hours. The dataset is split into
training, validation, and test sets by a 60:20:20 ratio, and the durations of
the three sets are 96.3, 33.2, and 30.6 minutes respectively.

Table 1: Mispronunciation localization results on Mismatch-AudioMNIST. Model | # Params | $\text{PR}_{\textbf{ML}}\%$ | $\text{RE}_{\textbf{ML}}\%$ | $\text{F1}_{\textbf{ML}}\%$
---|---|---|---|---
FA | 3.3M | 0.00 | 0.00 | 0.00
Two-Pass-FA | 3.3M | 6.22 | 2.43 | 2.28
w2v-CTC | 352.5M | 1.72 | 3.84 | 2.30
ML-VAE (ours) | 24M | 28.42 | 27.60 | 27.67
ML-VAE-RL (ours) | 25M | 30.67 | 30.32 | 30.28

##### Mispronunciation Localization Results

Table 1 shows the mispronunciation localization results of ML-VAE and ML-VAE-
RL on the synthetic dataset Mismatch-AudioMNIST. We can see that all three
baselines perform poorly for our mismatch localization task. More
specifically, the vanilla forced alignment model (FA) fails to locate any
mispronunciations due to its assumption that all the digits are correctly
pronounced. Both of our proposed models (ML-VAE and ML-VAE-RL) are much
superior to the baseline models, Two-Pass-FA and w2v-CTC, demonstrating the
effectiveness of the proposed method in handling the unsupervised mismatch
localization task. Note that in our problem settings, our training data
contains mislabeled spoken digits and such mislabelling is unknown. This poses
great challenges for acoustic model training. By further looking into the
recognition results of Two-Pass-FA and w2v-CTC, we find that their recognition
results from the acoustic model are very poor in terms of the mislabeled
spoken digits, consequently limiting the baseline models’ ability to detect
such mislabels in the second stage processing.

###  8.4 Real-World Dataset: L2-ARCTIC

We further apply ML-VAE and ML-VAE-RL to a real-world dataset: the L2-ARCTIC
dataset (Zhao et al., 2018), which is a non-native English corpus containing
11026 utterances from 24 non-native speakers. Note that to evaluate mismatch
localization, each annotated phoneme in the dataset needs to come with an
onset and offset time label; to the best of our knowledge, the L2-ARCTIC
dataset is currently _the only real-world dataset suitable for this problem
setting_.

##### Mispronunciation Localization Results

Table 2: Mispronunciation localization results on L2-ARCTIC. Model | # Params | $\text{PR}_{\textbf{ML}}\%$ | $\text{RE}_{\textbf{ML}}\%$ | $\text{F1}_{\textbf{ML}}\%$
---|---|---|---|---
FA | 3.3M | 0.00 | 0.00 | 0.00
Two-Pass-FA | 3.3M | 0.64 | 0.96 | 0.77
w2v-CTC | 352.5M | 1.29 | 2.26 | 1.64
ML-VAE (ours) | 24M | 6.46 | 11.97 | 8.39
ML-VAE-RL (ours) | 25M | 8.20 | 13.57 | 10.22

As shown in Table 2, similar to the results on Mismatch-AudioMNIST, Our ML-VAE
models dramatically outperform Two-Pass-FA and w2v-CTC in all three metrics.
Compared with original ML-VAE, the variant with REINFORCE algorithm yields
better localization performance of mispronunciation.

Interestingly, though w2v-CTC is based on a powerful wav2vec 2.0 acoustic
model, which shows superior performance for ASR, it does not work well on the
mispronunciation localization task; even with the largest number of
parameters, it only slightly outperforms Two-Pass-FA, and underperforms our
ML-VAE by a large margin. Our further analysis on w2v-CTC’s output shows that
such poor performance is mainly caused by the dataset which contains unknown
content mismatches between the speech data and the corresponding text; w2v-CTC
is fine-tuned on the speech data with mispronunciations, and therefore its
recognition results tend to contain mispronunciations as well. For example, if
a speaker always mispronounces the phoneme ‘b’ as ‘d’, a w2v-CTC model trained
with such data tends to predict ‘d’, instead of ‘b’ during inference. Another
reason for w2v-CTC’s poor performance is that the CTC-segmentation algorithm
assumes correct pronunciations and therefore does not work well on speech-text
sequences containing mispronunciations.

Some more experiments are performed to demonstrate how traditional alignment
methods fail on speech and text inputs with mispronunciations and such
experimental results can be found in Appendix F.

##### Ablation Study

We present ablation study results to demonstrate the effectiveness of our
proposed learning algorithm. Table 3 shows the results of ML-VAE-RL optimized
with $\hat{\mathcal{B}}$ instead of $\bar{\mathcal{B}}$. The first row shows
that directly using $\hat{\mathcal{B}}$ leads to very poor mispronunciation
localization performance. The second row shows that our proposed optimization
method effectively improves upon the traditional joint optimization method.

Table 3: Ablation study results. Model | $\text{PR}_{\textbf{ML}}\%$ | $\text{RE}_{\textbf{ML}}\%$ | $\text{F1}_{\textbf{ML}}\%$
---|---|---|---
Ablation 1: ML-VAE-RL using $\hat{\mathcal{B}}$ instead of $\bar{\mathcal{B}}$ for optimization | 2.31 | 1.53 | 1.84
Ablation 2: ML-VAE-RL w/ joint optimization | 4.90 | 2.89 | 3.64
Ablation 3: ML-VAE-RL w/ $\hat{\mathcal{B}}$ & $\hat{\mathcal{Y}}$ estimated separately | 7.31 | 10.85 | 8.73
ML-VAE (ours) | 6.46 | 11.97 | 8.39
ML-VAE-RL (ours) | 8.20 | 13.57 | 10.22

The third ablation study compares our estimation method with a traditional
one, which estimates $\hat{\mathcal{B}}$ and $\hat{\Pi}$ separately based on a
single-path FSA. Details on this traditional algorithm can be found in
Appendix B. Compared with this ablation model, our proposed ML-VAE-RL yields
better results. This proves the effectiveness of our proposed estimation
algorithm, which is specifically designed for the mismatch localization task.

##  9 Conclusions and Future Work

In this work, we present a hierarchical Bayesian deep learning model to
address the mismatch localization problem in cross-modal sequential data. More
specifically, two variants are proposed, namely ML-VAE and ML-VAE-RL. We also
propose a learning algorithm to optimize our proposed ML-VAE. We focus on
applying ML-VAE to the speech-text mismatch localization problem and propose a
set of new metrics to evaluate the model’s mispronunciation localization
performance. Our experimental results show that it can achieve superior
performance than existing methods, including the powerful wav2vec 2.0 acoustic
model. We also perform ablation studies to verify the effectiveness of our
training algorithm. One of the future research directions of this project is
how to extend ML-FSA to address the case where multiple correct pronunciations
exist.

##  10 Acknowledgement

We would like to thank our action editor Brian Kingsbury and the reviewers for
their detailed and constructive comments and feedback. HW is partially
supported by NSF Grant IIS-2127918 and an Amazon Faculty Research Award. This
project is funded in part by a research grant A-0008153-00-00 from the
Ministry of Education in Singapore.

## References

  * Baevski et al. (2020) Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _arXiv preprint arXiv:2006.11477_ , 2020.
  * Becker et al. (2018) Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek.  Interpreting and Explaining Deep Neural Networks for Classification of Audio Signals.  7 2018.  URL http://arxiv.org/abs/1807.03418.
  * Bell & Renals (2015) Peter Bell and Steve Renals.  A system for automatic alignment of broadcast media captions using weighted finite-state transducers.  In _2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)_ , pp. 675–680. IEEE, 2015.
  * Bordel et al. (2012) Germán Bordel, Mikel Penagarikano, Luis Javier Rodriguez-Fuentes, and Amparo Varona.  A simple and efficient method to align very long speech signals to acoustically imperfect transcriptions.  In _Thirteenth Annual Conference of the International Speech Communication Association_ , 2012.
  * Braunschweiler et al. (2010) Norbert Braunschweiler, Mark J F Gales, and Sabine Buchholz.  Lightly supervised recognition for automatic alignment of large coherent speech recordings.  In _Eleventh Annual Conference of the International Speech Communication Association_ , 2010.
  * Chung et al. (2015) Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio.  A recurrent latent variable model for sequential data.  In _Advances in neural information processing systems_ , pp. 2980–2988, 2015.
  * Chung et al. (2018) Yu-An Chung, Wei-Hung Weng, Schrasing Tong, and James Glass.  Unsupervised cross-modal alignment of speech and text embedding spaces.  _arXiv preprint arXiv:1805.07467_ , 2018.
  * Dwibedi et al. (2019) Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman.  Temporal cycle-consistency learning.  In _CVPR_ , 2019.
  * Favory et al. (2021) Xavier Favory, Konstantinos Drossos, Tuomas Virtanen, and Xavier Serra.  Learning contextual tag embeddings for cross-modal alignment of audio and tags.  In _ICASSP_ , 2021.
  * Finke & Waibel (1997) Michael Finke and Alex Waibel.  Flexible transcription alignment.  In _1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings_ , pp. 34–40. IEEE, 1997.
  * Forney (1973) G David Forney.  The viterbi algorithm.  _Proceedings of the IEEE_ , 61(3):268–278, 1973\.
  * Hazen (2006) Timothy J Hazen.  Automatic alignment and error correction of human generated transcripts for long speech recordings.  In _Ninth International Conference on Spoken Language Processing_ , 2006.
  * Hsu et al. (2017) Wei-Ning Hsu, Yu Zhang, and James Glass.  Unsupervised learning of disentangled and interpretable representations from sequential data.  In _Advances in neural information processing systems_ , pp. 1878–1889, 2017.
  * Huang et al. (2020) Hengguan Huang, Fuzhao Xue, Hao Wang, and Ye Wang.  Deep graph random process for relational-thinking-based speech recognition.  In _ICML_ , 2020.
  * Huang et al. (2021) Hengguan Huang, Hongfu Liu, Hao Wang, Chang Xiao, and Ye Wang.  Strode: Stochastic boundary ordinary differential equation.  In _ICML_ , 2021.
  * Huang et al. (2022) Hengguan Huang, Xiangming Gu, Hao Wang, Chang Xiao, Hongfu Liu, and Ye Wang.  Extrapolative continuous-time bayesian neural network for fast training-free test-time adaptation.  In _NeurIPS_ , 2022.
  * Jang et al. (2016) Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_ , 2016.
  * Jo et al. (2019) Dae Ung Jo, ByeongJu Lee, Jongwon Choi, Haanju Yoo, and Jin Young Choi.  Cross-modal variational auto-encoder with distributed latent spaces and associators.  _arXiv preprint arXiv:1905.12867_ , 2019.
  * Johnson et al. (2016) Matthew J Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta.  Composing graphical models with neural networks for structured representations and fast inference.  _Advances in neural information processing systems_ , 29:2946–2954, 2016.
  * Khemakhem et al. (2020) Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen.  Variational autoencoders and nonlinear ica: A unifying framework.  In _AISTATS_ , 2020.
  * Kingma & Welling (2013) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kundu et al. (2020) Jogendra Nath Kundu, Ambareesh Revanur, Govind Vitthal Waghmare, Rahul Mysore Venkatesh, and R Venkatesh Babu.  Unsupervised Cross-Modal Alignment for Multi-Person 3D Pose Estimation.  In _ECCV_ , 2020.
  * Kürzinger et al. (2020) Ludwig Kürzinger, Dominik Winkelbauer, Lujun Li, Tobias Watzel, and Gerhard Rigoll.  Ctc-segmentation of large corpora for german end-to-end speech recognition.  In _SPECOM_ , 2020.
  * Lei et al. (2018) Tao Lei, Yu Zhang, and Yoav Artzi.  Training rnns as fast as cnns.  2018\.
  * Leung et al. (2019) Wai-Kim Leung, Xunying Liu, and Helen Meng.  CNN-RNN-CTC based end-to-end mispronunciation detection and diagnosis.  In _ICASSP_ , 2019.
  * Liu et al. (2021) Jingyuan Liu, Mingyi Shi, Qifeng Chen, Hongbo Fu, and Chiew-Lan Tai.  Normalized Human Pose Features for Human Action Video Alignment.  In _ICCV_ , 2021.
  * Locatello et al. (2019) Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem.  Challenging common assumptions in the unsupervised learning of disentangled representations.  In _ICML_ , 2019.
  * McAuliffe et al. (2017) Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger.  Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi.  In _INTERSPEECH_ , 2017.
  * Mescheder et al. (2017) Lars Mescheder, Sebastian Nowozin, and Andreas Geiger.  Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks.  In _ICML_ , 2017.
  * Mohri et al. (2002) Mehryar Mohri, Fernando Pereira, and Michael Riley.  Weighted finite-state transducers in speech recognition.  _Computer Speech \ & Language_, 16(1):69–88, 2002.
  * Moon (1996) Todd K Moon.  The expectation-maximization algorithm.  _IEEE Signal processing magazine_ , 13(6):47–60, 1996.
  * Moreno & Alberti (2009) Pedro J Moreno and Christopher Alberti.  A factor automaton approach for the forced alignment of long speech recordings.  In _2009 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pp. 4869–4872. IEEE, 2009.
  * Moreno et al. (1998) Pedro J Moreno, Christopher F Joerg, Jean-Manuel Van Thong, and Oren Glickman.  A recursive algorithm for the forced alignment of very long audio segments.  In _ICSLP_ , volume 98, pp. 2711–2714, 1998.
  * Oord et al. (2017) Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning.  _arXiv preprint arXiv:1711.00937_ , 2017.
  * Rolfe (2016) Jason Tyler Rolfe.  Discrete variational autoencoders.  _arXiv preprint arXiv:1609.02200_ , 2016.
  * Song et al. (2016) Young Chol Song, Iftekhar Naim, Abdullah Al Mamun, Kaustubh Kulkarni, Parag Singla, Jiebo Luo, Daniel Gildea, and Henry A Kautz.  Unsupervised Alignment of Actions in Video with Text Descriptions.  In _IJCAI_ , pp. 2025–2031, 2016.
  * Stan et al. (2012) Adriana Stan, Peter Bell, and Simon King.  A grapheme-based method for automatic alignment of speech and text data.  In _2012 IEEE Spoken Language Technology Workshop (SLT)_ , pp. 286–290. IEEE, 2012.
  * Tebelskis (1995) Joseph Michael Tebelskis.  _Speech recognition using neural networks_.  Carnegie Mellon University, 1995.
  * Theodoridis et al. (2020) Thomas Theodoridis, Theocharis Chatzis, Vassilios Solachidis, Kosmas Dimitropoulos, and Petros Daras.  Cross-modal variational alignment of latent spaces.  In _CVPR_ , 2020.
  * Wang & Yeung (2016) Hao Wang and Dit-Yan Yeung.  Towards bayesian deep learning: A framework and some existing methods.  _TDKE_ , 28(12):3395–3408, 2016.
  * Wang & Yeung (2020) Hao Wang and Dit-Yan Yeung.  A survey on bayesian deep learning.  _ACM Computing Surveys (CSUR)_ , 53(5):1–37, 2020\.
  * Wang et al. (2015) Hao Wang, Naiyan Wang, and Dit-Yan Yeung.  Collaborative deep learning for recommender systems.  In _KDD_ , pp. 1235–1244, 2015.
  * Williams (1992) Ronald J Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine learning_ , 8(3):229–256, 1992.
  * Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In _International conference on machine learning_ , pp. 2048–2057. PMLR, 2015.
  * Young et al. (2002) Steve Young, Gunnar Evermann, Mark Gales, Thomas Hain, Dan Kershaw, Xunying Liu, Gareth Moore, Julian Odell, Dave Ollason, Dan Povey, and others.  The HTK book.  _Cambridge university engineering department_ , 3(175):12, 2002.
  * Zhao et al. (2018) Guanlong Zhao, Sinem Sonsaat, Alif O Silpachai, Ivana Lucic, Evgeny Chukharev-Khudilaynen, John Levis, and Ricardo Gutierrez-Osuna.  L2-ARCTIC: A non-native English speech corpus.  _Perception Sensing Instrumentation Lab_ , 2018.

##  Appendix A MAP estimation of $\hat{\mathcal{B}}$ and $\hat{\Pi}$

Given the sentence-level ML-FSA introduced in Sec. 5, the MAP estimate of
$\hat{\mathcal{B}}$ and $\hat{\Pi}$ can be obtained by maximizing the
following posterior distribution:

$\begin{split}p(y|x)&\propto p(y)p(x|y)\\\
&=\prod^{T}_{t=1}p(y_{t}|y_{1},...,y_{t-1})p(x_{t}|y_{t})\\\
&=\prod^{T}_{t=1}p(y_{t}|y_{1},...,y_{t-1})\frac{p(y_{t}|x_{t})p(x_{t})}{p(y_{t})}\\\
&\stackrel{{\scriptstyle
a}}{{\approx}}\prod^{T}_{t=1}p(y_{t}|y_{1},...,y_{t-1})\frac{q_{\phi_{p}}(y_{t}|x_{t})p(x_{t})}{p(y_{t})}\\\
&\stackrel{{\scriptstyle
b}}{{\propto}}\prod^{T}_{t=1}p(y_{t}|y_{1},...,y_{t-1})\frac{q_{\phi_{p}}(y_{t}|x_{t})}{p(y_{t})},\end{split}$
(9)

where in step $a$, $p(y_{t}|x_{t})$ is approximated by
$q_{\phi_{p}}(y_{t}|x_{t})$. In step $b$, the constant term $p(x_{t})$ is
dropped. $p(y_{t})$ is the prior of $y_{t}$, which is usually estimated with
the frequencies of different phonemes in the training dataset.
$q_{\phi_{p}}(y_{t}|x_{t})$ is the approximate posterior, which can be
computed using the phoneme estimator. $p(y_{t}|y_{1},...,y_{t-1})$ is the
transition probability which is determined by $p(b_{t})$ and $p(\pi_{t})$.
Concretely, when calculating the transition probability
$p(y_{t}|y_{1},...,y_{t-1})$ given the sentence-level FSA, three different
cases are considered: (1) the consecutive frames belong to the same segment;
(2) the consecutive two frames belong to different segments, and the current
segment _matches_ the phoneme sequence; (3) the consecutive two frames belong
to different segments, and the current segment contains content _mismatch_
(i.e., a mispronounced phoneme). Corresponding to these three cases, we have:

$p(y_{t}|y_{1},...,y_{t-1})=\begin{cases}p(b_{t}=0),&\text{if
}y_{t}=y_{t-1}\\\ p(b_{t}=1)p(\pi_{t}=0),&\text{if }y_{t}\neq y_{t-1}\text{
and }y_{t}\in{\mathcal{C}}\\\ p(b_{t}=1)p(\pi_{t}=1),&\text{if }y_{t}\neq
y_{t-1}\text{ and }y_{t}\in{\mathcal{C}}^{*},\end{cases}$ (10)

where ${\mathcal{C}}^{*}=(c_{1}^{*},...,c_{L}^{*})$ denotes the mismatched
elements in the phoneme sequence ${\mathcal{C}}$ (i.e. mispronounced
phonemes)222Note that strictly speaking, here we are approximating
$p(y_{t}|y_{1},...,y_{t-1})$ using
$p(y_{t}|y_{1},...,y_{t-1},c_{1},...,c_{t})$, where $c_{1},...,c_{t}$ are the
known canonical phonemes.. $p(\pi_{t}=0)$ and $p(\pi_{t}=1)$ are approximated
by $q_{\phi_{h}}(\pi_{t}=0|x_{t},y_{t},b_{t})$ and
$q_{\phi_{h}}(\pi_{t}=1|x_{t},y_{t},b_{t})$, which can be calculated using the
speech generator. Similarly, $p(b_{t}=0)$ and $p(b_{t}=1)$ are approximated by
$q_{\phi_{b}}(b_{t}=0|x_{t})$ and $q_{\phi_{b}}(b_{t}=1|x_{t})$, which can be
obtained from the boundary detector.

Generally, the MAP estimation of $\hat{\mathcal{B}}$ and $\hat{\Pi}$ can be
written as:

$\begin{split}\hat{\mathcal{B}},\hat{\Pi}&=\mathop{\rm
argmax}_{{\mathcal{B}},\Pi}{\mathcal{J}}_{{\mathcal{B}}\text{-}\Pi}({\mathcal{B}},\Pi;\Phi)\\\
&=\mathop{\rm
argmax}_{{\mathcal{B}},\Pi}\prod^{T}_{t=1}p(y_{t}|y_{1},...,y_{t-1})\frac{q_{\phi_{p}}(y_{t}|x_{t})}{p(y_{t})}.\end{split}$
(11)

In practice, MAP estimation is achieved by following Eq. 11 and searching for
the optimal path in ML-FSA, which can be solved by a DP algorithm.

##  Appendix B Estimating $\hat{\mathcal{B}}$ and $\hat{\Pi}$ separately

In this section, we describe the algorithm which estimates the hard
assignments for $\hat{\mathcal{B}}$ and $\hat{\Pi}$ separately. This algorithm
is used to perform the ablation study.

###  B.1 Estimation of $\hat{\mathcal{B}}$

Being different from our joint estimation algorithm discussed in the main
paper, the boundary variable $\hat{\mathcal{B}}$ is first estimated without
considering the correctness of the pronunciation.

When estimating $\hat{\mathcal{B}}$, we adopt the single-path FSA. Fig. 6
demonstrates the partial FSA for the $l$-th phoneme. From the initial state 0,
there is only one path pointing to state 1, standing for the $l$-th phoneme,
$c_{l}$. Then at each time step, since each phoneme may last for several
frames, it either still holds at state 1 (denoted by $H$), or moves forward to
the final state 2 (denoted by $S$). The sentence-level FSA can be constructed
by connecting all phoneme-level FSA together, according to the input phoneme
sequence ${\mathcal{C}}$.

Figure 6: Single-path FSA.

Based on such an FSA, the weights on the transitions can be estimated by ML-
VAE in a similar manner as described in Sec. 6.1. Then a DP algorithm can be
used to find the optimal path in the sentence-level FSA, after which the
estimated $\hat{\mathcal{B}}$ can be obtained by backtracking along the
optimal path.

###  B.2 Estimation of $\hat{\Pi}$

We then estimate $\hat{\Pi}$ given the $\hat{\mathcal{B}}$ obtained in the
above procedure. If we are given the $l$-th phoneme segment which starts at
$u$-th frame and ends at the $v$-th frame, we assume that all frames within
this segment share the same pronunciation correctness label, that is,
$\pi_{u}=\pi_{u+1}=\dots=\pi_{v}=\pi^{\prime}_{l}$, where $\pi^{\prime}_{l}$
denotes the pronunciation correctness of the $l$-th phoneme.

Therefore, similar to Eq. 11, we can obtain the MAP estimation of
$\pi^{\prime}_{l}$ per segment:

$\hat{\pi}^{\prime}_{l}=\mathop{\rm
argmax}_{\pi^{\prime}_{l}}\prod_{t=u}^{v}p(y_{t}|y_{1},...,y_{t-1})\frac{q_{\phi_{p}}(y_{t}|x_{t})}{p(y_{t})}=\mathop{\rm
argmax}_{\pi^{\prime}_{l}}\begin{cases}p(\pi_{s}=0)\prod_{t=u}^{v}\frac{q_{\phi_{p}}(y_{t}=c_{l}|x_{t})}{p(y_{t}=c_{l})},&\text{if
}\pi^{\prime}_{l}=0\\\ p(\pi_{s}=1)\prod_{t=u}^{v}\frac{q_{\phi_{p}}(y_{t}\neq
c_{l}|x_{t})}{p(y_{t}\neq c_{l})},&\text{if }\pi^{\prime}_{l}=1,\end{cases}$
(12)

where $q_{\phi_{p}}(y_{t}|x_{t})$, $q_{\phi_{p}}(y_{t}=c_{l}|x_{t})$, and
$q_{\phi_{p}}(y_{t}\neq c_{l}|x_{t})$ are the approximate posteriors.
$y_{t}=c_{l}$ denotes that the $t$-th frame is correctly pronounced
($\pi^{\prime}_{l}=0$). $y_{t}\neq c_{l}$ denotes that the $t$-th frame is
mispronounced ($\pi^{\prime}_{l}=1$), that is, $q_{\phi_{p}}(y_{t}\neq
c_{l}|x_{t})=1-q_{\phi_{p}}(y_{t}=c_{l}|x_{t})$, and $p(y_{t}\neq
c_{l})=1-p(y_{t}=c_{l})$.

##  Appendix C Case Study of ML-VAE

Figure 7: Case study for the last ten phonemes of the utterance YKWK_a0442.
The first and second rows show the input phonemes and the actual phonemes
pronounced by the speaker, respectively. The last row shows ML-VAE’s predicted
mispronunciation localization result.

Besides quantitative analysis, we also provide a case study to showcase ML-
VAE-RL in Fig. 7. Each rectangle in Fig. 7 represents one phoneme segment. The
first and second rows show the input phoneme sequence and the actual phonemes
pronounced by the speaker, respectively. The last row shows ML-VAE-RL’s
mispronunciation localization result. In the last row, a phoneme with a star
sign (e.g. ‘t*’) indicates that ML-VAE-RL detects a mispronounced phoneme
(e.g. a mispronounced ‘t’). It is shown that ML-VAE successfully detects three
out of four mispronunciations with a reasonable localization result.

##  Appendix D Case Study of Ablation Models

Figure 8: Case study for the last ten phonemes of the utterance YKWK_a0442.
The first and second rows show the input phonemes and the actual phonemes
pronounced by the speaker, respectively. The third row shows ML-VAE’s
predicted mispronunciation localization result. The last three rows show the
outputs of the three ablation models as presented in Table 3. To save space,
the phoneme err is denoted as e in this figure.

Fig. D shows the case study with the outputs of three ablation models
presented in Table 3. It is shown that the joint optimization model yields the
worst outputs because the poor alignment results. Compared with the other two
ablation models, ML-VAE clearly outperforms them in terms of localizing the
mispronounced phonemes.

##  Appendix E Implementation Details

In this section, we introduce the implementation details of ML-VAE. The input
feature sequence ${\mathcal{X}}$ is obtained by calculating the 40-dimension
FBANK feature Young et al. (2002) from the speech signal. The phoneme sequence
${\mathcal{C}}$ is obtained from the text which is read by the speaker.

###  E.1 Boundary Detector

The boundary detector includes two LSTM layers, each with 512 nodes, followed
by two fully connected (FC) layers with 128 nodes and ReLU activations. They
are followed by two separate output layers, each of which has only one node
and a Softplus activation function. The output of these two layers is used to
estimate the parameters $\alpha$ and $\beta$ to sample $v_{t}$, which is
further used to sample $b_{t}$. The weight of the KL term $\lambda_{b}$ is set
as 0.01.

###  E.2 Phoneme Estimator

The phoneme estimator contains two LSTM layers; each LSTM layer has 512 nodes,
followed by two FC layers, each with 128 nodes and ReLU. They are followed by
a Softmax layer to give the estimation of the phoneme.

###  E.3 Speech Generator

We adopt an encoder-decoder architecture to implement the speech generator.
The encoder consists of three FC layers, each with 32 nodes, followed by four
LSTM layers, each with 512 nodes. We use a FC layer with 512 nodes and ReLU to
estimate the mean and variance of the Gaussian components. The decoder
contains four bidirectional SRU layers (Lei et al., 2018), each with 512
nodes. They are followed by two FC layers, each with 120 nodes, to estimate
the mean and variance of the data distribution. During training, $\lambda_{r}$
and $\lambda_{l}$ are set to 1 and 0.001, respectively.

##### Gaussian Component Selection

As described in the generative process, the Gaussian component indicator
$z_{t}$ is sampled from $z_{t}|y_{t},\pi_{t},b_{t}\sim
Categorical(f_{z}(y_{t},b_{t},\pi_{t}))$. In this section, we describe the
implementation of $f_{z}(\cdot)$, which is defined as
$f_{z}(y_{t},b_{t},\pi_{t})=\text{softmax}(\rho_{t}*\epsilon+\delta_{t})$,
where $\rho_{t}$ is a scalar estimated by a two-layer multilayer perceptron
(MLP) taking as inputs $y_{t}$, $b_{t}$, and $\pi_{t}$. Each layer of the MLP
contains 128 nodes and a Sigmoid activation function; $\epsilon$ is a small
constant, which is set as $1\times 10^{-6}$ in our experiments; $\delta_{t}$
is an one-hot variable, i.e. $\delta_{t}[s]=1$, and $s$ is computed by:

$s=\begin{cases}(j-1)*(N_{m}+1)+1,&\text{if }\pi_{t}=0\\\
(j-1)*(N_{m}+1)+1+k,&\text{if }\pi_{t}=1,\end{cases}$ (13)

where $j$ denotes the phoneme label of $y_{t}$, i.e. $y_{t}[j]=1$. In case
that $y_{t}$ is mispronounced, $k\in[1,N_{m}]$ denotes the $k$-th
mispronunciation variant of the phoneme $y_{t}$, which is implemented by a
Gumbel Softmax function (Jang et al., 2016), i.e.
$\tau_{t}=\text{Gumbel}(\text{NN}(x_{t},y_{t}))$, where $\tau_{t}[k]=1$ and NN
is a simple neural network with three 128-node FC layers .

##  Appendix F Experimental Results of the Alignment Task

Table 4: Experimental results of aligning speech and text inputs with misproununciation on the L2-ARCTIC dataset. Model | Average IoU (%)
---|---
Two-Pass-FA | 4.68
w2v-CTC | 5.73
ML-VAE (ours) | 44.27
ML-VAE-RL (ours) | 53.02

This section provides some experimental results of aligning speech and text
inputs that contain mispronunciation, as shown in Table 4. Experiments are
performed on the real-world dataset L2-ARCTIC. The intersection-over-union
(IoU) is calculated for every phoneme in the input text and then averaged to
evaluate the alignment performance. The experimental results show that the
baselines, both the traditional forced alignment method and the CTC alignment
algorithm, fail to yield reasonable alignment results, while our proposed ML-
VAE and ML-VAE-RL can outperform both baselines in this task. Such results
demonstrate the traditional alignment results would fail when there is
mismatch in the inputs. For example, if there is a mispronunciation in the
middle of a sentence, it may cause errors in traditional alignment methods
(e.g., forced alignment). Then such errors may accumulate and further degrade
the alignment performance of the phonemes after the mispronounced one.


# Denoising Fast X-Ray Fluorescence Raster Scans of Paintings

###### Abstract

Macro x-ray fluorescence (XRF) imaging of cultural heritage objects, while a
popular non-invasive technique for providing elemental distribution maps, is a
slow acquisition process in acquiring high signal-to-noise ratio XRF volumes.
Typically on the order of tenths of a second per pixel, a raster scanning
probe counts the number of photons at different energies emitted by the object
under x-ray illumination. In an effort to reduce the scan times without
sacrificing elemental map and XRF volume quality, we propose using dictionary
learning with a Poisson noise model as well as a color image-based prior to
restore noisy, rapidly acquired XRF data.

pt© 2022 IEEE. Personal use of this material is permitted. Permission from
IEEE must be obtained for all other uses, in any current or future media,
including reprinting/republishing this material for advertising or promotional
purposes, creating new collective works, for resale or redistribution to
servers or lists, or reuse of any copyrighted component of this work in other
works.

Index Terms—  x-ray fluorescence imaging, image denoising, image restoration,
cultural heritage science

##  1 Introduction

In the growing field of applying scientific methods to cultural heritage
research, macro x-ray fluorescence (XRF) imaging is frequently used as a non-
invasive tool to analyze works of art. This approach leverages the insights
gained from XRF point analysis in providing elemental information on a per-
pixel basis. These elemental distribution maps provide information as to what
chemical elements compose the layers of paint. With these maps for example, an
art conservator can better preserve paintings [1], or an art historian can
deduce an artist’s painting techniques—sometimes revealing hidden paintings
[2].

In macro XRF imaging, a source excites a small target area of the painting by
irradiating it with x-rays. An inner orbital electron can be ejected if the
impinging x-ray has greater energy than the electron’s binding energy. An
electron at an outer orbital then drops to fill the inner orbital vacancy by
emitting a photon of energy equal to the energy difference of the orbitals.
Each element has characteristic orbital energy levels (and therefore a
characteristic XRF spectrum). A detector and digital post processor records
and bins each photon according to its energy.

Fig. 1: Jan Davidsz. de Heem’s Bloemen en Insecten, $49\times 67$ cm, Royal
Museum of Fine Arts Antwerp, inv. no. 54, oil on canvas.

While macro XRF is a powerful, increasingly popular technique, acquiring
elemental maps for entire paintings with good signal-to-noise ratios, often
translates to long acquisition times. Depending on the painting size, spot
size, and dwell time, it can take many hours to acquire the XRF volume. Take
as an example a modest painting of size $600x720$ mm. If we specify a scan
with spot size 1 mm2 and dwell time $0.2$ s/px, it would take exactly $1$ day
to scan. There are two problems in that (1) access to paintings often occur in
short time windows when they are off-view, en route to other sites, etc., and
(2) the x-ray exposure time should be minimized to best preserve the painting.

Analysis of XRF volumes uses photon count rates instead of photon counts, as
the dwell time can vary by scan. These volumes are then separated into
elemental maps using a least squares fit where the feature matrix is composed
of known elemental XRF responses. Before collecting XRF data, a trade-off
between image quality (e.g. mean-square error (MSE)) and time must be taken
into account: the longer the dwell time, the more accurate the measured photon
count rates from the count-limited photon data. Our goal here is to develop an
XRF denoising algorithm where we test it on simulated scans at different dwell
times based on real XRF data. We focus on Jan Davidsz. de Heem’s Bloemen en
Insecten as shown in figure (1), the data of which has been generously shared
by de Keyser et al. [3].

##  2 Related Work

Dictionary learning approaches frequently appear in XRF literature since each
element emits a characteristic set of discrete fluorescent lines. Limiting the
number of spectral representations to the number of elements makes intuitive
sense, as each pixel is then a linear combination of different elemental
spectra. Martins et al. proposed denoising XRF volumes using multivariate
curve resolution-alternating least squares (MCR-ALS), a simple dictionary
learning approach in the spectral domain to separate elemental compositions
[4, 5]. Kogou et al. used an unsupervised learning method called self-
organizing maps (SOMs) that also extracts a set of spectral dictionary atoms
to decompose the XRF volumes into a representative basis [6]. This method
effectively uses k-means clustering to generate the set of dictionary
endmembers. More elaborate dictionary methods have been explored by Dai et al.
whereby joint RGB and XRF dictionaries inpaint a spatially selective
subsampled XRF volume [7].

Even though photons arrive according to a Poisson process [8], each of these
methods (implicitly) uses a white Gaussian noise model since the dwell times
are assumed to be long. This noise model was shown to be a good approximation
in XRF denoising due to the central limit theorem, but can break down with
short dwell times when white Gaussian noise is no longer an accurate
approximation as our experiments show.

PURE-LET from Luisier et al. is an algorithm specifically for Poisson image
denoising that minimizes the Poisson unbiased risk estimate in the Haar-
wavelet domain [9]. This method was originally published using tests on
conventional images, MRI brain data, and fluorescence-microscopy of biological
samples. To the best of our knowledge, it has not been applied to XRF data,
but is another tool that can be used as it partially addresses the concerns of
current dictionary learning approaches for XRF denoising.

Our method merges the best characteristics of the two solution approaches: a
spectral dictionary learning approach with a Poisson model (instead of a
Gaussian model) to denoise XRF volumes. An RGB image prior and sparsity coding
are also used for denoising the data.

##  3 Algorithm

Assume for now that we have the XRF volume from a fast raster scan,
$\mathbf{X}\in\mathbb{N}^{0pt\times 0pt\times C}$ of height $0pt$, width
$0pt$, and channels (i.e. energy bins) $C$. Each pixel has an identical dwell
time $t$. The photons arrive with unknown underlying photon arrival rate
$\mathbf{\Lambda}\in\mathbb{R}^{0pt\times 0pt\times C}_{+}$. Additionally,
assume we have an RGB image of the painting
$\mathbf{I}\in\left[0,1\right]^{0pt\times 0pt\times 3}$ registered with the
XRF data. We estimate the count rate
$\mathbf{\Lambda}^{*}\approx\mathbf{\Lambda}$ using $\mathbf{X}$,
$\mathbf{I}$, and $t$ in our optimization formula detailed here.

###  3.1 Formulation

(a) Pb L3 | (b) Cu K | (c) Ca K | (d) Co K | (e) As K | (f) Cl K | (g) Si K
---|---|---|---|---|---|---
Fast Scan |  |  |  |  |  |
898149 | (-10797) |  18724 | (-7328) |  6208 | (-1245) |  4116 | (-151.95) |  336360 | (173.01) |  312.40 | (-4.78) |  137.24 | (0.7262)
PURE-LET |  |  |  |  |  |
36777 | (-11369) |  55864 | (-7326) |  4912 |  (-1255) |  299 | (-183.82) |  9108 |  (-103.04) |  22.55 |  (-14.99) |  12.89 | (-3.6336)
MCR-ALS |  |  |  |  |  |
420317 | (-11186) |  19160 | (-7326) |  6178 | (-1250) |  332 | (-183.83) |  191172 | (110.99) |  24.26 | (-14.91) |  12.32 |  (-3.6893)
Ours |  |  |  |  |  |
21904 |  (-11373) |  13747 |  (-7333) |  8240 | (-1247) |  290 |  (-184.06) |  10792 | (-96.61) |  23.49 | (-14.95) |  12.42 | (-3.6813)
Ground Truth |  |  |  |  |  |
0 | (-11380) |  0 | (-7342) |  0 | (-1262) |  0 | (-186.66) |  0 | (-172.48) |  0 | (-16.10) |  0 | (-5.4176)
Fig. 2: Visual comparison of seven elemental maps in the same display range
per column. Row 1: the simulated fast XRF scan at $14.25$ ms/px. Row 2: PURE-
LET denoising. Row 3: MCR-ALS denoising. Row 4: our denoising algorithm. Row
5: Ground truth XRF scan at $285$ ms/px. Numbers below are the PSNR and PNLL
(in parentheses) respectively.

Recall that the XRF signal is a combination of elemental spectra. Each element
has its own unique XRF response that we can exploit for sparse coding, which
has been shown to be effective in signal denoising [10]. Let

$\bar{\mathbf{X}}=DA$ (1)

be a matrix reordering of $\mathbf{X}$ where $D\in\mathbb{R}^{C\times M}_{+}$
is the dictionary with $M$ non-negative atoms representing spectral responses,
and $A\in\mathbb{R}^{M\times N}_{+}$ is the sparse abundance matrix. Each
pixel of $\bar{\mathbf{X}}\in\mathbb{R}^{C\times N}_{+}$ is organized as
$N=0pt\cdot 0pt$ column vectors with $C$ features. Learning the non-negative
dictionary $D$ and abundance matrix $A$ provides both a spectrally smooth XRF
volume and a more accurate representation of the chemical processes governing
XRF data acquisition.

When scanning each pixel, photons of different energies arrive according to a
Poisson sum model, which can be split into multiple independent Poisson
processes [11]. Each process here describes the number of photon arrivals at
each energy. Each pixel we also assume to be spatially independent from one
another. Thus, we use the Poisson negative log likelihood (PNLL) loss as the
data fidelity term:

$\mathcal{P}\left(\mathbf{X}\right)=\sum_{h=1}^{0}pt\sum_{w=1}^{0}pt\sum_{c=1}^{C}\mathbf{X}_{h,w,c}-t\,\mathbf{\Lambda}_{h,w,c}\cdot\ln\left(\mathbf{X}_{h,w,c}\right).$
(2)

Since $\mathbf{\Lambda}$ is unknown, we instead try to best match the
decomposition results with the data we record, namely:

$\mathcal{P}\left(D,A\right)=\sum_{c=1}^{C}\sum_{n=1}^{N}\left(DA\right)_{c,n}-\bar{\mathbf{X}}_{c,n}\cdot\ln\left(\left(DA\right)_{c,n}\right)$
(3)

which combines eq. (1) and eq. (2) together.

The RGB image, $\mathbf{I}$, provides valuable and rudimentary insight into
the spatial structure of each channel of the XRF volume. Local areas similar
in color likely have similar elemental profiles, and local areas of different
colors likely have different elemental profiles. The spatial gradient of
$\mathbf{I}$ contains this information. Define a total variation (TV)
regularizer that adapts to the RGB gradient by:

$\displaystyle\text{TV}\left(\tilde{A}\right)=$
$\displaystyle\sum_{h=1}^{0pt-1}\sum_{w=1}^{0}pt\sum_{c=1}^{C}\frac{\Omega^{0}pt_{h,w}}{t^{2}}\left(\tilde{A}_{h+1,w,c}-\tilde{A}_{h,w,c}\right)^{2}+$
$\displaystyle\sum_{h=1}^{0}pt\sum_{w=1}^{0pt-1}\sum_{c=1}^{C}\frac{\Omega^{0}pt_{h,w}}{t^{2}}\left(\tilde{A}_{h,w+1,c}-\tilde{A}_{h,w,c}\right)^{2}$
(4)

where $\tilde{A}\in\mathbb{R}^{0pt\times 0pt\times M}$ is the volumetric
representation of $A$ with the same pixel ordering as $\mathbf{X}$, and

$\displaystyle\Omega^{0}pt_{h,w}$
$\displaystyle=\exp\left(-\beta\sum_{c=1}^{3}\left(\mathbf{I}_{h+1,w,c}-\mathbf{I}_{h,w,c}\right)^{2}\right),$
(5) $\displaystyle\Omega^{0}pt_{h,w}$
$\displaystyle=\exp\left(-\beta\sum_{c=1}^{3}\left(\mathbf{I}_{h,w+1,c}-\mathbf{I}_{h,w,c}\right)^{2}\right)$
(6)

with $\beta>0$ as a hyperparameter. Adaptive weights $\Omega^{H}$ and
$\Omega^{W}$ are large when the RGB gradient is small and vice-versa. The
division by $t^{2}$ normalizes the TV regularizer result by time. This TV
regularizer is adapted from Dai et al. [7] with the dwell time factored in.

We now define the full optimization problem as a weighted sum of eqs. (3) and
(4) with the addition of a weighted $l_{0}$ norm of $A$ to enforce sparsity
constraints:

$D^{*},A^{*}=\operatorname*{arg\,min}_{D,A\geq
0}\;\mathcal{P}\left(D,A\right)+\lambda_{\text{TV}}\,\text{TV}\left(\tilde{A}\right)+\lambda_{l_{0}}\,\lVert
A\rVert_{0}.$ (7)

Once we have the optimized sparse representation of the signal, we can then
find the optimized XRF photon rates by:

$\bar{\mathbf{\Lambda}}^{*}=\frac{1}{t}\,D^{*}A^{*}.$ (8)

$\bar{\mathbf{\Lambda}}^{*}$ can be reshaped to the volumetric
$\mathbf{\Lambda}^{*}$ for analysis.

###  3.2 Solution

In order to solve eq. (7), we need a good initialization of $D$ and $A$ as
well as a way to relax the $l_{0}$ norm. We use K-means clustering of the
spectral domain of all the pixels to initialize the dictionary. A non-negative
least squares fit of this dictionary and the XRF data quickly finds an optimal
abundance matrix in a least-squares sense. Elastic Net loss [12] and the Least
Absolute Shrinkage and Selection Operator (LASSO) [13] replace the $l_{0}$
norm. Elastic Net penalizes elements of $A$ by a combination of the $l_{1}$
and $l_{2}$ norms:

$\mathcal{L}_{\text{EN}}\left(A\right)=\alpha\,\lVert
A/t\rVert_{2}^{2}+\left(1-\alpha\right)\lVert A/t\rVert_{1}.$ (9)

LASSO sets values in $A$ below a threshold to zero and removes those elements
from future updates. The implemented optimization equation is then updated
from eq. (7) as:

$D^{*},A^{*}=\operatorname*{arg\,min}_{D,A\geq
0}\;\mathcal{P}\left(D,A\right)+\lambda_{\text{TV}}\,\text{TV}\left(\tilde{A}\right)+\lambda_{\text{EN}}\,\mathcal{L}_{\text{EN}}\left(A\right).$
(10)

Abundance matrix $A$ is updated using the Adam optimizer [14] until
convergence which we define as when $i=i_{\text{min}}+j$ where $i$ is the
current iteration number, $i_{\text{min}}$ is the iteration number with the
smallest loss, and $j$ is a threshold. Dictionary $D$ is updated also using
Adam in an alternating fashion with $A$. The optimal $D$ given a fixed $A$
cannot easily be found analytically due to the nature of the PNLL loss. Thus,
we turn to gradient descent-based methods.

##  4 Experiments & Results

Jan Davidsz. de Heem’s Bloemen en Insecten as shown in fig. (1) was scanned by
de Keyser et al. [3]. It consists of 2048 photon energy channels and has a
resolution of $578\times 673$ after registering the RGB image to the target
XRF volume. In our experiments, we treat this volume
$\mathbf{Y}\in\mathbb{R}^{578\times 673\times 2048}$ as the ground truth
photon count. The dwell time per pixel for this acquisition was reported as
$t_{\mathbf{Y}}=285$ ms/px. Scanning an area of $578\times 673$ spots with
this dwell time would require over 30 hours of scanning. The ground truth rate
$\mathbf{\Lambda}_{\mathbf{Y}}$ can be found by dividing $\mathbf{Y}$ by
$t_{\mathbf{Y}}$.

We identified 37 elements likely to compose a painting, leading to our choice
of $M=37$ dictionary atoms. Additionally, we set $\lambda_{\text{TV}}=10^{-2}$
and $\lambda_{EN}=10^{-4}$. Hyperparameters $\beta=16$ of eqs. (5) and (6),
and $\alpha=0.2$ of eq. (9).

Using $\mathbf{\Lambda}_{\mathbf{Y}}$, we simulate raster scans at various
dwell times from a $5$-fold speedup ($57$ ms/px, about 6 hours scanning) to a
$100$-fold speedup ($2.85$ ms/px, about 19 minutes scanning). The MSE and mean
PNLL error are our comparison metrics. We compare against (1) PURE-LET2 with
cycle-spinning ($5$ cyclic shifts) and $5$ Haar wavelet scales [9, 15], (2)
our implementation of MCR-ALS [4] also with $37$ dictionary endmembers, and
(3) the original simulated data without optimization. PyMca [16], a platform
for XRF analysis, is used to generate the elemental maps for all the XRF
volumes.

Fig. (2) shows some elemental maps of varying count rates and their
corresponding MSEs and PNLLs. The dwell time for those elemental maps is
$0.05\,t_{\mathbf{Y}}\approx 14.25$ ms/px ($\approx 92$ minute scan time). A
volumetric comparison of the results are shown in the plots of fig. (3).

We see in fig. (2) that our method performs best overall when considering
error metrics and visual quality. PURE-LET does well in reducing the error,
especially for the lower dwell times as seen in fig. (3(b)), but qualitatively
the maps tend to be oversmoothed (see Pb L3, Cl K, and Si K), making it
difficult for further analysis. MCR-ALS has the opposite problem in that the
visual quality is high, but when examining fig. (3(a)) and the elemental map
metrics, the error on average is much higher than the other methods. Noise
tends to be most present in this method (see Pb L3 and As K), and some
artifacts may be present. For example, the Co K line, while visually
appealing, is overestimated in some areas as opposed to something more uniform
as the ground truth might suggest (see orange flower in the top left quadrant
and central white flower). Similarly in Cl K the same orange flower is
overestimated as the ground truth shows a lower count rate at the bottom of
the flower.

Our method generally shows well-denoised elemental maps both numerically and
visually. In terms of denoising the XRF volume as a whole, we outperform all
the algorithms in MSE starting at about $11.5$ ms and in PNLL starting at
about $7.5$ ms. Visually, we are most consistent with the ground truth as
well.

(a) Errors of the optimized XRF volumes. The minimum PNLL for
$\mathbf{\Lambda}_{\mathbf{Y}}$ is $-6.6956$.

(b) Close up error of the optimized XRF volumes only

Fig. 3: Error of the XRF volumes

##  5 Conclusion

We introduced a new method for denoising XRF volumes that combines a Poisson
noise model with sparse dictionary learning. Our algorithm outperforms methods
designed for XRF denoising and Poisson denoising in general in quantitative
and qualitative terms. Speedups of a factor of 20 can not only ease time-
related issues for accessing works of art, but could also open the opportunity
for researchers to scan more paintings in a session. Despite the speedup, our
algorithm can still recover high-quality elemental maps more denoised than
even the original data itself. This allows more paintings to be analyzed for
historical research and more quickly address conservation concerns.

## References

  * [1] K. Janssens, G. Van Der Snickt, M. Alfeld, P. Noble, A. van Loon, J. Delaney, D. Conover, J. Zeibel, and J. Dik,  “Rembrandt’s ‘saul and david’ (c. 1652): Use of multiple types of smalt evidenced by means of non-destructive imaging,”  Microchemical Journal, vol. 126, pp. 515–523, 2016.
  * [2] E Pouyet, K. Brummel, S. Webster-Cook, J. Delaney, C. Dejoie, G. Pastorelli, and M. Walton,  “New insights into pablo picasso’s la miséreuse accroupie (barcelona, 1902) using x-ray fluorescence imaging and reflectance spectroscopies combined with micro-analyses of samples,”  SN Applied Sciences, vol. 2, no. 8, pp. 1–6, 2020.
  * [3] N. De Keyser, G. Van der Snickt, A. Van Loon, S. Legrand, A. Wallert, and K. Janssens,  “Jan davidsz. de heem (1606–1684): a technical examination of fruit and flower still lifes combining ma-xrf scanning, cross-section analysis and technical historical sources,”  Heritage Science, vol. 5, no. 1, pp. 1–13, 2017.
  * [4] A. Martins, J. Coddington, G. Van der Snickt, B. van Driel, C. McGlinchey, D. Dahlberg, K. Janssens, and J. Dik,  “Jackson pollock’s number 1a, 1948: a non-invasive study using macro-x-ray fluorescence mapping (ma-xrf) and multivariate curve resolution-alternating least squares (mcr-als) analysis,”  Heritage Science, vol. 4, no. 1, pp. 1–13, 2016.
  * [5] A. Martins, C. Albertson, C. McGlinchey, and J. Dik,  “Piet mondrian’s broadway boogie woogie: non invasive analysis using macro x-ray fluorescence mapping (ma-xrf) and multivariate curve resolution-alternating least square (mcr-als),”  Heritage Science, vol. 4, no. 1, pp. 1–16, 2016.
  * [6] S. Kogou, L. Lee, G. Shahtahmassebi, and H. Liang,  “A new approach to the interpretation of xrf spectral imaging data using neural networks,”  X-Ray Spectrometry, vol. 50, no. 4, pp. 310–319, 2021.
  * [7] Q. Dai, H. Chopp, E. Pouyet, O. Cossairt, M. Walton, and A. K. Katsaggelos,  “Adaptive image sampling using deep learning and its application on x-ray fluorescence image reconstruction,”  IEEE Transactions on Multimedia, vol. 22, no. 10, pp. 2564–2578, 2020.
  * [8] T. J. Holmes and Y. H. Liu,  “Acceleration of maximum-likelihood image restoration for fluorescence microscopy and other noncoherent imagery,”  JOSA A, vol. 8, no. 6, pp. 893–907, 1991.
  * [9] F. Luisier, C. Vonesch, T. Blu, and M. Unser,  “Fast interscale wavelet denoising of poisson-corrupted images,”  Signal processing, vol. 90, no. 2, pp. 415–427, 2010.
  * [10] M. Elad,  Sparse and redundant representations: from theory to applications in signal and image processing,  Springer, 2010.
  * [11] R. G. Gallager,  Stochastic processes: theory for applications,  Cambridge University Press, 2013.
  * [12] H. Zou and T. Hastie,  “Regularization and variable selection via the elastic net,”  Journal of the royal statistical society: series B (statistical methodology), vol. 67, no. 2, pp. 301–320, 2005.
  * [13] R. Tibshirani,  “Regression shrinkage and selection via the lasso,”  Journal of the Royal Statistical Society: Series B (Methodological), vol. 58, no. 1, pp. 267–288, 1996.
  * [14] D. P. Kingma and J. Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.
  * [15] R. R. Coifman and D. L. Donoho,  “Translation-invariant de-noising,”  in Wavelets and statistics, pp. 125–150. Springer, 1995.
  * [16] V. A. Solé, E. Papillon, M. Cotte, P. Walter, and J. Susini,  “A multiplatform code for the analysis of energy-dispersive x-ray fluorescence spectra,”  Spectrochimica Acta Part B: Atomic Spectroscopy, vol. 62, no. 1, pp. 63–68, 2007.