# Amplifying the Chirp: Using Deep Learning (U-Nets) to filter signal from
noise in LIGO data

Akshay Ghalsasi<EMAIL_ADDRESS>Pittsburgh Particle Physics, Astrophysics, and
Cosmology Center, Department of Physics and Astronomy, University of
Pittsburgh, Pittsburgh, USA

###### Abstract

The direct detection of gravitational waves by LIGO has heralded a new era for
astronomy and physics. Typically the gravitational waves observed by LIGO are
dominated by noise. In this work we use Deep Convolutional Neural Networks
(specifically U-Nets) to filter a clean signal from noisy data. We present two
realizations of U-Net filters, the Noise2Clean U-Net filter which is trained
using noisy and clean realizations of the same signal, as well as Noise2Noise
U-Net which is trained on two separate noisy realization of the same signal.
We find that the U-Nets successfully filter signal from noise. We also
benchmark the performance of U-Nets by using them to detect the binary
presence or absence of gravitational wave signals in data.

††preprint: PITT-PACC-2119

##  I Introduction

The observation of gravitational waves by Laser Interferometer Gravitational-
wave Observatory (LIGO) [1] has opened up an entirely new observational
window, with far reaching implications for astrophysics, particle physics and
cosmology. The LIGO-Virgo-Kagra (LVK) gravitational observatory [1, 2] has
detected $\mathcal{O}(100)$ events so far, with the O4 run expected to detect
several hundred more events.

The events observed in LIGO have a very small amplitude compared to noise. In
order to reliably detect a signal, the process of matched filtering is used
(see for e.g. [3]), where the measured data is compared against a template
bank of simulated signals. This process is expensive since the template bank
contains $250,000$ different signals, corresponding to different parameters of
the merger i.e. black hole masses, inclination of the binary etc.

Matched filtering provides valuable information regarding the merging binary
system. In this work, we will focus on a simpler task, i.e. detection of
presence or absence of gravitational waves in a given measured time series. We
take our inspiration and data from the Kaggle G2Net competition which focuses
on the same task [4]. We will benchmark our answers to the dataset provided in
the Kaggle competition (thus henceforth we will refer to this as the benchmark
dataset). The metric for binary classification in the Kaggle competition is
the area under the curve (AUC) for our receiver operating characteristic (ROC)
curve, and we will use the same here.

Increasing compute and advances in deep learning has resulted in deep neural
networks being used to solve a variety of previously tough problems. Deep
learning has also been previously used for detecting signal in LIGO
observations as well as parameter estimation [5, 6]. The signal observed at
LIGO can be converted into the so-called “chirp plot” by performing a
continuous wavelet transformation (CWT) of the signal. The result of the CWT
transformation is an image which shows the presence of frequencies as time
progresses (see Fig. 1 for an example). In the presence of a gravitational
wave signal shows the presence of a “chirp” i.e. a signal with increasing
frequency as time progresses. This makes gravitational waves a particularly
nice targets for convolutional neural networks, especially deep vision models
which excel at the task of parsing a given image. This is the strategy we will
use in this work, with the added innovation of filtering out the noise and
making it easy for the vision CNN to detect the presence of the gravitational
wave.

The remaining paper will be divided as follows. In Sec. II, we will describe
the data we use as well as the basic preprocessing we perform to enhance the
signal compared to noise. Here we will also describe the synthetic data we
generate to train our Noise2Clean U-Net. In Sec. III we establish a baseline
against which we will benchmark the effects of our U-Net filters. In Sec. IV
we will describe the architecture of the convolutional neural network U-Net
and our modification of it to filter the signal from the noise. In Sec.IV.1 we
will describe the training of our Noise2Clean U-Net using synthetic data we
generate. In Sec.IV.2, we will describe an alternate filtering technique of
training a U-Net through the Noise2Noise procedure. In Sec. V, we will discuss
our results by using our procedure on our benchmark dataset provided in the
Kaggle competition. We will benchmark our results against the standard
procedure of simply using a vision net to classify the presence of a
gravitational wave signal. Finally we will conclude in Sec.VI and discuss the
multiple avenues of future work.

##  II Data and Preprocessing

The data that we will use to benchmark our U-Net filtering procedure will come
from the Kaggle competition [4] (see here). The data is of 560,000 noisy time-
series with labels, half of which have the presence of a GW signal and half do
not. However to reduce the computational time needed, we will only use 1/4th
of the total dataset for our benchmark dataset. The data consists of mock
observations from the Hanford and Livingston sites of the LIGO detector as
well as the Virgo observatory. Each datapoint is a time series of 2 seconds
sampled at a frequency of 2048 Hz for each of the LIGO detector Livingston
(L), Hanford (H) and Virgo. For the purposes of this work, we will not
consider the signal from the Virgo observatory111This will result in a sub-
optimal classification, however the two LIGO detectors have same or similar
noise templates making our job of filtering the noise easier. We will split
the data as $90\%$ for training and $10\%$ for testing (validation).

In the benchmark dataset described above, we do not have access to clean
signals. To get around this problem we generate synthetic LIGO signals using
the GGWD python package [7] (see this Github repo) for code. GGWD package is a
wrapper around LIGO’s PyCBC package [8] which is used by the LIGO
collaboration to generate synthetic gravitational waves from mergers of black
holes.

While the benchmark dataset is itself a synthetic dataset, the parameters used
to generate the gravitational wave signals in that dataset are not known. The
exact shape and amplitude of the gravitational wave signal depends on several
parameters, such as the mass of the two black holes, the inclination of their
orbit to the detectors and the distance of the merger event. For of our
synthetic dataset we use $M_{1,2}\subset[10,80]M_{\odot}$, a uniform prior on
spin between $(0,0.998)$, a signal-to-noise ratio of 10, a sin-angle prior on
inclination, and uniform prior for all other parameters. We generate 140,0000
synthetic timeseries (70,000 each for Hanford and Livingston observatories) of
3 seconds interval each. The benchmark dataset consists of ”measurements” over
a 2 second interval with the merger showing up in approximately the last half
a second. To conform to the length of the measurement as well as the location
of the merger, we chop our synthetic waveform into 2 second interval, while
ensuring that the merger happens between (1.45, 1.95) seconds. We expect that
this provides adequate coverage on the benchmark dataset.

To summarzie, we use two different dataseta

  * •

Benchmark dataset – This dataset comes from Kaggle G2Net competition and we
use it to benchmark our U-Net filtering procedure as well as to train our
Noise2Noise U-Net. This dataset consists of noisy time-series half of which
have a gravitational signal present.

  * •

Synthetic dataset – This dataset comes is generated by us to have access to
clean signal time-series i.e. without time-series without the presence of
noise. We use this dataset to train our Noise2Clean U-Net.

Typically at the raw noisy timeseries it is impossible to determine by eye,
the presence of a gravitational wave signal since the noise completely
dominates. Plotting the average power spectral density of the time series, we
can see that the timeseries is completely dominated by low frequency noise as
well as a few single frequency sources at higher frequency (See Fig. 8 ).
Gravitational waves from mergers are expected to have frequencies between
$20-500$ Hz, with the amplitude of the signal growing with higher frequency
(the “chirp”).

Figure 1: Raw as well as whitened time series of a particularly prominent
signal from our benchmark dataset. Left: The raw gravitational wave time
series and its corresponding CWT transform. The signal is not visible to the
eye in the time series and is completely overwhelmed by low frequency noise.
Right: Whitened time series. The signal is visible in the time series as well
as the CWT transform.

To make the time series smoothly go to zero at the endpoints, we convolve the
filtered signal with a tukey window with $\alpha=0.2$. To minimize the low
frequency noise, we perform the procedure known as whitening. In Fig. 1 we
have plotted an example from the raw data from our benchmark dataset. As can
be seen, whitening the signal enhances the gravitational wave signal compared
to noise. Further details on the whitening procedure have been provided in the
Appendix A. The above prepossessing is performed before training in all of our
procedures described below.

In all of the work, we do not use the time series signal itself but rather the
Complex Wavelet Transform (CWT) of the signal. The CWT is obtained by
convolving the time series signal with a wavelet that is continuous in time
and frequency space. The output of a CWT of a time series is a 2 dimensional
“image” which shows the presence of frequencies as a function of time. The CWT
transform of a binary BH merger gives rise to a a characteristic “chirp” plot
(see eg. Fig. 1) where the prevalent frequencies increase with increasing time
until the BHs merge. Note that our complex CWT transform has both the real and
imaginary part, but we only consider the magnitude here, thus losing the phase
information. Most of our work will involve enhancing the chirp with respect to
the background noise to make it more prominent and detectable.

##  III Establishing a Baseline

To quantify the performance of our methods, we need to establish a baseline to
compare ourselves to. Here we turn to the standard technique which involves
converting the time series signals to an image via a CWT transformation, and
then use transfer learning using vision nets to detect the presence of a
chirp, thereby classifying the presence or absence of gravitational waves
222This method was the most prevalent in the Kaggle G2Net competition, and
performs almost as well as the best solution.. In particular, we will use
EfficientNet architecture [9] ( specifically EfficientNetB7)for classifying
the presence of gravitational waves.

For our EfficientNetB7, we remove the top layer and replace it with a fully
connected binary classifier (a single neuron as output). We will establish two
benchmarks, the first one where only the last layer of EfficientNet is trained
and all the other layers are frozen, and one where we train the entire
EfficientNet. The EfficientNet is initialized with ImageNet weights. Our
training pipeline is shown in Fig. See Table. 1 for baseline benchmark
results.

The EfficientNet takes an input of three channels (RGB). The U-Net filtered
CWT images of Hanford and Livingston serve as two of the three channels. The
third channel is an “average” constructed out of the the Hanford and
Livingston CWT images defined as $2\frac{H*L}{max(H+L)}$, where $*$ refers to
an elementwise multiplication.

Figure 2: Training pipeline to establish our benchmark.

##  IV U-Net as a filter

Even with the data preprocessing described in Section II, the limitation on
achieving a perfect classification of the presence of gravitational wave
signals comes from the presence of noise which in most cases overwhelms the
signal. While the EfiicientNet does perform the task of detecting features
within images, and is effectively “denoising” the image by focusing on
features that indicate the presence of a gravitational wave signal, we want to
investigate whether better classification can be obtained if we present the
EfficientNet classifier with a denoised image.

In this work, we will use U-Nets to filter noise from signal. U-Nets were
originally invented for the purposes of biomedical image segmentation to
detect the presence of tumors in medical images [10]. They are also used for
image segmentation in autonomous driving. Since they segment images into
relevant and irrelevant parts, they are a natural choice for filtering signal
from noise in an image. The U-Net architecture we use is described in more
detain in the Appendix B.

Since the input and output dimensions of a U-Net are the same we can take two
approaches to filter signal from noise. The first one involves training a
noisy GW signal as input and the corresponding clean GW signal as the target.
Then we can train the neural network to filter out the signal from the noise.
This is commonly referred to as Noise2Clean (N2C) in literature. The second
approach involves training the U-Net where both input and output are separate
noisy realizations of the same signal. This method is known as Noise2Noise
(N2N) [11]. The trained U-Net then learns the statistical average of noisy
signals which is a clean signal. We will implement both N2C and N2N neureal
networks to filter noise from signal.

###  IV.1 U-Net Noise2Clean Filter

The image from the CWT transformation of the clean signal from our synthetic
dataset acts as the target for training of our neural network 333We add a
small amount of Gaussian random noise to the clean signal. This helps the
neural network to not get stuck in the false trivial minima where it simply
convolves the input image to zero.. For input, we use the CWT image of the
noisy (signal+noise) signal measurement. The signal is the synthetic signal we
generated, whereas the noise part of the measurement is sampled randomly from
the pure noise (no gravitational wave present) data of the benchmark dataset.
The noisy measurement is then preprocessed by the prescrpition given in Sec.
II and the CWT of the processed noisy signal is the used as the input in the
U-Net training. The U-Net is the trained by iterating over the signals
(130,000 training, 10,000 validation) in the synthetic dataset for several
epochs, until the relevant metric (‘Validation RMSE’) converges. Each time the
metric converges, we lower the SNR of the signal by multiplying it with a
$\mathcal{O}(1)$ number and retrain. This allows us to filter even weak
signals. This process is terminated once we confirm that a large part of the
validation data-set is no longer being filtered appropriately.

In Fig. 3 we have plotted the trained U-Net filter applied to the validation
dataset of the synthetic GW dataset we generated. It can be seen that the
U-Net filters out the signal that is not obviously visible in the CWT images.
In Fig. 4, we show the effect of the U-Net N2C filter on our benchmark
dataset. As can be seen the U-Net generalizes well to signal not generated by
us and successfully filters the signal for our benchmark dataset.

Figure 3: In the figure above we show the effects of our N2C U-Net filter on
the validation dataset. The left figure (the input to the U-Net) shows the CWT
image of a noisy signal from the validation data-set. The signal is not
prominently visible through the noise. Middle figure (the target of the U-Net)
is the CWT image of the true clean signal. The right figure (the output of the
U-Net) shows the effect of the N2C U-Net applied to the noisy CWT image. As
can be seen it reconstructs the true signal reliably.

In order to quantify the effect of the U-Net in classification of the presence
of gravitational waves, we add the U-Net to our classification pipeline, by
filtering the CWT images of the noisy signals from the benchmark dataset
through our trained U-Net before feeding it to the EfficientNet for
classification (See Fig. 5). Since we train the filter only on the presence of
a signal, we will be prone to false positives. As mentioned in the previous
section, the three channels of the input to the EfficientNet are the U-Net
filtered images of Hanford (H) Livingston (L) and the average
$\frac{H*L}{max(H+L)}$. Since the gravitational wave signals at Hanford and
Livingston signals will overlap in the U-Net filtered image, the averaging
procedure we perform seeks to remove any false positives which would not
necessarily overlap. We train the EfficientNet for the same benchmarks as our
baseline model. Our results are reported in Table 1

Figure 4: In the figure above we show the effects of our N2C U-Net filter on
the validation data-set. The top figures (the input to the N2C U-Net) shows
the CWT image of a whitened noisy signal from our benchmark validation
dataset. The signal is not prominently visible through the noise. The bottom
figures are the corresponding N2N filtered images. As can be seen the N2N
U-Net succeeds in filtering out the signal “chirp”, even thought its not
visible in the noisy images.  Figure 5: The training pipeline we use to
benchmark with the presence of the U-Net filter. The U-Net seeks to filter the
signal from noise potentially making the detection of gravitational wave
easier.

###  IV.2 U-Net Noise2Noise

Given two separate noisy realizations of the same signal, if the noise has
zero mean, one can train a U-Net using one noisy realization as input and the
other as target. Since noise in uncorrelated while the signal is correlated,
the network learns to predict the average of the noisy realizations, which is
the clean signal. This method of filtering a noisy signal is called
Noise2Noise [11] since it is based on training on a pair of noisy signals.

Given the fact that LIGO has two detectors (Hanford and Livingston) with
similar noise profiles, the Noise2Noise method seems at first like an ideal
approach to filter the noisy data. However, though Hanford and Livingston are
observing the same merger event, the time of arrival of the signal at Hanford
and Livingston can be as different as 10 ms. Thus although the low frequency
part of the signal will still be reasonably correlated in the time series, the
high frequency part, corresponding to the actual merger will not be correlated
and can be significantly out of phase. However the CWT transform trades
accuracy in frequency for accuracy in time, thus the appearance of the signal
in the CWT transform image of the Hanford and Livingston signal will overlap
reasonably well and can be used in training a Noise2Noise (N2N) filter.

The image of the CWT transform of the time series only carries information of
the amplitude of the signal and the phase is ignored. Thus our Noise2Noise
images do not contain noise that has zero mean and the filter will learn to
project the noise from one image to another thus learning the “average” of the
CWT of the noise in the process. To get rid of the noise we train two
Noise2Noise filters. The first one is trained purely on data with the presence
of a noisy signal and the other is trained purely on noise only data. Thus
applying both the filters and taking their difference should effectively
subtract the noise revealing the presence (or absence) of a signal. Since we
already have access to two noisy realizations of the same signal (Hanford and
Livingston) in our benchmark dataset, we do not need to generate a synthetic
dataset to train our U-Net N2N filter, we simply train using our entire
benchmark dataset (i.e. we don’t restrict ourselves to 1/4th of subset of the
data we use for benchmarking). To augment our training we randomly swap our
input and target to be the Hanford and Livingston signal. It is important to
note that while training a Noise2Noise U-Net, initially the metric of interest
(’Validation RMSE’) will converge within two epochs and then remain constant.
However that does not mean the Noise2Noise U-Net is no longer being trained.
In fact it is essential to keep training for a large number of epochs though
the training metric does not change. Here we train our Noise2Noise U-Net for
10 epochs. Below we demonstrate the filter effect of our U-Net N2N filter on
our benchmark validation dataset (see Fig. 6). As is the case with the N2C
filter, we then benchmark the effect of adding our filter to the training
pipeline as demonstrated in Fig 5. The results are presented in Table 1.
A few words about the Noise2Noise filtering approach are in order. The real
power of this approach lies in the fact that we did not need to generate a
mock dataset of binary mergers in order to train our U-Net filter. In this
sense a N2N filtering approach is completely data driven (apart from knowing
the binary presence or absence of signal in the dataset). Thus this approach
can be in principle be used on real LIGO data to detect signals for which a
template is not known. With more LIGO detectors coming online, our ability to
pair input and target in our U-Net grows combinatorially, presumably
increasing the effectiveness of the N2N filtering method (although further
study is required to quantify this statement).

Figure 6: In the figure above we show the effects of our N2N U-Net filter on
the validation data-set. The top figures (the input and target to the N2N
U-Net) shows the CWT image of a whitened noisy signal from our benchmark
validation dataset. The signal is not prominently visible through the noise.
Bottom figures are the corresponding N2N filtered images. As can be seen the
N2N U-Net succeeds in filtering out the signal “chirp”, even thought its not
visible in the noisy images.

##  V Results

We tabulate the results of our experiment to quantify the effect of our U-Net
filter 1.

Model | EfficientNet Size | Validation AUC | Validation Accuracy
---|---|---|---
EfficientNet | Last Layer Only | 0.7355 | 0.6788
EfficientNet | All Layers | 0.8344 | 0.7657
U-Net N2C + EfficientNet | Last Layer Only | 0.8206 | 0.7535
U-Net N2C + EfficientNet | All Layers | 0.8329 | 0.7577
U-Net N2N + EfficientNet | Last Layer Only | 0.8086 | 0.7497
U-Net N2N + EfficientNet | All Layers | 0.8209 | 0.7604
Table 1: Results of our experiments. The model refers to the pipeline we use
for training our classifier. The EfficientNet Size refers to the size of the
EfficientNet we use in training (i.e. train only the last layer vs. all of the
EfficientNet). We report the validation AUC and accuracy of our model.

First we train our benchmark dataset on the last fully connected layer of
EfficientNet. The worst performance comes from only training the last fully
connected layer of our EfficientNet. This is to be expected since this
includes only a few trainable parameters. Moreover the ImageNet weights we use
to intialize our EfficientNet and which we keep frozen during our training are
trained on everyday images which looks very different than the “chirp” of the
gravitational wave signal. Thus since the initial layers don’t do a good job
on extracting features relevant to the presence of a gravitational waves,
training the last layer only adds limited value to our binary classification.
This can be seem in the small validation AUC and accuracy in Table 1.

This is mostly remedied by applying a U-Net filter to our CWT images of the
gravitational wave signals. Both U-Net N2C and N2N manage to clean up the
signal well as well as evidenced by the leap in the AUC after training the
last layer of the EfficientNet on the filtered images. This is confirmation of
the results we found in Secs. IV.1, IV.2.

While the U-Net parameters are frozen during the training the EfficientNet, we
have already trained the U-Net parameters to extract a signal if present. In
priciple we can replace the EfficientNet that we use for classification after
U-Net filtering with a custom CNN architecture. This is because the U-Net
filtered images have a pretty simple representation if a signal is present.
However, to keep comparisons valid we stick to using EfficientNet for
classification.

While the above results are encouraging they are also expected since we are
effectively adding more trainable parameters in our classification problem. To
really see if our U-Net filters are performing beyond what’s expected we need
to ask if the U-Net can filter or amplify signal to a greater extent than what
the EfficientNet itself can do. Towards this end we unfreeze all the layers of
EfficientNet and train the CWT images of our benchmark dataset on the whole of
EfficientNet itself.

Training the whole of EfficientNet is training the network to extract features
relevant to the presence (or absence) of a GW signal in the noisy data. Thus
the EfficientNet is itself performing the job of “filtering” the signal.
However it’s unlikely it’s doing so in a physics interpretable way the U-Nets
filter signal.

Training all the layers of the EfficientNet gives the best score in our metric
of choice i.e validation AUC. Adding either a U-Net N2C or N2N filter to
preprocess the images before they are used to train all of the EfficientNet
seems to decrease the validation AUC as well as accuracy slightly. Although
the differences between using and not using a U-Net to filter our noisy CWT
images is very small (less than 1/5th of the percentage points), this result
is surprising if not a little disappointing since we had hoped the U-Nets
would enhance the presence of signals the EfficientNet cannot detect by
itself.

To further diagnose the issue with our U-Net filtering method we try to
ascertain if the U-Nets are missing very weak signals (that the EfficientNet
architecture can in fact detect by itself) i.e. false negatives, or our U-Net
filtering method is hallucinating a signal where none is present i.e. false
positives. Looking at the confusion matrices (using a threshold of 0.5) shown
in Fig. 7 it is evident that using a N2C filter before using the EfficientNet
increases the false positives while reducing the false negatives 444We choose
U-Net N2C for plotting the confusion matrix since it performs better.. Thus it
seems that our method of U-Net filtering introduces additional false positives
(while reducing false negatives) despite our efforts to reduce them by making
sure that the filtered “signals” coincide (which they wouldn’t necessarily if
the filter hallucinated a signal). In Sec. VI we will briefly discuss
strategies to mitigate false positives as part of our future work.

As can be seen the U-Net N2N model performs the worst. Some of this is
expected since we train the Noise2Noise U-Net on noisy data. However the score
of U-Net N2N filter is limited by the number of training samples and can be
improved by training on a larger dataset. Moreover the Noise2Noise strategy is
expected to give better results if the phase information of the CWT transform
is included in our training. We discuss this further in Sec. VI

Figure 7: The confusion matrix using all of the EfficientNet and using a N2C
filter before using the EfficientNet. Clockwise the confusion matrix reports
fractions of true negative, false positives, true positive and false
negatives. As can be seen using the N2C filter increases false positives while
reducing false negatives.

##  VI Conclusions and Future Work

In this work we have shown that U-Nets can be used to filter a clean
gravitational wave signal from noisy data. We explored two ways of
constructing a U-Net. The first method relied on using a Noise2Clean approach
where we used noisy data as input and clean data as target and trained the
U-Net to filter out the noise. For this method we had to create synthetic
gravitational wave signals to train our network and hoped it generalized well
to the benchmark dataset we used (which it did to a large extent). Our second
approach (Noise2Noise) was to use to separate noisy realizations of the same
signal. This was easily accomplished since LIGO has two detectors with a
similar noise PSD (i.e. Hanford and Livingston). Although the signals are not
coincident in time in the two detectors, their CWT transformed images are
coincident enough for us to use this approach. We used two U-Nets, one trained
on positive samples only and the other on negative samples only. In this
approach, there was no need generate mock dataset, instead we used our
benchmark dataset from the Kaggle G2Net competition. Since we did not need
access to clean timeseries the N2N approach is more data-driven than the N2C
approach.

We found that the U-Net succeeded in filtering signal from the noise. However
filtering the signal with U-Nets produces too many false positives. Additional
detectors would certainly help with the problem of false positives.
Additionally including the phase information should also help in establishing
a coincidence between the two signals. If including the phase information we
are able to reconstruct all or part of the merger in one detector, then we can
use that waveform to perform matched filtering on the other detector(s) to
make sure the signal appears in other detectors as well. These and other
standard strategies to mitigate overfitting and hence false positives will be
left for upcoming work.

We also propose the possibility of using the U-Nets as a trigger for detecting
the presence of gravitational waves (since once trained, inference using the
U-Nets will be quick). If the waveform time-series can be constructed phase
information, we can use this to provide an initial guess for the matched
filter thus speeding up the matched filtering procedure. Further investigation
along these lines is needed and we leave that for future work.

The fundamental shortcoming of our approach was that we discarded the phase
information of the signal and only trained using the amplitude of the complex
CWT transform. Keeping the phase information will allow us to reconstruct the
time series of the clean signal by performing an inverse CWT transform. Doing
this will allow us to define a loss function by comparing the time series that
the U-Net outputs with the expected one and might lead to better filtering. In
fact such an approach was used for speech enhancement in noisy audio [12] by
implementing a Deep Complex U-Net [13]. We leave this implementation for our
upcoming work.

The U-Net N2N method can be used to extract a signal when no template is
available. Thus it can be used to detect more exotic signals than mergers. If
the binary presence of a signal is known i.e. the time series is not
consistent with noise only in all of LIGO detectors, then such a dataset can
be used to find the new exotic signal. Thus we expect an approach that uses
anomaly detection (to detect the binary presence of a signal) combined with a
N2N U-Net can reveal exotic sources if they are present. On the other hand, if
a signal is always present such as a continuous gravitational wave signal, the
U-Net N2N approach can be used to enhance such a signal.

The U-Net N2N approach can also be used in other astrophysical data where a
signal with unknown shape is known to be present but buried under noise. An
example would be pulsar observations where the pulse shape and hence the
position (exact time of arrival) is not known. One can use the Noise2Noise
approach on pulses within a window of observations where the pulsar properties
are not expected to change significantly. Whether this approach provides
benefit over the standard stacking/folding procedure to will need to be
investigated.

##  VII Acknowledgements

The work of A.G. is supported by the U.S. Department of Energy under grant No.
DE–SC0007914. We would like to thank the Aspen Center for Physics where part
of this work was completed. We would also like to thank Ben Michel, Jared
Evans and David Shih for helpful discussions. We would also like to thank
Brian Batell, Matthew Low and Jared Evans for providing comments on the draft.

##  Appendix A Whitening

Typically the power spectrum of a time series, i.e. the contribution of each
frequency to a given signal typically depends on frequency. For our
gravitational wave time series the power spectrum density (PSD) of the noise
is dominated at low frequencies by seismic noise and at high frequencies by
the shot noise of the laser. It also includes certain dominant peak
frequencies. The average PSD of the noise only time series are plotted in Fig.
8

Whitening as the name suggests is a process that removing the frequency
dependence of the PSD (white noise has no frequency dependence). In order to
do this we take the FFT of the time series, divide by the PSD and take the
inverse Fourier transform. This gives us the whitened time series.

Figure 8: The averaged PSD of the normalized time series of noise only data.

##  Appendix B U-Net Architecture

The U-Net architecture consists of a contractive (downsampling) path followed
by an expansive (upsampling) path (See Fig.). The basic building block of the
contractive path is two successive applications of a convolution consisting of
$n$ filters and a rectified linear unit (ReLu) activation. $n$ is chosen as
$n_{0}\times 2^{l-1}$, where $l$ is the layer number. We choose $n_{0}=32$. We
choose padding as ‘same’ for the convolutions. In a standard U-Net this is
usually followed by a MaxPooling application. However we replace this with
another convolution and a stride of 2. This contractive block is then applied
in sequence to the input image for a prescribed number of times (we choose 6).
In the final application of the contractive path, we do not apply the final
convolutional pooling with stride of two. This creates a the contractive path
of the U-Net. At each level of the contractive path (each application of the
building block of the contractive path), spatial information is sacrificed in
favor of features.

This is followed by the expansive path, the building block of which is
deconvolution (with $n=n_{0}\times 2^{l-1}$ filters, stride of two, and kernel
size of three) followed by concatenation with a block from the at the same
level. This concatenation is done through skip connections between the
contractive and expansive path. Thus the upsampling block combines the spatial
information from a given level with high-level features from the preceeding
deeper level. We then follow with two more convolutions (with $n=n_{0}\times
2^{l-1}$ filters, stride of two, and kernel size of three). This upsampling is
repeated for the same number of layers. Finally we perform a convolution with
$n=1$ to recover the same dimension as the input. See Fig. 9 for an
illustration of the U-Net.

We then train this U-Net based on our choice of inputs and target i.e. N2C or
N2N.

Figure 9: The U-Net architecture. We use a 6 layer U-Net, but have shown only
3 layers for purposes of illustration

## References

  * Aasi  _et al._ [2015] J. Aasi _et al._ (LIGO Scientific), Class. Quant. Grav. 32, 074001 (2015), arXiv:1411.4547 [gr-qc] .
  * Acernese  _et al._ [2015] F. Acernese _et al._ (VIRGO), Class. Quant. Grav. 32, 024001 (2015), arXiv:1408.3978 [gr-qc] .
  * Schmitz [2021] K. Schmitz, JHEP 01, 097 (2021), arXiv:2002.04615 [hep-ph] .
  * Messenger  _et al._ [2021] C. Messenger _et al._ , (2021).
  * George and Huerta [2018] D. George and E. A. Huerta, Phys. Lett. B 778, 64 (2018), arXiv:1711.03121 [gr-qc] .
  * Yan  _et al._ [2022] J. Yan, M. Avagyan, R. E. Colgan, D. Veske, I. Bartos, J. Wright, Z. Márka, and S. Márka, Phys. Rev. D 105, 043006 (2022), arXiv:2104.03961 [astro-ph.IM] .
  * Gebhard  _et al._ [2019] T. D. Gebhard, N. Kilbertus, I. Harry, and B. Schölkopf, Phys. Rev. D 100, 063015 (2019), arXiv:1904.08693 [astro-ph.IM] .
  * Usman  _et al._ [2016] S. A. Usman _et al._ , Class. Quant. Grav. 33, 215004 (2016), arXiv:1508.02357 [gr-qc] .
  * Tan and Le [2019] M. Tan and Q. Le, in  _International conference on machine learning_ (PMLR, 2019) pp. 6105–6114.
  * Ronneberger  _et al._ [2015] O. Ronneberger, P. Fischer, and T. Brox, in  _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ (Springer, 2015) pp. 234–241.
  * Lehtinen  _et al._ [2018] J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala, and T. Aila, arXiv preprint arXiv:1803.04189 (2018).
  * Kashyap  _et al._ [2021] M. M. Kashyap, A. Tambwekar, K. Manohara, and S. Natarajan, arXiv preprint arXiv:2104.03838 (2021).
  * Choi  _et al._ [2018] H.-S. Choi, J.-H. Kim, J. Huh, A. Kim, J.-W. Ha, and K. Lee, in  _International Conference on Learning Representations_ (2018).


# Audio to score matching by combining phonetic and duration information

###### Abstract

We approach the singing phrase audio to score matching problem by using
phonetic and duration information – with a focus on studying the jingju a
cappella singing case. We argue that, due to the existence of a basic melodic
contour for each mode in jingju music, only using melodic information (such as
pitch contour) will result in an ambiguous matching. This leads us to propose
a matching approach based on the use of phonetic and duration information.
Phonetic information is extracted with an acoustic model shaped with our data,
and duration information is considered with the Hidden Markov Models (HMMs)
variants we investigate. We build a model for each lyric path in our scores
and we achieve the matching by ranking the posterior probabilities of the
decoded most likely state sequences. Three acoustic models are investigated:
(i) convolutional neural networks (CNNs), (ii) deep neural networks (DNNs) and
(iii) Gaussian mixture models (GMMs). Also, two duration models are compared:
(i) hidden semi-Markov model (HSMM) and (ii) post-processor duration model.
Results show that CNNs perform better in our (small) audio dataset and also
that HSMM outperforms the post-processor duration model.

##  1 Introduction

Figure 1: System design framework of the entire research project. The modules
with bold border lines are addressed in this paper.

The ultimate goal of our research project is to automatically evaluate the
jingju a cappella singing of a student in the scenario of jingju singing
education – see Figure 1. Jingju, a traditional Chinese performing art also
known as Peking or Beijing opera, is extremely demanding in the clear
pronunciation and accurate intonation of each syllabic or phonetic singing
unit. To this end, during the initial learning stages, students are required
to completely imitate tutor’s singing. Therefore, the automatic jingju singing
evaluation system we envision is based on this training principle and measures
the intonation and pronunciation similarities between the student’s and the
tutor’s singings. Before measuring similarities, the singing phrase should be
automatically segmented into syllabic or phonetic units in order to capture
the temporal details. Jingju music scores, which contain the phonetic and
duration information for each singing syllable, will be beneficial for this
segmentation. In the application scenario, the score of a query audio could be
selected from the database by the user itself. However, to avoid manual
intervention and improve the user experience, we tackle the problem of
automatically finding the corresponding music score for a given query audio
(bold in Figure 1). Note that achieving successful methods for audio to score
matching might be beneficial for several music informatics research (MIR)
tasks, such as: score-informed automatic syllable/phoneme segmentation [5] or
score-informed source separation [14]. The objective of this research task is
to find the corresponding score for a given singing audio query. We restrict
this research to the “matching” scope by pre-segmenting both the singing
audios and the music scores into the phrase units.

Xipi and erhuang are the main modes in jingju music. Each has two basic
melodic contours – an opening phrase and a closing phrase. Each basic melodic
contour is constructed upon characteristic pitch progressions for each mode
[22]. Therefore, singing phrases from different arias sharing the same mode
are likely to have a similar melodic contour. Figure 2 shows an example of
this fact.

Figure 2: An example of different phrases having a similar Xipi melodic
contour in our score dataset. The lengths of these contours are normalized to
a fixed sample size

However, melodic information tends to be intuitively used for such matching
tasks. For example in Query-by-Singing/Humming (QBSH) [15], melodic
similarities can be obtained by comparing the distance between the F0 contour
of the query audio and those synthesized from the candidate scores. Then, the
best matched music score can be retrieved by selecting the most similar
melody. But note that using this approach for jingju music would bring
matching errors since the melodic contours of the same mode are similar in
this sense. In this case, it is more appropriate to use another notion of
similarity. We propose using the lyrics since the stories narrated in
different jingju arias are distinctive and lyrics tend to change through
different jingju arias – even when they share the same mode. Therefore,
phonetic information might be useful to identify a similar score given a query
audio.

QBSH is the most related research task to our study, which retrieves a song by
singing a portion of itself. Most of the studies use melody information as the
only cue. The typical process of such systems was introduced by Molina et al.
[15]: firstly, the F0 contour and/or a note-level transcription for a given
singing query are extracted; and then, a set of candidate songs are retrieved
from a large database using a melodic matcher module. The most successful QBSH
system, which obtained the best results in MIREX 2016 contest, is based on the
method of multiple similarity measurements fusion [21]. This system proposed a
melodic matcher which combines several similarities that are note-based and
frame-based. The authors claim that the fusion mechanism improves the query
performance because no similarity measurement is perfect. Therefore,
information sources that are complementary to each other might be beneficial
for this approach. Very few studies have explored the capability of the
phonetic information for QBSH. Guo et al. [8] and Wang et al. [20] both used a
lyric recognizer based on Hidden Markov Models (HMMs). Their recognition
networks111The topology of the HMM is defined by the recognition network. were
constructed with the phonetic information from the query candidates database.
They used frame-based MFCCs to create the acoustic models with GMMs. Then, the
Viterbi algorithm was executed over the recognition networks to either obtain
the most likely phonetic state sequence (for Wang et al. [20]) or the
posterior probability of each possible decoding path (for Guo et al. [8]). The
final score of a query candidate is either based on semantic similarity [20]
or based on the posterior probability of its corresponding lyrics [8].

Another research task related to our study is singing keyword spotting. The
main goal of this task is to search for one or more keywords in a singing
query. The system proposed by Kruspe [12] searches for a specific singing
keyword on the resulting phoneme observations. A keyword-filler HMMs is
employed for this purpose. She used two phoneme duration models: the HSMM and
the post-processor duration model.

Finally, both phonetic and duration information extracted from the score have
been extensively used in alignment-related tasks, such as audio-to-score
alignment and audio-to-lyrics alignment. For example, Gong et al. [4]
construct a left-to-right HSMM using phonetic and duration information. Or
Dzhambazov et al. [3] use a similar approach for aligning polyphonic audio.
Analogously, the proposed approach explores the use of both phonetic and
duration information (available in scores) to tackle the matching ambiguity
problem existing in jingju music.

The remainder of this paper is organized as follows: the used dataset is
introduced in section 2, section 3 explains the modules of the proposed
approach – detailing how to incorporate phonetic and duration information.
Experiments and results are reported in section 4, and section 5 concludes and
points out future work.

##  2 Dataset

The jingju a cappella singing dataset is composed of two overlapping parts:
(i) audio and (ii) score datasets.

The audio dataset [1] used for this study consists of two role-types singing:
dan (young woman) and laosheng (old man). The dan part of this dataset has 42
recordings sung by 7 singers and the laosheng part has 23 recordings sung by 7
laosheng singers. The boundary annotations of the audio dataset have been done
in Praat format (textgrid) considering a hierarchy of three levels: phrase,
syllable and phoneme – using Chinese characters, pinyin notations and X-SAMPA
notations, respectively. 32 phoneme classes are used in the phoneme-level
annotation. Two Mandarin native speakers and a jingju musicologist have been
devoted to this annotating work. Annotations and more detailed information can
be found online222http://doi.org/10.5281/zenodo.344932. Some statistics about
the dataset are reported in Table 1. The average phrase, syllable and voiced
phoneme length of dan singing are ostensibly greater than those of laosheng
singing (bold numbers in Table 1), which might indicate that dan singing tends
to have more pitch variation and ornamentation – as we could observe
empirically by listening to the data.

Table 1: Detailed information of the jingju a cappella singing audio dataset: dan (top), laosheng (bottom). | Num. | Avg. len (s) | Std. len (s)
---|---|---|---
Phrases | 325 | 16.42 | 14.11
Syllables | 2933 | 1.58 | 2.82
Voiced phonemes | 7198 | 0.61 | 0.97
Unvoiced phonemes | 2014 | 0.10 | 0.67
Phrases | 247 | 9.47 | 8.14
Syllables | 2289 | 0.88 | 1.48
Voiced phonemes | 4948 | 0.39 | 0.78
Unvoiced phonemes | 1454 | 0.07 | 0.05

The audio dataset, along with their boundary annotations, is split into three
parts: training set, development (dev) set and test set. We define the
training set to be the non-overlapping part with the score dataset, see Figure
3. The training set will be used for calculating the phonetic duration
(duration information) and training acoustic models (phonetic information).
After taking the training set out, we define the development set to be the
half of the remaining phrases in the audio dataset (randomly selected) – it
will be used for parameters optimization. The test set consists on the
remaining phrases of the audio dataset – it will be used for testing the
acoustic models performance and the matching performance.

Figure 3: The intersection between the audio and the score datasets. The
partition of the audio dataset.

On the other hand, the score dataset contains 435 dan phrases and 481 laosheng
phrases. The scores have been typed in stave notation (including lyrics in
Chinese characters) using MuseScore from different printed sources in jianpu
notation. Since tempo is usually not clearly noted in the printed score, we do
not include this information in the dataset. The relative syllabic durations
are indicated by the note durations corresponding to the lyrics, which will be
used to calculate the phonetic duration (duration information) and the
matching network. The whole score dataset will be used as candidates for
testing the matching performance and for parameter optimization.

##  3 Approach

The proposed approach aims to match the query audio to its score by using
phonetic and duration information. During the training process (red boxes in
Figure 4): the acoustic models of each phoneme are shaped by using the audio
training set and its phonetic boundary annotations; the score dataset is used
to construct a matching network; and phoneme duration distributions are
estimated by using both audio training set and scores. During the matching
process (green boxes in Figure 4): two duration models –HSMM and post-
processor– are explored for the Viterbi decoding step. Finally, the best
matched phrase is found by ranking the decoded state sequence probabilities.

Figure 4: Diagram of the proposed approach.

###  3.1 Acoustic models

Here presented acoustic models aim to represent the relationship between an
audio signal and the 32 phoneme classes present in our dataset. The output of
these models yield probability scores for each phoneme class.

The most popular way to approach acoustic modeling is by using GMMs and MFCCs
features [8, 20]. For that reason, we set as baseline a 40-component GMM with
the following input vector: 13 MFCCs, their deltas and delta-deltas. Moreover,
DNNs have been found very useful for acoustic modeling [9, 13]. Therefore, we
propose an additional baseline: a DNN with 2 hidden layers followed by the
32-way softmax output layer – the input is set to be a log-mel spectrogram.

However, DNNs are very prone to over-fitting and the available dataset is
relatively small. For that reason we propose using CNNs since these are more
robust against over-fitting – note that CNNs allow parameter sharing.
Additionally, Pons et al. [17] have successfully used spectrograms-based CNNs
for learning music timbre representations from small datasets. Given that
timbre is an important feature for acoustic modeling, we propose using the
same architecture: a single convolutional layer with filters of various sizes
[17, 16]. The input is set to be a log-mel spectrogram. We use $128$ filters
of sizes $50{\times}1$ and $70{\times}1$, $64$ filters of sizes $50{\times}5$
and $70{\times}5$, and $32$ filters of sizes $50{\times}10$ and $70{\times}10$
– where the first and second numbers denote the frequential and temporal size
of the filter, respectively. A max-pool layer of $2{\times}N^{\prime}$ is
followed by a 32-way softmax output layer with 30% dropout – where
$N^{\prime}$ denotes the temporal dimension of the resulting feature map.
$2{\times}N^{\prime}$ max-pool layer was chosen to achieve time-invariant
representations while keeping the frequency resolution. And same padding is
used to preserve the dimensions of the feature maps so that these are
concatenable. Filter shapes are designed so that filters can capture the
relevant time-frequency contexts for learning timbre representations –
according to the design strategy proposed by Pons et al. [17]

Log-mel spectrograms are of size $80{\times}21$ – the network takes a decision
for a frame given its context: $\pm$10ms, 21 frames in total. Activation
functions are ELUs [2] for all deep learning models and these are optimized
with stochastic gradient descent (batch size: 64), using ADAM [11] and early
stopping – when validation loss (categorical cross-entropy) does not decrease
after 10 epochs.

Spectrograms are computed from audio recordings sampled at 44.1 kHz. STFT is
performed using a window length of 25ms (2048 samples with zero-padding) with
a hop size of 10ms. The 80 log-mel bands energies are calculated on
frequencies between 0Hz and 11000Hz and these are standardized to have zero
mean and unit variance.

The acoustic models are trained separately for each role-type and their
performance is reported in section 4.2.

###  3.2 Matching network

The matching network defines the topology of the hidden Markov model. By using
each candidate phrase in the score dataset as an isolated unit, isolated-
phrase matching networks can be constructed. Figure 5 shows the structure of
this matching network, which has $K=916$ lyric paths.

Figure 5: The structure of the $K$ paths isolated-phrase matching network.
Path 3 shows an example of the left-to-right state chain structure of an
individual lyric path.

The matching network uses HMMs or HSMMs, depending on how the internal
duration is modeled. Each path is a left-to-right state chain which represents
the phoneme transcription of its lyrics. In order to construct the lyric path,
pinyin lyrics are segmented into phonetic units and transcribed into X-SAMPA
notations by using a predefined dictionary. For example, a path which has the
lyrics yan jian de hong ri in pinyin is a chain consisting of 12 states: j,
En, c, j, En, c, 7, x, UN, N, r\’, 1 in X-SAMPA notation. When the decoding
process has finished, each lyric path can get a posterior probability which
will be used as the similarity measure between the query phrase and the
candidate phrase.

###  3.3 Phonetic duration distributions

Phonetic duration information comes from two sources: the boundary annotations
of audio training dataset and the score dataset. The phonetic duration is not
directly indicated in the score. However, it is indispensable for modeling the
phonetic duration distribution for each state in the matching network. The
syllable, of which duration can be deduced by the corresponding note(s), is
used to restrict the durations of the phonemes.

Figure 6: Flowchart example of estimating the phonetic durations of a
syllable.

In the following, we propose a method for estimating the absolute phonetic
duration given: (i) the score, and (ii) the phonemes duration histograms
computed from the audio dataset annotations. First, we omit silence parts in
the query audio (with a simple voice activity detection method [19]) and also
in the score by removing the rest notes. Second, we compute the duration
histogram and its duration centroid for each phoneme class – by aggregating
the phonetic durations indicated in the boundary annotations of the audio
training dataset. Then, we segment each syllabic duration in the score dataset
into phonetic durations according to the proportion of their duration
centroids. Finally, as the scores do not contain tempo, we normalize the
phonetic durations of each phrase such that their summation is equal to the
duration of the query audio. See Figure 6 for an equivalent graphic
explanation. In Figure 6, the centroid durations of these three phonemes are:
0.46s, 0.9s and 0.1s, summing: 1.46s – alternatively, these can be expressed
as a proportion of 1.46s: 0.32, 0.62 and 0.06. With these proportions and the
absolute syllable duration (2s), we can compute the absolute phoneme
durations: 0.32$\cdot$2s = 0.64s, 0.62$\cdot$2s=1.24s and 0.06$\cdot$2s=0.12s.

The phonetic duration distribution needs to be calculated for each state in
the matching network in order to incorporate the a priori phonetic duration
information. We model it by Gaussian distributions:

$\mathcal{N}(x;\mu_{l},\sigma_{l}^{2})=\frac{1}{\sqrt{2\pi}\sigma_{l}}\exp\left(-\frac{(x-\mu_{l})^{2}}{2\sigma_{l}^{2}}\right).$
(1)

where $\mu_{l}$ is the duration of the phoneme $l$ deduced by the above method
and the standard deviation $\sigma_{l}$ is proportional to $\mu_{l}$:
$\sigma_{l}=\gamma\mu_{l}$. The proportionality constant $\gamma$ will be
optimized in section 4.3 for each role-type.

###  3.4 Duration modeling

Standard Markovian state do not impose explicitly duration distribution,
instead, imposing an implicit state occupancy distribution which corresponds
to a “1-shifted” geometric distribution [6]:

$d_{j}(u)=(1-\tilde{p}_{jj})\tilde{p}_{jj}^{u-1}$ (2)

where $u$ denotes the occupancy or sojourn time in a Markovian state $j$ and
$\tilde{p}_{jj}$ denotes the self-transition probability of the state $j$.
Because of the implicity of the Markovian state occupancy, the phonetic
duration distribution introduced in section 3.3 can not be imposed. Kruspe
[12] presents two duration modeling techniques for HMMs: Hidden semi-markov
model (HSMM) and post-processor duration model.

####  3.4.1 Hidden semi-markov model

Guédon [6] defined a semi-Markov chain $S_{t}$ with finite state space
$0,...,J-1$ by the following parameters:

  * -

initial probabilities $\pi_{j}=P(S_{0}=j)$ with $\sum_{j}\pi_{j}=1$

  * -

transition probability of semi-Markovian state $j$: for each $k\neq
j,p_{jk}=P(S_{t+1}=k|S_{t+1}\neq j,S_{t}=j)$ with $\sum_{k\neq j}p_{jk}=1$ and
$p_{jj}=0$

An explicit occupancy distribution is attached to each semi-Markovian state:

$\displaystyle d_{j}(u)=P($ $\displaystyle S_{t+u+1}\neq
j,S_{t+u-v}=j,v=0,...,u-2$ (3) $\displaystyle|S_{t+1}=j,S_{t}\neq
j),u=1,...,M_{j}$

where $M_{j}$ denotes the upper bound to the time spent in state $j$.
$d_{j}(u)$ defines the conditional probability of leaving state $j$ at time
$t+u+1$ and entering state $j$ at time $t+1$.

To apply HSMMs to the matching network, we first use the matching network as
the HSMMs topology, thus the state occupancy distribution is set to its
corresponding phonetic duration distribution. Then the probabilities of each
left-to-right state transition are set to 1 because all self-transition
probabilities in HSMMs are 0. The goal is to find the most likely sequence of
hidden states for each lyric path and collect its posterior probability. The
Viterbi algorithm meets this exact goal and its complete implementation is
provided in [7].

####  3.4.2 Post-processor duration model

The post-processor duration model was first introduced by Juang et al. [10].
It was then experimentally proved in Kruspe’s paper [12] that this duration
model works better than HSMMs for the keyword spotting task in English pop
singing voice. The post-processor duration model uses the original HMMs
Viterbi algorithm – therefore, during the decoding process no explicit
occupancy duration distribution is imposed.

The log posterior probability of the decoded most likely state sequence is
augmented by the log duration probabilities:

$\log\hat{f}=\log
f+\alpha\sum\limits_{j=1}^{N}\mathcal{N}(u_{j};\mu_{j},\sigma_{j}^{2})$ (4)

where $f$ is the HMMs posterior probability, $\alpha$ is a weighting factor
which will be optimized in section 4.3, $j=1,...,N$ is the decoded state
number in the most likely state sequence, and
$\mathcal{N}(u_{j};\mu_{j},\sigma_{j}^{2})$ is the occupancy probability of
being in state $j$ for the occupancy $u_{j}$.

##  4 Experiments and results

###  4.1 Performance metrics

Two experiments333
Code:https://github.com/ronggong/jingjuSingingPhraseMatching/tree/v0.1.0 are
performed: the first is to evaluate the performance of the acoustic models,
and the second is to evaluate the proposed matching approaches. For the first
task, we use one simple evaluation metric: the overall classification accuracy
which is defined as the fraction of instances that are correctly classified.
For the second task, our goal is to evaluate the ability of matching the
ground-truth phrase in the score dataset to the query one, which is almost
identical to the goal of a QBSH system: ”finding the ground-truth song in a
song database from a given singing/humming query”. Therefore, we borrow the
standard performance metrics used in QBSH task to evaluate our approaches:
Top-M hit and Mean Reciprocal Rank (MRR) [8]. The Top-M hit rate is the
proportion of queries for which $r_{i}\leq M$, where $r_{i}$ denotes the rank
of the ground-truth score phrase. MRR is the average of the reciprocal ranks
across all queries, $n$ is the number of queries, and ${rank}_{i}$ is the
posterior probability rank of the ground-truth phrase corresponding to the
i-th query.

$MRR={\frac{1}{n}}\sum\limits_{i=1}^{n}{\frac{1}{rank_{i}}}$ (5)

###  4.2 Acoustic models

CNN, DNN and GMM acoustic models yield probability scores for each phoneme
class. In order to evaluate the classification accuracy, we choose the phoneme
class with the maximum probability score as the prediction. Table 2 reports
the performance of CNN, DNN and GMM acoustic models evaluated on the test set.

Table 2: Overall classification accuracies of CNN and baseline acoustical models for dan and laosheng datasets. |  dan(#parameters) |  laosheng(#parameters)
---|---|---
CNNs | 0.484(222k) | 0.432(222k)
DNNs | 0.284(481k) | 0.282(430k)
GMMs | 0.290(-) | 0.322(-)

The relatively low classification accuracies for all three models show that
modeling the phonetic characteristics for jingju singing voice is a
challenging problem. Our best results are achieved with CNNs – and GMMs
perform better than DNNs. Interestingly, these results contrast with the
literature where Hinton et al. [9] describe that DNNs acoustic models largely
outperform GMMs for automatic speech recognition, and Maas et al. [13] showed
that CNNs perform worse than DNNs for building speech acoustic models. First,
we argue that in our case DNNs perform worse than GMMs and CNNs because a
small amount of training data is available. DNNs require a lot of training
data to achieve good performance and note that large amounts of training data
are typically not available for most MIR tasks. And second, note the CNNs used
here are specifically designed to efficiently learn timbre representations
[17] while Maas et al. [13] used small squared filters, which proved
successful in computer vision tasks. These results show that using CNN
architectures designed for the task at hand is specially beneficial in small
data scenarios. A CNN model is used in the following experiments

###  4.3 Parameters optimization

The parameters which need to be optimized for dan and laosheng role-types are:
the weighting factor $\alpha$ for the post-processor duration model, and the
proportionality constant $\gamma$ for both models: HSMMs and post-processor
duration model. Table 3 reports the optimal values we obtained by doing grid
search on the development set – MRR metric was maximized.

Table 3: Parameters to be optimized, search bounds and resulting optimal values (dan / laosheng). | Search | Optimal
---|---|---
Parameters | bounds | values
$\alpha$ |  $[0.25,2]$ with step 0.25 | 1.0 / 1.0
$\gamma$ HSMMs |  $[0.1,2]$ with step 0.1 | 0.1 / 0.1
$\gamma$ post-processor |  $[0.1,2]$ with step 0.1 | 0.7 / 1.5

###  4.4 Duration modeling

To highlight the advantage of using duration modeling methods for audio to
score matching, a standard HMM without explicitly imposing the occupancy
distribution is used as a baseline. Results in Figure 7 show that its
performance is inferior than the HSMM duration model.

Figure 7: Phrase matching performance of HSMM and post-processor duration
model with CNN acoustic model: dan (top), laosheng (bottom).

One can also observe in Figure 7 that HSMM performs the best, improving the
baseline MRR metric performance by 13.2% for dan role-type and 15.1% for
laosheng role-type. This means that HSMMs explicit duration modeling can help
achieve a better audio to score matching by using phonetic information.

The post-processor duration model does not significantly improve the baseline
performance. This result contrasts with the literature, where the post-
processor duration model worked better than HSMMs for singing voice keyword
spotting [12]. This inconsistency might result from (i) the length difference
of the matching unit (singing-words vs. singing-phrases), and (ii) the large
standard deviation of the jingju singing phonemes length. First, in Kruspe’s
work [12], the matching unit is the singing keyword – which usually contains
less phonemes than a singing phrase (as in our case). And second, the vowel
length standard deviation of the a cappella dataset used by Kruspe [12]
(around 0.3s) is much short than in our dataset (dan: 0.97s, laosheng: 0.78s)
– denoting less vowel duration variance than in our study case. Moreover, a
significant deficiency of the post-processor duration model is that it does
not provide the most likely state sequence by internally considering the
durations, but it computes a new weighted likelihood given the obtained
sequence [12]. If the most likely state sequence is decoded badly, it can’t be
restored by the post-processor duration model.

##  5 Conclusions and future work

In this paper we presented an audio to score matching approach that uses
phonetic and duration information.

We explored two duration models: HSMM and post-processor duration model. HSMMs
achieved better results than the post-processor duration model – probably due
to (i) the matching units length, (ii) the large standard deviation of the
considered phonemes, and (iii) because for the post-processor duration model
it is hard to recover a decoding mistake. Moreover HSMMs achieved a better
matching performance than the baseline-HMMs approach, which only took into
account phonetic information, denoting the utility of using duration
information.

We also compared CNN, DNN and GMM acoustic models, and CNNs have shown to be
superior in our small singing voice audio dataset. The used CNN architecture
was specifically designed to learn timbral representations efficiently [17] –
this being the key factor for enabling CNNs (a deep learning method requiring
large amounts of data) to perform so well in such a small dataset.

There are many possibilities to improve our approach. It has been shown in the
speech research field that LSTM RNNs achieved the best acoustic modeling
performance [18]. However, this method requires a large training dataset in
order prevent from over-fitting. Another possibility to improve our acoustic
model is to go deeper with the current single-layer CNN architecture, but this
will also require more training data. We plan to collect more jingju a
cappella singing recordings and perform data augmentation to leverage the
capability of the acoustic models. Furthermore, in order to take advantage of
the melodic information existing in both audio and score datasets, we also
plan to investigate methods which can fuse melodic, phonetic and duration
information.

##  6 Acknowledgements

We are grateful for the GPUs donated by NVidia. This work is partially
supported by the Maria de Maeztu Programme (MDM-2015-0502) and by the European
Research Council under the European Union’s Seventh Framework Program, as part
of the CompMusic project (ERC grant agreement 267583).

## References

  * [1] D. A. A. Black, M. Li, and M. Tian.  Automatic Identification of Emotional Cues in Chinese Opera Singing.  In ICMPC, Seoul, South Korea, August 2014.
  * [2] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus).  arXiv:1511.07289, 2015.
  * [3] G. B. Dzhambazov and X. Serra.  Modeling of phoneme durations for alignment between polyphonic audio and lyrics.  In SMC, Maynooth, Ireland, 2015.
  * [4] R. Gong, P. Cuvillier, N. Obin, and A. Cont.  Real-Time Audio-to-Score Alignment of Singing Voice Based on Melody and Lyric Information.  In Interspeech, Dresden, Germany, September 2015.
  * [5] Rong Gong, Nicolas Obin, Georgi Dzhambazov, and Xavier Serra.  Score-informed syllable segmentation for jingju a cappella singing voice with mel-frequency intensity profiles.  In International Workshop on Folk Music Analysis, Málaga, Spain, June 2017.
  * [6] Y. Guédon.  Hidden hybrid markov/semi-markov chains.  Computational Statistics & Data Analysis, 49(3):663 – 688, 2005\.
  * [7] Yann Guédon.  Exploring the state sequence space for hidden markov and semi-markov chains.  Computational Statistics & Data Analysis, 51(5):2379–2409, 2007\.
  * [8] Z. Guo, Q. Wang, G. Liu, J. Guo, and Y. Lu.  A music retrieval system using melody and lyric.  In 2012 IEEE International Conference on Multimedia and Expo Workshops, pages 343–348, July 2012.
  * [9] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, et al.  Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups.  IEEE Signal Processing Magazine, 29(6):82–97, 2012.
  * [10] B. Juang, L. Rabiner, S. Levinson, and M. Sondhi.  Recent developments in the application of hidden markov models to speaker-independent isolated word recognition.  In ICASSP, volume 10, pages 9–12, Apr 1985.
  * [11] D. Kingma and J. B. Adam.  Adam: A method for stochastic optimization.  arXiv:1412.6980, 2014.
  * [12] A. M. Kruspe.  Keyword spotting in singing with duration-modeled hmms.  In EUSIPCO, pages 1291–1295, August 2015.
  * [13] Andrew L Maas, Peng Qi, Ziang Xie, Awni Y Hannun, Christopher T Lengerich, Daniel Jurafsky, and Andrew Y Ng.  Building dnn acoustic models for large vocabulary speech recognition.  Computer Speech & Language, 41:195–213, 2017.
  * [14] Marius Miron, Julio J. Carabias-Orti, Juan J. Bosch, Emilia Gómez, and Jordi Janer.  Score-informed source separation for multichannel orchestral recordings.  JECE, 2016, December 2016.
  * [15] E. Molina, L. J. Tardón, I. Barbancho, and A. M. Barbancho.  The importance of f0 tracking in query-by-singing-humming.  In ISMIR, Taipei, Taiwan, 2014.
  * [16] Jordi Pons and Xavier Serra.  Designing efficient architectures for modeling temporal features with convolutional neural networks.  In ICASSP, New orleans, USA, 2017.
  * [17] Jordi Pons, Olga Slizovskaia, Rong Gong, Emilia Gómez, and Xavier Serra.  Timbre analysis of music audio signals with convolutional neural networks.  arxiv.org:1703.06697, 2017.
  * [18] Hasim Sak, Andrew W. Senior, Kanishka Rao, and Françoise Beaufays.  Fast and accurate recurrent neural network acoustic models for speech recognition.  CoRR, abs/1507.06947, 2015.
  * [19] J. Sohn, N. S. Kim, and W. Sung.  A statistical model-based voice activity detection.  IEEE signal processing letters, 6(1):1–3, 1999.
  * [20] C. C. Wang and J. S. R. Jang.  Improving query-by-singing/humming by combining melody and lyric information.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(4):798–806, April 2015.
  * [21] L. Wang, S. Huang, S. Hu, J. Liang, and B. Xu.  An effective and efficient method for query by humming system based on multi-similarity measurement fusion.  In International Conference on Audio, Language and Image Processing, pages 471–475, July 2008.
  * [22] E. Wichmann.  Listening to Theatre: The Aural Dimension of Beijing Opera.  University of Hawaii Press, 1991.


11institutetext: Massachusetts General Hospital and Harvard Medical School

# A Prompt-driven Universal Model for View-Agnostic Echocardiography Analysis

Sekeun Kim 11 Hui Ren 11 Peng Guo 11 Abder-Rahman Ali 11 Patrick Zhang 11
Kyungsang Kim 11 Quanzheng Li 11 Xiang Li 11

###### Abstract

Echocardiography segmentation for cardiac analysis is time-consuming and
resource-intensive due to the variability in image quality and the necessity
to process scans from various standard views. While current automated
segmentation methods in echocardiography show promising performance, they are
trained on specific scan views to analyze corresponding data. However, this
solution has a limitation as the number of required models increases with the
number of standard views. To address this, in this paper, we present a prompt-
driven universal method for view-agnostic echocardiography analysis.
Considering the domain shift between standard views, we first introduce a
method called prompt matching, aimed at learning prompts specific to different
views by matching prompts and querying input embeddings using a pre-trained
vision model. Then, we utilized a pre-trained medical language model to align
textual information with pixel data for accurate segmentation. Extensive
experiments on three standard views showed that our approach signiﬁcantly
outperforms the state-of-the-art universal methods and achieves comparable or
even better performances over the segmentation model trained and tested on
same views.

###### Keywords:

Universal Model Prompt learning Visual-Language Echocardiography

##  1 Introduction

Echocardiography is the most frequently used imaging modality performed in
cardiology, which helps assessment of cardiac function by examining different
views by multiple standard views scanning. Given the complexity in image
analysis and workload of sonographers, there is a growing interest in
developing automated methods for segmentation in Echo [8, 9, 11]. Existing
methods have demonstrated superior performance in accurately delineating
anatomical structures within specific views when they are trained on
corresponding datasets. This process involves a step of identifying the
desired views in the patient study before conducting the analysis, which
requires an additional steps to select the appropriate files in the scan files
[3, 6]. The exploration of a general model capable of independently performing
echocardiography segmentation tasks across multiple standard views has not
been explored.
Currently, the general solution is training N models on N standard views.
However, this solutions have a limitation as the number of models are
increases with the number of standard views. One naive approach to creating a
universal model is to train the same network on data from various standard
view scans. This approach may result in a performance degradation of the
model, as each standard view manifests its distinct visual characteristics [9,
15]. Echocardiography poses distinct challenges, including view domain shift
among scan views and sparse annotation across frames. While there may not be
identical solutions to these problems, similar universal models have been
developed [23, 2, 13, 22]. DoDNet[23] introduces a dynamic head with an
encoder-decoder architecture. Different task information is encoded to one-hot
vector, and it controls the task-specific controller. CLIP-drvien universal
model[13] extends this idea by utilizing a pre-trained text model and managing
segmentation heads using semantically embedded class features. Although this
CLIP-based strategy has demonstrated success in CT organ segmentation tasks,
it has limitations in the medical framework due to disparities between natural
and medical texts. UniSeg[22] approaches with learnable prompts have been
developed to address various segmentation tasks in CT,MR, and PET. However, it
introduces a universal method using three anatomically similar datasets, where
the images may differ only in texture while the underlying anatomy remains the
same. Consequently, approach is likely to be less suitable for addressing view
shifts in echocardiography, potentially resulting in suboptimal performance as
in Table 2.
To address above problems, we propose a prompt-driven universal model that
allows superior segmentation of cardiac structures with state-of-the-art
performance. Our model incorporates a prompt learning with prompt pool and
pre-trained language model’s knowledge by pixel-text alignment. We first
employ a prompt pool-based prompt learning approach enables the development of
a universal model capable of handling various scan view data. Accordingly, it
effectively tackles the aforementioned issue by enabling dynamic adaptation to
diverse inputs. Second, The score maps facilitates pixel-text alignment, which
allows our model to fully leverage language information for medical
segmentation tasks. It effectively address the above problem by enabling input
video to select particular view specific prompt and focus on particular
semantic features with language model. To the best of our knowledge, this is
the first work that approaches unified model segmentation in echocardiography.
Our method simplifies current cardiac analysis by removing the necessity for a
view identification step to acquire the desired view from various DICOM scans
of the patient. Our method is evaluated on three standard views from three
different datasets and show promising performance compared to other universal
methods.
Our contributions can be summarized as follows:
• We present a prompt-driven universal model, comprising a prompt pool to
accommodate different standard views, and leveraging pixel-text alignment with
the prior knowledge of a pre-trained text model for view-agnostic
echocardiography segmentation.
• The proposed method streamlines cardiac analysis by minimizing the
requirement for a view identification step during the retrieval of the desired
view from patient scans.
• We demonstrate that our model achieves SOTA performance for cardiac
segmentation tasks compared to the previous universal approach through
extensive experiments on various datasets.

Figure 1: Over all framework of our proposed universal model for view-agnostic
segmentation. The pre-trained model and query model remain frozen, while the
other models are trainable.

##  2 Method

As illustrated in Figure 1, our proposed approach comprises the following
components: text, video encoder, a prompt pool consisting of trainable key and
value, an MLP layer, and a video decoder. We utilize a clinicalBert [1] for
enhanced extraction of medical text representations. Our goal is to segment
objects in all frames across different scan views. In order to achieve this
goal, we introduce two key components in our model: 1) a pixel-text dense
alignment mechanism that effectively bridges the gap between a pre-trained
language model and pixel-level representations for dense prediction tasks, and
2) a prompt matching technique that leverages a prompt pool to adaptively
select the optimal view-specific prompt for each task.

###  2.1 Problem Definition

Given total number of N datasets D = $\\{D_{1},D_{2},\ldots,D_{N}\\}$, each
dataset $D_{i}=\left\\{X_{ij},Y_{ij}\right\\}_{j=1}^{n_{i}}$, $X_{ij}$ and
$Y_{ij}$ are the video with total frame of ${F}$ and the corresponding ground
truth, represents total of $n_{i}$ pixels. Each videos $X_{ij}\in V_{k}$ , V
with total number of K views $V\in\\{V_{1},V_{2},\ldots,V_{K}\\}$. If there
are $\forall\textit{F}$ annotated in $Y_{ij}$, $D_{i}$ is fully labeled
dataset; otherwise $D_{i}$ is a partially labeled dataset. The objective is to
train a model F(·) using the partially labeled dataset $D_{i}$ =
$\\{D_{1},D_{2},\ldots,D_{N}\\}$, where the model is capable of making dense
predictions for all K classes along all frame ${F}$.

###  2.2 Pixel-text dense alignment

In the computer vision field, a series of works on vision and language models
(VLM) have emerged. Further in medical domain, previous studies have
successfully adapted CLIP embeddings for medical applications [18, 13].
However, the use of CLIP trained on natural image-text pairs weakens the
semantic meaning embedding for medical prompt into the models as shown in
Table 3. To fully leverage the knowledge encoded in the pre-trained medical
language model, we utilized ClinicalBert [1] for dense prediction. We generate
text embeddings by converting $N$ classes into text prompts using a template
of “An echocardiography of [Classes].” to generate text embeddings
$\mathcal{F}(c)\in\mathbb{R}^{N\times D}$. An input video is encoded through
the backbone video encoder to embed intermediate local video embedding
$\mathcal{G}(x)\in\mathbb{R}^{T_{i}H_{i}W_{i}\times D}$, i = 1, …, L where
$T_{i}$, $H_{i}$, and $W_{i}$ are the frame, height and width of the local
embeddings from i-th layer and D indicates embedding dimension. Then we
compute the score maps with pixel-text alignment using the text embedding and
the vision embedding by:

$\mathcal{S}=\bar{\mathcal{G}(x)}\bar{\mathcal{F}(c)^{T}}$

where superscript - refers the normalization along channel dimension and T
denotes the transpose operation. The score map $\mathcal{S}$ can be employed
for auxiliary segmentation at a lower resolution denoted as pixel-text loss.
We concatenate pixel-text score map $\mathcal{S}$ to local embeddings ${f}$ to
incorporate text priors. We utilized the chamber class for the text encoder
without incorporating any view information.

###  2.3 Prompt matching and text-driven parameter generation

Given an 2D-t input $x\in\mathbb{R}^{T\times H\times W\times C}$ and
$\mathfrak{Q}$ is a pre-trained Vision Transformer(ViT) of Segment Anything
Model [10], first frame of video is divided into patches and then embed as
patch embeddigs $\mathfrak{Q}:\mathbb{R}^{L\times(S^{2}\times
C)}\rightarrow\mathbb{R}^{L\times D}$ where S indicates the patch size and C
denotes input channels, and D is embedding dimension. The prompt pool consist
of two learnable key
$\left\\{(k_{1},P_{1}),(k_{2},P_{2}),...,(k_{M},P_{M})\right\\}$, where
$k_{i}\in\mathbb{R^{D}}$ and learnable value
$\left\\{P_{1},P_{2},...,P_{M}\right\\}$, where $P_{i}\in\mathbb{R^{L\times
D}}$. In our settings, the total number of entries in the prompt pool, denoted
by $M$, is equal to the number of views multiplied by a pre-defined prompt
size assigned to each view, which is set to 3. The queried input embeddings
and prompt keys for each views are drawn towards each other to maximize their
cosine distance in training steps, denoted as $\mathcal{L}_{pr}$. We use a
global average pooling (GAP) layer on the last encoder features to obtain a
global representation of the current video input. Then we utilize the text
embeddings together with the prompt values and global embedding to generate
parameters for the chamber segmentation heads, $\theta_{N}$. These parameters
used in video decoder heads and generate binary prediction for ${N}$ classess
[21]. This design facilitates view-agnostic prompt and preserving a view
information during test time.

Table 1: Details of the publicly available datasets for training and evaluation. Dataset |  Scan View |  Annotation |  #Total Scan (train/test)
---|---|---|---
CAMUS [12] |  A2C |  LVendo, LVepi |  500 (450/50)
|  A4C |  LVendo, LVepi |  500 (450/50)
EchoNet-Pediatric [19] |  A4C |  LVendo |  3284 (2580/704)
|  PSAX |  LVendo |  4526 (3559/967)
EchoNet-Dynamic [16] |  A4C |  LVendo |  10036 (8753/1277)

###  2.4 Loss Function

####  2.4.1 Video Masked Back-propagation

In our problem, the labels are distributed extremely sparsely across the
frames which is difference from previous work [13]. In this work, we designed
the video masked back propagation to address the partially labeled issue.
Specifically, we masked frames that does not have class label and only back
propagate loss to update parameter in our network. In this way, we can utilize
sparsely labeled problem and do accurate segmentation in video with partially
labeled dataset.

####  2.4.2 Total loss

Our objective is to achieve segmentation by minimizing two loss terms
including a prompt matching loss and segmentation loss with masked
backpropagation through optimization of the following loss function:

$\mathcal{L}_{\text{seg}}=\lambda_{1}\mathcal{L}_{\text{pixel-
text}}+\lambda_{2}\mathcal{L}_{\text{BCE}},\quad\mathcal{L}_{\text{pr}}=<\mathfrak{Q}(X_{i0}),P_{key}>$
$\mathcal{L}_{\text{total}}=(1-\lambda(t))\mathcal{L}_{\text{seg}}-\lambda(t)\mathcal{L}_{\text{pr}}$

where $\mathcal{L}_{seg}$ denotes the segmentation loss combining different
two losses, $\mathcal{L}_{pixel-text}$ represents CE loss with score maps, and
$\mathcal{L}_{BCE}$ represents the binary cross-entropy loss, respectively.
Throughout the experiments, $\lambda_{1}$ and $\lambda_{2}$ are set equally.
$\mathcal{L}_{pr}$ denotes the cosine similarity between queried input and
prompt keys assigned by view types during training steps. $\lambda$ is
scheduled by the time dependent ramp up Gaussian function
$\lambda(t)=\exp^{-5(1-t/t_{max})^{2}}$ where t is the current iteration and
$t_{max}$ is the maximal iteration. Since our prompt keys are converged in the
early stage, we reduce the weights on prompt matching loss during this initial
phase.

Table 2: Quantitative comparison across all datasets. The Dice scores [%] are presented, with the best results highlighted in bold. Method | A2C | A4C | PSAX | Indiv A4C perf.
---|---|---|---|---
|  LV${}_{\text{endo}}$ |  LV${}_{\text{epi}}$ |  LV${}_{\text{endo}}$ |  LV${}_{\text{epi}}$ | LV${}_{\text{endo}}$ |  Camus |  Pedia |  Dynamic
View-specific |  |  |  |  |  |  |  |
SwinUNETR [5] |  90.5 |  86.7 |  85.9 |  86.7 | 87.7 |  90.4 |  83.1 |  84.2
U-Transformer [17] |  93.3 |  88.1 |  88.3 |  88.5 | 88.1 |  92.8 |  86.4 |  85.9
View-integrated |  |  |  |  |  |  |  |
SwinUNETR [5] |  88.8 |  83.1 |  82.9 |  84.7 | 85.1 |  87.3 |  80.4 |  81.1
U-Transformer [17] |  89.3 |  83.6 |  85.7 |  84.1 | 85.4 |  87.5 |  84.8 |  84.9
DoDNet [23] |  90.8 |  87.1 |  84.9 |  85.3 | 87.2 |  87.9 |  85.3 |  81.5
Clip-driven [13] |  91.1 |  87.6 |  85.1 |  86.4 | 88.5 |  89.1 |  82.4 |  83.9
UniSeg [22] |  92.3 |  86.5 |  87.7 |  86.3 | 88.3 |  93.1 |  84.9 |  85.1
UniverSeg [2] |  83.3 |  80.7 |  82.1 |  81.8 | 81.0 |  83.2 |  80.3 |  82.9
Ours |  93.2 |  88.8 |  88.5 |  88.3 | 89.4 |  93.7 |  85.3 |  86.6
Figure 2: Qualitative visualization of segmentation results generated from our
method and state-of-the-art universal methods on representative image. Red and
blue represents LVendo and LVepi, respectively.

##  3 Experiments and Results

Materials. We evaluated the proposed method using three publicly available
datasets [12][19][16]. These datasets consist of 2D B-mode scan data annotated
different cardiac chambers at end-diastole (ED) and end-systole (ES). The
annotation includes the left ventricle endocardium (LV${}_{\text{endo}}$) and
the left ventricle epicardium (LV${}_{\text{epi}}$) in apical two-chamber
(A2C), apical four-chamber (A4C), and parasternal short-axis (PSAX) views. We
followed a predefined split set as shown in Table 1.
Implementation and Evaluation Metric To guarantee a fair comparison across all
experiments, we standardized the training and testing settings. The
experiments were conducted using PyTorch, with a consistent batch size of 5
over 100 epochs with Nvidia A100. We utilized the Unet architecture [20] as
the backbone to incorporate our key components. For optimization, we employed
the MADGRAD optimizer [4], setting the learning rate to 1e-4. Images are
resized to 224$\times$224 pixels with 16 frames and normalized to ensure a
zero mean and unit variance. To enhance the robustness of our model, we apply
various augmentation technique, including random flip, rotation within a range
of $-$30 to +30, and shearing along x-y dimensions. We utilize the Dice
Similarity Coefficient (DSC) for the evaluation of our model’s performance. We
compared our method across three scan views: A4C, A2C, and PSAX. These scan
views were selected based on the currently available echo dataset. The
performance of the models was evaluated on ED and ES cardiac phases, where
annotations were available.
Comparison study We present the performance of the proposed method for cardiac
segmentation across different scan views. To the best of our knowledge, our
method is the first approach capable of performing cardiac segmentation with
view-agnostic input that can perform universal segmentation. We compared our
proposed method under two different settings: 1) trained and tested on the
same views noted as view-specific model approach, and 2) trained on all views
and tested on all views noted as view-integrated model approach. We chose two
methods, SwinUNETR [5] and U-transformer [17] for baseline segmentation model
based on previous study [7]. We also compare the performance of proposed
method against universal models, including DoDNet [19], CLIP-driven universal
model [13], UniSeg [22], and UniverSeg [2] for cardiac segmentation tasks on
three datasets. The CLIP-driven universal model [13] substitutes one-hot
embeddings [23] with CLIP text embeddings. UniSeg [22] employs a learnable
prompt for generating task-specific embeddings. Additionally, our comparision
includes a few-shot based universal segmentation model called UniverSeg [2].
As depicted in Table 2, our method exhibits the capability to generate
excellent segmentation results even with view-agnostic input, as illustrated
in Fig. 2. In comparison to baseline models with view-integrated approaches,
our model demonstrates better performance compared to the view-specific based
U-transfomrer except on LVendo (93.2 vs. 93.3) and LVepi (88.3 vs. 88.5) in
A2C and A4C, respectively. Moreover, by incorporating prompts to adaptively
integrate input view information, our model outperforms all universal models
in delineating regions of interest (ROI) across all views. Since, our model
adapts the prompt to match input view types, resulting in enhanced
segmentation results and superior performance. Furthermore, our method
improves mean segmentation performance compared to employing a few-shot
segmentation method (89.64 vs 81.7). These experimental findings demonstrate
that our approach effectively leverages adaptable prompts to produce superior
segmentation results.
Ablation study To evaluate the effectiveness of the each component, we
conducted an ablation study to quantify the impact of different elements in
our model on segmentation performance. First, we evaluated performance metrics
without text-encoder path, and then with text-encoder path employing various
designs including one-hot encoding, as well as leveraging prior knowledge from
language models including CLIP and ClinicalBert. In the absence of text-
encoder path, which removes pixel-text alignment for auxiliary loss, we
observed a degradation in performance (89.6 to 85.6) showing that an pixel-
text alignment is crucial for our model. Additionally, we compared different
types of text encoders trained on both natural and medical text. The findings
indicate that the CLIP language model is not as effective in representing
medical text compared to the ClinicalBert (88.8 vs. 89.6). Secondly, we
assessed the classification performance based on selected prompt keys. We
examined the assigned prompt keys with majority voting determined by
$\arg\max_{G\in{A,B,C}}\sum_{i=1}^{3}\mathbb{I}(x_{i}\in G)$. T-SNE
visualization [14] is presented in Figure 3. We observed that the accuracy for
distinguishing between apical views and parasternal views was 0.96. However,
the accuracy for distinguishing between A2C and A4C was 0.54 and 0.6,
respectively. Secondly, there is variability between view classes annotated by
human readers due to ambiguous scan angles between A2C and A4C. In fact, the
probe, guided by manual manipulation, often fails to accurately capture both
the A2C and A4C angles when rotated from a single position. This is evident in
Table 4, where providing view information instead of selecting keys from the
prompt pool resulted in a model performance of 89.4, lower than the
performance of 89.6 achieved when view information was not provided.

Encoder design | Dice [%]
---|---
w/o text-encoder | 85.6
w/ One-hot | 88.6
CLIP | 88.8
ClinicalBert | 89.6

Table 3: Comparison of mean model performance on different encoder design.

View | Dice [%]
---|---
w/ | 89.4
w/o | 89.6

Table 4: Comparison of model performance with and without explicit view
information.

Figure 3: T-SNE visualization of prompt keys.

##  4 Conclusion

In this study, we introduce an innovative prompt-driven universal
echocardiography segmentation model that is capable of learning cardiac
segmentation across different standard views using partially labeled data.
This model incorporates the knowledge of pre-trained language models by
aligning text representation with visual pixel data. We also suggest a prompt
matching technique through a prompt pool to achieve view-agnostic
echocardiography segmentation. Our research utilized three standard views to
demonstrate the feasibility of our proposed model, highlighting its potential
to be expanded to additional standard views to create a universal model for
echocardiography. This method simplifies the process by eliminating the need
for a separate step to identify views, thereby reducing the variability
introduced by humans in selecting views for analysis in a patient’s study.
Extensive experiments on the echocardiography segmentation benchmark across
various scan views have shown that our approach not only performs superiorly
but also proves to be highly effective.

## References

  * [1] Alsentzer, E., Murphy, J.R., Boag, W., Weng, W.H., Jin, D., Naumann, T., McDermott, M.: Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323 (2019)
  * [2] Butoi, V.I., Ortiz, J.J.G., Ma, T., Sabuncu, M.R., Guttag, J., Dalca, A.V.: Universeg: Universal medical image segmentation. arXiv preprint arXiv:2304.06131 (2023)
  * [3] Charton, J., Ren, H., Kim, S., Gonzalez, C.M., Khambhati, J., Cheng, J., DeFrancesco, J., Waheed, A., Marciniak, S., Moura, F., et al.: Multi-task learning for hierarchically-structured images: Study on echocardiogram view classification. In: International Workshop on Advances in Simplifying Medical Ultrasound. pp. 185–194. Springer (2023)
  * [4] Defazio, A., Jelassi, S.: Adaptivity without compromise: a momentumized, adaptive, dual averaged gradient method for stochastic optimization. The Journal of Machine Learning Research 23(1), 6429–6462 (2022)
  * [5] Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H.R., Xu, D.: Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In: International MICCAI Brainlesion Workshop. pp. 272–284. Springer (2021)
  * [6] Jeon, J., Ha, S., Jang, Y., Yoon, Y.E., Kim, J., Jeong, H., Jeong, D., Hong, Y., Chang, S.A.L.H.J.: Improving out-of-distribution detection in echocardiographic view classication through enhancing semantic features (2023)
  * [7] Kim, S., Kim, K., Hu, J., Chen, C., Lyu, Z., Hui, R., Kim, S., Liu, Z., Zhong, A., Li, X., et al.: Medivista-sam: Zero-shot medical video analysis with spatio-temporal sam adaptation. arXiv preprint arXiv:2309.13539 (2023)
  * [8] Kim, S., Park, H.B., Jeon, J., Arsanjani, R., Heo, R., Lee, S.E., Moon, I., Yoo, S.K., Chang, H.J.: Fully automated quantification of cardiac chamber and function assessment in 2-d echocardiography: clinical feasibility of deep learning-based algorithms. The International Journal of Cardiovascular Imaging 38(5), 1047–1059 (2022)
  * [9] Kim, T., Hedayat, M., Vaitkus, V.V., Belohlavek, M., Krishnamurthy, V., Borazjani, I.: Automatic segmentation of the left ventricle in echocardiographic images using convolutional neural networks. Quantitative Imaging in Medicine and Surgery 11(5), 1763 (2021)
  * [10] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [11] Leclerc, S., Smistad, E., Østvik, A., Cervenansky, F., Espinosa, F., Espeland, T., Berg, E.A.R., Belhamissi, M., Israilov, S., Grenier, T., et al.: Lu-net: a multistage attention network to improve the robustness of segmentation of left ventricular structures in 2-d echocardiography. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 67(12), 2519–2530 (2020)
  * [12] Leclerc, S., Smistad, E., Pedrosa, J., Østvik, A., Cervenansky, F., Espinosa, F., Espeland, T., Berg, E.A.R., Jodoin, P.M., Grenier, T., et al.: Deep learning for segmentation using an open large-scale dataset in 2d echocardiography. IEEE transactions on medical imaging 38(9), 2198–2210 (2019)
  * [13] Liu, J., Zhang, Y., Chen, J.N., Xiao, J., Lu, Y., A Landman, B., Yuan, Y., Yuille, A., Tang, Y., Zhou, Z.: Clip-driven universal model for organ segmentation and tumor detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 21152–21164 (2023)
  * [14] Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(11) (2008)
  * [15] Mitchell, C., Rahko, P.S., Blauwet, L.A., Canaday, B., Finstuen, J.A., Foster, M.C., Horton, K., Ogunyankin, K.O., Palma, R.A., Velazquez, E.J.: Guidelines for performing a comprehensive transthoracic echocardiographic examination in adults: recommendations from the american society of echocardiography. Journal of the American Society of Echocardiography 32(1), 1–64 (2019)
  * [16] Ouyang, D., He, B., Ghorbani, A., Yuan, N., Ebinger, J., Langlotz, C.P., Heidenreich, P.A., Harrington, R.A., Liang, D.H., Ashley, E.A., et al.: Video-based ai for beat-to-beat assessment of cardiac function. Nature 580(7802), 252–256 (2020)
  * [17] Petit, O., Thome, N., Rambour, C., Themyr, L., Collins, T., Soler, L.: U-net transformer: Self and cross attention for medical image segmentation. In: Machine Learning in Medical Imaging: 12th International Workshop, MLMI 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, September 27, 2021, Proceedings 12. pp. 267–276. Springer (2021)
  * [18] Qin, Z., Yi, H., Lao, Q., Li, K.: Medical image understanding with pretrained vision language models: A comprehensive study. arXiv preprint arXiv:2209.15517 (2022)
  * [19] Reddy, C.D., Lopez, L., Ouyang, D., Zou, J.Y., He, B.: Video-based deep learning for automated assessment of left ventricular ejection fraction in pediatric patients. Journal of the American Society of Echocardiography 36(5), 482–489 (2023)
  * [20] Ronneberger, O., Fischer, P., Brox, T.: U-net: c networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. pp. 234–241. Springer (2015)
  * [21] Tian, Z., Shen, C., Chen, H.: Conditional convolutions for instance segmentation. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16. pp. 282–298. Springer (2020)
  * [22] Ye, Y., Xie, Y., Zhang, J., Chen, Z., Xia, Y.: Uniseg: A prompt-driven universal segmentation model as well as a strong representation learner. arXiv preprint arXiv:2304.03493 (2023)
  * [23] Zhang, J., Xie, Y., Xia, Y., Shen, C.: Dodnet: Learning to segment multi-organ and tumors from multiple partially labeled datasets. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1195–1204 (2021)