# Innovations in Cover Song Detection: A Lyrics-Based Approach

Maximilian Balluff1, Peter Mandl1, Christian Wolff2,

1Hochschule Muenchen University of Applied Science, 2University of Regensburg,
Correspondence<EMAIL_ADDRESS>
###### Abstract

Cover songs are alternate versions of a song by a different artist. Long being
a vital part of the music industry, cover songs significantly influence music
culture and are commonly heard in public venues. The rise of online music
platforms has further increased their prevalence, often as background music or
video soundtracks. While current automatic identification methods serve
adequately for original songs, they are less effective with cover songs,
primarily because cover versions often significantly deviate from the original
compositions. In this paper, we propose a novel method for cover song
detection that utilizes the lyrics of a song. We introduce a new dataset for
cover songs and their corresponding originals. The dataset contains 5078 cover
songs and 2828 original songs. In contrast to other cover song datasets, it
contains the annotated lyrics for the original song and the cover song. We
evaluate our method on this dataset and compare it with multiple baseline
approaches. Our results show that our method outperforms the baseline
approaches.

Innovations in Cover Song Detection: A Lyrics-Based Approach

Maximilian Balluff1, Peter Mandl1, Christian Wolff2, 1Hochschule Muenchen
University of Applied Science, 2University of Regensburg, Correspondence:
<EMAIL_ADDRESS>

##  1 Introduction

Cover versions, or cover songs, represent reinterpretations of original
compositions by different artists, holding a significant place in the music
industry and influencing musical culture. These renditions are frequently
featured in public venues such as bars, clubs, festivals, and restaurants. The
rise of online music platforms has further amplified their ubiquity, often
serving as background music or soundtracks for videos Vervielfältigungsrechte
(2021).

Identifying and monetizing cover songs poses a significant challenge for
rights holders, leading to an increasing demand for efficient methods to
detect cover songs on a large scale. Composers and performers, often members
of collecting societies, rely on accurate song identification for royalty
distribution. While existing automatic identification methods effectively
handle original songs, detecting cover songs remains challenging due to the
substantial variations introduced by different artists in their renditions Sha
(2023).

##  2 Related Work

Cover Song Detection (CSD) plays a pivotal role in Music Information
Retrieval, especially in recent years. Existing approaches are predominantly
centered around audio analysis. For instance, Tsai et al. Tsai et al. (2005)
focused on tempo, transposition, and accompaniment to compare original songs
with their covers. Serrà introduced a method using audio features to compute a
similarity matrix between original and cover songs Serrà Julià (2007). In
another study Du et al. (2021), advanced song embeddings were generated using
a convolutional neural network and utilized for similarity computation. These
methods commonly extract audio features, ranging from high-level attributes
like tempo and beats per minute Tsai et al. (2005) to low-level features like
Harmonic Pitch Class Profile Serrà Julià (2007) and Constant-Q-Transform Du et
al. (2021). Some approaches combine both, incorporating features like the mean
and standard deviation of Mel-Frequency Cepstral Coefficient Yu et al. (2008).

In contrast, only a few methods leverage song lyrics for CSD. Correya et al.
employed a unique approach using the lyrics and title of songs through a bag-
of-words algorithm Correya et al. (2018). Similarly, Vaglio et al. proposed a
method combining lyrics and audio for cover song detection Vaglio et al.
(2021). They processed the audio in two branches—one for lyrics and the other
for audio. After detecting singing voice, lyrics were transcribed, and string
matching identified corresponding cover versions. The results from both
branches were then combined for the final output.

##  3 Idea and Dataset

Our central proposition asserts that cover song lyrics exhibit notable
similarities with their original counterparts. Given the pivotal role of
lyrics in music, especially in genres like Pop, Hip Hop/Rap,
Rock/Hardrock/Heavy Metal, and Kids Music, where over 70% of sales are
concentrated e. V. (2021), investigating this hypothesis becomes imperative.

To substantiate our claim, we introduce a novel dataset comprising 5078 cover
songs and 2828 original songs. Collected from genius.com between June and
September 2023, this dataset stands out by providing annotated lyrics for both
the original and cover songs. Language analysis, conducted using a transformer
for language detection Conneau et al. (2020), reveals that approximately 85%
of the original songs are in English, followed by Spanish (4.7%) and
Portuguese (2.3%).

To assess lyrical similarity, we employ Levenshtein distance Levenshtein
(1996) and word error rate (WER) Klakow and Peters (2002). The mean
Levenshtein distance between a cover song and its original 302.62 is notably
lower than the mean (689.25) and minimum (504.76) distances to other songs;
for more details, compare figure 1. Surprisingly, 23.6% of non-matching songs
exhibit a lower distance than the actual original, with 301 instances where
the detected language of the cover and original differs. Further analysis
reveals instances of low distances in non-matching songs, often attributed to
poor annotations, including incomplete cover lyrics or variations in
annotation styles (e.g., filler words like "Uh, uh, uh…" or paragraph
descriptions like [Verse], [Refrain]). The average WER distance between a
cover song and its original, denoted as 0.54, is significantly less than the
average (1.33) and minimum (0.87) distances observed with other songs. These
findings underscore the challenges posed by these differences for classical
distance metrics, necessitating thorough preprocessing for accurate analysis.

Figure 1: Levenshtein distance between (upper) and WER (lower) between cover
and original songs, and between cover and other songs

The dataset will be published in the future. Currently, we are working on
increasing the size of the dataset and improving the quality. If possible we
will merge our dataset with other datasets like Second Hand Songs (SHS)
Bertin-Mahieux et al. (2011).

###  3.1 Baseline

As we are the first to utilize our new dataset, our primary objective is to
establish multiple baseline algorithms for comparison purposes. We utilize the
string similarity distances, namely Levenshtein distance ($L_{D}$) and WER
($WER$), along with a Bag-of-Words algorithm combined with Cosine distance
($BOW$), to calculate the similarity between lyrics. We opted for this set of
distances due to their relatively low complexity, and we benefit from existing
implementations for Levenshtein and WER Bachmann (2023); pyt (2023).

###  3.2 Proposed method

Since the initial introduction of transformer neural networks Vaswani et al.
(2017), these networks have become the standard for text preprocessing. We
utilize a transformer as a feature extractor for the lyrics, which generates
an embedding. Lyrics with similar content should yield similar embeddings. The
embeddings for each original song are stored in a database. For each cover
song, we compute the similarity between the cover song’s embedding and the
embeddings of the original songs. We order the songs by their similarity, the
original song with the highest similarity score is predicted to be the
original song for the cover version.

####  3.2.1 Model

In our approach, we use a pre-trained model for sequence similarity Müller et
al. (2022), which is based on XLM-RoBERTa Conneau et al. (2020) and is pre-
trained on the SNLI Bowman et al. (2015), MNLI Williams et al. (2018), ANLI
Nie et al. (2020) and XNLI Conneau et al. (2018) datasets. On top of the base
model, we add two fully connected and an embedding layer. The embedding layer
serves the purpose of transforming the sequence of features into a fixed-
length floating-point embedding vector, providing a condensed representation
of the lyrics that enables subsequent similarity calculations. The model
architecture is shown in figure 2. We did not further optimize the model
architecture. The dimensions of the layers are based on our experience in
previous text matching projects.

Figure 2: Model architecture

Various methods can be employed to train embeddings, and a common approach is
to use Siamese neural networks Chicco (2021). These networks consist of two
identical neural networks that share the same weights. The inputs for these
networks are the two samples that need to be compared, with the objective of
minimizing the distance between similar inputs and maximizing the distance
between dissimilar ones.

In our approach, we adopt the triplet loss method, originally introduced for
face recognition Chechik et al. (2009). The goal is to minimize the distance
between the anchor and the positive sample while maximizing the distance
between the anchor and the negative sample. Achieving this involves sampling
triplets, which consist of an anchor, a positive sample, and a negative sample
Chechik et al. (2009). To address the challenge of the relatively high cost of
finding good triplets, we employ the online mining strategy Schroff et al.
(2015). Instead of mining triplets before passing them to the model, the
triplets are mined within the training batch. This is accomplished by passing
a batch of samples to the model, calculating the embeddings and distances for
all embeddings in the batch, and then searching for each sample used as an
anchor for other positive and negative samples within the batch.

####  3.2.2 Evaluation

The evaluation of our approach relies on three widely-applied metrics in the
field of Cover Song Detection (CSD): mean average precision (mAP), mean rank
(MR), and precision at one (P@1). These metrics have been extensively used in
the evaluation of various CSD approaches Du et al. (2021, 2022); Chicco
(2021). mAP calculates the average precision of a ranked list of documents,
where precision is defined as the ratio of relevant documents among the
retrieved documents Christopher et al. (2008). MR provides the position of the
first relevant document in the ranked list of documents, with lower values
indicating better performance (ideally one) Osmalskyj (2017). P@1 represents
the precision at the first rank of the ranked list of documents Xu et al.
(2018).

Given the limited size of our dataset, we chose to implement k-fold cross-
validation with five folds, as recommended by Dat (2011). In order to maintain
consistency and comparability across all experiments, the same folds were
utilized. Each fold encompassed several cover lyrics and their corresponding
originals. The fold was further divided into training (64%), validation (16%),
and testing (20%) sets. The validation set is necessary to prevent our model
from overfitting the training data, using EarlyStopping and ReduceLearningRate
callbacks that monitore the validation loss.

It is crucial to note that no original version is duplicated among these sets,
ensuring that each set contains a unique set of original songs. This resulted
in varying sizes for each fold, due to the uneven distribution of the number
of covers per original. Table 1 provides an overview of the number of unique
lyrics (original and cover) for each fold.

To evaluate the model’s performance, we utilized the test set. For each lyric,
whether it be a cover or an original, we generated embeddings. The embeddings
from the original lyrics formed our database. Subsequently, for each cover
embedding, we calculated the Manhattan distance Royer (2001) to all original
embeddings, sorting them based on their distances. Our metrics were then
computed using this ranking. Model training was conducted on an Nvidia T4 GPU,
utilizing a n1-standard-8 instance on Google Cloud. The training process
spanned approximately 22 hours for all five folds, averaging around 19 epochs.
Notably, our model boasts 278 million parameters, of which 831 thousand are
trainable.

| Training | Validation | Test
---|---|---|---
Fold 1 | 5058 | 1313 | 1535
Fold 2 | 4975 | 1397 | 1534
Fold 3 | 5069 | 1224 | 1613
Fold 4 | 4953 | 1334 | 1619
Fold 5 | 5026 | 1275 | 1605
Table 1: Unique lyrics (original and cover) for each fold

Our baseline approaches do not require training, and therefore, we only use
the test set for evaluation. We compute the distance between the cover song
and all original songs. The original song with the smallest distance is then
predicted as the original for the cover song.

####  3.2.3 Results

The results of our experiments are summarized in Table 2. The Bag-of-Words
($BOW$) approach emerges as the best baseline, yielding a mAP of 85.74%, a MR
of 46.29, and Precision@1 of 83.65%. The triplet model outperforms the
baseline approaches in all metrics except precision, achieving a mAP of
87.17%, a MR of 18.51, and Precision@1 of 83.57%. Notably, all methods yield
approximately the same mAP, while the MR varies significantly. Remarkably, the
triplet model, with its substantially lower rank, outperforms the $BOW$ model
by more than a factor of two.

Although we did not initially plan to evaluate the computation time of the
approaches, it is worth noting that WER distance is significantly slower. It
took over 8 hours to compute the distance for all splits, whereas for all
other approaches, it was a matter of minutes.

| mAP | MR | P@1
---|---|---|---
$\mathbf{L_{D}}$ |  | 81.65
---
± 0.74%
| 82.57
---
± 4.74
| 80.23
---
± 0.54%
$\mathbf{WER}$ |  | 82.91
---
± 1.27%
| 75.01
---
± 8.43
| 81.52
---
± 0.97%
$\mathbf{BOW}$ |  | 85.74
---
± 0.67%
| 46.29
---
± 1.80
| 83.65
---
± 0.71%
| Triplet
---
Model
| 87.17
---
± 1.50%
| 18.51
---
± 2.35
| 83.57
---
± 1.80%
Table 2: Average results on the 5-fold cross-validation

##  4 Limitations and Future Work

While our approach shows promise, it is not without limitations. Firstly,
direct comparisons with other CSD approaches that focus on audio features pose
challenges, given our method’s exclusive operation on our introduced dataset.
A meaningful comparison would necessitate additional steps, including
downloading and extracting audio files from YouTube, conducting speech-to-text
transcriptions, and comparing lyrics. Importantly, a comparison of our
approaches to methods that use datasets like SHS Bertin-Mahieux et al. (2011)
requires evaluating how well current speech-to-text algorithms handle singing
voice, potentially involving training models like Whisper Radford et al.
(2022) for this task. This leads to the second limitation, lyrics extracted
from the audio may differ from the annotated lyrics, introducing potential
errors, or missing annotation. As already mentioned it is necessary to
evaluate our approaches on extract lyrics. Lastly, the context window of our
model, while suitable for average-sized lyrics, may prove insufficient for
larger lyrics, potentially leading to information loss and impacting cover
song detection accuracy.

In future work, addressing these limitations is crucial. Expanding the dataset
size could enhance the model’s performance. Furthermore, we intend to extend
training to the entire model, encompassing not only the fully connected and
embedding layers. Experimenting with direct lyric extraction from audio,
especially from datasets like SHS, would facilitate meaningful comparisons
with other CSD methods. To tackle the context window limitation, research
efforts could focus on methods for handling larger lyrics, such as expanding
the context window or exploring alternative solutions like splitting lyrics
into smaller segments for more effective processing. It might also be possible
to partly merge our dataset with other datasets like SHS to increase the size
of the dataset.

##  5 Responsible NLP

In light of the current state of our methodology, it has been conclusively
ascertained that the implementation of a large-scale CSD system in a
professional environment is currently unfeasible. The metrics provided herein
illustrate specific circumstances wherein our methodology fails to produce
accurate results, thus introducing a potential risk of incorrect matches. Such
inaccuracies could potentially disrupt the fair distribution of royalties.
Nevertheless, there are potential solutions that could mitigate this risk. For
instance, imposing a distance limit and only considering automated matches
within close proximity could be a viable strategy. We acknowledge that
transformer-based models inherently possess biases that may facilitate more
effective matching of certain words or phrases over others. Although there is
no concrete evidence at present to suggest bias in our model, we unequivocally
pledge to comprehensively investigate this issue in our future research
endeavors. With respect to the reproducibility of our approach’s results, we
do not have concerns. Given that our model does not generate text and as such
does not possess a hyperparameter for output generation, our results are
reproducible. For the same given lyrics, the same embedding vector will be
produced, thereby yielding the identical original lyrics. In the development
of our model, we utilized GitHub Copilot. It is important to note that this
tool did not exert any influence on our research, and we carefully reviewed
the generated code. Additionally, we employed GPT-4 to check for spelling and
grammatical inaccuracies present in our manuscript and correct them.

## References

  * Dat (2011) 2011\.  Front Matter.  In Ian H. Witten, Eibe Frank, and Mark A. Hall, editors, _Data Mining: Practical Machine Learning Tools and Techniques (Third Edition)_ , The Morgan Kaufmann Series in Data Management Systems, pages i–iii. Morgan Kaufmann.
  * Sha (2023) 2023\.  Shazam.
  * pyt (2023) 2023\.  _WordErrorRate - Torch Eval_.  PyTorch.
  * Bachmann (2023) Max Bachmann. 2023.  Levenshtein.  https://github.com/maxbachmann/Levenshtein.
  * Bertin-Mahieux et al. (2011) Thierry Bertin-Mahieux, Daniel P.W. Ellis, Brian Whitman, and Paul Lamere. 2011.  The million song dataset.  In _Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR 2011)_.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642. Association for Computational Linguistics.
  * Chechik et al. (2009) Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. 2009.  Large Scale Online Learning of Image Similarity through Ranking.  In Helder Araujo, Ana Maria Mendonça, Armando J. Pinho, and María Inés Torres, editors, _Pattern Recognition and Image Analysis_ , volume 5524, pages 11–14. Springer Berlin Heidelberg.
  * Chicco (2021) Davide Chicco. 2021.  Siamese Neural Networks: An Overview.  In Hugh Cartwright, editor, _Artificial Neural Networks_ , Methods in Molecular Biology, pages 73–94. Springer US.
  * Christopher et al. (2008) Manning Christopher, Prabhakar Raghavan, and Hinrich Schütze. 2008.  _Introduction to Information Retrieval_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised Cross-lingual Representation Learning at Scale.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  Xnli: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Correya et al. (2018) Albin Andrew Correya, Romain Hennequin, and Mickaël Arcos. 2018.  Large-Scale Cover Song Detection in Digital Music Libraries Using Metadata, Lyrics and Audio Features.
  * Du et al. (2022) Xingjian Du, Ke Chen, Zijie Wang, Bilei Zhu, and Zejun Ma. 2022.  Bytecover2: Towards Dimensionality Reduction of Latent Embedding for Efficient Cover Song Identification.  In _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 616–620. IEEE.
  * Du et al. (2021) Xingjian Du, Zhesong Yu, Bilei Zhu, Xiaoou Chen, and Zejun Ma. 2021.  Bytecover: Cover Song Identification Via Multi-Loss Training.  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 551–555.
  * e. V. (2021) Bundesverband Musikindustrie e. V. 2021.  Musikindustrie in Zahlen im Überblick: BVMI - 2021.
  * Klakow and Peters (2002) Dietrich Klakow and Jochen Peters. 2002.  Testing the correlation of word error rate and perplexity.  _Speech Communication_ , 38(1):19–28.
  * Levenshtein (1996) V. I. Levenshtein. 1996.  Binary Codes Capable of Correcting Deletions, Insertions and Reversals.  10:707.
  * Müller et al. (2022) Thomas Müller, Guillermo Pérez-Torró, and Marc Franco-Salvador. 2022.  Few-Shot Learning with Siamese Networks and Label Tuning.  In _ACL (to appear)_.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
  * Osmalskyj (2017) Julien Osmalskyj. 2017.  A Combining Approach to Cover Song Identification.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022.  Robust speech recognition via large-scale weak supervision.
  * Royer (2001) Christian Royer. 2001.  _Simultane Optimierung von Produktionsstandorten, Produktionsmengen und Distributionsgebieten_.  Herbert Utz Verlag.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015.  FaceNet: A Unified Embedding for Face Recognition and Clustering.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 815–823.
  * Serrà Julià (2007) Joan Serrà Julià. 2007.  Music similarity based on sequences of descriptors tonal features applied to audio cover song identification.
  * Tsai et al. (2005) Wei-Ho Tsai, Hung-Ming Yu, and Hsin-min Wang. 2005.  A Query-By-Example Technique for Retrieving Popular Songs with Similar Melodies.  In _ISMIR 2005, 6th International Conference on Music Information Retrieva_ , page 190.
  * Vaglio et al. (2021) Andrea Vaglio, Romain Hennequin, Manuel Moussallam, and Gael Richard. 2021.  THE WORDS REMAIN THE SAME: COVER DETECTION WITH LYRICS TRANSCRIPTION.  In _22nd International Society for Music Information Retrieval Conference ISMIR 2021_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Vervielfältigungsrechte (2021) GEMA-Gesellschaft für musikalische Aufführungs-und mechanische Vervielfältigungsrechte. 2021.  Diskotheken-monitoring.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Xu et al. (2018) Xiaoshuo Xu, Xiaoou Chen, and Deshun Yang. 2018.  Key-Invariant Convolutional Neural Network Toward Efficient Cover Song Identification.  In _2018 IEEE International Conference on Multimedia and Expo (ICME)_ , pages 1–6.
  * Yu et al. (2008) Yi Yu, J. Stephen Downie, Fabian Moerchen, Lei Chen, and Kazuki Joe. 2008.  Using Exact Locality Sensitive Mapping to Group and Detect Audio-Based Cover Songs.  In _2008 Tenth IEEE International Symposium on Multimedia_ , pages 302–309.

  *[CSD]: Cover Song Detection
  *[WER]: word error rate
  *[SHS]: Second Hand Songs
  *[mAP]: mean average precision
  *[MR]: mean rank
  *[P@1]: precision at one


# Local Background Estimation for Improved Gas Plume Identification in
Hyperspectral Images

###### Abstract

Deep learning identification models have shown promise for identifying gas
plumes in Longwave IR hyperspectral images of urban scenes, particularly when
a large library of gases are being considered. Because many gases have similar
spectral signatures, it is important to properly estimate the signal from a
detected plume. Typically, a scene’s global mean spectrum and covariance
matrix are estimated to whiten the plume’s signal, which removes the
background’s signature from the gas signature. However, urban scenes can have
many different background materials that are spatially and spectrally
heterogeneous. This can lead to poor identification performance when the
global background estimate is not representative of a given local background
material. We use image segmentation, along with an iterative background
estimation algorithm, to create local estimates for the various background
materials that reside underneath a gas plume. Our method outperforms global
background estimation on a set of simulated and real gas plumes. This method
shows promise in increasing deep learning identification confidence, while
being simple and easy to tune when considering diverse plumes.

Index Terms—  Hyperspectral Imaging, Gas Plume Identification, Deep Learning,
Local Background Estimation

##  1 Introduction

Hyperspectral imaging (HSI) has become increasingly popular due to its
application in fields ranging from national security to medical research.
Longwave IR (LWIR) HSI has been shown to be particularly useful for the task
of gas plume detection and identification [1]. This process involves first
using a model to detect potential gas plumes followed by using a model to
identify the gas in the plume.

There are several well established methods for identifying gases, such as
Step-wise Linear Regression and Bayesian Model Averaging, which work well for
identifying gases from a relatively small library of candidate gases [2].
Machine learning and deep learning classification models have proven to be
superior for identification, particularly when there are many candidate gases
and many images to analyze [3, 4, 5]. However, applying deep learning
classifiers to real world data can be difficult, thus it is important to
properly preprocess the data before classification.

Whitening is a common preprocessing step when working with nadir observed gas
plumes. Whitening uses the equation
$\mathbf{z}=C^{-1/2}(\mathbf{x}-\mathbf{\mu})$, where $C$ is the scene
covariance matrix, $\mu$ is the scene average spectral signature, and
$\mathbf{x}$ is the spectrum of interest. Typically a “global” estimate of
$\mu$ and $C$ are used, where every pixel in the scene, except for pixels in
the region of interest (ROI), are included in the calculation. However, urban
images often contain many diverse background materials which are not
represented well by the global $\mu$. This can lead to mischaracterization of
important identifying wavelengths for the gas, making it more difficult to
classify correctly.

Since identification with deep learning models is already a difficult task on
real world plumes, it is crucial to have an accurate estimate of the whitened
gas signature. This paper proposes an iterative background estimation approach
with image segmentation to create local estimates of background spectra for
plumes. We demonstrate our approach to be beneficial in estimating gas
signatures in both simulated and real world plumes when compared to the global
whitening approach.

##  2 Method

A standard approach for whitening is to assume that all spectral signatures in
the image are normally distributed. The normality assumption is not guaranteed
to be satisfied, and in images with many different background materials, such
as in urban scenes, the global background may not sufficiently represent the
variety of background signatures [6]. Whitening with an incorrect background
signature can lead to inaccurate whitened gas spectra, making it more
difficult to classify the plume correctly. For this reason, it is beneficial
to whiten each part of a plume according to the distinct background materials
it spans within the scene.

One option to distinguish between various background materials is to use image
segmentation to break the image into groups of similar pixels. We use
Watershed Segmentation (WS) as it has an appropriate adaption for use with HSI
[7]. WS determines segments by finding boundaries between “different colored”
regions, thus it is well suited for the purpose of distinguishing between
various background materials. We choose to use non-marker based WS which
results in over-segmentation, reducing the chance that a given segment will
contain more than one distinct background material. Since some segments may
have a relatively small number of pixels, we focus on locally estimating the
means while using a globally estimated covariance matrix [8].

WS produces segments that are treated independently which can cause problems
for segments that are completely masked by a plume, as whitening with the
segment’s mean will remove important characteristics belonging to the gas. For
this reason, we need to be able to estimate the background of a contaminated
segment using non-contaminated pixels. Inspired by [9], we represent each
spectral signature $y$ with an additive model, $y=b+\alpha t$, where $b$ is
the background signature, $t$ is the target gas signature, and $\alpha$ is the
non-negative signal strength.

Given a set of pixels of the same background material, we want to estimate $b$
and $t$ which should be the same for all pixels in the set, and estimate
$\alpha$ for each individual pixel. One way to do this is by trying to solve
the following optimization problem:

$\min_{b,t,\alpha_{i}\geq 0}\frac{1}{N}\sum_{i}||y_{i}-(b+\alpha_{i}t)||^{2}.$
(1)

This optimization problem minimizes the mean squared error (MSE) between each
pixel and its modeled signature. An issue with trying to solve this
optimization problem is that $b$, $t$, and $\alpha_{i}$ are all unknown. Thus,
we opt to use an alternating approach to simplify the problem. Using Lagrange
multipliers, the solution at each step for $\alpha_{i}$, $b$, and $t$ are:

$\displaystyle\alpha_{i}$
$\displaystyle=\max\left(0,\frac{(y_{i}-b)^{T}t}{t^{T}t}\right),$ (2)
$\displaystyle b$ $\displaystyle=\frac{1}{N}\sum y_{i}-\alpha_{i}t,$ (3)
$\displaystyle t$
$\displaystyle=\frac{\sum(y_{i}-b)\alpha_{i}}{\sum\alpha_{i}^{2}}.$ (4)

We call this algorithm iBATE (iterative Background Alpha Target Estimation).
Natural initializations for $b$ and $t$ are the global estimates, and
$\alpha_{i}=1$. Each step can be repeated until the error reaches a limit, or
ceases to decrease; in nearly all of our test cases we observed convergence
within 10 iterations. Although iteratively solving these equations to minimize
the error will produce an “optimal” estimate for $b$, it will not produce a
good estimate if all of the pixels are contaminated by the gas. For this
reason, and the observation that most segments are small or are completely
masked by the gas, we need to incorporate additional pixels from other
segments that are of the same background material but that are NOT
contaminated by the gas. Once such pixels are found, the initializations and
optimizations are simplified since we can fix $\alpha_{i}=0$ for all non-plume
pixels, and we can average the non-plume pixels to create a clean initial
estimate of $b$.

To quantify segment similarity, we use linkage functions from Hierarchical
Clustering, which are used to find the “distance” between two collections of
objects. We propose the Truncated Average Linkage function (TAL), a
modification of the Average Linkage function, where a proportion of the
largest distances are removed [10]. TAL is defined as:

$TAL(A,B,\beta)=\frac{1}{N_{\beta}}\sum_{i=0}^{N_{\beta}}d(\vec{a},\vec{b})_{(i)},$
(5)

where $A$ and $B$ are two segments, $N_{\beta}=(1-\beta)|A||B|$, and
$d(\vec{a},\vec{b})_{(i)}$ is the $i^{\text{th}}$ smallest distance between
two pixels $\vec{a}\in A$ and $\vec{b}\in B$. The primary benefit of TAL is
that it is robust to comparisons between segments that have more than one
distinct background material. Additionally, as $\beta\to 0$, TAL becomes
Average Linkage, and as $\beta\to 1$, TAL becomes Single Linkage, thus
allowing us to compare the performance of three linkage functions at once.

TAL requires a distance function to compare two spectral signatures, of which
there are many options [11]. However, pixels in a plume segment are
contaminated with the gas, which will influence the distance calculated with a
non-contaminated pixel. We propose a distance function designed to be robust
to contaminated signatures called Truncated Euclidean Distance (TED), which
removes a proportion of wavelengths with the largest squared error. TED is
defined as

$TED(\vec{a},\vec{b},\gamma)=\sqrt{\sum_{i=0}^{n_{\gamma}}(a-b)^{2}_{(i)}},$
(6)

where $n_{\gamma}=(1-\gamma)c$, $c$ is the number of channels, and
$(a-b)^{2}_{(i)}$ is the channel with the $i^{\text{th}}$ smallest squared
difference. Because gas signatures have relatively few wavelengths that are
distinguishing, $\gamma$ can be chosen to account for these wavelengths,
allowing proper comparison between contaminated and non-contaminated
signatures.

We call the combination of the above methods LEBEAUS (Local itErative
Background Estimation, Additive, Using Segmentation), which provides the
ability to estimate various backgrounds for gas plumes in rural scenes. Though
LEBEAUS can be applied to any shape or size of plume, it is designed to be
most useful for plumes that are subtended by either several background
materials, or background materials that are not well represented by the global
background estimate.

##  3 Data & Results

Because it is difficult to collect HSI with ground truth, it is useful to be
able to simulate plumes where the ground truth is known. To implant a gas
signal into an individual pixel in a scene, we use the following equation:

$L=L_{bg}+\tau_{a}\left[B(T_{p}),-\epsilon_{bg}B(T_{bg})-\rho_{bg}L_{d}\right].$
(7)

For each pixel, $T_{bg},L_{d}$, and $\tau_{a}$ are given, while
$\epsilon_{bg},T_{bg}$, and $\rho_{bg}$ are estimated. A Gaussian plume
dispersion model [12] is used to simulate the concentration of the simulated
plume, which is then used to define the shape and temperature of the plume.
For temperature, the relative concentration is used to smoothly transition
from max stack temperature to ambient air temperature.

We simulated two large and dense plumes, one consisting of Sulfur Hexafluoride
(SF6) and another of Ammonia (NH3). SF6 will act as a baseline plume as it is
generally easier to identify compared to NH3. These plumes where implanted
into the scene over background pixels that were found to be notably different
from the global background signature. In addition to these two simulated
plumes, we also have two real plumes of SF6 and NH3. All scenes are LWIR
captures of the Los Angeles basin area with 128 sampled wavelengths from
$7.56-13.16\mu\text{m}$. Figure 1 shows the four plume ROIs with context from
the false colored image.

Fig. 1: Plume ROIs shown in red over false-colored context images. First row
are the simulated plumes, second row are the real plumes.

There are several hyperparameters that will impact the performance of the
local whitening process. They are the TEDγ (range 0-100%), TALβ (range
0-100%), iBATE, and $\min K$ (from 1 to 1024) which is the minimum number of
additional non-plume pixels to include during the iBATE process. Note that if
iBATE is off then the initial local background estimate is used, i.e. the
average of the most similar non-plume pixels found using TAL and TED. It would
be ideal if the optimal values for all these hyperparameters were roughly
equal across the different plumes, as this would mean hyperparameters do not
need to be tuned on a per plume basis. We find the optimal values for each
plume by conducting hyperparameter searches. For the simulated plumes, we
search for parameters that minimize the MSE between the locally whitened plume
and the whitened ground truth signal. For the real plumes, we find parameters
that maximize the prediction confidence of our deep learning identification
model for the correct gas.

We use Optuna to conduct a grid search to explore the range of the
hyperparameter space (54 trials), followed by a Tree-structured Parzen
Estimator (TPE) search to narrow down to the optimal hyperparameters (74
trials) [13]. Figure 2 shows the results of the hyperparameter searches, and
the best results are summarized in Table 1. There are several conclusions we
can make from these figures. Looking at the first column, we see that the
optimal TEDγ is about 20% for each plume. The general trend shows that a
smaller TEDγ is optimal, while using TED${}_{\gamma}=0$, which is equivalent
to Euclidean distance, is least optimal.

Fig. 2: Hyperparameter search slice plots. The orange x marks the optimal hyperparameters, and the horizontal line marks the baseline performance from global whitening. The first two rows are the simulated gases where lower MSE is preferred. The last two rows are the real plumes, where higher confidence is preferred. | Simulated (MSE) | Real (Conf %)
---|---|---
Method | $\text{SF}_{6}$ | $\text{NH}_{3}$ | $\text{SF}_{6}$ | $\text{NH}_{3}$
Global | 0.00454 | 0.06932 | 90.00% | 62.06%
LEBEAUS | 0.00282 | 0.03423 | 91.62% | 81.87%
Table 1: Best performance for each plume and method. Lower MSE is preferred
for the simulated plumes, and higher confidence is preferred for the real
plumes.

The second column shows the optimal TALβ. For the simulated plumes, the clear
trend is that a high TALβ is better than a small TALβ. In other words, Single
Linkage is better than Average Linkage. Though the real SF6 plume prefers
TAL${}_{\beta}=50\%$, generally a larger $\beta$ is preferable to a smaller
$\beta$. The most important takeaway is that the optimal values are similar
across all four plumes.

The third column shows optimal values for $\min K$, and there is a clear
trend: including fewer pixels consistently outperforms including many
additional pixels for iBATE. This suggests that if too many additional
segments are considered, the diversity of the segments increases leading to a
worse estimate of the specific local background.

Lastly, the fourth column shows that iBATE typically does not perform
significantly worse than not using iBATE, and that in some situations it
offers large improvements (e.g. row 3 of Figure 2). From this small sampling
of plumes, most hyperparameters outperform global whitening and there are
similar optimal hyperparameters across the different plumes. This suggests
that for a variety of plumes, little tuning will be required to obtain
satisfactory performance.

##  4 Conclusions

Machine learning for automatic plume detection and identification is becoming
more important as the number of images, the resolution of images, and the
library of candidate gases increase. Since real world plumes in urban scenes
typically cover several background materials, it is invaluable to
appropriately characterize each background material. With appropriate
background estimates, gas spectra can be better estimated, leading to
increased confidence when models or spectral analysts identify the gas.

Our LEBEAUS method is able to locally estimate each different background
material under a plume. Using image segmentation to break the image into
different background materials, identifying sets of similar segments, then
iteratively estimating the local background signatures, we demonstrate
LEBEAUS’s superiority compared to standard global whitening on both simulated
and real plumes. Furthermore, we find that the optimal hyperparameters are
similar across both simulated and real plumes, and among plumes of different
gases. This suggests that a set of “universal” default hyperparameters can be
applied to any plume and it will likely outperform global whitening.

Several important studies are needed to understand the robustness and
generalizability of these results. First, robust and optimal distance
functions for clustering segments, as well as alternative linkage functions
and segmentation methods, exist that may further improve iterative background
estimates. Second, expanding the study to a greater variety of gas species and
plume morphologies is needed to determine how well the iterative method
converges with variation in parameters such as plume size, gas concentration,
and background heterogeneity.

## References

  * [1] Muhammad Jaleed Khan, Hamid Saeed Khan, Adeel Yousaf, Khurram Khurshid, and Asad Abbas,  “Modern trends in hyperspectral image analysis: A review,”  IEEE Access, vol. 6, pp. 14118–14129, 2018.
  * [2] Dimitris G. Manolakis, Steven E. Golowich, and Robert S. DiPietro,  “Long-wave infrared hyperspectral remote sensing of chemical clouds: A focus on signal processing approaches,”  IEEE Signal Processing Magazine, vol. 31, pp. 120–141, 2014.
  * [3] Shutao Li, Weiwei Song, Leyuan Fang, Yushi Chen, Pedram Ghamisi, and Jón Atli Benediktsson,  “Deep learning for hyperspectral image classification: An overview,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 9, pp. 6690–6709, 2019.
  * [4] Natalie Klein, Adra Carr, Zigfried Hampel-Arias, Allison Zastrow, Amanda Ziemann, and Eric Flynn,  “Hyperspectral target identification using physics-guided neural networks with explainability and feature attribution,”  IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, pp. 946–949, 2023.
  * [5] Utsav B. Gewali, Sildomar T. Monteiro, and Eli Saber,  “Machine learning based hyperspectral image analysis: A survey,” 2019.
  * [6] Stefania Matteoli, Marco Diani, and James Theiler,  “An overview of background modeling for detection of targets and anomalies in hyperspectral remotely sensed imagery,”  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 7, no. 6, pp. 2317–2336, 2014.
  * [7] Yuliya Tarabalka, Jocelyn Chanussot, and Jón Atli Benediktsson,  “Segmentation and classification of hyperspectral images using watershed transformation,”  Pattern Recognit., vol. 43, pp. 2367–2379, 2010.
  * [8] Scout C. Jarman, Tory Carr, Zigfried Hampel-Arias, Eric Flynn, and Kevin Moon,  “Ensemble segmentation for improved background estimation and gas plume identification in hyperspectral images,”  Applications of Machine Learning 2023, 2023.
  * [9] James Theiler and Amanda Ziemann,  “Local background estimation and the replacement target model,”  in Defense + Security, 2017.
  * [10] Frank Nielsen,  Hierarchical Clustering, pp. 195–211,  Springer International Publishing, Cham, 2016.
  * [11] Hilda Deborah, Noël Richard, and Jon Yngve Hardeberg,  “A comprehensive evaluation of spectral distance functions and metrics for hyperspectral image processing,”  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 8, pp. 3224–3234, 2015.
  * [12] N.S. Holmes and L. Morawska,  “A review of dispersion modelling and its application to the dispersion of particles: An overview of different dispersion models available,”  Atmospheric Environment, vol. 40, no. 30, pp. 5902–5928, 2006.
  * [13] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama,  “Optuna: A next-generation hyperparameter optimization framework,”  Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019.