# LyricSIM: A novel dataset and benchmark for similarity detection in Spanish
song lyrics††thanks: Accepted to Congreso Internacional de la Sociedad
Española para el Procesamiento del Lenguaje Natural 2023 (SEPLN2023).

Alejandro Benito-Santos1, Adrián Ghajari2, Pedro Hernández3, Víctor Fresno2
Salvador Ros2, Elena González-Blanco3
1Universidad de Salamanca
2Universidad Nacional de Educacion a Distancia
3IE University
<EMAIL_ADDRESS>

##  1 Introduction

The success of music streaming services is mainly based on the tailor-made
playlist they offer to users based on their listening habits and 54% of
consumers say playlists are replacing albums in their listening
habits111https://midiaresearch.com/blog/music-subscriber-market-shares-2022.
To build those lists and fit them to the user’s preferences, music streaming
providers have developed recommender systems, which provide personalized
suggestions based on the user´s behavior and some specific parameters related
to the musicality of a song [Fell,2020], oftentimes overlooking the song
lyrics.

A song is comprised of two distinct components: the music and the lyrics. The
lyrics consist of text, typically presented in poetic form, and describe the
artist‘s message and embody a unique combination of linguistic, artistic, and
cultural elements, making them a special expression of writing that contains
different features that could help us to improve song recommender systems. The
lyrics thus provide a rich source of unstructured data and qualitative
information that is not captured by metadata traditionally used in song
recommendation such as beat, tempo, pitch, instrument, or mood (acoustic).

Due to their unique writing style, the song lyrics present distinct challenges
for semantic similarity modeling, a task that has traditionally been evaluated
in the SemEVal tasks that were organized between 2012 and 2017 [Agirre et
al.,2012, Agirre et al.,2013, Agirre et al.,2014, Agirre et al.,2015]. This
evaluation framework operates under the assumption that a model that performs
well for the general STS task is also likely to perform well for tasks that
are tailored towards specific applications. However, there has been a
significant discrepancy between the performance of models in STS and their
performance in specific STS-based tasks such as MT Metrics (MTM) or Passage
Retrieval (PR) [Abe et al.,2022]. We argue existing STS-based
evaluations222http://nlpprogress.com/english/semantic_textual_similarity.html
may not effectively capture their nuances and specificities, leading to
suboptimal performance in lyric-related tasks such as recommendation, search,
and cultural analysis, which hinders research and development efforts in this
area. In addition, there is currently a gap in the availability of benchmark
datasets that are specific to the domain of song lyrics [Chandrasekaran y
Mago,2022].

Therefore, a dataset of song lyrics annotated for similarity would not only
facilitate the development of more accurate and applicable semantic similarity
models but also enable a deeper understanding of the relationships between
lyrics and music in this rich and diverse linguistic context. Since Spanish as
a language contains a rich cultural and linguistic diversity and the Latin
music industry represents a significant portion of the global music market,
with a vast array of genres and styles, ranging from traditional folk music to
contemporary pop, rock, or urban styles, the lyrics songs in Spanish offer
unique challenges and opportunities for the exploration of semantic similarity
in the domain of music.

In addition, the development of a dataset comprising Spanish song lyrics
annotated for similarity tailored to the Spanish language and culture is
crucial to advance the understanding and modeling of semantic similarity in
the context of music and lyrics within the Spanish-speaking world.

In this paper, we introduce LyricSIM, a novel dataset for Spanish song lyric
similarity designed to address the domain-specific characteristics of lyrics
and facilitate the development of more accurate and applicable semantic
similarity models for this domain. Our dataset comprises a diverse collection
of paired song lyrics in Spanish, annotated with similarity scores based on
various aspects such as theme, message, emotions, literal meaning, and
cultural context.

On the other hand, we also have assessed the performance of various state-of-
the-art models on our dataset for the semantic similarity task taking into
account the unique features of song lyrics. The obtained results make up a new
benchmark for the semantic similarity tasks based on the lyrics of songs.

By developing a dataset specifically tailored to song lyric similarity and
assessing the performance of SOTA models over this dataset, this paper aims to
bridge the gap between general-purpose semantic similarity tasks and domain-
specific applications, ultimately contributing to the advancement of NLP
research in the context of music and lyrics research and analysis.

##  2 Related Work

To the best of our knowledge, no benchmark datasets currently exist for the
study of semantic similarity between song lyrics. Although datasets containing
song lyrics are available in different languages, they do not include semantic
similarity annotations between song pairs, only collecting the lyrics. This
information makes our dataset a very good fit to design models aimed at
identifying plagiarism or detecting unlicensed versions of copyrighted songs,
among others.

The benchmark dataset we can consider most closely related to ours is the
4MuLA dataset introduced in [da Silva, Silva, y Marcacini,2020], which
contains structured information that can be applied to several tasks. This
dataset is obtained from a lyrics-focused platform and includes additional
user-provided annotations. It includes

Latin music genres that are underrepresented in other benchmark datasets. The
dataset provides acoustic features, extracted tags, and lyrics in English,
Portuguese, or Spanish, making it suitable for lyrics-, audio-, or multimodal-
based genre classification, music and artist similarity, and popularity
regression. Furthermore, the lyrics in the dataset can be used for cross- or
multilingual text analysis, such as discourse analysis or measuring the
differences between emotion transmitted by audio and lyrics. However, it lacks
annotations about the semantic similarity between the song lyrics, which is
what the benchmark dataset we propose provides.

Given that song lyrics often share similarities with poetry more than other
literary genres, it stands to reason that poetry evaluation collections could
be leveraged in the study of similarity between song lyrics, even when
accounting for the differences between songs and poems.

In [Li et al.,2021], a dataset was proposed in order to evaluate the semantic
understanding of poetry models through poem matching. The objective was to
advance research efforts focused on integrating deep semantics into the
generation and comprehension system of Chinese classical poetry. In [Haider et
al.,2020] the annotation with experts leaded to an agreement of kappa = .70,
resulting in a dataset for large scale analysis. The authors conducted first
emotion classification experiments based on BERT, showing that identifying
aesthetic emotions was challenging.

##  3 Annotation Task

Given that this was our first attempt at characterizing similarity in song
lyrics, and that we wanted to obtain a broad vision of the problem, we decided
to limit the number of annotators to three per pair and to favor diversity in
the song lyrics. Through a crowdsourcing platform, participants were chosen
from a pool of a total of 63 annotators who took part in the study. They were
asked to rate the similarity between pairs of lyrics of Spanish songs using a
six-point semantic differential scale ranging from 0, for completely different
items, to 5 for outstandingly similar items (see details in Section 3.2). They
were instructed to evaluate the similarity of pairs of song lyrics based on
various criteria, such as the primary theme or context of the lyrics, the
message conveyed, the emotions or feelings expressed, the literal meaning, the
vocabulary employed, the relationship between the sender and receiver, the
language style, and the sociocultural context of the song (see Annex 1 and 2
for a complete task description in Spanish and a translation into English,
respectively). Participants were advised to follow their intuition if they had
doubts or if the instructions were insufficient to provide a response. The aim
of this task was to obtain a comprehensive dataset of similarity annotations
for Spanish song lyrics, taking into account multiple dimensions of lyrical
content.

###  3.1 Dataset Description

The dataset we prepared for the annotation task contains 75 song lyrics in
Spanish that were selected for their diversity and popularity and for
representing a wide range of music genres and themes. Also, we included song
lyrics of varying lengths ($M=77.61,SD=34.88$, see histogram in Figure 1), to
check whether this variable had any influence in the participants’ perception
of similarity.

Figure 1: Histogram of song lyrics length in the dataset used to conduct the
annotation task. The lengths were approximately normally distributed with a
mean length of $77.61$ and a standard deviation of $34.88$.

Although pairs were randomly assigned from the pool of 75 songs, we ensured
that enough pair combinations with disparate lengths were paired together. A
representation of these pairs can be seen in Figure 2.

Figure 2: Scatterplot showing composition of pairs in the filtered dataset.
The circles represent the size of combination cases (axes X and Y). The size
of the circle depicts the number (in white) of particular cases matching
lyrics of length $X$ to lyrics of length $Y$. The color scale represents the
average of mean scores for items in that

###  3.2 Semantic Differential Scale

We employed a 6-point numeric scale in our annotation experiment, similar to
the one that was conceived for the SemEval tasks [Agirre et al.,2012]. It is
worth noting that, as in the original SemEval tasks, our scale devotes one
level to total dissimilarity (level 0), and five other different grades to
capture a subtler range of semantic similarity (levels 1-5) in an increasing
order of intensity. However, we changed the wording of the categories to fit
the broader context of similarity between song lyrics. A description of the
full scale is provided below:

  * •

Completely different (0): the lyrics are entirely dissimilar.

  * •

Barely any similarity (1): the lyrics share minor aspects without semantic
importance, such as language style or sociocultural context.

  * •

Little similarity (2): there is no semantic similarity (lyrical situation,
message, feelings), but the lyrics can be considered thematically (literal
meaning) related.

  * •

Basic similarity (3): the lyrics resemble each other in message, feelings of
the protagonist/singer, lyrical situation, or literal meaning.

  * •

Notable similarity / missing details (4): the lyrics share the same message
and feelings but differ in lyrical situations and/or literal meaning.

  * •

Outstanding similarity (5): the lyrics share the same message, emotions,
intentions, and lyrical situation, differing only in lexicon and genre.

Upon collection of the results, we obtained more 8,325 pair-wise similarity
values corresponding to the comparison of 2775 pairs by three different
participants each. An overview of the collected annotations can be seen to the
left of Figure 3.

Figure 3: Comparison of the number of labels before (left, in blue) and after
(right, in red) applying the filtering criteria. Although the difference in
size between the two datasets is notable (approximately 75% of the original
annotations were discarded in the process), the filtered dataset contains only
high-quality annotations.

##  4 Data Refinement

To maximize cost-effectiveness, we did not set any restrictions on the number
of pairs that could be annotated by each participant, which resulted in an
unbalanced distribution of annotation authoring. To mitigate this potential
bias, we took the necessary steps to ensure that the resulting dataset
included annotations that could be used as ground truth in future studies.
This meant that we kept only those pairs in which a high degree of consensus
(see 4.2) between the three annotators could be established.

Figure 4: Comparison of rating distributions between the STS datasets and
LyricSIM via kernel density estimation (KDE). In both cases, despite
differences in the scale interpretation, the distributions are positively
skewed, with STS datasets showing a higher bias towards the first point of the
scale.

###  4.1 Refinement

In order to obtain high-quality annotations, we filtered the collected data
using different criteria aimed at reducing rating variability. To model and
filter the annotation data, as in the STS Core tasks, we departed from the
assumption that annotations of similarity (scores from 1 to 5) are
fundamentally different from those of dissimilarity (0). This effect can be
seen in Figure 4, which shows the kernel density estimations (KDEs) of rating
distributions in the STS and LyricSIM datasets. The chart reveals that in all
cases, ratings are biased towards the inferior ratings of the scale, specially
in the STS data (around 50% probability of seeing a rating of 0). Thus, we
relied on the observation that the probability of seeing a dissimilar pair was
hypothetically much higher, and consequently we made a clear distinction
between dissimilar and similar items in an attempt to further characterize the
tail of the distribution.

For dissimilarities, we take only those pairs in which all three annotators
agreed that the pair was dissimilar (i.e., all annotators assigned a score of
0 to the pair), giving out a total of 837 pair-wise dissimilarities. In case
of similarities, we aimed to capture pairs in which 2 out of 3 annotators
agreed exactly on the same score. To avoid including doubtful cases, we chose
to exclude pairs in which the third annotator assigned a very different score
(i.e., 2 or more points apart from the mode), resulting in the selection of
676 high-quality similarity pairs or 24.36% of the original dataset. In total,
the refined dataset contains 2,028 high-quality annotations of pairwise
similarity and dissimilarity judgments for a total of 75 distinct song lyrics.
The 676 high-quality pairs are compared side by side to the unfiltered data in
3, providing a visual estimation of the reduction in size per rating label in
the refined dataset. This same data can be found in table format in Table 1.

dataset | rating | count | percent
---|---|---|---
Original | 0 | 3058 | 36.73%
1 | 3014 | 36.20%
2 | 1058 | 12.71%
3 | 746 | 8.96%
4 | 347 | 4.17%
5 | 102 | 1.23%
Filtered | 0 | 837 | 41.27%
1 | 705 | 34.76%
2 | 360 | 17.75%
3 | 88 | 4.34%
4 | 34 | 1.68%
5 | 4 | 0.20%
Table 1: Number of ratings in the original and filtered datasets.

###  4.2 Reliability

To further assess the reliability of our dataset, we calculated inter-
annotator agreement using Krippendorff’s reliability alpha ($\alpha_{k}$),
which gave a value of 0.90 (in the unrefined data, it was 0.27).
Krippendorff’s reliability alpha is a metric that generalizes other metrics
that are responsible for quantifying the reliability between annotators
(inter-rater reliability). It can be used for both ordinal and nominal
annotations, as well as with any number of annotators. K-alpha yields a value
between 0 and 1, where 1 represents full agreement. However, there are
different criteria regarding when to consider that there is enough agreement
between annotators. According to the general consensus, a common threshold is
to consider that there is enough agreement when K-alpha is greater than 0.8.
Using this metric, the resulting inter-annotator agreement of the high-quality
annotations dataset was found to be substantial, with a coefficient alpha of
0.90, indicating that the dataset is reliable and thus can be used for future
research in the field. Table 2 shows a comparison between STS gold standard
datasets used in the SemEval 2014 STS Core task [Agirre et al.,2014] and ours,
including dataset size (in sentence pairs), number of annotators per pair and
computed Krippendorff’s alpha-reliability scores [Krippendorff,2004], when
possible. Unfortunately, we could not find the results of the STS Spanish
tasks disaggregated by annotator. Thus, we derived our results from the
English tasks data (available at
http://ixa2.si.ehu.es/stswiki/images/2/21/STS2015-en-rawdata-scripts.zip).

Dataset | Size | Avg.Length | K-
---|---|---|---
STS-images | 1500 | $9.6\pm 3.04$ | 0.82
STS-students | 1500 | $10.44\pm 3.34$ | 0.72
STS-headlines | 2000 | $7.5\pm 2.24$ | 0.79
STS-belief | 2000 | $13.01\pm 6.83$ | 0.63
STS-forums | 1500 | $15.04\pm 3.28$ | 0.66
Avg. STS |  |  | 0.72
LyricSIM | 676 | $77.61\pm 34.88$ | 0.90
Table 2: Comparison between STS datasets and LyricSIM displaying dataset size
(in pairs), average sentence/utterance length and calculated Krippendorff
alpha (K-alpha). Text sizes in LyricSIM are considerably larger than their STS
counterparts and K-alpha varies greatly in the STS datasets, ranging from 0.63
(min) to 0.82 (max).

##  5 Evaluation

In this section, we use the refined dataset introduced in the previous section
to analyze how SOTA language models perform in the similarity detection task.
We have followed a similar 85-5-10 split as the one used in other studies
[Gutiérrez-Fandiño et al.,2022, Agerri y Agirre,2023], resulting in 638 song
pairs for the train set, 38 song pairs for the development set and 68 song
pairs for the test set. To ensure balanced representation of each class, we
used stratified sampling during the splitting process. The metrics used for
model assessment were Spearman’s Rank Correlation and Pearson’s Correlation
[Cer et al.,2017, Devlin et al.,2019]. For the purpose of creating a reference
point for our dataset’s behavior when subjected to traditional, less complex
models, a Support Vector Machine (SVM) model was also trained. Further, to
obtain a more holistic measure of the models’ performance, a combined score
was also calculated, computed by taking the arithmetic mean of both metrics
[Gutiérrez-Fandiño et al.,2022] [Agerri y Agirre,2023].

###  5.1 Language Models

We present a brief overview of the models that were selected for evaluation
and discuss their respective training and architectural characteristics. By
exploring these models in greater depth, we can better understand the nuances
of their performance on the refined dataset, ultimately allowing us to make
informed decisions regarding their suitability for similarity detection tasks
in the Spanish language.

The language model architectures we have selected are BERTIN, RoBERTa-base-bne
(MarIA base), RoBERTa-large-bne (MarIA large), Sentence Transformer, ALBERTI,
DeBERTa, XML-RoBERTa base and XML-RoBERTa large. There are several reasons
that led us to choose these five models for our study. First, we aimed to
include a significant representation of both monolingual Spanish models, as
our dataset comprises Spanish song lyrics, and multilingual models, which have
demonstrated superior performance over monolingual models in Spanish tasks.
Additionally, we selected both the base and large versions of these models to
analyze the resulting metrics after training.

Our selection was influenced by an article that evaluated Spanish Language
Models[Agerri y Agirre,2023]. The models we chose exhibited high performance
in terms of their STS official combined scores, as shown in Table 2 of the
referenced article. BERTIN and MarIA have become prominent monolingual models
for the Spanish language, as they have been trained on extensive Spanish
datasets. BERTIN was trained on the Spanish portion of mC4, which contains
approximately 416 million documents and 235 billion words in 1TB of
uncompressed data, along with other datasets such as Wikipedia, OpenSubtitles,
and Europarl. MarIA was trained on a 570GB corpus of clean and deduplicated
texts extracted from the Spanish Web Archive, built by the National Library of
Spain between 2009 and 2019.

Sentence Transformer was chosen due to its core focus on STS tasks during
training, which is highly relevant to our study. ALBERTI was selected because
it is domain-adapted for poetry, a semantic context closely related to song
lyrics. The reasons for choosing DeBERTa, XML-RoBERTa base, and XML-RoBERTa
large models are manifold. DeBERTa is selected due to its superior training
scores compared to RoBERTa and BERT, incorporating cutting-edge techniques
such as Disentangled Attention and Enhanced Mask Decoder, which enhance its
performance. XML-RoBERTa base and XML-RoBERTa large models are chosen for
their extensive parameter counts, with the base version comprising 270M
parameters and the large version containing 550M parameters. Additionally,
these models are multilingual, trained on 100 languages, making them highly
versatile and capable of handling diverse language tasks.

In the following, we provide an overview of these models representing both
monolingual and multilingual approaches. All these models were trained on
various corpora with distinct architectures and training parameters which we
discuss in the following section.

  * •

BERTIN [Rosa et al.,2022] employs a novel technique called ”perplexity
sampling” for pre-training Spanish language models. This method reduces the
amount of data and training steps needed compared to traditional approaches,
while still achieving competitive results. BERTIN utilizes a two-step training
process with different sequence lengths and batch sizes.

  * •

The MarIA [Gutiérrez-Fandiño et al.,2022], RoBERTa-base-bne and RoBERTa-large-
bne models [Liu et al.,2019], are state-of-the-art NLP models for the Spanish
language. They have been trained on a massive corpus of Spanish text data
derived from the National Library of Spain’s selective crawls. The MarIA
models employ a single-epoch training approach with no dropout, focusing on
tasks such as sentiment analysis, part-of-speech tagging, and named entity
recognition.

  * •

Sentence-Transformer [Reimers y Gurevych,2019] is a modification of pre-
trained BERT and RoBERTa networks that produce semantically meaningful
sentence embeddings. It uses a siamese and triplet network structure, reducing
computational overhead while maintaining accuracy.

  * •

ALBERTI333https://huggingface.co/flax-community/alberti-bert-base-
multilingual-cased is a BERT-based multilingual model trained on poetry
datasets, including Spanish resources. Although no publication detailing the
corpus and training methodology exists, the model leverages domain adaptation
to capture patterns and features specific to poetry.

  * •

DeBERTa [He et al.,2021], Decoding-enhanced BERT with Disentangled Attention,
refines BERT and RoBERTa through disentangled attention and superior masked
decoding. The model’s virtual adversarial training enhances generalization,
improving efficiency in pre-training and performance in NLU and NLG tasks. An
expansive DeBERTa version surpasses human performance on the SuperGLUE
benchmark, a notable achievement in macro-average score.

  * •

XML-RoBERTa [Conneau et al.,2020] enhances cross-lingual understanding (XLU)
through extensive study of unsupervised cross-lingual representations.
Introducing XLM-R, a transformer-based multilingual model pre-trained on text
in 100 languages, achieving state-of-the-art results in cross-lingual
classification, sequence labeling, and question answering. The authors delve
into key factors and trade-offs between positive transfer, capacity dilution,
and performance across languages. They demonstrate that a single large model
can effectively encompass all languages without sacrificing per-language
performance.

###  5.2 Training Parameters Details

In this section, we delve deeper into the training parameters employed by the
monolingual and multilingual models, highlighting the similarities and
differences that may contribute to their respective performance in the
similarity detection task.

We want to compile the training parameters in a list format, which we have
obtained after reading the papers of each of these models. These parameters
serve as a starting point for replicating and investigating the results
obtained by the research and development teams who worked on each of these
models. The training parameters for the selected models reveal substantial
differences in the pre-training procedures employed by the monolingual and
multilingual models. For example, BERTIN follows a two-step training process
with varying sequence lengths and batch sizes, while the MarIA models employ a
single-epoch training approach with no dropout. These variations in training
procedures and corpora used for pre-training contribute to the distinct
performance characteristics of each model. As researchers continue to develop
and evaluate Spanish language models, it is crucial to assess how these
differences in training parameters impact the models’ effectiveness in
downstream tasks. The selected Spanish language models offer valuable insights
into the development and application of transformer-based language models for
the Spanish language.

###  5.3 Fine-tuning

The fine-tuning process was carried out following the same practices found in
the aforementioned studies. We used the same scripts as the MarIA
team444https://github.com/PlanTL-GOB-ES/lm-spanish [Gutiérrez-Fandiño et
al.,2022], which are based on HuggingFace Transformers library [Wolf et
al.,2020], with minor modifications to adapt them to our dataset structure. To
maintain consistency across the models, we initialized each one of them with a
random head and employed a fixed seed for reproducibility. We conducted a grid
search over the following search space:

  * •

Weight decay: 0.1, 0.01

  * •

Learning rate: 1e-5, 2e-5, 3e-5, 5e-5

  * •

Batch size: 8, 16, 32

Due to memory constraints, especially for larger models, in cases where the
batch size exceeded capacity, gradient accumulation was used achieving the
same effective batch size; the rest of the hyperparameters remain the same as
the HuggingFace defaults. The maximum sequence length was 512 tokens for all
models, chosen to accommodate all sentence pairs in the dataset. To prevent
overfitting, we trained each model for a maximum of 5 epochs with Adam
optimizer and a linear decaying learning rate, selecting the checkpoint with
the highest score according to development set. Finally, we perform evaluation
on the test set for each model, with the best checkpoint from the previous
step. The optimal configuration of hyperparameters per model is shown in Table
5.

###  5.4 Results

The results are shown in Table 3 for each model, fine-tuned as described.
Among the eight models investigated, MarIA large demonstrated the highest
performance in terms of the combined score metric, while MarIA’s base
counterpart ranked fourth; mDeBERTa model closely followed MarIA large,
exhibiting competitive results. The XLM-RoBERTa large and base models are
fifth and sixth respectively, showing again a larger model performing better
than its base variant, while Sentence Transformer based on the same
architecture performed better than both, achieving third place. ALBERTI, the
only BERT based model of this study, was only slightly behind some of the
other models, placing seventh. BERTIN, although not dramatically behind the
SVM, placed last. These findings suggest that the MarIA large model is the
most effective in capturing semantic textual similarity on songs, with
mDeBERTa as a strong contender.

The results for the evaluation on the development set, used as criteria for
selecting the best checkpoint from the fine-tuning process and hyperparameter
selection, can be found in Table 4 of the appendix. A comparative analysis of
the development and test set results may provide further insights into the
generalization capabilities of the models and the effectiveness of the fine-
tuning process.

model name | STS combined | combined | spearmanr | pearson
---|---|---|---|---
BERTIN | 79.45 | 85.72 | 86.32 | 85.12
MarIA large | 84.11 | 90.02 | 89.94 | 90.11
MarIA base | 85.33 | 86.75 | 89.02 | 84.5
XLM-RoBERTa base | 83.47 | 86.45 | 88.19 | 84.72
XLM-RoBERTa large | 84.04 | 86.74 | 88.15 | 85.33
Sentence Transformer XLM-R | - | 88.91 | 89.55 | 88.28
mDeBERTa3 | 83.61 | 89.15 | 89.07 | 89.23
ALBERTI | - | 86.35 | 88.29 | 84.42
SVM (RBF) | - | 86.18 | 86.57 | 85.78
Table 3: Test combined scores for all the models considered (best in bold).
STS dataset results from other studies [Agerri y Agirre,2023] have been added
for illustration purposes.

##  6 Discussion

In this section, we will delve into the results obtained from the data
refining process and the fine-tuning of the baseline models, highlighting the
insights and implications of these outcomes for our understanding of semantic
similarity in Spanish song lyrics.

###  6.1 Evaluation Results

It is noteworthy that the highest performing model, MarIA large, is a Spanish-
specific model. This observation suggests that models trained on a particular
language may have an advantage in capturing the subtleties of that language
for this particular task, although multilingual models can still achieve
competitive performance. Further studies are recommended to provide a more
comprehensive understanding. Furthermore, our results validate that larger
models (e.g. Maria large) lead to improved performance compared to their base
counterparts. This result emphasizes the need to factor in model size and
computational demands when choosing a model for practical applications.

Another interesting observation is the difference in performance between the
BERT-based model, ALBERTI, and the other RoBERTa and XLM-RoBERTa-based models.
Surprisingly, even though this model was trained on poems, which exhibit a
structure more closely related to songs than the data used to train the other
models, performs relatively worse. A possible avenue for exploration would be
the tokenization process used by this model, as it does not take into account
line-break characters (\n) commonly used to delimit verses and stanzas, which
in turn could impact the model’s ability to learn the structure of a song.
This also accentuates the importance of considering the structure and
formatting of the input data when selecting a model for a specific task.

Similarly, the advantage in performance of Sentence Transformer, which is
based on the XLM-RoBERTa architecture, in comparison to the other XLM-RoBERTa
models may be partially attributed to the supplementary fine-tuning process
applied to sentence pairs for semantic textual similarity. We argue this fine-
tuning might have the potential to refine the model’s ability to capture
semantic relationships more effectively, which would explain this observation
in the data.

The results also revealed that the performance of the SVM model, a
comparatively simpler machine learning architecture, was not significantly
different from some of the more complex transformer-based models (e.g.
BERTIN). This could potentially indicate that the dataset size might not be
large enough or sufficiently complex to highlight the strengths of transformer
models.

These findings have practical implications for the development of real-world
applications related to the task at hand: the performance of the MarIA large
and mDeBERTa models suggests that they may be well-suited for tasks such as
song recommendation, lyric analysis, or music information retrieval.

###  6.2 Similarity vs Length

In addition to the findings discussed earlier, an interesting observation that
we could derive from the annotation data was that similarity scores increased
with the total length of the lyrics in a pair (see Figure 5), suggesting that
longer lyrics may contain more opportunities for shared vocabulary and
thematic elements to contribute to overall semantic similarity. Although more
research is needed in this area, current evidence points to the potential
utility of considering vocabulary overlap as an important factor in
determining semantic similarity between pieces of text [Abe et al.,2022]. As
we discuss in the next section, there is a currently a lack of examples of
annotated pairs exhibiting varying degrees of similarity (low, medium or
high), which we aim to resolve in future studies.

Figure 5: X axis: word sum of pairs in the annotation dataset. Y axis: average
similarity rating with 95% CI error bars. Despite disparity in sample size
across lengths, in our experiment, the perceived similarity in song lengths
increased with the lyrics length, indicating a possible interaction effect.

##  7 Conclusion and Future Work

In this study, we have laid the groundwork for assessing semantic similarity
in the context of Spanish song lyrics by presenting a dataset and baseline
results from pre-trained SOTA models. Our results provide insights into the
performance of these models, revealing potential strengths and weaknesses as
well as opportunities for future research. Beyond that, we have provided a
reflection on the data collection process typical of the similarity annotation
task, and a detailed characterization of the filtering process that we
followed to increase the quality of the annotation data. Finally, we provide
all the code and data necessary to reproduce our research at the repository
located at https://github.com/linhd-postdata/lyricsim.

To further advance this research area, during the course of our research we
identified several avenues for future work that can be pursued. One possible
direction is expanding the dataset to make it multilingual by incorporating
song lyrics from different languages and cultures. Furthermore, it is
essential to investigate the appropriateness of the psychometric scales
typically used in similarity annotation studies, as different scales or
measurement techniques could prove more effective in capturing and quantifying
semantic similarity in song lyrics.

Addressing the issue of imbalanced datasets, it is necessary to collect more
data on similarity pairs, composing them in a manner that allows for an
approximately uniform distribution of similarity ratings. This can be
achieved, for example, by precalculating similarities using one of the models
discussed in this paper. Building on the knowledge that language models can
partially capture semantic similarity between song lyrics, researchers can
design more targeted experiments to gain a deeper understanding of the
underlying factors that contribute to similarities and differences in lyrical
content, such as the relationship between song length and similarity
detection.

Moreover, conducting more user studies is crucial to assess the reliability
and validity of the models’ inferences. These studies should encompass not
only the annotation task and the training and evaluation processes of the
models, but also a user-driven evaluation of the models’ performance in real-
world applications, a practice that is already attracting the interest of the
NLP community [Schuff et al.,2023]. Lastly, continued analysis of the
experimental results obtained in this study will lead to a better
understanding of the strengths and limitations of the models used for
capturing semantic similarity in song lyrics, continuing with recent research
in the field [Zhang et al.,2021].

To summarize, this research has made significant progress in comprehending
semantic similarity within Spanish song lyrics, laying the groundwork for
further investigation. By addressing potential areas of future work, as
mentioned above, our aim is to develop a more all-encompassing understanding
of the factors that influence similarities and differences in lyrical content
across diverse languages and cultures. Ultimately, these advancements will
enhance models and applications for capturing and measuring semantic
similarity within song lyrics not only for the Spanish-speaking community but
also on a global scale. As we persist with refining our methods and exploring
new approaches, we remain devoted to promoting natural language processing
advancement while appreciating human linguistic diversity as well as musical
expression.

## Acknowledgments

  * •

This research has been carried out in the framework of the Grant LyrAIcs Grant
agreement ID: 964009 funded by ERC-POC-LS

  * •

This research has been carried out in the framework of the Grant CLS INFRA
reference 101004984 funded by H2020-INFRAIA-2020-1.

  * •

This research has received funding from the project ISL: Intelligent Systems
for Learning (GID2016-39) in the call PID 22/23 of the Vice-rectorate for
Educational Innovation of the University for National Distance Education of
Spain (UNED).

  * •

Alejandro Benito-Santos acknowledges support from the postdoctoral grant
”Margarita Salas”, awarded by the Spanish Ministry of Universities.

##

  * [Abe et al.,2022] Abe, K., S. Yokoi, T. Kajiwara, y K. Inui.  2022\.  Why is sentence similarity benchmark not predictive of application-oriented task performance?  En Proceedings of the 3rd Workshop on Evaluation and Comparison of NLP Systems, páginas 70–87, Online, Noviembre. Association for Computational Linguistics.
  * [Agerri y Agirre,2023] Agerri, R. y E. Agirre.  2023\.  Lessons learned from the evaluation of Spanish Language Models.  Procesamiento del Lenguaje Natural, 70(0):157–170, Marzo.
  * [Agirre et al.,2015] Agirre, E., C. Banea, C. Cardie, D. Cer, M. Diab, A. Gonzalez-Agirre, W. Guo, I. Lopez-Gazpio, M. Maritxalar, R. Mihalcea, G. Rigau, L. Uria, y J. Wiebe.  2015\.  SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability.  En Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2015), páginas 252–263, Denver, Colorado, Junio. Association for Computational Linguistics.  108 citations (Crossref) [2023-03-27].
  * [Agirre et al.,2014] Agirre, E., C. Banea, C. Cardie, D. Cer, M. Diab, A. Gonzalez-Agirre, W. Guo, R. Mihalcea, G. Rigau, y J. Wiebe.  2014\.  SemEval-2014 task 10: Multilingual semantic textual similarity.  En Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), páginas 81–91, Dublin, Ireland, Agosto. Association for Computational Linguistics.
  * [Agirre et al.,2012] Agirre, E., D. Cer, M. Diab, y A. Gonzalez-Agirre.  2012\.  SemEval-2012 task 6: A pilot on semantic textual similarity.  En *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), páginas 385–393, Montréal, Canada, 7-8 Junio. Association for Computational Linguistics.
  * [Agirre et al.,2013] Agirre, E., D. Cer, M. Diab, A. Gonzalez-Agirre, y W. Guo.  2013\.  *SEM 2013 shared task: Semantic Textual Similarity.  En Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, páginas 32–43, Atlanta, Georgia, USA, Junio. Association for Computational Linguistics.
  * [Cer et al.,2017] Cer, D., M. Diab, E. Agirre, I. Lopez-Gazpio, y L. Specia.  2017\.  SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation.  En Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), páginas 1–14, Vancouver, Canada, Agosto. Association for Computational Linguistics.
  * [Chandrasekaran y Mago,2022] Chandrasekaran, D. y V. Mago.  2022\.  Evolution of Semantic Similarity – A Survey.  ACM Computing Surveys, 54(2):1–37, Marzo.  arXiv:2004.13820 [cs].
  * [Conneau et al.,2020] Conneau, A., K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, y V. Stoyanov.  2020\.  Unsupervised Cross-lingual Representation Learning at Scale, Abril.  arXiv:1911.02116 [cs].
  * [da Silva, Silva, y Marcacini,2020] da Silva, A. C. M., D. F. Silva, y R. M. Marcacini.  2020\.  4mula: A multitask, multimodal, and multilingual dataset of music lyrics and audio features.  En Proceedings of the Brazilian Symposium on Multimedia and the Web, WebMedia ’20, página 145–148, New York, NY, USA. Association for Computing Machinery.
  * [Devlin et al.,2019] Devlin, J., M.-W. Chang, K. Lee, y K. Toutanova.  2019\.  BERT: Pre-training of deep bidirectional transformers for language understanding.  En NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, volumen 1, páginas 4171–4186.
  * [Fell,2020] Fell, M.  2020\.  Natural language processing for music information retrieval : deep analysis of lyrics structure and content.  phdthesis, Université Côte d’Azur, Mayo.
  * [Gutiérrez-Fandiño et al.,2022] Gutiérrez-Fandiño, A., J. Armengol-Estapé, M. Pàmies, J. Llop-Palao, J. Silveira-Ocampo, C. P. Carrino, C. Armentano-Oller, C. Rodriguez-Penagos, A. Gonzalez-Agirre, y M. Villegas.  2022\.  MarIA: Spanish Language Models.  Procesamiento del Lenguaje Natural, 68(0):39–60, Marzo.  Number: 0.
  * [Haider et al.,2020] Haider, T., S. Eger, E. Kim, R. Klinger, y W. Menninghaus.  2020\.  PO-EMO: Conceptualization, annotation, and modeling of aesthetic emotions in German and English poetry.  En Proceedings of the Twelfth Language Resources and Evaluation Conference, páginas 1652–1663, Marseille, France, Mayo. European Language Resources Association.
  * [He et al.,2021] He, P., X. Liu, J. Gao, y W. Chen.  2021\.  DeBERTa: Decoding-enhanced BERT with Disentangled Attention, Octubre.  arXiv:2006.03654 [cs].
  * [Krippendorff,2004] Krippendorff, K.  2004\.  Reliability in Content Analysis.  Human Communication Research, 30(3):411–433.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1468-2958.2004.tb00738.x.
  * [Li et al.,2021] Li, W., F. Qi, M. Sun, X. Yi, y J. Zhang.  2021\.  Ccpm: A chinese classical poetry matching dataset.  arXiv preprint arXiv:2106.01979.
  * [Liu et al.,2019] Liu, Y., M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, y V. Stoyanov.  2019\.  RoBERTa: A Robustly Optimized BERT Pretraining Approach, Julio.  arXiv:1907.11692 [cs].
  * [Reimers y Gurevych,2019] Reimers, N. y I. Gurevych.  2019\.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, Agosto.  arXiv:1908.10084 [cs].
  * [Rosa et al.,2022] Rosa, J. D. l., E. G. Ponferrada, M. Romero, P. Villegas, P. G. d. P. Salas, y M. Grandury.  2022\.  BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling.  Procesamiento del Lenguaje Natural, 68(0):13–23, Marzo.  Number: 0.
  * [Schuff et al.,2023] Schuff, H., L. Vanderlyn, H. Adel, y N. T. Vu.  2023\.  How to do human evaluation: A brief introduction to user studies in NLP.  Natural Language Engineering, páginas 1–24, Febrero.  0 citations (Crossref) [2023-03-31] Publisher: Cambridge University Press.
  * [Wolf et al.,2020] Wolf, T., L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, y A. Rush.  2020\.  Transformers: State-of-the-Art Natural Language Processing.  En Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, páginas 38–45, Online, Octubre. Association for Computational Linguistics.  979 citations (Crossref) [2023-03-30].
  * [Zhang et al.,2021] Zhang, X., M. Sun, J. Liu, y X. Li.  2021\.  Optimal Embedding Calibration for Symbolic Music Similarity, Septiembre.

##  Appendix A Annex 1: Original Task Description in Spanish

Instrucciones: Para esta tarea, comparará las letras de 2 canciones y evalúa
su similitud teniendo en cuenta las siguientes reglas de interpretación de las
letras:

  * •

Tema principal / contexto de la letra (por ejemplo: guerra, engaño, conocer a
una nueva persona)

  * •

Mensaje: opinión destacable del tema de la letra (por ejemplo: ”La vida es muy
corta”, ”Disfruta tu vida”, ”Uno puede amar solo una vez”)

  * •

Sentimientos / emociones del tema de la letra (decepción, tristeza,
esperanza,enamoramiento)

  * •

Significado literal de la letra /del vocabulario / ambiente (cosmos,
navegación, prado, vida en la gran ciudad, etc.)

  * •

Relación entre el emisario y el receptor de la letra (el hombre le canta a una
niña, etc.)

  * •

Estilo de lenguaje (jerga de adolescentes)

  * •

Conocimiento sobre el contexto social cultural de la canción (biografía del
autor/género musical, edad de la creación, etc.)

Teniendo en cuenta estas consideraciones puntúa la similitud ente los pares de
letras de canción entre 0 y 5 siendo:

  1. 0.

Completamente diferentes: Las letras son totalmente diferentes.

  2. 1.

Apenas existe similitud: se parecen en cualquiera de los cualquiera de los
aspectos sin importancia semántica como el estilo del lenguaje, o el contexto
social/cultural del contenido de las letras.

  3. 2.

Poca similitud: no hay similitud semántica (situación lírica, mensaje,
sentimientos) pero las letras se pueden considerar temáticamente (significado
literal) relacionadas entre sí.

  4. 3.

Similitud básica: solo se parecen en el mensaje, sentimientos del
protagonista/cantante, situación lírica o significado literal.

  5. 4.

Similitud notable/faltan detalles: el par de letras comparten el mismo mensaje
y sentimientos del sujeto, pero difieren en situaciones líricas y/o
significado literal.

  6. 5.

Similitud Sobresaliente: El par de letras tienen el mismo mensaje, comparten
la misma situación lírica, emociones e intenciones. La diferencia solo puede
estar en el léxico y en los géneros.

CONSIDERACION FINAL: Si tiene dudas y las instrucciones no son suficientes
para responder, siga su intuición.

##  Appendix B Annex 2: Task Description Translated into English

  * •

Main theme/context of the lyrics (e.g., war, deceit, meeting a new person)

  * •

Message: notable opinion on the subject of the lyrics (e.g., ”Life is very
short,” ”Enjoy your life,” ”One can love only once”)

  * •

Feelings/emotions of the subject of the lyrics (disappointment, sadness, hope,
infatuation)

  * •

Literal meaning of the lyrics/vocabulary/setting (cosmos, sailing, meadow,
life in the big city, etc.)

  * •

Relationship between the sender and the receiver of the lyrics (man singing to
a girl, etc.)

  * •

Language style (teenage slang)

  * •

Knowledge about the sociocultural context of the song (author’s
biography/music genre, age of creation, etc.)

Considering these factors, rate the similarity between the pairs of song
lyrics from 0 to 5 as follows:

  1. 0.

Completely different: the lyrics are entirely dissimilar.

  2. 1.

Barely any similarity: the lyrics share minor aspects without semantic
importance, such as language style or sociocultural context.

  3. 2.

Little similarity: there is no semantic similarity (lyrical situation,
message, feelings), but the lyrics can be considered thematically (literal
meaning) related.

  4. 3.

Basic similarity: the lyrics resemble each other in message, feelings of the
protagonist/singer, lyrical situation, or literal meaning.

  5. 4.

Notable similarity / missing details: the lyrics share the same message and
feelings but differ in lyrical situations and/or literal meaning.

  6. 5.

Outstanding similarity: the lyrics share the same message, emotions,
intentions, and lyrical situation, differing only in lexicon and genre.

FINAL CONSIDERATION: If you have doubts and the instructions are not
sufficient to provide an answer, follow your intuition.

##  Appendix C Annex 3: Summary of training parameters

A summary of the training parameters for each of the selected models is
provided below:

  * •

BERTIN

    * –

Total steps: 250k

    * –

Step 1:

      * *

Steps: 230k

      * *

Sequence length: 128

      * *

Batch size: 2048

    * –

Step 2:

      * *

Steps: 20k

      * *

Sequence length: 512

      * *

Batch size: 384

    * –

Epochs: Not specified

  * •

RoBERTa-base-bne (MarIA base)

    * –

Total steps: Not specified

    * –

Sequence length: 512

    * –

Batch size: 2048

    * –

Epochs: 1

    * –

Adam optimization with $\beta_{1}=0.9$, $\beta_{2}=0.98$, $\epsilon=1e^{-6}$,
and L2 weight decay of 0.01

  * •

RoBERTa-large-bne (MarIA large)

    * –

Total steps: Not specified

    * –

Sequence length: 512

    * –

Batch size: 2048

    * –

Epochs: 1

    * –

Adam optimization with $\beta_{1}=0.9$, $\beta_{2}=0.98$, $\epsilon=1e^{-6}$,
and L2 weight decay of 0.01

  * •

Sentence Transformer

    * –

Total steps: 1.5M

    * –

Sequence length: 512

    * –

Batch size: 8192

    * –

Epochs: Not specified

    * –

Adam optimization with $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=1e^{-4}$,
and L2 weight decay of 0.01

  * •

ALBERTI

    * –

Total steps: Not specified

    * –

Sequence length: Not specified

    * –

Batch size: Not specified

    * –

Epochs: Not specified

  * •

DeBERTa

    * –

Total steps: 1M

    * –

Sequence length: 768

    * –

Batch size: 64

    * –

Epochs: 10

    * –

Adam optimization with $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=1e^{-6}$,
and L2 weight decay of 0.01

  * •

XML-RoBERTa

    * –

Total steps: Not specified

    * –

Sequence length: Not specified

    * –

Batch size: 8192

    * –

Epochs: Not specified

##  Appendix D Annex 4: Table of test combined scores

model name | combined | spearmanr | pearson
---|---|---|---
BERTIN | 87.61 | 88.19 | 87.04
MarIA large | 89.12 | 89.97 | 88.26
MarIA base | 87.65 | 89.46 | 85.84
XLM-RoBERTa base | 90.63 | 92.23 | 89.03
XLM-RoBERTa large | 89.04 | 89.34 | 88.74
Sentence Transformer XLM-R | 88.31 | 89.46 | 87.17
mDeBERTa3 | 88.14 | 88.59 | 87.68
ALBERTI | 87.65 | 88.92 | 86.38
Table 4: Development set combined scores for all the models considered. model name | Batch size | Weight decay | Learning rate | Eval | Test
---|---|---|---|---|---
BERTIN | 32 | 0.01 | 0.00005 | 87.61 | 85.72
MarIA large | 32 | 0.1 | 0.00003 | 89.12 | 90.02
MarIA base | 8 | 0.01 | 0.00003 | 87.65 | 86.75
XLM-R base | 8 | 0.1 | 0.00001 | 90.63 | 86.45
XLM-R large | 16 | 0.01 | 0.00005 | 89.04 | 86.74
ST XLM-R | 8 | 0.01 | 0.00005 | 88.31 | 88.91
mDeBERTa3 | 16 | 0.01 | 0.00005 | 88.14 | 89.15
ALBERTI | 16 | 0.1 | 0.00003 | 87.65 | 86.35
Table 5: Best configuration for each model with combined test and evaluation
scores.


Keeping up with the bits: tracking physical layer latency in millimeter-wave
Wi-Fi networks

Alexander Marinšek | Liesbet Van der Perre
---|---
KU Leuven
ESAT-WaveCore
Ghent Technology Campus, 9000 Ghent, Belgium
<EMAIL_ADDRESS>

††footnotetext: This work has been accepted to the 41st WIC Symposium On
Information Theory and Signal Processing in the Benelux (SITB 2021).

###### Abstract

The wireless communications landscape is anticipated to offer new service
levels following the introduction of the millimeter-wave (mmWave) spectrum to
consumer electronics. With their broad bandwidths and corresponding multi-Gbps
data rates, these mmWaves are a perfect fit for data hungry applications, such
as streaming video to extended reality devices. However, the latter are also
bound by maximal latency constraints as low as 1 ms. Understanding where such
minuscule time delays lurk requires a close-up study of individual layers in
the network stack. Starting from the bottom up, the present work describes an
endeavor at uncloaking the origins of physical layer (PHY) latency in mmWave
Wi-Fi networks. It proposes a newly designed simulation framework and sheds
light on how any conventional laboratory can be turned into a virtual
experiment setting, speeding up computation. A case study based on the IEEE
802.11ad standard demonstrates the framework’s ability to track packet latency
at the PHY-level and identify individual bottlenecks. In particular, it
evaluates the impact of the number of LDPC decoding iterations on latency in
short transmission sequences.

Keywords— Simulation framework, Wi-Fi, millimeter-wave, IEEE 802.11ad

##  1 Introduction

A growing number of latency sensitive applications are imposing ever stricter
end-to-end (E2E) time delay constraints on wireless networks. The use cases
range from more traditional ones such as voice over IP (VoIP) [1], to future
connected vehicles, industry 4.0, and extended reality. The applications
contained therein require extremely low network time delays. For example,
coordinated driving [2], cooperative robots [3], and immersive media [4] may
all impose E2E latency constraints as low as 1 ms. Moreover, they are starting
to increasingly rely on computation offloading [5], giving rise to their
second requirement: high data rates.

Within the broad range of mmWave frequencies, several gigahertz of bandwidth,
centered around 60 GHz, have been allocated to unlicensed communications. One
of the standards operating in the 60 GHz band is the IEEE 802.11ad mmwave Wi-
Fi [6], also referred to as WiGig. Supporting 64QAM and high LDPC code rates,
it manages to attain data rates of up to 8.085 Gbps while conforming to the
1.76 GHz channel bandwidth limitations. This reduces the transmission times of
even the longest physical layer (PHY) payloads (262 KB) to well below 1 ms;
however, the finite transfer rate of the wireless network is not the only
factor determining E2E latency.

The IEEE 802.11ad PHY features several components, every one of which may
either add a small delay to the propagation of data, or in some cases, become
a bottleneck. For both analysis purposes and the conception of adequate
latency mitigation strategies and solutions, a simulation framework has been
developed.

###  1.1 State of the art simulation frameworks

Among the existing simulations tools, ns-3 is one of the most widespread
publicly available network simulators. It is backed by a lively community,
thanks to which it also supports the IEEE 802.11ad standard. Described in [7],
the WiGig ns-3 model features a detailed MAC layer on top of an abstract PHY
implementation. It is a versatile tool for higher-level studies, concerning
beamforming and fast session transfer among others. It is also able to keep
track of transmission delays and it can induce data corruption by leveraging
one of the underlying PHY error models. However, it does not cover the PHY
components in greater detail, meaning their contribution to latency is
unknown.

A substitute for ns-3 that offers more control over the PHY is the set of
tools contained within MATLAB and its Communications Toolbox. These offer
direct access to PHY component implementations, such as the LDPC codec. The
visualization and manual analysis of intermediate results is also simplified
with the use of MATLAB’s built-in debugging tools. While the toolbox offers
control over individual components, some effort was made in the past to
encapsulate them in an IEEE 802.11ad PHY simulation framework [8]. The
framework resides in an open online repository and its functionality is
verified against measurement data; yet, it does not support PHY latency
tracking.

Another endeavor to provide reliable WiGig simulations was presented in [9],
where the OPNET network simulator served as the foundation for the IEEE
802.11ad model. While the authors explicitly stated the timing results
obtained during simulation, the values ranging up to 5 ms include MAC layer
time delays as latency accumulation within the PHY remains unaddressed.

###  1.2 Contribution

With the progressively stricter latency requirements of future real-time
applications, gaining better insight into latency accumulation across the
network stack is growing in importance. As the foundation of the network
stack, the PHY is also the first layer contributing to latency accumulation.
The present work outlines the design principles of a fine-grained simulation
framework for tracking latency in the IEEE 802.11ad PHY. Furthermore, it
presents a potential usage of the framework for evaluating the impact of LDPC
decoding on transmission latency. The contributions are, therefore, two-fold:

  * •

Design of an open source IEEE 802.11ad PHY latency simulation framework in
Python, made publicly accessible through an online repository [10].

  * •

Evaluation of the contribution of iterative LDPC decoding to the total latency
during the transmission of short payloads (100 KB).

The work is structured as follows. Section 2 describes the working principles
of the simulation framework and presents a multi-processing approach to
reducing execution times both on the local machine and by using otherwise idle
remote hardware. Employing the framework on an example, section 3 demonstrates
latency tracking in practice and evaluates the delays induced by the LDPC
decoder. Finally, section 4 summarizes the main findings and outlines future
research prospects.

##  2 Simulation framework

The key features associated with PHY latency accumulation are finite
transmitter (TX) data rate and delays within the receiver (RX) digital
baseband (DBB). Simulating data propagation through individual components, the
simulation framework is not bound to a particular RX DBB implementation.
Assuming basic programming knowledge, the components can be added, removed, or
placed at different positions. Moreover, their latency performance figures are
freely adjustable. The RX DBB components are represented by individual blocks
and buffers within the simulation framework, the inner workings and invocation
of which are outlined in the following two sections.

In providing an illustrative example, the present work implements an RX DBB
that follows a particular design pattern. It assumes frequency domain
equalization and, with minor exceptions, the RX DBB is loosely based on the
one shown in Figure 1.

Figure 1: Overview of the simulated RX DBB (highlighted in red) within the
PHY, as presented in [11].

###  2.1 Buffers, blocks, and events

The propagation of information through the RX DBB takes the form of data
passing between the input buffers of individual blocks (components). If the
block fails to process the data in time, then they start piling up in its
input buffer. Several blocks also feature a second input buffer. Its function
is to halt the block from processing the data before being allowed to do so.
For example, the equalizer can only start processing the input data once
channel estimation has concluded. Corresponding to Figure 1 and Figure 2,
solid lines represent pathways leading to input buffers, while dashed lines
show the propagation of flag inputs.

Figure 2: Illustration of a block, representing an individual RX DBB
component.

Every time data is passed to a buffer, an event is triggered. The simulation
framework stores the event type (put, get, request) and the time at which it
occurred. Timekeeping is achieved using the SimPy***SimPy:
https://simpy.readthedocs.io/ Python package. The latter steps through
simulated time at a sub-nanosecond resolution and enables time-based data
propagation. The pace at which information is passed through the RX DBB is
dictated by the performance figures of individual components, sourced from
state of the art literature. These are covered in the corresponding module in
the simulation framework’s online repository.

###  2.2 Reducing simulation time

Executing simulations for multiple payload sizes at different modulation and
coding scheme (MCS) settings can present a time consuming task. Especially if
only one core on the host machine is employed. Bridging such bottlenecks is
often achieved using multithreading. However, being based on Python and
tightly connected to SimPy, the simulation framework must instead rely on
multiprocessing for parallel computing. Upon starting, the simulation is
automatically split into concurrent processes, and as they execute, new ones
are spawned based on the simulated parameter combinations. The size of the
process pool is manually adjustable. As hinted by figure 3, the execution time
is further reduced when multiple machines are used, such as otherwise idle
computers in a classroom.

Figure 3: Example of applying multiprocessing and manual distributed computing
for shorter execution times.

While process spawning is built into the framework, distributed computing must
be set up manually. A typical workflow is to clone the simulation framework to
all of the processing machines, configuring the input parameters (payload
length, MCS), and starting the simulation in the background. Once finished,
the results are manually aggregated before the data analysis is carried out.
This is somewhat cumbersome compared to using purpose made distributed
computing frameworks, yet, it reduces the number of dependencies and aids
overall simplicity of implementation.

##  3 Simulation framework usage examples

The simulation framework includes detailed event logging in the background. It
stores the exact time of every single block buffer input or output operation.
Upon simulation execution, the logs are stored on the host machined, together
with their corresponding metadata.

###  3.1 Latency accumulation in the physical layer

Illustrating the buffer events during a single transmission, Figure 4 analyzes
how data propagate through the RX DBB and where they spend the most time.
Observing buffer occupancy, there are two components where data are seemingly
piling up: the demapper and the descrambler. This sudden increase in the item
count is caused by the equalizer and decoder both outputting entire data
blocks. These are then processed as individual symbols and bits by the
demapper and descrambler. However, since the descrambler processes all of the
data before the arrival of a new dataword, it is not a bottleneck. The same
applies to the demapper, which, on the other hand, barely manages to keep up
with the incoming data. Mildly reducing its throughput would already make it a
bottleneck.

Figure 4: Example of a packet bearing 100 B of payload being transmitted using
MCS 2.0 and setting the LDPC decoder iteration count to 10. The colored
spheres represent buffer events, among which the first one is always the
initial data request. Where offset from zero (e.g. Joint CFO and IQ imbalance
estimation), there is an additional flag buffer, temporarily preventing the
component from processing the incoming data.

Depending on the transmission parameters (MCS, payload length) and RX DBB
configuration (component performance figures), the overview in Figure 4 might
take an entirely different form.

###  3.2 Evaluating iterative LDPC decoder latencies

A practical example, where the simulation framework can provide valuable
insight, is studying the effects of iterative decoding on the overall latency
of the PHY. This is made possible by associating a time delay function with
the decoder, dependent on the code rate and number of iterations. The latency
results for short transmission sequences are summarized in Figure 5. For each
modulation rate, the values obtained using code rate 0.5 feature the highest
latency since they spawn the most overhead. Furthermore, the 0.5 and 0.625
code rate lines exhibit a similar slope because of the decoder defining a
similar number of needed processing cycles for either of the two code rates
[12]. The remaining two code rates, 0.75 and 0.8125, reduce the processing
complexity and, therefore, the time delays diverge with the increasing number
of iterations.

Figure 5: Time delay incurred in the PHY during the transmission and reception
of PHY frames bearing 100 B payloads, dependent on the number of LDPC decoding
iterations. Line colors represent modulation rate (BPSK, QPSK, 16QAM, 64QAM),
while line types stand for different code rates (0.5, 0.625, 0.75, 0.8125).

From another perspective, the topmost and bottom-most lines are the
constraints of a new latency region. It outlines the range of expected
transmission time delays, regardless of the selected MCS, number of decoder
iterations, or both of them.

##  4 Conclusion

The present work describes the design and usage of an IEEE 802.11ad PHY
simulation framework, intended for PHY time delay analysis and the exploration
of latency mitigation strategies to meet strict latency requirements. The core
principles of how PHY components are reflected in the simulation framework
blocks are outlined, and an approach to reducing execution times using
parallel processing is discussed. The results demonstrate the simplicity of
tracking data propagation through the PHY components and identifying potential
bottlenecks. Finally, the evaluation of LDPC decoder latencies for small
payload sizes highlights how the framework can be used in specific studies.

While the presented simulation framework manages to keep track of PHY time
delays, it does not address data integrity. Since transmission errors are
expected, the next step should focus on associating latency with other
performance metrics, such as the bit error rate (BER), and evaluating the
trade-offs between the two. Future work can also consider studying specific
low-latency applications and the time delays of transmission sequences
typically associated with them.

##  5 Acknowledgement

This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No 861222 (MINTS MSCA-
ITN).

The authors also wish to acknowledge Gilles Callebaut for providing support
during different parts of debugging.

## References

  * [1] Murizah Kassim et al. “Performance analysis of VoIP over 3G and 4G LTE network” In _2017 International Conference on Electrical, Electronics and System Engineering (ICEESE)_ Kanazawa: IEEE, 2017, pp. 37–41 DOI: 10.1109/ICEESE.2017.8298391
  * [2] Athanasios Kanavos, Dimitrios Fragkos and Alexandros Kaloxylos “V2X Communication over Cellular Networks: Capabilities and Challenges” In _Telecom_ 2.1, 2021, pp. 1–26 DOI: 10.3390/telecom2010001
  * [3] 5G Alliance for Connected Industries and Automation “5G for Connected Industries and Automation”, 2019 URL: https://5g-acia.org/wp-content/uploads/2021/04/WP_5G_for_Connected_Industries_and_Automation_Download_19.03.19.pdf
  * [4] Peter Lincoln et al. “From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality” In _IEEE Transactions on Visualization and Computer Graphics_ 22.4, 2016, pp. 1367–1376 DOI: 10.1109/TVCG.2016.2518038
  * [5] Ashkan Yousefpour et al. “All one needs to know about fog computing and related edge computing paradigms: A complete survey” In _Journal of Systems Architecture_ 98, 2019, pp. 289–330 DOI: 10.1016/j.sysarc.2019.02.009
  * [6] IEEE Computer Society “Directional multi-gigabit (DMG) PHY specification” ISBN: 9781504436458 In _802.11-2016 - IEEE Standard for Information technology—Telecommunications and information exchange between systems Local and metropolitan area networks—Specific requirements - Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications_ IEEE, 2016, pp. 2436–2496 DOI: 10.1109/IEEESTD.2016.7786995
  * [7] Hany Assasa and Joerg Widmer “Implementation and Evaluation of a WLAN IEEE 802.11ad Model in ns-3” In _Proceedings of the Workshop on ns-3 - WNS3 ’16_ Seattle, WA, USA: ACM Press, 2016, pp. 57–64 DOI: 10.1145/2915371.2915377
  * [8] Jiri Blumenstein, Jiri Milos, Ladislav Polak and Christoph Mecklenbrauker “IEEE 802.11ad SC-PHY Layer Simulator: Performance in Real-world 60 GHz Indoor Channels” In _2019 IEEE Nordic Circuits and Systems Conference (NORCAS): NORCHIP and International Symposium of System-on-Chip (SoC)_ Helsinki, Finland: IEEE, 2019, pp. 1–4 DOI: 10.1109/NORCHIP.2019.8906960
  * [9] Carlos Cordeiro, Dmitry Akhmetov and Minyoung Park “Ieee 802.11ad: introduction and performance evaluation of the first multi-gbps wifi technology” In _Proceedings of the 2010 ACM international workshop on mmWave communications: from circuits to networks_ Chicago, Illinois, USA: Association for Computing Machinery, 2010, pp. 3–8 DOI: 10.1145/1859964.1859968
  * [10] Alexander Marinšek “PhyPy-802dot11ad/latency-simulator” Zenodo, 2021 DOI: 10.5281/zenodo.4749534
  * [11] Zulkuf Genc, Wim Van Thillo, Andre Bourdoux and Ertan Onur “60 GHz PHY Performance Evaluation with 3D Ray Tracing under Human Shadowing” In _IEEE Wireless Communications Letters_ 1.2, 2012, pp. 117–120 DOI: 10.1109/WCL.2012.022012.120033
  * [12] Meng Li et al. “A processor based multi-standard low-power LDPC engine for multi-Gbps wireless communication” In _2013 IEEE Global Conference on Signal and Information Processing_ Austin, TX, USA: IEEE, 2013, pp. 1254–1257 DOI: 10.1109/GlobalSIP.2013.6737136