# Small, Sparse, but Substantial: Techniques for Segmenting Small Agricultural
Fields Using Sparse Ground Data

Smit Marvaniya1, Umamaheswari Devi1, Jagabondhu Hazra1, Shashank Mujumdar1,
Nitin Gupta1
1IBM Research - India
{smarvani, umamadev, jahazra1, shamujum<EMAIL_ADDRESS>
###### Abstract

The recent thrust on digital agriculture (DA) has renewed significant research
interest in the automated delineation of agricultural fields. Most prior work
addressing this problem have focused on detecting medium to large fields,
while there is strong evidence that around 40% of the fields world-wide and
70% of the fields in Asia and Africa are small. The lack of adequate labeled
images for small fields, huge variations in their color, texture, and shape,
and faint boundary lines separating them make it difficult to develop an end-
to-end learning model for detecting such fields. Hence, in this paper, we
present a multi-stage approach that uses a combination of machine learning and
image processing techniques. In the first stage, we leverage state-of-the-art
edge detection algorithms such as holistically-nested edge detection (HED) to
extract first-level contours and polygons. In the second stage, we propose
image-processing techniques to identify polygons that are non-fields, over-
segmentations, or noise and eliminate them. The next stage tackles under-
segmentations using a combination of a novel “cut-point” based technique and
localized second-level edge detection to obtain individual parcels. Since a
few small, non-cropped but vegetated or constructed pockets can be
interspersed in areas that are predominantly croplands, in the final stage, we
train a classifier for identifying each parcel from the previous stage as an
agricultural field or not. In an evaluation using high-resolution imagery, we
show that our approach has a high F-Score of 0.84 in areas with large fields
and reasonable accuracy with an F-Score of 0.73 in areas with small fields,
which is encouraging.

##  1 Introduction

Digital agriculture (DA) encompasses technologies to enhance the productivity
and efficiency of agriculture, thereby bringing critical advantages for
farmers and wider social benefits around the world UN and Volvans (2017). A
first step in realizing the promise to the farmers is to digitize and create
electronic records of their fields. An electronic field record is the “digital
twin” of a physical field, which can be associated with detailed field
information such as boundary, soil type and moisture, pest attacks, crops
grown, and yields realized in addition to activities such as pesticide and
fertilizer application. By frequently collecting field data and analyzing it,
a farmer can be enabled to make more informed decisions. Further, the
availability of detailed field activities over the life-cycle of a crop can
help better predict yield quantity and quality, which can better inform
stakeholders downstream, such as commodity traders and financial
organizations, to optimize their operations.

Given the urgency of the digitization of field records, several research
efforts are underway to detect the boundaries of fields in an automated
manner. A comprehensive overview of these efforts is provided in Sec. 2. As
discussed therein, most of the proposed techniques have been developed and
evaluated in the context of large farms, found in countries in the Americas,
Europe, and Australia. However, according to Lesiv et al. (2019), about 40% of
the fields worldwide are small ranging 2 hectares in area while more than 70%
(resp. 50%) of the fields in Asia and Africa are small (resp., very small,
ranging less than 0.64 hectares in area). It is in the developing countries in
Asia and Africa, such as India and Kenya, that digitization of field records
is most lacking and currently involves significant manual effort and hence
prone to human and GPS errors, among others. Thus, in these countries,
comprehensive and accurate information on fields under cultivation and their
status, including field-level agricultural practices, is unavailable.

In this paper, we specifically tackle the problem of delineating the
boundaries of small fields, including some very small ones, at 0̃.4 ha. Due to
their small size and indistinct boundaries delineating them, we use aerial RGB
images captured at a high-resolution of 1.19 m. The images we use are
DigitalGlobe-based and sourced using Google’s Static Maps API.111The imagery
is accessed from https://maps.googleapis.com/maps/api/staticmap. For our
purpose, a field is a single farm-holding with adjacent fields separated by
structures such as a road, a fence, or a very thin strip of uncultivated land.
Our objective is to delineate the boundaries and identify each individual
field.

Identifying fields from images is a special instance of the image segmentation
problem, for which several traditional non-learning based algorithms Zhu et
al. (2015) as well as deep-learning models Garcia-Garcia et al. (2017) have
been developed. In comparison to images of 3-d objects that these methods seek
to detect, agricultural fields in aerial images are much simpler.
Nevertheless, detecting them, especially small fields, poses challenges and
the existing approaches are not directly applicable. Further, because the
fields are small, annotating them to generate sufficient ground truth to train
an end-to-end learning model is tedious and time-consuming. Hence, we use a
combination of image-processing and machine learning techniques to solve the
problem.

Contributions. We present a multi-stage approach for identifying small fields
in aerial imagery using limited ground data. We use the state-of-the-art
holistically nested edge detection (HED) Xie and Tu (2015) algorithm to
initially segment an image into first-level polygons. Since HED is a generic
edge-detection network and the open pre-trained model we use has not been
trained specifically using features related to crop fields, the output of HED,
when applied to field imagery, is a set of weakly-connected contours
containing both false and missing edges. Further, the detected contours are
significantly thick, diminishing the size of detected fields considerably,
especially when the fields are small. Hence, we develop and apply the
following novel techniques for culling out final agricultural fields from the
output contours of HED.

  * •

“Shape-complexity criteria” for detecting non-fields, over-segmentations, and
noise that need to be eliminated.

  * •

A method for detecting cut-points, cuts, and min-cuts in non-convex shapes,
along which an under-segmented polygon may need to be fragmented.

  * •

Second-level “localized” edge detection, tuned to image characteristics local
to each first-level polygon, to identify missing edges in it for further
segmentation.

  * •

A classifier trained using shape, color, intensity, and texture-based features
of individual parcels to classify the final parcels as agricultural (Ag) or
not (non-Ag).

Figure 1: Multi-stage framework for field boundary detection.

##  2 Related Work

Delineating boundaries and identifying individual fields is a special instance
of the image segmentation problem, with a rich body of literature. The
approaches developed therein can broadly be classified into (1) non-learning
approaches that use low-level, image-saliency features Zhu et al. (2015) and
(2) deep-learning approaches Garcia-Garcia et al. (2017); Badrinarayanan et
al. (2017); Chen et al. (2018). The non-learning approaches can further be
classified into edge- or boundary-based, region-based, or hybrid approaches.
Some of these approaches have been re-used and adapted for extracting
agricultural fields, as reviewed below.

Work on cropland extraction can broadly be classified based on the resolution
of the images and the spectral information used, the nature of the region it
applies to, and the algorithmic techniques used. Early work in this area
primarily used remote-sensed satellite imagery with resolution as coarse as 30
m and was hence confined to regions with large fields such as in Evans et al.
(2002). In Mueller et al. (2004), an approach that combines edge- and region-
based methods for identifying fields with boundaries that are typically long
with a straight object shape and have high brightness contrast to neighboring
regions is presented. The approach presented uses very high resolution (VHR)
imagery.

In later work, Yan and Roy (2014) presents another combination of edge- and
region-based method, in which, unlike traditional approaches, a time series of
Landsat satellite Williams et al. (2006) images at 30 m resolution is used.
The fields considered are large again. Crommelinck et al. (2017) investigates
the transferability of gPb contour detection to remotely-sensed VHR UAV
imagery (of the order of a few cms) and UAV-based cadastral mapping. Garcia-
Pedrero et al. (2017) solves the problem using ML techniques applied to SLIC
superpixels. Here, the idea is to first over segment the image into
superpixels and then train a classifier for each pair of superpixels with a
label denoting whether the pair is a part of the same field and can be merged.
To tackle sensitivity to intra-plot variability, a subset of authors later
explore combining segmentation at different scales using superpixels and
multi-temporal images from the same growing season to obtain a single
segmentation of the agricultural plots Garcia-Pedrero et al. (2018). These
techniques require high resolution multi-spectral imagery including bands in
the non-visible spectrum, which is hard to acquire, and labeled data to train
a supervised classifier.

Fueled probably by the growing imperative of DA, there has been a significant
surge in the work in this area in the past two years. A method for accurately
detecting large and mostly rectangular pastures in New Zealand with step edges
or linear features over a long distance for boundaries using time series SPOT
satellite imagery is presented in North et al. (2018). In Xia et al. (2019),
the authors explore the potential of deep Fully Convolutional Networks (FCNs)
for cadastral boundary detection (which is not limited to crop fields) and
compare it with traditional segmentation algorithms such as Multi-Resolution
Segmentation and gPb in two study sites in Rwanda. To deal with the small
field sizes, UAV images with a ground sampling distance of 2 cm, resulting in
VHR imagery, are used.

Other recent works exploring CNN’s include Garcia-Pedrero et al. (2019);
Persello et al. (2019); Xia et al. (2018); Waldner and Diakogiannis (2019). In
Xia et al. (2018), detecting hard and soft cropland edges from very high
resolution (0.8 m) GF-2 using RCF Liu et al. (2017) and U-net Ronneberger et
al. (2015) models is considered. Garcia-Pedrero et al. (2019) uses the U-Net
architecture to delineate crop fields and open data from the Land Parcel
Identification System of Spain for training. Images are again of VHR at 25 cm.
Results obtained are compared to that of gPb-UCM Arbelaez et al. (2011).
Persello et al. (2019) uses an FCN-based SegNet architecture to classify
boundary pixels and trains it using pan-sharpened multispectral bands of 0.5 m
resolution. The fragmented contours are closed to yield closed fields using
the OWT-UCM procedure. Finally, Waldner and Diakogiannis (2019) uses
ResUNet-a, a deep CNN with a fully connected UNet backbone to compute three
values for each pixel: the probability of belonging to a field, the
probability of being part of a boundary, and the distance to the closest
boundary. The computed values are then used to obtain closed field boundaries.

The recent works described above are quite interesting, but they all require
images that have sub-metre resolution or large-labeled datasets, or have
mostly been tested at select regions with large fields, and hence do not suit
our purpose.

##  3 Field Boundary Delineation System and Methods

This section describes the end-to-end stages and steps of our multi-stage
pipeline for segmenting field objects. Our input consists of ortho-rectified
pan-chromatic aerial or satellite imagery of the larger area of interest. The
processing steps have been designed to leverage characteristics unique to
field objects to improve the final segmentation. At a high-level, our pipeline
consists of four stages as in Fig. 1.

###  3.1 Stage 1: Initial Segmentation into Candidate Polygons

The first stage consists of (i) a pre-processing step to cull out, at a high-
level, parts of an image that correspond to croplands in which to identify
fields and (ii) an initial contour detection and segmentation step to identify
regions that are candidates for being individual small-holdings, as described
below.

Detecting croplands. In addition to cropland, a larger geography, such as a
state or county, typically consists of non-agricultural land, such as forests,
grassland, urban constructed areas, and water bodies. To eliminate processing
such regions, the first step of our pipeline masks away non-cropland areas
using an open global land cover mask provided by Copernicus Buchhorn et al.
(2019) at 100 m resolution.

Contour detection. The next step detects edges within regions identified as
croplands. While any edge detection algorithm can be used for this step, in
this work, as mentioned in Secs. 1 and 2, we apply the state-of-the-art HED
technique as it yielded the best results in our evaluation. HED uses a single
stream deep neural network with multiple side outputs that are at different
scales. Though HED can be retrained using aerial field imagery, due to lack of
adequate labeled data, in this work, we use a pre-trained HED network trained
on the BSDS500 dataset. As explained in Xie and Tu (2015),

$\vspace{-0.1cm}(Y_{\mathrm{fuse}},Y^{1}_{\mathrm{side}},...,Y^{N}_{\mathrm{side}})=\mathrm{CNN}(I^{\mathrm{input}},(W,w,h)^{*}),$

where $\mathrm{CNN}(\cdot)$ represents the edge map produced by the HED
network, $Y^{i}_{\mathrm{side}}$ denote the various side outputs, and
$Y_{\mathrm{fuse}}$, the weighted fusion output of the side outputs. $W$, $w$,
and $h$ denote the standard network layer parameters, weights for side output
layers, and fusion weights (for combining the side outputs), resp., which are
learned as a part of the pre-trained model. The final edge map is obtained by
further combining the edge maps generated at multiple scales by the side
outputs as
$Y_{\mathrm{hed}}=\mathrm{Average}(Y_{\mathrm{fuse}},Y^{1}_{\mathrm{side}},...,Y^{N}_{\mathrm{side}}).$

We clean-up the edge map generated by HED by applying image operations such as
erosion, dilation, and explicit edge-thinning. From the cleaned-up edge map,
an initial set of polygons, which are candidates for being fields, are
extracted by performing a connected-components extraction step.

The candidate polygons obtained require further processing due to the
following reasons: (i) Some polygons may not correspond to crop fields but
could be isolated buildings or built-up areas or non-crop but pockets with
natural vegetation that are interspersed within regions that are predominantly
croplands, and hence should not be classified as crop fields (or Ag fields),
(ii) the polygons could be under-segmented fields, wherein multiple adjacent
fields are agglomerated, or (iii) the polygons could be over-segmented wherein
a single field is split into multiple pieces. In what follows, we describe the
techniques we use to address the above issues. When the fields are small,
under-segmentation is more predominant than over-segmentation, and as such, we
focus more on handling the former.

###  3.2 Stage 2: Shape-based non-Ag and Spurious Field Elimination

We identify non-Ag regions using two different methods in two stages. We now
describe the first method, which deals with small spurious polygons and noise.
Method 2 is applied in Stage 4 after all significant parcels have been
detected.

The first method consists of a set of heuristics (listed below) that leverage
the layout characteristics of crop fields, which are typically convex, and
have a minimum size and aspect ratio (ratio or width to length) based on the
geography.

  1. 1.

Very small, noisy polygons removal: Both the perimeter and area of a field are
typically reasonably large, with the threshold depending on the geographic
location. Hence, the first rule drops all polygons with perimeter and area
less than specified thresholds.

  2. 2.

Convexity threshold: The next condition ensures that the contour of the
detected polygon does not deviate much from being convex. (Perfect convexity
cannot be expected because of some irregularities in the edges.) For this, for
polygons with area below a specified threshold, we verify that the ratio
between the areas of the convex hull drawn around the polygon and the polygon
per se. is not large.222Polygons that are significantly non-convex but larger
in area than the threshold are addressed by the sub-segmentation technique
discussed in Sec. 3.3

  3. 3.

Small noisy, elongated polygons removal: To eliminate small polygons that pass
the above two tests, but are still noisy, we require that for each polygon
with area below a specified minimum, the ratio between its area and perimeter
exceeds a threshold. This condition caters to removing within-field over-
segmentations due to small constructions, prominent water channels, etc.

  4. 4.

Long, thin strips removal: A long, thin strip with a regular shape, e.g., a
rectangle, can be easily identified by its aspect ratio. On the other hand,
identifying strips with irregular shapes without well-defined length or width,
as shown in yellow in Fig. 2(b), is a bit tricky. Such long thin strips are
mostly non-croplands and can correspond to footpaths in fields, roads, long
waterways, etc. To detect such strips, the width of a polygon is computed at
each of its boundary pixels as the minimum of the horizontal and vertical
distances (along the $x$ and $y$ axes, resp.) spanned by the polygon at that
pixel. The width of the polygon is then given by the average width at all the
boundary pixels. The approximate length of the polygon can then be obtained
using its average width and perimeter, from which its aspect ratio can be
computed. Polygons with aspect ratio below a specified threshold are then
discarded.

Fig. 2 shows a sample image with spurious fields identified using the above
constraints.

Figure 2: Sample image with spurious fields identified using shape constraints
of Sec. 3.2. (a) Input image. (b) Spurious fields identified via the four
conditions of Sec. 3.2 marked in order using blue, green, red, and yellow
colors.

###  3.3 Stage 3: Splitting Under-Segmented Polygons

In regions with small fields, first-level polygons could be grossly under-
segmented with several missing edges due to indistinct boundaries, noise,
coarse image resolution, etc.. Further, the edge detector we use, HED, is
trained on a generic set of images and oblivious to the distinct aspects of
fields and their bounding structures. Hence, in this stage, we identify and
fragment under-segmented polygons.

#### Splitting via Min-Cuts

The first method splits large polygons via high-curvature analysis of its
contour. Points on the contour are analyzed to identify cut points, from which
min-cuts – lines along which to split the polygon – are determined.

Many a time, faint boundaries separating fields are (i) only partly detected
by edge detection algorithms resulting in distinct fields that are held
together by short strips or (ii) largely undetected, leaving the fields mostly
connected, leading to odd, non-convex polygons. (See Fig. 3(b).) The purpose
of this step is to identify such instances of missing edges and fragment the
polygons along those.

Curvature on the contour at one or both the endpoints of missing edges in
cases as described above will, in general, be quite high. So, to detect the
missing edges, we first detect high-curvature points on the contour, which we
call cut points. Each cut point $c$ is paired with another point on the
contour to which its euclidean distance is the shortest and the contour
distance to which is greater than the euclidean distance to it. Each such pair
is termed a cut. Cuts for which the euclidean distance between its end points
is less than a threshold whereas the contour distance between the same points
is greater than a second threshold are chosen as the set of candidate cuts.
(Denoted by blue-yellow pairs in Fig. 3(d).) From this set, the final min-cuts
are culled out recursively as follows. The initial min-cut selected is the one
with maximum value for
$\alpha_{\mathrm{cut}(p_{i},p_{j})}\cdot\mathcal{C}_{s}(p_{i},p_{j})$, where
(i) $p_{i}$ and $p_{j}$ are the end points of $\mathrm{cut}(p_{i},p_{j})$,
(ii) $\alpha_{\mathrm{cut}(p_{i},p_{j})}$ is a binary flag that is set to 1 if
the line between $p_{i}$ and $p_{j}$ is within the polygon and the contour
lengths of the two sub-polygons produced by the cut exceed a threshold, and
(iii) $\mathcal{C}_{s}$ measures the strength of edge $(p_{i},p_{j})$ using a
combination of edge distance metric $C_{\mathrm{dist}}$ and edge probability
metric $\mathcal{C}_{\mathrm{prob}}$ as
$\mathcal{C}_{s}(\mathbf{p_{i}},\mathbf{p_{j}})=\beta\cdot\mathcal{C}_{\mathrm{dist}}(\mathbf{p_{i}},\mathbf{p_{j}})+(1-\beta)\cdot\mathcal{C}_{\mathrm{prob}}(\mathbf{p_{i}},\mathbf{p_{j}})$,
where $0\leq\beta\leq 1$. $\mathcal{C}_{dist}$ is computed by measuring the
distance between the cut segment and the contour map using directional chamfer
distance Liu et al. (2010) whereas $\mathcal{C}_{prob}$ is computed by
estimating the normalized edge probability along the cut.

We recursively apply the above procedure to each of the sub-polygons produced
by the initial min-cut to identify additional min-cuts to split the sub-
polygons. We terminate when there are no more qualifying cuts.

Fig. 3 shows an example in which multiple Ag and non-Ag fields are
agglomerated into a single polygon and illustrates the steps involved in
appropriately splitting them using min-cuts.

Figure 3: Splitting under-segmented fields using min-cuts. (a) Input Image
with Ag fields (yellow) and non-Ag fields (purple) of interest. (b) Under-
segmented polygon agglomerating multiple Ag fields and non-Ag polygons. (c)
Identified high curvature points. (d) Initial cut points, candidate cuts, and
min-cuts (yellow lines) in the order selected. (e) Splitting at the min-cuts
and discarding noisy fragments for the final Ag set.

#### Splitting via Localized Second-Level Contour Detection

The performance of edge-detection algorithms is impacted by the heterogeneity
of color, texture etc. within an image, and if based on a supervised model, on
the differences between the images in the training and test sets. Most
algorithms are dependent on a number of configurable parameters for
effectiveness, which are typically based on the mean values of the pixels in
the training or test images or both. This kind of averaging manifests as
under-segmentation, as the edges are sharper only when the intensity gradient
is large.

Figure 4: Sample localized second-level contour detection.

We deal with heterogeneous landscapes by performing a second-level “localized”
contour detection separately on each of the polygons detected at the end of
the previous step. The intuition is that configuring the edge detection
algorithms using local means (of the individual polygons) should help in
better detecting less distinct edges present within homogeneous regions. This
indeed turns out to be the case. Fig. 4 shows an example of localized second-
level edge detection using canny edge detector Canny (1986), wherein
hysteresis thresholds for the algorithm are based on the mean of pixels in the
polygon under consideration, as opposed to the entire image.

###  3.4 Stage 4: Ag vs. Non-Ag Classification

As discussed in Sec. 1, land cover classification maps are of coarse
resolution of 100 m or worse, and hence cannot be used to detect small pockets
of constructions or natural vegetation interspersed in predominantly
agricultural regions. The parcels identified in the previous stage can hence
be either crop fields or others. In this final stage we identify parcels that
are crop-based.

The spectral and shape features of agricultural (Ag) fields are considerably
different from the others (non-Ag). E.g., Ag fields have a reasonably definite
shape, which is mostly convex with only a few vertices, and distinct shade and
texture. Thus, we build a training set consisting of Ag and non-Ag parcel
images for each of which we extract its (i) shape features such as perimeter
and area, convex hull perimeter and area, (ii) color distribution in the form
of color histogram computed using Datta et al. (2006), and (iii) texture
histogram, capturing the distribution of local binary patterns Ojala et al.
(2000) of pixels in the parcels (computed using a radius of 3).

The above features are then used to train a random forest model for
classifying a detected parcel as Ag or non-Ag. Each resultant parcel at the
end of the previous stage is classified using this model, yielding the final
set of Ag fields.

##  4 Empirical Evaluation

This section evaluates our approach and presents its results.

###  4.1 Datasets and Experimental Setup

We evaluate our approach on two different datasets given below covering
multiple geographical locations. (Though our objective is to effectively
delineate small farms, we evaluate it on regions with larger farms, too, to
show that our approach is generic and extensible with minimal tuning.)

  1. 1.

Large Field Region (USA): Contains 10 images from different regions of the
United States. The average area captured by each image is around 5 sq. km. and
the average area of each field is around 11 hectares. The images together span
a total area of 50 sq. km.

  2. 2.

Small Field Region (India and Kenya): Contains 15 images from different
regions of developing countries such as India and Kenya. In this case, the
images are smaller by a factor of 5 at 1 sq. km. while the fields are smaller
by a factor of about 10 at 1 hectare. About 50% of the fields are very small
ranging less than 0.5 hectares in area. The images span around 15 sq. km.

We used the online image annotation tool _ImgLab_
333https://github.com/davisking/dlib/tree/master/tools/imglab developed by MIT
to manually mark boundaries of fields in each of the images in the dataset to
generate the ground truth.

Stage | Large Field Region (USA) | Small Field Region (India)
---|---|---
Precision | Recall | F1 | Precision | Recall | F1
PP | 0.6552 | 0.9536 | 0.7675 | 0.5615 | 0.8751 | 0.6787
PP+MC | 0.6836 | 0.9521 | 0.7812 | 0.5979 | 0.8716 | 0.7047
PP+LCD | 0.8721 | 0.8394 | 0.8463 | 0.6060 | 0.8679 | 0.7095
PP+MC+LCD | 0.8710 | 0.8409 | 0.8467 | 0.6858 | 0.7912 | 0.7290
Table 1: Ablation study of the multi-stage approach on large and small fields.
PP: non-Ag fields and noise removal using post-processing step, MC: Min-cuts,
LCD: Localized level-2 contour detection.

###  4.2 Results

Figure 5: Accuracy metrics for small (India) and large (USA) fields. The
numbers on the chart denote F1 scores.  Figure 6: Sample visual results for
field boundary delineation using the multi-stage approach for (a)–(d) small
fields and (e)–(h) large fields. Top line has vanila images while the bottom
line is with overlaid fields.

#### Overall boundary detection

To compute the accuracy metrics, we generate a mapping from the set of fields
in the ground truth to the fields in the detected output. Our mapping
effectively partitions the union of the two sets into three subsets: the first
subset identifies instances of over-segmentation and hence each ground truth
field in this set is mapped to more than one detected field, the second does
the inverse of identifying instances of under-segmentation and hence each
detected field in this set is mapped to multiple ground truth fields, while
the third subset contains fields that are rightly segmented and hence has a
one-to-one mapping from the ground truth subset to the detected subset. It is
possible for a ground-truth or detected field to be part of multiple under- or
over- or right segmentation instances or a combination. In such cases, a field
is assigned to the instance in which there is maximal overlap between the
ground truth and detected fields.

With the fields partitioned as above, accuracy numbers are generated for each
mapping instance. Pixels that belong to both the ground truth and detected
fields are labeled true positive pixels and those that belong only to a ground
truth (resp., detected) fields as false negative (resp., false positive)
pixels, using which precision, recall, and F-score (micro F1) are generated
for each mapping instance. The ratio of the number of ground truth (resp.,
detected) fields mapped to the detected (resp., ground truth) fields in each
instance provides the agglomeration (resp., fragmentation) metric. If $k$ is
the number of sub fields in an agglomeration or fragmentation, then the
corresponding metric is given by $\frac{1}{1+\log(k)}$. The numbers are
averaged over all the mapping instances weighted by the total number of ground
truth pixels in each instance to arrive at per-image accuracy metrics.

Image-level metrics are shown in Fig. 5. Overall, we detect more than 98% of
the large fields and 95% of the small fields. The average F1 score for the
detected large fields is around 84.5% while it is 73% for small fields (see
Table 1). Fragmentations are minimal for both large and small fields, with
0.92 and 0.94, resp., as averages. On the other hand, agglomerations are
higher for small fields with 0.79 as the average metric. This is due to the
faint boundaries between small fields and is as expected. Given the ambiguous
nature of the small-field images, the results are quite promising. Qualitative
visual results are shown in Figs. 6(a)–(d) for small fields and Figs. 6
(e)–(h) for large fields.

| Predicted
---|---
| Ag Fields | Non Ag Fields
Ground Truth | Ag Fields | 179 | 3
Non Ag Fields | 5 | 25
Macro-F1: 92%, Accuracy: 96.22%

Table 2: Confusion matrix for Ag vs non-Ag field classification.

#### Ag vs. Non-Ag Classification

We also assessed the accuracy of the random forest classifier that we use in
the final stage for culling out agricultural parcels. For this, we build a
dataset consisting of around 1025 fields captured in a small set of 25 images
(which are orthogonal to the set used above). The images are run through our
pipeline to delineate the parcels and also extract the needed features (see
Sec. 3.4). We manually label each identified parcel as Ag or non-Ag and
perform 5-fold cross-validation using the extracted features. Accuracy numbers
are reported in Table 2.

###  4.3 Ablation Study

We also performed an ablation study to assess the relative importance and
contribution of some of the important steps of the multi-stage approach. The
steps we evaluate are post-processing in Stage 2 to remove small non-Ag fields
and noise (PP), splitting under-segmented fields using min-cuts (MC) and
localized level-2 contour detection (LCD). Accordingly, we run the pipeline
with PP, PP+MC, PP+LCD, with the final Ag vs. non-Ag (NONAG) enabled in each
case. NONAG is evaluated separately (in Sec. 4.2 above). Numerical results are
provided in Table 1. Our key observations are as follows:

  * •

PP is able to identify most of the noisy fields and has very high recall of
0.95 and 0.88 for large and small fields, respectively.

  * •

Adding min-cuts based splitting of under-segmented fields (MC) improves the
precision and F1 for both the datasets without adversely impacting recall.

  * •

Performing localized level-2 contour detection (LCD) to handle under-
segmentation improves both precision and F1 significantly for both large and
small fields.

  * •

Precision and F1 improve substantially by 8 and 5 points for large and small
fields, resp., when MC and LCD are combined, in comparison to terminating with
PP in Stage 2.

##  5 Discussion & Conclusion

In this paper, we addressed the problem of accurately segmenting small fields
with indistinct boundaries, which dominate the agricultural landscape in
developing countries and constitute a substantial chunk globally. Unlike most
other objects, field objects lack a well-defined shape and can assume varying
shades of color and texture. Further, there is also a paucity of labeled
ground truth that can adequately represent small fields world-wide. Due to
these reasons, building an end-to-end learning model for segmenting small
fields is not effective, and we instead developed a multi-stage framework, in
which we leverage generic image-segmentation techniques and devise novel
domain-specific techniques. Our approach only required a small set of 25
minimally annotated images for training in the final stage as opposed to
thousands that would be required for an end-to-end learning-based approach.

In an empirical evaluation over fields spread across widely differing
geographies, the accuracy of our approach is promising. There is also scope to
enhance the approach using, for instance, multi-temporal data, which we plan
to consider in future work. Further, the work in this paper can be used to
generate much-needed ground truth for seeding end-to-end data-driven learning
and bootstrap the creation of a classifier based on it.

## References

  * Arbelaez et al. (2011) A. Arbelaez, M. Maire, C. Fowlkes, and J. malik.  Contour detection and hierarchical image segmentation.  PAMI, 33(5):898–916, 2011.
  * Badrinarayanan et al. (2017) Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  PAMI, pages 2481–2495, 2017.
  * Buchhorn et al. (2019) M. Buchhorn, B. Smets, L. Bertels, M. Lesiv, N. E. Tsendbazar, M. Herold, and S. Fritz.  Copernicus global land service: Land cover 100m: epoch 2015: Globe. dataset of the global component of the copernicus land monitoring service, 2019\.
  * Canny (1986) J. Canny.  A computational approach to edge detection.  PAMI, 8(6):679–698, 1986.
  * Chen et al. (2018) Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In ECCV, 2018.
  * Crommelinck et al. (2017) S. Crommelinck, R. Bennett, M. Gerke, M. Y. Yang, and G. Vosselman.  Contour detection for uav-based cadastral mapping.  Remote Sensing, 9(171):16–29, 2017.
  * Datta et al. (2006) Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z Wang.  Studying aesthetics in photographic images using a computational approach.  In ECCV, pages 288–301. Springer, 2006.
  * Evans et al. (2002) C. Evans, R. Jones, I. Svalbe, and M. Berman.  Segmenting multispectral landsat tm images into field units.  TGRS, 40(5):16–29, 2002.
  * Garcia-Garcia et al. (2017) A. Garcia-Garcia, S. Orts-Escolano, S.O. Oprea, V. Villena-Martinez, and J. Garcia-Rodriguez.  A review on deep learning techniques applied to semantic segmentation, 2017.
  * Garcia-Pedrero et al. (2017) Angel Garcia-Pedrero, C Gonzalo-Martín, and M Lillo-Saavedra.  A machine learning approach for agricultural parcel delineation through agglomerative segmentation.  IJRS, 38(7):1809–1819, 2017.
  * Garcia-Pedrero et al. (2018) A. Garcia-Pedrero, C. Gonzalo-Martin, M. Lillo-Saavedra, and D. Rodriguez-Esparragon.  The outlining of agricultural plots based on spatiotemporal consensus segmentation.  Remote Sensing, 10(1991), 2018.
  * Garcia-Pedrero et al. (2019) A. Garcia-Pedrero, M. Lillo-Saavedra, D. Rodriguez-Esparragon, and C. Gonzalo-Martin.  Deep learning for automatic outlining agricultural parcels: Exploiting the land parcel identification system.  Access, 7:158223–158236, 2019.
  * Lesiv et al. (2019) Myroslava Lesiv, Juan Carlos Laso Bayas, Linda See, Martina Duerauer, Domian Dahlia, Neal Durando, Rubul Hazarika, Parag Kumar Sahariah, Mar’yana Vakolyuk, Volodymyr Blyshchyk, et al.  Estimating the global distribution of field size using crowdsourcing.  Global change biology, 25(1):174–186, 2019.
  * Liu et al. (2010) Ming-Yu Liu, Oncel Tuzel, Ashok Veeraraghavan, and Rama Chellappa.  Fast directional chamfer matching.  In CVPR, pages 1696–1703. IEEE, 2010.
  * Liu et al. (2017) Y. Liu, M.-M. Cheng, X. Hu, K. Wang, and X. Bai.  Richer convolutional features for edge detection.  In CVPR, pages 5872–5881. IEEE, 2017.
  * Mueller et al. (2004) Marina Mueller, Karl Segl, and Hermann Kaufmann.  Edge-and region-based segmentation technique for the extraction of large, man-made objects in high-resolution satellite imagery.  PR, 2004.
  * North et al. (2018) Heather C North, David Pairman, and Stella E Belliss.  Boundary delineation of agricultural fields in multitemporal satellite imagery.  JSTARS, 12(1):237–251, 2018.
  * Ojala et al. (2000) Ojala, T. Pietikainen, and M. Maenpaa.  Gray scale and rotation invariant texture classification with local binary patterns.  Lecture Notes in Computer Science, 1842:404–420, 2000.
  * Persello et al. (2019) C Persello, VA Tolpekin, JR Bergado, and RA de By.  Delineation of agricultural fields in smallholder farms from satellite images using fully convolutional networks and combinatorial grouping.  Remote Sensing of Environment, 231:111253, 2019.
  * Ronneberger et al. (2015) O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In MICCAI, 2015.
  * UN and Volvans (2017) UN and Volvans.  Digital agriculture - feeding the future.  http://breakthrough.unglobalcompact.org/disruptive-technologies/digital-agriculture/, 2017\.
  * Waldner and Diakogiannis (2019) François Waldner and Foivos Diakogiannis.  Deep learning on edge: extracting field boundaries from satellite images with a convolutional neural network.  arXiv, 2019.
  * Williams et al. (2006) Darrel L Williams, Samuel Goward, and Terry Arvidson.  Landsat.  Photogrammetric Engineering & Remote Sensing, 72(10):1171–1178, 2006.
  * Xia et al. (2018) L. Xia, J. Luo, Y. Sun, and H. Yang.  Deep extraction of cropland parcels from very high resolution remotely sensed imagery.  In ICAG, 2018.
  * Xia et al. (2019) X. Xia, C. Persello, and M. Koeva.  Deep fully convolutional networks for cadastral boundary detection from uav images.  Remote Sensing, 11(1725), 2019.
  * Xie and Tu (2015) Saining Xie and Zhuowen Tu.  Holistically-nested edge detection.  In ICCV, pages 1395–1403, 2015.
  * Yan and Roy (2014) L Yan and DP Roy.  Automated crop field extraction from multi-temporal web enabled landsat data.  Remote Sensing of Environment, 144:42–64, 2014.
  * Zhu et al. (2015) H. Zhu, F. Meng, J. Cai, and S. Lu.  Beyond pixels: A comprehensive survey from bottom-up to semantic image segmentation and cosegmentation, 2015.


# A Computational Analysis of Lyric Similarity Perception

Haven Kim
&Taketo Akama

###### Abstract

In musical compositions that include vocals, lyrics significantly contribute
to artistic expression. Consequently, previous studies have introduced the
concept of a recommendation system that suggests lyrics similar to a user’s
favorites or personalized preferences, aiding in the discovery of lyrics among
millions of tracks. However, many of these systems do not fully consider human
perceptions of lyric similarity, primarily due to limited research in this
area. To bridge this gap, we conducted a comparative analysis of computational
methods for modeling lyric similarity with human perception. Results indicated
that computational models based on similarities between embeddings from pre-
trained BERT-based models, the audio from which the lyrics are derived, and
phonetic components are indicative of perceptual lyric similarity. This
finding underscores the importance of semantic, stylistic, and phonetic
similarities in human perception about lyric similarity. We anticipate that
our findings will enhance the development of similarity-based lyric
recommendation systems by offering pseudo-labels for neural network
development and introducing objective evaluation metrics.

A Computational Analysis of Lyric Similarity Perception

Haven Kim Taketo Akama

##  1 Introduction

The popularization of music streaming services has expanded the use and
distribution of music recommendation systems, which are instrumental in
facilitating the discovery of music within vast music libraries, catering to
diverse user preferences. While collaborative filtering (CF) remains a common
method in various recommendation domains, music recommendation systems have
largely shifted towards content-driven approaches by learning similarities in
audio signals McFee et al. (2012); Bogdanov et al. (2013) and calculating the
similarity between a user’s query and potential matches in the dataset Deldjoo
et al. (2021); Pearce and Müllensiefen (2017). These approaches have
necessitated consideration of aspects that influence music similarity
perception Schedl et al. (2018), including ones that model human perception of
similarities between two musical audio signals Berenzweig et al. (2004);
Novello et al. (2011), which enriches our understanding of how people
understand music and what they expect from music search systems Casey et al.
(2008).

In musical tracks accompanied with vocal performances, however, lyrics often
play a pivotal role in enhancing the appeal of the song as they express
emotions that resonate with listeners by conveying messages Watanabe and Goto
(2020). Consequently, there has been a growing interest in developing a lyric
recommender system that suggests lyrics akin to a user’s favorites or custom
choices. The earliest that we know was based on similarity between vectors
that represent the inverse document frequency (IDF) of words Mahedero et al.
(2005). Some systems employed unsupervised learning algorithms for clustering
lyrics. These include models based on Latent Dirichlet Allocation (LDA) Sasaki
et al. (2014); Tsukuda et al. (2017), the Doc2Vec, and the Self-Organizing
Feature Maps (SOFMs) Patra et al. (2017).

Although promising, such systems lack the consideration as to what constitutes
perceptual lyric similarity despite the suggestion of prior study that
integrating a model of human perception and cognition into the algorithms
could significantly mitigate the limitations inherent in purely feature-based
algorithms Pampalk (2004). Moreover, considering that these recommender
systems aim to suggest lyrics similar to the input in terms of perception, the
gap in understanding what humans anticipate from music searches becomes a
critical issue.

This oversight stems from a dearth of research in this specific area. Lyric
similarity is a concept that lacks a clear definition, and to our knowledge,
no scientific research has been conducted to pinpoint the factors influencing
perceptual lyrical similarity. Moreover, due to the lack of attention in this
domain, objective evaluation metrics have yet to be established. Consequently,
the evaluation of previous systems has depended on subjective assessments
Sasaki et al. (2014); Patra et al. (2017), which are time-consuming and
costly, or on indirect measures like perplexity Tsukuda et al. (2017).

Notably, there is relatively little focus on computationally modeling
perceptual lyric similarity. To address this, we compared six computational
methods, including ones influenced by existing systems and our own novel
approaches, against human perceptions of lyric similarity. This led us to
investigate whether computational metrics accurately represent perceptual
lyric similarity. Finally, we identified three computational metrics that show
statistically significant correlations with human perception: one based on
pre-trained language models, one on the music tracks from which the lyrics are
derived, and one on phonetic characteristics such as the degree of phoneme
repetition and the choice of phonemes. We expect these findings to be
beneficial for building systems that suggest similar lyrics to input by
generating pseudo-labels for neural model development or evaluating similar-
lyric recommendation systems.

The subsequent section will specify the selection and details of these six
computational models (Section 2). Following that, we will outline the design
of the human perception survey we conducted (Section 3). The results of this
survey will be presented and analyzed in the next section (Section 4). The
last section will conclude the paper, highlighting insights gained from our
human perception experiments and the potential applications of our
findings(Section 5).

##  2 Computational Modeling of Lyric Similarity

In this section, we introduce several factors that have been suggested as
contributing to characterizing lyrics. We then describe computational models
used to calculate the values of similarities in these factors, which will be
compared with human perceptions of lyric similarity along with motivation from
previous studies for selecting each model. Python implementations of these
metrics are provided as supplementary materials and will be released publicly
upon acceptance.

###  2.1 Topic

The concept of topic is pivotal in studies on lyrics, as suggested by models
for on lyrics generation based on topic transitions Watanabe et al. (2014,
2017) and automatic topic detection Choi and Downie (2018). Particularly, an
earlier study postulated that thematic content significantly influences lyric
similarity, leading to the development of a system rooted in a model initially
devised for topic modeling Sasaki et al. (2014); Tsukuda et al. (2017). This
suggests a potential impact of topic on the perceived similarity of lyrics.
Consequently, our experiment employs a computational model to gauge lyric
similarity from this perspective.

Topic Similarity ($\text{sim}_{\text{top}}$): Latent Dirichlet Allocation
(LDA) Blei et al. (2003), an unsupervised machine learning algorithm designed
for topic modeling, has been applied to a system that automatically recommends
to lyrics akin to a user’s input Sasaki et al. (2014); Tsukuda et al. (2017).
This algorithm operates on the premise that topics can be defined by their
word usage, learning how topics are distributed within one unit of lyrics and
how words are distributed across a given topic. Following this, we trained a
LDA model to categorize lyric topics into 50 classes. In order to numerically
represent topic similarity between pairs of lyrics, we followed the convention
of calculating cosine similarity between the LDA vectors Towne et al. (2016).
When the predicted topic probabilities for a given lyric set $\mathbf{L}$ is
represented as a vector
$t(\mathbf{{L}})=[t_{0}(\mathbf{L}),…,t_{49}(\mathbf{L})]$, where
$t_{n}(\mathbf{L})$ is the probability of $\mathbf{L}$ being classified under
the $n$-th topic, we define the topic similarity ($\text{sim}_{\text{top}}$)
between two lyric sets $\mathbf{L}$ and $\mathbf{\tilde{L}}$, as a cosine
similarity between $t(\mathbf{L})$ and $t(\mathbf{\tilde{L}})$, as follows.

$\displaystyle\scalebox{1.0}{$\text{sim}_{\text{top}}(\mathbf{L},\mathbf{\tilde{L}})=\frac{t(\mathbf{L})\cdot
t(\mathbf{\tilde{L}})}{|t(\mathbf{L})||t(\mathbf{\tilde{L}})|}$}$ (1)

###  2.2 Meaning

Semantic factors play a crucial role in determining the perceptual similarity
of textual content. In a preliminary study, we asked four participants to
assess lyric similarity and to explain their criteria; the meaning was the
most frequently mentioned factor. This indicates a substantial effect of
meaning on how people perceive lyric similarity. Therefore, our experiment
incorporates a computational model to automatically assess lyric similarity
based on semantics.

Semantic Similarity ($\text{sim}_{\text{sem}}$): A popular approach to obtain
numerical representation of semantics is to utilize embeddings of a pre-
trained BERT Devlin et al. (2019)-based language model. For our experiments,
we used a pre-trained Sentence BERT model Reimers and Gurevych (2019) 111We
used all-MiniLM-L6-v2 Wang et al. (2020), where we call the 384-dimensional
contextual embeddings of lyrics $\mathbf{L}$,
$s(\mathbf{{L}})=[s_{0}(\mathbf{L}),…,s_{383}(\mathbf{L})]$. We define the
semantic similarity ($\text{sim}_{\text{sem}}$) as the cosine similarity
between embeddings, as below.

$\displaystyle\scalebox{1.0}{$\text{sim}_{\text{sem}}(\mathbf{L},\mathbf{\tilde{L}})=\frac{s(\mathbf{L})\cdot
s(\mathbf{\tilde{L}})}{|s(\mathbf{L})||s(\mathbf{\tilde{L}})|}$}$ (2)

###  2.3 Mood

Mood is a widely studied concept in studies on lyrics, with mood
classification and analysis based on lyrics Laurier et al. (2008); Hu et al.
(2009) standing out as prominent topics in the field of music information
retrieval, and a prior study even suggested that lyrics are more effective for
classifying music moods than audio Hu and Downie (2010). Furthermore, a recent
lyric recommendation system relies on emotion prompts, classified into four:
happy, angry, relaxed, and sad Revathy et al. (2023). This suggests the
potential role of mood in determining lyric similarity. In response, we have
incorporated a computational model that accounts for this aspect.

Mood Difference ($\text{diff}_{\text{mood}}$): To numerically represent mood
for lyrics, we utilized the Deezer Mood Detection Dataset Delbouys et al.
(2018), a dataset characterizes a musical track $\mathbf{T}$ with a two-
element vector, $[v(\mathbf{T}),a(\mathbf{T})]$, representing its valence and
arousal levels. Here, high valence with high arousal corresponds to happiness,
low valence with high arousal indicates anger, low valence and low arousal
denotes sadness, and high valence and low arousal represents relaxation. To
compare two sets of lyrics $\mathbf{L}$ from track $\mathbf{T}$ and
$\mathbf{\tilde{L}}$ from $\mathbf{\tilde{T}}$ in terms of mood, we calculated
the euclidean distance between these two-element vectors
$[v(\mathbf{T}),a(\mathbf{T})]$ and
$[v(\mathbf{\tilde{T}}),a(\mathbf{\tilde{T}})]$. This distance is defined as
the mood difference ($\text{diff}_{\text{mood}}$) between $\mathbf{L}$ and
$\mathbf{\tilde{L}}$.

$\displaystyle\scalebox{0.9}{$\text{diff}_{\text{mood}}(\mathbf{L},\mathbf{\tilde{L}})=\sqrt{\left(v(\mathbf{T})-v(\mathbf{\tilde{T}})\right)^{2}+\left(a(\mathbf{T})-a(\mathbf{\tilde{T}})\right)^{2}}$}$
(3)

###  2.4 Style

Style, often considered to be defined by an artist, is an ambiguous but
crucial factor in characterizing lyrics. In response, previous research has
delved into style classification based on lyrics Li and Ogihara (2004) and
lyric generation models that emulate the style of a particular artist Barbieri
et al. (2012). This indicates that style similarity could play a significant
role in the perceived similarity of lyrics. Consequently, we have included a
computational model in our study to address this aspect.

Audio Similarity ($\text{sim}_{\text{aud}}$): Musical characteristics affects
a song’s lyrical style Mayer et al. (2008). For example, lyrics within the
same genre tend to exhibit stylistic similarities. Therefore, we used audio
features in order to represent similarity between lyrics. In order to obtain
features for lyrics $\mathbf{L}$, we obtained the 200-sized embedding
$e(\mathbf{T})$ for the corresponding musical track $\mathbf{T}$, generated by
the penultimate layer of a CNN-based music feature extractor Pons and Serra
(2019). The audio similarity ($\text{sim}_{\text{aud}}$) between two sets of
lyrics $\mathbf{L}$ and $\mathbf{\tilde{L}}$, each for musical tracks
$\mathbf{T}$ and $\mathbf{\tilde{T}}$, is defined as defined as a cosine
similarity between two corresponding embeddings, $e(\mathbf{T})$ and
$e(\mathbf{\tilde{T}})$.

$\displaystyle\scalebox{1.0}{$\text{sim}_{\text{aud}}(\mathbf{L},\mathbf{\tilde{L}})=\frac{e(\mathbf{T})\cdot
e(\mathbf{\tilde{T}})}{|e(\mathbf{T})||e(\mathbf{\tilde{T}})|}$}$ (4)

###  2.5 Sound

In our study, we also concentrated on the sound, which comes from the phonetic
features of lyrics, rather than their meaning. Although the concept of sound
similarity is somewhat nebulous, certain studies have investigated it by
focusing on phonetic features, such as rhyme, for musical genre classification
Mayer et al. (2008). Despite the concept being relatively underexplored in
lyrics research, we identified two computational models from previous studies
that address this aspect within the realms of poetry and lyric translation
evaluation. The two computational models employed in our experiments are based
on these studies.

Phonetic Similarity ($\text{sim}_{\text{pho}}$): One of our computational
models was inspired by a previous study that assessed sound similarity in
poetry Parrish (2017). This study initially decomposed 113,852 words into
phonemes (e.g., ring into [R, IH, NG]) and represented each phoneme with pre-
defined corresponding features (e.g., [R, IH, NG] with [{alv, apr},
{fnt,smh,unr,vwl}, {nas, vel}]). Consequently, two pseudo-features were added
at the beginning and end of each feature list, and pairs of consecutive
features were formed. (e.g., ({beg}×{alv, apr}) $\cup$ ({alv,
apr}×{fnt,smh,unr,vwl}) $\cup$ ({fnt,smh,unr,vwl}×{nas, vel}) $\cup$ ({nas,
vel}×{end})). The distribution of these feature pairs was then represented as
a vector, and PCA analysis Abdi and Williams (2010) was applied to reduce the
dimensionality of these vectors to 50. Adopting this methodology, we generated
50-dimensional phonetic vectors using lyrics, rather than individual words.
For two sets of lyrics $\mathbf{L}$, $\mathbf{\tilde{L}}$, with their
corresponding phonetic vectors
$p(\mathbf{L})=[p_{0}(\mathbf{L}),\dots,p_{49}(\mathbf{L})]$ and
$p(\mathbf{\tilde{L}})=[p_{0}(\mathbf{\tilde{L}}),\dots,p_{49}(\mathbf{\tilde{L}})]$,
we defined the cosine similarity between $p(\mathbf{L})$ and
$p(\mathbf{\tilde{L}})$ as the phonetic similarity between $\mathbf{L}$ and
$\mathbf{\tilde{L}}$ ($\text{sim}_{pho}$), consistent with the methodology of
the prior study, as follows.

$\displaystyle\scalebox{1.0}{$\text{sim}_{pho}(\mathbf{L},\mathbf{\tilde{L}})=\frac{p(\mathbf{L})\cdot
p(\mathbf{\tilde{L}})}{\|p(\mathbf{T})\|\|p(\mathbf{\tilde{L}})\|}$}$ (5)

Musical Difference ($\text{diff}_{\text{mus}}$): A previous study on lyric
translation suggested that a set of lyrics’ musicality is influenced by two
factors: phoneme repetition and phoneme choice Kim et al. (2023). To quantify
the phoneme repetition degree, they decomposed the lyrics $\mathbf{L}$ into
phonetic components and grouped them into bi-grams, and divided the unique bi-
gram counts by the total bi-gram counts. This value, called $pho$, indicates
higher repetition with a lower value, and vice versa. The difference in
repetition between two sets of lyrics, $\mathbf{L}$ and $\mathbf{\tilde{L}}$,
is determined by comparing $pho(\mathbf{L})$ and $pho(\mathbf{\tilde{L}})$. To
assess the similarity in phoneme usage between $\mathbf{L}$ and
$\mathbf{\tilde{L}}$, they calculated the $pho$ of the combined lyrics
$pho(\mathbf{L}\mathbin{{+}\mspace{-8.0mu}{+}}\mathbf{\tilde{L}})$, where
lower values imply greater shared phonemes. The musical difference between two
sets of lyrics was defined as the sum of the differences in repetition degrees
($|pho(\mathbf{L})-pho(\tilde{\mathbf{L}})|$) and the $pho$ of the
concatenated lyrics
($pho(\mathbf{L}{\mathbin{{+}\mspace{-8.0mu}{+}}}\tilde{\mathbf{L}})$). We
adopted this approach to calculate the musical difference
($\text{diff}_{\text{mus}}$) between $\mathbf{L}$ and $\mathbf{\tilde{L}}$.

$\displaystyle\scalebox{0.8}{$\text{diff}_{\text{mus}}(\mathbf{L},\mathbf{\tilde{L}})=\left|pho(\mathbf{L})-pho(\tilde{\mathbf{L}})\right|+pho(\mathbf{L}\oplus\tilde{\mathbf{L}})$}$
(6)

Figure 1: A matrix displaying the Pearson correlation coefficients between the
six metrics No pair exhibits a correlation exceeding 0.20, suggesting a
relative independence among these metrics.

(a) Topic Similarity

(b) Semantic Similarity

(c) Mood Difference

(d) Audio Similarity

(e) Phonetic Similarity

(f) Musical Difference

Figure 2: Distribution analysis of six metrics across 100,000 randomly sampled
lyric pairs. Red lines represents the boundaries separating the low (L),
medium (M), and high (H) similarity groups for each metric.

##  3 Experimental Setup

###  3.1 Data Collection

We collected lyrics along with their metadata from the Deezer Mood Detection
Dataset Delbouys et al. (2018) and the Million Song Dataset Bertin-Mahieux et
al. (2011). Additionally, we obtained 30 seconds audio previews that are
matched against metadata from the Million Song Dataset. This comprehensive
collection yielded a total of 17,865 tracks. Each song, on average, contained
7.07 lyrical sections, and we ended up with 126,336 total sets of lyrics.

###  3.2 Grouping Lyric Pairs

The number of feasible lyric pair combinations from the 126,336 lyric sets,
excluding pairs derived from the same song, was 7,979,812,331. To observe the
statistical tendency of these pairs, we randomly sampled 100,000 pairs and
evaluated them using six different metrics that were introduced in the
previous section.

Our primary goal was to categorize lyric pairs into three distinct groups-
high, medium, and low similarity (H, M, L)-for each metric. Due to the unique
range and distribution characteristics of the six metrics, we avoided uniform
cutoffs for categorization. Instead, we defined the “medium” category (M) as
pairs whose scores fell within one standard deviation from the mean. Pairs
with scores beyond this range were categorized as either “high” (H) or “low”
similarity (L) . For metrics indicating similarity,
($\text{sim}_{\text{top}}$, $\text{sim}_{\text{sem}}$,
$\text{sim}_{\text{aud}}$, $\text{sim}_{\text{pho}}$), pairs with higher
scores were assigned to the “high” similarity group ($H_{top}$, $H_{sem}$,
$H_{aud}$, and $H_{pho}$) and those with lower scores were to the “low” group
($L_{top}$, $L_{sem}$, $L_{aud}$, and $L_{pho}$). Conversely, for metrics
indicating differences ($\text{diff}_{\text{mood}}$ and
$\text{diff}_{\text{mus}}$), pairs with higher scores were categorized as
having “low” similarity ($L_{mood}$ and $L_{mus}$), and vice versa. The
distribution and categorization cutoffs are detailed in Figure 2.

(a) Topic Similarity ($\text{sim}_{\text{top}}$)

(b) Semantic Similarity ($\text{sim}_{\text{sem}}$)

(c) Mood Difference ($\text{diff}_{\text{mood}}$)

(d) Audio Similarity ($\text{sim}_{\text{aud}}$)

(e) Phonetic Similarity ($\text{sim}_{pho}$)

(f) Musical Difference ($\text{diff}_{\text{mus}}$)

Figure 3: Statistical analysis of similarity groups: High, Medium, and Low.
The red lines represent the mean ranking values, with 1 indicating the highest
similarity and 3 the lowest. The bounds mark the 95% confidence interval above
and the -95% interval below, respectively.

###  3.3 Variable Control

Using the same sample of 100,000 lyrics, we assessed the absolute Pearson
correlation between metrics, identifying four mildly correlated pairs with the
correlation larger than or equal to 0.1-{$\text{sim}_{\text{top}}$,
$\text{sim}_{\text{sem}}$}, {$\text{sim}_{\text{top}}$,
$\text{diff}_{\text{mus}}$}, {$\text{sim}_{\text{sem}}$,
$\text{diff}_{\text{mus}}$}, and $\\{\text{sim}_{pho}$,
$\text{diff}_{\text{mus}}$}. The mild correlation between
$\text{sim}_{\text{top}}$ and $\text{sim}_{\text{sem}}$ (0.18) is expected, as
topics identified by an LDA model often reflect semantic coherence. Their mild
correlations with $\text{diff}_{\text{mus}}$ (0.14 and 0.19, respectively) are
attributed to the tendency that shared words significantly affect these
metrics by resulting in higher $\text{sim}_{\text{top}}$ and
$\text{sim}_{\text{sem}}$ and lower $\text{diff}_{\text{mus}}$. On the other
hand, the correlation between $\text{diff}_{\text{mus}}$ and
$\text{sim}_{pho}$ appears to be attributed to their shared focus on phonetic
aspects.

Despite these correlations, no pair exceeded a 0.20 correlation, indicating
relative independence among metrics. Therefore, we were able to maintain
consistency in the other five metrics when assessing the impact of a single
metric, thereby minimizing external variable influence. Given the limited
array of lyric pairs, finding sets with identical scores across five metrics
while differing in one was nearly impossible. We instead selected pairs that
share closely matched scores across five metrics. To determine “closeness”, we
used the concept of item similarity, defined as one minus the ratio of their
differences to the maximum possible difference Bansal and Jain (2020).
Although 0.8 is suggested in previous literature, we employed a more stringent
threshold of 0.99 for each metric, to ensure higher precision in our
experiment. For instance, if the topic similarity score
($\text{sim}_{\text{top}}$) between a reference lyric set ($\mathbf{L}$) and
two comparison sets ($\mathbf{\tilde{L_{1}}}$ and $\mathbf{\tilde{L_{2}}}$)
are 0.67 and 0.68 respectively, these are considered “close” in terms of
$\text{sim}_{\text{top}}$ for our experimental purposes, given that the
maximum $\text{sim}_{\text{top}}$ score is 1 and the minimum is 0.

###  3.4 Survey

Each survey question was designed to evaluate the impact of one specific
metric. Participants were presented with a set of reference lyrics and three
comparison sets, corresponding to high, medium, and low levels of similarity
(H, M, L) as measured by the selected metric. To ensure consistency, the three
comparison sets achieved closeness across the other five metrics. For
instance, in questions assessing topic similarity, three comparison sets were
selected to maintain over 99% closeness in terms of semantic similarity, mood
difference, audio similarity, phonetic similarity, and musical difference with
reference lyris while each having high, medium, and low levels of topic
similarity with reference.

Among various methods employed in perceptual experiments to evaluate
similarity, we chose ranking, instead of rating, as it has proven its
simplicity and robustness in various domains Burton and Nerlove (1976);
Novello et al. (2011); participants, at each question, were asked to identify
which comparison sets most closely resembled the reference lyrics and which
were the most dissimilar. The survey comprised six questions for each metric,
each with different reference lyrics from a variety of genres. These genres,
including rock, pop, country, hip-hop, electronic, and R&B, were selected
based on prior studies that discuss genre diversity in lyrics Tee et al.
(2022). This resulted in a total of 36 unique question sets (6 genres × 6
metrics). The orders of both the questions and the comparison lyrics within
each question were randomized. Our survey involve four lyricists proficient in
English and their average age are 32.25 years.

##  4 Results

In this section, we examine the survey results presented in Figure 3 and Table
1. Figure 3 illustrates the statistical comparisons among three similarity
groups (H, M, L) in terms of mean ranking scores and their 95% confidence
intervals. Table 1 provides the Pearson correlations between the six metrics
and the ranking scores, along with their p-scores. A ranking score of 1
implies the highest perceptual similarity, while a score of 3 suggests the
lowest.

| Correlation |  $p$-value
---|---|---
Topic Similarity ($\text{sim}_{\text{top}}$) | -0.13 | 0.56
Semantic Similarity ($\text{sim}_{\text{sem}}$) | -0.65 | 0.00
Mood Difference ($\text{diff}_{\text{mood}}$) | 0.06 | 0.79
Audio Similarity ($\text{sim}_{\text{aud}}$) | -0.48 | 0.03
Phonetic Similarity ($\text{sim}_{pho}$) | -0.41 | 0.09
Musical Difference ($\text{diff}_{\text{mus}}$) | 0.74 | 0.00

Table 1: Pearson Correlation Coefficients and Corresponding $p$-Values Between
Six Metrics and Ranking Scores.

Topic Similarity ($\text{sim}_{\text{top}}$) : We observed that the group with
the lowest topic similarity ($L_{top}$) exhibited the lowest perceptual
similarity. However, the high $\text{sim}_{\text{top}}$ group ($H_{top}$) did
not demonstrate better perceptual similarity compared to the medium group
($M_{top}$); the latter even exhibited marginally higher mean perceptual
similarity. This finding suggests potential limitations in lyric retrieval
systems that rely soley on LDA models, given that they are suppoed to
recommend highly similar lyrics. The Pearson correlation of -0.13 and a
$p$-value of 0.56 further indicate a non-significant statistical relationship
between this metric and perceptual similarity.

Semantic Similarity ($\text{sim}_{\text{sem}}$) : Conversely, a significant
correlation was found between semantic similarity ($\text{sim}_{\text{sem}}$)
and perceptual similarity, evidenced by the correlation coefficient of -0.65
and a near-zero $p$-value, showing the robustness of this relationship. The
group with high $\text{sim}_{\text{sem}}$ scores ($H_{sem}$) showed markedly
better perceptual similarity compared to the medium ($M_{sem}$) and low groups
($L_{sem}$), which had similar average rankings. This finding further suggests
the importance of $\text{sim}_{\text{sem}}$ value in order to achieve high
perceptual lyric similarity.

Mood Difference ($\text{diff}_{\text{mood}}$) : Contrary to expectations, mood
did not exhibit a statistically significant correlation with perceptual
similarity, as indicated by a correlation of 0.061, a $p$-value of 0.79, and
the lack of differences across the high, medium, and low similarity groups
($H_{mood}$, $M_{mood}$, $L_{mood}$) in mean ranking values. This suggests
that a mood-based similar-lyric recommendation system may not align well with
human perceptual judgments.

As an example, we take one question from the survey with reference lyrics that
represents a sad mood with valence and arousal levels of -1.94 and -0.66,
respectively, as annotated in the Deezer Mood Detection Dataset Delbouys et
al. (2018).

[ indentfirst=false, leftmargin=rightmargin=] Reference lyrics: 222
Songwriters: Michael Dulaney, Neil Thrasher
Seventeen years old
She was out with her friends
They started drinking at some party
Until she was three sheets to the wind
Her momma always told her she could call no matter what
She was crying on the front steps
When her mom showed up

Participants compared the reference lyrics with two sets of lyrics: the first
set, Lyrics from $H_{mood}$, exhibits high mood similarity with them,
reflecting sadness by matching the reference lyrics in both valence and
arousal levels. The second set, Lyrics from $L_{mood}$, demonstrates low mood
similarity, representing happiness with distinct valence and arousal levels of
1.54 and 1.70, respectively.

[ indentfirst=false, leftmargin=rightmargin=] Lyrics from $H_{mood}$: 333
Songwriters: Mick Jagger, Keith Richards
And you know who’s the 2000 man
And your kids they just won’t understand you at all

Lyrics from $L_{mood}$: 444 Songwriter: Rhett Miller
Skin all over you
I’m a worrier it’s true
I came over to cushion your breakables
Breathe in you are stronger
You don’t have to wonder
You know my conviction’s unshakable
Although lyrics from $H_{mood}$ share the same arousal and valence values as
the reference lyrics, participants did not perceive them to be similar. In
fact, three out of four participants found lyrics from $L_{mood}$—which
significantly differs in mood levels—to be more similar to the reference
lyrics.

Audio Similarity ($\text{sim}_{\text{aud}}$) : Rather, our findings suggest
that humans tend to focus on stylistic elements of lyrics, which are often
determined by musical features (e.g., genre), as shown by the moderate
correlation of -0.48 between audio similarity ($\text{sim}_{\text{audio}}$)
and perceptual similarity ($p$-value = 0.03). This relationship is further
shown in the distinct perceptual similarity rankings among the $H_{aud}$,
$M_{aud}$, and $L_{aud}$ groups.

Phonetic Similarity ($\text{sim}_{p}ho$) : The relationship between phonetic
similarity ($\text{sim}_{pho}$) and perceptual similarity showed a moderate
correlation with a correlation efficient of -0.41, with the high
$\text{sim}_{pho}$ group ($H_{pho}$) achieving the highest perceptual
similarity. However, a $p$-value of 0.09 suggests unreliability in this
metric.

Musical Difference ($\text{diff}_{\text{mus}}$) : The musical difference
metric ($\text{diff}_{\text{mus}}$), another sound-related measure,
demonstrated a more stable correlation (0.743) with perceptual similarity
($p$-value = 0.00). Notably, small perceptual difference was observed between
the high and medium similarity groups ($H_{mus}$ and $M_{mus}$), yet a marked
decrease in perceptual similarity was apparent in the low similarity group
($L_{mus}$). This suggests a relative insensitivity of human perception to
small $\text{diff}_{\text{mus}}$ values, but a heightened sensitivity to
larger differences.

##  5 Conclusions and Future Work

In this study, we evaluated six computational models that quantify lyric
similarity, including both ones influenced by prior studies and novel
approaches proposed by us, against human perception. Our results reveal that
three metrics—semantic similarity ($\text{sim}_{\text{sem}}$), audio
similarity ($\text{sim}_{\text{aud}}$), and musical difference
($\text{diff}_{\text{mus}}$)—demonstrate statistically significant
correlations with human perceptual assessments. The relatively low correlation
among these metrics (less than 0.2) suggests that lyric similarity is a multi-
dimensional concept. Notably, while semantic, stylistic, and sound aspects
significantly influence perceptual lyric similarity, as indicated by the
correlations with $\text{sim}_{\text{sem}}$, $\text{sim}_{\text{aud}}$, and
$\text{diff}_{\text{{mus}}}$, respectively, mood did not emerge as a
significant factor. This was evidenced by the negligible correlation of 0.06
between perceptual similarity and mood difference
($\text{diff}_{\text{mood}}$). Moreover, we showed that while unsupervised
algorithms aiming to capture thematic ($\text{sim}_{\text{top}}$) or phonetic
coherence ($\text{sim}_{\text{{pho}}}$) showed some correlations with human
perception, if failed to prove robustness and reliability.

We expect that our exploration of the alignment between various computational
models and human perception regarding lyric similarity will catalyze further
research in this area. Potential applications of three computational models,
which have demonstrated correlations with human perception, include the
development of basic lyric recommendation algorithms based on a simple
calculation, the generation of pseudo-labels for neural network development,
and the automatic evaluation of similarity-based lyric recommendation systems.

## References

  * Abdi and Williams (2010) Hervé Abdi and Lynne J Williams. 2010.  Principal component analysis.  _Wiley interdisciplinary reviews: computational statistics_ , 2(4):433–459.
  * Bansal and Jain (2020) Kriti Bansal and Vinod Jain. 2020.  Performance analysis of collaborative filtering based recommendation system on similarity threshold.  In _2020 Fourth International Conference on Computing Methodologies and Communication (ICCMC)_ , pages 442–448. IEEE.
  * Barbieri et al. (2012) Gabriele Barbieri, François Pachet, Pierre Roy, and Mirko Degli Esposti. 2012.  Markov constraints for generating lyrics with style.  In _Ecai_ , volume 242, pages 115–120.
  * Berenzweig et al. (2004) Adam Berenzweig, Beth Logan, Daniel PW Ellis, and Brian Whitman. 2004.  A large-scale evaluation of acoustic and subjective music-similarity measures.  _Computer Music Journal_ , pages 63–76.
  * Bertin-Mahieux et al. (2011) Thierry Bertin-Mahieux, Daniel PW Ellis, Brian Whitman, and Paul Lamere. 2011.  The million song dataset.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.  Latent dirichlet allocation.  _Journal of machine Learning research_ , 3(Jan):993–1022.
  * Bogdanov et al. (2013) Dmitry Bogdanov et al. 2013.  From music similarity to music recommendation: Computational approaches based on audio features and metadata.
  * Burton and Nerlove (1976) Michael L Burton and Sara B Nerlove. 1976.  Balanced designs for triads tests: Two examples from english.  _Social Science Research_ , 5(3):247–267.
  * Casey et al. (2008) Michael A Casey, Remco Veltkamp, Masataka Goto, Marc Leman, Christophe Rhodes, and Malcolm Slaney. 2008.  Content-based music information retrieval: Current directions and future challenges.  _Proceedings of the IEEE_ , 96(4):668–696.
  * Choi and Downie (2018) Kahyun Choi and J Stephen Downie. 2018.  Exploratory investigation of word embedding in song lyric topic classification: promising preliminary results.  In _Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries_ , pages 327–328.
  * Delbouys et al. (2018) Rémi Delbouys, Romain Hennequin, Francesco Piccoli, Jimena Royo-Letelier, and Manuel Moussallam. 2018.  Music mood detection based on audio and lyrics with deep neural net.  _arXiv preprint arXiv:1809.07276_.
  * Deldjoo et al. (2021) Yashar Deldjoo, Markus Schedl, and Peter Knees. 2021.  Content-driven music recommendation: Evolution, state of the art, and challenges.  _arXiv preprint arXiv:2107.11803_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , pages 4171–4186.
  * Hu and Downie (2010) Xiao Hu and J Stephen Downie. 2010.  When lyrics outperform audio for music mood classification: A feature analysis.  In _Ismir_ , pages 619–624.
  * Hu et al. (2009) Xiao Hu, J Stephen Downie, and Andreas F Ehmann. 2009.  Lyric text mining in music mood classification.  _American music_ , 183(5,049):2–209.
  * Kim et al. (2023) Haven Kim, Kento Watanabe, Masataka Goto, and Juhan Nam. 2023.  A computational framework for singable lyric translation.  In _Proceedings of the 24th International Society for Music Information Retrieval (ISMIR)_.
  * Laurier et al. (2008) Cyril Laurier, Jens Grivolla, and Perfecto Herrera. 2008.  Multimodal music mood classification using audio and lyrics.  In _2008 seventh international conference on machine learning and applications_ , pages 688–693. IEEE.
  * Li and Ogihara (2004) Tao Li and Mitsunori Ogihara. 2004.  Music artist style identification by semi-supervised learning from both lyrics and content.  In _Proceedings of the 12th annual ACM international conference on Multimedia_ , pages 364–367.
  * Mahedero et al. (2005) Jose PG Mahedero, Álvaro MartÍnez, Pedro Cano, Markus Koppenberger, and Fabien Gouyon. 2005.  Natural language processing of lyrics.  In _Proceedings of the 13th annual ACM international conference on Multimedia_ , pages 475–478.
  * Mayer et al. (2008) Rudolf Mayer, Robert Neumayer, and Andreas Rauber. 2008.  Rhyme and style features for musical genre classification by song lyrics.  In _Ismir_ , pages 337–342.
  * McFee et al. (2012) Brian McFee, Luke Barrington, and Gert Lanckriet. 2012.  Learning content similarity for music recommendation.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 20(8):2207–2218.
  * Novello et al. (2011) Alberto Novello, Martin MF McKinney, and Armin Kohlrausch. 2011.  Perceptual evaluation of inter-song similarity in western popular music.  _Journal of New Music Research_ , 40(1):1–26.
  * Pampalk (2004) Elias Pampalk. 2004.  A matlab toolbox to compute music similarity from audio.  In _ISMIR_.
  * Parrish (2017) Allison Parrish. 2017.  Poetic sound similarity vectors using phonetic features.  In _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ , volume 13, pages 99–106.
  * Patra et al. (2017) Braja Gopal Patra, Dipankar Das, and Sivaji Bandyopadhyay. 2017.  Retrieving similar lyrics for music recommendation system.  In _Proceedings of the 14th International Conference on Natural Language Processing (ICON-2017)_ , pages 290–297.
  * Pearce and Müllensiefen (2017) Marcus Pearce and Daniel Müllensiefen. 2017.  Compression-based modelling of musical similarity perception.  _Journal of New Music Research_ , 46(2):135–155.
  * Pons and Serra (2019) Jordi Pons and Xavier Serra. 2019.  musicnn: Pre-trained convolutional neural networks for music audio tagging.  _arXiv preprint arXiv:1909.06654_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992.
  * Revathy et al. (2023) VR Revathy, Anitha S Pillai, and Fatemah Daneshfar. 2023.  Lyemobert: Classification of lyrics’ emotion and recommendation using a pre-trained model.  _Procedia Computer Science_ , 218:1196–1208.
  * Sasaki et al. (2014) Shoto Sasaki, Kazuyoshi Yoshii, Tomoyasu Nakano, Masataka Goto, and Shigeo Morishima. 2014.  Lyricsradar: A lyrics retrieval system based on latent topics of lyrics.  In _Ismir_ , pages 585–590.
  * Schedl et al. (2018) Markus Schedl, Hamed Zamani, Ching-Wei Chen, Yashar Deldjoo, and Mehdi Elahi. 2018.  Current challenges and visions in music recommender systems research.  _International Journal of Multimedia Information Retrieval_ , 7:95–116.
  * Tee et al. (2022) Tze Huat Tee, Belicia Qiao Bei Yeap, Keng Hoon Gan, and Tien Ping Tan. 2022.  Learning to automatically generating genre-specific song lyrics: A comparative study.  In _Iberoamerican Knowledge Graphs and Semantic Web Conference_ , pages 62–75. Springer.
  * Towne et al. (2016) W Ben Towne, Carolyn P Rosé, and James D Herbsleb. 2016.  Measuring similarity similarly: Lda and human perception.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 8(1):1–28.
  * Tsukuda et al. (2017) Kosetsu Tsukuda, Keisuke Ishida, and Masataka Goto. 2017.  Lyric jumper: A lyrics-based music exploratory web service by modeling lyrics generative process.  In _ISMIR_ , pages 544–551.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 33:5776–5788.
  * Watanabe and Goto (2020) Kento Watanabe and Masataka Goto. 2020.  Lyrics information processing: Analysis, generation, and applications.  In _Proceedings of the 1st Workshop on NLP for Music and Audio (NLP4MusA)_ , pages 6–12, Online. Association for Computational Linguistics.
  * Watanabe et al. (2014) Kento Watanabe, Yuichiroh Matsubayashi, Kentaro Inui, and Masataka Goto. 2014.  Modeling structural topic transitions for automatic lyrics generation.  In _Proceedings of The 28th Pacific Asia Conference on Language, Information and Computation_ , pages 422–431. Waseda University.
  * Watanabe et al. (2017) Kento Watanabe, Yuichiroh Matsubayashi, Kentaro Inui, Tomoyasu Nakano, Satoru Fukayama, and Masataka Goto. 2017.  Lyrisys: An interactive support system for writing lyrics based on topic transition.  In _Proceedings of the 22nd international conference on intelligent user interfaces_ , pages 559–563.