# Automatic Neural Lyrics and Melody Composition

Gurunath Reddy M  Yi Yu  Florian Harscoët  Simon Canales  Suhua Tang Gurunath
Reddy M is with Indian Institute of Technology, Kharagpur, India (email:
mgurunathreddy@sit.iitkgp.ernet.in)Yi Yu is with National Institute of
Informatics, Japan, (email: yiyu@nii.ac.jp)Florian Harscoët is with ISIMA,
France, (email: florian.harscoet@etu.uca.fr)Suhua Tang is with The University
of Electro-Communications, Japan, (email: shtang@uec.ac.jp)Simon Canales is
with EPFL, Switzerland, (email: simon.canales@epfl.ch)This paragraph will
include the Associate Editor who handled your paper.

###### Abstract

In this paper, we propose a technique to address the most challenging aspect
of algorithmic songwriting process, which enables the human community to
discover original lyrics, and melodies suitable for the generated lyrics. The
proposed songwriting system, Automatic Neural Lyrics and Melody Composition
(AutoNLMC) is an attempt to make the whole process of songwriting automatic
using artificial neural networks. Our lyric to vector (lyric2vec) model
trained on a large set of lyric-melody pairs dataset parsed at syllable, word
and sentence levels are large scale embedding models enable us to train data
driven model such as recurrent neural networks for popular English songs.
AutoNLMC is a encoder-decoder sequential recurrent neural network model
consisting of a lyric generator, a lyric encoder and melody decoder trained
end-to-end. AutoNLMC is designed to generate both lyrics and corresponding
melody automatically for an amateur or a person without music knowledge. It
can also take lyrics from professional lyric writer to generate matching
melodies. The qualitative and quantitative evaluation measures revealed that
the proposed method is indeed capable of generating original lyrics and
corresponding melody for composing new songs.

Recently, we can see a surge in growing research interest in automatically
generating music after the successful application of deep neural networks for
sequential modeling. Most of the works focus on either generating melodies or
lyrics. However, to the best of our knowledge, there are no works which focus
on automating the song creation process which has potential application in
cinema and gaming industries, and educational pedagogy. The proposed AutoNLMC
can be used by both amateur and professionals to create songs with both lyrics
and melodies automatically. Further, generated lyric melody pairs can be used
to synthesize songs using synthesizers which can be readily used in gaming and
cinema applications. Also, using AutoNLMC one can discover new melodies for
already existing lyrics without melodies which enables a faster song creation
process.

Lyrics, Melody, Notes, Singing voice

##  1 Introduction

Music composition is a human creative process that requires a wide range of
strong musical knowledge and expertise to create soothing music which
continues to remain in our heart forever. Given the vast majority of music
lovers and the limited availability of professional music composers, there is
a strong need for machines to assist human creativity. Recent advancement in
the software based music creation technology helped the professional and
amateur music creators to produce music with great joy and ease of production
in masses to be consumed by the music consumers with personal computers and
hand-held devices. Though there exists a plenty of machine assistance to
create high quality music with relative ease of production, the process of
songwriting that is automatically generating lyrics, composing melody
corresponding to the generated lyrics and synthesizing singing voice
corresponding to the generated melody and lyrics remained as mutually
exclusive tasks. Till date, the construction of novel/original songs is
limited to the individuals who possess the following skills: the ability to
create lyrics, compose melody and combine lyrics and melody to create a
rational, relevant and soothing final complete songs [1].

Figure 1: Example music score

In literature, we can find considerable amount of research work published on
automatic music generation (without conditioning on lyrics). Early machine
assisted music generation is mostly based on music theory and expert domain
knowledge to create novel works. With the advent of data driven approaches and
exploded public music collections in the internet, data driven methods such as
Hidden Markov models [2], graphic models [3] and deep learning models [4, 5,
6, 7, 8, 9] showed a potential for music creation. Though there exists
substantial amount of research on unconditional music generation, there exists
considerably less amount of work done so far on generating melody from lyrics
given in the form of text, which we call conditional melody/song generation
from lyrics. The primary reasons for substantially less research on
conditional melody generation can be attributed to i) the non-availability of
the direct source for lyrics-melody pair dataset to train the data driven
models, ii) a lyrics composition can have multiple melodic representations,
which makes it hard to learn the correlation between the lyrics and melodies,
and iii) it is hard to evaluate the generated melodies by objective measures.

This paper focuses on the most challenging aspect of algorithmic songwriting
process which enables the human community to discover original lyrics, and
melodies suitable for the generated lyrics. To the best our knowledge, the
proposed AutoNLMC is the first attempt to make the whole process of
songwriting automatic using artificial neural networks. We also present the
lyrics to vector model which is trained on a large dataset of popular English
songs to obtain the dense representation of lyrics at syllables, words and
sentence levels. The proposed AutoNLMC is an attention based encoder-decoder
sequential recurrent neural network model consists of a lyric generator, lyric
encoder and melody decoders trained end-to-end. We train several encoder-
decoder models on various dense representations of the lyric tokens to learn
the correlation between lyrics and corresponding melodies. Further, we prove
the importance of dense representation of lyrics by various qualitative and
quantitative measures. AutoNLMC is designed in such a way that it can generate
both lyrics and corresponding melodies automatically for an amateur or a
person without music knowledge by accepting a small piece of initial seed
lyrics as input. It can also take lyrics from professional lyrics writer to
generate the matching meaningful melodies.

##  2 Related Work

Here, we briefly present closely related work on lyrics-conditional music
generation frameworks developed by researchers in past years. Orpheus [10] is
a dynamic programming based melody composition algorithm for Japanese lyrics.
Orpheus is designed as an optimal melody search problem for the given lyrics
under the prosodic constraints of the Japanese lyrics. The authors design two
individual models such as rhythm and probabilistic pitch inference model to
generate melodies from the given lyrics. A Finish song generating system
called Sucus-Apparatusf is developed by Toivanen et al. [11]. Sucus-Apparatusf
is designed to randomly choose rhythm from the rhythm patterns actually found
in the Finish art songs. Further, a second order Markov model is designed to
generate the chord progression for the given lyrics. The pitches are generated
from the joint probabilistic distribution of previously generated notes and
the chords. In [12], the authors propose a system for automatically generating
melodic accompaniments from a given lyrical text. The system is designed to
generate the pitches modeled as n-gram models from the melodies of songs with
similar style. Further, the rhythm for the melodic accompaniment is derived
from the cadence information present in the text. The proposed method
generated hundreds of melodies by giving random options driven by set of
rules, followed by selecting among the generated options with an objective
measure that incorporates expert music knowledge. Both melody and rhythm are
generated by the same process for a given lyrics. Rhythm suggestion from
lyrics by using set of rules is studied by Nichols [13] while Oliveira [14]
proposed the inverse process of lyrics generation from the rhythm of melody.
ALYSIA [1] is the first fully data driven model based on random forests to
generate melody from the lyrics. ALYSIA is trained on a large set of features
manually extracted from Music-XML files. ALYSIA is designed to suggest
multiple melodies as output for a given lyrical piece thus giving user the
ability to choose more pleasing melody for the given lyrics. ALYSIA consists
of two independent melody prediction models to predict duration of the note
and scale of the note independently. An encoder-decoder based RNN sequential
model for lyrics-conditional melody generation for Chinese pop songs is
presented in [15]. The sequential model called Songwriter consists of two
encoders and one hierarchical decoder. The encoders are designed to encode the
lyric syllables and context melody of the prior generated melody. The
hierarchical decoder is designed to decode the note attributes such as pitch,
duration and syllable-note alignment labels since most of the syllables in
Chinese songs had more than one note.

##  3 Music Background

Melody can be defined as a sequence of meaningful musical notes and musical
rests. Let $N$ be the number of notes of a given melody. Each note of the
melody has two attributes:

  1. 1.

its pitch – called note or tone in common musical language – which is
represented by a letter followed by some number (e.g. C4) (and eventually a
$\flat$ (flat) or $\sharp$ (sharp) symbol (e.g. F$\sharp$5));

  2. 2.

its duration, which depends on the note type.

The common unit used for the duration is the beat. In most modern Western
music, four beats form a measure. A note which lasts for one beat (or a
quarter measure) is called a quarter-note. Accordingly other notes can be
created: sixteenth, eighth, etc. Then, between successive notes there is
either a rest or no rest. Again, the rest duration unit is the beat.

Therefore, a melody of length $N$ can be defined as
$Z=\\{\mathbf{z}_{1},\mathbf{z}_{2},\dots,\mathbf{z}_{N}\\}$, where each
$\mathbf{z}_{i}$’s are {note pitch, note duration, rest duration} triplets
(which is the representation used for this work). The $i$-th rest duration
value is the duration of the rest before the $i$-th note. A null rest duration
means no rest before the note.

We can define lyrics as sequence of natural language tokens which can
eventually make up song consisting of choruses and verses. The lyrics token
representation adopted in our framework is a syllable. A syllable is a part of
word or a single-syllable word. Each syllable is associated with melody
attributes: pitch, duration and rest to from the lyric-melody pairs.

We can represent lyrics-melody pair graphically via music scores. Each
syllable is associated with the corresponding note (see Figure 1). The shape
of the notes gives their duration, and their vertical position their pitch.
The duration of a rest is given by its shape. Most of the time, melodies have
perfect scale consistency, meaning that the pitches of the notes composing the
melody all belong to the same scale. A scale is a subset of pitches which have
properties such that they sound good when consecutively played.

##  4 Probabilistic Modeling of AutoNLMC

The proposed AutoNLMC is designed to first generate lyrics given a piece of
seed lyrics in the form of text and then composes the melody for the generated
lyrics sequentially one sentence at a time. We can model the AutoNLMC as a
probabilistic model with two conditionally dependent components: a lyrics
generator and a melody composer sequential model. Lyrics generator is modeled
as a conditional distribution to predict the next lyrical token given the
previous tokens of the lyric sequence modeled as
$p(s_{t}|s_{1},s_{2},...,s_{t-1})$. Here, the tokens are the sequence of
syllables of the lyrics given by $S=\\{s_{1},s_{2},...,s_{T}\\}$. We can learn
a probability distribution such as

$p(S)=\prod_{t=1}^{T}p(s_{t}|s_{1},s_{2},...,s_{t-1})$ (1)

the learned probability distribution is sampled one token at every time to
generate the full sequence lyrics.

We can model the Lyrics generator using recurrent neural network (RNN) to
implicitly learn the conditional probability distribution of the lyrics
generator. RNN can be trained to learn the probability distribution of a
sequence to predict the next token of the sequence at time $t$. RNN is
essentially an artificial neural network consisting of a hidden state $h$, an
output $y$, operates on a sequence of tokens to update the hidden states at
each time step $t$ given by

$h_{t}=g(h_{t-1},s_{t})$ (2)

where $g$ is a non-linear function mapping from input and previous state to
the present state. The non-linear function $g$ can be a simple sigmoid
function [16] or it can be a more complex gated unit such as long short-term
memory (LSTM) [17]. Melody composer is modeled as a conditional distribution
model over melody sequence on a lyrics sequence given by
$p(m_{1},m_{2},..,m_{T}|s_{1},s_{2},...,s_{T})$ where
$M=\\{m_{1},m_{2},..,m_{T}\\}$ is a melody sequence. The melody composer is
modeled as a sequential encoder-decoder model [18]. The encoder is an RNN
network which takes the lyrics tokens generated by lyrics generator model Eq.
1 at each time step sequentially to produce the encoder hidden states at time
$t$ defined in Eq. 3. The encoder also generates a variable context vector
$C_{i}$ which encodes the parts of the lyrics for which most attention to be
paid during melody decoding. The decoder is another RNN trained to predict the
melody token at time $t$ given the hidden state of the decoder RNN. The hidden
state of the decoder is computed recursively by

$h_{t}=g(h_{t-1},m_{t-1},C_{i})$ (3)

The next token distribution of the decoder can be similarly defined as

$p(m_{t}|m_{1},m_{2},....,m_{t-1},C_{i})=f(h_{t},m_{t-1},C_{i})$ (4)

The melody composer can be jointly trained to maximize the likelihood

$\operatorname*{max}_{\theta}\frac{1}{N}\sum_{i=1}^{N}\log
p_{\theta}(m_{i}|s_{i})$ (5)

where $\theta$ is the model parameters and the pair $m_{i},s_{i}$ is the input
and output to the melody composer.

Figure 2: Skip-gram model to extract lyrics embedding.

Figure 3: Figure shows the proposed Neural lyrics and melody composer
(AutoNLMC) model. AutoNLMC consists of a lyrics prediction module, encoder and
melody decoders. During inference, lyrics are predicted from the lyrics
prediction LSTM module shown in the rectangular red box in the lower half part
of the lyrics predictor with the help of softmax layers. While melody is
predicted by bypassing the softmax layers to predict pitch, duration and rest
with individual decoders as shown in figure.

##  5 Lyric to Vector (lyric2vec)

We train continuous skip-gram models to obtain the dense representation of the
input lyrics text. This enables us to learn high-quality real valued vector
representation of text tokens in an efficient way from large corpus of text
data [19]. The vector representations computed from the skip-gram model
explicitly encode many linguistic regularities and patterns which can be
represented as linear translations. We exploit the linear translation property
of embeddings to obtain the lyrics embedding representation to train deep
generative models.

The objective of the skip-gram model is to predict the surrounding context
tokens given a token at position $t$ in a piece of input text. We can define
the objective function more formally given a sequence of tokens such as
$s_{1},s_{2},...,s_{T}$. We want the model to maximize the log probability
[20] of the surrounding tokens given by

$\frac{1}{T}\sum_{t=1}^{T}\sum_{-c\leq i\leq c,i\neq 0}\log
p({s_{t+i}}|{s_{t}})$ (6)

where $c$ is the length of the context of $t^{th}$ token $s_{t}$.

The vanilla skip-gram model formulates the objective function as softmax
function given by

$p(s_{O}|s_{I})=\frac{\exp(v_{s_{0}}^{T}v_{s_{I}})}{\sum_{t=1}^{T}\exp(v_{s_{0}}^{T}v_{s_{I}})}$
(7)

where $v_{s_{I}}$ and $v_{s_{0}}$ are input and output vector representations
of token $s$, and $T$ is the number of unique tokens in the corpus. This
formulation is impractical given the large number of unique tokens/vocabulary
of the dataset. Hence, we use negative sampling [19] defined by

$\log\sigma(v_{s_{O}}^{T}v_{s_{I}})+\sum_{j=1}^{j=k}\operatorname{\mathbb{E}}_{s_{j}\sim
p_{n}(s)}[\log\sigma(v_{s_{j}}^{T}v_{s_{I}})]$ (8)

Here the objective of the negative sampling function is to discriminate the
actual token $s_{O}$ from the $k$ negative tokens drawn from the smoothed
noise distribution [21] $p_{n}(s)$ given by

$p_{n}(s)=\frac{f(s)^{\alpha}}{\sum_{c}f(s)^{\alpha}}$ (9)

where $f(s)$ is token frequency $s$, and $\alpha$ is real valued distribution
smoothing parameter. We train skip-gram models to obtain the embedding vectors
of lyrics at syllable and word level. We divide the lyrics of each song into
sentences, each sentence into words and each word into syllables. We treat
syllables $S=\\{s_{1},s_{2},...,s_{n}\\}$ as tokens to train the syllable
level embedding model and words $W=\\{w_{1},w_{2},w_{3},...,w_{n}\\}$ as
tokens for training word level embedding model. We train skip-gram model as a
logistic regression with stochastic gradient decent as optimizer, learning
rate with an initial value 0.03 is gradually decayed every epoch until 0.0007.
We use $c=7$ tokens context window and negative sampling distribution
parameter $\alpha$ is set to 0.75. We trained the models to obtain the
syllable and world level embedding vectors of dimensions $v=50$. The skip-gram
model to extract the lyric embeddings is shown in Fig. 2. The input one-hot
vector representing the syllable or word is compressed to a low dimensional
dense representation by linear units to predict the context tokens. The
following embedding representations are explored individually to train the
lyrics prediction and melody composition model: 1) syllable embeddings (SE)
where each syllable $s_{i}$ is encoded with $v$ dimensional vector from
syllable embedding model. 2) syllable and corresponding word embedding
concatenation (SWC) where a syllable $s_{i}$ is concatenated with
corresponding word embedding $w_{i}$ of $v$ dimension. 3) addition of syllable
and word vector (ASW) where syllable vector is added element wise to the word
vector. 4) concatenated syllable, word and syllable projected word vector
(CSWP). In CSWP, we project the syllable embedding vector $s_{i}$ onto the
corresponding word vector $w_{i}$ to give the wordness to syllable $s_{i}$ by

$proj_{w_{i}}s_{i}=\frac{s_{i}.w_{i}}{|w_{i}|^{2}}w_{i}$ (10)

finally, we concatenate $s_{i}$, $w_{i}$ and $proj_{w_{i}}s_{i}$ to form an
embedding vector for syllable $s_{i}$.

##  6 Lyrics and Melody Composer

The proposed lyrics and melody composer is a sequential encoder-decoder model
trained end-to-end to predict lyrics and compose the melody for the predicted
lyrics. Initially, AutoNLMC takes a piece of seed lyrics as input and starts
generating the lyrics one sentence at a time with syllables as tokens. For
each generated lyrics, the melody is composed until the last sentence. At the
end, all pairs of generated lyrics and melody are stitched together to form
the complete song.

###  6.1 Neural Lyrics Generator and Melody Composer

The proposed neural lyrics and melody composer (AutoNLMC) is shown in Fig. 3.
The AutoNLMC is a sequential RNN encoder-decoder network consists of a lyric
generator, encoder and melody decoder. The RNN sequential lyrics generator and
encoder are clubbed together during training as shown in Fig. 3. The melody
decoder is designed to decode the melody attributes in the form of MIDI.
Specifically, the lyric generator takes the sequence of lyric tokens $S$ as
input where each token is embedded into a fixed size dense vector by the skip-
gram model discussed in section 5 to generate the next lyrics token. Each
generated lyrics token is encoded by a RNN encoder to a fixed dimensional
hidden vector $h_{t}$ at each time step. The melody decoder uses the encoded
hidden vector $h_{t}$ and the dynamic context vector $C_{t}$ to generate the
melody decoder hidden state at each time step to generate the melody sequence.
The decoder consists of three independent MIDI decoders for each attribute
receives the lyric context vector $C_{t}$ and decoder state to generate the
pitch, duration and rest MIDI attributes for each syllable. We design
independent decoder for each attribute to reduce the number of classes and
hence to increase the accuracy of the predicted output, which is partially
motivated by the separate pitch and rhythm models in [1]. Each unit in the RNN
is modeled with gated LSTM [16]. The hidden state $h_{t}$ of the lyrics
generator is computed by

$h_{t}=(1-z_{t})\circ h_{(t-1)}+z_{t}\circ\bar{h}_{t}$ (11)

$\bar{h}_{t}=\tanh(Ws_{t}-1+U[r_{i}\circ h_{t-1}])$ (12)

$z_{t}=\sigma(W_{z}s_{t}+U_{z}h_{t-1})$ (13)

$r_{t}=\sigma(W_{r}s_{t}+U_{r}h_{t-1})$ (14)

where $z_{t}$ and $r_{t}$ are the reset and update gates of the LSTM unit.
$\sigma$ is the non-linear sigmoid function. The reset gate makes the hidden
state to forgot the past sequence information irrelevant to predict the future
sequence. While, the update gate controls the information flow from previous
hidden states to current state, and further acts as memory cell to remember
long term sequential dependencies which helps to learn the semantic
information of sequences. The variables $W,W_{z},W_{r},U,U_{z},U_{r}$ are the
weights of the LSTM unit. The weights are learned automatically during
training. The lyrics generator is trained to maximize the log conditional
probability

$\operatorname*{max}_{\theta}\frac{1}{N}\sum_{i=1}^{N}\log
p_{\theta}(s_{i}|s_{<i})$ (15)

where $s_{i}$ denotes the target lyrics token of the $i^{th}$ example and
$s_{<i}$ is shorthand notation for $\\{s_{1},s_{2},...,s_{t-1}\\}$.

The melody decoder decodes the melody attributes
$m_{i}=\\{p_{i},d_{i},r_{i}\\}$ i.e., pitch, duration, and rest independently
by maximizing the log conditional probability

$\operatorname*{max}_{\theta}\frac{1}{N}\sum_{i=1}^{N}\log
p_{\theta}(m_{i}|s_{i})$ (16)

The decoder states $\tilde{h}_{t}$ where
$\tilde{h}_{t}=\\{\tilde{h}_{pt},\tilde{h}_{dt},\tilde{h}_{rt}\\}$ are the
states of individual decoders given the encoder hidden state $h_{t}$ is
computed by

$\tilde{h}{{}_{t}}=(1-z_{t})\circ\tilde{h}_{t-1}+z_{t}\circ\bar{\tilde{h}}_{t}$
(17)

$\bar{\tilde{h}}_{t}=\tanh(Wm_{t-1}+U[r_{i}\circ\tilde{h}_{t-1}]+Cc_{t})$ (18)

$z_{t}=\sigma(W_{z}m_{t-1}+U_{z}\tilde{h}_{t-1}+C_{z}c_{t})$ (19)

$r_{t}=\sigma(W_{r}m_{t-1}+U_{r}\tilde{h}_{t-1}+C_{r}c_{t})$ (20)

The free variables $W,W_{z},W_{r},U,U_{z},U_{r},C,C_{z},C_{r}$ are all
learnable parameters.

The attention vector $c_{t}$ to align the input lyrics with the melody by
computing alignment model [22]

$c_{t}=\sum_{j=1}^{T}\alpha^{tj}h_{t}$ (21)

$\alpha^{tj}=\frac{exp(e^{tj})}{\sum_{k=1}^{T}exp(e_{tk})}$ (22)

$e^{tj}=V^{T}_{a}\tanh(W_{a}\tilde{h}_{t-1}+U_{a}h_{t})$ (23)

the parameters $V,W_{a},U_{a}$ are the learnable weights.

The neural lyric and melody composition model is trained end-to-end to
minimize the total loss function defined by

$\begin{split}L=-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{|M^{(i)}|}\log
p(M^{(i)}|\theta,S^{(i)},m_{<k}^{(i)},c^{(i)})\\\
-\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{|S^{(i)}|}\log
p(S^{(i)k}|S^{(i)}_{<k})\end{split}$ (24)

$\begin{split}L=-\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{|M^{(i)}|}\log
p(p^{(i)k},d^{(i)k},r^{(i)k}|\theta,S^{(i)},m_{<k}^{(i)},c^{(i)})\\\
-\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{|S^{(i)}|}\log
p(S^{(i)k}|S^{(i)}_{<k})\end{split}$ (25)

$\begin{split}L=-\frac{1}{N}\sum_{i=1}^{N}\bigg{(}\sum_{j=1}^{|M^{(i)}|}\log
p(p^{(i)k}|\theta,S^{(i)},p_{<k}^{(i)},c^{(i)})\\\ +\sum_{j=1}^{|M^{(i)}|}\log
p(d^{(i)k}|\theta,S^{(i)},d_{<k}^{(i)},c^{(i)})\\\ +\sum_{j=1}^{|M^{(i)}|}\log
p(r^{(i)k}|\theta,S^{(i)},r_{<k}^{(i)},c^{(i)})\bigg{)}\\\
-\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{|S^{(i)}|}\log
p(S^{(i)k}|S^{(i)}_{<k})\end{split}$ (26)

where $N$ is the set of $(S^{(i)},M^{(i)})$ lyric melody pairs,
$S^{(i)}=\\{s^{(i)1},s^{(i)2},....,s^{|S^{(i)}|}\\}$,
$M^{(i)}=\\{m^{(i)1},m^{(i)2},....,m^{|M^{(i)}|}\\}$ and
$m^{(i)1}=\\{p^{(i)1},d^{(i)1},r^{(i)1}\\}$.

The number of LSTM units used for all the models is 128. The initial state of
lyrics generator model is initialized with zero vector, where as the encoder
initial state is initialized with the last hidden state of lyrics generator
model. The pitch decoder, duration decoder and rest decoder of the melody
prediction model is initialized with the last hidden state of encoder model.
The loss function is minimized by Adam optimizer with initial learning rate of
0.0001 and linearly decayed after every 10 epochs. The model is trained to
minimize the cross entropy loss function with 32 batch size. All weight
matrices are initialized from zero mean, 0.02 variance Gaussian distribution.

###  6.2 Lyrics and Melody Inference

Lyrics inference is a process of generating new lyrics by combining the
previously generated lyrics from the conditional probability distribution to
generate the whole lyrics. Specifically, we obtain the next lyric token by
multinomial distribution output by the softmax non-linear activation function
as shown in the lower part of lyrics predictor of Fig. 3. At each time step,
we generate a lyrics token and feedback to the RNN unit at next time step to
generate new token. We use the following strategy to generate the lyrics from
the lyrics prediction model and evaluate the effectiveness of each strategies
in the evaluation section. i) Greedy search: at each time step, we pick the
most probable token from the softmax probability distribution given by

$\operatorname*{arg\,max}_{j}\
p(s_{(t,j)}=1|s_{(t-1)},...,s_{1})=\frac{\exp(w_{j}h_{t})}{\sum_{j^{{}^{\prime}}=1}^{K}exp(w_{j^{{}^{\prime}}}h_{t})}$
(27)

where $j=1,...,K$ are the unique lyric tokens, $w_{j}$ are the rows of the
LSTM weight matrix $W$. ii) Temperature sampling: The probability of each
lyric token is transformed to a freezing function $f_{\tau}$ [23] with a
controllable parameter $\tau$ before sampling most likely lyric token given by

$p_{j}=f_{\tau}(p_{j})=\frac{p_{j}^{\frac{1}{\tau}}}{\sum_{j=1}^{K}p_{j}^{\frac{1}{\tau}}}$
(28)

the value of $\tau\in[0,1]$ makes the model to predict more robust and diverse
lyrics.

The melody inference to predict the most probable melody sequence of the
generated lyrics is done at two modes: i) composing melody from the lyrics
generated from the lyrics generator and ii) composing melody from the original
lyrics created by the human lyrics writer. In the former case, we obtain the
lyrics from the lyrics generator. For each lyric token, we obtain the dense
representation from the skip-gram model. The attention vector at each time
step which aligns the input lyrics to the output pitch is computed from the
encoder hidden vectors. The initial state of the pitch decoder is initialized
with the last state of the encoder. At each time step, we feed the variable
attention vector and the decoder hidden state to the softmax probability
distribution function to predict the pitch corresponding to the lyric token.
To generate the full length sequence at each time step, we feed the previously
generated pitch tokens to generate the next tokens. We iterate this process
until we generate all pitch tokens corresponds to lyrics. Similarly, we follow
the same steps to decode the duration and rest attributes of all lyric tokens.
Finally, we club the predicted melody triplets i.e., pitch, duration and rest
and assign to the corresponding lyric syllable to form the lyrics melody
pairs. In case of composing melody from the original lyrics created by the
human lyrics writer, we feed the melody inference model with the lyrics
created by the human writer (here, lyrics prediction model is turned off.),
where one sentence at a time is generated to match the corresponding melody
form the model.

Figure 4: Pitch distribution.

Figure 5: Pitch interval distribution.

Figure 6: Pitch range distribution.

Figure 7: Histogram distribution of duration.

Figure 8: Histogram distribution of rest.

(a) Greedy search (b) $\tau=0.5$ (c) $\tau=0.6$ (d) $\tau=0.7$

(e) $\tau=0.8$ (f) $\tau=0.9$ (g) $\tau=1.0$

Figure 9: Pitch histogram distribution of the generated lyrics predicted from
melody decoder for greedy and temperature sampling methods with lyrics encoded
only with syllable embeddings (SE).

(a) Greedy search (b) $\tau=0.5$ (c) $\tau=0.6$ (d) $\tau=0.7$

(e) $\tau=0.8$ (f) $\tau=0.9$ (g) $\tau=1.0$

Figure 10: Pitch histogram distribution of the generated lyrics predicted from
melody decoder for greedy and temperature sampling methods with lyrics encoded
by concatenating syllable and word embeddings (SWC).

(a) Greedy search (b) $\tau=0.5$ (c) $\tau=0.6$ (d) $\tau=0.7$

(e) $\tau=0.8$ (f) $\tau=0.9$ (g) $\tau=1.0$

Figure 11: Pitch histogram distribution of the generated lyrics predicted from
melody decoder for greedy and temperature sampling methods with lyrics encoded
by adding syllable and word embeddings (ASW).

(a) Greedy search (b) $\tau=0.5$ (c) $\tau=0.6$ (d) $\tau=0.7$

(e) $\tau=0.8$ (f) $\tau=0.9$ (g) $\tau=1.0$

Figure 12: Pitch histogram distribution of the generated lyrics predicted from
melody decoder for greedy and temperature sampling methods with lyrics encoded
by concatenating syllable, word and syllable projected word vector embedding
(CSWP).

(a) Syllable embeddings (SE) (b) syllable and word embedding concatenation
(SWC) (c) addition of syllable and word embeddings (ASW) (d) concatenated
syllable, word and syllable projected word vector (CSWP)

Figure 13: Pitch histogram distribution of melody generated from the lyrics
from test dataset for various lyrics embedding representations.

Figure 14: Objective measures for various $\tau$ values and embedding
representations for the melody composed from generated lyrics.

Figure 15: Objective measures for lyric embedding representations for test set
lyrics.

Figure 16: BLEU score evaluation of pitch, duration and rest melody attributes
for SE, SWC, ASW and CSWP embedding representations. Abscissa represents the
1, 2, 3, 4, and 5 gram BLEU scores respectively.

(a)

(b)

(c)

(d)

Figure 17: Music sheet scores for generated lyrics-melody pairs for $\tau=0.8$
for SE, SWC, ASW and CSWP are show in Figs. a), b), c) and d) respectively.

##  7 Dataset Description

The dataset used in our experiments initially created in [9], which come from
two different sources: a) the ”LMD-full dataset” of the Lakh MIDI Dataset v0.1
111https://colinraffel.com/projects/lmd/ and a dataset found on a reddit
thread called ”the largest midi collection on the
internet”222https://https://www.reddit.com/r/datasets/comments/3akhxy/the_largest_midi_collection_on_the_internet/.
The LMD-full dataset contains a total of 176,581 different MIDI files but,
after taking only MIDI files with sufficient English lyrics, only 7,998 files
could be used. The largest midi collection on the internet dataset contains
130,000 different MIDI files but only 4,025 with enough English lyrics. These
two datasets were merged together to get a total of 12,023 MIDI files. For our
experiments, we parsed the dataset into three different formats: a) The
syllable level: This format is the lowest level that pair together every notes
and the corresponding syllable and its attributes. b) The word level: This
format regroups every notes of a word and gives the attributes of every
syllables that makes the word. c) The Sentence level: Similarly, this format
put together every notes that forms a syllable (or in most case, a lyric line)
and its corresponding attributes. Each syllable is represented with discrete
attributes: a) The Pitch of the note: In music, the pitch is what decide of
how the note should be played. We use Midi note number as an unit of pitch, it
can take any integer value between 0 and 127. b) The duration of the note: The
duration of the note in number of staves. It can be a quarter note, a half
note, a whole note or more. The exhaustive set of values it can take in our
parsing is [0.125, 0.25, 0.5, 0.75, 1, 1.5, 2, 4, 8, 16, 32]. and c) The
duration of the rest before the note: This value can take the same numerical
values as the duration but it can also be null (or zero).

The algorithm to parse the MIDI files to extract the lyric syllables and the
corresponding melody attributes is given in Algorithm 1. For the word and
sentence level, the algorithm is extremely similar, with just the part about
recognizing what is a word and what is a sentence that is added.

open a midi file;

$i\leftarrow 0$;

$TrackNumber\leftarrow 0$;

while  _note’s timestamp and first lyric’s timestamp not almost equal_ do

// The first 5 notes of every tracks are checked until there is an equality

if  _$i=5$_ then

$i\leftarrow 0$;

$TrackNumber\leftarrow TrackNumber+1$;

else

$i\leftarrow i+1$;

end if

end while

// Here the TrackNumber is found Go on i-th first note of the found Track;

$ListNote\leftarrow[\ ]$;

// The iteration and parsing of the song begins here

while  _The song and the lyrics has not reached their end_ do

if  _The next note and lyric’s timestamp match_ then

$ListNote\leftarrow ListNote+CurrentNote$;

Iterate to the next note and next lyric;

else

if  _The note is a lot before the lyrics’s start_ then

Iterate to the next note;

else

Iterate to the next lyric;

end if

end if

end while

close the file;

Algorithm 1 Syllable level parsing

The pitch distribution of the dataset is shown in Fig. 4. From distribution
plot in Fig. 4, we can observe that most of our songs have pitch range between
60 and 80 midi values. With mean pitch value of approximately 75. The
histogram distribution of the duration of all our notes is shown in Fig. 7.
From Fig 7, we can observe that most of our notes are quarter notes followed
by half notes and other notes. Similarly, we can observe the distribution of
the rest attribute of our dataset in Fig. 8. From the rest distribution in
Fig. 8, we can observe that almost 90% of our songs have zero rest between the
notes. The only significant rest attribute between the notes is quarter, half
and full duration rests. We also plot the histogram distribution of the pitch
interval in the dataset which is show in Fig. 5. From Fig. 5, we can observe
that the pitch interval distribution can be approximated with a Gaussian with
zero mean and standard deviation of approximately 5. i.e., most of our
consecutive note sequences have smooth pitch changes and they span form 0 to 5
midi pitch values. This also signifies that we have a sufficient data which
can be modeled to fit the distribution. The pitch range distribution of all
songs in the dataset is shown Fig. 6. We can observe that an average, we have
approximately 15 tones gap between the minimum and the maximum note value in
each song with the variation of $\pm 5$ tones.

Table 1: Seed lyrics and the generate full length songs ($\tau$ = 0.6). Seed lyrics | Generated lyrics
---|---
i never needed you to be strong |  | i never needed you to be strong i never
---
saw you you any time i could we be got
i believe you i will always will be the one
i love you if you know it’s just a girl i can stop
the music in the night you can make me
feel the way now i know you will see something
that i need i feel alone i wanna be alone with
out a love to be a part how am i supposed to
chances are find me some where on your road |  | chances are find me some where on your road
---
so the morning comes before i believe katy she
walks the one is me pa pa pa to three years
to rock me pretending let me be the one we fly
i will be there when we have to let the music
we belong together i don’t know now i know
i know but i know i know i need i got
to believe i would n’t cry i would n’t let me
our world is miles of endless roads |  | our world is miles of end less roads a drummer
---
walks we always be mellow nun y happy
it’s i never mind in love and in your heart
that’s that ing i don’t need to know if are i find
your self i can never be my baby but i
can’t believe that i’m not the one i’m waiting for
the rain on the rain once makes me cry on t
he rain on the rain looking out that i still have

Figure 18: Subjective evaluation results for lyrics and lyric-melody pairs.
Lyrics and melodies are generated from ASW and CSWP of the proposed AutoNLMC
for $\tau=0.6$.

##  8 Experiments

We have experimented with greedy search and temperature sampling as discussed
in subsection 6.2 methods to generate the lyrics from the lyrics prediction
model. We generate 10 full length songs by constraining the number of
syllables generated in each song to be 100 syllable length. We use ten
variable length seed lyrics i.e., lyrics to start with generating full song
lyrics form the test dataset. Few examples of seed lyrics used to generate the
full length song lyrics and the generated lyrics are shown in Table 1. We
evaluate the quality of the generated lyrics qualitatively by comparing the
distributions of the melody composed from the generated lyrics and the dataset
melody distributions. That is, for all the generated lyrics, we compose the
melody by the melody composition model and compare the individual melody
attribute distributions: pitch, duration and rest with the dataset
distributions described in section 7. Also, as discussed in section 5, we
represent the lyrics tokens in various encoding forms and train the proposed
lyrics generator and melody decoder model one for each representation. The
histogram distribution of the pitch decoded from the generated lyrics for
greedy search and various temperature values $\tau$ of temperature sampling is
shown in Figs. 9, 10, 11, 12 for lyrics encoded with syllable embedding (SE),
addition of syllable and word vector (ASW), syllable and word embedding
concatenation (SWC) and concatenated syllable, word and syllable projected
word vector (CSWP) respectively. From Fig. 9, we can observe that the model
trained with syllable embeddings alone shows quite interesting distribution
similar to the dataset. But a careful observation reveals that the predicted
pitch is biased towards the lower pitch range. This signifies that the
syllables alone cannot capture the whole dataset distribution. The model
trained by concatenating syllable and word embeddings with an anticipation to
improve the distribution range does not succeeded in capturing underlying
distribution shown in Fig. 10. Instead, it confines the distribution to a
small range thus the model helps in preventing accidental abrupt change in
pitch between notes. Though this type of model is quite stable in generating
stable lyrics with no sudden jumps in pitch values between notes, it will be
less interesting because the generated songs will have very low pitch span and
less number of unique notes results in mostly monotonous melody for any given
lyrics. We found that the model trained with the sum of syllable and word
embedding vectors (ASW) shows interesting results with dense distribution
round the mean pitch value of the dataset distribution. Unlike the
distributions of models trained with SE and CSWP, ASW shows a more bell like
distribution similar to dataset distribution results in picking other pitches
with more likelihood. The pitch distribution for the model trained with CSWP
embedding vectors is shown in Fig. 12. We have concatenated the projection
syllable vector on world vector in order to give the wordness to the syllable
vector since syllable and word embedding models trained separately. The pitch
distribution of the model trained with CSWP captures the distribution range
similar to the dataset distribution but the model most likely predicts the
mean pitch of the dataset distribution resulting in predicting average melody
of the dataset. We can conclude that ASW and CSWP embeddings are most likely
suitable embeddings for generating meaningful melodies qualitatively. We can
also note that greedy search can generate at most one lyric for a seed lyric
but temperature based sampling can generate original lyric and melody pair for
each $\tau$. It should be noted that for various $\tau$, the range of the
distribution remains same with slight changes in the shape of the distribution
that signifies that the lyrics generated by temperature sampling is not random
and we can generate novel lyrics-melody pairs with each value of $\tau$. We
also noted that the $\tau\in[0.5-1.0]$ is the best range to generate the
lyrics-melody pairs which follows dataset distribution. Further, we can also
conclude that ASW representation of lyrics can learn the correlation between
the generated lyrics and melody in an effective manner. Since we can also
generate the melody by feeding the original human generated lyrics to our
model, we have composed the melody for 10 randomly chosen full length lyrics.
The pitch histogram distribution shown in Fig. 13 also confirms that the ASW
and CSWP lyric embedding representations are better representations for
capturing the lyric-melody correlations. We also investigate the musical note
attribute related objective measures proposed in [6]: 1) unique tones, which
measures the total number of unique tones generated by the model, 2) maximum
tone value, which gives the highest tone value generated, 3) minimum tone
value, which represents the minimum tone value generated and 4) tone span,
which is the range of the pitch values present in the generated melody to
validate the claim. Fig. 14 shows the tone related objective measures for the
melody composed from generated lyrics for various $\tau$ values and lyric
embedding representations. From the left top Fig. 14, we can observe that ASW
and CSWP embedding representations generate a large number of unique tones
compare to other representations. Similarly, ASW and CSWP representations are
capable of generating very low and very high tone values which is consistent
with the dataset distribution which can be observed from left bottom and right
top of Fig. 14. The tone span plotted in the bottom right of Fig. 14 also
shows that ASW and CSWP representations are capable of generating larger tone
span melodies compared to other representations. The tone related objective
measures in Fig. 14 also reveal that irrespective of $\tau$ value, the
objective measures remains constant for SE and SWC which indicated that these
representations are not capable of generating expressive melodies for
different values of $\tau$ even though generated lyrics are quite different
for each $\tau$ value. Whereas ASW and CSWP are capable of generating
interesting tones for various values of $\tau$. The objective measures for the
pitch generated from the test set lyrics is shown in Fig. 14 also confirms
that ASW and CSWP are indeed good representations for generating melodies. In
[15], authors proposed to use BLEU (bilingual evaluation understudy) [24] as a
metric for evaluating the predicted pitch from lyrics. We also evaluate our
model with BLEU score for predicted pitch, duration and rest. BLEU is a single
real number value to evaluate the translation quality of the model from input
to output. Here, we compute the BLEU scores for the melodies generated from
the test set lyrics and corresponding ground truth melodies. Higher BLEU score
indicates better correlation between the generated and ground truth melodies.
The 1-gram, 2-gram, 3-gram, 4-gram and 5-gram BLEU scores of pitch, duration
and rest melody attributes for each embedding representations are shown in
Fig. 16. From Fig. 16, we can observe that ASW and CSWP representations out
performs other representations. Few samples of music sheet scores for the
generated lyrics and melody pairs by the proposed AutoNLMC is shown in Fig.
17. More samples can be found at
https://drive.google.com/file/d/1NTvo19CRzUqiUZokJaXM_4WVogXj1sdb/view?usp=sharing.

##  9 Subjective Evaluation

Although statistical and objective measures indicates that the model succeeds
in capturing the underlying data distribution for generating novel lyrics and
melodies, it is still difficult to conclude that the automatically generated
composition pleases human ears. Further, lyric writing and melody composition
are human creative process hence, it is very challenging to precisely quantify
the automatically generated lyrics and melodies objectively and statistically.
Hence, we adopt the subjective evaluation method proposed in [9] and [25] for
evaluating generated lyrics and melodies by our AutoNLMC. We ask the following
questions to participants during subjective evaluation

  * •

How meaningful are the predicted lyrics?

  * •

How well does the melodies fit the lyrics?

Participants rate the given samples on a five point discrete scale from 1 to 5
(where 1 corresponds to ”very bad”, 2 to ”bad”, 3 to ”ok”, 4 to ”good”, and 5
to ””very good”). We consider five full length lyrics and melodies generated
from AutoNLMC for subjective evaluation. We evaluate the generated lyrics and
melodies with baseline method and the ground truth human compositions. We
follow [9] and [25] to create baseline lyrics and melodies. The attributes of
the baseline melodies i.e., pitch, duration and rest are sampled from the
respective data distributions except for pitch whose MIDI values sampled
between 55-80, as most of the pitch distribution is concentrated in this range
in our dataset. The baseline lyrics are sampled from the most frequent
syllable lyrics vocabulary. We separately conduct the subjective evaluation
for generated lyrics and lyric-melody pairs. The subjective evaluation is
conducted through the Google sheets where Google sheets consists of the
samples understudy, radio buttons indicating scores to choose from, and clear
instructions on how to rate the samples based on 5 point scale. An example
evaluation sheet used for lyrics evaluation can be accessed from here
https://forms.gle/vZ6KnPDR6FgStizG9. We requested 20 adults who had at least
basic knowledge about lyrics, melodies and specifically about the music
composition for subjective evaluation. Out of 20 request participants, we have
obtained response from about 11 participants over the two weeks survey period.
We randomize 5 full length generated lyrics from the proposed AutoNLMC,
baseline and the ground truth lyrics. We ask the subjects to rate for the
question ”how meaningful are the presented lyrics”. Similarly, for lyrics-
melody pairs, we randomly choose lyrics-melody pairs from AutoNLMC, baseline
and the ground truth human compositions. We ask the raters to rate for the
question ”how well does the melodies fit the lyrics” on a five point scale.
The subjective evaluation results for ASW and CSWP for $\tau=0.6$ are shown in
Fig. 18. We can see that ASW is marginally better than CSWP. We can also
observe that AutoNLMC is close to the human compositions. The baseline
performs worst than the other methods without any surprise. We also see that
the ASW performs better than CSWP which is in correlation with the objective
measures of Section 8. From the subjective evaluation measures, we can find
that there is a gap between the human compositions and generated lyrics and
melodies by the proposed model. Which indicated that there is a lot of scope
to explore injecting prior musical knowledge to improve the current model.

##  10 Summary

In this paper, we proposed Automatic Neural Lyrics and Melody Composition
(AutoNLMC) to enable the human community to discover the original lyrics, and
the corresponding matching melodies. We trained lyrics to vector (lyric2vec)
models on a large set of lyrics-melody parallel dataset parsed at syllable,
word and sentence level to extract the lyric embeddings. The proposed AutoNLMC
is a encoder-decoder sequential recurrent neural network model consisted of
lyric generator, lyrics encoder and melody decoder for each note attribute
trained end-to-end to learn the correlation between the lyrics and melody
pairs with attention mechanism. AutoNLMC is designed to operate in two modes
such that it can generate both lyrics and corresponding melody automatically
for an amateur or person with no music knowledge, or it can take seed lyrics
from professional lyric writer to generate the corresponding melodies. The
qualitative and quantitative evaluation measures reveled that the proposed
method indeed capable of generating original lyrics and corresponding melody
for generating new songs.

## References

  * [1] M. Ackerman and D. Loker, “Algorithmic songwriting with alysia,” in _International Conference on Evolutionary and Biologically Inspired Music and Art_. Springer, 2017, pp. 1–16.
  * [2] F. Pachet and P. Roy, “Markov constraints: steerable generation of markov sequences,” _Constraints_ , vol. 16, no. 2, pp. 148–172, 2011.
  * [3] F. Pachet, A. Papadopoulos, and P. Roy, “Sampling variations of sequences for structured music generation.” in _ISMIR_ , 2017, pp. 167–173.
  * [4] E. Waite _et al._ , “Generating long-term structure in songs and stories,” _Magenta Bolg_ , 2016.
  * [5] H. Chu, R. Urtasun, and S. Fidler, “Song from pi: A musically plausible network for pop music generation,” _arXiv preprint arXiv:1611.03477_ , 2016\.
  * [6] O. Mogren, “C-rnn-gan: Continuous recurrent neural networks with adversarial training,” _arXiv preprint arXiv:1611.09904_ , 2016.
  * [7] H.-W. Dong, W.-Y. Hsiao, L.-C. Yang, and Y.-H. Yang, “Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [8] Y. Yu, S. Tang, F. Raposo, and L. Chen, “Deep cross-modal correlation learning for audio and lyrics in music retrieval,” _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ , vol. 15, no. 1, p. 20, 2019.
  * [9] Y. Yu and S. Canales, “Conditional lstm-gan for melody generation from lyrics,” _arXiv preprint arXiv:1908.05551_ , 2019.
  * [10] S. Fukayama, K. Nakatsuma, S. Sako, T. Nishimoto, and S. Sagayama, “Automatic song composition from the lyrics exploiting prosody of the japanese language,” in _Proc. 7th Sound and Music Computing Conference (SMC)_ , 2010, pp. 299–302.
  * [11] J. Toivanen, H. Toivonen, A. Valitutti _et al._ , “Automatical composition of lyrical songs,” in _The Fourth International Conference on Computational Creativity_ , 2013.
  * [12] K. Monteith, T. R. Martinez, and D. Ventura, “Automatic generation of melodic accompaniments for lyrics.” in _ICCC_ , 2012, pp. 87–94.
  * [13] E. Nichols, “Lyric-based rhythm suggestion,” in _ICMC_ , 2009.
  * [14] H. G. Oliveira, “Tra-la-lyrics 2.0: Automatic generation of song lyrics on a semantic domain,” _Journal of Artificial General Intelligence_ , vol. 6, no. 1, pp. 87–110, 2015.
  * [15] H. Bao, S. Huang, F. Wei, L. Cui, Y. Wu, C. Tan, S. Piao, and M. Zhou, “Neural melody composition from lyrics,” _arXiv preprint arXiv:1809.04318_ , 2018\.
  * [16] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” _arXiv preprint arXiv:1406.1078_ , 2014.
  * [17] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [18] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in _Advances in neural information processing systems_ , 2014, pp. 3104–3112.
  * [19] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” _arXiv preprint arXiv:1301.3781_ , 2013\.
  * [20] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in _Advances in neural information processing systems_ , 2013, pp. 3111–3119.
  * [21] H. Caselles-Dupré, F. Lesaint, and J. Royo-Letelier, “Word2vec applied to recommendation: Hyperparameters matter,” in _Proceedings of the 12th ACM Conference on Recommender Systems_. ACM, 2018, pp. 352–356.
  * [22] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” _arXiv preprint arXiv:1409.0473_ , 2014\.
  * [23] R. Stewart, “Maximum likelihood decoding with rnns - the good, the bad, and the ugly,” _https://nlp.stanford.edu/blog/maximum-likelihood-decoding-with-rnns-the-good-the-bad-and-the-ugly/_ , 2016\.
  * [24] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th annual meeting on association for computational linguistics_. Association for Computational Linguistics, 2002, pp. 311–318.
  * [25] H.-P. Lee, J.-S. Fang, and W.-Y. Ma, “icomposer: An automatic songwriting system for chinese popular music,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , 2019, pp. 84–88.