# Benchmarking Generated Poses: How Rational is Structure-based Drug Design
with Generative Models?

Charles Harris
University of Cambridge
<EMAIL_ADDRESS>
Kieran Didi
University of Cambridge
<EMAIL_ADDRESS>
Arian R. Jamasb
University of Cambridge
<EMAIL_ADDRESS>
Chaitanya K. Joshi
University of Cambridge
<EMAIL_ADDRESS>
Simon V. Mathis
University of Cambridge
<EMAIL_ADDRESS>
Pietro Lio
University of Cambridge
<EMAIL_ADDRESS>Tom L. Blundell
University of Cambridge
<EMAIL_ADDRESS>
Correspondance to<EMAIL_ADDRESS>
###### Abstract

Deep generative models for structure-based drug design (SBDD), where molecule
generation is conditioned on a 3D protein pocket, have received considerable
interest in recent years. These methods offer the promise of higher-quality
molecule generation by explicitly modelling the 3D interaction between a
potential drug and a protein receptor. However, previous work has primarily
focused on the quality of the generated molecules themselves, with limited
evaluation of the 3D molecule _poses_ that these methods produce, with most
work simply discarding the generated pose and only reporting a “corrected”
pose after after redocking with traditional methods. Little is known about
whether generated molecules satisfy known physical constraints for binding and
the extent to which redocking alters the generated interactions. We introduce
POSECHECK, an extensive analysis of multiple state-of-the-art methods and find
that generated molecules have significantly more physical violations and fewer
key interactions compared to baselines, calling into question the implicit
assumption that providing rich 3D structure information improves molecule
complementarity. We make recommendations for future research tackling
identified failure modes and hope our benchmark can serve as a springboard for
future SBDD generative modelling work to have a real-world impact.

##  1 Introduction

Structure-based drug design (SBDD) [1, 2, 3] is a cornerstone of drug
discovery. It uses the 3D structures of target proteins as a guide to
designing small molecule therapeutics. The intricate atomic interactions
between proteins and their ligands shed light on the molecular motifs
influencing binding affinity, selectivity, and drug-like properties. Employing
computational methods such as molecular docking [4, 5], molecular dynamics
simulations [6], and free energy calculations [7], SBDD aids in the
identification and optimization of potential drug candidates.

Deep generative models for SBDD have recently attracted considerable attention
in the ML community [8, 9]. These models learn from vast compound databases to
generate novel chemical structures with drug-like properties [10]. By
explicitly integrating protein structure information, these models aim to
generate ligands with a higher likelihood of binding to the target protein. In
particular, advancements in geometric deep learning [11, 12, 13] have led to a
new suite of generative methods, enabling the design of 3D molecules directly
within the confines of the target protein [14, 15, 16, 17, 18]. These methods,
which concurrently generate a molecular graph and 3D coordinates, provide the
significant advantage of obviating the need for determining the 3D pose post
hoc through traditionally slow molecular docking programs – at least in
theory.

Assessing the quality of molecules generated by these methodologies is not
straightforward, with little work on experimental validation, especially for
de novo design [19]. Typical evaluation metrics (Figure 1a) focus primarily on
the 2D graph of the generated molecules themselves, measuring their
physicochemical properties (e.g. QED [20]) and adherence to drug discovery
heuristics (e.g. Lipinski’s Rule of Five [21]). For effective SBDD, we argue
that it’s equally essential to assess the quality of the generated _binding
poses_ and their capacity to satisfy known biophysical prerequisites for
binding (Figure 1b). This perspective is essential if these methods are to
serve as practical alternatives to traditional virtual screening approaches in
SBDD.

We hypothesise that multiple failure modes, undetected by currently applied
metrics, are inherent within these methods. The situation is further
complicated by the common practice of disregarding the initially generated
pose and then redocking the molecule to attain a potentially enhanced pose.
This strategy tends to focus on presenting only the outcomes of the redocked
molecule, making the accurate assessment of pose quality an increasingly
intricate challenge.

Our primary contributions are summarized as follows: We introduce POSECHECK, a
set of new biophysical benchmarks for SBDD models, expanding the traditional
‘pipeline-wide’ framework by integrating ‘component-wise’ metrics (i.e.
generated and redocked poses), leading to comprehensive and precise model
assessment. Utilizing this new framework, we evaluate a selection of high-
performing machine learning SBDD methods, revealing two key findings: (1)
generated molecules and poses often contain nonphysical features such as
steric clashes, hydrogen placement issues, and high strain energies, and (2)
redocking masks many of these failure modes. Based on these evaluations, we
propose targeted recommendations to rectify the identified shortcomings. Our
work thus provides a roadmap for addressing critical issues in SBDD generative
modelling, informing future research efforts.

Figure 1:  Top: Overview of a conventional pipeline of SBDD with 3D generative
modelling. A generative model is usually trained using experimental or
synthetic protein-ligand complexes, from which new molecules and poses can be
sampled de novo. Typically, generated poses are discarded and redocked into
the receptor, and primarily evaluated on 2D molecular graphs (e.g. QED). The
effect of redocking on the final complex is often unknown, preventing
understanding of the common failure modes of the trained model and therefore
inhibiting progress. Bottom: the POSECHECK benchmarks for generated poses
include pipeline-wide as well as component-wise metrics, enabling a targeted
evaluation of each model component guiding further model development.

##  2 Background

#### Deep Generative Models for 3D Structure-based Drug Design

Many works have recently tried to recast the SBDD problem as learning the 3D
conditional probability of generating molecules given a receptor, allowing
users to sample new molecules completely de novo inside a pocket. Common
methods utilize Variational AutoEncoders (VAEs) [22], Generative Adversarial
Networks (GANs) [23], Autoregressive (AR) models and recently Denoising
Diffusion Probabilistic Models (DDPMs) [24]. LiGAN [14] uses a 3D
convolutional neural network combined with a VAE model and GAN-style training.
3DSBDD [15] introduced an autoregressive (AR) model that iteratively samples
from an atom probability field (parameterised by a Graph Neural Network) to
construct a whole molecule, with an auxiliary network deciding when to
terminal generation. Pocket2Mol [16] extended this work with a more efficient
sampling algorithm and better encoder. DiffSBDD [17] and TargetDiff [17] are
both conditional DDPMs conditioned on the 3D target structure. All methods
extract a point cloud from the end of the generative process and then draw
bonds based on the distances between generated atoms using OpenBabel [25].

#### Related work

Guan et al. [17] perform limited analysis of small chemical sub-features, such
as agreement to experimental atom-atom distances and the correctness of
aromatic rings within the generated molecule. Baillif et al. [19] emphasize
the necessity of 3D benchmarks for 3D generative models. However, both of
these works study the molecules in isolation rather than the protein-ligand
context.

##  3 Methods

In order to evaluate the quality of generated poses and their capacity to
facilitate high-affinity protein-ligand interactions, we present a variety of
computational methods and benchmarks in this section. These methodologies
provide a thorough perspective on the poses produced and illuminate the
ability of generative models to generate trustworthy and significant ligand
conformations. Full implementation details are given in Appendix A.

#### Interaction fingerprinting

Interaction fingerprinting is a computational method utilized in SBDD to
represent and analyze the interactions between a ligand and its target
protein. This approach encodes specific molecular interactions, such as
hydrogen bonding and hydrophobic contacts, in a compact and easily comparable
format – typically as a bit vector, known as a _interaction fingerprint_ [26,
27]. Each element in the interaction fingerprint corresponds to a particular
type of interaction between the ligand and a specific residue within the
protein binding pocket. In a similar way to chemical similarity [28], encoding
these interactions enables rapid comparison of different ligands or binding
poses by computing the Tanimoto similarity between fingerprints:

$T(A,B)=\frac{{|A\cap B|}}{{|A|+|B|-|A\cap B|}}$

where $A$ and $B$ are sets, and $|A|$ represents the cardinality (size) of set
$A$. We compute interactions using the ProLIF library [26].

#### Steric clashes

In the context of molecular interactions, the term _steric clash_ is used when
two neutral atoms come into closer proximity than the combined extent of their
van der Waals radii [29]. This event indicates an energetically unfavourable
[30], and thus physically implausible, interaction. The presence of such a
clash often points towards the current conformation of the ligand within the
protein being less than optimal, suggesting possible inadequacies in the pose
design or a fundamental incompatibility in the overall molecular topology.
Hence, the total number of clashes serves as a vital performance metric in the
realm of SBDD. We stipulate a clash to occur when the pairwise distance
between a protein and ligand atom falls below the sum of their van der Waals
radii, allowing a clash tolerance of 0.5 Å.

#### Strain-energy

Strain energy refers to the internal energy stored within a ligand as a result
of conformational changes upon binding. When a ligand binds to a protein, both
the ligand and the protein may undergo conformational adjustments to
accommodate each other, leading to changes in their bond lengths, bond angles,
and torsional angles. These changes can cause strain within the molecules,
which can affect the overall binding affinity and stability of the protein-
ligand complex [31]. Whilst there is always a trade-off between enthalpy and
entropy, generally speaking, lower strain energy results in more favourable
binding interactions and potentially more effective therapeutics. We calculate
the strain energy as the difference between the internal energy of a relaxed
pose and the generated pose (without pocket). Both relaxation and energy
evaluation are computed using the Universal Force Field (UFF) [32] using
RDKit.

#### Evaluation by redocking

Our final assessment involves measuring the level of agreement between the
docking programs and the molecules produced by the learned distribution in the
generative model. Although this is the most coarse-grained approach we employ
and docking programs come with their inherent limitations, they nevertheless
contain useful proxies and serve as valuable tools for comparison. In this
procedure, we redock the generated pose into the pocket using QuickVina2 [5].
Following this, we compute the Root Mean Squared Deviation (RMSD) between the
generated pose and the docking-predicted one across all generated molecules,
thereby obtaining a distribution of RMSD values.

##  4 Results

Figure 2: RSMD between the generated and redocked poses using the popular
docking framework Vina. Left: Violin plots of all redocking RMSDs. CrossDocked
violin is where we have redocking the poses from the original dataset. Right:
Illustrative examples (generated in magenta, redocked in green). We observe
that the generated and redocked poses often differ significantly. The top
figure shows an example of RMSD 2Å, which already results in a clearly
different pose. Failure modes such as entire flips of a generated molecule
upon re-docking are not uncommon, indicating that the generated pose is often
in disagreement with the principles behind common docking software.

###  4.1 Experimental Setup

In our study, we evaluate the quality of poses from five recent methods: LiGAN
[14], 3DSBDD [15], Pocket2Mol [16], TargetDiff [17] and DiffSBDD [18]. All
models were trained on the CrossDocked2020 [33] dataset using the dataset
splits computed in Peng et al. [16], which used a train/test split of 30%
sequence identity to give a test set of 100 target protein-ligand complexes
which we use for evaluation. For each model, we sampled 100 molecules per
target. During inference, the model is given a reduced PDB file containing
only the atoms for a single pocket within the test set, so there is no element
of blind docking during generation or subsequent redocking111Note illustrative
figures may show full proteins..

###  4.2 Agreement with docking functions

#### Results

In an effort to discern the impact of redocking within 3D generative SBDD
post-processing pipelines, our preliminary action involves determining the
extent to which the redocked pose preserves accurate information from the
initially generated binding mode. Therefore, we proceed to measure the RMSD
between the pose our model has generated and the pose that has been redocked
using QuickVina2 [5]. A lower RMSD value would denote a higher degree of
agreement between the generated binding modes and the scoring function. To
provide perspective, it’s worth noting that a carbon-carbon bond generally
measures 1.54 Å in length.

The density distributions of various methods are illustrated in Figure 2. All
methods exhibit remarkably high average redocking RMSD scores, with no method
achieving a median below 2 Å. Interestingly, the diffusion methods perform
worse than the AR (Auto-Regression) methods, as evidenced by DiffSBDD and
TargetDiff achieving median scores of 2.83 and 3.19 Å, respectively, while
Pocket2Mol and 3DSBDD achieve 2.57 and 2.83 Å, respectively. LiGAN performs
similarly to the diffusion methods, with a median score of 3.27. It is worth
noting that DiffSBDD shows a significantly wider range of scores compared to
all other methods, with a standard deviation of 1.54Å, whereas other methods
have values ranging from 0.83 to 1.04 Å. Full extended results for all section
are given in Appendix B.

#### Discussion

Figure 3: Interactions between protein and ligands as seen in reference
datasets (green) and generated molecules (orange). The frequency of (a)
hydrogen bond acceptors, (b) hydrogen bond donors, (c) Van der Waals contacts
and (d) hydrophobic interactions are considered. We find that generative
models have significant trouble making hydrogen bond interactions compared to
baseline.

These findings raise concerns for several reasons. They expose the minimal
concordance between the binding models learned by these methods and the
established QuickVina2 methodology [5]. More critically, they underline the
lack of accurate evaluations of generative models’ capability to produce
realistic binding poses; instead, these models tend to generate drug-like
molecules with vague binding modes, later rectified through docking.
Simultaneously, we should consider that the similar performance observed in
the RMSD distribution (median 3.07 ÅRMSD) when redocking the CrossDocked test
set using QuickVina2 might suggest two potential scenarios. The CrossDocked
dataset could be plagued by quality issues, or this could reflect a variance
in the docking programs used - SMINA is utilized by CrossDocked [34], whereas
QuickVina2 is favoured by the broader ML community [15, 16, 17, 18].

#### Limitations

Furthermore, it may be that all methods have learnt excellent binding models
(or at least as good as smina [34]), but that we cannot resolve this due to
the use of QuickVina2 [5] as the scoring function. Future work could consider
performing this benchmarking using the same docking protocol as CrossDocked
[33] or higher-fidelity docking programs such as Glide [35]. However, we think
this further underlines the issues of the currently used evaluation paradigm.

###  4.3 Protein-ligand interaction analysis

Figure 4: Left: Interaction similarity between the generated
molecules/interactions and the CrossDocked test set molecules. Right: examples
of a molecule from the test set (green) and a generated pose from TargetDiff
(magenta).

#### Evaluation

Below describe the classes of interaction that we evaluate. Hydrogen bonds
(HBs) are a type of interaction that occurs between a hydrogen atom that is
bonded to a highly electronegative atom, such as nitrogen, oxygen, or fluorine
[36]. They are key to many protein-ligand interactions [37] and require very
specific geometries to be formed [38]. The directionality of HBs confers
unique identities upon the participating atoms: hydrogen atoms attached to
electronegative elements are deemed ‘donors’, whilst the atom accepting the HB
is termed an ‘acceptor’. Van der Waals contacts (vdWs) are interactions that
occur between atoms that are not bonded to each other. These forces can be
attractive or repulsive and are typically quite weak [39]. However, they can
be significant when many atoms are involved, as is typical in protein-ligand
binding [40]. Hydrophobic interactions are non-covalent interactions that
occur between non-polar molecules or parts of molecules in a water-based
environment. They are driven by the tendency of water molecules to form
hydrogen bonds with each other, which leads to the exclusion of non-polar
substances. This exclusion principle prompts these non-polar regions to orient
away from the aqueous environment and towards each other [41], thereby
facilitating the association between protein and ligand molecules [42].

#### Results

Distributions of various kinds of interactions are shown in Figure 3. We
consider whether our generative models can design molecules with adequate
hydrogen bonding and find that no method can match or exceed the baseline. In
the reference set, CrossDocked, the mode of HBs for both acceptors and donors
is 1, with means of 2.23 and 1.66 for acceptors and donors respectively.
Strikingly, we find that in all generated poses for all models (except LiGAN
HB acceptors), the _most common number of HB acceptors and donors is 0_ , with
means varying between 0.68-1.73 for HB acceptors and 0.39-0.85 for HB donors.
We find an average difference of 0.50 and 0.81 HBs between the best-performing
models and the baseline for acceptors and donors respectively. Results for Van
der Waals contacts and hydrophobic interactions are closer to the dataset
baseline.

#### Discussion

Conventional wisdom would suggest that many minor imperfections in the
generated pose would be simply fixed by redocking the molecule (e.g. moving an
oxygen atom slightly to complete a hydrogen bond.) We find this is in fact
rarely the case, with redocking sometimes being significantly deleterious (see
examples of LiGAN in Figure 3), suggesting that there are either limitations
in the docking function used or, more likely, the generated interaction was
physically implausible to begin with.

#### Interaction similarities to reference poses

We also consider the interaction similarities between the generated
poses/interactions and the baseline test set in order to study generated
binding modes (see Figure 4). Here, we take the bit vectors constructed for
Figure 3 and compute the Tanimoto distance between the reference pose and all
generated poses. We observe that all methods perform similarly (both generated
and redocked) except for LiGAN, which achieves the highest similarity before
being completely degenerated by redocking, suggesting their model has done
well to maximise placing atoms where they form interactions with the pocket,
but the poses themselves are not meaningful.

###  4.4 Clash scores

Figure 5: Clash scores.

#### Results

Figure 5 presents the results of the steric clash analysis. In summary, the
latest methods, particularly those employing diffusion models, exhibit poor
performance in terms of steric clashes compared to the baseline, with a
significant number of outliers. Although redocking mitigates clashes to some
extent, it does not always resolve the most severe cases.

The CrossDocked test set has a low number of clashes with few extreme
examples, with a mean of 4.59, upper quantile of 6 and maximum value of 17. In
terms of generated poses, the older methods perform best, with 3DSBDD and
LiGAN having means of 3.79 and 3.40 clashes respectively. Pocket2Mol, an
extension of 3DSBDD, performs worse with a mean clash score of 5.62 and upper
quantile of 8 clashes. Finally, the diffusion-based approaches perform the
worst with mean clash scores of 9.08 and 15.33 for TargetDiff and DiffSBDD
respectively. The tail end of their distributions is also high, with both
methods having upper quantiles of 11 and 18 respectively, with TargetDiff
having the worst case of 264 steric clashes. Redocking the molecules generally
fixed many clashes and improved scores. The mean clash score for Pocket2Mol
improves from 5.62 to 2.98, TargetDiff from 9.08 to 5.79 and DiffSBDD from
15.34 to 3.61.

#### Discussion

Interestingly, DiffSBDD and TargetDiff, which are considered state-of-the-art
based on mean docking score evaluations [17, 18], exhibit subpar performance
in their number of clashes. They aim to learn atom position distributions
without explicit constraints on final placements. While DiffSBDD starts with a
performance deficit, its enhanced clash mitigation during redocking elevates
its results to match the baseline, highlighting methodological distinctions
between it and TargetDiff. Notably, 3DSBDD and LiGAN show low clash scores,
with the former positioning atoms within a predefined voxel grid [15] and the
latter applying a clash loss [14].

Our findings affirm the assumption that redocking alleviates many minor
clashes, akin to the force-field relaxation step in AlphaFold2 [43]. We
initially speculated that molecules with clashes exceeding 100 had been
mistakenly generated inside the protein pocket. Yet, we often discovered
fragments within highly constrained nooks, especially worsened with the
addition of hydrogen atoms.

#### Limitations

An important consideration to bear in mind is that proteins are not entirely
rigid receptors. They can often experience limited conformational
rearrangements to accommodate molecules of varying shapes and sizes [44].
Consequently, conducting generation and redocking in a rigid receptor
environment may not yield accurate scores for potentially plausible molecules.
Note all these results are with a _generous_ clash tolerance of 0.5 Å (roughly
half the ddW radii of a hydrogen atom), in order to be able to resolve
differences between methods.

###  4.5 Strain energy

Figure 6: Left: CDF of strain energies. Right: Examples of molecules with high
strain energy.

#### Results

To conclude our study, we provide an analysis of the strain energy [31] of the
generated poses. Force field relaxation before docking is a common post-
processing step of many generative SBDD pipelines, masking some potential
issues with the generated geometries less clear. This allows us to evaluate
the generated molecules for things like unrealistic bond distance or
impossible geometries in rings.

Figure 6 displays the cumulative density function (CDF) of strain energy for
the generated molecules, with the CrossDocked dataset serving as a baseline
(Note: the x-axis is on a logarithmic scale). We focus on median values in our
discussion since they are more representative in this context due to the
presence of extreme outliers, with _mean_ values ranging from approximately
$10^{4}$ to $10^{15}$ kcal/mol. None of the generative methods yields
molecules exhibiting strain energy close to that of the test set, which has a
median strain energy of 102.5 kcal/mol.

#### Discussion

Intriguingly, both of the diffusion-based methodologies (DiffSBDD and
TargetDiff) perform similarly poorly, reporting median values of 1243.1 and
1241.7 kcal/mol, respectively. This could suggest issues with the currently
used noised schedules [45] of these methods for ultra-precise atom position
refinement (discussed in Section 5). 3DSBDD performs to the same order of
magnitude, with a median strain energy of 592.2 kcal/mol, suggesting that
placing atoms into a discretized voxel space [15], while good for avoiding
clashes, has a detrimental impact on the strain energy.

Pocket2Mol performs by far the best in terms of strain energy, with a median
of 194.9 kcal/mol. The method provides perhaps the finest-grained control over
exact coordinates generated, by first choosing a focal atom and then
generating a new atom coordinate directly using an equivariant neural network
[13, 16], which may allow for more precise placement. LiGAN exhibits the
highest strain energy, with a median value of 18693.8 kcal/mol, indicating the
poorest performance in this context.

#### Limitations

The exceedingly high strain energy values observed in this scenario should be
approached with considerable prudence. For comparison, the combustion of TNT
releases approximately 815 kcal/mol. [46]. This data is not to be perceived as
absolute, but rather illustrative of the extent to which our generated
geometries deviate markedly from the standard distribution for the force
field. This further underscores the existing issues. It is also conceivable
that these poses might not even be initialized within more sophisticated,
high-fidelity force fields [47].

##  5 Recommendations for future work

#### Exploring reduced-noise sampling strategies

Interestingly, both diffusion-based works (DiffSBDD and TargetDiff) performed
similarly in terms of strain energy (see Section 4.5). We hypothesize this may
be due to the injection of random noise into the coordinate features at all
but the last step of stochastic gradient Langevin dynamics samplings [48],
making it challenging to construct precise bond angles etc. Here, inspiration
could be taken from protein design. For example, Chroma develops a low-
temperature sampling regime to reduce the effect of noise [49], FrameDiff
effectively scales down injected noise [50], both resulting in a substantial
increase in sample quality with an acceptable decrease in sample diversity.

#### Heavily penalise steric clashes during training

All evaluated methods frequently create steric clashes, resulting in
physically unrealizable samples. We suggest that mitigating steric clashes is
key for the next generation of SBDD models. This could be done via extra loss
terms, for example, by including a distogram loss as in AlphaFold2 [43] or the
steric clash loss in LiGAN [14]. A similar loss-based approach has been
effective in mitigating chain-breaks diffusion models for protein backbone
design [50].

#### Consider representing hydrogens

Virtually all work in ML for structural biology chooses to not explicitly
represent hydrogen atoms [43, 16, 15, 50, 18, 17], under the assumption that
they can be _implicitly_ learned and reasoned over with deep neural networks.
However, our analysis of hydrogen bond networks within generated molecules
found that generative methods struggle to handle the precise geometries
required to make a hydrogen bond [38] (even when redocked). Despite the
increased computational cost, we therefore recommend that future work explores
their inclusion.

##  6 Conclusion

In conclusion, this work presents a comprehensive exploration of structure-
based drug design (SBDD) methodologies with deep generative models. We
advocate for the need to consider _both_ the quality of the generated
molecules _and_ the quality of the binding poses in these models, calling for
an expanded evaluation of SBDD. The application of deep generative models in
SBDD holds promise for developing innovative drug-like molecules. However, for
SBDD approaches to realise that potential, we must establish a rigorous
evaluation regimen of both the generated molecules and their interaction with
the target – as proposed in this paper. Our research provides a solid
evaluation regimen for future advancements in this field and we hope that it
stimulates further development towards more efficient drug discovery
processes.

##  7 Acknowledgements

The authors would like to thank Arne Schneuing and Yuanqi Du for their
discussions and help refining the ideas presented here.

##  8 Code Availability

We will make the code available for POSECHECK in due course.

## References

  * Blundell [1996] Tom L Blundell.  Structure-based drug design.  _Nature_ , 384(6604 Suppl):23–26, 1996.
  * Ferreira et al. [2015] Leonardo G Ferreira, Ricardo N Dos Santos, Glaucius Oliva, and Adriano D Andricopulo.  Molecular docking and structure-based drug design strategies.  _Molecules_ , 20(7):13384–13421, 2015.
  * Anderson [2003] Amy C Anderson.  The process of structure-based drug design.  _Chemistry & biology_, 10(9):787–797, 2003\.
  * Trott and Olson [2010] Oleg Trott and Arthur J Olson.  Autodock vina: improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading.  _Journal of computational chemistry_ , 31(2):455–461, 2010.
  * Alhossary et al. [2015] Amr Alhossary, Stephanus Daniel Handoko, Yuguang Mu, and Chee-Keong Kwoh.  Fast, accurate, and reliable molecular docking with quickvina 2.  _Bioinformatics_ , 31(13):2214–2216, 2015.
  * Klepeis et al. [2009] John L Klepeis, Kresten Lindorff-Larsen, Ron O Dror, and David E Shaw.  Long-timescale molecular dynamics simulations of protein structure and function.  _Current opinion in structural biology_ , 19(2):120–127, 2009.
  * Chipot and Pohorille [2007] Christophe Chipot and Andrew Pohorille.  _Free energy calculations_ , volume 86.  Springer, 2007.
  * Du et al. [2022] Yuanqi Du, Tianfan Fu, Jimeng Sun, and Shengchao Liu.  Molgensurvey: A systematic survey in machine learning models for molecule design.  _arXiv preprint arXiv:2203.14500_ , 2022.
  * Isert et al. [2023] Clemens Isert, Kenneth Atz, and Gisbert Schneider.  Structure-based drug design with geometric deep learning.  _Current Opinion in Structural Biology_ , 79:102548, 2023\.
  * Gómez-Bombarelli et al. [2018] Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik.  Automatic chemical design using a data-driven continuous representation of molecules.  _ACS central science_ , 4(2):268–276, 2018.
  * Bronstein et al. [2017] Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst.  Geometric deep learning: going beyond euclidean data.  _IEEE Signal Processing Magazine_ , 34(4):18–42, 2017.
  * Atz et al. [2021] Kenneth Atz, Francesca Grisoni, and Gisbert Schneider.  Geometric deep learning on molecular representations.  _Nature Machine Intelligence_ , 3(12):1023–1032, 2021.
  * Jing et al. [2020] Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror.  Learning from protein structure with geometric vector perceptrons.  _arXiv preprint arXiv:2009.01411_ , 2020.
  * Masuda et al. [2020] Tomohide Masuda, Matthew Ragoza, and David Ryan Koes.  Generating 3d molecular structures conditional on a receptor binding site with deep generative models.  _arXiv preprint arXiv:2010.14442_ , 2020.
  * Luo et al. [2021] Shitong Luo, Jiaqi Guan, Jianzhu Ma, and Jian Peng.  A 3d generative model for structure-based drug design.  _Advances in Neural Information Processing Systems_ , 34:6229–6239, 2021.
  * Peng et al. [2022] Xingang Peng, Shitong Luo, Jiaqi Guan, Qi Xie, Jian Peng, and Jianzhu Ma.  Pocket2mol: Efficient molecular sampling based on 3d protein pockets.  In _International Conference on Machine Learning_ , pages 17644–17655. PMLR, 2022.
  * Guan et al. [2023] Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma.  3d equivariant diffusion for target-aware molecule generation and affinity prediction.  _arXiv preprint arXiv:2303.03543_ , 2023.
  * Schneuing et al. [2022] Arne Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blundell, Pietro Lió, Carla Gomes, Max Welling, et al.  Structure-based drug design with equivariant diffusion models.  _arXiv preprint arXiv:2210.13695_ , 2022.
  * Baillif et al. [2023] Benoit Baillif, Jason Cole, Patrick McCabe, and Andreas Bender.  Deep generative models for 3d molecular structure.  _Current Opinion in Structural Biology_ , 80:102566, 2023\.
  * Bickerton et al. [2012] G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins.  Quantifying the chemical beauty of drugs.  _Nature chemistry_ , 4(2):90–98, 2012.
  * Lipinski et al. [2012] Christopher A Lipinski, Franco Lombardo, Beryl W Dominy, and Paul J Feeney.  Experimental and computational approaches to estimate solubility and permeability in drug discovery and development settings.  _Advanced drug delivery reviews_ , 64:4–17, 2012.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Goodfellow et al. [2014] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _arXiv preprint arXiv:1406.2661_ , 2014.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * O’Boyle et al. [2011] Noel M O’Boyle, Michael Banck, Craig A James, Chris Morley, Tim Vandermeersch, and Geoffrey R Hutchison.  Open babel: An open chemical toolbox.  _Journal of cheminformatics_ , 3(1):1–14, 2011\.
  * Bouysset and Fiorucci [2021] Cédric Bouysset and Sébastien Fiorucci.  Prolif: a library to encode molecular interactions as fingerprints.  _Journal of Cheminformatics_ , 13:1–9, 2021.
  * Marcou and Rognan [2007] Gilles Marcou and Didier Rognan.  Optimizing fragment and scaffold docking by use of molecular interaction fingerprints.  _Journal of chemical information and modeling_ , 47(1):195–207, 2007.
  * Willett et al. [1998] Peter Willett, John M Barnard, and Geoffrey M Downs.  Chemical similarity searching.  _Journal of chemical information and computer sciences_ , 38(6):983–996, 1998.
  * Ramachandran et al. [2011] Srinivas Ramachandran, Pradeep Kota, Feng Ding, and Nikolay V Dokholyan.  Automated minimization of steric clashes in protein structures.  _Proteins: Structure, Function, and Bioinformatics_ , 79(1):261–270, 2011.
  * Buonfiglio et al. [2015] Rosa Buonfiglio, Maurizio Recanatini, and Matteo Masetti.  Protein flexibility in drug discovery: from theory to computation.  _ChemMedChem_ , 10(7):1141–1148, 2015.
  * Perola and Charifson [2004] Emanuele Perola and Paul S Charifson.  Conformational analysis of drug-like molecules bound to proteins: an extensive study of ligand reorganization upon binding.  _Journal of medicinal chemistry_ , 47(10):2499–2510, 2004.
  * Rappé et al. [1992] Anthony K Rappé, Carla J Casewit, KS Colwell, William A Goddard III, and W Mason Skiff.  Uff, a full periodic table force field for molecular mechanics and molecular dynamics simulations.  _Journal of the American chemical society_ , 114(25):10024–10035, 1992.
  * Francoeur et al. [2020] Paul G Francoeur, Tomohide Masuda, Jocelyn Sunseri, Andrew Jia, Richard B Iovanisci, Ian Snyder, and David R Koes.  Three-dimensional convolutional neural networks and a cross-docked data set for structure-based drug design.  _Journal of chemical information and modeling_ , 60(9):4200–4215, 2020.
  * Koes et al. [2013] David Ryan Koes, Matthew P Baumgartner, and Carlos J Camacho.  Lessons learned in empirical scoring with smina from the csar 2011 benchmarking exercise.  _Journal of chemical information and modeling_ , 53(8):1893–1904, 2013.
  * Friesner et al. [2004] Richard A Friesner, Jay L Banks, Robert B Murphy, Thomas A Halgren, Jasna J Klicic, Daniel T Mainz, Matthew P Repasky, Eric H Knoll, Mee Shelley, Jason K Perry, et al.  Glide: a new approach for rapid, accurate docking and scoring. 1. method and assessment of docking accuracy.  _Journal of medicinal chemistry_ , 47(7):1739–1749, 2004.
  * Pimentel and McClellan [1971] George C Pimentel and AL McClellan.  Hydrogen bonding.  _Annual Review of Physical Chemistry_ , 22(1):347–385, 1971.
  * Chen et al. [2016] Deliang Chen, Numan Oezguen, Petri Urvil, Colin Ferguson, Sara M Dann, and Tor C Savidge.  Regulation of protein-ligand binding affinity by hydrogen bond pairing.  _Science advances_ , 2(3):e1501240, 2016.
  * Brown [1976] ID Brown.  On the geometry of o–h-o hydrogen bonds.  _Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography_ , 32(1):24–31, 1976\.
  * Andersson et al. [1998] Ylva Andersson, Erika Hult, Henrik Rydberg, Peter Apell, Bengt I Lundqvist, and David C Langreth.  Van der waals interactions in density functional theory.  _Electronic Density Functional Theory: Recent Progress and New Directions_ , pages 243–260, 1998.
  * Barratt et al. [2005] Elizabeth Barratt, Richard J Bingham, Daniel J Warner, Charles A Laughton, Simon EV Phillips, and Steve W Homans.  Van der waals interactions dominate ligand- protein association in a protein binding site occluded from solvent water.  _Journal of the American Chemical Society_ , 127(33):11827–11834, 2005.
  * Meyer et al. [2006] Emily E Meyer, Kenneth J Rosenberg, and Jacob Israelachvili.  Recent progress in understanding hydrophobic interactions.  _Proceedings of the National Academy of Sciences_ , 103(43):15739–15746, 2006.
  * Patil et al. [2010] Rohan Patil, Suranjana Das, Ashley Stanley, Lumbani Yadav, Akulapalli Sudhakar, and Ashok K Varma.  Optimized hydrophobic interactions and hydrogen bonding at the target-ligand interface leads the pathways of drug-designing.  _PloS one_ , 5(8):e12029, 2010.
  * Jumper et al. [2021] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al.  Highly accurate protein structure prediction with alphafold.  _Nature_ , 596(7873):583–589, 2021.
  * Davis and Teague [1999] Andrew M Davis and Simon J Teague.  Hydrogen bonding, hydrophobic interactions, and failure of the rigid receptor hypothesis.  _Angewandte Chemie International Edition_ , 38(6):736–749, 1999.
  * Chen [2023] Ting Chen.  On the importance of noise scheduling for diffusion models.  _arXiv preprint arXiv:2301.10972_ , 2023.
  * Rinkenbach [1930] Wm H Rinkenbach.  The heats of combustion and formation of aromatic nitro compounds.  _Journal of the American Chemical Society_ , 52(1):115–120, 1930.
  * Brooks et al. [2009] Bernard R Brooks, Charles L Brooks III, Alexander D Mackerell Jr, Lennart Nilsson, Robert J Petrella, Benoît Roux, Youngdo Won, Georgios Archontis, Christian Bartels, Stefan Boresch, et al.  Charmm: the biomolecular simulation program.  _Journal of computational chemistry_ , 30(10):1545–1614, 2009.
  * Welling and Teh [2011] Max Welling and Yee W Teh.  Bayesian learning via stochastic gradient langevin dynamics.  In _Proceedings of the 28th international conference on machine learning (ICML-11)_ , pages 681–688, 2011.
  * Ingraham et al. [2022] John Ingraham, Max Baranov, Zak Costello, Vincent Frappier, Ahmed Ismail, Shan Tie, Wujie Wang, Vincent Xue, Fritz Obermeyer, Andrew Beam, et al.  Illuminating protein space with a programmable generative model.  _bioRxiv_ , pages 2022–12, 2022.
  * Yim et al. [2023] Jason Yim, Brian L Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola.  Se (3) diffusion model with application to protein backbone generation.  _arXiv preprint arXiv:2302.02277_ , 2023.

##  Appendix A Extended Implementation

All generative methods accessed were trained using the same dataset and splits
as proposed in Peng et al. [16]. For all methods, docking was performed using
QuickVina2. For generated poses, we sourced molecules from Schneuing et al.
[18] for DiffSBDD, and Guan et al. [17] for CrossDocked, TargetDiff,
Pocket2Mol, 3DSBDD and LiGAN (where they also performed redocking). We will
provide an updated version of this manuscript where we perform all redocking
using entirely our own pipeline.

##  Appendix B Extended results

In Figure 7, we provide the per target redocking RMSDs per method. Figure 8
and 9 show the number of steric clashes per target for the generated and
redocked poses respectively.

Figure 7: Redocking RMSD per method per target for CrossDocked test set. Order
is determined arbitrarily by median score per target for DiffSBDD.  Figure 8:
Steric clashes per method per target for generated poses in the CrossDocked
test set. Order is determined arbitrarily by median score per target for
DiffSBDD.  Figure 9: Steric clashes per method per target for redocked poses
in the CrossDocked test set. Order is determined arbitrarily by median score
per target for DiffSBDD.


# K-pop Lyric Translation: Dataset, Analysis, and Neural-Modelling

Haven Kim  Graduate School of Culture Technology
KAIST
<EMAIL_ADDRESS>Jongmin Jung  Dasaem Jeong  Department of Artificial
Intelligence
Sogang University
<EMAIL_ADDRESS>Department of Art and Technology
Sogang University
<EMAIL_ADDRESS>Juhan Nam  Graduate School of Culture Technology
KAIST
<EMAIL_ADDRESS>
###### Abstract

Lyric translation, a field studied for over a century, is now attracting
computational linguistics researchers. We identified two limitations in
previous studies. Firstly, lyric translation studies have predominantly
focused on Western genres and languages, with no previous study centering on
K-pop despite its popularity. Second, the field of lyric translation suffers
from a lack of publicly available datasets; to the best of our knowledge, no
such dataset exists. To broaden the scope of genres and languages in lyric
translation studies, we introduce a novel singable lyric translation dataset,
approximately 89% of which consists of K-pop song lyrics. This dataset aligns
Korean and English lyrics line-by-line and section-by-section. We leveraged
this dataset to unveil unique characteristics of K-pop lyric translation,
distinguishing it from other extensively studied genres, and to construct a
neural lyric translation model, thereby underscoring the importance of a
dedicated dataset for singable lyric translations.

###### Index Terms:

Lyric Translation, K-pop Translation, Lyrics Information Processing

##  I Introduction

Singable lyric translation is a common practice to bolster the global
resonance and appeal of music across diverse genres, from opera and animated
musical songs (such as those from Disney) to children’s songs and hymns [1].
With the continuous globalization of music, the importance and popularity of
singable lyric translation are increasing [2], particularly on social media
platforms like YouTube.

Despite its widespread appeal, singable lyric translation is acknowledged as a
challenging discipline as it calls for a sufficient understanding of
musicology and linguistics [3, 2]. Moreover, previous research emphasizes that
lyric translation also requires solid cultural consideration, given the
distinct poetic norms of each language [4, 5, 6]. Consequently, due to these
inherent intricacies, the study of lyric translation remains largely
unexplored. While some studies have endeavored to investigate singable lyric
translations, their research has primarily centered on Western languages,
predominantly English and German, and Western genres, such as opera and
animated musical songs [7, 8, 4, 9, 10]. To our knowledge, there has been no
comprehensive analysis of Korean pop (K-pop) translations, despite their
substantial popularity on social platforms.

Figure 1: An illustration of K-pop translation, featuring “ID Peace B” by BoA,
with English singable lyrics, Korean singable lyrics, and their corresponding
English translations.

Another challenge in lyric translation studies is the absence of a publicly
available dataset. As far as we can tell, no public singable lyric translation
dataset currently exists, creating a barrier to fully deciphering the art of
lyric translation. Hence, systematic analysis of singable lyric translation
has primarily relied on individual case studies [7, 6, 11, 10]. Moreover,
while automatic lyric translation has gained popularity, the development of
neural lyric translation models has been largely dependent on semi-supervised
methods [12, 13] or privately sourced datasets [14].

To address these issues, we have compiled a Korean-English lyric translation
dataset, of which approximately 89% comprises lyrics for K-pop songs. This
dataset, which contains lyrics for a thousand songs, has been meticulously
aligned on a line-by-line and section-by-section basis by humans. The
following section of this paper will delve into the construction methodology
and details of the dataset. Moving further, we will uncover the unique
characteristics of K-pop translation that differentiate it from previously
extensively analyzed genres, and demonstrate the application of our dataset to
the neural lyric translation task, revealing the necessity of singable lyrics
dataset for the enhancement of model performance. We will conclude the paper
by highlighting the insights acquired through our experiments using our
dataset.

Figure 2:  An example of the alignment task, using “Beautiful” by Amber, along
with its English and Korean lyrics and their alignments with syllable counts
in each language. We obtained the syllable counts by employing the syllables
library for English. For Korean, we simply counted the number of characters,
as each character in Korean corresponds to one syllable.

##  II Dataset

In order to facilitate systematic analysis of lyric translation and advance
the development of neural lyric translation models, we introduce a novel
singable lyric translation dataset, which comprises pairs of Korean-English
lyrics for a thousand songs: 886 K-pop songs, 62 animated musical songs, 34
theatre songs, etc. This dataset incorporates essential metadata such as the
name of artist and track, and genre. In addition, it presents meticulous line-
by-line and section-by-section alignments of English and Korean lyrics. As we
will later show, these alignments, although requiring substantial manual
effort as automation is not possible, play a pivotal role in the processes of
analysis, neural model development, and evaluation. Although the lyrics cannot
be directly downloaded due to copyright issues, they can be accessed via
public APIs or URLs. Additionally, we provide an alignment code to aid in both
line-wise and section-wise analysis. Our dataset primarily concentrates on
K-pop, which constitutes about 89% of the total data. Nevertheless, we have
intentionally included lyrics from other well-studied genres, like animated
musical songs (e.g., songs for Disney animation) or theatre songs, to enable
comparative analysis across genres. We believe that this dataset offers value
as it reveals insights into K-pop lyric translation, a topic not sufficiently
explored in prior research, and the translation of lyrics between Korean and
English, languages with substantial grammatical differences. Despite the
limited scope of genre and language, we aim to provide insights into the
importance of a singable lyric dataset for a diverse array of academic
pursuits. A snippet of the sample data is depicted in Table I. A more
comprehensive view of the sample data is shown in Appendix -A and the
statistical details of the dataset are presented in Appendix -B. This dataset
is available for download via the provided link
111https://github.com/havenpersona/k-pop.

| Sec-
---
tion
#
| Line
---
#
English |  | Korean
---
(English Translation)
1 | 1 | You don’t know me | You don’t know me
2 | L-O-V-E or hatred | L-O-V-E or hatred
3 | Hit you with a smile, not goodbye |  | 이별 대신 난 순진한 미소만
---
(Instead of a breakup,
I only have an innocent smile)
4 |  | All the while,
---
I’ll be sure to leave you wonderin’
| 오늘도 네 품에 안길래, oh
---
(So today, I’ll embrace you
in your arms again)
2 | 5 | Oh, on the outside I’ll be all calm |  | 아무것도 모르는 척
---
(Pretending to not know anything)
6 | Baby no more real love | Baby, no more real love
7 | Imma pretend we’re going strong |  | 너의 곁에 있어줄게
---
(I’ll stay by your side)
8 | Then at the end, break your heart |  | 마지막엔 break your heart
---
(In the end, I’ll break your heart.)

TABLE I: Sample data illustrating the English versions and Korean versions of
“Cry for Me” by Twice

###  II-A Source Corpora Collection

Our dataset includes both pairs of official lyrics for the same songs, that
have been officially released in both Korean and English (e.g., a pair of
Korean and English versions of “Cry for Me” by Twice) and pairs of official
Korean lyrics and high-quality unofficial English singable translations (e.g.,
the official Korean lyrics of “Attention” by New Jeans and their singable
translation by YouTuber Emily Dimes). Including unofficial translations, which
take up 65.2% of the entire dataset, significantly enhances the size of our
dataset.

###  II-B Human Alignment

Owing to the subjective nature of lyric structure—with no universal agreement
on what to call a line and what to call a section—the internet-sourced lyrics
for the same song in English and Korean are not aligned on a line-by-line or
section-by-section basis. Despite this, the neural model development,
evaluation, and analysis of singable lyric translations require these
alignments to identify the line-wise and section-wise correspondence and
relationship. We observed, however, that automatically generating these
alignments is currently unattainable. Some might propose syllable counting as
a potential alignment method. However, as it is common to modify melodies to
fit varying syllable counts, this method is not practical [4, 15].
Furthermore, the inclusion of non-lexical vocables, like “oohs” and “aahs,” in
one language’s lyrics, and their absence in the other, creates additional
inconsistency and challenges in auto-alignment. Consequently, we manually
aligned lyrics line-by-line and section-by-section to ensure that lyrics on
the same line share the same melodies and sections are divided by the same
criteria. To demonstrate this alignment task, we provide Figure 2, which uses
“Beautiful” by Amber as an example. This figure illustrates three main points:
1) the different ways sections and lines can be separated, 2) how the same
line can consist of a varying number of syllables, and 3) how the same
nonlexical sound can be represented in different ways (e.g., “yeah” in English
lyrics and “yeah yeah” in Korean lyrics), which makes the auto-alignment
unfeasible.

##  III Unpacking K-pop Translation

In this section, we aim to quantitatively compare the attributes of K-pop
translations with those of other extensively researched genres and identify
the unique features of K-pop that set its translation process apart. We are
basing our comparison on official translations of 234 K-pop songs, 62 animated
musical songs, and 34 musical theatre songs in our dataset.

###  III-A Semantic Pattern

A unique characteristic of K-pop lies in its incorporation of both Korean and
English within song lyrics. Upon analyzing the K-pop songs in our dataset, we
found 30.2% of the lines are entirely in English, and a blend of English and
Korean is observed in 20.7% of lines. In K-pop song translations into English,
English lyrics often remain untranslated, as illustrated in line 1, 2, and 6
of sample data provided in Table I. As a result, a superficial comparison
between K-pop and other genres could lead to the misconception that K-pop has
a high line-wise semantic similarity between the original and translated
lyrics. However, we observed that this differs from real-world K-pop
translation practices, which tend to focus on section-by-section
relationships. Using sample data from Table I as an example, the English
lyrics in line 5, “Oh, on the outside I’ll be all calm,” don’t directly align
semantically with the Korean lyrics, “Pretending to not know anything.”
However, when viewed at the section level, the English and Korean lyrics of
section 2 show the semantical relatedness by sharing a love theme and a
playful mood.

Genre | Similarity
---|---
$Sem_{line}$ | $Sem_{sec}$
K-pop (included) | 0.65 | 0.59
K-pop (excluded) | 0.30 | 0.56
Animation | 0.45 | 0.60
Theatre | 0.32 | 0.53

TABLE II: Line-wise and section-wise semantic similarity between the original
and translated lyrics for K-pop songs, considering both instances where
untranslated English lyrics are included and where they are excluded, animated
musical songs, and theatre songs.

For our analysis, we numerically assessed semantic textual similarity (sts)
between English and Korean lyrics. In order to do this, we followed a method
previously suggested [16]: calculating the cosine similarity between the
embeddings of the original and translated lyrics, generated by a pre-trained
sentence embedding model [17]. 222all-MiniLM-L6-v2 [18]. Because this model
was trained using English language data, Korean lyrics were automatically
translated into English using Google Translator 333We utilized the translate
package (version 3.6.1) in Python in order to use its API. before getting
their embeddings. For instance, the semantic textual similarity between
$x_{i}$ = “하나로 담긴 세상 (a world encapsulated as one)” and $\tilde{x_{i}}$ = “log
in together as one” ($sts(x_{i},\tilde{x_{i}})$) would be determined by
calculating the cosine similarity between the embeddings of “a world
encapsulated as one” and “log in together as one”. For a comparative analysis,
we obtained both line-wise and section-wise semantic similarity. Given a pair
of lyrics $\mathbf{X}$ and $\tilde{\mathbf{X}}$, each comprising $m$ sections
$\\{X_{1},...,X_{m}\\}$, $\\{\tilde{X_{1}},...,\tilde{X_{m}}\\}$ and $n$ lines
$\\{x_{1},...,x_{n}\\}$, $\\{\tilde{x_{1}},...,\tilde{x_{n}}\\}$, previous
study defined the line-wise semantic similarity ($Sem_{line}$) and the
section-wise semantic similarity ($Sem_{sec}$) as below [16], where
$\mathsf{n}(X_{i})$ refers to the number of lines in the $i$-th section.
(Therefore, $n$ = $\mathsf{n}(X_{1})$ \+ …+ $\mathsf{n}(X_{m})$.)

$\displaystyle\scalebox{0.9}{$Sem_{line}(\mathbf{X},\tilde{\mathbf{X}})=\sum_{i=1}^{n}(\frac{1}{n}sts(x_{i},\tilde{x_{i}}))$}.$
(1)
$\displaystyle\scalebox{0.88}{$Sem_{sec}(\mathbf{X},\tilde{\mathbf{X}})=\sum_{i=1}^{m}(\frac{\mathsf{n}(X_{i})}{n}sts(X_{i},\tilde{X_{i}}))$}.$
(2)

Table II provides the line-wise ($Sem_{line}$) and section-wise ($Sem_{sec}$)
semantic similarities between the original and translated lyrics for K-pop
(both that include and exclude untranslated English lyrics), animated musical
and theatre songs. As expected, K-pop displays the highest line-wise semantic
similarity ($Sem_{line}$) when untranslated lyrics are included. However, when
those duplicated parts are excluded, this similarity dramatically drops to the
lowest level among all the groups. This is contrasted to animated musical
songs, which typically need to maintain a strong line-by-line semantic
similarity due to their close ties with corresponding visual animations (see
Fig 3). In contrast, the section-wise semantic similarity ($Sem_{sec}$) stays
relatively steady, even with the exclusion of untranslated parts. Therefore,
it can be inferred that the semantic relatedness between the original and
translated K-pop lyrics exists on a section-by-section basis rather than on a
line-by-line basis.

Figure 3: Density plots showing the distribution of line-by-line semantic
textual similarity ($sts$) between the English and Korean lyrics for K-pop
songs, considering both instances where untranslated English lyrics are
included and where they are excluded, animated musical songs, and theatre
songs.

Genre | $Pho_{deg}$ | $Pho_{var}$
---|---|---
en | kr | en | kr
K-pop | 0.69 | 0.67 | 0.15 | 0.15
Animation | 0.79 | 0.79 | 0.10 | 0.10
Theatre | 0.79 | 0.77 | 0.09 | 0.09

TABLE III: The average values of $Pho_{deg}$ and $Pho_{var}$. Each signifies
the degree and variability of phoneme repetition, respectively.

###  III-B Phoneme Repetition Pattern

The highly repetitive nature of K-pop is reflected not only in melodies but
also in the corresponding lyrics, as they are composed to complement music
[19]. To quantify the degree of phoneme repetition, we leveraged the concept
of _phoneme distinct- $2$_ ($pho$), defined as the ratio of distinct phoneme
bigrams to the total number of phoneme bigrams [16]. For example, consider a
section with two lines $X_{1}$ = {“On the winds can change”, “I’ll stray off
the path I’m walking”}. First, the section is decomposed into phonemes and
<eos> token is added to each line: “AA”, “N”, …,“CH”, “EY”, “N”, “JH”,
“<eos>”, “AY”, …, “W”, “AO”, “K”, “IH”, “NG”, and “<eos>”. Next, the
decomposed components are grouped into bi-grams: “AAN”, …, “CHEY”, “EYN”,
“NJH”, “JH<eos>”, …, “WAO”, “AOK”, “KIH”, “IHNG”, “NG<eos>”. Finally, the
_pho_ of the section $X_{1}$ ($pho(X_{1})$) is obtained by dividing the number
of unique bigrams by the total number of bigrams. Given that the number of
unique bigrams decreases with the repetition of phonemes, a low _pho_ value
implies a higher degree of repetition, and a higher ratio suggests the
opposite.

Figure 4: Data utilization order during the training phase

To numerically represent the degree of phoneme repetition ($Pho_{deg}$) for a
single song, we averaged out the _pho_ value across all sections. To capture
the variability of phoneme repetition degree ($Pho_{var}$) for a song, we
computed the standard deviation of the _pho_ values for each section within
the song. Table III presents the average degree and variability ($Pho_{deg}$
and $Pho_{var}$) of phoneme repetition for K-pop, animated musical, and
theatre songs in our dataset. K-pop displays the lowest average value, which
suggests a high level of repetition inherent to the genre. Additionally, it is
noteworthy that K-pop has the highest $Pho_{var}$ value. This denotes
significant variability, implying that a typical K-pop song features a mix of
highly repetitive sections and others that are less so. As suggested by
previous studies, the extent of phoneme repetition in original lyrics is
mirrored in translated versions [20, 4]. This is evidenced by the similar
$Pho_{deg}$ and $Pho_{var}$ values between the English and Korean versions in
each genres.

##  IV Neural K-pop Translation

One of the research opportunities that our dataset provides is the development
of a model that can automatically generate singable translations of lyrics,
using only textual data. Although past studies have relied on semi-supervised
approaches with human-translated non-singable lyrics due to the unavailability
of a public singable lyric translation dataset [12, 13], we present an example
of building a neural network model that automatically translates Korean pop
lyrics into English, using our dataset. To underscore the potential role of a
singable lyric translation dataset, we will further contrast the outcomes of a
fully semi-supervised approach versus a fine-tuning approach. The usage
examples are provided with two different approaches, line-wise and section-
wise. The results of these approaches will be compared with those of a pre-
trained English to Koren translation model which is not specifically designed
for lyric translation but shares the same architecture as our models.

###  IV-A Training

Due to the scarcity of both non-singable and singable lyric translation
datasets, a previous study initially trained an English-to-Mandarin lyric
translation model with a general Mandarin-English machine translation dataset
[12]. Subsequently, the model was trained using non-singable human-translated
lyrics, treating the original lyrics as the target and their translations as
the source. In parallel to this previous approach, we began training a
transformer-based model [21], which adopts the architecture of the Marian MT
model [22], using general translation dataset, with tokenizing source and
target lyrics with a pre-trained Korean-English tokenizer [23]. This was
followed by the use of non-singable machine-translated lyrics (instead of non-
singable human-translated lyrics, owing to the scarcity of aligned data pairs
of English lyrics and human-translated non-singable Korean lyrics). Finally,
we fine-tuned the model with our singable lyric translation dataset. Unlike
the previous methodology that used melody information along with non-singable
lyrics as input, we only used textual data. The training details are explained
in Appendix -C.

###  IV-B Data Preprocessing

In order to match the syllable count, the previous study built an English-to-
Mandarin lyric translation model that integrated syllable tokens, which
represent the number of syllables, at the beginning of each source and target
lyric line [12]. Because the efficacy of these tokens has been only proven in
Mandarin text generation, where one character consistently corresponds to one
syllable, we compare models with syllable tokens against those without them in
order to study the impact of syllable tokens on English text generation, where
the count of characters does not necessarily reflect the number of syllables.

We modified data in two different ways: line-wise and section-wise, as seen in
Figure 4. Below are the details of data modification methods for models that
incorporate syllable tokens (<SYL>). The methods to construct data for
building models without <SYL> remain the same except for the omission of the
syllable tokens, and that sections are split using <SEP> tokens instead of
<SYL> tokens in the section-wise approach.

General We obtained 500,000 pairs of Korean sentences and their corresponding
English translations [24]. For line-wise training, we simply incorporated
syllable tokens <SYL$s$>, where the value of $s$ represents the total syllable
count of each target sentence. As an example, “annyeonghaseyo” and its English
correspondence “Hello” would be presented as “<SYL2> annyeonghaseyo” and
“<SYL2> Hello” because “Hello” consists of two syllables. Note that the $s$
value does not have any relation with syllable counts for Korean segments. For
section-wise training, we randomly divided both Korean and English sentences
into $n$ segments. Given that the syllable counts for each English segment are
{$s_{1},...,s_{n}$}, we inserted tokens {<SYL$s_{1}$>, …, <SYL$s_{n}$>} prior
to each segment, both in English and Korean.

Non-singable Lyrics We sourced 10,000 English lyrics randomly from the
internet, which were then automatically translated into Korean using Google
Translator. Here, the original English lyrics acted as the target, while the
machine-translated Korean lyrics were used as the source. For line-wise
training, syllable tokens representing the total syllable count of each line
were inserted. For section-wise training, we determined the syllable counts of
each target line {$s_{1},...,s_{n}$} within a section. Consequently, tokens
{<SYL$s_{1}$>, …, <SYL$s_{n}$>} were placed preceding each respective source
and target line. The order of lines within a section was randomly shuffled, so
that the model can learn to translate source sentences in varying order.

Singable Lyrics We employed our singable lyric translation dataset to fine-
tune our models. For both line-wise and section-wise training, we inserted
syllable tokens before each line. In the line-wise approach, each line was
treated as individual data, while in the section-wise approach, the entire
section was considered one data unit and the order of lines within a section
was randomly shuffled.

###  IV-C Evaluation Metrics

Researchers have suggested that traditional methods for analyzing conventional
text generation are not appropriate for lyrics, given their unique linguistics
characteristics [25]. As a result, prior study for automatically assessing
lyric translations concentrated on comparing lyrical features of the original
lyrics to those of the translated lyrics without using a reference [16] rather
than using traditional metrics for machine translation evaluation. In
alignment with these previously suggested methods, we compared the source
lyrics to those automatically translated by our models, focusing on syllable
count, semantics, and phoneme repetition.

To numerically assess the generated lyrics’ syllable count, which is one of
the most important factors that determine the singability, we used two
metrics: the error rate and the syllable count distance (SCD). The error rate
is defined as the rate at which the model generates lines with incorrect
syllable counts. On the other hand, the SCD is defined in the following way
[16]. Suppose that we have a pair of original lyrics $\mathbf{X}$ and
translated lyrics $\tilde{\mathbf{X}}$, each consisting of $n$ lines, where
syllable counts for each line are represented as ${\\{s_{1},...,s_{n}\\}}$ and
${\\{\tilde{s_{1}},...,\tilde{s_{n}}\\}}$. For example, if “I’ll stray off the
path I’m walking” is the second line of the Korean lyrics, and “haneureul-
pihae-sumji (하늘을 피해 숨지)” is its equivalent line in the English lyrics, then
$s_{2}$ equals 8 and $\tilde{s_{2}}$ equals 7. The SCD is calculated as shown
below.

$\displaystyle\scalebox{0.9}{${SCD{(\textbf{X},\tilde{\textbf{X}})}=\frac{1}{2n}\sum_{i=1}^{n}(\frac{|{s_{i}}-{\tilde{s_{i}}}|}{s_{i}}+}{\frac{|s_{i}-{\tilde{s_{i}}}|}{\tilde{s_{i}}}})$}$
(3)

As for the semantics, we evaluated section-wise semantic similarity by using
Equation 2 ($Sem_{sec}$) and semantic coherence between lines by using the
BERT-based next sentence prediction (NSP) model [26]. While the NSP task was
originally proposed to predict whether two sentences given are logically
connected, we used the model to evaluate whether two consecutive lines are
generated in a coherent manner. To achieve this, we fine-tuned a pre-trained
NSP model, bert-based-uncased. using English lyrics from 7,103 songs that are
not included in our training and evaluation data. Given all pairs of
consecutively generated (translated) lines, we averaged out the predicted
probability of whether two lines are consecutive or not, which we will call
the NSP score in this paper.

Finally, we quantitatively assessed the degree and variability of phoneme
repetition by obtaining the average value and standard deviation of the _pho_
across all sections ($Pho_{deg}$ and $Pho_{var}$) for each song as we did in
the previous section.

###  IV-D Quantitative Results

Given that none of our evaluation methods require a reference, we used
external data: lyrics from 2,038 K-pop songs that are not accompanied with
corresponding English translations. We ensured that these lyrics did not
duplicate any songs present in our training data. When drawing inferences from
models with <SYL$s$> tokens, the value of $s$ during the inference phase was
determined based on the syllable count of each line in the source lyrics
contrary to the training phase, where the $s$ value was introduced based on
the syllable count of the target lyrics. To generate inferences for a single
section, composed of $n$ lines, the line-wise approach models inferred on a
line-by-line basis, thereby requiring $n$ inference iterations, while the
generation of section-wise approach models involved one iteration. The
baseline model made inference only on a line-by-line basis, as it does not
have the ability to split a translated section into lines. To draw inference,
we used the beam search method, one of the most popular search methods in
machine translation tasks [27], with four beams.

Input/ Output Form |  Training Approach | Syllable Count | Semantics
---|---|---|---
SCD |  | Error
---
Rate
$Sem_{sec}$ | NSP
| Baseline
---
(Junczys-Dowmunt et al. (2018))
| 0.34
---
(0.10)
0.77 |  | 0.71
---
(0.08)
| 0.47
---
(0.08)
Line-wise | Semi-supervised |  | 0.29
---
(0.31)
0.75 |  | 0.64
---
(0.09)
| 0.51
---
(0.07)
| Semi-supervised
---
(+<SYL>)
| 0.08
---
(0.20)
0.48 |  | 0.68
---
(0.09)
| 0.50
---
(0.08)
Fine-tuned |  | 0.16
---
(0.16)
0.69 |  | 0.66
---
(0.10)
| 0.49
---
(0.07)
| Fine-tuned
---
(+<SYL>)
| 0.08
---
(0.38)
0.42 |  | 0.62
---
(0.12)
| 0.51
---
(0.07)
Section-wise | Semi-supervised |  | 0.45
---
(0.56)
0.78 |  | 0.64
---
(0.12)
| 0.58
---
(0.07)
| Semi-supervised
---
(+<SYL>)
| 0.18
---
(0.12)
0.71 |  | 0.68
---
(0.09)
| 0.54
---
(0.07)
Fine-tuned |  | 0.23
---
(0.28)
0.73 |  | 0.59
---
(0.12)
| 0.57
---
(0.08)
| Fine-tuned
---
(+<SYL>)
| 0.09
---
(0.11)
0.56 |  | 0.60
---
(0.11)
| 0.54
---
(0.07)
Dataset |  | 0.10
---
(0.04)
- |  | 0.59
---
(0.15)
| 0.57
---
(0.07)

TABLE IV: Comparative evaluation of syllable count and semantics, presenting
the average value and associated standard deviation for each metric.

Syllable Count and Semantics When the models were fine-tuned with our dataset,
there was a notable reduction in the average SCD and error rate (see Table
IV). For instance, after fine-tuning the section-wise model using our dataset
without <SYL>, the SCD dropped from 0.45 to 0.23, and the error rate decreased
from 0.78 to 0.73. This suggests that the model could adapt to match syllable
counts when learning from pairs of singable lyrics, even without explicit
training for syllable count matching. The ability to match syllable counts was
further enhanced when models were trained with <SYL> tokens, as evidenced by
the significantly low average SCD and error rate values compared to those
without and the baseline model. This underscores the effectiveness of <SYL>
tokens in matching syllable counts in English textual data.

These improvements were achieved at the expense of semantic similarity, as
suggested by the decline in the $Sem_{sec}$ score from the baseline model to
semi-supervised models and further decline from the semi-supervised to fine-
tuned models. We interpret that they have learned to make a balance between
semantics and singability. This interpretation aligns with real-world lyric
translation practices where semantics are often sacrificed to enhance
singability. This is evidenced by the statistical results of the fine-tuned
models, especially those trained with <SYL>, that closely mirror those of
K-pop songs in our dataset. Therefore, we conclude that the models most
accurately emulate real-world translation patterns when trained with our
dataset using <SYL>. (Note that while they achieved a degree of SCD comparable
to that of the dataset, they still struggled to maintain the exact syllable
counts, as indicated by significant error rate values. This is due to the
nature of English, distinct from that of Mandarain, where identical tokens
don’t consistently equate to the same number of syllables and the same phrases
aren’t always perceived to have an equivalent syllable count.)

While achieving decent performance in terms of syllable count and semantic
similarity, the line-wise models displayed low semantic coherence, as
indicated by a smaller NSP score than the section-wise models. This is due to
the inherent characteristic of the line-wise models that requires making
inferences without considering preceding or subsequent lines. It is also
noteworthy that the baseline model showed the lowest NSP score assumably
because the model failed to coherently capture the lyrical nuances of each
line.

| Input/
---
Output
Form
| Training
---
Approach
$Pho_{deg}$ | $Pho_{var}$
| Baseline [22]
---
0.70 | 0.14
Line-wise | Semi-supervised | 0.70 | 0.14
| Semi-supervised (+<SYL>)
---
0.68 | 0.13
Fine-tuned | 0.66 | 0.13
| Fine-tuned (+<SYL>)
---
0.64 | 0.13
Section-wise | Semi-supervised | 0.61 | 0.18
| Semi-supervised (+<SYL>)
---
0.66 | 0.14
Fine-tuned | 0.53 | 0.18
| Fine-tuned (+<SYL>)
---
0.63 | 0.13
Source | 0.64 | 0.13

TABLE V: Comparative evaluation of phonetic patterns.

Phonetic Pattern The baseline model struggles to replicate the repetitive
phonetic pattern, an important lyrical characteristic of K-pop, as suggested
by a notably higher $Pho_{deg}$ value compared to the source lyrics (see Table
V). A similar trend is observed in line-wise models without <SYL> tokens. On
the other hand, the section-wise models without <SYL> also displayed
significant disparities from the source lyrics in $Pho_{deg}$ value, but in a
different manner: these models exhibited excessive repetition with a markedly
high degree of variability because they occasionally generated the identical
phrase too frequently within a section.

Figure 5: Automatic translations of “In & Out” by Red Velvet, generated by the
baseline model [22] as well as the semi-supervised and fine-tuned models
provided with the original lyrics, their pronunciation, and meanings for
comparison. When the syllable count of the generated lyrics exceeds the target
count, two or more syllables are put under one note considered to be easily
arranged by a music expert. When the syllable count of the generated lyrics is
less than the target count, one or more notes, considered “musically
removable”, are not accompanied by lyrics in the score.

When trained with <SYL> tokens, the models adeptly emulated the repetitive
nuances of K-pop lyrics with stability, as they know when to continue and when
to halt generation. The ability of these models to mirror the repetitive
patterns was further enhanced when fine-tuned with our dataset by learning
from real-world examples. As a result, they produced English lyrics with
$Pho_{deg}$ values aligning closely with the source Korean lyrics in both
line-wise and section-wise approaches. Based on our observation that the
$Pho_{deg}$ and $Pho_{var}$ values of the original lyrics are reflected in the
translated lyrics, we can infer that the model learns the unique phonetic
pattern of K-pop when fine-tuned with our dataset using <SYL> tokens.

###  IV-E Qualitative Results

We present inference examples drawn by the baseline model as well as self-
supervised and fine-tuned models trained with <SYL> for a section from the
K-pop song “In & Out” by Red Velvet in Figure 5. For simplicity, we will
denote line-wise and section-wise semi-supervised models as SS-Line and SS-
Sect, and line-wise and section-wise fine-tuned models as FT-Line and FT-Sect,
respectively.

Syllable Counts and Semantics The baseline model’s generated lyrics showed a
significant difference in syllable counts with the original lyrics, failing to
maintain singability. For instance, the fifth bar contains six notes and
therefore, the original lyrics corresponding to this part have six syllables
(Jo-a-ha-neun-geo-ya). However, the baseline model produced lyrics with only
three syllables, making them unsingable, even with minor melody adjustments.
On the other hand, the translations of semi-supervised and fine-tuned models
generated lyrics with syllable counts comparable to those of the source
lyrics. 444Refer to https://github.com/havenpersona/k-pop for the vocalized
inference examples.

The original lyrics in the second bar, “I want to make you mad (화나게 하고 싶어)”,
were translated by the baseline and semi-supervised section-wise model (SS-
Sect) into “I wanna piss you off”, a direct translation of the original
meaning. Despite successfully reflecting the original meaning, it contains 6
syllables, while the original lyrics consist of 7 syllables. Conversely, the
FT-Sect model successfully generated a 7-syllable line, “I think I’m crazy for
you”, which is not an accurate translation of the corresponding line. However,
given that the original lyrics express deep affection for someone, we
interpret that the FT-Sect model effectively captured the overall mood and
topic of the song, yielding a decent (though not higher than SS-Sect) section-
wise semantic similarity ($Sem_{sec}$). Moreover, the output from the FT-Sect
model employs more lyrical expressions typically found in English lyrics about
love, whereas the SS-Sect model, while achieving literal accuracy, fails to
express love naturally in English. A similar tendency is observed in the line-
wise models. The semi-supervised line-wise model (SS-line)’s generated lyrics,
“I wanna make you mad”, failed to have accurate syllable counts while
achieving semantic accuracy. Conversely, the lyrics translated by the fine-
tuned line-wise model (FT-line), “I want you to stay with me” maintained the
number of syllables but not in semantic accuracy, while successfully capturing
the topic, mood, and lyrical expression. These examples further suggest that
the models, when fine-tuned with a singable lyrics translation dataset, have
learned to prioritize singability over semantic accuracy, reflecting real-
world lyrics translation practices.

This example further illustrates the semantic incoherence of line-wise models,
particularly the self-supervised model. For example, the consecutive lines,
“He was like this I was I” and “I don’t know my mind, yeah”, not only lack
sensibility but also a logical connection. Conversely, the FT-sect model
consistently focuses on expressing love for someone, without performing a
direct word-for-word translation. This results in lower semantic accuracy but
a reasonably good level of semantic coherence.

Phonetic Pattern The original (source) lyrics and melody lines in Figure 5
feature highly repetitive characteristics of K-pop. Similarly, both semi-
supervised and fine-tuned models show the repetitive phonetic pattern.
However, the ability of the line-wise model to create a sense of repetition is
naturally limited to line-wise repetition, as seen in phrases like “I miss
you, I miss you” in bar 6. Conversely, the section-wise model can generate a
sense of repetition on a section-wise basis, as demonstrated in phrases like
“Oh baby I want you” repeated in bars 1, 5, and 6. This capability results in
the FT-sect model having a lower $Pho_{deg}$ value than the FT-line model, as
it captures the repeating patterns across the section.

##  V Conclusions

In this paper, we introduced a novel singable lyrics dataset that precisely
aligns Korean and English lyrics for a thousand songs on a line-by-line and
section-by-section basis. As we demonstrated, this alignment is pivotal for
analyzing and evaluating lyric translations. Unlike previous translation
studies that primarily focused on Western languages and genres, our study
targets Korean pop. We utilized this dataset to analyze the unique
characteristics of K-pop translations in terms of semantic and phonetic
patterns. Additionally, we first suggested that a singable lyrics dataset can
be used to build a neural model that translates lyrics into singable forms,
even without musical information given, as the model draws inferences from
lyrics that are already singable. We compared two approaches to construct a
neural lyric translation model, line-wise and section-wise, along with
observing the effectiveness of <SYL> for these approaches, offering insights
into the development of neural models capable of translating text akin to
lyrics with structured line-by-line and section-by-section characteristics,
such as poetry. We hope that this paper will expand the boundaries of singable
lyric translation studies and offer valuable insights into this field.

## References

  * [1] M. Mateo, “Music and translation,” Handbook of Translation Studies, vol. 3, pp. 115–121, 2012.
  * [2] Ş. Susam-Sarajeva, “Translation and music: Changing perspectives, frameworks and significance,” The Translator, vol. 14, no. 2, pp. 187–200, 2008.
  * [3] S. Spaeth, “Translating to music,” The Musical Quarterly, vol. 1, no. 2, pp. 291–298, 1915.
  * [4] P. Low, “Translating songs that rhyme,” Perspectives: Studies in Translatology, vol. 16, no. 1-2, pp. 1–20, 2008.
  * [5] E. E. Davies and A. Bentahila, “Translation and code switching in the lyrics of bilingual popular songs,” The Translator, vol. 14, no. 2, pp. 247–272, 2008.
  * [6] H. P. Cintrão, “Translating “under the sign of invention”: Gilberto gil’s song lyric translation,” Meta, vol. 54, no. 4, pp. 813–832, 2009.
  * [7] J. Franzon, “Musical comedy translation: Fidelity and format in the scandinavian my fair lady,” in Song and Significance, pp. 263–297, Brill, 2005.
  * [8] M. Snell-Hornby, Theatre and Opera Translation, pp. 106–119.  Bristol, Blue Ridge Summit: Multilingual Matters, 2007.
  * [9] G. Anderman, Europe on stage: Translation and theatre.  Bloomsbury Publishing, 2017.
  * [10] C. Leni and A. S. Pattiwael, “Analyzing translation strategies utilized in the translation of song “Do You Want to Build a Snowman?”,” Journal of Language and Literature, vol. 19, no. 1, pp. 55–64, 2019.
  * [11] J. Åkerström, “Translating song lyrics: a study of the translation of the three musicals by Benny Andersson and Björn Ulvaeus,” 2010.
  * [12] F. Guo, C. Zhang, Z. Zhang, Q. He, K. Zhang, J. Xie, and J. Boyd-Graber, “Automatic song translation for tonal languages,” in Findings of the Association for Computational Linguistics (ACL), (Dublin, Ireland), pp. 729–743, Association for Computational Linguistics, May 2022.
  * [13] L. Ou, X. Ma, M.-Y. Kan, and Y. Wang, “Songs across borders: Singable and controllable neural lyric translation,” arXiv preprint arXiv:2305.16816, 2023.
  * [14] C. Li, K. Fan, J. Bu, B. Chen, Z. Huang, and Z. Yu, “Translate the beauty in songs: Jointly learning to align melody and translate lyrics,” arXiv preprint arXiv:2303.15705, 2023.
  * [15] E. C. Hui-tung, “Translation of songs,” An Encyclopedia of Practical Translation and Interpreting, p. 351, 2019.
  * [16] H. Kim, K. Watanabe, M. Goto, and J. Nam, “A computational framework for singable lyric translation,” in Proceedings of the 24th International Conference on Music Information Retrieval (ISMIR), 2023.
  * [17] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982–3992, 2019.
  * [18] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,” Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 5776–5788, 2020.
  * [19] J. P. G. Mahedero, A. MartÍnez, P. Cano, M. Koppenberger, and F. Gouyon, “Natural language processing of lyrics,” in Proceedings of the 13th Annual ACM International Conference on Multimedia, pp. 475–478, 2005.
  * [20] H. S. Drinker, “On translating vocal texts,” The Musical Quarterly, vol. 36, no. 2, pp. 225–240, 1950.
  * [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the Advances in neural information processing systems (NeurIPS), 2017.
  * [22] M. Junczys-Dowmunt, R. Grundkiewicz, T. Dwojak, H. Hoang, K. Heafield, T. Neckermann, F. Seide, U. Germann, A. F. Aji, N. Bogoychev, A. F. T. Martins, and A. Birch, “Marian: Fast neural machine translation in C++,” in Proceedings of ACL 2018, System Demonstrations, (Melbourne, Australia), pp. 116–121, Association for Computational Linguistics, July 2018.
  * [23] J. Tiedemann and S. Thottingal, “OPUS-MT – building open translation services for the world,” in Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, (Lisboa, Portugal), pp. 479–480, European Association for Machine Translation, Nov. 2020.
  * [24] C. Park, M. Shim, S. Eo, S. Lee, J. Seo, H. Moon, and H. Lim, “Empirical analysis of parallel corpora and in-depth analysis using LIWC,” Applied Sciences, vol. 12, no. 11, 2022.
  * [25] K. Watanabe and M. Goto, “Lyrics information processing: Analysis, generation, and applications,” in Proceedings of the 1st Workshop on NLP for Music and Audio (NLP4MusA), pp. 6–12, 2020.
  * [26] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): Human Language Technologies, Volume 1, pp. 4171–4186, June 2019.
  * [27] R. Leblond, J.-B. Alayrac, L. Sifre, M. Pislar, L. Jean-Baptiste, I. Antonoglou, K. Simonyan, and O. Vinyals, “Machine translation decoding beyond beam search,” pp. 8410–8434, Association for Computational Linguistics, 2021.
  * [28] E. L. Park and S. Cho, “KoNLPy: Korean natural language processing in Python,” in Proceedings of the 26th Annual Conference on Human & Cognitive Language Technology, (Chuncheon, Korea), October 2014.
  * [29] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proceedings of the International Conference on Learning Representations (ICLR), 2014.

| Sec-
---
tion
#
| Line
---
#
English (EN) | Korean (KR)
1 | 1 | I know | I know
2 | Our love ain’t anything to fight for | 고쳐 쓸 가치도 없단 걸
3 | But I’ll never break out of the cycle | 하지만 그녀와 달리 난 널
4 |  | I don’t really wanna let you go
---
(Never let go)
쉽게 놔줄 맘이 없거든 (Never let go)
2 | 5 | You don’t know me | You don’t know me
6 | L-O-V-E or hatred | L-O-V-E or hatred
7 | Hit you with a smile, not goodbye | 이별 대신 난 순진한 미소만
8 |  | All the while, I’ll be sure to leave you
---
wonderin’
오늘도 네 품에 안길래, oh
3 | 9 | Oh, on the outside I’ll be all calm | 아무것도 모르는 척
10 | Baby no more real love | Baby, no more real love
11 | Imma pretend we’re going strong | 너의 곁에 있어줄게
12 | Then at the end, break your heart | 마지막엔 break your heart
4 | 13 | Bad boy, bad boy | Bad boy, bad boy
14 |  | Yeah, you really make me a mad
---
girl, mad girl
| Yeah, you really make me a mad
---
girl, mad girl
15 | Woah-oh-oh | Woah-oh-oh
5 | 16 | I want you to cry, cry for me | I want you to cry, cry for me
17 |  | Thе way I cried for you, baby, cry
---
for me
내가 울었던 것처럼 cry for me
18 | Make your rain fall, cry for mе | Make your rain fall, cry for me
19 | But again | But, again
6 | 20 |  | Somehow you keep me goin’ round
---
and round
조금씩 조금씩 또 빠져가
21 |  | All the walls I built around me come
---
crashin’ down
사랑에 내 결심이 또 무너져가
22 | Makin’ excuses, gotta drown ’em out | 용서할 핑계를 만들어가
23 |  | I want you to, I want you to, I want
---
you to cry
| I want you to, I want you to, I want
---
you to cry for me
24 | Hmm, yeah | Hmm, yeah
7 | 25 | I don’t know if I’m just | I don’t know 너란 놈
26 | In too deep and I’m confused | 미워질 줄 모르고
27 |  | All my friends hate your guts but I’m
---
still defending you (Ooh; Yah yah,
yah yah)
| 친구들한텐 또 너를 감싸주는 중
---
(Ooh; Yah yah, yah yah)
28 |  | I can’t seem to cut you loose (Yah
---
yah, yah yah; Mmm, yeah)
| 바보가 돼 버렸군 (Yah yah, yah
---
yah; Mmm, yeah)
8 | 29 |  | Ooh, don’t know how you keep on
---
laughin’ everyday
Ooh, 너 왜 자꾸 나를 보며 웃는데
30 |  | Just a single tear from you, I’d be
---
okay (Ooh)
딱 한 번의 눈물이면 되는데 (Ooh)
31 |  | Cry for me, let me please forgive
---
you, oh
| Cry for me, let me please forgive
---
you, oh
9 | 32 | Oh on the outside I’ll be all calm | 아무것도 모르는 척
33 | Baby just like real love | Baby, just like real love
34 | Tellin’ myself we’re going strong | 마지막 기회야 어서
35 | If it’s the end, break your heart | 보여줘봐 your true love
10 | 36 | Bad boy, bad boy | Bad boy, bad boy
37 |  | Yeah, you really make me a sad girl,
---
sad girl (Sad girl, sad girl)
| Yeah, you really make me a sad girl,
---
sad girl (Sad girl, sad girl)
38 | Woah-oh-oh | Woah-oh-oh
11 | 39 | I want you to cry, cry for me | I want you to cry, cry for mе
40 |  | Thе way I cried for you, baby, cry
---
for me
내가 울었던 것처럼 cry for me
41 | Make your rain fall, cry for mе | Make your rain fall, cry for mе
42 | But again | But again
12 | 43 |  | Somehow you keep me goin’ round
---
and round
조금씩 조금씩 또 빠져가
44 |  | All the walls I built around me come
---
crashin’ down
사랑에 내 결심이 또 무너져가
45 | Makin’ excuses, gotta drown ’em out | 용서할 핑계를 만들어가
46 |  | I want you to, I want you to, I want
---
you to cry for me
| I want you to, I want you to, I want
---
you to cry for me

13 | 47 | If love is a game | 사랑이란 게
---|---|---|---
48 | Don’t want to play | 너무 혹독해
49 | You poison my veins | 미운 마음도
50 | Then take it all away | 다 녹아버리게 해
51 | I’m chasin’ that taste | 또 다시 원해
52 | I want your kiss, yeah, yeah, yeah | 널 내 곁에 yeah, yeah, yeah
14 | 53 | I want you to cry, cry for me | I want you to cry, cry for me
54 |  | Can you at least pretend, baby, cry
---
for me
너 연기라도 해 빨리 cry for me
55 | Make your rain fall | Make your rain fall
56 | Fall and fall now, yeah | Fall and fall now, yeah
15 | 57 | I want you to cry, cry for me | I want you to cry, cry for me
58 |  | The way I cried for you, baby, cry
---
for me
내가 울었던 것처럼 cry for me
59 | Make your rain fall, cry for me | Make your rain fall, cry for me
60 | But again | But again
16 | 61 |  | Somehow you keep me goin’ round
---
and round
조금씩 조금씩 또 빠져가
62 |  | All the walls I built around me come
---
crashin’ down
사랑에 내 결심이 또 무너져가
63 | Makin’ excuses, gotta drown ’em out | 용서할 핑계를 만들어가
64 |  | I want you to, I want you to, I want
---
you to die for me
| I want you to, I want you to, I want
---
you to die for me

TABLE VI: A pair of lyrics for official English and Korean versions of “Cry
for Me” by Twice

###  -A Sample Data

Table VI shows a pair of official English and Korean versions of “Cry for Me”
by Twice. Please note that we do not own the rights to these lyrics. Our
dataset simply provides public API access to the lyrics and the code for their
line-by-line and section-by-section alignment.

Property | Count
---|---
Songs | 1000
Total sections | 11330
Unique sections (Korean) | 9119
Unique sections (English) | 9040
Total lines | 59054
Unique lines (Korean) | 39134
Unique lines (English) | 38536
Unique vocabulary (Korean) | 7570
Unique vocabulary (English) | 16086

TABLE VII: Dataset Statistics.

Hyperparameters | Value
---|---
# of Layers | 6
# of Attention heads | 8
Batch size | 8
Max position embeddings | 1024
Warmup steps | 500
Weight decay | 0.1
Model size | 2048
Vocabulary size | 65051

TABLE VIII: Model Hyperparameters.

###  -B Dataset Statistics

Table VII shows the key properties of our dataset, along with their respective
counts. To determine the number of Korean vocabulary items during the
statistical gathering process, we used a morphological analyzer [28] with
excluding the English parts in the Korean version lyrics. On the other hand,
counting the number of English vocabulary was achieved through simple
whitespacing. Owing to the repetitive nature of lyrics, the numbers of unique
sections and lines are less than the total count of sections and lines.

###  -C Training Details

Each dataset (General, Machine Translated Lyrics, Singably Translated Lyrics)
was split in a 9:1 ratio for the training and validation sets. We trained our
models to minimize the Negative Log Likelihood (NLL) by comparing the
predicted probability distribution to the ground truth at each epoch. We
stopped the training if there was no reduction in the loss function observed
for three consecutive epochs when tested on the validation set, with
optimization achieved through the Adam optimizer [29]. Key hyperparameters
utilized during this process are detailed in Table VIII.