11institutetext: Bahcesehir University,Turkey
11email<EMAIL_ADDRESS>22institutetext: Bahcesehir
University,Turkey
22email<EMAIL_ADDRESS>
# NiNformer: A Network in Network Transformer with Token Mixing Generated
Gating Function

Abdullah Nazhat Abdullah 11 0000-0002-1757-0785 Tarkan Aydin 22
0000-0002-2018-405X

###### Abstract

The Attention mechanism is the main component of the Transformer architecture,
and since its introduction, it has led to significant advancements in Deep
Learning that span many domains and multiple tasks. The Attention Mechanism
was utilized in Computer Vision as the Vision Transformer ViT, and its usage
has expanded into many tasks in the vision domain, such as classification,
segmentation, object detection, and image generation. While this mechanism is
very expressive and capable, it comes with the drawback of being
computationally expensive and requiring datasets of considerable size for
effective optimization. To address these shortcomings, many designs have been
proposed in the literature to reduce the computational burden and alleviate
the data size requirements. Examples of such attempts in the vision domain are
the MLP-Mixer, the Conv-Mixer, the Perciver-IO, and many more. This paper
introduces a new computational block as an alternative to the standard ViT
block that reduces the compute burdens by replacing the normal Attention
layers with a Network in Network structure that enhances the static approach
of the MLP-Mixer with a dynamic system of learning an element-wise gating
function by a token mixing process.Extensive experimentation shows that the
proposed design provides better performance than the baseline architectures on
multiple datasets applied in the image classification task of the vision
domain.

###### Keywords:

Deep Learning Computer Vision Transformers

##  1 Introduction

Since the advent of the Transformer architecture [1] and the introduction of
the Attention mechanism as its main computational component within the context
of Natural Language Processing (NLP), large advancements have been achieved
not only in language-related tasks but across all aspects related to the
research and application of Machine Learning (ML). Transformers changed the
landscape of NLP with the adoption of their architecture in designing highly
successful and capable Large Language Models (LLM) [2] such as GPT [3],LLama
[4], Falcon [5] and Mistral [6]. The Computer Vision (CV) domain also
experienced rapid adoption of Transformer architectures. Vision-specific
implementations such as ViT [7], MLP-Mixer [8], Conv-Mixer [9], and Swin
Transformer [10] were introduced, along with many application-oriented designs
that utilize such architectures, such as Detection Transformer (DETR) [11],
Perceiver-IO [12], Unified-IO [13], DINO [14], and Segment Anything Model
(SAM) [15].Given such wide adoption on many tasks and modalities, the need for
more efficient implementations of Transformer has increased in importance, and
in that direction, significant research attempts have been introduced, such as
Linformer [16], FNets [17], Local-ViT [18], Max-ViT [19], and Nystromformer
[20].The MLP-Mixer implementation of Transformers is of interest as it
introduces a unique design choice that targets a more efficient architecture
with emphasis on the process of ”token mixing.” This mixing is applied in two
stages [8]; the first stage applies the mixing within the input token
representations, while the second stage applies the mixing process between
corresponding positions in each token. A drawback of the MLP-Mixer design is
that the mentioned mixing processes are performed with static weight matrices,
which limits the capabilities of the architecture in comparison to the
traditional Transformers that utilize the dynamic process of the Softmax
Attention mechanism. At the same time, the traditional Transformer
architecture has its own drawback of quadratic complexity in input size [21],
which imposes a considerable cost in both training and inference when
selecting the architecture. It is notable that in the literature there is a
lack of a design that adopts the efficiency measures introduced in the MLP-
Mixer model while also maintaining a dynamic information filtering mechanism,
as with the traditional Transformer design. In this paper, we introduce a
newly formulated computational block that can be used as a core process in
constructing Transformer architectures that blends both efficient elementary
operations and dynamic information filtering. The new proposal utilizes the
MLP-Mixer token mixing to learn a generator of dynamic per input gating
function that selectively filters the input representation tokens that are
then passed to the per token MLP stage as in traditional Transformers, which
results in a block that contains two levels of processing, an inner and an
outer, hence the chosen name for the proposal as a Network in Network
Transformer, or (NiNformer). In this work, the newly proposed architecture was
trained and its performance evaluated with respect to multiple baselines that
represent different architectural directions and a variety of design choices.
The comparison was conducted on three datasets, and the experimentation was
performed in an equalized setting with the same computational resources to
ensure a fair evaluation. From the experiments conducted, it was observed that
the NiNformer architecture was the most performing, and the obtained results
verified the validity and capability of the underlying assumptions employed in
our proposed computational block.

##  2 Related Work

The literature is rich with attempts to improve on the qualities and
capabilities of the traditional Transformer architecture design
[22],[23],[24],[25]. Guo et al. introduced Star Transformer [26], combining
band attention and global attention. This formulation of the Transformer has a
global node on which a band attention of width 3 is applied. Also, a shared
global node connects a pair of non-adjacent nodes, while adjacent nodes are
connected to each other. Beltagy et al. introduced Longformer[27], which also
uses a combination of band attention and internal global-node attention.
Classification tokens are selected as global nodes. The architecture
substitutes the band attention heads in the upper layers with dilated window
attention, thus increasing the receptive field without increasing computation.
Kitaev et al. introduced Reformer [28] as a modified Transformer that employs
locality-sensitive hashing (LSH). The LSH is used to select the key and value
pairs for each query, therefore allowing each token to attend to tokens that
exist in the same hashing bucket. BigBird architecture by Zaheer et al.[29]
utilizes random attention to approximate full attention with a sparse encoder
and sparse decoder, and it was shown by the analysis that this design can
simulate any Turing Machine, explaining the capability of such architecture.
Katharopoulos et al. proposed the Linear Transformer [30] with feature maps
that target an approximation of the full Softmax Dot Product Attention and
showed comparable performance in empirical tests.Wang et al. introduced
Linformer [16], showing an approximation to the Attention mechanism by a low-
rank matrix, thus lowering the computational requirement while maintaining
comparable performance.Choromanski et al. proposed Performer [31], which uses
random feature maps as an approximate to the Traditional Attention function.
Wang et al. introduced the Cascade Transformer [32] By using a sliding window
attention, the window size is exponentially increased when increasing the
number of layers, leading to a reduction in complexity. Li et al. introduced
the LogSparse Transformer [33] that facilitates long-term dependency on time
series analysis by using Eponym attention. Qiu et al. introduced BlockBERT
[34], which uses block-wise attention to split the input sequence into non-
overlapping blocks. Tay et al. introduced the sparse Sinkhorn attention [35].
This mechanism is essentially block-wise attention, but the keys are sorted
block-wise, therefore learning the permutations. Dai et al. proposed the
Transformer-XL[36]. This design uses a recurrence between the windows that is
segment-based. by storing the representations of the previous window and
storing them in first-in, first-out memory (FIFO). After this step, the
Transformer-XL applies attention to the sorted representations that have been
stored in memory. Clustered Attention, proposed by Vyas et al. [37] clusters
the quires, then calculates the attention distributions for cluster
centroids.Zhang et al. proposed PoolingFormer [38], which utilizes a two-level
attention, a sliding window attention, and a compressed memory attention. The
compressed memory module is used after first applying the sliding window
attention, then applying a compressed memory module for the purpose of
increasing the receptive field. Liu et al. proposed Memory Compressed
Attention (MCA) [39], which complements local attention with strided
convolution, thus reducing the number of keys and values. This allows the
architecture to process much longer sequences compared to traditional
Transformers.Xiong et al. used the Nyström method to modify Transformer with
the introduction of Nyströmformer [20]. This design selects landmark nodes by
the process of strided average pooling and then processes these selected
queries and keys with an approximation to attention by the Nyström method.
Funnel Transformer [40] was proposed by Dai et al. by employing a funnel-like
encoder that has a gradual reduction of the hidden sequence length using
pooling along the sequence dimension; the proper length is then restored with
an up-sampling process.Max-ViT [19] was introduced by Tu et al., which repeats
the basic building block over multiple stages. The basic block consists of two
aspects: blocked local attention and dilated global attention. Ho et al.
proposed the Axial Transformer [41]. This architecture computes a sequence of
attention functions with each one applied along a single axis of the input,
reducing the computational cost. Swin Transformer [10] is an architecture
proposed by Liu et al., and this design reduced the cost by splitting the
image input into non-overlapping patches. These patches are then embedded as
tokens for processing by Attention. FNets [17] was introduced by Lee-Thorp et
al., and it proposes an attention-free Transformer architecture that
substitutes the Softmax Dot Product function used in each The Fourier sublayer
applies a 2D DFT to the embedded input in two steps: one 1D DFT along the
sequence dimension and another 1D DFT along the hidden dimension. gMLP [42]
was introduced by Liu et al., and this architecture is comprised of a series
of blocks that are homogeneous in size and width. Each block layout is highly
reminiscent of inverted bottlenecks. Another feature of this architecture
compared to traditional Transformers is that it does not require position
embeddings. Local-ViT [18] was introduced by Li et al. This architecture
incorporates 2D depth-wise convolutions instead of the feed-forward network as
in ViT. This design choice was inspired by the inverted residuals of
MobileNets. Synthesizer [43] was proposed by Tay et al. as an architecture
that learns synthetic attention weights and does not rely on interactions
between tokens. The results showed competitive performance in relation to
other linear transformer designs.Transformer iN Transformer (TNT) [44] was
introduced by Han et al. This design treats the input images in a similar
manner to a paragraph of text and divides them into several patches as “visual
sentences” and then further divides them into sub-patches as “visual words.”
With this hierarchical division, the architecture is divided into conventional
transformer blocks for extracting features and attentions on the visual
sentence level, and then a sub-transformer is introduced in order to extract
the features of smaller visual words.

##  3 Methodology

###  3.1 Baselines

For an extensive comparative analysis of capability, our proposed architecture
is contrasted to multiple baseline architectures that represent a variety of
functional principles. The ViT follows the principles of a traditional NLP
Transformer, which represented the first iteration of designs that adopted
such architecture. At its core, it relies on the Softmax Dot Product
Attention, and as with NLP-oriented Transformers, the Vit also introduced the
homogeneous layer structure. The following are the main equations for the ViT
block:

$\text{Attention(Q,K,V)}=\text{softmax}\left(\dfrac{QK^{T}}{\sqrt{d_{k}}}\right)V$
(1)

$\text{Y(X)}=\text{Attention}(\text{LayerNorm(X)})+\text{X}$ (2)

$\text{Z(Y)}=\text{MLP}(\text{LayerNorm(Y)})+\text{Y}$ (3)

Below is the pseudocode for the ViT block:

Algorithm 1 ViT

1:Input: Image $I$, number of classes $C$, patch size $ps$, embedding
dimension $d_{m}odel$, number of Transformer blocks $B$, hidden dimension of
MLP $d_{mlp}$, learning rate $\eta$

2:Output: Predicted class probabilities

3:Steps: 1\. Divide $I$ into patches of size $ps\times ps$. 2\. Flatten each
patch and embed it into a $d_{model}$-dimensional vector using patch embedding
layer. 3\. Concatenate the embedded patches into a sequence $X$. 4\. for $i=1$
to $B$ do: * divide $X$ into residual and nonresidual paths. * normalize the
nonresidual path and Apply Attention. * Add the residual path. * divide
Attention result into residual and nonresidual paths. * normalize the
nonresidual path and Apply MLP block. * Add the residual path. 5\. Apply
global average pooling to the output of the last Transformer block. 6\. Use a
fully connected layer with $C$ output units and softmax activation to obtain
class probabilities. 7\. Train the model by minimizing the loss between
predicted and true labels using gradient descent with learning rate $\eta$.

The MLP-Mixer adopts the homogeneous layer structure as with the ViT but
introduces efficiency-oriented computational operations of mixing
(interacting) the token representation with the application of MLP that are
applied in two successive stages: first, an MLP mixing of per token
representation, and second, a per position (channel) MLP mixing of
representations in between the tokens.

The following is the main equation for the MLP-Mixer block:

$\text{Y(X)}=\text{Transpose}(\text{MLP}(\text{Transpose}(\text{LayerNorm(X)})))+\text{X}$
(4)

$\text{Z(Y)}=\text{MLP}(\text{LayerNorm(Y)})+\text{Y}$ (5)

Below is the pseudocode for the MLP-Mixer block:

Algorithm 2 MLP-Mixer

1:Input: Image $I$, number of classes $C$, patch size $ps$, embedding
dimension $d_{m}odel$, number of Transformer blocks $B$, hidden dimension of
MLP $d_{mlp}$, learning rate $\eta$

2:Output: Predicted class probabilities

3:Steps: 1\. Divide $I$ into patches of size $ps\times ps$. 2\. Flatten each
patch and embed it into a $d_{model}$-dimensional vector using patch embedding
layer. 3\. Concatenate the embedded patches into a sequence $X$. 4\. for $i=1$
to $B$ do: * divide $X$ into residual and nonresidual paths. * normalize the
nonresidual path and Transpose. * Apply MLP block. * Transpose. * Add the
residual path. * divide result into residual and nonresidual paths. *
normalize the nonresidual path and Apply MLP block. * Add the residual path.
5\. Apply global average pooling to the output of the last Transformer block.
6\. Use a fully connected layer with $C$ output units and softmax activation
to obtain class probabilities. 7\. Train the model by minimizing the loss
between predicted and true labels using gradient descent with learning rate
$\eta$.

The Local-ViT adopts a conservative design choice to introduce a more
lightweight variant of the original ViT by replacing the per-token MLP layer
in the ViT block with convolutions.

The following are the main equations for the Local-ViT block:

$\text{Y}(X)=\text{Attention}(\text{LayerNorm(X)})+\text{X}$ (6)

$\text{Z}(Y)=\text{CONV}(\text{LayerNorm(Y)})+\text{Y}$ (7)

Below is the pseudocode for the Local-ViT block:

Algorithm 3 Local-ViT

1:Input: Image $I$, number of classes $C$, patch size $ps$, embedding
dimension $d_{m}odel$, number of Transformer blocks $B$, hidden dimension of
MLP $d_{mlp}$, learning rate $\eta$

2:Output: Predicted class probabilities

3:Steps: 1\. Divide $I$ into patches of size $ps\times ps$. 2\. Flatten each
patch and embed it into a $d_{model}$-dimensional vector using patch embedding
layer. 3\. Concatenate the embedded patches into a sequence $X$. 4\. for $i=1$
to $B$ do: * divide $X$ into residual and nonresidual paths. * normalize the
nonresidual path and Apply Attention. * Add the residual path. * divide
Attention result into residual and nonresidual paths. * normalize the
nonresidual path and Apply CONV block. * Add the residual path. 5\. Apply
global average pooling to the output of the last Transformer block. 6\. Use a
fully connected layer with $C$ output units and softmax activation to obtain
class probabilities. 7\. Train the model by minimizing the loss between
predicted and true labels using gradient descent with learning rate $\eta$.

###  3.2 Proposed Architecture

The proposed computational block of this paper is comprised of two levels: an
outer network that resembles a Transformer block by including a token-wise
MLP, which provides the design with an optimization-driven token mapping
capability. The token-wise MLP of the outer network is preceded in the
proposed block by a substitute for the Attention mechanism, which has a gating
function process on the outer network level that extends the concept of Gated
Linear Unit (GLU) [45] by employing a network in network structure. In the
proposed Gating-Unit, the gating signal is generated by a sub-unit in the
inner network, where the inner sub-unit uses a token-mixing architecture of
the MLP-Mixer. The proposed design significantly differs from TNT architecture
[44] in that the two levels in our proposal are different in form and
function, and both inner and outer levels apply their transformations to the
input context as a whole, while the TNT architecture has two levels of the
same traditional Attention mechanism that are applied on two separate scales,
the visual word scale and the visual sentence scale within the input context.
Such distinction of scales omits processing of the global correlations that
may exist between parts of the context in the case of TNT, and our design
utilizes the full context on both of its two levels to capture the global
correlations of the input.In addition, the newly introduced gating mechanism
has the advantage of using the non-dynamic, fixed-weight MLP-Mixer as an inner
sub-unit to learn the interdependencies from the input representation, which
is then used by the outer level as a dynamic gating signal that functions on
an input by input basis to scale the values of its linearly projected
representation, thus facilitating further information processing by the outer
level MLPs without the use of the Softmax Dot Product Attention employed in
generic Transformer architectures.

The following equations describe the operation of the proposed block:

$\text{Gating(I)}=(\text{MLPMixer(I)})*\text{Linear(I)}$ (8)

$\text{Y(X)}=\text{Gating}(\text{LayerNorm(X)})+\text{X}$ (9)

$\text{Z(Y)}=\text{MLP}(\text{LayerNorm(Y)})+\text{Y}$ (10)

The following is the pseudocode of the proposed design:

Algorithm 4 NiNformer

1:Input: Image $I$, number of classes $C$, patch size $ps$, embedding
dimension $d_{m}odel$, number of Transformer blocks $B$, hidden dimension of
MLP $d_{mlp}$, learning rate $\eta$

2:Output: Predicted class probabilities

3:Steps: 1\. Divide $I$ into patches of size $ps\times ps$. 2\. Flatten each
patch and embed it into a $d_{model}$-dimensional vector using patch embedding
layer. 3\. Concatenate the embedded patches into a sequence $X$. 4\. for $i=1$
to $B$ do: * divide $X$ into residual and nonresidual paths. * normalize the
nonresidual path * generate the gating signal by the application of the MLP-
Mixer sub-unit on the nonresidual path. * Apply the Gating by multiplying the
liearly projected nonresidual path with the MLP-Mixer sub-unit output. * Add
the residual path. * divide Gating result into residual and nonresidual paths.
* normalize the nonresidual path and Apply MLP block. * Add the residual path.
5\. Apply global average pooling to the output of the last Transformer block.
6\. Use a fully connected layer with $C$ output units and softmax activation
to obtain class probabilities. 7\. Train the model by minimizing the loss
between predicted and true labels using gradient descent with learning rate
$\eta$.

Figure 1 shows the NiNformer overall architecture in comparison to the Vit and
Local-ViT architectures, while Figure 2 compares the proposed NiNformer block
with that of ViT.

(a) NiNformer Architecture

(b) ViT Architecture

(c) Local-ViT Architecture

Figure 1: Comparing NiNformer Architecture With ViT and Local-ViT

---
(a) NiNformer Gating-Unit and Mixer Sub-Unit

---
(a) Multi-Head Self Attention

Figure 2: Comparing NiNformer Block with Attention Block

##  4 Experiments and Results

For the purposes of this work, three data sets have been selected: The
CIFAR-10 [46] dataset consists of 60000 color images in 32 by 32 resolution
provided for 10 classes, with 6000 images per class. There are 50000 training
images and 10000 test images. The CIFAR-100 [46]dataset consists of 60000
color images in 32 by 32 resolution; the number of classes is 100, resulting
in 600 images per class. Similar to CIFAR-10, there are 50000 training images
and 10000 test images. The MNIST [47] dataset consists of 70,000 grayscale
images in 28 by 28 resolution. The number of classes is 10, as it is a dataset
of handwritten numerical digits. There are 60000 training images and 10000
test images. The utilized software platform is Pytorch, version 1.13,
supported by the CUDA toolkit, version 11.6.2. The version used for the Python
programming language is 3.9. The available hardware system is a desktop
personal computer with an Intel i9-9900k CPU, 32 Gigabytes of system RAM, an
Nvidia RTX 2080ti GPU with 12 Gigabytes of RAM, and running an UBUNTU 20 LTS
operating system. The implementation details of the selected Transformer
architectures in this work are as follows: For the ViT architecture, the
chosen patch size was 4 with a token dimension of 256, and the number of
layers chosen was 4 with 4 Attention heads and an MLP dimension of 512. For
the MLP-Mixer architecture, the chosen patch size was 4 with a token dimension
of 256, and the number of layers chosen was 4 with a token-wise MLP dimension
of 512 and a channel-wise MLP dimension of 512. For Local-ViT, the chosen
patch size was 4 with a token dimension of 256, the number of layers chosen
was 4, 4 Attention heads were selected, and the chosen channel dimension of
the feedforward part was 512. For NiNformer, the chosen patch size was 4, the
number of layers chosen was 4, the token dimension selected was 256, and the
MLP dimension was 512 in the outer network. The inner sub-unit was designed
with a token-wise MLP dimension of 512 and a channel-wise MLP dimension of
512. All models were fitted with a training loop comprised of 100 epochs with
a batch size of 128. All experiments adopted the recommended learning rate for
the Adam optimizer of 0.001 [48].

After performing the experimentation on the selected data sets applied to the
baseline architectures and NiNformer architecture, the following results have
been obtained:

Table 1: Experimentally obtained test accuracy metric on the utilized datasets Dataset |  Test accuracy (ViT) (%) |  Test accuracy (MLP-Mixer)(%) |  Test accuracy (LocalViT)(%) |  Test accuracy (Ninformer) (%) (Ours)
---|---|---|---|---
MNIST |  97.12 |  97.73 |  97.79 |  98.61
CIFAR-10 |  65.74 |  70.12 |  77.71 |  81.59
CIFAR-100 |  34.87 |  39.16 |  41.61 |  53.78

Figures 3 and 4 show respectively, the accuracy and loss curves obtained on
NiNformer for the CIFAR-10, CIFAR-100, and MNIST datasets.

(a) CIFAR-10 accuracy curve

(b) CIFAR-100 accuracy curve

(c) MNIST accuracy curve

Figure 3: Accuracy Curves for NiNformer Architecture

(a) CIFAR-10 Loss curve

(b) CIFAR-100 Loss curve

(c) MNIST Loss curve

Figure 4: Loss Curves for NiNformer Architecture

##  5 Conclusion

In this work, a newly designed Network in Network block was introduced that
substitutes the Attention block used traditionally by Transformer
architectures. By extending the token mixing approach presented in the MLP-
Mixer to function as a gating signal generator and taking advantage of the
gating mechanism to introduce dynamic behavior in the newly proposed block,
thus enhancing the static weight approach of the MLP-Mixer by utilizing its
layers as a sub-unit network within the outer network formulation, The
experimental results show that in comparison to baseline architectures that
were chosen from different families of Transformer formulations, our proposed
block significantly outperforms, showing noticeable improvements on those
baselines, specifically showing a great enhancement of accuracy compared to
the standalone MLP-Mixer architecture that acts as a sub-unit, which validates
that our proposal, with its dynamic gating of the upstream representation,
properly enhances and circumvents the shortcoming of the static weight
approach of the standalone MLP-Mixer while still providing more simplicity of
operations in contrast to the vanilla ViT Transformer architecture. Future
directions of this work are to investigate a multitude of sub-unit network
selections, aiming for further enhancements and capabilities.

## References

  * [1] Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. ”Attention is all you need.” Advances in neural information processing systems 30 (2017).
  * [2] Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever and Dario Amodei. “Language Models are Few-Shot Learners.” (2020).
  * [3] Radford, Alec and Karthik Narasimhan. “Improving Language Understanding by Generative Pre-Training.” (2018).
  * [4] Touvron, Hugo, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave and Guillaume Lample. “LLaMA: Open and Efficient Foundation Language Models.” (2023).
  * [5] Penedo, Guilherme, Quentin Malartic, Daniel Hesslow, Ruxandra-Aimée Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei and Julien Launay. “The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only.” (2023).
  * [6] Jiang, Albert Qiaochu, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L’elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix and William El Sayed. “Mistral 7B.” (2023).
  * [7] Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit and Neil Houlsby. “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.”(2020).
  * [8] Tolstikhin, Ilya O., Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic and Alexey Dosovitskiy. “MLP-Mixer: An all-MLP Architecture for Vision.” Neural Information Processing Systems (2021).
  * [9] Trockman, Asher and J. Zico Kolter. “Patches Are All You Need?” Trans. Mach. Learn. Res. 2023 (2022).
  * [10] Liu, Ze, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin and Baining Guo. “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.” 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021).
  * [11] Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. “End-to-End Object Detection with Transformers.” (2020).
  * [12] Jaegle, Andrew, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Andrew Brock, Evan Shelhamer, Olivier J. H’enaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals and João Carreira. “Perceiver IO: A General Architecture for Structured Inputs & Outputs.” (2021).
  * [13] Lu, Jiasen, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi and Aniruddha Kembhavi. “Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks.” (2022).
  * [14] Zhang, Hao, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun-Juan Zhu, Lionel Ming-shuan Ni and Heung-yeung Shum. “DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection.” (2022).
  * [15] Kirillov, Alexander, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár and Ross B. Girshick. “Segment Anything.” 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (2023).
  * [16] Wang, Sinong, Belinda Z. Li, Madian Khabsa, Han Fang and Hao Ma. “Linformer: Self-Attention with Linear Complexity.” (2020).
  * [17] Lee-Thorp, James, Joshua Ainslie, Ilya Eckstein and Santiago Ontañón. “FNet: Mixing Tokens with Fourier Transforms.” (2021).
  * [18] Li, Yawei, K. Zhang, Jie Cao, Radu Timofte and Luc Van Gool. “LocalViT: Bringing Locality to Vision Transformers.” (2021).
  * [19] Tu, Zhengzhong, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Conrad Bovik and Yinxiao Li. “MaxViT: Multi-Axis Vision Transformer.” European Conference on Computer Vision (2022).
  * [20] Xiong, Yunyang, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Moo Fung, Yin Li and Vikas Singh. “Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention.” Proceedings of the … AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence 35 16 (2021).
  * [21] Keles, Feyza Duman, Pruthuvi Maheshakya Wijewardena and Chinmay Hegde. “On The Computational Complexity of Self-Attention.” International Conference on Algorithmic Learning Theory (2022).
  * [22] Lin, Tianyang, Yuxin Wang, Xiangyang Liu and Xipeng Qiu. “A Survey of Transformers.” AI Open 3 (2021).
  * [23] Tay, Yi, Mostafa Dehghani, Dara Bahri and Donald Metzler. “Efficient Transformers: A Survey.” ACM Computing Surveys 55 (2020).
  * [24] Fournier, Quentin, Gaétan Marceau Caron and Daniel Aloise. “A Practical Survey on Faster and Lighter Transformers.” ACM Computing Surveys 55 (2021).
  * [25] Khan, Salman Hameed, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan and Mubarak Shah. “Transformers in Vision: A Survey.” ACM Computing Surveys (CSUR) 54 (2021).
  * [26] Guo, Qipeng, Xipeng Qiu, Pengfei Liu, Yunfan Shao, X. Xue and Zheng Zhang. “Star-Transformer.” (2019).
  * [27] Beltagy, Iz, Matthew E. Peters and Arman Cohan. “Longformer: The Long-Document Transformer.” (2020).
  * [28] Kitaev, Nikita, Lukasz Kaiser and Anselm Levskaya. “Reformer: The Efficient Transformer.” (2020).
  * [29] Zaheer, Manzil, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang and Amr Ahmed. “Big Bird: Transformers for Longer Sequences.” (2020).
  * [30] Katharopoulos, Angelos, Apoorv Vyas, Nikolaos Pappas and Franccois Fleuret. “Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention.” International Conference on Machine Learning (2020).
  * [31] Choromanski, Krzysztof, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy J. Colwell and Adrian Weller. “Rethinking Attention with Performers.” (2020).
  * [32] Chenguang Wang, Zihao Ye, Aston Zhang, Zheng Zhang, and Alexander J. Smola. 2020. ”Transformer on a Diet.” (2020).
  * [33] LI, SHIYANG, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang and Xifeng Yan. “Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting.” (2019).
  * [34] Qiu, Jiezhong, Hao Ma, Omer Levy, Scott Yih, Sinong Wang and Jie Tang. “Blockwise Self-Attention for Long Document Understanding.” (2019).
  * [35] Tay, Yi, Dara Bahri, Liu Yang, Donald Metzler and Da-Cheng Juan. “Sparse Sinkhorn Attention.” International Conference on Machine Learning (2020).
  * [36] Dai, Zihang, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le and Ruslan Salakhutdinov. “Transformer-XL: Attentive Language Models beyond a Fixed-Length Context.” (2019).
  * [37] Vyas, Apoorv, Angelos Katharopoulos and Franccois Fleuret. “Fast Transformers with Clustered Attention.” (2020).
  * [38] Zhang, Hang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan and Weizhu Chen. “Poolingformer: Long Document Modeling with Pooling Attention.” International Conference on Machine Learning (2021).
  * [39] Liu, Peter J., Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser and Noam M. Shazeer. “Generating Wikipedia by Summarizing Long Sequences.” (2018).
  * [40] Dai, Zihang, Guokun Lai, Yiming Yang and Quoc V. Le. “Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing.” (2020).
  * [41] Ho, Jonathan, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. ”Axial attention in multidimensional transformers.” (2019).
  * [42] Liu, Hanxiao, Zihang Dai, David So, and Quoc V. Le. ”Pay attention to mlps.” Advances in Neural Information Processing Systems 34 (2021).
  * [43] Tay, Yi, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao and Che Zheng. “Synthesizer: Rethinking Self-Attention for Transformer Models.” International Conference on Machine Learning (2020).
  * [44] Han, Kai, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. ”Transformer in transformer.” Advances in Neural Information Processing Systems 34 (2021).
  * [45] Dauphin, Yann, Angela Fan, Michael Auli and David Grangier. “Language Modeling with Gated Convolutional Networks.” International Conference on Machine Learning (2016).
  * [46] Krizhevsky, Alex, Vinod Nair, and Geoffrey Hinton. ”Cifar-10 and cifar-100 datasets.” URl: https://www. cs. toronto. edu/kriz/cifar. html 6, no. 1 (2009)
  * [47] LeCun, Yann, Léon Bottou, Yoshua Bengio and Patrick Haffner. “Gradient-based learning applied to document recognition.” Proc. IEEE 86 (1998)
  * [48] Steiner, Andreas, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. ”How to train your vit? data, augmentation, and regularization in vision transformers (2021).


# Automatic Song Translation for Tonal Languages

Fenfei Guo
University of Maryland
<EMAIL_ADDRESS>Zhang
Zhejiang University
<EMAIL_ADDRESS>Zhang
Tencent AI Lab
<EMAIL_ADDRESS>He
Purdue University
<EMAIL_ADDRESS>Zhang
Zhejiang University
<EMAIL_ADDRESS>Xie
Alibaba DAMO Academy
<EMAIL_ADDRESS>Boyd-Graber
cs, iSchool, umiacs, lsc
University of Maryland
<EMAIL_ADDRESS>
###### Abstract

This paper develops automatic song translation (ast) for tonal languages and
addresses the unique challenge of aligning words’ tones with melody of a song
_in addition to_ conveying the original meaning. We propose three criteria for
effective ast—preserving meaning, singability and intelligibility—and design
metrics for these criteria. We develop a new benchmark for English–Mandarin
song translation and develop an unsupervised ast system, Guided AliGnment for
Automatic Song Translation (gagast), which combines pre-training with three
decoding constraints. Both automatic and human evaluations show gagast
successfully balances semantics and singability.

##  1 Introduction

Suppose you are asked to translate the lyrics “Let it go” from the Disney
musical Frozen into Mandarin Chinese. Some good, literal translations of this
would be A) “fàng shǒu”, B) “fàng shǒu ba” or C) “ràng tā qù ba” (Figure 1);
these get the meaning across and are the domain of traditional machine
translation. However, what if you needed to sing this song _in Mandarin_?
These literal translations simply do not work: Translations A and C do not
match the number of notes and break the original rhythm; while the tones of
Translation B does not match the pitch flow of the original melody.

Figure 1: Example Mandarin translations for “Let it go” in Frozen. Of these,
only the official human song translation is something a singer could actually
sing: it fits the length of the notes and matches the tones with the pitch of
notes. gagast finds translations that satisfy these constraints.

Song translation, unlike translation lyrics for understanding (subtitling),
aims to translate the lyrics so that it can be _sung_ with the original
melody. Therefore, the translated lyrics must match the prosody of the pre-
existing music in addition to retaining the original meaning. In Singable
Translations of Songs, Low (2003) says, this is an uncommon and an unusually
complex task, a translator consider rhythm, notes’ pitches, phrasing, and
stress. Nonetheless, there are cultural and commercial incentives for more
efficient song translation; Frozen alone made over a half a billion dollars in
non-English box office receipts111https://www.the-
numbers.com/movie/Frozen-(2013)#tab=international and the musical Les
Misérables has been performed in over a dozen languages on stage.

As we discuss in Section 2, while translating Western songs resembles poetry
translation, translating into _tonal_ languages (e.g., Mandarin, Zulu and
Vietnamese) introduces new problems. In tonal languages, a word’s pitch
contributes to its meaning (Figure 2); when singing in tonal languages, the
tones of translated words must align with the “flow” of the pitches in the
music (Section 2.1). For example, if “fáng shǒu” were sung instead of “fàng
shǒu” (because notes are going up), a listener might hear “defensive” instead
of the intended meaning.

Figure 2: In total languages like Mandarin, the pitch changes the meaning of
the words (left). Each of the four tones in Mandarin (right) has a different
pitch profile. Figure from Xu (1997).

This paper builds the first system for automatic song translation (ast) for
one tonal language—Mandarin. Section 3 proposes three criteria—preserving
semantics, singability and intelligibility—needed in an ast system.

Guided by those goals, we propose an unsupervised ast system, Guided AliGnment
for Automatic Song Translation (gagast). gagast begins with an out-of-domain
translation system (Section 4.1) and adds song alignment constraints that
favor translations that are the appropriate length and whose tones match the
underlying music (Section 4.2). Naturally, such constraints trade-off between
semantic meaning and singability/intelligibility. Section 5.4 discusses this
trade-off between song alignment scores and the standard machine translation
metric, bleu.

These criteria also form the evaluation for our initial evaluation (Section
5.3). However, we go beyond an automatic evaluation through a human-centered
evaluation from musicology students. gagast creates singable songs that make
sense given the original text, and our proposed alignment scores correlate
with human judgements (Section 5.4.3).222Examples of translated songs by
gagast at https://gagast.github.io/posts/gagast.

Figure 3: If a song’s music doesn’t match the tones of the lyrics, it can
cause the hearer to misunderstand the lyrics. In this example, someone can
hear “sǐ zài” instead of “sì zài”, because the notes are going up and “sì zài”
is going down.

##  2 Background: Prose, Poetry, and Song Translation

A spoken language can be divided into two forms: prose, which corresponds to
natural conversation and conventional grammatical structure; and
verse—typically rhythmic and broken into stanzas–such as poetry and song
lyrics.

The vast majority of machine translation research has been focused on prose
translation and has made huge progress; in contrast verse translation is more
difficult as it must obey the rhythmic constraints and is less developed. In
his tour de force work Le Ton Beau de Marot, Douglas Hofstadter presents
eighty-nine translations of a single poem to capture the panoply of
considerations of what makes the task difficult (Hofstadter, 1997).

In western verse, the rhythmic structure are mostly defined by meter, such as
the iambic pentameter for sonnets, which defines the length of each line, the
patterns of long syllables versus short ones and the stressed ones versus weak
ones. Existing work Greene et al. (2010); Ghazvininejad et al. (2018) use
finite-state constraints to encode both meter and rhyme.

Song translation, on the other hand, can be viewed as a translation where the
_melody_ defines the constraints. Reproducing _all_ of the essential values of
a song—perfectly matching the meaning, perfectly singable, and perfectly
understandable—is an impossible ideal (Franzon, 2008). Thus, tradeoffs are
unavoidable. Low (2003) argues for prioritizing singability over other
qualities such as sense and rhyme since “effectiveness on stage” is a
practical necessity. Tonal languages (e.g., Mandarin, Zulu and Vietnamese)
dramatically increases the complexity of singability, and introduces a new
factor that could hamper intelligibility.

###  2.1 Song Translation for Tonal Languages

For tonal languages, pitch contributes to the meaning of words. In a
conservative estimation, fifty to sixty percent of the world’s languages are
tonal (Yip, 2002) and cover over 1.5 billion people. For the lyrics to be
intelligible, the speech tone and music tone should be correlated (Schneider,
1961). If not, the pitch contour could override the intended tone, which could
produce different meanings. This is not just a theoretical consideration;
Figure 3 shows how lyrics can be and have been misunderstood.333More examples
at https://gagast.github.io/posts/gagast/#misunderstanding_examples

###  2.2 Mandarin Tones and how to Sing them

Schellenberg (2013) summarizes the rules of singing with tone with a focus on
Chinese dialects. The tonal system of Mandarin has two components:

  * •

The pitch level and shape of tones. Four Mandarin tones are used since the 19
century (Figure 2). We denote tones with a diacritic over the vowel whose
shape roughly matches the shape of the tone. The four tones are a high level
(tone 1, e.g., shūo), rising (tone 2, yú), falling-rising (tone 3, wǒ) and
falling (tone 4, huài).

  * •

The sandhi of tones. Some combinations of tones have difficult articulatory
patterns, so words that might normally have one tone might take another
depending on the context. For example “nǐ” (you) and “hǎo” (good) are
typically both third tone, but when they are together it is pronounced as “ní
hǎo” (hello), with the first syllable changing to a _second_ tone. These
changes are called sandhi (Xu, 1997; Hu, 2017).

Mandarin tones interact with a sung melody in two ways (Yinliu et al., 1983;
Schellenberg, 2013) to ensure lyrics are intelligible. First, at a local
level, the shape of tones of individual syllables should be consistent with
the musical notes they are matched with; for example, in “Love Island” (Figure
4), “shàng” in the blue block has the “falling” shape and the group of notes
it assigned to it also falls from an A to a E. Second, and a global level, the
music’s pitch contour should align with the tones of the corresponding
syllables (taking sandhi into account). In practice, we align the transitions
between successive syllables and successive notes (Figure 5) ensuring that the
tone matches the relative pitch change (Schellenberg, 2013).

Figure 4: The output of a song translation needs to align syllables to the
reference melody. There are several options, as evinced by the song “Love
Island (xīn dǎo)”. Orange (top): REST notes; Blue (bottom left): one syllable
is assigned to a group of multiple notes (which needs tone shape alignment:
the down arrow matches with falling tone of “ràng”); Green (bottom right): one
syllable is assigned with one note.

##  3 AST for Tonal Languages

This section formally defines automatic song translation (ast) for tonal
languages and introduce three criteria for what makes for a good song
translation. These criteria form the foundation for the quantitative metrics
we use in the experiment.

###  3.1 Criteria

There are three criteria that a singable song translation needs to fulfil.

##### Preserve meaning.

The translated lyrics should be faithful to the original source lyrics.

##### Singability.

Low (2003) defines singability as the phonetic compatability of translated
lyrics and music. The translated song needs to be sung without too much
difficulty; difficult consonant clusters, cramming too many syllables into a
line, or incompatible tones all impair the singability.

##### Intelligibility.

The translated song need to be understood by the listener. This quality has
two components. First, could a listener produce _any_ transcription of the
lyrics. If the lyrics are too fast or garbled because the keywords do not fit
well with the music, the lyrics are unintelligible. Beyond this basic test of
recognizability, the lyrics must also be accurate: does this transcription
match the intended meaning. Both aspects matter for a stage performance, since
the audience should understand the content to follow the plot. For pop songs,
not understanding all contents could be acceptable for some audiences; for
example, Adriano Celentano’s Prisencolinensinainciusol sacrifices all
intelligibility for singability Bellos (2013). However, in more traditional
media, hilarious misheard lyrics can ruin the audience’s experience (Figure
3).

notes | A3 | C4 | D4 | REST | F4 | G4 | F4 | F4
---|---|---|---|---|---|---|---|---
pitch level | 57 | 60 | 62 |  | 65 | 67 | 65 | 65
duration | $\frac{1}{4}$ | $\frac{1}{4}$ | $\frac{1}{2}$ | 1 | 3 | $\frac{3}{2}$ | $\frac{1}{2}$ |  $\frac{1}{2}$
syllables | How | a- | bout |  |  love?

Table 1: A snippet of the song “Seasons of love” from the musical Rent that
shows the input into gagast. Notes are converted into integer pitches with a
duration, and syllables are aligned to notes: the “a” from “about” has one
note but “love” has four.

###  3.2 Task Definition

We define the ast task as follows: given an aligned pair of melody $M$ and
source lyrics $X$, generate translated text $Y$ in the target language that
aligns with the input melody $M$.

Specifically, $X=[x_{1},...,x_{L}]$ are the input lyrics with $L$ syllables.
Each syllable $x_{i}$ is aligned to a snippet of the melody (Table 1)
represented by a sequence of notes. To represent this to our algorithm, each
syllable is aligned to three components of the melody:

  1. 1.

A sequence of pitch values $\mathbf{p}_{i}\equiv[p_{i}^{0},\dots]$ with
$|\mathbf{p}_{i}|\geq 1$ where an integer value of 1.0 means a semitone (e.g.,
between C and C-sharp).

  2. 2.

The duration of those notes $\mathbf{d}_{i}\equiv[d_{i}^{0},...]$, where 1.0
is a quarter note. Because it encodes the duration of each note, the length of
$\mathbf{d}_{i}$ must be the same as the length of $\mathbf{p}_{i}$.

  3. 3.

Sometimes there is a rest (pause) before a lyric is sung. We align this to the
following syllable $i$. The scalar $r_{i}$ is the real-valued duration of the
REST note before note group $\mathbf{p}_{i}$. If no REST exists before
$\mathbf{p}_{i}$, $r_{i}=0.0$.

###  3.3 Constraints for Aligning Lyrics to Music

To make translated songs singable and intelligible, we summarize three
desirable properties of that the ast lyric outputs should have if they are to
match the underlying melody. Each of these induces a score function which we
will use both in our objective functions for constrained translation and for
our evaluation metrics.

####  3.3.1 Length Alignment

The number of syllables $L_{y}$ in translated lyrics $Y$ need to match the
number of groups of notes $\mathbf{p}_{i}$ in the melody $M$, so that it can
be sung with the music. Within the scope of this paper, we either keep the
original grouping in the melody $M$ and have $L_{y}=L_{x}$ for reproducing the
original music; or strictly produce one target syllable for each single note
in the melody.

####  3.3.2 Pitch Alignment

For tonal languages, pitch of the music must match the lyrics. As in Section
2.2, there are two types of pitch alignments: 1) intra-syllable, the tone
shape of each syllable (Figure 4 blue box) should align with the shape of the
assigned group of notes; 2) inter-syllable, the overall pitch contour of the
music phrase should align with the tones of lyrics.

##### Intra-syllable alignment.

For an _individual_ syllable, if it is assigned to more than one note (e.g.,
“love” in Table 1), those notes must be consistent with the shape of the
syllable’s tone (Wee, 2007). For Mandarin, there are four tones (Xu, 1997,
Figure 2). We estimate the shape of the multi-note sequence $\mathbf{p}_{i}$
by least-square estimation and classify it into one of five categories: level,
rising, falling, rising-falling, falling-rising.

Specifically, for each group $\mathbf{p}_{i}$ that $|\mathbf{p}_{i}|>1$, we
classify it as,

“level”, if $p_{max}^{i}-p_{min}^{i}\leq 1.0$; otherwise, we fit
$\mathbf{p}_{i}$ into $ax^{2}+bx+c$ via least-square estimation, and compute
the axis of symmetry $l=-b/2a$,

“rising”, if ($l\leq p_{i}^{0}$ and $a>0.0$) or ($l\geq p_{i}^{-1}$ and
$a<0.0$);

“falling”, if ($l\leq p_{i}^{0}$ and $a<0.0$) or ($l\geq p_{i}^{-1}$ and
$a>0.0$);

“rising-falling”, if $p_{i}^{0}<l<p_{i}^{-1}$ and $a<0.0$;

“falling-rising”, if $p_{i}^{0}<l<p_{i}^{-1}$ and $a>0.0$;

We compare the shape with that of syllable $y_{i}$, and compute the intra-
syllable alignment score $S_{\mbox{intra}}^{i}$:

$S_{\mbox{intra}}^{i}=\begin{cases}1.0&\text{if the shape matches},\\\
\epsilon&\text{otherwise},\\\ \end{cases}$ (1)

where $\epsilon$ is a small parameter that allows for mismatches. Of the five
patterns, “level” can match with any tone, “rising” matches with tone 2 (yú),
“falling” matches with tone 4 (huài), “falling-rising” matches with tone 3
(wǒ) while “rising-falling” matches no Chinese tones.

Figure 5: For translated songs in Mandarin to be singable, music notes should
align the tones of successive characters; this becomes our inter-syllable
pitch alignment. The arrows show acceptable transitions in music for two
successive Mandarin characters ($w_{i-1},w_{i}$) based on the shape of
Mandarin tones including sandhi.

##### Inter-syllable alignment.

The second constraint compares the transition directions between consecutive
tones ($t_{i-1}$, $t_{i}$) of successive syllables ($y_{i-1}$, $y_{i}$) that
belong to the same word (see arrows in Figure 3). These must match the
transition directions of music notes ($\mathbf{p}_{i-1}$,
$\mathbf{p}_{i}$).444We compute the directions of two notes group
($\mathbf{p}_{i-1}$, $\mathbf{p}_{i}$) by the first notes ($p_{i-1}^{0}$,
$p_{i}^{0}$) for simplicity. Each transition (the movement from one
syllable/note to the next) can be categorized as level, step up, jump up, step
down and jump down. We summarize the acceptable transitions for each pair of
successive syllables in Figure 5 based on analysis by Yinliu et al. (1983) and
we discuss our choices with more details in Appendix A.2. Given two syllables
($y_{i-1}$, $y_{i}$), we compute the local pitch contour
$S_{\mbox{inter}}^{i}$:

$S_{\mbox{inter}}^{i}=\begin{cases}1.0&\text{if contour matches},\\\
\epsilon&\text{otherwise},\\\ \end{cases}$ (2)

where $\epsilon$ again is a small value to allow mismatches.

####  3.3.3 Rhythmic Alignment with Word Segmentation in Mandarin

A musical REST is a silence separating music. Recall that in our setup of the
data, a scalar $r_{i}$ denotes if a note precedes syllable $i$. In any
language, it is uncommon for a rest to break up a word’s syllables. Thus a
good translation should avoid this. For Mandarin, creating metrics that
capture this are slightly more complicated because translation systems
typically do not explicitly generate word boundaries. Thus, we must rely on
the output of segmentation systems to know where word boundaries are.

An exception to this is punctuation (Figure 4). If a comma, period, or other
punctuation is attached to the previous syllable $y_{i-1}$, then that is a
clear signal that it’s fine to pause between them. Thus, our rest score a
syllable $y_{i}$ following $y_{i-1}$ that are part of different words with
probability $P_{\text{seg}}$555In practice, we use the cut output by the Jieba
toolkit., the rest score is:

$S_{R}^{i}=\begin{cases}1.0&\text{if }r_{i}>0.0\text{ and [punc] after
}y_{i-1},\\\ 1.0&\text{if }r_{i}=0.0,\\\ P_{\text{seg}}&\text{if
}r_{i}>0.0\text{ and not [punc]},\\\ \epsilon&\text{otherwise},\end{cases}$
(3)

where $\epsilon$ is a parameter that represents our tolerance of having a rest
within a word.

Figure 6: Overview of gagast for English–Mandarin song translation. We first
pre-train a lyrics translation model with mixture domain data (left); and then
add alignment constraints in decoding scoring function during inference
(right), we use unconstrained version as our baseline in the experiment.

##  4 GagaST

Ideally, we would build an ast system for English–Mandarin song translation
with data-driven models from parallel data, i.e., aligned triples ($M$, $X$,
$Y$). However, these data are not available in the quantity or quality
necessary for Mandarin: there is not enough data of any quality, and those
that do exist have errors in the syllable-notes alignment. Thus, we propose an
unsupervised ast system, Guided AliGnment for Automatic Song Translation
(gagast). For the pre-training, we collect non-parallel lyrics data in both
English and Mandarin, as well as a small set of lyrics translation data
(Section 5.1).

###  4.1 Song-Text Style Translation

To produce faithful translations in song-text style, we pre-train a
transformer-based translation model with cross-domain data: translation data
in the general domain, the collected monolingual lyrics data, and a small set
of lyric translation data. We append domain tags (Figure 6) before each input
example to control the model to produce translations merely in lyrics domain
during song translation. For monolingual lyrics data, we adopt bart pre-
training Lewis et al. (2020).

###  4.2 Music Guided Alignment Constraints

Without available parallel data to learn the lyric-melody alignments, we
impose constraints (Section 3.3) in the decoding phase. Specifically, since
all constraints are applied at the unigram (intra-syllable, REST) or bigram
(inter-syllable, REST) level, we apply them at each step of beam search as
rewards and penalties in the scoring function:

$\displaystyle\log$
$\displaystyle{P(Y\,|\,X,M)}=\sum_{i=0}^{L}\left[\log{P(y_{i}\,|\,y_{i-1:0},X)}\right.$
$\displaystyle+\lambda_{\mbox{inter}}\log{S_{\mbox{inter}}^{i}}+\lambda_{\mbox{intra}}\log{S_{\mbox{intra}}^{i}}$
$\displaystyle+\left.\lambda_{R}\log{S_{R}^{i}}\right.],$ (4)

where $S_{\mbox{inter}}$, $S_{\mbox{intra}}$, and $S_{R}$ refer to the
alignment scores for inter-syllable pitch alignment, intra-syllable pitch
alignment and the rhythm alignment by REST. We introduce three tunable
parameters—$\lambda_{\mbox{inter}}$, $\lambda_{\mbox{intra}}$, and
$\lambda_{R}$—that control the importance of each of the song-specific
constraints.

###  4.3 Length Control in Pre-training

To meet the length constraints, we pre-define the syllable-notes assignments
with two strategies:666A dynamic mapping between the note sequence and the
syllables changes the original rhythm and increases the search space
exponentially. We leave this to future work. 1) note-to-syllable, i.e., for
each note, we produce one syllable; 2) syllable-to-syllable, we use the
original notes grouping in the input melody, and assign one syllable to each
note group. In this case, the length of target translation is known. Following
Lakew et al. (2019), we use length tag “[LEN$i]” to control the length of
outputs during pre-training, where $i refers to the length of the target
sequence.

##  5 Generating Melody-constrained Lyrics and Validating Singability

This section details data sets, model configuration, and proposed evaluation
metrics. Then we analyze the results and the trade-offs inherent in song
translation. Our code and data are open-sourced at https://github.com/GagaST.

###  5.1 Training Datasets and Model Configuration

##### WMT dataset:

news commentary and back-translated news datatsets from wmt14 (29.6 million
en2zh sentence pairs). No Cantonese texts included and the official Chinese
texts can be pronounced in Mandarin by default.

##### Monolingual lyrics data:

monolingual lyrics in both Mandarin and English collected from the web (12.4
million lines of lyrics for Mandarin and 109.5 million for English after
removing duplicates).

##### Lyrics translation data:

a small set of lyrics translation data crawled from the web
777https://lyricstranslate.com/ (140 thousands pairs of English-to-Mandarin
lines). These translations are not singable.

We preprocess all training data with fastBPE (Sennrich et al., 2016) and a
code size of 50,000. We use encoder-decoder Transformer (Vaswani et al., 2017)
with 768 hidden units, 12 heads, gelu activation, 512 max input length, 12-12
layers structure (Appendix B for more details).

###  5.2 Evaluation Datasets

For evaluation, we need aligned triples (melody $M$, source lyrics $X$, target
reference lyrics $Y$), where two conditions hold: 1) $M$ and $X$ are syllable-
to-note aligned; 2) the reference $Y$ should be singable and intelligible.
With the confluence of digitization and copyright making such resources rare,
we choose fifty songs from the lyrics translation dataset that have open-
source music sheets on the web and create aligned triples manually. However,
the reference lyrics in this dataset are not singable (our primary goal!), we
use them to validate that the translations preserve the original meaning.
Twenty songs comprise the validation set (464 lines) and thirty songs comprise
the test set (713 lines).

###  5.3 Evaluation Metrics

An ast system for tonal languages should generate translated songs that are
singable and intelligible while conveying the original meaning. Evaluating
such system is an intrinsically hard task since all three qualities can be
qualitative. Especially for preserving meaning, the lack of gold references
and the greater tolerance for a loose translation in songs make it difficult
to say how much semantic divergence is acceptable. Therefore, we first
establish evaluations based on the relationship between lyrics and music and
then design human annotations for more qualitative evaluations.

####  5.3.1 Objective Evaluation

Section 3.3 outlines three constraints inspired by music and linguistic
theory. Because these constraints are directly incorporated into the decoding
objective (Equation 4), these will be better than an unconstrained
translation. However, we want to understand the trade-off between these new
objectives and traditional translation evaluations.

To control for the length of the sentence, we normalize the score to 0–1.0 by
the length of alignment pairs $L_{i}$, that is, based on Equation 1,2 and 3,

$s_{[\cdot]}=\sum_{1}^{L_{i}}S_{[\cdot]}^{i}/L_{i},$ (5)

For the length constraint, we compute: 1) $N_{l}$, the number of samples that
has length longer than the predefined length $L_{i}$; 2) $N_{s}$, that are
shorter than $L_{i}$. For each case we compute the average error ratio of
$\\{\Delta l_{i}/L_{i}\\}_{1}^{N_{[\cdot]}}$. For meaning, although we lack
gold singable translations, we follow the common practice and calculate bleu
(Papineni et al., 2002) between the translated songs and the prose
translation.

Figure 7: Trade-off between meaning ($y$-axis) and lyric-music alignments
($x$-axis) while adjusting the tuning parameter $\lambda$ on the validation
set. The selected value for the tuning parameter $\lambda$ for downstream
experiments is shown in red (preceeded by $\lambda=$). rest constraints do not
affect BLEUs, but increase the number of [punc]s, which impairs the fluency of
the lyrics, thus we select its parameter based on number of [punc]s.

Syllable-notes | Model | Pitch | Rhythm | Length | Meaning
---|---|---|---|---|---
Assignment | inter $\uparrow$ | intra $\uparrow$ | avg # of missed rests $\downarrow$ | longer $\downarrow$ | shorter $\downarrow$ | BLEU $\uparrow$
note-to-syllable |  gagast w/o constraints | 0.28 | - | 0.53 | 9 (0.09) | 0 | 24.0
gagast | 0.51 | - | 0.31 | 26 (0.21) | 0 | 16.9
–only inter-syllable | 0.51 | - | 0.45 | 26 (0.21) | 0 | 16.8
–only rest | 0.28 | - | 0.31 | 11 (0.09) | 0 | 23.8
syllable-to-syllable |  gagast w/o constraints | 0.29 | 0.49 | 0.62 | 4 (0.12) | 0 | 22.1
gagast | 0.50 | 0.55 | 0.28 | 13 (0.13) | 0 | 15.9
–only inter-syllable | 0.51 | 0.50 | 0.42 | 7 (0.12) | 0 | 15.8
–only intra-syllable | 0.29 | 0.56 | 0.44 | 4 (0.12) | 0 | 21.6
–only rest | 0.29 | 0.49 | 0.28 | 5 (0.12) | 0 | 21.6

Table 2: Our song-specific constraints with two syllable alignment techniques.
All results here use the same pre-training checkpoint and length tags are
applied. For length score, 9 (0.09) means that 9 out of 713 samples are longer
than the predefined length with an average ratio 0.09. All constraints have an
effect, but inter-syllable pitch alignment has the largest.

###  5.4 Trade-offs between Meaning and Melody-lyric Alignments

gagast adds constraints in the decoding scoring functions to enforce lyric-
music alignments; however, there are trade-offs between preserving meaning and
adhering to these constraints. To select the importance of these constraints
in decoding, we vary the value of the corresponding parameter $\lambda$
(Equation 4) and analyze how much the bleu score falls on the validation set
as we increase the influence of the parameter. We set the hyper-parameters
where the alignment scores increase fast while the bleu decreases slowly. The
REST constraint does not affect the bleu (Table 2) but does alter ammount of
punctuation. Working off the assumption that excessive punctuation is bad, we
select a parameter that minimizes the mismatches between the REST and word
boundaries. We choose (Figure 7) $\lambda_{\mbox{inter}}=0.5;\
\lambda_{\mbox{intra}}=1.0;\ \lambda_{R}=1.5$ for all subsequent experiments.

####  5.4.1 BLEU Evaluation

Table 2 compares gagast as we ablate constraints with our two syllable to note
alignment strategies (Section 4): note-to-syllable and syllable-to-syllable.
As in previous work, the length tag “[LEN$i]” helps lyrics fit the notes
available. In all cases, less than 30 out of 713 lines produces a longer
sentence with ratio less than 0.22; and no short cases. Thus, because it most
closely resembles prior work in controlled translation and works well in this
task, we adopt gagast with only length tags and no other constraints as our
baseline. With all of the constraints, gagast indeed increases both pitch and
rhythm alignments. It almost doubles the pitch contour alignment score, which
affect the intelligibility the most.

However, these gains come at the cost of bleu score.888Due the paucity of
reliable references, bleu scores do not correlate with human judgement. For
example, three official Disney Mandarin song translations have a lower bleu
score (12.3) than our more literal but demonstrably worse automatic
translations. While we believe that the audience would be more accepting of a
less-than-literal translation in a song if it sounds better, we need a
qualitative evaluation to validate that hypothesis.

####  5.4.2 Qualitative Evaluation

The true test of whether ast works is whether the songs can be sung,
understood, and enjoyed. Thus, we follow Sheng et al. (2021) and show
annotator from a music school students the resulting sheet music, ask their
opinion, and ask them to sing the songs. We randomly select five songs from
the test set and show the music sheets (see Appendix C) of the first ten
sentences of each translated song to five annotators.

Following mean opinion score (Rec, 1994, mos) in speech synthesis, we use
five-point Likert scales (1 for bad and 5 for excellent). And we evaluate the
songs on four dimensions: 1) sense, fidelity to the meaning of the source
lyric; 2) style, whether the translated lyric resembles song-text style; 3)
listenability, whether the translated lyric sounds melodious with the given
melody; 4) intelligibility, whether the audience can easily comprehend the
translated lyrics if sung with provided melody. The last two dimensions
require the annotators to sing the song.

Model | Song | sense | style | listenability | intelligibility
---|---|---|---|---|---
| Song1 | 3.4 | 3.0 | 3.2 | 3.4
Song2 | 3.6 | 3.9 | 3.4 | 3.8
gagast | Song3 | 3.7 | 3.6 | 3.4 | 3.5
w/o constraints | Song4 | 3.2 | 3.0 | 2.8 | 3.0
| Song5 | 3.7 | 3.6 | 3.4 | 3.8
Average |  3.5 ±0.14 | 3.4 ±0.14 | 3.2 ±0.12 | 3.5 ±0.13
gagast | Song1 | 3.5 | 3.1 | 3.3 | 3.5
Song2 | 3.4 | 3.7 | 3.5 | 4.0
Song3 | 3.2 | 3.6 | 3.3 | 3.6
Song4 | 2.9 | 3.0 | 3.1 | 3.5
Song5 | 3.4 | 3.6 | 3.2 | 3.9
Average | 3.3 ±0.15 | 3.4 ±0.15 |  3.3 ± 0.12 |  3.7 ±0.13

Table 3: Qualitative evaluation results for gagast w/o constraints and gagast.

####  5.4.3 Qualitative Evaluation Results

To examine whether the proposed constraints improve the singability and
intelligibility, our qualitative evaluation compares gagast with only length
constraints to fully constrained gagast (Table 3) with syllable-to-syllable
assignment. While the constraints significantly improve the intelligibility
and slightly improve the singability (listening experience), these constraints
make it harder for the original meaning to come through. Overall, the
annotators are satisfied with the translated songs by the proposed baseline
gagast. All aspects receive an average score around 3.5 out of 5. These case
studies and three translated songs by gagast sung by an amateur singer are
available on https://gagast.github.io/posts/gagast.

##  6 Related Work

##### Verse Generation and Translation.

Generating verse text began through rule-based implementations Milic (1970)
and developed through the next forty years. Manurung (1999) design a chart
system that generate strings that match a given stress pattern. Gervás (2000)
build a forward reasoning rule-based system. Manurung (2004) address poetry
generation with stochastic search based on evolutionary algorithms. Oliveira
(2012) create a template-based platform that allows user to define features
and create templates. He et al. (2012) adopt statistical machine translation
models for Chinese poetry generation. Yan et al. (2013) compose poetry based
on generative summarization framework. Zhang and Lapata (2014), Wang et al.
(2016), and Hopkins and Kiela (2017) adopt recurrent neural networks for
poetry generation and incorporate rhythmic constraints. Ghazvininejad et al.
(2016, 2017) represent rhythm and rhyme with finite-state machines. Poetry
translation using these frameworks and statistical machine translation thus
offers elegant solutions: Genzel et al. (2010) intersect the finite state
representation of the meter and rhyme scheme with the synchronous context-free
grammar of the translation model under the phrase-based machine translation
framework. Ghazvininejad et al. (2018) apply the finite-state constraints to
neural translation model. However, these representations of the rhythmic and
lexical constraints are not flexible enough to encode the real-valued
representation of a _song_ as required for translation in tonal languages.

##### Constrained Text Generation.

Most natural language generation tasks, including machine translation
(Bahdanau et al., 2015; Vaswani et al., 2017; Hassan et al., 2018), dialogue
system (Shang et al., 2015; Li et al., 2016; Wang et al., 2021) and
abstractive summarization (Rush et al., 2015; Paulus et al., 2018), are free
text generation. However, there is a need to generate text with constraints
for special tasks (Lakew et al., 2019; Li et al., 2020; Zou et al., 2021).
Hokamp and Liu (2017); Post and Vilar (2018); Hu et al. (2019) attempt to
constrain the beam search with dictionary. In the training procedure, Li et
al. (2020) add format embedding. Lakew et al. (2019) introduce length tag.
Saboo and Baumann (2019) address length control via rescoring the results of
beam search for machine translation under dubbing constraints.

##### Lyrics Generation.

As one of the most important tasks in automatic songwriting, lyrics generation
has received more attention recently. Sheng et al. (2021), Lee et al. (2019)
and Chen and Lerch (2020) generate lyrics via pure data driven models without
adding constraints based on expert knowledge. Oliveira et al. (2007) build a
rule-based lyrics generation system to handle rhyme and rhythm with designed
heuristics. Malmi et al. (2016) address rap lyrics generation via information-
retrieval approach and propose a rhyme-density measure. Watanabe et al. (2018)
add conditions in standard RNNLM with a featurized input melody for rhythmic
alignment. Ma et al. (2021) develop a SeqGAN-based lyrics generator to address
various properties, such as rhythmic alignment, theme and genre. Xue et al.
(2021) use transformer-based model to generate rap lyrics with a reverse
order, address rhymes with vowel embeddings and add extra beat tokens for
rthymic alignment. We are the first paper that formally address the importance
of aligning melody pitch with languages tones in lyrics generation for tonal
languages. We introduce two vital qualities of songs, singability and
intelligibility, and design three types of melody-lyric alignment scores to
improve the two qualities.

##  7 Conclusion

This paper addresses automatic song translation (ast) for tonal languages and
the unique challenge of aligning words’ tones with melody. And we build the
first English-Mandarin ast system – gagast. Both objective and subjective
evaluations demonstrate that gagast successfully improves the singability and
intelligibility of translated songs.

More constraints are left in the future work such as rhymes and style. We aim
to build a systematic framework that address all constraints. With the help of
newly developed singing voice synthesize tools such as X
Studio,999https://singer.xiaoice.com we can perform human evaluation with
actual singing voice with a larger scale to provide more reliable analysis.
Moreover, our system can also be applied in lyrics and song generation
applications without translation input.

## Ethical Considerations

gagast improves singability and intelligibility of the translated songs in
Mandarin via constraining the decoding of a pretrained lyrics translation
model. This methodology has limitations by imposing a direct trade-offs
between the original objective and the constraints. In terms of negative
impact or risks, the inaccurate translations may cause misunderstandings in
applications like Musical Theatre.

This paper collects lyrics data that are publicly available and are parsed
from the Web. We use these data for research purposes only. To prevent any
abuse or piracy of these data, we chose the dataset license Attribution-
NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0).

## Acknowledgements

This work was initially supported by Alibaba damo Academy through Alibaba
Innovative Research Program. We thank the anonymous reviewers for helpful
feedback on previous versions of this work. We thank Carol, Xin Xu and igp for
the helpful discussions about music. Finally, we thank the singer/songwriter
Ayanga for his professional discussion about the difficulties in musical
translations, which was the genesis of this paper. Boyd-Graber and Guo’s work
is supported in part by a gift from Adobe.

## References

  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _Proceedings of the International Conference on Learning Representations_.
  * Bellos (2013) David Bellos. 2013.  _Fictions of the Foreign: The Paradox of “Foreign-Soundingness”_, pages 31–43. Columbia University Press.
  * Chen and Lerch (2020) Yihao Chen and Alexander Lerch. 2020.  Melody-conditioned lyrics generation with seqgans.  In _2020 IEEE International Symposium on Multimedia (ISM)_.
  * Franzon (2008) Johan Franzon. 2008.  Choices in song translation: Singability in print, subtitles and sung performance.  _The Translator_ , 14(2):373–399.
  * Genzel et al. (2010) Dmitriy Genzel, Jakob Uszkoreit, and Franz Och. 2010.  “poetic” statistical machine translation: Rhyme and meter.  In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_ , pages 158–166, Cambridge, MA. Association for Computational Linguistics.
  * Gervás (2000) Pablo Gervás. 2000.  WASP: Evaluation of different strategies for the automatic generation of spanish verse.  In _Proceedings of the AISB-00 symposium on creative & cultural aspects of AI_.
  * Ghazvininejad et al. (2018) Marjan Ghazvininejad, Yejin Choi, and Kevin Knight. 2018.  Neural poetry translation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 67–71. Association for Computational Linguistics.
  * Ghazvininejad et al. (2016) Marjan Ghazvininejad, Xing Shi, Yejin Choi, and Kevin Knight. 2016.  Generating topical poetry.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1183–1191. Association for Computational Linguistics.
  * Ghazvininejad et al. (2017) Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017.  Hafez: an interactive poetry generation system.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 43–48. Association for Computational Linguistics.
  * Greene et al. (2010) Erica Greene, Tugba Bodrumlu, and Kevin Knight. 2010.  Automatic analysis of rhythmic poetry with applications to generation and translation.  In _Proceedings of Empirical Methods in Natural Language Processing_.
  * Hassan et al. (2018) Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William D. Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018.  Achieving human parity on automatic Chinese to English news translation.  _ArXiv_ , abs/1803.05567.
  * He et al. (2012) Jing He, Ming Zhou, and Long Jiang. 2012.  Generating Chinese classical poems with statistical machine translation models.  In _Association for the Advancement of Artificial Intelligence_.
  * Hofstadter (1997) Douglas R Hofstadter. 1997.  _Le ton beau de Marot: In praise of the music of language_.  Basic Books New York.
  * Hokamp and Liu (2017) Chris Hokamp and Qun Liu. 2017.  Lexically constrained decoding for sequence generation using grid beam search.  In _Proceedings of the Association for Computational Linguistics_ , pages 1535–1546. Association for Computational Linguistics.
  * Hopkins and Kiela (2017) Jack Hopkins and Douwe Kiela. 2017.  Automatically generating rhythmic verse with neural networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 168–178. Association for Computational Linguistics.
  * Hu (2017) Fangzhou Hu. 2017.  Lexical tones in mandarin sung words: A phonetic and psycholiguistic investigation.  Master’s thesis, Shanghai International Studies University.
  * Hu et al. (2019) J. Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. 2019.  Improved lexically constrained decoding for translation and monolingual rewriting.  In _Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 839–850. Association for Computational Linguistics.
  * Lakew et al. (2019) Surafel Melaku Lakew, Mattia Di Gangi, and Marcello Federico. 2019.  Controlling the output length of neural machine translation.  In _Proceedings of the 16th International Conference on Spoken Language Translation_. Association for Computational Linguistics.
  * Lee et al. (2019) Hsin-Pei Lee, Jhih-Sheng Fang, and Wei-Yun Ma. 2019.  iComposer: An automatic songwriting system for Chinese popular music.  In _Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 84–88. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the Association for Computational Linguistics_ , pages 7871–7880. Association for Computational Linguistics.
  * Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Georgios P. Spithourakis, Jianfeng Gao, and William B. Dolan. 2016.  A persona-based neural conversation model.  _ArXiv_ , abs/1603.06155.
  * Li et al. (2020) Piji Li, Haisong Zhang, Xiaojiang Liu, and Shuming Shi. 2020.  Rigid formats controlled text generation.  In _Proceedings of the Association for Computational Linguistics_ , pages 742–751. Association for Computational Linguistics.
  * Low (2003) Peter Low. 2003.  Singable translations of songs.  _Perspectives: Studies in Translatology_ , 11(2):87–103.
  * Ma et al. (2021) Xichu Ma, Ye Wang, Min-Yen Kan, and Wee Sun Lee. 2021.  Ai-lyricist: Generating music and vocabulary constrained lyrics.  In _Proceedings of the 29th ACM International Conference on Multimedia_.
  * Malmi et al. (2016) Eric Malmi, Pyry Takala, Hannu Toivonen, Tapani Raiko, and Aristides Gionis. 2016\.  Dopelearning: A computational approach to rap lyrics generation.  In _Knowledge Discovery and Data Mining_.
  * Manurung (2004) Hisar Manurung. 2004.  _An evolutionary algorithm approach to poetry generation_.  Ph.D. thesis, University of Edinburgh.
  * Manurung (1999) Hisar Maruli Manurung. 1999.  A chart generator for rhythm patterned text.  In _Proceedings of the First International Workshop on Literature in Cognition and Computer_.
  * Milic (1970) Louis T. Milic. 1970.  The possible usefulness of poetry generation.  In _Symposium on the Uses of Computers in Literary Research_.
  * Oliveira (2012) Hugo Gonçalo Oliveira. 2012.  Poetryme: a versatile platform for poetry generation.  _Computational Creativity, Concept Invention, and General Intelligence_ , 1:21.
  * Oliveira et al. (2007) Hugo R Gonçalo Oliveira, F Amilcar Cardoso, and Francisco C Pereira. 2007\.  Tra-la-lyrics: An approach to generate text based on rhythm.  In _Proceedings of the 4th. International Joint Workshop on Computational Creativity_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the Association for Computational Linguistics_.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018.  A deep reinforced model for abstractive summarization.  In _Proceedings of the International Conference on Learning Representations_.
  * Post and Vilar (2018) Matt Post and David Vilar. 2018.  Fast lexically constrained decoding with dynamic beam allocation for neural machine translation.  In _Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 1314–1324. Association for Computational Linguistics.
  * Rec (1994) ITUT Rec. 1994.  P. 85. a method for subjective performance assessment of the quality of speech voice output devices.  _International Telecommunication Union, Geneva_.
  * Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  In _Proceedings of Empirical Methods in Natural Language Processing_ , pages 379–389. Association for Computational Linguistics.
  * Saboo and Baumann (2019) Ashutosh Saboo and Timo Baumann. 2019.  Integration of dubbing constraints into machine translation.  In _Proceedings of the Fourth Conference on Machine Translation_.
  * Schellenberg (2013) Murray Henry Schellenberg. 2013.  _The realization of tone in singing in Cantonese and Mandarin_.  Ph.D. thesis, University of British Columbia.
  * Schneider (1961) Marius Schneider. 1961.  Tone and tune in West African music.  _Ethnomusicology_ , 5(3):204–215.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the Association for Computational Linguistics_ , pages 1715–1725. Association for Computational Linguistics.
  * Shang et al. (2015) Lifeng Shang, Zhengdong Lu, and Hang Li. 2015.  Neural responding machine for short-text conversation.  In _Proceedings of the Association for Computational Linguistics_ , pages 1577–1586. Association for Computational Linguistics.
  * Sheng et al. (2021) Zhonghao Sheng, Kaitao Song, Xu Tan, Yi Ren, Wei Ye, Shikun Zhang, and Tao Qin. 2021\.  Songmass: Automatic song writing with pre-training and alignment constraint.  In _Association for the Advancement of Artificial Intelligence_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of Advances in Neural Information Processing Systems_.
  * Wang et al. (2016) Qixin Wang, Tianyi Luo, Dong Wang, and Chao Xing. 2016.  Chinese song iambics generation with neural attention-based model.  In _International Joint Conference on Artificial Intelligence_.
  * Wang et al. (2021) Weizhi Wang, Zhirui Zhang, Junliang Guo, Yinpei Dai, Boxing Chen, and Weihua Luo. 2021.  Task-oriented dialogue system as natural language generation.  _ArXiv_ , abs/2108.13679.
  * Watanabe et al. (2018) Kento Watanabe, Yuichiroh Matsubayashi, Satoru Fukayama, Masataka Goto, Kentaro Inui, and Tomoyasu Nakano. 2018.  A melody-conditioned lyrics language model.  In _Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 163–172. Association for Computational Linguistics.
  * Wee (2007) Lian Hee Wee. 2007.  Unraveling the relation between mandarin tones and musical melody.  _Journal of Chinese Linguistics_ , 35(1):128.
  * Xu (1997) Yi Xu. 1997.  Contextual tonal variations in mandarin.  _Journal of phonetics_ , 25(1):61–83.
  * Xue et al. (2021) Lanqing Xue, Kaitao Song, Duocai Wu, Xu Tan, Nevin L Zhang, Tao Qin, Wei-Qiang Zhang, and Tie-Yan Liu. 2021.  Deeprapper: Neural rap generation with rhyme and rhythm modeling.  In _Proceedings of the Association for Computational Linguistics_.
  * Yan et al. (2013) Rui Yan, Han Jiang, Mirella Lapata, Shou-De Lin, Xueqiang Lv, and Xiaoming Li. 2013\.  I, poet: automatic Chinese poetry composition through a generative summarization framework under constrained optimization.  In _International Joint Conference on Artificial Intelligence_.
  * Yinliu et al. (1983) Yang Yinliu, Sun Congyin, and Wu Junda. 1983.  _Language and Music_.  People’s Music Publishing House.
  * Yip (2002) Moira Yip. 2002.  _Tone_.  Cambridge University Press.
  * Yu (2021) Weimin Yu. 2021.  The study of the fourth tone in kunqu opera.  _Journal of National Academy of Chinese Theatre Art_.
  * Zhang and Lapata (2014) Xingxing Zhang and Mirella Lapata. 2014.  Chinese poetry generation with recurrent neural networks.  In _Proceedings of Empirical Methods in Natural Language Processing_ , pages 670–680. Association for Computational Linguistics.
  * Zhuang (1982) Yongping Zhuang. 1982.  The relation of the sung words in peking opera.
  * Zou et al. (2021) Xu Zou, Da Yin, Qingyang Zhong, Hongxia Yang, Zhilin Yang, and Jie Tang. 2021.  Controllable generation from pre-trained language models via inverse prompting.  In _Knowledge Discovery and Data Mining_ , page 2450–2460. Association for Computing Machinery.

## Appendix A

### A.1 Illustration of Tonal Alignment by Frequency

Translating songs into tonal languages faces a unique challenge,i.e., the
tones of the translated lyrics should align with the music pitch for
singability and intelligibility (Section 2.1). Figure 8 provides visual
illustration of the main problem.

To help researchers who speak non-tonal languages understand better how the
tones of lyrics in tonal languages should align with the music/sung voice, we
record both sung and spoken voice of a piece of lyrics from one of the most
popular songs in Mandarin, transform the sound into the frequency space, and
compare the shape of the sound with that of the music in Figure 9. The
original music of the chosen song is from an American song “Dreaming of Home
and Mother”, and was rewritten in Mandarin. Despite that this is not a
translation task and do not have to convey the original meaning, we can see
how the tonal contour of the lyrics in Mandarin align with that of music.

### A.2 Acceptable Pitch Transition Directions Table

In Section 2.2, we explain that in practice, the relative relationship of the
pitch of the tones of the successive syllables/characters that belongs to the
same word affect the most to the singability and intelligibility. And we
summarize the acceptable transition directions in Figure 5 under the
assumption that only relative relationship of successive notes matters. It
should be noted that we intend merely to provide a workable solution but not a
perfect one. For example, the handle of the fourth notes of Mandarin is
actually very tricky. It is a continuous fall with a large range (see Figure
2), therefore it doesn’t represent one note. If it were to be represented by
one note, it might represent the onset or offset part of the tone, and the
falling trend is hinted by the pitch contour with proceeding and/or following
note (Zhuang, 1982; Yu, 2021).

Parameter | Value
---|---
encoder layer | 12
decoder layer | 12
max source position | 512
max target position | 512
layernorm embedding | True
criterion | label smoothed cross entropy
learning rate | 3e-4
label smoothing | 0.2
min lr | 1e-9
lr scheduler | inverse sqrt
warmup updates | 4000
warmup initial lr | 1e-7
optimizer | adam
adam epsilon | 1e-6
adam betas | (0.9, 0.98)
weight decay | 0.01
dropout | 0.1
attention dropout | 0.1
text infilling
mask rate | 0.3
poisson lambda | 3.5
replace length | 1

Table 4: Pretraining hyper-parameters

## Appendix B

### B.1 Training Details

We pretrain our transformer-based model with reconstruction objective and
corrupt our input sequence with text infilling (Lewis et al., 2020). More
detailed pretraining hyper-parameters can be found in Table 4.

## Appendix C

### C.1 Human Evaluation Instruction

In this paper, we conduct subjective evaluations by collecting annotations
about the qualities of the translated songs from music school students
(Section 5.4.2).

### C.2 Music Sheets

As describe in Section 5.4.2 and shown in the instructions (Figure 10), we
distributes music sheets of the translated songs to the annotators. All music
sheets can be found on https://gagast.github.io/posts/human_eval.

Figure 8: The pitch contour of the prose translation (bottom line, in
Mandarin) of lyrics do not match that of the original music (upper line). The
directions showed in figure is estimated by the base frequency of spoken sound
by text-to-speech tools. Such mismatch in pitch contour makes the sung lyrics
sound unnatural and hard to understand. Figure 9: An example of a piece of a
popular rewritten song in Mandarin “Farewell (sòng bié)”. The original music
is from an American song “Dreaming of Home and Mother”. We record the sung and
spoken voice and plot the actual base frequency of the sound. We can see how
the tone shape and overall tonal contour aligns with the sung voice (by the
music pitch). Figure 10: Instructions for human evaluation