# SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint

Zhonghao Sheng1,111The first two authors contribute equally to this work.,
Kaitao Song2,∗, Xu Tan222Corresponding Author: Xu Tan<EMAIL_ADDRESS>and
Wei Ye<EMAIL_ADDRESS>Yi Ren4, Wei Ye1, Shikun Zhang1, Tao Qin3

###### Abstract

Automatic song writing aims to compose a song (lyric and/or melody) by
machine, which is an interesting topic in both academia and industry. In
automatic song writing, lyric-to-melody generation and melody-to-lyric
generation are two important tasks, both of which usually suffer from the
following challenges: 1) the paired lyric and melody data are limited, which
affects the generation quality of the two tasks, considering a lot of paired
training data are needed due to the weak correlation between lyric and melody;
2) Strict alignments are required between lyric and melody, which relies on
specific alignment modeling. In this paper, we propose SongMASS to address the
above challenges, which leverages masked sequence to sequence (MASS) pre-
training and attention based alignment modeling for lyric-to-melody and
melody-to-lyric generation. Specifically, 1) we extend the original sentence-
level MASS pre-training to song level to better capture long contextual
information in music, and use a separate encoder and decoder for each modality
(lyric or melody); 2) we leverage sentence-level attention mask and token-
level attention constraint during training to enhance the alignment between
lyric and melody. During inference, we use a dynamic programming strategy to
obtain the alignment between each word/syllable in lyric and note in melody.
We pre-train SongMASS on unpaired lyric and melody datasets, and both
objective and subjective evaluations demonstrate that SongMASS generates lyric
and melody with significantly better quality than the baseline method without
pre-training or alignment constraint.

##  1 Introduction

Automatic song writing is an interesting and challenging task in both research
and industry. Two most important tasks in automatic song writing are lyric-to-
melody generation (L2M) (Bao et al. 2019; Yu and Canales 2019; Lee, Fang, and
Ma 2019) and melody-to-lyric generation (M2L) (Watanabe et al. 2018; Lu et al.
2019; Lee, Fang, and Ma 2019). L2M and M2L can be regarded as sequence to
sequence learning tasks and can be modeled by the techniques in natural
language processing since both melody and lyric can be represented as discrete
token sequence. However, L2M and M2L have distinctive characteristics that
differ them from other sequence to sequence learning tasks: 1) lyric and
melody are weakly correlated in L2M and M2L while in other tasks (Bahdanau,
Cho, and Bengio 2014; Rush, Chopra, and Weston 2015), source and target
sequences are strongly correlated in semantics; 2) one word or syllable in
lyric always strictly aligns with one or more notes in melody, while other
tasks do not require strict alignments. An example of aligned lyric and melody
piece is shown in Figure 1.

The above distinctive characteristics throw several challenges in modeling L2M
and M2L: 1) They require large amount of paired melody and lyric data to learn
the mapping relationship between lyric and melody due to weak correlation.
However, it is difficult to collect such large amount of paired data, and thus
both tasks suffer from limited paired data; 2) They need additionally generate
strict alignments between word/syllable in lyric and note in melody, and thus
how to model the alignments well is critical to ensure the generation quality
of lyric and melody. Previous works (Bao et al. 2019; Li et al. 2020; Watanabe
et al. 2018; Lee, Fang, and Ma 2019) on L2M and M2L have not considered the
scenario of limited paired data, and only leverage some greedy decisions for
lyric and melody alignment, which cannot well address these challenges. In
this paper, we propose SongMASS, an automatic song writing system for L2M and
M2L, which addresses the first challenge with masked sequence to sequence pre-
training and the second challenge with attention based alignment constraint.

Figure 1: A song fragment “Another day has gone, I’m still all alone” with its
melody. The table shows the alignment of the lyric and melody (pitch and
duration).

Specifically, to handle the challenges of limited paired data, we leverage
self-supervised pre-training on large amount of unpaired lyric and melody
data. Since L2M and M2L are both sequence to sequence learning tasks, we adopt
masked sequence to sequence pre-training (MASS) (Song et al. 2019), which is a
popular pre-training method by masking a segment in the input sequence and
predicting this segment in the output using an encoder-decoder framework.
However, simply using original MASS in L2M and M2L cannot well handle the long
lyric and melody sequence in song level and the diversity between lyric and
melody modality. Therefore, we introduce two extensions on MASS: 1) Instead of
masking in a single sentence in original MASS, we design a song-level masked
pre-training strategy to capture longer contextual information, since music
usually has repeat structure and relies on long context. 2) Unlike original
MASS, we use separate encoder-decoder for lyric-to-lyric and melody-to-melody
masked pre-training since they are in different modalities. However, separate
training of lyric-to-lyric and melody-to-melody cannot ensure to learn a
shared latent space between lyric and melody and thus could harm the
transformation between them. Therefore, we add supervised training with paired
lyric and melody data to guide the pre-training towards learning a shared
latent representation between lyric and melody modality.

To address the challenges of lyric-melody alignment, we propose to align the
word/syllable in lyric and note in melody based on the encoder-decoder
attention. Due to long melody and lyric sequence in a song, we split the
alignment into sentence level and token level. To ensure sentence-level
alignment, we constrain each sentence in target sequence to only attend to the
corresponding sentence in source sequence during training and inference. We
add an additional [SEP] token as the sentence boundary in each sequence, and
during inference, once a [SEP] token is predicted in the target side, we
switch the attention to the next source sentence. For token-level alignment,
we add constraints on the attention matrix using the ground-truth alignment in
the paired training data during training, and use a dynamic programming
algorithm on the generated attention matrix during inference to obtain the
final alignments.

The contributions of our method are as follows:

  * •

We are the first to leverage pre-training to address the low-resource
challenge on L2M and M2L, by introducing two extensions on MASS including
song-level masked pre-training and using supervised pre-training to guide the
separate encoder-decoder of lyric and melody to the same latent space.

  * •

To handle the alignment between word/syllable in lyric and note in melody, we
design the attention-based sentence-level and token-level alignment
constraints and a dynamic programming algorithm to obtain precise alignments.

  * •

Experimental results with objective and subjective evaluations demonstrate
that SongMASS significantly improves the quality of lyric and melody
generation with the help of pre-training and alignment constraint.

##  2 Background

#### Automatic Song Writing

Automatic song writing usually covers several tasks including lyric generation
(Malmi et al. 2015), melody generation (Zhu et al. 2018), lyric-to-melody
generation (L2M) (Choi, Fazekas, and Sandler 2016; Yu and Canales 2019) and
melody-to-lyric generation (M2L) (Bao et al. 2019; Li et al. 2020). In this
work, we focus on L2M and M2L. Choi, Fazekas, and Sandler (2016); Yu and
Canales (2019) generated melody conditioned on the lyrics with RNN-based
language model. Lee, Fang, and Ma (2019); Bao et al. (2019) used sequence to
sequence model for L2M and M2L. However, these works on L2M and M2L usually
only used limited paired data, without leveraging large-scale unpaired data.
On the other hand, some works only focused on L2M and M2L on the sentence
level, assuming there are strict one-to-one mapping in the training data,
which cannot compose a complete song. Some other works (Bao et al. 2019;
Watanabe et al. 2018) explicitly predicted the alignment flag (e.g., whether
switches to next word/syllable when predicting notes or not) in the model,
with a greedy decision in the word/syllable or note level, which is not
flexible and fail to capture the global alignment in the whole sentence. In
this paper, we propose SongMASS, which uses sequence to sequence pre-training
method to leverage the unpaired lyric and melody data, and attention-based
alignment constraints for global and precise lyric-melody alignment.

#### Pre-training Methods

Pre-trained language models (_e.g._ , BERT (Devlin et al. 2019), GPT (Radford
et al. 2018), XLNet (Yang et al. 2019), MASS (Song et al. 2019) and _etc_)
have achieved significant progress in natural language processing. They
usually employ specific self-supervised tasks and pre-train on large-scale
unlabeled data corpus to improve the understanding and generation capability.
MASS (Song et al. 2019) is the first and one of most successful pre-training
methods for sequence to sequence learning tasks, and several pre-training
methods such as BART (Lewis et al. 2019) and T5 (Raffel et al. 2019) are also
proposed to handle this kind of task. In this paper, we build our pre-training
method upon MASS considering its popularity for sequence to sequence learning
tasks and suitability for different modalities. Given a sequence from the
unpaired sentence corpus, MASS randomly replaces a segment of tokens with mask
tokens and takes the masked sequence as the encoder input and predicts the
masked segment in the decoder. We leverage the basic idea of MASS and extend
it with several improvements to address the distinctive challenges in the pre-
training of L2M and M2L.

#### Alignment Modeling

Alignment modeling builds the correlation between the tokens in source and
target sequences, which plays an important role in sequence to sequence tasks.
In L2M and M2L tasks, previous works usually used greedy alignment mechanisms
to handle the correlation between lyric and melody. For example, Watanabe et
al. (2018) used the Needleman-Wunsch algorithm (Needleman and Wunsch 1970) to
count the alignment of lyric and melody. Bao et al. (2019) predicted how many
syllables in the predicting word given current note input. However, these
greedy alignment strategies cannot provide flexible and global alignments in
the sentence level. In other sequence to sequence learning tasks like neural
machine translation, Bahdanau, Cho, and Bengio (2014); Luong, Pham, and
Manning (2015) introduced attention mechanism to learn the relationship
between source and target languages. In this paper, we leverage the attention
mechanism to build the global and soft alignment between lyric and melody, and
finally design a dynamic programming method to obtain the strict alignment
between word/syllable and note.

##  3 Method

###  3.1 System Overview

The overall architecture of SongMASS for L2M and M2L is shown in Figure 2,
which adopts the Transformer (Vaswani et al. 2017) based encoder-decoder
framework. We employ separated encoders and decoders for lyric and melody
respectively due to the large diversity between lyric and melody. To leverage
the knowledge from large-scale unlabeled lyrics or melodies, we perform MASS
pre-training for lyric-to-lyric and melody-to-melody in our framework. We pre-
train our model in song level to better capture long contextual information
for lyric or melody sequence, and incorporate supervised learning (lyric-to-
melody and melody-to-lyric) into our pre-training to learn a shared latent
space between different modalities. To learn the alignment between
word/syllable in lyric and note in melody, we further leverage sentence-level
constraint and token-level constraint into our model to guide the alignment
between lyric and melody. We use a dynamic programming strategy to obtain the
final strict alignment between the lyric and melody. In below, we describe the
details of our pre-training methods and alignment strategies in SongMASS.

Figure 2:  The overall architecture of our SongMASS framework. The red line
means unsupervised pre-training on lyric-to-lyric or melody-to-melody. The
blue dotted line is supervised pre-training on lyric-to-melody or melody-to-
lyric. Figure 3:  The song-level MASS pre-training.

###  3.2 Pre-training Methods

In this subsection, we introduce our pre-training methods, including song-
level MASS pre-training to capture long contextual information from the whole
song, and supervised pre-training to learn a shared latent space between lyric
and melody modality.

#### Song-Level MASS Pre-training

As mentioned in Section 2, the original MASS pre-training is designed to help
the model understand and generate sequence in sentence level. However, instead
of using sentence-level information, we expect model to capture long
contextual information in song level (_i.e._ , the whole song). Therefore, we
introduce the song-level MASS pre-training to address this issue.

Denote $\mathcal{X^{\prime}}$ and $\mathcal{Y^{\prime}}$ as the corpus of
unpaired lyrics and melodies in the song level respectively. For any
$x\in\mathcal{X^{\prime}}$ and $y\in\mathcal{Y^{\prime}}$, we split the song-
level sequence into multiple sentences and insert a special token $\rm{[SEP]}$
in the boundary of adjacent sentences. For every sentence from the song-level
sequence, we perform the same mask strategy as in the original MASS (as
mentioned in Section 2). The details of the masking strategy are shown in
Figure 3. The encoder takes the masked song-level sequence as input and the
decoder predicts masked fragments corresponding to all the sentences in this
song. The formulation of song-level MASS is as follows:

$\displaystyle
L(\mathcal{X};\theta^{enc},\theta^{dec})=\sum_{x\in\mathcal{X}}\sum_{i=1}^{S}\log
P(x^{u_{i}:v_{i}}|x^{\backslash\\{u_{i}:v_{i}\\}^{S}_{i=1}};\theta^{enc},\theta^{dec})$
(1)
$\displaystyle=\sum_{x\in\mathcal{X}}\sum_{i=1}^{S}\log\prod_{t=u_{i}}^{v_{i}}P(x^{u_{i}:v_{i}}_{t}|x^{u_{i}:v_{i}}_{<t},x^{\backslash\\{u_{i}:v_{i}\\}^{S}_{i=1}};\theta^{enc},\theta^{dec}),$

where $S$ represents the number of sentences in sequence $x$,
$x^{\backslash\\{u_{i}:v_{i}\\}^{S}_{i=1}}$ represents the masked song-level
sequence, and $x^{u_{i}:v_{i}}$ represents the masked segment in the $i$-th
sentence. We define $\theta_{x}^{enc}$, $\theta_{x}^{dec}$,
$\theta_{y}^{enc}$, $\theta_{y}^{dec}$ as the parameters of lyric encoder,
lyric decoder, melody encoder and melody decoder. The loss for lyric-to-lyric
generation is $L_{x}=L(\mathcal{X};\theta^{enc}_{x},\theta^{dec}_{x})$ and the
loss for melody-to-melody is
$L_{y}=L(\mathcal{Y};\theta^{enc}_{y},\theta^{dec}_{y})$.

#### Supervised Pre-training

Although MASS pre-training can help the model understand and generate lyric
and melody respectively, the model cannot learn to generate melody from lyric
and lyric from melody. What is worse is that the encoder-decoder models for
lyric and melody cannot align in the same latent space and may deviate from
each other, which will harm the transformation between lyric and melody. To
prevent them from deviating and help align them together, we leverage the
supervised training on lyric-melody paired data in the pre-training process.
Given paired corpus $(\mathcal{X},\mathcal{Y})$, the loss of the supervised
pre-training is

$\displaystyle{L}(\mathcal{X},\mathcal{Y};\theta^{enc},$
$\displaystyle\theta^{dec})=\sum_{(x,y)\in(\mathcal{X},\mathcal{Y})}\log
P(y|x;\theta^{enc},\theta^{dec})$ (2)
$\displaystyle=\sum_{(x,y)\in(\mathcal{X},\mathcal{Y})}\log\prod_{t=1}^{|y|}P(y_{t}|y_{<t},x;\theta^{enc},\theta^{dec}).$

The supervised pre-training is applied on both lyric-to-melody and melody-to-
lyric generation. The loss for lyric-to-melody generation is
$L_{xy}=L(\mathcal{X},\mathcal{Y};\theta^{enc}_{x},\theta^{dec}_{y})$ and the
loss for melody-to-lyric is
$L_{yx}=L(\mathcal{Y},\mathcal{X};\theta^{enc}_{y},\theta^{dec}_{x})$.

Finally, the total pre-training loss is

$L_{pt}=L_{x}+L_{y}+L_{xy}+L_{yx},$ (3)

where $L_{x}$ and $L_{y}$ are the unsupervised MASS pre-training loss
described in Equation 1, and $L_{xy}$ and $L_{yx}$ are the supervised pre-
training loss. During fine-tuning, we only use $L_{xy}$ for lyric-to-melody
generation and $L_{yx}$ for melody-to-lyric generation.

###  3.3 Alignment Strategy

In this subsection, we describe how to learn the alignment between lyric and
melody in SongMASS. The basic idea is to leverage the encoder-decoder
attention to infer the alignment between each word/syllable in lyric and note
in melody. In order to extract strict alignment, we explicitly add constraints
on the attention to learn effective attention patterns during training and
inference. Due to the long song-level sequence, we divide our alignment
strategy into sentence-level constraint and token-level constraint.

Figure 4:  Sentence-level attention mask.

#### Sentence-Level Constraint

A song consists of multiple lyric sentences and melody phrases. In song
writing, the given lyrics and melodies are naturally grouped into sentences or
phrases. The lyric sentences and the melody phrases are strictly aligned in
the training data. So we constrain that each sentence (lyric sentence or
melody phrase) in the target sequence can only attend to the corresponding
sentence (melody phrase or lyric sentence) in source sequence. Specifically,
we apply a sentence-level constraint mask on the encoder-decoder attention. We
denote $y_{i}$ and $x_{j}$ as the $i$-th token in the target sequence and
$j$-th token in the source sequence respectively. We assume the
representations of $x_{j}$ and $y_{i}$ from the previous Transformer layer as
$h_{j}^{enc}$ and $h_{i}^{dec}$. So the attention score between target token
$y_{i}$ and source token $x_{j}$ is computed as:

$f(i,j)=\frac{h_{i}^{dec}W^{Q}(h_{j}^{enc}W^{K})^{T}}{\sqrt{d_{z}}}+M(i,j),\\\
$ (4) $A(i,j)=\frac{\exp f(i,j)}{\sum_{j}\exp f(i,j)}$ (5)

where $A(i,j)$ calculates the attention score between the $y_{i}$ and $x_{j}$.
$W^{Q}$,$W^{K}$ $\in\mathbb{R}^{d_{z}\times d_{z}}$ are model parameters, and
$d_{z}$ is the dimension of the hidden representations. $M(i,j)$ represents
the mask element between $y_{i}$ and $x_{j}$, whose value is set as follows:

$M(i,j)=\begin{cases}0&\rm{ID}(y_{i})=\rm{ID}(x_{j})\\\
-\infty&\rm{ID}(y_{i})\neq\rm{ID}(x_{j})\end{cases}.$ (6)

where $\rm{ID}(x)$ gets the index of the sentence that the token $x$ belongs
to. $M$ is used as our sentence-level alignment constraint, as shown in Figure
4. Besides, we insert a special token [SEP] in the sentence boundary of the
input and output sequences as shown in Figure 3, to help the model better
capture the sentence boundary information and identify which sentence of the
input sequence should to be attended to at the current step during the
inference stage. Benefiting from such design, we guarantee the number of
sentences in the generated sequences is consistent with the input sequence.

#### Token-Level Constraint

Unlike sentence-level alignment, the alignment choices between each
word/syllable and note are more flexible. Therefore, we propose a
regularization term on the encoder-decoder attention during the training on
paired data, and apply a dynamic programming algorithm on the attention matrix
to obtain the final strict alignment during inference. We expect the attention
weight between $y_{i}$ and $x_{j}$ to follow:

$u(i,j)=\begin{cases}\frac{1}{T}&\text{if $y_{i}$ is aligned to $x_{j}$},\\\
0&\text{Otherwise},\end{cases}$ (7)

where $T$ is the number of tokens in the source sentence that $y_{i}$ is
aligned to. As shown in Figure 5, we add a regularization term to constrain
the attention weights:

$L_{att}=\frac{1}{N*M}\sum_{i=1}^{M}\sum_{j=1}^{N}\|A(i,j)-u(i,j)\|_{2},$ (8)

where $\|\cdot\|$ represents L2-Norm. $N$ and $M$ are the number of tokens in
the source and target sentence respectively. Finally, the loss function is:

$L=L_{pt}+\alpha\cdot L_{att},$ (9)

where $\alpha$ is the hyper-parameter of $L_{att}$, and $L_{pt}$ is the pre-
training loss defined in Equation 3.

Figure 5:  Token-level guided attention mask.

When all tokens in a sentence are generated and the attention matrix $A$ is
obtained, we infer the global alignment by applying a dynamic programming
algorithm as shown in Algorithm 1. We consider the following cases: a target
token is aligned to one or many source tokens, and a source token is aligned
to one or many target tokens. For the first case, as shown in Line 7 - 12 in
Algorithm 1, we search for a $k$ that the alignment between $y_{i}$ and
$x_{[K+1:j]}$ reaches the highest score, which is calculated by summing all
the corresponding attention weights. Similarly for the second case, as shown
in Line 13 - 18 in Algorithm 1, we search for a $k$ that the alignment between
$y_{[k+1:i]}$ and $x_{[j]}$ reaches the highest score. We take the average
weights as score in the second case, since the weights of the target sequence
dimension are not normalized like that of the source sequence. We choose the
higher score of the two cases and save the aligned pair.

Algorithm 1 DP for Melody-Lyric Alignment

1: Input: Attention matrix $A\in\mathbb{R}^{N\times M}$, score matrix
$F\in\mathbb{R}^{(N+1)\times(M+1)}$, path matrix Path, source sequence $x$ and
target sequence $y$. $N$ and $M$ are the length of $x$ and $y$.

2: Output: The aligned pairs list $D$.

3: Initialize: F is initialized as $-\infty$. Path is initialized as an empty
matrix with a shape of $(N+1)\times(M+1)$.

4: $F[0][0]=0$

5: for $i=1$ to $T$ do

6: for $j=1$ to $S$ do

7: for $k=0$ to $j-1$ do

8: $\text{score}=F[i-1][k]+\sum_{h=k+1}^{j}A[i][h]$

9: if $\text{score}\geq F[i][j]$ then

10: $F[i][j]=\text{score}$, $\text{Path}[i][j]=(i-1,k)$

11: end if

12: end for

13: for $k=0$ to $i-1$ do

14: $\text{score}=F[k][j-1]+\sum_{h=k+1}^{i}\frac{A[h][j]}{i-k}$

15: if $\text{score}\geq F[i][j]$ then

16: $F[i][j]=\text{score}$, $\text{Path}[i][j]=(k,j-1)$.

17: end if

18: end for

19: end for

20: end for

21: $m,n=M,N$

22: while $m\neq 0$ and $n\neq 0$ do

23: $i,j=\text{Path}[m][n]$

24: add the aligned pair ($x_{[j+1:n]}$,$y_{[i+1:m]}$) to  D

25: $m,n=i,j$

26: end while

27: return $D$

##  4 Experiments and Results

| Lyric-to-Melody | Melody-to-Lyric
---|---|---
| PD ($\%$) $\uparrow$ | DD ($\%$) $\uparrow$ | MD $\downarrow$ | PPL $\downarrow$ | PPL $\downarrow$
Baseline | 38.20 | 52.00 | 2.92 | 3.27 | 37.50
SongMASS | 57.00 | 65.90 | 2.28 | 2.41 | 14.66
$-$ pre-training | 43.50 | 57.00 | 2.79 | 3.72 | 45.10
$-$ separate encoder-decoder | 55.00 | 64.80 | 2.32 | 2.53 | 15.57
$-$ supervised loss | 47.20 | 53.60 | 3.29 | 2.92 | 27.50
$-$ alignment | 56.10 | 65.20 | 2.36 | 2.07 | 8.54
Table 1: Results of lyric-to-melody and melody-to-lyric generation in
objective evaluation.

###  4.1 Experimental Setup

#### Dataset

Unpaired Lyric and Melody. We use “380,000+ lyrics from
MetroLyrics”333https://www.kaggle.com/gyani95/380000-lyrics-from-metrolyrics
as our unpaired lyrics for pre-training, which contains 362,237 songs. The
lyrics in each song are split into sentences by the line break. For unpaired
melodies, we choose “The Lakh MIDI Dataset” (Raffel
2016)444https://colinraffel.com/projects/lmd. The dataset contains 176,581
MIDI files with complete tracks, and we extract the melody tracks by Midi-
miner555https://github.com/ruiguo-bio/midi-miner. Finally, we get 65,954
melodies as our unpaired data for pre-training. According to the
characteristics of vocal melody, we consider the pitch and duration tokens of
each note as the melody sequence. Each melody is transposed to the scale of C
major or A minor. All the notes are shifted by octave so that the most pitches
of the song fall into one-lined octave (MIDI pitch from 60 to 71). For
unpaired melody MIDI file, we calculate the starting beat and duration of the
note based on the absolute time and the BPM (Beats Per Minute), all the notes
are aligned to 1/16 notes as paired data. We spread the melodies into
sequences of pitch-duration patterns, as melody sequences for our model. For
example, the melody in Figure 1 will be represented as “R, 7/16, G3, 1/16, E4,
1/8 …”. During pre-training, we simply split the unpaired melodies into
phrases according to the average phrase length in paired data, since there is
no natural phrase segmentation symbol in the MIDI files. Paired Lyric and
Melody. We use the LMD dataset (Yu and Canales
2019)666https://github.com/yy1lab/Lyrics-Conditioned-Neural-Melody-Generation
which contains aligned melodies and lyrics from 7,998 songs. We apply the same
operation, as aforementioned, to process melody and lyric data. The
lyrics/melodies are split into sentences/phrases based on the annotations.

#### Model Configuration and Training

We choose Transformer (Vaswani et al. 2017) as our basic model structure,
which consists of 6 encoder/decoder layers. The hidden size and filter size of
each layer are set as 512 and 2048. The number of attention heads is 8. We use
the same masking strategy as in Song et al. (2019). We use Adam optimizer
(Kingma and Ba 2015) with a learning rate of 5e-4. The model is trained on a
NVIDIA Tesla T4 GPU card, and each mini-batch contains 4096 tokens. During
training, we apply dropout with the rate of 0.1. The hyper-parameter $\alpha$
is set as 0.5. The dataset is split as training/valid/test set with a ratio of
8:1:1. Our baseline is a standard Transformer model, using the same model
configuration with SongMASS but without any pre-training or alignment
constraints.

###  4.2 Evaluation Metrics

In this subsection, we introduce the objective and subjective metrics used in
this paper to evaluate the quality of lyric-to-melody and melody-to-lyric
generation.

#### Objective Evaluation

We mainly measure the similarity between the generated melody and ground-truth
melody in lyric-to-melody generation, in terms of pitch and duration
distribution and melody sequence, which are described below. We use perplexity
(PPL) to measure the model fitness for both lyric-to-melody and melody-to-
lyric generations. Besides, we also use alignment accuracy to measure
alignment quality in two generation tasks, which is also described below.

  * •

PD and DD (Pitch and Duration Distribution Similarity): We calculate the
distribution (frequency histogram) of pitches and durations in melodies, and
measure the similarity (average overlapped area (Ren et al. 2020)) of the
distribution between generated melodies and ground-truth melodies:
$\frac{1}{N_{\text{s}}}\sum_{i=1}^{N_{\text{s}}}OA(\text{Dis}_{i},\hat{\text{Dis}}_{i})$,
where $\text{Dis}_{i}$ and $\hat{\text{Dis}}_{i}$ represent the pitch or
duration distribution of the $i$-th generated and ground-truth song
respectively, $N_{s}$ is the number of songs in the testset, OA represents the
average overlapped area.

  * •

MD (Melody Distance): To evaluate the pitch trend of the melody, we spread out
the notes into a time series of pitch according to the duration, with a
granularity of 1/16 note. We subtract each pitch with the average pitch of the
entire sequence for normalization. To measure the similarity between the
generated and ground-truth time series with different lengths, we use dynamic
time warping (Berndt and Clifford 1994) to measure their distance.

  * •

Alignment Accuracy: To evaluate the alignment between melodies and lyrics, for
each token in the source sequence, we calculate how many tokens in the target
sequence (generated or ground-truth) are aligned to it, and check if the
number of the tokens in the generated sequence equals to that in the ground-
truth sequence. We calculate the ratio of equals among all source tokens and
all songs in the test set to obtain the alignment accuracy.

(a)  Left: without sentence-level constraints. Right: with sentence-level
constraints.

(b)  Left: without token-level constraints. Right: with token-level
constraints.

Figure 6:  Attention visualization. All of the results are displayed on the
average attention score of all heads in the last layer of the encoder-decoder
attention in Transformer. In Figure 6(b), the red blocks are the alignments
searched by our dynamic programming algorithm while the yellow blocks are by
the greedy algorithm described in the second paragraph in Section 4.4.

#### Subjective Evaluation

For subjective evaluation, we invite 5 participants with professional
knowledge in music and singing as human annotators to evaluate 10 songs (338
pairs of generated lyric sentences and melody phrases) randomly selected from
our test set. We require each annotator to answer some questions using a five-
point scale, from 1 (Poor) to 5 (Perfect). The whole evaluation is conducted
in a blind-review mode. Inspired by Watanabe et al. (2018), the metrics to
evaluate the generated lyrics are as follows: 1) Listenability: Is the lyric
sounds natural with the melody? 2) Grammaticality: It the lyric grammatically
correct? 3) Meaning: Is the lyric meaningful? 4) Quality: What is the overall
quality of the lyric? The metrics to evaluate the melody are as follows: 1)
Emotion (Bao et al. 2019): Does the melody represent the emotion of the
lyrics? 2) Rhythm (Zhu et al. 2018): Are the note durations and pauses of the
melody sound natural? 3) Quality (Watanabe et al. 2018): What is the overall
quality of the melody?

Metric | Baseline | SongMASS
---|---|---
_Lyric_ |  |
Listenability | 1.67 $\pm$ 0.62 | 2.00 $\pm$ 0.65
Grammaticality | 3.00 $\pm$ 0.76 | 3.27 $\pm$ 0.59
Meaning | 2.20 $\pm$ 0.68 | 3.20 $\pm$ 0.68
Quality | 2.27 $\pm$ 0.46 | 3.00 $\pm$ 0.38
_Melody_ |  |
Emotion | 2.40 $\pm$ 1.06 | 3.53 $\pm$ 0.64
Rhythm | 2.33 $\pm$ 1.18 | 2.87 $\pm$ 0.74
Quality | 2.33 $\pm$ 1.05 | 2.93 $\pm$ 0.70
Table 2:  Subjective evaluation results. Average scores and standard
deviations are shown for each measure.

###  4.3 Results

The main results of the objective evaluation of lyric-to-melody and melody-to-
lyric generations are shown in Table 1. The baseline model uses the same model
structure with SongMASS, but does not leverage unsupervised melody and lyric
data for pre-training and does not leverage attention-based alignment
constraints. It can be seen that SongMASS greatly outperforms the baseline
model in all objective metrics. The subjective evaluations are shown in Table
2, from which we can see that the lyrics and melodies generated by SongMASS
obtain better average scores in all subjective metrics. These results
demonstrate the effectiveness of SongMASS in generating high-quality lyric and
melody777Melody and lyric samples are available at:
https://musicgeneration.github.io/SongMASS/. We further conduct ablation study
to verify the effectiveness of pre-training and alignment constraint in
SongMASS. As shown in Table 1, removing each component results in worse
performance than SongMASS888Removing alignment constraint causes slightly
better performance in PPL, which indicates that attention constraint may harm
the fitting capability of the model, but still result in better generation
accuracy in terms of PD, DD and MD. We also demonstrate in Table 3 that
alignment constraint indeed improves the alignment accuracy of the generated
results., demonstrating the contribution of pre-training and alignment
constraint.

###  4.4 Method Analysis

#### Pre-training Method

We further investigate the effectiveness of each design in pre-training
method, including using separate encoder-decoder for lyric-to-lyric and
melody-to-melody pre-training and using supervised pre-training to learn a
shared latent space between lyric and melody. From Table 1, removing separate
encoder-decoder (i.e., using shared encoder-decoder) and removing supervised
loss both result in worse performance than SongMASS, which demonstrates the
effectiveness of the two designs.

| L2M Acc $\uparrow$ | M2L Acc $\uparrow$
---|---|---
SongMASS | 62.6 | 45.4
\- TC | 62.1 | 44.8
\- SC | 56.2 | 44.0
\- TC - SC | 55.3 | 43.8
\- TC - SC - PT | 48.3 | 37.1
\- DP | 15.7 | 11.3
Table 3: Analyses of the designs in alignment constraints.

#### Alignment Strategy

We study the effectiveness of the sentence-level and token-level alignment
constraints (denoted as SC and TC respectively) on the alignment accuracy
(denoted in Section 4.2) between melodies and lyrics. The results are shown in
Table 3. It can be seen that both token-level and sentence-level (especially
sentence-level) constraints can improve alignment accuracy. It is interesting
that pre-training (PT) also benefits alignment, which is probably because the
patterns of lyrics and melodies are better captured with pre-training.
Finally, we investigate the alignment accuracy without dynamic programming
(DP) algorithm. In this case, we implement a naive alignment algorithm on
attention weight matrix, which greedily decides to add another token to the
current one-to-many or many-to-one alignment or to start a new alignment pair
at each time step. When the sequence reaches the last token, we align all the
remaining tokens of the other sequence to that token to ensure all tokens are
aligned. We find that the alignment accuracy is drastically decreased without
DP in Table 3, showing the importance of DP for accurate alignments.

#### Alignment Visualization

To better highlight the advantages of our alignment strategy, we further
visualize some cases from the lyric-to-melody tasks, as shown in Figure 6.
Figure 6(a) shows the attention weights of the whole song with and without
sentence-level alignment constraints. It can be seen that the attention
weights without sentence-level constrains are dispersed in all positions of
the whole long sequence, and the target token cannot attend to the correct
source sentences. When using sentence-level constraints, there are monotonous
alignments between source and target sequence, which demonstrates the
effectiveness of sentence-level alignments. Figure 6(b) shows the differences
of whether using token-level constraints or not. We find that the attention
distributions without the token-level constraints are chaotic. When applying
token-level attention constraints, there are obvious diagonal trend in the
attention weights, which further enable the dynamic programming algorithm to
find a better alignment path as marked in red rectangles. These results
demonstrate the effectiveness of token-level alignment constraints.

##  5 Conclusion

In this paper, we have proposed SongMASS, an automatic song writing system for
both lyric-to-melody and melody-to-lyric generation, which leverages masked
sequence to sequence pre-training and attention-based alignment constraint. We
introduce some specific designs based on MASS for lyric-to-lyric and melody-
to-melody pre-training, including song-level unsupervised pre-training and
supervised pre-training loss to learn a shared latent space between lyric and
melody. Furthermore, we introduce the sentence-level and token-level alignment
constraints, and a dynamic programming algorithm to obtain accurate alignments
between lyric and melody. Experimental results show that our proposed SongMASS
greatly improves the quality of lyric-to-melody and melody-to-lyric generation
compared with the baseline. For future work, we will investigate other
sequence to sequence pre-training methods and more advanced alignment
algorithms for lyric-to-melody and melody-to-lyric generation.

## Acknowledgments

This research was supported by the National Key Research And Development
Program of China (No.2019YFB1405802).

## References

  * Bahdanau, Cho, and Bengio (2014) Bahdanau, D.; Cho, K.; and Bengio, Y. 2014.  Neural Machine Translation by Jointly Learning to Align and Translate.  _arXiv e-prints_ abs/1409.0473.
  * Bao et al. (2019) Bao, H.; Huang, S.; Wei, F.; Cui, L.; Wu, Y.; Tan, C.; Piao, S.; and Zhou, M. 2019\.  Neural Melody Composition from Lyrics.  In _NLPCC_ , volume 11838, 499–511.
  * Berndt and Clifford (1994) Berndt, D. J.; and Clifford, J. 1994.  Using dynamic time warping to find patterns in time series.  In _KDD workshop_ , volume 10, 359–370. Seattle, WA, USA:.
  * Choi, Fazekas, and Sandler (2016) Choi, K.; Fazekas, G.; and Sandler, M. B. 2016.  Text-based LSTM networks for Automatic Music Composition.  _CoRR_ abs/1604.05358.
  * Devlin et al. (2019) Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _NAACL_ , 4171–4186.
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015.  Adam: A Method for Stochastic Optimization.  In Bengio, Y.; and LeCun, Y., eds., _ICLR_.
  * Lee, Fang, and Ma (2019) Lee, H.-P.; Fang, J.-S.; and Ma, W.-Y. 2019.  iComposer: An Automatic Songwriting System for Chinese Popular Music.  In _NAACL_ , 84–88.
  * Lewis et al. (2019) Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ .
  * Li et al. (2020) Li, P.; Zhang, H.; Liu, X.; and Shi, S. 2020.  Rigid Formats Controlled Text Generation.  In _ACL_ , 742–751.
  * Lu et al. (2019) Lu, X.; Wang, J.; Zhuang, B.; Wang, S.; and Xiao, J. 2019.  A Syllable-Structured, Contextually-Based Conditionally Generation of Chinese Lyrics.  In _PRICAI_ , volume 11672, 257–265.
  * Luong, Pham, and Manning (2015) Luong, T.; Pham, H.; and Manning, C. D. 2015.  Effective Approaches to Attention-based Neural Machine Translation.  In _EMNLP_ , 1412–1421.
  * Malmi et al. (2015) Malmi, E.; Takala, P.; Toivonen, H.; Raiko, T.; and Gionis, A. 2015.  DopeLearning: A Computational Approach to Rap Lyrics Generation.  _CoRR_ abs/1505.04771.
  * Needleman and Wunsch (1970) Needleman, S. B.; and Wunsch, C. D. 1970.  A general method applicable to the search for similarities in the amino acid sequence of two proteins.  _J. Mol. Biol._ 48: 443–453.
  * Radford et al. (2018) Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018.  Improving language understanding with unsupervised learning.  _Technical report, OpenAI_ .
  * Raffel (2016) Raffel, C. 2016.  Learning-Based Methods for Comparing Sequences, with Applications to Audio-to-MIDI Alignment and Matching.  _PhD Thesis_ .
  * Raffel et al. (2019) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ .
  * Ren et al. (2020) Ren, Y.; He, J.; Tan, X.; Qin, T.; Zhao, Z.; and Liu, T. 2020.  PopMAG: Pop Music Accompaniment Generation.  _CoRR_ abs/2008.07703.  URL https://arxiv.org/abs/2008.07703.
  * Rush, Chopra, and Weston (2015) Rush, A. M.; Chopra, S.; and Weston, J. 2015.  A Neural Attention Model for Abstractive Sentence Summarization.  In _EMNLP_ , 379–389.
  * Song et al. (2019) Song, K.; Tan, X.; Qin, T.; Lu, J.; and Liu, T. 2019.  MASS: Masked Sequence to Sequence Pre-training for Language Generation.  In _ICML_ , volume 97, 5926–5936.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention is All you Need.  In _NIPS_ , 5998–6008.
  * Watanabe et al. (2018) Watanabe, K.; Matsubayashi, Y.; Fukayama, S.; Goto, M.; Inui, K.; and Nakano, T. 2018.  A Melody-Conditioned Lyrics Language Model.  In _NAACL_ , 163–172.
  * Yang et al. (2019) Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R. R.; and Le, Q. V. 2019\.  XLNet: Generalized Autoregressive Pretraining for Language Understanding.  In _NIPS_ , 5753–5763.
  * Yu and Canales (2019) Yu, Y.; and Canales, S. 2019.  Conditional LSTM-GAN for Melody Generation from Lyrics.  _CoRR_ abs/1908.05551.
  * Zhu et al. (2018) Zhu, H.; Liu, Q.; Yuan, N. J.; Qin, C.; Li, J.; Zhang, K.; Zhou, G.; Wei, F.; Xu, Y.; and Chen, E. 2018.  XiaoIce Band: A Melody and Arrangement Generation Framework for Pop Music.  In _KDD_ , 2837–2846.

## Appendix

### Visualization Examples

To demonstrate the advantages of our methods in alignment, we further randomly
choose some cases for attention visualization. The results are shown Figure 7.
From Figure 7, we observe obvious monotonous alignments in each case, and the
dynamic programming algorithm achieves more precise alignments than greedy
alignments.

Figure 7: Attention visualization. The meaning of red and yellow blocks are
same as Figure 7.


mode = titleTensor Decomposition Based Attention Module for Spiking Neural
Networks

cor

cor [cor]Corresponding authors

# Tensor Decomposition Based Attention Module for Spiking Neural Networks

Haoyu Deng<EMAIL_ADDRESS>Ruijie Zhu<EMAIL_ADDRESS>Xuerui Qiu
<EMAIL_ADDRESS>Yule Duan<EMAIL_ADDRESS>Malu Zhang
<EMAIL_ADDRESS>Liang-jian Deng<EMAIL_ADDRESS>University
of Electronic Science and Technology of China, 611731, China  University of
California, Santa Cruz, 95064, The United States

###### Abstract

[SUMMARY] The attention mechanism has been proven to be an effective way to
improve spiking neural network (SNN). However, based on the fact that the
current SNN input data flow is split into tensors to process on GPUs, none of
the previous works consider the properties of tensors to implement an
attention module. This inspires us to rethink current SNN from the perspective
of tensor-relevant theories. Using tensor decomposition, we design the
projected full attention (PFA) module, which demonstrates excellent results
with linearly growing parameters. Specifically, PFA is composed by the linear
projection of spike tensor (LPST) module and attention map composing (AMC)
module. In LPST, we start by compressing the original spike tensor into three
projected tensors using a single property-preserving strategy with learnable
parameters for each dimension. Then, in AMC, we exploit the inverse procedure
of the tensor decomposition process to combine the three tensors into the
attention map using a so-called connecting factor. To validate the
effectiveness of the proposed PFA module, we integrate it into the widely used
VGG and ResNet architectures for classification tasks. Our method achieves
state-of-the-art performance on both static and dynamic benchmark datasets,
surpassing the existing SNN models with Transformer-based and CNN-based
backbones.

###### keywords:

Spiking neural network Attention mechanism Tensor decomposition Neuromorphic
computing

##  1 Introduction

Spiking neural network (SNN) is currently attracting the interest of academics
due to its lower energy consumption and greater bio-interpretability compared
to traditional artificial neural network (ANN). The brain-inspired computation
model gives it significant potential in processing temporal data. Though
training a deep SNN is still a challenge, recent developments in SNN introduce
backpropagation [3] to relieve this issue to some degree. Moreover, it makes
the extension of ANN modules into SNN possible, such as batch normalization
and residual blocks.

As it is now feasible to incorporate ANN modules into SNNs while maintaining
their inherent power efficiency, we can leverage these modules to augment the
overall performance of SNNs. For instance, Zheng et al. [45] propose
threshold-dependent batch normalization to solve the problems of gradient
vanishing and threshold-input balance. The well-known residual block proposed
by He et al. [14] makes it possible to train an ultra-deep ANN network. To
introduce residual block into the field of SNN, Hu et al. [17] investigate
converting residual networks and propose a new technique to adjust the
parameter weight correspondingly. Furthermore, Fang et al. [11] propose the
SEW-ResNet to solve gradient vanishing problems and achieve identity mapping.
In addition, Hu et al. [18] design several novel residual blocks and test
their applicability, greatly extending the depth of a directly trained SNN. In
summary, several strategies in ANN, such as normalization, architecture, and
training methods, could favor the performance of SNN effectively, which also
motivates us to explore more useful techniques in ANNs for better-conducting
SNNs.

Figure 1: Accuracy on CIFAR10 (left) and CIFAR100 (right). Compared with other
methods, PFA significantly improves the network performance.

As one of the most promising techniques in ANNs, the attention mechanism can
mimic the human ability to focus on certain things while ignoring others.
Recently, three representative works have proven that the attention mechanism
can successfully work in SNNs. Especially, Yao et al. [42] switch channel
attention to the temporal dimension, revealing that paying attention to the
temporal dimension is effective for performance enhancement. Zhu et al. [48]
apply two 1D convolutions to a generated 2D tensor to achieve temporal-channel
attention. Inspired by experimental observations on predictive attentional
remapping, Cai et al. [4] design the SCTFA module to assess the input tensor
from temporal, channel, and spatial dimensions, achieving attention among
these three dimensions.

Current software frameworks, i.e. PyTorch, and tensorflow, are boosted by GPU
using parallel computation. This makes it hard to realize event-driven as
events occur asynchronously. Based on these frameworks, as a trade-off, a
popular approach is splitting the input event streams into slices and
composing tensors to convenient later computation. This approach is adopted by
all attention mechanism-related works, showing a promising outcome. Since we
can now only efficiently process event-converted tensors, it is natural for us
to consider involving tensor analysis theories to implement an attention
module for SNN, which is ignored by previous work. Simultaneously, this
analysis highlights the inherent resilience within SNN modules. It
demonstrates that even when employing low-rank execution for Attention
mechanisms, substantial performance enhancements can be achieved. This aligns
perfectly with the sparsity philosophy of SNNs.

In this paper, we propose the projected-full attention (PFA), which considers
projecting one input tensor into three separate lower-dimensional tensors,
then calculate a full-attention map by utilizing the inverse procedure of the
tensor decomposition, fully exploiting the peculiar temporal dimension in SNN
with minimal computing complexity. The input tensor is first projected along
three separate dimensions by the linear projection of spike tensor (LPST)
module. Then, the projections are composed into an attention map by the
attention map composing (AMC) module. The input tensor is fused with the
attention map by the Hadamard product to obtain the refined tensor. The main
contributions of this paper can be summarized as follows:

  * •

We propose the projected full-attention (PFA) module to achieve three-
dimensional attention. Compared with previous works, PFA is a lightweight
module with fewer learnable parameters, and the number of parameters grows
linearly as the size of input data increases. PFA is founded on a well-
established mathematical tool, thus enhancing its reliability and mathematical
interpretability.

  * •

We present a comprehensive theoretical analysis that encompasses two key
aspects: the rank of tensors and connecting factor. Based on these analyses,
we propose a selection criterion for that is tailored for distinct datasets.
This criterion effectively elucidates the experimental outcomes and offers
valuable guidance for future applications.

  * •

We conduct extensive experiments on both static and dynamic datasets.
Experimental results show that our PFA is very effective and achieves state-
of-the-art (SOTA) accuracy on both dynamic and static datasets. In addition,
ablation studies demonstrate the structural reasonableness of PFA. We
visualize the attention map obtained by PFA to intuitively illustrate the
attention distribution among temporal-channel-spatial dimensions.

##  2 Related Works and Motivation

Spiking Neural Network: Spiking neural network (SNN), the third generation of
artificial neural network [29], has the potential to behave as efficiently as
the human brain. In an SNN, information is transmitted through spikes, which
are binary events that represent the firing of a neuron. Unlike traditional
ANNs, which use continuous signals, SNN neurons communicate over time using
discrete spikes. Compared with ANN, SNN is more biologically realistic by
adopting models from neural dynamics and is more hardware-friendly thanks to
the discrete spike. Due to its temporal property, it is supposed to have
better temporal expressiveness than ANN. Quantities of studies have already
been conducted. For neurodynamic models, Hodgkin and Huxley proposed the
Hodgkin-Huxley (H-H) model [15] in 1952. Models like Izhikevich [19] and Leaky
Integrate and Fire (LIF) [25] are suggested to strike a balance between
computing complexity and bio-interpretability. Since the spiking behavior is
non-differentiable, training an SNN is challenging. Several methods have been
explored to address this difficulty. Some techniques [38; 39; 44] are
gradient-based optimization. Others, like Rueckauer et al. [36] and Diehl et
al. [9], are based on converting an ANN to SNN. There also exists ANN-SNN
collaborative networks [41]. Besides the conventional datasets (static
datasets) such as MNIST [7], and CIFAR10 [23], SNNs also support datasets
collected from event flow cameras (dynamic datasets) like NCaltech-101 [32].
The concept behind converted datasets, of which ES-ImageNet [28], is an
example to transform ordinary datasets into neural forms through specific
algorithms. Another way to convert ordinary data is to recapture it using
dynamic vision sensor (DVS) cameras. NCaltech-101 [32] and CIFAR10DVS [26] are
in this category. As aforementioned, SNNs are hardware-friendly, making
hardware implementation one of the frontier areas. Related developments
include TrueNorth [2], Loihi [6], and Tianjic [33] chips. Nonetheless, there
are still many obstacles in applying SNNs on various tasks, especially in its
performance. In this paper, we successfully increases the classification
accuracy of SNNs with the proposed PFA module.

Attention Mechanism in SNN: Since Google proposed recurrent models of visual
attention in 2014 [31], the attention mechanism become a potent tool to
increase the performance of a neural network, which gives distinct parts of
the input data varied weights. For instance, the squeeze-and-excitation
network (SENet) [16] and other attention blocks have been frequently used in
traditional ANNs. From this fact, it is of vital significance to introduce the
attention mechanism into SNNs. To achieve temporal-wise attention, inspired by
SENet, TA-SNN [42] is proposed. Besides, TCJA [48] use two 1D convolutions to
pay attention to both channel dimension and temporal dimension to prolong the
attention mechanism. Nevertheless, they fail to considered three crucial
aspects of SNN, _i.e.,_ temporal, channel, and spatial. By changing SENet and
leaving affection in the temporal direction, Cai et al. [4] and Yao et al.
[43] successfully apply temporal-channel-spatial attention to SNN. These
researches reveal the fact that attention mechanism does improve the
performance of SNN.

Motivation: As aforementioned, the application of the attention mechanism has
demonstrated promising ability in SNN-related tasks. Regardless of the
dimension(s) considered, the attention mechanism is always beneficial [42; 48;
4; 43]. Especially, the temporal-channel-spatial attention could faithfully
depict the relationship of temporal, channel and spatial dimensions, thus it
becomes a quite promising approach in SNNs. For this type of attention,
although 3D convolution is used to accomplish a comparable effect [5], the
processing complexity is intolerable. Besides, two other excellent works,
_i.e.,_ Cai et al. [4] and Yao et al. [43], have fully considered the
temporal-channel-spatial attention and gain exciting outcome. Nonetheless, the
above approach does not take into account the nature of the input data as a
tensor, which is unavoidable since the training process is accelerated by GPU
using parallel computation. In this case, we naturally hope to discover
properties that we can use from the relevant theories of tensors to implement
the attention mechanism. Drawing inspiration from the tensor CANDECOMP/PARAFAC
(CP) decomposition [22], the inverse process of the tensor CP decomposition
can be used to calculate the attention map. Therefore, we propose the PFA,
which mainly comprises two distinct components, _i.e._ , LPST for the tensor
compression and AMC for the attention calculation by tensor fusion. The goal
of LPST is to learn parameters that help project the most important
information into three 2D tensors. The AMC part uses these three projections
to construct an attention map through the inverse process of CP decomposition.

##  3 Method

In this part, we first introduce the neuron model and the training techniques
we adopt. Next, we introduce PFA and its two components, _i.e._ LPST and AMC,
in detail. Finally, a quantitative theoretical analysis on the adoption of
connecting factor $r$ is presented.

###  3.1 Leaky Integrate and Fire Model

In this work, we employ the Leaky Integrate and Fire (LIF) neuron due to its
efficiency and simplicity. The LIF neuron accumulates external input in its
membrane voltage, which can be described as follows:

$\tau\frac{dV(t)}{dt}=-(V(t)-V_{reset})+I(t),$ (1)

where $\tau$ denotes the time constant, $V(t)$ is the membrane potential at
the moment $t$, and $I(t)$ is presynaptic input at $t$ moment. With this
formula, the entire model is summarized as:

$\begin{cases}V_{t}^{n}=H_{t-1}^{n}+\frac{1}{\tau}[I_{t-1}^{n}-(H_{t-1}^{n}-V_{reset})]&\\\
S_{t}^{n}=\Theta(V_{t}^{n}-V_{thr})\\\
H_{t}^{n}=V_{t}^{n}\cdot(1-S_{t}^{n}),\end{cases}$ (2)

where $n$ and $t$ are the layer and moment indices respectively. Besides,
$\tau$ represents the time constant, $V$ is membrane potential, $S$ is the
spike tensor in binary form, $I$ is the input from the previous layer,
$\Theta$ stands for Heaviside step function, $H$ represents the reset behavior
after spiking, and $V_{thr}$ means the threshold membrane potential. The LIF
model has relatively lower computation complexity and higher representational
ability, which now has become a mainstream model in the field of SNN. Here, we
use the spatial-temporal backpropagation technique to train our network,
aiming to fully make use of the information from spatial and temporal domains.
Particularly, the backpropagation on the spatial and temporal domains can be
described as follows:

$\begin{cases}\frac{\partial Loss}{\partial
S_{i}^{t,l}}=\sum\limits^{n^{l+1}}_{j=1}\frac{\partial Loss}{\partial
S_{j}^{t,l+1}}\frac{\partial S_{j}^{t,l+1}}{\partial
S_{i}^{t,l+1}}+\frac{\partial Loss}{\partial S_{j}^{t+1,l}}\frac{\partial
S_{j}^{t+1,l}}{\partial S_{i}^{t,l}}\\\ \frac{\partial Loss}{\partial
V_{i}^{t,l}}=\frac{\partial Loss}{\partial S_{i}^{t,l}}\frac{\partial
S_{i}^{t,l}}{\partial V_{i}^{t,l}}+\frac{\partial Loss}{\partial
S_{i}^{t+1,l}}\frac{\partial S_{i}^{t+1,l}}{\partial V_{i}^{t,l}}.\end{cases}$
(3)

Note that the Heaviside function in (2) is non-differentiable in
backpropagation. To address the issue, we employ the following approximation
of the Heaviside function in backpropagation for easy calculation:

$ATan(V_{i})=\frac{1}{\pi}arctan[\frac{\pi}{2}\alpha(V_{i}-V_{thr})]+\frac{1}{2},$
(4)

where $V_{i}$ denotes the neuron membrane potential, $V_{thr}$ is the reset
membrane potential and $\alpha$ is an adjustment factor to adjust the slope.

###  3.2 Projected Full-Attention (PFA)

Figure 2: A simple comparison curve of parameter quantity growth among TCJA
[48], TA-SNN [42], and PFA. The parameter scale of PFA increases linearly. In
this figure, the channel number is fixed at 128. Figure 3: The detailed
workflow of PFA. The input tensor is first sent to Linear Projection of Spike
Tensor (LPST) unit to generate three projections and split the three
projections into corresponding vectors. In Attention Map Composing (AMC) unit,
these vectors are composed into the final attention map through the inverse
process of CP decomposition. The attention is fused with the input tensor to
obtain the refined tensor by Hadamard product.

In this section, we first introduce the overall idea of PFA. Then we present
the details of two crucial components of PFA, _i.e.,_ linear projection of
spike tensor (LPST) and attention map composing (AMC).

The core of the attention mechanism is to give essential parts of the input
data higher weights while ignoring less-related parts to some degree. This is
particularly important for SNNs, especially for data collected by DVS cameras.
DVS camera data often has more noise, which greatly affects network
performance. Thanks to the feature of CP-decomposition, PFA can effectively
suppress noise. In our proposed PFA, we first design the LPST module to
project the higher-dimensional tensor (generally 3D) to three lower-
dimensional tensors, _i.e.,_ 2D matrices. By using learnable parameters, LPST
can selectively reserve essential information while ignoring irrelevant one.
In AMC, we split the matrices into $r$ (called connecting factor here)
vectors. Three vectors from three matrices of temporal, channel and spatial
dimensions respectively are used to conduct the outer product to compose a
part of the final attention map. The final attention map is the sum of these
$r$ parts. We fuse this attention map with the origin input to obtain the
refined tensor. For an input tensor with the shape of $H\times W\times C\times
T$, the overall parameter amount is $C\times r+T\times r+k^{2}\times T\times
r$, where $k$ is the convolution kernel size. Note that $r$ is limited to a
small fixed value according to our theoretical analysis in Section 3.3, the
parameter size grows linearly with $T$ and $C$. Figure 2 shows the parameter
growth curve of PFA and compares several other techniques. The workflow of PFA
is plotted in Figure 3. In what follows, we will give the details of LPST and
AMC.

####  3.2.1 Linear Projection of Spike Tensor (LPST)

Tensor decomposition is a vital approach to represent a tensor and its latent
features. Therefore it is natural for us to utilize tensor decomposition
related techniques to explore the data characteristics. A typical technique is
the CANDECOMP/PARAFAC (CP) decomposition. Whatever way we adopt, it is
essential to generate a “projection” to extract significant components of the
input tensor to yield an attention map later.

In this part, we need to construct three linear projections of the input data
stream along spatial, channel, and temporal dimensions. In LPST, we apply two
fully connected (FC) layers and a convolution layer to project the input
tensor for simplicity. Note that here we simply use single-layer FCs and a
convolution layer. What we do here is just a simple projection with learnable
parameters rather than complex function fitting. In this way, we not only
retain the required information, but also reduce the number of parameters.

For temporal dimension, we first squeeze the input tensor ($H\times W\times
C\times T$) in the following way:

$Y^{T}_{(c,t)}=\frac{1}{HW}\sum\limits_{i=1}^{H}\sum\limits_{j=1}^{W}X_{(i,j,c,t)},$
(5)

where $Y^{T}_{(c,t)}$ denotes the output matrix, $H$ and $W$ denote the
spatial sizes, $X_{(i,j,c,t)}$ denotes the input tensor. After the squeeze
operation, the tensor is converted to a matrix in the shape of $C\times T$.
Then, we apply an FC layer to reserve $r$ necessary information in each
column. A sigmoid activation function is applied after the FC layer. Finally,
we obtain a projection matrix in the shape of $r\times T$.

For channel dimension, the operation is similar to that of temporal dimension.
We squeeze the input tensor by

$Y^{C}_{(c,t)}=\frac{1}{HW}\sum\limits_{i=1}^{H}\sum\limits_{j=1}^{W}X_{(i,j,c,t)},$
(6)

where $Y^{C}_{(c,t)}$ denotes the output matrix, $C$ denotes the number of
channel. Then, we pass the $T\times C$ matrix through an FC layer to acquire
the projected matrix in the shape of $r\times T$ with the sigmoid activation
function applied.

For spatial dimension, the channel is squeezed by,

$Y^{S}_{(x,y,t)}=\frac{1}{C}\sum\limits_{k=1}^{C}X_{(x,y,k,t)}.$ (7)

Afterward, we employ a convolution to the squeezed tensor, treating $T$ as the
input channel and $r$ as the output channel, and then reshape the output
tensor from the convolution into a matrix in the shape of $HW\times r$. The
sigmoid activation function is applied.

####  3.2.2 Attention Map Composing (AMC)

Once we obtain the three projections corresponding to three dimensions, a
necessary step is to compose an attention map using the three projections.
Here, we adopt the reverse process of the tensor CP decomposition. In the CP
decomposition, a tensor $\mathcal{X}$ can be represented as:

$\mathcal{X}\approx\sum_{i=1}^{r}v_{si}\circ v_{ci}\circ v_{ti},$ (8)

where $v_{si}$, $v_{ci}$ and $v_{ti}$ denote the $i$-th vectors in the
spatial, channel and temporal projection tensors respectively, and $\circ$ is
the outer product of vectors. Especially, $r$ is the connecting factor, which
can be viewed as the rank concept in the tensor CP decomposition (this process
is plotted in Figure 3). It is worth noting that the tensor CP decomposition
is just an approximate representation of tensors, which perfectly meets our
demands. The accuracy of our representation of the input tensor can be
controlled by r. That is the larger $r$ is, the more accurate the
representation is, and vice versa, which perfectly realizes the attention
mechanism we need. In the experiment part, we discuss the choice of connecting
factor $r$ and its effect in details.

###  3.3 Quantitative Theoretical Analysis

Figure 4: A toy example of the effect of different ranks on the approximation
outcome. error is measured under $\ell_{2}$ norm. The norm starts to cover
when the rank exceeds 30.

In this section, we first introduce an approach to estimate the rank of the
input spike tensor. Following such analysis, we propose a principle to select
the proper $r$ value, which is supported by our experiments.

To better understand the feature of PFA and facilitate later application, we
follow the work by Acar et al. [1] to investigate the rank of the input data.
Given that computing the CP decomposition of a tensor is recognized as an NP-
hard problem [13; 22], direct calculation from the tensor becomes infeasible.
Therefore, to address this challenge, we adopt a gradient descent technique to
approximate the original tensor. We manually set the rank to multiple values,
and consider the value with the best approximation as the rank of the tensor.
To be more specific, an element of the tensor can be presented as:

$x_{ijk}\approx\sum\limits_{r=1}^{R}a_{i,r}b_{j,r}c_{k,r}.$ (9)

We treat it as an optimization problem, in which $a_{i,r},b_{j,r},c_{k,r}$ are
considered as parameters. Then we optimize the $Loss$ in the following
expression using the gradient descent approach:

$Loss=\frac{1}{2}\sum_{(i,j,k)\in\Omega}(x_{i,j,k}-\sum\limits_{r=1}^{R}a_{i,r}b_{j,r}c_{k,r})^{2}.$
(10)

Find the first order partial derivative of $Loss$ about the parameter
($a_{i,r}$ as an example here), namely:

$\frac{\partial Loss}{\partial a_{i,r}}=-\frac{1}{2}\cdot
2\cdot(x_{i,j,k}-\sum\limits_{r=1}^{R}a_{i,r}b_{j,r}c_{k,r})\cdot
b_{j,r}c_{k,r}.$ (11)

Lastly, apply the gradient descent method to execute optimization in the
following way:

$a_{i,r}^{\prime}\leftarrow a_{i,r}-\mu\cdot\frac{\partial Loss}{\partial
a_{i,r}},$ (12)

where $\mu$ denotes the step size of the gradient descent. We take a piece of
data from the dataset CIFAR10DVS as a toy example and discover its rank. In
this case, we adopt $\mu=0.0001$ and iterate the process 1000 times to get the
final optimized parameters. We utilize $\ell_{2}$ norm as the metric to judge
the approximation effect. As shown in Figure 4, the error reduces as the rank
increases, and starts to converge when the rank exceeds 30. This toy example
shows that the rank of the input tensor is around 30, indicating that we need
a rank exceeding 30 to precisely represent it using the CP decomposition
technique. It should be noted that for the attention mechanism, representing
the original tensor too accurately is not conducive to improving network
performance. The core idea of the attention mechanism is to focus on the
important parts while ignoring the secondaries. Representing raw tensors too
precisely is counterproductive. Therefore, a good choice for $r$ should not be
too large. This is a very useful conclusion. On the one hand, it tells us the
range of finding $r$. On the other hand, it also ensures that PFA can obtain
good results without excessive calculation.

As shown in our experiment (Figure 6), though the rank of the attention map
differs on dynamic datasets like CIFAR10DVS and static datasets like CIFAR100,
the best option for $r$ is not very big. Further observing the experimental
results, we find that when the value of $r$ exceeds a certain threshold
(approximately $\frac{T}{2}$, $T$ is the time step), the larger $r$ is, the
worse the effect will be. The above analysis of the experimental results
proves the correctness of our conclusion. Considering the above analysis, we
propose a principle for selecting $r$: For dynamic datasets, the search for
the optimal value of $r$ is performed within the range of no more than $T$.
Specifically, we focus on identifying the best $r$ value around $\frac{T}{2}$,
considering the varying content across different time steps. In the case of
static datasets, the search for the optimal $r$ value begins from 1 and
extends until the best value is determined. Since static datasets involve
duplicated frames, this search aims to find the most suitable $r$ value for
the low-rank condition.

##  4 Experiment

We evaluate the classification performance of our PFA on two static datasets:
CIFAR10 [24], CIFAR100 [24] and two dynamic datasets: CIFAR10DVS [26],
NCaltech-101 [32]. Details of the datasets, network architecture, data
augmentation, loss function, and pre-process procedure is introduced in this
section. Extensive experiment results are also presented.

Dataset | Learning Rate | Epoch | Batch Size
---|---|---|---
CIFAR10 | 0.1 | 200 | 128
CIFAR100 | 0.1 | 200 | 64
CIFAR10DVS | 0.0001 | 200 | 32
NCaltech-101 | 0.0001 | 200 | 32
Table 1: Hyper-parameter settings of PFA.

###  4.1 Datasets and Training Details

####  4.1.1 Datasets

We have conducted experiments on both static datasets and dynamic datasets for
object classification. We train and test PFA on a workstation equipped with
one RTX 3090. For different datasets, the hyper-parameters are listed in Table
1. The summaries of datasets and augmentation involved in the experiment are
listed below.

CIFAR 10/100 consist of 50k training images and 10k testing images with the
size of 32 × 32 [24]. We use ResNet-19 for both CIFAR10 and CIFAR100. Random
horizontal flips and crops are applied to the training images for
augmentation. Moreover, Cutout is also used for augmentation which is the same
as the Ref [8]
CIFAR10-DVS converts 10,000 frame-based images of 1010 classes into event
streams with the dynamic vision sensor. Since the CIFAR10DVS dataset [26] does
not divide training and testing sets, we split the dataset into 9k training
images and 1k test images and reduced the spatial resolution from $128\times
128$ to $48\times 48$ [8]. We use VGGSNN for neuromorphic datasets CIFAR10DVS.
In each frame, horizontal flipping and mixup are adopted, where the
probability of Flipping is set to 0.5. Then, we randomly select one
augmentation among Rolling, Rotation, Cutout, and Shear, where the random
Rolling range is 5 pixels, and the degree of Rotation is sampled from the
uniform distribution, which is the same as the Ref [8].
NCALTECH-101 is also converted from the original version of Caltech-101 [32]
with a slight change in object classes to avoid confusion. The NCaltech-101
consists of 100 object classes plus one background class. We apply the 9: 1
train-test split as CIFAR10DVS. And the VGGSNN is applied to neuromorphic
datasets NCALTECH-101. Moreover, the augmentation is identical to CIFAR10DVS.

####  4.1.2 Loss function

We used the cross-entropy loss for the neuromorphic datasets CIFAR10DVS and
NCALTECH-101. And in order to get a better performance in static datasets
(CIFAR 10 / 100), we used a Temporal Efficient Training [8] loss function. It
can be concluded as follows:

$\mathcal{L}_{\mathrm{TET}}=\frac{1-\lambda}{T}\cdot\sum_{t=1}^{T}\mathcal{L}_{\mathrm{CE}}[\boldsymbol{O}(t),\boldsymbol{y}]+\frac{\lambda}{T}\cdot\sum_{t=1}^{T}\mathcal{L}_{\mathrm{MSE}}(\mathbf{O}(t),\phi),$
(13)

where $T$ is the total time steps, $\mathcal{L}_{\mathrm{CE}}$ and
$\mathcal{L}_{\mathrm{MSE}}$ denote the cross-entropy loss and mean square
error. And $\boldsymbol{y}$ represents the target label. Moreover,
$\mathbf{O}(t)$ and $\phi$ are the output and a constant used to regularize
the membrane potential distribution. Additionally, we set $\phi=V_{th}$ in our
experiments. In practice, we use a hyperparameter $\lambda$ to adjust the
proportion of the regular term.

Figure 5: Schematic diagram of the network structure. On the left is the
convolutional layer of the VGG network, and on the right is the convolutional
layer of Resnet.

####  4.1.3 Network Architectures

To evaluate PFA, we integrate it into VGG-type [37] and ResNet [14]
architectures, which are previously utilized in [12; 48; 8; 43]. Figure 5
presents the basic blocks involved in VGG and ResNet architectures. Five PFAs
are inserted into a VGG-type network after each average pooling operation. In
ResNet architecture, PFA is inserted before the residual connection in the
last two layers. When splitting the event flow data into frames, we simply
integrate the input in the time domain. Particularly, our work is based on the
software framework SpikingJelly [10]. Taking notice that frames in CIFAR10-DVS
and NCaltech-101 are entirely occupied by their content, we utilized a pooling
layer as the first layer of the network to reduce memory consumption.

###  4.2 Comparison with Existing SOTA Works

To verify the effectiveness of our PFA, we compare it with several current
best SNNs, including hybrid training and spike-based backpropagation. The
results for the static image data and the dynamic data classification task are
listed in Table 2. The results for the static image dataset are at time steps
2, 4, and 6.

For CIFAR10 and CIFAR100, PFA achieves higher accuracy than previous works. In
particular, our PFA achieves superior performance on CIFAR10 and CIFAR100,
with 1.2% and 4.4% improvements over the TET [8] method, respectively, using
the same network structure and time steps. Moreover, our PFA surpasses the DSR
[30] method while requiring 10$\times$ fewer time steps. Additionally, we
demonstrate that our attention-based SNN outperforms the Spiking-ViT [47]
backbone.

On the dynamic dataset CIFAR10DVS, our method outperforms the previous works
with binary spikes by 3.1%, while using fewer time-steps. For the NCaltech101
dataset, we achieved 80.5% top-1 accuracy, achieving a performance far beyond
previous work [48]. To further illustrate the advantages of our PFA, we show
the comparison with previous works at different time steps. As shown in Figure
1, we compared PFA with Spike-based BP SNNs. Specifically, our PFA shows a
significant advantage at shorter simulation lengths due to its more vital
representation ability.

Dataset Method Type Architecture Time steps Accuracy CIFAR10/CIFAR100 SIDP
[35]ICLR-2019 Hybrid training ResNet-20/VGGSNN 250/125 95.4/76.4 Diet
[34]TNNLS-2021 Spike-based BP ResNet-20 10/5 92.5/64.0 tdBN [46]AAAI-2021
Spike-based BP ResNet-19 6 93.1/71.1 Dspike [27]NeurIPS-2021 Spike-based BP
ResNet-18 6 94.2/74.2 NAS-SNN [21]ECCV-2022 Spike-based BP NAS 5 92.7/73.0 TET
[8]ICLR-2022 Spike-based BP ResNet-19 6 94.5/74.7 DSR [30]CVPR-2022 Spike-
based BP ResNet-19 20 95.4/78.5 Spikformer [47]ICLR-2023 Spike-based BP
Spiking-ViT 4 95.5/78.2 PFA Spike-based BP ResNet-19 6 95.7/79.1 4 95.7/78.1 2
95.6/76.7 CIFAR10DVS LIAF-Net [40]TNNLS-2021 Spike-based BP LIAF-Net 10 70.4
tdBN [46]AAAI-2021 Spike-based BP ResNet-19 10 67.8 TA-SNN [42]ICCV-2021
Spike-based BP LIAF-Net 10 72.0 Dspike [27]NeurIPS-2021 Spike-based BP VGGSNN
10 75.4 TET [8]ICLR-2022 Spike-based BP VGGSNN 10 83.2 DSR [30]CVPR-2022
Spike-based BP VGGSNN 10 77.3 TCJA [48]arXiv-2022 Spike-based BP VGGSNN 10
80.7 Spikformer [47]ICLR-2023 Spike-based BP Spiking-ViT 16 80.9 PFA Spike-
based BP VGGSNN 14 84.0 NCaltech-101 SALT [20]Neural Netw-2021 Spike-based BP
VGGSNN 20 55.0 TCJA [48]arXiv-2022 Spike-based BP VGGSNN 14 78.5 PFA Spike-
based BP VGGSNN 14 80.5

Table 2: Compare with existing works. Our method improves network performance
across all tasks.

###  4.3 Discussion on $r$

CIFAR10

CIFAR100

CIFAR10DVS

NCaltech101

Figure 6: Accuracy on four tested datasets with different $r$. PFA achieves
best performance when $r=3,1,8,8$ correspondingly.

Figure 6 demonstrates the performance with different $r$. The outcome reveals
that for dynamic datasets, the best rank option falls around $\frac{T}{2}$
while for static datasets, the best value of $r$ (also known as connecting
factor) is very small. Dynamic datasets exhibit significant variations in
content across different time steps. Conversely, static datasets involve the
duplication of frames, where each frame is replicated $T$ times as input to
the network. Consequently, this duplication results in the attention map being
subjected to a low-rank condition. Considering a comprehensive assessment of
both experimental outcomes and theoretical insights, our conclusion asserts
that the precision of the attention map in faithfully mirroring the input
tensor is unnecessary. Such fidelity does not inherently facilitate the
effective concentration on pivotal components. Furthermore, our findings
unveil an intriguing phenomenon: even in scenarios where SNNs exhibit
considerable sparsity, the rank of the attention map can still remain markedly
low. This revelation implies the presence of untapped optimization potential
within the realm of SNN sparsity.

###  4.4 Ablation Study

In this section, we conduct ablation experiments from both global and local
perspectives to verify the reasonableness of PFA. We first try adding
different numbers of PFA layers to the network to see how PFA improves
classification accuracy. Then we set the three projections obtained inside the
PFA to a constant matrices with each element assigned to 1 respectively to
eliminate the attention on this dimension to explore the impact of different
dimensions of attention on the classification results.

We insert different numbers of PFAs into the VGG network and obtain the
results in Figure 7 (left). In this experiment, $r$ is fixed at 8. We can
observe that as more and more PFA layers are inserted, the accuracy of the
network is getting higher and higher. Even inserting only one layer can
improve the accuracy of the network.

In order to explore how attention in temporal-channel-spatial dimensions
affects classification accuracy, we conduct an ablation by fixing
corresponding projection matrix obtained by LPST to a all-one tensor. As shown
in Figure 7 (right), whichever dimension is ablated will adversely affect the
results. Ablating temproal and ablating channel have similar effects. This
means that PFA treats temporal and channel dimensions equally. However, when
we ablate the spatial dimension, the accuracy of the network drops
significantly (79.60%) but still outperforms the baseline (77.90%). Whichever
dimension we ablate, the performance drops. This observation highlights that
PFA effectively directs attention to distinct regions across multiple
dimensions, aligning with the intended purpose of attention modules.
Specifically, PFA exhibits attention distribution across all three dimensions,
albeit with varying weights. Notably, the spatial dimension emerges as the
most significant, followed by the time and channel dimensions, in terms of
attention importance.

Figure 7: Results of ablation experiments. The left one is ablation study on
PFA layers. The x-axis represents the amount of PFA inserted. The right one is
the result of ablation study on temporal-channel-spatial ablation. Full
denotes the original PFA without any modification as a reference. w.o. T/C/S
denotes setting the temporal/channel/spatial projection to a constant matrix
with each element assigned to 1.

###  4.5 Attention Visualization

In order to visually represent the attention distribution of the PFA module,
we have generated an attention map in PFA and depicted it in Figure 8. The
attention maps corresponding to the temporal and channel dimensions in the
figure are obtained from the deep layer of the network. This choice was made
because deep layers tend to provide a more accurate reflection of the
attention’s role. Conversely, the spatial attention map is obtained from the
input layer of the network, where the shallow graphic features are more
distinct and conspicuous, facilitating analysis and observation. Figure 8 also
illustrates the distinctive attention distributions in the temporal, channel,
and spatial dimensions achieved by PFA. This finding aligns with the
conclusions drawn from our ablation study, further validating our earlier
assertions.

##  5 Conclusions

In this paper, we introduce the novel approach named PFA, which combines
learning and generation of a temporal-channel-spatial attention map through CP
decomposition. Specifically, we propose the LPST to learn a projection of the
input tensor that can extract essential information for AMC to compose the
attention map. In the AMC, we utilize the reverse process of CP decomposition
to generate the attention map. After analyzing the rank of the raw input
tensor and the best option for connecting factor $r$, we propose a principle
to search for the best $r$. Remarkably, our findings suggest that even a low
rank can lead to outstanding performance. Regardless the form of the dataset,
dynamic or static, PFA achieves the SOTA performance. Additional ablation
studies prove the reasonableness of PFA while visualization of PFA attention
distribution helps us intuitively understand the significance effect of PFA.

Figure 8: Attention visualization on CIFAR10DVS dataset. Temporal and channel
parts are taken from the deep layer of VGGSNN. The spatial layer
visualizations are extracted from the shallow layer of the network due to the
inherent complexity of the information contained in the deeper layers, which
may impede human comprehension and hinder the interpretability of our
analysis.

## References

  * Acar et al. [2011] Evrim Acar, Daniel M. Dunlavy, Tamara G. Kolda, and Morten Mørup.  Scalable tensor factorizations for incomplete data.  _Chemometrics and Intelligent Laboratory Systems_ , 106(1):41–56, March 2011.  10.1016/j.chemolab.2010.08.004.  URL https://doi.org/10.1016/j.chemolab.2010.08.004.
  * Akopyan et al. [2015] Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-Icaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, Brian Taba, Michael Beakes, Bernard Brezzo, Jente B. Kuang, Rajit Manohar, William P. Risk, Bryan Jackson, and Dharmendra S. Modha.  Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , 34(10):1537–1557, 2015.  10.1109/TCAD.2015.2474396.
  * Bohte et al. [2002] Sander M. Bohte, Joost N. Kok, and Han La Poutré.  Error-backpropagation in temporally encoded networks of spiking neurons.  _Neurocomputing_ , 48(1-4):17–37, October 2002.  10.1016/s0925-2312(01)00658-0.  URL https://doi.org/10.1016/s0925-2312(01)00658-0.
  * Cai et al. [2022] Wuque Cai, Hongze Sun, Rui Liu, Yan Cui, Jun Wang, Yang Xia, Dezhong Yao, and Daqing Guo.  A spatial-channel-temporal-fused attention for spiking neural networks, 2022.  URL https://arxiv.org/abs/2209.10837.
  * Chen et al. [2018] Rong Chen, Yuanlong Yu, and ZhiYong Huang.  Event recognition based on 3d convolutional networks.  In _2018 IEEE 8th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER)_ , pages 45–50, 2018.  10.1109/CYBER.2018.8688320.
  * Davies et al. [2018] Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, Yuyun Liao, Chit-Kwan Lin, Andrew Lines, Ruokun Liu, Deepak Mathaikutty, Steven McCoy, Arnab Paul, Jonathan Tse, Guruguhanathan Venkataramanan, Yi-Hsin Weng, Andreas Wild, Yoonseok Yang, and Hong Wang.  Loihi: A neuromorphic manycore processor with on-chip learning.  _IEEE Micro_ , 38(1):82–99, 2018.  10.1109/MM.2018.112130359.
  * Deng [2012] Li Deng.  The mnist database of handwritten digit images for machine learning research [best of the web].  _IEEE Signal Processing Magazine_ , 29(6):141–142, 2012.  10.1109/MSP.2012.2211477.
  * Deng et al. [2021] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu.  Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Diehl et al. [2015] Peter U. Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer.  Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing.  In _2015 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8, 2015.  10.1109/IJCNN.2015.7280696.
  * Fang et al. [2020] Wei Fang, Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Yonghong Tian, and other contributors.  Spikingjelly.  https://github.com/fangwei123456/spikingjelly, 2020.  Accessed: 2023-01-16.
  * Fang et al. [2021a] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian.  Deep residual learning in spiking neural networks.  _Advances in Neural Information Processing Systems_ , 34:21056–21069, 2021a.
  * Fang et al. [2021b] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian.  Incorporating Learnable Membrane Time Constant To Enhance Learning of Spiking Neural Networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 2661–2671, 2021b.
  * Håstad [1990] Johan Håstad.  Tensor rank is NP-complete.  _Journal of Algorithms_ , 11(4):644–654, December 1990.  10.1016/0196-6774(90)90014-6.  URL https://doi.org/10.1016/0196-6774(90)90014-6.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, June 2016.  10.1109/cvpr.2016.90.  URL https://doi.org/10.1109/cvpr.2016.90.
  * Hodgkin and Huxley [1952] A. L. Hodgkin and A. F. Huxley.  A quantitative description of membrane current and its application to conduction and excitation in nerve.  _The Journal of Physiology_ , 117(4):500–544, August 1952.  10.1113/jphysiol.1952.sp004764.  URL https://doi.org/10.1113/jphysiol.1952.sp004764.
  * Hu et al. [2018a] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7132–7141, 2018a.
  * Hu et al. [2018b] Yangfan Hu, Huajin Tang, and Gang Pan.  Spiking deep residual network, 2018b.  URL https://arxiv.org/abs/1805.01352.
  * Hu et al. [2022] Yifan Hu, Yujie Wu, Lei Deng, and Guoqi Li.  Advancing deep residual learning by solving the crux of degradation in spiking neural networks, 2022.  URL https://arxiv.org/abs/2201.07209.
  * Izhikevich [2003] Eugene M Izhikevich.  Simple model of spiking neurons.  _IEEE Transactions on neural networks_ , 14(6):1569–1572, 2003.
  * Kim and Panda [2021] Youngeun Kim and Priyadarshini Panda.  Optimizing Deeper Spiking Neural Networks for Dynamic Vision Sensing.  _Neural Networks_ , 144:686–698, 2021.
  * Kim et al. [2022] Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, and Priyadarshini Panda.  Neural architecture search for spiking neural networks.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIV_ , pages 36–56. Springer, 2022.
  * Kolda and Bader [2009] Tamara G. Kolda and Brett W. Bader.  Tensor decompositions and applications.  _SIAM Review_ , 51(3):455–500, August 2009.  10.1137/07070111x.  URL https://doi.org/10.1137/07070111x.
  * [23] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  Cifar-10 (canadian institute for advanced research).  URL http://www.cs.toronto.edu/~kriz/cifar.html.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Lapique [1907] Louis Lapique.  Recherches quantitatives sur l’excitation electrique des nerfs traitee comme une polarization.  _Journal of Physiology and Pathology_ , 9:620–635, 1907.
  * Li et al. [2017] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi.  CIFAR10-DVS: An event-stream dataset for object classification.  _Frontiers in Neuroscience_ , 11, May 2017.  10.3389/fnins.2017.00309.  URL https://doi.org/10.3389/fnins.2017.00309.
  * Li et al. [2021] Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu.  Differentiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 34, pages 23426–23439, 2021.
  * Lin et al. [2021] Yihan Lin, Wei Ding, Shaohua Qiang, Lei Deng, and Guoqi Li.  ES-ImageNet: A million event-stream classification dataset for spiking neural networks.  _Frontiers in Neuroscience_ , 15, November 2021.  10.3389/fnins.2021.726582.  URL https://doi.org/10.3389/fnins.2021.726582.
  * Maass [1997] Wolfgang Maass.  Networks of spiking neurons: The third generation of neural network models.  _Neural Networks_ , 10(9):1659–1671, 1997.  ISSN 0893-6080.  https://doi.org/10.1016/S0893-6080(97)00011-7.  URL https://www.sciencedirect.com/science/article/pii/S0893608097000117.
  * Meng et al. [2022] Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo.  Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation.  _ArXiv preprint arXiv:2205.00459_ , 2022.
  * Mnih et al. [2014] Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu.  Recurrent models of visual attention, 2014.  URL https://arxiv.org/abs/1406.6247.
  * Orchard et al. [2015] Garrick Orchard, Ajinkya Jayawant, Gregory K. Cohen, and Nitish Thakor.  Converting static image datasets to spiking neuromorphic datasets using saccades.  _Frontiers in Neuroscience_ , 9, November 2015.  10.3389/fnins.2015.00437.  URL https://doi.org/10.3389/fnins.2015.00437.
  * Pei et al. [2019] Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, Feng Chen, Ning Deng, Si Wu, Yu Wang, Yujie Wu, Zheyu Yang, Cheng Ma, Guoqi Li, Wentao Han, Huanglong Li, Huaqiang Wu, Rong Zhao, Yuan Xie, and Luping Shi.  Towards artificial general intelligence with hybrid tianjic chip architecture.  _Nature_ , 572(7767):106–111, Aug 2019.  ISSN 1476-4687.  10.1038/s41586-019-1424-8.  URL https://doi.org/10.1038/s41586-019-1424-8.
  * Rathi and Roy [2021] Nitin Rathi and Kaushik Roy.  Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization.  _IEEE Transactions on Neural Networks and Learning Systems (TNNLS)_ , 2021.
  * [35] Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy.  Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation.  In _International Conference on Learning Representations (ICLR)_.
  * Rueckauer et al. [2017] Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu.  Conversion of continuous-valued deep networks to efficient event-driven networks for image classification.  _Frontiers in Neuroscience_ , 11, December 2017.  10.3389/fnins.2017.00682.  URL https://doi.org/10.3389/fnins.2017.00682.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _CoRR_ , abs/1409.1556, 2015.
  * Wu et al. [2018] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi.  Spatio-temporal backpropagation for training high-performance spiking neural networks.  _Frontiers in Neuroscience_ , 12, May 2018.  10.3389/fnins.2018.00331.  URL https://doi.org/10.3389/fnins.2018.00331.
  * Wu et al. [2019] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi.  Direct Training for Spiking Neural Networks: Faster, Larger, Better.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , pages 1311–1318, 2019.  10.1609/aaai.v33i01.33011311.
  * Wu et al. [2021] Zhenzhi Wu, Hehui Zhang, Yihan Lin, Guoqi Li, Meng Wang, and Ye Tang.  LIAF-Net: Leaky Integrate and Analog Fire Network for Lightweight and Efficient Spatiotemporal Information Processing.  _IEEE Transactions on Neural Networks and Learning Systems_ , pages 1–14, 2021.  10.1109/TNNLS.2021.3073016.
  * Xu et al. [2022] Qi Xu, Yaxin Li, Jiangrong Shen, Pingping Zhang, Jian K. Liu, Huajin Tang, and Gang Pan.  Hierarchical spiking-based model for efficient image classification with enhanced feature extraction and encoding.  _IEEE Transactions on Neural Networks and Learning Systems_ , pages 1–9, 2022.  10.1109/TNNLS.2022.3232106.
  * Yao et al. [2021] Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang, Yihan Lin, Zhaoxu Yang, and Guoqi Li.  Temporal-wise attention spiking neural networks for event streams classification.  In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. IEEE, October 2021.  10.1109/iccv48922.2021.01006.  URL https://doi.org/10.1109/iccv48922.2021.01006.
  * Yao et al. [2023] Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li.  Attention spiking neural networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Zhang et al. [2022] Malu Zhang, Jiadong Wang, Jibin Wu, Ammar Belatreche, Burin Amornpaisannon, Zhixuan Zhang, Venkata Pavan Kumar Miriyala, Hong Qu, Yansong Chua, Trevor E. Carlson, and Haizhou Li.  Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks.  _IEEE Transactions on Neural Networks and Learning Systems_ , 33(5):1947–1958, 2022.  10.1109/TNNLS.2021.3110991.
  * Zheng et al. [2020] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li.  Going deeper with directly-trained larger spiking neural networks, 2020.  URL https://arxiv.org/abs/2011.05280.
  * Zheng et al. [2021] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li.  Going Deeper With Directly-Trained Larger Spiking Neural Networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , pages 11062–11070, 2021.
  * Zhou et al. [2023] Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng YAN, Yonghong Tian, and Li Yuan.  Spikformer: When spiking neural network meets transformer.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=frE4fUwz_h.
  * Zhu et al. [2022] Rui-Jie Zhu, Qihang Zhao, Tianjing Zhang, Haoyu Deng, Yule Duan, Malu Zhang, and Liang-Jian Deng.  Tcja-snn: Temporal-channel joint attention for spiking neural networks, 2022.  URL https://arxiv.org/abs/2206.10177.