# CLaMP: Contrastive Language-Music Pre-training for Cross-Modal Symbolic
Music Information Retrieval

###### Abstract

We introduce CLaMP: Contrastive Language-Music Pre-training, which learns
cross-modal representations between natural language and symbolic music using
a music encoder and a text encoder trained jointly with a contrastive loss. To
pre-train CLaMP, we collected a large dataset of 1.4 million music-text pairs.
It employed text dropout as a data augmentation technique and bar patching to
efficiently represent music data which reduces sequence length to less than
10%. In addition, we developed a masked music model pre-training objective to
enhance the music encoder’s comprehension of musical context and structure.
CLaMP integrates textual information to enable semantic search and zero-shot
classification for symbolic music, surpassing the capabilities of previous
models. To support the evaluation of semantic search and music classification,
we publicly release WikiMusicText (WikiMT), a dataset of 1010 lead sheets in
ABC notation, each accompanied by a title, artist, genre, and description. In
comparison to state-of-the-art models that require fine-tuning, zero-shot
CLaMP demonstrated comparable or superior performance on score-oriented
datasets. Our models and code are available at
https://github.com/microsoft/muzic/tree/main/clamp.

##  1 Introduction

Symbolic Music Information Retrieval (MIR) is a field that deals with the
automatic analysis and retrieval of music based on symbolic representations
such as sheet music or MIDI files. Symbolic MIR has numerous practical
applications, including music genre classification [1, 2], automatic music
transcription [3, 4], and music recommendation systems [5]. However,
traditional symbolic MIR approaches based on handcrafted features are often
limited in their ability to capture the complex nature of music.

Deep learning has become increasingly popular in symbolic MIR [6, 7, 8, 9] due
to its ability to extract complex and abstract music features from large
datasets. However, obtaining sufficient labelled data can be costly and time-
consuming, as most labelled symbolic music datasets are small in size [10, 11,
12]. To address this issue, semantic search and zero-shot classification
techniques can be used to retrieve and label extensive unlabelled data. These
techniques enable the search for music by a given open-domain query (e.g.,
"upbeat music with a fast tempo"), or the automatic identification of music
characteristics based on customized labels without the need for training data.

Figure 1: The architecture of CLaMP, including two encoders - one for music
and one for text - trained jointly with a contrastive loss to learn cross-
modal representations.

To enable semantic search and zero-shot classification for symbolic music, it
is necessary to establish a connection between music and language. This can be
achieved through the use of contrastive learning [13, 14, 15, 16, 17] and pre-
training [18, 19, 20]. Contrastive learning trains models to learn a feature
space where similar sample pairs are grouped and dissimilar pairs are
separated, while pre-training involves training a model on a large dataset
that can be fine-tuned or directly applied to a specific task.

In this paper, we introduce a solution for cross-modal symbolic MIR that
utilizes contrastive learning and pre-training. The proposed approach, CLaMP:
Contrastive Language-Music Pre-training, is inspired by the success of vision-
language models [13]. Unlike prior models that rely solely on symbolic music
[9, 21, 12], CLaMP learns semantically rich representations of musical
concepts from both sheet music and natural language. The contributions of this
paper are as follows:

Figure 2: Text dropout is a data augmentation technique that involves a
process in which candidate texts are shuffled randomly and then selected to
form a concatenated text. In this example, three candidate texts were randomly
selected and concatenated to produce the input text.

  * •

CLaMP is a cross-modal model for symbolic MIR, which is pre-trained on
WebMusicText (WebMT), a dataset of 1.4 million music-text pairs. To the best
of our knowledge, this is the first model of its kind and it achieves
comparable or better performance than existing state-of-the-art models without
training.

  * •

We propose multiple techniques to improve contrastive language-music pre-
training. Our proposed techniques include applying text dropout as a data
augmentation method, utilizing bar patching for efficient music
representation, and implementing the masked music model pre-training
objective.

  * •

The cross-modal pre-training empowers CLaMP to perform tasks beyond the
capabilities of unimodal models. It possesses unique features such as semantic
search for desired music using open-domain text queries and zero-shot
classification for new music.

  * •

To facilitate the evaluation of semantic search and music classification, we
release the WikiMusicText (WikiMT) dataset, which consists of 1010 music-text
pairs sourced from Wikifonia and Wikipedia.

##  2 Methodology

This section presents CLaMP and its cross-modal symbolic MIR abilities.
Additionally, we describe the WebMT dataset, which we created to pre-train our
model.

###  2.1 Model Design

####  2.1.1 Contrastive Learning Objective

CLaMP jointly trains music and text encoders to represent the structural and
semantic aspects of both modalities in a shared feature space. This is
achieved using a batch construction method and objective [22, 23], as
illustrated in Fig. 1, whereby the correct pairings of a batch of $N$ music-
text pairs are predicted. The music and text encoders employ global average
pooling to obtain corresponding features from the last hidden states.

The objective of CLaMP is to minimize the distance between $N$ paired music-
text examples while maximizing the distance between $N^{2}-N$ unpaired
examples. We denote a batch of $N$ music-text pairs as
${(m_{i},t_{i})}_{i=1}^{N}$, where $m_{i}$ and $t_{i}$ represent the $i$-th
music and text inputs, respectively. The music and text encoders are
represented as $f_{m}$ and $f_{t}$. The contrastive loss for
${(m_{i},t_{i})}_{i=1}^{N}$ is defined as follows:

$\begin{split}\mathcal{L}_{CL}=-\frac{1}{2N}\sum_{i=1}^{N}(\log\frac{\exp(f_{m}(m_{i})\cdot
f_{t}(t_{i})/\tau)}{\sum_{j=1}^{N}\mathbb{1}_{i\neq j}\exp(f_{m}(m_{i})\cdot
f_{t}(t_{j})/\tau)}+\\\ \log\frac{\exp(f_{m}(m_{i})\cdot
f_{t}(t_{i})/\tau)}{\sum_{j=1}^{N}\mathbb{1}_{i\neq j}\exp(f_{m}(m_{j})\cdot
f_{t}(t_{i})/\tau)}),\end{split}$ (1)

where $\tau$ is a temperature hyper-parameter that controls the sharpness of
the softmax distribution, and $\mathbb{1}_{i\neq j}$ is an indicator function
that equals 1 if $i\neq j$, and 0 otherwise. The two terms in Eq. 1 consider
either music-to-text or text-to-music logits.

####  2.1.2 Text Encoder

CLaMP includes a text encoder to extract musically relevant features from the
input text. To achieve optimal performance, a pre-trained language model is
used to initialize the text encoder. Furthermore, text dropout is employed as
a data augmentation technique to prevent overfitting and improve the
generalization ability of the text encoder.

Pre-trained Language Model RoBERTa [24] is a transformer-based language model
pre-trained on a large corpus of English text using the Masked Language
Modeling (MLM) objective [18]. This model is designed to be fine-tuned on
downstream tasks and has demonstrated excellent performance as a text encoder
for the contrastive language-audio pre-training [25]. To improve training
efficiency, we used DistilRoBERTa [26] instead, which has fewer parameters
(82M) compared to RoBERTa-base (125M) while achieving comparable performance.

Text Dropout Text dropout is a data augmentation technique that encourages
models to learn robust features from input texts. This technique involves
using a dataset consisting of multiple paired candidate texts from various
sources for each musical composition. Similar to [27], for a given composition
with $L$ candidates, text dropout shuffles the set of candidate texts and
randomly selects $K$ texts, where $K$ is uniformly and randomly sampled from
integers ranging from 1 to $L$. These selected texts are concatenated to form
a single input text for the text encoder, as shown in Fig. 2. Text dropout
offers a wider range of possible text combinations and allows the model to
learn more complex and diverse textual features.

####  2.1.3 Music Encoder

The CLaMP music encoder is designed to understand the complex musical
structure and context within ABC notation. As a text-based format for symbolic
music, ABC notation incorporates a wide range of musical symbols commonly used
in sheet music. To keep all musical information while shortening sequence
length, the encoding process utilizes the bar patching technique. To optimize
performance, the music encoder is specifically designed for symbolic music
understanding based on bar patching.

Table 1: The average number of tokens per lead sheet in the WikiMT dataset
with different encoding methods.

Encoding |  Bar Patching |  ABC Notation |  OctupleMIDI [9]
---|---|---|---
Tokens |  47.07±21.60 |  749.16±379.56 |  469.09±256.43

Bar Patching The bar in musical notation groups phrases by defining a fixed
number of beats and each bar can be read and played as a single unit. It is
separated by vertical lines, providing reference points for locating positions
within a score.

Previous models [28, 29, 30, 31] for ABC notation utilized character-based
tokenization, resulting in sequences that are too lengthy to process
efficiently. On the other hand, MeasureVAE [32] demonstrated the feasibility
of encoding scores at the bar-level for music generation. To improve the
efficiency of processing, we proposed bar patching, inspired by patch-based
techniques in computer vision [33].

Bar patching divides a score into several small segments corresponding to bars
or headers (i.e. meta-information) in ABC notation. In our implementation,
each patch is assigned a maximum of 64 characters, covering 98.8% of the
headers or bars in the pre-training dataset. We add an [END] token at the end
of each patch to indicate the end of the sequence. Patches with fewer than 64
are padded with [PAD] tokens, while those with over 64 characters are
truncated. For the vocabulary, 95 ASCII printable characters and three special
tokens (i.e., [PAD], [MASK], and [END]) are considered, resulting in a total
of 98 tokens. Thus, each patch can be represented as a 64$\times$98 matrix.
These patches are then flattened and projected into 768 dimensions embeddings
and used as input tokens, as illustrated in Fig. 3.

Bar patching effectively reduces the average sequence length of the encoded
music to less than 10% of the original ABC notation, as shown in Table 1. This
technique improves the efficiency of representing music and facilitates faster
computation while preserving all musical information in the notation.

Figure 3: The masked music model architecture, where the encoder takes in a
sequence of patches, and the decoder reconstructs character information of
noise-added patches.

Figure 4: The processes of CLaMP performing cross-modal symbolic MIR tasks,
including semantic search and zero-shot classification for symbolic music,
without requiring task-specific training data.

Masked Music Model The Masked Music Model (M3) is a self-supervised model for
symbolic MIR based on bar patching representation. The primary concept of M3
is to introduce random noise to certain patches of the input music, and then
reconstruct the characters in the noise-added bar patches based on the
context. This pre-training enables M3 to learn from unlabelled musical data,
making it useful for initializing the CLaMP music encoder.

M3 is based on an asymmetric encoder-decoder architecture, similar to MAE
[34], as shown in Fig. 3. It uses an encoder to extract contextualized
features of individual patches, along with a decoder, which is lightweight and
autoregressively reconstructs the characters for each patch. After pre-
training, the decoder is discarded and the encoder is used to initialize the
music encoder of CLaMP.

The pre-training objective is inspired by MLM [18]. We first randomly select
$M\%$ of the bar patches in the input music, and then the noise is added in
three different ways:

  * •

Masking: $80\%$ of the selected bar patches are replaced with a special patch
filled with [MASK] tokens. This encourages the model to learn to fill in
missing information and understand the relationship between different musical
elements.

  * •

Shuffling: $10\%$ of the selected bar patches are randomly shuffled
internally. For example, a bar patch "|: F |" may be randomly shuffled to "F
:| |" as shown in Fig. 3. This forces the model to learn the patterns and
structures within bar patches.

  * •

Unchanged: $10\%$ of the selected bar patches are left unchanged. This can
narrow down the gap between pre-training and fine-tuning.

M3 is trained to predict the original characters in the noise-added bar
patches based on contextualized patch features. The model is optimized using
the cross-entropy loss, which compares the predicted characters with the
ground truth characters. The final objective is to minimize the average loss
over all the noise-added bar patches in the training set. By denoising these
bar patches, M3 learns to capture the dependencies and relationships between
different musical elements and structures, allowing it to extract meaningful
features from ABC notation.

###  2.2 Cross-Modal Symbolic MIR

CLaMP is capable of aligning symbolic music and natural language, which can be
used for various cross-modal retrieval tasks, including semantic search and
zero-shot classification for symbolic music.

Semantic search is a technique for retrieving music by open-domain queries,
which differs from traditional keyword-based searches that depend on exact
matches or meta-information. This involves two steps: 1) extracting music
features from all scores in the library, and 2) transforming the query into a
text feature. By calculating the similarities between the text feature and the
music features, it can efficiently locate the score that best matches the
user’s query in the library.

Zero-shot classification refers to the classification of new items into any
desired label without the need for training data. It involves using a prompt
template to provide context for the text encoder. For example, a prompt such
as "This piece of music is composed by {composer}." is utilized to form input
texts based on the names of candidate composers. The text encoder then outputs
text features based on these input texts. Meanwhile, the music encoder
extracts the music feature from the unlabelled target symbolic music. By
calculating the similarity between each candidate text feature and the target
music feature, the label with the highest similarity is chosen as the
predicted one.

###  2.3 WebMusicText Dataset

To facilitate the learning of relationships between natural language and
symbolic music, we developed a dataset named WebMusicText (WebMT) by crawling
an extensive collection of music-text pairs from the web. Our dataset
comprises 1,448,750 pairs of music-text data, where all music files are in
score-oriented formats (e.g., MusicXML, LilyPond, and ABC notation). To reduce
the disparity between scores in different notations, we first converted all
music files to MusicXML and then to ABC
notation111https://wim.vree.org/svgParse/xml2abc.html. In addition, to avoid
information leakage, we removed any natural language (e.g., titles, composers,
and lyrics) in ABC notation. The text parts of each pair were obtained from
corresponding meta-information (e.g., title and composer) or user comments,
and are all in English. WebMT features diverse musical compositions, from
monophonic folk music to polyphonic orchestral music, which enables the model
to learn a wide range of musical information.

##  3 Experiments

###  3.1 Settings

####  3.1.1 Models

  * •

MusicBERT [9]: This model combines unsupervised pre-training with supervised
fine-tuning, which achieved state-of-the-art results. MusicBERT is available
in two settings: MusicBERT-S/1024 ($\rm MusicBERT_{small}$), and
MusicBERT-B/1024 ($\rm MusicBERT_{base}$). MusicBERT-S/1024 consists of 4
layers and was pre-trained on the small-scale Lakh MIDI Dataset (LMD, 148,403
pieces) [35], while MusicBERT-B/1024 has 12 layers and was pre-trained on the
large-scale Million MIDI Dataset (MMD, 1,524,557 pieces). Both models have a
maximum length of 1024.

  * •

M3: Our proposed music encoder is used to compare the performances of unimodal
and multimodal models trained on the same dataset (i.e., WebMT). M3 comes with
two settings: M3-S/512 and M3-S/1024, with maximum lengths of 512 and 1024,
respectively. In the following experiments, both settings use the 6 encoder
layers only.

  * •

CLaMP: Several variants were tested to verify the effectiveness of the
proposed techniques for improving contrastive language-music pre-training.
These include CLaMP-S/512 which is the full model, CLaMP-S/512 (w/o TD) which
removes text dropout, CLaMP-S/512 (w/o M3) which has a randomly initialized
music encoder, and CLaMP-S/512 (w/o M3, BP) which removes both M3 and bar
patching, and uses char-level tokenization to encode raw ABC notation instead.
CLaMP-S/1024 was included to verify the effectiveness of an extended maximum
length.

####  3.1.2 Pre-training

The text encoder was initialized using DistilRoBERTa [26], with a maximum
length of 128, and the music encoder was initialized using two settings:
M3-S/512 and M3-S/1024. A length of 512 resulted in truncating 17.29% of
compositions in WebMT, while a length of 1024 reduced truncation to 7.7%. Both
models were trained for 40 epochs with 6 encoder layers and 3 decoder layers,
an embedding size of 768, and a noise ratio of 45%. Based on these two M3
encoders, we developed CLaMP-S/512 and CLaMP-S/1024. Both of them were trained
for 20 epochs, using the AdamW optimizer [36] with $\beta_{1}=0.9$,
$\beta_{2}=0.999$, $\epsilon=10^{-8}$, and a weight decay coefficient of 0.01.
The batch size is set to 640, and the temperature $\tau=0.2$. The training
process was accelerated and memory was saved by using mixed precision [37].

####  3.1.3 Evaluation Datasets

We introduce WikiMusicText (WikiMT)222https://huggingface.co/datasets/sander-
wood/wikimt, a new dataset for the evaluation of semantic search and music
classification. It includes 1010 lead sheets (melodies with harmonies) in ABC
notation sourced from Wikifonia, each accompanied by a title, artist, genre,
and description. The title and artist information is extracted from the score,
whereas the genre labels are obtained by matching keywords from the Wikipedia
entries and assigned to one of the 8 classes that loosely mimic the GTZAN
genres [38]. The description is obtained by utilizing BART-large [39] to
summarize and clean the corresponding Wikipedia entry. Additionally, following
WebMT, the natural language information within the ABC notation is removed.

In addition to WikiMT, we use two other datasets to evaluate music
classification: VGMIDI and Pianist8. VGMIDI [11] includes 204 score-oriented
MIDI arrangements that were classified according to the valence-arousal model.
Pianist8 [12] contains symbolic piano performances of 411 pieces from 8
composers with distinct styles, which were automatically transcribed from
audio using a model presented in [8].

####  3.1.4 Metrics

We use the following three metrics to evaluate the effectiveness of models in
various downstream tasks:

  * •

Mean Reciprocal Rank (MRR) is used to evaluate ranking systems. This metric
calculates the average of the reciprocal ranks of the correct answers, which
measures the effectiveness of the ranking.

  * •

Hit Ratio at K (HR@K) measures the accuracy of the model by checking if the
correct item is among the top K recommendations, which is often used in
recommendation systems.

  * •

F1-macro score is a metric that assesses the overall effectiveness of a
classification model. It is computed using the arithmetic mean (i.e.,
unweighted mean) of all the per-class F1 scores.

Table 2: Semantic search performance of CLaMP on WikiMT (1010 music-text
pairs) under different settings.

Setting |  MRR |  HR@1 |  HR@10 |  HR@100
---|---|---|---|---
S/512 |  0.2561 |  0.1931 |  0.3693 |  0.7020
S/1024 |  0.2016 |  0.1436 |  0.3109 |  0.6554
S/512 (w/o TD) |  0.1841 |  0.1248 |  0.2911 |  0.6188
S/512 (w/o M3) |  0.1262 |  0.0802 |  0.1960 |  0.5119
S/512 (w/o M3, BP) |  0.0931 |  0.0525 |  0.1584 |  0.4426

Table 3: Classification performance of different models on three datasets:
WikiMT (1010 pieces, 8 genres), VGMIDI (204 pieces, 4 emotions), and Pianist8
(411 pieces, 8 composers).

Model | WikiMT | VGMIDI [11] | Pianist8 [12]
---|---|---|---
F1-macro | Accuracy | F1-macro | Accuracy | F1-macro | Accuracy
Linear Probe MusicBERT-S/1024 |  0.2401 |  0.3507 |  0.4662 |  0.5350 |  0.8047 |  0.8102
Linear Probe MusicBERT-B/1024 |  0.1746 |  0.3219 |  0.5127 |  0.5850 |  0.8379 |  0.8413
Zero-shot CLaMP-S/512 |  0.2660 |  0.3248 |  0.5217 |  0.6176 |  0.2180 |  0.2512
Zero-shot CLaMP-S/1024 |  0.2248 |  0.3406 |  0.4678 |  0.5049 |  0.1509 |  0.2390
Linear Probe M3-S/512 |  0.2832 |  0.3990 |  0.5991 |  0.6667 |  0.6773 |  0.6909
Linear Probe M3-S/1024 |  0.3079 |  0.4020 |  0.5966 |  0.6522 |  0.6844 |  0.6958
Linear Probe CLaMP-S/512 |  0.3452 |  0.4267 |  0.6453 |  0.6866 |  0.7067 |  0.7152
Linear Probe CLaMP-S/1024 |  0.3449 |  0.4416 |  0.6345 |  0.6720 |  0.7271 |  0.7298

###  3.2 Results

####  3.2.1 Semantic Search

In the semantic search evaluation, we assessed different versions of CLaMP for
semantic search, aiming to test the efficacy of contrastive language-music
pre-training techniques. The pre-training dataset WebMT and the evaluation
dataset WikiMT have no overlap, thus guaranteeing the validity of our
evaluation results. In addition, as semantic search requires no additional
training for this dataset, it demonstrates the generalizability of CLaMP.

Table 2 shows that our full model (CLaMP-S/512) outperforms all other models
across all metrics. Interestingly, we discovered that increasing the maximum
sequence length to 1024 (CLaMP-S/1024) did not lead to an improvement in
performance. We attribute this to the fact that all lead sheets in the WikiMT
dataset, once encoded with bar patching, have a length smaller than 512, which
limits the potential advantages of the longer sequence length of CLaMP-S/1024.
We also observed that the removal of the proposed techniques from CLaMP had a
considerable negative impact on semantic search performance. Notably, the
removal of M3 pre-training had the greatest effect on model performance,
followed by text dropout and bar patching.

In conclusion, our evaluation of CLaMP on WikiMT shows that CLaMP-S/512 with
all proposed contrastive language-music pre-training techniques is the most
effective for the semantic search task. This highlights the importance of
these techniques for effective pre-training and semantic search tasks.
Additionally, increasing the sequence length (CLaMP-S/1024) did not improve
the model’s performance. These results emphasize the significance of using
appropriate pre-training techniques in language-music models and suggest that
a longer sequence length may not necessarily result in better outcomes.

####  3.2.2 Music Classification

The goal of the classification evaluation is to assess how well the zero-shot
CLaMP models perform compared to other fine-tuned models. In addition, to
evaluate pre-trained models, linear probes are used to train a linear
classifier for the classification based on the features from pre-trained
models. Despite being less powerful and relying on pre-trained model features,
linear classifiers offer a valuable means of quantitatively assessing feature
quality [40].

WikiMT was converted into the MIDI format using music21 [41] to be compatible
with MusicBERT. In contrast, for VGMIDI and Pianist8, we employed MuseScore3’s
batch conversion tool333https://musescore.org/en/project/batch-convert to
convert the scores into the MusicXML format, which were then converted into
ABC notation for use with M3 and CLaMP.

We conducted 5-fold cross-validation with the same folds to assess all linear
probe models, using identical fine-tuning settings and a batch size of 10 to
ensure consistency, given the limited size of the evaluation datasets. The
linear probe CLaMP models used the music encoder only, while the text encoder
was discarded. In the zero-shot classification setting, CLaMP had no previous
exposure to these evaluation datasets during pre-training. We utilized
manually designed prompts for the zero-shot CLaMP models.

The top half of Table 3 presents the comparison of the performance between
linear probe MusicBERT and zero-shot CLaMP. The results found that the zero-
shot CLaMP models demonstrated comparable or even superior performance
compared to the linear probe MusicBERT models on WikiMT and VGMIDI datasets.
Interestingly, the smaller zero-shot CLaMP-S/512 outperformed the larger
linear probe MusicBERT-B/1024, indicating that the pre-training of CLaMP has
enabled it to learn more generalizable features that are useful for zero-shot
music classification. However, this trend was not observed on Pianist8, where
MusicBERT models performed much better than zero-shot CLaMP models. This
difference in performance can be attributed to the source of the datasets, as
WikiMT and VGMIDI primarily focus on score information, whereas Pianist8
contains performance MIDI data derived from audio. Since both CLaMP and M3
were trained exclusively on score information, they lack knowledge of
performance MIDI. However, we noticed that the performances of linear probe
CLaMP models on Pianist8 significantly improved after fine-tuning compared to
the zero-shot ones. This suggests that incorporating ABC notation from
performance MIDI into the pre-training of CLaMP may enhance its ability to
comprehend such data.

The linear probe CLaMP models show better performance compared to the linear
probe M3 models, as indicated in the bottom half of Table 3, despite being
pre-trained on the same dataset with the same architecture. This is attributed
to the use of contrastive learning, which aligns the music encoder of CLaMP
with the text modality, thus implicitly introducing textual information to the
music encoder. Furthermore, we found that CLaMP-S/1024 performed better on
Pianist8 than CLaMP-S/512, suggesting that a larger maximum length is
beneficial for models to learn performance MIDI.

In summary, our evaluation demonstrates that zero-shot CLaMP performs
comparably to state-of-the-art models in music classification. Furthermore,
the incorporation of contrastive learning and textual information enhances the
music encoder’s performance, resulting in better classification accuracy when
compared to M3 which employed the same architecture. These results highlight
the potential of CLaMP as a pre-training framework for symbolic MIR.

##  4 Conclusions

This paper introduces CLaMP, a pre-trained model that utilizes contrastive
language-music pre-training techniques to build cross-modal representations
between natural language and symbolic music. The model was trained on a
dataset containing 1.4 million music-text pairs and has demonstrated unique
abilities of semantic search and zero-shot classification for symbolic music.
Compared to state-of-the-art models that require fine-tuning, zero-shot CLaMP
exhibits comparable or superior performance in score-oriented music
classification tasks without any training. However, the current version of
CLaMP has limited comprehension of performance MIDI, and still has room for
improvement. Future research will aim to expand its capabilities by scaling it
up and pre-training it on larger datasets that incorporate a wider range of
symbolic music formats beyond score-oriented ones. We expect that its cross-
modal representations will facilitate research on new topics in music
analysis, retrieval, and generation, and provide a foundation for the
development of innovative systems and applications that integrate music and
language.

## References

  * [1] I. Karydis, “Symbolic music genre classification based on note pitch and duration,” in _Advances in Databases and Information Systems, 10th East European Conference, ADBIS 2006, Thessaloniki, Greece, September 3-7, 2006, Proceedings_ , ser. Lecture Notes in Computer Science, Y. Manolopoulos, J. Pokorný, and T. K. Sellis, Eds., vol. 4152. Springer, 2006, pp. 329–338. [Online]. Available: https://doi.org/10.1007/11827252_25
  * [2] C. Kofod and D. O. Arroyo, “Exploring the design space of symbolic music genre classification using data mining techniques,” in _2008 International Conferences on Computational Intelligence for Modelling, Control and Automation (CIMCA 2008), Intelligent Agents, Web Technologies and Internet Commerce (IAWTIC 2008), Innovation in Software Engineering (ISE 2008), 10-12 December 2008, Vienna, Austria_ , M. Mohammadian, Ed. IEEE Computer Society, 2008, pp. 43–48. [Online]. Available: https://doi.org/10.1109/CIMCA.2008.223
  * [3] J. P. Bello, G. Monti, and M. B. Sandler, “Techniques for automatic music transcription,” in _ISMIR 2000, 1st International Symposium on Music Information Retrieval, Plymouth, Massachusetts, USA, October 23-25, 2000, Proceedings_ , 2000. [Online]. Available: http://ismir2000.ismir.net/papers/bello_paper.pdf
  * [4] C. Raphael, “Automatic transcription of piano music,” in _ISMIR 2002, 3rd International Conference on Music Information Retrieval, Paris, France, October 13-17, 2002, Proceedings_ , 2002. [Online]. Available: http://ismir2002.ismir.net/proceedings/02-FP01-2.pdf
  * [5] C. Walshaw, “A statistical analysis of the abc music notation corpus: Exploring duplication,” 2014.
  * [6] C. Hawthorne, I. Simon, R. Swavely, E. Manilow, and J. H. Engel, “Sequence-to-sequence piano transcription with transformers,” in _Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021_ , J. H. Lee, A. Lerch, Z. Duan, J. Nam, P. Rao, P. van Kranenburg, and A. Srinivasamurthy, Eds., 2021, pp. 246–253. [Online]. Available: https://archives.ismir.net/ismir2021/paper/000030.pdf
  * [7] J. Gardner, I. Simon, E. Manilow, C. Hawthorne, and J. H. Engel, “MT3: multi-task multitrack music transcription,” in _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022. [Online]. Available: https://openreview.net/forum?id=iMSjopcOn0p
  * [8] Q. Kong, B. Li, X. Song, Y. Wan, and Y. Wang, “High-resolution piano transcription with pedals by regressing onset and offset times,” _IEEE ACM Trans. Audio Speech Lang. Process._ , vol. 29, pp. 3707–3717, 2021. [Online]. Available: https://doi.org/10.1109/TASLP.2021.3121991
  * [9] M. Zeng, X. Tan, R. Wang, Z. Ju, T. Qin, and T. Liu, “Musicbert: Symbolic music understanding with large-scale pre-training,” in _Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021_ , ser. Findings of ACL, C. Zong, F. Xia, W. Li, and R. Navigli, Eds., vol. ACL/IJCNLP 2021. Association for Computational Linguistics, 2021, pp. 791–800. [Online]. Available: https://doi.org/10.18653/v1/2021.findings-acl.70
  * [10] A. Ferraro and K. Lemström, “On large-scale genre classification in symbolically encoded music by automatic identification of repeating patterns,” in _Proceedings of the 5th International Conference on Digital Libraries for Musicology, DLfM 2018, Paris, France, September 28, 2018_ , K. R. Page, Ed. ACM, 2018, pp. 34–37. [Online]. Available: https://doi.org/10.1145/3273024.3273035
  * [11] L. N. Ferreira and J. Whitehead, “Learning to generate music with sentiment,” 2019\.
  * [12] Y. Chou, I. Chen, C. Chang, J. Ching, and Y. Yang, “Midibert-piano: Large-scale pre-training for symbolic music understanding,” _CoRR_ , vol. abs/2107.05223, 2021. [Online]. Available: https://arxiv.org/abs/2107.05223
  * [13] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 2021, pp. 8748–8763. [Online]. Available: http://proceedings.mlr.press/v139/radford21a.html
  * [14] W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” in _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 2021, pp. 5583–5594. [Online]. Available: http://proceedings.mlr.press/v139/kim21k.html
  * [15] J. Li, D. Li, C. Xiong, and S. C. H. Hoi, “BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation,” in _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, Eds., vol. 162. PMLR, 2022, pp. 12 888–12 900. [Online]. Available: https://proceedings.mlr.press/v162/li22n.html
  * [16] B. Elizalde, S. Deshmukh, M. A. Ismail, and H. Wang, “CLAP: learning audio concepts from natural language supervision,” _CoRR_ , vol. abs/2206.04769, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2206.04769
  * [17] Q. Huang, A. Jansen, J. Lee, R. Ganti, J. Y. Li, and D. P. W. Ellis, “Mulan: A joint embedding of music audio and natural language,” _CoRR_ , vol. abs/2208.12415, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2208.12415
  * [18] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , J. Burstein, C. Doran, and T. Solorio, Eds. Association for Computational Linguistics, 2019, pp. 4171–4186. [Online]. Available: https://doi.org/10.18653/v1/n19-1423
  * [19] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020\. [Online]. Available: https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
  * [20] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _J. Mach. Learn. Res._ , vol. 21, pp. 140:1–140:67, 2020. [Online]. Available: http://jmlr.org/papers/v21/20-074.html
  * [21] Z. Wang and G. Xia, “Musebert: Pre-training music representation for music understanding and controllable generation,” in _Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021_ , J. H. Lee, A. Lerch, Z. Duan, J. Nam, P. Rao, P. van Kranenburg, and A. Srinivasamurthy, Eds., 2021, pp. 722–729. [Online]. Available: https://archives.ismir.net/ismir2021/paper/000090.pdf
  * [22] K. Sohn, “Improved deep metric learning with multi-class n-pair loss objective,” in _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain_ , D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, Eds., 2016, pp. 1849–1857. [Online]. Available: https://proceedings.neurips.cc/paper/2016/hash/6b180037abbebea991d8b1232f8a8ca9-Abstract.html
  * [23] A. van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _CoRR_ , vol. abs/1807.03748, 2018. [Online]. Available: http://arxiv.org/abs/1807.03748
  * [24] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” _CoRR_ , vol. abs/1907.11692, 2019. [Online]. Available: http://arxiv.org/abs/1907.11692
  * [25] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” _CoRR_ , vol. abs/2211.06687, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2211.06687
  * [26] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” _ArXiv_ , vol. abs/1910.01108, 2019.
  * [27] S. Doh, M. Won, K. Choi, and J. Nam, “Toward universal text-to-music retrieval,” in _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5.
  * [28] B. L. Sturm, J. F. Santos, O. Ben-Tal, and I. Korshunova, “Music transcription modelling and composition using deep learning,” _CoRR_ , vol. abs/1604.08723, 2016. [Online]. Available: http://arxiv.org/abs/1604.08723
  * [29] C. Geerlings and A. Meroño-Peñuela, “Interacting with gpt-2 to generate controlled and believable musical sequences in abc notation,” in _NLP4MUSA_ , 2020.
  * [30] S. Wu and M. Sun, “Tunesformer: Forming tunes with control codes,” _CoRR_ , vol. abs/2301.02884, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2301.02884
  * [31] ——, “Exploring the efficacy of pre-trained checkpoints in text-to-music generation task,” in _The AAAI-23 Workshop on Creative AI Across Modalities_ , 2023. [Online]. Available: https://openreview.net/forum?id=QmWXskBhesn
  * [32] A. Pati, A. Lerch, and G. Hadjeres, “Learning to traverse latent spaces for musical score inpainting,” in _Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, Delft, The Netherlands, November 4-8, 2019_ , A. Flexer, G. Peeters, J. Urbano, and A. Volk, Eds., 2019, pp. 343–351. [Online]. Available: http://archives.ismir.net/ismir2019/paper/000040.pdf
  * [33] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021. [Online]. Available: https://openreview.net/forum?id=YicbFdNTTy
  * [34] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. B. Girshick, “Masked autoencoders are scalable vision learners,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_. IEEE, 2022, pp. 15 979–15 988. [Online]. Available: https://doi.org/10.1109/CVPR52688.2022.01553
  * [35] C. Raffel, “Learning-based methods for comparing sequences, with applications to audio-to-midi alignment and matching,” Ph.D. dissertation, PhD Thesis, 2016\.
  * [36] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7
  * [37] P. Micikevicius, S. Narang, J. Alben, G. F. Diamos, E. Elsen, D. García, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, “Mixed precision training,” in _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net, 2018. [Online]. Available: https://openreview.net/forum?id=r1gs9JgRZ
  * [38] B. L. Sturm, “An analysis of the GTZAN music genre dataset,” in _Proceedings of the second international ACM workshop on Music information retrieval with user-centered and multimodal strategies, MIRUM ’12, Nara, Japan, October 29 - November 02, 2012_ , C. C. S. Liem, M. Müller, S. K. Tjoa, and G. Tzanetakis, Eds., 2012, pp. 7–12. [Online]. Available: https://doi.org/10.1145/2390848.2390851
  * [39] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, Eds. Association for Computational Linguistics, 2020, pp. 7871–7880. [Online]. Available: https://doi.org/10.18653/v1/2020.acl-main.703
  * [40] Y. M. Asano, C. Rupprecht, and A. Vedaldi, “A critical analysis of self-supervision, or what we can learn from a single image,” in _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020. [Online]. Available: https://openreview.net/forum?id=B1esx6EYvr
  * [41] M. S. Cuthbert and C. Ariza, “Music21: A toolkit for computer-aided musicology and symbolic music data,” in _Proceedings of the 11th International Society for Music Information Retrieval Conference, ISMIR 2010, Utrecht, Netherlands, August 9-13, 2010_ , J. S. Downie and R. C. Veltkamp, Eds. International Society for Music Information Retrieval, 2010, pp. 637–642. [Online]. Available: http://ismir2010.ismir.net/proceedings/ismir2010-108.pdf

Table 4: The genre keywords we used for automatically assigning genre labels
based on Wikipedia entries.

Genre |  Keywords
---|---
Jazz |  jazz/bebop/swing
Country |  country/bluegrass/honky-tonk
Folk |  folk/singer-songwriter/acoustic
R&B |  r&b/motown/funk/soul/gospel/blues/boogie-woogie
Pop |  pop/bubblegum
Rock |  rock/punk
Dance |  dance/disco/electronic
Latin |  latin

Table 5: Token count statistics for the WikiMT dataset with different encoding
methods and data types.

Encoding |  Avg |  Max |  Min
---|---|---|---
ABC notation (music) |  749.16±379.56 |  2912 |  107
Bar Patching (music) |  47.07±21.60 |  188 |  11
Whitespace (text) |  54.63±13.94 |  99 |  22
BPE (text) |  72.96±17.60 |  127 |  31

Figure 5: The genre distribution of the WikiMT dataset.

##  Appendix A WikiMusicText dataset

###  A.1 Data Collection and Curation

To collect the data for the WikiMusicText (WikiMT) dataset, the following
steps were taken: 1) converting all the MusicXML sheet music in Wikifonia to
ABC notation, 2) extracting the title and artist name from each piece of
music, 3) crawling the corresponding Wikipedia entry based on the title and
artist name, 4) checking the most frequent genre keywords in the entry (as
given in Table 4), and assigning the corresponding genre label, and 5) using
BART-large to summarize the entry to keep them concise while also in a good
form.

To be included in the WikiMT dataset, each piece of music had to meet the
following criteria: 1) it must have a title and artist information, 2) the
original entry must contain genre keywords, and only one genre could have the
most matches, 3) after summarization by BART-large, the entry must still
include the artist name, and 4) to ensure that there were sufficient data for
each genre, any genres that had less than 10 files were removed from the
dataset (e.g., classical, hip-hop, metal, and reggae).

All of the scores included in WikiMT were believed to be in the public domain
to the best of our knowledge. However, it is important to acknowledge that
copyright issues cannot be entirely ruled out. Therefore, users of the dataset
should exercise caution when using it.

###  A.2 Dataset Statistics

WikiMT comprises 1010 pieces of music in ABC notation converted from
Wikifonia. Each composition is accompanied by its title, artist, genre, and a
summarized Wikipedia entry. The dataset covers eight genres, and their
respective piece counts are displayed in Fig. 5.

Table 5 shows that the token counts for the dataset vary significantly
depending on the encoding method and data type. ABC notation yields the
highest token count due to its detailed and complex representation of sheet
music. In contrast, the proposed bar patching representation reduces the
length of music to less than 10%. However, as all scores in WikiMT are lead
sheets, the average sequence length of ABC notation is still acceptable for
most models.

In terms of text encoding, whitespace encoding has a lower average token count
than Byte-Pair Encoding (BPE). This is to be expected as whitespace encoding
simply breaks text into tokens based on whitespace characters in the text,
whereas BPE is subword tokenization, which decomposes rare words into smaller
subwords.

WikiMT is a unique resource to support the evaluation of semantic search and
music classification. However, it is important to acknowledge that the dataset
was curated from publicly available sources, and there may be limitations
concerning the accuracy and completeness of the genre and description
information. Further research is needed to explore the potential biases and
limitations of the dataset and to develop strategies to address them.

##  Appendix B Prompts for Music Classification

###  B.1 WikiMT

  * •

Country: This piece of music is characterized by its roots in the traditional
music of the American South, with its distinctive twangy sound and
storytelling lyrics that often touch on themes of love, heartbreak, and rural
life.

  * •

Folk: This piece of music draws on the traditional music of a particular
culture or region, often featuring acoustic instruments and simple, catchy
melodies that are easy to sing along to. Folk music often tells stories of
everyday life and the struggles of ordinary people.

  * •

Dance: Whether it’s disco, hip-hop, or EDM, dance music is all about creating
a high-energy atmosphere and bringing people together through the universal
language of dance.

  * •

Latin: This piece of music draws on the rich musical traditions of Latin
America, with its vibrant rhythms, colorful melodies, and passionate lyrics
that often touch on themes of love, passion, and cultural identity.

  * •

Jazz: This piece of music is characterized by its improvisational nature, with
musicians often taking turns soloing over a complex and syncopated rhythm
section. Jazz music often draws on elements of blues, swing, and Bebop music,
and is known for its sophisticated harmonies and inventive melodies.

  * •

Pop: This piece of music is designed to be catchy and easy to sing along to,
with simple, memorable melodies and lyrics that often touch on themes of love,
relationships, and self-expression. Pop music can take many forms, from
bubblegum pop and boy bands to synthpop and EDM.

  * •

Rock: Whether it’s classic rock, punk, or grunge, rock music is all about
pushing boundaries and challenging the status quo. Rock music has a rich
history that spans decades, with iconic bands and legendary performances that
continue to inspire new generations of musicians and fans.

  * •

R&B: R&B music can take many forms, from classic Motown to contemporary hip-
hop and trap soul.

###  B.2 VGMIDI

  * •

Joy (high valence high arousal): This piece of music is a jubilant celebration
that radiates a contagious sense of happiness and joy.

  * •

Anger (low valence high arousal): This piece of music is a visceral experience
that unleashes a torrent of anger and fear.

  * •

Sadness (low valence low arousal): This piece of music is a poignant
reflection that evokes a deep sense of sadness and melancholy, taking the
listener on an emotional journey through the depths of human experience.

  * •

Calmness (high valence low arousal): This piece of music is a soothing balm
that washes over the listener with a gentle wave of calmness and tranquility.

###  B.3 Pianist8

  * •

Clayderman: This music was performed by Richard Clayderman, a French pianist
known for his romantic and sentimental style, whose repertoire includes a mix
of original compositions, classical pieces, and popular music covers.

  * •

Yiruma: This music was composed by Yiruma, a South Korean pianist and composer
whose music has gained popularity through YouTube and social media, and whose
style combines classical and contemporary elements with a strong emotional
core.

  * •

Hancock: This music was composed by Herbie Hancock, an American jazz pianist
and composer who is known for his innovative approach to jazz music, and for
incorporating elements of funk, rock, and electronic music into his
compositions.

  * •

Einaudi: This music was composed by Ludovico Einaudi, an Italian pianist and
composer who is known for his minimalist and meditative music that often
incorporates elements of classical, rock, and electronic music.

  * •

Hisaishi: This music was composed by Hisaishi Joe, a Japanese composer known
for his emotionally deep music that often features a blend of classical and
traditional Japanese elements, and who has worked on numerous films and anime
soundtracks.

  * •

Ryuichi: This music was composed by Ryuichi Sakamoto, a Japanese musician and
composer known for his eclectic and experimental approach to music, which
often blends elements of classical, electronic, and traditional Japanese
music, and who has worked on a variety of projects including film scores, solo
albums, and collaborations with other musicians.

  * •

Bethel: This music was created by the band Bethel Music, an American worship
music collective based in California, known for its contemporary sound and
strong Christian themes.

  * •

Hillsong: This music was created by Hillsong, an Australian-based worship
music collective that has become one of the most well-known and influential
Christian music groups in the world, characterized by its powerful lyrics,
modern sound, and uplifting messages of faith and hope.


Based on the general solution of the nonequilibrium Green functions we
obtained recently Zhang2012 , if there is no localized bound states (modes),
the dissipative propagating Green function vanishes in the steady-state limit,
namely

$\displaystyle\bm{u}(t\rightarrow\infty,t_{0})=0.$ (93)

This solution is valid for arbitrary continuous spectral density matrix of
multiple reservoirs $J_{\alpha ij}(\omega)$ that cover every point of the
whole energy frequency domain. Then the coefficients in the propagating
function of the reduced density matrix, Eq. (92), is largely simplified:
$\bm{J}_{1}(t,t_{0})=0$,
$\bm{J}_{2}(t)=\bm{1}-\bm{w}(t)=\pm\bm{v}(t,t)/[\bm{1}\pm\bm{v}(t,t)]$ and
$\bm{J}_{3}(t,t_{0})=\bm{1}$. Thus, the propagating function is simply reduced
to

$\displaystyle\mathcal{J}(\bm{\alpha}^{\dagger}_{t},\bm{\alpha}^{\prime}_{t},$
$\displaystyle
t\rightarrow\infty;\bm{\alpha}_{0},{\bm{\alpha}^{\prime}_{0}}^{{\dagger}},t_{0})=\lim_{t\rightarrow\infty}\big{(}{\rm
det}[\bm{w}(t)]\big{)}^{\pm 1}$
$\displaystyle\times\exp\Big{\\{}\pm{\bm{\alpha}_{0}^{\prime}}^{\dagger}\bm{\alpha}_{0}\pm\bm{\alpha}_{t}^{{\dagger}}[\bm{1}-\bm{w}(t)]\bm{\alpha}^{\prime}_{t}\Big{\\}}.$
(94)

Substituting this result into Eq. (88), we obtain the exact steady-state
reduced density matrix,

$\displaystyle\rho_{{}_{\\!S}}(\bm{\alpha}_{t},$
$\displaystyle\bm{\alpha}_{t}^{\prime},t\rightarrow\infty)=\\!\lim_{t\rightarrow\infty}\\!\int\\!\\!d\mu\left(\bm{\alpha}_{0}\right)d\mu\left(\bm{\alpha}_{0}^{\prime}\right)\rho_{{}_{\\!S}}\left(\bm{\alpha}_{0},\bm{\alpha}_{0}^{\prime},t_{0}\right)$
$\displaystyle\times\\!\big{(}{\rm det}[\bm{w}(t)]\big{)}^{\pm
1}\\!\\!\exp\Big{\\{}\pm{\bm{\alpha}_{0}^{\prime}}^{\dagger}\bm{\alpha}_{0}\pm\bm{\alpha}_{t}^{{\dagger}}[\bm{1}-\bm{w}(t)]\bm{\alpha}^{\prime}_{t}\Big{\\}}.$
(95)

Notice the normalization condition

$\displaystyle\int\\!\\!d\mu\left(\bm{\alpha}_{0}\right)d\mu\left(\bm{\alpha}_{0}^{\prime}\right)\rho_{{}_{\\!S}}\left(\bm{\alpha}_{0},\bm{\alpha}_{0}^{\prime},t_{0}\right)\exp\Big{\\{}\\!\\!\pm{\bm{\alpha}_{0}^{\prime}}^{\dagger}\bm{\alpha}_{0}\Big{\\}}=1,$
(96)

we have

$\displaystyle\langle\bm{\alpha}_{t}|$
$\displaystyle\rho_{{}_{\\!S}}\left(t\rightarrow\infty\right)|\bm{\alpha}^{\prime}_{t}\rangle$
$\displaystyle=\lim_{t\rightarrow\infty}\big{(}{\rm
det}[\bm{w}(t)]\big{)}^{\pm
1}\\!\\!\exp\Big{\\{}\pm\bm{\alpha}_{t}^{{\dagger}}[\bm{1}-\bm{w}(t)]\bm{\alpha}^{\prime}_{t}\Big{\\}}$
$\displaystyle=\lim_{t\rightarrow\infty}\Big{(}{\rm
det}[\frac{\bm{1}}{\bm{1}\pm\bm{v}(t,t)}]\Big{)}^{\pm
1}\\!\\!\langle\bm{\alpha}_{t}|\frac{\bm{v}(t,t)}{\bm{1}\pm\bm{v}(t,t)}\bm{\alpha}^{\prime}_{t}\rangle.$
(97)

This shows that the steady-state reduced density matrix is independent of its
initial states, as a consequence of thermalization. Equation (97) directly
results in the operator form of the steady-state reduced density matrix

$\displaystyle\rho_{{}_{\\!S}}\\!\left(t\rightarrow\infty\right)=$
$\displaystyle\lim_{t\rightarrow\infty}\Big{(}\frac{1}{\det[\bm{1}\pm\bm{v}(t,t)]}\Big{)}^{\pm
1}$
$\displaystyle\times\\!\exp\Big{\\{}\bm{a}^{\dagger}\\!\Big{(}\\!\ln\frac{\bm{v}(t,t)}{\bm{1}\pm\bm{v}(t,t)}\Big{)}\bm{a}\Big{\\}},$
(98)

where $\bm{a}\equiv(a_{1},a_{2},\cdots,a_{N})^{T}$ is a one-column matrix
operator. This is the exact steady-state solution of Eq. (47), which is also
recently derived from the general solution of the reduced density matrix
Xiong2020 . As one can see, the solution of Eq. (19) derived alternatively in
Appendix A is a special case of the above general solution. We should also
point out that the above solution remains the same for the initial coupled
system-reservoir state Yang2015 ; Huang2020 .

## References

  * (1) L. D. Landau and E. M. Lifshitz, _Statistical Physics_ , Vol. 5 of Course of Theoretical Physics (2nd Ed., Pergamon Press, 1969).
  * (2) R. Kubo, M. Toda and N. Hashitsume, _Statistical Physics II: Nonequilibrium statistical mechanic_ (Springer Series in Solid-State Sciences, Vol. 31, 2nd Ed., 1991).
  * (3) K. Huang, _Statistical Mechanics_ (2nd Ed., John Wiley and Sons, 1987). Chap. 8.7, pp. 189-191.
  * (4) R. P. Feynman and F. L. Vernon, _The theory of a general quantum system interacting with a linear dissipative system_ , Ann. Phys. 24, 118 (1963).
  * (5) A. O. Caldeira and A. J. Leggett, _Path integral approach to quantum Brownian motion_ , Physica 121 A, 587 (1983).
  * (6) M. Srednicki, _Chaos and Quantum Thermalization_. Phys. Rev. E. 50, 888 (1994).
  * (7) W. H. Zurek, _Decoherence, einselection, and the quantum origins of the classical_ , Rev. Mod. Phys. 75, 715 (2003).
  * (8) J. Gemmer, M. Michel, and G. Mahler, _Quantum Thermodynamics: Emergence of Thermodynamic Behavior within Composite Quantum Systems_ (2nd Ed. Springer, Berlin, 2009).
  * (9) C. Jarzynski, _Equalities and Inequalities: Irreversibility and the Second Law of Thermodynamics at the Nanoscale_ , Ann. Rev. Condens. Matter Phys. 2, 329 (2011).
  * (10) W. M. Zhang, P. Y. Lo, H. N. Xiong, M. W. Y. Tu, and F. Nori, _General non-Markovian dynamics of open quantum systems_ , Phys. Rev. Lett. 109, 170402 (2012).
  * (11) R. Kosloff, _Quantum Thermodynamics: A Dynamical Viewpoint_ , Entropy 15, 2100 (2013).
  * (12) H. N. Xiong, P. Y. Lo, W. M. Zhang, D. H. Feng, and F. Nori, _Non-Markovian Complexity in the Quantum-to-Classical Transition_ , Sci. Rep. 5, 13353 (2015).
  * (13) R. Nandkishore, D. A. Huse, D. A. Abanin, M. Serbyn and Z. Papić, _Many-Body Localization and Thermalization in Quantum Statistical Mechanics_. Ann. Rev. of Condens. Matter Phys. 6, 15 (2015).
  * (14) J. Millen and A. Xuereb, _Perspective on quantum thermodynamics_ , New J. Phys. 18, 011002 (2016).
  * (15) J. Anders and M. Esposito, _Focus on quantum thermodynamics_ , New J. Phys. 19, 010201 (2017).
  * (16) F. Binder, L. A. Correa, C. Gogolin, J. Anders, and G. Adesso, _Thermodynamics in the Quantum Regime_ (Springer, Cham, Switzerland, 2018).
  * (17) S. Deffner and S. Campbell, _Quantum Thermodynamics: An Introduction to the Thermodynamics of Quantum Information_ (Morgan & Claypool, San Rafael, CA, 2019).
  * (18) F. L. Xiong and W. M. Zhang, _Exact dynamics and thermalization of open quantum systems coupled to reservoirs through particle exchanges_ , Phys. Rev. A 102, 022215 (2020).
  * (19) P. Talkner and P. Hänggi, _Colloquium: Statistical mechanics and thermodynamics at strong coupling: Quantum and classical_ , Rev. Mod. Phys. 92, 041002 (2020).
  * (20) A. E. Allahverdyan and Th. M. Nieuwenhuizen, _Extraction of work from a single thermal bath in the quantum regime_ , Phys. Rev. Lett. 85, 1799 (2000)
  * (21) M. O. Scully et al. _Extracting work from a single heat bath via vanishing quantum coherence_ , Science 299, 862 (2003)
  * (22) M. Esposito, R. Kawai, K. Lindenberg, and C. Van den Broeck, _Efficiency at maximum power of low-dissipation Carnot engines_ , Phys. Rev. Lett. 105, 150603 (2010)
  * (23) M. O. Scully, et al. _Quantum heat engine power can be increased by noise-induced coherence_ , PNAS 108, 15097 (2011).
  * (24) S. Trotzky, et al. _Probing the relaxation towards equilibrium in an isolated strongly correlated one-dimensional Bose gas_ , Nat. Phys. 8, 325 (2012).
  * (25) S. Jezouin, et al. _Quantum limit of heat flow across a single electronic channel_ , Science 342, 601 (2013).
  * (26) K. Zhang, F. Bariani, and P. Meystre, _Quantum optomechanical heat engine_ , Phys. Rev. Lett. 112, 150602 (2014).
  * (27) C. Bergenfeldt, P. Samuelsson, B. Sothmann, C. Flindt, and M. Büttiker, _Hybrid microwave-cavity heat engine_ , Phys. Rev. Lett. 112, 076803 (2014).
  * (28) T. Langen, et al., _Experimental observation of a generalized Gibbs ensemble_ , Science 348, 207 (2015).
  * (29) J. P. Pekola, _Towards quantum thermodynamics in electronic circuits_ , Nat. Phys. 11, 118 (2015).
  * (30) A. M. Kaufman, et al. _Quantum thermalization through entanglement in an isolated many-body system_ , Science 353, 794 (2016).
  * (31) D. Gelbwaser-Klimovsky, W. Niedenzu, and G. Kurizki, _Thermodynamics of Quantum Systems Under Dynamical Control_ , Advances in AMO Physics, 64, 329 (2015).
  * (32) J. Roßnagel, et al. _A single-atom heat engine_. Science 352, 325 (2016).
  * (33) S. Vinjanampathy and J. Anders, _Quantum thermodynamics_ , Contemporary Physics 57, 545 (2016)
  * (34) M. A. Ochoa, N. Zimbovskaya, and A. Nitzan, _Quantum thermodynamics for driven dissipative bosonic systems_ , Phys. Rev. B 97, 085434 (2018).
  * (35) M. Campisi, P. Talkner, and P. Hänggi, _Thermodynamics and fluctuation theorems for a strongly coupled open quantum system: an exactly solvable case_ , J. Phys. A: Math. Theor. 42, 392002 (2009).
  * (36) Y. Subas, C. H. Fleming, J. M. Taylor, and B. L. Hu _Equilibrium states of open quantum systems in the strong coupling regime_ , Phys. Rev. E 86, 061132 (2012).
  * (37) M. Esposito, M. A. Ochoa, and M. Galperin, _On the nature of heat in strongly coupled open quantum systems_ , Phys. Rev. B 92, 235440 (2015).
  * (38) U. Seifert, _First and second law of thermodynamics at strong coupling_ , Phys. Rev. Lett. 116, 020601 (2016).
  * (39) M. Carrega, P. Solinas, M. Sassetti, and U. Weiss, _Energy exchange in driven open quantum systems at strong coupling_ , Phys. Rev. Lett. 116, 240403 (2016).
  * (40) M. A. Ochoa, A. Bruch, and A. Nitzan, _Energy distribution and local fluctuations in strongly coupled open quantum systems: the extended resonant level model_ , Phys. Rev. B 94, 035420 (2016).
  * (41) C. Jarzynski, _Stochastic and macroscopic thermodynamics of strongly coupled systems_ , Phys. Rev. X 7, 011008 (2017).
  * (42) S. Marcantoni, S. Alipour, F. Benatti, R. Floreanini, and A. T. Rezakhani, _Entropy production and non-Markovian dynamical maps_ , Sci. Rep. 7, 12447 (2017).
  * (43) A. Bruch, C. Lewenkopf, and F. von Oppen, _Landauer-Büttiker Approach to strongly coupled quantum thermodynamics: Inside-outside duality of entropy evolution_ , Phys. Rev. Lett. 120, 107701 (2018).
  * (44) M. Perarnau-Llobet, H. Wilming, A. Riera, R. Gallego, and J. Eisert, _Strong coupling corrections in quantum thermodynamics_ , Phys. Rev. Lett. 120, 120602 (2018).
  * (45) H. J. D. Miller and J. Anders _Energy-temperature uncertainty relation in quantum thermodynamics_ , Nat. Communications 9, 2203 (2018).
  * (46) J.-T. Hsiang, and B.-L. Hu, _Quantum thermodynamics at strong coupling: operator thermodynamic functions and relations_ , Entropy 20, 423 (2018).
  * (47) P. Strasberg, _Repeated interactions and quantum stochastic thermodynamics at strong coupling_ , Phys. Rev. Lett. 123,180604 (2019).
  * (48) Á. Rivas, _Strong coupling thermodynamics of open quantum systems_ , Phys. Rev. Lett. 124, 160601 (2020).
  * (49) D. Newman, F. Mintert, and A. Nazir, _Quantum limit to nonequilibrium heat-engine performance imposed by strong system-reservoir coupling_ , Phys. Rev. E 101, 052129 (2020).
  * (50) M. M. Ali, W. M. Huang, and W. M. Zhang, _Quantum thermodynamics of single particle systems_ , Sci. Rep. 10, 13500 (2020).
  * (51) M. Esposito, K. Lindberg, and C. van den Broek, _Entropy production as correlation between system and reservoir_ , New J. Phys. 12, 013013 (2010).
  * (52) F. C. Binder, S. Vinjanampathy, K. Modi, and J. Goold, _Quantum thermodynamics of general quantum processes_ , Phys. Rev. E 91, 032119 (2015).
  * (53) S. Alipour, F. Benatti, F. Bakhshinezhad, M. Afsary, S. Marcantoni, and A. T. Rezakhani, _Correlations in quantum thermodynamics: Heat, work, and entropy production_ , Sci. Rep. 6, 35568 (2016).
  * (54) P. Strasberg, G. Schaller, T. Brandes, and M. Esposito, _Quantum and Information Thermodynamics: A Unifying Framework Based on Repeated Interactions_ , Phys. Rev. X 7, 021003 (2017).
  * (55) M. W. Y. Tu and W. M. Zhang, _Non-Markovian decoherence theory for a double-dot charge qubit_ , Phys. Rev. B 78, 235311 (2008).
  * (56) J. Jin, M. W. Y. Tu, W. M. Zhang, and Y. J. Yan _Non-equilibrium quantum theory for nanodevices based on the Feynman-Vernon influence functional_ , New. J. Phys. 12, 083013 (2010).
  * (57) C. U Lei and W. M. Zhang, _A quantum photonic dissipative transport theory_ , Ann. Phys. 327, 1408 (2012).
  * (58) P. Y. Yang, C. Y. Lin, and W. M. Zhang, _Master equation approach to transient quantum transport in nanostructures incorporating initial correlations_ , Phys. Rev. B 92, 165403 (2015).
  * (59) P. Y. Yang and W. M. Zhang, _Master equation approach to transient quantum transport in nanostructures_ , Front. Phys. 12, 127204 (2017)
  * (60) H. L. Lai, P. Y. Yang, Y. W. Huang and W. M. Zhang, _Exact master equation and non-Markovian decoherence dynamics of Majorana zero modes under gate-induced charge fluctuations_ , Phys. Rev. B 97, 054508 (2018).
  * (61) W. M. Zhang, _Exact master equation and general non-Markovian dynamics in open quantum systems_ , Eur. Phys. J. Spec. Top. 227, 1849 (2019).
  * (62) Y. W. Huang, P. Y. Yang, and W. M. Zhang, _Quantum theory of dissipative topological systems_ , Phys. Rev. B 102, 165116 (2020).
  * (63) W. M. Zhang, D. H. Feng, and R. Gilmore, _Coherent States: Theory and some applications_ , _Rev. Mod. Phys_. 62, 867 (1990).
  * (64) A. O. Caldeira and A. J. Leggett, _Quantum Tunnelling in a Dissipative System_ , Ann. Phys. 149, 374 (1983).
  * (65) N. Nagaosa, _Quantum Field Theory in Strongly Correlated Electronic Systems_ , (Springer-Verlag, Berlin, 2010).
  * (66) U. Fano, _Effects of Configuration Interaction on Intensities and Phase Shifts_ , Phys. Rev. 124, 1866 (1961).
  * (67) S. Putz, D. O. Krimer, R. Amsüss, A. Valookaran, T. Nöbauer, J. Schmiedmayer, S. Rotter, and J. Maje, _Protecting a spin ensemble against decoherence in the strong-coupling regime of cavity QED_ , Nat. Phys. 10, 720 (2014).
  * (68) K. T. Chiang and W. M. Zhang, _Non-Markovian decoherence dynamics of strong-coupling hybrid quantum systems: A master equation approach_ , Phys. Rev. A 103, 013714 (2021).
  * (69) A. E. Miroshnichenko, S. Flach, and Y. S. Kivshar, _Fano resonances in nanoscale structures_ , Rev. Mod. Phys. 82, 2257 (2010).
  * (70) J. von Neumann, _Mathematische Grundlagen der Quantenmechanik._ (Springer, Berlin, 1932), also _Mathematical foundations of quantum mechanics_ , translated by R. T. Beyer, (Princeton university press, 1955).
  * (71) G. Lindblad, _On the generators of quantum dynamical semigroups_ , Commun. Math. Phys. 48, 119 (1976).
  * (72) V. Gorini, A. Kossakowski, and E. C. G. Sudarshan, _Completely positive dynamical semigroups of n-level systems_ , J. Math. Phys. 17, 821 (1976).
  * (73) P. Breuer and F. Petruccione, _The theory of open quantum systems_ , (Clarendon Press, Oxford, 2006).
  * (74) J. Paavola, J. Piilo, K.-A. Suominen, and S. Maniscalco, _Environment-dependent dissipation in quantum Brownian motion_ , Phys. Rev. A 79, 052120 (2009).
  * (75) A. Rajesh and M. Bandyopadhyay, _Quantum Brownian magneto-oscillator: Role of environmental spectrum and external magnetic field in decoherence and decay processes_ , Phys. Rev. A 92, 012105 (2015).
  * (76) J. Kolodyński, J. B. Brask, M. Perarnau-Llobet, and B. Bylicka, _Adding dynamical generators in quantum master equations_ , Phys. Rev. A 97, 062124 (2018).
  * (77) M. H. Wu, C. U. Lei, W. M. Zhang, and H. N. Xiong, _Non-Markovian dynamics of a microcavity coupled to a waveguide in photonic crystals_ , Optical Express, 18, 18407 (2010).
  * (78) H. N. Xiong, W. M. Zhang, X. Wang, and M. H. Wu, _Exact non-Markovian cavity dynamics strongly coupled to a reservoir_ , Phys. Rev. A 82, 012105 (2010)
  * (79) C. U Lei and W. M. Zhang, _Decoherence suppression of open quantum systems through a strong coupling to non-Markovian reservoirs_ , Phys. Rev. A 84, 052116 (2011).
  * (80) When there exist localized modes, the reduced density matrix is always dependent on the initial state of the system so that thermalization cannot be reached, as shown in Refs. Xiong2015 ; Xiong2020 ; Ali2020a .
  * (81) H. T. Tan and W. M. Zhang, _Non-Markovian dynamics of an open quantum system with initial system-reservoir correlations: A nanocavity coupled to a coupled-resonator optical waveguide_ , Phys. Rev. A 83, 032102 (2011).
  * (82) A. J. Leggett, S. Chakravarty, A. T. Dorsey, M. P. A. Fisher, A. Garg, and W. Zwerger, _Dynamics of the dissipative two-state system_ , Rev. Mod. Phys. 59, 1 (1987).
  * (83) H. B. Callen, _Thermodynamics and an Introduction to Thermostatistics_ (Wiley, New York, 1985).
  * (84) In Ref. Ali2020a , the renormalization of the Hamiltonian has not been taken into account, where the heat and work cannot be properly defined. After included the renormalization of the internal energy, the quantum thermodynamics can be completed, as shown in the present paper. The results in Ref. Ali2020a  are quantitatively modified after the renormalized Hamiltonian is included.
  * (85) M. W. Zemansky, and R. Dittman, _Heat and thermodynamics: an intermediate textbook_ (New York, McGraw-Hill, 1997), pp.329.
  * (86) D. J. Thouless, _The Quantum Mechanics of Many-Body Systems_ (Academic Press, New York, 1972).
  * (87) H. Haug and A.-P. Jauho, _Quantum Kinetics in Trans- port and Optics of Semiconductors_ , 2nd Ed. (Springer Series in Solid-State Sciences 123, Berlin, 2007).
  * (88) C. N. Yang, _Concept of Off-Diagonal Long-Range Order and the Quantum Phases of Liquid He and of Superconductors_ , Rev. Mod. Phys. 34, 694 (1962).
  * (89) M. W. Y. Tu, W. M. Zhang, and J. Jin, _Intrinsic coherence dynamics and phase localization in nanoscale Aharonov-Bohm interferometers_ , Phys. Rev. B 83, 115318 (2011).
  * (90) P. Y. Yang and W. M. Zhang, _Buildup of Fano resonances in the time domain in a double quantum dot Aharonov-Bohm interferometer_ , Phys. Rev. B 97, 054301 (2018).
  * (91) Y. Meir, N. S. Wingreen, and P. A. Lee, _Low-temperature transport through a quantum dot: The Anderson model out of equilibrium_ , Phys. Rev. Lett. 70, 2601 (1993).
  * (92) L.P. Kadanoff, and G. Baym, _Quantum Statistical Mechanics_ (Benjamin, New York, 1962).
  * (93) L.V. Keldysh, _Diagram Technique for Nonequilibrium Processes_ , Sov. Phys. JETP 20, 1018 (1965)
  * (94) W. M. Zhang, _A note on the general nonequilibrium theory of many-body interacting open quantum systems_ (unpublished).
  * (95) M. E. Peskin and D. V. Schroeder, _An Introduction to Quantum Field Theory_ (Addison Wesley, New York, 1995).
  * (96) Usually one takes the Coulomb gauge $\nabla\cdot\bm{A}=0$, then the Hamiltonian should contain a Coulomb interaction between electrons: $U\\!\\!=\\!\\!\frac{e^{2}}{4\pi\epsilon_{0}}\\!\\!\int\\!\\!d^{3}rd^{3}r^{\prime}\frac{\varrho(\bm{r})\varrho(\bm{r}^{\prime})}{|\bm{r}-\bm{r}^{\prime}|}$, where $\varrho(\bm{r})$ is the electron charge density at position $\bm{r}$. This is the origin of the second term in Eq. (66).
  * (97) W. M. Zhang, and L. Wilets, _A Transport Theory of Relativistic Heavy-Ion Collisions with Chiral Symmetry_ , Phys. Rev. C 45, 1900 (1992).


\- order (Type: string): The sort order. Possible values are ‘asc’ or ‘des’.
Returns:
\- result (Type: object): The updated worksheet data or an error message.
Name: merge_cells(start_position, end_position)
Description: Merge cells in sheet
Parameters:
\- start_position (Type: string): Starting cell position(top left) in A1
annotation.
\- end_position (Type: string): Ending cell position(bottom right) in A1
annotation.
Returns:
\- result (Type: object): The updated worksheet data or an error message.
Table 31: Prompt details for Sheet (Part 3/4). Name: update_note(position,
content)
---
Description: Update a note in a certain cell
Parameters:
\- position (Type: string): cell position in A1 annotation.
\- content (Type: string): The text note to insert.
Returns:
\- result (Type: string): The updated note or an error message.
Name: get_all_values()
Description: Display all cell values in current sheet
Returns:
\- result (Type: array[array[Any]]): Return all cell values or an error
message.
Name: get_range_values(start_position, end_position)
Description: Returns a list of cell data from a specified range.
Parameters:
\- start_position (Type: string): Starting cell position in A1 annotation.
\- end_position (Type: string): Ending cell position in A1 annotation.
Returns:
\- result (Type: array[array[Any]]): List of cell data from the specified
range or an error message.
Name: get_cell_value(position)
Description: Get the value of a specific cell
Parameters:
\- position (Type: string): Cell position in A1 annotation.
Returns:
\- result : Cell value or an error message.
Name: get_value_by_formula(start_position, end_position, position_list,
operator)
Description: Calculate a value applying formulas on specified cells. Note:
Either specify position_list or start_position and end_position.
Parameters:
\- start_position (Type: string): The starting position of the range. Default:
‘B1’.
\- end_position (Type: string): The ending position of the range. Default:
‘D2’.
\- position_list (Type: array[string]): A list of cell positions in A1
notation.
\- operator (Type: string): The operator to be applied on selected cells.
Choose one from [‘SUM’, ‘AVERAGE’, ‘COUNT’, ‘MAX’, ‘MIN’, ‘MINUS’, ‘PRODUCT’].
Returns:
\- result (Type: string): Calculated result or an error message.
Name: filter_cells(query, in_row, in_column)
Description: Find all cells matching the query, return all cells’ position.
Parameters:
\- query (Type: [‘string’, ‘re.RegexObject’]): A string to match or compiled
regular expression.
\- in_row (Type: [‘integer’, ‘None’]): Row number to scope the search. Default
is all rows
\- in_column (Type: [‘integer’, ‘None’]): Column number to scope the search.
Default is all columns
Returns:
\- result (Type: array[string]): List of cell addresses that match the query
or an error message.
Name: get_note(position)
Description: Get the note at the certain cell, or return empty string if the
cell does not have a note.
Parameters:
\- position (Type: string): Cell position in A1 annotation.
Returns:
\- result (Type: string): Note content or an error message.
Table 32: Prompt details for Sheet (Part 4/4). Name: finish()
---
Description: Return an answer and finish the task
Returns:
\- result (Type: array[array[Any]]): Return all cell values or an error
message.
Instruction:
Now new trial starts. You should perform actions to accomplish the goal:
Product Update: The table in “Sheet1” contains the product inventory
information, and [[‘Product’, ‘Today Sold’], [‘beef’, ‘5’], [‘pork’, ‘2’],
[‘chicken’, ‘8’], [‘lamb’, ‘12’], [‘duck’, ‘3’], [‘fish’, ‘23’], [‘shrimp’,
‘21’], [‘salmon’, ‘12’], [‘apple’, ‘100’], [‘banana’, ‘287’], [‘orange’,
‘234’], [‘carrot’, ‘12’]] is today’s sales data. Please update the product
information in “Sheet1” in time and then sort by “Quantity” in descending
order. Give me one action.