# Geneverse: A collection of Open-source Multimodal Large Language Models for
Genomic and Proteomic Research

Tianyu Liu
Yale University
New Haven, CT 06511
<EMAIL_ADDRESS>
&Yijia Xiao
UCLA
Los Angeles, CA 90095
<EMAIL_ADDRESS>
&Xiao Luo
UCLA
Los Angeles, CA 90095
<EMAIL_ADDRESS>
&Hua Xu
Yale University
New Haven, CT 06511
<EMAIL_ADDRESS>
&W. Jim Zheng
UT Health
Houston, TX 77030
<EMAIL_ADDRESS>
&Hongyu Zhao
Yale University
New Haven, CT 06511
<EMAIL_ADDRESS>

###### Abstract

The applications of large language models (LLMs) are promising for biomedical
and healthcare research. Despite the availability of open-source LLMs trained
using a wide range of biomedical data, current research on the applications of
LLMs to genomics and proteomics is still limited. To fill this gap, we propose
a collection of finetuned LLMs and multimodal LLMs (MLLMs), known as
Geneverse, for three novel tasks in genomic and proteomic research. The models
in Geneverse are trained and evaluated based on domain-specific datasets, and
we use advanced parameter-efficient finetuning techniques to achieve the model
adaptation for tasks including the generation of descriptions for gene
functions, protein function inference from its structure, and marker gene
selection from spatial transcriptomic data. We demonstrate that adapted LLMs
and MLLMs perform well for these tasks and may outperform closed-source large-
scale models based on our evaluations focusing on both truthfulness and
structural correctness. All of the training strategies and base models we used
are freely accessible.

##  1 Introduction

Foundation Models (FMs) [7] have attracted great attention recently because of
their superb functionality for handling multiple downstream tasks by adaption,
especially in the landscape of Natural Language Processing (NLP) [60, 65]. In
this area, we focus on one specific type of FMs, known as generative Large
Language Models (LLMs) [65]. LLMs have demonstrated their strong ability to
perform various tasks and domains in NLP [7, 65], and they have revolutionized
the approaches of productivity improvement with machine intelligence [10] as
well. Powerful LLMs including ChatGPT [1] and Bard [49] exhibit the ability to
understand and reason as a human, and they are capable of complicated problem-
solving tasks. However, since these well-trained models are both closed-
source, the emergence of open-source LLM development is also important.
Powerful open-source LLMs including LLaMA2 [52] and Mistral [21] have already
become the base models of many LLMs for scientific research [24, 57].

Beyond the success of LLMs, we are also interested in extending the input of
LLMs with datasets from different modalities, including images [32] and
scientific graphs [56]. Such extensions of LLMs are meaningful because
different modalities offer different aspects of a good representation [59,
20]. Therefore, Multimodal LLMs (MLLMs) are emerging as a new research hotspot
[62]. Models like Gemini (closed-source) [49], GPT-4 (closed-source) [1] and
LLaVA (open-source) [32] allow the combination of images and texts as input,
thus making MLLMs understand non-text data and generate responses accordingly
[34]. The performance of these models demonstrates that treating images as a
special language can extend the capabilities of LLMs to solve more tasks
including image-based conversation, image description and complex reasoning
[27, 38].

Although we have seen successful examples related to the development of LLMs
and MLLMs, the application of LLMs towards healthcare and biomedical research
presents both challenges and opportunities [17, 66]. First, different from
other areas, biomedical research highlights strict requirements for the
factualness or truthfulness of model outputs [24], as well as validation of
outputs by clinical or biological experiments [51]. When LLMs are used to
assist physicians in the diagnosis of patients, an incorrect diagnosis of a
patient’s condition can dramatically affect the process and even cause the
patient to miss the optimal time for treatment. Second, the output of LLMs
should also comply with established scientific knowledge [3]. Third, the
performance of finetuned open-source LLMs also has not outperformed large or
proprietary models [24] for such research. Regarding the opportunities, most
of the LLMs for biomedical research focus on precision healthcare [57] and are
finetuned with information from the corpus of biomedical articles [36], while
the LLMs or MLLMs for genomic or proteomic research are not well-discussed and
remain to be explored. Therefore, we aim to contribute such FMs to accelerate
research related to Central Dogma [47] at the molecular level [9].

To fill the gap in this area, we propose a collection of LLMs and MLLMs, named
as Geneverse 111The name is inspired by the combination of GeneGPT and
universe., for solving genomic and proteomic problems. Our collection mainly
focuses on the functions of genes and proteins, as well as their interactions.
Overall, our contributions can be summarized as follows:

1\. By performing a comprehensive benchmarking analysis for different base
models finetuned with the gene functional description data, we created a set
of LLMs with different scales for genomic research. Our benchmark results
included evaluations for both the correctness of structure and the
truthfulness. We selected the best base models and created an Artificial
Intelligence (AI) assistant.

2\. By leveraging the training datasets from both official databases as well
as synthetic data from advanced LLMs, we proposed a new adaptation framework
for biomedical research and built a set of powerful LLMs based on the best
base model for generating the summary of gene functions and protein functions
after benchmarking.

3\. We introduced MLLMs in genomic and proteomic research by incorporating the
structural information from proteins and expression information from genes, as
well as their functional descriptions, as input. We then finetuned MLLMs for
two tasks including protein identification and marker gene identification.

4\. We explored the possibility of handling more tasks with finetuned FMs as
well as improving the performances of models with more advanced techniques,
which offered guidance for future biomedical AI research.

Figure 1: The landscape of Geneverse. To generate LLMs for genomic and
proteomic analysis, we incorporate the training datasets from rephrased
descriptions for gene functions as well as synthetic descriptions from GPT
3.5. We then adjust the base model with different strategies and select the
best candidate. To generate MLLMs for genomic and proteomic analysis, we
incorporate the training datasets from known databases, including both
descriptions and corresponding images. We then finetune the base model with
different strategies and select the best candidate. The logo of Geneverse is
generated by DALLE [40].

##  2 Related Work

###  2.1 LLMs and MLLMs for general purpose

General purpose models including GPT-4 [1], Gemini [49] and Claude [13] have
demonstrated exceptional performances across various tasks in language
processing, under both zero-shot and few-shots learning frameworks. However,
all of the models above are closed-source. Meanwhile, there are also
attractive open-source LLMs including LLaMA series [52, 39], Mistral [21] and
Gemma [15], which also demonstrate comparable performances with proprietary
models under certain tasks. Furthermore, the capacities of LLMs with
multimodal information (e.g., audio, images, and DNA sequences, etc.) as
inputs/outputs, known as MLLMs, have also shown remarkable performances under
many multi-modal tasks. However, the models that serve as generalists often
fail to answer certain domain-specific questions, for example, in the
biological domain.

###  2.2 LLMs and MLLMs for biomedical research

The adaptation of LLMs and MLLMs to address tasks in biology and medicine also
attracts research communities. This topic studies how to leverage these
models’ abilities to address biomedical problems. Encoder-based models
including BioBERT [25] and MedBERT [45] have been designed for answering
biomedical questions. Recently, decoder-based models including BioGPT [35],
BioMedLM [6], ClinicalGPT [55], BioMedGPT [36], MEDITRON [11], BioMistral [24]
and Med-PALM 2 [44] are pre-trained or fine-tuned with biomedical databases
and used for answering questions in genomics, proteomics, and clinical
informatics, etc. Other methods including GeneGPT [22] and scELMo [33] utilize
tool augmentation or prompt engineering to address similar questions without
extra training. By jointly modelling biomedical images and texts, methods
including LLaVA-Med [27] and Med-MLLM [31], have been designed to address
multimodal problems. However, some of the models limit the accessibility while
others do not show satisfactory performances in our targeted tasks. Therefore,
we still need a better foundation model for healthcare and biology research.

##  3 Methods

###  3.1 Problem statement

In this manuscript, we focus on an LLM or MLLM $\mathcal{M}$, which accepts
the prompt $P(T,I,m)$ as input, where $T$ represents the text description, $I$
represents the image and $m$ represents the given task. For an LLM, we set $I$
as an empty item. For a MLLM, we use both $T$ and $I$. For each prompt, we
have a corresponding model output as $O=\mathcal{M}(P(T,I,m))$. For a
validation dataset, we compare the difference between $O$ and the ground truth
information $G$ to evaluate model performance. The more similar $O$ and $G$
are (The higher their similarity score is), the better the model is.

###  3.2 Overview of Geneverse

Our model collection Geneverse is based on a group of pre-training open-source
foundation models including LLaMA2-7B, LLaMA2-13B, Mistral-7B, Gemma-7B [15],
LLaMAPro-8B [58], LLaMA3-8B [39] and LLaVA-7B. The first six models are
designed for one text-based task, while the last one is designed for two
image-text-based tasks. We finetuned these models based on both Parameter-
Efficient FineTuning (PEFT) technology and full-parameter finetuning
technology, and the finetuning setting is consistent with the Supervised
Instruct Finetuning (SIFT) approach. The PEFT technology we used is Low-rank
Adaption (LoRA) [19]. Our training datasets are constructed based on augmented
datasets from the National Center of Biotechnology Information (NCBI) [46] and
the generated synthetic datasets from GPT 3.5 with designed instructions. In
the inference process, we did not use the sampling mode and thus the outputs
were not affected by random seeds. The training mechanism is summarized in
Figure 1. We include the details of problem ettings and training design in the
following subsections.

###  3.3 Supervised finetuning process of LLM

To finetune a LLM as an AI assistant for genomic and proteomic information
queries, we prepare the training dataset with both real data and synthetic
data with a data augmentation policy. To collect the real data, we use scripts
to access the gene information from NCBI pages and rephrase the content of
each gene with GPT 3.5 to normal sequence. To collect the synthetic data, we
use the same prompt to ask GPT 3.5 to generate responses for protein-encoding
genes. By combining two different datasets together, we finalize our
finetuning datasets. During the finetuning process, we search the optimized
hyper-parameter settings and follow the default evaluation settings of
Stanford-alpaca [48] and Alpaca-lora [54].

###  3.4 Supervised finetuning process of MLLM

To finetune a MLLM as a multimodal AI assistant for genomic and proteomic
information query, we prepare the training datasets for two representative
tasks in biology. Learning a protein’s function based on its structure (the
extra modality) and identifying marker genes for certain cell types based on
the locations of cells (the extra modality) are difficult but meaningful [5,
64]. Therefore, we consider the protein function classification task (known as
the protein task) for protein structures and the maker genes identification
task for spatial transcriptomics (known as the marker gene task). For the
protein task, we downloaded the protein structure images from the Deepmind
Alphafold2 website [23] with the fixed capture angle. We load the image
information using Pymol [63] and construct the instruction tuning dataset. We
then finetune LLaVA with LoRA based on our datasets. We search the optimized
hyper-parameter settings and use the default evaluation settings from LLaVA
v1.5. For the marker gene task, the pipeline is the same but we utilize
different image datasets [30] as well as different instructions.

###  3.5 Post-processing steps

To tackle the two problems of the outputs of LLMs, known as the incorrect
inference of numerical features of genes and redundant descriptions (also
known as degeneration) of genes, we design two approaches as post-processing
methods to improve the quality of model outputs.

Our post-processing methods are based on the idea of tool-augmented design
[28]. To handle the first problem, we collect the gene ID information from
public databases [46] and replace the information in the outputs of LLMs with
the numerical information from the database. To handle the second problem,
since the redundant information is related to the aliases of genes, we delete
the content related to aliases in the model outputs and insert the aliases
based on the correct information from our database.

###  3.6 Evaluation

In the evaluation process, we consider two different aspects for evaluation,
known as grader for truthfulness and grader for structural correctness. The
former grader works for evaluating whether the outputs from LLMs or MLLMs
match the fact, while the latter grader works for evaluating whether the
outputs from LLMs or MLLMs match the structure requirements from the prompt.
Higher scores mean better model performance for each grader.

We first consider the grader for the truthfulness. For the gene description
task, we assign 1 for the outputs of LLMs which match the major gene function
and gene name, while 0 otherwise. For the protein task, we compute the length
of the largest common string between the output of MLLMs and the correct
result and divide it by the length of the correct result as the metric. For
the marker gene task, we assign 1 for the outputs of MLLMs which match the
ground truth marker gene-cell type relation, while 0 otherwise. We finally
average the scores of different samples as the final score.

We then consider the grader for structural correctness. For all of the tasks
we evaluate, we assign 1 for the outputs if they match the structural
requirements from the prompt, and 0 otherwise. We finally average the scores
of different samples as the final score.

In order to validate our proposed metric, we also considered the use of
BiLingual Evaluation Understudy (BLEU) [41] and ROUGE$k$ (k Grams Recall-
Oriented Understudy for Gisting Evaluation) [29] scores as additional
evaluations. BLEU score is based on the overlap value of $n-grams$ for two
strings. ROUGE$k$ score considers the overlap of $k$ grams between the
reference text and generated text, and here $k=1$. In order to evaluate the
quality of embeddings from generated texts, we computed the Normalized Mutual
Information (NMI) score [42] between Leiden clusters [53] and observed labels.
All metrics are in [0,1] and higher values mean better model performances.

##  4 Results

###  4.1 Benchmarking LLMs for summarizing of gene functions

We first performed benchmarking analysis by fixing the training datasets to
find the best base model and the best training strategy. Our task is to
generate the summary of gene functions based on the prompt only containing the
task description. Out of $\sim$ 20,000 genes [12] in total, we randomly
selected 20 genes referred from [22] and their observed descriptions to
construct the evaluation datasets. Since our prompt also contains requirements
for the structure of model outputs, we consider a novel evaluation of the
output structure as well as the truthfulness of the contents. The illustration
of evaluators is shown in Extended Data Figure 1. The criteria for computing
the accuracy are summarized in the Methods section. We used accuracy as a
metric to evaluate these two types of metrics. We also included the BLEU score
as an additional metric. The methods we included in this section for
benchmarking are LLaMA2-7B (LoRA), LLaMA2-7B (full), LLaMA2-7B (RAG),
LLaMA2-13B (LoRA), LLaMA2-13B (full), Mistral-7B (LoRA), Mistral-7B (RAG),
Gemma-7B (LoRA), Gemma-7B (RAG), LLaMAPro-8B (LoRA), LLaMAPro-8B (RAG),
LLaMA3-8B (LoRA) and LLaMA3-8B (RAG). Here RAG represents the retrieval-
augmented generation technology [26]. We also considered including state-of-
the-art models for general models and more biomedical-focused models,
including GPT 3.5 [8], GPT 3.5 (RAG), GPT 4 [1], BioMedLM [6], bioGPT [35],
GeneGPT [22], and BioMistral [24]. Details of baselines are included in
Appendix A. Our evaluations are shown in Table 1.

Model | Factual Score | Structural Score | Average | BLEU | ROUGE1 | Finetuning?
---|---|---|---|---|---|---
LLaMA2-7B (LoRA) | 0.850 | 1.000 | 0.925 | 0.395 | 0.580 | Y
LLaMA2-7B (full) | 0.900 | 0.900 | 0.900 | 0.345 | 0.546 | Y
LLaMA2-7B (RAG) | 0.650 | 0.800 | 0.725 | 0.155 | 0.324 | N
LLaMA2-13B (LoRA) | 0.900 | 1.000 | 0.950 | 0.374 | 0.600 | Y
LLaMA2-13B (RAG) | 0.600 | 1.000 | 0.800 | 0.196 | 0.358 | N
Mistral-7B (LoRA) | 0.950 | 1.000 | 0.975 | 0.349 | 0.592 | Y
Mistral-7B (RAG) | 0.650 | 0.050 | 0.350 | 0.127 | 0.269 | N
Gemma-7B (LoRA) | 0.950 | 1.000 | 0.975 | 0.425 | 0.613 | Y
Gemma-7B (RAG) | 0.350 | 1.000 | 0.675 | 0.073 | 0.183 | N
LLaMAPro-8B (LoRA) | 0.950 | 1.000 | 0.975 | 0.387 | 0.627 | Y
LLaMAPro-8B (RAG) | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | N
LLaMA3-8B (LoRA) | 0.900 | 1.000 | 0.950 | 0.324 | 0.541 | Y
LLaMA3-8B (RAG) | 0.250 | 0.400 | 0.325 | 0.011 | 0.196 | N
GPT 3.5 | 0.900 | 1.000 | 0.950 | 0.123 | 0.306 | N
GPT 3.5 (RAG) | 0.850 | 1.000 | 0.925 | 0.144 | 0.312 | N
GPT 4 | 0.850 | 1.000 | 0.925 | 0.118 | 0.309 | N
BioMedLM | 0.000 | 0.000 | 0.000 | 0.046 | 0.193 | N
bioGPT | 0.000 | 0.000 | 0.000 | 0.041 | 0.186 | N
GeneGPT | 0.750 | 1.000 | 0.875 | 0.141 | 0.304 | N
BioMistral | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | N

Table 1: Evaluations for the task: Gene Function Description. We boldfaced the
top three methods ranked by Average scores, BLEU scores and ROUGE1 scores.

Based on our evaluation results, we found that Geneverse based on finetuning
Mistral-7B, Gemma-7B, and LLaMAPro achieved the best results by averaging the
scores from both content level and structure level. The latter two models also
had better performances evaluated by BLEU and ROUGE1 scores. We computed the
Pearson correlation between BLEU (ROUGE1) scores and average scores, which
showed significant positive correlations (coefficient=0.76, p-value=9.5e-5 for
BLEU, and coefficient=0.80, p-value=1.6e-5 for ROUGE1), which lends support of
our proposed metric. We note that finetuned Mistral-7B outperformed finetuned
models from the LLaMA family with different scales, but also outperformed
large-scale general LLMs including GPT 3.5 and GPT 4. On the other hand, other
methods including bioGPT and BioMistral did not generate meaningful
descriptions for gene functions. Meanwhile, methods based on RAG also did not
satisfy our requirements and the generated outputs sometimes have multiple
paragraphs. Therefore, the structure score of Mistral-7B (RAG) is low. Our
conclusions also agree with recent research [61] challenging the capacity of
RAG in addressing biomedical problems. Therefore, we still need to finetune
models for domain-specific tasks.

Moreover, the model outputs under different cases all failed in summarizing
the numerical information of genes including Ensemble id [18] and HUGO gene
nomenclature committee (HGNC) id [43]. To tackle this problem, we have
developed a retrieval database to replace the incorrect information of model
output as an important post-processing step. By adding this component, our
generated summary is precise in describing both the functional and
identifiable information of genes. The details of the post-processing method
are included in the Methods section. The outputs of different models for
evaluations are in Supplementary File 1.

###  4.2 Leveraging the diverse of datasets to finetune a novel LLM

As shown in Figure 1, we also used the outputs generated by GPT-3.5 to enrich
the data diversity for model training. To ensure the correctness of training
datasets, we only focused on protein-encoding genes for the outputs of GPT
3.5. While for non-coding genes and pseudogenes, we only used the descriptions
from NCBI. There are two advantages brought by such a data augmentation
approach. The first advantage is that GPT 3.5 has specific knowledge of
certain genes, thus helping our AI assistant answer questions related to genes
that are not collected in the NCBI database. The other advantage is that GPT
3.5’s output is more like human language and has more comprehensive functional
information to enhance the quality of the model output. A comparison of
outputs for models trained with different datasets is shown in Extended Data
Figure 3. From this figure, we can find that introducing more data can help
LLMs generate more informative summaries for the gene GLI1 by reducing
redundant information. Details of model outputs are summarized in
Supplementary File 1.

To visualize the outputs of Geneverse after finetuning, we utilize the
embeddings layer from OpenAI to transfer the descriptions of genes into
numerical vectors (as gene embeddings) and visualize the gene embeddings using
Uniform Manifold Approximation and Projection (UMAP) [37]. In Figure 2 (a), we
label the genes using gene functional information from [50]. We can find that
genes with similar functions display shared patterns. Details of labels are
summarized in Extended Data Figure 4. Furthermore, the Gene Ontology
Enrichment Analysis (GOEA) [4, 2, 14] results are summarized in Extended Data
Figure 2, which include top 10 pathways ranked by $-log(\text{Adjusted
P-value})$ (P-value corrected by Bonferroni method) for the top 10 clusters
ranked by the number of genes. We find many pathways related to important
biological activities such as RNA transcription and metabolism. Therefore, our
sentences cover the similarities and differences of the genes’ functions.

Figure 2: UMAPs for the gene embeddings colored by gene functional
information. Panels (a)-(c) represent the outputs of LLMs trained based on
datasets from NCBI+GPT 3.5, NCBI only and GPT 3.5. We report the NMI score of
each embeddings followed by their sources.

###  4.3 Finetuning a MLLM for genomic and proteomic application

By incorporating the image information into the model training process, we can
construct domain-specific MLLMs for several downstream tasks. We designed two
tasks and demonstrated the potential of MLLMs for analyzing proteins and
spatial transcriptomics by finetuning LLaVA on these two novel tasks. The
first task is known as protein classification, which means we intend to
identify the specific protein with its image. The images of protein structure
for training and testing come from the databases of AlphaFold [23]. The second
task is known as spatial marker gene identification, which means that for a
gene $g$, we intend to know whether it is a maker gene of cell type $c$ based
on the spatial transcriptomic data. The spatial transcriptomic data for
training and testing come from [30] collected from human breast tissue. Both
tasks have not been discussed previously and are important for biomedical
research. As for baselines, we included LLaVA-7B, MoE-LLaVA, GPT-4, and
Gemini. Our finetuned version of LLaVA is known as LLaVA-7B (LoRA). Details of
baselines are included in Appendix A.

For the first task, we compared the three baseline models with our finetuned
model, and the results are summarized in Table 2. Based on this table, we
found that the baselines without finetuning cannot generate correct labels for
proteins from images. Moreover, the output of GPT-4 even does not follow the
structure requirements from our prompts, which questioned the contributions of
large-scale models for this task. However, if we finetuned LLaVA based on the
training dataset, the factual score increases while the high structural score
is also preserved. Therefore, leveraging domain-specific knowledge is an
important step to handle this classification problem. However, only partial
protein names were matched for every sample, thus the finetuning process still
needs further investigation for performance improvement.

Model | Factual score | Structural score | Average | Finetuning?
---|---|---|---|---
LLaVA-7B | 0 | 1 | 0.5 | N
MoE-LLaVA | 0 | 1 | 0.5 | N
GPT-4v | 0 | 0 | 0 | N
GPT-4o | 0 | 0 | 0 | N
Gemini | 0 | 0 | 0 | N
LLaVA-7B (LoRA) | 0.29 | 1 | 0.645 | Y

Table 2: Evaluations for the task: Protein Classification.

For the second task, we compared the three baseline models with our finetuned
model, and the results are summarized in Table 3. According to this table, the
results from GPT-4 are slightly better than random guesses, and all the
baseline models did not show a strong ability to identify marker genes for
different cell types given the spatial figures colored by gene expression
levels. However, LLaVA-7B and GPT-4 did not perform well in following the
structural requirements. Moreover, prior knowledge in the training process
might help in this prediction process. For example, all the methods predicted
gene CD4 as a marker gene of CD4 T cell, which matched the biological
knowledge, and the reason is from the existing information of the pre-training
text databases. Moreover, our finetuned LLaVA outperformed other models again
in the factual scores, which had 90% accuracy. In addition, the outputs of our
finetuned LLaVA also followed the requirements of structure existing in the
prompts. Therefore, finetuning an MLLM with image information could boost the
ability of models to learn about specific biological problems, especially for
tasks related to spatial transcriptomics.

Model | Factual Score | Structural Score | Average | Finetuning?
---|---|---|---|---
LLaVA-7B | 0.5 | 0.425 | 0.4625 | N
MoE-LLaVA | 0.5 | 1 | 0.75 | N
GPT-4v | 0.575 | 0.125 | 0.35 | N
GPT-4o | 0 | 0 | 0 | N
Gemini | 0.6 | 0.025 | 0.3125 | N
LLaVA-7B (LoRA) | 0.9 | 1 | 0.95 | Y

Table 3: Evaluations for the task: Marker genes.

###  4.4 Sensitivity analysis

In this section, we analyzed the factors that may affect the finetuning
process, including hyper-parameters and data ablation. The results are
summarized in Figure 3. We utilized default hyper parameters to finetune
different LLMs and MLLMs, and selected the best base model used in this
section for sensitivity analysis.

We first discuss the sensitivity of LLM training. For hyper-parameter tuning,
we considered tuning the number of epochs and cut-off length (it means we use
different lengths to truncate one sentence). For data ablation, we considered
three conditions: 1. only NCBI data (NCBI), 2. only GPT 3.5 data (GPT 3.5),
and 3. the combination of NCBI and GPT data (NCBI$+$GPT 3.5). Extended Data
Figure 5 shows the results of data ablation, which suggests that integrating
both NCBI and GPT 3.5 leads to the best performance of LLM in this task. The
LLM based on GPT 3.5 to finetune does not accurately summarise the function of
non-protein-encoding genes, as shown in Supplementary File 1. Moreover, LLMs
trained based on the combination of datasets also generated a better
representation. Comparing with outputs of LLMs trained based on the second
condition (shown in Figure 2 (b)) and the third condition (shown in Figure 2
(c)), embeddings from our current design also have the highest NMI score.
Figure 3 (a) shows the relation between epochs and model performance, which
suggests that more epochs lead to better model performance in the description
generation task. Figure 3 (b) shows the relation between cut-off length and
model performance, which suggests that cut-off length does not affect the
performance of LLM in this task.

We then discuss the sensitivity of MLLM training. Based on our observations in
the training process of LLMs, we only considered hyper-parameter tuning of the
number of epochs. The relation between the number of epochs and model
performance is illustrated in Figures 3 (c) and (d) for the protein task and
in Figure for the gene task. According to Figure 3 (c), increasing the number
of epochs leads to a model performance drop in the training process of MLLMs
for the protein task, reflected in the average score and the truthful score.
However, according to Figure 3 (d), increasing the number of epochs leads to
better model performance in the training process of MLLMs for the gene task,
reflected in the average score and the score for truthfulness. Therefore, task
variability affects the relationship between hyper-parameters and model
performance, possibly due to the complexity of biological data and questions.

We also illustrate the relation between top-k candidates (It means the number
of top samples retrieved from the reference data ranked by vector similarity)
and model performance for RAG-based LLMs in Extended Data Figure 6. This
figure shows that increasing top-k candidates leads to a model performance
drop, especially for the score of structure. The RAG-based LLM using the
largest top-k generated results free of structure, which was not suitable for
the application of a useful AI assistant. Moreover, we considered using
Lora$+$ [16] to replace LoRA in the training process, but the average score
did not change. The running time of the finetuning process is summarized in
Supplementary File 2.

Figure 3: Results of sensitivity analysis for the training of different
models. (a) The relation between the number of epochs and model performance of
LLMs. (b) The relation between the number of cut-off length and model
performance of LLMs. (c) The relation between the number of epochs and model
performance of MLLMs for the protein task. (d) The relation between the number
of epochs and model performance of MLLMs for the gene task.

##  5 Discussion

Foundation models, especially LLMs, are powerful tools for scientific
research. In this manuscript, we introduced Geneverse, a collection of LLMs
and MLLMs tailored for genomic and proteomic tasks. We compared different
strategies and base models for generating accurate function descriptions of
genes based on the adapted LLMs. Moreover, we also explored the applications
of MLLMs for a protein-level task and a gene-level task, as an example of
leveraging the multimodal information. The models in Geneverse demonstrated
better performance in the selected tasks than closed-source models, thus
advocating the contributions of open-source base models for scientific
research. We also discussed the factors affecting model adaptation, which
advanced this field by offering suggestions.

Our future work aims to increase the generation quality of outputs from
Geneverse with more advanced techniques. We will also investigate why RAG-
based LLMs fail and improve them. Moreover, we will collaborate with other
institutes to access more high-quality datasets for model adaptation. The
next-generation multimodal AI assistant for genomics and proteomics will come
soon.

##  6 Limitations

First, finetuning an FM or LLM requires a large amount of computing resources.
Based on our testing results, we need at least one H100 or equivalent GPU
cores to perform finetuning with LoRA. Secondly, to utilize methods based on
closed-source large models, including GPT series and GeneGPT, we need to use
the OpenAI API to call the method, thus we need additional resources for
deploying such a method. Finally, a good domain-specific FM should align with
the version of state-of-the-art base models. Therefore, we should keep our
model updated by aligning the best version of Geneverse with the best open-
source base model. Moreover, due to the limitation of computing resources, we
only considered 7B-level and 13B-level models in the current version. In the
future, we need to explore LLMs or MLLMs with larger scales. Finally, our
evaluations for the domain-specific FMs only focus on three tasks and more
tasks should be included in the future.

##  7 Ethics Statement

The users are solely responsible for the content they generate with models in
Geneverse, and there are no mechanisms in place for addressing harmful,
unfaithful, biased, and toxic content disclosure. Any modifications of the
models should be released under different version numbers to keep track of the
original models related to this manuscript.

The target of current Geneverse only serves for academic research. The users
cannot use it for other purposes. Finally, we are not responsible for any
effects of the use of the model.

##  8 Codes and Reproductivities

The codes we used in this manuscript can be found in
https://github.com/HelloWorldLTY/Geneverse. We used the MIT licenses. All of
the model weights can be found in the Huggingface Space. We will release our
training datasets after peer review. We used two NVIDIA H100 GPU cores for
model training, while the maximum system memory for training is 100 GB.

##  9 Contribution

T.L. proposed this study. T.L., Y.X. and X.L. designed the model. T.L. ran all
the experiments. T.L., X.L. and H.Z. wrote the manuscript. H.X., W.Z., and
H.Z. supervised this study.

## References

  * Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Aleksander et al. [2023] Suzi A Aleksander, James Balhoff, Seth Carbon, J Michael Cherry, Harold J Drabkin, Dustin Ebert, Marc Feuermann, Pascale Gaudet, Nomi L Harris, et al.  The gene ontology knowledgebase in 2023.  _Genetics_ , 224(1):iyad031, 2023.
  * Arora and Goyal [2023] Sanjeev Arora and Anirudh Goyal.  A theory for emergence of complex skills in language models.  _arXiv preprint arXiv:2307.15936_ , 2023.
  * Ashburner et al. [2000] Michael Ashburner, Catherine A Ball, Judith A Blake, David Botstein, Heather Butler, J Michael Cherry, Allan P Davis, Kara Dolinski, Selina S Dwight, Janan T Eppig, et al.  Gene ontology: tool for the unification of biology.  _Nature genetics_ , 25(1):25–29, 2000.
  * Bernhofer et al. [2021] Michael Bernhofer, Christian Dallago, Tim Karl, Venkata Satagopam, Michael Heinzinger, Maria Littmann, Tobias Olenyi, Jiajun Qiu, Konstantin Schütze, Guy Yachdav, et al.  Predictprotein-predicting protein structure and function for 29 years.  _Nucleic acids research_ , 49(W1):W535–W540, 2021.
  * Bolton et al. [2024] Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Roxana Daneshjou, Jonathan Frankle, Percy Liang, Michael Carbin, et al.  Biomedlm: A 2.7 b parameter language model trained on biomedical text.  _arXiv preprint arXiv:2403.18421_ , 2024.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Brownlee et al. [2023] Alexander EI Brownlee, James Callan, Karine Even-Mendoza, Alina Geiger, Carol Hanna, Justyna Petke, Federica Sarro, and Dominik Sobania.  Enhancing genetic improvement mutations using large language models.  In _International Symposium on Search Based Software Engineering_ , pages 153–159. Springer, 2023.
  * Byun et al. [2023] Courtni Byun, Piper Vasicek, and Kevin Seppi.  Dispensing with humans in human-computer interaction research.  In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_ , pages 1–26, 2023.
  * Chen et al. [2023] Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al.  Meditron-70b: Scaling medical pretraining for large language models.  _arXiv preprint arXiv:2311.16079_ , 2023.
  * Dolgin [2017] Elie Dolgin.  The most popular genes in the human genome.  _Nature_ , 551(7681):427–432, 2017.
  * Enis and Hopkins [2024] Maxim Enis and Mark Hopkins.  From llm to nmt: Advancing low-resource machine translation with claude, 2024.
  * Fang et al. [2023] Zhuoqing Fang, Xinyuan Liu, and Gary Peltz.  Gseapy: a comprehensive package for performing gene set enrichment analysis in python.  _Bioinformatics_ , 39(1):btac757, 2023.
  * Google [2024] Google.  Gemma.  https://huggingface.co/google/gemma-7b, 2024.
  * Hayou et al. [2024] Soufiane Hayou, Nikhil Ghosh, and Bin Yu.  Lora+: Efficient low rank adaptation of large models.  _arXiv preprint arXiv:2402.12354_ , 2024.
  * He et al. [2023] Kai He, Rui Mao, Qika Lin, Yucheng Ruan, Xiang Lan, Mengling Feng, and Erik Cambria.  A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics.  _arXiv preprint arXiv:2310.05694_ , 2023.
  * Howe et al. [2021] Kevin L Howe, Premanand Achuthan, James Allen, Jamie Allen, Jorge Alvarez-Jarreta, M Ridwan Amode, Irina M Armean, Andrey G Azov, Ruth Bennett, Jyothish Bhai, et al.  Ensembl 2021.  _Nucleic acids research_ , 49(D1):D884–D891, 2021.
  * Hu et al. [2022] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=nZeVKeeFYf9.
  * Huh et al. [2024] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola.  The platonic representation hypothesis.  _arXiv preprint arXiv:2405.07987_ , 2024.
  * Jiang et al. [2023] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_ , 2023.
  * Jin et al. [2024] Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu.  Genegpt: Augmenting large language models with domain tools for improved access to biomedical information.  _Bioinformatics_ , 40(2):btae075, 2024.
  * Jumper et al. [2021] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al.  Highly accurate protein structure prediction with alphafold.  _Nature_ , 596(7873):583–589, 2021.
  * Labrak et al. [2024] Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour.  Biomistral: A collection of open-source pretrained large language models for medical domains.  _arXiv preprint arXiv:2402.10373_ , 2024.
  * Lee et al. [2020] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240, 2020.
  * Lewis et al. [2020] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474, 2020.
  * Li et al. [2024] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao.  Llava-med: Training a large language-and-vision assistant for biomedicine in one day.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Li et al. [2023] Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li.  API-bank: A comprehensive benchmark for tool-augmented LLMs.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 3102–3116, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.187.  URL https://aclanthology.org/2023.emnlp-main.187.
  * Lin [2004] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81, 2004.
  * Lin et al. [2020] Yingxin Lin, Yue Cao, Hani Jieun Kim, Agus Salim, Terence P Speed, David M Lin, Pengyi Yang, and Jean Yee Hwa Yang.  scclassify: sample size estimation and multiscale classification of cells using single and multiple reference.  _Molecular systems biology_ , 16(6):e9389, 2020.
  * Liu et al. [2023a] Fenglin Liu, Tingting Zhu, Xian Wu, Bang Yang, Chenyu You, Chenyang Wang, Lei Lu, Zhangdaihong Liu, Yefeng Zheng, Xu Sun, et al.  A medical multimodal large language model for future pandemics.  _NPJ Digital Medicine_ , 6(1):226, 2023a.
  * Liu et al. [2024] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _Advances in neural information processing systems_ , 36, 2024.
  * Liu et al. [2023b] Tianyu Liu, Tianqi Chen, Wangjie Zheng, Xiao Luo, and Hongyu Zhao.  scelmo: Embeddings from language models are good learners for single-cell data analysis.  _bioRxiv_ , pages 2023–12, 2023b.
  * Liu et al. [2023c] Zhengliang Liu, Aoxiao Zhong, Yiwei Li, Longtao Yang, Chao Ju, Zihao Wu, Chong Ma, Peng Shu, Cheng Chen, Sekeun Kim, et al.  Radiology-gpt: A large language model for radiology.  _arXiv preprint arXiv:2306.08666_ , 2023c.
  * Luo et al. [2022] Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu.  Biogpt: generative pre-trained transformer for biomedical text generation and mining.  _Briefings in Bioinformatics_ , 23(6):bbac409, 2022.
  * Luo et al. [2023] Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie.  Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine, 2023.
  * McInnes et al. [2018] Leland McInnes, John Healy, and James Melville.  Umap: Uniform manifold approximation and projection for dimension reduction.  _arXiv preprint arXiv:1802.03426_ , 2018.
  * Meskó [2023] Bertalan Meskó.  The impact of multimodal large language models on health care’s future.  _Journal of Medical Internet Research_ , 25:e52865, 2023.
  * Meta [2024] Meta.  Llama3, 2024.
  * OpenAI [2024] OpenAI.  Dalle.  https://help.openai.com/en/articles/6640875-how-should-i-credit-dall-e-in-my-work, 2024.
  * Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318, 2002.
  * Pedregosa et al. [2011] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al.  Scikit-learn: Machine learning in python.  _the Journal of machine Learning research_ , 12:2825–2830, 2011.
  * Povey et al. [2001] Sue Povey, Ruth Lovering, Elspeth Bruford, Mathew Wright, Michael Lush, and Hester Wain.  The hugo gene nomenclature committee (hgnc).  _Human genetics_ , 109:678–680, 2001.
  * Qian et al. [2024] Jili Qian, Zhengyu Jin, Quan Zhang, Guoqing Cai, and Beichang Liu.  A liver cancer question-answering system based on next-generation intelligence and the large model med-palm 2.  _International Journal of Computer Science and Information Technology_ , 2(1):28–35, 2024.
  * Rasmy et al. [2021] Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi.  Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction.  _NPJ digital medicine_ , 4(1):86, 2021.
  * Sayers et al. [2021] Eric W Sayers, Jeffrey Beck, Evan E Bolton, Devon Bourexis, James R Brister, Kathi Canese, Donald C Comeau, Kathryn Funk, Sunghwan Kim, William Klimke, et al.  Database resources of the national center for biotechnology information.  _Nucleic acids research_ , 49(D1):D10, 2021.
  * Shapiro [2009] James A Shapiro.  Revisiting the central dogma in the 21st century.  _Annals of the New York Academy of Sciences_ , 1178(1):6–28, 2009.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Team et al. [2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Theodoris et al. [2023] Christina V Theodoris, Ling Xiao, Anant Chopra, Mark D Chaffin, Zeina R Al Sayed, Matthew C Hill, Helene Mantineo, Elizabeth M Brydon, Zexian Zeng, X Shirley Liu, et al.  Transfer learning enables predictions in network biology.  _Nature_ , 618(7965):616–624, 2023.
  * Thirunavukarasu et al. [2023] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting.  Large language models in medicine.  _Nature medicine_ , 29(8):1930–1940, 2023.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Traag et al. [2019] Vincent A Traag, Ludo Waltman, and Nees Jan Van Eck.  From louvain to leiden: guaranteeing well-connected communities.  _Scientific reports_ , 9(1):5233, 2019.
  * Wang [2023] Eric J. Wang.  Alpaca-lora.  https://github.com/tloen/alpaca-lora, 2023.
  * Wang et al. [2023] Guangyu Wang, Guoxing Yang, Zongxin Du, Longjun Fan, and Xiaohu Li.  Clinicalgpt: large language models finetuned with diverse medical data and comprehensive evaluation.  _arXiv preprint arXiv:2306.09968_ , 2023.
  * Wei et al. [2023] Lanning Wei, Zhiqiang He, Huan Zhao, and Quanming Yao.  Unleashing the power of graph learning through llm-based autonomous agents.  _arXiv preprint arXiv:2309.04565_ , 2023.
  * Wu et al. [2024a] Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang.  Pmc-llama: toward building open-source language models for medicine.  _Journal of the American Medical Informatics Association_ , page ocae045, 2024a.
  * Wu et al. [2024b] Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ping Luo, and Ying Shan.  Llama pro: Progressive llama with block expansion.  _arXiv preprint arXiv:2401.02415_ , 2024b.
  * Wu et al. [2023a] Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip.  Multimodal large language models: A survey.  In _2023 IEEE International Conference on Big Data (BigData)_ , pages 2247–2256. IEEE, 2023a.
  * Wu et al. [2023b] Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Derek F Wong, and Lidia S Chao.  A survey on llm-gernerated text detection: Necessity, methods, and future directions.  _arXiv preprint arXiv:2310.14724_ , 2023b.
  * Wu et al. [2024c] Kevin Wu, Eric Wu, and James Zou.  How faithful are rag models? quantifying the tug-of-war between rag and llms’ internal prior.  _arXiv preprint arXiv:2404.10198_ , 2024c.
  * Wu et al. [2023c] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  Next-gpt: Any-to-any multimodal llm.  _arXiv preprint arXiv:2309.05519_ , 2023c.
  * Yuan et al. [2017] Shuguang Yuan, HC Stephen Chan, and Zhenquan Hu.  Using pymol as a platform for computational drug design.  _Wiley Interdisciplinary Reviews: Computational Molecular Science_ , 7(2):e1298, 2017.
  * Yuan et al. [2024] Zhiyuan Yuan, Fangyuan Zhao, Senlin Lin, Yu Zhao, Jianhua Yao, Yan Cui, Xiao-Yong Zhang, and Yi Zhao.  Benchmarking spatial clustering methods with spatially resolved transcriptomics data.  _Nature Methods_ , pages 1–11, 2024.
  * Zhao et al. [2023] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_ , 2023.
  * Zhou et al. [2023] Hongjian Zhou, Boyang Gu, Xinyu Zou, Yiru Li, Sam S Chen, Peilin Zhou, Junling Liu, Yining Hua, Chengfeng Mao, Xian Wu, et al.  A survey of large language models in medicine: Progress, application, and challenge.  _arXiv preprint arXiv:2311.05112_ , 2023.

##  Appendix A Explanations of baseline models.

Here we highlight the summary of different selected open-source LLMs and MLLMs
used in this manuscript for either baselines or base models.

For LLMs, we have the following models:

  * •

LLaMA2: LLaMA2 is a collection of pre-trained and finetuned LLMs for chatting
and other downstream tasks. The base model we choose for finetuning is from
LLaMA2-Chat, which is optimized for dialogue use cases.

  * •

LLaMA3: LLaMA3 is a collection of pre-trained and finetuned LLMs for chatting
and other downstream tasks. The base model we choose for finetuning is from
LLaMA3-Instruct, which is optimized for dialogue use cases.

  * •

Mistral: Mistral is an open-source LLM that is based on a different framework
compared with LLaMA2. It can also be finetuned into different cases and we
also use its chat mode as the base model.

  * •

Gemma: Gemma is also an open-source LLM which is the state-of-the-art 7B-base
model. It can also be finetuned into different cases and we also use its chat
mode as the base model.

  * •

LLaMAPro: LLaMAPro is also an open-source LLM modified based on LLaMA2. It
offers extra blocks for finetuning and thus this approach can reduce the
catastrophic forgetting of a finetuned LLM.

  * •

GPT 3.5: GPT 3.5 is a generative pretraining model finetuned for the chatting
task. We used the OpenAI API to access this model.

  * •

GPT 4: GPT 4 is a generative pretraining model finetuned for the chatting
task. We used the OpenAI API to access this model. GPT 4 also has the ability
to handle image data as input.

  * •

bioGPT: bioGPT is a finetuned GPT-2 model based on medical data. It is
designed for the applications of LLMs in biomedical cases.

  * •

BioMedLM: BioMedLM is a finetuned GPT-2 model based on PubMed data. It is
designed for the applications of LLMs in biomedical cases, especially for
question-answering (QA) problems.

  * •

GeneGPT: GeneGPT is a tool-augmented method for tasks related to medical
queries. GeneGPT is based on searching documents to enrich the information in
the prompts. We use the OpenAI API to access the base model of this method.

  * •

bioMistral: bioMistral is a finetuned Mistral model based on medical data. It
is designed for the applications of LLMs in biomedical cases.

For MLLMs, we have the following models:

  * •

LLaVA: LLaVA is a collection of pre-trained and finetuned MLLMs for chatting
and other downstream tasks. The base model we choose for finetuning is from
LLaVA-v1.5, which is recommended in the tutorials of LLaVA.

  * •

MoE-LLaVA: MoE-LLaVA is the mixture-of-experts (MOE) version LLaVA. It
utilizes different experts in the fine-tuning stages with the same inputs from
embeddings of images and embeddings of words. We access this model based on
the Huggingface Space resource.

  * •

Gemini: Gemini is a closed-source MLLM for various tasks. We use an
interactive platform from Google to access this model.

  * •

GPT-4: Introduced before. Here the GPT-4 model includes GPT-4v and GPT-4o,
which represent their abilities to handle image-based data as prompts. GPT-4o
is the most advanced model from OpenAI.

## Supplementary figures

This section contains all the supplementary figures, starting from next page.

Extended Data Fig. 1: Definition of different evaluators or scorers. For the
scorer focusing on truthfulness, we evaluate the matching level of model
outputs for the description of gene properties and gene functions. For the
scorer focusing on structural correctness, we evaluate the correctness of the
structure of model outputs by comparing them to the limitations in the prompt.
The logos of scorers are generated by DALLE [40].  Extended Data Fig. 2:
Figures of GOEA results. Each figure represents top 10 pathways in one
cluster, and the pathways are ranked by $-\text{log(Adjusted P-value)}$.
Extended Data Fig. 3: An example to illustrate the contribution of data
augmentation. We compared the description of gene GLI1 from two models trained
with different datasets but from the same instruction/prompt. We highlighted
the redundant part in red.  Extended Data Fig. 4: Annotation of gene functions
by different colors. Extended Data Fig. 5: The results of data ablation for
the training process.  Extended Data Fig. 6: Relation between top-k candidates
and model performance.


# Spin Radiation of Electrons, Excitons, and Phonons

Chengyuan Cai  School of Physics, Huazhong University of Science and
Technology, Wuhan 430074, China  Tao Yu<EMAIL_ADDRESS>School of
Physics, Huazhong University of Science and Technology, Wuhan 430074, China

###### Abstract

In the celebrated Stern-Gerlach experiment an inhomogeneous static magnetic
field separates a beam of charge-neutral atoms with opposite spins, thereby
driving a “spin current” normal to the propagation direction. Here we
generalize it to the dynamic scenario by demonstrating a spin transfer between
an AC inhomogeneous magnetic field and intraband electrons or charge-neutral
excitons and phonons. We predict that parametric pumping can efficiently
radiate their DC spin currents from local AC magnetic sources with van der
Waals semiconductors as prototypes. This mechanism brings a unified and
efficient paradigm in the spin transport of distinct mobile carriers.

Introduction.—The flow of electron spins or spin current is a fundamental
physical concept that plays an important role in understanding the conversion
between angular momentum in different disguises [1, 3, 4, 2, 5]. Besides
electrons, angular momentum and magnetic moments can also be carried by
bosonic information carriers such as magnons [6, 7, 9, 8], photons [10, 11],
chiral phonons [12, 13, 14], as well as excitons [15, 16, 17, 18]. Spin
pumping by magnetic contacts [19, 20] and the spin-Hall effect [22, 21, 23]
are popular approaches generating electron spin currents in metals, but are
less appealing for semiconductors and low-dimensional van der Waals materials
due to Schottky barriers and electronic structure mismatch [24]. These
approaches are difficult to apply to charge-neutral carriers as well such as
phonons and excitons, which have become important information carriers
utilized in modern quantum nanodevices [12, 13, 14, 15, 16, 17, 18].

Generation of spin current of charge-neutral carriers may be historically
traced to the celebrated Stern-Gerlach experiment [25], where the gradient of
static magnetic fields separates a beam of charge-neutral atoms with opposite
spins, thereby driving a spin current normal to the propagation direction.
Still, a similar effect appears to disappear when the magnetic field and,
thereby, the “force” oscillates. Instead, specific interband optical selection
rules or spin-orbit coupling optical fields can generate spin polarization or
accumulation for charge-neutral excitons [26, 27, 28] and chiral phonons [29],
as well as electrons [33, 34, 31, 32, 30]. Their interplay in semiconductors
and van der Waals materials inspires functionalities in opto-spintronic [35,
36] and magnonic [6, 7, 9, 8, 37] devices. However, plane-wave optical photons
hold very little momentum and cannot directly generate a spin current for the
charge-neutral carriers and electrons. This raises the issue of whether it is
possible to generalize the Stern-Gerlach effect to an AC magnetic field to
generate spin currents of distinct mobile carriers.

In this Letter, we predict intraband angular momentum transfer between a
focused radio-frequency (rf) or terahertz (THz) radiation and the electrons or
the charge neutral carriers in conductors, semiconductors, and van der Waals
materials, which is very different from the creation of interband electron-
hole pairs by the polarized electric fields of THz/infrared radiation.
Strongly localized near fields may be generated by, e.g., proximity excited
nanomagnets [39, 40, 38], focused laser beams, metallic nanostructures [41,
42] or a scanning near-field optical microscope (SNOM) [43, 44, 45, 46]. We
predict a parametric pumping mechanism that efficiently generates DC spin
currents carried by electrons, charge-neutral excitons, and phonons (Fig. 1).
Since the spin current is radiated from the local source, we term this
phenomenon as “spin radiation” for short.

Figure 1: Radiation of DC spin current of electrons or excitons when pumped by
a focused magnetic (optical or microwave) field with circular polarization.

Inelastic spin-flip by photons.—We first sketch the key physical processes by
electrons and estimate the magnitude of DC spin currents emitted by localized
AC magnetic (microwave or optical) fields in a setup as in Fig. 1. We consider
a monochromatic magnetic field
$\mathbf{h}({\boldsymbol{\rho}},t)=\sum_{\mathbf{q}}\left(\mathbf{h}^{(+)}(\mathbf{q})e^{-i\omega
t}+\mathbf{h}^{(-)}(\mathbf{q})e^{i\omega
t}\right)e^{i\mathbf{q}\cdot{\boldsymbol{\rho}}}$ [47] with frequency $\omega$
applied to a two-dimensional electron gas (2DEG) with an in-plane position
vector ${\boldsymbol{\rho}}=x\hat{\mathbf{x}}+y\hat{\mathbf{y}}$. The
electric-field component may be disregarded for our purpose since it does not
couple to the spin degree of freedom. The Fourier components of a strongly
localized field at the origin with polarization or “spin” along the out-of-
plane $\hat{\mathbf{z}}$-direction read

$\mathbf{h}^{(\pm)}(\mathbf{q})\approx(h_{0}/S)(1,\pm i,0)^{T},$ (1)

where $h_{0}$ is the amplitude, $S$ is the sample area, and “$+$” (“$-$”)
corresponds to the positive (negative) circular polarization. It couples with
the electrons by the Zeeman interaction
$\hat{V}(t)=\mu_{0}\gamma_{e}\int\hat{\mathbf{s}}(\boldsymbol{\rho})\cdot\mathbf{h}(\boldsymbol{\rho},t)d{\boldsymbol{\rho}}$,
in which $\mu_{0}$ is the vacuum permeability and $\gamma_{e}$ is the
effective gyromagnetic ratio of electrons, and excites a non-equilibrium spin
accumulation in the conduction band, which is the the expectation value of the
spin density operator
$\hat{\mathbf{s}}({\boldsymbol{\rho}})=\sum_{\eta,\epsilon}(\hbar/2){\boldsymbol{\sigma}}_{\eta\epsilon}|{\boldsymbol{\rho}},\eta\rangle\langle{\boldsymbol{\rho}},\epsilon|$,
where ${\boldsymbol{\sigma}}$ are Pauli matrices,
$\\{\eta,\epsilon\\}=\\{\uparrow,\downarrow\\}$ denote electron spins along
the $\hat{\mathbf{z}}$-direction, and $|{\boldsymbol{\rho}},\eta\rangle$ is an
electron ket. We decompose the coupling into
$\hat{V}(t)=\hat{V}^{(+)}e^{-i\omega t}+\hat{V}^{(-)}e^{i\omega t}$ and expand
it into electron eigenstates $|\mathbf{k},\eta\rangle$ as
$\hat{V}^{(\pm)}=\sum_{\mathbf{k},\mathbf{k}^{\prime}}\sum_{\eta,\epsilon}{\cal
G}^{(\pm)}_{\mathbf{k}^{\prime}-\mathbf{k}}|_{\eta\epsilon}e^{\mp i\omega
t}|\mathbf{k}^{\prime},\eta\rangle\langle\mathbf{k},\epsilon|$, where
$\mathbf{k}=k_{x}\hat{\mathbf{x}}+k_{y}\hat{\mathbf{y}}$ is electron wave
vector and ${\cal
G}^{(\pm)}_{\mathbf{q}}|_{\eta\epsilon}=(\mu_{0}\gamma_{e}\hbar/2)\sum_{\alpha=\\{x,y,z\\}}\sigma_{\eta\epsilon}^{\alpha}h_{\alpha}^{(\pm)}(\mathbf{q})$
are matrix elements. On substituting Eq. (1), only ${\cal G}^{(+)}_{\bf
q}|_{\uparrow\downarrow}$ and ${\cal G}^{(-)}_{\bf q}|_{\downarrow\uparrow}$
are non-zero, which reflects the conservation of angular momentum.

The localized or focused magnetic field coherently couples electron states of
different wave vectors since the operator $\hat{V}^{(+)}$ has finite matrix
elements between an occupied initial state $|\mathbf{k},\uparrow\rangle$ of
energy $\varepsilon_{\mathbf{k}}$ and an empty state
$|\mathbf{k}^{\prime},\downarrow\rangle$ of higher energy
$\varepsilon_{\mathbf{k^{\prime}}}$. Inversely, at finite temperatures
$\hat{V}^{(-)}$ emits a photon when the higher energy state is occupied and
the lower empty. These processes are captured by the time evolution of the
$2\times 2$ spin density matrix
$\boldsymbol{\rho}_{\mathbf{k^{\prime}k}}|_{\eta\epsilon}=\langle\mathbf{k^{\prime}},\eta|\hat{\rho}|\mathbf{k},\epsilon\rangle$,
in which diagonal terms represent the population and off-diagonal terms the
coherence between opposite spins. As detailed in the Supplemental Material
(SM) [48] the density matrix obeys the Liouville equation

$\displaystyle i\hbar\frac{\partial\rho_{\mathbf{k^{\prime}k}}}{\partial t}$
$\displaystyle=(\varepsilon_{\mathbf{k^{\prime}}}-\varepsilon_{\mathbf{k}})\rho_{\mathbf{k^{\prime}k}}+\sum_{\zeta=\pm}\sum_{\mathbf{q}}\mathcal{G}_{\mathbf{k^{\prime}-q}}^{(\zeta)}\rho_{\mathbf{qk}}e^{-i\zeta\omega
t}$
$\displaystyle-\sum_{\zeta=\pm}\sum_{\mathbf{q}}\rho_{\mathbf{k^{\prime}q}}\mathcal{G}_{\mathbf{q-k}}^{(\zeta)}e^{-i\zeta\omega
t}.$ (2)

A uniform field cannot induce inter-momentum coherence since when
$\mathcal{G}_{\mathbf{q}-\mathbf{k}}\propto\delta_{\mathbf{q}\mathbf{k}}$,
$\rho_{\mathbf{k}^{\prime}\mathbf{k}}\propto\delta_{\mathbf{k}^{\prime}\mathbf{k}}$
is diagonal in the wave vector space.

The carrier population and spin coherence induced by external fields follow
from the solutions of the equation of motion (2). Without drive,
${\rho}_{\mathbf{k^{\prime}k}}\approx
f_{\mathbf{k}}\delta_{\mathbf{k^{\prime}k}}$ is the Fermi-Dirac distribution
$f_{\mathbf{k}}=1/(e^{(\varepsilon_{\mathbf{k}}-\mu)/(k_{B}T)}+1)$ at
temperature $T$ and chemical potential $\mu$. With drive, the series expansion
up to $V^{2}$ to Eq. (2) reads

$\displaystyle{\rho}_{\mathbf{k^{\prime}k}}=f_{\mathbf{k}}\delta_{\mathbf{k^{\prime}k}}+\sum_{\zeta=\pm}\frac{\mathcal{G}_{\mathbf{k^{\prime}-k}}^{(\zeta)}(f_{\mathbf{k}}-f_{\mathbf{k^{\prime}}})e^{-i\zeta\omega
t}}{\varepsilon_{\mathbf{k}}+\zeta\hbar\omega-\varepsilon_{\mathbf{k^{\prime}}}+i0_{+}}+\sum_{\zeta_{1},\zeta_{2}=\pm}\sum_{\mathbf{q}}$
$\displaystyle\times\Big{[}\frac{\mathcal{G}_{\mathbf{k^{\prime}-q}}^{(\zeta_{1})}\mathcal{G}_{\mathbf{q-k}}^{(\zeta_{2})}(f_{\mathbf{k}}-f_{\mathbf{q}})e^{-i(\zeta_{1}+\zeta_{2})\omega
t}}{(\varepsilon_{\mathbf{k}}+(\zeta_{1}+\zeta_{2})\hbar\omega-\varepsilon_{\mathbf{k^{\prime}}}+i0_{+})(\varepsilon_{\mathbf{k}}+\zeta_{2}\hbar\omega-\varepsilon_{\mathbf{q}}+i0_{+})}$
$\displaystyle+\frac{\mathcal{G}_{\mathbf{k^{\prime}-q}}^{(\zeta_{1})}\mathcal{G}_{\mathbf{q-k}}^{(\zeta_{2})}(f_{\mathbf{k^{\prime}}}-f_{\mathbf{q}})e^{-i(\zeta_{1}+\zeta_{2})\omega
t}}{(\varepsilon_{\mathbf{k}}+(\zeta_{1}+\zeta_{2})\hbar\omega-\varepsilon_{\mathbf{k^{\prime}}}+i0_{+})(\varepsilon_{\mathbf{q}}+\zeta_{1}\hbar\omega-\varepsilon_{\mathbf{k^{\prime}}}+i0_{+})}\Big{]}.$
(3)

When disregarding the interference between different wave vectors in Eq. (3)
and taking $\zeta_{1}=-\zeta_{2}$ for DC spin-injection processes, the spin
injection rate is then constant and governed by the rate equation [48]

$\displaystyle\frac{\partial{\bf s}}{\partial t}\Big{|}_{\mathrm{DC}}$
$\displaystyle=\frac{\hbar}{2}\sum_{\mathbf{k}}\sum_{\eta,\epsilon=\\{\uparrow,\downarrow\\}}\frac{\partial{\rho}^{I}_{\mathbf{kk}}(t)|_{\epsilon\eta}}{\partial
t}\boldsymbol{\sigma}_{\eta\epsilon}$
$\displaystyle=\pi\sum_{\mathbf{k},\mathbf{q}}\left(f_{\mathbf{q}}-f_{\mathbf{k}}\right)\mathrm{Tr}\left(\mathcal{G}_{\mathbf{q-k}}^{(+)\dagger}{\cal
G}_{\mathbf{q-k}}^{(+)}\boldsymbol{\sigma}\right)$
$\displaystyle\times\delta\left(\varepsilon_{\mathbf{k}}+\hbar\omega-\varepsilon_{\mathbf{q}}\right)+{\rm
H.c.},$ (4)

noting in the interaction representation
${\rho}^{I}_{\mathbf{kk}}(t)={\rho}_{\mathbf{kk}}(t)$.

The spin injection is driven by the spin-flip induced by the AC magnetic field
(1) that may be understood in terms of photon absorption processes in Fig. 2
for a parabolic electron dispersion $\varepsilon_{\bf
k}={\hbar^{2}k^{2}}/(2m^{\ast})$, where $m^{\ast}$ is effective mass of
electrons. The red curves sketch an electron with spin “$\downarrow$” that
under absorption of a photon with energy $\hbar\omega$ flips to a “$\uparrow$”
under energy and momentum conservation, i.e., a transition from
$|\mathbf{k},\downarrow\rangle$ to $|\mathbf{q}_{+},\uparrow\rangle$. Here
$\varepsilon(\mathbf{q}_{\pm})=\varepsilon(\mathbf{k})\pm\hbar\omega$. The
blue curves indicate the photon emission process from
$|\mathbf{k},\uparrow\rangle$ to $|\mathbf{q}_{-},\downarrow\rangle$.

Figure 2: Spin-transfer process in an electron gas induced by a circularly
polarized photon. An electron at the Fermi energy can absorb a photon with
energy to be excited to a higher energy state or emit a photon to a lower
energy state under energy and linear/angular momentum conservation. Here state
k is occupied while states q must be empty. The Feynman diagrams depict the
photon absorption and emission processes with spin angular momentum
conservation.

At zero temperature electrons occupy states below the Fermi energy $E_{f}$ and
Fermi wave number $k_{f}$. Substituting the parabolic electron dispersion and
the field (1) into (4),

$\displaystyle\frac{\partial\mathbf{s}}{\partial t}\Big{|}_{\mathrm{DC}}$
$\displaystyle=\sum_{\mathbf{k}}\int_{0}^{2\pi}d\varphi_{\mathbf{q}}\int_{0}^{\infty}d\varepsilon_{\bf
q}\frac{iSm^{\ast}}{\pi\hbar^{2}}\left(f(\varepsilon_{\mathbf{q}})-f(\varepsilon_{\mathbf{k}})\right)$
$\displaystyle\times\left(\mathbf{h}_{\mathbf{k-q}}^{(+)\ast}\times\mathbf{h}_{\mathbf{k-q}}^{(+)}\right)\delta\left(\varepsilon_{\mathbf{k}}+\hbar\omega-\varepsilon_{\mathbf{q}}\right)+{\rm
H.c.}$
$\displaystyle={\omega(m^{\ast}\mu_{0}\gamma_{e}h_{0})^{2}}/{(2\pi\hbar)}\hat{\bf
z}.$ (5)

It indicates that the photon absorption in Fig. 2 contribute to the spin
injection. The number of absorbed photons with frequency $\hbar\omega$ is
proportional to $\hbar\omega D_{\mathrm{2DEG}},$ where the density of states
of 2DEG $D_{\mathrm{2DEG}}=Sm^{\ast}/(2\pi\hbar^{2})$. In the ballistic
regime, the energy and angular momentum injection rate into the electron
system equals the total spin and energy currents flowing through a circle
around the source. We then estimate the spin current density by [48]

$\displaystyle\frac{\partial\mathbf{s}}{\partial t}\Big{|}_{\mathrm{DC}}$
$\displaystyle=\varoiint\boldsymbol{\cal
J}_{s}^{\mathrm{DC}}(\boldsymbol{\rho})\cdot d\mathbf{S}$
$\displaystyle=2\pi\rho\mathcal{J}_{s}^{\mathrm{est}}({\rho}){\hat{\mathbf{z}}}=\omega(m^{\ast}\mu_{0}\gamma_{e}h_{0})^{2}/(2\pi\hbar){\hat{\mathbf{z}}},$
(6)

where $\rho=|\boldsymbol{\rho}|$. The process is proportional to the
absorption coefficient of the light intensity $\sim h_{0}^{2}$.

Quantum formalism.—Below we substantiate the magnetic spin pumping found by
rate equation as sketched above by a full quantum formalism. The Hamiltonian
of 2DEG in the $x$-$y$ plane subject to an inhomogeneous AC magnetic field
$\mathbf{h}({\boldsymbol{\rho}},t)$ of frequency $\omega$ reads

$\displaystyle\hat{H}_{e}$
$\displaystyle=\hat{H}_{0}+\hat{V}(t)=\sum_{\mathbf{k,\eta}}\left(\varepsilon_{\mathbf{k}}-\mu\right)|\mathbf{k},\eta\rangle\langle\mathbf{k},\eta|$
$\displaystyle+\sum_{\zeta=\pm}\sum_{\mathbf{k},\mathbf{k}^{\prime}}\sum_{\eta,\epsilon=\\{\uparrow,\downarrow\\}}{\cal
G}^{(\zeta)}_{\mathbf{k}^{\prime}-\mathbf{k}}|_{\eta\epsilon}e^{-i{\zeta}\omega
t}|\mathbf{k}^{\prime},\eta\rangle\langle\mathbf{k},\epsilon|,$ (7)

where all symbols have been defined above. Referring to the SM [48] for
details, the time evolution operator $\hat{U}_{I}(t,t_{0})$ in the interaction
representation is expanded into Dyson series of which we again retain the
lowest two order of $\hat{V}$. The electron wavefunction evolves under the
perturbation according to [49, 50, 51]
$\psi_{\eta}(\boldsymbol{\rho})=\sum_{\mathbf{k^{\prime}}}\sum_{\epsilon}\langle\boldsymbol{\rho}|e^{-i\hat{H}_{0}t/\hbar}|\mathbf{k}^{\prime},\epsilon\rangle\langle\mathbf{k^{\prime}},\epsilon|\hat{U}_{I}(t,t_{0}\rightarrow-\infty)|\mathbf{k},\eta\rangle$.
The field operator of such driven eigenstates in terms of the electron
annihilation operator $\hat{a}_{\eta}(\mathbf{k})$ of an unperturbed state
with wave vector $\mathbf{k}$ and spin $\eta$ [48]

$\displaystyle\hat{\psi}_{\eta}(\boldsymbol{\rho},t)=\frac{1}{\sqrt{S}}\left(\sum_{\mathbf{k}}\hat{a}_{\eta}(\mathbf{k})e^{i(\mathbf{k}\cdot\boldsymbol{\rho}-\varepsilon_{\mathbf{k}}t/\hbar)}+\sum_{\zeta=\pm}\sum_{\epsilon}\sum_{\mathbf{k,k^{\prime}}}\hat{a}_{\epsilon}(\mathbf{k})\right.$
$\displaystyle\times\frac{{\cal
G}_{\mathbf{k^{\prime}-k}}^{(\zeta)}|_{\eta\epsilon}e^{i(\mathbf{k^{\prime}}\cdot\boldsymbol{\rho}-(\varepsilon_{\mathbf{k}}+\zeta\hbar\omega)t/\hbar)}}{\varepsilon_{\mathbf{k}}+\zeta\hbar\omega-\varepsilon_{\mathbf{k^{\prime}}}+i0_{+}}+\sum_{\zeta_{1},\zeta_{2}=\pm}\sum_{\mathbf{k^{\prime}},\mathbf{k},\mathbf{q}}\sum_{\xi,\epsilon=\uparrow,\downarrow}\hat{a}_{\epsilon}(\mathbf{k})$
$\displaystyle\left.\times\frac{{\cal
G}_{\mathbf{k}^{\prime}-\mathbf{q}}^{(\zeta_{1})}|_{\eta\xi}{\cal
G}_{\mathbf{q}-\mathbf{k}}^{(\zeta_{2})}|_{\xi\epsilon}e^{i(\mathbf{k^{\prime}}\cdot\boldsymbol{\rho}-(\varepsilon_{\mathbf{k}}+(\zeta_{1}+\zeta_{2})\hbar\omega)t/\hbar)}}{\left(\varepsilon_{\mathbf{k}}+(\zeta_{1}+\zeta_{2})\hbar\omega-\varepsilon_{\mathbf{k^{\prime}}}+i0_{+}\right)\left(\varepsilon_{\mathbf{k}}+\zeta_{2}\hbar\omega-\varepsilon_{\mathbf{q}}+i0_{+}\right)}\right),$

where the Cartesian position (wave) vector $\boldsymbol{\rho}$
${(\mathbf{q}}_{\zeta})$ transforms to polar coordinates as
$\boldsymbol{\rho}=\rho\cos\varphi\hat{\mathbf{x}}+\rho\sin\varphi\hat{\mathbf{y}}$
($\mathbf{q}_{\zeta}=q_{\zeta}\cos\varphi_{q_{\zeta}}\hat{\mathbf{x}}+q_{\zeta}\sin\varphi_{q_{\zeta}}\hat{\mathbf{y}}$).

The spin-current density carried by the excited states is then

$\boldsymbol{\cal
J}_{s}(\boldsymbol{\rho},t)=\left\langle\frac{\hbar^{2}}{4im^{\ast}}\sum_{\eta\epsilon}\hat{\psi}_{\eta}^{\dagger}{\boldsymbol{\sigma}}_{\eta\epsilon}\nabla\hat{\psi}_{\epsilon}+\mathrm{H.c.}\right\rangle=\sum_{n\geq
0}\boldsymbol{\cal J}_{s}^{(n)}(\boldsymbol{\rho},t),$

where the second step indicates a perturbation expansion $\boldsymbol{\cal
J}_{s}^{(n)}\propto V^{n}$. Since the ensemble average
$\left\langle\hat{a}_{\eta}^{\dagger}(\mathbf{k}_{1})\hat{a}_{\epsilon}(\mathbf{k}_{2})\right\rangle=\delta_{\mathbf{k}_{1}\mathbf{k}_{2}}\delta_{\eta\epsilon}f_{\mathbf{k}}$
in terms of the Fermi-Dirac distribution $f_{\mathbf{k}}$, the zero-order
(equilibrium) spin current $\boldsymbol{\cal
J}_{s}^{(0)}(\boldsymbol{\rho},t)=\hbar^{2}/({4m^{\ast}S})\sum_{\mathbf{k}}f_{\mathbf{k}}\Tr\left(\boldsymbol{\sigma}\otimes\mathbf{k}\right)+\mathrm{H.c.}$
vanishes. In the linear response, the tensor

$\displaystyle\boldsymbol{\cal J}_{s}^{(1)}(\boldsymbol{\rho})$
$\displaystyle=\sum_{\mathbf{k}}\frac{-i}{8\pi
S}\int_{\varphi-\frac{\pi}{2}}^{\varphi+\frac{\pi}{2}}d\varphi_{q_{+}}(f_{\mathbf{k}}-f_{\mathbf{q}_{+}})e^{i((\mathbf{q}_{+}-\mathbf{k})\cdot{\boldsymbol{\rho}}-\omega
t)}$
$\displaystyle\times\Tr\left(\mathcal{G}_{\mathbf{q}_{+}-\mathbf{k}}^{(+)\dagger}\boldsymbol{\sigma}\otimes(\mathbf{k}+\mathbf{q}_{+})\right)+\mathrm{H.c.}$

oscillates at frequency $\omega$ and can be detected via the AC spin Hall
effect [52, 53]. However, the linear response conserves energy and spin and
vanishes on time average. Analogous to the spin pumping by magnetization
dynamics, DC spin currents emerge in the second-order term of the perturbation
series:

$\displaystyle\boldsymbol{\cal
J}_{s}^{(2)}(\boldsymbol{\rho},t)=\sum_{\zeta_{1},\zeta_{2}=\pm}\sum_{\mathbf{q},\mathbf{q}^{\prime},\mathbf{k}}\frac{\hbar^{2}}{4m^{\ast}}\mathrm{Tr}\left(\mathcal{G}_{\mathbf{q}-\mathbf{k}}^{(\zeta_{1})\dagger}(\boldsymbol{\sigma}\otimes\mathbf{q}^{\prime})\mathcal{G}_{\mathbf{q}^{\prime}-\mathbf{k}}^{(\zeta_{2})}\right)$
$\displaystyle\times\left[\frac{f_{\mathbf{k}}}{(\varepsilon_{\mathbf{k}}+\zeta_{2}\hbar\omega-\varepsilon_{\mathbf{q^{\prime}}}+i0_{+})(\varepsilon_{\mathbf{k}}+\zeta_{1}\hbar\omega-\varepsilon_{\mathbf{q}}-i0_{+})}\right.$
$\displaystyle+\frac{f_{\mathbf{q}^{\prime}}}{(\varepsilon_{\mathbf{q}}+(\zeta_{2}-\zeta_{1})\hbar\omega-\varepsilon_{\mathbf{q^{\prime}}}+i0_{+})(\varepsilon_{\mathbf{k}}+\zeta_{2}\hbar\omega-\varepsilon_{\mathbf{q^{\prime}}}+i0_{+})}$
$\displaystyle\left.+\frac{f_{\mathbf{q}}}{(\varepsilon_{\mathbf{q}}+(\zeta_{2}-\zeta_{1})\hbar\omega-\varepsilon_{\mathbf{q^{\prime}}}+i0_{+})(\varepsilon_{\mathbf{q}}-\zeta_{1}\hbar\omega-\varepsilon_{\mathbf{k}}+i0_{+})}\right]$
$\displaystyle\times
e^{i((\mathbf{q^{\prime}}-\mathbf{q})\cdot\boldsymbol{\rho}-(\zeta_{2}-\zeta_{1})\omega
t)}+\mathrm{H.c.},$

in which the $\zeta_{1}=\zeta_{2}$ contribution is constant in time and leads
to the DC spin current

$\displaystyle\boldsymbol{\cal J}_{s}^{\mathrm{DC}}(\boldsymbol{\rho})$
$\displaystyle\approx\sum_{\zeta=\pm}\sum_{\mathbf{k}}i\frac{m^{\ast}\mu_{0}^{2}\gamma_{e}^{2}}{32\pi^{2}S}\int_{\varphi-\frac{\pi}{2}}^{\varphi+\frac{\pi}{2}}d\varphi_{p_{\zeta}}d\varphi_{q_{\zeta}}e^{i(\mathbf{q}_{\zeta}-\mathbf{p}_{\zeta})\cdot\boldsymbol{\rho}}$
$\displaystyle\times\left(\mathbf{h}^{(\zeta)}(\mathbf{q}_{\zeta}-\mathbf{k})\times\mathbf{h}^{(\zeta)\ast}(\mathbf{p}_{\zeta}-\mathbf{k})\right)\otimes\mathbf{q}_{\zeta}$
$\displaystyle\times(f_{\mathbf{k}}-f_{\mathbf{q}_{\zeta}})+\mathrm{H.c.},$
(8)

in the approximation $f_{\mathbf{q}}=f_{\mathbf{q^{\prime}}}$ due to the
factor
$1/(\varepsilon_{\mathbf{q}}-\varepsilon_{\mathbf{q^{\prime}}}+i0_{+})$. We
derive the same result by the density-matrix approach in the SM [48]. The
circular polarization
$\mathbf{h}^{(+)}(\mathbf{q}_{+}-\mathbf{k})\times\mathbf{h}^{(+)\ast}(\mathbf{p}_{+}-\mathbf{k})$
or “photon spin” [10, 11] governs the electron spin polarization, which
depends on the optical/microwave source and is flexibly tunable.

The above formalism holds for arbitrary magnetic field profiles, frequencies,
and electron densities. It is convenient to derive specific results from a
line source $\mathbf{h}(\boldsymbol{\rho},t)=\mathbf{h}(x,t)$ with Fourier
components
$\mathbf{h}^{(\zeta)}(\mathbf{q})=2\pi\delta(q_{y})\mathbf{H}^{(\zeta)}(q_{x})$.
In the far-field $x\rightarrow+\infty$, the DC spin current [54]

$\displaystyle\boldsymbol{\cal J}^{\mathrm{DC}}_{s,{\rm
1D}}(\boldsymbol{\rho})\approx\sum_{\zeta=\pm}\sum_{k_{x},k_{y}}\frac{im^{*}\mu_{0}^{2}\gamma_{e}^{2}}{8\kappa_{\zeta}S}\left(f(k_{x},k_{y})-f(\kappa_{\zeta},k_{y})\right)$
$\displaystyle\times\left(\mathbf{H}^{(\zeta)}(\kappa_{\zeta}-k_{x})\times\mathbf{H}^{(\zeta)*}(\kappa_{\zeta}-k_{x})\right)\otimes\hat{\mathbf{x}}+\mathrm{H.c.},$

where $\kappa_{\zeta}\equiv\sqrt{k_{x}^{2}+2\zeta m^{*}\omega/\hbar}$.

Numerical results.—Here we illustrate the 2D spin radiation by the THz field
(1) of strong localization. Substituting into Eq. (8),

$\displaystyle\boldsymbol{\cal J}^{\mathrm{DC}}_{s}(\boldsymbol{\rho})$
$\displaystyle=\sum_{\mathbf{k}}\frac{m^{*}\mu_{0}^{2}\gamma_{e}^{2}h_{0}^{2}}{8S}\left(f_{\mathbf{k}}-f_{\mathbf{q}_{+}}\right)\left(\hat{\mathbf{z}}\otimes\hat{\mathbf{e}}_{\rho}\right)$
$\displaystyle\times\left(kF(k\rho)+q_{+}F(q_{+}\rho)\right)$
$\displaystyle\approx\frac{m^{*2}\mu_{0}^{2}\gamma_{e}^{2}h_{0}^{2}}{8\pi\hbar}\omega
k_{f}k_{f}F(k_{f}\rho)\left(\hat{\mathbf{z}}\otimes\hat{\mathbf{e}}_{\rho}\right),$
(9)

where $q_{+}=|\mathbf{q}_{+}|$ and in
$F(x)=J_{0}(x)H_{-1}(x)+J_{1}(x)H_{0}(x)$, $J_{n}(x)$ and $H_{n}(x)$ are
$n$-order Bessel function of the first kind and Struve function. In the second
step, we assume the degenerate limit, in which only electrons near the Fermi
surface contribute.

Figure 3 plots the spatial distribution of radiated $\boldsymbol{\cal
J}_{s}^{\mathrm{DC}}(\boldsymbol{\rho})$. In the calculation, we consider a
monolayer $n$-doped $\mathrm{MoS_{2}}$ with effective electron mass
$m^{*}=0.48m_{e}$ [57] and $g$-factor $|g_{e}|=2.16$ [58]. The typical
electron density $n_{e}=5.6\times 10^{12}$ $\mathrm{cm}^{-2}$ [59] corresponds
to a Fermi energy $E_{f}\sim 28.5$ meV. The mobility of electrons is high [55]
and their out-of-plane spin polarization has a long lifetime [56]. The field
frequency $\omega=10$ THz and its amplitude $\mu_{0}h_{0}\approx\pi\times
10^{-16}$ $\mathrm{T}\cdot\mathrm{m}^{2}$ is equivalent to a spot of magnetic
field $0.4$ mT of radius $500$ nm, which could be generated by THz near-field
from metallic nanoparticles [41, 42] or SNOM [43, 44, 45, 46]. As shown in
Fig. 3(a), the spin current radiates outward from the optical spot and decays
according to $1/\rho$. In Fig. 3(b), $2\pi\rho{\boldsymbol{\cal
J}}_{s}^{\mathrm{DC}}(\rho)$ is almost a constant with the increase of radius
$\rho$, which agrees well with the estimation from the spin transfer in Eq.
(6). The effect is robust and persisting at different temperatures and
electron densities, as shown in Fig. 3(c) and (d).

Referring to the SM [48], the spin current can be efficiently generated by
realistic magnetic-field spots and for the spin radiation by a line source,
the directed spin current does not decrease with the distance.

Figure 3: Radiated DC spin current $\boldsymbol{\cal
J}^{\mathrm{DC}}_{s}(\boldsymbol{\rho})$ under a spot of magnetic field. (a)
illustrates the magnitude (the color) and direction (the arrows) of the
radiated spin current. (b) plots $2\pi\rho{\boldsymbol{\cal
J}}^{\mathrm{DC}}_{s}(\rho)/\hbar$ as a function of radius $\rho$ and compares
with the estimation of spin transfer rate via Fermi’s golden rule [Eq. (6)].
(c) and (d) plot the efficiency of the spin transfer with different
temperatures and electron densities.

Optical radiation of exciton spin/valley current.—We then generalize the above
mechanism to the spin-current generation of charge-neutral angular momentum-
carriers such as excitons and chiral phonons [12, 13, 14]. The energy-
degenerate excitons in opposite valleys of monolayer MoS2 carry opposite spins
$\\{\uparrow,\downarrow\\}$. Their coupling with magnetic fields is referred
to as “valley Zeeman effect” [16, 17, 18]. Exciton with a valley, thereby
spin, polarization can be pumped by a laser of circular polarization via
direct-band photon absorption. We demonstrate here that an exciton pure spin
or valley current radiates when the exciton distribution is subjected to a
focused THz magnetic field, differing from previous proposals based on
interference [60, 61] or dispersion warping effect [62].

Here an optical laser generates a Gaussian distribution
$f_{\mathrm{ex}}^{\uparrow,\downarrow}(\mathbf{k})=\alpha_{\rm
ex}^{\uparrow,\downarrow}\exp(-(\varepsilon_{\mathbf{k}}^{\mathrm{ex}}-\varepsilon_{p})^{2}/(2\delta_{\varepsilon}^{2}))$
of excitons with spin $\\{\uparrow,\downarrow\\}$ in opposite valleys,
centered around the laser energy $\varepsilon_{p}$ and broadened by
$\delta_{\varepsilon}$ [63, 64, 65].
$\varepsilon_{\mathbf{k}}^{\mathrm{ex}}=\hbar^{2}k^{2}/(2m_{\mathrm{ex}}^{\ast})$
with exciton mass $m_{\mathrm{ex}}^{\ast}$. The normalization factor
$\alpha_{\rm
ex}^{\uparrow,\downarrow}={n_{\mathrm{ex}}^{\uparrow,\downarrow}}/{\sum_{\mathbf{k}}\exp[-(\varepsilon_{\mathbf{k}}^{\mathrm{ex}}-\varepsilon_{p})^{2}/\left(2\delta_{\varepsilon}^{2}\right)]}$
depends on the exciton densities $n_{\mathrm{ex}}^{\uparrow,\downarrow}$ in
opposite valleys, tunable by the ellipticity of light polarization. We then
subject the exciton to a strongly localized magnetic field
$\mathbf{h}(\boldsymbol{\rho},t)$ with circular polarization along the
$\hat{\mathbf{z}}$-direction. For the excitons of spin $s$ and gyromagnetic
ratio $\gamma_{\mathrm{ex}}=g_{\mathrm{ex}}\mu_{B}$ with exciton $g$-factor
$g_{\mathrm{ex}}$, the exciton spin current pumped by a focused magnetic field
reads, analogous to Eq. (9),

$\displaystyle\boldsymbol{\cal J}_{s}^{\mathrm{ex}}(\boldsymbol{\rho})$
$\displaystyle=\sum_{\mathbf{k}}\frac{m_{\mathrm{ex}}^{\ast}\mu_{0}^{2}\gamma_{\mathrm{ex}}^{2}h_{0}^{2}s^{2}}{2S}\left(f_{\mathrm{ex}}^{\downarrow}(\mathbf{k})-f_{\mathrm{ex}}^{\uparrow}(\mathbf{q}_{+})\right)$
$\displaystyle\times\left(kF(k\rho)+q_{+}F(q_{+}\rho)\right)\left(\hat{\mathbf{z}}\otimes\hat{\mathbf{e}}_{\rho}\right).$
(10)

This exciton spin current vanishes when
$n_{\mathrm{ex}}^{\uparrow}=n_{\mathrm{ex}}^{\downarrow}$, but exists in the
presence of valley polarization, which can be understood from the pumping
process depicted in Fig. 3(a): The exciton with polarization “$\downarrow$” is
pumped to the “$\uparrow$” states through the photon absorption $V^{(+)}$
process, and inversely, the exciton with polarization “$\uparrow$” is driven
to the “$\downarrow$” states by photon emission $V^{(-)}$ process. The net
spin transfer by absorption and emission then depends on the exciton
distribution $f^{\uparrow}_{\rm ex}$ and $f^{\downarrow}_{\rm ex}$.

Figure 4: Optical radiation of exciton spin current. (a) addresses the spin
transfer process in the opposite valleys, in which exciton distributions
$f^{\uparrow}_{\rm ex}\neq f^{\downarrow}_{\rm ex}$. (b) shows pumped exciton
spin current $2\pi\rho\boldsymbol{\cal J}^{\rm ex}_{s}(\rho)/\hbar$.

Figure 3(b) plots the exciton spin current under a laser irradiation of energy
$\varepsilon_{p}\approx 100$ meV and bandwidth $\delta_{\varepsilon}\approx 4$
meV [66], and for the THz field $\omega=10$ THz and
$\mu_{0}h_{0}\approx\pi\times 10^{-16}$ $\mathrm{T}\cdot\mathrm{m}^{2}$. For
excitons, $m_{\mathrm{ex}}^{\ast}\approx 0.19m_{e}$ [67], $g_{\mathrm{ex}}=-4$
[68], and $n_{\mathrm{ex}}^{\uparrow}=0.7n_{\mathrm{ex}}^{\downarrow}=7\times
10^{9}$ $\mathrm{cm}^{-2}$ [69].

For phonon spin current, we only need to replace the distribution function of
exciton (10) with that of chiral phonons [12, 13, 14]. We will address such
effects and those of different materials in the future.

Conclusion and discussion.—In conclusion, we generalize the spin-current
generation in static Stern-Gerlach effect to a dynamic scenario or “magnetic
spin pumping”, in which a focused AC magnetic field provides “forces” to
radiate the spin current of charge-neutral excitons and phonons, as well as
electrons. This effect is free of charge, optical selection rules, and spin-
orbit coupling. Its efficiency is measurable: the pumped spin current is of
the same order as the spin Hall current generated by an electric field of 0.1
kV/cm and a common spin Hall conductivity
$\sigma_{x}^{y}=10^{5}~{}(\Omega\cdot{\rm m})$ [22, 23]. The polarization of
pumped spin currents is governed by the angular momentum of optical/microwave
fields, which is thereby tunable and beyond that limited by the magnetization
direction in the spin pumping and the spin-Hall conductivity tensor. The
transfer of optical or microwave spin is, thereby, a unified and efficient
mechanism for distinct mobile carriers.

###### Acknowledgements.

This work is financially supported by the National Key Research and
Development Program of China under Grant No. 2023YFA1406600, the National
Natural Science Foundation of China under Grant No. 12374109, and the startup
grant of Huazhong University of Science and Technology. We thank Gerrit Bauer
for the useful discussions.

## References

  * [1] B. Lenk, H. Ulrichs, F. Garbs, and M. Münzenberg, The building blocks of magnonics, Phys. Rep. 507, 107 (2011).
  * [2] D. Go, D. Jo, H.-W. Lee, M. Kläui, and Y. Mokrousov, Orbitronics: Orbital currents in solids, Europhy. Lett. 135, 37001 (2021).
  * [3] S. Maekawa, T. Kikkawa, H. Chudo, J. Ieda, and E. Saitoh, Spin and spin current-From fundamentals to recent progress, J. Appl. Phys. 133, 020902 (2023).
  * [4] T. Yu, Z. C. Luo, and G. E. W. Bauer, Chirality as generalized spin-orbit interaction in spintronics, Phys. Rep. 1009, 1 (2023).
  * [5] A. Fert, R. Ramesh, V. Garcia, F. Casanova, and M. Bibes, Electrical control of magnetism by electric field and current-induced torques, Rev. Mod. Phys. 96, 015005 (2024).
  * [6] A. V. Chumak, V. I. Vasyuchka, A. A. Serga, and B. Hillebrands, Magnon spintronics, Nat. Phys. 11, 453 (2015).
  * [7] A. Brataas, B. van Wees, O. Klein, G. de loubens, and M. Viret, Spin Insulatronics, Phys. Rep. 885, 1 (2020).
  * [8] A. Barman, G. Gubbiotti, S. Ladak, A. O. Adeyeye, M. Krawczyk, J. Gräfe, C. Adelmann, S. Cotofana, A. Naeemi, V. I. Vasyuchka et al., The 2021 magnonics roadmap, J. Phys.: Condens. Matter 33, 413001 (2021).
  * [9] T. Yu, J. Zou, B. W. Zeng, J. W. Rao, and K. Xia, Non-Hermitian topological magnonics, Phys. Rep. 1062, 1 (2024).
  * [10] K. Y. Bliokh and F. Nori, Transverse and longitudinal angular momenta of light, Phys. Rep. 592, 1 (2015).
  * [11] S. M. Lloyd, M. Babiker, G. Thirunavukkarasu, and J. Yuan, Electron vortices: Beams with orbital angular momentum, Rev. Mod. Phys. 89, 035004 (2017).
  * [12] S. V. Vonsovskii and M. S. Svirskii, Phonon spin, Sov. Phys. Solid State 3, 1568 (1962).
  * [13] L. Zhang and Q. Niu, Angular momentum of phonons and the Einstein-de Haas effect, Phys. Rev. Lett. 112, 085503 (2014).
  * [14] X. Zhang, G. E. W. Bauer, and T. Yu, Unidirectional pumping of phonons by magnetization dynamics, Phys. Rev. Lett. 125, 077203 (2020).
  * [15] A. A. High, A. T. Hammack, J. R. Leonard, S. Yang, L. V. Butov, T. Ostatnický, M. Vladimirova, A. V. Kavokin, T. C. H. Liew, K. L. Campman, and A. C. Gossard, Spin Currents in a Coherent Exciton Gas, Phys. Rev. Lett. 110, 246403 (2013).
  * [16] W. Li, X. Lu, J. Wu, and A. Srivastava, Optical control of the valley Zeeman effect through many-exciton interactions, Nat. Nanotechnol. 16, 148 (2021).
  * [17] R. Schmidt, A. Arora, G. Plechinger, P. Nagler, A. G. del Águila, M. V. Ballottin, P. C. M. Christianen, S. M. de Vasconcellos, C. Schüller, T. Korn, and R. Bratschitsch, Phys. Rev. Lett. 117, 077402 (2016).
  * [18] E. C. Regan, D. Wang, E. Y. Paik, Y. Zeng, L. Zhang, J. Zhu, A. H. MacDonald, H. Deng, and F. Wang, Emerging exciton physics in transition metal dichalcogenide heterobilayers, Nat. Rev. Mater. 7, 778 (2022).
  * [19] Y. Tserkovnyak, A. Brataas, and G. E. W. Bauer, Enhanced Gilbert Damping in Thin Ferromagnetic Films, Phys. Rev. Lett. 88, 117601 (2002).
  * [20] Y. Tserkovnyak, A. Brataas, G. E. W. Bauer, and B. I. Halperin, Nonlocal magnetization dynamics in ferromagnetic heterostructures, Rev. Mod. Phys. 77, 1375 (2005).
  * [21] J. E. Hirsch, Spin Hall Effect, Phys. Rev. Lett. 83, 1834 (1999).
  * [22] J. Sinova, S. O. Valenzuela, J. Wunderlich, C. H. Back, and T. Jungwirth, Spin Hall effects, Rev. Mod. Phys. 87, 1213 (2015).
  * [23] T. Jungwirth, J. Wunderlich, and K. Olejník, Spin Hall effect devices, Nat. Mater. 11, 382 (2012).
  * [24] A. Avsar, H. Ochoa, F. Guinea, B. Özyilmaz, B. J. van Wees, and I. J. Vera-Marun, Colloquium: Spintronics in graphene and other two-dimensional materials, Rev. Mod. Phys. 92, 021003 (2020).
  * [25] W. Gerlach and O. Stern, Der experimentelle Nachweis der Richtungsquantelung im Magnetfeld, Z. Physik 9, 349 (1922).
  * [26] Y. J. Bae, J. Wang, A. Scheie, J. Xu, D. G. Chica, G. M. Diederich, J. Cenker, M. E. Ziebel, Y. Bai, H. Ren, C. R. Dean, M. Delor, X. Xu, X. Roy, A. D. Kent, and X. Zhu, Exciton-coupled coherent magnons in a 2D semiconductor, Nature 609, 282 (2022).
  * [27] G. M. Diederich, J. Cenker, Y. Ren, J. Fonseca, D. G. Chica, Y. J. Bae, X. Zhu, X. Roy, T. Cao, D. Xiao, and X. Xu, Tunable interaction between excitons and hybridized magnons in a layered semiconductor, Nat. Nanotechnol. 18, 23 (2023).
  * [28] N. P. Wilson, K. Lee, J. Cenker, K. Xie, A. H. Dismukes, E. J. Telford, J. Fonseca, S. Sivakumar, C. Dean, T. Cao, X. Roy, X. Xu, and X. Zhu, Interlayer electronic coupling on demand in a 2D magnetic semiconductor, Nat. Mater. 20, 1657 (2021).
  * [29] Y. Ren, M. Rudner, and D. Xiao, Light-Driven Spontaneous Phonon Chirality and Magnetization in Paramagnets, Phys. Rev. Lett. 132, 096702 (2024).
  * [30] E. L. Ivchenko and G. E. Pikus, New photogalvanic effect in gyrotropic crystals, JETP Lett. 27, 604 (1978).
  * [31] A. G. Aronov and Y. B. Lyanda-Geller, Nuclear electric resonance and orientation of carrier spins by an electric field, JETP Lett. 50, 431 (1989).
  * [32] V. M. Edelstein, Spin polarization of conduction electrons induced by electric current in two-dimensional asymmetric electron systems, Solid State Commun. 73, 233 (1990).
  * [33] J. Rioux and J. E. Sipe, Optical injection processes in semiconductors, Phys. E 45, 1 (2012).
  * [34] D. Pan and H. Xu, Polarizing Free Electrons in Optical Near Fields, Phys. Rev. Lett. 130, 186901 (2023).
  * [35] C. Tang, L. Alahmed, M. Mahdi, Y. Xiong, J. Inman, N. J. McLaughlin, C. Zollitsch, T. H. Kim, C. R. Du, H. Kurebayashi, E. J. G. Santos, W. Zhang, P. Li, and W. Jin, Spin dynamics in van der Waals magnetic systems, Phys. Rep. 1032, 24 (2023).
  * [36] J. F. Sierra, J. Fabian, R. K. Kawakami, S. Roche, and S. O. Valenzuela, Van der Waals heterostructures for spintronics and opto-spintronics, Nat. Nanotechnol. 16, 856 (2021).
  * [37] J. Li, C. B. Wilson, R. Cheng, M. Lohmann, M. Kavand, W. Yuan, M. Aldosary, N. Agladze, P. Wei, M. S. Sherwin, and J. Shi, Spin current from sub-terahertz-generated antiferromagnetic magnons, Nature 578, 70 (2020).
  * [38] T. Yu and G. E. W. Bauer, in Chirality, Magnetism and Magnetoelectricity: Separate Phenomena and Joint Effects in Metamaterial Structures, edited by E. Kamenetskii (Springer, Cham, 2021).
  * [39] H. Wang, J. Chen, T. Yu, C. Liu, C. Guo, S. Liu, K. Shen, H. Jia, T. Liu, J. Zhang, M. A. Cabero, Q. Song, S. Tu, M. Wu, X. Han, K. Xia, D. Yu, G. E. W. Bauer, and H. Yu, Nonreciprocal coherent coupling of nanomagnets by exchange spin waves, Nano Res. 14, 2133 (2021).
  * [40] C. Cai, D. M. Kennes, M. A. Sentef, and T. Yu, Edge and corner skin effects of chirally coupled magnons characterized by a topological winding tuple, Phys. Rev. B 108, 174421 (2023).
  * [41] P. Dombi, Z. Pápa, J. Vogelsang, S. V. Yalunin, M. Sivis, G. Herink, S. Schäfer, P. Groß, C. Ropers, and C. Lienau, Strong-field nano-optics, Rev. Mod. Phys. 92, 025003 (2020).
  * [42] C. Girard, C. Joachim, and S. Gauthier, The physics of the near-field, Rep. Prog. Phys. 63, 893 (2000).
  * [43] E. Betzig and R. J. Chichester, Single Molecules Observed by Near-Field Scanning Optical Microscopy, Science 262, 1422 (1993).
  * [44] L. Wang and X. G. Xu, Scattering-type scanning near-field optical microscopy with reconstruction of vertical interaction, Nat. Commun. 6, 8973 (2015).
  * [45] H. Wang, L. Wang, D. S. Jakob, and X. G. Xu, Tomographic and multimodal scattering-type scanning near-field optical microscopy with peak force tapping mode, Nat. Commun. 9, 2005 (2018).
  * [46] T. Vincent, Scanning near-field infrared microscopy, Nat. Rev. Phys. 3, 537 (2021).
  * [47] The local optical field can be equivalently expressed in the form $\mathbf{h}({\boldsymbol{\rho}},t)=\sum_{\mathbf{q}}\left(\mathbf{H}^{*}(\mathbf{-q})e^{i\omega t}+\mathbf{H}(\mathbf{q})e^{-i\omega t}\right)e^{i\mathbf{q}\cdot{\boldsymbol{\rho}}}$.
  * [48] See Supplemental Material […] for the details of the density-matrix approach and time-dependent perturbation theory for the optical spin pumping.
  * [49] J. J. Sakurai and J. Napolitano, Modern Quantum Mechanics Second Edition (Cambridge University Press, Cambridge, UK, 2017).
  * [50] G. D. Mahan, Many Particle Physics (Plenum, New York, 1990).
  * [51] G. F. Giuliani and G. Vignale, Quantum Theory of the electron Liquid (Cambridge University Press, Cambridge, 2005).
  * [52] H. Jiao and G. E. W. Bauer, Spin Backflow and ac Voltage Generation by Spin Pumping and the Inverse Spin Hall Effect, Phys. Rev. Lett. 110, 217602 (2013).
  * [53] D. Wei, M. Obstbaum, M. Ribow, C. H. Back, and G. Woltersdorf, Spin Hall voltages from a.c. and d.c. spin currents, Nat. Commun. 5, 3768 (2014).
  * [54] T. Yu and G. E. W. Bauer, Noncontact Spin Pumping by Microwave Evanescent Fields, Phys. Rev. Lett. 124, 236801 (2020).
  * [55] B. Radisavljevic and A. Kis, Mobility engineering and a metal-insulator transition in monolayer $\mathrm{MoS_{2}}$, Nat. Mater. 12, 815 (2013).
  * [56] L. Wang and M. W. Wu, Electron spin relaxation due to D’yakonov-Perel’ and Elliot-Yafet mechanisms in monolayer $\mathrm{MoS_{2}}$: Role of intravalley and intervalley processes, Phys. Rev. B 89, 115302 (2014).
  * [57] W. S. Yun, S. W. Han, S. C. Hong, I. G. Kim, and J. D. Lee, Thickness and strain effects on electronic structures of transition metal dichalcogenides: 2H-$\mathit{\mathbf{MX}}_{2}$ semiconductors ($\mathit{\mathbf{M}}$=Mo, W; $\mathit{\mathbf{X}}$=S, Se, Te), Phys. Rev. B 85, 033305 (2012).
  * [58] K. Marinov, A. Avsar, K. Watanabe, T. Taniguchi, and A. Kis, Resolving the spin splitting in the conduction band of monolayer $\mathrm{MoS_{2}}$, Nat. Commun. 8, 1938 (2017).
  * [59] M. D. Siao, W. C. Shen, R. S. Chen, Z. W. Chang, M. C. Shih, Y. P. Chiu, and C.-M. Cheng, Two-dimensional electronic transport and surface electron accumulation in $\mathrm{MoS_{2}}$, Nat. Commun. 9, 1442 (2018).
  * [60] R. Asgari and D. Culcer, Unidirectional valley-contrasting photocurrent in strained transition metal dichalcogenide monolayers, Phys. Rev. B 105, 195418 (2022).
  * [61] S. Sharma, P. Elliott, and S. Shallcross, THz induced giant spin and valley currents, Sci. Adv. 9, eadf3673 (2023).
  * [62] W.-Y. Shan, J. Zhou, and D. Xiao, Optical generation and detection of pure valley current in monolayer transition-metal dichalcogenides, Phys. Rev. B 91, 035402 (2015).
  * [63] T. Yu, and M. W. Wu, Valley depolarization due to intervalley and intravalley electron-hole exchange interactions in monolayer $\mathrm{MoS_{2}}$, Phys. Rev. B 89, 205303 (2014).
  * [64] C. Mai, A. Barrette, Y. Yu, Y. G. Semenov, K. W. Kim, L. Cao, and K. Gundogdu, Many-Body Effects in Valleytronics: Direct Measurement of Valley Lifetimes in Single-Layer $\mathrm{MoS_{2}}$, Nano Lett. 14, 202 (2014).
  * [65] Q. Wang, S. Ge, X. Li, J. Qiu, Y. Ji, J. Feng, and D. Sun, Valley Carrier Dynamics in Monolayer Molybdenum Disulfide from Helicity-Resolved Ultrafast Pump–Probe Spectroscopy, ACS Nano 7, 11087 (2013).
  * [66] D. Lagarde, L. Bouet, X. Marie, C. R. Zhu, B. L. Liu, T. Amand, P. H. Tan, and B. Urbaszek, Carrier and Polarization Dynamics in Monolayer $\mathrm{MoS_{2}}$, Phys. Rev. Lett. 112, 047401 (2014).
  * [67] T. Cheiwchanchamnangij and W. R. L. Lambrecht, Quasiparticle band structure calculation of monolayer, bilayer, and bulk $\mathrm{MoS_{2}}$, Phys. Rev. B 85, 205302 (2012).
  * [68] A. V. Stier, K. M. McCreary, B. T. Jonker, J. Kono, and S. A. Crooker, Exciton diamagnetic shifts and valley Zeeman effects in monolayer $\mathrm{WS_{2}}$ and $\mathrm{MoS_{2}}$ to 65 Tesla, Nat. Commun. 7, 10643 (2016).
  * [69] P. Vancsó, G. Z. Magda, J. Pető, J.-Y. Noh, Y.-S. Kim, C. Hwang, L. P. Biró, and L. Tapasztó, The intrinsic defect structure of exfoliated $\mathrm{MoS_{2}}$ single layers revealed by Scanning Tunneling Microscopy, Sci. Rep. 6, 29726 (2016).