# Molecular Identification and Peak Assignment: Leveraging Multi-Level
Multimodal Alignment on NMR

Hao Xu1,† Zhengyang Zhou1,† Pengyu Hong1,∗
1 Department of Computer Science
Brandeis University
Waltham, MA 02453
† Equal Contribution * Corresponding Author
<EMAIL_ADDRESS>

###### Abstract

Nuclear magnetic resonance (NMR) spectroscopy plays an essential role across
various scientific disciplines, providing valuable insights into molecular
dynamics and interactions. Despite the promise of AI-enhanced NMR prediction
models, challenges persist in the interpretation of spectra for tasks such as
molecular retrieval, isomer recognition, and peak assignment. In response,
this paper introduces Multi-Level Multimodal Alignment with Knowledge-Guided
Instance-Wise Discrimination (K-M3AID) to establish meaningful correspondences
between two heterogeneous modalities: molecular graphs (structures) and NMR
spectra. In particular, K-M3AID employs a dual-coordinated contrastive
learning architecture, and incorporates a graph-level alignment module, a
node-level alignment module, and a communication channel. Notably, the
framework introduces knowledge-guided instance-wise discrimination into
contrastive learning within the node-level alignment module, significantly
enhancing accuracy in cross-modal alignment. Additionally, K-M3AID showcases
its capability of meta-learning by demonstrating that skills acquired during
node-level alignment positively impact graph-level alignment. Empirical
validation underscores K-M3AID’s effectiveness in addressing multiple zero-
shot tasks, offering a promising solution to bridge the gap between structural
information and spectral data in complex NMR scenarios.

##  1 Introduction

Nuclear magnetic resonance (NMR) spectroscopy has found a board applications
in various scientific domains, such as organic chemistry, pharmaceuticals,
biochemistry, food science, and material science, by providing insights into
molecular dynamics and interactions (Gunther & Gunther (1994); Claridge
(2016); Yu et al. (2021)). The positions and shapes of signal peaks in NMR
spectra, influenced by through-bond and through-space interactions, serve as
”fingerprints” for deducing atomic connectivity, relative stereochemistry, and
conformations. The conventional approach for elucidating molecular structures
and attributing peaks has long relied on manual determination by organic
chemists (Guan et al. (2021)). However, the interpretation of NMR spectra is
not straightforward, particularly when dealing with isomers and complex
molecules consisting of multiple stereogentic (chiral) centers (Wu et al.
(2023); Chhetri et al. (2018)). Even an expert chemist may encounter
significant difficulties in accurately assigning isomeric compounds with
extremely similar NMR spectra due to this complexity (Nicolaou & Snyder
(2005)).

Recent advancements in AI-enhanced NMR prediction models show promise in
generating spectra from candidate structures (Chen et al. (2020); Jonas et al.
(2022); Kuhn (2022)). However, real-world scenarios often demand spectral
interpretation before detailed structural information is available. Ideally,
the interpretation of spectra should transition from spectral data to
structural elucidation. Consequently, these prediction models are not
applicable in highly demanding real-world tasks, such as molecular retrieval,
candidate ranking, and peak assignment (see Figure 1.a). In scenarios with
limited knowledge of an unknown compound, such as identifying a compound from
a plant, molecular retrieval is crucial, as the retrieved analogs can aid in
structural hypothesis formulation. In cases where pre-existing knowledge is
available, such as identifying reaction products, the quantitative ranking of
candidate possibilities becomes essential. Peak assignment represents the
decisive stage in the structural identification process, providing detailed
insights into the local atomic context. These prediction models fall short in
identifying compounds without prior knowledge, quantitatively assessing
candidates (especially in isomer recognition), and guiding accurate peak
assignments. How might one employ or interrogate these advanced AI
methodologies to enhance molecular retrieval, candidate ranking, and peak
assignment in NMR spectra interpretation?

NMR spectroscopy can be viewed as a modality, capturing molecular structural
and electronic details through an NMR spectrometer and expressing this
information in sequence format (chemical shifts). On the other hand, these
structural and electronic information can also be presented in a molecular
graph, which encapsulates molecular attributes in a graph-based format. The
tasks of determining molecular structure and assigning peaks require clear
correspondence across these two heterogeneous modalities: NMR spectra and
molecular graphs. This imperative is met by multimodal learning, particularly
multimodal alignment (see fundamentals of multimodal alignment in Preliminary
2). However, multimodal alignment commonly encounters challenges in addressing
data heterogeneity, semantic gaps among modalities, collecting paired data,
designing effective cross-modal attention mechanisms, and formulating
appropriate evaluation metrics (Liang et al. (2023); Jabeen et al. (2023)).

Molecules is composed of atoms, and each molecule, with its unique arrangement
of atoms and bonding patterns, gives rise to a distinct spectrum.
Consequently, it is impractical to incorporate all molecules and their spectra
in the training dataset. One of the potential solutions is to take into
account the constrained possibilities of atomic diversity for interpreting
spectra. This, in turn, motivates us to leverage the power of AI for
interpreting spectra at the atomic level, establishing a solid foundation for
molecular identification. This approach aligns tightly with the core
principles of meta-learning (see principles of meta-learning in Preliminary
2). However, meta-learning typically concerns in meta-overfitting, distance
evaluation metrics, model interpretablity, transferability and generalization
(Huisman et al. (2021); Feng et al. (2022); Ma et al. (2022); Vettoruzzo et
al. (2023)).

In view of these challenges and opportunities, we propose a novel framework
K-M3AID (Multi-Level Multimodal Alignment with Knowledge-Guided Instance-Wise
Discrimination), aiming at achieving reliable analog retrieval, candidate
ranking, and peak assignment in the interpretation of NMR spectra (see Figure
1.b). The overview of our K-M3AID framework is a dual-coordinated contrastive
learning architecture, which contains three key components: graph-level
alignment module, node-level alignment module, and communication channel.
Graph-level alignment module establishes the correspondences of molecules with
their individual 13C NMR spectra. Given that each unique molecule gives rise
to a distinct spectral signature, this module is equipped with a
straightforward cross-entropy loss designed for effective contrastive
learning. Node-level alignment module aligns each C atom within the molecules
with their signal peaks on the spectrum. In contrast to the diverse and
distinctive molecular spectral signatures, many atoms exhibit chemical
symmetric and magnetic equivalence within the same molecule, corresponding to
the same peaks. Meanwhile, atoms with different local surroundings can still
presents significant similarity on the spectrum, which introduces heightened
level of complexity. In view of these complex scenarios, we come up with
knowledge-guided instance wise discrimination based contrastive learning in
graph-level alignment module (see Figure 4). The communication channel
dynamically adjusts the flow of gradients between the node encoder and the
graph encoder from two modules during the training process.

In summary, our contribution comprises three major aspects: Conceptually: we
integrate two architectural levels alignment, graph- and node-levels, into the
K-M3AID framework. This integration facilitates rapid adaptation,
significantly boosting the efficiency of learning for multimodal zero-shot
tasks. Methodologically: We present knowledge-guided instance wise
discrimination for cross-modal contrastive learning, which take advantage of
continuous and domain-specific features with natural ordering. To the best of
our knowledge, this is the first to demonstrate knowledge-guided instance wise
discrimination based cross-modal contrastive learning, which transforms a
discrete comparison into a continuous one. Empirically: We demonstrate the
effectiveness of K-M3AID through its successful application to multiple zero-
shot tasks, including molecular retrieval, isomer recognition, and peak
assignment.

Figure 1: a. Demands for interpreting NMR spectra in real-world scenarios:
molecular retrieval, candidate ranking, and peak assignment; b. Zero-shot
applications of the K-M3AID model: molecular retrieval, isomer recognition,
and peak assignment; c. The framework of K-M3AID model: the molecular
alignment module is responsible for optimizing the the correspondence between
modalities at the molecular level, the atomic alignment module focus on the
fine-tuning of atomic positioning on the spectrum, and the communication
channel dynamically adjusts the flow of gradients between node encoder and
graph encoder during the training process. S for spectrum embedding, G for
graph embedding, P for peak embedding and N for node embedding.

##  2 Preliminaries

Multimodal Alignment: Multimodal alignment, as defined in the literature
(Baltrusaitis et al. (2017)), involves establishing relationships and
correspondences among sub-components of instances from two or more modalities.
A typical example is identifying specific regions in an image that correspond
to words or phrases in a given caption (Karpathy & Fei-Fei (2015)). This
approach offers numerous benefits, including enhanced data interpretation,
heightened accuracy and robustness, overcoming limitations of single-modal
systems, and better addressing real-world complexity (Baltrusaitis et al.
(2017), Summaira et al. (2021), Akkus et al. (2023)). CLIP (Contrastive
Language-Image Pretraining) (Radford et al. (2021); Li et al. (2021)) is one
of the most widely adopted frameworks for multimodal alignment. As highlighted
in the introduction, molecular information originates from diverse sources
such as molecule graphs and NMR spectroscopy. To leverage effective alignments
of this multifaceted information across different modalities, we adopt CLIP
framework with graph neural networks (GNN) (Xu et al. (2018), Wu et al.
(2022)) to encoder molecular information and neural network encoders (Serra et
al. (2018)) to encapsulate NMR information.

Meta-Learning: Meta-learning is defined as learning how to learn across tasks
(Vilalta & Drissi (2002)). More specifically, it leverages skills previously
acquired from related tasks to the current one (Lake et al. (2017). With more
skills learned, learning new skills becomes easier, requiring fewer examples
and less trial-and-error (Vanschoren (2018); Finn et al. (2017)). A meta-
learner is trained on a diverse set of object recognition tasks. During this
training, it learns common features, patterns, and strategies for recognizing
objects. Once trained, when presented with a new, previously unseen object
category, the meta-learner can rapidly adapt and achieve high recognition
accuracy, leveraging the knowledge it acquired from the diverse training tasks
to perform in this novel recognition task (Finn et al. (2017). A profound
understanding of atom properties allows us to extend our vision to previously
unseen molecules, aligning with the principles of meta-learning in artificial
intelligence.

Contrastive Learning: Contrastive learning focus on learning how to
distinguish between similar and dissimilar items (Le-Khac et al. (2020b);
Jaiswal et al. (2021); Liu et al. (2021)). A fundamental aspect of this
process involves instance-wise discrimination (Wu et al. (2018)). Models
incorporating instance-wise discrimination not only foster an understanding of
the inherent data structure but also enhance generalization capabilities. This
is attributed to the contrastive learning approach, which prioritizes
distinguishing between instances rather than memorizing specific labeled
examples. In chemistry, each molecule/atom is treated as a distinct instance.
The learning algorithm focuses on distinguishing each molecule/atom based on
its context.

##  3 Experiments

To thoroughly evaluate the performance of K-M3AID, we compare it with other
baselines through various zero-shot downstream tasks, such as molecular
retrieval, isomer recognition and peak assignment.

###  3.1 Experimental Setting

####  3.1.1 Pre-training

Dataset: We use 13C NMR spectra of about 20,000 molecules sourced from
nmrshiftdb2 (Steinbeck et al. (2003)), a public access database that contains
NMR spectra of organic molecules. In the collected dataset, molecule are
aligned with their respective 13C NMR spectra, and atomic alignments with
peaks are also included. Notably, the dataset contains 12,771 molecules with
fewer than 10 carbon atoms, 7,043 molecules featuring carbon atom counts
ranging from 10 to 20, and 1,138 molecules incorporating more than 20 carbon
atoms. The quality of the dataset was further validated by experienced organic
chemists. We randomly sample 80% of the molecules for training and the rest
for evaluation.

Training: We concurrently leverage both graph- and node-level alignment tasks
in the pre-training of K-M3AID. Graph-level alignment focuses on aligning
molecules with their spectra, accompanied by the utilization of cross-entropy
loss for contrastive purposes. On the other hand, node-level alignment entails
aligning atoms with their corresponding peaks, implemented through knowledge-
guided instance-wise discrimination to achieve contrastive loss. A diverse set
of molecular features is employed for training, including atomic number (node
feature), chiral tags (node feature), hybridization (node feature), bond types
(edge feature), and bond direction (edge feature). The spectral features are
derived from peak intensity, peak position (chemical shift measured in ppm),
and peak type.

####  3.1.2 Zero-Shot Molecular Retrieval

Dataset: We randomly collected 1 million molecules from PubChem (Kim et al.
(2023)) to form a molecular reference library. Subsequently, we carefully
selected 1000 spectra, ensuring that they had not appeared in the training
dataset, from an external dataset to serve as query spectra. Following this,
the corresponding molecules associated with these 1000 spectra were added into
the existing reference library.

Evaluation: We perform molecular retrieval using each of the selected spectra
to determine if the correct corresponding molecular entity can be retrieved
from the reference library. The model’s performance is assessed at top-1%
accuracy, as well as at accuracy of 5%, 10%, and 25%.

####  3.1.3 Zero-Shot Isomer Recognition

Dataset: We categorize isomers from the validation dataset to guarantee their
absence from the training dataset. To assess effectiveness, we perform isomer
recognition on each isomer group containing at least 10 molecules. Within the
same group, isomers may be structural or spatial isomers of each other.
Structural isomers refer to molecules with the same molecular formula but
different structural arrangements of atoms, resulting in distinct chemical
structures. On the other hand, spatial isomers, also known as stereoisomers,
have the same molecular formula and arrangement of atoms but differ in the
spatial orientation of their atoms in three-dimensional space, leading to
different stereoisomeric forms. (For an elucidation of an isomer group and in-
depth insights into isomers with NMR, please refer to the details provided in
Appendix D.)

Evaluation: We conduct isomer recognition for each isomer group, aiming to
assess the correct alignment of each spectrum with its respective molecule
within each isomer group.

####  3.1.4 Zero-Shot Peak Assignment

Dataset: The dataset utilized for evaluating the overall performance of
K-M3AID on zero-shot peak assignment is the validation dataset from the pre-
training phase. In order to highlight the capabilities of K-M3AID in zero-shot
peak assignment, the case studies include complex natural products featuring
multiple fused rings, stereogenic (chiral) centers, and symmetric structures.

Evaluation: We conduct peak assignment within each molecule, aiming to assess
the accurate alignment of each atom with its corresponding peak on the
spectrum. It’s important to note that this alignment process is confined to
each individual molecule and not across different molecules.

####  3.1.5 Baselines

In contrast to communicative mechanism of K-M3AID, one of the baselines is
established without the utilization of a communication channel. Additionally,
we explore two alternative baselines where the knowledge-guided instance-wise
discrimination in the node-level alignment module is replaced with either
strong-pair-based (SP-ID) or weak-pair-based (WP-ID) instance-wise
discrimination. SP-ID enforces a precise match in node-level (atom-peak)
alignment, ensuring that only correct pairs established during the training
process are considered. On the other hand, WP-ID broadens the matching
criteria, allowing for multiple matches within a specified threshold set for
the distance of their corresponding parts per million (ppm, referenced in
Section 6.4). The mathematical definitions of strong pair and weak pair are as
follows:

$\textit{Strong Pair: }|ppm_{i}-ppm_{j}|=0,$ (1) $\textit{Weak Pair:
}|ppm_{i}-ppm_{j}|\leq th,$ (2)

Here, $A_{i}\in A$ represents atoms, $P_{j}\in P$ represents peaks, and $th$
is the abbreviation for the threshold. Additionally, $ppm_{i}$ denotes the
parts per million for the $i^{th}$ Carbon Atom, and $ppm_{j}$ represents the
corresponding parts per million for the $j^{th}$ peak.

###  3.2 Results

####  3.2.1 Validation Performance

The K-M3AID model demonstrates an impressive validation accuracy of 95.5% for
molecule-spectrum alignments within the graph-level alignment module. In
direct comparison, K-M3AID surpasses alternative models such as SP-ID, WP-ID,
and the model without a communication mechanism, exhibiting superior
performance by a margin ranging from about 1% to 6% in the graph-level
alignment module (refer to Table 1). Notably, SP-ID outperforms WP-ID
significantly, and as the matching criteria threshold widens, the performance
of the latter degrades.

In the context of peak-atom alignment within the node-level alignment module,
K-M3AID and the model without a communication mechanism exhibit comparable
accuracies (refer to Table 1). However, K-M3AID showcases slightly better
stability across 5-fold cross-validation. Moreover, K-M3AID demonstrates
superiority in peak-atom alignment when compared to SP-ID and WP-ID. This
superiority may arise from the inherent limitations of both strong and weak
pair definitions, which is failing to precisely calibrate the diverse
relationships among the elements. This finding is further supported by the
significant decreases in the accuracy of peak-atom alignment as the threshold
of weak pair increases.

Table 1: Batch-wise validation accuracy (%) of K-M3AID and baselines with $epochs=200$. Alignment | SP-ID |  WP-ID(th=1) |  WP-ID(th=5) |  WP-ID(th=10) | No Comm. | K-M3AID
---|---|---|---|---|---|---
Graph-Level |  93.5$\pm$0.6 |  91.3$\pm$0.8 |  90.3$\pm$0.6 |  88.4$\pm$1.4 |  94.6$\pm$0.4 | 95.5$\pm$0.4
Node-Level |  89.3$\pm$0.4 |  83.7$\pm$0.6 |  79.8$\pm$0.5 |  66.1$\pm$2.5 |  90.4$\pm$0.2 | 90.3$\pm$0.1

####  3.2.2 Performance on Zero-Shot Molecular Retrieval

We conduct a systematic evaluation of the effectiveness of our K-M3AID model,
comparing it with baseline models in the zero-shot molecular retrieval task
across datasets of varying magnitudes. Detailed results are presented in Table
2. The K-M3AID model consistently attains an impressive top-1 accuracy of
approximately 95.8% in molecular retrieval when the molecular reference
library comprises 100 entries. This performance surpasses that of alternative
mechanisms such as SP-ID (95.3%), WP-ID (92.9%), and No Communication (94.8%).
As the molecular reference library expands to 1000 entries, the K-M3AID model
exhibits notable superiority, achieving accuracy levels of 80.4%, 1.8%, 8.7%,
and 2.8% higher than SP-ID, WP-ID, and No Communication mechanisms,
respectively. The advantage of K-M3AID becomes even more pronounced when the
library size reaches 10,000 entries. In this scenario, K-M3AID yields 46.3% at
top-1 accuracy, showcasing advancements of 10.5%, 13.6%, and 6.2% over SP-ID,
WP-ID, No Communication mechanisms, respectively. Even with larger molecular
reference libraries, such as 100,000 and 1,000,000 entries, K-M3AID
consistently outshines SP-ID, WP-ID, and No Communication mechanisms. These
compelling results distinguish the K-M3AID model as an exceptional choice in
scenarios demanding robust performance in molecular retrieval tasks.

Table 2: Zero-shot molecular retrieval at top 1(%) accuracy across datasets of varying sizes. (For more statistics, such as top 5(%), top 10(%) and top 25(%), please refer to Appendix Table C.1.) Method | $10^{2}$ | $10^{3}$ | $10^{4}$ | $10^{5}$ | $10^{6}$
---|---|---|---|---|---
K-M3AID |  95.8$\pm$1.0 |  80.4$\pm$3.9 |  46.3$\pm$1.2 |  18.0$\pm$0.8 |  5.8$\pm$1.7
SP-ID |  95.3$\pm$0.8 |  78.6$\pm$2.7 |  35.8$\pm$3.8 |  12.9$\pm$1.6 |  3.4$\pm$0.9
WP-ID (th = 1) |  92.9$\pm$0.6 |  71.7$\pm$1.0 |  32.7$\pm$1.3 |  10.7$\pm$0.5 |  3.6$\pm$0.7
No Comm. |  94.8$\pm$1.2 |  77.6$\pm$1.4 |  40.1$\pm$1.2 |  14.4$\pm$0.9 |  4.1$\pm$1.1

####  3.2.3 Performance on Zero-Shot Isomer Recognition

K-M3AID stands out prominently when compared to SP-ID, WP-ID, and no
communication approaches in the task of zero-shot isomer recognition,
achieving an exceptional 100% accuracy across given groups of isomers (refer
to Table 3). These empirical observations underscore the advantages of K-M3AID
in the context of isomer recognition. The superiority of K-M3AID over the no
communication baseline demonstrates the positive impact of node-level
alignment on graph-level alignment, emphasizing the potency of meta-learning.

Table 3: Zero-shot isomer recognition accuracy (%) of K-M3AID and baselines. Formula | #Isomers | SP-ID | WP-ID (th=1) | No Comm. | K-M3AID
---|---|---|---|---|---
$\text{C}_{4}\text{H}_{6}\text{O}$ | 15 | 86.7 | 86.7 | 86.7 | 100.0
$\text{C}_{9}\text{H}_{9}\text{N}$ | 15 | 86.7 | 80.0 | 100.0 | 100.0
$\text{C}_{7}\text{H}_{11}\text{NO}_{3}$ | 14 | 78.6 | 85.7 | 85.7 | 100.0
$\text{C}_{6}\text{H}_{13}\text{NO}$ | 23 | 91.3 | 91.3 | 100.0 | 100.0
$\text{C}_{8}\text{H}_{7}\text{NO}_{4}$ | 13 | 92.3 | 84.6 | 92.3 | 100.0
$\text{C}_{15}\text{H}_{24}\text{O}$ | 16 | 93.8 | 93.8 | 100.0 | 100.0
$\text{C}_{11}\text{H}_{14}$ | 10 | 90.0 | 80.0 | 70.0 | 100.0
$\text{C}_{7}\text{H}_{15}\text{NO}$ | 14 | 85.7 | 85.7 | 100.0 | 100.0
$\text{C}_{10}\text{H}_{16}\text{O}_{2}$ | 26 | 92.3 | 84.6 | 100.0 | 100.0
$\text{C}_{8}\text{H}_{15}\text{N}$ | 11 | 81.8 | 90.9 | 100.0 | 100.0

####  3.2.4 Performance on Zero-Shot Peak Assignment

K-M3AID model demonstrates a validation accuracy surpassing 90% for peak
assignment (peak-atom alignment) within the node-level alignment module after
200 epochs (see Figure 2.A). Notably, the model achieves a 100% accuracy rate
in 74.1% of molecules containing fewer than 10 carbon atoms (see Figure 2.B).
For molecules with carbon atom counts ranging from 10 to 20, the model attains
100% accuracy in 37.2% of cases (see Figure 2.C). Furthermore, it achieves an
accuracy exceeding 80% in more than 50% of cases pertaining to molecules
containing more than 20 carbon atoms (see Figure 2.D). Additionally, to
further illustrate the power of K-M3AID on peak assignment, we present two
complex natural product molecules featuring multiple rings (4 and 4,
respectively) and stereogenic (chiral) centers (6 and 8, respectively) in
Figure E.1.

Figure 2: The statistics of zero-shot peak assignment.

K-M3AID demonstrates superior performance in peak assignment compared to SP-ID
and WP-ID. Our case studies reveal that the limitations of SP-ID and WP-ID
become particularly pronounced in two scenarios: 1) when local contexts of
specific atoms exhibit a high degree of similarity; 2) when certain atoms
display symmetric mapping within the same molecule. In the former scenario,
exemplified by molecular A in Figure 3, atom 0 and atom 4 are secondary
carbons (attaching to 2 carbons and 2 hydrogens), nearly symmetric on the same
5-member ring, corresponding to the peak position measured in ppm of 27.0 and
29.8, respectively. (For the definition of ppm, please refer to Section 6.4.)
The similar local content of these two atoms fools SP-ID and WP-ID. Meanwhile,
atom 1 and atom 3 are tertiary carbons (attaching to 3 carbons and 1
hydrogen), nearly symmetric on the same 5-member ring, corresponding to the
peak position measured in ppm of 54.5 and 44.1, respectively. Only WP-ID fails
to distinguish and align them. In the later scenario, exemplified by molecular
B in Figure 3, there exist instances one-to-one and one-to-many for atomic-
level alignment within the molecular configuration. Both SP-ID and WP-ID
method misaligns certain atoms with other atoms with small ppm differences
(less than 3 ppm in this case), rather than aligning them with themselves or
their symmetric counterparts. In contrast, K-M3AID approach excels in both
scenario by discerning each one of the atoms, which is attributed to the full
utilization of ppm difference distance learning. (For additional cases, please
refer to Appendix Figure E.2)

Figure 3: Case study of peak assignment. Yellow cells in PPM difference
represent the ground truth alignment, and red cross represents the wrong
alignment. For the definition of ppm, please refer to 6.4. For additional
cases, please refer to Appendix Figure E.2

##  4 Related Work

Multimodal Instance-Wise Discrimination: As mentioned in the prelimtaries,
instance discrimination (Le-Khac et al. (2020a); Zolfaghari et al. (2021);
Morgado et al. (2021); Liu et al. (2023)), an important part of contrastive
learning, distinguishes individual instances without explicit class labels.
Moving into multimodal contrastive learning, it can be categorized into two
general approaches: strong-pair-based (van den Oord et al. (2019); Jaiswal et
al. (2021); Liu et al. (2023)) and weak-pair-based (Salakhutdinov & Hinton
(2007); Frosst et al. (2019); Liang et al. (2021)) instance-wise
discrimination. The strong-pair-based such as NCE method enforces a precise
one-to-one correspondence for real samples with artificially generated noise
samples. An example of positive pair can be a noise-added picture of zebra
with the text description of zebra. Instead of one-to-one correspondences,
weak-pair-based approach relaxes the positive pairs to more boarder semantic
correspondences. An example of positive pair can be a picture of zebra with
the text description of horse, but not with the text description of tiger.

Multimodal Meta-Alignment: Within the scope of multimodal alignment,
multimodal meta-alignment is a novel method to align representation spaces
using paired cross-modal data with different similarity while ensuring quick
generalization to new tasks across different modalities (Liang et al. (2021)).
This approach can be observed at different levels, including the intermediate
and fundamental (irreducible) element level. Examples of this method at the
intermediate level can be seen in research on Cross-Modal Generalization (Chen
et al., 2017; Li et al., 2020; Liang et al., 2021; Zhang et al., 2021) and
Livestreaming Product Recognition (Yang, 2023). While these studies showcase
how multimodal meta-alignment operates at a broad objective level, the
application of multimodal meta-alignment at the most fundamental element level
remains underexplored in current research.

##  5 Conclusion and Future Work

In this paper, we introduced K-M3AID (Knowledge-Guided Multi-Level Multimodal
Alignment with Instance-Wise Discrimination) framework, incorporating graph-
level alignment and node-level alignment. Its effectiveness was demonstrated
through multiple zero-shot tasks: molecular retrieval, isomer recognition and
peak assignment. The significance of knowledge-guided instance-wise
discrimination is underscored through various metrics and case studies.
Moreover, the findings from molecular retrieval and isomer recognition
underscored the favorable influence of the node-level alignment on the graph-
level alignment. It emphasizes the successful integration of meta-learning
within our hierarchical alignment framework. While our framework achieves an
atomic-level alignment overall accuracy of 100% for 55% of cases, it drops
significantly to 9.8% when handling molecules with more than 20 carbon atoms.
Presently, our graph encoder operates on 2D molecular graphs with basic node
and edge features. This implementation potentially constrains its ability to
generate precise node embeddings for distinguishing atoms in highly complex
scenarios. Future developments could benefit from incorporating a 3D-based
graph, holding substantial potential to enhance performance in such complex
situations.

##  6 Our Method

In this section, we firstly present the architecture of K-M3AID framework, an
end-to-end system designed for multi-level multimodal alignment. Then, we
introduce the constrastive learning loss in K-M3AID along with the principles
of Knowledge-Guided Instance-Wise Discrimination.

###  6.1 Architecture

The K-M3AID framework is a dual-CLIP architecture (see Figure 1), which
consists of three critical components: graph-level alignment module, node-
level alignment module and communication channel. The graph-level alignment
module adapts a gradient-asymmetric CLIP mechanism. While two unimodal
encoders work in conjunction, only the from-scratch graph encoder (GIN, Xu et
al. (2018)) undergoes dynamic training throughout the process, the pre-trained
spectrum encoder (Yang et al. (2021)) remains fixed. Both encoders are
complemented by dedicated projection layers, which facilitate the mapping of
embeddings into a joint space. The node-level alignment module adapts a
gradient-symmetric CLIP mechanism. It is equipped with two from-scratch
unimodal encoders, node encoder and peak encoder, as well as their dedicated
projection layers. The graph encoder in graph-level alignment module shares
part of the weights with node encoder in node-level alignment module, serving
as the communication channel.

###  6.2 Contrastive Learning Loss

The synergy between these two modules is pivotal, collectively contributing to
our loss function, expressed as

$L=CL_{graph}+CL_{node},$ (3)

where $CL_{graph}$ represents the contrastive learning loss in the graph-level
alignment module by Equation 5, and $CL_{node}$ represents the contrastive
learning loss in the node-level alignment module by Equation 12.

Let $i$ denote the $i^{th}$ instance, and $j$ denote the $j^{th}$ instance.
Then $x_{i}$ denotes the raw input in modality A for the $i^{th}$ instance and
$y_{j}$ denotes the raw input in modality B for the $j^{th}$ instance. Suppose
$f_{x}\left(\cdot\right)$ represent the encoding function for modality A, and
$f_{y}\left(\cdot\right)$ denote the encoding function for modality B. In
graph-level alignment module, these two encoding functions, should map $x_{i}$
and $y_{j}$ to a proximate location in the joint embedding (inter-modality) if
$i=j$.

$\displaystyle CL_{graph}(i)$
$\displaystyle=-\log\frac{e^{\delta(x_{i},y_{i})}}{\displaystyle\sum_{1\leq
j\leq N}e^{\delta(x_{i},y_{j})}}$ (4)
$\displaystyle=-\text{log}(\text{softmax}(\delta(x_{i},y_{i}))$ (5)

Where $\delta(x_{i},y_{j})=\left(f_{x}(x_{i})^{T}\cdot f_{y}(y_{j})\right)$,
$N$ is the total number of instances from the current batch.

Thus, the total $CL_{graph}$ is expressed as following:

$\displaystyle CL_{graph}=\frac{1}{N}\displaystyle\sum_{1\leq i\leq
N}CL_{graph}(i)$ (6)

This design for the loss aims to match the same instance cross different
modalities.

###  6.3 Knowledge-Guided Instance-Wise Discrimination Contrastive Learning

Figure 4: Knowledge span mechanism. $K_{i}$ and $K_{j}$ represents the
corresponding knowledge span label for $i^{th}$ and $j^{th}$ items.

Knowledge Span, in which we define as some continuous and domain-specific
feature exhibiting natural ordering and offering guidance, can potentially
offer valuable insights into the contrastive learning process. As such, we
propose a novel and general approach to contrastive learning called Knowledge-
Guided Instance-Wise Discrimination (see Figure 4). This approach expands the
scope of contrastive learning from confined comparisons (pre-determined
negative and positive pairs) to unrestricted comparisons (no need for pre-
determination). This extension removes the necessity of explicitly defining
such pairs, thus mitigating the potential introduction of human bias.

Suppose $\mathcal{M}$ is the set of instances.
$\mathcal{A}\subset\mathbb{R}^{d_{1}}$ is the set of tunable instances’
embeddings in modality A, $\mathcal{B}\subset\mathbb{R}^{d_{1}}$ is the set of
tuable instances’ embeddings in modality B, and
$\mathcal{K}\subset\mathbb{R}^{d_{2}}$ is the corresponding fixed knowledge
span label that can guide the relative distance learning between components in
$\mathcal{A}$ and $\mathcal{B}$. Thus, the size of $\mathcal{A}$,
$\mathcal{B}$, $\mathcal{K}$ are $|\mathcal{M}|$, respectively.

Let $\mathcal{A}_{i}$ be the $i^{th}$ instance embedding of $\mathcal{A}$, and
$\mathcal{B}_{j}$ be the $j^{th}$ instance embedding of $\mathcal{B}$. We
define the distance function between $\mathcal{A}_{i}$ and $\mathcal{B}_{j}$
as
$d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})=\mathcal{A}_{i}\cdot\mathcal{B}_{j}\rightarrow\mathbb{R}^{+}$,
and calibration function
$d(\mathcal{K}_{i},\mathcal{K}_{j})\rightarrow\mathbb{R}^{+}$ with a monotonic
property and constraint
$\sum_{j=1}^{|\mathcal{M}|}d(\mathcal{K}_{i},\mathcal{K}_{j})=1$, in which
$\mathcal{K}_{i}$ and $\mathcal{K}_{j}$ serve as the designated Knowledge Span
Label. We introduce the Knowledge Span Guided Loss (KSGL) as follows:

$\displaystyle KSGL(i)$
$\displaystyle=-\displaystyle\sum_{\begin{subarray}{c}1\leq
j\leq|\mathcal{M}|\end{subarray}}d(\mathcal{K}_{i},\mathcal{K}_{j})\log\frac{e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})}}{\displaystyle\sum_{\begin{subarray}{c}1\leq
k\leq|\mathcal{M}|\end{subarray}}e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{k})}}$
(7) $\displaystyle=-\displaystyle\sum_{\begin{subarray}{c}1\leq
j\leq|\mathcal{M}|\end{subarray}}d(\mathcal{K}_{i},\mathcal{K}_{j})\log(\text{softmax}(d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})))$
(8)

In particular, when it reaches ideal optimum,
$d(\mathcal{K}_{i},\mathcal{K}_{j})$ and
$d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})$ reaches the following relation:

$d(\mathcal{K}_{i},\mathcal{K}_{j})=\text{softmax}(d_{E}(\mathcal{A}_{i},\mathcal{B}_{j}))$
(9)

For detail proof, please refer to Appendix A. As a result, the corresponding
$CL_{instance}$ is expressed as following:

$\displaystyle CL_{instance}$
$\displaystyle=\frac{1}{|\mathcal{M}|}\displaystyle\sum_{1\leq
i\leq|\mathcal{M}|}KSGL(i)$ (10)

###  6.4 Chosen Knowledge Span-ppm

13C NMR uncovers molecular structures by providing the chemical environments
of carbon atoms and their magnetic responses to external fields, and
quantifies these features in parts per million (ppm) relative to a reference
compound like tetramethylsilane (TMS), simplifying comparisons across
experiments. Thus, continuous peak positions, measured in ppm, can serve as a
robust knowledge span to facilitate instance-wise discrimination for this
contrastive learning task.

For node-level alignment module, $\mathcal{A}$ is the set of node embeddings
for Carbon atoms in molecular graph modality and $\mathcal{B}$ is the set of
peak embeddings for respective Carbon atoms in NMR modality, $\mathcal{K}$ is
the set of ppm value for each corresponding Carbon atom in $\mathcal{A}$ and
$\mathcal{B}$. Suppose $ppm_{i}$ is the ppm for the $i^{th}$ Carbon Atom and
$ppm_{j}$ is the corresponding ppm for $j^{th}$ peak. $d(\cdot,\cdot)$ is then
defined as following:

$\displaystyle d(\mathcal{K}_{i},\mathcal{K}_{j})=d(ppm_{i},ppm_{j})$
$\displaystyle=softmax(\frac{\tau_{2}}{|ppm_{i}-ppm_{j}|+\tau_{1}})$ (11)

where $\tau_{1}$ and $\tau_{2}$ are temperature hyper-parameter. For further
discussion of selection about $\tau_{1}$ and $\tau_{2}$, please refer to
Appendix B.2. Then, the final form of contrastive loss for node-level
alignment according to Equation 8 and Equation 10 is as following:

$CL_{node}=-\frac{1}{|\mathcal{M}|}\sum_{\begin{subarray}{c}1\leq
j\leq|\mathcal{M}|\end{subarray}}d(ppm_{i},ppm_{j})\log\frac{e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})}}{\sum_{\begin{subarray}{c}1\leq
k\leq|\mathcal{M}|\end{subarray}}e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{k})}}$
(12)

## References

  * Akkus et al. (2023) Cem Akkus, Luyang Chu, Vladana Djakovic, Steffen Jauch-Walser, Philipp Koch, Giacomo Loss, Christopher Marquardt, Marco Moldovan, Nadja Sauter, Maximilian Schneider, Rickmer Schulte, Karol Urbanczyk, Jann Goschenhofer, Christian Heumann, Rasmus Hvingelby, Daniel Schalk, and Matthias Aßenmacher.  Multimodal deep learning, 2023.
  * Baltrusaitis et al. (2017) Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency.  Multimodal machine learning: A survey and taxonomy, 2017.
  * Chen et al. (2020) Dicheng Chen, Zi Wang, Di Guo, Vladislav Orekhov, and Xiaobo Qu.  Review and prospect: deep learning in nuclear magnetic resonance spectroscopy.  _Chemistry–A European Journal_ , 26(46):10391–10401, 2020.
  * Chhetri et al. (2018) Bhuwan Khatri Chhetri, Serge Lavoie, Anne Marie Sweeney-Jones, and Julia Kubanek.  Recent trends in the structural revision of natural products.  _Natural product reports_ , 35(6):514–531, 2018.
  * Claridge (2016) Timothy DW Claridge.  _High-resolution NMR techniques in organic chemistry_ , volume 27.  Elsevier, 2016.
  * Feng et al. (2022) Yong Feng, Jinglong Chen, Jingsong Xie, Tianci Zhang, Haixin Lv, and Tongyang Pan.  Meta-learning as a promising approach for few-shot cross-domain fault diagnosis: Algorithms, applications, and prospects.  _Knowledge-Based Systems_ , 235:107646, 2022.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International conference on machine learning_ , pp. 1126–1135. PMLR, 2017.
  * Frosst et al. (2019) Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton.  Analyzing and improving representations with the soft nearest neighbor loss, 2019.
  * Guan et al. (2021) Yanfei Guan, SV Shree Sowndarya, Liliana C Gallegos, Peter C St John, and Robert S Paton.  Real-time prediction of 1 h and 13 c chemical shifts with dft accuracy using a 3d graph neural network.  _Chemical Science_ , 12(36):12012–12026, 2021.
  * Gunther & Gunther (1994) Harald Gunther and Harald Gunther.  _NMR spectroscopy: basic principles, concepts, and applications in chemistry_.  John Wiley & Sons Chichester, UK, 1994.
  * Huisman et al. (2021) Mike Huisman, Jan N Van Rijn, and Aske Plaat.  A survey of deep meta-learning.  _Artificial Intelligence Review_ , 54(6):4483–4541, 2021.
  * Jabeen et al. (2023) Summaira Jabeen, Xi Li, Muhammad Shoib Amin, Omar Bourahla, Songyuan Li, and Abdul Jabbar.  A review on methods and applications in multimodal deep learning.  _ACM Transactions on Multimedia Computing, Communications and Applications_ , 19(2s):1–41, 2023.
  * Jaiswal et al. (2021) Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon.  A survey on contrastive self-supervised learning.  _Technologies_ , 9(1), 2021.  ISSN 2227-7080.  doi: 10.3390/technologies9010002.  URL https://www.mdpi.com/2227-7080/9/1/2.
  * Jonas et al. (2022) Eric Jonas, Stefan Kuhn, and Nils Schlörer.  Prediction of chemical shift in nmr: A review.  _Magnetic Resonance in Chemistry_ , 60(11):1021–1031, 2022.
  * Karpathy & Fei-Fei (2015) Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3128–3137, 2015.
  * Kim et al. (2023) Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, et al.  Pubchem 2023 update.  _Nucleic acids research_ , 51(D1):D1373–D1380, 2023.
  * Kuhn (2022) Stefan Kuhn.  Applications of machine learning and artificial intelligence in nmr, 2022.
  * Lake et al. (2017) Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman.  Building machines that learn and think like people.  _Behavioral and brain sciences_ , 40:e253, 2017.
  * Le-Khac et al. (2020a) Phuc H. Le-Khac, Graham Healy, and Alan F. Smeaton.  Contrastive representation learning: A framework and review.  _IEEE Access_ , 8:193907–193934, 2020a.  doi: 10.1109/ACCESS.2020.3031549.
  * Le-Khac et al. (2020b) Phuc H Le-Khac, Graham Healy, and Alan F Smeaton.  Contrastive representation learning: A framework and review.  _Ieee Access_ , 8:193907–193934, 2020b.
  * Li et al. (2021) Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  _arXiv preprint arXiv:2110.05208_ , 2021.
  * Liang et al. (2021) Paul Pu Liang, Peter Wu, Liu Ziyin, Louis-Philippe Morency, and Ruslan Salakhutdinov.  Cross-modal generalization: Learning in low resource modalities via meta-alignment.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450386517.  doi: 10.1145/3474085.3475247.  URL https://doi.org/10.1145/3474085.3475247.
  * Liang et al. (2023) Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency.  Foundations and trends in multimodal machine learning: Principles, challenges, and open questions, 2023.
  * Liu et al. (2021) Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang.  Self-supervised learning: Generative or contrastive.  _IEEE transactions on knowledge and data engineering_ , 35(1):857–876, 2021.
  * Liu et al. (2023) Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang.  Self-supervised learning: Generative or contrastive.  _IEEE Transactions on Knowledge and Data Engineering_ , 35(1):857–876, 2023.  doi: 10.1109/TKDE.2021.3090866.
  * Ma et al. (2022) Yao Ma, Shilin Zhao, Weixiao Wang, Yaoman Li, and Irwin King.  Multimodality in meta-learning: A comprehensive survey.  _Knowledge-Based Systems_ , 250:108976, 2022.
  * Morgado et al. (2021) Pedro Morgado, Nuno Vasconcelos, and Ishan Misra.  Audio-visual instance discrimination with cross-modal agreement.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12475–12486, 2021.
  * Nicolaou & Snyder (2005) KC Nicolaou and Scott A Snyder.  Chasing molecules that were never there: misassigned natural products and the role of chemical synthesis in modern structure elucidation.  _Angewandte Chemie International Edition_ , 44(7):1012–1044, 2005.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Salakhutdinov & Hinton (2007) Ruslan Salakhutdinov and Geoff Hinton.  Learning a nonlinear embedding by preserving class neighbourhood structure.  In Marina Meila and Xiaotong Shen (eds.), _Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics_ , volume 2 of _Proceedings of Machine Learning Research_ , pp. 412–419, San Juan, Puerto Rico, 21–24 Mar 2007. PMLR.  URL https://proceedings.mlr.press/v2/salakhutdinov07a.html.
  * Serra et al. (2018) Joan Serra, Santiago Pascual, and Alexandros Karatzoglou.  Towards a universal neural network encoder for time series.  In _CCIA_ , pp. 120–129, 2018.
  * Steinbeck et al. (2003) Christoph Steinbeck, Stefan Krause, and Stefan Kuhn.  Nmrshiftdb constructing a free chemical information system with open-source components.  _Journal of chemical information and computer sciences_ , 43(6):1733–1739, 2003.
  * Summaira et al. (2021) Jabeen Summaira, Xi Li, Amin Muhammad Shoib, Songyuan Li, and Jabbar Abdul.  Recent advances and trends in multimodal deep learning: A review, 2021.
  * van den Oord et al. (2019) Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding, 2019.
  * Vanschoren (2018) Joaquin Vanschoren.  Meta-learning: A survey.  _arXiv preprint arXiv:1810.03548_ , 2018.
  * Vettoruzzo et al. (2023) Anna Vettoruzzo, Mohamed-Rafik Bouguelia, Joaquin Vanschoren, Thorsteinn Rögnvaldsson, and KC Santosh.  Advances and challenges in meta-learning: A technical review.  _arXiv preprint arXiv:2307.04722_ , 2023.
  * Vilalta & Drissi (2002) Ricardo Vilalta and Youssef Drissi.  A perspective view and survey of meta-learning.  _Artificial intelligence review_ , 18:77–95, 2002.
  * Wu et al. (2023) Anan Wu, Qing Ye, Xiaowei Zhuang, Qiwen Chen, Jinkun Zhang, Jianming Wu, and Xin Xu.  Elucidating structures of complex organic compounds using a machine learning model based on the 13c nmr chemical shifts.  _Precision Chemistry_ , 1(1):57–68, 2023.
  * Wu et al. (2022) Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui.  Graph neural networks in recommender systems: a survey.  _ACM Computing Surveys_ , 55(5):1–37, 2022.
  * Wu et al. (2018) Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3733–3742, 2018.
  * Xu et al. (2018) Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.  How powerful are graph neural networks?  _arXiv preprint arXiv:1810.00826_ , 2018.
  * Yang et al. (2021) Zhuo Yang, Jianfei Song, Minjian Yang, Lin Yao, Jiahua Zhang, Hui Shi, Xiangyang Ji, Yafeng Deng, and Xiaojian Wang.  Cross-modal retrieval between 13c nmr spectra and structures for compound identification using deep contrastive learning.  _Analytical Chemistry_ , 93(50):16947–16955, 2021.
  * Yu et al. (2021) Hyo-Yeon Yu, Sangki Myoung, and Sangdoo Ahn.  Recent applications of benchtop nuclear magnetic resonance spectroscopy.  _Magnetochemistry_ , 7(9):121, 2021.
  * Zolfaghari et al. (2021) Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, and Thomas Brox.  Crossclr: Cross-modal contrastive learning for multi-modal video representations.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 1450–1459, 2021.

Appendix

##  Appendix A Revisiting Knowledge Span Guided Loss

######  Theorem 1 (Knowledge Span Guided Loss).

Suppose $\mathcal{M}$ is the set of instances.
$\mathcal{A}\subset\mathbb{R}^{d_{1}}$ is the set of tunable instances’
embeddings in modality A, $\mathcal{B}\subset\mathbb{R}^{d_{1}}$ is the set of
tuable instances’ embeddings in modality B, and
$\mathcal{K}\subset\mathbb{R}^{d_{2}}$ is the corresponding fixed knowledge
span label that can guide the relative distance learning between components in
$\mathcal{A}$ and $\mathcal{B}$. Thus, the size of $\mathcal{A}$,
$\mathcal{B}$, $\mathcal{K}$ are $|\mathcal{M}|$, respectively.

Let $\mathcal{A}_{i}$ be the $i^{th}$ instance embedding of $\mathcal{A}$, and
$\mathcal{B}_{j}$ be the $j^{th}$ instance embedding of $\mathcal{B}$. We
define the distance function between $\mathcal{A}_{i}$ and $\mathcal{B}_{j}$
as
$d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})=\mathcal{A}_{i}\cdot\mathcal{B}_{j}\rightarrow\mathbb{R}^{+}$,
and calibration function
$d(\mathcal{K}_{i},\mathcal{K}_{j})\rightarrow\mathbb{R}^{+}$ with a monotonic
property and constraint
$\sum_{j=1}^{|\mathcal{M}|}d(\mathcal{K}_{i},\mathcal{K}_{j})=1$, in which
$\mathcal{K}_{i}$ and $\mathcal{K}_{j}$ serve as the designated Knowledge Span
Label. We introduce the Knowledge Span Guided Loss (KSGL) as follows:

$\displaystyle KSGL(i)$
$\displaystyle=-\displaystyle\sum_{\begin{subarray}{c}1\leq
j\leq|\mathcal{M}|\end{subarray}}d(\mathcal{K}_{i},\mathcal{K}_{j})\log\frac{e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})}}{\displaystyle\sum_{\begin{subarray}{c}1\leq
k\leq|\mathcal{M}|\end{subarray}}e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{k})}}$
(A.1) $\displaystyle=-\displaystyle\sum_{\begin{subarray}{c}1\leq
j\leq|\mathcal{M}|\end{subarray}}d(\mathcal{K}_{i},\mathcal{K}_{j})\log(\text{softmax}(d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})))$
(A.2)

###### Proof.

In order to optimize the loss $KSGL(i)$, we need to set the following partial
derivative to be 0 for each $d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})$ with
$1\leq j\leq|\mathcal{M}|$. Here are the detail process:

$\displaystyle\frac{\partial
KSGL(i)}{\partial{d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})}}$
$\displaystyle=\frac{\partial}{\partial
d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})}\underbrace{\left(-d(\mathcal{K}_{i},\mathcal{K}_{j})\log\frac{e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})}}{e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})}+\sum_{\begin{subarray}{c}k\neq
j\end{subarray}}e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{k})}}\right)}_{\text{When
the numerator includes }e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})}}$
$\displaystyle\quad+\frac{\partial}{\partial
d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})}\underbrace{\left(\sum_{\begin{subarray}{c}k\neq
j\end{subarray}}-d(\mathcal{K}_{i},\mathcal{K}_{k})\log\frac{e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{k})}}{e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})}+\sum_{\begin{subarray}{c}k\neq
j\end{subarray}}e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{k})}}\right)}_{\text{When
the numerator does not include }e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})}}$
$\displaystyle=-(d(\mathcal{K}_{i},\mathcal{K}_{j})-d(\mathcal{K}_{i},\mathcal{K}_{j})\cdot\text{softmax}(d_{E}(\mathcal{A}_{i},\mathcal{B}_{j}))$
$\displaystyle\quad-\sum_{\begin{subarray}{c}k\neq
j\end{subarray}}d(\mathcal{K}_{i},\mathcal{K}_{k})\cdot\text{softmax}(d_{E}(\mathcal{A}_{i},\mathcal{B}_{j}))$
$\displaystyle=-\left(d(\mathcal{K}_{i},\mathcal{K}_{j})-(d(\mathcal{K}_{i},\mathcal{K}_{j})+\sum_{\begin{subarray}{c}k\neq
j\end{subarray}}d(\mathcal{K}_{i},\mathcal{K}_{k}))\cdot\text{softmax}(d_{E}(\mathcal{A}_{i},\mathcal{B}_{j}))\right)$

Since $\sum_{l=1}^{|\mathcal{M}|}d(\mathcal{K}_{i},\mathcal{K}_{l})$ = 1, we
can further simplify it as

$\displaystyle\frac{\partial
KSGL(i)}{\partial{d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})}}=-(d(\mathcal{K}_{i},\mathcal{K}_{j})-\text{softmax}(d_{E}(\mathcal{A}_{i},\mathcal{B}_{j}))$

In order to optimize, we need to set the respective partial derivative to be
0:

$\displaystyle\frac{\partial
KSGL(i)}{\partial{d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})}}=-(d(\mathcal{K}_{i},\mathcal{K}_{j})-\text{softmax}(d_{E}(\mathcal{A}_{i},\mathcal{B}_{j}))=0$

In addition, the corresponding second partial derivative denoted as
$\frac{\partial
KSGL(i)}{\partial{d_{E}^{2}(\mathcal{A}_{i},\mathcal{B}_{j})}}$ manifests as
follows:

$\displaystyle\frac{\partial
KSGL(i)}{\partial{d_{E}^{2}(\mathcal{A}_{i},\mathcal{B}_{j})}}=\text{softmax}(d_{E}(\mathcal{A}_{i},\mathcal{B}_{j}))(1-\text{softmax}(d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})))$

As $\text{softmax}(d_{E}(\mathcal{A}_{i},\mathcal{B}_{j}))$ takes values
within the open interval (0,1), it follows that $\frac{\partial
KSGL(i)}{\partial{d_{E}^{2}(\mathcal{A}_{i},\mathcal{B}_{j})}}$ is always
positive. Consequently, the pinnacle of optimization emerges as a global
minimum.
Furthermore, when it comes to optimum:

$\displaystyle d(\mathcal{K}_{i},\mathcal{K}_{j})$
$\displaystyle=\text{softmax}(d_{E}(\mathcal{A}_{i},\mathcal{B}_{j}))$
$\displaystyle d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})$
$\displaystyle=\log(d(\mathcal{K}_{i},\mathcal{K}_{j}))+\log\left(\sum_{\begin{subarray}{c}1\leq
l\leq|\mathcal{M}|\end{subarray}}e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{l})}\right)$

It is easy to show that when it reaches optimum, $d_{E}(A_{i},B_{j})$ is
consistent with Knowledge Span Guidance $d(\mathcal{K}_{i},\mathcal{K}_{j})$.
Without loss of generosity, suppose
$d(\mathcal{K}_{i},\mathcal{K}_{j})>d(\mathcal{K}_{i},\mathcal{K}_{j^{\prime}})$
:

$\displaystyle
d_{E}(\mathcal{A}_{i},\mathcal{B}_{j})-d_{E}(\mathcal{A}_{i},\mathcal{B}_{j^{\prime}})$
$\displaystyle=\log(d(\mathcal{K}_{i},\mathcal{K}_{j}))+\log\left(\sum_{\begin{subarray}{c}1\leq
l\leq|\mathcal{M}|\end{subarray}}e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{l})}\right)$
$\displaystyle\quad-\left(\log(d(\mathcal{K}_{i},\mathcal{K}_{j^{\prime}}))+\log\left(\sum_{\begin{subarray}{c}1\leq
l\leq|\mathcal{M}|\end{subarray}}e^{d_{E}(\mathcal{A}_{i},\mathcal{B}_{l})}\right)\right)$
$\displaystyle=\log(d(\mathcal{K}_{i},\mathcal{K}_{j}))-\log(d(\mathcal{K}_{i},\mathcal{K}_{j^{\prime}}))$
$\displaystyle=\log\left(\frac{d(\mathcal{K}_{i},\mathcal{K}_{j})}{d(\mathcal{K}_{i},\mathcal{K}_{j^{\prime}})}\right)>0$

∎

##  Appendix B Further ablation study about parameter choices

###  B.1 Ablation study about the choice of GIN structure and projection.

We choose GIN(Xu et al. (2018)) as our graph encoder. By Table B.1, ”GIN
Depth” signifies the number of layers in the GIN, ”GIN Embedding Dim” denotes
the dimensionality of the embeddings generated by the GIN model, and
”Projection Dim” indicates the resulting dimensionality after transforming the
GIN-produced embeddings. In particular, the best performance is observed when
the GIN model has 5 layers, GIN Embedding Dim is 128, and projection Dim is
512.

Table B.1: GIN structure and projection ablation study GIN Depth | GIN Embedding Dim | Projection Dim | Validation accuracy (%)
---|---|---|---
3 | 128 | 128 | 86.6
3 | 256 | 128 | 86.8
3 | 512 | 128 | 86.3
5 | 128 | 128 | 89.4
5 | 256 | 128 | 89.6
5 | 512 | 128 | 89.3
3 | 128 | 256 | 86.6
3 | 256 | 256 | 86.8
3 | 512 | 256 | 86.3
5 | 128 | 256 | 89.4
5 | 256 | 256 | 89.6
5 | 512 | 256 | 89.3
3 | 128 | 512 | 86.6
3 | 256 | 512 | 86.5
3 | 512 | 512 | 86.2
5 | 32 | 512 | 84.0
5 | 64 | 512 | 87.5
5 | 128 | 512 | 90.0
5 | 256 | 512 | 89.4
5 | 512 | 512 | 88.9

###  B.2 Ablation Study on the Choice of $\tau_{1}$ and $\tau_{2}$

We also conducted a further ablation study exploring different combinations of
$\tau_{1}$ and $\tau_{2}$ as shown in Table B.2. For this analysis, we fixed
the GIN depth at 5, set the GIN embedding dimensionality to 128, and
maintained a projection dimension of 512. We observe that the best performance
is achieved when $\tau_{1}$ = $10^{-5}$ and $\tau_{2}$ = $10^{1}$.

Table B.2: Ablation study about $tau_{1}$ and $tau_{2}$. We have 5 layers and 128 dimension as the final representation. $\tau_{1}$ | $\tau_{2}$ | Molecular Alignment Accuracy (%) | Atom Alignment Accuracy (%)
---|---|---|---
$10^{-1}$ | $10^{1}$ | 94.9 | 89.6
$10^{-1}$ | $10^{2}$ | 95.2 | 89.8
$10^{-1}$ | $10^{3}$ | 95.6 | 89.6
$10^{-1}$ | $10^{4}$ | 95.1 | 88.9
$10^{-1}$ | $10^{5}$ | 95.0 | 89.3
$10^{-2}$ | $10^{1}$ | 95.5 | 89.8
$10^{-2}$ | $10^{2}$ | 94.8 | 89.8
$10^{-2}$ | $10^{3}$ | 95.4 | 88.8
$10^{-2}$ | $10^{4}$ | 94.8 | 87.2
$10^{-2}$ | $10^{5}$ | 95.1 | 89.4
$10^{-3}$ | $10^{1}$ | 95.0 | 89.2
$10^{-3}$ | $10^{2}$ | 95.1 | 89.1
$10^{-3}$ | $10^{3}$ | 95.2 | 89.0
$10^{-3}$ | $10^{4}$ | 95.3 | 89.7
$10^{-3}$ | $10^{5}$ | 95.0 | 89.4
$10^{-4}$ | $10^{1}$ | 95.0 | 89.8
$10^{-4}$ | $10^{2}$ | 95.1 | 89.7
$10^{-4}$ | $10^{3}$ | 95.0 | 89.8
$10^{-4}$ | $10^{4}$ | 95.3 | 89.5
$10^{-4}$ | $10^{5}$ | 95.1 | 88.4
$10^{-5}$ | $10^{1}$ | 95.4 | 90.0
$10^{-5}$ | $10^{2}$ | 95.0 | 89.5
$10^{-5}$ | $10^{3}$ | 95.8 | 89.6
$10^{-5}$ | $10^{4}$ | 95.2 | 89.7
$10^{-5}$ | $10^{5}$ | 95.0 | 89.7

##  Appendix C Additional Results on Molecular Retrieval

Table C.1: Zero-shot molecular retrieval top 5, 10, 25 accuracy (%) with K-M3AID and baselines Method | Accuracy | $10^{2}$ | $10^{3}$ | $10^{4}$ | $10^{5}$ | $10^{6}$
---|---|---|---|---|---|---
Knowledge Guide | Top 1(%) |  95.8$\pm$1.0 |  80.4$\pm$3.9 |  46.3$\pm$1.2 |  18.0$\pm$0.8 |  5.8$\pm$1.7
Top 5(%) |  99.8$\pm$0.2 |  96.8$\pm$0.5 |  77.8$\pm$1.1 |  41.6$\pm$1.6 |  16.6$\pm$2.3
Top 10(%) |  100.0$\pm$0.0 |  98.8$\pm$0.3 |  87.7$\pm$1.0 |  53.9$\pm$1.8 |  25.2$\pm$3.2
Top 25(%) |  100.0$\pm$0.0 |  99.6$\pm$0.2 |  94.8$\pm$0.6 |  71.6$\pm$0.6 |  37.8$\pm$4.1
SP-ID | Top 1(%) |  95.3$\pm$0.8 |  78.6$\pm$2.7 |  35.8$\pm$3.8 |  12.9$\pm$1.6 |  3.4$\pm$0.9
Top 5(%) |  95.4$\pm$0.1 |  77.3$\pm$0.7 |  44.7$\pm$2.3 |  16.2$\pm$2.4 |  4.4$\pm$1.5
Top 10(%) |  100.0$\pm$0.0 |  97.3$\pm$0.7 |  77.5$\pm$2.3 |  40.2$\pm$2.4 |  12.3$\pm$1.5
Top 25(%) |  100.0$\pm$0.0 |  99.1$\pm$0.2 |  85.9$\pm$1.0 |  53.1$\pm$3.0 |  18.5$\pm$1.8
WP-ID(th=1) | Top 1(%) |  92.9$\pm$0.6 |  71.7$\pm$1.0 |  32.7$\pm$1.3 |  10.7$\pm$0.5 |  3.6$\pm$0.7
Top 5(%) |  99.6$\pm$0.1 |  93.8$\pm$0.8 |  63.9$\pm$1.5 |  29.3$\pm$1.5 |  10.2$\pm$1.2
Top 10(%) |  99.9$\pm$0.0 |  97.1$\pm$0.4 |  76.8$\pm$0.7 |  39.3$\pm$0.9 |  15.7$\pm$1.5
Top 25(%) |  100.0$\pm$0.0 |  99.1$\pm$0.2 |  88.2$\pm$0.6 |  55.7$\pm$1.1 |  26.5$\pm$2.0
No communication | Top 1(%) |  94.8$\pm$1.2 |  77.6$\pm$1.4 |  40.1$\pm$1.2 |  14.4$\pm$0.9 |  4.1$\pm$1.1
Top 5(%) |  99.8$\pm$0.1 |  96.2$\pm$0.5 |  73.6$\pm$2.2 |  35.8$\pm$1.3 |  11.4$\pm$1.0
Top 10(%) |  99.9$\pm$0.1 |  98.6$\pm$0.3 |  84.1$\pm$1.4 |  47.3$\pm$1.8 |  17.3$\pm$1.8
Top 25(%) |  100.0$\pm$0.0 |  99.7$\pm$0.2 |  92.9$\pm$0.9 |  65.1$\pm$2.2 |  27.9$\pm$2.5

##  Appendix D Additional Discussion about Isomers

###  D.1 Isomer Category

Isomers typically fall into two main categories: constitutional (structural)
isomers, which share the same chemical formula but display distinct atom
connectivity, and stereoisomers (spatial isomers), which share the same
topology graph but diverge in their three-dimensional arrangement (see Figure
D.1). Constitutional isomers are NMR-variant, meaning that different isomers
produce distinct NMR spectrum. In the sub-categories of stereoisomers,
enantiomers are NMR-invariant, but diastereomers and cis-trans isomers are
NMR-variant.

Figure D.1: NMR Variability in Isomers

###  D.2 Isomers Group for $\text{C}_{7}\text{H}_{11}\text{NO}_{3}$

Here is an example for isomer groups. In this isomer group of
$\text{C}_{7}\text{H}_{11}\text{NO}_{3}$, they all share the same chemical
formula in Figure D.2. The first 10 are constitutional (structural) isomers of
each other (cycled green), the last 4 are two pairs of diastereomers (cycled
brown). Each of these isomers corresponds to a distinct NMR spectrum.

Figure D.2: Isomer demo for $\text{C}_{7}\text{H}_{11}\text{NO}_{3}$

##  Appendix E Additional Results on Peak Assignment

In complex natural product molecules, it is a common situation that the local
contents of some atoms within the same molecule exhibit a high degree of
similarity. It gives rise to challenges for the atomic alignment, as some
atoms correspond to ppm values in close proximity. However, our K-M3AID model
is capable of recognizing each of the atoms with effective learnt embeddings
and deciphering the correspondences among the atoms and the peaks at zero-
shot. Two complex natural product molecules with multiple rings (4 and 4,
respectively) and multiple chiral centers (6 and 8, respectively) are taken to
showcase the effectiveness of atomic alignment (see Appendix Figure E.1).

In molecular A in Figure E.2, atom 13 and atom 14 are tertiary carbons
(attaching to 3 carbons and 1 hydrogen) and on the same 5-member ring,
corresponding to the ppm of 34.3 and 35.6, respectively. The similar local
content of these two atoms fools SP-ID and WP-ID. In addition, WP-ID fails
with more atomic alignments. The molecular B is chemical symmetric regarding
atom 0. Thus, atom 1 and atom 3 correspond to the same peak on the spectra.
The ppm of atom 1 and atom 3 is 114.2, the ppm of atom 2 and atom 4 is 110.0.
While there is 4.2 difference, SP-ID and WP-ID fails to pick up right
alignment for atom 1 and atom 3. In contrast, K-ID succeed to align the atoms
with peaks in both molecules.

Figure E.1: Examples of Zero-shot Atomic Alignment for Complex Natural
Products. Yellow cells in the PPM difference represent the ground truth
alignment. Figure E.2: Extra case studies of IE-Meta-MMA.Yellow cells in the
PPM differerence represent the ground truth alignment, and red cross
represents the wrong alignment.