[1]Shi Yin [2,1]Lixin He

[1]Institute of Artificial Intelligence, Hefei Comprehensive National Science
Center, Hefei, 230088, People’s Republic of China 2]CAS Key Laboratory of
Quantum Information, University of Science and Technology of China, Hefei,
230026, People’s Republic of China 3]University of Science and Technology of
China, Hefei, 230026, People’s Republic of China

# A Framework of SO(3)-equivariant Non-linear Representation Learning and its
Application to Electronic-Structure Hamiltonian Prediction

<EMAIL_ADDRESS>Xinyang Pan  Fengyan Wang  Feng Wu<EMAIL_ADDRESS>*
[  [

###### Abstract

We present both a theoretical and a methodological framework that addresses a
critical challenge in applying deep learning to physical systems: the
reconciliation of non-linear expressiveness with SO(3)-equivariance in
predictions of SO(3)-equivariant quantities. Inspired by covariant theory in
physics, we address this problem by exploring the mathematical relationships
between SO(3)-invariant and SO(3)-equivariant quantities and their
representations. We first construct theoretical SO(3)-invariant quantities
derived from the SO(3)-equivariant regression targets, and use these invariant
quantities as supervisory labels to guide the learning of high-quality
SO(3)-invariant features. Given that SO(3)-invariance is preserved under non-
linear operations, the encoding process for invariant features can extensively
utilize non-linear mappings, thereby fully capturing the non-linear patterns
inherent in physical systems. Building on this foundation, we propose a
gradient-based mechanism to induce SO(3)-equivariant encodings of various
degrees from the learned SO(3)-invariant features. This mechanism can
incorporate non-linear expressive capabilities into SO(3)-equivariant
representations, while theoretically preserving their equivariant properties
as we prove. We apply our theory and method to the electronic-structure
Hamiltonian prediction tasks, experimental results on eight benchmark
databases covering multiple types of elements and challenging scenarios show
dramatic breakthroughs on the state-of-the-art prediction accuracy, with
improvements of up to 40% in predicting Hamiltonians and up to 76% in
predicting downstream physical quantities such as occupied orbital energy. Our
approach goes beyond handling physical systems and offers a promising general
solution to the critical dilemma between equivariance and non-linear
expressiveness for the deep learning paradigm.

###### keywords:

Deep Learning, Neural Networks, SO(3)-equivariance, Atomic Modeling,
Hamiltonian Prediction

##  1 Introduction

With the advantages in computational complexity and generalization
capabilities, deep learning paradigm has vigorously driven advancements in
physics research [1]. For example, in predicting electronic-structure
Hamiltonians, traditional Density Functional Theory (DFT) methods [2, 3]
suffer from computational complexities of $\mathcal{O}(N^{3})$, where $N$ is
the number of atoms in a system. In contrast, deep learning approaches have
significantly reduced this complexity to $\mathcal{O}(N)$ [4, 5, 6, 7, 8],
accelerating processes by thousands or even millions of times. This reduction
in computational demand not only unlocks possibilities for analyzing extremely
large atomic systems but also paves the way for efficient material simulation,
discovery, and design that were previously unimaginable.

However, deep learning methods still face substantial challenges when
processing physical systems. To align with fundamental physical laws, these
methods must strictly adhere to symmetry principles. For example, physical
quantities such as force fields and electronic-structure Hamiltonians must be
equivariant under 3D rotational transformations, i.e., elements from the SO(3)
group. Traditional deep learning methods [9, 10] typically approximated
SO(3)-equivariance through rotational data augmentation. However, this
approach does not provide the strict assurance of SO(3)-equivariance required
for physical research. Besides, the calculation of physical quantities calls
for high numerical accuracy, necessitating that the neural networks possess
strong capabilities to express the complex non-linear mappings from atomic
structures to the regression targets. However, existing deep learning methods
are unable to simultaneously ensure strict SO(3)-equivariance and high
numerical accuracy. The root cause of this problem lies in the conflict
between SO(3)-equivariance and non-linear expressiveness: specifically,
directly applying non-linear activation functions on SO(3)-equivariant
features (with degree $l\geq 1$) may lead to the loss of equivariance, while
bypassing non-linear mappings severely restricts the network’s expressive
capabilities and thereby lowering down the achievable accuracy. As pointed out
in previous literature [11], this issue is commonly found in machine learning
tasks that demand both strict equivariance and fine-grained generalization
performance. Although several efforts have been made to alleviate this issue
by introducing gated activation mechanisms [8, 7, 12, 13, 14], developing
local coordinate strategy [6], and employing discrete spherical channels [11,
15, 16], they still, to varying degrees, compromised either non-linear
expressiveness [7, 8, 12, 13, 14] or strict SO(3)-equivariance [11, 15, 16]
when processing physical systems. Please refer to Supplementary Information A
for an in-depth analysis.

To address the equivariance-expressiveness dilemma, we make theoretical and
methodological explorations on unifying strict SO(3)-equivariance with strong
non-linear expressiveness within the realm of deep representation learning. We
are inspired by the insight that invariant quantities in transformation often
reflect the mathematical nature of physical laws and can induce other
quantities with equivariant properties. For example, in special relativity
[18], the spacetime interval between events is an invariant under Lorentz
transformations. This invariance is foundational in formulating physical laws
that involve covariant quantities. Furthermore, from the perspective of deep
representation learning, the attribute that invariance is preserved under non-
linear operations is a significant advantage, given its compatibility with
non-linear expressive capabilities. Built upon these insights, we
theoretically solve the equivariance-expressiveness dilemma by intensively
exploring and making use of the intrinsic relationships between
SO(3)-invariant and SO(3)-equivariant quantities and representations: we first
dedicate efforts to learning high-quality SO(3)-invariant features with ample
non-linear expressiveness, and subsequently, we derive SO(3)-equivariant non-
linear representations from these SO(3)-invariant ones. Specifically:

First, we construct quantities that are theoretically SO(3)-invariant, namely
$tr(\mathbf{Q}\cdot\mathbf{Q}^{\dagger})$, where $\mathbf{Q}$ are the
SO(3)-equivariant regression targets, $tr(\cdot)$ signifies the trace
operation, and $\dagger$ denotes the conjugate transpose operation. A
significant advantage of these SO(3)-invariant quantities lies in the fact
that they are directly derived from the SO(3)-equivariant target labels and
can serve as unique supervision labels for the effective learning of
informative SO(3)-invariant features that capture the intrinsic symmetry
properties of the mathematical structure of $\mathbf{Q}$ without requiring
additional labeling resources.

Second, we propose a gradient-based mechanism to induce SO(3)-equivariant
representations for predicting the regression target $\mathbf{Q}$, from high-
quality non-linear SO(3)-invariant features learned under the supervision of
$tr(\mathbf{Q}\cdot\mathbf{Q}^{\dagger})$. This mechanism can incorporate non-
linear expressive capabilities into SO(3)-equivariant representations while
preserving their equivariant properties, as we prove.

We develop our theory into a unified SO(3)-equivariant non-linear
representation learning method, and apply it to the computation of electronic-
structure Hamiltonian, which occupies a central position in quantum mechanics
to induce key physical quantities such as electronic wavefuction, orbital
energy, and band structure, bringing deeply understanding of the electrical,
optical, magnetic, and transport properties of atomic systems, while posing
significant difficulties for machine learning techniques due to its intrinsic
symmetry with respect to the SO(3) group as well as the high-dimensional
complexity, as shown in Fig. 1. Our method dramatically enhances the state-of-
the-art performance on eight benchmark Hamiltonian databases from the renowned
DeepH and QH9 benchmark series [6, 8, 19], showing excellent generality to
various challenging scenarios such as thermal motions, bilayer twists, scale
variations, and new trajectories, with an accuracy improvement of Hamiltonians
of up to 40%. Moreover, our approach substantially enhances the prediction
accuracy of downstream physical quantities of Hamiltonians including occupied
orbital energies and electronic wavefunctions, with an accuracy improvement by
a maximum of 76% and 17%, respectively. Our leading performance
comprehensively demonstrate that our method, while satisfying
SO(3)-equivariance, possesses excellent expressive power and generalization
performance, providing an effective deep learning tool for efficient and
accurate electronic-structure calculations of atomic systems.

Figure 1: Illustration of the high-dimensional complexity of the Hamiltonian.
In an atomic system, a Hamiltonian matrix is defined between each pair of
atoms within each other’s cutoff radius, constituting a large cluster of
Hamiltonian matrices for all atom pairs. Besides, each Hamiltonian matrix
contains multiple basic blocks, with the $\mathbf{Q}=\mathbf{Q}^{l_{p}\otimes
l_{q}}$ defined in Section 2 corresponding to a basic block
$\mathbf{H}^{l_{p}\otimes l_{q}}$ here. Thus, for a Hamiltonian prediction
model, the space of the regression target is typically huge.

##  2 Problem Formalization

Physical quantities that are SO(3)-equivariant typically possess their
respective equivariant formulas under both matrix-formed direct-product states
and vector-formed direct-sum states, which are mathematically equivalent to
each other.

Without loss of generality, let ${\bf{Q}}^{l_{p}\otimes l_{q}}$ denote an
SO(3)-equivariant quantity in direct-product state formed by $l_{p}\otimes
l_{q}$, i.e., the direct product between degrees $l_{p}$ and $l_{q}$. It obeys
the following SO(3)-equivariant law:

${\bf{Q}}({\bf{R}})^{l_{p}\otimes
l_{q}}={\bf{D}}^{l_{p}}({\bf{R}})\cdot{\bf{Q}}^{l_{p}\otimes
l_{q}}\cdot{({\bf{D}}^{l_{q}}({\bf{R}}))^{\dagger}},\;1\leq p\leq P,\,1\leq
q\leq Q$ (1)

where $\dagger$ denotes the conjugate transpose operation.
${\bf{R}}\in\mathbb{R}^{3\times 3}$ is the rotational matrix and
${\bf{D}}^{l_{p}}({\bf{R}})\in\mathbb{R}^{(2l_{p}+1)\times(2l_{p}+1)}$,
${\bf{D}}^{l_{q}}({\bf{R}})\in\mathbb{R}^{(2l_{q}+1)\times(2l_{q}+1)}$ are the
Wigner-D matrices of degrees $l_{p}$ and $l_{q}$, respectively,
${\bf{Q}}({\bf{R}})^{l_{p}\otimes
l_{q}}\in\mathbb{R}^{(2l_{p}+1)\times(2l_{q}+1)}$ denotes the transformed
results of ${\bf{Q}}^{l_{p}\otimes
l_{q}}\in\mathbb{R}^{(2l_{p}+1)\times(2l_{q}+1)}$ through the rotational
operation by ${\bf{R}}$.

${\bf{Q}}^{l_{p}\otimes l_{q}}$ in the direct-product state can be further
decomposed into a series of direct-sum state components with vector forms,
i.e., ${\bf{q}}^{l}(|l_{p}-l_{q}|\leq l\leq l_{p}+l_{q})$, which follows
SO(3)-equivariant law mathematically equivalent to Eq. (1) but with a simpler
form:

${\bf{q}}({\bf{R}})^{l}={\bf{D}}^{l}({\bf{R}})\cdot{\bf{q}}^{l},|l_{p}-l_{q}|\leq
l\leq l_{p}+l_{q}$ (2)

where ${\bf{q}}^{l}\in\mathbb{R}^{2l+1}$ and
${\bf{q}}({\bf{R}})^{l}\in\mathbb{R}^{2l+1}$ respective denote the components
with degree $l$ before and after the rotational operation by ${\bf{R}}$.

Eq. (1) and Eq. (2) are mathematical equivalent to each other, meaning that
fulfilling one under the corresponding state naturally satisfies the other.
Due to the simplicity and ease of handling of the direct-sum state form,
existing deep learning methods [6, 8, 7] typically first predict
$\mathbf{q}^{l}(|l_{p}-l_{q}|\leq l\leq l_{p}+l_{q})$, then convert it to
$\mathbf{Q}^{l_{p}\otimes l_{q}}$ through Clebsch-Gordan matrices. In the
direct-sum state, to obey the equivariant law of Eq. (2) for the regression
target, the hidden representations of neural networks must also satisfy the
same form of equivariance:

${\bf{f}}({\bf{R}})^{(k)l}={\bf{D}}^{l}({\bf{R}})\cdot{\bf{f}}^{(k)l}$ (3)

where ${\bf{f}}^{(k)l}\in\mathbb{R}^{2l+1}$ and
${\bf{f}}({\bf{R}})^{(k)l}\in\mathbb{R}^{2l+1}$ respectively denote one
channel of features with degree $l$ before and after the rotational operation
by ${\bf{R}}$, at the $k$ th hidden layer.

Due to the intrinsic complexity and non-linearity of physic quantities, neural
networks on regressing these quantities are supposed to equip with non-linear
mappings denoted as $g_{nonlin}(\cdot)$ to fully capture the intrinsic
patterns of the physical quantities, which is crucial for precise and
generalizable prediction performance. Meanwhile, $g_{nonlin}(\cdot)$ must also
preserve SO(3)-equivariance, which is expressed as:

${\bf{f}}({\bf{R}})^{(k+1)l}={\bf{D}}^{l}({\bf{R}})\cdot{\bf{f}}^{(k+1)l},{\bf{f}}^{(k+1)l}=g_{nonlin}({\bf{f}}^{(k)l})$
(4)

However, directly implementing $g_{nonlin}(\cdot)$ as neural network module
with non-linear activation functions, such as $Sigmoid$, $Softmax$ and $Silu$,
may result in the destruction of strict equivariance. How to make
$g_{nonlin}(\cdot)$ both theoretically SO(3)-equivariant and capable of non-
linear expressiveness, is the core problem this paper aims at solving.

##  3 Theory

This section establishes a rigorous theoretical foundation for integrating
SO(3)-equivariance with non-linear expressiveness in the deep learning
paradigm. First, we propose a theoretical construct to derive SO(3)-invariant
quantities from the SO(3)-equivariant targets, aiming at facilitating the
learning of high-quality SO(3)-invariant features. Second, we propose a
gradient-based mechanism encoding features maintaining theoretically
SO(3)-equivariance while possessing the non-linear expressive capability
induced from SO(3)-invariant non-linear features to capture the complex non-
linear patterns inherent in physical quantities.

######  Theorem 1.

The quantity $\mathbf{T}=tr(\mathbf{Q}\cdot\mathbf{Q}^{\dagger})$ is
SO(3)-invariant, where $\mathbf{Q}$ is the simplified representation (without
superscripts) of $\mathbf{Q}^{l_{p}\otimes l_{q}}$ defined in Section 2, and
$\dagger$ denotes the conjugate transpose operation, $tr(\cdot)$ is the trace
operation.

######  Theorem 2.

The non-linear neural mapping defined as the following is SO(3)-equivariant,
satisfying the requirements specified in Eq. (4):

$\mathbf{v}=g_{nonlin}(\mathbf{f})=\frac{\partial
z}{\partial\mathbf{f}}\quad\text{s.t.}\quad
z=s_{nonlin}(u),u=\text{CGDecomp}(\mathbf{f}\otimes\mathbf{f},0)$ (5)

where $\mathbf{f}$ is an input SO(3)-equivariant feature with degree $l$ in
the direct-sum state, $\otimes$ denotes the direct-product operation of
tensors, $\text{CGDecomp}(\cdot,0)$ refers to performing a Clebsch-Gordan
decomposition of the tensor and returning the scalar component of degree 0,
$s_{nonlin}(\cdot)$ represents non-linear neural modules, and $\mathbf{v}$ is
the outputted feature encoded by $g_{nonlin}(\cdot)$.

Proofs of the two theorems are included in Supplementary Information B.

######  Remark 1.

In Theorem 1, the term $tr(\mathbf{Q}\cdot\mathbf{Q}^{\dagger})$ is equal to
the square of the Frobenius norm of the matrix $\mathbf{Q}$, i.e.,
$tr(\mathbf{Q}\cdot\mathbf{Q}^{\dagger})=\|\mathbf{Q}\|_{F}^{2}=\sum_{m,n}|\mathbf{Q}_{mn}|^{2}$.
The invariance of Frobenius norm under unitary transformations reflects the
profound connections between modulus and invariance in physics.

######  Remark 2.

In Theorem 2, $g_{\text{nonlin}}(\cdot)$ can handle equivariant features of
any degree and the returned feature $\mathbf{v}$ has the same degree $l$ as
the input feature $\mathbf{f}$. $\mathbf{v}$ is endowed with non-linear
expressiveness with the effect of $s_{nonlin}(\cdot)$, which can be any
differentiable non-linear network module, including non-linear activation
function as a part. The gradient operation $\frac{\partial
z}{\partial\mathbf{f}}$ in $g_{\text{nonlin}}(\cdot)$ is used during the
forward inference process to promote feature encoding, expanding the usage and
significance of gradients within the deep learning paradigm.

Taking $z$ in Eq. (5) as the bridge connecting Theorem 1 and Theorem 2, we
present a general method for learning non-linear representations that are
equivariant to SO(3), as shown in Fig. 2. The core of our method can be
abstractly referred to as TraceGrad. At the label level, it incorporates the
SO(3)-invariant trace quantity $tr(\mathbf{Q}\cdot\mathbf{Q}^{\dagger})$
introduced in Theorem 1 as a supervisory signal for learning SO(3)-invariant
non-linear feature $z$. Given the attribute that invariance is preserved under
non-linear operations, $z$ is encoded by $s_{nonlin}(\cdot)$ to obtain non-
linear expressive capabilities. At the representation level, by taking the
gradient of $z$ with respect to $\mathbf{f}$, the non-linear expressiveness of
$z$ is transferred to the equivariant feature $\mathbf{v}$, while maintaining
strict SO(3)-equivariance as we prove. Subsequently, merging $\mathbf{v}$ and
$\mathbf{f}$, and applying $g_{\text{nonlin}}(\cdot)$ in a stacked manner can
yield rich SO(3)-equivariant non-linear representations to predict the
SO(3)-equivariant regression target. For details of the method, please refer
to Section 6.

Figure 2: Methodological framework for learning SO(3)-equivariant
representations with non-linear expressiveness to regress complex
SO(3)-equivariant targets.

##  4 Results and Analyses

We apply our theory and method to the electronic-structure Hamiltonian
prediction task, and collect results on eight benchmark databases, i.e.,
Monolayer Graphene ($MG$), Monolayer MoS2 ($MM$), Bilayer Graphene ($BG$),
Bilayer Bismuthene ($BB$), Bilayer Bi2Te3 ($BT$), Bilayer Bi2Se3 ($BS$),
QH9-Stable (QS), and QH9-Dynamic (QD). The first six databases, consisting of
crystal systems with elements like C, Mo, S, Bi, Te and Se, are from the DeepH
benchmark series [6, 8]; while the last two databases, composed of molecular
systems of C, H, O, N, F, are from the QH9 benchmark series [19] . These
databases present diverse and complex challenges to the generality of a
regression model. Regarding $MG$, $MM$, and QD, as their samples are prepared
from a 300K environment, the thermal motions lead to complex non-rigid
deformations, increasing the difficulty of Hamiltonian prediction. For $BG$,
$BB$, $BT$, and $BS$, their twisted samples with an interplay of
SO(3)-equivariant effects and van der Waals (vdW) force variations bring
significant generalization challenges, which are further exacerbated by the
absence of any twisted samples in the training sets. Besides, $BB$, $BT$, and
$BS$ exhibit strong spin-orbit coupling (SOC) effects, which further increase
the complexity of Hamiltonian modeling. For QS and QD, as we select the most
difficult subset split strategies provided in their paper, namely ’ood’ and
’mol’, respectively, yielding differences in atom numbers and trajectories
between training and testing samples, challenges in scale and trajectory
extrapolation become prominent. Detailed information of these databases,
including their statistics and visualizations, can be found in the
Supplementary Information C.

We use a comprehensive set of metrics to deeply evaluate the performance of
electronic-structure Hamiltonian prediction models. On the databases from the
DeepH benchmark series [6, 8], we adpot the metrics of Mean Absolute Error
(MAE) between predicted and ground truth Hamiltonians introduced by Yin et al.
[17], including $MAE^{H}_{all}$ for measuring average MAE of all samples and
matrix elements, $MAE^{H}_{cha\\_s}$ for measuring the MAE of challenging
samples where the baseline model performs the worst, $MAE^{H}_{block}$ for
measuring the MAE of different basic blocks in the Hamiltonian matrix, and
$MAE^{H}_{cha\\_b}$ for measuring the MAE on the most challenging Hamiltonian
block where the baseline model shows the poorest performance (with the largest
$MAE^{H}_{block}$). These metrics comprehensively reflect the accuracy
performance, covering not only the average accuracy but also the accuracy on
difficult samples and challenging blocks of the Hamiltonian matrices. On the
two databases from the QH9 benchmark series, we adopt the metrics introduced
by their original paper [19], including MAE of Hamiltonian matrices, which are
further subdivided into $MAE^{H}_{all}$ for measuring average MAE,
$MAE^{H}_{diag}$ for measuring MAE of Hamiltonian matrix formed by an atom
with itself, and $MAE^{H}_{non\\_diag}$ for measuring MAE of Hamiltonian
matrices formed by different atoms; as well as the MAE ($MAE^{\epsilon}$) of
occupied orbital energies $\epsilon$ induced by the predicted Hamiltonians and
compared to the ground truth ones, and the cosine similarity ($Sim(\psi)$)
between the electronic wavefunctions $\psi$ induced by the predicted and
ground truth Hamiltonians. $\epsilon$ and $\psi$ are crucial downstream
physical quantities for determining multiple properties of the atomic systems
as well as their dynamics, highly reflecting the application values of the
Hamiltonian regression model.

We compare experimental results from two setups: the first one is the
experimental results of the baseline SO(3)-equivariant regression model [8, 7]
for Hamiltonian prediction, and the second one is the experimental results of
extending the architecture and pipeline of the baseline model through the
proposed TraceGrad method, which incorporates non-linear expressiveness into
the SO(3)-equivariant features of the baseline model with the gradient
operations of SO(3)-invariant non-linear features learned by the trace
targets. We choose DeepH-E3 [8] as the baseline model for databases from the
DeepH benchmark series [6, 8]; and we choose QHNet [7] as the baseline model
for databases from the QH9 benchmark series [19]. The results of DeepH-E3 to
be compared are taken from the reproduction of Yin et al. [17]. The
reproduction, based on DeepH-E3’s newest open-source
project111https://github.com/Xiaoxun-Gong/DeepH-E3, showed slightly better
results than that in the original publication, and was evaluated more
comprehensively across multiple metrics. The results of QHNet to be compared
are taken from their original paper [19], and for the unification of MAE
units, we convert the units of MAE from $10^{-6}$ Hartree ($E_{h}$) in the
original paper to meV222 $1E_{h}=27211.4$ meV. The results of our methods are
the average from 10 independent repeated experiments. To ensure
reproducibility, we used a fixed random seed for all random processes. As a
result, the standard deviation of the Hamiltonian prediction MAE across
repeated experiments does not exceed $0.007$ meV, which is negligible.

We list the results of DeepH-E3 and DeepH-E3+TraceGrad in Tables 1 and 2 for
monolayer structure databases ($MG$, $MM$) and bilayer structure databases
($BG$, $BB$, $BT$, $BS$) from the DeepH benchmark series, reporting the values
of $MAE^{H}_{all}$, $MAE^{H}_{cha\\_s}$, and $MAE^{H}_{cha\\_b}$. Regarding
the metric of $MAE^{H}_{block}$ for every Hamiltonian block, due to its large
data volume, we just present its values for representative datasets here in
Fig. 3, and attach its values for all databases from the DeepH series in
Supplementary Information E. From these results, we could find that the
proposed TraceGrad method dramatically enhances the accuracy performance of
the baseline method DeepH-E3, both on average and for challenging samples or
blocks, both on the non-twisted samples and the twisted samples. Specifically,
on the corresponding datasets, TraceGrad lowers down the $MAE^{H}_{all}$ and
$MAE^{H}_{cha}$ of DeepH-E3 with relative ratios of up to $34\%$ and $35\%$,
respectively, and achieving better results in the vast majority of basic
blocks. Particularly, for the blocks where DeepH-E3 perform the worst,
TraceGrad reduces the $MAE$ ($MAE^{H}_{cha\\_b}$) by a maximum of $48\%$. The
leading performance on the $MG$ and $MM$ databases prepared at 300K
demonstrates the robustness of our method against thermal motion. The high
accuracy on the $BB$, $BT$, and $BS$ databases, which have strong SOC effects,
indicates our method’s strong capability to model such effects. The excellent
performance on the $BG^{t}$, $BB^{t}$, $BT^{t}$, and $BS^{t}$ subsets
showcases the method’s superior generalization to twisted structures, which
are not present in the training data. The outstanding performance on such
samples highlights the good potential for studying twist-related phenomena,
given interlayer twists are hot research topics because they may bring new
electrical and transport properties [20, 21, 22]. Additionally, the $BG^{t}$
and $BB^{t}$ testing subsets contain significantly larger atomic systems
compared to the training set (see Supplementary Information C for statistics
of their sizes), yet our method still excels on these subsets for all metrics,
demonstrating its good scalability on the sizes of atomic systems it handles.

We list the results of QHNet and QHNet+TraceGrad in Tables 3 and 4, for the
$QS$ and $QD$ databases, respectively, reporting the values of
$MAE^{H}_{all}$, $MAE^{H}_{diag}$, $MAE^{H}_{non\\_diag}$, $MAE^{\epsilon}$,
and $Sim(\psi)$. From these results, we can observe that the proposed
TraceGrad method significantly enhances the accuracy performance of the
baseline method QHNet across all of these metrics on both of the two
databases. Specifically, TraceGrad lowers down the $MAE^{H}_{all}$,
$MAE^{H}_{non\\_diag}$, $MAE^{H}_{diag}$, $MAE^{\epsilon}$ and $Sim(\psi)$ of
QHNet with relative ratios of up to 40%, 39%, 40%, 76%, and 17%, on the
corresponding databases. The significant improvements on the $QS$ database,
partitioned using the ’ood’ split strategy [19] without scales overlapping
among the training, validation, and testing sets, once again demonstrate the
method’s strong generalization capabilities across different scales.
Meanwhile, the improvements on the $QD$ database under the ’mol’ strategy [19]
partitioning the training, validation, and testing sets with samples from
totally different thermal motion trajectories, highlight the method’s
robustness in generalizing to new thermal motion sequences. Furthermore, the
significant promotion on the prediction accuracy of $\epsilon$, i.e., occupied
orbital energies which are crucial for determining electronic properties such
as optical characteristics and conductivity in atomic systems, and $\psi$,
i.e., the electronic wavefunctions essential for understanding electron
distribution and interactions, underscore good application potential of our
method for tasks like material design and quantum computing.

The experimental results above indicate that the proposed representation
learning framework, while maintaining strict SO(3)-equivariance, significantly
enhances the expressiveness of equivariant representations, thus excellently
capturing the complexity of physical systems and substantially breakthroughs
the SOTA accuracy on the eight benchmark databases. The consistent
improvements brought by our framework to different baseline models (DeepH-E3
or QHNet) and in various challenging scenarios, such as thermal motions,
bilayer twists, scale variations, and new trajectories, fully demonstrate the
universality and significance of our proposed theory and method.

Table 1: Experimental results measured by the $MAE^{H}_{all}$, $MAE^{H}_{cha\\_s}$, and $MAE^{H}_{cha\\_b}$ metrics (meV) on the Monolayer Graphene ($MG$) and Monolayer MoS2 ($MM$) databases. $\downarrow$ means lower values of the metrics correspond to better accuracy. Methods | $MG$ | $MM$
---|---|---
$MAE$ $(\downarrow)$
$MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$ | $MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$
DeepH-E3 (Baseline) | 0.251 | 0.357 | 0.362 | 0.406 | 0.574 | 1.103
DeepH-E3+TraceGrad | 0.175 | 0.257 | 0.228 | 0.285 | 0.412 | 0.808
Table 2: Experimental results measured by the $MAE^{H}_{all}$, $MAE^{H}_{cha\\_s}$ and $MAE^{H}_{cha\\_b}$ metrics (meV) on the Bilayer Graphene ($BG$), Bilayer Bismuthene ($BB$), Bilayer Bi2Te3 ($BT$) and Bilayer Bi2Se3 ($BS$) databases, where the superscripts $nt$ and $t$ respectively denote the non-twisted and twisted subsets. $\downarrow$ means lower values of the metrics correspond to better accuracy. Methods | $BG^{nt}$ | $BG^{t}$
---|---|---
$MAE$ $(\downarrow)$
$MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$ | $MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$
DeepH-E3 (Baseline) | 0.389 | 0.453 | 0.644 | 0.264 | 0.429 | 0.609
DeepH-E3+TraceGrad | 0.291 | 0.323 | 0.430 | 0.198 | 0.372 | 0.406
Methods | $BB^{nt}$ | $BB^{t}$
$MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$ | $MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$
DeepH-E3 (Baseline) | 0.274 | 0.304 | 1.042 | 0.468 | 0.602 | 2.399
DeepH-E3+TraceGrad | 0.226 | 0.256 | 0.740 | 0.384 | 0.503 | 1.284
Methods | $BT^{nt}$ | $BT^{t}$
$MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$ | $MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$
DeepH-E3 (Baseline) | 0.447 | 0.480 | 1.387 | 0.831 | 0.850 | 4.572
DeepH-E3+TraceGrad | 0.295 | 0.312 | 0.718 | 0.735 | 0.755 | 4.418
Methods | $BS^{nt}$ | $BS^{t}$
$MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$ | $MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$
DeepH-E3 (Baseline) | 0.397 | 0.424 | 0.867 | 0.370 | 0.390 | 0.875
DeepH-E3+TraceGrad | 0.300 | 0.332 | 0.644 | 0.291 | 0.302 | 0.674
Figure 3: Visualization of $MAE^{H}_{block}$ on each basic block of the Hamiltonian matrices for representative datasets, where the superscripts $nt$ and $t$ respectively denote the non-twisted and twisted subsets. Lower value of $MAE^{H}_{block}$ corresponds to better accuracy. The $MAE^{H}_{block}$ for other datasets from DeepH series can be found in Supplementary Information E.  Table 3: Experimental results measured by the $MAE^{H}_{all}$, $MAE^{H}_{diag}$, $MAE^{H}_{non\\_diag}$, $MAE^{\epsilon}$, and $Sim(\psi)$ metrics on the QH9-Stable (QS) database using the ’ood’ split strategy [19]. $\downarrow$ means lower values of the metrics correspond to better accuracy, while $\uparrow$ means higher value of the metric corresponds to better performance. The units of MAE metrics are meV, while $Sim(\psi)$ is the cosine similarity which is dimensionless. Methods | MAE $(\downarrow)$ | $Sim(\psi)$ $(\uparrow)$
---|---|---
$MAE^{H}_{all}$ | $MAE^{H}_{diag}$ | $MAE^{H}_{non\\_diag}$ | $MAE^{\epsilon}$
QHNet | 1.962 | 3.040 | 1.902 | 17.528 | 0.937
QHNet+TraceGrad | 1.191 | 2.125 | 1.139 | 8.579 | 0.948
Table 4: Experimental results measured by the $MAE^{H}_{all}$, $MAE^{H}_{diag}$, $MAE^{H}_{non\\_diag}$, $MAE^{\epsilon}$, and $Sim(\psi)$ metrics on the QH9-Dynamic (QD) database using the ’mol’ split strategy [19]. $\downarrow$ means lower values of the metrics correspond to better accuracy, while $\uparrow$ means higher value of the metric corresponds to better performance. The units of MAE metrics are meV, while $Sim(\psi)$ is the cosine similarity which is dimensionless. Methods | MAE $(\downarrow)$ | $Sim(\psi)$ $(\uparrow)$
---|---|---
$MAE^{H}_{all}$ | $MAE^{H}_{diag}$ | $MAE^{H}_{non\\_diag}$ | $MAE^{\epsilon}$
QHNet | 4.733 | 11.347 | 4.182 | 264.483 | 0.792
QHNet+TraceGrad | 2.819 | 6.844 | 2.497 | 63.375 | 0.927

We present experimental results of ablation study in Supplementary Information
F. These fine-grained experiments assess the individual and combined
contributions of the two core mechanisms of the proposed TraceGrad method on
the six databases from the DeepH series. These mechanisms are the
SO(3)-invariant trace supervision mechanism (Trace) at the label level and the
gradient-based induction mechanism (Grad) at the representation layer. Results
indicate that each mechanism can contribute individually to the performance.
Moreover, their combination provides even better performance. This is because,
on one hand, with the gradient-based induction mechanism as a bridge, the
SO(3)-invariant features learned from the trace label can be transformed into
the SO(3)-equivariant representations during inference; on the other hand,
with trace label, the SO(3)-invariant network branch has a strong supervisory
signal, enabling it to learn the intrinsic symmetry and complexity of the
regression targets, enhancing the quality of SO(3)-invariant features and
ultimately benefits the encoding of SO(3)-equivariant features. The value of
such complementarity has been fully demonstrated in the experimental results.

##  5 Future Work

In future research, various extensions are conceivable across theoretical,
methodological, and application fields:

First, from a theoretical perspective, we can generalize SO(3) group symmetry
to a broader range and more complex groups. This will enable deep learning
frameworks to accommodate richer mathematical structures.

Second, from a methodological perspective, we plan to combine our framework
with advanced Transformer [23] and KAN [24] architectures to continuously
enhance the expressive capabilities of equivariant features.

Third, from the application perspective, the current methodology extends
beyond predicting electronic-structure Hamiltonians and is capable of
generally predicting multiple physical quantities and properties that require
equivariance. Furthermore, the methodology is not confined to physical
research alone and may find applications in areas such as robotics, autonomous
vehicles, and motion tracking systems. For instance, in the vision tasks of
autonomous vehicles [25], where there is a high demand for robustness against
coordinate transformations of cameras, the mainstream approach has been to
achieve approximate SO(3)-equivariance through data augmentation. However,
this does not guarantee absolute reliability and safety. Our work holds great
potential to construct deep models with high generalization performance under
the premise of strict SO(3)-equivariance, potentially revolutionizing
techniques in such tasks. Looking forward to future endeavors, the
foundational and pivotal role of this work is clear.

##  6 Method

###  6.1 Encoding and Decoding Framework

As shown in Fig. 2, our encoding framework corresponds to a total of $K$
encoding modules, which are sequentially connected to form a cascading stack.
For the $k$-th module ($1\leq k\leq K$), it first introduces an
SO(3)-equivariant base encoder, e.g., the encoder from DeepH-E3 [8] or QHNet
[7] which is composed of recently developed equivariant operators [26, 27]
like linear scaling, direct products, direct sums, Clebsch-Gordan
decomposition, gated activation, equivariant normalization, and etc., to
encode the physical system’s initial representations, such as spherical
harmonics [28], or representations passed from the previous neural layers, as
equivariant feature $\textbf{f}^{(k)}$ in the current hidden layer. Next, we
construct the feature $\textbf{v}^{(k)}=g_{\text{nonlin}}(\textbf{f}^{(k)})$,
to achieve sufficient non-linear expressiveness while maintaining
SO(3)-equivariance, where $g_{\text{nonlin}}(\cdot)$ is the function defined
in Eq. (5). $s_{nonlin}(\cdot)$, i.e., $g_{\text{nonlin}}(\cdot)$’s built-in
non-linear mechanism, which can be any differentiable non-linear neural
module, is implemented here as a feed-forward module with a hidden layer size
of $1024$, using Layer Normalization for stabilization and regularization, and
employing SiLU as the non-linear mapping. To facilitate learning and
optimization, we connect the input and output of $g_{\text{nonlin}}(\cdot)$ in
a residual manner [29], serving as the output of the current encoding module,
i.e., $\textbf{o}^{(k)}=\textbf{f}^{(k)}+\textbf{v}^{(k)}$. $\textbf{o}^{(k)}$
is then fed into the next encoder module to yield $\textbf{f}^{(k+1)}$,
$\textbf{o}^{(k+1)}$, …., until the last encoder module, which is followed by
the decoder modules.

For an expressive feature encoder handling a complex physical system,
$\textbf{f}^{(k)}$ is usually not a feature of single degree but a direct-sum
concatenation of series of components $\textbf{f}^{(k)l_{1}}$,
$\textbf{f}^{(k)l_{2}}$, …, $\textbf{f}^{(k)l_{k}}$ at multiple degrees, i.e.,
$l_{1}$, $l_{2}$, …, $l_{k}$, where some components share the same degree
while others differ. In this case, it becomes necessary to extend the
decomposition operator, i.e., $\text{CGDecomp}(\cdot)$ in Eq. (5), to
accommodate various components of degrees. Moreover, in the context of neural
networks, introducing learnable parameters $W$ may improve the model capacity.
Based on these considerations, when constructing the encoding module, the
$\text{CGDecomp}(\cdot)$ operation in Eq. (5) can be expanded as
$\text{CGDecomp}_{ext}(\cdot)$:

$u^{(k)}=\text{CGDecomp}_{\text{ext}}(\textbf{f}^{(k)}\otimes\textbf{f}^{(k)},0,W)=\sum_{1\leq
i,j\leq
k,l_{i}=l_{j}}W_{ij}\cdot\text{CGDecomp}(\textbf{f}^{(k)l_{i}}\otimes\textbf{f}^{(k)l_{j}},0)$
(6)

where $W$ represents learnable parameters. This expansion does not change the
equivariance or invariance of the features. It is obvious that, the output
($u^{(k)}$) of $\text{CGDecomp}_{\text{ext}}(\cdot)$ is also SO(3)-invariant,
just like the original $\text{CGDecomp}(\cdot)$. Thus, even with such an
expansion, the feature $z^{(k)}=s_{\text{nonlin}}(u^{(k)})$ remains
SO(3)-invariant and $\textbf{v}^{(k)}=\frac{\partial
z^{(k)}}{\partial\mathbf{f}^{(k)}}$ remain SO(3)-equivariant.

The outputted SO(3)-equivariant features, i.e., $\textbf{o}^{(K)}$, as well as
the outputted SO(3)-invariant feature, i.e., $\textbf{z}^{(K)}$, from the
final ($K$ th) layer of the encoding modules, are fed into the corresponding
decoding modules to yield the predictions of $\mathbf{Q}$ and
$\mathbf{T}=tr(\mathbf{Q}\cdot\mathbf{Q}^{\dagger})$, respectively.
Specifically, for the Hamiltonian prediction task, $\mathbf{Q}$ corresponds to
$\mathbf{H}^{l_{p}\otimes l_{q}}$ ($1\leq p\leq P,\,1\leq q\leq Q$) in Fig. 1,
while $\mathbf{T}=\text{tr}(\mathbf{H}^{l_{p}\otimes
l_{q}}\cdot(\mathbf{H}^{l_{p}\otimes l_{q}})^{\dagger})$.

Since complex non-linear transformations have already been incorporated in the
encoding framework, the decoders can be simple: the SO(3)-equivariant decoder
is implemented using an SO(3)-equivariant linear mapping [27], and the
SO(3)-invariant decoder is implemented as a shallow feed-forward module.

###  6.2 Training and Inference

The training loss function is shown as the following:

$\begin{split}\min_{\theta}loss&=loss_{Q}+\mu(loss_{Q},loss_{T})\cdot
loss_{T},\\\ loss_{Q}&=MSE(\widehat{\mathbf{Q}},\mathbf{Q}^{*}),\\\
loss_{T}&=MSE(\widehat{\mathbf{T}},\mathbf{T}^{*})\end{split}$ (7)

where $\theta$ denotes all of the learnable parameters of our framework,
$\widehat{\mathbf{Q}}$, $\widehat{\mathbf{T}}$ and $\mathbf{Q}^{*}$,
$\mathbf{T}^{*}$ respectively denote the predictions and labels of
$\mathbf{Q}$ and $\mathbf{T}$. In order that the both of the SO(3)-equivariant
and SO(3)-invariant branches can be stably converged, we introduce
$\mu(loss_{Q},loss_{T})$, i.e., a function returning a coefficient to
regularize the relative scale of the two loss terms:

$\mu(loss_{Q},loss_{T})=\lambda\cdot No\\_Grad(\frac{loss_{Q}}{loss_{T}})$ (8)

where $No\\_Grad(\cdot)$ denotes gradient discarding when calculating such
coefficient, as this coefficient is only used for adjusting the relative scale
of the two loss terms and should not itself be a source of training gradients,
otherwise it would counteract $loss_{T}$ in Eq. (7). The introduction of
$\mu(loss_{Q},loss_{T})$ effectively prevents the numerical disparity between
the two loss terms to stabilize the training for both of them. All of the
encoding and decoding modules are trained jointly by Eq. (7), and the training
configurations are included in Supplementary Information D.

In our framework, the SO(3)-invariant non-linear features $z^{(k)}(1\leq k\leq
K)$ learned with the help of $\mathbf{T}$ contribute to the learning of
SO(3)-equivariant representations $\textbf{v}^{(k)}(1\leq k\leq K)$ and the
prediction of $\mathbf{Q}$. To achieve this goal, in the training phase, both
the SO(3)-invariant feature encoder and the SO(3)-invariant decoder, used for
predicting $\mathbf{T}$ from $z^{(k)}$, need to be trained. While in the
inference phase, since $\mathbf{T}$ is not the final regression target, the
SO(3)-invariant decoder can be removed, yet the SO(3)-invariant encoder still
should be retained, used to induce SO(3)-equivariant features.

###  6.3 Time Complexity

Typically, one of the key advantage of deep learning algorithms for
electronic-structure calculation lies in their low computational complexity
which can asymptotically scale as $O(N)$, where $N$ is the number of atoms of
a system. This is a significant efficiency advantage compared to the
$O(N^{3})$ computational complexity of traditional DFT method, enabling
efficient and low-cost prediction of electronic structures for large-scale
atomic systems. Our method improves the accuracy of Hamiltonian predictions
within the deep learning paradigm while maintaining this advantage of
asymptotic complexity. The baseline models we select, whether DeepH-E3 or
QHNet, are efficient models with an asymptotic computational complexity of
$O(N)$ as $N$ increases. The neural module $g_{nonlin}(\cdot)$ we introduce
efficiently operates on each individual feature f from the baseline models to
obtain the corresponding non-linear SO(3)-equivariant feature v, rather than
relying on complex interactions between multiple features to derive new ones.
This ensures that the computational complexity remains asymptotically $O(N)$,
with good potential to efficiently handle large atomic systems.

##  7 Acknowledgements

This work was supported by the National Natural Science Foundation of China
(Grant Number 12134012) and the Strategic Priority Research Program of Chinese
Academy of Sciences (Grant Number XDB0500201).

## References

  *   * Zhang et al. [2023] Zhang, X., Wang, L., Helwig, J., Luo, Y., Fu, C., Xie, Y., Liu, M., Lin, Y., Xu, Z., Yan, K., Adams, K., Weiler, M., Li, X., Fu, T., Wang, Y., Yu, H., Xie, Y., Fu, X., Strasser, A., Xu, S., Liu, Y., Du, Y., Saxton, A., Ling, H., Lawrence, H., Stärk, H., Gui, S., Edwards, C., Gao, N., Ladera, A., Wu, T., Hofgard, E.F., Tehrani, A.M., Wang, R., Daigavane, A., Bohde, M., Kurtin, J., Huang, Q., Phung, T., Xu, M., Joshi, C.K., Mathis, S.V., Azizzadenesheli, K., Fang, A., Aspuru-Guzik, A., Bekkers, E., Bronstein, M., Zitnik, M., Anandkumar, A., Ermon, S., Liò, P., Yu, R., Günnemann, S., Leskovec, J., Ji, H., Sun, J., Barzilay, R., Jaakkola, T., Coley, C.W., Qian, X., Qian, X., Smidt, T., Ji, S.: Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv preprint arXiv:2307.08423 (2023)
  * Hohenberg and Kohn [1964] Hohenberg, P., Kohn, W.: Inhomogeneous electron gas. Physical Review 136(3B), 864 (1964)
  * Kohn and Sham [1965] Kohn, W., Sham, L.J.: Self-consistent equations including exchange and correlation effects. Physical Review 140(4A), 1133 (1965)
  * Unke et al. [2021] Unke, O., Bogojeski, M., Gastegger, M., Geiger, M., Smidt, T., Müller, K.-R.: Se(3)-equivariant prediction of molecular wavefunctions and electronic densities, vol. 34, pp. 14434–14447 (2021)
  * Gu et al. [2022] Gu, Q., Zhang, L., Feng, J.: Neural network representation of electronic structure from ab initio molecular dynamics. Science Bulletin 67(1), 29–37 (2022)
  * Li et al. [2022] Li, H., Wang, Z., Zou, N., Ye, M., Xu, R., Gong, X., Duan, W., Xu, Y.: Deep-learning density functional theory hamiltonian for efficient ab initio electronic-structure calculation. Nature Computational Science 2(6), 367–377 (2022)
  * Yu et al. [2023] Yu, H., Xu, Z., Qian, X., Qian, X., Ji, S.: Efficient and equivariant graph networks for predicting quantum hamiltonian. In: International Conference on Machine Learning, pp. 40412–40424 (2023)
  * Gong et al. [2023] Gong, X., Li, H., Zou, N., Xu, R., Duan, W., Xu, Y.: General framework for e (3)-equivariant neural network representation of density functional theory hamiltonian. Nature Communications 14(1), 2848 (2023)
  * Guo et al. [2020] Guo, Y., Wang, H., Hu, Q., Liu, H., Liu, L., Bennamoun, M.: Deep learning for 3d point clouds: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(12), 4338–4364 (2020)
  * Qian et al. [2022] Qian, R., Lai, X., Li, X.: 3d object detection for autonomous driving: A survey. Pattern Recognition 130, 108796 (2022)
  * Zitnick et al. [2022] Zitnick, L., Das, A., Kolluru, A., Lan, J., Shuaibi, M., Sriram, A., Ulissi, Z.W., Wood, B.M.: Spherical channels for modeling atomic interactions. In: Conference on Neural Information Processing Systems (2022)
  * Liao and Smidt [2023] Liao, Y., Smidt, T.E.: Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. In: International Conference on Learning Representations (2023)
  * Musaelian et al. [2023] Musaelian, A., Batzner, S., Johansson, A., Sun, L., Owen, C.J., Kornbluth, M., Kozinsky, B.: Learning local equivariant representations for large-scale atomistic dynamics. Nature Communications 14(1) (2023)
  * Zhong et al. [2023] Zhong, Y., Yu, H., Su, M., Gong, X., Xiang, H.: Transferable equivariant graph neural networks for the hamiltonians of molecules and solids. npj Computational Materials 9(1), 182 (2023)
  * Passaro and Zitnick [2023] Passaro, S., Zitnick, C.L.: Reducing SO(3) convolutions to SO(2) for efficient equivariant gnns. In: International Conference on Machine Learning, pp. 27420–27438 (2023)
  * Liao et al. [2024] Liao, Y., Wood, B.M., Das, A., Smidt, T.E.: Equiformerv2: Improved equivariant transformer for scaling to higher-degree representations. In: International Conference on Learning Representations (2024)
  * Yin et al. [2024] Yin, S., Zhu, X., Gao, T., Zhang, H., Wu, F., He, L.: A hybrid deep learning framework for electronic-structure hamiltonian prediction. arXiv: 2401.00744 (2024)
  * Resnick [1991] Resnick, R.: Introduction to Special Relativity, (1991)
  * Yu et al. [2023] Yu, H., Liu, M., Luo, Y., Strasser, A., Qian, X., Qian, X., Ji, S.: QH9: A quantum hamiltonian prediction benchmark for QM9 molecules. In: Conference on Neural Information Processing Systems (2023)
  * Cao et al. [2018] Cao, Y., Fatemi, V., Fang, S., Watanabe, K., Taniguchi, T., Kaxiras, E., Jarillo-Herrero, P.: Unconventional superconductivity in magic-angle graphene superlattices. Nature 556(7699), 43–50 (2018)
  * Wang et al. [2024] Wang, C., Zhang, X.-W., Liu, X., He, Y., Xu, X., Ran, Y., Cao, T., Xiao, D.: Fractional chern insulator in twisted bilayer mote 2. Physical Review Letters 132(3) (2024)
  * He et al. [2024] He, M., Cai, J., Zheng, H., Seewald, E., Taniguchi, T., Watanabe, K., Yan, J., Yankowitz, M., Pasupathy, A., Yao, W., et al.: Dynamically tunable moiré exciton rydberg states in a monolayer semiconductor on twisted bilayer graphene. Nature Materials (2024)
  * Han et al. [2023] Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., Xu, Y., Yang, Z., Zhang, Y., Tao, D.: A survey on vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(1), 87–110 (2023)
  * Liu et al. [2024] Liu, Z., Wang, Y., Vaidya, S., Ruehle, F., Halverson, J., Soljacic, M., Hou, T.Y., Tegmark, M.: KAN: kolmogorov-arnold networks. CoRR abs/2404.19756 (2024) 2404.19756
  * Wang et al. [2023] Wang, Y., Mao, Q., Zhu, H., Deng, J., Zhang, Y., Ji, J., Li, H., Zhang, Y.: Multi-modal 3d object detection in autonomous driving: a survey. International Journal of Computer Vision 131(8), 2122–2152 (2023)
  * Thomas et al. [2018] Thomas, N., Smidt, T., Kearnes, S., Yang, L., Li, L., Kohlhoff, K., Riley, P.: Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219 (2018)
  * Geiger and Smidt [2022] Geiger, M., Smidt, T.E.: e3nn: Euclidean neural networks. CoRR abs/2207.09453 (2022)
  * Schrodinger [1926] Schrodinger, E.: Quantisierung als eigenwertproblem. Annalen der Physik 384(4), 361–376 (1926)
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE/CVF Computer Vision and Pattern Recognition Conference, pp. 770–778 (2016)
  * Fuchs et al. [2020] Fuchs, F., Worrall, D.E., Fischer, V., Welling, M.: Se(3)-transformers: 3d roto-translation equivariant attention networks. In: Conference on Neural Information Processing Systems (2020)
  * Schütt et al. [2019] Schütt, K.T., Gastegger, M., Tkatchenko, A., Müller, K.-R., Maurer, R.J.: Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions. Nature Communications 10(1), 5024 (2019)
  * Wang et al. [2024] Wang, Y., Li, H., Tang, Z., Tao, H., Wang, Y., Yuan, Z., Chen, Z., Duan, W., Xu, Y.: Deeph-2: Enhancing deep-learning electronic structure via an equivariant local-coordinate transformer. arXiv preprint arXiv:2401.17015 (2024)

##  Appendix A The Equivariance-Expressiveness Dilemma in Existing Deep
Learning Methods

In this section, we introduce existing efforts to harmonize the deep learning
paradigm with SO(3)-equivariance, analyzing the dilemma between strict
SO(3)-equivariance and non-linear expressive capabilities. This dilemma leads
to an unsatisfactory trade-off between strict equivariance and non-linear
expressiveness, ultimately limiting the achievable performance and scopes of
application for these regression models.

###  A.1 Approaches with Strict SO(3)-equivariance but Weak Expressiveness

Most existing methods have focused on achieving SO(3)-equivariance at the
neural mechanism level by applying group theory-guaranteed SO(3)-equivariant
operators [26, 27]. These operators, including linear scaling, element-wise
sum, direct products, direct sums, and Clebsch-Gordan decomposition, have been
used to build graph neural network architectures for tasks in 3D point cloud
analysis [30], molecular dynamic simulation and property prediction [13, 12],
as well as Hamiltonian prediction [31, 4, 8, 14]. However, these operators
could not provide sufficient expressiveness to fit complex non-linear
mappings. The majority of these operators are linear operations, and the only
one (direct product) that introduces some non-linearity can only approximate a
simple squaring function, and cannot be extended to more complex non-linear
forms due to the huge computational burden and difficulty in convergence. Even
worse, directly applying highly-expressive non-linear activation functions,
such as $Sigmoid$, $Softmax$, and $SiLU$, to SO(3)-equivariant features may
result in the loss of equivariance. To alleviate this issue, these methods
introduced a gated activation mechanism that processes SO(3)-invariant
features ($l=0$) using non-linear activations and employs them as linear
gating coefficients for SO(3)-equivariant features ($l\geq 1$), aiming to
enhance their expressive power while maintaining strict equivariance. However,
from the perspective of SO(3)-equivariant features, this mechanism is a
combination of linear operations and cannot fundamentally increase their non-
linear expressive capabilities.

###  A.2 Approaches with Remarkable Expressiveness but Approximate
SO(3)-equivariance

Li et al. [6] proposed a local coordinate strategy to achieve
SO(3)-equivariance. This strategy relied on the transformation relationship
between local and global coordinates to achieve equivariance, without imposing
restrictions on the neural network, allowing the use of network modules that
possess non-linear expressiveness. However, this strategy, solely depending on
coordinate transformations, only works under holistic rigid rotations of the
atomic system. Without exploration of a neural mechanism for equivariance,
when the local coordinates experience non-rigid perturbations such as thermal
movements or bilayer twists, this strategy offers no guarantees on maintaining
symmetry properties; Zitnick et al. [11] decomposed features into weighted
sums of spherical harmonic basis functions, each corresponding to a discrete
point on a unit sphere, through inner-product operations. Such discrete
decomposition enables non-linear operations on the coefficients to enhance
expressiveness. In subsequent developments [15, 16, 32], this approach has
demonstrated a remarkable capacity to fit complex functions. However, the
inner-product operations with discrete basis make such approach degenerate
from continuous to discrete rotational equivariance, no longer maintaining
strict equivariance to continuous rotational operations, as pointed out by
Zhang et al. [1]. Additionally, due to the high complexity of its discrete
sampling strategies, tensor orders ($m$) corresponding to large degrees ($l$)
are often partially pruned in architectures such as Equiformerv2 [16], further
sacrificing the equivariance of features. In summary, although these methods
with approximate SO(3)-equivariance achieve high numerical accuracy due to
their extensive use of non-linear mechanisms, they are best suited for the
applications that primarily require numerical approximations but not for tasks
that demand theoretical depth in physics. In some applications, the demands
for symmetry are extremely high. Even minor deviations from perfect symmetry
can result in incorrect physical results.

Combining these analyses, existing deep learning methods have compromised
either strict SO(3)-equivariance or non-linear expressiveness when processing
physical systems. In this paper, we develop a novel framework that
theoretically combines strict SO(3)-equivariance with the non-linear
expressiveness of neural networks. This solution resolves the equivariance-
expressiveness dilemma by deeply exploring the intrinsic relationships between
SO(3)-invariant and SO(3)-equivariant quantities and their representations.

##  Appendix B Proof of Theory

###  B.1 Proof of Theorem 1

###### Proof.

Under an SO(3) rotation represented by the rotational matrix $\mathbf{R}$,
$\mathbf{Q}=\mathbf{Q}^{l_{p}\otimes l_{q}}$ is transformed as
$\mathbf{Q}(\mathbf{R})$:

$\mathbf{Q}(\mathbf{R})=\mathbf{D}^{l_{p}}(\mathbf{R})\cdot\mathbf{Q}\cdot\mathbf{D}^{l_{q}}(\mathbf{R})^{\dagger},$

where $\mathbf{D}^{l_{p}}(\mathbf{R})$ and $\mathbf{D}^{l_{q}}(\mathbf{R})$
are the Wigner-D matrices for the degrees of $l_{p}$ and $l_{q}$,
respectively, corresponding to the rotation $\mathbf{R}$.

The conjugate transpose of the transformed quantity is:

$\mathbf{Q}(\mathbf{R})^{\dagger}=\mathbf{D}^{l_{q}}(\mathbf{R})\cdot\mathbf{Q}^{\dagger}\cdot\mathbf{D}^{l_{p}}(\mathbf{R})^{\dagger}.$

Thus, the product
$\mathbf{Q}(\mathbf{R})\cdot\mathbf{Q}(\mathbf{R})^{\dagger}$ is:

$\mathbf{Q}(\mathbf{R})\cdot\mathbf{Q}(\mathbf{R})^{\dagger}=(\mathbf{D}^{l_{p}}(\mathbf{R})\cdot\mathbf{Q}\cdot\mathbf{D}^{l_{q}}(\mathbf{R})^{\dagger})\cdot(\mathbf{D}^{l_{q}}(\mathbf{R})\cdot\mathbf{Q}^{\dagger}\cdot\mathbf{D}^{l_{p}}(\mathbf{R})^{\dagger}).$

Using the cyclic property of the trace, which states that the trace of a
product of matrices remains unchanged under cyclic permutations (i.e.,
$tr(ABC)=tr(BCA)=tr(CAB)$), and combining the properties that
$\mathbf{D}^{l_{p}}(\mathbf{R})\cdot\mathbf{D}^{l_{p}}(\mathbf{R})^{\dagger}=\mathbf{I}$
and
$\mathbf{D}^{l_{q}}(\mathbf{R})\cdot\mathbf{D}^{l_{q}}(\mathbf{R})^{\dagger}=\mathbf{I}$,
we can rearrange the terms inside the trace as follows:

$\mathbf{T}(\mathbf{R})=tr(\mathbf{Q}(\mathbf{R})\cdot\mathbf{Q}(\mathbf{R})^{\dagger})=tr((\mathbf{D}^{l_{p}}(\mathbf{R})\cdot\mathbf{Q}\cdot\mathbf{D}^{l_{q}}(\mathbf{R})^{\dagger})\cdot(\mathbf{D}^{l_{q}}(\mathbf{R})\cdot\mathbf{Q}^{\dagger}\cdot\mathbf{D}^{l_{p}}(\mathbf{R})^{\dagger}))$
$=tr(\mathbf{D}^{l_{p}}(\mathbf{R})\cdot\mathbf{Q}\cdot\mathbf{Q}^{\dagger}\cdot\mathbf{D}^{l_{p}}(\mathbf{R})^{\dagger})=tr(\mathbf{Q}\cdot\mathbf{Q}^{\dagger}\cdot\mathbf{D}^{l_{p}}(\mathbf{R})^{\dagger}\cdot\mathbf{D}^{l_{p}}(\mathbf{R}))=tr(\mathbf{Q}\cdot\mathbf{Q}^{\dagger})=\mathbf{T}.$

Therefore, $\mathbf{T}=tr(\mathbf{Q}\cdot\mathbf{Q}^{\dagger})$ is invariant
under SO(3) transformations, its SO(3)-invariance is proved. ∎

###  B.2 Proof of Theorem 2

###### Proof.

The input feature $\mathbf{f}$ is SO(3)-equivariant, meaning that under an
SO(3) rotation represented by $\mathbf{R}$, it transforms as follows:

$\mathbf{f}(\mathbf{R})=\mathbf{D}^{l}(\mathbf{R})\cdot\mathbf{f}$

where $\mathbf{D}^{l}(\mathbf{R})$ is the Wigner-D matrix corresponding to
degree $l$.

First, according to group theory,
$u=\text{CGDecomp}(\mathbf{f}\otimes\mathbf{f},0)$ is an SO(3)-invariant
scalar because the degree-zero component from the Clebsch-Gordan decomposition
is invariant under rotations. Since applying a non-linear operation to an
SO(3)-invariant quantity does not change its invariance,
$z=s_{\text{nonlin}}(u)$ is also SO(3)-invariant, independent to the specific
form of $s_{\text{nonlin}}(\cdot)$. It formally holds that:

$z=z(\mathbf{R})$ (9)

Next, we apply the chain rule in Jacobian form. Considering
$\mathbf{f}(\mathbf{R})$ is a column vector, to facilitate the application of
the chain rule in vector form, we first transpose it into a row vector
$\mathbf{f}(\mathbf{R})^{T}$, then differentiate:

$\frac{\partial
z(\mathbf{R})}{\partial\mathbf{f}^{T}(\mathbf{R})}=\frac{\partial
z}{\partial\mathbf{f}^{T}(\mathbf{R})}=\frac{\partial
z}{\partial\mathbf{f}^{T}}\frac{\partial\mathbf{f}^{T}}{\partial\mathbf{f}(\mathbf{R})^{T}}=\frac{\partial
z}{\partial\mathbf{f}^{T}}\cdot\mathbf{D}^{l}(\mathbf{R})^{-1}=\frac{\partial
z}{\partial\mathbf{f}^{T}}\cdot\mathbf{D}^{l}(\mathbf{R})^{T}$ (10)

Here we utilize the property that
$\mathbf{D}^{l}(\mathbf{R})^{-1}=\mathbf{D}^{l}(\mathbf{R})^{T}$. Since the
representations of neural networks are generally real numbers, the Wigner-D
matrix is also real unitary 333In Theorem 1 and Theorem 2, the Wigner-D
matrices are in the complex and real fields, respectively, because the output
space may be complex while the internal representations of neural networks are
typically real. Although the internal representations of neural network
modules including $g_{\text{nonlin}}(\cdot)$ are in the real field, they are
still able to predict targets in complex field with equivariance. Literature
[8] has provided mechanisms for converting the network outputs in the real
field into regression targets with real and imaginary parts..

Finally, we transpose the result back to a column vector:

$\mathbf{v}(\mathbf{R})=g_{nonlin}(\mathbf{f}(\mathbf{R}))=(\frac{\partial
z(\mathbf{R})}{\partial\mathbf{f}^{T}(\mathbf{R})})^{T}=(\frac{\partial
z}{\partial\mathbf{f}^{T}}\cdot\mathbf{D}^{l}(\mathbf{R})^{T})^{T}=\mathbf{D}^{l}(\mathbf{R})\cdot\frac{\partial
z}{\partial\mathbf{f}}=\mathbf{D}^{l}(\mathbf{R})\cdot\mathbf{v}$ (11)

This proves that $g_{\text{nonlin}}(\cdot)$ is an SO(3)-equivariant non-linear
operator: when applying its non-linearity to a SO(3)-equivariant feature
$\mathbf{f}$, the output feature $\mathbf{v}$ remains SO(3)-equivariant. ∎

##  Appendix C Information of Experimental Databases

In this part, we provide detailed information about the experimental
databases, including the statistical information of the six databases from the
DeepH series [6, 8], and the two databases from the QH9 series [19], listed in
Table 5 and Table 6, respectively. Additionally, we visualize two types of
challenging testing samples: samples with non-rigid deformation from thermal
motions, as well as the bilayer samples with interlayer twists, which are
shown in Fig. 4 and Fig. 5, respectively.

Table 5: Statistic information of the six benchmark databases, i.e., Monolayer Graphene ($MG$), Monolayer MoS2 ($MM$), Bilayer Graphene ($BG$), Bilayer Bismuthene ($BB$), Bilayer Bi2Te3 ($BT$), Bilayer Bi2Se3 ($BS$), from the DeepH benchmark series [6, 8]. SOC: effects of Spin-Orbit Coupling. $m$: number of samples in the current dataset; $a_{max}$: maximum number of atoms for a sample in the current dataset. $a_{min}$: minimum number of atoms for a sample in the current dataset. $nt$: non-twisted samples. $t$: twisted samples. Statistic Types | MG | MM | BG | BB | BT | BS
---|---|---|---|---|---|---
Elements | C | Mo, S | C | Bi | Bi, Te | Bi, Se
SOC | weak | weak | weak | strong | strong | strong
Training ($nt$) | $m$ | 270 | 300 | 180 | 231 | 204 | 231
$a_{max}$ | 72 | 75 | 64 | 36 | 90 | 90
$a_{min}$ | 72 | 75 | 64 | 36 | 90 | 90
Validation ($nt$) | $m$ | 90 | 100 | 60 | 113 | 38 | 113
$a_{max}$ | 72 | 75 | 64 | 36 | 90 | 90
$a_{min}$ | 72 | 75 | 64 | 36 | 90 | 90
Testing ($nt$) | $m$ | 90 | 100 | 60 | 113 | 12 | 113
$a_{max}$ | 72 | 75 | 64 | 36 | 90 | 90
$a_{min}$ | 72 | 75 | 64 | 36 | 90 | 90
Testing ($t$) | $m$ | - | - | 9 | 4 | 2 | 2
$a_{max}$ | - | - | 1084 | 244 | 130 | 190
$a_{min}$ | - | - | 28 | 28 | 70 | 70
Table 6: Basic statistics of the two benchmark databases, QH9-Stable ($QS$) and QH9-Dynamic ($QD$), from the QH9 benchmark series [19]. For the $QS$ database, the ’ood’ strategy from the official settings is used to split the training, validation, and test sets, ensuring that the atom number of samples do not overlap across the three subsets. For the $QD$ database, the ’mol’ strategy from the official settings is employed to split the training, validation, and test sets, ensuring that there are no thermal motion samples from the same temporal trajectory across the three subsets. These split strategies aim to assess the regression model’s extrapolation capability with respect to the number of atoms as well as the temporal trajectories. $m$: number of samples in the current dataset. $a_{max}$: maximum number of atoms for a sample in the current dataset. $a_{min}$: minimum number of atoms for a sample in the current dataset. Statistic Types | QS | QD
---|---|---
Elements | C, H, O, N, F | C, H, O, N, F
Training | $m$ | 104,001 | 79,900
$a_{max}$ | 20 | 19
$a_{min}$ | 3 | 10
Validation | $m$ | 17,495 | 9,900
$a_{max}$ | 22 | 19
$a_{min}$ | 21 | 10
Testing | $m$ | 9,335 | 10,100
$a_{max}$ | 29 | 19
$a_{min}$ | 23 | 10
Figure 4: Visualization of testing samples exhibiting non-rigid deformations
due to thermal motions Figure 5: Visualization of testing samples with
interlayer twists.

##  Appendix D Training Configurations

$\lambda$ in the training loss function is set as $0.3$ according to parameter
selection on the validation sets. The initial learning rates for experiments
on the $MG$, $MM$, $BG$, $BB$, $BT$ and $BS$ databases are set as $0.003$,
$0.005$, $0.003$, $0.005$, $0.004$, and $0.005$, respectively, the training
batch size for them is set as $1$ and the optimizer is selected as Adam; the
initial learning rate for experiments on the $QS$ and $QD$ databases is set as
$0.0005$, the training batch size is set as $16$, and the optimizer is
selected as AdamW.

##  Appendix E Visualization of Block-level MAE Statistics

The MAE metrics (denoted as $MAE^{H}_{block}$) for the two setups, i.e.,
DeepH-E3 and DeepH-E3+TraceGrad, across different blocks of the Hamiltonian
matrix for six databases from the DeepH benchmark series are illustrated in
Fig. 6 and 7. Fig. 6 presents the results for monolayer structures, while Fig.
7 focuses on bilayer structures. From these figures, it can be observed that
our method, TraceGrad, brings significant accuracy improvements over the
baseline method, i.e., DeepH-E3, on most of the blocks of the Hamiltonian
matrix, especially on those blocks where DeepH-E3 performs poorly.

Figure 6: Visualization of $MAE^{H}_{block}$ on each basic block of the
Hamiltonian matrices for the $MG$ and $MM$ databases. Figure 7: Visualization
of $MAE^{H}_{block}$ on each basic block of the Hamiltonian matrices for the
non-twisted (marked with superscripts $nt$) and twisted (marked with
superscripts $t$) structures of $BG$, $BB$, $BT$ and $BS$.

##  Appendix F Ablation Study

We conduct fine-grained ablation study on the six databases from DeepH
benchmark series, comparing results from the four setups:

• DeepH-E3 [8]: the baseline model.

• DeepH-E3+Trace: this experimental setup, an ablation term, only implements
half part of our method. Specifically, it extends the architecture of DeepH-E3
by adding our SO(3)-invariant encoding and decoding branches and using the
trace quantity
$\mathbf{T}=\text{tr}(\mathbf{Q}\cdot\mathbf{Q}^{\dagger})=\text{tr}(\mathbf{H}^{l_{p}\otimes
l_{q}}\cdot(\mathbf{H}^{l_{p}\otimes l_{q}})^{\dagger})$ to train them. As for
ablation study, this setup does not include the gradient-based mechanism for
encoding SO(3)-equivariant non-linear features; instead, it directly uses the
SO(3)-equivariant features outputted by DeepH-E3 for Hamiltonian regression.

• DeepH-E3+Grad: this setup is also an ablation term and implements the other
half part of our method in contrast to the previous ablation term.
Specifically, it incorporates our SO(3)-invariant encoder branch as well as
the gradient-induced operator to yield SO(3)-equivariant non-linear features
based on the representations provided by the baseline encoders of DeepH-E3 for
Hamiltonian prediction. As for ablation study, this setup continues to use the
single-task training pipeline of DeepH-E3, predicting only the Hamiltonian
without joint supervision training through the trace label.

• DeepH-E3+TraceGrad: this is a complete implementation of our framework
extending beyond of the architecture and training pipeline of DeepH-E3, at the
label level, we introduce the trace quantity to guide the learning of
SO(3)-invariant features; Meanwhile, at the representation level, we leverage
the gradient operator to yield SO(3)-equivariant non-linear features for
Hamiltonian prediction.

Table 7: Ablation study MAE results (meV) on the $MG$ and $MM$ databases. $\downarrow$ means lower values of the metrics correspond to better accuracy.  Methods | $MG$ | $MM$
---|---|---
$MAE$ $(\downarrow)$
$MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$ | $MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$
DeepH-E3 (Baseline) | 0.251 | 0.357 | 0.362 | 0.406 | 0.574 | 1.103
DeepH-E3+Trace | 0.230 | 0.344 | 0.348 | 0.378 | 0.537 | 1.091
DeepH-E3+Grad | 0.185 | 0.269 | 0.258 | 0.308 | 0.453 | 0.924
DeepH-E3+TraceGrad | 0.175 | 0.257 | 0.228 | 0.285 | 0.412 | 0.808
Table 8: Ablation study MAE results (meV) on $BG$, $BB$, $BT$and $BS$. The superscripts $nt$ and $t$ respectively denote the non-twisted and twisted subsets. $\downarrow$ means lower values of the metrics correspond to better accuracy. Methods | $BG^{nt}$ | $BG^{t}$
---|---|---
$MAE$ $(\downarrow)$
$MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$ | $MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$
DeepH-E3 (Baseline) | 0.389 | 0.453 | 0.644 | 0.264 | 0.429 | 0.609
DeepH-E3+Trace | 0.362 | 0.417 | 0.593 | 0.251 | 0.401 | 0.480
DeepH-E3+Grad | 0.320 | 0.356 | 0.511 | 0.222 | 0.389 | 0.446
DeepH-E3+TraceGrad | 0.291 | 0.323 | 0.430 | 0.198 | 0.372 | 0.406
Methods | $BB^{nt}$ | $BB^{t}$
$MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$ | $MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$
DeepH-E3 (Baseline) | 0.274 | 0.304 | 1.042 | 0.468 | 0.602 | 2.399
DeepH-E3+Trace | 0.259 | 0.285 | 0.928 | 0.429 | 0.570 | 1.782
DeepH-E3+Grad | 0.243 | 0.272 | 0.824 | 0.406 | 0.542 | 1.431
DeepH-E3+TraceGrad | 0.226 | 0.256 | 0.740 | 0.384 | 0.503 | 1.284
Methods | $BT^{nt}$ | $BT^{t}$
$MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$ | $MAE^{H}_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$
DeepH-E3 (Baseline) | 0.447 | 0.480 | 1.387 | 0.831 | 0.850 | 4.572
DeepH-E3+Trace | 0.406 | 0.462 | 1.239 | 0.784 | 0.812 | 4.520
DeepH-E3+Grad | 0.342 | 0.365 | 0.750 | 0.742 | 0.786 | 4.463
DeepH-E3+TraceGrad | 0.295 | 0.312 | 0.718 | 0.735 | 0.755 | 4.418
Methods | $BS^{nt}$ | $BS^{t}$
$MAE_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$ | $MAE_{all}$ | $MAE^{H}_{cha\\_s}$ | $MAE^{H}_{cha\\_b}$
DeepH-E3 (Baseline) | 0.397 | 0.424 | 0.867 | 0.370 | 0.390 | 0.875
DeepH-E3+Trace | 0.382 | 0.397 | 0.843 | 0.351 | 0.367 | 0.838
DeepH-E3+Grad | 0.343 | 0.365 | 0.696 | 0.324 | 0.339 | 0.746
DeepH-E3+TraceGrad | 0.300 | 0.332 | 0.644 | 0.291 | 0.302 | 0.674

Experimental results of the four setups are listed in Table 7 and 8. From the
results of ablation terms, we can obtain a more fine-grained experimental
analysis. By comparing among the results of DeepH-E3, DeepH-E3+Trace,
DeepH-E3+Grad, and DeepH-E3+TraceGrad, we can conclude that the two core
mechanisms of our method, i.e., the SO(3)-invariant trace supervision
mechanism (Trace) at the label level as well as the gradient-based induction
mechanism (Grad) at the representation layer, can contribute to the
performance individually. Moreover, their combination provides even better
performance.