# Invariant classes for families of complexes

Daniel Levin& & Mathematical Institute
University of Oxford
Andrew Wiles Building
Radcliffe Observatory Quarter (550)
Woodstock Road
Oxford
OX2 6GG
United Kingdom  and  Alexander Zuevsky∗ ∗ Institute of Mathematics
Czech Academy of Sciences
Zitna 25, 11567
Prague
Czech Republic<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

We consider families of chain-cochain infinite complexes $\mathcal{C}$ of
spaces with elements depending on a number of parameters, and endowed with a
converging associative multiple product. The existence of left/right
local/non-local square-vanishing ideals is assumed for subspaces of
$\mathcal{C}$-spaces. We show that a set of differential and orthogonality
relations together with coherence conditions on indices of a chain-cochain
complex $\mathcal{C}$ elements generates families of graded differential
algebras. With the appropriate orthogonality conditions on completions of
$\mathcal{C}$ elements in the multiple product, we define the equivalence
classes of cohomology invariants.

AMS Classification: 53C12, 57R20, 17B69

###### Key words and phrases:

Cohomology invariants; orthogonality condition for chain-cochain complexes;
graded differential algebras

Conflict of Interest and Data availability Statements:

The authors state that:

1.) The paper does not contain any potential conflicts of interests.

2.) The paper does not use any datasets. No dataset were generated during
and/or analysed during the current study.

3.) The paper includes all data generated or analysed during this study.

4.) Data sharing is not applicable to this article as no datasets were
generated or analysed during the current study.

5.) The data of the paper can be shared openly.

6.) No AI is used to write this paper.

##  1\. Introduction

The cohomology of complexes endowed with associative products is an important
way to study various algebraic and geometric structures. Computation of
cohomology classes of invariants associated to such complexes is difficult
problem. For differential forms considered on smooth manifolds, the Frobenius
theorem for a distribution leads to the orthogonality condition. Motivated by
the notion of integrability for differential forms on foliated manifolds [8,
10], we introduce in this paper the notion of orthogonality with respect to a
multiple product of elements of a chain-cochain complex. Differential and
orthogonality conditions then endow a complex with the structure of a graded
differential algebra. Assuming existence of square-vanishing ideals in
subspaces of families of complexes, and requiring natural orthogonality
conditions for completions of elements, cohomology classes are derived and
their independence is proven. The classes obtained constitute generalization
of Godbillon-Vey classes for foliations [8, 10].

As for possible applications of the material presented in this paper, we would
like to mention computations of higher cohomology of grading-restricted vertex
algebras [11] endowed with a sewing procedure [9, 15], constructions of more
complicated cohomology invariants, and applications in differential geometry
and algebraic topology. In particular, it will be possible to compute
cohomology invariants of vertex operator algebra bundles [5]. It would be
interesting to study possible applications of invariants we constructed to
cohomology of manifolds, continual Lie algebras [13], conformal field theory
[6, 14]. In differential geometry there exist various approaches to the
construction of cohomology classes (cf., in particular, [12]). The results
proven in this paper are useful in computations of foliations cohomology [1,
3].

##  2\. Setup and the main results

Introduce a system of families of horizontal chain-cochain complexes
$\mathcal{C}$ $=$ $\left(C^{n,\kappa}_{m}\right.$, $\delta^{n,\kappa}_{m}$,
$\left.\Delta^{n,\kappa}_{m}\right)$ with $n$, $m$, $\kappa\in\mathbb{Z}$. The
horizontal complexes with differentials
$\delta^{n,\kappa}_{m}:C^{n,\kappa}_{m}\to C^{n+1,\kappa}_{m-1}$ are supposed
to be chain-cochain, i.e.,
$\delta^{n+1,\kappa}_{m-1}\circ\delta^{n,\kappa}_{m}=0$, for all
$\kappa\in\mathbb{Z}$. In certain cases we take the vertical complexes with
differentials $\Delta^{n,\kappa}_{m}:C^{n,\kappa}_{m}\to C^{n,\kappa+1}_{m}$,
to be chain complexes with $\Delta^{n,\kappa}_{m}$ commuting with
$\delta^{n^{\prime},\kappa^{\prime}}_{m^{\prime}}$. When we write
$c\in\mathcal{C}$ that means that an element $c$ belongs to a space of
$\mathcal{C}$. Instead of families of $C^{n,\kappa}_{m}$-complexes for various
$\kappa$, one can consider families $C^{\bm{n}}_{\bm{m}}$ with multiple
indices ${\bm{n}}=(n_{1},\ldots,n_{M_{1}})$,
${\bm{m}}=(n_{1},\ldots,n_{M_{2}})$, for some $M_{1}$ and $M_{2}\geq 1$. Each
space of the multiple complex has two types of parameter with respect to the
action of differentials: one set of growing parameters $\bm{n}$, and one set
of lowering parameters $\bm{m}$. Such a construction will be considered in a
separate paper.

Let us give an example of a $\mathcal{C}$-complex with three parameters $n$,
$m$, $\kappa\in\mathbb{Z}$ which corresponds to the diagram

$\displaystyle\qquad\qquad\quad\vdots\qquad\qquad\qquad\qquad\vdots$ (2.1)
$\displaystyle\qquad\qquad\quad\downarrow\Delta^{n,\kappa-1}_{m}\qquad\;\qquad\downarrow\Delta^{n,\kappa-1}_{m-1}$
$\displaystyle\ldots\stackrel{{\scriptstyle\delta^{n-1,\kappa}_{m+1}}}{{\longrightarrow}}\quad
C^{n,\kappa}_{m}\qquad\stackrel{{\scriptstyle\delta^{n,\kappa}_{m}}}{{\longrightarrow}}\qquad
C^{n+1,\kappa}_{m-1}\stackrel{{\scriptstyle\delta^{n+1,\kappa}_{m-1}}}{{\longrightarrow}}\cdots$
$\displaystyle\qquad\qquad\quad\downarrow\Delta^{n,\kappa}_{m}\qquad\quad\qquad\downarrow\Delta^{n+1,\kappa}_{m-1}$
$\displaystyle\ldots\stackrel{{\scriptstyle\delta^{n-1,\kappa+1}_{m+1}}}{{\longrightarrow}}\quad
C^{n,\kappa+1}_{m}\stackrel{{\scriptstyle\delta^{n,\kappa+1}_{m}}}{{\longrightarrow}}\quad
C^{n+1,\kappa+1}_{m-1}\stackrel{{\scriptstyle\delta^{n+1,\kappa+1}_{m-1}}}{{\longrightarrow}}\cdots$
$\displaystyle\qquad\qquad\quad\downarrow\Delta^{n,\kappa+1}_{m}\qquad\quad\quad\downarrow\Delta^{n+1,\kappa+1}_{m-1}$
$\displaystyle\qquad\qquad\quad\vdots\qquad\qquad\qquad\qquad\vdots$

Since the differentials $\delta^{n,\kappa}_{m}$ and $\Delta^{n,\kappa}_{m}$
are supposed to act on elements of appropriate spaces $C^{n,\kappa}_{m}$ with
the same set of indices we will often omit indices and write $\delta$ and
$\Delta$.

For all combinations of $l$ elements of various spaces of $\mathcal{C}$ we
assume existence of a formal multiple associative product

$\cdot_{l}=\cdot_{i=1}^{l}:\times_{i=1}^{l}C^{n_{i},\kappa_{i}}_{m_{i}}\to
C^{n^{\prime},\kappa^{\prime}}_{m^{\prime}},$ (2.2)

for $n^{\prime}=\sum_{i=1}^{l}n_{i}$,
$\kappa^{\prime}=\sum_{i=1}^{l}\kappa_{i}$, $m^{\prime}=\sum_{i=1}^{l}m_{i}$.
It is assumed that a result of the $\cdot_{l}$-product which is an element of
a $\mathcal{C}$-space, is well-defined (e.g., converging) for all values of
parameters defining the product. The associativity of a $\cdot_{l}$-product
means
$(a\cdot_{l}\ldots\cdot_{l^{\prime}}b)\cdot_{l^{\prime\prime}}\ldots\cdot_{l^{\prime\prime\prime}}c=a\cdot_{l}\ldots\cdot_{l^{\prime}}(b\cdot_{l^{\prime\prime}}\ldots\cdot_{l^{\prime\prime\prime}}c)$.
for elements $a$, $b$, $c\in\mathcal{C}$.

In certain cases, a product (2.2) may lead to coincidence of parameters of
multiplied elements of a complex (2.1) spaces. Resulting product may not allow
such a coincidence resulting in possible overcounting of the number of
parameters (e.g., for a product of vertex operator algebra complexes [11]). In
order to avoid such a possibility, we take into account one coinciding
elements/parameters only in the resulting product. For an element $\phi_{i}\in
C^{n_{i},\kappa_{i}}_{m_{i}}$ in a particular $\cdot_{l}$-product let $r_{i}$
and $t_{i}$ be the numbers of common parameters with other elements in the
product corresponding to upper and lower indices for $\phi_{i}$. In that case,
the conditions for indices for the resulting element of a $\cdot_{l}$-product
are $n_{i}=\sum_{j=1}^{l}n_{j}-r_{j}$, and $m_{i}=\sum_{j=1}^{l}n_{j}-t_{j}$.

Note that $l$ can be infinite if we assure that the result of the product
$\times_{i\geq 1}$ is converging. We call the product formal since elements of
a complex $\mathcal{C}$ spaces can be formal geometric objects. Then the
result of a $\cdot_{l}$-product is a superposition of geometrical objects,
e.g., Riemann surfaces. In that case, convergence of the product means that
corresponding superposition leads to a well-defined formal object. Note that
we do not assume commutation relations for elements of a $\mathcal{C}$-spaces
to be known. All constructions of this paper are independent of actual
commutation formulas for elements inside a $\cdot_{l}$-product. We assume that
in the spaces of a complex $\mathcal{C}$ there exist a net of subspaces
constituting (left, right) square-vanishing ideals
$\mathcal{I}(2)\subset\mathcal{C}$. I.e., the if for elements $a^{\prime}$,
$a\in\mathcal{I}(2)\subset\mathcal{C}$, satisfying the condition
$\cdot_{l}\left(a^{\prime},b,a\right)=0$, then there exists $c$, $c^{\prime}$,
$c^{\prime\prime}$, $d$, $d^{\prime}\in\mathcal{C}$,
$b=\cdot_{l}(a^{\prime},c,d)$ or $b=\cdot_{l}(d^{\prime},c^{\prime},a)$, or
$b=\cdot_{l}(a^{\prime},c^{\prime\prime},a)$, for $b\in\mathcal{I}_{L}$ or
$b\in\mathcal{I}_{R}$ or $b\in\mathcal{I}_{L,R}$ correspondingly. We define
(left, right) local $\mathcal{I}_{l}(2)$ and non-local $\mathcal{I}_{nl}(2)$
ideals of order $2$ as the set of $\theta$, $\theta^{\prime}\in\mathcal{C}$
such that $\cdot_{l}(\ldots,\theta,\theta^{\prime},\ldots)=0$ or
$\cdot_{l}(\ldots,\theta,\ldots,\theta^{\prime},\ldots)=0$ correspondingly.
The cases of higher order ideals $\mathcal{I}(k)$, $k>2$ will be considered in
another paper.

Introduce the following notations. Since differentials and elements of
$\mathcal{C}$-complex are always assumed coherent, let us denote
$\delta=\delta^{i,k}_{m}$, $\Delta=\Delta^{i,k}_{m}$. Let $d=d^{i,k}_{m}$ be a
choice of $\delta$ or $\Delta$, and $\overline{d}$ denotes the alternative
choice. In addition to all that above it is set that a product (2.2) is
coherent with respect to all vertical and horizontal differentials of the
diagram (2.1). That means that for every $\kappa=\sum_{i=1}^{l}\kappa_{i}$,
$n=\sum_{i=1}^{l}n_{i}$, and $m=\sum_{i=1}^{l}m_{i}$, and corresponding
$\phi\in C^{n,\kappa}_{m}$ given by a $\cdot_{l}$-product, the action of
corresponding differential $d=d^{n,\kappa}_{m}$ has Leibniz formula

$d^{n,\kappa}_{m}\phi=d^{n,\kappa}_{m}\cdot_{l}\left(\phi_{1},\ldots,\phi_{i},\ldots,\phi_{l}\right)=\sum\limits_{i=1}^{l}\cdot_{l}\left(\ldots,d^{n_{i},\kappa_{i}}_{m_{i}}\phi_{i},\ldots\right),$
(2.3)

with respect to the elements $d^{n_{i},\kappa_{i}}_{m_{i}}\phi_{i}$,
$\phi_{i}\in C^{n_{i},\kappa_{i}}_{m_{i}}$.

For an element $\gamma\in\mathcal{C}$ and a set $J=(j_{1},\ldots j_{l})$ of
choices $j_{i}$ of $(0,1)$, let us require that for $l$ chain-cochain spaces
$C^{n_{i},\kappa_{i}}_{m_{i}}$ of the complex (2.1), there exist subspaces
$C^{{}^{\prime}n_{i},\kappa_{i}}_{m_{i}}\subset C^{n_{i},\kappa_{i}}_{m_{i}}$
such that for all $\phi_{i}\in C^{{}^{\prime}n_{i},\kappa_{i}}_{m_{i}}$,

$\delta\gamma=\cdot_{l}\left(\delta^{j_{1}}\phi_{1},\ldots,\delta^{j_{i}}\phi_{i},\ldots,\delta^{j_{l}}\phi_{l},\right).$
(2.4)

We call (2.4) a differential condition. Symmetrizing with respect to all
possible choices of $J_{k}$, $k\geq 1$, with a fixed $l$, we obtain a set of
differential conditions

$\left\\{Symm_{J_{l_{k}}}\left\\{\delta\gamma_{k}=\;\cdot_{l_{k}}\left(\delta^{j_{1,k}}\phi_{1,k},\ldots,\delta^{j_{i,k}}\phi_{i,k},\ldots,\delta^{j_{l_{k},k}}\phi_{l_{k},k}\right)\right\\}\right\\},$
(2.5)

for $\gamma_{k}\in\mathcal{C}$, $l_{k}\geq 1$, $k\geq 1$. With all
$\delta\gamma_{k}=0$, we call (2.5) the set of orthogonality conditions. For
each differential condition above of the set (2.5) we have the coherence
condition for corresponding upper and lower indices of $\mathcal{C}$-spaces,
i.e.,
$n_{\delta\gamma_{k}}=\sum_{i=1}^{l_{k}}n_{\delta^{j_{i},k}\phi_{i,k}}-r_{\delta^{j_{i},k}\phi_{i,k}}$,
$m_{\delta\gamma_{k}}=\sum_{i=1}^{l_{k}}m_{\delta^{j_{i},k}\phi_{i,k}}-t_{\delta^{j_{i},k}\phi_{i,k}}$,
and
$\kappa_{\delta\gamma_{k}}=\sum_{i=1}^{l_{k}}\kappa_{\delta^{j_{i},k}\phi_{i,k}}$.
We will skip these coherence relations for all differential conditions below.

The first result of this paper is the following Lemma 1 which will be
illustrated in Section 3.

######  Lemma 1.

Actions of the differentials $d$, $\overline{d}$ to differential and
orthogonality conditions with respect to a $\cdot_{l}$-multiplication (2.2),
$l\geq 0$, endow families of chain-cochain complexes $\mathcal{C}$ with the
structure of a multiple graded differential algebra.

For a set of $p$ elements $\phi_{i}$, $1\leq i\leq p$, in a
$\cdot_{l}$-product

$\cdot_{l}(\Phi_{1},\phi_{1},\Phi_{2},\ldots,\Phi_{i},\phi_{i},\Phi_{i+1},\ldots,\Phi_{p},\phi_{p},\Phi_{p+1}),$
(2.6)

we call the union of sets of $\mathcal{C}$-elements
$\left\\{\Phi_{j}\right\\}$, $1\leq j\leq p+1$, the completion of a set of
elements $\phi_{i}$ in a $\cdot_{l}$-product. Due to associativity of the
$\cdot_{l}$-product we can consider each set $\Phi_{j}$ as one element. The
completion $\left\\{\Phi_{j}\right\\}$, $1\leq j\leq p+1$, in (2.6) for a set
of elements $\phi_{i}$, $1\leq i\leq p$, with respect to the differential $d$
is called closed if

$\sum\limits_{j=1}^{p+1}d_{j}.\cdot_{l}(\Phi_{1},\phi_{1},\Phi_{2},\ldots,\Phi_{i},\phi_{i},\Phi_{i+1},\ldots,\Phi_{p},\phi_{p},\Phi_{p+1})=0,$
(2.7)

where $d_{j}$ acts only on $\Phi_{j}$-elements. Denote by $\xi_{\sigma(j)}$,
$1\leq j\leq 3$ the permutations with repetitions of the set of two
$\mathcal{C}$-element $(\phi,\psi)$. Let us define an auxiliary operator
$\epsilon:\mathcal{C}\times\mathcal{C}\to\mathcal{C}$, such that
$\epsilon(.,.)=(.,.)$, when $d$ or $\overline{d}$ is chain-cochain and
$\epsilon(.,.)={\rm Id}_{\mathcal{C}}$ otherwise. Consider the case when
$\overline{d}d\phi\neq 0$, for all $\phi\in\mathcal{C}$. We then formulate
then the second result of this paper:

######  Theorem 1.

For arbitrary elements $\phi$, $\psi\in\mathcal{C}$, such that $d\phi$,
$d\psi\in\mathcal{I}_{nl}(2)$, any set of closed completions
$\Phi_{j}\in\mathcal{C}$, $1\leq j\leq 4$, with respect to the choice $d$ of
differentials $\delta$ or $\Delta$ for the permutations $\xi_{\sigma(1\leq
j_{1},j_{2},j_{3}\leq 3)}$ with repetitions of the set $(\phi,\psi)$, i.e.,

$\left\\{{Symm\atop{\small 1\leq j_{1},j_{2},j_{3}\leq
3}}\left\\{0=\sum\limits_{j=1}^{4}d_{j}.\cdot_{l}\left(\epsilon\left(\Phi_{1},\overline{d_{i}}\xi_{\sigma(j_{1})}\right),\Phi_{2},d_{i}\xi_{\sigma(j_{2})},\Phi_{3},\xi_{\sigma(j_{3})},\Phi_{4}\right)\right\\}\right\\},$
(2.8)

the cohomology classes are given by

$Symm_{J}\;\left\\{\;\left[\cdot_{l}\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\phi,\Phi_{3},\phi,\Phi_{4}\right)\right]\;\right\\},$
(2.9)

independent on the choice of $\phi\in\mathcal{C}$.

The proof of Theorem 1 is given in Section 4. The result of Theorem 1
generalizes results of [8] for Godbillon-Vey invariants for codimension one
foliations, and results of [10] for differential forms defined on foliations.
The set of conditions (2.8) generalizes corresponding orthogonality conditions
assumed in [8, 10]. Due to the associativity property of the
$\cdot_{l}$-product mention above, we will skip the notation $\cdot_{l}$ and
denote all products for various $l$ as $(\ldots,\ldots,\ldots)$. Since all
actions of differentials of are supposed to be coherent with the indexing of
$\mathcal{C}$-spaces, we will skip also the upper and lower indices from
$\delta^{n,\kappa}_{m}$. Let us underline that we work with a complex spaces
as grading subspaces of an associative algebra. No commutation relations for
element of various subspaces are assumed known. In particular, the multiple
product $\cdot_{l}$ can be introduces for geometric objects of general kind.
Thus, we keep the constructions in this paper independent of commutation
relations.

##  3\. Graded differential algebras associated to families of chain-cochain
complexes

The way of manipulation with sequences of differential and orthogonality
conditions follows from the chain-cochain and Leibniz (2.3) conditions for
differentials, as well as the assumption that some of $\mathcal{C}$ belong to
the ideals $\mathcal{I}(2)$ or $\mathcal{I}_{nl}(2)$. Now we provide a
specific example illustrating Lemma 1.

######  Example 1.

Due to associativity, we consider each differential in a $\cdot_{l}$ product
for $1\leq i\leq l$ separately. A particular element $\delta^{j_{i}}\phi_{i}$
may be situated at an arbitrary position in the $\cdot_{l}$-product in a
differential or orthogonality condition defined by a set
$J=(j_{1},\ldots,j_{l})$, where $j_{i}=(0,1)$ is a choice of $0$ of $1$. In
this example we consider splittings of the product into two and three parts.
In general, starting from a particular orthogonality or differential
condition, we act consequently by $d$ and $\overline{d}$ differentials. For
elements belonging to the left of right ideals $\mathcal{I}_{l}(2)$ or
$\mathcal{I}_{nl}(2)$ of order two, the squares of any pair of their elements
vanish. Using the $\mathcal{I}(2)$ ideal vanishing properties applied to
orthogonality conditions, we express particular elements
$\delta^{j_{i}}\phi_{i}$ in terms of other $\mathcal{C}$ elements. Finally, by
taking into account corresponding coherence conditions, we arrive to the full
structure of differential relations. A sequence of relations does not stop as
long as coherence conditions on indices are fulfilled, or until the sequence
gives identical zero.

Let us reproduce the general structure of relations following from particular
orthogonality conditions. It is quite clear that differential conditions
always follow from some orthogonality conditions. In this example we consider
only two first types of cases where differential conditions with respect to
one differential $\delta$ follow from its applications to the orthogonality
conditions of the form $(\Phi,\phi)$ and $(\Phi^{\prime},\phi,\Phi)$ with one
picked element $\phi\in\mathcal{C}$ in a $\cdot_{l}$-product. The
consideration below extends to the case of higher number of picked elements in
a product. The case of arbitrary action of both differentials $d$ and
$\overline{d}$ on arbitrary number of elements in a multiple product will be
considered in a separate paper.

Let us start with the form $(00)$ and the differential $\delta$. Similar
computations can be done for the differential $\Delta$ and for combinations of
these two differentials. The differential part:

$\displaystyle(00):\;0=(\Phi^{\prime},\Phi);\;0=(\delta\Phi^{\prime},\Phi)+(\Phi^{\prime},\delta\Phi);\;0=(\delta\Phi^{\prime},\delta\Phi)+(\delta\Phi^{\prime},\delta\Phi);\;0\;\mbox{or}\;(II).$

The square vanishing-part

$\displaystyle(00):\;0=(\Phi^{\prime},\Phi);\;\Phi^{\prime}=(\beta^{\prime},\Phi);$
$\displaystyle
1)\;\delta\Phi^{\prime}=(\delta\beta^{\prime},\Phi)+(\beta^{\prime},\delta\Phi);\;0\;\mbox{or}\;(II);\;2)\;\delta\Phi=(\Phi^{\prime},\beta);\;(II).$

The case $(I0)$, differential part: $0=(\delta\phi,\Phi);\;(II)$. Non-
differential part:

$\displaystyle
1)\;\delta\phi=(\beta^{\prime},\Phi);\;0=(\delta\beta^{\prime},\Phi)+(\beta^{\prime},\delta\Phi);\;(II);\;2)\;\Phi=(\delta\phi,\beta);\;\delta\Phi=(\delta\phi,\delta\beta);\;0.$

Case $(0I)$: differential part: $0=(\Phi^{\prime},\delta\phi);\;(II)$. Non-
differential part:

$\displaystyle
1)\;\Phi^{\prime}=(\beta^{\prime},\delta\phi);\;\delta\Phi^{\prime}=(\delta\beta^{\prime},\delta\phi);\;0;\;2)\;\delta\phi=(\Phi^{\prime},\beta);\;0=(\delta\Phi^{\prime},\beta)+(\Phi^{\prime},\delta\beta),\;(II).$

Let us now consider the case $(II)\to(II)$. The differential part is trivial.
The square vanishing-part is:

$\displaystyle(II):\;0=(\delta\Phi^{\prime},\delta\Phi);\;1)\;\;\delta\Phi^{\prime}=(\beta^{\prime},\delta\Phi);\;(II):\;0=(\delta\beta^{\prime},\delta\Phi);\;2)\;\delta\Phi=(\delta\Phi^{\prime},\beta);\;(II).$

Let us continue with the differentiation of the form $(000)$:

$\displaystyle(000):\;0=(\Phi^{\prime},\phi,\Phi),\;0=(\delta\Phi^{\prime},\phi,\Phi)+(\Phi^{\prime},\delta\phi,\Phi)+(\Phi^{\prime},\phi,\delta\Phi),$
$\displaystyle
0=(\delta\Phi^{\prime},\delta\phi,\Phi)+(\delta\Phi^{\prime},\phi,\delta\Phi)+(\delta\Phi^{\prime},\delta\phi,\Phi)+(\Phi^{\prime},\delta\phi,\delta\Phi)+(\delta\Phi^{\prime},\phi,\delta\Phi)+(\delta\Phi^{\prime},\delta\phi,\delta\Phi),$
$\displaystyle(III):\;0=(\delta\Phi^{\prime},\delta\phi,\delta\Phi);\;1)\;\delta\Phi^{\prime}=(\beta^{\prime},\delta\phi);\;(II):\;0=(\delta\beta^{\prime},\delta\phi);$
$\displaystyle
2a)\;\delta\phi=(\delta\Phi^{\prime},\gamma);\;(II):0=(\delta\Phi^{\prime},\delta\gamma);\;2b)\;\delta\phi=(\gamma^{\prime},\delta\Phi);\;(II):\;0=(\delta\gamma^{\prime},\delta\Phi),$
$\displaystyle
3)\;\delta\Phi=(\beta,\delta\phi);\;(II):\;0=(\delta\beta,\delta\phi).$

Next we continue with the square vanishing-part of elements of $(000)$ and
differentiations:

$\displaystyle(000):\;0=(\Phi^{\prime},\phi,\Phi);\;1)\;\Phi^{\prime}=(\beta,\phi);\;\delta\Phi^{\prime}=(\delta\beta,\phi)+(\beta,\delta\phi);\;(II):\;0=(\delta\beta,\delta\phi);$
$\displaystyle
2a)\;\phi=(\Phi^{\prime},\beta),\;\delta\phi=(\delta\Phi^{\prime},\beta)+(\Phi^{\prime},\delta\beta);\;(II):\;0=(\delta\Phi^{\prime},\delta\beta).$
$\displaystyle
2b)\;\phi=(\beta,\Phi);\;\delta\phi=(\delta\beta,\Phi)+(\beta,\delta\Phi);\;(II):\;0=(\delta\beta,\delta\Phi).$
$\displaystyle
3)\;\phi=(\Phi^{\prime},\psi,\Phi);\;\delta\phi=(\delta\Phi^{\prime},\psi,\Phi)+(\Phi^{\prime},\delta\psi,\Phi)+(\Phi^{\prime},\psi,\delta\Phi);\;(III).$

The case $(I00)$ is equivalent to $(I0)$. The case $(00I)$ is equivalent to
$(0I)$. Consider the case $(I0I)$. The differential part:
$(I0I):\;0=(\delta\Phi^{\prime},\phi,\delta\Phi);\;(III)$. Non-differential
part:

$\displaystyle(I0I):\;0=(\delta\Phi^{\prime},\phi,\delta\Phi);\;1)\;\delta\Phi^{\prime}=(\beta^{\prime},\phi);\;(II).$
$\displaystyle
2a)\;\phi=(\delta\Phi^{\prime},\beta);\;\delta\phi=(\delta\Phi^{\prime},\delta\beta);\;0;\;2b)\;\phi=(\beta^{\prime},\delta\Phi);\;\delta\phi=(\delta\beta^{\prime},\delta\Phi);\;0;$
$\displaystyle
3)\;\phi=(\delta\Phi^{\prime},\gamma,\delta\Phi),\;\delta\phi=(\delta\Phi^{\prime},\delta\gamma,\delta\Phi);\;0;\;4)\;\delta\Phi=(\phi,\beta);\;(II).$

The case $(II0)$. The differential part:
$(II0):\;0=(\delta\Phi^{\prime},\delta\phi,\Phi);\;(III)$. Non-differential
part:

$\displaystyle(II0):\;0=(\delta\Phi^{\prime},\delta\phi,\Phi);\;1)\;\delta\Phi^{\prime}=(\beta^{\prime},\delta\phi);\;(II).$
$\displaystyle
2a)\;\delta\phi=(\delta\Phi^{\prime},\beta);\;(II);\;2b)\;\delta\phi=(\beta^{\prime},\Phi);\;(II);$
$\displaystyle
2c)\;\delta\phi=(\delta\Phi^{\prime},\gamma,\Phi);\;0;\;3)\;\Phi=(\delta\phi,\beta);\;\delta\Phi=(\delta\phi,\delta\beta);\;0.$

The case $(0II)$: differential part:
$(0II):\;0=(\Phi^{\prime},\delta\phi,\delta\Phi);\;(III)$. Non-differential
part:

$\displaystyle(0II):\;0=(\Phi^{\prime},\delta\phi,\delta\Phi);\;1)\;\Phi^{\prime}=(\beta^{\prime},\delta\phi);\;\delta\Phi^{\prime}=(\delta\beta^{\prime},\delta\phi);\;0;$
$\displaystyle
2a)\;\delta\phi=(\Phi^{\prime},\beta);\;(II);\;2b)\;\delta\phi=(\beta^{\prime},\delta\Phi);\;(II);\;2c)\;\delta\phi=(\Phi^{\prime},\gamma,\delta\Phi);$
$\displaystyle
0=(\delta\Phi^{\prime},\gamma,\delta\Phi)+(\Phi^{\prime},\delta\gamma,\delta\Phi);\;0=(III);\;3)\;\delta\Phi=(\delta\phi,\beta);\;(II).$

In the case of the sequence ending by $(II)$, we arrive at an infinite series
of differential and orthogonality conditions. Namely, with $k\in\mathbb{Z}$
(since a $(II)$ sequence can be seen infinite in both directions), we have the
sequence

$\displaystyle
0=(\alpha_{k},\beta_{k}),\quad\delta\alpha_{k}=(\alpha_{k+1},\beta_{k}),\quad\delta\beta_{k}=(\alpha_{k},\beta_{k+1}).$
(3.1)

Recall, that for each differential relation in this example, there exists the
coherence relation for corresponding $\mathcal{C}$-indices. Here
$\alpha_{k+1}$ and $\beta_{k+1}$ belong to the spaces of $\mathcal{C}$
corresponding to the relations (3.1) via the coherence relations. The
sequences of differential and orthogonality conditions derived above together
with the corresponding coherences relations for indices taken for all choices
of $J$ provide the full set relations defining a multiply graded differential
algebra.

The structure of relations for a resulting differential algebra as well as the
structure of corresponding invariants given by Theorem 1 depends which
position on the tree of differential and orthogonality conditions we start
with. Then the resulting multiple graded differential algebra is a reduction
of the full graded differential algebra. In addition to that, one can also
restrict the values of indices of gradings for vertical and horizontal parts
of a complex. E.g., one can set $n\geq 0$ instead of $\mathbb{Z}$. This would
affect coherence conditions for corresponding identities.

##  4\. The invariants

In this Section we give a proof of Theorem 1. Recall the notations given in
Section 2. The relations (2.8) play the role of orthogonality conditions on
$\Phi_{j}$, $1\leq j\leq 4$ for invariants (2.9).

###### Proof.

Let $\phi\in\mathcal{C}$. Now let us show that for $\cdot_{l}$-product
completions $\Phi_{j}$, $1\leq j\leq 4$, satisfying quite general restriction
(2.8) for the completion, we get invariant cohomology classes (2.9) not
depending on the choice of $\phi$ such that
$\delta\phi\in\mathcal{I}_{nl}(2)$. For some fixed choice of $J$, consider the
product
$\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right)\Phi_{2},d\phi,\Phi_{3},\phi,\Phi_{4}\right)$.
We have

$\displaystyle
d\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\phi,\Phi_{3},\phi,\Phi_{4}\right)=\left(\epsilon\left(d\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\phi,\Phi_{3},\phi,\Phi_{4}\right)$
$\displaystyle\quad+\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),d\Phi_{2},d\phi,\Phi_{3},\phi,\Phi_{4}\right)+\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\phi,d\Phi_{3},\phi,\Phi_{4}\right)$
$\displaystyle\quad+\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\phi,\Phi_{3},d\phi,\Phi_{4}\right)+(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\phi,\Phi_{3},\phi,d\Phi_{4})=0,$

when $d\phi\in\mathcal{I}_{nl}(2)$, and (2.8) is satisfied, i.e.,

$\sum_{j=1}^{4}d_{j}.\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\phi,\Phi_{3},\phi,\Phi_{4}\right)=0,$
(4.2)

so that the differential of the total completion is zero. Then (4) defines the
cohomology class
$\left[\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\phi,\Phi_{3},\phi,\Phi_{4})\right]$.

Now, let us check that for completions $\Phi_{j}$, $1\leq j\leq 4$, satisfying
(2.8), the cohomology classes (2.9) do not depend on the choice of $\phi$.
Indeed, let us substitute $\phi$ by $\phi+\eta$, with
$\delta\eta\in\mathcal{I}_{nl}(2)$, and satisfying conditions (2.8). Then,

$\displaystyle\left(\epsilon\left(\Phi_{1},\overline{d}d(\phi+\eta)\right),\Phi_{2},d(\phi+\eta),\Phi_{3},\phi+\eta,\Phi_{4}\right)$
$\displaystyle=\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\phi,\Phi_{3},\phi,\Phi_{4}\right)+\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\phi,\Phi_{3},\eta,\Phi_{4}\right)$
$\displaystyle+\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\eta,\Phi_{3},\phi,\Phi_{4}\right)+\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\eta,\Phi_{3},\eta,\Phi_{4}\right)$
$\displaystyle+\left(\epsilon\left(\Phi_{1},\overline{d}d\eta\right),\Phi_{2},d\phi,\Phi_{3},\phi,\Phi_{4}\right)+\left(\epsilon\left(\Phi_{1},\overline{d}d\eta\right),\Phi_{2},d\phi,\Phi_{3},\eta,\Phi_{4}\right)$
$\displaystyle+\left(\epsilon\left(\Phi_{1},\overline{d}d\eta\right),\Phi_{2},d\eta,\Phi_{3},\phi,\Phi_{4}\right)+\left(\epsilon\left(\Phi_{1},\overline{d}d\eta\right),\Phi_{2},d\eta,\Phi_{3},\eta,\Phi_{4}\right).$

Note that

$\displaystyle\left(\epsilon\left(d\Phi_{1},\overline{d}\phi\right),\Phi_{2},d\phi,\Phi_{3},\eta,\Phi_{4}\right)+\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\phi,\Phi_{3},\eta,\Phi_{4}\right)$
$\displaystyle+\left(\epsilon\left(\Phi_{1},\overline{d}\phi\right),d\Phi_{2},d\phi,\Phi_{3},\eta,\Phi_{4}\right)+\left(\epsilon\left(\Phi_{1},\overline{d}\phi\right),\Phi_{2},d\phi,d\Phi_{3},\eta,\Phi_{4}\right)$
$\displaystyle+\left(\epsilon\left(\Phi_{1},\overline{d}\phi\right),\Phi_{2},d\phi,\Phi_{3},d\eta,\Phi_{4}\right)+\left(\epsilon\left(\Phi_{1},\overline{d}\phi\right),\Phi_{2},d\phi,\Phi_{3},\eta,d\Phi_{4}\right)$
$\displaystyle=d\left(\epsilon\left(\Phi_{1},\overline{d}\phi\right),\Phi_{2},d\phi,\Phi_{3},\eta,\Phi_{4}\right)=\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\phi,\Phi_{3},\eta,\Phi_{4}\right),$

and similarly

$\displaystyle
d\left(\epsilon\left(\Phi_{1},\overline{d}\phi\right),\Phi_{2},d\eta,\Phi_{3},\phi,\Phi_{4}\right)=\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\eta,\Phi_{3},\phi,\Phi_{4}\right),$
$\displaystyle
d\left(\epsilon\left(\Phi_{1},\overline{d}\phi\right),\Phi_{2},d\eta,\Phi_{3},\eta,\Phi_{4}\right)=\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\eta,\Phi_{3},\eta,\Phi_{4}\right),$
$\displaystyle
d\left(\epsilon\left(\Phi_{1},\overline{d}\eta\right),\Phi_{2},d\phi,\Phi_{3},\phi,\Phi_{4}\right)=\left(\epsilon\left(\Phi_{1},\overline{d}d\eta\right),\Phi_{2},d\phi,\Phi_{3},\phi,\Phi_{4}\right),$
$\displaystyle
d\left(\epsilon\left(\Phi_{1},\overline{d}\eta\right),\Phi_{2},d\phi,\Phi_{3},\eta,\Phi_{4}\right)=\left(\epsilon\left(\Phi_{1},\overline{d}d\eta\right),\Phi_{2},d\phi,\Phi_{3},\eta,\Phi_{4}\right),$
$\displaystyle
d\left(\epsilon\left(\Phi_{1},\overline{d}\eta\right),\Phi_{2},d\eta,\Phi_{3},\phi,\Phi_{4}\right)=\left(\epsilon\left(\Phi_{1},\overline{d}d\eta\right),\Phi_{2},d\eta,\Phi_{3},\phi,\Phi_{4}\right),$
$\displaystyle
d\left(\epsilon\left(\Phi_{1},\overline{d}\eta\right),\Phi_{2},d\eta,\Phi_{3},\eta,\Phi_{4}\right)=\left(\epsilon\left(\Phi_{1},\overline{d}d\eta\right),\Phi_{2},d\eta,\Phi_{3},\eta,\Phi_{4}\right),$

due to (4.2) and $d\phi$, $d\eta\in\mathcal{I}_{nl}(2)$. Therefore,

$\displaystyle\left(\epsilon\left(\Phi_{1},\overline{d}d(\phi+\eta)\right),\Phi_{2},d(\phi+\eta),\Phi_{3},\phi+\eta,\Phi_{4}\right)$
$\displaystyle=$
$\displaystyle\left(\epsilon\left(\Phi_{1},\overline{d}d\phi\right),\Phi_{2},d\phi,\Phi_{3},\phi,\Phi_{4}\right)+d\left(\left(\epsilon\left(\Phi_{1},\overline{d}\phi\right),\Phi_{2},d\phi,\Phi_{3},\eta,\Phi_{4}\right)\right.$
$\displaystyle+$
$\displaystyle\left(\epsilon\left(\Phi_{1},\overline{d}\phi\right),\Phi_{2},d\eta,\Phi_{3},\phi,\Phi_{4}\right)+\left(\epsilon\left(\Phi_{1},\overline{d}\phi\right),\Phi_{2},d\eta,\Phi_{3},\eta,\Phi_{4}\right)$
$\displaystyle+$
$\displaystyle\left(\epsilon\left(\Phi_{1},\overline{d}\eta\right),\Phi_{2},d\phi,\Phi_{3},\phi,\Phi_{4}\right)+\left(\epsilon\left(\Phi_{1},\overline{d}\eta\right),\Phi_{2},d\phi,\Phi_{3},\eta,\Phi_{4}\right)$
$\displaystyle+$
$\displaystyle\left.\left(\epsilon\left(\Phi_{1},\overline{d}\eta\right),\Phi_{2},d\eta,\Phi_{3},\phi,\Phi_{4}\right)+\left(\epsilon\left(\Phi_{1},\overline{d}\eta\right),\Phi_{2},d\eta,\Phi_{3},\eta,\Phi_{4}\right)\right).$

Thus, the cohomology class is preserved under the transformations
$\phi\to\phi+\eta$. In the opposite choice of $d$, we use the symmetry of
$\delta$ and $\Delta$ for vertical chain-cochain complexes with differentials
$\Delta$ commuting with differentials $\delta$. The considerations above
extend to the symmetrized expression (2.9). ∎

## Acknowledgment

The second author is supported by the Institute of Mathematics, Academy of
Sciences of the Czech Republic (RVO 67985840).

## References

  * [1] Bazaikin, Ya. V., Galaev, A. S. Losik classes for codimension one foliations, J. Inst. Math. Jussieu 21 (2022), no. 4, 1391–1419.
  * [2] Ya. V. Bazaikin, A. S. Galaev, and P. Gumenyuk. Non-diffeomorphic Reeb foliations and modified Godbillon-Vey class, Math. Z. (2022), no. 2, 1335–1349.
  * [3] Ya. V. Bazaikin, A. S. Galaev, N. I. Zukova. Chaos in Cartan foliations. Chaos 30 (2020), no. 10, 103116.
  * [4] M. Crainic and I. Moerdijk. ${\rm\check{C}}$ech-De Rham theory for leaf spaces of foliations. Math. Ann. 328 (2004), no. 1–2, 59–85.
  * [5] E. Frenkel, D. Ben-Zvi. Vertex algebras and algebraic curves. Mathematical Surveys and Monographs, 88\. American Mathematical Society, Providence, RI, 2001. xii+348 pp.
  * [6] Ph. Francesco, P. Mathieu, and D. Senechal. Conformal Field Theory. Graduate Texts in Contemporary Physics. 1997.
  * [7] A. S. Galaev. Comparison of approaches to characteristic classes of foliations. arXiv:1709.05888.
  * [8] E. Ghys. L’invariant de Godbillon-Vey. Seminaire Bourbaki, 41–eme annee, n 706, S. M. F. Asterisque 177–178 (1989).
  * [9] B. Gui. Convergence of sewing conformal blocks. Comm. Contemp. Math. Vol. 26, No. 03, 2350007 (2024).
  * [10] D. Kotschick. Godbillon-Vey invariants for families of foliations. Symplectic and contact topology: interactions and perspectives (Toronto, ON/Montreal, QC, 2001), 131–144, Fields Inst. Commun., 35, Amer. Math. Soc., Providence, RI, 2003.
  * [11] Y.-Zh. Huang. A cohomology theory of grading-restricted vertex algebras. Comm. Math. Phys. 327 (2014), no. 1, 279–307.
  * [12] M. V. Losik. On some generalization of a manifold and its characteristic classes (Russian), Funcional. Anal. i Prilozhen. 24(1990), no 1, 29-37 ; English translation in Functional Anal. Appl. 24 (1990), 26–32.
  * [13] M. V. Saveliev, A. M. Vershik. Continuum analogues of contragredient Lie algebras. Commun. Math. Phys. 126, 367, 1989.
  * [14] A. Tsuchiya, K. Ueno, and Y. Yamada, Y. Conformal field theory on universal family of stable curves with gauge symmetries, Adv. Stud. Pure. Math. 19 (1989), 459–566.
  * [15] A. Yamada. Precise variational formulas for abelian differentials. Kodai Math.J. 3 (1980), 114–143.


# Graph Contrastive Learning for Materials

Teddy Koker Keegan Quigley Will Spaeth Nathan C. Frey Lin Li
MIT Lincoln Laboratory
Lexington, MA 02421-6426
<EMAIL_ADDRESS>

###### Abstract

Recent work has shown the potential of graph neural networks to efficiently
predict material properties, enabling high-throughput screening of materials.
Training these models, however, often requires large quantities of labelled
data, obtained via costly methods such as ab initio calculations or
experimental evaluation. By leveraging a series of material-specific
transformations, we introduce CrystalCLR, a framework for constrastive
learning of representations with crystal graph neural networks. With the
addition of a novel loss function, our framework is able to learn
representations competitive with engineered fingerprinting methods. We also
demonstrate that via model finetuning, contrastive pretraining can improve the
performance of graph neural networks for prediction of material properties and
significantly outperform traditional ML models that use engineered
fingerprints. Lastly, we observe that CrystalCLR produces material
representations that form clusters by compound class.

##  1 Introduction

The discovery of novel materials is a problem of considerable interest.
Machine learning (ML) has demonstrated potential in accelerating material
discovery with accurate prediction performance and lower computational cost
[1]. Traditional ML models used in material discovery rely on expert knowledge
to construct material fingerprints as inputs to ML models [2]. The development
of fingerprint-based representations require manual design which is often
limited to specific material families and properties and can be time-consuming
to compute.

Recent advances in deep learning provide an alternative way of learning
feature representations automatically from data. In particular, graph neural
networks (GNNs) have been developed for modeling crystalline materials to
predict material properties [3, 4, 5, 6, 7, 8]. In addition, Magar et al. [9]
have introduced several augmentations specific to crystalline structures,
including perturb-structure and supercell. They find that training GNNs on
augmented crystal data results in improved performance in prediction of
several material properties. While GNNs are becoming increasingly popular,
like in other deep learning domains, adoption is limited by the availability
of high quality labelled data [10]. Oftentimes, we only have access to very
limited labelled data, such as materials’ physical properties (e.g., melting
point, thermal conductivity) that require laborious and time-consuming
experimentation or computationally expensive ab initio calculations. One way
to tackle this problem is to transfer the learned model trained on a related
base task with large labelled data to the target task [11]. However, this
approach is limited by the availability of large labelled data for the base
task. In the absence of labelled data, self-supervised methods provide a way
to learn representations from samples alone.

Contrastive learning is one such method of self-supervised learning, which
seeks to learn representations such that similar pairs remain close in
embedding space, while others remain distant. In recent years, this has become
a powerful method of both learning of representations, as well as self-
supervised pre-training for transfer learning, primarily in the image domain
[12]. You et al. [13] introduced a framework for applying contrastive learning
to graphs, using GNNs and broadly applicable transformations to biochemical
molecules, social network, and image super-pixel graphs. Similar work has
shown that contrastive pretraining can improve supervised performance in
predicting the chemical properties of molecules [14]. Lastly, Khosla et al.
[15] showed how supervised labels can be incorporated into the contrastive
learning framework, allowing these labels to be incorporated into the
representations. Despite the success of contrastive learning in a broad range
of domains, very little work has been done to apply contrastive learning to
crystalline materials.

In this work, we introduce CrystalCLR, a framework for the constrastive
representation learning of crystalline materials. We highlight the importance
of augmentation selection, and show that contrastive learning is capable of
learning representations competitive with fingerprinting methods. We also
incorporate a novel loss function into the contrastive learning objective,
which further improve representation quality. Furthermore, we show transfer
learning with the pretrained contrastive model can outperform supervised
learning for the prediction of material properties.

##  2 Method

Following [12, 13], we construct our constrastive learning framework with four
major components: augmentation, GNN encoder, projection head and contrastive
loss. Let $V$ and $E$ denote the atomic and bond attributes of a material’s
crystal structure, respectively. Each crystal graph $\mathcal{G}=(V,E)$ is
first augmented into a similar pair $\hat{\mathcal{G}}_{i}$ and
$\hat{\mathcal{G}}_{j}$ where
$\hat{\mathcal{G}}_{i}=(\hat{V}_{i},\hat{E}_{i})$ and the augmentations used
to transform $\mathcal{G}$ into $\hat{\mathcal{G}}$ are specific to the domain
of crystalline materials (Sec. 2.1). A GNN encoder, $f(\cdot)$, maps crystal
graphs $\hat{\mathcal{G}}_{i},\hat{\mathcal{G}}_{j}$ into representations
$\boldsymbol{h}_{i},\boldsymbol{h}_{j}$. For our work we use the CGCNN
architecture as the encoder, following the same graph representation of
crystal structure as inputs [3]. The projection head, a two layer MLP
$g(\cdot)$, projects representations $\boldsymbol{h}_{i},\boldsymbol{h}_{j}$
into 128-dimensional space to create projections
$\boldsymbol{z}_{i},\boldsymbol{z}_{j}$. The addition of a non-linear
projection prior to the loss function has shown to improve representation
quality [12]. The contrastive loss function seeks to maximize the agreement
between representations $\boldsymbol{z}_{i},\boldsymbol{z}_{j}$ augmented from
the same crystalline material while minimizing the agreement among the rest of
the pairs augmented from different crystalline materials. We use the NT-Xent
loss [16], written for pair $\boldsymbol{z}_{i},\boldsymbol{z}_{j}$ in a batch
of $N$ pairs as:

$\mathcal{L}^{\text{CLR}}_{i}=-\log\frac{\exp(\textrm{sim}(\boldsymbol{z}_{i},\boldsymbol{z}_{j})/\tau)}{\sum^{2N}_{k=1}\mathbbm{1}_{[k\neq
i]}\exp(\textrm{sim}(\boldsymbol{z}_{i},\boldsymbol{z}_{k})/\tau)}$ (1)

where $\textrm{sim}(\boldsymbol{u},\boldsymbol{v})$ denotes cosine similarity
($\boldsymbol{u}^{\top}\boldsymbol{v}/\lVert\boldsymbol{u}\rVert\lVert\boldsymbol{v}\rVert$).
The final loss is the sum across all positive pairs in the batch.

###  2.1 Crystal Augmentations

(a) Original

(b) Strain

(c) Perturb

(d) Column replace

(e) Supercell

Figure 1: Visualizations of the crystal transformations used. All transforms
are stochastic by nature with the exception of supercell.

We investigate four transformations specific to crystalline materials (Fig.
1). Each transformation is isoelectronic, preserving the valence electron
configuration of the material: (1) perturb structure: each site is perturbed
by a distance uniformly sampled between 0 and 50% of the minimum pairwise
distance within the structure; (2) strain: a random, anisotropic tensile
strain deformation is applied to the structure. Each lattice vector is
increased by a factor uniformly sampled from $[0,0.05]$; (3) column replace:
with probability $1$, a single, randomly selected site within the structure is
replaced with an element from the same column in the periodic table as the
original element; and (4) supercell: a supercell of the original crystal is
created, scaling each lattice vector by a factor of $3$. The perturb structure
strength range is selected based on values used in prior work [9], and the
strain strength range is selected based on physically reasonable values.

###  2.2 Optimizing for composition similarity

Under the notion that materials with similar composition may tend to have
similar properties, we hypothesize that explicitly guiding representations of
similarly composed materials together may improve the representation quality.
We propose an additional loss term, composition similarity (CS), to promote
representation similarity among materials that have one or more common
elements. Each crystal is assigned a binary vector $a\in\\{0,1\\}^{100}$,
indicating which of the possible 100 atoms (that exist in the dataset used)
are contained in the crystal. The set $P(i)=\\{p\in I:a_{i}\cdot a_{p}>0,i\neq
p\\}$ is the set of indices that share one or more atom with the $i^{th}$
crystal. The loss can then be defined as:

$\mathcal{L}^{\text{CS}}_{i}=-\log\Bigl{\\{}\frac{1}{|P(i)|}\sum_{j\in
P(i)}\frac{\exp(\text{sim}(z_{i},z_{j})/\tau)}{\sum^{2N}_{k=1}\mathbbm{1}_{[k\neq
i]}\exp(\text{sim}(z_{i},z_{k})/\tau)}\Bigr{\\}}$ (2)

Where $|P(i)|$ is the cardinality of the set. While prior work in [15] has
shown that placement of the summation over positives outside the $\log$ is
more theoretically optimal, we found the above formulation to produce better
representations empirically. In the final model, (2) is used in combination
with the original NT-Xent loss (1) and weighted equally.

###  2.3 Learning Task Description

Our self-supervised training dataset consists of 90,160 crystal structures
downloaded from Materials Project in April, 2021 [17]. For evaluation, we use
additional melting temperature, thermal conductivity, and bulk moduli (K_VRH)
datasets of sizes 3,014, 5,540, and 13,121 respectively. The melting
temperature data were experimental data scraped from MatWeb [18]. Thermal
conductivity data were downloaded from AFLOW [19]; all values are at 300K.
Bulk moduli data were downloaded from Materials Project. Each dataset is split
80/10/10 into train, test, and validation sets.

##  3 Experimental Setup and Results

###  3.1 Augmentation Study

Figure 2: Linear regression evaluation of mean-absolute-error (MAE) of melting
temperature. Single transformations are applied for elements along the
diagonal, while other elements contain two transformations applied
sequentially.

To evaluate the effects of individual and combinations of augmentations, we
train models under all of the pairwise combinations of augmentations. For this
study, we apply augmentations to only one sample in each positive pair. Each
model is evaluated by performing a linear regression on the learned
representations for each of the studied material properties. In our work, we
use no augmentations for evaluation. Similar to [12, 13], we observe that the
composition of transformations is crucial for representation quality. As shown
in Figure 2 for melting point prediction, the best representations are
obtained using the perturb-structure and column-replace transformations, and
the addition of supercell often deteriorates performance. We observe a similar
trend across all of the studied material properties.

###  3.2 Model Evaluation

We selected perturb-structure, strain, and column-replace as the final set of
data augmentations; each augmentation has a 50% chance of being applied to
each training sample. We trained two self-supervised models; each model was
trained for 5000 epochs, with a batch size of $512$. The first model
(CrystalCLR) uses $\mathcal{L}^{\text{CLR}}$ alone, while the second model
(CrystalCLR+CS) uses a final loss of
$\mathcal{L}^{\text{CLR}}+\mathcal{L}^{\text{CS}}$. We use the Adam optimizer
[20] with a learning rate of 1e-4.

To quantitatively evaluate the learned representations, we trained random-
forest regression models [21] to predict material properties. We compare the
predictive performance of the learned embeddings to several other material
fingerprinting methods, all obtainable without supervision: (1) Site stats
fingerprint that aggregates statistics of the CrystalNNFingerprint [22] using
structural order parameters, (2) Sine Coulomb fingerprint which is a variant
of the Coloumb matrix for periodic crystals [23], and (3) Element property
which is a weighted mean, standard deviation, minimum and maximum of element
properties obtained from Magpie [24]. See Table 2 for the list of properties
used. We use the default hyperparameters for the random forest from scikit-
learn [25].

Table 1 shows the representations learned from contrastive training perform at
or above traditional fingerprinting methods, with the exception of element
properties. Element properties contain additional information, including
melting point, which explain its high performance. Furthermore, we observe
that the addition of the composition loss yields an improvement in performance
across all properties.

Figure 3: UMAP [26] visualizations of CrystalCLR + CS and Element Properties
embeddings, colored by compound classes.

We also evaluate the use of contrastive learning as a pretraining task for
material property prediction. For each of aforementioned properties, we train
CGCNN models; one randomly initialized, and one initialized with the CGCNN
encoder weights from each of the two contrastive training methods. For each
model, we use the Adam optimizer [20] to fine-tune the models, using the
learning rate with the best validation performance.

Table 1 shows by transferring weights from the contrastive-pretrained encoder,
we are able to obtain more accurate predictions of material properties than a
randomly initialized model for melting temperature and bulk moduli. The
benefit of the composition loss is more evident for melting temperature
prediction than the thermal conductivity and bulk moduli properties.

Lastly, in Figure 3 we visualize produced crystal embeddings using UMAP [26].
CrystalCLR learns representations that form clusters by compound class,
despite training in a self-supervised manor.

| Melting temp (∘C) | Thermal cond. (Wm-1K-1) | K_VRH (GPa)
---|---|---|---
Random Forest evaluation:
Site Stats Fingerprint | $415.7\pm 2.2$ | $5.41\pm 0.02$ | $43.02\pm 0.11$
Sine Coulomb Fingerprint | $336.7\pm 2.2$ | $3.50\pm 0.03$ | $25.74\pm 0.13$
Element Properties | $\textbf{106.1}\pm 1.6$ | $\textbf{3.39}\pm 0.03$ | $\textbf{17.64}\pm 0.07$
CrystalCLR (ours) | $161.7\pm 2.5$ | $3.73\pm 0.03$ | $21.98\pm{0.11}$
CrystalCLR + CS (ours) | $156.7\pm 1.8$ | $3.51\pm 0.03$ | $20.79\pm 0.07$
Fine-tune:
Random init | $119.9\pm 2.0$ | $3.08\pm 0.16$ | $13.13\pm 0.25$
CrystalCLR (ours) | $106.8\pm 6.0$ | $\textbf{3.04}\pm 0.33$ | $\textbf{12.89}\pm 0.19$
CrystalCLR + CS (ours) | $\textbf{102.8}\pm 2.6$ | $3.11\pm 0.13$ | $12.95\pm 0.28$
Table 1: Evaluation of models, measured in mean absolute error. Each reported
value is the mean over five random seeds, along with standard deviation

##  4 Conclusion

In this work, we introduce a method for the contrastive training of graph
neural networks for representation learning of materials. First, we establish
several crystalline material specific transformations, and study the effects
of the composition of transformations on representation quality. In addition,
we introduce a novel loss function to explicitly maximize the similarity among
materials that share common elements. We show that our framework is capable of
producing representations competitive with engineered material fingerprinting
techniques. Finally we show that constrastive pretraining improves performance
over random initialization for downstream tasks. Future work will investigate
additional contrastive learning methods, the ability to use CrystalCLR
representations for material retrieval, and the applicability of equivariant
graph neural networks.

## Acknowledgments and Disclosure of Funding

DISTRIBUTION STATEMENT A. Approved for public release. Distribution is
unlimited. This material is based upon work supported by the Under Secretary
of Defense for Research and Engineering under Air Force Contract No.
FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations
expressed in this material are those of the author(s) and do not necessarily
reflect the views of the Under Secretary of Defense for Research and
Engineering.© 2022 Massachusetts Institute of Technology. Delivered to the
U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013
or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government
rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as
detailed above. Use of this work other than as specifically authorized by the
U.S. Government may violate any copyrights that exist in this work.

## References

  * Schleder et al. [2019] Gabriel R. Schleder, Antonio C. M. Padilha, Carlos Mera Acosta, Marcio Costa, and Adalberto Fazzio.  From dft to machine learning: recent approaches to materials science–a review.  _Journal of Physics: Materials_ , 2(3):032001, 2019.
  * Ramprasad et al. [2017] Rampi Ramprasad, Rohit Batra, Ghanshyam Pilania, Arun Mannodi-Kanakkithodi, and Chiho Kim.  Machine learning in materials informatics: recent applications and prospects.  _npj Computational Materials_ , 3(54), 2017.
  * Xie and Grossman [2018] Tian Xie and Jeffrey C Grossman.  Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties.  _Physical review letters_ , 120(14):145301, 2018\.
  * Chen et al. [2019] Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong.  Graph networks as a universal machine learning framework for molecules and crystals.  _Chemistry of Materials_ , 31(9):3564–3572, 2019\.  doi: 10.1021/acs.chemmater.9b01294.  URL https://doi.org/10.1021/acs.chemmater.9b01294.
  * Louis et al. [2020] Steph-Yves Louis, Yong Zhao, Alireza Nasiri, Xiran Wang, Yuqi Song, Fei Liu, and Jianjun Hu.  Graph convolutional neural networks with global attention for improved materials property prediction.  _Physical Chemistry Chemical Physics_ , 22(32):18141–18148, 2020.  ISSN 1463-9084.  doi: 10.1039/d0cp01474e.  URL http://dx.doi.org/10.1039/D0CP01474E.
  * Griggs et al. [2020] Petar Griggs, Lin Li, and Rajmonda Caceres.  Unified gnn architecture design for high-throughput material screening.  2020\.
  * Batzner et al. [2022] Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky.  E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials.  _Nature communications_ , 13(1):1–11, 2022.
  * Musaelian et al. [2022] Albert Musaelian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron J Owen, Mordechai Kornbluth, and Boris Kozinsky.  Learning local equivariant representations for large-scale atomistic dynamics.  _arXiv preprint arXiv:2204.05249_ , 2022.
  * Magar et al. [2021] Rishikesh Magar, Yuyang Wang, Cooper Lorsung, Chen Liang, Hariharan Ramasubramanian, Peiyuan Li, and Amir Barati Farimani.  Auglichem: Data augmentation library ofchemical structures for machine learning, 2021.
  * Fung et al. [2021] Victor Fung, Jiaxin Zhang, Eric Juarez, and Bobby G Sumpter.  Benchmarking graph neural networks for materials chemistry.  _npj Computational Materials_ , 7(1):1–8, 2021\.
  * Lee and Asahi [2021] Joohwi Lee and Ryoji Asahi.  Transfer learning for materials informatics using crystal graph convolutional neural network.  _Computational Materials Science_ , 190:110314, 2021.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020.
  * You et al. [2020] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen.  Graph contrastive learning with augmentations.  _Advances in Neural Information Processing Systems_ , 33:5812–5823, 2020.
  * Wang et al. [2022] Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani.  Molecular contrastive learning of representations via graph neural networks.  _Nature Machine Intelligence_ , 4(3):279–287, 2022.
  * Khosla et al. [2020] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  _Advances in Neural Information Processing Systems_ , 33:18661–18673, 2020.
  * Sohn [2016] Kihyuk Sohn.  Improved deep metric learning with multi-class n-pair loss objective.  _Advances in neural information processing systems_ , 29, 2016.
  * Jain et al. [2013] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al.  Commentary: The materials project: A materials genome approach to accelerating materials innovation.  _APL materials_ , 1(1):011002, 2013.
  * Gao et al. [2013] Zhi-Yu Gao, Guo-Quan Liu, et al.  Recent progress of web-enable material database and a case study of nims and matweb.  _Journal of Materials Engineering_ , 3(11):89–96, 2013.
  * Curtarolo et al. [2012] Stefano Curtarolo, Wahyu Setyawan, Shidong Wang, Junkai Xue, Kesong Yang, Richard H. Taylor, Lance J. Nelson, Gus L.W. Hart, Stefano Sanvito, Marco Buongiorno-Nardelli, Natalio Mingo, and Ohad Levy.  Aflowlib.org: A distributed materials properties repository from high-throughput ab initio calculations.  _Computational Materials Science_ , 58:227–235, 2012.  ISSN 0927-0256.  doi: https://doi.org/10.1016/j.commatsci.2012.02.002.  URL https://www.sciencedirect.com/science/article/pii/S0927025612000687.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Breiman [2001] Leo Breiman.  Random forests.  _Machine learning_ , 45(1):5–32, 2001.
  * Zimmermann and Jain [2020] Nils ER Zimmermann and Anubhav Jain.  Local structure order parameters and site fingerprints for quantification of coordination environment and crystal structure similarity.  _RSC advances_ , 10(10):6063–6081, 2020.
  * Faber et al. [2015] Felix Faber, Alexander Lindmaa, O Anatole von Lilienfeld, and Rickard Armiento.  Crystal structure representations for machine learning models of formation energies.  _International Journal of Quantum Chemistry_ , 115(16):1094–1101, 2015.
  * Ward et al. [2016] Logan Ward, Ankit Agrawal, Alok Choudhary, and Christopher Wolverton.  A general-purpose machine learning framework for predicting properties of inorganic materials.  _npj Computational Materials_ , 2(1):1–7, 2016\.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * McInnes et al. [2018] Leland McInnes, John Healy, and James Melville.  Umap: Uniform manifold approximation and projection for dimension reduction.  _arXiv preprint arXiv:1802.03426_ , 2018.

##  5 Appendix

###  5.1 Element Properties List

Table 2 lists the element properties used for the Element Property classifier.

Feature Name |  Description
---|---
Number |  Atomic Number
MendeleevNumber |  Mendeleev Number (position on the periodic table, counting columnwise from H)
AtomicWeight |  Atomic weight
MeltingT |  Melting temperature of element
Column |  Column on periodic table
Row |  Row on periodic table
CovalentRadius |  Covalent radius of each element
Electronegativity |  Pauling electronegativity
NsValence |  Number of filled s valence orbitals
NpValence |  Number of filled p valence orbitals
NdValence |  Number of filled d valence orbitals
NfValence |  Number of filled f valence orbitals
NValence |  Number of valence electrons
NsUnfilled |  Number of unfilled s valence orbitals
NpUnfilled |  Number of unfilled p valence orbitals
NdUnfilled |  Number of unfilled d valence orbitals
NfUnfilled |  Number of unfilled f valence orbitals
NUnfilled |  Number of unfilled valence orbitals
GSvolume_pa |  DFT volume per atom of T=0K ground state
GSbandgap |  DFT bandgap energy of T=0K ground state
GSmagmom |  DFT magnetic moment of T=0K ground state
SpaceGroupNumber |  Space group of T=0K ground state structure
Table 2: Element properties used. Feature names and descriptions are
reproduced from the supplementary materials of Ward et al. [24].