# Leveraging language representation for material recommendation, ranking, and
exploration

Jiaxing Qu1
& Yuxuan Richard Xie211footnotemark: 1
& Kamil M. Ciesielski4
& Claire E. Porter4
& Eric S. Toberer4
& Elif Ertekin1,3
These authors share equal contributions to this work.

###### Abstract

Data-driven approaches for material discovery and design have been accelerated
by emerging efforts in machine learning. However, general representations of
crystals to explore the vast material search space remain limited. We
introduce a material discovery framework that uses natural language embeddings
derived from language models as representations of compositional and
structural features. The discovery framework consists of a joint scheme that
first recalls relevant candidates, and next ranks the candidates based on
multiple target properties. The contextual knowledge encoded in language
representations conveys information about material properties and structures,
enabling both representational similarity analysis for recall, and multi-task
learning to share information across related properties. By applying the
framework to thermoelectrics, we demonstrate diversified recommendations of
prototype structures and identify under-studied high-performance material
spaces. The recommended materials are corroborated by first-principles
calculations and experiments, revealing novel materials with potential high
performance. Our framework provides a task-agnostic means for effective
material recommendation and can be applied to various material systems.

††footnotetext: 1 Department of Mechanical Science and Engineering, 2 Beckman
Institute for Advanced Science and Technology, 3 Materials Research
Laboratory, University of Illinois Urbana-Champaign, Urbana, IL, 61802, 4
Colorado School of Mines, Golden, CO, 80401

_Teaser_ Natural language representations provide opportunities to discover
functional materials with desired properties.

##  1 Introduction

Rapid growth of data in materials science has opened a data-centric paradigm
(Hey et al. (2009)) for discovery of novel materials. In this paradigm,
machine learning (ML) models trained on large material data sets can
computationally screen candidates for field-specific applications such as
batteries (Aykol et al. (2020)), thermoelectrics, (Wang et al. (2020)) and
solar cells (Mahmood and Wang (2021)), etc. The key objective of the model-
driven approach is to identify candidates that exhibit targeted, desirable
material properties. Extracting representative features of materials to
capture attributes is therefore a key to success of accurate model performance
and property prediction. Conventionally, material feature extraction has
consisted of hand-crafted descriptors that contain essential information
related to composition and crystal structure, relying on physical and
mathematical intuition (Schmidt et al. (2019); Behler and Parrinello (2007);
Isayev et al. (2017)). Until recently, materials’ atomic structures have been
treated as graphs, where convolution operations extract features from local
chemical environments for accurate property predictions (Xie and Grossman
(2018)). Subsequently, several models have been proposed to directly learn
features from material compositions or structures for supervised prediction
tasks (Chen et al. (2019); Choudhary and DeCost (2021); Yan et al. (2022); Jha
et al. (2018)). An outstanding challenge, however, is to identify a universal
and task-agnostic representation that can enables generalized efficient search
and navigation of the vast and largely unlabeled material space.

Advances in natural language processing have allowed information mining from
the large corpus of material science related literature in an unsupervised
fashion. A pioneering work utilizes word embeddings trained on a large
material text corpus to encode material science knowledge into information-
dense vector representations (Tshitoyan et al. (2019)). Given a context word
for technological application, e.g. “thermoelectrics”, candidate materials are
ranked by similarity to the word embedding of the context word. Word
embeddings obtained on material compositions have also shown competitive
performance on material property prediction tasks (Wang et al. (2021a)).
However, word embedding, such as in Word2Vec (Tshitoyan et al. (2019)), does
not capture the contextual meaning of the word that is present in a sentence.
Progress on contextual embedding models has been enabled by masked language
modeling to train Transformer-based language models (MatBERT – Trewartha et
al. (2022), MatSciBERT – Gupta et al. (2022)) for material discovery and
knowledge extraction from millions of unstructured material science
literatures. By employing pretrained BERT models, latent knowledge learnt from
the material science text corpus can be encoded into the representation and
then subjected to a number of subsequent prediction tasks.

Figure 1: a, The proposed funnel-based recommender framework in which
candidate materials are recalled, and ranked based on similarity to the query
material. b, The schematic workflow to screen candidate materials including
constructing language representations, recalling candidates, and multi-task
prediction for ranking.

In the context of new high-performance materials, some essential factors
should be included in the recommendation pipeline: (i) effective
representations of both chemical and structural complexity in the large
material space, (ii) successful recall of relevant candidates to the query
material or property of interest, and (iii) accurate candidate ranking based
on multiple desired functional properties. Previously, recommender-like
systems for materials were developed to filter by identifying materials for
which predicted confidence levels of target properties fall within a desirable
range for thermoelectrics (Gaultois et al. (2016)), to predict chemically
relevant compositions for pseudo-ternary systems (Seko et al. (2018a, b)), and
to propose experimental synthesis conditions (Hayashi et al. (2019)). However,
a systematic and generalizable recommendation approach, which incorporates the
factors mentioned above for representation, recall, and ranking, could
accelerate discovery of desirable material candidates across diverse
applications.

Here we present a material recommendation framework that leverages language
representations of composition and structure to explore a large space and
identify similar candidate materials, given a query material with targeted
desired properties. The framework invokes a funnel-based architecture
comprising a candidate generation (“recall”) step and a subsequent property
evaluation (“ranking”) step (Figure 1a). We first constructed representations
for $\sim$116,000 materials using text description as the input to the
transformer based language models. By evaluating different embedding methods
on various downstream tasks, we found that material language representations
are both highly potent in recalling relevant material candidates, and capable
of predicting properties with comparable performance to state-of-the-art
specialized ML models. For improved ranking, we introduced a multi-gate
mixture-of-experts (MMoE) model, a multi-task learning strategy, to exploit
correlations between material property prediction tasks (Figure 1b). As a
demonstration example of material discovery, we applied our framework to
search and recommend high-performance thermoelectrics (TEs) – materials that
convert waste heat into electricity. Using this framework, we successfully
identify structurally-diversified TE candidates that are relevant to query
materials. Additionally, we identify and further explore several under-
searched high-performance materials spaces including halide perovskite,
delafossite-like, and spinel-like structures as promising TE candidates. As an
evaluation of the effectiveness of this framework, we performed first-
principles calculations and experiments on the recommended materials and
successfully identified CuZn2GaTe4 as a new TE material that, under further
optimization, may demonstrate high performace.

Figure 2: a, Distributions of predicted $zT$ of the top-100 recalled
candidates for PbTe as the query material and randomly sampled 100 materials.
The distributions are also visualized in kernel density estimation. Predicted
$zT$ are obtained from the MMoE models (section 2.4). b, Adjusted $p$-values
of the candidates for the top-100 highest $zT$ materials from experimental
UCSB and ESTM datasets. Out of 100 materials, 94 show significantly different
$zT$ distributions from random. c, Recall results of seven high-performing TE
materials are highlighted on the UMAP projection of 116K material
representations. Each color corresponds to first 100 materials recalled via
cosine similarity. The UMAP projection is obtained on the best performing
embeddings (MatBERT, structure embedding) from Section 2.2. d, UMAP overlaid
with the predicted $zT$. All known materials with $zT$>0.4 from the
experimental datasets are indicated by stars.

##  2 Results and discussions

###  2.1 A language-based framework enables material recommendations and
discovery.

Machine learning-based recommender systems leverage a large corpus of training
data to provide precise suggestions when querying for items among a large
candidate pool (Covington et al. (2016); Gomez-Uribe and Hunt (2015)). During
the recommendation process, a funnel-based architecture is typically applied
for initial screening, followed by more fine-grained ranking steps. Inspired
by the standard design of recommender systems, we adapted our framework to
material science to effectively search a large space and recommend relevant
materials with similar functional performance to a query material.
Specifically, we designed a funnel-based architecture that can be decoupled
into a recall step and a ranking step (Figure 1a). To enable candidate recall
for a query material, we converted each material into text-based descriptions
that include both compositional and structural information. Using language
models (Trewartha et al. (2022); Gupta et al. (2022)) pretrained on material
science literature, we then obtained output embeddings on these text-based
material descriptions. These embeddings encode contextual representations to
capture compositional and structural features with high-level interactions
arising from self-attention (Devlin et al. (2018)). In the recall step
(“candidate generation”), candidates can be searched via cosine similarity
against the query material in the representation space (Figure 1b). In the
ranking step, recalled candidate materials are evaluated and ranked using a
multi-objective scoring function trained on the encoded representations to
simultaneously predict multiple material properties through neural networks.
For this work, we exploited task correlations between predicting five TE
properties by training multi-task learning MMoE models, which provided
improved accuracy compared to models trained on single tasks.

To obtain compositional and structural level representations for the database
consisting of 116K materials (Section 4.1), we embedded all material formulae
(e.g., "PbTe") and sentence descriptors automatically generated
(Robocrystallographer – Ganose and Jain (2019)) from the structures (e.g.,
"PbTe is Halite, Rock Salt structured and crystallizes in the cubic
$Fm\bar{3}m$ space group…") as the input to pretrained language models.
Embedding each formula or structure generates a dense vector output from the
model hidden layer, which contains latent material-specific knowledge learnt
during unsupervised pretraining. In Figure 2, we demonstrate that recalled
candidates in the representation space are not only compositionally and
structurally related to the query material, but also can exhibit similar
functional performance to a query material. Starting with known materials with
favorable properties for TEs such as PbTe, we analyzed the top recalled
candidates and found significantly different predicted figure-of-merit $zT$
distributions from random sampling as indicated by $p$-values (Figure 2a). We
repeated this experiment for a total of 100 materials with the known highest
$zT$; 94 of these show statistical significance with $p$<0.05 (Figure 2b),
showing that recalled materials show distributions that are distinct from
random. Moreover, low-dimensional Uniform Manifold Approximation and
Projection (UMAP) (McInnes et al. (2018)) of the material representations
display latent signatures of seven high-performing TE materials along with
their top-100 recalled materials, each indicated by a different color (Figure
2c). We further observed a distinct clustering pattern, in which known
materials with good $zT$ (>0.4) form a “band” in the projection (Figure 2d).
Additionally, the observed “band” overlaps with the MMoE-predicted high $zT$
(also Figure 2d). The distribution of $zT$ in the representation space
provides opportunities to explore under-explored material spaces, such as the
region enclosed in the grey box with high predicted $zT$.

To understand how individual steps contribute to the performance of our
material recommendation framework, in the following we assess the
effectiveness of different representation strategies, recall ability, and
property prediction via multi-task learning. Further, we demonstrate our
framework to search, ranking, and exploration tasks for TE materials.

###  2.2 Language models offer effective representations of material
composition, structure, and properties.

Figure 3: UMAP projections of 116K materials using different embedding models.
Materials are colored a, by anionic groups, and b, by cosine similarity to
PbTe under the current embedding model.

Effective representations require rendering information about material design
principles and intrinsic properties. We evaluated several strategies for
material representation, focusing on unsupervised generation of features to
convey diverse chemical and structural information. In total, we investigated
six embedding methods. For composition level representation, we embed the
material formula using pretrained word embedding Mat2Vec (Tshitoyan et al.
(2019)) and contextualized word embedding from MatSciBERT (Gupta et al.
(2022)) and MatBERT (Trewartha et al. (2022)). For structural level
representation, we obtained local environment based structure fingerprints
(Zimmermann and Jain (2020)) and sentence embeddings of text-based material
descriptions from MatSciBERT and MatBERT. Note that for BERT models, we
constructed embedding vectors from entire passages of text consisting of
human-readable crystal structure characteristics (Ganose and Jain (2019)), as
described in Section 4.3.

To assess whether the embedding models have encoded material knowledge in the
representations, we projected the six different material embedding vectors
into low-dimensional spaces with UMAP, as visualized in Figure 3. Embedded
materials consisting of groups 15 (pnictogen), 16 (chalcogen), and 17
(halogen) on the periodic table are indicated by color (Figure 3a). Overall,
structure level representations exhibit more distinct separation (well-defined
domains) by material groups, apart from fingerprints which are solely
determined by structural similarity and include information only about local
but not semi-local and global environments. By contrast, composition level
representations retain the expected chemical differences, but form more
disperse and heterogeneous clusters.

To better interpret the embedding results, we picked three well-studied TE
materials, including SnTe – a rock-salt structural analog of PbTe with highest
reported $zT$ of $\sim$1.8 (Tang et al. (2018)), CuGaTe2 – a diamond-like
semiconductor in chalcopyrite structure that achieves a $zT$ of 1.5 (Wu et al.
(2022)), as well as Mg3Sb2 – a layered Zintl phase with the highest $zT$ of
$\sim$1.65 (Zhang et al. (2017); Ohno et al. (2018)), and visualized their
proximity to PbTe in the representation space (Figure 3b). All three materials
have demonstrate high $zT$ around 1.5, but the high performance arises from
different combinations of properties relevant to TEs (i.e. electronic and
thermal transport) due to their different structures. Embedded materials in
the representation space follow our anticipated similarity (PbTe$\approx$SnTe
> CuGaTe2 > Mg3Sb2), apart from fingerprints and composition embeddings from
MatBERT.

Next, we quantitatively evaluated material embedding performance on downstream
property prediction tasks. We applied a feature-based-approach to train
regression models directly on the derived embeddings, instead of optimizing
BERT parameters on the task-specific loss, i.e., fine-tuning (Devlin et al.
(2018)). This approach is more computationally efficient due to fixed
features, and grants flexibility to adapt task-specific architectures or
combine features of various sources across different models. We list the cross
validation performance on predicting six material properties for 5,700
materials in Table 1. The task models were multi-layer perceptrons (MLPs) with
mean-absolute-error (MAE) training loss. The tasks consisted of band gap,
energy per atom, bulk modulus, shear modulus, Debye temperature, and
coefficient of thermal expansion from AFLOW dataset (Curtarolo et al. (2012)).
Performance metrics of models trained using several embeddings, such as
structure embeddings extracted from MatBERT, achieved accurate performance.
Moreover, by leveraging latent material science knowledge embeddings from
pretrained large language models, the language representation supports
learning in the face of data scarcity, a ubiquitous challenge in applying ML
to materials science. For small data with only 200 training materials, models
trained using these embeddings outperform graph neural networks (CGCNN – Xie
and Grossman (2018)) when tested on 100 independent materials (Supplementary
Figure S1). These results suggest that pretrained language models, in
combination with text-based structure descriptions, provide a competitive
avenue to generate features for material representations.

Table 1: Benchmarking six different embedding models on six regression property prediction tasks with MAEs and R2-scores. ($E$/atom – energy per atom (eV), $E_{\mathrm{g}}$ – band gap (eV), $K$ – bulk modulus (GPa), $G$ – shear modulus (GPa), $\theta$ – Debye temperature (K), $\alpha$ – coefficient of thermal expansion (K-1))  | Composition embedding | Structure embedding
---|---|---
Property |  Metric |  Mat2Vec |  MatSciBERT |  MatBERT |  Fingerprint |  MatSciBERT |  MatBERT
$E$/atom |  MAE |  0.47$\pm$0.02 |  0.42$\pm$0.01 |  0.37$\pm$0.01 |  1.13$\pm$0.02 |  0.32$\pm$0.02 |  0.29$\pm$0.03
|  R2 |  0.81$\pm$0.02 |  0.86$\pm$0.01 |  0.88$\pm$0.01 |  0.283$\pm$0.02 |  0.95$\pm$0.01 |  0.96$\pm$0.01
$E_{\mathrm{g}}$ |  MAE |  0.15$\pm$0.01 |  0.20$\pm$0.02 |  0.19$\pm$0.01 |  0.54$\pm$0.03 |  0.25$\pm$0.01 |  0.23$\pm$0.01
|  R2 |  0.92$\pm$0.02 |  0.88$\pm$0.02 |  0.88$\pm$0.01 |  0.45$\pm$0.04 |  0.88$\pm$0.01 |  0.89$\pm$0.01
log_$K$ |  MAE |  0.18$\pm$0.01 |  0.18$\pm$0.01 |  0.17$\pm$0.01 |  0.45$\pm$0.01 |  0.16$\pm$0.01 |  0.15$\pm$0.01
|  R2 |  0.83$\pm$0.01 |  0.83$\pm$0.03 |  0.85$\pm$0.02 |  0.26$\pm$0.02 |  0.90$\pm$0.01 |  0.93$\pm$0.01
log_$G$ |  MAE |  0.20$\pm$0.01 |  0.23$\pm$0.01 |  0.22$\pm$0.01 |  0.48$\pm$0.01 |  0.24$\pm$0.01 |  0.23$\pm$0.01
|  R2 |  0.82$\pm$0.01 |  0.80$\pm$0.01 |  0.81$\pm$0.02 |  0.29$\pm$0.03 |  0.83$\pm$0.01 |  0.84$\pm$0.01
log10_$\theta$ |  MAE |  0.06$\pm$0.01 |  0.07$\pm$0.01 |  0.06$\pm$0.01 |  0.13$\pm$0.01 |  0.07$\pm$0.01 |  0.06$\pm$0.01
|  R2 |  0.81$\pm$0.02 |  0.82$\pm$0.03 |  0.84$\pm$0.02 |  0.34$\pm$0.05 |  0.85$\pm$0.03 |  0.88$\pm$0.02
log10_$\alpha$ |  MAE |  0.07$\pm$0.01 |  0.07$\pm$0.01 |  0.07$\pm$0.01 |  0.15$\pm$0.01 |  0.07$\pm$0.01 |  0.06$\pm$0.01
|  R2 |  0.78$\pm$0.03 |  0.81$\pm$0.02 |  0.81$\pm$0.02 |  0.19$\pm$0.02 |  0.87$\pm$0.03 |  0.90$\pm$0.01

###  2.3 Unsupervised candidate recall extracts highly relevant materials.

For the recall (candidate generation) step, we use an unsupervised approach,
as the models need to generalize to unseen query materials without further
training while correctly recalling relevant candidates. In the unsupervised
context, each query material is an individual prediction task, where the goal
is to find a set of related materials. While material recall is strictly based
on cosine similarity in embedding space from text-based descriptions of
composition and/or structure, we hypothesize that these embeddings contain
latent material science knowledge, in which recalled materials will also share
some similarities in properties to the query material.

For commercial recommender systems, online learning (Xiao et al. (2018); Song
et al. (2014)) makes data collection and model evaluation straightforward. For
our framework, we evaluate recalled TE materials in an offline setting with
predefined ‘relevancy’ (Section 4.5) as a measure of the the composite
differences in TE properties. We considered five TE properties – power factor,
Seebeck coefficient, electrical conductivity, thermal conductivity, and $zT$
for 826 unique host materials (Section 4.1) For each query material, the
relevancy is obtained as the summation of absolute differences of these five
properties, i.e. candidates with similarity across all properties are
considered most relevant.

For evaluation, precision and normalized discounted cumulative gain (nDCG)
were used as recall performance metrics (see Section 4.5). Specifically, we
calculated precision$@$15% to assess the recall accuracy by defining the top
15% of 826 materials (124 materials) as ‘relevant’ to the query material based
on experimental TE properties. We then evaluated the overlap between the top
124 recalled and relevant materials. A higher precision$@$15% score indicates
that more relevant materials have been recalled. On the other hand, nDCG
evaluates the ranking from the perspective of relative positions of items in
the similarity-based list. The two metrics are jointly visualized in Figure 4
and analyzed separately for composition and structure embeddings. Each scatter
point denotes performance for one queried material. Ideally, candidates should
have high precision$@$15% and nDCG (top right corner). Using composition
embeddings, Mat2Vec and MatBERT gave similar performance on both
precision$@$15% and nDCG, indicating the effectiveness of Mat2Vec word
embedding in capturing latent material science knowledge. For structure
embeddings, however, MatBERT recalls significantly more relevant materials
than using fingerprint. This performance is nor surprising, since fingerprints
only contain information about local structure at the motif level, but lack
information at the semi-local level (i.e. motif connectivity) and global level
(e.g. space group). From both composition and structure MatBERT embeddings, a
considerable number of materials achieved precision$>$0.25 and nDCG$>$0.7,
suggesting that the representations extracted similarity preserving signals
which could be utilized for unsupervised search for similarly performing
materials.

Figure 4: Evaluation metric of candidate generation stage for 116K materials.
Each square represents a different query material. The X and Y-axis represent
precision at 15% and nDCG, which measure the accuracy and relative ranking
order respectively. Evaluations are performed for composition embedding and
structure embedding.

###  2.4 Multi-task learning exploits cross-task correlations for improved
property predictions.

For a more accurate candidate material ranking, in the second stage of the
funnel approach of Figure 1 we improved multi-property predictions through
multi-task learning. Learning from multiple related tasks provides superior
performance over single-task learning by modeling task-specific objectives and
cross-task relationships (Ma et al. (2018); Caruana (1997)). Multi-task
learning is thus ideal to learn the underlying commonalities across different
yet correlated material properties, improving performance for each task.
Sanyal et al. (2018) showed that joint-training on several material properties
leads to better model performance in prediction tasks. A mixture-of-experts
framework (Chang et al. (2022)) demonstrates transferability between models
trained on different material properties, thereby improving task performance.

To this aim, we introduce multi-task learning with the MMoE model, which
contains a set of expert networks and gating networks (Figure 5a). Through
task-specific tower networks, the gating network for each property prediction
allows the model to learn mixture contributions from different experts, thus
exploiting the interconnections between tasks (Section 4.4). In the approach
adopted here, the input representations for MMoE models, discussed later in
this section, are concatenated composition and structure embeddings, as well
as context features for growth conditions (Figure 5a). We first benchmarked
MMoE with single-task prediction to predict the six properties shown in Table
1. As shown in Figure 5b, the MMoE results are within error of the single-task
results, but show modest improvement by around 5-10% for most cases. MMoE does
show notably better model stability, indicated by lower variance in cross-
validation performance. The complete single-task and MMoE performance can be
found in Supplementary Figure S2, S3.

Figure 5: a, Schematic of the MMoE model. The gating networks learn
contributions from different experts through the task-specific tower networks.
b, Comparison of model performance for 6 material properties prediction
between single-task models and MMoE using composition or structure embeddings.

Next, we purposed MMoE for multi-task learning of thermoelectric properties.
The efficiency of TE energy conversion is given by figure of merit ($zT$) as:
$zT$=$S^{2}\sigma T$/$\kappa$, where $S$ is Seebeck coefficient, $\sigma$ is
electrical conductivity, $\kappa$ is thermal conductivity, and $T$ is the
temperature. A high $zT$ indicates a good thermoelectric, however, the
properties that lead to high $zT$ are inter-dependent and often conflicting
(Snyder and Toberer (2008)). For example, thermal conductivity increases with
electrical conductivity as carrier concentrations approach the degenerate
regime. Optimizing for TE performance is thus a challenging task that requires
a balance of several properties. For this reason, we speculate that multi-task
learning can naturally leverage the TE task correlations for better model
performance. We found moderate Pearson correlation ranging from 0.15 to 0.5
between the five TE properties considered here (Supplementary Figure S4),
which is considered ideal for multi-task learning. Interestingly, we found
that multi-task learning significantly enhances the predictive performance of
Seebeck coefficient by 71% compared with single-task prediction, with close
performance for the other four tasks within variance from cross-validation
(Supplementary Figure S5).

Table 2: MAEs and R2-scores of four representation methods as the input for MatBERT model to predict the thermoelectric properties on the UCSB and ESTM dataset. Note that for all representations, context features for temperature are also included. (Power factor – PF (S2/m2), Seebeck coefficient – $S$ ($\mu$V/K), Electrical conductivity – $\sigma$ (S/m), Thermal conductivity – $\kappa$ (W/mK), figure-of-merit – $zT$) Property |  Evaluation |  Host |  Host |  Doped |  Host structure$+$
---|---|---|---|---|---
|  |  composition |  structure |  composition |  Doped composition
log_PF |  MAE |  0.580$\pm$0.035 |  0.566$\pm$0.045 |  0.471$\pm$0.039 |  0.433$\pm$0.024
|  R2 |  0.584$\pm$0.080 |  0.624$\pm$0.095 |  0.740$\pm$0.060 |  0.778$\pm$0.063
$S$ |  MAE |  52.3$\pm$6.1 |  53.3$\pm$8.0 |  36.8$\pm$5.1 |  35.4$\pm$3.4
|  R2 |  0.741$\pm$0.069 |  0.753$\pm$0.032 |  0.862$\pm$0.070 |  0.872$\pm$0.046
log_$\sigma$ |  MAE |  1.151$\pm$0.076 |  1.157$\pm$0.063 |  0.693$\pm$0.040 |  0.654$\pm$0.074
|  R2 |  0.576$\pm$0.080 |  0.585$\pm$0.076 |  0.813$\pm$0.036 |  0.832$\pm$0.044
log_$\kappa$ |  MAE |  0.270$\pm$0.020 |  0.272$\pm$0.029 |  0.237$\pm$0.014 |  0.221$\pm$0.022
|  R2 |  0.779$\pm$0.051 |  0.772$\pm$0.049 |  0.824$\pm$0.018 |  0.841$\pm$0.025
$zT$ |  MAE |  0.098$\pm$0.009 |  0.099$\pm$0.009 |  0.094$\pm$0.007 |  0.088$\pm$0.010
|  R2 |  0.678$\pm$0.068 |  0.668$\pm$0.055 |  0.708$\pm$0.034 |  0.741$\pm$0.065
Figure 6: Multi-task prediction results of five TE properties from the best
performing MMoE model with 5-fold cross validation on the experimental
dataset. Composition embeddings, structure embeddings, and context features
are concatenated as the model input. Color indicates density.

The accuracy of the property predictions is rooted in the quality of the data
representations. In addition to the embeddings derived from language models,
we added further information based on context features as model input.
Materials science optimization techniques including doping/defect-engineering
(Toriyama et al. (2021)), alloying and phase-boundary mapping (Ohno et al.
(2018); Ortiz et al. (2019)) are widely utilized and critical to enhance the
performance of TE materials. The composition of a material after optimization
(e.g. doping) is different from the original composition of the host material
via the introduction of dopants and other defects. A small degree of doping
can substantially affect TE performance. For example, in the experimental
database the reported $zT$ of PbTe can vary from as low as 0.10 to as high as
1.56 depending on doping/alloying strategy according to the UCSB
dataset(Gaultois et al. (2013)). Moreover, material properties resulting from
different synthesis conditions (especially temperature) can vary
substantially.

For these reasons, we devised different material representations that can
include up to three components: (i) host material structural features, (ii)
composition features accounting for doping, phase boundary mapping, and
alloying (on normalized cheical formulae), and (iii) context features (one-hot
encoded temperature). For property prediction, structure embeddings carry
important information regarding host material structure, while composition
embeddings bring in information about off-stoichiometry. We summarize how
different input material representations affect MMoE performance for TE tasks
in Table 2. With context features being included, the best performing multi-
task TE model was achieved by concatenating doped composition, host structure,
embeddings (Table 2), whereas host composition embeddings alone gave the worst
performance metrics. While doping and alloying often present significant
challenges for first-principles modeling, the language representation accounts
for such material complexity naturally, through the contextual knowledge
contained in the embedding. In general, models trained with both structure and
composition representations perform consistently better than those with only
composition embeddings. Therefore, modeling TE properties requires accurate
representations of both structural and doped compositions, which can be
effectively extracted through BERT-based language models. The multi-task
learning results from our best-performing material representation and MMoE is
shown in Figure 6. In all five prediction tasks, MMoE accurately predicts the
TE properties for the input material under each one-hot encoded temperature
category with $R^{2}>0.7$. Despite being trained directly on general
representations of crystals, this model achieves comparable accuracy to recent
domain-specific models in the TE field (Na et al. (2021); Na and Chang
(2022)).

###  2.5 Search ranking of TE materials with similar potential

Figure 7: Ranking results of top 15 materials that exhibit most similar TE
potential to a, CoSb3, b, CuGaTe2, and c, Mg3Sb2. The color of each data point
denotes the structure prototype as shown on the right panel for each query
material. The recommended structure prototypes share similar structural
features with the query material.

To interpret and evaluate the ranking performance, we demonstrated the ranking
outcomes from our recommendation framework on seven representative TE
materials. Candidates were ranked by their relevancy score (Section 4.5),
which is defined as the reciprocal of the summed absolute percent difference
of five properties from the query material. Figure 7 shows the ranking results
for CoSb3, CuGaTe2, and Mg3Sb2, representing skutterudite, diamond-like
semiconductors (DLS), and Zintl phases. For each query material, the top 15
ranked materials that exhibit the most similar TE potential are shown. A full
list of the search ranking results for the other materials (PbTe, BiCuSeO,
Cu2Se, Bi2Te3) can be found in Supplementary Figure S6.

In Figure 7, each candidate is colored by its structure prototype to visualize
the structural diversity. The distribution of prototype structures is shown by
the pie chart. For skutterudite CoSb3, the top 15 recommendations consist of 5
prototype structures, and 9 out of the 15 top are different from the space
group of CoSb3 (I$\bar{m}$3, No. 204). As expected, several AX3 skutterudites
(grey in Figure 7a) appear in the list, sharing the same prototype structure
with CoSb3. Several novel structure prototypes also appear, including pyrite
(P$\bar{a}$3, No. 205, red in Figure 7a) and marcasite structures (Pnnm, No.
58, green in Figure 7a), as close relevant TE materials to CoSb3. Moreover,
two more prototype structures – covellite (P63/mmc, No. 194, orange) and
carrollite-like AX2 structure (F$\bar{d}$3m, No. 227, blue) are also
recommended, both of which have received limited attention historically but
may warrant further investigation (Mukherjee et al. (2019)). Note that all
above structure prototypes have corner-sharing octahedral motifs, a local
structural feature shared with query material CoSb3 that may correlate to
similar TE properties. The recommendations based on querying of diamond-like
chalcopyrite material CuGaTe2 (I$\bar{4}$2d, No. 122, grey in Figure 7b)
render diversified outcomes with 5 different structure prototypes. In addition
to four more ABX2 chalcopyrites, the framework selected quaternary stannite
(I$\bar{4}$2m, No. 121, orange in Figure 7b), sulvanite (P$\bar{4}$3m, No.
215, green in Figure 7b), defect kesterite ($\bar{I}$4, No. 79, blue in Figure
7b), and chalcopyrite-like (P$\bar{4}$m2, No. 115, red in Figure 7b)
structures. For Zintl phase Mg3Sb2, the top 15 recommendations comprise 8
unique prototype structures (5 of which are shown in Figure 7c).
Interestingly, the prototypes do not exhibit the layered structure of query
material Mg3Sb2. Instead, the common local structural feature of octahedral
motifs is present throughout the recommended prototypes. Unlike other
computational materials discovery strategies which generate candidate
materials by applying chemical substitutions to a single prototype structure
(Wang et al. (2021b); Qu et al. (2020)), our framework is able to suggest
candidates with diversified structures that are different from, but still
related to, the prototype. Such capability can offer insights and
understanding of structural similarity between different prototypes and
structure-to-property mappings for ML tasks.

To evaluate the performance of the ranking tasks, we performed first-
principles calculations on the TE properties of top recommended candidates
(see computational details in 4.6.2). As shown in Figure S10, the calculated
properties of the recommended materials resemble those of the query material.
For example, both CuGaTe2 and its top ranked candidates exhibit high $p$-type
TE performance that outperforms the $n$-type counterparts. Upon experimental
evaluation of several top ranked candidates, we identified CuZn2GaTe4 as a
novel $p$-type TE material with high Seebeck coefficient ($S=250\mu$V/K at 575
K) and overall decent TE properties, see Figure S11. This immediate positive
result arose from self-doping that yielded an optical carrier concentration
near 4.5$\times 10^{19}$ cm-3 at 473 K. Preliminary experimental measurements
on other candidates, while not demonstrating good performance immediately,
revealed individual features that are beneficial to TEs and the potential to
achieve good performance upon further optimization. The most important
features are, e.g., strongly suppressed thermal conductivities at room
temperature of HgGa2Te4 (0.36 W/mK) and CuGaGeSe4 (0.62 W/mK). Cu2ZnGeTe4 also
belongs to the group of CuGaTe2-like materials and was previously reported to
show decent mobility (ca. 30 cm2/Vs at 550 K, Ortiz et al. (2018)). Lastly,
TiS2 suggested as relevant to CoSb3, has previously been reported with large
Seebeck and $zT$ = 0.3 at 700 K (Bourges et al. (2016)).

Figure 8: a, Zoomed-in region of UMAP projected embedding space as shown by
the dashed line in Figure 3, colored by predicted $zT$ from the MMoE model. b,
Groups of materials that exhibit high predicted $zT$ are colored in the UMAP.
c, Distributions of the predicted $zT$ from each material group in b. d, Top
10 candidates ranked by their predicted $zT$ from each material group.

###  2.6 Exploration of under-studied materials in the representation space.

Through material language representations, we noticed that the distributions
of materials for both known and predicted high $zT$ appear within the same
“band”-like region in the UMAP (Figure 3d). Despite their high predicted $zT$,
materials at the bottom right corner of the “band” (shown by the dashed grey
box in Figure 3) are under-explored with no records from the experimental
datasets (Figure 8a). In that region, we have identified high-$zT$ clusters
composed of halide perovskites (fluoride, chloride, bromide, and iodide),
oxide perovskites, spinel-like, and delafossite-like structures, as labelled
in Figure 8b. The distributions of predicted $zT$ for all materials in each
cluster are visualized in Figure 8c. Among halide perovskites, bromides have
the highest predicted $zT$ with a mean above 1.0, while fluorides, chlorides,
and iodides are close to each other in the predicted $zT$ distributions. The
top 10 highest $zT$ candidates from bromide and fluoride perovskite clusters
(Figure 8d) are mostly Cs- and K-containing double pervoskites
A2BB${}^{{}^{\prime}}$X6, with a few single perovskites ABX3. The high TE
performance of halide perovskites can likely be attributed to low thermal
conductivity. Lee et al. (2017) revealed that inorganic halide perovskites
exhibit ultra-low thermal conductivity due to a unique cluster rattling
mechanism, resulting in thermal conductivities comparable to the amorphous
limit. One of the top predicted candidates CsPbBr3, in particular, has
attracted wide attention in the TE field (Yan et al. (2020)). Recent first-
principles calculations (Mahmood et al. (2022); Saeed et al. (2022)) support
our findings of high TE performance for several double perovskites. It is
worth noting that all of the recommended candidates are lead-free and have
high-temperature stability (Gao et al. (2021a)), good oxidation resistance,
and lower processing costs. With recent experimental advances in improving the
stability of perovskites (Niu et al. (2015); Tiep et al. (2016)), halide
perovskites may become more appealing new TE candidates. Oxide perovksites,
interestingly, show inferior TE performance to halide counterparts. This
observation aligns with chemical intuition, since oxides are in general more
ionic and insulating materials, rendering them hard to dope to optimize the TE
power factor.

Delafossite-like and spinel-like structures are also under-explored structural
spaces with potential to host TE candidates. Unlike perovskites which form
isolated clusters in the representation space, these two material groups
neighbor the more well-explored chalcopyrites and AB2X2 Zintl phases.
Delafossite-like structures, also known as caswellsivlerites, refer to ABX2 (X
= O, S, Se, Te) materials crystallizing in the trigonal structure, with CuFeO2
(R$\bar{3}$m, No.166) being the prototype. So far, extensive research efforts
have focused on delafossite-type oxides as TE materials (Hayashi et al.
(2008); Van Hoang et al. (2020)), while a recent high-throughput computational
study (Shi et al. (2017)) revealed that sulfide, selenide and telluride
delafossite-like structures are also thermodynamically stable. For the top 10
predicted $zT$ candidates from the delafossite group (Figure 8d), all
candidates are sulfides and selenides that we recommend for further
investigation. Similarly, less attention has been focused on sulfide,
selenide, and telluride spinels compared to oxide counterparts, while all top
10 spinels from the recommended list (Figure 8d) are sulfides and selenides.
Recent theoretical works have suggested several high-performance spinel
sulfides Tm2MgS4 ($zT\sim$0.8, Nazar et al. (2022)), Y2CdS4 ($zT\sim$0.8,
Yakhou et al. (2019)), MgIn2Se4 ($zT\sim$0.7, Mahmood et al. (2019)),
suggesting that the discovery framework is able to select good candidates from
a large and diverse search space. As evaluation of the recommended under-
explored material groups, we performed first-principles calculations on the
top candidates from each group in Figure 8d, which are summarized in Figure
S10. These calculations corroborated the promising TE potential of several
candidates, e.g. delafossite-like CdPbS2 ($n$-type, $zT_{\mathrm{max}}$=1.7 at
800 K), halide perovskite Cs2InSbF6 ($n$-type, $zT_{\mathrm{max}}$=1.0 at 800
K), etc.

##  3 Discussion

While representation learning has facilitated extraction of more meaningful
features from large unlabeled data, methods for learning material
representations have also gained substantial momentum (Xu et al. (2021); Gupta
et al. (2021); Na and Kim (2022)). On the other hand, language-based models
have achieved remarkable outcomes in prediction and generation tasks across an
extensive array of domain areas. In this work, we demonstrated the use of
language representations in the inorganic crystalline materials domain.
Specifically, we introduced a language-based framework to extract composition
and structure embeddings as material representations via pretrained language
models. The discovery framework is designed to be task-agnostic. We anticipate
that it can be expanded upon and utilized to search and explore vast chemical
and structural spaces, towards functional materials design and discovery.

Representing materials in the format of natural language enables effective
utilization of material science knowledge learnt from ever-growing
unstructured scientific texts. Indeed, the extracted embeddings form a
chemically meaningful representation space without task-specific supervision.
We find that knowledge can be extracted from representations by unsupervised
recall on embedding vectors and supervised neural networks, together enabling
the funnel-based approach. In particular, the recall step allows reliable
recommendation by constraining the ranking on candidates that are similar to
the query material in the representation space. A benefit of such pre-
screening is the avoidance of common pitfalls where materials exhibit similar
properties that arise for inherently different reasons, i.e. far from each
other in the representation space. For the use case of thermoelectrics, for
example, high $zT$ can arise from either high power factor or low thermal
conductivity. Another strength of language representations is that they can
effectively handle off-stoichiometric material compositions to account for
alloying and doping, which typically require complicated computational
techniques (e.g. disorder modelling) for accurate predictions in first-
principles simulations.

Exploitation and exploration trade-off has been a common phenomenon in
recommender systems (Gao et al. (2021b); Vanchinathan et al. (2014)). For our
recommendation framework, while exploitation refers to seeking maximum reward,
exploration may be thought of as consideration of new structural prototypes
present in the top-ranked candidates that share structural features with, but
are distinct from, the query material. A reasonable balance between
exploitation and exploration, which can be tuned by the number of candidates
recalled from the candidate generation step, will diversify the recommendation
while still proposing structurally-related materials. For example, the top-15
ranked materials for both CoSb3 and CuGaTe2 contain 5 different prototype
structures when 100 recalled materials from the candidate generation step are
considered for ranking, while the number of prototypes increases to 14 and 9
respectively if the number of recalled materials considered is increased to
1000 (Supplementary Figure S7).

As future directions for language representation for crystals, we suggest to
enrich the material representations by diversifying both text-based input and
structures. The automatically generated text descriptions from
Robocrystallographer are monotonic with little variation between descriptive
words/phrasing (Sayeed et al. (2023)). These descriptions can possibly be
diversified via paraphrasing or developing structure to sentence machine
translation models to describe crystal structures in text. On the other hand,
the structural complexity in the representation space can be diversified via
generative models, e.g., diffusion models (Xie et al. (2021); Lyngby and
Thygesen (2022)), to design new prototype structures beyond simple lattice
decoration of known crystals.

##  4 Methods

###  4.1 Data preparation

The training dataset was collected from the Materials Project (Jain et al.
(2013)) to include 116,216 materials are likely to be thermodynamically
stable. Using decomposition enthalpy < 0.5 eV as a query criteria, we utilized
Materials Project API (Ong et al. (2015)) and Pymatgen (Ong et al. (2013))
library to collect materials for use in this study.

In this work we considered five different datasets, all of which includes
properties relevant to thermoelectric materials; UCSB dataset (Gaultois et al.
(2013)) – an experimental dataset from Materials Research Laboratory (MRL)
about 1092 materials (500 unique materials) with their thermoelectric
properties; ESTM dataset (Na and Chang (2022)) – an experimental dataset
containing 5205 materials (880 unique materials) with their thermoelectric
properties; ChemExtracter dataset (Sierepeklis and Cole (2022)) – a mixture of
experimental and theory dataset by auto-generation from the scientific
literature spanning 10,641 unique chemical names; TEDesignLab dataset (Gorai
et al. (2016)) – a theory dataset containing lattice thermal conductivity for
3278 materials; Citrine dataset (Ward et al. (2018)) – an experimental dataset
from Matminer (Ward et al. (2018)) containing thermal conductivity records for
871 materials. In all five datasets, 826 materials that have records for five
TE properties are used for evaluation of recall performance in Section 4.5. We
calculated the numeric mean for materials with repeated entry for certain
properties and properties at different temperatures. For MMoE model training
and testing, UCSB and ESTM dataset are utilized as ground-truth labels .
During training, the TE properties are matched to corresponding temperature
range via one-hot encoding.

###  4.2 Embedding models

Three model-based and one model-free embedding methods were used in this work.
For the model-based approach, we obtained pretrained weights for Mat2Vec
(Tshitoyan et al. (2019)), MatsciBERT (Gupta et al. (2022)), and MatBERT
(Trewartha et al. (2022)). For Mat2Vec, it was trained similarly as Word2vec
training through skip-gram with negative sampling. Each word is embedded into
a 200-dimensional vector. For the BERT-based models, MatsciBERT was pretrained
on whole sections of more than 1 million material science articles, whereas
MatBERT was trained by sampling 50 million paragraphs from 2 million articles.
Both models were trained with masked language modeling (15% dynamic whole word
masking) and next-sentence prediction as the unsupervised training objectives.
Both models are uncased, and have maximum 512 input token size with 768 hidden
dimensions. The vocabulary size for the tokenizer is 30,522. For the
fingerprint generation, it was generated using CrystalNN (Zimmermann and Jain
(2020)) algorithm as implemented in Matminer (Ward et al. (2018)) package. The
fingerprint contains statistical information about local motifs with a size
dimension of 122.

###  4.3 Material language representations

We acquired compositional and structural level representations for 116K
materials in total. To acquire structural level representations for each
individual material, we applied robocrystallographer (Ganose and Jain (2019)),
an open-source toolkit that converts the material structure into a human-
readable text passage describing local, semi-local and global structural
features of the given material. We used robocrystallographer descriptions from
Sayeed et al. (2023). Similar to material descriptions found in literature,
such material passage encodes naturally interpretable structural information.
The whole passage is processed by tokenizers and fed into the pretrained BERT
models (MatsciBERT and MatBERT) for output embeddings from hidden layers. The
output embeddings are $L$ by 768 dimensional matrix, where $L\in$(0,512] is
the total number of tokens within the passage. We partitioned passages with
more than 512 tokens to fit the maximum input token size. The final embeddings
for each material are constructed by averaging output embeddings across all
tokens, resulting in a fixed length of vector representations with 768
dimensions.

For the compositional level representations, Mat2Vec embeddings are directly
obtained as the 200-dimensional word embedding vectors of the material
formulas. With BERT models, we performed same tokenization and embedding
procedures on material formulas only. This results in the same number of
768-dimensional embedding vectors but only contains information related to the
material composition. For composition embeddings of the doped material
formulas (UCSB dataset), we normalized the compositions to the element with
the most number of atoms in the unit cell. The output embeddings are obtained
on the normalized formulas.

###  4.4 MMoE and TE property prediction

A shared-bottom multi-task network was first introduced by Caruana (1997) and
widely applied for multi-task learning. The basic network formulation is:

$y_{k}=h^{k}(f(x))$ (1)

where $k=1,2,3...K$ for $K$ number of tasks, $f$ is the shared-bottom network,
$h^{k}$ is the tower network for task $k$, and $y_{k}$ is the output for task
$k$. The key difference in MMoE network is to substitute the shared-bottom $f$
with MoE layer $f^{k}(x)$ for a specific task $k$, which is defined as:

$f^{k}(x)=\Sigma^{n}_{i=1}g^{k}(x)_{i}f_{i}(x)$ (2)
$g^{k}(x)=\mathrm{softmax}(W_{gk}x)$ (3)

where $i$ = 1, 2, 3… $n$ for $n$ number of experts, $g^{k}(x)$ is the gating
network for each task $k$, and $W_{gk}$ is the trainable matrix. In our
implementation, all expert network is a three-layered MLP with 128, 64, and 32
dimensions. The gating network is a two-layered MLP with 32 and 16 dimensions.
In all of our experiments, networks are trained for 500 epochs with learning
rate = $10^{-3}$, weight decay = $10^{-5}$, and batch size=64. We used k-fold
cross-validation method to train and evaluate the model performance. For all
datasets, we employed 5-fold cross validation by splitting the dataset into 5
nonoverlapping portions. The number of experts is set to 8 for both AFLOW
benchmark dataset and TE dataset.

Recent works (Na and Chang (2022); Na et al. (2021)) reported that doping and
alloying information, as well as context features greatly enhance the model
performance for TE predictions. As for context features for MMoE, we first
sorted the continuous temperatures into four ranges (0, 300], (300, 600],
(600, 900], (900, $+\infty$], which were one-hot encoded into sparse feature
vectors and passed to embedding layers of the MMoE model. Since our structure
embeddings are restricted to the host materials, dopant or alloying
information will be derived from composition embeddings to delineate the
compositional effect. To match doped materials to their hosts, we encoded the
normalized doped formulas into composition vectors (sparse vector with number
of corresponding elements at each site), followed by mapping to existing host
composition vectors via cosine similarity. Host materials with the highest
cosine similarities were selected.

###  4.5 Ranking score and exploratory analysis

Once candidates are recalled for the query, their predicted properties are
used to compute total absolute percent difference (TAPD) defined as:

$\mathrm{TAPD}=\Sigma^{K}_{k=1}\left(\frac{|y_{k}^{c}-y_{k}^{q}|}{y_{k}^{q}}\right)$
(4)

where $K$ is the total number of material properties, $y^{c}$ and $y^{q}$ are
the candidate and query properties respectively. This measures the composite
deviation of candidate properties from the query properties. All properties
need to be close to those of the query to have a low TAPD. We define relevancy
score as the reciprocal of TAPD:

$\textrm{relevancy}=\frac{1}{\mathrm{TAPD}}$ (5)

In our experiments, 100 candidates were recalled per query material. We ranked
the candidates based on their relevancy score. The scores presented in the
figure were normalized by the maximum score within the recalled list. For the
exploratory analysis, clusters were hand-selected based on localization of
materials with high predicted $zT$. Within each selected cluster, we extracted
and ranked the materials according to their $zT$. All predictions were made at
high temperature (900, $+\infty$] as the context features.

###  4.6 Evaluation

####  4.6.1 Unsupervised recall of relevant materials

Recalling relevant material candidates is an unsupervised process which does
not require training labels. First, candidates are searched in the
representation space by computing cosine similarities between the embedding
vector of the query and the rest of the embedding vectors. The similarity-
sorted top candidates are returned as the relevant materials. Metrics
including Precision$@k$ and Normalized Discounted Cumulative Gain (nDCG) are
used to evaluate the recall performance. Such evaluation metrics are common
for recommender system, where the goal is to maximize the number of relevant
items in the recalled list, i.e., the top$@$k items with $k$ being the size of
the list, as well as the relative order of recalled items. Precision$@k$
measures the percentage of the relevant materials in the first $k$ recalled
materials:

$\textrm{precision}@k=\frac{\textrm{relevant items}@k}{k}$ (6)

while nDCG is an evaluation method which compares the ideal ranking of a test
set (iDCG), with the ranking assigned by the recommendation algorithm (DCG –
Equ.7).

$\textrm{DCG}=\Sigma^{n}_{i=1}\frac{\textrm{relevance}}{\log_{2}(i+1)}$ (7)
$\textrm{nDCG}=\frac{\textrm{DCG}}{\textrm{iDCG}}$ (8)

####  4.6.2 First-principles calculations

The ab initio scattering and transport (AMSET, Ganose et al. (2021)) software
package was used to estimate scattering rates (or lifetime) and transport
properties based on momentum relaxation time approximation (MRTA), which has
been shown to give comparable results to state-of-art EPW code (Poncé et al.
(2016)). The carrier mobility was simulated by considering three scattering
processes, including acoustic phonon scattering (ADP), polar optical phonon
scattering (POP), and ionized impurity scattering (IMP). Each component of
carrier lifetime was evaluated by Fermi’s golden rule, with total
characteristic scattering time following Matthiessen’s rule. The associated
Seebeck coefficient, electrical conductivity, and electronic component of the
thermal conductivity were were calculated by solving the Boltzmann transport
equation (BoltzTraP) using Onsager transport coefficients. All ab initio
inputs are computed from density functional theory (DFT) using the GGA-PBE
(Perdew et al. (1996)) exchange-correlation functional. Lattice thermal
conductivity ($\kappa_{\mathrm{L}}$) was calculated using a semi-empirical
model based on a modified Debye-Callaway model (Miller et al. (2017)) which
captures anharmonicity. Bulk modulus ($B$) was determined by fitting the
Birch-Murnaghan equation of state to a set of total energies computed at
different volumes that were expanded and contracted around the equilibrium
volume. Other parameters of the semi-empirical model are directly accessible
from the relaxed structures, including density, average atomic mass, volume
per atom, average coordination number, and number of atoms in the primitive
cell. The expression for lattice thermal conductivity is give by

$\kappa_{\mathrm{L,ac}}=A_{2}\frac{\bar{M}v_{\mathrm{s}}^{y}}{TV^{2/3}n^{1/3}}+A_{3}\frac{v_{\mathrm{s}}}{V^{2/3}}(1-\frac{1}{n^{2/3}})\hskip
5.0pt,$ (9)

where $A_{1}$ and $A_{2}$ are fitted parameters, $\bar{M}$ is the average
atomic mass, $v_{\mathrm{s}}$ is the speed of sound, $T$ is the temperature,
$V$ is the volume per atom, and $n$ is the number of atoms in the primitive
cell. $v_{\mathrm{s}}$ is approximated as $v_{\mathrm{s}}$ $\sim$
$(B/d)^{\frac{1}{2}}$.

####  4.6.3 Experiments

CuZn2GaTe4, CuGaGeSe4, and HgGa2Te4 samples were prepared from elements: Cu
(99.9%), Hg (99.999%), Ga (99.999%), Zn (99.999%), Ge (99.999%), and Te
(99.999%), Se (99.999%). The stoichiometric weights were first sealed in
evacuated silica ampoules and melted at 1000oC for several hours. Next, the
ingots were milled in high-energy mechanical mill Spex 8000D for 90 min in an
inert environment. The powders were consolidated in an induction heating hot
press at 500o C, 40 MPa for at least 2 hours. Electrical resistivity and Hall
coefficient were studied under vacuum on a home-built apparatus with van-der
Pauw geometry (Borup et al. (2012)). Seebeck coefficient measurements were
carried out using a custom-built device (Iwanaga et al. (2011)) in 300 Torr of
nitrogen gas. Diffusivity coefficient ($D$) measurements were performed on
Netzsch LFA 467 apparatus. To obtain thermal conductivity ($\kappa$), we used
formula $\kappa$ = $DC_{p}d_{\mathrm{exp}}$, where $C_{p}$ is heat capacity
and $d_{\mathrm{exp}}$ is experimental density. Values of $C_{p}$ were
obtained from Dulong-Petit law, while density of the samples was measured with
geometric method. For all obtained materials $d_{\mathrm{exp}}$ was ca. 90% of
the theoretical value or higher.

## Data Availability

The preprocessed AFLOW and thermoelectric datasets used for training and
testing the models, as well as material embeddings obtained in this work, are
available at https://doi.org/10.6084/m9.figshare.22718668.v1.

## Code Availability

The code and the model weights are available under the MIT license at:
https://github.com/ertekin-research-group/Material_Recommender

## Acknowledgement

This work was funded with support from the U.S. National Science Foundation
(NSF) via Grant No. 2118201 (HDR Institute for Data-Driven Dynamical Design)
and Grant No. 1922758 (DIGI-MAT). This work used the Extreme Science and
Engineering Discovery Environment (XSEDE) Bridges-2 at the Pittsburgh
Supercomputing Center through allocation TG-MAT220011P.

## References

  * Hey et al. [2009] Anthony JG Hey, Stewart Tansley, Kristin Michele Tolle, et al.  _The fourth paradigm: data-intensive scientific discovery_ , volume 1.  Microsoft research Redmond, WA, 2009.
  * Aykol et al. [2020] Muratahan Aykol, Patrick Herring, and Abraham Anapolsky.  Machine learning for continuous innovation in battery technologies.  _Nature Reviews Materials_ , 5(10):725–727, 2020\.
  * Wang et al. [2020] Tian Wang, Cheng Zhang, Hichem Snoussi, and Gang Zhang.  Machine learning approaches for thermoelectric materials research.  _Advanced Functional Materials_ , 30(5):1906041, 2020.
  * Mahmood and Wang [2021] Asif Mahmood and Jin-Liang Wang.  Machine learning for high performance organic solar cells: current scenario and future prospects.  _Energy & environmental science_, 14(1):90–105, 2021.
  * Schmidt et al. [2019] Jonathan Schmidt, Mário RG Marques, Silvana Botti, and Miguel AL Marques.  Recent advances and applications of machine learning in solid-state materials science.  _npj Computational Materials_ , 5(1):83, 2019\.
  * Behler and Parrinello [2007] Jörg Behler and Michele Parrinello.  Generalized neural-network representation of high-dimensional potential-energy surfaces.  _Physical review letters_ , 98(14):146401, 2007\.
  * Isayev et al. [2017] Olexandr Isayev, Corey Oses, Cormac Toher, Eric Gossett, Stefano Curtarolo, and Alexander Tropsha.  Universal fragment descriptors for predicting properties of inorganic crystals.  _Nature communications_ , 8(1):15679, 2017.
  * Xie and Grossman [2018] Tian Xie and Jeffrey C Grossman.  Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties.  _Physical review letters_ , 120(14):145301, 2018\.
  * Chen et al. [2019] Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong.  Graph networks as a universal machine learning framework for molecules and crystals.  _Chemistry of Materials_ , 31(9):3564–3572, 2019\.
  * Choudhary and DeCost [2021] Kamal Choudhary and Brian DeCost.  Atomistic line graph neural network for improved materials property predictions.  _npj Computational Materials_ , 7(1):185, 2021\.
  * Yan et al. [2022] Keqiang Yan, Yi Liu, Yuchao Lin, and Shuiwang Ji.  Periodic graph transformers for crystal material property prediction.  _arXiv preprint arXiv:2209.11807_ , 2022.
  * Jha et al. [2018] Dipendra Jha, Logan Ward, Arindam Paul, Wei-keng Liao, Alok Choudhary, Chris Wolverton, and Ankit Agrawal.  Elemnet: Deep learning the chemistry of materials from only elemental composition.  _Scientific reports_ , 8(1):1–13, 2018.
  * Tshitoyan et al. [2019] Vahe Tshitoyan, John Dagdelen, Leigh Weston, Alexander Dunn, Ziqin Rong, Olga Kononova, Kristin A Persson, Gerbrand Ceder, and Anubhav Jain.  Unsupervised word embeddings capture latent knowledge from materials science literature.  _Nature_ , 571(7763):95–98, 2019.
  * Wang et al. [2021a] Anthony Yu-Tung Wang, Steven K Kauwe, Ryan J Murdock, and Taylor D Sparks.  Compositionally restricted attention-based network for materials property predictions.  _Npj Computational Materials_ , 7(1):77, 2021a.
  * Trewartha et al. [2022] Amalie Trewartha, Nicholas Walker, Haoyan Huo, Sanghoon Lee, Kevin Cruse, John Dagdelen, Alexander Dunn, Kristin A Persson, Gerbrand Ceder, and Anubhav Jain.  Quantifying the advantage of domain-specific pre-training on named entity recognition tasks in materials science.  _Patterns_ , 3(4):100488, 2022.
  * Gupta et al. [2022] Tanishq Gupta, Mohd Zaki, and NM Anoop Krishnan.  Matscibert: A materials domain language model for text mining and information extraction.  _npj Computational Materials_ , 8(1):102, 2022\.
  * Gaultois et al. [2016] Michael W Gaultois, Anton O Oliynyk, Arthur Mar, Taylor D Sparks, Gregory J Mulholland, and Bryce Meredig.  Perspective: Web-based machine learning models for real-time screening of thermoelectric materials properties.  _Apl Materials_ , 4(5):053213, 2016.
  * Seko et al. [2018a] Atsuto Seko, Hiroyuki Hayashi, and Isao Tanaka.  Compositional descriptor-based recommender system for the materials discovery.  _The Journal of chemical physics_ , 148(24):241719, 2018a.
  * Seko et al. [2018b] Atsuto Seko, Hiroyuki Hayashi, Hisashi Kashima, and Isao Tanaka.  Matrix-and tensor-based recommender systems for the discovery of currently unknown inorganic compounds.  _Physical Review Materials_ , 2(1):013805, 2018b.
  * Hayashi et al. [2019] Hiroyuki Hayashi, Katsuyuki Hayashi, Keita Kouzai, Atsuto Seko, and Isao Tanaka.  Recommender system of successful processing conditions for new compounds based on a parallel experimental data set.  _Chemistry of Materials_ , 31(24):9984–9992, 2019\.
  * Covington et al. [2016] Paul Covington, Jay Adams, and Emre Sargin.  Deep neural networks for youtube recommendations.  In _Proceedings of the 10th ACM conference on recommender systems_ , pages 191–198, 2016.
  * Gomez-Uribe and Hunt [2015] Carlos A Gomez-Uribe and Neil Hunt.  The netflix recommender system: Algorithms, business value, and innovation.  _ACM Transactions on Management Information Systems (TMIS)_ , 6(4):1–19, 2015.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Ganose and Jain [2019] Alex M Ganose and Anubhav Jain.  Robocrystallographer: automated crystal structure text descriptions and analysis.  _MRS Communications_ , 9(3):874–881, 2019.
  * McInnes et al. [2018] Leland McInnes, John Healy, and James Melville.  Umap: Uniform manifold approximation and projection for dimension reduction.  _arXiv preprint arXiv:1802.03426_ , 2018.
  * Zimmermann and Jain [2020] Nils ER Zimmermann and Anubhav Jain.  Local structure order parameters and site fingerprints for quantification of coordination environment and crystal structure similarity.  _RSC advances_ , 10(10):6063–6081, 2020.
  * Tang et al. [2018] Jing Tang, Bo Gao, Siqi Lin, Juan Li, Zhiwei Chen, Fen Xiong, Wen Li, Yue Chen, and Yanzhong Pei.  Manipulation of band structure and interstitial defects for improving thermoelectric snte.  _Advanced Functional Materials_ , 28(34):1803586, 2018.
  * Wu et al. [2022] Mengyue Wu, Lujun Zhu, Shixuan Liu, Mingzhen Song, Fudong Zhang, Pengfei Liang, Xiaolian Chao, Zupei Yang, Jiaqing He, and Di Wu.  Significantly enhanced thermoelectric performance achieved in cugate2 through dual-element permutations at cation sites.  _ACS Applied Materials & Interfaces_, 14(26):30046–30055, 2022.
  * Zhang et al. [2017] Jiawei Zhang, Lirong Song, Steffen Hindborg Pedersen, Hao Yin, Le Thanh Hung, and Bo Brummerstedt Iversen.  Discovery of high-performance low-cost n-type mg3sb2-based thermoelectric materials with multi-valley conduction bands.  _Nature communications_ , 8(1):13901, 2017.
  * Ohno et al. [2018] Saneyuki Ohno, Kazuki Imasato, Shashwat Anand, Hiromasa Tamaki, Stephen Dongmin Kang, Prashun Gorai, Hiroki K Sato, Eric S Toberer, Tsutomu Kanno, and G Jeffrey Snyder.  Phase boundary mapping to obtain n-type mg3sb2-based thermoelectrics.  _Joule_ , 2(1):141–154, 2018.
  * Curtarolo et al. [2012] Stefano Curtarolo, Wahyu Setyawan, Gus LW Hart, Michal Jahnatek, Roman V Chepulskii, Richard H Taylor, Shidong Wang, Junkai Xue, Kesong Yang, Ohad Levy, et al.  Aflow: An automatic framework for high-throughput materials discovery.  _Computational Materials Science_ , 58:218–226, 2012.
  * Xiao et al. [2018] Jun Xiao, Minjuan Wang, Bingqian Jiang, and Junli Li.  A personalized recommendation system with combinational algorithm for online learning.  _Journal of ambient intelligence and humanized computing_ , 9:667–677, 2018.
  * Song et al. [2014] Linqi Song, Cem Tekin, and Mihaela Van Der Schaar.  Online learning in large-scale contextual recommender systems.  _IEEE Transactions on Services Computing_ , 9(3):433–445, 2014.
  * Ma et al. [2018] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi.  Modeling task relationships in multi-task learning with multi-gate mixture-of-experts.  In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 1930–1939, 2018.
  * Caruana [1997] Rich Caruana.  Multitask learning.  _Machine learning_ , 28:41–75, 1997.
  * Sanyal et al. [2018] Soumya Sanyal, Janakiraman Balachandran, Naganand Yadati, Abhishek Kumar, Padmini Rajagopalan, Suchismita Sanyal, and Partha Talukdar.  Mt-cgcnn: Integrating crystal graph convolutional neural network with multitask learning for material property prediction.  _arXiv preprint arXiv:1811.05660_ , 2018.
  * Chang et al. [2022] Rees Chang, Yu-Xiong Wang, and Elif Ertekin.  Towards overcoming data scarcity in materials science: unifying models and datasets with a mixture of experts framework.  _npj Computational Materials_ , 8(1):242, 2022\.
  * Snyder and Toberer [2008] G Jeffrey Snyder and Eric S Toberer.  Complex thermoelectric materials.  _Nature materials_ , 7(2):105–114, 2008.
  * Toriyama et al. [2021] Michael Y Toriyama, Jiaxing Qu, G Jeffrey Snyder, and Prashun Gorai.  Defect chemistry and doping of bicuseo.  _Journal of Materials Chemistry A_ , 9(36):20685–20694, 2021.
  * Ortiz et al. [2019] Brenden R Ortiz, Kiarash Gordiz, Lídia C Gomes, Tara Braden, Jesse M Adamczyk, Jiaxing Qu, Elif Ertekin, and Eric S Toberer.  Carrier density control in cu 2 hggete 4 and discovery of hg 2 gete 4 via phase boundary mapping.  _Journal of Materials Chemistry A_ , 7(2):621–631, 2019.
  * Gaultois et al. [2013] Michael W Gaultois, Taylor D Sparks, Christopher KH Borg, Ram Seshadri, William D Bonificio, and David R Clarke.  Data-driven review of thermoelectric materials: performance and resource considerations.  _Chemistry of Materials_ , 25(15):2911–2920, 2013\.
  * Na et al. [2021] Gyoung S Na, Seunghun Jang, and Hyunju Chang.  Predicting thermoelectric properties from chemical formula with explicitly identifying dopant effects.  _npj Computational Materials_ , 7(1):106, 2021\.
  * Na and Chang [2022] Gyoung S Na and Hyunju Chang.  A public database of thermoelectric materials and system-identified material representation for data-driven discovery.  _npj Computational Materials_ , 8(1):214, 2022\.
  * Mukherjee et al. [2019] Bodhoday Mukherjee, Tarachand, S Hussain, and Gunadhor S Okram.  Effect of different surfactants on thermoelectric properties of cus nanoparticles.  In _AIP Conference Proceedings_ , volume 2100, page 020091. AIP Publishing LLC, 2019.
  * Wang et al. [2021b] Hai-Chen Wang, Silvana Botti, and Miguel AL Marques.  Predicting stable crystalline compounds using chemical similarity.  _npj Computational Materials_ , 7(1):12, 2021b.
  * Qu et al. [2020] Jiaxing Qu, Vladan Stevanovic, Elif Ertekin, and Prashun Gorai.  Doping by design: finding new n-type dopable abx 4 zintl phases for thermoelectrics.  _Journal of Materials Chemistry A_ , 8(47):25306–25315, 2020.
  * Ortiz et al. [2018] Brenden R Ortiz, Wanyue Peng, Lídia C Gomes, Prashun Gorai, Taishan Zhu, David M Smiadak, G Jeffrey Snyder, Vladan Stevanovic, Elif Ertekin, Alexandra Zevalkink, et al.  Ultralow thermal conductivity in diamond-like semiconductors: selective scattering of phonons from antisite defects.  _Chemistry of Materials_ , 30(10):3395–3409, 2018\.
  * Bourges et al. [2016] Cedric Bourges, Tristan Barbier, Gabin Guelou, Paz Vaqueiro, Anthony V Powell, Oleg I Lebedev, Nicolas Barrier, Yoshiaki Kinemuchi, and Emmanuel Guilmeau.  Thermoelectric properties of tis2 mechanically alloyed compounds.  _J. Eur. Ceram. Soc._ , 36(5):1183–1189, 2016\.
  * Lee et al. [2017] Woochul Lee, Huashan Li, Andrew B Wong, Dandan Zhang, Minliang Lai, Yi Yu, Qiao Kong, Elbert Lin, Jeffrey J Urban, Jeffrey C Grossman, et al.  Ultralow thermal conductivity in all-inorganic halide perovskites.  _Proceedings of the National Academy of Sciences_ , 114(33):8693–8697, 2017.
  * Yan et al. [2020] Lifu Yan, Mingchao Wang, Chenxi Zhai, Lingling Zhao, and Shangchao Lin.  Symmetry breaking induced anisotropic carrier transport and remarkable thermoelectric performance in mixed halide perovskites cspb (i1–x br x) 3.  _ACS Applied Materials & Interfaces_, 12(36):40453–40464, 2020.
  * Mahmood et al. [2022] Q Mahmood, M Hassan, N Yousaf, Abeer A AlObaid, Tahani I Al-Muhimeed, Manal Morsi, Hind Albalawi, and Osama A Alamri.  Study of lead-free double perovskites halides cs2ticl6, and cs2tibr6 for optoelectronics, and thermoelectric applications.  _Materials Science in Semiconductor Processing_ , 137:106180, 2022.
  * Saeed et al. [2022] Muhammad Saeed, Izaz Ul Haq, Awais Siddique Saleemi, Shafiq Ur Rehman, Bakhtiar Ul Haq, Aijaz Rasool Chaudhry, and Imad Khan.  First-principles prediction of the ground-state crystal structure of double-perovskite halides cs2agcrx6 (x= cl, br, and i).  _Journal of Physics and Chemistry of Solids_ , 160:110302, 2022.
  * Gao et al. [2021a] Zhengyang Gao, Hanwen Zhang, Guangyang Mao, Jianuo Ren, Ziheng Chen, Chongchong Wu, Ian D Gates, Weijie Yang, Xunlei Ding, and Jianxi Yao.  Screening for lead-free inorganic double perovskites with suitable band gaps and high stability using combined machine learning and dft calculation.  _Applied Surface Science_ , 568:150916, 2021a.
  * Niu et al. [2015] Guangda Niu, Xudong Guo, and Liduo Wang.  Review of recent progress in chemical stability of perovskite solar cells.  _Journal of Materials Chemistry A_ , 3(17):8970–8980, 2015.
  * Tiep et al. [2016] Nguyen Huy Tiep, Zhiliang Ku, and Hong Jin Fan.  Recent advances in improving the stability of perovskite solar cells.  _Advanced Energy Materials_ , 6(3):1501420, 2016\.
  * Hayashi et al. [2008] Kei Hayashi, Ken-ichi Sato, Tomohiro Nozaki, and Tsuyoshi Kajitani.  Effect of doping on thermoelectric properties of delafossite-type oxide cucro2.  _Japanese journal of applied physics_ , 47(1R):59, 2008.
  * Van Hoang et al. [2020] Dung Van Hoang, Anh Tuan Thanh Pham, Hanh Kieu Thi Ta, Truong Huu Nguyen, Ngoc Kim Pham, Lai Thi Hoa, Vinh Cao Tran, Ohtaki Michitaka, Quang Minh Nhat Tran, Jong-Ho Park, et al.  Effects of multi-scale defects on the thermoelectric properties of delafossite cucr1-xmgxo2 materials.  _Journal of Alloys and Compounds_ , 844:156119, 2020.
  * Shi et al. [2017] Jingming Shi, Tiago FT Cerqueira, Wenwen Cui, Fernando Nogueira, Silvana Botti, and Miguel AL Marques.  High-throughput search of ternary chalcogenides for p-type transparent electrodes.  _Scientific reports_ , 7(1):43179, 2017.
  * Nazar et al. [2022] Mubashar Nazar, Shatha A Aldaghfag, Muhammad Yaseen, Mudassir Ishfaq, Rasheed Ahmad Khera, Saima Noreen, Magda H Abdellattif, et al.  First-principles calculations to investigate structural, magnetic, optical, electronic and thermoelectric properties of x2mgs4 (x= gd, tm) spinel sulfides.  _Journal of Physics and Chemistry of Solids_ , 166:110719, 2022.
  * Yakhou et al. [2019] H Yakhou, A Maachou, H Riane, and M Sahnoun.  Theoretical investigation of electronic and thermoelectric properties of spinel sulfides a2bs4 (a= sc and y; b= cd and zn).  _Computational Condensed Matter_ , 21:e00417, 2019.
  * Mahmood et al. [2019] Q Mahmood, M Rashid, NA Noor, M Gul Bahar Ashiq, Shahid M Ramay, Asif Mahmood, et al.  Opto-electronic and thermoelectric properties of mgin2x4 (x= s, se) spinels via ab-initio calculations.  _Journal of Molecular Graphics and Modelling_ , 88:168–173, 2019.
  * Xu et al. [2021] Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang.  Self-supervised graph-level representation learning with local and global structure.  In _International Conference on Machine Learning_ , pages 11548–11558. PMLR, 2021.
  * Gupta et al. [2021] Vishu Gupta, Kamal Choudhary, Francesca Tavazza, Carelyn Campbell, Wei-keng Liao, Alok Choudhary, and Ankit Agrawal.  Cross-property deep transfer learning framework for enhanced predictive analytics on small materials data.  _Nature communications_ , 12(1):6595, 2021.
  * Na and Kim [2022] Gyoung S Na and Hyun Woo Kim.  Contrastive representation learning of inorganic materials to overcome lack of training datasets.  _Chemical Communications_ , 58(47):6729–6732, 2022.
  * Gao et al. [2021b] Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua.  Advances and challenges in conversational recommender systems: A survey.  _AI Open_ , 2:100–126, 2021b.
  * Vanchinathan et al. [2014] Hastagiri P Vanchinathan, Isidor Nikolic, Fabio De Bona, and Andreas Krause.  Explore-exploit in top-n recommender systems via gaussian processes.  In _Proceedings of the 8th ACM Conference on Recommender systems_ , pages 225–232, 2014.
  * Sayeed et al. [2023] Hasan M Sayeed, Sterling G Baird, and Taylor D Sparks.  Structure feature vectors derived from robocrystallographer text descriptions of crystal structures using word embeddings.  2023\.
  * Xie et al. [2021] Tian Xie, Xiang Fu, Octavian-Eugen Ganea, Regina Barzilay, and Tommi Jaakkola.  Crystal diffusion variational autoencoder for periodic material generation.  _arXiv preprint arXiv:2110.06197_ , 2021.
  * Lyngby and Thygesen [2022] Peder Lyngby and Kristian Sommer Thygesen.  Data-driven discovery of 2d materials by deep generative models.  _npj Computational Materials_ , 8(1):232, 2022\.
  * Jain et al. [2013] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al.  Commentary: The materials project: A materials genome approach to accelerating materials innovation.  _APL materials_ , 1(1):011002, 2013.
  * Ong et al. [2015] Shyue Ping Ong, Shreyas Cholia, Anubhav Jain, Miriam Brafman, Dan Gunter, Gerbrand Ceder, and Kristin A Persson.  The materials application programming interface (api): A simple, flexible and efficient api for materials data based on representational state transfer (rest) principles.  _Computational Materials Science_ , 97:209–215, 2015.
  * Ong et al. [2013] Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L Chevrier, Kristin A Persson, and Gerbrand Ceder.  Python materials genomics (pymatgen): A robust, open-source python library for materials analysis.  _Computational Materials Science_ , 68:314–319, 2013.
  * Sierepeklis and Cole [2022] Odysseas Sierepeklis and Jacqueline M Cole.  A thermoelectric materials database auto-generated from the scientific literature using chemdataextractor.  _Scientific Data_ , 9(1):648, 2022.
  * Gorai et al. [2016] Prashun Gorai, Duanfeng Gao, Brenden Ortiz, Sam Miller, Scott A Barnett, Thomas Mason, Qin Lv, Vladan Stevanovic, and Eric S Toberer.  Te design lab: A virtual laboratory for thermoelectric material design.  _Computational Materials Science_ , 112:368–376, 2016.
  * Ward et al. [2018] Logan Ward, Alexander Dunn, Alireza Faghaninia, Nils ER Zimmermann, Saurabh Bajaj, Qi Wang, Joseph Montoya, Jiming Chen, Kyle Bystrom, Maxwell Dylla, et al.  Matminer: An open source toolkit for materials data mining.  _Computational Materials Science_ , 152:60–69, 2018.
  * Ganose et al. [2021] Alex M Ganose, Junsoo Park, Alireza Faghaninia, Rachel Woods-Robinson, Kristin A Persson, and Anubhav Jain.  Efficient calculation of carrier scattering rates from first principles.  _Nature Communications_ , 12(1):1–9, 2021.  doi:10.1038/s41467-021-22440-5.  URL https://doi.org/10.1038/s41467-021-22440-5.
  * Poncé et al. [2016] Samuel Poncé, Elena R Margine, Carla Verdi, and Feliciano Giustino.  Epw: Electron–phonon coupling, transport and superconducting properties using maximally localized wannier functions.  _Computer Physics Communications_ , 209:116–133, 2016.
  * Perdew et al. [1996] John P. Perdew, Kieron Burke, and Matthias Ernzerhof.  Generalized gradient approximation made simple.  _Phys. Rev. Lett._ , 77:3865, 1996.  doi:10.1103/PhysRevLett.77.3865.  URL https://doi.org/10.1103/PhysRevLett.77.3865.
  * Miller et al. [2017] Samuel A. Miller, Prashun Gorai, Brenden R. Ortiz, Anuj Goyal, Duanfeng Gao, Scott A. Barnett, Thomas O. Mason, G. Jeffrey Snyder, Qin Lv, Vladan Stevanovic, and Eric S. Toberer.  Capturing anharmonicity in a lattice thermal conductivity model for high-throughput predictions.  _Chemistry of Materials_ , 29(6):2494–2501, 2017\.  doi:10.1021/acs.chemmater.6b04179.  URL https://doi.org/10.1021/acs.chemmater.6b04179.
  * Borup et al. [2012] Kasper A Borup, Eric S Toberer, Leslie D Zoltan, George Nakatsukasa, Michael Errico, Jean-Pierre Fleurial, Bo B Iversen, and G Jeffrey Snyder.  Measurement of the electrical resistivity and hall coefficient at high temperatures.  _Rev. Sci. Instrum._ , 83(12):123902, 2012.
  * Iwanaga et al. [2011] Shiho Iwanaga, Eric S Toberer, Aaron LaLonde, and G Jeffrey Snyder.  A high temperature apparatus for measurement of the seebeck coefficient.  _Rev. Sci. Instrum._ , 82(6):063905, 2011.

## Supplementary Information

Model performance benchmark on small datasets of materials properties

Figure S1: Model performance on a small dataset containing 200 training and
100 testing materials for band gap and formation energy predictions. Scatter
plots show the comparisons between CGCNN and our models trained with language
representations as input.

Single-task predictions on general material properties

Figure S2:  Performance of models trained on individual tasks, evaluated by
5-fold cross validation. Rows correspond to different embedding methods used
for model input. Scatter plots show groud truth vs. predicted values on six
properties of 5,700 materials from AFLOW dataset.

Multi-task learning on general material properties

Figure S3: Multi-task MMoE model performance evaluated by 5-fold cross
validation. Scatter plots show groud truth vs. predicted values on six
properties of 5,700 materials from AFLOW dataset.

Correlation between thermoelectric (TE) properties for MMoE prediction tasks

Figure S4:  Correlation for five TE properties of the experimental UCSB and
ESTM datasets, as indicated by their pairwise Pearson correlation scores
($\rho$). An average of $\sim$0.3 pairwise Pearson correlation presents strong
evidence for MMoE to leverage task correlations.

Single task prediction vs. multi-task learning on TE properties

Figure S5:  Comparisons of single-task learning (top row) and multi-task
learning (bottom row) on TE properties.

Ranking results of top 15 materials (recall top-100 candidates)

Figure S6:  Top-15 candidates from the ranking results of PbTe, Cu2Se,
BiCuOSe, Mg3Sb2, and Bi2Te3.

Ranking results of top 15 materials (recall top-1000 candidates)

Figure S7:  Ranking results of CoSb3 and CuGaTe2 by setting the number of
recalled candidates to 1000. More diverse structural prototypes can be seen in
the top-15 lists.

Predicted $zT$ distribution of recalled materials for well-known TE materials

Figure S8: Predicted $zT$ distributions of 100 recalled candidates for seven
representative TE materials when they are used as query material for the
recommendation framework.

Exploration: Top 10 candidates ranked by their predicted $zT$

Figure S9:  Top-10 ranked candidates with the highest predicted $zT$ in Cl, I,
and oxide perovskite groups.

Evaluation: First-principles calculations of TE properties

Figure S10:  First-principles calculations of TE properties for recommended
materials from search ranking tasks and exploration tasks.

Evaluation: Experimental transport measurements on TE properties of CuZn2GaTe4

Figure S11:  Experimental results on temperature-dependent TE properties of
CuGaTe2 ‘relevant’ material CuZn2GaTe4. Panel display electrical resistivity
(a), carrier concentration (b), carrier mobility (c), Seebeck coefficient (d),
thermal conductivity (e), and figure of merit calculated with parabolic band
model formalism using the experimental input parameters (f).

Evaluation: Synthesis (experiment) on top candidates from ranking tasks

Figure S12: All samples show porosity but excellent x-ray diffraction
patterns. Left column of SEM images (ETD) corresponds to secondary electron
emission, and right column (SSD) corresponds to imaging performed with back-
scattered electrons. We observe good agreement between expected stoichiometry
and stoichiometry measured from energy dispersive spectroscopy (EDS), which
was performed over several points across the sample and averaged.