# Termite: A System for Tunneling Through Heterogeneous Data

Raul Castro Fernandez, Samuel Madden  CSAIL, MIT<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

Data-driven analysis is important in virtually every modern organization. Yet,
most data is underutilized because it remains locked in silos inside of
organizations; large organizations have thousands of databases, and billions
of files that are not integrated together in a single, queryable repository.
Despite 40+ years of continuous effort by the database community, data
integration still remains an open challenge.

In this paper, we advocate a different approach: rather than trying to infer a
common schema, we aim to find another common representation for diverse,
heterogeneous data. Specifically, we argue for an embedding (i.e., a vector
space) in which all entities, rows, columns, and paragraphs are represented as
points. In the embedding, the distance between points indicates their degree
of relatedness. We present Termite, a prototype we have built to _learn_ the
best embedding from the data. Because the best representation is learned, this
allows Termite to avoid much of the human effort associated with traditional
data integration tasks. On top of Termite, we have implemented a _Termite-
Join_ operator, which allows people to identify related concepts, even when
these are stored in databases with different schemas and in unstructured data
such as text files, webpages, etc. Finally, we show preliminary evaluation
results of our prototype via a user study, and describe a list of future
directions we have identified.

††conference: ; March 18;

##  1\. Introduction

Data integration – combining diverse data sets, from different organizations
or with heterogeneous schemas – has been a long standing challenge for the
database community, which we continue to struggle with today. At the core of
this challenge is the fact that modern relational query processors require
data to be carefully organized into a uniform schema. In particular, for
relational operators to provide meaningful results, different columns that
reference the same concept in different data sets must use exactly the same
values and syntax. Duplicates must be eliminated. Values must be normalized.
Errors must be cleaned. Although these challenges have been a boon to academic
researchers who have published hundreds of papers on each of these topics,
they also mean that most data integration projects are hugely time consuming
and expensive, and that many data sets that should be integrated never are due
to the complexity of creating sufficiently uniform data for relational
operations to produce well-defined answers.

In this paper, we advocate an alternative approach. Instead of insisting on
clean data and a standardized schema, we argue that we should accept that many
closely related data sets will never be fully integrated into a single
relational system. Instead, we propose Termite, a“dirt-loving” database system
that provides as much of the power of declarative querying as possible, but on
top of these non-uniform datasets.

The desiderata for a dirt-loving database are clear:

  * •

It should able to query structured but differently-schema’d tabular data,
retrieving related rows from these different tables.

  * •

It should be able to relate structured to unstructured data (i.e., text
files), highlighting portions of the text files that are related to specific
records in the structured data.

Termite supports these goals through a novel _Termite-Join_ capability. Unlike
a conventional relational query processor, where most joins are based on exact
equality matches, in Termite, the join operation retrieves data that is in
_close_ proximity to some input query. That proximity indicates degree of
relatedness, and it is measured as the distance between vectors of an
embedding that represents all cells, rows, columns from relations as well as
text from web pages, and emails, and other files. These _relational
embeddings_ are similar to the word embeddings used for modelling language in
text processing (Mikolov et al., 2013; Pennington et al., 2014), but are
specially constructed to work well for tabular datasets that are relational in
nature.

The key advantage of the embedding representation is that because both
structured and unstructured data are represented as points (vectors),
understanding whether a tuple in a relation is related to a text file boils
down to measuring the distance between their vector representation. The key
challenge is to assign vectors to data in such a way that distance between
data points in the embedding indicates data is indeed related.

Figure 1. Distance in the embedding indicates relatedness Figure 2. End-to-end
overview of the 5 components of the Termite system

Consider the example of Fig. 1, which contains a relation and an excerpt from
Wikipedia. Suppose we want to know what awards have been granted to professors
working at CSAIL. If we only looked at the table, we would miss the Godel
prize being awarded to Nir Shavit. In the embedding, the vector representation
for _Godel prize_ is close to the vector for _Turing award_ because: i) it is
an award, i.e., it is _granted_ , or _won by_ someone, and; ii) it is, in this
case, awarded to someone who works at _CSAIL_. Note that a traditional TF-IDF
based retrieval approach would not be able to identify this relationship. The
_Termite-Join_ operator relates these two together because they _share_
relationships (granted/won award) and entities (CSAIL). With Termite, we can
build an embedding on which the _Termite-Join_ operator works without writing
manual rules. Instead, Termite only needs pairs of elements that are related
to each other; the pairs can be generated automatically from the sentence or
tuple in which the elements appear.

Building the embedding. A principled way of building the embedding is to
represent relational and unstructured data in some multi-dimensional
tensor—which would be very sparse—where dimensions correspond to the different
entities in the data, and then factorize the tensor to obtain a dense
embedding that would contain information about how the entities are related to
each other. This approach is so far only theoretical because we do not know
how to precisely represent all data in a tensor form, or how to factorize it
in such a way that the resulting embedding possesses the desired equivalence
between vector distance and data relatedness. Instead, we propose to _learn_
the embedding directly from the data. The central theme of this paper is
_Termite_ , a system to train and build the embedding, and an operator,
_Termite-Join_ to query the embedding to relate structured and unstructured
data.

As an initial step towards Termite, we have built a proof of concept focused
on helping with discovery problems. We conducted a user study to understand
the benefits of the _Termite-Join_ operator to discover data across structured
and unstructured data such as MIT News, Wikipedia, personal webpages as well
as relational data from the MIT datawarehouse and DBPedia. With Termite, users
found faster more relevant content than with a baseline solution consisting of
a full-text search index carefully built. We complement our evaluation with
results on record linkage and concept expansion, two tasks closely related to
the discovery problem.

We discuss the Termite’s architecture rationale in section 2, the current
learning pipeline in 3, followed by evaluation results (4), related work 5 and
a brief discussion in 6.

##  2\. Termite’s Architecture Rationale

The idea of building an embedding with Termite was inspired by the impact of
statistical language models such as word embeddings (Mikolov et al., 2013;
Pennington et al., 2014) on the NLP, speech recognition and information
retrieval communities. We quickly discovered that it is not straightforward to
directly use these existing techniques, mainly because the assumption that all
those models make—that words that appear often together are related to each
other—does not translate to the set-oriented relational world of tuples,
attributes, and tables, which carry much more structured information.
Furthermore, it is not clear how to merge relational data with unstructured
sources.

We have conceptualized the challenges faced by Termite into 5 loosely coupled
components (Fig. 2). Each stage presents a number of research opportunities.
Rather than exploring in depth each one of them, we decided to first build an
end-to-end prototype so we can learn how the different stages are
interconnected to each other. We describe these stages and our initial
implementation below:

Extraction. The data extraction component converts raw relations and text into
a set of bag-of-words (BoW) representation, e.g., one BoW per triple extracted
from a sentence, or per cell value from a relation. To do that, it uses
different _connectors_. For unstructured data, we use state-of-the-art
information extraction platforms such as (Gabor et al., 2015; Finkel et al.,
2005; H. et al., 2002) to extract entity-relationship-entity triples. For
relational data, a relational discovery tool guesses (Castro Fernandez et al.,
2018; Chen et al., 2014) each relation’s key and uses it along with the
attribute values to produce triples, e.g.,  _John:value - age:attribute -
22:value_.

Encoding. The encoding component transforms each BoW into a vector. The
vectors must be fixed-size so they can be used as the input to the learning
component. A straightforward fixed-sized representation such as one-hot
encoding has two big drawbacks. First, its dimensionality depends on the
vocabulary size, which is large even for small datasets. Second, the
vocabulary size must be known a-priori in order to size the vectors, which is
inconvenient.

Our encoding component, instead, dictionary-encodes the vocabulary terms as
integers, which are assigned incrementally as new words appear. These integers
are indexed into a fixed-sized vector of length F using a hash function in
$1\ldots F$. We size the vector to minimize the number of collisions, which
can be achieved using the birthday paradox and the expected number of words
per BoW. Because collisions will occur anyway, we make a second attempt to
insert the integer using a different hash function. With this encoding
strategy we have seen performance improvements during learning of up to 2
orders of magnitude compared with one-hot encoding for a vocabulary size of 1M
terms.

Learning and Refinement. These components are explained in detail in section
3. Here, we only mention that given the current extractor and encoder
components, which produce triples of the form _subject-predicate-object_ , the
training dataset is built by generating pairs from such triples: _subject-
predicate_ , _predicate-object_ and _subject-object_. The pairs from the
extracted triples are the positive pairs. Suppose we have positive pairs that
always relate a professor to a phone number, and a phone number to an office.
Even if we do not have an explicit pair relating the professor to the the
office, both entities will appear closer to each other in the embedding:
that’s a key advantage of _joining_ in the embedding.

To obtain negative samples we randomly assemble pairs that are not part of the
positive training set, similar to the approach used in (Bordes et al., 2013).
This makes it easy to generate negative pairs, but it introduces anomalies
during the learning process i.e., unrelated points that end up close to each
other not because they are related, but because they were not explicitly
provided as negative pairs. The refinement component ameliorates some of the
anomalies.

Serving. The serving component is Termite’s raison d’etre. It makes the
embedding available to answer database queries. Applications that
traditionally take most of the time from analysts who need to perform them—and
that are therefore not available to organizations without the luxury of
dedicated analysts—become straightforward to perform if an embedding is
available.

One example is data exploration, which we refer as the process of
_visualizing_ schemas to learn the content they represent, _summarizing_
relations to get a glimpse of the information they convey, _understanding_ how
two relations are related to each other without going through the process of
figuring out how to join them. Each of these tasks would take a long time to
solve, but are really simple to solve in a vector space: 1) plot vectors in a
reduced dimensionality to visualize the schema; 2) find a subset of diverse
vectors to summarize a relation; 3) find vectors from the two relations we
want to join that are close to each other in the embedding.

Another example is the task of discovering how data from relations and
unstructured sources is related to each other. Useful for discovery, filling
missing values and verifying information that appears in a table among others.
This is the first application we have focused on and the reason for the
_Termite-Join_ operator we have implemented and focus on in the rest of this
paper.

##  3\. Building a Good Data Embedding

We have explained how to transform a collection of text and relations into a
collection of triples. Here, we explain how to turn the triples into a
collection of vectors, (learning component of Fig. 2) in (3.1). We then
explain how to refine the embedding (component 4 in the figure) and briefly
the _Termite-Join_ operator.

###  3.1. Obtaining a Basic Embedding

Methods such as (Krompaß et al., 2013; Socher et al., 2013; Bordes et al.,
2013) consume triples from a knowledge base to learn the latent variables that
_explain_ observable data, which helps among other tasks, with knowledge base
completion. Our goal is different. We want to learn a distance metric to
measure the relatedness of data coming from databases and text.

The entities we want to represent are the union of the sets of subjects, $S$,
predicates, $P$, and objects $O$ of the triples, $X=S\cup P\cup O$. We want to
find a vector representation for each entity, $f(x_{i})|x_{i}\in X$. In
addition, given a distance function, $d()$, we want the vector representation
of two related entities, $f(x_{i})$ and $f(x_{j})$, to be closer to each other
according to $d()$ than to a third, unrelated vector $f(x_{k})$. Finally, our
training data consists of related and unrelated pairs. How can we find such
vector representation $f()$?

Can’t we just use Word Embeddings?. Word embeddings (Mikolov et al., 2013;
Pennington et al., 2014) assume that words that appear often together are
related to each other. Using very large text corpora—where words are used many
times in different contexts—it is possible to learn a vector representation
for each word, and measuring the distance between word vectors, to determine
whether they are similar or not. The notion of similarity in word embeddings
stems from the usage of words in the _same context_ , e.g.,  _handsome_ and
_pretty_ will be similar to each other because they are often used together in
sentences. In our setting, we have the advantage of knowing precisely which
entities are related to each other (the pairs): there is no need to infer this
from their appearance together. However, we also have the disadvantage that
entities won’t occur many times in many different contexts. We need to find an
alternative.

Vector assignment. Our proposal is to frame the assignment of vectors to
entities in $X$ as an optimization problem amenable to learning, so we can
train it efficiently by feeding the pairs we have in the training dataset. In
particular we want to train a deep network that, when given two input
entities, $x_{i}$ and $x_{j}$, assigns a vector to each of them, $f(x_{i})$
and $f(x_{j})$, computes their distance and predicts whether these two
entities are close to each other. Unlike traditional machine learning models
built for generalization, i.e., to predict output for unseen data, we are only
truly interested in the representation $f()$ learned by the network, so we can
use it to encode our data into the embedding. So, how do we train $f()$?

Figure 3. Architecture of the siamese network

Deep metric learning. We were inspired by the siamese networks used in
(Bromley et al., 1994) for identifying images of similar faces by using a
metric learned by showing examples of similar faces, a task known as deep
metric learning. We used a network such as the one in Fig. 3 to learn a metric
for data. Once the network is trained and it has learned $f()$, we apply it to
each element in $X$, obtaining the embedding representation of our data. And
with that, we are back into database territory.

###  3.2. Refining the Embedding

We can use a repertoire of techniques from databases to store, index and query
the embedding efficiently. We are exploring the best ways to manage and
manipulate the embedding, but in this section we focus on how we can further
improve the embedding quality.

The learned embedding will contain anomalies: vectors that are close together
but are unrelated. This is because we only use a limited number of negative
samples during training and because of the curse of dimensionality. If we
leave the embedding untouched, we will produce wrong results when querying it.
Next, we explain the technique we have implemented in the refinement component
as well as ongoing work:

Curse of dimensionality. Since we are working with a high-dimensional
embedding, we suffer from the curse of dimensionality (Aggarwal et al., 2001).
The worst consequence is the phenomenon known as _hubness_ , which is the
tendency of certain points to be close to many other points. This means that
certain data will be artificially related (close) to a lot of other data,
which is directly against the quality metric we desire for our embedding.

The good news is that we can largely ameliorate this problem. The main
intuition of our technique is that it is possible to compute a _hubness
factor_ for each entity represented in the embedding, and then remove entities
with a high hubness factor in the top-k results. In particular, we compute how
many times each point appears close to other points in the embedding. We then
take the 75 percentile of the number of appearances as a cutoff parameter. At
query time, we filter out those entities of the top-k results with a hubness
factor higher than the cutoff parameter and pad the ranking with additional
entities until we have K elements. Empirically, this improves the quality of
the returned rankings.

Ongoing work: TDA. Can we learn more from the data once it’s in a vector
format? It seems intuitively interesting to understand the shapes the data
forms in the learned embedding, and that may help us further refine the
embedding itself. Whether two or more vectors are related—whether they have a
shape—boils down to determining if they are within a specific distance,
$\delta$, of each other. Then, if they remain close as $\delta$ grows, it is
possible to determine the strength of the relationship. Using the same
intuition, vectors that do not remain close may be categorized as noise.
Topological data analysis (TDA) is a mathematical tool that permits reasoning
about shapes in high-dimensions algebraically. A central concept in TDA is
persistent homology, which indicates which shapes remain in the high-
dimensional embedding as $\delta$ changes. A straightforward application of
persistent homology to our embedding gives us a degree of confidence for each
of the results of the top-k list, depending on how _persistent_ they are in
the embedding. We are currently investigating additional applications of the
technique, as well as how to best use TDA to curate the embedding.

###  3.3. Termite-Join Operator

Given an input entity, _Termite-Join_ returns the $K$ closest entities in the
embedding. All entities representations in the embedding are computed offline
by feeding the entities to the learned network and obtaining their
representation.

At query time, given an input query $x_{i}$: 1) Obtain the embedding
representation, $f(x_{i})$ from the collection. 2) Retrieve the $K$-closest
vectors to $f(x_{i})$ from the collection. 3) Remove each $k_{i}\in K$ whose
hubness factor is beyond the cutoff parameter computed by the refinement
component. 4) Fill in the top-K list if some element has been removed in step
3. 5) Obtain the string representation of the top-k vectors and present the
results to the user.

##  4\. Early Experience

We demonstrate how Termite helps users identify related data across
heterogeneous schemas and unstructured data (section 4.1). Then, we show
additional microbenchmark results on record linkage and concept expansion
(4.2).

Dataset and setup. We built a dataset with information of faculty at CSAIL.
The dataset contains both structured data with different schemas, e.g., MIT
datawarehouse and DBPedia, as well as unstructured data, e.g., Wikipedia
pages, online news articles. We then used Termite to learn the embedding,
using a laptop with 4 cores, 8GB RAM and without access to a GPU. The whole
process took around 1 hour.

###  4.1. Data Discovery with Termite-Join

We did a user study to evaluate _Termite-Join_ :

Study Goals. The goals of the study were to determine: i) whether the
embedding is an appropriate abstraction to discover data across structured and
unstructured data sources; and ii) whether the semantic distance learned is
more appropriate for discovery tasks than a traditional full-text search
interface based on TF-IDF relevance.

To answer these questions, we built two different interfaces to discover data.
One of them, Full-Text-Search (FTS), receives all the data from the data
extractor from Fig. 2 and indexes it in elasticsearch (elastic, [n. d.]). FTS
has an API to perform keyword queries and find the matching documents from the
system. The second interface, Termite-Search (TMT) is built on top of the
embedding. It is similar to the first in that people can query with keywords,
but those keywords are used as input to the _Termite-Join_ operator. Our goal
was to understand which interface was better for a set of discovery tasks we
describe next.

Study Procedure. We recruited 8 users with a CS background and that are daily
users of web search engines. We asked them to solve 3 tasks. The first is used
as a training exercise, and the remaining two, (Task 1 and Task 2), are part
of the experiment. We split the users in two groups of 4 people each, and
showed a different interface to each group to avoid cross-learning effects,
i.e., a person learning the results with one interface and reverse engineering
the right query when using the second interface. We then measured the coverage
of the results obtained by each group and asked for their feedback on both the
questions and the interfaces they used.

We gave each user a 7 minute introduction to the corresponding interface,
along with an example walk-through to illustrate the process. An experimenter
was present at all times with the user, to clarify questions about the task
goal, as well as to suggest ways of using the API when the user had doubts. We
first explained the example task, without telling them it was an example.
Users were asked to _create a list of forms of recognition (i.e., awards) that
have been given to CSAIL faculty_. We asked the users to write their results
in a text file, and explained that they could make as many search requests as
they needed. We let them use the interface for 5 minutes (we did not stop them
abruptly if they were engaged in preparing a query) and then moved on to the
next two tasks: Task 1: _Create a list of contributions associated with CSAIL
faculty_ ; and Task 2: _A list of institutions associated with CSAIL faculty_.

Results. The users of TMT were far more successful than the users of FTS as
the results of table 1 demonstrate. The table shows the percentage of results
found by the users when using each interface, distinguishing between the worst
result achieved by any of the 4 users using the same interface and the average
one. Remarkably, for Task 1, the users that used TMT found a query that led to
all results. In the case of Task 2, only one user found all of them. We
measured the time the users took to perform each task and found that users of
TMT took 2 min less on average than users of FTS. The times were similar,
however, for Task 2.

Table 1. Results from User Study | FTS | TMT
---|---|---
| (worst,
---
avg)
| Avg.
---
Time
| (worst,
---
avg)
| Avg.
---
Time
Task 1 | 1%, 27% | 6.1m | 100% | 4.2m
Task 2 | 9%, 18% | 5.1m | 18%, 43% | 5.3m

When we asked the users for the difficulty of the different tasks they were
solving, users of FTS were consistent in finding Task 1 harder than Task 2,
and Task 2 harder than the example task. The users of TMT mentioned that all
three tasks were similarly simple.

Examples. We show example results obtained by users of both interfaces in
table 2 for both tasks. For Task 1, the users of TMT found algorithmic
contributions, such as _LSH, Zero-knowledge proof_ when inserting software
artifacts such as _Vertica_ or _Postgres_ , see the second column of table 2.
Users of FTS had to _guess_ keywords that would indicate contributions, and
this led naturally to a lower recall. Similar results were found in the case
of Task 2, in which users of the FTS had to try different keywords such as
_university_ , _degree_ , while users of TMT quickly identified that using a
known example, e.g.,  _Harvard_ led quickly to many other relevant results.

Table 2. Example results by users of the study Task 1: Contributions of CSAIL Faculty | Task 2: Associated Organizations of CSAIL Faculty
---|---
FTS | TMT | FTS | TMT
| arvind co founded company
---
Robert_Tappan_Morris
Y_Combinator_(company)
stonebraker focused aurora
hari balakrishnan commercializing
research medusa/aurora project
| Shor’s_algorithm, karger algorithm
---
Chord_(peer-to-peer), haystack project
simultaneous multithreading
Multics, VoltDB, Ingres_(database)
Wait-free, Public-key
Zeroknowledge_proof
RSA_(algorithm)
Alloy_(specification_language)
MOOC
(aprox. 25 more results)
| morris delbarton school
---
dewitt university michigan
morris harvard university
demaine phd university waterloo
meyer phd harvard university
| Princeton_University
---
California_Institute_of_Technology
University_of_Pennsylvania
University_of_California _Berkeley
Stanford_University
Colgate_University
Carnegie_Mellon_University
massachusetts institute technology
La_Sapienza_University_of_Rome
University_of_Michigan
IBM_Almaden_Research_Center
Rice_University, harvard university
(aprox. 10 more results)

Conclusion. With TMT, users discover relevant information for their tasks from
both unstructured and structured sources more efficiently and easier than with
a full-text search index.

###  4.2. Record Linkage and Concept Expansion

Record Linkage (Koudas et al., 2006) is about finding syntactically distinct
records that refer to the same real-world entity, e.g.,  _Samuel R. Madden_
and _Sam Madden_. Concept Expansion (Wang et al., 2015) is about obtaining
instances of a given concept, e.g., given _Harvard_ , obtain _MIT_ , _Caltech_
, _Stanford_ , etc. Our hypothesis is that _Termite-Join_ can help with these
tasks. To understand this empirically, we retrieved ground truth for both
tasks on the same CSAIL faculty dataset mentioned above, and then implemented
functions on Termite to perform each task. We discuss the results next:

Record Linkage. Our dataset contains 52 faculty members. The minimum number of
representations for each faculty member was 2, the average 4, and the maximum
was 7. In total there are 210 different representations for the 52 faculty. To
conduct this experiment we take one representation of each member and use it
to query the embedding with the _Termite-Join_ operator. We then measure how
many of the found results are true alternative representations of the original
query. We could identify 77% (163/210) different representations used to refer
to CSAIL faculty.

In particular, the Termite identifies different spellings of faculty such as
_David DeWitt_ and _Dave DeWitt_ , as well as those that include middle names,
such as _David J. DeWitt_. More important, the embedding helped to identify
entities that were not syntactically similar, such as _Liskov_ and _Barbara
Jane Huberman_. The first appears in the relational data, while the last name
only appears in an unstructured source, but since both representations share
relationships and entities they are placed closed to each other in the
embedding.

Concept Expansion. For this experiment we compiled ground truth following a
procedure similar to the one described in (Wang et al., 2015). We found
instances of the same concept for 10 different concepts, which are shown in
the first column of table 3. The concepts had a number of instances that
ranged from 10 to 80. Unlike the original definition of the concept expansion
problem (Wang et al., 2015), which takes both a concept and an example
instance as input, we only provide an example instance to the _Termite-Join_
operator. We then run queries retrieving lists of different sizes, 2 and 4
times the size of the original (referred to as 2x Top and 4x Top in the
table), and reporting the total percentage of values in the ranking result
that were correct. The first query returns a list of size equal to the number
of instances for the concept. The second and third query subsequently double
the previous size.

Table 3. Results for concept expansion Concept |  | #
---
Instances
| Found
---
Top
| Found
---
2x Top
| Found
---
4x Top
| known
---
for
77 |  | 27
---
35%
| 46
---
59%
| 65
---
84%
| faculty
---
name
52 |  | 30
---
57%
| 46
---
88%
| 51
---
98%
| Institutions
---
44 |  | 26
---
59%
| 33
---
75%
| 37
---
84%
| birth
---
place
35 |  | 14
---
40%
| 18
---
51%
| 24
---
68%
award | 33 |  | 19
---
57%
| 24
---
72%
| 28
---
84%
| academic
---
children
54 |  | 41
---
75%
| 53
---
98%
—
field | 21 |  | 13
---
61%
| 14
---
66%
| 16
---
76%
nationality | 10 |  | 10
---
100%
— | —
| doctoral
---
advisor
31 |  | 12
---
38%
| 16
---
51%
| 23
---
74%
| thesis title
---
19 |  | 8
---
42%
| 13
---
68%
| 16
---
84%

Conclusions. The best news is that we obtained these results by only pointing
out Termite to a data repository. No manual domain-specific engineering was
necessary.

##  5\. Related Work

Automatic Knowledge Base Completion. RESCAL (Krompaß et al., 2013) models
triples from a knowledge base via the pairwise interactions of latent
features. Similarly, Structured Embeddings and subsequent work (Bordes et al.,
2011, 2013; Socher et al., 2013) learns embeddings for each relation from the
triples. These approaches focus on learning the latent variables that describe
the triples, to later fill in values of an incomplete knowledge base. In
contrast, Termite learns a metric we use to relate structured and unstructured
data based on their distance in the embedding.

Universal schema. (Riedel et al., 2013; Verga and McCallum, 2016) decomposes a
matrix in which rows represent entity pairs and columns represent relations
between entities. Similar to our embedding, they show how to jointly embed
text and knowledge bases. They are not focused on data discovery, but rather
information extraction applications.

Word Embeddings and Relational Data. In (Bordawekar and Shmueli, 2017), the
authors propose a method to learn a vector representation of data items from
relational data based on word embeddings (Mikolov et al., 2013), and then use
those vectors to augment traditional SQL queries with so-called _cognitive_
capabilities such as finding elements within a column or row that are similar
to an input data item. In contrast, our embedding’s goal is to find a
relatedness metric for discovery applications beyond only relational data. Our
embedding could be applied to extend SQL queries as well, which is interesting
future work.

Other related techniques. There are a myriad of applications in NLP which
share some characteristics with out goal of learning a good embedding of both
structured and unstructured data. We can benefit from: i) sequence learning
(Hochreiter and Schmidhuber, 1997; Sutskever et al., 2014), ii) alternative
deep metric methods (Hoffer and Ailon, 2015; Sohn, 2016); iii) alternative
embedding methods such as holographic (Nickel et al., 2016) and hyperbolic
embeddings (Dhingra et al., 2018). A recent paper (Mudgal et al., 2018)
explores deep learning for entity resolution, a common data integration task.
Termite’s focus is on leeting users operate on structured and unstructured
data without a high upfront cost.

##  6\. Research Agenda

In this paper we have shown how, by operating on a vector space, we can
understand relationships between structured and unstructured data sources. We
have evaluated Termite, our proof of concept, with a few common applications
in data integration as well as with a user study. The applications of
embedding data into a common vector space, as well as the challenges of doing
so, however, span beyond what we presented here. We use the remainder of this
section to present challenges and additional applications:

  * •

Learning a relational embedding.

The siamese network presented in this paper permits merging relations and text
easily, but it is slow to train. We have performed preliminary results with
neural network architectures designed for modeling language, such as Skipgram
and continuous bag of words (CBOW), which are simpler and therefore faster to
train. Adapting these models to work with structured data is not trivial
because they have been designed to use the sequence nature of text and not the
row- and column-wise relationships represented in tables. One important
challenge, then, is to understand the different learning solutions and
identify tradeoffs between different models.

  * •

Cleaning embeddings.  All of the learning methods we consider lead to high
dimensional vector spaces. Operating on these spaces is challenging for
several reasons. First, we suffer from the hubness problem (Tomasev et al.,
2014), where spurious vectors appear to be close to most other vectors. This
behavior interferes with out goal of relying on distance between vectors to
understand properties of the underlying data. A challenge in this setting is
to detect and clean hubs from the embedding efficiently. For that, we are
exploring techniques from shape analysis as well as topological data
processing, which we think may help us identify and correct errors in data.

  * •

Reducing storage overhead Because every token is mapped to a high dimensional
vector, we suffer a severe storage overhead (sometimes up to 20x) because we
need to store many floating points in addition to the original token. We
cannot learn a low dimensional embedding directly because we need high
capacity networks to learn good embeddings. However, once we possess the high
dimensional embedding, it is possible to apply dimensionality reduction
techniques to reduce the storage overhead. The challenge at this point is to
apply the dimensionality reduction techniques on millions of vectors, a
process that becomes too slow. To tackle this problem, we are exploring
techniques to reduce the dimensionality of a sample of the data only, and then
derive vectors in a low dimensional space by taking into consideration the
relative distance of points in the original embedding.

  * •

Identifying intrinsic evaluation criteria. In this paper we have measured the
quality of the learned embedding by using downstream tasks that are well
defined, and for which we can compare metrics easily. An open question is
whether we can identify features of vector embedding we can use to perform
intrinsic evaluations, which would help with speeding up the rate of
innovation.

Despite the challenges, we are motivated to pursue this line of work, not only
because of the promise of merging structured and unstructured data, but also
because the promise of many other applications which we briefly discuss here:

  * •

Database exploration.  Many techniques exist to explore databases, from
clustering of tables, to summarizing relations, to visualizing what tables
look more like each other, and many more. Each of these applications requires
the design an implementation of a system. We are exploring how we can
translate all the previous applications into mere vector operations, and
therefore simplify the database exploration process end to end.

  * •

Statistics retrieval.  Databases, and in particular, query optimizers, rely on
data statistics to perform well their job. A relational embedding captures
data properties that can be beneficial to databases.

  * •

Performance benchmarking.  When customers of commercial databases have a
performance problem and the vendors are not allowed to see the data,
troubleshooting the underlying problem becomes difficult. To tackle this
problem we can train a generative model from the embedding, and then use the
generative model to create databases of different sizes for benchmarking.
Furthermore, we can prevent certain security risks by obfuscating the tokens
in the dictionary that maps them to vectors.

We believe the vision exposed in this paper along with the encouraging results
justify investing more time in understanding how a vector representation of
data may benefit difficult data management problems.

## References

  * (1)
  * Aggarwal et al. (2001) Charu C. Aggarwal, Alexander Hinneburg, et al. 2001\.  On the Surprising Behavior of Distance Metrics in High Dimensional Spaces. In _ICDT_.
  * Bordawekar and Shmueli (2017) Rajesh Bordawekar and Oded Shmueli. 2017.  Using Word Embedding to Enable Semantic Queries in Relational Databases. In _DEEM_.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, et al. 2013\.  Translating Embeddings for Modeling Multi-relational Data. In _NIPS_.
  * Bordes et al. (2011) Antoine Bordes, Jason Weston, et al. 2011\.  Learning Structured Embeddings of Knowledge Bases.. In _AAAI_.
  * Bromley et al. (1994) Jane Bromley, Isabelle Guyon, et al. 1994\.  Signature verification using a ”siamese” time delay neural network. In _NIPS_.
  * Castro Fernandez et al. (2018) Raul Castro Fernandez, Abedjan Ziawasch, et al. 2018\.  Aurum: A Data Discovery System. In _ICDE_.
  * Chen et al. (2014) Zhimin Chen, Vivek Narasayya, et al. 2014\.  Fast Foreign-key Detection in Microsoft SQL Server PowerPivot for Excel.  _VLDB_ (2014).
  * Dhingra et al. (2018) Bhuwan Dhingra, Christopher Shallue, et al. 2018\.  Embedding Text in Hyperbolic Spaces. In _TextGraphs-12_.
  * elastic ([n. d.]) elastic [n. d.].  ElasticSearch.  http://www.elastic.com.
  * Finkel et al. (2005) J. R. Finkel, Grenager T., et al. 2005\.  Incorporating non-local information into information extraction systems by Gibbs sampling.  _ACL_ (2005).
  * Gabor et al. (2015) Angeli Gabor, Johnson Melvin, et al. 2015\.  Leveraging Linguistic Structure For Open Domain Information Extraction.  _ACL_ (2015).
  * H. et al. (2002) Cunningham H., Bontcheva K., and Maynard D. 2002.  GATE: an architecture for development of robust HLT applications.  _ACL_ (2002).
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Comput._ (1997).
  * Hoffer and Ailon (2015) Elad Hoffer and Nir Ailon. 2015.  Deep metric learning using triplet network. In _SIMBAD_.
  * Koudas et al. (2006) Nick Koudas, Sunita Sarawagi, and Divesh Srivastava. 2006\.  Record Linkage: Similarity Measures and Algorithms. In _SIGMOD_.
  * Krompaß et al. (2013) Denis Krompaß, Maximilian Nickel, et al. 2013\.  Non-negative tensor factorization with rescal. In _ECML workshop_.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, et al. 2013\.  Distributed Representations of Words and Phrases and Their Compositionality. In _NIPS_.
  * Mudgal et al. (2018) Sidharth Mudgal, Han Li, et al. 2018\.  Deep Learning for Entity Matching: A Design Space Exploration. In _SIGMOD_.
  * Nickel et al. (2016) Maximilian Nickel, Lorenzo Rosasco, et al. 2016\.  Holographic Embeddings of Knowledge Graphs. In _AAAI_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, et al. 2014\.  GloVe: Global Vectors for Word Representation. In _EMNLP_.
  * Riedel et al. (2013) Sebastian Riedel, Limin Yao, et al. 2013\.  Relation extraction with matrix factorization and universal schemas. In _NAACL-HLT_.
  * Socher et al. (2013) Richard Socher, Danqi Chen, et al. 2013\.  Reasoning with Neural Tensor Networks for Knowledge Base Completion. In _NIPS_.
  * Sohn (2016) Kihyuk Sohn. 2016\.  Improved Deep Metric Learning with Multi-class N-pair Loss Objective.  In _NIPS_.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014\.  Sequence to Sequence Learning with Neural Networks. In _NIPS_.
  * Tomasev et al. (2014) N. Tomasev, M. Radovanovic, et al. 2014\.  The Role of Hubness in Clustering High-Dimensional Data.  _IEEE TKDE_ (2014).
  * Verga and McCallum (2016) Patrick Verga and Andrew McCallum. 2016.  Row-less Universal Schema.  _AKBC_ (2016).
  * Wang et al. (2015) Chi Wang, Kaushik Chakrabarti, et al. 2015\.  Concept Expansion Using Web Tables. In _WWW_.


# Structure-property relations of silicon oxycarbides studied using a machine
learning interatomic potential

Niklas Leimeroth  Institut für Materialwissenschaft, Technische Universität
Darmstadt, Otto-Berndt-Strasse 3, 64287 Darmstadt, Germany  Corresponding
author email<EMAIL_ADDRESS>Jochen Rohrer  Institut für
Materialwissenschaft, Technische Universität Darmstadt, Otto-Berndt-Strasse 3,
64287 Darmstadt, Germany  Karsten Albe  Institut für Materialwissenschaft,
Technische Universität Darmstadt, Otto-Berndt-Strasse 3, 64287 Darmstadt,
Germany

###### Abstract

Silicon oxycarbides show outstanding versatility due to their highly tunable
composition and microstructure. Consequently, a key challenge is a thorough
knowledge of structure-property relations in the system. In this work, we fit
an atomic cluster expansion potential to a set of actively learned DFT
training data spanning a wide configurational space. We demonstrate the
ability of the potential to produce realistic amorphous structures and
rationalize the formation of different morphologies of the turbostratic free
carbon phase. Finally, we relate the materials stiffness to its composition
and microstructure, finding a delicate dependence on Si-C bonds that
contradicts commonly assumed relations to the free carbon phase.

##  1 Introduction

SiOC are highly versatile materials that combine remarkable structural and
functional properties. Among them are high temperature resistance, good
mechanical strength [1, 2], great creep- and corrosion-resistance [3, 4], as
well as piezoresistivity and the ability to reversibly store Li+, Na+ and K+
[5, 6, 7, 8]. These properties make them interesting for applications in very
different fields, such as protective coatings [9], energy storage [10] and
biomedicine [11].

Despite the plethora of desirable properties and intensive research, open
questions about structural features and their relation even to basic
characteristics such as the Young’s modulus (E) remain. From NMR measurements
it is known that silicon oxycarbides consist of corner-shared SiOxC4-x
tetrahedra with carbidic sp3-hybridized carbon [12, 13] and a segregated
secondary phase with sp2-hybridized turbostratic carbon [14, 15]. The detailed
nanostructure, however, remains elusive and sensitively depends on the
composition, precursor structure and processing conditions. For example, it is
still unclear in which form the turbostratic carbon is present in Si-O-C.
Scarmi et al. [16] and Saha et al. [17] argued that interpenetrating networks
of graphene-like carbon and silica-rich mixed tetrahedra domains are formed,
based on the high creep resistance of the material. In contrast, Widgeon et
al. [12] found that a model with graphitic inclusions embedded in a silica
rich matrix result in a better match of the mass fractal dimensions of mixed
tetrahedra.

Here, atomistic simulations may help in understanding structure formation and
structure-property relations of Si-O-C compounds at the nanoscale. As shown
for example in a series of studies by Kroll [18, 19, 20], ab-initio molecular
dynamics (MD) simulations can be employed to investigate structural details,
energetics and elastic properties of Si-O-C, but are limited to structures
consisting of a few hundred atoms and simulation times on the order of tens of
picoseconds. Consequently, the heterogeneity on the nanoscale intrinsic to Si-
O-C can not be reproduced in full extend. Large-scale MD simulations, on the
other hand, require suitable interatomic potentials and the complex nature of
the strongly directional covalent bonds is hard to capture in empirical
formulas like bond order potentials. Recent studies showed that the ReaxFF
framework [21] allows to study specific aspects of the Si-O-C system. For
example, Newsome et al. simulated the oxidation of silicon carbide [22]. Soria
et al. investigated organic molecules on silicon surfaces [23], Gao et al. and
Ponomarev et al. simulated the pyrolysis of specific polymers to amorphous Si-
O-C [24, 25]. These potentials, however, have a limited scope and require a
reparametrization for each application. Furthermore, only two out of the four
parameter sets are publicly available [22, 23]. Here, modern machine learning
interatomic potentials offer an alternative approach to describe complex
systems over a wide compositional and structural range at similar
computational cost, but at the expanse of requiring more training data. Recent
studies have shown the successful application of MLIPs to carbon [26, 27, 28,
29, 30, 31, 32, 33], silicon [34, 35, 36, 37, 38, 39], SiO2 [40, 41, 42, 43]
and SiC [44, 45]. In this work, we present an Atomic Cluster Expansion (ACE)
potential for the Si-O-C system fitted to an extensive database generated
using the active learning (AL) capabilities of Moment Tensor potentials [46]
and ACE [47]. We show that the potential achieves a high accuracy for a wide
compositional range and that it can be used to produce realistic amorphous Si-
O-C structures. We investigate structural features, formation energies and
Young’s moduli for samples with varying compositions and precursor
configurations. Thereby, we find that the structure model containing graphene
like sheets and the graphitic inclusions in a silica matrix are both likely to
describe the structure of Si-O-C, but occur at different stages of processing.
Furthermore, we establish relations of Young’s modulus to the fraction of Si-C
bonds in mixed SiOxC4-x tetrahedra and the silica volume.

##  2 Methods

(a)

(b)

(c)

Figure 1:  Si-O-C structure generation and active-learning strategy. Most initial structures for the AL process were generated by densely packing smaller building blocks using the PACKMOL program. These building blocks and the nomenclature used to describe them throughout this work are shown in (1(a)). Si is shown in beige, O in red and C in gray. PD structures are stripped of H atoms. GrAts building blocks were used to produce large sample structures, but not in the AL process. The AL procedure employed to iteratively improve the training data is schematically shown in (1(b)). Structures for AL and samples used in the analysis were produced using a cook and quench simulations depicted in (1(c)). The compressions step shown as dotted line was only applied for PD and GrAts structures, where it was necessary to obtain nonporous bulk samples. For structures made from BFs or Ats it was not necessary, because the initial packing already leads to reasonable densities.  | XSi | XO | XC
---|---|---|---
Si2O2C | 0.4 | 0.4 | 0.2
PMSQ | 0.25 | 0.5 | 0.25
RD-212 | 0.25 | 0.25 | 0.5
SILRES-604 | 0.125 | 0.125 | 0.75
RD-684 |  $\approx$0.121 |  $\approx$0.121 |  $\approx$0.758
Table 1: Compositions of hydrogen stripped polymers, sorted from low to high
carbon content. These compositions were used to generate polymer derived (PD),
bulk fragment (BF), isolated atoms (Ats) and graphite flakes and isolated
atoms (GrAts) samples (See Fig. 1(a)).

###  2.1 Training and testing data

The training data for the potential was produced as follows. Initial Si, O and
Si-O structures were taken from an ACE potential previously fitted for the
Si-O system [48]. Pure C structures were generated using the ASE package [49]
and SiC structures were taken from the materials project database [50]. Si-O-C
structures with varying compositions were generated with two different
procedures. Firstly, structures were produced to match the expected
coordination, i.e. 4-fold for Si and C and 2-fold for O [51, 52]. Additional
structures were then generated using an AL procedure, schematically shown in
Fig. 1. We employed PACKMOL [53], which implements an algorithm to densely
pack structural fragments while keeping adjustable minimal distances between
atoms [54]. With this approach structures based on polymers inspired from
Polymethylsilsesquioxane (PMSQ), the polyorganosiloxanes RD-212, RD-684 and
SILRES-604 and a fully artificial polymer made from Si2O2C monomers were
generated with removed H atoms. In the following we will use the compositions,
instead of polymer names as shown in Tab. 1. Stripping H from the polymers
avoids a massive extension of the necessary training data, as the
configuration space for a quarternary system is much larger than for a
ternary, but keeps the polymeric backbone. We expect that this has a
negligible influence for final structures produced with pyrolysis temperatures
of $1273\text{\,}\mathrm{K}1523\text{\,}\mathrm{K}$ and higher, because H
evaporates during synthesis and the remaining amount of H atoms in them is
very small [55, 56, 57].

In order to cover a wider variety of compositions and structural features, we
also produced configurations made from isolated atoms (Ats) and bulk fragments
(BFs) SiO2, SiC4, CSi4 and C6 as shown in Fig. 1(a). For the structures
generated directly from Ats and BFs the composition was varied, with Si and C
concentrations ranging from $0\text{\,}\%100\text{\,}\%$ and the oxygen
concentration from $02.2$ times the Si concentration. The AL capabilities of
MTPs [58] and ACE potentials [47] were employed to iteratively generate new
Si-O-C configurations as shown in Fig. 1(b). Here, we started with MTPs
because the AL capabilities for ACE were implemented only recently [47].
Furthermore, we covered a wider variety of structures by using both codes. The
training data set was considered complete when no new structures, that had a
maximum density-functional theory (DFT) force of less than
$150\text{\,}\mathrm{e}\mathrm{V}\mathrm{/}\mathrm{\text{Å}}$, were discovered
in cook and quench simulations with temperatures up to
$3000\text{\,}\mathrm{K}$ and pressures up to
$200\text{\,}\mathrm{G}\mathrm{P}\mathrm{a}$.

For the test data set the test data of the Si-O potential [48] was
supplemented with a separately created set of Si-O-C structures. These Si-O-C
test structures were created with PACKMOL at varying densities and
compositions. Consequently, the atoms were randomly displaced.

Finally, the training and test data sets were filtered to not contain
structures with a maximum force of more than
$150\text{\,}\mathrm{e}\mathrm{V}\mathrm{/}\mathrm{\text{Å}}$, a minimal
distance between atoms smaller than $0.6\text{\,}\frac{\mathrm{\text{Å}}}{}$
or greater than $4\text{\,}\frac{\mathrm{\text{Å}}}{}$ or an energy of more
than
$20\text{\,}\mathrm{e}\mathrm{V}\mathrm{/}\mathrm{a}\mathrm{t}\mathrm{o}\mathrm{m}$
above the convex hull.

###  2.2 Details of potentials

For the potentials we used a cutoff of $5\text{\,}\frac{\mathrm{\text{Å}}}{}$.
The ACE and MTPs were fitted using the pacemaker [39, 59] and MLIP packages
[46]. For the AL process with MTPs a level 26 potential was employed. The
intermediate ACE potentials for AL were fitted with the triple embedding
$\rho_{1}^{0.5}+\rho_{2}^{1}+\rho_{3}^{2}$ and a total of 2325 basis
functions. In principle, the most straightforward way of increasing the
accuracy of ACE potentials is to increase the number of basis functions.
However, this considerably increases the computational cost [59]. Instead, we
tested different embedding terms for the final potential, which is
computationally very cheap. Here, we found a highly nonlinear sum of
expansions $\sum_{i}^{n}\phi_{i}^{\alpha_{i}}$ with $n=10$ and exponents
$\alpha_{i}$ 0.125, 0.25, 0.375, 0.5, 0.75, 0.875, 1, 1.25, 1.5 and 2 to
result in the best testing errors.

###  2.3 SiOC sample structures

The structure samples used to analyze formation energies, structural features
and elastic properties were produced in a cook and quench process, as
schematically shown in Fig. 1(c). Here we tested two different degrees of
freedom, the influence of the composition and the effect of the precursor on
the structure and properties of the final sample. The compositions of the
structures correspond to the 5 polymeric compositions shown in Tab. 1. As
precursors, we employed 4 different types of building blocks shown in Fig.
1(a). The graphite sheets in GrAts structures consist of 160 atom 2-layer
graphite. The resulting 20 structures were used in cook and quench simulations
with annealing temperatures of $1000$, $1500$ and $2000\text{\,}\mathrm{K}$ to
obtain a total of 60 sample structures. Here, the annealing time was
$1\text{\,}\mathrm{n}\mathrm{s}$. As shown in the supplemental material
(section S1) this time is sufficient to reach a steady state regarding
different structural features. The employed quench rate was
$1\text{\times}{10}^{12}\text{\,}\mathrm{K}\mathrm{/}\mathrm{s}$. PD and GrAts
based structures required an additional compression step before the cook and
quench process to obtain nonporous initial structures. For this purpose, they
were equilibrated at $500\text{\,}\mathrm{K}$ with an applied isotropic
pressure of $10\text{\,}\mathrm{G}\mathrm{P}\mathrm{a}$ for
$10\text{\,}\mathrm{p}\mathrm{s}$ before heating them up to the annealing
temperature.

###  2.4 Simulations

DFT calculations were carried out with the same settings as used for a silica
potential previously fitted [43] to keep the training data consistent. The
plane wave code VASP [60, 61, 62] with projector-augmented wave [63]
pseudopotentials and the SCAN [64] meta-GGA exchange-correlation were employed
with a plane-wave cutoff of $900\text{\,}\mathrm{e}\mathrm{V}$ and a k-spacing
of $0.23\text{\,}\mathrm{\text{Å}}^{-1}$. Classical MD simulations were
carried out with LAMMPS [65], applying GPU accelerated KOKKOS versions where
possible. If not otherwise noted an NPT ensemble with isotropic
$0\text{\,}\mathrm{P}\mathrm{a}$ pressure, Nosé-Hoover thermo- and barostats
and a timestep of $1\text{\,}\mathrm{f}\mathrm{s}$ were employed in the
simulations.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2:  Training data and performance of the potential. Training data
resulting from the AL procedure (2(a)) with the black box indicating the
magnified region shown in (2(b)). Error of energies as function of the
formation energy distance from the convex hull (2(c)) and correlation of
forces (2(d)) as calculated with DFT and the ACE potential for the training
and test datasets. Training and test RMSEs of energies (2(e)) and forces
(2(e)) during optimization of the potential.

##  3 Results

In the following we will start by shortly presenting the results of the
applied AL procedure and evaluate the newly developed MLIP. Then we will
discuss the structure of the produced Si-O-C samples based on their
composition and precursors going from low to high C contents. Finally, we will
relate structural features to the Young’s modulus of the samples.

###  3.1 Training and performance of the potential

The flexibility of the ACE formalism allows for an accurate description of
highly complex materials, under the condition that similar atomic
configurations have been part of the data used in their training procedure. In
this work, an AL procedure was employed to ensure that the training data
covers a large phase space volume. Here, we started with MTPs as implemented
in the MLIP package [46], and continued with ACE potentials for which AL
capabilities were implemented only recently [47]. We included structures with
varying compositions and at high temperatures and pressures, as well as
different defective structures that can form during MD simulations. This makes
the potential applicable to a wide area of problems. Fig. 2(a) and 2(b) show
the resulting distribution of the training data in terms of atomic energies
and volumes.

The mostly actively learned Si-O-C structures are scattered widely in this 2D
representation of phase space. The pure elements and SiC structures were
mostly made by hand. In combination with less compositional degrees of freedom
this leads to a comparatively narrow distribution for them. To prevent the
occurrence of unphysically large forces on single atoms during MD simulations
at high temperatures or pressures we found it helpful to add SiC structures
with a high density, leading to a second area with aggregated SiC structures
around
$4\text{\,}\mathrm{e}\mathrm{V}\mathrm{/}\mathrm{a}\mathrm{t}\mathrm{o}\mathrm{m}$
above the convex hull.

Energies and forces predicted using the potential agree well with those
calculated using DFT over the whole range of structures, as shown in the
scatter plots in Fig. 2(c) and 2(d). For energies an RMSE of $24$ and
$36\text{\,}\mathrm{m}\mathrm{e}\mathrm{V}\mathrm{/}\mathrm{a}\mathrm{t}\mathrm{o}\mathrm{m}$
for the training and test sets were obtained. For forces the RMSEs were $479$
and $650\text{\,}\mathrm{m}\mathrm{e}\mathrm{V}\mathrm{/}\mathrm{\text{Å}}$
respectively. We want to note that the higher errors for the testing set are a
result from the different distribution of the structures regarding their
composition and energy and not from overfitting as shown by the continuous
decrease of testing errors in Fig. 2(e) and 2(f).

###  3.2 Structure and energetics

####  3.2.1 Si0.4O0.4C0.2

Figure 3: Si0.4O0.4C0.2 assembled precursors and final structures. Shown are
samples as packed with PACKMOL (upper row) and after the cook and quench
process (lower row) with an annealing temperature of $1500\text{\,}\mathrm{K}$
for all structure prototypes. The PD, BF and Ats structures appear to be very
similar upon visual inspection. The graphite like flakes in the GrAts based
structure are still present after processing, showing that they are
kinetically stabilized, as one would expect a decomposition into SiO2 and SiC
from an energetic viewpoint.

The microstructure and properties of Si-O-C compounds depend strongly on the
precursor material and processing conditions like pyrolysis temperature [7].
Time and lengths scales of the experimental processes can not be reproduced
directly in MD simulations. Instead, we used different building blocks to
produce a variety of microstructures in cook and quench simulations as
previously described. Examples of these initial structures are shown in the
upper row of Fig. 3 Exemplary samples produced via the cook and quench
protocol described in section 2.3 with an annealing temperature of
$1500\text{\,}\mathrm{K}$ are shown in the row below. Qualitatively, the PD,
BF and Ats structures are very similar. Major differences are only found for
GrAts. Here, large graphite areas are still present in the final structure.
From a purely thermodynamic viewpoint this is surprising, as one would expect
that Si0.4O0.4C0.2 decomposes into SiO2 and SiC, but at the tested
temperatures and timescales the kinetics do not allow for such a phase
separation.

Figure 4: Structure analysis and formation energies of Si-O-C samples. The
left column shows the free carbon volume fraction, defined as Voronoi volume
of all 3-fold coordinated C atoms, the center column shows the silica volume
fraction (Voronoi volume of SiO4 tetrahedra. The right column contains the
formation energies of samples with respect to $\alpha$-quartz, $\beta$-SiC and
graphite. Rows correspond to different sample compositions. Si0.125O0.125C0.75
and Si0.121O0.121C0.758 are shown in the same row, as a direct comparison of
them is interesting especially in the case of PD structures due to their
compositional similarity.

To quantify differences between the structures we calculated the Voronoi
volume fractions occupied by 3-fold coordinated C atoms, i.e. the free carbon
phase and SiO4 tetrahedra, i.e. the silica phase. The results are shown in
Fig. 4. Only the structures based on GrAts contain a meaningful amount of free
carbon. As discussed previously there is no thermodynamic driving force for
the formation of graphene or graphite like carbon, so its volume fraction is
determined by the kinetics of the system and this behavior can be expected. In
the GrAts structure the volume fraction stays constant between annealing
temperatures $1000$ and $1500\text{\,}\mathrm{K}$. At
$2000\text{\,}\mathrm{K}$ a slight decrease can be observed, showing that the
graphite phase is kinetically stabilized up to Temperatures greater than
$1500\text{\,}\mathrm{K}$. Regarding the silica volume fraction a continuous
increase at increasing temperatures can be observed for all structures,
indicating a higher degree of phase separation. However, the data shows large
differences with regard to the absolute values. The largest amount is observed
for the GrAts structure, followed by BFs, Ats and finally the PD
configuration. Again this observation can be rationalized by the
thermodynamics and kinetics of the system. In the GrAts structure the unpaired
Si and O atoms can quickly form silica without interference of the carbon,
because it is already bound in the graphite phase. Si-C can only form in the
relatively small interface regions or requires long range diffusion. In the BF
based structure SiO2, SiC4 and Si4C units are already present, only requiring
rotations and minor rearrangement to form the thermodynamically favored
products, while the diffusion paths necessary to achieve phase separation in
the atom based structure are long and intermediate bonds can form and need to
brake again. Similarly, the PD structure contains Si bonded to C and O, which
needs to break before forming pure SiO4 or SiC4 tetrahedra.

The right column of Fig. 4 shows the formation energy of samples with respect
to $\alpha$-quartz, $\beta$-SiC and graphite. In the case of Si0.4O0.4C0.2
they are very close in energy, despite the significant structural differences.
Especially for the structures annealed at $2000\text{\,}\mathrm{K}$ the
difference of nearly 0 for may irritate, when considering that the system
still contains a considerable amount of free carbon in the case GrAts, but
this energetically unfavorable state is apparently compensated by the high
fraction of very favorable silica in the system.

####  3.2.2 Si0.25O0.5C0.25

The carbon content of Si0.25O0.5C0.25 is very close to the previously
discussed Si0.4O0.4C0.2, but the different Si:O ratio leads to a different
thermodynamic situation. In equilibrium Si0.25O0.5C0.25 should split into a
pure silica and a graphite phase, while no SiC should form. Consequently, the
formation energies of structures based on GrAts are considerably lower than
those of the other precursors for Si0.25O0.5C0.25. The difference between the
Ats and GrAts structures with an annealing temperature of
$1000\text{\,}\mathrm{K}$ is about
$300\text{\,}\mathrm{m}\mathrm{e}\mathrm{V}\mathrm{/}\mathrm{a}\mathrm{t}\mathrm{o}\mathrm{m}$,
i.e. the driving forces for phase separation are very high. Since the
structures are in a steady state, this also indicates a very low mobility. For
an annealing temperature of $2000\text{\,}\mathrm{K}$ the difference between
precursors becomes much smaller, reaching around
$100\text{\,}\mathrm{m}\mathrm{e}\mathrm{V}\mathrm{/}\mathrm{a}\mathrm{t}\mathrm{o}\mathrm{m}$.
It is expected that the differences shrink, because the increased mobility at
higher temperatures allows coming closer to the thermodynamic equilibrium. The
free carbon and silica volume in Si0.25O0.5C0.25 samples increases
significantly with increasing annealing temperatures. The least changes are
observed for the GrAts structure, which is already close to the phase
separation. Generally, the arguments regarding the kinetics of the different
precursors discussed for Si0.4O0.4C0.2 also apply for Si0.25O0.5C0.25, so
similar trends can be observed, with the main differences determined by the
decomposition products.

####  3.2.3 Si0.25O0.25C0.5, Si0.125O0.125C0.75 and Si0.121O0.121C0.758

Si0.25O0.25C0.5, Si0.125O0.125C0.75 and Si0.121O0.121C0.758 contain Si and O
in a 1:1 ratio, but, in contrast to Si0.4O0.4C0.2, also excess carbon. Thus we
expect the formation of a varying amount of free carbon phase in the system
and not just SiO2 and SiC. The compositional similarity of Si0.125O0.125C0.75
and Si0.121O0.121C0.758 allows us to investigate the influence of different
polymer-like precursor structures. In the Si0.121O0.121C0.758 polymer C6 rings
are stacked very closely (cf. Fig. 1(a)), which could ease the formation of
graphite. However, this can not be observed in our data. Generally the three
compounds show high free carbon contents and qualitatively similar temperature
dependence for both, the silica and free carbon fractions. The energetically
most favorable sample is the one made from GrAts. Similar to Si0.4O0.4C0.2
this can be explained by the high degree of phase separation, which
corresponds to the thermodynamic equilibrium. Si0.125O0.125C0.75 and
Si0.121O0.121C0.758 behave similar.

Figure 5: Structure formation of free carbon phase in Si-O-C. For low
pyrolysis temperatures and times graphene like carbon forms as argued by
Scarmi et al. [16] and Saha et al. [17] based on high creep resistance. At
higher temperatures energetically more favorable graphite like agglomerates
form, which are consistent with the mass fractal dimension as found by Widgeon
et al. [12]. The figures show the free carbon phase as found for
Si0.25O0.25C0.5PD sample annealed at $2000\text{\,}\mathrm{K}$ (a), GrAts
sample annealed at $2000\text{\,}\mathrm{K}$ (b) and an additional PD sample
annealed at $3500\text{\,}\mathrm{K}$ (c). Their formation energies suggests
that the Si-O-C system would evolve from PD to GrAts structures, so we
consider it as an intermediate step even though timescales accessible in MD do
not allow a direct observation.

####  3.2.4 Relation to model structures

As discussed previously, two models for the nanostructure of Si-O-C exist. One
suggests silica rich nanodomains, which are separated by an interconnected
graphene like carbon network [16, 17]. The other describes the structure as
graphitic inclusions in a silica rich matrix [12]. The results from our
simulations are shown in Fig. 5. For structures that should contain graphite
upon decomposition we found the GrAts models to be energetically favorable,
favoring the latter model. However, in the other samples no graphitic carbon
could be found and instead graphene like layers spread through the system,
pointing to the former. This allows us to conclude that both models are
representative for two distinct stages. In early stages of structure formation
the structure is likely described by the first model, because the slow
kinetics in the system prevent the formation of graphitic inclusions. With
increasing temperatures and pyrolysis times the structure evolves towards the
latter model argued for by Widgeon et al., as it is lower in energy.

(a)

(b)

(c)

Figure 6: Dependence of E on structural features. Calculated E values for
samples in dependence of silica volume fraction (6(a)), average amount of
carbon in mixed tetrahedra (6(b)) and free carbon volume fraction (6(c)).
Experimental data in (6(c)) is taken from [66, 67, 68, 69, 3, 70] as collected
in [70] and from [71].

###  3.3 Elastic properties

The structure of Si-O-C is highly tunable, depending on the pyrolysis
conditions and precursors. An understanding of structure-property relations
can therefore guide the search for processing routes that best match specific
needs. Here, we investigate the structural features influencing the elastic
properties of Si-O-C. For this purpose we calculated the elastic tensors of
our samples using pymatgen [72] and derived the Young’s modulus

$E=\mu(3\lambda+2\mu)/(\lambda+\mu)$ (1)

with the Lamè constants $\mu=C_{44}$ and $\lambda=C_{12}$, as valid for
isotropic materials. Here, the assumption of isotropy is very accurate with
differences between $C_{ij}$ that should be equivalent due to symmetry in the
range of $\pm 3\text{\,}\%$. Only the GrAts structures show larger
anisotropies, due to the limited amount of randomly orientated graphite
flakes, leading to differences in the range of $\pm 15\text{\,}\%$. To reduce
resulting errors we averaged over the supposedly equivalent directions. The
deformations applied to obtain elastic tensors are not fully reversible
because the relaxation of atomic positions leads to small energy barriers on
the path back to the initial state that cannot be overcome in static
relaxations. Corresponding energy-strain relations can be found in the
supplemental material.

Fig. 6 shows the dependence of E on the SiO4 volume, the average amount of
carbon in SiO4-xCx tetrahedra and the free carbon volume in the samples.
Generally the modulus ranges from $70\text{\,}\mathrm{G}\mathrm{P}\mathrm{a}$,
which corresponds to E of silica glass [73] up to around
$145\text{\,}\mathrm{G}\mathrm{P}\mathrm{a}$, which agrees well with
experiments. Also, it scatters significantly between different compositions
and building blocks, showing the multidimensional nature of the problem.
Strong correlations of E can be seen for the SiO4 volume fraction and the
amount of carbon within SiO4-xCx tetrahedra. As expected the modulus decreases
for higher silica volume fractions, approaching that of the pure amorphous
phase and increases with an increasing amount of carbon in the mixed bonds. As
discussed previously, GrAts based structures contain high amounts of SiO4
tetrahedra and the C atoms are bound within the graphite inclusions, i.e. they
do not participate much in mixed bonding. Consequently, these structures have
lower Young’s moduli than structures based on other building blocks at the
same compositions. Experimental studies have established empirical relations
between structural features of Si-O-C and its elastic properties. Typically,
it is observed that the Young’s modulus increases with the amount of SiC and C
within the SiO4-xCx tetrahedra and decreases with increasing free carbon
volume [67, 71]. However, as shown by Stabler et al. [70] the results can
differ significantly based on the kind of sample and applied measurement
method. Furthermore, a recent simulation based study by Haseen and Kroll [74]
found a strong decrease of E with decreasing sample densities employing a
Tersoff type potential. They argued that this could overlap the direct effect
of the Cfree phase, because the density of samples also strongly correlates
with the amount of Cfree. Indeed, they observed an increase of E with higher
free carbon contents for samples with same density. These results fit well to
our observation of a weak overall correlation of E with Cfree. If anything,
when taking into account all data points we observe a slight increase.
Considering each composition individually can also help to resolve the
discrepancy with experimental results, as all of them but Si0.25O0.5C0.25 show
downwards trends. Therefore, we conclude that the amount of free carbon only
weakly influences E, but at similar compositions more free carbon is
equivalent to lower amounts of the strong Si-C bonds, indirectly leading to a
lower stiffness.

##  4 Conclusion

Using multiple iterations of AL techniques implemented for MTPs and ACE
potentials we produced a highly diverse set of amorphous Si-O-C structures,
spanning a large area of phase space. With the converged dataset, we fitted a
nonlinear ACE potential to energies and forces calculated with DFT and showed
that the potential can accurately reproduce them. Due to the diversity of the
training data, the potential has a large applicability range and can describe
the formation of Si-O-C compounds and their properties in a large temperature
and pressure range.

Applying the potential, we produced amorphous Si-O-C samples with various
compositions in a cook and quench procedure. Here, we tested the influence of
different initial structures on the final configuration. We found that
manually added graphite agglomerates are kinetically stabilized in MD
simulation times and lead to energetically favorable states compared to
graphene like sheets, if the structure contains excess C. However, their
formation is kinetically hindered, and we assume that an interpenetrating
network of silica rich domains and graphene like sheets is formed as an
intermediate step during synthesis of amorphous Si-O-C.

To test and establish structure-property relations in Si-O-C we calculated
Young’s moduli of the samples. We found that the silica volume fraction and
the average amount of carbon in mixed SiO4-xCx tetrahedra correlate well with
the stiffness across all samples. Silica reduces the stiffness and high
amounts of Si-C bonds increase it. Furthermore, the free carbon volume
fraction is a good indicator of E for samples with similar compositions, but
not in general.

We believe that the presented potential will allow detailed investigations of
structure formation and properties in Si-O-C. In a broader context, we have
shown that modern MLIPs can be employed to study multi element material
systems that form highly complex structures. Therefore, they can greatly
benefit the understanding of glassy and ceramic materials on the atomistic
scale.

## Acknowledgments

N. L. acknowledges Linus C. Erhard for helpful discussions. N. L. recieved
funding from the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under grant number 405621137 and from the German Federal Ministry
of Education and Research (BMBF) under project HeNa (FKZ 03XP0390A). J. R.
acknowledges funding from the European Unions Horizon 2020 research and
innovation programme under Grant Agreement no. 963542. K. A. acknowledges
funding by the DFG under project number 413956820. The authors gratefully
acknowledge the computing time provided to them on the high-performance
computer Lichtenberg at the NHR Centers NHR4CES at TU Darmstadt. This is
funded by the German Federal Ministry of Education and Research, and the state
governments participating.

## Data availability

The potential and datasets are available on zenodo upon journal acceptance
[75].

## Author contributions

N. L. performed most computations and analysis. J. R. provided hand crafted
Si-O-C structures and performed early AL iterations with MTPs. All authors
contributed to the research design and interpretation of results. N. L. wrote
the paper with input from J. R. and K. A.

## Competing interests

The authors declare no competing interests.

## References

  * [1] Ziyong Li et al.  “Additive Manufacturing of Lightweight and High-Strength Polymer-Derived SiOC Ceramics”  In _Virtual and Physical Prototyping._ 15.2 Taylor & Francis, 2020, pp. 163–177  DOI: 10.1080/17452759.2019.1710919
  * [2] Elizabeth Barrios and Lei Zhai  “A Review of the Evolution of the Nanostructure of SiCN and SiOC Polymer Derived Ceramics and the Impact on Mechanical Properties”  In _Molecular Systems Design & Engineering._ 5.10 Royal Society of Chemistry, 2020, pp. 1606–1641  DOI: 10.1039/D0ME00123F
  * [3] Benjamin Papendorf et al.  “High-Temperature Creep Behavior of Dense SiOC-Based Ceramic Nanocomposites: Microstructural and Phase Composition Effects”  In _Journal of the American Ceramic Society._ 96.1, 2013, pp. 272–280  DOI: 10.1111/jace.12067
  * [4] Christina Stabler et al.  “High-Temperature Creep Behavior of a SiOC Glass Ceramic Free of Segregated Carbon”  In _Journal of the European Ceramic Society._ 36.15, Preparation and Application of Ultra-high Temperature Ceramic Matrix Composites, 2016, pp. 3747–3753  DOI: 10.1016/j.jeurceramsoc.2016.04.015
  * [5] Marco Melzi d’Eril et al.  “Effect of Ultra-Fast Pyrolysis on Polymer-Derived SiOC Aerogels and Their Application as Anodes for Na-ion Batteries”  In _Open Ceramics._ 14, 2023, pp. 100354  DOI: 10.1016/j.oceram.2023.100354
  * [6] Pradeep Vallachira Warriam Sasikumar et al.  “Structural Design of Polymer-Derived SiOC Ceramic Aerogels for High-Rate Li Ion Storage Applications”  In _Journal of the American Ceramic Society._ 99.9, 2016, pp. 2977–2983  DOI: 10.1111/jace.14323
  * [7] Christina Stabler et al.  “Silicon Oxycarbide Glasses and Glass-Ceramics: “All-Rounder” Materials for Advanced Structural and Functional Applications”  In _Journal of the American Ceramic Society._ 101.11, 2018, pp. 4817–4856  DOI: 10.1111/jace.15932
  * [8] Christian Chandra et al.  “Understanding Lithium, Sodium, and Potassium Storage Mechanisms in Silicon Oxycarbide”  In _Chemical Engineering Journal._ 428, 2022, pp. 131072  DOI: 10.1016/j.cej.2021.131072
  * [9] Kathy Lu and Donald Erb  “Polymer Derived Silicon Oxycarbide-Based Coatings”  In _International Materials Reviews._ 63.3 Taylor & Francis, 2018, pp. 139–161  DOI: 10.1080/09506608.2017.1322247
  * [10] Qingbo Wen et al.  “Si-Based Polymer-Derived Ceramics for Energy Conversion and Storage”  In _Journal of Advanced Ceramics._ 11, 2022, pp. 197–246  DOI: 10.1007/s40145-021-0562-2
  * [11] Marcela Arango-Ospina et al.  “Review: Silicon Oxycarbide Based Materials for Biomedical Applications”  In _Applied Materials Today._ 18, 2020, pp. 100482  DOI: 10.1016/j.apmt.2019.100482
  * [12] S.. Widgeon et al.  “29Si and 13C Solid-State NMR Spectroscopic Study of Nanometer-Scale Structure and Mass Fractal Characteristics of Amorphous Polymer Derived Silicon Oxycarbide Ceramics”  In _Chemistry of Materials._ 22.23 American Chemical Society, 2010, pp. 6221–6228  DOI: 10.1021/cm1021432
  * [13] Gabriela Mera et al.  “Polymer -Derived SiCN and SiOC Ceramics – Structure and Energetics at the Nanoscale”  In _Journal of Materials Chemistry A._ 1.12 Royal Society of Chemistry, 2013, pp. 3826–3836  DOI: 10.1039/C2TA00727D
  * [14] Gian Domenico Sorarù et al.  “Structural Characterization and High-Temperature Behavior of Silicon Oxycarbide Glasses Prepared from Sol-Gel Precursors Containing Si-H Bonds”  In _Journal of the American Ceramic Society._ 78.2, 1995, pp. 379–387  DOI: 10.1111/j.1151-2916.1995.tb08811.x
  * [15] Felix Roth et al.  “UV Raman Spectroscopy of Segregated Carbon in Silicon Oxycarbides”  In _Journal of the Ceramic Society of Japan._ 124.10, 2016, pp. 1042–1045  DOI: 10.2109/jcersj2.16100
  * [16] A. Scarmi, G.D. Sorarù and R. Raj  “The Role of Carbon in Unexpected Visco(an)Elastic Behavior of Amorphous Silicon Oxycarbide above 1273K”  In _Journal of Non-Crystalline Solids._ 351.27-29, 2005, pp. 2238–2243  DOI: 10.1016/j.jnoncrysol.2005.06.008
  * [17] Atanu Saha, Rishi Raj and Don L. Williamson  “A Model for the Nanodomains in Polymer-Derived SiCO”  In _Journal of the American Ceramic Society._ 89.7, 2006, pp. 2188–2195  DOI: 10.1111/j.1551-2916.2006.00920.x
  * [18] Peter Kroll  “Modelling and Simulation of Amorphous Silicon Oxycarbide”  In _Journal of Materials Chemistry._ 13.7 Royal Society of Chemistry, 2003, pp. 1657–1668  DOI: 10.1039/B301389H
  * [19] Peter Kroll  “Modeling the ‘Free Carbon’ Phase in Amorphous Silicon Oxycarbide”  In _Journal of Non-Crystalline Solids._ 351.12, Proceedings of the International Conference on Non-Crystalline Materials (CONCIM), 2005, pp. 1121–1126  DOI: 10.1016/j.jnoncrysol.2005.01.010
  * [20] Peter Kroll  “Searching Insight into the Atomistic Structure of SiCO Ceramics”  In _Journal of Materials Chemistry._ 20.46 Royal Society of Chemistry, 2010, pp. 10528–10534  DOI: 10.1039/C0JM01583K
  * [21] Adri C.. Duin, Siddharth Dasgupta, Francois Lorant and William A. Goddard  “ReaxFF: A Reactive Force Field for Hydrocarbons”  In _The Journal of Physical Chemistry A._ 105.41 American Chemical Society, 2001, pp. 9396–9409  DOI: 10.1021/jp004368u
  * [22] David A. Newsome et al.  “Oxidation of Silicon Carbide by O2 and H2O: A ReaxFF Reactive Molecular Dynamics Study, Part I”  In _The Journal of Physical Chemistry C._ 116.30 American Chemical Society, 2012, pp. 16111–16121  DOI: 10.1021/jp306391p
  * [23] Federico A. Soria et al.  “Si/C/H ReaxFF Reactive Potential for Silicon Surfaces Grafted with Organic Molecules”  In _The Journal of Physical Chemistry C._ 122.41 American Chemical Society, 2018, pp. 23515–23527  DOI: 10.1021/acs.jpcc.8b07075
  * [24] Hongfei Gao et al.  “Reactive Dynamics Simulation Study on the Pyrolysis of Polymer Precursors To Generate Amorphous Silicon Oxycarbide Structures”  In _The Journal of Physical Chemistry C._ 122.10 American Chemical Society, 2018, pp. 5767–5773  DOI: 10.1021/acs.jpcc.7b12287
  * [25] Ilia Ponomarev, Adri C.. Duin and Peter Kroll  “Reactive Force Field for Simulations of the Pyrolysis of Polysiloxanes into Silicon Oxycarbide Ceramics”  In _The Journal of Physical Chemistry C._ 123.27, 2019, pp. 16804–16812  DOI: 10.1021/acs.jpcc.9b03810
  * [26] Rustam Z. Khaliullin et al.  “Graphite-Diamond Phase Coexistence Study Employing a Neural-Network Mapping of the Ab Initio Potential Energy Surface”  In _Physical Review B._ 81.10 American Physical Society, 2010, pp. 100103  DOI: 10.1103/PhysRevB.81.100103
  * [27] Volker L. Deringer and Gábor Csányi  “Machine Learning Based Interatomic Potential for Amorphous Carbon”  In _Physical Review B._ 95.9 American Physical Society, 2017, pp. 094203  DOI: 10.1103/PhysRevB.95.094203
  * [28] Mingjian Wen and Ellad B. Tadmor  “Hybrid Neural Network Potential for Multilayer Graphene”  In _Physical Review B._ 100.19 American Physical Society, 2019, pp. 195419  DOI: 10.1103/PhysRevB.100.195419
  * [29] Miguel A. Caro  “Optimizing Many-Body Atomic Descriptors for Enhanced Computational Performance of Machine Learning Based Interatomic Potentials”  In _Physical Review B._ 100.2 American Physical Society, 2019, pp. 024112  DOI: 10.1103/PhysRevB.100.024112
  * [30] Patrick Rowe et al.  “An Accurate and Transferable Machine Learning Potential for Carbon”  In _The Journal of Chemical Physics._ 153.3 American Institute of Physics, 2020, pp. 034702  DOI: 10.1063/5.0005084
  * [31] Yusuf Shaidu et al.  “A Systematic Approach to Generating Accurate Neural Network Potentials: The Case of Carbon”  In _npj Computational Materials._ 7.1 Nature Publishing Group, 2021, pp. 1–13  DOI: 10.1038/s41524-021-00508-6
  * [32] Jinjin Wang et al.  “A Deep Learning Interatomic Potential Developed for Atomistic Simulation of Carbon Materials”  In _Carbon._ 186, 2022, pp. 1–8  DOI: 10.1016/j.carbon.2021.09.062
  * [33] Minaam Qamar et al.  “Atomic Cluster Expansion for Quantum-Accurate Large-Scale Simulations of Carbon”  In _Journal of Chemical Theory and Computation._ American Chemical Society, 2023  DOI: 10.1021/acs.jctc.2c01149
  * [34] Albert P. Bartók, James Kermode, Noam Bernstein and Gábor Csányi  “Machine Learning a General-Purpose Interatomic Potential for Silicon”  In _Physical Review X._ 8.4 American Physical Society, 2018, pp. 041048  DOI: 10.1103/PhysRevX.8.041048
  * [35] X. Qian et al.  “Thermal Conductivity Modeling Using Machine Learning Potentials: Application to Crystalline and Amorphous Silicon”  In _Materials Today Physics._ 10, 2019, pp. 100140  DOI: 10.1016/j.mtphys.2019.100140
  * [36] T. Yokoi, Y. Noda, A. Nakamura and K. Matsunaga  “Neural-Network Interatomic Potential for Grain Boundary Structures and Their Energetics in Silicon”  In _Physical Review Materials._ 4.1 American Physical Society, 2020, pp. 014605  DOI: 10.1103/PhysRevMaterials.4.014605
  * [37] Janine George et al.  “Combining Phonon Accuracy with High Transferability in Gaussian Approximation Potential Models”  In _The Journal of Chemical Physics._ 153.4 American Institute of Physics, 2020, pp. 044104  DOI: 10.1063/5.0013826
  * [38] A. Hamedani et al.  “Insights into the Primary Radiation Damage of Silicon by a Machine Learning Interatomic Potential”  In _Materials Research Letters._ 8.10 Taylor & Francis, 2020, pp. 364–372  DOI: 10.1080/21663831.2020.1771451
  * [39] Yury Lysogorskiy et al.  “Performant Implementation of the Atomic Cluster Expansion (PACE) and Application to Copper and Silicon”  In _npj Computational Materials._ 7.1, 2021, pp. 97  DOI: 10.1038/s41524-021-00559-9
  * [40] Ivan S. Novikov and Alexander V. Shapeev  “Improving Accuracy of Interatomic Potentials: More Physics or More Data? A Case Study of Silica”  In _Materials Today Communications._ 18, 2019, pp. 74–80  DOI: 10.1016/j.mtcomm.2018.11.008
  * [41] I.. Balyakin, S.. Rempel, R.. Ryltsev and A.. Rempel  “Deep Machine Learning Interatomic Potential for Liquid Silica”  In _Physical Review E._ 102.5 American Physical Society, 2020, pp. 052125  DOI: 10.1103/PhysRevE.102.052125
  * [42] Keita Kobayashi, Yuki Nagai, Mitsuhiro Itakura and Motoyuki Shiga  “Self-Learning Hybrid Monte Carlo Method for Isothermal–Isobaric Ensemble: Application to Liquid Silica”  In _The Journal of Chemical Physics._ 155.3 American Institute of Physics, 2021, pp. 034106  DOI: 10.1063/5.0055341
  * [43] Linus C. Erhard, Jochen Rohrer, Karsten Albe and Volker L. Deringer  “A Machine-Learned Interatomic Potential for Silica and Its Relation to Empirical Models”  In _npj Computational Materials._ 8.1, 2022, pp. 90  DOI: 10.1038/s41524-022-00768-w
  * [44] M.. Kutzhanov et al.  “Al/SiC Nanocomposites with Enhanced Thermomechanical Properties Obtained from Microwave Plasma-Treated Nanopowders”  In _Materials Science and Engineering: A._ 824, 2021, pp. 141817  DOI: 10.1016/j.msea.2021.141817
  * [45] Yong Liu et al.  “Deep Learning Inter-Atomic Potential for Irradiation Damage in 3C-SiC”, 2023  DOI: 10.48550/arXiv.2305.19516
  * [46] Ivan S. Novikov, Konstantin Gubaev, Evgeny V. Podryabinkin and Alexander V. Shapeev  “The MLIP Package: Moment Tensor Potentials with MPI and Active Learning”  In _Machine Learning: Science and Technology._ 2.2 IOP Publishing, 2021, pp. 025002  DOI: 10.1088/2632-2153/abc9fe
  * [47] Yury Lysogorskiy, Anton Bochkarev, Matous Mrovec and Ralf Drautz  “Active Learning Strategies for Atomic Cluster Expansion Models”  In _Physical Review Materials._ 7.4 American Physical Society, 2023, pp. 043801  DOI: 10.1103/PhysRevMaterials.7.043801
  * [48] Linus C. Erhard, Jochen Rohrer, Karsten Albe and Volker L. Deringer  “Modelling Atomic and Nanoscale Structure in the Silicon-Oxygen System through Active Machine Learning”, 2023  DOI: 10.48550/arXiv.2309.03587
  * [49] Ask Hjorth Larsen et al.  “The Atomic Simulation Environment—a Python Library for Working with Atoms”  In _Journal of Physics: Condensed Matter._ 29.27, 2017, pp. 273002  DOI: 10.1088/1361-648X/aa680e
  * [50] Anubhav Jain et al.  “Commentary: The Materials Project: A Materials Genome Approach to Accelerating Materials Innovation”  In _APL Materials._ 1.1 American Institute of Physics, 2013, pp. 011002  DOI: 10.1063/1.4812323
  * [51] Jochen Rohrer et al.  “Si- and Sn-containing SiOCN-based Nanocomposites as Anode Materials for Lithium Ion Batteries: Synthesis, Thermodynamic Characterization and Modeling”  In _International Journal of Materials Research._ 108.11 De Gruyter, 2017, pp. 920–932  DOI: 10.3139/146.111517
  * [52] Edina Šić et al.  “SiCO Ceramics as Storage Materials for Alkali Metals/Ions: Insights on Structure Moieties from Solid-State NMR and DFT Calculations”  In _ChemSusChem._ n/a.n/a, pp. e202202241  DOI: 10.1002/cssc.202202241
  * [53] L. Martínez, R. Andrade, E.. Birgin and J.. Martínez  “PACKMOL: A package for building initial configurations for molecular dynamics simulations”  In _Journal of Computational Chemistry._ 30.13, 2009, pp. 2157–2164  DOI: 10.1002/jcc.21224
  * [54] José Mario Martínez and Leandro Martínez  “Packing Optimization for Automated Generation of Complex System’s Initial Configurations for Molecular Dynamics and Docking”  In _Journal of Computational Chemistry._ 24.7, 2003, pp. 819–825  DOI: 10.1002/jcc.10216
  * [55] Qingbo Wen, Zhaoju Yu and Ralf Riedel  “The Fate and Role of in Situ Formed Carbon in Polymer-Derived Ceramics”  In _Progress in Materials Science._ 109, 2020, pp. 100623  DOI: 10.1016/j.pmatsci.2019.100623
  * [56] O. Delverdier, M. Monthioux, D. Mocaer and R. Pailler  “Thermal Behavior of Polymer-Derived Ceramics. I. Si-C and Si-C-O Systems from Both Commercial and New Polycarbosilane (PCS) Precursors”  In _Journal of the European Ceramic Society._ 12.1, 1993, pp. 27–41  DOI: 10.1016/0955-2219(93)90068-3
  * [57] Carlo G. Pantano, Anant K. Singh and Hanxi Zhang  “Silicon Oxycarbide Glasses”  In _Journal of Sol-Gel Science and Technology._ 14.1, 1999, pp. 7–25  DOI: 10.1023/A:1008765829012
  * [58] Evgeny V. Podryabinkin and Alexander V. Shapeev  “Active Learning of Linearly Parametrized Interatomic Potentials”  In _Computational Materials Science._ 140, 2017, pp. 171–180  DOI: 10.1016/j.commatsci.2017.08.031
  * [59] Anton Bochkarev et al.  “Efficient Parametrization of the Atomic Cluster Expansion”  In _Physical Review Materials._ 6.1 American Physical Society, 2022, pp. 013804  DOI: 10.1103/PhysRevMaterials.6.013804
  * [60] G. Kresse, J. Furthmüller and J. Hafner  “Theory of the Crystal Structures of Selenium and Tellurium: The Effect of Generalized-Gradient Corrections to the Local-Density Approximation”  In _Physical Review B._ 50.18 American Physical Society, 1994, pp. 13181–13185  DOI: 10.1103/PhysRevB.50.13181
  * [61] G. Kresse and J. Furthmüller  “Efficiency of Ab-Initio Total Energy Calculations for Metals and Semiconductors Using a Plane-Wave Basis Set”  In _Computational Materials Science._ 6.1, 1996, pp. 15–50  DOI: 10.1016/0927-0256(96)00008-0
  * [62] G. Kresse and J. Furthmüller  “Efficient Iterative Schemes for Ab Initio Total-Energy Calculations Using a Plane-Wave Basis Set”  In _Physical Review B._ 54.16 American Physical Society, 1996, pp. 11169–11186  DOI: 10.1103/PhysRevB.54.11169
  * [63] G. Kresse and D. Joubert  “From Ultrasoft Pseudopotentials to the Projector Augmented-Wave Method”  In _Physical Review B._ 59.3 American Physical Society, 1999, pp. 1758–1775  DOI: 10.1103/PhysRevB.59.1758
  * [64] Jianwei Sun, Adrienn Ruzsinszky and John P. Perdew  “Strongly Constrained and Appropriately Normed Semilocal Density Functional”  In _Physical Review Letters._ 115.3 American Physical Society, 2015, pp. 036402  DOI: 10.1103/PhysRevLett.115.036402
  * [65] Aidan P. Thompson et al.  “LAMMPS - a Flexible Simulation Tool for Particle-Based Materials Modeling at the Atomic, Meso, and Continuum Scales”  In _Computer Physics Communications._ 271, 2022, pp. 108171  DOI: 10.1016/j.cpc.2021.108171
  * [66] Gary M. Renlund, Svante Prochazka and Robert H. Doremus  “Silicon Oxycarbide Glasses: Part II. Structure and Properties”  In _Journal of Materials Research._ 6.12, 1991, pp. 2723–2734  DOI: 10.1557/JMR.1991.2723
  * [67] Gian Domenico Sorarù, Elisabetta Dallapiccola and Gennaro D’Andrea  “Mechanical Characterization of Sol–Gel-Derived Silicon Oxycarbide Glasses”  In _Journal of the American Ceramic Society._ 79.8, 1996, pp. 2074–2080  DOI: 10.1111/j.1151-2916.1996.tb08939.x
  * [68] S Walter, G. Soraru, H Bréquel and S Enzo  “Microstructural and Mechanical Characterization of Sol Gel-Derived Si–O–C Glasses”  In _Journal of the European Ceramic Society._ 22.13, 2002, pp. 2389–2400  DOI: 10.1016/S0955-2219(01)00537-4
  * [69] Claude Moysan et al.  “Mechanical Characterization of a Polysiloxane-Derived SiOC Glass”  In _Journal of the European Ceramic Society._ 27.1, 2007, pp. 397–403  DOI: 10.1016/j.jeurceramsoc.2006.01.016
  * [70] Christina Stabler et al.  “Effect of Composition and High-Temperature Annealing on the Local Deformation Behavior of Silicon Oxycarbides”  In _Journal of the European Ceramic Society._ 39.7, 2019, pp. 2287–2296  DOI: 10.1016/j.jeurceramsoc.2019.02.024
  * [71] Gian Domenico Sorarù, Lakshminath Kundanati, Balanand Santhosh and Nicola Pugno  “Influence of Free Carbon on the Young’s Modulus and Hardness of Polymer-Derived Silicon Oxycarbide Glasses”  In _Journal of the American Ceramic Society._ 102.3, 2019, pp. 907–913  DOI: 10.1111/jace.16131
  * [72] Shyue Ping Ong et al.  “Python Materials Genomics (Pymatgen): A Robust, Open-Source Python Library for Materials Analysis”  In _Computational Materials Science._ 68, 2013, pp. 314–319  DOI: 10.1016/j.commatsci.2012.10.028
  * [73] Seiji Inaba, Shigeru Fujino and Kenji Morinaga  “Young’s Modulus and Compositional Parameters of Oxide Glasses”  In _Journal of the American Ceramic Society._ 82.12, 1999, pp. 3501–3507  DOI: 10.1111/j.1151-2916.1999.tb02272.x
  * [74] Shariq Haseen and Peter Kroll  “Analyzing the Effect of Composition, Density, and the Morphology of the “Free” Carbon Phase on Elastic Moduli in Silicon Oxycarbide Ceramics”  In _Journal of the European Ceramic Society._ 43.4, 2023, pp. 1432–1441  DOI: 10.1016/j.jeurceramsoc.2022.11.025
  * [75] Niklas Leimeroth, Jochen Rohrer and Karsten Albe  “Potential and Training Data for ’Structure-Property Relations of Silicon Oxycarbides Studied Using a Novel Machine Learning Interatomic Potential”’  DOI: 10.5281/zenodo.10131450

  *[E]: Young’s modulus
  *[Si-O-C]: silicon oxycarbide
  *[MD]: molecular dynamics
  *[MLIPs]: machine learning interatomic potential
  *[ACE]: Atomic Cluster Expansion
  *[AL]: active learning
  *[PD]: polymer derived
  *[BF]: bulk fragment
  *[Ats]: isolated atoms
  *[GrAts]: graphite flakes and isolated atoms
  *[PMSQ]: Polymethylsilsesquioxane
  *[BFs]: bulk fragment
  *[MTPs]: Moment Tensor potential
  *[DFT]: density-functional theory
  *[MLIP]: machine learning interatomic potential