# Adversarial Deep Structured Nets for
Mass Segmentation from Mammograms

###### Abstract

Mass segmentation provides effective morphological features which are
important for mass diagnosis. In this work, we propose a novel end-to-end
network for mammographic mass segmentation which employs a fully convolutional
network (FCN) to model a potential function, followed by a conditional random
field (CRF) to perform structured learning. Because the mass distribution
varies greatly with pixel position, the FCN is combined with a position
priori. Further, we employ adversarial training to eliminate over-fitting due
to the small sizes of mammogram datasets. Multi-scale FCN is employed to
improve the segmentation performance. Experimental results on two public
datasets, INbreast and DDSM-BCRP, demonstrate that our end-to-end network
achieves better performance than state-of-the-art approaches.
111https://github.com/wentaozhu/adversarial-deep-structural-networks.git

Index Terms—  Adversarial deep structured networks, segmentation, adversarial
fully convolutional networks

##  1 Introduction

According to the American Cancer Society, breast cancer is the most frequently
diagnosed solid cancer and the second leading cause of cancer death among U.S.
women. Mammogram screening has been proven to be an effective way for early
detection and diagnosis, which significantly decrease breast cancer mortality.
Mass segmentation provides morphological features, which play crucial roles
for diagnosis.

Traditional studies on mass segmentation rely heavily on hand-crafted
features. Model-based methods build classifiers and learn features from masses
[1, 2]. There are few works using deep networks for mammogram [3]. Dhungel et
al. employed multiple deep belief networks (DBNs), Gaussian mixture model
(GMM) classifier and a priori as potential functions, and structured support
vector machine (SVM) to perform segmentation [4]. They further used CRF with
tree re-weighted belief propagation to boost the segmentation performance [5].
A recent work used the output from a convolutional network (CNN) as a
complimentary potential function, yielding the state-of-the-art performance
[6]. However, the two-stage training used in these methods produces potential
functions that easily over-fit the training data.

In this work, we propose an end-to-end trained adversarial deep structured
network to perform mass segmentation (Fig. 1). The proposed network is
designed to robustly learn from a small dataset with poor contrast
mammographic images. Specifically, an end-to-end trained FCN with CRF is
applied. Adversarial training is introduced into the network to learn robustly
from scarce mammographic images. Different from DI2IN-AN using a generative
framework [7], we directly optimize pixel-wise labeling loss. To further
explore statistical property of mass regions, a spatial priori is integrated
into FCN. We validate the adversarial deep structured network on two public
mammographic mass segmentation datasets. The proposed network is demonstrated
to outperform other algorithms for mass segmentation consistently.

Our main contributions in this work are: (1) We propose an unified end-to-end
training framework integrating FCN+CRF and adversarial training. (2) We employ
an end-to-end network to do mass segmentation while previous works require a
lot of hand-designed features or multi-stage training. (3) Our model achieves
the best results on two most commonly used mammographic mass segmentation
datasets.

Fig. 1: The proposed adversarial deep FCN-CRF network with four convolutional
layers followed by CRF for structured learning.

##  2 FCN-CRF Network

Fully convolutional network (FCN) is a commonly used model for image
segmentation, which consists of convolution, transpose convolution, or pooling
[8]. For training, the FCN optimizes maximum likelihood loss function

$\mathcal{L}_{FCN}=-\frac{1}{N\times
N_{i}}\sum_{n=1}^{N}\sum_{i=1}^{N_{i}}\log
P_{fcn}{(y_{n,i}|\textbf{I}_{n};\bm{\theta})},$ (1)

where $y_{n,i}$ is the label of $i$th pixel in the $n$th image
$\textbf{I}_{n}$, $N$ is the number of training mammograms, $N_{i}$ is the
number of pixels in the image, and $\bm{\theta}$ is the parameter of FCN. Here
the size of images is fixed to $40\times 40$ and $N_{i}$ is 1,600.

CRF is a classical model for structured learning, well suited for image
segmentation. It models pixel labels as random variables in a Markov random
field conditioned on an observed input image. To make the annotation
consistent, we use $\textbf{y}=(y_{1},y_{2},\dots,y_{i},\dots,y_{1,600})^{T}$
to denote the random variables of pixel labels in an image, where
$y_{i}\in\\{0,1\\}$. The zero denotes pixel belonging to background, and one
denotes it belonging to mass region. The Gibbs energy of fully connected
pairwise CRF is [9]

$E(\textbf{y})=\sum_{i}\psi_{u}(y_{i})+\sum_{i<j}\psi_{p}(y_{i},y_{j}),$ (2)

where unary potential function $\psi_{u}(y_{i})$ is the loss of FCN in our
case, pairwise potential function $\psi_{p}(y_{i},y_{j})$ defines the cost of
labeling pair $(y_{i},y_{j})$,

$\psi_{p}(y_{i},y_{j})=\mu(y_{i},y_{j})\sum_{m}w^{(m)}k_{G}^{(m)}(\textbf{f}_{i},\textbf{f}_{j}),$
(3)

where label compatibility function $\mu$ is given by the Potts model in our
case, $w^{(m)}$ is the learned weight, pixel values $I_{i}$ and positions
$p_{i}$ can be used as the feature vector $\textbf{f}_{i}$, $k_{G}^{(m)}$ is
the Gaussian kernel applied to feature vectors [9],

$k_{G}(\textbf{f}_{i},\textbf{f}_{j})=[\exp(-|I_{i}-I_{j}|^{2}/2),\exp(-|p_{i}-p_{j}|^{2})/2]^{T}.$
(4)

Efficient inference algorithm can be obtained by mean field approximation
$Q(\textbf{y})=\prod_{i}Q_{i}(y_{i})$ [9]. The update rule is

$\displaystyle\tilde{Q}_{i}^{(m)}(l)\leftarrow\sum_{i\neq
j}k_{G}^{(m)}(\textbf{f}_{i},\textbf{f}_{j})Q_{j}(l)\text{ for all }m,$ (5)
$\displaystyle\check{Q}_{i}(l)\leftarrow\sum_{m}w^{(m)}\tilde{Q}_{i}^{(m)}(l),\quad\hat{Q}_{i}(l)\leftarrow\sum_{l^{\prime}\in\mathcal{L}}\mu(l,l^{\prime})\check{Q}_{i}(l),$
$\displaystyle\breve{Q}_{i}(l)\leftarrow\exp(-\psi_{u}(y_{i}=l))-\hat{Q}_{i}(l),\quad
Q_{i}\leftarrow\frac{1}{Z_{i}}\exp\left(\breve{Q}_{i}(l)\right),$

where the first equation is the message passing from label of pixel $i$ to
label of pixel $j$, the second equation is re-weighting with the learned
weights $w^{(m)}$, the third equation is compatibility transformation, the
fourth equation is adding unary potentials, and the last step is
normalization. Here $\mathcal{L}=\\{0,1\\}$ denotes background or mass. The
initialization of inference employs unary potential function as
$Q_{i}(y_{i})=\frac{1}{Z_{i}}\exp(-\psi_{u}(y_{i}))$. The mean field
approximation can be interpreted as a recurrent neural network (RNN) [10].

##  3 Adversarial FCN-CRF Nets

The shape and appearance priori play important roles in mammogram mass
segmentation [11, 6]. The distribution of labels varies greatly with position
in the mammographic mass segmentation. From observation, most of the masses
are located in the center of region of interest (ROI), and the boundary areas
of ROI are more likely to be background (Fig. 2(a)).

(a)

(b)

Fig. 2: The empirical estimation of a priori on INbreast (left) and DDSM-BCRP
(right) training datasets (a). Trimap visualizations on the DDSM-BCRP dataset,
segmentation groundtruth (first column), trimap of width $2$ (second column),
trimaps of width $3$ (third column) (b).

The conventional FCN provides independent pixel-wise predictions. It considers
global class distribution difference corresponding to bias in the last layer.
Here we employ a priori for position into consideration

$P(y_{i}|\textbf{I};\bm{\theta})\propto
w_{i}P_{fcn}{(y_{i}|\textbf{I};\bm{\theta})},$ (6)

where $w_{i}$ is the empirical estimation of mass varied with the pixel
position $i$, and $P_{fcn}{(y_{i}|\textbf{I};\bm{\theta})}$ is the predicted
mass probability of conventional FCN. In the implementation, we added an image
sized bias in the softmax layer as the empirical estimation of mass for FCN to
train network. The $-\log P(y_{i}|\textbf{I};\bm{\theta})$ is used as the
unary potential function for $\psi_{u}(y_{i})$ in the CRF as RNN. For multi-
scale FCN as potential functions, the potential function is defined as
$\psi_{u}(y_{i})=\sum_{u^{\prime}}w_{(u^{\prime})}\psi_{u^{\prime}}(y_{i})$,
where $w_{(u^{\prime})}$ is the learned weight for unary potential function,
$\psi_{u^{\prime}}(y_{i})$ is the potential function provided by FCN of each
scale.

Adversarial training provides strong regularization for deep networks. The
idea of adversarial training is that if the model is robust enough, it should
be invariant to small perturbations of training examples that yield the
largest increase in the loss (adversarial examples [12]). The perturbation
$\bm{R}$ can be obtained as $\min_{\textbf{R},\|\textbf{R}\|\leq\epsilon}\log
P(\bm{y}|\textbf{I}+\textbf{R};\bm{\theta})$. In general, the calculation of
exact $\bm{R}$ is intractable especially for complicated models such as deep
networks. The linear approximation and $L_{2}$ norm box constraint can be used
for the calculation of perturbation as
$\bm{R}_{adv}=-\frac{\epsilon\bm{g}}{\|\bm{g}\|_{2}}$, where
$\bm{g}=\nabla_{\bm{I}}\log P(\bm{y}|\bm{I};\bm{\theta})$. For adversarial
FCN, the network predicts label of each pixel independently as
$p(\bm{y}|\bm{I};\bm{\theta})=\prod_{i}P(y_{i}|\bm{I};\bm{\theta})$. For
adversarial CRF as RNN, the prediction of network relies on mean field
approximation inference as
$p(\bm{y}|\bm{I};\bm{\theta})=\prod_{i}Q(y_{i}|\bm{I};\bm{\theta})$.

The adversarial training forces the model to fit examples with the worst
perturbation direction. The adversarial loss is

$\mathcal{L}_{adv}(\bm{\theta})=-\frac{1}{N}\sum_{n=1}^{N}\log
P(\bm{y}_{n}|\bm{I}_{n}+\bm{R}_{adv,n};\bm{\theta}).$ (7)

In training, the total loss is defined as the sum of adversarial loss and the
empirical loss based on training samples as

$\mathcal{L}(\bm{\theta})=\mathcal{L}_{adv}(\bm{\theta})-\frac{1}{N}\sum_{n=1}^{N}\log
P(\bm{y}_{n}|\bm{I}_{n};\bm{\theta})+\frac{\lambda}{2}\|\bm{\theta}\|^{2},$
(8)

where $\lambda$ is the $l_{2}$ regularization factor for $\bm{\theta}$,
$P(\bm{y}_{n}|\bm{I}_{n};\bm{\theta})$ is either mass probability prediction
in the FCN or a posteriori approximated by mean field inference in the CRF as
RNN for the $n$th image $\bm{I}_{n}$.

##  4 Experiments

We validate the proposed model on two most commonly used public mammographic
mass segmentation datasets: INbreast [13] and DDSM-BCRP dataset [14]. We use
the same ROI extraction and resize principle as [4, 6, 5]. Due to the low
contrast of mammograms, image enhancement technique is used on the extracted
ROI images as the first 9 steps in [15], followed by pixel position dependent
normalization. The preprocessing makes training converge quickly. We further
augment each training set by flipping horizontally, flipping vertically,
flipping horizontally and vertically, which makes the training set 4 times
larger than the original training set.

For consistent comparison, the Dice index metric is used to evaluate
segmentation performance and is defined as $\frac{2\times TP}{2\times
TP+FP+FN}$. For a fair comparison, we re-implement a two-stage model [6], and
obtain similar result (Dice index $0.9010$) on the INbreast dataset.

  * •

FCN is the network integrating a position priori into FCN (denoted as FCN 1 in
Table 1).

  * •

Adversarial FCN is FCN with adversarial training.

  * •

Joint FCN-CRF is the FCN followed by CRF as RNN with an end-to-end training
scheme.

  * •

Adversarial FCN-CRF is the Jointly FCN-CRF with end-to-end adversarial
training.

  * •

Multi-FCN, Adversarial multi-FCN, Joint multi-FCN-CRF, Adversarial multi-FCN-
CRF employ 4 FCNs with multi-scale kernels, which can be trained in an end-to-
end way using the last prediction.

The prediction of Multi-FCN, Adversarial multi-FCN is the average prediction
of the 4 FCNs. The configurations of FCNs are in Table 1. Each convolutional
layer is followed by $2\times 2$ max pooling. The last layers of the four FCNs
are all two $40\times 40$ transpose convolution kernels with soft-max
activation function. We use hyperbolic tangent activation function in middle
layers. The parameters of FCNs are set such that the number of each layer’s
parameters is almost the same as that of CNN used in the work [6]. We use Adam
with learning rate 0.003. The $\lambda$ is $0.5$ in the two datasets. The
$\epsilon$ used in adversarial training are $0.1$ and $0.5$ for INbreast and
DDSM-BCRP datasets respectively. Because the boundaries of masses on the DDSM-
BCRP dataset are smoother than those on the INbreast dataset, we use larger
perturbation $\epsilon$. For the CRF as RNN, we use 5 time steps in the
training and 10 time steps in the test phase empirically.

Table 1: Kernel sizes of sub-nets (#kernel$\times$#width$\times$#height). Net. | First layer | Second layer | Third layer
---|---|---|---
FCN 1 | $6\times 5\times 5$ |  $12\times 5\times 5$ conv. | $588\times 7\times 7$
FCN 2 | $9\times 4\times 4$ |  $12\times 4\times 4$ conv. | $588\times 7\times 7$
FCN 3 | $16\times 3\times 3$ |  $13\times 3\times 3$ conv. | $415\times 8\times 8$
FCN 4 | $37\times 2\times 2$ |  $12\times 2\times 2$ conv. | $355\times 9\times 9$
|  |  |
Table 2: Dices (%) on INbreast and DDSM-BCRP datasets. Methodology | INbreast | DDSM-BCRP
---|---|---
| Cardoso et al. [2]
---
88 | N/A
| Beller et al. [1]
---
N/A | 70
| Deep Structure Learning [4]
---
88 | 87
| TRW Deep Structure Learning [5]
---
89 | 89
| Deep Structure Learning + CNN [6]
---
90 | 90
FCN | 89.48 | 90.21
| Adversarial FCN
---
89.71 | 90.78
| Joint FCN-CRF
---
89.78 | 90.97
| Adversarial FCN-CRF
---
90.07 | 91.03
| Multi-FCN
---
90.47 | 91.17
| Adversarial multi-FCN
---
90.71 | 91.20
| Joint multi-FCN-CRF
---
90.76 | 91.26
| Adversarial multi-FCN-CRF
---
90.97 | 91.30
|  |

The INbreast dataset is a recently released mammographic mass analysis
dataset, which provides more accurate contours of lesion region and the
mammograms are of high quality. For mass segmentation, the dataset contains
116 mass regions. We use the first 58 masses for training and the rest for
test, which is of the same protocol as [4, 6, 5]. The DDSM-BCRP dataset
contains 39 cases (156 images) for training and 40 cases (160 images) for
testing [14]. After ROI extraction, there are 84 ROIs for training, and 87
ROIs for test. We compare schemes with other recently published mammographic
mass segmentation methods in Table 2.

Table 2 shows the CNN features provide superior performance on mass
segmentation, outperforming hand-crafted feature based methods [2, 1]. Our
enhanced FCN achieves 0.25% Dice index improvement than the traditional FCN on
the INbreast dataset. The adversarial training yields 0.4% improvement on
average. Incorporating the spatially structured learning further produces 0.3%
improvement. Using multi-scale model contributes the most to segmentation
results, which shows multi-scale features are effective for pixel-wise
classification in mass segmentation. Combining all the components together
achieves the best performance with 0.97%, 1.3% improvement on INbreast, DDSM-
BCRP datasets respectively. The possible reason for the improvement is
adversarial scheme eliminates the over-fitting.We calculate the p-value of
McNemar’s Chi-Square Test to compare our model with [6] on the INbreast
dataset. We obtain p-value $<0.001$, which shows our model is significantly
better than model [6].

To better understand the adversarial training, we visualize segmentation
results in Fig. 3. We observe that the segmentations in the second and fourth
rows have more accurate boundaries than those of the first and third rows. It
demonstrates the adversarial training improves FCN and FCN-CRF.

Fig. 3: Visualization of segmentation results using the FCN (first row),
Adversarial FCN (second row), Joint FCN-CRF (third row), Adversarial FCN-CRF
(fourth row) on the test sets of INbreast dataset. Each column denotes a test
sample. Red lines denote the ground truth. Green lines or points denote the
segmentation results. Adversarial training provides sharper and more accurate
segmentation boundaries than methods without adversarial training.

We further employ the prediction accuracy based on trimap to specifically
evaluate segmentation accuracy in boundaries [16]. We calculate the accuracies
within trimap surrounding the actual mass boundaries (groundtruth) in Fig. 4.
Trimaps on the DDSM-BCRP dataset is visualized in Fig. 2(b). From the figure,
accuracies of Adversarial FCN-CRF are 2-3 % higher than those of Joint FCN-CRF
on average and the accuracies of Adversarial FCN are better than those of FCN.
The above results demonstrate that the adversarial training improves the FCN
and Joint FCN-CRF both for whole image and boundary region segmentation.

(a)

(b)

Fig. 4: Accuracy comparisons among FCN, Adversarial FCN, Joint FCN-CRF and
Adversarial FCN-CRF in trimaps with pixel width $1$, $2$, $3$, $4$, $5$ on the
INbreast dataset (a) and the DDSM-BCRP dataset (b). The adversarial training
improves segmentation accuracy around boundaries.

##  5 Conclusion

In this work, we propose an end-to-end adversarial FCN-CRF network for
mammographic mass segmentation. To integrate the priori distribution of masses
and fully explore the power of FCN, a position priori is added to the network.
Furthermore, adversarial training is used to handle the small size of training
data by reducing over-fitting and increasing robustness. Experimental results
demonstrate the superior performance of adversarial FCN-CRF on two commonly
used public datasets.

## References

  * [1] M. Beller et al.,  “An example-based system to support the segmentation of stellate lesions,”  Springer, 2005.
  * [2] J. S Cardoso et al.,  “Closed shortest path in the original coordinates with an application to breast cancer,”  IJPRAI, 2015.
  * [3] W. Zhu et al,  “Deep multi-instance networks with sparse label assignment for whole mammogram classification,”  MICCAI, 2017.
  * [4] N. Dhungel et al.,  “Deep structured learning for mass segmentation from mammograms,”  in ICIP. IEEE, 2015.
  * [5] N. Dhungel et al.,  “Tree re-weighted belief propagation using deep learning potentials for mass segmentation from mammograms,”  in ISBI. IEEE, 2015.
  * [6] N. Dhungel et al.,  “Deep learning and structured prediction for the segmentation of mass in mammograms,”  in MICCAI, 2015.
  * [7] D. Yang et al.,  “Automatic liver segmentation using an adversarial image-to-image network,”  in MICCAI. Springer, 2017.
  * [8] J. Long, E. Shelhamer, and T. Darrell,  “Fully convolutional networks for semantic segmentation,”  in CVPR, 2015.
  * [9] P. Krähenbühl and V. Koltun,  “Efficient inference in fully connected crfs with gaussian edge potentials,”  in NIPS, 2011.
  * [10] S. Zheng et al.,  “Conditional random fields as recurrent neural networks,”  in ICCV, 2015.
  * [11] M. Jiang et al.,  “Mammographic mass segmentation with online learned shape and appearance priors,”  in MICCAI, 2016.
  * [12] C. Szegedy et al.,  “Intriguing properties of neural networks,”  ICLR, 2014.
  * [13] I. C Moreira et al.,  “Inbreast: toward a full-field digital mammographic database,”  Academic radiology, 2012.
  * [14] M. Heath et al.,  “Current status of the digital database for screening mammography,”  in Digital mammography. 1998.
  * [15] J. Ball et al.,  “Digital mammographic computer aided diagnosis using adaptive level set segmentation,”  in EMBI, 2007.
  * [16] P. Kohli et al.,  “Robust higher order potentials for enforcing label consistency,”  IJCV, 2009.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

SEMANTICS 2023 EU: 19th International Conference on Semantic Systems,
September 20-22, 2023, Leipzig, Germany

[orcid=0000-0001-5260-5181<EMAIL_ADDRESS>[orcid=0000-0003-3127-0815][1] [orcid=0000-0003-1337-2770][1]
[orcid=0009-0008-5245-6655][1] [orcid=0000-0002-1459-3754]
[orcid=0000-0003-0054-5003] [orcid=0000-0003-0762-8688]

[1]Corresponding author. [1]These authors contributed equally.

# Developing a Scalable Benchmark for Assessing Large Language Models in
Knowledge Graph Engineering

Lars-Peter Meyer  Johannes Frey  Kurt Junghanns  Felix Brei  Kirill Bulert
Sabine Gründer-Fahrer  Michael Martin  Institute for Applied Informatics,
Goerdelerring 9, 04109 Leipzig, Germany, https://infai.org Agile Knowledge
Engineering and Semantic Web (AKSW), https://aksw.org Leipzig University,
Institute for Informatics, Germany, https://www.uni-leipzig.de

(2023)

###### Abstract

As the field of Large Language Models (LLMs) evolves at an accelerated pace,
the critical need to assess and monitor their performance emerges. We
introduce a benchmarking framework focused on knowledge graph engineering
(KGE) accompanied by three challenges addressing syntax and error correction,
facts extraction and dataset generation. We show that while being a useful
tool, LLMs are yet unfit to assist in knowledge graph generation with zero-
shot prompting. Consequently, our _LLM-KG-Bench_ framework provides automatic
evaluation and storage of LLM responses as well as statistical data and
visualization tools to support tracking of prompt engineering and model
performance.

###### keywords:

Large Language Model Knowledge Graph Engineering Large Language Model
Benchmark

##  1 Introduction

Large Language Models (LLMs) hold the potential to change the way how we
interact with data and technology. Especially models like GPT-3 and GPT-4 have
shown proficient capabilities in solving textual assignments [1] and spawned a
wave of subsequent models and the field of _prompt engineering_.

But the fast evolution and rapidly growing landscape of different LLMs make it
challenging to keep track of their individual capabilities and to choose the
best model and best prompt for the job. There exist efforts on generic LLM
benchmarks (e.g. [2]). However, despite these advancements, the application
and (automated) assessment of LLMs in the context of knowledge graph
engineering (KGE) and the Semantic Web is still a highly under-explored area.
In response to this gap, this paper proposes a first LLM KGE benchmarking
framework LLM-KG-Bench111Repository: https://github.com/AKSW/LLM-KG-Bench or
doi:10.5281/zenodo.8251944 that follows our vision of an automated and
continuous evaluation platform for different tasks in KGE scenarios. A test of
the framework is presented by comparing three LLMs for three exemplary KGE
tasks.

##  2 Related Work

The utilization of an LLM in the semantic web domain benefits from its
capability to handle RDF-related syntaxes such as JSON-LD, Turtle and SPARQL.
A comprehensive amalgamation of LLMs and knowledge graphs (KGs) is described
in _Dagstuhl Seminar_ [3] and [4]. The _Knowledge Base Construction from Pre-
trained Language Models (LM-KBC) Challenge_ 222Website: https://lm-
kbc.github.io/challenge2023/ emphasises the relevance of this combination.

The basis of this study is [5], where ChatGPT’s use in knowledge graph
engineering is assessed. Impressive capabilities were revealed, suggesting two
conclusions: Firstly, such studies offer insight into LLMs’ potential and
limitations, aiding knowledge graph engineers. Secondly, comparing different
LLMs can lead to superior results by addressing inherent model issues.

Recognizing the potential of Large Language Models (LLMs) in knowledge graph
engineering, it’s vital to evaluate their performance across diverse tasks.
Google’s _Beyond the Imitation Game (BIG-bench) Benchmark_ 333Repository:
https://github.com/google/BIG-bench[2] and the Large Model Systems (LMSys)
leaderboard444Blogpost: https://lmsys.org/blog/2023-06-22-leaderboard/ are
community efforts that assess the performance of various models with regard to
a plethora of tasks. The Language Model Evaluation Harness555Repository:
https://github.com/EleutherAI/lm-evaluation-harness offers further testing of
generative language models on various evaluation tasks. However all of them
are not perfect for assessing an LLM’s use for KGE. They are missing KGE
specific scoring and do not evaluate scores relative to problem size. The size
seems to be relevant for KGE as KGs get quite big in relation to current LLMs
context sizes[5]. Acknowledging the existing appraoches limitations we
introduce the _LLM-KG-Bench_ framework.

##  3 The LLM-KG-Bench Framework

Our current (and ongoing) work presented in this paper is comprising the
design and implementation of the modular _LLM-KG-Bench_ framework1 for
benchmarking LLMs in the context of knowledge graph engineering. The main
focus is on automated evaluation procedures to allow for many repeated test
executions. The framework supports configurable task sizing, as prior work[5]
suggest the relevance of the LLM’s context size for KGE tasks.

As we aim for as much compatibility as possible, especially in the direction
of _BIG-bench_ 3, the _LLM-KG-Bench_ framework is organized around _benchmark
tasks_ and _LLM model connectors_ , glued together by some code for execution
organisation and result persistence. _LLM model connectors_ encapsulate the
connection to a specific LLM and offer the function `generate_text`. With this
function a benchmark task can send a prompt to LLM and get its answer.
_Benchmark tasks_ handle the LLM evaluation for a single task. In the function
`evaluate_model` they usually build a prompt or task description for the LLM,
hand this task over to a given LLM via an _LLM model connector_ and evaluate
the given answer. If necessary the _benchmark task_ could send additional
prompts to the LLM in the evaluation process. The evaluation results in score
values for the task specific defined score types and additional information.

Due to _LLM-KG-Bench_ ’s modularization, as shown in Figure 1, additional
benchmark tasks and LLM model connectors can be added by just adding
corresponding python class definitions. The framework supports basic result
visualization with the help of _seaborn_ 666Website:
https://seaborn.pydata.org/. The plots shown in Figure 2 are generated this
way.

Figure 1:  Basic _LLM-KG-Bench_ framework architecture. The Benchmark runner
takes a _benchmark configuration_ and organizes the repeated execution of
_benchmark tasks_ with _LLM model connectors_ and given size parameters.
Results generated get stored and can be visualized.

##  4 Initial Evaluation of the Framework with first Tasks

To test the _LLM-KG-Bench_ framework we added a couple of benchmark tasks and
evaluated three of the currently highest ranking LLMs at the LLMSYS Chatbot
Arena Leaderboard4. The test setup is detailed in Table 1(b).

Table 1: Setup used for testing the LLM-KG-Bench framework.

Model | Version
---|---
Claude | claude-1.3-100k
GPT 3.5 | gpt-3.5-turbo-0613 (4k)
GPT 4 | gpt-4-0613 (8k)
(a) LLMs evaluated

| Task a | Task b | Task c
---|---|---|---
Repetitions: | 20 x 1 size | 20 x 1 size | 20 x 8 sizes
plot type: | F1 measure | F1 measure | Mean error
plot generated: | Figure 2(a) | Figure 2(b) | Figure 2(c)
(b) test configuration per task

### Task a: Fixing of Errors in Turtle Files:

Turtle is a common serialization format for knowledge graphs. By asking the
LLMs to fix errors in given manipulated turtle files we test the knowledge of
turtle syntax as well as strict adhering to the given task and facts. One of
the scores calculated during evaluation is the F1 measure on parsable
normalized triples, comparing LLM’s answer with a perfect answer. A plot on
the F1 measure results for this task is shown in Figure 2(a). GPT-3.5 often
claims that file would be correct and returns no turtle. This accounts for the
high frequency of zero-value F1 scores. The answers given by Claude-1.3 and
GPT-4 score better.

### Task b: KG Creation from Factsheet Plaintext:

To evaluate knowledge extraction and modelling capabilities, we use a
plaintext excerpt of a PDF factsheet. The text describes various
specifications of a 3D printer in a key-value style, including usual
formatting irregularities associated with PDF extraction. We ask the model to
generate a Turtle file, that captures a subset of the information. The prompt
is engineered very specific with regard to which properties or ontologies have
to be used and how IRI identifiers and Literals should be represented.
Subsequently, we can evaluate the quality of a single response using the F1
measure, counting the set of parsable triples that (mis)match or are missing
compared to a manually curated reference document. Fig. 2(b) shows that the
GPT models outperform Claude in this task. While GPT4 has a better mean, due
to one very good response, it however replied often with unparseable content,
which in turn did not happen for GPT3.5, leading to a slightly better median
for that.

### Task c: Synthetic Dataset Generation:

Creating example data is an important task and the help of LLMs would be
highly appreciated. We created a basic test for this capability. We ask the
LLM to generate some synthetic dataset using well known `foaf:Person` and
`foaf:knows` with a varying number of desired objects and links in the final
KG. In the evaluation we used beside other scores the _persons_relative_error_
indicating the difference between the actual number person objects generated
and the number asked for. This value is normalized to be $=0$ if they match,
$>0$ if there are more persons than asked for and $<0$ if there are less
persons, with the special case of $-1$ meaning an empty graph. The results
presented in Figure 2(c) show a relation between the _persons_relative_error_
and the problem size, in this case number of person objects to generate.

(a) Turtle Fixing

(b) Fact Extraction

(c) Mean Error Dataset Generation

Figure 2: Subset of metrics from initial tasks. Shown are the F1 scores and
mean error of person count

##  5 Conclusion and Future Work

We showed that there is a need for measuring the knowledge graph engineering
capabilities of the rapidly evolving LLMs. We proposed and describe the novel
_LLM-KG-Bench_ framework for this task. A first evaluation of three high
ranking LLMs with first benchmarks shows the benefit of the automated
evaluation with the new framework.

The _LLM-KG-Bench_ framework is prepared to enable dialogs between benchmark
tasks and LLMs. It will be interesting to evaluate LLMs capabilities to fix
their answers with some feedback like e.g. error codes in improved or
additional tasks. We are looking forward to extending to more LLMs and more
benchmark tasks with the help of a bigger community.

###### Acknowledgements.

This work was partially supported by grants from the German Federal Ministry
for Economic Affairs and Climate Action (BMWK) to the CoyPu project
(01MK21007A) and KISS project (01MK22001A) as well as from the German Federal
Ministry of Education and Research (BMBF) to the projects StahlDigital
(13XP5116B) and KupferDigital (F13XP5119F).

## References

  * OpenAI [2023] OpenAI, Gpt-4 technical report, 2023\. arXiv:2303.08774.
  * Srivastava et al. [2023] A. Srivastava, et al.,  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,  Transactions on Machine Learning Research (2023). arXiv:2206.04615.
  * Groth et al. [2023] P. Groth, E. Simperl, M. van Erp, D. Vrandečić,  Knowledge graphs and their role in the knowledge engineering of the 21st century (dagstuhl seminar 22372) (2023). doi:10.4230/DAGREP.12.9.60.
  * Pan et al. [2023] S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, X. Wu, Unifying large language models and knowledge graphs: A roadmap, 2023. arXiv:2306.08302.
  * Meyer et al. [2023] L.-P. Meyer, C. Stadler, J. Frey, N. Radtke, K. Junghanns, R. Meissner, G. Dziwis, K. Bulert, M. Martin, Llm-assisted knowledge graph engineering: Experiments with chatgpt, 2023. arXiv:2307.06917, to appear in proceedings of AI-Tomorrow track on Data Week 2023 in Leipzig.

##  Appendix A Online Resources

  * •

_LLM-KG-Bench_ repository: https://github.com/AKSW/LLM-KG-Bench
or doi:10.5281/zenodo.8251944

  * •

experiment data: https://github.com/AKSW/LLM-KG-Bench-
Results/tree/main/2023-SEMANTICS_LLM-KGE-Bench-Results or
doi:10.5281/zenodo.8250646


# Numerical Study of Cleavable Bond-Modified Polyethylene for Circular Polymer
Design

Maria Ley-Flores  Pritzker School of Molecular Engineering, University of
Chicago  Archit Chabbi  Department of Bioengineering, Rice University
Riccardo Alessandri  Pritzker School of Molecular Engineering, University of
Chicago  Sam Marsden  Pritzker School of Molecular Engineering, University of
Chicago  Isabella Vettese  Department of Chemistry, University of Chicago
Stuart J. Rowan  Department of Chemistry, University of Chicago  Juan J. de
Pablo  Pritzker School of Molecular Engineering, University of Chicago
<EMAIL_ADDRESS>
###### Abstract

There is considerable interest in designing new polymeric materials with
built-in mechanisms for recycling. In this work, we present a systematic
exploration of several polyethylene-based polymers that contain strategically
incorporated cleavable bonds that are susceptible to chemical degradation. We
consider ten distinct telechelic functionalities across varying chain lengths:
ester, aromatic ester, anhydride, carbonate, urethane, siloxane, oxalate,
acetal, urea, and amide linkages. Our study is focused on the effects of
cleavable bonds on density, diffusion, enthalpy, and melting temperatures. We
also examine the crystallization process. These are key properties for polymer
processing and, by comparing them to those of linear polyethylene, we
establish a platform that will help guide future synthetic efforts towards the
design of sustainable plastics with performance comparable to that of
traditional, high-performance polyolefin-based materials.

Pritzker School of Molecular Engineering, University of Chicago Materials
Science Division, Argonne National Lab

##  1 Introduction

Plastic waste management has emerged as a pressing global challenge.
Polyolefins constitute the largest fraction of global plastics production.1,
2, 3 The majority of current plastics recycling efforts rely on mechanical
processes, often leading to performance losses and short usage cycles.4
Chemical recycling technologies have been developed 5, 6, 7, 8, 9 to break
down polymers into small, more versatile molecules that can be used for a wide
range of potential applications, including as feedstocks for new polymers.10
However, due to the high energetic cost associated with breaking down the
carbon-carbon $\sigma$ bond, progress in chemical recycling of polyolefins has
been particularly challenging. Promising new approaches have been developed
for enzymatic based degradation of plastics but, to date, such processes have
only been demonstrated for polyethylene terephthalate (PET)-based polymers.11

One approach to address the challenge of polyolefin recycling is to introduce
cleavable functional groups within the structure of linear polyethylene
(PE).10, 12, 13 Such long-spaced aliphatic polycondensates, also termed
”polyolefin-like materials,” offer the potential to open the door to polymer
circularity while displaying properties similar to those of commodity
plastics. In a previous review paper, Stempfle _et al_12 provide a
comprehensive account of the physical properties of the long-spaced aliphatic
condensates that have been studied to date. Some trends have been observed;
for example the melting point increases with increasing hydrocarbon chain
length between cleavable bonds in some of the long-spaced aliphatic
polycondensates considered in the literature. There are, however, disparities
in the reported values among various authors, which can probably be attributed
to differences in synthetic approaches and sample preparation methods.
Importantly, key properties such as the density or transport properties of the
materials, which are critically important to their performance in plastic
processing equipment, have not been examined before in a systematic manner.

Past work has established that molecular models provide a valuable tool to
bridge the gap between limited experimental data and property prediction for
PE-like materials. Harmandaris _et al_ ,14 for example, have computed the
self-diffusion coefficient of polyethylene melts to determine the crossover
from the Rouse to the entangled melt regimes. Mondello _et al_15 performed
non-equilibrium molecular dynamics (NEMD) simulations to calculate the
Newtonian viscosity of higher n-alkanes at 448 K. Ramos _et al_16 combined
computer simulations and experiments to describe the viscoelastic properties
of linear PE. However, although PE is undoubtedly the most studied polymer in
the polymer physics literature,17 there is no precedent for computational
studies of long-spaced aliphatic polycondensates.

In this work we present a computational study of the effect of cleavable bonds
in ten different linear polycondensates. We consider anhydride, amide,
urethane, urea, acetal, carbonate, ester, siloxane, and oxalate
functionalities. These functional groups are significantly high in energy, and
they lower the energy barriers for degradation via solvolysis. By using MD
simulations, we study the effect of chain length on the density, heat of
vaporization, and diffusivity in the melt. We also simulate the
crystallization process of the resulting semi-crystalline samples of the PE-
like candidates considered here. The properties of these polycondensates are
compared to those of linear polyethylene at different temperatures, thereby
providing a basis to evaluate their potential as sustainable alternatives in
the context of polymer processing.

##  2 Methods

### Molecular Models

In order to study a wide range of chain lengths, a united atom (UA) model was
used to simulate the PE and PE-like chains. In this model, the hydrogen atoms
are not explicitly simulated. Rather, they are grouped with the carbon to
which they are bonded into a single interaction site. Previous studies have
shown that united-atom models are better than all-atom models at reproducing
thermodynamic and transport properties for long hydrocarbons.18, 19 In
particular, we used the GROMOS 54A720 force field. All molecules were
parametrized using semiempirical QM calculations from the MOPAC software 21,
22 implemented by the Automated Topology Builder (ATB)23 server tool. Models
will be made available via the polyply 24 library. We modeled chains
containing an individual cleavable bond, surrounded by methylene segments from
20 to 80 united atoms. A total of 47 different chains were considered (4 chain
lengths for 10 functionalities, and 7 chain lengths for PE). To facilitate
comparison to furture expermental work, each of the PE-like chains were
modeled as symmetric molecules with the cleavable bond placed in the middle of
the chain (Figure 1).

Figure 1: Polymer chains studied in this work: polyethylene and cleavable
bond-modified polyethylenes featuring ten different functionalities at the
center of the backbone.

### Simulation Methods

We seek to describe the temperature and molecular weight dependence of the
density, heat of vaporization, and self-diffusion coefficient for PE-like
materials featuring different functionalities proposed in the literature. MD
simulations were performed with the GROMACS 25, 26, 27, 28 software version
2022.4. The equations of motion were integrated using the velocity Verlet
algorithm with a 5 fs time step.

The simulation box was generated by randomly inserting between 100 and 200
chains of a particular polymer in a cubic cell of 40 nm on a side. We used a
box with 100 chains for the longest chain and 200 chains for the smallest
molecules; these systems are sufficiently large to avoid finite-size effects.
After an initial energy minimization at 298 K to remove high-energy
configurations of the initial structures, the system was rapidly condensed at
523 K and 500 bar in the isothermal-isobaric (NPT) ensemble for 500 ps. The
last configuration was then used to equilibrate individual systems for 10 ns
at the corresponding pressure and temperature condition using the NPT
ensemble. The last 5 ns were used to compute the average density of the liquid
polymer melt.

In general, barostats, which manipulate positions through volume changes, can
interfere with the dynamics of the system, leading to biased transport
properties.29 Basconi and Shirts30 revealed that the estimates of self-
diffusivity in the microcanonical (NVE) ensemble are statistically
indistinguishable from those obtained by canonical (NVT) simulations with the
velocity-scaling thermostats employed in this work. Therefore, the final
configurations of the NPT simulations were used as starting configurations for
a 100 ns NVT run to compute the self-diffusion coefficient of each system by
means of the Einstein relation (Equation 1). Altogether, the simulations
spanned a temperature range from 373 K to 573 K. Whenever a temperature
control was necessary, the velocity-rescaling thermostat 31 was employed. For
controlling the pressure, the Parrinello-Rahman barostat 32, 33 was used.

$\lim_{t\to\infty}\langle\|\mathbf{r_{i}}(t)-\mathbf{r_{i}}_{0}\|^{2}\rangle_{i}=6Dt$
(1)

Additionally, three single-molecule simulations of each PE-like chain at 150
K, 373 K, 473 K, and 573 K were performed for 100 ns with an NVT ensemble to
calculate the average potential energy of the gas phase. This value was then
combined with the potential energy of the liquid simulations to estimate the
heat of vaporization by means of Equation 2:

$\Delta H_{vap}=E^{single}_{pot}-\frac{E^{bulk}_{pot}}{N_{mol}}+RT$ (2)

To examine the crystallization processes of the polymer chains, a cooling
process was performed from an initial amorphous melt at 423 K to a final
temperature of 150 K using a piece-wise linear function at a cooling rate of
0.05 K/ns. All annealing simulations were performed for the longest chains
consisting of a cleavable bond and 80 methylene units. The crystallization
temperature was determined by estimating the intersection point of the two
fitted-lines from plotting the density of the polymer against the temperature
(Figure 2).

Figure 2: Crystallization temperature of the polymers, determined by
estimating the intersection of two lines fitting the melt and the semi-
crystalline regions of the material from a plot of density vs temperature.

##  3 Results and discussion

### Density

Figure 3 shows the relative density of each modified polyethylene compared to
that of linear polyethylene at 373, 473, and 573 K. Overall, by increasing the
backbone length, the density converges to a value close to that of linear
polyethylene in almost all cases. The intermolecular interactions coming from
the single cleavable bond are not strong enough to produce a substantial
change in the relative density of modified PE melts. At higher temperatures,
however, the differences between the density of different chains become more
pronounced.

Figure 3: Density of cleavable bond-modified polyethylene normalized by the
density of linear polyethylene at 373 K, 473 K, and 573 K.

The aromatic ester-linked PE shows the most significant increase in density,
with deviations of 8.7%, 9.1%, and 9.4% at 373, 473, and 573 K, respectively,
for the longest chain length. In contrast, for the ester-linked PE, which also
features two ester groups separated by two methylene units, the differences in
density are less significant, and similar to the behavior of the chains
featuring carbonate, amide, anhydride, urea, urethane, siloxane, and oxalate
functionalities. These findings suggest that the introduction of the aromatic
ester linkages leads to a more efficient packing of the chains, due to the
$\pi$-$\pi$ stacking interactions associated with the two aromatic rings in
its structure.

After the aromatic-ester, the urethane linkage exhibits the highest deviations
in density with respect to polyethylene. The urethane group displays strong
hydrogen bonding interactions, which become more important at shorter chain
lengths. At longer backbone lengths, the urethane-linked PE reaches a density
that is closer to that of polyethylene, with a deviation of only 4.5% at the
highest temperature. The amide-linked PE, which has a structure similar to
that of the urethane, where the functionalities are spaced by six methylene
units, displays a less pronounced change in density as a function of backbone
length. This is likely due to the higher flexibility that stems from the extra
oxygens in the urethane-linked PE, and the higher strength and directionality
of the hydrogen bonding in urethane compared to those of the amide groups.

Acetal functional groups are not sufficiently electronegative to participate
in strong van der Waals bonding interactions. As a result, the acetal-linked
PE displays the closest behavior to PE. The siloxane linkage, although nearly
non-polar, displays a higher density compared to the acetal-linked PE. This is
likely due to the differences in molecular weight between the heavier silicon
atom and the carbon atom. For the shortest chain length, we can see that the
density of the siloxane-linked PE becomes even greater than that of the
oxalate-linked PE. Siloxane groups are expected to increase the flexibility of
the chain as the Si-O bond length is longer and rotation about Si-O bond is
extremely facile.34 More flexible chains can adopt a wider variety of
conformations that allow them to pack more efficiently and reduce free volume.
This is especially true at higher temperatures, where chains have increased
mobility.

At lower temperatures, the shortest chains are more densely packed than the
reference polyethylene with the same backbone length with differences close to
30% for the case of the aromatic ester-linked PE. With the exception of the
aromatic ester-linked PE, all chains approach the density of polyethylene as
the number of backbone atoms increases with differences of less than 5%. For
the aromatic ester-linked PE, the converged density is close to 9% higher than
the predicted value for the reference polyethylene.

### Heat of Vaporization

While the density relates to how tightly the polymer chains are packed, the
heat of vaporization provides a measure of the energy required to overcome
intermolecular forces that hold molecules together in the melt phase. In
general, stronger intermolecular interactions in the form of hydrogen bonding,
van der Waals forces, and dipole-dipole interactions result in higher values
for the heat of vaporization.

Figure 4 shows the heat of vaporization of the cleavable bond-modified PE
relative to PE at 373, 473, an 573 K. Similar to the relative density, the
heat of vaporization of almost all modified PE-like molecules converges to a
value close to the heat of vaporization of the unmodified PE as the backbone
length increases. Likewise, the deviations increase as the temperature
increases, leading to values closer to those of polyethylene at lower
temperatures and longer backbone lengths.

Figure 4: Relative heat of vaporization for cleavable bond-modified
polyethylene with respect to linear polyethylene at 373 K, 473 K, 573 K.

Just as for the density, the biggest deviations come from the aromatic ester-
modified PE. At the shortest chain length, the heat of vaporization of the
modified PE is close to 90% greater than the one of the unmodified PE. As the
chain length increases, this deviation reduces to 21%, compared to the 9.4%
increase in density (Table 1). This shows how the aromatic ester linkage
significantly enhances the strength of the intermolecular interactions through
dipole-dipole interactions and $\pi$-$\pi$ stacking without considerably
affecting the overall packing efficiency.

Polymer |  $\Delta H_{vap}/H_{vap,PE}$ (%) | $\Delta\rho/\rho_{{}_{PE}}$
---|---|---
Aromatic Ester-linked PE | 21.0% | 8.7%
Urethane-linked PE | 11.6% | 4.4%
Amide-linked PE | 9.7% | 3.01%
Oxalate-linked PE | 4.8% | 3.4%
Ester-linked PE | 4.6% | 3.2%
Urea-linked PE | 4.2% | 1.9%
Carbonate-linked PE | 4.08% | 2.7%
Siloxane-linked PE | 4.0% | 2.6%
Anhydride-linked PE | 2.4% | 2.5%
Acetal-linked PE | 0.8% | 1.5%
Table 1: Heat of vaporization and density increment relative to polyethylene
at 373 K and a PE-backbone length of 80 united atoms.

The modified PE with the next most significant increment is the urethane-
linked PE with a value 11.6% higher than that of PE. Next, at 373 K and for
the longest backbone length, the amide-linked PE closely follows the trend of
the urethane-liked PE with an increment of 9.7%, almost 3 times higher than
the increment in density. Both the urethane-linked PE and amide-linked PE
share a very similar structure (Figure 1) with two functional groups separated
by 6 methylene units in the center of its structure. Both urethanes and amides
are able to form strong hydrogen bonds. The major difference lies in the
additional oxygen in the urethane’s chemical structure.

The oxalate, ester, urea, carbonate, and siloxane-linked PE show a very
similar increment of around 4% compared to polyethylene. Both the oxalate and
the ester linkage contain two ester groups in their chemical structure. The
ester linkage has two methylene units spacing their ester groups, whereas the
oxalate linkage has no spacing between them. Even though the oxalate’s rigid
structure may facilitate better packing, the results show very insignificant
differences between the heat of vaporization and the density of these two
cleavable bonds. This implies that when the spacers between functionalities
are short enough, the differences in the strength of intermolecular
interactions and packing are negligible. Although the carbonate-linked PE
would be expected to have lower values for heat of vaporization than an
oxalate group or an ester-linkage, urea groups often form stronger hydrogen
bond interactions due to the presence of two hydrogen bond donors on the
nitrogen atoms. However, the results show that having two ester groups are
seemingly equivalent to having an urea group in terms of the strength of
intermolecular interactions. Surprisingly, the siloxane group has a relative
increase in heat of vaporization comparable to that of the urea group. Since
siloxanes do not generally form strong hydrogen bonds, this may seem
counterintuitive. However, this could be explained by the polarizability of
silicon atoms, which leads to an increase in London dispersion forces. The
semi-polar Si-O bonds within a non-polar hydrocarbon could induce local
dipoles and therefore increase the value for the heat of vaporization.

The anhydride-linked PE and the acetal-linked PE have the closest agreement
with the unmodified PE. The anhydride, containing two carbonyl groups,
enhances intermolecular forces through dipole-dipole interactions compared to
the unmodified PE. For acetals, the presence of oxygen also introduces some
lower dipole-dipole interactions. However, at longer chain lengths such
contributions become insignificant.

Figure 5: Self-diffusion coefficient for cleavable bond-modified polyethylene
at 373, 473, and 573 K as a function of number of PE backbone united atoms and
molecular weight.

### Diffusion

Figure 5 shows diffusion coefficients for all cleavable bond-modified PEs at
373, 473, and 573 K, for PE backbone lengths up to 80 united atoms. At
elevated temperatures, 573 K, for most of the PE-like melt, all curves
collapse into a single line. At such temperatures, the segmental motion of the
chains is less influenced by the specific nature of the modifying groups. The
thermal energy is sufficiently high to overcome different intermolecular
forces present in the modified PEs, leading to an increment of chain
flexibility and similar diffusion values. As temperature decreases, the
differences between diffusion coefficients of molecules with the same
molecular weight become more important.

At lower temperatures, the structural features of each of the cleavable bond-
modified PEs affect the diffusion of the molecules. The chains with more
hindrances to molecular motion due to stronger intermolecular forces, such as
the aromatic ester-linked and urethane-linked PE, exhibit lower self-diffusion
coefficients at lower temperatures. Conversely, polymers with more flexible
structures like the unmodified linear PE and the acetal-linked PE exhibit
relatively higher self-diffusion coefficients under the same conditions. Lower
diffusion values and differences in slopes of diffusion as a function of
molecular weight indicate that the chemical structure of the cleavable bonds
does have a significant impact on the physical behavior of the polymer closer
to the glass transition or crystallization temperatures.

### Crystallization Temperature

Figure 6 shows the evolution of the density as a function of temperature for
the different cleavable bond-modified polyethylenes, obtained through an
annealing process. A sudden drop in density indicates a phase transition, in
this case crystallization. A summary of the crystallization temperatures for a
cooling rate of 0.05 K/ns is shown in Figure 7. Note that the experimental
cooling rates accessible to simulations are several orders of magnitude lower
than those used in experiments.

All polymers, except for the aromatic ester-linked PE, display a lower
crystallization temperature than PE. The presence of the bulky aromatic ester
groups allows the material to retain its semi-crystalline state over a wider
temperature range. Since most of the hydrogen bonding linkages have lower
crystallization temperatures, the higher value is likely explained by the
strength and rigidity of $\pi$-$\pi$ stacking interactions imparted by the
aromatic rings, which facilitate a more ordered packing and help nucleate
crystallization early on.

Figure 8 shows snapshots of the semi-crystalline structures resulting from the
annealing of PE ordered from lowest to highest crystallization temperature.
From the annealed structures, we can observe that even the semi-polar siloxane
linkages exhibit a tendency to aggregate in the resulting semi-crystalline
structures. In a previous study, Menges _et al_35 examined the crystalline
structure of a long-chain aliphatic polyester featuring cleavable bonds with
the same ester linkage considered here. The structures obtained at 150 K are
in good qualitative agreement with the structures reported from X-ray
Scattering and Nuclear Magnetic Resonance. The semi-crystalline behavior
observed here supports the findings of these authors regarding intermolecular
interactions: a few functional groups introduced into a purely aliphatic
polymer are sufficient to control the morphology of polyethylene-like
polymers.

Figure 6: Crystallization temperatures for all cleavable bond-modified PE
chains (blue) and unmodified PE (red) at a cooling rate of 0.05 K/ns. Figure
7: Crystallization temperatures for all cleavable bond-modified PE chains
(blue) and unmodified PE (red) at a cooling rate of 0.05 K/ns.

Figure 8: Cleavable bond-modified PE and PE structures at 150 K obtained
through an annealing process at a rate of 0.05 K/ns.

##  4 Conclusions

In this work, we compared the effect of different cleavable bonds on the
structure of various PE chains. As expected, as the number of methylene units
increase in the chain, all properties converge to those of polyethylene. A
notable exception is provided by for the aromatic ester linkage, which
exhibits a density that is considerably and consistently higher than that of
unmodified PE. Such a higher density could be turned to an advantage in
several applications of polyethylene. More generally, in all cases the
observed differences in density are sufficiently important to lead to
differences in processability, particularly as the molecular weight between
cleavable bonds decreases. In this regard, the acetal linkages appear to
exhibit the smallest disruption to the underlying PE backbone, whereas the
aromatic ester linkage is the most disruptive. Similarly, at elevated
temperatures the diffusion coefficients of all polymers are similar (except
the aromatic ester linked materials), and are proportional to the molecular
weight. At lower temperatures, the differences in diffusion coefficient become
more pronounced. The aromatic ester-linked PE has the closest crystallization
temperature to polyethylene and also the highest crystallization temperature
of all structures studied in this work. For all the semi-crystalline
structures considered here, we find that the cleavable bonds are excluded from
the crystalline domains of the material, leading to the formation of
aggregates within the material. We speculate, however, that in the case of
aromatic-ester linkages the presense of $\pi$-$\pi$ stacking interactions
helps nucleate the formation of ordered structures that facilitate
crystallization.

This work was supported as part of the Center for Plastics Innovation, an
Energy Frontier Research Center funded by the U.S. Department of Energy,
Office of Science, Basic Energy Sciences at the University of Chicago under
Grant Number DE-SC0021166. R.A. acknowledges support by the Dutch Research
Council (NWO Rubicon 019.202EN.028).

## References

  * Giacovelli 2018 Giacovelli, C. _Single-use plastics: a roadmap for sustainability_ ; International Environmental Technology Centre, 2018
  * Geyer et al. 2017 Geyer, R.; Jambeck, J. R.; Law, K. L. Production, use, and fate of all plastics ever made. _Science advances_ 2017, _3_ , e1700782
  * Jambeck et al. 2015 Jambeck, J. R.; Geyer, R.; Wilcox, C.; Siegler, T. R.; Perryman, M.; Andrady, A.; Narayan, R.; Law, K. L. Plastic waste inputs from land into the ocean. _Science_ 2015, _347_ , 768–771
  * Ragaert et al. 2017 Ragaert, K.; Delva, L.; Van Geem, K. Mechanical and chemical recycling of solid plastic waste. _Waste management_ 2017, _69_ , 24–58
  * Vollmer et al. 2020 Vollmer, I.; Jenks, M. J.; Roelands, M. C.; White, R. J.; van Harmelen, T.; de Wild, P.; van Der Laan, G. P.; Meirer, F.; Keurentjes, J. T.; Weckhuysen, B. M. Beyond mechanical recycling: Giving new life to plastic waste. _Angewandte Chemie International Edition_ 2020, _59_ , 15402–15423
  * Lopez et al. 2017 Lopez, G.; Artetxe, M.; Amutio, M.; Bilbao, J.; Olazar, M. Thermochemical routes for the valorization of waste polyolefinic plastics to produce fuels and chemicals. A review. _Renewable and Sustainable Energy Reviews_ 2017, _73_ , 346–368
  * Epps III et al. 2021 Epps III, T. H.; Korley, L. T.; Yan, T.; Beers, K. L.; Burt, T. M. Sustainability of synthetic plastics: Considerations in materials life-cycle management. _JACS Au_ 2021, _2_ , 3–11
  * Hinton et al. 2022 Hinton, Z. R.; Talley, M. R.; Kots, P. A.; Le, A. V.; Zhang, T.; Mackay, M. E.; Kunjapur, A. M.; Bai, P.; Vlachos, D. G.; Watson, M. P.; others Innovations toward the valorization of plastics waste. _Annual Review of Materials Research_ 2022, _52_ , 249–280
  * Zhao et al. 2022 Zhao, X.; Korey, M.; Li, K.; Copenhaver, K.; Tekinalp, H.; Celik, S.; Kalaitzidou, K.; Ruan, R.; Ragauskas, A. J.; Ozcan, S. Plastic waste upcycling toward a circular economy. _Chemical Engineering Journal_ 2022, _428_ , 131928
  * Coates and Getzler 2020 Coates, G. W.; Getzler, Y. D. Chemical recycling to monomer for an ideal, circular polymer economy. _Nature Reviews Materials_ 2020, _5_ , 501–516
  * Tournier et al. 2020 Tournier, V.; Topham, C.; Gilles, A.; David, B.; Folgoas, C.; Moya-Leclair, E.; Kamionka, E.; Desrousseaux, M.-L.; Texier, H.; Gavalda, S.; others An engineered PET depolymerase to break down and recycle plastic bottles. _Nature_ 2020, _580_ , 216–219
  * Stempfle et al. 2016 Stempfle, F.; Ortmann, P.; Mecking, S. Long-chain aliphatic polymers to bridge the gap between semicrystalline polyolefins and traditional polycondensates. _Chemical reviews_ 2016, _116_ , 4597–4641
  * Häußler et al. 2021 Häußler, M.; Eck, M.; Rothauer, D.; Mecking, S. Closed-loop recycling of polyethylene-like materials. _Nature_ 2021, _590_ , 423–427
  * Harmandaris et al. 2003 Harmandaris, V.; Mavrantzas, V.; Theodorou, D.; Kröger, M.; Ramirez, J.; Öttinger, H. C.; Vlassopoulos, D. Crossover from the rouse to the entangled polymer melt regime: signals from long, detailed atomistic molecular dynamics simulations, supported by rheological experiments. _Macromolecules_ 2003, _36_ , 1376–1387
  * Mondello et al. 1998 Mondello, M.; Grest, G. S.; Webb III, E. B.; Peczak, P. Dynamics of n-alkanes: Comparison to Rouse model. _The Journal of chemical physics_ 1998, _109_ , 798–805
  * Ramos et al. 2008 Ramos, J.; Vega, J. F.; Theodorou, D. N.; Martinez-Salazar, J. Entanglement relaxation time in polyethylene: Simulation versus experimental data. _Macromolecules_ 2008, _41_ , 2959–2962
  * Ramos et al. 2018 Ramos, J.; Vega, J. F.; Martínez-Salazar, J. Predicting experimental results for polyethylene by computer simulation. _European Polymer Journal_ 2018, _99_ , 298–331
  * da Silva et al. 2020 da Silva, G. C.; Silva, G. M.; Tavares, F. W.; Fleming, F. P.; Horta, B. A. Are all-atom any better than united-atom force fields for the description of liquid properties of alkanes? _Journal of Molecular Modeling_ 2020, _26_ , 1–17
  * da Silva et al. 2022 da Silva, G. C.; Silva, G. M.; Tavares, F. W.; Fleming, F. P.; Horta, B. A. Are all-atom any better than united-atom force fields for the description of liquid properties of alkanes? 2. A systematic study considering different chain lengths. _Journal of Molecular Liquids_ 2022, _354_ , 118829
  * Schuler et al. 2001 Schuler, L. D.; Daura, X.; Van Gunsteren, W. F. An improved GROMOS96 force field for aliphatic hydrocarbons in the condensed phase. _Journal of computational chemistry_ 2001, _22_ , 1205–1218
  * Dewar and Thiel 1977 Dewar, M. J.; Thiel, W. Ground states of molecules. 38. The MNDO method. Approximations and parameters. _Journal of the American Chemical Society_ 1977, _99_ , 4899–4907
  * Stewart 2013 Stewart, J. J. Optimization of parameters for semiempirical methods VI: more modifications to the NDDO approximations and re-optimization of parameters. _Journal of molecular modeling_ 2013, _19_ , 1–32
  * Stroet et al. 2018 Stroet, M.; Caron, B.; Visscher, K. M.; Geerke, D. P.; Malde, A. K.; Mark, A. E. Automated topology builder version 3.0: Prediction of solvation free enthalpies in water and hexane. _Journal of chemical theory and computation_ 2018, _14_ , 5834–5845
  * Grünewald et al. 2022 Grünewald, F.; Alessandri, R.; Kroon, P. C.; Monticelli, L.; Souza, P. C.; Marrink, S. J. Polyply; a python suite for facilitating simulations of macromolecules and nanomaterials. _Nature Communications_ 2022, _13_ , 68
  * Van Der Spoel et al. 2005 Van Der Spoel, D.; Lindahl, E.; Hess, B.; Groenhof, G.; Mark, A. E.; Berendsen, H. J. GROMACS: fast, flexible, and free. _Journal of computational chemistry_ 2005, _26_ , 1701–1718
  * Pronk et al. 2013 Pronk, S.; Páll, S.; Schulz, R.; Larsson, P.; Bjelkmar, P.; Apostolov, R.; Shirts, M. R.; Smith, J. C.; Kasson, P. M.; Van Der Spoel, D.; others GROMACS 4.5: a high-throughput and highly parallel open source molecular simulation toolkit. _Bioinformatics_ 2013, _29_ , 845–854
  * Berendsen et al. 1995 Berendsen, H. J.; van der Spoel, D.; van Drunen, R. GROMACS: A message-passing parallel molecular dynamics implementation. _Computer physics communications_ 1995, _91_ , 43–56
  * Abraham et al. 2015 Abraham, M. J.; Murtola, T.; Schulz, R.; Páll, S.; Smith, J. C.; Hess, B.; Lindahl, E. GROMACS: High performance molecular simulations through multi-level parallelism from laptops to supercomputers. _SoftwareX_ 2015, _1_ , 19–25
  * Maginn et al. 2019 Maginn, E. J.; Messerly, R. A.; Carlson, D. J.; Roe, D. R.; Elliot, J. R. Best practices for computing transport properties 1. Self-diffusivity and viscosity from equilibrium molecular dynamics [article v1. 0]. _Living Journal of Computational Molecular Science_ 2019, _1_ , 6324–6324
  * Basconi and Shirts 2013 Basconi, J. E.; Shirts, M. R. Effects of temperature control algorithms on transport properties and kinetics in molecular dynamics simulations. _Journal of chemical theory and computation_ 2013, _9_ , 2887–2899
  * Bussi et al. 2007 Bussi, G.; Donadio, D.; Parrinello, M. Canonical sampling through velocity rescaling. _The Journal of chemical physics_ 2007, _126_
  * Nosé and Klein 1983 Nosé, S.; Klein, M. Constant pressure molecular dynamics for molecular systems. _Molecular Physics_ 1983, _50_ , 1055–1076
  * Parrinello and Rahman 1981 Parrinello, M.; Rahman, A. Polymorphic transitions in single crystals: A new molecular dynamics method. _Journal of Applied physics_ 1981, _52_ , 7182–7190
  * Sun and Rigby 1997 Sun, H.; Rigby, D. Polysiloxanes: ab initio force field and structural, conformational and thermophysical properties. _Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy_ 1997, _53_ , 1301–1323
  * Menges et al. 2007 Menges, M.; Penelle, J.; Le Fevere de Ten Hove, C.; Jonas, A. M.; Schmidt-Rohr, K. Characterization of long-chain aliphatic polyesters: Crystalline and supramolecular structure of PE22, 4 elucidated by X-ray scattering and nuclear magnetic resonance. _Macromolecules_ 2007, _40_ , 8714–8725