# Neural Data-to-Text Generation with LM-based Text Augmentation

†Ernie Chang, $\odot$ Xiaoyu Shen , †Dawei Zhu, †Vera Demberg, $\otimes$ Hui
Su
†Dept. of Language Science and Technology, Saarland University
<EMAIL_ADDRESS>
$\odot$ Amazon Alexa AI, Berlin
$\otimes$ Pattern Recognition Center, Wechat AI, Tencent Inc, China
Work done prior to joining Amazon.

###### Abstract

For many new application domains for data-to-text generation, the main
obstacle in training neural models consists of a lack of training data. While
usually large numbers of instances are available on the data side, often only
very few text samples are available. To address this problem, we here propose
a novel few-shot approach for this setting. Our approach automatically
augments the data available for training by (i) generating new text samples
based on replacing specific values by alternative ones from the same category,
(ii) generating new text samples based on GPT-2, and (iii) proposing an
automatic method for pairing the new text samples with data samples. As the
text augmentation can introduce noise to the training data, we use cycle
consistency as an objective, in order to make sure that a given data sample
can be correctly reconstructed after having been formulated as text (and that
text samples can be reconstructed from data).

On both the E2E and WebNLG benchmarks, we show that this weakly supervised
training paradigm is able to outperform fully supervised seq2seq models with
less than $10\%$ annotations. By utilizing all annotated data, our model can
boost the performance of a standard seq2seq model by over 5 BLEU points,
establishing a new state-of-the-art on both datasets.

##  1 Introduction

Neural data-to-text generation has been the subject of much recent research.
The task aims at transforming _source-side_ structured data into _target-side_
natural language text Reiter and Dale (2000); Barzilay and Lapata (2005).
While neural end-to-end systems afford the advantage of easy adaptability
Lebret et al. (2016); Wiseman et al. (2017), huge amounts of data-text pairs
are still necessary to perform on par with their rule-based counterparts van
der Lee et al. (2018). This makes using neural systems less appealing:
oftentimes, in-domain text samples are not readily available, and there is a
high cost to collecting in-domain texts which fit the data samples, and
annotating these texts with the data labels – the cost for collecting this
data might hence even outweigh the efforts of designing a rule-based system
Gkatzia (2016). The goal of this work is to improve the performance of neural
data-to-text models in scenarios where only very few text samples exist (we
assume that these text samples are paired with corresponding data samples). We
aim to answer how we can make the most of the scarce annotations, together
with large amounts of unlabelled data, in order to push the limit of the
neural data-to-text models. Figure 1 illustrates the scenario.

Figure 1:  Few-shot scenario: The model is expected to learn data-to-text
generation with few labeled instances (i.e. table-text pairs). The example is
taken from the E2E dataset.

To address the limited-data challenge, we propose a simple yet effective way
of augmenting the text side with the pretrained language model (LM) GPT-2
Radford et al. (2019). Unlike other text augmentation work employed in data-
to-text generation systems Freitag and Roy (2018); Agarwal et al. (2018), our
proposal assumes little to no domain-dependent heuristics. It consists of two
steps: (1) information augmentation by slot-value replacement and (2) LM
augmentation by GPT-2 generation.

Once we have augmented the set of text samples, we are essentially in a
similar setting as previously proposed semi-supervised approaches to data-to-
text generation Schmitt and Schütze (2019); Qader et al. (2019); Su et al.
(2020b), which assume the presence of vast amounts of unpaired data and text
instances. These approaches exploit a cycle consistency objective in order to
learn a pairing for the data samples. The cycle consistency objective tries to
make sure that data samples can be reconstructed correctly from their textual
formulations, and similarly that texts can be reconstructed after having been
parsed into a data representation.

As the automatically generated text samples from GPT-2 might be very noisy and
not pair well with data samples, we align each augmented text sample with its
most similar unlabeled data sample, as defined in their encoded vector space.
This idea is inspired by recent work on representation matching in MT Artetxe
and Schwenk (2019); Ruiter et al. (2019). To ensure good quality of the
training data, only pairs above a certain similarity threshold $\epsilon$ are
retained as pseudo pairs for training. The quality of the pseudo pairs will
gradually improve as the encoder improves in the training process. In return,
the learning of the encoder will also be facilitated with the improved quality
of pseudo pairs as a virtuous cycle.

On two data-to-text benchmarks E2E Novikova et al. (2017) and WebNLG Gardent
et al. (2017), we show that our LM-augmented weakly supervised model succeeds
on outperforming fully supervised seq2seq model, though utilizing less than
$10\%$ of the data annotations. It even outperforms previous work which
_additionally has access to all unpaired text samples_. When trained with full
data annotations, it is able to boost the model performance by up to 5 BLEU
points, establishing a new state-of-the-art on both datasets.

In summary, this work makes the following contributions:

  1. 1.

We study the few-shot data-to-text scenario where, unlike previous works, no
further target-side text is available.

  2. 2.

We present an effective way of automatically augmenting target text by
resorting to the pretrained LM GPT-2.

  3. 3.

We propose utilizing the augmented text by a combination of cycle consistency
and representation matching. The resulting model outperforms standard seq2seq
model with less than 10% data annotations.

  4. 4.

The proposed model is shown to be complementary with current seq2seq
pretraining techniques, and can offer orthogonal improvements when combining
both.

##  2 Related Work

Building neural data-to-text systems with few paired samples (but a large set
of unpaired samples) has been a hot research topic recently. Most works adopt
the idea of cycle consistency Zhu et al. (2017), which has been used in many
text generation tasks like machine translation Artetxe et al. (2017); Lample
et al. (2017) and style transfer Prabhumoye et al. (2018); Subramanian et al.
(2018). Schmitt and Schütze (2019); Qader et al. (2019); Su et al. (2020b);
Chang et al. (2020b, 2021a, 2021b) applied this idea to the task of data-to-
text generation and reported promising results. Ma et al. (2019) separate the
generation process into few-shot content selection and surface realization
components and learn them separately. Nonetheless, all of these approaches
assume the existence of _huge quantity of unpaired text samples_ , which, as
we mentioned, is an unrealistic assumption for the task of data-to-text
generation. Freitag and Roy (2018) proposes to reconstruct usable sequences
re-written from data with rules for unsupervised data-to-text generation.
Unfortunately, designing these rules require efforts similar to building a
template-based system. Budzianowski and Vulić (2019); Chen et al. (2020); Peng
et al. (2020) tackle the few-shot challenge by finetuning a pretrained LM to
incorporate prior knowledge from general-domain text or data-text pairs. We
show that our technique is complementary with them and can offer orthogonal
improvements when combining both.

##  3 Problem Formulation

We represent the data samples as D and the text samples as T. In our work, we
do not restrict the format of the data. Each $d\in D$ can be a set of key-
value pairs, as in Figure 1, or in form of RDF triples as in Gardent et al.
(2017). Each text $t\in T$ consists of a sequence of words. In few-shot
settings, we are assumed to have (1) $k$ labeled pairs $(D_{L},T_{L})$ and (2)
large quantities of unlabeled data $D_{U}$ where $|D_{U}|\gg k>0$111We force
$k>0$ as we believe a reasonable generation system needs a least a few
demonstrations of the annotation.. This, we believe, is a more realistic
setting as unlabeled data are usually abundant and also can be easily
fabricated from predefined schemata. Notably, we assume no access to outside
resources containing in-domain text. The $k$ annotations are all we know about
the text side.

##  4 Approach

In this section, we first explain our proposed new method for text sample
augmentation, and then discuss methods to remove noise and automatically align
the data by elaborating on the ideas of cycle consistency and representation
matching. Finally, we summarize the approach and present the detailed
algorithm.

Figure 2: Depiction of _text augmentation_ and _representation matching_. Each
text sample first goes through information augmentation by slot-value
replacement, then passed to GPT-2 with iterative conditional generation. The
augmented text samples are paired with the most similar data from the corpus
with a threshold cutoff.

###  4.1 Text Augmentation

To mitigate the paucity of the set of text samples $T$, we propose a pipeline
approach to augment the text samples by (1) information augmentation and (2)
LM augmentation.

####  4.1.1 Information Augmentation

We generate additional text samples by performing slot-value replacements. As
many data values are exactly copied to the text samples, these copied
information can be easily detected and replaced with other values (for the
same slot type) to enrich the information space of the text samples. This can
be considered as a simplified version of traditional methods of template
mining where key words are extracted to construct templates Kondadadi et al.
(2013); Oya et al. (2014). An example is shown in Figure 2. Each text sample
is augmented with 10 more distinct text samples or with all possible values
being replaced.

The slot-value replacement is efficient to implement. However, it can only
detect identical values and augment text with the same combinatorial patterns
as the few-shot annotations. To enrich the linguistic realizations of text
sentences and enable new combinations of information, we further propose a LM
augmentation approach using GPT-2.

####  4.1.2 LM Augmentation

GPT-2 Radford et al. (2019) is a language model pretrained on the collected
WebText. It has demonstrated remarkable zero-shot multitask adaptability by
simply feeding the input of each task into the LM and continuing to generate
words. People have also also shown that GPT-2 is able to improve
classification tasks via in-domain text augmentation Papanikolaou and
Pierleoni (2020); Sun et al. (2020). We use a similar technique by first fine-
tuning GPT-2 in the few-shot annotations Wolf et al. (2019), and then applying
it to produce synthetic text through an iterative conditional generation
process: With initial seeds being samples of $T_{L}$ plus new samples from
information augmentation, the LM iteratively conditions on the previous output
sentence to generate in-domain text222We adopt the Top-$k$ random sampling
setting with $k=2$ to encourage diversity and reduce repetition Radford et al.
(2019). Each synthetic sentence is pruned if it (1) is shorter than 5 words or
(2) contains only special tokens. The iterative generation is terminated when
all tokens in the initial seeds are covered or if the maximum of 100 runs is
reached. All the unpruned synthetic text samples are added into the space of
$T$ to benefit the learning direction of $t\rightarrow d^{\prime}\rightarrow
t$ and $\tilde{t}\rightarrow t$. Figure 2 depicts the generation process of
GPT-2.

In practice, obtaining clean in-domain text requires extreme efforts of
designing heuristic rules. Nonetheless, the synthetic text from GPT-2 makes
decent sense and can already provide useful signals to drive the learning
process.

###  4.2 Cycle Consistency

The core idea of encouraging cycle consistency is that starting from one
sample in a domain, the model first maps it into the other domain, then maps
it back He et al. (2016). The resulting sample should be identical to the
original sample. Specifically, let $p_{\theta}(t|d)$ be the probability
distribution to map a data sample $d$ to its corresponding text $t$, and
$p_{\phi}(d|t)$ be the probability distribution to map text back to data.
Starting from a data sample $d\in D$, its objective is:

$\begin{split}\max_{\phi}\mathbb{E}_{d\sim p(D)}\log
p_{\phi}(d|t^{\prime});\;t^{\prime}\sim p_{\theta}(t|d)\end{split}$ (1)

which basically ensures the consistency in the direction of $d\rightarrow
t^{\prime}\rightarrow d$. Note that only $p_{\phi}$ is updated in this
direction and $p_{\theta}$ serves only as as an auxiliary function to provide
pseudo samples $t^{\prime}$ from $d$. Though it is also possible to update
$\theta$ at the same time through tricks like Gumbel-softmax Jang et al.
(2016) or REINFORCE Williams (1992), we find it did not lead to better
performance, yet complicated the training. Similar observations have been made
in Lample et al. (2018); He et al. (2020); Garcia et al. (2020).

Similarly, starting from a text $t\in T$, the objective is to ensure the
consistency in the direction of $t\rightarrow d^{\prime}\rightarrow t$:333In
the MT community, the equivalent step is usually called _back translation_
Sennrich et al. (2016); Lample et al. (2018).

$\begin{split}\max_{\theta}\mathbb{E}_{t\sim p(T)}\log
p_{\theta}(t|d^{\prime});\;d^{\prime}\sim p_{\phi}(d|t)\end{split}$ (2)

Finally, we further add two denoising autoencoding objectives on both the data
and text sides:

$\max_{\theta,\phi}\mathbb{E}_{d\sim p(D),t\sim p(T)}\log
p_{\phi}(d|\tilde{d})p_{\theta}(t|\tilde{t})$ (3)

where $\tilde{d}$ and $\tilde{t}$ are the corrupted versions of $d$ and $t$.
We use the same noise function as in Lample et al. (2018) which randomly
permutes and pads a portion of the input. This can encourage the encoder to
learn meaningful latent representations by reconstructing the input itself
Currey et al. (2017); Lample et al. (2018).

Figure 3 illustrates all the four directions of the cycle consistency
objective.

Figure 3: Four directions of cycle consistency. Gradients are backpropagated
only through solid lines.

We use one shared encoder $Enc$ for both the data and text sides. Each data
sample is flattened into a sequence by making a list of slot value pairs and
fed into the same encoder. Using the same encoder for both types of input
gives the model an inductive bias to project similar data/text into
surrounding latent space.

We will show later that encoder sharing is essential for a good performance
under the few-shot scenario. From the shared encoded space, two separate
decoders $Dec_{d}$ and $Dec_{t}$ are used to decode $d$ and $t$
respectively444The shared encoding has also been shown effective in other
tasks like machine translation Lample et al. (2018) and image transition Zhu
et al. (2017). We further tried sharing the decoder as in Johnson et al.
(2017) but find no improvement (see Table 2)..

###  4.3 Representation Matching

Apart from training under the cycle consistency, we further consider matching
each synthetic text with its most similar data sample and treating them as
supplementary training pairs. Compared with the pseudo $d^{\prime}$ obtained
from back translation (Eq. 2), the matched data samples are extracted from the
existing corpus $D_{U}$ and thereby are guaranteed to be clean. This can
provide a much more stable training signal especially at the initial training
stage555In theory, as we can fabricate arbitrary possible data samples from
the predefined schema and add to the corpus, we can always find one matched
data for a text samples.. Previous work has used representation matching to
automatically extract pseudo training pairs for machine translation Artetxe
and Schwenk (2019); Ruiter et al. (2019). Baziotis et al. (2019); Chu and Liu
(2019) also demonstrate that the representation similarity between input-
output pairs can serve as a useful regularization for unsupervised text
summarization. We adopt a similar idea to create pseudo pairs based on their
cosine similarity in the representation space. To summarize, the process of
representation matching can be described as:

$\begin{split}\max_{\theta,\phi}\mathbb{E}_{t\sim
p(T^{\prime})}\mathbbm{1}_{cos(d^{\ast},t)>\varepsilon}(&\log
p_{\theta}(t|d^{\ast})\\\ +&\log p_{\phi}(d^{\ast}|t));\\\
d^{\ast}=\operatorname*{arg\,max}_{d\in D}\;&\text{cos}(d,t)\end{split}$ (4)

where $T^{\prime}$ is augmented text from the LM and $\mathbbm{1}$ is the
indicator function. We also perform mean pooling over the encoded
representations before matching them. $\varepsilon$ is a threshold. Pseudo
pairs with a cosine similarity less than $\varepsilon$ will be discarded.
Ideally, as the encoder improves, the pseudo pairs created by representation
matching will make more sense, which can in turn benefit the training of the
encoder.

###  4.4 Summary

Apart from the above unsupervised objective, on the few annotated data-text
pairs, we can impose the supervised objective:

$\max_{\theta,\phi}\mathbb{E}_{d,t\sim p(D_{L},T_{L})}\log
p_{\theta}(t|d)+\log p_{\phi}(d|t)$ (5)

where $(D_{L},T_{L})$ contains the $k$ data annotations. Putting all together,
we summarize it in Algorithm 1. In the training stage, we optimize the
objectives of cycle consistency, representation matching and supervised
learning sequentially to maintain a constant ratio of signals from all sides.

Algorithm 1 Few-shot Data-to-text Framework

1: Input: $D_{U},(D_{L},T_{L})$

2: Create $(D_{a},T_{a})$ by information augmentation ;

3: $(D_{L},T_{L})\leftarrow(D_{L},T_{L})\cup(D_{a},T_{a})$;

4: Create $T^{\prime}$ by LM augmentation ;

5: $T\leftarrow T_{L}\cup T^{\prime}$;

6: repeat

7: Sample batch data from $(D_{L},T)$ ;

8: Cycle consistency:

9: Optimize by Eq. 1 \+ Eq. 2 \+ Eq. 3;

10: Representation Matching:

11: Optimize by Eq. 4;

12: Supervised Training:

13: Optimize by Eq. 5;

14: until convergence

##  5 Experiment Setting

##### Data

We conduct experiments on the E2E Novikova et al. (2017) and WebNLG Colin et
al. (2016) datasets. E2E is a crowd-sourced dataset containing 50k instances
in the restaurant domain. The inputs are dialogue acts consisting of three to
eight slot-value pairs. WebNLG contains 25k instances describing entities
belonging to fifteen distinct DBpedia categories. The inputs are up to seven
RDF triples of the form _(subject, relation, object)_.

##### Configuration

The model is implemented based on fairseq Ott et al. (2019). We use
$600$-dimensional token embedding and Adam optimizer with initial learning
rate at $0.0002$. Batch size is kept at $48$ with a dropout rate at $0.3$. We
employ beam search with size $3$ for decoding and select models based on
BLEU-4 scores on the development set. The score is averaged over 10 random
initialization runs. In this work, the seq2seq models are built upon the long
short-term memory (LSTM) Hochreiter and Schmidhuber (1997). For LSTM cells,
both the encoder and decoder have $3$ layers, amounting to 18M parameters for
the seq2seq model (600-dimension and 1024 hidden units). Maximum sequence
length is set as 100 for E2E and 200 for WebNLG (SPM-based). All encoder
parameters are shared between data and text samples. All models were trained
on 1 Nvidia V100 GPUs (32GB and CUDA Version 10.2) for 4k steps. The total
batch size is around $48$K tokens per GPU and we use the Adam optimizer
($\epsilon=1\mathrm{e}{-6}$, $\beta_{2}=0.98$) along with linear learning rate
decay scheduling. The total number of updates is set to $8000$ for all
training and models are selected based on optimal validation BLEU4. At
decoding time, sentences are generated using greedy decoding.

##  6 Results and Analysis

Model | E2E - 10% | E2E - 100%
---|---|---
BLEU | NIST | METEOR | ROUGE-L | BLEU | NIST | METEOR | ROUGE-L
SLUG | - | - | - | - | 66.19 | 8.61 | 44.54 | 67.72
Seq2seq | 53.38 | 6.10 | 38.10 | 60.53 | 63.32 | 6.81 | 41.25 | 62.91
Qader et al. (2019) | 58.10 | 6.24 | 41.32 | 62.84 | 64.20 | 7.14 | 44.68 | 65.31
Chen et al. (2020) | 59.10 | 7.49 | 40.25 | 63.23 | 63.72 | 7.76 | 40.25 | 66.23
Proposed (LSTM) | 64.24 | 7.71 | 43.53 | 66.81 | 68.88 | 8.89 | 48.53 | 72.12
| WebNLG - 10% | WebNLG - 100%
Model | BLEU | NIST | METEOR | ROUGE-L | BLEU | NIST | METEOR | ROUGE-L
Melbourne | - | - | - | - | 44.93 | 8.98 | 36.58 | 60.40
Seq2seq | 36.54 | 7.3 | 35 | 54.61 | 44.60 | 8.49 | 38.23 | 59.67
Qader et al. (2019) | 38.66 | 7.81 | 34.1 | 56.95 | 47.19 | 8.71 | 37.90 | 58.61
Chen et al. (2020) | 39.40 | 7.84 | 37.25 | 56.23 | 46.15 | 8.52 | 39.1 | 58.5
Proposed (LSTM) | 43.75 | 8.29 | 33.58 | 58.49 | 50.26 | 8.86 | 40.71 | 61.29

Table 1:  Performance on E2E and WebNLG with 10% and 100% data. Qader et al.
(2019) utilizes all ground-truth unpaired text samples while our proposed
model only gets access to the few-shot data annotations.

In this section, we present experiment results and analysis. We first compare
our model with other baselines on both datasets, then perform a set of
ablation studies on the E2E dataset to see the effects of each component.
Finally, we analyze how text augmentation helps improves the model, include
example outputs and show the human evaluation results in the end.

##### Comparison with Other Models

In Table 1, we compare our model with (1) seq2seq baseline, (2) cycle
consistency model as in Qader et al. (2019)666The author did not open-source
their code. We reproduced their model based on our implementation. The results
on 10k annotations matches their reports in the paper. and (3) finetuned GPT-2
model as in Chen et al. (2020)777https://github.com/czyssrs/Few-Shot-NLG. For
all models, we try running with 10% and 100% annotations to see how they
perform under different data sizes. Our model is implemented both with LSTM
encoder-decoders, same as the seq2seq baseline for a fair comparison. Note
that Qader et al. (2019) further _utilized all the ground-truth unpaired text
samples_ , while the other models run only on the few-shot annotations. We
also include the results of SLUG Juraska et al. (2018) and MELBOURNE Gardent
et al. (2017), the overall winner on automatic metrics in the E2E and WebNLG
challenge respectively(both seq2seq-based). SLUG uses a heuristic slot aligner
based on a set of handcrafted rules and combines a complex pipeline of data
augmentation, selection, model ensemble and reranker.

The results show that our proposed model significantly improves over the
baseline on both the few-shot and fully supervised setting. The improvement is
more evident when only 10% annotations are available, with a leap of 11 and 7
BLEU scores on E2E and WebNLG respectively. It also outperforms systems
relying on task-dependent heuristics. In comparison, Qader et al. (2019),
though with access to all text samples at all percentages, still underperforms
our model with tangible margin. On the fully supervised setting, it brings
little to no difference compared with the seq2seq baseline as no more extra
data is incorporated in the training process. As such, we also observe that
the text augmentation from finetuned GPT-2 model helps the proposed model on
the few-shot setting, but its advantage also vanishes when all data
annotations are available.

Figure 4: Model performance with varying number of data annotations. Our model
with 10% annotations outperforms the seq2seq model trained on $100\%$ pairs
(dotted line) on both datasets. Shades are the sample standard deviation based
on 10 runs of different model initializations.

In Figure 4, we draw the model performance with varying number of data
annotations. All models are trained from scratch with 10 different random
initializations and the standard deviation of the BLEU-4 score is visualized.
We can see our model (LSTM-based), though with a relatively larger standard
deviation due to the uncertainty of text augmentation sampling, still
consistently outperforms other baselines significantly and even surpasses the
fully supervised seq2seq model with less than 10% of data annotations.

Model/Share | None | Enc | Dec | Both
---|---|---|---|---
Supervised | 53.20 | - | - | -
\+ $t\rightarrow d^{\prime}\rightarrow t$ | 53.19 | 53.28 | 53.17 | 53.29
\+ $d\rightarrow t^{\prime}\rightarrow d$ | 53.15 | 56.12 | 53.49 | 56.07
\+ $t\rightarrow t$ | 53.74 | 56.39 | 55.29 | 55.73
\+ $d\rightarrow d$ | 53.37 | 56.44 | 56.09 | 56.11
\+ Noise | 54.13 | 57.37 | 56.59 | 57.04

Table 2: Ablation study for cycle consistency (10% annotations). BLEU-4 score
is reported. Each line adds one condition on top of the previous one.
Supervised is a supervised seq2seq baseline.

##### Ablation Study on Cycle Consistency

In Table 2, we study how the four directions, input noise and parameter
sharing affect the performance of cycle-consistency. The experiments are
conducted with 10% annotations and _no further unpaired text samples are
available_.

As can be observed, adding the training direction $t\rightarrow
d^{\prime}\rightarrow t$ (i.e. back translation) has little effects on top of
the supervised seq2seq baseline. This is expected since back translation is
naturally designed to incorporate additional unpaired text samples. When run
only on the few-shot annotations, its power is very limited. The backward
direction $d\rightarrow t^{\prime}\rightarrow d$ is surprisingly useful when
the encoder is shared between the data and text. Though this direction will
not affect the text decoder at all, the improvement suggests the model can
benefit a lot by simply structuring its encoded space and mapping aligned
data-text pairs to similar vector space. The autoencoding directions brings a
little improvement. When combined with input noise, the performance further
increases. This is similar to previous findings that denoising autoencoding is
more helpful in inducing meaningful latent space Lample et al. (2018) in
comparison to simply learning to copy the original input.

The results also suggest encoder sharing is important for the cycle
consistency objective to work in our few-shot setting. Decoder sharing, in
contrast, makes little or even negative influence. This is kinda similar as in
multilingual machine translation where sharing the decoder among languages
might negatively interfere with the performance Johnson et al. (2017).

Text Augmentation | 1% | 5% | 10% | 20%
---|---|---|---|---
None | 44.18 | 50.22 | 57.37 | 63.28
Random | 41.30 | 49.62 | 57.71 | 62.79
UDA | 44.24 | 50.09 | 57.66 | 61.30
Info | 45.63 | 52.22 | 58.80 | 63.22
\+ LM | 48.67 | 53.53 | 59.04 | 64.78
Reference | 55.33 | 54.92 | 59.11 | 64.26
Random (+RM) | 42.32 | 50.10 | 58.53 | 63.29
UDA (+RM) | 44.32 | 52.22 | 58.80 | 61.27
Info (+RM) | 48.63 | 56.66 | 60.80 | 63.52
\+ LM (+RM) | 53.18 | 59.12 | 64.24 | 65.38
Reference (+RM) | 62.74 | 63.28 | 64.93 | 65.27

Table 3: Ablation study for text augmentation with varying number of
annotations. Experiments are performed on the E2E dataset. LM augmentation
outperforms Random by a large margin, and even outperforming augmentation with
ground-truth references on some occasions. Representation matching (RM) boosts
the overall performance further.

Data: [name] Blue Spice [eat type] restaurant [food] Chinese [area] city
centre [family friendly] no [near] Rainbow Vegetarian Café Reference: at the
[city centre], there is a [restaurant] called the [Blue Spice]. seq2seq/info-
aug: Blue Spice restaurant near Rainbow Vegetarian Café has a _5 star rating.
prices start at £30_. +LM-aug: located near Rainbow Vegetarian Café is a
Chinese theme eatery and restaurant called Blue Spice. It is in the city
centre area.

Figure 5: Generation examples with different text augmentation techniques.
Trained on 5 data annotations (see the above toy training set). The attribute
combination of input data is unseen in the 5 annotations. Hallucinated
contents are _italitized_.  Figure 6:  Additional number of decoded unique
tokens (non-copied) and unique combinations of information on the testset with
varying number of annotations.

Model | E2E | WebNLG
---|---|---
Fluency | Miss | Wrong | Fluency | Miss | Wrong
Seq2Seq | 3.68 | 49 | 63 | 3.95 | 57 | 48
Cycle-only | 4.08 | 46 | 66 | 4.23 | 48 | 44
Finetune GPT-2 | 4.21 | 43 | 57 | 4.10 | 39 | 45
Proposed (LSTM) | 4.33 | 39 | 44 | 4.27 | 31 | 39

Table 4:  Human Evaluation on the sampled outputs (100 instances) for models
with 10% annotated data. Cycle-only indicates the approach in Qader et al.
(2019); and Finetuned GPT-2 is refers to Chen et al. (2020).

##### Ablation Study on Text Augmentation

On top of the four-direction cycle consistency training, we study the effects
of text augmentation in Table 3. We compare our proposed info + LM
augmentation with (1) random augmentation, where a random text from Wikipedia
is sampled to the augmented text space, (2) unsupervised data augmentation Xie
et al. (2019) where text samples are augmented with paraphrases of current
annotations and (3) ground-truth augmentation with reference obtained from the
left training corpus, which can serve as an upper bound of text augmentation
techniques. We test the performance with 1%, 5%, 10% and 20% annotations to
see the effects with varying number of supervisions.

As can be seen, the random augmentation even harms the model performance,
suggesting reasonable in-domain text augmentation are necessary for the model
improvement. UDA augmentation also makes rather little difference as it simply
paraphrases the current available annotations but cannot bring any new
information. The information augmentation by slot-value replacement helps
improve a bit. When combined with LM, the performance can be further boosted,
especially for lower-resource scenarios. The representation matching always
helps lift the performance, with gains of up to 10 BLEU points. As expected,
the benefit from text augmentation gradually vanishes as more annotations are
collected, especially for datasets with relatively simple patterns as E2E.

##### How text augmentation helps

Intuitively the GPT-2 augmentation is expected to impose new tokens and
combination patterns to the few-shot annotations. To investigate whether this
is the case, for the decoded text in the test phase, we count the number of
unique tokens (excluding copied data values) and unique information
combination patterns (attribute combinations in E2E). The results in Fig. 6
show that LM-augmentation indeed greatly enriches the vocabulary space, even
doubling the generated unique tokens in low-resource scenarios. The same
happens for new combination patterns. In contrast, _info-aug_ cannot insert
new tokens or combinations at all since all it does is replacing data values
based on the same text annotation. UDA can impose new tokens by paraphrasing
the annotations, but it hardly helps the model generalize to new combinations
of information. Moreover, when trained on a toy dataset, we observe from the
generation outputs that Seq2seq and info-aug produce the wrong outputs and
overfit to the information in the 5 training instances. With LM augmentation,
it adapts to the new combination and connects information correctly. Figure 5
shows a generation example with different text augmentation techniques. We
train the systems in a toy setting with only 5 data annotations (Trainset in
the Appendix). We pick an input data with an unseen attribute combination to
test if models can generalize correctly. Seq2seq and info-aug produce the
wrong generation overfit to the information in the 5 training instances. With
LM augmentation, it adapts to the new combination and connects information
correctly.

##### Human Evaluation

We further run a human evaluation on the model outputs to closely check the
generation quality. We compared four types of models: the seq2seq baseline,
seq2seq plus cycle-consistency as in Qader et al. (2019), finetuned GPT-2 as
in Chen et al. (2020) and our proposed model. All models are LSTM-based apart
from the finetuned GPT-2 one. We sample 100 data instances from the test set
and apply all the models to generate corresponding text. The data and
generated text are evaluated by $50$ crowdworkers on
Prolific888https://www.prolific.co/. For each data-text pair, the annotator is
instructed to evaluate (1) if the text is fluent (score 0-5 with 5 being fully
fluent), (2) if it misses information contained in the source data and (3) if
it includes wrong information. The average fluency scores, count of
information miss and wrong information are presented in Table 4. The scores
are generally consistent with the automatic evaluation results, our proposed
model outperforms other ones by a large margin, even though cycle-only can
access all unpaired text and finetuned GPT-2 is significantly larger than our
LSTM-based seq2seq. The generated text are more fluent, yet maintaining the
information completeness and correctness to a large extent.

##  7 Conclusion

We study few-shot data-to-text generation with only limited annotated data. We
propose text augmentation with slot-value replacement followed by GPT-2
generation. The augmented text, when combined with cycle consistency and
representation matching, is shown to help the model to generalize to unseen
new tokens and patterns of token combinations. With less than 10% annotations,
it outperforms supervised seq2seq model trained on 100% annotations and is
extensible enough to be combined with pretraining techniques. For future
works, we hope to apply the techniques to annotation scenarios Hong et al.
(2019); de Souza et al. (2018); Zhuang and Chang (2017); Chang et al. (2020a);
Wiehr et al. (2020); Shen et al. (2020); Chang et al. (2020b); Su et al.
(2020a); Chang et al. (2021a, b); Crook et al. (2018).

## Acknowledgements

This research was funded in part by the German Research Foundation (DFG) as
part of SFB 248 “Foundations of Perspicuous Software Systems”. We sincerely
thank the anonymous reviewers for their insightful comments that helped us to
improve this paper.

## References

  * Agarwal et al. (2018) Shubham Agarwal, Marc Dymetman, and Eric Gaussier. 2018.  Char2char generation with reranking for the e2e nlg challenge.  In _Proceedings of the 11th International Conference on Natural Language Generation_ , pages 451–456.
  * Artetxe et al. (2017) Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017.  Unsupervised neural machine translation.  _arXiv preprint arXiv:1710.11041_.
  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Margin-based parallel corpus mining with multilingual sentence embeddings.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3197–3203.
  * Barzilay and Lapata (2005) Regina Barzilay and Mirella Lapata. 2005.  Modeling local coherence: An entity-based approach.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 141–148, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Baziotis et al. (2019) Christos Baziotis, Ion Androutsopoulos, Ioannis Konstas, and Alexandros Potamianos. 2019.  Seqˆ3: Differentiable sequence-to-sequence-to-sequence autoencoder for unsupervised abstractive sentence compression.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 673–681.
  * Budzianowski and Vulić (2019) Paweł Budzianowski and Ivan Vulić. 2019.  Hello, it’s gpt-2-how can i help you? towards the use of pretrained language models for task-oriented dialogue systems.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , pages 15–22.
  * Chang et al. (2020a) Ernie Chang, David Adelani, Xiaoyu Shen, and Vera Demberg. 2020a.  Unsupervised pidgin text generation by pivoting english data and self-training.  In _In Proceedings of Workshop at ICLR_.
  * Chang et al. (2020b) Ernie Chang, Jeriah Caplinger, Alex Marin, Xiaoyu Shen, and Vera Demberg. 2020b.  Dart: A lightweight quality-suggestive data-to-text annotation tool.  In _COLING 2020_ , pages 12–17.
  * Chang et al. (2021a) Ernie Chang, Vera Demberg, and Alex Marin. 2021a.  Jointly improving language understanding and generation with quality-weighted weak supervision of automatic labeling.  In _EACL 2021_.
  * Chang et al. (2021b) Ernie Chang, Hui-Syuan Yeh, and Vera Demberg. 2021b.  Does the order of training samples matter? improving neural data-to-text generation with curriculum learning.  In _EACL 2021_.
  * Chen et al. (2020) Zhiyu Chen, Harini Eavani, Yinyin Liu, and William Yang Wang. 2020.  Few-shot nlg with pre-trained language model.  _ACL_.
  * Chu and Liu (2019) Eric Chu and Peter Liu. 2019.  Meansum: a neural model for unsupervised multi-document abstractive summarization.  In _International Conference on Machine Learning_ , pages 1223–1232.
  * Colin et al. (2016) Emilie Colin, Claire Gardent, Yassine M’rabet, Shashi Narayan, and Laura Perez-Beltrachini. 2016.  The WebNLG challenge: Generating text from DBPedia data.  In _Proceedings of the 9th International Natural Language Generation conference_ , pages 163–167, Edinburgh, UK. Association for Computational Linguistics.
  * Crook et al. (2018) Paul A Crook, Alex Marin, Vipul Agarwal, Samantha Anderson, Ohyoung Jang, Aliasgar Lanewala, Karthik Tangirala, and Imed Zitouni. 2018.  Conversational semantic search: Looking beyond web search, q&a and dialog systems.  In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_ , pages 763–766.
  * Currey et al. (2017) Anna Currey, Antonio Valerio Miceli-Barone, and Kenneth Heafield. 2017.  Copied monolingual data improves low-resource neural machine translation.  In _Proceedings of the Second Conference on Machine Translation_ , pages 148–156.
  * Freitag and Roy (2018) Markus Freitag and Scott Roy. 2018.  Unsupervised natural language generation with denoising autoencoders.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3922–3929.
  * Garcia et al. (2020) Xavier Garcia, Pierre Foret, Thibault Sellam, and Ankur P Parikh. 2020.  A multilingual view of unsupervised machine translation.  _arXiv preprint arXiv:2002.02955_.
  * Gardent et al. (2017) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017.  The webnlg challenge: Generating text from rdf data.  In _Proceedings of the 10th International Conference on Natural Language Generation_ , pages 124–133.
  * Gkatzia (2016) Dimitra Gkatzia. 2016.  Content selection in data-to-text systems: A survey.  _arXiv preprint arXiv:1610.08375_.
  * He et al. (2016) Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016.  Dual learning for machine translation.  In _Advances in neural information processing systems_ , pages 820–828.
  * He et al. (2020) Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg-Kirkpatrick. 2020.  A probabilistic formulation of unsupervised text style transfer.  _ICLR_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Hong et al. (2019) Xudong Hong, Ernie Chang, and Vera Demberg. 2019.  Improving language generation from feature-rich tree-structured data with relational graph convolutional encoders.  In _Proceedings of the 2nd Workshop on Multilingual Surface Realisation (MSR 2019)_ , pages 75–80.
  * Jang et al. (2016) Eric Jang, Shixiang Gu, and Ben Poole. 2016.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_.
  * Johnson et al. (2017) Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017.  Google’s multilingual neural machine translation system: Enabling zero-shot translation.  _Transactions of the Association for Computational Linguistics_ , 5:339–351.
  * Juraska et al. (2018) Juraj Juraska, Panagiotis Karagiannis, Kevin Bowden, and Marilyn Walker. 2018.  A deep ensemble model with slot alignment for sequence-to-sequence natural language generation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 152–162.
  * Kondadadi et al. (2013) Ravi Kondadadi, Blake Howald, and Frank Schilder. 2013.  A statistical nlg framework for aggregated planning and realization.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 1406–1415.
  * Lample et al. (2017) Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017\.  Unsupervised machine translation using monolingual corpora only.  _arXiv preprint arXiv:1711.00043_.
  * Lample et al. (2018) Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, et al. 2018.  Phrase-based & neural unsupervised machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5039–5049.
  * Lebret et al. (2016) Rémi Lebret, David Grangier, and Michael Auli. 2016.  Neural text generation from structured data with application to the biography domain.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1203–1213.
  * van der Lee et al. (2018) Chris van der Lee, Emiel Krahmer, and Sander Wubben. 2018.  Automated learning of templates for data-to-text generation: comparing rule-based, statistical and neural methods.  In _Proceedings of the 11th International Conference on Natural Language Generation_ , pages 35–45.
  * Ma et al. (2019) Shuming Ma, Pengcheng Yang, Tianyu Liu, Peng Li, Jie Zhou, and Xu Sun. 2019.  Key fact as pivot: A two-stage model for low resource table-to-text generation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2047–2057.
  * Novikova et al. (2017) Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017.  The e2e dataset: New challenges for end-to-end generation.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 201–206.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of NAACL-HLT 2019: Demonstrations_.
  * Oya et al. (2014) Tatsuro Oya, Yashar Mehdad, Giuseppe Carenini, and Raymond Ng. 2014.  A template-based abstractive meeting summarization: Leveraging summary and source text relationships.  In _Proceedings of the 8th International Natural Language Generation Conference (INLG)_ , pages 45–53.
  * Papanikolaou and Pierleoni (2020) Yannis Papanikolaou and Andrea Pierleoni. 2020.  Dare: Data augmented relation extraction with gpt-2.  _arXiv preprint arXiv:2004.13845_.
  * Peng et al. (2020) Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, and Jianfeng Gao. 2020.  Few-shot natural language generation for task-oriented dialog.  _arXiv preprint arXiv:2002.12328_.
  * Prabhumoye et al. (2018) Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018\.  Style transfer through back-translation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 866–876.
  * Qader et al. (2019) Raheel Qader, François Portet, and Cyril Labbé. 2019.  Semi-supervised neural text generation by joint learning of natural language generation and natural language understanding models.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 552–562.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8).
  * Reiter and Dale (2000) Ehud Reiter and Robert Dale. 2000.  _Building natural language generation systems_.  Cambridge university press.
  * Ruiter et al. (2019) Dana Ruiter, Cristina España-Bonet, and Josef van Genabith. 2019.  Self-supervised neural machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1828–1834, Florence, Italy. Association for Computational Linguistics.
  * Schmitt and Schütze (2019) Martin Schmitt and Hinrich Schütze. 2019.  Unsupervised text generation from structured data.  _arXiv preprint arXiv:1904.09447_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Improving neural machine translation models with monolingual data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 86–96.
  * Shen et al. (2020) Xiaoyu Shen, Ernie Chang, Hui Su, Jie Zhou, and Dietrich Klakow. 2020.  Neural data-to-text generation via jointly learning the segmentation and correspondence.  In _ACL 2020_. https://www.aclweb.org/anthology/2020.acl-main.641.pdf.
  * de Souza et al. (2018) José GC de Souza, Michael Kozielski, Prashant Mathur, Ernie Chang, Marco Guerini, Matteo Negri, Marco Turchi, and Evgeny Matusov. 2018.  Generating e-commerce product titles and predicting their quality.  In _INLG_ , pages 233–243.
  * Su et al. (2020a) Hui Su, Xiaoyu Shen, Zhou Xiao, Zheng Zhang, Ernie Chang, Cheng Zhang, Cheng Niu, and Jie Zhou. 2020a.  Moviechats: Chat like humans in a closed domain.  In _EMNLP 2020_ , pages 6605–6619.
  * Su et al. (2020b) Shang-Yu Su, Chao-Wei Huang, and Yun-Nung Chen. 2020b.  Towards unsupervised language understanding and generation by joint dual learning.  _ACL_.
  * Subramanian et al. (2018) Sandeep Subramanian, Guillaume Lample, Eric Michael Smith, Ludovic Denoyer, Marc’Aurelio Ranzato, and Y-Lan Boureau. 2018.  Multiple-attribute text style transfer.  _arXiv preprint arXiv:1811.00552_.
  * Sun et al. (2020) Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2020.  {LAMAL}: {LA}nguage modeling is all you need for lifelong language learning.  In _International Conference on Learning Representations_.
  * Wiehr et al. (2020) Frederik Wiehr, Anke Hirsch, Florian Daiber, Antonio Kruger, Alisa Kovtunova, Stefan Borgwardt, Ernie Chang, Vera Demberg, Marcel Steinmetz, and Hoffmann Jorg. 2020.  Safe handover in mixed-initiative control for cyber-physical systems.  In Proceedings of Workshop at CHI.
  * Williams (1992) Ronald J Williams. 1992.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine learning_ , 8(3-4):229–256.
  * Wiseman et al. (2017) Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017.  Challenges in data-to-document generation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2253–2263.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _ArXiv_ , abs/1910.03771.
  * Xie et al. (2019) Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2019.  Unsupervised data augmentation for consistency training.  _arXiv preprint arXiv:1904.12848_.
  * Zhu et al. (2017) Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2223–2232.
  * Zhuang and Chang (2017) WenLi Zhuang and Ernie Chang. 2017.  Neobility at semeval-2017 task 1: An attention-based sentence similarity model.  In _In Proceedings of SemEval-2017 at ACL 2017._


Juami H. M. van Gils*  Halima Mouhib  Erik van Dijk  Maurits Dijkstra  Isabel
Houtkamp Arthur Goetzee Sanne Abeln*  K. Anton Feenstra*  * editorial
responsability

##  Chapter 13 Thermodynamics of Protein Folding

###### Contents

  1. 13 Thermodynamics of Protein Folding
    1. 13.1 Equilibrium and Dynamics
    2. 13.2 Thermodynamic laws
    3. 13.3 Entropy
    4. 13.4 Enthalpy
    5. 13.5 Free energy
      1. 13.5.1 Temperature Dependence of Free Energy Landscapes
    6. 13.6 From Microstates to Macrostates
      1. 13.6.1 Order Parameters
      2. 13.6.2 Ensemble Average
    7. 13.7 Ensembles
    8. 13.8 Conclusion
    9. Author contributions

In the previous chapter, Chapter “LABEL:ch:ChIntroDyn”, we introduced the
concept of _free energy_ in the context of protein folding. The goal of that
chapter was to start building an intuitive feeling of the meaning of the
fundamental concepts that are important for this topic. Given a free energy
landscape for the conformational space of a protein, we explained that
conformations with low free energy are the most stable states of a protein
(see Figure LABEL:fig:ChIntroDyn-FreeEnergy). In other words, these represent
the states that are most probable, and proteins will typically spend most of
the time in these low free energy states. The most stable state of a protein
under normal conditions (e.g., room temperature and neutral pH) is often
referred to as the native state of that protein. This is typically also the
functional state of the protein. Although the protein is mostly in its native
state under normal conditions, some fraction of the molecules may in fact be
unfolded (typically as little as 1% or even less).

In this chapter, we will take a step back and try to get a deeper, more formal
understanding of the concept of free energy in terms of the _entropy_ and
_enthalpy_ ; to this end, we will first need to better define the meaning of
equilibrium, entropy and enthalpy. When we understand these concepts, we will
come back for a more quantitative explanation of protein folding and dynamics.

###  13.1 Equilibrium and Dynamics

In molecular simulations of proteins, we consider proteins in a _dynamic
equilibrium_. We do this under the assumption that over time, all systems move
towards equilibrium. This means that the system will eventually reach a state
where there is no net flow of energy or molecules between different parts of
the system.

Take for example thermal equilibrium. When a cold metal object is placed into
a warm water bath Figure 13.1, the two subsystems will exchange energy in the
form of heat. However, initially the warm water will have higher thermal
energy than the metal object, and there will be larger energy flow from the
water into the metal. Therefore, the water will (gradually) cool down and the
metal object will become warmer. At a certain point, the temperature of the
water and the metal object have become equal, and there will be no net energy
flow. The system has now reached thermal equilibrium. Note that this is a
macroscopic property of the system; although there is no net flow of energy in
the water bath, at the microscopic level, individual atoms in the system can
(and will) still exchange energy when they bump into one another. We will
elaborate on macro- vs microscopic properties later in this chapter.

Figure 13.1: Thermal Equilibrium. If a cold metal object ($T_{1}=10^{\circ}C$)
is placed into a warm-water bath ($T_{2}=20^{\circ}C$), the two components
will eventually exchange energy until they have reached the same temperature
($T\textsubscript{equilibrium}=15^{\circ}C$).

Equilibrium also applies to protein folding and unfolding. When the system is
in equilibrium, individual molecules can still (stochastically) switch between
the folded and the unfolded state, but the total number of molecules in each
state remains constant. In other words, although protein molecules are still
switching between states, the probability to be in a certain state is
constant. As highlighted before in Chapter “LABEL:ch:ChIntroDyn”, the
probability of being in a given state in the system is directly related to the
free energy of that state: states with a low free energy have a high
probability to be encountered, and are therefore considered to have high
stability.

Finally, before we dive into the detailed material, we will provide a brief
explanation of the difference between classical and statistical
thermodynamics. We will need both, and understanding this difference will help
you while reading the rest of the chapter. Classical thermodynamics is used to
describe macroscopic properties of the system, such as the distribution of
energy in the warm water bath described above, but does not consider
microscopic fluctuations, such as the folding or unfolding of individual
protein molecules. Statistical thermodynamics, also referred to as
(equilibrium) statistical mechanics, describes how fluctuations at the
microscopic (molecular) level of the system lead to the macroscopic behaviors
that can be observed in experiments. Therefore, if we want to gain more
mechanistic insight into properties of a process like protein folding,
stability or function, we need to include the principles of statistical
thermodynamics.

In the next sections, we begin by explaining the fundamental laws of
thermodynamics, followed by the entropy and enthalpy. Finally, we will bring
these concepts together to understand protein folding and dynamics from a
thermodynamic perspective.

###  13.2 Thermodynamic laws

The laws of thermodynamics were first formulated in the 19th and early 20th
century. They have several fundamental implications for protein dynamics, and
many checks, tricks and assumptions used in simulations are based on these
laws. (Note that in literature the numbering of these rules is not always
consistent.)

  * Zeroth law of thermodynamics.

The zeroth law of thermodynamics states that if we have three states (A, B and
C) in a system, and both A & B and B & C are in equilibrium, then A & C must
also be in equilibrium.

  * First law of thermodynamics.

The first law is equivalent to the law of conservation of energy; energy
cannot be created or destroyed, just transformed from one form to another.
This can be rephrased by saying that the amount of total energy in the system
does not change:

$\frac{\partial E_{tot}}{\partial t}=0$ (13.1)

Examples of consequences of the first law:

    1. 1)

Motor proteins in a cell use ATP to move along a microtubule Hirokawa et al.
(1998); Mondal et al. (2017). This process converts internal chemical energy
(in the form of ATP) into kinetic energy or work (moving forward with a
certain velocity), and heat.

    2. 2)

The F-ATPase found in bacteria and mitochondria can use ATP to move protons
(H+) across the cell membrane, or, reversibly, use a concentration gradient of
protons to synthesize ATP Mitchell (1961); Frasch et al. (2022). Here,
chemical energy from the ATP molecule is converted into a different kind of
chemical energy in the form of an electrochemical (charge & pH) gradient
across the cell membrane.

    3. 3)

Friction converts kinetic energy (movement) into thermal energy (heat). For
example, if you rub your hands together because you are feeling cold), they
become warmer.

  * Second law of thermodynamics.

The second law of thermodynamics states that, for an isolated system, the
entropy will never decrease (we will define entropy in the next section; for
now, think of it as a measure for the amount of chaos/disorder). In other
words, the entropy will keep increasing until it reaches the maximum possible
value within the physical constraints of the system and then remain constant:

$\frac{\partial S}{\partial t}\geq 0$ (13.2)

This also means that an isolated system always evolves to thermodynamic
equilibrium (a state with no net flow of energy or molecules).

Examples of consequences of the second law:

    1. 1)

The gas atoms in a room will not spontaneously go to one corner in that room.

    2. 2)

A purely hydrophilic peptide sequence will not spontaneously fold in a
hydrophilic solution, as the folded state is entropically unfavorable.

    3. 3)

Mixed red and blue marbles in a box will not separate spontaneously when
shaking the box (we will use this example in Section 13.3 “Entropy”).

  * Third law of thermodynamics.

The entropy of a system at absolute zero (zero Kelvin), is constant and zero
(actually close to zero due to quantum mechanical effects, and kinetically
trapped states in so-called glassy systems) Kittel and Kroemer (1980). The
third law has an interesting consequence: later in this chapter, we will see
that entropy becomes unimportant near zero Kelvin (as the product of $T$ and
$S$ approaches zero in Equation 13.6).

Examples of consequences of the third law:

    1. 1)

There is no vibration of atoms in molecules at zero Kelvin (aside from
quantum-mechanical effects).

    2. 2)

At low temperature, a protein will optimize its internal interactions, i.e.,
minimise its total internal energy. (This property is the foundation of a
simulation technique called ‘simulated annealing’, as we will see later in
Chapter “LABEL:ch:ChMD” and Chapter “LABEL:ch:ChMC”).

    3. 3)

At high temperature, a protein will unfold, i.e., maximise its entropy.

Note that these laws apply to all physical systems, not just to the atoms,
molecules and other details that we find important when investigating
biological protein structures. Think of for example steam engines, the weather
or galactic systems.

###  13.3 Entropy

In this section we will explain the concept of entropy in more detail. Entropy
is a quantification of the amount of conformational freedom of the system.
Loosely speaking, one may think of entropy as quantifying the amount of chaos
or disorder in the system. More strictly, it is the number of possible
microstates (structural conformations) that are accessible in a given
macrostate (e.g., folded/unfolded state of a protein), as determined by the
physical conditions.

Figure 13.2: Box with marbles. Initially, the marbles are sorted with the red
marbles on one side and the blue marbles on the other side of the box (left
panel). If the box is shaken, the marbles will move around randomly. In the
equilibrium state, the marbles are distributed homogeneously over the box
(right panel).

To illustrate this concept on a simple system, take for example a box with
spherical red and blue marbles that do not interact (i.e., there are no
attractive or repulsive forces between the marbles). Initially, the marbles
are sorted and all red marbles are at one side of the box and all blue marbles
are at the other side (left panel Figure 13.2). If you shake the box, the
marbles will start moving around. After the box has been shaken for a
sufficient amount of time, the red and blue marbles will be randomly
distributed over the box. The number of red and blue marbles at each side of
the box is now approximately equal. If you keep shaking the box, marbles will
keep moving from one side to the other and back, but the total number of red
and blue marbles at each side will no longer change. The system has reached
equilibrium (right panel Figure 13.2).

#### How to calculate the multiplicity of a state

The multiplicity of a state $A$ can be calculated by
$\Omega_{A}=\frac{N!}{k_{A}!(N-k_{A})!}$ (13.3) where $N$ equals the total
number of conformations and $k_{A}$ equals the number of conformations in one
particular state. In our example of the marbles, imagine we have ten red and
ten blue marbles in the box and that only ten marbles can fit into one side of
the box (see Figure 13.2). In this case, the number of conformations equals
the total number of marbles at one side of the box, hence ${N=10}$. $k_{A}$
represents the number of red marbles at that side of the box. There are only
two ways to have all red marbles at the same side, namely ${k_{A}=0}$ and
${k_{A}=10}$. However, there are 252 ways to distribute the marbles equally
over the two sides of the box (if ${k_{A}=5}$, $\Omega_{A}=\frac{10!}{5!\cdot
5!}=\frac{10\cdot 9\cdot 8\cdot 7\cdot 6\cdot 5\cdot 4\cdot 3\cdot 2\cdot
1}{5\cdot 4\cdot 3\cdot 2\cdot 1\,\cdot\,5\cdot 4\cdot 3\cdot 2\cdot
1}=\frac{30240}{120}=252$). Note that we do not care exactly which marbles are
in which position, but only the total distribution of the differently coloured
marbles over the sides of the box.

Is it impossible for the red and blue marbles to spontaneously return to the
initial (sorted) state? Technically, no. However, the probability of such an
event is extremely low. The probability of being in a certain state is
dependent on the number of ways to reach that state. This is called the
multiplicity of a state, and is denoted by $\Omega$ (omega). For example, if
four out of ten blue marbles are on the left side of the box, it does not
matter which four these are, you will observe the same state, namely four blue
marbles on the left side and six on the right side of the box.

Another example is coin tossing. If you flip a coin three times and throw two
heads and a tail, it does not matter if you throw two heads first and then a
tail or first a tail and then two heads. Either way you reach the same state,
namely two out of three heads. In the context of protein folding, the
multiplicity would indicate the number of conformations in each state (e.g.,
folded/unfolded). The probability of each state can be calculated by dividing
the multiplicity of that state by the sum of the multiplicities of all states
in the system (i.e., the total number of ways to arrange the marbles in the
box – see Panel “How to calculate the multiplicity of a state”):

${p_{A}}=\frac{\Omega_{A}}{\sum\limits_{X}{\Omega_{X}}}$ (13.4)

where ${p_{A}}$ equals the probability of state $A$, $\Omega_{A}$ is the
multiplicity of state $A$ and $\sum\limits_{X}{\Omega_{X}}$ is the sum of the
multiplicities of all states in the system.

From this equation one can see that in a system lacking attracting or opposing
forces the largest multiplicity also has the largest probability. A large
multiplicity implies a lot of conformational freedom, i.e., high entropy,
which is directly related to the multiplicity $\Omega$ by:

$S_{A}={k_{B}}\ln{\Omega_{A}}$ (13.5)

where $S_{A}$ is the entropy of state $A$, ${k_{B}}$ is the Boltzmann constant
(which relates temperature to energy per molecule Fischer (2019)) and
$\Omega_{A}$ is the multiplicity of state $A$. Thus, a state with a high
multiplicity also has a high entropy, which makes that state more favorable.
Note that in complex continuous systems like proteins in solution, the number
of conformations is infinite and so we cannot count the absolute number of
conformations. However, we can use the relation between multiplicity and
probability to get an estimate of entropic differences between states from
simulations (more on that later, in Section 13.5 “Free energy”).

###  13.4 Enthalpy

The enthalpy is the internal energy of the system plus the product of pressure
and volume. The internal energy consists of kinetic energy (movement), thermal
energy (heat), potential and interaction energies, amongst others. The more
favourable interactions a molecule has, the lower the enthalpy. An example of
interaction energy is the Van der Waals interaction, which can be described by
the Lennard-Jones interaction potential (more detail on interaction
potentials, including the Lennard-Jones in Chapter “LABEL:ch:ChMD”). Other
forms of interactions are polar interactions (e.g., hydrogen bonds),
electrostatic interactions and hydrophobic interactions (e.g.,
$\pi$-stacking).

Note, that a favourable interaction reduces the enthalpy. The interaction
energy at infinite distance is typically defined as zero; this means that a
favorable interaction will have a negative energy.

###  13.5 Free energy

So far, we have explained two thermodynamic concepts that affect the systems
of interest: entropy and enthalpy. Now we will combine these two concepts to
understand the stability of states in the system. As we have seen in Chapter
“LABEL:ch:ChIntroDyn”, the more stable a state is, the lower the free energy
associated with that state is. If the entropy and enthalpy of the state are
known, we can directly calculate the free energy of that state using

$F=H-TS$ (13.6)

where $H$ is the enthalpy, $T$ is the temperature in Kelvin and $S$ is the
entropy. One can see from this formula that a macroscopic state with a high
number of favorable interactions (low enthalpy) and/or low number of
unfavorable interactions (which would increase the enthalpy), has a low free
energy. Conversely, a state with fewer favorable interactions or more
unfavourable interactions, will have a higher free energy.

Additionally, Equation 13.6 shows that increasing the entropy of the system
also reduces the free energy. Usually there is a trade-off between reducing
the enthalpy and increasing the entropy of a system. Having many favorable
interactions (low enthalpy) reduces the conformational freedom of the protein,
which will lead to a lower entropy, which is entropically unfavourable. On the
other hand, the state with the maximum entropy usually consists of unfolded
conformations, where there are more interactions with the solvent. But for a
protein this also means more interactions between hydrophobic residues and the
solvent, which is enthalpically unfavourable. An example of this is shown in
Figure 13.3. The figure shows a model of a six-residue peptide on a square
lattice, where the orange circles indicate hydrophobic residues, and blue
circles hydrophilic residues. There is one conformation with the lowest
enthalpy (the highest number of favourable internal interactions), this is the
folded state at the bottom in Figure 13.3. The highest entropy state is the
unfolded state with fourteen conformations, which all have no internal
interactions, shown at the top of Figure 13.3. Depending on the conditions of
the system (e.g., temperature, pH, other solvents), the folded, unfolded or
one of the intermediate states will be the most stable.

Figure 13.3: Conformational enthalpy and entropy in an hydrophobic-polar (HP)
lattice model of protein folding. Here, atoms can only be at the intersections
of a two-dimensional square lattice. The lowest energy is shown at the bottom,
which corresponds to the ‘folded’ state. Energy here is counted as the number
of interactions, where H-H and P-P are favorable and H-P is not. The top row
has no interactions, the second row one P-P contact, the third row one H-H
contact, and the bottom (native state) has one H-H plus one P-P contact. The
number of ‘possible’ conformations per energy level (state) decreases as well,
going from top (unfolded) to bottom (native/folded). Image adapted from Martin
Gruebele, University of Illinois, USA Ballew et al. (1996).

Now that we know the relation between free energy, entropy and enthalpy
(Equation 13.6), the next question is then, how do we determine these
properties from a system. As explained in Section 13.3, the entropy $S_{A}$ of
a state A is directly related to its multiplicity via Equation 13.5. The total
energy of the state $E_{A}$ can be calculated from the weighted sum of the
energies of the conformations in state A:

$E_{A}=\sum\limits_{i\in A}{E_{i}p_{i}}$ (13.7)

However, the calculation of the absolute values of the free energy
contributions is far too complex for continuous systems such as proteins in
solution, because the number of conformations to be considered is infinite.
Nevertheless, the probability of a macroscopic state, such as the folded or
unfolded state of a protein, can be approximated from simulations by
determining the fraction of time spent in that state (see Panel “Derivation of
free energy using statistical thermodynamics” to see how we get here using
statistical thermodynamics):

$F_{A}\propto-k_{B}T\ln{(p_{A})}$ (13.8)

where $F_{A}$ is the free energy of state $A$, $k_{B}$ is the Boltzmann
constant, $T$ is the temperature in Kelvin and $p_{A}$ is the probability of
the system to be in state $A$. Note that $F_{A}$ is proportional, not equal,
to $p_{A}$. Thus, we cannot calculate the absolute value of the free energy of
state $A$ using this approach. In practice, this is not a problem, since we
are usually only interested in the _free energy difference_ between two states
(e.g folded vs unfolded) to see which one is more favourable, rather than the
absolute free energy of an individual state. This difference can be obtained
using:

$\displaystyle\Delta F_{A\rightarrow B}$ $\displaystyle=$ $\displaystyle
F_{B}-F_{A}$ (13.9) $\displaystyle=$ $\displaystyle-
k_{B}T\ln{(p_{B})}-\left(-k_{B}T\ln{(p_{A})}\right)$

Thus, the free energy difference between two states equals:

$\Delta F_{A\rightarrow B}=-k_{B}T\ln{\left(\frac{p_{B}}{p_{A}}\right)}$
(13.10)

Equation 13.10 is very important, since it tells us that the only thing we
need to know to calculate the free energy difference between two states, is
the relative amount of time proteins spend in each state; $p_{A}$ and $p_{B}$.
As we will see in Chapter “LABEL:ch:ChMD” and Chapter “LABEL:ch:ChMC”, these
probabilities can easily be determined from simulations (provided all states
are sampled sufficiently).

#### Derivation of free energy using statistical thermodynamics

Combining Equation 13.5, Equation 13.6, and Equation 13.7, the formula for the
free energy of state $A$ ($F_{A}$) becomes
$F_{A}=E_{A}-TS_{A}=\sum_{i}{E_{i}p_{i}}-k_{B}T\ln{\left(\Omega_{A}\right)}$
(13.11) where $E_{i}$ is the energy of a conformation in the ensemble of
conformations belonging to state A, $p_{i}$ is the probability of that
conformation, $k_{B}$ is the Boltzmann constant, $T$ is the temperature in
Kelvin and $\Omega$ is the multiplicity of the state. In a complex system such
as a protein in solution, the multiplicity can usually not be calculated
explicitly. However, in Panel “How to calculate the multiplicity of a state”,
we explained that $\Omega_{A}=N!/(k_{A}!\left(N-k_{A}\right))$. Applying this
to Equation 13.5 ($S_{A}={k_{B}}\ln{\Omega_{A}}$), using Stirling’s
approximation, which says that the logarithm of the factorial of a very large
number can be approximated as $\ln(N!)\approx N\ln(N)-N$, Glazer et al. (2002)
and plugging the results into Equation 13.11, eventually yields:
$F_{A}=\sum_{i}{E_{i}p_{i}}+k_{B}T\sum_{i}{p_{i}\ln{(p_{i})}}$ (13.12)
Equation 13.4 shows how to calculate the probability of a state if there are
no interactions between the particles in the system (i.e., no enthalpy, so the
system is entropy-driven), such as the example with the marbles in Section
13.3. In a system where the particles do interact, such as a protein in
solution, the probability of a conformation also depends on the energy of that
conformation. This is captured in the Boltzmann distribution, which describes
the probability $p_{i}$ of a state $i$ as function of its energy $E_{i}$ in an
equilibrium situation:
$p_{i}=\frac{e^{-E_{i}/k_{B}T}}{\sum_{i}{e^{-E_{i}/k_{B}T}}}$ (13.13) where
$p_{i}$ is the probability of a conformation, $E_{i}$ is the energy of that
conformation, $k_{B}$ is the Boltzmann constant and $T$ is the temperature (in
Kelvin). The denominator is called the _Partition function_ , denoted as $Z$:
$Z~{}=~{}\sum_{i}{e^{-\frac{E_{i}}{k_{B}T}}}$ (13.14) which simplifies
Equation 13.13 to: $p_{i}=\frac{1}{Z}{e^{-\frac{E_{i}}{k_{B}T}}}$ (13.15)
Substituting Equation 13.15 back into Equation 13.12 and simplifying the
result gives the total free energy $F$ of the system: $F_{A}=-k_{B}T\ln{(Z)}$
(13.16) We can now directly relate the free energy to the probability of being
in a certain state. Equation 13.14 defines the partition $Z$ as the sum of all
Boltzmann factors of the conformations in the system. For a system like a
solution of protein molecules, we can think of this as describing the how
likely it is to find one molecule in a particular microstate at any given
moment in time, or in other words, how the system is divided, _partitioned_ ,
across the different microstates. Then, we can reverse Equation 13.15 to
obtain the probability $p_{i}$ of being in that state $i$, then $F_{i}\propto-
k_{B}T\ln{(p_{i})}$ (13.17) This probability can, at least in principle, be
obtained from simulations or experiments. We will show examples of this, from
simulation, in the next section, “Temperature Dependence of Free Energy
Landscapes”, and more in Chapter “LABEL:ch:ChMD” and Chapter “LABEL:ch:ChMC”.

####  13.5.1 Temperature Dependence of Free Energy Landscapes

Figure 13.4: Temperature dependence of protein stability. (a) RMSD vs. time
for 5 different temperatures: 298, 320, 340, 350 and 360 K. (b) Fraction
folded as function of temperature, derived from the simulations shown in (a).
At room temperature the protein is almost always in its folded state. As the
temperature increases the protein is more in its unfolded state. (c) Schematic
Free Energy diagrams corresponding to the lowest temperature (298K). The
reaction coordinate used is the same as for (a): the RMSD to native. The
folded state (left) has a lower RMSD, while for the unfolded state (right) it
is high. The free energy of the folded state is lower, which indicates it is
more stable than the unfolded state at this temperature. The barrier between
folded and unfolded states limits the rate at which folding and unfolding
events may happen. (d) Same, but for the highest temperature (360K). Now, the
free energy of the unfolded state is lower, which indicates it is more stable
than the folded state at 360K. The barrier between folded and unfolded states
is somewhat lower, reflecting the higher rate at which folding and unfolding
can be observed in panel (a). Panel (a), data for panel (b) and structures in
(c) and (d) with permission from Daura & Oostenbrink Daura et al. (1998).

One of the reasons we are interested in the thermodynamics of protein folding
is so we can understand the temperature dependence of the stability. Proteins
can fold reversibly, as we know from Anfinsen (1973). Figure 13.4a shows
simulations of a small (7 amino acids) peptide to illustrate this phenomenon.
Each of the colors in the graph represents a time trace of the RMSD to the
native structure over time for simulations at a different temperature. One can
see in the figure that throughout the simulations, the RMSD increases and
decreases, indicating that the protein unfolds and refolds several times. It
is clear that as the temperature increases the protein spends more time in the
unfolded state (high RMSD in Figure 13.4a) which can be quantified by the
probability (time spent) of the folded state as a function of temperature
(Figure 13.4b).

Let us see if we can understand this from a thermodynamic perspective. Recall,
from Equation 13.6, that $\Delta F=\Delta H-T\Delta S$. In this example, we
are talking about the free energy of folding, so the $\Delta$ here indicates
the difference between the folded state and the unfolded state. A negative
value of $\Delta F$ indicates the unfolded state is more favourable, whereas a
positive value for $\Delta F$ indicates the folded state is more favourable.
For the first term of the equation, the unfolded state has fewer favorable
internal contacts than the folded state, so the unfolded state has a higher
enthalpy: $\Delta H=H_{unfolded}-H_{folded}>0$. For the second term of the
equation, the unfolded state has more conformational freedom and thus a higher
entropy than the folded state: $\Delta S=S_{unfolded}-S_{folded}>0$. Since
temperature in Kelvin ($T$) is always positive, in the formula $\Delta
F=\Delta H-T\Delta S$, both $\Delta H$ and $T\Delta S$ are positive.

At low $T$, the absolute value of $T\Delta S$ will be small. Therefore, the
favorable enthalpy of the folded protein structure to $\Delta F$ will outweigh
the unfavourable entropy of the folded state at low temperatures, and the
protein will spend most time in the folded conformation, as shown in Figure
13.4c.

As $T$ increases, the absolute value of $T\Delta S$ will increase as well. At
a sufficiently high value of $T$, the favorable enthalpy of the folded protein
structure to $\Delta F$ will no longer outweigh the unfavourable entropy of
the folded state, and at higher temperatures, the protein will spend most time
in the unfolded conformation, as shown in Figure 13.4d. This decrease in
stability of the folded state at higher temperatures is exactly what is
observed in Figure 13.4a and b: as temperature increases, the protein spends
relatively more time in the unfolded state because the unfolded state becomes
more stable than the folded state. Thus, the importance of entropy increases
with increasing temperature.

Figure 13.5: Temperature dependence of protein folding. The horizontal axis
shows temperature (the results are from a simplified lattice model of protein
folding with reduced units for temperature). The vertical axis shows the
extent of folding, 1 meaning fully folded and 0 unfolded (measured by the
fraction of native contacts formed). At low temperature (T$<$0.2), the native
state is stable and therefore the protein is folded. This folding is driven by
the energetically favourable conformation where hydrophobic residues (yellow)
are ‘shielded’ in the interior of the protein structure. At high temperature
(T$>$0.4), entropic effects win out over the energetic effects. This makes the
unfolded state more stable. The unfolded state, naturally, has a higher
entropy than the folded state, but has to pay the energetic cost of exposing
hydrophobic residues to the water van Dijk et al. (2015, 2016).

Figure 13.5 shows another example of temperature dependence of protein
folding, but here a 3D lattice model is used. Despite its simplicity, we can
observe the transition from a stable folded state at low temperature, through
a transition temperature at which both states are stable to some extent, to
high temperatures where the unfolded state is the most stable van Dijk et al.
(2015). Although the model is very simple, it is realistic in the sense that
such transition temperature effects are also observed for real proteins
Thiriot et al. (2005).

###  13.6 From Microstates to Macrostates

So far, we have discussed the formal definitions of the free energy, enthalpy
and energy of a system and how they are related. An important distinction that
has been mentioned, but not yet emphasized, is the distinction between
_microstates_ and _macrostates_. Microstates are individual conformations (in
the previous sections of this chapter indicated with the subscript $i$),
whereas macrostates describe the behavior of ensembles of conformations with
very similar properties (indicated with the subscripts $A$ and $B$). For
example, the folded and unfolded states with local energy minima in Figure
13.4c,d are macrostates. The probability of a conformation calculated using
the Boltzmann equation is an example of the probability of a microstate.
Another example can be found in Figure LABEL:fig:ChIntroDyn-ensemblea, where
each individual conformation shown of the structure is a microstate, and the
collection of conformations is a macrostate. In the next two subsections we
will explain two concepts that are important to understand how to relate
microstates to macrostates, namely _order parameters_ and _ensemble averages_.

####  13.6.1 Order Parameters

In a free energy profile (such as shown in Figure 13.4c,d) the free energy is
determined for different values of an _order parameter_. An order parameter is
a quantitative measure that can distinguish the relevant states of a system
(also see panel Panel “LABEL:panel:ChMD:Analysis”). For protein folding,
frequently used order parameters are for example the number of native contacts
(e.g., the number of internal hydrogen bonds), the RMSD to the native
structure or the radius of gyration (a measure of the diameter of a flexible
molecule, which indicates how compact or extended a protein conformation is).
Order parameters are used to define the macrostates in the system. For
example, we can define an RMSD threshold to determine whether a protein is
folded or unfolded.

Choosing a suitable order parameter is very important when setting up a
research project involving simulations. Not every order parameter is able to
help answer every research question. For example, say that we are interested
in the folded and unfolded state of a protein with a large disordered loop
region (e.g., a protein with two domains separated by a linker or some
transmembrane proteins). During simulation, this region would be relatively
flexible and move around a lot. As a result, the RSMD of this structure would
be relatively high in both the folded and the unfolded state. Thus, the RMSD
might not be able to distinguish the states of interest that well, and would
not be a suitable order parameter. The number of native contacts on the other
hand, would be less sensitive to the dynamics of the loop region(s), and might
be more suitable in this case.

For the simple two-state examples of protein folding considered so far, a
single order parameter suffices, but in many cases two or several more order
parameters are used to establish a multi-dimensional free energy landscape. We
will discuss an example of this in Chapter “LABEL:ch:ChMC”, where we use
native vs. non-native contacts to describe an two-dimensional free energy
landscape (see Section LABEL:sec:ChMC:lattice “LABEL:sec:ChMC:lattice”).

####  13.6.2 Ensemble Average

Apart from free energies, we may also be interested in other properties that
are characteristic of the system (e.g., enthalpy, radius of gyration,
secondary structure content); Especially properties that can be measured
experimentally, because those show that the simulations can reproduce behavior
observed in vitro or in vivo, while providing mechanistic insights into the
observed processes.

Because the values of the property of interest may not equal for all
conformations within a state, and the probabilities of observing each
conformation within a state may not be equal, we calculate a weighted average
to describe that property for a state. This weighted average is called the
_ensemble average_. The ensemble average for a property $X$ in state $A$ is
calculated by:

$\langle X\rangle_{A}=\frac{\sum_{i\in A}{X_{i}p_{i}}}{\sum_{i\in A}{p_{i}}}$
(13.18)

where $X_{i}$ is the value of the property of interest $X$ in conformation $i$
and $p_{i}$ is the probability of conformation $i$. $i\in A$ is a notation
indicating all conformations $i$ in state $A$. Note that $i$ here denotes
microstates, and $A$ the (observable) macrostate, as introduced at the start
of this section. From our simulation, we can directly calculate an ensemble
average, by averaging a certain property $X$ over all the sampled
conformations.

One example of the application of ensemble averages is the calculation of the
enthalpy of a state. The enthalpy of a state is the ensemble average of the
internal energies of all the conformations in the ensemble of that state:

$H_{A}=\langle E\rangle_{A}=\frac{\sum_{i\in A}{E_{i}p_{i}}}{\sum_{i\in
A}{p_{i}}}$ (13.19)

where $H$ is the enthalpy and $E$ is the internal energy (more on the relation
between the two in the next section). To calculate the difference in enthalpy
we can subtract the ensemble averages of the energies of the two states:

$\Delta H_{A,B}=\langle E\rangle_{B}-\langle E\rangle_{A}$ (13.20)

Thus, ensemble averages can help us quantify structural properties related to
macrostates.

Figure 13.6: A schematic representation of different ensembles. A: NVT
ensemble, B: NPT ensemble, C: $\mu$VT ensemble, D: NVE ensemble.

###  13.7 Ensembles

In molecular simulations, we generally consider the molecules of interest in
an isolated environment (e.g., protein in a box of water with constant
temperature and volume). We will not go into detail here, but generally this
helps significantly simplify the calculations needed during the simulations.
It is important to realise that thermodynamic relations are slightly different
depending on the physical conditions chosen for the system. This gives rise to
what is known in thermodynamics as different ensembles (not to be confused
with ensemble averages), each with their own specific definition of free
energy. In this section, we will (briefly) discuss several of these ensembles,
as shown schematically in Figure 13.6. Each of them are defined by the three
parameters that are constant under the specific conditions:

  1. NVT:

Constant number of particles ($N$), volume ($V$) and temperature ($T$),
typically encountered in Monte Carlo simulations (Chapter “LABEL:ch:ChMC”).
Known as the “canonical” ensemble.

  2. NVE:

The “microcanonical ensemble” is the natural situation in molecular dynamics
simulations (Chapter “LABEL:ch:ChMD”), where we typically define a fixed
system ($N$), in a fixed size environment ($V$) and no exchange of energy
allowed ($E$).

  3. NPT:

Instead of volume, here the pressure ($P$) is constant, as well as the number
of particles ($N$) and temperature ($T$). This is closer to laboratory
conditions, and still convenient for simulations. Therefore, in MD
simulations, usually a thermostat and barostat are applied to the system such
that the simulations are performed in NPT rather than NVE (Chapter
“LABEL:ch:ChMD”).

  4. $\mu$VT:

Instead of number of particles, here the chemical potential ($\mu$) is
constant, which presumes exchange of particles with the surroundings.
Additionally, the volume ($V$) and temperature ($T$) are constant. This so-
called “grand canonical” ensemble is close to typical laboratory conditions,
but hard to achieve in simulations.

Note, that we do not go into why there are three parameters, and why they
occur in these combinations (although there are more possible). For this,
please refer to Schroeder (1999).

We will consider two ensembles, the NVT and NPT, in some more detail, as these
are common in simulations. In the NVT ensemble, the number of particles $N$,
the volume $V$ and the temperature $T$ are kept constant. This corresponds to
the so called Helmholtz ensemble or canonical ensemble, and the Helmholtz free
energy; which is what we defined in Equation 13.6, and is often written as:

$\Delta F=\Delta E-T\Delta S$ (13.21)

Here, $F$ is the Helmholtz free energy. $E$ is the internal energy in the
system. We typically cannot calculate or measure absolute values for energies,
but only differences between states. This is not a problem since these
relative free energies between states determines their stability, and
therefore, the difference is the variable that needs to be determined. In
protein folding for instance, these can be between the folded and unfolded
states (e.g., Figure LABEL:fig:ChIntroDyn-FreeEnergy), or for the case of
protein interactions, between the bound and unbound states.

In the NPT ensemble, the number of molecules $N$, pressure $P$ and temperature
$T$ are constant, and the Gibbs free energy $G$ is minimized. The Gibbs free
energy is defined as

$\Delta G=\Delta E-T\Delta S+PV=\Delta H-T\Delta S$ (13.22)

The Gibbs free energy is the Helmholtz free energy plus the product of the
pressure and volume. The term $E+PV$ is called the enthalpy and is typically
denoted as $H$. Due to the low compressibility of water, the $PV$ term can
often be neglected in the systems we are interested in. Therefore, we
sometimes consider the free energy without specifying whether we use the
Helmholtz or the Gibbs ensemble.

###  13.8 Conclusion

In this chapter we have elaborated on the concepts introduced in Chapter
“LABEL:ch:ChIntroDyn” in a more formal way using principles from
thermodynamics and statistical mechanics. We have mainly focused on how the
free energy is related to the entropy and enthalpy, and how this affects the
probability that a system is in a certain state. We have also illustrated how
changes in the temperature affect the free energy landscape. Finally, we
highlighted some key differences between microstates (individual
conformations) and macrostates (ensembles of conformations with similar
properties). In the next chapters, we will go into more depth on how
simulation methods can be applied to study free energy landscapes.

### Key concepts

  * •

Free energy $F$ is a function of the internal energy $E$, entropy $S$ and
temperature $T$: $F=E-TS$

  * •

Boltzmann’s relation gives us the probability $p_{i}$ of conformation $i$ as
function of its energy $E_{i}$: $p_{i}=\frac{1}{Z}{e^{-E_{i}/k_{B}T}}$

  * •

The free energy of a state is proportional to the probability of encountering
that state: $F_{A}\propto-k_{B}T\ln(p_{A})$

  * •

The free energy difference between two states can be calculated by: $\Delta
F_{A\rightarrow B}=-k_{B}T\ln{\left(p_{B}/p_{A}\right)}$

  * •

Ensemble averages can be used to describe the general behaviour of the
microstates belonging to a macrostate

  * •

Different thermodynamic ensembles are used to describe the physical properties
of the system of interest

  * •

An order parameter is needed to distinguish different states of the system;
only then does it become possible to calculate free energies

### Further Reading

  * •

“Physical Biology of the Cell” – Phillips et al. (2012)

  * •

“Statistical Mechanics - A survival guide” – Glazer et al. (2002)

  * •

“The Real Reason Why Oil and Water Don’t Mix” – Silverstein (1998)

  * •

“An Introduction to Thermal Physics” – Schroeder (1999)

### Author contributions

Wrote the text: | JvG, EvD, HM, KAF, SA
---|---
Created figures: | JvG, AG, KAF, SA
Review of current literature: | JvG, JV, IH, KAF, SA
Critical proofreading: | JvG, HM, KAF, JV, SA
Non-expert feedback: | JB, JG
Editorial responsibility: | JvG, KAF, SA

## References

  * Anfinsen (1973) Anfinsen, C. B. (1973).  Principles that govern the folding of protein chains.  Science, 181(4096), 223–230.
  * Ballew et al. (1996) Ballew, R. M., Sabelko, J., and Gruebele, M. (1996).  Direct observation of fast protein folding: the initial collapse of apomyoglobin.  Proceedings of the National Academy of Sciences of the United States of America, 93(12), 5759–64.
  * Daura et al. (1998) Daura, X., Jaun, B., Seebach, D., van Gunsteren, W. F., and Mark, A. E. (1998).  Reversible Peptide Folding in Solution by Molecular Dynamics Simulation.  J. Mol. Biol., 280(5), 925–932.
  * Fischer (2019) Fischer, J. (2019).  The Boltzmann Constant for the Definition and Realization of the Kelvin.  Annalen der Physik, 531(5), 1800304.
  * Frasch et al. (2022) Frasch, W. D., Bukhari, Z. A., and Yanagisawa, S. (2022).  F1FO ATP synthase molecular motor mechanisms.  Frontiers in Microbiology, 13.
  * Glazer et al. (2002) Glazer, M., Wark, J., and Schmittmann, B. (2002).  Statistical Mechanics: A Survival Guide.  American Journal of Physics, 70(12), 1274–1275.
  * Hirokawa et al. (1998) Hirokawa, T., Boon-Chieng, S., and Mitaku, S. (1998).  SOSUI: classification and secondary structure prediction system for membrane proteins.  Bioinformatics, 14(4), 378–379.
  * Kittel and Kroemer (1980) Kittel, C. and Kroemer, H. (1980).  Thermal physics.  WIT Freman: San Fransisco.
  * Mitchell (1961) Mitchell, P. (1961).  Coupling of phosphorylation to electron and hydrogen transfer by a chemi-osmotic type of mechanism.  Nature, 191(4784), 144–148.
  * Mondal et al. (2017) Mondal, P., Khamo, J. S., Krishnamurthy, V. V., Cai, Q., Zhang, K., Diao, J., Wang, Y., and Qiu, R. (2017).  Drive the Car(go)s-New Modalities to Control Cargo Trafficking in Live Cells.
  * Phillips et al. (2012) Phillips, R., Kondev, J., Theriot, J., Garcia, H. G., and Orme, N. (2012).  Physical biology of the cell.
  * Schroeder (1999) Schroeder, D. V. (1999).  An Introduction to Thermal Physics.  Addison-Wesley Publishing Company, San Francisco, CA.
  * Silverstein (1998) Silverstein, T. P. (1998).  The Real Reason Why Oil and Water Don’t Mix.  Journal of Chemical Education, 75(1), 116.
  * Thiriot et al. (2005) Thiriot, D. S., Nevzorov, A. A., and Opella, S. J. (2005).  Structural basis of the temperature transition of Pf1 bacteriophage.  Protein Science, 14(4), 1064–1070.
  * van Dijk et al. (2015) van Dijk, E., Hoogeveen, A., and Abeln, S. (2015).  The Hydrophobic Temperature Dependence of Amino Acids Directly Calculated from Protein Structures.  PLOS Computational Biology, 11(5), e1004277.
  * van Dijk et al. (2016) van Dijk, E., Varilly, P., Knowles, T. P. J., Frenkel, D., and Abeln, S. (2016).  Consistent Treatment of Hydrophobicity in Protein Lattice Models Accounts for Cold Denaturation.  Physical Review Letters, 116(7), 078101.