# Regressive Side Effects of Training
Language Models to Mimic Student Misconceptions

Shashank Sonkar
Rice University
Houston, TX
<EMAIL_ADDRESS>
&Naiming Liu
Rice University
Houston, TX
<EMAIL_ADDRESS>
&Richard G. Baraniuk
Rice University
Houston, TX
<EMAIL_ADDRESS>

###### Abstract

This paper presents a novel exploration into the regressive side effects of
training Large Language Models (LLMs) to mimic student misconceptions for
personalized education. We highlight the problem that as LLMs are trained to
more accurately mimic student misconceptions, there is a compromise in the
factual integrity and reasoning ability of the models. Our work involved
training an LLM on a student-tutor dialogue dataset to predict student
responses. The results demonstrated a decrease in the model’s performance
across multiple benchmark datasets, including the ARC reasoning challenge and
TruthfulQA, which evaluates the truthfulness of model’s generated responses.
Furthermore, the HaluEval Dial dataset, used for hallucination detection, and
MemoTrap, a memory-based task dataset, also reported a decline in the model
accuracy. To combat these side effects, we introduced a “hallucination token”
technique. This token, appended at the beginning of each student response
during training, instructs the model to switch between mimicking student
misconceptions and providing factually accurate responses. Despite the
significant improvement across all datasets, the technique does not completely
restore the LLM’s baseline performance, indicating the need for further
research in this area. This paper contributes to the ongoing discussion on the
use of LLMs for student modeling, emphasizing the need for a balance between
personalized education and factual accuracy.

##  1 Introduction

Personalized education, an approach that caters to the unique learning needs
of individuals, is increasingly becoming a key aspiration in educational
technology [1, 2, 3]. With the advent of advanced Large Language Models
(LLMs), this aspiration is inching closer to reality. LLMs, such as Llama [4]
and GPT [5, 6] models, are playing a pivotal role in this domain,
demonstrating significant potentials in various applications, including the
simulation of student behavior [7] and learning patterns [8]. However, the
road to leveraging LLMs for personalized education is challenging [9, 10, 11].

In this paper, we have identified regressive side effects arising from
training LLMs to mimic student behavior. We find that training LLMs to
replicate student misconceptions accurately is a double-edged sword. On one
hand, it creates a model that can mimic student behavior, making it a
potentially effective tool for personalized learning. On the other hand, it
significantly compromises the model’s factual integrity and reasoning ability.
These regressive side effects are a critical issue, as the primary role of any
educational model is to provide accurate and reliable information.

Figure 1: This figure illustrates a typical student-tutor interaction from the
CLASS [12] dataset. Unlike the CLASS methodology, which focuses on training a
tutor model, our study trains a ‘student model’, with the LLM predicting
student responses. This approach is motivated by the potential of personalized
education, where understanding and mimicking student behavior can lead to more
effective learning interventions. However, this approach, while effectively
mimicking student misconceptions, leads to regressive side effects such as
compromising the model’s factual integrity and diminishing its reasoning
abilities. The conversation shown here exemplifies this issue, where the
student’s response, while partially correct, contains misconceptions. To
mitigate these side effects, we introduce hallucination tokens ([hal] and
[/hal]) appended to student responses during training. These tokens instruct
the model to switch between mimicking student misconceptions and providing
factually accurate responses. Despite significant improvements, the technique
does not fully restore the model’s baseline performance, highlighting the
complexity of the issue and the need for further research.

To investigate this issue further, we conducted a comprehensive exploration
involving training LLMs on a student-tutor dialogue dataset. This dataset,
derived from the CLASS framework [12, 13, 2], comprises dialogues on biology
questions sourced from college-level textbooks. An example of the student-
tutor interaction from the dataset is illustrated in figure 1. It provides a
realistic representation of student learning patterns, featuring student
misconceptions and the tutor’s rectifications.

We used the dataset to train the latest Vicuna models (7B and 13B) [14],
state-of-the-art Llama [4] variants, to mimic student responses. However, the
training process significantly decreased the model’s performance across
various benchmark datasets, including the ARC reasoning challenge [15],
TruthfulQA [16], Hallucination Evaluation Dialogue [17], and MemoTrap [18]. We
present a detailed analysis across nine key benchmarks using the Eleuther AI
Language Model Evaluation Harness [19], a widely used framework [20] which
provides a thorough and fair assessment of generative models across a spectrum
of reasoning and general knowledge tasks. We present a detailed analysis
across nine key benchmarks using the Eleuther AI Language Model Evaluation
Harness [19], a widely used framework [20] to test generative language models
on a large number of different evaluation tasks.

To further understand the regressive side effects, we conducted a control
experiment to compare the model trained to predict tutor responses versus one
trained to predict student responses. The results showed that training the LLM
on tutor responses did not lead to any performance decline observed when
mimicking student responses. This trends highlight that the regressive side
effects are a unique challenge specific to training LLMs to replicate student
misconceptions.

To counteract the side effects, we propose to incorporate novel start and end
hallucination tokens ([hal] and [/hal]) into the LLM training process. These
tokens, placed at the beginning and end of each student response, serve as
cues to the model, instructing it when to differentiate between providing
accurate responses and replicating student misconceptions. Our results
indicate a substantial improvement in the model’s performance across all
datasets after introducing this token. However, these tokens do not fully
restore the model’s baseline performance, underscoring the complexity of the
issue.

Through our research, we have brought to the following critical contributions
in the realm of personalized education leveraging LLMs:

1\. We have uncovered and thoroughly investigated regressive side effects in
the LLMs trained for student modeling. This research highlights the
paradoxical challenge when LLMs are trained to mimic student misconceptions,
potentially compromising their factual integrity and reasoning ability.

2\. We have proposed hallucination tokens to mitigate these regressive
effects. These tokens, added to the training process, instruct the LLMs to
differentiate between mimicking student misconceptions and providing factually
accurate responses, substantially improving the model’s performance.

3\. Despite the improvements achieved with the hallucination tokens, our
research indicates that it does not fully counteract the regressive side
effects. This points to the complexity of this issue and underscores the need
for further research in this area.

Our research marks a significant step towards understanding the complexities
of using LLMs for student modeling. The findings and contributions of this
study will fuel further exploration and innovation in this domain, ultimately
refining the use of LLMs in personalized learning environments.

##  2 Related Work

The intersection of artificial intelligence and education has been an area of
active research, with a focus on developing systems that can adapt to and
support individual learners. Our work touches upon several research domains,
including student modeling, the design of intelligent tutoring systems, and
the deployment of Large Language Models (LLMs) in educational contexts.

###  2.1 Student Modeling

Student modeling has long been the cornerstone of personalized learning, with
early attempts using rule-based and Bayesian systems to predict student
knowledge and behaviors [21]. Recent advancements have shifted towards
utilizing machine learning to create more sophisticated models that can adapt
to student learning patterns over time [22, 7]. Our work builds upon these
foundations by exploring how LLMs can simulate not only the knowledge but also
the typical errors and misconceptions students have during the learning
process.

###  2.2 Intelligent Tutoring Systems (ITS)

Intelligent tutoring systems have been designed to provide immediate and
personalized instruction or feedback to learners without human intervention
[23]. The application of LLMs in ITS presents a novel opportunity to create
systems that can engage in more natural and meaningful dialogues with students
[3, 12]. Our approach diverges from traditional ITS by focusing on the
intentional generation of errors to mimic a student’s learning trajectory,
rather than solely providing expert-level instructions [24].

###  2.3 Large Language Models in Education

The use of LLMs like GPT [5] in education is a relatively new but rapidly
growing field of study [25]. These models have been employed for various
educational purposes, from generating educational content to serving as
conversational agents [26, 12]. However, the challenge of ensuring the
truthfulness and reliability of the information provided by LLMs is a
recurring concern [27]. Our research contributes to this dialogue by
investigating the impact of training LLMs to produce student-like errors and
proposing a novel ‘hallucination token’ to manage this trade-off.

###  2.4 Truthfulness and Reliability in AI

The TruthfulQA benchmark has been instrumental in highlighting the issues of
truthfulness in AI-generated content [28]. The ARC challenge further
emphasizes the complexity of reasoning required from AI systems beyond simple
fact retrieval [29]. Our work is aligned with these challenges, as we seek to
understand and improve the truthfulness and reasoning capacity of LLMs when
they are trained to replicate student behaviors.

In conclusion, our study intersects with and contributes to the existing body
of work in these areas by addressing the unique challenge of training LLMs to
authentically mimic student learning processes, including the generation of
errors. Our introduction of the “hallucination token” represents a step
forward in this domain, suggesting a new direction for future research and
development.

##  3 Methodology

Our methodology is divided into three main parts: data preparation, model
training, and the incorporation of hallucination tokens.

###  3.1 Data Preparation

The first step in our methodology involves preparing the dataset for training
the LLMs. We denote the conversation dataset as $\mathcal{D}$, which consists
of ordered pairs of tutor-student conversational turns:
$\mathcal{D}=\\{(\mathbf{x}_{1},\mathbf{y}_{1}),(\mathbf{x}_{2},\mathbf{y}_{2}),\ldots,(\mathbf{x}_{N},\mathbf{y}_{N})\\}$,
where $N$ is the total number of conversational turns. Each $\mathbf{x}$
represents a sequence of tutor utterances, and each corresponding $\mathbf{y}$
represents the student response.

The dataset is derived from the CLASS framework [12], which provides a
realistic representation of student learning patterns, featuring student
misconceptions and the tutor’s rectifications. This dataset provides a rich
source of student-tutor dialogues on biology questions sourced from college
textbooks.

###  3.2 Model Training

The second step in our methodology involves training LLMs. The LLMs are
designed to predict the next utterance given the previous conversational
context. Unlike traditional approaches that focus on the correct responses
typically output by a tutoring system, our model centers on student outputs,
which may possess a mix of correctness and misconceptions.

For an input sequence $\mathbf{x}_{i}$, the LLM aims to generate an output
sequence $\hat{\mathbf{y}}_{i}$ that resembles a student’s response. The
language modeling loss for a single data pair is defined by the negative log
likelihood:

$\mathcal{L}(\mathbf{y}_{i},\hat{\mathbf{y}}_{i})=-\sum_{t=1}^{|\mathbf{y}_{i}|}\log
p\left(y_{i,t}\middle|\mathbf{x}_{i},\mathbf{y}_{i,<t};\theta\right)$

where $\mathbf{y}_{i,<t}$ indicates the tokens in the true response preceding
the current token $y_{i,t}$, and $\theta$ encapsulates the parameters of the
LLM. The overall training loss is the sum over the entire dataset:

$\mathcal{L}_{\text{total}}=\sum_{i=1}^{N}\mathcal{L}\left(\mathbf{y}_{i},\hat{\mathbf{y}}_{i}\right)$

###  3.3 Incorporation of Hallucination Tokens

The third step in our methodology involves the incorporation of hallucination
tokens. To enhance the LLM’s ability to generate responses that simulate
student behaviors, including providing incorrect or uncertain information, we
introduce hallucination token markers. Each student response in the dataset is
enriched with these markers to indicate the beginning and the end of the
potentially inaccurate content.

Let $\mathbf{y}_{i}$ be an original student response sequence from the
dataset. The augmented student response $\tilde{\mathbf{y}}_{i}$ used for
training is constructed by prepending and appending hallucination tokens [hal]
and [/hal], respectively:

$\tilde{\mathbf{y}}_{i}=\left[\texttt{[hal]},\mathbf{y}_{i,1},\mathbf{y}_{i,2},\ldots,\mathbf{y}_{i,|\mathbf{y}_{i}|},\texttt{[/hal]}\right]$

In the modified training regime, the LLM predicts the sequence
$\hat{\mathbf{y}}_{i}$ such that it learns to include these tokens,
effectively grasping the context of student uncertainty or errors. These
tokens serve as cues to the model, instructing it when to differentiate
between providing accurate responses and replicating student misconceptions.

##  4 Experiments and Discussion

Table 1: Performance of Vicuna models on TruthfulQA tasks. The table compares
the performance of the original vicuna model, the control model trained to
mimic tutor responses in biology (tutor) the model trained to mimic tutor
responses in biology (tutor), and the model trained with hallucination tokens
(student-hal). The results are presented for three different settings: MC1,
MC2, and Generation. MC1 refers to a setting where there is only one correct
answer to a question, while MC2 refers to a setting where there are multiple
correct answers. For these settings, the performance is measured in terms of
accuracy. The generation setting involves the model generating 1-2 sentence
answers, with performance evaluated using BLEU and ROUGE scores. The results
highlight the significant drop in performance when the model is trained to
mimic student responses, demonstrating a regressive side effect in terms of
truthfulness. However, the substantial recovery in performance with the
introduction of hallucination tokens suggests a promising strategy to mitigate
these regressive effects.

Dataset |  | TQA MC1
---
(Single-true)
| TQA MC2
---
(Multi-true)
TruthfulQA (TQA) Generation
Metric | Accuracy | Accuracy | BLEU |  | ROUGE
---
(unigram)
| ROUGE
---
(bigram)
| ROUGE
---
(LCS)
vicuna-7b-v1.5 | 32.93 | 50.37 | 49.69 | 51.41 | 45.90 | 50.55
tutor-7b | 34.64 | 52.43 | 42.72 | 47.12 | 37.94 | 45.29
student-7b | 23.75 | 36.14 | 24.60 | 29.74 | 14.32 | 28.89
student-hal-7b | 29.25 | 44.68 | 43.94 | 47.61 | 36.47 | 45.53
vicuna-13b-v1.5 | 35.01 | 50.87 | 47.12 | 50.18 | 44.92 | 49.08
tutor-13b | 34.76 | 52.20 | 42.84 | 48.71 | 38.80 | 46.76
student-13b | 22.15 | 33.93 | 15.18 | 18.12 | 6.12 | 17.75
student-hal-13b | 27.91 | 41.46 | 39.29 | 42.35 | 33.66 | 42.96

In this section, we present our experimental methodology and discuss the
findings in detail. The experiments were designed to explore the regressive
side effects of training LLMs to mimic student behavior and to assess the
effectiveness of our proposed hallucination tokens in mitigating these
effects.

###  4.1 Experimental Setup

We trained the Vicuna 7B and 13B models [14], one of the best open-source
LLMs, on a student-tutor dialogue dataset derived from the CLASS [12]
framework. This dataset, which provides a realistic representation of student
learning patterns, misconceptions, and the tutor’s rectifications, was used to
fine-tune the models to generate outputs that mimic student dialogue. The
dataset contains $648$ conversations, which sums up to a total of 20K student-
tutor interactions. Average length of conversations is around $~{}400$ words,
only including the student and tutor fields in the conversation template.

The models were evaluated across seven key benchmarks using the Eleuther AI
Language Model Evaluation Harness [19]. These benchmarks include the
TruthfulQA [16], ARC [15], HellaSwag [30], Winogrande [31], MMLU [32],
HaluEval Dialogue [17], and MemoTrap [18]. Each of these benchmarks tests
different aspects of the model’s performance, including its truthfulness,
reasoning abilities, ability to recognize hallucinations, and memory-based
task performance.

###  4.2 In-depth Analysis: TruthfulQA

In the realm of educational technology, the veracity of information provided
by a model is of paramount importance. Misinformation or misconceptions can
lead to significant learning detriments, making the truthfulness of a model’s
responses a critical factor in its effectiveness as an educational tool.
Therefore, we chose to conduct an in-depth analysis of our models’ performance
on the TruthfulQA benchmark.

TruthfulQA is a benchmark specifically designed to measure the truthfulness of
a language model’s responses across a wide range of categories. It tests the
model’s ability to avoid generating false answers learned from imitating human
texts, a challenge that is particularly relevant to our study. Given the
importance of truthfulness in educational contexts and the unique challenges
posed by training models to mimic student misconceptions, we believe that a
rigorous analysis of our models’ performance on TruthfulQA is warranted.

In this section, we present our findings from the TruthfulQA benchmark,
exploring the impact of training models to mimic student behavior and the
effectiveness of our proposed hallucination tokens in mitigating any negative
effects. We delve into the results from the multiple-choice and generation
tasks within TruthfulQA, providing a comprehensive view of our models’
truthfulness in different contexts.

TruthfulQA Multiple-Choice Setting 1 (MC1) Findings. In the first multiple-
choice setting, where there is a single correct label, the student-7b model’s
accuracy decreased by 15 points compared to the vicuna-7b model. However, the
introduction of hallucination tokens led to a significant recovery in
performance. This finding is particularly relevant in the context of
education, where maintaining the truthfulness of responses is crucial. The
improvement with hallucination tokens suggests that it is possible to train
models that can both simulate student behavior and adhere to factual accuracy,
a key consideration for deploying LLMs in educational settings.

TruthfulQA Multiple-Choice Setting 2 (MC2) Findings. In the second multiple-
choice setting, where multiple correct labels are possible, we observed a
similar trend to the MC1 setting. The student-7b model experienced a
significant drop in accuracy, from 50.37% in the vicuna-7b model to 36.14%
when trained to mimic student responses. However, the introduction of
hallucination tokens led to a notable improvement in performance, with the
student-7b model’s accuracy recovering to 44.68%.

This recovery is particularly relevant in the context of education, where
multiple perspectives or answers might be correct. The ability of the model to
navigate such complexities while maintaining truthfulness is crucial. The
improvement with hallucination tokens suggests that it is possible to train
models that can both simulate student behavior and adhere to factual accuracy,
a key consideration for deploying LLMs in educational settings.

TruthfulQA Generation Findings. For the TruthfulQA generation task, where the
model is tasked with generating 1-2 sentence answers, we employed ROUGE scores
to evaluate performance due to the generative nature of the task. The
student-7b model saw a significant decrease in ROUGE scores, from 51.41 in the
vicuna-7b model to 29.74, indicating a substantial loss in the ability to
generate truthful, relevant responses. However, the introduction of
hallucination tokens led to a significant recovery in performance, with ROUGE
scores improving to 47.61.

This finding is crucial for educational technology as LLMs are increasingly
used as generative agents to create educational content, provide explanations,
and engage in dialogue with students. The ability to generate truthful,
accurate responses is fundamental to their utility in these contexts. The
recovery observed with hallucination tokens highlights their potential to
enable LLMs to simulate student misconceptions for personalized learning
without sacrificing the quality and truthfulness of their output.

Table 2: Comparative performance of Large Language Models (LLMs) on various
benchmarks before and after the introduction of hallucination tokens, with a
control experiment involving tutor models. The table presents the performance
of Vicuna 7B models across five key benchmarks: ARC Reasoning, Hallucination
Evaluation Dialogue (HaluDial), Hallucination Memorization Trap (MemoTrap),
TruthfulQA (TQA), HellaSwag (HSwag), MMLU, and Winogrande (WinoG). The numbers
in parentheses (e.g., 25-S in ARC) represent the number of few-shot examples
provided to the model during evaluation. The performance is measured in terms
of accuracy percentage. The table compares the performance of the original
vicuna models, tutor models, student models, and student models trained with
hallucination tokens (student-hal). The results highlight the significant drop
in performance when the model is trained to mimic student responses,
demonstrating regressive side effects across multiple tasks. However, the
introduction of hallucination tokens leads to a substantial recovery in
performance across all benchmarks, underscoring their potential in mitigating
these regressive effects.

Model | Avg |  | ARC
---
(25-S)
| HaluDial
---
(0-S)
| MemoTrap
---
(0-S)
| TQA
---
(6-S)
| HSwag
---
(10-S)
| MMLU
---
(5-S)
| WinoG
---
(5-S)
vicuna-7b-v1.5 | 60.8 | 53.24 | 69.08 | 68.48 | 50.34 | 77.39 | 51.04 | 72.14
tutor-7b | 61.0 | 52.13 | 68.81 | 69.23 | 52.3 | 78.07 | 51.32 | 71.19
student-7b | 55.4 | 40.61 | 65.39 | 65.28 | 36.87 | 76.72 | 50.77 | 71.9
student-hal-7b | 58.0 | 45.48 | 70.73 | 66.88 | 44.83 | 77.21 | 51.54 | 72.03
vicuna-13b-v1.5 | 64.2 | 57.08 | 73.78 | 67.2 | 51.51 | 81.24 | 56.67 | 74.66
tutor-13b | 64.7 | 57.34 | 73.92 | 66.13 | 52.99 | 81.51 | 57.02 | 74.35
student-13b | 58.2 | 46.5 | 66.97 | 65.81 | 35.0 | 80.36 | 57.06 | 72.22
student-hal-13b | 60.3 | 48.63 | 72.98 | 66.13 | 42.75 | 80.28 | 56.4 | 73.16

###  4.3 Benchmark Evaluation

Following the exploration of TruthfulQA settings, we delve into the
performance of our models across a broader range of benchmarks as detailed in
Table 2. These benchmarks—ARC, HaluEval Dial, MemoTrap, MMLU, HellaSwag, and
Winogrande—offer a comprehensive view of the models’ capabilities in
reasoning, detecting hallucinations, avoiding memorization traps, and
understanding commonsense, respectively.

AI2 Reasoning Challenge (ARC) Findings. ARC serves as a rigorous benchmark to
evaluate a model’s reasoning capabilities through a set of grade-school
science questions. These questions are designed to test not just the factual
knowledge of the models but also their ability to apply this knowledge in
reasoning through complex, multi-step problems. The ARC dataset is
particularly relevant in educational contexts as it mirrors the type of
critical thinking and problem-solving skills students are expected to develop.

In our experiments, the performance of models trained to mimic student
responses on the ARC benchmark experienced a notable decline. Specifically,
the vicuna-7b model saw its accuracy decrease from 53.24% to 40.61% when
trained on student dialogues. This significant drop in performance highlights
a critical concern: training LLMs to replicate student behavior, including
misconceptions, can severely impair their reasoning abilities.

However, our introduction of hallucination tokens into the training process
presents a silver lining. Our approach led to a partial recovery in the ARC
performance, with accuracy improving to 45.48%. While this does not fully
restore the model’s baseline performance, it represents a significant step
towards mitigating the regressive side effects of training LLMs on student
data.

Hallucination Evaluation (HaluEval) Dialogue Findings. The HaluEval Dial
benchmark is designed to assess a model’s ability to recognize and avoid
hallucinations in generated responses, particularly in the context of
knowledge grounded dialogue tasks. Hallucinations in this context refer to the
model generating information that is not supported by the input data or
general knowledge, a critical issue when models are used in educational
settings where accuracy is paramount. Our findings indicate that training
models to mimic student responses led to a decrease in performance on the
HaluEval Dial benchmark. Specifically, the vicuna-7b model saw its accuracy
drop from 69.0% to 65.39%. However, the introduction of hallucination tokens
demonstrated a remarkable ability to counteract this effect, with the
student-7b model’s accuracy improving to 70.73%.

Memorization Traps (MemoTrap) Findings. MemoTrap is a benchmark designed to
test whether language models can avoid memorization traps by prompting them to
complete well-known proverbs with endings that deviate from the commonly used
ones. This benchmark is particularly relevant for evaluating a model’s ability
to generate creative and contextually appropriate responses rather than
relying on rote memorization.

In our experiments, training models to mimic student responses resulted in a
decrease in performance on the MemoTrap benchmark. The vicuna-7b model’s
accuracy decreased from 68.48% to 65.28%, indicating that training on student
dialogues might encourage the model to rely more on memorization rather than
understanding and applying knowledge flexibly. The introduction of
hallucination tokens led to a slight improvement, with accuracy increasing to
66.88%.

MMLU, HellaSwag, and Winogrande Findings. The performance of models on the
MMLU, HellaSwag, and Winogrande benchmarks remained relatively stable,
regardless of whether they were trained to mimic tutor or student responses.

The nuanced impact observed in other benchmarks underscores the importance of
carefully considering the training data and methodologies used when developing
LLMs for educational purposes. The introduction of hallucination tokens
emerges as a promising strategy for mitigating some of the regressive side
effects associated with training models to mimic student behavior, ensuring
that they can still serve as effective tools for personalized learning without
compromising on factual accuracy or reasoning capabilities.

###  4.4 Control Models: Tutor Models

To further understand the regressive side effects of training LLMs to mimic
student behavior, we conducted a control experiment by training models to
predict tutor responses. This experiment aimed to compare the performance of
models trained to predict tutor responses versus those trained to predict
student responses. The tutor models were trained using the same student-tutor
dialogue dataset derived from the CLASS framework [12]. However, instead of
training the models to mimic student responses, we trained them to predict the
responses of the tutor. Our findings, as shown in Table 2, revealed that
training the LLMs on tutor responses did not lead to the same performance
decline observed when mimicking student responses. This result underscores
that the regressive side effects are a unique challenge specific to training
LLMs to replicate student misconceptions.

##  5 Conclusion

In this study, we have delved into the challenges of training LLMs to mimic
student behavior, with a particular focus on the regressive side effects that
emerge in this process. Our findings reveal a complex paradox: as LLMs become
more adept at replicating student misconceptions, they tend to compromise on
their factual integrity and reasoning ability. Our experiments demonstrated a
notable decrease in the model’s performance across various key benchmark
datasets like ARC Reasoning Challenge and TruthfulQA. To mitigate these
regressive side effects, we introduced a novel technique involving the use of
hallucination tokens during the training process. Our results indicate that
the introduction of these tokens leads to a substantial improvement in the
model’s performance across all datasets. However, it’s important to note that
despite the significant improvements achieved with the hallucination tokens,
they do not fully restore the model’s baseline performance. This outcome
underscores the complexity of the problem and highlights the need for a more
nuanced approach when training LLMs to mimic student behavior. While we have
made some strides in addressing the regressive side effects, our work is just
the beginning. We believe that our findings will pave the way for further
research in this domain, ultimately contributing to the refinement of LLMs in
personalized learning environments.

##  6 Ethics

The ethical implications of training models on incorrect data are profound and
demand conscientious exploration in future work. As our models find
application in real-world educational settings, the delineation between the
effective simulation of student behaviors and the propagation of
misinformation will need to be continually assessed and refined. Our research
has thus laid the groundwork for a new pedagogical paradigm, where AI becomes
a symbiotic partner in the complex choreography of learning and teaching.

## Acknowledgments

This work was supported by NSF grants 1842378, ONR grant N0014-20-1-2534,
AFOSR grant FA9550-22-1-0060, and a Vannevar Bush Faculty Fellowship, ONR
grant N00014-18-1-2047.

## References

  * [1] Sankalan Pal Chowdhury, Vilém Zouhar, and Mrinmaya Sachan.  Scaling the authoring of autotutors with large language models.  arXiv preprint arXiv:2402.09216, 2024.
  * [2] Shashank Sonkar, Kangqi Ni, Sapana Chaudhary, and Richard G Baraniuk.  Pedagogical alignment of large language models.  arXiv preprint arXiv:2402.05000, 2024.
  * [3] Robin Schmucker, Meng Xia, Amos Azaria, and Tom Mitchell.  Ruffle&riley: Towards the automated induction of conversational tutoring systems.  arXiv preprint arXiv:2310.01420, 2023.
  * [4] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [5] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.  Sparks of Artificial General Intelligence: Early experiments with GPT-4.  arXiv preprint arXiv:2303.12712, 2023.
  * [6] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  * [7] Naiming Liu, Zichao Wang, Richard Baraniuk, and Andrew Lan.  Open-ended knowledge tracing for computer science education.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3849–3862, 2022.
  * [8] Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al.  Chatgpt for good? on opportunities and challenges of large language models for education.  Learning and Individual Differences, 103:102274, 2023.
  * [9] Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D Hwang, et al.  Faith and fate: Limits of transformers on compositionality.  arXiv preprint arXiv:2305.18654, 2023.
  * [10] Shashank Sonkar and Richard G. Baraniuk.  Deduction under perturbed evidence: Probing student simulation (knowledge tracing) capabilities of large language models.  CEUR Workshop Proceedings, 3487:26–33, 2023.
  * [11] Allison Wang, Ethan Prihar, and Neil Heffernan.  Assessing the ality of large language models in generating mathematics explanations.  Learning@ Scale (L@ S’23), July 20–22, 2023, Copenhagen, Denmark., 2023.
  * [12] Shashank Sonkar, Naiming Liu, Debshila Mallick, and Richard Baraniuk.  CLASS: A Design Framework for Building Intelligent Tutoring Systems Based on Learning Science principles.  In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1941–1961, Singapore, December 2023.
  * [13] Shashank Sonkar, Xinghe Chen, Myco Le, Naiming Liu, Debshila Basu Mallick, and Richard Baraniuk.  Code soliloquies for accurate calculations in large language models.  In Proceedings of the 14th Learning Analytics and Knowledge Conference, pages 828–835, 2024.
  * [14] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  * [15] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
  * [16] Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods, 2022.
  * [17] Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen.  HaluEval: A large-scale hallucination evaluation benchmark for large language models.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464, Singapore, December 2023. Association for Computational Linguistics.
  * [18] Ian McKenzie, Alexander Lyzhov, Alicia Parrish, Ameya Prabhu, Aaron Mueller, Najoung Kim, Sam Bowman, and Ethan Perez.  The inverse scaling prize, 2022.
  * [19] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, 12 2023.
  * [20] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf.  Open llm leaderboard, 2023.
  * [21] Martha C Polson and J Jeffrey Richardson.  Foundations of intelligent tutoring systems.  Psychology Press, 2013.
  * [22] Ryan SJD Baker, Kalina Yacef, et al.  The state of educational data mining in 2009: A review and future visions.  Journal of educational data mining, 1(1):3–17, 2009.
  * [23] Beverly Park Woolf.  Building intelligent interactive tutors: Student-centered strategies for revolutionizing e-learning.  Morgan Kaufmann, 2010.
  * [24] Kurt VanLehn.  The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems.  Educational psychologist, 46(4):197–221, 2011.
  * [25] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [26] Neil T Heffernan and Cristina Lindquist Heffernan.  The assistments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching.  International Journal of Artificial Intelligence in Education, 24:470–497, 2014.
  * [27] Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods.  arXiv preprint arXiv:2109.07958, 2021.
  * [28] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  arXiv preprint arXiv:1803.05457, 2018.
  * [29] Oren Etzioni, Anthony Fader, Janara Christensen, Stephen Soderland, and Mausam Mausam.  Open information extraction: The second generation.  In IJCAI, volume 11, pages 3–10, 2011.
  * [30] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?, 2019.
  * [31] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  WINOGRANDE: an adversarial winograd schema challenge at scale, 2019.
  * [32] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  CoRR, abs/2009.03300, 2020.


OPES, MetaD, CVs, FES

# Exploration vs Convergence Speed
in Adaptive-bias Enhanced Sampling

Michele Invernizzi  Freie Universität Berlin, 14195 Berlin, Germany
<EMAIL_ADDRESS>Michele Parrinello  Italian Institute of
Technology, 16163 Genova, Italy

###### Abstract

In adaptive-bias enhanced sampling methods, a bias potential is added to the
system to drive transitions between metastable states. The bias potential is a
function of a few collective variables and is gradually modified according to
the underlying free energy surface. We show that when the collective variables
are suboptimal, there is an exploration-convergence tradeoff, and one must
choose between a quickly converging bias that will lead to fewer transitions,
or a slower to converge bias that can explore the phase space more efficiently
but might require a much longer time to produce an accurate free energy
estimate. The recently proposed On-the-fly Probability Enhanced Sampling
(OPES) method focuses on fast convergence, but there are cases where fast
exploration is preferred instead. For this reason, we introduce a new variant
of the OPES method that focuses on quickly escaping metastable states, at the
expense of convergence speed. We illustrate the benefits of this approach on
prototypical systems and show that it outperforms the popular metadynamics
method.

###### keywords:

opes, metadynamics, umbrella sampling, collective variables, free energy

##  1 Introduction

Molecular dynamics has become a valuable tool in the study of a variety of
phenomena in physics, chemistry, biology, and materials science. One of the
long-standing challenges in this important field is the sampling of rare
events, such as chemical reactions or conformational changes in biomolecules.
To simulate effectively such systems, many enhanced sampling methods have been
developed. An important class of such methods is based on an adaptive-bias
approach and includes adaptive umbrella sampling1, metadynamics (MetaD) 2, 3,
and the recently developed on-the-fly probability enhanced sampling (OPES)4,
5, 6. Adaptive-bias methods operate by adding to the system’s energy
$U(\mathbf{R})$ an external bias potential $V=V(\mathbf{s})$, that is a
function of a set of collective variables (CVs), $\mathbf{s}$. The CVs,
$s=s(\mathbf{R})$, depend on the atomic coordinates $\mathbf{R}$ and are meant
to describe the slow modes associated with the rare event under study. They
also define a free energy surface (FES), $F(\mathbf{s})=-\frac{1}{\beta}\log
P(\mathbf{s})$, where $\beta=(k_{B}T)^{-1}$ is the inverse Boltzmann factor
and $P(\mathbf{s})$ the marginal $\mathbf{s}$ distribution,
$P(\mathbf{s})\propto\int e^{-\beta
U(\mathbf{R})}\delta[\mathbf{s}-\mathbf{s}(\mathbf{R})]\,d\mathbf{R}$. The
bias is periodically updated until it converges to a chosen form. A popular
choice is to have it exactly offset the underlying FES,
$V(\mathbf{s})=-F(\mathbf{s})$, so that the resulting $\mathbf{s}$
distribution is uniform.

The main limitation of adaptive-bias methods is that finding good collective
variables is sometimes difficult and a bad choice of CVs might not promote the
desired transitions in an affordable computer time. In practical applications
one generally has to live with suboptimal CVs7 that still can drive
transitions, but do not include some of the slow modes. In this case, applying
a static bias cannot speed up the slow modes that are not accounted for, and
thus transitions remain quite infrequent. It is sometimes possible to achieve
a faster transition rate by using a rapidly changing bias, which can push the
system out of a metastable state through a high free energy pathway, different
from the energetically favoured one. However, unless one wishes to deal
explicitly with out-of-equilibrium statistics8, 9, 10, it is not possible to
obtain reliable information about the system while the bias changes in a non-
adiabatic fashion. To estimate the FES and other observables one must let the
adaptive-bias method approach convergence, and as the bias becomes quasi-
static, transitions inevitably become less frequent.

We refer to this situation as an exploration-convergence tradeoff that every
adaptive-bias enhanced sampling method has to deal with, when suboptimal CVs
are used. Some methods, like OPES, focus more on quickly converging to a
quasi-static bias potential and thus obtaining an efficiently reweighted FES,
while others, like metadynamics, focus more on escaping metastable states and
exploring the phase space. We will demonstrate this qualitative difference on
some prototypical systems. For simplicity, in the paper we only consider the
well-tempered variant of metadynamics3, but in the Supporting Information (SI)
we provide examples that use the original non-tempered MetaD2 and other
popular variants, such as parallel-bias MetaD11.

We propose here a variant of OPES, named OPES-explore, that focuses on rapid
exploration, rather than on fast convergence. It shares many features with the
original OPES, and is designed to be an easy-to-use tool requiring few input
parameters. To this end, we also introduce an adaptive bandwidth algorithm
that can be used in both OPES variants, and further reduces the number of
input parameters that need to be specified. The detailed description of the
adaptive bandwidth algorithm is left to the SI. All OPES simulations presented
make use of this algorithm.

##  2 The OPES method

The enhanced sampling method OPES works by adding an adaptive-bias potential
to the energy of the system, so as to modify the Boltzmann probability
distribution into a desired target one. Most adaptive-bias methods aim at
sampling uniformly the CV space, but it has been shown that choosing a
different target distribution could be advantageous12, 13. There are two
different classes of target distributions that can be sampled with OPES;
metadynamics-like and replica-exchange-like. We will consider here only the
former type, introduced in Ref. 4, but the interested reader can find in Ref.
5 information about OPES for replica-exchange-like sampling.

To define a metadynamics-like target distribution, one has to choose a set of
collective variables, $s=s(\mathbf{R})$. As stated in the introduction, the
unbiased marginal probability along such CVs is $P(\mathbf{s})\propto\int
e^{-\beta
U(\mathbf{R})}\delta[\mathbf{s}-\mathbf{s}(\mathbf{R})]\,d\mathbf{R}$, where
$U(\mathbf{R})$ is the potential energy. The target distribution is then
defined by requiring a specific marginal probability distribution over the
CVs, $p^{\text{tg}}(\mathbf{s})$. Consequently, the desired bias potential is
written as:

$V(\mathbf{s})=-\frac{1}{\beta}\log\frac{p^{\text{tg}}(\mathbf{s})}{P(\mathbf{s})}\,,$
(1)

so that $\int
e^{-\beta[U(\mathbf{R})+V(\mathbf{s})]}\delta[\mathbf{s}-\mathbf{s}(\mathbf{R})]\,d\mathbf{R}\propto
p^{\text{tg}}(\mathbf{s})$. A typical choice for $p^{\text{tg}}(\mathbf{s})$
is the well-tempered distribution3:

$p^{\text{WT}}(\mathbf{s})\propto[P(\mathbf{s})]^{1/\gamma}\,,$ (2)

where the bias factor $\gamma>1$ controls how much the original distribution
is smoothed out. In the limit of $\gamma=\infty$ one targets a uniform
distribution.

The core idea of OPES is to update self-consistently the estimate of the
probability distributions and of the bias potential, in an on-the-fly fashion
similar to self-healing umbrella sampling 14. The estimate of the unbiased
probability is obtained via a weighted kernel density estimation, so that at
step $n$ one has:

$P_{n}(\mathbf{s})=\frac{\sum_{k}^{n}w_{k}G(\mathbf{s},\mathbf{s}_{k})}{\sum_{k}^{n}w_{k}}\,,$
(3)

where the weights $w_{k}$ are given by $w_{k}=e^{\beta
V_{k-1}(\mathbf{s}_{k})}$, and the Gaussian kernels
$G(\mathbf{s},\mathbf{s}^{\prime})=h\exp\left[-\frac{1}{2}(\mathbf{s}-\mathbf{s}^{\prime})^{T}\boldsymbol{\Sigma}^{-1}(\mathbf{s}-\mathbf{s}^{\prime})\right]$
have a diagonal covariance matrix $\Sigma_{ij}=\sigma^{2}_{i}\delta_{ij}$ and
fixed height $h=\prod_{i}\left(\sigma_{i}\sqrt{2\pi}\right)^{-1}$. The number
of kernels to represent $P_{n}(\mathbf{s})$ would grow linearly with
simulation time, but this is avoided thanks to an on-the-fly kernel
compression algorithm15, as described in detail in the supporting information
of Ref. 4. The compression algorithm also allows for the bandwidth of the
kernels to shrink over time, as the effective sample size
$N_{\text{eff}}^{(n)}=\left(\sum_{k}^{n}w_{k}\right)^{2}/\sum_{k}^{n}w_{k}^{2}$
grows. The idea is to start with a coarse estimate of $P(\mathbf{s})$ and then
refine it as more data are available. The kernel bandwidth of the $i$-th CV at
step $n$ is:

$\sigma_{i}^{(n)}=\sigma_{i}^{(0)}[N_{\text{eff}}^{(n)}(d+2)/4]^{-1/(d+4)}\,,$
(4)

where $d$ is the total number of CVs.

The instantaneous bias is based on the probability estimate
$P_{n}(\mathbf{s})$, following Eq. (1) and using the approximation
$p^{\text{WT}}(\mathbf{s})\propto[P_{n}(\mathbf{s})]^{1/\gamma}$, one has:

$V_{n}(\mathbf{s})=(1-1/\gamma)\frac{1}{\beta}\log\left(\frac{P_{n}(\mathbf{s})}{Z_{n}}+\epsilon\right)\,,$
(5)

where $\epsilon$ is a regularization term that limits the maximum possible
absolute value of the bias potential, and $Z_{n}$ can be understood as a
normalization of $P_{n}(\mathbf{s})$ over the CV space thus far explored,
$\Omega_{n}$:

$Z_{n}=\frac{1}{|\Omega_{n}|}\int_{\Omega_{n}}P_{n}(\mathbf{s})\,d\mathbf{s}\,.$
(6)

This integral is calculated approximately as a sum of $P_{n}$ over the
compressed kernels, as explained in the supplementary information of Ref. 4.
The intuitive idea is that new kernels are added to the compressed
representation only when a new region of CV space is sampled (otherwise they
are merged with existing ones), thus the explored CV-space volume
$|\Omega_{n}|$, is approximately proportional to the total number of
compressed kernels.

The introduction of the $Z_{n}$ term is one of the key innovations of OPES. In
similar methods, once a new metastable state is found one often sees a
dramatic increase of the exit time, compared to the first one16 (see SI, Fig.
S5). This exit time problem is present also when the CVs are optimal, and
should not be confused with the exploration-convergence tradeoff that is the
primary concern of this paper. Other convergence-focused methods introduce
extra parameters to tackle this problem, for example in transition-tempered
metadynamics17 prior knowledge of the position of all metastable states is
required. Instead, OPES avoids the exit time problem by taking into account
the expansion of the CV space via the $Z_{n}$ term, which allows the bias to
adjust more quickly when a new CV-space region is sampled4.

At the start of an OPES simulation only a handful of parameters needs to be
chosen, namely the initial kernel bandwidth, the pace at which the bias is
updated, and the approximate FES barrier height that needs to be overcome.
From this last information a prescription is given to automatically set the
values $\gamma$ and $\epsilon$. The number of parameters can be reduced even
further if one uses, as we shall do here, the adaptive bandwidth algorithm
discussed in the SI.

##  3 An exploratory OPES variant

We present now a new OPES variant called OPES-explore which, compared to the
original OPES formulation, leads to a faster exploration of the phase space at
the cost of a slower convergence. We have recalled that the $Z_{n}$ term
allows OPES to quickly adapt the bias when a metastable state is found in a
previously unexplored region of CV space. However, if the CVs used are
suboptimal, it may happen that a new metastable state is found in an already
explored $\mathbf{s}$ region18, 19. In such a case, the $Z_{n}$ term remains
constant and is therefore ineffective in accelerating the exit time. Instead,
to encourage a rapid exit, one would need a method that allows the bias to
significantly change shape again. Fortunately, it is possible to achieve this
exploratory behaviour simply by making a minimal change to the OPES protocol,
which gives rise to the OPES-explore variant.

In formulating OPES-explore, we restrict ourselves to the case of using as
target the well-tempered distribution,
$p^{\text{tg}}(\mathbf{s})=p^{\text{WT}}(\mathbf{s})$, Eq. (2). In OPES, the
bias is expressed as a function of $P_{n}(\mathbf{s})$, the on-the-fly
estimate of the unknown equilibrium distribution $P(\mathbf{s})$. At the
beginning of the simulation this estimate is not reliable, but it improves
over time and converges in a self-consistent way. In OPES-explore instead, one
builds the bias starting from the on-the-fly estimate of the distribution that
is being sampled in the biased simulation:

$p^{\text{WT}}_{n}(\mathbf{s})=\frac{1}{n}\sum_{k}^{n}G(\mathbf{s},\mathbf{s}_{k})\,,$
(7)

where $\mathbf{s}_{k}$ is the CVs value sampled at step $k$. As the simulation
converges, $p^{\text{WT}}_{n}(\mathbf{s})$ approaches the target well-tempered
distribution $p^{\text{WT}}(\mathbf{s})$. Thus, analogously to Sec. 2, we use
the approximation
$P(\mathbf{s})\propto[p^{\text{WT}}_{n}(\mathbf{s})]^{\gamma}$ and write the
bias according to Eq. (1):

$V_{n}(\mathbf{s})=(\gamma-1)\frac{1}{\beta}\log\left(\frac{p^{\text{WT}}_{n}(\mathbf{s})}{Z_{n}}+\epsilon\right)\,,$
(8)

where $\epsilon$ and $Z_{n}$ have been added for the same reasons as in Eq.
(5). We notice that the expressions in Eqs. (3) and (7), which define the
probability estimates used in the two OPES schemes, converge respectively to
$P(\mathbf{s})$ and $p^{\text{WT}}(\mathbf{s})$ only within the self-
consistent scheme where the simulation runs with a bias that is updated on-
the-fly according to Eqs. (5) and (8) respectively. Both OPES variants are
applications of the general Eq. (1), but OPES estimates on-the-fly
$P(\mathbf{s})$ and uses it to calculate the bias, while OPES-explore does the
same but with $p^{\text{WT}}(\mathbf{s})\propto[P(\mathbf{s})]^{1/\gamma}$.

The free energy surface as a function of the CVs can be estimated in two
distinct ways, either directly from the probability estimate,
$F_{n}(\mathbf{s})=-\gamma\frac{1}{\beta}\log p_{n}^{\text{WT}}(\mathbf{s})$,
or via importance sampling reweighting, e.g. using a weighted kernel density
estimation,

$F_{n}(\mathbf{s})=-\frac{1}{\beta}\log\sum_{k}^{n}e^{\beta
V_{k-1}(\mathbf{s}_{k})}G(\mathbf{s},\mathbf{s}_{k})\,.$ (9)

In standard OPES these two estimates are equivalent, while in OPES-explore
(similarly to MetaD) they can differ significantly in the first part of the
simulation until they eventually converge to the same estimate.

|
---|---
(a) OPES | (b) OPES EXPLORE
Figure 1: Time evolution of a typical simulation of alanine dipeptide in
vacuum using the two OPES variants with the dihedral angles $\phi$ and $\psi$
as CVs. For each method, the compressed kernels are shown on the left with the
point size indicating the adaptive bandwidth, and the corresponding free
energy estimate $F_{n}(\phi,\psi)$ on the right. (a) In the original OPES,
kernels make up the unbiased distribution estimate $P_{n}(\phi,\psi)$ and
$F_{n}(\phi,\psi)=-\frac{1}{\beta}\log P_{n}(\phi,\psi)$, while (b) in OPES-
explore kernels make up the sampled distribution estimate
$p^{\text{WT}}_{n}(\phi,\psi)$ and
$F_{n}(\phi,\psi)=-\gamma\frac{1}{\beta}\log p^{\text{WT}}_{n}(\phi,\psi)$.
All $F_{n}(\phi,\psi)$ are shifted to have zero minimum. Notice how OPES-
explore requires fewer kernels and visits higher FES regions.

In figure 1 we contrast an OPES and OPES-explore simulation of alanine
dipeptide in vacuum, which has become a standard test for enhanced sampling
methods. Both simulations have the same input parameters and use the adaptive
bandwidth scheme described in the SI. The bias is initially quite coarse, but
the width of the kernels reduces as the simulation proceeds and the details of
the FES are increasingly better described. It can clearly be seen that the
OPES-explore variant employs fewer kernels compared to the original OPES. This
is due to the fact that in OPES-explore the kernel density estimation is used
for $p^{\text{WT}}(\mathbf{s})\propto[P(\mathbf{s})]^{1/\gamma}$ that is a
smoothed version of $P(\mathbf{s})$, and thus requires less details. This more
compact representation can be useful especially in higher dimensions, where
the number of kernels can greatly increase despite the compression algorithm.
However, as a drawback it can result in a less accurate bias estimate,
especially for large values of $\gamma$.

##  4 Fewer transitions can lead to better convergence

Figure 2: (a) The Müller potential energy surface, $U(x,y)$. (b) The free
energy surface along the $x$ coordinate, $F(x)$, with and without the addition
of the bias potential $V(x)=-(1-1/\gamma)F(x)$, where $\gamma=20$. (c) The
potential energy modified by the bias potential, $U(x,y)+V(x)$. It can be seen
that, despite the almost flat profile along $x$, the transition region between
the states remains at high energy.

The difference in performance between OPES and OPES-explore cannot be judged
from the alanine dipeptide example, because in this case the CVs chosen are
extremely efficient. In order to highlight the difference between the two
methods, we study a simple two-dimensional model potential that is known as
the Müller potential20, see Fig. 2a, using the $x$ coordinate as collective
variable. This is a clear example of suboptimal CV, since it can discriminate
the metastable states, but not the transition state.

For two-dimensional systems the free energy along the CV, $F(x)$, can be
computed precisely with numerical integration, Fig. 2b. From $F(x)$, the free
energy difference between the two metastable states can be calculated as

$\Delta F=-\frac{1}{\beta}\log\frac{\int_{0}^{1}e^{-\beta
F(x)}dx}{\int_{-1.3}^{0}e^{-\beta F(x)}dx}\,.$ (10)

While it is possible to distinguish better the two states by using also the
$y$ coordinate, this does not result in a significant difference in the
$\Delta F$ value (see SI, Sec. S3). On the other hand, $x$ does a poor job of
identifying the transition state, which is around $x\approx-0.7$ and $y\approx
0.6$, and not at $x\approx 0$ as it would seem from $F(x)$. As a consequence,
it is not possible to significantly increase the transition rate between the
states using a static bias that is a function of $x$ only.

To show this, we consider the effect of adding to the system the converged
well-tempered bias $V(x)=-(1-1/\gamma)F(x)$, with $\gamma=20$. In Fig. 2b, we
can see the effect of the bias on the FES along $x$, which becomes almost
completely flat. However, when we consider the full 2D landscape, Fig. 2c, we
can see that such bias does not really remove the barrier between the two
states. From the height of the barrier at the transition state, one can
roughly estimate that adding $V(x)$ improves the transition rate of about one
order of magnitude. Nevertheless, transitions remain quite rare, around one
every $10^{6}$ uncorrelated samples (see SI, Sec. S3).

We want to compare the two OPES variants and well-tempered metadynamics in
this challenging setting, where CVs are suboptimal and the total simulation
time is not enough to reach full convergence. This type of situation is not
uncommon in practical applications, and it is thus of great interest. Given
enough time, all the method considered converge to the same bias potential and
sample the same target distribution, but we shall see that before reaching
this limit they behave very differently.

|
---|---
(a) $x$ trajectory | (b) $\Delta F$ estimate
Figure 3: Typical simulations of the Müeller potential using different methods
for biasing the $x$ coordinate. Given more time, the three methods will
converge to the same bias potential and will sample the same target
distribution. In (a) is the trajectory along the CV and in (b) is the
corresponding $\Delta F_{n}$, Eq. 10, calculated using the FES estimate
obtained directly from the applied bias,
$F_{n}(x)=-(1-1/\gamma)^{-1}V_{n}(x)$. The correct $\Delta F$ value is
highlighted by a blue stripe 1 k${}_{\text{B}}$T thick.

Figure 3a shows a typical run of the Müller potential obtained by biasing the
$x$ coordinate with OPES, OPES-explore or MetaD. As a simple way to visualize
the evolution of the bias, we also report in Fig. 3b the $\Delta F_{n}$
estimate obtained directly from the applied bias, by using
$F_{n}(x)=-(1-1/\gamma)^{-1}V_{n}(x)$ in Eq. (10). We can see a qualitative
difference between OPES and the other two methods.

OPES reaches a quasi-static bias that is very close to the converged one, but
samples a distribution that is far from the well-tempered one, where the two
basins would be about equally populated. On the other hand, the $x$
distribution sampled by OPES-explore is closer to the target well-tempered
one, but its bias is far from converged, and makes ample oscillations around
the correct value. Metadynamics behaves similarly to OPES-explore. This is the
exploration-convergence tradeoff described in the introduction. Since the CV
is suboptimal, even when using the converged bias $V(x)$, to see a transition
occur one has to wait for an average number of steps $\tau\approx 10^{6}$,
which is more than the total length of the simulation. However, it is possible
to greatly accelerate transitions by using a time-dependent bias that forces
the system into higher energy pathways, that are not accessible at
equilibrium.

In OPES-explore the bias is based on the estimate of the sampled probability
$p^{\text{WT}}_{n}(\mathbf{s})$, and pushes to make it similar to the almost
flat well-tempered target. This means that in order to have a quasi-static
bias, $p^{\text{WT}}_{n}(\mathbf{s})$ should both be almost flat and not
change significantly as the simulation proceeds. Clearly, this cannot happen
unless the simulation is longer than $\tau$, otherwise most of the time would
be spent in the same basin and $p^{\text{WT}}_{n}(\mathbf{s})$ would be far
from flat. On the contrary, in OPES the bias is based on the reweighted
estimate $P_{n}(\mathbf{s})$, and thus it can reach a quasi-static regime even
before sampling the target distribution.

|
---|---
(a) Direct $\Delta F$ | (b) Reweighted $\Delta F$
Figure 4: Estimate of the free energy difference $\Delta F$ for the Müeller
potential obtained by averaging 25 independent runs for each biasing method.
The standard deviation is also shown for each estimate. Given more time, all
these estimates will converge to the correct $\Delta F$. All simulations start
from the main basin, $x<0$, but with different initial conditions. In (a) is
the estimate obtained directly from the applied bias, as in Fig. 3b, while in
(b) is the corresponding estimate obtained via reweighting. For metadynamics
two different reweighting schemes are considered, bias-offset 21, 22 and last-
bias reweighting23, 19. The correct $\Delta F$ value is highlighted by a blue
stripe 1 k${}_{\text{B}}$T thick.

In figure 4a we show the $\Delta F_{n}$ estimate averaged over 25 independent
runs, all starting from the main basin $x<0$. We can see that on average OPES
provides the best $\Delta F_{n}$ estimate at any $n$ in spite of the fact that
it induces far less transitions. In fact, most of the time only one full back-
and-forth transition is observed (see SI). One should notice that after a
single transition the $\Delta F_{n}$ estimate is far from being accurate (see
Fig. 3b) but, since the bias quickly becomes quasi-static, it is possible to
collect equilibrium samples and reliably reweight them, and the average
estimate becomes more accurate the more simulations are run. Instead in OPES-
explore and MetaD, despite starting from independent initial conditions, the
runs are highly correlated, due to the transitions being mostly driven by the
strong changes in the bias rather than the natural fluctuations of the system.
As a further consequence of this, a systematic error is present in the average
estimate, even if $\Delta F_{n}$ is further averaged over time, to remove the
oscillatory behaviour of OPES-explore and MetaD. Such systematic error depends
on the characteristic of the system and the chosen CVs, and is hard to predict
weather it will be relevant or small. Nevertheless, one can be sure that it
reduces over time as the bias converges24.

Estimates of $\Delta F_{n}$ using different reweighting schemes are shown in
Fig. 4b. For OPES and OPES-explore the simple Eq. (9) has been used, while for
MetaD we consider two of the most popular reweighiting schemes, namely last-
bias reweighting23, 19 and bias-offset reweighting21, 22. As expected, the
reweighting estimate of OPES is virtually identical to the direct estimate
obtained from the bias, while for the other two methods the two estimates
differ. The reweighing of OPES-explore has very small statistical uncertainty,
which further highlights the presence of a systematic error in the free energy
difference estimate. Like others before us22, 25, 26, we observe empirically
that the last-bias reweighting for MetaD tends to always be in agreement with
the direct estimate, even when the simulation is far from converged, while the
bias-offset reweighting provides a very unreliable estimate if the MetaD bias
has not reached a quasi-static regime and the initial part of the simulation
is not discarded. Once again, it must be noted that the simulations considered
here are not fully converged, otherwise all the different estimates of the
various methods would have yielded the correct result, without systematic
errors. However, for most practical purposes they behave very differently,
thus it is important to choose between an exploration-focused or a
convergence-focused enhanced sampling method, depending on the specific aim of
the simulation.

##  5 Sometimes exploration is what matters

In the examples of the previous paragraph, it was shown in that OPES converges
to a quasi-static bias faster than OPES-explore and provides more accurate FES
estimates. However, FES estimation is not the only goal of an enhanced
sampling simulation. In complex systems where good CVs are not available,
convergence can remain out of reach, still one might be interested in
exploring the phase space and find all the relevant metastable basins. In such
situation, OPES-explore can be a useful tool.

Figure 5: The eight metastable basins of alanine tetrapeptide in vacuum
sampled via OPES-explore by biasing the three $\psi$ angles, a suboptimal set
of CVs. Each basin is identified by the sign of the three $\phi$ angles, for a
total of $2^{3}$ possible combinations. The most stable basin has
$\phi_{1},\phi_{2},\phi_{3}<0$, while for the least stable
$\phi_{1},\phi_{2},\phi_{3}>0$.  Figure 6: Exploration time of the eight
metastable basins of alanine tetrapeptide over 100 ns. The lines are an
average over 10 independent runs for each method, showing the total number of
visited basins. In (a) the bias is a function of the three $\phi$ angles,
$V=V(\phi_{1},\phi_{2},\phi_{3})$, while in (b) the three $\psi$ angles are
used, $V=V(\psi_{1},\psi_{2},\psi_{3})$. See SI for results with different
input parameters and other MetaD variants, such as parallel-bias MetaD11.

We consider here as test system alanine tetrapeptide in vacuum, as in Ref. 4.
It has three $\phi$ dihedral angles, each of them can change from positive to
negative values and vice versa with a relatively low probability. This leads
to $2^{3}=8$ distinct metastable basins, each corresponding to a different
combination of $\phi$ angles signs, as shown in Fig. 5. Here we are not
interested in estimating the FES, but rather we want to compare the ability of
different methods to explore this space and discover all metastable states.

Figure 6 shows the number of explored basins averaged over 10 independent
simulations for each enhanced sampling method. The simulations in the top
panel (Fig. 6a) use as CVs the $\phi$ angles,
$V=V(\phi_{1},\phi_{2},\phi_{3})$, which are good CVs, while in the bottom
(Fig. 6b) the suboptimal $\psi$ angles are used,
$V=V(\psi_{1},\psi_{2},\psi_{3})$. In all methods, the exploration time
increases approximately by two orders of magnitude when suboptimal CVs are
used (please note the horizontal logarithmic scale). As expected, OPES and
OPES-explore have similar exploration speed when using good CVs, while with
suboptimal CVs OPES struggles to find all the metastable basins. This is
because the same region of CV space might correspond to two different
metastable basins, or to a basin and a transition state, as for the Müller
potential18, 19. In this situation, the previously estimated bias must change
considerably for the simulation to escape quickly the current metastable
state.

The exploration speed of MetaD depends critically on the input parameters and
requires a trial-and-error tuning. We report here only the outcome of MetaD
simulations in which a standard choice of the input parameters has been made.
As can be seen in Fig. 6, in these simulations the exploration speed is
roughly one order of magnitude slower than that of OPES-explore. However, the
performance of MetaD simulations can be improved by using different settings,
as shown in the SI. In the SI we also report and briefly discuss results
obtained with non-tempered metadynamics2, adaptive-Gaussians metadynamics23
and parallel-bias metadynamics11. None of these MetaD variants significantly
improve the exploration speed, and some make it even worse.

Finally, in the SI we show how a preliminary OPES-explore run can be combined
with a multithermal OPES simulation5, 6 to sample efficiently alanine
tetrapeptide and reach a converged FES, even without explicitly biasing the
$\phi$ angles.

##  6 Conclusion

We have shown with the help of model systems that there is an exploration-
convergence tradeoff in adaptive-bias methods when suboptimal CVs are used.
This tradeoff should not be confused with the exit time problem, that is
present also with optimal CVs, and is discussed in Sec. 2 and Refs. 17, 16.
Contrary to the exit time problem, the exploration-convergence tradeoff cannot
be solved and is an intrinsic limitation of CV-based adaptive-bias methods,
that is a consequence of suboptimal CVs. We believe the best way to handle
this tradeoff is to have separate methods that clearly focus on one or the
other aspect, so that they can be used depending on the application. In a
convergence-focused method the bias soon becomes quasi-static to allow for
accurate reweighting and free energy estimation. However, with suboptimal CVs
this leads to a slow transition rate and a long time is required to sample the
target distribution. As discussed, even if one knows the true $F(\mathbf{s})$
and directly applies the converged bias, one would not obtain a faster
exploration. In an exploration-focused method, it is possible to improve the
exploration speed by letting the bias change substantially even in a CV region
that has already been visited. While this may increase the number of
transitions, it comes at the cost of a less accurate estimate of the free
energy.

The original OPES method focuses on fast convergence to provide an accurate
estimate of the free energy surface and reweighted observables. As a
consequence, it is very sensitive to the quality of the CVs (see e.g. Fig. 3a)
and any improvement in the CVs results in a clear acceleration of the
transition rate. This is a particularly useful property when developing
machine learning-based CVs, and in fact OPES has already been used several
times in this context27, 28, 29, 30, 31, 32.

In other situations, improving the CVs may require first a better exploration
of the phase space33, 34, 35, 28. Furthermore, one may be interested simply in
exploring the metastable states of a system rather than estimating an accurate
FES36, 37, 38, 39. For this reason, we have introduced a variant of the OPES
method, OPES-explore, that focuses on quickly sampling the target distribution
and exploring the phase space.

We have shown that also well-tempered metadynamics is an exploration-focused
method. One of the main advantages of OPES-explore over MetaD is that it is
easier to use, since it requires fewer input parameters and it has a more
straightforward reweighting scheme (but more advanced ones can also be used25,
40). Another important difference between the two methods is that OPES-
explore, similarly to OPES, by default provides a maximum threshold to the
applied bias potential, thus it avoids unreasonably high free energy regions.
To obtain the same effect with MetaD, one typically has to define some ad hoc
static bias walls by trial and error. This last feature of OPES-explore has
been recently leveraged by Raucci et al. to systematically discover reaction
pathways in chemical processes41.

Finally, we should clarify that OPES-explore, just as metadynamics, might not
be able to exit any metastable state if the CVs are too poor42, 19, and its
improved exploration capability can only be harnessed if the CVs are close
enough to the correct ones to make such transitions possible. The speed and
small number of input parameters of OPES-explore are extremely helpful for
quickly testing several candidate CVs, to find out which can drive transitions
and discard the bad ones.

We believe that OPES-explore is an important addition to the OPES family of
methods and will become a useful tool for researchers as it pushes forward the
trend for more robust and reliable enhanced sampling methods.

We thank Valerio Rizzi and Umberto Raucci for useful discussions. M.I.
acknowledges support from the Swiss National Science Foundation through an
Early Postdoc.Mobility fellowship. Calculations were carried out on Euler
cluster at ETH Zurich and on workstations provided by USI Lugano.

## Data availability

An open-source implementation of the OPES and OPES-explore methods is
available in the enhanced sampling library PLUMED from version 2.843. All the
data and input files needed to reproduce the simulations presented in this
paper are available on PLUMED-NEST (www.plumed-nest.org), the public
repository of the PLUMED consortium 44, as plumID:22.003 .

Description of the adaptive bandwidth algorithm, computational details
regarding the Müller potential and further biased trajectories, exploration
speed for alanine tetrapeptide using other methods, and description of a
multithermal-multiumbrella simulation to improve upon the OPES-explore run.

## References

  * Mezei 1987 Mezei, M. Adaptive umbrella sampling: Self-consistent determination of the non-Boltzmann bias. _Journal of Computational Physics_ 1987, _68_ , 237–248
  * Laio and Parrinello 2002 Laio, A.; Parrinello, M. Escaping free-energy minima. _Proceedings of the National Academy of Sciences_ 2002, _99_ , 12562–12566
  * Barducci et al. 2008 Barducci, A.; Bussi, G.; Parrinello, M. Well-Tempered Metadynamics: A Smoothly Converging and Tunable Free-Energy Method. _Physical Review Letters_ 2008, _100_ , 020603
  * Invernizzi and Parrinello 2020 Invernizzi, M.; Parrinello, M. Rethinking Metadynamics: From Bias Potentials to Probability Distributions. _The Journal of Physical Chemistry Letters_ 2020, _11_ , 2731–2736
  * Invernizzi et al. 2020 Invernizzi, M.; Piaggi, P. M.; Parrinello, M. Unified Approach to Enhanced Sampling. _Physical Review X_ 2020, _10_ , 41034
  * Invernizzi 2021 Invernizzi, M. OPES: On-the-fly Probability Enhanced Sampling Method. _Il Nuovo Cimento C_ 2021, _44_ , 112
  * Invernizzi and Parrinello 2019 Invernizzi, M.; Parrinello, M. Making the Best of a Bad Situation: A Multiscale Approach to Free Energy Calculation. _Journal of Chemical Theory and Computation_ 2019, _15_ , 2187–2194
  * Jarzynski 1997 Jarzynski, C. Nonequilibrium Equality for Free Energy Differences. _Physical Review Letters_ 1997, _78_ , 2690–2693
  * Donati and Keller 2018 Donati, L.; Keller, B. G. Girsanov reweighting for metadynamics simulations. _The Journal of Chemical Physics_ 2018, _149_ , 072335
  * Bal 2021 Bal, K. M. Reweighted Jarzynski Sampling: Acceleration of Rare Events and Free Energy Calculation with a Bias Potential Learned from Nonequilibrium Work. _Journal of Chemical Theory and Computation_ 2021, _17_ , 6766–6774
  * Pfaendtner and Bonomi 2015 Pfaendtner, J.; Bonomi, M. Efficient Sampling of High-Dimensional Free-Energy Landscapes with Parallel Bias Metadynamics. _Journal of Chemical Theory and Computation_ 2015, _11_ , 5062–5067
  * Valsson and Parrinello 2014 Valsson, O.; Parrinello, M. Variational Approach to Enhanced Sampling and Free Energy Calculations. _Physical Review Letters_ 2014, _113_ , 090601
  * White et al. 2015 White, A. D.; Dama, J. F.; Voth, G. A. Designing Free Energy Surfaces That Match Experimental Data with Metadynamics. _Journal of Chemical Theory and Computation_ 2015, _11_ , 2451–2460
  * Marsili et al. 2006 Marsili, S.; Barducci, A.; Chelli, R.; Procacci, P.; Schettino, V. Self-healing Umbrella Sampling: A Non-equilibrium Approach for Quantitative Free Energy Calculations. _The Journal of Physical Chemistry B_ 2006, _110_ , 14011–14013
  * Sodkomkham et al. 2016 Sodkomkham, D.; Ciliberti, D.; Wilson, M. A.; Fukui, K.-I.; Moriyama, K.; Numao, M.; Kloosterman, F. Kernel density compression for real-time Bayesian encoding/decoding of unsorted hippocampal spikes. _Knowledge-Based Systems_ 2016, _94_ , 1–12
  * Fort et al. 2017 Fort, G.; Jourdain, B.; Lelièvre, T.; Stoltz, G. Self-healing umbrella sampling: convergence and efficiency. _Statistics and Computing_ 2017, _27_ , 147–168
  * Dama et al. 2014 Dama, J. F.; Rotskoff, G.; Parrinello, M.; Voth, G. A. Transition-tempered metadynamics: Robust, convergent metadynamics via on-the-fly transition barrier estimation. _Journal of Chemical Theory and Computation_ 2014, _10_ , 3626–3633
  * Pietrucci 2017 Pietrucci, F. Strategies for the exploration of free energy landscapes: Unity in diversity and challenges ahead. _Reviews in Physics_ 2017, _2_ , 32–45
  * Bussi and Laio 2020 Bussi, G.; Laio, A. Using metadynamics to explore complex free-energy landscapes. _Nature Reviews Physics_ 2020,
  * Müller and Brown 1979 Müller, K.; Brown, L. D. Location of saddle points and minimum energy paths by a constrained simplex optimization procedure. _Theoretica Chimica Acta_ 1979, _53_ , 75–93
  * Tiwary and Parrinello 2015 Tiwary, P.; Parrinello, M. A time-independent free energy estimator for metadynamics. _Journal of Physical Chemistry B_ 2015, _119_ , 736–742
  * Valsson et al. 2016 Valsson, O.; Tiwary, P.; Parrinello, M. Enhancing Important Fluctuations: Rare Events and Metadynamics from a Conceptual Viewpoint. _Annual Review of Physical Chemistry_ 2016, _67_ , 159–184
  * Branduardi et al. 2012 Branduardi, D.; Bussi, G.; Parrinello, M. Metadynamics with Adaptive Gaussians. _Journal of Chemical Theory and Computation_ 2012, _8_ , 2247–2254
  * Dama et al. 2014 Dama, J. F.; Parrinello, M.; Voth, G. A. Well-Tempered Metadynamics Converges Asymptotically. _Physical Review Letters_ 2014, _112_ , 240602
  * Marinova and Salvalaglio 2019 Marinova, V.; Salvalaglio, M. Time-independent free energies from metadynamics via mean force integration. _The Journal of Chemical Physics_ 2019, _151_ , 164115
  * Giberti et al. 2020 Giberti, F.; Cheng, B.; Tribello, G. A.; Ceriotti, M. Iterative Unbiasing of Quasi-Equilibrium Sampling. _Journal of Chemical Theory and Computation_ 2020, _16_ , 100–107
  * Bonati et al. 2020 Bonati, L.; Rizzi, V.; Parrinello, M. Data-Driven Collective Variables for Enhanced Sampling. _The Journal of Physical Chemistry Letters_ 2020, _11_ , 2998–3004
  * Bonati et al. 2021 Bonati, L.; Piccini, G.; Parrinello, M. Deep learning the slow modes for rare events sampling. _Proceedings of the National Academy of Sciences_ 2021, _118_ , e2113533118
  * Karmakar et al. 2021 Karmakar, T.; Invernizzi, M.; Rizzi, V.; Parrinello, M. Collective variables for the study of crystallisation. _Molecular Physics_ 2021, _119_ , e1893848
  * Trizio and Parrinello 2021 Trizio, E.; Parrinello, M. From Enhanced Sampling to Reaction Profiles. _The Journal of Physical Chemistry Letters_ 2021, _12_ , 8621–8626
  * Rizzi et al. 2021 Rizzi, V.; Bonati, L.; Ansari, N.; Parrinello, M. The role of water in host-guest interaction. _Nature Communications_ 2021, _12_ , 93
  * Ansari et al. 2021 Ansari, N.; Rizzi, V.; Carloni, P.; Parrinello, M. Water-Triggered, Irreversible Conformational Change of SARS-CoV-2 Main Protease on Passing from the Solid State to Aqueous Solution. _Journal of the American Chemical Society_ 2021, _143_ , 12930–12934
  * Branduardi et al. 2007 Branduardi, D.; Gervasio, F. L.; Parrinello, M. From A to B in free energy space. _The Journal of Chemical Physics_ 2007, _126_ , 054103
  * McCarty and Parrinello 2017 McCarty, J.; Parrinello, M. A variational conformational dynamics approach to the selection of collective variables in metadynamics. _The Journal of Chemical Physics_ 2017, _147_ , 204109
  * Mendels et al. 2018 Mendels, D.; Piccini, G.; Parrinello, M. Collective Variables from Local Fluctuations. _Journal of Physical Chemistry Letters_ 2018, _9_ , 2776–2781
  * Piaggi and Parrinello 2018 Piaggi, P. M.; Parrinello, M. Predicting polymorphism in molecular crystals using orientational entropy. _Proceedings of the National Academy of Sciences_ 2018, _115_ , 10251–10256
  * Capelli et al. 2019 Capelli, R.; Carloni, P.; Parrinello, M. Exhaustive Search of Ligand Binding Pathways via Volume-Based Metadynamics. _The Journal of Physical Chemistry Letters_ 2019, _10_ , 3495–3499
  * Ahlawat et al. 2021 Ahlawat, P.; Hinderhofer, A.; Alharbi, E. A.; Lu, H.; Ummadisingu, A.; Niu, H.; Invernizzi, M.; Zakeeruddin, S. M.; Dar, M. I.; Schreiber, F.; Hagfeldt, A.; Grätzel, M.; Rothlisberger, U.; Parrinello, M. A combined molecular dynamics and experimental study of two-step process enabling low-temperature formation of phase-pure $\alpha$-FAPbI 3. _Science Advances_ 2021, _7_ , eabe3326
  * Francia et al. 2021 Francia, N. F.; Price, L. S.; Salvalaglio, M. Reducing crystal structure overprediction of ibuprofen with large scale molecular dynamics simulations. _CrystEngComm_ 2021, _23_ , 5575–5584
  * Carli and Laio 2021 Carli, M.; Laio, A. Statistically unbiased free energy estimates from biased simulations. _Molecular Physics_ 2021, _119_
  * Raucci et al. 2022 Raucci, U.; Rizzi, V.; Parrinello, M. Discover, Sample, and Refine: Exploring Chemistry with Enhanced Sampling Techniques. _The Journal of Physical Chemistry Letters_ 2022, 1424–1430
  * Bussi and Branduardi 2015 Bussi, G.; Branduardi, D. _Free-Energy Calculations with Metadynamics: Theory and Practice_ ; John Wiley & Sons, Inc, 2015; pp 1–49
  * Tribello et al. 2014 Tribello, G. A.; Bonomi, M.; Branduardi, D.; Camilloni, C.; Bussi, G. PLUMED 2: New feathers for an old bird. _Computer Physics Communications_ 2014, _185_ , 604–613
  * The PLUMED consortium 2019 The PLUMED consortium, Promoting transparency and reproducibility in enhanced molecular simulations. _Nature Methods_ 2019, _16_ , 670–673

See pages - of SupportingInformation.pdf