# Interactive Evolution: A Neural-Symbolic Self-Training Framework For Large
Language Models

Fangzhi Xu$\diamondsuit$$\heartsuit$, Qiushi Sun$\heartsuit$, Kanzhi Cheng,
Jun Liu$\diamondsuit$, Yu Qiao$\heartsuit$, Zhiyong Wu$\heartsuit$
$\diamondsuit$Xi’an Jiaotong University $\heartsuit$Shanghai Artificial
Intelligence Laboratory
The University of Hong Kong Nanjing Univerisity
<EMAIL_ADDRESS><EMAIL_ADDRESS>Work done during internship at
Shanghai AI Lab. Corresponding Author.

###### Abstract

One of the primary driving forces contributing to the superior performance of
Large Language Models (LLMs) is the extensive availability of human-annotated
natural language data, which is used for alignment fine-tuning. This inspired
researchers to investigate self-training methods to mitigate the extensive
reliance on human annotations. However, the current success of self-training
has been primarily observed in natural language scenarios, rather than in the
increasingly important neural-symbolic scenarios. To this end, we propose an
environment-guided neural-symbolic self-training framework named ENVISIONS. It
aims to overcome two main challenges: (1) the scarcity of symbolic data, and
(2) the limited proficiency of LLMs in processing symbolic language. Extensive
evaluations conducted on three distinct domains demonstrate the effectiveness
of our approach. Additionally, we have conducted a comprehensive analysis to
uncover the factors contributing to ENVISIONS’s success, thereby offering
valuable insights for future research in this area. Code will be available at
https://github.com/xufangzhi/ENVISIONS.

##  1 Introduction

Large Language Models (LLMs) [1, 2] have undergone extensive training using
massive data, enabling them to possess remarkable capabilities across diverse
domains. One of the main recipes of LLMs’ success is the post-pretraining
effort to achieve alignment with downstream tasks [3, 4]. The effective
alignment primarily relies on the accessibility of a substantial volume of
expensive human-annotated data, employing techniques such as Supervised Fine-
Tuning (SFT) [5] or Reinforcement Learning from Human Feedback (RLHF) [6].
Recently, there has been a growing interest in developing self-training
methods that enable fine-tuning of LLMs without human annotations, thereby
reducing cost and streamlining the training process [7].

Notable progress has been made in self-training techniques for natural
language (NL) scenarios [8, 9], where researchers focus on improving LLMs by
synthesizing their own natural language input-output pairs. However, in recent
years, there has been a growing emphasis on delegating tasks to external
tools/environments to expand the capability boundaries of LLMs. The shift in
focus necessitates the generation of a symbolic intermediate representation a
that can be executed in the environment to faithfully produce the desired
output y. This neural-symbolic framework [10] has achieved significant success
in complex planning [11], mathematical reasoning [12], robotic planning [13],
and agentic tasks [14, 15]. In contrast to the abundance of NL annotation data
(x-y), curating symbolic annotation (x-a-y) is significantly more challenging
and costly due to the scarcity and inherent complexity of symbolic language
(SL). In this paper, we delve into the exploration of effective self-training
methods for LLMs within complex neural-symbolic scenarios, all without human-
annotated symbolic data.

Figure 1: Weak-to-strong paradigm comparison. (a) Distill-then-Finetune
paradigm. (b) Reinforced Self-Training methods. (c) Env-guided Self-Training
paradigm.

Current self-training approaches in empowering LLMs in SL-centric scenarios
fall into two categories, each with its own drawbacks. _Distill-then-Finetune_
[5, 16], shown in Fig. 1(a), entails fine-tuning a less powerful LLM using
distilled data obtained from a teacher LLM, such as GPT-4 [1]. Although this
method is simple yet effective, its application is constrained by the
requirement of an already existing stronger LLM and the associated high costs.
Furthermore, the performance of the student LLM is upper-bounded by the
capabilities of the teacher LLM. _Reinforced Self-Training_ [17, 18], as shown
in Fig. 1 (b), iteratively improves a weak LLM by leveraging reinforcement
learning algorithms [19], guided by customized reward models. Nevertheless,
reinforced methods are constrained by their inefficiency in training and/or
reliance on human annotations for reward model training.

To address the limitations of previous approaches, this work focuses on two
key challenges: enhancing the proficiency of LLMs in processing SL and
eliminating the requirement for human-annotated data. Illustrated in Figure 1
(c), the proposed approach, called _Environment-guided (Env-guided) self-
training_ , involves iterative training of LLMs through interactions with an
embodied environment. Built upon the approach, we propose an ENV-guIded Self-
traIning framework fOr Neural Symbolic scenarios, named ENVISIONS. As an
example, consider the training of LLMs for web browsing tasks, i.e., training
a web agent. Given a web manipulation task $x$, the agent generates multiple
candidate actions $a\in\mathcal{A}$ and executes these actions within the web
browser, resulting in both correct and incorrect outcomes. A self-rewarding
algorithm is designed to post-process the agent’s trajectories and create
contrastive training pairs. These correct-incorrect trajectory pairs, along
with a self-refining loss, are utilized to empower the LLMs to self-correct
and improve their performance.

Through the Env-guided self-training approach, the LLMs leverage the
interactive nature of the embodied environment to generate trajectories and
learn symbolic language processing abilities, mitigating the need for human
annotations. Through extensive evaluation, we found that ENVISIONS can
consistently convert an existing LLM to a stronger one without reliance on
existing stronger models or reward models. It’s also worth noting that
ENVISIONS and previous methods are not mutually exclusive, but we leave it as
a future work to explore their synergy.

We highlight our contributions as follows:

(1) A neural-symbolic self-training framework: We propose a novel framework
ENVISIONS for neural-symbolic self-training. The proposed framework can
eliminate the need for human annotation or a stronger teacher model during
self-training.

(2) Comprehensive evaluations and analysis: We perform extensive evaluation
across three different domains to demonstrate the effectiveness of ENVISIONS
over previous state-of-the-art self-training methods. Additionally, we
conducted an in-depth analysis to uncover the underlying reasons for the
remarkable performance of ENVISIONS and to highlight its potential as a new
paradigm for neural-symbolic self-training.

(3) Insights on Env-guided neural-symbolic self-training: Our research
provides valuable insights, supported by evidence, into the training process
of Env-guided neural-symbolic self-training. These findings pave the way for
future developments in this area of research.

##  2 Related Work

#### Self-Training Methods.

Self-training, particularly when integrated with RL [20, 6], offers a
promising avenue for models to learn from their own outputs, minimizing the
need for extensive human annotations. Typically, recent advances [17] leverage
well-trained reward models to filter better training samples, and further
decouples the data collection and policy optimization steps for reinforced
self-training [18, 21]. However, these approaches heavily rely on a strong
reward model, which limits its applicability and training efficiency.
Following the success of DPO [19], self-play frameworks have emerged as a new
path that implicitly models the preferences among unlabeled rationales in
iterative DPO styles [8, 9]. More recently, [7] incorporates the self-
rewarding strategy into the DPO step, which reduces the reliance on external
models. Nevertheless, such methods still face efficiency issues [22]. Beyond
RL, previous works [23, 24] optimize policy models within iterative SFT
frameworks, yet neglecting the value of negative samples. Moreover, current
approaches solely integrate the neural model into self-training, constraining
the faithfulness and generalization to executable symbolic environments.

#### Data Synthesis with LLMs.

Obtaining high-quality human-annotated reasoning traces to optimize LLMs has
been a long-standing challenge [25]. Beyond well-established approaches
utilizing data augmentation strategies to obtain diversified training data
[26, 27]. Recent efforts [28, 29, 30] primarily distill strong LLMs [1, 31] to
generate novel samples in the given format. They either generate more diverse
samples from seed data through self-instruct [32] or enhance diversity through
sample rephrasing and rewriting [33, xu2023large]. Combining environments,
learning from the trajectory of LLMs for behavior cloning has also emerged as
a promising approach [34, 35]. However, current works mainly employ close-
sourced LLMs (e.g., GPT-4) for data synthesis, which is a cost. We propose to
achieve data bootstrapping from weak LLMs without relying on external models.

#### Neural-Symbolic Integration for LLMs.

Neural-symbolic methods synergize the powerful generation capacity of LLMs
with the reliability and interpretability of symbolic systems. Typically,
PAL/PoT [36, 37] synthesize executable programs as intermediate reasoning
steps to solve numerical problems. This strategy of delegating problems to
external solvers (e.g., Python interpreter), has gained significant traction
[38]. ToRA [12] and Logic-LM [39] apply neural code generation and symbolic
execution on math and logical reasoning respectively, enhanced with necessary
refinement. Moreover, Symbol-LLM [10] unifies neural-symbolic applications
under a _Symbol+Delegation_ setting. Beyond reasoning, recent endeavors have
extended the application into agent scenarios [40, 41] and leverage external
feedback from the environment [14, 42] for refinement. However, these
approaches mainly optimize LLM usage rather than providing autonomous self-
improvement.

##  3 Methodology

Figure 2: The neural-symbolic self-training framework ENVISIONS. $\rightarrow$
denotes _self-exploration_ process (Step ①-③), $\rightarrow$ indicates _self-
refinement_ (Step ④-⑥), and $\rightarrow$ is _self-rewarding_ (Step ⑦).

###  3.1 Preliminaries

In neural-symbolic scenarios, based on the NL input $x$, LLMs are required to
produce symbolic solution $a$ to obtain the desired output $y$ through the
execution in the environment Env. To adapt the weak LLMs to such complex
settings and curate extensive $(x,a,y)$ pairs, we propose to iteratively
interact with Env, thus boosting LLMs through self-training. For each
iteration $i$, the LLM $\pi_{\theta_{i}}$ will be provided with the task data
set $\\{(x^{(i)},y^{(i)})\\}$, with $J$ input-output pairs. Without loss of
generality, we assume the samples keep static between iterations.

Figure 2 presents the iterative self-training framework, named ENVISIONS. Our
framework is specifically designed to address two key challenges: (1) the
scarcity of SL data and (2) the limited proficiency of LLMs in SL. Data
scarcity limitation is addressed by the online exploration stage (Step ①-⑦).
To convert LLMs from weak to strong in addressing SL, we employ LLM training
using a carefully designed loss function and filtered data (Step ⑧-⑩). To
simplify the expression, we omit the indicator of iteration $i$ in the
symbols. The overall procedure of ENVISIONS is also concluded in the
pseudocode of Appendix B.

###  3.2 Mitigating SL Data Scarcity with Online Exploration

With the lack of well-annotated SL data for training, ENVISIONS empowers
policy LLM to produce candidate symbolic solutions through autonomous
interaction with the environment Env. This process is named _Online
Exploration_ , which includes three main aspects 1) self-exploration (Step
①-③); 2) self-refinement (Step ④-⑥); and 3) self-rewarding (Step ⑦).

#### Self-exploration

Given the NL input $x$, the policy model $\pi_{\theta}$ first generates $K$
diverse symbolic outputs (Step ①), formulated as
$\\{a_{k}\\}_{k=1}^{K}\sim\pi_{\theta}(\cdot|x)$. These intermediate outputs
will be executed in Env (Step ②) to obtain the binary feedback
$\\{b_{k}\\}_{k=1}^{K}$ based on $y$ (Step ③). The above procedure allows
$\pi_{\theta}$ to explore the environment autonomously and search for diverse
symbolic solutions.

#### Self-refinement.

Considering the complexity of SL, solutions generated by the LLM may contain
mistakes in symbolic format, which significantly impair the efficiency of
exploration. To address this, we utilize the above self-explored solutions
$\\{a_{k}\\}_{k=1}^{K}$ as references to regenerate new refined symbolic
solutions (Step ④), formulated as
$\\{\widetilde{a}_{k}\\}_{k=1}^{K}\sim\pi_{\theta}(\cdot|x;a_{k})$. Similarly,
these outputs will be executed in Env (Step ⑤) and receive the corresponding
binary reward $\\{\widetilde{b}_{k}\\}_{k=1}^{K}$ (Step ⑥).

#### Self-rewarding.

Feedback from Env merely gives the binary rewards. However, it remains
challenging to discern preferences among various positive solutions or obtain
valuable feedback from negative solutions. Motivated by it, we propose a soft
reward score through sequence output probabilities with the following
calculation:

$r=\overline{p}_{\theta}(a|x)=\frac{\sum\limits_{t}{\mathrm{log}}\,p_{\theta}(a_{t}|x;a_{<t})}{||a||},$
(1)

where $||a||$ is the length of the symbolic solution $a$. Based on this
definition, the soft self-rewards of $a_{k}$ and $\widetilde{a}_{k}$ are
derived respectively as $r_{k}$ and $\widetilde{r}_{k}$. Considering that no
extra reward model is involved, we name it _self-rewarding_ 111 _Self-
rewarding_ step in ENVISIONS is different from [7], though they share the same
name. (Step ⑦).

###  3.3 Data Selection and Training Strategies

After the online exploration stage, the candidate trajectories are constructed
as $T_{k}=(x,y,a_{k},b_{k},r_{k})$ and
$\widetilde{T}_{k}=(x,y,\widetilde{a}_{k},\widetilde{b}_{k},\widetilde{r}_{k})$,
which are sourced from _self-exploration_ and _self-refinement_ respectively.
Next, we select premium trajectories for training the LLM $\pi_{\theta_{i}}$.

#### Trajectory filtering and candidate pool updating.

To control the candidate number to maintain trajectories with high quality, we
select the superior one from $T_{k}$ and $\widetilde{T}_{k}$ to update the
candidate trajectory pool (Step ⑧). To facilitate automatic selection, we
incorporate binary rewards and self-rewards for assessment. Following the
principle of prioritizing execution correctness, we derive the filtered
trajectory $T_{k}^{*}$:

$T_{k}^{*}=(x,y,a_{k}^{*},b_{k}^{*},r_{k}^{*})=\begin{cases}(x,y,a_{k},b_{k},r_{k}),&\text{if
}b_{k}=1\text{ and }\widetilde{b}_{k}=0,\vspace{0.1em}\\\
(x,y,a_{k},b_{k},r_{k}),&\text{if }b_{k}=\widetilde{b}_{k}\text{ and
}r_{k}>\widetilde{r}_{k},\vspace{0.1em}\\\
(x,y,\widetilde{a}_{k},\widetilde{b}_{k},\widetilde{r}_{k}),&\text{otherwise}.\end{cases}$
(2)

Notably, our filter strategy still maintains some trajectories with incorrect
solutions but relatively higher rewards. These trajectories will serve as hard
negative samples for the subsequent steps.

#### Supervised fine-tuning on positive solutions.

As we have explored diverse trajectories in Env, an intuitive way to bootstrap
the performance of LLMs is fine-tuning with the positive solutions. Therefore,
for each input $x$, we can retrieve the positive trajectories (i.e., $b=1$)
from the candidate pool. Giving priority to more valuable solutions, we rank
the trajectories in descending order based on self-rewards, resulting in the
positive set $S^{+}$. To mitigate overfitting, we enforce a maximum of $N_{1}$
positive-only solutions sampled for each input $x$:

$U_{1}=\\{(x,a_{m}^{+})\mid m\leq\min(N_{1},|S^{+}|)\textrm{ and }T_{m}^{+}\in
S^{+}\\}$ (3)

where $m\in\mathbb{Z}^{+}$ means the index in the ranked set and $|\cdot|$
returns the number of trajectories in the given set.
$T_{m}^{+}=(x,y,a_{m}^{+},b_{m}^{+}r_{m}^{+})$ denotes the trajectories in
$S^{+}$. Following the principle of MLE, the optimized loss function can be
written as:

$\mathcal{L}_{1}=-\sum\limits_{(x,a^{+})\sim{U}_{1}}{\mathrm{log}}\,p_{\theta}(a^{+}|x)$
(4)

#### RL-free loss to learn from mistakes.

Under the neural-symbolic setting, negative solutions may comprise a
substantial portion of exploration trajectories, while also offering valuable
insights for model enhancement. ENVISIONS explores motivating the policy LLM
to learn from mistakes during the weak-to-strong process. We can obtain the
ranked negative set $S^{-}$ from the candidate pool. For each input $x$, at
most $N_{2}$ positive-negative pairs will be constructed from $S^{+}$ and
$S^{-}$:

$U_{2}=\\{(x,a_{m}^{+},a_{m}^{-})\mid
m\leq\min(N_{2},|S^{+}|-N_{1},|S^{-}|)\textrm{ and }T_{m+|U_{1}|}^{+}\in
S^{+}\textrm{ and }T_{m}^{-}\in S^{-}\\}$ (5)

where $T_{m}^{-}=(x,y,a_{m}^{-},b_{m}^{-}r_{m}^{-})$ denotes the trajectories
in $S^{-}$. Limited by the difficulty and complexity of optimizing models in
an RL manner (e.g., DPO [19] and PPO [20]), it is challenging for reinforced-
based methods such as SPIN [8] and DNO [9] to quickly adapt to the SL
scenarios. Therefore, we design the following RL-free loss function in a
contrastive manner:

$\mathcal{L}_{2}=-\sum\limits_{(x,a^{+},a^{-})\sim{U}_{2}}{\mathrm{log}}\,p_{\theta}(a^{+}|x;a^{-})$
(6)

It brings two main advantages: (1) the ability of self-refinement is acquired,
which benefits the scalability to complex cases; (2) compared to reinforced
losses, superior training efficiency is achieved. Finally, the overall loss
function of each iteration is simply designed as
$\mathcal{L}=\mathcal{L}_{1}+\mathcal{L}_{2}$. We leave the discussion of the
weighted one in the future.

##  4 Experiments

###  4.1 Datasets

We evaluate the proposed framework on three distinct domains, each with its
own environment: web agents (Chrome browser), math reasoning (Python
compiler), and logical reasoning (Pyke engine). For agentic tasks, we select
the widely-used web navigation benchmark MiniWob++ [43]. For the math
reasoning domain, we include 5 tasks: GSM8K [44], MATH [45], GSM-Hard [36],
SVAMP [46], and AsDiv [47]. ProofWriter [48] and RuleTaker [49] are used to
evaluate logic reasoning performance. To evaluate the generalization
capability of our method, we reserve some datasets for out-of-distribution
evaluation as shown in Table 1. Please refer to Appendix A.2 for more details.

Domains | Held-in Tasks | Held-out Tasks | #Samples | Static ? | Env.
---|---|---|---|---|---
Web Agent | MiniWob++ | - | 2,200 | No | Chrome browser
Math Reasoning | GSM8K, MATH | GSM-H, SVAMP, AsDiv | 13,492 | Yes | Python compiler
Logic Reasoning | ProofWriter | RuleTaker | 3,600 | Yes | Pyke engine

Table 1: Details and statistics of evaluated domains. _#Samples_ denotes the
number of input samples per iteration. _Static?_ indicates whether the input
data remains the same across all iterations. _Env._ column presents the
external solver for execution.

###  4.2 Baselines and Training Details

Following the categorization of Figure 1, we consider the respective three
lines of baselines. All baselines are reproduced under the same codebase for a
fair comparison.

Distill-then-Finetune. GPT-4 and Claude-2 are selected as strong teacher LLMs
in this approach. By prompting teacher LLMs, we obtain the symbolic
trajectories with correct answers to fine-tune the LLM. In light of the high
time and financial cost of calling API, each input will be prompted only once.

Reinforced Self-Training. We implement two RL-based self-training baselines:
_Self-Rewarding_ [7] and _iterative SFT+DPO_. For the former, we follow the
official implementation to first warm up the weak LLM using human annotation
from OpenAssistant [50]. The latter is a simple variation of ENVISIONS that
mainly separates the training into two stages, with positive solutions for SFT
and positive-negative pairs for DPO.

Env-guided Self-Training. Since there are no existing baselines for this
approach, we consider extending the NL-centric self-training method STaR [23]
to support neural-symbolic scenarios. It is worth noting that STaR only uses
positive samples for behavior cloning. For the methods under this paradigm
(including ENVISIONS) we optimize LLM from scratch in each iteration with the
updated training samples.

Except for _Distill-then-Finetune_ baselines, all other methods, ours
included, utilize few-shot prompting to acquire training samples as a cold
start. The few-shot numbers for the web agent, math reasoning, and logic
reasoning domains are set to 1, 3, and 1 respectively. We also include few-
shot results on weak LLM for comparison. For a fair evaluation, all baselines
are optimized to generate symbolic outputs (e.g., Python code) rather than
natural language outputs, following PoT style [37]. Please refer to Appendix A
for other details.

We use LLaMA2-Chat 7B/13B models for the evaluation. At each generation step
(i.e., Step ①,④), the candidate size $K$ is set to 5. The total iteration
number for web agent, math, and logic tasks is set to 5, 10, and 8
respectively, unless otherwise stated. For each input, $N_{1}$ and $N_{2}$ are
fixed to 10 and 2 respectively. All the self-training experiments are
implemented on 8*A100 of 80GB VRAM. Please refer to Appendix A.1 for other
details.

Models | Agent | Math Reasoning | Logical Reasoning | Avg.
---|---|---|---|---
MiniWob++ | GSM8K | MATH | GSM-H | SVAMP | ASDiv | ProofWriter | RuleTaker
Is Held-out ? |  2717 |  2717 |  2717 | ✓ | ✓ | ✓ |  2717 | ✓ |
LLaMA2-Chat (7B)
LLaMA2-Chat (few-shot) | 51.14 | 12.21 | 1.32 | 10.69 | 22.00 | 25.86 | 34.83 | 47.44 | 25.69
Distill-then-Finetune |  |  |  |  |  |  |  |  |
GPT-4 + LLaMA2-Chat | 81.14 | 53.07 | 18.84 | 47.84 | 66.80 | 68.75 | 34.33 | 48.88 | 52.46
Claude-2 + LLaMA2-Chat | 82.80 | 52.69 | 18.17 | 44.88 | 70.50 | 69.85 | 36.17 | 49.17 | 53.03
Reinforced Self-Training |  |  |  |  |  |  |  |  |
Self-Rewarding | 69.39 | 40.03 | 10.70 | 31.69 | 58.20 | 61.55 | 32.17 | 50.04 | 44.22
Iterative SFT+DPO | 77.05 | 54.81 | 14.75 | 47.08 | 70.10 | 66.22 | 49.00 | 58.82 | 54.73
Env-guided Self-Training |  |  |  |  |  |  |  |  |
STaR + Env. | 83.71 | 58.23 | 15.97 | 46.63 | 67.50 | 68.46 | 50.17 | 58.60 | 55.91
ENVISIONS | 85.38 | 58.98 | 19.00 | 48.52 | 72.40 | 69.80 | 52.83 | 62.63 | 58.69
LLaMA2-Chat (13B)
LLaMA2-Chat (few-shot) | 60.00 | 34.87 | 6.07 | 28.96 | 45.00 | 46.61 | 35.83 | 51.50 | 38.61
Distill-then-Finetune |  |  |  |  |  |  |  |  |
GPT-4 + LLaMA2-Chat | 80.15 | 62.85 | 23.64 | 53.98 | 73.00 | 73.52 | 34.17 | 50.61 | 56.49
Claude-2 + LLaMA2-Chat | 84.77 | 62.24 | 23.47 | 52.08 | 76.30 | 74.05 | 36.00 | 48.45 | 57.17
Reinforced Self-Training |  |  |  |  |  |  |  |  |
Self-Rewarding | 74.55 | 50.80 | 13.97 | 41.24 | 74.10 | 71.37 | 37.33 | 56.66 | 52.50
Iterative SFT+DPO | 82.73 | 63.84 | 22.32 | 50.57 | 77.30 | 70.94 | 51.00 | 59.47 | 59.77
Env-guided Self-Training |  |  |  |  |  |  |  |  |
STaR + Env. | 85.15 | 63.61 | 20.57 | 53.37 | 74.70 | 74.76 | 52.33 | 60.33 | 60.60
ENVISIONS | 87.12 | 68.31 | 26.04 | 57.54 | 78.30 | 75.52 | 54.83 | 60.84 | 63.56

Table 2: Main Results on Agent, Math Reasoning and Logical Reasoning domain.
Notably, we report the average performance across extensive tasks in MiniWob++
benchmark (refer to Appendix D.3 for details). _Is Held-out?_ row
distinguishes the held-in and held-out tasks. _Avg._ column reports the
averaged performances on all tasks.

###  4.3 Main Results

Table 2 presents the evaluation results. We include an LLaMA2-Chat (few-shot)
baseline for reference purposes, all other methods are tested under the zero-
shot setting.

####  ENVISIONS presents consistent superiority over strong baselines.

Evolving from LLaMA2-Chat, ENVISIONS significantly enhances average
performances by 30.00% and 24.95% for the 7B and 13B variants, respectively.
Compared with _Distill-then-Finetune_ methods, 5.66%-7.13% improvements are
obtained. In addition to its superior performance, ENVISIONS offers greater
scalability compared with the significant costs associated with using strong
LLMs (e.g., GPT-4). Besides it, the obvious advantages over _Reinforced Self-
Training_ and other _Env-guided Self-Training_ methods are observed, with
2.78%-14.47% average gains. The combination of the performance as well as the
training efficiency makes ENVISIONS stand out among these strong baselines.

#### Env-guided Self-Training exhibits strong scalability to neural-symbolic
scenarios.

Compared to the other two approaches, _Env-guided Self-Training_ is more
applicable to complex neural-symbolic scenarios, especially in agentic tasks
where NL-centric methods inherently exhibit limitations. Besides the great
performances of ENVISIONS, previous methods _STaR_ can also benefit from the
supervision signals acquired in Env, which helps the evolution progress.

###  4.4 Evolution Progress for Self-Training Frameworks

Figure 3: Performance evolution of self-training methods on LLaMA2-Chat 13B
model. _Reinforced Self-Training_ approaches are represented by dashed lines,
while _Env-guided_ ones are in solid lines.

In Figure 3, we present the iterative evolution curves of the self-training
frameworks with LLaMA2-Chat (13B) as the LLM, which clearly shows the
procedure of weak-to-strong transformation. Limited by space, we leave the
discussion on the evolution of both performance and explored sample numbers
with the 7B version in Appendix D.1.

####  ENVISIONS combines high evolutionary efficiency and sustainability.

In the initial iterations, ENVISIONS demonstrates swift adaptability to
different scenarios. This indicates that exceptional performance can be
achieved with minimal time for data collection in ENVISIONS. Additionally,
ENVISIONS stands out as a more sustainable option when compared to other
baselines. For instance, in math reasoning tasks of Fig. 3(b), all baseline
methods achieve saturated performance levels by $6^{th}$ iteration. However,
our framework continues to exhibit evolutionary progress.

#### Reinforced baselines are largely flawed during iterations.

The incorporation of reinforced loss (e.g., DPO) brings difficulty in
optimization and greatly restricts the evolutionary scales of the LLM to adapt
to the neural-symbolic scenarios. _Self-Rewarding_ exhibits largely reduced
benefits during iterations, in contrast to its impressive performance in NL-
centric tasks. For _Iterative SFT+DPO_ , the SFT stage boosts the ability in
effective exploration. However, the subsequent DPO stage imposes a slight
improvement while significantly reduce the training efficiency.

###  4.5 Generalization to Various Backbones

To demonstrate the generalizability, we apply ENVISIONS to enhance two
additional base LLMs on mathematical reasoning tasks: (1) DeepSeek-Chat [51]
model of 7B size,

Figure 4: Generalization to different LLMs. The performances on math reasoning
tasks are reported.

which is a foundational LLM and (2) Llemma [52], a competent domain-specific
LLM optimized for math reasoning. Figure 4 shows the comparisons with _Few-
shot Prompting_ and _Distill GPT4-then-Finetune_. It is observed that our
framework still works for strong foundation LLMs, with 9.20% and 14.78%
performance boosts for DeepSeek-Chat and Llemma respectively. This
demonstrates that our framework can not only convert LLMs from weak to strong,
but also elevate LLMs from strong to stronger.

##  5 Analysis

This section will make an in-depth analysis of the underlying reason behind
ENVISIONS’s success.

Models | Agent | Math Reasoning | Logical Reasoning | Avg.
---|---|---|---|---
MiniWob++ | GSM8K | MATH | GSM-H | SVAMP | ASDiv | ProofWriter | RuleTaker
LLaMA-2-Chat (7B)
ENVISIONS | 85.38 | 58.98 | 19.00 | 48.52 | 72.40 | 69.80 | 52.83 | 62.63 | 58.69
w/o self-refine | 84.92 | 56.86 | 18.20 | 48.14 | 68.70 | 67.89 | 42.00 | 58.60 | 55.66
w/o self-reward | 84.47 | 58.61 | 18.75 | 47.92 | 71.10 | 68.46 | 47.33 | 59.61 | 57.03
w/o candidate pool | 83.86 | 57.77 | 17.55 | 47.16 | 70.90 | 68.03 | 49.17 | 59.18 | 56.70
w/o $\mathcal{L}_{2}$ loss | 81.89 | 55.88 | 18.90 | 47.16 | 67.60 | 67.75 | 47.67 | 57.88 | 55.59
LLaMA-2-Chat (13B)
ENVISIONS | 87.12 | 68.31 | 26.04 | 57.54 | 78.30 | 75.52 | 54.83 | 60.84 | 63.56
w/o self-refine | 84.24 | 65.96 | 24.95 | 55.34 | 77.70 | 73.90 | 51.00 | 57.59 | 61.34
w/o self-reward | 85.45 | 67.02 | 25.59 | 55.57 | 77.80 | 74.05 | 51.50 | 60.69 | 62.21
w/o candidate pool | 85.61 | 66.89 | 24.19 | 53.07 | 77.20 | 72.90 | 51.33 | 58.96 | 61.27
w/o $\mathcal{L}_{2}$ loss | 81.59 | 63.08 | 20.00 | 51.18 | 74.30 | 71.23 | 50.33 | 60.19 | 58.99

Table 3: Ablation studies on key components.

###  5.1 What is the Impact of Key Components in ENVISIONS?

Some key components are ablated independently to verify their effectiveness in
Table 3. _w/o self-refine_ ablates both the self-refinement process (i.e.,
Step ④-⑥) and $\mathcal{L}_{2}$. _w/o self-rewards_ replaces the trajectory
ranking on the self-rewarding strategy with random sampling. _w/o long-term
memory_ only utilizes the generated trajectories from the current iteration
for training. _w/o $\mathcal{L}_{2}$ loss_ ablates the optimization with
positive-negative trajectory pairs.

Of these components, self-refine-oriented optimizations (i.e., self-refinement
and $\mathcal{L}_{2}$ loss) play key roles in boosting the performances. As
one of the key contributions, the design of $\mathcal{L}_{2}$ loss leads to
3.10%-4.57% improvements in ENVISIONS. It makes full use of negative
trajectories while maintaining training efficiency in an RL-free style.
Especially in agent tasks, ENVISIONS benefits a lot from $\mathcal{L}_{2}$
loss, with 3.49%-5.53% performance gains.

Figure 5: In-depth analysis from three perspectives. The first row (i.e.,
(a),(b),(c)) and the second row (i.e., (d),(e),(f)) represent the results on
LLaMA2-Chat (7B) and LLaMA2-Chat (13B) respectively. Plots in fig.(b),(e)
correspond to the methods represented by the same colors in fig.(a),(d).

###  5.2 What is Behind the Superiority of ENVISIONS?

We provide in-depth evidence and analysis on the superiority of ENVISIONS from
three distinctive views: (1) exploratory ability and stability; (2) log
probability margin between positive and negative solutions; and (3) diversity
of synthetic samples.

#### Balanced exploratory ability and stability are key to success in weak-to-
strong.

To effectively navigate the environment and acquire new skills autonomously,
two factors are crucial: 1) promptly resolving extensive samples to collect
correct trajectories, and 2) minimizing the potential loss of knowledge from
previously-solved samples. We employ two metrics _exploratory ability_ and
_stability_ to evaluate the LLM (both of them are the higher, the better).
Refer to Appendix C for definition details. In Figure 5(a)(d), ENVISIONS
demonstrates remarkable performance in achieving a balance between exploratory
ability and stability. By leveraging the candidate trajectory pool and self-
rewards, ENVISIONS effectively retains high-quality positive solutions during
training, significantly mitigating the issue of forgetting previous
trajectories. Additionally, the RL-free loss $\mathcal{L}_{2}$ enables
flexible updates of the LLM, enhancing its exploration capabilities.

#### Clearly distinguishing positive and negative solutions can help the LLM
optimization.

During the optimization process, it is inevitable for the log probability of
both positive and negative trajectories to increase simultaneously [53].
However, clearly keeping the probability margins ($\Delta\mathrm{logp}$)
between positive-negative pairs is crucial to facilitate the optimization.
Fig. 5(b)(e) shows the analysis of $\Delta\mathrm{logp}$ during iterations. It
is observed ENVISIONS keeps the margin within a reasonable range, while
reinforced methods exhibit a rapid decrease in $\Delta\mathrm{logp}$. It
indicates the unsuitablity of DPO to the exploration setting and the
importance of feedback from Env. Notably, _STaR+Env._ fails to keep the stable
margins in the math domain, since it merely utilizes positive data for
training, which fails to distinguish negative ones and leads to overfiting on
the limited number of solutions. Such finding corresponds to the lack of
exploratory ability in Fig. 5(a)(d).

#### Diverse trajectories are what you need for self-training.

In Fig. 5(c)(f), we compare the number of correct and unique trajectories by
the last iteration. It demonstrates the huge strengths of ENVISIONS in
synthesizing diverse trajectories. It largely surpasses _Reinforced Self-
Training_ approaches, which is one of the underlying reasons for our
superiority. In fact, the LLM updates in RL methods are restricted by _KL_
constraints, which ultimately impact the diversity of the generated
trajectories. Moreover, _Distill GPT-4_ and _Distill Claude2_ lead to 10,831
and 8,561 diverse trajectories with one iteration. Since repeatedly calling
strong LLMs involves extremely high cost and cumbersome prompt optimizations,
they are far from sustainable compared with ENVISIONS.

###  5.3 How does the Training Recipe Matter in Iterative Self-Exploration?

Tasks | Cont. | ENVISIONS | $\Delta$
---|---|---|---
LLaMA-2-Chat (7B)
Agent | 78.18 | 85.38 | +7.20
Math Reasoning | 51.20 | 53.74 | +2.54
Logic Reasoning | 46.20 | 57.73 | +11.53
Average | 53.32 | 58.69 | +5.37
Table 4: Comparisons between training strategies. _Cont._ column denotes the
performances of ENVISIONS under the continual training setting.

In each iteration of ENVISIONS, we optimize the policy LLM from scratch (e.g.,
LLaMA2-Chat) with the updated training trajectories. Such a training recipe is
expected to bring stability to the training process, compared with the
strategy of continuous training based on previous checkpoints. Table. 4
presents the performance comparisons. Obvious superiority of ENVISIONS is
observed across these three domains, with an average improvement of 5.37%.
Training from previous checkpoints does affect the exploration. For the RL-
based self-training method, the training of the policy LLM is constrained
within the range of the reference model by the KL term. In order to enable
continuous evolution, the policy LLM is required to be updated from the
checkpoint of the previous iteration. It is also one of the main causes of
their sub-optimal performances.

##  6 Conclusion

This paper focuses on converting LLMs from weak to strong in increasingly
promising neural-symbolic scenarios, without human-annotated symbolic training
data. In view of two key challenges, i.e., 1) the scarcity of symbolic
training data, and 2) the inherent weakness of LLMs in addressing SL, we
conclude the env-guided self-training approach. Built on it, we propose a
novel neural-symbolic self-training framework ENVISIONS. Extensive experiments
across three domains verify the remarkable performances. In-depth analysis on
the superiority of ENVISIONS from three distinctive views provide novel
insights for future researches.

## References

  * Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Team et al. [2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Yin et al. [2023] Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang.  Do large language models know what they don’t know?  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 8653–8665, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-acl.551.  URL https://aclanthology.org/2023.findings-acl.551.
  * Ivison et al. [2023] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi.  Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ , 35:27730–27744, 2022.
  * Yuan et al. [2024] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston.  Self-rewarding language models.  _arXiv preprint arXiv:2401.10020_ , 2024.
  * Chen et al. [2024] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu.  Self-play fine-tuning converts weak language models to strong language models.  _arXiv preprint arXiv:2401.01335_ , 2024.
  * Rosset et al. [2024] Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie.  Direct nash optimization: Teaching language models to self-improve with general preferences.  _arXiv preprint arXiv:2404.03715_ , 2024.
  * Xu et al. [2023a] Fangzhi Xu, Zhiyong Wu, Qiushi Sun, Siyu Ren, Fei Yuan, Shuai Yuan, Qika Lin, Yu Qiao, and Jun Liu.  Symbol-llm: Towards foundational symbol-centric interface for large language models.  _arXiv preprint arXiv:2311.09278_ , 2023a.
  * Liu et al. [2023a] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone.  Llm+ p: Empowering large language models with optimal planning proficiency.  _arXiv preprint arXiv:2304.11477_ , 2023a.
  * Gou et al. [2023] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al.  Tora: A tool-integrated reasoning agent for mathematical problem solving.  _arXiv preprint arXiv:2309.17452_ , 2023.
  * Hu et al. [2023] Hanxu Hu, Hongyuan Lu, Huajian Zhang, Yun-Ze Song, Wai Lam, and Yue Zhang.  Chain-of-symbol prompting elicits planning in large langauge models.  _arXiv preprint arXiv:2305.10276_ , 2023.
  * Zheng et al. [2023] Longtao Zheng, Rundong Wang, and Bo An.  Synapse: Leveraging few-shot exemplars for human-level computer control.  _arXiv preprint arXiv:2306.07863_ , 2023.
  * Wu et al. [2024] Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong.  Os-copilot: Towards generalist computer agents with self-improvement.  _arXiv preprint arXiv:2402.07456_ , 2024.
  * Xu et al. [2023b] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  Wizardlm: Empowering large language models to follow complex instructions.  _arXiv preprint arXiv:2304.12244_ , 2023b.
  * Gulcehre et al. [2023] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al.  Reinforced self-training (rest) for language modeling.  _arXiv preprint arXiv:2308.08998_ , 2023.
  * Singh et al. [2023] Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al.  Beyond human data: Scaling self-training for problem-solving with language models.  _arXiv preprint arXiv:2312.06585_ , 2023.
  * Rafailov et al. [2024] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms, 2017.
  * Liu et al. [2023b] Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu.  Statistical rejection sampling improves preference optimization.  In _The Twelfth International Conference on Learning Representations_ , 2023b.
  * Wang et al. [2023a] Jianing Wang, Qiushi Sun, Nuo Chen, Chengyu Wang, Jun Huang, Ming Gao, and Xiang Li.  Uncertainty-aware parameter-efficient self-training for semi-supervised language understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 7873–7884, Singapore, December 2023a. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.528.  URL https://aclanthology.org/2023.findings-emnlp.528.
  * Zelikman et al. [2022] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman.  Star: Bootstrapping reasoning with reasoning.  _Advances in Neural Information Processing Systems_ , 35:15476–15488, 2022.
  * Ni et al. [2022] Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Alex Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao.  Learning math reasoning from self-sampled correct and partially-correct solutions.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Mukherjee et al. [2023] Subhabrata (Subho) Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah.  Orca: Progressive learning from complex explanation traces of gpt-4.  _arXiv: Computation and Language_ , June 2023.  URL https://www.microsoft.com/en-us/research/publication/orca-progressive-learning-from-complex-explanation-traces-of-gpt-4/.
  * Deng et al. [2023] Yihe Deng, Weitong Zhang, Zixiang Chen, and Quanquan Gu.  Rephrase and respond: Let large language models ask better questions for themselves.  _arXiv preprint arXiv:2311.04205_ , 2023.
  * Lee et al. [2024] Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipali, Michael W Mahoney, Kurt Keutzer, and Amir Gholami.  Llm2llm: Boosting llms with novel iterative data enhancement.  _arXiv preprint arXiv:2403.15042_ , 2024.
  * Yue et al. [2023] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.  Mammoth: Building math generalist models through hybrid instruction tuning.  _arXiv preprint arXiv:2309.05653_ , 2023.
  * Zeng et al. [2023] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang.  Agenttuning: Enabling generalized agent abilities for llms, 2023.
  * Cheng et al. [2024] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu.  Seeclick: Harnessing gui grounding for advanced visual gui agents.  _arXiv preprint arXiv:2401.10935_ , 2024.
  * Anil et al. [2023] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_ , 2023.
  * Wang et al. [2023b] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language models with self-generated instructions.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13484–13508, Toronto, Canada, July 2023b. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.754.  URL https://aclanthology.org/2023.acl-long.754.
  * Wei et al. [2023] Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, and Quoc Le.  Symbol tuning improves in-context learning in language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 968–979, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.61.  URL https://aclanthology.org/2023.emnlp-main.61.
  * Lifshitz et al. [2023] Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila McIlraith.  STEVE-1: A generative model for text-to-behavior in minecraft.  In _NeurIPS 2023 Foundation Models for Decision Making Workshop_ , 2023.  URL https://openreview.net/forum?id=QO2f1XqgiR.
  * Zhao et al. [2024] Zhonghan Zhao, Ke Ma, Wenhao Chai, Xuan Wang, Kewei Chen, Dongxu Guo, Yanting Zhang, Hongwei Wang, and Gaoang Wang.  Do we really need a complex agent system? distill embodied agent into a single model, 2024.
  * Gao et al. [2023] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.  PAL: Program-aided language models.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 10764–10799. PMLR, 23–29 Jul 2023.  URL https://proceedings.mlr.press/v202/gao23f.html.
  * Chen et al. [2023] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen.  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=YfZ4ZPt8zd.
  * Sun et al. [2024] Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al.  A survey of neural code intelligence: Paradigms, advances and beyond.  _arXiv preprint arXiv:2403.14734_ , 2024.
  * Pan et al. [2023] Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang.  Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 3806–3824, 2023.
  * Xu et al. [2023c] Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang.  On the tool manipulation capability of open-sourced large language models.  In _NeurIPS 2023 Foundation Models for Decision Making Workshop_ , 2023c.
  * Qin et al. [2023] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al.  Toolllm: Facilitating large language models to master 16000+ real-world apis.  _arXiv preprint arXiv:2307.16789_ , 2023.
  * Yang et al. [2024] Zonghan Yang, Peng Li, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu.  React meets actre: Autonomous annotations of agent trajectories for contrastive self-training.  _arXiv preprint arXiv:2403.14589_ , 2024.
  * Liu et al. [2018] Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang.  Reinforcement learning on web interfaces using workflow-guided exploration.  In _International Conference on Learning Representations (ICLR)_ , 2018.  URL https://arxiv.org/abs/1802.08802.
  * Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the math dataset.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.
  * Patel et al. [2021] Arkil Patel, Satwik Bhattamishra, and Navin Goyal.  Are nlp models really able to solve simple math word problems?  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, 2021.
  * Miao et al. [2020] Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su.  A diverse corpus for evaluating and developing english math word problem solvers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 975–984, 2020.
  * Tafjord et al. [2021] Oyvind Tafjord, Bhavana Dalvi, and Peter Clark.  Proofwriter: Generating implications, proofs, and abductive statements over natural language.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3621–3634, 2021.
  * Clark et al. [2021] Peter Clark, Oyvind Tafjord, and Kyle Richardson.  Transformers as soft reasoners over language.  In _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_ , pages 3882–3890, 2021.
  * Köpf et al. [2024] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al.  Openassistant conversations-democratizing large language model alignment.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * DeepSeek-AI [2024] DeepSeek-AI.  Deepseek llm: Scaling open-source language models with longtermism.  _arXiv preprint arXiv:2401.02954_ , 2024.  URL https://github.com/deepseek-ai/DeepSeek-LLM.
  * Azerbayev et al. [2023] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck.  Llemma: An open language model for mathematics.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Hong et al. [2024] Jiwoo Hong, Noah Lee, and James Thorne.  Reference-free monolithic preference optimization with odds ratio.  _arXiv preprint arXiv:2403.07691_ , 2024.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.

##  Appendix A Implementation Details

In this section, we provide some details of the implementation.

###  A.1 Training Details

The SFT training in both our framework and baselines is conducted on 8*A100
with a maximum length of 2,048. They are optimized and accelerated with
Deepspeed Zero3 and FlashAttention2. The AdamW optimizer [54] is leveraged
with a _Linear_ learning rate of 2e-5. The SFT training epoch number of each
iteration is set to 2, 1, 2 for agent, math reasoning, and logic reasoning
tasks respectively.

For the DPO training stage in baseline methods, it is also conducted on 8*A100
with a maximum length of 2,048. The _Linear_ learning rate is 5e-7 with a
warm-up ratio of 0.1. The epoch number for each domain is the same as the SFT
stage.

###  A.2 Test Tasks and Benchmark

The experiments in the main paper primarily cover three domains: web agent,
math reasoning, and logic reasoning. We have concluded some key details in
Table 1. In Table 5, we attach extra information on the test tasks and
benchmark.

Domains | Task name | Is Held-out? | #Test Samples | Beam Size | Max. Length | Sources
---|---|---|---|---|---|---
Web Agent | MiniWob++ |  |  30 ($\times$44 tasks) | 1 | 2,048 |  Liu et al. [43]
Math Reasoning | GSM8K |  | 1,319 | 2 | 2,048 | Cobbe et al. [44]
MATH |  | 4,001 | 2 | 2,048 | Hendrycks et al. [45]
GSM-Hard | ✓ | 1,319 | 2 | 2,048 | Gao et al. [36]
SVAMP | ✓ | 1,000 | 2 | 2,048 | Patel et al. [46]
AsDiv | ✓ | 2,096 | 2 | 2,048 | Miao et al. [47]
Logic Reasoning | ProofWriter |  | 600 | 1 | 4,096 | Tafjord et al. [48]
RuleTaker | ✓ | 1,389 | 1 | 4,096 | Clark et al. [49]

Table 5: Details of test tasks and benchmarks.

Unless otherwise stated, all these test tasks are evaluated under the zero-
shot setting. For MiniWob++ benchmark, we select 44 tasks for the test [30],
each with 30 randomly generated samples. All the above settings are consistent
among all baseline methods.

##  Appendix B Pseudocode of ENVISIONS

The self-training framework ENVISIONS can be expressed through the following
pseudocode.

Input: Data pair $\\{(x,y)\\}$, environment Env, candidate trajectory pool
POOL, weak LLM $\pi_{\theta_{0}}$, number of generated samples $K$, number of
iteration $T$.

Output: Strong LLM $\pi_{\theta}^{*}$.

// Initialize

$\pi_{\theta}\leftarrow\pi_{\theta_{0}}$

// Start the Loop

for  _$i=1$ to $T$_ do

for  _each $x$ in the input_ do

// 1-Online Exploration

Generate $K$ symbolic solutions with self-rewards:
$\\{a_{k}\\}_{k=1}^{K},\\{r_{k}\\}_{k=1}^{K}\sim\pi_{\theta}(\cdot|x)$.

Get binary rewards by executing in Env:
$\\{b_{k}\\}_{k=1}^{K}\longleftarrow\mathbb{I}[{\textbf{{Env}}(a_{k})==y}]$.

Generate self-refined solutions with self-rewards:
$\\{\widetilde{a}_{k}\\}_{k=1}^{K},\\{\widetilde{r}_{k}\\}_{k=1}^{K}\sim\pi_{\theta}(\cdot|x;a_{k})$.

Get binary rewards by executing in Env:
$\\{\widetilde{b}_{k}\\}_{k=1}^{K}\longleftarrow\mathbb{I}[{\textbf{{Env}}(\widetilde{a}_{k})==y}]$.

Let
$T_{k}=(x,y,a_{k},b_{k},r_{k}),\widetilde{T}_{k}=(x,y,\widetilde{a}_{k},\widetilde{b}_{k},\widetilde{r}_{k})$
denote the collected trajectories.

// 2-Traj. Filtering and Candidate Pool Updating

Filter the superior trajectory $T_{k}^{*}$ from $T_{k}$ and
$\widetilde{T}_{k}$ with binary rewards and self-rewards.

Update the candidate pool with $T_{k}^{*}$.

end for

// 3-Training

Rank and retrieve positive-only training set $U_{1}$ and positive-negative
pairs $U_{2}$ from POOL.

Optimize $\pi_{\theta_{0}}$ to $\pi_{\theta}^{*}$ with
$\mathcal{L}=-\sum\limits_{(x,a^{+})\sim{U}_{1}}{\mathrm{log}}\,p_{\theta_{0}}(a^{+}|x)-\sum\limits_{(x,a^{+},a^{-})\sim{U}_{2}}{\mathrm{log}}\,p_{\theta_{0}}(a^{+}|x;a^{-})$.

Update the policy LLM for the next iteration:
$\pi_{\theta}\leftarrow\pi_{\theta}^{*}$

end for

// Output the enhanced LLM

Return $\pi_{\theta}^{*}$;

Algorithm 1 A Neural-Symbolic Self-Training Framework ENVISIONS

##  Appendix C Definition of _Exploratory Ability_ and _Stability_

(1) Whether the policy LLM can rapidly explore large amounts of correct
samples, and (2) whether it can mitigate the issue of forgetting previously-
solved samples are two key factors to evaluate LLMs in interacting with the
environment. We define _Exploratory Ability_ (EA) and _Stability_ (STB)
respectively as the metrics. The calculation of the metrics is defined as
follows:

Suppose that we have the input set $M$. In the $i^{th}$ iteration, the solved
sample (with correct trajectories) constitute of set $M_{i}$.
$\bigcup_{j=1}^{i-1}{M_{j}}$ contains all the previously-solved samples from
the iteration $1$ to $i-1$. And $M_{i}\cup\bigcup_{j=1}^{i-1}{M_{j}}$
comprises the overlapped successful samples between the current iteration and
preceding iterations. $M_{i}\setminus\bigcup_{j=1}^{i-1}{M_{j}}$ denotes the
sample set that are newly solved in the current iteration $i$. Based on the
definition, we accumulate to obtain the overall EA and STB of the entire
process:

$\textrm{EA}=\sum\limits_{i=2}^{T}\frac{|M_{i}\setminus\bigcup_{j=1}^{i-1}{M_{j}}|}{|\bigcup_{j=1}^{i-1}{M_{j}}|},\quad\textrm{STB}=\sum\limits_{i=2}^{T}\frac{|M_{i}\cap\bigcup_{j=1}^{i-1}{M_{j}}|}{|\bigcup_{j=1}^{i-1}{M_{j}}|}$
(7)

where $|\cdot|$ is the number of samples in the given set. $T$ is the total
number of iterations.

Take the process of 2 iterations as an example, suppose the iteration 1
explores 1,000 correct samples. Iteration 2 obtains 1200 correct samples,
including 800 previously-solved samples and 400 newly-solved samples. Then,
$\textrm{EA}=400/1000$ and $\textrm{STB}=800/1000$.

##  Appendix D Supplementary Results

###  D.1 Evolution Progress

Apart from the performance evolution curves with the LLaMA2-Chat 13B model
presented in Figure 3, we expand the discussion on the 7B version. In Figure
6, we visualize the evolution progress of self-training methods on both the
model performance and the number of explored samples. The explored sample
denotes that one input $x$ is solved by at least one generated symbolic
solution $a_{k}$ (i.e., $b_{k}=1$). We count the number of explored samples at
each iteration to make the figure.

(a) Evolution of performance.

(b) Evolution of explored sample numbers.

Figure 6: Evolution curves on LLaMA2-Chat 7B version across agent, math, and
logic reasoning domains. (a) is the performance evolution progress. (b)
denotes the evolution of explored sample numbers.

From the results, the performances of the frameworks are positively correlated
with the ability to continuously explore correct trajectories. ENVISIONS
presents great superiority, especially in the logic reasoning tasks. Compared
with our proposed _Env-guided Self-Training_ approach, _Reinforced Self-
Training_ approach appears to be weaker at exploring new samples. This finding
is consistent with Figure 5 in the main paper.

###  D.2 Scaling of $K$

Models | Agent | Math Reasoning | Logical Reasoning | Avg.
---|---|---|---|---
MiniWob++ | GSM8K | MATH | GSM-H | SVAMP | ASDiv | ProofWriter | RuleTaker
K=2 | 78.56 | 53.60 | 17.37 | 44.96 | 67.20 | 66.84 | 35.17 | 49.82 | 51.69
K=5 | 85.38 | 58.98 | 19.00 | 48.52 | 72.40 | 69.80 | 52.83 | 62.63 | 58.69
K=10 | 79.24 | 58.30 | 21.89 | 48.29 | 67.90 | 69.75 | 53.50 | 61.99 | 57.61
K=15 | 79.55 | 57.47 | 23.72 | 46.63 | 69.80 | 70.28 | 54.83 | 59.54 | 57.73

Table 6: Scaling of $K$ with LLaMA2-Chat (7B) as the base LLM. In the main
results, we implement $K=5$ for illustration.

The hyper-parameter $K$ controls the number of generated candidate symbolic
solutions at each generation step. In the main results, we only implement
$K=5$ for illustration.

In Table 6, we present performances under various choices of $K$, including 2,
5, 10, and 15. Considering the training cost, we only include LLaMA2-Chat (7B)
as the base LLM. From the results, we conclude the following takeaways:

#### Moderate value of $K$ leads to the optimal performances.

When $K=5$, ENVISIONS demonstrates superior performances, especially on
agentic tasks (i.e., MiniWob++ benchmark). However, when reducing the value of
$K$ (i.e., $K=2$), the overall performances of ENVISIONS drop a lot. It
indicates that keeping a moderate number of candidate solutions in each
generation step benefits the self-training process.

#### Scaling of $K$ does not bring significant improvements.

Scaling $K$ from 5 to 10 and 15 does bring improvements on some challenging
tasks (e.g., MATH). However, this observation is not consistent across various
tasks. Generally, the average performances remain stable with $K$ increasing.

###  D.3 MiniWob++ Results Per Tasks

Table 7 shows the performance of ENVISIONS on each of the 44 MiniWob++ tasks.

| 1-shot | Distill GPT4 | Distill Claude2 | STAR+Env. | Self-Rewarding | iter. SFT+DPO | Ours
---|---|---|---|---|---|---|---
LLaMA-2-Chat (7B)
choose-date | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
choose-list | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-button | 0.00 | 100.00 | 100.00 | 100.00 | 100.00 | 96.67 | 100.00
click-button-sequence | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 96.67
click-checkboxes | 20.00 | 100.00 | 96.67 | 100.00 | 100.00 | 100.00 | 100.00
click-checkboxes-large | 20.00 | 86.67 | 96.67 | 86.67 | 66.67 | 100.00 | 100.00
click-checkboxes-soft | 0.00 | 6.67 | 30.00 | 50.00 | 0.00 | 63.33 | 76.67
click-checkboxes-transfer | 56.67 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-collapsible | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-color | 53.33 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-dialog | 100.00 | 100.00 | 100.00 | 100.00 | 0.00 | 100.00 | 100.00
click-dialog-2 | 0.00 | 26.67 | 73.33 | 100.00 | 73.33 | 100.00 | 100.00
click-link | 73.33 | 93.33 | 93.33 | 93.33 | 93.33 | 93.33 | 93.33
click-option | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-scroll-list | 56.67 | 100.00 | 100.00 | 100.00 | 96.67 | 100.00 | 100.00
click-shades | 93.33 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-shape | 0.00 | 70.00 | 53.33 | 63.33 | 16.67 | 50.00 | 70.00
click-tab | 100.00 | 100.00 | 100.00 | 96.67 | 26.67 | 56.67 | 100.00
click-test | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-test-2 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-widget | 96.67 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
copy-paste | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
copy-paste-2 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
enter-date | 3.33 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
enter-password | 96.67 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
enter-text | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
enter-text-dynamic | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
enter-time | 0.00 | 30.00 | 0.00 | 0.00 | 0.00 | 0.00 | 43.33
focus-text | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
focus-text-2 | 33.33 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
guess-number | 6.67 | 0.00 | 6.67 | 10.00 | 6.67 | 10.00 | 10.00
identify-shape | 0.00 | 56.67 | 80.00 | 100.00 | 56.67 | 50.00 | 100.00
multi-layouts | 3.33 | 96.67 | 86.67 | 100.00 | 76.67 | 96.67 | 100.00
multi-orderings | 0.00 | 93.33 | 100.00 | 100.00 | 80.00 | 100.00 | 100.00
navigate-tree | 60.00 | 60.00 | 60.00 | 60.00 | 60.00 | 60.00 | 60.00
read-table | 70.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
search-engine | 3.33 | 100.00 | 100.00 | 100.00 | 43.33 | 0.00 | 100.00
simple-algebra | 6.67 | 50.00 | 63.33 | 80.00 | 6.67 | 3.33 | 73.33
simple-arithmetic | 0.00 | 86.67 | 90.00 | 100.00 | 40.00 | 73.33 | 96.67
social-media-all | 30.00 | 100.00 | 100.00 | 30.00 | 0.00 | 30.00 | 30.00
text-transform | 66.67 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
unicode-test | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
use-slider | 0.00 | 6.67 | 6.67 | 6.67 | 6.67 | 6.67 | 6.67
use-spinner | 0.00 | 6.67 | 6.67 | 6.67 | 6.67 | 0.00 | 0.00
Average | 51.14 | 81.14 | 82.80 | 83.71 | 69.47 | 77.05 | 85.38
LLaMA-2-Chat (13B)
choose-date | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
choose-list | 96.67 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-button | 96.67 | 100.00 | 100.00 | 100.00 | 100.00 | 96.67 | 100.00
click-button-sequence | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-checkboxes | 30.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-checkboxes-large | 26.67 | 86.67 | 96.67 | 90.00 | 43.33 | 90.00 | 93.33
click-checkboxes-soft | 0.00 | 3.33 | 46.67 | 90.00 | 20.00 | 60.00 | 90.00
click-checkboxes-transfer | 10.00 | 100.00 | 96.67 | 100.00 | 100.00 | 100.00 | 100.00
click-collapsible | 100.00 | 100.00 | 96.67 | 100.00 | 100.00 | 100.00 | 100.00
click-color | 56.67 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-dialog | 100.00 | 100.00 | 100.00 | 100.00 | 0.00 | 100.00 | 100.00
click-dialog-2 | 0.00 | 26.67 | 73.33 | 100.00 | 73.33 | 100.00 | 100.00
click-link | 70.00 | 93.33 | 93.33 | 93.33 | 93.33 | 93.33 | 93.33
click-option | 0.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-scroll-list | 63.33 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-shades | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-shape | 10.00 | 76.67 | 56.67 | 86.67 | 10.00 | 66.67 | 73.33
click-tab | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-test | 100.00 | 100.00 | 96.67 | 100.00 | 100.00 | 100.00 | 100.00
click-test-2 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
click-widget | 96.67 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
copy-paste | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
copy-paste-2 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 96.67
enter-date | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
enter-password | 100.00 | 96.67 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
enter-text | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
enter-text-dynamic | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
enter-time | 0.00 | 0.00 | 23.33 | 0.00 | 0.00 | 0.00 | 93.33
focus-text | 0.00 | 100.00 | 100.00 | 100.00 | 96.67 | 100.00 | 100.00
focus-text-2 | 63.33 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
guess-number | 6.67 | 0.00 | 6.67 | 3.33 | 6.67 | 10.00 | 6.67
identify-shape | 0.00 | 10.00 | 90.00 | 100.00 | 20.00 | 90.00 | 100.00
multi-layouts | 66.67 | 100.00 | 100.00 | 100.00 | 86.67 | 96.67 | 100.00
multi-orderings | 56.67 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
navigate-tree | 60.00 | 60.00 | 60.00 | 60.00 | 60.00 | 60.00 | 56.67
read-table | 76.67 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
search-engine | 90.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
simple-algebra | 23.33 | 76.67 | 80.00 | 80.00 | 43.33 | 36.67 | 83.33
simple-arithmetic | 56.67 | 100.00 | 100.00 | 100.00 | 100.00 | 96.67 | 100.00
social-media-all | 93.33 | 100.00 | 100.00 | 30.00 | 0.00 | 30.00 | 30.00
text-transform | 96.67 | 83.33 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
unicode-test | 93.33 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
use-slider | 0.00 | 6.67 | 6.67 | 6.67 | 6.67 | 6.67 | 10.00
use-spinner | 0.00 | 6.67 | 6.67 | 6.67 | 6.67 | 6.67 | 6.67
Average | 60.00 | 80.15 | 84.77 | 85.15 | 74.24 | 82.73 | 87.12

Table 7: Detailed performances on 44 MiniWob++ tasks.


††thanks: Email address for correspondence<EMAIL_ADDRESS>
# Nonanomalous heat transport in a one-dimensional composite chain.

Piero Olla  ISAC-CNR and INFN, Sez. Cagliari, I–09042 Monserrato, Italy.

###### Abstract

Translation-invariant low-dimensional systems are known to exhibit anomalous
heat transport. However, there are systems, such as the coupled-rotor chain,
where translation invariance is satisfied, yet transport remains diffusive. It
has been argued that the restoration of normal diffusion occurs due to the
impossibility of defining a global stretch variable with a meaningful
dynamics. In this Letter, an alternative mechanism is proposed, namely, that
the transition to anomalous heat transport can occur at a scale that, under
certain circumstances, may diverge to infinity. To illustrate the mechanism, I
consider the case of a composite chain that conserves local energy and
momentum as well as global stretch, and at the same time obeys, in the
continuum limit, Fourier’s law of heat transport. It is shown analytically
that for vanishing elasticity the stationary temperature profile of the chain
is linear; for finite elasticity, the same property holds in the continuum
limit.

##  I Introduction

Heat transport in solids is described on a phenomenological level by Fourier’s
law; the description fails, however, in low-dimensional systems, where heat
transport takes an anomalous character such that the thermal conductivity of
the material diverges with the sample size [1, 2, 3]. A standard method for
the evaluation of thermal conductivity in solids is provided by the Green-Kubo
formula [4] (see [2] for a simple derivation). In low dimensional systems,
however, the heat current fluctuation correlation, on which the formula is
based, diverges at large-scale, which prevents direct application of the
method and suggests breakup of normal transport. Indeed, analysis of such
divergences by renormalization techniques first allowed researchers to
determine the anomalous scaling exponent for the thermal conductivity in low-
dimensional systems [5] and implied that a coarse-grained description of the
fluctuations in terms of field variables in a laboratory reference frame must
take into account advection terms analogous to those in the Eulerian
description of a fluid.

The analogy in the relation between Lagrangian and Eulerian description in a
fluid, and the dynamics in the continuum limit, of a low-dimensional solid,
was recognized in [6] and constitutes the basis for the derivation of the
Nonlinear Fluctuating Hydrodynamic (NFH) theory [7]. The relevance of the
fluid mechanic point of view in the description of heat transport in a low-
dimensional solid is corroborated by the fact that the same anomalous
behaviors are observed in one-dimensional particle models where the only
interaction is provided by collisions, and which, at a coarse-grained level,
can be described as bona fide one-dimensional fluids [8, 9].

The key mechanism leading to the divergence of the field equations, and hence
to anomalous heat conduction in the systems under consideration, appears to be
the simultaneous conservation locally of energy and momentum [10, 11, 5]. More
recently, an additional condition has been identified in the fact that the
global stretch of the system must have a dynamical content [12, 13]. If any
such condition is violated—e.g. if the atoms in the chain interact with a
substrate, leading to translation invariance violation, or if, as in the case
of the coupled-rotor chain [14, 15], it is not possible to define a total
stretch for the system—normal diffusion is recovered. In the same way,
systems, such as the zero-range model [16] and the Kipnis-Presutti model [17]
to name a few, in which energy is randomly exchanged between atoms without
momentum conservation, are characterized by normal heat conduction.

To date, all analytical models of low-dimensional heat transport are based on
mimicking the role of anharmonicity in spatially redistributing the vibration
energy along the chain, by adding a stochastic component to the dynamics. The
strategy to microscopically implement stochasticity is not unique. In [18],
random collisions are assumed, with pairs of neighboring atoms exchanging
momentum while their total energy remains constant. In other models, three-
atom interactions are required to accommodate the joint conditions of energy
and momentum conservation. In [19], the stochastic component of the dynamics
is realized by a random walk in momentum space on the constant energy surface
of the system. NFH predicts that for generic interaction potentials, the
large-scale dynamics of energy and momentum preserving one-dimensional chains
should fall in the universality class of the Kardar-Parisi-Zhang model [7].
There are special cases, however, in which the predictions of the NFH theory
do not apply [7, 20], with finite-size effects, as well as weak chaos in the
interaction, making the detection of universal behaviors difficult [21].

The situation as regards experiments and numerical (atomistic) simulation of
more realistic systems is equally complicated. Results are indeed often
dependent on the properties of the material and the experimental or numerical
technique adopted (see [3] and references therein for an extended discussion).
Of particular interest is the possible presence of a diffusive range at small
scales, complicating the measurement of the anomalous scaling exponents
predicted by the theory; such crossover behaviors are indeed predicted in
particle systems [9].

The purpose of the present Letter is to study the crossover from small-scale
thermal diffusion to large-scale anomalous heat conduction in the specific
example of a “composite” chain, in which atoms interact with their neighbors
through harmonic forces and inclusions acting as random sources and sinks of
kinetic energy. Composite materials such as e.g. semiconductor perovskites
find application in photovoltaics, and proper characterization of their
thermal properties is particularly important [22]. The total momentum and
energy of the atoms and the inclusion involved in an interaction are
conserved, the ends of the chain are fixed, and thus all the conditions for
anomalous heat conduction in the system are satisfied. Yet, the analysis that
follows shows that the range in which heat transport is diffusive can greatly
exceed the range in which the dynamics of the chain is viscous. In particular,
heat transport becomes diffusive at all scales in the continuum limit.

##  II Outline of the model

The geometry of the system is illustrated in Fig. 1.

Figure 1:  Sketch of the composite chain: the two-bead assemblies (blue
online) represent the individual atoms; the cells underneath (magenta online)
represent the zero-mass mobile inclusions and the thermal baths.

Each two-bead assembly represents an atom, with the bars joining the beads
assumed rigid. The cells in the middle represent the inclusions, which for
simplicity are taken to be massless; the cells at the extremes of the chain
are the heat baths, whose position is fixed. Indicate with $N$ the number of
atoms in the chain and with $L=N\eta$ the chain’s length. The inclusions act
on the beads as Langevin baths with friction coefficient $\Gamma/2$ and noise
amplitude

$\displaystyle\langle\xi_{k+1/2}(0)\xi_{j+1/2}(t)\rangle=2m\Gamma\langle\epsilon\rangle_{k+1/2}\delta_{kj}\delta(t).$
(1)

The adopted Langevin dynamics may be interpreted as the result of coarse
graining the fast internal degrees of freedom in the inclusions, with
$\epsilon_{k+1/2}$ an energy variable 111In the case of an approximately
linear internal dynamics, $\epsilon_{k+1/2}$ would be the energy per degree of
freedom of the inclusion.  that is going to be determined dynamically from the
condition of local energy conservation (see below).

The thermal baths at the chain extremes act on the respective atoms in the
same way as the inclusions, with energy variables $\epsilon_{1/2}$ and
$\epsilon_{N+1/2}$ replaced in Eq. (1) by fixed temperatures $T_{L}/2$ and
$T_{R}/2$, $T_{L}-T_{R}=2\Delta T$ (the Boltzmann constant $k_{B}$ is set
equal to 1 throughout the calculation).

Indicate with $q_{k}$ the displacement of the atoms from their equilibrium
position and with $p_{k}=m\dot{q}_{k}$ the associated momentum. As illustrated
in Fig. 1, the elastic force acts in parallel with that by the inclusion; the
chain dynamics is then described by the system of equations, in the bulk
$1<k<N$,

$\displaystyle\dot{p}_{k}$ $\displaystyle=$
$\displaystyle\Gamma(p_{k+1}+p_{k-1}-2p_{k})/2$ (2) $\displaystyle+$
$\displaystyle\alpha(q_{k+1}+q_{k-1}-2q_{k})/2+\xi_{k-1/2}-\xi_{k+1/2},$

while at the ends of the chain,

$\displaystyle\dot{p}_{1}$ $\displaystyle=$
$\displaystyle\Gamma(p_{2}-2p_{1})/2$ (3) $\displaystyle+$
$\displaystyle\alpha(q_{2}-2q_{1})/2+\xi_{1/2}-\xi_{3/2},$
$\displaystyle\dot{p}_{N}$ $\displaystyle=$
$\displaystyle\Gamma(p_{N-1}-2p_{N})/2$ (4) $\displaystyle+$
$\displaystyle\alpha(q_{N-1}-2q_{N})/2+\xi_{N-1/2}-\xi_{N+1/2}$

(Itô’s prescription is assumed throughout the Letter). The conservative nature
of the noise in Eqs. (2-4) should be noted, which distinguishes the present
model from ones in which local heat baths force the dynamics, such as e.g.
[24, 25].

It is possible to identify a microscopic elastic timescale
$\omega_{\eta}^{-1}=\sqrt{m/\alpha}$, with the magnitude of the ratio

$\displaystyle r=\omega_{\eta}/\Gamma$ (5)

determining whether the microscopic dynamics is dominated by elasticity or by
the effective friction generated by the inclusions. We can take the continuum
limit of Eq. (2), and the result is 222Note that $x$ in Eqs. (6) and (7) is a
Lagrangian variable; note also that the equations are linear, which implies
that neither viscosity nor sound speed renormalization is required in the
analysis that follows.:

$\displaystyle\partial_{t}^{2}q$ $\displaystyle=$
$\displaystyle\partial_{x}^{2}(c_{s}^{2}+\nu\partial_{t})q+\partial_{x}\xi,$
(6) $\displaystyle\langle\xi(x,t)\xi(0,0)$ $\displaystyle=$ $\displaystyle
2(\eta\langle\epsilon\rangle\nu/m)\delta(x)\delta(t),$ (7)

which describes wave propagation in a viscoelastic (Kelvin-Voigt) medium with
sound speed and viscosity, respectively,

$\displaystyle c_{s}=\eta\omega_{\eta}\quad{\rm and}\quad\nu=\eta^{2}\Gamma.$
(8)

From here, it is possible to define a viscous scale

$\displaystyle l_{\nu}=\nu/c_{s}=\eta/r,$ (9)

which identifies the upper limit of the viscosity-dominated range for the
dynamics.

To study the fluctuation dynamics, one needs an equation for the energy
variable $\epsilon_{k+1/2}$. One obtains such equation by imposing energy
conservation in the interaction between atoms and inclusions. The energy
budget in the interaction between atoms $k$ and $k+1$, and inclusion $k+1/2$
is obtained, for $1<k<N$, by evaluating the contribution to the variation of
kinetic energy of the two atoms,
$K_{k+1/2}=(p_{k+1}-p_{k})^{2}/(4m):=p_{k+1/2}^{2}/(4m)$, from the forcing by
the inclusion. One can write in general

$\displaystyle\dot{K}_{k+1/2}=-\dot{E}_{k+1/2}+\ldots,$ (10)

where $E_{k+1/2}=E(\epsilon_{k+1/2})$ is the internal energy of the inclusion
and the dots stand for the contributions from the elastic forces and the
neighboring inclusions. Substituting Eq. (2) in the left hand side of Eq. (10)
yields then

$\displaystyle\dot{E}_{k+1/2}$ $\displaystyle=$ $\displaystyle
2\Gamma\Big{(}\frac{p^{2}_{k+1/2}}{4m}-\epsilon_{k+1/2}\Big{)}-\frac{p_{k+1/2}\xi_{k+1/2}}{m},$
(11)

where one recognizes in the term $\Gamma p_{k+1/2}^{2}/(2m)$ the work by the
friction forces, and in $2\Gamma\epsilon_{k+1/2}$ the average energy provided
to the two atoms by the fluctuating force.

##  III The purely viscous chain

In the $\alpha\to 0$ limit, the system provides an example of heat transport
by Brownian motion. Exactly as in the elastic case [27], it is possible to
evaluate the heat flow from the dynamics of the one-time correlations 333 Note
that once a constitutive relation $E_{k+1/2}=E(\epsilon_{k+1/2})$ is selected,
Eqs. (1-4) and (11), together with the kinematic condition $p_{k}=m\dot{q}$,
constitute a system of stochastic differential equation with multiplicative
noise. Linearity of the stochastic equations, nevertheless, makes the
equations for the correlations analytically solvable.. Define

$\displaystyle\Pi_{ik}=\langle p_{i}p_{k}\rangle,\ Z_{ik}=\langle
q_{i}p_{k}\rangle,\ Q_{ik}=\langle q_{i}q_{k}\rangle.$ (12)

For $\alpha=0$, $q_{i}$ is an irrelevant variable, and the equation for $\Pi$
and those for $Z$ and $Q$ decouple. The last two variables can then, for the
moment, be disregarded.

Let us focus first on the bulk; from Eq. (2), one obtains the equation for
$\Pi_{ik}$

$\displaystyle 2\Gamma^{-1}\dot{\Pi}_{k,k+l}$ $\displaystyle=$
$\displaystyle\Pi_{k+1,k+l}+\Pi_{k-1,k+l}-2\Pi_{k,k+l}$ (13) $\displaystyle+$
$\displaystyle\Pi_{k,k+1+l}+\Pi_{k,k-1+l}-2\Pi_{k,k+l}$ $\displaystyle+$
$\displaystyle 4m[(\delta_{l0}-\delta_{l1})\langle\epsilon\rangle_{k+1/2}$
$\displaystyle+$
$\displaystyle(\delta_{l0}-\delta_{l,-1})\langle\epsilon\rangle_{k-1/2}],$

where $1<k,k+l<N$. At stationarity, one gets from Eq. (11)

$\displaystyle 4m\langle\epsilon\rangle_{k+1/2}$ $\displaystyle=$
$\displaystyle\langle p_{k+1/2}^{2}\rangle$ (14) $\displaystyle=$
$\displaystyle\Pi_{kk}+\Pi_{k+1,k+1}-2\Pi_{k,k+1},$

which, substituted into Eq. (13), yields

$\displaystyle-2\Pi_{kk}+\Pi_{k+1,k+1}+\Pi_{k-1,k-1}$ $\displaystyle=$
$\displaystyle 0,$ (15) $\displaystyle-2\Pi_{k,k+1}+\Pi_{k,k+2}+\Pi_{k+1,k-1}$
$\displaystyle=$ $\displaystyle 0,$ (16)
$\displaystyle\Pi_{k+1,k+l}+\Pi_{k-1,k+l}+\Pi_{k,k+1+l}$
$\displaystyle\qquad+\Pi_{k,k-1+l}-4\Pi_{k,k+l}$ $\displaystyle=$
$\displaystyle 0,\quad|l|>1.$ (17)

The same procedure can be carried out at $k=1$,

$\displaystyle\Pi_{22}-3\Pi_{11}+2mT_{L}$ $\displaystyle=$ $\displaystyle 0,$
(18) $\displaystyle\Pi_{13}-2\Pi_{12}$ $\displaystyle=$ $\displaystyle 0,$
(19) $\displaystyle\Pi_{2l}-4\Pi_{1l}+\Pi_{1,l-1}+\Pi_{1,l+1}$
$\displaystyle=$ $\displaystyle 0,\quad l>2,$ (20)

and a similar set of equations is produced at $k=N$. The system of equations
(15-20) has the remarkable property that Eqs. (16,17,19,20), which involve
out-of-diagonal terms, decouple from Eqs. (15) and (18) on the diagonal.
Equations (15) and (18) tell us that $\Pi_{kk}$ has a linear profile:

$\displaystyle 2(mT_{L}-\Pi_{11})=\Pi_{kk}-\Pi_{k+1,k+1}$
$\displaystyle=2(\Pi_{NN}-mT_{R})=2m\Delta T/N.$ (21)

On the other hand, Eqs. (16,17,19,20) admit the zero solution $\Pi_{kl}=0$,
$k\neq l$, which is also necessarily unique, since a nonzero solution could
have arbitrary amplitude and lead to negative
$\langle(p_{l}+p_{k})^{2}\rangle$). From Eq. (21), it is then possible to
write 444 The same result could be obtained by imposing a linear profile for
$\langle\epsilon\rangle_{k+1/2}$, rather than solving self-consistently Eq.
(11). The operation would be equivalent to replacing the inclusions with zero-
mass thermal baths, and would result in the modification of the the higher-
order correlations and time-dependent component of the statistics.

$\displaystyle\Pi_{kl}=m\Big{(}T_{L}+\frac{2k-1}{N}\Delta
T\Big{)}\delta_{kl};$ (22)

The temperature profile along the purely viscous chain,
$T_{k}\equiv\Pi_{kk}/m\simeq 2\langle\epsilon_{k+1/2}\rangle$, is thus linear.

For $\alpha=0$, the heat transfer along the chain is mediated by the work on
the atoms by the inclusions; the average work by inclusion $k-1/2$ on the atom
to its right thus coincides with the heat flux at site $k$

$\displaystyle J_{k}$ $\displaystyle=$
$\displaystyle\Gamma\Big{[}\langle\epsilon\rangle_{k-1/2}-\frac{\langle
p_{k}(p_{k}-p_{k-1})\rangle}{2m}\Big{]},$ (23)

where it is understood that $p_{0}=0$. At stationarity, from Eqs. (14) and
(22),

$\displaystyle\frac{J_{k}}{\Gamma}=\frac{T_{L}}{2}-\frac{\Pi_{11}}{2m}=\frac{\Pi_{k-1,k-1}-\Pi_{kk}}{4m}=\frac{\Pi_{NN}}{2m}-\frac{T_{R}}{2},$

which, using again Eq. (22), implies Fourier’s law; after reinstating
Boltzmann’s constant,

$\displaystyle
J=\kappa\frac{T_{L}-T_{R}}{L},\qquad\kappa=\frac{k_{B}\nu}{4\eta}.$ (24)

Setting $T_{L}=T_{R}=T$, it is easy to verify from Eqs. (14), (23) and (22),
that at equilibrium $J_{k}=0$ and equipartition holds:
$\langle\epsilon\rangle_{k+1/2}=\Pi_{kk}/(2m)=T/2$.

##  IV The effect of finite elasticity

For finite $\alpha$, part of the heat transfer is mediated by the elastic
forces, with a contribution to the heat flux [1]:

$\displaystyle\delta J^{el}_{k}=\alpha(Z_{kk}-Z_{k+1,k})/(2m).$ (25)

To evaluate $\delta J^{el}_{k}$, we need an equation for $Z_{ik}$. Indicate

$\displaystyle A_{ik}=\Gamma Z_{ik}+\alpha Q_{ik}.$ (26)

The stationarity conditions $\dot{Z}_{k,k+l}+\dot{Z}_{k+l,k}=0$ and
$\dot{Z}_{k,k+l}-\dot{Z}_{k+l,k}=0$ take the form, in the bulk, from Eq. (2):

$\displaystyle-4\Pi_{k+l,k}/m=A_{k+l,k+1}+A_{k+l,k-1}$
$\displaystyle-2A_{k+l,k}+A_{k,k+l+1}+A_{k,k+l-1}-2A_{k,k+l},$ (27)
$\displaystyle A_{k+l,k+1}+A_{k+l,k-1}-2A_{k+l,k}$
$\displaystyle=A_{k,k+l+1}+A_{k,k+l-1}-2A_{k,k+l}.$ (28)

Equations (27) and (28) imply
$-2\Pi_{k+l,k}/m=A_{k+l,k+1}+A_{k+l,k-1}-2A_{k+l,k}$, better rewritten as

$\displaystyle\partial_{k}^{2}A_{lk}=-2\Pi_{lk}/m,\quad 1<k,l<N.$ (29)

where $\partial_{k}$ indicates finite difference:
$\partial_{k}^{2}f=f_{k+1}+f_{k-1}-2f_{k}$. Identify with a bar the zero-
viscosity component of quantities and with a tilde the respective correction,
$\Pi=\bar{\Pi}+\tilde{\Pi}$, and make the ansatz (to be verified a posteriori)
that if the chain is sufficiently short viscosity will dominate thermal
fluctuations. It is then possible to set in Eq. (29)
$\Pi_{lk}\simeq\bar{\Pi}_{lk}=\bar{\Pi}_{ll}\delta_{lk}$ [see Eq. (22)], which
yields the solution

$\displaystyle A_{lk}\simeq a_{l}+b_{l}k-\bar{\Pi}_{ll}|k-l|/m.$ (30)

To determine the coefficients $a_{l}$ and $b_{l}$, one needs boundary
conditions, which are provided by imposing stationarity at the ends of the
chain, $\dot{Z}_{1k}=\dot{Z}_{Nk}=0$; exploiting Eqs. (2), (3) and (4):

$\displaystyle 2\Pi_{l1}/m$ $\displaystyle=$ $\displaystyle 2A_{l1}-A_{l2},$
$\displaystyle 2\Pi_{lN}/m$ $\displaystyle=$ $\displaystyle
2A_{lN}-A_{l,N-1},\quad 1<l<N.$ (31)

From $\Pi_{1l}\simeq\Pi_{lN}\simeq 0$, $1<l<N$, the following relations are
then obtained:

$\displaystyle A_{l2}\simeq 2A_{l1},\quad A_{l,N-1}\simeq 2A_{lN},\quad
1<l<N.$ (32)

Substituting Eq. (30) into Eq. (32) yields

$\displaystyle a_{l}\simeq\frac{l\bar{\Pi}_{ll}}{m},\quad
b_{l}\simeq\frac{\bar{\Pi}_{ll}}{m}\frac{N+1-2l}{N+1},\quad 1<l<N.$ (33)

Exploiting Eq. (30) and the relation
$\bar{\Pi}_{kk}-\bar{\Pi}_{k+1,k+1}=2m\Delta T/N$ [see Eq. (21)], we obtain
the expression, valid for $1<l\leq k<N$,

$\displaystyle Z_{lk}$ $\displaystyle\simeq$
$\displaystyle\frac{A_{lk}-A_{kl}}{2\Gamma}=\frac{l(N+1-k)(\bar{\Pi}_{ll}-\bar{\Pi}_{kk})}{m\Gamma(N+1)}$
(34) $\displaystyle=$ $\displaystyle\frac{2l(N+1-k)(l-k)\Delta T}{\Gamma
N(N+1)}.$

Substituting Eq. (34) into Eq. (25) finally yields

$\displaystyle\delta J_{k}^{el}\simeq-\frac{\alpha(1+k)(N+1-k)\Delta T}{\Gamma
mN(N+1)},$ (35)

and hence, by comparing with Eq. (24), the estimate

$\displaystyle\delta J^{el}/J\sim r^{2}N=\eta L/l_{\nu}^{2},$ (36)

which tells us that as long as

$\displaystyle L\ll l_{\kappa}=l^{2}_{\nu}/\eta,$ (37)

heat transport remains diffusive. A similar estimate holds for the momentum
fluctuation amplitude, $\tilde{\Pi}/\bar{\Pi}\sim r^{2}N$ (see Supplemental
Material [30]), which confirms the ansatz at the basis of Eq. (30). Note that
in the continuum limit $\eta\to 0$ (all macroscopic quantities $L$, $Nk_{B}T$,
$Nm$, $\nu$ and $c_{s}$ fixed and finite), the diffusive scale $l_{\kappa}$
goes to infinity and Fourier’s law holds irrespective of the sample size.

An interesting question concerns the role of possible violations of stretch
conservation in the regime $L\ll l_{\kappa}$. By construction, for finite
$\alpha$, the total stretch $\delta L$ is controlled by elasticity, which
means that, strictly speaking, the total stretch is conserved. One may
nevertheless argue that if, for $l_{\nu}\ll L\ll l_{\kappa}$ (that is the
range where the dynamics of the chain is elastic), the ratio $\delta L/L$ in
the continuum limit were to diverge to infinity, the chain would be behaving
as if its endpoints were unconstrained. In other words, diffusive transport
for $l_{\nu}\ll L\ll l_{\kappa}$ could be the consequence of insufficient
conservation of stretch. We show below that this is not the case.

The continuum limit $\eta\to 0$ ($L$, $Nk_{B}T$, $Nm$, $\nu$ and $c_{s}$ fixed
and finite) corresponds to a regime $L\ll l_{\kappa}$ such that Eq. (22)
applies. It is then possible to estimate for the stretch $\delta
L\sim\sqrt{N\langle q_{k+1/2}^{2}\rangle}$, where $q_{k+1/2}=q_{k+1}-q_{k}$;
for small deviations from equilibrium, $T_{L}\simeq T_{R}\sim T$, $\langle
q_{k+1/2}^{2}\rangle\sim v_{th}^{2}m/\alpha=(v_{th}/\omega_{\eta})^{2}$, where
$v_{th}=\sqrt{k_{B}T/m}$ is the thermal velocity, which remains finite in the
limit. From Eqs. (8) and (9) one then gets

$\displaystyle\frac{\delta
L}{L}\sim\frac{N^{1/2}v_{th}}{L\omega_{\eta}}=\frac{N^{1/2}\eta
v_{th}}{Lc_{s}}=\frac{\sqrt{l_{\nu}\eta}}{L}\frac{v_{th}}{c_{s}},$ (38)

which tells us that the relative stretch vanishes in the continuum limit (all
quantities in the right hand side of the formula remain finite, except $\eta$,
which vanishes); this suggests that stretch conservation violations do not
play a role in the dynamics under consideration. (An alternative derivation of
the result is provided in the Supplemental Material [30].)

##  V Conclusion

The present analysis shows that heat transport in a composite chain with a
purely viscous microscopic dynamics, obeys Fourier’s law. By construction, no
internal forces are generated in response to stretching of the system, and we
thus have another example, beside that of the coupled-rotor chain, of a system
conserving local energy and momentum, in which global stretch is not conserved
and heat transport is diffusive.

If elasticity is finite, but there is a viscous range extending to macroscopic
scales, the range of scales where heat transport is diffusive greatly exceeds
the viscous range, and extends to infinity in the continuum limit. This is the
main result of the Letter. The global stretch fluctuations vanish in the
limit, which means that diffusive heat transport in such composite chain,
contrary to the case of the coupled-rotor chain, cannot be explained by non-
conservation of the stretch.

Some questions remain open. Is anomalous heat transport in the composite chain
recovered at scales larger than the diffusion length $l_{\kappa}$? Is the
existence of an extended diffusive range a common property of viscoelastic
one-dimensional chains? The answer is probably yes in both cases 555Note added
after publication: Eqs. (15-17) and Eq. (29) in the present Letter coincide
with Eqs. (14-15) and Eq. (36) in Ref. [18]; the answer to the first question
is therefore yes; the answer to the second question is again yes in the case
of the model of Ref. [18]. , but, to prove the statement, going beyond the
present perturbative—model-dependent analysis, would be required.

## References

  * Lepri  _et al._ [2003] S. Lepri, R. Livi, and A. Politi, Thermal conduction in classical low-dimensional lattices, Physics reports 377, 1 (2003).
  * Dhar [2008] A. Dhar, Heat transport in low-dimensional systems, Advances in Physics 57, 457 (2008).
  * Benenti  _et al._ [2022] G. Benenti, D. Donadio, S. Lepri, and R. Livi, Non-fourier heat transport in nanosystems, arXiv preprint arXiv:2212.09374 (2022).
  * Kubo  _et al._ [2012] R. Kubo, M. Toda, and N. Hashitsume,  _Statistical physics II: nonequilibrium statistical mechanics_ , Springer Series in Solid-State Sciences, Vol. 31 (Springer Science & Business Media, 2012).
  * Narayan and Ramaswamy [2002] O. Narayan and S. Ramaswamy, Anomalous heat conduction in one-dimensional momentum-conserving systems, Physical Review Letters 89, 200601 (2002).
  * Chetrite and Gawȩdzki [2009] R. Chetrite and K. Gawȩdzki, Eulerian and lagrangian pictures of non-equilibrium diffusions, Journal of Statistical Physics 137, 890 (2009).
  * Spohn [2014a] H. Spohn, Nonlinear fluctuating hydrodynamics for anharmonic chains, Journal of Statistical Physics 154, 1191 (2014a).
  * Kundu  _et al._ [2016] A. Kundu, O. Hirschberg, and D. Mukamel, Long range correlations in stochastic transport with energy and momentum conservation, Journal of Statistical Mechanics: Theory and Experiment 2016, 033108 (2016).
  * Miron  _et al._ [2019] A. Miron, J. Cividini, A. Kundu, and D. Mukamel, Derivation of fluctuating hydrodynamics and crossover from diffusive to anomalous transport in a hard-particle gas, Physical Review E 99, 012124 (2019).
  * Prosen and Campbell [2000] T. Prosen and D. K. Campbell, Momentum conservation implies anomalous energy transport in 1d classical lattices, Physical Review Letters 84, 2857 (2000).
  * Bonetto  _et al._ [2000] F. Bonetto, J. L. Lebowitz, and L. Rey-Bellet, Fourier’s law: a challenge to theorists, in  _Mathematical physics 2000_ (World Scientific, 2000) pp. 128–150.
  * Spohn [2014b] H. Spohn, Fluctuating hydrodynamics for a chain of nonlinearly coupled rotators, arXiv preprint arXiv:1411.3907 (2014b).
  * Das and Dhar [2014] S. G. Das and A. Dhar, Role of conserved quantities in normal heat transport in one dimenison, arXiv preprint arXiv:1411.5247 (2014).
  * Giardina  _et al._ [2000] C. Giardina, R. Livi, A. Politi, and M. Vassalli, Finite thermal conductivity in 1d lattices, Physical Review Letters 84, 2144 (2000).
  * Gendelman and Savin [2000] O. V. Gendelman and A. V. Savin, Normal heat conductivity of the one-dimensional lattice with periodic potential of nearest-neighbor interaction, Physical Review Letters 84, 2381 (2000).
  * Evans and Hanney [2005] M. R. Evans and T. Hanney, Nonequilibrium statistical mechanics of the zero-range process and related models, Journal of Physics A: Mathematical and General 38, R195 (2005).
  * Kipnis  _et al._ [1982] C. Kipnis, C. Marchioro, and E. Presutti, Heat flow in an exactly solvable model, Journal of Statistical Physics 27, 65 (1982).
  * Lepri  _et al._ [2009] S. Lepri, C. Mejia-Monasterio, and A. Politi, A stochastic model of anomalous heat transport: analytical solution of the steady state, Journal of Physics A: Mathematical and Theoretical 42, 025001 (2009).
  * Basile  _et al._ [2006] G. Basile, C. Bernardin, and S. Olla, Momentum conserving model with anomalous thermal conductivity in low dimensional systems, Physical Review Letters 96, 204303 (2006).
  * Lee-Dadswell [2015] G. R. Lee-Dadswell, Universality classes for thermal transport in one-dimensional oscillator systems, Physical Review E 91, 032102 (2015).
  * Lepri  _et al._ [2020] S. Lepri, R. Livi, and A. Politi, Too close to integrable: Crossover from normal to anomalous heat diffusion, Physical Review Letters 125, 040604 (2020).
  * Caddeo  _et al._ [2016] C. Caddeo, C. Melis, M. I. Saba, A. Filippetti, L. Colombo, and A. Mattoni, Tuning the thermal conductivity of methylammonium lead halide by the molecular substructure, Physical Chemistry Chemical Physics 18, 24318 (2016).
  * Note [1] In the case of an approximately linear internal dynamics, $\epsilon_{k+1/2}$ would be the energy per degree of freedom of the inclusion.
  * Bolsterli  _et al._ [1970] M. Bolsterli, M. Rich, and W. Visscher, Simulation of nonharmonic interactions in a crystal by self-consistent reservoirs, Physical Review A 1, 1086 (1970).
  * Bonetto  _et al._ [2004] F. Bonetto, J. L. Lebowitz, and J. Lukkarinen, Fourier’s law for a harmonic crystal with self-consistent stochastic reservoirs, Journal of Statistical Physics 116, 783 (2004).
  * Note [2] Note that $x$ in Eqs. (6) and (7) is a Lagrangian variable; note also that the equations are linear, which implies that neither viscosity nor sound speed renormalization is required in the analysis that follows.
  * Rieder  _et al._ [1967] Z. Rieder, J. Lebowitz, and E. Lieb, Properties of a harmonic crystal in a stationary nonequilibrium state, Journal of Mathematical Physics 8, 1073 (1967).
  * Note [3] Note that once a constitutive relation $E_{k+1/2}=E(\epsilon_{k+1/2})$ is selected, Eqs. (1-4) and (11), together with the kinematic condition $p_{k}=m\dot{q}$, constitute a system of stochastic differential equation with multiplicative noise. Linearity of the stochastic equations, nevertheless, makes the equations for the correlations analytically solvable.
  * Note [4] The same result could be obtained by imposing a linear profile for $\langle\epsilon\rangle_{k+1/2}$, rather than solving self-consistently Eq. (11). The operation would be equivalent to replacing the inclusions with zero-mass thermal baths, and would result in the modification of the the higher-order correlations and time-dependent component of the statistics.
  * [30] See supplemental material at URL_will_be_inserted_by_publisher for evaluation of the elastic correction to the momentum correlation, and for an alternative derivation of the expression for the relative stretch.
  * Note [5] Note added after publication: Eqs. (15-17) and Eq. (29) in the present Letter coincide with Eqs. (14-15) and Eq. (36) in Ref. [18]; the answer to the first question is therefore yes; the answer to the second question is again yes in the case of the model of Ref. [18].

SUPPLEMENTAL MATERIAL

A: Finite elasticity correction to the momentum fluctuation amplitude

An equation for the stationary momentum correlation in the bulk, valid for
generic $\alpha$, can be obtained from Eqs. (2) and (11):

$\displaystyle\partial_{k}^{2}\Pi_{kk}+(\alpha/\Gamma)(Z_{k+1,k}+Z_{k-1,k})=0.$
(39)

By exploiting Eq. (34), Eq. (39) takes the form, valid for small $r$ and large
$N$,

$\displaystyle\partial_{k}^{2}\Pi_{kk}\simeq\frac{4\alpha\Delta
T(N/2-k)}{\Gamma^{2}N^{2}}.$ (40)

Equation (40) has the general solution

$\displaystyle\Pi_{kk}\simeq\frac{8\alpha\Delta
T}{3\Gamma^{2}N^{2}}(N/2-k)^{3}+c_{0}+c_{1}k.$ (41)

The homogeneous part of the solution is fixed by the boundary conditions

$\displaystyle\Pi_{11}=mT_{L},\qquad\Pi_{NN}=mT_{R}.$ (42)

From Eq. (22) one obtains then the expression for the elastic correction

$\displaystyle\tilde{\Pi}_{kk}=\Pi_{kk}-\bar{\Pi}_{kk}\simeq\frac{8\alpha\Delta
T}{3\Gamma^{2}N^{2}}[(N/2-k)^{3}-N^{2}(N/2-k)/4].$ (43)

and hence the estimate [see Eq. (36)]

$\displaystyle\frac{\tilde{\Pi}_{kk}}{m\Delta T}\sim r^{2}N\sim\frac{\delta
J^{el}_{k}}{J}.$ (44)

B: Stretch fluctuation amplitude — alternative calculation.

The stretch fluctuation can be estimated as $\delta L\sim\langle
q_{j}^{2}\rangle^{1/2}$, where $\langle q_{j}^{2}\rangle\equiv Q_{jj}$. By
switching to Fourier space in both space and time, we easily obtain from Eqs.
(6) and (7) the fluctuation spectrum

$\displaystyle Q_{k\omega}=\frac{2\nu\epsilon
k^{2}}{\rho[(\omega^{2}-c_{s}^{2}k^{2})^{2}+\nu^{2}k^{4}]}=\frac{\nu
k_{B}Tk^{2}}{\rho[(\omega^{2}-c_{s}^{2}k^{2})^{2}+\nu^{2}k^{4}]},$ (45)

where $\rho=m/\eta$ is the chain density and $k$ indicates here wavenumbers.
The denominator of $Q_{k\omega}$ has roots

$\displaystyle kl_{\nu}\gg 1:\qquad\omega_{1}\simeq\pm{\rm i}\nu
k^{2},\quad\omega_{2}=\pm{\rm i}c_{s}^{2}/\nu,$ $\displaystyle kl_{\nu}\ll
1:\qquad\omega_{1}\simeq c_{s}k\pm{\rm i}\nu
k^{2},\quad\omega_{2}=-c_{s}k\pm{\rm i}\nu k^{2};$

we thus get for $kl_{\nu}\gg 1$

$\displaystyle Q_{k}=\int\frac{{\rm d}\omega}{2\pi}\frac{\nu
k_{B}Tk^{2}}{\rho(\omega^{2}+\nu^{2}k^{4})(\omega^{2}+c_{s}^{4}/\nu^{2})}=\frac{\nu
k_{B}T}{2\rho c_{s}^{2}(c_{s}^{2}/\nu+\nu k^{2})}\simeq\frac{k_{B}T}{2\rho
c_{s}^{2}k^{2}},$ (46)

while for $kl_{\nu}\ll 1$ we have

$\displaystyle Q_{k}$ $\displaystyle=$ $\displaystyle\int\frac{{\rm
d}\omega}{2\pi}\frac{\nu k_{B}Tk^{2}}{\rho[(\omega-
c_{s}k)^{2}+\nu^{2}k^{4}][(\omega+c_{s}k)^{2}+\nu^{2}k^{4}]}$ (47)
$\displaystyle=$
$\displaystyle\frac{k_{B}T}{4\rho\nu(\nu^{2}k^{4}+c_{s}^{2}k^{2})}\simeq\frac{k_{B}T}{4\rho
c_{s}^{2}k^{2}}.$

From here we get the displacement fluctuation amplitude in a chain of length
$L$

$\displaystyle\langle q^{2}\rangle\sim\int_{kL>1}{\rm
d}kQ_{k}\sim\frac{k_{B}TL}{\rho c_{s}^{2}}\sim(v_{th}/c_{s})^{2}\eta L,$ (48)

and hence the estimate for the stretch, valid for both $l_{\nu}\ll L$ and
$l_{\nu}\gg L$,

$\displaystyle\delta L\sim\langle q^{2}\rangle^{1/2}\sim\frac{v_{th}\sqrt{\eta
L}}{c_{s}}=\frac{v_{th}N^{1/2}\eta}{c_{s}}$ (49)

[compare with Eq. (38)].