# Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning

Chaojie Wang111footnotemark: 1 Yanchen Deng211footnotemark: 1 Zhiyi Lv2 Liang
Zeng1 Jujie He1

Shuicheng Yan1 Bo An12

1Skywork AI 2Nanyang Technological University
The first two authors contributed equally, and the order of authors was
determined by a coin toss.

###### Abstract

Large Language Models (LLMs) have demonstrated impressive capability in many
natural language tasks. However, the auto-regressive generation process makes
LLMs prone to produce errors, hallucinations and inconsistent statements when
performing multi-step reasoning. In this paper, by casting multi-step
reasoning of LLMs as a heuristic search problem, we aim to alleviate the
pathology by introducing Q*, a general, versatile and agile framework for
guiding LLMs decoding process with deliberative planning. By learning a plug-
and-play Q-value model as heuristic function for estimating expected future
rewards, our Q* can effectively guide LLMs to select the most promising next
reasoning step without fine-tuning LLMs for the current task, which avoids the
significant computational overhead and potential risk of performance
degeneration on other tasks. Extensive experiments on GSM8K, MATH and MBPP
demonstrate the superiority of our method, contributing to improving the
reasoning performance of existing open-source LLMs.

##  1 Introduction

Large Language Models (LLMs) have exhibited impressive capabilities in solving
various reasoning tasks encoded in natural languages, including math word
problems [1, 2, 3, 4, 5, 6], code generation [7, 8, 9] and planning [10, 11,
12]. Unfortunately, even the most advanced LLMs still face significant
challenges and are prone to introduce errors, hallucinations and inconsistent
statements as the number of reasoning steps grows due to their auto-regressive
nature [13, 14]. In fact, the auto-regressive generation process of LLMs can
be characterized by “System 1" [15], a mode of thought which is fast,
instinctive but less accurate. Most of recent works focus on improving LLMs’
“System 1” capability by (1) constructing sophisticated prompts with extensive
expertise to trigger the potential capacities of LLMs without modifying their
parameters [16, 17, 18, 19], (2) fine-tuning LLMs with massive task-specific
corpus at the price of significant computational burdens and the potential
risk of performance degeneration on other tasks [5, 6, 20, 21], or (3)
training reward models to rank the candidate responses [22, 23, 4, 24].

On the other hand, solving complex reasoning problems requires more in-depth,
deliberative and logical thinking steps, _i.e._ , the “System 2" mode [15].
Taking solving math word problems as an example, any incorrect intermediate
reasoning step (_e.g._ , calculation errors, mis-interpretations) can
potentially lead to incorrect final answers. Prior attempts [25, 26, 27, 28]
for enhancing “System 2” reasoning capability includes performing deliberation
with basic tree search algorithms (_e.g._ , BFS or DFS), Monte Carlo Tree
Search (MCTS) [29], and A* [30]. Nonetheless, the utility functions used in
these methods often require laborious expertise to design for each specific
task, which are difficult to be extended to new scenarios. Furthermore,
deliberation with MCTS would require significant number of rollouts before
finding high-quality responses when solving the problems with many reasoning
steps, which substantially slows down the overall decoding process.

In light of this, we propose Q*, a general, versatile and agile framework for
improving the multi-step reasoning capability of LLMs with deliberative
planning. Different from the existing deliberation methods, our method does
not rely on domain knowledge to design the heuristic function. Besides, by
leveraging plug-and-play Q-value models as heuristic function, our Q* can
effectively solve various tasks via guiding LLMs to select the most promising
next step without fine-tuning LLMs beforehand, which avoids the significant
computational overhead and potential risk of performance degeneration in other
tasks. Finally, Q* considers only one single step when performing
deliberation, which is much cheaper than completing rollouts in MCTS.
Specifically, the main contributions of our work are summarized as follows:

  * •

We formalize the multi-step reasoning of LLMs as a Markov Decision Process
(MDP) where the state is the concatenation of input prompt and the reasoning
steps generated so far, the action is the next reasoning step and the reward
measures how well the task is solved.

  * •

We present several general approaches to estimate the optimal Q-value of
state-action pairs, _i.e._ , offline reinforcement learning, the best sequence
from rollouts, and completion with stronger LLMs. It is noteworthy that our
methods only need the ground-truth of training problems and can be easily
applied to various reasoning tasks without modification.

  * •

We cast solving multi-step reasoning tasks as a heuristic search problem,
where the objective is to find the most proper reasoning trace with maximum
utility. Built upon A* search, our deliberation framework, Q*, leverages plug-
and-play Q-value models as heuristic function and guides LLMs to select the
most promising next reasoning step in best-first fashion.

  * •

We conduct extensive experiments on math reasoning and code generation tasks,
demonstrating that Q* can significantly improve the multi-step reasoning
capability of existing open-sourced LLMs.

##  2 Related Works

#### LLM alignment.

Alignment has become an important technique to prevent the output of LLMs
deviates from human’s expectation. Supervised Fine-Tuning (SFT) is probably
the most fundamental alignment approach that directly minimizes the cross-
entropy loss between the output and ground-truth. Reinforcement learning from
Human Feedback (RLHF) [31], on the other hand, firstly learns a reward model
(RM) from human preferences and then optimizes the SFT model with
reinforcement learning algorithms to maximize the cumulative rewards from RM.
Direct Preference Optimization (DPO) [32] aligns LLMs directly according to
the ranking information from human feedback without explicitly learning RM.
Recently, Aligner [33] came out as a model-agnostic alignment method by
learning to re-write LLMs’ output. Compared to these methods, our Q* achieves
the goal of alignment with distinct merits. Different from SFT and Aligner, Q*
does not rely on massive human annotated preference pairs which are expensive
to collect; different from RLHF and DPO, our Q* does not modify the parameters
of LLMs, which avoids the potential risk of performance degeneration on other
tasks.

#### Enhancing LLMs with planning.

Tree-of-thoughts (ToT) [25] improves the LLMs’ reasoning capability by
exploring the intermediate steps towards problem solving with basic tree-
search algorithms. In the same vein, A* search and MCTS are applied to serve
as a planning technique to enhance the performance of LLMs when solving
challenging complex reasoning problems [26, 27, 28, 34]. Unfortunately, the
utility function used in these methods is either constructed from LLMs’
feedback (_e.g._ , [25, 27]), which could be highly-inaccurate in complex
problems, or specific to each individual task (_e.g._ , [28, 34]). Moreover,
planning with MCTS often requires to perform costly rollout, which can
significantly slow down the overall decoding process. In contrast, our Q*
solely relies on training a Q-value model to guide LLMs to select the most
promising next reasoning step and the pipeline can be easily applied to
various reasoning tasks without modification. Besides, we consider only a
single step each time in Q*, which is much cheaper than complete rollout in
MCTS-based methods.

#### LLMs for math reasoning & code generation.

Math reasoning and code generation require LLMs to perform multi-step
reasoning on relations, quantities and logics which are inherently
challenging. Current techniques include: 1) prompt engineering which triggers
the potential capacities of LLMs with sophisticated prompts [16, 17, 18, 19,
35, 36]. However, constructing such prompt needs extensive expertise and case-
by-case tuning, which is difficult to generalize to different tasks; 2) Fine-
tuning LLMs with massive math/code corpus [5, 6, 20, 21, 8, 9, 37], which
usually comes at the price of significant computational burden and may
compromise the performance on other tasks; 3) training RMs/verifiers to rank
the candidate solutions without providing any guidance in intermediate steps
[22, 23, 4, 24]. Differently, our Q* leverages a plug-and-play Q-value model
to direct the deliberation process of LLMs, which effectively provides
guidance for each intermediate step without modifying the parameters of LLMs.
Moreover, by casting multi-step reasoning of LLMs as a heuristic search
problem, our method can be generalized to various reasoning tasks without
laborious prompt engineering.

##  3 Preliminary

###  3.1 Formulate the Multi-step Reasoning of LLMs as an MDP

Taking the question $q$ as input, the answer generation process of LLMs can be
broken down into multiple reasoning steps, where the final answer sequence
$\mathbf{a}$ can be treated as the concatenation of these $T$ single-step
reasoning steps, formulated as $\mathbf{a}=[a_{1};a_{2};\dots;a_{T}]$. Each
step can be a single line or fixed number of tokens outputted by LLMs. Under
this perspective, we can conceptualize the multi-step reasoning process of
LLMs as a Markov Decision Process (MDP)
$\langle\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\gamma\rangle$, where
the state $s_{t}\in\mathcal{S}$ denotes the concatenation of the input
question and the partial reasoning trace already generated by timestep $t-1$
(_i.e._ , $s_{t}=[q;a_{1};\dots;a_{t-1}]$) with the special case $s_{1}=q$,
the action $a_{t}\in\mathcal{A}$ denotes the next reasoning step generated by
LLMs taking the current state $s_{t}$ as input, the deterministic state
transition $\mathcal{T}$ from the current state $s_{t}$ to the next state
$s_{t+1}$ is accomplished through a simple operation of concatenation,
$\mathcal{R}$ is the reward function to measure how well the question is
solved and $\gamma$ is the discount factor. The reward function is often
outcome-based. That is, it gives reward by comparing the final results with
ground-truth:

$\mathcal{R}(s_{t},a_{t})=\begin{cases}1&t=T\land[s_{t};a_{t}]\;\text{matches
the ground-truth}\\\ 0&\text{otherwise}\end{cases},$ (1)

In particular, we assign a reward of 1 if the generated code passes all test
cases (for code generation tasks) or the final answer matches the ground-truth
(for math reasoning tasks) which is a common practise in previous studies [4,
22]. Finally, the policy $\pi_{\theta}$ is embodied by an LLM, which produces
reasoning sequence conditioned on the input question:

$\displaystyle\pi_{\theta}(a_{t}|s_{t})=\text{LLM}(a_{t}|s_{t}),~{}\pi_{\theta}(\mathbf{a}|q)=\prod_{t=1}^{T}\pi_{\theta}(a_{t}|s_{t}).$
(2)

Given the MDP and LLM policy $\pi_{\theta}$, the value of state-action pair
$(s_{t},a_{t})$ is given by a Q-function
$Q^{\pi_{\theta}}(s_{t},a_{t})=\mathbb{E}_{\pi_{\theta}}\left[\sum_{t^{\prime}=t}^{T}\gamma^{T-t^{\prime}}\mathcal{R}(s_{t^{\prime}},a_{t^{\prime}})\right]$.
The Q-function of an optimal policy $\pi^{*}$ is called optimal Q-function and
satisfies the Bellman optimality equation:

$Q^{*}(s_{t},a_{t})=\mathcal{R}(s_{t},a_{t})+\gamma\max_{a_{t+1}\in\mathcal{A}}Q^{*}(s_{t+1},a_{t+1}),$
(3)

which gives the value of starting state $s_{t}$, taking action $a_{t}$ and
then following the optimal policy $\pi^{*}$.

###  3.2 A* Search

A* [30] is an important heuristic search algorithm in deliberative planning
[38], multi-agent pathfinding [39], and constraint reasoning [40]. Originally,
A* is proposed for finding the shortest path from source $s$ to goal $g$ in
path planning problems. It associates each frontier vertex $n$ with a value
$f(n)=g(n)+h(n)$, where $g(n)$ is the accumulated path cost from source $s$
and $h(n)$ is a heuristic value that estimates the cost of the shortest path
from $n$ to goal $g$. The algorithm adopts a best-first search strategy,
_i.e._ , in each iteration it always picks the vertex with minimum $f$-value
to explore until reaching the goal. When the heuristic $h(\cdot)$ is
admissible [41], A* guarantees to find the optimal path.

Figure 1: Overview of Q*. (a): the deliberation process of Q*. Each state is
associated with an $f$-value which is the weighted sum of the aggregated
utility (cf. Eq. (5)) and the heuristic value (cf. Eq. (6)). (b-d): estimating
optimal Q-value with fitted-Q-iteration, rollout and completion with stronger
LLMs.

##  4 Q*: A General, Versatile and Agile Deliberation Framework for LLMs

Most of modern LLMs generate natural languages in an auto-regressive way,
_i.e._ , predict the next token in a sequence given the previously generated
tokens (cf. Eq. (2)). Therefore, when applied to multi-step reasoning problem,
LLMs can potentially introduce errors, hallucinations and inconsistent
statements in the subsequent reasoning trace if any previous step is
incorrect, which may fail to solve the current problem. Indeed, given the fact
that LLMs produce each token with limited computation resources, there is no
way to devote more computational efforts to solve difficult problems. In
short, LLMs cannot perform in-depth deliberation which is essential for
solving complex multi-step reasoning problems.

We address this issue by presenting Q*, a general, versatile and agile
deliberation framework based on A* to effectively guide LLMs to select the
most promising next step when performing multi-step reasoning without costly
fine-tuning LLMs for each task beforehand. In more detail, we cast finding the
most proper reasoning sequence for a given problem as a heuristic search
process, where each state $s_{t}$ is associated with a $f$-value estimating
how much utility will be attained if we expand $s_{t}$:

$\displaystyle f(s_{t})=g(s_{t})+\lambda h(s_{t}),$ (4)

where $g(s_{t})$ denotes the aggregated utility from the initial state
$s_{1}$; $h(s_{t})$ is the heuristic value for measuring the probability of
reaching the correct answer derived from $s_{t}$; $\lambda$ is a coefficient
to balance the importance of $g(s_{t})$ and $h(s_{t})$ terms.

Specifically, we propose to use process-based reward function
$\mathcal{R}_{P}$ that encodes the prior knowledge or preference of the
reasoning task to compute the aggregated utility $g(s_{t})$. That is,

$g(s_{t})=\text{Agg}(\mathcal{R}_{P}(s_{1}),\dots,\mathcal{R}_{P}(s_{i}),\dots,\mathcal{R}_{P}(s_{t})),$
(5)

where Agg $\in\\{\min,\max,\sum,[-1]\\}$, with $[-1]$ standing for assigning
the reward of last state as the utility, is the aggregation function to
summarize the rewards in the path from $s_{1}$ to $s_{t}$, and $s_{i-1}$ is
the prefix of $s_{i},\;1<i\leq t$. Such process-based reward function
$\mathcal{R}_{P}$ could be learned from human feedback [22, 23, 42], ground-
truth [4, 24], rules, or simply be the logits of a reasoning step which
reflects the confidence of the LLM. Furthermore, we use the optimal Q-value of
state $s_{t}$ (cf. Eq. (3)) as the heuristic value $h(s_{t})$. In other words,
the $f$-value is given by:

$\displaystyle
f(s_{t})=g(s_{t})+\lambda\max_{a_{t}\in\mathcal{A}}Q^{*}(s_{t},a_{t}).$ (6)

Since enumerating all possible next reasoning steps is intractable, in
practice we restrict the alternatives to the top-K of all step candidates
returned by LLM, and thus Eq. (6) can be written as
$f(s_{t})=g(s_{t})+\lambda\max_{a_{t}\in\text{top-K}(\pi_{\theta}(\cdot|s_{t}))}Q^{*}(s_{t},a_{t})$.

###  4.1 Estimation of Optimal Q-value

A critical challenge of implementing Q* is to estimate the optimal Q-value of
state-action pairs (cf. Eq. (6)) with a frozen LLM policy $\pi_{\theta}$ which
could be suboptimal on the given reasoning problems. Therefore, we aim to
learn a proxy Q-value model $\hat{Q}$ to approximate $Q^{*}$ from a dataset
$D=\\{q_{i},\\{\mathbf{a}_{i_{j}}\\}_{j=1}^{M}\\}_{i=1}^{N}$, where $q_{i}$ is
a training problem and $\mathbf{a}_{i_{j}}\sim\pi_{\theta}(\cdot|q_{i})$ is a
trajectory sampled from the LLM policy $\pi_{\theta}$. Formally:

$\hat{Q}=\arg\min_{Q}\frac{1}{NMT}\sum_{i=1}^{N}\sum_{j=1}^{M}\sum_{a_{t}\in\mathbf{a}_{i_{j}}}\left(Q(s_{t},a_{t})-\hat{y}(s_{t},a_{t})\right)^{2},$
(7)

where $s_{t}=[q_{i};a_{1};\dots;a_{t-1}]$ is the partial reasoning trace by
timestep $t-1$ in $\mathbf{a}_{i_{j}}$ and $\hat{y}(s_{t},a_{t})$ is the label
that approximates the true optimal Q-value, specifically $Q^{*}(s_{t},a_{t})$.

In more detail, we effectively construct Q-value labels $\hat{y}(s_{t},a_{t})$
for question $q_{i}$ in the following ways:

#### Offline reinforcement learning.

Given the offline dataset $D$, we learn the proxy Q-value model $\hat{Q}$
using Fitted Q-iteration [43]. Specifically, for each iteration $\ell$, we
construct Q-value label as:

$\hat{y}_{\ell}(s_{t},a_{t})=\begin{cases}\mathcal{R}(s_{t},a_{t})&t=T\\\
\mathcal{R}(s_{t},a_{t})+\gamma\max_{a_{t+1}\in\text{top-K}(\pi_{\theta}(\cdot|s_{t+1}))}\hat{Q}_{\ell-1}(s_{t+1},a_{t+1})&\text{otherwise}\end{cases},$
(8)

where $\hat{Q}_{\ell-1}$ is the proxy Q-value model learned in iteration
$\ell-1$. After that, we train a new proxy model $\hat{Q}_{\ell}$ according to
Eq. (7). Such two phases will be alternated for $L$ iterations, and we use
$\hat{Q}_{L}$ as the proxy Q-value model when performing deliberation.

#### Learning from rollout.

Staring from the state-action pair $(s_{t},a_{t})$, we first perform random
rollout or MCTS [27, 26] with policy $\pi_{\theta}$ for enough iterations,
forming a trajectory pool $P$. After that, we use the best reasoning sequence
with the highest accumulated rewards to construct the Q-value label:

$\hat{y}(s_{t},a_{t})=\mathcal{R}(s_{t},a_{t})+\max_{\begin{subarray}{c}\tau\sim
P\\\
(s_{t^{\prime}},a_{t^{\prime}})\in\tau\end{subarray}}\left[\sum_{t^{\prime}=t+1}^{T}\gamma^{T-t^{\prime}}\mathcal{R}(s_{t^{\prime}},a_{t^{\prime}})\right].$
(9)

#### Approximating optimal policy with stronger LLMs.

By exploiting the fact that the optimal Q-values are derived from an optimal
policy, one can estimate the optimal Q-value of state-action pair
$(s_{t},a_{t})$ by completing the trajectory with another stronger LLM
$\pi_{\theta^{*}}$ (_e.g._ , GPT-4) if available:

$\hat{y}(s_{t},a_{t})=\mathcal{R}(s_{t},a_{t})+\sum_{t^{\prime}=t+1}^{T}\gamma^{T-t^{\prime}}\mathcal{R}(s^{*}_{t^{\prime}},a^{*}_{t^{\prime}}),$
(10)

where
$s^{*}_{t^{\prime}}=[s_{t};a_{t};a^{*}_{t+1};\dots;a^{*}_{t^{\prime}-1}]$ and
$a^{*}_{t^{\prime}}\sim\pi_{\theta^{*}}(\cdot|s^{*}_{t^{\prime}})$.

###  4.2 Deliberative Planning with A*

Once obtaining the proxy Q-value model $\hat{Q}$, we can plug it to Eq. (6) to
compute the $f$-value of each state and perform best-first search with A*.
Alg. 1 illustrates the deliberative planning process.

Algorithm 1 Deliberative planning for LLMs with A*

1:question $q$, LLM policy $\pi_{\theta}$, proxy Q-value model $\hat{Q}$

2:$unvisited\leftarrow\\{q\\}$, $visited\leftarrow\emptyset$

3:while $unvisited\neq\emptyset$ do

4: $s\leftarrow\arg\max_{s^{\prime}\in unvisited}f(s^{\prime})$

5: $unvisited\leftarrow unvisited\backslash\\{s\\}$, $visited\leftarrow
visited\cup\\{s\\}$

6: if $s$ is a terminal state then return the complete trajectory $s$

7: for each $a\in\text{top-K}(\pi_{\theta}(\cdot|s))$ do

8: $s^{\prime}\leftarrow[s;a]$

9: if $s^{\prime}\notin visited$ then $unvisited\leftarrow
unvisited\cup\\{s^{\prime}\\}$

Specifically, we maintain a set for storing state candidates to be explored,
denoted as $unvisited$, which initially only contains the input question $q$,
and another set $visited$ to record the visited states. Each step we pick the
state $s$ with the maximum $f$-value from the set $unvisited$ and expand it by
querying the top-K alternatives with the LLM policy $\pi_{\theta}$. After
that, both $visited$ and $unvisited$ sets will be updated and this process
repeats until the terminal state (a complete trajectory) is reached. Finally,
we extract the answer part of the terminal state $s$ as the result.

##  5 Experiments

###  5.1 Experimental Settings

Datasets. We evaluate the effectiveness of Q* on two math reasoning and one
code generation tasks, where the dataset statistics have been summarized in
Table 1. 1) GSM8K [2] is a dataset of grade school math problems, where the
solution is given in a one-line-per-step format with an exact numerical answer
in the last line; 2) MATH [3] is a dataset consisting of math problems of high
school math competitions, where the solutions are given in a format that mixes
latex code and natural language; 3) MBPP [44] is an entry-level Python
programming dataset, where the questions are coding challenges along with a
test case that defines the function format. The solutions are Python code that
is excepted to pass the pre-collected test cases of each question.

Table 1: Statistics of datasets.  Dataset | GSM8K | MATH | MBPP
---|---|---|---
Domain | Math Reasoning | Math Reasoning | Code Generation
Training | 5000 | 8000 | 374
Testing | 1319 | 5000 | 500
Average Steps | 4.5 | 11.0 | 7.0

Implementation Details. The implementation of Q* method mainly includes three
steps: 1) Q-value estimation. As discussed in Section 4.1, we propose several
ways for estimating Q values and will illustrate the implementation details in
the next paragraph; 2) Utility aggregation for each step (cf. Eq. (5)). For
GSM8K dataset, we adopt a process reward model (PRM) trained on PRM800K [22]
to model $\mathcal{R}_{P}$ to provide an intermediate signal for each
reasoning step, and use $\min$ as the aggregation function; For MATH dataset,
we set $g(s_{t})=0$ for all passed states $\\{s_{i}\\}_{i=1}^{t}$ in each
trajectory for fairness, because PRM800K contains data samples constructed
from MATH testing set and there is a potential risk of data leakage; For MBPP
dataset, we tokenize the code generated so far with function
`tokenize.generate_tokens` and give a penalty of -0.5 if `TokenError` is
raised, which is often the case that there are mismatched delimiters (e.g.,
parentheses, quotation marks) and invalid indention in the code. We use $[-1]$
as the aggregation function to cancel the previous penalties since the code is
generated on-the-fly and mismatched delimiters may be fixed in subsequent
steps. 3) A* planning. For GSM8K and MATH datasets, we treat a single line
outputted by the LLM as an action, while the action in MBPP is defined as a
code snippet with 24 tokens when planning. Besides, in all experiments, we set
$\lambda=1$ when computing $f$-values and expand a state with $K=6$ actions at
each reasoning step. Finally, following the common practice of Best-of-$N$, we
perform planning to collect $N=6$ trajectories for each question, and select
the one with the maximum $f$-value as the final result for evaluation.

For Q-value estimation, in our practice, we find that learning from rollout
could be the most effective and robust way to collect precise Q-value labels.
Specifically, given the prompt of questions, we will firstly perform random
rollout to obtain complete trajectories with LLM, denoted as $\pi_{\theta}$,
under the setting of high temperature, _e.g._ , $\tau=0.9$ for math reasoning
and $\tau=0.2$ for code generation, and split each trajectory into a series of
states according to the newline token “$\textbackslash n$”. Then, for each
state-action pair in a trajectory, denoted as $(s_{t},a_{t})$, we can perform
random rollout or MCTS with the same LLM to collect a pool $P$ of
trajectories, and then select the best reasoning path with the highest
accumulated rewards to construct the corresponding Q-value label of the
current state-action pair. Note that the reward $\mathcal{R}(s_{t},a_{t})$ is
given as 1 only if the obtained math numerical answer is correct or the
program passes all test cases, indicating that the Q value of a state-action
pair can be 1 only if it has the potential to generate a trajectory containing
the correct answer.

###  5.2 Experimental Results

GSM8K. For the comparison on GSM8K dataset, we select Llama-2-7b [45] as our
base model, whose accuracy can achieve 65.2% after finetuning on MetaMath [5].
Then, we treat Llama-2-7b finetuned on MetaMath as policy $\pi_{\theta}$, and
perform random rollout to collect Q-value labels for training Q-value model
(QVM). For utility aggregation, we train a process reward model (PRM) on
PRM800K [22] to provide intermediate signal for each reasoning step. With PRM
and QVM in hand, traditional methods tend to treat either of them as a
verifier to select the Best-of-$N$ trajectory or utilize them to perform PPO
training of RLHF. As the results shown in Table 2, we can find that with the
same PRM/QVM, using it for verification performs significantly better than
using it for alignment. Further, in the comparison of planning-based methods,
we can find that with the same QVM, Q* method with constant aggregated utility
can still outperform Best-of-$N$ method. With the PRM trained on PRM800K
determining whether the intermediate reasoning steps are correct, Q* method
that combines PRM and QVM achieves the best performance among all methods
based on the same LLM, helping Llama-2-7b surpass the performance of close-
sourced ChatGPT-turbo [46] and reaching an accuracy of 80.8%.

MATH. As the results shown in Table 3, considering the weak performance of
Llama-2-7b fine-tuned with MetaMath for the MATH dataset, we seek for two
other stronger LLMs to evaluate the effectiveness of our Q* method. One is
Llama-2-7b fine-tuned on Synthetic Data [47], which is constructed following
the instruction of scaling up the SFT data, and achieves 41.9% accuracy on
MATH dataset, approaching the performance of GPT-4 [48]. The other base model
is DeepSeek-Math-7b [49], which could be the most powerful open-source 7b
model for math reasoning on MATH dataset, achieving 50.8% accuracy in our
evaluation. From the results shown in the second and third blocks of Table 3,
we can find that Q* can still lead to further performance improvement compared
to the Best-of-$N$ method on either of base models. Additionally, it is
noteworthy that the performance of DeepSeek-Math-7b enhanced with Q* has
already surpassed a series of closed-source models on the leaderboard of MATH
dataset 111https://paperswithcode.com/sota/math-word-problem-solving-on-math,
such as Gemini Ultra (4-shot), reaching an accuracy of 55.4% .

Table 2: Comparison of Q* and other baselines on GSM8K dataset.

Base Model | SFT | Alignment | Verification | Accuracy
---|---|---|---|---
GPT-3.5 (5-shot) [50] |  Unknown |  PPO (RM) [31] |  - |  57.1%
ChatGPT-instruct (0-shot) [46] |  Unknown |  PPO (RM) [31] |  - |  71.3%
ChatGPT-turbo (0-shot) [46] |  Unknown |  PPO (RM) [31] |  - |  77.7%
GPT-4 (0-shot) [46] |  Unknown |  PPO (RM) [31] |  - |  91.9%
GPT-4 (5-shot) [50] |  Unknown |  PPO (RM) [31] |  - |  92.0%
Llama-2-7b (0-shot) |  - |  - |  - |  49.5%
Llama-2-7b (0-shot) |  MetaMath[5] |  - |  - |  65.2%
Llama-2-7b (0-shot) |  MetaMath[5] |  PPO (PRM) [31] |  - |  67.2%
Llama-2-7b (0-shot) |  MetaMath[5] |  PPO (QVM) [31] |  - |  67.6%
Llama-2-7b (0-shot) |  MetaMath[5] |  - |  Best-of-$N$ (PRM) [22] |  72.1%
Llama-2-7b (0-shot) |  MetaMath[5] |  - |  Best-of-$N$ (QVM) [22] |  74.5%
Llama-2-7b (0-shot) |  MetaMath[5] |  - |  Q* (QVM) |  78.8%
Llama-2-7b (0-shot) |  MetaMath[5] |  - |  Q* (PRM+QVM) |  80.8%

Table 3: Comparison of Q* and other baselines on MATH dataset.

Base Model | SFT | Alignment | Verification | Accuracy
---|---|---|---|---
GPT-3.5 (0-shot) [48] |  Unknown |  PPO (RM) [31] |  - |  23.5%
GPT-4 (0-shot) [48] |  Unknown |  PPO (RM) [31] |  - |  42.5%
Gemini Ultra (4-shot) [51] |  Unknown |  PPO (RM) [31] |  - |  53.2%
Llama-2-7b (0-shot) |  - |  - |  - |  7.9%
Llama-2-7b (0-shot) |  MetaMath[5] |  - |  - |  20.0%
Llama-2-7b (0-shot) |  Synthetic Data[47] |  - |  - |  41.9%
Llama-2-7b (0-shot) |  Synthetic Data[47] |  PPO (QVM) [31] |  - |  42.5%
Llama-2-7b (0-shot) |  Synthetic Data[47] |  - |  Best-of-$N$ (QVM) [22] |  46.8%
Llama-2-7b (0-shot) |  Synthetic Data[47] |  - |  Q* (QVM) |  49.1%
DeepSeek-Math-7b (0-shot) |  Unknown |  PPO (QVM) [31] |  - |  50.8%
DeepSeek-Math-7b (0-shot) |  Unknown |  PPO (QVM) [31] |  Best-of-$N$ (QVM) [22] |  54.3%
DeepSeek-Math-7b (0-shot) |  Unknown |  PPO (QVM) [31] |  Q* (QVM) [22] |  55.4%

MBPP. As for the comparison on MBPP dataset, we also choose the most powerful
open-source LLM in the aspect of code generation, specifically
CodeQwen1.5-7b-Chat, as our base model for evaluating the effectiveness of Q*.
Following a similar procedure of math reasoning, we train a QVM for Q-value
estimation and manually construct the utility function as described in the
previous part of implementation details (tokenize the code generated so far
with function `tokenize.generate_tokens` and give a penalty of -0.5 if
`TokenError` is raised). From the results shown in Table 4, we can find that
Q* can still outperform Best-of-$N$ method in the aspect of code generation,
and help CodeQwen1.5-7b-Chat to achieve 77.0% accuracy on MBPP dataset, which
is also a promising performance in the leaderboard of MPBB
222https://paperswithcode.com/sota/code-generation-on-mbpp.

Table 4: Comparison of Q* and other baselines on MBPP dataset.

Base Model | SFT | Alignment | Verification | Accuracy
---|---|---|---|---
GPT-3.5 Turbo (self-debug) [52] |  Unknown |  PPO (RM) [31] |  - |  72.8%
GPT-4 (self-debug) [52] |  Unknown |  PPO (RM) [31] |  - |  80.2%
CodeQwen1.5-7b-Chat (0-shot) |  Unknown |  PPO (QVM) [31] |  - |  74.6%
CodeQwen1.5-7b-Chat (0-shot) |  Unknown |  PPO (QVM) [31] |  Best-of-$N$ (QVM) [22] |  75.0%
CodeQwen1.5-7b-Chat (0-shot) |  Unknown |  PPO (QVM) [31] |  Q* (PRM+QVM) [22] |  77.0%

##  6 Conclusion

Solving challenging multi-step reasoning problems requires LLMs to perform in-
depth deliberation beyond auto-regressive token generation. In this paper, we
present Q*, a general, versatile and agile deliberation framework for LLMs.
Unlike existing deliberation methods which need extensive expertise to design
a utility function for each specific task, Q* relies on ground-truth solely to
train value model and can be easily applied to various reasoning tasks without
modification. Moreover, by leveraging plug-and-play Q-value models as the
heuristic function, Q* can effectively guide LLMs to solve various tasks
without fine-tuning LLMs beforehand, which avoids potential performance
degeneration on other tasks. Finally, our Q* is agile because we consider only
a single step each time rather than complete rollouts (_e.g._ , simulation in
MCTS). Extensive empirical evaluations on math reasoning and code generation
tasks confirm the superiority of our method.

## References

  * Ahn et al. [2024] Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin.  Large language models for mathematical reasoning: Progresses and challenges.  _arXiv preprint arXiv:2402.00157_ , 2024.
  * Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the math dataset.  _arXiv preprint arXiv:2103.03874_ , 2021.
  * Wang et al. [2023] Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui.  Math-shepherd: A label-free step-by-step verifier for LLMs in mathematical reasoning.  _arXiv preprint arXiv:2312.08935_ , 2023.
  * Yu et al. [2023] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.  Metamath: Bootstrap your own mathematical questions for large language models.  _arXiv preprint arXiv:2309.12284_ , 2023.
  * Luo et al. [2023a] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang.  Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.  _arXiv preprint arXiv:2308.09583_ , 2023a.
  * Luo et al. [2023b] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang.  Wizardcoder: Empowering code large language models with evol-instruct.  _arXiv preprint arXiv:2306.08568_ , 2023b.
  * Roziere et al. [2023] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_ , 2023.
  * CodeGemma Team et al. [2024] CodeGemma Team, Ale Jakse Hartman, Andrea Hu, Christopher A. Choquette-Choo, Heri Zhao, Jane Fine, Jeffrey Hui, Jingyue Shen, et al.  Codegemma: Open code models based on gemma.  2024\.  URL https://goo.gle/codegemma.
  * Xie et al. [2024] Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su.  TravelPlanner: A benchmark for real-world planning with language agents.  _arXiv preprint arXiv:2402.01622_ , 2024.
  * Liu et al. [2023] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone.  LLM+P: Empowering large language models with optimal planning proficiency.  _arXiv preprint arXiv:2304.11477_ , 2023.
  * Guan et al. [2023] Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati.  Leveraging pre-trained large language models to construct and utilize world models for model-based task planning.  In _NeurIPS_ , pages 79081–79094, 2023.
  * Valmeekam et al. [2023] Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati.  On the planning abilities of large language models - A critical investigation.  In _NeurIPS_ , pages 75993–76005, 2023.
  * Stechly et al. [2024] Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati.  Chain of thoughtlessness: An analysis of CoT in planning.  _arXiv preprint arXiv:2405.04776_ , 2024.
  * Daniel [2011] Kahneman Daniel.  _Thinking, Fast and Slow_.  Macmillan, 2011.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  In _NeurIPS_ , pages 24824–24837, 2022.
  * Wang et al. [2022] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_ , 2022.
  * Fu et al. [2022] Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot.  Complexity-based prompting for multi-step reasoning.  In _ICLR_ , 2022.
  * Zhou et al. [2022] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al.  Least-to-most prompting enables complex reasoning in large language models.  _arXiv preprint arXiv:2205.10625_ , 2022.
  * Azerbayev et al. [2023] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck.  Llemma: An open language model for mathematics.  _arXiv preprint arXiv:2310.10631_ , 2023.
  * Yue et al. [2023] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.  Mammoth: Building math generalist models through hybrid instruction tuning.  _arXiv preprint arXiv:2309.05653_ , 2023.
  * Lightman et al. [2023] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.  Let’s verify step by step.  _arXiv preprint arXiv:2305.20050_ , 2023.
  * Uesato et al. [2022] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins.  Solving math word problems with process-and outcome-based feedback.  _arXiv preprint arXiv:2211.14275_ , 2022.
  * Khalifa et al. [2023] Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang.  Discriminator-guided multi-step reasoning with language models.  _arXiv preprint arXiv:2305.14934_ , 2023.
  * Yao et al. [2023] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan.  Tree of thoughts: Deliberate problem solving with large language models.  In _NeurIPS_ , 2023.
  * Feng et al. [2023] Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang.  AlphaZero-like tree-search can guide large language model decoding and training.  _arXiv preprint arXiv:2309.17179_ , 2023.
  * Hao et al. [2023] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu.  Reasoning with language model is planning with world model.  _arXiv preprint arXiv:2305.14992_ , 2023.
  * Zhuang et al. [2023] Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A Rossi, Somdeb Sarkhel, and Chao Zhang.  Toolchain*: Efficient action space navigation in large language models with A* search.  _arXiv preprint arXiv:2310.13227_ , 2023.
  * Browne et al. [2012] Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton.  A survey of monte carlo tree search methods.  _IEEE Transactions on Computational Intelligence and AI in Games_ , 4(1):1–43, 2012.
  * Hart et al. [1968] Peter E Hart, Nils J Nilsson, and Bertram Raphael.  A formal basis for the heuristic determination of minimum cost paths.  _IEEE Transactions on Systems Science and Cybernetics_ , 4(2):100–107, 1968.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  In _NeurIPS_ , pages 27730–27744, 2022.
  * Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_ , 2023.
  * Ji et al. [2024] Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang.  Aligner: Achieving efficient alignment through weak-to-strong correction.  _arXiv preprint arXiv:2402.02416_ , 2024.
  * Hazra et al. [2024] Rishi Hazra, Pedro Zuidberg Dos Martires, and Luc De Raedt.  SayCanPay: Heuristic planning with large language models using learnable domain knowledge.  In _AAAI_ , pages 20123–20133, 2024.
  * Huang et al. [2023] Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui.  Agentcoder: Multi-agent-based code generation with iterative testing and optimisation.  _arXiv preprint arXiv:2312.13010_ , 2023.
  * Shinn et al. [2023] Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.  Reflexion: Language agents with verbal reinforcement learning.  _arXiv preprint cs.AI/2303.11366_ , 2023.
  * Team [2024] Qwen Team.  Code with codeqwen1.5, April 2024.  URL https://qwenlm.github.io/blog/codeqwen1.5/.
  * Bonet and Geffner [2001] Blai Bonet and Héctor Geffner.  Planning as heuristic search.  _Artificial Intelligence_ , 129(1-2):5–33, 2001.
  * Silver [2005] David Silver.  Cooperative pathfinding.  In _AAAI-AIIDE_ , pages 117–122, 2005.
  * Pezeshki et al. [2022] Bobak Pezeshki, Radu Marinescu, Alexander Ihler, and Rina Dechter.  AND/OR branch-and-bound for computational protein design optimizing K.  In _UAI_ , pages 1602–1612, 2022.
  * Russell and Norvig [2016] Stuart J Russell and Peter Norvig.  _Artificial Intelligence: A Modern Approach_.  Pearson, 2016.
  * Wu et al. [2023] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi.  Fine-grained human feedback gives better rewards for language model training.  _arXiv preprint arXiv:2306.01693_ , 2023.
  * Riedmiller [2005] Martin Riedmiller.  Neural fitted Q iteration–first experiences with a data efficient neural reinforcement learning method.  In _ECML_ , pages 317–328, 2005.
  * Austin et al. [2021] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_ , 2021.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Shridhar et al. [2023] Kumar Shridhar, Koustuv Sinha, Andrew Cohen, Tianlu Wang, Ping Yu, Ram Pasunuru, Mrinmaya Sachan, Jason Weston, and Asli Celikyilmaz.  The art of LLM refinement: Ask, refine, and trust.  _arXiv preprint arXiv:2311.07961_ , 2023.
  * Li et al. [2024] Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng.  Common 7b language models already possess strong math capabilities.  _arXiv preprint arXiv:2403.04706_ , 2024.
  * Bubeck et al. [2023] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.  Sparks of artificial general intelligence: Early experiments with GPT-4.  _arXiv preprint arXiv:2303.12712_ , 2023.
  * Shao et al. [2024] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo.  Deepseekmath: Pushing the limits of mathematical reasoning in open language models.  _arXiv preprint arXiv:2402.03300_ , 2024.
  * Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  GPT-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Team et al. [2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: A family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Chen et al. [2023] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou.  Teaching large language models to self-debug.  _arXiv preprint arXiv:2304.05128_ , 2023.


# Interacting Random-field Dipole Defect Model for Heating in Semiconductor-
based Qubit Devices

Yujun Choi  Department of Physics, University of Wisconsin-Madison, Madison,
WI 53706, United States  Robert Joynt<EMAIL_ADDRESS>Department of Physics,
University of Wisconsin-Madison, Madison, WI 53706, United States

###### Abstract

Semiconductor qubit devices suffer from the drift of important device
parameters as they are operated. The most important example is a shift in
qubit operating frequencies. This effect appears to be directly related to the
heating of the system as gate operations are applied. We show that the main
features of this phenomenon can be explained by the two-level systems that can
also produce charge noise, if these systems are considered to form an
interacting random-field glass. The most striking feature of the theory is
that the frequency shift can be non-monotonic in temperature. The success of
the theory narrows considerably the possible models for the two-level systems.

##  I Introduction

There has been considerable progress in semiconductor quantum computing, with
significant strides in scaling up and in gate fidelities [1, 2]. The chief
difficulty is decoherence, with charge noise as the main culprit [3, 4, 5, 6,
7, 8, 9, 10, 11]. The precise nature of the two-level systems (TLS) that
produce the noise remains elusive. There have been extensive characterizations
of the noise spectrum [12, 13], and the spatial correlations in the noise [14,
15] in different devices. These observations can help in the elucidation of
the nature of the TLS, but the data are not currently sufficient to pin things
down precisely.

Another problem, at first sight quite separate, that interferes with qubit
operation is the pulse-induced resonance shift (PIRS). This is a shift in the
operating frequency of the qubits as a computation proceeds. This is highly
problematic, since continual recalibration of the system is not practical.
Quadrature control [16] and prepulsing [17] can mitigate but not eliminate
this issue. PIRS has recently received an intensive experimental study [18].

Here we propose that the source of PIRS is also a group of TLS, perhaps the
same group that gives charge noise. Hence detailed observations of PIRS may
provide additional insight into the microscopic origin of the noise. We
proceed in the time-honored fashion of proposing a phenomenological model that
explains the data, and then seeing what constraints the model places on the
underlying physics of the system.

Let us take a concrete situation in which the system is in a resting state at
low temperature $T$ for times $t<0$ and the operations, which involve
microwave pulses that feed energy into the system, begin at $t=0$ and end at
some later time $t_{f}$. PIRS is a time-dependent shift $\Delta f(t)$ with
$\Delta f(t=0)=0$ by definition. $\Delta f$ is a function of time that
ultimately reverts to the base state some time after the operations have
ceased. PIRS appears to be rather ubiquitous in semiconductor qubits, but
there is considerable variability in how it manifests itself. Early
observations found positive shifts ($\Delta f\geq 0)$ of order a few MHz [17].
The magnitude of the shift was an increasing function of the energy injected
by the pulses. It also depended on the details of the electron wavefunctions
in the dots, for example on the dot occupations. The MHz magnitude of the
shifts is fairly typical for quantum dot qubits. Importantly, $\Delta f$ can
also be negative [16, 19]. The decay time after $t_{f}$ varies in the dot
systems, with values from 0.5 ms [1] to 38 $\mu$s [19] having been observed.
PIRS also occurs in donor-based qubit systems [20], though $|\Delta f|$ is
much smaller, of order 10s of kHz. Effects of a similar magnitude are seen in
flip-flop qubits [21]. In this work we concentrate on experiments in dots, but
we expect the theory to apply more broadly.

Our interpretation is based on the fact that dot and donor systems share the
feature that the qubit operating frequency depends on the spatial position of
the qubit. The arbitrary sign of $\Delta f$ then suggests that a change in the
electric field on the qubit is the origin of PIRS. Experimentally, it now
appears to be clear that PIRS is essentially a thermal heating effect rather
than a mechanical effect [1, 18]. This is also supported by the characteristic
return to a base state, most naturally interpreted as a return to thermal
equilibrium. The most striking feature of the results is that the magnitude of
$\Delta f$ is typically not monotonic in temperature ($T$), instead rising to
a maximum at about 200-300 mK, then decreasing [18].

##  II Model

To explain these observations, we introduce a model based on the charged TLS
that are known to exist in these devices, and that in fact are also
responsible, at least in part, for the decoherence of the qubits. These
charged defects or traps are modeled as a collection of $N$ fluctuating
electric dipoles. The $j$th dipole fluctuates between states
$s_{j}\mathbf{p}_{j}$, where $s_{j}=\pm 1$ and $\mathbf{p}_{j}$ is a fixed
vector for each $j$. For simplicity we assume that the dipoles all have the
same magnitude: $|\mathbf{p}_{j}|=p_{0}$. This is reasonable if all the TLS
have the same physical origin. The dipoles can have a non-zero equilibrium
moment which is random in direction and they interact via the long-range
Coulomb interaction. We call this the Interacting Random-field Glass Model
(IRGM). Somewhat similar models have been introduced to understand charge
noise (rather than equilibrium fields) in dot systems [22] and also in the
context of superconducting qubit systems to explain fluctuations in the
relaxation time $T_{1}$ [23, 24].

The electric field $\langle\mathbf{F}_{q}\rangle$ on a qubit at the origin of
coordinates is

$\langle\mathbf{F}_{q}\rangle=\frac{1}{4\pi\varepsilon}\sum_{j=1}^{N}\langle
s_{j}\rangle\frac{3(\mathbf{p}_{j}\cdot\mathbf{r}_{j})\,\mathbf{r}_{j}-\mathbf{p}_{j}|\mathbf{r}_{j}|^{2}}{|\mathbf{r}_{j}|^{5}}\equiv\sum_{j=1}^{N}\langle
s_{j}\rangle\mathbf{F}_{j}.$ (1)

The angle brackets indicate a thermal average. In the devices in question, the
qubit operation frequency depends linearly on the electric field at its
position. The frequency is a quasi-equilibrium quantity, so
$\langle\mathbf{F}_{q}\rangle$ is the object of interest for our purposes. The
relation between field and frequency is platform-dependent. In the set-up of
Refs. [17, 16, 1], $\langle\mathbf{F}_{q}\rangle$ causes the displacement of
the spin qubit in a magnetic field gradient, while in the flip-flop qubit
architecture motion of the qubit caused by $\langle\mathbf{F}_{q}\rangle$
would change the hyperfine coupling or the g-factor [25]. In all cases the
displacement changes the qubit operating frequency. The qubit frequency is
$f(T)=f_{0}+\mathbf{c}_{q}\cdot\langle\mathbf{F}_{q}\rangle(T=0)+\Delta f(T)$,
where $f_{0}$ is the $T$-independent part from the applied magnetic field,
$\langle\mathbf{F}_{q}\rangle(T=0)\neq 0$ is a constant that comes from the
ground state configuration of the TLS, and $\Delta f(T)$ is the PIRS effect
and all the $T$ dependence of $f$ comes from it. Thus $\Delta
f(T)=\mathbf{c}_{q}\cdot[\langle\mathbf{F}_{q}\rangle(T)-\langle\mathbf{F}_{q}\rangle(T=0)]$
is the quantity of interest. $\mathbf{c}_{q}$ depends on the particular type
of qubit and the position of the qubit in the device. Its direction is
determined by the condition that the effective magnetic field produced by
$\langle\mathbf{F}_{q}\rangle$ should be parallel to the applied field.

The Hamiltonian of the TLS in our model contains a random-field term $H_{r}$
and an interaction term $H_{int}$:

$H=H_{r}+H_{int}=-p_{0}\sum_{j=1}^{N}s_{j}\mathbf{E}_{j}\cdot\hat{p}_{j}-\frac{p_{0}^{2}}{8\pi\varepsilon}\sum_{j\neq
k=1}^{N}s_{j}s_{k}V_{jk}.$

Here

$V_{jk}=\frac{3\hat{p}_{j}\cdot(\mathbf{r}_{j}-\mathbf{r}_{k})\,\hat{p}_{k}\cdot(\mathbf{r}_{j}-\mathbf{r}_{k})-(\hat{p}_{j}\cdot\hat{p}_{k})\,|\mathbf{r}_{j}-\mathbf{r}_{k}|^{2}}{|\mathbf{r}_{j}-\mathbf{r}_{k}|^{5}}.$

The random effective electric fields $\mathbf{E}_{j}$ if interpreted in a
double-well picture of the TLS are related to the energy asymmetry
(‘detuning’) of the two wells. However, the physical origin of the
$\mathbf{E}_{j}$ may not be the same in all cases. For example, they could be
actual external electric fields coming from the gate electrodes, strain
fields, asymmetric microscopic defects, etc. For our purposes they are
considered to be phenomenological parameters that must be fit, since they are
very difficult to estimate in the absence of a real microscopic model. We
expect $N$ to be a number in the range of perhaps 10 to 100 and to be sample-
dependent [12]. The dipoles may well be the same TLS that give rise to the
noise in the system, but here we are interested in their equilibrium behavior,
not their fluctuations. This assumes that measurement of PIRS takes place over
a time interval longer than the characteristic switching times of the TLS.
However, the intersection of the set of TLS that causes qubit dephasing and
the set that causes PIRS need not be complete.

##  III Non-monotonicity of PIRS

The $T$ dependence of $\langle\mathbf{F}_{q}\rangle$ in a non-interacting
model with $H_{int}=0$ is already interesting, so we discuss it in detail. In
this case the problem is exactly solvable, once the positions of the dipoles
are specified: $\langle
s_{j}\rangle=\textrm{sgn}(\hat{p}_{j}\cdot\mathbf{E}_{j})\tanh(p_{0}\,\hat{p}_{j}\cdot\mathbf{E}_{j}/k_{B}T)$.
$\langle s_{j}\rangle$ has a definite sign at $T=0$ but eventually $\langle
s_{j}\rangle\rightarrow 0$ as $T\rightarrow\infty$. We can identify a turn-off
temperature $T_{j}=p_{0}|\mathbf{E}_{j}|/k_{B}$ for each TLS. Substitution
into Eq. 1 and performing the sum gives the equilibrium electric field
$\langle\mathbf{F}_{q}\rangle(T)$ at the qubit.

To understand the qualitative $T$ dependence of $\langle\mathbf{F}_{q}\rangle$
in the IRGM we begin with $T=0$. We divide the TLS into two groups. In the set
$S^{+}$ we have the indices $j$ for which
$\langle\mathbf{F}_{q}\rangle(T=0)\cdot\mathbf{F}_{j}\langle
s_{j}\rangle(T=0)>0$ while in group $S^{-}$ we have the indices $j$ for which
$\langle\mathbf{F}_{q}\rangle(T=0)\cdot\mathbf{F}_{j}\langle
s_{j}\rangle(T=0)<0$. That is, the dipoles in $S^{+}$ are aligned with the
ground state resultant field, while those in $S^{-}$ are anti-aligned. The
electric field at the qubit is the result of a random walk of the vectors
$\langle s_{j}\rangle\,\mathbf{F}_{j}$ with a resultant vector

$\langle\mathbf{F}_{q}\rangle=\sum_{j\in S^{+}}\langle
s_{j}\rangle\mathbf{F}_{j}+\sum_{j\in S^{-}}\langle
s_{j}\rangle\mathbf{F}_{j}$ (2)

The vectors in group $S^{+}$ are in the direction of the final result of the
walk, while those in group $S^{-}$ are in the opposite direction. Due to the
randomness in the asymmetry, the various components of the walk turn off at
different temperatures, and there will be some average turn-off temperature
$T^{+}$ for group $S^{+}$ and a different average turn-off temperature $T^{-}$
for group $S^{-}$. We now increase $T$ from zero. If $T^{+}>T^{-}$ then the
total field strength $|\langle\mathbf{F}_{q}\rangle|$ will first increase and
eventually vanish when $T\gg T^{+}$. In this case we have a non-monotonic
$T$-dependence of the field strength $|\langle\mathbf{F}_{q}\rangle|$, a
rather surprising result. $T^{+}$ and $T^{-}$ are not expected to be very
different if the dipoles have the same physical structure, but even in this
case the relatively small value of $N$ implies that random fluctuations will
make $T^{+}\neq T^{-}$. If $T^{+}<T^{-}$ then the total field strength
$|\langle\mathbf{F}_{q}\rangle|$ will decrease as the dominant dipoles turn
off and eventually vanish when $T\gg T^{-}$. If there is a gross mismatch
between the $T^{+}$ and $T^{-}$ components of $|\langle\mathbf{F}_{q}\rangle|$
could even reverse sign, but overall we would expect a monotonic decrease. The
relative magnitudes of $|\sum_{j\in S^{+}}\langle s_{j}\rangle\mathbf{F}_{j}|$
and $|\sum_{j\in S^{-}}\langle s_{j}\rangle\mathbf{F}_{j}|$ are also
important. If $|\sum_{j\in S^{+}}\langle s_{j}\rangle\mathbf{F}_{j}|$
dominates, then there is little cancellation in the sum and the non-monotonic
behavior will be suppressed. If the fields from $S^{+}$ and $S^{-}$ are
comparable, then non-monotonicity is more likely.

Now we turn to the effects of interactions. Overall, dipolar interactions are
always antiferromagnetic. This favors depolarization - a smaller net moment
$|\langle\mathbf{P}\rangle|=|\sum_{j}\langle s_{j}\rangle\mathbf{p}_{j}|$. If
the TLS are located to one side of the qubit, then the correlation between
$|\langle\mathbf{P}\rangle|$ and $|\langle\mathbf{F}_{q}\rangle|$ will be
strong, but even if the qubit is surrounded symmetrically by the TLS,
fluctuations will still give some correlation in a given sample. Small
$|\langle\mathbf{P}\rangle|$ comes from cancellation in the directions of the
individual moments.

There is an additional temperature scale associated with the interactions,
which is the average change in the interaction energy from flipping one spin:
$T_{int}=|\langle H_{int}\rangle|/Nk_{B}$. If $T_{int}<T^{+},T^{-}$ then the
interactions will turn off before the random field effects as $T$ is
increased, and $|\langle\mathbf{P}\rangle|$ increases. If $T_{int}\gg
T^{+},T^{-}$ then the system is frozen by the interactions and we expect
little change in $|\langle\mathbf{P}\rangle|$ until $T\gg T_{int}$.

Overall, the effect of interactions is to make cancellation more likely due to
their antiferromagnetic character. Unless the interactions are extremely
strong, they make the non-monotonic $T$ dependence of
$|\langle\mathbf{F}_{q}\rangle|$ and therefore of $\Delta f(T)$ more likely.

##  IV Numerical Results

This analytic analysis of the IRGM is semi-quantitative. To make the arguments
more firm we perform numerical simulations. We do this for three different
physical pictures of the TLS. In all simulations there is a single qubit at
the origin.

In the first picture the TLS are charge traps near the surface of a two-
dimensional electron gas. The trap is positively charged when empty and then
relatively negatively charged when full. We include the image charge. This can
be described as a fluctuating dipole in the z direction, z being the growth
direction. These dipoles are uniformly distributed in a thin layer at
positions $\mathbf{r}_{j}=(x_{j},y_{j},z_{j})$ with
$-150\,\mathrm{nm}<x_{j},y_{j}<150\,\mathrm{nm}$ and $z_{j}=50\,\mathrm{nm}$.
This is the trap picture.

The second picture conceives of the TLS as point defect dipoles in the oxide
with orientations uniformly distributed in direction. Their positions
$\mathbf{r}_{j}=(x_{j},y_{j},z_{j})$ are uniformly distributed in a layer with
coordinates satisfying $\mathbf{r}_{j}=(x_{j},y_{j},z_{j})$ with
$-150\,\mathrm{nm}<x_{j},y_{j}<150\,\mathrm{nm}$ and
$30\,\mathrm{nm}<z_{j}<50\,\mathrm{nm}$. This is the random dipole picture.

In the third picture the TLS are distributed in the neighborhood of the qubit.
For the simulations, we take the TLS to be uniformly distributed in a
spherical shell with the qubit at the center.
$\mathbf{r}_{j}=(r_{j},\theta_{j},\phi_{j})$ satisfy
$60\,\mathrm{nm}<r_{j}<80\,\mathrm{nm}$, $0\leq\theta_{j}<\pi$, and
$0\leq\phi_{j}<2\pi$. The radii are chosen to make $T_{int}\sim 1$ K. This is
the spherical shell picture.

Figure 1: Temperature dependence of qubit frequency shifts in the trap picture
of the TLS in some representative samples. The shifts are proportional to one
component of the equilibrium electric field $\langle\mathbf{F}_{q}\rangle(T)$
at the qubit. Without Int. and With Int. mean the non-interacting and fully
interacting case, respectively. (a) Both non-interacting and interacting cases
show non-monotonic shifts. (b) Example in which the interaction causes non-
monotonic behavior.

We define a random field temperature scale $T_{r}=|\langle
H_{r}\rangle|/Nk_{B}$ in addition to $T_{int}$. We use the parameter values
$p_{0}=48$ Debye $\approx 1|e|$-nm and $N=30$, which are chosen because they
give $T_{2}$ of order $10^{-6}$s in the correct experimental range [26]. The
TLS density from these values is also consistent with those computed from the
magnitude of measured power spectra [27, 28]. The random field strength is
taken as $\Delta E_{0}=10^{4}$ V/m, which denotes the standard deviation of a
distribution centered on zero. With these values, $T_{r}\sim 0.1$ K and
$T_{int}\sim 1$ K in most samples, but while $T_{r}$ is almost independent of
the disorder because its strength is roughly fixed, $T_{int}$ can vary from
0.1 K to 10 K because it depends sensitively on the separations and
orientations of the TLS. Overall, these values are chosen to be representative
of experiments on semiconductor qubit systems, in the sense that most analyses
give something like our value for $N$ which together with value of $p_{0}$
that we use gives a reasonable dephasing time. A key observation here is that
$T_{int}$ comes out approximately correct when the distances of the TLS from
each and from the qubit, and the dipole magnitude, are chosen to fit $T_{2}$
and other noise experiments. Some further details are given in Appendix A. The
IRGM passes the self-consistency check that $T_{int}$ does indeed match the
temperature scale on which $\Delta f$ varies.

In Fig. 1 we plot one example of $\Delta f(T)$ in the trap picture for both
the non-interacting case $H_{int}=0$, which is exactly solvable, and the fully
interacting case, which is computed by a standard Monte Carlo (MC) algorithm.
We use arbitrary units for $\Delta f$ since the conversion factor
$\mathbf{c}_{q}$ is platform-dependent. For the interacting case, a moving
average over 11 neighboring temperatures is applied to obtain stable results,
and a smaller number of neighboring temperatures are used for the moving
average at the ends of the curves.

We stress that $\langle\mathbf{F}_{q}\rangle(T)$ is sample-dependent for all
three pictures in that changing the parameters in natural ranges can alter
$\Delta f(T)$ qualitatively. In particular, non-monotonic behavior of
$\langle\mathbf{F}_{q}\rangle$ is by no means universal. It is even possible
for a single sample that one component of the field is non-monotonic and
another is monotonic. Some idea of the variety of possible behaviors is given
in Appendix B.

Fig. 1 demonstrates that non-monotonicity can arise already even when the TLS
do not interact. This comes simply from the fact that there can be
cancellation of the random fields at $T=0$ that is lessened as $T$ increases
in certain circumstances, as explained above. The interactions tend to enhance
non-monotonicity as in Fig. 1(a), though this effect is not universal. In
fact, as we will see below, only a minority of samples for the trap picture
show non-monotonicity. Interactions can create non-monotonicity when the non-
interacting picture shows monotonicity as seen in Fig. 1(b). Again, this is
consistent with the idea of cancellation as the active ingredient in non-
monotonicity. Additional examples of the different types of behavior that can
occur for $\Delta f(T)$ are given in Appendix B.

Once universality of the non-monotonicity is ruled out, the question becomes
whether it is likely or not. To answer this we did simulations over 10,000
samples for each of the three pictures and determined whether a monotonic or
non-monotonic behavior was observed for each component of the electric field.
The precise criterion for monotonicity or its absence is given in Appendix C.

Trap | $\langle F_{q,x}\rangle$ | $\langle F_{q,y}\rangle$ | $\langle F_{q,z}\rangle$
---|---|---|---
$T_{r}<T_{int}$
Without Int. | 15.9% | 16.4% | 12.0%
With Int. | 48.0% | 48.6% | 40.1%
$T_{r}\sim T_{int}$
Without Int. | 11.0% | 10.9% | 7.4%
With Int. | 30.5% | 30.2% | 24.2%
Random dipole | $\langle F_{q,x}\rangle$ | $\langle F_{q,y}\rangle$ | $\langle F_{q,z}\rangle$
$T_{r}<T_{int}$
Without Int. | 14.7% | 15.9% | 12.4%
With Int. | 61.9% | 61.9% | 57.2%
$T_{r}\sim T_{int}$
Without Int. | 10.0% | 9.4% | 7.8%
With Int. | 42.5% | 42.3% | 36.9%
Spherical shell | $\langle F_{q,x}\rangle$ | $\langle F_{q,y}\rangle$ | $\langle F_{q,z}\rangle$
$T_{r}<T_{int}$
Without Int. | 25.0% | 25.3% | 28.0%
With Int. | 89.5% | 89.4% | 90.1%
$T_{r}\sim T_{int}$
Without Int. | 21.0% | 21.8% | 25.8%
With Int. | 79.4% | 79.0% | 81.4%
Table 1: The fractional number of samples showing non-monotonicity obtained
from Monte Carlo simulation and exact solutions for three physical pictures.
$\langle F_{q,i}\rangle$ is the $i$ component of the vector
$\langle\mathbf{F}_{q}\rangle(T)$. Without Int. and With Int. mean the non-
interacting and fully interacting case, respectively. $T_{r}<T_{int}$
indicates $T_{r}\sim$ 0.1 K and $T_{int}\sim$ 1 K, while $T_{r}\sim T_{int}$
indicates $T_{r}\sim$ 0.1 K and $T_{int}\sim$ 0.1K.

The results in Table 1 support the physical picture explained above. In the
trap picture, the cancellation of fields from different dipoles is relatively
small, since the vectors leading from the qubit to the TLS, while not
collinear, generally do not make large angles with each other. Similar
statements apply to the random dipole picture, but there is some additional
cancellation due to the different orientations. The most interesting result is
for the spherical shell picture. Here we see the appearance of non-monotonic
behavior for non-interacting TLS in about one quarter of the cases, as would
be suggested by the above arguments. With dipoles on all sides of the qubit,
the cancellation effect is quite strong.

Interactions do promote non-monotonicity in all cases, as expected, especially
when the interactions are strong: $T_{int}>T_{r}$. Overall, non-monotonicity
increases as we proceed from the trap to the random dipole to the spherical
shell pictures. The interaction enhancement reinforces this pattern of non-
monotonicity.

Figure 2: PIRS data: theory and experiment. The points are measured frequency
shifts for six qubits Q1-Q6 from Ref. [18] and the dashed lines are
theoretical fits. The qubits are situated in a one-dimensional array. The
fitting procedure is described in detail in the text. The applied magnetic
field is assumed to be in the $y$-direction.

We turn now to comparison of theory and experiment. In nearly all measurements
$\Delta f$ is measured as a function of time, not temperature. Comparing to
data of this type would require detailed modeling of the heat flow in the
system, which is outside the scope of this paper. We therefore compare only to
the temperature data for $\Delta f(T)$ for 6 qubits in a single device
reported in Ref. [18]. We plot these data with a theoretical fit in Fig. 2.
The fit was done as follows. We first chose a set of TLS positions and random
fields so that the parameters were in a range where non-monotonicity could be
expected, and peak in $\Delta f$ would be around 250 mK. Then, since the
curves have a fairly similar shape but differ in vertical scale, we varied
$\mathbf{c}_{q}$ for each of the curves. Finally, the positions of the TLS
were adjusted to fit each curve individually. The main feature that needed to
be accounted for in this final step was the sharper peak and steeper falloff
at high $T$ that is seen in qubits 3-6. Some further details are given in
Appendix D.

The fits are quite good quantitatively, but not too much should be made of
this, since the number of parameters far exceeds the number of qualitative
features to be fit for each curve. However, even given this, the fit does
provide evidence for the correctness of the IRGM. The non-monotonicity arises
naturally, but also the linear behavior at small $T$, (which is caused by the
uniform distribution of the $\mathbf{E}_{j}$ near zero) and finally the $1/T$
behavior at large $T$.

##  V Conclusion

We conclude by summarizing the strong and weak points of the IRGM as a model
for PIRS. There are three evaluation categories; qualitative phenomenological
understanding, semi-quantitative self-consistency, and quantitative fit of
theory and experiment.

In qualitative terms, the surprising non-monotonic $T$ dependence and some of
the other features of the $T$-dependence of the qubit frequencies finds a
natural explanation in the IRGM. The important theoretical ingredients are the
cancellations due to vector summations that involve only a relatively small
number of variables combined with the natural temperature dependence of the
TLS fluctuations and the depolarizating effects of interactions. The
explanation of non-monotonicity is quite defies the usual expectation that
thermal effects in the absence of phase transitions tend to be monotonic.
Overall, this is quite strong evidence for the IRGM. Aside from the non-
monotonicity, there is the observation that $\Delta f$ is sometimes negative
[16, 19]. This is also somewhat surprising if one assumes that the heating
affects the qubits directly in some fashion. In the IRGM, there is nothing
that constrains the sign of the components of $\mathbf{c}_{q}$, so the sign of
the effect is not determined. Similarly, the fact that the effect is not
resonant with qubit frequencies suggests that an ancillary part of the device
is driven by the heating - in the IRGM, the system of TLS is driven.
Hysteresis does not seem to be a feature of PIRS. This might seem to argue
against the IRGM but in fact with only a few tens of TLS involved, this aspect
of glassiness does not argue against the model. In contrast, no T-dependent
electric field shows up in measurements at the charge sensor [18], which is
not explained in the model as it stands.

There are two experimental scales that must be consistent with the theory: the
overall magnitude of $\Delta f$ (1 MHz in dot systems), and the temperature of
the peak in $\Delta f$ (about 0.2 to 0.4 K in dot systems). The first number
is very consistent with the roughly known numbers for the size of the dipoles
and their presumed positions. The second depends on the distribution of the
random local fields $\mathbf{E}_{j}$ and indeed the distribution must be such
that $|p_{0}\mathbf{E}_{j}/k_{B}|$ is clustered near 0.3 K. There is no
obvious reason why this should be so, so the IRGM does include at least one ad
hoc element.

The fit to the data in Fig. 2 is strikingly accurate, but it raises questions.
If interactions are relatively weak for some reason (such that the TLS are
particularly far apart), then why do six out of six qubits all show non-
monotonicity? This is only consistent with the spherical shell picture, which
in turn is not very consistent with the usual idea that the TLS are associated
with the oxide layer. Furthermore, the non-monotonicity in all six qubits
means that there must be correlations between the positions of the TLS and the
strength and direction of their random fields. Specifically, the TLS closer to
the qubits must have stronger random fields for each qubit. In addition, in
order for the shift to be positive in all qubits, the random fields for the
nearby TLS must all have the same orientation across all six qubits.

We conclude that the basic mechanism of PIRS is explained by the IRGM, but
that the explanation is far from complete at this stage. Most likely the model
needs to supplemented by a better picture of the positions of the TLS and a
better understanding of their physical nature. This would limit the model to a
smaller region of its parameter space than we have investigated here.

###### Acknowledgements.

We acknowledge helpful discussions and correspondence with S.N. Coppersmith,
M.A. Eriksson, M. Friesen, A. Laucht, A. Morello, A. Saraiva, L.M.K.
Vandersypen, and H. Yang. This research was sponsored by the Army Research
Office (ARO) under Awards No. W911NF-17-1-0274 and No. W911NF-22-1-0090. The
views, conclusions, and recommendations contained in this document are those
of the authors and are not necessarily endorsed nor should they be interpreted
as representing the official policies, either expressed or implied, of the
Army Research Office (ARO) or the U.S. Government. The U.S. Government is
authorized to reproduce and distribute reprints for Government purposes
notwithstanding any copyright notation herein. This research was performed
using the computer resources and assistance of the UW-Madison Center for High
Throughput Computing (CHTC) in the Department of Computer Sciences. The CHTC
is supported by UW-Madison, the Advanced Computing Initiative, the Wisconsin
Alumni Research Foundation, the Institutes for Discovery, and the National
Science Foundation, and is an active member of the Open Science Grid, which is
supported by the National Science Foundation and the U.S. Department of
Energy’s Office of Science.

##  Appendix A Magnitude of frequency shift in quantum dot architecture

Here we give an order-of-magnitude estimate of the PIRS effect and some of the
intermediate quantities involved in it for a Si/SiGe hetrostructure quantum
dot device with a micromagnet. In this case, $\Delta f$ is due to the shift in
position of the electron in the non-uniform magnetic field caused by the
micromagnet. Let there be an electron spin qubit in a quantum dot at the
origin of coordinates. The electron is at the bottom of a circularly symmetric
two-dimensional harmonic potential $k(x^{2}+y^{2})/2$. At this point there is
a magnetic field gradient $\partial B_{i}/\partial x_{j}$, where $i$ and $j$
are Cartesian indices.

The qubit frequency shift is given by $\Delta
f=\mathbf{c}_{q}\cdot[\langle\mathbf{F}_{q}(T)\rangle-\langle\mathbf{F}_{q}(T)\rangle(T=0)]$,
with [26]:

$\mathbf{c}_{q}=\frac{g\mu_{B}}{h}\frac{q}{m_{t}\omega_{orb}^{2}}\
\left(\frac{\partial B_{y}}{\partial x}\hat{x}+\frac{\partial B_{y}}{\partial
y}\hat{y}\right).$ (3)

We have assumed that the applied field is in the $y$-direction. A typical
device of this kind that was particularly well-characterized was described in
Ref. [Kawakami:2014p666]. The magnetic field gradients for that device in
units of mT(nm)-1 were $\partial B_{y}/\partial x=-0.05$ and $\partial
B_{y}/\partial y=0.18$. The transverse effective mass
$m_{t}=0.19\,m_{e}=1.73\times 10^{-31}\,$kg. We take the lowest orbital
excitation frequency as $\omega_{orb}\sim 2meV/\hbar$, which is related to the
spring constant by $k=m_{t}\omega_{orb}^{2}$, and an average value of $|\nabla
B|$ as 0.1 mT (nm)-1. These numerical values should be more or less typical of
micromagnet-based Si/SiGe devices, but variations from device to device can
certainly alter our estimate.

A single component of $\langle\mathbf{F}_{q}\rangle$ at $T=0$ is the result of
a random-walk summation of the same component of the field exerted at the
position of the qubit by the N TLSs. It is therefore given by $\sqrt{N}$ times
the rms value of the individual contributions to one component in the sum in
Eq. 1. There is an additional angular average over the directions of
$\mathbf{p}_{i}$ with the result that $\Delta
f(T=0)\sim\sqrt{2N/3}\,p_{0}/4\pi\varepsilon\varepsilon_{r}d^{3}$, where $d$
is an average distance from the TLS to the qubit, and we use $d\sim 50$ nm and
$\varepsilon_{r}\sim$11\. Combining this with Eq. 3 we find $\Delta f(T=0)\sim
1.4$ MHz, not too far from what is observed for the maximum $\Delta f$, which
should be roughly comarable with the computed quantity. With these parameters,
the qubit moves about 0.5 nm due to $\langle\mathbf{F}_{q}\rangle$, a field of
about 4500 V/m.

##  Appendix B Further examples of the temperature dependence of the electric
field

In this section we give a few representative examples of the temperature
dependence of the qubit frequency for single TLS configurations in the
different pictures of TLS positions and directions. In each figure we show two
configurations but with the interaction turned off and on. In fact we compute
the $y$ component of $\langle\mathbf{F}_{q}\rangle$ and leave
$\mathbf{c}_{q},y$ arbitrary. Then we enforce the condition that $\Delta
f(T=0)=0$. This means that the ground state configurations for the interacting
and non-interacting cases and their resultant $\langle\mathbf{F}_{q}\rangle$
may be quite different. The curves for the interacting case are smoothed by
averaging over the 11 points centered at the plotted point, except at the ends
of the curve. Note that $\langle\mathbf{F}_{q}\rangle\rightarrow 0$ as
$T\rightarrow\infty$, but this asymptote is usually off the plotted region.

In Fig. 1 we plot results for the random dipole picture. Fig. 1(a) is an
example where $T^{+}$ considerably exceeds $T^{-}$, leading in the non-
interacting case to a peak at relatively low T. Interactions mainly shift the
peak but leave non-monotonicity intact. In Fig. 1(b) the needed cancellation
pattern does not occur for the non-interacting case but there is non-
monotonicity in the interacting case because of increased cancellation.
Interactions have a strong influence on the ground state configuration for
this particular sample, as indicated by the change in sign for the interacting
and non-interacting cases.

In Fig. 2 we see $\Delta f(T)$ for the spherical shell picture for two
different TLS configurations. In this picture the interactions are more
effective in producing non-monotonicity and once more we see that the non-
monotonicity can be present already in the non-interacting case or it can be
induced by the interactions. $\Delta f(T)$ can show somewhat surprising
behavior in the interacting case, in this case a change of sign as a function
of T. This has not been observed to date. If it were, it would be a sign that
interactions are important.

In Fig. 3 we plot $\Delta f(T)$ for two samples belonging to the trap and
random dipole pictures. These plots are mainly included to dispel any
impression that non-monotonicity is universal in the IRGM. Both the non-
interacting and the interacting cases can show monotonic behavior as a
function of temperature. This can happen as in Fig. 3(a), where the ground
state configuration of the TLS changes drastically when interactions are
turned on, or as in Fig. 3(b), where the two ground states are apparently
rather similar.

Figure 3: Temperature dependence of qubit frequency shifts in the random
dipole picture of the TLS. The shifts are computed from the equilibrium
electric field
$\langle\mathbf{F}_{q}\rangle(T)-\langle\mathbf{F}_{q}\rangle(T=0)$ at the
position of the qubit for two configurations of the TLS. Without Int. and With
Int. mean the non-interacting ($H_{int}=0$) and fully interacting case,
respectively. (a) Both non-interacting and interacting cases show non-
monotonic shifts. (b) Example in which the interaction causes non-monotonic
behavior. The applied magnetic field is assumed to be in the $y$-direction.
Figure 4: Temperature dependence of qubit frequency shifts in the spherical
shell picture of the TLS. The shifts are computed from the equilibrium
electric field
$\langle\mathbf{F}_{q}\rangle(T)-\langle\mathbf{F}_{q}\rangle(T=0)$ at the
position of the qubit for two configurations of the TLS. Without Int. and With
Int. mean the non-interacting ($H_{int}=0$) and fully interacting case,
respectively. (a) Both non-interacting and interacting cases show non-
monotonic shifts. (b) Example in which interaction causes non-monotonic
behavior. The applied magnetic field is assumed to be in the $y$-direction.
Note the change in vertical scale from (a) to (b). Figure 5: Temperature
dependence of qubit frequency shifts in the trap and random dipole picture of
the TLS. The shifts are computed from the equilibrium electric field
$\langle\mathbf{F}_{q}\rangle(T)-\langle\mathbf{F}_{q}\rangle(T=0)$ at the
position of the qubit for two configurations of the TLS. Without Int. and With
Int. mean the non-interacting ($H_{int}=0$) and fully interacting case,
respectively. Both non-interacting and interacting cases show monotonic shifts
for (a) trap and (b) random dipole pictures. The applied magnetic field is
assumed to be in the $y$-direction. Note the change in vertical scale from (a)
to (b).

##  Appendix C Non-monotonicity criterion

To determine whether a component $F(T)$ of $\langle\mathbf{F_{q}}\rangle$ has
non-monotonic T dependence, we set up a criterion as follows: from exact or
numerical results, we define a set of differences

$F_{diff}=\\{F_{m+1}-F_{m}|\;m=1,2,3,...,M-1\\}$ (4)

where $M=100$ is the number of temperature points for evaluation. The average
slope magnitude is defined as

$s=\frac{1}{M-1}\sum_{m=1}^{M-1}|F_{m+1}-F_{m}|.$ (5)

To avoid false positives from small random fluctuations (particularly
important for MC simulations), small slope elements are excluded from
$F_{diff}$ so that the smaller set is

$F_{large}=\\{F_{m+1}-F_{m}|\;\frac{s}{2}<|F_{m+1}-F_{m}|\\}.$ (6)

Defining signs of the differences as $\sigma_{m}=\mathrm{sgn}(F_{m+1}-F_{m})$,
we have groups of positive and negative slopes:

$\displaystyle F_{pos}$
$\displaystyle=\\{F_{m+1}-F_{m}|\;\sigma_{m}>0\;\mathrm{and}\;F_{m+1}-F_{m}\in
F_{large}\\},$ (7) $\displaystyle F_{neg}$
$\displaystyle=\\{F_{m+1}-F_{m}|\;\sigma_{m}<0\;\mathrm{and}\;F_{m+1}-F_{m}\in
F_{large}\\}.$

The final non-monotonicity criterion is

$\frac{\min(|F_{pos}|,|F_{neg}|)}{|F_{pos}|+|F_{neg}|}>0.1\;\mathrm{and}\;s>5$
(8)

where the first inequality requires that $\langle\mathbf{F}_{q}\rangle(T)$ has
non-negligible positive and negative parts of slopes and the second one
demands that the overall frequency shift in the temperature range is not so
small. The tolerance values, 0.1 and 5, are empirically chosen and can be
adjusted for different systems.

##  Appendix D Fitting procedure for Fig. 2 of main text

In Fig. 2 of the main text we give a comparison of theory and the PIRS
experiment of Undseth et al. in which $\Delta f$ was measured for each of six
qubits in a row. We found that the best fit is obtained by taking the case
that $T_{r}\gg T_{int}$, which amounts to a non-interacting model. We used a
trap picture, but the other two pictures could also have been used for the
fit. The parameters are the same as those in the description of the trap
picture in the main text except for the $z$ coordinates of the TLS, which are
now taken as $z_{j}=36$ nm following Ref. [18, 1], and $\Delta E_{0}=10^{5}$
V/m. The six samples are generated by varying the conversion factor
$\mathbf{c}_{q,y}$ and the positions of TLS in the $x-y$ plane, which are
randomly assigned within circles whose centers are TLS positions of a
reference sample and radii are 5 nm. The applied magnetic field is assumed to
be in the $y$-direction. As we showed in Sec. 1, the order of magnitude of the
effect is consistent between theory and experiment. The best fit for the
conversion factors for the qubits $Q_{i}$ with $i=$ 1, 2, 3, 4, 5, 6 were in
the ratios 0.105 : 0.114 : 0.072 : 0.077 : 0.103 : 0.059. All fit parameters
are available from the authors on request.

## References

  * Philips  _et al._ [2022] S. G. Philips, M. T. Mądzik, S. V. Amitonov, S. L. de Snoo, M. Russ, N. Kalhor, C. Volk, W. I. Lawrie, D. Brousse, L. Tryputen, B. Paquelet Wuetz, A. Sammak, M. Veldhorst, G. Scappucci, and L. M. K. Vandersypen, Universal control of a six-qubit quantum processor in silicon, Nature 609, 919–924 (2022).
  * Mills  _et al._ [2022] A. R. Mills, C. R. Guinn, M. J. Gullans, A. J. Sigillito, M. M. Feldman, E. Nielsen, and J. R. Petta, Two-qubit silicon quantum processor with operation fidelity exceeding 99%, Science Advances 8, eabn5130 (2022).
  * Dekker  _et al._ [1991] C. Dekker, A. J. Scholten, F. Liefrink, R. Eppenga, H. van Houten, and C. T. Foxon, Spontaneous resistance switching and low-frequency noise in quantum point contacts, Phys. Rev. Lett. 66, 2148 (1991).
  * Sakamoto  _et al._ [1995] T. Sakamoto, Y. Nakamura, and K. Nakamura, Distributions of single-carrier traps in GaAs/Alx Ga1-xAs heterostructures, Applied Physics Letters 67, 2220 (1995).
  * Kurdak  _et al._ [1997] I. Kurdak, C.-J. Chen, D. C. Tsui, S. Parihar, S. Lyon, and G. W. Weimann, Resistance fluctuations in GaAs/Alx Ga1-xAs quantum point contact and hall bar structures, Phys. Rev. B 56, 9813 (1997).
  * Buizert  _et al._ [2008] C. Buizert, F. H. L. Koppens, M. Pioro-Ladrière, H.-P. Tranitz, I. T. Vink, S. Tarucha, W. Wegscheider, and L. M. K. Vandersypen, Insitu reduction of charge noise in GaAs/AlGaAs Schottky-gated devices, Phys. Rev. Lett. 101, 226603 (2008).
  * Pioro-Ladrière  _et al._ [2005] M. Pioro-Ladrière, J. H. Davies, A. R. Long, A. S. Sachrajda, L. Gaudreau, P. Zawadzki, J. Lapointe, J. Gupta, Z. Wasilewski, and S. Studenikin, Origin of switching noise in GaAs/Alx Ga1-xAs lateral gated devices, Phys. Rev. B 72, 115331 (2005).
  * Takeda  _et al._ [2013] K. Takeda, T. Obata, Y. Fukuoka, W. M. Akhtar, J. Kamioka, T. Kodera, S. Oda, and S. Tarucha, Characterization and suppression of low-frequency noise in Si/SiGe quantum point contacts and quantum dots, Appl. Phys. Lett. 102, 123113 (2013).
  * Croot  _et al._ [2019] X. Croot, S. Pauka, M. Jarratt, H. Lu, A. Gossard, J. Watson, G. Gardner, S. Fallahi, M. Manfra, and D. Reilly, Gate-sensing charge pockets in the semiconductor-qubit environment, Phys. Rev. Applied 11, 064027 (2019).
  * Campbell and Lenahan [2002] J. Campbell and P. Lenahan, Density of states of Si/SiO2 interface trap centers, Appl. Phys. Lett. 80, 1945 (2002).
  * Hofheinz  _et al._ [2006] M. Hofheinz, X. Jehl, M. Sanquer, G. Molas, M. Vinet, and S. Deleonibus, Individual charge traps in silicon nanowires, The European Physical Journal B-Condensed Matter and Complex Systems 54, 299 (2006).
  * Connors  _et al._ [2019] E. J. Connors, J. Nelson, H. Qiao, L. F. Edge, and J. M. Nichol, Low-frequency charge noise in Si/SiGe quantum dots, Phys. Rev. B 100, 165305 (2019).
  * Connors  _et al._ [2022] E. J. Connors, J. Nelson, L. F. Edge, and J. M. Nichol, Charge-noise spectroscopy of Si/SiGe quantum dots via dynamically-decoupled exchange oscillations, Nature Communications 13, 1 (2022).
  * Boter  _et al._ [2020] J. M. Boter, X. Xue, T. Krähenmann, T. F. Watson, V. N. Premakumar, D. R. Ward, D. E. Savage, M. G. Lagally, M. Friesen, S. N. Coppersmith, M. A. Eriksson, R. Joynt, and L. M. K. Vandersypen, Spatial noise correlations in a Si/SiGe two-qubit device from Bell state coherences, Phys. Rev. B 101, 235133 (2020).
  * Rojas-Arias  _et al._ [2023] J. Rojas-Arias, A. Noiri, P. Stano, T. Nakajima, J. Yoneda, K. Takeda, T. Kobayashi, A. Sammak, G. Scappucci, D. Loss, and S. Tarucha, Spatial noise correlations beyond nearest-neighbor in 28 Si/SiGe spin qubits, arXiv preprint arXiv:2302.11717 (2023).
  * Takeda  _et al._ [2018] K. Takeda, J. Yoneda, T. Otsuka, T. Nakajima, M. Delbecq, G. Allison, Y. Hoshi, N. Usami, K. M. Itoh, S. Oda, T. Kodera, and S. Tarucha, Optimized electrical control of a Si/SiGe spin qubit in the presence of an induced frequency shift, npj Quantum Inf. 4, 54 (2018).
  * Watson  _et al._ [2018] T. F. Watson, S. G. J. Philips, E. Kawakami, D. R. Ward, P. Scarlino, M. Veldhorst, D. E. Savage, M. G. Lagally, M. Friesen, S. N. Coppersmith, M. A. Eriksson, and L. M. K. Vandersypen, A programmable two-qubit quantum processor in silicon, Nature 555, 633 (2018).
  * Undseth  _et al._ [2023] B. Undseth, O. Pietx-Casas, E. Raymenants, M. Mehmandoost, M. T. Mądzik, S. Philips, S. de Snoo, D. Michalak, S. Amitonov, L. Tryputen, B. Wuetz, V. Fezzi, D. Esposti, A. Sammak, G. Scappucci, and L. Vandersypen, Hotter is easier: unexpected temperature dependence of spin qubit frequencies, arXiv preprint arXiv:2304.12984 (2023).
  * Zwerver  _et al._ [2022] A. Zwerver, T. Krähenmann, T. Watson, L. Lampert, H. C. George, R. Pillarisetty, S. Bojarski, P. Amin, S. Amitonov, J. Boter, R. Caudillo, D. Correas-Serrano, J. P. Dehollain, G. Droulers, E. M. Henry, R. Kotlyar, M. Lodari, F. Lüthi, D. J. Michalak, B. K. Mueller, S. Neyens, J. Roberts, N. Samkharadze, G. Zheng, O. K. Zietz, G. Scappucci, M. Veldhorst, L. M. K. Vandersypen, and J. S. Clarke, Qubits made by advanced semiconductor manufacturing, Nature Electronics 5, 184 (2022).
  * Freer  _et al._ [2017] S. Freer, S. Simmons, A. Laucht, J. T. Muhonen, J. P. Dehollain, R. Kalra, F. A. Mohiyaddin, F. E. Hudson, K. M. Itoh, J. C. McCallum, D. N. Jamieson, A. S. Dzurak, and A. Morello, A single-atom quantum memory in silicon, Quantum Sci. Tech. 2, 015009 (2017).
  * Savytskyy  _et al._ [2023] R. Savytskyy, T. Botzem, I. F. de Fuentes, B. Joecker, F. E. Hudson, K. M. Itoh, A. M. Jakob, B. C. Johnson, D. N. Jamieson, A. S. Dzurak, and A. Morello, An electrically-driven single-atom ‘flip-flop’ qubit, arXiv:2305.14515 (2023).
  * Mickelsen  _et al._ [2023] D. Mickelsen, H. M. Carruzzo, S. N. Coppersmith, and C. C. Yu, Effects of temperature fluctuations on charge noise in quantum dot qubits, arXiv preprint arXiv:2305.14515 (2023).
  * Müller  _et al._ [2015] C. Müller, J. Lisenfeld, A. Shnirman, and S. Poletto, Interacting two-level defects as sources of fluctuating high-frequency noise in superconducting circuits, Phys. Rev. B 92, 035442 (2015).
  * Müller  _et al._ [2019] C. Müller, J. H. Cole, and J. Lisenfeld, Towards understanding two-level-systems in amorphous solids: insights from quantum circuits, Repts. on Prog. in Phys. 82, 124501 (2019).
  * Rahman  _et al._ [2009] R. Rahman, S. H. Park, T. B. Boykin, G. Klimeck, S. Rogge, and L. C. L. Hollenberg, Gate-induced g-factor control and dimensional transition for donors in multivalley semiconductors, Phys. Rev. B 80, 155301 (2009).
  * Choi and Joynt [2022] Y. Choi and R. Joynt, Anisotropy with respect to the applied magnetic field of spin qubit decoherence times, npj Quantum Information 8, 70 (2022).
  * Shehata  _et al._ [2023] M. M. E. K. Shehata, G. Simion, R. Li, F. A. Mohiyaddin, D. Wan, M. Mongillo, B. Govoreanu, I. Radu, K. De Greve, and P. Van Dorpe, Modeling semiconductor spin qubits and their charge noise environment for quantum gate fidelity estimation, Phys. Rev. B 108, 045305 (2023).
  * Kępa  _et al._ [2023] M. Kępa, Ł. Cywiński, and J. A. Krzywda, Correlations of spin splitting and orbital fluctuations due to 1/f charge noise in the Si/SiGe quantum dot, arXiv preprint arXiv:2305.06011 (2023).