#  _Diffusion-RWKV_ :
Scaling RWKV-Like Architectures for Diffusion Models

Zhengcong Fei, Mingyuan Fan, Changqian Yu
Debang Li, Junshi Huang*
Kunlun Inc.
<EMAIL_ADDRESS>
###### Abstract

Transformers have catalyzed advancements in computer vision and natural
language processing (NLP) fields. However, substantial computational
complexity poses limitations for their application in long-context tasks, such
as high-resolution image generation. This paper introduces a series of
architectures adapted from the RWKV model used in the NLP, with requisite
modifications tailored for diffusion model applied to image generation tasks,
referred to as Diffusion-RWKV. Similar to the diffusion with Transformers, our
model is designed to efficiently handle patchnified inputs in a sequence with
extra conditions, while also scaling up effectively, accommodating both large-
scale parameters and extensive datasets. Its distinctive advantage manifests
in its reduced spatial aggregation complexity, rendering it exceptionally
adept at processing high-resolution images, thereby eliminating the necessity
for windowing or group cached operations. Experimental results on both
condition and unconditional image generation tasks demonstrate that Diffison-
RWKV achieves performance on par with or surpasses existing CNN or
Transformer-based diffusion models in FID and IS metrics while significantly
reducing total computation FLOP usage.

Figure 1:  Diffusion models with RWKV-like backbones achieve comparable image
quality. Selected samples generated by class-conditional Diffusion-RWKV
trained on the ImageNet with resolutions of 256$\times$256 and 512$\times$512,
respectivly.

##  1 Introduction

Transformers [81, 64, 53, 10, 25, 44], which have gained prominence due to
their adaptable nature and proficient information processing capabilities,
have set new standards across various domains including computer vision and
NLP. Notably, they have demonstrated exceptional performance in tasks like
image generation [57, 8, 9, 42, 65, 4, 14, 58]. However, the self-attention
operation in Transformer exhibits a quadratic computational complexity,
thereby limiting their efficiency in handling long sequences and poses a
significant obstacle to their widespread application [39, 80, 89, 87, 14].
Consequently, there is a pressing need to explore architectures that can
effectively harness their versatility and robust processing capabilities while
mitigating the computational demands. It becomes even more crucial in the
context of high-resolution image synthesis or the generation of lengthy
videos.

Figure 2: Overall framework of diffusion models with RWKV-like architectures.
(a) The Diffusion-RWKV architecture comprises $L$ identical Bi-RWKV layers, a
patch embedding, and a projection layer. A skip connection is established
between shallow and deep stacked Bi-RWKV layers for information flow. (b) The
detailed composition of Bi-RWKV layers, includes a shift method and a
bidirectional RNN cell in spatial mix, and a shift with two activate functions
in channel mix.

In recent developments, models such as RWKV [59] and Mamba [21], have emerged
as popular solutions for enhancing efficiency and processing lengthy textual
data with comparable capacity. These innovative models exhibit characteristics
akin to transformers [3, 6, 43, 45, 63, 64, 73, 78, 15, 85, 12], encompassing
to handle long-range dependencies and parallel processing. Moreover, they have
demonstrated scalability, performing admirably with large-scale NLP and CV
datasets [90, 18]. However, given the substantial dissimilarities between
visual and textual data domains, it remains challenging to envision complete
replacement of Transformers with RWKV-based methods for vision generation
tasks [11]. It becomes imperative to conduct an in-depth analysis of how these
models are applied to image generation tasks. This analysis should investigate
their scalability in terms of training data and model parameters, evaluate
their efficiency in handling visual data sequentially, and identify the
essential techniques to ensure model stability during scaling up.

This paper introduces Diffusion-RWKV, which is designed to adapt the RWKV
architecture in diffusion models for image generation tasks. The proposed
adaptation aims to retain the fundamental structure and advantages of RWKV
[59] while incorporating crucial modifications to tailor it specifically for
synthesizing visual data. Specifically, we employ Bi-RWKV [11] for backbone,
which enables the calculation within linear computational complexity in an RNN
form forward and backward. We primarily make the architectural choices in
diffusion models, including condition incorporation, skip connection, and
finally offer empirical baselines that enhance the model’s capability while
ensuring scalability and stability. Building on the aforementioned design, a
diverse set of Diffusion-RWKV models is developed, as a broad range of model
scales, ranging from tiny to large. These models are training on CIFAR-10,
Celebrity to ImageNet-1K using unconditional and class-conditioned training at
different image resolutions. Moreover, performance evaluations are conducted
in both raw and latent spaces. Encouragingly, under the same settings,
Diffusion-RWKV has comparable performance to competitor DiT [58] in image
generation, with lower computational costs while maintaining stable
scalability. This achievement enables Diff-RWKV training parallelism, high
flexibility, excellent performance, and low inference cost simultaneously,
making it a promising alternative in image synthesis. The contribution can be
summarized as:

  * •

In a pioneering endeavor, we delve into the exploration of a purely RWKV-based
diffusion model for image generation tasks, positioning as a low-cost
alternative to Transformer. Our model not only inherits the advantages of RWKV
for long-range dependency capture, but also reduces complexity to a linear
level.

  * •

To cater to the demands of image synthesis, we have conducted a comprehensive
and systematic investigation of Diffusion-RWKV models by exploring various
configuration choices pertaining to conditioning, block design, and model
parameter scaling.

  * •

Experimental results indicate that Diffusion-RWKV performs comparably to well-
established benchmarks DiTs and U-ViTs, exhibiting lower FLOPs and faster
processing speeds as resolution increases. Notably, Diffusion-RWKV achieves a
2.95 FID score trained only on ImageNet-1k. Code and model are available at
https://github.com/feizc/Diffusion-RWKV.

##  2 Methodology

This section commences by providing an overview of the foundational concepts
in Section 2.1. Subsequently, we delve into a comprehensive exposition of the
RWKV-based diffusion models for image generation in Section 2.2. It
encompasses various aspects such as image patchnify, stacked Bi-RWKV block,
skip connections, and condition incorporation. Lastly, we perform
computational analysis and establish optimal model scaling configurations.

###  2.1 Preliminaries

#### Diffusion models.

Diffusion models have emerged as a new family of generative models that
generate data by iterative transforming random noise through a sequence of
deconstructible denoising steps. It usually includes a forward noising process
and a backward denoising process. Formally, given data $x_{0}$ sampled from
the distribution $p(x_{0})$, the forward noising process involves iteratively
adding Gaussian noise to the data, creating a Markov Chain of latent variables
$x_{1},\ldots,x_{T}$, where:

$q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}{I}),$
(1)

and $\beta_{1},\ldots,\beta_{T}$ are hyperparameters defining the noise
schedule. After a pre-set number of diffusion steps, $x_{T}$ can be considered
as standard Gaussian noise. A denoising network $\epsilon_{\theta}$ with
parameters $\theta$ is trained to learn the backward denoising process, which
aims to remove the added noise according to a noisy input. During inference, a
data point can be generated by sampling from a random Gaussian noise
$x_{T}\sim\mathcal{N}(0;{I})$ and iteratively denoising the sample by
sequentially sampling $x_{t-1}$ from $x_{t}$ with the learned denoising
process, as:

$x_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}(x_{t}-\frac{1-\alpha_{t}}{1-\overline{\alpha}_{t}}\epsilon(x_{t},t))+\sigma_{t}z,$
(2)

where $\overline{\alpha}t=\prod{s=1}^{t}\alpha_{s}$, $\alpha_{t}=1-\beta_{t}$,
and $\sigma_{t}$ denotes the noise scale. In practice, the diffusion sampling
process can be further accelerated with various sampling techniques [48, 74,
49].

#### RWKV-like structures.

RWKV [59] brought improvements for standard RNN architecture [30], which is
computed in parallel during training while inference like RNN. It involves
enhancing the linear attention mechanism and designing the receptance weight
key value (RWKV) mechanism. Generally, RWKV model consists of an input layer,
a series of stacked residual blocks, and an output layer. Each residual block
is composed of time-mix and channel-mix sub-block.

(i) The Time-Mix Block aims to improve the modeling of dependencies and
patterns within a sequence. It is achieved by replacing the conventional
weighted sum calculation in an attention mechanism with hidden states. The
time-mix block can effectively propagate and updates information across
sequential steps with hidden states and the updation can be expressed as
follows:

$\displaystyle q_{t}$ $\displaystyle=({\mu}_{q}\odot x_{t}+(1-{\mu}_{q})\odot
x_{t-1})\cdot W_{q},$ (3) $\displaystyle k_{t}$ $\displaystyle=({\mu}_{k}\odot
x_{t}+(1-{\mu}_{k})\odot x_{t-1})\cdot W_{k},$ (4) $\displaystyle v_{t}$
$\displaystyle=({\mu}_{v}\odot x_{t}+(1-{\mu}_{v})\odot x_{t-1})\cdot W_{v},$
(5) $\displaystyle o_{t}$ $\displaystyle=(\sigma(q_{t})\odot
h(k_{t},v_{t}))\cdot W_{o},$ (6)

where $q_{t}$, $k_{t}$, and $v_{t}$ are calculated by linearly interpolating
between the current input and the input at the previous time step. The
interpolation, determined by the token shift parameter $\mu$, ensures coherent
and fluent token representations. Additionally, a non-linear activation
function $\sigma$ is applied to $q_{t}$, and the resulting value is combined
with the hidden states $h(k_{t},v_{t})$ using element-wise multiplication. The
hidden states, which serve as both the reset gate and a replacement for the
traditional weighted sum value, can be computed as:

$\displaystyle p_{t}$ $\displaystyle=\max(p_{t-1},k_{t}),$ (7) $\displaystyle
h_{t}$ $\displaystyle=\frac{\text{exp}(p_{t-1}-p_{t})\odot
a_{t-1}+\text{exp}(k_{t}-p_{t})\odot v_{t}}{\text{exp}(p_{t-1}-p_{t})\odot
b_{t-1}+\text{exp}(k_{t}-p_{t})},$ (8)

where $a_{0},b_{0},p_{0}$ are zero-initialized. Intuitively, the hidden states
are computed recursively, and the vector $p$ serves as the reset gate in this
process.

(ii) Channel-Mix Block aims to amplify the outputs of time-mix block, which
can be given by:

$\displaystyle r_{t}$ $\displaystyle=({\mu}_{r}\odot o_{t}+(1-{\mu}_{r})\odot
o_{t-1})\cdot W_{r}$ (9) $\displaystyle z_{t}$ $\displaystyle=({\mu}_{z}\odot
o_{t}+(1-{\mu}_{z})\odot o_{t-1})\cdot W_{z}$ (10)
$\displaystyle\tilde{x}_{t}$
$\displaystyle=\sigma(r_{t})\odot(\max(z_{t},0)^{2}\cdot W_{v})$ (11)

The output $o_{t}$ contains historical information up to time $t$, and the
interpolation weight $\mu$ is derived from $o_{t}$ and $o_{t-1}$, similar to
the time-mix block, which also enhances the historical information
representation. Note that the calculations of hidden states may lead to
information loss and failure to capture long-range dependencies [59].

| #Params | $L$ | $D$ | $E$ | Gflops
---|---|---|---|---|---
Small | 38.9M | 25 | 384 | 4 | 1.72
Base | 74.3M | 25 | 768 | 4 | 3.32
Medium | 132.0M | 49 | 768 | 4 | 5.90
Large | 438.5M | 49 | 1024 | 4 | 19.65
Huge | 779.1M | 49 | 1536 | 4 | 34.95

Table 1: Scaling law model size. The model sizes and detailed hyperparameter
settings for scaling experiments. In between, $L$ is the number of stacked Bi-
RWKV layers, $D$ is the hidden state size, and $E$ is the embedding ratio.

###  2.2 Model Structure Design

We present Diffusion-RWKV, a variant of RWKV-like diffusion models, as a
simple and versatile architecture for image generation. Diffusion-RWKV
parameterizes the noise prediction network $\epsilon_{\theta}({x}_{t},t,{c})$,
which takes the timestep $t$, condition ${c}$ and noised image ${x}_{t}$ as
inputs and predicts the noise injected into data point ${x}_{t}$. As our goal
follows the cutting-edge RWKV architecture to maintain its scalability
characteristics, Diffusion-RWKV is grounded in the bidirectional RWKV [11]
architecture which operates on sequences of tokens. Figure 2 illustrates an
overview of the complete Diffusion-RWKV architecture. In the following, we
elaborate on the forward pass and the components that constitute the design
space of this model class.

#### Image tokenization.

The initial layer of Diffusion-RWKV performs a transformation of the input
image ${I}\in\mathbb{R}^{H\times W\times C}$ into flattened 2-D patches
${X}\in\mathbb{R}^{J\times(p^{2}\cdot C)}$. Subsequently, it converts these
patches into a sequence of $J$ tokens, each with $D$ dimension, by linearly
embedding each image patch in the input. Consistent with [10], learnable
positional embeddings are applied to all input tokens. The number of tokens
$J$ generated by the tokenization process is determined by the hyperparameter
patch size $p$, calculated as $\frac{H\times W}{p^{2}}$. The tokenization
layer supports both raw pixel and latent space representations.

#### Bi-directional RWKV block.

Subsequent to the embedding layer, the input tokens undergo processing through
a succession of identical Bi-RWKV blocks. Considering that the original RWKV
block was designed for one-dimensional sequence processing, we resort to [90],
which incorporates bidirectional sequence modeling tailored for vision tasks.
This adaptation preserves the core structure and advantages of RWKV [59] while
integrating critical modifications to tailor it for processing visual data.
Specifically, it employs a quad-directional shift operation tailored for two-
dimensional vision data and modifies the original causal RWKV attention
mechanism to a bidirectional global attention mechanism. The quad-directional
shift operation expands the semantic range of individual tokens, while the
bidirectional attention enables the calculation of global attention within
linear computational complexity in an RNN-like forward and backward manner. As
illustrated in the right part of Figure 2, the forward pass of Bi-RWKV blocks
amalgamates both forward and backward directions in the spatial and channel
mix modules. These alterations enhance the model’s long-range capability while
ensuring scalability and stability.

#### Skip connection.

Considering a series of $L$ stacked Bi-RWKV blocks, we categorize the blocks
into three groups: the first $\lfloor\frac{L}{2}\rfloor$ blocks as the shallow
group, the middle block as the central layer, and the remaining
$\lfloor\frac{L}{2}\rfloor$ blocks as the deep group as [18, 1]. Let
$h_{shallow}$ and $h_{deep}$ represent the hidden states from the main branch
and long skip branch, respectively, both residing in $\mathbb{R}^{J\times D}$.
We propose concatenating these hidden states and applying a linear projection,
expressed as Linear(Concate($h_{shallow},h_{deep}$)), before propagating them
to the subsequent block.

#### Linear decoder.

Upon completion of the final Bi-RWKV block, it becomes essential to decipher
the sequence of hidden states to generate an output noise prediction and
diagonal covariance prediction. These resulting outputs maintain a shape
equivalent to the original input image. To achieve this, a standard linear
decoder is employed, wherein the final layer norm is applied, and each token
is linearly decoded into a $p^{2}\cdot C$ tensor. Finally, the decoded tokens
are rearranged to match their original spatial layout, yielding the predicted
noise and covariance.

Figure 3: Ablation experiments and model analysis for different designs with
DRWKV-S/2 model on the CIFAR10 dataset. We report FID metrics on 10K generated
samples every 50K steps. We can find that: (a) Patch size. A smaller patch
size can improve the image generation performance. (b) Skip operation.
Combining the long skip branch can accelerate the training as well as optimize
generated results. (c) Variants of condition incorporation. AdaLN-Zero is an
effective strategy for conditioning. (d) Model parameters scaling. As we
expected, holding the patch size constant, increasing the model parameters can
consistently improve the generation performance.

#### Condition incorporation.

In addition to the noised image inputs, diffusion models process supplementary
conditional information, such as noise timesteps $t$ and condition c, which
usually encompass class labels or natural language data. In order to
incorporate additional conditions effectively, this study employs three
distinct designs as referred from [58, 18]:

  * •

_In-context conditioning._ A straightforward strategy of appending the vector
embeddings of timestep $t$ and condition c as two supplementary tokens within
the input sequence. These tokens are treated on par with the image tokens.
Implementing this technique allows for the utilization of Bi-RWKV blocks
without requiring any adjustments. Note that the conditioning tokens are
removed from the sequence in the spatial mix module in each Bi-RWKV block and
after the final block.

  * •

_Adaptive layer norm (adaLN) block._ We explore replacing the standard norm
layer with adaptive norm layer. Rather than directly learning scale and shift
parameters on a per-dimension basis, these parameters are deduced from the
summation of the embedding vectors of $t$ and c.

  * •

_adaLN-Zero block._ In addition to regressing $\gamma$ and $\beta$, we also
regress dimension-wise scaling parameters $\alpha$ that are applied
immediately prior to any residual connections within the Bi-RWKV block. The
MLP is initialized to produce a zero-vector output for all $\alpha$.

###  2.3 Computation Analysis

In summary, the hyper-parameters of the Diffusion-RWKV model encompass crucial
components including embedding dimension $E$, hidden dimension $D$ in linear
projection, and depth $L$. Central to the bi-directional RWKV block’s
architecture is the generation of attention results for each token through
individual update steps, culminating in the requisite $T$ steps for the
complete processing of the WKV matrix. Here, $T$ is the sequence length.
Considering the input $K$ and $V$ are matrices with the shape of $J\times D$,
where $D$ is the dimension of hidden learnable vectors, the computational cost
of calculating the WKV matrix is given by:

$\text{FLOPs}(\text{Bi\mbox{-}WKV}(K,V))=13\times J\times D.$ (12)

Here, the number 13 is approximately from the updates of four hidden states,
the computation of the exponential, and the calculation of wkvt matrix. $J$ is
the total number of update steps and is equal to the number of image tokens.
The above approximation shows that the complexity of the forward process is
$O(J\cdot D)$. The backward propagation of the operator can still be
represented as a more complex RNN form, with a computational complexity of
$O(J\cdot D)$. It demonstrates a superiority of linear increasing compared
with self-attention operation in Transformer structure. Finally, the different
model variants are specified in Table 1. In between, we use five configs, from
small to huge, to cover a wide range of model sizes and flop allocations, from
1.72 to 34.95 Gflops, allowing us to gauge scaling performance.

##  3 Experiments

In this section, we shall delve into the intricacies of the design space and
thoroughly examine the scaling properties inherent in our Diffusion-RWKV model
class. In order to simplify our discourse, each model within our class is
denoted by its specific configurations and patch size $p$. As an example, we
refer to the Large version configuration with $p=2$ as DRWKV-L/2.

###  3.1 Experimental Settings

#### Datasets.

For unconditional image generation, two datasets are considered: CIFAR10 [40]
and CelebA 64x64 [46]. CIFAR10 comprises a collection of 50k training images,
while CelebA 64x64 encompasses 162,770 images depicting human faces. As for
the class-conditional image generation, the ImageNet dataset [5] is employed.
This dataset consists of 1,281,167 training images, distributed across 1,000
distinct classes. In terms of data augmentation, only horizontal flips are
employed. The training process involves 500k iterations on both CIFAR10 and
CelebA 64$\times$64, utilizing a batch size of 128 in pixel space. In the case
of ImageNet, two scenarios are considered as resolution of 256$\times$256 and
512$\times$512\. For the former, 500k iterations are conducted, while for the
latter, 1M iterations are performed. The batch size is set as 512 in both
cases.

#### Implementation details.

We followed the same training recipe from DiT [58] to ensure consistent
settings across all models. We choose to incorporate an exponential moving
average (EMA) of model weights with a fixed decay rate of 0.9999. All reported
results have been obtained using the EMA model. We use the AdamW optimizer
[36] without weight decay across all datasets and maintain a learning rate of
1e-4 to 3e-5 in stages. Our models are trained on the Nvidia A100 GPU. During
training on the ImageNet dataset at a resolution of 256$\times$256 and
512$\times$512, we also adopt classifier-free guidance [27] following [67] and
use an off-the-shelf pre-trained variational autoencoder (VAE) model [38] from
playground V2 provided in huggingface111https://huggingface.co/playgroundai
with corresponding settings. The VAE encoder component incorporates a
downsampling factor of 8. We maintain the diffusion hyperparameters from [58],
employing a $t_{max}=1000$ linear variance schedule ranging from $1\times
10^{-4}$ to $2\times 10^{-2}$ and parameterization of the covariance. In order
to adapt the model to an unconditional context, we just removed the class
label embedding component.

#### Evaluation metrics.

The performance evaluation of image generation is conducted using the Fréchet
Inception Distance (FID) [26], an extensively employed metric for assessing
the quality of generated images. In accordance with established conventions
for comparative analysis with previous works, we present FID-50K results
obtained through 250 DDPM sampling steps [56], following [7]. Furthermore, we
provide supplementary metrics such as the Inception Score [69], sFID [54], and
Precision/Recall [41] to complement the evaluation.

###  3.2 Model Analysis

We first conduct a systematical empirical investigation into the fundamental
components of Diffusion-RWKV models. Specifically, we ablate on the CIFAR10
dataset, evaluate the FID score every 50K training iterations on 10K generated
samples, instead of 50K samples for efficiency [1], and determine the optimal
default implementation details.

#### Effect of patch size.

We train patch size range over (8, 4, 2) in Small configuration on the CIFAR10
dataset. The results obtained from this experimentation are illustrated in
Figure 3 (a). It indicates that the FID metric exhibits fluctuations in
response to a decrease in patch size while maintaining a consistent model
size. Throughout the training process, we observed discernible improvements in
FID values by augmenting the number of tokens processed by Diffusion-RWKV,
while keeping the parameters approximately fixed. This observation leads us to
the conclusion that achieving optimal performance requires a smaller patch
size, specifically 2. We hypothesize that this requirement arises from the
inherently low-level nature of the noise prediction task in diffusion models.
It appears that smaller patches are more suitable for this task compared to
higher-level tasks such as classification.

#### Effect of long skip.

To assess the efficacy of the skipping operation, we investigate three
different variants, namely: (i) Concatenation, denoted as
Linear(Concat($h_{shallow}$, $h_{deep}$)); (ii) Addition, represented by
$h_{shallow}+h_{deep}$; and (iii) No skip connection. Figure 3 (b) illustrates
the outcomes of these variants. It is evident that directly adding the hidden
states from the shallow and deep layers does not yield any discernible
benefits. Conversely, the adoption of concatenation entails a learnable linear
projection on the shallow hidden states and effectively enhances performance
in comparison to the absence of a long skip connection.

Model | #Params |  FID$\downarrow$
---|---|---
DDPM [28] | 36M | 3.17
EDM [34] | 56M | 1.97
GenViT [86] | 11M | 20.20
U-ViT-S/2 [1] | 44M | 3.11
DiS-S/2 [18] | 28M | 3.25
DRWKV-S/2 | 39M | 3.03
Table 2: Benchmarking unconditional image generation on CIFAR10. Diffusion-RWKV-S/2 model obtains comparable results with fewer parameters.  Model | #Params |  FID$\downarrow$
---|---|---
DDIM [74] | 79M | 3.26
Soft Trunc. [35] | 62M | 1.90
U-ViT-S/4 [1] | 44M | 2.87
DiS-S/2 [18] | 28M | 2.05
DRWKV-S/2 | 39M | 1.92
Table 3: Benchmarking unconditional image generation on CelebA 64$\times$64.
Diffusion-RWKV-S/2 maintains a superior generation performance in small model
settings.

#### Effect of condition combination.

We examine three approaches for incorporating the conditional timestep $t$
into the network, as discussed in the preceding method section. The
integration methods are depicted in Figure 3 (c). Among these strategies, the
adaLN-Zero block exhibits a lower FID compared to the in-context conditioning
approach, while also demonstrating superior computational efficiency.
Specifically, after 500k training iterations, the adaLN-Zero model achieves an
FID that is approximately one-third of that obtained by the in-context model,
underscoring the critical influence of the conditioning mechanism on the
overall quality of the model. Furthermore, it should be noted that the
initialization process holds significance in this context. Additionally, it is
worth mentioning that due to the inclusion of a resize operation in the design
of the Bi-RWKV in spatial channel mix, only the in-context token is provided
to the channel mix module.

#### Scaling model size.

We investigate scaling properties of Diffusion-RWKV by studying the effect of
depth, _i.e._ , number of Bi-RWKV layers, and width, e.g. the hidden size.
Specifically, we train 5 Diffusion-RWKV models on the ImageNet dataset with a
resolution of 256$\times$256, spanning model configurations from small to huge
as detailed in Table 1, denoted as (S, B, M, L, H) for simple. As depicted in
Figure 3 (d), the performance improves as the depth increases from 25 to 49.
Similarly, increasing the width from 384 to 1024 yields performance gains.
Overall, across all five configurations, we find that similar to DiT models
[58], large models use FLOPs more efficient and scaling the DRWKV will improve
the FID at all stages of training.

###  3.3 Main Results

Model |  FID$\downarrow$ |  sFID$\downarrow$ |  IS$\uparrow$ |  Precision$\uparrow$ |  Recall$\uparrow$
---|---|---|---|---|---
BigGAN-deep [2] | 6.95 | 7.36 | 171.4 | 0.87 | 0.28
StyleGAN-XL [70] | 2.30 | 4.02 | 265.12 | 0.78 | 0.53
ADM [7] | 10.94 | 6.02 | 100.98 | 0.69 | 0.63
ADM-U | 7.49 | 5.13 | 127.49 | 0.72 | 0.63
ADM-G | 4.59 | 5.25 | 186.70 | 0.82 | 0.52
ADM-G, ADM-U | 3.94 | 6.14 | 215.84 | 0.83 | 0.53
CDM [29] | 4.88 | - | 158.71 | - | -
LDM-8 [67] | 15.51 | - | 79.03 | 0.65 | 0.63
LDM-8-G | 7.76 | - | 209.52 | 0.84 | 0.35
LDM-4 | 10.56 | - | 103.49 | 0.71 | 0.62
LDM-4-G | 3.60 | - | 247.67 | 0.87 | 0.48
VDM++ [37] | 2.12 | - | 267.70 | - | -
U-ViT-H/2 [1] | 2.29 | 5.68 | 263.88 | 0.82 | 0.57
DiT-XL/2 [58] | 2.27 | 4.60 | 278.24 | 0.83 | 0.57
SiT-XL/2 [50] | 2.06 | 4.50 | 270.27 | 0.82 | 0.59
DiffuSSM-XL/2 [84] | 2.28 | 4.60 | 278.24 | 0.83 | 0.57
DiS-H/2 [18] | 2.10 | 4.55 | 271.32 | 0.82 | 0.58
DRWKV-H/2 | 2.16 | 4.58 | 275.36 | 0.83 | 0.58

Table 4: Benchmarking class-conditional image generation on ImageNet
256$\times$256. Diffusion-RWKV-H/2 achieves state-of-the-art FID metrics
towards best competitors.

We compare to a set of previous best models, includes: GAN-style approaches
that previously achieved state-of-the-art results, UNet-architectures trained
with pixel space representations, and Transformers and state space models
operating in the latent space. Note that our aim is to compare, through a
similar denoising process, the performance of our model with respect to other
baselines.

#### Unconditional image generation.

We evaluate the unconditional image generation capability of our model in
relation to established baselines using the CIFAR10 and CelebA datasets within
the pixel-based domain. The outcomes of our analysis are presented in Table 2
and Table 3, respectively. The results reveal that our proposed model,
Diffusion-RWKV, attains FID scores comparable to those achieved by
Transformer-based U-ViT and SSM-based DiS models, while utilizing a similar
training budget. Notably, our model achieves this with fewer parameters and
exhibits superior FID scores. These findings emphasize the practicality and
effectiveness of RWKV across various image generation benchmarks.

#### Class-conditional image generation.

We also compare the Diffusion-RWKV model with state-of-the-art class-
conditional models in the ImageNet dataset, as listed in Table 4 and Table 5.
When considering a resolution of 256, the training of our DRWKV model exhibits
a 25% reduction in Total Gflops compared to the DiT (1.60$\times$1011 vs.
2.13$\times$1011). Additionally, our models achieve similar sFID scores to
other DDPM-based models, outperforming most state-of-the-art strategies except
for SiT and DiS. This demonstrates that the images generated by the Diffusion-
RWKV model are resilient to spatial distortion. Furthermore, in terms of FID
score, Diffusion-RWKV maintains a relatively small gap compared to the best
competitor. It is noteworthy that SiT is a transformer-based architecture that
employs an advanced strategy, which could also be incorporated into our
backbone. However, this aspect is left for future research, as our primary
focus lies in comparing our model against DiT. Moreover, we extend our
comparison to a higher-resolution benchmark of size 512. The results obtained
from the Diffusion-RWKV model demonstrate a relatively strong performance,
approaching that of some state-of-the-art high-resolution models. Our model
outperforms all models except for DiS, while achieving comparable FID scores
with a lower computational burden.

###  3.4 Case Study

In Figure 1 and Figure 4, a curated selection of samples from the ImageNet
datasets is presented. These samples are showcased at resolutions of
256$\times$256 and 512$\times$512, effectively illustrating clear semantic
representations and exhibiting high-quality generation. To delve deeper into
this topic, the project page offers a collection of additional generated
samples, encompassing both class-conditional and random variations.

Model |  FID$\downarrow$ |  sFID$\downarrow$ |  IS$\uparrow$ |  Precision$\uparrow$ |  Recall$\uparrow$
---|---|---|---|---|---
BigGAN-deep [2] | 8.43 | 8.13 | 177.90 | 0.88 | 0.29
StyleGAN-XL [70] | 2.41 | 4.06 | 267.75 | 0.77 | 0.52
ADM [7] | 23.24 | 10.19 | 58.06 | 0.73 | 0.60
ADM-U | 9.96 | 5.62 | 121.78 | 0.75 | 0.64
ADM-G | 7.72 | 6.57 | 172.71 | 0.87 | 0.42
ADM-G, ADM-U | 3.85 | 5.86 | 221.72 | 0.84 | 0.53
VDM++ [37] | 2.65 | - | 278.10 | - | -
U-ViT-H/4 [1] | 4.05 | 6.44 | 263.79 | 0.84 | 0.48
DiT-XL/2 [58] | 3.04 | 5.02 | 240.82 | 0.84 | 0.54
DiffuSSM-XL/2 [84] | 3.41 | 5.84 | 255.06 | 0.85 | 0.49
DiS-H/2 [18] | 2.88 | 4.74 | 272.33 | 0.84 | 0.56
DRWKV-H/2 | 2.95 | 4.95 | 265.20 | 0.84 | 0.54

Table 5: Benchmarking class-conditional image generation on ImageNet
512$\times$512. DRWKV-H/2 demonstrates a promising performance compared with
both CNN-based and Transformer-based UNet for diffusion. Figure 4: Image
results generated from Diffusion-RWKV model.  Selected samples on ImageNet
512$\times$512 with sample classes and different seeds. We can see that
Diffusion-RWKV can generate high-quality images while keeping integrated
condition alignment.

##  4 Related Works

#### Image generation with diffusion.

Diffusion and score-based generative models [33, 75, 76, 77] have demonstrated
significant advancements in various tasks, particularly in the context of
image generation [66, 67, 68]. The DDPM has been primarily attributed to
improvements in sampling techniques [28, 34, 55, 16, 17], and the
incorporation of classifier-free guidance [27]. Additionally, [74] introduced
a more efficient sampling procedure called Denoising Diffusion Implicit Model
(DDIM). Latent space modeling is another core technique in deep generative
models. Variational autoencoders [38] pioneered learning latent spaces with
encoder-decoder architectures for reconstruction. The concept of compressing
information in latent spaces was also adopted in diffusion models, as
exemplified by the state-of-the-art sample quality achieved by latent
diffusion models [67], which train deep generative models to reverse a noise
corruption process within a latent space. Additionally, recent advancements
have incorporated masked training procedures, enhancing denoising training
objectives through masked token reconstruction [88]. Our work is fundamentally
built upon existing standard DDPMs.

#### Architectures for diffusion models.

Early models for diffusion employed U-Net style architectures [7, 28].
Subsequent studies endeavored to enhance U-Nets by incorporating various
techniques, such as the addition of attention layers at multiple scales [55],
residual connections [2], and normalization [60, 83]. However, U-Nets
encounter difficulties when scaling to high resolutions due to the escalating
computational demands imposed by the attention mechanism [71]. Recently,
vision transformers [10] have emerged as an alternative architecture,
showcasing their robust scalability and long-range modeling capabilities,
thereby challenging the notion that convolutional inductive bias is always
indispensable. Diffusion transformers [1, 58, 19] demonstrated promising
results. Other hybrid CNN-transformer architectures were proposed [47] to
improve training stability. More recently, state space-based model [31, 18,
84] have obtain a advanced performance with computation efficiency. Our work
aligns with the exploration of recurrent sequence models and the associated
design choices for generating high-quality images while mitigating text
similarity.

#### Efficient long sequence modeling.

The standard transformer architecture employs attention to comprehend the
interplay between individual tokens. However, it faces challenges when dealing
with lengthy sequences, primarily due to the quadratic computational
complexity it entails. To address this issue, various attention approximation
methods have been proposed [32, 51, 72, 79, 82, 13], which aim to approximate
self-attention while utilizing sub-quadratic computational resources. Notably,
Mega [52] combines exponential moving average with a simplified attention
unit, surpassing the performance of the baseline transformer models. What’s
more, researchers have also explored alternatives that are capable of
effectively handling long sequences. One involves employing state space
models-based architectures, as exemplified by [23, 22, 24], which have
demonstrated significant advancements over contemporary state-of-the-art
methods in tasks such as LRA and audio benchmarking [20]. Moreover, recent
studies [22, 59, 61, 62] have provided empirical evidence supporting the
potential of non-attention architectures in achieving commendable performance
in language modeling. Motivated by this evolving trend of recurrence designs,
our work draws inspiration from these advancements and predominantly leverages
the backbone of RWKV.

##  5 Conclusion

This paper presents Diffusion-RWKV, an architecture designed for diffusion
models featuring sequential information with linear computational complexity.
The proposed approach effectively handles long-range hidden states without
necessitating representation compression. Through comprehensive image
generation tasks, we showcase its potential as a viable alternative backbone
to the Transformer. Experimentally, Diffusion-RWKV demonstrates comparable
performance and scalability while exhibiting lower computational complexity
and memory consumption. Leveraging its reduced complexity, Diffusion-RWKV
outperforms the Transformer model in scenarios where the latter struggles to
cope with high computational demands. We anticipate that it will serve as an
efficient and cost-effective substitute for the Transformer, thereby
highlighting the substantial capabilities of transformers with linear
complexity in the realm of multimodal generation.

## References

  * Bao et al. [2023] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu.  All are worth words: A vit backbone for diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22669–22679, 2023.
  * Brock et al. [2018] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chang et al. [2022] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman.  Maskgit: Masked generative image transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11315–11325, 2022.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Ding et al. [2021] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al.  Cogview: Mastering text-to-image generation via transformers.  _Advances in Neural Information Processing Systems_ , 34:19822–19835, 2021.
  * Ding et al. [2022] Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang.  Cogview2: Faster and better text-to-image generation via hierarchical transformers.  _Advances in Neural Information Processing Systems_ , 35:16890–16902, 2022.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Duan et al. [2024] Yuchen Duan, Weiyun Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Hongsheng Li, Jifeng Dai, and Wenhai Wang.  Vision-rwkv: Efficient and scalable visual perception with rwkv-like architectures.  _arXiv preprint arXiv:2403.02308_ , 2024.
  * Fei [2021] Zhengcong Fei.  Partially non-autoregressive image captioning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 1309–1316, 2021.
  * Fei [2022] Zhengcong Fei.  Attention-aligned transformer for image captioning.  In _proceedings of the AAAI Conference on Artificial Intelligence_ , pages 607–615, 2022.
  * Fei et al. [2022a] Zhengcong Fei, Mingyuan Fan, Li Zhu, and Junshi Huang.  Progressive text-to-image generation.  _arXiv preprint arXiv:2210.02291_ , 2022a.
  * Fei et al. [2022b] Zhengcong Fei, Xu Yan, Shuhui Wang, and Qi Tian.  Deecap: Dynamic early exiting for efficient image captioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12216–12226, 2022b.
  * Fei et al. [2023a] Zhengcong Fei, Mingyuan Fan, and Junshi Huang.  Gradient-free textual inversion.  In _Proceedings of the 31st ACM International Conference on Multimedia_ , pages 1364–1373, 2023a.
  * Fei et al. [2023b] Zhengcong Fei, Mingyuan Fan, Li Zhu, Junshi Huang, Xiaoming Wei, and Xiaolin Wei.  Uncertainty-aware image captioning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 614–622, 2023b.
  * Fei et al. [2024] Zhengcong Fei, Mingyuan Fan, Changqian Yu, and Junshi Huang.  Scalable diffusion models with state space backbone.  _arXiv preprint arXiv:2402.05608_ , 2024.
  * Fei [2019] Zheng-cong Fei.  Fast image caption generation with position alignment.  _arXiv preprint arXiv:1912.06365_ , 2019.
  * Goel et al. [2022] Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré.  It’s raw! audio generation with state-space models.  In _International Conference on Machine Learning_ , pages 7616–7633. PMLR, 2022.
  * Gu and Dao [2023] Albert Gu and Tri Dao.  Mamba: Linear-time sequence modeling with selective state spaces.  _arXiv preprint arXiv:2312.00752_ , 2023.
  * Gu et al. [2020] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré.  Hippo: Recurrent memory with optimal polynomial projections.  _Advances in neural information processing systems_ , 33:1474–1487, 2020.
  * Gu et al. [2021] Albert Gu, Karan Goel, and Christopher Ré.  Efficiently modeling long sequences with structured state spaces.  _arXiv preprint arXiv:2111.00396_ , 2021.
  * Gupta et al. [2022] Ankit Gupta, Albert Gu, and Jonathan Berant.  Diagonal state spaces are as effective as structured state spaces.  _Advances in Neural Information Processing Systems_ , 35:22982–22994, 2022.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 16000–16009, 2022.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Ho et al. [2022] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  _The Journal of Machine Learning Research_ , 23(1):2249–2281, 2022.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Hu et al. [2024] Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Fischer, and Bjorn Ommer.  Zigma: Zigzag mamba diffusion model.  _arXiv preprint arXiv:2403.13802_ , 2024.
  * Hua et al. [2022] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le.  Transformer quality in linear time.  In _International conference on machine learning_ , pages 9099–9117. PMLR, 2022.
  * Hyvärinen and Dayan [2005] Aapo Hyvärinen and Peter Dayan.  Estimation of non-normalized statistical models by score matching.  _Journal of Machine Learning Research_ , 6(4), 2005.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _Advances in Neural Information Processing Systems_ , 35:26565–26577, 2022.
  * Kim et al. [2021] Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon.  Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation.  _arXiv preprint arXiv:2106.05527_ , 2021.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kingma and Gao [2023] Diederik P Kingma and Ruiqi Gao.  Understanding diffusion objectives as the elbo with simple data augmentation.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kitaev et al. [2020] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  _arXiv preprint arXiv:2001.04451_ , 2020.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Kynkäänniemi et al. [2019] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Improved precision and recall metric for assessing generative models.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Lee et al. [2022] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han.  Autoregressive image generation using residual quantization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11523–11532, 2022.
  * Lewis et al. [2019] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ , 2019.
  * Lin et al. [2022] Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu.  A survey of transformers.  _AI open_ , 3:111–132, 2022.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Liu et al. [2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _Proceedings of the IEEE international conference on computer vision_ , pages 3730–3738, 2015.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021.
  * Lu et al. [2022a] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _Advances in Neural Information Processing Systems_ , 35:5775–5787, 2022a.
  * Lu et al. [2022b] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.  _arXiv preprint arXiv:2211.01095_ , 2022b.
  * Ma et al. [2024] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie.  Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers.  _arXiv preprint arXiv:2401.08740_ , 2024.
  * Ma et al. [2021] Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer.  Luna: Linear unified nested attention.  _Advances in Neural Information Processing Systems_ , 34:2441–2453, 2021.
  * Ma et al. [2022] Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer.  Mega: moving average equipped gated attention.  _arXiv preprint arXiv:2209.10655_ , 2022.
  * Mann et al. [2020] Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ , 2020.
  * Nash et al. [2021] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia.  Generating images with sparse representations.  _arXiv preprint arXiv:2103.03841_ , 2021.
  * Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pages 8162–8171. PMLR, 2021.
  * Parmar et al. [2022] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu.  On aliased resizing and surprising subtleties in gan evaluation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11410–11420, 2022.
  * Parmar et al. [2018] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran.  Image transformer.  In _International conference on machine learning_ , pages 4055–4064. PMLR, 2018.
  * Peebles and Xie [2023] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4195–4205, 2023.
  * Peng et al. [2023] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al.  Rwkv: Reinventing rnns for the transformer era.  _arXiv preprint arXiv:2305.13048_ , 2023.
  * Perez et al. [2018] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville.  Film: Visual reasoning with a general conditioning layer.  In _Proceedings of the AAAI conference on artificial intelligence_ , 2018.
  * Poli et al. [2023] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré.  Hyena hierarchy: Towards larger convolutional language models.  In _International Conference on Machine Learning_ , pages 28043–28078. PMLR, 2023.
  * Qin et al. [2024] Zhen Qin, Songlin Yang, and Yiran Zhong.  Hierarchically gated recurrent neural network for sequence modeling.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International conference on machine learning_ , pages 8821–8831. Pmlr, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Salimans et al. [2016] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  _Advances in neural information processing systems_ , 29, 2016.
  * Sauer et al. [2022] Axel Sauer, Katja Schwarz, and Andreas Geiger.  Stylegan-xl: Scaling stylegan to large diverse datasets.  In _ACM SIGGRAPH 2022 conference proceedings_ , pages 1–10, 2022.
  * Shaham et al. [2018] Uri Shaham, Kelly Stanton, Henry Li, Boaz Nadler, Ronen Basri, and Yuval Kluger.  Spectralnet: Spectral clustering using deep neural networks.  _arXiv preprint arXiv:1801.01587_ , 2018.
  * Shen et al. [2021] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li.  Efficient attention: Attention with linear complexities.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , pages 3531–3539, 2021.
  * Smith et al. [2022] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al.  Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model.  _arXiv preprint arXiv:2201.11990_ , 2022.
  * Song et al. [2020a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ , 32, 2019.
  * Song et al. [2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Song et al. [2023] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  _arXiv preprint arXiv:2303.01469_ , 2023.
  * Stickland and Murray [2019] Asa Cooper Stickland and Iain Murray.  Bert and pals: Projected attention layers for efficient adaptation in multi-task learning.  In _International Conference on Machine Learning_ , pages 5986–5995. PMLR, 2019.
  * Tay et al. [2020] Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan.  Sparse sinkhorn attention.  In _International Conference on Machine Learning_ , pages 9438–9447. PMLR, 2020.
  * Tay et al. [2022] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler.  Efficient transformers: A survey.  _ACM Computing Surveys_ , 55(6):1–28, 2022.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2020] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-attention with linear complexity.  _arXiv preprint arXiv:2006.04768_ , 2020.
  * Wu and He [2018] Yuxin Wu and Kaiming He.  Group normalization.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 3–19, 2018.
  * Yan et al. [2023] Jing Nathan Yan, Jiatao Gu, and Alexander M Rush.  Diffusion models without attention.  _arXiv preprint arXiv:2311.18257_ , 2023.
  * Yan et al. [2021] Xu Yan, Zhengcong Fei, Zekang Li, Shuhui Wang, Qingming Huang, and Qi Tian.  Semi-autoregressive image captioning.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pages 2708–2716, 2021.
  * Yang et al. [2022] Xiulong Yang, Sheng-Min Shih, Yinlin Fu, Xiaoting Zhao, and Shihao Ji.  Your vit is secretly a hybrid discriminative-generative diffusion model.  _arXiv preprint arXiv:2208.07791_ , 2022.
  * Zamir et al. [2022] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang.  Restormer: Efficient transformer for high-resolution image restoration.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5728–5739, 2022.
  * Zheng et al. [2023] Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar.  Fast training of diffusion models with masked transformers.  _arXiv preprint arXiv:2306.09305_ , 2023.
  * Zhou et al. [2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang.  Informer: Beyond efficient transformer for long sequence time-series forecasting.  In _Proceedings of the AAAI conference on artificial intelligence_ , pages 11106–11115, 2021.
  * Zhu et al. [2024] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang.  Vision mamba: Efficient visual representation learning with bidirectional state space model.  _arXiv preprint arXiv:2401.09417_ , 2024.


# Nucleation at finite temperature: a gauge-invariant, perturbative framework

Johan Löfgren<EMAIL_ADDRESS>Department of Physics and
Astronomy, Uppsala University, Box 516, SE-751 20 Uppsala, Sweden  Michael J.
Ramsey-Musolf<EMAIL_ADDRESS><EMAIL_ADDRESS>Amherst Center for
Fundamental Interactions, Department of Physics, University of Massachusetts,
Amherst, MA 01003, USA  Kellogg Radiation Laboratory, California Institute of
Technology, Pasadena, CA 91125 USA  Tsung-Dao Lee Institute and School of
Physics and Astronomy, Shanghai Jiao Tong University, 800 Dongchuan Road,
Shanghai, 200240 China  Shanghai Key Laboratory for Particle Physics and
Cosmology, Key Laboratory for Particle Astrophysics & Cosmology (MOE),
Shanghai Jiao Tong University, Shanghai 200240, China  Philipp Schicho
<EMAIL_ADDRESS>Department of Physics and Helsinki Institute of
Physics, PL 64, FI-00014 University of Helsinki, Finland  Tuomas V. I.
Tenkanen<EMAIL_ADDRESS>Tsung-Dao Lee Institute and School of Physics
and Astronomy, Shanghai Jiao Tong University, 800 Dongchuan Road, Shanghai,
200240 China  Shanghai Key Laboratory for Particle Physics and Cosmology, Key
Laboratory for Particle Astrophysics & Cosmology (MOE), Shanghai Jiao Tong
University, Shanghai 200240, China  Nordita, KTH Royal Institute of Technology
and Stockholm University, Roslagstullsbacken 23, SE-106 91 Stockholm, Sweden

###### Abstract

We present a gauge-invariant framework for bubble nucleation in theories with
radiative symmetry breaking at high temperature. As a procedure, this
perturbative framework establishes a practical, gauge-invariant computation of
the leading order nucleation rate, based on a consistent power counting in the
high-temperature expansion. In model building and particle phenomenology, this
framework has applications such as the computation of the bubble nucleation
temperature and the rate for electroweak baryogenesis and gravitational wave
signals from cosmic phase transitions.

††preprint: ACFI-T21-15††preprint: HIP-2021-44/TH††preprint: NORDITA 2021-110

Achieving a rigorous understanding of the thermal history of the electroweak
(EW) symmetry breaking has been a long-standing challenge at the interface
between particle physics and cosmology. The standard history of the Standard
Model (SM) of particle physics contains no electroweak phase transition.
Instead a smooth crossover between high and low temperature phases occurs [1,
2]. A non-standard history with a first order phase transition is conceivable
in theories beyond the SM (BSM) that are well motivated from the electroweak
to TeV scale. BSM scenarios with extended scalar sectors can contain rich
patterns of symmetry breaking [3] and potentially produce EW baryogenesis [4,
5, 6], and also rich phenomenology for high energy collider physics [7] and
production of primordial gravitational waves [8]. These phenomena have been
actively studied for decades and recently sparked increased interest in next-
generation gravitational wave detector experiments [9, 10, 11] such as LISA
[12].

One key ingredient for both EW baryogenesis and stochastic gravitational wave
background production is a first order phase transition in the primordial
plasma of particles that proceeds via nucleating bubbles of the low-
temperature phase. The thermo- and bubble dynamics of such transitions can be
reliably described non-perturbatively using lattice simulations [13, 14, 15].
However, exploring BSM scenarios and their thermodynamics invariably requires
the use of perturbation theory, since fully comprehensive non-perturbative
analyses of a multidimensional parameter space are computationally out of
reach. A qualitative and quantitative assessment of the perturbative
reliability requires robust theory computations when comparing to lattice
data. In that respect, an unphysical gauge-dependence of the bubble nucleation
rate has plagued particle physics phenomenology applications of the thermal
phase transition literature for decades (c.f. [16, 8] and references therein).
While by now resolved at zero temperature [17, 18, 19, 20, 21, 22], a similar,
long-standing and open problem exists at finite temperature if the potential
barrier between the phases is radiatively generated.

We resolve this long-standing problem by providing a practical, gauge-
invariant framework for thermal bubble nucleation, intended for model-building
and particle phenomenology applications. The reliability of the approach can
be assessed by “benchmarking”, against lattice analyses in the future. Such
benchmarking is incompatible with the conventional approach since there the
nucleation rate is gauge-dependent and an ill-defined unphysical quantity.

The bubble nucleation rate per unit volume, $\Gamma$, has the semiclassical
approximation [23, 24, 25, 26]

$\displaystyle\Gamma$ $\displaystyle=Ae^{-\mathcal{B}}\;.$ (1)

Here, the exponent $\mathcal{B}=S_{3}/T$ with $S_{3}$ being the three-
dimensional Euclidean effective action evaluated at the “bounce” solution that
solves the classical Euclidean field equations [24, 27]. The prefactor $A$ is
dimensionful and given by the characteristic mass scales of the theory,
resulting from computing the functional determinants. The leading behaviour of
the rate $\Gamma$ is encoded in the exponent $\mathcal{B}$.

The effective action is computable using the background field ($\phi$) method.
Therein, $V^{{\mbox{\scriptsize eff}}}(\phi,T)$ is the thermal effective
potential describing the equilibrium free energy of the system and
$Z^{\frac{1}{2}}(\phi,T)$ is the field renormalization factor. Both
$V^{{\mbox{\scriptsize eff}}}(\phi,T)$ and $Z(\phi,T)$ admit an expansion in
the weak gauge coupling, denoted generically here as $g$. After the gradient
expansion in powers of spatial derivatives $(\partial_{i}\phi)$, the effective
action reads [28, 29]

$S_{3}=\int{\rm d}^{3}x\Bigl{[}V^{{\mbox{\scriptsize
eff}}}(\phi,T)+\frac{1}{2}Z(\phi,T)\left(\partial_{i}\phi\right)^{2}+\dots\Bigr{]}\;,$
(2)

where the ellipsis contains terms of additional $(\partial_{i}\phi)$-powers.
While the gradient expansion in general does not converge [15, 30, 31, 32],
one may obtain a self-consistent, gauge-invariant estimate of $\Gamma$ by
ensuring the two following conditions apply

  * (A):

Fields that generate the barrier must be parametrically heavier than the
nucleating field. In this case leading orders can be correctly described by a
derivative expansion,

  * (B):

Considering temperatures in the vicinity of the critical temperature, $T_{\rm
c}$, such that a specific power counting in $g$ applies, wherein the leading
order potential has a radiatively generated barrier.

Hence, we focus on a temperature regime where the first two leading terms of
the effective action are enhanced by powers of $g$ and presentable
schematically as

$\displaystyle S_{3}$
$\displaystyle=a_{0}g^{-\frac{3}{2}}+a_{1}g^{-\frac{1}{2}}+\Delta\;.$ (3)

Here $a_{0,1}$ are numerical coefficients that are computable in the gradient
expansion at leading (LO) and next-to-leading orders (NLO), respectively.
$\Delta$ encodes formally higher order corrections – that are
$\mathcal{O}({1})$, up to potential logarithms of $g$ [31] – that we presently
do not compute. The bubble nucleation rate then reads

$\displaystyle\Gamma$
$\displaystyle=A^{\prime}e^{-\left(a_{0}g^{-\frac{3}{2}}+a_{1}g^{-\frac{1}{2}}\right)}\;.$
(4)

We formally collected higher order effects encoded in $\Delta$ in Eq. (3), to
a yet unspecified prefactor $A^{\prime}\equiv Ae^{-\Delta}$. The expression
(4) results from an expansion based on the chain of scale hierarchies at high
temperature

$\displaystyle\pi T\gg gT\gg g^{\frac{3}{2}}T\;,$ (5)

where $\pi T$ is the thermal scale of non-zero Matsubara modes, $gT$ is the
intermediate scale of zero modes that are thermally screened, and
$g^{\frac{3}{2}}T$ is a characteristic scale of bubble nucleation, related to
an effective mass of the nucleating field. For a systematic treatment of
thermodynamics in the framework of three-dimensional, high temperature
effective field theory (3d EFT) see [33, 34]. This framework was recently
extended [31] by presenting an approach for an effective description for
bubble nucleation.

Here, we work at the leading high temperature expansion [35] that allows us to
perform the computation without formally constructing a 3d EFT. Our goal is to
compute LO and NLO terms in $\mathcal{B}$ and show their gauge invariance. We
emphasize that this computation agrees with the generic EFT approach [31] at
leading orders, and an accompanying article [32] revisits this computation in
the EFT context, which allows to pursue improvement in terms of higher order
corrections.

As in earlier literature both at zero [17, 36] and high [30] temperature, we
use the Abelian Higgs model as a concrete illustration. The Euclidean
Lagrangian density for the Abelian Higgs model is

$\displaystyle\mathcal{L}$
$\displaystyle=\frac{1}{4}F_{\mu\nu}F_{\mu\nu}+(D_{\mu}\Phi)^{*}(D_{\mu}\Phi)$
$\displaystyle+\mu^{2}\Phi^{*}\Phi+\lambda(\Phi^{*}\Phi)^{2}+\mathcal{L}_{{\mbox{\tiny\rm{GF}}}}+\mathcal{L}_{{\mbox{\tiny\rm{FP}}}}\;,$

with a U(1) gauge field $B_{\mu}$ and a complex scalar field $\Phi$. The
covariant derivative for complex Higgs reads $D_{\mu}\Phi=\partial_{\mu}\Phi-
igB_{\mu}\Phi$, where $g$ is the gauge coupling and the field strength tensor
$F_{\mu\nu}=\partial_{\mu}B_{\nu}-\partial_{\nu}B_{\mu}$. We expand $\Phi$ in
real fields

$\Phi=\frac{\phi}{\sqrt{2}}+\frac{1}{\sqrt{2}}\bigl{(}H+i\chi\bigr{)}\;,$ (6)

where $\phi\equiv\phi(x)$ is a spatially dependent classical background field
and $H$ and $\chi$ are propagating quantum degrees of freedom. In perturbation
theory the gauge can be fixed using $R_{\xi}$-gauge [37, 30]

$\displaystyle\mathcal{L}_{{\mbox{\tiny\rm{GF}}}}$
$\displaystyle=\frac{1}{2\xi}\bigl{[}-\bigl{(}\partial_{\mu}B_{\mu}+ig\xi(\tilde{\phi}^{*}\Phi-\Phi^{*}\tilde{\phi})\bigr{)}\bigr{]}^{2}\;,$
(7)

with Fadeev-Popov ghost ($c,\bar{c}$) Lagrangian

$\displaystyle\mathcal{L}_{{\mbox{\tiny\rm{FP}}}}$
$\displaystyle=\bar{c}\Big{(}-\square+\xi
g^{2}(\tilde{\phi}^{*}\Phi+\Phi^{*}\tilde{\phi})\Big{)}c\;.$ (8)

Although one need not relate $\tilde{\phi}$ directly to $\phi$, it is
convenient to identify $\tilde{\phi}=\phi/\sqrt{2}$ to eliminate mixing
between the gauge field and Goldstone mode $\chi$.

Equilibrium thermodynamics are described by the effective potential, whose
minima should be separated by a barrier for a first order phase transition.
While the Abelian Higgs model admits no tree-level barrier, the barrier arises
radiatively through loop corrections. Similar to [17, 30, 35, 38], we show
that one-loop gauge field contributions can arise at the same order as the
tree-level potential provided that model parameters assume the following power
counting. First, we assume that

$\displaystyle\phi$ $\displaystyle\sim T\;,$ (9)

which sets the size of the background field to the characteristic mass scale
of the problem, the temperature. Second, we relate the size of the scalar
self-coupling and gauge coupling by [35]

$\displaystyle\lambda$ $\displaystyle\sim g^{3}\;.$ (10)

By assuming that the quartic coupling is sufficiently small compared to gauge
coupling, the cubic term induced by the gauge field will contribute at leading
order, as seen below. Third, we assume

$\displaystyle\mu^{2}$ $\displaystyle\sim(gT)^{2}\;,$ (11)

which installs the high-temperature expansion of $\mu^{2}\ll T^{2}$. Finally,
the effective thermally corrected mass of the scalar at leading order is

$\displaystyle\mu^{2}_{{\mbox{\tiny\rm{eff}}}}\equiv\mu^{2}+\bigl{(}4\lambda+3g^{2}\bigr{)}\frac{T^{2}}{12}$
$\displaystyle\sim\mathcal{O}({g^{2+N}T^{2}})\;.$ (12)

Here, the first and third terms are individually of
$\mathcal{O}({g^{2}T^{2}})$. However, for $\mu^{2}<0$, a cancellation between
the first and remaining terms can render the overall sum in
$\mu_{{\mbox{\tiny\rm{eff}}}}^{2}$ smaller – parametrised by $N>0$ – for some
temperature range. Henceforth, we assume a temperature window where $N=1$ and
argue below that this occurs in the vicinity of the phase transition critical
temperature [35, 39].

The assumptions of Eqs. (9)–(12) induce the chain of thermal scale hierarchies
in Eq. (5). Non-zero Matsubara modes have masses $\sim\pi T$. In the unbroken
phase ($\phi=0$) zero Matsubara modes have the following scales: Higgs and
Goldstone fields have masses $\sim\mu_{{\mbox{\tiny\rm{eff}}}}\sim
g^{\frac{3}{2}}T$. At zero temperature it is sufficient to talk of transverse
and longitudinal modes of the gauge field, but at high temperatures there is a
further splitting of the three transverse modes into two spatial modes and one
temporal. The gauge field temporal mode, $B_{0}$, has a mass
$\bar{m}_{{\mbox{\tiny\rm{$B_{0}$}}}}\sim gT$; while spatial gauge fields
remain massless ($m_{{\mbox{\tiny\rm{$B$}}}}=0$). In the broken phase
($\phi>0$), spatial and temporal gauge bosons, Goldstone and ghost fields have
masses $\sim g\phi\sim gT$, while the Higgs field that undergoes nucleation
has parametrically lighter mass $\sim g^{\frac{3}{2}}T$.

The full one-loop effective potential is found using the following background
field dependent mass eigenvalues

$\displaystyle\bar{m}^{2}_{{\mbox{\tiny\rm{$H$}}}}$
$\displaystyle=\mu^{2}_{{\mbox{\tiny\rm{eff}}}}+3\lambda\phi^{2}\;,$ (13)
$\displaystyle\bar{m}^{2}_{\chi}$
$\displaystyle=\mu^{2}_{{\mbox{\tiny\rm{eff}}}}+\lambda\phi^{2}+g^{2}\xi\phi^{2}\;,$
(14) $\displaystyle\bar{m}_{{\mbox{\tiny\rm{$B_{0}$}}}}^{2}$
$\displaystyle=\frac{1}{3}g^{2}T^{2}+g^{2}\phi^{2}\;,$ (15) $\displaystyle
m^{2}_{c}$ $\displaystyle=g^{2}\xi\phi^{2}\;,\quad
m_{{\mbox{\tiny\rm{$B$}}}}^{2}=g^{2}\phi^{2}\;,$ (16)

where masses for the zero Matsubara modes of the Higgs, Goldstone and temporal
gauge field $B_{0}$ include resummed thermal corrections e.g. the $T^{2}$-term
in the third line describes Debye screening. The mass of the longitudinal
component of the gauge field equals the ghost mass. Neither spatial gauge
fields, nor ghost fields, develop thermal masses. Based on the power counting
in $g$, the leading contribution to the effective potential is of
$\mathcal{O}({g^{3}T^{4}})$ and reads

$\displaystyle V^{{\mbox{\scriptsize eff}}}_{{\mbox{\tiny\rm{LO}}}}$
$\displaystyle=\frac{1}{2}\mu^{2}_{{\mbox{\tiny\rm{eff}}}}\phi^{2}+\frac{1}{4}\lambda\phi^{4}$
$\displaystyle-\frac{g^{3}T}{12\pi}\Bigl{[}2\phi^{3}+\Bigl{(}\frac{1}{3}T^{2}+\phi^{2}\Bigr{)}^{\frac{3}{2}}\Bigr{]}\;,$
(17)

where the second line is the transverse gauge field contribution and the
second term therein corresponds to the Debye mass (15) of the temporal mode.
Near the phase transition, all terms in the potential should be approximately
of the same order of magnitude which is assured by construction, given the
assumed power counting in $g$ in Eqs. (9)–(12). The leading-order potential of
Eq. (Nucleation at finite temperature: a gauge-invariant, perturbative
framework) at $\mathcal{O}({g^{3}T^{4}})$ is gauge invariant.

The one-loop contribution to the effective potential from the longitudinal
gauge field, Goldstone field, and ghost fields is

$\displaystyle-\frac{T}{12\pi}\left(\bar{m}_{\chi}^{3}-m_{c}^{3}\right)\sim\mathcal{O}({g^{4}T^{4}})\;.$
(18)

Note that ghosts contribute with a relative minus sign to other fields. These
contributions give rise to an explicit $\xi$ dependence, but these terms are
of higher order compared to Eq. (Nucleation at finite temperature: a gauge-
invariant, perturbative framework) due to a cancellation at leading order.
Below we include the remaining $\mathcal{O}({g^{4}T^{4}})$ terms at NLO in the
effective potential which expands as

$\displaystyle V^{{\mbox{\scriptsize eff}}}$
$\displaystyle=V^{{\mbox{\scriptsize
eff}}}_{{\mbox{\tiny\rm{LO}}}}+V^{{\mbox{\scriptsize
eff}}}_{{\mbox{\tiny\rm{NLO}}}}+\mathcal{O}({g^{\frac{9}{2}}T^{4}})\;,$ (19)

where

$\displaystyle V^{{\mbox{\scriptsize eff}}}_{{\mbox{\tiny\rm{NLO}}}}$
$\displaystyle=\frac{1}{(4\pi)^{2}}\bigg{\\{}g^{4}T^{2}\phi^{2}\bigg{(}-1+\ln\Big{(}\frac{4g^{2}\phi^{2}}{\Lambda^{2}}\Big{)}\bigg{)}$
$\displaystyle+\sqrt{\xi}gT\phi\Big{(}g^{3}T\phi-2\pi(\mu^{2}_{{\mbox{\tiny\rm{eff}}}}+\lambda\phi^{2})\Big{)}$
$\displaystyle+g^{4}T^{2}\bigg{[}\frac{1}{2}\sqrt{\xi}\phi\sqrt{\frac{1}{3}T^{2}+\phi^{2}}$
$\displaystyle+\frac{1}{2}\phi^{2}\bigg{(}-1+\ln\Big{(}\frac{4g^{2}(\frac{1}{3}T^{2}+\phi^{2})}{\Lambda^{2}}\Big{)}\bigg{)}\bigg{]}\bigg{\\}}\;.$
(20)

The last term on the second line originates from the one-loop Goldstone-ghost
contribution in Eq. (18), and the last two lines correspond to the two-loop
contributions of the $B_{0}$ field. The remaining terms result from two-loop
diagrams with spatial gauge fields, ghosts and scalars ($\chi$,$H$). The
$\mathcal{O}({g^{\frac{9}{2}}T^{4}})$ term in Eq. (19) arises at one-loop
order from the Higgs field.

Consistent treatment of the effective action (2) at NLO also requires the
inclusion of field renormalization:

$\displaystyle Z$
$\displaystyle=1+Z_{{\mbox{\tiny\rm{NLO}}}}+\mathcal{O}({g^{\frac{3}{2}}})\;,$
(21)

where

$Z_{{\mbox{\tiny\rm{NLO}}}}(\phi)=\frac{gT}{48\pi}\biggl{[}-\frac{22}{\phi}+\frac{\phi^{2}}{(\frac{1}{3}T^{2}+\phi^{2})^{\frac{3}{2}}}\biggr{]}\;.$
(22)

The first term is from the two spatial modes and ghosts and the second term is
from the temporal mode. Notably, at order $\mathcal{O}({g})$ the field
renormalisation is independent of the gauge-fixing parameter. Higher order
terms at $\mathcal{O}({g^{\frac{3}{2}}})$ arise from two-loop diagrams
involving gauge, ghost and Goldstone fields, and one-loop diagrams with
internal Higgs legs.

In the semiclassical approximation, the background field extremizes the
leading-order action and can be found from the equation of motion for the
leading-order potential

$\nabla^{2}\phi_{b}(x)=\frac{\partial V^{{\mbox{\scriptsize
eff}}}_{{\mbox{\tiny\rm{LO}}}}}{\partial\phi}\biggr{|}_{\phi=\phi_{b}}\;,\quad\begin{cases}\phi_{b}(\infty)&=0\\\
\phi_{b}^{\prime}(0)&=0\end{cases}\;,$ (23)

where $\nabla^{2}\equiv\partial_{i}\partial_{i}$ is the three-dimensional
Laplacian operator, and $\phi_{b}$ is the “bounce solution” [40]. We expand
the exponent of the nucleation rate as
$\mathcal{B}=\mathcal{B}_{0}+\mathcal{B}_{1}$:

$\displaystyle\mathcal{B}_{0}$ $\displaystyle=\beta\int{\rm
d}^{3}x\left[V^{{\mbox{\scriptsize
eff}}}_{{\mbox{\tiny\rm{LO}}}}(\phi_{b})+\frac{1}{2}\left(\partial_{i}\phi_{b}\right)^{2}\right]\;,$
(24) $\displaystyle\mathcal{B}_{1}$ $\displaystyle=\beta\int{\rm
d}^{3}x\left[V^{{\mbox{\scriptsize
eff}}}_{{\mbox{\tiny\rm{NLO}}}}(\phi_{b})+\frac{1}{2}Z_{{\mbox{\tiny\rm{NLO}}}}(\phi_{b})\left(\partial_{i}\phi_{b}\right)^{2}\right]\;,$
(25)

where $\beta\equiv 1/T$. The characteristic length scale $R$ for nucleation is
related to the typical bubble size, given by the inverse mass of the
nucleating field $R\sim m_{\mbox{\tiny\rm{$H$}}}^{-1}\sim
g^{-\frac{3}{2}}T^{-1}$. This gives rise to the formal scaling $\int{\rm
d}{}^{3}x\sim g^{-\frac{9}{2}}T^{-3}$. Together with the power counting for
the effective potential and field renormalisation, this establishes the
relative importance of the first two leading exponent terms of the nucleation
rate: $\mathcal{B}_{0}\sim g^{-\frac{3}{2}}$, $\mathcal{B}_{1}\sim
g^{-\frac{1}{2}}$ as already foreshadowed in Eqs. (3) and (4). Despite the
$1/\phi$ behaviour of $Z_{{\mbox{\tiny\rm{NLO}}}}(\phi_{b})$, the contribution
of the term
$Z_{{\mbox{\tiny\rm{NLO}}}}(\phi_{b})\left(\partial_{i}\phi_{b}\right)^{2}$ is
finite also in the region of vanishing $\phi_{b}$ [32] (c.f. also [41, 42,
31]).

Because the bounce solution has to be solved numerically from Eq. (23), the
exponents $\mathcal{B}_{0,1}$ are necessarily obtained by numerical
integration. Nevertheless, their gauge-independence can still be proven
analytically. The gauge-fixing parameter is absent at leading order since $Z$
and $V^{\mathrm{eff}}$ are both $\xi$-independent at this order, implying
gauge-independence of both the bounce solution $\phi_{b}$ and exponent
$\mathcal{B}_{0}$. The gauge-invariance of $\mathcal{B}_{1}$ is not
immediately obvious since $V^{{\mbox{\scriptsize
eff}}}_{{\mbox{\tiny\rm{NLO}}}}$ in Eq. (Nucleation at finite temperature: a
gauge-invariant, perturbative framework) explicitly depends on $\xi$. To
proceed, we utilise the Nielsen identities [43, 37] in analogy to [17]. These
identities have been discussed in the context of finite temperature in e.g.
[44, 30, 45, 36]. The Nielsen identity (in $d$-dimensional Euclidean space)

$\displaystyle\xi\frac{\partial S^{{\mbox{\scriptsize eff}}}}{\partial\xi}$
$\displaystyle=-\int{\rm d}^{d}{\bf x}\frac{\delta S^{{\mbox{\scriptsize
eff}}}}{\delta\phi(x)}\,\mathcal{C}(x)\;,$ (26)

relates the variation of the effective action with the gauge parameter to the
corresponding Nielsen functional

$\displaystyle\mathcal{C}(x)$ $\displaystyle=\frac{ig}{2}\int{\rm d}^{d}{\bf
y}\Big{\langle}\chi(x)c(x)\bar{c}(y)$
$\displaystyle\hphantom{{}\frac{ig}{2\sqrt{2}}\int{\rm d}^{d}{\bf
y}}\times\Bigl{[}\partial_{i}B_{i}(y)+\sqrt{2}g\xi\phi\chi(y)\Bigr{]}\Big{\rangle}\;,$
(27)

which admits a derivative expansion [30]

$\displaystyle\mathcal{C}(x)$
$\displaystyle=C(\phi)+D(\phi)(\partial_{\mu}\phi)^{2}-\partial_{\mu}\bigl{(}\tilde{D}(\phi)\partial_{\mu}\phi\bigr{)}+\mathcal{O}({\partial^{4}})\;.$
(28)

Together with the expansion of the effective action (2), this yields the
Nielsen identities for the effective potential and field renormalization
factor

$\displaystyle\xi\frac{\partial}{\partial\xi}V^{{\mbox{\scriptsize eff}}}=$
$\displaystyle-C\frac{\partial}{\partial\phi}V^{{\mbox{\scriptsize eff}}}\;,$
(29) $\displaystyle\xi\frac{\partial}{\partial\xi}Z=$
$\displaystyle-C\frac{\partial}{\partial\phi}Z-2Z\frac{\partial}{\partial\phi}C$
$\displaystyle-2D\frac{\partial}{\partial\phi}V^{{\mbox{\scriptsize
eff}}}-2\tilde{D}\frac{\partial^{2}}{\partial\phi^{2}}V^{{\mbox{\scriptsize
eff}}}\;.$ (30)

To employ these relations, we expand them in powers of $g$ by first quoting
the scaling of the Nielsen coefficients

$\displaystyle C$
$\displaystyle=C_{{\mbox{\tiny\rm{LO}}}}+\mathcal{O}(g^{\frac{3}{2}}T)\;,\quad
C_{{\mbox{\tiny\rm{LO}}}}=T\frac{\sqrt{\xi}}{16\pi}g\;,$ (31) $\displaystyle
D$
$\displaystyle=\mathcal{O}(g^{-1}T^{-3})\;,\quad\tilde{D}=\mathcal{O}(g^{-1}T^{-2})\;.$
(32)

These coefficients are computed at leading order in [30, 32]. We do not need
the explicit expressions of $D,\tilde{D}$ because the terms on the second line
of Eq. (Nucleation at finite temperature: a gauge-invariant, perturbative
framework) appear at $\mathcal{O}({g^{2}})$, and are hence suppressed relative
to those on the first line at $\mathcal{O}({g})$. The leading order
$C_{{\mbox{\tiny\rm{LO}}}}\sim gT$ is independent of the scalar background
field $\phi$ at finite temperature. An explicit counting in powers of $g$ in
the identities (29) and (Nucleation at finite temperature: a gauge-invariant,
perturbative framework) yields

$\displaystyle\xi\frac{\partial}{\partial\xi}V^{{\mbox{\scriptsize
eff}}}_{{\mbox{\tiny\rm{NLO}}}}$
$\displaystyle=-C_{{\mbox{\tiny\rm{LO}}}}\frac{\partial}{\partial\phi}V^{{\mbox{\scriptsize
eff}}}_{{\mbox{\tiny\rm{LO}}}}\;,$ (33)
$\displaystyle\xi\frac{\partial}{\partial\xi}Z_{{\mbox{\tiny\rm{NLO}}}}$
$\displaystyle=-2\frac{\partial}{\partial\phi}C_{{\mbox{\tiny\rm{LO}}}}\;,$
(34)

at $\mathcal{O}({g^{4}T^{4}})$ in the first and $\mathcal{O}({gT})$ in the
second Nielsen identity. A combination of the explicit expressions (Nucleation
at finite temperature: a gauge-invariant, perturbative framework), (Nucleation
at finite temperature: a gauge-invariant, perturbative framework), (22), and
(31) readily verifies both identities. In particular, the equality (34) holds
since the NLO field renormalisation is $\xi$-independent and at LO $C$ is
$\phi$-independent.

Using the above Nielsen identities, we demonstrate gauge independence of
$\mathcal{B}_{1}$:

$\displaystyle\xi\frac{\partial}{\partial\xi}\mathcal{B}_{1}$
$\displaystyle=\xi\frac{\partial}{\partial\xi}\beta\int{\rm
d}^{3}x\Bigl{[}V^{{\mbox{\scriptsize
eff}}}_{{\mbox{\tiny\rm{NLO}}}}(\phi_{b})+\frac{1}{2}Z_{{\mbox{\tiny\rm{NLO}}}}\left(\partial_{\mu}\phi_{b}\right)^{2}\Bigr{]}$
$\displaystyle\stackrel{{\scriptstyle(A)}}{{=}}\beta\int{\rm
d}{}^{3}x\Bigl{[}-C_{{\mbox{\tiny\rm{LO}}}}\frac{\partial}{\partial\phi}V^{{\mbox{\scriptsize
eff}}}_{{\mbox{\tiny\rm{LO}}}}(\phi_{b})\Bigr{]}$
$\displaystyle\stackrel{{\scriptstyle(B)}}{{=}}-C_{{\mbox{\tiny\rm{LO}}}}\,\beta\int{\rm
d}{}^{3}x\Bigl{[}\square\phi_{b}\Bigr{]}$
$\displaystyle\stackrel{{\scriptstyle(C)}}{{=}}-C_{{\mbox{\tiny\rm{LO}}}}\,\beta\int{\rm
d}^{2}S\cdot(\partial\phi_{b})$
$\displaystyle\stackrel{{\scriptstyle(D)}}{{=}}0\;.$ (35)

Step $(A)$ uses the Nielsen identity (33) and $\xi$-independence of
$Z_{{\mbox{\tiny\rm{NLO}}}}$; $(B)$ applies the equation of motion (23) and
moves $C_{{\mbox{\tiny\rm{LO}}}}$ outside the integrand due to its
$\phi$-independence; $(C)$ applies Gauss’s theorem; $(D)$ follows from the
asymptotic behaviour of the bounce solution at the boundary,

$\displaystyle\square\phi_{b}$
$\displaystyle\sim\mu^{2}_{{\mbox{\tiny\rm{eff}}}}\phi_{b}\;,\quad\phi_{b}(\infty)=0\;,$
(36) $\displaystyle\implies\phi_{b}(r)$ $\displaystyle\sim
c\frac{e^{-\mu_{{\mbox{\tiny\rm{eff}}}}r}}{r}\qquad\text{as }r\to\infty\;.$
(37)

This completes our proof of gauge invariance of the leading exponential of the
nucleation rate.

Figure 1:  An illustration of a gauge independent, perturbative determination
of the thermodynamics for a first order phase transition, with $g^{2}=0.42$
and $M=100$, where $M$ and $T$ are in arbitrary units of mass. Left: The
dimensionless ratio $L/T^{4}_{c}$ as function of $\lambda$ at LO (dashed line)
and NLO (solid line). Right: The action
$S_{3}/T=\mathcal{B}_{0}+\mathcal{B}_{1}$ as function of temperature at
$\lambda=0.005$, is shown by a black solid curve. For comparison, both panels
illustrate the conventional, inconsistent gauge dependent determination,
wherein the light grey band shows the variation of the gauge-fixing parameter
$\xi=(0,\dots,5)$.

Figure 1 illustrates the quantitative impact of applying our framework as
compared to the conventional gauge-dependent approach. Fig. 1 (left) gives the
$\lambda$-dependence of the strength of the transition, characterized by
latent heat released in the transition, $L$, scaled to $T_{\rm c}^{4}$. Fig. 1
(right) shows the $T$-dependence of $S_{3}/T$. We fix $g^{2}=0.42$ and
$\mu^{2}_{{\mbox{\tiny\rm{eff}}}}=-M^{2}/2$, where $M=100$, and where $M$ and
$T$ are in arbitrary units of mass. Furthermore, we define $\lambda$ and $g$
at an initial renormalisation scale $\Lambda_{0}=100$ and run to a scale
$\Lambda=T$ using one-loop renormalization group evolution [32].

The dashed (solid) line in Fig. 1 (left) corresponds to the gauge-invariant
result for $L/T_{\rm c}^{4}$ at LO (LO$+$NLO). For comparison, we also
illustrate the conventional analysis based on direct minimization of the full
one-loop effective potential (in the high-$T$ expansion). The latter approach
is gauge-dependent and inconsistent, as illustrated by the light grey band in
which the gauge-fixing parameter ranges over $\xi=(0,\dots,5)$. The
qualitative behaviour – the transition is stronger for smaller $\lambda$ –
agrees with the gauge-independent determination, but the quantitative result
is ambiguous.

Fig. 1 (right) shows the $T$-dependence of
$S_{3}/T=\mathcal{B}_{0}+\mathcal{B}_{1}$ for fixed $\lambda=0.005$ as the
black solid curve. We have obtained this result using FindBounce [46] to find
the bounce solution $\phi_{b}$ of Eq. (23). For comparison, the light grey
band significantly varies in $S_{3}/T$ when varying $\xi=(0,\dots,5)$ as
manifestation of an inconsistent computation that is often encountered in the
past literature (e.g. [47, 8]). The latter result is based on computing Eq.
(1) using Eq. (2), where the effective potential is directly computed at one-
loop order in a fixed gauge. Therein, the bounce solution is found from the
same gauge-dependent one-loop effective potential. This leftover gauge-
dependence immediately signals an inconsistency, as the computation of a
physical quantity should contain no $\xi$-dependence. For a discussion of
other theoretical inconsistencies encountered in the “conventional” approach,
such as the appearance of an imaginary part in $S_{3}$ and double counting of
contributions from the nucleating field, see [48, 31]. The renormalization
scale-dependence of the results in Fig. 1, that can be used to monitor the
accuracy of perturbation theory (c.f. [48, 49]), is further discussed in [32].

The framework presented in this letter provides a way to obtain a gauge-
independent, perturbative estimate of the thermal nucleation rate in the
presence of radiative barriers. While we have worked in the Abelian Higgs
model along the lines of previous literature [17, 30], our framework readily
generalises to more complicated gauge field theories, with radiatively
generated barrier, and our practical approach can facilitate corresponding
model-building phenomenological studies. Ultimately, one must assess the
quantitative and qualitative reliability of perturbative nucleation rate
computations through comparison with non-perturbative calculations (see [15,
50]). A meaningful comparison requires a well-defined, gauge-invariant
perturbative computation, which the framework presented herein provides.

###### Acknowledgements.

Acknowledgments.—We acknowledge enlightening discussions with Andreas Ekstedt,
Oliver Gould, Joonas Hirvonen and Juuso Österman and specially thank Suntharan
Arunasalam for his contributions during early stages of the project. MJRM and
TT are supported in part under National Science Foundation of China grant No.
19Z103010239. PS has been supported by the European Research Council, grant
no. 725369, and by the Academy of Finland, grant no. 1322507. TT thanks Aleksi
Vuorinen and Helsinki Institute of Physics for their hospitality and support
during part of this work.

## References

  * Kajantie  _et al._ [1996a] K. Kajantie, M. Laine, K. Rummukainen, and M. E. Shaposhnikov, Phys. Rev. Lett. 77, 2887 (1996a), arXiv:hep-ph/9605288 .
  * Csikor  _et al._ [1999] F. Csikor, Z. Fodor, and J. Heitger, Phys. Rev. Lett. 82, 21 (1999), arXiv:hep-ph/9809291 .
  * Weinberg [1974] S. Weinberg, Phys. Rev. D 9, 3357 (1974).
  * Kuzmin  _et al._ [1985] V. A. Kuzmin, V. A. Rubakov, and M. E. Shaposhnikov, Phys. Lett. B 155, 36 (1985).
  * Shaposhnikov [1987] M. E. Shaposhnikov, Nucl. Phys. B 287, 757 (1987).
  * Morrissey and Ramsey-Musolf [2012] D. E. Morrissey and M. J. Ramsey-Musolf, New J. Phys. 14, 125003 (2012), arXiv:1206.2942 [hep-ph] .
  * Ramsey-Musolf [2020] M. J. Ramsey-Musolf, JHEP 09, 179 (2020), arXiv:1912.07189 [hep-ph] .
  * Caprini  _et al._ [2020] C. Caprini _et al._ , JCAP 03, 024 (2020), arXiv:1910.13125 [astro-ph.CO] .
  * Kawamura  _et al._ [2011] S. Kawamura _et al._ , Class. Quant. Grav. 28, 094011 (2011).
  * Harry  _et al._ [2006] G. M. Harry, P. Fritschel, D. A. Shaddock, W. Folkner, and E. S. Phinney, Classical and Quantum Gravity 23, 4887 (2006).
  * Ruan  _et al._ [2020] W.-H. Ruan, Z.-K. Guo, R.-G. Cai, and Y.-Z. Zhang, Int. J. Mod. Phys. A 35, 2050075 (2020), arXiv:1807.09495 [gr-qc] .
  * Amaro-Seoane  _et al._ [2017] P. Amaro-Seoane _et al._ (LISA), (2017), arXiv:1702.00786 [astro-ph.IM] .
  * Farakos  _et al._ [1995] K. Farakos, K. Kajantie, K. Rummukainen, and M. E. Shaposhnikov, Nucl. Phys. B 442, 317 (1995), arXiv:hep-lat/9412091 .
  * Kajantie  _et al._ [1996b] K. Kajantie, M. Laine, K. Rummukainen, and M. E. Shaposhnikov, Nucl. Phys. B 466, 189 (1996b), arXiv:hep-lat/9510020 .
  * Moore and Rummukainen [2001] G. D. Moore and K. Rummukainen, Phys. Rev. D63, 045002 (2001), arXiv:hep-ph/0009132 [hep-ph] .
  * Caprini  _et al._ [2016] C. Caprini _et al._ , JCAP 04, 001 (2016), arXiv:1512.06239 [astro-ph.CO] .
  * Metaxas and Weinberg [1996] D. Metaxas and E. J. Weinberg, Phys. Rev. D 53, 836 (1996), arXiv:hep-ph/9507381 .
  * Baacke and Heitmann [1999] J. Baacke and K. Heitmann, Phys. Rev. D 60, 105037 (1999), arXiv:hep-th/9905201 .
  * Andreassen  _et al._ [2015] A. Andreassen, W. Frost, and M. D. Schwartz, Phys. Rev. D 91, 016009 (2015), arXiv:1408.0287 [hep-ph] .
  * Andreassen  _et al._ [2014] A. Andreassen, W. Frost, and M. D. Schwartz, Phys. Rev. Lett. 113, 241801 (2014), arXiv:1408.0292 [hep-ph] .
  * Garbrecht and Millington [2015] B. Garbrecht and P. Millington, Phys. Rev. D 92, 125022 (2015), arXiv:1509.08480 [hep-ph] .
  * Endo  _et al._ [2017] M. Endo, T. Moroi, M. M. Nojiri, and Y. Shoji, Phys. Lett. B 771, 281 (2017), arXiv:1703.09304 [hep-ph] .
  * Langer [1969] J. S. Langer, Annals Phys. 54, 258 (1969).
  * Coleman [1977] S. R. Coleman, Phys. Rev. D 15, 2929 (1977), [Erratum: Phys.Rev.D 16, 1248 (1977)].
  * Linde [1980] A. D. Linde, Phys. Lett. 96B, 289 (1980).
  * Affleck [1981] I. Affleck, Phys. Rev. Lett. 46, 388 (1981).
  * Linde [1983] A. D. Linde, Nucl. Phys. B 216, 421 (1983), [Erratum: Nucl.Phys.B 223, 544 (1983)].
  * Langer [1974] J. Langer, Physica 73, 61 (1974).
  * Moss  _et al._ [1992] I. Moss, D. Toms, and A. Wright, Phys. Rev. D 46, 1671 (1992).
  * Garny and Konstandin [2012] M. Garny and T. Konstandin, JHEP 07, 189 (2012), arXiv:1205.3392 [hep-ph] .
  * Gould and Hirvonen [2021] O. Gould and J. Hirvonen, Phys. Rev. D 104, 096015 (2021), arXiv:2108.04377 [hep-ph] .
  * Hirvonen  _et al._ [2021] J. Hirvonen, J. Löfgren, M. J. Ramsey-Musolf, P. Schicho, and T. V. I. Tenkanen, forthcoming (2021).
  * Kajantie  _et al._ [1996c] K. Kajantie, M. Laine, K. Rummukainen, and M. E. Shaposhnikov, Nucl. Phys. B 458, 90 (1996c), arXiv:hep-ph/9508379 .
  * Braaten and Nieto [1995] E. Braaten and A. Nieto, Phys. Rev. D51, 6990 (1995), arXiv:hep-ph/9501375 [hep-ph] .
  * Arnold and Espinosa [1993] P. B. Arnold and O. Espinosa, Phys. Rev. D47, 3546 (1993), [Erratum: Phys. Rev.D50,6662(1994)], arXiv:hep-ph/9212235 [hep-ph] .
  * Arunasalam and Ramsey-Musolf [2021] S. Arunasalam and M. J. Ramsey-Musolf, (2021), arXiv:2105.07588 [hep-ph] .
  * Fukuda and Kugo [1976] R. Fukuda and T. Kugo, Phys. Rev. D 13, 3469 (1976).
  * Ekstedt and Löfgren [2020] A. Ekstedt and J. Löfgren, JHEP 12, 136 (2020), arXiv:2006.12614 [hep-ph] .
  * Gynther and Vepsäläinen [2006] A. Gynther and M. Vepsäläinen, JHEP 03, 011 (2006), arXiv:hep-ph/0512177 .
  * Coleman and Weinberg [1973] S. R. Coleman and E. J. Weinberg, Phys. Rev. D 7, 1888 (1973).
  * Baacke and Lavrelashvili [2004] J. Baacke and G. Lavrelashvili, Phys. Rev. D 69, 025009 (2004), arXiv:hep-th/0307202 .
  * Ekstedt [2021] A. Ekstedt, (2021), arXiv:2104.11804 [hep-ph] .
  * Nielsen [1975] N. K. Nielsen, Nucl. Phys. B 101, 173 (1975).
  * Patel and Ramsey-Musolf [2011] H. H. Patel and M. J. Ramsey-Musolf, JHEP 07, 029 (2011), arXiv:1101.4665 [hep-ph] .
  * Espinosa  _et al._ [2017] J. R. Espinosa, M. Garny, T. Konstandin, and A. Riotto, Phys. Rev. D 95, 056004 (2017), arXiv:1608.06765 [hep-ph] .
  * Guada  _et al._ [2020] V. Guada, M. Nemevšek, and M. Pintar, Comput. Phys. Commun. 256, 107480 (2020), arXiv:2002.00881 [hep-ph] .
  * Delaunay  _et al._ [2008] C. Delaunay, C. Grojean, and J. D. Wells, JHEP 04, 029 (2008), arXiv:0711.2511 [hep-ph] .
  * Croon  _et al._ [2021] D. Croon, O. Gould, P. Schicho, T. V. I. Tenkanen, and G. White, JHEP 04, 055 (2021), arXiv:2009.10080 [hep-ph] .
  * Gould and Tenkanen [2021] O. Gould and T. V. I. Tenkanen, JHEP 06, 069 (2021), arXiv:2104.04399 [hep-ph] .
  * Moore  _et al._ [2001] G. D. Moore, K. Rummukainen, and A. Tranberg, JHEP 04, 017 (2001), arXiv:hep-lat/0103036 [hep-lat] .