* [104] L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, and R. Fergus, “Regularization of neural networks using dropconnect,” in _International conference on machine learning_ , 2013, pp. 1058–1066.
  * [105] T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” _arXiv preprint arXiv:1708.04552_ , 2017.
  * [106] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 6023–6032.
  * [107] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” _arXiv preprint arXiv:1710.09412_ , 2017.
  * [108] H.-Y. Tseng, L. Jiang, C. Liu, M.-H. Yang, and W. Yang, “Regularizing generative adversarial networks under limited data,” _arXiv preprint arXiv:2104.03310_ , 2021.
  * [109] Z. Zhao, Z. Zhang, T. Chen, S. Singh, and H. Zhang, “Image augmentations for gan training,” _arXiv preprint arXiv:2006.02595_ , 2020.
  * [110] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila, “Training generative adversarial networks with limited data,” _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [111] N.-T. Tran, V.-H. Tran, N.-B. Nguyen, T.-K. Nguyen, and N.-M. Cheung, “Towards good practices for data augmentation in gan training,” _arXiv preprint arXiv:2006.05338_ , 2020.
  * [112] J. Jeong and J. Shin, “Training gans with stronger augmentations via contrastive discriminator,” 2021.
  * [113] I. Anokhin, K. Demochkin, T. Khakhulin, G. Sterkin, V. Lempitsky, and D. Korzhenkov, “Image generators with conditionally-independent pixel synthesis,” 2020.
  * [114] Q. Xie, Z. Dai, E. Hovy, M.-T. Luong, and Q. V. Le, “Unsupervised data augmentation for consistency training,” _arXiv preprint arXiv:1904.12848_ , 2019.
  * [115] K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” _arXiv preprint arXiv:2001.07685_ , 2020\.
  * [116] M. Gao, Z. Zhang, G. Yu, S. O. Arik, L. S. Davis, and T. Pfister, “Consistency-based semi-supervised active learning: Towards minimizing labeling cost,” _arXiv preprint arXiv:1910.07153_ , 2019.
  * [117] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [118] A. Jolicoeur-Martineau, “The relativistic discriminator: a key element missing from standard gan,” _arXiv preprint arXiv:1807.00734_ , 2018.
  * [119] J. Su, “Training generative adversarial networks via turing test,” _arXiv preprint arXiv:1810.10948_ , 2018.
  * [120] Y. Xiangli, Y. Deng, B. Dai, C. C. Loy, and D. Lin, “Real or not real, that is the question,” _arXiv preprint arXiv:2002.05512_ , 2020.
  * [121] Y.-G. Shin, Y.-J. Yeo, and S.-J. Ko, “Simple yet effective way for improving the performance of gan,” _arXiv preprint arXiv:1911.10979_ , 2019.
  * [122] Y. Mroueh, T. Sercu, and V. Goel, “Mcgan: Mean and covariance feature matching gan,” _arXiv preprint arXiv:1702.08398_ , 2017.
  * [123] R. Durall, M. Keuper, and J. Keuper, “Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7890–7899.
  * [124] T. Ohkawa, N. Inoue, H. Kataoka, and N. Inoue, “Augmented cyclic consistency regularization for unpaired image-to-image translation,” _arXiv preprint arXiv:2003.00187_ , 2020.
  * [125] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9729–9738.
  * [126] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio, “Learning deep representations by mutual information estimation and maximization,” _arXiv preprint arXiv:1808.06670_ , 2018.
  * [127] C. Doersch, A. Gupta, and A. A. Efros, “Unsupervised visual representation learning by context prediction,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 1422–1430.
  * [128] S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting image rotations,” _arXiv preprint arXiv:1803.07728_ , 2018\.
  * [129] H. Lee, S. J. Hwang, and J. Shin, “Rethinking data augmentation: Self-supervision and self-distillation,” _arXiv preprint arXiv:1910.05872_ , 2019.
  * [130] A. Kolesnikov, X. Zhai, and L. Beyer, “Revisiting self-supervised visual representation learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1920–1929.
  * [131] X. Zhai, A. Oliver, A. Kolesnikov, and L. Beyer, “S4l: Self-supervised semi-supervised learning,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1476–1485.
  * [132] R. Huang, W. Xu, T.-Y. Lee, A. Cherian, Y. Wang, and T. Marks, “Fx-gan: Self-supervised gan learning via feature exchange,” in _Proceedings of the IEEE Winter Conference on Applications of Computer Vision_ , 2020, pp. 3194–3202.
  * [133] G. Baykal and G. Unal, “Deshufflegan: A self-supervised gan to improve structure learning,” _arXiv preprint arXiv:2006.08694_ , 2020.
  * [134] P. Patel, N. Kumari, M. Singh, and B. Krishnamurthy, “Lt-gan: Self-supervised gan with latent transformation detection,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021, pp. 3189–3198.
  * [135] F. M. Carlucci, A. D’Innocente, S. Bucci, B. Caputo, and T. Tommasi, “Domain generalization by solving jigsaw puzzles,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2229–2238.
  * [136] N.-T. Tran, V.-H. Tran, B.-N. Nguyen, L. Yang _et al._ , “Self-supervised gan: Analysis and improvement with multi-class minimax game,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 13 253–13 264.
  * [137] K. S. Lee, N.-T. Tran, and N.-M. Cheung, “Infomax-gan: Improved adversarial image generation via information maximization and contrastive learning,” _arXiv preprint arXiv:2007.04589_ , 2020.
  * [138] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [139] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio, “Learning deep representations by mutual information estimation and maximization,” 2019.
  * [140] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” _arXiv preprint arXiv:1607.08022_ , 2016\.
  * [141] Y. Wu and K. He, “Group normalization,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 3–19.
  * [142] T. Salimans and D. P. Kingma, “Weight normalization: A simple reparameterization to accelerate training of deep neural networks,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 901–909.
  * [143] T. Miyato and M. Koyama, “cgans with projection discriminator,” _arXiv preprint arXiv:1802.05637_ , 2018.
  * [144] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” _arXiv preprint arXiv:1805.08318_ , 2018.
  * [145] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1501–1510.
  * [146] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4401–4410.
  * [147] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu, “Semantic image synthesis with spatially-adaptive normalization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2337–2346.
  * [148] Y. Wang, Y.-C. Chen, X. Zhang, J. Sun, and J. Jia, “Attentive normalization for conditional image generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5094–5103.
  * [149] S. Xiang and H. Li, “On the effects of batch and weight normalization in generative adversarial networks,” _arXiv preprint arXiv:1704.03971_ , 2017\.
  * [150] D. Yang, S. Hong, Y. Jang, T. Zhao, and H. Lee, “Diversity-sensitive conditional generative adversarial networks,” _arXiv preprint arXiv:1901.09024_ , 2019.
  * [151] A. Odena, J. Buckman, C. Olsson, T. B. Brown, C. Olah, C. Raffel, and I. Goodfellow, “Is generator conditioning causally related to gan performance?” _arXiv preprint arXiv:1802.08768_ , 2018.
  * [152] X. Gong, S. Chang, Y. Jiang, and Z. Wang, “Autogan: Neural architecture search for generative adversarial networks,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 3224–3234.
  * [153] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8110–8119.
  * [154] S. Zhao, J. Cui, Y. Sheng, Y. Dong, X. Liang, E. I. Chang, and Y. Xu, “Large scale image completion via co-modulated generative adversarial networks,” _arXiv preprint arXiv:2103.10428_ , 2021.
  * [155] E. Richardson, Y. Alaluf, O. Patashnik, Y. Nitzan, Y. Azar, S. Shapiro, and D. Cohen-Or, “Encoding in style: a stylegan encoder for image-to-image translation,” _arXiv preprint arXiv:2008.00951_ , 2020.


# CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models

Hyungjin Chung∗, Jeongsol Kim∗, Geon Yeong Park∗, Hyelin Nam∗, Jong Chul Ye
KAIST
∗: Equal Contribution
{hj.chung, jeongsol, pky3436, hyelin.nam<EMAIL_ADDRESS>
###### Abstract

Classifier-free guidance (CFG) is a fundamental tool in modern diffusion
models for text-guided generation. Although effective, CFG has notable
drawbacks. For instance, DDIM with CFG lacks invertibility, complicating image
editing; furthermore, high guidance scales, essential for high-quality
outputs, frequently result in issues like mode collapse. Contrary to the
widespread belief that these are inherent limitations of diffusion models,
this paper reveals that the problems actually stem from the off-manifold
phenomenon associated with CFG, rather than the diffusion models themselves.
More specifically, inspired by the recent advancements of diffusion model-
based inverse problem solvers (DIS), we reformulate text-guidance as an
inverse problem with a text-conditioned score matching loss, and develop
CFG++, a novel approach that tackles the off-manifold challenges inherent in
traditional CFG. CFG++ features a surprisingly simple fix to CFG, yet it
offers significant improvements, including better sample quality for text-to-
image generation, invertibility, smaller guidance scales, reduced mode
collapse, etc. Furthermore, CFG++ enables seamless interpolation between
unconditional and conditional sampling at lower guidance scales, consistently
outperforming traditional CFG at all scales. Experimental results confirm that
our method significantly enhances performance in text-to-image generation,
DDIM inversion, editing, and solving inverse problems, suggesting a wide-
ranging impact and potential applications in various fields that utilize text
guidance. Project Page: https://cfgpp-diffusion.github.io/.

Figure 1:  (Top) Comparison of T2I results by SDXL for the prompt "kayak in
the water, optical color, aerial view, rainbow". The CFG-guided image has a
broken oar, an artifact significantly reduced in the CFG++ version. (Middle)
DDIM Inversion results under CFG show noticeable artifacts at various CFG
scales, which are significantly reduced by CFG++. (Bottom) The evolution of
denoised estimates differs between CFG and CFG++. CFG exhibits sudden shifts
and intense color saturation early in reverse diffusion, while CFG++
transitions smoothly from low to high-resolution details.

##  1 Introduction

Classifier-free guidance (CFG) (Ho & Salimans, 2021) forms the key basis of
modern text-guided generation with diffusion models (Dhariwal & Nichol, 2021;
Rombach et al., 2022). Nowadays, it is common practice to train a diffusion
model with large-scale paired text-image data (Schuhmann et al., 2022), so
that sampling (i.e. generating) a signal (e.g. image, video) from a diffusion
model can either be done unconditionally from
$p_{\theta}({\boldsymbol{x}}|\varnothing)\equiv p_{\theta}({\boldsymbol{x}})$,
or conditionally from $p_{\theta}({\boldsymbol{x}}|{\boldsymbol{c}})$, where
${\boldsymbol{c}}$ is the text conditioning. Once trained, it seems natural
that one would acquire samples from the conditional distribution by simply
solving the probability-flow ODE or SDE sampling (Song et al., 2021b; a;
Karras et al., 2022) with the conditional score function. In practice,
however, it is observed that the conditioning signal is insufficient when used
naively. To emphasize the guidance, one uses the guidance scale $\omega>1$,
where the direction can be defined by the direction from the unconditional
score to the conditional score (Ho & Salimans, 2021).

In modern text-to-image (T2I) diffusion models, the guidance scale $\omega$ is
typically set within the range of $[5.0,30]$, referred to as the moderately
high range of CFG guidance (Chen et al., 2024b; Podell et al., 2023). The
insufficiency in guidance also holds for classifier guidance (Dhariwal &
Nichol, 2021; Song et al., 2021b) so that a scale of 10 was used. While using
a high guidance scale yields higher-quality images with better alignment to
the condition, it is also prone to mode collapse, reduces sample diversity,
and yields an inevitable accumulation of errors during the sampling process.
One example is DDIM inversion (Dhariwal & Nichol, 2021), a pivotal technique
for controllable synthesis and editing (Mokady et al., 2023), where running
the inversion process with $\omega>1.0$ leads to significant compromise in the
reconstruction performance (Mokady et al., 2023; Wallace et al., 2023).
Another extreme example would be score distillation sampling (SDS) (Poole et
al., 2022), where the guidance scale in the order of a few hundred is chosen.
Using such a high guidance scale leads to better asset quality to some extent,
but induces blurry and saturated results. Several research efforts have been
made to mitigate this downside by exploring methods where using a smaller
guidance scale suffices (Wang et al., 2024; Liang et al., 2023). Although
recent progress in SDS-type methods has reduced the necessary guidance scale
to a range that is similar to those of ancestral samplers, using a moderately
large $\omega$ is considered an inevitable choice.

In this work, we aim to give an answer to this conundrum by revisiting the
geometric view of diffusion models. In particular, inspired by the recent
advances in diffusion-based inverse problem solvers (DIS) (Kadkhodaie &
Simoncelli, 2021; Chung et al., 2023a; Song et al., 2023a; Kim et al., 2024;
Chung et al., 2024), we reformulate the text-guidance as an inverse problem
with a text-conditioned score matching loss and derive reverse diffusion
sampling strategy by utilizing the decomposed diffusion sampling (DDS) (Chung
et al., 2024). This results in a surprisingly simple fix of CFG to the
sampling process without any computational overhead. The resulting process,
which we call CFG++, works with a small guidance scale, typically
$\lambda\in[0.0,1.0]$, that smoothly interpolates between unconditional and
conditional sampling, with $\lambda=1.0$ having a similar effect as using CFG
sampling with $\omega\sim 12.5$ at 50 neural function evaluation (NFE).
Furthermore, DDIM inversion with CFG++ is invertible up to the discretization
error, simplifying image editing. Comparing CFG++ against CFG shows that we
achieve consistently better sample quality for text-to-image (T2I) generation,
significantly better DDIM inversion capabilities that lead to enhanced
reconstruction and editing, and enabling the incorporation of CFG guidance to
diffusion inverse solvers (DIS) (Chung et al., 2023a). While the applications
of CFG++ that we show in this work are limited, we believe that our work will
have a broad impact that can be applied to all applications that leverage text
guidance through the traditional CFG.

Algorithm 1 Reverse Diffusion with CFG

1:${\boldsymbol{x}}_{T}\sim{\mathcal{N}}(0,{\mathbf{I}}_{d}),0\leq{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\omega}\in\mathbb{R}$

2:for $i=T$ to $1$ do

3:
${\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})={\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})+{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\omega}[{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-{\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})]$

4:
${\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})\leftarrow({\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t}))/\sqrt{\bar{\alpha}_{t}}$

5:
${\boldsymbol{x}}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}{\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})+\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})$

6:end for

7:return ${\boldsymbol{x}}_{0}$

Algorithm 2 Reverse Diffusion with CFG++

1:${\boldsymbol{x}}_{T}\sim{\mathcal{N}}(0,{\mathbf{I}}_{d}),{\color[rgb]{0.165,0.616,0.561}\definecolor[named]{pgfstrokecolor}{rgb}{0.165,0.616,0.561}\lambda}\in[0,1]$

2:for $i=T$ to $1$ do

3:
${\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})={\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})+{\color[rgb]{0.165,0.616,0.561}\definecolor[named]{pgfstrokecolor}{rgb}{0.165,0.616,0.561}\lambda}[{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-{\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})]$

4:
${\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})\leftarrow({\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t}))/\sqrt{\bar{\alpha}_{t}}$

5:
${\boldsymbol{x}}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}{\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})+\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})$

6:end for

7:return ${\boldsymbol{x}}_{0}$

Figure 2: Comparison between reverse diffusion process by CFG and CFG++. CFG++
proposes a simple but surprisingly effective fix: using
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})$
instead of
${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})$
in updating ${\boldsymbol{x}}_{t-1}$.

##  2 Background

Diffusion models. Diffusion models (Ho et al., 2020; Song et al., 2021b;
Karras et al., 2022) are generative models designed to learn the reversal of a
forward noising process. This process starts with an initial distribution
$p_{0}({\boldsymbol{x}})$ where ${\boldsymbol{x}}\in{\mathbb{R}}^{n}$, and
progresses towards the standard Gaussian distribution
$p_{T}({\boldsymbol{x}})\approx\mathcal{N}(\bm{0},{\bm{I}})$, utilizing
forward Gaussian perturbation kernels. Sampling from the data distribution can
be performed by solving either the reverse stochastic differential equation
(SDE) or the equivalent probability-flow ordinary differential equation (PF-
ODE) (Song et al., 2021b). For example, under the choice
$p({\boldsymbol{x}}_{t}|{\boldsymbol{x}}_{0})=\mathcal{N}({\boldsymbol{x}}_{0},t^{2}{\bm{I}})$,
the generative PF-ODE reads

$\displaystyle d{\boldsymbol{x}}_{t}=-t\nabla_{{\boldsymbol{x}}_{t}}\log
p({\boldsymbol{x}}_{t})\,dt=\frac{{\boldsymbol{x}}_{t}-\mathbb{E}[{\boldsymbol{x}}_{0}|{\boldsymbol{x}}_{t}]}{t}\,dt,\,\,{\boldsymbol{x}}_{T}\sim
p_{T}({\boldsymbol{x}}_{T}),$ (1)

where Tweedie’s formula (Efron, 2011)
${\mathbb{E}}[{\boldsymbol{x}}_{0}|{\boldsymbol{x}}_{t}]={\boldsymbol{x}}_{t}+t^{2}\nabla_{{\boldsymbol{x}}_{t}}\log
p({\boldsymbol{x}}_{t})$ is applied to achieve the second equality. Thus, the
score function $\nabla_{{\boldsymbol{x}}_{t}}\log p({\boldsymbol{x}}_{t})$
plays the key role in diffusion models. One can approximate an equivalent
parameterization of the score function via denoising score matching (Hyvärinen
& Dayan, 2005) or epsilon-matching (Ho et al., 2020), etc. When aiming for a
text-conditional diffusion model that can condition on arbitrary
${\boldsymbol{c}}$, one can extend epsilon matching to a conditional one as:

$\displaystyle\min_{\theta}{\mathbb{E}}_{{\boldsymbol{x}}_{t}\sim
p({\boldsymbol{x}}_{t}|{\boldsymbol{x}}_{0}),\\{{\boldsymbol{x}}_{0},{\boldsymbol{c}}\\}\sim
p({\boldsymbol{x}}_{0},{\boldsymbol{c}}),{\boldsymbol{\epsilon}}\sim{\mathcal{N}}(0,{\bm{I}})}\left[\|{\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},{\boldsymbol{c}})-{\boldsymbol{\epsilon}}\|_{2}^{2}\right],$
(2)

where the condition is dropped with a certain probability (Ho & Salimans,
2021) so that null conditioning with ${\boldsymbol{c}}=\varnothing$ is
possible. The neural network architecture of
${\boldsymbol{\epsilon}}_{\theta}$ is designed so that the condition
${\boldsymbol{c}}$ can modulate the output through cross attention (Rombach et
al., 2022). For simplicity in notation throughout the paper, we define
${\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}:={\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},{\boldsymbol{c}})$
and
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}:={\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},\varnothing)$
by dropping $\theta$ and ${\boldsymbol{x}}_{t}$.

Extending the result of the Tweedie’s formula to the unconditional case under
the variance preserving (VP) framework of DDPMs (Ho et al., 2020), we have

$\displaystyle{\mathbb{E}}[{\boldsymbol{x}}_{0}|{\boldsymbol{x}}_{t},\varnothing]=\hat{\boldsymbol{x}}_{\varnothing}({\boldsymbol{x}}_{t}):=({\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t}))/\sqrt{\bar{\alpha}_{t}}.$
(3)

Leveraging (3), it is common to use DDIM sampling (Song et al., 2021b) to
solve the conditional probability-flow ODE (PF-ODE) of the generative process.
Specifically, a single iteration reads

$\displaystyle{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}$
$\displaystyle=({\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}})/\sqrt{\bar{\alpha}_{t}}$
(4) $\displaystyle{\boldsymbol{x}}_{t-1}$
$\displaystyle=\sqrt{\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}+\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}},$
(5)

where
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}:=\hat{\boldsymbol{x}}_{\varnothing}({\boldsymbol{x}}_{t})={\mathbb{E}}[{\boldsymbol{x}}_{0}|{\boldsymbol{x}}_{t},\varnothing]$
is the denosed signal by Tweedie’s formula, and (5) corresponds to the
renoising step. This is repeated for $t=T,T-1,\dots,1$.

For modern diffusion models, it is common to train a diffusion model in the
latent space (Rombach et al., 2022) with the latent variable
${\boldsymbol{z}}$. While most of our experiments are performed with latent
diffusion models (LDM), as our framework holds both for pixel- and latent-
diffusion models, we will simply use the notation ${\boldsymbol{x}}$
regardless of the choice.

Classifier free guidance. For a conditional diffusion, ho2021classifierfree
considered the sharpened posterior distribution
$p^{\omega}({\boldsymbol{x}}|{\boldsymbol{c}})\propto
p({\boldsymbol{x}})p({\boldsymbol{c}}|{\boldsymbol{x}})^{\omega}$. Using Bayes
rule for some timestep $t$,

$\displaystyle\nabla_{\boldsymbol{x}}\log
p^{\omega}({\boldsymbol{x}}_{t}|{\boldsymbol{c}})=\nabla_{{\boldsymbol{x}}_{t}}\log
p({\boldsymbol{x}}_{t})+\omega(\nabla_{{\boldsymbol{x}}_{t}}\log
p({\boldsymbol{x}}_{t}|{\boldsymbol{c}})-\nabla_{{\boldsymbol{x}}_{t}}\log
p({\boldsymbol{x}}_{t}))$ (6)

Parametrizing the score function with ${\boldsymbol{\epsilon}}_{\theta}$, as
in DDPM (ho2020denoising), we have

$\displaystyle{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t}):={\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})+\omega[{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})]$
(7)

where we introduce a compact notation
${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}$
that guides the sampling from the sharpened posterior. When sampling with CFG
guidance with DDIM sampling, one replaces
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}$
with
${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}$
for both the Tweedie estimate (4) and the subsequent update step (5), leading
to Algorithm 1.

Diffusion model-based inverse problem solvers. Diffusion model-based inverse
problem solvers (DIS) aim to perform posterior sampling from an unconditional
diffusion model (kadkhodaie2021stochastic; chung2023diffusion;
song2023pseudoinverseguided; kim2024dreamsampler). Specifically, for a given
loss function $\ell({\boldsymbol{x}})$ which often stems from the likelihood
for measurement consistency111For example, for a given measurement model
${\boldsymbol{y}}=A{\boldsymbol{x}}+{\boldsymbol{w}}$ with
${\boldsymbol{w}}\sim\mathcal{N}({\boldsymbol{x}}_{0},\sigma^{2}{\bm{I}})$,
the loss function is the data consistency (DC) loss given by
$\ell({\boldsymbol{x}})=\|{\boldsymbol{y}}-A{\boldsymbol{x}}\|^{2}/2\sigma^{2}$.
, the goal of DIS is to address the following optimization problem

$\displaystyle\min_{{\boldsymbol{x}}\in{\mathcal{M}}}\ell({\boldsymbol{x}})$
(8)

where ${\mathcal{M}}$ represents the clean data manifold sampled from
unconditional distribution $p_{0}({\boldsymbol{x}})$. Consequently, it is
essential to navigate in a way that minimizes cost while also identifying the
correct clean manifold.

Recently, chung2023diffusion proposed a general technique called diffusion
posterior sampling (DPS), where the updated estimate from the noisy sample
${\boldsymbol{x}}_{t}\in{\mathcal{M}}_{t}$ is constrained to stay on the same
noisy manifold ${\mathcal{M}}_{t}$. This is achieved by computing the manifold
constrained gradient (MCG) (chung2022improving) on a noisy sample
${\boldsymbol{x}}_{t}\in{\mathcal{M}}_{t}$ as
$\nabla^{mcg}_{{\boldsymbol{x}}_{t}}\ell({\boldsymbol{x}}_{t}):=\nabla_{{\boldsymbol{x}}_{t}}\ell(\hat{\boldsymbol{x}}_{t})$,
where $\hat{\boldsymbol{x}}_{t}$ is the denoised sample in (3) through
Tweedie’s formula. The resulting algorithm can be stated as follows:

$\displaystyle{\boldsymbol{x}}_{t-1}$
$\displaystyle=\sqrt{\bar{\alpha}_{t-1}}\left({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}-\gamma_{t}\nabla_{{\boldsymbol{x}}_{t}}\ell({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}})\right)+\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}},$
(9)

where $\gamma_{t}>0$ denotes the step size. Under the linear manifold
assumption (chung2022improving; chung2023diffusion), this allows precise
transition to ${\mathcal{M}}_{t-1}$. Therefore, by performing (9) from $t=T$
to $t=0$, we can solve the optimization problem (8) with
${\boldsymbol{x}}_{0}\in{\mathcal{M}}$. Unfortunately, the computation of MCG
requires computationally expensive backpropagation and is often unstable
(poole2022dreamfusion; du2023reduce).

In another work, chung2024decomposed shows that (9) can be equivalently
represented as

$\displaystyle{\boldsymbol{x}}_{t-1}$
$\displaystyle=\sqrt{\bar{\alpha}_{t-1}}P_{\mathcal{M}}\left({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}-\gamma_{t}\nabla_{{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}}\ell({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}})\right)+\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}},$
(10)

where $P_{\mathcal{M}}$ denotes the projection on the clean manifold
${\mathcal{M}}$. This implies that the data consistency update step becomes a
standard gradient step, followed by the projection on the clean manifold.
Importantly, under the commonly used linear manifold assumption in DPS, the
one step update by
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}-\gamma_{t}\nabla_{{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}}\ell({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}})$
or multi-step update employing the conjugate gradient (CG) method are
guaranteed to remain within a linear subspace, thus obviating the need for
explicit computation of $P_{\mathcal{M}}$, i.e.

$\displaystyle{\boldsymbol{x}}_{t-1}$
$\displaystyle\simeq\sqrt{\bar{\alpha}_{t-1}}\left({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}-\gamma_{t}\nabla_{{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}}\ell({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}})\right)+\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}.$
(11)

This method, often referred to as the decomposed diffusion sampling (DDS),
bypasses the computation of the score Jacobian, similar to
poole2022dreamfusion, making it stable and suitable for large scale medical
imaging inverse problems (chung2024decomposed). In the following, we leverage
the insight from DDS to propose an algorithm to improve upon the CFG
algorithm.

Figure 3: Off-manifold phenomenon of CFG arise from: (a) the typical CFG scale
$\omega>1.0$ which leads to extrapolation and deviation from the piecewise
linear data manifold, and (b) CFG’s renoising process, which introduces a
nonzero offset $\Delta^{\omega}$ from the correct manifold. CFG++ effectively
mitigates all these artifacts.

##  3 CFG++ : Manifold-Constrained CFG

###  3.1 Derivation of CFG++

Instead of uncritically adopting the sharpened posterior distribution
$p^{\omega}({\boldsymbol{x}}|{\boldsymbol{c}})\propto
p({\boldsymbol{x}})p({\boldsymbol{c}}|{\boldsymbol{x}})^{\omega}$ as
introduced by ho2021classifierfree and further explored in the Bayesian CFG
formulation, we adopt a fundamentally different strategy by reformulating
text-guidance as an optimization problem. Specifically, our focus is on
identifying a loss function $\ell({\boldsymbol{x}})$ in (8) such that, when
minimized under the condition set by the text, enables the reverse diffusion
process to progressively generate samples that increasingly satisfy the text
condition.

One of the most significant contributions of this paper is to reveal that the
text-conditioned score matching loss or score distillation sampling (SDS) loss
is also ideally suited for our purpose. Specifically, we are interested in
solving the following inverse problem through diffusion models:

$\displaystyle\min_{{\boldsymbol{x}}\in{\mathcal{M}}}\ell_{sds}(x),\quad\ell_{sds}({\boldsymbol{x}}):=\|{\boldsymbol{\epsilon}}_{\theta}(\sqrt{\bar{\alpha}_{t}}{\boldsymbol{x}}+\sqrt{1-\bar{\alpha}_{t}}{\boldsymbol{\epsilon}},{\boldsymbol{c}})-{\boldsymbol{\epsilon}}\|_{2}^{2}$
(12)

This implies that our goal is to identify solutions on the clean manifold
${\mathcal{M}}$ that optimally align with the text condition
${\boldsymbol{c}}$.

Figure 4: Enhanced T2I results by SDXL ($\omega=9.0,\lambda=0.8$) with CFG++.
Under CFG, the lion cub is not visible (top-left), the dog appears with two
tails (top-right), the goggles have an unusual shape (bottom-left), and the
tree trunk is folded (bottom-right). These artifacts are absent in those
produced by CFG++.

To avoid the Jacobian computation, in this paper we attempt to solve (12)
through decomposed diffusion sampling (DDS). The resulting sampling process
from reverse diffusion is then given by

$\displaystyle{\boldsymbol{x}}_{t-1}$ $\displaystyle=$
$\displaystyle\sqrt{\bar{\alpha}_{t-1}}\left({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}-\gamma_{t}\nabla_{{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}}\ell_{sds}({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}})\right)+\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}$
(13)

which is analogous to (11). Consequently, the remaining technical issue is to
compute
$\nabla_{{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}}\ell_{sds}({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}})$.
To answer this, by using the forward diffusion to generate
${\boldsymbol{x}}_{t}=\sqrt{\bar{\alpha}_{t}}{\boldsymbol{x}}+\sqrt{1-\bar{\alpha}_{t}}{\boldsymbol{\epsilon}}$
from the clean image ${\boldsymbol{x}}\in{\mathcal{M}}$, we convert the
objective function in (12) into the following equivalent loss:

$\displaystyle\ell_{sds}({\boldsymbol{x}})$
$\displaystyle=\|{\boldsymbol{\epsilon}}-{\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},{\boldsymbol{c}})\|^{2}=\left\|\frac{{\boldsymbol{x}}_{t}-\sqrt{\bar{\alpha}_{t}}{\boldsymbol{x}}}{\sqrt{1-\bar{\alpha}_{t}}}-\frac{{\boldsymbol{x}}_{t}-\sqrt{\bar{\alpha}_{t}}{\hat{\boldsymbol{x}}_{\boldsymbol{c}}}}{\sqrt{1-\bar{\alpha}_{t}}}\right\|^{2}$
(14)
$\displaystyle=\frac{\bar{\alpha}_{t}}{1-\bar{\alpha}_{t}}\|{\boldsymbol{x}}-{\hat{\boldsymbol{x}}_{\boldsymbol{c}}}\|^{2}$
(15)

where we use the conditional Tweedie formula derived by replacing
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}$
in (4) by the conditioned score
${\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}$. Similar observation has been
made in DreamSampler (kim2024dreamsampler) and HiFA (zhu2023hifa).
Consequently,
$\nabla_{{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}}\ell_{sds}({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}})$
can be readily computable and the resulting DDS sampling process can be
simplified as

$\displaystyle{\boldsymbol{x}}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\left({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}+\lambda({\hat{\boldsymbol{x}}_{\boldsymbol{c}}}-{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}})\right)+\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}$
(16)

where $\lambda:=\frac{2{\bar{\alpha}_{t}}}{{1-\bar{\alpha}_{t}}}\gamma_{t}$.
Although concise, (16) can be further simplified to a similar form of the
traditional CFG as described in Algorithm 1. Specifically, note that

$\displaystyle{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}+\lambda({\hat{\boldsymbol{x}}_{\boldsymbol{c}}}-{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}})=({\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t}))/\sqrt{\bar{\alpha}_{t}}$
(17)

where

$\displaystyle{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t}):={\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})+{\lambda}[{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-{\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})]$
(18)

Therefore, (16) can be equivalently represented by:

$\displaystyle{\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})$
$\displaystyle=({\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t}))/\sqrt{\bar{\alpha}_{t}}$
(19) $\displaystyle{\boldsymbol{x}}_{t-1}$
$\displaystyle=\sqrt{\bar{\alpha}_{t-1}}{\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})+\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})$
(20)

which is summarized in Algorithm 2.

By examining Algorithm 1 and Algorithm 2, we observe a notable similarity
between CFG and CFG++. However, a crucial difference exists. Specifically, in
CFG++, the renoising process after applying Tweedie’s formula should utilize
the unconditional noise
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})$
instead of
${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})$.
This surprisingly simple fix to the original CFG algorithm leads to smoother
trajectory of generation, (Fig. CFG++: Manifold-constrained Classifier Free
Guidance for Diffusion Models bottom) and generation with superior quality
(Fig. CFG++: Manifold-constrained Classifier Free Guidance for Diffusion
Models top). Moreover, CFG++ can also be easily extended to significantly
improve DDIM inversion, which is known to be prone to failure by standard CFG
(mokady2023null). See Fig. CFG++: Manifold-constrained Classifier Free
Guidance for Diffusion Models (middle) for representative results and Sec. 4.2
for further discussions. In the following section, we present a theoretical
analysis of CFG++ to investigate the origin of this improvement.

###  3.2 Geometry of CFG++

Mitigating off-manifold phenomenon. In Fig. CFG++: Manifold-constrained
Classifier Free Guidance for Diffusion Models (bottom), we illustrate the
evolution of the posterior mean through Tweedie’s formula during the reverse
diffusion process. Notably, in the early phases of reverse diffusion sampling
under CFG, there is a sudden shift in the image and intense color saturation,
followed by only minor changes until later stages. This abrupt transition in
the posterior mean is somewhat unexpected, as the reverse diffusion process is
designed to progressively refine resolution, moving from a noisier to a
clearer manifold. Similar phenomena are depicted in Fig. 13 in Appendix. We
refer to this as the off-manifold phenomenon, which will be clarified shortly.
Conversely, under CFG++, the posterior mean evolves gradually, transitioning
smoothly from a low-resolution image to high-resolution details. This
indicates that CFG++ is less prone to the off-manifold phenomenon observed
with CFG. In the following, we provide a theoretical analysis of the origin of
the off-manifold phenomenon and explain why CFG++ can mitigate this issue.

By examining (17) and (19) as well as Algorithm 1, we can easily see that the
Tweedie’s formula for the CFG++ and CFG can be equivalently represented by

$\displaystyle{\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})=(1-\lambda){\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t})+\lambda{\hat{\boldsymbol{x}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t}),\quad{\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})=(1-\omega){\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t})+\omega{\hat{\boldsymbol{x}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})$
(21)

This implies that $\lambda,\omega\in[0,1]$ faciliates an interpolation between
the unconditional and the conditional posterior means. However, given the
typical CFG scale $\omega>1.0$, we observe that CFG extrapolates beyond the
unconditional and the conditional posterior means established by Tweedie’s
formula. Consequently, under the usual assumption that the clean manifold to
be piecewise linear around the posterior mean derived via Tweedie’s formula,
the conditional posterior mean estimates from traditional CFG, obtained with a
guidance scale outside the range of $[0,1]$, can extend beyond the piecewise
linear manifold. This may lead to the estimates potentially ‘falling off’ the
data manifold, as depicted by an orange arrow pointing downwards in Fig. 3(a).
We conjecture that this is one of the primary contributors to the off-manifold
phenomenon. Thus, we select $\lambda\in[0,1]$ as the guidance scale for CFG++
to ensure it remains an interpolation between the unconditional and
conditional Tweedie, thus preventing it from ‘falling off’ the clean data
manifold.

An additional source of the off-manifold phenomenon in CFG occurs during the
transition from the clean manifold ${\mathcal{M}}$ to the subsequent noisy
manifold ${\mathcal{M}}_{t-1}$. Specifically, even from the same point in the
clean manifold, the use of
${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})$
for this transition introduces a nonzero offset
$\Delta^{\omega}:=\omega\sqrt{1-\bar{\alpha}_{t-1}}({\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t}))$
from ${\mathcal{M}}_{t-1}$, as illustrated in Fig. 3 (b). This leads to the
invalid lifts from ${\mathcal{M}}$ to ${\mathcal{M}}_{t-1}$. On the other
hand, we note that the renoising process of CFG++ follows the standard DDIM,
resulting in the accurate transition to ${\mathcal{M}}_{t-1}$.

Figure 5: The normalized text-conditioned score matching loss throughout the
reverse diffusion sampling process for both CFG and CFG++.

Text-Image alignment. In Fig. CFG++: Manifold-constrained Classifier Free
Guidance for Diffusion Models (top), we display images generated by SDXL using
the prompt “Kayak in the water, optical color, areal view, rainbow". The
generated image on the left is guided by the traditional CFG, while the one on
the right is produced with our CFG++. Although both images look impressive and
appear to align well with the given text prompt, a closer examination reveals
a significant difference. For instance, one of the oars in the CFG-guided
image is broken in the middle, which shows a misalignment with the text
prompt. This type of artifact is substantially reduced in the CFG++ version,
where the oars are depicted intact. The examples provided by SDXL in Fig. 4
further demonstrate the enhanced text-to-image alignment achieved with CFG++.
For instance, the lion cub is not visible in Fig. 4(top-left), the dog appears
with two tails in Fig. 4(top-right), the goggles have an unusual shape in Fig.
4(bottom-left), and the tree trunk is folded in Fig. 4(bottom-right). These
artifacts, observed in the images generated by CFG, are absent in those
produced by CFG++.

In fact, this enhanced alignment capability is a natural consequence of CFG++,
which directly minimizes the text-conditioned score-matching loss as shown in
(12). In contrast, CFG indirectly seeks text alignment through the sharpened
posterior distribution $p^{\omega}({\boldsymbol{x}}|{\boldsymbol{c}})\propto
p({\boldsymbol{x}})p({\boldsymbol{c}}|{\boldsymbol{x}})^{\omega}$. Therefore,
CFG++ inherently outperform CFG in terms of text alignment due to its
fundamental design principle. For example, Fig. 5 displays the normalized
text-conditioned score matching loss, represented as
$(1-\bar{\alpha}_{t})\|{\boldsymbol{\epsilon}}-{\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},{\boldsymbol{c}})\|^{2}/\bar{\alpha}_{t}=\|{\boldsymbol{x}}-{\hat{\boldsymbol{x}}_{\boldsymbol{c}}}\|^{2}$,
throughout the reverse diffusion sampling process for both CFG and CFG++. The
loss plot associated with CFG shows fluctuations and maintains a noticeable
gap compared to CFG++ even after the completion of the reverse diffusion
process. In fact, the fluctuation associated with CFG is also related to the
off-manifold issue, and from Fig. 5 we can easily see that the off-manifold
phenomenon is more dominant at early stage of reverse diffusion sampling.
Conversely, the loss trajectory for CFG++ demonstrates a much smoother
variation, particularly during the early stages of reverse diffusion.

DDIM inversion. As discussed in (song2020denoising), the denoising process for
unconditional DDIM is approximately invertible, meaning that
${\boldsymbol{x}}_{t}$ can generally be recovered from
${\boldsymbol{x}}_{t-1}$ Specifically, from (4) and (5), we have the following
approximate inversion formula for unconditional DDIM:

$\displaystyle{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t})$
$\displaystyle=({{\boldsymbol{x}}_{t-1}-\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})})/{\sqrt{\bar{\alpha}_{t-1}}}$
$\displaystyle\simeq({{\boldsymbol{x}}_{t-1}-\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t-1})})/{\sqrt{\bar{\alpha}_{t-1}}}$
(22) $\displaystyle{\boldsymbol{x}}_{t}$
$\displaystyle=\sqrt{\bar{\alpha}_{t}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t})+\sqrt{1-\bar{\alpha}_{t}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})$
$\displaystyle\simeq\sqrt{\bar{\alpha}_{t}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t})+\sqrt{1-\bar{\alpha}_{t}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t-1})$
(23)

where the approximation arises from
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})\simeq{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t-1})$.
A similar inversion procedure has been employed for conditional DDIM inversion
under CFG By assuming
${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})\simeq{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t-1})$,
and replacing
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}$
by
${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}$
as detailed in Algorithm 3.

On the other hand, by examining (19) and (20), we can obtain the following
approximate DDIM inversion formula under CFG++:

$\displaystyle{\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})$
$\displaystyle\simeq({{\boldsymbol{x}}_{t-1}-\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t-1})})/{\sqrt{\bar{\alpha}_{t-1}}}$
(24) $\displaystyle{\boldsymbol{x}}_{t}$
$\displaystyle\simeq\sqrt{\bar{\alpha}_{t}}{\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})+\sqrt{1-\bar{\alpha}_{t}}{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t-1})$
(25)

where we apply the approximation
${\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})\simeq{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t-1})$
alongside the usual assumption
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})\simeq{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t-1})$.
This formulation underpins the CFG++ guided DDIM inversion algorithm,
presented in Algorithm 4.

Algorithm 3 DDIM Inversion with CFG

1:${\boldsymbol{x}}_{0},0\leq{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\omega}\in\mathbb{R}$

2:for $i=0$ to $T-1$ do

3:
${\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})={\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})+{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\omega}[{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-{\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})]$

4:
${\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})\leftarrow({\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t}))/\sqrt{\bar{\alpha}_{t}}$

5:
${\boldsymbol{x}}_{t+1}=\sqrt{\bar{\alpha}_{t+1}}{\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})+\sqrt{1-\bar{\alpha}_{t+1}}{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})$

6:end for

7:return ${\boldsymbol{x}}_{T}$

Algorithm 4 DDIM Inversion with CFG++

1:${\boldsymbol{x}}_{0},{\color[rgb]{0.165,0.616,0.561}\definecolor[named]{pgfstrokecolor}{rgb}{0.165,0.616,0.561}\lambda}\in[0,1]$

2:for $i=0$ to $T-1$ do

3:
${\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})={\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})+{\color[rgb]{0.165,0.616,0.561}\definecolor[named]{pgfstrokecolor}{rgb}{0.165,0.616,0.561}\lambda}[{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-{\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})]$

4:
${\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})\leftarrow({\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t}))/\sqrt{\bar{\alpha}_{t}}$

5:
${\boldsymbol{x}}_{t+1}=\sqrt{\bar{\alpha}_{t+1}}{\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})+\sqrt{1-\bar{\alpha}_{t+1}}{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})$

6:end for

7:return ${\boldsymbol{x}}_{T}$

Figure 6: Comparison between DDIM inversion. CFG++ proposes a simple yet
effective fix: using
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})$
instead of
${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})$
in Tweedie’s denoising step.

In practice, for small time step size the error
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})\simeq{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t-1})$
is relatively small, so the unconditional DDIM inversion by (22) and (23) lead
to relatively insignificant errors. Unfortunately, the corresponding inversion
from conditional diffusion under CFG is quite distorted as noted in
(mokady2023null; wallace2023edict). In fact, this distortion is originated
from the inaccuracy of the approximation
${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})\simeq{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t-1})$.
More specifically, even when
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})\simeq{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t-1})$,
the approximation error from the CFG is given by

$\displaystyle\boldsymbol{\varepsilon}_{cfg}$
$\displaystyle:={\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})-{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t-1})=({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})-{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t-1}))+\omega\big{(}\delta{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-\delta{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t-1})\big{)}$
$\displaystyle\simeq\omega\big{(}\delta{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-\delta{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t-1})\big{)},$
(26)

where
$\delta{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t}):={\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})$
denotes the directional component from unconditional to the conditional
diffusion. Since the directional component by the text guidance is not
negligible, this error becomes significant for high guidance scale $\omega$.
Accordingly, the guidance scale must be heavily downweighted in order for
inversions on real world images to be stable, thus limiting the strength of
edits. To mitigate this issue, the authors in (mokady2023null;
wallace2023edict) developed null text optimization and coupled transform
techniques, respectively.

On the other hand, under the usual DDIM assumption
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})\simeq{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t-1})$,
the approximation error of CFG++ mainly arises from (25), which is smaller
than that of CFG since we have

$\displaystyle\|\boldsymbol{\varepsilon}_{cfg++}\|=\lambda\|\delta{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-\delta{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t-1})\|<\|\boldsymbol{\varepsilon}_{cfg}\|$
(27)

thanks to $\lambda<\omega$. Therefore, CFG++ significantly improves the DDIM
inversion as shown Fig. CFG++: Manifold-constrained Classifier Free Guidance
for Diffusion Models (middle) for representative results and Sec. 4.2 for
further discussions.

Figure 7: T2I using SD v1.5, CFG vs CFG++ ($\omega=9.0,\lambda=0.8$).
Unnatural depictions of human hands, and incorrect renderings of the text by
CFG are corrected in CFG++

| $\omega=2.0,\lambda=0.2$ | $\omega=5.0,\lambda=0.4$ | $\omega=7.5,\lambda=0.6$ | $\omega=9.0,\lambda=0.8$ | $\omega=12.5,\lambda=1.0$
---|---|---|---|---|---
Method | FID $\downarrow$ | CLIP $\uparrow$ | FID $\downarrow$ | CLIP $\downarrow$ | FID $\downarrow$ | CLIP $\uparrow$ | FID $\downarrow$ | CLIP $\uparrow$ | FID $\downarrow$ | CLIP $\uparrow$
CFG (ho2021classifierfree) | 13.84 | 0.298 | 15.08 | 0.310 | 17.71 | 0.312 | 20.01 | 0.312 | 21.23 | 0.313
CFG++ (ours) | 12.75 | 0.303 | 14.95 | 0.310 | 17.47 | 0.312 | 19.34 | 0.313 | 20.88 | 0.313

Table 1:  Quantitative evaluation (FID, CLIP-similarity) of T2I with SD v1.5

##  4 Experimental Results

In this section, we design experiments to show the limitation of CFG and how
CFG++ can effectively mitigate these downsides. We select Stable Diffusion
(rombach2022high) (SD) as our baseline model for its widespread use.
Experiments are conducted using SD v1.5 from diffusers (patil2022stable)
unless specified otherwise. Notice that we use the conditioning scale
$\lambda\in[0,1]$ for CFG++ whereas for standard CFG, the typical range is
$\omega>1.0$. When searching for the matched guidance values of $\omega$ and
$\lambda$ for a fair comparison, we fix $\lambda=0.2,0.4,0.6,0.8,1.0$ and find
the $\omega$ values that produce the images that are of closest proximity in
terms of LPIPS distance given the same seed. We found that the corresponding
values were $\omega=2.0,5.0,7.5,9.0,12.5$, respectively, as shown in Appendix
Fig. 14.

###  4.1 Text-to-Image Generation

Using the corresponding scales for $\omega$ and $\lambda$, we directly compare
the performance of the T2I task using SD v1.5 and SDXL. In Tab. 1, we observe
a constant improvement of the FID metric across all guidance scales, with
approximately the same level of CLIP similarity or better. The improvements
can also be clearly seen in Fig. 7 (SD v1.5), where the unnatural components
of the generated images are corrected. Specifically, we see that unnatural
depictions of human hands, and incorrect renderings of the text are corrected
in CFG++, a long-standing research question in and of its own (podell2023sdxl;
pelykh2024giving; chen2023textdiffuser).

Figure 8: Interpolating behavior of T2I along various guidance scales using
SDXL with CFG++. The generated visuals align more significantly with the given
textual description as the guidance scale increase to 1.

In Fig. 8, we visualize multiple images generated by CFG++ as we increase the
guidance scale $\lambda$, illustrating a smooth transition from unconditional
to highly conditional sampling. Specifically, at higher guidance scales, the
target concept is more pronounced, with less emphasis on noisy, unrelated
elements. This results in images that more closely reflect the specified text
conditions. For example, in the second row, the images increasingly resemble a
"painting by Van Gogh," demonstrating how the generated visuals align more
significantly with the given textual description as the guidance scale
increase to 1.

###  4.2 Diffusion image Inversion and Editing

Figure 9: Real image DDIM inversion results using SDv1.5. (a) Reconstructed
samples after inversion by CFG and CFG++. (b) Quantitative comparison between
CFG and CFG++. Figure 10: Real image inversion results at various CFG and
CFG++ scales using SDv1.5. The image is from COCO dataset. We use the text
prompt “a white dog is sleeping on a street and a bicycle” during the
inversion.

We further explore the effect of CFG++ on DDIM inversion
(dhariwal2021diffusion), where the source image is reverted to a latent vector
that can reproduce the original image through generative process. DDIM
inversion is well-known to break down in the usual CFG setting as CFG
magnifies the accumulated error of each inversion step (mokady2023null),
violating local linearization assumptions (wallace2023edict). Remarkably,
CFG++ may mitigate this issue. This improves inversion and enables better
image editing compared to inversion with the original CFG. We evaluate our
method following the experimental setups in park2024energy and
kim2024dreamsampler. More details follow.

Diffusion Image Inversion. Using the matched set of scales for $\omega$ and
$\lambda$, we demonstrate the effect of CFG++ on the diffusion image inversion
task. Specifically, we reconstruct the images after inversion and evaluate it
through PSNR and RMSE, following the methodology of wallace2023edict. In Fig.
9, we illustrate the reconstructed examples (9a) from a real image from COCO
data set and computed metrics (9b) for 5k COCO-2014 (lin2014microsoft)
validation set. Both qualitative and quantitative evaluation demonstrate a
consistent improvement on reconstruction performance induced by CFG++.
Notably, DDIM inversion with CFG++ leads to consistent reconstruction of the
source image across all guidance scales, while DDIM inversion with CFG fails
to reconstruct it, as shown in Fig. 10 for another real image.

Image Editing. Fig. 11 compares the image editing result using CFG and CFG++
followed by image inversion. In the image editing stage, a word in the source
text on the left side is swapped with the target concept, and this modified
text is used as condition for sampling. For both synthetic and real images,
CFG++ successfully edits the target concept while preserving other concepts,
such as background. In particular, the water splashing or brush in the
background, which tend to disappear in the conventional CFG, are maintained
through the inversion process by CFG++. This emphasizes CFG++’s superior
ability to retain specific scene elements that are frequently lost in the
standard CFG approach.

Figure 11: Comparison on image editing via DDIM inversion using SDXL: (top)
synthetic image editing, (bottom) real image editing.

###  4.3 Text-Conditioned Inverse Problems

Inverse problem involves restoring the original data ${\boldsymbol{x}}$ from a
noisy measurement
${\boldsymbol{y}}={\mathcal{A}}({\boldsymbol{x}})+{\boldsymbol{n}}$, where
${\mathcal{A}}$ represents an imaging operator introduces distortions (e.g,
Gaussian blur) and ${\boldsymbol{n}}$ denotes the measurement noise. Diffusion
inverse solvers (DIS) address this challenge by leveraging pre-trained
diffusion models as implicit priors and performing posterior sampling
${\boldsymbol{x}}\sim p({\boldsymbol{x}}|{\boldsymbol{y}})$. Methods that
leverage latent diffusion have gained recent interest (rout2024solving;
song2024solving), but leveraging texts for solving these problems remains
relatively underexplored (chung2023prompt; kim2023regularization). One of the
main reasons for this is that naively using CFG on top of latent DIS leads to
diverging samples (chung2023prompt). Several heuristic modifications such as
null prompt optimization (kim2023regularization) with modified sampling
schemes were needed to mitigate this drawback. This naturally leads to the
question: Is it possible to leverage CFG guidance as a plug-and-play component
of existing solvers? Here, we answer this question with a positive by showing
that CFG++ enables the incorporation of text prompts into a standard solver.
Specifically, we focus on comparing the performance of PSLD (rout2024solving)
combined with CFG and CFG++ in solving linear inverse problems. This
evaluation utilizes the FFHQ (karras2019style) 512x512 dataset and the text
prompt "a high-quality photo of a face". Further details about the
experimental settings can be found in the Appendix A.

As shown in Tab. 2, our method mostly outperforms both the vanilla PSLD and
PSLD with CFG. The superiority is also evident from the Fig. 12, where CFG++
consistently delivers high-quality reconstructions across all tasks. PSLD with
CFG often suffer from artifacts and blurriness. Conversely, CFG++ achieves
better fidelity, clearly distinguishing between faces and faithfully
reproducing fine details like eyelids and hair texture. For more results,
please refer to the Appendix B.

Figure 12: Qualitative comparison on various inverse problems using PSLD
(rout2024solving) under CFG and CFG++.

| SR (x8) | Deblur (motion) | Deblur (gauss) | Inpaint
---|---|---|---|---
Method | FID $\downarrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | FID $\downarrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | FID $\downarrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | FID $\downarrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$
PSLD | 46.24 | 0.413 | 24.41 | 97.51 | 0.500 | 21.83 | 41.65 | 0.388 | 26.88 | 10.27 | 9.36 | 30.15
PSLD + CFG | 41.24 | 0.394 | 24.91 | 91.90 | 0.493 | 22.29 | 41.52 | 0.390 | 26.94 | 9.36 | 0.055 | 30.27
PSLD + CFG++ (ours) | 36.58 | 0.385 | 24.87 | 65.67 | 0.482 | 21.93 | 39.85 | 0.400 | 26.90 | 9.78 | 0.052 | 30.31

Table 2: Quantitative comparison (FID, LPIPS, PSNR) of PSLD, PSLD with CFG,
and PSLD with CFG++ on Latent Diffusion Inverse Solver.

##  5 Related Works and Discussion

Related works. kynkaanniemi2024applying observed that applying CFG guidance at
the earlier stages of sampling always led to detrimental effects and
drastically reduced the diversity of the samples. The guidance at the later
stages of sampling had minimal effects. Drawing upon these observations, it
was empirically shown that applying CFG only in the limited time interval near
the middle led to the best performance. Similar observations were made in the
SD community (howard2022; birch2023) in adjusting the guidance scale across
$t$. These findings are orthogonal to ours in that these methods keep the
sampling trajectory the same and try to empirically adjust the strength of the
guidance, while we aim to design a different trajectory.

Implications. In several studies, attempts at designing a better trajectory of
the generative path were made (liu2023flow; lipman2023flow). As diffusion
models can only estimate the tangent vector of the trajectory at time $t$, it
is now a well regarded consensus that minimizing the curvature of the
trajectory leads to faster sampling (liu2023flow; lee2023minimizing) and
better distillation results (liu2023instaflow). In this regard, the
excessively high curvature and oscillating behavior of CFG with high guidance
scales would hamper the performance of fast sampling and distillation, which
was constantly reported in the literature as difficulties in extending
distillation methods (salimans2022progressive; song2023consistency) to the
latent space (meng2023distillation; luo2023latent; chen2024pixart).
Straightening the diffusion path prior to distillation with Reflow
(liu2023flow) exhibited much better results than aiming for direct
distillation. In this regard, CFG++ can also be considered a better
alternative to CFG for enhanced distillation performance.

##  6 Conclusion

This paper revises the most widespread guidance method, classifier-free
guidance (CFG), highlighting that a small and reasonable guidance scale, e.g.
$\lambda\in[0.0,1.0]$, might suffice successful guidance with a proper
geometric correction. Observing that the original CFG suffers from off-
manifold issues during sampling, we propose a simple but surprisingly
effective fix: using
${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})$
instead of
${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})$
in updating ${\boldsymbol{x}}_{t-1}$. This change translates the conditional
guidance from extrapolating to interpolating between unconditional and
conditional sampling trajectories, leading to more interpretable guidance in
contrast to the conventional CFG which heavily relies on heuristic $\omega$
scaling. Given that CFG++ mitigates off-manifold issues, it may be beneficial
for other downstream applications requiring accurate latent denoised estimate
representations, e.g. DIS, as demonstrated in our experiments.

## References

  * Birch (2023) Alex Birch.  Turning off classifier-free guidance at low noise levels.  https://twitter.com/Birchlabs/status/1640033271512702977, 2023.  Idea mentioned on Twitter.
  * Chen et al. (2023) Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei.  Textdiffuser: Diffusion models as text painters.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=ke3RgcDmfO.
  * Chen et al. (2024a) Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li.  Pixart-delta: Fast and controllable image generation with latent consistency models.  _arXiv preprint arXiv:2401.05252_ , 2024a.
  * Chen et al. (2024b) Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li.  PixArt-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis.  In _The Twelfth International Conference on Learning Representations_ , 2024b.  URL https://openreview.net/forum?id=eAKmQPe3m1.
  * Chung et al. (2022) Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye.  Improving diffusion models for inverse problems using manifold constraints.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=nJJjv0JDJju.
  * Chung et al. (2023a) Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye.  Diffusion posterior sampling for general noisy inverse problems.  In _International Conference on Learning Representations_ , 2023a.  URL https://openreview.net/forum?id=OnD9zGAGT0k.
  * Chung et al. (2023b) Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio.  Prompt-tuning latent diffusion models for inverse problems.  _arXiv preprint arXiv:2310.01110_ , 2023b.
  * Chung et al. (2024) Hyungjin Chung, Suhyeon Lee, and Jong Chul Ye.  Decomposed diffusion sampler for accelerating large-scale inverse problems.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=DsEhqQtfAG.
  * Dhariwal & Nichol (2021) Prafulla Dhariwal and Alexander Quinn Nichol.  Diffusion models beat GANs on image synthesis.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems_ , 2021.
  * Du et al. (2023) Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl.  Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc.  In _International Conference on Machine Learning_ , pp. 8489–8510. PMLR, 2023.
  * Efron (2011) Bradley Efron.  Tweedie’s formula and selection bias.  _Journal of the American Statistical Association_ , 106(496):1602–1614, 2011.
  * Ho & Salimans (2021) Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  In _NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications_ , 2021.  URL https://openreview.net/forum?id=qw8AKxfYbI.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Howard & Prashanth (2022) Jeremy Howard and Rekil Prashanth.  Adjusting guidance weight as a function of time.  https://twitter.com/jeremyphoward/status/1584771100378288129, 2022.  Idea mentioned on Twitter.
  * Hyvärinen & Dayan (2005) Aapo Hyvärinen and Peter Dayan.  Estimation of non-normalized statistical models by score matching.  _Journal of Machine Learning Research_ , 6(4), 2005.
  * Kadkhodaie & Simoncelli (2021) Zahra Kadkhodaie and Eero P Simoncelli.  Stochastic solutions for linear inverse problems using the prior implicit in a denoiser.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=x5hh6N9bUUb.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4401–4410, 2019.
  * Karras et al. (2022) Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  In _Proc. NeurIPS_ , 2022.
  * Kim et al. (2023) Jeongsol Kim, Geon Yeong Park, Hyungjin Chung, and Jong Chul Ye.  Regularization by texts for latent diffusion inverse solvers.  _arXiv preprint arXiv:2311.15658_ , 2023.
  * Kim et al. (2024) Jeongsol Kim, Geon Yeong Park, and Jong Chul Ye.  Dreamsampler: Unifying diffusion sampling and score distillation for image manipulation.  _arXiv preprint arXiv:2403.11415_ , 2024.
  * Kynkäänniemi et al. (2024) Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen.  Applying guidance in a limited interval improves sample and distribution quality in diffusion models.  _arXiv preprint arXiv:2404.07724_ , 2024.
  * Lee et al. (2023) Sangyun Lee, Beomsu Kim, and Jong Chul Ye.  Minimizing trajectory curvature of ode-based generative models.  In _International Conference on Machine Learning_ , pp. 18957–18973. PMLR, 2023.
  * Liang et al. (2023) Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen.  Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching.  _arXiv preprint arXiv:2311.11284_ , 2023.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pp. 740–755. Springer, 2014.
  * Lipman et al. (2023) Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le.  Flow matching for generative modeling.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=PqvMRDCJT9t.
  * Liu et al. (2023a) Xingchao Liu, Chengyue Gong, and qiang liu.  Flow straight and fast: Learning to generate and transfer data with rectified flow.  In _The Eleventh International Conference on Learning Representations_ , 2023a.  URL https://openreview.net/forum?id=XVjTT1nw5z.
  * Liu et al. (2023b) Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al.  Instaflow: One step is enough for high-quality diffusion-based text-to-image generation.  In _The Twelfth International Conference on Learning Representations_ , 2023b.
  * Luo et al. (2023) Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao.  Latent consistency models: Synthesizing high-resolution images with few-step inference.  _arXiv preprint arXiv:2310.04378_ , 2023.
  * Meng et al. (2023) Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans.  On distillation of guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 14297–14306, 2023.
  * Mokady et al. (2023) Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6038–6047, 2023.
  * Park et al. (2024) Geon Yeong Park, Jeongsol Kim, Beomsu Kim, Sang Wan Lee, and Jong Chul Ye.  Energy-based cross attention for bayesian context update in text-to-image diffusion models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Patil et al. (2022) Suraj Patil, Pedro Cuenca, Nathan Lambert, and Patrick von Platen.  Stable diffusion with diffusers.  _Hugging Face Blog_ , 2022.  https://huggingface.co/blog/stable_diffusion.
  * Pelykh et al. (2024) Anton Pelykh, Ozge Mercanoglu Sincan, and Richard Bowden.  Giving a hand to diffusion models: a two-stage approach to improving conditional human image generation.  _arXiv preprint arXiv:2403.10731_ , 2024.
  * Podell et al. (2023) Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_ , 2023.
  * Poole et al. (2022) Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv_ , 2022.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10684–10695, 2022.
  * Rout et al. (2024) Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai.  Solving linear inverse problems provably via posterior sampling with latent diffusion models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In _ACM SIGGRAPH 2022 conference proceedings_ , pp. 1–10, 2022.
  * Salimans & Ho (2022) Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=TIdIXIpzhoI.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Song et al. (2024) Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen.  Solving inverse problems with latent diffusion models via hard data consistency.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=j8hdRqOUhN.
  * Song et al. (2021a) Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _9th International Conference on Learning Representations, ICLR_ , 2021a.
  * Song et al. (2023a) Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz.  Pseudoinverse-guided diffusion models for inverse problems.  In _International Conference on Learning Representations_ , 2023a.  URL https://openreview.net/forum?id=9_gsMA8MRKQ.
  * Song et al. (2021b) Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _9th International Conference on Learning Representations, ICLR_ , 2021b.
  * Song et al. (2023b) Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  _arXiv preprint arXiv:2303.01469_ , 2023b.
  * Wallace et al. (2023) Bram Wallace, Akash Gokul, and Nikhil Naik.  EDICT: Exact diffusion inversion via coupled transformations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 22532–22541, 2023.
  * Wang et al. (2024) Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Zhu et al. (2023) Junzhe Zhu, Peiye Zhuang, and Sanmi Koyejo.  Hifa: High-fidelity text-to-3d generation with advanced diffusion guidance.  In _The Twelfth International Conference on Learning Representations_ , 2023.

##  Appendix A Experimental details

###  A.1 Text-Conditioned Inverse Problems

Problem settings. We evaluate our approach across following degradation types:
1) Super-resolution with a scale factor of x8, 2) Motion deblurring from an
image convolved with a 61 x 61 motion kernel, randomly sampled with an
intensity value $0.3^{2}$, 3) Gaussian deblurring from an image convolved with
a 61 x 61 kernel with an intensity value 0.5, and 4) Inpainting from 10-20%
free-form masking, as implemented in (saharia2022palette). In inpainting
evaluations, regions outside the mask are overlaid with the ground truth
image.

Datasets, Models. For evaluation, we use the FFHQ (karras2019style) 512x512
dataset and follow (chung2023diffusion) by selecting the first 1,000 images
for testing. For the pre-trained latent diffusion model including baseline
methods, we choose SD v1.5 trained on the LAION dataset. As a baseline for
latent DIS, we consider PSLD (rout2024solving). It enforces fidelity by
projecting onto the subspace of ${\mathcal{A}}$ during the intermediate step
between decoding and encoding, in conjunction with the DPS
(chung2023diffusion) loss term. For gradient updates in both vanilla PSLD and
PSLD with CFG, we use static step sizes of $\eta=1.0$ and $\gamma=0.1$ as
recommended in (rout2024solving). For CFG scale $\omega$, we applied the
corresponing scales that we found to be corresponded for CFG++ scale
$\lambda$. Please refer to the Tab. 3 for the hyperparameters used for PSLD
with CFG++.

| SR(x8) | Deblur(motion) | Deblur(gauss) | Inpaint
---|---|---|---|---
$\eta$ | 1.3 | 0.4 | 1.0 | 1.0
$\gamma$ | 0.1 | 0.025 | 0.12 | 0.1
$\lambda$ | 0.1 | 0.2 | 0.2 | 0.6
$\omega$ | 1.5 | 2.0 | 2.0 | 7.5

Table 3: Hyperparameters for PSLD (rout2024solving) with CFG++ and
corresponding CFG scale $\omega$.

##  Appendix B Further experimental results

###  B.1 T2I

Since the range of guidance scales for CFG and CFG++ is different, we matched
the guidance values of $\omega$ and $\lambda$ for comparison by computing the
LPIPS distance using the same seed. Fig. 14 displays an example of generated
images given the same text prompt and seed, while varying the guidance scale.

Fig. 13 shows the clean estimates at each diffusion sampling time $t$,
computed using Tweedie’s formula, both with and without CFG. It is evident
that the original CFG induces significant error, particularly at earlier
times, leading to off-manifold samples. However, the CFG++ addresses this
issue by adjusting the DDIM re-noising step and the guidance scale.

Figure 13: The discrete evolution of the posterior mean in CFG and CFG++.
Denoised estimates in latent space are decoded into pixel space at each
timestep using SDXL. Figure 14: Example of generated images with various
guidance scales by CFG and CFG++. The respective guidance scales are chosen to
best match the generated images by CFG and CFG++.

###  B.2 Real Image Editing

In Fig. 15-17, we provide qualitative comparison on real image editing via
DDIM inversion with CFG and CFG++. For the DDIM inversion stage, we use "a
photography of [source concept]" as conditioning prompt. For the sampling
stage, we swap [source concept] to [target concept] and use it as conditioning
prompt for generation. For example, in Fig. 15, we set [source concept] to
"dog" and [target concept] to "cat". For all experiments, we set the guidance
scale as $\omega=9.0$ and $\lambda=0.8$ as described in the main paper. The
comparison demonstrates that CFG++ successfully edits the given image which
was not possible by CFG. This results also support our claim on reduced error
during DDIM inversion by CFG++.

Figure 15: Comparison on real image editing via DDIM inversion under CFG and
CFG++ using SDXL. Cat $\rightarrow$ Dog. Figure 16: Comparison on real image
editing via DDIM inversion under CFG and CFG++ using SDXL. Horse $\rightarrow$
Zebra. Figure 17: Comparison on real image editing via DDIM inversion under
CFG and CFG++ using SDXL. Cat $\rightarrow$ Cat with glasses.

###  B.3 Text-Conditioned Inverse Problems

In Fig. 18-21, we display additional qualitative comparison for text-
conditioned inverse problem solver with CFG and CFG++. Experiments are
conducted with FFHQ (512x512) validation set and CFG++ consistently leads to
better reconstruction of true solution for various tasks.

Figure 18: Results of PSLD using CFG and CFG++ on the FFHQ dataset at Super-
Resolution (x8). Figure 19: Results of PSLD using CFG and CFG++ on the FFHQ
dataset at Motion Deblurring. Figure 20: Results of PSLD using CFG and CFG++
on the FFHQ dataset at Gaussian Deblurring. Figure 21: Results of PSLD using
CFG and CFG++ on the FFHQ dataset at Inpainting.