# Upsample Guidance: Scale Up Diffusion Models without Training

Juno Hwang  Yong-Hyun Park  Junghyo Jo

###### Abstract

Diffusion models have demonstrated superior performance across various
generative tasks including images, videos, and audio. However, they encounter
difficulties in directly generating high-resolution samples. Previously
proposed solutions to this issue involve modifying the architecture, further
training, or partitioning the sampling process into multiple stages. These
methods have the limitation of not being able to directly utilize pre-trained
models as-is, requiring additional work. In this paper, we introduce upsample
guidance, a technique that adapts pretrained diffusion model (e.g., $512^{2}$)
to generate higher-resolution images (e.g., $1536^{2}$) by adding only a
single term in the sampling process. Remarkably, this technique does not
necessitate any additional training or relying on external models. We
demonstrate that upsample guidance can be applied to various models, such as
pixel-space, latent space, and video diffusion models. We also observed that
the proper selection of guidance scale can improve image quality, fidelity,
and prompt alignment.

Machine Learning, ICML

Figure 1:  High-resolution samples with upsample guidance. The original
trained resolution is increased ($\geq 2$ times) through upsample guidance.
(a) Images sampled at twice the resolution for the models trained on CIFAR-10
and CelebA-HQ datasets at $32^{2}$ and $256^{2}$ resolutions, respectively.
The adjacent image pairs are sampled from the same initial noise. (b) High-
resolution images of latent diffusion models using upsample guidance. (c)
Upsampled snapshots of text-to-video models. The upper panel represents
spatial upsampling, while the lower panel represents temporal upsampling.

##  1 Introduction

Diffusion models are generative models that generate samples by progressively
restoring the original distribution from prior noise distribution, by modeling
the reverse process of data diffusion. (Ho et al., 2020; Song et al., 2020a,
b) Recently, diffusion models have demonstrated state-of-the-art performance
in various domains such as image (Ho et al., 2022a; Ramesh et al., 2022;
Rombach et al., 2022; Podell et al., 2023), video (Ho et al., 2022c; Guo et
al., 2023; Blattmann et al., 2023b, a), audio, and 3D generation (Poole et
al., 2022).

Despite its effectiveness in image generation, generating high-resolution
images remains a challenging problem. To circumvent this issue, researcher
suggest to operate in lower dimensional latent space (latent diffusion models,
LDMs) (Rombach et al., 2022), or generate low-resolution images and then
upscale them with super-resolution diffusion models (cascaded diffusion model,
CDM) (Ho et al., 2022b) or mixtures-of-denoising-experts (eDiff-I) (Balaji et
al., 2022). Recently, end-to-end high resolution image generation have been
proposed, either by improving the training loss (Hoogeboom et al., 2023) or by
generating multiple resolutions simultaneously (Gu et al., 2023).

The aforementioned solutions involve require from-scratch training or fine-
tuning, requiring additional computation cost. In this paper, we introduce a
novel technique, upsample guidance, which enables higher resolution sampling
without any additional training or external models, but simply by adding a
single term involving minimal computation.

As shown in Figure 1, upsample guidance can be universally introduced to any
types of diffusion model, including pixel-space, latent-space, or even video
diffusion model. Moreover, it is fully compatible with any diffusion models or
all previously proposed techniques that improve or control diffusion models
such as SDEdit (Meng et al., 2021), ControlNet (Zhang et al., 2023), LoRA (Hu
et al., 2021; Ryu, 2022), and IP-Adapter (Ye et al., 2023). Surprisingly, our
method can even allows to generate higher resolution images that never shown
in the training dataset, such as $64^{2}$ resolution images of CIFAR-10
dataset (Krizhevsky et al., 2009), which has $32^{2}$ resolution images.

We demonstrate the results of applying upsample suidance across numerous pre-
trained diffusion models and compare these with cases where our method was not
applied. Additionally, we show the feasibility of spatial and temporal
upsampling in video generation models. Finally, we conduct experiments on the
selection of an appropriate guidance scale.

##  2 Related Works

Various ideas have been proposed for generating high-resolution samples using
diffusion models. However, many of these require modifications to the
architecture or traning from scratch. Here, we focus on a method that
leverages pre-trained models to generate at resolutions higher than their
trained resolution.

###  2.1 Super-Resolution

An intuitive solution is to use pre-trained models to generate low-resolution
samples and then upscale them to high resolution through a super-resolution
model. Cascaded diffusion models (CDM) perform super-resolution using
diffusion models that take low-resolution images as a condition (Ho et al.,
2022b) . This method has been applied to high-performance text-to-image models
such as IMAGEN (Ho et al., 2022a) and DeepFloyd-IF (StabilityAI, 2023).
However, this approach involves muptiple diffusion models and sampling
process, requiering additional training and heavy computational cost.

A similar method in practice involves upscaling an image generated by a
diffusion model using a relatively lightweight super-resolution model,
followed by applying SDEdit (Meng et al., 2021) with the same diffusion model
to enhance details in the high-resolution image. This technique is implemented
under the name ”HiRes.fix” in a well-known web-based diffusion model UI
(Automatic1111, 2022). However, a drawback is the additional encoding and
decoding operations required when it is combined with LDMs.

###  2.2 Fine-Tuning

Even models trained at a fixed low resolution can generate higher resolution
images better when fine-tuned on datasets with higher resolutions and various
aspect ratios (Zheng et al., 2023). For instance, although the Stable
Diffusion v1.5 model (Rombach et al., 2022) was trained at a $512^{2}$
resolution, several models fine-tuned near a $768^{2}$ resolution are widely
used. However, as the resolution increases, the computational cost required
for fine-tuning also rises considerably, making it challenging to train on
higher resolutions.

##  3 Background

In this section, we introduce the fundamental concepts of diffusion models and
guidance, which are essential for understanding our method.

###  3.1 Diffusion Models

Diffusion models transform an original sample $x_{0}$ from the dataset into a
noised sample $x_{t}$ through a forward diffusion process, eventually reaching
pure noise $x_{T}$ that can be easily sampled. Many diffusion models follow
the formalization of denoising diffusion probabilistic models (DDPMs) (Ho et
al., 2020) that use Gaussian noise. Specifically, the noised sample is given
by:

$x_{t}=\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\epsilon_{t},$ (1)

which represents a linear combination of the signal $x_{0}$ and noise
$\epsilon_{t}\sim\mathcal{N}(0,I)$. The term $\alpha_{t}$ is a noise schedule
that affects the signal-to-noise ratio
$\textrm{SNR}=\frac{\alpha_{t}}{1-\alpha_{t}}$ and monotonically decreases
with respect to time $t$.

The generation in DDPMs corresponds to a backward diffusion process, starting
from $x_{T}$ and approximately sampling the distribution $p(x_{t-1}|x_{t})$.
The expected value of this conditional distribution is estimated by a noise
predictor $\epsilon(x_{t},t)$, which takes the noised sample $x_{t}$ and time
$t$ as inputs to predict $\epsilon_{t}$. Note that the noise predictor has a
U-Net architecture (Ronneberger et al., 2015), allowing it to accept inputs at
resolutions other than the trained resolution. This flexibility underscores
the adaptability of the model to handle a variety of image sizes.

###  3.2 Guidances for Diffusion Models

Techniques have been proposed for conditionally sampling images corresponding
to specific classes or text prompts by adding a guidance term to the predicted
noise. Ho & Salimans (2022) added the gradient of the log probability
predicted by a classifier to $\epsilon(x_{t},t)$, enabling an unconditional
diffusion model to generate class-conditioned images. Subsequently,
classifier-free guidance (CFG) was proposed. Instead of using a classifier,
the noise predictor’s architecture was modified to accept condition $c$ as an
input. The following formula is then used as the predicted noise:

$\tilde{\epsilon}(x_{t},t;c)={\underbrace{\epsilon(x_{t},t)}_{\text{denoise}}}+w{\underbrace{[\epsilon(x_{t},t;c)-\epsilon(x_{t},t)]}_{\text{guidance}}}.$
(2)

Here, $w$ represents the guidance scale. It has been commonly observed that
proper adjustment of the scale improves the alignment with the condition and
the fidelity of the generated images.

##  4 Method

In this section, we first introduce signal-to-noise ratio (SNR) matching, a
core concept crucial for understanding our method, and then derive upsample
guidance based on it. We also explain the considerations when applying it to
LDMs that include an encoder-decoder structure.

###  4.1 SNR Matching

Figure 2:  Consistency between different resolutions. (a) Downsampled image
generated by the diffusion model at the target resolution. (b) Image generated
at the trained resolution. The noise reduction due to downsampling creates a
significant difference in the recognizability between the central and right
images at the trained resolution, indicating a change in their signal-to-noise
ratio. For this example, $\alpha_{t}=0.85$ is used.

When a diffusion model is trained at a resolution of $r\times r$, consider the
scenario of generating images at a target (high) resolution that is $m$ times
higher, namely $mr\times mr$. If images can be ideally generated at all
resolutions, it is reasonable to expect that the result of downsampling the
target resolution by a scale of $1/m$ should resemble the outcome sampled at
the trained (low) resolution. In this context, let’s assume that the same
model takes a downsampled image $\mathbf{D}[x_{t}]$ as input during the
generation process. Here, $\mathbf{D}$ represents a downsampling operator that
takes the average of $m\times m$ pixels.

As Figure 2 demonstrates, this downsampled image $\mathbf{D}[x_{t}]$ (the
second image) significantly differs from $x^{\textrm{low}}_{t}$ (the third
image) that was generated from the diffusion model trained at the low
resolution. More specifically, the trained image follows

$x^{\textrm{low}}_{t}=\sqrt{\alpha_{t}}\mathbf{D}[x_{0}]+\sqrt{1-\alpha_{t}}\epsilon^{\textrm{low}}_{t},$
(3)

where $\epsilon^{\textrm{low}}_{t}$ represents the standard Gaussian noise
with a size of trained resolution. However, the downsampled image can be
obtained from Equation (1) by applying the linear downsampling operator
$\mathbf{D}$,

$\mathbf{D}[x_{t}]=\sqrt{\alpha_{t}}\mathbf{D}[x_{0}]+\frac{1}{m}\sqrt{1-\alpha_{t}}\epsilon^{\textrm{low}}_{t}.$
(4)

Here, the standard deviation of noise is reduced to $1/m$, because
$\mathbf{D}$ averages every $m\times m$ pixels. This directly results form the
central limit theorem. Therefore, some adjustments are necessary to make
$x^{\textrm{low}}_{t}$ and $\mathbf{D}[x_{t}]$ equivalent (Hwang et al.,
2023). This requires matching both SNR and overall power:

$\displaystyle\mathrm{SNR}^{\mathrm{low}}=\frac{m\alpha_{t}}{1-\alpha_{t}}=m\cdot\mathrm{SNR},$
(5) $\displaystyle P=\alpha_{t}+\frac{1}{m^{2}}(1-\alpha_{t}).$ (6)

Since the SNR is a function of time determined by the noise schedule
$\alpha_{t}$, we can find the adjusted time $\tau$ such that
$\mathrm{SNR}^{\mathrm{low}}(\tau)=m\cdot\mathrm{SNR}(t)$. Furthermore, by
multiplying by $1/\sqrt{P}$, we can make the overall power equivalent to that
of the target resolution. Therefore, the proper noise predictors of high and
low resolutions are associated with time and power adjustments as follows

$\mathbf{D}[\epsilon^{\mathrm{adj}}(x_{t},t)]=\frac{1}{m}\epsilon\left(\frac{1}{\sqrt{P}}\mathbf{D}[x_{t}],\tau\right),$
(7)

where the factor of $1/m$ is multiplied to adjust the variance of trained
resolution to 1.

###  4.2 Upsample Guidance

Figure 3:  Conceptual illustration of upsample guidance. The model receives
the same noised images at two different resolutions in parallel, but time and
power are adjusted at the trained resolution. The difference between the two
predicted noises then acts as guidance, which is added to the total noise.

Considering the consistency across diverse resolutions, we explore the
decomposition of predicted noise at the target (high) resolution. Suppose that
the target noise comprises both a component from the trained (low) resolution
and its corresponding residual part:

$\epsilon(x_{t},t)=\mathbf{U}\underbrace{\mathbf{D}[\epsilon(x_{t},t)]}_{\textrm{trained
resolution}}+\underbrace{\\{\epsilon(x_{t},t)-\mathbf{U}\mathbf{D}[\epsilon(x_{t},t)]\\}}_{\textrm{target
resolution}}.$ (8)

In this context, $\mathbf{U}$ represents the nearest upsampling operator with
a scale factor of $m$, utilized to align dimensions between target and trained
noise predictors. The residual noise,
$\epsilon-\mathbf{U}\mathbf{D}[\epsilon]$, corresponds to the part that
remains after removing the contribution of the low resolution.

Now, recognizing the need for adjustments to ensure consistency among noise
predictors at various resolutions, we substitute the term about trained
resolution with the adjusted noise predictor in Equation (7), as follows:

$\displaystyle\epsilon^{\mathrm{adj}}(x_{t},t)=$
$\displaystyle\mathbf{U}\underbrace{\left[\frac{1}{m}\epsilon\left(\frac{1}{\sqrt{P}}\mathbf{D}[x_{t}],\tau\right)\right]}_{\textrm{trained
resolution}}+$
$\displaystyle\underbrace{\\{\epsilon(x_{t},t)-\mathbf{U}\mathbf{D}[\epsilon(x_{t},t)]\\}}_{\textrm{target
resolution}}.$ (9)

This model parallelly sees and predicts noises at both resolutions.

Finally, we consider interpolation between the naive sampling at the targe
resolutin with the parallel sampling at the trained resolution,

$\displaystyle\tilde{\epsilon}(x_{t},t)=(1-w_{t})\epsilon(x_{t},t)+w_{t}\epsilon^{\mathrm{adj}}(x_{t},t)=$
$\displaystyle\epsilon(x_{t},t)+w_{t}\underbrace{\mathbf{U}\left[\frac{1}{m}\epsilon\left(\frac{1}{\sqrt{P}}\mathbf{D}[x_{t}],\tau\right)-\mathbf{D}[\epsilon(x_{t},t)]\right]}_{\textrm{upsample
guidance}}.$ (10)

This structure resembles Equation (2) and can be interpreted similarly as
guidance. Consequently, we named it “upsample guidance (UG),” where $w_{t}$
functions as the guiding scale, which may generally depend on time. Similar to
how CFG incorporates the shift from unconditional to conditional noise, UG
represents the influence pushing the model towards consistency with the
trained low-resolution component.

###  4.3 Adaptation on LDMs

Figure 4:  Artifacts of encoder-decoder in a LDM. When an image is upsampled
or downsampled in the latent space of an LDM and then decoded back into pixel
space, artifacts are introduced. The variational autoencoder introduces
nonlinearity in the implementation of upsample guidance, and significant
degradation can be observed in both cases.

Figure 5: Upsampling across various image generation models, resolutions, and
conditional generation methods. Unconditional image generation, such as
CIFAR-10 and CelebA-HQ, was sampled in the pixel space. For the text-to-image
models, the left side of the images represents results without UG, while the
right side shows results with UG. We used DreamShaper (Lykon, 2023) as an
example of fine-tuned LDM. The paired images are all generated from the same
initial noise. Across different models, resolutions, prompts, and
conditioning, consistently better images were obtained with UG. Notably, our
method effectively resolved artifacts where multiple subjects were generated
or bad anatomy was present.

The aforementioned derivation heavily relies on the linearity of operators
$\mathbf{U}$ and $\mathbf{D}$. Nonetheless, in the context of LDMs, the pixel
space undergoes a transformation into the latent space using a nonlinear
variational autoencoder (VAE) (Kingma & Welling, 2013). Consequently, it is
crucial to proceed with caution, as the latent space of a downsampled image at
the target resolution may not align with the latent space of the resolution it
was originally trained on. The outcomes of downsampling in latent space and
subsequently decoding back into pixel space are shown in Figure 4.

However, we discovered a viable solution to this challenge by heuristically
tailoring $w_{t}$ to be time-dependent. Specifically, we designed $w_{t}$ to
decrease or be set to zero when $t$ is close to zero, preventing the upsample
guidance from introducing artifacts. While various designs for $w_{t}$ are
conceivable, in Section 5.4, we introduce the most straightforward
parameterized design using the Heaviside step function $H$ to investigate the
influence of scale magnitude $\theta$ and time threshold $\eta$. The
formulation is expressed as follows:

$w_{t}=\theta\cdot H(t-(1-\eta)T).$ (11)

##  5 Experiments

The core concept behind upsample guidance lies in the SNR matching during the
downsampling process. As a result, it can be extended to diverse data
generation tasks, not confined to images alone. Moreover, its compatibility
extends to any pre-trained model, conditional generation, and application
techniques. In this section, we showcase the outcomes when applied to various
image generation models and applications. Specifically, we explore spatial and
temporal upsampling in video generation. Subsequently, we conduct an ablation
study to evaluate scenarios where the adjustments proposed in Section 4.2 are
not implemented. Lastly, a quantitative analysis on the guidance scale is
performed to help the design of $w_{t}$.

###  5.1 Image Upsampling

As upsample guidance requires only a straightforward linear operation on the
predicted noise, it exhibits compatibility with a wide array of models and
applications. In our study, we used a pre-trained unconditional model trained
on CIFAR-10 and CelebA-HQ $256^{2}$ (Karras et al., 2018) datasets to generate
images at twice the resolution using a constant $w_{t}$. We also sampled using
UG with $m=2$ for text-to-image models based on stable diffusion v1-5, and
checked its capability on a fine-tuned model, different aspect ratios and
image conditioning techniques.

Figure 5 presents images that are slightly cherry-picked to aid in
understanding the impact of upsample guidance. Each image pair contains images
generated from the same initial noise. For images with different resolutions,
initial noise was resized and its variance was adjusted accordingly. Upon
carefully examining some samples from CIFAR-10, UG sometimes alters coarse
contents (overall colors and shapes) between $32^{2}$ and $64^{2}$
resolutions, with details emerging at higher resolutions that were not present
at lower ones. This suggests that UG does more than just interpolation or
sharpening; it actually generates new meaningful features. For the comparison
between low and high resolution in LDMs and more extensive non-cherry-picked
samples, please refer to Appendix B.

We roughly measured the additional computational time introduced by UG. In
Stable Diffusion v1-5, using an RTX3090 GPU at $1024^{2}$ resolution with a
scale factor of $m=2$ and $\eta=0.5$, we measured the wall time from sampling
in the latent space to converting into an RGB image, as shown in the Figure 6.
The extra computation for UG is minimal, given that the dimension of the noise
prediction $\epsilon(\frac{1}{\sqrt{P}}\mathbf{D}[x_{t}],\tau)$ at trained
resolution is $1/m^{2}$ times the dimension of $\epsilon(x_{t},t)$.
Additionally, this supplementary computation is only applied when
$t\geq(1-\eta)T$. Therefore, the cost is less than $\eta/m^{2}=1/8$ of the
naive sampling time. However, in LDMs, decoding also consumes the time, so the
portion of cost due to UG decreases as the sampling step becomes shorter. With
the recent advancements in sampling methods (Song et al., 2020a; Liu et al.,
2022; Luo et al., 2023) leading to a reduction in the number of inference
steps, our method becomes more competitive, requiring only $\leq 10\%$
additional computation cost within 20 inference steps.

Figure 6:  Computational cost comparison for upsample guidance (UG). Wall time
for computation is compared with and without the use of UG. The percentages on
the bars indicate the proportion of additional time attributed to UG.

###  5.2 Video Upsampling

Figure 7:  Spatial and temporal video upsampling. Frames of videos are
generated using AnimateDiff with UG applied. (a) Spatial upsampling by a
factor of 2, similar to images. (b) Temporal upsampling with the number of
frames upsampled by a factor of 2. Note that for visibility, only odd-numbered
frames from each sequence are displayed.

Upsample guidance can also enhance video upsampling by addressing both spatial
and temporal resolution. To illustrate this, we employ AnimateDiff (Guo et
al., 2023), a video generation model that integrates a motion module into a
text-to-image model. In AnimateDiff, a video is represented as a sequence of
color, time, width, and height in latent space, basically a tensor with the
shape [C, T, W, H]. While we can upsample in the spatial dimensions [W, H] as
above, it’s also possible to upsample in the temporal dimension T, increasing
the number of frames by a factor of $m$. Assuming UG gives robustness for
temporal resolution, we expect an increase in frames per second rather than an
extension of time length, similar to the case with images.

Figure 7 shows the results of applying UG across these two dimensions. For
spatial upsampling, issues like multiple subjects appearing and misalignment
with text prompts were resolved thanks to UG, indicating that spatial UG works
similarly in video generation as it does for images.

For temporal upsampling, we kept the spatial size constant and generated 32
frames, double the 16 frames AnimateDiff was trained on. Without UG, there was
a complete failure in maintaining temporal consistency, and sometimes even
adjacent frames lost continuity. However, with UG, the videos were overall
consistent at a level similar to the trained temporal resolution, and greater
continuity was also appeared in the subject’s movements. This difference is
more pronounced when viewing the videos in playback rather than as listed
frames.

###  5.3 Ablation Study on Time and Power Adjustments

Figure 8:  Effects of time and power adjustments in UG is indicated by red and
blue dashed boxes, respectively. Two images are generated from (a) the CelebA-
HQ model and (b) the text-to-image model, respectively, with and without
either time adjustment, power adjustment, or both.

So far, we have seen that our method effectively suppresses artifacts that
could occur at higher resolutions. However, some might question the necessity
of the time and power adjustment presented in Equation 7. Therefore, we
illustrate here that each adjustment is indeed essential, and how the images
ruined when either one or both adjustments are not made.

As illustrated in Figure 8, both adjustments are essential, and the absence of
each leads to image degradation in different ways. Without $\tau$ in Equation
7 (outside the red dashed boxes), the model fails in denoising due to the
mismatch between the learned SNR at the time and the SNR of the input noised
sample, resulting in residual noise. Without $1/\sqrt{P}$ (outside the blue
dashed boxes), the model confronts samples with variances it has never
learned, resulting in complete failure. This suggests that the noise predictor
$\epsilon(x_{t},t)$ is highly sensitive to time, variance, and SNR, making our
method crucial.

###  5.4 Analysis on Guidance Scale

We observed that for diffusion models in pixel space, it is acceptable to keep
the guidance scale constant, but for LDMs, the guidance scale needs to be
reduced near $t=0$ to eliminate artifacts as shown in Figure 4. To
quantitatively analyze the impact of the guidance scale, we measured changes
in LDMs using a time-independent $w_{t}$ and by parameterizing it as in
Equation 11.

####  5.4.1 Sampling on Pixel Space

Figure 9: Fidelity of generated images across different guidance scale $w_{t}$
(numbers in upper left corners) measured by the FID score (lower is better).
The label “LANCZOS” refers to the images originally generated at a size of
$256^{2}$ by the model and then upsampled to $512^{2}$ using Lanczos
resampling. The green dashed line represents the FID between images generated
by the model at $256^{2}$ trained resolution and the CelebA-HQ $256^{2}$
dataset.

We empirically found that for pixel space diffusion models, keeping the
guidance scale constant is effective. Thus, we recommend using a constant
$w_{t}$, with $\eta=1$ and varying $\theta$ only in Equation (11). After
generating $512^{2}$ resolution images from a model trained on the CelebA-HQ
$256^{2}$ using UG, we measured the fidelity via Fréchet inception distance
(FID) (Heusel et al., 2017) to CelebA-HQ $512^{2}$. Results showed that as the
guidance scale increased, the features and contrast became clearer.
Astonishingly, at the optimal point ($\theta\approx 1.3$), the model
outperformed not only resized images from the trained resolution but also
achieved better fidelity compared to the dataset of the originally trained
size of $256^{2}$, demonstrating that UG serves a role beyond simple
interpolation or sharpening.

####  5.4.2 LDMs and Text-to-Image

Figure 10: Impact of guidance scale with CLIP score and NIQE metric. Specific
image examples at three parameter points are selected as Maximal CLIP
(triangle), Minimal NIQE (diamond), and Balanced (star). More Images are
presented in Appendix C.

For LDMs, it’s crucial to reduce the guidance scale to zero during the mid-
stages of sampling to prevent artifacts as shown in Figure 4. However, if we
tolerate the artifacts, the coarse structure of images can be better aligned
with the text prompt. Therefore, choosing the guidance scale involves a trade-
off between prompt alignment and image quality. To evaluate alignment and
quality, we used CLIP score (Radford et al., 2021) and naturalness image
quality evaluator (NIQE) (Mittal et al., 2012) respectively.

As shown in Figure 10, the CLIP score tends to increase with stronger
guidance, indicating better alignment with the prompt. Conversely, NIQE scores
worsen with strong guidance. At the optimal point for CLIP, image’s coarse
features align well with the prompt but lose photorealism. At NIQE’s optimal
point, the image appears locally natural and realistic but deviate
significantly from the prompt. This trend was consistent across samples and
prompts. We heuristically recommend $(\theta,\eta)\approx(1,0.6)$ as a
balanced setting.

##  6 Conclusion

In conclusion, we introduced upsample guidance, a training-free technique
enabling the generation of high-fidelity images at high resolutions not
originally trained on, demonstrating its applicability across various models
and applications. Our method, derived from the diffusion process and not
dependent on architecture, holds synergistic potential with any other
techniques for high-resolution image generation. Moreover, UG uniquely enables
the creation of images for datasets like CIFAR-10 $64^{2}$, where high-
resolution data may not originally exist.

In our experiments, we used a simple design of guidance scale for clarity, but
there’s room for enhancement through replacing it with more elaborated
functions. While focusing on spatial upsampling, further exploration into the
best practice for temporal upsampling in video and audio models is needed.
Especially, for audio, careful implementation is necessary as temporal
downsampling may shift pitch.

The computational cost of UG is marginal, and ongoing research aimed at
reducing inference steps further minimize the portion of time consumption due
to UG in LDMs. We consider our method a universally beneficial add-on for
generating high-resolution samples due to its ease of implementation and cost-
effectiveness.

##  7 Impact Statements

This paper presents work whose goal is to advance the field of Machine
Learning. There are many potential societal consequences of our work, none
which we feel must be specifically highlighted here.

## References

  * Automatic1111 (2022) Automatic1111.  Stable diffusion web ui.  2022\.  URL https://github.com/AUTOMATIC1111/stable-diffusion-webui.
  * Balaji et al. (2022) Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Kreis, K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., et al.  ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.  _arXiv preprint arXiv:2211.01324_ , 2022.
  * Bar-Tal et al. (2023) Bar-Tal, O., Yariv, L., Lipman, Y., and Dekel, T.  Multidiffusion: Fusing diffusion paths for controlled image generation.  _arXiv preprint arXiv:2302.08113_ , 2023.
  * Blattmann et al. (2023a) Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al.  Stable video diffusion: Scaling latent video diffusion models to large datasets.  _arXiv preprint arXiv:2311.15127_ , 2023a.
  * Blattmann et al. (2023b) Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K.  Align your latents: High-resolution video synthesis with latent diffusion models.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023b.
  * Gu et al. (2023) Gu, J., Zhai, S., Zhang, Y., Susskind, J., and Jaitly, N.  Matryoshka diffusion models.  _arXiv preprint arXiv:2310.15111_ , 2023.
  * Guo et al. (2023) Guo, Y., Yang, C., Rao, A., Wang, Y., Qiao, Y., Lin, D., and Dai, B.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  _arXiv preprint arXiv:2307.04725_ , 2023.
  * Heusel et al. (2017) Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho & Salimans (2022) Ho, J. and Salimans, T.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. (2020) Ho, J., Jain, A., and Abbeel, P.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Ho et al. (2022a) Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ , 2022a.
  * Ho et al. (2022b) Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T.  Cascaded diffusion models for high fidelity image generation.  _The Journal of Machine Learning Research_ , 23(1):2249–2281, 2022b.
  * Ho et al. (2022c) Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J.  Video diffusion models.  _arXiv preprint arXiv:2204.03458_ , 2022c.
  * Hoogeboom et al. (2023) Hoogeboom, E., Heek, J., and Salimans, T.  simple diffusion: End-to-end diffusion for high resolution images.  _arXiv preprint arXiv:2301.11093_ , 2023.
  * Hu et al. (2021) Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Hwang et al. (2023) Hwang, J., Park, Y.-H., and Jo, J.  Resolution chromatography of diffusion models.  _arXiv preprint arXiv:2401.10247_ , 2023.
  * Karras et al. (2018) Karras, T., Aila, T., Laine, S., and Lehtinen, J.  Progressive growing of gans for improved quality, stability, and variation.  In _International Conference on Learning Representations_ , 2018.
  * Kingma & Welling (2013) Kingma, D. P. and Welling, M.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Krizhevsky et al. (2009) Krizhevsky, A., Hinton, G., et al.  Learning multiple layers of features from tiny images.  2009\.
  * Liu et al. (2022) Liu, L., Ren, Y., Lin, Z., and Zhao, Z.  Pseudo numerical methods for diffusion models on manifolds.  _arXiv preprint arXiv:2202.09778_ , 2022.
  * Luo et al. (2023) Luo, S., Tan, Y., Huang, L., Li, J., and Zhao, H.  Latent consistency models: Synthesizing high-resolution images with few-step inference.  _arXiv preprint arXiv:2310.04378_ , 2023.
  * Lykon (2023) Lykon.  Dreamshaper.  https://huggingface.co/Lykon/dreamshaper-8, 2023.  Accessed: 2024-02-01.
  * Meng et al. (2021) Meng, C., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S.  Sdedit: Image synthesis and editing with stochastic differential equations.  _arXiv preprint arXiv:2108.01073_ , 2021.
  * Mittal et al. (2012) Mittal, A., Soundararajan, R., and Bovik, A. C.  Making a “completely blind” image quality analyzer.  _IEEE Signal processing letters_ , 20(3):209–212, 2012.
  * Podell et al. (2023) Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_ , 2023.
  * Poole et al. (2022) Poole, B., Jain, A., Barron, J. T., and Mildenhall, B.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv preprint arXiv:2209.14988_ , 2022.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pp. 8748–8763. PMLR, 2021.
  * Ramesh et al. (2022) Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Rombach et al. (2022) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10684–10695, 2022.
  * Ronneberger et al. (2015) Ronneberger, O., Fischer, P., and Brox, T.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pp. 234–241. Springer, 2015.
  * Ryu (2022) Ryu, S.  Low-rank adaptation for fast text-to-image diffusion fine-tuning.  2022\.  URL https://github.com/cloneofsimo/lora.
  * Song et al. (2020a) Song, J., Meng, C., and Ermon, S.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song et al. (2020b) Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * StabilityAI (2023) StabilityAI.  If by deepfloyd lab at stabilityai.  2023\.  URL https://github.com/deep-floyd/IF.
  * Ye et al. (2023) Ye, H., Zhang, J., Liu, S., Han, X., and Yang, W.  Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.  _arXiv preprint arXiv:2308.06721_ , 2023.
  * Zhang et al. (2023) Zhang, L., Rao, A., and Agrawala, M.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 3836–3847, 2023.
  * Zheng et al. (2023) Zheng, Q., Guo, Y., Deng, J., Han, J., Li, Y., Xu, S., and Xu, H.  Any-size-diffusion: Toward efficient text-driven synthesis for any-size hd images.  _arXiv preprint arXiv:2308.16582_ , 2023.
  * Zhou et al. (2022) Zhou, S., Chan, K. C., Li, C., and Loy, C. C.  Towards robust blind face restoration with codebook lookup transformer.  In _NeurIPS_ , 2022.

##  Appendix A Calculation of Adjusted Time $\tau$

Time adjustment is for matching SNR, so analytically obtaining tau is possible
by finding the inverse function of SNR over time. However, as most
implementations encode time as an integer, it’s sufficient to numerically
approximate values rather than compute exact ones. Below is a Python code for
numerically calculating time adjustment for integer times, and Figure 11
visualizes results calculated from a real model’s $\alpha_{t}$.

⬇

# m : scale factor

# alphas : list of alpha_t

snr = alphas / (1 - alphas)

snr_low = alphas / (1 - alphas) * m**2

log_snr, log_snr_low = np.log(snr), np.log(snr_low)

def getSingleMatch(t):

differences = np.abs(log_snr_low[t] - log_snr)

tau = np.argmin(differences)

return tau

return [getSingleMatch(t) for t in range(len(alphas))]

Figure 11: Time adjstment with scale factor $m=2$ for the noise schedule of
Stable Diffusion v1.5.

##  Appendix B More Upsampling Examples

Figure 12: Images generated with and without UG, sampled on $1280^{2}$
resolutions with $m=2$.

Figure 13: Comparison of samples generated at a resolution of $512^{2}$ using
different upscaling techniques, based on a model trained on the CelebA dataset
at a resolution of $256^{2}$. ‘Lanczos’ and ‘CodeFormer (Zhou et al., 2022)’
refer to images upscaled from $256^{2}$ using their respective methods, while
‘UG’ denotes samples generated with $w_{t}=1.35$, optimally chosen based on
the experiments described in Section 5.4.1

Figure 14: Samples generated at a resolution of $1024^{2}$ using different
upscaling methods applied to the same text-to-image model. ‘HiRes.Fix’ refers
to the use of CodeFormer (Zhou et al., 2022) as a super-resolution model,
employing the method mentioned in Section 2.1. ‘MultiDiffusion (Bar-Tal et
al., 2023)’ was used to generate samples via a panoramic approach. The numbers
below each column title represent the average elapsed time to generate one
sample on an RTX 3090 GPU. UG demonstrates not only competitive quality and
fidelity but also relatively faster generation speeds.

##  Appendix C Grid Images for Varying Guidance Scale

Figure 15:  Full images obtained by varying the parameters involving the
guidance scale in the experiments from Section 5.4.2


# Bijections between sets of invariant ideals, via the ladder technique

Matthew Gillespie  School of Mathematical and Statistical Sciences
Arizona State University
Tempe, Arizona 85287<EMAIL_ADDRESS>,  S. Kaliszewski  School of
Mathematical and Statistical Sciences
Arizona State University
Tempe, Arizona 85287<EMAIL_ADDRESS>,  John Quigg  School of
Mathematical and Statistical Sciences
Arizona State University
Tempe, Arizona 85287<EMAIL_ADDRESS>and  Dana P. Williams  Department of
Mathematics
Dartmouth College
Hanover, NH 03755-3551 USA<EMAIL_ADDRESS>
(Date: May 21, 2024)

###### Abstract.

We present a new method of establishing a bijective correspondence — in fact,
a lattice isomorphism — between action- and coaction-invariant ideals of
$C^{*}$-algebras and their crossed products by a fixed locally compact group.
It is known that such a correspondence exists whenever the group is amenable;
our results hold for any locally compact group under a natural form of
coaction invariance.

###### Key words and phrases:

action, coaction, crossed product duality, ideal, Morita equivalence

###### 2000 Mathematics Subject Classification:

Primary 46L55

##  1\. Introduction

One of the most fundamental things to know about a $C^{*}$-algebra is its
ideal structure. If the $C^{*}$-algebra arises as the crossed product
$A\rtimes_{\alpha}G$ of an action $(A,G,\alpha)$, it is natural to compare the
ideal structures of $A$ and $A\rtimes_{\alpha}G$. This is most easily done
when we restrict attention to ideals that are related in some way to the
action of $G$. More precisely, we focus on ideals $I$ of $A$ that are
$\alpha$-invariant. Then the crossed product is (isomorphic to) an ideal
$I\rtimes_{\alpha}G$ of $A\rtimes_{\alpha}G$, and the obvious question is
which ideals of $A\rtimes_{\alpha}G$ arise in this way. It turns out that they
are precisely those ideals that are invariant under the dual coaction
$\widehat{\alpha}$, and there is a bijection between the two sets of invariant
ideals. When $G$ is amenable this is an old result of Gootman and Lazar
([gootman]), and in this paper we prove it in complete generality (Theorem 3.2
(a)). Dually, starting with a coaction $(A,\delta)$ of $G$, the
$\delta$-invariant ideals of $A$ correspond to $\widehat{\delta}$-invariant
ideals of the crossed product $A\rtimes_{\delta}G$. Again, Gootman–Lazar
proved this for amenable $G$, and in this paper we prove it in general
(Theorem 3.2 (b)). Nilsen ([nilsen]) has a related pair of results, but their
relationship to ours is complicated because she used a different notion of
coaction invariance than we do.

Perhaps more important, though, is our method of proof—we introduce what we
call the “ladder technique”, which leads to quick proofs of the aforementioned
bijections, using only crossed-product duality and the Rieffel correspondence
between ideals of Morita equivalent $C^{*}$-algebras. We expect the ladder
technique to be useful in other situations; for example three of us are
currently working on an application to ideals of Fell bundles over groupoids.

We begin in Section 2 with a detailed overview of the crossed-product duality
theorems we need, the essential definitions and facts regarding invariant
ideals, and an identification of the associated Rieffel correspondences. Then
in Section 3 we prove our main theorem, which actually comprises four
versions: starting with actions we can consider either full or reduced crossed
products, and dually we can start with either maximal or normal coactions.
Finally, in Section 4 we close with a brief discussion comparing our results
to those of Gootman–Lazar and Nilsen. And we point out that one of our
theorems was proved, using somewhat more technical methods (unrelated to the
ladder technique), in a recent paper two of us wrote with Tron Omland.

##  2\. Preliminaries

Below we will recall suitable versions of the Imai–Takai and Katayama duality
theorems for crossed products. But to prepare for this we start with some
background on actions and coactions. Further details about action crossed
products can be found in [danacrossed]. Our main references for coactions and
their crossed products are [enchilada, Appendix A], [maximal], [raeburn],
[clda], [klqfunctor], and [nilsen].

Throughout, $G$ is a locally compact group and $A$ is a $C^{*}$-algebra. We
write $L^{2}$ for $L^{2}(G)$, $\lambda$ and $\rho$ for the left and right
regular representations of $G$ on $L^{2}$, respectively, $\mathcal{K}$ for the
compact operators $\mathcal{K}(L^{2})$, and $M$ (sometimes) for the
representation of $C_{0}(G)$ on $L^{2}$ by multiplication operators. We write
$w_{G}$ for the unitary element of $M(C_{0}(G)\otimes
C^{*}(G))=C_{b}^{\beta}(G,M(C^{*}(G)))$ given by the norm-bounded strictly
continuous function $w_{G}(s)=s$ for $s\in G$, where the $\beta$ signifies
that we are using the strict topology on $M(C^{*}(G))$.

We use $(A,\alpha)$ to denote an action $\alpha$ of $G$ on $A$. A _coaction_
$(A,\delta)$ of $G$ on $A$ is an injective nondegenerate homomorphism
$\delta:A\to M(A\otimes C^{*}(G))$ satisfying:

  1. (a)

$\overline{\operatorname{span}}\,\delta(A)(1\otimes C^{*}(G))=A\otimes
C^{*}(G)$

  2. (b)

$(\delta\otimes\operatorname{id})\circ\delta=(\operatorname{id}\otimes\delta_{G})\circ\delta$,

where $\otimes$ always denotes the minimal $C^{*}$-tensor product, and where
$\delta_{G}:C^{*}(G)\to M(C^{*}(G)\otimes C^{*}(G))$ is the usual
comultiplication of $C^{*}(G)$, i.e., the integrated form of the strictly
continuous unitary homomorphism $s\mapsto s\otimes s$. Here, and similarly
throughout the paper (unlike the convention used in [enchilada]),
$\delta(A)(1\otimes C^{*}(G))$ represents the set of products
$\\{\delta(a)(1\otimes z)\mid a\in A,z\in C^{*}(G)\\}$. (Also note that by
definition, our coactions are _nondegenerate_ in the sense of [enchilada,
Definition 2.10].) The (full) crossed product of an action $(A,\alpha)$ is
$A\rtimes_{\alpha}G$, which comes with a universal covariant representation
$(i_{A},i_{G})$ and a dual coaction $\widehat{\alpha}$. The reduced crossed
product of $(A,\alpha)$ is $A\rtimes_{\alpha,r}G$. The crossed product of a
coaction $(A,\delta)$ is $A\rtimes_{\delta}G$, which comes with a universal
covariant representation $(j_{A},j_{G})$ and a dual action $\widehat{\delta}$.

For any action $(A,\alpha)$, the _canonical surjection_ $\Phi_{\alpha}\colon
A\rtimes_{\alpha}G\rtimes_{\widehat{\alpha}}G\to A\otimes\mathcal{K}$ is
determined by

  1. (a)

$\Phi_{\alpha}\circ j_{A\rtimes_{\alpha}G}\circ
i_{A}=(\operatorname{id}\otimes M)\circ\alpha^{-1}$

  2. (b)

$\Phi_{\alpha}\circ j_{A\rtimes_{\alpha}G}\circ i_{G}=1\otimes\lambda$

  3. (c)

$\Phi_{\alpha}\circ j_{G}=1\otimes M$.

In item (a), “$\alpha^{-1}$” refers to the map that sends $a\in A$ to the
element of $M(A\otimes C_{0}(G))$ determined by the function
$s\mapsto\alpha_{s^{-1}}(a)$ (see [raeburn]). For a coaction $(A,\delta)$, the
_canonical surjection_
$\Phi_{\delta}:A\rtimes_{\delta}G\rtimes_{\widehat{\delta}}G\to
A\otimes\mathcal{K}$ is determined by

  1. (a)

$\Phi_{\delta}\circ i_{A\rtimes_{\delta}G}\circ
j_{A}=(\operatorname{id}\otimes\lambda)\circ\delta$

  2. (b)

$\Phi_{\delta}\circ i_{A\rtimes_{\delta}G}\circ j_{G}=1\otimes M$

  3. (c)

$\Phi_{\delta}\circ i_{G}=1\otimes\rho$.

An _equivariant homomorphism_ $\phi:(A,\delta)\to(B,\epsilon)$ between
coactions is a possibly degenerate homomorphism $\phi$ mapping $A$ into $B$
(not $M(B)$) such that the diagram

${A}$${\widetilde{M}(A\otimes C^{*}(G))}$${B}$${\widetilde{M}(B\otimes
C^{*}(G))}$$\scriptstyle{\delta}$$\scriptstyle{\phi}$$\scriptstyle{\phi\otimes\operatorname{id}}$$\scriptstyle{\epsilon}$

commutes (see [klqfunctor, Definition 3.2]). Immediately following
[klqfunctor, Corollary 3.13] it is noted that “a routine adaptation of the
usual arguments” (i.e., carefully applying the definitions) shows that every
$\delta-\epsilon$ equivariant homomorphism $\phi:A\to B$ gives rise to a
$\widehat{\delta}-\widehat{\epsilon}$ equivariant homomorphism

$\phi\rtimes G=(j_{B}\circ\phi)\times j_{G}:A\rtimes_{\delta}G\to
B\rtimes_{\epsilon}G.$

A coaction $(A,\delta)$ is _maximal_ if $\Phi_{\delta}$ is an isomorphism, and
is _normal_ if $j_{A}:A\to M(A\rtimes_{\delta}G)$ is injective. Equivalently,
$\delta$ is normal exactly when $\Phi_{\delta}$ factors through an isomorphism
of $A\rtimes_{\delta}G\rtimes_{\widehat{\delta},r}G$ onto
$A\otimes\mathcal{K}$.

Every dual coaction $(A\rtimes_{\alpha}G,\widehat{\alpha})$ is maximal. Every
coaction $(A,\delta)$ has a _normalization_ $(A^{n},\delta^{n})$, which is
unique up to isomorphism, such that $A^{n}$ is a quotient of $A$, the coaction
$\delta^{n}$ is normal, the quotient map $\psi_{\delta}:A\to A^{n}$ is
$\delta-\delta^{n}$ equivariant, and the homomorphism $\psi_{\delta}\rtimes
G:A\rtimes_{\delta}G\to A^{n}\rtimes_{\delta^{n}}G$ is an isomorphism. For a
dual coaction $\widehat{\alpha}$ on an action crossed product
$A\rtimes_{\alpha}G$, the associated map $\psi_{\widehat{\alpha}}$ is the
regular representation $A\rtimes_{\alpha}G\to A\rtimes_{\alpha,r}G$. Thus, the
dual coaction on the reduced crossed product is the normalization
$\widehat{\alpha}^{n}$, and the double-dual action on
$A\rtimes_{\alpha,r}G\rtimes_{\widehat{\alpha}^{n}}G$ is
$\widehat{\widehat{\alpha}^{n}}$.

Theorem 2.1 below is _Imai–Takai duality_. The original version was stated in
[imaitakai] for reduced crossed products, and did not use the word ‘coaction’.
Version (a) below for full crossed products appears in [raeburn, Theorem 7].
We use [enchilada] as a convenient reference.

######  Theorem 2.1 ([enchilada, Theorem A.67]).

For any action $(A,\alpha)$:

  1. (a)

(full-crossed-product version) The canonical map $\Phi_{\alpha}$ is an
$\widehat{\widehat{\alpha}}-(\alpha\otimes\operatorname{Ad}\rho)$ equivariant
isomorphism of $A\rtimes_{\alpha}G\rtimes_{\widehat{\alpha}}G$ onto
$A\otimes\mathcal{K}$.

  2. (b)

(reduced-crossed-product version) There is an
$\widehat{\widehat{\alpha}^{n}}-(\alpha\otimes\operatorname{Ad}\rho)$
equivariant isomorphism $\Phi_{\alpha,r}$ of
$A\rtimes_{\alpha,r}G\rtimes_{\widehat{\alpha}^{n}}G$ onto
$A\otimes\mathcal{K}$ such that
$\Phi_{\alpha}=\Phi_{\alpha,r}\circ(\psi_{\widehat{\alpha}}\rtimes G)$.

Theorem 2.2 below is _Katayama duality_ , which is the dual version of Theorem
2.1. The original version was stated in [katayama, Theorem 8] for reduced
coactions, but we prefer to work with full coactions.

######  Theorem 2.2 ([enchilada, Theorem A.69]).

For any coaction $(A,\delta)$:

  1. (a)

(maximal coaction version) If $\delta$ is maximal, the canonical surjection
$\Phi_{\delta}$ is a $\widehat{\widehat{\delta}}-\epsilon$ equivariant
isomorphism of $A\rtimes_{\delta}G\rtimes_{\widehat{\delta}}G$ onto
$A\otimes\mathcal{K}$.

  2. (b)

(normal coaction version) If $\delta$ is normal, there is a
${\widehat{\widehat{\delta}}\,}^{n}-\epsilon$ equivariant isomorphism
$\Phi_{\delta,r}$ of $A\rtimes_{\delta}G\rtimes_{\widehat{\delta},r}G$ onto
$A\otimes\mathcal{K}$ such that
$\Phi_{\delta}=\Phi_{\delta,r}\circ\psi_{\widehat{\widehat{\delta}}}$.

The coaction $(A\otimes\mathcal{K},\epsilon)$ associated to $(A,\delta)$ in
Theorem 2.2 is defined by

$\epsilon=\operatorname{Ad}\bigl{(}1\otimes(M\otimes\operatorname{id})(w_{G}^{*})\bigr{)}\circ(\operatorname{id}\otimes\Sigma)\circ(\delta\otimes\operatorname{id}),$

where $\Sigma:C^{*}(G)\otimes\mathcal{K}\to\mathcal{K}\otimes C^{*}(G)$ is the
flip isomorphism determined on elementary tensors by $a\otimes b\mapsto
b\otimes a$. The statement about equivariance in part (a) is actually missing
from [enchilada]; it follows from the analogous result for normal coactions in
[nilsen, Remark 5] and the equivalence of maximal and normal coactions ([clda,
Theorem 3.3]).

Given an action $(A,\alpha)$, we say that a (closed, two-sided) ideal $I$ of
$A$ is _$\alpha$ -invariant_ if $\alpha$ restricts to an action $\alpha|$ on
$I$; that is, if $\alpha_{g}(I)\subseteq I$ for each $g\in G$. (See
[danacrossed, Section 3.4] for further discussion.) We write
$\mathscr{I}_{\alpha}(A)$ for the set of $\alpha$-invariant ideals of $A$.

For a coaction $(A,\delta)$, an ideal $I$ of $A$ is _$\delta$ -invariant_ if
$\delta$ restricts to a coaction $\delta|$ on $I$. By results in Section 2 of
[nilsenbundle], this is equivalent to the condition that

(2.1) $\overline{\operatorname{span}}\,\delta(I)(1\otimes C^{*}(G))=I\otimes
C^{*}(G).$

(See also the discussion preceding Definition 3.17 in [klqfunctor].) We write
$\mathscr{I}_{\delta}(A)$ for the set of $\delta$-invariant ideals of $A$.

Somewhat surprisingly (to us), the fact that crossed products of invariant
ideals are _invariant_ ideals has not been clearly stated or entirely
justified elsewhere in the literature.

######  Proposition 2.3.

  1. (a)

For any action $(A,\alpha)$, for each $I\in\mathscr{I}_{\alpha}(A)$ the
inclusion map $\phi:I\hookrightarrow A$ is $\alpha|-\alpha$ equivariant, and
$\phi\rtimes G$ is an isomorphism of $I\rtimes_{\alpha|}G$ onto an
$\widehat{\alpha}$-invariant ideal $I\rtimes_{\alpha}G$ of
$A\rtimes_{\alpha}G$. Moreover, the image $I\rtimes_{\alpha,r}G$ of
$I\rtimes_{\alpha}G$ under the regular representation of $A\rtimes_{\alpha}G$
is an $\widehat{\alpha}^{n}$-invariant ideal of the reduced crossed product
$A\rtimes_{\alpha,r}G$.

  2. (b)

For any coaction $(A,\delta)$, for each $I\in\mathscr{I}_{\delta}(A)$ the
inclusion map $\phi:I\hookrightarrow A$ is $\delta|-\delta$ equivariant, and
$\phi\rtimes G$ is an isomorphism of $I\rtimes_{\delta|}G$ onto a
$\widehat{\delta}$-invariant ideal $I\rtimes_{\delta}G$ of
$A\rtimes_{\delta}G$.

###### Proof.

(a) Except for invariance, this is a consequence of [danacrossed, Proposition
3.19]. For the $\widehat{\alpha}$-invariance of $I\rtimes_{\alpha}G$, we have

$I\rtimes_{\alpha}G=\phi\rtimes
G(I\rtimes_{\alpha|}G)=\overline{\operatorname{span}}\,i_{A}(\phi(I))i_{G}(C^{*}(G)),$

so that

$\displaystyle\widehat{\alpha}(I\rtimes_{\alpha}G)$
$\displaystyle=\overline{\operatorname{span}}\,\widehat{\alpha}\bigl{(}i_{A}(\phi(I))\bigr{)}\widehat{\alpha}\bigl{(}i_{G}(C^{*}(G))\bigr{)}$
$\displaystyle=\overline{\operatorname{span}}\,\bigl{(}i_{A}(\phi(I))\otimes
1\bigr{)}(i_{G}\otimes\operatorname{id})(\delta_{G}(C^{*}(G)))$

by definition of $\widehat{\alpha}$. Thus,

$\displaystyle\overline{\operatorname{span}}\,$
$\displaystyle\widehat{\alpha}(I\rtimes_{\alpha}G)(1\otimes C^{*}(G))$
$\displaystyle=\overline{\operatorname{span}}\,\bigl{(}i_{A}(\phi(I))\otimes
1\bigr{)}(i_{G}\otimes\operatorname{id})(\delta_{G}(C^{*}(G)))(1\otimes
C^{*}(G))$
$\displaystyle=\overline{\operatorname{span}}\,\bigl{(}i_{A}(\phi(I))\otimes
1\bigr{)}(i_{G}\otimes\operatorname{id})\bigl{(}\delta_{G}(C^{*}(G))(1\otimes
C^{*}(G))\bigr{)}$
$\displaystyle=\overline{\operatorname{span}}\,\bigl{(}i_{A}(\phi(I))\otimes
1\bigr{)}(i_{G}\otimes\operatorname{id})\bigl{(}C^{*}(G)\otimes
C^{*}(G)\bigr{)}$
$\displaystyle=\overline{\operatorname{span}}\,\bigl{(}i_{A}(\phi(I))\otimes
1\bigr{)}\bigl{(}i_{G}(C^{*}(G))\otimes C^{*}(G)\bigr{)}$
$\displaystyle=\overline{\operatorname{span}}\,\bigl{(}i_{A}(\phi(I))i_{G}(C^{*}(G))\bigr{)}\otimes
C^{*}(G)$ $\displaystyle=\bigl{(}\phi\rtimes
G(I\rtimes_{\alpha|}G)\bigr{)}\otimes C^{*}(G)$
$\displaystyle=(I\rtimes_{\alpha}G)\otimes C^{*}(G),$

which shows that $I\rtimes_{\alpha}G$ is $\widehat{\alpha}$-invariant.
Invariance of $I\rtimes_{\alpha,r}G$ now follows from
$\widehat{\alpha}-\widehat{\alpha}^{n}$ equivariance of the regular
representation.

(b) Some of this — apart from invariance of $I\rtimes_{\delta}G$ — is
addressed in [nilsenbundle, Proposition 2.1], but with a different convention
regarding equivariant maps.

For us, the restriction $\delta|$ is a coaction on $I$ by definition of
invariance, and the inclusion $\phi$ is trivially $\delta|-\delta$
equivariant. The verification that the induced
$\widehat{\delta|}-\widehat{\delta}$ equivariant homomorphism $\phi\rtimes
G:I\rtimes_{\delta|}G\to A\rtimes_{\delta}G$ is injective follows a standard
computation with covariant representations on Hilbert space (see
[nilsenbundle, proof of Proposition 2.1], for example).

To see that $I\rtimes_{\delta}G=\phi\rtimes G(I\rtimes_{\delta|}G)$ is an
ideal of $A\rtimes_{\delta}G$, first note that [klqfunctor, Lemma 3.8] implies

$A\rtimes_{\delta}G=\overline{\operatorname{span}}\,j_{A}(A)j_{G}(C_{0}(G))=\overline{\operatorname{span}}\,j_{G}(C_{0}(G))j_{A}(A),$

so that

$\displaystyle(I\rtimes_{\delta}G)(A\rtimes_{\delta}G)$
$\displaystyle=\overline{\operatorname{span}}\,j_{A}(I)j_{G}(C_{0}(G))j_{A}(A)$
$\displaystyle=\overline{\operatorname{span}}\,j_{A}(I)j_{A}(A)j_{G}(C_{0}(G))$
$\displaystyle=\overline{\operatorname{span}}\,j_{A}(I)j_{G}(C_{0}(G))$
$\displaystyle=I\rtimes_{\delta}G,$

and similarly for $(A\rtimes_{\delta}G)(I\rtimes_{\delta}G)$.

Finally, to see that $I\rtimes_{\delta}G$ is $\widehat{\delta}$-invariant, for
any $s\in G$ we compute:

$\displaystyle\widehat{\delta}_{s}\left(I\rtimes_{\delta}G\right)$
$\displaystyle=\widehat{\delta}_{s}\left(\phi\rtimes
G(I\rtimes_{\delta|}G)\right)$ $\displaystyle=\phi\rtimes
G\bigl{(}\widehat{\delta|}_{s}(I\rtimes_{\delta|}G)\bigr{)}$
$\displaystyle=\phi\rtimes G(I\rtimes_{\delta|}G)$
$\displaystyle=I\rtimes_{\delta}G.\qed$

For our main result, Theorem 3.2, we refer to the _Rieffel correspondence_
(see, for example, [tfb, Proposition 3.24]). This is the lattice isomorphism
between the ideals of $A$ and the ideals of $B$ that arises from a $B-A$
imprimitivity bimodule $X$ by associating each ideal $I$ of $A$ with the ideal
$J$ of $B$ given by

$J=\overline{\operatorname{span}}\,\mathopen{{}_{\hbox{\vrule
height=8.5pt,width=0.0pt}\scriptscriptstyle{B}}{\mskip-3.0mu}\langle}X\cdot
I\nobreak\mathrel{,}\nobreak X\rangle.$

If $A$ and $B$ are equipped with actions or coactions of $G$ and $X$ has a
suitably compatible action or coaction (see [enchilada, Definitions 2.5 and
2.10]), then the Rieffel correspondence preserves invariance of ideals, and
therefore restricts to a bijection between the appropriately-invariant ideals
of $A$ and those of $B$. Again, since we could not find a precise statement of
this fact in the literature, we provide one here.

######  Lemma 2.4.

Let $X$ be a $B-A$ imprimitivity bimodule, let $I$ be an ideal of $A$, and let
$J$ be the ideal of $B$ associated to $I$ under the Rieffel correspondence.

  1. (a)

If $(A,\alpha)$ and $(B,\beta)$ are actions of $G$ and there exists a
$\beta-\alpha$ compatible action $\gamma$ on $X$, then $I$ is
$\alpha$-invariant if and only if $J$ is $\beta$-invariant.

  2. (b)

If $(A,\delta)$ and $(B,\epsilon)$ are coactions of $G$ and there exists a
nondegenerate $\epsilon-\delta$ compatible coaction $\zeta$ on $X$, then $I$
is $\delta$-invariant if and only if $J$ is $\epsilon$-invariant.

###### Proof.

Part (a) is straightforward, so we only address part (b). Moreover, by
symmetry it suffices to prove only one implication; so suppose $I$ is
$\delta$-invariant. Then $\overline{\operatorname{span}}\,(1\otimes
C^{*}(G))\delta(I)=I\otimes C^{*}(G)$ (after taking adjoints in (2.1)), and
nondegeneracy of $\zeta$ means that $\overline{\operatorname{span}}\,(1\otimes
C^{*}(G))\zeta(X)=X\otimes C^{*}(G)$. So, writing $M$ for $M(B\otimes
C^{*}(G))$, we have:

$\displaystyle\overline{\operatorname{span}}\,(1\otimes C^{*}(G))\epsilon(J)$
$\displaystyle=\overline{\operatorname{span}}\,(1\otimes
C^{*}(G))\epsilon\bigl{(}{}_{B}\langle X\cdot I,X\rangle\bigr{)}$
$\displaystyle=\overline{\operatorname{span}}\,(1\otimes
C^{*}(G)){}_{B}\langle\zeta(X\cdot I),\zeta(X)\rangle$
$\displaystyle=\overline{\operatorname{span}}\,{}_{M}\langle(1\otimes
C^{*}(G))\zeta(X)\delta(I),(1\otimes C^{*}(G))\zeta(X)\rangle$
$\displaystyle=\overline{\operatorname{span}}\,{}_{M}\langle(X\otimes
C^{*}(G))\delta(I),X\otimes C^{*}(G),\rangle$
$\displaystyle=\overline{\operatorname{span}}\,{}_{M}\langle(X\otimes
C^{*}(G))(1\otimes C^{*}(G))\delta(I),X\otimes C^{*}(G)\rangle$
$\displaystyle=\overline{\operatorname{span}}\,{}_{M}\langle(X\otimes
C^{*}(G))(I\otimes C^{*}(G)),X\otimes C^{*}(G)\rangle$
$\displaystyle=\overline{\operatorname{span}}\,{}_{B}\langle X\cdot
I,X\rangle\otimes C^{*}(G)$ $\displaystyle=J\otimes C^{*}(G).$

Thus $J$ is $\epsilon$-invariant. ∎

The isomorphisms from the duality theorems 2.1 and 2.2 allow us to make the
$A\otimes\mathcal{K}-A$ imprimitivity bimodule $A\otimes L^{2}$ into an
$A\rtimes G\rtimes G-A$ imprimitivity bimodule (for each of the four types of
crossed products), and when we do this we can identify the associated ideals
explicitly:

######  Proposition 2.5.

  1. (a)

Let $(A,\alpha)$ be an action, and let $I$ be an $\alpha$-invariant ideal of
$A$. Then the ideal of $A\rtimes_{\alpha}G\rtimes_{\widehat{\alpha}}G$
associated to $I$ by the Rieffel correspondence is
$I\rtimes_{\alpha}G\rtimes_{\widehat{\alpha}}G$. The ideal of
$A\rtimes_{\alpha,r}G\rtimes_{\widehat{\alpha}^{n}}G$ associated to $I$ by the
Rieffel correspondence is
$I\rtimes_{\alpha,r}G\rtimes_{\widehat{\alpha}^{n}}G$.

  2. (b)

Let $(A,\delta)$ be a coaction, and let $I$ be a $\delta$-invariant ideal of
$A$. If $\delta$ is maximal, then the ideal of
$A\rtimes_{\delta}G\rtimes_{\widehat{\delta}}G$ associated to $I$ by the
Rieffel correspondence is $I\rtimes_{\delta}G\rtimes_{\widehat{\delta}}G$. If
$\delta$ is normal, the ideal of
$A\rtimes_{\delta}G\rtimes_{\widehat{\delta},r}G$ associated to $I$ by the
Rieffel correspondence is $I\rtimes_{\delta}G\rtimes_{\widehat{\delta},r}G$.

###### Proof.

(a) The key observation is that image of
$I\rtimes_{\alpha}G\rtimes_{\widehat{\alpha}}G$ in $A\otimes\mathcal{K}$ under
the canonical map $\Phi_{\alpha}$ coincides with the image of
$I\rtimes_{\alpha|}G\rtimes_{\widehat{\alpha|}}G$ under the canonical map
$\Phi_{\alpha|}$, and this latter image is precisely $I\otimes\mathcal{K}$ by
Theorem 2.1 (a). Since the ideal of $A\otimes\mathcal{K}$ associated to $I$ by
the Rieffel correspondence is also precisely $I\otimes\mathcal{K}$, it follows
that the ideal of $A\rtimes_{\alpha}G\rtimes_{\widehat{\alpha}}G$ associated
to $I$ is

$\Phi_{\alpha}^{-1}(I\otimes\mathcal{K})=I\rtimes_{\alpha}G\rtimes_{\widehat{\alpha}}G.$

The other part of (a), and both parts of (b), are quite similar. ∎

##  3\. The ladder technique

Our main result, Theorem 3.2 below, rests upon the following basic observation
concerning maps between sets. We record it only for convenient reference.

######  Lemma 3.1.

If

${W}$${Z}$${Y}$${X}$$\scriptstyle{h}$$\scriptstyle{g}$$\scriptstyle{v}$$\scriptstyle{f}$$\scriptstyle{u}$

is a commutative diagram of sets and maps such that $u$ and $v$ are
bijections, then $f$, $g$, and $h$ are also bijections.

Recall that for any $C^{*}$-algebra $A$, the set $\mathscr{I}(A)$ of all
(closed, two-sided) ideals of $A$ is a lattice when ordered by inclusion, with
$I\wedge J=I\cap J$ and $I\vee J=I+J$. For any fixed action $\alpha$ or
coaction $\delta$ of $G$ on $A$, the intersection and sum of two invariant
ideals are themselves invariant, so the sets $\mathscr{I}_{\alpha}(A)$ and
$\mathscr{I}_{\delta}(A)$ of appropriately invariant ideals of $A$ each form a
sublattice of $\mathscr{I}(A)$.

######  Theorem 3.2.

  1. (a)

For any action $(A,\alpha)$, the assignments $I\mapsto I\rtimes_{\alpha}G$ and
$I\mapsto I\rtimes_{\alpha,r}G$ define lattice isomorphisms of
$\mathscr{I}_{\alpha}(A)$ onto
$\mathscr{I}_{\widehat{\alpha}}(A\rtimes_{\alpha}G)$ and
$\mathscr{I}_{\widehat{\alpha}^{n}}(A\rtimes_{\alpha,r}G)$, respectively.

  2. (b)

For any maximal or normal coaction $(A,\delta)$, the assignment $I\mapsto
I\rtimes_{\delta}G$ defines a lattice isomorphism of $\mathscr{I}_{\delta}(A)$
onto $\mathscr{I}_{\widehat{\delta}}(A\rtimes_{\delta}G)$.

###### Proof.

For the first part of (a), consider the _ladder diagram_

(3.1)
${\mathscr{I}_{G}(A\rtimes_{\alpha}G\rtimes_{\widehat{\alpha}}G\rtimes_{\widehat{\widehat{\alpha}}}G)}$${\mathscr{I}_{G}(A\rtimes_{\alpha}G\rtimes_{\widehat{\alpha}}G)}$${\mathscr{I}_{G}(A\rtimes_{\alpha}G)}$${\mathscr{I}_{G}(A)}$$\scriptstyle{h}$$\scriptstyle{g}$$\scriptstyle{v}$$\scriptstyle{f}$$\scriptstyle{u}$

(For simplicity here we’re writing $\mathscr{I}_{G}$ for the lattice of
appropriately-invariant ideals in each $C^{*}$-algebra.) The diagonal maps
$f$, $g$, and $h$ (the “rungs” of the ladder) are defined by Proposition 2.3;
so for example $f(I)=I\rtimes_{\alpha}G$ for $I\in\mathscr{I}_{\alpha}(A)$,
and $g(J)=J\times_{\widehat{\alpha}}G$ for
$J\in\mathscr{I}_{\widehat{\alpha}}(A\rtimes_{\alpha}G)$. The vertical maps
$u$ and $v$ come from the Rieffel correspondence using the imprimitivity
bimodules implicit in the duality theorems 2.1 and 2.2; since those bimodules
have suitably compatible actions and coactions (respectively), $u$ and $v$ are
bijections by Lemma 2.4, and they make the diagram commute by Proposition 2.5.
It follows from Lemma 3.1 that all the maps in the diagram are bijections.

Now $u$ and $v$ are in fact lattice isomorphisms because they are bijective
restrictions of lattice isomorphisms. Moreover, it’s routine to check that any
order-preserving bijection of lattices _whose inverse is also order-
preserving_ is a lattice isomorphism. Since here it’s evident by construction
that $f$ and $g$ are order-preserving, we see that $f^{-1}=u^{-1}\circ g$ is
too, and it follows that $f$ is a lattice isomorphism.

The other part of (a), and both parts of (b), are quite similar. ∎

##  4\. Conclusion

We emphasize that our primary intent in this paper is to introduce the “ladder
technique”; the bijections in Theorem 3.2, while significant in their own
right, are to some extent intended to serve as illustrative examples of the
technique. Theorem 3.2 itself is not completely new: for example, Gootman and
Lazar proved a version for amenable $G$. Their results are clearly immediate
corollaries of Theorem 3.2:

######  Theorem 4.1 ([gootman, Theorems 3.4 & 3.7]).

Assume that $G$ is amenable.

  1. (a)

For any action $(A,\alpha)$ of $G$, an ideal $J$ of $A\rtimes_{\alpha}G$ is
$\widehat{\alpha}$-invariant if and only if it is of the form
$I\rtimes_{\alpha}G$ for an $\alpha$-invariant ideal $I$ of $A$. Moreover, $I$
is uniquely determined.

  2. (b)

For any coaction $(A,\delta)$ of $G$, an ideal $J$ of $A\rtimes_{\delta}G$ is
$\widehat{\delta}$-invariant if and only if it is of the form
$I\rtimes_{\delta}G$ for a $\delta$-invariant ideal $I$ of $A$. Moreover, $I$
is uniquely determined.

The full-crossed-product part of Theorem 3.2 (a) has also appeared in
[pedersen2, Theorem 8.2], where it is proved using different techniques
(involving Landstad duality).

Nilsen has proved a pair of related results (for arbitrary locally compact
$G$) which are _not_ corollaries of Theorem 3.2; nevertheless, it seems
relevant to include Nilsen’s results here for comparison. One difference is
that Nilsen used a different notion of coaction-invariance for ideals: if
$(A,\delta)$ is a coaction, we say an ideal $I$ of $A$ is _weakly invariant_
if $\delta$ passes to a coaction on the quotient $A/I$. This is properly
weaker than the notion of invariance used in the current paper (although the
two coincide for amenable $G$). More importantly, and Nilsen’s bijection was
different from ours: she maps an ideal $J$ in a coaction crossed-product
$A\rtimes_{\delta}G$ to its _restriction_ $j_{A}^{-1}(J)\subseteq A$, and
similarly for an action and $i_{A}^{-1}(J)$. Thus, for example, given a (non-
normal) coaction $\delta$ of $G$ on $A$, Nilsen makes the (nonzero) kernel of
the canonical homomorphism $j_{A}:A\to M(A\rtimes_{\delta}G)$ correspond to
the zero ideal of $A\rtimes_{\delta}G$, which is _not_ the way the bijection
in Theorem 3.2 (b) works.

######  Theorem 4.2 ([nilsen, Corollaries 3.2 and 3.4]).

Let $G$ be a locally compact group.

  1. (a)

For any coaction $(A,\delta)$ of $G$, restriction gives a bijection between
the $\widehat{\delta}$-invariant ideals of $A\rtimes_{\delta}G$ and the weakly
$\delta$-invariant ideals of $A$.

  2. (b)

For any action $(A,\alpha)$ of $G$, restriction gives a bijection between the
weakly $\widehat{\alpha}$-invariant ideals of $A\rtimes_{\alpha}G$ and the
$\alpha$-invariant ideals of $A$.

## References