#
Load schedulers and models

Diffusion pipelines are a collection of interchangeable schedulers and models that can be mixed and matched to tailor a pipeline to a specific use case. The scheduler encapsulates the entire denoising process such as the number of denoising steps and the algorithm for finding the denoised sample. A scheduler is not parameterized or trained so they don’t take very much memory. The model is usually only concerned with the forward pass of going from a noisy input to a less noisy sample.

This guide will show you how to load schedulers and models to customize a pipeline. You’ll use the runwayml/stable-diffusion-v1-5 checkpoint throughout this guide, so let’s load it first.

    import torch
    from diffusers import DiffusionPipeline

    pipeline = DiffusionPipeline.from_pretrained(
        "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16, use_safetensors=True
    ).to("cuda")

You can see what scheduler this pipeline uses with the `pipeline.scheduler` attribute.

    pipeline.scheduler
    PNDMScheduler {
      "_class_name": "PNDMScheduler",
      "_diffusers_version": "0.21.4",
      "beta_end": 0.012,
      "beta_schedule": "scaled_linear",
      "beta_start": 0.00085,
      "clip_sample": false,
      "num_train_timesteps": 1000,
      "set_alpha_to_one": false,
      "skip_prk_steps": true,
      "steps_offset": 1,
      "timestep_spacing": "leading",
      "trained_betas": null
    }

##
Load a scheduler

Schedulers are defined by a configuration file that can be used by a variety of schedulers. Load a scheduler with the SchedulerMixin.from_pretrained() method, and specify the `subfolder` parameter to load the configuration file into the correct subfolder of the pipeline repository.

For example, to load the DDIMScheduler:

    from diffusers import DDIMScheduler, DiffusionPipeline

    ddim = DDIMScheduler.from_pretrained("runwayml/stable-diffusion-v1-5", subfolder="scheduler")

Then you can pass the newly loaded scheduler to the pipeline.

    pipeline = DiffusionPipeline.from_pretrained(
        "runwayml/stable-diffusion-v1-5", scheduler=ddim, torch_dtype=torch.float16, use_safetensors=True
    ).to("cuda")

##
Compare schedulers

Schedulers have their own unique strengths and weaknesses, making it difficult to quantitatively compare which scheduler works best for a pipeline. You typically have to make a trade-off between denoising speed and denoising quality. We recommend trying out different schedulers to find one that works best for your use case. Call the `pipeline.scheduler.compatibles` attribute to see what schedulers are compatible with a pipeline.

Let’s compare the LMSDiscreteScheduler, EulerDiscreteScheduler, EulerAncestralDiscreteScheduler, and the DPMSolverMultistepScheduler on the following prompt and seed.

    import torch
    from diffusers import DiffusionPipeline

    pipeline = DiffusionPipeline.from_pretrained(
        "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16, use_safetensors=True
    ).to("cuda")

    prompt = "A photograph of an astronaut riding a horse on Mars, high resolution, high definition."
    generator = torch.Generator(device="cuda").manual_seed(8)

To change the pipelines scheduler, use the from_config() method to load a different scheduler’s `pipeline.scheduler.config` into the pipeline.

LMSDiscreteScheduler
EulerDiscreteScheduler
EulerAncestralDiscreteScheduler
DPMSolverMultistepScheduler

LMSDiscreteScheduler typically generates higher quality images than the default scheduler.

    from diffusers import LMSDiscreteScheduler

    pipeline.scheduler = LMSDiscreteScheduler.from_config(pipeline.scheduler.config)
    image = pipeline(prompt, generator=generator).images[0]
    image

LMSDiscreteScheduler
EulerDiscreteScheduler
EulerAncestralDiscreteScheduler
DPMSolverMultistepScheduler

Most images look very similar and are comparable in quality. Again, it often comes down to your specific use case so a good approach is to run multiple different schedulers and compare the results.

###
Flax schedulers

To compare Flax schedulers, you need to additionally load the scheduler state into the model parameters. For example, let’s change the default scheduler in FlaxStableDiffusionPipeline to use the super fast `FlaxDPMSolverMultistepScheduler`.

The `FlaxLMSDiscreteScheduler` and `FlaxDDPMScheduler` are not compatible with the FlaxStableDiffusionPipeline yet.

    import jax
    import numpy as np
    from flax.jax_utils import replicate
    from flax.training.common_utils import shard
    from diffusers import FlaxStableDiffusionPipeline, FlaxDPMSolverMultistepScheduler

    scheduler, scheduler_state = FlaxDPMSolverMultistepScheduler.from_pretrained(
        "runwayml/stable-diffusion-v1-5",
        subfolder="scheduler"
    )
    pipeline, params = FlaxStableDiffusionPipeline.from_pretrained(
        "runwayml/stable-diffusion-v1-5",
        scheduler=scheduler,
        revision="bf16",
        dtype=jax.numpy.bfloat16,
    )
    params["scheduler"] = scheduler_state

Then you can take advantage of Flax’s compatibility with TPUs to generate a number of images in parallel. You’ll need to make a copy of the model parameters for each available device and then split the inputs across them to generate your desired number of images.

    # Generate 1 image per parallel device (8 on TPUv2-8 or TPUv3-8)
    prompt = "A photograph of an astronaut riding a horse on Mars, high resolution, high definition."
    num_samples = jax.device_count()
    prompt_ids = pipeline.prepare_inputs([prompt] * num_samples)

    prng_seed = jax.random.PRNGKey(0)
    num_inference_steps = 25

    # shard inputs and rng
    params = replicate(params)
    prng_seed = jax.random.split(prng_seed, jax.device_count())
    prompt_ids = shard(prompt_ids)

    images = pipeline(prompt_ids, params, prng_seed, num_inference_steps, jit=True).images
    images = pipeline.numpy_to_pil(np.asarray(images.reshape((num_samples,) + images.shape[-3:])))

##
Custom Timestep Schedules

With all our schedulers, you can choose one of the popular timestep schedules using configurations such as `timestep_spacing`, `interpolation_type`, and `use_karras_sigmas`. Some schedulers also provide the flexibility to use a custom timestep schedule. You can use any list of arbitrary timesteps, we will use the AYS timestep schedule here as example. It is a set of 10-step optimized timestep schedules released by researchers from Nvidia that can achieve significantly better quality compared to the preset timestep schedules. You can read more about their research here.

    from diffusers.schedulers import AysSchedules
    sampling_schedule = AysSchedules["StableDiffusionXLTimesteps"]
    print(sampling_schedule)

    [999, 845, 730, 587, 443, 310, 193, 116, 53, 13]

You can then create a pipeline and pass this custom timestep schedule to it as `timesteps`.

    pipe = StableDiffusionXLPipeline.from_pretrained(
        "SG161222/RealVisXL_V4.0",
        torch_dtype=torch.float16,
        variant="fp16",
    ).to("cuda")

    pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config, algorithm_type="sde-dpmsolver++")

    prompt = "A cinematic shot of a cute little rabbit wearing a jacket and doing a thumbs up"

    generator = torch.Generator(device="cpu").manual_seed(2487854446)

    image = pipe(
        prompt=prompt,
        negative_prompt="",
        generator=generator,
        timesteps=sampling_schedule,
    ).images[0]

The generated image has better quality than the default linear timestep schedule for the same number of steps, and it is similar to the default timestep scheduler when running for 25 steps.

AYS timestep schedule 10 steps
Linearly-spaced timestep schedule 10 steps
Linearly-spaced timestep schedule 25 steps

🤗 Diffusers currently only supports `timesteps` and `sigmas` for a selected list of schedulers and pipelines, but feel free to open a feature request if you want to extend feature to a scheduler and pipeline that does not currently support it!

##
Models

Models are loaded from the ModelMixin.from_pretrained() method, which downloads and caches the latest version of the model weights and configurations. If the latest files are available in the local cache, from_pretrained() reuses files in the cache instead of re-downloading them.

Models can be loaded from a subfolder with the `subfolder` argument. For example, the model weights for runwayml/stable-diffusion-v1-5 are stored in the unet subfolder.

    from diffusers import UNet2DConditionModel

    unet = UNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5", subfolder="unet", use_safetensors=True)

They can also be directly loaded from a repository.

    from diffusers import UNet2DModel

    unet = UNet2DModel.from_pretrained("google/ddpm-cifar10-32", use_safetensors=True)

To load and save model variants, specify the `variant` argument in ModelMixin.from_pretrained() and ModelMixin.save_pretrained().

    from diffusers import UNet2DConditionModel

    unet = UNet2DConditionModel.from_pretrained(
        "runwayml/stable-diffusion-v1-5", subfolder="unet", variant="non_ema", use_safetensors=True
    )
    unet.save_pretrained("./local-unet", variant="non_ema")

< > Update on GitHub