#
Reproducibility

Before reading about reproducibility for Diffusers, it is strongly recommended to take a look at PyTorch’s statement about reproducibility.

PyTorch states that

> completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms. While one can never expect the same results across platforms, one can expect results to be reproducible across releases, platforms, etc… within a certain tolerance. However, this tolerance strongly varies depending on the diffusion pipeline and checkpoint.

In the following, we show how to best control sources of randomness for diffusion models.

##
Inference

During inference, diffusion pipelines heavily rely on random sampling operations, such as the creating the gaussian noise tensors to be denoised and adding noise to the scheduling step.

Let’s have a look at an example. We run the DDIM pipeline for just two inference steps and return a numpy tensor to look into the numerical values of the output.

    from diffusers import DDIMPipeline
    import numpy as np

    model_id = "google/ddpm-cifar10-32"

    # load model and scheduler
    ddim = DDIMPipeline.from_pretrained(model_id)

    # run pipeline for just two steps and return numpy tensor
    image = ddim(num_inference_steps=2, output_type="np").images
    print(np.abs(image).sum())

Running the above prints a value of 1464.2076, but running it again prints a different value of 1495.1768. What is going on here? Every time the pipeline is run, gaussian noise is created and step-wise denoised. To create the gaussian noise with `torch.randn`, a different random seed is taken every time, thus leading to a different result. This is a desired property of diffusion pipelines, as it means that the pipeline can create a different random image every time it is run. In many cases, one would like to generate the exact same image of a certain run, for which case an instance of a PyTorch generator has to be passed:

    import torch
    from diffusers import DDIMPipeline
    import numpy as np

    model_id = "google/ddpm-cifar10-32"

    # load model and scheduler
    ddim = DDIMPipeline.from_pretrained(model_id)

    # create a generator for reproducibility
    generator = torch.Generator(device="cpu").manual_seed(0)

    # run pipeline for just two steps and return numpy tensor
    image = ddim(num_inference_steps=2, output_type="np", generator=generator).images
    print(np.abs(image).sum())

Running the above always prints a value of 1491.1711 - also upon running it again because we define the generator object to be passed to all random functions of the pipeline.

If you run this code snippet on your specific hardware and version, you should get a similar, if not the same, result.

It might be a bit unintuitive at first to pass `generator` objects to the pipelines instead of just integer values representing the seed, but this is the recommended design when dealing with probabilistic models in PyTorch as generators are random states that are advanced and can thus be passed to multiple pipelines in a sequence.

Great! Now, we know how to write reproducible pipelines, but it gets a bit trickier since the above example only runs on the CPU. How do we also achieve reproducibility on GPU? In short, one should not expect full reproducibility across different hardware when running pipelines on GPU as matrix multiplications are less deterministic on GPU than on CPU and diffusion pipelines tend to require a lot of matrix multiplications. Let’s see what we can do to keep the randomness within limits across different GPU hardware.

To achieve maximum speed performance, it is recommended to create the generator directly on GPU when running the pipeline on GPU:

    import torch
    from diffusers import DDIMPipeline
    import numpy as np

    model_id = "google/ddpm-cifar10-32"

    # load model and scheduler
    ddim = DDIMPipeline.from_pretrained(model_id)
    ddim.to("cuda")

    # create a generator for reproducibility
    generator = torch.Generator(device="cuda").manual_seed(0)

    # run pipeline for just two steps and return numpy tensor
    image = ddim(num_inference_steps=2, output_type="np", generator=generator).images
    print(np.abs(image).sum())

Running the above now prints a value of 1389.8634 - even though we’re using the exact same seed! This is unfortunate as it means we cannot reproduce the results we achieved on GPU, also on CPU. Nevertheless, it should be expected since the GPU uses a different random number generator than the CPU.

To circumvent this problem, we created a `randn_tensor` function, which can create random noise on the CPU and then move the tensor to GPU if necessary. The function is used everywhere inside the pipelines allowing the user to always pass a CPU generator even if the pipeline is run on GPU:

    import torch
    from diffusers import DDIMPipeline
    import numpy as np

    model_id = "google/ddpm-cifar10-32"

    # load model and scheduler
    ddim = DDIMPipeline.from_pretrained(model_id)
    ddim.to("cuda")

    # create a generator for reproducibility
    generator = torch.manual_seed(0)

    # run pipeline for just two steps and return numpy tensor
    image = ddim(num_inference_steps=2, output_type="np", generator=generator).images
    print(np.abs(image).sum())

Running the above now prints a value of 1491.1713, much closer to the value of 1491.1711 when the pipeline is fully run on the CPU.

As a consequence, we recommend always passing a CPU generator if Reproducibility is important. The loss of performance is often neglectable, but one can be sure to generate much more similar values than if the pipeline would have been run on CPU.

Finally, we noticed that more complex pipelines, such as UnCLIPPipeline are often extremely susceptible to precision error propagation and thus one cannot expect even similar results across different GPU hardware or PyTorch versions. In such cases, one has to make sure to run exactly the same hardware and PyTorch version for full Reproducibility.

##
Randomness utilities

###
randn_tensor

####
diffusers.utils.randn_tensor

< source >

( shape: typing.Union[typing.Tuple, typing.List] generator: typing.Union[typing.List[ForwardRef('torch.Generator')], ForwardRef('torch.Generator'), NoneType] = None device: typing.Optional[ForwardRef('torch.device')] = None dtype: typing.Optional[ForwardRef('torch.dtype')] = None layout: typing.Optional[ForwardRef('torch.layout')] = None )

This is a helper function that allows to create random tensors on the desired `device` with the desired `dtype`. When passing a list of generators one can seed each batched size individually. If CPU generators are passed the tensor will always be created on CPU.