# DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models

Chong Mou1 Xintao Wang2 Jiechong Song1 Ying Shan2 Jian Zhang†1
1School of Electronic and Computer Engineering, Shenzhen Graduate School,
Peking University 2ARC Lab, Tencent PCG

###### Abstract

Despite the ability of existing large-scale text-to-image (T2I) models to
generate high-quality images from detailed textual descriptions, they often
lack the ability to precisely edit the generated or real images. In this
paper, we propose a novel image editing method, DragonDiffusion, enabling
Drag-style manipulation on Diffusion Models. Specifically, we construct
classifier guidance based on the strong correspondence of intermediate
features in the diffusion model. It can transform the editing signals into
gradients via feature correspondence loss to modify the intermediate
representation of the diffusion model. Based on this guidance strategy, we
also build a multi-scale guidance to consider both semantic and geometric
alignment. Moreover, a cross-branch self-attention is added to maintain the
consistency between the original image and the editing result. Our method,
through an efficient design, achieves various editing modes for the generated
or real images, such as object moving, object resizing, object appearance
replacement, and content dragging. It is worth noting that all editing and
content preservation signals come from the image itself, and the model does
not require fine-tuning or additional modules. Our source code will be
available at https://github.com/MC-E/DragonDiffusion.

††footnotetext:  † Corresponding author.

##  1 Introduction

Thanks to the large-scale training data and huge computing power, generative
models have developed rapidly, especially large-scale text-to-image (T2I)
diffusion models [29, 27, 23, 26, 10, 43, 22, 42], which aims to generate
images conditioned on a given text/prompt. However, this generative capability
is often diverse, and it is challenging to design suitable prompts to generate
images consistent with what the user has in mind, let alone further editing
based on existing images.

Compared to image generation, image editing has broader application demands.
Methods based on GANs [1, 2, 3] are widely used in the image editing domain
due to the compact and editable latent space (e.g., StyleGAN [17]). Recently,
DragGAN [24] proposes a point-to-point dragging scheme, which can achieve
refined content dragging. However, it is constrained by the capacity and
generalization of GAN models. Compared to GAN models, Diffusion [14] has
higher stability and superior generation quality. In this paper, we aim to
investigate whether the diffusion model can achieve a similar drag-style
ability. This ability should be a more generalized editing capability, not
limited to point dragging, such as object moving, object resizing, and cross-
image content dragging.

In implementation, the primary challenge lies in the lack of a concise and
modifiable latent space amenable to editing. Numerous diffusion-based image
editing methods (e.g., Prompt2Prompt [13], [12], [5]) are built based on the
correspondence between intermediate text and image features. They find that
the cross-attention map between the feature of words and object has a notable
local similarity, which can be used as an editing medium. Recently, self-
guidance [11] proposes a differentiable approach that employs cross-attention
maps to locate and calculate the size of objects within images. Then, gradient
backpropagation is utilized to edit these properties. However, the
correspondence between text and image features is weak, heavily relying on the
design of prompts. Moreover, in complex or multi-object scenarios, text
struggles to build accurate local similarity with a specific object. In this
paper, we aim to explore a more fine-grained editable space than text-image
correspondence for generalized image editing tasks.

In the large-scale T2I diffusion generation process, besides the strong
correspondence between text features and intermediate image features, there is
also a strong correspondence between intermediate image features. This
characteristic has been explored in DIFT [37], which demonstrates that this
feature correspondence is high-level, facilitating point-to-point
correspondence of relevant content in different images. Therefore, we are
intrigued by the possibility of utilizing this strong correspondence between
image features to achieve image editing. In this paper, we present our
solution. Specifically, our method involves two sets of features (i.e.,
guidance features and generation features) during the diffusion process. We
use the guidance features as the target, employing strong image feature
correspondence to constrain and edit the generation features. Additionally,
the content consistency between the edited result and the original image is
also maintained through the strong image feature correspondence. Here, we also
notice that there is a concurrent work, Drag-Diffusion [30], studying this
issue. It utilizes LORA [28] to maintain consistency with the original image
and optimizes one intermediate step in the diffusion process to perform
editing. Unlike Drag-Diffusion, our method is based on classifier-guidance
[9], and all editing and content consistency signals come from the image
itself, without the need for fine-tuning or training the model. In addition,
we use the intermediate feature correspondence to explore generalized image
editing capabilities, such as object moving, object resizing, object
appearance replacement, and content dragging. In summary, the contributions of
this paper are as follows:

  * •

We propose a classifier-guidance image editing strategy based on the strong
correspondence of intermediate features in diffusion models. In this design,
we also study the roles of the feature in different layers and develop a
multi-scale feature matching scheme that considers both semantic and geometric
correspondence.

  * •

All content editing and preservation signals in our proposed method come from
the image itself. It allows for a direct translation of T2I generation ability
in diffusion models to image editing tasks without the need for any model
fine-tuning or training.

  * •

Extensive experiments demonstrate that our DragonDiffusion can perform various
fine-grained image editing tasks, including object moving, object resizing,
object appearance replacement, and content dragging.

##  2 Related Work

###  2.1 Diffusion Models

In recent years, the diffusion model [14] has achieved great success in the
community of image synthesis. It is designed based on thermodynamics [32, 34],
including a diffusion process and a reverse process. In the diffusion process,
a natural image $\mathbf{X}_{0}$ is converted to a Gaussian distribution
$\mathbf{X}_{T}$ by adding random Gaussian noise with $T$ iterations. Each
step of adding noise is defined as:

$\mathbf{x}_{t}=\sqrt{1-\beta_{t}}\mathbf{x}_{t-1}+\sqrt{\beta_{t}}\bm{\epsilon}_{t-1},\
t\in[1,T],$ (1)

where $\beta_{t}\in[0,1]$ is a gradually increasing hyperparameter.
$\bm{\epsilon}_{t-1}\sim\mathcal{N}(0,\mathbf{I})$ is the random Gaussian
noise. The reverse process is to recover $\mathbf{x}_{0}$ from
$\mathbf{x}_{T}$ by several denoising steps. Therefore, the diffusion model is
training a denoiser, conditioned on the current noisy image and time step:

$L(\theta)=\mathbb{E}_{\mathbf{x}_{0},t,\epsilon\sim\mathcal{N}(0,1)}\left[||\epsilon_{t}-\epsilon_{\theta}(\mathbf{x}_{t},t)||_{2}^{2}\right],$
(2)

where $\theta$ is the model parameters of the denoiser.

Recently, some text-conditioned diffusion models (e.g., GLID [23] and SD [27])
have been proposed, which mostly inject text condition into the denoiser
through a cross-attention strategy.

Figure 1: Illustration of our model design. Our proposed method consists of
two branches, i.e., the guidance branch and the generation branch. The
guidance branch provides editing and consistency guidance to the generation
branch through the correspondence of intermediate features. Our
DragonDiffusion is built based on Stable Diffusion [27], without model fine-
tuning or training.

###  2.2 Classifier guidance in Diffusion Model

From a continuous perspective [36], diffusion models can be viewed as a score
function, i.e., $\nabla\mathbf{x}_{t}\rm{log}q(\mathbf{x}_{t})$, that samples
from the corresponding distribution [35] according to Langevin dynamics [32,
34]. The conditional diffusion process, on the other hand, can be seen as
using a joint score function, i.e.,
$\nabla\mathbf{x}_{t}\rm{log}q(\mathbf{x}_{t},y)$, to sample from a more
enriched distribution, where $y$ is the external condition. The joint score
function can be further decomposed into:

$\nabla\mathbf{x}_{t}\rm{log}q(\mathbf{x}_{t},y)=\nabla\mathbf{x}_{t}\rm{log}q(\mathbf{x}_{t})+\nabla\mathbf{x}_{t}\rm{log}q(y|\mathbf{x}_{t}),$
(3)

where the first term is the original unconditional diffusion denoiser, and the
second term corresponds to the classifier guidance to be added to the
diffusion process, also known as the energy function. The energy function can
be selected based on the generation target, such as a classifier [9] to
specify the category of generation results.

Classifier guidance has been applied to numerous controllable image generation
tasks, such as sketch-guided generation [38], mask-guided generation [31],
universal guided generation [41, 6], and image editing [11]. These methods,
based on classifier guidance, inspire us to transform editing signals into
gradients through score functions, achieving fine-grained image editing.

###  2.3 Image Editing

Image editing methods traditionally targeted a translation between image
domains [15, 16, 19]. Numerous editing approaches [1, 2, 3] invert images into
a latent space of StyleGAN [17] and then edit specific content (e.g., hair and
age) by manipulating latent vectors. Recently, DragGAN [24] proposes a point-
to-point dragging scheme, which can achieve more refined content dragging.
Diffusion [14], as a more stable generative model compared to GANs, has led to
several diffusion-based image editing methods [4, 13, 18, 20, 7]. Most of them
use text as the edit signal. For example, Prompt2Prompt [13] achieves specific
object editing by replacing the correspondence between text features and
intermediate features. SDEdit [20] performs image editing by adding noise to
the original image and then denoising under new text conditions.
InstructionP2P [7] achieves image editing by fine-tuning the model and using
text as an editing instruction. Recently, Self-guidance [11] transforms
editing signals into gradients through the correspondence between text and
intermediate features to achieve image editing. However, the correspondence
between image and text is coarse-grained. How to perform fine-grained and
generalized image editing with diffusion models is still an open challenge.

##  3 Method

###  3.1 Preliminary: Stable Diffusion

In this paper, we implement our method based on the recent state-of-the-art
T2I diffusion model (i.e., Stable Diffusion (SD) [27]). SD is a latent
diffusion model (LDM), which contains an autoencoder and an UNet denoiser. The
autoencoder can convert natural images $\mathbf{x}_{0}$ into latent space
$\mathbf{z}_{0}$ and then reconstruct them. The diffusion process of SD is
conducted in the latent space. The training objective of SD is the same as
that of common diffusion models (i.e., Eq. 3), except that the denoiser
operates on the latent $\mathbf{z}_{t}$ instead of the image $\mathbf{x}_{t}$.
During inference, $\mathbf{Z}_{T}$ is generated from random Gaussian
distribution. The final result $\mathbf{z}_{0}$, as the clean latent, is fed
into the decoder of the autoencoder to generate the natural image
$\mathbf{x}_{0}$. In the conditional part, SD utilizes the pre-trained CLIP
[25] text encoder to embed text inputs as embedding sequences $\mathbf{y}$.

###  3.2 Overview

The objective of our DragonDiffusion is to achieve fine-grained image editing
of real images by SD, which involves two issues: changing the content to be
edited and preserving other content. For example, if a user wants to move the
bread in an image, the generated result only needs to change the position of
the bread, while the appearance of the bread and other image content should
not change. In this paper, inspired by DIFT [14], we utilize the strong
correspondence of intermediate features in diffusion models to address both
issues simultaneously. An overview of our design is presented in Fig. 1.
First, we invert the original image $\mathbf{x}_{0}$ to the latent
representation $\mathbf{z}_{T}$ through the reverse diffusion process [33,
21]. Then, we input $\mathbf{z}_{T}$ into two parallel branches, i.e., the
guidance branch and the generation branch. The guidance branch is the standard
diffusion generation process, which can reconstruct $\mathbf{x}_{0}$. The
generation branch needs to generate the corresponding editing result according
to the demand. To preserve the content of the original image, we utilize the
correspondence between the intermediate features of the two branches,
transferring the content information from the guidance branch to the
generation branch through a cross-branch self-attention design. Similarly,
using the strong features correspondence, we design a score function [36, 35]
that transforms the editing signal into gradients through classifier guidance
[9], modifying the intermediate representation $\mathbf{z}_{t}$ of the
generation branch. Our entire editing process only applies the correspondence
of intermediate features in diffusion models, without the need for model fine-
tuning or training.

###  3.3 Classifier-guidance-based Editing Design

In this article, inspired by classifier guidance [9], we aim to update the
intermediate representation (i.e., $\mathbf{z}_{t}$) of the diffusion process
by transforming editing signals into gradients through score functions,
thereby achieving image editing.

Figure 2: Illustration of using features from different layers as guidance to
reconstruct the original image. In this experiment, we set $\mathbf{z}_{T}$ as
random Gaussian noise, and we set $\mathbf{m}^{gen}$, $\mathbf{m}^{gud}$ as
zeros matrix and $\mathbf{m}^{share}$ as a ones matrix.

####  3.3.1 Score Function

As illustrated in Eq. 3, to utilize classifier guidance, we first need to
construct a score function that matches the target. The recent work, DIFT
[37], discovers that the intermediate features of diffusion models have a
strong correspondence, which can be used for point-to-point matching between
different images. Inspired by this work, in each iteration, we use the same
denoiser to map the intermediate representations (i.e.,
$\mathbf{z}_{t}^{gen}$, $\mathbf{z}_{t}^{gud}$) of the two branches to the
feature space (i.e., $\mathbf{F}_{t}^{gen}$, $\mathbf{F}_{t}^{gud}$). The
subscripts “gen” and “gud” represent the generation branch and the guidance
branch, respectively. Note that the features here come from the decoder in the
denoiser. $\mathbf{F}_{t}^{gud}$ contains the features of the original image,
and $\mathbf{F}_{t}^{gen}$ contains the features of the edited image. Here, we
use two masks (i.e., $\mathbf{m}^{gud}$ and $\mathbf{m}^{gen}$) to represent
the positions of certain content in the original and edited images,
respectively. Based on the strong correspondence between the features, the two
regions in $\mathbf{F}_{t}^{gen}$ and $\mathbf{F}_{t}^{gud}$ need to have high
similarity. Here, we utilize the cosine distance ($cos(\cdot)\in[-1,1]$) to
measure the similarity and normalize it to $[0,1]$:

$\small\mathcal{S}(\mathbf{m}^{gen},\mathbf{m}^{gud})=\frac{cos(\mathbf{F}_{t}^{gen}[\mathbf{m}^{gen}],\
Sg(\mathbf{F}_{t}^{gud}[\mathbf{m}^{gud}]))+1}{2},$ (4)

where $Sg$ is the gradient clipping operation. The larger the value, the
higher the similarity. $[\cdot]$ represents retrieving values in non-zero
regions. When we want to constrain the content appearing in the position of
$\mathbf{m}^{gud}$ to appear in the target position $\mathbf{m}^{gen}$, our
optimization goal is to make the similarity in Eq. 4 as large as possible.

In addition to editing, we hope that other areas of the editing result remain
consistent with the original image. Given a mask $\mathbf{m}^{share}$, marking
areas with no editing, the similarity between the editing result and the
original image in these areas can also be defined using the cosine similarity
as $\mathcal{S}(\mathbf{m}^{share},\mathbf{m}^{share})$. Finally, the loss
function, combining editing and content preserving, is defined as:

$\small\mathcal{L}=\frac{w_{e}}{\alpha+\beta\cdot\mathcal{S}(\mathbf{m}^{gen},\mathbf{m}^{gud})}+\frac{w_{p}}{\alpha+\beta\cdot\mathcal{S}(\mathbf{m}^{share},\mathbf{m}^{share})},$
(5)

where $\alpha$ and $\beta$ are two hyper-parameters. $w_{e}$ and $w_{p}$ are
two weights to balance the editing and consistency parts. Finally, the joint
score function in Eq. 6 can be written as:

$\displaystyle\begin{split}&\nabla\mathbf{z}_{t}^{gen}\rm{log}q(\mathbf{z}_{t}^{gen},\mathbf{m}^{gen},\mathbf{m}^{share})=\\\
&\nabla\mathbf{z}_{t}^{gen}\rm{log}q(\mathbf{z}_{t}^{gen})+\nabla\mathbf{z}_{t}^{gen}\rm{log}q(\mathbf{m}^{gen},\mathbf{m}^{share}|\mathbf{z}_{t}^{gen}).\end{split}$
(6)

The classifier guidance
$\nabla\mathbf{z}_{t}^{gen}\rm{log}q(\mathbf{m}^{gen},\mathbf{m}^{share}|\mathbf{z}_{t}^{gen})$
can be computed as $\eta\frac{d\mathcal{L}}{d\mathbf{z}_{t}^{gen}}$, where
$\eta$ is the learning rate.

Figure 3: Visualization of the roles that contrastive loss and inpainting loss
play in the object movement task. The contrastive loss we designed can
eliminate the multi-object phenomenon, while the inpainting loss can generate
more natural content in the missing areas.

####  3.3.2 Multi-scale Guidance

The decoder of the Unet denoiser contains four blocks of different scales.
DIFT [37] finds that the second layer contains more semantic information,
while the third layer contains more geometric information. We also studied the
role of features from different layers in editing tasks, as shown in Fig. 2.
In the experiment, we set $\mathbf{z}_{T}$ as random Gaussian noise, and we
set $\mathbf{m}^{gen}$, $\mathbf{m}^{gud}$ as zeros matrixes and
$\mathbf{m}^{share}$ as a ones matrix. In this way, the generation branch is
guided to reconstruct the original image from the random Gaussian
distribution. We can find that the feature in the first layer is too high-
level to reconstruct the original image accurately. The features in the fourth
layer have weak feature correspondence, resulting in significant differences
between the reconstructed image and the original. The features in the second
and third layers are more suitable for reconstructing the original image, and
each has its own specialty. The second layer of features contains more
semantic information and can reconstruct images that are semantically similar
to the original but with some differences in content details. The features in
the third layer tend to express low-level visual features. The reconstructed
images are closer to the original, but they cannot provide effective
supervision for high-level texture features, resulting in blurry reconstructed
images. In our design, we aim to combine these two levels (i.e., high and low)
of guidance and propose a multi-scale supervision approach based on the second
and third layers of features. The reconstructed results in Fig. 2 also
demonstrate that this combination can balance the generation of low-level and
high-level visual features. Therefore, $\mathbf{F}_{t}^{gen}$ and
$\mathbf{F}_{t}^{gud}$ contain two sets of features from layer 2 and layer 3.

####  3.3.3 Implementation Details for Each Application

Object moving. In the task of object moving, $\mathbf{m}^{gen}$ and
$\mathbf{m}^{gud}$ locate the same object in different spatial positions.
$\mathbf{m}^{share}$ is the complement of the union of $\mathbf{m}^{gen}$ and
$\mathbf{m}^{gud}$, i.e.,
$\mathbf{m}^{share}=Cu(\mathbf{m}^{gen}\cup\mathbf{m}^{gud})$. We define the
points with a value of 1 in the binary mask as belonging to this mask. Using
only the editing and preserving losses in Eq. 5 can lead to some issues,
especially in the multiple objects phenomenon. As shown in the second image of
Fig. 3, although the bread has been moved according to the editing signal,
some of the bread content is still preserved in its original position in the
generated result. Therefore, in the object moving task, we need to constrain
the generated results to avoid previous image content in the original
position. To address this, we added a contrastive loss to Eq. 5 to provide an
additional constraint:

$\mathcal{L}_{c}=w_{c}\cdot\mathcal{S}(\mathbf{m}^{inpaint},\mathbf{m}^{inpaint}),$
(7)

where $\mathbf{m}^{inpaint}=\mathbf{m}^{gud}-\mathbf{m}^{gen}$, i.e.,
$\mathbf{m}^{inpaint}=\\{p|p\in\mathbf{m}^{gud}\ and\
p\notin\mathbf{m}^{gen}\\}$. $w_{c}$ is a hyper-parameter of the loss weight.

As illustrated in the third image of Fig. 3, although the contrastive loss
function can address the multi-object phenomenon, it lacks guidance during the
inpainting process, resulting in somewhat disordered inpainting. Here, we
design an inpainting loss, using content outside of the object as guidance to
constrain the features of the inpainting region. Mathematically, the loss
function is defined as:

$\displaystyle\begin{split}\left\\{\begin{array}[]{ll}\mathcal{L}_{i}=\frac{w_{i}}{\alpha+\beta\cdot\mathcal{S}_{glob}}\\\
\mathcal{S}_{glob}=\frac{cos(\frac{\sum\mathbf{F}_{t}^{gen}[\mathbf{m}^{inpaint}]}{\sum\mathbf{m}^{inpaint}},\
Sg(\frac{\sum\mathbf{F}_{t}^{gud}[\mathbb{I}-\mathbf{m}^{gud}]}{\sum\mathbf{m}^{gud}}))+1}{2},\end{array}\right.\end{split}$
(8)

where $w_{i}$ is a hyper-parameter of the loss weight. After equipping
$\mathcal{L}_{c}$ and $\mathcal{L}_{i}$, our method can effectively inpaint
the gaps of the object in the original image, as shown in the fourth image of
Fig. 3.

Object resizing. In this task, we use interpolation to transform
$\mathbf{m}^{gud}$ and $\mathbf{F}^{gud}_{t}$ to the target size, and then
extract the intermediate feature $\mathbf{F}^{gud}_{t}[\mathbf{m}^{gud}]$ as
the feature of the object after resizing. To guide the generation branch to
produce a target object with the same size, we perform local resizing on
$\mathbf{m}^{gen}$. Then, we use $\mathbf{F}^{gud}_{t}[\mathbf{m}^{gud}]$ to
supervise and guide the features within this region. Local resizing refers to
interpolating the input and then restoring it to its original size with center
cropping/expansion. Finally, in this task, Eq. 4 is reformulated as:

$\displaystyle\begin{split}\small&\mathcal{S}(\mathbf{m}^{gen},\mathbf{m}^{gud})=\\\
&\frac{cos(\mathbf{F}_{t}^{gen}[\mathcal{C}(\mathcal{R}(\mathbf{m}^{gen}))],\
Sg(\mathcal{R}(\mathbf{F}_{t}^{gud})[\mathcal{R}(\mathbf{m}^{gud})]))+1}{2},\end{split}$
(9)

where $\mathcal{R}$ and $\mathcal{C}$ represent the interpolation and center
cropping/expansion operation, respectively. The other constraints remain
consistent with default.

Figure 4: Visualization of the object moving with and without cross-branch
self-attention.

Appearance replacement. This task aims to replace the appearance between
objects of the same category. Similar to the inpainting loss (i.e., Eq. 8) in
object moving, we use the features mean of the corresponding region to
represent the object appearance. Therefore, the guidance branch will involve
the diffusion of two guidance images, the original image and the appearance
reference image. The appearance reference image only edits the generation
through gradients, generated from appearance similarity. We use
$\mathbf{F}_{t}^{app}$ and $\mathbf{m}^{app}$ to represent the intermediate
features of the appearance reference image and the mask corresponding to the
reference object, respectively. Therefore, the appearance similarity is
defined as:

$\displaystyle\begin{split}\small&\mathcal{S}_{app}(\mathbf{m}^{gud},\mathbf{m}^{gen})=\\\
&\frac{cos(\frac{\sum\mathbf{F}_{t}^{gen}[\mathbf{m}^{gen}]}{\sum\mathbf{m}^{gen}},\
Sg(\frac{\sum\mathbf{F}_{t}^{app}[\mathbf{m}^{app}]}{\sum\mathbf{m}^{app}}))+1}{2}.\end{split}$
(10)

The other constraints remain consistent with default.

Point dragging. In this task, we want to drag the image content via a specific
point in the image. In this case, $\mathbf{m}^{gen}$ and $\mathbf{m}^{gud}$
denote the destination and starting points, as well as their adjacent points
within a small range surrounding them. Unlike the previous tasks, the
$\mathbf{m}^{share}$ here is manually defined. The gradient guidance comes
from Eq. 5 without other specific designs.

###  3.4 Cross-branch Self-attention

To maintain consistency between the generated result and the original image,
we use two strategies: DDIM inversion [33] and a cross-branch self-attention
design. For DDIM inversion, we can also use the more accurate Null-text
inversion [21] to improve consistency. However, it is still challenging to
maintain high consistency between the editing result and the original image
solely through DDIM inversion. Here, inspired by the consistency preservation
in some video and image editing works [40, 39, 8], we design a cross-branch
self-attention guidance. Specifically, we replace the key and value in the
self-attention module of the denoiser in the generation branch with the
corresponding key and value from the guidance branch. Note that since the
feature correspondence in denoiser encoder is relatively weak [14], we only
use this operation in the denoiser decoder. The modified self-attention module
is defined as:

$\displaystyle\begin{split}\small\left\\{\begin{array}[]{ll}\mathbf{Q}=\mathbf{W}_{Q}^{gen}*\mathbf{F}^{gen};\
\mathbf{K}=\mathbf{W}_{K}^{gud}*\mathbf{F}^{gud};\
\mathbf{V}=\mathbf{W}_{V}^{gud}*\mathbf{F}^{gud}\\\
Attention(\mathbf{Q},\mathbf{K},\mathbf{V})=softmax(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d}})\mathbf{V},\end{array}\right.\end{split}$
(11)

where $\mathbf{W}_{Q}$, $\mathbf{W}_{K}$, and $\mathbf{W}_{V}$ are learnable
projection matrices. $*$ refers to the convolution operator. A comparison of
our method with and without cross-branch self-attention is shown in Fig. 4.
One can see that the design can effectively close the distance between the
generated result and the original image.

Figure 5: Visualization of our object moving and resizing applications. It can
be seen that our DragonDiffusion is capable of effectively moving objects on
real images, and at the same time, the region of the original object can also
be well inpainted. During the object moving process, we can also selectively
enlarge or shrink the object.

Figure 6: Visualization of object appearance replacement. Our method can
extract the appearance features of objects within the same category from a
reference image, and subsequently replace the appearance of objects in the
edited image accordingly.

Figure 7: Visualization of content dragging. Our method allows dragging image
content using one or multiple points. The results of continuous dragging
demonstrate the promising editing capabilities and stability of our
DragonDiffusion.

##  4 Experiments

In this paper, our DragonDiffusion can perform various image editing tasks,
including object moving, object resizing, object appearance replacement, and
content dragging. In Fig. 5, we demonstrate the application of object moving
and resizing. As can be seen, our method can naturally move objects within the
image, and the edited objects can blend well with the other content in the
original image. In Fig. 6, we present the performance of object appearance
replacement. It is obvious that our method can replace the appearance with
that of a same-category object from a reference image while preserving the
original outline. In Fig. 7, we present the content dragging performance of
our method. As can be seen, our method can drag the content within the image
using a single point or multiple points. The dragging results are consistent
and reasonable with the editing direction, and at the same time, the content
remains consistent with the original image.

##  5 Conclusion

Recent studies have shown that intermediate features in diffusion models
exhibit strong correspondence relationships. Compared to the correspondence
between text and image features, the correspondence between image and image
features is more stable and fine-grained. In this paper, we aim to develop a
fine-grained image editing scheme based on the strong correspondence of
intermediate features in diffusion models. To this end, we design a
classifier-guidance-based method to transform the editing signals into
gradients via feature correspondence loss to modify the intermediate
representation of the diffusion model. The feature correspondence loss is
designed with multiple scales to consider both semantic and geometric
alignment. Moreover, a cross-branch self-attention is added to maintain the
consistency between the original image and the editing result. Extensive
experiments demonstrate that our proposed DragonDiffusion can perform various
image editing applications for the generated or real images, including object
moving, object resizing, object appearance replacement, and content dragging.
At the same time, our DragonDiffusion does not require model fine-tuning or
additional modules.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4432–4441, 2019.
  * [2] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan++: How to edit the embedded images?  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8296–8305, 2020.
  * [3] Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano.  Hyperstyle: Stylegan inversion with hypernetworks for real image editing.  In Proceedings of the IEEE/CVF conference on computer Vision and pattern recognition, pages 18511–18521, 2022.
  * [4] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208–18218, 2022.
  * [5] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al.  ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.  arXiv preprint arXiv:2211.01324, 2022.
  * [6] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Universal guidance for diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 843–852, 2023.
  * [7] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023.
  * [8] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng.  Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing.  arXiv preprint arXiv:2304.08465, 2023.
  * [9] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in neural information processing systems, 34:8780–8794, 2021.
  * [10] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al.  Cogview: Mastering text-to-image generation via transformers.  Advances in Neural Information Processing Systems, 34:19822–19835, 2021.
  * [11] Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski.  Diffusion self-guidance for controllable image generation.  arXiv preprint arXiv:2306.00986, 2023.
  * [12] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang.  Training-free structured diffusion guidance for compositional text-to-image synthesis.  arXiv preprint arXiv:2212.05032, 2022.
  * [13] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [14] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [15] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz.  Multimodal unsupervised image-to-image translation.  In Proceedings of the European conference on computer vision (ECCV), pages 172–189, 2018.
  * [16] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [17] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [18] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007–6017, 2023.
  * [19] Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz.  Few-shot unsupervised image-to-image translation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10551–10560, 2019.
  * [20] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  arXiv preprint arXiv:2108.01073, 2021.
  * [21] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023.
  * [22] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  arXiv preprint arXiv:2302.08453, 2023.
  * [23] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  In International Conference on Machine Learning, pages 16784–16804. PMLR, 2022.
  * [24] Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt.  Drag your gan: Interactive point-based manipulation on the generative image manifold.  arXiv preprint arXiv:2305.10973, 2023.
  * [25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763, 2021.
  * [26] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
  * [27] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [28] Simo Ryu.  Low-rank adaptation for fast text-to-image diffusion fine-tuning, 2023\.
  * [29] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [30] Yujun Shi, Chuhui Xue, Jiachun Pan, Wenqing Zhang, Vincent YF Tan, and Song Bai.  Dragdiffusion: Harnessing diffusion models for interactive point-based image editing.  arXiv preprint arXiv:2306.14435, 2023.
  * [31] Jaskirat Singh, Stephen Gould, and Liang Zheng.  High-fidelity guided image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5997–6006, 2023.
  * [32] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  * [33] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [34] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  Advances in neural information processing systems, 32, 2019.
  * [35] Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models.  Advances in neural information processing systems, 33:12438–12448, 2020.
  * [36] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  arXiv preprint arXiv:2011.13456, 2020.
  * [37] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan.  Emergent correspondence from image diffusion.  arXiv preprint arXiv:2306.03881, 2023.
  * [38] Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or.  Sketch-guided text-to-image diffusion models.  arXiv preprint arXiv:2211.13752, 2022.
  * [39] Wen Wang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen.  Zero-shot video editing using off-the-shelf image diffusion models.  arXiv preprint arXiv:2303.17599, 2023.
  * [40] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  arXiv preprint arXiv:2212.11565, 2022.
  * [41] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang.  Freedom: Training-free energy-guided conditional diffusion model.  arXiv preprint arXiv:2303.09833, 2023.
  * [42] Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models, 2023.
  * [43] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun.  Lafite: Towards language-free training for text-to-image generation.  arXiv preprint arXiv:2111.13792, 2021.


# Mean-Field Assisted Deep Boltzmann Learning with Probabilistic Computers

Shuvro Chowdhury, Shaila Niazi, Kerem Y. Camsari

Department of Electrical and Computer Engineering
University of California Santa Barbara, Santa Barbara, CA 93106, USA
{schowdhury, sniazi, camsari} @ ucsb.edu

(September 2023)

###### Abstract

Despite their appeal as physics-inspired, energy-based and generative nature,
general Boltzmann Machines (BM) are considered intractable to train. This
belief led to simplified models of BMs with _restricted_ intralayer
connections or layer-by-layer training of deep BMs. Recent developments in
domain-specific hardware – specifically probabilistic computers (p-computer)
with probabilistic bits (p-bit) – may change established wisdom on the
tractability of deep BMs. In this paper, we show that deep and _unrestricted_
BMs can be trained using p-computers generating hundreds of billions of Markov
Chain Monte Carlo (MCMC) samples per second, on sparse networks developed
originally for use in D-Wave’s annealers. To maximize the efficiency of
learning the p-computer, we introduce two families of Mean-Field Theory
assisted learning algorithms, or xMFTs (x = Naive and Hierarchical). The xMFTs
are used to estimate the averages and correlations during the _positive phase_
of the contrastive divergence (CD) algorithm and our custom-designed
p-computer is used to estimate the averages and correlations in the negative
phase. A custom Field-Programmable-Gate Array (FPGA) emulation of the
p-computer architecture takes up to 45 billion flips per second, allowing the
implementation of CD-$n$ where $n$ can be of the order of millions, unlike
RBMs where $n$ is typically 1 or 2. Experiments on the full MNIST dataset with
the combined algorithm show that the positive phase can be efficiently
computed by xMFTs without much degradation when the negative phase is computed
by the p-computer. Our algorithm can be used in other scalable Ising machines
and its variants can be used to train BMs, previously thought to be
intractable.

##  1 Introduction

Since their introduction by Hinton and colleagues [1], Boltzmann Machines (BM)
have received sustained interest over the years [2]. Most recently, BMs have
found renewed interest in the representation of quantum many-body
wavefunctions as an alternative to quantum Monte Carlo algorithms (see, for
example, [3]). Meanwhile, the nearing end of Moore’s Law has been driving the
development of domain-specific computers tailored for specific applications
and algorithms. A notable class of such computers deals with probabilistic
computers (p-computer) with probabilistic bits (p-bit) (see, for example, [4,
5]). Probabilistic computers have been implemented at various sizes and in
different physical substrates. Magnetic nanodevices exploiting the ambient
thermal noise to build p-bits have been implemented in small scales (10-80
p-bits, [6, 7]). Custom-made digital _emulators_ using Field Programmable Gate
Arrays (FPGA) has been scaled much further, up to 5,000 - 10,000 p-bits [8].
Despite their small sizes at table-top experiments, nanodevice-based
p-computers are arguably the most scalable option to gain more performance and
energy-efficiency, with projections up to million p-bit densities [9], given
the success of magnetic memory technology [10]. The high costs of pseudorandom
number generators required for each p-bit make it prohibitively expensive to
get to such million-bit densities using existing CMOS technology [11].

Figure 1: p-computing overview: (a) Analogy between interacting bodies in
nature and interacting p-bit networks we build in this work. In stochastic MTJ
(sMTJ) based implementations of p-bits, a low energy barrier magnet is used to
generate natural noise. (b) Typical output of a p-bit against time fluctuating
randomly between $+1$ and $-1$. (c) Input/output characteristic of a p-bit.
The output (blue curve) is pinned to $\pm 1$ at strong positive and negative
inputs. The average (orange) has a tanh behavior. (d) In this work, we emulate
the p-bit in a digital system (FPGA) with a pseudorandom number generator
(PRNG), a lookup table for the tanh and a comparator. (e) The digital
emulation of the synapse with MUXes is also shown. (f) A p-computer consisting
of a network of such p-bits is then realized in an FPGA.

Nonetheless, CMOS emulators of p-computers are useful in demonstrating the
architectural and algorithmic potential of physics-inspired scalable
p-computers. In this paper, we use such a custom-designed, highly efficient
FPGA-based p-computer emulator (Fig. 1d-f) that can take 45 billion MCMC
samples every second (or flips per second, fps) to train deep Boltzmann
machines. Notably, this flips per second is about 4 - 5X faster than custom
GPU/TPU implementations implemented on far simpler networks with $\pm 1$
weights (see, for example, [12, 13, 14, 15]). Boltzmann Machines are trained
by the contrastive divergence algorithm, assuming a quadratic energy function
[16, 17],

$\Delta W_{ij}=\langle m_{i}m_{j}\rangle^{\mbox{data}}-\langle
m_{i}m_{j}\rangle^{\mbox{model}}\qquad\text{and}\qquad\Delta h_{i}=\langle
m_{i}\rangle^{\mbox{data}}-\langle m_{i}\rangle^{\mbox{model}}$ (1)

One difficulty in training unrestricted Boltzmann machines is the need to
perform explicit MCMC sampling in the positive (data) phase. To perform the
visible-to-hidden layer inference in one shot, BMs typically restrict
connections within a layer, removing the need for MCMC sampling. Removing
these connections, however, hurts the representational ability of the network.
The need for two separate phases is also detrimental to hardware development
[18] where each input in a batch needs to be clamped followed by MCMC
sampling, _serially_. In this paper, we propose a hybrid algorithm to
circumvent the positive phase sampling of _unrestricted_ Boltzmann machines.
Our main contributions are as follows:

(1) We implement a fast FPGA-based digital MCMC sampler emulating physical
probabilistic computers that are able to take up to 45 billion Gibbs samples
per second, communicating with a classical computer in a closed-loop setup
capable of training a deep _unrestricted_ BM with 2560 nodes and 17984
parameters to learn the full MNIST data set entirely in hardware, which is
rarely performed in direct hardware implementations of BMs.

(2) We propose a hybrid mean-field theory (MFT) assisted contrastive
divergence (CD) algorithm to ease the positive phase computation of
_unrestricted_ and _deep_ BMs. Going beyond naive MFTs (NMFT), we also propose
a _hierarchical_ MFT (HMFT), improving correlation estimations at the cost of
making $\mathcal{O}(N^{2})$ more NMFT calls.

(3) We demonstrate that the hybrid algorithm we design does not result in
significant degradation compared to the MCMC method since _positive_ phase
correlations are much more easily handled by MFTs as opposed to _negative_
phase correlations.

Figure 2: Hybrid computing scheme for ML: A hybrid computing scheme with
probabilistic and classical computers is shown. Inside the classical computer,
the positive phase is performed with the help of mean-field theory derivative
algorithms. At the beginning of the negative phase, the classical computer
sends weights and biases required to our probabilistic computer (PC) where we
perform Gibbs sampling. The probabilistic computer can generate a measured 45
billion Gibbs flips in a second (FPGA). The PC returns samples to the CPU
which computes the gradient. This process is repeated until convergence.

##  2 Gibbs Sampling with p-bits and Mean Field Theories

A p-bit randomly fluctuates between two states (say, in between $+1$ and $-1$)
with a continuous-valued input. Mathematically, an interconnected network of
p-bits is represented by the following two equations:

$\displaystyle I_{i}=\sum_{j}{W_{ij}m_{j}}+h_{i}\qquad\text{and}\qquad
m_{i}=\mathrm{sgn}\left(\tanh{\left(\beta I_{i}\right)-r_{[-1,1]}}\right)$ (2)

where $m_{i}\in\\{-1,+1\\}$ and $r_{[-1,1]}$ is a uniform random number drawn
from the interval $[-1,1]$. $\\{W_{ij}\\}$ are the weights, $\\{h_{i}\\}$ are
the biases and $\beta$ is the inverse temperature. When solved iteratively
also known as Gibbs sampling [19], Eq. (2) approximately follows the Boltzmann
distribution [20]:

$\displaystyle p(\\{m\\})=\frac{1}{Z}\,\exp{[-\beta
E(\\{m\\})]}\qquad\text{and}\qquad
E(\\{m\\})=-\sum_{i<j}W_{ij}m_{i}m_{j}-\sum_{i}h_{i}m_{i}$ (3)

The second equation in Eq. (2) is also known as the “binary stochastic neuron”
in machine learning. In the present context, the iterated evolution of these
equations represents a _dynamical system_ directly implemented in hardware. As
long as $I_{i}$ computation time is faster than $m_{i}$ computation time in
Eq. (2), the update order does not matter and can be random, supporting
asynchronous and massively parallel designs. In general-purpose computers
where Gibbs sampling is usually performed on software, it can become
computationally expensive, especially without the help of any accelerator.
Therefore, in many fields of physics and statistics, mean-field theory (MFT)
is instead widely used where one tries to approximate the behavior of a many-
body system by using an average field instead of individual interactions among
the components [21],[22]. This simplification of the system description to a
mere average significantly reduces the computational load. In the present
context, the relevant MFT equations to be solved self-consistently are the
following [23] where $\langle m\rangle\in(-1,1)$:

$\displaystyle\langle I_{i}\rangle=\sum_{j}{W_{ij}\langle
m_{i}\rangle}+h_{i}\qquad\text{and}\qquad\langle
m_{i}\rangle=\tanh{(\beta\langle I_{i}\rangle)}$ (4)

It is also worthwhile to note that although MFT yields a solution of a complex
system with less computational effort, the estimates from MFT are not always
accurate [23].

##  3 Hierarchical Mean Field Assisted CD Algorithm

In the context of Boltzmann machine learning, the idea of replacing the
correlations in Eq. (1) with MFTs was first introduced by Petersen et al.
[24]. Improvements to this idea such as linear response correction [25, 26,
27] and its higher order extensions [28] were also made. Hinton et al. [29]
proposed a deterministic variant of the contrastive divergence algorithm.
Recently, a variational mean field theory has also been proposed in [30].

Different from all these approaches considered earlier, in this paper we
propose a hybrid approach (Fig. 2): unlike most MFT approaches, the free
running phase is performed with Gibbs sampling but on a fast p-computer and
for the positive phase in the spirit of [2], we use an alternative method to
compute the correlations from the vanilla MFT method which we call
hierarchical mean-field theory (HMFT). In traditional MFT methods, the
correlations are calculated assuming independence between interacting bodies,
i.e., $\langle m_{i}m_{j}\rangle=\langle m_{i}\rangle\langle m_{j}\rangle$
[27]. In our approach, we do not use this assumption. Rather, we start from
the basic definition of correlation, i.e.,

$\displaystyle\langle m_{i}m_{j}\rangle$ $\displaystyle=\sum_{m_{i}=\pm
1,m_{j}=\pm 1}p(m_{i},m_{j})m_{i}m_{j}\qquad\text{with}\qquad
p(m_{i},m_{j})=p(m_{i}|m_{j})\,p(m_{j})$ (5)

When we compute MFT, we get an estimate for $p(m_{j})$. However, to use Eq.
(5), we also need to know or compute $p(m_{i}|m_{j})$. This can be done by
_clamping_ p-bit $j$ to $\pm 1$ and then performing another MFT estimating
$p(m_{i}|m_{j})$. After making $\Theta(2n)$ such MFT calls, we can estimate
the second-order correlations using Eq. (5). Our HMFT approach is presented in
Algorithm 1. HFMT improves correlation estimations by not baking in the
independence assumption $\langle m_{i}m_{j}\rangle=\langle m_{i}\rangle\langle
m_{j}\rangle$ as Naive MFT does (see Supplementary Information 1 for a
discussion on how HMFT can capture correlations completely missed by MFT
assuming independence, in a toy example). In fact, this Bayesian trick can be
used in conjunction with other methods that approximate marginals, such as
Loopy Belief Propagation [31] or with corrections to naive MFT (for example,
see [25]), improving the HFMT method. Moreover, higher-order correlations of
the form $\langle m_{i}m_{j}m_{k}\rangle$, $\langle
m_{i}m_{j}m_{k}m_{l}\rangle$, $\ldots$ can be _hierarchically_ estimated.
These can then be used to train higher-order Boltzmann machines [32], trading
off parallelizable MFT computations with the fundamentally serial Gibbs
sampling. In the experiments below, we investigate how the positive phase of
the contrastive divergence algorithm could be performed by the MFT and the
HFMT method we propose.

Input : weights and biases $J,h$, update factor $\lambda$, tolerance $\delta$,
max. iteration $T_{\mathrm{max}}$

Output : estimates for averages $\langle m_{\text{i}}\rangle$ and correlations
$\langle m_{i}m_{j}\rangle$

1 $\epsilon\leftarrow 1000$, $N\leftarrow\text{length}(h)$, $T\leftarrow 1$,
$m_{\text{old}}\leftarrow 0.01\,\text{rand}(-1,1)$

2 for  _$i\leftarrow 1$ to $N+1$_ do

3 for  _$j\leftarrow-1$ to $+1$ By $2$_ do

4 if  _$i\neq 1$_ then

$m_{\text{old},i-1}\leftarrow j$

$\triangleright$ clamping to $\pm 1$ to get conditional probability

5

6 while  _$\epsilon\geq\delta$_ do

7 $I\leftarrow Jm_{\text{old}}+h$

8 $m_{\text{new}}\leftarrow\tanh{(I)}$, $m_{\text{new},i-1}\leftarrow j$

9
$\epsilon\leftarrow\left(\sum_{i}|m_{\text{new,i}}-m_{\text{old,i}}|\right)/\left(\sum_{i}|m_{\text{new,i}}+m_{\text{old,i}}|\right)$

10 $m_{\text{old}}\leftarrow\lambda m_{\text{new}}+(1-\lambda)m_{\text{old}}$

11

12

$m_{\text{avg}}\leftarrow m_{\text{new}}$

$\triangleright$ spin averages

$p(m_{k}=\pm 1)\leftarrow 1/\left(1+\exp{(\mp 2I_{k})}\right)$

$\triangleright$ individual probabilities

$p(m_{k}=\pm 1|m_{i}=j)\leftarrow 1/\left(1+\exp{(\mp 2I_{k})}\right)$

$\triangleright$ conditional probabilities

13

14

$\langle m_{i}m_{j}\rangle\leftarrow$ Compute correlations from Eq. (5)

Algorithm 1 The Hierarchical Mean-field Algorithm

##  4 Experiments

We have used the MNIST dataset (handwritten digits, [33]) to train sparse,
deep and unrestricted Boltzmann networks without any downsampling, typically
performed on hardware implementations by D-Wave and others [12, 13, 14, 15].
We have used black/white images by thresholding the MNIST dataset and we
choose a Pegasus graph [34] with up to 2560 p-bits (nodes) as the sparse DBM
network model in this paper. The graph density of this Pegasus is 0.55% and
the maximum number of neighbors is 15. The network has 834 visible p-bits
(including 5 sets of labels each containing 10 p-bits) and 1726 hidden p-bits
that are arranged in 2 layers as shown in the inset of FIG. 3b. The total
number of connections (network parameters) in this graph is 17984. Using our
fast Gibbs sampler (p-computer), we accomplish the contrastive divergence
algorithm to train the MNIST dataset divided into 1200 mini-batches with 50
images in each batch. We used $10^{5}$ sweeps in the negative phases of each
epoch.

Similarly, we train the full MNIST using our hybrid MFT algorithm with naive
MFT in the positive phase and Gibbs sampling ($10^{5}$ sweeps) in the negative
phase. To find the classification accuracy, we perform a softmax
classification over 50 label p-bits to get the 10 labels. The p-bit with the
highest probability of being ‘1’ indicates the classified digit. FIG. 3a shows
that our sparse DBM with 2560 p-bits reaches around 87% accuracy with Gibbs
sampling and 70% accuracy (with MFT tolerance $=10^{-2}$ and this accuracy may
further improve with lower tolerance) with hybrid MFT technique in 100 epochs
despite having a significantly lesser number of parameters than typical RBMs.
For the full MNIST dataset, the computational expense of HMFT prevented us
from comparing it with the results of MFT at this time but in future this
could be made possible with more parallel resources like using GPUs. Moreover,
sophisticated techniques like layer-by-layer learning [2] should further
improve the accuracy reported in this work. Although our reported accuracy is
comparable with models with less parameters (e.g., regression models), the
real value of Boltzmann machines is their generative properties and has been
shown recently in [18].

To be able to evaluate both the efficiency of MFT and HFMT methods in the
positive phase, we performed the simpler task of training MNIST/100 (100
images randomly chosen from the MNIST dataset). Three different schemes used
the same hyper-parameters where the positive phase of training is accomplished
with naive MFT, HMFT (on CPU), and Gibbs sampling (on p-computer). The
negative phase is performed in our probabilistic computer where we are
naturally doing persistent CD algorithm (PCD) [35, 36]. This hybrid computing
scheme is illustrated in Fig. 2 and the details of the experimental setup can
be found in [18]. The training accuracy of 100 images reaches 100% with Gibbs
sampling and naive MFT and HMFT also perform similarly. Supplementary Table S2
indicates that despite a large difference in the training set log-likelihood,
the test set shows roughly similar results, indicating how the hybrid approach
does not degrade the performance significantly. The performance of this
approach on larger datasets and networks remains to be seen.

It is interesting to note here that when Gibbs sampling in the negative phase
is replaced by xMFTs, both training and test set accuracies degrade severely
and, in fact, do not work at all. The supplementary Table S1 shows the poorer
performance of xMFTs in the negative phases.

Figure 3: MNIST accuracy with different methods: (a) Full MNIST (60,000
images) is trained on sparse DBM (Pegasus 2560 p-bits) with Gibbs sampling
(CD-$10^{5}$) and naive MFT where batch size = 50, learning rate = 0.003,
momentum = 0.6. Around 87% accuracy is achieved in 100 epochs for Gibbs
sampling and 70% for the naive MFT. Test accuracy represents the accuracy of
all 10,000 images from the MNIST test set, while the training accuracy
corresponds to the accuracy of 10,000 images randomly drawn from the training
set. (b) Training accuracy of MNIST/100 with the three different schemes:
naive MFT, HMFT, and Gibbs sampling where they perform similarly. Here the
batch size = 10, momentum = 0.6 and learning rate varies from 0.06 to 0.006
over 1000 epochs.

##  5 Conclusions and Outlook

The end of Moore’s law is driving the development of physics-based
probabilistic computers that can accelerate MCMC algorithms by orders of
magnitude. In this paper, we showed an FPGA emulation of such a computer that
can take up to 45 billion Gibbs samples per second. Experimentally-validated
projections indicate up to $10^{15}$ samples per second are possible with
truly physical p-computers. Such a large increase in MCMC speeds may allow the
direct training of deep and _unrestricted_ BMs. As an example of this
approach, we trained a 2-layer unrestricted deep BM on a sparse Pegasus graph
used by D-Wave, showing promising results (near 90% classification accuracy
with only $\approx$18k parameters). To aid with the positive phase training of
unrestricted and deep BMs, we also proposed a hierarchical mean-field theory
assisted learning algorithm. In accordance with common wisdom, we found that
MFTs fail to estimate model correlations, especially when the weights become
large. Surprisingly, however, we observed that MFTs accurately approximate
data correlations, during the positive phase, greatly simplifying training in
hardware. With the development of new physical computers, our results may
allow the training of deep and unrestricted BMs previously thought to be
intractable where hybrid MFT approaches could be used during pretraining or as
a supplement to the computationally expensive Gibbs sampling. Extensions of
the model by fully exploiting the parallelism offered by the MFTs in deeper
networks to train harder datasets are left for future study.

## Acknowledgments and Disclosure of Funding

Authors acknowledge support from the Office of Naval Research Young
Investigator Program and National Science Foundation grants.

## References

  * Ackley et al. [1985] David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski.  A learning algorithm for Boltzmann machines, 1985.
  * Salakhutdinov and Hinton [2009] Ruslan Salakhutdinov and Geoffrey Hinton.  Deep boltzmann machines.  In David van Dyk and Max Welling, editors, _Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics_ , volume 5 of _Proceedings of Machine Learning Research_ , pages 448–455, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16–18 Apr 2009. PMLR.
  * Carleo et al. [2019] Giuseppe Carleo, Ignacio Cirac, Kyle Cranmer, Laurent Daudet, Maria Schuld, Naftali Tishby, Leslie Vogt-Maranto, and Lenka Zdeborová.  Machine learning and the physical sciences.  _Reviews of Modern Physics_ , 91(4):045002, 2019.
  * Camsari et al. [2017a] Kerem Yunus Camsari, Rafatul Faria, Brian M Sutton, and Supriyo Datta.  Stochastic p-bits for invertible logic.  _Physical Review X_ , 7(3):031014, 2017a.
  * Chowdhury et al. [2023] Shuvro Chowdhury, Andrea Grimaldi, Navid Anjum Aadit, Shaila Niazi, Masoud Mohseni, Shun Kanai, Hideo Ohno, Shunsuke Fukami, Luke Theogarajan, Giovanni Finocchio, et al.  A full-stack view of probabilistic computing with p-bits: devices, architectures and algorithms.  _IEEE Journal on Exploratory Solid-State Computational Devices and Circuits_ , 2023.
  * Borders et al. [2019] William A Borders, Ahmed Z Pervaiz, Shunsuke Fukami, K. Y. Camsari, Hideo Ohno, and Supriyo Datta.  Integer factorization using stochastic magnetic tunnel junctions.  _Nature_ , 2019.
  * Si et al. [2023] Jia Si, Shuhan Yang, Yunuo Cen, Jiaer Chen, Zhaoyang Yao, Dong-Jun Kim, Kaiming Cai, Jerald Yoo, Xuanyao Fong, and Hyunsoo Yang.  Energy-efficient superparamagnetic Ising machine and its application to traveling salesman problems.  _arXiv preprint arXiv:2306.11572_ , 2023.
  * Aadit et al. [2022] Navid Anjum Aadit, Andrea Grimaldi, Mario Carpentieri, Luke Theogarajan, John M Martinis, Giovanni Finocchio, and Kerem Y Camsari.  Massively parallel probabilistic computing with sparse Ising machines.  _Nature Electronics_ , 5(7):460–468, 2022.
  * Sutton et al. [2020] Brian Sutton, Rafatul Faria, Lakshmi Anirudh Ghantasala, Risi Jaiswal, Kerem Yunus Camsari, and Supriyo Datta.  Autonomous probabilistic coprocessing with petaflips per second.  _IEEE Access_ , 8:157238–157252, 2020.
  * Lin et al. [2009] CJ Lin, SH Kang, YJ Wang, K Lee, X Zhu, WC Chen, X Li, WN Hsu, YC Kao, MT Liu, et al.  45nm low power cmos logic compatible embedded stt mram utilizing a reverse-connection 1t/1mtj cell.  In _Electron Devices Meeting (IEDM), 2009 IEEE International_ , pages 1–4. IEEE, 2009.
  * Kobayashi et al. [2023] Keito Kobayashi, Nihal Singh, Qixuan Cao, Kemal Selcuk, Tianrui Hu, Shaila Niazi, Navid Anjum Aadit, Shun Kanai, Hideo Ohno, Shunsuke Fukami, et al.  CMOS+ stochastic nanomagnets: heterogeneous computers for probabilistic inference and learning.  _arXiv preprint arXiv:2304.05949_ , 2023.
  * Adachi and Henderson [2015] Steven H Adachi and Maxwell P Henderson.  Application of quantum annealing to training of deep neural networks.  _arXiv preprint arXiv:1510.06356_ , 2015.
  * Manukian et al. [2019] Haik Manukian, Fabio L Traversa, and Massimiliano Di Ventra.  Accelerating deep learning with memcomputing.  _Neural Networks_ , 110:1–7, 2019.
  * Dixit et al. [2021] Vivek Dixit, Raja Selvarajan, Muhammad A Alam, Travis S Humble, and Sabre Kais.  Training Restricted Boltzmann Machines With a D-Wave Quantum Annealer.  _Frontiers in Physics_ , 9:589626, 2021.
  * Böhm et al. [2022] Fabian Böhm, Diego Alonso-Urquijo, Guy Verschaffelt, and Guy Van der Sande.  Noise-injected analog Ising machines enable ultrafast statistical sampling and machine learning.  _Nature Communications_ , 13(1):5847, 2022.
  * Hinton [2002] Geoffrey E Hinton.  Training products of experts by minimizing contrastive divergence.  _Neural computation_ , 14(8):1771–1800, 2002.
  * Larochelle et al. [2007] Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio.  An empirical evaluation of deep architectures on problems with many factors of variation.  In _Proceedings of the 24th international conference on Machine learning_ , pages 473–480, 2007.
  * Niazi et al. [2023] Shaila Niazi, Navid Anjum Aadit, Masoud Mohseni, Shuvro Chowdhury, Yao Qin, and Kerem Y Camsari.  Training Deep Boltzmann Networks with Sparse Ising Machines.  _arXiv preprint arXiv:2303.10728_ , 2023.
  * Koller and Friedman [2009] Daphne Koller and Nir Friedman.  _Probabilistic Graphical Models: Principles and Techniques - Adaptive Computation and Machine Learning_.  The MIT Press, 2009.  ISBN 0262013193.
  * Camsari et al. [2017b] Kerem Yunus Camsari, Rafatul Faria, Brian M Sutton, and Supriyo Datta.  Stochastic p-bits for invertible logic.  _Physical Review X_ , 7(3):031014, 2017b.
  * Chaikin and Lubensky [1995] P. M. Chaikin and T. C. Lubensky.  _Mean-field theory_ , page 144–212.  Cambridge University Press, 1995.  doi: 10.1017/CBO9780511813467.005.
  * Kardar [2007] Mehran Kardar.  _Statistical Physics of Particles_.  Cambridge University Press, 2007.  doi: 10.1017/CBO9780511815898.
  * Sakthivadivel [2022] Dalton A R Sakthivadivel.  Magnetisation and Mean Field Theory in the Ising Model.  _SciPost Phys. Lect. Notes_ , page 35, 2022.  doi: 10.21468/SciPostPhysLectNotes.35.
  * Petersen and Anderson [1987] Carsten Petersen and James R Anderson.  A mean field theory learning algorithm for neural networks.  _Complex Systems_ , 1:995–1019, 1987.
  * Kappen and de Borja Rodríguez Ortiz [1997] Hilbert Kappen and Francisco de Borja Rodríguez Ortiz.  Boltzmann machine learning using mean field theory and linear response correction.  In M. Jordan, M. Kearns, and S. Solla, editors, _Advances in Neural Information Processing Systems_ , volume 10. MIT Press, 1997.
  * Kappen and Rodríguez [1997] H.J. Kappen and F.B. Rodríguez.  Mean field approach to learning in Boltzmann Machines.  _Pattern Recognition Letters_ , 18(11):1317–1322, 1997.  ISSN 0167-8655.  doi: https://doi.org/10.1016/S0167-8655(97)00096-2.
  * Kappen and Rodríguez [1998] Hilbert J. Kappen and FDB Rodríguez.  Efficient learning in Boltzmann machines using linear response theory.  _Neural Computation_ , 10(5):1137–1156, 1998.
  * Tanaka [1998] Toshiyuki Tanaka.  Mean-field theory of Boltzmann machine learning.  _Phys. Rev. E_ , 58:2302–2310, Aug 1998.  doi: 10.1103/PhysRevE.58.2302.
  * Welling and Hinton [2002] Max Welling and Geoffrey E. Hinton.  A New Learning Algorithm for Mean Field Boltzmann Machines.  In José R. Dorronsoro, editor, _Artificial Neural Networks — ICANN 2002_ , pages 351–357, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg.  ISBN 978-3-540-46084-8.
  * Huang [2020] Haiping Huang.  Variational mean-field theory for training restricted Boltzmann machines with binary synapses.  _Phys. Rev. E_ , 102:030301, Sep 2020.  doi: 10.1103/PhysRevE.102.030301.
  * Mezard and Montanari [2009] Marc Mezard and Andrea Montanari.  _Information, physics, and computation_.  Oxford University Press, 2009.
  * Sejnowski [1986] Terrence J Sejnowski.  Higher-order Boltzmann machines.  In _AIP Conference Proceedings_ , volume 151, pages 398–403. American Institute of Physics, 1986.
  * LeCun [1998] Yann LeCun.  The MNIST database of handwritten digits.  _http://yann. lecun. com/exdb/mnist/_ , 1998.
  * Dattani et al. [2019] Nike Dattani, Szilard Szalay, and Nick Chancellor.  Pegasus: The second connectivity graph for large-scale quantum annealing hardware.  _arXiv preprint arXiv:1901.07636_ , 2019.
  * Hinton [2012] Geoffrey E. Hinton.  _A Practical Guide to Training Restricted Boltzmann Machines_ , pages 599–619.  Springer Berlin Heidelberg, Berlin, Heidelberg, 2012.
  * Tieleman [2008] Tijmen Tieleman.  Training restricted Boltzmann machines using approximations to the likelihood gradient.  In _Proceedings of the 25th international conference on Machine learning_ , pages 1064–1071, 2008.

## Supplementary Information

##  1 HMFT vs NMFT in a toy 2-spin example

Consider a simple two-spin antiferromagnetic system with weights
$W_{ij}=\delta_{ij}-1$ and no biases. At any temperature, the marginals of
these two spins will be identically, i.e., $\langle m_{i}\rangle=0$. As such,
the naive mean field method which uses conditional independence assumption
$\langle m_{i}m_{j}\rangle=\langle m_{i}\rangle\langle m_{j}\rangle$ will
estimate the correlation between these two spins to be identically zero, which
is clearly incorrect, since at any non-zero temperature the spins will be
anti-correlated. On the other hand, the hierarchical mean field method which
_does not_ assume the conditional independence and rather uses Eq. (4) to
compute correlations, estimates a non-zero correlation. For example, at $T=1$,
the exact correlation (from Boltzmann law) between the two spins for the given
weights and biases is $-0.7616$. The hierarchical method also estimates the
same correlation value. Higher-order correlations (hierarchically obtained)
can also be shown to be better estimated by the HMFT method.

##  2 Contrastive divergence algorithm

In this section, we briefly outline the contrastive divergence algorithm in
our hybrid approach which implements Eq. (1):

Input : number of samples $N$, batch size $B$, number of batches $N_{B}$,
epochs $N_{\text{L}}$, learning rate $\varepsilon$, mean-field update factor
$\lambda$, mean-field tolerance $\delta$, maximum iteration for mean-field
$T_{\mathrm{max}}$

Output : trained weights $J_{\text{out}}$ and biases $h_{\text{out}}$

1 $J_{\text{out}}\leftarrow\mathcal{N}(0,0.01)$,
$h_{\text{out,hidden}}\leftarrow 0$,
$h_{\text{out,visible}}\leftarrow\log{(p_{i}/(1-p_{i}))}$

2 for  _$i\leftarrow 1$ to $N_{\text{L}}$_ do

3 for  _$j\leftarrow 1$ to $N_{\text{B}}$_ do

/* positive phase */

4 for  _$k\leftarrow 1$ to $B$_ do

5 $h_{\text{$B$}}\leftarrow\text{ clamping to batch images}$

6 $\langle m_{i}\rangle^{(k)},\langle
m_{i}m_{j}\rangle^{(k)}\leftarrow\text{{x}MFT$\\_$module}(J_{\text{out}},h_{B},\lambda,\delta,T_{max})$

7

$\langle m_{i}\rangle_{\text{data}}=\text{mean}\left(\\{\langle
m_{i}\rangle^{(k)}\\}\right)$, $\langle
m_{i}m_{j}\rangle_{\text{data}}=\text{mean}\left(\\{\langle
m_{i}m_{j}\rangle^{(k)}\\}\right)$

$\triangleright$ CPU

/* negative phase */

8 $h_{\text{Sampler}}\leftarrow h_{\text{out}}$, $J_{\text{Sampler}}\leftarrow
J_{\text{out}}$

$\\{m\\}\leftarrow\text{GibbsSampler}(N)$

$\triangleright$ p-computer

$\langle m_{i}\rangle_{\text{model}}=\text{mean}(\\{m\\})$, $\langle
m_{i}m_{j}\rangle_{\text{model}}=\\{m\\}\\{m\\}^{\text{T}}/(N)$

$\triangleright$ CPU

/* update weights and biases */

$J_{\text{out},ij}\leftarrow J_{\text{out},ij}+\epsilon\left(\langle
m_{i}m_{j}\rangle_{\text{data}}-\langle
m_{i}m_{j}\rangle_{\text{model}}\right)$

$\triangleright$ CPU

$h_{\text{out},i}\leftarrow h_{\text{out},i}+\epsilon\left(\langle
m_{i}\rangle_{\text{data}}-\langle m_{i}\rangle_{\text{model}}\right)$

$\triangleright$ CPU

9

10

Algorithm 2 Mean-field assisted training of sparse DBMs

##  3 Evolution of correlations over epochs

Figure S1: Typical predictions of correlations from different methods: The
correlations predicted by the three different schemes - naive MFT, HMFT, and
Gibbs sampling (the putative exact method since we cannot obtain exact
Boltzmann correlations in general spin-glasses) are shown during a typical
epoch in the training of a sparse DBM. We used a batch size of 10 images and
for the positive phase, we obtained correlations by showing only one batch of
10 images. For Gibbs sampling, we used $10^{4}$ sweeps in both positive and
negative phases. We chose a relative error tolerance of $10^{-2}$ for both MFT
and HMFT. 20 bins were used in both histograms. MFT algorithms do
significantly better in the positive phase than in the negative phase allowing
their use in the positive phase training of deep and unrestricted BMs, instead
of the more expensive Gibbs sampling.

In this section, we discuss the evolution of correlations over epochs for
naive and hierarchical MFT. Typical predictions of correlations from xMFT
algorithms are shown in Supplementary Fig. S1. It can be clearly seen that in
the positive phase, xMFT algorithms provide nearly accurate estimations for
correlations. The clamping of many pbits during the positive phase helps xMFT
algorithms to boost their performance but in the absence of such clamps, their
performances degrade severely in the negative phase. In a hybrid setting, the
probabilistic computer suffers from the communication delay caused by the fact
that images are to be sent repeatedly whereas in the negative phase, there is
no such delay. These two considerations justify the desire to replace Gibbs
sampling in the positive phase with xMFT algorithms.

In order to provide a quantitative measure of the performance between the two
xMFT algorithms discussed in this paper, we define the relative average error
between the two correlation matrices estimated from two different approaches
as
$\epsilon=\sqrt{\sum_{ij}{(A_{ij}-B_{ij})^{2}}}/\sqrt{\sum_{ij}{B_{ij}^{2}}}$
where $A$ is the correlation matrix predicted by the xMFT algorithms and $B$
is the corresponding matrix for the Gibbs sampling. We do not include the zero
correlation values in this measure. Since at the 2560 p-bits level, it is
impossible to obtain correlations from the exact distribution, we use Gibbs
sampling as the “putative” reference for comparison. Supplementary Table S1
lists this measure for the xMFT algorithms both in the positive and negative
phases. The performance of the xMFT algorithms in both phases is consistent
with Supplementary Fig. S1. As mentioned in the main text, in our HMFT
implementations we do not modify the average estimations from naive MFT
therefore both xMFT algorithms show the same accuracy for averages. For
correlations, HMFT estimates are slightly better than naive MFT. It is also
interesting to note that both models (MFT/HFMT) perform better at lower epochs
when the weights are small which may suggest their use in possible pre-
training supplementing the exact Gibbs sampling approach.

Table S1: Relative average error is shown in positive and negative phases for two MFT algorithms. The error is measured with respect to Gibbs sampling at each phase (we take $10^{4}$ sweeps for both positive and negative phases). The tolerance of the MFT algorithms is set to $10^{-2}$. xMFT predicted averages and correlations are significantly better in the positive phase than in the negative phase. | Positive phase | Negative phase
---|---|---
epoch | averages | correlations | averages | correlations
| $\langle m_{i}\rangle$ | $\langle m_{i}m_{j}\rangle$ | $\langle m_{i}\rangle$ | $\langle m_{i}m_{j}\rangle$
| MFT | HMFT | MFT | HMFT | MFT | HMFT | MFT | HMFT
1 | 0.72% | 0.72% | 1.55% | 1.19% | 2.00% | 2.00% | 3.82% | 2.967%
5 | 0.69% | 0.69% | 1.39% | 1.11% | 5.82% | 5.82% | 8.82% | 8.00%
10 | 0.63% | 0.63% | 1.24% | 1.00% | 11.73% | 11.73% | 16.40% | 16.07%
50 | 0.59% | 0.59% | 1.11% | 0.92% | 35.85% | 35.85% | 46.80% | 46.61%
100 | 0.61% | 0.61% | 1.11% | 0.93% | 44.44% | 44.44% | 55.4% | 55.16%

##  4 Log-likelihood measure for xMFT algorithms

Table S2: Train (100 images) and test set (20 images) log-likelihood for different samplers i.e., Gibbs-Gibbs, naive MFT-Gibbs, HMFT-Gibbs for sparse deep Boltzmann machines. MNIST/100 | Gibbs-Gibbs | HMFT-Gibbs | MFT-Gibbs
---|---|---|---
Train set | -35.08 | -71.07 | -89.82
Test set | -33.87 | -37.71 | -37.63

In this section, we report the log-likelihood ($\mathcal{L}$) measure defined
as

$\mathcal{L}=\sum_{i=1}^{N}\sum_{c=1}^{C}y_{i,c}\cdot\log(p_{i,c})$ (S.1)

for the xMFT algorithms in both the training and the test set. Here, $y_{i,c}$
is the true label of the $i^{\text{th}}$ sample for class $c$ (1 if the sample
belongs to class $c$ and 0 otherwise), and $p_{i,c}$ is the predicted
probability that the $i^{\text{th}}$ sample belongs to class $c$. We measure
the performance after training MNIST/100, with three different choices of
algorithms to be used in the positive phase namely, Gibbs sampling, naive MFT
and HMFT. The negative phase is always performed with Gibbs sampling.
Supplementary Table S2 lists this measure.

As can be seen, the HMFT-trained model performs better than the naive MFT-
trained model in both the training and test set although in the test set case,
this difference is minimal.