# Deep Quantized Representation for Enhanced Reconstruction

###### Abstract

While machine learning approaches have shown remarkable performance in
biomedical image analysis, most of these methods rely on high-quality and
accurate imaging data. However, collecting such data requires intensive and
careful manual effort. One of the major challenges in imaging the Shoot Apical
Meristem (SAM) of Arabidopsis thaliana, is that the deeper slices in the
$z-$stack suffer from different perpetual quality related problems like poor
contrast and blurring. These quality related issues often lead to disposal of
the painstakingly collected data with little to no control on quality while
collecting the data. Therefore, it becomes necessary to employ and design
techniques that can enhance the images to make it more suitable for further
analysis. In this paper, we propose a data-driven Deep Quantized Latent
Representation (DQLR) methodology for high-quality image reconstruction in the
Shoot Apical Meristem (SAM) of Arabidopsis thaliana. Our proposed framework
utilizes multiple consecutive slices in the $z$-stack to learn a low
dimensional latent space, quantize it and subsequently perform reconstruction
using the quantized representation to obtain sharper images. Experiments on a
publicly available dataset validate our methodology showing promising results.
Our code is available at github.com/agupt013/enhancedRec.git.

Index Terms—  Cell reconstruction, quantized representation, shoot apical
meristem, arabidopsis thaliana

##  1 Introduction

Fig. 1: Conceptual Overview of DQLR. The latent representation of the
collected image is quantized using $k-$means over the entire dataset [1]. This
quantized representation is then used to reconstruct the enhanced image.

Automated analysis in biomedical research is critical to provide researchers
with concrete evidence to prove any proposed hypothesis without any bias.
However, automated image analysis requires high-quality imaging data. Image
quality related problems are often encountered while imaging deeper layers of
the Shoot Apical Meristem (SAM) of arabidopsis thaliana [2]. These quality
related problems hinder automated analysis and often lead to disposal of
painstakingly collected data. To this end, we propose a data driven Deep
Quantized Latent Representation (DQLR) framework for high-quality imaging data
reconstruction of the $z-$stack of the SAM. In this work, we propose to
project noisy stack in a latent space, quantize the latent representations and
utilize the quantized latent representations for reconstruction of enhanced
$z-$stack (see Fig. 1 for conceptual overview).

Overview. An architectural overview of our approach is illustrated in Fig. 2.
During training, the encoder $\mathsf{E}$ compresses $i^{th}$ input slice
image to a latent representation $x_{i}$. The consecutive slices in the
$z$-stack are correlated which implies that they must be correlated in the
latent space as well. We employ a recurrent neural network (RNN) $\mathsf{R}$
to learn this correlated representation $\\{y_{i},y_{i+1},\cdots,y_{i+n}\\}$
by passing the latent vector $\\{x_{1},x_{2},\cdots,x_{n}\\}$ through
$\mathsf{R}$. The compressed representation $x_{i}$ is processed through
$\mathsf{R}_{i}$ to learn the inter-correlation between this latent
representation of the consecutive slices
$\\{{x_{i},x_{i+1},\cdots,x_{i+n}}\\}$ during training. RNN generated latent
codes $\\{y_{i},y_{i+1},\cdots,y_{i+n}\\}$ are then used as input to
quantization module $\mathsf{Q}_{i}$. $\mathsf{Q}_{i}$ learns a vector
dictionary for quantized representation of the network and generates a
quantized latent code $\\{y^{q}_{i},y^{q}_{i+1},\cdots,y^{q}_{i+n}\\}$. In our
proposed method, the quantization of the latent code will remove the noisy
component of $\\{y_{i}\\}$ and the reconstructed/predicted images using the
quantized latent codes by generator $\mathsf{G}$ should be enhanced. During
testing, we pass one slice at a time from the $z-$stack, compress it using the
encoder, predict the correlated latent codes using the RNN, and finally
quantize it using the quantization dictionary learned during the training
stage using $\mathsf{Q}$. This quantized code is then used to reconstruct and
predict enhanced consecutive slices from the given $z-$stack.

##  2 Related Work

In this section we describe prior works closely related to the our proposed
method. Our method closely relates to reconstruction using auto-encoders [3]
and enhancement in the compressed domain [1, 4, 5].

Auto-Encoders. Variations of auto-encoders are extensively used in
reconstruction tasks by compressing the input to a latent representation and
using the latent representation to retrieve the input as close as possible [1,
3, 6]. However, often the reconstructed images are blurry due to inherent
nature of Mean Square Error (MSE) loss to produce blurry results. In the
proposed approach we also include Structural Similarity Index (SSIM) [7] loss
to enhance the visual results.

Compressed Domain Enhancement. Some works have tried to enhance the images in
the compressed domain. In [4] a method based on a contrast measure defined
within the discrete cosine transform (DCT) domain is proposed to enhance the
image. Attention based video enhancement is proposed in [8]. Authors in [1]
propose a vector quantized variations auto-encoder for reconstruction of
various media input. We adopt their approach of vector quantization in our
framework. However, we exploit the input data correlation using RNN for
enhancement task as opposed to reconstruction in [1] where ground truth data
was available.

##  3 Methodology

Fig. 2: Architectural Overview of DQLR (for one slice of the stack). Encoder
$\mathsf{E}$ encodes input image to $x_{i}$. Recurrent Neural Network (RNN)
module generates correlated codes for reconstruction ($y_{i}$) and prediction
$\left(\\{y_{i},y_{i+1},\cdots,y_{i+n}\\}\right)$. Quantizer module
$\mathsf{Q}_{i}$ quantizes the latent codes and Generator $\mathsf{G}$
reconstructs/predicts the images.

We propose a Deep Quantized Latent Representation (DQLR) framework for
enhancing $z-$stack imaging in SAM of Arabidopsis thaliana. We apply
quantization in the latent space of the noisy $z-$stack for enhanced
reconstruction. In this section, we first formulate the problem statement and
then explain our proposed approach in details.

###  3.1 Problem Formulation

Given a $z-$stack $\mathbf{Z}$ =
$\\{\mathbf{z}_{1},\mathbf{z}_{2},\cdots,\mathbf{z}_{n}\\}$, with
$\textbf{z}_{i}$ being the $i^{th}$ slice in the stack from the top, we aim to
reconstruct $\widehat{\mathbf{Z}}$ =
$\\{\widehat{\mathbf{z}}_{1},\widehat{\mathbf{z}}_{2},\cdots,\widehat{\mathbf{z}}_{n}\\}$
such that $\widehat{\mathbf{z}}_{i}$ is the visually enhanced slice compared
to $\mathbf{z}_{i},\forall i=1,2,\cdots,n$. Let there be a latent
representation of input noisy $z-$stack
$\textbf{X}_{\mathbf{Z}}=\\{x_{1},x_{2},\cdots,x_{n}\\}$ where $x_{i}$ is the
latent representation corresponding to the $i^{th}$ slice $\mathbf{z}_{i}$.
Since the slices in $z-$stack are correlated in the pixel space, their latent
representations should inherit the same property in the latent space.
Therefore, corresponding to each latent representation
$\textbf{X}_{\mathbf{Z}}$ let there be a latent representation
$\textbf{Y}_{\mathbf{Z}}=$ $\\{y_{1},y_{2},\cdots,y_{n}\\}$ such that all
$\\{y_{i}\\}$ are correlated.

We propose to generate visually enhanced $z-$stack by quantizing the latent
representation of the noisy input stack. Our hypothesis is that each
correlated latent representation $y_{i}$ of a slice in the $z-$stack consists
of two components; the quantized representation $y^{q}_{i}$ and the noise
representation $y^{\text{noise}}_{i}$ of $y_{i}$, such that
$y_{i}=y^{q}_{i}+y^{\text{noise}}_{i}$. Hence, noise component
$y^{\text{noise}}_{i}$ can be removed by applying quantization on the
correlated latent codes leaving the representation $y^{q}_{i}$ required to
generate the enhanced image
$\widehat{\textbf{z}}_{i}~{}\forall~{}i=1,2,\cdots,n$.

###  3.2 Proposed Approach

Our proposed framework is shown in Figure 2. It consists of four components:
the encoder network $\mathsf{E}$, the recurrent neural network $\mathsf{R}$,
the quantization module $\mathsf{Q}$ and the generator network $\mathsf{G}$.
The encoder network is used to extract latent representation for each slice in
the noisy input stack. The recurrent neural network utilizes the latent
representations to generate correlated latent representations. These
correlated representations are quantized to reduce noise in the latent space
by the quantization module. Finally, the quantized representations are used to
generate an enhanced $z-$stack.

Input Latent Representation. We employ a convolutional neural network as an
encoder $\mathsf{E}$ which extracts the latent representation for each slice
in a given noisy $z-$stack such that

$\displaystyle\mathsf{E}\left(\mathbf{Z}\right)=\mathsf{E}\left(\\{~{}\textbf{z}_{1},~{}\textbf{z}_{2},\cdots,~{}\textbf{z}_{n}\\}\right)=\\{~{}x_{0},~{}x_{1},\cdots,~{}x_{n}\\}$
(1)

where $x_{i}$ is latent representation corresponding to slice
$\textbf{z}_{i}$. A set of correlated representations is generated by the
recurrent neural network for the latent representations extracted from the
encoder $\mathsf{E}$ to incorporate the z-resolution dynamics of the $z-$stack
in the latent representations.

Recurrent Neural Network (RNN). The consecutive slices in a $z-$stack capture
3D-structure of any cell in the plant. Thus, there must be a correlation
between the consecutive slices. The latent representation
$\textbf{X}_{\mathbf{Z}}$ of the noisy input $\mathbf{Z}$ should also be
correlated in some space $\textbf{Y}_{\mathbf{Z}}$. Thus, we employ a
recurrent neural network $\mathsf{R}_{i}$ to transform the $i^{th}$ noisy
latent representation to the correlated latent representation as RNN can
capture dynamics of the sequence given by

$\displaystyle y_{i+1}=\mathsf{R}_{i}\left(y_{i},h_{i}\right)$ (2)

where $h_{0}$ is the hidden state sampled randomly from a Gaussian
distribution and $h_{i}=x_{i-1}~{}\forall~{}i>0$. Here, we aim to capture the
$z-$resolution dynamics of the stack unlike traditional recurrent neural
network where temporal dynamics of the sequence is captured.

Deep Quantized Latent Representation. We propose that a data driven
quantization of the latent representation can reduce the average noise in the
stack and enhance it visually. In order to quantize the latent representation,
we employ vector quantization dictionary learning algorithm as proposed in
[1], represented as $\mathsf{Q_{i}}$ in our framework.

Enhanced Stack Generation. We employ a generative model $\mathsf{G}$ to
transform the quantized representations into an enhanced stack
$\widehat{\mathbf{Z}}$. The quantized representations
$\textbf{Y}^{q}_{\mathbf{Z}}$ are used by the generator $\mathsf{G}$ to
synthesize enhanced stack
$\widehat{\mathbf{Z}}=\\{\widehat{\mathbf{z}}_{1},\widehat{\mathbf{z}}_{2},\cdots,\widehat{\mathbf{z}}_{n}\\}$
such that $\widehat{\mathbf{z}_{i}}$ is the visually enhanced image of the
slice $\mathbf{z}_{i}$ in the noisy stack $\mathbf{Z}$.

###  3.3 Optimization

Our optimization function consists of the Mean Squared Error (MSE) pixel
reconstruction loss, the Structural Similarity (SSIM) loss [7] and
quantization loss as defined in [1]. Please note that we do not have de-noised
image as ground truth. We assume that the quantized latent codes should reduce
noise when it is used by generator $\mathsf{G}$ to reconstruct the stack.
Results in section 4 demonstrate the validity of this assumption.

$\displaystyle\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{mse}}+\lambda_{\text{s}}\mathcal{L}_{\text{ssim}}+\lambda_{\text{q}}\mathcal{L}_{\text{quant}}$
(3)

We briefly describe the loss functions below. Define $P$ as the total number
of non-overlapping patches in a given image, $N$ as total number of pixels in
$P$, and $\alpha$ and $\beta$ as the generated and ground truth image,
respectively.

$\displaystyle\mathcal{L}_{\text{mse}}(P)=\dfrac{1}{N}\sum_{p\in
P}\|\alpha(p)-\beta(p)\|_{2}$
$\displaystyle\mathcal{L}_{\text{ssim}}(P)=\dfrac{1}{N}\sum_{p\in
P}1-\text{SSIM}(p),$
$\displaystyle\text{with},~{}\text{SSIM}(p)=\left(\dfrac{2\mu_{\alpha}\mu_{\beta}+C_{1}}{\mu_{\alpha}^{2}+\mu_{\beta}^{2}+C_{1}}\right)\left(\dfrac{2\sigma_{\alpha}\sigma_{\beta}+C_{2}}{\sigma_{\alpha}^{2}+\sigma_{\beta}^{2}+C_{2}}\right)$

where, $\mu_{\left(\cdot\right)}$ and $\sigma_{\left(\cdot\right)}$ are
computed with a Gaussian filter with standard deviation $\sigma_{G}$,
$C_{1}<1$ and $C_{2}<1$ are constants introduced to handle division by zero
issue, $\lambda_{\text{s}}$ and $\lambda_{\text{q}}$ weights for SSIM and
quantization loss, respectively. For $\mathcal{L}_{\text{quant}}$, we use the
loss function as proposed in [1] on the correlated latent space
$\textbf{Y}_{\mathbf{Z}}$ and dictionary
$\mathbf{D}=\\{d^{1},d^{2},\cdots,d^{k}\\}$, where $k=128$ is length of
dictionary to learn for quantization.

##  4 Experimentation and Results

Datasets. We used the publicly available Confocal Membrane dataset [9]
consisting of six plants. We train our model using four plant stacks, and use
one plant stack each for validation and testing.

Qualitative Results. Fig. 3 shows few examples of the reconstructed slices
from the $z-$stack using the our approach along with the input slice. It can
be observed that our proposed method is able to generate sharper cell
boundaries. Since we learn the quantization dictionary using all the slices in
various $z-$stacks, our method is able to generate cleaner images.
Deconvolution is a standard technique used by many researchers to enhance
microscopy images. We compare our proposed approach with deconvolution
operation used to denoise microscopy images using ImageJ [10]. It is performed
on 2D slices using Gaussian Point Spread Function (PSF) with standard values.
It can be seen from Fig. 5 that our proposed approach reconstructs visually
enhanced slices compared to deconvolution operation in ImageJ. A key reason
that deconvolution doesn’t work well is due to the selection of PSF which
highly depends on the capturing instrument. This demonstrates the advantage of
our approach with respect to existing algorithms. Note that in Fig. 3, Fig. 4,
and Fig. 5, input slice is shown inside  and the reconstructed slice using the
proposed approach is shown inside  H . Results are best viewed when zoomed-in.

Qualitative Ablation. To evaluate the impact of quantization in the latent
space, we perform an experiment without applying quantization keeping all
other parameters same in the proposed method. Fig. 4 qualitatively shows the
contribution on quantization in latent space. The image generated without
quantization is less sharp than with quantization. This is due to inherent
property of mean square loss to produce blurry results which dominates the
reconstruction in absence of latent representation quantization loss.

(a)

(b)

Fig. 3: Qualitative Results of Proposed Method. Original image (left) and
Reconstructed image (right) with corresponding zoomed parts are presented
here. The proposed method is able to generate sharper images from the given
blurry image slices. Fig. 4: Reconstruction Results without Quantization.
Reconstructed image without quantization (left) and Reconstructed image
(right) with quantization with corresponding zoomed parts are presented here.
This demonstrates that the quantization module in our proposed approach is
effective in deblurring the data. Fig. 5: Comparison of Reconstructed Results
with ImageJ [10].(a) Original Image, (b) Reconstructed using DQLR (ours) and
(c) Reconstructed using deconvolution by ImageJ.

##  5 Conclusion

Micro-imaging data collected for various bio-medical research suffers from
inherent blurriness and using this data for further analysis is a challenging
task. We present an approach for enhanced reconstruction of microscopic
sequential data by leveraging the information from consecutive image slices
and using quantization of their latent representation to alleviate blurriness.
Our data driven approach demonstrates visually superior results on a publicly
available benchmark. The proposed approach would be useful for bio-medical
researchers to enhance images where data is scarce and consequently, avoid
unwanted laborious efforts for re-imaging the data.

Acknowledgement. We thank Prof. B.S. Manjunath from University of California,
Santa Barbara for valuable discussions and helpful suggestions.

## References

  * [1] Aaron van den Oord, Oriol Vinyals, et al.,  “Neural discrete representation learning,”  in Advances in Neural Information Processing Systems, 2017, pp. 6306–6315.
  * [2] Min Liu, Anirban Chakraborty, et al.,  “Adaptive cell segmentation and tracking for volumetric confocal microscopy images of a developing plant meristem,”  Molecular Plant, vol. 4, no. 5, pp. 922–931, 2011.
  * [3] Jürgen Schmidhuber,  “Deep learning in neural networks: An overview,”  Neural networks, vol. 61, pp. 85–117, 2015.
  * [4] Jinshan Tang, Eli Peli, and Scott Acton,  “Image enhancement using a contrast measure in the compressed domain,”  IEEE Signal Processing Letters, vol. 10, no. 10, pp. 289–292, 2003\.
  * [5] Yan Wu, Mihaela Rosca, and Timothy Lillicrap,  “Deep compressed sensing,”  in International Conference on Machine Learning, 2019, pp. 6850–6860.
  * [6] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and Ian J. Goodfellow,  ,”  in Adversarial Autoencoders, 2015, vol. abs/1511.05644.
  * [7] Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz,  “Loss functions for image restoration with neural networks,”  IEEE Transactions on Computational Imaging, vol. 3, no. 1, pp. 47–57, 2016.
  * [8] Akash Gupta, Abhishek Aich, and Amit K Roy-Chowdhury,  “Alanet: Adaptive latent attention network for joint video deblurring and interpolation,”  in Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 256–264.
  * [9] Lisa Willis et al.,  “Cell size and growth regulation in the arabidopsis thaliana apical stem cell niche,”  Proceedings of the National Academy of Sciences, pp. E8238–E8246, 2016.
  * [10] Tony J Collins,  “ImageJ for microscopy,”  Biotechniques, vol. 43, no. S1, pp. S25–S30, 2007.


Yuetian Luo†<EMAIL_ADDRESS>and Anru R. Zhang†∗<EMAIL_ADDRESS>
†Department of Statistics, University of Wisconsin-Madison
∗Department of Biostatistics and Bioinformatics, Duke University

# Open Problem: Average-Case Hardness of Hypergraphic Planted Clique Detection

###### Abstract

We note the significance of hypergraphic planted clique (HPC) detection in the
investigation of computational hardness for a range of tensor problems. We ask
if more evidence for the computational hardness of HPC detection can be
developed. In particular, we conjecture if it is possible to establish the
equivalence of the computational hardness between HPC and PC detection.

##  1 Introduction

The analysis of tensors has emerged as an active topic in recent years.
Numerous datasets in the form of tensors emerge in various applications, such
as collaborative filtering, neuroimaging analysis, hyperspectral imaging. In
addition, tensor methods have been applied to many machine learning problems
where the observations are not necessarily tensors, such as additive index
models, high-order interaction pursuit, topic and latent variable models
(Anandkumar et al., 2014). Compared with the inference for vectors or
matrices, the tensor problems often possess distinct characteristics that
bring significant computational challenges. As pointed out by the seminal work
of Hillar and Lim (2013), extensions of many matrix concepts such as operator
norm, singular values and eigenvalues are possible but computationally NP-
hard. There often exist intrinsic gaps between the computational barriers and
statistical limits in many tensor problems, as observed in tensor completion
(Barak and Moitra, 2016), tensor PCA (Richard and Montanari, 2014; Hopkins et
al., 2015; Wein et al., 2019; Zhang and Xia, 2018; Lesieur et al., 2017;
Brennan and Bresler, 2020), high-order clustering (Luo and Zhang, 2020), etc.
The analysis of computational barriers has attracted enormous attention
because of its crucial role in the understanding of the computational
feasibility of a wide range of tensor problems.

In the past decades, several lines of works have been devoted to the analysis
of computational complexity in high-dimensional matrix problems. Such works
include the average-case reduction (Berthet and Rigollet, 2013), analysis of
restricted model of computation (e.g., statistical query Kearns (1998)),
analysis of algorithms such as sum-of-squares, belief propagation, approximate
message passing, and analysis of optimization landscape. In particular, the
average-case reduction requires randomized polynomial-time reduction from the
conjectured hard problem in distribution to the target problem up to the
conjectured computational barrier (Brennan et al., 2018). Once the average-
case reduction is established, all hardness results of the conjectured hard
problem can be automatically inherited to the target problem. This provides a
one-shot solution to the hardness of the target problem.

It would be ideal to use the same logic and establish the computational
barriers for various tensor problems rigorously by average-case reduction from
commonly raised conjectures, such as the planted clique (PC) detection or
Boolean satisfiability problem (SAT), to the target tensor problem so that all
of the hardness results of these well-studied conjectures could be naturally
inherited. However, this route is often complicated by the multiway structure
of the tensor. To overcome this, an alternative way of average-case reduction
from hypergraphic planted clique (HPC) has been proposed in Zhang and Xia
(2018); Luo and Zhang (2020). HPC has a more natural tensor structure that
enables a more straightforward average-case reduction. A better understanding
of the computational hardness relationship between PC detection and HPC
detection can make a significant contribution to our knowledge of the average-
case complexity in various tensor problems.

##  2 Planted Clique Detection and Hypergraphic Planted Clique Detection

A $d$-hypergraph can be seen as an order-$d$ extension of regular graph. In a
$d$-hypergraph $G=(V(G),E(G))$, each hyperedge $e\in E$ includes an unordered
group of $d$ different vertices in $V$. Define $\mathcal{G}_{d}(N,1/2)$ as the
Erdős-Rényi $d$-hypergraph with $N$ vertices, where each hyperedge
$(i_{1},\ldots,i_{d})$ is independently included in $E$ with probability
$1/2$. Given a $d$-hypergraph $G$, define its adjacency tensor
${\mathbfcal{A}}={\mathbfcal{A}}(G)\in(\\{0,1\\}^{N})^{\otimes d}$ as

${\mathbfcal{A}}_{[i_{1},\ldots,i_{d}]}=\begin{cases}1,&\text{ if
}(i_{1},\ldots,i_{d})\in E;\\\ 0,&\text{ otherwise}.\end{cases}$

We define $\mathcal{G}_{d}(N,1/2,\kappa)$ as the hypergraphic planted clique
(HPC) model with the clique size $\kappa$. To generate
$G\sim\mathcal{G}_{d}(N,1/2,\kappa)$, we sample a random hypergraph from
$\mathcal{G}_{d}(N,1/2)$, pick $\kappa$ vertices uniformly at random from
$[N]$, denote them as $K$, and connect all hyperedges $e$ if all vertices of
$e$ are in $K$. The hypergraphic planted clique detection (HPC) can be
formulated as the following hypothesis testing problem:

$H_{0}:G\sim\mathcal{G}_{d}\left(N,1/2\right)\quad\text{v.s.}\quad
H_{1}:G\sim\mathcal{G}_{d}\left(N,1/2,\kappa\right).$ (1)

We say a test $\phi$ solves the HPC detection if
$\limsup_{N\to\infty}\mathbb{P}_{H_{0}}\left(\phi({\mathbfcal{A}})=1\right)+\mathbb{P}_{H_{1}}\left(\phi({\mathbfcal{A}})=0\right)=0$,
i.e., the sum of type-I, II errors converges to zero.

When $d=2$, HPC detection reduces to the well-regard planted clique (PC)
detection problem. The PC detection is statistically solvable if
$\kappa\geq(2+\epsilon)\log_{2}N$ for $\epsilon>0$ by searching over all
subgraphs of size $(2+\epsilon)\log_{2}N$ (Bollobás and Erdös, 1976). However,
the best known polynomial-time algorithm requires $\kappa=\Omega(\sqrt{N})$
(Dekel et al., 2014; Deshpande and Montanari, 2015a). It has been widely
conjectured that no polynomial-time algorithm can solve the PC detection when
$\kappa=o(\sqrt{N})$. This conjecture has been strengthened in various types
of computational models (Jerrum, 1992; Hopkins, 2018; Feldman et al., 2017;
Deshpande and Montanari, 2015b; Meka et al., 2015; Barak et al., 2019;
Gamarnik and Zadik, 2019). As a result, the hardness conjecture of PC
detection has become a key assumption in obtaining computational lower bounds
for many problems, such as sparse PCA (Berthet and Rigollet, 2013; Wang et
al., 2016), submatrix detection (Ma and Wu, 2015), community detection (Hajek
et al., 2015), etc.

Compared to PC detection, HPC detection ($d\geq 3$) has received far less
attention in literature. Bollobás and Erdös (1976) proved that
$K^{d}_{N}/\left(d!\log_{2}(N)\right)^{1/(d-1)}\overset{a.s.}{\to}1$ if
$K_{N}^{d}$ is the size of the largest fully connected subgraph in
$G\sim\mathcal{G}_{d}(N,1/2)$, which implies the HPC detection problem can be
solved by exhaustive search in super-polynomial time when
$\kappa\geq\left((d!+\epsilon)\log_{2}(N)\right)^{1/(d-1)}$ for any
$\epsilon>0$. Zhang and Xia (2018) observed that applying SVD on the
matricization/unfolding of ${\mathbfcal{A}}$ solves HPC detection efficiently
if $\kappa=\Omega(\sqrt{N})$ but fails when $\kappa=N^{1/2-\epsilon}$ for any
$\epsilon>0$. Then they made a version of the following conjecture indicating
that $\kappa=\Omega(\sqrt{N})$ is essential for solving HPC detection in
polynomial time.

######  Conjecture 1 (HPC detection)

Suppose $d\geq 2$ is a fixed integer. If

$\limsup_{N\to\infty}\log\kappa/\log\sqrt{N}\leq 1-\tau\quad\text{for any
}\tau>0,$

for any sequence of polynomial-time tests $\\{\phi\\}_{N}:G\to\\{0,1\\}$,
$\liminf_{N\to\infty}\mathbb{P}_{H_{0}}\left(\phi(G)=1\right)+\mathbb{P}_{H_{1}}\left(\phi(G)=0\right)>1/2$.

Recently, Luo and Zhang (2020) provides evidence for HPC detection conjecture
by showing that _Metropolis algorithms_ (Jerrum, 1992) and _low-degree
polynomial algorithms_ (Hopkins, 2018) can not solve the problem in
polynomial-time when $\kappa=O(N^{\frac{1}{2}-\tau})$ for any $\tau>0$.

##  3 How hard is HPC detection?

We raise the following challenges on the average-case computational hardness
of HPC detection.

###### Challenge 1

Can we establish the sharp limit of $\kappa$ to guarantee that HPC can be
solved by SVD on the matricization/unfolding of ${\mathbfcal{A}}$? Is it
possible to develop another polynomial-time algorithm for HPC that works below
this limit?

Such results for PC detection can be implied by Knowles and Yin (2013) and
Deshpande and Montanari (2015a).

###### Challenge 2

Can we provide more evidence for HPC detection conjecture under more classes
of algorithms, such as statistical query (SQ), sum-of-squares (SOS), belief
propagation (BP), approximate message passing (AMP), etc, in addition to the
evidence in Luo and Zhang (2020)?

###### Challenge 3

At the threshold level of $\kappa=O(N^{1/2-\tau})$ for $\tau>0$, what is
relationship between HPC detection and PC detection in their computational
hardness?

First, HPC detection is no harder than PC detection. Given the adjacency
tensor ${\mathbfcal{A}}$ of hypergraph $G$, by fixing $(d-2)$ indices and
varying the remaining two, we get an adjacency matrix ${\mathbf{A}}$ from
${\mathbfcal{A}}$ which is a PC detection problem. So we can do a randomized
polynomial-time reduction from HPC detection to PC detection without changing
the planted clique size $\kappa$.

Now Challenge 3 becomes, how easier is HPC detection compared to PC detection?
A one-shot approach to tackle this challenge is to do randomized polynomial-
time reduction from PC to HPC. But this seems to be a very difficult task,
upon which we have made several failed attempts. If reduction from PC to HPC
could be proved, we would be able to arrive at the conclusion that PC and HPC
detection are equivalently hard and all hardness results established in PC
detection can be applied to HPC. The resulting connection between PC and HPC
would enable solutions to a huge class of tensor computational hardness
problems through existing matrix arguments. Such a connection can also provide
solid evidence for the computational hardness assumptions of generalized
planted clique problems proposed in (Brennan and Bresler, 2020), based on
which we can establish various computational lower bounds more easily.

## References

  * Anandkumar et al. (2014) Animashree Anandkumar, Rong Ge, Daniel Hsu, Sham M Kakade, and Matus Telgarsky.  Tensor decompositions for learning latent variable models.  _The Journal of Machine Learning Research_ , 15(1):2773–2832, 2014.
  * Barak and Moitra (2016) Boaz Barak and Ankur Moitra.  Noisy tensor completion via the sum-of-squares hierarchy.  In _Conference on Learning Theory_ , pages 417–445, 2016.
  * Barak et al. (2019) Boaz Barak, Samuel Hopkins, Jonathan Kelner, Pravesh K Kothari, Ankur Moitra, and Aaron Potechin.  A nearly tight sum-of-squares lower bound for the planted clique problem.  _SIAM Journal on Computing_ , 48(2):687–735, 2019\.
  * Berthet and Rigollet (2013) Quentin Berthet and Philippe Rigollet.  Complexity theoretic lower bounds for sparse principal component detection.  In _Conference on Learning Theory_ , pages 1046–1066, 2013.
  * Bollobás and Erdös (1976) Béla Bollobás and Paul Erdös.  Cliques in random graphs.  In _Mathematical Proceedings of the Cambridge Philosophical Society_ , volume 80, pages 419–427. Cambridge University Press, 1976.
  * Brennan and Bresler (2020) Matthew Brennan and Guy Bresler.  Reducibility and statistical-computational gaps from secret leakage.  _arXiv preprint arXiv:2005.08099_ , 2020.
  * Brennan et al. (2018) Matthew Brennan, Guy Bresler, and Wasim Huleihel.  Reducibility and computational lower bounds for problems with planted sparse structure.  In _Conference On Learning Theory_ , pages 48–166, 2018.
  * Dekel et al. (2014) Yael Dekel, Ori Gurel-Gurevich, and Yuval Peres.  Finding hidden cliques in linear time with high probability.  _Combinatorics, Probability and Computing_ , 23(1):29–49, 2014.
  * Deshpande and Montanari (2015a) Yash Deshpande and Andrea Montanari.  Finding hidden cliques of size $\sqrt{N/e}$ in nearly linear time.  _Foundations of Computational Mathematics_ , 15(4):1069–1128, 2015a.
  * Deshpande and Montanari (2015b) Yash Deshpande and Andrea Montanari.  Improved sum-of-squares lower bounds for hidden clique and hidden submatrix problems.  In _Conference on Learning Theory_ , pages 523–562, 2015b.
  * Feldman et al. (2017) Vitaly Feldman, Elena Grigorescu, Lev Reyzin, Santosh S Vempala, and Ying Xiao.  Statistical algorithms and a lower bound for detecting planted cliques.  _Journal of the ACM (JACM)_ , 64(2):8, 2017.
  * Gamarnik and Zadik (2019) David Gamarnik and Ilias Zadik.  The landscape of the planted clique problem: Dense subgraphs and the overlap gap property.  _arXiv preprint arXiv:1904.07174_ , 2019.
  * Hajek et al. (2015) Bruce Hajek, Yihong Wu, and Jiaming Xu.  Computational lower bounds for community detection on random graphs.  In _Conference on Learning Theory_ , pages 899–928, 2015.
  * Hillar and Lim (2013) Christopher J Hillar and Lek-Heng Lim.  Most tensor problems are np-hard.  _Journal of the ACM (JACM)_ , 60(6):45, 2013.
  * Hopkins (2018) Samuel Hopkins.  Statistical inference and the sum of squares method.  _Cornell thesis_ , 2018.
  * Hopkins et al. (2015) Samuel B Hopkins, Jonathan Shi, and David Steurer.  Tensor principal component analysis via sum-of-square proofs.  In _Conference on Learning Theory_ , pages 956–1006, 2015.
  * Jerrum (1992) Mark Jerrum.  Large cliques elude the metropolis process.  _Random Structures & Algorithms_, 3(4):347–359, 1992.
  * Kearns (1998) Michael Kearns.  Efficient noise-tolerant learning from statistical queries.  _Journal of the ACM (JACM)_ , 45(6):983–1006, 1998.
  * Knowles and Yin (2013) Antti Knowles and Jun Yin.  The isotropic semicircle law and deformation of wigner matrices.  _Communications on Pure and Applied Mathematics_ , 66(11):1663–1749, 2013.
  * Lesieur et al. (2017) Thibault Lesieur, Léo Miolane, Marc Lelarge, Florent Krzakala, and Lenka Zdeborová.  Statistical and computational phase transitions in spiked tensor estimation.  In _2017 IEEE International Symposium on Information Theory (ISIT)_ , pages 511–515. IEEE, 2017.
  * Luo and Zhang (2020) Yuetian Luo and Anru R Zhang.  Tensor clustering with planted structures: Statistical optimality and computational limits.  _arXiv preprint arXiv:2005.10743_ , 2020.
  * Ma and Wu (2015) Zongming Ma and Yihong Wu.  Computational barriers in minimax submatrix detection.  _The Annals of Statistics_ , 43(3):1089–1116, 2015.
  * Meka et al. (2015) Raghu Meka, Aaron Potechin, and Avi Wigderson.  Sum-of-squares lower bounds for planted clique.  In _Proceedings of the forty-seventh annual ACM symposium on Theory of computing_ , pages 87–96. ACM, 2015.
  * Richard and Montanari (2014) Emile Richard and Andrea Montanari.  A statistical model for tensor PCA.  In _Advances in Neural Information Processing Systems_ , pages 2897–2905, 2014.
  * Wang et al. (2016) Tengyao Wang, Quentin Berthet, and Richard J Samworth.  Statistical and computational trade-offs in estimation of sparse principal components.  _The Annals of Statistics_ , 44(5):1896–1930, 2016.
  * Wein et al. (2019) Alexander S Wein, Ahmed El Alaoui, and Cristopher Moore.  The kikuchi hierarchy and tensor pca.  In _2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS)_ , pages 1446–1468. IEEE, 2019.
  * Zhang and Xia (2018) Anru Zhang and Dong Xia.  Tensor SVD: Statistical and computational limits.  _IEEE Transactions on Information Theory_ , 64(11):7311–7338, 2018.


11institutetext: Artificial Intelligence and Learning Systems Laboratory
School of Electrical and Computer Engineering
National Technical University of Athens
11email: {iliasmits<EMAIL_ADDRESS>
# U-Sketch: An Efficient Approach for Sketch to Image Diffusion Models

Ilias Mitsouras  Eleftherios Tsonis  Paraskevi Tzouveli  Athanasios Voulodimos

###### Abstract

Diffusion models have demonstrated remarkable performance in text-to-image
synthesis, producing realistic and high resolution images that faithfully
adhere to the corresponding text-prompts. Despite their great success, they
still fall behind in sketch-to-image synthesis tasks, where in addition to
text-prompts, the spatial layout of the generated images has to closely follow
the outlines of certain reference sketches. Employing an MLP latent edge
predictor to guide the spatial layout of the synthesized image by predicting
edge maps at each denoising step has been recently proposed. Despite yielding
promising results, the pixel-wise operation of the MLP does not take into
account the spatial layout as a whole, and demands numerous denoising
iterations to produce satisfactory images, leading to time inefficiency. To
this end, we introduce U-Sketch, a framework featuring a U-Net type latent
edge predictor, which is capable of efficiently capturing both local and
global features, as well as spatial correlations between pixels. Moreover, we
propose the addition of a sketch simplification network that offers the user
the choice of preprocessing and simplifying input sketches for enhanced
outputs. The experimental results, corroborated by user feedback, demonstrate
that our proposed U-Net latent edge predictor leads to more realistic results,
that are better aligned with the spatial outlines of the reference sketches,
while drastically reducing the number of required denoising steps and,
consequently, the overall execution time.

###### Keywords:

Generative Models Diffusion Image-to-Image Translation Sketch-to-Image
Synthesis

##  1 Introduction

Freehand sketches serve as means of expressing the creativity and imagination
of individuals, providing an abstract and relatively simple way of capturing
and depicting different aspects of the highly detailed modern world. In this
context, the task of transforming them into realistic images, is gaining
increasing importance and offers individuals a way to see their imagination
come alive. Despite its significance, the task of sketch-to-image synthesis
poses a great challenge, as it tries to bridge the gap between the abstract
nature of sketches and the rich details of real-world images. This connection
requires methods capable of understanding and extrapolating features from the
sparse information provided by sketches.

In this context, various GAN-based methods have been proposed to tackle the
task of sketch-to-image synthesis [8, 9, 17, 33, 35, 34, 4]. Despite their
promising results, these methods are limited by the absence of textual
guidance, which leads to lack of versatility in the synthesized images. These
limitations, in combination with the large amount of paired data that most of
them require for training, necessitate the exploration of different methods to
tackle the task.

In recent years, diffusion models [6, 10, 11, 26] have garnered attention for
their ability to generate high-fidelity images based on corresponding textual
descriptions, exhibiting a clear superiority over GANs in text-to-image
synthesis tasks [20]. Consequently, it is only normal to examine their
performance in the task of sketch-to-image synthesis. With this notion in
mind, we propose, to the best of our knowledge, the first framework for
sketch-guided text-to-image synthesis utilizing a U-Net architecture [21] in
conjunction with diffusion models.

Our method comprises a U-Net latent edge predictor which extracts and
encapsulates spatial pixel correlations, in contrast to recent work of Voynov
_et al_. [30], which utilizes a per-pixel MLP architecture. The U-Net latent
edge predictor takes as input a vector of concatenated activations, extracted
from the intermediate layers of the denoising U-Net used in Stable Diffusion
[20] at a certain denoising step and outputs a prediction of the edges of the
synthesized image. This prediction is then used to guide the synthesis
process, so that the generated image faithfully adheres to the outline of the
input sketch. The whole guidance procedure takes place during inference time
and thus doesn’t require any further training of the pre-trained diffusion
model used as backbone.

In addition, our framework incorporates a sketch simplification network [24,
23], which refines and smooths out the edges of the input sketches. In this
way, we offer the user the option to preprocess the input before feeding it to
the diffusion model.

The experimental results, as well as feedback received by users, show that our
proposed U-Sketch:

  * i)

Produces more realistic and high-fidelity images, that faithfully follow the
spatial outline of the input sketches.

  * ii)

Reduces the total number of inference steps required to obtain satisfactory
results by $\sim$80%. This improvement leads to an equally significant
reduction in total execution time.

  * iii)

Produces images that are preferred by the vast majority of the public opinion,
in terms of realism, edge fidelity and overall structure.

##  2 Related Work

Sketch-guided image synthesis has been a topic of great interest in the fields
of computer vision and image processing. Early methods primarily relied on
traditional image processing techniques (e.g. BoW, descriptors, edge
detection) to retrieve images from reference sketches [7, 12, 2, 13, 15, 29,
31]. These methods, while showing promising results, suffer from several
shortcomings, such as limited expressiveness, difficulty in handling complex
sketches, and lack of photorealism.

With the advent of deep learning, researchers started to explore GAN-based
approaches to tackle the task of sketch-guided image synthesis. By introducing
conditional GANs, Mirza _et al_. [19] paved the way for many image-to-image
translation tasks. To this end, Isola _et al_. [14] implemented ’pix2pix’, a
conditional GAN capable of learning a mapping from the domain of the input
image to a desired output domain in a supervised manner, based on a dataset of
paired samples. The first comprehensive effort to guide image generation based
on sketches was carried out by Chen _et al_. [4], with their proposed
SketchyGAN. Their framework consists of a generator, with an encoder-decoder
architecture and a discriminator. The network is trained in an adversarial
manner, where in addition to the adversarial loss, perceptual losses are
incorporated to ensure that the generated images are visually similar to the
real images. Subsequently, many other GAN-based methods have been proposed [8,
9, 17, 33, 35, 34] with some of them focusing explicitly on generating human
faces from sketch portraits [3, 16].

Overall, GAN-based methods constituted the first effective way for addressing
the task of sketch-guided image synthesis. However, they do not provide the
ability for textual guidance. The absence of textual description in
combination with mode collapse [18] and instability issues that are present
because of the adversarial training of GANs, has prompted the exploration of
new methods, based on diffusion models. These likelihood-based models are
capable of seamlessly integrating textual guidance, mitigating the
shortcomings associated with GANs and providing more robust image synthesis
capabilities.

The utilization of diffusion models for sketch-guided text-to-image synthesis
has not yet been widely studied, with very few diffusion-based methods being
proposed. Voynov _et al_. [30] implemented an MLP latent edge predictor, which
estimates an edge map at each step of the denoising process. The edge map is
then used to guide the spatial layout of the generated image based on the
reference sketch. The edge predictor is trained in a self-supervised, domain
agnostic manner. However, their method requires multiple inference steps to
achieve satisfactory results, which still fall short compared to the
performance of our proposed U-Sketch. Wang _et al_. [32], train a diffusion
model using a hybrid objective function. This objective consists of an
identity loss term calculated between the input image and its reconstructed
version after the denoising process and a perceptual loss term defined between
the input sketch and the one extracted from the reconstructed image. One
notable limitation of their approach is the necessity for retraining the model
from scratch, which proves to be both costly and time-intensive. In contrast,
our approach leverages a pre-trained diffusion backbone directly and the whole
sketch-guidance process takes place at inference time.

##  3 Method

In this section we describe our sketch-guidance framework, U-Sketch, along
with the individual components it comprises. We start with a brief reference
to the basic principles of diffusion models and we continue with the analysis
of our proposed method.

###  3.1 Background

Diffusion Models Diffusion models are a class of generative probabilistic
models. They work by gradually adding noise to the original data during the
forward diffusion process and then attempting to methodically reverse this
process. Given a sample $\mathbf{x}_{0}\sim q(\mathbf{x}_{0})$, a variance
schedule $\beta_{t}$, the forward diffusion process can be formulated as a
Markov chain of fixed length $T$ ($t=1,\ldots,T$), as follows:

$\displaystyle q(\mathbf{x}_{1:T}|\mathbf{x}_{0})$
$\displaystyle=\prod\limits_{t=1}^{T}q(\mathbf{x}_{t}|\mathbf{x}_{t-1}),$ (1)
$\displaystyle q(\mathbf{x}_{t}|\mathbf{x}_{t-1})$
$\displaystyle=\mathcal{N}(\mathbf{x}_{t};\bm{\mu}_{t}=\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\mathbf{\Sigma}_{t}=\beta_{t}\mathbf{I}),$
(2)

Then, during the reverse diffusion process, which is also modelled as a Markov
chain, we start from a normal distribution
$p(\mathbf{x}_{T})=\mathcal{N}(\mathbf{x}_{T};\mathbf{0},\mathbf{I})$ and try
to estimate the transition kernels
$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$. This can be done by training a
denoising autoencoder ${\bm{\epsilon}}_{\theta}(\mathbf{x}_{t},t)$ which
estimates a denoised variant $\mathbf{x}_{t-1}$ of its input latent
$\mathbf{x}_{t}$. Finally using a sequence of these denoising autoencoders for
$t=T,\ldots,1$ we can sample from the desired distribution $\mathbf{x}_{0}\sim
p_{\theta}(\mathbf{x}_{0})$.

Many methods have been proposed for the implementation and acceleration of
reverse diffusion process. Song Y. _et al_. [27] model the forward process as
a SDE and then try to sample by solving the reverse-time SDE, while Song J.
_et al_. [25] use a non-Markovian formulation of the forward and reverse
processes, which allows sampling using only a subset of the total steps $T$.

###  3.2 U-Net Latent Edge Predictor

At the core of the proposed sketch-guidance framework is a U-Net latent edge
predictor $\mathbf{U}_{\mathrm{LEP}}$, that estimates the edge map of a latent
representation $\mathbf{z}_{t}$ of an image at a given step of the denoising
process. Following the example of [1], the network takes as input a vector
$\mathbf{F}$ of concatenated activations $\bm{l}_{i},i=1\ldots,n$, extracted
from the intermediate levels of the denoising U-Net ${\bm{\epsilon}}_{\theta}$
used in Stable Diffusion, along with the noise level $\mathbf{t}$ and its
positional encoding $\mathbf{p}=[\sin(2\pi\mathbf{t}\cdot
2^{-i})],i=0,\ldots,p$, as shown below:

$\mathbf{F}(\mathbf{z}_{t}|t,\mathbf{y})=\left[\bm{l}_{1}(\mathbf{z}_{t}|t,\mathbf{y}),\ldots,\bm{l}_{n}(\mathbf{z}_{t}|t,\mathbf{y}),\mathbf{t},\mathbf{p}\right].$
(3)

Since the activation maps $\bm{l}_{i}$ come from different layers of the
denoising U-Net, they may have varying dimensions. To overcome this issue,
before their concatenation we resize them, along channel axis, to match the
dimensions of the original input.

Based on vector $\mathbf{F}$, the edge predictor outputs an estimation
$\hat{\mathbf{{E}}}(\mathbf{z}_{t}|t,\mathbf{y})=\mathbf{U}_{\mathrm{LEP}}(\mathbf{F}(\mathbf{z}_{t}|t,\mathbf{y}))$
of the edge map of the latent representation $\mathbf{z}_{t}$ of the
synthesized image, at the current denoising step $t$. This estimation can be
later used for the guidance of the synthesis process, based on its similarity
with the target edge map.

The use of a U-Net type architecture in our latent edge predictor aims to take
advantage of its convolutional nature, which enables it to capture and extract
spatial correlations between the pixels of the input tensors. Taking into
account that our goal is to estimate spatial outlines, in combination with the
fact that edges are in general observed in areas with intense discontinuities
in pixel values, it is reasonable to utilize architectures that are capable of
processing the input tensors as whole and not in a per-pixel way.

The training procedure of the U-Net latent edge predictor is similar to the
one used in [30]. Our training dataset $\bm{{D}}$ is composed of triplets of
the form $(\mathbf{x},\mathbf{e},\mathbf{y})$, where $\mathbf{x}$ refers to an
input image, $\mathbf{e}$ to its edge map and $\mathbf{y}$ to the
corresponding text-prompt. Given triplets of this form, the training process
works by adding noise to the latent representation of the input image
$\bm{\mathcal{{E}}}(\mathbf{x})$ and then passing the noisy latent
$\mathbf{z}_{t}$ through the denoising U-Net ${\bm{\epsilon}}_{\theta}$. The
extracted intermediate activations are then used to construct the vector
$\mathbf{F}$ (eq. 3), which is fed to the U-Net latent edge predictor to
estimate the edge map $\hat{\mathbf{{E}}}(\mathbf{z}_{t}|t,\mathbf{y})$.
Finally, the gradient step is taken on the objective function
$\mathcal{L}(\hat{\mathbf{{E}}}(\mathbf{z}_{t}|t,\mathbf{y}),\bm{\mathcal{E}(\mathbf{e})})$.
The whole procedure is presented in detail in Algorithm 1.

Algorithm 1 U-Net Latent Edge Predictor Training

1:Datatset $\bm{{D}}$, noise_scheduler$\triangleright$ Require dataset and
noise scheduler

2:repeat

3: $(\mathbf{x},\mathbf{e},\mathbf{y})\sim\bm{{D}}$

4: Pass $\mathbf{x},\mathbf{e}$ through the variational encoder to get
$\bm{\mathcal{{E}}}(\mathbf{x})$ and $\bm{\mathcal{{E}}}(\mathbf{e})$

5: $t\sim\mathcal{U}[1,T]$ $\triangleright$ Uniform sampling of noise level

6: Add noise to get noisy latent
$\mathbf{z}_{t}=\text{noise\\_scheduler.add\\_noise}(\bm{\mathcal{{E}}}(\mathbf{x}),t)$

7: Pass $\mathbf{z}_{t}$ through the denoising U-Net
${\bm{\epsilon}}_{\theta}(\mathbf{z}_{t}|t,\mathbf{y})$

8: Extract intermediate activations to construct vector
$\mathbf{F}(\mathbf{z}_{t}|t,\mathbf{y})$ $\triangleright$ (eq. 3)

9: Estimate edge map
$\hat{\mathbf{{E}}}(\mathbf{z}_{t}|t,\mathbf{y})=\mathbf{U}_{\mathrm{LEP}}(\mathbf{F}(\mathbf{z}_{t}|t,\mathbf{y}))$

10: Take gradient step on
$\mathcal{L}(\hat{\mathbf{{E}}}(\mathbf{z}_{t}|t,\mathbf{y}),\bm{\mathcal{E}(\mathbf{e})})=\sum\limits_{i,j}\left\|\hat{\mathbf{{E}}}(\mathbf{z}_{t}|t,\mathbf{y})_{ij}-\bm{\mathcal{E}}(\mathbf{e})_{ij}\right\|^{2}\vspace*{-0.4\baselineskip}$
(4)

11:until converge

Figure 1: U-Sketch framework. Given an input sketch $\mathbf{e}$ and a text-
prompt $\mathbf{y}$, we start by producing random noise $\mathbf{z}_{t}$ and
then we iteratively pass it through the denoising autoencoder
${\bm{\epsilon}}_{\theta}(\mathbf{z}_{t}|t,\mathbf{y})$ to remove noise.
During the first $S$ of the total $T$ denoising steps we guide the synthesis
process using our U-Net latent edge predictor and sketch’s latent
representation $\bm{\mathcal{E}}(\mathbf{e})$, according to the process
described in section 3.3. Finally we return the decoded synthesized image
$\mathbf{x}_{0}=\bm{\mathcal{D}}(\mathbf{z}_{0})$. Sketch simplification
network $\mathbf{S}$ offers the user the choice of smoothing the input sketch
before feeding it in to the pipeline.

###  3.3 Image Synthesis using U-Net Latent Edge Predictor

After training our U-Net latent edge predictor, we can now use it to guide the
reverse diffusion process. As mentioned in section 3.1, reverse diffusion
consists of a sequence of denoising autoencoders
${\bm{\epsilon}}_{\theta}(\mathbf{x}_{t},t)$ for $t=T,\ldots 1$ which estimate
a denoised variant of their input $\mathbf{x}_{t}$. In our implementation, we
use Stable Diffusion as the backbone of image synthesis. This model transfers
the whole diffusion process in latent space and thus, given a noisy latent
representation $\mathbf{z}_{t}$ at step $t$ along with the corresponding text-
prompt $\mathbf{y}$, the denoising autoencoder
${\bm{\epsilon}}_{\theta}(\mathbf{z}_{t}|t,\mathbf{y})$ estimates a denoised
version $\mathbf{z}_{t-1}$ at step $t-1$.

To guide the aforementioned synthesis process towards the spatial outline of
the input sketch $\mathbf{e}$ we use a method similar to [30, 6]. More
precisely, at each denoising step $t$, given the noisy latent $\mathbf{z}_{t}$
we first estimate its denoised version $\mathbf{z}_{t-1}$. We, then, use the
U-Net latent edge predictor to estimate the corresponding edge map
$\hat{\mathbf{{E}}}(\mathbf{z}_{t}|t,\mathbf{y})$, based on the intermediate
activations extracted from the denoising U-Net ${\bm{\epsilon}}_{\theta}$.
This estimation is compared to the original edge map and a similarity between
the two is defined as follows,

$\mathrm{L}(\hat{\mathbf{{E}}}(\mathbf{z}_{t}|t,\mathbf{y}),\bm{\mathcal{E}}(\mathbf{e}))=\|\hat{\mathbf{{E}}}(\mathbf{z}_{t}|t,\mathbf{y})-\bm{\mathcal{E}}(\mathbf{e})\|^{2}.$
(5)

The similarity expresses how closely the spatial layout of the synthesized
image at step $t$ adheres to the corresponding layout of the reference sketch.
To guide the synthesis process, we calculate the gradient of the similarity
measure (eq. 5) and then use it to modify the original estimation
$\mathbf{z}_{t-1}$ in the following way,

$\mathbf{z}_{t-1}\leftarrow\mathbf{z}_{t-1}-\alpha\mathbf{\nabla}_{{\mathbf{z}}_{t}}\mathrm{L}(\hat{\mathbf{{E}}}(\mathbf{z}_{t}|t,\mathbf{y}),\bm{\mathcal{E}}(\mathbf{e})),$
(6)

where $\alpha$ is a parameter that controls the sketch-guidance strength and
defined as follows,

$\alpha=\frac{\|\mathbf{z}_{t}-\mathbf{z}_{t-1}\|_{2}}{\|\mathbf{\nabla}_{\mathbf{z}_{t}}\mathrm{L}(\hat{\mathbf{{E}}}(\mathbf{z}_{t}|t,\mathbf{y}),\bm{\mathcal{E}}(\mathbf{e}))\|_{2}}\cdot\beta.$
(7)

According to the above equation, the magnitude of the gradient is relative to
the magnitude of the difference between the original estimation and the noisy
latent of the previous step. $\beta$ functions as a weighting factor within
$\alpha$ to balance the aspects of edge fidelity and realism in the generated
images.

The impact of sketch-guidance is present during the first $S$ steps of the
reverse diffusion process, while afterwards, and for the rest $T-S$ steps we
lift the sketch’s restriction and leave the diffusion process to continue
unaffected.

Finally, our framework incorporates a sketch simplification network
$\mathbf{S}$ [23] which simplifies and smooths the edges of the sketches. This
symmetric convolutional network operates in pixel space and is trained in a
hybrid way, utilizing pairs of sketches and their simplified versions along
with a discriminator, that aims to distinguish true edges from generated ones.
This network offers the choice of preprocessing the input sketch before
feeding it to the synthesis pipeline.

The total sketch-guided synthesis process using our U-Net latent edge
predictor is described in Algorithm 2 and depicted in Figure 1.

Algorithm 2 Sketch-guided Image Synthesis

1:($\mathbf{y},\mathbf{e}$), $S$, $\beta$, noise_scheduler

2:$\mathbf{z}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$$\triangleright$
Initialize random noise

3:Get latent representation $\bm{\mathcal{E}}(\mathbf{e})$

4:for $t=T,\ldots,1$ do

5: Calculate $\mathbf{z}_{t-1}$ based on $\mathbf{z}_{t}$,
$\mathbf{z}_{t-1}=\text{noise\\_scheduler.step}(\mathbf{z}_{t},t,{\bm{\epsilon}}_{\theta}(\mathbf{z}_{t}|t,\mathbf{y}))$

6: Estimate edge map
$\hat{\mathbf{{E}}}(\mathbf{z}_{t}|t,\mathbf{y})=\mathbf{U}_{\text{LEP}}(\mathbf{F}(\mathbf{z}_{t}|t,\mathbf{y}))$

7: if $T-t\leq S$ then

8: Calculate $\alpha$$\triangleright$ (eq.7)

9:
${{\mathbf{z}}}_{t-1}\leftarrow\mathbf{z}_{t-1}-\alpha\mathbf{\nabla}_{\mathbf{z}_{t}}\mathrm{L}(\hat{\mathbf{{E}}}(\mathbf{z}_{t}|t,\mathbf{y}),\bm{\mathcal{E}}(\mathbf{e}))$

10: end if

11:end for

12:return $\mathbf{x}_{0}=\bm{\mathcal{D}}(\mathbf{z}_{0})$$\triangleright$
Pass latent $\mathbf{z}_{0}$ through variational decoder $\bm{\mathcal{D}}$

##  4 Experiments and Results

In this section we delve into the implementation details of our approach and
conduct a series of experiments to evaluate the performance of our proposed
framework. For the qualitative part of the evaluation we present various
results of synthesized images and compare them with the corresponding results
obtained by [30]. Finally, after taking into account the lack of suitable
evaluations metrics, we conduct user studies to quantitatively evaluate our
results.

###  4.1 Experimental Setup

Model Configuration: To generate high-fidelity images based on text-prompts,
we utilize Stable Diffusion v1.5. The variational encoder used in Stable
Diffusion outputs latent representations with a total of 4 channels.

Our proposed U-Net latent edge predictor adopts the following architecture:
the encoder block consists of a sequence of 4 convolutional levels, each
containing 2 convolutional layers followed by ReLU activation and a max
pooling operation, with 64, 128, 256 and 512 filters, respectively. The
bottleneck comprises 2 convolutional layers with 1024 filters each. The
decoder block mirrors the encoder’s architecture in a symmetric way. The
output block is a convolutional layer with 4 output channels. For the MLP
latent edge predictor we adopt the architecture

used in the original implementation, which consists of 4 fully-connected
layers with ReLU activations, followed by batch normalization layers, hidden
dimensions 512, 256, 128, 64, and output dimension 4.

The intermediate activations $\bm{l}_{i},i=1,\ldots,n$ are extracted from a
total of $n=9$ different intermediate levels of Stable Diffusion’s denoising
U-Net, as follows: $\bm{l}_{1},\ldots,\bm{l}_{3}$ are extracted from layers 2,
4 and 8 of the input block, $\bm{l}_{4},\ldots,\bm{l}_{6}$ from layers 0, 1, 2
of the middle block and $\bm{l}_{7},\ldots,\bm{l}_{9}$ from layers 2, 4, 8 of
the output block. For the positional encoding we set $p=9$.

Training: Our training dataset consists of images collected from ImageNet [5].
More specifically, we randomly sample 40 images from 150 different classes, to
obtain a total of 6000 images. To extract the corresponding edge maps we
utilize PiDiNet [28], a convolutional neural network that estimates edges
based on pixel difference. The edge maps extracted from PiDiNet are then
binarized using a threshold value of 0.5 and the resulting binarized images
form the desired edge maps. Finally, to get the textual description of each
image we use the name of the class to which the image is included. Both MLP
and U-Net latent edge predictors are trained for a total of 10 epochs.

Inference: As discussed in section 3.3, the inference process involves the
tuning of several hyperparameters. During our experiments we employ a DDIM
scheduler to update the samples at each denoising step, deterministically. For
all the experiments involving our proposed U-Net latent edge predictor, the
denoising process consists of a total of $T=50$ steps, while $\beta$ is set to
$1.6$ for a total of $S=0.5\hskip 1.0ptT=25$ steps. Following thorough
experimentation, we deduce that values of $\beta$ in range $[1.5,1.8]$, and
values of $S$ in range $[0.45\hskip 1.0ptT,0.55\hskip 1.0ptT]$ lead to
satisfactory results, with lower values of $S$ being better suited to sketches
of higher spatial complexity. Finally, we set guidance scale equal to 8 for
the classifier-free guidance, to increase the adherence to the text-prompt.
All the sketches used during inference are part of the Sketchy Dataset [22].

###  4.2 Qualitative Evaluation

For our first experiment we conduct a comparative qualitative evaluation
between the results obtained by the MLP and our proposed U-Net latent edge
predictor. In Figure 2 we present a series of sketch-guided text-to-image
synthesis examples. Each triplet of images is structured from left to right:
the input sketch with the corresponding text-prompt, the image produced by the
MLP and the image produced by our U-Net latent edge predictor, using the same
seed number for both.

Upon scrutinizing the images shown in Figure 2, we discern that our proposed
framework outperforms the MLP, in terms of both realism and edge fidelity.
Specifically, our framework generates images that exhibit a higher degree of
realism, with the depicted objects resembling more closely their real-world
counterparts. Additionally, the background in which these objects are
situated, demonstrates wide versatility, blending into real-world sceneries.
At the same time, our results are, for the most part, better aligned with the
spatial layout of the reference sketches and encapsulate the character of the
textual description.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure 2: Sketch-guided text-to-image synthesis examples. For each triplet we
have from left to right: the reference sketch along with the text-prompt, the
image generated using the MLP and the image generated using our proposed
U-Sketch framework.

Even in cases where the MLP manages to capture the edges of the input sketches
more effectively (e.g. Figure 2(h), 2(i)), this spatial adherence comes at the
expense of realism, with the synthesized images exhibiting a more anime-like
appearance, with a solid and relatively plain background.

This behaviour is attributed to the per-pixel nature of the MLP, which treats
each pixel individually, disregarding the correlations between them. In
contrast, our U-Net architecture can effectively capture these type of
correlations. This spatial knowledge enables the U-Net to provide better
estimations of the edge maps at earlier steps of the denoising process. As a
result, the sketch-guidance process is performed more effectively, while
exerting minimal impact on the background elements of the synthesized images.
Generally, our framework tends

to prioritize the preservation of realism over attaining absolute spatial
precision. Thereby, when the input sketches exhibit a high level of complexity
(e.g. Figure 2(c), 2(d)), our framework generates images, that despite being
slightly less accurate in terms of the spatial outline, possess a heightened
sense of realism.

At this point, in order to ensure that our comparison is thorough and
impartial, we provide additional examples, utilizing more denoising steps for
the case of the MLP latent edge predictor. For each example in Figure 3 we
have from left to right: the input sketch with the corresponding text-prompt,
the image produced by the MLP using $T=50$ and $T=250$ steps and the image
produced by our U-Net using $T=50$ steps.

Based on these results, we observe that the increase of denoising steps leads
to an overall improvement in the performance of the MLP latent edge predictor.
The newly synthesized images are better aligned with the input sketches and
the borders of their individual components are more efficiently
differentiated. Despite this overall improvement, the quality of the images is
significantly lower in comparison to those obtained using our U-Net latent
edge predictor. It is worth noting that our approach boasts an 80% reduction
in denoising steps and calls to the latent edge predictor. At the same time,
as shown in Figure 3(c), the increase of denoising steps does not guarantee
the realism of the final result. Finally, in terms of time efficiency,
increasing the number of denoising steps from 50 to 250, results to a
proportional increase in the overall execution time. For instance, the total
execution time required to produce an image using our framework with 50
denoising steps is approximately 50 seconds on a T4 NVIDIA GPU, whereas when
using the MLP latent edge predictor with 250 steps, it increases to
approximately 250 seconds.

(a)

(b)

(c)

Figure 3: Sketch-guided text-to-image synthesis examples. For each example we
have from left to right: the reference sketch along with the text-prompt, the
image generated using the MLP with $T=50$ and $T=250$ denoising steps and the
image generated using our proposed U-Sketch framework with $T=50$ denoising
steps.

###  4.3 Impact of Sketch Simplification Network

As mentioned in section 3.3, a fundamental component of our proposed U-Sketch
framework is a sketch simplification network, that allows users to simplify
and smoothen their sketches. This network is particularly useful in cases of
rough and poorly drawn sketches, where it is used to alleviate the impact of
the overlapping and indistinct lines to the sketch-guidance process. To assess
the contribution of the network to the overall performance of our framework,
in Figure 4 we present some results, obtained both with and without its
utilization.

Based on these results, we can conclude that the sketch simplification network
enhances the generated images, in various aspects. From Figure 4(a) we note
that the refinement applied to the input sketch, leads to a better capture of
the position and width of its dense parallel lines. Furthermore, in Figure
4(b) we can clearly observe that the simplification of the input sketch
reduces the impact of the intermediate abstract, disorderly lines, resulting
in better alignment with the input spatial outlines. Finally, examples 4(c)
and 4(d) demonstrate an overall improvement after simplifying the input
sketches, both in terms of realism and edge fidelity, corroborating the
aforementioned observations.

(a)

(b)

(c)

(d)

Figure 4: Sketch-guided text-to-image synthesis examples using U-Sketch. For
each triplet we have from left to right: the reference sketch along with the
text-prompt, the image generated without the use and with the use of the
sketch simplification network.

###  4.4 Impact of Noise Initialization

Noise initialization is a pivotal part of the diffusion process, serving as
the foundation for generating new images. Despite its random and unstructured
nature, noise possesses a basic intrinsic layout that is not obvious with
simple observation. By stabilizing the seed and text-prompt of the diffusion
process, we generate images where the depicted objects exhibit consistent
layout and positioning.

In this section we aim to examine how the initial random noise affects the
sketch-guidance process and the overall results. Figure 5 presents a series of
examples, where for each input sketch and text-prompt, we use 4 different
noise initializations to get the resulting images. Based on these results, it
is evident that the initial noise significantly affects the generated images,
both in terms of background scenery and edge fidelity. More precisely, in
examples 5(a), 5(c) the variation of noise initialization leads to images with
different background sceneries and color palettes, while maintaining similar
spatial layouts for the objects of interest. In contrast, in example 5(b)
apart from the versatility in background, we also observe a divergence in
spatial layout, with some initializations leading to better adhering to the
outlines of the input sketch (first and fourth generated images), while others
to a slight deviation from the reference geometry (second and third generated
images).

In conclusion, we can infer that the initial noise used in sketch-guided text-
to-image synthesis poses a factor of great importance for the quality and
fidelity of the generated images. In cases where its intrinsic layout is
better aligned with the spatial outlines of the reference sketch, the guidance
process is significantly aided and thus leads to better capturing of this
geometry. Conversely, when the initial intrinsic layout diverges substantially
from the reference spatial topology, the whole process becomes notably harder,
with the resulting images either slightly diverging or completely failing to
adhere to the input sketches.

(a)

(b)

(c)

Figure 5: Sketch-guided text-to-image synthesis examples using U-Sketch and
different noise initializations.

###  4.5 Quantitative Evaluation

Up to this point, our evaluation has primarily focused on qualitative
assessments. In this section we proceed to conduct a quantitative evaluation
to enrich the performance study of the MLP and our proposed U-Net latent edge
predictor.

A major challenge in sketch-guided text-to-image synthesis is the absence of
metrics that effectively integrate the aspects of edge fidelity, realism and
overall quality of the synthesized images. To overcome this limitation, we
conduct two user studies and utilize public opinion to quantify the
performance of the two latent edge predictors. Additionally, in order to
obtain a rudimentary quantification of the spatial aspect of the synthesized
images, we employ recall metric.

Recall: In our study, recall is employed to compare the input sketch with the
edge map extracted from the corresponding synthesized image. To extract this
edge map we once again utilize PiDiNet, as it has the capability of
disregarding some edges that are not of interest (e.g. complex edges that are
situated in the background). Considering that our goal is to focus our
comparison on the areas containing the objects of interest and taking into
account that the synthesized images exhibit a high level of complexity in
terms of the depicted sceneries, using edge detectors like Canny or Sobel
filter, yields edge maps with proportionally high complexity. In our
experiments using both the MLP and U-Net latent edge predictor, this
redundancy in detected edges results in notably low values of the recall
metric due to the significant amount of false positive pixels.

Prior to comparing the extracted edge maps to the input sketches, it is
beneficial to perform an additional preprocessing step. The lines composing
the input sketches are handwritten and may not have the desired width for
direct comparison to the edge maps extracted from the generated images. To
alleviate this issue, we refine the input sketches by using an erosion
operation with a structuring element $B$, which, in our case, corresponds to a
$3\times 3$ all-ones matrix. The eroded sketch is then used as a reference to
calculate the recall metric. Finally, based on the contours of the depicted
objects in the input sketch, a bounding box is extracted and used to crop the
eroded sketch and the edge map of the generated image. This helps to limit the
comparison to the area of interest.

The recall score was computed using 1400 generated images, with U-Sketch
showing superior performance compared to the MLP baseline, as evidenced by the
results presented in Table 1.

User Studies: In order to evaluate the effectiveness of our proposed U-Sketch
in the sketch-to-image task, we conducted two comprehensive user studies.
These studies were structured to compare the performance of our proposed
framework, comprising the U-Net latent edge predictor, against the MLP
baseline. The user studies were conducted anonymously, with participants
unaware of the methodologies underlying the generation of the presented
samples.

In our User Preference Evaluation Study, a cohort of 37 participants was
provided with sketches accompanied by textual prompts, alongside outputs
generated by both the U-Sketch and MLP frameworks, using the same seed number
for fair comparison. The provided examples were randomly selected and
participants were not informed about which framework produced each image.
Subsequently, participants were tasked with assessing each example, to
indicate which of the generated images was (a) more realistic, (b) had better
edge fidelity, _i.e_. adherence to the sketch input, and (c) had better
overall structural coherence relative to the provided sketch and textual
guidance.

Subsequently, 1110 comparative evaluations were collected, reflecting strong
user preference for the U-Sketch framework, as noted in Table 1. Specifically,
the samples generated by the U-Sketch framework were adjudged to be more
realistic in 60.9% of evaluations. Moreover, in terms of edge fidelity, our
proposed method prevailed over the MLP baseline in 70.5% of cases. Finally, in
terms of overall structural coherence, 70.7% of evaluations indicated the
superiority of our method. In summary, the outcomes of the User Preference
Evaluation Study substantiated the superiority of our proposed method over the
baseline.

In our User Rating Evaluation Study, a cohort of 31 participants was presented
with sketch and textual prompt pairs. Each input was accompanied by a single
output, generated either by U-Sketch or MLP framework. Once again,
participants had no prior knowledge of the framework used to generate the
presented image. Each participant was asked to rate the generated image on a
scale of 1 to 5 (bad $\rightarrow$ excellent) with regard to (a) realism, (b)
edge fidelity, and (c) overall structural coherence. Ultimately, 930
evaluations were collected, with an equal distribution between evaluations
associated with U-Sketch and MLP.

The findings from the User Rating Evaluation Study are presented in Table 1,
wherein the Mean Opinion Score (MOS) is computed by taking the mean ($\mu$)
and standard deviation ($\sigma$) of all responses. Notably, our proposed
U-Sketch outperforms the MLP baseline across all assessed criteria.
Specifically, our method exhibits an improvement of 20.4% in realism, 33.8% in
edge fidelity, and 26.1% in structural coherence relative to the baseline.
Furthermore, our approach yields reduced variance in MOS, indicative of
heightened consensus among participants regarding the quality of its outputs.

Table 1: Results of quantitative evaluation.

Framework |  Recall ($\uparrow$) |  User Preference Evaluation Study ($\uparrow$) |  User Rating Evaluation Study ($\uparrow$)
---|---|---|---
| Realism |  Edge fidelity |  Overall structural coherence |  Realism ($\pm\sigma$) |  Edge fidelity ($\pm\sigma$) |  Overall structural coherence ($\pm\sigma$)
MLPLEP |  0.595 | 39.1% | 29.5% | 29.3% | 3.29 $\pm$ 1.04 | 2.89 $\pm$ 1.02 | 3.20 $\pm$ 1
U-Sketch |  0.645 | 60.9% | 70.5% | 70.7% | 3.97 $\pm$ 1 | 3.86 $\pm$ 0.95 | 4.03 $\pm$ 0.93
|  |  |  |  |  |  |

##  5 Conclusion

In this paper we introduce U-Sketch, the first, to the best of our knowledge,
diffusion-based sketch-to-image framework, comprising a U-Net architecture for
edge map prediction, during inference time. The convolutional nature of U-Net
facilitates precise estimation of edge maps for generated images, yielding
high-quality samples that closely adhere to the reference sketches. We
conducted thorough qualitative evaluations and extensive user studies,
solidifying the efficacy and superiority of U-Sketch over existing methods.
Our framework achieves promising results with significantly fewer inference
steps, demonstrating higher efficiency while preserving image realism.

As future work, there remains a pressing need to develop better evaluation
metrics tailored specifically for the sketch-to-image task. Additionally, the
impact of initial noise on the quality and structure of generated images
warrants further investigation. Understanding and mitigating these factors
will undoubte- dly contribute to advancing the field, enhancing the
performance and reliability of sketch-to-image frameworks, such as U-Sketch.

## References

  * [1] Baranchuk, D., Rubachev, I., Voynov, A., Khrulkov, V., Babenko, A.: Label-efficient semantic segmentation with diffusion models. CoRR abs/2112.03126 (2021), https://arxiv.org/abs/2112.03126
  * [2] Cao, Y., Wang, C., Zhang, L., Zhang, L.: Edgel index for large-scale sketch-based image search. In: CVPR 2011. pp. 761–768 (2011). https://doi.org/10.1109/CVPR.2011.5995460
  * [3] Chen, S.Y., Su, W., Gao, L., Xia, S., Fu, H.: Deepfacedrawing: deep generation of face images from sketches. ACM Transactions on Graphics 39 (07 2020). https://doi.org/10.1145/3386569.3392386
  * [4] Chen, W., Hays, J.: Sketchygan: Towards diverse and realistic sketch to image synthesis. CoRR abs/1801.02753 (2018), http://arxiv.org/abs/1801.02753
  * [5] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848
  * [6] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis (2021)
  * [7] Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: Sketch-based image retrieval: Benchmark and bag-of-features descriptors. IEEE Transactions on Visualization and Computer Graphics 17(11), 1624–1636 (2011). https://doi.org/10.1109/TVCG.2010.266
  * [8] Gao, C., Liu, Q., Xu, Q., Wang, L., Liu, J., Zou, C.: Sketchycoco: Image generation from freehand scene sketches. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5173–5182 (2020). https://doi.org/10.1109/CVPR42600.2020.00522
  * [9] Ghosh, A., Zhang, R., Dokania, P.K., Wang, O., Efros, A.A., Torr, P.H.S., Shechtman, E.: Interactive sketch & fill: Multiclass sketch-to-image translation (2019)
  * [10] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models (2020)
  * [11] Ho, J., Saharia, C., Chan, W., Fleet, D.J., Norouzi, M., Salimans, T.: Cascaded diffusion models for high fidelity image generation. CoRR abs/2106.15282 (2021), https://arxiv.org/abs/2106.15282
  * [12] Hu, R., Barnard, M., Collomosse, J.: Gradient field descriptor for sketch based retrieval and localization. pp. 1025–1028 (09 2010). https://doi.org/10.1109/ICIP.2010.5649331
  * [13] Hu, R., Wang, T., Collomosse, J.: A bag-of-regions approach to sketch-based image retrieval. In: 2011 18th IEEE International Conference on Image Processing. pp. 3661–3664 (2011). https://doi.org/10.1109/ICIP.2011.6116513
  * [14] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks (2018)
  * [15] James, S., Fonseca, M.J., Collomosse, J.: Reenact: Sketch based choreographic design from archival dance footage (04 2014). https://doi.org/10.1145/2578726.2578766
  * [16] Li, Y., Chen, X., Yang, B., Chen, Z., Cheng, Z., Zha, Z.J.: Deepfacepencil: Creating face images from freehand sketches. In: Proceedings of the 28th ACM International Conference on Multimedia. MM ’20, ACM (Oct 2020). https://doi.org/10.1145/3394171.3413684, http://dx.doi.org/10.1145/3394171.3413684
  * [17] Liu, R., Yu, Q., Yu, S.: Unsupervised sketch-to-photo synthesis (2020)
  * [18] Metz, L., Poole, B., Pfau, D., Sohl-Dickstein, J.: Unrolled generative adversarial networks. CoRR abs/1611.02163 (2016), http://arxiv.org/abs/1611.02163
  * [19] Mirza, M., Osindero, S.: Conditional generative adversarial nets. CoRR abs/1411.1784 (2014), http://arxiv.org/abs/1411.1784
  * [20] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models (2022)
  * [21] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation (2015)
  * [22] Sangkloy, P., Burnell, N., Ham, C., Hays, J.: The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions on Graphics 35, 1–12 (07 2016). https://doi.org/10.1145/2897824.2925954
  * [23] Simo-Serra, E., Iizuka, S., Ishikawa, H.: Mastering sketching: Adversarial augmentation for structured prediction (2017)
  * [24] Simo-Serra, E., Iizuka, S., Sasaki, K., Ishikawa, H.: Learning to simplify: fully convolutional networks for rough sketch cleanup 35(4) (jul 2016). https://doi.org/10.1145/2897824.2925972, https://doi.org/10.1145/2897824.2925972
  * [25] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models (2022)
  * [26] Song, Y., Ermon, S.: Generative modeling by estimating gradients of the data distribution (2020)
  * [27] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations (2021)
  * [28] Su, Z., Liu, W., Yu, Z., Hu, D., Liao, Q., Tian, Q., Pietikäinen, M., Liu, L.: Pixel difference networks for efficient edge detection (2021)
  * [29] Turmukhambetov, D., Campbell, N., Goldman, D., Kautz, J.: Interactive sketch-driven image synthesis. Computer Graphics Forum 34 (08 2015). https://doi.org/10.1111/cgf.12665
  * [30] Voynov, A., Aberman, K., Cohen-Or, D.: Sketch-guided text-to-image diffusion models (2022)
  * [31] Wang, F., Kang, L., Li, Y.: Sketch-based 3d shape retrieval using convolutional neural networks. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 1875–1883 (2015), https://api.semanticscholar.org/CorpusID:14343656
  * [32] Wang, Q., Kong, D., Lin, F., Qi, Y.: Diffsketching: Sketch control image synthesis with diffusion models (05 2023)
  * [33] Wang, S., Bau, D., Zhu, J.: Sketch your own GAN. CoRR abs/2108.02774 (2021), https://arxiv.org/abs/2108.02774
  * [34] Xu, P., Huang, Y., Yuan, T., Pang, K., Song, Y., Xiang, T., Hospedales, T.M., Ma, Z., Guo, J.: Sketchmate: Deep hashing for million-scale human sketch retrieval. CoRR abs/1804.01401 (2018), http://arxiv.org/abs/1804.01401
  * [35] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks (2020)