colorlinks=true, linkcolor=blue, filecolor=blue, urlcolor=blue,
citecolor=blue, 11institutetext: School of Data Science, The Chinese
University of Hong Kong (Shenzhen)
Longxiang Avenue, Shenzhen, 518172, China
11email<EMAIL_ADDRESS>
# Learning Word Embedding with Better Distance Weighting and Window Size
Scheduling

Chaohao Yang  0009-0002-6198-2469

###### Abstract

Distributed word representation (a.k.a. word embedding) is a key focus in
natural language processing (NLP). As a highly successful word embedding
model, Word2Vec offers an efficient method for learning distributed word
representations on large datasets. However, Word2Vec lacks consideration for
distances between center and context words. We propose two novel methods,
Learnable Formulated Weights (LFW) and Epoch-based Dynamic Window Size (EDWS),
to incorporate distance information into two variants of Word2Vec, the
Continuous Bag-of-Words (CBOW) model and the Continuous Skip-gram (Skip-gram)
model. For CBOW, LFW uses a formula with learnable parameters that best
reflects the relationship of influence and distance between words to calculate
distance-related weights for average pooling, providing insights for future
NLP text modeling research. For Skip-gram, we improve its dynamic window size
strategy to introduce distance information in a more balanced way. Experiments
prove the effectiveness of LFW and EDWS in enhancing Word2Vec’s performance,
surpassing previous state-of-the-art methods.

###### Keywords:

Natural language processing Word embedding Word2Vec Learnable weights Window
size scheduling

##  1 Introduction

NLP researchers have long aimed to obtain high-quality word vector
representations. One traditional approach is one-hot encoding, where a vector
has a “1" at the index corresponding to the word and “0"s elsewhere. However,
this approach suffers from the curse of dimensionality when dealing with large
vocabulary sizes that can reach millions [2]. Additionally, one-hot encoding
fails to capture syntactic or semantic properties because all word distances
in the vector space are equal.

Distributed word representations have been developed to overcome the
limitations of one-hot encoding [8]. In this approach, words are represented
by lower-dimensional vectors, typically a few hundred dimensions, where each
element can take on various values [18]. These distributed word vectors can
capture both syntactic and semantic properties, allowing syntactic and
semantic similarities to be measured using Euclidean distance or cosine
similarity between vectors. Distributed word representations may also preserve
analogical relationships between words [12]. For example, the vector
subtraction $vector(``king")-vector(``man")+vector(``woman")$ is most likely
to result in a vector closest to $vector(``queen")$. Among all distributed
word representation models, the Word2Vec model [12] is the most successful
one, with outstanding performance in both modeling effectiveness and training
efficiency.

(a) (b)

Figure 1: Illustrations of two previous methods for improving Word2Vec with
example window size 3. The distance-related weights method in (a) combines all
context words ($w_{t+i}$) with their corresponding weights ($\lambda_{i}$)
when averaging them up to predict the center word ($w_{t}$). The dynamic
window size strategy in (b) uses dynamically selected window sizes (indicated
by the dashed box in the figure) to sample more from nearby context words,
allowing them to contribute more to the training process.

However, both variants of Word2Vec, namely CBOW and Skip-gram, discard the
distance information between center and context words when modeling text [16,
4]. Specifically, CBOW predicts the center word (also known as the current
word) based on the unweighted average of context word embeddings within a
context window around it, while Skip-gram gives the prediction for one context
word in the window each time based on the center word. All context words,
regardless of their distances to the center word, are treated equally in those
predictive tasks. There are two major problems concerning this design: 1)
Words closer to each other in distance are usually also closer in their
relationships, and their predictive powers with respect to each other are
generally larger compared to words that are farther from them, which should be
considered in Word2Vec’s predictive tasks. 2) The distance between words
reflects the order and relative positions of words in a text, which is
essential for extracting the meaning of the text. For example, the meanings of
“people can save water” and “water can save people” are greatly different
despite the fact that they contain the same words. Therefore, Word2Vec
discarding word distance information encounters difficulties in both its
predictive training process and semantic modeling ability.

Therefore, many researchers have been working on introducing distance
information into these models. For CBOW, some researchers have proposed adding
distance-related weights to the context average to make the model distance-
sensitive [16, 4]. For Skip-gram, exactly one center word and one context word
are used each time for modeling and no average is needed. Therefore, The most
preferred way among researchers is to dynamically select a random context
window size for each center word [12] and only use the context words with
distances within this dynamic window size for prediction. Since context words
closer to the center word are more likely to be in the window, the probability
of each context word being used will decrease linearly as their distance to
the center word increases, allowing context words closer to the center to
contribute more to the training process. Figure 1 provides some illustrations
for these two methods.

Despite their effectiveness, some problems still remain for these distance
information introducing methods. For distance-related weights, the major
concern is how to construct weights that can utilize a reasonable prior
relationship with distances and are adaptive to specific situations. For the
dynamic window size strategy, the problem is the frequent and irregular
changes in window size caused by random selections, which ruins the training
balance for each word and reduces modeling performance.

We propose two novel methods, Learnable Formulated Weights (LFW) and Epoch-
based Dynamic Window Size (EDWS), to solve these problems and improve the
performance of Word2Vec. LFW introduces a prior formula with a small number of
learnable parameters to calculate the distance-related weights. EDWS cancels
the random selection while preserving the dynamicity by gradually increasing
the window size by the number of training epochs. The effectiveness of both
our methods has been demonstrated by various experiments. Specifically, we
accomplish an accuracy improvement of 15.3% with LFW on CBOW and 2.5% with
EDWS on Skip-gram.

##  2 Related Work

The distributed representation of words has a long history since [8] and [7].
In its subsequent development, it gradually accepted the distributional
hypothesis presented in distributional representation papers such as Latent
Semantic Indexing [3] and Latent Dirichlet Allocation [5] and derived a series
of methods that use context words to obtain the distributed representation of
center words. A famous neural network language model (NNLM) architecture for
generating word representation of this kind is proposed in [2], which uses a
window of context words to predict the word right after them. In NNLM, context
words are first embedded into distributed vector representations, then
sequentially concatenated together as input to the following hyperbolic
tangent hidden layer, and finally transformed into probability distributions
in the vocabulary space at the output layer as the predicted result. In this
process, the order of the sequentially concatenated word vectors reflects the
order of the context words, which takes the distance information into account.

Thanks to its outstanding modeling ability of contextual relationships, NNLM
has been applied in fields like speech recognition and machine translation [6,
20] and in research of developing new NLP training methods [21]. However, NNLM
runs too slowly and is difficult to train on large datasets. [19] and [14]
point out this problem, and many attempts to solve it have been made since
then. Among all attempts, [12] creatively uses part of the NNLM architecture
(mainly removing the hidden layer of NNLM) to train word vectors, which
effectively improves the training efficiency and makes the new model called
Word2Vec suitable for large datasets. [13] also proposes hierarchical softmax,
negative sampling, and subsampling methods to further enhance Word2Vec’s
performance. Since then, Word2Vec has become one of the most popular
lightweight techniques for learning word embeddings [9] and is still widely
used in recent applications like fake news detection, sentiment analysis, and
depression expression classification [11, 1, 17, 15].

However, one major issue concerning Word2Vec is that both of its architecture
variants, namely CBOW and Skip-gram, discard the concatenation process of the
NNLM word embedding layer and its distance-modeling ability. [12] proposes the
dynamic window size strategy to solve this problem, giving each context word a
distance-related sample probability by randomly changing the window size, but
at the cost of ruining the training balance for each word. Other studies like
[16] and [4] attempt to make CBOW distance-sensitive by introducing distance-
related weights. The former provides an independent learnable weight to each
context word position, while the latter uses fixed weights calculated by a
prescribed distance-related formula. However, These methods either do not
model the prior relationships between distance and weights or lose the
adaptability of weights. Another approach presented in [10] introduces
different projection layers for different word positions to capture their
distance information, which suffers from a much more complex architecture and
higher training costs. In summary, further research is still needed to explore
more effective methods to incorporate distance information into the Word2Vec
model.

##  3 Proposed Methods

###  3.1 Learnable Formulated Weights

In the CBOW model, on a sliding window of the text segment
$(w_{t-r},...,w_{t-2},w_{t-1},\\\ w_{t},w_{t+1},w_{t+2},...,w_{t+r})$ with
window size $r$, we predict the center word $w_{t}$’s occurrence probability
using the context words $C_{t}=(w_{t-r},...,w_{t-1},w_{t+1},...,w_{t+r})$ as

$P(w_{t}|C_{t})=\exp(u_{w_{t}}\cdot u_{C_{t}})/\sum_{w^{\prime}\in
V}exp(u_{w^{\prime}}\cdot u_{C_{t}})$ (1)

where $u_{w_{t}}=Ex_{w_{t}}$, $u_{C_{t}}=(\sum_{|i|\leq r,i\neq
0}u_{t+i})/|C_{t}|$, and $V$ is the vocabulary set.

Here, $E$ is the neural network weight matrix of size $d\times|V|$, and
$x_{t}$ is the one hot representation of word $w_{t}$ with element 1 at
position $t$ and zero on all other positions. So, $u_{w_{t}}=Ex_{w_{t}}$ picks
the $t$-th column of the $E$ matrix, and this column therefore can be viewed
as a distributed representation of the word $w_{t}$, i.e., the $d$-dimensional
embedding of $w_{t}$.

The above model is not distance-sensitive, since the distance between a
context word and the target (center) word is absent in Eq.1. However, Our
experience says that $w_{t+1}$’s relation with $w_{t}$ is more significant
than $w_{t+2}$’s relation with $w_{t}$. Consequently, $w_{t+1}$’s predictive
power with respect to $w_{t}$ is generally larger than $w_{t+2}$’s predictive
power w.r.t $w_{t}$.

For this reason, it is reasonable to add a distance-related weight
$\lambda_{i}$ for each context word $u_{t+i}$ to reflect their influence on
predicting $w_{t}$ when averaging them up for $u_{C_{t}}$. Specifically

$u_{C_{t}}=\frac{1}{Z}\sum_{|i|\leq r,i\neq 0}\lambda_{i}u_{t+i}$ (2)

where $Z=\sum_{|i|\leq r,i\neq 0}\lambda_{i}$ is the normalization factor.

The main question to be answered for this strategy is how the distance-related
weights are constructed. We propose the Learnable Formulated Weights (LFW)
method to address this issue, which constructs a formula with a small number
of learnable parameters that takes distance as input to calculate weights at
different distances. This method combines the prior form of the formula with
the posterior learning results of the parameters, allowing us to directly
model the relationship between weights and distances and perform dynamic
adjustments on specific values of parameters.

There are two major aspects of the prior relationship between weights and
distances we will model in this paper, each corresponding to two assumptions.
The first aspect is whether or not the two context words on both sides of
$w_{t}$ with the same distance share the same weight, and the second aspect is
which kind of mathematical relationship (exponential or power) better models
the relationship between weights and distances on a single side. By combining
assumption choices for each of the aspects, we will be able to model the prior
relationship in four different ways and find out the best combination for
CBOW.

Now we will give the formula for each of the combinations of assumptions
mentioned above. Note that for each formula there is a learnable constant
added to the end to model the basic amount of relation between any two words
in the same text to keep them integrated together. The first combination of
assumptions is the same weights on both sides and the negative power law decay
of weights with respect to distance, and the formula representing this is

$\lambda_{i}=|i|^{-\alpha}+\beta$ (3)

where $0<|i|\leq r$ represents the distance between a context word and
$w_{t}$, and $\alpha$ and $\beta$ are two learnable parameters initialized to
0 before training.

The second combination of assumptions is different weights on both sides and
the negative power law decay of weights with respect to distance, so the
formula for this should be

$\lambda_{i}=\begin{cases}\quad|i|^{-\alpha_{0}}+\beta_{0}\quad\text{if
}-r\leq i<0,\\\ \quad|i|^{-\alpha_{1}}+\beta_{1}\quad\text{if }0<i\leq
r\end{cases}$ (4)

where $\alpha_{0}$, $\beta_{0}$, $\alpha_{1}$, and $\beta_{1}$ are four
learnable parameters initialized to 0 before training.

Based on the content above, the formulas corresponding to the third and fourth
assumption combinations can be given accordingly as Eq.5 and Eq.6, except that
the weights decrease exponentially with respect to distance:

$\lambda_{i}=e^{-\alpha|i|}+\beta$ (5) $\lambda_{i}=\begin{cases}\quad
e^{-\alpha_{0}|i|}+\beta_{0}\quad\text{if }-r\leq i<0,\\\ \quad
e^{-\alpha_{1}|i|}+\beta_{1}\quad\text{if }0<i\leq r\end{cases}$ (6)

In the experimental part, we will use the above four formulas for weight
calculation respectively to demonstrate which of those formulas is the most
effective, and at the same time prove the superiority of our method over other
approaches to adding weights to CBOW.

###  3.2 Epoch-based Dynamic Window Size

For the Skip-gram model, each center word $w_{t}$ is used as an input to
predict context words in a sliding window of size $r$. That is, for each
context word $w_{t+i}$ with $0<|i|\leq r$, we predict its occurrence
probability as

$P(w_{t+i}|w_{t})=exp(u_{w_{t+i}}\cdot u_{w_{t}})/\sum_{w^{\prime}\in
V}exp(u_{w^{\prime}}\cdot u_{w_{t}})$ (7)

Because no average of context words appears in this model, we cannot make the
model distance-sensitive by adding weights for context words. However, since
the context words that appear more in the prediction have more influence on
how the word embeddings are trained, we can make the model sensitive to
distance by sampling more from context words that are closer to the center
word and sampling less otherwise. In other words, the probability of being
sampled will act in a similar way as the weights added to the CBOW model.

A naive way to do this is to dynamically select a random context window size
$r^{\prime}$ from 1 to $r$ for each center word $w_{t}$, and use $w_{t}$ to
predict only context words within distance $r^{\prime}$. In this case, the
probability of each $w_{t+i}$ with $0<|i|\leq r$ to be used for prediction is

$P(w_{t+i})=P(|i|\leq r^{\prime})=\frac{r-|i|+1}{r}$ (8)

We can learn from Eq.8 that the probability of each $w_{t+i}$ to be used for
prediction decreases linearly as the distance $|i|$ increases. Therefore, the
context words closer to $w_{t}$ will obtain larger weights than those farther
away.

One major problem concerning this dynamic window size strategy is the frequent
and irregular changes in window size for each center word resulting from
random selections, which ruins the training balance for each word and reduces
modeling performance. For example, imagine that the window sizes selected for
two center words $w_{t_{1}}$ and $w_{t_{2}}$ are 1 and r respectively, then
the number of predictions made by $w_{t_{2}}$ is $r$ times larger than that of
$w_{t_{2}}$, causing the model to pay special attention to the center words in
some parts while ignoring the training of others, which will lead to an
unreasonable allocation of calculations and a decrease in model performance.

We propose the Epoch-based Dynamic Window Size (EDWS) to solve this problem,
which cancels the random selection while preserving the dynamicity by
gradually increasing the window size by the number of training epochs. In our
specific implementation, we use three different window sizes with a ratio of
$1:2:3$ to represent the beginning, middle, and end of the training process.
The smallest window size appears first, then the middle and largest one, each
running for the same number of epochs. Therefore, the window size
$r^{\prime}_{k}$ on the $k$th epoch is given by

$r^{\prime}_{k}=\lceil\frac{3k}{K}\rceil\frac{r}{3}$ (9)

where $K$ is the total number of epochs and r is the maximum window size, both
are multiples of 3 in this case, but can be other values if under a different
implementation.

The EDWS strategy has two major advantages over the original dynamic window
size. First, EDWS eliminates the irregularity brought by random selections at
its source. Second, EDWS specifies the window sizes to appear in ascending
order, forming a gradual learning process from nearby to distant contexts,
which aligns with general learning principles. Given these advantages, EDWS is
theoretically more effective than the original dynamic window size, which will
be demonstrated through experiments in later parts of this paper.

##  4 Experiments

Model | Window Size | Semantic | Syntactic | Total
---|---|---|---|---
CBOW | 5 | 960 | 2987 | 3947
10 | 1317 | 2938 | 4255
15 | 1687 | 2814 | 4501
20 | 1756 | 2867 | 4623
Skip-gram | 5 | 1995 | 2590 | 4585
10 | 2179 | 2673 | 4852
15 | 2666 | 2368 | 5034
20 | 2710 | 2429 | 5139
Table 1: Performance of CBOW and Skip-gram with different window sizes.
Because some (2392 out of 19544) analogical reasoning questions involve words
that do not appear in the vocabulary of the training dataset, the number of
correct answers is reported rather than the accuracy.

In this section, we first find the appropriate maximum window size via
multiple pre-experiments. Then, we test the effectiveness of LFW on CBOW and
EDWS on Skip-gram, each compared with its original model.

###  4.1 Datasets

We use two corpora constructed from English Wikipedia for training: enwik9 and
text8 111Datasets and text preprocessing script are from
http://mattmahoney.net/dc/textdata.html. The enwik9 dataset contains the first
$10^{9}$ bytes of English Wikipedia with about 120 million words after being
processed by Matt Mahoney’s text preprocessing script, and text8 contains the
first $10^{8}$ bytes of the preprocessed enwik9 with about 17 million words.
We discard words that appear no more than 5 times in each dataset to focus on
more valuable words while reducing computation overhead, forming two
vocabularies with sizes of 194,377 and 63,643. The enwik9 dataset is much
larger than text8, so we conduct pre-experiments on text8 to determine the
hyperparameter choice, i.e., the window size. Then, we conduct comparative
experiments on enwik9 based on the chosen hyperparameter to demonstrate the
effectiveness of our proposed methods.

As for test results, We use the analogical reasoning task dataset proposed in
[12] to measure the quality of the trained word embeddings. By evaluating the
embeddings’ analogical inference ability, this task reflects the embeddings’
mastery of language and forces a more precise distribution of the trained word
embeddings in the vector space to meet the requirements of linguistic
relations. There are 8,869 semantic and 10,675 syntactic questions in the test
set in total. The semantic questions are divided into five categories, while
the syntactic questions are divided into nine categories, as shown in Table 4.
An example of these questions is finding the word with the same relative
relationship with the word “woman” as the word “man” has with the word “king”.
The question should be answered by finding the word embedding that is closest
to $vector(``king")-vector(``man")+vector(``woman")$ in cosine distance. The
quality of word embeddings is then measured by the total accuracy of all
questions in this task.

###  4.2 Window Size Choice

We list the results of CBOW and Skip-gram with different window sizes in Table
1. Since the dynamic window size strategy is proposed as an original design of
Skip-gram, we will use this strategy for all original Skip-gram models used in
the experiments, and the window size represents its maximum window size in
random selection. We set the word vector dimension to a relatively small value
of 128 in this step to reduce the calculation cost of pre-experiments.
Consistent with the results reported in [4], we find that the improvement in
model performance is significantly related to the increase in window size
before the window size reaches 15. After that, the correlation becomes less
significant. Considering the impact of increasing the window size on the
calculation amount, we choose the window size to be 15 for formal experiments.

###  4.3 Experimental Results

Model | Semantic | Syntactic | Total
---|---|---|---
CBOW | 27.26% | 45.70% | 37.34%
LFW CBOW Eq.3 |  48.39% | 56.24% |  52.68%
LFW CBOW Eq.4 | 47.32% | 55.46% | 51.77%
LFW CBOW Eq.5 | 44.51% | 55.77% | 50.66%
LFW CBOW Eq.6 | 43.14% | 54.75% | 49.48%
Table 2: Performance of LFW Model | Semantic | Syntactic | Total
---|---|---|---
Skip-gram | 58.43% | 29.63% | 42.70%
EDWS Skip-gram | 61.60% |  31.59% |  45.21%
Table 3: Performance of EDWS

In this step, we conduct experiments to show our methods’ effectiveness for
Word2Vec on the enwik9 dataset. We set the word vector dimension to 600,
following the configuration in [16]. Table 2 and Table 3 show positive
experimental results for both proposed methods, where the fonts in bold
indicate the highest accuracy for each column. The LFW methods following Eq.3,
Eq.4, Eq.5, and Eq.6 improve the overall accuracy of CBOW by 15.3%, 14.4%,
13.3%, and 12.1%, respectively, and EDWS improves that of Skip-gram by more
than 2.5%.

Results suggest that formula Eq.3 models the relationship between weight and
distance best among all proposed formulas. This may be because: 1) There is no
significant difference in the relationship between context on two sides and
the center word and using the same parameters on both sides can reduce
overfitting. 2) The closest context words have a significantly higher
influence on the center word than the other, while the influence of the other
words decreases relatively slowly with distance, which can be modeled better
with the power function that decreases greatly from distance 1 to 2 and gently
at farther distances. To establish an intuitive impression of the relationship
between weights and distances obtained using power and exponential functions,
we provide Figure 2, which shows the curves formed by the relationship between
weights and distances learned from the enwik9 dataset when using Eq.3 (power
law decay) and Eq.5 (exponential decay). The curves in the figure take their
points at integer distances within the window size, and the sum of the weights
for each curve is normalized to 1 for ease of comparison. The behavior of
those curves solidifies our analysis of the success of Eq.3. Therefore, we
suggest using Eq.3 as the formula for LFW and future context modeling efforts.
to achieve the largest performance improvement.

Under their best version, Our methods’ performance exceeds the state-of-the-
art accuracy improvement of 13.5% on CBOW and 1.8% on Skip-gram achieved by
independent learnable weights presented in [16]. Additionally, the LFW method
consumes less than 15% of extra training time, which is a huge improvement
compared with the 72% (computed from its 42% slower speed) consumed by the
method in [16]. The test results and the improvement amounts of LFW Eq.3 (best
version) detailed to each category are shown in Table 4 to illustrate its
effectiveness better. Significant relative improvements are brought to nearly
every category by our proposed method. The experimental results prove the
effectiveness of both our methods.

Figure 2: The curves of normalized weights for power law decay and exponential decay Categories | CBOW | LFW CBOW Eq.3 |  $\Delta$ Acc (%)
---|---|---|---
Semantic | 27.26% (2418/8869) | 48.39% (4292/8869) | 21.13% (77.50%)
capital-common-countries | 80.04% (405/506) | 92.49% (468/506) | 12.45% (15.56%)
capital-world | 31.06% (1405/4524) | 62.07% (2808/4524) | 31.01% (99.86%)
currency | 3.00% (26/866) | 3.93% (34/866) | 0.92% (30.77%)
city-in-state | 15.08% (372/2467) | 28.70% (708/2467) | 13.62% (90.32%)
family | 41.50% (210/506) | 54.15% (274/506) | 12.65% (30.48%)
Syntactic | 45.70% (4879/10675) | 56.24% (6004/10675) | 10.54% (23.06%)
adjective-to-adverb | 11.49% (114/992) | 17.34% (172/992) | 5.85% (50.88%)
opposite | 17.49% (142/812) | 31.03% (252/812) | 13.55% (77.46%)
comparative | 65.62% (874/1332) | 76.73% (1022/1332) | 11.11% (16.93%)
superlative | 21.75% (244/1122) | 38.86% (436/1122) | 17.11% (78.69%)
present-participle | 44.22% (467/1056) | 45.17% (477/1056) | 0.95% (2.14%)
nationality-adjective | 75.05% (1200/1599) | 84.24% (1347/1599) | 9.19% (12.25%)
past-tense | 41.60% (649/1560) | 45.38% (708/1560) | 3.78% (9.09%)
plural | 62.31% (830/1332) | 81.68% (1088/1332) | 19.37% (31.08%)
plural-verbs | 41.26% (359/870) | 57.70% (502/870) | 16.44% (39.83%)
Total | 37.34% (7297/19544) | 52.68% (10296/19544) | 15.34% (41.10%)
Table 4: Performance of LFW Eq.3 on each category of the analogical reasoning
task. $\Delta$ indicates the absolute and relative performance improvement of
LFW Eq.3 compared to original CBOW.

##  5 Conclusion

In this paper, we propose two novel methods, LFW and EDWS, to improve the
performance of Word2Vec. For CBOW, We introduce distance-related weights
calculated by a prior formula with a small number of learnable parameters, the
best version of which can be widely adopted in NLP text modeling tasks and
research regarding how the influence and relevance between words change with
distance. For Skip-gram, we invent an epoch-based dynamic window size method
to eliminate the irregularity brought by dynamic window size at its source.
Experiments prove that both methods effectively improve the performance of
their corresponding models, surpassing state-of-the-art approaches to
incorporating distance information into Word2Vec. For future work, we will
focus on combining the two methods to improve model performance, such as
specifying the running epochs of different window sizes based on the learnable
formulated weights at different distances. Research related to this area
should be a valuable direction in the future.

## References

  * [1] Aoumeur, N.E., Li, Z., Alshari, E.M.: Improving the polarity of text through word2vec embedding for primary classical arabic sentiment analysis. Neural processing letters 55(3), 2249–2264 (2023)
  * [2] Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C.: A neural probabilistic language model. Journal of Machine Learning Research 3, 1137–1155 (2003)
  * [3] Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. Journal of machine Learning research 3(Jan), 993–1022 (2003)
  * [4] Chang, C.Y., Lee, S.J., Lai, C.C.: Weighted word2vec based on the distance of words. In: 2017 International Conference on Machine Learning and Cybernetics (ICMLC). vol. 2, pp. 563–568. IEEE (2017)
  * [5] Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. Journal of the American society for information science 41(6), 391–407 (1990)
  * [6] Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., Makhoul, J.: Fast and robust neural network joint models for statistical machine translation. In: proceedings of the 52nd annual meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1370–1380 (2014)
  * [7] Elman, J.L.: Distributed representations, simple recurrent networks, and grammatical structure. Machine learning 7, 195–225 (1991)
  * [8] Hinton, G.E., et al.: Learning distributed representations of concepts. In: Proceedings of the eighth annual conference of the cognitive science society. vol. 1, p. 12. Amherst, MA (1986)
  * [9] Karani, D.: Introduction to word embedding and word2vec. Towards Data Science 1 (2018)
  * [10] Ling, W., Dyer, C., Black, A.W., Trancoso, I.: Two/too simple adaptations of word2vec for syntax problems. In: Proceedings of the 2015 conference of the North American chapter of the association for computational linguistics: human language technologies. pp. 1299–1304 (2015)
  * [11] Mallik, A., Kumar, S.: Word2vec and lstm based deep learning technique for context-free fake news detection. Multimedia Tools and Applications 83(1), 919–940 (2024)
  * [12] Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
  * [13] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems 26 (2013)
  * [14] Morin, F., Bengio, Y.: Hierarchical probabilistic neural network language model. In: International workshop on artificial intelligence and statistics. pp. 246–252. PMLR (2005)
  * [15] Nugraha, M.R.A., Sibaroni, Y.: Classification of depression expressions on twitter using ensemble learning with word2vec. Inform: Jurnal Ilmiah Bidang Teknologi Informasi dan Komunikasi 9(1), 67–74 (2024)
  * [16] Qiu, L., Cao, Y., Nie, Z., Yu, Y., Rui, Y.: Learning word representation considering proximity and ambiguity. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 28 (2014)
  * [17] Rakshit, P., Sarkar, A.: A supervised deep learning-based sentiment analysis by the implementation of word2vec and glove embedding techniques. Multimedia Tools and Applications pp. 1–34 (2024)
  * [18] Schakel, A.M., Wilson, B.J.: Measuring word significance using distributed representations of words. arXiv preprint arXiv:1508.02297 (2015)
  * [19] Schwenk, H., Gauvain, J.L.: Training neural network language models on very large corpora. In: Proceedings of human language technology conference and conference on empirical methods in natural language processing. pp. 201–208 (2005)
  * [20] Shi, Y., Zhang, W.Q., Cai, M., Liu, J.: Efficient one-pass decoding with nnlm for speech recognition. IEEE Signal Processing Letters 21(4), 377–381 (2014)
  * [21] Turian, J., Ratinov, L., Bengio, Y.: Word representations: a simple and general method for semi-supervised learning. In: Proceedings of the 48th annual meeting of the association for computational linguistics. pp. 384–394 (2010)


# Representative Feature Extraction During Diffusion Process
for Sketch Extraction with One Example

Kwan Yun∗ Youngseo Kim∗ Kwanggyoon Seo Chang Wook Seo Junyong Noh
KAIST, Visual Media Lab

###### Abstract

We introduce DiffSketch, a method for generating a variety of stylized
sketches from images. Our approach focuses on selecting representative
features from the rich semantics of deep features within a pretrained
diffusion model. This novel sketch generation method can be trained with one
manual drawing. Furthermore, efficient sketch extraction is ensured by
distilling a trained generator into a streamlined extractor. We select
denoising diffusion features through analysis and integrate these selected
features with VAE features to produce sketches. Additionally, we propose a
sampling scheme for training models using a conditional generative approach.
Through a series of comparisons, we verify that distilled DiffSketch not only
outperforms existing state-of-the-art sketch extraction methods but also
surpasses diffusion-based stylization methods in the task of extracting
sketches.

---

Figure 1: Results of DiffSketch and distilled $\text{DiffSketch}_{distilled}$,
trained with one example. The left sketches were generated by DiffSketch,
while the right sketches were extracted from images using
$\text{DiffSketch}_{distilled}$.

**footnotetext: These authors contributed equally to this work

##  1 Introduction

Sketching is performed in the initial stage of artistic creation of drawing,
serving as a foundational process for both conceptualizing and conveying
artistic intentions. It also serves as a preliminary representation that
visualizes the core structure and content of the eventual artwork. As sketches
can exhibit distinct styles despite their basic form composed of simple lines,
many studies in computer vision and graphics have attempted to train models
for automatically extracting stylized sketches [55, 25, 2, 5, 43] that differ
from abstract lines [30, 51, 54].

Majority of current sketch extraction approaches utilize image-to-image
translation techniques to produce high-quality results. These approaches
typically require a large dataset when training an image translation model
from scratch, making it hard to personalize the sketch auto-colorization [6,
17, 66, 63] or sketch-based editting [24, 67, 44, 33]. On the other hand,
recent research has explored the utilization of diffusion model [36, 40]
features for downstream tasks [60, 16, 64, 50]. Features derived from
pretrained diffusion models are known to contain rich semantics and spatial
information [50, 60], which is known to help the training with limited data
[3]. Previous studies have utilized these features extracted from a subset of
layers, certain timesteps, or every specific intervals. Unfortunately, these
hand-selected features often do not contain most of the information generated
during the entire diffusion process.

To this end, we propose Diffsketch, a new method that can extract
representative features from a pretrained diffusion model and train the sketch
extraction model with one data. For feature extraction from the denoising
process, we statistically analyze the features and select those that can
represent the whole feature information from the denoising process. Our new
generator aggregates the features from multiple timesteps, fuses them with VAE
features, and decodes these fused features.

The way we train the generator with synthetic features differs from that
employed by previous diffusion-based stylization methods in that our method is
specially designed for sketch extraction. While most diffusion-based
stylization methods adopt the original pretrained diffusion model by swapping
features [11, 50] or by inverting style into a certain prompt [10, 39], these
techniques do not provide fine control over the style of the sketch, making
them unsuitable for extracting sketches in a desired style. In contrast,
DiffSketch trains a generator model from scratch specifically for sketch
extraction of a desired style.

In addition to the newly proposed model architecture, we introduce a method
for effective sampling during training. It is easy to train a network with
data that share similar semantic information to ground truth data. However,
relying solely on such data for training will hinder the full utilization of
the capacity provided by the diffusion model. Therefore, we adopt a new
sampling method to ensure training with diverse examples while enabling
effective training. Finally, we distill our network into a streamlined image-
to-image translation network for improved inference speed and efficient memory
usage. The resulting $\text{DiffSketch}_{distilled}$ is the final network that
is capable of performing a sketch extraction task. The contributions can be
summarized as follows:

  * •

We propose DiffSketch, a novel method that utilizes features from a pretrained
diffusion model to generate sketches, learning from one manual sketch data.

  * •

Through analysis, we select the representative features during the diffusion
process and utilize the VAE features as fine detailed input to the sketch
generator.

  * •

We propose a new sampling method to train the model effectively with synthetic
data.

##  2 Related Work

###  2.1 Sketch Extraction

At its core, sketch extraction utilizes edge detection. Edge detection serves
as the foundation not only for sketch extraction but also for tasks like
object detection and segmentation [65, 1]. Initial edge detection studies
primarily focused on identifying edges based on abrupt variations in color or
brightness [4, 55]. Although these techniques are direct and efficient without
requiring extensive datasets to train on, they often produce outputs with
artifacts, like scattered dots or lines.

To make extracted sketches authentic, learning-based strategies have been
introduced. These strategies excel at identifying object borders or rendering
lines in distinct styles [57, 58, 25, 22, 21]. Chan et al. [5] took a step
forward from prior techniques by incorporating the depth and semantic
information of images to procure superior-quality sketches. In a more recent
development, Ref2sketch [2] permits to extract stylized sketches using
reference sketches through paired training. Semi-Ref2sketch [43] adopted
contrastive learning for semi-supervised training. All of these methods share
the same limitation; they require a large amount of sketch data for training,
which is hard to gather. Due to data scarcity, training a sketch extraction
model is generally challenging. To address this challenge, our method is
designed to train a sketch generator using just one manual drawing.

###  2.2 Diffusion Features for Downstream Task

Diffusion models [12, 31] have shown cutting-edge results in tasks related to
generating images conditioned on text prompt [36, 40, 35]. There have been
attempts to analyze the features for utilization in downstream tasks such as
segmentation [3, 60, 16], image editing [50], and finding dense semantic
correspondence [26, 64, 48]. Most earlier studies chose a specific subset of
features for their own downstream tasks. Recently, Luo et al. [26] proposed an
aggregator that learns features from all layers and that uses equally sampled
time steps. We advance a step further by analyzing and selecting the features
from multiple timesteps, which represent the overall features. We also propose
a two-stage aggregation network and feature-fusing decoder utilizing
additional information from VAE to generate finer details.

###  2.3 Deep Features for Sketch Extraction

Most of recent sketch extraction methods utilize the deep features of a
pretrained model for sketch extraction training [2, 43, 61, 62]. While the
approach of utilizing deep features from a pretrained classifier [14, 68] is
widely used to measure perceptual similarity, vision-language models such as
CLIP [34] are used to measure semantic similarity [5, 51]. These methods
indirectly use the features by comparing them for the loss calculation during
the training process instead of using them directly to generate a sketch.
Unlike the previous approaches, we directly use the denoising diffusion
features that contain rich information to extract sketches for the first time.

##  3 Diffusion Features

Figure 2: Analysis on sampled features. PCA is applied to DDIM sampled
features from different classes. (a) : features colored with human-labeled
classes. (b) : features colored with denoising timesteps.

During a backward diffusion process, a latent or image with noise repeatedly
invokes a UNet [37] to reduce the noise. The UNet produces several
intermediate features with different shapes. This collection of features
contains rich information about texture and semantics, which can be used to
generate an image in various domains. For instance, features from the lower to
intermediate layers of the UNet reveal global structures and semantic regions,
while features from higher layers exhibit fine and high-frequency information
[50, 26]. Furthermore, features become more fine-grained over time steps [11].
As these features have different information depending on their embedded
layers and processed timesteps, it is important to select diverse features to
fully utilize the information they provide.

###  3.1 Diffusion Features Selection

Here, we first present a method for selecting features by analysis. Our
approach involves selecting representative features from all the denoising
timesteps and building our novel sketch generator, $G_{sketch}$ to extract a
sketch from an image by learning from a single data. To perform analysis for
this purpose, we first sampled 1,000 images randomly and collected all the
features from multiple layers and timesteps during Denoising Diffusion
Implicit Model (DDIM) sampling, with a total of 50 steps [47].

We conducted Principal component analysis (PCA) on these features from
multiple classes and all timesteps to examine the distribution of features
depending on their semantics and timesteps. The PCA results are visualized in
Figure 2. For our experiments, we manually classified the sampled images and
their corresponding features into 17 classes with human perception, where each
class contains more than 5 images. As illustrated by the left graphs in Figure
2 (a), features from the same class tend to have similar characteristics,
which can be seen as an additional proof to the previous literature finding
that features contain semantic information [64, 3, 60]. There is also a smooth
trajectory across timesteps as shown in Figure 2 (b). Therefore, selecting
features from a hand-crafted interval can be more beneficial than using a
single feature, as it provides richer information, as previously suggested
[26]. Upon further examination, we can observe that features tend to start at
a similar point in their initial timesteps ($t\approx 50$) and diverge
thereafter (cyan box). In addition, during the initial steps, nearby values do
not show a large difference compared to those in the middle (black box), while
the final features exhibit distinct values even though they are on the same
trajectory (orange box).

These findings provide insights that can guide the selection of representative
features. As we aim to capture the most informative features across timesteps
instead of using all features, we first conducted a K-means cluster analysis
(K-means) [13] with Within Clusters Sum of Squares distance (WCSS) to
determine the number of representative clusters. One way to compute the
K-means cluster with WCSS distance is to use the elbow method. However, we
could not identify a clear visual elbow when 30 PCA components were used.
Therefore, we used a combination of the Silhouette Score (SS) [38] and the
Davies-Bouldin Index (DBI) [7]. For all features from each sampled image, we
chose the first $K$ that matched both $k^{\prime}$’th highest SS score and
$k^{\prime}$’th lowest DBI score.

From this process, we chose our $K$ as 13 although this $K$ value may vary
with the number of diffusion sampling processes. We select the representative
features from the center of each cluster to use them as input to our sketch
generation network. To verify that the selected features indeed offer better
representation compared to those selected from equal timesteps and random
features, we calculated the minimum Euclidean distance from each projected
feature to the selected 13 features across 1,000 images. We found that our
method led to the minimum distance (18,615.6) among the distances achieved by
using the equal timestep selection (19,004.9) and random selection (23,957.2).
More explanations are provided in the supplementary material.

###  3.2 Diffusion Features Aggregation

Inspired by feature aggregation networks for downstream tasks [60, 26], we
build our two-level aggregation network and feature fusing decoder (FFD), both
of which constitute our new sketch generator ${G}_{sketch}$. The architectures
of $G_{sketch}$ and FFD are shown in Figure 4 (b) and (d), respectively. The
diffusion features $f_{l,t}$, generated on layer $l$ and timestep $t$, are
passed through the representative feature gate $G^{*}$. They are then
upsampled to a certain resolution by $U_{md}$ and $U_{tp}$, and passed through
a bottleneck layer $B_{l}$ followed by being assigned with mixing weights $w$.
The second aggregation network receives the first fused feature $F_{fst}$ as
an additional input feature.

$\displaystyle F_{fst}$
$\displaystyle=\sum_{t=0}^{T}\sum_{l=1}^{l_{md}}w_{l,t}\cdot
B_{l}(U_{md}(G^{*}(f_{l,t}))),$ (1) $\displaystyle F_{fin}$
$\displaystyle=\sum_{t=0}^{T}\sum_{l={l_{md}+1}}^{L}w_{l,t}\cdot
B_{l}(U_{tp}(G^{*}(f_{l,t})))$
$\displaystyle+\sum_{l={l_{md}+1}}^{L}w_{l}\cdot B_{l}(U_{tp}(F_{fst}))$

Here, $L$ is the total number of UNet layers, while $l_{md}$ indicates the
middle layer, which are set to be 12 and 9, respectively. Bottleneck layer
$B_{l}$ is shared across timesteps. $T$ is the total number of timesteps.
$F_{fst}$ denotes the first level aggregated features and $F_{fin}$ denotes
the final aggregated features. These two levels of aggregation allow us to
utilize the features in a memory efficient manner by mixing the features
sequentially in a lower resolution first and then in a higher resolution.

###  3.3 VAE Decoder Features

Unlike recent applications on utilizing diffusion features, where semantic
correspondences are more important than high-frequency details, sketch
generation utilizes both semantic information and high-frequency details such
as texture. As shown in Figure 3, VAE decoder features contain high-frequency
details such as hair and wrinkles. From this observation, we designed our
network to utilize VAE features following the aggregation of UNet features.
Extended visualizations are provided in the supplementary material.

Figure 3: Visualization of features from UNet and VAE in lower and higher
resolution layers. Lower resolution layers are the first layers while higher
resolution layers are the 11th for UNet and the 9th for VAE.

We utilize all the VAE features from the residual blocks to build FFD. The
aggregated features $F_{fin}$ and VAE features are fused together to generate
the output sketch. Specifically, in the fusing step $i$, VAE features with the
same resolution are passed through the channel reduction layer followed by the
convolution layer. These processed features are concatenated to the previously
fused feature $x_{i}$ and the result is passed through the fusion layer to
output $x_{i+1}$. For the first step ($i=0$), $x_{0}$ is $F_{fin}$. All
features in the same step has same resolution. We denote the number of total
features at $i$ as $N$ without subscript for simplicity. This process is shown
in Figure 4 (d) and can be expressed as follows:

$\displaystyle x_{i+1}$
$\displaystyle=\text{FUSE}\left[\left\\{\sum_{n=1}^{N}\text{Conv}(\text{CH}(v_{i,n}))\right\\}+x_{i}\right]$
(2) $\displaystyle\hat{I}_{sketch}$
$\displaystyle=\text{OUT}\left[\left\\{\sum_{n=1}^{N}\text{Conv}(\text{CH}(v_{M,n}))\right\\}+x_{M}+{I}_{source}\right]$

where $CH$ is the channel reduction layer, Conv is the convolution layers,
FUSE is the fusion layer, OUT is the final convolution layer applied before
outputting a $\hat{I}_{sketch}$, $\sum$ and addition represent concatenation
in the channel dimension. Only at the last step ($i=M$), the source image,
$I_{source}$ is also concatenated to generate the output sketch.

##  4 DiffSketch

DiffSketch learns to generate a pair of image and sketch through the process
described below, which is also shown in Figure 4.

  1. 1.

First, the user generates an image using a prompt with Stable Diffusion (SD)
[36] and draws a corresponding sketch while its diffusion features $F$ are
kept.

  2. 2.

The diffusion features $F$, its corresponding image $I_{source}$, and drawn
sketch $I_{sketch}$ constitute a triplet data to train the sketch generator
$G_{sketch}$ with directional CLIP guidance.

  3. 3.

With trained $G_{sketch}$, paired image and sketch can be generated with a
condition. This becomes the input for the distilled network for fast sketch
extraction.

In the following subsections, we will describe the structure of sketch
generator $G_{sketch}$ (Sec. 4.1), its loss functions (Sec. 4.2), and the
distilled network (Sec. 4.4).

Figure 4: Overview of Diffsketch. The UNet features generated during the
denoising process are fed to the Aggregation networks to be fused with the VAE
features to generate a sketch corresponding to the image that Stable Diffusion
generates.

###  4.1 Sketch Generator

Our sketch generator $G_{sketch}$ is built to utilize the features from the
denoising diffusion process by performed UNet and the VAE as described in
Secs. 3.2 and 3.3.  $G_{sketch}$ takes the representative features from UNet
as input, and aggregate them and fuse them with the VAE decoder features
${v}_{i,n}$ to synthesizes the corresponding sketch $\hat{I}_{sketch}$. Unlike
other image-to-image translation-based sketch extraction methods in which the
network takes an image as input [2, 5, 43], our method accepts multiple deep
features that have different spatial resolutions and channels.

###  4.2 Objectives

To train $G_{sketch}$, we utilize the following loss functions:

$L=L_{\text{rec}}+\lambda_{\text{across}}L_{\text{across}}+\lambda_{\text{within}}L_{\text{within}}$
(3)

where $\lambda_{\text{across}}$ and $\lambda_{\text{within}}$ are the
balancing weights. $L_{\text{across}}$ and $L_{\text{within}}$ are directional
CLIP losses proposed in Mind-the-gap (MTG) [69], where $L_{\text{within}}$
preserves the direction across the domain, by enforcing the difference between
$I_{samp}$ and $I_{source}$ to be similar to that between $I_{sampsketch}$ and
$I_{sketch}$ in CLIP embedding space. Similarly, $L_{\text{across}}$ enforces
the difference between $I_{sampsketch}$ and $I_{samp}$ to be similar to that
between $I_{sketch}$ and $I_{source}$. $L_{\text{rec}}$ enforces the generated
sketch from one known feature $F$ and the ground truth sketch $I_{sketch}$ to
be similar. While MTG uses an MSE loss for the pixel-wise reconstruction, we
use an L1 distance to avoid blurry sketch results, which is important in the
generation of stylized sketches. Our $L_{\text{rec}}$ can be expressed as
follows:

$L_{\text{rec}}=\lambda_{\text{L1}}L_{\text{L1}}+\lambda_{\text{LPIPS}}L_{\text{LPIPS}}+\lambda_{\text{CLIPsim}}L_{\text{CLIPsim}}$
(4)

where $\lambda_{\text{L1}}$, $\lambda_{\text{LPIPS}}$, and
$\lambda_{\text{CLIPsim}}$ are the balancing weights. $L_{\text{CLIPsim}}$
calculates the semantic similarity in the cosine distance, $L_{\text{LPIPS}}$
[68] captures the perceptual similarity, and $L_{\text{L1}}$ calculates the
pixel-wise reconstruction. More details can be found in Sec. 5.1.

###  4.3 Sampling Scheme for Training

Our method uses one source image and its corresponding sketch as the only
ground truth when guiding the sketch style, using the direction of CLIP
embeddings. Therefore, our losses rely on well-constructed CLIP manifold. When
the domains of two images $I_{source}$ and $I_{samp}$ differ largely, the
confidence in the directional CLIP loss becomes low in general (experiment
details are provided in the supplementary material). To fully utilize the
capacity of the diffusion model and produce sketches in diverse domains,
however, it is important to train the model on diverse examples.

To ensure learning from diverse examples without decreasing the CLIP loss
confidence, we propose a novel sampling scheme, condition diffusion sampling
for training (CDST). We envision that this sampling can be useful when
training a model with a conditional generator. This method initially samples a
data $I_{samp}$ from one known condition $C$ and gradually changes the
sampling distribution to random by using a diffusion algorithm when training
the network. The condition on the iteration $iter$ ($0\leq iter\leq S$) can be
described as follows:

$\displaystyle\alpha_{iter}$
$\displaystyle=\sqrt{(1-\frac{iter}{S})},\beta_{iter}=\sqrt{\frac{iter}{S}},$
(5) $\displaystyle C_{iter}$
$\displaystyle=\frac{\alpha_{iter}}{\alpha_{iter}+\beta_{iter}}C+\frac{\beta_{iter}}{\alpha_{iter}+\beta_{iter}}D_{SD},$

where $D_{SD}$ represents the distribution of the pretrained SD, while $S$
indicates the number of total diffusion duration during training.

###  4.4 Distillation

Once the sketch generator $G_{sketch}$ is trained, DiffSketch can generate
pairs of images and sketches in the trained style. This generation can be
performed either randomly or with a specific condition. Due to the nature of
the denoising diffusion model, however, in which the result is refined through
the denoising process, long processing time and high memory usage are
required. Moreover, when extracting sketches from images, the quality can be
degraded because of the inversion process. Therefore, to perform image-to-
sketch extraction efficiently while ensuring high-quality results, we train
$\text{DiffSketch}_{distilled}$ using Pix2PixHD [52].

To train $\text{DiffSketch}_{distilled}$, we extract 30k pairs of image and
sketch samples using our trained DiffSketch, adhering to CDST. Additionally,
we employ regularization to ensure that the ground truth sketch $I_{sketch}$
can be generated and discriminated effectively during the training of
$\text{DiffSketch}_{distilled}$. With this trained model, images can be
extracted in a given style much more quickly than with the original
DiffSketch.

Table 1: Quantitative results on ablation with LPIPS and SSIM. Best scores are denoted in bold. Sketch Styles | anime-informative | HED | XDoG | Average
---|---|---|---|---
Methods | LPIPS↓ | SSIM↑ | LPIPS↓ | SSIM↑ | LPIPS↓ | SSIM↑ | LPIPS↓ | SSIM↑
Ours | 0.2054 | 0.6835 | 0.2117 | 0.5420 | 0.1137 | 0.6924 | 0.1769 | 0.6393
Non-representative features 1 | 0.2154 | 0.6718 | 0.2383 | 0.5137 | 0.1221 | 0.6777 | 0.1919 | 0.6211
Non-representative features 2 | 0.2042 | 0.6869 | 0.2260 | 0.5281 | 0.1194 | 0.6783 | 0.1832 | 0.6311
One timestep features (t=0) | 0.2135 | 0.6791 | 0.2251 | 0.5347 | 0.1146 | 0.6962 | 0.1844 | 0.6367
W/O CDST | 0.2000 | 0.6880 | 0.2156 | 0.5341 | 0.1250 | 0.6691 | 0.1802 | 0.6304
W/O L1 | 0.2993 | 0.3982 | 0.2223 | 0.5011 | 0.1203 | 0.6547 | 0.2140 | 0.5180
FFD W/O VAE features | 0.2650 | 0.5044 | 0.2650 | 0.4061 | 0.2510 | 0.3795 | 0.2603 | 0.4300

##  5 Experiments

###  5.1 Implementation Details

We implemented DiffSketch and trained generator $G_{sketch}$ on an Nvidia
V-100 GPU for 1,200 iterations. When training DiffSketch, we applied CDST with
$S$ in Eq. 5 to be 1,000. The model was trained with a fixed learning rate of
1e-4. The balancing weights $\lambda_{\text{across}}$,
$\lambda_{\text{within}}$, $\lambda_{\text{L1}}$, $\lambda_{\text{LPIPS}}$,
and $\lambda_{\text{CLIPsim}}$ are fixed at 1, 1, 30, 15, and 30,
respectively. $\text{DiffSketch}_{distilled}$ was trained on two A6000 GPUs
using the same architecture and parameters from its original paper except for
the output channel, where ours was set to one. We also added regularization on
every 16 iterations. $\text{DiffSketch}_{distilled}$ was trained with 30,000
pairs that were sampled from DiffSketch with CDST ($S=30,000$).

LPIPS [68] and SSIM [53] were used for evaluation metrics, in both ablation
study and comparison with baselines. LPIPS was to calculate perceptual
similarity with pre-trained classifier. SSIM was calculated for structural
similarity of sketch image.

###  5.2 Datasets

For training, DiffSketch requires a sketch corresponding to an image generated
from SD. To facilitate a numerical comparison, we established the ground truth
for given images. Specifically, three distinct styles were employed for
quantitative evaluation: 1) HED [59] utilizes nested edge detection and is one
of the most widely used edge detection methods. 2) XDoG [56] takes an
algorithmic approach of using a difference of Gaussians to extract sketches.
3) Informative-anime [5] employs informative learning. This method is the
state-of-the-art among single modal sketch extraction methods and is trained
on the Anime Colorization dataset [18], which consists of 14,224 sketches. For
qualitative evaluation, we added hand-drawn sketches of two more styles.

For testing, we employed the test set from BSDS500 dataset [29] and also
randomly sampled an additional 1,000 images from the test set of Common
Objects in Context (COCO) dataset [23]. As a result, our training set
consisted of 3 sketches and the test dataset consisted of 3,600 pairs (1,200
pairs for each style) of image-sketch. Two hand-drawn sketches were used only
for perceptual study because there is no ground truth to compare with.

###  5.3 Ablation Study

Figure 5: Visual examples of the ablation study. Ours generates higher quality
results with details such as face, separated with hair region, compared to the
alternatives. Figure 6: Visualization of additional ablation: Ours were
trained and sampled with CDST. In contrast, W/O CDST were trained and sampled
randomly.

We conducted an ablation study on each component of our method compared to the
baselines as shown in Table 1. Experiment were performed to verify the
contribution of each component; feature selections, CDST, losses, and FFD. To
perform the ablation study, we randomly sampled 100 images and extracted
sketches with HED, XDog, and Anime-informative and paired them with all 100
images. All seeds were fixed to generate sketches from the same sample.

The ablation study was conducted as follows. For Non-representative features,
we randomly selected the features from the denoising timesteps while keeping
the number of timesteps equal to ours (13). We performed this random selection
and analysis twice. For one timestep feature, we only used the features from
the final timestep $t=0$. To produce a result without CDST, we executed random
text prompt guidance for the diffusion sampling process during training. For
the alternative loss approach, we contrasted L1 Loss with L2 Loss for pixel-
level reconstruction, as proposed in MTG. To evaluate the effect of the FFD,
we produced sketches after removing the VAE features.

The quantitative and qualitative results of the ablation study are shown in
Table 1 and Figure 5, respectively. Ours achieved the higest average scores on
both indices. Both Non-representative features achieved overall low scores
indicating that representative feature selection helps obtain rich
information. Similarly, using one time step features achieved lower scores
than ours on average, showing the importance of including diverse features.
W/O CDST scored lower than ours on both HED and XDoG styles. W/O L1 and FFD
W/O features performed the worst due to the blurry and blocky output,
respectively. The blocky results are due to lack of fine information from VAE.

#### Condition Diffusion Sampling for Training

While we tested on randomly generated images for quantitative evaluation, our
CDST can be applied to both training DiffSkech and sampling for training
$\text{DiffSketch}_{distilled}$. Therefore, we performed an additional
ablation study on CDST, comparing Ours (trained and sampled with CDST), with
W/O CDST (trained and sampled randomly). The outline of the sketch is clearly
reproduced, following the style, when CDST is used as shown in Figure 6.

###  5.4 Comparison with Baselines

We compared our method with 5 different alternatives including state-of-the-
art sketch extraction methods [2, 43] and diffusion based methods [39, 19, 9].
Ref2sketch [2] and Semi-Ref2sketch [43] are methods specifically designed to
extract sketches in the style of a reference from a large pretrained network
on diverse sketches in a supervised (Ref2sketch) and a semi-supervised (Semi-
Ref2sketch) manner. DiffuseIT [19] is designed for image-to-image translation
by disentangling style and content. DreamBooth [39] finetunes a Stable
Diffusion model to generate personalized images, while Textural Inversion [10]
optimizes an additional text embedding to generate a personalized concept for
a style or object. For DreamBooth and Textual Inversion, DDIM inversion was
conducted to extract sketches.

Table 2 presents the result of the quantitative evaluation that used BSDS500
and COCO datasets in a one-shot setting. Overall, ours achieved the best
scores. While Semi-Ref2sketch scored higher on some of SSIM scores, the method
relies on a large sketch dataset to train while ours requires only one. Figure
7 presents visual results produced by different methods. While Semi-Ref2sketch
and Ref2sketch generated superior quality sketches to the results produced by
others, they do not faithfully follow the style of the reference sketches,
especially for dense styles. Diffusion-based methods sometimes overfit to the
style image (DiffuseIT) or change the content of the images (DreamBooth,
Textual Inversion). $\text{DiffSketch}_{distilled}$ generated superior results
compared to these baselines, effectively maintaining its styles and content.

Figure 7: Qualitative comparison with alternative sketch extraction methods.
Table 2: Quantitative comparison of different methods on BSDS500 and COCO
datasets.

| BSDS500 - anime | BSDS500 - HED | BSDS500 - XDoG | BSDS500 - average
---|---|---|---|---
Methods | LPIPS↓ | SSIM↑ | LPIPS↓ | SSIM↑ | LPIPS↓ | SSIM↑ | LPIPS↓ | SSIM↑
$\text{Ours}_{distilled}$ | 0.21746 | 0.49343 | 0.22706 | 0.59314 | 0.14280 | 0.64874 | 0.19577 | 0.57844
Ref2sketch | 0.33621 | 0.46932 | 0.41993 | 0.31448 | 0.57096 | 0.13095 | 0.44237 | 0.30492
Semi-Ref2sketch | 0.23916 | 0.50972 | 0.39675 | 0.34200 | 0.50447 | 0.30918 | 0.38013 | 0.38697
DiffuseIT | 0.48365 | 0.29789 | 0.49217 | 0.19104 | 0.57335 | 0.11030 | 0.51639 | 0.19974
DreamBooth | 0.80608 | 0.30149 | 0.74550 | 0.18523 | 0.72326 | 0.19465 | 0.75828 | 0.22712
Textual Inversion | 0.82789 | 0.26373 | 0.77098 | 0.16416 | 0.64662 | 0.21953 | 0.74850 | 0.21581

| COCO - anime | COCO - HED | COCO - XDoG | COCO - average
---|---|---|---|---
Methods | LPIPS↓ | SSIM↑ | LPIPS↓ | SSIM↑ | LPIPS↓ | SSIM↑ | LPIPS↓ | SSIM↑
$\text{Ours}_{distilled}$ | 0.17634 | 0.36021 | 0.20039 | 0.36093 | 0.14806 | 0.38319 | 0.17493 | 0.36811
Ref2sketch | 0.32142 | 0.50517 | 0.37764 | 0.37230 | 0.56012 | 0.16835 | 0.41973 | 0.34861
Semi-Ref2sketch | 0.21337 | 0.64732 | 0.32920 | 0.39487 | 0.47974 | 0.31894 | 0.34077 | 0.45371
DiffuseIT | 0.46527 | 0.36092 | 0.47905 | 0.24611 | 0.56360 | 0.14595 | 0.50264 | 0.25099
DreamBooth | 0.76399 | 0.30517 | 0.72278 | 0.22066 | 0.67909 | 0.21655 | 0.72195 | 0.24746
Textual Inversion | 0.81458 | 0.29168 | 0.78835 | 0.19952 | 0.63215 | 0.22074 | 0.74503 | 0.23731

###  5.5 Perceptual Study

We conducted a user study to evaluate different sketch extraction methods on
human perception. We recruited 45 participants to complete a survey that used
test images from two datasets, processed in five different styles, to extract
sketches. Each participant was presented with a total of 20 sets of source
image, target sketch style, and resulting sketch. Participants were asked to
choose one that best follows the given style while preserving the content of
the source image. The result should not depend on demographics distribution,
therfore we did not focus on group of people as previous sketch studies [2,
43, 5]. As shown in Table 3, our method received the highest scores when
compared with the alternative methods. Ours outperformed the diffusion-based
methods by a large margin and even received a higher preference rating than
the specialized sketch extraction method that was trained on a large sketch
dataset.

Table 3: Results from the user perceptual study given style example and the source image. The percentage indicates the selected frequency. Methods | User Score
---|---
Ours | 68.67%
Ref2sketch | 6.00%
Semi-Ref2sketch | 18.56%
DiffuseIT | 0.22%
DreamBooth | 0.00%
Textual Inversion | 0.22%

##  6 Limitation and Conclusion

We proposed DiffSketch, a novel method to train a sketch generator using
representative features and extract sketches in diverse styles. For the first
time, we conducted the task of extracting sketches from the features of a
diffusion model and demonstrated that our method outperforms previous state-
of-the-art methods in extracting sketches. The ability to extract sketches in
a diverse style, trained with one example, will have various use cases not
only for artistic purposes but also for personalizing sketch-to-image
retrieval and sketch-based image editing.

We built our generator network specialized for generating sketches by fusing
aggregated features with the features from a VAE decoder. Consequently, our
method works well with diverse sketches including dense sketches and outlines.
Because our method not directly employ a loss function to compares stroke
styles, however, it fails to generate highly abstract sketches or pointillism.
One possible research direction could involve incorporating a new sketch style
loss that does not require additional sketch data, such as penalizing based on
stroke similarity in close-ups.

Although we focused on sketch extraction, our analysis of selecting
representative features and the proposed training scheme are not limited to
the domain of sketches. Extracting representative feature holds potential to
improve applications leveraging diffusion features, including semantic
segmentation, visual correspondence, and depth estimation. We believe this
research direction promises to broaden the impact and utility of diffusion
feature-based applications.

## References

  * Arbelaez et al. [2010] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik.  Contour detection and hierarchical image segmentation.  _IEEE transactions on pattern analysis and machine intelligence_ , 33(5):898–916, 2010.
  * Ashtari et al. [2022] Amirsaman Ashtari, Chang Wook Seo, Cholmin Kang, Sihun Cha, and Junyong Noh.  Reference based sketch extraction via attention mechanism.  _ACM Transactions on Graphics (TOG)_ , 41(6):1–16, 2022.
  * Baranchuk et al. [2021] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko.  Label-efficient semantic segmentation with diffusion models.  _arXiv preprint arXiv:2112.03126_ , 2021.
  * Canny [1986] John Canny.  A computational approach to edge detection.  _IEEE Transactions on pattern analysis and machine intelligence_ , (6):679–698, 1986.
  * Chan et al. [2022] Caroline Chan, Frédo Durand, and Phillip Isola.  Learning to generate line drawings that convey geometry and semantics.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7915–7925, 2022.
  * Ci et al. [2018] Yuanzheng Ci, Xinzhu Ma, Zhihui Wang, Haojie Li, and Zhongxuan Luo.  User-guided deep anime line art colorization with conditional adversarial networks.  In _Proceedings of the 26th ACM international conference on Multimedia_ , pages 1536–1544, 2018.
  * Davies and Bouldin [1979] David L Davies and Donald W Bouldin.  A cluster separation measure.  _IEEE transactions on pattern analysis and machine intelligence_ , (2):224–227, 1979.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on_ , pages 248–255. IEEE, 2009.
  * Gal et al. [2022a] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ , 2022a.
  * Gal et al. [2022b] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ , 2022b.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Hotelling [1933] Harold Hotelling.  Analysis of a complex of statistical variables into principal components.  _Journal of educational psychology_ , 24(6):417, 1933.
  * Johnson et al. [2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14_ , pages 694–711. Springer, 2016.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _CVPR_ , 2019.
  * Khani et al. [2023] Aliasghar Khani, Saeid Asgari Taghanaki, Aditya Sanghi, Ali Mahdavi Amiri, and Ghassan Hamarneh.  Slime: Segment like me.  _arXiv preprint arXiv:2309.03179_ , 2023.
  * Kim et al. [2019] Hyunsu Kim, Ho Young Jhoo, Eunhyeok Park, and Sungjoo Yoo.  Tag2pix: Line art colorization using text tag with secat and changing loss.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9056–9065, 2019.
  * Kim [2018] Taebum Kim.  Anime sketch colorization pair.  https://www.kaggle.com/ktaebum/anime-sketch-colorization-pair, 2018.
  * Kwon and Ye [2023] Gihyun Kwon and Jong Chul Ye.  Diffusion-based image translation using disentangled style and content representation.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Levina and Bickel [2001] Elizaveta Levina and Peter Bickel.  The earth mover’s distance is the mallows distance: Some insights from statistics.  In _Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001_ , pages 251–256. IEEE, 2001.
  * Li et al. [2017] Chengze Li, Xueting Liu, and Tien-Tsin Wong.  Deep extraction of manga structural lines.  _ACM Transactions on Graphics (SIGGRAPH 2017 issue)_ , 36(4):117:1–117:12, 2017.
  * Li et al. [2019] Mengtian Li, Zhe Lin, Radomir Mech, Ersin Yumer, and Deva Ramanan.  Photo-sketching: Inferring contour drawings from images.  In _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 1403–1412. IEEE, 2019.
  * Lin et al. [2015] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár.  Microsoft coco: Common objects in context, 2015.
  * Liu et al. [2022] Feng-Lin Liu, Shu-Yu Chen, Yukun Lai, Chunpeng Li, Yue-Ren Jiang, Hongbo Fu, and Lin Gao.  Deepfacevideoediting: Sketch-based deep editing of face videos.  _ACM Transactions on Graphics_ , 41(4):167, 2022.
  * lllyasviel [2017] lllyasviel.  sketchkeras.  https://github.com/lllyasviel/sketchKeras, 2017.
  * Luo et al. [2023] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell.  Diffusion hyperfeatures: Searching through time and space for semantic correspondence.  _arXiv preprint arXiv:2305.14334_ , 2023.
  * Mardia [1970] Kanti V Mardia.  Measures of multivariate skewness and kurtosis with applications.  _Biometrika_ , 57(3):519–530, 1970.
  * Mardia [1974] Kanti V Mardia.  Applications of some measures of multivariate skewness and kurtosis in testing normality and robustness studies.  _Sankhyā: The Indian Journal of Statistics, Series B_ , pages 115–128, 1974.
  * Martin et al. [2001] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In _Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001_ , pages 416–423. IEEE, 2001.
  * Mo et al. [2021] Haoran Mo, Edgar Simo-Serra, Chengying Gao, Changqing Zou, and Ruomei Wang.  General virtual sketching framework for vector line art.  _ACM Transactions on Graphics (TOG)_ , 40(4):1–14, 2021.
  * Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pages 8162–8171. PMLR, 2021.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Portenier et al. [2018] Tiziano Portenier, Qiyang Hu, Attila Szabo, Siavash Arjomand Bigdeli, Paolo Favaro, and Matthias Zwicker.  Faceshop: Deep sketch-based face image editing.  _arXiv preprint arXiv:1804.08972_ , 2018.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Rousseeuw [1987] Peter J Rousseeuw.  Silhouettes: a graphical aid to the interpretation and validation of cluster analysis.  _Journal of computational and applied mathematics_ , 20:53–65, 1987.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs, 2021.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Seo et al. [2023] Chang Wook Seo, Amirsaman Ashtari, and Junyong Noh.  Semi-supervised reference-based sketch extraction using a contrastive learning framework.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–12, 2023.
  * Seo et al. [2022] Junyoung Seo, Gyuseong Lee, Seokju Cho, Jiyoung Lee, and Seungryong Kim.  Midms: Matching interleaved diffusion models for exemplar-based image translation.  _arXiv preprint arXiv:2209.11047_ , 2022.
  * Shapiro and Wilk [1965] Samuel Sanford Shapiro and Martin B Wilk.  An analysis of variance test for normality (complete samples).  _Biometrika_ , 52(3/4):591–611, 1965.
  * sharpei pups [2014] sharpei pups.  6.5 weeks old sharpei puppies.  https://www.youtube.com/watch?v=plIyQg6llp8, 2014.  Accessed: 23-11-2023.
  * Song et al. [2020] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020.
  * Tang et al. [2023] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan.  Emergent correspondence from image diffusion.  _arXiv preprint arXiv:2306.03881_ , 2023.
  * TheSaoPauloSeries [2013] TheSaoPauloSeries.  São paulo city mini-documentary: (full hd) the são paulo series.  https://www.youtube.com/watch?v=A3pBJTTjwCM, 2013.  Accessed: 23-11-2023.
  * Tumanyan et al. [2023] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1921–1930, 2023.
  * Vinker et al. [2022] Yael Vinker, Ehsan Pajouheshgar, Jessica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir.  Clipasso: Semantically-aware object sketching.  _ACM Transactions on Graphics (TOG)_ , 41(4):1–11, 2022.
  * Wang et al. [2018] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * Willett et al. [2023] Nora S Willett, Fernando de Goes, Kurt Fleischer, Mark Meyer, and Chris Burrows.  Stylizing ribbons: Computing surface contours with temporally coherent orientations.  _IEEE Transactions on Visualization and Computer Graphics_ , 2023.
  * Winnemöller [2011] Holger Winnemöller.  Xdog: advanced image stylization with extended difference-of-gaussians.  In _Proceedings of the ACM SIGGRAPH/eurographics symposium on non-photorealistic animation and rendering_ , pages 147–156, 2011.
  * Winnemöller et al. [2012] Holger Winnemöller, Jan Eric Kyprianidis, and Sven C Olsen.  Xdog: An extended difference-of-gaussians compendium including advanced image stylization.  _Computers & Graphics_, 36(6):740–753, 2012.
  * Xiang et al. [2021] Xiaoyu Xiang, Ding Liu, Xiao Yang, Yiheng Zhu, and Xiaohui Shen.  Anime2sketch: A sketch extractor for anime arts with deep networks.  https://github.com/Mukosame/Anime2Sketch, 2021.
  * Xie and Tu [2015a] Saining Xie and Zhuowen Tu.  Holistically-nested edge detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1395–1403, 2015a.
  * Xie and Tu [2015b] Saining Xie and Zhuowen Tu.  Holistically-nested edge detection, 2015b.
  * Xu et al. [2023] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello.  Open-vocabulary panoptic segmentation with text-to-image diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2955–2966, 2023.
  * Yi et al. [2019] Ran Yi, Yong-Jin Liu, Yu-Kun Lai, and Paul L Rosin.  Apdrawinggan: Generating artistic portrait drawings from face photos with hierarchical gans.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10743–10752, 2019.
  * Yi et al. [2020] Ran Yi, Yong-Jin Liu, Yu-Kun Lai, and Paul L Rosin.  Unpaired portrait drawing generation via asymmetric cycle mapping.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8217–8225, 2020.
  * Yuan and Simo-Serra [2021] Mingcheng Yuan and Edgar Simo-Serra.  Line art colorization with concatenated spatial attention.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3946–3950, 2021.
  * Zhang et al. [2023a] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang.  A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence.  _arXiv preprint arXiv:2305.15347_ , 2023a.
  * Zhang et al. [2015] Kaihua Zhang, Lei Zhang, Kin-Man Lam, and David Zhang.  A level set approach to image segmentation with intensity inhomogeneity.  _IEEE transactions on cybernetics_ , 46(2):546–557, 2015.
  * Zhang et al. [2018a] Lvmin Zhang, Chengze Li, Tien-Tsin Wong, Yi Ji, and Chunping Liu.  Two-stage sketch colorization.  _ACM Transactions on Graphics (TOG)_ , 37(6):1–14, 2018a.
  * Zhang et al. [2023b] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023b.
  * Zhang et al. [2018b] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018b.
  * Zhu et al. [2022] Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka.  Mind the gap: Domain gap control for single shot domain adaptation for generative adversarial networks.  In _International Conference on Learning Representations_ , 2022.

Supplementary Material

Figure 8: Visualization of WCSS values according to the number used for
K-means clustering. The left plots are the WCSS of the features from an
randomly sampled image while the right plot shows the average WCSS values of
the features from 1,000 randomly sampled images.

## Overview

This supplementary material consists of 5 Sections. Section A describes
implementation details (Sec. A). Sec. B provides additional details and
findings on diffusion features selection. Sec. C presents extended details of
VAE decoder features. Sec. D contains the results of additional experiments on
CDST. Lastly, Sec. E presents additional qualitative results with various
style sketches.

## A. Implementation Details

#### DiffSketch

DiffSketch leverages the Stable Diffusion v1.4 sampled with DDIM [47]
pretrained with the LAION-5B [42] dataset, which produced images of resolution
512 $\times$ 512\. With the pretrained Stable Diffusion, we use a total of 50
time steps T for sampling. The training of DiffSketch was performed for 1200
iterations which required less than 3 hours on an Nvidia V100 GPU. For the
training using HED [59], we concatenated the first two layers with the first
three layers to stylize sketch. In the case of XDoG [55], we used Gary Grossi
style.

#### $\text{DiffSketch}_{distilled}$

$\text{DiffSketch}_{distilled}$ was developed to conduct sketch extraction
efficiently with the streamlined generator. The training of
$\text{DiffSketch}_{distilled}$ was performed for 10 epochs for 30,000 sketch-
image pairs generated from DiffSKetch, following the CDST. The training of
$\text{DiffSketch}_{distilled}$ required approximately 5 hours on two Nvidia
A6000 GPUs. The inference time of both DiffSketch and
$\text{DiffSketch}_{distilled}$ was 4.74 seconds and 0.0139 seconds,
respectively, when tested on an Nvidia A5000 GPU with image with same
resolutions.

#### Comparison with Baselines

For the baselines, the settings used in our study were based on the official
code provided by the authors and information obtained from their respective
papers. For both Ref2Sketch [2] and Semi-ref2sketch [43], we used the given
checkpoint, the official pre-trained model provided by the authors. For
DiffuseIT [19], we also used the official code and checkpoint given by the
authors in which the diffusion model was trained with the Imagenet [8]
dataset, not FFHQ [15] because our comparison is not constrained to the face.
For Dreambooth [39] and Textual Inversion [10], we used DDIM inversion [47] to
invert the source image to the latent code of Stable Diffusion.

## B. Diffusion Features Selection

To conduct K-means clustering for diffusion feature selection, we first
employed the elbow method, visualizing the results. However, a distinct elbow
was not visually apparent, as shown in Figure 8. The left 6 images are WCSS
values from randomly selected images out of our 1,000 test images. All 6 plots
show similar patterns, making it hard to select a definitive elbow as stated
in the main paper. The right image, which exhibits similar results, shows the
average of WCSS on all 1,000 images.

Therefore, we chose to use the Silhouette score [38] and Davies-Bouldin index
[7], which are two of the most widely used numerical method when choosing the
optimal number of clusters. However, they are two different methods, whose
results do not always match with each other. We first visualized and found the
contradicting results of these two methods as shown in Figure 9. Therefore, we
chose to use the one that first matches the $i^{\text{th}}$ highest silhouette
score and the $i^{\text{th}}$ lowest Davies-Bouldin index simultaneously. This
process of choosing the optimal number of clusters can be written as follows :

Algorithm 1 Finding the Optimal Number of Clusters

1:$MAX\\_clusters=Total\\_time\\_steps/2$

2:$sil\\_indices\leftarrow\text{sorted}(\text{range}(MAX\\_clusters),\text{key}=\lambda
k:silhouette\\_scores[k],\text{reverse}=True)$

3:$db\\_indices\leftarrow\text{sorted}(\text{range}(MAX\\_clusters),\text{key}=\lambda
k:db\\_scores[k],\text{reverse}=False)$

4:for $i\leftarrow 0$ to $MAX\\_clusters$ do

5: if $sil\\_indices[i]$ in $db\\_indices[:i+1]$ then

6: $k\\_optimal$ = $sil\\_indices[i]$+1

7: break

8: end if

9:end for

We conducted this process twice with two different numbers of PCA components
(10 and 30), yielding the results shown in Figure 10. The averages (13.26 and
13.34) and standard deviations (0.69 and 0.69) were calculated. As the mode
value with both PCA components was 13, and the rounded average was also 13, we
chose our optimal k to be 13. Using this number of clusters, we chose the
representative feature as the one nearest to the center of each cluster.

Figure 9: Visualization of contradicting results of Silhouette scores and
Davis Bouldin indices on five different images.

From this process, we ended up with the following t values:
[0,3,8,12,16,21,25,28,32,35,39,43,47]. To verify the process, if an optimal
number of clusters in each image can really be globally adjusted, we compared
our selected features against the baselines. These baselines included sampling
at equal time intervals (t=[i*4+1 for i in the range of (0,13)]) and randomly
selecting 13 values. We calculated the minimum Euclidean distance from each
feature and confirmed that our method resulted in the minimum distance across
1,000 randomly sampled images. This is illustrated in Table 4.

Table 4: Sum of the minimum distances from all features Method | Euclidean Distance
---|---
Ours | 18,615.6
Equal time steps | 19,004.9
Random sample | 23,957.2
Figure 10: Visualization on histogram for optimal k value with different
number of PCA components. Figure 11:  Additional analysis on sampled features.
PCA is applied to DDIM sampled features from different classes. Up : features
colored with human-labeled classes. Down : features colored with denoising
timesteps

In the main paper, we found several key insights through the visualization of
features within the manually selected classes, which we summarize extensively
here. First, semantically similar images lead to similar trajectories,
although not identical. Second, features in the initial stage of the diffusion
process (when t is approximately 50) retain similar information despite
significant differences in the resulting images. Third, features in the middle
stage of the diffusion process (when t is around 25) exhibit larger
differences between adjacent features in their time steps. Lastly, the feature
at the final time step (t=0) possesses distinctive information, varying
significantly from previous values. This is also evident in the additional
visualization presented in Figure 11.

Our automatically selected features indicate a prioritization of the final
feature (t=0), and that selection was made more from the middle stages than
from the initial steps (t=[21,25,28] versus t=[43,47]). Our finding offer some
guidance for manual feature selecting to consider the time steps, especially
when memory is constrained. The order of the preference will on the last
feature (t=0), a middle one (t is near 25), and the middle to final time steps
while the features from initial steps are preferred less in general. For
instance, when selecting four features from 50 time steps, a possible
selection could be t=[0,12,25,37].

### B.2 Features From Additional Models

While we focused on T=50 DDIM sampling, for generalization, we examined
different intervals (T=25, T=100) and different model. For these experiments,
we randomly sampled 100 images. While our main experiments were conducted with
manually classified images, we utilized DINOv2 [32], which was contrastively
trained in a self-supervised manner and has learned visual semantics. With
DINOv2, we separated the data into 15 different clusters and followed the
process described in the main paper to plot the features. Here, we used 15
images from each cluster to calculate the PCA axis while we used 17 classes in
the main experiments. The results, as shown in Figure 12 and Figure 13,
indicate that even with different sampling methods, the same conclusions
regarding the sampling method can be drawn. The last feature exhibits a
distinct value, while the features from the initial time step show similar
values.

In addition, we also tested on different model, Stable diffusion V2.1 which
produce 768$\times$768 images. Following the same process, we randomly sampled
100 images and clustered with DINOv2 and plot as shown in Figure 14. This
result also shows that even with different model with different resolution,
the same conclusions can be drawn, showing the scalability of our analysis.

Figure 12:  Additional analysis on sampled features. PCA is applied to 25
steps of DDIM sampled features with different clusters. Up : features colored
with DINOv2 clusters. Down : features colored with denoising timesteps.
Figure 13:  Additional analysis on sampled features. PCA is applied to 100
steps of DDIM sampled features with different clusters. Up : features colored
with DINOv2 clusters. Down : features colored with denoising timesteps. Figure
14:  Additional analysis on Stable diffusion v2.1 sampled features. PCA is
applied to 50 steps of DDIM sampled features with different clusters. Up :
features colored with DINOv2 clusters. Down : features colored with denoising
timesteps.

## C. VAE Decoder Features

VAE features fused with the Aggregation network features for FFD in the
proposed model architecture. Figure 15 shows a visualization of the VAE
features. We used a set of 20 generated face images and extracted features
from different decoder layers of the UNet and VAE decoder, at the last time
step (t=0) similar to that of PNP [50]. We observe that the use of VAE decoder
resulted in higher-frequency details than the UNet decoder. While the feature
from UNet decoder contains semantic information, the features from VAE decoder
produces finer details such as hair, wrinkles, and small letters.

Figure 15: Extended visualization of features from UNet and VAE. (a) shows the
UNet decoder features in lower resolution (layers 1), intermediate resolution
(layers 5), and higher resolution (layers 11). (b) shows the VAE decoder
features in lower resolution (layers 1), intermediate resolution (layers 6),
and higher resolution (layers 9).

## D. Condition Diffusion Sampling for Training

### D.1 Rationale Behind CDST

An underlying assumption of CDST is that for a directional CLIP loss, two
images with a similar domain ($I_{source}$ and $I_{samp}$ in the main paper)
leads to higher confidence compared to two images with a different domain. To
examine this, we performed a confidence score test using 4SKST [43] which
consists of four different sketch styles paired with color images. 4KST is
suitable for the confidence score test because it contains images from two
different domains, photos, and anime images, in four different styles.

We manually separated into photos and anime images since it was not labeled.
Here, we computed a confidence score to determine if the directional clip loss
is more reliable when the calculated source images are in the same domain. We
performed a test with three settings, measuring cosine similarity between the
images $I_{A}$ (Photo) and $I_{B}$ (Anime) from different domains with the
corresponding sketches $S_{A}$ and $S_{B}$. All these images were encoded into
the CLIP embedding. We employed two similarity scores $Sim_{within}$ and
$Sim_{across}$ in the same manner as the main paper (Sec.4.2). We calculated
the similarity of the features in the photo domain, in the anime domain, and
across the two domains. The equation can be expressed as follows:

Table 5: Confidence scores on 4SKST with four different styles. Similarity | Style1 | Style2 | Style3 | Style4 | Average
---|---|---|---|---|---
confidence(Anime,Anime) | 104.2608 | 102.8716 | 108.2026 | 101.3530 | 104.1720
confidence(Photo,Photo) | 101.9346 | 98.8005 | 102.4516 | 100.5453 | 100.9330
confidence(Photo,Anime) | 94.5036 | 94.0189 | 98.1867 | 92.3874 | 94.7742

$\displaystyle
Sim(X,Y)={\cos(\overrightarrow{I_{X}I_{Y}}\cdot\overrightarrow{S_{X}S_{Y}})+\cos(\overrightarrow{I_{X}S_{X}}\cdot\overrightarrow{I_{Y}S_{Y}})\over{N}}$
(6)

where $cos(a\cdot b)$ is the cosine similarity and N is the total number of
$cos$ calculation. X,Y corresponds to the images in each domain.

With these computed similarities, the confidence score in domain A and domain
B can be written as follows where $Sim_{(}ALL,ALL)$ denotes the average
similarity of all images:

$\displaystyle\textit{confidence(A,B)}={{Sim(A,B)}\over{Sim_{(}ALL,ALL)}}\times
100$ (7)

In Table 5, we show the confidence test results on four different style
sketches. For all four styles, calculating the directional CLIP loss in the
same domain produced higher confidence compared to the confidence computed
across a different domain. Accordingly, we propose a sampling scheme, CDST to
train the generator in the same domain at the initial stage of the training,
which leads to higher confidence while widening its capacity in the latter
iterations of training.

### D.2 Additional Experiment on CDST

In the main paper, we used $D_{SD}$ for CDST. However, the distribution of the
condition of a pretrained stable diffusion network is not known. Therefore, we
approximate $D_{SD}$ by randomly sampling 1,000 text prompts from the
LAION-400M [41], which is a subset of the trained text-image pairs of the SD
model. We then tokenized and embedded these prompts for preprocessing,
following the process of the pretrained SD model. We conducted PCA on these
1,000 sampled embeddings to extract 512 principal components. We then checked
the normality of the sampled embeddings with all 512 principal component axes
using the Shapiro-Wilk test [45] with a significance level of $\alpha=5\%$.

As a result, 214 components rejected the null hypothesis of normality. This
indicates that each of its marginals cannot be assumed to be univariate
normal. Next, we conducted the Mardia test [27, 28] with the same 1,000
samples, taking into account skewness and kurtosis to check if the
distribution is multivariate. The results failed to reject the null hypothesis
of normality with a significance level of $\alpha=5\%$. Therefore, we assumed
$D_{SD}$ as a multivariate normal distribution for our sampling during
training.

We examined whether our calculated distribution of stable diffusion ($D_{SD}$)
is similar to the ground truth embedding distribution of LAION-400M. For
verification, we sampled 100k data from the embedded LAION-400M as a subset of
ground truth. We also sampled same amount of embeddings from the multivariate
normal distribution (Ours), univariate normal distribution for each axis, and
a uniform distribution between the max and min values of the sampled embedded
LAION-400M as a baseline. We used Earth moving distance (EMD) [20] and found
out that the multivariate normal distribution lead the lowest distance, as
shown in Table 6.

$\displaystyle M_{ij}=\lVert{{dist}}_{i}-{{dist_{GT}}}_{j}\rVert_{2},$ (8)
$\displaystyle a_{i}=\frac{1}{{{len(dist)}}},\quad
b_{j}=\frac{1}{{{len(dist_{GT})}}},$ $\displaystyle
W({{dist}},{{GTdist}})={\text{EMD}}(a,b,M).$

This result does not prove that $D_{SD}$ has multivariate normality, and the
difference with the normal distribution is marginal. However, it is sufficient
for our usage of the condition diffusion sampling for training.

Table 6: Distance from GT embeddings. Method | EMD
---|---
Multivariate normal (Ours) | 244.22
normal distribution for each axis | 244.31
uniform distribution | 1480.57

## E. Qualitative Results

We present additional results from the baseline comparisons in Figure 16 and
17. Each figure shows the results that compared
$\text{DiffSketch}_{distilled}$ and the baseline methods on the COCO dataset
[23] and the BSDS500 dataset [29], respectively. Addition to this, we also
provide visual examples of video sketch extraction results on diverse domain
including buildings, nature, and animals [46, 49] using
$\text{DiffSketch}_{distilled}$ in Figure 18 and supplementary video.

Figure 16: Qualitative comparison with alternative sketch extraction methods
on COCO dataset. Figure 17: Qualitative comparison with alternative sketch
extraction methods on BSDS500 dataset. Figure 18: Qualitative examples of
video sketch extraction.