# AirSketch: Generative Motion to Sketch

Hui Xian Grace Lim111Authors contributed equally.
Center for Research in Computer Vision
University of Central Florida
Orlando, Florida 32816
<EMAIL_ADDRESS>
&Xuanming Cui††footnotemark:
Center for Research in Computer Vision
University of Central Florida
Orlando, Florida 32816
<EMAIL_ADDRESS>
&Yogesh S Rawat
Center for Research in Computer Vision
University of Central Florida
Orlando, Florida 32816
<EMAIL_ADDRESS>
&Ser-Nam Lim
Center for Research in Computer Vision
University of Central Florida
Orlando, Florida 32816
<EMAIL_ADDRESS>

###### Abstract

Illustration is a fundamental mode of human expression and communication.
Certain types of motion that accompany speech can provide this illustrative
mode of communication. While Augmented and Virtual Reality technologies
(AR/VR) have introduced tools for producing drawings with hand motions (air
drawing), they typically require costly hardware and additional digital
markers, thereby limiting their accessibility and portability. Furthermore,
air drawing demands considerable skill to achieve aesthetic results. To
address these challenges, we introduce the concept of AirSketch, aimed at
generating faithful and visually coherent sketches directly from hand motions,
eliminating the need for complicated headsets or markers. We devise a simple
augmentation-based self-supervised training procedure, enabling a controllable
image diffusion model to learn to translate from highly noisy hand tracking
images to clean, aesthetically pleasing sketches, while preserving the
essential visual cues from the original tracking data. We present two air
drawing datasets to study this problem. Our findings demonstrate that beyond
producing photo-realistic images from precise spatial inputs, controllable
image diffusion can effectively produce a refined, clear sketch from a noisy
input. Our work serves as an initial step towards marker-less air drawing and
reveals distinct applications of controllable diffusion models to AirSketch
and AR/VR in general.

Figure 1: Given a hand drawing video, we first use a hand detection algorithm
to extract the tracking image (Hand Tracking). We then take this noisy
tracking image and generate a clean sketch (Generated), which faithfully and
aesthetically resembles the intended sketch (Ground-truth).

##  1 Introduction

Hand gestures are an essential element in communication[42]. In particular,
iconic hand motions (i.e. air drawing) can depict visual aspects of an object.
This form of expression is frequently used to visually supplement verbal
communication and is used in various practical applications, including
conceptual discussions, overcoming language barriers, and aiding in visual
design.

Popular AR/VR tools like Google’s Tiltbrush [19] and HoloARt [5] create
visuals via hand motions, but are inconvenient. These applications generally
require head-mounted displays and digital hand markers, which are costly and
hinder portability. Furthermore, their weight and temperature make them
unsuitable for continuous use [59, 71, 12], and so are inconvenient for
spontaneous usage. Yet, these devices provide accurate positioning,
stabilization, and varied brush controls and are crucial to producing high-
quality drawings.

Can we generate sketches from hand motions without additional sensors or
markers? In order to enhance accessibility and convenience, we aim to generate
sketches from hand motions videos captured using any standard camera embedded
in devices like smartphones or smart glasses.

One could clearly deploy hand tracking algorithms [9, 73] to turn these hand
motion videos into sketches. However, creating air drawings with a hand
tracking algorithm alone presents several challenges. These include the user’s
drawing ability, physical fatigue, and inaccuracies in hand tracking. Noise in
hand tracking can severely distort a sketch, rendering it almost
unrecognizable.

The objective, therefore, is to generate clean sketches that faithfully
represent the user’s intent, from highly noisy and distorted hand motion
input. This task requires the model to possess a deep understanding of shape
and object priors, enabling it to discern and correct deformed motion cues
while filtering out undesirable noise. We refer to this task as Generative
Motion to Sketch.

There are many approaches to this task, with different architecture and data
modalities. The input modality might include learned video representations,
coordinate sequences from a hand tracking algorithm, or a rasterized image.
Depending on the modality, the task may also be reformulated as video-to-
sketch, sequence-to-sequence, image-to-image, or a combination thereof. This
diversity introduces interesting opportunities for rich exploration of all
these diverse approaches.

We explore the use of controllable image Diffusion Models (DM) in generating
sketches from motion. Existing work such as ControlNet [75] and T2IAdapter
[43] generate photo-realistic images given spatially-precise conditioning
images. We explore a different use case by using controllable DMs to
“reconstruct” clean sketches from severely distorted and noisy input images
obtained with a hand tracking algorithm. We propose a simple, augmentation-
based, self-supervised training procedure and construct two air drawings
datasets for evaluation purposes.

Our experiments show that with our augmentation-based training, controllable
image DMs are able to recognize and interpret correct visual cues from noisy
tracking images, some of which even appear to be nearly unrecognizable to the
human eye, and generate sketches faithfully and aesthetically, while being
robust to unseen objects. Moreover, we show that through simple augmentations,
the model is capable of sketch-completion and text-instructed stroke styling.
Finally, we conduct ablations to investigate 1) the effects of different
augmentations, 2) the contribution of text prompts to sketch generation and 3)
the effect of different level of input ‘chaos’ on the quality of resulting
generations.

In summary, in this paper:

  1. 1.

We conduct a pilot study of AirSketch, sketch generation from marker-less air
drawing, and provide two air drawing datasets for evaluation.

  2. 2.

We propose a controllable DM approach that generates faithful and aesthetic
sketches from air-drawn tracking images with a self-supervised, augmentation-
based training procedure, and conduct ablation studies to prove its
effectiveness and robustness.

  3. 3.

We explore a different way of using spatially-conditioned DMs and reveal some
of their interesting properties in the context of sketch generation. We hope
our experiments shed new light on understanding the properties of controllable
DMs and facilitate future research.

##  2 Related Works

###  2.1 Sketching in AR/VR

There are many methods for drawing in AR and VR. Applications such as Google’s
Tilt Brush [19], Mozilla’s A-Painter [44], Oculus’ Quill [47], and HoloARt [5]
display user strokes as lines or tubes that can extend in all three
dimensions. Many sketching applications such as these require a combination of
VR/AR headsets and controllers to use. Since drawing freehand with six degrees
of freedom makes it difficult to draw steady lines and surfaces, applications
like AdaptiBrush [58] and Drawing on Air [27] use trajectory-based motion of
ribbons to render strokes predictably and reliably. Just-a-Line [20] and
ARCore Drawing [25] are smartphone-based AR drawing applications, where users
draw lines by moving a smartphone in the air [20], or drawing on a smartphone
screen [25].

###  2.2 Sketch Applications

#### Representing Sketches.

A sketch can be represented in both vector and pixel space, and with varying
levels of abstraction. A sketch can be represented as a rasterized image [39,
74, 49, 36, 31], a sequence of coordinates [22, 55, 37], a set of Bezier
curves [66, 17], or a combination [8]. These representation methods are suited
for different tasks. For example, the rasterized image representation is
commonly used in Sketch-Based Image Retrieval (SBIR) in order to compare
sketches with images, while the coordinate sequence representation is often
used for sketch generation. On the other hand, a sketch can also depict the
same object at varying abstraction levels, thereby imposing further challenges
on downstream tasks, especially sketch-based retrieval in 2D [36, 31, 61] and
3D [68, 40, 13, 30].

#### Sketch Generation.

Most existing sketch generation methods adopt the vector representation [22,
55, 37], and view sketch generation as an auto-regressive problem on
coordinate sequences. These sequences are typically represented by lists of
tuple $(\delta x,\delta y,\mathbf{p})$, where $\delta x$ and $\delta y$
represent the offset distances of x and y from the previous point, and
$\mathbf{p}$ is a one-hot vector indicating the state of the pen. Sketch-RNN
[22] uses a Variational Autoencoder (VAE) with a bi-directional RNN as the
encoder and an auto-regressive RNN as the decoder. Sketch-Bert [37] follows
the BERT [14] model design and training regime. Instead of auto-regressive
sketch generation, SketchGAN [38] takes in a rasterized sketch image and uses
a cascade GAN to iteratively complete the sketch.

Nonetheless, these methods generate sketches in the input sketch modality by
taking in a sketch and reconstructing the exact same sketch, or by predicting
endings given incomplete sketches. In contrast, our work considers the task
for generating sketches from hand motions. Specifically, we adopt the image
representation for sketch generation as opposed to using coordinate sequences.
This offers several advantages: it 1) maintains constant computation
complexity with regards to sketch length, 2) is drawing-order invariant, which
is especially favorable in our case as we consider extremely noisy sketches as
input, and 3) allows us to utilize large pretrained image generative models.

###  2.3 Image Diffusion Models

#### Diffusion Probabilistic Model.

First introduced by Sohl-Dickstein et al. [63], diffusion probabilistic models
have been widely applied in image generation [23, 64, 28]. To reduce
computational cost, Latent Diffusion Models [56] project input images to lower
dimension latent space, where diffusion steps are performed. Text-to-image
diffusion models [46, 54, 53, 60, 50] achieve text control over image
generation by fusing text embeddings obtained by pre-trained language models
such as CLIP [51] and T5 [52] into diffusion UNet [57] via cross attention.

#### Controllable Image Generation.

Beyond text, various recent works have focused on allowing more fine-grained
controls over the diffusion process. This can be done via prompt engineering
[72, 35, 77], manipulating cross-attention [6, 10, 26, 70], or training
additional conditioning adapters [43, 75]. ControlNet [75] learns a trainable
copy of the frozen UNet encoder and connects it to the original UNet via zero
convolution. T2IAdapter [43] trains a lightweight adapter to predict residuals
that are added to the UNet encoder at multiple scales.

In particular, both ControlNet and T2IAdapter take spatially-precise
conditional images such as canny edge, depth, and skeleton images extracted
from the original image. Koley et al. [32] considers human sketches as a
condition and observes that human sketches have shapes that are deformed
compared to canny edge or depth images, resulting in generations that lack
photo-realism. Therefore, they avoid using the spatial-conditioning approach
by training a sketch adapter to transform the sketch into language tokens that
replace the text embeddings as the input to the cross attention.

In our work, we show that the spatial-conditioning does not have to be a
“hard” constraint, at least not in the domain of sketch generation. Through
proper augmentations, we discover that we can indeed teach ControlNet to map
from a noisy input condition to a clean output.

###  2.4 Hand tracking and gesture recognition

Hand tracking and gesture recognition are used for several purposes, including
communication and interaction in an AR environment [41] [33] and VR [1], for
image-based pose estimation in sign language recognition [7] [3] [4], and many
others [34] [11]. Many of these require depth-sensing hardware, such as work
done by Oikonomidis et. al. using a Kinect sensor [48], or use deep learning
for pose estimation [67] [18], making it difficult to integrate them into
lightweight systems.

Hand pose estimation such as MediaPipe Hands [73] and OpenPose [9] take in RGB
images or video and return the coordinates of 21 landmarks for each hand
detected, and MediaPipe Hands is lightweight enough to integrate into even
mobile devices.

##  3 Air-Drawing Dataset

In order to thoroughly evaluate our model, we need datasets with sketch-video
pairs. Popular sketch datasets include Sketchy [62], TUBerlin [16], and Quick,
Draw! [21]. There are also hand motion datasets that associate hand motions
with semantic meaning, such as Sign Language MNIST [65], How2Sign [15], and
the American Sign Language Lexicon Video Dataset [45]. However, there are no
datasets that associate sketches with air drawing hand motions, prompting us
to create our own sketch-video pair datasets for evaluation purposes.

Figure 2: Examples of air drawing videos. Left: synthetic hand drawing; Right:
real hand drawing.

#### Synthetic Air-Drawing Dataset.

We use samples from the Quick, Draw! dataset [21] as the intended ground-truth
sketch; each stroke is represented as a sequence of timestamps and
coordinates. A 3D human arm asset is animated in the Unity engine [29] using
inverse kinematics and rotation constraints, see Figure 2 (left). While
following stroke coordinates, the index finger is extended, and when the
stroke ends, the hand is in a closed fist. The videos have an aspect ratio of
1920 by 1080 pixels and were recorded at 60 frames per second. We choose 50
common object categories from Quick, Draw! dataset, each with 100 sketches to
form our synthetic dataset with a total of 5000 sketch-video pairs. This
synthetic Air-Drawing dataset simulates the scenario where users have perfect
drawing ability, and the errors are solely introduced by the tracking
algorithm.

#### Real Air-Drawing Dataset.

A human user attempts to replicate sketches from the Quick, Draw! dataset by
moving their index finger through the air. The videos were recorded with
aspect ratio of 1280 by 720 pixels and at 30 frames per second. We take 10
samples per category used in the synthetic dataset, resulting in 500 video-
sketch pairs.

##  4 Methods

Our method trains a controllable DM to recover the clean sketch from a noisy
one, which we discuss in Sections 4.1 and 4.2. As input at training time, we
simulate noisy sketches produced from the direct application of a hand
tracking algorithm, discussed in Section 4.3. Finally, we evaluate our model
using two different datasets: a dataset of 3D animated hand motion videos, and
a small dataset of real hand motion videos. The creation of these two datasets
is discussed in Section 3.

###  4.1 Preliminary: Controllable DMs

Diffusion models involve a forward and inverse diffusion process. Given an
input image $\bm{x}_{0}$, the forward process gradually adds noise to
$\bm{x}_{0}$ to form a noisy input $\bm{x}_{t}$ at time step $\bm{t}$ as:

$\bm{x}_{t}=\sqrt{\bar{\alpha}_{t}}\bm{x}_{0}+(\sqrt{1-\bar{\alpha}_{t}})\epsilon,\epsilon\sim\mathcal{N}(0,\bm{I})$
(1)

where $\bar{\alpha}_{t}:=\prod^{t}_{i=1}\alpha_{i}$, and
$\alpha_{t}=1-\beta_{t}$ is determined by a pre-defined noise scheduler [23].

The reverse process trains a denoising UNet $\epsilon_{\theta}(\cdot)$ to
predict the noise $\epsilon$ added to input $\bm{z}_{0}$. In the context of
controllable generation, as in ControlNet [75] and T2IAdapter [43], with a set
of conditions including a text prompt $\bm{c}_{t}$ and an additional condition
$\bm{c}_{\text{f}}$, the overall loss can be defined as:

$\mathcal{L}=\mathbb{E}_{\bm{x}_{0},\bm{t},\bm{c}_{t},\bm{c}_{\text{f}},\epsilon\sim\mathcal{N}(0,1)}\Big{[}\|\epsilon-\epsilon_{\theta}(\bm{x}_{t},\bm{t},\bm{c}_{t},\bm{c}_{\text{f}})\|_{2}^{2}\Big{]},$
(2)

###  4.2 Training Controllable DMs for Sketch Recovery

Figure 3: The overall pipeline for training and inference. During training, we
randomly apply augmentations to the original ground-truth sketch to form a
distorted view. The distorted view is passed to ControlNet, which is asked to
generate the original, undistorted sketch. During inference, a hand-tracking
algorithm is used on a hand motion video to create the input.

We adopt ControlNet [75] as our primary controlling approach. Our training
procedure is illustrated in Figure 3. Due to the lack of sketch-video pair
datasets, we devise a self-supervised, augmentation-based training procedure.
During training, for each sketch image, we randomly sample combinations of
augmentations $\mathcal{A}(\cdot)$ and apply to $\bm{x}_{0}$ to get the
distorted view $\mathcal{A}(\bm{x}_{0})$. It is then used as the input to
ControlNet’s conditioning adapter. Hence, the loss function 2 can be re-
written as:

$\mathcal{L}=\mathbb{E}_{\bm{x}_{0},\bm{t},\bm{c}_{t},\bm{c}_{\text{f}},\epsilon\sim\mathcal{N}(0,1)}\Big{[}\|\epsilon-\epsilon_{\theta}(\bm{x}_{t},\bm{t},\bm{c}_{t},\mathcal{A}(\bm{x}_{0}))\|_{2}^{2}\Big{]}.$
(3)

Therefore, unlike regular controllable DMs where the conditioning adapter
takes in edge-like maps and predicts spatial-conditioning signals to be
injected to the UNet, our adapter learns both the spatial-conditioning signals
and a mapping from the distorted to the clean input:
$\mathcal{A}(\bm{x}_{0})\shortrightarrow\bm{x}_{0}$.

###  4.3 Sketch Augmentations

We categorize the prevalent errors from air drawings into three types: 1)
user-induced artifacts such as hand jitters and stroke distortions, 2) hand
tracking errors such as inaccurate hand landmark predictions, unintended
strokes, and 3) aesthetic shortcomings related to the user’s drawing
proficiency. In order to closely replicate these artifacts, we carefully
examine typical noise found in real tracking samples and divide them into 3
categories: local, structural, and false strokes. For each category, we
observe several types of artifacts, and apply augmentations to introduce each
artifact to an input sketch. Visual examples for these augmentations are shown
in Figure 4.

Figure 4: Visual examples of different augmentations.

Local artifacts include jitters, stroke-wise distortion, and random spikes.
Jitters are common and become especially obvious when drawing speed is low.
Stroke-wise distortion happens when users fail to properly draw the desired
shape primitive (e.g. drawing a rectangle but ending up with unwanted curvy
edges), potentially due to lack of friction and visual landmarks. Random
spikes can arise from an accidental jerk of a user’s hand or an incorrect
detection from the tracking algorithm.

Structural artifacts include sketch-level distortion, incorrect stroke size,
and misplacement. Sketch-level distortion refers to an overall distorted
aspect ratio; incorrect stroke size and misplacement occur when the user
unintentionally draws a stroke too small/large, and/or at the wrong position.

False strokes refer to unintentionally drawn strokes which commonly occur in
three situations: entering or exiting the canvas, transitioning between
strokes, and hesitating during drawing.

Unlike real images where shapes and scales are strictly defined, sketches are
more versatile and thus do not have a clear boundary for correctness. For
example, a slight change of scale during augmentation does not falsify the
sketch. Therefore, we carefully tune the augmentation parameters such that the
resulting augmented sketches are, in general, obviously incorrect.

Table 1: Results on the similarity between generated and ground-truth sketches. “Tracking” refers to hand tracking images, and “Gen.” refers to generated images. “w/ Aug.” refers to whether sketch augmentations have been applied. “CLIP I2I/I2T” refers to CLIP Image-to-Image/Image-to-Text similarity. | Dataset | Backbone | w/ Aug. |  SSIM ($\uparrow$) |  CD ($\downarrow$) |  LPIPS ($\downarrow$) |  CLIP I2I ($\uparrow$) |  CLIP I2T ($\uparrow$)
---|---|---|---|---|---|---|---|---
Seen Categories
Tracking | synth. | – | – | 0.59 | 20.12 | 0.36 | 0.80 | 0.22
Gen. | synth. | SDXL | ✗ | 0.59 | 20.11 | 0.37 | 0.79 | 0.23
Gen. | synth. | SD1.5 | ✓ | 0.60 | 17.98 | 0.35 | 0.80 | 0.26
Gen. | synth. | SDXL | ✓ | 0.64 | 17.39 | 0.33 | 0.85 | 0.28
Tracking | real | – | – | 0.55 | 32.36 | 0.42 | 0.76 | 0.21
Gen. | real | SDXL | ✗ | 0.55 | 31.99 | 0.41 | 0.79 | 0.21
Gen. | real | SD1.5 | ✓ | 0.59 | 27.59 | 0.38 | 0.80 | 0.27
Gen. | real | SDXL | ✓ | 0.64 | 25.46 | 0.36 | 0.84 | 0.29
Unseen Categories
Tracking | synth. | – | – | 0.59 | 20.47 | 0.36 | 0.80 | 0.22
Gen. | synth. | SDXL | ✗ | 0.59 | 20.32 | 0.35 | 0.81 | 0.22
Gen. | synth. | SD1.5 | ✓ | 0.60 | 17.50 | 0.35 | 0.80 | 0.26
Gen. | synth. | SDXL | ✓ | 0.64 | 17.27 | 0.34 | 0.85 | 0.27
Tracking | real | – | – | 0.54 | 33.92 | 0.42 | 0.76 | 0.21
Gen. | real | SDXL | ✗ | 0.55 | 33.53 | 0.41 | 0.78 | 0.21
Gen. | real | SD1.5 | ✓ | 0.61 | 27.67 | 0.38 | 0.80 | 0.27
Gen. | real | SDXL | ✓ | 0.63 | 24.26 | 0.38 | 0.85 | 0.28

##  5 Experiments

#### Datasets and Implementation Details.

In training, we use a subset of 100 categories from the Quick, Draw! dataset
[21]. Because a large portion of Quick, Draw! sketches are not well-drawn, we
first calculate the CLIP Image-Text similarity between each sketch and their
corresponding category and select the top 5% from each category, resulting in
60K sketches. Note that the sketches used for training are mutually exclusive
with the sketches used for generating synthetic and real tracking images used
during evaluation. To test for generalizability, we select 10 categories with
similar statistics 222Details in Appendix A.2 from the rest and exclude them
from training.

We primarily use Stable Diffusion XL [50] (SDXL) in our experiments, and
adhere to the original ControlNet training and inference procedures. During
both training and inference phases, we use text prompts in the format of “a
black and white sketch of a <category>” to guide the model generation. We also
finetune SDXL on the Quick, Draw! dataset with Low-Rank Adaptation [24] (LoRA)
in order to “equip” the model with the basic ability to generate sketches in
the appropriate style.

Figure 5: Inference results on unseen categories. For the left group, the
categories are: angel, cat, cow, sun; for the right group, the classes are:
cow, candle, car, snail (top to bottom).

#### Evaluation Metrics.

We primarily focus on using faithfulness, or the similarity between the
generated sketch and the ground-truth sketch, as our model performance. Due to
the versatility of sketches, we adopt multiple metrics to ensure comprehensive
measurements. On the pixel-level, we use SSIM [69] to measure detailed local
structural similarity, and Chamfer Distance [2] (CD) for global comparison, as
CD is less sensitive to local density mismatch. Taking a perceptual
perspective, we adopt LPIPS [76], CLIP [51] Image-to-Image similarity (I2I),
and CLIP Image-to-Text similarity (I2T) between sketches and their
corresponding text prompts to measure “recognizability”.

We benchmark our model on the similarity between the ground-truth sketch and
the hand tracking image. We then train a ControlNet on sketches but without
any augmentation as our second baseline.

###  5.1 Results and Analysis

#### Faithfulness.

In Figure 1 and Figure 6 we can clearly observe the ControlNet trained with
augmentations successfully identifies the visual cues from the noisy tracking
image, removes artifacts, and generates the clean sketches, while being
aesthetic and semantically coherent. Unsurprisingly, ControlNet trained
without augmentations fails to make any improvement from the tracking.

Table 1 shows quantitative results for measuring the faithfulness of generated
sketch images to ground-truth. For both synthetic and real datasets, we
observe a noticeable performance gain. For example, with the real dataset and
SDXL, SSIM increases by 10% in SSIM, LPIPS decreases by 6%, and CD decreases
by 21%.

#### Generalizability.

From Figure 5 we can see that our model generalizes well to categories which
are not seen during training, suggesting that the trained ControlNet learns a
robust category-agnostic mapping from the noisy to clean sketch. The
generalizability is also verified in Table 1, where the resulting scores for
unseen categories are close to scores obtained on the seen categories.

Table 2: Model trained on seen categories with different combinations of augmentations. All the experiments are conducted using Stable Diffusion1.5 and trained for 5K steps. Local | Structural | False Strokes |  SSIM ($\uparrow$) |  CD ($\downarrow$) |  LPIPS ($\downarrow$) |  CLIP I2I ($\uparrow$) |  CLIP I2T ($\uparrow$)
---|---|---|---|---|---|---|---
Orig. vs Tracking | 0.55 | 32.36 | 0.42 | 0.76 | 0.21
✓ |  |  | 0.60 | 31.49 | 0.39 | 0.83 | 0.27
| ✓ |  | 0.55 | 31.14 | 0.41 | 0.80 | 0.24
|  | ✓ | 0.57 | 29.04 | 0.40 | 0.82 | 0.24
✓ | ✓ |  | 0.60 | 31.21 | 0.39 | 0.83 | 0.28
✓ |  | ✓ | 0.61 | 29.82 | 0.37 | 0.83 | 0.28
| ✓ | ✓ | 0.60 | 28.97 | 0.39 | 0.82 | 0.23
✓ | ✓ | ✓ | 0.62 | 27.33 | 0.37 | 0.84 | 0.29
Figure 6: Qualitative results when different combinations of augmentations are
applied during training. Each column represents the generated sketches when
the model is trained with one combination of augmentations, e.g. Column 2 are
the generated sketches when only structural augmentations are applied during
training.

###  5.2 Ablations

#### Role of Augmentations.

In Table 2 we provide similarity scores between ground-truth and generated
sketches when different combinations of augmentations are applied in training.
We observe that local augmentations plays an important role in detailed local
similarity: with only local augmentation applied, SSIM increases more than
other metrics, with a 10% improvement to the baseline, yet increases by only
4% and 0% when only false strokes and structural augmentations are applied.
Conversely, false strokes and structural augmentation have larger effects on
the global similarity of the generated sketch: CD decreases by 9% and 3% when
only false strokes and structural augmentations are applied, respectively.

In Figure 6 we provide visual results when different combinations of sketch
augmentations are applied during training. We observe that local augmentations
are indeed crucial to removing jitters and correcting deformed lines, while
false stroke augmentations ensure that the model does not falsely follow the
spatial-conditions introduced by these false strokes. The structural
augmentations are less significant, likely because structural artifacts are
not as common as local artifacts and false strokes.

#### Effect of Text Prompts.

Input tracking image conditions are extremely noisy and may not even possess
obvious visual cues about the nature of the intended sketch. It is then
important to investigate the effect of text guidance on the generated output
and examine if our augmentation-based training significantly contributes to
the generation, or if it is guided purely by text prompts.

In Table 8 we show the similarity scores between the ground-truth and
generated sketches with or without prompt on seen or unseen categories. When
no augmentation is applied during training, there is little to no performance
gain, even with a text prompt. When augmentations are present, in most cases,
there is noticeable improvement across all metrics. Such results verify the
necessity of our augmentation-based training procedure.

Figure 7: Examples of incorrect generation on unseen categories due to the
absence of text prompt (w/o Prompt), comparing to correct generations when
prompt is present (w/ Prompt).

We find that the model tends to rely more on text prompts when working with
unseen categories. When prompts are not present, CLIP I2T and CD drop 24.7%
and 21.4% respectively on the unseen categories, versus a decrease of 4.9% and
10.3% on the seen categories. In Figure 7, we show examples of the model
failing to generate the correct sketch when no prompt is present, under unseen
categories. Consider the bottom left example of Figure 7: the ground-truth
sketch (Ground-truth) represents a snail. With the corresponding text prompt,
the model successfully generates a sketch of a snail, following visual cues in
the tracking image (Tracking) such as the spiral shape. When no prompt is
given, the model falsely, yet reasonably, generates a fan (w/o Prompt) – a
category seen during training. In fact, in the absence of a text prompt and
given an input from an unseen category, we find that the failure cases tend to
be generated sketches of the seen categories.

Table 3: Quantitative results on real air drawing dataset with/without
prompts. “w/ Aug.” refers to with/without augmentations. “w/ P.” refers to
with/without prompts. “I2I” refers to CLIP Image-Image similarity between
tracking/generated and ground-truth sketch, and “I2T” refers to CLIP Image-
Text similarity between sketches and texts associated with their class labels.

w/ Aug. | w/ P. |  SSIM ($\uparrow$) |  CD ($\downarrow$) |  LPIPS ($\downarrow$) |  I2I ($\uparrow$) |  I2T ($\uparrow$)
---|---|---|---|---|---|---
Seen Categories
Tracking | 0.55 | 32.36 | 0.42 | 0.76 | 0.21
✗ | ✓ | 0.55 | 31.99 | 0.41 | 0.79 | 0.21
✗ | ✗ | 0.56 | 32.09 | 0.40 | 0.78 | 0.21
✓ | ✓ | 0.64 | 25.46 | 0.36 | 0.85 | 0.29
✓ | ✗ | 0.63 | 26.70 | 0.36 | 0.81 | 0.26
Unseen Categories
Tracking | 0.54 | 33.92 | 0.42 | 0.76 | 0.21
✗ | ✓ | 0.55 | 33.53 | 0.43 | 0.78 | 0.21
✗ | ✗ | 0.56 | 33.66 | 0.40 | 0.78 | 0.22
✓ | ✓ | 0.63 | 24.26 | 0.38 | 0.84 | 0.28
✓ | ✗ | 0.61 | 30.47 | 0.39 | 0.80 | 0.23

Figure 8: The impact of level of chaos of the tracking on the faithfulness of
the generated sketch, based on CD. We divide CD between tracking and ground-
truth sketch equally into four bins. Within each bin, we plot the mean of CD
between ground-truth sketch, tracking, and generated image with/without
prompt.

#### Effect of level of chaos on conditional input.

In Figure 8, we investigate how the amount of chaos in the input conditioning
tracking image, measured by the CD between the ground-truth and tracking
images, affects generation. The improvements are more obvious when the CD
between the ground-truth and tracking images is large. On the other hand, the
CD between ground-truth and generated sketches are almost the same when the
tracking image is already close to the ground-truth sketch, indicating that
the generation is being faithful to the ground-truth sketch. We also observe
that when the level of chaos is high for the unseen categories (bottom right),
the performance gain with text guidance is most obvious, suggesting that the
model relies more on text guidance for correct generation. Conversely, the
model depends less on the text prompts when the input chaos level is low or
under seen categories. Such results are also in line with Figure 7 and Table 8
as discussed above.

##  6 Conclusions

In this paper, we tackle the problem of marker-less air drawing by leveraging
a spatially-controlled diffusion model. We devise a simple augmentation-based
data-free training procedure to learn a mapping from noisy to clean sketches.
We collect two hand drawing datasets and verify that the trained model can
effectively generate coherent and faithful sketch from an extremely noisy
tracking image, and exhibits decent generalizability.

#### Limitations and Future Work.

In this work we provide a stepping stone for the proposed task by creating a
framework that establishes a correspondence between hand motions and clean,
pleasing, and coherent illustrations, and providing datasets to evaluate these
tasks. However, this work does not explore the possibility of using hand
gestures to create complex, full color images that image diffusers are known
for. It also assumes that the desired output sketch is simple and often
cartoon-like, as a majority of the Quick, Draw! sketches are drawn with few
lines and simple shapes. These are interesting future research directions.

## References

  * Adkins [2022] Alex Adkins.  _The Importance of Hand Motions for Communication and Interaction in Virtual Reality_.  PhD thesis, Clemson University, 2022.
  * Akmal Butt and Maragos [1998] M. Akmal Butt and P. Maragos.  Optimum design of chamfer distance transforms.  _IEEE Transactions on Image Processing_ , 7(10):1477–1484, 1998.  doi: 10.1109/83.718487.
  * Al-Obodi et al. [2020] Alaa Al-Obodi, Ameerh Al-Hanine, Khalda Al-Harbi, Maryam Al-Dawas, and Amal Al-Shargabi.  A saudi sign language recognition system based on convolutional neural networks.  _Building Services Engineering Research and Technology_ , 13, 11 2020.  doi: 10.37624/IJERT/13.11.2020.3328-3334.
  * Alawwad et al. [2021] Rahaf Alawwad, Ouiem Bchir, and Mohamed Maher Ben Ismail.  Arabic sign language recognition using faster r-cnn.  _International Journal of Advanced Computer Science and Applications_ , 12:692–700, 03 2021.  doi: 10.14569/IJACSA.2021.0120380.
  * Amores Fernandez and Lanier [2017] Judith Amores Fernandez and Jaron Lanier.  Holoart video showcase: Painting with holograms in mixed reality.  pages 466–466, 05 2017.  doi: 10.1145/3027063.3049787.
  * Avrahami et al. [2022] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, June 2022.  doi: 10.1109/cvpr52688.2022.01767.  URL http://dx.doi.org/10.1109/CVPR52688.2022.01767.
  * Bani Baker et al. [2023] Qanita Bani Baker, Nour Alqudah, Tibra Alsmadi, and Rasha Awawdeh.  Image-based arabic sign language recognition system using transfer deep learning models.  _Applied Computational Intelligence and Soft Computing_ , 2023:1–11, 12 2023.  doi: 10.1155/2023/5195007.
  * Bhunia et al. [2021] Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Yongxin Yang, Timothy M. Hospedales, Tao Xiang, and Yi-Zhe Song.  Vectorization and rasterization: Self-supervised learning for sketch and handwriting, 2021.
  * Cao et al. [2018] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields.  _CoRR_ , abs/1812.08008, 2018.  URL http://arxiv.org/abs/1812.08008.
  * Chen et al. [2023] Minghao Chen, Iro Laina, and Andrea Vedaldi.  Training-free layout control with cross-attention guidance, 2023.
  * Chen et al. [2020] Weiya Chen, Chenchen Yu, Chenyu Tu, Zehua Lyu, Jing Tang, Shiqi Ou, Yan Fu, and Zhidong Xue.  A survey on hand pose estimation with wearable sensors and computer-vision-based methods.  _Sensors_ , 20:1074, 02 2020.  doi: 10.3390/s20041074.
  * Chihara and Seo [2018] Takanori Chihara and Akihiko Seo.  Evaluation of physical workload affected by mass and center of mass of head-mounted display.  _Applied ergonomics_ , 68:204–212, 04 2018.  doi: 10.1016/j.apergo.2017.11.016.
  * Chowdhury et al. [2023] Pinaki Nath Chowdhury, Ayan Kumar Bhunia, Aneeshan Sain, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song.  Democratising 2d sketch to 3d shape retrieval through pivoting.  In _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 23218–23229, 2023.  doi: 10.1109/ICCV51070.2023.02127.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * Duarte et al. [2020] Amanda Cardoso Duarte, Shruti Palaskar, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giró-i-Nieto.  How2sign: A large-scale multimodal dataset for continuous american sign language.  _CoRR_ , abs/2008.08143, 2020.  URL https://arxiv.org/abs/2008.08143.
  * Eitz et al. [2012] Mathias Eitz, James Hays, and Marc Alexa.  How do humans sketch objects?  _ACM Trans. Graph. (Proc. SIGGRAPH)_ , 31(4):44:1–44:10, 2012.
  * Gal et al. [2023] Rinon Gal, Yael Vinker, Yuval Alaluf, Amit H. Bermano, Daniel Cohen-Or, Ariel Shamir, and Gal Chechik.  Breathing life into sketches using text-to-video priors.  2023\.
  * Ge et al. [2016] Liuhao Ge, Hui Liang, Junsong Yuan, and Daniel Thalmann.  Robust 3d hand pose estimation in single depth images: from single-view cnn to multi-view cnns.  06 2016.  doi: 10.1109/CVPR.2016.391.
  * Google [2016] Google.  Tiltbrush, 2016.  https://www.tiltbrush.com/.
  * Google Creative Lab and Uncorked Studios [2018] Google Creative Lab and Uncorked Studios.  Just a line, March 2018.  https://experiments.withgoogle.com/justaline.
  * Google Research [2018] Google Research.  Introducing the kaggle “quick, draw!” doodle recognition challenge, September 2018.  https://blog.research.google/2018/09/introducing-kaggle-quick-draw-doodle.html.
  * Ha and Eck [2017] David Ha and Douglas Eck.  A neural representation of sketch drawings, 2017.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models, 2020.
  * Hu et al. [2021] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models, 2021.
  * Jongejan [2017] Jonas Jongejan.  ARCore Drawing, August 2017.  https://experiments.withgoogle.com/arcore-drawing.
  * Ju et al. [2023] Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, and Qiang Xu.  Humansd: A native skeleton-guided diffusion model for human image generation, 2023.
  * Keefe et al. [2007] Daniel Keefe, Robert Zeleznik, and David Laidlaw.  Drawing on air: Input techniques for controlled 3d line illustration.  _IEEE Transactions on Visualization and Computer Graphics_ , 13(5):1067–1081, 2007.  doi: 10.1109/TVCG.2007.1060.
  * Kingma et al. [2023] Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models, 2023.
  * Knife Entertainment [2018] Knife Entertainment.  Realistic FPS hands, 2018.  https://assetstore.unity.com/packages/3d/characters/realistic-fps-hands-107409.
  * Koley et al. [2024a] Subhadeep Koley, Ayan Kumar Bhunia, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang, and Yi-Zhe Song.  How to handle sketch-abstraction in sketch-based image retrieval?, 2024a.
  * Koley et al. [2024b] Subhadeep Koley, Ayan Kumar Bhunia, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang, and Yi-Zhe Song.  You’ll never walk alone: A sketch and text duet for fine-grained image retrieval, 2024b.
  * Koley et al. [2024c] Subhadeep Koley, Ayan Kumar Bhunia, Deeptanshu Sekhri, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang, and Yi-Zhe Song.  It’s all about your sketch: Democratising sketch control in diffusion models, 2024c.
  * Lee and Höllerer [2007] Taehee Lee and Tobias Höllerer.  Handy ar: Markerless inspection of augmented reality objects using fingertip tracking.  pages 83–90, 11 2007.  ISBN 978-1-4244-1452-9.  doi: 10.1109/ISWC.2007.4373785.
  * Li et al. [2019] Rui Li, Zhenyu Liu, and Jianrong Tan.  A survey on 3d hand pose estimation: Cameras, methods, and datasets.  _Pattern Recognition_ , 93, 04 2019.  doi: 10.1016/j.patcog.2019.04.026.
  * Li et al. [2023] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee.  Gligen: Open-set grounded text-to-image generation, 2023.
  * Lin et al. [2023] Fengyin Lin, Mingkang Li, Da Li, Timothy Hospedales, Yi-Zhe Song, and Yonggang Qi.  Zero-shot everything sketch-based image retrieval, and in explainable style, 2023.
  * Lin et al. [2020] Hangyu Lin, Yanwei Fu, Yu-Gang Jiang, and Xiangyang Xue.  Sketch-bert: Learning sketch bidirectional encoder representation from transformers by self-supervised learning of sketch gestalt, 2020.
  * Liu et al. [2019] Fang Liu, Xiaoming Deng, Yu-Kun Lai, Yong-Jin Liu, Cuixia Ma, and Hongan Wang.  Sketchgan: Joint sketch completion and recognition with generative adversarial network.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5823–5832, 2019.  URL https://api.semanticscholar.org/CorpusID:182164433.
  * Liu et al. [2017] Li Liu, Fumin Shen, Yuming Shen, Xianglong Liu, and Ling Shao.  Deep sketch hashing: Fast free-hand sketch-based image retrieval, 2017.
  * Luo et al. [2022] Ling Luo, Yulia Gryaditskaya, Tao Xiang, and Yi-Zhe Song.  Structure-aware 3d vr sketch to 3d shape retrieval, 2022.
  * Marchand et al. [2016] Eric Marchand, Hideaki Uchiyama, and Fabien Spindler.  Pose estimation for augmented reality: A hands-on survey.  _IEEE Transactions on Visualization and Computer Graphics_ , 22, 01 2016.  doi: 10.1109/TVCG.2015.2513408.
  * McNeill [1992] David McNeill.  _Hand and Mind: What Gestures Reveal about Thought_.  University of Chicago Press., Chicago, Illinois, USA, 1992.
  * Mou et al. [2023] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  _arXiv preprint arXiv:2302.08453_ , 2023.
  * Mozilla [2016] Mozilla.  A-painter, 2016.  https://blog.mozvr.com/a-painter/.
  * Neidle et al. [2012] Carol Neidle, Ashwin Thangali, and Stan Sclaroff.  Challenges in the development of the american sign language lexicon video dataset (asllvd) corpus.  05 2012.
  * Nichol et al. [2022] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022.
  * Oculus and Smoothstep [2016] Oculus and Smoothstep.  Quill, 2016.  https://quill.art/.
  * Oikonomidis et al. [2011] Iason Oikonomidis, Nikolaos Kyriazis, and Antonis Argyros.  Efficient model-based 3d tracking of hand articulations using kinect.  volume 1, 01 2011.  doi: 10.5244/C.25.101.
  * Pang et al. [2019] Kaiyue Pang, Ke Li, Yongxin Yang, Honggang Zhang, Timothy M. Hospedales, Tao Xiang, and Yi-Zhe Song.  Generalising fine-grained sketch-based image retrieval.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 677–686, 2019.  doi: 10.1109/CVPR.2019.00077.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * Raffel et al. [2023] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer, 2023.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents, 2022.
  * Ribeiro et al. [2020] Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, and Moacir Ponti.  Sketchformer: Transformer-based representation for sketched structure, 2020.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation, 2015.
  * Rosales et al. [2021] Enrique Rosales, Chrystiano Araújo, Jafet Rodriguez, Nicholas Vining, DongWook Yoon, and Alla Sheffer.  Adaptibrush: adaptive general and predictable vr ribbon brush.  _ACM Trans. Graph._ , 40(6), 2021.  doi: 10.1145/3478513.3480511.  URL https://doi.org/10.1145/3478513.3480511.
  * Rupp [2024] Michael Rupp.  Is it getting hot in here? the effects of vr headset microclimate temperature on perceived thermal discomfort, vr sickness, and skin temperature.  _Applied Ergonomics_ , 114, 2024.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding, 2022.
  * Sain et al. [2023] Aneeshan Sain, Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song.  Clip for all things zero-shot sketch-based image retrieval, fine-grained or not, 2023.
  * Sangkloy et al. [2016] Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays.  The sketchy database: learning to retrieve badly drawn bunnies.  _ACM Transactions on Graphics_ , 35:1–12, 07 2016.  doi: 10.1145/2897824.2925954.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In Francis Bach and David Blei, editors, _Proceedings of the 32nd International Conference on Machine Learning_ , volume 37 of _Proceedings of Machine Learning Research_ , pages 2256–2265, Lille, France, 07–09 Jul 2015. PMLR.  URL https://proceedings.mlr.press/v37/sohl-dickstein15.html.
  * Song et al. [2021] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=St1giarCHLP.
  * tecperson [2016] tecperson.  Sign language mnist: Drop-in replacement for mnist for hand gesture recognition tasks, 2016.  https://www.kaggle.com/datasets/datamunge/sign-language-mnist.
  * Vinker et al. [2022] Yael Vinker, Ehsan Pajouheshgar, Jessica Y. Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir.  Clipasso: Semantically-aware object sketching, 2022.
  * Wan et al. [2019] Chengde Wan, Thomas Probst, Luc Van Gool, and Angela Yao.  Self-supervised 3d hand pose estimation through training by fitting.  pages 10845–10854, 06 2019.  doi: 10.1109/CVPR.2019.01111.
  * Wang et al. [2015] Fang Wang, Le Kang, and Yi Li.  Sketch-based 3d shape retrieval using convolutional neural networks, 2015.
  * Wang et al. [2004] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ , 13(4):600–612, 2004.  doi: 10.1109/TIP.2003.819861.
  * Xie et al. [2023] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou.  Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion, 2023.
  * Yan et al. [2019] Yan Yan, Ke Chen, Yu Xie, Song Yiming, and Yonghong Liu.  _The Effects of Weight on Comfort of Virtual Reality Devices_ , pages 239–248.  01 2019.  ISBN 978-3-319-94705-1.  doi: 10.1007/978-3-319-94706-8_27.
  * Yang et al. [2022] Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.  Reco: Region-controlled text-to-image generation, 2022.
  * Zhang et al. [2020] Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, and Matthias Grundmann.  Mediapipe hands: On-device real-time hand tracking.  _CoRR_ , abs/2006.10214, 2020.  URL https://arxiv.org/abs/2006.10214.
  * Zhang et al. [2018a] Jingyi Zhang, Fumin Shen, Li Liu, Fan Zhu, Mengyang Yu, Ling Shao, Heng Tao Shen, and Luc Van Gool.  Generative domain-migration hashing for sketch-to-image retrieval.  In _Proceeding of European Conference on Computer Vision (ECCV)_ , 2018a.
  * Zhang et al. [2023a] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models, 2023a.
  * Zhang et al. [2018b] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _CVPR_ , 2018b.
  * Zhang et al. [2023b] Tianjun Zhang, Yi Zhang, Vibhav Vineet, Neel Joshi, and Xin Wang.  Controllable text-to-image generation with gpt-4, 2023b.

##  Appendix A Appendix

###  A.1 Details for Sketch Augmentations

#### Implementation.

Below we provide implementation details for each augmentation:

  * •

Local Augmentation includes 3 types of sub-augmentations: stroke distortion,
random spikes, and jitters. For stroke distortion, we generate a number of
wave functions each with random frequency, shift, and amplitude. We aggregate
them to form a random curve, whose value is used to distort the ground-truth
sketch. Spikes are implemented in two modes sharp spikes and smooth spikes.
Sharp spikes are determined by the width and height which are randomly sampled
from a normal distribution; smooth spikes are implemented with Bezier curve,
and the control points are randomly sampled from a uniform distribution within
predefined range along the gradient of the ground-truth sketch. Jitters are
simply small perturbations sampled from a pre-defined normal distribution and
added to random locations along the ground-truth sketch.

  * •

Structural Augmentation also include 3 types of sub-augmentations: sketch-wise
distortion & relocation, stroke-wise misplacement, and stroke-wise resize. For
sketch-wise distortion& relocation, we randomly shrink and change aspect ratio
of the whole sketch and reposition the sketch in the canvas. For stroke-wise
misplacement/resize, we randomly move/resize each stroke within a pre-defined
range.

  * •

False Strokes includes two types: transitional and random false strokes. For
transitional false strokes, we simply draw lines between the transition of
each stroke; for random false strokes, we randomly draw a number of extra
lines on the canvas.

In the above augmentations, misplacement and stroke resize under structural
augmentation are mutually-exclusively applied. This is because it is highly
probable that the combination of the two will completely destroy the visual
cues of the resulting sketch image. For all other augmentations, we set a 50%
chance for each to be applied to each sketch sample during training.

###  A.2 Training Details

#### Held-out categories.

Because different object categories have different drawing difficulty, sketch
and tracking samples belonging to different categories exhibit large variance
in statistics (CLIP Image-Text similarity between ground-truth sketch and
corresponding text, CLIP Image-Image similarity, Chamfer Distance, or SSIM
between ground-truth and tracking image). To allow for fair comparison of the
performance between seen and held-out categories, we first partition
categories into 10 clusters using K-Means Clustering, and randomly sample one
category from each cluster as the held-out categories. The held-out categories
are:

{car, face, cow, snail, diamond, candle, angel, cat, grapes, sun}

#### Training configurations.

All training is conducted on two Nvidia H100 GPUs. For finetuning the
diffusion model with LORA, we set LORA rank to 4, per device batch size to 16,
learning rate to $5\text{e}^{-5}$, gradient accumulation steps to 4, and train
for 6K steps on the Quick, Draw! dataset. For our augmentation-based
ControlNet training, we set per device batch size to 8, learning rate to
$2\text{e}^{-5}$, gradient accumulation steps to 4, and the proportion of
empty prompts to 25%.

###  A.3 Comparison on Egocentric Hand Tracking Algorithm

An example of hand landmarking with MediaPipe, OpenPose, and NRSM is shown in
Figure 9. We can see that only MediaPipe’s hand landmarker comparatively
accurate and consistent, while OpenPose often fails to detect landmarks, and
NSRM is less accurate. However, even though MediaPipe seems to accurately
predict hand landmarks, the resulting tracking image, as shown in Figure 1 and
Figure 12, still contains large amount of noise.

Figure 9: A comparison of hand landmarking done by MediaPipe, OpenPose, and
NSRM respectively. MediaPipe in general provides most accurate hand landmarks,
while OpenPose often struggling to detect the hand, and NSRM not able to
provide accurate landmarks.

###  A.4 Sketch Completion & Text-conditioned styling.

With simple augmentations, we can also perform other sketch-based tasks such
as sketch completion and text-conditioned styling. Unlike prior works [38, 37]
on sketch completion that require sophisticated architectures or training
procedures, we could achieve sketch completion by simply adding random erasure
to the set of augmentations during training. In Figure 10, we show two types
of sketch completion: completion with small missing line segments (column
1&5), and completion with whole strokes missing (column 3&7). In addition, as
shown in Figure 11, we can also utilize the text-conditioning capacity of DMs
to prompt for different stroke styles, such as different color and thickness,
therefore alleviating the need for fine-grained style control on the user’s
part.

Figure 10: Auto-completion given a partially-drawn sketch in two modes: line
segment missing (Column 1&5) and whole strokes missing (Column 3&7). Bordered
images are the generated results. Figure 11: Generation with text instructions
for different line colors and brush thickness. Bordered images are the
generated results. We use the prompt: “A sketch of a <category>, <color>
lines, <thickness> brush.”

###  A.5 Visual Comparison between Controllable DMs

In Figure 12 we compare the generated sketches between ControlNet and
T2IAdapter, when trained under our augmentation-based procedure. We can
observe that both ControlNet and T2IAdapter generate visually coherent
sketches and generally follow the input tracking image. Nonetheless, by
looking into details, ControlNet’s generations are more faithful to the visual
cues from the tracking images. For example, the butterfly (bottom right)
generated by ControlNet resembles the shapes of the wings from tracking image
closely, while the generation from T2IAdapter looks much less similar.

Figure 12: More inference results on ControlNet with our augmentation-based
training procedure. We additionally show inference results by using T2IAdapter
in our method.