# Hand-Object Interaction Reasoning

Jian Ma
University of Bristol
<EMAIL_ADDRESS>Dima Damen
University of Bristol
<EMAIL_ADDRESS>
###### Abstract

This paper proposes an interaction reasoning network for modelling spatio-
temporal relationships between hands and objects in video. The proposed
interaction unit utilises a Transformer module to reason about each acting
hand, and its spatio-temporal relation to the other hand as well as objects
being interacted with. We show that modelling two-handed interactions are
critical for action recognition in egocentric video, and demonstrate that by
using positionally-encoded trajectories, the network can better recognise
observed interactions. We evaluate our proposal on EPIC-KITCHENS and
Something-Else datasets, with an ablation study.

##  1 Introduction

Different from general actions (e.g. jumping), object interactions involve
actors influencing objects (e.g. playing an instrument or kicking a ball). Of
particular interest to this work is hand-object interactions (HOI) which
feature regularly in the activities of daily living. HOIs include one-handed
(e.g. “open drawer”) as well as two-handed interactions (e.g. “open bottle”),
and many interactions include tools that extend our hands’ abilities (e.g.
cutting a vegetable requires a knife). However, most video understanding
methods aim to recognise both actions and interactions alike as general video
datasets involve a mix of classes [4, 18]. Recently, a handful of large-scale
datasets [17, 7, 6] that focus on HOIs have fueled works that specifically
reason about interactions.

Recent progresses in interaction reasoning have been driven by the success of
object detectors, e.g. [31]. Due to the datasets used, previous works [35, 15,
29, 1] detect the person in the middle frame of the video or all the objects
that appear in the video. This means that trajectories of interactions are not
explicitly emphasised.

Figure 1: Illustration of Spatio-Temporal HOI reasoning for the action of
stirring food in the pan. Both hands play critical roles. The left hand
($H_{L}$, in blue) steadies the pan, which we refer to as the left object
($O_{L}$, in yellow) given its direct interaction with the left hand. The
right hand ($H_{R}$, in red) holds the wooden spoon ($O_{R}$, in green). While
$H_{L}$, $O_{L}$ are steady over time, the red and green trajectories
($H_{R},O_{R}$) demonstrate the stirring motion relative to the pan.

In this work, we focus on the interaction between hands and objects, through
their trajectories that encode motion and positions, thus discriminating
between different interactions (Fig. 1). We propose an Interaction Reasoning
Network (IRN) that jointly reasons about interactions between both hands and
active objects.

We propose encoders that learn hand-object and hand-hand interactions and
decoders that enrich this learned interaction with action representation
knowledge. We automatically detect hands and active objects, i.e. those with
which the hand interacts, using the approach from [32], and link these
detections over time to form trajectories with pooled features from a spatio-
temporal backbone. We then reason about pairwise interactions, distinguishing
left from right hand interactions. The proposed framework is trained end to
end.

Figure 2: Proposed Architecture for Interaction Reasoning Network (IRN). A
base 3D ConvNet extracts temporal-spatial features. These are ROI pooled based
on hand-object detections and combined with positional encoding to form
trajectories. The trajectories are fed into interaction unit to reason about
hand-object relations. A decoder combines the action representation and two-
hand encoders for classification (Fig. 4 for details) . $\circ$ is for
multiple outputs, $\oplus$ is for summation and _C_ for concatenation. Dashed
box for ‘Hand & Object Detector’ highlights frozen weights. Thicker lines
highlight backpropagation pathways.

Our contributions: (i) this paper proposes to separately reason about
interactions between the left and right hands with their corresponding objects
in HOI through an encoder-decoder transformer module; (ii) to the best of the
authors’ knowledge, the proposed method is the first work that uses the
trajectories of hands and objects to enrich the relational representation of
interactions; (iii) we showcase the importance of this proposed reasoning on
the large-scale egocentric HOI dataset EPIC-KITCHENS-100 [6] as well as on the
crowd-sourced HOI dataset Something-Else [26].

##  2 Related Work

Action Recognition. In recent years, the success of deep learning in computer
vision has promoted the rapid development of action recognition models. From
the 2D ConvNets [44, 8, 40, 45, 23] and multi-stream networks [33, 12, 4] to
3D ConvNets [21, 37, 11, 10] and transformer-based networks [27, 3], these
models have progressively improved the understanding of actions. Of relevance
to our model, SlowFast network [11] is a 3D convolutional model that can
combine the spatial features with temporal information from two rates of
sampling the input video. SlowFast results remain competitive particularly for
HOI datasets we analyse. These approaches form the base for interaction
reasoning which we review next.

Interaction Recognition. Recently, interest in actor-centric video
understanding employed increasingly reliable person detectors, to localise
actors in movies [18] or players in sports [34]. Many works [16, 5, 24, 39,
14, 13, 29, 43, 15, 26, 32, 46] utilise an object detector (_e.g._ Faster
R-CNN) to locate bounding boxes of human or objects for input into a
relational module. However, the detector is likely to yield object proposals
that are not relevant to the interaction due to the lack of annotations for
training. Instead, [16] learns to predict the location of related objects
based on the appearance of actors. Similarly, [5, 14] introduce pairwise
streams for interaction patterns to encode the spatial relative locations of
human and objects. Based on pairwise streams, human poses are considered for
modeling HOIs in [24, 39]. Additionally, Graph Convolutional Network (GCN) is
another way to explore image-based object interactions [46, 13].

Different from image-based methods, the motion is also particularly important
in video analysis. Given person detections from the middle frame of a video,
in [15], detections are pooled as a query to attend to the whole frame’s 3D
features, in a transformer encoder block. Longer-term reasoning is proposed in
[43] by learning contextual information through short-term person feature
banks and long-term feature banks from non-local blocks [41]. Apart from the
contextual interaction in the temporal dimension, [1] study relational
interactions between objects via training a Gate Recurrent Unit with pairing
current and previous frames. Similarly, [42] regard video as a graph of
objects, conducting interaction recognition reasoning on a graph network.
Moreover, [26] generalise the performance to unseen actions, decomposing each
action into a verb, subject, and one or more goals and proposing the
Something-Else dataset for exploring hand-object interactions. Detections over
time are combined through the Hungarian algorithm to track the detected
persons and objects. However, detected objects may not be part of the
interaction, which introduces noise in all but the simplest scenes. Besides,
in [26], additional expensive-to-collect annotations of objects are required
to train the model.

IRN is inspired by [1, 15], but we focus on hands as actors, thus requiring to
model two-hand interactions including hand-to-hand interactions as well as
hand-to-object interactions. Different from [26], we do not require additional
annotations and instead use automatic detection of hands and active objects
from [32]. This allows us to reason about interactions in busy scenes
efficiently. We detail our method next.

##  3 Methodology

In this section, we firstly give an overall of the Interaction Reasoning
Network (IRN) architecture (visualised in Fig. 2). We then discuss detections
and the spatial position encoder followed by details of our proposed
interaction unit (IU).

IRN captures the relationships between hands and active (i.e. action-relevant)
objects. We model interactions through i) encoders that focus on hand-object
and hand-hand interaction; and ii) decoders that enrich the action
representation, trained jointly. We formulate the interaction unit as:

$\displaystyle E$ $\displaystyle=e(H,O,P;\omega_{e}),$ (1) $\displaystyle I$
$\displaystyle=d(F,E;\omega_{d}),$

where $\omega_{e}$ and $\omega_{d}$ represent the parameters of encoders $e()$
and decoders $d()$. The encoder aims to model the hand $H$ and interacting
object $O$ representations along with their spatial position encoding $P$. The
encoder’s output $E$ and global action representation $F$ are fed into
decoders.

Figure 3: Spatial Position Encoder. Detections, per object, over a sequence
are represented by binary maps. A 3-layer 3D ConvNet produces per-frame
spatial encoding.

###  3.1 Detections and Positional Encoding

We use a pretrained hand and object detector, which we run per frame, and
distinguish four bounding box detections of
$\left\\{b_{H_{L}},b_{H_{R}},b_{O_{L}},b_{O_{R}}\right\\}$ where $H/O$ mean
hand and object, $L/R$ denote the side, left or right. We pool a 3D ConvNet to
extract frame-level hands and objects features, using RoI average pooling as
in [1, 15]. Given a layer in the backbone with $C$ channels, we extract a
feature $a$ per detection of size $\mathbb{R}^{C}$ by RoI pooling, of which we
have $\left\\{a_{H_{L}},a_{H_{R}},a_{O_{L}},a_{O_{R}}\right\\}$.

Figure 4: The Architecture of proposed Interaction Unit (IU). This module
includes encoders and decoders. Encoders reason about hand-object interactions
by multi-head transformers with hands as query and objects as key and value.
Encoders outputs from left and right hands are concatenated and fed to
decoders. Decoders model the interaction with action representation.

In addition to the features, the absolute positions of hands and objects offer
significant information to distinguish interactions. We thus propose to use
spatial positional representation $p$ for each detection, such that
$\left\\{p_{H_{L}},p_{H_{R}},p_{O_{L}},p_{O_{R}}\right\\}$. Inspired by [20],
we consider a binary map for each detection $b$. These show the absolute
position and scale of an object in the image. We learn position (and scale)
encodings from this binary input and use the consecutive binary maps to
capture the positional/scale changes. Note that while we use local 3D
convolution to model motion, we have the position encoding per frame to
correspond to the bounding box $b$. We learn a weights-shared 3D ConvNet as
shown in Fig. 3.

As interactions focus on the temporal evolution of the relationships between
hands and objects, we combine features and their positional encodings to form
trajectories. We use standard summation but also ablate this against
concatenation. We represent each trajectory
${H=(a^{1}+p^{1},\cdots,a^{T}+p^{T})}$ over $T$ frames, and similarly for
objects. We thus have four trajectories
$\left\\{H_{L},H_{R},O_{L},O_{R}\right\\}$, which form the input for our
interaction unit. When hands or objects are not present in a frame or are not
detected, we set the feature $a$ to 0 and the binary map to blank. We showcase
experimentally that our method can recover from missed detections.
Additionally, some trajectories might be missing altogether. We consider the
presence as well as the absence of hands and objects as informative evidence
for interaction reasoning.

We next describe how we can reason about interactions using these
trajectories.

###  3.2 Interaction Unit

In order to reason about HOIs, we consider a pair of trajectories ($H$, $O$)
where $H$ is the actor - a left/right hand and $O$ is the object with which
the hand interacts. Importantly $O$ can be the other hand or an object. As
Fig. 4 demonstrates, we have up to 3 interactions per actor/hand. For the left
hand, these would be $(H_{L},O_{L})$, $(H_{L},O_{R})$ and $(H_{L},H_{R})$. We
intuitively describe these using the example from Fig. 1 on stirring food in
the pan.

  * •

$(H_{L},O_{L})$ captures the left hand holding the pan. The hand gesture and
relative positions of hand and pan would be captured using this interaction.

  * •

$(H_{L},O_{R})$ captures the left hand versus the spoon as it stirs through
the food in the pan held by the left hand.

  * •

$(H_{L},H_{R})$ captures the absolute positions and gestures of both hands as
one holds the pan and the other stirs the food.

We first describe our encoders. These are concatenated and passed to stacked
decoders described after.

Interaction Encoder. We explain the encoder for one interaction pair namely
$(H_{L},O_{L})$.

$\displaystyle Q_{H_{L},O_{L}}$ $\displaystyle=q(H_{L}),$ (2) $\displaystyle
K_{H_{L},O_{L}}$ $\displaystyle=k(O_{L}),$ $\displaystyle V_{H_{L},O_{L}}$
$\displaystyle=v(O_{L}),$

where $q,k,v$ all linearly project $\mathbb{R}^{T\times C}$ input to
$\mathbb{R}^{N}$. Fig. 4 shows the module of left-hand interactions. The
encoder is then a residual attention unit:

$E^{{}^{\prime}}_{H_{L},O_{L}}=\sigma\left(\frac{Q_{H_{L},O_{L}}K_{H_{L},O_{L}}}{\sqrt{N}}\right)V_{H_{L},O_{L}}+Q_{H_{L},O_{L}},$
(3)

where $E^{{}^{\prime}}_{H_{L},O_{L}}$ is interaction representation between
actor $H_{L}$ and object $O_{L}$ and $\sigma$ is the softmax operator.
Following [38], we add a linear feedforward network $FFN()$:

$E^{{}^{\prime\prime}}_{H_{L},O_{L}}=\delta(FFN(E^{{}^{\prime}}_{H_{L},O_{L}}))+E^{{}^{\prime}}_{H_{L},O_{L}},$
(4)

where $\delta$ is the dropout operation and
$E^{{}^{\prime\prime}}_{H_{L},O_{L}}\in\mathbb{R}^{N}$ is the encoded
representation for $(H_{L},O_{L})$.

Similarly, $E^{{}^{\prime\prime}}_{H_{L},O_{R}}$ and
$E^{{}^{\prime\prime}}_{H_{L},H_{R}}$ are computed for the left hand (see Fig.
4 Left Encoder) as well as three encoders for $E^{{}^{\prime\prime}}_{H_{R}}$.
We concatenate all outputs to form overall encoding
${\mathbf{E^{{}^{\prime\prime}}}\in\mathbb{R}^{6N}}$. Subsequently, the
dimension of $\mathbf{E}$ is reduced from 6N to M. We set M to the size of
action representation $F$ by a linear projection.

Interaction Decoders. Having reasoned about all pairwise interactions using
the encoder, we use $\mathbf{E}$ to enrich the action representation, as in
[15]. The pipeline of the decoder is similar to the encoder. Specifically, the
features $\mathbf{E}$ from the encoder is projected to key
$K^{{}^{\prime}}\in\mathbb{R}^{M}$ and value
$V^{{}^{\prime}}\in\mathbb{R}^{M}$. Different from the initial transformer
[38], we remove the self attention from the decoder due to action
representation features $F\in\mathbb{R}^{M}$ pooled on temporal dimension at
backbone features. Instead, we directly map $F$ linearly to
$Q^{{}^{\prime}}\in\mathbb{R}^{M}$. Similarly, we adopt dropout, a feedforward
network as well as residual connections, like in Eq. 4, to learn decoder’s
output $I\in\mathbb{R}^{M}$.

At last, we stack the multi-head interaction encoders and decoders. The output
$I$ is fed to a fully connected layer, and trained to classify actions using
standard cross entropy loss. During tracking, we backpropagate through all
components of the IRN including IU and convolutional positional encoding, as
well as the base network. The weights of the detector remain frozen.

##  4 Experiments and Results

In this section, we experimentally evaluate the model on two datasets
featuring hand-object interactions from egocentric and crowd sourced videos
respectively.

###  4.1 Datasets, Evaluation and Implementation Details

EPIC-KITCHENS-100 [6] is the largest video dataset in egocentric vision. We
report on the full dataset but the conduct ablation study on a subset of the
validation set. The subset was collected by the first participant (P01) and
contains 5,509 / 885 action segments for the training and validation set,
respectively. We use a fixed random seed and a single run during ablations to
ensure results are directly comparable.

Something-Else [26] is based on [17] proposing a new split for novel verb-noun
combinations in the test set. Active objects have been densely annotated for
training by [26] – however hands are not annotated for side L/R and
accordingly the active objects are not associated with a hand. We use this
proposed split without the spatial annotations, instead using automatic,
potentially noisy, detections.

Evaluation We use evaluation metrics proposed for both dataset - Top-1/5
accuracy for verb, noun and action classes. For Something-Else, we use a
single output classifier.

Implementation Details. The hand-object detector [32] aims to find links
between hands and interacting objects by optimising an offset vector. It was
trained on 100 days dataset that contains both first-person and third-person
videos. We use the publicly available trained weights, and detect frame-level
hands and objects with the confidence threshold of 0.5.

We use SlowFast [11] R-50 8×8 due to its performance on both datasets. The
input videos are 32-frame clips, where we sample $T=8$ frames with a temporal
stride $\tau=8$ for the slow pathway, and 32 frames for the fast pathway. We
pool hands and objects features $C=640$ based on the third block of SlowFast
as when the layer goes deeper, the features get sparser. We set $N=5120$ used
in the encoder and $M=2304$ so that it matches the the globally pooled feature
size of the last block of SlowFast. For the interaction unit, 16 heads are
used in the multi-head attention and 3-layer encoders and decoders are
implemented.

For EPIC-KITCHENS-100, the backbone is pre-trained on the training set and
$e()$ and $d()$ are trained from random initialisation. We train for 24 epochs
with learning rate 0.001 using SGD with 0.0001 weight decay and 0.9 momentum,
the learning rate is decayed by the factor of 10 at epochs 10 and 20. For
Something-Else the backbone is pretrained on Kinetics-400 [22]. Similarly, the
network was optimised by SGD with initial learning rate 0.01 dropped at epochs
12 and 18, 0.0001 weight decay, 0.9 momentum for 20 epochs.

Figure 5: Visualisation of STD and SCR for data augmentation. The action is
“take plate”.

It is important to note that we changed the random cropping typically used for
data augmentation. This is because randomly cropping the image in EPIC-
KITCHENS-100 results in hands and objects being frequently cropped out of the
main frame. As we show in Fig. 5, the standard STD crops the right hand and
plate out. This significantly harms the interaction unit. In egocentric
footage, hands can be in the bottom half of the image or towards a corner. To
avoid this while maintaining the power of augmentation, we first randomly
scale (S) frames to ${H}^{\prime}\times{W}^{\prime}$, then crop (C) frames to
${H}^{\prime}\times{H}^{\prime}$ and finally resize (R) frames to the target
resolution $224\times 224$. We refer to this augmentation approach as SCR,
which we use in all our experiments on EPIC-KITCHENS-100. The two data
augmentations perform comparably on Something-Else dataset as the action is
typically at the centre of the image. We thus use the standard random cropping
data augmentation for this dataset.

###  4.2 Something-Else Dataset Results

Tab. 1 shows the performance of our method outperforms SotA reasoning
approaches STIN, STRG and CAF. Specifically, we outperform STIN Combined with
I3D, when trained jointly or separately. Notably, our class-agnostic IRN is
superior to methods using object labels among non-ensemble models. We also
report ensemble methods. These metods are not directly comparable but our
model remains competitive.

Method | Ens. | Obj. | Top-1 | Top-5
---|---|---|---|---
STIN [26] | ✗ | ✓ | 37.2 | 62.4
I3D+STIN [26] | ✗ | ✓ | 48.2 | 72.6
CAF [30] | ✗ | ✓ | 52.3 | 78.9
STRG [42] | ✗ | ✗ | 52.3 | 78.3
IRN (Ours) | ✗ | ✗ | 52.9 | 80.8
I3D-STIN [26] | ✓ | ✓ | 51.5 | 77.1
STRG-STIN [26] | ✓ | ✓ | 56.2 | 81.3
CACNF [30] | ✓ | ✓ | 56.9 | 82.5

Table 1: Results on Something-Else Datasets. +: jointly trained. -: trained
separately. Ens.: Ensemble. Obj.: use manual object labels.

Importantly, we showcase that our model particularly benefits from the usage
of a SlowFast backbone. We evaluate our interaction units with different
backbones in the Tab. 2. We get the best performance when combining SlowFast
as a base network with our interaction unit, but also report improvement on
the I3D backbone.

I3D | SlowFast | $IRN$ | Top-1 | Top-5
---|---|---|---|---
✓ |  |  | 46.8 | 72.2
✓ |  | ✓ | 47.5 | 73.8
| ✓ |  | 52.2 | 80.3
| ✓ | ✓ | 52.9 | 80.8

Table 2: Results of different backbones on Something-Else.

In the Something-Else dataset, we find that only 39.63% of clips have both
left and right hands, while 50.11% and 10.26% of clips have only one hand or
no hands respectively. When no actor/hand is detected, our interaction unit is
likely to struggle. Despite this, our proposed method can enrich the
representation for interaction reasoning.

###  4.3 EPIC-KITCHENS-100 Dataset Results

As this dataset is egocentric, i.e. participants are using a wearable camera,
more actions involve both hands making it more suitable to assess our
proposal. To manage the size, we conduct an ablation on a selected
subset111We’ll make the subset available for direct comparisons.

Interaction Components. To evaluate the contribution of the various encoding
interactions, we ablate the results by removing one at a time, as well as
left/right hand encoders in Tab. 3. We first note that removing right hand
interaction encoders (row 6) results in a larger drop than left hand (row 2).
This is anticipated with most participants being right-handed. Similarly, the
largest drop is associated with removing the encoder of $H_{R},O_{R}$, which
is critical for one-handed interactions (row 7) followed by $H_{R},H_{L}$ (row
9) which is critical for hand-only interactions (e.g. wash hands). The encoder
with the least contribution is that for $H_{R},O_{L}$ (row 8), as it is
probably compensated by the other pairwise encoders.

Figure 6: Qualitative results of correctly-recognised interactions from EPIC-
KITCHENS-100 (col 1) and Something-Else (col 2). $L/R$ indicate the side and
$H/O$ denote hands and objects. $GT$ and $Pred$ are Ground Truth and
Prediction.

Trajectory. We evaluate the importance of using trajectories. Previous work
[15] only uses one actor (person) of the middle frame ($middle$) along with
video action representation and a recent work [29] detects persons in the
middle frame and then duplicate these detections across time ($duplicate$). We
compare these options to our proposed approach that encodes object
trajectories as well as two-handed interactions. The best performance is
achieved when using the complete trajectory of detections, including both
hands and objects. For more details, please refer to the supplementary.

$(H_{L},O_{L})$ | $(H_{L},O_{R})$ | $(H_{L},H_{R})$ | $(H_{R},O_{R})$ | $(H_{R},O_{L})$ | $(H_{R},H_{L})$ | Top-1
---|---|---|---|---|---|---
✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 42.37
✗ | ✗ | ✗ | ✓ | ✓ | ✓ | 43.28
✗ | ✓ | ✓ | ✓ | ✓ | ✓ | 42.60
✓ | ✗ | ✓ | ✓ | ✓ | ✓ | 43.28
✓ | ✓ | ✗ | ✓ | ✓ | ✓ | 42.94
✓ | ✓ | ✓ | ✗ | ✗ | ✗ | 42.82
✓ | ✓ | ✓ | ✗ | ✓ | ✓ | 41.47
✓ | ✓ | ✓ | ✓ | ✗ | ✓ | 44.07
✓ | ✓ | ✓ | ✓ | ✓ | ✗ | 41.69
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 44.52

Table 3: Ablation study for Interaction Components. Det. | $H_{R}$ | Act. Rep. | $H_{L}$ | Objects | Top-1
---|---|---|---|---|---
middle [15] | ✓ | ✓ |  |  | 43.05
duplicate [29] | ✓ | ✓ |  |  | 44.07
trajectory | ✓ | ✓ |  |  | 42.37
trajectory | ✓ | ✓ |  | ✓ | 43.28
trajectory | ✓ | ✓ | ✓ | ✓ | 44.52
Table 4: Comparison to prior works without trajectory representations as well
as active object trajectories.

SPE | Top-1
---|---
none | 43.39
concat | 42.71
sum | 44.52
Table 5: Ablation study for Spatial Position Encoding.

Act.Rep. | Top-1
---|---
none | 31.98
concat | 43.73
decoder | 44.52
Table 6: Ablation study for Action Representation.

Det.Rep | Top-1
---|---
MLP | 43.95
CNN | 44.52
Table 7: Ablation study for Detection Representation.

Spatial Positional Encoding. We adopt two ways to fuse positional information
with visual features of hands or objects. As we expected in Tab. 7, spatial
position encoding $SPE$ improves the performance. Compared to the network
without $SPE$ ($none$ in Tab. 7) or concatenations of $SPE$. This is due to
the sparsity of positional features. Concatenating lots of zeros to the visual
features may introduce noise. Our proposed approach to $sum$ the positional
encoding yields the best action performance.

Action Representation. In this ablation study, we evaluate the importance of
globally pooled action representation. First, We remove decoders and train the
network with only interaction features, i.e. $none$. Tab. 7 shows that results
drop significantly if the network does not use the globally-pooled action
representation. We also concatenate the hand-object interaction features with
action representation, in a late-fusion fashion. What stands out in the Tab. 7
is that the Top-1 action result of the decoder surpasses concatenation.

Detection Representation. In all other experiments, we have used RoI pooling
to get feature representations of our hands’ and objects’ detections. Instead,
inspired by popular MLPs [36], we also explore using MLP projection directly
from image patches. We crop the input image given bounding boxes $b$ and
resize to a fixed size by bilinear interpolation. The fixed size patches are
flatten, and projected using an MLP. Hence, representations of any detections
can be learned by a parameter-shared MLP. Notably, these features only contain
the spatial information, without any motion cues as they are projected per
image. We compare performances of MLPs and pooled 3D-CNN as the features
extractors. Tab. 7 shows that the accuracy of 3D-CNN pooling is superior to
MLP projection, indicating that temporal features are significant for video-
based hand-to-object interactions.

Detections Feature Representation. In all other experiments, we have used RoI
pooling to get feature representations of our detections. As presented in Sec
3, we also explore using MLP projection. Inspired by popular MLPs [36], we
compare performances of MLPs and ConvNets as to get the visual features of our
detections. Tab. 7 shows that MLP has better modelling capabilities than
ConvNets only for verb recognition. However, ConvNets are more advantageous
for noun recognition and their accuracy is far better than MLPs by about 2.7%.
Hence, ConvNets are more accurate in action recognition. All previous and
subsequent results thus use 3D ConvNet RoI pooling to represent the visual
features of hand and object detections.

|  | Top-1 Accuracy (%)
---|---|---
Method | DataAug. | Verb | Noun | Act.
Chance [6] | STD | 10.42 | 1.70 | 0.51
IRN (Ours) | STD | 60.94 | 43.97 | 31.97
TSN [6] | STD | 60.18 | 46.03 | 33.19
SlowFast [6] | SCR | 63.64 | 48.58 | 36.76
IRN (Ours) | SCR | 63.68 | 48.94 | 37.11

Table 8: Quantitative results on Validation set. STD and SCR denote standard
and our data augmentation strategies.

We next report results on the validation set of EPIC-KITCHENS-100. In Tab. 8,
our method $IRN$ improves over published baselines. Adding the interaction
unit to SlowFast improves results for verb, noun and action top-1 accuracy by
0.04%, 0.36% and 0.35%, respectively. We explain the data augmentation
strategies in supplementary, where STD performs random cropping often cropping
the hands out of the image. Comparing row 2 and row 5, we demonstrate that the
random cropping harms the performance of IRN.

###  4.4 Qualitative Results

Figure 7: Failure cases of Something-Else (row 1) and EPIC-KITCHENS-100 (rows
2, 3).

We demonstrate the results on clips containing only one hand, or both hands in
Fig. 6. It illustrates correct action recognition examples from both datasets.
It is also important to highlight that our method recovers from partial
failures in automatic detections of hands and objects, as in rows 3 and 4. IRN
is robust to two types of detection errors: (a) incorrect active objects
detection; (b) incorrect side of hand and objects. Specifically, row 3
illustrates “chop onion” and “pull two ends of something so that it gets
stretched”. The onion is not detected in several frames due to occlusion.
Similarly. the knife and rubber are missed in several frames along the
trajectory. For row 4 (col 1) the drawer switches between being a left-object
or a right object, while in row 4 (col 2) the hand sides are swapped in
various frames. Both errors (a) and (b) have little impact on our method, due
to our usage of trajectories and the attention mechanism that selects the
relevant hand and object representations to reason about interactions.

Moreover, we show failure cases in Fig. 7. The main reason for failure is
undetected or unobserved hands throughout the videos (row 1). In both
examples, the hands are not visible throughout. Row 2 (col 1) shows that our
focus on both hands might result in detecting another concurrent action,
paying more attention to the object in the other hand (salt). In row 2 (col 2)
the method fails to recognise the interaction with small movement. In row 3,
the less frequent action of replacing the bin bag is incorrectly mistaken as
the frequent throwing action. A clear limitation of our approach is evident in
row 3 (col 2) where the person is checking the time on their hand watch.
Evidently, the watch is never recognised as the interacting object, as it is
always part of the hand detection.

##  5 Conclusion

In this paper, we present a framework for hand-object interaction reasoning
that separately attends to actors (hands) and interacting objects, through
encoders, as well as action representation which includes contextual
information through a decoder. We present results on two hand-object
interaction datasets, demonstrating generality and competitive performance,
with an ablation study.

Acknowledgements. This work used public datasets and is supported by EPSRC
UMPIRE (EP/T004991/1).

## References

  * [1] Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori.  Object level visual reasoning in videos.  In ECCV, 2018.
  * [2] Iz Beltagy, Matthew E Peters, and Arman Cohan.  Longformer: The long-document transformer.  arXiv preprint arXiv:2004.05150, 2020.
  * [3] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In ICML, 2021.
  * [4] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In CVPR, pages 6299–6308, 2017.
  * [5] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng.  Learning to detect human-object interactions.  In WACV, pages 381–389, 2018.
  * [6] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Rescaling egocentric vision.  CoRR, abs/2006.13256, 2020.
  * [7] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Scaling egocentric vision: The epic-kitchens dataset.  In ECCV, 2018.
  * [8] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell.  Long-term recurrent convolutional networks for visual recognition and description.  In CVPR, pages 2625–2634, 2015.
  * [9] Haoqi Fan, Yanghao Li, Bo Xiong, Wan-Yen Lo, and Christoph Feichtenhofer.  Pyslowfast.  https://github.com/facebookresearch/slowfast, 2020.
  * [10] Christoph Feichtenhofer.  X3d: Expanding architectures for efficient video recognition.  In CVPR, June 2020.
  * [11] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In ICCV, pages 6202–6211, 2019.
  * [12] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.  Convolutional two-stream network fusion for video action recognition.  In CVPR, pages 1933–1941, 2016.
  * [13] Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang.  Drg: Dual relation graph for human-object interaction detection.  In ECCV, pages 696–712, 2020.
  * [14] Chen Gao, Yuliang Zou, and Jia-Bin Huang.  ican: Instance-centric attention network for human-object interaction detection.  In BMVC, 2018.
  * [15] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman.  Video action transformer network.  In CVPR, pages 244–253, 2019.
  * [16] Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He.  Detecting and recognizing human-object interactions.  In CVPR, pages 8359–8367, 2018.
  * [17] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al.  The” something something” video database for learning and evaluating visual common sense.  In ICCV, pages 5842–5850, 2017.
  * [18] Chunhui Gu, Chen Sun, David A. Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik.  Ava: A video dataset of spatio-temporally localized atomic visual actions.  In CVPR, June 2018.
  * [19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, pages 2961–2969, 2017.
  * [20] Md Amirul Islam, Sen Jia, and Neil DB Bruce.  How much position information do convolutional neural networks encode?  In ICLR, 2020.
  * [21] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu.  3d convolutional neural networks for human action recognition.  TPAMI, 35(1):221–231, 2012.
  * [22] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950, 2017.
  * [23] Chao Li, Qiaoyong Zhong, Di Xie, and Shiliang Pu.  Collaborative spatiotemporal feature learning for video action recognition.  In CVPR, pages 7872–7881, 2019.
  * [24] Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu.  Transferable interactiveness knowledge for human-object interaction detection.  In CVPR, pages 3585–3594, 2019.
  * [25] Ji Lin, Chuang Gan, and Song Han.  Tsm: Temporal shift module for efficient video understanding.  In IOC], pages 7083–7093, 2019.
  * [26] Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell.  Something-else: Compositional action recognition with spatial-temporal interaction networks.  In CVPR, pages 1049–1059, 2020.
  * [27] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann.  Video transformer network.  arXiv preprint arXiv:2102.00719, 2021.
  * [28] Junting Pan, Siyu Chen, Mike Zheng Shou, Yu Liu, Jing Shao, and Hongsheng Li.  Actor-context-actor relation network for spatio-temporal action localization.  In CVPR, pages 464–474, 2021.
  * [29] Junting Pan, Siyu Chen, Zheng Shou, Jing Shao, and Hongsheng Li.  Actor-context-actor relation network for spatio-temporal action localization.  2020\.
  * [30] Gorjan Radevski, Marie-Francine Moens, and Tinne Tuytelaars.  Revisiting spatio-temporal layouts for compositional action recognition.  arXiv preprint arXiv:2111.01936, 2021.
  * [31] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In NeurIPS, pages 91–99, 2015.
  * [32] Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey.  Understanding human hands in contact at internet scale.  In CVPR, 2020.
  * [33] Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  In NeurIPS, 2014.
  * [34] Khurram Soomro and Amir R Zamir.  Action recognition in realistic sports videos.  In the Computer Vision in Sports, pages 181–208. 2014.
  * [35] Chen Sun, Abhinav Shrivastava, Carl Vondrick, Kevin Murphy, Rahul Sukthankar, and Cordelia Schmid.  Actor-centric relation network.  In ECCV, pages 318–334, 2018.
  * [36] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al.  Mlp-mixer: An all-mlp architecture for vision.  arXiv preprint arXiv:2105.01601, 2021.
  * [37] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In ICCV, pages 4489–4497, 2015.
  * [38] A Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [39] Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He.  Pose-aware multi-level feature network for human object interaction detection.  In ICCV, pages 9469–9478, 2019.
  * [40] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In ECCV, pages 20–36, 2016.
  * [41] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In CVPR, pages 7794–7803, 2018.
  * [42] Xiaolong Wang and Abhinav Gupta.  Videos as space-time region graphs.  In ECCV, pages 399–417, 2018.
  * [43] Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krähenbühl, and Ross Girshick.  Long-Term Feature Banks for Detailed Video Understanding.  In CVPR, 2019.
  * [44] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici.  Beyond short snippets: Deep networks for video classification.  In CVPR, pages 4694–4702, 2015.
  * [45] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba.  Temporal relational reasoning in videos.  In ECCV, pages 803–818, 2018.
  * [46] Penghao Zhou and Mingmin Chi.  Relation parsing neural network for human-object interaction detection.  In ICCV, pages 843–851, 2019.


# VectorPainter: A Novel Approach to Stylized Vector Graphics Synthesis with
Vectorized Strokes

Juncheng Hu  Beihang UniversityBeijingChina<EMAIL_ADDRESS>,  Ximing
Xing  Beihang UniversityBeijingChina<EMAIL_ADDRESS>,  Zhengqi Zhang
Beihang UniversityBeijingChina<EMAIL_ADDRESS>,  Jing Zhang  Beihang
UniversityBeijingChina<EMAIL_ADDRESS>and  Qian Yu  Beihang
UniversityBeijingChina<EMAIL_ADDRESS>
###### Abstract.

We propose a novel method, VectorPainter, for the task of stylized vector
graphics synthesis. Given a text prompt and a reference style image,
VectorPainter generates a vector graphic that aligns in content with the text
prompt and remains faithful in style to the reference image. We recognize that
the key to this task lies in fully leveraging the intrinsic properties of
vector graphics. Innovatively, we conceptualize the stylization process as the
rearrangement of vectorized strokes extracted from the reference image.
VectorPainter employs an optimization-based pipeline. It begins by extracting
vectorized strokes from the reference image, which are then used to initialize
the synthesis process. To ensure fidelity to the reference style, a novel
style preservation loss is introduced. Extensive experiments have been
conducted to demonstrate that our method is capable of aligning with the text
description while remaining faithful to the reference image.

Stylized Vector Graphics Synthesis, Vector Line Art Generation, Diffusion
Model, Image Vectorization

††conference: Make sure to enter the correct conference title from your rights
confirmation email; October 28 - November 1, 2024; Melbourne,
Australia.††submissionid: 5488††ccs: Computing methodologies Computer
graphics††ccs: Computing methodologies Visual inspection††ccs: Computing
methodologies Machine learning algorithms††ccs: Computing methodologies
Parametric curve and surface models††ccs: Computing methodologies
Rasterization Figure 1. Stylized vector graphics synthesized by our
VectorPainter.

##  1\. Introduction

”Every artist dips his brush in his soul and paints his own nature into his
paintings.” -Henry Ward Beecher

In recent years, there has been a growing number of studies focused on SVG
synthesis (Ma et al., 2022; Frans et al., 2022; Schaldenbrand et al., 2022;
Mirowski et al., 2022; Vinker et al., 2022, 2023; Jain et al., 2023; Xing et
al., 2024, 2023) due to the superior compatibility of vector graphics with
visual design applications compared to raster images. Particularly driven by
advancements in text-to-image (T2I) models (Rombach et al., 2022), recent
works such as VectorFusion (Jain et al., 2023) and SVGDreamer (Xing et al.,
2023) have explored the task of text-guided SVG synthesis by integrating T2I
models with a differentiable rasterizer (Li et al., 2020). Despite these
advancements, a persistent challenge remains: it is difficult to precisely
control the style of generated vector graphics when relying solely on textual
descriptions.

Figure 2. Motivation of our method. (a) Artists have different stroke
preferences. Patches highlighted in the same color exhibit a similar stroke
style. (b) A Tangram puzzle can form different objects using a set of basic
elements. This motivates us to treat the stylization process as rearranging
strokes extracted from the reference image.

Using reference images has shown to be an effective method for achieving
precise control over the style of synthesized images. However, research on
stylized vector graphics generation remains limited. The sole existing study
in this domain, StyleCLIPDraw (Schaldenbrand et al., 2022), marks the first
attempt to generate stylized vector graphics based on both a text prompt and a
reference image. Nevertheless, this approach primarily adheres to the style
transfer pipeline used for raster images. During the generation process, the
synthesized vector graphics are rendered and their style is compared with
reference images in the raster image space. Unfortunately, this method often
yields less than satisfactory results.

We recognize that the key to generating high-quality stylized vector graphics
lies in fully leveraging the intrinsic properties of vector graphics. First,
SVGs utilize primitives such as Bézier curves and colors to depict images,
aligning perfectly with the nature of artistic paintings which are generally
composed of individual strokes. Each stroke in a painting can be effectively
represented as a vector primitive. Second, artists often employ distinctive
stroke styles in their work. For example, as illustrated in Fig. 2(a), Vincent
van Gogh favored short strokes, whereas Edvard Munch preferred longer ones.
This diversity in stroke styles inspires the use of stroke information to
characterize and replicate artistic styles effectively. Third, the stylization
process can be conceptualized as rearranging the strokes of a style image to
create new content. This concept is analogous to the way Tangram puzzles use a
fixed set of shapes to form various objects, as depicted in Fig. 2(b).
Similarly, if a reference image is deconstructed into a collection of vector
strokes, these strokes can serve as foundational elements for crafting new
content. However, these properties have not been explored in existing works.

We propose a novel framework, termed VectorPainter, for generating stylized
vector graphics. Given a text prompt and a style reference image,
VectorPainter is capable of producing a vector graphic whose content aligns
with the text and whose style remains faithful to the reference image. Our
model comprises two main components: style stroke extraction and stylized SVG
generation. Firstly, we introduce a new method to extract a set of vectorized
strokes from the reference image. Drawing inspiration from Berger et
al.(Berger et al., 2013) and recent advancements in stroke-based
rendering(Kotovenko et al., 2021; Zou et al., 2021; Nolte et al., 2022; Hu et
al., 2023), our technique identifies local color features and the
directionality of strokes to facilitate precise stroke extraction. Secondly,
we follow the prior works (Jain et al., 2023; Xing et al., 2023) to adopt an
optimization-based pipeline by combining a T2I model, i.e., LDM(Rombach et
al., 2022), and the differentiable rasterizer (Li et al., 2020). Unlike the
aforementioned models, VectorPainter utilizes the extracted strokes as the
initial step in the generation process. During this process, strokes—including
their positions and colors—are rearranged to create new content as specified
by the text prompt, entirely within the vector space. Furthermore, to preserve
the style of the extracted strokes throughout the optimization process, we
introduce a style preservation loss. This loss function is designed to
maintain the shape integrity of the strokes in the synthesized images.
Additionally, we incorporate the Score Distillation Sampling (SDS) (Jain et
al., 2023; Xing et al., 2024) and JVSP losses (Xing et al., 2024) to further
enhance the quality of the synthesized images.

In summary, the contributions of this work are threefold:

  * •

We introduce a new model, VectorPainter, which innovatively conceptualizes the
task of stylized vector graphics generation as a process of rearranging
reference strokes. This approach allows for greater fidelity in style
replication and content generation.

  * •

We propose a novel algorithm for extracting a set of vectorized strokes from
the reference image. These strokes serve as the basic elements for forming new
content. Additionally, we introduce a style preservation loss to maintain the
integrity of stroke shapes throughout the generation process.

  * •

We conducted extensive experiments to assess the effectiveness of our model.
The results demonstrate the superiority of VectorPainter in producing high-
quality stylized SVGs.

##  2\. Related Work

###  2.1. Vector Graphics Synthesis

Scalable Vector Graphics (SVGs) comprise essential components such as Bézier
curves, lines, shapes, and colors to represent images. The latest technique
for generating SVGs involves using a Differentiable Rasterizer called DiffVG
(Li et al., 2020). DiffVG bridges the gap between vector graphics and raster
image spaces. Unlike traditional vector graphics synthesis methods, which
require access to vector graphics datasets, DiffVG allows the generation of
vector graphics without the need for any such datasets.

Numerous studies have utilized visual text embedding CLIP (Radford et al.,
2021) (Contrastive Language-Image Pre-training) model to supervise the
synthesis of SVGs. Some notable examples of these studies include CLIPDraw
(Frans et al., 2022), StyleCLIPDraw (Schaldenbrand et al., 2022), CLIP-CLOP
(Mirowski et al., 2022), CLIPasso (Vinker et al., 2022), CLIPascene (Vinker et
al., 2023). Other studies have combined the text-to-image (T2I) diffusion
model (Ho et al., 2020) with a differentiable rasterizer for SVG synthesis,
such as VectorFusion (Jain et al., 2023), DiffSketcher (Xing et al., 2024) and
SVGDreamer (Xing et al., 2023), all of which have achieved impressive results.

###  2.2. Style Transfer

Style transfer (Jing et al., 2019) is a task in computer vision that involves
combining a content image and a style image to create a new image that
preserves the former’s content and the latter’s style patterns. Over the
years, many researchers have proposed various models (Gatys et al., 2016;
Huang and Belongie, 2017; Kolkin et al., 2019; Sohn et al., 2024; Zhang et
al., 2023; Chung et al., 2023) to improve the quality and speed of style
transfer.

Although style transfer has been extensively studied, research has mainly
focused on raster images, with little work on vector graphics. This is because
the basic units of vector graphics are primitives like mathematical formulas
and geometric descriptions, different from the pixels in raster images. As a
result, the style transfer methods used for raster images are not very
effective for vector images.

It has been noted that StyleCLIPDraw (Schaldenbrand et al., 2022) attempted to
add a style loss term to CLIPDraw (Frans et al., 2022) using VGG (Simonyan and
Zisserman, 2014) network to achieve stylized vector graphics synthesis.
However, the result was messy, and the style was poorly represented. On the
other hand, Efimova et al. (Efimova et al., 2023) attempted to transfer the
style of an image to another vector graphic. However, this method requires a
vector graphic as a content image, which is impractical for most use cases and
does not produce high-quality results.

###  2.3. Diffusion Model

Denoising diffusion probabilistic models (Ho et al., 2020; Sohl-Dickstein et
al., 2015; Song et al., 2021b) (DDPMs), especially improved by Classifier-Free
Guidance (Ho and Salimans, 2022) (CFG), have shown impressive ability of T2I
synthesis. As the diffusion model matures (Dhariwal and Nichol, 2021; Song et
al., 2021a; Lu et al., 2022), large image generation models like Stable
Diffusion (Rombach et al., 2022), Imagen (Saharia et al., 2022), DALL·E 2
(Ramesh et al., 2022), DeepFloyd IF (StabilityAI, 2023), and GLIDE (Nichol et
al., 2022) become more prevalent. Our proposed VectorPainter employs a
diffusion model as prior to supervising the stylized vector graphics synthesis
process. However, the application of diffusion model to the vector domain has
not yet been explored. One potential approach is to employ Score Distillation
Sampling (SDS).

Many 3D generative approaches use optimization-based methods to train a Neural
Radiance Fields (Mildenhall et al., 2021) (NeRF) such as DreamFusion (Poole et
al., 2023). It utilizes a pixel space T2I diffusion model to optimize 3D
volumes and proposes Score Distillation Sampling loss, which can be used to
optimize an image or a differentiable image parameterization (Mordvintsev et
al., 2018) (DIP). Inspired by DreamFusion (Poole et al., 2023), we can adopt
SDS loss in vector graphics synthesis. Vector graphics are composed of
parameterized curves and shapes, which makes the synthesis a process of
optimizing SVG parameters. To achieve this, we utilize a differentiable
rasterizer to render the parameters of vector graphics, which allows the
raster image to be represented by the differentiable rasterizer.

##  3\. Methodology

In this section, we introduce VectorPainter, a method based on optimization
for generating vector graphics that exhibit consistent stylization while
remaining faithful to textual descriptions. As depicted in Figure 3,
VectorPainter operates on three inputs: a set of SVG parameters
$\\{s_{i},c_{i},w_{i}\\}_{i=1}^{n}$, a text prompt $\mathcal{P}$ and a
reference image $\mathcal{I}_{s}$. Specifically, we input the text prompt into
a frozen pre-trained diffusion model (Rombach et al., 2022) to obtain the
latent vector z, render the SVG parameters using DiffVG (Li et al., 2020) to
produce a raster image $\mathcal{I}$, and subsequently utilize both z and
$\mathcal{I}$ as inputs for SDS (Poole et al., 2023) to compute the gradient.
And we optimize over $\theta=\\{s_{i},c_{i},w_{i}\\}_{i=1}^{n}$ such that
$\mathcal{I}=\mathcal{R}(\theta)$ is close to a sample from the frozen
diffusion model. While this represents a generic text-to-SVG pipeline for
generating SVGs (as highlighted in the dashed box), it currently lacks the
inclusion of an additional reference image for style processing
$\mathcal{I}_{s}$.

Figure 3. Overview of our method: VectorPainter is built upon an optimization-
based vector graphics synthesis pipeline (Xing et al., 2024), as highlighted
in the dashed box. It also accepts a reference image to control the style. The
key to our method is the extraction of vectorized strokes from the reference
image for SVG initialization. The reference image is also used for loss
computation to maintain stroke style during optimization. Figure 4.  The
workflow of our VectorPainter: Starting with a text prompt $\mathcal{P}$ and a
raster reference image $\mathcal{I}_{s}$, VectorPainter optimizes the
parameters of the vector graphic $\mathcal{S}$. Initially, it extracts
vectorized strokes from $\mathcal{I}_{s}$, which are then used to initialize
the synthesis process. During this process, a novel style preservation loss is
computed to ensure fidelity to the original style.

In this work, VectorPainter aims to generate a vector graphic $\mathcal{S}$
while upholding fidelity to $\mathcal{P}$ and preserving the stylistic
characteristics of $\mathcal{I}_{s}$. To precisely maintain style consistency
within the vector space, we propose a method to extract vector strokes from
the reference image and utilize them as the initial SVG parameters. In order
to prevent the style from being overshadowed by text content, we have
introduced an additional style preservation loss. This loss function
supervises SVG synthesis from two perspectives: local stroke-level deformation
constraint and global painting awareness-level style constraint. Finally, we
jointly optimize the aforementioned weighted losses. Experimental results
demonstrate that our approach effectively aligns with both the textual
description and the stylistic characteristics of the reference image.

###  3.1. Style Stroke Extraction

We define a stylized vector graphic as a collection of strokes
$\\{s_{i}\\}_{i=1}^{n}$ accompanied by corresponding color and width
attributes. To represent each stroke, a two-dimensional quadratic Bézier curve
is employed, defined by three control points denoted as
$P_{i}=\\{p_{i}^{j}\\}_{j=1}^{3}=\\{(x_{i},y_{i})^{j}\\}_{j=1}^{3}$, where $x$
and $y$ represent the coordinates within the canvas. Additionally, each stroke
is defined by an RGBA vector representing color $c_{i}=\\{r,g,b,a\\}$ and a
value indicating width $w_{i}$.

We extract and learn the painting style of the reference image from the
perspective of brush strokes, enabling an accurate representation of the
reference image’s style in vector graphics. In certain painting styles,
intriguing and concise brushwork is indispensable. Artists utilize strokes of
varying lengths and thicknesses throughout different regions of the painting.
Accurate brushstroke representation in vector space is critical for
synthesizing vector drawings in a style consistent with the reference image.

Similar strokes within a piece of artwork exhibit analogous attributes like
texture, color, and brightness, consequently forming irregular pixel clusters
with visual significance. Leveraging this observation, we employ a super-pixel
method to delineate these pixel clusters, facilitating the representation of
strokes in groups. As depicted in Figure 5, when given a style image, we
employ the SLIC method (Achanta et al., 2010) to partition strokes exhibiting
similar features within the image into distinct regions. We vectorize these
stroke groups, with each stroke group corresponding to an SVG path group. And
these SVG path groups serve as the basic representation units of the
vectorized reference image. In addition, we propose a novel stroke extraction
algorithm for representing the extracted region as vectorized strokes. As
shown in Algorithm 14, in order to extract strokes within each segmented
region, we propose employing the pair of points most distant from each other
as the initial and terminal control points, calculating the mean color of the
region to serve as the stroke color, and computing the average distance
between border points and the connections between control points.

Figure 5. The process of style stroke extraction. We excavate and extract
strokes with similar characteristics, and vectorize these strokes as a capture
of the style of the reference image. Algorithm 1 Stroke extraction algorithm

1:Distinct regions $T$ of segmented reference image $\mathcal{I}_{\text{s}}$

2:List of initialized strokes $S=\\{s_{i}\\}_{i=1}^{n}$

3:Initialize: An empty set $S$

4:for each $region\in T$ do

5: Calculate the border points $border$ of $region$

6: Calculate distance $dist$ of each point pair

7: Find the point pair $<p_{1},p_{3}>$ where $dist(<p_{1},p_{3}>)=\max(dist)$

8: $p_{2}\leftarrow\dfrac{p_{1}+p_{3}}{2}$

9: Stroke control points $s_{i}\leftarrow\\{p_{1},p_{2},p_{3}\\}$

10: Stroke width
$w_{i}\leftarrow\frac{1}{M}\sum_{j=1}^{M}\|border[j],\vec{p_{1}p_{3}}\|$,
where $M$ indicates the number of border points

11: Stroke color
$c_{i}\leftarrow\frac{1}{N}\sum_{k=1}^{N}region.point[k].color$, where $N$
represents the number of points in $region$

12: $S.append(\\{s_{i},c_{i},w_{i}\\})$

13:end for

14:return $S$

###  3.2. Vector Stroke Initialization

As depicted in Figure 6, the extracted vector strokes enable the accurate
reconstruction of the original reference image, effectively preserving its
intricate details. These vector strokes contain the style of reference image,
and can be used to provide style prior information for the generation of
vector graphics. Specifically, we propose using vector strokes obtained from
the reference image as the initial SVG for text-to-SVG generation. Compared to
the random initialization of vector stroke shapes, our method excels in
preserving the style of the reference image with maximum detail. Figure 10
illustrates a qualitative comparison of the two initialization methods. Our
method demonstrates a higher fidelity to the reference image at stroke level.

###  3.3. SVG Synthesis with Style Supervision

To preserve style consistency without significantly altering the overall
optimization process—thus retaining the final style while minimizing changes
to individual strokes—we introduce the Style Preservation loss. This loss
function monitors SVG synthesis from two perspectives: local stroke-level
deformation constraints and global paint-conscious style constraints. The
Style Preservation loss comprises two components: one involves stroke-to-
stroke-level constraints implemented through control points, while the other
pertains to global-level style constraints determined by the perceived
similarity between the rendered and the reference image.

Specifically, we record a collection of vector strokes extracted from the
reference image and use this as Ground-truth (GT) for the stroke-level
constraint. During the optimization process, each updated stroke will be
calculated one-to-one with the GT changes,

(1)
$\begin{split}&\mathcal{L}_{\text{stroke}}(s^{\prime},s)=\frac{1}{N}\sum_{i=1}^{N}\|d(s^{\prime}_{i})-d(s_{i})\|^{2}_{2}\\\
&\text{where}\;d(s)=\|\vec{p_{1}p_{2}}\|+\|\vec{p_{2}p_{3}}\|+\cos<\vec{p_{1}p_{2}},\vec{p_{2}p_{3}}>\end{split}$

where $s^{\prime}\in\\{s_{i}^{\prime}\\}_{i=1}^{N}$ indicates a stroke from
stroke collection, and $s_{i}$ indicates the updated stroke. $p_{i}$ indicates
$i$-th control points of $s$ and $s^{\prime}$. The
$\mathcal{L}_{\text{stroke}}$ loss discourages excessive stroke variation in
synthesized vector graphics, preserving the style of the reference image on a
micro-scale.

Figure 6. VectorPainter is capable of reconstructing paintings in vector
format.

The stroke-level loss ensures that vector graphics in optimization maintain
strokes consistent with the reference image. However, there remains a lack of
global style consistency restrictions. Additionally, we incorporate the LPIPS
(Zhang et al., 2018) loss in our Style Preservation loss. The LPIPS loss helps
to make the generated stylized vector graphics similar to the reference image
$\mathcal{I}_{\text{s}}$ in terms of perceptual similarity level.

(2)
$\mathcal{L}_{\text{style}}=\lambda_{s}\mathcal{L}_{\text{stroke}}(s^{\prime},s)+\lambda_{g}\mathcal{L}_{\text{LPIPS}}(\mathcal{I},\mathcal{I}_{\text{s}})$

where $\lambda_{s}$ and $\lambda_{g}$ are the weights of the Style
Preservation loss. And $\mathcal{I}=\mathcal{R}(\theta)$ is rendering result.
The $\mathcal{L}_{\text{LPIPS}}$ loss encourages the synthesized stylized
vector graphics to look more similar from a macro perspective, for example,
making the overall color of the two images closer. Experimental results
demonstrate that our generated vector graphics match the style of the
reference image at both micro and macro levels.

###  3.4. Total Loss Objectives

We incorporate our proposed Style Preservation loss with the JVSP and ASDS
suggested in DiffSketcher (Xing et al., 2024) to produce SVG content that is
in line with the text and maintains the style of the reference image.
Specifically, we utilize the VAE (Esser et al., 2021) decoder $\mathcal{D}$ to
obtain the pixel representation of
$\hat{\epsilon}_{\phi}(\textbf{z}_{t}|y;t)$. At the same time, we render the
parameters $\theta$ of the SVG using the differentiable rasterizer
$\mathcal{R}$, which provides the raster image
$\mathcal{I}=\mathcal{R}(\theta)$. And, we augment the two images and apply
the CLIP visual encoders (Radford et al., 2021) as visual semantic similarity
metrics. Specifically, we use the following loss function:

(3)
$\mathcal{L}_{\text{JVSP}}=\mathcal{L}_{\text{CLIP}}(\mathcal{I},\mathcal{I}^{\prime})+\mathcal{L}_{\text{Perceptual}}(\mathcal{I},\mathcal{I}^{\prime})$

where
$\mathcal{I}^{\prime}=\mathcal{D}(\hat{\epsilon}_{\phi}(\mathbf{z}_{t}|y;t))$.
where $\hat{\epsilon}_{\phi}$ indicates the frozen pre-trained diffusion model
(Rombach et al., 2022). The $\mathcal{L}_{\text{JVSP}}$ loss encourages vector
graphics synthesis to match underlying semantic details of samples from LDM,
leading to more realistic and visually appealing results. Although vectorizing
a rasterized diffusion sample can lead to loss, the input augmentation version
of the SDS (ASDS) loss can be utilized to either fine-tune vector results or
optimize vector graphics from scratch.

Figure 7. Various results synthesized by VectorPainter from identical inputs.

We optimize vector graphics parameters $\theta$ directly and use a
differentiable rasterizer $\mathcal{R}$ to acquire the raster image
$\tilde{\mathcal{S}}\in\mathbb{R}^{H\times W\times 3}$. Then, we apply data
augmentation techniques to obtain the augmented version
$\tilde{\mathcal{S}}_{a}\in\mathbb{R}^{H\times W\times 3}$. The LDM employs a
VAE encoder (Esser et al., 2021) to encode $\tilde{\mathcal{S}}_{a}$ into a
latent representation
$\mathbf{z}=\mathcal{E}(\tilde{\mathcal{S}_{a}})\in\mathbb{R}^{(H/f)\times(W/f)\times
4}$, where $f$ denotes the downsample factor. Finally, we use the ASDS loss
function given below:

(4)
$\begin{split}\nabla_{\theta}\mathcal{L}_{\text{ASDS}}&(\phi,\mathcal{S}=\mathcal{R}(\theta))\triangleq\\\
&\mathbb{E}_{t,\epsilon,a}\left[w(t)(\hat{\epsilon}_{\phi}(\mathbf{z}_{t};y,t)-\epsilon)\frac{\partial\mathbf{z}}{\partial\tilde{\mathcal{S}}_{a}}\frac{\partial\tilde{\mathcal{S}}_{a}}{\partial\theta}\right]\end{split}$

where $w(t)$ is the weighting function. And noised to form
$\mathbf{z}_{t}=\alpha_{t}\mathbf{x}_{a}+\sigma_{t}\mathbf{\epsilon}$ where
$t\sim\mathcal{U}(0.05,0.95)$ preventing excessive noise. In practice, the
gradients of the loss are used instead of the scalar loss. The
$\mathcal{L}_{\text{ASDS}}$ loss encourages the synthesis process to match the
given text prompt.

To summarize, we combine Style Preservation loss linearly with text-guided
content losses. Our optimization objectives are as follows:

(5)
$\min_{\theta}\;\mathcal{L}_{\text{style}}+\lambda_{\text{JVSP}}\mathcal{L}_{\text{JVSP}}+\lambda_{\text{ASDS}}\nabla_{\theta}\mathcal{L}_{\text{ASDS}}$

where $\lambda_{\text{JVSP}}$ is weights of the corresponding loss functions,
and $\lambda_{ASDS}$ is the gradient scale.

Figure 8. Results of the qualitative evaluation. The format of the generated
results is indicated in parentheses.

##  4\. Experiments

In this section, we employ our VectorPainter model to synthesize various
stylized vector graphics. We compared our VectorPainter to recent vector
graphics synthesis methods (Schaldenbrand et al., 2022; Xing et al., 2024;
Efimova et al., 2023), stroke-based rendering methods (Kotovenko et al.,
2021), and style transfer methods on raster images (Kolkin et al., 2019).
Finally, we conducted some ablation studies to demonstrate the effectiveness
of each part of our approach.

###  4.1. Qualitative Evaluation

As shown in Fig. 1, our proposed VectorPainter can produce stylized vector
graphics considering both the prompt’s semantics and the reference image’s
style representation. Besides, given the same inputs, our model is also
capable of generating various stylized vector images, as illustrated in Fig.
7.

Table 1. Results of the quantitative evaluation. The format of the generated results is indicated in parentheses. |  Ours (vector) |  StyleCLIPDraw 111If text-guided synthesis uses the samples of the LDM, we employ it as the content image. Since StyleCLIPDraw does not utilize LDM, we do not calculate its LPIPS score. (vector) |  DiffSketcher + STROSS (vector) |  VectorNST (vector) |  STROTSS + LIVE (vector) |  Kotovenko et al. (raster) |  STROTSS (raster)
---|---|---|---|---|---|---|---
ArtFID$\downarrow$ | 38.807 | - | 45.298 | 57.126 | 56.525 | 51.828 | 40.352
FID$\downarrow$ | 24.069 | 45.112 | 27.933 | 30.851 | 33.902 | 28.075 | 22.893
LPIPS$\downarrow$ | 0.548 | - | 0.566 | 0.794 | 0.667 | 0.783 | 0.689
CLIP score($\times 100$)$\uparrow$ | 30.533 | 28.312 | 28.780 | 26.130 | 26.864 | 29.450 | 28.445
Figure 9. Comparison of results with style control solely through text
prompts. Figure 10. Effect of stroke initialization strategy.

We compare our VectorPainter with several baseline methods. To achieve a
stylized vector graphic from a text prompt and a reference image, there are
some possible solutions: i) Directly stylized vector graphics synthesis:
Generate stylized vector graphics directly from text and reference images,
such as our VectorPainter and StyleCLIPDraw (Schaldenbrand et al., 2022). ii)
Vector graphics synthesis + Raster style transfer methods: Use a general text-
guided vector graphics synthesis process and adopt a raster image style
transfer method. For example, DiffSketcher (Xing et al., 2024) \+ STROTSS
(Kolkin et al., 2019). iii) Vectorized T2I model sample + Vector style
transfer method: First generate a raster image using the T2I model providing
text prompt, and then transfer the style of reference image into vectorized
T2I model sample. For instance, generate a raster image by the LDM (Rombach et
al., 2022), then use LIVE (Ma et al., 2022) to vectorize it, and finally
employ VectorNST (Efimova et al., 2023). iv) Raster image style transfer
method + Vectorization: First generate a raster image using the T2I model
providing text prompt, then utilize a raster style transfer method to obtain a
stylized raster image, and finally vectorize it. For example, STROTSS (Kolkin
et al., 2019) \+ LIVE (Ma et al., 2022). v) Stroke-based rendering: Use
parameterized brushstrokes to mimic the human painting style, which is similar
to the idea of placing strokes. However, this method can only generate raster
outputs instead of vector graphics. We choose Kotovenko et al.  (Kotovenko et
al., 2021) proposed method, where we select the outputs before its pixel
optimization process for fairness. The results of the comparison,
differentiated by whether the output is raster or vector, are presented in
Figure 8. StyleCLIPDraw (Schaldenbrand et al., 2022) may be unable to
accurately express the meaning of the prompt because it uses the CLIP model to
supervise semantics, and the subject in the image may become challenging to
identify. DiffSketcher (Xing et al., 2024) \+ STROTSS (Kolkin et al., 2019)
struggles with capturing the unique characteristics of vector graphics very
well, leading to distorted curves and untidy sketches on the canvas, which
loses the feeling of smearing strokes in the painting. VectorNST (Efimova et
al., 2023) results are too messy and abstract, with only the colors being
close. Kotovenko et al.  (Kotovenko et al., 2021) falls short in processing
strokes effectively. This results in strokes that do not cover the entire
canvas, blurring the content of the output image. STROTSS (Kolkin et al.,
2019) performs effectively in transferring style within raster space. However,
its effectiveness diminishes when vectorized, and loses its capability to
transfer texture styles from raster space. In contrast, our VectorPainter
handles the stroke information of the reference image precisely by style
stroke extraction and our style supervision method, and incorporates LDM
(Rombach et al., 2022) to supervise the semantics of the generated vector
images, achieving the best results in the generation of stylized vector
graphics.

Figure 11. The effect of different loss terms. The top row illustrates the
results of the VectorPainter full model, and without the
$\mathcal{L}_{\text{stroke}}$, $\mathcal{L}_{\text{LPIPS}}$, and
$\mathcal{L}_{\text{JVSP}}$, respectively. The bottom row illustrates the
effect of ASDS grad scale on the results.

We further compare the results of ours with SVGDreamer (Xing et al., 2023),
which synthesizes vector graphics with text control only. As illustrated in
Fig. 9(a), using prompts alone to control the style of an image cannot match
the accuracy of using a reference image. We also show the results of
vectorizing an LDM sample, as shown in Fig. 9(b). We can observe that both
methods can generate correct content, but the style is not aligned with that
indicated in the text prompt. In comparison, using a reference image can
realize more precise control, thus our results are better.

###  4.2. Quantitative Evaluation

To illustrate the effectiveness of our method, we employ the CLIP score
(Radford et al., 2021) to measure the alignment of prompt and synthesized SVG
and a recently introduced style transfer metric, ArtFID (Wright and Ommer,
2022), to evaluate the effectiveness by considering the preservation of both
content and style. ArtFID is acknowledged for its strong alignment with human
judgment. Specifically, the ArtFID value is computed as (ArtFID = (1 + LPIPS)
· (1 + FID)), where LPIPS (Zhang et al., 2018) quantifies the content fidelity
between the stylized image and its corresponding content image. Concurrently,
FID (Heusel et al., 2017) measures the style fidelity between the stylized
image and its respective style image. Table 1 presents a comparison of the
methods above. Our VectorPainter performs the best in the vector domain for
ArtFID and CLIP score.

###  4.3. Implementation

Our method is based on an optimization-based vector graphics synthesis
pipeline (Xing et al., 2024) and accepts a reference image to control the
style. Specifically, we define the synthesized stylized vector graphic as
$\mathcal{S}=\\{s_{i},c_{i},w_{i}\\}_{i=1}^{n}$, where $s_{i}$, $c_{i}$, and
$w_{i}$ refer to stroke path, stroke color, and stroke width, respectively.
When provided with a reference image, we utilize a stroke extraction algorithm
to derive the corresponding collection of strokes, consisting of $n$ extracted
strokes. The number of strokes $n$ is determined by the user. If the strokes
in the style image are short, users may choose to define a greater number of
strokes; conversely, fewer strokes may be defined if the strokes are longer.
For paintings by Van Gogh, we typically set this number to approximately
4,000.

Furthermore, given a text prompt, we employ the Stable Diffusion (Rombach et
al., 2022) V1.5 as a pre-trained diffusion model, along with a DDIM solver
(Song et al., 2021a), to sample a raster image. The parameters of this model
are kept frozen during the optimization process. During the optimization
process, we typically assign the following values for most cases:
$\lambda_{s}=1.0$, $\lambda_{g}=0.5$, $\lambda_{\text{JVSP}}=1.0$, and
$\lambda_{\text{ASDS}}=5\times 10^{-6}$. We configure the other hyper-
parameters as utilized in DiffSketcher (Xing et al., 2024), setting the DDIM
sample steps to 1,000, the CFG weight $\omega=7.5$, and the noise level
$t\sim\mathcal{U}(0.05,0.95)$. Additionally, we set the learning rate for the
control point at 1.0, for color at 0.1, and for width at 0.01. In the initial
600 steps, we exclusively apply the JVSP loss to ensure that the extracted
strokes accurately represent the content described in the textual input.
Subsequently, we introduce the Style Preservation Loss into the optimization
process. In the final 800 steps, we integrate the ASDS loss to fine-tune the
synthesized results.

Throughout the optimization process, VectorPainter requires a minimum of 16 GB
of memory on a Tesla V100 GPU to generate a stylized vector graphic consisting
of 4,000 strokes. This process takes approximately 2.5 hours to complete.

###  4.4. Ablation Studies

####  4.4.1. Stroke Initialization

As depicted in Figure 10, randomly initialized strokes result in a
disorganized aggregation where the delineations between different objects are
obscured, and the primary subject of the image becomes indistinct, because it
cannot discover the style information hidden in the strokes. Conversely,
applying our stroke initialization method enhances the clarity of strokes in
the synthesized vector graphics, imbuing them with an artistic smearing effect
reminiscent of traditional painting techniques.

####  4.4.2. Style Preservation Loss Ablation

Based on the observations in Fig. 11, we can infer that not using stroke-level
loss to generate vector graphics can result in messy strokes in the
synthesized vector graphics. Although the colors may appear similar at a
glance, a closer examination reveals that the smearing effect of using short
strokes in the reference image is not considered. Instead, it uses long curves
to approximate the overall color for a more accurate representation.

Similarly, if we do not contain the LPIPS loss, the synthesized vector
graphics might have the appearance of short smearing strokes. However, from a
broader perspective, there is a significant difference in color between the
synthesized and reference images. This makes it difficult for observers to
quickly distinguish whether the style information of the reference image has
been included in the synthesized vector graphics.

Based on these observations, we can summarize the key characteristics of
Stroke-level loss and LPIPS loss as follows: Stroke-level loss: operates at
the micro level, meaning it applies to each primitive of vector graphics. By
imposing constraints on each stroke of the vector image, we can ensure that
the final generated strokes do not deviate significantly from the strokes
extracted from the reference image. This supports our hypothesis that strokes
can be a means of expressing style. LPIPS loss: operates on a macro level. Its
purpose is to regulate the overall color of the generated vector graphics to
resemble the reference image. This ensures that the visual perception of the
generated vector graphics is similar to that of the reference image.

####  4.4.3. JVSP and ASDS Ablation

As illustrated in Fig. 11, the JVSP loss depends on the sample of LDM, which
takes advantage of the capability of the T2I model for comprehending text
prompts, providing control over the synthesized vector graphics. While the
ASDS loss function does not depend on the outcomes of the LDM and facilitates
for one-stage optimization process. This approach enhances the diversity of
the synthesized vector graphics. However, it is critical to maintain the
gradient scale within an optimal range. A gradient scale that is too low may
result in imagery characterized by faint colors and indistinct content,
whereas an excessively high scale can cause colors to become overly saturated,
leading to a disparity in the hues of the reference image. These two losses
can be used simultaneously to generate more diverse and detailed outcomes.

##  5\. Conclusion

In this study, we introduce VectorPainter, a novel and effective methodology
for synthesizing stylized vector graphics using text prompts and reference
images. VectorPainter posits that the style of brush strokes uniquely
characterizes the overall style of a painting. This study is the first to
conceptualize the stylization process as the reorganization of vectorized
strokes extracted from the reference image. Comprehensive experimental results
validate the effectiveness of each component within our proposed model. We
believe our method not only advances the field but also provides a novel
perspective on understanding and addressing the broader task of style
transfer.

Limitations. Our method has a few limitations that need to be considered.
Firstly, since our model is based on rendering, it can take considerable time
to synthesize a vector graphic, which also exists in other optimization-based
methods. Secondly, because we need to extract brush strokes, generating non-
painting styles whose brush strokes are not clearly defined leads to
unsatisfactory results.

## References

  * (1)
  * Achanta et al. (2010) Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurélien Lucchi, Pascal Fua, and Sabine Süsstrunk. 2010.  SLIC Superpixels.  (2010), 15.  http://infoscience.epfl.ch/record/149300
  * Berger et al. (2013) Itamar Berger, Ariel Shamir, Moshe Mahler, Elizabeth Carter, and Jessica Hodgins. 2013.  Style and abstraction in portrait sketching.  _ACM Transactions on Graphics (TOG)_ 32, 4 (2013), 1–12.
  * Chung et al. (2023) Jiwoo Chung, Sangeek Hyun, and Jae-Pil Heo. 2023.  Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer.  _arXiv preprint arXiv:2312.09008_ (2023).
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Nichol. 2021.  Diffusion Models Beat GANs on Image Synthesis. In _Advances in Neural Information Processing Systems (NIPS)_ , Vol. 34. 8780–8794.
  * Efimova et al. (2023) Valeria Efimova, Artyom Chebykin, Ivan Jarsky, Evgenii Prosvirnin, and Andrey Filchenkov. 2023.  Neural Style Transfer for Vector Graphics.  _arXiv preprint arXiv:2303.03405_ (2023).
  * Esser et al. (2021) Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021.  Taming transformers for high-resolution image synthesis. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 12873–12883.
  * Frans et al. (2022) Kevin Frans, Lisa Soros, and Olaf Witkowski. 2022.  Clipdraw: Exploring text-to-drawing synthesis through language-image encoders.  _Advances in Neural Information Processing Systems_ 35 (2022), 5207–5218.
  * Gatys et al. (2016) Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016.  Image style transfer using convolutional neural networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2414–2423.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ 30 (2017).
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ 33 (2020), 6840–6851.
  * Ho and Salimans (2022) Jonathan Ho and Tim Salimans. 2022.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ (2022).
  * Hu et al. (2023) Teng Hu, Ran Yi, Haokun Zhu, Liang Liu, Jinlong Peng, Yabiao Wang, Chengjie Wang, and Lizhuang Ma. 2023.  Stroke-based Neural Painting and Stylization with Dynamically Predicted Painting Region. In _Proceedings of the 31st ACM International Conference on Multimedia_. 7470–7480.
  * Huang and Belongie (2017) Xun Huang and Serge Belongie. 2017.  Arbitrary style transfer in real-time with adaptive instance normalization. In _Proceedings of the IEEE international conference on computer vision_. 1501–1510.
  * Jain et al. (2023) Ajay Jain, Amber Xie, and Pieter Abbeel. 2023.  Vectorfusion: Text-to-svg by abstracting pixel-based diffusion models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1911–1920.
  * Jing et al. (2019) Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. 2019.  Neural style transfer: A review.  _IEEE transactions on visualization and computer graphics_ 26, 11 (2019), 3365–3385.
  * Kolkin et al. (2019) Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich. 2019.  Style transfer by relaxed optimal transport and self-similarity. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 10051–10060.
  * Kotovenko et al. (2021) Dmytro Kotovenko, Matthias Wright, Arthur Heimbrecht, and Bjorn Ommer. 2021.  Rethinking style transfer: From pixels to parameterized brushstrokes. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 12196–12205.
  * Li et al. (2020) Tzu-Mao Li, Michal Lukáč, Michaël Gharbi, and Jonathan Ragan-Kelley. 2020.  Differentiable vector graphics rasterization for editing and learning.  _ACM Transactions on Graphics (TOG)_ 39, 6 (2020), 1–15.
  * Lu et al. (2022) Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. 2022.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _Advances in Neural Information Processing Systems (NIPS)_ 35 (2022), 5775–5787.
  * Ma et al. (2022) Xu Ma, Yuqian Zhou, Xingqian Xu, Bin Sun, Valerii Filev, Nikita Orlov, Yun Fu, and Humphrey Shi. 2022.  Towards layer-wise image vectorization. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16314–16323.
  * Mildenhall et al. (2021) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Commun. ACM_ 65, 1 (2021), 99–106.
  * Mirowski et al. (2022) Piotr Mirowski, Dylan Banarse, Mateusz Malinowski, Simon Osindero, and Chrisantha Fernando. 2022.  Clip-clop: Clip-guided collage and photomontage.  _arXiv preprint arXiv:2205.03146_ (2022).
  * Mordvintsev et al. (2018) Alexander Mordvintsev, Nicola Pezzotti, Ludwig Schubert, and Chris Olah. 2018.  Differentiable Image Parameterizations.  _Distill_ (2018).  https://doi.org/10.23915/distill.00012 https://distill.pub/2018/differentiable-parameterizations.
  * Nichol et al. (2022) Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. 2022.  GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In _Proceedings of the 39th International Conference on Machine Learning (ICML)_ , Vol. 162. 16784–16804.
  * Nolte et al. (2022) Florian Nolte, Andrew Melnik, and Helge Ritter. 2022.  Stroke-based Rendering: From Heuristics to Deep Learning.  _arXiv preprint arXiv:2302.00595_ (2022).
  * Poole et al. (2023) Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. 2023.  DreamFusion: Text-to-3D using 2D Diffusion. In _The Eleventh International Conference on Learning Representations (ICLR)_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ (2022).
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 10684–10695.
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022\.  Photorealistic text-to-image diffusion models with deep language understanding. In _Advances in Neural Information Processing Systems (NIPS)_ , Vol. 35. 36479–36494.
  * Schaldenbrand et al. (2022) Peter Schaldenbrand, Zhixuan Liu, and Jean Oh. 2022.  Styleclipdraw: Coupling content and style in text-to-drawing translation.  _arXiv preprint arXiv:2202.12362_ (2022).
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015.  Deep unsupervised learning using nonequilibrium thermodynamics. In _International conference on machine learning_. PMLR, 2256–2265.
  * Sohn et al. (2024) Kihyuk Sohn, Lu Jiang, Jarred Barber, Kimin Lee, Nataniel Ruiz, Dilip Krishnan, Huiwen Chang, Yuanzhen Li, Irfan Essa, Michael Rubinstein, et al. 2024\.  Styledrop: Text-to-image synthesis of any style.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Song et al. (2021a) Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021a.  Denoising Diffusion Implicit Models. In _International Conference on Learning Representations (ICLR)_.
  * Song et al. (2021b) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2021b.  Score-Based Generative Modeling through Stochastic Differential Equations. In _International Conference on Learning Representations (ICLR)_.
  * StabilityAI (2023) StabilityAI. 2023.  IF by DeepFloyd Lab at StabilityAI.  https://github.com/deep-floyd/IF.
  * Vinker et al. (2023) Yael Vinker, Yuval Alaluf, Daniel Cohen-Or, and Ariel Shamir. 2023.  Clipascene: Scene sketching with different types and levels of abstraction. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 4146–4156.
  * Vinker et al. (2022) Yael Vinker, Ehsan Pajouheshgar, Jessica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. 2022.  Clipasso: Semantically-aware object sketching.  _ACM Transactions on Graphics (TOG)_ 41, 4 (2022), 1–11.
  * Wright and Ommer (2022) Matthias Wright and Björn Ommer. 2022.  Artfid: Quantitative evaluation of neural style transfer. In _DAGM German Conference on Pattern Recognition_. Springer, 560–576.
  * Xing et al. (2024) Ximing Xing, Chuang Wang, Haitao Zhou, Jing Zhang, Qian Yu, and Dong Xu. 2024.  Diffsketcher: Text guided vector sketch synthesis through latent diffusion models.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Xing et al. (2023) Ximing Xing, Haitao Zhou, Chuang Wang, Jing Zhang, Dong Xu, and Qian Yu. 2023.  SVGDreamer: Text Guided SVG Generation with Diffusion Model.  _arXiv preprint arXiv:2312.16476_ (2023).
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018.  The unreasonable effectiveness of deep features as a perceptual metric. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 586–595.
  * Zhang et al. (2023) Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng Xu. 2023.  Inversion-based style transfer with diffusion models. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 10146–10156.
  * Zou et al. (2021) Zhengxia Zou, Tianyang Shi, Shuang Qiu, Yi Yuan, and Zhenwei Shi. 2021.  Stylized neural painting. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 15689–15698.