# ArtFusion: Controllable Arbitrary Style Transfer using
Dual Conditional Latent Diffusion Models

Dar-Yen Chen
<EMAIL_ADDRESS>
###### Abstract

Arbitrary Style Transfer (AST) aims to transform images by adopting the style
from any selected artwork. Nonetheless, the need to accommodate diverse and
subjective user preferences poses a significant challenge. While some users
wish to preserve distinct content structures, others might favor a more
pronounced stylization. Despite advances in feed-forward AST methods, their
limited customizability hinders their practical application. We propose a new
approach, ArtFusion, which provides a flexible balance between content and
style. In contrast to traditional methods reliant on biased similarity losses,
ArtFusion utilizes our innovative Dual Conditional Latent Diffusion
Probabilistic Models (Dual-cLDM). This approach mitigates repetitive patterns
and enhances subtle artistic aspects like brush strokes and genre-specific
features. Despite the promising results of conditional diffusion probabilistic
models (cDM) in various generative tasks, their introduction to style transfer
is challenging due to the requirement for paired training data. ArtFusion
successfully navigates this issue, offering more practical and controllable
stylization. A key element of our approach involves using a single image for
both content and style during model training, all the while maintaining
effective stylization during inference. ArtFusion outperforms existing
approaches on outstanding controllability and faithful presentation of
artistic details, providing evidence of its superior style transfer
capabilities. Furthermore, the Dual-cLDM utilized in ArtFusion carries the
potential for a variety of complex multi-condition generative tasks, thus
greatly broadening the impact of our research.

Figure 1: Results of our ArtFusion using classifier-free guidance along the
style and content conditions. We can adjust the degree of content and style
fusion during inference, dynamically ranging from under- to over-stylization.
From left to right, the content and style guidance scales are
$[0.15,0.25,0.5,1,2,3,4]$ and $[0.15,0.25,0.5,1,3,5,7]$, respectively.

##  1 Introduction

The objective of style transfer is to synthesise an image $I_{cs}$ that aptly
integrates the content from image $I_{c}$ with the unique stylistic patterns
of a given artistic work, $I_{s}$. Seminal work by Gatys _et al_. [17]
introduced an optimization-based approach. It iteratively enhances the
similarity of content and style features using a pretrained deep neural
network. Despite the influence [41, 54, 69] of this method, it has certain
inherent limitations, most notably its time-consuming nature. This shortcoming
sparked a shift towards research into feed-forward networks for direct, rapid
stylized $I_{cs}$ generation, initiated by Johnson _et al_. [29].

While style transfer models have progressed from transferring a singular style
[29, 37, 67] or a limited number of styles [6, 15, 42, 39, 74, 8, 61] to
arbitrary styles [26, 2, 34, 68, 62, 40, 71, 43, 28, 18, 73, 38, 20],
contemporary AST models still grapple with major issues. Challenges include a
lack of adjustable results tailored to user subjective demands, leading to
undesirably rigid results, under-stylization, or over-stylization [9] that
often disappoint the users. Furthermore, these models frequently suffer from
repetitive artifacts and a poignant loss of artistic details owing to bias in
style similarities [45, 8, 75, 12, 1, 7].

Diffusion probabilistic models (DM) [21] are growing in popularity in the
field of computer vision (CV), celebrated for their high-quality, diverse
image generation. With the incorporation of various inference guidance
techniques [14, 23, 48], conditional DMs (cDMs) can offer flexible control
over output results, suggesting a promising pathway for addressing AST’s
challenges. Nevertheless, direct training of cDMs for style transfer
encounters a roadblock: the necessity for paired data in maximum likelihood
learning, a condition unsatisfied in many complex multi-condition generative
tasks, including style transfer. While disentangled inference guidance [35]
and optimization-based algorithms [30] attempt to solve this, they require
heavy computation and careful hyperparameter tuning. Hence, we raise the
question: Can a cDM be trained effectively for AST?

We present ArtFusion, the first diffusion-based AST model, built upon the
latent diffusion model (LDM) [55]. ArtFusion introduces the dual conditional
conditional LDM (Dual-cLDM) that treats both content and style as conditions.
During the training phase, our model transforms the style transfer task into a
self-reconstruction task while retaining robust stylization capacity during
the inference phase. With likelihood learning, we can avoid biased similarity
loss, where the similarity measure does not accurately reflect human
perception, and artifacts followed. This novel approach, coupled with the
proposed two-dimensional classifier-free guidance (2D-CFG) during sampling
(refer to Fig. 1 and 4), facilitates balanced control between the content and
style, thereby catering to users’ subjective preferences effectively.
Furthermore, ArtFusion capitalizes on DM’s intrinsic ability to generate
diverse and highly coherent stylization, outperforming previous feed-forward
approaches in deftly expressing subtle style characteristics and showcasing
efficiency over inference-only DM methods.

In summary, we offer the following key contributions:

  1. 1.

ArtFusion, the first diffusion-based feed-forward AST model, provides an
effective solution for AST.

  2. 2.

Dual-cLDM, which breaks the paired data limitation in cDM training, promises
to catalyze advancements in other multi-condition generative tasks.

  3. 3.

2D-CFG offers an adjustable tradeoff between content and style, enhancing the
applicability of AST.

  4. 4.

Comprehensive experiments demonstrating the effectiveness of our approach,
showcasing its ability to faithfully transfer style without bias.

Figure 2: The inference framework of the Dual-cLDM for style transfer.
Initiated from isotropic Gaussian-distributed noise $\hat{z}_{T}$, the dual
conditional backbone progressively denoises using both content and style as
conditions. Post-denoising, the $\hat{z}_{0}$ is decoded using the first-stage
decoder. Figure 3: Left: Dual-cLDM Architecture. Pretrained VGG extracts style
features $f_{s}$, while content features $z_{c}$ are encoded using the first-
stage VAE encoder. The content refiner processes $z_{c}$ into $z_{r}$,
refining content from inherent style. The refined $z_{r}$ is then concatenated
with the noisy latent $z_{t}$. Style features $f_{s}$, along with timestep
embeddings, are injected via adaptive normalization. Right: The architecture
of the Content Refiner. This design aims to reduce the depth dimension of
$z_{c}$.

##  2 Limitations and Biases of Style Similarity

While pretrained VGG [63] has traditionally played a pivotal role in style
transfer, it brings with its drawbacks and biases. Having been trained on
natural images for classification tasks, VGG’s repurposing for style
extraction from artistic images encounters certain obstacles. Specifically,
its capabilities in capturing certain aspects of style, such as color hue and
geometric patterns, does not effectively extend to more abstract elements
crucial to art, such as brush strokes, textures of various painting mediums
(_e.g_. oil-painting, watercolor, sketch), or genre nuances. Such essential
artistic features might be infrequent in natural images, or may even be
consciously disregarded by classification-oriented models. As a result,
although prior works might show strength with geometric or vivid styles, they
falter with more abstract ones, leading to a loss of intricate art details.
Consequently, the output might deviate from the intended stylistic vision,
restricting its adaptability to varied artistic demands.

Beyond the inherent issues of classification-pretrained VGG models, another
limitation arises from the widespread use of second-order statistics style
loss in style transfer. While beneficial for matching feature statistics, this
loss inadvertently encourages repetitive artifacts [7]. The second-order
statistics mean/variance or Gram matrix approach to style representation
captures the statistical distribution of features, but neglects their real
distribution and spatial arrangements. This results in style transfer that may
overemphasize certain aspects, particularly dominant textures of the style
image, to minimise the statistical similarity. Consequently, statistics style
loss often leads to a lack of global coherence, creates annoying artifacts,
and misses subtle artistic characteristics [75], once again. This points to an
implicit and flawed aspect of this conventional choice of objective - it does
not explicitly define what constitutes style, relying instead on a statistical
measure that is assumed, but not assured, to encapsulate style.

These limitations underscore the necessity for novel approaches in style
transfer that can more adeptly handle the complexity and subtlety inherent to
artistic styles.

##  3 Related Work

###  3.1 Feed-forward Arbitrary Style Transfer

Feed-forward networks, brimming with potential, have been a focal point of
research in Arbitrary Style Transfer (AST). A myriad of researchers [11, 13,
33, 34, 50, 9, 12, 75, 7, 45, 26, 72, 66, 60] have made significant
contributions to this field. Typically, AST models operate with an objective
function describing the similarities between content and style representations
of output and input images. Nonetheless, this approach confronts two key
challenges: firstly, the bias in content and style representations, and
secondly, the lack of flexibility in output control, inevitably limiting the
capability to cater to diverse aesthetic preferences.

Several methods have attempted to counter the bias issue [45, 11, 13, 50, 72,
12] by exploring self-attention mechanisms in AST. Deng _et al_. [12] notably
developed a pure transformer-based architecture to tackle content bias. Cheng
_et al_. [9] adjusted the style loss to alleviate style bias. To improve the
quality of stylized images, adversarial loss [19] has been incorporated into
AST [3, 53, 27, 7, 75]. Recently, Chen _et al_. [7] and Zhang _et al_. [75]
have used contrastive learning to mitigate bias from pretrained feature
extractors and statistics style loss. Despite the efficacy of these solutions,
the issue of style bias persists as a challenge. Moreover, there has been
minimal improvement in the area of output controllability. To address these
issues, we propose a novel approach that employs diffusion probabilistic
models with maximum likelihood learning for AST, eliminating the necessity for
computing biased similarities and ensuring remarkably versatile and
manipulable outputs.

###  3.2 Diffusion Probabilistic Model

In recent years, diffusion probabilistic models (DM) have gained prestige.
They have shown the capacity for generating high-quality images. As a result,
more research [65, 14, 4, 49, 24, 22, 32, 47, 59, 23, 48] is being invested in
this area. Among them, controlling the progressive inference process is a
significant direction [14, 23]. It provides DM with unprecedented
controllability. On the other hand, Rombach _et al_. [55] and Hu _et al_. [25]
integrate VQ-GAN [16] with DM. This integration allows the dimensional
reduction of images through first-stage VAEs, making the denoising process
less time-consuming.

Conditional DM (cDM) have found widespread application in numerous generative
tasks [58, 70, 57, 36, 46, 48, 55, 31]. SR3 was proposed by Saharia _et al_.
[58] for super-resolution. Wang _et al_. [70] achieved success in semantic
synthesis. Inpainting was researched by Lugmayr _et al_. [46] and Rombach _et
al_. [55]. Furthermore, Nichol _et al_. [48] and Rombach _et al_. [55]
developed stunning text-to-image diffusion models. Despite these impressive
accomplishments, cDMs are confronted with a significant challenge - the need
for paired data for training. This requirement often poses an obstacle for
complex generative tasks like AST that require alignment with multiple
conditions, _e.g_. content and style. Some progress has been made by
developing post hoc approaches using pretrained DMs. For instance, Kwon and Ye
[35] proposed a content/style inference guidance. Also, Kawar _et al_. [30]
presented optimization-based methods. Nevertheless, these methods demand
extensive computational inference resources and carefully tuned
hyperparameters. Our work introduces the pioneering learning-based diffusion
model for style transfer tasks, designed to generate stylized images directly,
hence significantly enhancing the efficiency and effectiveness of the AST.

Figure 4: Our 2D-CFG results enable simultaneous content and style
manipulation for optimized output, demonstrating flexibility and diversity,
thereby offering users with freedom of choice.

##  4 Approach

Our proposed ArtFusion is built on a variant of LDM [55], delivering high-
fidelity stylizations that express subtle artistic elements that are often
overlooked in previous works. This is facilitated by a step-by-step denoising
process throughout the stylization process (refer to Fig. 2). Moreover, we
empower users with the flexibility to balance between source content and
reference style in the outputs, catering to diverse stylization preferences.
Moving forward, this section initially offers an overview of LDM, followed by
a detailed explanation of our proposed framework, and its constituent
components. Lastly, we elucidate novel techniques for manipulating the results
of stylization.

Preliminaries. LDM works with a two-stage framework that combines a VAE and a
diffusion backbone. The key VAE decreases the spatial dimensionality of the
image while preserving its semantic essence, resulting in a concise, low-
dimensional latent space. The diffusion backbone operates within this latent
space, eliminating the need to handle redundant data in the high-dimensional
pixel space, thus alleviating the computational burden. Denote the encoder and
decoder of the first-stage VAE as $E$ and $D$ respectively, the image as $I$,
and the diffusion backbone as $\epsilon_{\theta}$. LDM can be viewed as
sequential denoising autoencoders $\epsilon_{\theta}(z_{t},t)$, for
$t=1,...,T$. The training objective is to predict the noise at stage $t$ and
yield a less noisy version, $z_{t-1}$. Here, $z_{t}$ is derived from a
diffusion process on $z_{0}=E(I)$, this process is modelled as a Markov Chain
of length $T$, wherein each step involving a slight Gaussian perturbation of
the preceding state. To keep the notation simple, we will use
$\epsilon_{\theta}(z_{t})$ to denote the time-dependent
$\epsilon_{\theta}(z_{t},t)$

By applying the reweighted variational lower bound [14], the objective of LDM
become:

$\mathcal{L}_{LDM}=\mathbb{E}_{z,\epsilon\sim\mathcal{N}(0,\mathbf{I}),t\sim\mathcal{U}({1,...,T})}\left[\lVert\epsilon-\epsilon_{\theta}(z_{t})\rVert_{2}^{2}\right]$
(1)

###  4.1 Dual Conditional LDM

As demonstrated in Fig. 3, we establish our approach on the dual conditional
LDM (Dual-cLDM) backbone, leveraging a U-Net[56]-based structure, similar to
the one employed in [55]. Our training method diverges from conventional style
transfer procedures that utilize separate inputs for the content and style
images. Instead, a single image serves the dual purpose of providing both the
content and the style input, such that $I_{cs}=I_{c}=I_{s}$. Consequently, our
task shifts from style transfer to self-reconstruction.

First-stage VAE. We draw on a pretrained VAE from LDM [55], which has a
downsampling factor of 16 and a latent dimension of 16. Consequently, for an
image $I$ with a shape of $3\times 256\times 256$, the encoded latent $z=E(I)$
takes on a shape of $16\times 16\times 16$.

Conditioning Mechanisms. We derive the style feature $f_{s}$ for the style
image $I_{s}$ by concatenating means and variances from layers within the pre-
trained VGG [63] network. Propagating $f_{s}$ through an MLP and subsequently
integrating it into the timestep embedding allows us to condition the model
using adaLN-Zero [51]. An intuitive approach for conditioning the content
image $I_{c}$ involves using $z_{c}:=E(I_{c})$ as the content feature and
combining it with the noisy version $z_{t}$ via concatenation. However, an
unintended consequence could arise during training. The model might overly
rely on $z_{c}$ and neglect $f_{s}$, resulting in the compromisation of the
stylization ability. This issue stems from the fact that $z_{c}$ not only
contains the content information but also the complete style information. To
circumvent this problem, we introduce a content refiner module that assists
the model in refining pure content information from $z_{c}$.

VGG Style Feature Extractor. We utilize the pre-trained VGG-16 [63], which has
been trained on ImageNet [10], to extract features from the style image
$I_{s}$. The style features are formed by concatenating the means and
variances of each feature map in the five style layers [29] relu1_2, relu2_2,
relu3_3, relu4_3 and relu5_3, which results in a $f_{s}$ with a length of
2944.

Content Refiner. The content refiner, a critical component of our model,
serves to refine content and eliminate style from the latent representation
$z_{c}$, producing $z_{r}$. During training, both content and style are
encapsulated within a single image input. By applying two layers of point-wise
convolutions, the content refiner strategically reduces the depth dimension of
$z_{c}$, forcing the elimination of certain information. Since the model can
extract style information from the $f_{s}$ during training, the content
refiner naturally leans towards preserving content while discarding style.
Hence, the $z_{r}$ output is a refined representation, primarily comprising
content with lessened style influence. Unless stated otherwise, the content
refiner in our approach reduces the original depth dimensions from 16 to 12.

Training Algorithm. To harness classifier-free guidance for both content and
style, we use shared weights for training the dual conditional and two partial
conditional models. Specifically, $\epsilon_{\theta}(z_{t},z_{c},\O_{s})$ and
$\epsilon_{\theta}(z_{t},\O_{c},f_{s})$ solely use content or style as
condition, respectively. $\O_{s}$ is the learnable null style, and $\O_{c}$ is
the all zero null content. Throughout the training, we use probabilities
$p_{c}=0.1$ and $p_{s}=0.5$ for the content-only and style-only models,
respectively.

Objective. In our training process, $I_{cs}=I_{c}=I_{s}$ act as the condition,
thus transforming Eqation 1 into:

$\mathcal{L}=\mathbb{E}_{z,\epsilon\sim\mathcal{N}(0,\mathbf{I}),t,I_{cs}}\left[\lVert\epsilon-\epsilon_{\theta}(z_{t},z_{c},f_{s})\rVert_{2}^{2}\right]$
(2)

Inference Algorithm. The inference denoising process is visually depicted in
Fig. 2. Stylization results are generated by progressively denoising the
randomly initialized $\hat{z}_{T}$ with
$\epsilon{\theta}(\hat{z}_{t},z_{c},f_{s})$. We have provided a detailed
explanation of the denoising process in the supplementary materials A. Despite
being trained for self-reconstruction, our model can still effectively utilize
content and style features to achieve remarkable style transfer during
inference when fed with different content and style images.

###  4.2 Two-Dimensional Classifier-free Guidance

Earlier feed-forward AST models have developed several manipulation methods,
such as style interpolation [45, 13, 50] and spatial control [50]. Our
proposed model, ArtFusion, not only accommodates these functions but also
introduces a more flexible adjustment – the two-dimensional classifier-free
guidance (2D-CFG), which is an extension of the classifier-free guidance [23].
Using 2D-CFG, users can guide the inference process to lean towards either
content or style. With two scaling factors, $s_{cnt}$ and $s_{sty}$, assigned
for content and style respectively, the innovative two-dimensional guidance
provides a competitive element in the gradual denoising sampling, driving
content and style vie for dominance:

Figure 5: Style visualization from partial
$\epsilon_{\theta}(z_{t},\O_{c},f_{s})$ reveals ArtFusion’s faithful
expression of style features. Figure 6: Comparison with SOTA results.

$\tilde{\epsilon}_{\theta,s_{cnt}}=s_{cnt}\epsilon_{\theta}(\hat{z}_{t},z_{c},f_{s})-(s_{cnt}-1)\epsilon_{\theta}(\hat{z}_{t},\O_{c},f_{s})$
(3)

$\tilde{\epsilon}_{\theta,s_{sty}}=s_{sty}\epsilon_{\theta}(\hat{z}_{t},z_{c},f_{s})-(s_{sty}-1)\epsilon_{\theta}(\hat{z}_{t},z_{c},\O_{s})$
(4)

$\tilde{\epsilon}_{\theta}(\hat{z}_{t},z_{c},f_{s})=\tilde{\epsilon}_{\theta,s_{cnt}}+\tilde{\epsilon}_{\theta,s_{sty}}-\epsilon_{\theta}(\hat{z}_{t},z_{c},f_{s})$
(5)

##  5 Experiments

###  5.1 Qualitative Evaluation

In this section, we evaluate the controllability and fidelity of style
reproduction in our proposed method, ArtFusion. ArtFusion demonstrates
controllability by enabling stylization level adjustments. This aspect is
illustrated in Fig. 1, which presents a spectrum of stylization ranging from
vivid content to strong stylization. Additionally, ArtFusion’s two-dimensional
classifier-free guidance offers an unprecedented level of nuanced output
adjustments. This capability is showcased in Fig. 4, where ArtFusion
concurrently manipulates content and style, thus easily adapting to various
preferences. Moreover, ArtFusion demonstrates proficiency in integrating style
characteristics with the content, thereby yielding striking style transfer
results. Apart from controlling capacities, ArtFusion also manifests talent in
style representation. The model adeptly integrates distinctive style
characteristics, such as the blurry edges typical of Impressionist art, with
the content to yield compelling results. Fig. 5 serves as proof of this
ability, showcasing ArtFusion’s faithfulness in style representation.

Our style-conditional model, $\epsilon_{\theta}(z_{t},\O_{c},f_{s})$, is
central to this process. This model learns to reconstruct $I_{s}$
independently of $z_{c}$ content information, establishing a correspondence
with the arrangement of style inputs. For common patterns in the dataset, this
link becomes more pronounced. As exemplified in Fig. 5, the results are
logical, especially observable in the depiction of castles in the $2^{nd}$
row, and bottom-up growing trees in the $4^{th}$ row. By learning the
likelihood, our model can grasp the essential traits of various elements,
moving closer to comprehending the essence of ”real art,” a fundamental
challenge in style transfer.

Figure 7: Comparison with style learned by SOTA, using noise content and 20 rounds stylization. ArtFusion avoids repetitive patterns and demonstrates faithful depictions. Figure 8: Close-up view of ArtFusion’s superior fine style texture representation compared to SOTA models. Each example’s second row provides a magnified view. | Our | DiffuseIT | StyTr2 | StyleFormer | CAST | IEST | AdaAttn | ArtFlow | AdaIN
---|---|---|---|---|---|---|---|---|---
$\mathcal{L}_{\mu/\sigma}$ | 1.140 | 1.190 | 0.498 | 0.526 | 0.849 | 0.793 | 0.498 | 0.245 | 0.272
Table 1: VGG style similarity of last example (Stonehenge) in Fig. 8. This
highlights the inconsistency between visual perception and conventional style
similarity.

###  5.2 Comparison

In this section, we compare our method, ArtFusion, with DiffuseIT [35], the
cutting-edge diffusion-based approach, and seven representative feed-forward
AST models: StyTr2 [12], Styleformer [72], CAST [75], IEST [7], AdaAttn [45],
ArtFlow [1], and AdaIN [26]. On the basis of maintaining the content
semantics, the comparison is focused on the criteria: alignment with style
references. DiffuseIT [35], although innovative in its use of pretrained DMs
and DINO ViT [5] similarities, tends to struggle with content and style
degradation. This results in inferior outcomes to other models. The feed-
forward AST models [12, 72, 75, 7, 45, 1, 26] often demonstrate a noticeable
bias in style representation, leading to a divergence between their generated
outputs and original artworks. The presence of repetitive artifacts and a lack
of style texture detail in their generated outputs, as displayed in Fig. 8,
support this observation.

In contrast, ArtFusion effectively learns the correlation between style
conditions and actual artworks, leading to results with superior alignment to
style references. It can capture the core style characteristics that are
typically overlooked in prior similarity learning models. Enlarged details in
Fig. 8 reveal the original-like impression, the texture of oil painting, and
similar brush strokes in our results.

###  5.3 Comparison on Style

Fig. 7 presents a comparative analysis between ArtFusion and other models,
with a focus on how each model comprehends and reproduces styles. We intensify
the style and minimise the content impact by applying noise content and 20
consecutive stylization rounds. DiffuseIT [35] employs the [CLS] token of DINO
ViT for style similarity, which leads to a strong content structure and
results that closely resemble style images. This reveals a limitation in the
current DINO ViT similarity - an inability to effectively separate content
from style, a critical requirement for style transfer. The pretrained VGG
style similarity [12, 26] prompts models to replicate major patterns from the
style reference, leading to the creation of repetitive artifacts and
obstructing the capture of subtle style components. CAST [75] replaces the
statistics similarity with contrastive learning, but still shows a significant
style bias and struggles to grasp unique characteristics. The outcomes tend to
be structurally alike, indicating an issue within contrastive learning.

ArtFusion, in contrast, sidesteps these issues. It shuns repetitive patterns
and heavy content contexts, resulting in outputs that authentically reflect
the style references. This affirms the capacity for unbiased style learning,
accentuating its unique advantage in style transfer. Attributed to the
iterative nature of the denoising process, ArtFusion is able to capture the
fine-grained details and essence of style references, resulting in a high-
fidelity representation of styles.

###  5.4 Analysis of Bias in Style Similarity

We examine the alignment of visual perception with quantification results,
specifically focusing on the commonly used pretrained VGG mean/variance style
loss $\mathcal{L}_{\mu/\sigma}$ [26]. This evaluation is represented in Fig. 8
and Tab. 1. Remarkably, the $\mathcal{L}_{\mu/\sigma}$ metric produces results
that do not align with visual quality. Despite their seemingly superior loss
scores, both AdaIN [26] and ArtFlow[1] generate stylized images marked by
substantial color distortion and spurious artifacts, deviating visually from
the style reference. In contrast, our model presents the unique brush touch of
the style reference, not seen in other models, and stays faithful to other
aspects of the style. However, our model incurs a higher loss, comparable to
that of DiffuseIT [35], which displays clear distortion. These findings
underline the discrepancy between $\mathcal{L}_{\mu/\sigma}$ and the
perceptual quality, serving as a caution against the exclusive reliance on
style similarity for achieving optimal style transfer results.

Figure 9: Impact of compression ratio in the content refiner. Only the content
refiner retains content and eliminates style in $z_{c}$, the model can
effectively rely on $f_{s}$ for style transfer. The style guidance scale
increases from left to right.

###  5.5 Ablation on Content Refiner

In this section, we investigate the impact of the content refiner on
stylization output under varying compression levels of the content feature
$z_{c}$ (Fig. 9). ”Compression” here denotes the reduction of the depth
dimension of $z_{c}$ Analysis indicates that the absence of compression causes
$z_{c}$ to carry an excess of style information, which should ideally be
contributed by the style features $f_{s}$. As a result, the model over-relies
on $z_{c}$ and is unable to utilize the style information from $f_{s}$,
causing a significant drop in stylization performance during inference (see
the $5^{th}$ row in Fig. 9). On the other hand, over-compression of the
content feature leads to inadequate preservation of content semantics,
noticeable as an apparent loss of content structure in the output (see the
$2^{nd}$ and $3^{rd}$ rows in Fig. 9). Based on our empirical results,
compressing the 16-dimensional $z_{c}$ down to a 12-dimensional $z_{r}$
achieves an optimal equilibrium.

###  5.6 Interpolation

Interpolating predicted noise between styles in each intermediate latent space
allows for a smooth, gradual shift from one artistic style to another. Visual
examples of the two-dimensional interpolation process are provided in Fig. 10.
This process enables the seamless blending of artistic features, allowing for
the creation of unique, hybrid styles. Furthermore, when we utilize the
content image itself as one of the styles, a one-dimensional content-style
tradeoff is introduced. This spectrum empowers users to finetune the balance
between retaining content and adopting a new style, further showcasing the
model’s outstanding versatility.

Figure 10: Smooth style interpolation between two styles.

##  6 Conclusion

We have presented ArtFusion, a novel, controllable approach to arbitrary style
transfer (AST) that leverages dual conditional latent diffusion models (Dual-
cLDM). This framework overcomes the common data limitations associated with
diffusion models and avoids biases in feature extractors. With this
innovation, ArtFusion effectively expresses and mirrors unique artistic
attributes derived from style inputs. ArtFusion introduces a new level of
flexibility to AST through two-dimensional classifier-free guidance (2D-CFG)
and noise interpolation. Significantly, our results demonstrate that ArtFusion
effectively prevents common issues found in existing models, such as
repetitive patterns, and excels at reproducing nuanced artistic aspects.

The Dual-cLDM employed in ArtFusion harbors potential applications beyond AST,
opening the door to other complex generative tasks, and broadening the
horizons for diffusion models. While our model demonstrates a leap forward in
the realm of style transfer, a comprehensive understanding of artistic
characteristics continues to be a stimulating challenge for future research
and exploration.

## References

  * [1] Jie An, Siyu Huang, Yibing Song, Dejing Dou, Wei Liu, and Jiebo Luo.  Artflow: Unbiased image style transfer via reversible neural flows.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  * [2] Jie An, Tao Li, Haozhi Huang, Li Shen, Xuan Wang, Yongyi Tang, Jinwen Ma, Wei Liu, and Jiebo Luo.  Real-time universal style transfer on high-resolution images via zero-channel pruning.  CoRR, abs/2006.09029, 2020.
  * [3] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein gan, 2017.
  * [4] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg.  Structured denoising diffusion models in discrete state-spaces.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * [5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9650–9660, October 2021.
  * [6] Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua.  Stylebank: An explicit representation for neural image style transfer.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [7] Haibo Chen, Lei Zhao, Zhizhong Wang, Zhang Hui Ming, Zhiwen Zuo, Ailin Li, Wei Xing, and Dongming Lu.  Artistic style transfer with internal-external learning and contrastive learning.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * [8] Haibo Chen, Lei Zhao, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Ailin Li, Wei Xing, and Dongming Lu.  Dualast: Dual style-learning networks for artistic style transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 872–881, June 2021.
  * [9] Jiaxin Cheng, Ayush Jaiswal, Yue Wu, Pradeep Natarajan, and Prem Natarajan.  Style-aware normalized loss for improving arbitrary style transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 134–143, June 2021.
  * [10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [11] Yingying Deng, Fan Tang, Weiming Dong, Haibin Huang, Chongyang Ma, and Changsheng Xu.  Arbitrary video style transfer via multi-channel correlation.  Proceedings of the AAAI Conference on Artificial Intelligence, 35(2):1210–1217, May 2021.
  * [12] Yingying Deng, Fan Tang, Weiming Dong, Chongyang Ma, Xingjia Pan, Lei Wang, and Changsheng Xu.  Stytr2: Image style transfer with transformers.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [13] Yingying Deng, Fan Tang, Weiming Dong, Wen Sun, Feiyue Huang, and Changsheng Xu.  Arbitrary style transfer via multi-adaptation network.  In Acm International Conference on Multimedia. ACM, 2020.
  * [14] Prafulla Dhariwal and Alexander Quinn Nichol.  Diffusion models beat GANs on image synthesis.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * [15] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur.  A learned representation for artistic style.  In International Conference on Learning Representations, 2017.
  * [16] Patrick Esser, Robin Rombach, and Björn Ommer.  Taming transformers for high-resolution image synthesis, 2020.
  * [17] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [18] Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, and Jonathon Shlens.  Exploring the structure of a real-time, arbitrary neural artistic stylization network.  CoRR, abs/1705.06830, 2017.
  * [19] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
  * [20] Shuyang Gu, Congliang Chen, Jing Liao, and Lu Yuan.  Arbitrary style transfer with deep feature reshuffle.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8222–8231, 2018.
  * [21] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc., 2020.
  * [22] Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  CoRR, abs/2106.15282, 2021.
  * [23] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
  * [24] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling.  Argmax flows and multinomial diffusion: Learning categorical distributions.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * [25] Minghui Hu, Yujie Wang, Tat-Jen Cham, Jianfei Yang, and P. N. Suganthan.  Global context with discrete diffusion in vector quantised modelling for image generation, 2021.
  * [26] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In ICCV, 2017.
  * [27] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [28] Yongcheng Jing, Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, and Shilei Wen.  Dynamic instance normalization for arbitrary style transfer.  Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):4369–4376, Apr. 2020.
  * [29] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 694–711, Cham, 2016. Springer International Publishing.
  * [30] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models, 2022.
  * [31] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2426–2435, June 2022.
  * [32] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  On density estimation with diffusion models.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * [33] Dmytro Kotovenko, Artsiom Sanakoyeu, Sabine Lang, and Bjorn Ommer.  Content and style disentanglement for artistic style transfer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [34] Dmytro Kotovenko, Artsiom Sanakoyeu, Pingchuan Ma, Sabine Lang, and Bjorn Ommer.  A content transformation block for image style transfer.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [35] Gihyun Kwon and Jong Chul Ye.  Diffusion-based image translation using disentangled style and content representation.  In The Eleventh International Conference on Learning Representations, 2023.
  * [36] Bo Li, Kaitao Xue, Bin Liu, and Yu-Kun Lai.  Vqbb: Image-to-image translation with vector quantized brownian bridge, 2022.
  * [37] Chuan Li and Michael Wand.  Precomputed real-time texture synthesis with markovian generative adversarial networks.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 702–716, Cham, 2016. Springer International Publishing.
  * [38] Xueting Li, Sifei Liu, Jan Kautz, and Ming-Hsuan Yang.  Learning linear transformations for fast arbitrary style transfer.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [39] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang.  Diversified texture synthesis with feed-forward networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [40] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang.  Universal style transfer via feature transforms.  In Advances in Neural Information Processing Systems, 2017.
  * [41] Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou.  Demystifying neural style transfer.  In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 2230–2236, 2017.
  * [42] Minxuan Lin, Fan Tang, Weiming Dong, Xiao Li, Changsheng Xu, and Chongyang Ma.  Distribution aligned multimodal and multi-domain image stylization.  ACM Trans. Multimedia Comput. Commun. Appl., 17(3), jul 2021.
  * [43] Tianwei Lin, Zhuoqi Ma, Fu Li, Dongliang He, Xin Li, Errui Ding, Nannan Wang, Jie Li, and Xinbo Gao.  Drafting and revision: Laplacian pyramid network for fast high-quality artistic style transfer.  2021\.
  * [44] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft coco: Common objects in context.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 740–755, Cham, 2014. Springer International Publishing.
  * [45] Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Meiling Wang, Xin Li, Zhengxing Sun, Qian Li, and Errui Ding.  Adaattn: Revisit attention mechanism in arbitrary neural style transfer.  In Proceedings of the IEEE International Conference on Computer Vision, 2021.
  * [46] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11461–11471, June 2022.
  * [47] Eric Luhman and Troy Luhman.  Knowledge distillation in iterative generative models for improved sampling speed, 2021.
  * [48] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2021.
  * [49] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models, 2021.
  * [50] Dae Young Park and Kwang Hee Lee.  Arbitrary style transfer with style-attentional networks.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5873–5881, 2018.
  * [51] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  arXiv preprint arXiv:2212.09748, 2022.
  * [52] Fred Phillips and Brandy Mackintosh.  Wiki Art Gallery, Inc.: A Case for Critical Thinking.  Issues in Accounting Education, 26(3):593–608, 08 2011.
  * [53] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks, 2015.
  * [54] Eric Risser, Pierre Wilmot, and Connelly Barnes.  Stable and controllable neural texture synthesis and style transfer using histogram losses, 2017.
  * [55] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * [56] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  CoRR, abs/1505.04597, 2015.
  * [57] Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models, 2022.
  * [58] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  arXiv:2104.07636, 2021.
  * [59] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  In International Conference on Learning Representations, 2022.
  * [60] Artsiom Sanakoyeu, Dmytro Kotovenko, Sabine Lang, and Björn Ommer.  A style-aware content loss for real-time hd style transfer.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 698–714, 10 2018.
  * [61] Falong Shen, Shuicheng Yan, and Gang Zeng.  Neural style transfer via meta networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [62] Lu Sheng, Ziyi Lin, Jing Shao, and Xiaogang Wang.  Avatar-net: Multi-scale zero-shot style transfer by feature decoration.  In Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on, pages 1–9, 2018.
  * [63] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In International Conference on Learning Representations, 2015.
  * [64] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  CoRR, abs/1503.03585, 2015.
  * [65] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  ArXiv, abs/2010.02502, 2021.
  * [66] Jan Svoboda, Asha Anoosheh, Christian Osendorfer, and Jonathan Masci.  Two-stage peer-regularized feature recombination for arbitrary image style transfer.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [67] Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky.  Texture networks: Feed-forward synthesis of textures and stylized images.  In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, page 1349–1357. JMLR.org, 2016.
  * [68] Huan Wang, Yijun Li, Yuehai Wang, Haoji Hu, and Ming-Hsuan Yang.  Collaborative distillation for ultra-resolution universal style transfer.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [69] Pei Wang, Yijun Li, and Nuno Vasconcelos.  Rethinking and improving the robustness of image style transfer.  In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
  * [70] Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li.  Semantic image synthesis via diffusion models, 2022.
  * [71] Zhizhong Wang, Lei Zhao, Haibo Chen, Lihong Qiu, Qihang Mo, Sihuan Lin, Wei Xing, and Dongming Lu.  Diversified arbitrary style transfer via deep feature perturbation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7789–7798, 2020.
  * [72] Xiaolei Wu, Zhihao Hu, Lu Sheng, and Dong Xu.  Styleformer: Real-time arbitrary style transfer via parametric style composition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14618–14627, 2021.
  * [73] Yuan Yao, Jianqiang Ren, Xuansong Xie, Weidong Liu, Yong-Jin Liu, and Jun Wang.  Attention-aware multi-stroke style transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [74] Hang Zhang and Kristin Dana.  Multi-style generative network for real-time transfer.  In Laura Leal-Taixé and Stefan Roth, editors, Computer Vision – ECCV 2018 Workshops, pages 349–365, Cham, 2019. Springer International Publishing.
  * [75] Yuxin Zhang, Fan Tang, Weiming Dong, Haibin Huang, Chongyang Ma, Tong-Yee Lee, and Changsheng Xu.  Domain enhanced arbitrary image style transfer via contrastive learning.  In ACM SIGGRAPH, 2022.

Supplementary Material

Figure 11: Samples with size $1920\times 480$. 2D-CFG scales $s_{cnt}/s_{sty}$
from top to bottom: $0.4/3.0$, $0.55/2.0$ and $0.25/2.0$.

Figure 12: Samples with size $1280\times 640$. 2D-CFG scales $s_{cnt}/s_{sty}$
from top to bottom: $0.25/1.5$ and $0.25/2.0$.

##  A Details of Denoising Diffusion Probabilistic Models

Given the data $x_{0}$, the Gaussian diffusion process, denoted by $q$,
incrementally adds noise to $x_{0}$ to create noisy data at each timestep
$t=1,...,T$ as per the following equation:

$q(x_{t}|x_{t-1}):=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}},\beta_{t}\mathbf{I})$
(6)

In this equation, ${\beta_{t}}^{T}_{t=1}$ represents the hyper variance
schedule that dictates the extent of noise introduced at each timestep. We
denote $\alpha_{t}:=1-\beta_{t}$ and
$\bar{\alpha}_{t}:=\prod^{t}_{s=1}\alpha_{s}$ to express $q$ in an alternate
form:

$\displaystyle q(x_{t}|x_{0})$
$\displaystyle=\mathcal{N}(x_{t};\sqrt{\bar{\alpha}_{t}}x_{0},(1-\bar{\alpha}_{t})\mathbf{I})$
(7)
$\displaystyle=\sqrt{\bar{\alpha}}_{t}x_{0}+\epsilon\sqrt{1-\bar{\alpha}_{t}},\epsilon\sim\mathcal{N}(0,\mathbf{I})$

This assists in an efficient sampling of $x_{t}$. With an appropriate variance
schedule $\beta_{t}$ and sufficiently large $T$, the distribution $q(x_{T})$
will converge to $\mathcal{N}(0,\mathbf{I})$. In such a case, given
$x_{T}\sim\mathcal{N}(0,\mathbf{I})$, the Gaussian diffusion model
$p_{\theta}$ seeks to approximate and parametrize the reverse distribution
$q(x_{t-1}|x_{t})$. According to Sohl-Dickstein _et al_. [64],
$q(x_{t-1}|x_{t})$ can be treated as a diagonal Gaussian distribution as $T$
approaches infinity and $\beta_{t}$ tends to zero. Therefore, we can represent
the parametrized $p_{\theta}$ as:

$p_{\theta}(x_{t-1}|x_{t}):=\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t},t),\Sigma_{\theta}(x_{t},t))$
(8)

Here, $\mu_{\theta}(x_{t},t)$ and $\Sigma_{\theta}(x_{t},t)$ are learned
deviation and mean. Ho _et al_. [21] observe that instead of directly
optimizing the variational lower-bound for $p_{\theta}$ by learning both
$\mu_{\theta}$ and $\Sigma_{\theta}$, the model can fix
$\Sigma_{\theta}(x_{t},t)$ to either $\beta_{t}\mathbf{I}$ or
$\tilde{\beta}_{t}\mathbf{I}$, where
$\tilde{\beta}_{t}:=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}$
represents a rescaling of $\beta_{t}$. Moreover, we can represent
$\mu_{\theta}$ as:

$\mu_{\theta}(x_{t},t)=\frac{1}{\sqrt{\alpha_{t}}}(x_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(x_{t},t))$
(9)

With the prediction $\epsilon_{\theta}$ of the involved noise $\epsilon$ in
Eq. 7. The optimization goal can be transferred to minimize the difference
between $\epsilon_{\theta}$ and $\epsilon$. This simplified objective is:

$\mathcal{L}_{simple}:=\mathbb{E}_{x_{0}\sim
q(x_{0}),t\sim\mathcal{U}(\\{1,...,T\\}),\epsilon\sim\mathcal{N}(0,\mathbf{I})}\left[\lVert\epsilon-\epsilon_{\theta}(x_{t},t)\rVert_{2}^{2}\right]$
(10)

The main optimization goal, therefore, is to align the predicted and actual
noise terms as closely as possible.

##  B Implementation Details

We employed the MS-COCO dataset [44] to train the partial conditional
$\epsilon_{\theta}(z_{t},z_{c},\O_{s})$, the model that exclusively conditions
on content. Meanwhile, the WikiArt dataset [52] was selected for training both
$\epsilon_{\theta}(z_{t},z_{c},f_{s})$ and
$\epsilon_{\theta}(z_{t},\O_{c},f_{s})$ due to its diverse artistic styles.
All the images used in the training process were randomly cropped into a
256x256 size. Our model was trained on a single NVIDIA GeForce RTX 3080 Ti
GPU. Throughout the training process, we maintained an exponential moving
average (EMA) of ArtFusion with a decay rate of 0.9999. Unless otherwise
specified, our results were sampled using the EMA model with 250 DDIM [65]
steps and setting the 2D-CFG scales as $s_{cnt}/s_{sty}=0.6/3$. The
hyperparameters used for the architecture and training process of ArtFusion
are detailed in Tab. 2 and 3, respectively. We did not conduct hyperparameter
sweeps in this study.

$z$-shape |  $z_{r}$-shape | Channels | Channel Multi. | Head Dim. | Embed. Dim. | Attn. Resolution
---|---|---|---|---|---|---
$16\times 16\times 16$ | $12\times 16\times 16$ | 384 | 1, 2 | 64 | 1024 | 16, 8
Table 2: Hyperparameters for the architecture of ArtFusion. Diffusion Steps | Noise Schedule | Batch Size | Iterations | Optimizer | Learning Rate
---|---|---|---|---|---
1000 | Linear | 128 | 175K | AdamW | $1e-4$
Table 3: Hyperparameters for the training process of ArtFusion.

##  C Inference Analysing

###  C.1 Sampling Steps

Figure 13 showcases a series of stylized results from varying DDIM [65] steps.
Interestingly, our observations indicate that beyond 10 sampling steps, any
additional steps have only a marginal improvement in the visual quality of the
results. This implies that despite our default setting of 250 steps, a
reduction to merely 10 sampling steps does not lead to a noticeable
deterioration in the quality of the output.

Figure 13: Impact of DDIM sampling steps on stylization outcomes. 10 sampling
steps are enough for high-fidelity results.

###  C.2 Inference Time

We evaluated the inference time of our model in comparison to other SOTA
methods, as outlined in Table 4. These comparisons utilized images of 256x256
resolution on an RTX 3080 Ti GPU. DiffuseIT [35], another diffusion-based
method, requires notably extended inference times compared to our model. This
increased time is due to DiffuseIT’s reliance on DINO ViT [5], which requires
execution in both forward and backward directions during each sampling step to
provide guidance. On the other hand, our model maintains a competitive
inference time, approximately $3\times$ as long as ArtFlow [1] when sampling
with 10 steps. Considering the continued advancements in accelerated denoising
inference processes, we expect the current efficiency gap to diminish in the
near future.

| Ours | Ours 10 steps | DiffuseIT | StyTr2 | Styleformer | CAST | IEST | AdaAttn | ArtFlow | AdaIN
---|---|---|---|---|---|---|---|---|---|---
time | 3.658s | 0.196s | 60.933s | 0.051s | 0.022s | 0.018s | 0.018s | 0.024s | 0.063s | 0.017s
Table 4: Inference time comparison among SOTA methods.

##  D Limitation

Our model tends to overfit on the most recurring patterns in the WikiArt
dataset, namely, frontal human faces. Among various art categories, portraits
represent $15\%$ of the entire dataset. This overfitting is evident in
multiple cases, as showcased in style visualizations from
$\epsilon_{\theta}(z_{t},\O_{c},f_{s})$ in Fig. 14. For instance, images
comprising human faces or objects with human-like attributes, as well as
images featuring upside-down faces, appear to predominantly overfit on
horizontal facial patterns (as evident in the $1^{st}-6^{th}$ columns).
However, this issue seems to alleviate or even vanish when images incorporate
other discernible style features (as observable when comparing the $4^{th}$
and $7^{th}-9^{th}$ columns). Moving forward, we intend to overcome this
limitation through the implementation of more robust data augmentation
strategies.

Figure 14: Visualization of style composition with human faces. It highlights
cases of overfitting in style references that include face-like objects,
without strong style patterns.

##  E Additional Qualitative Results

High Resolution. We assess the scalability of ArtFusion by testing it on
higher-resolution content images, while keeping the size of the style images
at $256\times 256$ (refer to Fig. 11 and 12). ArtFusion adeptly scales to high
resolutions without any fine-tuning, preserving high levels of detail and
yielding aesthetically pleasing results.

Manipulation. The controlling capabilities of our model transcend conventional
limits, as demonstrated by the 2D-CFG samples (refer to Fig. 15) and style
interpolations between four styles (see Fig. 16). Additionally, we demonstrate
how the application of gradient masks in style interpolation allows for
precise spatial control over style proportions in Fig. 17. The suite of
manipulation methods we have introduced enhances the versatility and
practicality of AST for real-world applications.

Comparison. Additional comparison with SOTA approaches [35, 12, 72, 75, 7, 45,
1, 26] is illustrated in Fig. 18. We encourage a closer examination of these
figures, as the zooming-in details truly showcase the superior performance of
our model. It is here where the real strengths of ArtFusion shine – in its
fine-grained details.

Figure 15: Additional two-dimensional classifier-free guidance results. Figure
16: Interpolation results between four styles. Figure 17: Results of spatial
control with size $1024\times 480$. The first row is the content and style
images. The second row showcases the spatial control results along with the
corresponding gradient masks. Figure 18: Additional comparison with SOTA
results.