Figure 1. SonicDiffusion. Our framework introduces a novel approach where
audio inputs guide image generation and editing. Leveraging paired audio-
visual data, SonicDiffusion effectively learns to translate sounds into
corresponding audio tokens and use them to guide the diffusion process. Beyond
this, our method can combine audio with textual information for richer image
synthesis, and is capable of artistically altering paintings, demonstrating
its versatility and creative potential.

Description here.

# SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained
Diffusion Models

Burak Can Biner<EMAIL_ADDRESS>Koç UniversityTurkey ,  Farrin Marouf
Sofian  Koç UniversityTurkey ,  Umur Berkay Karakaş  Koç UniversityTurkey ,
Duygu Ceylan<EMAIL_ADDRESS>Adobe ResearchUnited Kingdom ,  Erkut
Erdem<EMAIL_ADDRESS>0000-0002-6744-8614 Hacettepe
UniversityTurkey and  Aykut Erdem<EMAIL_ADDRESS>0000-0002-6280-8422 Koç
UniversityTurkey

###### Abstract.

We are witnessing a revolution in conditional image synthesis with the recent
success of large scale text-to-image generation methods. This success also
opens up new opportunities in controlling the generation and editing process
using multi-modal input. While spatial control using cues such as depth,
sketch, and other images has attracted a lot of research, we argue that
another equally effective modality is audio since sound and sight are two main
components of human perception. Hence, we propose a method to enable audio-
conditioning in large scale image diffusion models. Our method first maps
features obtained from audio clips to tokens that can be injected into the
diffusion model in a fashion similar to text tokens. We introduce additional
audio-image cross attention layers which we finetune while freezing the
weights of the original layers of the diffusion model. In addition to audio
conditioned image generation, our method can also be utilized in conjuction
with diffusion based editing methods to enable audio conditioned image
editing. We demonstrate our method on a wide range of audio and image
datasets. We perform extensive comparisons with recent methods and show
favorable performance.

Latent Diffusion Models, Image Editing, Image Synthesis

##  1\. Introduction

As visual content synthesis evolves, the quest for more immersive and
authentic generation techniques becomes paramount. Text-driven approaches like
DALL·E 2 (Ramesh et al., 2022), Latent Diffusion Models (LDM) (Rombach et al.,
2022), and Imagen (Saharia et al., 2022), have shown remarkable results in
terms of image realism and creativity. However, the reliance on text for image
conditioning, while effective, introduces limitations due to the inherently
manual and sometimes incongruent nature of textual descriptions with their
visual counterparts. This observation underscores the potential of alternative
modalities that offer a more natural and cohesive integration with visual
content.

The integration of audio cues into the image generation process represents an
intriguing yet underexplored frontier. Audio, unlike text, shares a direct and
natural correlation with visual scenes, providing a rich tapestry of
information that text might overlook or find cumbersome to articulate.
Consider the complex soundscape of a bustling city street: audio not only
captures the cacophony of voices, vehicle noises, and the distant hum of urban
life but also conveys the atmosphere of the scene, from the brisk energy of a
morning commute to the subdued tones of an evening stroll. Such auditory cues
offer a depth of context and ambiance that text descriptions may struggle to
encapsulate fully. By leveraging these rich audio signals, our approach aims
to generate images that reflect the vibrancy and dynamism of visual content,
showing the potential of audio to provide a more nuanced and comprehensive
understanding of the scenes. To address these challenges, we propose
SonicDiffusion, a new diffusion model that can generate images semantically
aligned with accompanying audio, as illustrated in Fig. 1.

Prior efforts in audio-driven image synthesis, such as those based on
Generative Adversarial Networks (GANs) (Lee et al., 2022b, 2023b; Li et al.,
2022) and pre-trained text-to-image models (Qin et al., 2023; Yariv et al.,
2023b) have laid the groundwork for our research. These studies have
demonstrated the potential of audio to guide image generation. However, they
often fall short in sample quality or struggle with limitations in the depth
of audio’s semantic capture. SonicDiffusion advances beyond these initial
efforts by integrating audio cues within a diffusion model framework,
specifically adapting and extending the capabilities of the Stable Diffusion
model (Rombach et al., 2022) to embrace audio information.

Our approach is distinguished by its ability to process and translate audio
signals into visual representations, employing audio-image cross-attention
layers that serve as a conduit for this modality translation. This method not
only maintains the high-quality image generation synonymous with diffusion
models but also introduces a novel dimension of audio-driven creativity and
contextual richness with minimal additional training requirements.
Furthermore, our method can be easily extended to edit real images, performing
modifications that reflect the characteristics of the audio input through
feature injection.

We validate the efficacy of our approach through rigorous testing on three
diverse datasets. Our results show that our model surpasses current existing
methods for audio-to-image generation and audio-driven image editing, both in
qualitative and quantitative terms. Notably, our model exhibits exceptional
ability in capturing intricate high-frequency details in landscape generation,
accurately rendering human facial features influenced by sound, and precisely
mapping specific sounds to corresponding visual elements that reflect the
material properties of depicted objects.

To sum up, our contributions are threefold: (i) We introduce a new diffusion
model that extends the capabilities of the pre-trained Stable Diffusion model,
enabling sound-guided image generation. (ii) Our approach presents audio-image
cross-attention layers, strategically designed to require a minimal set of
trainable parameters. (iii) We demonstrate the versatility of our model in not
just generating, but also editing images to match auditory inputs. For our
code and models, please see https://cyberiada.github.io/SonicDiffusion/.

##  2\. Related Work

Diffusion-based Image Generation and Editing. Image generation and editing
have evolved significantly with diffusion models like Imagen (Saharia et al.,
2022), DALL$\cdot$E 2 (Ramesh et al., 2022), and Latent Diffusion (Rombach et
al., 2022), which generate realistic images from text prompts. These models
are also central in text-based image editing (Meng et al., 2021; Nichol et
al., 2022). Various approaches have been developed since then. PnP (Tumanyan
et al., 2023) injects image features and manipulates self-attention weights
during text guided editing. Imagic (Kawar et al., 2023) optimizes text
embeddings and fine-tunes the model for targeted edits. Prompt-to-Prompt
(Hertz et al., 2022) adjusts cross-attention maps to ensure spatial
consistency during editing. InstructPix2Pix (Brooks et al., 2023) employs
GPT-3 (Brown et al., 2020) to generate editing instructions which are then
used to train a diffusion model to enable instruction based editing. Pix2pix-
zero (Parmar et al., 2023) calculates edit directions from the text embeddings
of original and edited image pairs. Lastly, Mokady et al. (2023) focus on
optimizing null-text embeddings to facilitate better inversion and editing
performance. All these methods, however, focus on text guided generation and
editing while our goal is to unlock audio guidance.

Adapters. Adapters have proven to be highly effective for transfer learning
within large pre-trained NLP models, achieving results that closely approach
the state-of-the-art (Houlsby et al., 2019). For instance, adapters have
recently been utilized in (Zhang et al., 2023) to tune LLaMa (Zhang et al.,
2023) into an instruction model. In the text-to-image diffusion models,
adapters have also been instrumental. T2I-Adapter (Mou et al., 2023)
illustrates that adapters can offer a straightforward, cost-effective, and
flexible means of guiding pre-trained Stable Diffusion models while preserving
their core structure. IP-Adapter (Ye et al., 2023) introduces an adapter
architecture with a decoupled cross-attention mechanism, facilitating
multimodal image generation.

Figure 2. An overview of our proposed SonicDiffusion model. Our framework
allows for two core functionalities: (1) audio-driven image generation, and
(2) audio-guided image editing. In (1), both sound inputs and optional text
prompts are tokenized, guiding the denoising process via text-to-image and
audio-to-image cross-attention layers. For (2), the process begins with the
inversion of the input image using DDIM. Subsequently, extracted spatial
features and self-attention maps are integrated into the generation process,
complemented by audio-conditioned cross-attention maps to obtain the desired
changes.

Audio-Driven Image Generation and Editing. Incorporating audio into image
generation and editing has recently gained traction. Early studies
predominantly utilized GANs (Goodfellow et al., 2014). SGSIM (Lee et al.,
2022b) uses a two-step approach that augments the CLIP (Radford et al., 2021)
embedding space with audio embeddings via InfoNCE loss (Alayrac et al., 2020;
van den Oord et al., 2019), combining audio, image, and text. It then
manipulates StyleGAN’s (Karras et al., 2019) latent codes to produce images
compatible with the audio inputs. Robust-SGSIM enhances this method by adding
a KL divergence term to its loss function to better preserve image structure
and reduce bias. Sound2Scene (Sung-Bin et al., 2023) also utilizes InfoNCE
loss but combines it with BigGAN (Brock et al., 2018) for synthesis. AVStyle
(Li et al., 2022) introduces an audio-visual adversarial loss, focusing on
texture stylization and structural preservation using patch-wise contrastive
learning.

Diffusion-based methods have gained popularity recently. GlueGen (Qin et al.,
2023) is a flexible model handling multimodal inputs, merging linguistic
contexts via XLM-R (Conneau et al., 2020) and auditory elements through
AudioCLIP (Guzhov et al., 2022), aligning representations within the CLIP
framework to enable conditional image generation via (Rombach et al., 2022).
AudioToken (Yariv et al., 2023b) presents an audio embedder that replaces a
specific token’s CLIP embedding, enabling audio-guided image generation.
ImageBind (Girdhar et al., 2023) combines audio, image, and text modalities in
one embedding space using InfoNCE, also enabling audio-driven image synthesis.
Lee et al. (2023a) introduce a two-stage method, initially using an audio
captioning transformer for generating attention map, followed by direct sound
optimization to generate new images from the initial latent embeddings. Our
model distinguishes itself from these methods by adaptively tuning the
denoising process of the Stable Diffusion. In particular, we incorporate
audio-image cross-attention layers to enhance the semantic and visual
alignment between the synthesized images and the input audio.

Figure 3. Training pipeline of SonicDiffusion comprises two distinct stages:
(1) aligning audio features with CLIP’s semantic space, and (2) sound-driven
parameter-efficient tuning of the Stable Diffusion (SD) model. In stage (1),
the audio projector module is trained to transform audio clips into
semantically rich tokens, employing MSE and contrastive loss functions. Stage
(2) involves integrating gated cross-attention layers, which facilitate
interaction between image and audio modalities, into the existing SD
framework. Only these newly added layers, alongside the audio projector
module, are adjusted to enable audio-conditioned image synthesis.

Audio-Driven Video Generation. Recent years have seen growing interest in
generating videos from audio. Building on their earlier work (Lee et al.,
2022b), Lee et al. (2022a) developed a method combining a sound inversion
encoder with a StyleGAN generator, using recurrent blocks for fine texture
variations in video frames. TPoS (Jeong et al., 2023) similarly employs an
audio encoder with recurrent blocks, adding a temporal attention module for
better audio temporal understanding. This model uses the multimodal CLIP space
from (Lee et al., 2022b) for image synthesis. Another recent approach is
TempoTokens (Yariv et al., 2023a), which proposes to use specific tokens
generated from audio using an audio mapper. The model leverages an audio-
conditioned temporal attention mechanism and facilitates the generation of
videos by integrating these tokens into a pre-trained text-to-video diffusion
model.

##  3\. Method

In this work, we introduce SonicDiffusion, an approach that steers the process
of image generation and editing using auditory inputs. As depicted in Fig. 2,
our proposed approach has two principal components. The first module, termed
the Audio Projector, is designed to transform features extracted from an audio
clip into a series of inner space tokens. These tokens are subsequently
integrated into the image generation model through newly incorporated audio-
image cross-attention layers. Crucially, we maintain the original
configuration of the image generation model by freezing its existing layer
weights. This positions the added cross-attention layers as adapters, serving
as a parameter-efficient way to fuse the audio and visual modalities. Below,
we first present preliminaries about the Stable Diffusion (SD) model, which
serves as the basis for our implementation (Sec. 3.1). We then detail the
specifics of the audio projector (Sec. 3.3) and discuss the methodology
employed for the integration of audio tokens into the SD framework (Sec. 3.4).
Furthermore, we describe how we can extend our approach to sound-guided image
editing through feature injection (Sec. 3.5).

###  3.1. Preliminaries

Stable Diffusion (SD) is a latent text-to-image diffusion model (LDM) (Rombach
et al., 2022), integrating a variational autoencoder to project an input image
$\mathbf{x}$ into a reduced latent space $\mathbf{z}=E(\mathbf{x})$ via its
encoder $E$. The corresponding decoder $D$ reconstructs an image
$\hat{\mathbf{x}}=D(E(\mathbf{x}))$ from any given latent $\mathbf{z}$.
Central to SD is its diffusion UNet (Ronneberger et al., 2015), operating in
this latent space to progressively denoise input latents at each diffusion
step. An unconditional UNet $\epsilon_{\theta}$ is trained with the
established denoising loss introduced by Denoising Diffusion Probabilistic
Models (DDPM) (Ho et al., 2020):

(1) $\mathcal{L}_{LDM}=\mathbb{E}_{\mathbf{z}\sim E(\mathbf{x}),\epsilon\sim
N(0,1),t}\left[||\epsilon-\epsilon_{\theta}(\mathbf{z}_{t},t)||_{2}^{2}\right]$

where $\mathbf{z}_{t}$ represents the noise corrupted latent at time step $t$,
and $\mathbf{z}_{0}=E(\mathbf{x})$.

The conditional variant of LDM incorporates a cross-attention mechanism,
enabling it to be conditioned with different modalities. Specifically, SD
leverages text conditioning through embeddings generated by the CLIP text
encoder. Consequently, the denoising loss is adapted to accommodate this
conditioning, as follows:

(2) $\mathcal{L}_{LDM}=\mathbb{E}_{\mathbf{z}\sim E(\mathbf{x}),\epsilon\sim
N(0,1),t,\mathbf{y}}\left[||\epsilon-\epsilon_{\theta}(\mathbf{x},t,c_{\phi}((\mathbf{y}))||_{2}^{2}\right]$

where $\mathbf{y}$ is the conditioning text prompt, and $c_{\phi}$ is the text
encoder transforming $\mathbf{y}$ into an intermediate representation.

###  3.2. Training Pipeline

The overall training pipeline is illustrated in Fig. 3. SonicDiffusion employs
a two-stage training process, each serving a specific purpose in achieving
audio-driven image synthesis. The initial stage is dedicated to training the
audio projector module. The second stage of training focuses on integrating
gated cross-attention layers within the SD model. Below, we discuss the
details of data processing as well as the two-stage training framework.

###  3.3. Audio Projector

In training the Stable Diffusion model, a large text-image pair dataset was
used to learn a text-conditioning space semantically aligned with the image
space. To achieve similar alignment between audio and image domains, we
introduce the _Audio Projector_ module. This module, depicted in Fig. 4,
converts audio clips into tokens for additional conditioning input. It
utilizes the CLAP model (Elizalde et al., 2023), a state-of-the-art audio
encoder, to generate one-dimensional embeddings from audio inputs. These
embeddings are then transformed by an initial mapper with 1D convolutional and
deconvolutional layers into $K$ tokens of $C$ channels. These tokens are
further refined by four self-attention blocks to produce final
representations, which are compatible with the cross-attention layers of the
SD model, matching the dimensions of the text tokens.

Figure 4. Audio Projector utilizes CLAP audio encoder (Elizalde et al., 2023)
for initial feature extraction, followed by a mapper with 1D convolutions and
deconvolutions. Four self-attention layers are then integrated to enhance
learning effectiveness of the projector.

The training of Audio Projector includes a joint strategy involving both Mean
Squared Error (MSE) and contrastive loss functions. Specifically, for the
contrastive loss, we employ the InfoNCE (Alayrac et al., 2020) loss, treating
audio clips from the same class as positive pairs, and those from different
classes as negative pairs. Within each training batch, two audio clips,
$\mathbf{a}_{0}$ and $\mathbf{a}_{1}$, are randomly selected from the same
class, alongside $N$ audio clips, denoted as
$\left\\{\textbf{a}_{i}\right\\}_{i=2}^{N+1}$, from other classes. The
contrastive loss objective is thus formulated as:

(3)
$\mathcal{L}_{\text{InfoNCE}}=-\log\frac{\exp(\langle\mathbf{a}_{0},\mathbf{a}_{1}\rangle)}{\sum_{i=1}^{N+1}\exp(\langle\mathbf{a}_{0},\mathbf{a}_{i}\rangle)}$

This contrastive loss pushes different classes away from each other, resulting
in a well-defined semantic space. To enrich this space, we utilize audio-image
pairs from training dataset. For each audio clip, captions are extracted from
corresponding images using the BLIP v2 (Li et al., 2023) and their text
embeddings are computed using the text encoder of SD. An MSE loss is defined
as the squared norm difference between the audio tokens
$\mathbf{c}_{\text{audio}}$ and the text tokens $\mathbf{c}_{\text{text}}$:

(4)
$\mathcal{L}_{\text{MSE}}=\|\mathbf{c}_{\text{audio}}-\mathbf{c}_{\text{text}}\|_{2}^{2},$

The training of the audio projector combines these two losses with weights
$\alpha_{1}=1.0$ and $\alpha_{2}=0.25$, respectively:

(5)
$\mathcal{L}_{\text{AudioProjector}}=\alpha_{1}*\mathcal{L}_{\text{InfoNCE}}+\alpha_{2}*\mathcal{L}_{\text{MSE}}$

###  3.4. Gated Cross-Attention for Audio Conditioning

The diffusion UNet architecture of SD comprises blocks that include residual,
self-attention, and cross-attention layers. The self-attention layers capture
fine-grained spatial information, while the cross-attention layers establish
semantic relationships between the conditioning text and the generated image.
To effectively model the relationship between audio tokens and the image, we
introduce new gated cross-attention layers into this architecture (Fig. 5).
This modification aims to facilitate audio-conditioned image generation by
enabling image features to interact dynamically with audio tokens.

We have retained the encoder component of the UNet, while augmenting the
decoder with our newly introduced gated cross-attention layers. In this
configuration, only the audio projector and these new layers are set as
trainable, with the rest of the network remaining frozen. The pre-trained
audio projector is further refined through DDPM loss optimization, enhancing
the alignment between audio and image pairs. Within each decoder block of the
UNet, we process visual tokens $\mathbf{v}$, conditioning text tokens
$\mathbf{c}_{\text{text}}$, and audio tokens $\mathbf{c}_{\text{audio}}$ as
follows:

(6) $\displaystyle\mathbf{v}=\mathbf{v}+\text{Residual\\_Block}(\mathbf{v}),$
(7) $\displaystyle\mathbf{v}=\mathbf{v}+\text{Self\\_Attn}(\mathbf{v}),$ (8)
$\displaystyle\mathbf{v}=\mathbf{v}+\text{Cross\\_Attn}(\mathbf{v},\mathbf{c}_{\text{text}}),$
(9)
$\displaystyle\mathbf{v}=\mathbf{v}+\beta*\text{tanh}(\gamma)*\text{Cross\\_Attn}(\mathbf{v},\mathbf{c}_{\text{audio}})\;.$

Figure 5. Gated cross-attention module. Our method guides the diffusion
process by gated cross-attention and dense feed-forward layers added after
pre-trained text-conditioned layers, using audio tokens as keys/values and
ensuring training stability and quality.

A key novelty in our approach is the incorporation of a gating mechanism
within these new layers. This mechanism is crucial for maintaining the
original denoising efficiency of the model. We initialize the trainable
parameter $\gamma=0$ and set $\beta=1$. During training, $\gamma$ is optimized
to foster gradual learning. The conditioned DDPM loss objective is applied at
this stage, as defined below:

(10) $\mathcal{L}_{LDM}=\mathbb{E}_{\mathbf{z}\sim E(\mathbf{x}),\epsilon\sim
N(0,1),t,\mathbf{c}_{\text{audio}}}\left[||\epsilon-\epsilon_{\theta}(\mathbf{x},t,\mathbf{c}_{\text{audio}})||_{2}^{2}\right]$

Note that while null-text is used during tuning, our approach can incorporate
an optional text prompt at inference time to augment the generation process.

###  3.5. Audio-Guided Image Editing

By extending SD to perform audio conditioned image generation, our method can
be combined with existing SD-based image editing methods to enable audio-
guided image editing. We demonstrate this by using the Plug-and-Play features
method (Tumanyan et al., 2023). This approach involves injecting residual and
self-attention features, derived from inverting a source image, into a
subsequent image generation phase. This process ensures the preservation of
the structure of the source image while facilitating text-conditioned edits.

In line with this approach, we inject features from the residual layers
$f_{t}^{4}$ and attention maps $A_{t}^{l}$, obtained during the inversion step
$t$, into the corresponding editing phase as follows:

(11)
$\mathbf{z}_{t-1}=\epsilon_{\theta}(\mathbf{x}_{t},t,c_{\phi}(\mathbf{y}),{f_{t}^{4}},{{A_{t}^{l}}})$

Through the injection of self-attention features, our model captures the
relationship between spatial features, preserving critical layout and shape
details. By incorporating audio tokens as conditioning signals, it effectively
facilitates audio-guided image editing.

###  3.6. Implementation Details

Our experiments were conducted on a single NVIDIA V100 GPU. In both stages,
the AdamW optimizer with a weight decay of 1e-2 was utilized. During the first
stage, the audio projector was trained using a learning rate of 1e-4. For the
second stage, the cross-attention layers were trained at a learning rate of
1e-4, while the audio projector was adjusted to a lower rate of 1e-5. We used
a batch size of 6. For our image generation backbone, we used Stable Diffusion
version 1.4, the same backbone all our Stable-Diffusion based competitors
employ, to ensure a fair comparison in our evaluations. We employed the DDIM
sampler with 50 steps and set the classifier-free guidance scale as 8 for our
evaluation setup.

##  4\. Experimental Setup

Datasets. We carry out our experiments on the following datasets.

Landscape + Into the Wild. For the landscape scenes, we create a dataset by
combining videos from Into the Wild (Li et al., 2022) and High Fidelity Audio-
Video Landscape (Landscape) datasets (Lee et al., 2022a), both gathered from
YouTube. The Into the Wild dataset contains 94 hiking videos with nature
sounds, and the Landscape dataset is composed of 928 high-resolution videos of
varying lengths. The collected videos are split into 10 seconds, and a frame
is extracted from each interval along with its corresponding 10-second audio.
The final dataset contains 22,000 image-audio pairs.

Greatest Hits (Owens et al., 2016) is a video dataset that contains
distinctive sounds that come from various objects when a drumstick hits them.
Every hit in these videos is annotated by which material they have been hit
on. There are 17 material types, some of which are wood, metal, dirt, rock,
leaf, plastic, cloth, and paper. In total, there are 977 videos with 46,577
hit actions. Similar to the Landscape dataset, we extract 10 seconds of audio
clips and a frame from that interval to have matching audio and image pairs.
However, during the recording of this dataset, some materials are being hit by
turns in a random order within a couple of seconds. To have more consecutive
audios, we filter actions that have one material consecutively being hit. The
final dataset contains 3,385 image-audio pairs.

RAVDESS (Livingstone and Russo, 2018) dataset consists of 7,356 song-video
files of 24 male and female actors. The dataset includes speech samples
representing emotions like calm, happy, sad, angry, fearful, surprise, and
disgust. In our experiments, we took the song-video version of this dataset
and extracted a single frame along with its corresponding song from each video
since the videos were only 4-5 seconds long. The final dataset contains 1,008
frames with their paired audio.

Evaluation Metrics. In our study, we conduct a quantitative evaluation
focusing on two critical aspects: (i) the semantic relevance of generated
images to the input audio, and (ii) the photorealism of the samples. To assess
the semantic relevance, we employ three specific metrics: Audio-Image
Similarity (AIS), Image-Image Similarity (IIS), and Audio-Image Content (AIC),
as introduced in (Yariv et al., 2023b). For evaluating the sample quality, we
utilize the FID metric (Heusel et al., 2017). The detailed definitions of
these metrics are provided in the supplementary.

Table 1. Quantitative comparison of our proposed approach with existing audio-
conditioned image generation methods, focusing on AIS, AIC, ISS for semantic
consistency, and FID for image quality. Top scores are bolded, second best are
underlined. Pre-trained large-scale models and audio-to-video generation model
are highlighted with $*$ and ${\dagger}$ symbols, respectively.

| Model | AIS$\uparrow$ | AIC$\uparrow$ | IIS$\uparrow$ | FID$\downarrow$
---|---|---|---|---|---
Landscape + Into the Wild | SGSIM (Lee et al., 2022b) | .7224 | .2166 | .6898 | 220.3
Sound2Scene (Sung-Bin et al., 2023) | .7466 | .3672 | .8894 | 122.2
GlueGen (Qin et al., 2023) | .6632 | .4618 | .7357 | 133.0
ImageBind (Girdhar et al., 2023)∗ | .7209 | .4600 | .8044 | 159.1
CoDi (Tang et al., 2023)∗ | .7578 | .3618 | .7749 | 134.6
AudioToken (Yariv et al., 2023b) | .6983 | .2851 | .8592 | 141.3
TempoTokens (Yariv et al., 2023a)† | .7446 | .2242 | .6215 | 258.0
SonicDiffusion (Ours) | .7390 | .5436 | .8898 | 118.6
Greatest Hits | Sound2Scene (Sung-Bin et al., 2023) | .6536 | .3125 | .6693 | 143.1
SGSIM (Lee et al., 2022b) | .5065 | .2000 | .6612 | 239.2
GlueGen (Qin et al., 2023) | .5047 | .5050 | .5976 | 208.9
ImageBind (Girdhar et al., 2023)∗ | .5736 | .215 | .5889 | 186.0
CoDi (Tang et al., 2023)∗ | .5694 | .2325 | .5721 | 218.0
AudioToken (Yariv et al., 2023b) | .6552 | .2675 | .7541 | 123.6
| SonicDiffusion (Ours) | .6237 | .6050 | .7411 | 123.5
RAVDESS | Sound2scene (Sung-Bin et al., 2023) | 5053 | .1257 | 5301 | 140.3
SGSIM (Lee et al., 2022b) | .5090 | .1108 | .5094 | 155.4
Gluegen (Qin et al., 2023) | .5052 | .2252 | .5059 | 247.1
Imagebind∗ | .5802 | .2131 | .6332 | 248.8
CoDi (Tang et al., 2023)∗ | .5292 | .1396 | .5674 | 229.1
AudioToken (Yariv et al., 2023b) | .5009 | .1821 | .6409 | 279.4
| SonicDiffusion (Ours) | .5309 | .2316 | .8736 | 89.6

Competing Approaches. Our SonicDiffusion model is compared against seven
state-of-the-art methods: the GAN-based SGSIM (Lee et al., 2022b) and
Sound2Scene (Sung-Bin et al., 2023), Stable-Diffusion based GlueGen (Qin et
al., 2023), CoDi (Tang et al., 2023), AudioToken (Yariv et al., 2023b), and
TempoTokens (Yariv et al., 2023a), and DALLE$\cdot$2-based ImageBind (Girdhar
et al., 2023). Unlike other models, TempoTokens is designed for video
generation from audio clips; thus, we use the center frames of its generated
videos in our evaluation. SGSIM, GlueGen, AudioToken, and Sound2Scene are
fine-tuned on our datasets, while pre-trained models were used for large CoDi
and ImageBind models. No fine-tuning is needed for TempoTokens, as it is
already trained on the Landscape dataset. Further model and fine-tuning
details are in the supplementary material. We also compare our image editing
results with the text-based PnP model (Tumanyan et al., 2023), using audio
class labels as text prompts. This comparison shows the advantage of injecting
audio-derived semantic information into image manipulation, as opposed to
relying solely on text labels.

##  5\. Results

Figure 6. Comparison against the state-of-the-art audio-driven image synthesis
methods. Our model generates images that closely align with the semantics of
the input audio clips, surpassing all other existing methods in performance
and fidelity.

Audio-Driven Image Generation. Table 1 shows quantitative evaluation results.
Our model consistently outperforms competing approaches in photorealism,
achieving the best FID scores across all datasets. Notably, our landscape
images exhibit a good alignment with the audio semantics. In the Greatest Hits
dataset, our model achieves the highest AIC score and the second-highest IIS
score, further showing its effectiveness in audio-driven image generation.

In Fig. 6, we present a visual comparison between our model and the state-of-
the-art methods. Our model demonstrates a superior capability in generating
images that are not only visually coherent but also closely aligned with the
input audio cues. For example, landscape images synthesized by our model
capture the essence of the target scenes more accurately. When processing
audio encoding material properties, our model synthesizes images featuring
objects or elements that reflect the desired attributes. Additionally, the
human face images generated from speech audio effectively mirror the
corresponding vocal tonations, capturing the intended emotions.

Moreover, the versatility of our in handling mixed-modality inputs, blending
both text and audio, is illustrated in Fig. 7. These results demonstrate our
method’s proficiency in interpreting and fusing concepts from varied
modalities. For example, it successfully merges a ‘watercolor painting’ text
prompt with a ‘squishing water’ audio input, resulting in accurately composed
and distinct image.

Figure 7. Image generation results using both text and audio. Our model
enables mixing text and audio modalities while synthesizing novel images.
Figure 9. Comparison against the state-of-the-art audio-driven image editing
methods. Our model generates images that closely align with the semantics of
the input audio clips, surpassing all other existing methods in performance
and fidelity. Figure 8. Audio-guided image editing results. Our model
successfully performs manipulations that semantically reflect the audio
content while maintaining the content of the source images.

Audio-Driven Image Editing. Our model also excels in editing real images based
on audio clips. Fig. 8 illustrates various examples of our method’s image
editing results. These show our model’s ability in modifying image content to
align with the semantics of the provided audio. For example, when editing a
landscape image with the sounds of ‘waterfall burbling’, ‘fire cracking’ and
‘rain’, our method makes the necessary changes in the overall look of the
input image. In another instance, as shown in the second row, the model
successfully modifies the visual appearance of the objects based on the audio
inputs reflecting material characteristics like ‘plastic bag’, ‘rock’ and
‘paper’.

In Fig. 9, we show a comparative analysis of our model’s editing capabilities
against existing methods, including the text-based image editing PnP model,
which serves as a foundational component of our framework. For this
comparison, we employ class label information from audio inputs as text
prompts in the PnP model. The results show that SonicDiffusion surpasses
others in manipulation quality and style transfer accuracy. For instance, in
editing the second image, methods like AudioToken and GlueGen not only fall
short in achieving convincing manipulations but also introduce artifacts and
color shifts. Similarly, the text-based PnP model fails to capture the full
extent of the modifications implied by the audio inputs, unlike our model.

(a) Sound mixing

(b) Volume changes

Figure 10. Controlling Editing with Sound Manipulation: Our model can (a)
blend two sounds via linear interpolation of audio embeddings, and (b) adjust
manipulation intensity by varying sound volume.

Quantitative results in Table 2 further validate our approach. SonicDiffusion
achieves the highest scores in AIS and ISS metrics, and the second-highest in
IIS, confirming its ability to accurately manipulate images in response to
audio cues. It also leads in FID scores, indicating better image quality
compared to existing approaches. In a user study, our results are preferred by
a majority of participants over competing approaches. Specifically, 56.05% of
19 participants favored our method over GlueGen (17.37%) and AudioToken
(26.58%) for the Greatest Hits samples. Similarly, for the Landscape+Into the
Wild samples, 64.95% of 16 participants rated our results higher than those by
GlueGen (22.83%) and AudioToken (12.23%). Please check the supplementary
material for the details of the user study.

Table 2. Quantitative Results for Image Editing. We evaluate semantic
consistency with AIS, AIC, and ISS metrics, and image quality with FID. The
highest scores are in bold, with second-best scores underlined. The text-based
model is highlighted with a ${\ddagger}$ symbol.

| Model | AIS $\uparrow$ | AIC $\uparrow$ | IIS $\uparrow$ | FID $\downarrow$
---|---|---|---|---|---
L + ItW | PnP (Tumanyan et al., 2023)‡ | – | .6833 | – | 239.2
GlueGen (Qin et al., 2023) | .5869 | .2254 | .6491 | 250.3
AudioToken (Yariv et al., 2023b) | .6059 | .1849 | .7405 | 206.8
SonicDiffusion (Ours) | .6496 | .3745 | .7933 | 172.4
GH | PnP (Tumanyan et al., 2023)‡ | – | .6604 | – | 199.5
Gluegen (Qin et al., 2023) | .4680 | .2275 | .5425 | 208.9
AudioToken (Yariv et al., 2023b) | .4749 | .1450 | .5441 | 206.8
SonicDiffusion (Ours) | .5491 | .3575 | .6255 | 152.7

We also show the ability of our model to control image editing outcomes
through simple manipulations of input sounds. Our model can blend
characteristics from two distinct audio sources by linearly interpolating
their audio embeddings, as shown in Fig. 10(a). This results in outputs
representing seamlessly transitions between features corresponding to each
audio input. Additionally, as illustrated in Fig. 10(b), our model responds to
changes in audio volume, allowing for precise manipulation of image content.
The degree of change directly correlates with the volume adjustments, offering
users detailed control over the audio’s influence on the image.

##  6\. Ablations

To explore the influence of loss functions and training strategies on our
SonicDiffusion model, we conducted an extensive series of ablation studies.
These experiments aim to illuminate how different components and methodologies
affect the model’s efficacy in audio-visual alignment and overall performance.

Impact of Loss Functions in the Training of SonicDiffusion’s Audio Projector.
In the first training stage of our model, we employ two loss functions: (i)
contrastive loss and (ii) mean squared error (MSE) loss. We conducted ablation
studies to understand the effects of omitting one of these losses. Table 3
presents the results of these experiments. When the contrastive loss is
removed, our network struggles to distinguish between different audio
conditionings. On the other hand, omitting the MSE loss results in the model
still being able to differentiate between various audio cues, but with a
longer convergence time and reduced accuracy. Including the MSE loss is
particularly crucial as it facilitates the alignment of the audio space with
the pre-trained CLIP space, though not through a direct mapping due to the
simultaneous application of the contrastive loss. In essence, the contrastive
loss adds another layer of complexity to this alignment. After completing the
first stage of training, we continue to train the audio projector module but
with a reduced learning rate. This step is vital as it enables the network to
directly capture the semantic correlations between audio and image, enhancing
the overall effectiveness of our SonicDiffusion model.

Table 3. Impact of loss functions used in training Audio Projector on the performance of SonicDiffusion. We report the performance using both contrastive and MSE losses (first row), using only the MSE loss (second row), and using only the contrastive loss (third row). |  | AIS $\uparrow$ | AIC $\uparrow$ | IIS $\uparrow$ | FID $\downarrow$
---|---|---|---|---|---
L + ItW | SonicDiffusion | .7390 | .5436 | .8898 | 118.6
w/o Contrastive Loss | .7245 | .3919 | .8432 | 121.6
w/o MSE Loss | .6827 | .4018 | .7767 | 124.9
GH | SonicDiffusion | .6237 | .6050 | .7411 | 123.5
w/o Contrastive Loss | .6560 | .3694 | .7264 | 133.4
w/o MSE Loss | .6269 | .4027 | .6826 | 142.5

Impact of Different Training Strategies on the Performance. Our goal is to
establish a conditioning space that effectively guides the diffusion process.
Leveraging embeddings from a pre-trained, well-represented space is a
recognized strategy in diffusion models. Large text encoders such as CLIP
(Radford et al., 2021) and T5 (Raffel et al., 2020) are commonly employed for
this purpose due to their robust pre-trained capabilities. In our approach, we
extract audio features from CLAP (Elizalde et al., 2023) and consider an
additional learning phase to map these features from the CLAP embedding domain
to the CLIP embedding domain. To assess the impact of various training
strategies on our model’s performance, we conduct a series of experiments.
These experiments focus on evaluating changes in performance under different
conditions: representing audio features with a single token, mapping CLAP
embeddings to CLIP space solely using diffusion loss (thereby omitting the
first stage of training), and freezing the Audio Projector’s parameters during
the second stage of training. The outcomes of these experiments are summarized
in Table 4. We found that relying exclusively on the diffusion loss for
learning the mapping led to a notable decrease in the model’s performance.
Notably, the model seems to lose its ability to distinguish between different
audio inputs. Freezing the Audio Projector does not consistently improve
results and can even lower image quality, as indicated by the FID scores.
Additionally, we also observed that representing audio with just a single
token, as in AudioToken (Yariv et al., 2023b), leads to worse performance.

Table 4. Impact of various training choices on the performance of our proposed
SonicDiffusion model.

|  | AIS $\uparrow$ | AIC $\uparrow$ | IIS $\uparrow$ | FID $\downarrow$
---|---|---|---|---|---
L + ItW | SonicDiffusion | .7390 | .5436 | .8898 | 118.6
w/o 1st Stage Training | .6013 | .0600 | .5160 | 116.7
w/ Audio Proj. Frozen Stage 2 | .7488 | .6020 | .8920 | 128.3
|  w/ Single Token Audio Repr. | .7042 | .4618 | .8281 | 136.6
GH | SonicDiffusion | .6237 | .6050 | .7411 | 123.5
w/o 1st Stage Training | .4376 | .0888 | .5128 | 177.6
w/ Audio Proj. Frozen Stage 2 | .5849 | .5972 | .7188 | 128.3
|  w/ Single Token Audio Repr. | .6863 | .4625 | .7647 | 119.0

##  7\. Limitations and Failure Cases

Our SonicDiffusion model demonstrates an encouraging capacity for image
generation that aligns with the semantic context of provided audio input.
Nonetheless, as illustrated in Fig. 11, it is not without its shortcomings. In
particular, the model occasionally does not understand the essence of the
audio content. One significant source of these suboptimal outcomes is the
artifacts intrinsically produced by the underlying Stable Diffusion framework.
This is evident when processing the Greatest Hits dataset, where the model’s
rendition of hands and objects in motion often falls short. Similarly, with
the RAVDESS dataset, which primarily features human facial expressions, the
model sometimes struggles with accurately rendering facial features, such as
teeth, resulting in clearly noticeable anomalies. When it comes to image
editing, the shortcomings become more pronounced with the employment of the
DDIM inversion method. This technique is liable to obliterate delicate details
in intricate scenes or integrating extraneous elements, thereby skewing the
image’s authentic structure. Instances of such editing impediments are
exemplified in the third row of Fig. 11.

Figure 11. Limitations of the SonicDiffusion model. The figure illustrates
various scenarios where the SonicDiffusion model does not adequately perform
in the generation or modification of images based on audio cues. Issues arise
from imprecise semantic interpretations of audio signals, as well as from
artifacts introduced during the Stable Diffusion process. In the context of
image editing, our model may inadvertently replace original content, conduct
ineffective semantic modifications, or alter the subject’s identity. Notably,
some of the editing challenges are attributed to the use of DDIM inversion,
which can result in the insertion of extraneous elements, the omission of fine
details, or significant structural disruptions.

##  8\. Conclusion

Our SonicDiffusion model introduces an original framework for sound-guided
image generation and editing, leveraging the robustness of the SD model. We
achieve seamless integration of audio modality with minimal training overhead
by adapting cross-attention layers in SD to respond to audio prompts. Our
method allows for the incorporation of audio context that aligns with the
input sound, effectively enhancing both image generation and editing tasks.
The results demonstrate our framework’s strong performance in multimodal image
synthesis, demonstrating significant advancements in the field of multimodal
image synthesis.

###### Acknowledgements.

This work has been partially supported by AI Fellowships to B. C. Biner
provided by the KUIS AI Center, by BAGEP 2021 Award of the Science Academy to
A. Erdem, by GEBIP 2018 Award of the Turkish Academy of Sciences, and by an
Adobe research gift.

## References

  * (1)
  * Alayrac et al. (2020) Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. 2020.  Self-supervised multimodal versatile networks. In _Advances in Neural Information Processing Systems (NeurIPS)_ , Vol. 33. 25–37.
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018.  Large Scale GAN Training for High Fidelity Natural Image Synthesis. In _International Conference on Learning Representations (ICLR)_.
  * Brooks et al. (2023) Tim Brooks, Aleksander Holynski, and Alexei A Efros. 2023.  Instructpix2pix: Learning to follow image editing instructions. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 18392–18402.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners. In _Advances in Neural Information Processing Systems (NeurIPS)_ , Vol. 33. 1877–1901.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised Cross-lingual Representation Learning at Scale. In _The 58th Annual Meeting of the Association for Computational Linguistics (ACL)_. 8440–8451.
  * Elizalde et al. (2023) Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023.  CLAP: Learning audio concepts from natural language supervision. In _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 1–5.
  * Girdhar et al. (2023) Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023.  ImageBind: One embedding space to bind them all. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 15180–15190.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014.  Generative Adversarial Nets. In _Advances in Neural Information Processing Systems (NeurIPS)_ , Vol. 27.
  * Guzhov et al. (2022) Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. 2022.  AudioCLIP: Extending clip to image, text and audio. In _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 976–980.
  * Hertz et al. (2022) Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. 2022.  Prompt-to-Prompt Image Editing with Cross-Attention Control. In _International Conference on Learning Representations (ICLR)_.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models. In _Advances in Neural Information Processing Systems (NeurIPS)_ , Vol. 33. 6840–6851.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP. In _International Conference on Machine Learning (ICML)_. 2790–2799.
  * Jeong et al. (2023) Yujin Jeong, Wonjeong Ryoo, Seunghyun Lee, Dabin Seo, Wonmin Byeon, Sangpil Kim, and Jinkyu Kim. 2023.  The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion. In _IEEE/CVF International Conference on Computer Vision (ICCV)_. 7822–7832.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A style-based generator architecture for generative adversarial networks. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 4401–4410.
  * Kawar et al. (2023) Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. 2023.  Imagic: Text-based real image editing with diffusion models. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 6007–6017.
  * Lee et al. (2022a) Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Chanyoung Kim, Won Jeong Ryoo, Sang Ho Yoon, Hyunjun Cho, Jihyun Bae, Jinkyu Kim, and Sangpil Kim. 2022a.  Sound-guided semantic video generation. In _European Conference on Computer Vision (ECCV)_. Springer, 34–50.
  * Lee et al. (2023b) Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Sang Ho Yoon, Jinkyu Kim, and Sangpil Kim. 2023b.  Robust Sound-Guided Image Manipulation.  _arXiv preprint arXiv:2208.14114_ (2023).
  * Lee et al. (2022b) Seung Hyun Lee, Wonseok Roh, Wonmin Byeon, Sang Ho Yoon, Chanyoung Kim, Jinkyu Kim, and Sangpil Kim. 2022b.  Sound-Guided Semantic Image Manipulation. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 3377–3386.
  * Lee et al. (2023a) Taegyeong Lee, Jeonghun Kang, Hyeonyu Kim, and Taehwan Kim. 2023a.  Generating Realistic Images from In-the-wild Sounds. In _IEEE/CVF International Conference on Computer Vision (ICCV)_. 7160–7170.
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023.  BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.  _arXiv preprint arXiv:2301.12597_ (2023).
  * Li et al. (2022) Tingle Li, Yichen Liu, Andrew Owens, and Hang Zhao. 2022.  Learning visual styles from audio-visual associations. In _European Conference on Computer Vision (ECCV)_. Springer, 235–252.
  * Livingstone and Russo (2018) Steven R. Livingstone and Frank A. Russo. 2018.  The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) [Data set].  _PLoS ONE_ 13, 5 (2018), e0196391.  https://doi.org/10.5281/zenodo.1188976
  * Meng et al. (2021) Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. 2021.  SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. In _International Conference on Learning Representations (ICLR)_.
  * Mokady et al. (2023) Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2023.  Null-text inversion for editing real images using guided diffusion models. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 6038–6047.
  * Mou et al. (2023) Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. 2023.  T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models.  _arXiv preprint arXiv:2302.08453_ (2023).
  * Nichol et al. (2022) Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. 2022.  GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In _International Conference on Machine Learning (ICML)_. PMLR, 16784–16804.
  * Owens et al. (2016) Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William T Freeman. 2016.  Visually indicated sounds. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 2405–2413.
  * Parmar et al. (2023) Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. 2023.  Zero-shot image-to-image translation. In _ACM SIGGRAPH 2023 Conference Proceedings_. 1–11.
  * Qin et al. (2023) Can Qin, Ning Yu, Chen Xing, Shu Zhang, Zeyuan Chen, Stefano Ermon, Yun Fu, Caiming Xiong, and Ran Xu. 2023.  GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation. In _IEEE/CVF International Conference on Computer Vision (ICCV)_. 23085–23096.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International Conference on Machine Learning (ICML)_. 8748–8763.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ 21, 140 (2020), 1–67.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with CLIP latents.  _arXiv preprint arXiv:2204.06125_ 1, 2 (2022), 3.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-Resolution Image Synthesis With Latent Diffusion Models. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 10684–10695.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015.  U-net: Convolutional networks for biomedical image segmentation. In _Medical Image Computing and Computer-Assisted Intervention (MICCAI)_. 234–241.
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022\.  Photorealistic text-to-image diffusion models with deep language understanding. In _Advances in Neural Information Processing Systems (NeurIPS)_ , Vol. 35. 36479–36494.
  * Sung-Bin et al. (2023) Kim Sung-Bin, Arda Senocak, Hyunwoo Ha, Andrew Owens, and Tae-Hyun Oh. 2023.  Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 6430–6440.
  * Tang et al. (2023) Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. 2023.  Any-to-Any Generation via Composable Diffusion.  _arXiv preprint arXiv:2305.11846_ (2023).
  * Tumanyan et al. (2023) Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. 2023.  Plug-and-play diffusion features for text-driven image-to-image translation. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 1921–1930.
  * van den Oord et al. (2019) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019.  Representation Learning with Contrastive Predictive Coding.  _arXiv preprint arXiv:1807.03748_ (2019).
  * Yariv et al. (2023a) Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. 2023a.  Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation.  arXiv:2309.16429 [cs.LG]
  * Yariv et al. (2023b) Guy Yariv, Itai Gat, Lior Wolf, Yossi Adi, and Idan Schwartz. 2023b.  Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation. In _Annual Conference of the International Speech Communication Association (INTERSPEECH)_. 5446–5450.  https://doi.org/10.21437/Interspeech.2023-852
  * Ye et al. (2023) Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. 2023.  IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models.  _arXiv preprint arxiv:2308.06721_ (2023).
  * Zhang et al. (2023) Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. 2023.  LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention.  _arXiv preprint arXiv:2303.16199_ (2023).