# Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos

Kun Su1 Kaizhi Qian2 Eli Shlizerman1 Antonio Torralba3 Chuang Gan2,4
1University of Washington 2MIT-IBM Watson AI Lab 3MIT 4UMass Amherst
Work done while interning at MIT-IBM Watson AI Lab

###### Abstract

Modeling sounds emitted from physical object interactions is critical for
immersive perceptual experiences in real and virtual worlds. Traditional
methods of impact sound synthesis use physics simulation to obtain a set of
physics parameters that could represent and synthesize the sound. However,
they require fine details of both the object geometries and impact locations,
which are rarely available in the real world and can not be applied to
synthesize impact sounds from common videos. On the other hand, existing
video-driven deep learning-based approaches could only capture the weak
correspondence between visual content and impact sounds since they lack of
physics knowledge. In this work, we propose a physics-driven diffusion model
that can synthesize high-fidelity impact sound for a silent video clip. In
addition to the video content, we propose to use additional physics priors to
guide the impact sound synthesis procedure. The physics priors include both
physics parameters that are directly estimated from noisy real-world impact
sound examples without sophisticated setup and learned residual parameters
that interpret the sound environment via neural networks. We further implement
a novel diffusion model with specific training and inference strategies to
combine physics priors and visual information for impact sound synthesis.
Experimental results show that our model outperforms several existing systems
in generating realistic impact sounds. More importantly, the physics-based
representations are fully interpretable and transparent, thus enabling us to
perform sound editing flexibly. We encourage the readers to visit our project
page 111https://sukun1045.github.io/video-physics-sound-diffusion/ to watch
demo videos with audio turned on to experience the results.

##  1 Introduction

Figure 1: The physics-driven diffusion model takes physics priors and video
input as conditions to synthesize high-fidelity impact sound. Please also see
the supplementary video and materials with sample results.

Automatic sound effect production has become demanding for virtual reality,
video games, animation, and movies. Traditional movie production heavily
relies on talented Foley artists to record many sound samples in advance and
manually perform laborious editing to fit the recorded sounds to visual
content. Though we could obtain a satisfactory sound experience at the cinema,
it is labor-intensive and challenging to scale up the sound effects generation
of various complex physical interactions.

Recently, much progress has been made in automatic sound synthesis, which can
be divided into two main categories. The first category is physics-based modal
synthesis methods [52, 38, 39], which are often used for simulating sounds
triggered by various types of object interactions. Although the produced
sounds can reflect the rich variations of interactions and the geometry of the
sound objects, such approaches require a sophisticated designed environment to
perform physics simulation and compute a set of physics parameters for sound
synthesis. It is, therefore, impractical to scale up for a complicated scene
because of the time-consuming parameter selection procedure. On the other
hand, due to the availability of a significant amount of impact sound videos
in the wild, training deep learning models for impact sound synthesis turns
out to be a promising direction. Indeed, several works have shown promising
results in various audio-visual applications [66]. Unfortunately, most
existing video-driven neural sound synthesis methods [65, 8] apply end-to-end
black box model training and lack of physics knowledge which plays a
significant role in modeling impact sound because a minor change in the impact
location could exert a significant difference in the sound generation process.
As a result, these methods are prone to learning an average or smooth audio
representation that contains artifacts, which usually leads to generating
unfaithful sound.

In this work, we aim to address the problem of automatic impact sound
synthesis from video input. The main challenge for the learning-based approach
is the weak correspondence between visual and audio domains since the impact
sounds are sensitive to the undergoing physics. Without further physics
knowledge, generating high-fidelity impact sounds from videos alone is
insufficient. Motivated by physics-based sound synthesis methods using a set
of physics mode parameters to represent and re-synthesize impact sounds, we
design a physics prior that could contain sufficient physics information to
serve as a conditional signal to guide the deep generative model synthesizes
impact sounds from videos. However, since we could not perform physics
simulation on raw video data to acquire precise physics parameters, we
explored estimating and predicting physics priors from sounds in videos. We
found that such physics priors significantly improve the quality of
synthesized impact sounds. For deep generative models, recent successes in
image generation such as DALL-E 2 and Imagen [46] show that Denoising
Diffusion Probabilistic Models (DDPM) outperform GANs in fidelity and
diversity, without training instability and mode collapse issues. While the
idea of the denoising process is naturally fitted with sound signals, it is
unclear how video input and physics priors could jointly condition the DDPM
and synthesize impact sounds.

To address all these challenges, we propose a novel system for impact sound
synthesis from videos. The system includes two main stages. In the first
stage, we encode physics knowledge of the sound using physics priors,
including estimated physical parameters using signal processing techniques and
learned residual parameters interpreting the sound environment via neural
networks. In the second stage, we formulate and design a DDPM model
conditioned on visual input and physics priors to generate a spectrogram of
impact sounds. Since the physics priors are extracted from the audio samples,
they become unavailable at the inference stage. To solve this problem, we
propose a novel inference pipeline to use test video features to query a
physics latent feature from the training set as guidance to synthesize impact
sounds on unseen videos. Since the video input is unseen, we can still
generate novel impact sounds from the diffusion model even if we reuse the
training set’s physics knowledge. In summary, our main contributions to this
work are:

  * •

We propose novel physics priors to provide physics knowledge to impact sound
synthesis, including estimated physics parameters from raw audio and learned
residual parameters approximating the sound environment.

  * •

We design a physics-driven diffusion model with different training and
inference pipeline for impact sound synthesis from videos. To the best of our
knowledge, we are the first work to synthesize impact sounds from videos using
the diffusion model.

  * •

Our approach outperforms existing methods on both quantitative and qualitative
metrics for impact sound synthesis. The transparent and interpretable
properties of physics priors unlock the possibility of interesting sound
editing applications such as controllable impact sound synthesis.

##  2 Related Work

###  2.1 Sound Synthesis from Videos

Sound synthesis has been an ongoing research theme with a long history in
audio research. Traditional approaches mainly use linear modal synthesis to
generate rigid body sounds [52]. While such methods could produce sounds
reflecting the rich variations of interactions and the geometry of the
sounding objects, setting up good initial parameters for the virtual sounding
materials in the modal analysis is time-consuming and non-intuitive. When
faced with a complicated scene consisting of many different sounding
materials, the parameter selection procedure can quickly become prohibitively
expensive and tedious [43]. In recent years, deep learning approaches have
been developed for sound synthesis. Owens et al. [40] investigated predicting
the sound emitted by interacting objects with a drumstick. They first used a
neural network to predict sound features and then performed an exemplar-based
retrieval algorithm instead of directly generating the sound. Instead of
performing retrieval, our work directly generates the impact sounds. Chen et
al. [6] proposed using conditional generative adversarial networks for cross-
modal generation on lab-collected music performance videos. Zhou et al. [65]
introduced a SampleRNN-based method to directly predict a generated waveform
from an unconstrained video dataset that contains ten types of sound recorded
in the wild. Chen et al. [5] proposed a perceptual loss to improve the audio-
visual semantic alignment. Chen et al. [8] introduced an information
bottleneck to generate visually aligned sound. Several works [49, 11, 50] also
attempt to generate music sound from videos.

###  2.2 Audio-visual learning

In recent years, methods for multi-modality learning have shown significance
in learning joint representation for downstream tasks [42], and unlocked novel
cross-modal applications such as visual captioning [61, 31], visual question
answering (VQA) [56, 9], vision language navigation [2], spoken question
answering (SQA) [57, 7], healthcare AI [29, 32, 58, 60, 59], etc. In this
work, we are in the field of audio-visual learning which deals with exploring
and leveraging the correlation of both audio and video simultaneously. For
example, given unlabeled training videos, Owens et al. [41] used sound
clusters as supervision to learn visual feature representation, and Aytar et
al. [4] utilized the scene to learn the audio representations. Follow-up work
[3] further investigated jointly learning the visual and audio representation
using a visual-audio correspondence task. Instead of learning feature
representations, recent works have also explored localizing sound sources in
images or videos [24, 20, 47], biometric matching [36], visual-guided sound
source separation [63, 12, 16, 54], auditory vehicle tracking [15], multi-
modal action recognition [33, 34, 17], audio inpainting [64], emotion
recognition [1], audio-visual event localization [51], multi-modal physical
scene understanding [13], audio-visual co-segmentation [45], and audio-visual
embodied navigation [14].

###  2.3 Diffusion Model

The recently explored diffusion probabilistic models (DPMs) [48] has served as
a powerful generative backbone that achieves promising results in both
unconditional and conditional generation [26, 35, 27, 21, 37, 10, 22],
outperforming GANs in fidelity and diversity, without training instability and
mode collapse issues. Compared to the unconditional case, conditional
generation is usually applied in more concrete and practical cross-modality
scenarios. Most existing DPM-based conditional synthesis works [19, 10] learn
the connection between the conditioning and the generated data implicitly by
adding a prior to the variational lower bound. Most of these methods focus on
the image domain, while audio data differs in its long-term temporal
dependencies. Several works have explored to use of diffusion models for text-
to-speech (TTS) synthesis [25, 23]. Unlike the task of text-to-speech
synthesis, which contains a strong correlation between phonemes and speech,
the correspondences between impact sounds and videos are weak. Therefore it is
non-trivial to directly apply a conditional diffusion model to impact sound
synthesis from videos. In this work, we found that only video condition is
insufficient to synthesize high-fidelity impact sounds and additionally apply
physics priors significantly improve the results. Moreover, due to the
difficulty of predicting physics priors from video, we propose different
training and testing strategies that could benefit the information of physics
priors but also synthesize new impact sounds from the video input.

##  3 Method

Figure 2: Reconstruction of physics priors by two components: $1$) We estimate
a set of physics parameters (frequency, power, and decay rate) via signal
processing techniques. $2$) We predict residual parameters representing the
environment by a transformer encoder. A reconstruction loss is applied to
optimize all trainable modules.

Our method includes two main components: (a) physics priors reconstruction
from sound (shown in Fig. 2), and (b) a physics-driven diffusion model for
impact sound synthesis (shown in Fig. 3). We first show how we can acquire
physics priors from sounds In (a). Then in (b), we use reconstructed physics
priors as additional information to the video input and guide the diffusion
model to learn impact sound synthesis. Since no sound is available during the
test time, we use different training and inference strategies to keep the
benefit of physics priors and generate novel impact sounds.

###  3.1 Reconstruct Physics Priors From Sound

We aim to reconstruct physics from the sound. There are two modules: $1$) the
physics parameters estimation extracting modes parameters from audio waveform,
and $2$) the residual parameters prediction learning to encode the environment
information such as background noise and reverberation using neural networks.

Physics Parameters Estimation. The standard linear modal synthesis technique
is frequently used for modeling physics-based sound synthesis. The
displacement $x$ in such a system can be calculated with the following linear
deformation equation:

$\displaystyle M\ddot{x}+C\dot{x}+Kx=F,$ (1)

where $F$ represents the force, $M,C$ and $K$ respectively represent the mass,
damping and stiffness matrices. After solving the generalized eigenvalue
problem $KU=\Lambda MU$, the system can be decoupled into the following form:

$\displaystyle\ddot{q}+(\alpha I+\beta\Lambda)\dot{q}+\Lambda q=U^{T}F$ (2)

where $\Lambda$ is a diagonal matrix, containing the eigenvalues, $U$ is the
eigenvector matrix that transforms $x$ into the decoupled deformation bases
$q$ with $x=Uq$.

The solution to this decoupled system is a bank of modes, i.e., damped
sinusoidal waves. The $i$-th mode can be expressed by:

$\displaystyle q_{i}=p_{i}e^{-\lambda_{i}t}\sin(2\pi f_{i}t+\theta_{i})$ (3)

where $f_{i}$ is the frequency of the mode, $\lambda_{i}$ is the decaying
rate, $p_{i}$ is the excited power, and $\theta_{i}$ is the initial phase. It
is also common to represent $q_{i}$ under the decibel scale and we have

$\displaystyle q_{i}=10^{({p_{i}-\lambda_{i}t})/20}\sin(2\pi
f_{i}t+\theta_{i}).$ (4)

The frequency, power, and decay rate together define the physics parameter
feature $\phi$ of mode $i$: $\phi=(f_{i},p_{i},\lambda_{i})$ and we ignore
$\theta_{i}$ because it can be safely treated as zero in our estimation
process, where the object is initially at rest and struck at $t=0$.

Given a recorded audio waveform $s\in\mathbb{R}^{T}$, from which we first
estimate physics parameters, namely, a set of damped sinusoids with constant
frequencies, powers, and decay rates. We first compute the log-spectrogram
magnitude $S\in\mathbb{R}^{D\times N}$ of the audio by short-time-Fourier-
transform (STFT), where $D$ is the number of frequency bins and $N$ is the
number of frames. To capture sufficient physics parameters, we set the number
of modes to be equal to the number of frequency bins. Within the range of each
frequency bin, we identify the peak frequency $f$ from the fast Fourier
transform (FFT) magnitude result of the whole audio segment. Next, we extract
the magnitude at the first frame in the spectrogram to be the initial power
$p$. Finally, we compute the decay time $\lambda$ for the mode according to
the temporal bin when it reaches the silence ($-80$dB). At this point, we
obtain $D$ modes physics parameters
$\\{(f_{i},p_{i},\lambda_{i})\\}^{D}_{i=1}$ and we can re-synthesize an audio
waveform $\hat{s}$ using equation 4.

Residual Parameters Prediction. While the estimated modes capture most of the
components of the impact sound generated by physical object interactions, the
recorded audio in the wild has complicated residual components such as
background noise and reverberation dependent on the sound environment which is
critical for a real and immersive perceptual experience. Here we propose a
learning-based approach to model such residual parameters. We approximate the
sound environment component with exponentially decaying filtered noise. We
first randomly generate a Gaussian white noise $\mathcal{N}(0,1)$ signal and
perform a band-pass filter (BPF) to split it into $M$ bands. Then, for each
band $m$, the residual component is formulated as

$\displaystyle R_{m}=10^{(-\gamma t)/20}\text{BPF}(\mathcal{N}(0,1))_{m}$ (5)

The accumulated residual components $R$ is a weighted sum of subband residual
components

$\displaystyle R=\sum^{M}_{m=1}w_{m}R_{m},$ (6)

where $w_{m}$ is the weight coefficient of band $m$ residual component. Given
the log-spectrogram $S\in\mathbb{R}^{D\times N}$ as input, we use a
transformer-based encoder to encode each frame of the $S$. the output features
are then averaged and two linear projections are used to estimate
$\gamma\in\mathbb{R}^{M}$ and $w\in\mathbb{R}^{M}$. We minimize the error
between $\hat{s}+R$ and $s$ by a multi-resolution STFT loss $L_{\text{mr-
stft}}(\hat{s}+R,s)$ which has been shown effective in modeling audio signals
in the time domain [55]. By estimating physics parameters and predicting
residual parameters, we obtain the physics priors and it is ready to be a
useful condition to guide the impact sound synthesis model to generate high-
fidelity sounds from videos.

###  3.2 Physics-Driven Diffusion Models

Figure 3: Overview of the physics-driven diffusion model for impact sound
synthesis from videos. (left) During training, we reconstruct physics priors
from audio samples and encode them into a physics latent. Besides, we use a
visual encoder to extract visual latent from the video input. We apply these
two latents as conditional inputs to the U-Net spectrogram denoiser. (right)
During testing, we extract the visual latent from the test video and use it to
query a physics latent from the key-value pairs of visual and physics latents
in the training set. Finally, the physics and visual latents are used as
conditional inputs to the denoiser and the denoiser iteratively generates the
spectrogram.

With the physics priors and video inputs, we propose a conditional Denoising
Diffusion Probabilistic Model (DDPM) for impact sound synthesis. The model
converts the noise distribution to a spectrogram distribution corresponding to
the given physics priors and video content. We encode all physics and residual
parameters as a latent feature embedding with multi-layer perceptron (MLPs).
The resulting physics latent vector is denoted by $\mu$. For video inputs,
given a sequence of RGB frames, we use temporal-shift-module (TSM) [30] to
efficiently extract visual features, which are then average pooled to compute
a single visual latent representation $\nu$.

As shown in Fig. 3, the diffusion process gradually corrupts the spectrogram
with Gaussian noise and transforms it into latent variables. Considering a
sequence of the same dimensional variables $x_{1},...,x_{T}$ where
$t=0,1,...,T$ is the index for diffusion time steps, the diffusion process
transforms spectrogram $x_{0}$ into a Gaussian noise $x_{T}$ through a chain
of Markov transitions where a variance schedule
$\beta_{1},\beta_{2},...,\beta_{T}$ will use at each transition step. Assuming
the Markov transition probability $q(x_{t}|x_{t-1},\mu,\nu)$ is independent of
the physics latent $\mu$, visual latent $\nu$, we perform each transformation
as follows:

$\displaystyle
q(x_{t}|x_{t-1},\mu,\nu)=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I).$
(7)

Since the diffusion process $q(x_{1:T}|x_{0},\mu,\nu)$ is a Markov process, we
can factorize it as follows:

$\displaystyle
q(x_{1},...,x_{T}|x_{0},\mu,\nu)=\prod_{t=1}^{T}q(x_{t}|x_{t-1}).$ (8)

To generate a spectrogram, we need the reverse process that aims to recover a
spectrogram from Gaussian noise. The reverse process can be defined as the
conditional distribution $p_{\theta}(x_{0:T-1}|x_{T},\mu,\nu)$, and according
to Markov chain property, it can be factorized into multiple transitions as
follows:

$\displaystyle
p_{\theta}(x_{0},...,x_{T-1}|x_{T},\mu,\nu)=\prod_{t=1}^{T}p_{\theta}(x_{t-1}|x_{t},\mu,\nu).$
(9)

Given the diffusion time-step with physics latent and visual latent
conditions, a spectrogram is recovered from the latent variables by applying
the reverse transitions $p_{\theta}(x_{t-1}|x_{t},\mu,\nu)$. Considering the
spectrogram distribution as $q(x_{0}|\mu,\nu)$, we aim to learn a model
distribution $p_{\theta}(x_{0}|\mu,\nu)$ obtained from the reverse process to
approximate $q(x_{0}|\mu,\nu)$. In other words, the reverse process aims to
maximize the log-likelihood of the spectrogram. Since
$p_{\theta}(x_{0}|\mu,\nu)$ is computationally intractable, we follow the
parameterization trick in [21] to calculate the variational lower bound of the
log-likelihood in a closed form. Specifically, consider
$\alpha_{t}=1-\beta_{t}$ and
$\overline{\alpha}_{t}=\prod_{t^{\prime}=1}^{t}\alpha_{t^{\prime}}$, the
training objective of the diffusion model can be described as follows:

$\displaystyle\min_{\theta}L(\theta)=\mathbb{E}_{x_{0},\epsilon,t}||\epsilon-\epsilon_{\theta}(\sqrt{\overline{\alpha}_{t}}x_{0}+\sqrt{1-\overline{\alpha}_{t}}\epsilon,t,\mu,\nu)||_{1},$
(10)

where $t$ is uniformly taken from the entire diffusion time-step. Our model
only uses L1 loss function between the output of model
$\epsilon_{\theta}(\cdot)$ and Gaussian noise $\epsilon\sim\mathcal{N}(0,I)$.

###  3.3 Training and Inference

During training, we use physics priors extracted from the audio waveform as an
additional condition to guide the model to learning correspondence between
video inputs and impact sounds. However, since the ground truth sound clip is
unavailable during inference, we could not obtain the corresponding physics
priors for the video input as we did in the training stage. Therefore, we
propose a new inference pipeline to allow us to preserve the benefit of
physics priors. To achieve this goal, we construct key-value pairs for visual
and physics latents in our training sets. At the inference stage, we feed the
testing video input and acquire the visual latent vector $\nu^{\text{test}}$.
We then take $\nu^{\text{test}}$ as a query feature and find the key in
training data by computing the euclidean distance between the test video
latent $\nu^{\text{test}}$ and all training video latents
$\\{\nu^{\text{train}}_{j}\\}^{J}_{j=1}$. Given the key
$\nu^{\text{train}}_{j}$, we then use the value $\mu^{\text{train}}_{j}$ as
our test physics latent $\hat{\mu}^{\text{test}}$. Once we have both visual
latent $\nu^{\text{test}}$ and physics latent $\hat{\mu}^{\text{test}}$, the
model generates a spectrogram from a latent variable by iteratively predicting
the diffusing noise added at each forward transition with
$\epsilon_{\theta}(x_{t},t,\hat{\mu}^{\text{test}},\nu^{\text{test}})$ and
removing the corrupted part as follows:

$\displaystyle
x_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}(x_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\overline{\alpha}_{t}}}\epsilon_{\theta}(x_{t},t,\hat{\mu}^{\text{test}},\nu^{\text{test}}))+\sigma_{t}z_{t},$
(11)

where $z_{t}\sim\mathcal{N}(0,I)$,
$\sigma_{t}=\eta\sqrt{\frac{1-\overline{\alpha}_{t-1}}{1-\overline{\alpha}_{t}}\beta_{t}}$,
and $\eta$ is a temperature scaling factor of the variance. After iterative
sampling over all of the time steps, we can obtain the final spectrogram
distribution $p_{\theta}(x_{0}|\hat{\mu}^{\text{test}},\nu^{\text{test}})$. It
is worth noting that while we use the physics latent from the training set, we
can still generate novel sound since the diffusion model also takes additional
visual features as input.

##  4 Experiments

###  4.1 Dataset

To evaluate our method, we use the Greatest Hits dataset [40] in which people
interact with physical objects by hitting and scratching materials with a
drumstick, comprising 46,577 total actions in 977 videos. Human annotators
labeled the actions with material labels and the temporal location of impact.
According to the dataset annotation assumption that the time between two
consecutive object sounds is at least $0.25$ second, we segment all audios
into $0.25$ second clips based on the data annotation for training and
testing. We use the pre-defined train/test split for all experiments.

###  4.2 Implementation Details

We use Pytorch to implement all models in our method. For physics parameter
estimation, all audio waveforms are in 44.1Khz and we compute log-scaled
spectrograms with 2048 window size and 256 hop size, leading to a $1025\times
44$ spectrogram for each impact sound. Then we estimate $1025$ modes
parameters from the spectrogram as described in the Sec 3.1. For residual
parameter prediction, the transformer encoder is a 4-layer transformer encoder
with 4 attention heads. The residual weights and decay rate dimensions are
both $100$. In the physics-driven diffusion model, we feed $22$ video frames
centered at the impact event to the video encoder which is a ResNet-50 model
with TSM [30] to efficiently handle the temporal information. The physics
encoder consists of five parallel MLPs which take each of physics priors as
input and project into lower-dimension feature vectors. The outputs are
concatenated together into a $256$-dim physics latent vector $\mu$. The
spectrogram denoiser is an Unet architecture, which is constructed as a
spatial downsampling pass followed by a spatial upsampling pass with skip
connections to the downsampling pass activation. We use Griffin-Lim algorithm
to convert the spectrogram to the final audio waveform [18]. We use AdamW
optimizer to train all models on a A6000 GPU with a batch size of 16 until
convergence. The initial learning rate is set to $5e-4$, and it gradually
decreases by a factor of 0.95.

###  4.3 Baselines

We compare our physics-driven diffusion model against various state-of-the-art
systems. For fair comparison, we use the same video features extracted by TSM
[30].

  * •

ConvNet-based Model: With a sequence video features, we first up-sampled them
to have the same number of frames as the spectrogram. Then we perform a Unet
architecture to convert video features to spectrogram. Such a architecture has
shown successful results in spectrogram-based music generation [53].

  * •

Transformer-based Model: We implement a conditional Transformer Network which
has shown promising results in Text-to-Speech [28]. Instead of using text as
condition, here we use the extracted video features.

  * •

Video conditioned Diffusion model: We also compare our approach to two video
conditioned spectrogram diffusion model variants. In the first setup, we do
not include the physics priors and keep all other settings the same.

  * •

Video + Class Label conditioned Diffusion model: In the second variant, we
provide a class-label of the impact sound material as an additional condition
to the video features. All other settings are the same as ours.

  * •

Video + Other Audio Features Diffusion model: To show the importance of
physics latents, we replace the physics latent with spectrogram/mfcc latent by
extracting spectrogram/mfcc features from the raw audio and pass them to a
transformer encoder similar to the one used in physics parameters
reconstruction, and then we apply average pooling to obtain the latent vector.
During testing, we still use visual features to query the corresponding
spectrogram/mfcc latent in the training set and synthesize the final results.

###  4.4 Evaluation Metrics

We use four different metrics to automatically assess both the fidelity and
relevance of the generated samples. For automatic evaluation purpose, we train
an impact sound object material classifier using the labels in Great Hits
Dataset. The classifier is a ResNet-50 convolutional-based neural network and
we use the spectrogram as input to train the classifier.

  * •

Fréchet Inception Distance (FID) is used for evaluating the quality of
generated impact sound spectrograms. The FID score evaluates the distance
between the distribution of synthesized spectrograms and the spectrograms in
the test set. To build the distribution, we extract the features before the
impact sound classification layer.

  * •

Kernel Inception Distance (KID) is calculated via maximum mean discrepancy
(MMD). Again, we extract features from synthesized and real impact sounds. The
MMD is calculated over a number of subsets to both get the mean and standard
deviation of KID.

  * •

KL Divergence is used to individually compute the distance between output
distributions of synthesized and ground truth features since FID and KID
mainly rely on the distribution of a collection of samples.

  * •

Recognition accuracy is used to evaluate if the quality of generated impact
sound samples can fool the classifier.

###  4.5 Results

Model\Metric |  FID $\downarrow$ |  KID (mean, std)$\downarrow$ |  KL Div. $\downarrow$ |  Recog. Acc (%) $\uparrow$
---|---|---|---|---
ConvNet-based | 43.50 | 0.053, 0.013 | 4.65 | 51.69
Transformer-based | 34.35 | 0.036, 0.015 | 3.13 | 62.86
Video Diffusion | 54.57 | 0.054, 0.014 | 2.77 | 69.94
Video + Class label Diffusion | 31.82 | 0.026, 0.021 | 2.38 | 72.02
Video + MFCC Diffusion | 40.21 | 0.037, 0.010 | 2.84 | 67.87
Video + Spec Diffusion | 28.77 | 0.016, 0.009 | 2.55 | 70.46
Video + Physics Diffusion (Ours) | 26.20 | 0.010, 0.008 | 2.04 | 74.09
Table 1: Quantitative evaluations for different models. For FID, KID, and KL
Divergence, lower is better. For recognition accuracy, higher is better. Bold
font indicates the best value.

Quantitative evaluation results are shown in Table 1. Our proposed physics-
driven diffusion model outperforms all other methods across all metrics. It is
worth noting that without physics priors, using video features alone as
condition to the spectrogram denoiser is not sufficient to generate high
fidelity sounds. While this could be improved when class labels are available,
it is obvious that there is a large gap to reach the performance of the
physics-driven method. Fig. 4 illustrates a comparison of three examples of
generated spectrograms given a video by ConvNet-based, Transformer-based, and
our physics-driven approaches to the ground truth. While the ConvNet and
Transformer-based approaches could also capture some correspondences between
audio and video, it is obvious that a lot of noise appears in the generated
spectrogram because these approaches are prone to learn an average smoothed
representation, and thus introducing many artifacts in the end. In comparison,
our physics-driven diffusion approach does not suffer from this problem and
can synthesize high fidelity sound from videos. It is worth noting that the
interoperability of our approach could potentially unlock applications such as
controllable sound synthesis by manipulating the physics priors.

Figure 4: Qualitative Comparison results on sound spectrogram generated by
different methods.

###  4.6 Human Perceptual Evaluation

In addition to the objective evaluation of our method, we also perform human
perceptual surveys using Amazon Mechanical Turk (AMT). These surveys were
intended to evaluate the effectiveness of our generated samples to match the
video content and the fidelity of the generated samples. For all surveys, no
background on the survey or our approach was given to the participants to
avoid perceptual biases. We surveyed $50$ participants individually, where
each participant was asked to evaluate $10$ videos along with different
generated samples from various methods. A total of $500$ opinions were
collected in the end.

  * •

Matching. In the first survey, we asked people to watch the same video with
different synthesized sounds and answer the question: ”In which video the
sound best matches the video content?”. The given options of the soundtracks
were also the ConvNet-based, Transformer-based and Physics-driven diffusion
approaches. From results shown in Table. 2 (Top) (left column), we observe a
clear indication that the sound generated with our method are chosen to be the
best match to the visual content more frequently.

  * •

Quality. In the second survey, we asked people (non-expert) to choose the
video with the highest sound quality, including 3 variations of samples
generated by ConvNet-based, Transformer-based and Physics-driven diffusion
approaches. Results in Table clearly indicate our approach achieves the best
sound quality.

  * •

Perceptual Ablation Studies. In an additional survey, we performed a
perceptual ablation study to test how the physics priors influence the
perception of the generated impact sounds compared to the approaches without
it. Survey results shown in Table 2 (Bottom) and suggest that in comparison to
video only model, the physics priors improve the overall perception of
synthesized impact sounds.

Model\Metric | Matching | Quality
---|---|---
Comparison to Baselines
ConvNet-based | 18% | 17.6%
Transformer-based | 26.6% | 28.8%
Ours | 55.4% | 53.6%
Perceptual Ablation Studies
Video-only | 23.6% | 23.6%
Video+label | 37.8% | 35.8%
Ours | 38.6% | 40.6%
Table 2: (Top) Human perceptual evaluation on matching and quality metrics.
(Bottom) Ablation study on human perceptual evaluation. The value indicates
the percentage of Amazon Turkers who select the method.

###  4.7 Ablation Studies

We performed three ablation studies to answer the following questions. Q1: How
do residual parameters influence the physics priors? Q2: What is the
contribution of each component to our approach? Q3: Is our method better than
simple retrieval methods?

Q1. Since the physics and residual parameters are essential in our approach.
We have investigated different physics priors variants to find the most
informative physics priors and use the multi-resolution STFT loss of the
reconstructed sounds for evaluation. The results in Fig. 5(a) clearly show
that the loss decreases significantly with residual parameters. We also find
that using $100$ residual parameters achieves the best performance, while
fewer or more residual parameters may damage the performance.

Q2. We perform extensive experiments to understand the contribution of each
component. For all studies, we use the nearest physics parameters/priors
retrieved by visual latent to synthesize the sound. Results are shown in Fig.
5(b). We first observe that without residual components and diffusion models,
using estimated physics parameters to perform modal synthesis could not obtain
faithful impact sounds. The physics priors could re-synthesize impact sounds
with much better quality with learned residual parameters. We further show
that using physics priors as the condition input to the diffusion model
achieves even better performance. We have also performed an experiment to
predict physics latent from video input and use it as the condition for the
diffusion model but the quality of generated samples is poor. This is due to
the weak correspondence between video inputs and physics behind the impact
sounds and indicates the importance of using video inputs to query physics
priors of the training set at the inference stage.

Q3. We consider two retrieval baselines for comparison. The first one is a
simple baseline without physics priors and diffusion model. We only use visual
features extracted from the ResNet-50 backbone to search the nearest neighbor
(NN) in the training set and use the audio as output. In the second
experiment, we try our best to reproduce the model in [40] since no official
implementation is available. The model predicts sound features (cochleagrams)
from images via LSTM. For fair evaluation, a collection-based metric like FID
is invalid because the retrieved audios are in the real data distribution.
Therefore, we use sample-based metrics, including KL Divergence between
predicted and ground truth audio features and Mean Square Error on the
spectrogram level. The table 3 clearly shows that our approach outperforms the
retrieval baselines by a large margin.

Model\Metric |  KL Div. $\downarrow$ |  Spec. MSE$\downarrow$
---|---|---
NN via Visual Features | 10.60 | 0.307
NN via Predicted Sound Features [40] | 7.39 | 0.205
Ours | 2.04 | 0.149
Table 3: Comparison with retrieval methods. Figure 5: (a) Ablation study on
the importance and selection for the number of residual parameters by testing
multi-resolution STFT loss. (b) Ablation study on the contribution of each
component of our approach using FID score, the lower the better.

##  5 Conclusion

We present a physics-driven diffusion model for impact sound synthesis from
videos. Our model can effectively generate high fidelity sounds for physical
object interactions. We achieve such function by leveraging physics priors as
guidance for the diffusion model to generate impact sounds from video input.
Experimental results demonstrate that our approach outperforms other methods
quantitatively and qualitatively. Ablation studies have demonstrated that
physics priors are critical for generating high-fidelity sounds from video
inputs. The limitation of our approach naturally becomes that our approach
cannot generate impact sounds for unseen physics parameters due to the query
process (failure case demonstration is shown in Supplementary material), while
we can generate novel sounds given an unseen video.

#### Acknowledgements.

This work was supported by the MIT-IBM Watson AI Lab, DARPA MCS, DSO grant
DSOCO21072, and gift funding from MERL, Cisco, Sony, and Amazon.

## References

  * [1] Samuel Albanie, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman.  Emotion recognition in speech using cross-modal transfer in the wild.  In Proceedings of the 26th ACM international conference on Multimedia, pages 292–301, 2018.
  * [2] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel.  Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674–3683, 2018.
  * [3] Relja Arandjelovic and Andrew Zisserman.  Look, listen and learn.  In Proceedings of the IEEE International Conference on Computer Vision, pages 609–617, 2017.
  * [4] Yusuf Aytar, Carl Vondrick, and Antonio Torralba.  Soundnet: Learning sound representations from unlabeled video.  Advances in neural information processing systems, 29, 2016.
  * [5] Kan Chen, Chuanxi Zhang, Chen Fang, Zhaowen Wang, Trung Bui, and Ram Nevatia.  Visually indicated sound generation by perceptually optimized classification.  In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0–0, 2018.
  * [6] Lele Chen, Sudhanshu Srivastava, Zhiyao Duan, and Chenliang Xu.  Deep cross-modal audio-visual generation.  In Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pages 349–357, 2017.
  * [7] Nuo Chen, Chenyu You, and Yuexian Zou.  Self-supervised dialogue learning for spoken conversational question answering.  In INTERSPEECH, 2021.
  * [8] Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan.  Generating visually aligned sound from videos.  IEEE Transactions on Image Processing, 29:8292–8302, 2020.
  * [9] Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee Kenneth Wong, Joshua B Tenenbaum, and Chuang Gan.  Grounding physical concepts of objects and events through dynamic visual reasoning.  arXiv preprint arXiv:2103.16564, 2021.
  * [10] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  * [11] Chuang Gan, Deng Huang, Peihao Chen, Joshua B Tenenbaum, and Antonio Torralba.  Foley music: Learning to generate music from videos.  In European Conference on Computer Vision, pages 758–775. Springer, 2020.
  * [12] Chuang Gan, Deng Huang, Hang Zhao, Joshua B Tenenbaum, and Antonio Torralba.  Music gesture for visual sound separation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10478–10487, 2020.
  * [13] Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al.  Threedworld: A platform for interactive multi-modal physical simulation.  arXiv preprint arXiv:2007.04954, 2020.
  * [14] Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum.  Look, listen, and act: Towards audio-visual embodied navigation.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9701–9707. IEEE, 2020.
  * [15] Chuang Gan, Hang Zhao, Peihao Chen, David Cox, and Antonio Torralba.  Self-supervised moving vehicle tracking with stereo sound.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7053–7062, 2019.
  * [16] Ruohan Gao, Rogerio Feris, and Kristen Grauman.  Learning to separate object sounds by watching unlabeled video.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 35–53, 2018.
  * [17] Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani.  Listen to look: Action recognition by previewing audio.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10457–10467, 2020.
  * [18] Daniel Griffin and Jae Lim.  Signal estimation from modified short-time fourier transform.  IEEE Transactions on acoustics, speech, and signal processing, 32(2):236–243, 1984.
  * [19] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10696–10706, 2022.
  * [20] John Hershey and Javier Movellan.  Audio vision: Using audio-visual synchrony to locate sounds.  Advances in neural information processing systems, 12, 1999.
  * [21] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [22] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  J. Mach. Learn. Res., 23:47–1, 2022.
  * [23] Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren.  Prodiff: Progressive fast diffusion model for high-quality text-to-speech.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 2595–2605, 2022.
  * [24] Hamid Izadinia, Imran Saleemi, and Mubarak Shah.  Multimodal analysis for identification and segmentation of moving-sounding objects.  IEEE Transactions on Multimedia, 15(2):378–390, 2012.
  * [25] Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim.  Diff-tts: A denoising diffusion model for text-to-speech.  arXiv preprint arXiv:2104.01409, 2021.
  * [26] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  arXiv preprint arXiv:2009.09761, 2020.
  * [27] Junhyeok Lee and Seungu Han.  Nu-wave: A diffusion probabilistic model for neural audio upsampling.  arXiv preprint arXiv:2104.02321, 2021.
  * [28] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu.  Neural speech synthesis with transformer network.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6706–6713, 2019.
  * [29] Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Salimi-Khorshidi.  Behrt: transformer for electronic health records.  Scientific reports, 10(1):1–12, 2020.
  * [30] Ji Lin, Chuang Gan, and Song Han.  Tsm: Temporal shift module for efficient video understanding.  In Proceedings of the IEEE International Conference on Computer Vision, 2019.
  * [31] Fenglin Liu, Xian Wu, Chenyu You, Shen Ge, Yuexian Zou, and Xu Sun.  Aligning source visual and target language domains for unpaired video captioning.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [32] Fenglin Liu, Chenyu You, Xian Wu, Shen Ge, Xu Sun, et al.  Auto-encoding knowledge graph for unsupervised medical report generation.  Advances in Neural Information Processing Systems, 34, 2021.
  * [33] Xiang Long, Chuang Gan, Gerard De Melo, Jiajun Wu, Xiao Liu, and Shilei Wen.  Attention clusters: Purely attention based local feature integration for video classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7834–7843, 2018.
  * [34] Xiang Long, Chuang Gan, Gerard Melo, Xiao Liu, Yandong Li, Fu Li, and Shilei Wen.  Multimodal keyless attention fusion for video classification.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [35] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon.  Symbolic music generation with diffusion models.  arXiv preprint arXiv:2103.16091, 2021.
  * [36] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman.  Seeing voices and hearing faces: Cross-modal biometric matching.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8427–8436, 2018.
  * [37] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In International Conference on Machine Learning, pages 8162–8171. PMLR, 2021.
  * [38] James F O’Brien, Perry R Cook, and Georg Essl.  Synthesizing sounds from physically based motion.  In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 529–536, 2001.
  * [39] James F O’Brien, Chen Shen, and Christine M Gatchalian.  Synthesizing sounds from rigid-body simulations.  In Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation, pages 175–181, 2002.
  * [40] Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William T Freeman.  Visually indicated sounds.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2405–2413, 2016.
  * [41] Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba.  Ambient sound provides supervision for visual learning.  In European conference on computer vision, pages 801–816. Springer, 2016.
  * [42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [43] Zhimin Ren, Hengchin Yeh, and Ming C Lin.  Example-guided physically based modal sound synthesis.  ACM Transactions on Graphics (TOG), 32(1):1–16, 2013.
  * [44] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [45] Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh McDermott, and Antonio Torralba.  Self-supervised audio-visual co-segmentation.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2357–2361. IEEE, 2019.
  * [46] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [47] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon.  Learning to localize sound source in visual scenes.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358–4366, 2018.
  * [48] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  * [49] Kun Su, Xiulong Liu, and Eli Shlizerman.  Audeo: Audio generation for a silent performance video.  Advances in Neural Information Processing Systems, 33:3325–3337, 2020.
  * [50] Kun Su, Xiulong Liu, and Eli Shlizerman.  How does it sound?  Advances in Neural Information Processing Systems, 34:29258–29273, 2021.
  * [51] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu.  Audio-visual event localization in unconstrained videos.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 247–263, 2018.
  * [52] Kees Van Den Doel, Paul G Kry, and Dinesh K Pai.  Foleyautomatic: physically-based sound effects for interactive simulation and animation.  In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 537–544, 2001.
  * [53] Bryan Wang and Yi-Hsuan Yang.  Performancenet: Score-to-audio music generation with multi-band convolutional residual network.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1174–1181, 2019.
  * [54] Xudong Xu, Bo Dai, and Dahua Lin.  Recursive visual sound separation using minus-plus net.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 882–891, 2019.
  * [55] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim.  Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6199–6203. IEEE, 2020.
  * [56] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum.  Neural-symbolic vqa: Disentangling reasoning from vision and language understanding.  Advances in neural information processing systems, 31, 2018.
  * [57] Chenyu You, Nuo Chen, and Yuexian Zou.  MRD-Net: Multi-Modal Residual Knowledge Distillation for Spoken Question Answering.  In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI, pages 3985–3991, 2021.
  * [58] Chenyu You, Weicheng Dai, Fenglin Liu, Haoran Su, Xiaoran Zhang, Lawrence Staib, and James S Duncan.  Mine your own anatomy: Revisiting medical image segmentation with extremely limited labels.  arXiv preprint arXiv:2209.13476, 2022.
  * [59] Chenyu You, Weicheng Dai, Yifei Min, Fenglin Liu, Xiaoran Zhang, Chen Feng, David A Clifton, S Kevin Zhou, Lawrence Hamilton Staib, and James S Duncan.  Rethinking semi-supervised medical image segmentation: A variance-reduction perspective.  arXiv preprint arXiv:2302.01735, 2023.
  * [60] Chenyu You, Ruihan Zhao, Fenglin Liu, Siyuan Dong, Sandeep P Chinchali, Lawrence Hamilton Staib, James s Duncan, et al.  Class-aware adversarial transformers for medical image segmentation.  In Advances in Neural Information Processing Systems.
  * [61] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  arXiv preprint arXiv:2205.01917, 2022.
  * [62] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  arXiv preprint arXiv:1605.07146, 2016.
  * [63] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba.  The sound of pixels.  In Proceedings of the European conference on computer vision (ECCV), pages 570–586, 2018.
  * [64] Hang Zhou, Ziwei Liu, Xudong Xu, Ping Luo, and Xiaogang Wang.  Vision-infused deep audio inpainting.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 283–292, 2019.
  * [65] Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L Berg.  Visual to sound: Generating natural sound for videos in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3550–3558, 2018.
  * [66] Hao Zhu, Man-Di Luo, Rui Wang, Ai-Hua Zheng, and Ran He.  Deep audio-visual learning: A survey.  International Journal of Automation and Computing, 18(3):351–376, 2021.

##  Appendix A Generated Samples

Please see the attached video for a short overview and generated impact sound
samples using our physics-driven diffusion models. Please turn Audio ON and
use headphones for best perception of the audio.

##  Appendix B Fixed Physics Priors

As we mentioned in the main paper, while we use the physics latent from
training set, we can still generate novel sound since the diffusion model also
takes additional visual features as input. To show that, we fix one type
physics priors and use various video inputs to synthesize impact sounds. We
found that the synthesized sounds are strongly impacted by the physics priors
but different visual inputs can still introduce the diversity of the sounds.
Please check out the videos with prefix
fix_physics_priors_type_sample_num.mp4.

##  Appendix C Physics Priors Editing

Since the physics priors are transparent, we can manipulate them to control
the synthesized sounds. In the first experiment, we decrease values of power
and decay rate in the lowest $200$ modes to reduce low frequency components
and we also set the residual parameters be zero for better visualization. The
generated spectrogram results are shown in Fig. 6. In addition, we also try
highlighting the low frequency components by tuning up the power and decay
values of the lowest $200$ modes. The generated spectrogram results are shown
in Fig. 7. Both results demonstrate that the transparency of the physics
priors.

Figure 6: Spectrogram examples of removing low frequency parts by manipulating
physics priors. Figure 7: Spectrogram examples of adding low frequency parts
by manipulating physics priors.

##  Appendix D Additional Implementation Details

###  D.1 Physics Priors Representation & Encoder

We provide more details of how physics parameters and residual parameters are
scaled and fitted into the diffusion model. For each estimated frequency
feature $f$, we compute the distance to the corresponding central frequency of
the spectrogram bin and then normalize the distance by the bin resolution
(21Hz in our case) to obtain a value between $-1$ and $1$. The power feature
$p$ is extracted in dB scale. We normalize it by $2p/(-80)-1$ where $-80$dB
indicates the silence. The decay feature $\lambda$ is also applied min-max
normalization to map the value to the range of $-1$ and $1$. For residual
parameters, we use a transformer spectrogram encoder to learn $w$ and $\gamma$
by applying a sigmoid function to outputs of two linear projection layers. We
then use a physics priors encoder to take produce the physics latent. The
details of the physics priors encoder are shown in Fig. 8.

###  D.2 Diffusion Model Details

The architecture is a U-Net [44] based on a Wide ResNet [62]. Our models use
four feature map resolutions and each resolution level contains two
convolutional residual blocks and a spatial attention module. The channel
dimensions increase from $1$ to $64$, $128$, $256$, and $512$. We use
sinusoidal position embedding for diffusion time $t$ which is added into each
residual block. We use cosine schedule for diffusion noise and set the
sampling timesteps be $T=1000$.

Figure 8: Physics Priors Encoder Details

##  Appendix E Failure case demonstration

We show an example of having an impact on rock. In the following figure, the
(a, b) are common rock sounds in the training set. However, we find a rare
case in the testing set shown in (c). While our approach can generate new
realistic rock sound (d), it is different from the ground truth.