# DiffSED: Sound Event Detection with Denoising Diffusion

Swapnil Bhosale1*, Sauradip Nag1*, Diptesh Kanojia1, Jiankang Deng2, Xiatian
Zhu1

###### Abstract

Sound Event Detection (SED) aims to predict the temporal boundaries of all the
events of interest and their class labels, given an unconstrained audio
sample. Taking either the split-and-classify (i.e., frame-level) strategy or
the more principled event-level modeling approach, all existing methods
consider the SED problem from the discriminative learning perspective. In this
work, we reformulate the SED problem by taking a generative learning
perspective. Specifically, we aim to generate sound temporal boundaries from
noisy proposals in a denoising diffusion process, conditioned on a target
audio sample. During training, our model learns to reverse the noising process
by converting noisy latent queries to the ground-truth versions in the elegant
Transformer decoder framework. Doing so enables the model generate accurate
event boundaries from even noisy queries during inference. Extensive
experiments on the Urban-SED and EPIC-Sounds datasets demonstrate that our
model significantly outperforms existing alternatives, with 40+% faster
convergence in training.

## Introduction

Sound event detection (SED) aims to temporally localize sound events of
interest (i.e., the start and end time) and recognize their class labels in a
long audio stream (Mesaros et al. 2021). As a fundamental audio signal
processing task, it has become the cornerstone of many related recognition
scenarios, such as audio captioning (Xu et al. 2021; Bhosale, Chakraborty, and
Kopparapu 2023; Xie et al. 2023), and acoustic scene understanding (Igarashi
et al. 2022; Bear, Nolasco, and Benetos 2019).

Figure 1: Architectural comparison: (a) Conventional discriminative DETR-based
Sound Event Detector Transformer (SEDT) (Ye et al. 2021) incorporates a single
decoding step with clean queries. (b) Our diffusion-infused generative DETR-
based Sound Event Detector (DiffSED) conducts multi-step decoding/denoising
over noised queries.

In the literature, all existing SED methods can be grouped into two categories
namely, frame-level and event-level approaches. Frame-level approaches
classify each audio frame/segment into event classes and then aggregate the
consecutive frame-level predictions to identify sound event boundaries or
endpoints (Miyazaki et al. 2020a; Lin et al. 2019). They are often heavily
manually designed with plenty of heuristics and data-specific parameter
optimization, hence less scalable and reliable across different audio data.
Event-level approaches, on the other hand, directly model the temporal
boundaries of sound events, taking into account the correlation between
frames, thereby eliminating the mundane post-processing step and are more
generalizable (Ye et al. 2021). In both approaches, existing methods rely on
proposal prediction by regressing the start and end times of each, i.e.,
discriminative learning based.

Recently, generative learning models such as diffusion models (Ho, Jain, and
Abbeel 2020; Song, Meng, and Ermon 2020) have emerged strongly in computer
vision. Conceptually, we draw an analogy between the SED problem and image-
based object detection (Duan et al. 2019; Chen et al. 2019). We consider the
latest generative learning based object detection approach (Chen et al. 2022b)
represents a new direction for designing detection models in general. Although
conceptually similar to object detection, the SED problem still presents
unique challenges and complexity due to the presence of temporal dynamics.
Besides, there are several limitations with the detection diffusion
formulation in (Chen et al. 2022b). First, a two-stage pipeline (e.g., RCNN
(Chao et al. 2018)) is adopted, giving rise to localization-error propagation
from proposal generation to proposal classification (Nag et al. 2022a, b, c).
Second, as each event proposal is processed individually, their intrinsic
relationship modeling is overlooked, potentially hurting the learning
efficacy. To address these issues, we present two different designs: (a)
Adopting the one-stage detection pipeline (Tian et al. 2019; Wang et al. 2020)
that have already shown excellent performance with a relatively simpler
design, in particular, DETR (Carion et al. 2020). Even within the SED
literature, this simpler pipeline has shown to achieve higher accuracy than
frame-level models on a variety of sound event detection datasets due to the
better temporal resolution, as well as its ability to learn long-range
dependencies between sound events (Ye et al. 2021). (b) A unique challenge
with SED is big boundary ambiguity as compared to object detection. This is
because temporal audio events are continuous in time without clear start and
end points (e.g., non-zero momentum), and the transition between consecutive
events is often stochastic. Further, human perception of event boundaries is
also instinctive and subjective. For the above reasons, we reckon that
diffusion-based models could be a great fit for sound event detection.

Nonetheless, it is non-trivial to integrate denoising diffusion with existing
sound event detection models, due to several reasons. (1) Whilst efficient at
processing high-dimension data simultaneously, diffusion models (Dhariwal and
Nichol 2021; Li et al. 2022) have typically been shown to work with continuous
input data. But event boundaries in SED are discrete. (2) Denoising diffusion
and SED both suffer low efficiency, and their combination would even get
worse. Both of the problems have not been investigated systematically thus
far.

To address the aforementioned challenges, a novel conditioned event diffusion
method is proposed for efficiently tackling the SED task, abbreviated as
DiffSED. In the forward diffusion process, Gaussian noises are added to the
event latents iteratively. In the reverse denoising process, the noisy latents
are passed as queries to a denoiser (e.g., DETR (Carion et al. 2020)) for
denoising the event latents so that desired event proposals can be obtained,
with the condition on the observation of an input audio stream. The usage of
noisy latents allows our model to bypass the need for continuous input, as the
denoising diffusion process takes place in the designated latent space. During
inference, the model can take as input the noisy latents composed of noises
sampled from Gaussian distribution and learned components, and outputs the
event proposals of a given audio stream (i.e., the condition). The proposed
noise-to-queries strategy for denoising diffusion has several appealing
properties: (i) Evolutionary enhancement of queries during inference wherein
each denoising step can be interpreted as a unique distribution of noise thus
adding stochasticity to solve the boundary ambiguity problem. (ii) Integrating
denoising diffusion with this noisy-latent decoder design solves the typical
slow-convergence limitation.

We summarize the contributions of this work. (a) We reformulate sound event
detection (SED) as a generative denoising process in an elegant transformer
decoder framework. This is the first study to apply the diffusion model for
the SED task to the best of our knowledge. (b) The proposed generative
adaptation uses a noise-to-queries strategy with several appealing properties
such as evolutionary enhancement of queries and faster convergence. (c) Our
comprehensive experiments on the URBAN-SED (Salamon, Jacoby, and Bello 2014)
and the EPIC-Sounds (Huh et al. 2023) datasets validate the significant
performance advantage of our DiffSED over existing alternatives.

Figure 2: Overview of our proposed DiffSED. (Top) In the forward diffusion
process, Gaussian noises are added to the event latents iteratively to obtain
noisy latents $X_{T}$. (Bottom) In the reverse denoising process, an audio
melspectogram is passed as the condition along with random noisy latents
sampled from the Gaussian distribution. The noisy latents are passed as the
query to the denoiser for denoising the event latents in an iterative fashion
to obtain event proposals.

## Related work

### Sound event detection

The existing SED literature can be divided into two categories, namely, frame-
level approaches and event-level approaches. In frame-level approaches (Lim,
Park, and Han 2017; Turpault et al. 2019; Miyazaki et al. 2020a), the input
audio signal is first divided into short, fixed-length segments, and the sound
events within each segment are further classified independently. Despite
strong performance and good intuition, this split-and-classify strategy
requires plenty of heuristics designs, unscalable parameter settings (e.g.,
segment duration), as well as time-consuming post-processing (e.g.,
aggregating frame-level predictions). To overcome these limitations, event-
level approaches (Ye et al. 2021) present a more principled and scalable
solution with end-to-end learning frameworks, inspired by the model designs in
object detection (Carion et al. 2020; Zhu et al. 2020; Zhang et al. 2022) and
video action recognition domains (Tan et al. 2021; Shi et al. 2022). Whilst
being understudied, this strategy has shown to be more efficient and robust to
longer and more complex (overlapping) events, such as those in music and human
speech as well as short and frequently occurring events such as those in urban
soundscapes or environmental monitoring. Our DiffSED belongs to this category,
further pushing this forefront of performance.

Deep learning techniques have achieved excellent performance in SED. For
instance, convolutional neural networks (CNNs) have been widely investigated
for audio event classification (Cakır et al. 2017; Kumar, Khadkevich, and
Fügen 2018) owing to their ability to efficiently capture and analyze local
patterns within the acoustic waveform of sound. Additionally, recurrent neural
networks (RNNs) have been used for temporal modeling of audio signals in
arrears to their propensity to capture long-term temporal dependencies in
sequential data - an innate property of audio signals. Interestingly, apart
from the hybrid approaches (Li et al. 2020; Koh et al. 2021), that utilize
CNNs to extract features from the audio signal, which are then fed into an RNN
to model temporal dependencies, recently, transformer based architectures
(Wakayama and Saito 2022; Chen et al. 2022a) have been shown as equally
promising, particularly, leveraging the self-attention mechanisms to model
temporal relationships in audio signals and capturing complex patterns over
time. Commonly, all the prior methods consider the SED problem as
discriminative learning. In contrast, we treat for the first time this problem
in a unique perspective of generative learning. In particular, we generate the
sound event bounds and predict the class labels from noise latents, with the
condition to the input audio sample.

### Diffusion-based models for audio tasks

As a new class of deep generative models, diffusion models have been gaining
popularity in different fields. Beginning with a sample from a random
distribution, the diffusion model is optimized to gradually learn a denoising
schedule to obtain a noise-free target. This paradigm has yielded remarkable
results in audio processing tasks ranging from audio generation (Leng et al.
2022; Huang et al. 2022), audio enhancement (Lemercier et al. 2022), audio
separation (Lutati, Nachmani, and Wolf 2023) etc. To the best of our
knowledge, this is the first work that exploits a diffusion model for the SED
task.

## Methodology

Problem definition Sound event detection (SED) involves both classification
and temporal localization given an audio sequence. In this task, the audio
sequence is usually represented as a 2-dimensional feature, such as a mel-
spectrogram. We want a model to output the onset and offset times of all
target events and the corresponding event labels (Wakayama and Saito 2022). To
train the model, we collect a set of labeled audio sequence set
$D^{train}=\\{A_{i},\psi_{i}\\}$. Each audio $A_{i}\in\mathcal{R}^{T\times F}$
(where $T\times F$ represents the spectro-temporal dimension) is labeled with
temporal annotation $\psi_{i}=\\{(\Psi_{j},\xi_{j},y_{j})\\}_{j=1}^{M_{i}}$
where $\Psi_{j}/\xi_{j}$ represents onset/offset of an event and $y_{j}$
denotes the acoustic class event label.

### Preliminaries on diffusion model

Diffusion models are a class of generative models that use the diffusion
process to model complex probability distributions (Ho, Jain, and Abbeel 2020;
Song, Meng, and Ermon 2020). In a diffusion model, the forward process
generates samples by iteratively applying a diffusion equation to a starting
noise vector. The forward process can be represented by the following
equation:

$z_{t}=\sqrt{1-\beta_{t}}*z_{t-1}+\sqrt{\beta_{t}}*x_{t}$ (1)

where $z_{t}$ is the diffusion state at time $t$, $x_{t}$ is the input at time
$t$, and $\beta_{t}$ is the diffusion coefficient at time $t$. The noise scale
is controlled by $\beta_{t}$ which adopts a monotonically decreasing cosine
schedule (Ho, Jain, and Abbeel 2020; Song, Meng, and Ermon 2020) in every
different time step $t$.

Denoising in diffusion models is the process of generating a clean
representation from a noisy observation by reversing the diffusion process. In
other words, the goal is to obtain an estimate of the original representation
from the final diffusion state. The denoising process can be performed using
the reverse diffusion process, which can be represented by the following
equation:

$z_{T-t}=(z_{T-t+1}-\sqrt{\beta_{T-t}}*x_{T-t})/\sqrt{1-\beta_{T-t}}$ (2)

where $z_{T}$ is the final diffusion state, $x_{t}$ is the noisy input at time
$t$, and $\beta_{T-t}$ is the diffusion coefficient at time $T-t$. The
denoising process starts from the final diffusion state $z_{T}$ and
iteratively applies the reverse diffusion equation to obtain an estimate of
the original representation $x_{0}=z_{1}$.

$\displaystyle x_{t}=(z_{t+1}-\sqrt{\beta_{t}}*x_{t-1})/\sqrt{1-\beta_{t}}$
(3)

where $\forall t\in[1,T-1]$ such that $x_{t}$ is the estimate of the original
representation at time $t$. The denoising process can be improved by adding
regularization or constraints to the estimate of the original representation.

### DiffSED: Architecture design

Diffusion-based SED formulation In this work, we formulate the SED task in a
conditional denoising diffusion framework. In our setting, data samples are a
set of learnable event query embeddings $\textbf{z}_{0}=b$, where
$b\in\mathbb{R}^{N\times D}$ denotes $N$ event query embeddings at the
dimension of $D$. In our implementation, the event queries are retrieved from
a simple lookup table that stores embeddings of a fixed dictionary of size N
(initialized from $\mathcal{N}(0,1)$). A neural network
$f_{\theta}(\textbf{z}_{t},t,A)$ is trained to predict $\textbf{z}_{0}$ from
noisy proposals $\textbf{z}_{t}$, conditioned on the corresponding audio $A$.
The audio category $\hat{y}$ is predicted subsequently. See Algorithm 1 for
more details.

Algorithm 1 Training

⬇

1 def train_loss(audio, event_query):

2 """

3 audio: [B, T, F]

4 event_queries: [B, N, D]

5 # B: batch_size

6 # N: number of event queries

7 """

8 # Encode audio features

9 audio_feats = audio_encoder(audio)

10

11 # Signal scaling

12 event_queries = (event_queries * 2 - 1) * scale

13

14 # Corrupt event_queries

15 t = randint(0, T) # time step

16 eps = normal(mean=0, std=1) # noise: [B, N, D]

17 event_queries_crpt = sqrt( alpha_cumprod(t)) * event_queries +

18 sqrt(1 - alpha_cumprod(t)) * eps

19

20 # Predict bounding boxes

21 pb_pred = detection_decoder(event_queries_crpt, audio_feats, t)

22

23 # Set prediction loss

24 loss = set_prediction_loss(pb_pred, gt_boxes)

25

26 return loss

Since the diffusion model generates a data sample iteratively, it needs to run
the model $f_{\theta}$ multiple times in inference. It would be
computationally intractable to directly apply $f_{\theta}$ on the raw audio at
every iterative step. For efficiency, we propose to separate the whole model
into two parts, audio encoder and detection decoder, where the former runs
only once to extract a feature representation of the input audio $A_{i}$, and
the latter takes this feature as a condition to progressively refine the noisy
proposals $\textbf{z}_{t}$.

Audio encoder The audio encoder takes as input the pre-extracted audio mel-
spectograms and extracts high-level features for the following detection
decoder. In general, any audio encoder can be used. We follow (Ye et al. 2021)
for the audio encoder. More specifically, the raw audio is first encoded using
a CNN based encoder backbone (i.e., ResNet-50) to obtain the audio feature
$A_{f}\in\mathbb{R}^{T^{{}^{\prime}}\times F^{{}^{\prime}}}$ respectively.
This is followed by a multi-layered temporal transformer (Vaswani et al. 2017)
$\tau$ that performs global attention across the time dimension to obtain the
global feature as:

$C_{a}=\tau(A_{f})$ (4)

where query, key, and value of the transformer is set to $A_{f}$. We also
append positional encoding to $A_{f}$ before passing it into the transformer.

#### Detection decoder

Similar to SEDT (Ye et al. 2021), we use a transformer decoder (Vaswani et al.
2017) (denoted by $f_{\theta}$) for detection. Functionally, in our
formulation it serves as a denoiser. In traditional DETR (Lin et al. 2021),
the queries are learnable continuous embeddings with random initialization. In
DiffSED, however, we exploit the queries as the denoising targets.

As opposed to adding noises to object boundaries (Chen et al. 2022b; Nag et
al. 2023), we inject the Gaussian noise to the randomly initialized latent
queries. This is similar to the concept of event queries (Rombach et al.
2022). To detect multiple events occurring simultaneously, we sample $N$ such
noisy event queries to form $Q\in\mathbb{R}^{N\times D}$ which will be
subsequently passed on to the detection decoder for denoising. Taking $Q$ as
input, the decoder predicts $N$ outputs:

$F_{d}=f_{\theta}(Q;C_{a})\in\mathbb{R}^{N\times D}$ (5)

where $C_{a}$ is the encoded audio feature and the $F_{d}$ is the final
embedding. $F_{d}$ is finally decoded using two parallel heads namely (1)
event classification head and (2) event localization head respectively. The
first estimates the probability of a particular event within the event
proposal. The second estimates the onset and offset of event in the raw audio.

### Model training

During training, we first construct the diffusion process that corrupts the
event latents to noisy latents. We then train the model to reverse this
noising process. We add Gaussian noises to the learnable queries. The noise
scale is controlled by $\beta_{t}$ (Eq. (1)), which adopts a monotonically
decreasing cosine schedule in different timestep $t$, following (Ho, Jain, and
Abbeel 2020; Song, Meng, and Ermon 2020). The decoder uses the noisy event
queries (corresponding to $t$) and the global feature $C_{a}$ as the condition
(see Fig 1 (b)) to generate the denoised event queries (corresponding to
$t-1$) repeatedly until an approximation of $Q$ is obtained. The output from
the last denoising step (corresponding to each input event query) is projected
into sigmoidal onset and offset timestamps and an event probability
distribution using separate feedforward projection layers. We observe that SED
favors a relatively high signal scaling value than object detection (Chen et
al. 2022b) (see Table 5).

The event-based objective is defined as a combination of a binary
classification loss for event onset and offset prediction and a cross-entropy
loss for event class prediction. We compute Hungarian assignment between
ground truth boxes and the outputs of the model. We supervise the model
training using each pair of matched ground-truth/prediction (event class and
the temporal boundary).

### Model Inference

In inference, the noisy event queries are randomly sampled from a Gaussian
distribution. Starting from noisy latents sampled from a Gaussian
distribution, the model progressively refines the predictions. At each
sampling step, the random or estimated latents from the last sampling step are
sent into the detection decoder to predict the event category and the event
onset/offsets. After obtaining the event proposals of the current step, DDIM
(Song, Meng, and Ermon 2021) is adopted to estimate the proposals for the next
step. DiffSED has a simple event proposal generation pipeline without post-
processing (e.g., non-maximum suppression).

### Key Insights

##### One model multiple trade-offs

Once trained, DiffSED works under a varying number of event queries and
sampling steps in inference. While inferring, each sampling step involves,
estimating event queries from the last sampling step and sending them back
into the detection decoder to eventually predict the event classes and event
boundaries at the $t_{0}$ step, i.e., fully denoised. In general, better
accuracy can be obtained using more queries and fewer steps (see Table 3 and
Table 4). We discuss the multistep decoding experiments in detail in our
ablation study. Ultimately, it can be determined that a single DiffSED can
meet a number of different trade-off needs between speed and accuracy.

Figure 3: Convergence rates for SEDT and DiffSED on the URBAN-SED dataset. The
dotted lines represent the training epoch when the best-performing checkpoint
(the one with the best audio-tagging F1 score on the validation set) arrived.
DiffSED trains faster ($>$40%) and achieves better optimum than SEDT.

##### Faster convergence

DETR-style detection models suffer generally slow convergence (Liu et al.
2022) due to inconsistent matching of event queries to the event proposals.
Concretely, for the same audio, an event query is often matched with different
event boundaries in different epochs, making the optimization oscillating and
difficult. In DiffSED each query is designed as a proposal proxy – a noised
event query that can be regarded as a good event proposal due to staying close
to the corresponding ground truth boundary. Our query denoising task thus has
a definite optimization objective which is the ground truth proposal. We
validate that query denoising based DiffSED converges faster than SEDT (see
Fig 3), whilst achieving superior performance (Table 1).

Algorithm 2 Noise corruption

⬇

1 def add_noise():

2 """

3 gt_boxes: [B, *, 2]

4 event_queries: [B, N, D]

5 B: batch_size

6 N: number of event queries

7 """

8 if corrupt bounding_boxes: # Diff-SED-BB

9 # Padding (repeat) bounding boxes

10 pb = Pad(gt_boxes, N) #[B, N, 2]

11 # Signal scaling

12 pb = (pb * 2 - 1) * scale

13 # Corrupt bounding boxes

14 t = randint(0, T) #time step

15 eps = normal(mean=0, std=1) #noise: [B, N, 2]

16 pb_crpt = sqrt(alpha_cumprod(t)) * pb + sqrt(1 - alpha_cumprod(t)) * eps

17 event_queries_crpt = Project(pb_crpt)

18 #[B, N, 2] -> [B, N, D]

19 else: # DiffSED

20 # Signal scaling

21 event_queries = (event_queries * 2 - 1) * scale

22 # Corrupt event_queries

23 t = randint(0, T) #time step

24 eps = normal(mean=0, std=1) #noise: [B, N, D]

25 event_queries_crpt = sqrt(alpha_cumprod(t)) * event_queries +

26 sqrt(1 - alpha_cumprod(t)) * eps

27 return event_queries_crpt

## Experiments

##### Datasets

We present our results on two datasets namely, URBAN-SED (Salamon, Jacoby, and
Bello 2014) and EPIC-Sounds (Huh et al. 2023). URBAN-SED is a publicly
available dataset for SED in urban environments. It is accompanied by detailed
annotations, including onset and offset times for each sound event, along with
human generated accurate annotations. The EPIC-Sounds dataset consists of more
than 36,000 audio recordings of various lengths, totaling over 500 hours of
audio. The recordings were made in a variety of indoor and outdoor
environments, including office spaces, public places, and natural
environments. They cover a wide range of sound classes, including human
speech, animal sounds, environmental sounds, and music.

Table 1: Results on URBAN-SED (Test set) Model | Event-based $[\%]$ | Segment-based$[\%]$ | Audio tagging$[\%]$
---|---|---|---
| $\rm{F1}$ | $\rm{P}$ | $\rm{R}$ | $\rm{F1}$ | $\rm{P}$ | $\rm{R}$ | $\rm{F1}$
CRNN-CWin (Miyazaki et al. 2020b) | $36.75$ | $-$ | $-$ | $65.74$ | $-$ | $-$ | $74.19$
Ctrans-CWin (Miyazaki et al. 2020b) | $34.36$ | $-$ | $-$ | $64.73$ | $-$ | $-$ | $74.05$
SEDT (Ye et al. 2021) | $37.27$ | $43.32$ | $33.21$ | $65.21$ | $74.82$ | $58.46$ | $74.37$
DiffSED (Ours) | 43.89 | 48.46 | 37.82 | 69.24 | 77.49 | 62.05 | 77.87
Table 2: Results on EPIC-Sounds (Validation set) Model | Top-1 | Top-5 | mCA | mAP | mAUC
---|---|---|---|---|---
ASF (Kazakos et al. 2021) | 53.47 | 84.56 | 20.22 | 0.235 | 0.879
SSAST (Gong et al. 2022) | 53.75 | 84.54 | 20.11 | 0.254 | 0.873
DiffSED (Ours) | 56.85 | 87.45 | 20.75 | 0.277 | 0.861

##### Evaluation metrics

To evaluate the model’s performance on the URBAN-SED dataset, we measure
F1-score, precision, and recall for both event-level and segment-level
settings on the test split. For the EPIC-Sounds dataset, we report the top-1
and top-5 accuracy, as well as mean average precision (mAP), mean area under
ROC curve (mAUC), and mean per class accuracy (mCA) on the validation split,
following the protocol of (Huh et al. 2023).

### Implementation Details

##### Training schedule

We use a pre-trained encoder backbone ResNet-50 for feature extraction, for
fair comparisons with previous methods (Ye et al. 2021). Our model is trained
for 400 epochs, while re-initializing the weights from the best checkpoint for
every 100 epochs, using Adam optimizer with an initial learning rate of
$10^{-4}$ with a decay schedule of $10^{-2}$. The batch size is set to 64 for
URBAN-SED and 128 for EPIC-Sounds. All models are trained with 2 NVIDIA-A5500
GPUs.

##### Testing schedule

At the inference stage, the detection decoder iteratively refines the
predictions from Gaussian random latent queries. For efficiency, by default,
we denoise for a single time-step, i.e., $T_{0}\leftarrow T_{1000}$ timestep.

### Main Results

##### Results on URBAN-SED

We compare our model with previous end-to-end approaches under the supervised
learning setting. The primary contribution of our work lies in proposing a
diffusion-infused transformer decoder that provides a more robust
representation of grounded event boundaries in the encoded acoustic features.
From Table 1, we draw the following conclusions: (1) The diffusion-based
decoder of DiffSED performs significantly better than all the other methods
for both event-level and segment-level metrics, with a 6.62% and 4.03%
absolute improvement, respectively. (2) Additionally, our model outperforms
existing approaches in terms of audio-tagging results, with a 3.5% absolute
improvement. This validates our model formulation in exploiting the SED
problem as generative learning in the denoising diffusion framework.

##### Results on EPIC-Sounds

We use the publicly available pre-trained backbones ASF (Kazakos et al. 2021)
and SSAST (Gong et al. 2022) as competing models. We observe from Table 2
that: (1) DiffSED consistently outperforms both the alternatives with 3.1% and
2.89% improvement in the Top-1 and Top-5 accuracies, respectively; (2) Our
model performs competitively in the mAUC score.

### Ablation study

We conduct ablation experiments on URBAN-SED to study DiffSED in detail. All
experiments use the pre-trained ResNet-50 backbone features for training and
inference without further specification.

##### Denoising strategy

Due to the inherent query based design with the detection decoder, we discuss
and compare two denoising strategies: (1) Corrupting the event latents in the
continuous space and passing it as queries (referred as DiffSED, our choice).
(2) Corrupting discrete event proposals (i.e., ground-truth bounding boxes)
and projecting it as queries (denoted as DiffSED-BB, detailed in Algorithm 2).
Additionally, we corrupt the label queries using random shuffle as the noise
in the forward diffusion step. To evaluate the effect of the denoising
strategy experimentally, we test both variants using different numbers of
event proposals. It can be observed in Table 3 that both variants achieve the
best audio-tagging performance when using 30 event proposals as input to the
decoder. Also, the overall scores in both event-level and segment-level
metrics are lesser for DiffSED-BB compared to DiffSED. We hypothesize this is
caused by some adversarial effect in projecting the ground-truth bounding box
(2-dimensional) to the latent event query.

Table 3: Effect of the number of queries on the performance for URBAN-SED Test set. (AT: Audio Tagging performance) | #Queries | Event-F1$[\%]$ | Segment-F1$[\%]$ | AT$[\%]$
---|---|---|---|---
DiffSED-BB | 10 | $31.43$ | $58.85$ | $68.87$
20 | $35.32$ | $60.53$ | $68.84$
30 | 37.29 | 60.91 | 69.61
40 | $31.95$ | $58.79$ | $68.41$
50 | $31.81$ | $57.89$ | $68.31$
DiffSED | 10 | $40.78$ | $68.41$ | $77.22$
20 | 41.42 | 68.73 | $76.54$
30 | $41.3$ | $68.21$ | 77.46
40 | $38.65$ | $67.21$ | $75.21$
50 | $36.28$ | $64.22$ | $72.77$

##### Multistep decoding

We tabulate the results upon varying the number of denoising steps for both
DiffSED and DiffSED-BB in Table 4. We observe a steady improvement over the
event-level and segment-level F1 scores as we increase the number of denoising
steps from 1 to 5 and then gradually decrease when using 10 decoding steps.
However, the best audio tagging performance is achieved when performing a
single-step decoding. We hypothesize this is primarily because the event
boundaries have short-range temporal dependencies that might not benefit
significantly from multistep denoising. The noise addition mainly affects each
time step independently and doesn’t accumulate over multiple steps hence does
not yield substantial improvements. Denoising over multiple timesteps requires
more computing, while providing only a marginal gain thus not worthwhile.

Table 4: Effect of the number of denoising steps used while inference on the performance for URBAN-SED Test set. (AT: Audio Tagging performance) | #steps | Event-F1$[\%]$ | Segment-F1$[\%]$ | AT$[\%]$
---|---|---|---|---
DiffSED-BB | 1 | 39.78 | $64.74$ | 72.92
5 | $38.27$ | 65.72 | 71.88
10 | $38.3$ | $64.82$ | $72.17$
DiffSED | 1 | $43.89$ | $69.24$ | 77.87
5 | 44.35 | 70.75 | $77.07$
10 | $43.50$ | $69.05$ | $77.36$

##### Signal scaling

The signal scaling factor controls the signal-to-noise ratio (SNR) of the
diffusion process. We study the influence of scaling factors. The results in
Table 5 demonstrate that the scaling factor of 0.4 achieves the highest audio-
tagging performance as well as all other metrics for DiffSED, whereas for
DiffSED-BB the best audio tagging performance is obtained for a scaling factor
of 0.2 whilst achieving the best event-level and segment-level F1 score for a
scaling factor of 0.4. This suggests the relationship between optimal scaling
and the denoising strategy.

Table 5: Effect of scaling the noise factor on the performance for URBAN-SED Test set. (AT: Audio Tagging Performance) | Noise scale | Event-F1$[\%]$ | Segment-F1$[\%]$ | AT$[\%]$
---|---|---|---|---
DiffSED-BB | 0.1 | $32.61$ | $32.45$ | $73.49$
0.2 | $35.91$ | $35.73$ | 75.73
0.3 | $37.29$ | $60.91$ | $69.61$
0.4 | 39.78 | 64.74 | $72.92$
0.5 | $33.14$ | $61.79$ | $71.12$
DiffSED | 0.1 | $37.61$ | $54.63$ | $72.2$
0.2 | $39.65$ | $58.17$ | $73.89$
0.3 | $41.3$ | $68.21$ | $77.46$
0.4 | 43.89 | 69.24 | 77.87
0.5 | $39.23$ | $59.25$ | $72.78$
Table 6: Effect of changing the seed value for inducing noise during inference. Values inside $(.)$ indicate deviation from the mean calculated over 3 runs.  | Runs | Event-F1$[\%]$ | Segment-F1$[\%]$ | AT$[\%]$
---|---|---|---|---
DiffSED-BB | 1 | $38.6(\uparrow 0.2)$ | $64.32(\downarrow 0.09)$ | $72.48(0.0)$
2 | $39.45(\downarrow 0.57)$ | $64.15(\uparrow 0.07)$ | $72.88(\downarrow 0.4)$
3 | $38.57(\uparrow 0.3)$ | $64.21(\uparrow 0.01)$ | $72.08(\uparrow 0.4)$
Avg | 38.87 | 64.22 | 72.48
DiffSED | 1 | $43.12(\downarrow 0.2)$ | $68.38(\uparrow 0.5)$ | $77.62(\downarrow 0.01)$
2 | $42.35(\uparrow 0.5)$ | $68.97(\uparrow 0.01)$ | $77.59(\uparrow 0.02)$
3 | $43.29(\downarrow 0.3)$ | $69.54(\downarrow 0.5)$ | $77.62(\downarrow 0.01)$
Avg | 42.92 | 68.96 | 77.61

##### Random seed

DiffSED starts with random noisy event queries as input during inference. We
evaluate the stability of DiffSED and DiffSED-BB by training three models
independently with strictly the same configurations (30 noisy event proposals
as input to the decoder and a scaling factor of 0.4) except for random seed on
URBAN-SED dataset. Then, we evaluate each model instance with 3 different
random seeds to measure the distribution of performance, inspired by (Chen et
al. 2022b; Nag et al. 2023). As shown in Table 6, most evaluation results are
distributed closely to the average metrics for both variants. This
demonstrates that our models are robust to random event queries.

## Conclusion

In this work, we reformulate the Sound Event Detection (SED) problem from the
generative learning perspective, in particular under the diffusion-based
transformer framework. We introduce a diffusion adaptation method
characterized by noisy event latents denoising. This design has the advantage
of being able to model the global dependencies of sound events, while still
being computationally efficient. Our study verifies the efficacy of diffusion
models in a new problem context (i.e., SED), consistent with previous
findings. Experiments show that our method is superior to existing art
alternatives on standard benchmarks.

## References

  * Bear, Nolasco, and Benetos (2019) Bear, H. L.; Nolasco, I.; and Benetos, E. 2019.  Towards joint sound scene and polyphonic sound event recognition.  In _INTERSPEECH_.
  * Bhosale, Chakraborty, and Kopparapu (2023) Bhosale, S.; Chakraborty, R.; and Kopparapu, S. K. 2023.  A Novel Metric For Evaluating Audio Caption Similarity.  In _IEEE ICASSP_.
  * Cakır et al. (2017) Cakır, E.; Parascandolo, G.; Heittola, T.; Huttunen, H.; and Virtanen, T. 2017\.  Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_.
  * Carion et al. (2020) Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; and Zagoruyko, S. 2020.  End-to-end object detection with transformers.  In _European conference on computer vision_ , 213–229. Springer.
  * Chao et al. (2018) Chao, Y.-W.; Vijayanarasimhan, S.; Seybold, B.; Ross, D. A.; Deng, J.; and Sukthankar, R. 2018.  Rethinking the faster r-cnn architecture for temporal action localization.  In _IEEE CVPR_.
  * Chen et al. (2022a) Chen, K.; Du, X.; Zhu, B.; Ma, Z.; Berg-Kirkpatrick, T.; and Dubnov, S. 2022a.  HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection.  In _IEEE ICASSP_.
  * Chen et al. (2019) Chen, K.; Wang, J.; Pang, J.; Cao, Y.; Xiong, Y.; Li, X.; Sun, S.; Feng, W.; Liu, Z.; Xu, J.; Zhang, Z.; Cheng, D.; Zhu, C.; Cheng, T.; Zhao, Q.; Li, B.; Lu, X.; Zhu, R.; Wu, Y.; Dai, J.; Wang, J.; Shi, J.; Ouyang, W.; Loy, C. C.; and Lin, D. 2019.  MMDetection: Open MMLab Detection Toolbox and Benchmark.  _arXiv preprint arXiv:1906.07155_.
  * Chen et al. (2022b) Chen, S.; Sun, P.; Song, Y.; and Luo, P. 2022b.  Diffusiondet: Diffusion model for object detection.  _arXiv preprint arXiv:2211.09788_.
  * Dhariwal and Nichol (2021) Dhariwal, P.; and Nichol, A. 2021.  Diffusion models beat gans on image synthesis.  _NeurIPS_.
  * Duan et al. (2019) Duan, K.; Bai, S.; Xie, L.; Qi, H.; Huang, Q.; and Tian, Q. 2019.  Centernet: Keypoint triplets for object detection.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 6569–6578.
  * Gong et al. (2022) Gong, Y.; Lai, C.-I.; Chung, Y.-A.; and Glass, J. 2022.  Ssast: Self-supervised audio spectrogram transformer.  In _AAAI Conference on Artificial Intelligence_.
  * Ho, Jain, and Abbeel (2020) Ho, J.; Jain, A.; and Abbeel, P. 2020.  Denoising diffusion probabilistic models.  _Neural Information Processing Systems (NeurIPS)_.
  * Huang et al. (2022) Huang, R.; Lam, M. W.; Wang, J.; Su, D.; Yu, D.; Ren, Y.; and Zhao, Z. 2022.  Fastdiff: A fast conditional diffusion model for high-quality speech synthesis.  _International Joint Conferences on Artificial Intelligence (IJCAI)_.
  * Huh et al. (2023) Huh, J.; Chalk, J.; Kazakos, E.; Damen, D.; and Zisserman, A. 2023.  EPIC-SOUNDS: A Large-Scale Dataset of Actions that Sound.  In _IEEE ICASSP_.
  * Igarashi et al. (2022) Igarashi, A.; Imoto, K.; Komatsu, Y.; Tsubaki, S.; Hario, S.; and Komatsu, T. 2022\.  How Information on Acoustic Scenes and Sound Events Mutually Benefits Event Detection and Scene Classification Tasks.  In _IEEE Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_.
  * Kazakos et al. (2021) Kazakos, E.; Nagrani, A.; Zisserman, A.; and Damen, D. 2021.  Slow-fast auditory streams for audio recognition.  In _IEEE ICASSP_.
  * Koh et al. (2021) Koh, C.-Y.; Chen, Y.-S.; Liu, Y.-W.; and Bai, M. R. 2021.  Sound event detection by consistency training and pseudo-labeling with feature-pyramid convolutional recurrent neural networks.  In _IEEE ICASSP_.
  * Kumar, Khadkevich, and Fügen (2018) Kumar, A.; Khadkevich, M.; and Fügen, C. 2018.  Knowledge transfer from weakly labeled audio using convolutional neural network for sound events and scenes.  In _IEEE ICASSP_.
  * Lemercier et al. (2022) Lemercier, J.-M.; Richter, J.; Welker, S.; and Gerkmann, T. 2022.  Analysing Diffusion-based Generative Approaches versus Discriminative Approaches for Speech Restoration.  In _IEEE ICASSP_.
  * Leng et al. (2022) Leng, Y.; Chen, Z.; Guo, J.; Liu, H.; Chen, J.; Tan, X.; Mandic, D.; He, L.; Li, X.-Y.; Qin, T.; et al. 2022.  Binauralgrad: A two-stage conditional diffusion probabilistic model for binaural audio synthesis.  _Neural Information Processing Systems (NeurIPS)_.
  * Li et al. (2022) Li, X. L.; Thickstun, J.; Gulrajani, I.; Liang, P.; and Hashimoto, T. B. 2022.  Diffusion-LM Improves Controllable Text Generation.  _arXiv preprint arXiv:2205.14217_.
  * Li et al. (2020) Li, Y.; Liu, M.; Drossos, K.; and Virtanen, T. 2020.  Sound event detection via dilated convolutional recurrent neural networks.  In _IEEE ICASSP_.
  * Lim, Park, and Han (2017) Lim, H.; Park, J.-S.; and Han, Y. 2017.  Rare Sound Event Detection Using 1D Convolutional Recurrent Neural Networks.  In _Detection Classification Acoustic Scenes Events (DCASE) Workshop_.
  * Lin et al. (2019) Lin, L.; Wang, X.; Liu, H.; and Qian, Y. 2019.  Guided learning convolution system for dcase 2019 task 4.  _arXiv preprint arXiv:1909.06178_.
  * Lin et al. (2021) Lin, M.; Li, C.; Bu, X.; Sun, M.; Lin, C.; Yan, J.; Ouyang, W.; and Deng, Z. 2021\.  DETR for Crowd Pedestrian Detection.  arXiv:2012.06785.
  * Liu et al. (2022) Liu, S.; Li, F.; Zhang, H.; Yang, X.; Qi, X.; Su, H.; Zhu, J.; and Zhang, L. 2022\.  DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR.  In _International Conference on Learning Representations_.
  * Lutati, Nachmani, and Wolf (2023) Lutati, S.; Nachmani, E.; and Wolf, L. 2023.  Separate And Diffuse: Using a Pretrained Diffusion Model for Improving Source Separation.  _arXiv preprint arXiv:2301.10752_.
  * Mesaros et al. (2021) Mesaros, A.; Heittola, T.; Virtanen, T.; and Plumbley, M. D. 2021.  Sound event detection: A tutorial.  _IEEE Signal Processing Magazine_.
  * Miyazaki et al. (2020a) Miyazaki, K.; Komatsu, T.; Hayashi, T.; Watanabe, S.; Toda, T.; and Takeda, K. 2020a.  Convolution-augmented transformer for semi-supervised sound event detection.  In _Detection Classification Acoustic Scenes Events (DCASE) Workshop_.
  * Miyazaki et al. (2020b) Miyazaki, K.; Komatsu, T.; Hayashi, T.; Watanabe, S.; Toda, T.; and Takeda, K. 2020b.  Weakly-supervised sound event detection with self-attention.  In _IEEE ICASSP_.
  * Nag et al. (2023) Nag, S.; Zhu, X.; Deng, J.; Song, Y.-Z.; and Xiang, T. 2023.  DiffTAD: Temporal Action Detection with Proposal Denoising Diffusion.  _arXiv preprint arXiv:2303.14863_.
  * Nag et al. (2022a) Nag, S.; Zhu, X.; Song, Y.-Z.; and Xiang, T. 2022a.  Proposal-free temporal action detection via global segmentation mask learning.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part III_ , 645–662. Springer.
  * Nag et al. (2022b) Nag, S.; Zhu, X.; Song, Y.-z.; and Xiang, T. 2022b.  Semi-Supervised Temporal Action Detection with Proposal-Free Masking.  In _ECCV_.
  * Nag et al. (2022c) Nag, S.; Zhu, X.; Song, Y.-z.; and Xiang, T. 2022c.  Zero-Shot Temporal Action Detection via Vision-Language Prompting.  In _ECCV_.
  * Rombach et al. (2022) Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 10684–10695.
  * Salamon, Jacoby, and Bello (2014) Salamon, J.; Jacoby, C.; and Bello, J. P. 2014.  A dataset and taxonomy for urban sound research.  In _ACM International Conference on Multimedia_.
  * Shi et al. (2022) Shi, D.; Zhong, Y.; Cao, Q.; Zhang, J.; Ma, L.; Li, J.; and Tao, D. 2022.  ReAct: Temporal Action Detection with Relational Queries.  In _European conference on computer vision_.
  * Song, Meng, and Ermon (2020) Song, J.; Meng, C.; and Ermon, S. 2020.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_.
  * Song, Meng, and Ermon (2021) Song, J.; Meng, C.; and Ermon, S. 2021.  Denoising Diffusion Implicit Models.  In _International Conference on Learning Representations_.
  * Tan et al. (2021) Tan, J.; Tang, J.; Wang, L.; and Wu, G. 2021.  Relaxed transformer decoders for direct action proposal generation.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * Tian et al. (2019) Tian, Z.; Shen, C.; Chen, H.; and He, T. 2019.  Fcos: Fully convolutional one-stage object detection.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 9627–9636.
  * Turpault et al. (2019) Turpault, N.; Serizel, R.; Salamon, J.; and Shah, A. P. 2019.  Sound event detection in domestic environments with weakly labeled data and soundscape synthesis.  _Detection Classification Acoustic Scenes Events (DCASE) Workshop_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wakayama and Saito (2022) Wakayama, K.; and Saito, S. 2022.  CNN-Transformer with Self-Attention Network for Sound Event Detection.  In _IEEE ICASSP_.
  * Wang et al. (2020) Wang, X.; Kong, T.; Shen, C.; Jiang, Y.; and Li, L. 2020.  Solo: Segmenting objects by locations.  In _ECCV_.
  * Xie et al. (2023) Xie, Z.; Xu, X.; Wu, M.; and Yu, K. 2023.  Enhance Temporal Relations in Audio Captioning with Sound Event Detection.  _arXiv preprint arXiv:2306.01533_.
  * Xu et al. (2021) Xu, X.; Dinkel, H.; Wu, M.; and Yu, K. 2021.  Text-to-audio grounding: Building correspondence between captions and sound events.  In _IEEE ICASSP_.
  * Ye et al. (2021) Ye, Z.; Wang, X.; Liu, H.; Qian, Y.; Tao, R.; Yan, L.; and Ouchi, K. 2021.  Sound Event Detection Transformer: An Event-based End-to-End Model for Sound Event Detection.  _arXiv preprint arXiv:2110.02011_.
  * Zhang et al. (2022) Zhang, H.; Li, F.; Liu, S.; Zhang, L.; Su, H.; Zhu, J.; Ni, L.; and Shum, H. 2022\.  Dino: Detr with improved denoising anchor boxes for end-to-end object detection.  In _International Conference on Learning Representations (ICLR)_.
  * Zhu et al. (2020) Zhu, X.; Su, W.; Lu, L.; Li, B.; Wang, X.; and Dai, J. 2020.  Deformable DETR: Deformable Transformers for End-to-End Object Detection.  _arXiv preprint_.


0 Research

# Evaluation of cinematic volume rendering open-source and commercial
solutions for the exploration of congenital heart data

Irum Baseer
Universitat Pompeu Fabra e-mail<EMAIL_ADDRESS>Israel Valverde
Institute of Biomedicine of Seville e-mail<EMAIL_ADDRESS>Abdel H.
Moustafa
Hospital de la Santa Creu i Sant Pau<EMAIL_ADDRESS>Josep Blat
Universitat Pompeu Fabra<EMAIL_ADDRESS>Oscar Camara
Universitat Pompeu Fabra e-mail<EMAIL_ADDRESS>
###### Abstract

Detailed anatomical information is essential to optimize medical decisions for
surgical and pre-operative planning in patients with congenital heart disease.
The visualization techniques commonly used in clinical routine for the
exploration of complex cardiac data are based on multi-planar reformations,
maximum intensity projection, and volume rendering, which rely on basic
lighting models prone to image distortion. On the other hand, cinematic
rendering (CR), a three-dimensional visualization technique based on
physically-based rendering methods, can create volumetric images with high
fidelity. However, there are a lot of parameters involved in CR that affect
the visualization results, thus being dependent on the user’s experience and
requiring detailed evaluation protocols to compare available solutions. In
this study, we have analyzed the impact of the most relevant parameters in a
CR pipeline developed in the open-source version of the MeVisLab framework for
the visualization of the heart anatomy of three congenital patients and two
adults from CT images. The resulting visualizations were compared to a
commercial tool used in the clinics with a questionnaire filled in by clinical
users, providing similar definitions of structures, depth perception, texture
appearance, realism, and diagnostic ability.

Cinematic renderingopen-sourcecommercial toolcongenital heart data

Introduction

Congenital heart disease (CHD) is one of the most frequently diagnosed defects
afflicting approximately 0.8% to 1.2% of live births worldwide [2]. As the
cardiovascular morphology varies greatly between individual patients, it is
important for clinicians to have a comprehensive understanding of the spatial
relationship between the cardiac structures, in order to make optimal medical
decisions. As a result, there is a growing number of computer-aided software
tools available to assist radiologists in this process. One of the widely used
methods is volume rendering, which was found to represent human structures in
an artificial way, also being prone to image distortions [8] due to their
reliance on basic lighting models. On the other hand, cinematic rendering (CR)
is a novel post-processing tool [10] that renders the volumetric medical data
using physically-based advanced lighting models [12]. CR resembles the casting
of billions of light rays from all possible directions to create volumetric
images with a remarkable level of realism [6].

|  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
Figure 1: Comparison of volume rendering (VR) and cinematic rendering (CR)
visualizations for congenital heart disease patients (Cases 1-3) and normal
adults (Cases 4-5). Top row: VR. Middle row: CR from commercial solution.
Bottom row: CR from open-source solution.

Benefits of cinematic vs volume rendering are increasingly being reported in
medical applications, such as for faster comprehension of anatomy [9] and for
pre-operative planning[17]. Additionally, studies have shown that more
accurate visualization of medical data benefits imaging tasks including
delineating complex congenital heart pathologies[15]. Apart from its clinical
utility, CR also has the potential to be useful in patient communication [7]
and education [1].

The most widely used commercial implementation of CR[5] is offered by Siemens
Healthineers as part of their syngo.via
platform111https://www.siemenshealthineers.com/digital-health-
solutions/cinematic-rendering. Several other license-limited solutions include
Global Illumination Vitrea by Canon Medical
Informatics222https://www.vitalimages.com/global-illumination/ and
MeVisLab333https://www.mevislab.de/download/ are also available. Vitrea offers
global illumination methods for rendering volumetric data in a photo-realistic
manner, while MeVisLab offers a path tracer module, which is a significantly
enhanced version of the ExposureRender [12] framework by Thomas Kroes. For
instance, MeVisLab has recently been used for post-surgical assessment in
oncologic head and neck reconstructive surgery comparing path tracing and
volume rendering techniques [4]. While these vendor-provided solutions often
have high rendering capabilities and are utilized in advanced healthcare
centers, their cost can be prohibitive for smaller institutions or individual
researchers. However, there are some open-source alternatives.
Voreen444https://www.uni-muenster.de/Voreen and Inviwo555http://www.inviwo.org
offer better volumetric rendering capabilities by implementing ray casting
with global illumination. Yet, neither of these applications utilizes
volumetric path tracing or equivalent state-of-the-art volumetric rendering
techniques. Another open-source and freely available solution supporting CR in
web browsers is VolView [18]. These solutions are typically free to use,
customizable to meet specific needs, and can be used by anyone with an
internet connection, regardless of their location or financial resources.
While these solutions are affordable and flexible, there is a lack of research
comparing different solutions in cardiac applications, to identify the
strengths and weaknesses of open-source software tools as compared to
commercial solutions.

In this study, we utilized a free version of MeVisLab to design a pipeline for
cinematically rendering a CHD dataset. We conducted a detailed evaluation of
several critical parameters to enhance the shape and depth perception of the
heart anatomy. Furthermore, we assessed the performance of the developed open-
source rendering pipeline by comparing it with a commercial solution available
in clinics. This evaluation was conducted using a questionnaire filled out by
cardiology experts.

##  1 Materials and methods

###  1.1 Patient cases and reconstructions

We included clinical data of 3 congenital heart disease patients who underwent
CT imaging for diagnosis or treatment, along with CT data of two normal adult
hearts. A brief summary of patient data is provided in Table 1. For creating
CR visualizations from CT data, anonymized DICOM reconstructions were
transferred to both used environments: MeVisLab and a workstation with
prototype commercial software (syngo.via cinematic VRT, Siemens Healthineers).
Each case was carefully displayed using the same zooming and rotation features
in both software; the cutting tool was applied to eliminate any bones and
remaining lines that could obscure the view. To ensure comparability, the
generated reconstructions were captured using the same angle of view, color,
and opacity settings (see Figure 1).

Table 1: Patient cases and reconstructions. M/F: male/female.

Case | M/F | Age | Condition | Manufacturer | Voxel spacing
---|---|---|---|---|---
1 | F | 3 years | Pulmonary atresia | GE Medical | $0.273\times 0.273\times 0.625$
2 | M | 4 days | Ventricular septal defect | Canon Medical | $0.163\times 0.163\times 0.250$
3 | M | 2 years | Occluded arterial duct | Siemens | $0.246\times 0.246\times 0.400$
4 | F | 46 years | Normal heart | GE Medical | $0.559\times 0.559\times 0.625$
5 | M | 63 years | Normal heart | Philips | $0.576\times 0.576\times 0.329$

###  1.2 Cinematic rendering pipeline and sensitivity analysis

To build the cinematic rendering pipeline, the MeVisLab software (version
3.5.0) was installed on a personal computer (AMD FX (tm), eight-core
processor, 3.50GHz, 32 GB RAM, 64-bits operating system). The pipeline
involved a visual programming approach, combining various modules to load
imaging data, perform pre-processing, design transfer functions, set material
and lighting properties, and render the volume followed by post-processing.
These steps are briefly explained below.

Data loading and pre-processing. The first step of the rendering involved
loading patient data, usually in the form of a series of slices. MeVisLab’s
DirectDicomImport module was used to directly import the image files. To
reduce noise in the acquired data, the GaussSmoothing module in MeVisLab was
applied.

Transfer function. After pre-processing, the next step involved designing a
transfer function. A transfer function maps voxel values to visual properties
like color and opacity, enabling the distinction of anatomical structures in
the image. The SoLUTEditor module allowed interactive editing of RGBA lookup
tables to design transfer functions. We generated a range of preset transfer
functions, saving them in CSV format for easy loading via a Python script. The
module’s window level and width option allowed for setting a color range for
displaying specific anatomical structures, similar to conventional CT
reconstructions.

Shading and lighting. Once the data was mapped to the transfer function,
lighting and shading were applied to the volume using a physically-based
rendering (PBR) workflow. PBR has two principal workflows[14]: metal/roughness
and specular/glossiness. In PBR, shading is achieved using various
Bidirectional Reflectance Distribution Function (BRDF), which are mathematical
models that describe how light reflects off surfaces based on their physical
properties. The SoPathTracerMaterial module in MeVisLab provides a range of
different materials from which Material_Microfacet is based on the
specular/glossiness workflow of PBR, whereas Material_Principled is based on
the metal/roughness workflow. We used ’Material_Principled’, a physically-
based material whose parameters are based on the Disney BRDF model[3]. We
adjusted parameters like base color, metallic, roughness, and specular
properties to achieve realistic material appearances. For lighting, the
pipeline included the SoPathTracerAreaLight and SoPathTracerBackgroundLight
modules to simulate realistic lighting effects in the 3D scene. The intensity,
color, and position of both lights were carefully adjusted to create the
desired lighting effect.

|  |
---|---|---
(a) | (b) | (c)
Figure 2: Effects of different values of roughness (r): (a) r = 0, results in
a completely shiny and metallic surface (b) r = 0.5, introduces some subtle
variations in the surface texture, resulting in a more natural appearance. and
(c) r = 1, shows a highly rough surface, leading to a matte surface from left
to right, keeping metallic and specular equal to 0.5.

In order to determine which parameters should be adjusted and which values
achieved the most realistic appearance, a sensitivity analysis was performed.
Specifically, the roughness parameter was tested with constant metallic and
specular values of 0.5, producing three images with different roughness
levels, as shown in Figure 2. The impact of lighting on the final image was
also examined by varying the number and position of lights. For instance,
adding two light sources in the same position tends to create blurry
reflections (overexposure) as compared to a single light source, which focuses
better on the details of the image as can be seen in Figure 3. Moreover,
adjusting the position of light sources improved the visualization of specific
regions, including shadows and depth. We also tested the effects of area and
background lighting, where area lighting is positioned on the top right and
the background light source is positioned behind the objects being lit,
providing overall illumination of the scene. As illustrated in Figure 4, the
use of only area lighting tended to overexpose certain areas and obscure
details. On the other hand, the use of background white lighting alone
resulted in a more diffused and natural-looking, while combining area and
background lighting produced dynamic effects, highlighting specific features
of the volume with area light, and creating a sense of depth and
dimensionality using the background light.

|
---|---
(a) | (b)
Figure 3: Effects of single and multiple lights at the same position. Volume is illuminated with (a) single light, and (b) two lights. |  |
---|---|---
(a) | (b) | (c)
Figure 4: Effects of area and background lighting. Volume is illuminated with:
(a) area lighting positioned on the top right; (b) background white lighting;
and (c) combination of area and background lighting.

Image-based lighting (IBL). IBL is a computer graphics technique that employs
a high-dynamic-range image (HDRI) as a light source. In PBR, background light
refers to the light coming from the environment surrounding the rendered
scene. We utilized the SoPathTracerBackgroundLight module to support IBL with
cubemaps for rendering the volume (Figure 5), demonstrating how different
lighting strategies can influence the overall appearance of the rendered
image. For instance, the use of two area light sources in the first image
provides a more focused and detailed illumination, while IBL with HDRI creates
a more realistic and immersive illumination by capturing lighting and
reflection information from the surroundings.

|  |
---|---|---
|  |
(a) | (b) | (c)
Figure 5: Cinematic renderings using different lighting strategies: (a) use of
two area light sources, (b-c) use of image-based lighting with two different
high-dynamic range images; cloudy sky and a Vasamuseum Humus666This is the
work of Emil Persson, also known as Humus. http://www.humus.name

Rendering. Once color, material, and lighting are adjusted, we integrate the
SoPathTracer module for rendering. This module utilizes a Monte Carlo path
tracing method to simulate light transport through the anatomy and create
photo-realistic images. Path tracing is a common technique in computer
graphics [11], generating paths of scattering events from the camera to light
sources, resulting in multi-scattering. The Monte Carlo integration method
solves the following multi-dimensional and non-continuous rendering equation
1, considering the properties of the scene and the physical interactions of
light with the materials:

$L_{o}(x,\omega_{o})=L_{e}(x,\omega_{o})+\int_{\Omega}f_{r}(x,\omega_{i},\omega_{o})L_{i}(x,\omega_{i})|\cos\theta|\mathrm{d}\omega_{i},$
(1)

where $L_{o}(\mathbf{x},\omega_{o})$ and $L_{e}(\mathbf{x},\omega_{o})$ are
the emitted (from the surface) and outgoing radiances, respectively, at point
$\mathbf{x}$ in direction $\omega_{o}$,
$f_{r}(\mathbf{x},\omega_{i},\omega_{o})$ is the bidirectional reflectance
distribution function (BRDF) that describes how much light is reflected in
different directions $\omega_{o}$ from direction $\omega_{i}$,
$L_{i}(\mathbf{x},\omega_{i})$ is the incoming radiance at point $\mathbf{x}$
from direction $\omega_{i}$, and $|\cos\theta|$ where theta is the angle
between the surface normal and $\omega_{i}$.

After setting up all the light sources and material properties, the user can
interact with the rendering and adjust the camera projection type.
Furthermore, post-processing tools such as the SoPostEffectAmbientOcclusion
module were applied to improve the shadowing effect and depth perception.
Moreover, the SoVolumeCutting and clip plane modules were added to allow image
editing, enabling the exposure and display of specific regions of interest, as
well as isolating the heart from adjacent structures such as bones and
vessels.

Table 2: Likert’s scale evaluation of cinematic renderings across all cardiac
structures and characteristics from the commercial and
open-source solutions.

Characteristics $\longrightarrow$ Anatomical structure $\downarrow$ | Definition of structure | Depth perception | Texture appearance | Fidelity | Diagnostic ability
---|---|---|---|---|---
Commercial | Open | Commercial | Open | Commercial | Open | Commercial | Open | Commercial | Open
Atria | 4.00 | 3.67 | 4.67 | 4.33 | 3.67 | 3.33 | 3.67 | 3.33 | 4.67 | 4.00
Ventricles | 4.00 | 3.33 | 4.67 | 4.33 | 4.00 | 3.67 | 4.00 | 3.67 | 4.67 | 4.00
Great Arteries | 4.33 | 4.00 | 4.67 | 4.33 | 4.67 | 4.67 | 4.00 | 3.67 | 4.33 | 4.00
Coronary Arteries | 2.33 | 2.33 | 4.67 | 4.33 | 4.67 | 4.33 | 3.67 | 3.33 | 4.33 | 4.00
Average | 3.67 | 3.33 | 4.67 | 4.33 | 4.25 | 4.00 | 3.83 | 3.50 | 4.50 | 4.00

##  2 Evaluation of cinematic rendering

###  2.1 Assessment protocol

The overall evaluation consisted of subjective assessments of photo-realistic
static snapshots. Three independent domain-expert cardiologists, two cardiac
radiologists with 8 years of experience, and one pediatric cardiologist with
15 years of experience, conducted the evaluations.  The snapshots were
acquired in such a way as to enable independent ratings for various anatomical
structures, including the atria, ventricles, great arteries, and coronary
arteries. A score was required for the snapshots on a Likert scale from 1 to 5
(e.g., very unsatisfied to very satisfied). Specifically, five questions were
asked per case, related to the following visual characteristics, adapted from
[13, 16]:

  * •

_Definition of structure_ describes the sharpness of the edges, e.g., for
performing an anatomical measurement.

  * •

_Depth perception_ describes the ability to perceive spatial relationships in
3D (e.g., anterior/posterior).

  * •

_Texture appearance_ refers to the appearance of the surfaces in terms of
their degree of roughness and metalness.

  * •

_Fidelity_ is a characteristic analyzing the sensation of resembling real
cardiac tissue on screen.

  * •

_Diagnostic ability_ refers to the effectiveness of the rendered images in
supporting clinical diagnosis.

The questionnaire also included questions about using open-source and
commercial solutions for CR. Two questions used a Likert scale to assess the
reliability of open-source tools and whether the investigators would recommend
them to others. The third question asked for an open-ended opinion about the
preference for using open-source or commercial tools for advanced rendering.

###  2.2 Results

Figure 1 presents the conventional volume alongside cinematic renderings of
all five analyzed cases. The top row showcases the volume rendering, while the
middle and bottom rows display cinematic renderings created using the
commercial solution and open-source framework. In the open-source framework,
all cases underwent CR with two area light sources, background white light,
and distinct material properties, followed by a post-processing step.
Furthermore, noise removal was applied during preprocessing for the congenital
cases, as they often had higher noise levels compared to normal adult hearts.
Alongside these visual results, table 2 complements these visual results by
providing a comprehensive overview of our evaluation findings. The analysis
was performed by taking the average of all anatomical structure scores across
all characteristics. All investigators found the renderings from the
commercial solution to be superior or equivalent to the open-source
alternative reconstructions, for all analyzed anatomical structures and visual
characteristics. However, both solutions obtained satisfaction scores of a
similar scale, with no significant differences overall. Case 3 notably
highlights the importance of CR. Here, the occluded arterial duct (indicated
by the red arrow) is challenging to identify using only VR; however, both
alternative methods offer comparable visual results. The analysis further
reveals minor distinctions in each visual characteristic assessment for the
Great Arteries, with variances of approximately 0.33 and 0.34. Interestingly,
the texture appearance exhibits remarkable equivalence, emphasizing the
effectiveness of both solutions in capturing texture details. Moreover, case
2, involving a ventricular septal defect, exhibited enhanced visual appeal
when rendered using an open-source solution; however, the analysis indicates
that commercial CRs outperformed across all visual characteristics.
Furthermore, in perceiving the depth of anatomical structures both CRs
performed equally. Additionally, the evaluators found that the edges of
anatomical structures were better visualized with the commercial tool, with a
mean difference in the definition of structure characteristic of 0.34 with
respect to the open-source renderings. We can also observe that the lowest
scores for both CRs solutions corresponded to the definition of the coronary
arteries, due to their complex anatomy.

##  3 Discussion and conclusions

Cinematic rendering for complex cardiac data is a promising 3D visualization
tool, which is mainly performed on commercial tools in clinical environments.
These tools have high-performance capabilities but are relatively costly, thus
not being accessible to everyone. In the present study, we have designed a
photo-realistic rendering pipeline using the open-source SDK version of
MeVisLab as an alternative.

To achieve the best possible cinematic rendering visualizations, we performed
a sensitivity analysis to identify the most relevant parameters (e.g.,
material properties) and their effect on the final results. A limitation of
fixing values for material properties is that the same value will not result
in the same texture appearance for each case since materials behave
differently for each anatomy which also depends on the image quality [9]. We
also analyzed the effects of lighting and found that the number of light
sources and their positions are crucial for creating visually compelling and
informative images with higher depth perception. Moreover, area lights offer
focused and detailed lighting, while background lighting with HDRI provides
more realistic and immersive illumination by capturing lighting and reflection
information from the surrounding environment.

An evaluation protocol was jointly designed with cardiologists to compare the
CRs provided by the developed open-source pipeline and from a commercial
solution available in a hospital environment, assessing several visual
characteristics of different cardiac structures. Results obtained by three
independent cardiologists were consistent in recognizing an overall superior
performance of the commercial solution, with slightly higher scores on the
Likert scale, especially for the definition of structures, texture appearance,
and diagnostic ability. However, the open-source solution had several visual
characteristics with a 4.00 or beyond, being similar to the commercial
alternative for fidelity and depth perception. Both types of CRs performed
worst for the definition of structures due to the low scores given to the
coronary arteries. The cardiologists also expressed their satisfaction with
the reliability of open-source rendering tools and their recommendation to
others for advanced rendering purposes. Their overall preference was for open-
source tools, mainly due to their cost-effectiveness.

Computationally speaking, both solutions were highly dependent on suitable
hardware. The quality of the renderings was dependent on the number of rays to
be traced, which in turn depended on the computational power. For example, on
a workstation with a GPU RTX 1050, the image was completely rendered in 29 sec
(300 iterations) with the open-source solution. However, with a GPU NVIDIA RTX
A6000, it only took 3 seconds to completely render the same image.

The main strength of the open-source CR pipeline developed in the study is
that it allows continuous improvement (GitHub Project), being a ”white box”
tool enabling to tune parameters such as transfer functions, materials, and
lighting to obtain better visualizations, as per clinician’s requirement on
specific patients. Considering the accessibility of the open-source solution
and the similarity of the corresponding satisfaction scores to the commercial
tool, the developed pipeline is an exciting alternative, mainly for
educational purposes and to support medical diagnosis as a pre-operative
planning tool. Future work will be devoted to a more extensive evaluation with
more analyzed cases, evaluators, and used software tools, including web-based
solutions democratizing the use of open-source CR in hospitals independently
of their resources.

###### Acknowledgements.

This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No 101016496
(SimCardioTest).

## References

  * [1] J. Binder, C. Krautz, K. Engel, R. Grützmann, F. A. Fellner, P. H. Burger, and M. Scholz.  Leveraging medical imaging for medical education—a cinematic rendering-featured lecture.  Annals of Anatomy-Anatomischer Anzeiger, 222:159–165, 2019.
  * [2] B. J. Bouma and B. J. Mulder.  Changing landscape of congenital heart disease.  Circulation research, 120(6):908–922, 2017.
  * [3] B. Burley and W. D. A. Studios.  Physically-based shading at disney.  In ACM Siggraph, vol. 2012, pp. 1–7. vol. 2012, 2012.
  * [4] N. Cardobi, R. Nocini, G. Molteni, V. Favero, A. Fior, D. Marchioni, S. Montemezzi, and M. D’Onofrio.  Path tracing vs. volume rendering technique in post-surgical assessment of bone flap in oncologic head and neck reconstructive surgery: A preliminary study.  Journal of Imaging, 9(2), 2023.
  * [5] D. Comaniciu, K. Engel, B. Georgescu, and T. Mansi.  Shaping the future through innovations: From medical imaging to precision medicine.  Medical image analysis, 33:19–26, 2016.
  * [6] E. Dappa, K. Higashigaito, J. Fornaro, S. Leschka, S. Wildermuth, and H. Alkadhi.  Cinematic rendering–an alternative to volume rendering for 3D computed tomography imaging.  Insights into imaging, 7(6):849–856, 2016.
  * [7] L. C. Ebert, W. Schweitzer, D. Gascho, T. D. Ruder, P. M. Flach, M. J. Thali, and G. Ampanozi.  Forensic 3D visualization of CT data using cinematic volume rendering: a preliminary study.  American Journal of Roentgenology, 208(2):233–240, 2017.
  * [8] M. Eid, C. N. De Cecco, J. W. Nance Jr, D. Caruso, M. H. Albrecht, A. J. Spandorfer, D. De Santis, A. Varga-Szemes, and U. J. Schoepf.  Cinematic rendering in CT: a novel, lifelike 3D visualization technique.  American Journal of Roentgenology, 209(2):370–379, 2017.
  * [9] M. Elshafei, J. Binder, J. Baecker, M. Brunner, M. Uder, G. F. Weber, R. Grützmann, and C. Krautz.  Comparison of cinematic rendering and computed tomography for speed and comprehension of surgical anatomy.  JAMA surgery, 154(8):738–744, 2019.
  * [10] F. A. Fellner.  Introducing cinematic rendering: a novel technique for post-processing medical imaging data.  Journal of Biomedical Science and Engineering, 9(3):170–175, 2016\.
  * [11] J. T. Kajiya.  The rendering equation.  In Proceedings of the 13th annual conference on Computer graphics and interactive techniques, pp. 143–150, 1986.
  * [12] T. Kroes, F. H. Post, and C. P. Botha.  Exposure render: An interactive photo-realistic volume rendering framework.  PloS one, 7(7):e38586, 2012.
  * [13] B. Preim, A. Baer, D. Cunningham, T. Isenberg, and T. Ropinski.  A survey of perceptually motivated 3D visualization of medical image data.  In Computer Graphics Forum, vol. 35, pp. 501–525. Wiley Online Library, 2016.
  * [14] P. M. Rea, ed.  Biomedical Visualisation.  Springer International Publishing, 2022. doi: 10 . 1007/978-3-030-87779-8
  * [15] F. Röschl, A. Purbojo, A. Rüffer, R. Cesnjevar, S. Dittrich, and M. Glöckler.  Initial experience with cinematic rendering for the visualization of extracardiac anatomy in complex congenital heart defects.  Interactive cardiovascular and thoracic surgery, 28(6):916–921, 2019.
  * [16] T. Steffen, S. Winklhofer, F. Starz, D. Wiedemeier, U. Ahmadli, and B. Stadlinger.  Three-dimensional perception of cinematic rendering versus conventional volume rendering using CT and CBCT data of the facial skeleton.  Annals of Anatomy-Anatomischer Anzeiger, 241:151905, 2022.
  * [17] L. M. Wollschlaeger, J. Boos, P. Jungbluth, J.-P. Grassmann, C. Schleich, D. Latz, P. Kroepil, G. Antoch, J. Windolf, and B. M. Schaarschmidt.  Is CT-based cinematic rendering superior to volume rendering technique in the preoperative evaluation of multifragmentary intraarticular lower extremity fractures?  European Journal of Radiology, 126:108911, 2020.
  * [18] J. Xu, G. Thevenon, T. Chabat, M. McCormick, F. Li, T. Birdsong, K. Martin, Y. Lee, and S. Aylward.  Interactive, in-browser cinematic volume rendering of medical images.  Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, pp. 1–8, 2022.