# WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving
Scene Generation

Jiachen Lu Ze Huang Jiahui Zhang Zeyu Yang Li Zhang111 Corresponding to
<EMAIL_ADDRESS>
School of Data Science, Fudan University
https://github.com/fudan-zvg/WoVoGen

###### Abstract

Generating multi-camera street-view videos is critical for augmenting
autonomous driving datasets, addressing the urgent demand for extensive and
varied data. Due to the limitations in diversity and challenges in handling
lighting conditions, traditional rendering-based methods are increasingly
being supplanted by diffusion-based methods. However, a significant challenge
in diffusion-based methods is ensuring that the generated sensor data preserve
both intra-world consistency and inter-sensor coherence. To address these
challenges, we combine an additional explicit world volume and propose World
Volume-aware Multi-camera Driving Scene Generator (WoVoGen). This system is
specifically designed to leverage 4D world volume as a foundational element
for video generation. Our model operates in two distinct phases: (i)
envisioning the future 4D temporal world volume based on vehicle control
sequences, and (ii) generating multi-camera videos, informed by this
envisioned 4D temporal world volume and sensor interconnectivity. The
incorporation of the 4D world volume empowers WoVoGen not only to generate
high-quality street-view videos in response to vehicle control inputs but also
to facilitate scene editing tasks.

Figure 1:  Our WoVoGen is crafted to generate future world volumes (_i.e_., HD
maps and occupancy) and high-quality multi-camera street-view images, with the
input of past world-volumes. The bottom row shows the weather-based control of
WoVoGen. Specifically, with the predicted world volume at time $t_{4}$ and a
weather description, the multi-camera images transit from rainy to sunny
conditions, while maintaining the street layout. For comprehensive control,
editing, and diverse sample generation, please refer to the following article.

##  1 Introduction

The burgeoning field of vision-based autonomous driving perception [21, 26,
24] underscores the need for high-quality multi-camera street-view datasets
[2]. With the notorious costs of labeling in autonomous driving datasets,
there is a significant demand for generating high-quality multi-camera street-
view videos that accurately mirror real-world 3D data distributions and
maintain consistency across multiple sensors.

Recent driving scene content synthesis techniques can be categorized into two
main groups: rendering-based [42, 45, 10, 43, 3] and diffusion-based [34, 44,
9, 40, 19] methods. Rendering-based methods benefit from an explicit 3D or 4D
world structure, ensuring stringent 3D consistency. Yet, this approach tends
to offer limited diversity and requires considerable effort to produce multi-
camera videos that comply with real-world lighting, weather conditions, etc.
On the other hand, methods based on finetuned stable diffusion models boast
high diversity and can easily generate images that follow real-world
distributions. Despite the noted advantages, diffusion-based approaches [34,
44, 9, 19, 40] that employ bounding boxes or HD maps as control signals offer
a sparse representation of the driving scene. Consequently, the generated
videos often lack critical information about the scene, leading to an
incomplete understanding. This limitation underscores the necessity for a
diffusion-based model that incorporates a dense representation of the world as
a condition, ensuring a more comprehensive and accurate depiction of the
driving environment.

To overcome the limitations of diffusion-based methods, we introduce World
Volume-aware Multi-camera Driving Scene Generator (WoVoGen), a framework
designed to endow diffusion-based generative models with an explicit 4D world
volume. Our approach operates in two stages: initially, we envision a 4D world
volume using a reference scene combined with a future vehicle control
sequence. Subsequently, this volume guides the generation of multi-camera
footage. Concretely, our 4D world volume manifests as a dense voxel volume
spanning four dimensions: time, height, length, and width, corresponding to
the scope of the bird’s-eye view (BEV) domain. This representation
encapsulates the scene’s comprehensive data, comprising object occupancy,
high-definition maps, background details, and road attributes. Each voxel
within this 4D construct is annotated with a class label, providing a rich,
multi-faceted understanding of the environment.

In the preliminary stage, our method is to train an autoencoder model [38]
that encodes a single-frame 3D world volume into a 2D latent representation.
Subsequently, we stack these 2D latents along the temporal axis to form a 2D
temporal latent series. We further refine the UNet by introducing temporal
versions of its residual and cross-attention blocks, which can effectively
process the time-varying information, guided by the vehicle control sequence
as the conditional context. Upon generating the future 2D temporal latent, we
proceed to decode it back into the 4D world voxel volume using the
autoencoder’s decoder. Having generated the future 4D world voxel volume, we
employ a combination of CLIP and 3D sparse CNN to convert it into a 4D world
feature. This feature is then transformed geometrically to sample 3D image
volumes for each camera, corresponding to each time step. Subsequently, these
3D image volumes are condensed into 2D image features, which serve as
conditional inputs for ControlNet [46]. Alongside image features, we employ
textual prompts as scene guidance akin to those used in Stable Diffusion [28]
to govern the overall scene conditions such as weather, lighting, location,
and the scene at large. For more precise object location control within the
scene, textual prompts are utilized with greater specificity. We map the 4D
world volume labels onto each 2D pixel, utilizing the label names as textual
conditions to objectively guide over each pixel’s characteristics. For
achieving inter-sensor consistency, we concatenate surround-view images into a
meta-image and leverage the diffusion model to learn the distribution of real-
world multi-view image sequences. To ensure temporal coherence, we utilize the
same temporal Transformer blocks previously described.

In summary, we make the following contributions: (i) We propose WoVoGen, a
framework that leverages an explicit world volume to guide the diffusion model
generation, ensuring intra-world and inter-sensor consistency. (ii) We
introduce a two-phase strategy: initially envisioning the future 4D world
volume, followed by generating multi-camera videos based on this envisioned
world volume. (iii) Our system not only excels in producing street-view videos
that exhibit consistency within the world and across various cameras, driven
by vehicle control inputs, but also facilitates scene editing tasks.

##  2 Related work

Figure 2: Overall framework of WoVoGen. Top: world model branch. We finetune
the AutoencoderKL and train the 4D diffusion model from scratch to generate
future world volumes based on past world volumes and the actions of the ego
car. Bottom: world volume-aware synthesis branch. Leveraging the generated
future volumes as input, $\mathcal{F}_{w}$ are derived through the world
encoder. Subsequent sampling yields $\mathcal{F}_{img}$, which are then
aggregated. The process is finalized by applying panoptic diffusion to produce
future videos.

##### Diffusion model for visual content generation

Diffusion models, as demonstrated in [12, 33, 5, 28], exhibit the capability
to generate a diverse array of images through a learned denoising process.
Building upon the foundation of a well-trained image generation model, several
works have emerged to augment its controllable generation capabilities, such
as text-guided approaches [6, 27, 30, 29], as well as layout-guided approaches
[46, 15, 47, 20], etc. Further advancements in image generation extend to
video generation, emphasizing temporal consistency in recent works [32, 48,
41, 1]. Additionally, there has been a surge in interest in novel view
generation techniques [23, 31, 22, 25], focused on generating diverse
perspectives from existing images. These advancements necessitate models with
enhanced capabilities to ensure spatial consistency in generated outputs.

We consolidate various functionalities from preceding diffusion models and
introduce WoVoGen, designed specifically for generating driving scene content.
WoVoGen encompasses several innovative features: (i) Text-guided and world
volume-aware controllable image generation, (ii) Spatially consistent multi-
view image generation and novel view image generation, and (iii) Temporally
consistent video generation.

##### Rendering-based driving scene synthesis

The rendering-based approaches [43, 42, 45, 10] initially reconstruct
comprehensive world structures, followed by image or video rendering based on
this reconstructed data. While these methods yield high-quality synthetic
results, they often lack robust control capabilities. Efforts like
NeuralField-LDM [17] have attempted to introduce diffusion models into volume
rendering to address diversity issues and improve control in rendering-based
synthesis. However, they still rely heavily on numerous real-world samples
from various perspectives or additional supervision during model training,
raising concerns about their efficacy.

##### Diffusion-based driving scene synthesis

The diffusion-based approaches [34, 44, 9, 40, 19] leverage pre-trained
diffusion models and incorporate sparse conditions, i.e., HD maps and bounding
boxes, enabling precise control over driving scene content generation. Part of
the works [34, 44, 9] primarily focus on single-frame image generation. As a
unified pipeline, they encode ground truth condition information and inject
conditional features into the diffusion model to generate images corresponding
to the specified condition. Other works [40, 19] have ventured into driving
scene video generation. Building upon trained single-frame generation models,
they finetune the temporal module to ensure the temporal coherence of
generated elements across consecutive frames in the video.

Figure 3:  (a): an action attention block enhances the model by incorporating
essential action information. (b): a guidance attention block integrates the
CLIP feature of a specific object into the latent representation, further
refining the model’s capabilities.

The proposed WoVoGen excels in generating high-quality driving scene images
and videos. Distinguishing itself from other driving scene video generation
works [40, 19], WoVoGen possesses unique traits: (i) It employs a self-
organized dense world volume as a condition, facilitating the natural
generation of geometrically consistent images from multiple perspectives. (ii)
WoVoGen integrates a world model capable of generating future world volumes,
eliminating the necessity for ground truth conditions for each video frame
generation. (iii) With the incorporation of explicit 4D information, i.e.,
temporal world volumes, WoVoGen offers a comprehensive and accurate
representation of the driving environment, resulting in a strong correlation
with critical scene details in the generated content.

##  3 Method

###  3.1 Preliminary

The latent diffusion model (LDM) [28] is a generative model adept at
generating high-resolution images in two stages. Initially, an autoencoder
compresses data into a latent space (perceptual compression), where $z=E(x)$
and $\hat{x}=D(z)$ represent encoding and decoding processes, respectively.
Subsequently, a denoising diffusion probabilistic model (DDPM) [12] models
image distributions in this latent space.

The generative process reverses the diffusion sequence
$\mathbf{z}_{1},\ldots,\mathbf{z}_{T}$ using a learned Gaussian transition,
formalized as:

$q\left(\mathbf{z}_{\tau}|\mathbf{z}_{\tau-1}\right)=\mathcal{N}\left(\mathbf{z}_{\tau};\sqrt{1-\beta_{\tau}}\mathbf{z}_{\tau-1},\beta_{\tau}\mathbf{I}\right),$
(1)
$p_{\theta}\left(\mathbf{z}_{\tau-1}|\mathbf{z}_{\tau}\right)=\mathcal{N}\left(\mathbf{z}_{\tau-1};\mu_{\theta}(\mathbf{z}_{\tau},\tau),\tilde{\beta}_{\tau}\right).$
(2)

Here, $\mu_{\theta}(\mathbf{z}_{\tau},\tau)$ is given by a trainable noise
predictor $\epsilon_{\theta}(\mathbf{z}_{\tau},\tau)$.

DDPMs, viewed as a series of weight-sharing denoising autoencoders, train to
predict the initial noise from a noisy input. The training goal is to maximize
the variational lower bound of the negative log-likelihood:

$\mathbb{E}_{z,\epsilon,\tau}\left[\|\epsilon-\epsilon_{\theta}(\mathbf{z}_{\tau},\tau)\|^{2}_{2}\right].$
(3)

Finally, sampling from the latent distribution and then using the latent
decoder generates novel images from Gaussian noise.

###  3.2 Overall architecture

The proposed architecture, referred to as WoVoGen, comprises two distinct
operational branches: the world model branch and the world volume-aware
generation branch. The world model branch is responsible for generating future
world volumes, incorporating action inputs and several initial frames of the
world volume to inform its predictions. Meanwhile, the world volume-aware
generation branch focuses on the generation of multi-camera video outputs
based on temporal world volumes. An overview of our comprehensive pipeline is
visually represented in Figure 2.

###  3.3 World volume

To effectively harness the potential of rapidly evolving generative
architectures, it is imperative to transform the environmental context into a
standardized format. Our primary focus herein lies on the extraction of high-
level, abstract structural data pertinent to autonomous driving scenarios,
encompassing road layouts, semantic occupancies, and other related elements.

Recognizing the critical role of three-dimensional data in rendering processes
and the benefits derived from fine-grained constraints in view generation, we
propose the encoding of scene information within a three-dimensional world
volume, denoted as $\mathcal{W}\in\mathbb{R}^{Z\times H\times W\times C}$.
Specifically, at any given time instance $t$, we amalgamate various facets of
driving-related information to encapsulate the environmental context around
the ego vehicle within a predefined range. This integration is achieved
through the concatenation process, i.e.,
$\mathcal{W}=\text{concat}(\mathcal{O},\mathcal{M})$, where
$\mathcal{O}\in\mathbb{R}^{Z\times H\times W\times C_{\text{occ}}}$ represents
the three-dimensional semantic occupancy grid, with $C_{\text{occ}}$
indicating the number of semantic classes. Concurrently,
$\mathcal{M}\in\mathbb{R}^{1\times H\times W\times C_{\text{map}}}$ captures
the road map information, constrained to the voxel plane at zero height.

To maintain uniformity across the height dimension, we apply zero-padding to
$\mathcal{M}$ along the $Z$ axis. For a more streamlined representation, both
road elements and semantic classes are encoded into the RGB spectrum, i.e.,
setting $C_{\text{map}}=3$.

###  3.4 World model

Given a clip of multi-camera videos captured during driving, denoted as
$\\{\mathcal{I}_{0},\cdots,\mathcal{I}_{N_{\text{past}}}\\}$, we initially
employ off-the-shelf scene understanding models to infer the environment. This
allows us to obtain the 3D world volume at these moments, represented as
$\left\\{\mathcal{W}_{t}\right\\}_{t=1,\cdots,N_{\text{past}}}$. Subsequently,
the world volume for the time instance $N_{\text{past}}+1$ is inferred,
utilizing the past world volumes and the corresponding driving actions.

##### Latent world volume autoencoder

Given the world volume $\mathcal{W}\in\mathbb{R}^{Z\times H\times W\times C}$,
the encoder is designed to compress $\mathcal{W}$ into its latent
representation,
$z_{w}=E_{\mathcal{W}}(\mathcal{W})\in\mathbb{R}^{\frac{Z}{s}\times\frac{H}{s}\times\frac{W}{s}\times
C_{z}}$. Here, $s$ represents the downsampling factor, and is set equal to $Z$
to achieve a 2D latent representation. The decoder, $D_{\mathcal{W}}$, is then
employed to reconstruct the original world volume from the latent
representation $z_{w}$. The autoencoder is trained using a reconstruction loss
and vq-regularization [7], aimed at minimizing the variance within the latent
space.

##### Latent world diffusion

After the training of the latent autoencoder, we commence the training of the
world model, with both the encoder and the decoder remaining frozen. As
illustrated in Figure 2, our world model is a diffusion model that captures
the conditional distribution of the latent world volume in the future,
conditioned on the past world volumes and the driving actions of the ego
vehicle. The noise predictor $\epsilon_{\phi}$ in this stage is implemented as
a time-conditioned UNet [12]. The world volumes of preceding frames are first
encoded by the same latent encoder to dimensions commensurate with the noised
latent $z_{i}$, and then channel-wise concatenate with $z_{i}$ to serve as the
input for the noise predictor.

For the driving actions, we initially tokenize the velocity $v_{i}$ and the
steering angle $a_{i}$ of the ego vehicle through Fourier embedding [39],
resulting in a sequence of action tokens
$\mathcal{A}=\left\\{v_{1},a_{1},\cdots,v_{N_{\text{past}}},a_{N_{\text{past}}}\right\\}$.
These action tokens are then refined by a shallow Transformer. Finally, the
updated action tokens are injected into the noise predictor via cross-
attention layers.

##### Jointly modeling of the consecutive frames

To enhance the temporal consistency of predicted future world volumes and
achieve more realistic scene dynamics, we transition to generating consecutive
world volumes by considering a joint distribution of a world volume sequence,
$p(\mathcal{W}_{N_{\text{past}}+1},\cdots,\mathcal{W}_{N_{\text{past}}+N_{\text{future}}}|\mathcal{W}_{1},\cdots,\mathcal{W}_{N_{\text{past}}};\mathcal{A})$.

As shown in Figure 3, we build upon the basic spatial Transformer block of
Stable Diffusion [28], which comprises spatial convolution, spatial self-
attention, and spatial cross-attention. Our temporal variant augments this
framework by integrating temporal attention with residual connections into the
spatial Transformers.

Specifically, let us denote $z_{w}\in\mathbb{R}^{B\times
N_{\text{future}}\times C\times H\times W}$ as the sequence of latent world
volumes, adding the batch dimension for clarity. The computation within the
modified spatial-temporal Transformer is formalized as follows:

$\displaystyle\mathbf{z}_{w}$
$\displaystyle=\text{rearrange}\left(\mathbf{z}_{w},(b\ n)\ h\ w\
c\rightarrow(b\ n)\ (h\ w)\ c\right),$ $\displaystyle\mathbf{z}_{w}$
$\displaystyle=\text{MHSA}\left(\text{Norm}\left(\mathbf{z}_{w}\right)\right)+\mathbf{z}_{w},\quad\text{(Spatial)}$
$\displaystyle\mathbf{z}_{w}$
$\displaystyle=\text{rearrange}\left(\mathbf{z}_{w},(b\ n)\ (h\ w)\
c\rightarrow(b\ h\ w)\ n\ c\right),$ $\displaystyle\mathbf{z}_{w}$
$\displaystyle=\text{MHSA}\left(\text{Norm}\left(\mathbf{z}_{w}\right)\right)+\mathbf{z}_{w},\quad\text{(Temporal)}$
$\displaystyle\mathbf{z}_{w}$
$\displaystyle=\text{rearrange}\left(\mathbf{z}_{w},(b\ h\ w)\ n\
c\rightarrow(b\ n)\ (h\ w)\ c\right),$ $\displaystyle\mathbf{z}_{w}$
$\displaystyle=\text{MHCA}\left(\text{Norm}\left(\mathbf{z}_{w},\mathcal{A}\right)\right)+\mathbf{z}_{w},\quad\text{(Action)}$
$\displaystyle\mathbf{z}_{w}$
$\displaystyle=\text{FFN}\left(\text{Norm}\left(\mathbf{z}_{w}\right)\right)+\mathbf{z}_{w},$

where Norm represents the group normalization, MHSA stands for multi-head
self-attention, MHCA denotes multi-head cross-attention, and FFN refers to the
feed-forward network. Both MHSA and MHCA perform attention operations on the
second dimension $l$ of the $(b\ l\ c)$ configuration.

###  3.5 World volume-aware 2D feature

With the temporally continuous world volumes
$\mathcal{W}=\left\\{\mathcal{W}_{t}\right\\}_{t=1,2,\cdots}$ generated by our
diffusion-based world model, we further decode them into relational camera
images for autonomous driving video generating.

##### World volume encoding

The world volumes inherently possess semantic information, initially
represented in the form of simplistic labels. To enhance their
informativeness, we employ a featurization process utilizing CLIP [26]:

$\mathcal{F}_{w}=\text{SPConv}\left(\text{PCA}\left(\text{CLIP}\left(\mathcal{W}\right)\right)\right),$
(4)

where CLIP encodes the class name into the class feature. PCA is used to
decrease the dimension of the CLIP feature, thereby reducing the computational
cost. SPConv (Sparse Convolution) [4] then processes the reduced-dimensional
world volume feature.

Figure 4: WoVoGen excels in producing future world volumes (top two rows) with
temporal consistency. Subsequently, it utilizes the world volume-aware 2D
image features derived from the world model’s outputs to synthesize a driving
video (bottom two rows) with both multi-camera consistency and temporal
consistency.

##### Camera volume sampling

To incorporate the world volume into image generation, we sample from it using
dense rays emitted from the camera. We construct a 3D grid for each camera’s
frustum, denoted as $p_{c}$, with dimensions $D_{c}\times H_{c}\times W_{c}$.
This grid is based on the camera’s intrinsic and extrinsic properties. The
process is formalized as follows:

$\mathcal{F}_{cam}=\text{interpolate}\left(p_{c},\mathcal{F}_{w}\right),$ (5)

where $\mathcal{F}_{c}\in\mathbb{R}^{B\times N_{\text{future}}\times C\times
D_{c}\times H_{c}\times W_{c}}$ represents the camera frustum. We then apply a
squeeze-and-excitation operation [13] on the depth channel and sum along the
depth dimension to obtain the world volume-aware 2D image feature:

$\mathcal{F}_{img}=\sum_{i=1}^{D_{c}}\text{SE}(\mathcal{F}_{cam})_{[:,\ :,\
:,\ i,\ :,\ :]}.$ (6)

###  3.6 World volume-aware diffusion generation

Based on the ControlNet framework [46], we implement a controller that injects
the aforementioned world volume-aware 2D image feature
$\mathcal{F}_{\text{img}}$ into the pretrained latent diffusion model. The
Transformer architecture of the UNet is illustrated in Figure 3. However, we
initially omit the Temporal Attention Block when training the single-frame
Generation model.

##### Panoptic diffusion

We aggregate the world volume-aware 2D image feature from different view into
a single panoptic feature $\mathcal{F}_{pano}$ input to the diffusion model:

$\mathcal{F}_{pano}=\begin{bmatrix}\mathcal{F}_{img}^{\text{front
left}}&\mathcal{F}_{img}^{\text{front}}&\mathcal{F}_{img}^{\text{front
right}}\\\ \mathcal{F}_{img}^{\text{back
right}}&\mathcal{F}_{img}^{\text{back}}&\mathcal{F}_{img}^{\text{back
left}}\end{bmatrix}.$ (7)

Naturally, the decoding target is transformed into the corresponding panoptic
image. This operation shifts the focus from ensuring consistency between
latent codes for different views to maintaining inherent consistency within a
single latent code $z_{pano}\in\mathbb{R}^{C\times 2H_{c}\times 3W_{c}}$,
resulting in a unified and coherent appearance in multi-camera image
generation.

##### Scene guidance

Except for the conditional feature, we also introduce text prompt-based scene
guidance into the latent representations. This involves extracting the CLIP
feature from a text-based description of an image and mapping this text
feature to the intermediate layers of the latent diffusion model, following a
similar approach as LDM [28].

##### Object guidance

When it comes to objects that require specific placement within the generated
image, we emphasize their pixel-level locations in the latent space using a
cross-attention calculation.

Specifically, we employ the voxel-based projection of the initial world volume
onto each camera plane, resulting in the creation of preliminary masks
$m_{\text{class}}$ distinguished by the world volumes’ semantic categories.
Then, cross-attention is calculated by:

$\mathbf{z}_{pano}=\text{MHCA}(\mathbf{z}_{pano}(m_{\text{class}}=1),\text{CLIP}(\text{class}))+\mathbf{z}_{pano},$

where $\text{CLIP}(\text{class})$ represents the category-specific CLIP
feature, with semantic categories class being defined as bus, car, pedestrian,
truck, construction, and vegetation.

###  3.7 Video generation

Before we generate video, we first train the single-frame multi-camera
generation model, and then we add temporal attention block and only train the
temporal attention block as shown in the right side of Figure 3 and freeze the
other blocks.

Figure 5: Examples of conditional generation on nuScenes [2] validation
dataset. WoVoGen empowers diverse and controllable scene generation. Altering
the random seed allows for the generation of various scenarios. Additionally,
adjustment to weather (such as sunny, rainy, night, etc.) and location
(Singapore, Boston, etc.) within the prompt enables the modification of
weather conditions and city styles within the generated scene.

##### Temporal consistency

The temporal attention block is added to ensure the generation of multi-frame
images consistent, which calculates the attention by:

$\displaystyle\mathbf{z}_{pano}$
$\displaystyle=\text{rearrange}\left(\mathbf{z}_{pano},(b\ n)\ (h\ w)\
c\rightarrow(b\ h\ w)\ n\ c\right),$ $\displaystyle\mathbf{z}_{pano}$
$\displaystyle=\text{MHSA}\left(\text{Norm}\left(\mathbf{z}_{pano}\right)\right)+\mathbf{z}_{pano},\quad\text{(Temporal)}$

###  3.8 Training target

With all the conditions serving as inputs, the training objective can be
formulated as:

$\min_{\theta}\mathcal{L}=\mathbb{E}_{\mathbf{z},\epsilon\sim\mathcal{N}(0,1),\tau}\left[\|\epsilon-\epsilon_{\theta}(\mathbb{\mathbf{z}}_{\tau},\tau,c)\|^{2}_{2}\right],$
(8)

where $\theta$ is the learnable parameters in our network, $c$ denotes the
conditions.

Figure 6: Examples of controlled editing on nuScenes [2] validation dataset.
Top: the ability to selectively add or remove specific objects (such as trees,
buildings, cars, etc., highlighted by red circles in the figure) within the
world volume empowers the precise and coherent generation of diverse driving
scenarios across multiple cameras. Bottom: due to WoVoGen’s advanced 3D
understanding capabilities, the rotation of perspectives across multiple
cameras can be achieved by modifying the camera’s extrinsic parameters. This
functionality enables the generation of driving scenes from any desired angle.

##  4 Experiments

###  4.1 Experimental setup

##### Dataset

We conducted experiments using the nuScenes [2] dataset which comprises 700
training videos and 150 validation videos. Our study utilized images generated
by the six camera views trained on the nuScenes dataset.

For ground-truth occupancy data, we utilized CVPR 2023 3D Occupancy Prediction
Challenge [36, 35], aligned with semantic labels consistent with the nuScenes-
lidarseg dataset [8]. The ego car-centric local map is derived from the global
HD map using the vehicle’s ego pose and precisely aligned with the occupancy
data.

##### Data preprocessing

Images are resized to a resolution of 256 $\times$ 448, and six consecutive
frames are selected for video training. Text prompts are constructed using a
structured template: “Drive in {weather description} in {location}. The
driving scene is in {environment description}, captured by multi-view camera.”
The descriptive details enclosed in brackets are filled using scene
descriptions from nuScenes.

##### Training

The training process utilized eight NVIDIA A6000 GPUs, encompassing 30,000
iterations for the world volume auto-encoder, 50,000 iterations for volume
diffusion, 50,000 iterations for single-frame generation training, and 3,500
iterations for refining video generation.

##### Inferring

We begin with three initial world volumes organized from ground-truth data and
proceed to generate the subsequent three world volumes by incorporating
actions as input. This iterative process continues, producing a quasi-long
world volume video sequence. Ultimately, the world volume video sequence is
decoded to form a multi-view camera video.

##### Evaluation

We conducted a comparative analysis of our method’s image quality against
other approximate works in the following text. To quantitatively assess our
work, we employed established metrics such as Fréchet Inception Distance [11]
(FID) and Fréchet Video Distance [37] (FVD) for evaluating the quality of the
generated video. Additionally, we provided qualitative demonstrations that
underscore the advantages of our proposed method.

###  4.2 Results

####  4.2.1 World volume generation

The distinctive feature of WoVoGen lies in its ability to predict high-level
driving environments in the first stage, including occupancy and HD maps, as
vividly illustrated in the left two columns of Figure 1 and the top two rows
of Figure 4. Given vehicle actions from the dataset, WoVoGen successfully
foresees high-quality future driving environments. Notably, our model retains
its effectiveness in predicting single frames while ensuring a high degree of
consistency across the generated future frames. This dual capability of
WoVoGen — excelling in both short-term prediction and long-term scene
evolution — highlights its potential as a powerful tool for advanced driving
environment simulation.

####  4.2.2 Multi-camera single-frame image generation

Figure 5 shows the qualitative results of the multi-camera single-frame image
generation under diverse conditions. We can observe that the generated images
have highly consistency, which clearly shows that our model indeed learns to
understand the geometric and photometric relationships between different views
of the same object or scene. Furthermore, from the first row of Figure 5, we
can see that the generated images accurately correspond to the occupancy
guide. Beyond that, we can also control the style of the generated images
through natural language. In the last two rows of the Figure 5, our WoVoGen
demonstrates a profound comprehension of the specified textual conditions,
skillfully creating highly realistic simulations of rainy day scenarios.
Remarkably, this model also exhibits the capability to interpret and respond
to location-specific instructions through natural language processing. When
prompted with “Drive in Singapore Onenorth,” the model skillfully generates a
scene embodying the garden city essence typical of that region. This notable
feature not only illustrates the model’s advanced understanding of
geographical nuances but also highlights its potential to generate a wide
array of diverse and contextually accurate data.

The quantitative results in Table 1 underscore that WoVoGen showcases markedly
lower FID (27.6) compared to DriveGan [16] and DriveDreamer [40], while these
methods are limited to single-view camera image generation. This result
solidifies the improved capacity of our approach in generating more realistic
autonomous driving images.

####  4.2.3 Multi-camera single-frame image editing

Given the world volume-aware character of WoVoGen, we can realize editing
based on the edited world volume.

##### Rearranging objects:

We can add, delete, or transform the objects within the occupancy. As shown in
Figure 6, we can add trees, vehicles, or rearrange the location of objects.

##### Camera extrinsic editing:

More interestingly, we can rotate the camera stereo when projecting the world
volume to the camera volume and generate the corresponding scene under a new
camera setting.

Method | Multi-view | Multi-frame | FID$\downarrow$ | FVD$\downarrow$
---|---|---|---|---
DriveGan [16] | $\surd$ |  | 73.4 | 502.3
DriveDreamer [40] |  | $\surd$ | 52.6 | 452.0
Ours(single-frame) | $\surd$ |  | 27.6 | -
Ours(video) | $\surd$ | $\surd$ | - | 417.7
Table 1: Quantitative comparison of image/video generation quality on Nuscenes
validation set. WoVoGen achieve both multi-view and multi-frame generation,
demonstrating the lowest FID and FVD scores among all methods.

####  4.2.4 Multi-camera video generation

Figure 4 showcases the qualitative results of the multi-camera video
generation. Thanks to the high-quality world volume generation and finetuning
for temporal consistency, the generated video demonstrates a remarkable level
of temporal coherence. The appearance of objects and background within the
video maintains consistency throughout the sequence. Furthermore, the
quantitative results of video quality, indicated by the FVD (417.7) in Table
1, underscores the superiority of our method compared to the other works.

##  5 Conclusion

In this paper, we propose WoVoGen, which marks a significant advancement in
generating multi-camera driving scene videos. Leveraging a novel 4D world
volume, it adeptly integrates time with spatial data, addressing the
complexity of creating content from multi-sensor data while ensuring
consistency. This two-phase system not only produces high-quality videos based
on vehicle controls but also enables complex scene editing, showcasing its
potential as a comprehensive tool for advancing autonomous driving
technologies.

## References

  * Blattmann et al. [2023] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al.  Stable video diffusion: Scaling latent video diffusion models to large datasets.  _arXiv preprint_ , 2023.
  * Caesar et al. [2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In _CVPR_ , 2020.
  * Chen et al. [2023] Yurui Chen, Chun Gu, Junzhe Jiang, Xiatian Zhu, and Li Zhang.  Periodic vibration gaussian: Dynamic urban scene reconstruction and real-time rendering.  _arXiv preprint_ , 2023.
  * Contributors [2022] Spconv Contributors.  Spconv: Spatially sparse convolution library.  https://github.com/traveller59/spconv, 2022.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  In _NeurIPS_ , 2021.
  * Ding et al. [2021] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al.  Cogview: Mastering text-to-image generation via transformers.  In _NeurIPS_ , 2021.
  * Esser et al. [2021] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In _CVPR_ , 2021.
  * Fong et al. [2022] Whye Kit Fong, Rohit Mohan, Juana Valeria Hurtado, Lubing Zhou, Holger Caesar, Oscar Beijbom, and Abhinav Valada.  Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking.  In _IEEE Robotics and Automation Letters_ , 2022.
  * Gao et al. [2023] Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu.  Magicdrive: Street view generation with diverse 3d geometry control.  _arXiv preprint_ , 2023.
  * Guo et al. [2023] Jianfei Guo, Nianchen Deng, Xinyang Li, Yeqi Bai, Botian Shi, Chiyu Wang, Chenjing Ding, Dongliang Wang, and Yikang Li.  Streetsurf: Extending multi-view implicit surface reconstruction to street views.  _arXiv preprint_ , 2023.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _NeurIPS_ , 2017.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _NeurIPS_ , 2020.
  * Hu et al. [2018] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In _CVPR_ , 2018.
  * Huang et al. [2021] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du.  Bevdet: High-performance multi-camera 3d object detection in bird-eye-view.  _arXiv preprint_ , 2021.
  * Huang et al. [2023] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou.  Composer: Creative and controllable image synthesis with composable conditions.  _arXiv preprint_ , 2023.
  * Kim et al. [2021] Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler.  Drivegan: Towards a controllable high-quality neural simulation.  In _CVPR_ , 2021.
  * Kim et al. [2023] Seung Wook Kim, Bradley Brown, Kangxue Yin, Karsten Kreis, Katja Schwarz, Daiqing Li, Robin Rombach, Antonio Torralba, and Sanja Fidler.  Neuralfield-ldm: Scene generation with hierarchical latent diffusion models.  In _CVPR_ , 2023.
  * Kynkäänniemi et al. [2022] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen.  The role of imagenet classes in fr$\backslash$’echet inception distance.  _arXiv preprint_ , 2022.
  * Li et al. [2023a] Xiaofan Li, Yifu Zhang, and Xiaoqing Ye.  Drivingdiffusion: Layout-guided multi-view driving scene video generation with latent diffusion model.  _arXiv preprint_ , 2023a.
  * Li et al. [2023b] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee.  Gligen: Open-set grounded text-to-image generation.  In _CVPR_ , 2023b.
  * Li et al. [2022] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  In _ECCV_ , 2022.
  * Liu et al. [2023a] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In _ICCV_ , 2023a.
  * Liu et al. [2023b] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang.  Syncdreamer: Generating multiview-consistent images from a single-view image.  _arXiv preprint_ , 2023b.
  * Philion and Fidler [2020] Jonah Philion and Sanja Fidler.  Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d.  In _ECCV_ , 2020.
  * Qian et al. [2023] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al.  Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors.  _arXiv preprint_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint_ , 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _CVPR_ , 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  In _NeurIPS_ , 2022.
  * Shi et al. [2023] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang.  Mvdream: Multi-view diffusion for 3d generation.  _arXiv preprint_ , 2023.
  * Singer et al. [2022] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  _arXiv preprint_ , 2022.
  * Song et al. [2023] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _ICLR_ , 2023.
  * Swerdlow et al. [2023] Alexander Swerdlow, Runsheng Xu, and Bolei Zhou.  Street-view image generation from a bird’s-eye view layout.  _arXiv preprint_ , 2023.
  * Tian et al. [2023] Xiaoyu Tian, Tao Jiang, Longfei Yun, Yue Wang, Yilun Wang, and Hang Zhao.  Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving.  _arXiv preprint_ , 2023.
  * Tong et al. [2023] Wenwen Tong, Chonghao Sima, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, et al.  Scene as occupancy.  In _ICCV_ , 2023.
  * Unterthiner et al. [2018] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly.  Towards accurate generative models of video: A new metric & challenges.  _arXiv preprint_ , 2018.
  * Van Den Oord et al. [2017] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  _NeurIPS_ , 2017.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NeurIPS_ , 2017.
  * Wang et al. [2023] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu.  Drivedreamer: Towards real-world-driven world models for autonomous driving.  _arXiv preprint_ , 2023.
  * Wu et al. [2023a] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In _ICCV_ , 2023a.
  * Wu et al. [2023b] Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao Chen, Runyi Yang, et al.  Mars: An instance-aware, modular and realistic simulator for autonomous driving.  _arXiv preprint_ , 2023b.
  * Xie et al. [2023] Ziyang Xie, Junge Zhang, Wenye Li, Feihu Zhang, and Li Zhang.  S-nerf: Neural radiance fields for street views.  In _ICLR_ , 2023.
  * Yang et al. [2023a] Kairui Yang, Enhui Ma, Jibin Peng, Qing Guo, Di Lin, and Kaicheng Yu.  Bevcontrol: Accurately controlling street-view elements with multi-perspective consistency via bev sketch layout.  _arXiv preprint_ , 2023a.
  * Yang et al. [2023b] Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun.  Unisim: A neural closed-loop sensor simulator.  In _CVPR_ , 2023b.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _ICCV_ , 2023.
  * Zheng et al. [2023] Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li.  Layoutdiffusion: Controllable diffusion model for layout-to-image generation.  In _CVPR_ , 2023.
  * Zhou et al. [2022] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng.  Magicvideo: Efficient video generation with latent diffusion models.  _arXiv preprint_ , 2022.

##  Appendix A Masks for object guidance

We have devised an efficient approach to compute masks for object guidance.

Specifically, for the original occupancy voxel grids belongs to a certain
class $P_{class}\in\mathbb{R}^{N\times 3}$, our initial step involves
projecting them onto the camera imaging plane using the camera’s intrinsic
properties $K$ and extrinsic properties $T$, performed as follows:

$P_{proj}=K(RP_{class}+t),$ (9)

where $R=T^{-1}[:3,:3]$ and $t=T^{-1}[:3,3]$ represent the rotation matrix and
translation vector of the inverse of $T$, respectively. Following this, we
initialize an all-zero image $m_{class}$, matching the dimensions of the
camera image. For each normalized projected point $P_{proj}^{i}$, we
efficiently simulate voxel projections by marking a square-shaped region in
$m_{class}$ as occupied:

$m_{class}[P_{proj}^{i_{x}}-\delta:P_{proj}^{i_{x}}+\delta,P_{proj}^{i_{y}}-\delta:P_{proj}^{i_{y}}+\delta]=1,$
(10)

where $P_{proj}^{i_{x}}$ and $P_{proj}^{i_{y}}$ represent the normalized x and
y components of $P_{proj}^{i}$, while $\delta$ is inversely proportional to
the depth of the projection points $P_{proj}^{i_{z}}$:

$\delta=d/P_{proj}^{i_{z}},$ (11)

where $d$ is a hyperparameter set to 375. Several generated mask samples are
illustrated in Figure 7.

##  Appendix B Additional results

###  B.1 Downstream tasks

To enhance performance through more comprehensive data training, we integrate
our generation results with the nuScenes [2] training set as a supplementary
data source. We test our results on 3D object detection, with BEVDet [14] as
baseline. We evaluate our outcomes in 3D object detection, using BEVDet as our
baseline model. Our approach involves generating a complete training dataset
based on the ground truth. Subsequently, we train the 3D object detection
model using BEVDet on both the original nuScenes dataset and our newly
generated nuScenes dataset. Since our world volume class encompasses only
cars, trucks, and buses, our testing focuses exclusively on vehicles within
these categories.

Method | NDS${}_{\text{v}}\uparrow$ | mAP${}_{\text{v}}\uparrow$ | mATE${}_{\text{v}}\downarrow$ | mASE${}_{\text{v}}\downarrow$ | mAOE${}_{\text{v}}\downarrow$ | mAVE${}_{\text{v}}\downarrow$ | mAAE${}_{\text{v}}\downarrow$
---|---|---|---|---|---|---|---
original | 17.45 | 34.9 | 69.2 | 20.4 | 17.9 | 124.6 | 28.6
\+ generated | 18.10 | 36.2 | 68.6 | 20.1 | 15.7 | 123.4 | 28.1
Table 2: The enhancement brought by our generated data for 3D object
detection. The evaluation is on the vehicle classes of cars, trucks, and
buses.

As indicated in Table 2, we observed a significant improvement in the mean
Average Precision (mAP). Upon analyzing the error contributions, it’s evident
that the most substantial improvement arises from a reduction in orientation
error, which decreased from 17.9 to 15.7. This demonstrates that our generated
data indeed enhances the training of downstream tasks, such as 3D object
detection.

###  B.2 Ablation studies

##### Effectiveness of object guidance

We implement object guidance to emphasize the location of the generated
target. The qualitative comparison in Figure 8 illustrates the ablation study,
while the quantitative comparison, using the metric that more accurately
depict the alignment among image instances, i.e., CLIP FID [18], is presented
in Table 3.

Clearly, the integration of object guidance results in a more precise
alignment of the object within the generated image with the target image. This
refinement corresponds to a notable reduction of 7.5 in the metric,
$FID_{clip}$.

Method |  $FID_{clip}\downarrow$
---|---
w/o object guidance |  20.3
w/ object guidance |  12.8
Table 3: Quantitative comparison of image generation quality on the nuScenes
validation set with and without object guidance.

##### Effectiveness of temporal finetuning

To verify the effectiveness of the video consistency module, we illustrate the
simulation outcomes of the front and back cameras in Figure 9 after removing
the temporal finetuning.

Obviously, the finetuned model notably maintains consistent object and
background appearances across frames. In contrast, the single-frame model
struggles to achieve this level of consistency.

###  B.3 Additional single-frame image generation results

We provide additional single-frame image generation samples showcasing more
diverse editing and controlling capabilities in Figure 10.

Utilizing the Lego-style edit-friendly features of the world volume offers
boundless possibilities for editing and controlling, resulting in numerous
potential generation outcomes, some of which may even be rare in the real
world.

Figure 7: Object masks calculated by voxel projection. Origin images and per-
class masks are organized as: Top: front left, front, front right; Bottom:
back right, back, back left. Figure 8: Qualitative ablation study on object
guidance. The incorporation of object guidance significantly improves the
alignment of the generated image’s object with the target image. Figure 9:
Qualitative ablation study on temporal finetuning. Consistency in object and
background preservation across frames is evident when conduct temporal
finetuning. Figure 10: Additional controlling and editing samples. Figure 11:
Additional videos generated from the generated world volumes Figure 12:
Additional videos generated from the generated world volumes Figure 13:
Additional videos generated from the generated world volumes Figure 14:
Additional videos generated from the generated world volumes

###  B.4 Additional video generation results

More generated videos along with the corresponding generated world volume are
shown in Figure 11, Figure 12, Figure 13 and Figure 14.

The world model accurately predicts future world evolution and generates a
comprehensive world volume. Leveraging world volume-aware diffusion, it
produces high-quality videos that maintain cross-frame consistency and
coherence across multiple cameras.


# Two-step method for assessing dissimilarity of random sets

1111 University of Split, Croatia; 2 Czech Technical University in Prague,
Czech Republic (‡ email: heliskat@fel.cvut.cz); 3 Charles University, Czech
Republic; 4 University of Chemistry and Technology, Czech Republic Vesna
Gotovac Đogaš1, Kateřina Helisová2,‡, Bogdan Radović2, Jakub Staněk3,
Markéta Zikmundová4, Kateřina Brejchová2

###### Abstract

The paper concerns a new statistical method for assessing dissimilarity of two
random sets based on one realisation of each of them. The method focuses on
shapes of the components of the random sets, namely on the curvature of their
boundaries together with the ratios of their perimeters and areas. Theoretical
background is introduced and then, the method is described, justified by a
simulation study and applied to real data of two different types of tissue -
mammary cancer and mastopathy.

##  1 Introduction

In the last years, modelling and statistical analyses of random sets have
become very popular. It has been studied both from theoretical point of view
[10], [11], [17] as well as from the practical side, because they have many
applications in biology [13], medicine [8], material sciences [16] and other
branches. They can describe and explain many events, for example behaviour of
cells in organisms, particles in materials, presence of different plants etc.
Therefore mathematical methods dealing with random sets must be developed and
improved.

Usually, when we are given a realisation of a random set, we try to find,
based on the realisation, a model in order to make further statistical
analyses. However, there are situations when the knowledge about the concrete
model is not necessary, because the aim is to decide whether two realisations
are similar in some sense, i.e. whether they may come from the same process,
e.g. we need only to distinguish between two types of cells in tissue from
microscopic pictures, recognise different tendency of growth of some plants,
detect defects in materials etc.

In the presented paper, we focus on planar random sets, nevertheless, the
method described here can be easily extended to more dimensional spaces.
Although there exist classical tools for comparing random sets like covariance
function or contact distribution function [2], functions on morphological
operations, namely dilation, erosion, opening and closing [17], etc., there
are situations when these characteristics are not sufficient to distinguish
between two realisations. E.g. we can obtain the same estimate of contact
distribution function for different shapes or, on the other hand, two
realisations consisting of components of the same shapes but different mutual
distances have very different estimate of the covariance function etc., so
they cannot be used when the main objects of interest are the shapes of the
components, as in this paper. Another disadvantage of the mentioned
characteristics is that for one realisation, we obtain only one function.
Then, it is difficult to formulate the task of comparing two random sets when
we have only one realisation of each of them. In this case, it would be
helpful to have data consisting of more functions for each realisation.

New methods taking this into account have been developed in the last five
years, see [3], [4], [5] and [6]. In [3], morphological skeletons [17] of
compared realisations are constructed and then, similarity of two realisations
is defined through a function describing mass growth around selected points of
the skeletons. This method shows the highest power in simulation study,
compared to the methods in the remaining three papers. However, it is closely
tied to the placement of components in realisations, which is not always
desirable. Further approach can be found in [4], where the author also focuses
on similarity of shapes and positions of components in realisations, but the
positions can be omitted in special cases. The considered components are
either the connected components or some more specific set components that are
obtained by further decomposing of the observed set (e.g. cells in a tissue
which are connected but they can be determined in their binary image). The
positions of the components are described by the so-called neighbourhood
tessellation of the observation window constructed from the components and by
using the Hausdorff metric [2]. The samples of pairs of the components and
neighbouring tessellation cells are compared using the test based on the
$\mathcal{N}$-distance [9], where a suitable kernel involving the symmetric
difference of the sets is constructed. For omitting dependence on locations,
the components can be considered without their neighbourhoods. A disadvantage
of the method is that the test of similarity in this sense is weak, especially
when the neighbourhoods are omitted, so in order to achieve better accuracy,
the positions of the components must be taken into account again. An approach
independent of the positions of the components is introduced in [6] and
improved in [5]. The authors of the papers distinguish between two
realisations via a heuristic approach based on approximation of the components
by unions of convex compact sets using Voronoi tessellation [2] with respect
to a special hard-core point process on the realisations and consequent
comparison of the support functions of the convex compact cells of the
tessellation using the envelope test from [15] and the test based on
$\mathcal{N}$-distance [9] (sometimes also called the kernel test, see [7]),
respectively. The authors of [5] and [6] declare that the method focuses on
the structure of the components like clustering or repulsion tendencies,
generating rounded or angular objects, long and thin or short and thick
formations, etc., because it recognises how much mass is concentrated on the
boundaries of the components and what is the approximate shape of the
boundary. The results of simulation study in [5] and [6] are very
satisfactory, however, there are some weak parts in the heuristic procedure
coming from the fact that the approximation of the components is not unique,
but random, and there are quite large differences between the shapes of the
original realisations and their approximations. Moreover, some free parameters
must be chosen, which can affect the results.

In the present paper, we focus on testing similarity of two realisations of
random sets, while the similarity is given by similar shapes of their
components, analogously to [5] and [6]. However, instead of the heuristic
approximation of realisations, we use description by uniquely defined
characteristics of components in the realisations, namely the ratio of the
perimeter and the area of each component, and the curvature [1] of the
boundary of each component.

The paper is organised as follows. Section 2 summarises definitions and the
already existing theoretical results concerning curvatures of planar curves
and statistical testing via the $\mathcal{N}$-distance theory. In Section 3,
we define similarity of two random sets and describe the procedure of
assessing similarity from two realisations. This is the main result of the
paper. Section 4 justifies the procedure from Section 3 by a simulation study
and compares the results to the results obtained by the previous methods.

##  2 Theoretical background

###  2.1 Curvature of a planar curve

Consider a smooth 2D curve $c$ parameterised by a parameter
$\varphi\in[0,\phi]\subset\mathbb{R}$, i.e.
$c(\varphi)=(x(\varphi),y(\varphi))$. Then the curvature $\kappa$ of $c$ is
defined as

$\kappa(c(\varphi))=\frac{x^{\prime}(\varphi)y^{\prime\prime}(\varphi)-x^{\prime\prime}(\varphi)y^{\prime}(\varphi)}{(x^{\prime
2}(\varphi)+y^{\prime 2}(\varphi))^{3/2}},$

see e.g. [1]. It means that $\kappa(c(\varphi))=\pm 1/R(\varphi)$ , where
$R(\varphi)$ is the radius of the osculating circle touching the curve in the
point $[x(\varphi),y(\varphi)]$ and the choice between “$+$” and “$-$” is
determined by the local convexity convention.

Let us assume that the curve $c$ is continuous, closed (i.e. $c(0)=c(\phi)$)
and it does not intersect itself (i.e.
$c(\varphi_{1})=c(\varphi_{2})\Rightarrow\varphi_{1}=\varphi_{2}$). Consider a
(connected) planar set $X$ whose boundary is given by the curve $c$. It can be
shown [1] that for the curvature $\kappa(z)$ evaluated in a given point $z\in
c$ and for a disc $b(z,r)$ with the center in $z$ and a radius $r$ small
enough, it holds that

$\kappa(z)\approx\frac{3A^{*}_{b(z,r)}}{r^{3}}-\frac{3\pi}{2r}=\frac{3\pi}{r}\left(\frac{A^{*}_{b(z,r)}}{A_{b(z,r)}}-\frac{1}{2}\right),$
(1)

where $A_{b(z,r)}$ is the area of the disc $b(z,r)$ and $A^{*}_{b(z,r)}$ is
the area of $b(z,r)\cap X$.

This is used below in Section 3 when estimating the curvature of boundary of
binary image of the set $X$. The center of discretised disc $b$ with the
radius $r$ pixels is placed to the boundary pixel in which we want to evaluate
the curvature, and approximate the ratio $A^{*}_{b(z,r)}/A_{b(z,r)}$ by the
number of pixels of the disc $b$ inside the set $X$ divided by the number of
all pixels forming the disc $b$.

###  2.2 Testing equality in distribution based on $\mathcal{N}$-distance of
probability measures

In this paper, testing equality in distribution of random variables as well as
random functions comes from the theory of $\mathcal{N}$-distances (see [9])
that is briefly recalled in the following paragraphs.

Let $\mathcal{X}$ be a nonempty set. Consider a negative definite kernel
$\mathcal{L}:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{C}$ (see [9, pp.
16–19]).

###### Definition 2.1

The negative definite kernel $\mathcal{L}$ is called _strongly negative
definite kernel_ if for an arbitrary probability measure $\mu$ and an
arbitrary $f:\mathcal{X}\to\mathbb{R}$ such that
$\int_{\mathcal{X}}f(x)d\mu(x)=0$ holds and
$\int_{\mathcal{X}}\int_{\mathcal{X}}\mathcal{L}(x,y)f(x)f(y)d\mu(x)d\mu(y)$
exists and is finite, the relation

$\int_{\mathcal{X}}\int_{\mathcal{X}}\mathcal{L}(x,y)f(x)f(y)d\mu(x)d\mu(y)=0$

implies that $f(x)=0$ $\mu$-a.e.

For a map $\mathcal{L}:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{C}$,
denote $B_{\mathcal{L}}$ the set of all measures $\mu$ such that
$\int_{\mathcal{X}}\int_{\mathcal{X}}\mathcal{L}(x,y)d\mu(x)d\mu(y)$ exists.

######  Theorem 2.1 (Klebanov, 2006)

Let $\mathcal{L}(x,y)=\mathcal{L}(y,x)$. Then

$\displaystyle\mathcal{N}(\mu,\nu)=$ $\displaystyle
2\int_{\mathcal{X}}\int_{\mathcal{X}}\mathcal{L}(x,y)d\mu(x)d\nu(y)-\int_{\mathcal{X}}\int_{\mathcal{X}}\mathcal{L}(x,y)d\mu(x)d\mu(y)-\int_{\mathcal{X}}\int_{\mathcal{X}}\mathcal{L}(x,y)d\nu(x)d\nu(y)\geq
0$ (2)

holds for all measures $\mu,\nu\in\mathcal{B}_{\mathcal{L}}$ with equality in
the case $\mu=\nu$ only, if and only if $\mathcal{L}$ is a strongly negative
definite kernel.

In the sequel, the term $\mathcal{N}(\mu,\nu)$ from (2) is called the
$\mathcal{N}$-distance of the measures $\mu$ and $\nu$.

Some examples of strongly negative definite kernels $\mathcal{L}$, which can
be used for testing equality in distribution of real random variables, can be
found in [9, pp. 21–22]. Here, we use the Euclidean distance as negative
definite kernel when comparing only random values and a special kernel
developed for comparing random functions described below in this section when
comparing random functions. In both cases, we apply a permutation version of
test. The approach is as follows. Assume we have an observation
$x_{1},\ldots,x_{m_{1}}$ from an distribution $\mu$ and
$y_{1},\ldots,y_{m_{2}}$ from an distribution $\nu$. The
$\mathcal{N}$-distance of the measures $\mu$ and $\nu$ is estimated as

$\displaystyle\hat{\mathcal{N}_{1}}=\ $
$\displaystyle\frac{2}{m_{1}m_{2}}\sum\limits_{i=1}^{m_{1}}\sum\limits_{j=1}^{m_{2}}\mathcal{L}(x_{i},y_{j})-\frac{1}{m_{1}^{2}}\sum\limits_{i=1}^{m_{1}}\sum\limits_{j=1}^{m_{1}}\mathcal{L}(x_{i},x_{j})-\frac{1}{m_{2}^{2}}\sum\limits_{i=1}^{m_{2}}\sum\limits_{j=1}^{m_{2}}\mathcal{L}(y_{i},y_{j}).$
(3)

This value plays the role of the test characteristic. Then, we use a Monte
Carlo permutation test, i.e. we make $s$ permutations of all observed values
$x_{1},\ldots,x_{m_{1}},y_{1},\ldots y_{m_{2}}$, split each permutation into
two groups of the lengths $m_{1}$ and $m_{2}$, and, analogously to (3), we
calculate $\hat{\mathcal{N}}_{i}$ for the $i$-th permutation,
$i=2,\ldots,s+1$. Then the $p$-value of the test is

$p=\frac{\sharp\\{i\in\\{2,\ldots,s+1\\}:\hat{\mathcal{N}_{i}}\geq\hat{\mathcal{N}_{1}}\\}+1}{s+1}.$
(4)

For testing the equality in distribution of random functions is Section 3, we
use a kernel from [5] constructed especially for random functions. The test
works as follows. When we evaluate the testing functions $t^{(1)}$ and
$t^{(2)}$ in discrete arguments $u_{1},\ldots,u_{n},$ $n\in\mathbb{N}$, the
kernel is

$\mathcal{L}(t^{(1)},t^{(2)})=\sum\limits_{m=1}^{D}\sum\limits_{\left\\{k_{1},\ldots,k_{m}\right\\}\subseteq\left\\{1,\ldots,n\right\\}}\left(\sum\limits_{l=1}^{m}\left(t^{(1)}(u_{k_{l}})-t^{(2)}(u_{k_{l}}))\right)^{2}\right)^{1/2},$
(5)

where $D$ is a chosen constant specifying the depth of dependence, i.e. it is
suitable for testing the equality of finite-dimensional distributions of
random functions $t^{(1)}$ and $t^{(2)}$ for the dimensions less than or equal
to $D$ (for more details see [5]). The choice of the kernel (5) is based on
the simulation study from [5] which shows that it gives the largest test power
from all studied kernels. The estimate of the $\mathcal{N}$-distance of the
functions $t^{(1)}$ and $t^{(2)}$ based on the random samples $t^{(1)}_{i}$,
$i=1,\ldots,m_{1}$, and $t^{(2)}_{j}$, $j=1,\ldots,m_{2}$, respectively, is
evaluated analogously to the previous, i.e. as

$\displaystyle\hat{\mathcal{N}_{1}}=\ $
$\displaystyle\frac{2}{m_{1}m_{2}}\sum\limits_{i=1}^{m_{1}}\sum\limits_{j=1}^{m_{2}}\mathcal{L}(t^{(1)}_{i},t^{(2)}_{j})-\frac{1}{m_{1}^{2}}\sum\limits_{i=1}^{m_{1}}\sum\limits_{j=1}^{m_{1}}\mathcal{L}(t^{(1)}_{i},t^{(1)}_{j})-\frac{1}{m_{2}^{2}}\sum\limits_{i=1}^{m_{2}}\sum\limits_{j=1}^{m_{2}}\mathcal{L}(t^{(2)}_{i},t^{(2)}_{j}).$
(6)

Then, we again use a Monte Carlo permutation test making $s$ permutations of
all functions$t^{(1)}_{1}(u),\ldots,t^{(1)}_{m_{1}}(u),t^{(2)}_{1}(u),\ldots
t^{(2)}_{m_{2}}(u)$ in order to obtain $\hat{\mathcal{N}}_{i}$,
$i=2,\ldots,s+1$, and evaluate the $p$-value as described above.

##  3 Methodology

###  3.1 Testing characteristics

In this paper, we define the similarity of random sets through their
components, namely through the curvature of their boundaries and the ratios of
their perimeters and areas.

Consider a connected random set $\mathbf{X}$, i.e. the random set whose
realisations are connected. Denote $B_{\mathbf{X}}$ the boundary of
$\mathbf{X}$ and $\kappa_{\mathbf{X}}(z)$ the (random) curvature in the point
$z\in B_{\mathbf{X}}$. From (1), we can see that for a disc $b(z,r)$ with
suitable chosen radius r,

$\kappa_{\mathbf{X}}(z)\propto\frac{A^{*}_{b(z,r),\mathbf{X}}}{A_{b(z,r)}},$

where $A_{b(z,r)}$ is the area of the disc $b(z,r)$ and
$A^{*}_{b(z,r),\mathbf{X}}$ is the area of $b(z,r)\cap\mathbf{X}$. Therefore,
we focus only on the ratio of these two areas. Denote

$O_{\mathbf{X},b(z,r)}=\frac{A^{*}_{b(z,r),\mathbf{X}}}{A_{b(z,r)}}$

and define the function

$\tilde{\kappa}_{\mathbf{X},r}(u)=|B_{\mathbf{X}}|^{-1}\int_{B_{\mathbf{X}}}\mathbf{1}\\{O_{\mathbf{X},b(z,r)}\leq
u\\}dz,\quad u\in\langle 0,1\rangle,$

which is basically an analogy of the distribution function of the curvature at
points on the boundary, but it is evaluated for all boundary points, so it
describes the distribution for strongly dependent values. The object of our
interest is the function, analogous to density function, describing the
distribution of the curvature along the boundary, i.e.

$t_{\mathbf{X},r}(u)=\tilde{\kappa}_{\mathbf{X},r}^{\prime}(u).$ (7)

Finally, denote $R_{\mathbf{X}}$ the random variable describing the ratio of
the perimeter and the area of the random set $\mathbf{X}$.

###### Definition 3.1

Two connected random sets $\mathbf{X}$ and $\mathbf{Y}$ are considered to be
similar if the distributions of $\lim_{r\rightarrow 0}t_{\mathbf{X},r}$ and
$\lim_{r\rightarrow 0}t_{\mathbf{Y},r}$ as well as the distributions of
$R_{\mathbf{X}}$ and $R_{\mathbf{Y}}$ are equal.

In practice, we usually observe realisations $X$ and $Y$ of the random sets
$\mathbf{X}$ and $\mathbf{Y}$, respectively, in the form of binary images, so
we need to adjust the task of assessing dissimilarity of the realisations
consisting of black and white pixels. The pixels play the role of units in the
sequel. The ratio of the perimeter and the area is then simply given by the
number of boundary pixels divided by the number of all pixels of the
component. For evaluating of the function describing the curvature, fix a
radius $r\in\mathbb{N}$, denote $P$ the set of all pixels of the binary image
$X$, $z_{1},\ldots,z_{n}$ all boundary pixels, and for each boundary pixel
$z_{i}$, define

$K(z_{i})=\frac{\sharp\\{p\in P:p\in b(z_{i},r)\cap X\\}}{\sharp\\{p\in P:p\in
b(z_{i},r)\\}}.$

Then the approximation of the function (7) is

$t(u)=\frac{\sharp\\{i\in\\{1,\ldots,n\\}:K(z_{i})\in[u-1/l,u)\\}}{n},\quad
u=\frac{1}{l},\frac{2}{l},\ldots,1,$ (8)

which plays the role of testing function.

###  3.2 Testing similarity of connected random sets

Consider two samples, namely $X_{1},\ldots,X_{m_{1}}$ and
$Y_{1},\ldots,Y_{m_{2}}$, of realisations of connected random sets
$\mathbf{X}$ and $\mathbf{Y}$, respectively. We want to test the null
hypothesis that $\mathbf{X}$ and $\mathbf{Y}$ are similar. First we evaluate
the ratios $R_{X_{1}},\ldots,R_{X_{m_{1}}}$, $R_{Y_{1}},\ldots,R_{Y_{m_{2}}}$
of the perimeters and areas of the corresponding realisations. Based on these
values, we estimate the $\mathcal{N}$-distance of the ratios of $\mathbf{X}$
and $\mathbf{Y}$ by the formula (3), where we set $x_{i}=R_{X_{i}}$,
$i=1,\ldots,m_{1}$ and $y_{j}=R_{X_{j}}$, $j=1,\ldots,m_{2}$. Let us denote it
$\hat{\mathcal{N}^{R}_{1}}$. Then we evaluate the testing functions (8)
describing the boundary curvatures $t_{X_{1}}(u),\ldots,t_{X_{m_{1}}}(u)$,
$t_{Y_{1}}(u),\ldots,t_{Y_{m_{2}}}(u)$, calculate the $N$-distance of the
functions corresponding to $\mathbf{X}$ and $\mathbf{Y}$, respectively, by the
formulas (5) and (6), and denote this $\mathcal{N}$-distance as
$\hat{\mathcal{N}^{t}_{1}}$. The couple
$(\hat{\mathcal{N}^{R}_{1}},\hat{\mathcal{N}^{t}_{1}})$ is the test statistic.
Here, we use the Monte Carlo permutation test as described in Section 2.2,
i.e. we make $s$ permutations of all realisations $X_{1},\ldots,X_{m_{1}}$ and
$Y_{1},\ldots,Y_{m_{2}}$, and split them into two groups of the sizes $m_{1}$
and $m_{2}$, respectively, in order to obtain
$(\hat{\mathcal{N}}^{R}_{i},\hat{\mathcal{N}}^{t}_{i})$, $i=2,\ldots,s+1$, and
evaluate the $p$-value as

$p=\frac{\sharp\\{i\in\\{2,\ldots,s+1\\}:\hat{\mathcal{N}_{i}^{R}}\geq\hat{\mathcal{N}_{1}^{R}}\wedge\hat{\mathcal{N}_{i}^{t}}\geq\hat{\mathcal{N}_{1}^{t}}\\}+1}{s+1}.$
(9)

###  3.3 Similarity of random sets consisting of more components

Usually in practice, we have the data in the form of realisations consisting
of more than one component. If we can suppose that the components are
independent and come from the same distribution, then we can define similarity
of two random sets it the way that they are considered to be similar, if their
components are similar in the meaning of Definition 3.1. It is used in
simulation study below in Section 4. Nevertheless, the independence of the
components can be supposed in very specific cases, e.g. in germ-grain models
[2] in which the intensity of germs is low with respect to the volume of
grains. However, the components are usually dependent. In order to avoid this
complication, we make a suitable random sample of the components in each
realisation, and use these two samples as the input samples
$X_{1},\ldots,X_{m_{1}}$ and $Y_{1},\ldots,Y_{m_{2}}$ from Section 3.2.

##  4 Simulation study

In the simulation study, we first focus on four models which illustrates the
usage of the procedure. Then, we study models compared before in [3], [4], [5]
and [6]. The approach in the simulation study is analogous to that one
presented in [3], [4], [5] and [6]. For all considered models, we simulate 200
realisations and compare 100 vs 100 realisations of the same models as well as
100 vs 100 realisations of different models. Since the outputs of the tests
are the $p$-values, we obtain 100 $p$-values for each couple. Some of their
histograms are shown and commented below. Note that $p$-value close to zero
means that the equality of distributions of the corresponding testing
functions is rejected. Thus, the $p$-value should be concentrated close to
zero when comparing realisations of different models, while it should be
uniformly distributed in the interval $[0,1]$ when comparing realisations of
the same models.

In this simulation study, we moreover have to choose the radius of the disc
used to estimate the curvature. Briefly said, too large disc does not detect
localised changes in curvature in the sense that it can capture more than one
interface, on the other hand, too small disc has large error due to
discretisation. All realisation images in our simulation study are in
resolution of $400\times 400$ pixels (units). Some recommendations on how to
choose a suitable radius can be found in [1]. Based on these conditions and
personal consultation with Matěj Lébl (Institute of Information Theory and
Automation, Czech Academy of Science), we use the radii $r=3$ and $r=5$. Note
that the results of the simulation study are very similar for both radii. The
presented histograms show the $p$-values of the tests using $r=5$ (with one
exception mentioned below).

Examples of realisations of the illustrating models are shown in Fig. 1. The
first picture is a realisation of the random disc Boolean model used in
previous studies in [3], [4], [5] and [6] (see the last paper for details
about parameters of the model). The second realisation is simulated so that in
(another) realisation of the Boolean model, each connected component is
deleted with probability 1/2. It is called the reduced Boolean model in the
sequel. The third realisation is formed by disjoint squares whose ratio of the
perimeter and the area comes from the same distribution as the ratio for the
Boolean model (namely from the empirical distribution obtained from 100
realisations of Boolean model). We call it the square model in the sequel. The
fourth realisation is simulated as the process of disjoint rectangles with the
same distribution of perimeters as the square perimeters, while one side has
fixed length of 4 pixels (note that in this case, we use the disc with radius
$r=3$ only for estimating the boundary curvature). It is called the rectangle
model in the sequel.

Figure 1: Example of realisation of the Boolean, the reduced Boolean, the
square and the rectangle model, respectively.

We want to show that the method does not distinguish between the Boolean model
and reduced Boolean model since it is based only on the similarity of the
components, but it distinguishes between the Boolean models and the square
model due to the boundary curvature, as well as between the square model and
the rectangle model due to the ratios of the perimeter and the area of the
components. Indeed, we can see it in the first column of Fig. 2. The histogram
of $p$-values shows approximately uniform distribution when comparing the
Boolean model and reduced Boolean model, but the $p$-values are very close to
zero when testing the Boolean model vs the square model and the square model
vs the rectangle model. We moreover test the equality in distribution of the
curvature functions and of the ratios of perimeters and areas separately. The
histograms of the $p$-values of the test for the ratios are shown in the
second column, and the $p$-values of the test for the curvature functions are
shown in the third column. It is natural that the the $p$-values are
approximately uniformly distributed for both tests of the Boolean model vs
reduced Boolean model. The meaning of the procedure is clear from other
histograms. There, we can observe the equal distributions of the ratios of
perimeters and areas, but clearly different distributions of the curvature
functions when comparing the Boolean model and the square model, and
conversely the agreement of the distribution of the curvature functions and
different distributions of the ratios of perimeters and areas when testing the
square model vs the rectangle model. Thus, when we want to distinguish between
realisations with differently shaped components, both characteristics must be
taken into account.

Figure 2: Histograms of $p$-values when testing the Boolean model vs the
reduced Boolean model (the first row), the Boolean model vs the square model
(the second row) and the square morel vs the rectangle model using both
characteristics, i.e. the curvatures of the boundary and the ratios of the
area and perimeter of the components (the first colomn), only the ratios (the
second columns) and only the curvatures (the third columns).

Next thing we observe in our simulation study is that when we have all
components of the realisation in the sample, the $p$-values are greater than
we expect. It is seen when comparing the same models. The histograms of
$p$-values are located to the right, while they become more uniform when we
sample less components for testing, see Fig. 3. It is the effect of dependence
of components in each realisation. In realisations with densely placed
components, the shape of one component affects the shape of another one, so
they form something like a puzzle. Such sets of components are then more
similar then sets formed by independent components. From histograms in Fig. 3,
we conclude that in our case, we can take a sample of 10 components from
realisation of Boolean model to eliminate the effect of dependence of the
components.

Figure 3: Histograms of $p$-values when testing the Boolean model vs the
Boolean model using, respectively, 50, 30, 20 and 10 components from each
realisation.

Further, we consider models compared before in [3], [4], [5] and [6]. Except
the Boolean model mentioned above, which appears in all the mentioned
publications, we consider a model of partially repulsive particles (called the
repulsive model in the sequel) and a model of particles forming clusters
(called the cluster model in the sequel). Realisations of both these models
are simulated as realisations of the random disc Quermass-interaction process
[12] with suitable chosen parameters. More details about the parameters can be
found in [6] and [4]. Note that the same repulsive model is employed for
simulation studies in all above mentioned papers, similarly as the Boolean
model, while the same cluster model is used only in [4]. In [3], [5] and [6],
another cluster model is considered, which is not suitable for our current
study since its realisations consist of too few components. The fourth model
is the Boolean model with grains to be ellipses (called the ellipse model in
the sequel). It appears, similarly as the cluster model, only in [4], because
in the other papers, its application would not be interesting. Examples of
realisations of the four models are shown in Fig. 4.

Figure 4: Example of realisation of the Boolean, the repulsive, the cluster
and the ellipse model, respectively.

First, we test the similarity of the same models. For the Boolean model, we
can see in Fig. 3 that the $p$-values are approximately uniformly distributed
for samples of the size between 10 and 20 components. Such a large sample can
be viewed as a sample of weakly dependent components. Therefore, we make
samples of 10 and 20 components from each realisation of the remaining models.
The histograms in Fig. 5 shows that the $p$-values are uniformly distributed
when testing the samples of 20 components for the repulsive model and for the
ellipse model, while for the cluster model, the sample is not rare enough, it
has uniformly distributed $p$-values for the samples of 10 components.

Figure 5: Example of realisation of the Boolean, the reduced Boolean, the
square and the rectangle model, respectively.

Based on this observation, we use the samples of 10 components for testing
similarity of different models. Their histograms are shown in Fig. 6. We can
see that the $p$-values are more or less close to zero, but the rejection of
the similarity hypothesis is not very convincing.

Figure 6: Histograms of $p$-values when testing similarity of the Boolean
model vs the repulsive model (upper left), the Boolean model vs the cluster
model (upper middle), the repulsive model vs the cluster model (upper right),
the ellipse model vs the Boolean model (lower left), the ellipse model vs
repulsive model (lower left) and the ellipse model vs the cluster model (lower
left) using the samples of 10 components.

We assume that this is due to the small number of components in the test
sample, but we cannot create a larger sample from our simulated realisations
because of the interdependence of the components. Therefore, we try to apply
the boot strap method. We mix all the components from each model together and,
when testing the similarity of the two models, we randomly select 100
components of one model and 100 components of the second one and calculate the
$p$-value of the similarity test. We repeat this approach one hundred times to
get 100 $p$-values for construction of histogram. The histograms are shown in
Fig. 7. We can see that except comparing the repulsive model and the cluster
model, almost all $p$-values are less than 0.05 now. The reason for larger
$p$-values in the case of the repulsive model and the cluster model is the
fact that many components in these models are formed by isolated discs that
come from the same distribution, see the model parameters in [6].

Figure 7: Histograms of $p$-values when testing similarity of the Boolean
model vs the repulsive model (upper left), the Boolean model vs the cluster
model (upper middle), the repulsive model vs the cluster model (upper right),
the ellipse model vs the Boolean model (lower left), the ellipse model vs
repulsive model (lower left) and the ellipse model vs the cluster model (lower
left) using boot strap method and the samples of 100 components.

##  5 Application to real data

Finally, we apply the procedure to real data kindly provided by the authors of
[14]. We work with binary images of two different types of mammary tissue,
namely 8 images of mastopathy tissue and 8 images of mammary cancer, see Fig.
8 and 9. Note that the breast contains a branching system of ducts spanning
down from the nipple to glands. The tissue between the ducts and glands is
made of fat and fibrous tissue of different proportions. The morphology of
this tissue may indicate various malignant or benign changes.

The images are in resolution of 512 $\times$ 5120 pixels. From each image, we
randomly sample 20 components, and then, we test the similarity of the random
sets represented by the images each to each, using $R=5$. We repeat the
procedure 100 times, while we evaluate the mean $p$-value of the test for each
couple of images (including the image with itself) and the number of
$p$-values below 0.05 which indicate significant dissimilarity on the
classical level. The results are introduced in Tab. 1 and Tab. 2. We can
observe that the $p$-values are significantly lower and the number of
$p$-values below 0.05 is significantly higher when comparing pairs of
different types of tissue than that ones for pairs of the same types of
tissue. Just note that we repeated the same approach also for $R=3$ and for
the samples of 10 components from each image and the results were very
similar.

Figure 8: Samples of mastopathy breast tissue. Figure 9: Samples of mammary cancer. Table 1: Mean $p$-values (rounded to 2 decimal places) when comparing the corresponding samples 100 times. The values related to couples of different types of tissue are marked with italic font. | m1 | m2 | m3 | m4 | m5 | m6 | m7 | m8 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
m1 | .81 | .46 | .35 | .02 | .19 | .41 | .04 | .04 | .01 | .02 | .04 | .10 | .02 | .03 | .04 | .06
m2 |  | .60 | .10 | .00 | .05 | .38 | .07 | .03 | .01 | .04 | .06 | .20 | .03 | .05 | .06 | .11
m3 |  |  | .87 | .35 | .57 | .20 | .00 | .00 | .00 | .00 | .01 | .01 | .00 | .01 | .00 | .00
m4 |  |  |  | .90 | .13 | .04 | .00 | .00 | .00 | .00 | .00 | .00 | .00 | .01 | .00 | .00
m5 |  |  |  |  | .78 | .32 | .00 | .00 | .00 | .00 | .01 | .01 | .00 | .01 | .00 | .01
m6 |  |  |  |  |  | .65 | .03 | .01 | .01 | .03 | .03 | .14 | .01 | .09 | .04 | .10
m7 |  |  |  |  |  |  | .59 | .49 | .14 | .16 | .10 | .36 | .45 | .02 | .19 | .15
m8 |  |  |  |  |  |  |  | .61 | .10 | .19 | .10 | .28 | .40 | .01 | .15 | .13
c1 |  |  |  |  |  |  |  |  | .53 | .38 | .32 | .17 | .27 | .11 | .35 | .26
c2 |  |  |  |  |  |  |  |  |  | .55 | .43 | .41 | .35 | .17 | .50 | .37
c3 |  |  |  |  |  |  |  |  |  |  | .57 | .29 | .18 | .31 | .47 | .38
c4 |  |  |  |  |  |  |  |  |  |  |  | .57 | .25 | .20 | .35 | .40
c5 |  |  |  |  |  |  |  |  |  |  |  |  | .55 | .03 | .35 | .16
c6 |  |  |  |  |  |  |  |  |  |  |  |  |  | .54 | .24 | .40
c7 |  |  |  |  |  |  |  |  |  |  |  |  |  |  | .51 | .42
c8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | .54
Table 2: The number of $p$-values bellow $.05$ when comparing the corresponding samples 100 times. The values related to couples of different types of tissue are marked with italic font. | m1 | m2 | m3 | m4 | m5 | m6 | m7 | m8 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
m1 | 0 | 4 | 9 | 92 | 29 | 5 | 81 | 81 | 97 | 92 | 82 | 57 | 88 | 79 | 80 | 70
m2 |  | 3 | 67 | 100 | 71 | 13 | 71 | 84 | 95 | 86 | 78 | 37 | 86 | 65 | 74 | 57
m3 |  |  | 0 | 5 | 1 | 29 | 99 | 100 | 100 | 100 | 98 | 98 | 100 | 96 | 99 | 98
m4 |  |  |  | 0 | 47 | 77 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 96 | 100 | 100
m5 |  |  |  |  | 0 | 19 | 100 | 100 | 100 | 100 | 97 | 96 | 100 | 95 | 99 | 93
m6 |  |  |  |  |  | 0 | 87 | 96 | 96 | 91 | 89 | 54 | 97 | 62 | 84 | 62
m7 |  |  |  |  |  |  | 1 | 5 | 40 | 36 | 64 | 16 | 6 | 91 | 29 | 50
m8 |  |  |  |  |  |  |  | 1 | 52 | 29 | 60 | 15 | 17 | 97 | 33 | 50
c1 |  |  |  |  |  |  |  |  | 6 | 12 | 19 | 43 | 20 | 63 | 13 | 28
c2 |  |  |  |  |  |  |  |  |  | 2 | 13 | 8 | 20 | 52 | 3 | 14
c3 |  |  |  |  |  |  |  |  |  |  | 2 | 18 | 47 | 18 | 5 | 12
c4 |  |  |  |  |  |  |  |  |  |  |  | 1 | 21 | 39 | 20 | 12
c5 |  |  |  |  |  |  |  |  |  |  |  |  | 1 | 89 | 23 | 44
c6 |  |  |  |  |  |  |  |  |  |  |  |  |  | 7 | 35 | 16
c7 |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 2 | 12
c8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 3

##  6 Conclusion

A new statistical test for assessing (dis)similarity of two random sets has
been constructed. It works with two realisations - one realisation of each of
the random sets. The procedure focuses only on shapes of the components of the
random sets, namely on the curvature of their boundaries together with the
ratios of their perimeters and areas, and it does not take into account the
positions of the components in the realisations, since it is very often
required in practical applications.

The described method is equipped by a simulation study. The study shows that
under quite mild conditions, the test has large power when distinguishing
realisations of different models. The power is larger than the method in [6]
and one of the methods in [4], which also distinguishes realisations based on
the shape of the components, and comparable to another method in [4] and to
the method in [3], which, however, take into account the placement of
components. Another advantage with respect to the method in [6] is that it is
not heuristic, and moreover, it does not require a lot of input parameters as
needed in [6] and [3].

A small complication occurs when components of compared realisations are too
close to each other, because the method assumes independence of the
components, which is not satisfied when the components are too densely placed.
However, the problem has an easy solution - only a sample of the components
can be considered instead of all components. In this case, the dependence is
reduced and the method works very well.

Finally, the procedure is applied to real data. The data consists of pictures
of two different types of tissue, namely the tissue of mammary cancer and
mastopathic tissue. We have 8 pictures of each type. The aim is to distinguish
between different types and to assess the pictures of the same type of the
tissue as similar. Considering how different the images of the same type
appear to be at first glance and how difficult it is to identify specific
distinguishing features for different types, the results of the method are
very good.

From the above observations, we can conclude that the new method works very
well and has a high potential to be a useful tool for comparing realisations
of random sets.

#### Acknowledgment

Supported by The Czech Science Foundation, project No. 19-04412S.

## References

  * [1] Bullard J. W., Garboczi E. J., Carter W. C., and Fuller E. R. Jr.: Numerical methods for computing interfacial mean curvature, Computational Materials Science, vol. 4, pp. 103–116, 1995.
  * [2] S. N. Chiu, D. Stoyan, W. S. Kendall, and J. Mecke, Stochastic geometry and its applications. John Wiley & Sons, New York, 2013.
  * [3] J. Debayle, V. Gotovac Dogaš, K. Helisová, J. Staněk, and M. Zikmundová “Assessing similarity of random sets via skeletons,” Methodology and Computing in Applied Probability, DOI: 10.1007/s11009-020-09785-y,
  * [4] V. Gotovac, “Similarity between random sets consisting of many components.” Image Anal Stereol, vol.38, pp. 185–99, 2019.
  * [5] V. Gotovac Dogaš and K. Helisová, “Testing equality of distributions of random convex compact sets via theory of N-distances,” Methodology and Computing in Applied Probability, 2021+, DOI: 10.1007/s11009-019-09747-z
  * [6] V. Gotovac, K. Helisová, and I. Ugrina, “Assessing dissimilarity of random sets through convex compact approximations, support functions and envelope tests,” Image Anal Stereol, vol. 35, pp. 181–93, 2016.
  * [7] Gretton A, Borgwart KM, Rash MJ, Scholkopf B, Smola A (2012) A Kernel Two-Sample Test. J Mach Learn Res 13: 723–73
  * [8] P. Hermann, T. Mrkvička, T. Mattfeldt, M. Minárová, K. Helisová, O. Nicolis, F. Wartner, and M. Stehlík, “Fractal and stochastic geometry inference for breast cancer: a case study with random fractal models and Quermass-interaction process,” Stat in Med, vol. 34.18, pp. 2636–61, 2015.
  * [9] L. B. Klebanov, $\mathcal{N}$-distances and their applications. Karolinum Press, Charles University, Prague, 2006.
  * [10] G. Matheron, Random Sets and Integral Geometry. John Wiley & Sons Inc, New-York, 1975.
  * [11] I. Molchanov, Theory of random sets. Springer, New York, 2005.
  * [12] J. Møller and K. Helisová, “Power diagrams and Interaction processes for unions of discs,” Adv Appl Probab, vol. 40, pp. 321–47, 2008.
  * [13] J. Møller and K. Helisová, “Likelihood inference for unions of interacting discs,” Scand Stat, vol. 37, pp. 365–81, 2010.
  * [14] Mrkvička T, Mattfeldt T (2011): Testing histological images of mammary tissues on compatibility with the Boolean model of random sets. Image Analysis and Stereology 30(1), 11–18.
  * [15] M. Myllymäki, T. Mrkvička, P. Grabarnik, H. Henri Seijo, and U. Hahn, “Global envelope tests for spatial processes,” Journal of the Royal Stat Society: Series B (Statistical Methodology), vol. 79, pp. 381–404, 2017.
  * [16] M. Neumann, J. Staněk, O. M. Pecho, L. Holzer, V. Beneš, and V. Schmidt, “Stochastic 3D modeling of complex three-phase microstructures in SOFC-electrodes with completely connected phases,” Comp Mat Sci vol. 118, pp. 353–64, 2016.
  * [17] J. Serra, Image Analysis and Mathematical Morphology, vol.2: Theoretical Advances. Academic Press, 1982.