# DeepJSCC-l++: Robust and Bandwidth-Adaptive Wireless Image Transmission

Chenghong Bian, Yulin Shao, , Deniz Gündüz C. Bian and D. Gündüz are with the
Department of Electrical and Electronic Engineering, Imperial College London,
London SW7 2AZ, U.K. (e-mail: {c.bian22, d.gunduz}@imperial.ac.uk). Y. Shao is
with the State Key Laboratory of Internet of Things for Smart City, University
of Macau, Macau S.A.R. (e-mail: ylshao@um.edu.mo). This work was supported in
part by UKRI for the project AIR (ERC-CoG, EP/X030806/1), and in part by the
IOTSC-UM Conference Grant SKL-IoTSC(UM)-2021-2023 and the start-up grant
SRG2023-00038-IOTSC.

###### Abstract

This paper presents a novel vision transformer (ViT) based deep joint source
channel coding (DeepJSCC) scheme, dubbed DeepJSCC-l++, which can adapt to
different target bandwidth ratios as well as channel signal-to-noise ratios
(SNRs) using a single model. To achieve this, we treat the bandwidth ratio and
the SNR as channel state information available to the encoder and decoder,
which are fed to the model as side information, and train the proposed
DeepJSCC-l++ model with different bandwidth ratios and SNRs. The
reconstruction losses corresponding to different bandwidth ratios are
calculated, and a novel training methodology, which dynamically assigns
different weights to the losses of different bandwidth ratios according to
their individual reconstruction qualities, is introduced. Shifted window
(Swin) transformer is adopted as the backbone for our DeepJSCC-l++ model, and
it is shown through extensive simulations that the proposed DeepJSCC-l++ can
adapt to different bandwidth ratios and channel SNRs with marginal performance
loss compared to the separately trained models. We also observe the proposed
schemes can outperform the digital baseline, which concatenates the BPG
compression with capacity-achieving channel code. We believe this is an
important step towards the implementation of DeepJSCC in practice as a single
pre-trained model is sufficient to serve the user in a wide range of channel
conditions.

###### Index Terms:

Semantic communication, DeepJSCC, Swin Transformer, bandwidth adaptive,
dynamic weight assignment.

##  I Introduction

Thanks to recent advances in machine learning, there has been a growing
interest in developing data-driven joint source-channel coding (JSCC) systems.
Focusing on the wireless transmission of images, the DeepJSCC scheme proposed
in [1] is shown to achieve better performance and enhanced robustness against
channel variations compared to conventional separation-based baselines. Over
the last several years, DeepJSCC approach has been successfully extended to
many new scenarios, exhibiting its potential as a viable technology, from
multi-path fading channel [2, 3] to multi-input multi-output (MIMO) [4, 5] and
multi-user scenarios [6].

On the other hand, in most existing works, the DeepJSCC encoder/decoder pairs
are designed and trained for specific channel conditions, i.e., channel
bandwidth and signal-to-noise ratio (SNR). This is a limitation for the
adoption of DeepJSCC in practical systems, as it requires storing a large
number of DeepJSCC encoder/decoder parameters on mobile devices to be used in
different channel conditions, imposing significant memory requirements. In [7,
3], it has been shown that a single DeepJSCC network can adapt to different
channel SNRs. In this work, we will show that a single DeepJSCC
encoder/decoder pair can be trained to be used in any available channel
bandwidth and SNR. This not only shows the flexibility of DeepJSCC, but
significantly increases its potential to be applied in practice.

DeepJSCC with a varying bandwidth is also studied in [8]; however, [8]
considers a successive refinement scenario, where transmission takes place
over several channel blocks, and the image should be recovered from any number
of first $l$ channel blocks (Fig. 1(a)). This scenario is applicable when
broadcasting an image to several receivers, where each receiver can receive a
different number of channel blocks. Instead, we consider a single receiver,
where the available channel bandwidth can vary from image to image, but is
known for each image prior to transmission, dictated, for example, by the
latency constraint of the underlying application. Alternatively, variable-
length transmission is studied in [9, 10, 11], where the encoder decides how
much channel bandwidth to use based on the content of the input image.

In this paper, we propose a novel bandwidth and channel quality adaptive
scheme, named DeepJSCC-l++, which can map each input image to a desired
channel bandwidth - see Fig. 1(b). The DeepJSCC-l++ encoder takes the image as
well as the bandwidth ratio and the channel SNR as side information to produce
the codeword. We introduce a novel code architecture using the Swin
transformer [12] as the backbone. To balance the reconstruction qualities for
different bandwidth ratios, a novel training methodology, called dynamic
weight assignment (DWA), is introduced, which assigns different weights to the
losses corresponding to different bandwidth ratios. Simulation results show
the effectiveness of the proposed DeepJSCC-l++ scheme, which can be adaptive
to both the bandwidth ratio and the channel SNR with negligible gap from the
reconstruction performance obtained with separately trained models. This shows
that the proposed architecture is capable of acquiring and prioritising the
input image features, and sending only the most important features depending
on the available bandwidth, while employing the necessary amount of redundancy
against channel noise depending on the channel SNR.

We highlight that the proposed DeepJSCC-l++ architecture can also be applied
to the successive refinement problem. Our results show that DeepJSCC-l++
provides a significant performance improvement compared to the CNN-based
solution in [8] in this scenario. Finally, DeepJSCC-l++ outperforms the
separation-based baseline that combines BPG compression algorithm with a
capacity-achieving channel code. This result shows the potential of the
transformer architecture in designing state-of-the-art DeepJSCC solutions.

##  II System Model

We consider the wireless transmission of images over the AWGN channel. Let
$\bm{S}\in\mathbb{R}^{C\times H\times W}$ denote the input image, where
$C,H,W$ denote the number of color channels, height, and width of the image,
respectively. We define $N=CHW$ to denote the input dimension to facilitate
following definitions. The encoder maps the input image, $\bm{S}$, to a
complex codeword $\bm{z}\in\mathbb{C}^{\rho N}$, where $\rho N\in\mathbb{Z}$,
denotes the bandwidth used for transmission. Here, $\rho$ is defined as the
bandwidth ratio as it represents the average number of channel symbols
available per source dimension. The transmitted codeword $\bm{z}$ goes through
an AWGN channel, $\bm{y}=\bm{z}+\bm{w}$, where each element in $\bm{w}$
follows a complex Gaussian distribution with zero mean and variance equals to
$\sigma^{2}$, and $\bm{y}\in\mathbb{C}^{\rho N}$ denotes the noisy channel
output vector. We impose an input power constraint: $\frac{1}{\rho
N}\|\bm{z}\|_{2}^{2}\leq 1$ for each $\bm{z}$, which means that the channel
SNR is given by $\mathrm{SNR}=1/\sigma^{2}$.

The decoder maps the received vector directly to its estimate of the input
signal, $\widetilde{\bm{S}}\in\mathbb{R}^{C\times H\times W}$. The
reconstruction quality can be measured through a variety of distortion
measures. In this paper, we will consider the most commonly used peak signal-
to-noise ratio (PSNR), defined as:

PSNR
$\displaystyle=10\log_{10}\frac{255^{2}}{\frac{1}{N}||\bm{S}-\widetilde{\bm{S}}||^{2}_{F}}.$
(1)

glo In conventional separate source-channel coding schemes, we choose a pair
of compression and channel coding rates depending on the channel SNR and
bandwidth ratio, $\rho$. Channel coding rate depends on the channel SNR, and
is chosen to guarantee reliable transmission with high probability. The source
compression rate is dictated by the channel code rate and the available
bandwidth ratio, $\rho$. In practical systems, a mobile device chooses from a
list of prescribed list of modulation and coding schemes (MCSs) according to
the estimated channel SNR. In the case of JSCC, since we have a single code,
the code parameters will depend on both the SNR and the bandwidth ratio. The
initial works on DeepJSCC [1] considered a similar approach to these practical
systems, where a separate DeepJSCC encoder/decoder pair is trained for given
$(\mathrm{SNR},\rho)$ pairs. However, due to the high memory complexity of
DeepJSCC codes, it is not practically feasible to assume that each device can
store a large variety of code parameters to be used in different channel
conditions. Therefore, the goal in this paper is to train a single
encoder/decoder pair, which can dynamically adapt to the desired bandwidth
ratio and $\mathrm{SNR}$ level in an online manner at the time of
transmission.

Let $f_{\Theta}$ denotes the encoder function parameterized by DNN parameters
$\Theta$, while the decoder is denoted by $g_{\Phi}$, paramaterized by $\Phi$.
A single DNN model will be utilized to adapt to different $\mathrm{SNR}$ and
$\rho$ values, where the encoder $f_{\Theta}$ takes the image $\bm{S}$ as well
as the $\mathrm{SNR}$ and $\rho$ as input, and we have
$\bm{z}=f_{\Theta}(\bm{S},\mathrm{SNR},\rho)\in\mathbb{C}^{\rho N}$. The
receiver then reconstructs the original image as
$\widetilde{\bm{S}}=g_{\Phi}(\bm{y},\mathrm{SNR},\rho)\in\mathbb{R}^{C\times
H\times W}$. For the sake of simplicity, we will assume that only $L$
different bandwidth ratios are allowed, specified by
$\\{\rho_{1},\ldots,\rho_{L}\\}$, where we assume that
$\rho_{l}=l\cdot\rho_{1}$, $l\in[L]$.

((a)) Successive refinement: Each image is transmitted using the full
bandwidth, but can be recovered from any of the first $l$ channel blocks.

((b)) Bandwidth-adaptive transmission: Available channel bandwidth is dictated
by the application, and may change from image to image.

Figure 1: Successive refinement vs. bandwidth-adaptive transmission.
Figure 2: The detailed neural network architecture for the proposed
DeepJSCC-l++ model.

Next, we present the successive refinement problem formulation considered in
[8] and highlight the differences with respect to the adaptive-bandwidth
formulation considered here. As shown in Fig. 1(a), in the successive
refinement scheme studied in [8], the encoder maps the image into a latent
vector $\tilde{\bm{z}}\in\mathbb{C}^{\rho_{L}N}$, which is further power
normalized to $\bm{z}$ before transmission. Then, $L$ different receivers are
considered, where receiver $l$ receives only the first $l$ portions of the
noisy signal. That is, if we define
$\bm{y}^{\top}=\bm{y}^{\top}_{1:L}=[\bm{y}_{1}^{\top},\cdots,\bm{y}_{L}^{\top}]$,
then the receiver $l$ reconstructs the signal as
$\widetilde{\bm{S}}_{l}=g_{\Phi,l}(\bm{y}_{1:l},\mathrm{SNR})$. Therefore, in
this scheme, the goal of the transmitter is to generate a codeword that can
simultaneously satisfy $L$ receivers, each of which receives a different
amount of information. Accordingly, the additional bandwidth available to
receiver $l$ is used to refine the reconstruction generated by receiver
$(l-1)$. Additionally, the formulation in [8] imposes a separate power
normalization to each part of the transmitted codeword.

We note that the DeepJSCC-l scheme proposed in [8] with successive refinement
is also a solution to our problem, although we expect it to be a suboptimal
solution since in our problem we have a more relaxed requirement: the
transmitter knows the available bandwidth, and can adapt its transmission
accordingly, as a consequence, a single average power allocation can be used.

###  II-A Proposed solution: DeepJSCC-l++

Here, we propose a new architecture called DeepJSCC-l++. In the proposed
solution, the encoder maps the input image to an output of dimension
$\rho_{L}N$, denoted by $\tilde{\bm{z}}$, similarly to DeepJSCC-l. However,
instead of transmitting the full-bandwidth codeword, we adopt a simple mask at
the encoder to transmit only the first $\rho_{l}N$ elements, denoted by
$\bm{z}_{l}$, which is expressed as

$\displaystyle\bm{z}_{l}=\mathbb{M}(\tilde{\bm{z}},l),$ (2)

where $\mathbb{M}(\cdot,\cdot)$ represents the masking operation. Note that
$\bm{z}_{l}$ is subject to a more flexible power constraint compared with the
successive refinement scheme: $\frac{1}{\rho_{l}N}{||\bm{z}_{l}||^{2}_{2}}\leq
1$. The power normalized signal is then transmitted over the complex AWGN
channel. At the receiver, the decoder zero-pads the received signal
$\bm{y}_{l}$ to a length-$\rho_{L}N$ vector, $\tilde{\bm{y}}$, and takes both
$\tilde{\bm{y}}$ and the side information $\rho_{l},\mathrm{SNR}$ as input to
reconstruct
$\widetilde{\bm{S}}_{l}=g_{\Phi}(\tilde{\bm{y}},\mathrm{SNR},l)\in\mathbb{R}^{C\times
H\times W}$ using a decode function
$g_{\Phi}:\mathbb{R}^{\rho_{L}N}\times\mathbb{R}\times\mathbb{Z}\rightarrow\mathbb{R}^{C\times
H\times W}$.

##  III Methodology

In this section, we present the neural network architectures to parameterize
the encoder and decoder $f_{\Theta}$ and $g_{\Phi}$ for the DeepJSCC-l++. Then
we propose a novel training methodology called DWA to avoid substantial
performance loss at the higher bandwidth ratios.

###  III-A Neural Network Architectures

As shown in Fig. 2, we use one of the state-of-the-art vision transformer
models, the Swin transformer [12], as the backbone, and the data flow for the
proposed scheme is detailed as follows:

Data flow at the encoder. We start with the initial stage where the image
$\bm{S}$ is first split into non-overlapping patches (also known as ‘tokens’)
by a patch partition module followed by a linear embedding layer to project
each ‘token’ into feature space with dimension $c$. We use a patch size of
$2\times 2$ thus the aforementioned patch partition and linear embedding
modules transform the original image from dimension $C\times H\times W$ to a
feature tensor $\widetilde{\bm{X}}$ with dimension $c\times H/2\times W/2$.
Before feeding $\widetilde{\bm{X}}$ into the subsequent transformer layers, we
concatenate each of its tokens with the side information,
$\bm{u}\in\mathbb{R}^{n_{u}}$, to form a larger tensor $\bm{X}$ with dimension
$(c+n_{u})\times H/2\times W/2$. Note that $\bm{u}$ is obtained by feeding
both $\mathrm{SNR}$ and $\rho_{l}$ ($\rho_{l}$ is not needed if the successive
refinement scheme is adopted) to a fully connected layer and $\bm{X}$ can be
obtained as:

$\displaystyle\bm{u}=\mathrm{MLP}([\mathrm{SNR},\rho_{l}])$
$\displaystyle\bm{X}=\widetilde{\bm{X}}$
$\displaystyle\oplus\bm{U}_{e}\quad\text{with}\;\bm{U}_{e}[:,i,j]=\bm{u},$ (3)

where $\bm{U}_{e}\in\mathbb{R}^{n_{u}\times H/2\times W/2}$ is obtained by
duplicating the vector $\bm{u}$ and $\oplus$ denotes concatenation. Then
$M_{1}$ Swin transformer blocks111Note that $M_{1}$ is required to be an even
number since the shifted window self-attention is performed over two
consecutive transformer blocks. are applied to $\bm{X}$ before proceeding into
the next stage. We briefly introduce the operations of the Swin transformer
block and refer readers to [12] for more details.

The Swin transformer blocks resemble those in the standard vision transformer
[13] except the multi-head self-attention module is replaced by the one based
on shifted windows. We describe the operations of two consecutive Swin
transformer blocks as follows. Assume that the input feature tensor
$\bm{X}_{i}$ at stage $i$ has dimension $c\times h_{i}\times w_{i}$ and each
window contains $w\times w$ patches/tokens, then the first Swin transformer
block whose operation is denoted by W-MSA, evenly partitions $\bm{X}_{i}$ into
$(h_{i}/w,w_{i}/w)$ non-overlapping windows222We assume that $h_{i}$ and
$w_{i}$ are multiples of $w$. We can simply zero-pad the corresponding tensors
if the assumption does not hold. then performs multi-head self-attention
within each window. To allow communications between different windows, a
shifted window multi-head self-attention operation, denoted as SW-MSA, is
adopted, where a cyclic shift is applied to the window configuration in the
first transformer block followed by the multi-head self-attention operated
within the new (shifted) windows. We summarize the operations as follows:

$\displaystyle\hat{\bm{X}}_{i}^{(1)}$
$\displaystyle=\textbf{W-MSA}(\text{LN}(\bm{X}_{i}))+\bm{X}_{i}$
$\displaystyle{\bm{X}}_{i}^{(1)}$
$\displaystyle=\text{MLP}(\text{LN}(\hat{\bm{X}}_{i}^{(1)}))+\hat{\bm{X}}_{i}^{(1)}$
$\displaystyle\hat{\bm{X}}_{i}^{(2)}$ $\displaystyle=\textbf{SW-
MSA}(\text{LN}(\bm{X}_{i}^{(1)}))+\bm{X}_{i}^{(1)}$
$\displaystyle{\bm{X}}_{i}^{(2)}$
$\displaystyle=\text{MLP}(\text{LN}(\hat{\bm{X}}_{i}^{(2)}))+\hat{\bm{X}}_{i}^{(2)}.$
(4)

where LN denotes the layer normalization and we use the superscripts to
distinguish the features processed by the first and second Swin transformer
block. By applying the operations defined in (4) $M_{i}/2$ times, we obtain
the final output of the $i$-th stage.

As shown in Fig. 2, the output of the Swin transformer blocks in the $i$-th
stage is then fed to the patch merging module in the $(i+1)$-th stage, whose
output is of dimension $c\times h_{i+1}\times w_{i+1}$. Note that, in our
setting, the patch merging layer concatenates the features of $2\times 2$
neighboring patches and applies a linear layer to reduce the $4c$-dimensional
features to a dimension of $c$. Thus, we have
$(h_{i+1},w_{i+1})=(h_{i}/2,w_{i}/2)$. After passing all the $I$ stages, we
obtain the output with dimension $c\times h_{I}\times w_{I}$, which is
reshaped and linear projected to matrix $\widetilde{\bm{Z}}$ with dimension
$N_{F}\times N_{T}$, where $N_{F}N_{T}=2\rho_{L}N$ with $N_{T}=h_{I}w_{I}$ .

Varying Patches versus Varying Features. The matrix $\widetilde{\bm{Z}}$
contains $N_{T}$ tokens, each consisting of $N_{F}$ features for the maximum
bandwidth ratio $\rho_{L}$. To be adaptive to different $\rho_{l}$’s, one may
either transmit a reduced number of tokens $n_{t}<N_{T}$, while keeping the
dimension of features per token the same ($n_{f}=N_{F}$), which is called
varying patches, or reduces the dimension of features per token (we assume the
same $n_{f}$ for different tokens) while fixing the number of tokens
($n_{t}=N_{T}$), which is referred to as varying features. Note that a more
flexible bandwidth adaptive scheme is proposed in [11], where different
$n_{f}$’s are assigned to different tokens for better performance. However, we
argue that this requires transmitting additional digital information to inform
the decoder concerning how to partition the received signal for each token.
When the digital information is not correctly decoded, the decoder would
totally fail to reconstruct the original image. Our schemes, both the varying
patches and the varying features, on the other hand, do not require to
transmit any digital information333We assume that $\rho_{l}$ is available to
both the transmitter and receiver as part of the control channel information..
We evaluate the reconstruction performance for the two schemes, and find that
they yield similar performance, which will be detailed in Section IV. In the
following discussions, we employ the varying features scheme.

Data flow at the decoder. Upon receiving
$\bm{y}_{l}\in\mathbb{C}^{\rho_{l}N}$, the receiver first converts it to a
real tensor and then zero-pads it along the first dimension to obtain
$\widetilde{\bm{Y}}_{l}\in\mathbb{R}^{N_{F}\times h_{I}\times w_{I}}$, denoted
by $\widetilde{\bm{Y}}_{l}=\mathcal{P}(\bm{y}_{l})$, where $\mathcal{P}$
represents the reshaping and zero-padding operations. As shown in Fig. 2, the
same side information $\bm{u}$ for the encoder is first duplicated to
$\bm{U}_{d}$ and then concatenated to each token of $\widetilde{\bm{Y}}_{l}$
to form ${\bm{Y}}_{l}\in\mathbb{R}^{(N_{F}+n_{u})\times h_{I}\times w_{I}}$
whose $h_{I}\times w_{I}$ tokens will be further mapped to a $c$-dimensional
vector. The decoder also has $I$ stages and each stage consists of a patch
division block and Swin transformer blocks. To be precise, the patch division
block adopts pixel shuffling to upsample the spatial dimension of the input
tensors. The upsampled tensor is fed to the Swin transformer blocks in the
subsequent stage, whose structures are identical to those at the encoder.
After passing all $I$ stages, a 2d-convolutional layer converts the latent
tensor to the reconstructed image
$\widetilde{\bm{S}}_{l}\in\mathbb{R}^{C\times H\times W}$.

Algorithm 1 Overall Training Process for DeepJSCC-l++ Model with DWA.

1:Initialize $w_{l}^{1}=1,\forall l\in[L]$

2:for $t=1,\ldots,T$ do

3: Training Phase:

4: for each batch do

5: Sample $l\in[L],\mathrm{SNR}\in[\mathrm{SNR}_{min},\mathrm{SNR}_{max}]$

6: Encoder: $\bm{z}_{l}=f_{\Theta}(\bm{S},\mathrm{SNR},l)$

7: Decoder: $\widetilde{\bm{S}}_{l}=g_{\Psi}(\bm{y},\mathrm{SNR},l)$

8: Weighted Loss:
$\mathcal{L}_{l}^{t}=w_{l}^{t}||\bm{S}-\widetilde{\bm{S}}_{l}||^{2}_{2}$.

9: Optimize $\\{\Theta,\Psi\\}$ using $\mathcal{L}_{l}^{t}$.

10: Validation Phase:

11: for $l\in[L]$ do

12: Calculate $\text{PSNR}^{t}_{l},\Delta_{l}^{t}$ over validation set.

13: Update $w_{l}^{t}$ according to (5).

###  III-B Dynamic Weight Assignment (DWA)

In this subsection, we introduce a training methodology designed for both the
successive refinement and adaptive-bandwidth frameworks, which is essential
for good reconstruction performance with different bandwidth ratios.

To train a single model that is adaptive to the bandwidth ratio $\rho_{l}$ and
the channel SNR, we uniformly sample
$\rho_{l}\in\\{\rho_{1},\ldots,\rho_{L}\\}$ and
$\mathrm{SNR}\in[\mathrm{SNR}_{min},\mathrm{SNR}_{max}]$ and feed them along
with the image $\bm{S}$ to both the encoder and decoder during training. The
loss $\mathcal{L}_{l}$ for $\rho_{l}$ is evaluated using the mean square error
(MSE) between the input image and its reconstruction. For small $\rho_{l}$,
$\mathcal{L}_{l}$ is much larger compared with that of larger $\rho_{l}$. As
the losses for all bandwidth ratios are optimized together, the proposed
DeepJSCC-l++ tends to focus on the reconstruction performance for smaller
bandwidth ratios with significantly larger loss values. The reconstruction for
larger bandwidth ratios, however, becomes highly sub-optimal, which motivates
us to explore a better training methodology to improve the reconstruction
quality across all conditions.

Inspired by [14], we propose the DWA scheme, which assigns different weights
$w_{l}^{t}$ to the loss achieved for different $\rho_{l}$ values according to
their reconstruction qualities in the validation phase at the $t$-th epoch. To
be specific, we first obtain an average image reconstruction quality
$\text{PSNR}_{l}^{t}$ as defined in (1) for each bandwidth ratio $\rho_{l}$ in
the $t$-th epoch, where the images from the validation dataset along with
$\rho_{l}$ and $\mathrm{SNR}_{val}$ are fed to the DeepJSCC-l++ model. Without
loss of generality, we set the
$\mathrm{SNR}_{val}=(\mathrm{SNR}_{min}+\mathrm{SNR}_{max})/2$. To evaluate
the relative reconstruction performance for each $\rho_{l}$, we compare
$\text{PSNR}_{l}^{t}$ with the PNSR upper bound for that specific bandwidth
ratio $\rho_{l}$, denoted by $\text{PSNR}_{l}^{*}$, which is obtained by
training a non-adaptive model with fixed $\mathrm{SNR}_{val}$ and $\rho_{l}$.
Intuitively, a larger gap from $\text{PSNR}_{l}^{*}$, requires a larger weight
$w_{l}^{t}$ assigned to that bandwidth ratio $\rho_{l}$. Based on this
intuition, we introduce a policy to dynamically assign weights to the losses,
$\mathcal{L}_{l}^{t}$, for different bandwidth ratios:

$\displaystyle\Delta_{l}^{t}=\text{PSNR}_{l}^{*}-\text{PNSR}_{l}^{t},$
$\displaystyle w_{l}^{t}$
$\displaystyle=\text{clip}(2^{\alpha(\Delta_{l}^{t}-\beta)}\\!-\\!1,\>0,\Gamma),$
(5)

where $\alpha$ is set to be a positive number to ensure that a larger weight,
$w_{l}^{t}$, is assigned to the bandwidth ratio $\rho_{l}$ with a larger
$\Delta_{l}^{t}$. Since the reconstruction quality of the adaptive model is
expected to be upper bounded by separately (non-adaptive) trained models, we
would expect a PSNR gap of the adaptive model from these benchmarks. Thus, we
set a non-zero $\beta$, which allows the proposed DeepJSCC-l++ model to focus
on optimizing the reconstruction performance for other bandwidth ratios
$\rho_{i},i\neq l$ if $\Delta_{l}^{t}$ is smaller than $\beta$. Note that,
even when $\Delta_{l}^{t}<\beta$, a non-negative weight is required. Thus, we
clip the weights to have a minimum value $0$ and a maximum value $\Gamma$ to
ensure stable training. Empirically, we found that setting
$\alpha=2,\beta=0.25,\Gamma=10$ yields reasonable performance and we use these
values throughout the paper. Fig. 3 plots the function in (5) for the settings
given above. We can observe the weight assigned to a certain bandwidth ratio
decays smoothly with the reduction of $\Delta_{l}^{t}$ during the training
process. The overall training process with the DWA training methodology for
DeepJSCC-l++ is summarized in Algorithm 1.

##  IV Numerical Experiments

Figure 3: We plot the function in (5) for the parameters
$\alpha=2,\beta=0.25,\Gamma=10$.

Next, we evaluate the performance of the proposed DeepJSCC-l++ scheme for both
the adaptive-bandwidth and successive refinement scenarios, along with the
non-adaptive models trained at a fixed bandwidth ratio and channel SNR444Code
available at https://github.com/aprilbian/deepjscc-lplusplus..

###  IV-A Parameter Setting and Training Details

We consider the transmission of images from the CIFAR-10 dataset, which
includes 40000 training, 10000 validation, and 10000 test RGB images, each
with $32\times 32$ resolution. Both the encoder and decoder employ $I=2$
stages, the number of features $c$ is set to 256, the window size to $w=8$,
the numbers of Swin transformer blocks in each stage to $M_{1}=4,M_{2}=2$. The
dimension of the embedding is set to $n_{u}=2$. Finally, the GeLU activation
function is used within the Swin transformer blocks.

For training, we adopt the Adam optimizer and a varying learning rate
initialized at $10^{-4}$, which is reduced by a factor of $0.95$ if the
validation loss does not drop for 20 epochs. The maximum number of epochs is
set to $4\times 10^{3}$, and the early stopping module is included, where the
training process terminates if the validation loss does not improve in $80$
epochs for the adaptive schemes, whereas the patience is set to $60$ for the
non-adaptive models. Note that the settings mentioned above ensure that the
validation losses are saturated for both adaptive and non-adaptive schemes
when the training ends.

Throughout this section, we assume a maximum bandwidth ratio of
$\rho_{max}=1/4$, which corresponds to a maximum number of tokens $N_{T}=64$
and maximum number of features per token $N_{F}=24$. We train and evaluate the
proposed schemes with channel SNR ranging from $4$ dB to $10$ dB.

###  IV-B Performance Evaluation

We first compare the reconstruction performance of the bandwidth-adaptive
model with varying patches and varying features. In this experiment, we set
the number of supported bandwidth ratios to $L=4$ with
$\rho_{l}\in\\{1/16,1/8,3/16,1/4\\}$. The varying patches scheme adopts a
fixed $N_{F}=24$ while its $n_{t}$ changes from $16$ to $64$ to adjust for
different bandwidth ratios, the varying features scheme, on the other hand,
fixes $N_{T}=64$ with a varying number of features $n_{f}=\\{6,12,18,24\\}$.
The PSNR performance of the two adaptive schemes as well as the separately
trained (non-adaptive) models under a fixed channel quality $\mathrm{SNR}=7$
dB are shown in Table I. Note that both adaptive schemes adopt the DWA
training methodology proposed in Section III-B. We observe similar PSNR values
for the two schemes, and both of them achieve comparable performance with the
separately trained models where a maximum of $0.24$ dB PSNR gap is observed at
$\rho_{L}=1/4$. We use the varying features scheme for the remaining
simulations.

Table I: Evaluation for the varying patches and varying features DeepJSCC-l++ schemes at $\mathrm{SNR}=7$ $dB$ in terms of PSNR ($dB$). $\rho$ | 1/16 | 1/8 | 3/16 | 1/4
---|---|---|---|---
varying patches | 26.12 | 30.01 | 32.53 | 34.32
varying features | 26.14 | 30.01 | 32.53 | 34.31
separate training | 26.36 | 30.23 | 32.70 | 34.55

Figure 4: We show the effectiveness of the proposed DeepJSCC-l++ models: (a)
the PSNR performance over different $\rho_{l}$ with $\mathrm{SNR}=7$ dB; (b)
the PSNR performance over different evaluation SNRs with $\rho_{l}=1/6$; (c)
the weights $w_{l}^{t}$ for different bandwidth ratios $\rho_{l}$ with respect
to the number of epochs $t$.

Next we consider a more challenging case where we train the adaptive models
with $L=6$ supported bandwidth ratios and varying channel SNRs. The possible
bandwidth ratios are $\rho_{l}\in\\{1/24,1/12,1/8,1/6,5/24,1/4\\}$, which
correspond to $n_{f}=\\{4,8,12,16,20,24\\}$. We show the PSNR performance of
the proposed DeepJSCC-l++ applied to both bandwidth-adaptive and successive
refinement scenarios introduced in Section II (with and without DWA), along
with the DeepJSCC-l model originally proposed in [8], which follows the
successive refinement principle but adopts a CNN as its backbone.

As can be seen in Fig. 4 (a) and (b), where we fix the channel SNR to $7$ dB,
the proposed DeepJSCC-l++ model with DWA can be adaptive to different
bandwidth ratios with a negligible performance gap from the separately trained
(non-adaptive) benchmarks. Specifically, we observe that for the proposed
DeepJSCC-l++ models, the gap is more significant at larger $\rho_{l}$ values,
which is intuitive as the optimization task for reconstructing the image from
a higher dimensional latent is in general more challenging555This can also be
verified by checking the training curves for different $\rho_{l}$ values..
Further, it is shown that the DeepJSCC-l++ under successive refinement
scenario with DWA generates a reconstruction performance with a modest PSNR
gap of $\approx 0.5$ dB compared to the adaptive-bandwidth scenarios at high
$\rho_{l}$ values. This is due to the fact that the successive refinement
scheme imposes more stringent constraints as illustrated in Section II. The
DeepJSCC-l++ without the proposed training methodology, on the other hand, is
optimized with overwhelming focus on the performance of the smallest bandwidth
ratio, $\rho_{1}=1/24$, yielding $\Delta_{1}^{T}\approx 0$ dB where $T$
denotes the epoch where the training terminates. However, the reconstruction
performances at higher $\rho_{l},l\\!>\\!1$, are far from optimal, which
highlights the effectiveness of the proposed DWA training methodology. The
curve named ‘Succ. refn. scheme of [8]’ in Fig. 4 (a) represents the results
obtained in [8] for successive refinement, which has fallen short of the
performance of the DeepJSCC-l++ scheme which is mainly due to the less
powerful CNN backbone adopted in [8].

We then demonstrate that the proposed DeepJSCC-l++ model is also adaptive to
different channel SNRs. In this simulation, the settings are the same as those
in Fig. 4 (a). The bandwidth ratio $\rho_{l}$ is fixed at $1/6$ while the
evaluation $\mathrm{SNR}$ varies from $5$ dB to $9$ dB. As shown in Fig. 4
(b), the PSNR performance of the DeepJSCC-l++ models are compared with the
separately trained models. A fixed $\approx 0.25$ dB PSNR gap from the
separately trained models is observed for the DeepJSCC-l++ model, which
corresponds to the $\Delta_{l}^{T}$ value introduced in Section III-B and also
shown in Fig. 4 (a). All these schemes show significant gain compared with the
digital baseline which utilizes the BPG image compression and capacity
achieving code. Combining the results in Fig. 4 (a) and (b), we confirm the
proposed schemes are adaptive to different bandwidth ratios and channel SNRs.

Finally, for a comprehensive understanding of the DWA training methodology for
the DeepJSCC-l++, we present the weights $w_{l}^{t}$ versus the number of
epochs $t$ for different bandwidth ratios $\rho_{l}$’s in Fig. 4 (c). Note
that the settings for this simulation is identical to that in Fig. 4 (a). As
analyzed before, the reconstruction task for larger $\rho_{l}$ is generally
harder, thus for a fixed epoch $t$, the corresponding $\Delta_{l}^{t}$ is
larger leading to a larger $w_{l}^{t}$ which is verified in the figure. As the
reconstruction performance improves with more epochs, some of the
$w_{l}^{t}$’s reduce to 0 if $\Delta_{l}^{t}\leq\beta=0.25$. Since a PSNR gap
greater than $0.25$ dB remains when the training finishes, we can still
observe a non-zero weight for the $\rho_{L}=1/4$ case.

##  V Conclusion

We developed a flexible wireless image transmission scheme, called
DeepJSCC-l++, which can adapt to multiple bandwidth ratios and channel SNRs
using a single encoder/decoder neural network pair built upon the Swin
transformer architecture. Numerical simulations verify that the proposed
DeepJSCC-l++ model can be adaptive to both the bandwidth ratio and the channel
SNR with marginal performance loss with respect to the separately trained
models. It is also shown that the proposed architecture can be applied to the
successive refinement problem, outperforming the state-of-the-art by a
significant margin. It also outperforms the separation-based baseline
considering BPG compression with a capacity-achieving channel code, which
provides an upper bound on the performance achievable by a separation-based
scheme employing BPG for compression. These results not only show the superior
performance of DeepJSCC, but also highlight its potential for practical
systems through the use of a single pair of encoder/decoder parameters.

## References

  * [1] E. Bourtsoulatze, D. B. Kurka, and D. Gündüz, “Deep joint source-channel coding for wireless image transmission,” _IEEE Trans. Cognitive Commun. Netw._ , vol. 5, no. 3, pp. 567–579, 2019.
  * [2] M. Yang, C. Bian, and H.-S. Kim, “OFDM-guided deep joint source channel coding for wireless multipath fading channels,” _IEEE Trans. on Cogn. Comm. and Networking_ , vol. 8, no. 2, pp. 584–599, 2022.
  * [3] H. Wu, Y. Shao, K. Mikolajczyk, and D. Gündüz, “Channel-adaptive wireless image transmission with OFDM,” _IEEE Wireless Communications Letters_ , vol. 11, no. 11, pp. 2400–2404, 2022.
  * [4] H. Wu, Y. Shao, C. Bian, K. Mikolajczyk, and D. Gündüz, “Vision transformer for adaptive image transmission over MIMO channels,” in _IEEE International Conference on Communications (ICC)_ , 2023.
  * [5] C. Bian, Y. Shao, H. Wu, and D. Gunduz, “Space-time design for deep joint source channel coding of images over MIMO channels,” 2022. [Online]. Available: https://arxiv.org/abs/2210.16985
  * [6] C. Bian, Y. Shao, H. Wu, and D. Gunduz, “Deep joint source-channel coding over cooperative relay networks,” 2022. [Online]. Available: https://arxiv.org/abs/2211.06705
  * [7] J. Xu, B. Ai, W. Chen, A. Yang, P. Sun, and M. Rodrigues, “Wireless image transmission using deep source channel coding with attention modules,” _IEEE Trans. Circuits Syst. Video Technol_ , vol. 32, no. 4, pp. 2315–2328, 2021.
  * [8] D. B. Kurka and D. Gündüz, “Bandwidth-agile image transmission with deep joint source-channel coding,” _IEEE Trans. on Wireless Comm._ , vol. 20, no. 12, pp. 8081–8095, 2021.
  * [9] M. Yang and H.-S. Kim, “Deep joint source-channel coding for wireless image transmission with adaptive rate control,” in _ICASSP_ , 2022.
  * [10] W. Zhang, H. Zhang, H. Ma, H. Shao, N. Wang, and V. C. M. Leung, “Predictive and adaptive deep coding for wireless image transmission in semantic communication,” _IEEE Trans. on Wireless Comm._ , 2023.
  * [11] J. Dai, S. Wang, K. Tan, Z. Si, X. Qin, K. Niu, and P. Zhang, “Nonlinear transform source-channel coding for semantic communications,” _IEEE J. on Sel. Areas in Comm._ , vol. 40, no. 8, pp. 2300–2316, 2022.
  * [12] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _ICCV_ , October 2021, pp. 10 012–10 022.
  * [13] D. Alexey, B. Lucas, K. Alexander, W. Dirk, Z. Xiaohua, U. Thomas, D. Mostafa, M. Matthias, H. Georg, G. Sylvain, U. Jakob, and H. Neil, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _ICLR_ , 2021\.
  * [14] O. Rippel, A. G. Anderson, K. Tatwawadi, S. Nair, C. Lytle, and L. Bourdev, “Elf-vc: Efficient learned flexible-rate video coding,” in _ICCV_ , October 2021.


Omm$#2\text{\,}#\mathrm{3}$ Om$#\mathrm{2}$

# FastSurf: Fast Neural RGB-D Surface Reconstruction using Per-Frame Intrinsic
Refinement and TSDF Fusion Prior Learning

Seunghwan Lee Gwanmo Park Hyewon Son Jiwon Ryu Han Joo Chae
ROKIT Healthcare, Inc.
{seunghwan.lee, gwanmo.park, hyewon.son, jiwon.ryu<EMAIL_ADDRESS>
###### Abstract

We introduce FastSurf, an accelerated neural radiance field (NeRF) framework
that incorporates depth information for 3D reconstruction. A dense feature
grid and shallow multi-layer perceptron are used for fast and accurate surface
optimization of the entire scene. Our per-frame intrinsic refinement scheme
corrects the frame-specific errors that cannot be handled by global
optimization. Furthermore, FastSurf utilizes a classical real-time 3D surface
reconstruction method, the truncated signed distance field (TSDF) Fusion, as
prior knowledge to pretrain the feature grid to accelerate the training. The
quantitative and qualitative experiments comparing the performances of
FastSurf against prior work indicate that our method is capable of quickly and
accurately reconstructing a scene with high-frequency details. We also
demonstrate the effectiveness of our per-frame intrinsic refinement and TSDF
Fusion prior learning techniques via an ablation study.

Figure 1: Our method proposes per-frame intrinsic refinement and classical
TSDF Fusion [10] prior learning schemes for fast and high-quality 3D surface
reconstruction. We adopt the Neural RGB-D method [2], revised with a hybrid
representation through a dense feature grid and shallow MLPs. Our per-frame
intrinsic refinement scheme compensates for the frame-specific distortion
effects caused by the camera motion. The first phase of the training learns
geometric prior using the TSDF Fusion algorithm and the later phases adopt a
progressive learning technique [38].

##  1 Introduction

Most modern mobile devices are now capable of capturing RGB-D images and
videos without external sensors, and widely accessible RGB-D media expanded
the use of 3D geometric information in various fields from augmented reality
(AR) applications [5, 6, 16, 46] and 3D video streaming [41] to medical
surgeries [7, 48, 19]. While fast and efficient classical 3D reconstruction
methods [12, 27] based on the truncated signed distance field (TSDF) Fusion
[10] were able to reflect surface geometry in real-time, inaccuracies in the
reconstructed results due to the limitations of physical sensors hampered
their use in the real-world applications where precision is critical.
Recently, a neural RGB-D surface reconstruction method [2] incorporated a
depth-measurement-based implicit surface representation into the volume
rendering scheme of neural radiance field (NeRF) [24] and achieved high-
quality geometry estimates of the entire scenes.

However, like most variants of NeRF, Neural RGB-D [2] requires a long time to
optimize for each newly given scene; the study reports to take 9 to 13 hours
to optimize the networks depending on the size of the scene. While fast and
precise 3D surface reconstruction is necessary for many real-world
applications, methods proposed for quicker convergence of NeRF-style
approaches mostly utilized the volume density representation focusing more on
rendering unseen views rather than extracting 3D surface meshes [49, 26, 4,
52, 38, 25, 39], or sacrificed high-frequency details to achieve real-time
speed [30, 37].

One of the most recent studies [42] that adopted a hybrid volume rendering
scheme specifically for 3D surface reconstruction purposes showed impressive
speed-up in convergence, but it was achieved at the cost of losing high-
frequency details. Moreover, although some works [2] addressed the errors in
camera poses and view-dependent color effects, the innate problems of the
RGB-D video taken from the consumer-level cameras (_i.e_., motion blur and
rolling shutter problem) have not been considered.

In this work, we introduce _FastSurf_ 111 https://rokit-
healthcare.github.io/FastSurf , an accelerated NeRF framework that
incorporates depth information for 3D reconstruction. A dense feature grid and
shallow multi-layer perceptron are used for fast and accurate surface
reconstruction of the entire scene. Our per-frame intrinsic refinement scheme
corrects the frame-specific errors that cannot be handled by global
optimization. Specifically, four explicit representations (two for scaling and
the other two for translation) are added when casting rays to handle the
camera motion problems of RGB-D videos with a marginal effect on the training
speed. Furthermore, FastSurf utilizes a classical real-time 3D surface
reconstruction method, TSDF Fusion [10], as prior knowledge to pretrain the
feature grid to give the model a head start when training. FastSurf consists
of three optimization phases; it first leverages TSDF Fusion [10] to pretrain
the feature grid and then learns low- and high-frequency implicit surfaces
using coarse and fine feature grids, respectively. By comparing the
performance against prior work (_i.e_., Neural RGB-D [2] and GO-Surf [42]), we
show how FastSurf quickly and accurately reconstructs a scene with high-
frequency details. Moreover, we demonstrate the effectiveness of the TSDF
Fusion prior learning and per-frame intrinsic refinement techniques via an
ablation study.

##  2 Related work

Our method reconstructs 3D scene representation from a sequence of RGB-D video
frames. It relates to the previous works on TSDF Fusion-based multi-view 3D
reconstruction [10, 27, 12], coordinate-based implicit neural representations
[24, 43, 47, 29, 37, 30, 2], and optimization speedup techniques for the
neural models [38, 25, 42].

###  2.1 Classical 3D reconstruction

A wide range of methods was researched for multi-view 3D geometry
reconstruction including stereo matching and SLAM-based techniques [18, 35,
36, 14, 15, 17, 34, 50].

When reconstructing 3D surfaces from depth images, variations of TSDF Fusion
have been the most commonly used method [53], where the surfaces were
implicitly represented through TSDF in a uniform 3D grid [10]. Various
improvements to the method have been implemented over the years including
real-time application [27], efficient memory usage via spatial hashing [28],
and better reconstruction quality through camera pose estimation [12].

While the methods based on TSDF Fusion were fast and efficient, they had clear
limitations, often exhibiting holes or geometric artifacts in the
reconstruction results. Nevertheless, we found that using TSDF Fusion output
as the target while pretraining our voxel feature grid and MLPs accelerates
the convergence and improves the overall optimization time as well as the
reconstruction quality.

###  2.2 Neural radiance field and depth

Ever since Mildenhall _et al_. proposed NeRF [24], various attempts at
adapting the neural radiance field representation and the volume rendering
scheme to the depth images have been made. Some incorporated depth priors for
better novel view synthesis [8, 44, 13, 26], while others used implicit neural
representations for 3D surface reconstruction [43, 47, 29, 37, 30, 32, 9].

Among the methods for 3D reconstruction from RGB-D images, using a hybrid
representation with both TSDF and volumetric radiance fields to incorporate
the signed distance values into the classical volume rendering scheme saw the
most success. Neural RGB-D [2] used two separate MLP networks to reconstruct
3D scenes with great accuracy, demonstrating the effectiveness of mixing
photometric and depth loss terms. It additionally utilized pose optimization
and deformation field correction techniques to further improve the quality.
However, it takes a long time (9 to 13 hours) to optimize the networks, making
it impractical for any time-sensitive applications.

###  2.3 Fast neural radiance field

As the long optimization time has been the critical weakness of NeRF and its
variants, numerous methods for quicker convergence have been proposed [49, 26,
30, 37, 4, 52, 38, 25, 39]. In particular, DVGO [38] used a learnable feature
grid to reduce the size of MLP network and thus decrease the training time,
and Instant-NGP [25] employed multi-resolution hash encoding table for more
memory- and computationally-efficient approach.

For reconstructing 3D surfaces, iMAP [37] and iSDF[30] had shown that
representing implicit surfaces through MLP could be done in real-time.
However, they relied on keyframe selection and active sampling, sacrificing
much of the details. GO-Surf [42] combined a multi-resolution feature grid
with a hybrid volume rendering scheme akin to Neural RGB-D for much faster
optimization speed which took dozens of minutes while having decent accuracy.
It additionally used a smoothness prior which helped with quick convergence at
the cost of high-frequency details.

Taking a similar approach to GO-Surf, we incorporate a dense feature grid and
shallow MLPs to focus on recovering high-frequency details while preserving
the speedup. We additionally introduce a per-frame intrinsic refinement scheme
to fix the previously uncaught errors and a TSDF Fusion-guided pretraining
phase for faster optimization, outperforming GO-Surf as well as the more
accurate Neural RGB-D in terms of both reconstruction speed and quality.

#### Concurrent work

Voxurf [45] and Vox-Surf [21] are similar to our work in that they both used
voxel-based implicit surface representation and SDF-incorporated volume
rendering schemes to accelerate 3D reconstruction. However, they did not
utilize rich surface information that can be obtained from the depth channel
as they mainly focus on using only color information. In addition, our work is
primarily different as it corrects the camera intrinsic matrices frame-by-
frame and leverages the classical TSDF Fusion as prior knowledge.

##  3 Method

We propose a fast neural RGB-D surface reconstruction method using a dense
feature grid and shallow MLPs, with per-frame intrinsic refinement and TSDF
Fusion-guided pretraining schemes to obtain high-fidelity results without
sacrificing the speed. Similar to the previous RGB-D surface reconstruction
methods [2, 42], we leverage the depth and corresponding color information
from RGB-D video frames $\nu=\\{\nu_{i}\mid
i\in\mathbb{N},\,i\leq\textit{\\#}frames\\}$ to optimize our coordinate-based
networks. As depicted in Fig. 1, our explicit-implicit hybrid geometry
representation optimizes the volumetric radiance field and implicit surface
based on TSDF in a continuous region of interest. Our three-phase training
process leverages the TSDF Fusion algorithm and the progressive learning
technique for efficient optimization.

###  3.1 Hybrid geometry representation

We adopt a dense feature grid to explicitly learn local features in grid
cells. This explicit representation lowers the computational complexity
compared to an MLP, significantly reducing the training time. We dynamically
build the feature grid per scene:

$V^{(feat)}:(N_{x}\times N_{y}\times N_{z})\mapsto\mathbb{R}^{F},$ (1)

where $F$ is a fixed hyperparameter for feature vector length and $N_{x}$,
$N_{y}$, $N_{z}$ are the feature grid dimensions varied according to the scene
size. We use the depth frames and the estimated camera poses to calculate the
minimum bounding box size of the scene—$L_{x}$, $L_{y}$, $L_{z}$. With the
grid cell size $gs$, each of $N_{x}$, $N_{y}$, $N_{z}$ is set to
$\lceil{L_{x}/gs}\rceil,\lceil{L_{y}/gs}\rceil,\lceil{L_{z}/gs}\rceil$
respectively. The feature vector $\mathbf{\phi}_{\mathbf{p}}$ for a 3D point
$\mathbf{p}$ is simply the trilinear interpolation of the feature vectors of
the nearest 8 grid vertices $\mathbf{P}_{near}$.

$\mathbf{\phi}_{\mathbf{p}}=interp[V^{(feat)}(\mathbf{P}_{near})]\in\mathbb{R}^{F}$
(2)

With the dense grid representation efficiently encoding local geometric
features, we use shallow MLPs to decode the features into view-dependant color
$C_{\mathbf{p},\mathbf{d}}^{\,i}$ and truncated signed distance value
$\widehat{D}_{\mathbf{p}}$ of a 3D point $\mathbf{p}$. In order to decode the
view-dependant color information, positional-encoded ray direction
$\mathbf{d}$ as in [24] and view-dependent latent embedding vector
$\mathbf{\xi}$ introduced in [23, 2] are concatenated with the feature vector
before inputting to the color MLP. As described in Eqs. 3 and 4, two separate
MLPs were used for color and signed distance value, respectively.

$C_{\mathbf{p},\mathbf{d}}^{\,i}=MLP^{C}(\mathbf{\phi}_{\mathbf{p}},\Lambda(\mathbf{d}),\mathbf{\xi}_{i}),$
(3) $\widehat{D}_{\mathbf{p}}=MLP^{D}(\mathbf{\phi}_{\mathbf{p}}),$ (4)

where $i$ is the frame number and $\Lambda$ is the frequency positional
encoding function.

The neural rendering process is designed in a similar way as in [2]. Given a
known camera intrinsic matrix, the camera ray $\mathbf{r}$ for an image pixel
$(u,v)$ is cast along its casting direction $\mathbf{d}$:

$\mathbf{d}_{u,v}=\frac{\mathbf{\varrho}_{u,v}}{|\mathbf{\varrho}_{u,v}|}$ (5)
$\mathbf{\varrho}_{u,v}=\begin{bmatrix}{1}/{f_{x}}&0&{-c_{x}}/{f_{x}}\\\
0&{-1}/{f_{y}}&{c_{y}}/{f_{y}}\\\ 0&0&-1\end{bmatrix}\begin{bmatrix}u\\\ v\\\
1\end{bmatrix}$ (6)

where $f_{x},f_{y}$ are focal lengths and $(c_{x},c_{y})$ is the principal
point. For a 3D point $\mathbf{p}$, its weight value $\omega_{\mathbf{p}}$ for
rendering the color image is calculated from the signed distance value:

$\omega_{\mathbf{p}}=\sigma(\frac{\widehat{D}_{\mathbf{p}}}{tr})\cdot\sigma(-\frac{\widehat{D}_{\mathbf{p}}}{tr}),$
(7)

where $tr$ is the truncation distance and $\sigma$ is the sigmoid function.
Finally, the rendered color $\widehat{C}$ for a ray $\mathbf{r}$ with
direction $\mathbf{d}$ of the $i^{th}$ frame is the weighted sum of the
radiance values on the sampled points $\mathbf{p}$ along the ray:

$\widehat{C}_{\mathbf{r}}^{\,i}=\frac{1}{\sum\omega_{\mathbf{p}}}\sum\omega_{\mathbf{p}}C_{\mathbf{p},\mathbf{d}}^{\,i}$
(8)

###  3.2 Per-frame intrinsic refinement

Often, the camera intrinsic matrix is assumed to be the same across all
frames. However, unlike still images, video frames are vulnerable to camera
motion, resulting in motion blurs or rolling shutter effects (Fig. 2). As
these effects can be handled by transformations in the image-plane coordinate
[33, 3, 31], FastSurf is designed to correct the camera intrinsic matrix,
which effectively changes the scale and translation of the projected image
plane. The level of distortion varies from frame to frame and thus, having
learnable features that can quickly correct the distortion per frame is
critical.

(a)

(b)

(c)

Figure 2: Samples from the ScanNet V2 [11] dataset demonstrate the negative
impact of motion blurs [3] and rolling shutter effects [33]. The RGB frames
(a, b) are blurry and distorted. The depth frames (c) likewise contain
distorted object shapes.

Before handling this per-frame intrinsic refinement issue, we first adopt the
image-plane deformation field and pose optimization techniques from [2] to
correct the camera-pose errors and the potential global errors from the
intrinsic parameters as well as the camera lens distortion. We modify the
image-plane deformation field to use a two-layered shallow MLP to reduce the
training time.

Our per-frame intrinsic refinement scheme is performed in between the image-
plane deformation field and pose optimization to correct the frame-specific
errors unhandled by the global optimizations. Specifically, we introduce four
parameters per frame, two for scaling and the other two for translation
purposes. Scaling and translation are applied to the normalized image
coordinate before it is transformed into the world coordinate:

$\displaystyle\mathbf{\widehat{\varrho}}_{u,v}^{\,i}=\mathbf{s}^{i}\cdot(\mathbf{\varrho}_{u,v}+\mathbf{\tau}^{i})$
(9) $\displaystyle\mathbf{s}^{i}=\begin{bmatrix}s_{x}^{i}&0&0\\\
0&s_{y}^{i}&0\\\
0&0&1\end{bmatrix},\,\,\mathbf{\tau}^{i}=\begin{bmatrix}\tau_{x}^{i}\\\
\tau_{y}^{i}\\\ 0\end{bmatrix}$ (10)

where $\mathbf{s}^{i}$ and $\mathbf{\tau}^{i}$ are trainable parameters for
$i^{th}$ frame and optimized during the training. Accordingly, the refined
casting direction $\widehat{\mathbf{d}}$ is calculated from
$\mathbf{\widehat{\varrho}}$:

$\widehat{\mathbf{d}}_{u,v}^{\,i}=\frac{\widehat{\mathbf{\varrho}}_{u,v}^{\,i}}{|\widehat{\mathbf{\varrho}}_{u,v}^{\,i}|}$
(11)

Eq. 11 replaces Eq. 5 for generating rays and sampling points. These
parameters can be interpreted as correcting the principal point and focal
lengths of the intrinsic matrix.

###  3.3 Optimization

FastSurf is optimized through a three-phase training process as depicted in
Fig. 1. In the first phase, FastSurf quickly learns geometric prior using the
classical real-time algorithm, TSDF Fusion [10]. Specifically, FastSurf builds
a dense grid described in Eq. 1 with $F$=1 and runs TSDF Fusion with the
identical scene and grid sizes. We randomly sample grid cells, query the
center points, and compare them with the corresponding points from the results
of TSDF Fusion using mean squared error:

$\mathcal{L}_{pre}=\frac{1}{|\mathbf{P}_{b}|}\sum_{\mathbf{p}\in\mathbf{P}_{b}}(\widehat{D}_{\mathbf{p}}-D^{f}_{\mathbf{p}})^{2},$
(12)

where $\mathbf{P}_{b}$ is the set of sampled center points and
$D^{f}_{\mathbf{p}}$ is the signed distance value from the TSDF Fusion result.
Only $V^{(feat)}$ and $MLP^{D}$ are trained during this phase, while the other
parameters are fixed at their initial values.

For the second and third phases, FastSurf adopts the progressive learning
technique [38, 49] to successively learn low- and high-frequency details of
the scene step by step. All the parameters including those that were fixed
during the first phase are optimized during these phases.

During the second phase, FastSurf is trained with randomly sampled ray batch
$r_{b}$ from the input frames. Inspired by [2], we sample points $S_{c}$ from
the rays for every 15.625mm within the ray length where the ray length is
varied depending on the scene, ranging from 4m to 10m, to cover the entire
scene. Using the estimated signed distance values for each sampled point and
the rendered color value as described in Eqs. 4 and 8, our loss function is
defined similarly to [2]:

$\mathcal{L}=\lambda_{fs}\mathcal{L}_{fs}+\lambda_{sdf}\mathcal{L}_{sdf}+\lambda_{rgb}\mathcal{L}_{rgb}+\lambda_{reg}\mathcal{L}_{reg}$
(13)

$\mathcal{L}_{fs}$ denotes the ‘free space’ loss that forces the sample points
$\mathbf{p}\in S_{fs}\subset S_{c}$ that lie outside the truncated area to
have the signed distance value of $tr$:

$\mathcal{L}_{fs}=\frac{1}{|r_{b}|}\sum_{r\in
r_{b}}\frac{1}{|S_{fs}|}\sum_{\mathbf{p}\in
S_{fs}}(\widehat{D}_{\mathbf{p}}-tr)^{2}$ (14)

$\mathcal{L}_{sdf}$ compares the predicted signed distance value of the sample
points $\mathbf{p}\in S_{sdf}\subset S_{c}$ lying within the truncated area
against $D_{\mathbf{r}}^{i}$, the signed distance value observed in the depth
frame:

$\mathcal{L}_{sdf}=\frac{1}{|r_{b}|}\sum_{r\in
r_{b}}\frac{1}{|S_{sdf}|}\sum_{\mathbf{p}\in
S_{sdf}}(\widehat{D}_{\mathbf{p}}-D_{\mathbf{r}}^{i})^{2}$ (15)

$\mathcal{L}_{rgb}$ measures the difference between the predicted rendered
color of the rays and the observed color of the corresponding pixels:

$\mathcal{L}_{rgb}=\frac{1}{|r_{b}|}\sum_{r\in
r_{b}}(\widehat{C}_{\mathbf{r}}^{\,i}-C_{u,v}^{\,i})^{2}$ (16)

$\mathcal{L}_{reg}$ is the regularization term and consists of three parts:

$\displaystyle\mathcal{L}_{reg}$
$\displaystyle=\mathcal{R}_{\text{embed}}+\mathcal{R}_{\text{refine}}+\mathcal{R}_{\text{deform}}\,,$
(17) $\displaystyle\mathcal{R}_{\text{embed}}$
$\displaystyle=\frac{1}{|\nu|}\sum_{\nu_{i}\in\nu}\xi_{\nu_{i}}^{2}\,,$
$\displaystyle\mathcal{R}_{\text{refine}}$
$\displaystyle=\frac{1}{|r_{b}|}\sum_{r\in
r_{b}}((s_{r}-1)^{2}+\tau_{r}^{2})\,,$
$\displaystyle\mathcal{R}_{\text{deform}}$
$\displaystyle=\frac{1}{|r_{b}|}\sum_{r\in r_{b}}w_{ipdf}^{2}$

where $w_{ipdf}$ are the weights for the image-plane deformation field
network, and $s_{r}$ and $\tau_{r}$ denote the transform parameters (Section
3.2) of the frame to which the ray $r$ belongs.

FastSurf learns the fine details through the third phase, in which the dense
feature grid is divided into higher resolutions so that the feature grid
dimensions $N_{x},N_{y},N_{z}$ become
$\lceil{L_{x}/(gs/2)}\rceil,\lceil{L_{y}/(gs/2)}\rceil,\lceil{L_{z}/(gs/2)}\rceil$
respectively. We sample additional points $S_{f}$ in small neighborhoods
around the surfaces discovered from the previously sampled set of points
$S_{c}$. The third phase is trained with the same loss function, Eq. 13, using
both $S_{c}$ and $S_{f}$.

###  3.4 Implementation details

We implemented our network with PyTorch, and the network was optimized using
the ADAM optimizer [20] with $5\times 10^{-4}$ for learning rate, $0.9$ for
beta1, and $0.999$ for beta2. The learning rate was set to exponentially decay
so that it was decreased to a tenth for every 250K iterations. For the feature
grid, we chose $F$=12 and $gs$=10cm. Its weights were initialized to 0. The
signed distance values were truncated to $tr$=5cm. The MLP networks for the
color and truncated signed distance values both consisted of 2 hidden layers
with 128 nodes each. We conducted the positional encoding for the ray
direction vector in 4 frequency levels. The MLP for the image-plane
deformation field was built with 2 hidden layers with 64 nodes each. Per-frame
intrinsic refinement optimization parameters were initialized to 1 for scaling
and 0 for translation. Ray sampling number for $S_{f}$ was set to 16. The
$\lambda$ weights for the loss terms were set to 10 for $\lambda_{fs}$,
$6\times 10^{3}$ for $\lambda_{sdf}$, $0.5$ for $\lambda_{rgb}$, and $0.1$ for
$\lambda_{reg}$. We ran 3K, 7K, and 65K iterations for each training phase,
respectively. We used a CUDA implementation of the TSDF Fusion algorithm that
can process, on average, 231.7 frames per second on Tesla V100 GPU [51]. The
batch size of the grid cells in the first training phase was 1024. For the
rest of the training phases, we sampled 768 to 2048 rays in a batch depending
on the scene size.

##  4 Experiments

We demonstrate comparative studies of FastSurf against prior work. We designed
a qualitative study on the real-scene dataset to prove that FastSurf
outperforms its competitors in both the reconstruction quality and the
training time. Quantitative evaluation was conducted using 10 synthetic scenes
provided in [2] to show that our method achieves superior performance in
comparative metrics. Finally, we conducted an ablation study to prove the
effectiveness of the proposed framework components.

For the evaluation of each study, we extracted the truncated signed distance
values in $1\text{cm}^{3}$ resolution and ran MarchingCubes [22] algorithm to
get the triangular meshes.

###  4.1 Datasets

We used ScanNet V2 [11] as a real-scene dataset during the qualitative study.
ScanNet V2 contains RGB-D videos scanning 1,513 scenes from 707 unique indoor
landscapes. Each scene was scanned using an iOS device externally attached
with a Structure sensor. Since it used the rolling shutter method when
capturing the images, ScanNet V2 often contains motion blur and other
distortions as well as noisy depth measurements including holes and missing
thin or transparent objects.

To compare quantitative results against the baselines, we used 10 synthetic
scenes from [2]. They used indoor 3D models to photo-realistically render
color and depth frames with ground truth camera trajectories. Depth frames
were simulated with Kinect-like noises including holes and quantization
noises.

###  4.2 Baselines

We compare our results with Neural RGB-D [2] and GO-Surf [42]. Specifically,
we report our results at 20K and 75K iterations to respectively compare
against GO-Surf and Neural RGB-D to show that FastSurf outperforms in terms of
both efficiency and performance. We trained GO-Surf and Neural RGB-D with the
hyperparameters recommended by the respective papers on a Tesla V100 GPU.

(a) GO-Surf [42]

(b) Ours: 20K iterations

(c) Ours: 75K iterations

(d) Neural RGB-D [2]

Train time: 19m15s - 22m43s

11m08s - 14m30s

54m35s - 73m30s

452m10s - 546m50s

Figure 3:  We compare our method with GO-Surf [42] and Neural RGB-D [2] at
different points in time. The comparison was conducted using scenes 2, 5, 12,
and 50 from ScanNet V2 [11]. When trained for a shorter amount of time,
FastSurf-20K (b) recovers high-frequency details overlooked by GO-Surf (a) and
generates much less erroneous surfaces. Given a longer training time,
FastSurf-75K (c) achieves a quality on par with that of Neural RGB-D (d), but
at a significantly shorter amount of time ($\times$7.7 faster on average). It
also fixes the errors caused by the camera motion.

###  4.3 Results and discussion

#### Qualitative results

The qualitative results are illustrated in Fig. 3. For the comparison with GO-
Surf, our reconstruction results after 20K iterations—which took 11 to 15
minutes—is shown. The training time was less than that of GO-Surf (19-23
minutes) but FastSurf produced more fine details and fewer erroneous surfaces
across the scenes. After 75K iterations, our reconstruction qualities showed
better results than Neural RGB-D, recovering the structures Neural RGB-D
missed in some cases. Compared to Neural RGB-D, our training speed was 7.3 to
8.7 times faster.

#### Quantitative results

We compared our method with baselines in terms of Chamfer $\ell_{1}$ distance
(C-$\ell_{1}$), intersection-over-union (IoU), normal consistency (NC), and
F-score. In order to compute C-$\ell_{1}$, NC, and F-score, we sampled point
clouds from the output meshes in 1$\text{cm}^{2}$ resolution. We used a
threshold of 5cm for F-score. The evaluation meshes were voxelized with an
edge length of 10cm to compute the IoU. As shown in Table 1, our result after
20K iterations outperforms GO-Surf in C-$\ell_{1}$, IoU, and F-score with less
training time (21% faster on average). Our method took 94 minutes on average
to train 75K iterations, which was 7.15 times faster than what Neural RGB-D
took. At the same time, it showed superior C-$\ell_{1}$ and IoU scores and
comparable F-score.

While FastSurf showed outstanding performance on the three major measures, it
was less effective on the normal consistency, indicating that its results
contained relatively uneven surfaces. The quantitative metrics imply that our
method is best suited for quickly reconstructing complex geometries, rather
than simple flat surfaces.

Method |  C-$\ell_{1}$ $\downarrow$ |  IoU $\uparrow$ |  NC $\uparrow$ |  F-score $\uparrow$ | Time
---|---|---|---|---|---
GO-Surf [42] | 0.042 | 0.723 | 0.922 | 0.918 | 22m57s
Neural RGB-D [2] | 0.052 |  0.757 | 0.922 | 0.938 | 669m59s
Ours (20K) | 0.039 | 0.735 | 0.905 |  0.927 | 18m05s
Ours (75K) |  0.041 | 0.768 |  0.914 | 0.938 | 93m45s

Table 1: Quantitative results on the synthetic scene dataset. FastSurf shows
better C-$\ell_{1}$, IoU, and F-score than GO-Surf when trained for a shorter
amount of time. After training for more iterations, it achieves better
performances than Neural RGB-D with significantly less training time.

#### Ablation study

The ablation study shows the effects of per-frame intrinsic refinement and
TSDF Fusion-guided training phase. We compared results at the same iteration
points without the per-frame intrinsic refinement or the first phase of
training—TSDF Fusion prior learning (Fig. 4).

Our per-frame intrinsic refinement method fixes the erroneous parts of the
objects. Although the pose correction scheme was used in both baselines and
the refinement of camera parameters was used in Neural RGB-D, there still
exists reconstruction errors in some cases. In addition to the image-plane
deformation field which corrects the global parameters applied to all the
frames simultaneously, our per-frame intrinsic refinement further corrects the
frame-specific errors caused by camera motion. With only four additional
parameters per frame, our scheme allowed these parameters to be optimized in
the early stages of training, fixing the reconstruction errors in small
iterations while increasing the training time by only 2.7% on average.
Overall, the benefits of the per-frame intrinsic refinement outweigh the cost
of the increased training time.

Fast classical algorithms are useful sources of information to utilize as
prior knowledge of the scene despite their inaccuracies. The first phase of
our training process that uses this prior knowledge is extremely fast as the
signed distance values of 3D coordinates are directly learned instead of
casting rays for the rendering process. Subsequent training phases refine the
scene from this prior using the proposed optimization methods, correcting the
inaccuracies from the classical algorithms and ultimately requiring less time
to converge. As shown in Fig. 4, on all the scenes, the training time for the
same iteration took several more minutes without the TSDF Fusion prior
learning. TSDF Fusion prior learning phase also slightly improved the
reconstruction quality of the objects. The benefits of utilizing the prior
knowledge from TSDF Fusion can be maximized when fast optimization with a
small number of training iterations is required.

(a) Ours w/o PFIR 7s (TSDF) / 14m47s

(b) Ours w/o TSDF 17m06s

(c) Ours full 7s / 15m16s

15s / 34m05s

37m55s

15s / 34m51s

9s / 17m11s

19m21s

9s / 17m39s

9s / 16m42s

19m24s

9s / 16m66s

Figure 4: Ablation study. (a) Ours without per-frame intrinsic refinement
(PFIR). (b) Ours without TSDF Fusion prior learning in the first phase of
training (TSDF). (c) Ours with both methods applied. Timestamps below the
subfigures represent the TSDF Fusion prior learning time (if applicable) and
the total training time.

###  4.4 Limitation

Our main limitation comes from the use of the dense feature grid that requires
large amounts of memory and puts a limit to the available maximum scene size.
For example, in the evaluation dataset, 1.3GB of memory was required for the
scene size 18.0m $\times$ 16.8m $\times$ 12.3m. However, despite the memory
limitation, we leveraged the dense grid for efficient computation. It enabled
us to use lightweight decoding MLPs and achieve 7.3 to 8.7 times faster
convergence rate. Adopting other voxel representations introduced to handle
the memory inefficiency [40, 25, 28] may decrease the memory usage without
sacrificing the training speed or the performance.

Another limitation was the imbalance of ray density over the grid. A grid cell
is involved in the training only if a ray from the batch passes through it. In
other words, ray-abundant grid cells are prone to optimize better than ray-
sparse cells. Depending on how the video was taken, some cells may never be
learned in the training. In practice, one could scan more thoroughly around
the regions of interest to ensure completeness. Setting different learning
rates or loss weights as per ray density was also proposed in [38] which could
be applicable to our method as well.

Finally, we use simple affine transformation matrices to correct the
distortions from camera motion. While it proved to be effective at fixing
errors without increasing the computational burden, more complex models were
proposed to handle the motion blur or rolling shutter effects in the past [33,
1, 31]. Using more advanced or complicated modeling for the per-frame
intrinsic refinement might be beneficial, but it could possibly trade off the
speed with accuracy.

##  5 Conclusion

In this paper, we introduced FastSurf, an accelerated NeRF framework that
incorporates depth information for 3D reconstruction. A dense feature grid and
shallow MLPs were used for fast and accurate surface optimization of the
entire scene. Our per-frame intrinsic refinement scheme corrects the frame-
specific errors that cannot be handled by global optimization. Moreover,
FastSurf utilized TSDF Fusion, a classical real-time 3D surface reconstruction
method, as prior knowledge to pretrain the feature grid to accelerate the
training. The quantitative and qualitative experiments compared the
performances of FastSurf and prior work to show that our method is capable of
quickly and accurately reconstructing a scene with high-frequency details.
Specifically, FastSurf outperformed both GO-Surf [42] and Neural RGB-D [2] by
21% and 715% faster training time while having better reconstruction quality.
We also demonstrated the effectiveness of the per-frame intrinsic refinement
and TSDF Fusion prior learning techniques in the ablation study.

## References

  * [1] Cenek Albl, Zuzana Kukelova, Viktor Larsson, Michal Polic, Tomas Pajdla, and Konrad Schindler.  From two rolling shutters to one global shutter.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [2] Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies.  Neural rgb-d surface reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6290–6301, June 2022.
  * [3] Alastair Barber, Matthew Brown, Paul Hogbin, and Darren Cosker.  Inferring changes in intrinsic parameters from motion blur.  Computers & Graphics, 52:155–170, 2015.
  * [4] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields, 2021.
  * [5] Han Joo Chae, Youli Chang, Minji Kim, Gwanmo Park, and Jinwook Seo.  Arphy: Managing photo collections using physical objects in ar.  In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–7, 2020.
  * [6] Han Joo Chae, Jeong-in Hwang, and Jinwook Seo.  Wall-based space manipulation technique for efficient placement of distant objects in augmented reality.  In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, pages 45–52, 2018.
  * [7] Han Joo Chae, Seunghwan Lee, Hyewon Son, Seungyeob Han, and Taebin Lim.  Generating 3d bio-printable patches using wound segmentation and reconstruction to treat diabetic foot ulcers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2539–2549, June 2022.
  * [8] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo, 2021.
  * [9] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling, 2018.
  * [10] Brian Curless and Marc Levoy.  A volumetric method for building complex models from range images.  In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’96, page 303–312, New York, NY, USA, 1996. Association for Computing Machinery.
  * [11] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017\.
  * [12] Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt.  Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface re-integration, 2016.
  * [13] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised nerf: Fewer views and faster training for free, 2021\.
  * [14] Jakob Engel, Thomas Schöps, and Daniel Cremers.  Lsd-slam: Large-scale direct monocular slam.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 834–849, Cham, 2014. Springer International Publishing.
  * [15] Jakob Engel, Jürgen Sturm, and Daniel Cremers.  Semi-dense visual odometry for a monocular camera.  In 2013 IEEE International Conference on Computer Vision, pages 1449–1456, 2013.
  * [16] Sarah Faltaous, Gabriel Haas, Liliana Barrios, Andreas Seiderer, Sebastian Felix Rauh, Han Joo Chae, Stefan Schneegass, and Florian Alt.  Brainshare: A glimpse of social interaction for locked-in syndrome patients.  In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–6, 2019.
  * [17] Christian Forster, Matia Pizzoli, and Davide Scaramuzza.  Svo: Fast semi-direct monocular visual odometry.  In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 15–22, 2014.
  * [18] Michael Goesele, Noah Snavely, Brian Curless, Hugues Hoppe, and Steven Seitz.  Multi-view stereo for community photo collections.  pages 1–8, 11 2007.
  * [19] Hyunwoo Jo, Bo Young Choi, Giup Jang, Jung Pyo Lee, Ara Cho, Boyun Kim, Jeong Hwan Park, Jeonghwan Lee, Young Hoon Kim, and Jina Ryu.  Three-dimensional bio-printed autologous omentum patch ameliorates unilateral ureteral obstruction-induced renal fibrosis.  Tissue Engineering Part C: Methods, 28(12):672–682, 2022.
  * [20] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [21] Hai Li, Xingrui Yang, Hongjia Zhai, Yuqian Liu, Hujun Bao, and Guofeng Zhang.  Vox-surf: Voxel-based implicit surface representation, 2022.
  * [22] William E. Lorensen and Harvey E. Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’87, page 163–169, New York, NY, USA, 1987. Association for Computing Machinery.
  * [23] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7210–7219, June 2021.
  * [24] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis, 2020.
  * [25] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM Trans. Graph., 41(4):102:1–102:15, July 2022.
  * [26] T. Neff, P. Stadlbauer, M. Parger, A. Kurz, J. H. Mueller, C. R. A. Chaitanya, A. Kaplanyan, and M. Steinberger.  DONeRF: Towards real-time rendering of compact neural radiance fields using depth oracle networks.  Computer Graphics Forum, 40(4):45–59, jul 2021.
  * [27] Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon.  Kinectfusion: Real-time dense surface mapping and tracking.  In 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pages 127–136, 2011.
  * [28] Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger.  Real-time 3d reconstruction at scale using voxel hashing.  ACM Trans. Graph., 32(6), nov 2013.
  * [29] Michael Oechsle, Songyou Peng, and Andreas Geiger.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction, 2021.
  * [30] Joseph Ortiz, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoefer, and Mustafa Mukadam.  isdf: Real-time neural signed distance fields for robot perception, 2022\.
  * [31] Chandramouli Paramanand and Ambasamudram N. Rajagopalan.  Non-uniform motion deblurring for bilayer scenes.  In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pages 1115–1122, 2013.
  * [32] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger.  Convolutional occupancy networks, 2020.
  * [33] Vijay Rengarajan, Yogesh Balaji, and A. N. Rajagopalan.  Unrolling the shutter: Cnn to correct motion distortions.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2345–2353, 2017.
  * [34] Nikolay Savinov, Christian Häne, L’ubor Ladický, and Marc Pollefeys.  Semantic 3d reconstruction with continuous regularization and ray potentials using a visibility consistency constraint.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5460–5469, 2016.
  * [35] D. Scharstein, R. Szeliski, and R. Zabih.  A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.  In Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001), pages 131–140, 2001.
  * [36] Johannes L. Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4104–4113, 2016.
  * [37] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J. Davison.  imap: Implicit mapping and positioning in real-time, 2021.
  * [38] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction, 2021.
  * [39] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler.  Neural geometric level of detail: Real-time rendering with implicit 3d shapes, 2021.
  * [40] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler.  Neural geometric level of detail: Real-time rendering with implicit 3d shapes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11358–11367, June 2021.
  * [41] Telegie.  Telegie.  https://telegie.com/, 2023.  Accessed: 2023-02-17.
  * [42] Jingwen Wang, Tymoteusz Bleja, and Lourdes Agapito.  Go-surf: Neural feature grid optimization for fast, high-fidelity rgb-d surface reconstruction, 2022.
  * [43] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction, 2021.
  * [44] Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou.  Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo, 2021.
  * [45] Tong Wu, Jiaqi Wang, Xingang Pan, Xudong Xu, Christian Theobalt, Ziwei Liu, and Dahua Lin.  Voxurf: Voxel-based efficient and accurate neural surface reconstruction, 2022.
  * [46] Jianghao Xiong, En-Lin Hsiang, Ziqian He, Tao Zhan, and Shin-Tson Wu.  Augmented reality and virtual reality displays: emerging technologies and future perspectives.  Light: Science & Applications, 10(1):216, Oct 2021.
  * [47] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lipman.  Multiview neural surface reconstruction by disentangling geometry and appearance, 2020.
  * [48] Ahmet Ç Yastı, Ali E Akgun, Aziz A Surel, Jeehee Kim, and Merve Akın.  Graft of 3d bioprinted autologous minimally manipulated homologous adipose tissue for the treatment of diabetic foot ulcer.  Wounds: a Compendium of Clinical Research and Practice, 35(1):E22–E28, 2023.
  * [49] Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks, 2021.
  * [50] Christopher Zach, Thomas Pock, and Horst Bischof.  A globally optimal algorithm for robust tv-l1 range image integration.  In 2007 IEEE 11th International Conference on Computer Vision, pages 1–8, 2007.
  * [51] Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser.  3dmatch: Learning local geometric descriptors from rgb-d reconstructions.  In CVPR, 2017.
  * [52] Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu.  Nerfusion: Fusing radiance fields for large-scale scene reconstruction, 2022.
  * [53] Michael Zollhöfer, Patrick Stotko, Andreas Görlitz, Christian Theobalt, Matthias Nießner, Reinhard Klein, and Andreas Kolb.  State of the art on 3d reconstruction with rgb‐d cameras.  Computer Graphics Forum, 37:625–652, 05 2018.


# Image Generation with Multi-modal Priors using Denoising Diffusion
Probabilistic models ††thanks: Work in Progress

Nithin Gopalakrishnan Nair, Wele Gedara Chaminda Bandara
Student Member, IEEE
Johns Hopkins University
Baltimore
{ngopala2, wbandar1}jhu.edu
&Vishal M Patel
Senior Member, IEEE
Johns Hopkins University
Baltimore
<EMAIL_ADDRESS>

###### Abstract

Image synthesis under multi-modal priors is a useful and challenging task that
has received increasing attention in recent years. A major challenge in using
generative models to accomplish this task is the lack of paired data
containing all modalities (i.e. priors) and corresponding outputs. In recent
work, a variational auto-encoder (VAE) model was trained in a weakly
supervised manner to address this challenge. Since the generative power of
VAEs is usually limited, it is difficult for this method to synthesize images
belonging to complex distributions. To this end, we propose a solution based
on a denoising diffusion probabilistic models to synthesise images under
multi-model priors. Based on the fact that the distribution over each time
step in the diffusion model is Gaussian, in this work we show that there
exists a closed-form expression to the generate the image corresponds to the
given modalities. The proposed solution does not require explicit retraining
for all modalities and can leverage the outputs of individual modalities to
generate realistic images according to different constraints. We conduct
studies on two real-world datasets to demonstrate the effectiveness of our
approach.

##  1 Introduction

Precise user desired images could be generated using complementary information
describing the image obtained from multiple sources. Conditional image
generation has received much attention in recent years for its practical
utility in content creation tools[1]. As the number of conditions increases,
the task of image synthesis becomes more constrained, yet also happens
according to the user’s expectations. Hence, condtional image generation there
are practical utility in entertainment industry,surveillance and security,
etc. Consider the problem of face synthesis, some specific conditional
information, such as segmentation masks, can give a rough description of how
the generated face should be, and finer details can be added by conditioning
on specific text-based attributes. The task of generating images given
multiple constraints is called multi-modal image synthesis. Existing methods
use different generative models [2, 1, 3](such as VAE, GAN etc.) for
multimodal image synthesis.

One major challenge for training generative models for multi-modal image
synthesis is that the lack of paired data containing multiple modalities[4].
This is one of the primary reasons why most existing models constrain
themselves to one or two modalities[4, 3]. Few works perform multi-modal
generation with more than two modalities[1]. These methods are used to perform
high-resolution image synthesis and needs to be trained with paired data
across different domains to achieve good results.

A weakly supervised method for multi-modal generation has been recently
proposed that does not require paired data from all modalities[3], and the
model performs well when trained with sparse data. But if we need to increase
the number of modalities, the model needs to be retrained, hence not being
easily scalable. Scalable multi-modal generation is an area that has not been
properly explored due to the difficulty in obtaining the large amounts of
paired data needed to train models for the generative process. Modern multi-
modal generative methods use VAEs and GANs, which lack the ability to model
complex distributions.

Recently diffusion models thrived over other generative models for the task of
image generation[5]. This is due to the power of diffusion models to perform
exact sampling from very complex distributions[6]. One unique quality of
diffusion models over other generative processes is that the model performs
the generation through a Markovian process that happens through many
timesteps. The output at each timestep is easily accessible. Hence being more
flexible over other generative models and this form of generation allows
manipulation of images. Multiple techniques have used this interesting
property of diffusion models for low-level vision tasks like Image inpainting,
Image super-resolution, and other Image restoration problems.

In this paper, we exploit this flexible property of denoising diffusion
probabilistic models and use it to design solutions to multi-modal image
generation problems without explicitly retraining the network with paired data
from all modalities. Specifically, we develop a derivation for computing the
conditional score for the presence of multiple modes by using the conditional
score for each mode. The implications of the proposed multi-modal generative
method are manifold. We can easily extend multi-modal generation with
complementary information across datasets using the proposed method. The
labels/modalities present across can be combined for the generative problem
for a particular class. Therefore, the proposed method eliminates the need for
paired data for all multi-modal image generation task classes.

##  2 Related Work

###  2.1 Multimodal Image generation:

Recently multimdoal image synthesis has gained significant attention[1, 3, 7,
8, 4, 2, 9], where these method attempt to learn the posterior distribution of
an image when conditioned on the prior joint distribution of all the different
modalities. The approaches [3, 8, 4, 9, 7] follows a variational autoencoder
based solution where the all the input modalities are passed through their
respective encoders to obtain the means and variances of the underlying
Gaussian distributions and their means and variances in latent spaces are
combined using product of experts.Finally a new sample is obtained by sampling
using the new posterior means and variances. Some methods using GANs for
multimodal Image synthesis have also gained recent attention. Huang et al[1]
performs multimodal Image sysnthesis using introduces a new Local-Global
Adaptive Instance normalization to combine the modalities. Huang et al [1]
also uses Product of experts theory the combine encoded feature vectors and
use a feature decoder to obtain the final output. TediGAN[2] uses a StyleGAN
based framework where, the different visual-linguistic modalities are combined
in the feature space and decoded to obtain the final output. TediGAN allows
user defined Image manipulation according to the input of different
modalities.

###  2.2 Diffusion Probabilistic Models:

Diffusion probabilistic models belong to a class of generative models where
the a model is trained to generate images belonging to a distribution through
a Markovian process. The model is trained for reducing the lower bound of the
negative log likelihood of the data distribution. Diffusion models has a
forward process as well as a reverse process. In the forward process noise
with very small variances are added to the image over a large number of $T$
resulting in an isotropic Gaussian distribution. In the reverse Markovian
process, a network is utilized to perform iterative denoising based starting
from an isotropic Gaussian. At each step the network input is the output of
the previosu step as well as a particular timestep $t$. The sampling during
the forward process at a particular timestep $t$ is represented by

$\displaystyle q(z_{t}|z_{t-1})$
$\displaystyle=\mathcal{N}\left(z_{t};\sqrt{1-{\beta}_{t}}z_{t-1},{\beta}_{t}\mathbf{I}\right)$
(1)

Here $\beta_{1},\beta_{t},...\beta_{t}$ refers to a predefined variance
schedule for the diffusion process at a timestep $t$. The corresponding
reverse step is is defined by the distribution $p(z_{t-1}|z_{t}$ defined by,

$\displaystyle p(z_{t-1}|z_{t})$
$\displaystyle=\mathcal{N}\left(\mathbf{\mu_{\theta}(z_{t},t)},\mathbf{\Sigma_{\theta}(z_{t},t)}\right)$
(2)

Given the samples $z_{t-1}$ and $z_{t}$ at timesteps $t-1$ and $t$, a neural
network is used to estimate the mean of the distribution $p(z_{t-1}|z_{t})$.
The solution for $\mathbf{\Sigma_{\theta}(z_{t},t)}=\sigma_{t}^{2}\mathcal{I}$
can be set to $\mathbf{\beta_{t}}$ as mentioned in Ho et al[10].

###  2.3 Conditional Denoising Diffusion Models:

Recently Saharia et al[11] proposed a method for conditonal image generation
and extended its application to image super resolution. Saharia et al[11]
proposed that the conditonal generation of samples according to a particular
distribution $p(z|x)$ could be performed by adding the same condition to the
denoising network with the condition $x$ at each timestep during training and
inference The conditional distribution during the reverse time process and
condition with the image $x$ at each timestep. Extending this to our case,
given a particular modality $x_{i}$ The conditonal generation process could be
trained by using a newtwork with parameters $\theta$. The network
$f_{\theta}(.)$ predicts the scaled version of noise $\epsilon$ in the noisy
sample of the image $z_{t}$ . The training objective is defined as,

$L_{simple}=E_{t\sim[1,T],\epsilon\sim\mathcal{N}(0,\mathbf{I})}\left[\left\|\epsilon-
f_{\theta}\left(\mathbf{z}_{t},x_{i},t\right)\right\|^{2}\right]\\\ $ (3)

and the sampling process takes place through the equation

$z_{t-1}\leftarrow{}\frac{1}{\sqrt{1-\beta_{t}}}\left(z_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}f_{\theta}(z_{t},x_{i},t)\right)+\sigma_{t}^{2}\boldsymbol{\eta},\text{
where }\boldsymbol{\eta}=\mathcal{N}(\boldsymbol{0},\boldsymbol{I}),$ (4)

###  2.4 Langevian score based sampling from a diffusion process

An alternate interpretation of denoising diffusion probabilistic models is
denoising score based approach[12], where the sampling during inference is
performed using langevian dynamics. Here a network is used to compute the
score representing the gradient of likelihood of the data. The sampling during
the reverse timestep is can be represented by[12]

$\displaystyle
z_{t-1}\leftarrow{}\frac{1}{\sqrt{1-\beta_{t}}}\left(z_{t}-\beta_{t}s_{\theta}(z_{t},t)\right)+\sigma_{t}^{2}\boldsymbol{\eta},\text{
where }\boldsymbol{\eta}=\mathcal{N}(\boldsymbol{0},\boldsymbol{I}),$ (5)

Comparing Equations 4 and 5, we can see that the score value

$s_{\theta|}(z_{t},t)=\frac{f_{\theta}(z_{t},x_{i},t)}{\sqrt{1-\bar{\alpha}_{t}}}$
(6)

##  3 Proposed Method

In normal conditional denoising diffusion models[11], the input image is
concatenated along with the sampled noise while passing through the network.
When multiple modalities are present, one trivial way of finding the
conditional distribution, is to concatenate all the $N$ input modalities while
training the network. But In case we want to improve the functionality of the
network by conditioning on new modalities, the whole model needs to be
retrained by conditioning with the new $N+1$ modalities.

For a given generative task based on multimodal priors, given the $N$
modalities and models trained on them individually, the new modal for the
joint generation task can be defined by

###  3.1 Multimodal conditioning using Diffusion models(MCDM)

Let $z$ and $x_{i}$ denote a point in the space of the images of a particular
domain and $p(z|x_{i})$ denotes the distribution of the predicted image $z$
based on the modality $x_{i}$.Let the distribution of the image conditioned on
all modalities is denoted by $P(z|x_{1},x_{2},...,x_{N})$ and the distribution
of the image conditioning on the individual modalities be $P(z|x_{i})$.
Assuming that all the modalities are statistically independent,

$P(z|x_{1},x_{2},...,x_{N})=\frac{P(z)}{P(x_{1},x_{2},...,x_{N})}\prod_{i=1}^{N}P(x_{i}|z)\propto\frac{\prod_{i=1}^{N}P(z|x_{i})}{\prod_{i=1}^{N-1}P(z)}$
(7)

Assuming the individual distributions $P(z|x_{i})$ and $P(z)$ are Gaussian
distributions, the distribution $P(z|x_{1},x_{2},...,x_{N})$ will also be a
Gaussian distribution. Now Let’s assume that $N$ diffusion models trained to
generate samples the distributions $P(z|x_{i})$, conditioned on each modality
$x_{i}$ separately.

As mentioned in section, we can use the relationship between diffusion models
and score matching[12] to derive the effective score function for the
conditional distibution $P(z|x_{1},x_{2},...,x_{N})$ and perform inference
based on this.

$P(z_{t}|x_{1},x_{2},...,x_{N})=\frac{K.\prod_{i=1}^{N}P(z|x_{i})}{\prod_{i=1}^{N-1}P(z)}$
(8)
$\nabla_{z_{t}}logP_{\theta}(z_{t}|x_{1},x_{2},...,x_{N})=\nabla_{z_{t}}log\left(\frac{K.\prod_{i=1}^{N}P(z|x_{i})}{\prod_{i=1}^{N-1}P(z)}\right)$
(9)
$=\sum_{i=1}^{N}\nabla_{z_{t}}\text{log}P(z_{t}|x_{i})-(N-1)\nabla_{z_{t}}\text{log}P(z_{t})$
(10)

Where K is a term independent of $z_{t}$. Hence the effective score when
conditioned on all the modalities can be represented in terms of scores of the
individual conditional distribution as well as the score of the unconditonal
model. Hence the effective score $s_{c}$

$\displaystyle s_{c}=\frac{\epsilon_{c}}{\sqrt{1-\bar{\alpha}_{t}}}$ (11)

$\displaystyle\epsilon_{c}=\epsilon_{\theta}(z_{t},x_{1},x_{2},..,x_{N},t)=\sum_{i=1}^{N}\epsilon_{\theta}(z_{t},x_{i},t)-(N-1)\epsilon_{\theta}(z_{t},t)$
(12)

$\epsilon_{\theta}(z_{t},x_{i},t)$ denotes the output prediction of the
individual conditional networks and $\epsilon_{\theta}(z_{t},t)$ is the
prediction of the unconditional network. After computing the effective score,
sampling could be performed using equation by,

$z_{t-1}\leftarrow{}\frac{1}{\sqrt{1-\beta_{t}}}\left(z_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{c}\right)+\sigma_{t}^{2}\boldsymbol{\eta},\text{
where }\boldsymbol{\eta}=\mathcal{N}(\boldsymbol{0},\boldsymbol{I}),$ (13)

##  4 Experiments

For the multimodal generation task, we perform experiments on CelebA dataset.
As the different modalities. We choose edge maps of the images as well as
partial face segmentation masks from the CelebAHQ dataset. We also take the
text captions for images as another modality.

##  5 Results

In this section we detail on the qualitative experiments performed. As part of
the multimodal generation task, we perform three experiments combining three
different modalities taking two at a time and another experiment with all
three modalities.

Figure 1: Results using the modalities sketch and partial face mask.

Figure 2: Results using the modalities text and partial face mask.

Figure 3: Results using the modalities sketch and text.

.

Figure 4: Image samples generated with three modalities sketch, segment and
text. The text prompt used in all these results are "This person has black
hair".

###  5.1 Modalities- sketch and segment:

As can be seen in Figure 1 The method is able to extract complementary
information across modalities and fill in the missing regions.

###  5.2 Modalities- text and segment:

As can be seen in Figure 2 Since there is a lot of missing information in the
partial face masks. We are able to include key information such as hair
colour, gender, mouth orientation using text prompts.

###  5.3 Modalities- text and sketch:

As can be seen in Figure 3. The edge maps consists of a lot of information
from the original image. Hence it is difficult to add much text information to
this. For this reason, we choose hair colour as a property and try to change
it using the corresponding text prompts.

This is a raw version of the paper. The paper will be updated with more
results and experiments

##  6 Conclusion

In this paper we propose a method to combine multiple diffusion models during
inference time to perform multimodal image synthesis. Our method does hence
opens up an idea for utilizing annotated data across multiple datasets and use
these together for a better constrained generation. Extensive experiments on
CelebA dataset shows the working and proof of concept of our method.

## References

  * [1] Xun Huang, Arun Mallya, Ting-Chun Wang, and Ming-Yu Liu.  Multimodal conditional image synthesis with product-of-experts gans.  arXiv preprint arXiv:2112.05130, 2021.
  * [2] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu.  Tedigan: Text-guided diverse face image generation and manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2256–2265, 2021.
  * [3] Yuge Shi, Brooks Paige, Philip Torr, et al.  Variational mixture-of-experts autoencoders for multi-modal deep generative models.  Advances in Neural Information Processing Systems, 32, 2019.
  * [4] Mike Wu and Noah Goodman.  Multimodal generative models for scalable weakly-supervised learning.  Advances in Neural Information Processing Systems, 31, 2018.
  * [5] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34, 2021.
  * [6] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  * [7] Thomas M Sutter, Imant Daunhawer, and Julia E Vogt.  Generalized multimodal elbo.  arXiv preprint arXiv:2105.02470, 2021.
  * [8] Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo.  Joint multimodal learning with deep generative models.  arXiv preprint arXiv:1611.01891, 2016.
  * [9] Zhu Zhang, Jianxin Ma, Chang Zhou, Rui Men, Zhikang Li, Ming Ding, Jie Tang, Jingren Zhou, and Hongxia Yang.  M6-ufc: Unifying multi-modal controls for conditional image synthesis.  arXiv preprint arXiv:2105.14211, 2021.
  * [10] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [11] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  arXiv preprint arXiv:2104.07636, 2021.
  * [12] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  Advances in Neural Information Processing Systems, 32, 2019.