# ECoDepth: Effective Conditioning of Diffusion Models
for Monocular Depth Estimation

Suraj Patni Aradhye Agarwal11footnotemark: 1 Chetan Arora
Indian Institute of Technology Delhi
https://ecodepth-iitd.github.io Equal contribution.

###### Abstract

In the absence of parallax cues, a learning based single image depth
estimation (SIDE) model relies heavily on shading and contextual cues in the
image. While this simplicity is attractive, it is necessary to train such
models on large and varied datasets, which are difficult to capture. It has
been shown that using embeddings from pre-trained foundational models, such as
CLIP, improves zero shot transfer in several applications. Taking inspiration
from this, in our paper we explore the use of global image priors generated
from a pre-trained ViT model to provide more detailed contextual information.
We argue that the embedding vector from a ViT model, pre-trained on a large
dataset, captures greater relevant information for SIDE than the usual route
of generating pseudo image captions, followed by CLIP based text embeddings.
Based on this idea, we propose a new SIDE model using a diffusion backbone
which is conditioned on ViT embeddings. Our proposed design establishes a new
state-of-the-art (SOTA) for SIDE on NYU Depth v2 dataset, achieving Abs Rel
error of 0.059(14% improvement) compared to 0.069 by the current SOTA (VPD).
And on KITTI dataset, achieving Sq Rel error of 0.139 (2% improvement)
compared to 0.142 by the current SOTA (GED). For zero shot transfer with a
model trained on NYU Depth v2, we report mean relative improvement of (20%,
23%, 81%, 25%) over NeWCRF on (Sun-RGBD, iBims1, DIODE, HyperSim) datasets,
compared to (16%, 18%, 45%, 9%) by ZoEDepth. The code is available at this
link.

(a) Sun-RGBD [42]

(b) iBims1 [19]

(c) DIODE [49]

(d) HyperSim [35]

Figure 1: Qualitative results across four different datasets, demonstrating
the zero-shot performance of our model trained only on the NYU Depth v2
dataset. Corresponding quantitative results are presented in Table 3. The
first column displays RGB images, the second column depicts ground truth
depth, and the third column showcases our model’s predicted depths. Additional
images for each dataset are available in the Supplementary Material.

##  1 Introduction

Single Image Depth Estimation (SIDE) is the task of predicting per pixel depth
using a single RGB image from a monocular camera. It is a fundamental problem
in computer vision with applications in several domains, viz robotics,
autonomous driving, and augmented reality. The problem is typically formulated
in two flavors: metric depth estimation (MDE), and relative depth estimation
(RDE). As the names suggest, MDE deals with estimation in physical units such
as meters, whereas, RDE techniques focus on relative depth only, and require a
per-image affine transformation as post-processing to convert to a physical
unit.

Conventional geometric techniques for depth estimation typically rely on
feature correspondence, parallax, and triangulation from two or more views.
However, the problem becomes ill-posed for estimation from a single view.
Intuitively, depth map is a 3D representation of the scene, whereas an RGB
image is a 2D projection of the scene. Hence, it cannot be uniquely determined
from a single RGB image. Therefore, learning-based SIDE models rely on visual
cues like ‘shape from shading’ and other contextual priors for per-pixel depth
prediction.

A data-driven approach makes the SIDE pipeline simpler, but also makes the
learnt model dependent on the quality of training data. It has been observed
that such models overfit on a particular training distribution/domain and fail
to generalize on unseen data. This is especially true for MDE models when the
range of depth in the training dataset is limited. Hence, training on multiple
datasets, with wide variations in depth ranges has been proposed [4, 34].

On the other hand, development of large foundational models (LFMs) in recent
years has altered the preferred design approach for many computer vision
problems. These huge models are trained using extensive datasets of unlabeled
images and learning objectives that are agnostic to specific tasks. The learnt
embeddings from such pre-trained models have been shown to help generalization
and zero-shot transfer in many applications. We are aware of at least two
works for MDE in SIDE problem that have appeared in the last few months and
make use of such foundational models. VPD [54] uses a text-to-image diffusion
model pre-trained on LAION-400M [39] dataset having large-scale image-text
pairs as the backbone. The model prompts the denoising UNet with textual
inputs to make the visual contents interact with the text prompts. Since the
problem formulation doesn’t include text description as the input, the model
generates simple descriptions such as ‘‘A photo of a {scene name}’’ based on
the scene label given in the NYU Depth v2 dataset. Another work, TADP [20]
improves upon VPD. Instead of simple image descriptions, TADP uses BLIP-2 [23]
to generate image captions, and then uses CLIP [31] embeddings of the pseudo-
caption to condition the diffusion model.

We view the above works as providing robust semantic context to LFMs for the
actual task at hand, which helps in visual recognition in general, as well as
SIDE. While we do agree with the broad motivations of these works, the
question that we ask is if pseudo-captions are the most effective way to
provide the semantic context. Textual descriptions of an image typically focus
on large salient objects and emphasize on their relationships. On the other
hand, large vision models for image classification, typically contain
representation for even smaller objects present in the scene. Even when a
single object is present in the scene, the representations typically capture
uncertainties and ambiguities inherent in the scene. Hence, we posit that
using embeddings from a transformer model, pretrained on a large image dataset
unrelated to the SIDE task, captures more relevant information, and is a
better alternative to using pseudo-captions’ CLIP embedding. Since diffusion-
based models have shown their superiority for the dense prediction tasks in
recent works [16, 38, 54, 20, 8]. Hence, we propose a diffusion backbone for
our model, along with a novel CIDE module which employs ViT [7] to extract
semantic context embeddings. These embeddings are subsequently utilized to
condition the diffusion backbone.

Contributions. The key contributions of this work are:

  1. (1)

We propose a new model for MDE in a SIDE task. The proposed model uses a
conditional diffusion architecture with the semantic context being supplied
through embeddings generated using a ViT model. Achieving a new state-of-the-
art (SOTA) performance, our method outperforms existing approaches on
benchmark datasets, including the NYU Depth v2 indoor and KITTI outdoor
datasets. Notably, we report a significant improvement of 14% in absolute
relative error, achieving 0.059 compared to the current SOTA (VPD) performance
of 0.069 on NYU Depth v2. And we report an improvement of 2% in square
relative error, achieving 0.139 compared to the current SOTA (GED) performance
of 0.142 on KITTI.

  2. (2)

We show, qualitatively as well as quantitatively, that using ViT embeddings to
provide semantic context is a better alternative to generating pseudo captions
and then using its CLIP embeddings to condition a SIDE model. In contrast to
TADP [20], which uses pseudo captions, but only achieves a RMSE of 0.225 on
NYU Depth v2, we report a lower, and a new SOTA, error of 0.218 (VPD [54]
achieves 0.254).

  3. (3)

We show that providing ViT conditioning, helps our model perform better in a
zero shot transfer task. ZoEDepth [4], the current SOTA for zero-shot
transfer, reports an improvement of (16%, 18%, 45%, 9%) over NeWCRF on (Sun-
RGBD, iBims1, DIODE, HyperSim) datasets, after training their model on 12
other datasets and NYU Depth v2. In contrast, we only train on NYU Depth v2,
and report a much larger improvement of (21%, 23%, 81%, 25%).

Figure 2: An overview of our proposed model: The latent representation of the
input image undergoes a diffusion process, which is conditioned by our
proposed CIDE module. Within the CIDE module, the input image is fed through
the frozen ViT model. From this, a linear combination of the learnt embeddings
is computed, which is transformed to generate a 768-dimensional contextual
embedding. This embedding is utilized to condition the diffusion backbone.
Subsequently, hierarchical feature maps are extracted from the UNet’s decoder
which are concatenated and processed through a depth regressor to generate the
depth map.

##  2 Related Work

Traditional Methods. Earlier techniques for SIDE have used Markov Random
Fields [47], non-parametric depth sampling [18], and structural similarity
with prior depth map [13] to predict pixel-wise depth.

Deep Learning Techniques for SIDE. Modern techniques have approached the
problem as a dense regression problem. CNNs have been the dominant
architecture for the SIDE in the last decade, with global-local network stack
[9], and multi-scale [21], or encoder-decoder architecture [11] as some of the
popular solution strategies. Recently, PixelFormer [1] used a transformer-
based encoder-decoder architecture with skip connections from encoders to
decoders. MIM [51] proposed masked image modeling as a general-purpose pre-
training for geometric and motion tasks such as SIDE and pose estimation.
Similarly, AiT [28] used mask augmentation and proposed soft tokens to
generalize visual prediction tasks. While earlier works either focused on MDE
for a specific dataset or RDE for generalization on multiple datasets,
ZoEDepth [4] has proposed a generalized method for MDE that performs well in
zero-shot transfer. We outperform [4] by a large margin even when training on
a single dataset - NYU Depth v2 for indoor scenes or KITTI for outdoor scenes.
Whereas, [4] uses 12 datasets for pre-training and then fine-tunes on NYU
Depth v2 or KITTI for zero shot transfer.

Diffusion-based Methods with Pretraining on Large Datasets. Recently, many
techniques for SIDE has used diffusion architectures. These techniques [38,
16, 54, 20, 8], exploit prior knowledge acquired by pretraining on large
datasets like LAION-400M [39], which consists of 400 million image-text pairs.
In contrast, depth datasets, such as NYU Depth v2 and KITTI, contain around
20-30 thousand image-depth pairs. DepthGen [38] and DDP [16] work on a noise-
to-depthmap paradigm and use images for conditional guidance of the diffusion
process. DepthGen employs self-supervised pretraining on tasks like
colorization, inpainting, and JPEG artifact removal, followed by supervised
training on indoor and outdoor datasets [6, 26, 27, 48, 12]. DDP [16]
decouples the image encoder and map decoder, allowing the image encoder to run
just once, while the lightweight map decoder is run multiple times. VPD [54]
and TADP [20] use denoising UNet [37] as a backbone to extract the rich
features at multiple scales. Also, they utilize text instead of image for
conditioning the diffusion backbone.

Vision Transformer for Scene Understanding. The transformer architecture was
initially proposed for NLP tasks [50], but introduced to the computer vision
community as Vision Transformer (ViT) in [7]. Prior to this, CNNs dominated
computer vision problems due to their ability to capture spatial hierarchies
in image data. However, such architectures were constrained due to their
ability to learn spatially localized features only. Transformer architecture
has a weaker inductive bias and allows ViT to learn long-range dependencies,
and robust, generalizable features. ViT architectures have replaced CNNs for
SOTA performance on most computer vision tasks in recent years. We use a pre-
trained ViT model for providing semantic information to the diffusion backbone
in our model.

##  3 Proposed Methodology

###  3.1 Preliminaries

Problem Formulation. The objective of single image depth prediction task is to
predict continuous values, denoted as $\mathbf{y}\in\mathbb{R}^{H\times W}$,
for every pixel present in the input RGB image, $\mathbf{x}\in[0,255]^{3\times
H\times W}$. Here $H$ and $W$ represent the height and width respectively of
the input image.

Diffusion Model. Diffusion models are a class of generative models that
progressively inject noise into the input data (forward pass) and then learn
to reconstruct the original data in a reverse denoising process (reverse
pass). There are three formulations of diffusion models: denoising diffusion
probabilistic models (DDPMs) [14], score-based generative models [43, 44], and
those based on stochastic differential equations [46, 45]. DDPMs are of
relevance to our paper, and are described below. The architecture of DDPMs
consists of two Markov chains: a forward chain that adds noise to the data,
and a reverse chain that converts noise back to data by learning transition
kernels parameterized by deep neural networks. Formally, the forward pass is
modeled as a Markov process:

$\mathbb{P}\left(\mathbf{z}_{t}\mid\mathbf{z}_{t-1}\right)=\mathcal{N}\left(\mathbf{z}_{t};\sqrt{1-\beta_{t}}\mathbf{z}_{t-1},\beta_{t}\boldsymbol{I}\right).$
(1)

Here $\mathbf{z}_{t}$ denotes the random variable at the $t^{\text{th}}$ time
step, $\mathcal{N}(\mathbf{z};\boldsymbol{\mu},\boldsymbol{\sigma})$ denotes
Gaussian probability distribution, and $\beta_{t}$ is the noise schedule. The
above equation leads to the analytic form of
$\mathbb{P}(\mathbf{z}_{t}\mid\mathbf{z}_{0}),\forall t\in\\{0,1,\dots,T\\}$:

$\mathbf{z}_{t}=\sqrt{\bar{\beta}_{t}}\mathbf{z}_{0}+\sqrt{1-\bar{\beta}_{t}}\boldsymbol{\epsilon},$
(2)

where $\bar{\beta}_{t}=\prod_{s=1}^{t}\beta_{s}$ and
$\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\boldsymbol{I})$. The model
then gradually removes noise by executing a learnable Markov chain in the
reverse time direction, parameterized by a normal prior distribution
$\mathbb{P}\left(\mathbf{z}_{T}\right)=\mathcal{N}\left(\mathbf{z}_{T};0,\boldsymbol{I}\right)$
and a learnable transition kernel
$\mathbb{P}_{\theta}\left(\mathbf{z}_{t-1}\mid\mathbf{z}_{t}\right)$ given by:

$\mathbb{P}_{\theta}\left(\mathbf{z}_{t-1}\mid\mathbf{z}_{t}\right)=\mathcal{N}\left(\mathbf{z}_{t-1};\mu_{\theta}\left(\mathbf{z}_{t},t\right),\Sigma_{\theta}\left(\mathbf{z}_{t},t\right)\right).$
(3)

The goal of the training process is to approximately match the reverse Markov
chain with the actual time reversal of the forward Markov chain.
Mathematically, parameter $\theta$ is adjusted so that the joint distribution
of the reverse Markov chain
$\mathbb{P}_{\theta}\left(\mathbf{z}_{0},\mathbf{z}_{1},\cdots,\mathbf{z}_{T}\right):=\mathbb{P}\left(\mathbf{z}_{T}\right)\prod_{t=1}^{T}\mathbb{P}_{\theta}\left(\mathbf{z}_{t-1}\mid\mathbf{z}_{t}\right)$
closely approximates that of the forward Markov chain
$\mathbb{P}\left(\mathbf{z}_{0},\mathbf{z}_{1},\cdots,\mathbf{z}_{T}\right):=\mathbb{P}\left(\mathbf{z}_{0}\right)\prod_{t=1}^{T}\mathbb{P}\left(\mathbf{z}_{t}\mid\mathbf{z}_{t-1}\right)$.
This is achieved by minimizing the following loss:

$\mathbb{E}_{t\sim\mathcal{U}[1,T],\mathbf{z}_{0}\sim\mathbb{P}\left(\mathbf{z}_{0}\right),\boldsymbol{\epsilon}\sim\mathcal{N}(0,\boldsymbol{I})}\left[\left\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\theta}\left(\mathbf{z}_{t},t\right)\right\|^{2}\right],$
(4)

where $\mathbf{z}_{t}$ is computed using Eq. 2, and
$\boldsymbol{\epsilon}_{\theta}$ is predicted using a neural network,
typically a UNet architecture [37]. In a conditional diffusion model
$\boldsymbol{\epsilon}_{\theta}\left(\mathbf{z}_{t},t\right)$ gets replaced by
$\boldsymbol{\epsilon}_{\theta}\left(\mathbf{z}_{t},t,\mathbf{\mathcal{C}}\right)$,
where $\mathbf{\mathcal{C}}$ is a conditioning variable.

###  3.2 Our Architecture

Image Encoder and Latent Diffusion. Diffusion models typically take large
number of time steps to train, and are difficult to converge. Recently, [36]
proposed a new diffusion with improved convergence properties and is called
“stable-diffusion”. The key idea is to perform the diffusion in latent space,
with latent embedding learnt separately through a variational autoencoder
(VAE). The Encoder of VAE first transforms the input image $\mathbf{x}$ of
size $(H,W)$ to latent space, then we follow latent diffusion formulation and
utilize the UNet used in Stable Diffusion[36]. Utilizing latent diffusion
formulation enables our architecture to capture multi-resolution features.
Hence, we aggregate the feature maps from different layers of the UNet module
(implementing conditional diffusion) by bringing them all to 1/4th resolution
of the latent space, resulting in a feature map of size $8e\times H/32\times
W/32$.

(a)

(b)

Figure 3:  (a) Probabilistic graphical model corresponding to VPD. (b) The same corresponding to our formulation. Here, $\mathbf{\mathcal{C}}$ represents the semantic embedding derived from our CIDE module. This embedding is internally generated by passing $\mathbf{x}$ through the ViT, resulting in $\mathcal{E}$. Subsequently, $\mathcal{E}$ undergoes further processing to yield $\mathbf{\mathcal{C}}$, which is then utilized in the conditional diffusion module implementing $\mathbb{P}(\mathbf{z}_{0}\mid\mathbf{z}_{t},\mathbf{\mathcal{C}})$. The output of the conditional diffusion module is fed into the Depth Regressor module within our architecture, implementing $\mathbb{P}(\mathbf{y}\mid\mathbf{z}_{0})$. Table 1: Results on Indoor NYU Depth v2 [27] Dataset. Results that are bold perform best. $\uparrow$ means the metric should be higher, $\downarrow$ indicate lower is better. The evaluation uses an upper bound of 10 meters on the ground truth depth map. All the numbers for other works have been taken from the corresponding papers. For MIM, and ZoEDepth we have used SwinV2-L 1K, and ZoeDepth-M12-N versions respectively. We see an overall improvement against SOTA on all the metrics used for evaluation. Method | Venue | Abs Rel$\downarrow$ | RMSE$\downarrow$ | ${\log_{10}\downarrow}$ | Sq Rel${\downarrow}$ | $\delta_{1}\uparrow$ | $\delta_{2}\uparrow$ | $\delta_{3}\uparrow$
---|---|---|---|---|---|---|---|---
Eigen et al.[9] | NIPS’14 | 0.158 | 0.641 | - | - | 0.769 | 0.950 | 0.988
DORN[10] | CVPR’18 | 0.115 | 0.509 | 0.051 | - | 0.828 | 0.965 | 0.992
SharpNet[32] | ICCV’19 | 0.139 | 0.502 | 0.047 | - | 0.836 | 0.966 | 0.993
Chen et al.[5] | IJCAI-19 | 0.111 | 0.514 | 0.048 | - | 0.878 | 0.977 | 0.994
BTS[22] | Arxiv’19 | 0.110 | 0.392 | 0.047 | 0.066 | 0.885 | 0.978 | 0.994
AdaBins[2] | CVPR’21 | 0.103 | 0.364 | 0.044 | - | 0.903 | 0.984 | 0.997
DPT[33] | ICCV’21 | 0.110 | 0.357 | 0.045 | - | 0.904 | 0.988 | 0.998
P3Depth[30] | CVPR’22 | 0.104 | 0.356 | 0.043 | - | 0.898 | 0.981 | 0.996
NeWCRFs[53] | CVPR’22 | 0.095 | 0.334 | 0.041 | 0.045 | 0.922 | 0.992 | 0.998
SwinV2-B[24] | CVPR’22 | 0.133 | 0.462 | 0.059 | - | 0.819 | 0.975 | 0.995
SwinV2-L[24] | CVPR’22 | 0.112 | 0.381 | 0.051 | - | 0.886 | 0.984 | 0.997
Localbins[3] | ECCV’22 | 0.099 | 0.357 | 0.042 | - | 0.907 | 0.987 | 0.998
Jun et al.[17] | ECCV’22 | 0.098 | 0.355 | 0.042 | - | 0.913 | 0.987 | 0.998
PixelFormer[1] | WACV’23 | 0.090 | 0.322 | 0.039 | 0.043 | 0.929 | 0.991 | 0.998
DDP[16] | ICCV’23 | 0.094 | 0.329 | 0.040 | - | 0.921 | 0.990 | 0.998
MIM [51] | CVPR’23 | 0.083 | 0.287 | 0.035 | - | 0.949 | 0.994 | 0.999
AiT[28] | ICCV’23 | 0.076 | 0.275 | 0.033 | - | 0.954 | 0.994 | 0.999
ZoeDepth [4] | Arxiv’23 | 0.075 | 0.270 | 0.032 | 0.030 | 0.955 | 0.995 | 0.999
VPD[54] | ICCV’23 | 0.069 | 0.254 | 0.030 | 0.027 | 0.964 | 0.995 | 0.999
Ours | CVPR’24 | 0.059 | 0.218 | 0.026 | 0.013 | 0.978 | 0.997 | 0.999

RGBGTZoeDepth [4]VPD [54]Ours

Figure 4: Visual Comparison on NYU Depth v2 Indoor Dataset. Note, our method’s ability to delineate objects in terms of their depth, such as the table lamp in Row 5, even when such information is absent from the ground truth depth map. Table 2: Performance on the Outdoor KITTI [12] Dataset. Please refer to the caption of Tab. 1 for notation details. For ZoEDepth results, we use the ZoeDepth-M12-K version following the authors’ recommendation. In instances where results for certain methods were not reported in the respective works, denoted by “-”, and the code is unavailable, we were unable to generate the missing numbers. Despite the saturation of results on the outdoor KITTI dataset, our method consistently achieves comparable or superior performance to the state-of-the-art (SOTA) across all metrics. VPD[54] cannot be trained on KITTI dataset as it required a per image text label which is not present in KITTI. The symbol † indicates that the method utilizes additional information beyond RGB.  Method | Venue | Abs Rel$\downarrow$ | Sq Rel$\downarrow$ | RMSE${}_{log}\downarrow$ | RMSE$\downarrow$ | $\delta_{1}\uparrow$ | $\delta_{2}\uparrow$ | $\delta_{3}\uparrow$
---|---|---|---|---|---|---|---|---
Eigen et al.[9] | NIPS’14 | 0.203 | 1.517 | 0.282 | 6.307 | 0.702 | 0.898 | 0.967
DORN[10] | CVPR’18 | 0.072 | 0.307 | 0.120 | 2.727 | 0.932 | 0.984 | 0.994
BTS[22] | Arxiv’19 | 0.059 | 0.241 | 0.096 | 2.756 | 0.956 | 0.993 | 0.998
AdaBins[2] | CVPR’21 | 0.067 | 0.190 | 0.088 | 2.960 | 0.949 | 0.992 | 0.998
DPT[33] | ICCV’21 | 0.060 | - | 0.092 | 2.573 | 0.959 | 0.995 | 0.996
P3Depth[30] | CVPR’22 | 0.071 | 0.270 | 0.103 | 2.842 | 0.953 | 0.993 | 0.998
NeWCRFs[53] | CVPR’22 | 0.052 | 0.155 | 0.079 | 2.129 | 0.974 | 0.997 | 0.999
PixelFormer[1] | WACV’23 | 0.051 | 0.149 | 0.077 | 2.081 | 0.976 | 0.997 | 0.999
ZoeDepth [4] | Arxiv’23 | 0.054 | 0.189 | 0.083 | 2.440 | 0.97 | 0.996 | 0.999
DDP [16] | ICCV’23 | 0.050 | 0.148 | 0.076 | 2.072 | 0.975 | 0.997 | 0.999
URCDC [40] | ToM’23 | 0.050 | 0.142 | 0.076 | 2.032 | 0.977 | 0.997 | 0.999
IEBins [41] | NeurIPS’23 | 0.050 | 0.142 | 0.075 | 2.011 | 0.978 | 0.998 | 0.999
MIM [51] | CVPR’23 | 0.050 | 0.139 | 0.075 | 1.966 | 0.977 | 0.998 | 1.000
GEDepth[52]† | ICCV’23 | 0.048 | 0.142 | 0.076 | 2.044 | 0.976 | 0.997 | 0.999
Ours | CVPR’24 | 0.048 | 0.139 | 0.074 | 2.039 | 0.979 | 0.998 | 1.000

RGBNewCRF[53]Pixelformer[1]IEbins[41]URCDCdepth[40]Ours

Figure 5: Visual Comparison on KITTI Outdoor Dataset. Table 3: Quantitative results for zero-shot transfer to four unseen indoor datasets. mRIθ denotes the mean relative improvement with respect to NeWCRFs across all metrics ($\delta_{1}$, REL, RMSE). Evaluation depth is capped at 8m for SUN RGB-D, 10m for iBims and DIODE Indoor, and 80m for HyperSim. Best results are in bold, second best are underlined. Our mRIθ outperforms all methods across all datasets by a large margin. † denotes that ZoeD is trained on 12 datasets and our method is trained only on NYUv2. | SUN RGB-D | iBims-1 Benchmark | DIODE Indoor | HyperSim
---|---|---|---|---
Method |  $\delta_{1}$ $\uparrow$ |  REL $\downarrow$ |  RMSE $\downarrow$ |  mRIθ $\uparrow$ |  $\delta_{1}$ $\uparrow$ |  REL $\downarrow$ |  RMSE $\downarrow$ |  mRIθ $\uparrow$ |  $\delta_{1}$ $\uparrow$ |  REL $\downarrow$ |  RMSE $\downarrow$ |  mRIθ $\uparrow$ |  $\delta_{1}$ $\uparrow$ |  REL $\downarrow$ |  RMSE $\downarrow$ |  mRIθ $\uparrow$
BTS [22] | 0.740 | 0.172 | 0.515 | -14.2% | 0.538 | 0.231 | 0.919 | -6.9% | 0.210 | 0.418 | 1.905 | 2.3% | 0.225 | 0.476 | 6.404 | -8.6%
AdaBins [2] | 0.771 | 0.159 | 0.476 | -7.0% | 0.555 | 0.212 | 0.901 | -2.1% | 0.174 | 0.443 | 1.963 | -7.2% | 0.221 | 0.483 | 6.546 | -10.5%
LocalBins [3] | 0.777 | 0.156 | 0.470 | -5.6% | 0.558 | 0.211 | 0.880 | -0.7% | 0.229 | 0.412 | 1.853 | 7.1% | 0.234 | 0.468 | 6.362 | -6.6%
NeWCRFs [53] | 0.798 | 0.151 | 0.424 | 0.0% | 0.548 | 0.206 | 0.861 | 0.0% | 0.187 | 0.404 | 1.867 | 0.0% | 0.255 | 0.442 | 6.017 | 0.0%
VPD [54] | 0.861 | 0.121 | 0.355 | 14.7% | 0.627 | 0.187 | 0.767 | 11.5% | 0.480 | 0.392 | 1.295 | 63.4% | 0.333 | 0.531 | 5.111 | 8.5%
ZoeD-M12-N† [4] | 0.864 | 0.119 | 0.346 |  16.0% | 0.658 | 0.169 | 0.711 | 18.5% | 0.376 | 0.327 | 1.588 | 45.0% | 0.292 | 0.410 | 5.771 | 8.6%
Ours | 0.885 | 0.112 | 0.319 | 20.5% | 0.688 | 0.163 | 0.664 | 23.1% | 0.545 | 0.344 | 1.164 | 81.3% | 0.394 | 0.442 | 4.739 | 25.2%
Table 4: Effectiveness of different embeddings for guiding the diffusion process for depth estimation on NYU Depth v2 dataset. In the third row, rather than using pseudo caption embeddings generated from the scene label (as implemented by VPD [54]), we provide the one-hot vector representing the scene label as a condition to the diffusion model. We observe a slight improvement in the metrics, highlighting that information content in the caption is similar to that in one-hot label. Embeddings | RMSE$\downarrow$ | Abs Rel$\downarrow$ | $\delta_{1}\uparrow$
---|---|---|---
Scene label emb. [54] | 0.254 | 0.069 | 0.964
Text caption emb. [20] | 0.225 | 0.062 | 0.976
One-hot vector emb. | 0.244 | 0.067 | 0.968
Proposed scene emb. | 0.218 | 0.059 | 0.978

Figure 6:  Visualization of improvements over VPD [54] in our model due to ViT
embeddings passed down as conditional vectors (in blue and green). In the
above images, ViT detects the desktop computer (first image) and table lamp
(second image) with high probability, and they are thus better detected by our
model. Additional visualizations are provided in the supplementary material.

Exploiting Semantic Context with Conditional Diffusion. Recall that we
formulated the single image depth estimation as a dense regression problem,
predicting $\mathbb{P}(\mathbf{y}\mid\mathbf{x})$, where $\mathbf{y}$ and
$\mathbf{x}$ denote output depth, and input image respectively. In our
diffusion formulation, we predict $\mathbf{z}_{0}$ which is then used to
predict pixel-wise depth, $\mathbf{y}$. It has been shown that noise
prediction in a diffusion model, $\boldsymbol{\epsilon}_{\theta}$, can be seen
as predicting the gradient of the density function,
$\nabla_{\mathbf{z}_{t}}\log\mathbb{P}(\mathbf{z}_{t})$. Hence, overall
architecture for SIDE using a diffusion architecture can be seen as
factorizing the conditional probability, as shown in the probabilistic graph
model in Fig. 3(a). To utilize additional semantic context generated from a
ViT model, we condition it on the ViT embeddings as shown in Fig. 3(b).
Mathematically, we model:

$\displaystyle\mathbb{P}(\mathbf{y}\mid\mathbf{x},\mathcal{E})=$
$\displaystyle\mathbb{P}(\mathbf{y}\mid\mathbf{z}_{0})\mathbb{P}(\mathbf{z}_{0}\mid\mathbf{z}_{t},\mathbf{\mathcal{C}})\mathbb{P}(\mathbf{z}_{t}\mid\mathbf{x})\mathbb{P}(\mathbf{\mathcal{C}}\mid\mathbf{x}),$
where $\displaystyle\mathbb{P}(\mathbf{\mathcal{C}}\mid\mathbf{x})=$
$\displaystyle\mathbb{P}(\mathbf{\mathcal{C}}\mid\mathcal{E})\mathbb{P}(\mathcal{E}\mid\mathbf{x}).$
(5)

Here, the first term, $\mathbb{P}(\mathbf{y}\mid\mathbf{z}_{0})$, is
implemented through the Depth Regressor module explained earlier. Similarly,
$\mathbb{P}(\mathbf{z}_{t}\mid\mathbf{x})$ is implemented using the VAE’s
Encoder as described earlier. We generate conditional information,
$\mathbf{\mathcal{C}}$ using our _Comprehensive Image Detail Embedding_ module
(hereafter CIDE, and explained below). The CIDE module takes $\mathbf{x}$ as
input and generates embedding vector $\mathbf{\mathcal{C}}$ (of dimension 768
in our design) as the output, thus, implementing
$\mathbb{P}(\mathbf{\mathcal{C}}\mid\mathbf{x})$ given in Eq. 5. We use
$\mathcal{E}$ to denote the embedding vector of a ViT module, and
$\mathbb{P}(\mathcal{E}\mid\mathbf{x})$ is implemented through the ViT.
$\mathbb{P}(\mathbf{\mathcal{C}}\mid\mathcal{E})$ is implemented using
downstream modules in CIDE consisting of learnable embeddings. The second
term, $\mathbb{P}(\mathbf{z}_{0}\mid\mathbf{z}_{t},\mathbf{\mathcal{C}})$, is
implemented using conditional diffusion,

Comprehensive Image Detail Embedding (CIDE) Module. As described earlier, we
believe using pseudo-captions to generate the semantic context has limited
utility, as the textual descriptions typically focus on large salient objects
only. Instead we propose our CIDE module which use embeddings from a pre-
trained ViT, and extract detailed semantic context from these embeddings. For
this we take 1000 dimensional logit vector from ViT, and pass it through a two
layer MLP which converts it to a 100 dimensional vector. Subsequently, we
employ this vector to compute the linear combination of 100 learnable
embeddings. This resulting embedding undergoes a linear transformation to
yield a semantic context vector of dimension 768, which is then passed to
conditional diffusion module.

Depth Regressor. The output feature map undergoes through an Upsampling
Decoder, comprised of deconvolution layers, followed by a Depth Regressor. The
Depth Regressor is essentially a two-layer convolutional neural network (CNN),
with the initial layer having dimensions Conv$(3\times 3)$,192, and the
subsequent layer Conv$(3\times 3)$,1.

##  4 Experiments and Results

Datasets and Evaluation. We use NYU Depth v2 [27] and KITTI [12] as the
primary datasets for training. The NYU Depth v2 dataset is a widely used
indoor benchmark for monocular depth estimation, containing over 24k densely
labeled pairs of RGB and depth images in the train set and 654 in the test
set. The dataset covers a wide range of indoor scenes and includes challenging
scenarios such as reflective surfaces, transparent objects, and occlusions.
The ground truth depth maps are obtained using a structured light sensor and
are provided at a resolution of $640\times 480$. KITTI dataset is a widely
used outdoor benchmark for monocular depth estimation, containing over 24k
densely labeled pairs of RGB and depth images. The dataset covers outdoor
driving scenarios and includes varying lighting conditions, weather, and
occlusions. The ground truth depth maps are obtained using a Velodyne LiDAR
sensor and are provided at a resolution of $1242\times 375$. To demonstrate
generalizability, we evaluate zero-shot performance on the following datasets:
Sun-RGBD [42], iBims1 [19], DIODE [49], and HyperSim [35]. Whereas, the main
paper contains mostly quantitative, and only some representative visual
results, detailed visual results on each dataset are included in the
supplementary.

Implementation Details. Our model is implemented using PyTorch [29]. For
optimization, we have used AdamW optimizer [25] with $\beta_{0}$ values of 0.9
and 0.999, a batch size of 32, and a weight decay of $0.1$. We train our model
for 25 epochs for both KITTI and NYU Depth v2 datasets, with an initial
learning rate of $3\times 10^{-5}$. We first linearly increase learning rate
to $5\times 10^{-4}$, and then linearly decrease across training iterations.
We use usual data augmentation techniques, including random hue addition,
horizontal flipping, changing the image brightness, and Cut Depth [15]. Our
model takes approximately 21 minutes per epoch to train using 8 NVIDIA A100
GPUs.

###  4.1 Comparison on Benchmark Datasets

Comparison on NYU Depth v2. Tab. 1 shows the comparison of our proposed method
with SOTA methods on the indoor NYU Depth v2 dataset [27]. We achieve a new
state of the art on this dataset. Our methods perform better than the previous
SOTA ([54])) by a large margin of 14% in terms of RMSE. Fig. 4 provides a
qualitative comparison on the dataset.

Comparison on KITTI. Tab. 2 shows the comparison with various methods on the
KITTI dataset [12]. Fig. 5 shows the qualitative results. Unlike NYU Depth v2
dataset, KITTI is an outdoor dataset. On this dataset also, we achieve similar
or better performance than all existing state-of-the-art techniques.

###  4.2 Generalization and Zero Shot Transfer

Unlike state of the art on zero short transfer (ZoEDepth [4]), which requiring
training on many datasets (12 in their case) for effective zero shot transfer,
we show that our model generalizes well to other unseen dataset even when
trained on a single NYU Depth v2 dataset. Tab. 3 shows quantitative results to
back our findings.

###  4.3 Ablation Study

Effect of Contextual Information. As highlighted in the motivation, a key
observation of this study is the richness of information contained within the
output probability vector of the ViT, surpassing the textual embeddings
employed in current state-of-the-art (e.g., VPD [54]). To compare with the
utilization of scene label information (as implemented by VPD), we construct
the conditioning embedding as a one-hot vector using the scene label, and
subsequently transform it using an MLP. As illustrated in Tab. 4, we observe a
slight improvement over VPD, indicating that the information content in
pseudo-captions resembles that of one-hot labels. Furthermore, our proposed
method surpasses both the approaches.

Qualitative Results. Perhaps the most important task would be to verify that
use of rich probability vector instead of text embeddings actually results in
an improvement in depth as a direct consequence. We do this by considering the
top few objects predicted by the ViT and correlating this with the depth
predicted at these objects. We hypothesise when a particular class say _dog_
is predicted with a high probability (hence there must be a dog in the image),
then the corresponding depth must also be more accurate. We show this in Fig.
6.

##  5 Conclusion

We presented a new architecture block Comprehensive Image Detail Embedding
(CIDE) module for robust monocular depth estimation in this paper. Our key
idea is to highlight the limitations of using pseudo-captions to provide
contextual information, and instead propose to use richer class-wise
probability generated by a classification model, such as ViT. The motivation
is that, while textual embedding typically highlight salient objects, class-
wise probability vector preserves more details, including smaller objects in
the background also. We implement the idea using proposed CIDE module cascaded
with a conditional diffusion pipeline for monocular depth estimation. We
demonstrated the effectiveness of our approach on several benchmark datasets
and showed that it outperforms SOTA methods by a significant margin.

Acknowledgement. We thank Kartik Anand for assistance with some of the
experiments. This work has been supported by funding through Staqu
Technologies Pvt. Ltd., and Department of Science and Technology (DST),
Government of India. The travel for Suraj Patni is supported through Research
Acceleration funds of Department of Computer Science and Engineering, IIT
Delhi.

## References

  * Agarwal and Arora [2023] Ashutosh Agarwal and Chetan Arora.  Attention attention everywhere: Monocular depth prediction with skip attention.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 5861–5870, 2023.
  * Bhat et al. [2021] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka.  Adabins: Depth estimation using adaptive bins.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4009–4018, 2021.
  * Bhat et al. [2022] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka.  Localbins: Improving depth estimation by learning local distributions.  In _European Conference on Computer Vision_ , pages 480–496. Springer, 2022.
  * Bhat et al. [2023] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller.  Zoedepth: Zero-shot transfer by combining relative and metric depth, 2023\.
  * Chen et al. [2019] Xiaotian Chen, Xuejin Chen, and Zheng-Jun Zha.  Structure-aware residual pyramid network for monocular depth estimation.  _arXiv preprint arXiv:1907.06023_ , 2019.
  * Dai et al. [2017] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5828–5839, 2017.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Duan et al. [2023] Yiqun Duan, Xianda Guo, and Zheng Zhu.  Diffusiondepth: Diffusion denoising approach for monocular depth estimation, 2023.
  * Eigen et al. [2014] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  _Advances in neural information processing systems_ , 27, 2014.
  * Fu et al. [2018a] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao.  Deep ordinal regression network for monocular depth estimation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2002–2011, 2018a.
  * Fu et al. [2018b] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao.  Deep ordinal regression network for monocular depth estimation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2002–2011, 2018b.
  * Geiger et al. [2013] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  _International Journal of Robotics Research (IJRR)_ , 2013.
  * Herrera et al. [2018] José L. Herrera, Carlos R. del Blanco, and Narciso García.  Automatic depth extraction from 2d images using a cluster-based learning framework.  _IEEE Transactions on Image Processing_ , 27(7):3288–3299, 2018.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Ishii and Yamashita [2021] Yasunori Ishii and Takayoshi Yamashita.  Cutdepth: Edge-aware data augmentation in depth estimation.  _arXiv preprint arXiv:2107.07684_ , 2021.
  * Ji et al. [2023] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo.  Ddp: Diffusion model for dense visual prediction.  _arXiv preprint arXiv:2303.17559_ , 2023.
  * Jun et al. [2022] Jinyoung Jun, Jae-Han Lee, Chul Lee, and Chang-Su Kim.  Depth map decomposition for monocular depth estimation.  In _European Conference on Computer Vision_ , pages 18–34. Springer, 2022.
  * Karsch et al. [2014] Kevin Karsch, Ce Liu, and Sing Bing Kang.  Depth transfer: Depth extraction from video using non-parametric sampling.  _IEEE transactions on pattern analysis and machine intelligence_ , 36(11):2144–2158, 2014.
  * Koch et al. [2018] Tobias Koch, Lukas Liebel, Friedrich Fraundorfer, and Marco Korner.  Evaluation of cnn-based single-image depth estimation methods.  In _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_ , pages 0–0, 2018.
  * Kondapaneni et al. [2023] Neehar Kondapaneni, Markus Marks, Manuel Knott, Rogério Guimarães, and Pietro Perona.  Text-image alignment for diffusion-based perception.  _arXiv preprint arXiv:2310.00031_ , 2023.
  * Laina et al. [2016] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab.  Deeper depth prediction with fully convolutional residual networks.  In _2016 Fourth international conference on 3D vision (3DV)_ , pages 239–248. IEEE, 2016.
  * Lee et al. [2019] Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Il Hong Suh.  From big to small: Multi-scale local planar guidance for monocular depth estimation.  _arXiv preprint arXiv:1907.10326_ , 2019.
  * Li et al. [2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR, 2022.
  * Liu et al. [2022] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al.  Swin transformer v2: Scaling up capacity and resolution.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12009–12019, 2022.
  * [25] Ilya Loshchilov and Frank Hutter.  Decoupled Weight Decay Regularization.  In _ICML 2019_.
  * McCormac et al. [2016] John McCormac, Ankur Handa, Stefan Leutenegger, and Andrew J Davison.  Scenenet rgb-d: 5m photorealistic images of synthetic indoor trajectories with ground truth.  _arXiv preprint arXiv:1612.05079_ , 2016.
  * Nathan Silberman and Fergus [2012] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In _ECCV_ , 2012.
  * Ning et al. [2023] Jia Ning, Chen Li, Zheng Zhang, Chunyu Wang, Zigang Geng, Qi Dai, Kun He, and Han Hu.  All in tokens: Unifying output space of visual tasks via soft token.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 19900–19910, 2023.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2019.
  * Patil et al. [2022] Vaishakh Patil, Christos Sakaridis, Alexander Liniger, and Luc Van Gool.  P3depth: Monocular depth estimation with a piecewise planarity prior.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1610–1621, 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramamonjisoa and Lepetit [2019] Michael Ramamonjisoa and Vincent Lepetit.  Sharpnet: Fast and accurate recovery of occluding contours in monocular depth estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , pages 0–0, 2019.
  * Ranftl et al. [2021] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 12179–12188, 2021.
  * Ranftl et al. [2022] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(3), 2022.
  * Roberts et al. [2021] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind.  Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding.  In _International Conference on Computer Vision (ICCV) 2021_ , 2021\.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Saxena et al. [2023] Saurabh Saxena, Abhishek Kar, Mohammad Norouzi, and David J Fleet.  Monocular depth estimation using diffusion models.  _arXiv preprint arXiv:2302.14816_ , 2023.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Shao et al. [2023a] Shuwei Shao, Zhongcai Pei, Weihai Chen, Ran Li, Zhong Liu, and Zhengguo Li.  Urcdc-depth: Uncertainty rectified cross-distillation with cutflip for monocular depth estimation.  _arXiv preprint arXiv:2302.08149_ , 2023a.
  * Shao et al. [2023b] Shuwei Shao, Zhongcai Pei, Xingming Wu, Zhong Liu, Weihai Chen, and Zhengguo Li.  Iebins: Iterative elastic bins for monocular depth estimation.  _arXiv preprint arXiv:2309.14137_ , 2023b.
  * Song et al. [2015] Shuran Song, Samuel P. Lichtenberg, and Jianxiong Xiao.  Sun rgb-d: A rgb-d scene understanding benchmark suite.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 567–576, 2015.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ , 32, 2019.
  * Song and Ermon [2020] Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models.  _Advances in neural information processing systems_ , 33:12438–12448, 2020.
  * Song et al. [2020] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020.
  * Song et al. [2021] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon.  Maximum likelihood training of score-based diffusion models.  _Advances in Neural Information Processing Systems_ , 34:1415–1428, 2021.
  * Sun et al. [2009] M. Sun, A. Y. Ng, and A. Saxena.  Make3d: Learning 3d scene structure from a single still image.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 31(05):824–840, 2009.
  * Sun et al. [2020] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2446–2454, 2020.
  * Vasiljevic et al. [2019] Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z Dai, Andrea F Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R Walter, et al.  Diode: A dense indoor and outdoor depth dataset.  _arXiv preprint arXiv:1908.00463_ , 2019.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Xie et al. [2023] Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao.  Revealing the dark secrets of masked image modeling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14475–14485, 2023.
  * Yang et al. [2023] Xiaodong Yang, Zhuang Ma, Zhiyu Ji, and Zhe Ren.  Gedepth: Ground embedding for monocular depth estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 12719–12727, 2023.
  * Yuan et al. [2022] Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan.  Newcrfs: Neural window fully-connected crfs for monocular depth estimation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Zhao et al. [2023] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu.  Unleashing text-to-image diffusion models for visual perception.  _ICCV_ , 2023.

Supplementary Material

##  Appendix A Ablation Study

###  A.1 Effect of ViT Architecture

Table 5 investigates the impact of varying ViT sizes on the generation of
embeddings from RGB images. Our results for the NYU Depth v2 [27] dataset
suggest that ViT-base yields optimal performance. Additionally, our
observations in the KITTI dataset align with a similar trend.

Table 5: Ablation Study on ViT Sizes: Performance comparison of different ViT variants in terms of parameters and depth error metrics on the NYUv2 [27] dataset. The results guide the selection of ViT-base in our final architecture. Best results are in bold. Classifier |  $\\#$Parameters |  RMSE$\downarrow$ |  Abs Rel$\downarrow$ | $\delta_{1}\uparrow$
---|---|---|---|---
ViT-base | 86.6 M | 0.218 | 0.059 | 0.978
deit-base | 86.6 M | 0.218 | 0.059 | 0.978
ViT-large | 303.3 M | 0.218 | 0.060 | 0.978
ViT-huge | 630.8 M | 0.219 | 0.060 | 0.978
Table 6: Ablation Study on dimension of Learnable Scene Embeddings (N): The table shows the impact of varying the dimension of learnable scene embeddings on the depth error metrics. We observe a decrease in error with increasing N until saturation occurs at N=100, prompting us to limit the model parameters to N=100. Best results are highlighted in bold. N |  RMSE$\downarrow$ |  Abs Rel$\downarrow$ |  log${}_{10}\downarrow$ | $\delta_{1}\uparrow$
---|---|---|---|---
10 | 0.219 | 0.061 | 0.027 | 0.978
50 | 0.219 | 0.060 | 0.026 | 0.978
100 | 0.218 | 0.059 | 0.026 | 0.978
200 | 0.218 | 0.060 | 0.026 | 0.978

###  A.2 Additional Qualitative Ablation

In Fig. 8, we present supplementary qualitative ablation results that
highlight the correlation between value of ViT logits and the improvement in
the predicted depth. The visualization demonstrates that elevated value of ViT
logits for specific objects contribute to our model’s ability to focus on
those objects, enhancing the accuracy of predicted depth in corresponding
regions.

##  Appendix B Architectural Details

###  B.1 Image Encoder

Similar to Latent Diffusion [36], we employed the VQVAE’s encoder to
transition from image space to latent space.

###  B.2 Upsampling Decoder

After obtaining the hierarchical feature map from denoising UNet, the
concatenated feature map undergoes upsampling, transitioning from a resolution
of $64\times 64$ back to $H\times W$. Refer to Fig. 7 for a detailed view of
the upsampling decoder architecture.

Figure 7: Detailed architecture of the upsampling decoder, responsible for
upsampling the concatenated feature map to obtain the final feature map at a
resolution of $H\times W$, $e=192$

Figure 8: Enhanced visualizations showcasing improvements over VPD [54] in our
model, facilitated by ViT embeddings employed as conditional vectors for the
denoising procedure. In the presented images, our model demonstrates
heightened accuracy in detecting objects, such as the television (blue in
first image) and table lamp (green in second image) when these are detected
with high probability by ViT.

Table 7: Hyper-parameter settings for our model. Hyper-parameter | Value
---|---
Learning rate schedule | one cycle
Min learning Rate | $3\times 10^{-5}$
Max learning Rage | $5\times 10^{-4}$
Batch Size | $32$
Optimizer |  AdamW [25]
$\beta_{s}$ in optimizer |  ($0.9$, $0.999$)
Weight Decay | $0.1$
Layer Decay Rate | $0.9$
Embedding Dimension | $192$
Variance focus in SiLog loss | $0.85$
ViT Size | ViT-base
Number of learnable emb. | 100
epochs | 25

##  Appendix C Additional Experimental Details

###  C.1 Hyperparameters

For reproducibility of the results presented in the main paper and the
supplementary material, we provide a comprehensive list of the hyper
parameters employed in our experiments in Table 7.

RGB

GT

Zoed

Ours

RGB

GT

Zoed

Ours

Figure 9: Qualitative Comparison on the HyperSim [35] Dataset. Our depth
predictions are contrasted with those of Zoedepth[4]. The first row displays
RGB images, the second row shows groundtruth depth, the third row exhibits
Zoedepth[4]’s depth, and the fourth row showcases our depth predictions. To
facilitate visual comparison, the colormap scale remains consistent across
corresponding depth maps. Our model, trained only on NYU Depth v2, is compared
with Zoedepth[4], which is trained on 12 datasets and then fine-tuned on NYU
Depth v2.

RGB

GT

Zoed

Ours

RGB

GT

Zoed

Ours

Figure 10: Qualitative Comparison on the DIODE [49] Dataset. Our depth
predictions are contrasted with those of Zoedepth[4]. The first row displays
RGB images, the second row shows groundtruth depth, the third row exhibits
Zoedepth[4]’s depth, and the fourth row showcases our depth predictions. To
facilitate visual comparison, the colormap scale remains consistent across
corresponding depth maps. Our model, trained only on NYU Depth v2, is compared
with Zoedepth[4], which is trained on 12 datasets and then fine-tuned on NYU
Depth v2.

RGB

GT

Zoed

Ours

RGB

GT

Zoed

Ours

Figure 11: Qualitative Comparison on the Sun-RGBD [42] Dataset. Our depth
predictions are contrasted with those of Zoedepth[4]. The first row displays
RGB images, the second row shows groundtruth depth, the third row exhibits
Zoedepth[4]’s depth, and the fourth row showcases our depth predictions. To
facilitate visual comparison, the colormap scale remains consistent across
corresponding depth maps. Our model, trained only on NYU Depth v2, is compared
with Zoedepth[4], which is trained on 12 datasets and then fine-tuned on NYU
Depth v2.

RGB

GT

Zoed

Ours

RGB

GT

Zoed

Ours

Figure 12: Qualitative Comparison on the iBims1 [19] Dataset. Our depth
predictions are contrasted with those of Zoedepth[4]. The first row displays
RGB images, the second row shows groundtruth depth, the third row exhibits
Zoedepth[4]’s depth, and the fourth row showcases our depth predictions. To
facilitate visual comparison, the colormap scale remains consistent across
corresponding depth maps. Our model, trained only on NYU Depth v2, is compared
with Zoedepth[4], which is trained on 12 datasets and then fine-tuned on NYU
Depth v2.

##  Appendix D Qualitative Results for Zero-Shot Performance Across Datasets

In the main paper, we presented a quantitative comparison of our method’s
zero-shot performance. Here, we provide a qualitative assessment of our
method’s performance in comparison to ZoEDepth [4] across the HyperSim, DIODE,
Sun-RGBD and iBims1 datasets in Fig. 9, 10, 11 and 12.


# SynthPop++: A Hybrid Framework for Generating A Country-scale Synthetic
Population

Bhavesh Neekhra, Kshitij Kapoor & Debayan Gupta
Department of Computer Science
Ashoka University
India
<EMAIL_ADDRESS>

###### Abstract

Population censuses are vital to public policy decision-making. They provide
insight into human resource, demography, culture, and economic structure at
local, regional, and national levels. However, such surveys are very expensive
(especially for low and middle-income countries with high populations, such as
India), time-consuming, and may also raise privacy concerns, depending upon
the type of data collected.

In light of these issues, we introduce SynthPop++, a novel hybrid framework,
which can combine data from multiple real-world surveys (with different,
partially overlapping sets of attributes) to produce a real-scale synthetic
population of humans. Critically, our population maintains family structures
comprising individuals with demographic, socioeconomic, health, and
geolocation attributes: this means that our “fake” people live in realistic
locations, have realistic families, etc. Such data can be used for a variety
of purposes: we explore one such use case, Agent-based modelling of infectious
disease in India.

To gauge the quality of our synthetic population, we use machine learning and
statistical metrics. Our experimental results show that synthetic population
can realistically simulate the population for various administrative units of
India, producing real-scale, detailed data at the desired level of zoom – from
cities, to districts, to states, eventually combining to form a country-scale
synthetic population.

##  1 Introduction

Financial institutions, government agencies, think tanks, etc. are using
techniques like agent-based modelling(ABM) Bonabeau (2002) to simulate
increasingly complex scenarios for decision-making. However, the collection of
such extensive datasets needed for model training and Agent-based modelling
are constrained by real-world challenges, such as high costs of acquisition,
privacy laws, etc.

Figure 1: Our Framework : Pre-processed data is used as input for generating
data

Moreover, population censuses and various other surveys, e.g., for health,
economics, education etc., help inform public policy decision-making. Periodic
censuses and surveys provide aggregate statistics on variables describing the
demography, socioeconomic distribution and health status of the constituents
at various levels of zoom. They also help decision-makers examine the effects
of previous public policy decisions on the evolution of these statistics.
Surveys that provide data at an individual level instead of marginals are of
particular interest as once released, they allow for superior analysis of
correlations between different attributes. However, data collection for
censuses and surveys is a time-consuming and expensive task, especially for
low and middle-income countries. For example, the last Census in India, during
2011, involved a total of 2.7 million “enumerators” and “supervisors” (Office
of the Registrar General, India, 2019), costing approximately INR 22,000
million. Further, national censuses do not often release full-scale individual
data for public access due to privacy reasons which limit the potential for
analysis.

In light of these challenges, Privacy-preserving synthetic populations can be
used as an alternative to real individual-level datasets. To increase the
accuracy of downstream tasks, it is essential that the marginal and joint
distributions of the various attributes are similar across the original and
the synthetic population. Further, to address privacy concerns, it is
necessary to ensure that the synthetic dataset does not leak any personally
identifiable information about the individuals who were originally surveyed.

Additionally, with the growing complexity of Agent-based models, it is
necessary to generate synthetic datasets where individuals are parts of
networks, and the linkage is based on shared attributes like a household, a
workplace, a school, common direction of travel, etc. Preexisting frameworks
for synthetic population generation are limited in this regard.

To focus our discussion, we will explore the task of generating a synthetic
population with the use case of modelling disease spread (via Agent-based
modelling) throughout this paper. In this context, we make the following
contributions:

  * •

A novel hybrid framework which combines various state-of-the-art statistical
and machine learning models to generate a real-scale synthetic population with
network and geolocation data.

  * •

A combination of statistical and Machine Learning based metrics to gauge the
quality of the synthetic population

  * •

To promote transparency and reproducibility, our code and datasets are open-
sourced111https://github.com/k00lk0der1/synthpoppp/

The rest of the paper is organised as follows. Section 2 is about related work
to generate a synthetic population. In Section 3, we describe our model in
detail. The experiments and evaluation are presented in Section 4. We discuss
future work and conclude in Section 5.

##  2 Related Work

In the task of modelling and generating synthetic populations, there are
various challenges in modelling the tabular data. For example, a table might
have mixed data types with discrete and continuous columns. The columns in the
table (especially those with continuous values) might not follow, say, the
Gaussian distribution. Further, a column might have multi-modal distributions.
The data may also be highly imbalanced, resulting in insufficient training
opportunities for minor classes. Some ABMs require the synthetic population to
be realistically geolocated, which, to the best of our knowledge, has not been
attempted, especially not for India.

Though various methods have been suggested to tackle these issues, they do not
provide a solution to the issues specified. For example, Bayesian networks
Koller & Friedman (2009) combine low dimensional distributions to approximate
the full-dimensional distribution of a data set and are a simple but powerful
example of a graphical model. But, Bayesian methods can not model tabular data
effectively when both continuous and discrete columns are present. Deep
Generative models are generally more sophisticated but, in practice, can not
outperform Bayesian methods as the tabular data may contain Non-Gaussian
continuous columns and/or imbalanced discrete columns. Some methods have tried
to mitigate these issues. PrivBayes Zhang et al. (2014) can model the table
with discrete variables, but all continuous variables need to be discretised.
Additionally, noise injection to preserve privacy may result in low-quality
synthetic data. MedGAN Choi et al. (2017) has been used to generate health
records, but it does not support mixed data types. It also can not handle Non-
Gaussian, Multi-modal, imbalanced categorical columns, lack of training data
and missing data. Although CTGAN Xu et al. (2019) can work with tables with
mixed data types, high dimensionality, imbalanced categorical data etc., it
can not handle a lack of training data and missing values. Also, it can not
model realistic family structures. Iterative Proportional Updating (IPU) Ye et
al. (2009) can work with limited data and model realistic family structures,
but it does not scale well with high dimensional data. To tackle these issues,
we propose our framework, which uses a hybrid, as shown in diagram 1 of IPU
and CTGAN to take advantage of the best from these two models. Our framework
allows for combining data from various sources and generates a synthetic
population with required columns.

##  3 Our Framework

Our objective is to generate a real-scale synthetic population for an entire
country with distributions and joint distributions of various variables that
match those of the actual population. This is necessary to ensure that the
results of the downstream tasks are reliable.

A synthetic population is a limited individual-level representation of the
actual population. However, not all the attributes of an individual are
included (for example, hair colour or shoe size might be irrelevant for
modelling disease spread, while co-morbidities like diabetes would be
included). As such, a synthetic population does not aim to perfectly mimic
reality – this would be impossible. Instead, it attempts to sufficiently match
various statistical measures observed in the real population.

The components of any pipeline used for generating a synthetic population
would largely depend on the variables that are required for the downstream
task. As stated earlier, our pipeline, as shown in diagram 1, is built with a
focus on synthetic population data for district-level infectious disease
modelling.

For the chosen downstream task, the synthetic population should capture
various heuristics like the size of a family; the joint distribution of age
and sex of individuals within a family; administrative units like cities,
districts, and states; geographical distribution of households, workplaces,
and schools within an administrative unit; the number of people who are
associated with a workplace or school; and various other individual-level
statistical correlations that can be observed in any population. Iterative
Proportional Updating and Conditional Tabular GAN were used to generate
families and individuals and their related metadata. Further, reject sampling
was used to sample geo-location points within a city’s boundaries which
followed the per unit area population density distribution. A method for
random assignment of external locations like workplaces, schools and public
places was used where the probability of selecting any external location was
inversely proportional to the distance between a given individual’s household
and the external location Ministry of Education (2019).

###  3.1 Data Generation with Family Structure

IPU Ye et al. (2009) is a sampling method which ensures that the marginal
distributions for both households and individuals in the synthetic population
match with the marginals distributions of the real population. Hence, sampling
with IPU helps generate a population in which households have a realistic
distribution of the number of family members and their age and gender.
However, matching only marginal distribution is not enough. We must closely
resemble the joint distribution of various attributes in the population. To
achieve this, we use CTGAN Xu et al. (2019) for adding individual-level
attributes for which marginals are not available. The CTGAN model can be
conditioned on attributes present in the base population generated by IPU to
generate additional attributes. Thus using CTGAN, we can get a reasonable
joint distribution of various individual-level attributes in the synthetic
population. Additionally, multiple CTGAN models can also be used to join
survey datasets which have an overlapping set of attributes. This allows for
the expansion of attributes as per the needs of the downstream task.

###  3.2 Population Density Distribution Sampling

Given that the downstream task is to analyse the spread of infectious diseases
in a given region, it is necessary to have an accurate geographical
distribution of population density and workplace density. To this end, grid
population density data Gaughan et al. (2013) is used. A row in the grid
population density data consists of latitude ($X$), longitude ($Y$) and the
number of individuals ($Z$) who live within a square with edge length $S$ and
with its centre at the given latitude and longitude ($X$, $Y$). For the data
set used in the current pipeline, the length of the side of square $S=0.3$
$arctan$ translates to a length of about $1$ $km$ on the equator. The sampling
method also needs a polygon for the boundaries of the region for which data is
being generated. We use the geojson file for each district to get these
geographical boundaries.

Let $n$ be the number of latitude-longitude pairs that need to be sampled
within this region. A filtered subset of the population density dataset is
obtained by only retaining those rows for which the latitude $X$ and longitude
$Y$ are within the boundary polygon of the region. We sample with replacement
$k$ latitude longitude pairs from this filtered subset and use the number of
individuals($Z$) as the weight for this sampling. For a given row in the
sample, we then sample points within the squares corresponding to the
latitude-longitude pair by adding independent random uniform noise $A,B\sim
U(-S/2,S/2)$ to the latitude $X$ and longitude $Y$ respectively. We reject
those latitude-longitude pairs that are not within the polygon. Given this
method uses reject sampling, we initially need to sample more points than
required, and hence we need to use a suitable value for $k>n$ and then choose
at random from the pairs which were not rejected.

We use this method to assign geolocation to houses, workplaces, schools and
public places. Given we do not have access to workplace distribution data, we
assume that it follows the population density distribution as well. However,
this assumption is not valid in cities where people live in outer sub urban
areas and work closer to the city center and other possible scenarios. With
the appropriate dataset, we can substitute the population density data with
workplace density data and get a distribution similar to the ground reality
with the sampling method described earlier.

###  3.3 External Location Assignment

Since infectious disease modelling often includes analysis based on contact
network graphs, individuals in this synthetic population would need to be
associated with external locations like workplaces, schools and public places
which they might visit periodically. This assignment is based on the
assumption that adults work at workplaces, children go to schools, and people
are more likely to visit public places which are closer to their homes. We use
the L2 distance metric for this computation since the earth’s curvature within
a city is negligible, but a geodesic distance metric can also be used. Given
an individual with home latitude $X_{h}$ and longitude $Y_{h}$ and a list of
possible external locations with latitude longitude pairs ($X_{E_{1}}$,
$Y_{E_{1}}$), …, ($X_{E_{k}}$, $Y_{E_{k}}$), we calculate the $L_{2}$ distance
between the home of the individual and the external location. Then we choose a
function $f(x)$ which is strictly decreasing in the positive real domain. Then
we weigh the probability of the individual being assigned the external
location by the value $f(D((X_{h},Y_{h}),(X_{E_{k}},Y_{E_{k}})))$ where $D$ is
the $L_{2}$ distance in 2 dimensions.

##  4 Experiments and Evaluation

###  4.1 Datasets

As mentioned in Our Framework section, we use various datasets as input to our
framework. We use sample survey data (micro-data) from India Human Development
Survey-II Desai et al. (2018), marginals from Census, 2011 Office of the
Census Commissioner of India (2011a), Data for employment from NSS National
Sample Survey Office,NSSO (2012), and population density from GADM grid
population density dataset J. Hijmans (2018). The IHDS-II, 2011-12 is a
nationally representative, multi-topic survey of 42,152 households in 1,503
villages and 971 urban neighbourhoods across India. We have used micro-data
from two datasets from this survey, namely Individual and Household as input
to the CTGAN model. The Census of India 2011 was done across 28 states and 8
union territories, covering 640 districts, 5,924 sub-districts, 7,935 towns
and more than 600,000 villages. We use marginal data from Census 2011 as input
to IPU (The Census micro-data is not publicly available). The NSS collects
data through nationwide sample household surveys on various socio-economic
subjects. We have used employment data from this survey. GADM provides maps
and spatial data for all countries and their sub-divisions. We used population
density data for India to place the generated synthetic population across
geography. In the next section, we describe how to use these different
datasets as input to our hybrid model to generate a synthetic population for a
district of India.

###  4.2 Generating Synthetic Population

A synthetic population for Agent-based disease modelling should have
demographic, socioeconomic, health-related and geographic variables for each
individual in the population. We will now describe the steps for generating a
synthetic population for the district of Mumbai, located in the state of
Maharashtra in India. As discussed in the previous section, we use marginals
for individual and household attributes from Census 2011. For individuals, the
relevant attributes are age, sex 222We used the same terminology as used in
the Census of India, religion and caste. For households, the attribute is
household size, i.e. the number of members in the household. We also use a
subset of IHDS-II dataset obtained by filtering for individuals and households
which are situated in the state of Maharashtra. We use the filtered datasets
for Maharashtra because the Mumbai dataset in IHDS-II contains a few hundred
samples, which are not enough for our model to generate quality synthetic
population. We then use IPU which use the marginals and the microdata to
generate households of varying sizes. Each household has a certain number of
individual family members, each with their own age, sex, religion and caste.
There may be some shared attributes between family members like common
religion and caste. Further the distribution of age and sex of members within
a family of a given size resembles that of a typical family of the given size
that one would expect to see in the real world. We then sample latitude
longitude pairs using the method described in Section 3.2 for each household.
The same household latitude and longitude is attached to each member of the
household.
We add individual level attributes like weight, height, comorbidities and job
description. For weight, height and comorbidities, we use a CTGAN model
conditioned on age and sex. The CTGAN model is trained on the subset of IHDS-
II dataset containing individuals situated within the state of Maharashtra.
For individuals below the age of 3, ’Homebound’ is assigned as the job
description. For individuals above the age of 3 and below the age of 18,
’Student’ is assigned as the job description. For individuals, the job
description is sampled with replacement from the list of job descriptions of
the individuals in the IHDS-II subset. The weight assigned to each possible
job description is linearly proportional to the distribution observed in the
IHDS-II subset for Maharashtra.
Next, we generate synthetic workplaces, schools and public places for our
synthetic individuals. The number of workplaces, schools and public places is
a parameter that varies for each district. Each workplace has three
attributes, workplace type, latitude and longitude. The attribute workplace
type is generated by sampling (with replacement) from the list of jobs in the
IHDS-II subset. Latitude and longitude pairs are sampled with the method
described in Section 3.2. Schools are obtained by filtering for a subset of
workplaces with workplace types set to ’Teacher’. Next, we assign schools to
students and workplaces to individuals who are not homebound or students.
Assigning schools in straightforward as we use the method from Section 3.3.
The final synthetic population has demographic data (e.g., age, height etc.),
disease data (e.g., chronic heart disease, diabetes etc.), family data,
geographic location for workplaces, schools etc., and socio-economic
data(e.g., job, religion etc.). In total, we generate a synthetic population
with 44 columns. A sample of our synthetic population is shown in the
Appendix.

###  4.3 Evaluating Synthetic Population

According to Census of India, 2011 Office of the Census Commissioner of India
(2011b), the district of Mumbai had a total population of 3,085,411, and the
district of Mumbai Suburb had a total population of 9,356,962. We generated
the synthetic population for these two districts combined. We evaluate this
synthetic population and present our results here.

Table 1: Statistical Metrics Test | Score
---|---
Chi-square Test for Sex | 0.99
Kolmogorov-Smirnov for Age | 0.98
Kolmogorov-Smirnov for Height | 0.91
Kolmogorov-Smirnov for Weight | 0.95
Table 2: ML efficacy tests Test | Source Population | Synthetic Population
---|---|---
Linear Regression for Weight | 0.72 | 0.69
MLP Regressor for Weight | 0.78 | 0.76
Linear Regression for Height | 0.69 | 0.65
MLP Regressor for Height | 0.71 | 0.72

Though there are no standard benchmark techniques for evaluating the quality
of synthetic population, previous works have tried different methods. For
example, Wan et al. (2019) uses ML classification tasks to check the
usefulness of synthetic population augmented data. Yue et al. (2018) uses
classification and regression tasks to validate the synthetic population. We
evaluate the synthetic population based on the following three criteria:

  * •

Distribution of individual features in synthetic population match those in
sample surveys, i.e., histogram should look similar

  * •

Joint distribution of selected features in synthetic population match those in
sample surveys, i.e., scatter plot should look similar

  * •

Relationship between features and selected target variable in synthetic
population match the same in sample survey

Figure 2: Histogram for marginal distributions of age, height and weight:
Comparing source population with synthetic population for the combined
districts of Mumbai and Mumbai Suburban, India Figure 3: Violin plots for
marginal distributions of age, height and weight: Comparing source population
with synthetic population for the combined districts of Mumbai and Mumbai
Suburban, India Figure 4: Scatter plots for joint distributions of Age, Height
and Weight: Comparing source population (+) with synthetic population (x) for
the combined districts of Mumbai and Mumbai Suburban, India

Figure 2 and 3 help compare the marginal distributions of attributes of age,
height and weight across the real and the synthetic population for the
combined districts of Mumbai and Mumbai Suburban, India. The marginal
distribution has similar shapes even though the synthetic population is real
scale. Figure 4 juxtaposes the scatter plots of attributes of age, height and
weight for real and synthetic populations. The scatter plots depict the joint
distributions, which are roughly similar across the real and the synthetic
population. Figure 5 helps visualize the geographic distribution of synthetic
households, workplaces and schools in the combined districts of Mumbai and
Mumbai Suburban. Due to space constraints, the population pyramid and the box
plots are placed in the Appendix (Figure 7, Figure 8).

Table 1 shows results for two statistical tests, the two-sample
Kolmogorov–Smirnov test (KSTest) and the Chi-Squared test (CSTest). We use the
chi-square test for columns with discrete, categorical data (sex in our
population). This test returns a $p-$value. A high $p-$value indicates the
confidence that the synthetic data and the real data come from the same
distribution. We also use the Kolmogorov-Smirnov statistic to compare the
marginal distributions of continuous, numerical data (age, height, and weight
in our population). We report a score of 1-(KS statistic). A higher score
again indicates the confidence that the synthetic data and the real data come
from the same continuous distribution.

Another set of tests we conducted to evaluate if our synthetic population can
replace the real population to solve a machine learning problem. We first
attempt to predict the weight based on age, sex and height. Similarly, we
attempt to predict the height based on age, sex, and weight. For this
experiment, we use two ML models, Linear Regression and Multi-layer Perceptron
(MLP) Regressor. First, we train and test the models on real data. Next, we
train the models on the synthetic data and test the model performance on the
real data. The comparative results are shown in Table 2.

(a) Households

(b) Schools

(c) Workplaces

Figure 5: Geographical distribution of (A) Households, (B) Schools and (C)
Workplaces for the combined synthetic population for the districts of Mumbai
and Mumbai Suburban, India

###  4.4 Our Results on BharatSim

BharatSim, an Agent-based model, is capable of allowing for complex social
dynamics. Real-world systems involve interactions between individuals with
different attributes (age, weight, etc.) and geographies. These interactions
lead to emergent phenomena, while events like a pandemic affect individuals
according to their attributes. Agent-based modelling accounts for individual
differences and allows us to simulate scenarios of varying complexity. These
simulations can guide policy-level interventions (e.g. lockdowns).

The synthetic population gives us information on the homes and workplaces of
all agents in the population, and the individual’s movements in a day are
charted by defining a “schedule” specific to that individual, which decides
how individuals move back and forth between their homes and workplaces or
schools during a day. The infection spreads stochastically between individuals
who share a location at any given time at rates determined by the infection
parameters. In this model, we have constructed a simple SIR compartmental
structure Tolles & Luong (2020) describing a hypothetical disease: Susceptible
(S) individuals can be infected at a rate proportional to the fraction of
infected (I) individuals they are in contact with. The constant of
proportionality, which determines how infectious the disease is, is age-
stratified. Once an agent is infected, it can infect other individuals until
they recover. The time to recovery is assumed to be drawn from an exponential
distribution with a mean of 7 days. For a simulated lockdown, individual
agents’ schedules are modified so that they remain at home. All agents are
required to follow this rule, except those that are designated by the
synthetic population as essential workers and those who have a low probability
of adherence to policy-level interventions. Whether or not an individual falls
into the latter category is decided by comparing a random coin toss to their
”Adherence to interventions” value. The reduction in the number of people in
different network locations thus reduces contact between individuals and
restricts the spread of the disease. In our simulations, we have considered
two counterfactual scenarios, one in which a lockdown is imposed when the
number of active cases is 1% and 2% of the total population, respectively. As
can be seen in Figure 6, the lockdown flattens the curve, keeping the peak in
active cases low. Additionally, starting the lockdown earlier is much more
effective in curtailing the spread of the disease in our simulation. The
results are averaged over 20 runs, with the individual runs in the background.

Figure 6: No. of recovered individuals (left) and active infections over time

##  5 Conclusion and Future Work

In this paper, we propose, SynthPop++, a novel hybrid framework for generating
synthetic populations at various scales ranging from a few thousand to a
billion individuals. We also suggest a combination of metrics to verify the
generated synthetic population.

Future work would include the exploration of methods to generate a synthetic
population without the need for sample survey data, as well as the modelling
of complex features such as economic variables. Another target would be a
realistic intra-city geographical distribution of individuals depending on
their demographic and socio-economic attributes. Additionally, some analysis
tasks might require a dataset with a specific combination of attributes for
which a survey has not been conducted. A method, which can combine two or more
datasets with different, partially overlapping sets of attributes, can be used
to generate a synthetic population with the specific set of attributes.

#### Acknowledgments

This work was funded by Mphasis F1 foundation. The authors thank Philip
Cherian (Ashoka University) and Gautam I. Menon (Ashoka University and IMSc)
for useful discussions. The authors also thank the anonymous referees for
their useful suggestions.

## References

  * Bonabeau (2002) Eric Bonabeau.  Agent-based modeling: Methods and techniques for simulating human systems.  _Proceedings of the National Academy of Sciences_ , 99(3):7280–7287, 2002.
  * Choi et al. (2017) Edward Choi, Siddharth Biswal, Bradley A. Malin, Jon Duke, Walter F. Stewart, and Jimeng Sun.  Generating multi-label discrete patient records using generative adversarial networks.  volume 68, 2017.  URL http://dblp.uni-trier.de/db/conf/mlhc/mlhc2017.html#ChoiBMDSS17.
  * Desai et al. (2018) Sonalde Desai, Reeve Vanneman, and National Council of Applied Economic Research.  _India Human Development Survey-II (IHDS-II), 2011-12_.  Inter-university Consortium for Political and Social Research, 2018.
  * Gaughan et al. (2013) Andrea E Gaughan, Forrest R Stevens, Catherine Linard, Peng Jia, and Andrew J Tatem.  High resolution population distribution maps for southeast asia in 2010 and 2015.  _PloS one_ , 8(2):e55882, 2013.
  * J. Hijmans (2018) Robert J. Hijmans.  Database of Global Administrative Areas, 2018.  URL https://gadm.org/data.html.
  * Koller & Friedman (2009) D. Koller and N. Friedman.  _Probabilistic Graphical Models: Principles and Techniques_.  2009\.  URL https://books.google.co.in/books?id=7dzpHCHzNQ4C.
  * Ministry of Education (2019) GoI Ministry of Education.  Steps taken by government to provide education to poor student, Jul 2019\.  URL https://pib.gov.in/PressReleasePage.aspx?PRID=1578389.
  * National Sample Survey Office,NSSO (2012) National Sample Survey Office,NSSO.  NSS 68th Round, 2012.  URL http://www.icssrdataservice.in/datarepository/index.php/catalog/91.
  * Office of the Census Commissioner of India (2011a) Office of the Census Commissioner of India.  Census Tables, 2011a.  URL https://censusindia.gov.in/census.website/data/census-tables.
  * Office of the Census Commissioner of India (2011b) Office of the Census Commissioner of India.  Districts of Maharashtra, 2011b.  URL https://www.census2011.co.in/census/state/districtlist/maharashtra.html.
  * Office of the Registrar General, India (2019) Office of the Registrar General, India.  Centre of India, 2021 - Circular No. 6, 2019.  URL https://censusindia.gov.in/nada/index.php/catalog/40515/download/44147/ORGI_circular006_2021.pdf.
  * Tolles & Luong (2020) Juliana Tolles and ThaiBinh Luong.  Modeling Epidemics With Compartmental Models.  _JAMA_ , 323(24):2515–2516, 06 2020.  ISSN 0098-7484.  doi: 10.1001/jama.2020.8420.  URL https://doi.org/10.1001/jama.2020.8420.
  * Wan et al. (2019) Colin Wan, Zheng Li, Alicia Guo, and Yue Zhao.  Sync: A unified framework for generating synthetic population with gaussian copula, 2019.  URL https://arxiv.org/abs/1904.07998.
  * Xu et al. (2019) Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni.  Modeling tabular data using conditional gan.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Ye et al. (2009) Xin Ye, Karthik Konduri, Ram Pendyala, Bhargava Sana, and Paul Waddell.  Methodology to match distributions of both household and person attributes in generation of synthetic populations.  01 2009.
  * Yue et al. (2018) Yang Yue, Ying Li, Kexin Yi, and Zhonghai Wu.  Synthetic data approach for classification and regression.  2018\.  URL http://dblp.uni-trier.de/db/conf/asap/asap2018.html#YueLYW18.
  * Zhang et al. (2014) Jun Zhang, Graham Cormode, Cecilia M. Procopiuc, Divesh Srivastava, and Xiaokui Xiao.  Privbayes: private data release via bayesian networks.  2014\.  URL http://dblp.uni-trier.de/db/conf/sigmod/sigmod2014.html#ZhangCPSX14.

##  Appendix A Appendix

###  A.1 Synthetic Population Summary Statistics and Charts

Figure 7: Population pyramid for joint distribution of gender and age:
Comparing source population with synthetic population for the combined data
for the districts of Mumbai and Mumbai Suburban, India Figure 8: Box plots for
marginal distributions of age, height and weight: Comparing source population
with synthetic population for the combined data for the districts of Mumbai
and Mumbai Suburban, India

###  A.2 Sample Population

A sample of ten individuals in the synthetic population is reproduced below.
The columns are split to fit in the page.

Age | SexLabel | Height | Weight | HHID | H_Lat | H_Lon | District | AdminUnitName
---|---|---|---|---|---|---|---|---
56 | Male | 165.970 | 44.381 | 5.190e+10 | 19.024 | 72.911 | Mumbai | M/E
10 | Male | 138.150 | 34.388 | 5.190e+10 | 19.074 | 72.832 | Mumbai | H/W
3 | Female | 65.460 | 7.205 | 1.038e+11 | 19.258 | 72.867 | Mumbai | R/N
63 | Male | 170.400 | 59.962 | 1.038e+11 | 19.046 | 72.933 | Mumbai | M/E
37 | Female | 143.530 | 60.381 | 5.190e+10 | 19.189 | 72.837 | Mumbai | P/N
46 | Male | 163 | 63.064 | 1.038e+11 | 19.071 | 72.889 | Mumbai | L
35 | Male | 165.660 | 70.111 | 1.038e+11 | 19.073 | 72.906 | Mumbai | N
7 | Female | 119.680 | 21.093 | 1.038e+11 | 19.013 | 72.886 | Mumbai | M/W
29 | Male | 152.520 | 59.198 | 5.190e+10 | 19.131 | 72.898 | Mumbai | S
74 | Female | 169.560 | 37.874 | 5.190e+10 | 18.952 | 72.797 | Mumbai | D
AdminUnitLatitude | AdminUnitLongitude | Religion | Caste | JobLabel | JobID | WorkPlaceID
---|---|---|---|---|---|---
19.056 | 72.922 | Hindu | other | Carpenters | 81 | 2.001e+12
19.056 | 72.835 | Hindu | other | Student | 199 | 0
19.120 | 72.852 | Hindu | other | Homebound | 0 | 0
19.056 | 72.922 | Hindu | other | Homebound | 0 | 0
19.188 | 72.842 | Hindu | other | Homebound | 0 | 0
19.070 | 72.879 | buddhist | other | Labour nec | 99 | 2.001e+12
19.084 | 72.906 | Hindu | SC | Construction | 95 | 2.001e+12
19.061 | 72.899 | Hindu | other | Student | 199 | 0
19.139 | 72.930 | Hindu | other | Construction | 95 | 2.001e+12
18.963 | 72.813 | Hindu | other | Construction | 95 | 2.001e+12
W_Lat | W_Lon | essential_worker | Adherence_to_Intervention | PublicTransport_Jobs | school_id
---|---|---|---|---|---
19.036 | 72.867 | 0 | 0.900 | 1 | 0
nan | nan | 0 | 0.800 | 1 | 2.001e+12
nan | nan | 0 | 1 | 1 | 0
nan | nan | 0 | 1 | 1 | 0
nan | nan | 0 | 0 | 1 | 0
19.071 | 72.890 | 0 | 0.900 | 1 | 0
19.192 | 72.865 | 0 | 0 | 1 | 0
nan | nan | 0 | 1 | 1 | 2.001e+12
19.155 | 72.883 | 0 | 0.200 | 1 | 0
19.024 | 72.850 | 0 | 1 | 1 | 0
school_lat | school_long | public_place_id | public_place_lat | public_place_long | Agent_ID | PSUID
---|---|---|---|---|---|---
nan | nan | 3.001e+12 | 19.024 | 72.910 | 5.191e+10 | 1
19.177 | 72.867 | 3.001e+12 | 19.179 | 72.828 | 5.191e+10 | 20
nan | nan | 3.001e+12 | 19.098 | 72.844 | 5.191e+10 | 9
nan | nan | 3.001e+12 | 19.148 | 72.842 | 5.190e+10 | 14
nan | nan | 3.001e+12 | 19.189 | 72.830 | 5.190e+10 | 17
nan | nan | 3.001e+12 | 19.092 | 72.889 | 5.191e+10 | 5
nan | nan | 3.001e+12 | 19.059 | 72.895 | 5.190e+10 | 8
18.962 | 72.838 | 3.001e+12 | 19.223 | 72.866 | 5.191e+10 | 22
nan | nan | 3.001e+12 | 19.129 | 72.913 | 5.191e+10 | 13
nan | nan | 3.001e+12 | 19.046 | 72.851 | 5.190e+10 | 1

M_Fever | M_Diarrhea | M_Cataract | M_Heart_disease | M_Diabetes | M_Leprosy | M_Cancer
---|---|---|---|---|---|---
0 | 0 | 0 | 0 | 0 | 0 | 0
0 | 0 | 0 | 0 | 0 | 0 | 0
0 | 0 | 0 | 0 | 0 | 0 | 0
0 | 0 | 0 | 0 | 0 | 0 | 0
0 | 0 | 0 | 0 | 0 | 0 | 0
0 | 0 | 0 | 0 | 0 | 0 | 0
0 | 0 | 0 | 0 | 0 | 0 | 0
0 | 0 | 0 | 0 | 0 | 0 | 0
0 | 0 | 0 | 0 | 0 | 0 | 0
0 | 0 | 0 | 0 | 0 | 0 | 0

M_Asthma | M_Paralysis | M_Epilepsy
---|---|---
0 | 0 | 0
0 | 0 | 0
0 | 0 | 0
0 | 0 | 0
0 | 0 | 0
0 | 0 | 0
0 | 0 | 1
0 | 0 | 1
0 | 0 | 1
0 | 0 | 0