# Tele-Aloha: A Low-budget and High-authenticity Telepresence System Using
Sparse RGB Cameras

Hanzhang Tu  0009-0003-7555-9546 Tsinghua UniversityBeijingChina
<EMAIL_ADDRESS>,  Ruizhi Shao  0000-0003-2188-1348 Tsinghua
UniversityBeijingChina<EMAIL_ADDRESS>,  Xue Dong  BOE
Technology GroupBeijingChina<EMAIL_ADDRESS>,  Shunyuan Zheng
0000-0001-5056-614X Harbin Institute of TechnologyWeihaiChina
<EMAIL_ADDRESS>,  Hao Zhang  BOE Technology GroupBeijingChina
<EMAIL_ADDRESS>,  Lili Chen  BOE Technology GroupBeijingChina
<EMAIL_ADDRESS>,  Meili Wang  BOE Technology GroupBeijingChina
<EMAIL_ADDRESS>,  Wenyu Li  BOE Technology GroupBeijingChina
<EMAIL_ADDRESS>,  Siyan Ma  BOE Technology GroupBeijingChina
<EMAIL_ADDRESS>,  Shengping Zhang  0000-0001-5200-3420 Harbin Institute of
TechnologyWeihaiChina<EMAIL_ADDRESS>,  Boyao Zhou  0009-0004-4583-2676
Tsinghua UniversityBeijingChina<EMAIL_ADDRESS>and  Yebin Liu
0000-0003-3215-0225 Tsinghua UniversityBeijingChina
<EMAIL_ADDRESS>
(2024)

###### Abstract.

In this paper, we present a low-budget and high-authenticity bidirectional
telepresence system, Tele-Aloha, targeting peer-to-peer communication
scenarios. Compared to previous systems, Tele-Aloha utilizes only four sparse
RGB cameras, one consumer-grade GPU, and one autostereoscopic screen to
achieve high-resolution (2048x2048), real-time (30 fps), low-latency (less
than 150ms) and robust distant communication. As the core of Tele-Aloha, we
propose an efficient novel view synthesis algorithm for upper-body. Firstly,
we design a cascaded disparity estimator for obtaining a robust geometry cue.
Additionally a neural rasterizer via Gaussian Splatting is introduced to
project latent features onto target view and to decode them into a reduced
resolution. Further, given the high-quality captured data, we leverage
weighted blending mechanism to refine the decoded image into the final
resolution of 2K. Exploiting world-leading autostereoscopic display and low-
latency iris tracking, users are able to experience a strong three-dimensional
sense even without any wearable head-mounted display device. Altogether, our
telepresence system demonstrates the sense of co-presence in real-life
experiments, inspiring the next generation of communication.

videoconferencing, telepresence, telecommunication, real-time free-view
synthesis, human performance rendering

††journalyear: 2024††copyright: rightsretained††conference: Special Interest
Group on Computer Graphics and Interactive Techniques Conference Conference
Papers ’24; July 27-August 1, 2024; Denver, CO, USA††booktitle: Special
Interest Group on Computer Graphics and Interactive Techniques Conference
Conference Papers ’24 (SIGGRAPH Conference Papers ’24), July 27-August 1,
2024, Denver, CO, USA††doi: 10.1145/3641519.3657491††isbn:
979-8-4007-0525-0/24/07††ccs: Computing methodologies Perception††ccs:
Computing methodologies Mixed / augmented reality

##  1\. Introduction

Figure 1. We design a low-budget telepresence system using only 4 consumer RGB
cameras. The overall system is at an affordable price of around $15,000.

Since ancient times, communication technology has always been one of the most
important driving forces to promote innovation. Recent years have witnessed
the advancement of quality and availability of synchronous telecommunication
with the help of the extraordinary development of the Internet and consumer
electronics. High-quality video conferencing systems like Zoom, FaceTime, and
Teams have found their place in various scenarios, even holding dominant
positions in some. More recently, immersive telepresence systems (Clemm et
al., 2020) envisaged for 6G (Strinati et al., 2019) have attracted increasing
interest due to their potential to achieve co-presence, which means
individuals who are physically thousands of miles apart have the feeling of
occupying a shared space.

For interpersonal communication, upper-body motion, including but not limited
to facial expressions, eye gaze, hand gestures, and arm movement, dominates
the body language (Alleva et al., 2014; De Gelder et al., 2015). In other
words, the most critical visual cues for daily communication range mainly in
upper human body regions, which makes the telepresence system that
specifically focuses on the upper body a more reasonable choice for research.
The latest examples such as Starline (Lawrence et al., 2021) and VirtualCube
(Zhang et al., 2022) have showcased unprecedented levels of immersive
experience. However, these commercial systems necessity intricate hardware
devices and customized physical setups (i.e. booths or rooms), preventing them
from spreading to average consumers. Moreover, numerous input sensors that
appear in previous systems, especially depth sensors, also increase the burden
of computation and transmission, resulting in an urgent requirement for
multiple high-end graphics cards.

In this paper, we present Tele-Aloha, A low-budget and high-authenticity
telepresence prototype system equipped with only four RGB cameras, one
consumer-grade GPU, and one autostereoscopic screen targeting upper-body peer-
to-peer communication scenarios. With this simplest hardware configuration, we
are committed to enabling an immersive experience via a carefully designed
system construction. The four calibrated 4K RGB cameras, as all input sensors,
capture the detailed appearance of users, which is extremely sparse compared
to most of the existing systems. In addition to the sparsity of capture
devices, no depth sensor (neither TOF nor active structured light camera)
appears in our system for the reason that i) they are susceptible to the
environment including illumination conditions and reflection in the scene; ii)
depth observations suffer from incompleteness and noise in low reflectivity
areas and scenes with complex material characteristics (discussed in Sec.
4.1), resulting in a degradation of the robustness and versatility of the
system. Also, high-quality depth sensors are still costly compared to RGB
cameras. Notably, all mentioned components are readily available commercial
products at affordable prices, in total around $15,000, releasing the
potential of becoming a consumer-grade product and being mass-produced.

As the core of the proposed system, we introduce a novel view synthesis
algorithm, given two selected perspective views by eye tracking, to generate
photo-realistic and high-fidelity rendering images under a highly sparse RGB-
only setup. Recently, Gaussian Splatting (Kerbl et al., 2023; Luiten et al.,
2024; Wu et al., 2024; Zheng et al., 2024) has made a significant breakthrough
in the area of novel view synthesis for its high-resolution rendering and
real-time inference. However, it typically requires per-scene parameter
optimization for several minutes, which makes it unsuitable for real-world
application. In contrast, we aim to build an instant novel view synthesis
system that could generalize to any unseen person through a massive human data
learning process. In our system, we focus on upper body communication,
imposing a “zoom-in” effect of artifacts and jitters, which increases the
difficulty of novel view synthesis. Moreover, under a sparse camera setting,
the wide baseline of the reference cameras also challenges the effectiveness
of traditional stereo matching methods (Teed and Deng, 2020; Lipson et al.,
2021).

To this end, we carefully design a cascaded disparity estimator to obtain the
geometry of the target user, which handles the problem of disparity estimation
for stereo cameras with a wide baseline. Given a more robust geometry cue,
pixel-aligned features extracted from input images can be projected to the
selected novel views with our generalizable 3D Gaussian Splatting rasterizer
(Kerbl et al., 2023). We propose to splat latent features, instead of
projecting spherical harmonics within 3DGS, onto the target view in a reduced
resolution so that the rendering result can be further completed with a
decoder network. However, the decoded image is in a lower resolution for the
reason of fast and completed rendering. Thus, we further take advantage of the
high-resolution RGB input with a weighted blending mechanism according to
surface visibility in the source view, providing strong cues to a refinement
module for a high-quality rendering. The proposed algorithm balances well the
trade-off between rendering quality and efficiency. To summarize, our
contributions include:

(i) A lightweight, consumer-affordable 3D telepresence system, using only four
RGB cameras and no depth sensors to achieve high-resolution ($2048\times
2048$), real-time (30 fps), low-latency (less than 150 ms), and distant
communication.

(ii) A cascaded stereo matching strategy to improve the robustness of depth
estimation under wide baseline camera setting.

(iii) Rasterizing latent code from the source view to the target view via a
generalizable Gaussian Splatting, together with a decoder network, to
guarantee the rendering completeness.

(iv) Taking advantage of original high-resolution input by blending it into
the novel view, along with the decoded features rasterized by 3DGS, for
refinement to realize fast high-quality rendering.

Table 1. 3D Telepresence Systems. Since Holoportation, MetaStream, and Live4d capture and transmit volumetric videos rather than 2D free-view videos, the resolution value cannot be quantified exactly. F.B. and U.B. stand for full-body and upper-body respectively. ∗ LookinGood develops two systems for full-body and upper-body specifically, we only list the setting of the upper-body system. Systems | Input setting | Efficiency | Resolution | GPU equipment | Complexity
---|---|---|---|---|---
Holoportation (Orts-Escolano et al., 2016) | 8 $\times$ RGBD | 34 fps | - | 10 $\times$ Titan X | Medium (F.B.)
LookinGood∗ (Martin-Brualla et al., 2018) | 1 $\times$ RGBD | 34 fps | 512 $\times$ 1024 | 1 $\times$ Titan V | High (U.B.)
Starline (Lawrence et al., 2021) | 3 $\times$ RGBD | 60 fps | 1600 $\times$ 1200 | 2 $\times$ RTX 6000 + 2 $\times$ Titan RTX | High (U.B.)
VirtualCube (Zhang et al., 2022) | 6 $\times$ RGBD | 30 fps | 1280 $\times$ 960 | 2 $\times$ RTX 3090 + 1 $\times$ RTX 2080 | High (U.B.)
MetaStream (Guan et al., 2023) | 4 $\times$ RGBD | 30 fps | - | 4 $\times$ Jetson Nano + 1 $\times$ RTX 2080S | Medium (F.B.)
AI-mediated (Stengel et al., 2023) | 1 $\times$ RGB | 24 fps | 512 $\times$ 512 | 1 $\times$ RTX 6000 + 1 $\times$ RTX 4090 | Low (Head)
Live4d (Zhou et al., 2023) | 20 $\times$ RGB | 24 fps | - | 4 $\times$ RTX 3090 + 1 $\times$ RTX 4090 | Medium (F.B.)
Ours | 4 $\times$ RGB | 30 fps | 2048 $\times$ 2048 | 1 $\times$ RTX 4090 | High (U.B.)

##  2\. Related Work

The 3D telepresence was identified early on (Raskar et al., 1998; Gibbs et
al., 1999) and has sparked sustained interest due to its immersive user
experience. The existing 3D telepresence systems can mainly be categorized
into three types: head, upper-body, and full-body systems, as listed in Tab.
1. The system that only focuses on the head is the simplest setting due to the
highly symmetrical and non-occlusion nature of heads. (Stengel et al., 2023)
lifts a facial RGB image to 3D space using a triplane based 3D GAN (Trevithick
et al., 2023). The full-body setting (Orts-Escolano et al., 2016; Guan et al.,
2023; Zhou et al., 2023) poses challenges of self-occlusion issues.
Nonetheless, the small effective proportion of humans within image space
results in a higher tolerance for blurry or ghosting artifacts. We target an
upper-body telepresence system similar to Starline (Lawrence et al., 2021) and
VirtualCube (Zhang et al., 2022), which satisfies an ideal communication
scenario covering motion and gesture. However, complicated upper-body
movements introduce severe self-occlusion in this context, making it more
difficult.

The primitive telepresence systems (Gibbs et al., 1999; Kuster et al., 2012)
attempt to capture the appearance and geometry of participants, thus they
typically employ depth sensors (Izadi et al., 2011; Newcombe et al., 2015) to
serve as a geometry proxy. Leveraging depth sensors, (Novotny et al., 2019;
Nguyen-Ha et al., 2022; Neff et al., 2021; Stelzner et al., 2021) facilitate
and speed up the rendering process. For example, some attempts(Maimone et al.,
2012; Newcombe et al., 2015; Yu et al., 2018; Su et al., 2020) directly obtain
textured mesh via depth triangulation or depth fusion to demonstrate 3D
surface. With the development of learning methods, depth maps can be refined,
inpainted, and denoised with neural networks and serve as geometry to
accelerate the neural rendering process. Function4d (Yu et al., 2021b)
integrates dynamic fusion and implicit surface reconstruction to perform real-
time full-body human volumetric capture from four consumer RGBD sensors.
Equipped with similar sensors, VirtualCube (Zhang et al., 2022) builds real-
world cubicles to address immersive 3D video conferences under three various
scenarios. HVS-Net (Nguyen-Ha et al., 2022) warps latent feature of input view
image to target view by using un-projected points of depth map, the feature is
then decoded and refined with CNN based network. LookinGood (Martin-Brualla et
al., 2018) employs volumetric fusion (Curless and Levoy, 1996) of depth inputs
to generate geometry prior for novel view rendering, but accumulative geometry
error is inevitable in fusion methods when increasing image range. FWD (Cao et
al., 2022) refines the depth value of a captured map and warps input images
into the target view. Since the captured depth map plays the role of geometry
cue, the rendering results highly depend on the quality of the captured depth
map, which is sensitive to the lighting environment.

Given the increasing resolution of RGB cameras, more real-world applications
utilize only RGB cameras as input to generalize to different lighting
environments. Image-based rendering (IBR) (Hedman et al., 2018; Riegler and
Koltun, 2021) synthesizes novel views by reasoning a blending weight and a
geometry proxy, which are used to warp source image cues to novel viewpoints.
As for human novel view synthesis, NeuralHumanFVV (Suo et al., 2021) proposes
a neural blending mechanism to conduct image warping and texture blending
based on a neural geometry reconstruction (Saito et al., 2019) from sparse
views. Floren (Shao et al., 2022) realizes a real-time full-body $360^{\circ}$
free-view rendering system with eight RGB cameras. Neural radiance field
(NeRF) (Mildenhall et al., 2020; Zhang et al., 2020; Barron et al., 2022) has
shown impressive results in 4D performance capture (Pumarola et al., 2021;
Shao et al., 2023). However, such methods typically require per-scene
optimization, which restricts their applications in real-time telepresence
systems. The follow-up work (Yu et al., 2021a; Wang et al., 2021; Lin et al.,
2022a) combines the advantages of IBR and NeRF by replacing the input of
implicit function from scene-specific position encoding to image features
aggregated from source views. Despite the progress in generalization, these
methods rely on dense sampling points and still have difficulty achieving
photorealistic results. Recently, 3DGS (Kerbl et al., 2023; Luiten et al.,
2024) has introduced a new promising point-based representation. It
demonstrates a more reasonable mechanism for back-propagating the gradients
alongside real-time rendering efficiency. However, the original 3DGS requires
scene-specific training in minutes. Concurrent work (Zheng et al., 2024;
Charatan et al., 2024; Szymanowicz et al., 2024) attempt to address this
fragmentation by formulating 3DGS on 2D image planes, termed Gaussian maps.
The Gaussian maps trained on large-scale images determine the parameters of 3D
Gaussians in a feed-forward manner rather than an iterative optimization way,
which makes the representation generalizable to the domain of training data.
GPS-Gaussian (Zheng et al., 2024) is the most related method to ours. Although
they have showcased remarkable results in full-body cases, they have no
specific design for the complicated self-occlusion. We summarize the major
distinctions as follows. First, compared with GPS-Gaussian, cameras and
subjects are closer in our system, which leads to much larger disparity. To
address this issue, we propose cascaded disparity estimation for upper-body
setup to overcome the significantly larger parallax (Fig. 8). Second, we
propose to splat latent features followed by a neural decoder instead of RGB
values used in GPS-Gaussian, to mitigate the incompleteness caused by severe
self-occlusion (Fig. 7). Third, we propose a lightweight refinement module
that improves rendering quality without introducing much computational burden
(Fig. 6).

##  3\. System Overview

Figure 2. Schematic diagram of the overall system.

[sys model]System model

The schematic diagram of our system is shown in Fig. 2. We construct the
hardware and software architecture with concentrated attention to optimized
user experience. In general, our system consists of video capture,
transmission, stream decoding, novel view synthesis, and display components.

###  3.1. Overall system setup

In our design, we are committed to providing users with a seamless and
unencumbered communication experience. The users are accommodated in front of
the display with a typical 1.25m eye-to-display distance. The virtual remote
user is rendered with a $180^{\circ}$ rotation about the eye-to-display
midpoint as described in (Lawrence et al., 2021). To include important non-
verbal cues like hand gestures on a relatively smaller screen size (27-inch in
our implementation), we apply a scaling factor of 0.5 for rendering the upper
body. To avoid vergence-accommodation conflict and maximize the user’s
comfort, we choose to put the virtual user near the display plane rather than
behind the screen. The setup may break the eye contact to some extent, so we
use the center point of the two eyes as the scaling center to minimize the
defocus of eye contact.

During conversation, the optimal range that the user can move left and right
is about 0.8m, which means the view angle is about 40°. The range of forward
and backward movement is about 0.3m. The limitation is mainly due to the
characteristics of the display hardware and the capture volume of cameras.

###  3.2. Multiview capture system

Our system captures 4 video streams using 4 RGB cameras. Since we get rid of
costly and illumination-sensitive depth sensors, we propose to apply stereo
matching (Lipson et al., 2021) of dual-camera setting as a fast and accurate
geometry proxy. As shown in Fig 2, we place four hardware-synchronized RGB
cameras on the left, right, and above the displayer to provide complete
coverage of the user. Each camera(BFS-U3-123S6C-C) has a high resolution of
$4096\times 3000$ and operates at 30 Hz. Two upper cameras make up a dual-view
stereo with a narrow baseline, while the other pair has a wider one. The
former provides accurate geometric information to the latter as an
initialization for robust depth estimation. Notice that although we choose
cameras with global shutter in our implementation, our system is not
specifically designed for such cameras. Rolling shutter is also feasible when
subject motion is not extremely fast.

Precise geometric knowledge of the system including camera-camera and camera-
display poses is required for stereo matching and novel view synthesis. We
calibrate cameras adopting the method proposed by (Zhang, 2000), which
minimized reprojection error over images of a planar checkerboard-like target.
Also, bundle adjustment is utilized for overall refinement, providing the
final estimation of camera intrinsics and extrinsics. With respect to camera-
display calibration, method described in (Hesch et al., 2010) is adopted to
solve relative transform between the display and cameras. In addition to
camera geometries, we also color-calibrate four RGB cameras with a standard
ColorChecker. We counteract the effects of ambient lighting and color
characteristics by adjusting gamma and a $3\times 3$ color correction matrix
(CCM). The color consistency between cameras ensures the robustness and
fidelity of subsequent algorithms.

###  3.3. Data compression and transmission

Captured data will be encoded into H.265 streams and transmitted over the
Internet using low-latency WebRTC technology (Johnston and Burnett, 2012).
Hardware-based encoder and decoder (NVIDIA, 2024) are deployed for data stream
encoding and decoding. With end-to-end encoding/decoding offloaded to
NVENC/NCDEC, the graphics/CUDA cores and the CPU cores are free for other
operations. Before encoding, all four input images are square-cropped and
concatenated into a large one, resulting in a single input image into NVENC
with a resolution of $6000\times 6000$. As for audio, capture is performed in
two-channel stereo, 16-bit and 48000 Hz. Timestamps are also inserted into the
stream for audio-video synchronization. We measured an overall network
bandwidth of 100 Mbit/s which is feasible for most enterprise or even in-home
users.

###  3.4. Novel view synthesis and display

On receiving a new frame from remote client, a video matting module (Lin et
al., 2022b) is engaged as a pre-processing step. Meanwhile, we track the 3D
eye positions of the users, providing the parameters of the viewpoints for our
algorithm to generate novel views. For eye tracking, two of the existing
camera views on the top are used as input (cam0 and cam1 in Fig. 3) and we use
BlazeFace [Bazarevsky et al. 2019] finetuned with our own data to detect 2D
iris positions, which are further triangulated into 3D positions. Given
foreground images and tracked eye positions, the novel view synthesis
(elaborated in Sec. 4) is performed.

As for the display, we prefer an autostereoscopic screen powered by eye
tracking for its full-scale display field and much higher resolution with
respect to the quilt format in light field holography device (LookingGlass,
2021), displaying all views from leftmost to rightmost into one single image.
In theory, all autostereoscopic screens satisfy our system, e.g. 32-inch 3D
display screen of Beijing Shiyan Technology Co., Ltd or ThinkVision-27-3D
(Lenovo, 2023) with a cost lower than $3,000.

With the highly-optimized, well-implemented pipeline, our system has the
capability of handling all workloads with as little as one consumer-grade GPU
(NVIDIA RTX 4090) at over 30 FPS. This allows us to well organize the input
and output of our novel view synthesis algorithm. We achieve an end-to-end
latency of less than 150 ms in our prototype system, where two terminals are
in the same LAN, which gives good interactivity to the participants.

##  4\. Efficient novel view synthesis

In this section, we present an efficient novel view synthesis method for the
upper body with only four RGB images. Our algorithm achieves photo-realistic
novel view generation in real time. Instead of solving the highly ill-posed
problem of geometry reconstruction, we turn to stereo constraint across input
views and novel-view-centered neural rendering. All computationally intensive
neural networks focus on targets in the 2D domain and thus can be efficiently
executed. The pipeline of the proposed method is illustrated in Fig. 3. We
first introduce a Cascaded Disparity Estimation (Sec. 4.1), in which the
disparity of two closer cameras is predicted and serves as an initialization
of the disparity of two farther ones. In Sec. 4.2, we propose a Neural
Rasterizer, which projects latent features from source views to the target
view via Gaussian Splatting, and a decoder network for a completed rendering
in reduced resolution. Further, Occlusion-aware Rendering Refinement is
presented in Sec. 4.3, which fuses low-resolution rendering with weighted
blending images from high-resolution input views, to synthesize high-quality
images in resolution of $2048\times 2048$ less than 25ms with a single NVIDIA
RTX 4090.

Figure 3. The pipeline of the proposed novel view synthesis algorithm. Given
four input views that form two stereo pairs, we first estimate the disparity
maps in the perspectives of these input views in a cascaded manner. Then,
together with feature maps, scale maps, and opacity maps extracted by an image
encoder, the pixels are lifted to 3D to form pixel-aligned 3DGS points. After
that, feature vectors are rasterized to selected novel views, followed by a
decoder that turns feature maps back into RGB images. Finally, a refiner
module is applied to upsample the novel view rendering.

###  4.1. Cascaded Disparity Estimation

Obtaining a stable geometry is crucial for the robustness of novel view
synthesis, especially in the case of sparse input views (4 views in our
system). To address this, we design a cascaded multi-view capture system as
illustrated in Fig. 3. Two pairs of horizontally arranged cameras with
different lengths of baseline are present in our system, denoted as upper pair
(cam0, cam1) and lower pair (cam2, cam3), respectively. The upper pair has a
smaller baseline, which is more robust for stereo matching, while the lower
pair has a larger field of view. Therefore, we propose to estimate the
disparity of the upper pair first and adopt it as initialization to estimate
the disparity of the lower pair with more stability.

Specifically, we adopt RAFT-stereo (Lipson et al., 2021) to estimate a
disparity $\hat{d}_{u}$ for each pixel $\left(u,v\right)$ in reference view
that matches its counterpart $(u+\hat{d}_{u},v)$ in the target view, which is
formulated as

(1)
$\langle\hat{\mathbf{d}}_{1},\hat{\mathbf{d}}_{2}\rangle=\mathcal{F}_{d}\left(\mathbf{I}_{1},\mathbf{I}_{2},\mathbf{d}_{1}^{init},\mathbf{d}_{2}^{init}\right),$

where $\mathbf{d}_{1}^{\text{init}},\mathbf{d}_{2}^{\text{init}}$ are
disparity initialization for the iterative update process. We first apply the
module $\mathcal{F}_{d}$ in Eq. 1 to upper pair with zero initialization

(2)
$\langle\hat{\mathbf{d}}_{cam0},\hat{\mathbf{d}}_{cam1}\rangle=\mathcal{F}_{d}\left(\mathbf{I}_{cam0},\mathbf{I}_{cam1},\mathbf{0},\mathbf{0}\right).$

The predicted disparity of cam0 is converted to a depth map and then lifted to
a 3D point cloud. These points are rasterized to the viewpoints of cam2 and
cam3. Z-buffers are extracted and converted back to disparity maps
$\mathbf{d}_{cam\\{2,3\\}}^{\text{init}}$. Disparity maps of lower pair can
now be estimated with more robustness and stability

(3)
$\langle\hat{\mathbf{d}}_{cam2},\hat{\mathbf{d}}_{cam3}\rangle=\mathcal{F}_{d}\left(\mathbf{I}_{cam2},\mathbf{I}_{cam3},\mathbf{d}_{cam2}^{\text{init}},\mathbf{d}_{cam3}^{\text{init}}\right).$

As shown in Fig.3, three disparity maps
$\hat{\mathbf{d}}_{\\{cam0,cam2,cam3\\}}$ are converted to depth maps
$\hat{\mathbf{z}}_{\\{cam0,cam2,cam3\\}}$, and further transformed into
Gaussian Splatting point cloud in neural rasterization.

The visualization of the depth reconstruction is demonstrated in Fig.4, which
shows that using only RGB cameras (second row), our method produces
competitive depth maps in comparison to that captured by a TOF sensor (third
row) which costs up to 1500 USD (Lucid Helios 2 (Lucid, 2024)). Notably, depth
observations from the TOF sensor are incomplete in areas like the logo on
clothes, hair and glass jar.

Figure 4. Comparison of depth map sources. (a) Stereo rectified pair input to
the disparity estimator; (b) Depth maps predicted by the disparity estimator
mentioned in Sec. 4.1; (c) Depth maps captured by a TOF sensor.

###  4.2. Neural Rasterization via 3DGS

To synthesize novel views, we propose to transform three depth maps
$\hat{\mathbf{z}}_{\\{cam0,cam2,cam3\\}}$ into a latent-based 3D Gaussians and
train a pixel-wise image encoder $\mathcal{E}_{src}$ with a neural image
decoder $\mathcal{D}_{novel}$. The image encoder predicts the Gaussian
properties instantly. Then, the latent-based 3DGS is rendered as the latent
map at the novel view, and the neural image decoder is utilized to recover RGB
images.

#### Latent-based Gaussian Splatting.

In the latent-based Gaussian Splatting, each point $p_{i}$ is characterized by
four properties: position $\mathbf{x}_{i}$, latent appearance feature
$\mathbf{f}_{i}$, scale $\mathbf{s}_{i}$, opacity $\alpha_{i}$. The rotation
of Gaussian points is set to an identity matrix. Compared to the original
Gaussian Splatting, we propose to learn the latent feature instead of RGB
values. Our method compresses local content information from the original
image into the high-dimensional latent code. Consequently, each point can
effectively represent a local region, enhancing rendering robustness against
occlusion, depth estimation errors, and camera color noise.

Specifically, Given camera parameters $\Pi$, 3D positions $\mathbf{x}$ of
Gaussian points can be unprojected from depth maps. Then, the pixel-wise image
encoder $\mathcal{E}_{src}$ is constructed to extract image features from
source views. After a shared backbone, three different heads are applied to
the intermediate feature, producing appearance feature
$\mathcal{M}_{f}\in\mathbb{R}^{H\times W\times d}$, scale map
$\mathcal{M}_{s}\in\mathbb{R}^{H\times W\times 3}$ and opacity map
$\mathcal{M}_{\alpha}\in\mathbb{R}^{H\times W\times 1}$, respectively. To
emphasize the geometric awareness for more accurate Gaussian parameter
regression, depth maps are also introduced to the network

(4)
$\langle\mathcal{M}_{f},\mathcal{M}_{s},\mathcal{M}_{\alpha}\rangle=\mathcal{E}_{src}(\mathbf{I}\oplus\mathbf{z}).$

As illustrated in Fig. 3, foreground pixels in source views (i.e. cam0, cam2,
and cam3) are lifted to pixel-aligned Gaussian points $\\{\mathcal{G}\\}$.
These Gaussian points are then differentiably rasterized to the targeted novel
views (left and right perspective of the user) given their projection matrix
$\Pi_{novel}$:
$\mathbf{F}_{novel}=\mathcal{R}_{\mathcal{G}}(\\{\mathcal{G}\\},\Pi_{novel}),$
where $\mathcal{R}_{\mathcal{G}}$ denotes the differentiable Gaussian
splatting rasterizer (Kerbl et al., 2023). The projected feature maps
$\mathbf{F}_{novel}$ carry abundant appearance details from input images.

#### Neural Image Decoder.

Due to sparsity caused by self-occlusion, the projected feature maps are not
necessarily dense. Therefore, a deep neural image decoder
$\mathcal{D}_{novel}$ implemented as a 2D UNet is introduced to address the
artifacts and discontinuities. Since the encoder and the decoder are both
trained on a large-scale dataset, the decoder is capable of inpainting
invisible regions of the novel view from the latent map learned by the
encoder.

Considering runtime efficiency and the receptive field of the network, the
image decoder generates images with a reduced resolution ($1024\times 1024$,
while the full output resolution is $2048\times 2048$) to enhance the
completeness of the inpainted images. Also, the refined feature maps
$\mathbf{F}_{novel}^{ref}$ from the last convolutional layer are extracted for
further upsampling operation:

(5)
$\langle\mathbf{F}_{novel}^{ref},\mathbf{I}_{novel}^{lr}\rangle=\mathcal{D}_{novel}(\mathbf{F}_{novel}).$

We apply the perceptual loss to $\mathbf{I}_{novel}^{lr}$ for better
convergence due to a shorter gradient chain compared with the final full-
resolution rendering produced in Sec. 4.3.

###  4.3. Occlusion-aware Rendering Refinement

We have now obtained a complete and hole-free novel view rendering, but there
is still a gap between satisfaction in terms of resolution. A lightweight
refinement module $\mathcal{D}_{refine}$ is engaged to generate the final
output in full resolution. To fully leverage the high-resolution input of the
system, $\mathcal{D}_{refine}$ takes the refined image feature
$\mathbf{F}_{novel}^{ref}$ and a high-resolution image $\mathbf{I}_{blend}$
blended from source views as input, to produce high-resolution images. As
shown in Fig. 6, the blended image $\mathbf{I}_{blend}$ surpasses the
inpainted image $\mathbf{I}_{novel}^{lr}$ in respect of high-frequency
texture. Meanwhile, it is not necessarily complete due to self-occlusion and
inaccurate geometry prediction. Therefore, fusing the two can give full play
to their complementary advantages.

We first introduce the definite process of input views blending, which is
briefly illustrated in Fig. 5. To begin, we warp all predicted depth maps
$\hat{\mathbf{z}}_{i}$ in Sec. 4.1 onto the novel view, resulting in a
relatively dense fused depth map denoted as $\mathbf{z}_{fused}$. After that,
each pixel in $\mathbf{z}_{fused}$ is projected back to all input views:

(6) $\mathbf{x}_{i}=\Pi_{i}(\Pi_{novel}^{-1}(u,v,\mathbf{z}_{fused}(u,v))).$

Color value and depth value are fetched from each input view using the back-
projected coordinate:

(7)
$\mathbf{c}_{i}=\operatorname{Interp}(\mathbf{I}_{i},\mathbf{x}_{i.uv}),\quad
z_{i}=\operatorname{Interp}(\hat{\mathbf{z}}_{i},\mathbf{x}_{i.uv}),$

where $\operatorname{Interp}(\cdot)$ is a bi-linear sampling function. Similar
to (Lawrence et al., 2021), the blending weights of color values from
different input views are given by:

(8)
$w_{i}=\mathbf{M}_{i}(\mathbf{x}_{i.uv})\cdot\operatorname{Occ}(\mathbf{x}_{i})\cdot\cos\langle\mathbf{r}(u,v),\mathbf{n}_{i}\rangle\cdot\frac{1}{\|\mathbf{x}_{i}^{cam}\|_{2}},$

where $\mathbf{M}_{i}$ is a binary mask labeling valid pixels in input view
$i$, which is obtained by an AND operation of the input mask, appearance
consistency check mask, and edge mask, $\operatorname{Occ}(\cdot)$ is the
signed distance function (SDF) check function to avoid sampling on the
occluded regions

(9) $\operatorname{Occ}(\mathbf{x}_{i})=\begin{cases}1&\text{ if
}|\mathbf{x}_{i.z}-z_{i}|<\delta\\\ 0&\text{ otherwise. }\end{cases},$

$\cos\langle\mathbf{r}(u,v),\mathbf{n}_{i}\rangle$ evaluates the angle between
ray direction corresponded to pixel $(u,v)$ and the surface normal of input
view, and $\mathbf{x}_{i}^{cam}$ denotes the coordinate of the point in
$i^{t}h$ camera’s coordinate system. Finally, all $\mathbf{c}_{i}$ are
weighted-summed:

(10)
$\mathbf{I}_{blend}(u,v)=\frac{1}{\sum_{i=0}^{\upsilon_{num}}w_{i}}\cdot\sum_{i=0}^{\upsilon_{num}}w_{i}\cdot\mathbf{c}_{i}.$

Figure 5. Source views are blended in novel view to acquire high-frequency
texture with occlusion awareness. The blending weights are basically defined
by surface normal, view direction, and PSDF value.  Figure 6.  Qualitative
evaluation of refinement module. LR image (left) lacks details in blue square
while the blended image (middle) suffers from incompleteness in red square.
The final result (right) takes the advantage of both.

Finally, we fuse the two components together:

(11)
$\mathbf{I}_{novel}^{hr}=\mathcal{D}_{refine}(\mathbf{F}_{novel}^{refine}\oplus\mathcal{E}_{hr}(\mathbf{I}_{blend})),$

in which $\mathcal{E}_{hr}$ is an encoder composed of several edge-enhanced
diverse branch blocks (Wang, 2022), $\oplus$ stands for concatenation.
$\mathbf{I}_{novel}^{hr}$ is the overall synthesized high-fidelity novel view
image.

Figure 7.  Comparison on sparse-view synthetic dataset against Floren (Shao et
al., 2022), ENeRF (Lin et al., 2022a), and GPS-gaussian (Zheng et al., 2024).

##  5\. Experiment and Analysis

The proposed method is trained on THuman-Sit (Zhang et al., 2023) and
THuman2.0 (Yu et al., 2021b). We split the THuman-Sit scans into two parts,
4000 for training and 700 for testing, according to (Zhang et al., 2023). We
render source images parallel to our hardware setup and render novel views
randomly among them.

###  5.1. Analysis of Rendering Quality

The quantitative results on the synthetic dataset are listed in Tab. 2. Our
method outperforms other efficient NVS algorithms. The qualitative comparisons
are presented in Fig. 7. The first row shows that our method inpaints high-
quality results in the occlusion region, demonstrating the effectiveness of
our proposed neural rasterization. As shown in the second row, our method
achieves superior rendering quality thanks to the robust cascade disparity
estimation.

###  5.2. Analysis of Depth Reconstruction

For validation of the effectiveness of the cascaded manner, we conduct
experiments with synthetic datasets. As shown in Tab. 3, the cascaded
initialization improves the accuracy of disparity estimation of large baseline
cameras in terms of EPE (end point error) and percentage of small-error
pixels. Fig. 8 shows that with initialization, 3 iterative flow updates can
achieve the same accuracy as 16 iterative updates without initialization,
while fewer iterations without initialization cause significant accuracy
degeneration.

Figure 8. Evaluation of the cascaded disparity estimator. (a) Input views; (b)
w/o cascaded initialization, 3 iterative updates; (c) w/ cascaded
initialization, 3 iterative updates; (d) w/o cascaded initialization, 16
iterative updates.  Figure 9. Failure case on non-Lambertian objects.

###  5.3. Execution Time and System Latency

To maximize the performance, we implemented all components carefully with
CUDA. Neural networks are optimized using TensorRT with fp16 precision. The
overall system is optimized using CUDA Graphs. The disparity estimation takes
4.7 ms, the encoder takes 6.1 ms, the decoder and the refiner together take
9.3 ms, the input views blending takes 1.4 ms, the 3DGS rasterization takes 1
ms, and other operations take 1 ms, which is in total 23.5 ms.

We measured our system’s end-to-end latency in a system containing two
terminals in the same LAN. To determine the latency that users actually
sensed, we capture a timer and calculate the time difference between it and
the displayed result on the other side. We observe an average latency of
around 150 ms, which is acceptable according to (Chen et al., 2004). We report
the detailed latency breakdown of each component of the system in Tab. 4.

Table 2. Quantitative comparison on synthetic dataset. Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
---|---|---|---
FloRen (Shao et al., 2022) | 19.049 | 0.844 | 0.251
ENeRF (Lin et al., 2022a) | 24.203 | 0.891 | 0.160
GPS-Gaussian (Zheng et al., 2024) | 25.204 | 0.909 | 0.112
Ours w/o refinement | 25.642 | 0.916 | 0.171
Ours w/ refinement | 26.543 | 0.928 | 0.095
Table 3. Ablation study on the cascaded disparity module. |  EPE$\downarrow$ |  1pix$\uparrow$ |  3pix$\uparrow$ |  5pix$\uparrow$
---|---|---|---|---
Small baseline | 1.141 | 0.8610 | 0.9749 | 0.9861
Large baseline w/o init | 2.741 | 0.5350 | 0.8302 | 0.9006
Large baseline w/ init | 2.192 | 0.7169 | 0.9051 | 0.9472
Table 4. System latency breakdown.  Component | Latency | Component | Latency
---|---|---|---
Capture | 56 | RTC transmission | 2
Upload to vmem | 11 | Upload to vmem | 1
Debayering | 13 | Decoding | 5
Pre-processing | 5 | View synthesis | 28
Encoding | 5 | Display output | 27
Download to mem | 1 | Total | 149

##  6\. Conclusion

We propose a low budget real-time upper-body communication system, Tele-Aloha
with only four RGB inputs. We carefully design a novel view synthesis
algorithm for an autostereoscopic displayer, including a cascaded disparity
estimation and combination of 3DGS and weighted blending mechanism. On only
one RTX 4090 GPU, we process data capture, stream encoding/decoding, view
synthesis and 2K display with a latency of less than 150 ms.

#### Limitations

Our system sometimes fails on specular objects, e.g., eyeglasses, due to
strong anisotropy which leads to instability of disparity estimation. An
example is shown in Fig. 9. Also, our method potentially suffers from
inaccurate background segmentation, see Fig. 12 and for example.

###### Acknowledgements.

This paper is supported by National Key R&D Program of China (2022YFF0902200),
the NSFC project No.62125107.

## References

  * (1)
  * Alleva et al. (2014) Jessica M Alleva, Carolien Martijn, Anita Jansen, and Chantal Nederkoorn. 2014.  Body language: Affecting body satisfaction by describing the body in functionality terms.  _Psychology of Women Quarterly_ 38, 2 (2014), 181–196.
  * Barron et al. (2022) Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. 2022.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5470–5479.
  * Cao et al. (2022) Ang Cao, Chris Rockwell, and Justin Johnson. 2022.  Fwd: Real-time novel view synthesis with forward warping and depth. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 15713–15724.
  * Charatan et al. (2024) David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. 2024.  pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction. In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Chen et al. (2004) Yan Chen, Toni Farley, and Nong Ye. 2004.  QoS requirements of network applications on the Internet.  _Information Knowledge Systems Management_ 4, 1 (2004), 55–76.
  * Clemm et al. (2020) Alexander Clemm, Maria Torres Vega, Hemanth Kumar Ravuri, Tim Wauters, and Filip De Turck. 2020.  Toward truly immersive holographic-type communication: Challenges and solutions.  _IEEE Communications Magazine_ 58, 1 (2020), 93–99.
  * Curless and Levoy (1996) Brian Curless and Marc Levoy. 1996.  A volumetric method for building complex models from range images. In _Proceedings of the 23rd annual conference on Computer graphics and interactive techniques_. 303–312.
  * De Gelder et al. (2015) Beatrice De Gelder, Aline W de Borst, and Rebecca Watson. 2015.  The perception of emotion in body expressions.  _Wiley Interdisciplinary Reviews: Cognitive Science_ 6, 2 (2015), 149–158.
  * Gibbs et al. (1999) Simon J Gibbs, Constantin Arapis, and Christian J Breiteneder. 1999.  Teleport–towards immersive copresence.  _Multimedia Systems_ 7, 3 (1999), 214–221.
  * Guan et al. (2023) Yongjie Guan, Xueyu Hou, Nan Wu, Bo Han, and Tao Han. 2023.  MetaStream: Live Volumetric Content Capture, Creation, Delivery, and Rendering in Real Time. In _Proceedings of the 29th Annual International Conference on Mobile Computing and Networking_. 1–15.
  * Hedman et al. (2018) Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow. 2018.  Deep blending for free-viewpoint image-based rendering.  _ACM Transactions on Graphics (ToG)_ 37, 6 (2018), 1–15.
  * Hesch et al. (2010) Joel A Hesch, Anastasios I Mourikis, and Stergios I Roumeliotis. 2010.  Mirror-based extrinsic camera calibration. In _Algorithmic Foundation of Robotics VIII: Selected Contributions of the Eight International Workshop on the Algorithmic Foundations of Robotics_. Springer, 285–299.
  * Izadi et al. (2011) Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al. 2011\.  Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In _Proceedings of the 24th annual ACM symposium on User interface software and technology_. 559–568.
  * Johnston and Burnett (2012) Alan B Johnston and Daniel C Burnett. 2012.  _WebRTC: APIs and RTCWEB protocols of the HTML5 real-time web_.  Digital Codex LLC.
  * Kerbl et al. (2023) Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023.  3D Gaussian Splatting for Real-Time Radiance Field Rendering.  _ACM Transactions on Graphics_ 42, 4 (2023).
  * Kuster et al. (2012) Claudia Kuster, Nicola Ranieri, Henning Zimmer, Jean-Charles Bazin, Chengzheng Sun, Tiberiu Popa, Markus Gross, et al. 2012\.  Towards next generation 3D teleconferencing systems. In _2012 3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON)_. IEEE, 1–4.
  * Lawrence et al. (2021) Jason Lawrence, Danb Goldman, Supreeth Achar, Gregory Major Blascovich, Joseph G Desloge, Tommy Fortes, Eric M Gomez, Sascha Häberling, Hugues Hoppe, and Andy Huibers. 2021.  Project starline: a high-fidelity telepresence system.  _ACM Transactions on Graphics (TOG)_ 40, 6 (2021), 1–16.
  * Lenovo (2023) Lenovo. 2023.  ThinkVision 27 3D.  https://psref.lenovo.com/Product/ThinkVision/ThinkVision_27_3D.  [Online; accessed 2023].
  * Lin et al. (2022a) Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. 2022a.  Efficient neural radiance fields for interactive free-viewpoint video. In _SIGGRAPH Asia 2022 Conference Papers_. 1–9.
  * Lin et al. (2022b) Shanchuan Lin, Linjie Yang, Imran Saleemi, and Soumyadip Sengupta. 2022b.  Robust high-resolution video matting with temporal guidance. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 238–247.
  * Lipson et al. (2021) Lahav Lipson, Zachary Teed, and Jia Deng. 2021.  Raft-stereo: Multilevel recurrent field transforms for stereo matching. In _2021 International Conference on 3D Vision (3DV)_. IEEE, 218–227.
  * LookingGlass (2021) LookingGlass. 2021.  Looking Glass Factory.  https://lookingglassfactory.com/.  [Online; accessed 2021].
  * Lucid (2024) Lucid. 2024.  Helios2 (ToF) IP67 3D Camera.  https://thinklucid.com/product/helios2-time-of-flight-imx556/.  [Online; accessed 2024].
  * Luiten et al. (2024) Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. 2024.  Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis. In _3DV_.
  * Maimone et al. (2012) Andrew Maimone, Jonathan Bidwell, Kun Peng, and Henry Fuchs. 2012.  Enhanced personal autostereoscopic telepresence system using commodity depth cameras.  _Computers & Graphics_ 36, 7 (2012), 791–807.
  * Martin-Brualla et al. (2018) Ricardo Martin-Brualla, Rohit Pandey, Shuoran Yang, Pavel Pidlypenskyi, Jonathan Taylor, Julien Valentin, Sameh Khamis, Philip Davidson, Anastasia Tkach, Peter Lincoln, et al. 2018\.  LookinGood: enhancing performance capture with real-time neural re-rendering.  _ACM Transactions on Graphics (TOG)_ 37, 6 (2018), 1–14.
  * Mildenhall et al. (2020) B Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and R Ng. 2020.  Nerf: Representing scenes as neural radiance fields for view synthesis. In _European conference on computer vision_.
  * Neff et al. (2021) Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H Mueller, Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger. 2021.  DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks. In _Computer Graphics Forum_ , Vol. 40. Wiley Online Library, 45–59.
  * Newcombe et al. (2015) Richard A Newcombe, Dieter Fox, and Steven M Seitz. 2015.  Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 343–352.
  * Nguyen-Ha et al. (2022) Phong Nguyen-Ha, Nikolaos Sarafianos, Christoph Lassner, Janne Heikkilä, and Tony Tung. 2022.  Free-viewpoint rgb-d human performance capture and rendering. In _European Conference on Computer Vision_. Springer, 473–491.
  * Novotny et al. (2019) David Novotny, Ben Graham, and Jeremy Reizenstein. 2019.  Perspectivenet: A scene-consistent image generator for new view synthesis in real indoor environments.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * NVIDIA (2024) NVIDIA. 2024.  NVIDIA Video Codec SDK.  https://developer.nvidia.com/video-codec-sdk.  [Online; accessed 2024].
  * Orts-Escolano et al. (2016) Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong Dou, et al. 2016\.  Holoportation: Virtual 3d teleportation in real-time. In _Proceedings of the 29th annual symposium on user interface software and technology_. 741–754.
  * Pumarola et al. (2021) Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. 2021.  D-nerf: Neural radiance fields for dynamic scenes. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10318–10327.
  * Raskar et al. (1998) Ramesh Raskar, Greg Welch, Matt Cutts, Adam Lake, Lev Stesin, and Henry Fuchs. 1998.  The office of the future: A unified approach to image-based modeling and spatially immersive displays. In _Proceedings of the 25th annual conference on Computer graphics and interactive techniques_. 179–188.
  * Riegler and Koltun (2021) Gernot Riegler and Vladlen Koltun. 2021.  Stable view synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 12216–12225.
  * Saito et al. (2019) Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. 2019.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In _Proceedings of the IEEE/CVF international conference on computer vision_. 2304–2314.
  * Shao et al. (2022) Ruizhi Shao, Liliang Chen, Zerong Zheng, Hongwen Zhang, Yuxiang Zhang, Han Huang, Yandong Guo, and Yebin Liu. 2022.  Floren: Real-time high-quality human performance rendering via appearance flow using sparse rgb cameras. In _SIGGRAPH Asia 2022 Conference Papers_. 1–10.
  * Shao et al. (2023) Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. 2023.  Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16632–16642.
  * Stelzner et al. (2021) Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. 2021.  Decomposing 3d scenes into objects via unsupervised volume segmentation.  _arXiv preprint arXiv:2104.01148_ (2021).
  * Stengel et al. (2023) Michael Stengel, Koki Nagano, Chao Liu, Matthew Chan, Alex Trevithick, Shalini De Mello, Jonghyun Kim, and David Luebke. 2023.  AI-Mediated 3D Video Conferencing.  In _ACM SIGGRAPH 2023 Emerging Technologies_. 1–2.
  * Strinati et al. (2019) Emilio Calvanese Strinati, Sergio Barbarossa, Jose Luis Gonzalez-Jimenez, Dimitri Ktenas, Nicolas Cassiau, Luc Maret, and Cedric Dehos. 2019.  6G: The next frontier: From holographic messaging to artificial intelligence using subterahertz and visible light communication.  _IEEE Vehicular Technology Magazine_ 14, 3 (2019), 42–50.
  * Su et al. (2020) Zhuo Su, Lan Xu, Zerong Zheng, Tao Yu, Yebin Liu, and Lu Fang. 2020.  Robustfusion: Human volumetric capture with data-driven visual cues using a rgbd camera. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16_. Springer, 246–264.
  * Suo et al. (2021) Xin Suo, Yuheng Jiang, Pei Lin, Yingliang Zhang, Minye Wu, Kaiwen Guo, and Lan Xu. 2021.  Neuralhumanfvv: Real-time neural volumetric human performance rendering using rgb cameras. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 6226–6237.
  * Szymanowicz et al. (2024) Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. 2024.  Splatter Image: Ultra-Fast Single-View 3D Reconstruction. In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Teed and Deng (2020) Zachary Teed and Jia Deng. 2020.  Raft: Recurrent all-pairs field transforms for optical flow. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_. Springer, 402–419.
  * Trevithick et al. (2023) Alex Trevithick, Matthew Chan, Michael Stengel, Eric Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, and Koki Nagano. 2023.  Real-time radiance fields for single-image portrait view synthesis.  _ACM Transactions on Graphics (TOG)_ 42, 4 (2023), 1–15.
  * Wang et al. (2021) Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. 2021.  Ibrnet: Learning multi-view image-based rendering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4690–4699.
  * Wang (2022) Yan Wang. 2022.  Edge-Enhanced Feature Distillation Network for Efficient Super-Resolution. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_. 777–785.
  * Wu et al. (2024) Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 2024.  4d gaussian splatting for real-time dynamic scene rendering. In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Yu et al. (2021a) Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. 2021a.  pixelnerf: Neural radiance fields from one or few images. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4578–4587.
  * Yu et al. (2021b) Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu. 2021b.  Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 5746–5756.
  * Yu et al. (2018) Tao Yu, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li, Gerard Pons-Moll, and Yebin Liu. 2018.  Doublefusion: Real-time capture of human performances with inner body shapes from a single depth sensor. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7287–7296.
  * Zhang et al. (2023) Jiajun Zhang, Yuxiang Zhang, Hongwen Zhang, Boyao Zhou, Ruizhi Shao, Zonghai Hu, and Yebin Liu. 2023.  Ins-HOI: Instance Aware Human-Object Interactions Recovery.  _arXiv preprint arXiv:2312.09641_ (2023).
  * Zhang et al. (2020) Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. 2020.  Nerf++: Analyzing and improving neural radiance fields.  _arXiv preprint arXiv:2010.07492_ (2020).
  * Zhang et al. (2022) Yizhong Zhang, Jiaolong Yang, Zhen Liu, Ruicheng Wang, Guojun Chen, Xin Tong, and Baining Guo. 2022.  Virtualcube: An immersive 3d video communication system.  _IEEE Transactions on Visualization and Computer Graphics_ 28, 5 (2022), 2146–2156.
  * Zhang (2000) Z. Zhang. 2000.  A flexible new technique for camera calibration.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 22, 11 (2000).  https://doi.org/10.1109/34.888718
  * Zheng et al. (2024) Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu. 2024.  GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis. In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhou et al. (2023) Yifeng Zhou, Shuheng Wang, Wenfa Li, Chao Zhang, Li Rao, Pu Cheng, Yi Xu, Jinle Ke, Wenduo Feng, Wen Zhou, et al. 2023\.  Live4D: A Real-time Capture System for Streamable Volumetric Video.  In _SIGGRAPH Asia 2023 Technical Communications_. 1–4.

Figure 10. Main results of Tele-Aloha system. The first column shows 2 of 4
input views. The first two rows show the results of static scenes. The other
rows show the results of dynamic scenes.  Figure 11. Results of Tele-Aloha
system on human-object interaction scenes. Our method can synthesis most of
solid objects. Figure 12. Failure cases. Left one shows blurry eyeglasses.
Right one shows that inaccuracy of background matting causes artifacts in
novel views.


#  Exploring Text-based Realistic Building Facades Editing Applicaiton

Jing Wang, and Xin Zhang  HUST, China

###### Abstract

This paper explores the utilization of diffusion models and textual guidance
for achieving localized editing of building facades, addressing the escalating
demand for sophisticated editing methodologies in architectural design and
urban planning. Leveraging the robust generative capabilities of diffusion
models, this study presents a promising avenue for realistically synthesizing
and modifying architectural facades. Through iterative diffusion and text
descriptions, these models adeptly capture both the intricate global and local
structures inherent in architectural facades, thus effectively navigating the
complexity of such designs. Additionally, the paper examines the expansive
potential of diffusion models in various facets, including the generation of
novel facade designs, the enhancement of existing facades, and the realization
of personalized customization. Despite their promise, diffusion models
encounter obstacles such as computational resource constraints and data
imbalances. To address these challenges, the study introduces the innovative
Blended Latent Diffusion method for architectural facade editing, accompanied
by a comprehensive visual analysis of its viability and efficacy. Through
these endeavors, we aims to propel forward the field of architectural facade
editing, contributing to its advancement and practical application.

Figure 1:  Through the integration of the powerful editing capabilities of
diffusion models and user-drawn masks, we have achieved precise editing of
architectural facades. This innovative approach opens up new possibilities in
the field of architectural design, allowing designers to customize the
appearance of buildings more intuitively and efficiently. In this editing
process, users first provide textual descriptions of the architectural facade
editing effects they wish to achieve. Subsequently, users specify the editing
areas by drawing masks, such as for windows, doors, walls, etc.

Figure 2: We can control the editing areas and achieve local modifications of
building facades via user-drawn masks and prompts Figure 3: Diffusion models
possess powerful generative capabilities, allowing them to realistically
synthesize details and structures of architectural facades. By combining with
user-drawn masks, we can precisely control the editing areas and achieve local
modifications of building appearances. This method not only enables rapid
generation of edited images but also preserves the realism and quality of the
images.

##  I Introduction

In recent years, with the rapid development of artificial intelligence 2D/3D
technology [1, 2, 3, 4] and deep learning technologies [5, 6, 7],there has
been a growing demand for facade editing techniques in the fields of
architectural design and urban planning [8, 9, 10, 11, 12, 13, 14, 15, 16].
The facade of a building, as its external interface, not only directly impacts
the aesthetic appearance of the building but also bears many functional and
environmental responsibilities. Therefore, effectively synthesizing and
editing building facades has become a critical aspect of the architectural
design process.

Deep learning technology has made breakthroughs in the field of image
synthesis and editing [17, 18, 19, 20, 21, 22, 23, 24, 25]. Among them,
diffusion models have gained considerable attention as an emerging deep
generative model. Compared to traditional Generative Adversarial Networks
(GANs) [26, 27, 28], diffusion models demonstrate better stability and sample
quality in image generation, making them a potentially powerful application
for architectural facade editing.

Facade editing based on diffusion models involves iteratively diffusing and
updating the information of an image to gradually generate realistic images.
Unlike traditional generative models, diffusion models can gradually generate
images by iterating multiple times on each pixel, allowing them to better
capture the global and local structures of images, especially suitable for
editing the complex structures and details of architectural facades.

In the field of architectural facade editing [29, 30, 31, 32, 15, 33],
diffusion models hold broad application prospects [34, 35]. Firstly, they can
be used to generate new designs for building facades, providing architects
with more diverse design options. Secondly, diffusion models can also be used
to edit the appearance details of existing building facades, such as adjusting
colors, materials, textures, etc., thereby achieving fine control over the
building’s appearance. Additionally, diffusion models can perform local
editing based on user needs, such as adding or removing building elements at
specific locations, enabling personalized customization of the building’s
appearance. Compared to traditional methods of architectural facade editing,
diffusion model-based editing methods [34, 35] have many advantages. Firstly,
their powerful generative capability can produce more realistic and detailed
architectural facade images, enhancing the realism and visual quality of the
editing results. Secondly, some diffusion model-based methods can perform fine
adjustments to architectural facades while maintaining the integrity of the
image structure. Furthermore, diffusion models can also incorporate natural
language descriptions, achieving semantic-level editing of architectural
facades, thereby enhancing the flexibility and intelligence of editing.

However, despite the enormous potential of diffusion models in architectural
facade editing, they still face many challenges in practical applications.
Firstly, due to the complexity and diversity of architectural facades,
effectively representing and capturing their features and structures becomes a
key issue. Secondly, the training and inference processes of diffusion models
often require substantial computational resources and time, limiting their
scalability and practicality in real-world applications. Additionally, when
dealing with large-scale architectural facade data, diffusion models often
encounter problems such as data imbalance and sample scarcity, which also
constrain their effectiveness and performance in practical applications.

As shown in Fig. 1, this paper explores the latest image editing techniques,
applying the Blended Latent Diffusion method to architectural facade editing,
and provides a detailed visual analysis of its feasibility and generation
effects in architectural design. It aims to provide new editing tools and
methods for the field of architectural design and planning and proposes
corresponding solutions and improvement methods. Through this research, we aim
to provide a deeper understanding and exploration for the further development
and application of architectural facade editing.

##  II Related Method

In this section, we introduce a image editing method called Blended Latent
Diffusion [36, 37], which guides image generation and editing through text
descriptions. This method combines the capabilities of diffusion models [5]
and visual-language models, featuring the following key steps and
characteristics:

### Local Editing Scenario

Unlike most text-guided methods that focus on generating images from scratch
or globally modifying existing ones, Blended Latent Diffusion emphasizes local
editing scenarios. This means artists only need to modify part of an image
while retaining the rest.

### Diffusion Models

The method is based on diffusion models, known for their excellent performance
in generation, editing, and other tasks. However, the iterative diffusion
process at the pixel level results in longer inference times. Therefore,
recent research proposes conducting diffusion in lower-dimensional and higher-
level semantic latent spaces to achieve competitive performance in shorter
training and inference times.

### Text Guidance

Blended Latent Diffusion utilizes text prompts to guide image generation and
editing. Users can add new objects, modify existing ones, or inject textual
content into images by providing text prompts, achieving intuitive and
creative editing effects.

### Background Optimization

The method supports whether to optimize the background during generation,
allowing users to decide on this additional step as needed. Even considering
background optimization, Blended Latent Diffusion outperforms baseline methods
in standard batch inference scenarios, with inference speeds ten times faster
than Blended Diffusion and Local CLIP-guided diffusion.

### Ranking Effects

By ranking generated results using the CLIP model, the study found that the
top 20$\%$ of results are generally better than the bottom 20$\%$, although
not every image strictly surpasses another.

Blended Latent Diffusion [37] combines diffusion models and text-guided
techniques to achieve the goal of generating and editing images in local
editing scenarios, providing users with an intuitive and efficient image
processing tool. Users simply describe the architectural facade editing
effects they want to achieve, such as adjusting appearance details or
modifying materials and colors, and the system generates corresponding editing
results based on the provided text prompts. This text-based editing approach
streamlines the designer’s workflow, significantly speeding up the facade
design process. The following sections will detail how this algorithm is
applied to architectural facade design.

##  III Facade Editing

The application of the Blended Latent Diffusion [36] method in the field of
architectural facade design provides designers and architects with an
innovative way to achieve local editing of facades. By combining text guidance
and image generation techniques, this method enables users to make design
modifications quickly and intuitively, explore different design concepts, and
achieve personalized customization of architectural appearances.

Firstly, designers can input the architectural facade images that need to be
edited. These could be existing architectural sketches, actual building
photos, or digital modeling images. Using the Blended Latent Diffusion method,
designers can perform local editing on specific areas such as windows, doors,
decorative elements, while retaining the original architectural structure.
This flexibility in local editing allows designers to quickly experiment with
different design concepts and explore various possibilities.

Secondly, users can guide the editing process by providing text prompts. These
prompts describe the desired design effects, such as ”adding a large glass
window on the left side of the architectural facade” or ”adding a protruding
decorative element at the top of the building”. This text-guided approach
makes the editing process more intuitive and easy to understand, while also
providing users with more creative inspiration.

During the editing process, users can mark the areas to be edited to specify
the editing scope. This marking can be done using simple drawing tools or
editing software to ensure the accuracy and precision of the editing. By
combining text prompts and marking of editing areas, the Blended Latent
Diffusion method can generate edited architectural facade images according to
user needs.

After generating the results, users can adjust and optimize the edited
architectural facade images. This may involve fine-tuning aspects such as
color, texture, shape, etc., to ensure that the editing effects meet
expectations. Designers can further modify the generated images as needed to
meet design requirements and aesthetic standards.

As shown in Fig. 2 and Fig. 3, through the application of the Blended Latent
Diffusion method in the field of architectural facade design, designers and
architects can conduct local facade editing more quickly and efficiently,
achieving rapid iteration and validation of design ideas. This method not only
helps designers save time and effort but also provides them with more design
possibilities and creative space, thereby driving innovation and development
in the field of architectural design.

In summary, the Blended Latent Diffusion method brings new possibilities and
opportunities to the field of architectural facade design, providing designers
with an intuitive and efficient tool for local facade editing and promoting
innovation and personalized customization in architectural design. By
combining text guidance and image generation techniques, this method injects
new vitality and creativity into the field of architectural design, opening up
more possibilities and imagination for future architectural design.

##  IV Conclusion

In conclusion, this paper focus on the Blended Latent Diffusion method
demonstrates its potential for enhancing architectural facade editing,
providing a detailed visual analysis of its feasibility and generation
effects. By proposing solutions and improvement methods, the study contributes
to the advancement of architectural design and planning tools. the application
prospects of diffusion models in architectural facade editing are discussed
comprehensively, ranging from generating new designs to fine-tuning existing
facade details and performing localized edits based on user preferences.
Despite their advantages, diffusion models face challenges such as effectively
representing complex facade features and requiring significant computational
resources for training and inference. Overall, this paper underscores the
importance of exploring innovative image editing techniques like the Blended
Latent Diffusion method in addressing the evolving needs of architectural
facade editing, ultimately aiming to facilitate deeper understanding and
advancement in this critical area of architectural design.

## References

  * [1] J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young _et al._ , “Scaling language models: Methods, analysis & insights from training gopher,” _arXiv preprint arXiv:2112.11446_ , 2021.
  * [2] P. Li, Y. Ding, L. Li, J. Guan, and Z. Li, “Towards practical consistent video depth estimation,” in _Proceedings of the 2023 ACM International Conference on Multimedia Retrieval_ , 2023, pp. 388–397.
  * [3] R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du _et al._ , “LaMDA: Language models for dialog applications,” _arXiv preprint arXiv:2201.08239_ , 2022.
  * [4] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” in _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , 2020, pp. 1877–1901.
  * [5] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10 684–10 695.
  * [6] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [7] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [8] L. N. Groat and D. Wang, _Architectural research methods_. John Wiley & Sons, 2013.
  * [9] D. Garlan, R. Allen, and J. Ockerbloom, “Exploiting style in architectural design environments,” _ACM SIGSOFT software engineering notes_ , vol. 19, no. 5, pp. 175–188, 1994.
  * [10] O. O. Demirbaş and H. Demirkan, “Focus on architectural design process through learning styles,” _Design studies_ , vol. 24, no. 5, pp. 437–456, 2003.
  * [11] I. Caetano, L. Santos, and A. Leitão, “Computational design in architecture: Defining parametric, generative, and algorithmic design,” _Frontiers of Architectural Research_ , vol. 9, no. 2, pp. 287–300, 2020.
  * [12] D. Aliakseyeu, J.-B. Martens, and M. Rauterberg, “A computer support tool for the early stages of architectural design,” _Interacting with Computers_ , vol. 18, no. 4, pp. 528–555, 2006.
  * [13] Ö. Akin and C. Akin, “Frames of reference in architectural design: analysing the hyperacclamation (aha-!),” _Design studies_ , vol. 17, no. 4, pp. 341–361, 1996.
  * [14] K. J. Lomas, “Architectural design of an advanced naturally ventilated building form,” _Energy and Buildings_ , vol. 39, no. 2, pp. 166–181, 2007.
  * [15] M. Thalfeldt, E. Pikas, J. Kurnitski, and H. Voll, “Facade design principles for nearly zero energy buildings in a cold climate,” _Energy and Buildings_ , vol. 67, pp. 309–321, 2013.
  * [16] Y.-W. Lim, M. Z. Kandar, M. H. Ahmad, D. R. Ossen, and A. M. Abdullah, “Building façade design for daylighting quality in typical government office building,” _Building and Environment_ , vol. 57, pp. 194–204, 2012.
  * [17] P. Li, Q. Huang, Y. Ding, and Z. Li, “Layerdiffusion: Layered controlled image editing with diffusion models,” in _SIGGRAPH Asia 2023 Technical Communications_ , 2023, pp. 1–4.
  * [18] T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 18 392–18 402.
  * [19] G. Parmar, K. Kumar Singh, R. Zhang, Y. Li, J. Lu, and J.-Y. Zhu, “Zero-shot image-to-image translation,” in _ACM SIGGRAPH 2023 Conference Proceedings_ , 2023, pp. 1–11.
  * [20] R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or, “Null-text inversion for editing real images using guided diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 6038–6047.
  * [21] H. Chang, H. Zhang, J. Barber, A. Maschinot, J. Lezama, L. Jiang, M.-H. Yang, K. Murphy, W. T. Freeman, M. Rubinstein _et al._ , “Muse: Text-to-image generation via masked generative transformers,” _arXiv preprint arXiv:2301.00704_ , 2023.
  * [22] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman, “Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 22 500–22 510.
  * [23] H. Chefer, Y. Alaluf, Y. Vinker, L. Wolf, and D. Cohen-Or, “Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models,” _ACM Transactions on Graphics (TOG)_ , vol. 42, no. 4, pp. 1–10, 2023.
  * [24] N. Kumari, B. Zhang, R. Zhang, E. Shechtman, and J.-Y. Zhu, “Multi-concept customization of text-to-image diffusion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 1931–1941.
  * [25] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 4401–4410.
  * [26] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” _Communications of the ACM_ , vol. 63, no. 11, pp. 139–144, 2020.
  * [27] A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, “Generative adversarial networks: An overview,” _IEEE signal processing magazine_ , vol. 35, no. 1, pp. 53–65, 2018.
  * [28] M.-Y. Liu and O. Tuzel, “Coupled generative adversarial networks,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [29] G. Shalunts, Y. Haxhimusa, and R. Sablatnig, “Architectural style classification of building facade windows,” in _International Symposium on Visual Computing_. Springer, 2011, pp. 280–289.
  * [30] Y.-W. Lim, M. Z. Kandar, M. H. Ahmad, D. R. Ossen, and A. M. Abdullah, “Building façade design for daylighting quality in typical government office building,” _Building and Environment_ , vol. 57, pp. 194–204, 2012.
  * [31] S. Moghtadernejad, M. S. Mirza, and L. E. Chouinard, “Facade design stages: Issues and considerations,” _Journal of Architectural Engineering_ , vol. 25, no. 1, p. 04018033, 2019.
  * [32] S. M. Hosseini, M. Mohammadi, A. Rosemann, T. Schröder, and J. Lichtenberg, “A morphological approach for kinetic façade design process to improve visual and thermal comfort,” _Building and environment_ , vol. 153, pp. 186–204, 2019.
  * [33] V. Ž. Leskovar and M. Premrov, “An approach in architectural design of energy-efficient timber buildings with a focus on the optimal glazing size in the south-oriented façade,” _Energy and buildings_ , vol. 43, no. 12, pp. 3410–3418, 2011.
  * [34] P. Li, B. Li, and Z. Li, “Sketch-to-architecture: Generative ai-aided architectural design,” in _Proceedings of the 31st Pacific Conference on Computer Graphics and Applications_. The Eurographics Association, 2023.
  * [35] P. Li and B. Li, “Generating daylight-driven architectural design via diffusion models,” _arXiv preprint arXiv:2404.13353_ , 2024.
  * [36] O. Avrahami, D. Lischinski, and O. Fried, “Blended diffusion for text-driven editing of natural images,” 2022, pp. 18 208–18 218.
  * [37] O. Avrahami, O. Fried, and D. Lischinski, “Blended latent diffusion,” _ACM Transactions on Graphics (TOG)_ , vol. 42, no. 4, pp. 1–11, 2023.