# CityCraft: A Real Crafter for 3D City Generation

Jie Deng1, Wenhao Chai2,∗ Junsheng Huang1,∗ Zhonghan Zhao1,∗ Mingyan Gao1
Qixuan Huang1 Jianshu Guo1 Shengyu Hao1 Wenhao Hu1 Jenq-Neng Hwang 2 Xi Li1
Gaoang Wang1 🖂
1Zhejiang University 2University of Washington
https://github.com/djFatNerd/CityCraft Equal contribution; 🖂Corresponding
Author; Correspondence to<EMAIL_ADDRESS>
###### Abstract

City scene generation has gained significant attention in autonomous driving,
smart city development, and traffic simulation. It helps enhance
infrastructure planning and monitoring solutions. Existing methods have
employed a two-stage process involving city layout generation—typically using
Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), or
Transformers—followed by neural rendering. These techniques often exhibit
limited diversity and noticeable artifacts in the rendered city scenes. The
rendered scenes lack variety, resembling the training images, resulting in
monotonous styles. Additionally, these methods lack planning capabilities,
leading to less realistic generated scenes. In this paper, we introduce
CityCraft, an innovative framework designed to enhance both the diversity and
quality of urban scene generation. Our approach integrates three key stages:
initially, a diffusion transformer (DiT) model is deployed to generate diverse
and controllable 2D city layouts. Subsequently, a Large Language Model (LLM)
is utilized to strategically make land-use plans within these layouts based on
user prompts and language guidelines. Based on the generated layout and city
plan, we utilize the asset retrieval module and Blender for precise asset
placement and scene construction. Furthermore, we contribute two new datasets
to the field: 1) CityCraft-OSM dataset including 2D semantic layouts of urban
areas, corresponding satellite images, and detailed annotations. 2) CityCraft-
Buildings dataset, featuring thousands of diverse, high-quality 3D building
assets. CityCraft achieves state-of-the-art performance in generating
realistic 3D cities.

Figure 1: Overview of CityCraft. The Layout Generator generates realistic 2D
city layouts based on user input conditions; the CityPlanner process the
generated layouts, isolates instances, get image-level information, make land-
use plans for the urban region, and select appropriate assets from assets
library to craft the 3D city.

##  1 Introduction

The field of 3D content generation has made remarkable progress with
generative modeling, particularly in the automatic creation of 3D objects [57,
65, 46], avatars [84, 38], and comprehensive scenes [43, 79]. Generating 3D
city scenes is crucial for simulating realistic urban environments and
exploring innovative designs. Traditional methods like procedural [17, 45, 52,
70, 80] and image-based modeling [11, 6, 24, 30, 75, 12] have established
significant benchmarks, but they often lack diversity and flexibility in
generating varied layout designs, hindering innovation and adaptability in
urban planning.

The emerging field of text-to-3D generation combines text-to-image diffusion
models with 3D representations. Innovations like DreamFusion [57], Magic3D
[42], and ProlificDreamer [77] use techniques to refine 3D models and enhance
the realism of generated scenes. However, applying these techniques to urban-
scale scenes introduces complexities due to diverse architectural styles and
complex topologies.

Recent methodologies like 3D-GPT [69] and SceneCraft [35] aim to enhance
traditional approaches by combining instruction-driven 3D modeling with
procedural generation software, using Large Language Models (LLMs) to assist
human designers. However, challenges persist, such as the difficulty of
SceneCraft [35] in fine-grained editing of 3D assets and limitations of 3D-GPT
in utilizing available procedural generation resources for creating large-
scale, high-quality urban scenes.

Recent advancements in city scene generation [43, 79] have shown significant
progress but highlighted the persistent challenges in creating realistic
three-dimensional urban landscapes suitable for various applications.
Traditional methods [43, 79] struggle to offer enough diversity and control,
leading to gaps in realism and adaptability. LLMs have the potential to
revolutionize urban scene generation by providing explainable, logical, and
controllable planning, enhancing the process and resulting in more adaptable,
efficient, and visually appealing cityscapes.

CityCraft addresses these challenges by integrating state-of-the-art
technologies, including Diffusion Transformers (DiT) for layout generation,
Large Language Models (LLM) for strategic urban planning, and Blender [1] for
realistic city scene crafting and rendering. This integrated approach
significantly improves the generation of urban scenes that are not only
diverse and detailed but also highly controllable, as shown in Figure 4.

Given these advancements, the essential question we address is: How can we
generate infinitely scalable, highly detailed, and controllable 3D city scenes
efficiently? In response, CityCraft introduces innovative methodologies to
revolutionize city scene generation:

Advanced Layout Generation: We use an advanced Diffusion Transformer model
[53] to generate high-quality, diverse and detailed 2D city layouts. The
generator can take class-ratios and text as user controls and expand the
generated layouts infinitely.

Strategic Urban Planning: We use a Large Language Model(LLM) to implement
complex city planning strategies from text prompts, resulting in more rational
and realistic city generation.

High-Quality Asset Integration and Scene Construction: Combining a high
quality 3D assets library with Blender, CityCraft meticulously constructs city
scenes, ensuring high realism and aesthetic quality through precise asset
placement and advanced rendering techniques. Our main contributions are
summarized as follows:

  1. 1.

We present CityCraft, an innovative framework that significantly enhances
urban scene generation by combining advanced layout generation, strategic city
planning, and high-quality scene construction.

  2. 2.

We demonstrate how CityCraft achieves unprecedented control and diversity in
city scene generation, enabling applications requiring high detail and
customization levels.

  3. 3.

We contribute two datasets, CityCraft-OSM and CityCraft-Buildings, to the
community, enhancing the ability of researchers and practitioners to create
more realistic and varied city environments.

##  2 Related Works

###  2.1 Diffusion Models

Denoising Diffusion Probabilistic models (DDPM) [32] have demonstrated
exceptional success as generative models across various domains, including
images [19, 12, 11, 18], 3D objects [56, 51, 74], and videos [13, 47]. These
models have surpassed Generative Adversarial Networks (GANs) [28], which
previously dominated the field. Recent efforts in latent Diffusion Models
(DMs) [63] have shown promising applications in generating images, point
clouds, and text [73, 83, 40]. Diffusion models typically uses convolutional
U-Nets [64] as backbone for noise or image prediction in the denoising
procedure. Recently, [53] proposed to use pure transformers as the denoising
network.

###  2.2 Scene Planning with Large Language Models

Recent advancements in scene design involve learning spatial knowledge priors
from established 3D scene databases [16, 71, 48, 72, 87, 76, 78] or refining
3D scenes iteratively based on user input [15, 22]. However, approaches that
rely solely on datasets like 3D-FRONT [27] face limitations due to the
restricted variety of categories within these datasets. Incorporating Large
Language Models (LLMs), recent works such as LayoutGPT [25], Holodeck [81] and
others [44] have demonstrated the potential of LLMs in generating 3D scene
layouts and other agent-like tasks [88, 89, 90, 67, 68]. Nevertheless, direct
numerical outputs from LLMs can sometimes result in physically implausible
layouts, such as overlapping assets. To address this, our approach leverages
LLMs to define spatial relational constraints on definite semantic layouts,
utilizing a solver to optimize the arrangement. This ensures that the layouts
are inspired by the vast knowledge encoded in LLMs and adhere to physical
plausibility.

###  2.3 City Scene Generation

City scene generation combines detailed urban planning, including road
networks, land use, and building placement, using techniques from rule-based
designs [8, 10] to procedural tools like CityEngine [2] and Unreal Engine [5],
and deep learning [43, 79, 61]. While diffusion models [29, 36] often simplify
layouts to basic elements, limiting complexity, Neural Radiance Fields (NeRF)
[43, 79, 20] produce high-quality visuals but are computationally expensive.
In contrast, CityGen [23] combines Stable Diffusion [63] with Low-Rank
Adaptation (LoRA) [34] on the MatrixCity dataset [41], resulting in more
realistic and controllable city scenes through ControlNet [86]. Additionally,
new integrations of vision-language models like CLIP [60] with depth
prediction [26, 33, 85] push 3D scene generation further, though modularity
remains a challenge. Our approach utilizes an extensive 3D asset database to
enhance realism and interactivity, improving scene applicability for
simulation, city planning, and virtual reality.

##  3 Methods

###  3.1 City Layout Generation

Unconditional Generation. In the first stage, we generate a 2D city layout for
major objects in city scenes. We represent the city layout
$L\in\mathbb{R}^{C\times H\times W}$ as a $H\times W$ semantic mask for $C$
classes [20]. Considering diffusion model’s excellent generation and
outpainting ability, we utilize a DiT [53] as the layout generator
$\mathcal{D}$. We use the pretrained VAE of SDXL [55] to encode training
images into latent space during training and decode the denoised latents into
image space during sampling.

Conditional Generation. Condition guidance allows users to control the output,
enabling the generation of images that meet user specifications.

We introduce two types of control that can be added to our layout generator.
The first is a class ratio vector $R_{L}\in\mathbb{R}^{1\times C}$ used to
enhance semantic aware generation, where $C$ is the number of classes. For a
given semantic layout $L$, the element $R_{L}[i]$ represents the ratio of
class $i$, where $i\in\\{1,2,\ldots,C\\}$, $R_{L}[i]\in(0,1)$ and
$\sum_{i=1}^{C}R_{L}[i]=1$. The second is a textual description $t_{L}$ that
captures the desired city characteristics or specific structural elements for
a given layout $L$, introduced to allow customized style generation of
layouts. Directly preparing annotations for the semantic layout $L$ is
challenging due to the scarcity of such data for training large vision-
language models (LVMs) to generate captions. To address this issue, we utilize
the corresponding satellite image $L_{sat}$ and employ GPT-4V [82] to generate
the caption for the satellite images: $t_{L}=$ GPT-4V$(L_{sat})$. These
conditions enhance the diffusion model’s ability in reflecting user inputs,
thereby improving flexibility and controllability in the generation process.
The DiT blocks incorporate these conditions via adaptive layer norm [54]. The
class ratio and text embeddings are mapped to the same temporal embedding
space using MLP layers before being combined with the time embeddings.

Infinite Expansion Our method also includes an infinite expansion feature to
address the challenge of generating large-scale city layouts. We utilize
BlendedDiffusion [7] to achieve outpainting using the pretrained layout
generation models. During sampling, we iteratively expand the city layout by
generating new sections that seamlessly connect with previously generated
areas, thereby enabling the creation of expansive city layouts without the
limitations of fixed-size generation windows.

###  3.2 City Planning

Figure 2: Planning and Selection Process. An example process of planning and
selection. Starting from the 2D semantic layout, we isolate all instances and
build a basic information dictionary for all instances based on their scale,
type, spatial information, _etc._(only partial information are shown in the
figure for explanation, D2Road: distance to traffic roads). For each instance,
we feed its information to the planner and let the planner make decisions on
its characteristics. Then based on these characteristics, we retrieve the best
matching candidate from the asset library and use it to craft the 3D city.

Leveraging the 2D layouts generated in the first phase, we employ an LLM as
our city region planner $\mathcal{P}$ to plan the allocation of urban objects.
$\mathcal{P}$ analyzes $L$’s spatial distribution and semantic context to
suggest optimal space usage, which is crucial for realistic and functional
urban design. Given a generated layout $L$, we first isolate all instances and
retrieve information from $L$ to build a dictionary $D_{L}$ of information for
$L$, as shown in Figure 2. For example, for a selected instance $i$ where $i$
is a building, we extract the following information from the 2D layout: size,
area, location, surrounding neighbors, distance to traffic roads, etc. We
summarize this information into $D_{L}[i]$. Subsequently, we pass $D_{L}[i]$
to $\mathcal{P}$ to get plans, for instance, $i$, in this case where $i$ is a
building, we output suggested planned information including primary function
(residential, commercial, _etc._), secondary function (store, hospital,
school, _etc._), size, architectural style, as well as the reasoning
$\mathcal{P}$ use for making the decision. We summarize plans for instance $i$
into $P_{L}[i]$, where $P_{L}$ is the dictionary that stores plans for all
instances. We also build an information dictionary $D_{A}$ for the assets
library $A$, which contains image renders from multiple views, and text
annotations from multiple aspects. We retrieve the most matching asset from
the asset library based on the plan. During selection, we first use a tree-
searching algorithm to select a subset of candidates from the asset library
based on critical requirements like function and scale. Then, we calculate the
similarities between these candidates and the plan. For an asset $A$ and a
plan $P$, the total similarity score is calculated by Equation 1:

$\sigma(A,P)=\sum_{m=1}^{M}w_{m}\cdot\sigma_{m}(\mathcal{E}_{m}^{A},\mathcal{E}_{m}^{P})$
(1)

where $\mathcal{E}_{m}$ is the CLIP [60] or SBERT [62] embeddings of the
$m^{\text{th}}$ image or text, $w_{m}$ is the weight and $\sigma(\cdot)_{m}$
is the similarity function for pair-wised image or text embeddings. After
that, the asset $A$ with the highest similarity will be selected as the
candidate.

This planning procedure is repeated several rounds for all instances, and the
planner updates plans for each instance based on updated local and global
information. At each iteration of the planning, we tell $\mathcal{P}$ its
previous reasoning for decision and query if it wants to keep the original
plan or make a new plan. This step is necessary since, in the initial steps,
planner $\mathcal{P}$ only gets partial local and global information for
making decisions if an instance’s neighboring objects have not been planned,
so the plan might not be coherent and stable in early stages. We record the
number of changes made by the planner and the total number of plans at each
iteration and claim convergence when their ratio is smaller than a threshold.

After making plans for all instances, we utilize Blender [1] to translate the
2D layout into fully realized 3D city scenes. For each chosen asset, we use
Powell algorithm [58] to find the optimal scale and rotation factor to place
it onto the 2D layout. For other objects, including vegetation, water, and
roads, we add planned textures to them. We also add customized features like
trees and streetlights to make the scene look more realistic.

##  4 Datasets

We build two datasets to facilitate our study. The CityCraft-OSM dataset is
used for training the city layout generation model, and the CityCraft-
Buildings dataset is used for planning and crafting 3D cities.

Table 1: Quantitative comparison between CityCraft-OSM and CityDreamer-OSM
[79]. CityCraft-OSM includes a greater number of classes and a significantly
larger total area. Unlike CityDreamer-OSM [79], CityCraft-OSM also
incorporates satellite images and text annotations, enhancing its utility for
more detailed and accurate urban planning and analysis.

Dataset |  Number of Classes | Total Area | Satellite Images | Text Annotation
---|---|---|---|---
CityDreamer-OSM [79] | 6 | 6,000 km2 | ✗ | ✗
CityCraft-OSM(Ours) | 7 | 67,108 km2 | ✓ | ✓

#### CityCraft-OSM dataset

We build the CityCraft-OSM dataset upon OpenStreetMap(OSM) [3]. It contains
semantic layouts of real-world North America, Europe, and Asia cities. The
layouts contain objects of seven classes, including terrain, vegetation,
water, building, traffic road, rail, and footpath. Our dataset contains over
100,000 $768\times 768$ patches of pixel distance 0.5 meters. We also provide
semantic class ratios and the corresponding satellite images, and we annotate
a subset of the satellite images using GPT4-V [50], and GeoChat [39] with
human corrections. We compare our dataset with a similar dataset CityDreamer-
OSM [79], which was also built from OSM; we show the comparison in Table 1 and
qualitative comparisons in Figure I2.

#### CityCraft-Buildings dataset

We construct CityCraft-Buildings, consisting of 2,000 high-quality 3D building
assets sourced from online open resources [4]. This dataset encompasses
buildings of diverse functions, scales, styles, _etc._ We provide high-quality
renders for each asset from 12 angles and rich annotations detailing various
features. We show examples of the assets in Figure K4.

##  5 Experiments

###  5.1 Experiment Setups

We use DiT-B/2 [53] as the backbone for layout generation model. We use
gpt-4-vision-preview [82] for image annotations, gpt-4o-2024-05-13 [49] and
gpt-4-1106-preview [49] for planning. All training and experiments are
conducted on 8 $\times$ 4090-24GB Nvidia GPUs. The settings for all user
studies and preference score calculations are described in Section 10.

###  5.2 Results on City Layout Generation

We compare the performance of layout generation with other models that
generate city layouts as semantic masks, including Infinicity [23],
CityDreamer [79], and CityGen [23]. We do not include comparisons with other
methods that use object bounding boxes to represent city layouts, such as [59,
37, 36]. These methods typically limit object types to only buildings and
roads and impose constraints on object shapes. They fail to capture intricate
object shapes, diverse types, precise positions, and relationships in complex
urban scenarios , making them less suitable for city crafting.

#### Quantitative Comparison.

As shown in Table 2, CityCraft significantly outperforms all compared methods
in FID and KID scores. Specifically, CityCraft achieves an FID score of 27.60
and a KID score of 0.022, which are considerably lower than those of its
closest competitor, CityGen [23], which scores 88.38 and 0.089, respectively.
This substantial improvement highlights the advanced capabilities of CityCraft
in generating more realistic and diverse city layouts. CityCraft’s preference
score 8.6 further underscores its appeal, demonstrating its high user
favorability.

#### Qualitative Comparison.

We show samples from all models in Figure 3. InfiniCity [43] lacks structured
zoning, while CityDreamer [79] demonstrates enhancements with distinct grid
patterns and rich semantic zones, yet it retains repetitive and simplistic
features. Additionally, many buildings are interconnected, complicating
instance separation, and some objects (roads, water bodies) overlap with
buildings. CityGen [23] offers more consistency in grid layouts and a better
delineation of roads but remains limited in architectural diversity. Our
method, CityCraft, significantly enhances realism and diversity, featuring
detailed, varied architectural styles and sophisticated road networks that
closely mimic real city scenarios. The layouts demonstrate an advanced
integration of city elements, providing a dynamic and visually appealing
cityscape that outperforms other methods in aesthetics and practical layout
design.

ground  vegetation  building  rail  traffic roads  footpath  water

Figure 3: Qualitative comparison of city layouts. From top to bottom:
CityCraft (ours), InfiniCity [43], CityDreamer [79], and CityGen [23].
CityCraft shows superior detail and realism in city planning, highlighting
complex road networks and diverse architectural styles. Table 2: Quantitative
comparison on Layout and Scene Generation. CityGen-2 substantially outperforms
other methods in layout and scene generation. Layout generation is measured by
Fréchet Inception Distance(FID) [31] and Kernel Inception Distance(KID) [9]
(both the lower the better). DE and CE are for depth errors and camera errors
adopted from [79].

Method |  FID $(\downarrow)$ |  KID $(\downarrow)$ |  Preference $(\uparrow)$
---|---|---|---
City Layout Generation |  |  |
InfiniCity [43] | 175.68 | 0.175 | 2.3
CityDreamer [79] | 111.44 | 0.115 | 6.8
CityGen [23] | 88.38 | 0.089 | 7.5
CityCraft (ours) | 27.60 | 0.022 | 8.6
Method |  DE $(\downarrow)$ |  CE $(\downarrow)$ |  Preference $(\uparrow)$
City Scene Generation |  |  |
Infinicity [43] | N/A | N/A | 5.1
SGAM [66] | 0.575 | 239.291 | 6.6
CityDreamer [79] | 0.147 | 0.060 | 7.6
CityGen [23] | N/A | N/A | 5.8
CityCraft (ours) | 0 | 0 | 9.2

###  5.3 Results on City Scene Generation.

We compare the generated city scenes with other SOTA city scene generation
methods, including SGAM [66], Infinicity [43], CityDreamer [79], and CityGen
[23].

#### Quantitative Comparison.

We show quantitative results in Table 2. The metrics extend to Depth Error
(DE) [14] and Camera Error (CE) [21], essential for assessing the accuracy of
spatial and perspective representations in generated scenes. CityCraft has a
Depth Error (DE) and Camera Error (CE) of 0, showing no errors in depth or
camera placement. Our framework creates precise 3D city models, enabling
consistent views from any angle during sampling. In contrast, rendering-based
methods such as CityDreamer [79] and Infinicity [43] exhibit noticeable
discrepancies and limited consistency, with the diversity of generated scenes
closely resembling their training images. The CE and DE scores for Infinicity
[43] and CityGen [23] are not included since Infinicity [43] is not open-
sourced and CityGen [23] has no multi-view consistency as they only support
synthesizing image at single view. The user preference score of 9.2 for
CityCraft aligns with its technical metrics, further confirming user
satisfaction with the realism and technical accuracy of the generated scenes.

#### Qualitative Comparison.

As shown in Figure 4, the qualitative results underscore the superiority of
CityCraft. Unlike InfiniCity [43], which tends to produce blurred and
undetailed scenes. SGAM [66] and CityDreamer [79] are much clearer. However,
they still lack realism and exhibit repetitive architectural features across
frames.

Figure 4: Qualitative comparison of city scenes. From top to bottom: CityCraft
(ours), InfiniCity [43], CityDreamer [79], CityGen [23]. CityCraft
demonstrates superior architectural diversity and realism, leveraging Real 3D
Crafter technology for direct building growth and LLM-driven adaptive
modeling, resulting in a more authentic and varied city landscape.

###  5.4 Ablation Study

#### Layout Generation Controls

Table 3: Ablation Study of Conditional Generation.CityCraft (uncondition) are
random samples generated from the unconditional model; CityCraft (ratio) are
conditional samples generated from user input ratios; CityCraft (text) are
conditional samples generated from user input texts.

Method |  FID $(\downarrow)$ |  KID $(\downarrow)$ |  Preference $(\uparrow)$
---|---|---|---
CityCraft (uncondition) | 27.60 | 0.022 | 7.9
CityCraft (ratio) | 25.58 | 0.023 | 8.7
CityCraft (text) | 28.35 | 0.032 | 8.3

We conduct an ablation study on the layout generation model under conditions,
comparing the FID and KID of unconditional and conditional models by sampling
10,000 images from each. According to Table 3, while FID/KID scores showed no
significant differences among the models, users prefer the conditional models
due to their control over layout generation. Specifically, ratio control was
favored for its precise influence on the generation process. We discover that
ratio-based guidance outperforms text descriptions for creating semantic
layouts. We find that class ratios effectively improve layout generation and
semantic understanding by calculating the Average Class Error (ACE) through
the average absolute difference between generated sample class ratios and
those of 10,000 input conditions.

Table 4: Average Class Error(ACE) of ratio-controlled generation We calculate
the per average class error between the input condition class ratios and
output class ratios. Results demonstrate ratios can be used as effective
control for layout synthesis.

Object Class | Ground | Vegetation | Building | Rail | Traffic Roads | Footpath | Water
---|---|---|---|---|---|---|---
ACE(%) | 14.94 | 5.35 | 3.78 | 5.28 | 4.48 | 11.50 | 1.78

#### Scene Generation Controls

We demonstrate the necessity of multi-round refinement for city planning. We
generate 50 city scenes and measure the number of changes made by the planner
in 10 rounds of planning. The results are shown in Figure H1. From the
results, we verify our assumption that the planning procedure is less stable
in the initial steps and that multi-round refinement is necessary.

We also test the influence of global controls to generated plans. As shown in
Figure 5, CityCraft demonstrates its capability to tailor urban environments
through user-defined prompts. For the same city layouts, we prompt the planner
to generate residential v.s. commercial regions. The generated commercial
zones mainly comprise public service facilities (47%) and commercial
establishments (31%), indicating a vibrant business district with concentrated
economic activities. Healthcare facilities and some residential areas create a
comprehensive urban environment with easy access to essential services,
typical of economic hubs with strong public transport systems supporting heavy
foot traffic and commerce. The generated residential zones is mainly
residential (60%), with public service facilities (13%) and commercial spaces
(11%) for necessary amenities and leisure activities. Healthcare and
educational institutions (8% and 5% respectively) further enhance the area’s
livability, reflecting a well-rounded neighborhood designed to support a
thriving community. From the results, we observe the planner’s ability to
adapt to various user settings, proving its robustness across various scenes.

Figure 5: City functionality distribution with different prompts. Commercial
zones are mainly for business and public services, with strong public
infrastructure. Residential zones focus on living spaces, supplemented by key
urban functions like healthcare and education.

##  6 Conclusion and Broader Impacts

We introduce CityCraft, a novel framework for creating detailed 3D city scenes
from user-defined text and ratio specifications. CityCraft integrates advanced
techniques in layout generation, city planning, and scene construction to
ensure high fidelity and alignment with user requirements. Through
evaluations, it showed significant improvements over traditional methods in
diversity, controllability, and visual appeal. We also introduced two novel
datasets, CityGen-OSM and CityGen-Buildings, for research and development in
city scene generation. We plan to expand CityCraft’s capabilities to include
dynamic elements such as moving traffic and integrate real-time user feedback
mechanisms for interactive scene customization.

## References

  * [1] Blender.  https://www.blender.org/.
  * [2] Esri’s cityengine.  https://www.esri.com/en-us/arcgis/products/arcgis-cityengine/overview.
  * [3] Openstreetmap.  https://www.openstreetmap.org/.
  * [4] Sketchfab.  https://sketchfab.com/3d-models/.
  * [5] Unreal engine.  https://www.unrealengine.com/.
  * [6] Daniel G Aliaga, Carlos A Vanegas, and Bedrich Benes.  Interactive example-based urban layout synthesis.  ACM transactions on graphics (TOG), 27(5):1–10, 2008.
  * [7] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208–18218, 2022.
  * [8] Edmund N Bacon.  Design of cities: Revised edition.  Penguin books, 1976.
  * [9] Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying mmd gans.  arXiv preprint arXiv:1801.01401, 2018.
  * [10] Peter Calthorpe and William B Fulton.  The regional city: Planning for the end of sprawl, volume 18.  Island Press Washington, DC, 2001.
  * [11] Shidong Cao, Wenhao Chai, Shengyu Hao, and Gaoang Wang.  Image reference-guided fashion design with structure-aware transfer by diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3524–3528, 2023.
  * [12] Shidong Cao, Wenhao Chai, Shengyu Hao, Yanting Zhang, Hangyue Chen, and Gaoang Wang.  Difffashion: Reference-based fashion design with structure-aware transfer by diffusion models.  arXiv preprint arXiv:2302.06826, 2023.
  * [13] Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu.  Stablevideo: Text-driven consistency-aware diffusion video editing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23040–23050, 2023.
  * [14] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123–16133, 2022.
  * [15] Angel Chang, Manolis Savva, and Christopher D Manning.  Interactive learning of spatial knowledge for text to 3d scene generation.  In Proceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces, pages 14–21, 2014.
  * [16] Angel X Chang, Mihail Eric, Manolis Savva, and Christopher D Manning.  Sceneseer: 3d scene design with natural language.  arXiv preprint arXiv:1703.00050, 2017.
  * [17] Guoning Chen, Gregory Esch, Peter Wonka, Pascal Müller, and Eugene Zhang.  Interactive procedural street modeling.  In ACM SIGGRAPH 2008 papers, pages 1–10. 2008.
  * [18] Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li.  Pixart-$\\{$$\backslash$delta$\\}$: Fast and controllable image generation with latent consistency models.  arXiv preprint arXiv:2401.05252, 2024.
  * [19] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al.  Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.  arXiv preprint arXiv:2310.00426, 2023.
  * [20] Zhaoxi Chen, Guangcong Wang, and Ziwei Liu.  Scenedreamer: Unbounded 3d scene generation from 2d image collections.  arXiv preprint arXiv:2302.01330, 2023.
  * [21] Zhaoxi Chen, Guangcong Wang, and Ziwei Liu.  Scenedreamer: Unbounded 3d scene generation from 2d image collections.  In arXiv, 2023.
  * [22] Yu Cheng, Yan Shi, Zhiyong Sun, Dezhi Feng, and Lixin Dong.  An interactive scene generation using natural language.  In 2019 International Conference on Robotics and Automation (ICRA), pages 6957–6963. IEEE, 2019.
  * [23] Jie Deng, Wenhao Chai, Jianshu Guo, Qixuan Huang, Wenhao Hu, Jenq-Neng Hwang, and Gaoang Wang.  Citygen: Infinite and controllable 3d city layout generation.  arXiv preprint arXiv:2312.01508, 2023.
  * [24] Lubin Fan, Przemyslaw Musialski, Ligang Liu, and Peter Wonka.  Structure completion for facade layouts.  ACM Trans. Graph., 33(6):210–1, 2014.
  * [25] Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang.  Layoutgpt: Compositional visual planning and generation with large language models.  arXiv preprint arXiv:2305.15393, 2023.
  * [26] Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel.  Scenescape: Text-driven consistent scene generation.  arXiv preprint arXiv:2302.01133, 2023.
  * [27] Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al.  3d-front: 3d furnished rooms with layouts and semantics.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10933–10942, 2021.
  * [28] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [29] Liu He, Yijuan Lu, John Corring, Dinei Florencio, and Cha Zhang.  Diffusion-based document layout generation.  arXiv preprint arXiv:2303.10787, 2023.
  * [30] Olof Henricsson, Andre Streilein, and Armin Gruen.  Automated 3-d reconstruction of buildings and visualization of city models, 1996.
  * [31] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in neural information processing systems, pages 6626–6637, 2017.
  * [32] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [33] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner.  Text2room: Extracting textured 3d meshes from 2d text-to-image models.  arXiv preprint arXiv:2303.11989, 2023.
  * [34] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [35] Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A. Ross, Cordelia Schmid, and Alireza Fathi.  Scenecraft: An llm agent for synthesizing 3d scene as blender code, 2024.
  * [36] Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi.  Layoutdm: Discrete diffusion model for controllable layout generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10167–10176, 2023.
  * [37] Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori.  Layoutvae: Stochastic scene layout generation from a label set.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9895–9904, 2019.
  * [38] Nikos Kolotouros, Thiemo Alldieck, Andrei Zanfir, Eduard Gabriel Bazavan, Mihai Fieraru, and Cristian Sminchisescu.  Dreamhuman: Animatable 3d avatars from text.  arXiv preprint arXiv:2306.09329, 2023.
  * [39] Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abhijit Das, Salman Khan, and Fahad Shahbaz Khan.  Geochat: Grounded large vision-language model for remote sensing.  arXiv preprint arXiv:2311.15826, 2023.
  * [40] Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto.  Diffusion-LM improves controllable text generation.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * [41] Yixuan Li, Lihan Jiang, Linning Xu, Yuanbo Xiangli, Zhenzhi Wang, Dahua Lin, and Bo Dai.  Matrixcity: A large-scale city dataset for city-scale neural rendering and beyond.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3205–3215, 2023.
  * [42] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300–309, 2023.
  * [43] Chieh Hubert Lin, Hsin-Ying Lee, Willi Menapace, Menglei Chai, Aliaksandr Siarohin, Ming-Hsuan Yang, and Sergey Tulyakov.  Infinicity: Infinite-scale city synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22808–22818, October 2023.
  * [44] Yiqi Lin, Hao Wu, Ruichen Wang, Haonan Lu, Xiaodong Lin, Hui Xiong, and Lin Wang.  Towards language-guided interactive 3d generation: Llms as layout interpreter with generative feedback.  arXiv preprint arXiv:2305.15808, 2023.
  * [45] Markus Lipp, Daniel Scherzer, Peter Wonka, and Michael Wimmer.  Interactive modeling of city layouts using layers of procedural content.  In Computer Graphics Forum, volume 30, pages 345–354. Wiley Online Library, 2011.
  * [46] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298–9309, 2023.
  * [47] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan.  Videofusion: Decomposed diffusion models for high-quality video generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10209–10218, 2023.
  * [48] Rui Ma, Akshay Gadi Patil, Matthew Fisher, Manyi Li, Sören Pirk, Binh-Son Hua, Sai-Kit Yeung, Xin Tong, Leonidas Guibas, and Hao Zhang.  Language-driven synthesis of 3d scenes from scene databases.  ACM Transactions on Graphics (TOG), 37(6):1–16, 2018.
  * [49] OpenAI.  Gpt-4 technical report, 2023.
  * [50] OpenAI.  GPT-4V(ision) System Card, 2023.
  * [51] Yichen Ouyang, Wenhao Chai, Jiayi Ye, Dapeng Tao, Yibing Zhan, and Gaoang Wang.  Chasing consistency in text-to-3d generation from a single image.  arXiv preprint arXiv:2309.03599, 2023.
  * [52] Yoav IH Parish and Pascal Müller.  Procedural modeling of cities.  In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 301–308, 2001.
  * [53] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.
  * [54] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville.  Film: Visual reasoning with a general conditioning layer.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * [55] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  arXiv preprint arXiv:2307.01952, 2023.
  * [56] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [57] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  In International Conference on Learning Representations, 2023.
  * [58] Michael JD Powell.  An efficient method for finding the minimum of a function of several variables without calculating derivatives.  The computer journal, 7(2):155–162, 1964.
  * [59] Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nie, and Tat-Seng Chua.  Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image generation.  In Proceedings of the 31st ACM International Conference on Multimedia, pages 643–654, 2023.
  * [60] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [61] Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf.  Sayplan: Grounding large language models using 3d scene graphs for scalable task planning.  arXiv preprint arXiv:2307.06135, 2023.
  * [62] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084, 2019.
  * [63] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [64] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  * [65] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023.
  * [66] Yuan Shen, Wei-Chiu Ma, and Shenlong Wang.  Sgam: Building a virtual 3d world through simultaneous generation and mapping.  Advances in Neural Information Processing Systems, 35:22090–22102, 2022.
  * [67] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al.  Moviechat: From dense token to sparse memory for long video understanding.  arXiv preprint arXiv:2307.16449, 2023.
  * [68] Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, and Gaoang Wang.  Moviechat+: Question-aware sparse memory for long video question answering.  arXiv preprint arXiv:2404.17176, 2024.
  * [69] Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zishan Qin, and Stephen Gould.  3d-gpt: Procedural 3d modeling with large language models.  arXiv preprint arXiv:2310.12945, 2023.
  * [70] Jerry O Talton, Yu Lou, Steve Lesser, Jared Duke, Radomír Mech, and Vladlen Koltun.  Metropolis procedural modeling.  ACM Trans. Graph., 30(2):11–1, 2011.
  * [71] Fuwen Tan, Song Feng, and Vicente Ordonez.  Text2scene: Generating compositional scenes from textual descriptions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6710–6719, 2019.
  * [72] Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner.  Diffuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis.  arXiv preprint arXiv:2303.14207, 2023.
  * [73] Arash Vahdat, Karsten Kreis, and Jan Kautz.  Score-based generative modeling in latent space.  Advances in Neural Information Processing Systems, 34:11287–11302, 2021.
  * [74] Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, Karsten Kreis, et al.  Lion: Latent point diffusion models for 3d shape generation.  Advances in Neural Information Processing Systems, 35:10021–10039, 2022.
  * [75] Vladimir Vezhnevets, Anton Konushin, and Alexey Ignatenko.  Interactive image-based urban modeling.  In Proc. of PIA, pages 63–68, 2007.
  * [76] Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner.  Sceneformer: Indoor scene generation with transformers.  In 2021 International Conference on 3D Vision (3DV), pages 106–115. IEEE, 2021.
  * [77] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  arXiv preprint arXiv:2305.16213, 2023.
  * [78] Qiuhong Anna Wei, Sijie Ding, Jeong Joon Park, Rahul Sajnani, Adrien Poulenard, Srinath Sridhar, and Leonidas Guibas.  Lego-net: Learning regular rearrangements of objects in rooms.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19037–19047, 2023.
  * [79] Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, and Ziwei Liu.  Citydreamer: Compositional generative model of unbounded 3d cities.  arXiv preprint arXiv:2309.00610, 2023.
  * [80] Yong-Liang Yang, Jun Wang, Etienne Vouga, and Peter Wonka.  Urban pattern: Layout design by hierarchical domain splitting.  ACM Transactions on Graphics (TOG), 32(6):1–12, 2013.
  * [81] Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al.  Holodeck: Language guided generation of 3d embodied ai environments.  In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024), volume 30, pages 20–25. IEEE/CVF, 2024.
  * [82] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang.  The dawn of lmms: Preliminary explorations with gpt-4v (ision).  arXiv preprint arXiv:2309.17421, 9(1):1, 2023.
  * [83] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis.  Lion: Latent point diffusion models for 3d shape generation.  arXiv preprint arXiv:2210.06978, 2022.
  * [84] Chi Zhang, Yiwen Chen, Yijun Fu, Zhenglin Zhou, Gang Yu, Billzb Wang, Bin Fu, Tao Chen, Guosheng Lin, and Chunhua Shen.  Styleavatar3d: Leveraging image-text diffusion models for high-fidelity 3d avatar generation.  arXiv preprint arXiv:2305.19012, 2023.
  * [85] Jingbo Zhang, Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao.  Text2nerf: Text-driven 3d scene generation with neural radiance fields.  arXiv preprint arXiv:2305.11588, 2023.
  * [86] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3836–3847, October 2023.
  * [87] Yiqun Zhao, Zibo Zhao, Jing Li, Sixun Dong, and Shenghua Gao.  Roomdesigner: Encoding anchor-latents for style-consistent and shape-compatible indoor scene generation.  arXiv preprint arXiv:2310.10027, 2023.
  * [88] Zhonghan Zhao, Wenhao Chai, Xuan Wang, Li Boyi, Shengyu Hao, Shidong Cao, Tian Ye, Jenq-Neng Hwang, and Gaoang Wang.  See and think: Embodied agent in virtual environment.  arXiv preprint arXiv:2311.15209, 2023.
  * [89] Zhonghan Zhao, Kewei Chen, Dongxu Guo, Wenhao Chai, Tian Ye, Yanting Zhang, and Gaoang Wang.  Hierarchical auto-organizing system for open-ended multi-agent navigation.  arXiv preprint arXiv:2403.08282, 2024.
  * [90] Zhonghan Zhao, Ke Ma, Wenhao Chai, Xuan Wang, Kewei Chen, Dongxu Guo, Yanting Zhang, Hongwei Wang, and Gaoang Wang.  Do we really need a complex agent system? distill embodied agent into a single model.  arXiv preprint arXiv:2404.04619, 2024.

Supplementary Material

The supplementary material is structured as follows:

  * •

We begin with the “CityCraft Algorithm” section, detailing the CityCraft
method, encompassing input requirements, variables, functions, and the
procedure of the CityCraft Method algorithm in Section 7.

  * •

We provide “Detailed Performance” section for more detailed in evolution of
LLM performance over iterations.

  * •

We deliver “Dataset Example” section for more detailed examples of Dataset
Assets.

  * •

We supply “User Study Settings” section for details in the scoring method of
preference in experiment.

  * •

We list “Demo Visualization” section for more detailed demo of city scenes
generated by CityCraft.

##  7 CityCraft Algorithm

#### Preliminary

Diffusion Models [32, 63] are state-of-the-art generative models known for
their high-quality, photorealistic image synthesis. It operates on the
principle of simulating a stochastic process where Gaussian noise is gradually
added to an image at each step, and the model is trained to reverse this
process, removing the noise and reconstructing the original image. The reverse
process is typically learnt using an UNet with text conditioning support
enabling text-to-image generation during inference. Specifically, given an
initial noise map $\epsilon\sim\mathcal{N}(0,I)$ and a text-image pair
$(c,x)$, they are trained using a squared error loss to denoise a variably-
noised image as follows:

$\mathbb{E}_{x,c,\epsilon,t}[w_{t}||\hat{x}_{\theta}(\alpha_{t}x+\sigma_{t}\epsilon,c)-x||_{2}^{2}]$
(2)

where x is the ground-truth image, c is a text prompt, and
$\alpha_{t},\sigma_{t},w_{t}$ are terms that control the noise schedule and
sample quality.

#### Method

The CityCraft framework is designed to generate detailed and controllable
urban 3D scenes from user-defined parameters. CityCraft transforms initial
textual and ratio-based inputs into complex urban environments using layout
generation, urban planning, and scene construction techniques. Here, we
present the pseudo-code that encapsulates the entire process.

As outlined in Algorithm 1, CityCraft implements a three-stage framework
$\mathcal{F}$ designed to convert user requirements into detailed 3D urban
scenes. This transformation process is formulated as follows:

$Y=\mathcal{F}(X),\quad X\in\\{\mathbf{Text},\mathbf{Ratio}\\},$ (3)

where $\mathbf{Text}$ represents user requirements for urban design, and
$\mathbf{Ratio}\in\mathbb{R}^{1\times C}$ denotes the probabilities of each
class in a semantic layout $L$, with $C$ indicating the number of classes.

In detail, the process begins with generating semantic city $\mathbf{Layout}$,
employing a layout generator $\mathcal{G}$. This generator processes an
initial noise vector $\epsilon\sim\mathcal{N}(0,I)$ and a condition vector
$\mathbf{Condition}$:

$\mathbf{Layout}=\mathcal{G}(\epsilon,\mathbf{Condition}),\quad\mathbf{Condition}=\mathcal{E}(X),$
(4)

where $\mathbf{Condition}$ is defined as text embeddings and probabilities of
each prototype class from encoding the user input $X$ with encoder
$\mathcal{E}$.

After generating the initial layout, the urban planning module $\mathcal{P}$
refines $\mathbf{Layout}$ containing user requirement information to create a
more detailed $\mathbf{Plan}$:

$\mathbf{Plan}=\mathcal{P}(\mathbf{Layout}),$ (5)

Following the planning phase $\mathbf{Plan}$, the self-adapting system
$\mathcal{A}dapt$ selects and assigns appropriate assets for
$\mathbf{Layout}$. This system ensures that each asset is perfectly integrated
into the urban 3D scene $Y$, conforming to both user specifications and
spatial dynamics:

$Y=\mathcal{A}dapt(\mathbf{Plan},\mathbf{Layout}),$ (6)

where the final 3D scene $Y$ is rendered based on explainable $\mathbf{Plan}$
and original $\mathbf{Layout}$, completing the transformation from concept to
visualization.

11: Input: User input $X=\\{\mathbf{Text},\mathbf{Ratio}\\}$

22: Output: Generated urban scene $Y$

3: Initialize:

34: $\mathbf{Condition}\leftarrow\mathcal{E}(X)$ {Encode user input into
condition vector}

5: Step 1: Generate Initial Layout

6: $\epsilon\sim\mathcal{N}(0,I)$ {Generate initial noise vector}

47: $\mathbf{Layout}\leftarrow\mathcal{G}(\epsilon,\mathbf{Condition})$
{Generate layout using noise and condition}

8: Step 2: Refine Layout into Urban Plan

59: $\mathbf{Plan}\leftarrow\mathcal{P}(\mathbf{Layout})$ {Refine layout to
create detailed plan}

10: Step 3: Asset Selection and Scene Adaptation

611: $Y\leftarrow\mathcal{A}dapt(\mathbf{Plan},\mathbf{Layout})$ {Select and
place assets according to the plan and layout}

12: return $Y$ {Return the final rendered 3D urban scene}

Algorithm 1 CityGen-2 Urban Scene Generation Process $\mathcal{F}$

##  8 Detailed Performance

As shown in Figure H1, one represents the rate of change, and the other
illustrates the output quality as a function of the LLM’s iteration index. As
the index increases, indicating more iterations of the LLM, the rate of change
curve likely shows a decline or stabilization, suggesting that the model is
reaching a point of diminishing returns in terms of learning new patterns or
making significant adjustments to its outputs. Meanwhile, the quality curve is
expected to show a steady increase, indicating that the overall performance
and output quality of the LLM are improving with each iteration despite the
slower rate of change.

Figure H1: Evolution of LLM performance over iterations. The graph illustrates
the relationship between the iteration index and the LLM’s output dynamics. As
the iteration index increases, the rate of change stabilizes, indicating a
maturation in learning, while the quality of the outputs continues to rise
steadily. This trend underscores the LLM’s increasing efficiency and
effectiveness in generating high-quality results as it undergoes more
iterations.

##  9 Dataset Example

As shown in Figure I2, we compare the CityCraft-OSM dataset and the
CityDreamer-OSM dataset to highlight our dataset’s enhancements. The
CityCraft-OSM dataset focuses on enhancing the clarity and organization of map
data, improving legibility and computational efficiency. Additionally,
including real-world satellite imagery in the CityCraft-OSM-Satellite dataset
provides a ground-truth comparison, showcasing our dataset’s accuracy and
practical alignment with real-world urban geographies. These advancements make
CityCraft a valuable tool for urban planners, architects, and developers
looking to create more realistic and functional urban simulations.

As shown in Figure K4,

Figure I2: Comparison between the CityCraft-OSM and CityDreamer-OSM datasets.
Top row: CityDreamer-OSM [79] visualizations with dense urban details. Middle
row: CityCraft-OSM (ours) showcases enhanced clarity and organization in urban
mapping. Bottom row: CityCraft-OSM-Satellite (ours) provides real-world
satellite imagery, reflecting the accuracy of our city scene representations
compared to actual geographical layouts. This comparison highlights our
dataset’s clarity, usability, and real-world accuracy improvements.

##  10 User Study Settings

To better asses the quality of the generated city layouts and the generated
city scenes, we conduct user studies and invite 22 participants, including 12
undergraduate students, 8 graduate students, and 2 faculty members. We set the
following metrics for evaluating image quality: fidelity, style, consistency,
controllability, clarity, sharpness, and overall quality. Each metric has a
score from 0 ~10, where a score of 10 is perfect, 8 ~9 is excellent, 6 ~7 is
good, 4 ~5 is average, 2 ~3 is poor and 0 ~1 is terrible. In each experiments,
we provide the users a set of images generated from different models, and let
them give ratings for each model. We compute the final score of each model as
the average score of all user’s rating, excluding the highest and lowest
scores.

##  11 Demo Visualization

This section presents a series of demonstrative visualizations highlighting
the advanced capabilities of CityCraft in generating highly detailed and
realistic city scenes. As shown in Figure K3, these visualizations serve not
only as a testament to the technological prowess of CityCraft but also provide
insight into its practical applications in urban planning and architecture.

###  11.1 Comprehensive Cityscape Rendering

The top image offers a panoramic view of the urban landscape generated by
CityCraft, showcasing a diverse array of building architectures and
meticulously planned urban layouts. This view illustrates how CityCraft
handles complex spatial relationships and urban density precisely,
facilitating a realistic portrayal of large-scale urban environments.

###  11.2 Detailed Environmental Interactions

The bottom left image focuses on specific areas within the city, such as
public squares, parks, and individual buildings. This visualization emphasizes
the detailed textural and material qualities that CityCraft can achieve, from
the surface textures of roads and pathways to the varied facades of buildings.
The attention to detail in these components underscores the model’s utility in
simulating real-world conditions and enhancing urban design’s visual and
functional aspects.

###  11.3 Street-Level Perspectives

The bottom right image provides a street-level perspective that brings the
viewer into the urban environment created by CityCraft. This perspective is
crucial for evaluating the human-scale aspects of urban design, such as
walkability, the integration of greenery, and the interaction between
different urban elements. It allows planners and architects to assess how well
the virtual environment aligns with human-centered design principles.

These demonstrations collectively display the versatility and depth of
CityCraft’s rendering capabilities, offering stakeholders a powerful tool for
visualizing and planning future urban developments with unprecedented detail
and realism.

Figure K3: Demo visualization of city scenes generated by CityCraft. It
demonstrates the framework’s capacity to render complex city environments with
high realism. Top: An overall cityscape view showing varied architectural
styles and detailed urban planning. Bottom left: A closer view highlighting
the intricate modeling of public spaces and individual buildings. Bottom
right: A street-level perspective provides a realistic visualization of the
urban environment, emphasizing texture quality and integrating natural
elements like trees and parks. Figure K4: 3D model examples of architectural
assets with corresponding metadata. The top row features modern-style
commercial buildings with varied sections and roof heights, identified as mid-
rise (12 floors). The bottom row showcases a large, sprawling residential
estate with multiple roofs and intricate details, classified as low-rise (2
floors). Each asset is presented from multiple viewpoints alongside its
metadata, including function, architectural style, and object count.


# An Unsupervised Learning-Based Approach for Symbol-Level-Precoding

Abdullahi Mohammad1, Christos Masouros1 and Yiannis Andreopoulos1

e-mail: (abdullahi.mohammad.16; c.masouros<EMAIL_ADDRESS>1Department of Electronic and Electrical Engineering, University College
London, WC1E 7JE UK

###### Abstract

This paper proposes an unsupervised learning-based precoding framework that
trains deep neural networks (DNNs) with no target labels by unfolding an
interior point method (IPM) proximal ‘log’ barrier function. The proximal
‘log’ barrier function is derived from the strict power minimization
formulation subject to signal-to-interference-plus-noise ratio (SINR)
constraint. The proposed scheme exploits the known interference via symbol-
level precoding (SLP) to minimize the transmit power and is named strict
Symbol-Level-Precoding deep network (SLP-SDNet). The results show that SLP-
SDNet outperforms the conventional block-level-precoding (Conventional BLP)
scheme while achieving near-optimal performance faster than the SLP
optimization-based approach.

##  I Introduction

Recent studies on interference exploitation have shown that known inferences
can be effectively managed and transformed into valuable signals to improve
the system’s quality-of-service (QoS) [1]. The concept of constructive
interference (CI) is first introduced in [1], where instantaneous interference
is categorized into constructive and destructive. Traditionally, multi-user
interference (MUI) is suppressed in block-level precoding designs. However,
the symbol-level precoding (SLP) technique utilizes the transmitted symbol to
convert the MUI into useful signals [2]. Suboptimal strategies that exploit CI
are first introduced in [2]. Optimal SLP schemes using convex optimization-
based CI with strict phase constraints on the received constellation point are
proposed in [3, 4, 5]. Despite the performance benefits offered by the
optimization-based SLP schemes, computational complexity is still an issue in
their implementation on practical systems.

Due to the low computational cost of online training, there has been an
increasing interest in designing deep learning (DL) precoding schemes recently
for MU-MISO downlink transmission [6, 7, 8, 9, 10, 11]. For example, in [6]
the authors propose a DL-based coordinated beamforming technique to improve
the link reliability and low latency in millimeter-wave (mmWave)
communications. A deep neural network (DNN) precoding method is introduced in
[7] for decentralized decision making. An unsupervised learning (UL) based
beamforming scheme that explores the optimal solution of weighted-sum-rate is
proposed in [8]. Convolutional neural networks (CNNs) framework for downlink
beamforming optimization using expert knowledge based on the known structure
of optimal iterative solutions is investigated in [9]. The application of DL
on SLP is further studied in [10], where a deep autoencoder-based framework is
designed for robust SLP and symbol detection. While the approach has low
computational complexity, decision rules for symbol detection at the receivers
are practically challenging to implement. To further enhance training
efficiency, a computationally low-cost DNN-based SLP design is proposed in
[11].

However, most of the learning-based strategies mentioned above are based on
supervised learning, where the constraints are implicitly contained in the
training dataset obtained from conventional optimization solutions. This
requires solving the optimization problem twice, first by traditional
optimization and second by approximating the optimal solution using DNN.
However, if it was difficult to obtain the optimal solutions via conventional
optimization methods, the learning-based solutions may be impractical.

This paper proposes a learning-based precoding scheme that requires no target
labels for power minimization problems under signal-to-interference-noise-
ratio (SINR). The learning framework is designed by unfolding an IPM iterative
algorithm via IPM proximal log barrier function that considers the convexity
of the inequality constraint. A case scenario of strict phase angle rotation
is considered under a known perfect channel condition.

##  II System Model

Suppose a MISO downlink channel in a single cell with $N$ transmit antennas at
the base station (BS) serves $K$ single-antenna users. The channel between
users and the BS is assumed to be quasi-static flat-fading and is denoted by
$\mathbf{h}_{i}\in\mathbb{C}^{{N}\times 1}$.

###  II-A Conventional Power Minimization

Traditionally, the power minimization problem tries to minimize the average
transmit power by handling all interference as harmful subject to QoS
constraints, as described below [12]

$\displaystyle\underset{\mathbf{\\{v_{i}\\}}}{\text{min}}$
$\displaystyle{\sum_{i=1}^{K}\left\lVert\mathbf{v}_{k}\right\rVert_{2}^{2}}$
(1) s.t.
$\displaystyle\frac{|\mathbf{h}_{i}^{T}\mathbf{v}_{i}|^{2}}{\sum_{k=1,k\neq
i}|\mathbf{h}_{i}^{T}\mathbf{v}_{k}|^{2}+{N}_{0}}\geq\Gamma_{i}\ ,\
\forall{i}.$

where $\Gamma_{i}$ is the SINR of the i-th user. From an instantaneous
viewpoint, problem (1) does not consider the fact that interference can
additively improve the received signal power [13]. Therefore, the solution is
sub-optimal.

Figure 1: Generic geometrical optimization regions for interference
exploitation for Precoding design in QPSK [3]

Fig. 1 shows the the generic geometrical representation of the CI. The real
part of the received symbol $(\lambda_{Re})$ gives a measure of the received
constellation along the theoretical constellation axis. Likewise, the
imaginary part $(\lambda_{Im})$ shows the extent of the phase displacement
from the primary constellation point.

###  II-B Power Minimization via Symbol-Level Precoding

The instantaneous interference in a multi-user downlink channel scenario for M
array phase shift keying (M-PSK) modulation can be categorized into
constructive and destructive based on the known criteria defined in [14].
Therefore, CI is defined as the interference that pushes the received symbols
away from the modulated-symbol constellation’s decision edges [3]. For further
details on SLP and its formulation (3), we refer the reader to [15].
Therefore, the problem in (1) is modeled to incorporate CI in the power
minimization formulation. Consequently, the interfering signals align with the
symbol of interest constructively by precoding vectors, offering useful
signals. Hence, for M-PSK, the power minimization SLP-based optimization can
be reformulated based on the classification criteria explained in [3]

$\displaystyle\underset{\mathbf{\\{v_{i}\\}}}{\text{min}}$
$\displaystyle{\left\lVert\sum_{k=1}^{K}\mathbf{v}_{k}{e}^{j(\phi_{k}-\phi_{1})}\right\rVert_{2}^{2}}$
(2) s.t.
$\displaystyle\text{Im}{\left(\bm{h}_{i}^{T}\sum_{k=1}^{K}\mathbf{v}_{k}e^{j(\phi_{k}-\phi_{i})}\right)=0}\
,\ \forall{i}$
$\displaystyle\text{Re}{\left(\bm{h}_{i}^{T}\sum_{k=1}^{K}\mathbf{v}_{k}e^{j(\phi_{k}-\phi_{i})}\right)\geq\sqrt{\Gamma_{i}N_{0}}}\
,\ \forall{i}.$

##  III Learning-Based SLP for Power minimization problem

This section presents detailed formulations of a learning-based CI power
minimization problem for strict phase angle rotation assuming a perfect
channel state information (CSI) known at the BS. The power minimization
problem for the case where the phase angle of the interfering symbols strictly
aligns with the angle of the symbols of interest. If the maximum angle shift
in the constructive interference region is zero, i.e., all the interfering
signals completely overlap on the signal of interest ($\varphi=0$, see [15]
for details).

It is often difficult to derive the closed-form solution to problem
$\mathcal{}{P}_{\text{st}}$ due to the in-equality constraints. Therefore,
conventional iterative solvers are usually used to find sub-optimal solutions.
Motivated by the recent adoption of an IPM for image restoration [16], we
propose an unsupervised learning framework that unfolds a constrained
optimization problem into a sequence of neural network layers for a multi-user
MIMO beamforming. We first convert (2) to a general form of proximal IPM. The
measure of the fidelity of the solution to (2) is determined by learning a set
of penalty parameters in the form of Lagrange multipliers associated with the
constraints. We define the channel vector based on (2) as follows

$\tilde{\mathbf{h}}_{i}=\mathbf{h}_{i}\sum_{k=1}^{K}e^{j(\phi_{k}-\phi_{i})}$
(3)

${\mathbf{v}}=\sum_{k=1}^{K}\mathbf{v}_{k}.$ (4)

From (3) and (4), we have

$\tilde{\mathbf{h}}_{i}\mathbf{v}=(\tilde{\mathbf{h}}_{Ri}+j\tilde{\mathbf{h}}_{Ii})(\mathbf{v}_{R}+j\mathbf{v}_{I})$
(5)

where $\tilde{\mathbf{h}}_{R}=\text{Re}(\tilde{\mathbf{h}}_{i})$,
$\tilde{\mathbf{h}}_{I}=\text{Im}(\tilde{\mathbf{h}}_{i})$,
$\mathbf{v}_{R}=\text{Re}(\mathbf{v})$ and
$\mathbf{v}_{I}=\text{Im}(\mathbf{v})$. Let
$\bm{\Upsilon}=[\tilde{\mathbf{h}}_{R}\ ;\ \tilde{\mathbf{h}}_{I}]$,
$\mathbf{v}_{1}=[\mathbf{v}_{R}\ -\mathbf{v}_{I}]^{T}$ and
$\mathbf{v}_{2}=[\mathbf{v}_{I}\ \ \mathbf{v}_{R}]^{T}$. To simplify the
analysis, we partition the complex vectors into the real and imaginary parts
as follows:
$\text{Re}(\tilde{\mathbf{h}}_{i}^{T}\mathbf{v})=\bm{\Upsilon}_{i}^{T}\mathbf{v}_{1}$
and
$\text{Im}(\tilde{\mathbf{h}}_{i}^{T}\mathbf{v})=\bm{\Upsilon}_{i}^{T}\bm{\Omega}\mathbf{v}_{1}$,
where

${\mathbf{v}_{2}=\bm{\Omega}}\mathbf{v}_{1}\ \text{and}\
\bm{\Omega}=\begin{bmatrix}\mathbf{O}_{N}&-\mathbf{I}_{N}\\\
\mathbf{I}_{N}&\mathbf{O}_{N}\end{bmatrix};\ \in\mathbb{R}^{2N\times 2N}.$ (6)

Therefore, the multicast equivalent of (2) is

$\displaystyle\underset{\mathbf{\\{v_{1}\\}}}{\text{min}}$
$\displaystyle{\left\lVert\mathbf{v}_{1}\right\rVert^{2}}$ (7) s.t.
$\displaystyle{\bm{\Upsilon}_{i}^{T}\bm{\Omega}\mathbf{v}_{1}=0}\ ,\
\forall{i}$
$\displaystyle\bm{\Upsilon}_{i}^{T}\mathbf{v}_{1}\geq\sqrt{\Gamma_{i}N_{0}}\
,\ \forall{i}.$

###  III-A Interior Point Method (IPM)

Consider a general form of a nonlinear constrained optimization of the form
[17]:

$\displaystyle\underset{\mathbf{x\in{\mathbb{R}^{N}}}}{\text{min}}$
$\displaystyle{f(\mathbf{z})}$ (8) s.t.
$\displaystyle\mathcal{C(\mathbf{z})}=0$ $\displaystyle\mathbf{z}\geq 0.$

The reason for adopting IPM is to replace the initial constrained optimization
problem with a chain of unconstrained sub-problems of the form:

$\displaystyle\underset{\mathbf{x\in{\mathbb{R}^{N}}}}{\text{min}}f(\mathbf{z})+\lambda{\mathbcal{D(\mathbf{z})}}+\mu{\mathbcal{B(\mathbf{z})}}.$
(9)

where $\mathbcal{B}$ is the logarithmic barrier function associated with
inequality constraint with unbounded derivative at the boundary of the
feasible domain, $\mathbcal{D}$ is associated with equality constraint, $\mu$
and $\lambda$ are the Lagrangian multipliers for inequality and equality
constraints, respectively.

To facilitate the solution of (7), we introduce additional notations. For
every inequality constraint, $\gamma\in\\{0,+\infty\\}$ and
$\mathbf{v}_{1}\in\mathbb{R}^{2N\times 1}$, we define the proximity operator
as in [17] with respect to (9), which we shall later use to compute the
projected gradient descent as

$\begin{aligned}
\text{prox}_{\gamma\mathcal{G}}{(\mathbf{v}_{1})}=&\underset{\mathbf{v_{1}\in{\mathbb{R}^{2N\times
1}}}}{\text{argmin}}&&{\frac{1}{2}\left\lVert\mathbf{v}_{0}-\mathbf{v}_{1}\right\rVert}_{2}^{2}+\gamma\mathcal{G}({\mathbf{v}_{1}})\end{aligned},$
(10)

where $\gamma$ is the step-size taken for computing the gradients of the
objective function, $\mathcal{G}$ is the function that defines the barrier
operator and $\mathbf{v}_{0}$ is the initial value of the precoding vector.

To convert (7) into its equivalent barrier function problem, we get raid of
the inequality constrain and translate it into a barrier term of the form [18]

$\displaystyle\underset{\mathbf{x_{i}\in{\mathbb{R}^{n}}}}{\text{min}}$
$\displaystyle{f(\mathbf{z})}-\mu\sum_{i=1}^{p}\ln(\mathbf{z_{i})}$ (11) s.t.
$\displaystyle\mathcal{C(\mathbf{z})}=0.$

####  III-A1 Affine Constraints

Consider a half-space constraint expressed as [16]:

$\mathbcal{C}=\\{\mathbf{z}\in\mathbb{R}^{N}\lvert{b}^{T}\mathbf{z}\leq{c}\\}$
(12)

As shown in [16], the $\mathbcal{B}$ function associated to (12) is defined as

$\mathbcal{B(\mathbf{z})}=\left\\{\begin{aligned}
&-\ln{\left(c-b^{T}\mathbf{z}\right)},&&\text{if}\ b^{T}\mathbf{z}<c,\
\forall\ \mathbf{z}\in\mathbb{R}^{N}\\\
&+\infty,&&\text{otherwise}\end{aligned}\right.$ (13)

Following (11), we can express (7) as

$\displaystyle\underset{\mathbf{w_{1}}}{\text{min}}$
$\displaystyle{f(\mathbf{v}_{1})}-\mu\sum_{i=1}^{p}{\ln\left(\bm{\Upsilon}_{i}^{T}\mathbf{v}_{1}-\sqrt{\Gamma_{i}N_{0}}\right)}$
(14) s.t. $\displaystyle{\bm{\Upsilon_{i}^{T}\Omega}{\mathbf{v}_{1}}=0}.$

For all $\mu>0$, $\lambda>0$ and $\mathbf{v}_{1}$, we define $\mathbcal{B}$ as
in (13), so that the proximity operator can be defined as follows

$\Phi(\mathbf{v}_{1},\gamma,\mu)=\text{prox}_{\gamma\mu\bm{\mathbcal{B}}}{(\mathbf{v}_{1})}.$
(15)

In what follows, we provide the expression of $\Phi$ and its corresponding
derivatives with respect to the optimization variable $\mathbf{v}_{1}$, the
step-size and the barrier parameters $(\gamma,\mu)$ for affine constraint,
which will be used for training the neural network using a gradient
backpropagation algorithm. Finally, following the above formulations, the
proximal barrier function for the strict phase rotation is reduced to the
following expression

$\mathbcal{B(\mathbf{v}_{1})}=\left\\{\begin{aligned}
&-\ln{\left(\bm{\Upsilon}_{i}^{T}\mathbf{v}_{1}-\sqrt{{\Gamma}_{i}N_{0}}\right)},&&\text{if}\
\bm{\Upsilon}_{i}^{T}\mathbf{v}_{1}\geq\sqrt{{\Gamma}_{i}N_{0}}\\\
&+\infty,&&\text{otherwise.}\end{aligned}\right.$ (16)

It can be easily shown that for every precoding vector
$\mathbf{v}_{1}\in\mathbb{R}^{2N\times 1}$, the proximity operator of
$\mu\gamma\mathbcal{B}$ at $\mathbf{v}_{1}$ is given by

$\Phi(\mathbf{v}_{1},\mu,\gamma)=\mathbf{v}_{1}+\\\
\frac{\bm{\Upsilon_{i}}^{T}\mathbf{v}_{1}-\sqrt{\Gamma_{i}N_{0}}-\sqrt{(\bm{\Upsilon}_{i}^{T}\mathbf{v}_{1}-\sqrt{\Gamma_{i}N_{0}})^{2}+4\gamma\mu\left\lVert\bm{\Upsilon}_{i}^{T}\right\rVert_{2}^{2}}}{{{2\left\lVert\bm{\Upsilon}_{i}\right\rVert_{2}^{2}}}}\bm{\Upsilon}_{i}.$
(17)

Furthermore, the Jacobian matrix of $\Phi$ with respect to $\mathbf{v}_{1}$,
and the derivatives of $\Phi$ with respect to $\gamma$ and $\mu$ are as
follows

$\mathcal{J}_{\Phi}\mid_{(\mathbf{v}_{1})}=\mathbf{I}_{2N}+\frac{1}{2\left\lVert\bm{\Upsilon}_{i}\right\rVert_{2}^{2}}\
\times\\\
\left(1-\frac{\bm{\Upsilon}_{i}^{T}-\sqrt{\Gamma_{i}N_{0}}}{\sqrt{(\bm{\Upsilon}_{i}^{T}\mathbf{v}_{1}-\sqrt{\Gamma_{1}N_{0}})^{2}+4\gamma\mu\left\lVert\bm{\Upsilon}_{i}^{T}\right\rVert_{2}^{2}}}\right)\bm{\Upsilon_{i}\Upsilon_{i}^{T}}$
(18)

$\Delta_{\Phi}\mid_{({\mu})}=\frac{-\gamma}{{\sqrt{(\bm{\Upsilon}_{i}^{T}\mathbf{v}_{1}-\sqrt{\Gamma_{1}N_{0}})^{2}+4\gamma\mu\left\lVert\bm{\Upsilon}_{i}^{T}\right\rVert_{2}^{2}}}}\bm{\Upsilon_{i}}$
(19)

$\Delta_{\Phi}\mid_{({\gamma})}=\frac{-\mu}{{\sqrt{(\bm{\Upsilon}_{i}^{T}\mathbf{v}_{1}-\sqrt{\Gamma_{1}N_{0}})^{2}+4\gamma\mu\left\lVert\bm{\Upsilon}_{i}^{T}\right\rVert_{2}^{2}}}}\bm{\Upsilon_{i}}$
(20)

where $\mathbf{I}\in\mathbb{R}^{2(N\times N)}$ is identity matrix.

Finally, the learning algorithm for every update rule is thus the unfolded (7)
as a sequence of sub-problems with respect to the constraints as follows

$\displaystyle\underset{\mathbf{v_{1}\in{\mathbb{R}}^{2N\times
1}}}{\text{min}}$
$\displaystyle{\left\lVert\mathbf{v}_{1}\right\rVert}_{2}^{2}+\lambda(\bm{\Upsilon_{i}^{T}\Omega}{\mathbf{v}_{1}})+\mu{\mathbcal{B(\mathbf{v}_{1})}}.$
(21)

Using the proximity operator of the barrier, the update rule for every
iteration is given by

$\mathbf{v}_{1}^{[r+1]}=\text{prox}_{\gamma^{[r]}\mu^{[r]}\mathcal{B}}\left(\mathbf{v}_{1}^{[r]}-\gamma^{[r]}\Delta{\mathbcal{E}(\mathbf{v}_{1}^{[r]},\lambda^{[r]})}\right),$
(22)

where

$\mathbcal{E}(\mathbf{v}_{1}^{[r]},\lambda^{[r]})={\left\lVert\mathbf{v}_{1}\right\rVert}_{2}^{2}+\lambda(\bm{\Upsilon_{i}^{T}\Omega}{\mathbf{v}_{1}}).$
(23)

The update function can thus be expressed as

$\mathbcal{H}(\mathbf{v}_{1}^{[r]},\gamma^{[r]},\mu^{[r]},\lambda^{[r]})=\\\
\text{prox}_{\gamma^{[r]}\mu^{[r]}\mathcal{B}}\left(\mathbf{v}_{1}^{[r]}-\gamma^{[r]}\Delta{\mathbcal{E}(\mathbf{v}_{1}^{[r]},\lambda^{[r]})}\right),$
(24)

and
$\Delta=\frac{\partial{\mathbcal{E}(\mathbf{v}_{1}^{[r]},\lambda^{[r]})}}{\partial{\mathbf{v}_{1}^{[r]}}}$.

####  III-A2 Duality and Loss Function for the Strict Phase Formulation

Since we are interested in learning the optimal solution via unsupervised
learning (without target labels), we firstly formulate a primal-dual problem.
This formulation is then used to derive the optimization variable (precoding
vector) as a function of dual variables (Lagrangian multipliers) associated
with the constraints. The Lagrangian function can be expressed as

$\mathcal{L}_{\text{st}}(\mathbf{v}_{1},\lambda,\
\mu)=\|\mathbf{v}_{1}\|_{2}^{2}+\sum^{K}_{k=1}\lambda_{k}\bm{\Upsilon}^{T}_{i}\bm{\Omega}\mathbf{v}_{1}+\\\
\sum^{K}_{k=1}\mu_{k}\left(\sqrt{\Gamma_{i}N_{0}}-\bm{\Upsilon}^{T}_{i}\mathbf{v}_{1}\right).$
(25)

The optimal precoder can be found by minimizing (25) with respect to
$\mathbf{v}_{1}$ (differentiating $\mathcal{L}_{\text{st}}(\cdotp)$ w.r.t
$\mathbf{v}_{1}$). The optimal precoder is thus

$\mathbf{v}_{1}=\frac{\bm{\mu}^{T}\cdotp\bm{\Upsilon}_{i}-\bm{\lambda}^{T}\cdotp\bm{\Omega}\bm{\Upsilon}_{i}}{2}.$
(26)

The above expression in (26) is used to generate the training input (precoding
vector) by initializing the Lagrangian multipliers ($\lambda$ and $\mu$)
randomly and then train the neural network to learn their best values that
minimize the loss function (Lagrangian function). The loss function is
modified by adding $\mathbcal{l}2$-norm regularization over the weights to
adjust the learning coefficients to stabilize the learning process. The loss
function over $B$ training batches is finally expressed as

$\mathcal{L}_{\text{st}}(\mathbf{v}_{1},\lambda,\
\mu)=\frac{1}{B}\sum^{B}_{i=1}\left(\|\mathbf{v}_{1}\|_{2}^{2}+\bm{\lambda}\bm{\Upsilon}^{T}_{i}\bm{\Omega}\mathbf{v}_{1}\right)+\\\
\frac{1}{B}\sum^{B}_{i=1}\left(\bm{\mu}\left(\sqrt{\Gamma_{i}N_{0}}-\bm{\Upsilon}^{T}_{i}\mathbf{v}_{1}\right)\right)+\frac{\vartheta}{BL}\sum^{B}_{i=1}\sum_{i=1}^{L}\|\bm{\theta}_{i}\|_{2}^{2},$
(27)

where $\bm{\theta}$ is the learning parameter associated with the weights and
$\vartheta>0$ is the penalty parameter that controls the bias and variance of
the learning coefficients, $B$ and $L$ are training batch size (number of
channel realization) and the number of layers respectively.

Figure 2: Complete SLP-SDNet Architecture showing the internal structure of
the Barrier Operator.

###  III-B Deep Proximal Strict Symbol-Level Precoding Network (SLP-SDNet)

The optimization problem is unfolded over r-th iterations, and the Lagrange
multiplier associated with the equality constraint is wired across the network
to provide additional flexibility [16]. The unfolded neural network is trained
in an unsupervised fashion without target labels. We build the structure of
the learning framework based on (22) and the algorithm presented in [16],
which gives rise to Algorithm 1.

Algorithm 1 Feed-forward-Backward Proximal IPM

0: $\mathbf{v}_{1}^{[0]}$, ${\gamma}^{[0]}$, ${\bm{\lambda}}^{[0]}$ and
$\bm{\mu}^{[0]}$

0: $\mathbf{v}_{1}$ Initialization :

1: randomly initialize $\mathbf{v}_{1}^{[0]}\in{\mathbb{R}^{2N\times 1}}$,
$\bm{\mu}^{[0]}>0$, $\bm{\lambda}^{[0]}>0$ and $\gamma^{[0]}>0$ $\forall\
{i}=1,\ \cdots,\ K$ Loop over r-th iterations

2: for $r=0$ to $L$ do

3:
$\mathbf{v}_{1}^{[r+1]}=\text{prox}_{\gamma^{[r]}\mu^{[r]}\mathcal{B}}\left(\mathbf{v}_{1}^{[r]}-\gamma^{[r]}\Delta{\mathbcal{E}(\mathbf{v}_{1}^{[r]},\lambda^{[r]})}\right).$

4: end for

5: return $\mathbf{v}_{1}$

Algorithm 2 Proximity Barrier Operator for Strict phase rotation

0: $\mathbf{h}_{\text{{R}}}$, $\mathbf{h}_{\text{{I}}}$, $\Gamma_{i}$ and
${N}_{0}\ (\text{noise power})$

0: $\mathbf{v}_{1}$, $\gamma$, $\bm{\mu}$ and $\bm{\lambda}$ Initialization :

1: randomly initialize $\mathbf{v}_{0}\in{\mathbb{R}^{2N\times 1}}$,
$\bm{\mu}^{[0]}>0$, $\bm{\lambda}^{[0]}>0$ and $\gamma^{[0]}>0$ $\forall\
{i}=1,\ \cdots,\ K$.

2: Compute the Barrier function $\mathbcal{B(\mathbf{v}_{1})}$ using function
(16).

3: Compute the Proximity Operator of the Barrier at $\mathbf{v}_{0}$ using
(10), where $\mathcal{G}=\mu\mathbcal{B(\mathbf{v}_{1})}$ .

4: Compute the derivatives of the Proximity Operator w.r.t $\mathbf{v}_{1}$,
$\mu$ and $\gamma$ using (18), (19) and (20).

5: Update the training variables as follows:

  1. (a)

$\mu^{[r+1]}=\mu^{[r]}-\eta\frac{\partial{\Phi}}{\partial{\mu}}$

  2. (b)

$\gamma^{[r+1]}=\gamma^{[r]}-\eta\frac{\partial{\Phi}}{\partial{\gamma}}$

  3. (c)

$\lambda^{[r+1]}=\lambda^{[r]}-\eta\frac{\partial{\mathbcal{E}(\mathbf{v}_{1}^{[r]},\lambda^{[r]})}}{\partial{\lambda^{[r]}}}$
using (23)

where $\eta$ is the learning rate.

6: Use the results in step 5 and the Algorithm 1 to obtain the optimal
precoding tensor.

For every r-th iterations (r-th layer) $\mathcal{L}^{[r]}$, there exist three
latent structures associated with the learnable parameters ($\mu$, $\gamma$
and $\mu$) $\mathcal{L}_{\mu}^{[r]}$, $\mathcal{L}_{\gamma}^{[r]}$ and
$\mathcal{L}_{\lambda}^{[r]}$. As shown in Fig. 2, each of these structures
forms a learning block for computing the barrier parameter ($\mu$) associated
with the inequality constraint, the step-size for update rule ($\gamma$) and
finally ($\lambda$), which is related to the equality constraint and all of
them must be positive. To impose such constraint, a ‘Softplus sign’ function
is used. Hence, the step-size and the parameters associated with the
constraints can all be estimated as
${\gamma}^{[r]}=\mathcal{L}_{\gamma}^{[r}=\text{Softplus}(\mathbf{z}^{[r]})$.
The output of the last three hidden structures is connected to an auxiliary
processing module (APM) to convert it into the required transmit precoding
vector. The APM consists of 4 convolution layers and 3 activation layers, a
“Batch Normalization” layer placed between them. Therefore, the Proximal
Barrier function for a strict phase formulation is summarized in Algorithm 2.

Finally, the output from the auxiliary processing block is the precoding
vector in the real domain. The relation:
$\mathbf{v}_{1}=[\mathbf{v}_{R}\ -\mathbf{v}_{I}]^{T}$ is used to convert it
to its equivalent complex domain for every SINR value of the i-th user.

###  III-C The proposed Learning Structure and the general NN Architecture

Using (22) and Algorithm 1, we show a startling correlation between our scheme
and the universal feed-forward DNN. Generally, an open-chained neural network
(NN) structure can be derived from (22) as follows

$\mathbf{v}_{1}^{[r+1]}=\text{prox}_{\gamma^{[r]}\mu^{[r]}\mathcal{B}}\left[\left(\mathbf{I}_{2N}-2\gamma^{[r]}\right)\mathbf{v}_{1}^{[r]}+\lambda^{[r]}\Upsilon_{i}\Omega\right].$
(28)

By letting $\mathbf{W}_{r}=\mathbf{I}_{2N}-2\gamma^{[r]}$,
$\mathbf{b}_{r}=\lambda^{[r]}\Upsilon_{i}\Omega$ and
$\bm{\Pi}_{r}=\text{prox}_{\gamma^{[r]}\mu^{[r]}\mathcal{B}}$, the r-layer
network $\mathcal{L}^{[r-1]}\cdots\mathcal{L}^{[0]}$ will correspond to the
following

$\bm{\Pi}_{R-1}\left(\mathbf{W}_{R-1}\mathbf{v}_{1}^{[R-1]}+\mathbf{b}_{R-]}\right),\cdots,\
\bm{\Pi}_{0}\left(\mathbf{W}_{0}\mathbf{v}_{1}^{[0]}+\mathbf{b}_{0}\right)\\\
\forall\ 0\leq{r}\leq{R-1}\in\text{R-layers},$ (29)

where $[\mathbf{W}_{r}]_{0\leq{r}\leq{R-1}}$ and
$[\mathbf{b}_{r}]_{0\leq{r}\leq{R-1}}$ are described as weight and bias
parameters respectively. The identity square matrix is defined as
$\mathbf{I}_{2N}\in\mathbb{R}^{2(N\times N)}$. The nonlinear activation
functions are defined by $[\bm{\Pi}_{r}]_{0\leq{r}\leq{R-1}}$ and can be
obtained from the proximal operator. Furthermore, $\bm{\Pi}_{r}$ can be
expressed as sum of a bias and a proximal activation operator.

###  III-D SLP-SDNet Training and Testing

The SLP-SDNet has two modules; the parameter module and the auxiliary module.
The parameter module consists of three structures associated with Lagrangian
multipliers (equality and inequality constraints) and the training step-size.
The proximity barrier function is related to the inequality constraint and
forms the parameter module. It is constructed with one convolutional layer, an
average pooling layer, a fully connected layer, and a softPlus layer so that
the output is constrained to a positive real value. The parameter update
module contains r-th blocks and is trained block-wise for l-th number of
iterations. Similarly, the auxiliary unit is trained for k-th iterations. It
is important to note that the number of training iterations of the parameter
update module may not necessarily be equal to that of the auxiliary unit. We
train the parameter update unit with 15 iterations and the auxiliary unit for
10 iterations. During the inference, a feed-forward pass is performed over the
whole layers using the learned Lagrangian multipliers to calculate the
precoding vector using (26). The trained model is run over different SINR
values to output the optimal precoding matrix.

##  IV Results and Discussion

###  IV-A Simulation Setup

We consider a downlink scenario, where the BS has four antennas ($N=4$) that
serve $K$ single users, assuming a perfect known CSI. We generate 50,000
training samples and 2000 test samples of the channel coefficients randomly
drawn from a normal distribution with zero mean and unit variance using (3).
The datasets are normalized by the data symbol so that data entries are within
the nominal range. The transmit data symbols are modulated using a QPSK and
8PSK modulation schemes; and the SINR is randomly generated from uniform
distribution $\Gamma_{\text{train}}\sim\mathcal{U}(\Gamma_{\text{low}},\
\Gamma_{\text{high}})$. A stochastic gradient descent algorithm with Adam
optimizer is used to minimize the Lagrangian function (loss function). For
every training iteration, the learning rate is reduced by $\beta=0.65$ to help
the learning algorithm converge faster. The implementation is done on Pytorch
1.7.1 and Python 3.7.8 on a computer with the following specifications:
Intel(R) Core (TM) i7-6700 CPU Core, 32.0GB RAM.

Figure 3: Transmit Power vs SINR averaged over 2000 test samples for
Conventional BLP, SLP-St optimization-based and SLP-SDNet schemes, $N=4$,
$K=4$. Figure 4: Transmit Power vs SINR averaged over 2000 test samples for
SLP-St optimization-based and SLP-SDNet schemes for varying number of user;
$N=4$, $K=5$ and $K=6$.

###  IV-B Performance Evaluation of SLP-SDNet

We consider a SLP-SDNet for strict phase angle rotation problems (27). Our
proposed unsupervised learning framework’s performance is evaluated against
the benchmark precoding designs in [12, 3]. We compare the average transmit
power of the conventional BLP approach (1), the SLP-based problems (2), the
proposed SLP learning-based precoding scheme based on Algorithm 2. Fig. 3
shows that the SLP-SDNet gives less transmit power than the conventional BLP
scheme because, for a $4\times 4$ system, there is inadequately available
transmit power at the BS. It is also essential to note that the transmit power
given by an SLP-SDNet is the same as for an SLP optimization-based solution at
$SINR$ values below $30$dB. However, the transmit power increases by 8% for an
SLP-SDNet solution over SLP optimization-based approach at $SINR$ greater than
$30$dB.

Figs. 4(a) and 4(b) show the average transmit power for a given BS antennas
($N=4$) and varying number of users (5 and 6 users). We find from our
simulation that while conventional BLP is only feasible for $K\leq M$, both
SLP optimization-based algorithm and the proposed learning schemes are viable
for all sets of $N$ BS antennas and $K$ mobile users. Furthermore, we also
observe that the performance gap between the SLP optimization-based and
proposed learning-based schemes closes as more users are served.

###  IV-C Complexity Evaluation

For a fair comparison, we measure the complexities of our proposals and the
benchmark optimization-based precoding schemes in terms of the optimization
algorithms’ average execution time, as shown in Figs. 5. We observe that the
average execution time of the SLP-SDNet scheme per symbol averaged over 2000
test samples offers $2\times$ decrease in execution time per data symbol
because the predominant operations in SLP-SDNet during online training are
matrix-matrix or vector-matrix convolution. This shows that the proposed
unsupervised learning-based precoding scheme offers a desirable trade-off
between performance and computational complexity.

Figure 5: Comparison of average execution time per sample averaged over 2000
test samples for Conventional BLP, SLP-St optimization-based and SLP-SDNet
schemes.

##  V Conclusion

This paper proposes a fast unsupervised learning-based precoding framework for
a multi-user downlink MISO system. The proposed learning technique exploits
the constructive interference for the power minimization problem so that for
given QoS constraints, the transmit power available for transmission is
minimized. We use domain knowledge to develop an unsupervised learning
architecture by unfolding the proximal interior point method barrier ‘log
function. Proximal barrier function for strict phase rotation is derived based
on the nature and characteristics of the inequality constraints.

## References

  * [1] C. Masouros and E. Alsusa, “A novel transmitter-based selective-precoding technique for DS/CDMA systems,” in _2007 IEEE International Conference on Communications_. IEEE, 2007, pp. 2829–2834.
  * [2] C. Masouros and E. Alsusa, “Dynamic linear precoding for the exploitation of known interference in MIMO broadcast systems,” _IEEE Transactions on Wireless Communications_ , vol. 8, no. 3, pp. 1396–1404, 2009.
  * [3] C. Masouros and G. Zheng, “Exploiting known interference as green signal power for downlink beamforming optimization,” _IEEE Transactions on Signal processing_ , vol. 63, no. 14, pp. 3628–3640, 2015.
  * [4] M. Alodeh, S. Chatzinotas, and B. Ottersten, “Constructive multiuser interference in symbol level precoding for the MISO downlink channel,” _IEEE Transactions on Signal processing_ , vol. 63, no. 9, pp. 2239–2252, 2015.
  * [5] C. Masouros, “Harvesting signal power from constructive interference in multiuser downlinks,” in _Wireless Information and Power Transfer: A New Paradigm for Green Communications_. Springer, 2018, pp. 87–122.
  * [6] A. Alkhateeb, S. Alex, P. Varkey, Y. Li, Q. Qu, and D. Tujkovic, “Deep learning coordinated beamforming for highly-mobile millimeter wave systems,” _IEEE Access_ , vol. 6, pp. 37 328–37 348, 2018.
  * [7] P. de Kerret and D. Gesbert, “Robust decentralized joint precoding using team deep neural network,” in _2018 15th International Symposium on Wireless Communication Systems (ISWCS)_. IEEE, 2018, pp. 1–5.
  * [8] H. Huang, W. Xia, J. Xiong, J. Yang, G. Zheng, and X. Zhu, “Unsupervised learning-based fast beamforming design for downlink MIMO,” _IEEE Access_ , vol. 7, pp. 7599–7605, 2018.
  * [9] W. Xia, G. Zheng, Y. Zhu, J. Zhang, J. Wang, and A. P. Petropulu, “A deep learning framework for optimization of MISO downlink beamforming,” _IEEE Transactions on Communications_ , vol. 68, no. 3, pp. 1866–1880, 2019\.
  * [10] F. Sohrabi, H. V. Cheng, and W. Yu, “Robust symbol-level precoding via autoencoder-based deep learning,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 8951–8955.
  * [11] Z. Bo, R. Liu, M. Li, and Q. Liu, “Deep learning based efficient symbol-level precoding design for MU-MISO systems,” _IEEE Transactions on Vehicular Technology_ , vol. 70, no. 8, pp. 8309–8313, 2021.
  * [12] E. Björnson, M. Bengtsson, and B. Ottersten, “Optimal multiuser transmit beamforming: A difficult problem with a simple solution structure [lecture notes],” _IEEE Signal Processing Magazine_ , vol. 31, no. 4, pp. 142–148, 2014.
  * [13] C. Masouros, “Correlation rotation linear precoding for MIMO broadcast communications,” _IEEE Transactions on Signal Processing_ , vol. 59, no. 1, pp. 252–262, 2010.
  * [14] C. Masouros, T. Ratnarajah, M. Sellathurai, C. Papadias, and A. Shukla, “Known interference in wireless communications: a limiting factor or a potential source of green signal power?” _IEEE Comms. Mag_ , vol. 51, no. 10, pp. 162–171, 2013.
  * [15] A. Li, D. Spano, J. Krivochiza, S. Domouchtsidis, C. G. Tsinos, C. Masouros, S. Chatzinotas, Y. Li, B. Vucetic, and B. Ottersten, “A tutorial on interference exploitation via symbol-level precoding: Overview, state-of-the-art and future directions,” _IEEE Communications Surveys & Tutorials_, vol. 22, no. 2, pp. 796–839, 2020.
  * [16] C. Bertocchi, E. Chouzenoux, M.-C. Corbineau, J.-C. Pesquet, and M. Prato, “Deep unfolding of a proximal interior point method for image restoration,” _Inverse Problems_ , vol. 36, no. 3, p. 034005, 2020.
  * [17] R. Hauser, “Interior-point methods for inequality constrained optimization,” 2007\.
  * [18] N. Pustelnik and L. Condat, “Proximity operator of a sum of functions; application to depth map estimation,” _IEEE Signal Processing Letters_ , vol. 24, no. 12, pp. 1827–1831, 2017.