Cloned from stepfun-ai/Step1X-3D

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text
+*.jpg filter=lfs diff=lfs merge=lfs -text
+*.jpeg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,95 @@
+---
+license: apache-2.0
+---
+
+
+<h1 align="center"> Step1X-3D: Towards High-Fidelity and Controllable<br>Generation of Textured 3D Assets</h1>
+
+<p align="center">
+  <img src="assets/stepfun_illusions_logo.jpeg" width="100%">
+</p>
+
+<div align="center">
+<img width="" alt="demo" src="assets/step1x-3d-teaser.png">
+</div>
+
+<div align="left">
+<p><b>Step1X-3D demonstrates the capability to generate 3D assets with high-fidelity geometry and versatile texture maps, while maintaining exceptional alignment between surface geometry and texture mapping. From left to right, we sequentially present: the base geometry (untextured), followed by cartoon-style, sketch-style, and photorealistic 3D asset generation results.</b></p>
+</div>
+
+
+## 🔥🔥🔥 Latest News!!
+* May 13, 2025: 👋 Step1X-3D online demo is available on huggingface-enjoy yourself with generated 3D assets! [Huggingface web live](https://huggingface.co/spaces/stepfun-ai/Step1X-3D)
+* May 13, 2025: 👋 We release the 800K uids of high quality 3D assets (excluding self-collected assets) obtained with our rigorous data curation pipeline for both training 3D geometry and synthesis. [Huggingface dataset](https://huggingface.co/datasets/stepfun-ai/Step1X-3D-obj-data/tree/main)
+* May 13, 2025: 👋 We have also release the training code of both Step1X-3D geometry generation and texture synthesis. 
+* May 13, 2025: 👋 We have released the inference code and model weights of Step1X-3D geometry and Step1X-3D texture.
+* May 13, 2025: 👋 We have released Step1X-3D [technical report]() as open source.
+
+## Introduction
+While generative artificial intelligence has advanced significantly across text, image, audio, and video domains, 3D generation remains comparatively underdeveloped due to fundamental challenges such as data scarcity, algorithmic limitations, and ecosystem fragmentation. 
+To this end, we present Step1X-3D, an open framework addressing these challenges through: 
+(1) a rigorous data curation pipeline processing >5M assets to create a 2M high-quality dataset with standardized geometric and textural properties; 
+(2) a two-stage 3D-native architecture combining a hybrid VAE-DiT geometry generator 
+with an SD-XL-based texture synthesis module; and (3) the full open-source release of models, training code, and adaptation modules. For geometry generation, the hybrid VAE-DiT component produces watertight TSDF representations by employing perceiver-based latent encoding with sharp edge sampling for detail preservation. The SD-XL-based texture synthesis module then ensures cross-view consistency through geometric conditioning and latent-space synchronization.
+Benchmark results demonstrate state-of-the-art performance that exceeds existing open-source methods, while also achieving competitive quality with proprietary solutions. 
+Notebly, the framework uniquely bridges 2D and 3D generation paradigms by supporting direct transfer of 2D control techniques~(e.g., LoRA) to 3D synthesis.
+By simultaneously advancing data quality, algorithmic fidelity, and reproducibility, Step1X-3D aims to establish new standards for open research in controllable 3D asset generation.
+<img width="" alt="framework" src="assets/step1x-3d-framework-overall.jpg">
+
+
+## Usage
+
+```python
+# Stage 1: 3D geometry generation
+from step1x3d_geometry.models.pipelines.pipeline import Step1X3DGeometryPipeline
+
+# define the pipeline
+geometry_pipeline = Step1X3DGeometryPipeline.from_pretrained("stepfun-ai/Step1X-3D", subfolder='Step1X-3D-Geometry-1300m'
+).to("cuda")
+
+# input image
+input_image_path = "examples/test.png"
+
+# run pipeline and obtain the untextured mesh 
+generator = torch.Generator(device=geometry_pipeline.device).manual_seed(2025)
+out = geometry_pipeline(input_image_path，guidance_scale=7.5, num_inference_steps=50)
+
+# export untextured mesh as .glb format
+out.mesh[0].export("untexture_mesh.glb")
+
+
+# Stage 2: 3D texure synthsis
+from step1x3d_texture.pipelines.step1x_3d_texture_synthesis_pipeline import (
+    Step1X3DTexturePipeline,
+)
+from step1x3d_geometry.models.pipelines.pipeline_utils import reduce_face, remove_degenerate_face
+import trimesh
+
+# load untextured mesh
+untexture_mesh = trimesh.load("untexture_mesh.glb")
+
+# define texture_pipeline
+texture_pipeline = Step1X3DTexturePipeline.from_pretrained("stepfun-ai/Step1X-3D", subfolder="Step1X-3D-Texture")
+
+# reduce face
+untexture_mesh = remove_degenerate_face(untexture_mesh)
+untexture_mesh = reduce_face(untexture_mesh)
+
+# texture mapping
+textured_mesh = texture_pipeline(input_image_path, untexture_mesh)
+
+# export textured mesh as .glb format
+textured_mesh.export("textured_mesh.glb")
+```
+
+## Citation
+If you find our work helpful, please cite us
+
+```
+@article{li2025step1x3dhighfidelitycontrollablegeneration,
+      title={Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assets}, 
+      author={Weiyu Li and Xuanyang Zhang and Zheng Sun and Di Qi and Hao Li and Wei Cheng and Weiwei Cai and Shihao Wu and Jiarui Liu and Zihao Wang and Xiao Chen and Feipeng Tian and Jianxiong Pan and Zeming Li and Gang Yu and Xiangyu Zhang and Daxin Jiang and Ping Tan},
+      journal={arXiv preprint arxiv:2505.07747}
+      year={2025}
+}
+```

Step1X-3D-Geometry-1300m/model_index.json

@@ -0,0 +1,32 @@
+{
+  "_class_name": "Step1X3DGeometryPipeline",
+  "_diffusers_version": "0.32.2",
+  "caption_encoder": [
+    null,
+    null
+  ],
+  "label_encoder": [
+    null,
+    null
+  ],
+  "scheduler": [
+    "diffusers",
+    "FlowMatchEulerDiscreteScheduler"
+  ],
+  "transformer": [
+    "step1x3d_geometry.models.transformers.flux_transformer_1d",
+    "FluxDenoiser"
+  ],
+  "vae": [
+    "step1x3d_geometry.models.autoencoders.michelangelo_autoencoder",
+    "MichelangeloAutoencoder"
+  ],
+  "visual_eature_extractor": [
+    "transformers",
+    "BitImageProcessor"
+  ],
+  "visual_encoder": [
+    "step1x3d_geometry.models.conditional_encoders.dinov2_encoder",
+    "Dinov2Encoder"
+  ]
+}

Step1X-3D-Geometry-1300m/scheduler/scheduler_config.json

@@ -0,0 +1,16 @@
+{
+  "_class_name": "FlowMatchEulerDiscreteScheduler",
+  "_diffusers_version": "0.32.2",
+  "base_image_seq_len": 256,
+  "base_shift": 0.5,
+  "invert_sigmas": false,
+  "max_image_seq_len": 4096,
+  "max_shift": 1.15,
+  "num_train_timesteps": 1000,
+  "shift": 1.0,
+  "shift_terminal": null,
+  "use_beta_sigmas": false,
+  "use_dynamic_shifting": false,
+  "use_exponential_sigmas": false,
+  "use_karras_sigmas": false
+}

Step1X-3D-Geometry-1300m/transformer/config.json

@@ -0,0 +1,18 @@
+{
+    "weights": null,
+    "input_channels": 64,
+    "width": 1280,
+    "layers": 12,
+    "num_single_layers": 24,
+    "num_heads": 20,
+    "condition_dim": 1024,
+    "multi_condition_type": "in_context",
+    "use_visual_condition": true,
+    "visual_condition_dim": 1024,
+    "n_views": 1,
+    "use_caption_condition": false,
+    "caption_condition_dim": 1024,
+    "use_label_condition": false,
+    "label_condition_dim": 1024,
+    "identity_init": false
+}

Step1X-3D-Geometry-1300m/transformer/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8cfc27716009acc60035f2489a9b61de81790625b70aa27a682efc99a5137a9
+size 5265243816

Step1X-3D-Geometry-1300m/vae/config.json

@@ -0,0 +1,31 @@
+{
+    "weights": null,
+    "n_samples": 16384,
+    "use_downsample": true,
+    "downsample_ratio": 0.0625,
+    "num_latents": 2048,
+    "point_feats": 3,
+    "embed_point_feats": false,
+    "out_dim": 1,
+    "embed_dim": 64,
+    "embed_type": "fourier",
+    "num_freqs": 8,
+    "include_pi": false,
+    "width": 768,
+    "heads": 12,
+    "num_encoder_layers": 8,
+    "num_decoder_layers": 16,
+    "init_scale": 0.009021097956087902,
+    "qkv_bias": false,
+    "qk_norm": false,
+    "use_ln_post": true,
+    "use_flash": true,
+    "use_checkpoint": true,
+    "use_multi_reso": false,
+    "resolutions": null,
+    "sampling_prob": null,
+    "with_sharp_data": true,
+    "volume_decoder_type": "hierarchical",
+    "surface_extractor_type": "mc",
+    "z_scale_factor": 1.0
+}

Step1X-3D-Geometry-1300m/vae/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa1b17360f6614a54ab3b971c966cfe64ff87a5445186b02c666e634b59c8302
+size 766401652

Step1X-3D-Geometry-1300m/visual_eature_extractor/preprocessor_config.json

@@ -0,0 +1,27 @@
+{
+  "crop_size": {
+    "height": 518,
+    "width": 518
+  },
+  "do_center_crop": true,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "BitImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "shortest_edge": 518
+  }
+}

Step1X-3D-Geometry-1300m/visual_encoder/config.json

@@ -0,0 +1,15 @@
+{
+    "weights": null,
+    "encode_camera": false,
+    "camera_embeds_type": "sincos",
+    "camera_embeds_dim": null,
+    "n_views": 1,
+    "empty_embeds_ratio": 0.1,
+    "normalize_embeds": false,
+    "zero_uncond_embeds": true,
+    "freeze_modulation_dino": false,
+    "enable_gradient_checkpointing": false,
+    "image_size": 518,
+    "dino_type": "facebook/dinov2-with-registers-large",
+    "kwargs": null
+}

Step1X-3D-Geometry-1300m/visual_encoder/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c07f8abe3af5db214e3341c9f23fb2351cb15591afdae0da8e835dfabc42e850
+size 1217544216

Step1X-3D-Geometry-Label-1300m/label_encoder/config.json

@@ -0,0 +1,6 @@
+{
+    "hidden_size": 1024,
+    "empty_embeds_ratio": 0.1,
+    "normalize_embeds": false,
+    "zero_uncond_embeds": true
+}

Step1X-3D-Geometry-Label-1300m/label_encoder/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d7d76cfe4bc95f264421fa34076826bada5a218e63febabe69461d8c2607e4b
+size 33104

Step1X-3D-Geometry-Label-1300m/model_index.json

@@ -0,0 +1,32 @@
+{
+  "_class_name": "Step1X3DGeometryPipeline",
+  "_diffusers_version": "0.32.2",
+  "caption_encoder": [
+    null,
+    null
+  ],
+  "label_encoder": [
+    "step1x3d_geometry.models.conditional_encoders.label_encoder",
+    "LabelEncoder"
+  ],
+  "scheduler": [
+    "diffusers",
+    "FlowMatchEulerDiscreteScheduler"
+  ],
+  "transformer": [
+    "step1x3d_geometry.models.transformers.flux_transformer_1d",
+    "FluxDenoiser"
+  ],
+  "vae": [
+    "step1x3d_geometry.models.autoencoders.michelangelo_autoencoder",
+    "MichelangeloAutoencoder"
+  ],
+  "visual_eature_extractor": [
+    "transformers",
+    "BitImageProcessor"
+  ],
+  "visual_encoder": [
+    "step1x3d_geometry.models.conditional_encoders.dinov2_clip_encoder",
+    "Dinov2CLIPEncoder"
+  ]
+}

Step1X-3D-Geometry-Label-1300m/scheduler/scheduler_config.json

@@ -0,0 +1,16 @@
+{
+  "_class_name": "FlowMatchEulerDiscreteScheduler",
+  "_diffusers_version": "0.32.2",
+  "base_image_seq_len": 256,
+  "base_shift": 0.5,
+  "invert_sigmas": false,
+  "max_image_seq_len": 4096,
+  "max_shift": 1.15,
+  "num_train_timesteps": 1000,
+  "shift": 1.0,
+  "shift_terminal": null,
+  "use_beta_sigmas": false,
+  "use_dynamic_shifting": false,
+  "use_exponential_sigmas": false,
+  "use_karras_sigmas": false
+}

Step1X-3D-Geometry-Label-1300m/transformer/config.json

@@ -0,0 +1,17 @@
+{
+    "input_channels": 64,
+    "width": 1536,
+    "layers": 8,
+    "num_single_layers": 16,
+    "num_heads": 16,
+    "condition_dim": 1024,
+    "multi_condition_type": "in_context",
+    "use_visual_condition": true,
+    "visual_condition_dim": 1024,
+    "n_views": 1,
+    "use_caption_condition": false,
+    "caption_condition_dim": 1024,
+    "use_label_condition": true,
+    "label_condition_dim": 1024,
+    "identity_init": false
+}

Step1X-3D-Geometry-Label-1300m/transformer/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0df096288471320f3fd566e94db628c3a36a697ae6fa55e60152b236fd5918aa
+size 5086980952

Step1X-3D-Geometry-Label-1300m/vae/config.json

@@ -0,0 +1,30 @@
+{
+    "n_samples": 16384,
+    "use_downsample": true,
+    "downsample_ratio": 0.0625,
+    "num_latents": 2048,
+    "point_feats": 3,
+    "embed_point_feats": false,
+    "out_dim": 1,
+    "embed_dim": 64,
+    "embed_type": "fourier",
+    "num_freqs": 8,
+    "include_pi": false,
+    "width": 768,
+    "heads": 12,
+    "num_encoder_layers": 8,
+    "num_decoder_layers": 16,
+    "init_scale": 0.009021097956087902,
+    "qkv_bias": false,
+    "qk_norm": false,
+    "use_ln_post": true,
+    "use_flash": true,
+    "use_checkpoint": true,
+    "use_multi_reso": false,
+    "resolutions": null,
+    "sampling_prob": null,
+    "with_sharp_data": true,
+    "volume_decoder_type": "hierarchical",
+    "surface_extractor_type": "mc",
+    "z_scale_factor": 1.0
+}

Step1X-3D-Geometry-Label-1300m/vae/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9d764b4e3fdf377f1ba960215f2e91800fa6bf86ec5243340b96e22a0a7f781
+size 766401652

Step1X-3D-Geometry-Label-1300m/visual_eature_extractor/preprocessor_config.json

@@ -0,0 +1,27 @@
+{
+  "crop_size": {
+    "height": 518,
+    "width": 518
+  },
+  "do_center_crop": true,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "BitImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "shortest_edge": 518
+  }
+}

Step1X-3D-Geometry-Label-1300m/visual_encoder/config.json

@@ -0,0 +1,17 @@
+{
+    "encode_camera": false,
+    "camera_embeds_type": "sincos",
+    "camera_embeds_dim": null,
+    "n_views": 1,
+    "empty_embeds_ratio": 0.1,
+    "normalize_embeds": false,
+    "zero_uncond_embeds": true,
+    "freeze_modulation_clip": false,
+    "freeze_modulation_dino": false,
+    "enable_gradient_checkpointing": false,
+    "image_size": 518,
+    "fuse_type": "concat",
+    "dino_type": "facebook/dinov2-with-registers-large",
+    "clip_type": "openai/clip-vit-large-patch14",
+    "kwargs": null
+}

Step1X-3D-Geometry-Label-1300m/visual_encoder/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9b22f50296f1f63841c92a724c732bffee8051b59e830e9eac9a99af8d0e65e
+size 2928089212

Step1X-3D-Texture/step1x-3d-ig2v.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05cd28096e875712c1f2489bb325a9bc7324a41cd3f367d7a38c63f1bea77964
+size 3602537816