# ConTEXTure: Consistent Multiview Images to Texture

Jaehoon Ahn  Sogang UniversitySouth Korea<EMAIL_ADDRESS>,  Sumin Cho
Sogang UniversitySouth Korea<EMAIL_ADDRESS>,  Harim Jung  SwatchOn Co.,
LtdSouth Korea<EMAIL_ADDRESS>,  Kibeom Hong  Sookmyung Women’s
UniversitySouth Korea<EMAIL_ADDRESS>,  Seonghoon Ban  RECON Labs
Inc.South Korea<EMAIL_ADDRESS>and  Moon-Ryul Jung  Sogang
UniversitySouth Korea<EMAIL_ADDRESS>
###### Abstract.

We introduce ConTEXTure, a generative network designed to create a texture
map/atlas for a given 3D mesh using images from multiple viewpoints. The
process begins with generating a front-view image from a text prompt, such as
’Napoleon, front view’, describing the 3D mesh. Additional images from
different viewpoints are derived from this front-view image and camera poses
relative to it. ConTEXTure builds upon the TEXTure network, which uses text
prompts for six viewpoints (e.g., ’Napoleon, front view’, ’Napoleon, left
view’, etc.). However, TEXTure often generates images for non-front viewpoints
that do not accurately represent those viewpoints.To address this issue, we
employ Zero123++, which generates multiple view-consistent images for the six
specified viewpoints simultaneously, conditioned on the initial front-view
image and the depth maps of the mesh for the six viewpoints. By utilizing
these view-consistent images, ConTEXTure learns the texture atlas from all
viewpoint images concurrently, unlike previous methods that do so
sequentially. This approach ensures that the rendered images from various
viewpoints, including back, side, bottom, and top, are free from viewpoint
irregularities.

[“white humanoid robot, movie poster,
villain character of a science fiction movie”]  [“doctor in a lab coat with a
simple, modest hijab”]

[“person in red sweater, blue jeans”]  [“rock star with leather jacket”]

Figure 1. Four examples of texture maps generated using the ConTEXTure model,
being worn by its mesh. Each model is shown from three equidistant azimuth
angles of 0°, 120°, and -120°, offset from the front. The prompts used to
generate each texture are written directly beneath the images.

##  1\. Introduction

Recent advancements in 3D modeling and texture synthesis (Richardson et al.,
2023; Cao et al., 2023; Chen et al., 2023a; Metzer et al., 2023; Liu et al.,
2023b; Ceylan et al., 2024; Tang et al., 2024; Wu et al., 2024; Tang and He,
2023; Youwang et al., 2024; Wang et al., 2023; Gao et al., 2024) have led to
significant improvements in generating custom textures for any provided input
mesh. TEXTure (Richardson et al., 2023), a pioneering model in this domain,
utilizes text prompts to generate texture atlases using the Stable Diffusion
(Rombach et al., 2022) depth pipeline, which leverages depth maps of the mesh
rendered from diverse viewpoints. However, the images generated for different
viewpoints often lack ”consistency across all views.”

Consistency across all views means that the images generated for each
viewpoint (front, left, right, back, top, and bottom) is in harmony with each
other. Each image accurately reflects the 3D mesh from its specific
perspective without any discrepancies or distortions. This ensures that the
texture map learned from the multiple images appears uniform and correct from
every angle.

The problem that the generated multi-view images lack view-consistency may be
caused by several factors. The main one seems that most images in the dataset
used to train the depth pipeline are taken from the front viewpoint. This
issue is referred to by (Liu et al., 2023a) as the _viewpoint bias_ problem,
demonstrating how both Dall-E 2 (Ramesh et al., 2022) and Stable Diffusion,
when provided the prompt “ _a chair_ ”, consistently generate chair images in
a forward-facing canonical pose. Even when augmenting the text prompts with
directional information (e.g., “{front, back, left, right, overhead, bottom}
view”), the problem generally continues to persist.

(a) Front

(b) Left

(c) Back

(d) Right

Generated texture using TEXTure (Richardson et al., 2023)

(e) Front

(f) Left

(g) Back

(h) Right

Generated texture using our model Figure 2. Texture generated using the
original TEXTure model with the prompt “A photo of Spongebob, {} view” and a
guidance scale of 10. Note that the unique geometry of the SpongeBob character
causes faces to be erroneously generated on each side of the mesh. This
problem was resolved in our model. Figure 3. A visual representation of the
ConTEXTure model’s generation process. A latent image blending process is
performed in our custom implementation of Zero123++ which is shown in more
detail in Figure 4. Refer to Algorithm 1 for more information on the overall
texture generation process.

In our investigation of texture map generation, it was found that the problem
of view-consistency is generally caused by two factors. First, the less
detailed polygon representation of the mesh causes the view-consistency
problem. For example, the case of the mesh of a Napoleon Bonaparte bust, it
has a high level of polygonal detail. It plays a role in facilitating the
depth pipeline’s performance by providing a rich set of cues for image
generation from the text prompt. Conversely, cartoon-style meshes with fewer
details present ambiguity in depth maps, particularly for identifying distinct
features like the front or back of a character. Despite appending directional
cues to the text prompt, the lack of detailed geometry can cause the
generation of images which lack the consistency among views.

Second, the view-consistency problem stems from inherent viewpoint biases
within the Stable Diffusion dataset. As seen in Figure 2, this is particularly
evident in generating textures for characters like SpongeBob from non-standard
viewpoints. SpongeBob is selected as an exemplar due to his distinct geometry,
where his front and back sides become invisible when viewed from a
perpendicular 90-degree angle. This unique characteristic accentuates the
challenges in depth-based image generation, diverging significantly from
typical human or object meshes, thereby providing a stringent test case for
assessing the robustness of our texture synthesis approach in handling
atypical shapes and viewpoints. Given the model’s training on diverse
internet-scale datasets, it has a predisposition towards generating
recognizable features, such as faces, even when provided with depth maps from
unconventional viewpoints. This bias leads to the erroneous placement of these
features across multiple sides of a mesh, highlighting the limitations of
relying solely on text prompts and standard depth cues in guiding the texture
synthesis process for characters with iconic and highly recognizable features.

To address this problem, we introduce the ConTEXTure model, which adapts the
TEXTure model to use the Stable Diffusion depth pipeline (hereinafter referred
to as SD2-depth) alongside the version of Zero123++ (Shi et al., 2023) fine-
tuned with ControlNet (Zhang et al., 2023) to support depth-controlled novel
view image generation. Depth maps play a crucial role in texture atlas
generation, guiding the texture synthesis to ensure alignment with the mesh’s
geometry. SD2-depth is used to generate just the front view of the object.
Then we learn the texture atlas from the front view image, and renders the
mesh using the partially learned texture atlas from six viewpoints, producing
six viewpoint images. The front view image is reused as a condition image to
generate all remaining viewpoints using the depth-controlled Zero123++. The
six render images of the mesh are also used as a guide to the process of
generating the six viewpoint images by the depth-controlled Zero123++.

Zero123++ improves upon its predecessor models Zero 1-to-3 (Liu et al., 2023a)
and Zero123-XL (Deitke et al., 2024) by simultaneously generating six diverse
viewpoint images in one denoising loop as opposed to one viewpoint at a time,
significantly speeding up the texture map generation process. This
simultaneous generation of viewpoint images led us implement an inverse
renderer that learns the texture atlas from the multi-view images as the
target images simultaneously. It is a departure from the approach used in
(Richardson et al., 2023) which performs inverse rendering one viewpoint image
at a time. This approach initializes the texture map image as learnable
parameters, adjusting them by reducing the loss between the given seven target
viewpoint images (including the front viewpoint image) and the seven viewpoint
images rendered from the textured mesh rendered with the Kaolin
(Jatavallabhula et al., 2019) library. Several modifications had to be made to
the loss equation to support simultaneous multi-view inverse rendering. This
new modified approach is a natural one given the novel architecture of
Zero123++.

##  2\. Related works

###  2.1. Diffusion models

Although GANs historically held state-of-the-art performance for image
generation tasks (Brock et al., 2018; Karras et al., 2020), diffusion models
have recently proven to be a strong, superior alternative (Dhariwal and
Nichol, 2021) and have since overtaken GANs in popularity. Widely used image
generation models like Stable Diffusion (Rombach et al., 2022) and DALL-E
(Ramesh et al., 2022; Betker et al., 2023) are based on the diffusion model
architecture. Stable Diffusion is popular due to it being open source and
relatively easy to fine-tune with various types of conditions. Most
applications of image generation can be classified as text-to-image, but other
conditions can be used in conjunction with text prompts. Stable Diffusion
additionally provides a depth-to-image pipeline. The depth map condition of
the pipeline can be created from an image by means of MiDaS (Ranftl et al.,
2020, 2021) or rendered using a given mesh from a given viewpoint.

Although Stable Diffusion can be fine-tuned directly, it is a known issue
where fine-tuning the diffusion model with a small dataset can destroy its
performance. This problem was resolved with the release of ControlNet (Zhang
et al., 2023). This allows for fine-tuning to take place in the copied
backbone while the weights on the original model are preserved.

###  2.2. Texture generation models

The development of texture generation models has evolved significantly, with
diffusion models leveraging large-scale data marking a notable leap forward.
Early models like AUV-Net (Chen et al., 2022) and Texturify (Siddiqui et al.,
2022) facilitated 2D texture generation on 3D meshes by learning aligned UV
spaces and training 3D StyleGANs. These models effectively addressed
challenges in achieving global coherence and detailed texture representation.

PS-NeRF (Yang et al., 2022) estimates the geometry, PBR maps, lights from
multiple viewpoint images of a non-Lambertian object under multiple unknown
directional lights. From the learned volume-based radiance field, PS-NeRF
constructs the surface normals, spatially-varying BRDFs, and lights by means
of a shadow-aware differentiable rendering layer. While sophisticated PS-NeRF
requires multiple view-consistent images, which are not easy to acquire. We
aim at a simpler method of learning a texture atlas from a given mesh and a
text prompt.

Diffusion models have ushered in a new era in texture generation. DreamFusion
(Poole et al., 2022) and Magic3D (Lin et al., 2023) leveraged Score
Distillation Sampling (SDS) to optimize 3D representations using large-scale
2D text-to-image diffusion models, producing high-quality textures guided by
text prompts. However, these models required lengthy optimization processes
and faced limitations in color saturation and diversity due to high guidance
weights (Ho and Salimans, 2022).

The TEXTure model (Richardson et al., 2023) revolutionized texture synthesis
by employing depth-conditioned diffusion models to iteratively paint 3D
surfaces from different viewpoints, significantly enhancing efficiency and
texture quality. This method enabled the rapid generation of high-quality
textures without extensive optimization,offering substantial speed
improvements.

Text2Tex (Chen et al., 2023b) resembles TEXTure in many ways, synthesizing
partial textures from multiple viewpoint images. It introduces an automatic
view sequence generation method to determine the next best view for updating
the texture atlas.

TexFusion (Cao et al., 2023) proposes constructing the texture atlas in the
latent space while multi-view images are being denoised. At each denoising
step, each viewpoint image is sequentially denoised, and the noisy version of
the texture atlas is learned from the currently denoised viewpoint images.
This contrasts with previous approaches, which learn the texture atlas from a
set of clean images corresponding to multiple viewpoints. It leads to more
smoothed transition between the regions of the texture atlas corresponding to
different view images.

However, these models continue to suffer from viewpoint bias due to the use of
diffusion models trained on disproportionate amounts of images with front-
facing directions. TexRO (Wu et al., 2024) is similar to the model created in
our research in that it also utilizes Zero123++ (Shi et al., 2023), a model
used for novel view synthesis, to generate the texture for a given mesh.
Although the viewpoint bias problem was not explicitly mentioned in the paper,
its usage of Zero123++ would lead to the problem being resolved here. However,
the inverse rendering in TexRO is not simultaneous and uses an effective but
counter-intuitive recursive optimization approach to increase details in the
resulting texture atlas.

###  2.3. Novel view synthesis

Here we review techniques of synthesis of multiple view images which are the
backbone of the construction of texture atlas. Recent advancements in novel
view synthesis have significantly contributed to the field of 3D
reconstruction and image generation. Among these, the development of the Zero
1-to-3 model (Liu et al., 2023a) was specifically designed to address the
”viewpoint bias” problem commonly encountered in text-to-image models, such as
Stable Diffusion (Rombach et al., 2022) and DALL-E 2 (Ramesh et al., 2022;
Betker et al., 2023), where generated images often display objects in
canonical poses regardless of the specified viewpoint.

Prior to Zero 1-to-3, novel view synthesis primarily leveraged neural radiance
fields (NeRFs) (Mildenhall et al., [n. d.]) for rendering specific scenes from
novel viewpoints. NeRF-based methods, including DreamFields (Jain et al.,
2022), utilized a neural representation that required multiple views of an
individual scene for accurate synthesis, limiting their application to pre-
defined scenes. These methods demonstrated high fidelity in view synthesis but
were constrained by the need for extensive 3D annotations and specific scene
data.

The introduction of Zero 1-to-3 marked a significant shift by enabling zero-
shot novel view synthesis and 3D shape reconstruction from a single RGB image.
This model utilizes large diffusion models trained on extensive 2D image
datasets, allowing it to generalize to unseen images and achieve state-of-the-
art results without the need for 3D supervision. The model’s ability to fine-
tune on synthetic datasets and manipulate camera viewpoints through learned
geometric priors has set a new benchmark in the field.

Building on the success of Zero 1-to-3, subsequent models like Zero123-XL
(Deitke et al., 2024) and Zero123++ (Shi et al., 2023) have further enhanced
the capabilities of novel view synthesis. Zero123-XL leverages the extensive
Objaverse-XL dataset (Deitke et al., 2024), comprising over 10 million 3D
objects, to improve generalization and performance across a wide range of
modalities, including photorealistic assets, cartoons, and sketches. Zero123++
integrates depth control through a Depth ControlNet (Zhang et al., 2023),
enhancing its ability to handle complex geometric variations and achieve
superior image quality metrics such as LPIPS (Zhang et al., 2018). This model
not only improves on the synthesis quality but also extends the application to
more diverse and challenging datasets, showcasing the robustness and
adaptability of modern novel view synthesis techniques.

Figure 4. Our usage of Zero123++ features a custom diffusion process that was
inspired by the implementation of (Richardson et al., 2023). We perform
blending of the latent image $z_{grid,t-1}$ with noisy ground truth latent
$z_{Q_{grid,t-1}}$ after each denoising iteration $t-1$. Performing blending
according to the blending mask $m_{grid}$ ensures that the front view image
that was already projected back unto the texture atlas is not regenerated by
the Zero123++ model.

##  3\. Method

Algorithm 1 ConTEXTure Algorithm

1:Input: Text prompt $text$, Mesh $M$, camera viewpoints
$\\{v_{0},\ldots,v_{6}\\}$ where $v_{0}$ is the front viewpoint

2:Learn the $max\\_z\\_normals$ meta-texture map

3:$D_{0}$ $\leftarrow$ Render depth map of $M$ at $v_{0}$

4:$z_{0,N_{sd2}}\leftarrow\mathcal{N}(0,I)$

5:for $t=N_{sd2}$ to $0$ do

6: $z\leftarrow\mathcal{N}(0,I)$

7: $z_{0,t-1}\leftarrow\text{TEXTure\\_denoise}(z_{0,t},text,D_{0},t,z)$

8:end for

9:$Q_{0}\leftarrow x_{0,0}\leftarrow\text{SD2\\_VAE\\_decode}(z_{0,0})$

10:$T\leftarrow$ Project-back $Q_{0}$ to learn $T$ for $Q_{0}$

11:for $i=1\ldots 6$ do

12: $D_{i}\leftarrow\text{Render depth map of $M$ at $v_{i}$}$

13: $m_{i}\leftarrow$ Render object mask of $M$ at $v_{i}$ where the region of
$M$ projected by the learned part of $T$ is set to 0

14: $Q_{i}\leftarrow\text{Render image of $M$ wearing $T$ at $v_{i}$}$

15:end for

16:$D_{grid}\leftarrow\text{Create 2x3 grid image from $D_{1}\ldots D_{6}$}$

17:$m_{grid}\leftarrow\text{Create 2x3 grid image from $m_{1}\ldots m_{6}$}$

18:$Q_{grid}\leftarrow\text{Create 2x3 grid image from $Q_{1}\ldots Q_{6}$}$

19:$z_{Q_{grid},0}\leftarrow\text{Zero123plus\\_VAE\\_encode($Q_{grid}$)}$

20:$z_{grid,N_{zero123}}\leftarrow\mathcal{N}(0,I)$

21:for $t=N_{zero123}$ … $0$ do

22:
$z_{grid,t-1}\leftarrow\text{Zero123plus\\_denoise}(z_{grid,t},Q_{0},D_{grid},t-1)$

23: $z_{Q_{grid},t-1}\leftarrow\text{Add the noise of $t-1$ to
$z_{Q_{grid},0}$}$

24: $z_{grid,t-1}\leftarrow z_{grid,t-1}\cdot
m_{grid}+z_{Q_{grid},t-1}\cdot(1-m_{grid})$

25:end for

26:$x_{grid,0}\leftarrow\text{Zero123plus\\_VAE\\_decode}(z_{grid,0})$

27:$x_{1,0},x_{2,0},x_{3,0},x_{4,0},x_{5,0},x_{6,0}\leftarrow\text{Split
$x_{grid,0}$}$

28:$T\leftarrow$ Project-back all view images $x_{i,0},i=0\ldots 6,$ to learn
the texture map thereof, using $max\\_z\\_normals$ as the weights of each view
image to contribute to the texture map

###  3.1. Depth-based diverse view image generation

(a) Without blending with the ground truth render image

(b) With blending with the ground truth render image

Figure 5. There are slight misalignments among the novel view images by
Zero123++ which result in a low quality texture when projected onto the mesh
without further postprocessing. The blending technique, used in (Richardson et
al., 2023) for preventing already-generated regions of the image from being
overwritten, continues to prove effective in ConTEXTure. Prompt of “A photo of
Spiderman, front view” was used for both textures.

The original TEXTure model is designed to generate diverse view images
sequentially, using SD2-depth. This model generates a novel view image from
the depth map of the mesh, given a specific viewpoint and a text prompt that
includes the relevant view descriptor (e.g., ”Spiderman, left view”).
Initially, TEXTure ”projects back” the front view image to the texture atlas.
This process involves learning the texture atlas as trainable parameters of
the neural network from the front view image by minimizing the loss between
the target front view image and the image rendered using the current texture
atlas.

Subsequently, the next view image is generated using the depth pipeline. The
texture atlas is further refined by minimizing the loss between this new view
image and the image rendered from the updated texture atlas. At this stage,
the region of the texture atlas is re-learned only if the region of the mesh
onto which it is mapped is better viewed from the current viewpoint compared
to the previous one. This condition functions as a view-weight over the mesh
region, which may be observed from multiple viewpoints. The better a mesh
region is viewed, the more significantly it contributes to the texture atlas.

TEXTure further refines the latent image for a given viewpoint, obtained
through the depth pipeline, by incorporating the ground truth image of the
mesh rendered from that specific viewpoint. Specifically, the latent image
being denoised is blended with the ground truth rendered image as the
background, similar to the in-painting pipeline where the image being denoised
is blended with a background image. This approach ensures consistency between
the generated and rendered images by preventing the denoising process from
overwriting the valid parts of the texture atlas already learned from the
prior viewpoint. The blending process utilizes the _keep_ , _refine_ , and
_generate_ masks, which are created by comparing the z-normals (the
z-coordinates of the faces projected onto the screen in the current camera)
with a cache of the z-normals that holds the maximum z-normals from the
previously considered viewpoints.

Since Zero123++ requires a condition image to generate six viewpoints, we
decided to keep using SD2-depth to generate an image from the first viewpoint
$v_{0}$, which we always assume to be the front view. This generated frontal
image $Q_{0}$ is then used to create six novel views of the same object using
Zero123++. We employed the _Depth ControlNet for Zero123++_ to control the
generated image $Q_{0}$ with six depth maps $D_{1}\dots D_{6}$) corresponding
to the mesh from the six viewpoints $v_{1}\dots v_{6}$. The text prompt
$T_{front}$ used for SD2-depth has directional information (e.g.. “A picture
of Napoleon Bonaparte, front view”) concatenated to the end of the original
prompt $T$. Zero123++ uses FlexDiffuse (Speed, 2022) for utilizing global
image conditioning in order to minimize the amount of fine-tuning required for
Zero123++ to obtain the desired results. Although we tried using $T$ as the
prompt when using Zero123++ in an attempt to improve prompt fidelity, there
was a noticeable decrease in quality. We assume that this can be attributed to
the fact that public versions of Zero123++ were not trained with the text
prompt when using FlexDiffuse.

In ConTEXTure, the process of learning the texture atlas occurs in two steps.
First, the texture atlas is partially learned from the front view image, and
the ground truth render images are produced using this partially learned
texture atlas. Second, the remaining part of the texture atlas is learned
using the seven viewpoint images, including the front view image. Confer lines
10 and 28 of Algorithm 1. From the perspective of the texture itself, learning
the texture atlas from just $Q_{0}$ is redundant due to the subsequent
learning of the entire texture atlas using $Q_{0}\dots Q_{6}$. However, we
found that Zero123++ has the tendency to produce view images that suffer from
misalignment problems, leading to a low quality texture seen in Figure 5(a).
To resolve this problem, we borrowed the blending technique from TEXTure
(Richardson et al., 2023) to prevent modification of the area corresponding to
the front of the mesh. The texture map learned from $Q_{0}$ is used as the
image to blend with the image being denoised by the Zero123++ pipeline. This
blending process, which can be seen in Figure 4, forces the pipeline to leave
the frontal region unchanged.

$\alpha=1$

$\alpha=10$

$\alpha=100$

$\alpha=\infty$

Figure 6. A comparison between different values of $\alpha$. The first row
shows a limited region of the generated texture map. The second row shows a
close-up shot of the model when looked from the back. The version with
$\alpha=\infty$ corresponds to the version using the binary mask. The mesh and
texture shown here is the same as the one shown in Figure 5(b).

###  3.2. Simultaneous inverse rendering from multiple view images

TEXTure and TexFusion are designed to generate images for multiple viewpoints
one at a time. In contrast, the unique aspect of Zero123++ lies in its
simultaneous generation of view images. This highlights the need for a new and
more efficient approach for simultaneous view inverse rendering. However,
performing inverse rendering without considering overlapping regions results
in unnatural overlaps of inconsistent images in these areas. An obvious
solution involves creating a binary mask for each viewpoint, which holds the
maximum z-normals of the faces among all viewpoints. If the z-normals of a
face in a given viewpoint are the same as the maximum z-normals for that face,
then the face (and the corresponding pixel locations) is assigned a value of
1; otherwise, it is assigned 0. Confer Appendix A for more information on the
algorithm used to create these binary masks.

However, these binary view masks lead to the problem of seams in the
overlapping regions of the texture atlas, requiring the edges to be smoothed
to hide the resulting artifacts. To smooth these masks, we learn the meta-
texture $N_{i}$, which holds the maximum z-normals among all viewpoints (line
2 of Algorithm 1). When the values of $n_{i}$ (the z-normals at face $i$) are
close to $N_{i}$, it indicates that these regions are better viewed by the
camera and that their images are more valuable for contributing to the texture
map. The meta-texture map $N_{i}$ is learned so that when projected onto the
screen from each viewpoint, the projected $N_{i}$ is never less than $n_{i}$.
The learning of $N_{i}$ using just the mesh and camera information, allowing
it to be pre-computed before the generation of multi-view images and the
learning of the texture atlas. Equation 1 is the loss used to train
$N_{i_{p}}$, for every pixel $p$ among all viewpoints $i$.

(1) $\mathcal{L}=\sum_{i}\sum_{p}\text{ReLU}\left(n_{i,p}-N_{i,p}\right)$

After training of $N_{i}$, it is used to calculate view weights $W_{i}$ to be
used during the learning of the texture map. The calculation of $W_{i}$ can be
seen in Equation 2.

(2) $W_{i}=\exp(-\alpha*|N_{i}-n_{i}|)$

$\alpha$ is a hyperparameter that determines the roughness of the amount of
smoothing to be applied to the texture map. As can be seen in Figure 6, the
resulting texture is smooth when $\alpha$ is set to be a low number. However,
there is a problem where neon-colored pixels appear within the regions that
have been projected back onto from more than one view image, and this issue is
exacerbated for small values of $\alpha$. Multiple experiments led us to the
conclusion that setting this value to around 10 leads to a fair balance
between maximizing texture smoothness and minimizing the problem of neon-
colored pixels. As $\alpha$ reaches $\infty$, the more similar the result
should be to the binary masks.

##  4\. Experiments

###  4.1. Qualitative results

We have not once been able to reproduce the viewpoint bias problem in
constructing the texture atlas when using ConTEXTure. However, there remains
several remaining issues. The first issue is the fact that some pixels of the
texture map result in minor color distortions. This problem arises when a
pixel on the texture map is contributed by the pixels of more than one view
images. In a sense the multiple contributions to the texture map from multiple
view images fight against each other, leading to unsatisfactory compromise. It
means that the use of view-weights based on the difference between the
z-normals of each viewpoint and the max z-normals meta-texture map is not
completely satisfactory. A more sophisticated view-weights are needed. Note
that this problem disappears completely when we use the binary view-masks to
blend the contributions to the texture atlas from multiple view images. But it
introduces the problem of seams between regions of the texture atlas.

Another problem we encountered is that lighting information, such as shadows
and lights, is _baked into_ the generated image. The reason for this is
because Zero123++ was trained using rendered images of Objaverse models in an
environment with random HDRI lighting (Shi et al., 2023). This issue can be
exacerbated when the generated front view image $Q_{0}$ has more bright or
shadowy features than usual.

####  4.1.1. Runtime

In our environment, TEXTure takes 2 minutes and 54 seconds from start to
finish, compared to 27 seconds when running ConTEXTure. Our model also
surpasses TexFusion in terms of speed, which was reported to have a runtime of
2.2 minutes (approx. 2 minutes and 12 seconds) The biggest reason behind the
speed difference is due to the fact that Zero123++ is used to generate six
novel views in a single denoising process. This allows us to run just two
denoising processes (once with SD2-depth and once with Zero123++), compared to
ten SD2-depth denoising processes in TEXTure and eight in TexFusion. Although
we are not able to run TexFusion in our runtime speed experiment due to the
model being closed source, we used the same NVIDIA RTX A6000 GPU used by the
author and also experienced an equivalent runtime speed when running TEXTure
on our machine.

###  4.2. Quantitative results

####  4.2.1. FID score

(a) Front

(b) Back

Figure 7. When using the front- and back-side depth maps on the _person_ mesh using the prompt _“white humanoid robot, movie poster, villain character of a science fiction movie,”_ the viewpoint bias issue manifests in the generation of eyes on both the front and back side of the head. Criteria |  | Model
---|---|---
TEXTure | ConTEXTure
User study | Overall quality | 57.5% | 42.5%
Viewpoint consistency | 47.5% | 52.5%
Prompt fidelity | 50.0% | 50.0%
FID |  | 108.5 | 97.38
Runtime |  | 2 min 54 sec | 0 min 27 sec
Table 1. Comparison of TEXTure and ConTEXTure models. The user study was
conducted with 30 participants who were asked to compare 40 pairs of results
from both models.

It is an inherently difficult task to evaluate the quality of a mesh-wearable
texture map, which is perhaps why the original TEXTure model does not include
quantitative results other than a user survey (Richardson et al., 2023). The
TexFusion model calculated the Fréchet Inception Distance (FID) (Heusel et
al., 2017) to compare eight images of a proxy ground truth set with eight
rendered images of the texture when worn by the mesh (Cao et al., 2023). This
proxy ground set consists of eight images generated by using eight
corresponding depth maps of the mesh with SD2-depth. However, images generated
in this manner also suffer from viewpoint bias (see Figure 7), demonstrating
that using the same evaluation technique is not suitable in measuring how well
we resolved the viewpoint bias problem. To circumvent this flaw in the proxy
set, we used SD2-depth to generate an image from just the frontal viewpoint.
We used this image with six depth maps extracted from non-frontal viewpoints
as the condition image for Zero123++. The generated six view images were
grouped together with the initial front view image and saved as seven separate
image files. We then compared these images with the rendered images of the
mesh wearing the generated texture map from the same viewpoints. TexFusion
reported high FID scores and that the cause of this is due to the small size
of the proxy ground truth set (Cao et al., 2023). Our calculated FID scores
shown in Table 1 are also considerably high as the numbers reported in
TexFusion.

####  4.2.2. User study

Along with calculation of the FID score, we conducted a survey on Google Forms
in which respondents are to look at and compare images for two textured meshes
per section. One texture is generated with (Richardson et al., 2023) and the
other from ConTEXTure. To reduce bias in the answers provided by respondents,
the order in which the two textures are shown is shuffled in each part. Each
textured mesh is shown with four images and shows the front, left, back, and
right side. For each textured mesh pair, 30 respondents were asked to choose
between the two textures using the following criteria:

  1. (1)

Overall quality \- The higher the quality, the more natural the colors and
details appear, and the fewer unnatural artifacts there are.

  2. (2)

Viewpoint consistency \- The consistency is higher if features such as the
eyes, nose, and mouth are not positioned on the back or sides of the mesh.

  3. (3)

Prompt fidelity \- The higher the fidelity, the more accurately the texture
image reflects the prompt.

A total of 30 respondents were provided with an explanation on the meaning of
viewpoint consistency and prompt fidelity. The survey included 40 texture
pairs, where each pair included the generated texture results from both
TEXTure and ConTEXTure, the prompt used to generate the two textures, and a
question pertaining to each of the three criteria. Every question required the
respondent to choose which texture is superior in regards to the criterion.
Refer to Appendix B for more details.

The results of the user study demonstrated that ConTEXTure outperforms TEXTure
in viewpoint consistency, which demonstrates its superiority in addressing the
viewpoint bias problem. However, it was tied in terms of prompt fidelity and
fell short in regards to the overall quality. We analyzed the reasons this may
have been the case and realized that SD2-depth generates images with more
vibrant colors and more realistic details in object textures. Generated
textures using TEXTure that do not visibly suffer from viewpoint bias had the
tendency to score higher in the survey.

##  5\. Conclusion

In this paper, we presented a solution to the viewpoint bias problem that was
originally solved in novel view synthesis by Zero123 (Liu et al., 2023a) but
had yet to be attempted in texture atlas generation. Throughout the course of
this research, we discovered that the viewpoint bias problem is pervasive
throughout this domain, requiring us to make modifications to the evaluation
process presented in (Cao et al., 2023).

###  5.1. Future work

Although we were able to use Zero123++ for texture atlas generation, its
tendency to generate texture faces with shadows prevents it from being seen as
a pure texture generator. Although this does not end up being a problem when
the objective is to reconstruct a novel view of an object in the condition
image, this leads to problems in texture generation. Any future research would
have to involve separating lighting information from the base color
information of a given texture map. This will allow the image generation model
to have control over the lighting information.

The issue with undesirable artifacts on the generated texture atlas frequently
seen in TEXTure continues to be noticeable in ConTEXTure. Improving texture
generation to be more high quality and globally coherent was well achieved in
TexFusion. Although the code for TexFusion is closed source and our approach
largely diverges from that of TexFusion primarily due to our usage of
simultaneous multi-view inverse rendering, several approaches such as
interleaving and usage of latent texture maps would very likely improve
ConTEXTure in a similar manner.

As expressed in Section 4.2, using a “proxy ground truth dataset” for
evaluation, an approach that was originally proposed in (Cao et al., 2023),
has its shortcomings. This demonstrates a need for a more neutral evaluation
method, which we may attempt to create for future research.

## References

  * (1)
  * Betker et al. (2023) James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. 2023\.  Improving image generation with better captions.  _Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf_ 2, 3 (2023), 8.
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018.  Large scale GAN training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ (2018).
  * Cao et al. (2023) Tianshi Cao, Karsten Kreis, Sanja Fidler, Nicholas Sharp, and Kangxue Yin. 2023.  Texfusion: Synthesizing 3d textures with text-guided image diffusion models. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 4169–4181.
  * Ceylan et al. (2024) Duygu Ceylan, Valentin Deschaintre, Thibault Groueix, Rosalie Martin, Chun-Hao Huang, Romain Rouffet, Vladimir Kim, and Gaëtan Lassagne. 2024.  MatAtlas: Text-driven Consistent Geometry Texturing and Material Assignment.  _arXiv preprint arXiv:2404.02899_ (2024).
  * Chen et al. (2023a) Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner. 2023a.  Text2tex: Text-driven texture synthesis via diffusion models. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 18558–18568.
  * Chen et al. (2023b) Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner. 2023b.  Text2Tex: Text-driven Texture Synthesis via Diffusion Models. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 18558–18568.
  * Chen et al. (2022) Zhiqin Chen, Kangxue Yin, and Sanja Fidler. 2022.  Auv-net: Learning aligned uv maps for texture transfer and synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1465–1474.
  * Deitke et al. (2024) Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. 2024\.  Objaverse-xl: A universe of 10m+ 3d objects.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Nichol. 2021.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ 34 (2021), 8780–8794.
  * Gao et al. (2024) Chenjian Gao, Boyan Jiang, Xinghui Li, Yingpeng Zhang, and Qian Yu. 2024.  GenesisTex: Adapting Image Denoising Diffusion to Texture Space. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4620–4629.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ 30 (2017).
  * Ho and Salimans (2022) Jonathan Ho and Tim Salimans. 2022.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ (2022).
  * Jain et al. (2022) Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. 2022.  Zero-shot text-guided object generation with dream fields. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 867–876.
  * Jatavallabhula et al. (2019) Krishna Murthy Jatavallabhula, Edward Smith, Jean-Francois Lafleche, Clement Fuji Tsang, Artem Rozantsev, Wenzheng Chen, Tommy Xiang, Rev Lebaredian, and Sanja Fidler. 2019.  Kaolin: A pytorch library for accelerating 3d deep learning research.  _arXiv preprint arXiv:1911.05063_ (2019).
  * Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020.  Analyzing and improving the image quality of stylegan. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 8110–8119.
  * Lin et al. (2023) Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. 2023.  Magic3d: High-resolution text-to-3d content creation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 300–309.
  * Liu et al. (2023a) Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 2023a.  Zero-1-to-3: Zero-shot one image to 3d object. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 9298–9309.
  * Liu et al. (2023b) Yuxin Liu, Minshan Xie, Hanyuan Liu, and Tien-Tsin Wong. 2023b.  Text-guided texturing by synchronized multi-view diffusion.  _arXiv preprint arXiv:2311.12891_ (2023).
  * Metzer et al. (2023) Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. 2023.  Latent-nerf for shape-guided generation of 3d shapes and textures. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 12663–12673.
  * Michel et al. (2022) Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. 2022.  Text2mesh: Text-driven neural stylization for meshes. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 13492–13502.
  * Mildenhall et al. ([n. d.]) B Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and R Ng Nerf. [n. d.].  Representing scenes as neural radiance fields for view synthesis., 2021, 65.  _DOI: https://doi. org/10.1145/3503250_ ([n. d.]), 99–106.
  * Poole et al. (2022) Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. 2022.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv preprint arXiv:2209.14988_ (2022).
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ 1, 2 (2022), 3.
  * Ranftl et al. (2021) René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. 2021.  Vision Transformers for Dense Prediction.  _ArXiv preprint_ (2021).
  * Ranftl et al. (2020) René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. 2020.  Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ (2020).
  * Richardson et al. (2023) Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. 2023.  Texture: Text-guided texturing of 3d shapes.  _arXiv preprint arXiv:2302.01721_ (2023).
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 10684–10695.
  * Shi et al. (2023) Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. 2023.  Zero123++: a single image to consistent multi-view diffusion base model.  _arXiv preprint arXiv:2310.15110_ (2023).
  * Siddiqui et al. (2022) Yawar Siddiqui, Justus Thies, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. 2022.  Texturify: Generating textures on 3d shape surfaces. In _European Conference on Computer Vision_. Springer, 72–88.
  * Speed (2022) Tim Speed. 2022.  Flexdiffuse: An adaptation of stable diffusion with image guidance.  https://github.com/timspeed/flexdiffuse.  Accessed: 2022.
  * Tang et al. (2024) Jiaxiang Tang, Ruijie Lu, Xiaokang Chen, Xiang Wen, Gang Zeng, and Ziwei Liu. 2024.  InTeX: Interactive Text-to-texture Synthesis via Unified Depth-aware Inpainting.  _arXiv preprint arXiv:2403.11878_ (2024).
  * Tang and He (2023) Zhibin Tang and Tiantong He. 2023.  Text-guided high-definition consistency texture model.  _arXiv preprint arXiv:2305.05901_ (2023).
  * Wang et al. (2023) Tianfu Wang, Menelaos Kanakis, Konrad Schindler, Luc Van Gool, and Anton Obukhov. 2023.  Breathing new life into 3d assets with generative repainting.  _arXiv preprint arXiv:2309.08523_ (2023).
  * Wu et al. (2024) Jinbo Wu, Xing Liu, Chenming Wu, Xiaobo Gao, Jialun Liu, Xinqi Liu, Chen Zhao, Haocheng Feng, Errui Ding, and Jingdong Wang. 2024.  TexRO: Generating Delicate Textures of 3D Models by Recursive Optimization.  _arXiv preprint arXiv:2403.15009_ (2024).
  * Yang et al. (2022) Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang Chen, and Kwan-Yee K. Wong. 2022.  PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo. In _Computer Vision – ECCV 2022_ , Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer Nature Switzerland, Cham, 266–284.
  * Youwang et al. (2024) Kim Youwang, Tae-Hyun Oh, and Gerard Pons-Moll. 2024.  Paint-it: Text-to-texture synthesis via deep convolutional texture map optimization and physically-based rendering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4347–4356.
  * Zhang et al. (2023) Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023.  Adding conditional control to text-to-image diffusion models. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 3836–3847.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018.  The unreasonable effectiveness of deep features as a perceptual metric. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 586–595.

##  Appendix A Z-normal binary mask generation

Since ConTEXTure uses multiple novel view images to learn the texture map in a
simultaneous fashion, it is crucial to determine which images should apply to
which areas of the mesh. We follow the basic idea that every face of the mesh
should be _assigned_ to it the viewpoint in which its z-normal value is
maximized. To create this binary face view mask, we used a vectorized version
of Algorithm 2.

Algorithm 2 Binary mask creation

1:Input: Face index $F$, normal map $N$

2:$M\leftarrow\text{\\{\\}}$

3:$(V,H,W)\leftarrow\text{shape of }F$

4:$mask\leftarrow\text{Boolean matrix of shape }(V,H,W)$

5:for $v=0\text{ to }V-1$ do

6: for $i=0\text{ to }H-1$ do

7: for $j=0\text{ to }W-1$ do

8: $f\leftarrow F[v,i,j]$

9: if $f\text{ is valid}$ then

10: if $f\notin M$ then

11: $M[f]\leftarrow\text{\\{\\}}$

12: end if

13: if $v\notin M[f]$ then

14: $M[f][v]\leftarrow\text{[]}$

15: end if

16: Append $(i,j)$ to $M[f][v]$

17: end if

18: end for

19: end for

20:end for

21:for $f\text{ in }M$ do

22: if $M[f]\text{ is not empty}$ then

23: $V_{f}\leftarrow\text{keys of }M[f]$

24: $Z_{f}\leftarrow N[V_{f},2,f]$

25: $z_{\max}\leftarrow\max(Z_{f})$

26: for $v\text{ in }M[f]$ do

27: if $N[v,2,f]<z_{\max}$ then

28: $i,j\leftarrow M[f][v]$

29: $mask[v,i,j]\leftarrow\text{False}$

30: end if

31: end for

32: end if

33:end for

34:Output: $mask$

##  Appendix B User study details

The texture prompts used in the survey are listed in Table 2. Since each
texture pair includes three questions, there is a total of 120 questions in
the full survey. Due to the sheer length of the full survey, we created one
unique survey for each respondent where each survey asks about a random
selection of 10 prompts. After 30 respondents completed the survey, we
calculated the ratings for each texture. For each criteria of each texture
pair, the texture with more votes was given one point. Thus, the denominator
of the percentages in the user study results of Table 1 is the number of
prompts in the survey, or 40. Because most of the survey respondents were
Korean native speakers, all English text prompts were translated to Korean
using ChatGPT 4o.

Mesh | Text Prompts
---|---
| person.obj
---
(Text2Mesh (Michel et al., 2022))
| comic book superhero, red body suit
---
white humanoid robot, movie poster, villain character of a science fiction
movie
futuristic soldier, glowing armor, protagonist of an action game
steampunk adventurer, leather attire with brass accessories
astronaut in a sleek space suit, exploring alien worlds
cyberpunk hacker, neon-lit clothing, main character in a dystopian cityscape
| rp_alvin_rigged_003_yup_a.obj
---
(Renderpeople)
| person wearing black shirt and white pants
---
person wearing white t-shirt with a peace sign
person wearing a classic detective trench coat and fedora
surfer wearing board shorts with a tropical pattern
mountaineer in a thermal jacket and snow goggles
chef in a white jacket and checkered pants
pilot in a vintage leather jacket with aviator sunglasses
| rp_alexandra_rigged_004_yup_a.obj
---
(Renderpeople)
| person in red sweater, blue jeans
---
person in white sweater with a red logo, yoga pants
professional gamer in a team jersey and headphones
ballet dancer in a pink tutu and ballet slippers
rock star with leather jacket
vintage 1950s dress with polka dots and sunglasses
athlete in a running outfit with a marathon number
| rp_adanna_rigged_007_yup_a.obj
---
(Renderpeople)
| nunn in a black dress
---
nunn in a white dress, black headscarf
professional in a suit jacket, skirt, and elegant headscarf
athlete in sportswear with a sporty hijab
artist in a paint-splattered apron and a stylish hijab
student in a denim jacket, casual dress, and a colorful headscarf
doctor in a lab coat with a simple, modest hijab
| rp_aaron_rigged_001_yup_a.obj
---
(Renderpeople)
| railroad worker wearing high-vis vest
---
biker wearing red jacket and black pants
firefighter in full gear with reflective stripes
plumber in a blue jumpsuit
electrician with a tool belt and safety goggles
carpenter in overalls with a hammer in pocket
landscape gardener in a green t-shirt and cargo pants
| human.obj
---
| a photo of spiderman
---
a caricature of a pirate with a large hat and eye patch
a whimsical wizard with a pointed hat, dark shadow
a cartoon astronaut with a bubbly space helmet
a ninja turtle with a colorful mask
a cartoon zombie in tattered clothes
Table 2. Meshes and text prompts used for the user study. Most of these text prompts and meshes were also used in TexFusion (Cao et al., 2023). human.obj is a proprietary mesh and not included in any particular dataset. Bolded prompts are shown in Table 3 in order of the meshes listed here. |  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
Table 3. Six texture maps constructed using ConTEXTure. Each image on each row
was rendered from the texture map from four equidistant azimuth angles,
$0\degree$, $90\degree$, $180\degree$, and $270\degree$ (from left to right).