∎

11institutetext:  institute Changwen Zheng (Corresponding Author.)
22institutetext: 22email<EMAIL_ADDRESS>33institutetext: 1 Institute of
Software, Chinese Academy of Sciences, Beijing, China.
2 University of Chinese Academy of Sciences, Beijing, China.
3 JD Explore Academy, Beijing, China.

# 3DDesigner: Towards Photorealistic 3D Object Generation and Editing with
Text-guided Diffusion Models

Gang Li1,2 Heliang Zheng3 Chaoyue Wang3 Chang Li3
Changwen Zheng 1 Dacheng Tao3

(Received: October 7, 2023 / Accepted: xxx )

###### Abstract

Text-guided diffusion models have shown superior performance in image/video
generation and editing. While few explorations have been performed in 3D
scenarios. In this paper, we discuss three fundamental and interesting
problems on this topic. First, we equip text-guided diffusion models to
achieve 3D-consistent generation. Specifically, we integrate a NeRF-like
neural field to generate low-resolution coarse results for a given camera
view. Such results can provide 3D priors as condition information for the
following diffusion process. During denoising diffusion, we further enhance
the 3D consistency by modeling cross-view correspondences with a novel two-
stream (corresponding to two different views) asynchronous diffusion process.
Second, we study 3D local editing and propose a two-step solution that can
generate 360∘ manipulated results by editing an object from a single view.
Step 1, we propose to perform 2D local editing by blending the predicted
noises. Step 2, we conduct a noise-to-text inversion process that maps 2D
blended noises into the view-independent text embedding space. Once the
corresponding text embedding is obtained, 360∘ images can be generated. Last
but not least, we extend our model to perform one-shot novel view synthesis by
fine-tuning on a single image, firstly showing the potential of leveraging
text guidance for novel view synthesis. Extensive experiments and various
applications show the prowess of our 3DDesigner. The project page is available
at https://3ddesigner-diffusion.github.io/

###### Keywords:

Text-guided Diffusion Models 3D Object Generation Novel View Synthesis Local
Editing of 3D Objects.

Figure 1: “3DDisigner” – Text guided 3D object generation and editing. Given a
text, e.g., “A red sedan BMW 5 series 2021”, our method can 1) generate the
corresponding 3D images, 2) create “SUV” and “Sports” 3D counterparts, and 3)
support text-guided 3D local editing.

##  1 Introduction

Recently, the advent of text-guided diffusion models has brought about a
paradigm shift in the field of image synthesis, offering remarkable
advancements in the creation of high-fidelity, diverse, and highly
controllable images Nichol et al. (2022); Ramesh et al. (2022); Saharia et al.
(2022). These models have sparked the rise of what can be termed “AI artists,”
capable of producing intricate and imaginative artworks within mere seconds.
Moreover, substantial strides have been achieved in the domain of image
editing Kim et al. (2022); Avrahami et al. (2022); Lugmayr et al. (2022); Gal
et al. (2023); Ruiz et al. (2023); Kawar et al. (2023), enabling complex and
nuanced text-guided semantic modifications, even for objects with non-rigid
structures, using nothing more than a simple text prompt. Beyond static
images, text-guided diffusion models have been successfully extended to the
realm of video synthesis Ho et al. (2022b); Singer et al. (2023); Ho et al.
(2022a); Villegas et al. (2023), achieving remarkable results across multiple
dimensions, including enhanced data efficiency, the production of high-
definition video content, and the extension of video length.

Real-world objects are 3D, and it usually requires multi-view information to
represent every detail of an object. Creating 3D assets demands specialized
software and skills, and the process can be time-consuming compared to
generating 2D content. Consequently, a tool capable of 3D object generation
holds great significance for real-world applications. However, limited
exploration has been conducted on text-guided diffusion models in 3D
scenarios. CLIP-NeRF Wang et al. (2022) integrate a CLIP Radford et al. (2021)
model to guide the rendering of a Neural Radiance Field (NeRF) Mildenhall et
al. (2021), while without the generation capability of diffusion models, CLIP-
NeRF cannot achieve flexible control and local editing. 3DiM Watson et al.
(2023) firstly makes a diffusion model work well for 3D novel view synthesis,
while without text guidance, it is hard to perform novel object generation and
manipulation. Dreamfusion Poole et al. (2023) creatively adopt text-guided
diffusion models as supervision to train a NeRF, but such a solution is not
efficient, i.e., an optimized model can only generate one specific object.

In this paper, we delve into text-guided diffusion models for 3D object
generation and manipulation (Figure 1). We originally discuss three
fundamental and interesting problems. The quintessential and arguably the most
crucial concern: the attainment of 3D-consistent generation. Our proposed
model comprising a NeRF-based condition module and a two-stream asynchronous
diffusion module. The NeRF-based condition module takes camera views and
coarse text guidance (e.g., “A red sedan BMW”) and generates low-resolution
coarse results for the given camera views as inputs. Such results can provide
3D priors, serving as condition information for the following diffusion
process. The two-stream asynchronous diffusion module is designed to further
enhance the 3D consistency by modeling cross-view correspondences. This
approach entails the joint denoising of two distinctively noised images, each
derived from varying perspectives (one perspective per stream) of same object,
where the cross-view feature interactions can encourage the two streams to
generate images that are consistent with each other. During sampling, one of
the two streams loads previously generated views to guide the generation of
the current view. In this way, 360∘ consistent results can be obtained.

Secondly, we explore 3D local editing. To the best of our knowledge, we are
the first to perform 360∘ manipulated results by editing an object from a
single view. We achieve this in two steps. In Step 1, we perform 2D local
editing. We propose a “noise blending” pipeline to edit a specific region of a
generated image. Specifically, during each sampling step, we predict the noise
under the guidance of the provided text and replace a defined region of this
noise with the noise predicted by the target text. This allows us, for
example, to seamlessly combine the body of a BMW Series 5 car with the front
end of a Benz Class-E car. In Step 2, we design a noise-to-text inversion
process that maps 2D blended noises into the view-independent text embedding
space. Once the corresponding text embedding is obtained, 360∘ images can be
generated.

Last but not least, we discover that our model can also be easily extended to
perform one-shot novel view synthesis by simply fine-tuning on a single image.
Impressive results demonstrate the potential of leveraging text guidance for
novel view synthesis, and we hope our work provides valuable insights for the
novel view synthesis community.

##  2 Related Work

###  2.1 Text-guided diffusion models

Diffusion models Sohl-Dickstein et al. (2015); Song and Ermon (2019); Ho et
al. (2020) have made tremendous progress recently, showing superiority on
stable training, high-fidelity image/video synthesis, non-trivial semantic
editing, multi-modality fusion, etc Dhariwal and Nichol (2021); Nichol et al.
(2022); Rombach et al. (2022); Avrahami et al. (2022); Gal et al. (2023);
Kawar et al. (2023); Singer et al. (2023); Villegas et al. (2023); Ramesh et
al. (2022). Meanwhile, billions of (text, image) pairs Schuhmann et al. (2022)
enable the recent breakthroughs in text-to-image synthesis Ramesh et al.
(2021, 2022); Saharia et al. (2022); Rombach et al. (2022); Yu et al. (2022).
Thus not surprisingly, text-guided diffusion models have shown impressive
performance and attracted lots of attention. In this work, we further explore
the performance of text-guided diffusion models for 3D generation.

Once a text-guided diffusion model is trained, it can be further extended to
perform image editing. GLIDE Nichol et al. (2022) formulate an in-painting
task, where the text can guide the generation of the masked region. Blend
Diffusion Avrahami et al. (2022) and Repaint Lugmayr et al. (2022) spatially
blend noised versions of the input image with the local text-guided diffusion
latent at a progression of noise levels. Textual inversion Gal et al. (2023)
and DreamBooth Ruiz et al. (2023) propose the concept of “personalizing
generation”, which inverts an object to a special text token. Imagic Kawar et
al. (2023) propose a fine-tuning technique that enables various text-based
semantic edits on a single real input image. In this work, we extend noisy
image blending to noise blending and further conduct noise-to-text inversion
for 3D local editing.

Figure 2: An illustration of our framework for text-guided 3D-consistent
generation (training phase). (A) NeRF-based Condition Module, which takes
$<$one coarse text, two camera views$>$ pairs as inputs and generates low-
resolution coarse results. The coarse results are resized and concatenated
with noised images to provide conditions for denoising. (B) Two-stream
Asynchronous Diffusion Module, which takes $<$one full text, two coarse
results, two timesteps, two noised images$>$ quadruples as inputs and predicts
the added noises. Each stream is a vanilla text-guided diffusion model except
for the feature interaction module after each attention block. Note that the
timesteps are randomly generated and the parameters of these two streams are
shared.

###  2.2 3D novel view synthesis

Since NeRF Mildenhall et al. (2021) was firstly proposed to learn implicit 3D
representations for the task of novel view synthesis, plenty of variants have
been proposed to improve the efficiency Zhang et al. (2021); Müller et al.
(2022); Zhu et al. (2023), generalization Chen et al. (2021); Johari et al.
(2022); Yu et al. (2021); Sharma et al. (2022); Rebain et al. (2022); Lan et
al. (2023), controllability Yang et al. (2021); Wang et al. (2022); Lazova et
al. (2023), etc. For example, CLIP-NeRF Wang et al. (2022) integrates CLIP
Radford et al. (2021) with NeRF and makes the colors and shapes can be
controlled by text. Some recent literature also proposes to get rid of
explicit geometric inductive biases (e.g., introduced by volume rendering).
3DiM Watson et al. (2023) firstly makes a diffusion model work well for 3D
novel view synthesis. Moreover, Dreamfusion Poole et al. (2023) creatively
adopt text-guided diffusion models as supervision to train a NeRF. Although
GAUDI Bautista et al. (2022) also integrates NeRF with diffusion models, the
diffusion model here is only used to learn a controllable latent code,
preventing their model from performing local editing. Compared to these works,
we leverage text-guided diffusion models to achieve more controllable and
editable 3D generation.

##  3 Approach

In this section, we introduce the details of our proposed 3DDesigner,
including 1) 3D-consistent generation, i.e., learning a text-guided diffusion
model that can generate images from different camera views with consistent
content; 2) 3D local editing, i.e., generating 360∘ manipulated results by
editing an object from a single view; and 3) one-shot novel view synthesis,
i.e., synthesising 360∘ results based on a single image.

###  3.1 3D-consistent generation

Figure 2 is an illustration of our framework for text-guided 3D-consistent
generation (training phase). Specifically, we integrate a NeRF-like neural
field to generate 3D-consistent coarse results, which are leveraged in the
following denoising diffusion process in the form of image conditions. For the
denoising diffusion process, we propose to jointly denoise two noised images
of an object with different views. So that the cross-view feature interactions
can encourage the two streams to generate images that are consistent with each
other. For now, the two views’ consistency can be achieved. To further obtain
360∘ consistent results, we adopt an autoregressive sampling strategy Watson
et al. (2023), where one stream loads previously generated views to guide the
other stream to generate the current view.

NeRF-based condition module. The NeRF-based condition module takes $<$coarse
text, camera view$>$ pairs as inputs and generates coarse results. Note that
we do not use the full-text guidance here to avoid suppressing the creative
ability of the following denoising diffusion module (i.e., making it a simple
super-resolution module).

Learning MLP. We first calculate the 3D location and view direction of each
camera ray according to the given camera view. The camera ray can be denoted
as $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$, where $\mathbf{o}$ and $\mathbf{d}$
are the origin and direction of the ray, respectively. We use a multiple layer
perceptual (MLP) to predict the density $\sigma(t)$ and RGB color
$\mathbf{c}(t)$ of a given point in the NeRF implicit representation:

$\sigma(t),\mathbf{c}(t)=MLP(\mathbf{r}(t),\mathbf{d},\mathbf{y}_{c})$ (1)

where $\mathbf{y}_{c}$ denotes the coarse text embedding. In particular, we
concatenate the position embedded of $\mathbf{r}(t)$ and the text embedding
$\mathbf{y}_{c}$ before getting through the MLP.

Volume rendering. We can obtain coarse results by conducting volume rendering:

$\mathbf{x}_{c}(\mathbf{r})=\int_{t_{n}}^{t_{f}}T(t)\sigma(t)\mathbf{c}(t)dt$
(2)

where $t_{n}$ and $t_{f}$ are the near and far bounds, respectively.
$T(t)=exp(-\int_{t_{n}}^{t}\sigma(s)ds)$ handles occlusion, and
$\mathbf{x}_{c}(\mathbf{r})$ indicates the RGB value of the coarse result
$\mathbf{x}_{c}$ that rendered by camera ray $\mathbf{r}$. In our experiments,
we follow StyleNeRF Gu et al. (2021), 1) approximating the volume rendering
process to a GPU-friendly version that supports operations on a set of rays
and 2) removing the view direction condition to suppress spurious correlations
and dataset bias.

Two-stream asynchronous diffusion module. The two-stream asynchronous
diffusion module takes $<$full text, coarse results, timesteps, noised
images$>$ quadruples as inputs and predicts the added noises.

Diffusion. Given two images of an object sampled from different views,
$\mathbf{x}^{1}$ and $\mathbf{x}^{2}$, the diffusion processes (i.e., adding
noise) for these two images are independent, which can be denoted as:

$\begin{split}&q(\mathbf{x}^{1}_{t}|\mathbf{x}^{1}_{0})\coloneqq\mathcal{N}(\mathbf{x}^{1}_{t};\sqrt{\bar{\alpha}^{1}_{t}}\mathbf{x}^{1}_{0},(1-\bar{\alpha}^{1}_{t})\mathbf{I}),\\\
&q(\mathbf{x}^{2}_{t}|\mathbf{x}^{2}_{0})\coloneqq\mathcal{N}(\mathbf{x}^{2}_{t};\sqrt{\bar{\alpha}^{2}_{t}}\mathbf{x}^{2}_{0},(1-\bar{\alpha}^{2}_{t})\mathbf{I}),\end{split}$
(3)

where $t$ is the timestep, $\bar{\alpha}_{t}$ can be calculated by the
variance schedule, and $\mathbf{I}$ is an identity matrix.

Posterior. Now let’s move on to approximate the posterior (i.e., reverse
process), where all of the coarse results, cross-view information, and text
guidance can be leveraged:

$\begin{split}&p_{\theta}(\mathbf{x}^{1}_{t_{1}-1}|\mathbf{x}^{1}_{t_{1}},\mathbf{x}^{1}_{c},\mathbf{x}^{2}_{t_{2}},\mathbf{x}^{2}_{c},\mathbf{y})\coloneqq\\\
&\mathcal{N}(\mu_{\theta}(\mathbf{x}^{1}_{t_{1}},\mathbf{x}^{1}_{c},\mathbf{x}^{2}_{t_{2}},\mathbf{x}^{2}_{c},\mathbf{y}),\Sigma_{\theta}(\mathbf{x}^{1}_{t_{1}},\mathbf{x}^{1}_{c},\mathbf{x}^{2}_{t_{2}},\mathbf{x}^{2}_{c},\mathbf{y})),\end{split}$
(4)

where $\mathbf{y}$ is the full text, $\mathbf{x}_{c}$ is the coarse result
obtained by Equation 2, $t_{1}$ and $t_{2}$ are randomly generated timesteps,
$\mu_{\theta}$ and $\Sigma_{\theta}$ are models that predict the mean and
variance of the Gaussian distribution, respectively.

Optimization. In practice, we follow the widely used variational lower-bound
re-weighing to learn the posterior. Specifically, we generate samples
$\\{\mathbf{x}^{1}_{t_{1}},\mathbf{x}^{2}_{t_{2}}\\}$ by applying Gaussian
noise $\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ to
$\\{\mathbf{x}^{1}_{0},\mathbf{x}^{2}_{0}\\}$ using Equation 3. Then we train
a model $\epsilon_{\theta}$ to predict the added noise:

$L=\mathbb{E}_{\mathbf{y},\mathbf{x}^{1}_{0},\mathbf{x}^{2}_{0},t_{1},t_{2},\epsilon}\parallel\epsilon_{\theta}(\mathbf{x}^{1}_{t_{1}},\mathbf{x}^{1}_{c},\mathbf{x}^{2}_{t_{2}},\mathbf{x}^{2}_{c},\mathbf{y})-\epsilon\parallel.$
(5)

Architecture. We follow the design of multi-stream U-Nets that are widely
obtained in video generation Ho et al. (2022b); Singer et al. (2023) and 3DiM
Watson et al. (2023). Specifically, the two-stream asynchronous diffusion
module consists of two U-Nets and several feature interaction modules. The
interaction module can be the temporal convolution/attention used in Make-A-
Video Singer et al. (2023) or the cross-attention used in 3DiM Watson et al.
(2023) (the latter performs better on our task). The coarse results obtained
by Equation 2 are resized and concatenated with noised images obtained by
Equation 3. The text embedding is integrated by attention blocks, and the
timestep is specified by adding position embedding into each convolution block
Ho et al. (2020).

Sampling. The cross-view feature interaction can encourage the two streams to
generate images that are consistent with each other in the sampling phase.
Note that we adopt different timesteps to generate different-level noises. Our
insight is to make the “cleaner” (condition) stream to guide the generation of
the “noisier” (generation) stream. In particular, we adopt the DDIM Song et
al. (2021) sampling process:

$\mathbf{x}^{1}_{t_{1}-1}=\sqrt{\alpha^{1}_{t_{1}-1}}(\frac{\mathbf{x}^{1}_{t_{1}}-\sqrt{1-\alpha^{1}_{t_{1}}}\hat{\epsilon}_{t_{1}}}{\sqrt{\alpha^{1}_{t_{1}}}})+\sqrt{1-\alpha^{1}_{t_{1}-1}}\hat{\epsilon}_{t_{1}},$
(6)

where
$\hat{\epsilon}_{t_{1}}=\epsilon_{\theta}(\mathbf{x}^{1}_{t_{1}},\mathbf{x}^{1}_{c},\mathbf{x}^{2}_{t_{1}-\Delta
t},\mathbf{x}^{2}_{c},\mathbf{y})$ is the predicted noise, $\Delta t\geq 0$ is
a hyper-parameter to adjust the noise level of the two streams. We further
follow 3DiM Watson et al. (2023) to adopt an autoregressive sampling strategy,
where the second stream loads previously generated views to guide the
generation of the current view. In the training phase, two timesteps are
randomly sampled, and in the sampling phase, we set $\Delta t$ to be a
constant (except for the first view, where $\Delta t=0$).

Figure 3: An illustration of 3D local editing. We propose to blend noises in
each sampling step to achieve 2D local editing and conduct noise-to-text
inversion to generate 3D manipulated images. The notations are explained in
Sec. 3.2.

###  3.2 3D local editing

The model introduced above makes 3D-consistent visual content creation as
simple as preparing a text. Moreover, to further achieve iterative refinement
and fine-grained control, 3D local editing is required. 3D local editing takes
a generated image (actually, the corresponding text), a single view mask that
indicates the region to be manipulated, and a target text that guides the
manipulation as inputs. The output would be manipulated images from different
views. In the following, we first introduce our “noise blending” for 2D local
editing. After that, we introduce a noise-to-text inversion process that
enables our model to generate other views of the manipulated object. Figure 3
shows an illustration to make the editing process clearer.

Noise blending for editing a single view image. We conduct spatial blending
over the predicted noises to perform local editing for a given camera view.
Formally, in each sampling step, we have:

$\begin{split}\hat{\epsilon}^{b}_{t-1}&=(1-\mathbf{M})\odot\hat{\epsilon}^{o}_{t-1}+\mathbf{M}\odot\hat{\epsilon}^{n}_{t-1},\\\
\hat{\epsilon}^{o}_{t-1}&=\epsilon_{\theta}(\mathbf{x}^{b}_{t},\mathbf{y}^{o}),\\\
\hat{\epsilon}^{n}_{t-1}&=\epsilon_{\theta}(\mathbf{x}^{b}_{t},\mathbf{y}^{n}),\end{split}$
(7)

where $\hat{\epsilon}^{b}_{t-1}$ is the blended noise at timestep $t-1$,
$\mathbf{M}$ is the user-provided mask that indicates the region to be
manipulated, $\odot$ denotes the Hadamard product (i.e., element-wise
product), $\hat{\epsilon}^{o}_{t-1}$ and $\hat{\epsilon}^{n}_{t-1}$ are the
noise predicted based on the original $\mathbf{y}^{o}$ and new (target) text
$\mathbf{y}^{n}$, respectively. $\epsilon_{\theta}$ is the trained diffusion
model (we omit the coarse results and the other view for simplicity),
$\mathbf{x}^{b}_{t}$ is the image that sampled by Equation 6 with the blended
noise in the last timestep.

Noise-to-text inversion for generating 3D manipulated images. To generate
other views of the manipulated object, our insight is to invert the blended
noise into the view-independent text space. Specifically, we fix the diffusion
model and optimize a text embedding $\mathbf{y}^{b}$ to fit the blended noise
in each sampling step:

$L_{inv}=\Sigma_{t}\parallel\hat{\epsilon}^{b^{*}}_{t-1}-{\epsilon_{\theta}}^{*}({\mathbf{x}^{b^{*}}_{t}},\mathbf{y}^{b})\parallel,$
(8)

where $*$ indicates the fixed variables. In practice, we find that if the
manipulated region is small, the optimization tends to be dominated by the
original noise $\hat{\epsilon}^{o}_{t}$. Thus we extend the above Equation by
adding a loss weight $\lambda$ to the manipulated region:

$\displaystyle L_{inv}=$
$\displaystyle\Sigma_{t}\|(1-\mathbf{M})\odot({\hat{\epsilon}^{o^{*}}_{t-1}}-{\epsilon_{\theta}}^{*}({\mathbf{x}^{b^{*}}_{t}},\mathbf{y}^{b}))$
(9)
$\displaystyle+\lambda\mathbf{M}\odot({\hat{\epsilon}^{n^{*}}_{t-1}}-{\epsilon_{\theta}}^{*}({\mathbf{x}^{b^{*}}_{t}},\mathbf{y}^{b}))\|.$

Please refer to Equation 7 for the explanation of notations.

Figure 4: Our 3DDsigner can perform (A) fine-grained 3D generation, (B)
semantic meaningful interpolation in the text embedding space, and (C)
controllable generation that can change car types to create counterparts of
real-world car models.

Discussion and comparison. We undertake a comprehensive comparative analysis
and engage in in-depth discussions from three distinct perspectives: local
editing, inversion, and alternative solutions.

Local editing. Our “noise blending” pipeline is similar to Blended-diffusion
Avrahami et al. (2022), which conducts spatially blending over the noisy image
to edit a specific region of a given image. We propose to adapt the
manipulation space from the noisy image space to the noise space, thus we can
further invert the blended noise to a text embedding (as introduced in noise-
to-text inversion) to achieve 3D local editing.

Inversion. The high-level insight of our noise-to-text inversion process is
somehow similar to Textual inversion Gal et al. (2023), which learns a special
text token with a pretrained diffusion model (fixed) to represent a specific
concept (from 3-5 images). While an essential difference is that instead of
following the fine-tuning paradigm, we learn the text embedding by fitting a
whole DDIM sampling process, i.e., learning to fit every step that generates
the manipulated object. Figure 11 shows the effectiveness of our method.

An alternative solution. An alternative solution for 3D local editing would be
generating a 2D manipulated image and further synthesising other views. 1) As
discussed above, with a fixed diffusion model, image-to-text is much more
difficult than noise-to-text inversion. 2) Fine-tuning diffusion models
(Section 3.3) can achieve one-shot novel view synthesis but not efficient.
Since every time to fit an object, the diffusion model needs to be adjusted.

###  3.3 One-shot novel view synthesis

In this study, we delve into a compelling question, namely, ‘How does our
text-guided 3D generation model perform in the realm of novel view synthesis?’
Specifically, one-shot novel view synthesis aims to synthesize new views of an
object based on a single given image. Leveraging the prior object information
learned by our model, we find that achieving one-shot novel view synthesis is
easily attainable through fine-tuning on the provided image using Equation 5.
To elaborate, we introduce noise to the given image and fine-tune our model to
predict this added noise. During the fine-tuning process, we ensure that both
views of our model align with the given view, and the timesteps are
synchronized ($\Delta t=0$). For detailed results and further discussions,
please refer to Section 4.4.

It’s crucial to highlight that only one given view is available for optimizing
the two streams, and all parameters of our model are subject to training for
the task of one-shot novel view synthesis.

##  4 Experiments

###  4.1 Experiment setup

In this section, we present the experimental details of our approach. We
outline the datasets, implementation details, baseline methods and evaluation
metrics.

Figure 5: A set of examples consisting of image-text pairs sourced from our collected RealCars dataset. Table 1: Distribution of the various car types. Car Type | SUV | Sedan | Hatchback | Sports | MPV | Estate
---|---|---|---|---|---|---
Num | 5854 | 3693 | 1040 | 636 | 443 | 115
- | Fastback | Hardtop convertible | Convertible | Pickup | Minibus | Crossover
- | 114 | 95 | 19 | 16 | 10 | 4
Table 2: Distribution of the various car colors. Car Color | White | Blue | Black | Red | Silver | Gray | Orange | Yellow | Green
---|---|---|---|---|---|---|---|---|---
Num | 2737 | 2111 | 2029 | 1858 | 1307 | 909 | 615 | 549 | 161
Table 3: Distribution of the various car brands (only the top 50 brands with the highest quantity are shown). Car Brand | BMW | Audi | Mercedes Benz | Volkswagen | Benz | Toyota | Porsche | BYD
---|---|---|---|---|---|---|---|---
Num | 775 | 730 | 647 | 566 | 509 | 331 | 293 | 245
- | Changan | Geely | Honda | Chery | Volvo | Dongfeng | Ford | Skoda
- | 233 | 222 | 221 | 218 | 218 | 217 | 210 | 189
- | Havel | GAC | Nissan | Buick | Mini | Aston Martin | Land Rover | Hyundai
- | 177 | 172 | 171 | 166 | 162 | 156 | 155 | 151
- | Land Rover | Lexus | Cadillac | Jeep | Jaguar | Baojun | SAIC | BAIC
- | 149 | 137 | 128 | 126 | 125 | 122 | 120 | 114
- | Maserati | Roewe | Lincoln | Infiniti | Tesla | Mazda | Chevrolet | Qichen
- | 108 | 104 | 101 | 92 | 83 | 79 | 75 | 74
Figure 6: Visualization and comparison in terms of text-guided 3D generation.
Our model can generate high-fidelity images that are more controllable than
CLIP-NeRF Wang et al. (2022) and better 3D-consistent than extended 3DiM
Watson et al. (2023). Figure 7: The generated results of chair category on
3D-FUTURE dataset. We show the results of Image-to-3D, local loading, and
color-controllable chair generation, respectively. Figure 8: Visualization on
text-guided 3D local editing. The left column shows the given original images
and mask images, while the right column visualizes randomly selected 2D images
from 10 different perspectives after local editing. Figure 9: Visualization of
One-Shot Novel View Synthesis. The left column displays the given original
images, while the right column visualizes 2D images from 10 different
perspectives generated through one-shot novel view synthesis. Figure 10:
Ablation study on the two-stream asynchronous diffusion module: (a) single-
stream diffusion, (b) synchronistic timesteps, (c) clean condition, and (d)
ours. The red circles locate 3D-inconsistent parts. [Best viewed with zoom in]
Figure 11: Visualization of 3D local editing. Noise-to-text inversion
outperforms image-to-text inversion, and our model can generate 360∘ results
given a single view mask. Figure 12: Visualization and comparison on one-shot
novel view synthesis. Leveraging text guidance can significantly improve the
performance of novel view synthesis.

RealCars Dataset. We collected a total of 493,530 images(including relevant
textual information) from 11,696 cars, of which 4,755 cars have 60 views
(including 30 views from a 360-degree horizontal perspective and 30 views from
a 360-degree bird’s-eye perspective, both at 12-degree intervals), and the
remaining 6,941 cars have 30 views. We train two classifiers on CompCars
dataset to predict the color and type of each car to enrich text information.
We present the distribution of the RealCars dataset by car type and color in
Table 1 and Table 2, respectively. The collected text shows that our dataset
contains approximately 202 brands of cars, and we display the top 40 brands in
Table 3. We also randomly present a set of examples consisting of image-text
pairs sourced from our dataset in Figure 5. We collect a 3D car dataset, which
contains around 3k car models with corresponding texts (i.e., model names).
Each model is rendered into 30 images from different camera views (12∘/360∘
per view) and augmented by around 4 different colors. Some models are rendered
into 60 images with two different pitch angles. We also train a classification
model on CompCars Yang et al. (2015) dataset to predict the color and type of
each car model. To this end, we can obtain a 3D car dataset with 400k images
with texts, e.g., “A red sedan BMW 5 series 2021”. We will make this dataset
publicly released. For text-guided 3D generation, we create a testing set
where the texts are novel compositions of the color, types, and model names.
For novel view synthesis, the training and testing set contains different car
models.

3D-FUTURE dataset. In addition to conducting experiments on the RealCars
dataset that we proposed, we also conducted experiments on the chair category
of the 3D-FUTURE dataset Fu et al. (2021). 3D-FUTURE is an extensive
collection of industrial computer-aided design (CAD) models for furniture and
interior finishes. Each shape is associated with multiple textures and
materials, leading to the expansion of shape repositories. We opted for the
relatively clean ‘chair’ supercategory from this dataset for our experiments.
To align more closely with our method of reconstructing 3D objects based on 2D
images, we used the Blender tool to render textures of 3D models into 2D
images from various viewpoints. We captured viewpoints at 12-degree intervals,
resulting in a total of 30 different perspectives for each 3D model, which
served as input data for our model.

Implementation details. We use Pytorch Paszke et al. (2019) as our codebase
and the overall training of our model costs 14 days on 8 NVIDIA A100 GPUs. The
model size is around 600Mb. During inference, we adopt DDIM Song et al. (2021)
sampling with 100 steps. Note that the NeRF-based condition model only needs
to run one time (instead of 100), thus the inference time is around 2 times
longer than traditional single-stream diffusion models. The loss weight
$\lambda$ in Equation 9 is experimentally set to be 3.

Table 4: Quantitative Evaluation on text-guided 3D generation. 3DiM*: our extended version to support text-guided generation. Method | Components | Consistency | Quality | Controllability
---|---|---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | FID $\downarrow$ | Acc $\uparrow$(%)
CLIP-NeRF Wang et al. (2022) | NeRF | 35.10 | 0.95 | 62.97 | 10.90
3DiM* Watson et al. (2023) | Diffusion | 29.13 | 0.89 | _35.46_ | _60.13_
Ours | NeRF+Diffusion | _30.84_ | _0.93_ | 18.77 | 71.33

Baselines. Since there are few works exploring text-guided 3D object
generation, we re-implement and compare two related models on our task to
discuss the effectiveness and limits of NeRF and diffusion components. CLIP-
NeRF Wang et al. (2022) leverages text features and NeRF to achieve multi-
modal 3D object manipulation, we remove the CLIP branch and jointly learn a
text embedding and a NeRF on our dataset. 3DiM Watson et al. (2023) is a
diffusion model designed for novel view synthesis, we extend their model to a
text-guided version.

Evaluation metric. We evaluate our model from three dimensions, i.e., 3D
consistency, image quality, and controllability. Specifically, 1) we follow
3DiM Watson et al. (2023) to adopt a 3D consistency scoring, which can
evaluate the 3D consistency of multi-view images without requiring ground
truth images. 2) We adopt FID Heusel et al. (2017) to evaluate the image
quality. Moreover, 3) we use a pre-trained classification model Yang et al.
(2015) to evaluate whether the types of the generated images are consistent
with the given text.

Table 5: Ablation study (i.e., different $\Delta t$) of the asynchronous diffusion process on the PSNR and SSIM metrics. Although multi-stream U-Nets are also adopted in Make-A-Video Singer et al. (2023) and 3DiM Watson et al. (2023), they use synchronistic timesteps and clean conditions, respectively. Method | $\Delta t$ | PSNR$\uparrow$ | SSIM$\uparrow$
---|---|---|---
Single-stream | – | 28.56 | 0.87
Synchronistic timesteps | 0 | 30.62 | 0.92
Clean condition | 1000 | 30.68 | 0.91
Ours | 200 | 30.84 | 0.93

###  4.2 3D-consistent generation

Gallery. Figure 4 shows the visualization results of our 3DDesigner from three
aspects, i.e., fine-grained 3D generation, semantic meaningful interpolation,
and controllable generation. In particular, our model is capable of generating
very fine-grained 3D objects (e.g., different series of BMW as shown in Figure
4 (A)), showing the large capacity of our model and making it more practical
for real-world scenarios. Moreover, we also find that our model can produce
semantically meaningful interpolations as shown in Figure 4 (B), which opens a
convenient gateway to create novel car models. Last but not least, our
3DDesigner shows great performance in terms of controllability, e.g., we can
effectively change the type of cars from sedan to sports and SUV as shown in
Figure 4 (C). In addition to presenting the generated results on the RealCars
dataset, we also demonstrate the generated outcomes on the 3D-FUTURE dataset
in Figure 7. We showcase our model’s consistency in view generation, local
editing capabilities, and attribute modifications (e.g., changing the color of
chairs)

Comparison. To the best of our knowledge, we are the first to extend text-
guided diffusion models to achieve photorealistic 3D-consistent generation.
Thus there is no available previous work that can be compared directly. To
discuss the effectiveness and limits of NeRF and diffusion components, we re-
implement and extend CLIP-NeRF Wang et al. (2022) and 3DiM Watson et al.
(2023) to our task. It can be observed from Table 4 that CLIP-NeRF Wang et al.
(2022) performs best on 3D consistency, however, lacking diffusion model makes
it fail to achieve promising image quality and controllability. Adding GAN
loss may improve the visual quality, while the controllability is hard to be
improved in their design. Note that 3DiM originally cannot perform text-guided
generation, thus we integrate 3DiM with text guidance Nichol et al. (2022).
Compared to our model, the extended 3DiM: 1) lacks of the NeRF-based condition
module and 2) adopts a different two-stream diffusion process. The results
show that our model can significantly improve 3D consistency, image quality,
and controllability. Moreover, visualization results in Figure 6 also show the
superior performance of our model.

Ablation study. The comparison to extended 3DiM Watson et al. (2023) in Table
4 has shown the effectiveness of our NeRF-based condition model. Here we take
a closer look at our proposed two-stream asynchronous diffusion module, where
a cleaner condition stream is designed to guide a generation stream. The
results in Table 5 show that 1) without the two-stream architecture to learn
cross-view correspondence, the 3D consistency drops seriously; 2) without
asynchronous timesteps, the conditioned stream is as noisy as the generation
stream, leading to poor guidance; 3) conditioned on a clean image cannot well
leverage the guidance due to the large gap of noise level. Figure 10 shows
that our proposed two-stream asynchronous diffusion can generate more
consistent results.

###  4.3 3D local editing

We propose to extend the 2D editing method (i.e., noisy image blending
Avrahami et al. (2022)) to noise blending, thus the blended noise can further
be inverted into a view-independent text embedding for 3D local editing. The
motivation of such a design is that inverting noises in a specific DDIM
sampling process is much easier than inverting an image. It can be observed
from the third column of Figure 11 that obvious artifacts appear in image-to-
text inversion results. The last two columns of Figure 11 show that our
proposed method can generate high-fidelity 360∘ manipulated images with a
single view mask. We provide additional results of 3D local editing in Figure
8. A profusion of visualizations serves to underscore the broad applicability
of our methodology.

Table 6: Quantitative Evaluation of One-Shot Novel View Synthesis, comparing our approach with concurrent work based on diffusion models using PSNR and SSIM metrics. Method | PSNR $\uparrow$ | SSIM $\uparrow$
---|---|---
3DiM Watson et al. (2023) | 32.53 | 0.93
Ours | 33.14 | 0.94

###  4.4 One-shot novel view synthesis

Inspired by recent advances in fine-tuning text-guided diffusion models for
image inversion Gal et al. (2023); Kawar et al. (2023), we extend our text-
guided 3D generation model to perform novel view synthesis. Given an object
that the model has nerve seen, we fine-tune our model by adding noise to a
single-view image of the object and conduct denoising. After that, we can
obtain novel view images by changing the input camera pose. Table 6 and Figure
12 shows the qualitative and quantitative results, respectively. Figure 12
illustrates the comparative advantages of our model over 3DiM Watson et al.
(2023). It can be observed that our model can synthesis more realistic and
consistent novel view images. To comprehensively showcase our model’s
generation capabilities in one-shot novel view synthesis, we provide
additional visual results in Figure 9.

##  5 Conclusion

In this paper, we introduced 3DDesigner, a text-guided diffusion model which
is capable of generating and manipulating 3D objects. To our best knowledge,
the proposed 3DDesigner is the first text-guided generative model that can
perform 3D-consistent generation, 3D local editing, and one-shot novel view
synthesis together. Benefitting from the proposed NeRF-based Condition Module,
Two-stream Asynchronous Diffusion Module, and novel diffusion training,
sampling&blending strategies, our 3DDesigner outperforms existing methods and
achieves highly realistic, detailed, and controlled 3D object generation.

##  6 Data availability statement

The data that support the findings of this study are openly available,
including “3D-FUTUR” and “RealCars,” which are included in this published
article Fu et al. (2021) and the link:
https://pan.baidu.com/s/1d6zC3oN2pq2frd3bbnYt-Q?pwd=veeb, respectively.

## References

  * Avrahami et al. (2022) Avrahami O, Lischinski D, Fried O (2022) Blended diffusion for text-driven editing of natural images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 18208–18218
  * Bautista et al. (2022) Bautista MA, Guo P, Abnar S, Talbott W, Toshev A, Chen Z, Dinh L, Zhai S, Goh H, Ulbricht D, et al. (2022) Gaudi: A neural architect for immersive 3d scene generation. Advances in Neural Information Processing Systems 35:25102–25116
  * Chen et al. (2021) Chen A, Xu Z, Zhao F, Zhang X, Xiang F, Yu J, Su H (2021) MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 14124–14133
  * Dhariwal and Nichol (2021) Dhariwal P, Nichol A (2021) Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems 34:8780–8794
  * Fu et al. (2021) Fu H, Jia R, Gao L, Gong M, Zhao B, Maybank S, Tao D (2021) 3d-future: 3d furniture shape with texture. International Journal of Computer Vision pp 1–25
  * Gal et al. (2023) Gal R, Alaluf Y, Atzmon Y, Patashnik O, Bermano AH, Chechik G, Cohen-Or D (2023) An image is worth one word: Personalizing text-to-image generation using textual inversion. In: International Conference on Learning Representations
  * Gu et al. (2021) Gu J, Liu L, Wang P, Theobalt C (2021) StyleNeRF: A style-based 3D aware generator for high-resolution image synthesis. In: International Conference on Learning Representations
  * Heusel et al. (2017) Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) GANs trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems 30
  * Ho et al. (2020) Ho J, Jain A, Abbeel P (2020) Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33:6840–6851
  * Ho et al. (2022a) Ho J, Chan W, Saharia C, Whang J, Gao R, Gritsenko A, Kingma DP, Poole B, Norouzi M, Fleet DJ, et al. (2022a) Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:221002303
  * Ho et al. (2022b) Ho J, Salimans T, Gritsenko A, Chan W, Norouzi M, Fleet DJ (2022b) Video diffusion models. arXiv:220403458
  * Johari et al. (2022) Johari MM, Lepoittevin Y, Fleuret F (2022) GeoNeRF: Generalizing NeRF with geometry priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 18365–18375
  * Kawar et al. (2023) Kawar B, Zada S, Lang O, Tov O, Chang H, Dekel T, Mosseri I, Irani M (2023) Imagic: Text-based real image editing with diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6007–6017
  * Kim et al. (2022) Kim G, Kwon T, Ye JC (2022) Diffusionclip: Text-guided diffusion models for robust image manipulation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 2426–2435
  * Lan et al. (2023) Lan Y, Loy CC, Dai B (2023) Correspondence distillation from nerf-based gan. International Journal of Computer Vision pp 1–21
  * Lazova et al. (2023) Lazova V, Guzov V, Olszewski K, Tulyakov S, Pons-Moll G (2023) Control-nerf: Editable feature volumes for scene rendering and manipulation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp 4340–4350
  * Lugmayr et al. (2022) Lugmayr A, Danelljan M, Romero A, Yu F, Timofte R, Van Gool L (2022) Repaint: Inpainting using denoising diffusion probabilistic models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 11461–11471
  * Mildenhall et al. (2021) Mildenhall B, Srinivasan PP, Tancik M, Barron JT, Ramamoorthi R, Ng R (2021) NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65(1):99–106
  * Müller et al. (2022) Müller T, Evans A, Schied C, Keller A (2022) Instant neural graphics primitives with a multiresolution hash encoding. ACM Transaction on Graphics 41(4):102:1–102:15
  * Nichol et al. (2022) Nichol A, Dhariwal P, Ramesh A, Shyam P, Mishkin P, McGrew B, Sutskever I, Chen M (2022) Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In: International conference on machine learning, PMLR, pp 16784–16804
  * Paszke et al. (2019) Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, et al. (2019) PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems 32
  * Poole et al. (2023) Poole B, Jain A, Barron JT, Mildenhall B (2023) Dreamfusion: Text-to-3d using 2d diffusion. In: International Conference on Learning Representations
  * Radford et al. (2021) Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J, et al. (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning, PMLR, pp 8748–8763
  * Ramesh et al. (2021) Ramesh A, Pavlov M, Goh G, Gray S, Voss C, Radford A, Chen M, Sutskever I (2021) Zero-shot text-to-image generation. In: International Conference on Machine Learning, PMLR, pp 8821–8831
  * Ramesh et al. (2022) Ramesh A, Dhariwal P, Nichol A, Chu C, Chen M (2022) Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:220406125
  * Rebain et al. (2022) Rebain D, Matthews M, Yi KM, Lagun D, Tagliasacchi A (2022) LOLNeRF: Learn from one look. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 1558–1567
  * Rombach et al. (2022) Rombach R, Blattmann A, Lorenz D, Esser P, Ommer B (2022) High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 10684–10695
  * Ruiz et al. (2023) Ruiz N, Li Y, Jampani V, Pritch Y, Rubinstein M, Aberman K (2023) Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 22500–22510
  * Saharia et al. (2022) Saharia C, Chan W, Saxena S, Li L, Whang J, Denton EL, Ghasemipour K, Gontijo Lopes R, Karagol Ayan B, Salimans T, et al. (2022) Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35:36479–36494
  * Schuhmann et al. (2022) Schuhmann C, Vencu R, Beaumont R, Coombes T, Gordon C, Katta A, Kaczmarczyk R, Jitsev J (2022) Laion-5b: laion-5b: A new era of open large-scale multi-modal datasets
  * Sharma et al. (2022) Sharma P, Tewari A, Du Y, Zakharov S, Ambrus R, Gaidon A, Freeman WT, Durand F, Tenenbaum JB, Sitzmann V (2022) Seeing 3D objects in a single image via self-supervised static-dynamic disentanglement. arXiv preprint arXiv:220711232
  * Singer et al. (2023) Singer U, Polyak A, Hayes T, Yin X, An J, Zhang S, Hu Q, Yang H, Ashual O, Gafni O, et al. (2023) Make-a-video: Text-to-video generation without text-video data. In: International Conference on Learning Representations
  * Sohl-Dickstein et al. (2015) Sohl-Dickstein J, Weiss E, Maheswaranathan N, Ganguli S (2015) Deep unsupervised learning using nonequilibrium thermodynamics. In: International Conference on Machine Learning, PMLR, pp 2256–2265
  * Song et al. (2021) Song J, Meng C, Ermon S (2021) Denoising diffusion implicit models. In: International Conference on Learning Representations
  * Song and Ermon (2019) Song Y, Ermon S (2019) Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems 32
  * Villegas et al. (2023) Villegas R, Babaeizadeh M, Kindermans PJ, Moraldo H, Zhang H, Saffar MT, Castro S, Kunze J, Erhan D (2023) Phenaki: Variable length video generation from open domain textual description. In: International Conference on Learning Representations
  * Wang et al. (2022) Wang C, Chai M, He M, Chen D, Liao J (2022) CLIP-NeRF: Text-and-image driven manipulation of neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3835–3844
  * Watson et al. (2023) Watson D, Chan W, Martin-Brualla R, Ho J, Tagliasacchi A, Norouzi M (2023) Novel view synthesis with diffusion models. In: International Conference on Learning Representations
  * Yang et al. (2021) Yang B, Zhang Y, Xu Y, Li Y, Zhou H, Bao H, Zhang G, Cui Z (2021) Learning object-compositional neural radiance field for editable scene rendering. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 13779–13788
  * Yang et al. (2015) Yang L, Luo P, Change Loy C, Tang X (2015) A large-scale car dataset for fine-grained categorization and verification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3973–3981
  * Yu et al. (2021) Yu A, Ye V, Tancik M, Kanazawa A (2021) pixelnerf: Neural radiance fields from one or few images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 4578–4587
  * Yu et al. (2022) Yu J, Xu Y, Koh JY, Luong T, Baid G, Wang Z, Vasudevan V, Ku A, Yang Y, Ayan BK, et al. (2022) Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:220610789 2(3):5
  * Zhang et al. (2021) Zhang X, Srinivasan PP, Deng B, Debevec P, Freeman WT, Barron JT (2021) Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transaction on Graphics 40(6):1–18
  * Zhu et al. (2023) Zhu J, Zhu H, Zhang Q, Zhu F, Ma Z, Cao X (2023) Pyramid nerf: Frequency guided fast radiance field optimization. International Journal of Computer Vision pp 1–16


# Active Learning with Tabular Language Models

Martin Ringsquandl
Siemens AG
Munich, Germany
<EMAIL_ADDRESS>&Aneta Koleva
Siemens AG, LMU
Munich, Germany
<EMAIL_ADDRESS>
###### Abstract

Despite recent advancements in tabular language model research, real-world
applications are still challenging. In industry, there is an abundance of
tables found in spreadsheets, but acquisition of substantial amounts of labels
is expensive, since only experts can annotate the often highly technical and
domain-specific tables. Active learning could potentially reduce labeling
costs, however, so far there are no works related to active learning in
conjunction with tabular language models. In this paper we investigate
different acquisition functions in a real-world industrial tabular language
model use case for sub-cell named entity recognition. Our results show that
cell-level acquisition functions with built-in diversity can significantly
reduce the labeling effort, while enforced table diversity is detrimental. We
further see open fundamental questions concerning computational efficiency and
the perspective of human annotators.

##  1 Introduction

Recent tabular language models (TaLMs) which are based on pre-trained
transformers and modality-adapted to tabular data have shown promising results
on academic datasets [3]. However, using pre-trained transformer models for
complex tasks in industry still requires fine-tuning with substantial amounts
of labels. Although tabular data in spreadsheets is ubiquitous in industry,
the application of TaLMs in real-world settings is still marginal. One reason
is that acquiring labels for industrial spreadsheets, e.g. cell-level entity
or column annotations, quickly becomes prohibitively expensive considering
that often highly technical and domain specific language can only be annotated
by a handful of experts.

Settings with abundant unlabeled data and costly label acquisition are perfect
matches for active learning (AL), which aims to minimize the costs of
acquiring labeled data by maximizing model’s performance with each new labeled
instance. Deep neural networks, such as large-scale transformers, require
batch-based acquisition functions instead of classical one-by-one instance
selection, since these large models need to be trained for multiple epochs on
batches of labeled data to effectively change the model’s parameters for the
next iteration of candidate acquisition. Otherwise training is inefficient and
can lead to overfitting [7]. A popular approach here is to have hybrid
acquisition functions that weigh instance-wise uncertainty against some
within-batch similarity (diversity) of all instances [11].

In the text data modality, batch typically means a set of sentences or
documents. Even for a given token-level task such as named entity recognition
(NER), we still want to select the most informative sentences (not single
tokens), since all NER labels depend on the full sentence context. In AL
terms, NER is a so-called multi-instance problem [10]. Transferring this to
tabular data, a batch therefore should be a set of tables. Given a cell-level
task, where each cell may have multiple labels, this is now a nested multi-
instance problem. Since the instance is the full table, each table has
multiple cells and cells have multiple labels. Until now, the table modality
for AL has not been explored and it is unclear how to deal with such a novel
AL problem. In this paper, we study how well-known AL acquisition functions
can be adopted to TaLMs to solve an industrial sub-cell NER task where named
entities are mentioned in tables cells.

Our key contributions are the following:

  * •

We present an industrial NER use case for TaLMs in conjunction with AL.

  * •

We outline novel problems that arise with AL when dealing with tabular data.

  * •

We adopt well-known AL acquisition functions to tables and carry out an
empirical evaluation on a real-world industrial dataset.

##  2 Industrial Table NER Use Case

Our use case originates from the need for information extraction to support
data management in the process industry. Industrial plant operators have to
maintain data about plant equipment, such as actuators, sensors, vessels,
etc., processes, and ingredients among others. This data is used, for example,
for process monitoring, maintenance, or other regulatory reasons. Typically,
still today such data is collected and maintained by engineers in some form of
spreadsheets. The spreadsheets are roughly organized in a tabular format, as
shown in the example table in Figure 1. In these spreadsheets, each row
typically represents information about one or multiple equipment instances.
Some columns represent relevant physical properties of these equipment, while
others are non-informative. However, the engineers do neither comply to a
fixed schema, nor to unified spelling of equipment or properties. The NER task
is to automatically extract relevant entities for creating structured
specifications of the plant equipment. We phrase this problem as NER task
(sub-cell NER) with the following types of entities. The type TAG refers to a
systematic identifier of an equipment. There are some conventions for
generating equipment tags, but most plant operators customize them and some
sheets do not contain identifiers at all. Type EQ is for surface names of
equipment types. The type QUANT refers to the physical properties/quantities
describing the functional specifications of equipment and the type UoM stands
for unit of measurement.

Figure 1: Overview of our industrial sub-cell NER problem and the active
learning candidate acquisition.

As these tables can get quite large, e.g. hundreds of rows, labeling all cells
in a single table is already too time-consuming for human annotators. However,
table cells can contain lots of similar, repetitive content, where it may be
enough to provide small amount of labels to reach good performance. Therefore,
we are interested in AL acquisition functions that would select the most
informative instances to be labeled, but it is unclear on which level (token,
cell, table) to select instances, as shown in the bottom of Figure 1.

##  3 AL and TaLMs

In this section we give a general introduction of AL and define our TaLM for
sub-cell NER.

Given an unlabeled dataset $\mathcal{X}_{U}\sim\mathbb{P}$ sampled from some
true unknown distribution $\mathbb{P}$. The objective of pool-based AL is to
select candidates $\mathcal{X}_{S}\subset\mathcal{X}_{U}$ to be labeled with
restricted budget $m\geq|\mathcal{X}_{S}|$ and $m\ll|\mathcal{X}_{U}|$ such
that:

$\operatorname*{arg\,min}_{\mathcal{X}_{S}}\mathbb{E}_{x\sim\mathbb{P}}\left[\ell(f_{\theta}(x),y)\right],$
(1)

where $y$ is the corresponding label of instance $x$, $\ell(\cdot)$ is a loss
function and $f_{\theta}$ is a classification model trained on data
${\mathcal{X}_{S}}$ with revealed labels from an oracle. Finding the optimal
candidate set $\mathcal{X}_{S}$ w.r.t. 1 is infeasible since in AL by
definition we do not have access the true distribution nor any labels (except
for a small seed set). However, based on principles such as uncertainty
sampling AL research has proposed several heuristics to address this problem.
These heuristics are often called query strategies or acquisition functions
$f_{\text{acq}}(x_{u};f_{\theta}):\mathcal{X}_{U}\rightarrow\mathbb{R}$
assigning a score to every instance based on the current classification model
$f_{\theta}$. Typically, AL is performed in an iterative fashion where
candidates in the next iteration are sampled considering the model trained on
the previous candidates.

In our case $f_{\theta}$ is a TaLM and $\mathcal{X}_{U}$ consists of a set of
tables. We define a table as a tuple $T=(C,H)$, where
$C=\\{c_{1,1},c_{1,2},\dots c_{i,j},\dots,c_{n,m}\\}$ is the set of table body
cells for $n$ rows and $m$ columns. Every cell
$c_{i,j}=\left(w_{c_{i,j},1},w_{c_{i,j},2},\dots,w_{c_{i,j},t}\right)$ is a
sequence of tokens of length $t$. The table header
$H=\\{h_{1},h_{2},\dots,h_{m}\\}$ is the set of corresponding $m$ column
header cells, where
$h_{j}=~{}\left(w_{{h_{j}},1},w_{{h_{j}},2},\dots,w_{{h_{j}},t}\right)$ is a
sequence of header tokens.

Our TaLM consists of an encoder which will produce a table context-sensitive
representation for each token:

$\mathbf{w}_{h_{1},1},\dots,\mathbf{w}_{h_{m},t},\mathbf{w}_{c_{1,1},1},\dots,\mathbf{w}_{c_{n,m},t}=\textsc{Enc}(T),$
(2)

where Enc is a table-biased transformer encoder using the row-column
visibility matrix instead of vanilla attention and only within-cell positional
encoding, refer to [6] for a more detailed description of this architecture.
Note that most TaLM architectures like TURL [2] are not directly applicable
here, since they already aggregate tokens within cells to get cell-level
representations. This is why we resort to this TaLM based on a pre-trained
language model and modality adapt it to tables while fine-tuning for the sub-
cell NER task.

Each labeled cell has an NER-tag sequence: $(y_{1},y_{2},\dots,y_{t})$, where
each $y_{i}\in\mathcal{Y}$. We use IO tags, thus $\mathcal{Y}$ is
$\\{\textit{O}\\}\cup\\{\textit{I-ENT}\\}$, where
$\textit{ENT}\in\\{\textit{TAG},\textit{EQ},\textit{QUANT},\textit{UoM}\\}$.
Our TaLM decoder Dec is a classification layer that projects each token
representation into the label space, plus a Softmax activation to assign a
normalized score for each class.

$\hat{\mathbf{y}}_{i,j,z}=f_{\theta}(T)=\textsc{Dec}(\textsc{Enc}(T)),$ (3)

where $\hat{\mathbf{y}}_{i,j,z}$ is the probability distribution of the labels
of the $z$-th token in header or body cell $i,j$.

##  4 Table NER Acquisition Function

As mentioned above, AL is primarily concerned with the choice of an
acquisition function. This function assigns a score to each instance, thereby
determining which new (unlabeled) instances to select for labeling such that
they maximize the informativeness to the current model.

For single-instance tasks, such as text classification, the design of
acquisition functions is straightforward - every instance has one label - so
select the instance $x$ with the most informative label distribution
$\hat{\mathbf{y}}$. For example, the popular entropy-based
$f_{\text{acq}}^{\text{ent}}(x_{U};f_{\theta})=\mathbb{H}\left[\hat{\mathbf{y}}\mid
x_{U};f_{\theta}\right]$, simply ranks every $x_{U}$ based on the entropy
$\mathbb{H}$ of the label distribution.

In multi-instance tasks such as NER there might be multiple labels (entity
spans) in a single sentence instance. Therefore, we want to select the
instance $x_{U}$ where the acquisition function considers the joint label
distributions
$\hat{\mathbf{y}}_{1},\hat{\mathbf{y}}_{2},\dots,\hat{\mathbf{y}}_{t}$. In our
table NER case, we have a nested multi-instance problem. Every table instance
$T_{k}$ has multiple cells, for example body cells $c_{k,i,j}$, and every such
cell has a sequence of label distributions
$(\hat{\mathbf{y}}_{k,i,j,1},\hat{\mathbf{y}}_{k,i,j,2},\dots\hat{\mathbf{y}}_{k,i,j,t})$
associated with the $t$ tokens in the cell.

One could argue that in our table NER case, the acquisition function should
select whole tables with most informative set of cells. Unfortunately, it is
not reasonable to assume that a human annotator would have the time to
annotate every cell in a full table. Hence, since a single table would already
break the annotation budget, we need an acquisition function on cell-level.
One possibility is to do this hierarchically: first select a set of tables
using $f_{\text{acq}}(T;f_{\theta})$ (e.g. by clustering table
representations), then from this set of tables select a set of cells
$f_{\text{acq}}(c_{i,j};f_{\theta})$ (e.g. by well-known multi-instance
acquisition functions).

On the other hand, note that in our TaLM for NER $f_{\theta}(T)$ all cells
$c_{i,j}$ in table $T$ are informed and therefore depend on each other. Based
on this we argue that an acquisition function
$f_{\text{acq}}(c_{i,j};f_{\theta})$ on cell-level is enough since it already
captures the context of the full table $T$. In the following we adopt common
acquisition functions to select batches of most informative cells. We consider
different types of acquisition functions in our experiments - pure
uncertainty-based, real-world diversity and batch diversity.

### MNLP

A popular acquisition function for uncertainty-sampling in multi-instance
problems is the maximized normalized log-probability (MNLP) [9]. It considers
the probability of the model’s most likely label sequence to select new
instances (here cells) $c_{i,j}$ with mean normalization to account for the
bias towards longer sequences. Since the most likely label sequence in our
TaLM is simply the sequence of every most likely token label we have:

$\displaystyle
f_{\text{acq}}^{\text{MNLP}}(c_{i,j};f_{\theta})=\frac{1}{t}\sum_{z=1}^{t}\text{log}\text{
}y^{*}_{i,j,z},$ (4)

where $y^{*}_{i,j,z}$ is the probability of the class with the maximum
probability in the predicted label distribution. The sums of log-probabilities
are negative, hence the candidates are selected in ascending order w.r.t. to
4. MNLP has a major weakness in the batch-based AL setting. The sequences with
highest uncertainty for the current model tend to be highly correlated.
Therefore, when selecting a batch of candidates, there will be substantial
similarity within the batch, providing little additional value. For tables
this means that most candidates will likely come from the same table or even
the same row/column.

### MNLP+

To make MNLP less prone to selection of similar cells, we introduce a simple
procedure considering real-world diversity in the tables. Instead of selecting
the cells with the highest uncertainty from the set of all tables, we first
sort all cells within every table w.r.t. to 4. Then we employ a round-robin
selection across tables by taking the current table’s best available candidate
cell and moving to the next table until the budget is empty. In other words,
this acquisition is a combination of cell uncertainty with a maximum table-
diversity constraint.

### BADGE

As a batch-diverse acquisition function we consider Batch Active learning by
Diverse Gradient Embeddings (BADGE) [1]. Here, the authors propose to measure
uncertainty in the magnitude of the model’s last layer gradients. A diverse
set of candidates is then selected using the $k$-means++ initialization to
sample from the gradient embeddings. In our case, we take the mean of each
cell’s token gradient embeddings and feed this into $k$-means++ to get cell
candidates:

$\displaystyle
f_{\text{acq}}^{\text{BADGE}}(c_{i,j};f_{\theta})=k\text{-}\textsc{means}\text{++}(\frac{1}{t}\sum_{z}\mathbf{g}_{i,j,z}),$
(5)

where
$\mathbf{g}_{i,j,z}=\frac{\partial}{\partial\theta^{\text{out}}}\ell_{\text{CE}}(\hat{\mathbf{y}}_{i,j,z},y^{*}_{i,j,z})$
is the gradient of the cross-entropy loss $\ell_{\text{CE}}$ of the $z$-th
token label distribution and its predicted maximum label $y^{*}_{i,j,z}$
w.r.t. the decoder classification layer weights $\theta^{\text{out}}$.

### Rand

As a simple baseline we also employ an acquisition function that selects
candidate cells uniformly at random from the set of all cells in all available
tables.

##  5 Experiments

In this section we describe the experiments on our real-world industrial
dataset.

### Dataset

The dataset contains spreadsheets from multiple industrial plants. The rows in
the original sheets were downsampled, such that each resulting table had a
maximum of 5 rows. Expert annotators were then asked to label each table cell-
by-cell using the prodi.gy span-based NER annotation tool.

Table 1: Training tables statistics Statistic | Value
---|---
#tables | 55
#cells | 4,774
#labels | 1,112
labels per cell | 0.23

After performing a random train-test split, we end up with 55 tables in the
training set and 24 in the test set. The training set statistics are shown in
Table 1. We can see that in total they contain $\sim$ 1,1k NER span labels.
Dividing this by the number of cells in all tables, we observe that the vast
majority $\sim$ 77% of cells are only comprised of tokens with label O. This
is an extreme class-imbalance - typical for industry settings. Note that
Figure 1 is not representative of all tables, i.e. there are some with much
less densely populated NER labels.

### AL Setup

For every AL acquisition function, a single experiment starts with a set of
100 labeled cell candidates $\mathcal{X}_{L}$ as seed to initially train our
NER TaLM (iteration 0). Since tasks with high class-imbalance have strong
dependence on this initial seed, we make sure that it is stratified w.r.t. the
class distribution in the overall training set. This should mitigate the risk
of completely missing minority classes in the initial seed.

(a) F1-score at each iteration averaged over 5 seeds

(b) Number of tables sampled from at each iteration

Figure 2: Results of different cell acquisition functions

After seed selection, the AL acquisition function selects a batch
$\mathcal{X}_{S}$ of 50 unlabeled instances (cells) from $\mathcal{X}_{U}$
that are added to $\mathcal{X}_{L}$ along with their true labels in each
iteration. Following the suggested protocol from [4], we fine-tune in each
iteration from scratch to avoid overfitting data from previous rounds. In each
iteration we train for a maximum of 30 epochs and use a small portion of the
test set as validation for early stopping. In each experiment, all acquisition
functions start with the same initial seed. The reported results are the
average over 5 different experiments, i.e., 5 different initial seeds. As pre-
trained language model we use the ’bert-base-uncased’ from huggingface.

After every experiment we evaluate the model’s final performance against the
hold-out test set. As a ceiling benchmark we also train our NER TaLM on the
full training dataset. As performance metric we chose the micro-averaged
F1-score.

##  6 Results and Discussion

The test set performance throughout iterations is shown in Figure 2(a). BADGE
is the only acquisition function that clearly outperforms Rand and even
exceeds full training performance. Interestingly, MNLP has only slight
advantages over Rand after around 500 labels. MNLP+ is by far the worst
meaning that the forced maximum table diversity is too extreme and detrimental
to performance. We also notice higher variance for MNLP which suggests that it
is more dependent on the initial seed labels, which is a known issue in these
high-imbalance settings [10]. BADGE also seems to alleviate this problem to
some degree looking more robust across different seeds.

In terms of diversity of samples we show the number of different tables from
which each acquisition function selects cells in each iteration in Figure
2(b). This metric shows that MNLP suffers from selecting similar cells from
the same table with only slightly increasing number of tables. MNLP+ is by
design selecting from as many tables as the budget allows. Random sampling
only samples from roughly half the tables. This is due to an imbalance in
table sizes - some large ones dominating the uniform sampling procedure. BADGE
seems to strike a good balance between table diversity and cell uncertainty,
sitting between MNLP and Rand in the number of tables.

Also note that the full training performance is lower compared to the original
work [6], since we do not use data augmentation here. Although combining AL
with data augmentation looks like a promising direction [5].

### Computational Efficiency

Apart from pure classification performance, we may also consider computational
efficiency. As an example, in a cell-level task, an acquisition function that
retrieves a set of $k$ candidate cells from a single table versus $k$ cells
which span over $k$ tables (one cell per table) will perform much fewer
computations. Especially, for large tables we may want our acquisition to
balance the informativeness of instances versus the number of tables that fit
in a batch compute budget. Alternatively, we want the acquisition to select
the most informative sub-tables rows (context), such that large tables will
not impact the compute limits too much. Since TaLMs have a high memory
footprint, a very table-diverse acquisition function may be much more
expensive than others.

### Human Annotators

On the other hand, a table-diverse acquisition function may also impact human
annotators. When presented with many different tables, the annotator probably
needs more time per cell annotation, because every table’s unique schema and
context needs to be interpreted. This again raises the question if the full
table needs to be shown to the user to make an informed labeling decision or a
sub-table is sufficient.

##  7 Related Work

Pre-trained large-scale language models like BERT have been studied in AL
setups for different tasks, e.g. text classification [8]. While these models
can already show good results when fine-tuning on small labeled datasets, in
industry AL is still often necessary as argued in [4], where an empirical
study of AL for BERT in text classification was carried out. They consider the
challenging, practical scenario with high class-imbalance and small annotation
budget. Their results show that batch-diverse acquisitions handle this
scenario better than classical ones.

For NER deep conditional random field models have been studied with AL [9]
with the focus on computational efficiency, due to expensive training. Further
the missed-class effect due to the AL’s exploitative acquisition has been
studied in NER [10]. The authors propose acquisitions for more informed seed
instance selection that can alleviate this problem.

While all of these works are relevant to ours, it is non-trivial to transfer
their findings to AL on tables using TaLMs.

##  8 Conclusion

In this paper we describe an industrial sub-cell NER use case for table
language models and active learning. We define this novel nested multi-
instance AL problem and adopt some well-known acquisition functions to select
cell candidates from unlabeled tables. Our empirical results based on a real-
world dataset show that these acquisition functions can be applied to TaLMs
and outperform random selection. We find that balancing table diversity
against label uncertainty is crucial and more advanced acquisition functions
specifically for TaLMs and tables may be needed for other tasks. Since our
work is the first to address this novel problem setting, there are still some
open questions for future work regarding computational efficiency and impact
on human annotators.

## References

  * [1] Jordan T. Ash et al.  “Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds”  In _ArXiv_ abs/1906.03671, 2020
  * [2] Xiang Deng et al.  “TURL: Table Understanding through Representation Learning”  In _Proc. VLDB Endow._ 14.3 VLDB Endowment, 2020, pp. 307–319
  * [3] Haoyu Dong et al.  “Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks”  In _IJCAI’2022 SURVEY TRACK_ , 2022
  * [4] Liat Ein-Dor et al.  “Active Learning for BERT: An Empirical Study”  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ Association for Computational Linguistics, 2020, pp. 7949–7962
  * [5] Yoon-Yeong Kim, Kyungwoo Song, JoonHo Jang and Il-chul Moon  “LADA: Look-Ahead Data Acquisition via Augmentation for Deep Active Learning”  In _Advances in Neural Information Processing Systems_ 34, 2021, pp. 22919–22930
  * [6] Aneta Koleva et al.  “Named Entity Recognition in Industrial Tables using Tabular Language Models”  In _ArXiv_ 2209.14812, 2022
  * [7] Pengzhen Ren et al.  “A Survey of Deep Active Learning”  In _ACM Comput. Surv._ 54.9 ACM, 2021
  * [8] Christopher Schröder, Andreas Niekler and Martin Potthast  “Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers”  In _60th Annual Meeting of the Association for Computational Linguistics: Findings (ACL 2022)_ Association for Computational Linguistics, 2022
  * [9] Yanyao Shen et al.  “Deep Active Learning for Named Entity Recognition”  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_ Vancouver, Canada: Association for Computational Linguistics, 2017, pp. 252–256
  * [10] Katrin Tomanek, Florian Laws, Udo Hahn and Hinrich Schütze  “On Proper Unit Selection in Active Learning: Co-Selection Effects for Named Entity Recognition”  In _Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing_ Association for Computational Linguistics, 2009, pp. 9–17
  * [11] Fedor Zhdanov  “Diverse mini-batch active learning”  In _arXiv preprint arXiv:1901.05954_ , 2019