# 3DRealCar: An In-the-wild RGB-D Car Dataset with 360-degree Views

Xiaobiao Du 1,2,3 Haiyang Sun 3 Shuyun Wang 1 Zhuojie Wu 1 Hongwei Sheng 1
Jiaying Ying 1 Ming Lu 5 Tianqing Zhu 4 Kun Zhan 3 Xin Yu1

1 The University of Queensland 2 University of Technology Sydney 3 Li Auto
4 City University of Macau 5 Peking University
Corresponding author.

###### Abstract

3D cars are commonly used in self-driving systems, virtual/augmented reality,
and games. However, existing 3D car datasets are either synthetic or low-
quality, presenting a significant gap toward the high-quality real-world 3D
car datasets and limiting their applications in practical scenarios. In this
paper, we propose the first large-scale 3D real car dataset, termed 3DRealCar,
offering three distinctive features. (1) High-Volume: 2,500 cars are
meticulously scanned by 3D scanners, obtaining car images and point clouds
with real-world dimensions; (2) High-Quality: Each car is captured in an
average of 200 dense, high-resolution 360-degree RGB-D views, enabling high-
fidelity 3D reconstruction; (3) High-Diversity: The dataset contains various
cars from over 100 brands, collected under three distinct lighting conditions,
including reflective, standard, and dark. Additionally, we offer detailed car
parsing maps for each instance to promote research in car parsing tasks.
Moreover, we remove background point clouds and standardize the car
orientation to a unified axis for the reconstruction only on cars without
background and controllable rendering. We benchmark 3D reconstruction results
with state-of-the-art methods across each lighting condition in 3DRealCar.
Extensive experiments demonstrate that the standard lighting condition part of
3DRealCar can be used to produce a large number of high-quality 3D cars,
improving various 2D and 3D tasks related to cars. Notably, our dataset brings
insight into the fact that recent 3D reconstruction methods face challenges in
reconstructing high-quality 3D cars under reflective and dark lighting
conditions. Our dataset is available here.

Figure 1: Characteristics of our curated high-quality 3DRealCar dataset. 3DRealCar contains detailed annotations for various colors, car types, brands, and even car parsing maps. In particular, our dataset contains three lighting conditions on car surfaces, bringing challenges to existing methods. Table 1: The comparison of 3D car datasets. Lighting means the lighting conditions of the surfaces of cars. Point Cloud represents the point clouds with actual sizes in real-world scenes.  Dataset | Instances | Type | Views | Resolution | Brand | Lighting | Car Parsing | Depth | Point Cloud
---|---|---|---|---|---|---|---|---|---
SRN-Car | 2151 | Synthetic | 250 | 128$\times$128 | ✗ | ✗ | ✗ | ✗ | ✗
Objaverse-car | 511 | Synthetic | - | - | ✗ | ✗ | ✗ | ✗ | ✗
MVMC | 576 | Real |  $\sim$10 | 600$\times$450 |  $\sim$40 | ✗ | ✗ | ✗ | ✗
3DRealCar (Ours) | 2500 | Real |  $\sim$200 | 1920$\times$1440 | 100+ | 3 | 13 | ✓ | ✓
Figure 2: Visual comparisons of 3D car datasets and the results of a 3D
generative method. Our 3DRealCar is captured in real-world scenes and contains
more densely captured views. In addition, our dataset has annotations for
three different lighting conditions on the car surface. We also compare a
recent state-of-the-art text-to-3D model, MVDream [57] with a prompt “a modern
sedan”, demonstrating its failure to generate high-quality 3D car models.

##  1 Introduction

Cars, as both daily objects and vehicles, are of significant interest to
researchers, especially in the field of autonomous driving. Autonomous
perception systems are typically trained on daily scene datasets that are
collected frequently. However, these datasets often exhibit long-tailed
distributions, with far fewer instances of corner-case scenarios, like car
accidents. Consequently, this imbalance leads to the autonomous perception
system generalizing well in the most frequently occurring scenes. This means
that the system is likely to perform poorly in rare situations, posing
significant safety risks to drivers. To build a reliable system, it is
essential to have a simulator that can simulate photorealistic hazardous
scenes. Moreover, high-quality 3D cars are necessary for a realistic
simulator.

Recent 3D car reconstruction methods [67, 77, 72] mainly reconstruct cars from
self-driving datasets [61, 7, 21]. To apply reconstructed cars to real-world
scenes, the reconstructed 3D car should be high-quality. However, it is very
challenging to obtain such high-quality 3D cars for the following reasons: (1)
Previous 3D car reconstruction methods produce low-quality 3D cars, primarily
because they train on self-driving datasets with low-resolution car images and
a limited number of trainable views. (2) Manually crafting a high-quality 3D
car model requires specialized artists, which is time-consuming. (3) There is
no large-scale 3D real car dataset that can be utilized to produce a bulk of
3D cars.

Moreover, existing 3D car datasets are either synthetic or only contain a few
posed images, as shown in Figure 2. SRN-Car [8] and Objaverse-Car [16] collect
3D car computer-aided design (CAD) models from the Internet, but these models
are synthetic and contain non-photorealistic texture. Although MVMC [74] is a
real car dataset, it collects only ten views on average for each car. On the
contrary, our collected 3DRealCar dataset provides an average of 200 dense
RGB-D views per car for high-quality 3D car reconstruction.

We also show that the recent state-of-the-art 3D generative method, MVDream
[57], as depicted in Figure 2, fails to generate high-quality cars due to the
multi-view inconsistency introduced by generative models [54, 2, 39, 60].
Thus, the existing 3D generation methods cannot be employed to generate high-
quality 3D real car assets.

In this work, we collect a large-scale 3D real car dataset in the wild, termed
3DRealCar, which contains dense high-quality views and rich diversity. During
data collection, we employ 3D scanners to scan cars parked on roadsides or
parking lots, obtaining posed RGB-D images and point clouds of cars. In
particular, we scan around the cars in three loops to obtain dense views. Note
that we collect car data with the consent of owners. In Table 1 and Figure 1,
we show our dataset possesses striking characteristics compared with previous
3D car datasets. We capture dense RGB-D images in high resolution, which
promotes the reconstruction of high-quality 3D cars. Furthermore, we scan cars
under three different lighting conditions, resulting in the surfaces of cars
having different lighting effects, such as reflective, standard, and dark,
where we denote the standard as the smooth lighting condition without obvious
specular highlight. Figure 2 shows some examples of three lighting conditions
in our dataset. Note that the number of instances in our dataset is the
largest in existing datasets. Therefore, our collected 3DRealCar dataset has a
rich diversity in terms of car types, colors, brands, and lighting conditions.
We also provide car parsing map annotations with thirteen classes for each
instance, which enable our dataset to be applied in car component
understanding tasks.

To construct a high-quality dataset, we filter out the images that are out of
focus, occluded, or blurred. To facilitate the 3D reconstruction solely on
cars, we remove the point clouds of the background. We also adjust the
orientation of the car facing along the x-axis before the reconstruction for
controllable rendering. Based on the high-quality posed RGB-D images, point
clouds, and multi-grained annotations, we can apply the dataset to various
tasks related to cars. Figure 3 shows our dataset supports over 10 tasks,
including several popular 2D and 3D tasks to promote the advancement of car-
related research.

We leverage existing state-of-the-art methods to benchmark 3D reconstruction
and car parsing tasks of our 3DRealCar dataset. We also conduct extensive
experiments to demonstrate that the reflective and dark lighting conditions in
our dataset are challenging to existing methods, which brings a new challenge
for 3D reconstruction in awful lighting conditions. Furthermore, we
demonstrate that our 3DealCar dataset can bring real-car prior and enhance
existing 3D generation and downstream methods. Overall, the contributions of
this work can be summarized below:

$\bullet$ We propose the first large-scale 3D real car dataset, named
3DRealCar, which contains 2,500 car instances and their point clouds with
actual sizes in real-world scenes.

$\bullet$ 3DRealCar contains RGB-D images and point clouds with detailed
annotations, supporting researchers to investigate various tasks in both 2D
and 3D.

$\bullet$ We conduct 3D reconstruction and car parsing benchmarks to advance
car-related tasks. Notably, we observe that existing methods face challenges
under the extreme lighting conditions of 3DRealCar.

$\bullet$ Extensive experiments demonstrate our 3DRealCar dataset can enhance
real-car prior and improve the performance of existing 3D generation and novel
view synthesis methods.

Figure 3: The applicable tasks of our dataset. Our proposed 3DRealCar dataset
containing RGB-D images and point clouds can be applied to many popular 2D and
3D tasks.

##  2 Related Work

3D Car Datasets. There are several well-known large-scale autonomous driving
datasets so far, such as Nuscenes [7], KITTI [21], Waymo [61], Pandaset [70],
ApolloScape [25], and Cityscape [14]. These datasets are captured by multi-
view cameras and lidars mounted on ego cars. Various works [67, 20, 41, 72]
attempt to reconstruct 3D cars in these datasets. However, these methods fall
short of reconstructing high-quality 3D cars due to the lack of sufficient and
dense training views. SRN-Car [8] and Objaverse [16] collect 3D car models
from existing repositories and Internet sources. However, these datasets only
contain synthetic cars, which cannot produce realistic textures and geometry.
MVMC [74] is collected from car advertising websites, which contain a series
of car images, especially multi-view images of each car. However, the views of
images per car in MVMC are unposed and sparse, which is adverse to
reconstructing high-quality 3D car models. In this paper, we collect a high-
quality 3D real car dataset to fill the above gaps.

3D Reconstruction with Neural Field. 3D reconstruction aims to create a 3D
structure digital representation of an object or a scene from its multi-view
images, which is a long-standing task in computer vision. One of the most
representative works in 3D reconstruction is Neural Radiance Fields (NeRFs)
[46], which demonstrates promising performance for novel view synthesis.
Afterward, this method inspires a new wave of 3D reconstruction methods using
the volume rendering method, with subsequent works focusing on improving its
quality [65, 3, 4, 22, 58, 12, 68, 6], efficiency [18, 47, 53, 59, 30, 9, 19],
applying artistic effects [17, 66, 26, 75], and generalizing to unseen scenes
[73, 10, 69, 28, 62, 13]. Particularly, Kilonerf [53] accelerates the training
process of NeRF by dividing a large MLP into thousands of tiny MLPs.
Furthermore, Mip-NeRF [3] proposes a conical frustum rather than a single ray
to ameliorate aliasing. Mip-NeRF 360 [5] further improves the application
scenes of NeRF to the unbounded scenes. Although these NeRF-based methods
demonstrate powerful performance on various datasets, the training time always
requires several hours even one day more. Instant-NGP [47] uses a multi-
resolution hash encoding method, which reduces the training time by a large
margin. 3DGS [30] proposes a new representation based on 3D Gaussian
Splatting, which reaches real-time rendering for objects or unbounded scenes.
2DGS [24] proposes a perspective-accurate 2D splatting process that leverages
ray-splat intersection and rasterization to further enhance the quality of the
reconstructions. Scaffold-GS [43] proposes an anchor growing and pruning
strategy to accelerate the scene coverage, which effectively reduces redundant
Gaussians and improves rendering quality. However, there is not yet a large-
scale 3D real car dataset so far. Therefore, we present a 3D real car dataset,
named 3DRealCar in this work.

3D Generation with Diffusion Prior. Some current works [29, 49] leverage a 3D
diffusion model to learn the representation of 3D structure. However, these
methods lack generalization ability due to the scarcity of 3D data. To
facilitate 3D generation without direct supervision of 3D data, image or
multi-view diffusion models are often used to guide the 3D creation process.
Notable approaches like DreamFusion [50] and subsequent works [45, 36] use an
existing image diffusion model as a scoring function, applying Score
Distillation Sampling SDS loss to generate 3D objects from textual
descriptions. These methods, however, suffer from issues such as the Janus
problem [50, 45] and overly saturated textures. Inspired by Zero123 [39],
several recent works [2, 56, 42, 34, 76, 44, 38, 37, 52] refine image or video
diffusion models to better guide the 3D generation by producing more reliable
multi-view images. However, these generative methods fail to generate high-
quality cars due to the lack of the prior of real cars.

Figure 4: Illustration of our data collection and preprocessing. We first
circle a car three times while scanning the car with a 3D scanner for the
attainment of RGB-D images and its point clouds. Then we use Colmap [55] and
SAM [32] to obtain poses and remove the background point clouds. Finally, we
use the 3DGS [31] trained on the processed data to obtain 3D car model.

##  3 Proposed 3DRealCar Dataset

###  3.1 Data Collection and Annotation

As shown in Figure 4, our dataset is collected using smartphones, specifically
iPhone 14 models, equipped with a 3D scanner application to scan cars for
their point clouds and RGB-D images. The data collection process is conducted
under three distinct lighting conditions, such as standard, reflective, and
dark. These lighting conditions represent the lighting states of vehicle
surfaces. It is important to note that all data collection is performed with
the consent of owners. During the scanning process, the car should be
stationary while we meticulously circle the car three times to capture as many
views as possible. For each loop, we adjust the height of the smartphone to
obtain images from different angles. Furthermore, we try our best to make sure
captured images contain the entire car body without truncation. To preserve
the privacy of owners, we make license plates and other private information
obfuscated. To construct a high-quality dataset, we filter out some instances
with blurred, out-of-focus, and occluded images. We also provide detailed
annotations for car brands, types, and colors. Particularly, we provide the
car parsing maps for each car with thirteen classes in our dataset as shown in
Figure 1 for the advancement of car component understanding tasks.

###  3.2 Data Preprocessing

Background Removal. Since we only reconstruct cars for the 3D car
reconstruction task, the background should be removed. Recent Segment Anything
Model (SAM) [32] demonstrates powerful context recognition and segmentation
performance. However, SAM needs a bounding box, text, or point as a driving
factor for accurate segmentation. Therefore, we employ Grounding DINO [40] as
a text-driven detector with a detection prompt with “car” for the attainment
of car bounding boxes. With these bounding boxes, we use SAM to obtain the
masks from captured images. The point cloud initialization is demonstrated
useful for the convergence of 3D Gaussian Splatting [31]. Except for the
removal of the background in 2D images, we still need to remove the background
point clouds. Therefore, we first project the 3D point clouds into 2D space
with camera parameters. Then, we can eliminate background point clouds with
masks and save them for further processing.

Orientation Rectification. As shown in Figure 4, we utilize Colmap [55] to
reconstruct more dense point clouds and obtain accurate camera poses and
intrinsics because we find that the estimated poses by the 3D scanner are not
accurate. However, after the removal of the background point clouds, we find
that the car orientation of the point cloud is random, which leads to the
subsequent render task being uncontrollable. Given camera poses
$P=\\{p_{i}\\}_{1}^{\mathcal{N}}$, where $\mathcal{N}$ is the number of poses,
we use Principal Component Analysis (PCA) [1] to obtain a PCA component
$\mathcal{T}\in\mathbb{R}^{3\times 3}$. The PCA component is the principal
axis of the data in 3D space, which represents rotation angles to each axis.
Therefore, we leverage it to rectify the postures of cars parallel to the
x-axis. However, this process cannot guarantee cars facing along the x-axis.
Therefore, in some failure cases, we manually interfere and adjust the
orientation along the x-axis. With the fixed car orientation, we can control
rendered poses for the subsequent tasks.

Point Cloud Rescaling. The size of the point clouds reconstructed by Colmap
[55] does not match the real-world size, which inhibits the reconstruction of
a practically sized 3D car. To address this, we calculate the bounding box of
the scanned foreground point clouds to obtain its actual size in the real-
world scene. Then, we rescale the rectified point clouds into the real size.
In addition to the rescaling of the point clouds, we also need to adjust the
camera poses. We rescale translations of camera poses using a scale factor
calculated by the ratio of scanned point cloud size and Colmap point cloud
size. After these rescaling processes, we use rescaled point clouds to
reconstruct a 3D car model through recent state-of-the-art methods, like 3DGS
[31].

Figure 5: The distributions of our 3DRealCar dataset. We show distributions of
car types, lighting conditions, captured views, car colors, and car size. We
try our best to capture cars with various colors and types for the diversity
of our dataset.

###  3.3 Data Statistics

In our 3DRealCar, we provide detailed annotations for researchers to leverage
our dataset for different tasks. During the data annotating, we discard the
data with the number of views less than fifty. As we can observe in Figure 1
and 2, we collect our dataset under real-world scenes and meticulously scan
dense views. Therefore, cars in our dataset possess dense views and realistic
texture, which is necessary for the application in a real-world setting.

As shown in Figure 5, we conduct detailed statistical analysis to show the
features of our dataset. Our dataset mainly contains six different car types,
such as Sedan, SUV, MPV, Van, Lorry, and Sports Car. Among them, sedans and
SUVs are common to collect in real life, so their volume dominates in our
dataset. We also count the number of different lighting conditions on cars.
The standard condition means the car is well-lighting and without strong
specular highlights. The reflective condition means the car has strong
specular highlights. Glossy materials bring huge challenges to recent 3D
reconstruction methods. The dark condition means the car is captured in an
underground parking so not well-lighting. To promote high-quality
reconstruction, we save the captured images in high resolution
(1920$\times$1440) and also capture as many views as possible. The number of
captured images per car is an average of 200. The number of views ranges from
50 to 400. To enrich the diversity of our dataset, we try our best to collect
as many different colors as possible. Therefore, our dataset contains more
than twenty colors, but the white and black colors still take up most of our
dataset. In addition, we also show the distribution of car size, in terms of
their length, width, and height. We obtain their sizes by computing the
bounding boxes of the scanned point clouds. Thanks to different car types, the
sizes of cars are also diverse.

##  4 Overview of 3DRealCar Tasks

###  4.1 2D tasks

Corner-case scene 2D Detection [64]: Given a serial of images
$I=\\{I_{i}\\}_{1}^{\mathcal{N}}$, this task aims to detect vehicles as
accurately as possible. However, in some corner cases, like car accidents,
detectors sometimes fail to detect target vehicles since this kind of scene is
rare or not in the training set.

2D Car Parsing [11, 23, 71, 33]: Given a serial of images
$I=\\{I_{i}\\}_{1}^{\mathcal{N}}$ and their car parsing maps
$S=\\{S_{i}\\}_{1}^{\mathcal{N}}$, this task aims to segment car components.
With annotated parsing maps, we can train a model to understand and segment
each component of cars, which is very important in self-driving.

###  4.2 3D Tasks

3D Reconstruction [48, 31, 24]: Given a serial of images
$I=\\{I_{i}\\}_{1}^{\mathcal{N}}$ and matched poses
$P=\\{p_{i}\\}_{1}^{\mathcal{N}}$, where $\mathcal{N}$ is the number of images
and poses, the task of 3D reconstruction aims to reconstruct 3D model of a
object or a scene. The reconstructed 3D model is usually used to render 2D
images with different views for the evaluation of the performance of the 3D
model.

Novel View Synthesis [39, 42, 2]: Given a serial of reference images
$I^{ref}=\\{I_{i}^{ref}\\}_{1}^{\mathcal{N}}$, reference poses
$P^{ref}=\\{p_{i}^{ref}\\}_{1}^{\mathcal{N}}$, target images
$I^{target}=\\{I_{i}^{target}\\}_{1}^{\mathcal{N}}$, and target poses
$P^{target}=\\{p_{i}^{target}\\}_{1}^{\mathcal{N}}$, recent 3D generative
models, such as Zero123 [39], Syncdreamer [42], and Stable-Zero123 [2], take
relative poses and reference images as inputs and generate target images.
However, these models cannot generalize well to real car objects since they
are trained on large-scale synthetic datasets [16, 15]. In this work, we will
demonstrate that our dataset can improve the robustness of these generative
models to real cars.

3D Generation [51, 60, 63]: Given a text prompt or single image, recent 3D
generation methods generate 3D objects with Score Distillation Sampling (SDS)
[51] and diffusion generative models [54, 39, 2]. However, these methods
cannot generate high-quality 3D cars due to the lack of the prior of real cars
in 3D-based diffusion models. Therefore, we would demonstrate the value of our
dataset by improving recent 3D generation for real cars.

Figure 6: The simulated corner-case scenes. We use a red rectangle to
highlight the simulated vehicles. These corner-case scenes show some vehicles
have potential risks to traffic safety.

Table 2: Benchmark results on 2D car parsing of our 3DRealCar dataset. We use recent advanced image segmentation methods [11, 23, 71, 33] to benchmark our dataset. Method | DeepLabV3 | PointRend | DDRNet | SegFormer
---|---|---|---|---
mIOU $\uparrow$ | 0.556 | 0.562 | 0.603 | 0.613
mAcc $\uparrow$ | 0.616 | 0.619 | 0.659 | 0.663

Table 3: Quantitative comparisons of Dreamcraft3D [60] and its improved version by trained on our dataset.  CD denotes Chamfer Distance.  Method |  CLIP-I $\uparrow$ |  Hausdorff $\downarrow$ |  CD $\downarrow$
---|---|---|---
Dreamcraft3D | 0.812 | 1.572 | 0.587
+our dataset | 0.847 | 1.364 | 0.371

##  5 Experiments

###  5.1 Experimental Setup

Corner-case 2D Detection.  In this task, we leverage the reconstructed cars to
simulate rare and corner-case scenes. To be specific, we use Nuscenes [7] as
background to simulate corner-case scenes with reconstructed cars and leverage
recent popular detectors, like YOLOv8 [64], as detectors for evaluation. To
evaluate the robustness of detectors in corner-case scenes, we use the test
part of the corner-case dataset, CODA [35] as a testing set. Since we focus on
the corner-case scenes of cars, so we only evaluate a car class.

2D Car Parsing. In this task, we utilize DeepLabV3[11], DDRNet [23], SegFormer
[71], and PointRend [33] to benchmark our dataset. To be specific, we split
80% of our car parsing maps in 3DRealCar as the training set and the rest of
20% as the testing set.

3D Reconstruction. In this task, we randomly choose 100 instances from each
lighting condition in our dataset and split 80% of the views per instance as
the training set and the rest of 20% as the testing set. Specifically, we
employ Instant-NGP [48], 3DGS [31], GaussianShader [27], and 2DGS [24] to
benchmark our dataset.

Novel View Synthesis. We finetune Zero123-XL [39] on our 3DRealCar dataset to
enhance its generalization to real cars. Note that since the training of
diffusion-based models needs entire objects centered on images, we use the
images rendered by our trained 3D models as training images.

3D Generation. In this task, we exploit Dreamcraft3D [60] as our baseline.
Dreamcract3D exploits Stable-Zero123 [2] as a prior source for providing 3D
generative prior. By fine-tuning Stable-Zero123 on our dataset, we enable it
to obtain car-specific prior so it generalizes well to real cars.

Table 4:  Detection improvements by simulated data for corner-case scenes. We report the metric by calculating<EMAIL_ADDRESS>on CODA dataset [35].  Method | Simulated Data
---|---
500 | 1000 | 2000 | 3000 | 4000 | 5000
YOLOv8n | 0.261 | 0.299 | 0.312 | 0.357 | 0.386 | 0.386
YOLOv8s | 0.336 | 0.371 | 0.366 | 0.403 | 0.413 | 0.435
YOLOv8m | 0.347 | 0.354 | 0.362 | 0.416 | 0.419 | 0.441
YOLOv8l | 0.358 | 0.375 | 0.381 | 0.411 | 0.413 | 0.458

###  5.2 2D Tasks

Figure 7: Qualitative comparisons among recent advanced image segmentation
methods. We select the inputs from the testing set of our images and evaluate
the capacity of car component understanding for each method.

Corner-case 2D Detection. As shown in Table 4, we employ four variants of
YOLOv8 serial models as our detectors for training. To evaluate the
performance of models in corner-case scenes, we leverage the test part of the
CODA dataset [35] as our testing set. In particular, when we increase the
training simulated data from 500 to 5,000, the performance of detectors is
also improved by a large margin. This phenomenon demonstrates that our
simulated data is effective in improving a detector robust to corner-case
scenes. We provide the visualizations of simulated corner-case scenes in
Figure 6. The detailed simulation process and more visualizations can be seen
in the supplementary.

2D Car Parsing. We conduct benchmarks for car parsing maps of our dataset
using recent image segmentation methods, such as DeepLabV3[11], PointRend[33],
DDRNet[23], and SegFormer[71]. The quantitative performance for these methods
on our dataset is summarized in Table 3. Visual comparisons are provided in
Figure 7. Our high-quality dataset enables these methods to achieve promising
performance, highlighting its potential for application in self-driving
systems.

###  5.3 3D Tasks

3D Reconstruction. As depicted in Table 5, we show benchmark results of recent
state-of-the-art 3D reconstruction methods, such as Instant-NGP [48], 3DGS
[31], GaussianShader [27], and 2DGS [24] on our dataset. To the standard
lighting condition, we can find that recent methods are capable of achieving
PSNR more than 27 dB, which means these methods can reconstruct relatively
high-quality 3D cars from our dataset. However, the reflective and dark
condition results are lower than the standard. These two parts of our
3DRealCar bring two challenges to recent 3D methods. The first challenge is
the reconstruction of specular highlights. Due to the particular property of
cars, materials of car surfaces are generally glossy, which means it would
produce plenty of specular highlights if cars are exposed to the sun or strong
light. The second challenge is the reconstruction in a dark environment. The
training images captured in the dark environment lose plenty of details for
reconstruction. Therefore, how to achieve high-quality reconstruction results
from these two extremely lighting conditions is a challenge to recent methods.
3D visualizations can be found on our project page.

Novel View Synthesis. As illustrated in the left part of Figure 8, we show
visual comparisons of Zero123-XL [39] and our improved version by training on
our dataset. As we can see, given input images, we use Zero123-XL and our
improved version to synthesize novel views. In this figure, we can find that
Zero123-XL prefers to generate synthetic results with unrealistic texture and
geometry, due to the lack of prior for real objects. In contrast, our improved
version of Zero123-XL can generate photorealistic geometry and texture, which
demonstrates the effectiveness of our dataset.

3D Generation. As depicted in the right part of Figure 8, we visualize 3D
generation results of the recent state-of-the-art single-image-to-3D method,
Dreamcraft3D [60], along with its improved version by our dataset. This figure
shows that Dreamcraft3D sometimes fails to generate complete geometry or
realistic texture, due to the scarcity of the real car prior. As shown in
Table 3, we also show quantitative comparisons of Dreamcraft3D and its
improved version. CLIP-I means the similarity of rendered images with the
original input. The quantitative and qualitative results indicate our dataset
significantly improves 3D generation performance.

Table 5: Benchmark results on 3D reconstruction of our 3DRealCar dataset. We present the 3D reconstruction performance of recent state-of-the-art methods in three lighting conditions, standard, reflective, and dark, respectively. The best results are highlighted.  Method | Standard | Reflective | Dark
---|---|---|---
PSNR $\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
Instant-NGP [48] | 27.31 | 0.9315 | 0.1264 | 24.37 | 0.8613 | 0.1962 | 23.17 | 0.9152 | 0.1642
3DGS [31] | 27.47 | 0.9367 | 0.1001 | 24.58 | 0.8647 | 0.1852 | 23.51 | 0.9181 | 0.1613
GaussianShader [27] | 27.53 | 0.9311 | 0.1109 | 25.41 | 0.8684 | 0.1423 | 23.39 | 0.9172 | 0.1631
2DGS [24] | 27.58 | 0.9361 | 0.1084 | 23.19 | 0.8509 | 0.2041 | 22.63 | 0.9148 | 0.1681
Figure 8: Visualizations of novel view synthesis (left) and image-to-3D
generation (right).  we compare the results of the recent state-of-the-art
method, Zero123-XL [39], Dreamcraft3D [60], and their improvement by training
on our dataset.

##  6 Conclusion

In this paper, we propose the first large-scale high-quality 3D real car
dataset, named 3DRealCar. The collected dense and high-resolution 360-degree
views for each car can be used to reconstruct a high-quality 3D car. Extensive
experiments demonstrate the efficacy and challenges of our 3DRealCar in 3D
reconstruction. Thanks to the reconstructed high-quality 3D cars from our
dataset and car-part level annotations, our dataset can be utilized to support
various tasks related to cars. In addition, the benchmarking results can serve
as baselines for prospective research. Although 3DRealCar currently only has
car exterior views, we intend to provide both exterior and interior views in
the future to further promote the reconstruction of more intact 3D cars.

## References

  * [1] H. Abdi and L. J. Williams.  Principal component analysis.  Wiley interdisciplinary reviews: computational statistics, 2(4):433–459, 2010.
  * [2] S. AI.  Stable Zero123: Quality 3d object generation from single images.  https://stability.ai/news/stable-zero123-3d-generation, 2023.
  * [3] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
  * [4] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
  * [5] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman.  Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5460–5469, 2022.
  * [6] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman.  Zip-nerf: Anti-aliased grid-based neural radiance fields.  ICCV, 2023.
  * [7] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
  * [8] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [9] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su.  Tensorf: Tensorial radiance fields.  In European Conference on Computer Vision, pages 333–350. Springer, 2022.
  * [10] A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14124–14133, 2021.
  * [11] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [12] T. Chen, P. Wang, Z. Fan, and Z. Wang.  Aug-nerf: Training stronger neural radiance fields with triple-level physically-grounded augmentations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15191–15202, 2022.
  * [13] J. Chibane, A. Bansal, V. Lazova, and G. Pons-Moll.  Stereo radiance fields (srf): Learning view synthesis from sparse views of novel scenes.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, jun 2021.
  * [14] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [15] M. Deitke, R. Liu, M. Wallingford, H. Ngo, O. Michel, A. Kusupati, A. Fan, C. Laforte, V. Voleti, S. Y. Gadre, et al.  Objaverse-xl: A universe of 10m+ 3d objects.  Advances in Neural Information Processing Systems, 36, 2024.
  * [16] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi.  Objaverse: A universe of annotated 3d objects.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142–13153, 2023.
  * [17] Z. Fan, Y. Jiang, P. Wang, X. Gong, D. Xu, and Z. Wang.  Unified implicit neural stylization.  In European Conference on Computer Vision, pages 636–654. Springer, 2022.
  * [18] S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa.  Plenoxels: Radiance fields without neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501–5510, 2022.
  * [19] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin.  Fastnerf: High-fidelity neural rendering at 200fps.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14346–14355, 2021.
  * [20] C. J. García Orellana, R. Gallardo Caballero, H. M. González Velasco, and F. J. López Aligué.  Neusim: a modular neural networks simulator for beowulf clusters.  In Bio-Inspired Applications of Connectionism: 6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001 Granada, Spain, June 13–15, 2001 Proceedings, Part II 6, pages 72–79. Springer, 2001.
  * [21] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun.  Vision meets robotics: The kitti dataset.  The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  * [22] Y.-C. Guo, D. Kang, L. Bao, Y. He, and S.-H. Zhang.  Nerfren: Neural radiance fields with reflections.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18409–18418, 2022.
  * [23] Y. Hong, H. Pan, W. Sun, and Y. Jia.  Deep dual-resolution networks for real-time and accurate semantic segmentation of road scenes.  arXiv preprint arXiv:2101.06085, 2021.
  * [24] B. Huang, Z. Yu, A. Chen, A. Geiger, and S. Gao.  2d gaussian splatting for geometrically accurate radiance fields.  arXiv preprint arXiv:2403.17888, 2024.
  * [25] X. Huang, X. Cheng, Q. Geng, B. Cao, D. Zhou, P. Wang, Y. Lin, and R. Yang.  The apolloscape dataset for autonomous driving.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 954–960, 2018.
  * [26] A. Jain, B. Mildenhall, J. T. Barron, P. Abbeel, and B. Poole.  Zero-shot text-guided object generation with dream fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 867–876, 2022.
  * [27] Y. Jiang, J. Tu, Y. Liu, X. Gao, X. Long, W. Wang, and Y. Ma.  Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces.  arXiv preprint arXiv:2311.17977, 2023.
  * [28] M. M. Johari, Y. Lepoittevin, and F. Fleuret.  Geonerf: Generalizing nerf with geometry priors.  Proceedings of the IEEE international conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [29] H. Jun and A. Nichol.  Shap-E: Generating conditional 3D implicit functions, 2023.
  * [30] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis.  3d gaussian splatting for real-time radiance field rendering.  ACM Transactions on Graphics (ToG), 42(4):1–14, 2023.
  * [31] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis.  3d gaussian splatting for real-time radiance field rendering.  ACM Transactions on Graphics (ToG), 42(4):1–14, 2023.
  * [32] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al.  Segment anything.  arXiv preprint arXiv:2304.02643, 2023.
  * [33] A. Kirillov, Y. Wu, K. He, and R. Girshick.  Pointrend: Image segmentation as rendering.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9799–9808, 2020.
  * [34] X. Kong, S. Liu, X. Lyu, M. Taher, X. Qi, and A. J. Davison.  Eschernet: A generative model for scalable view synthesis.  arXiv preprint arXiv:2402.03908, 2024.
  * [35] K. Li, K. Chen, H. Wang, L. Hong, C. Ye, J. Han, Y. Chen, W. Zhang, C. Xu, D.-Y. Yeung, et al.  Coda: A real-world road corner case dataset for object detection in autonomous driving.  arXiv preprint arXiv:2203.07724, 2022.
  * [36] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin.  Magic3d: High-resolution text-to-3d content creation.  In CVPR, 2023.
  * [37] M. Liu, R. Shi, L. Chen, Z. Zhang, C. Xu, X. Wei, H. Chen, C. Zeng, J. Gu, and H. Su.  One-2-3-45++: Fast single image to 3D objects with consistent multi-view generation and 3D diffusion.  arXiv preprint arXiv:2311.07885, 2023.
  * [38] M. Liu, C. Xu, H. Jin, L. Chen, Z. Xu, H. Su, et al.  One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization.  arXiv preprint arXiv:2306.16928, 2023.
  * [39] R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298–9309, 2023.
  * [40] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  arXiv preprint arXiv:2303.05499, 2023.
  * [41] T. Liu, H. Zhao, Y. Yu, G. Zhou, and M. Liu.  Car-studio: Learning car radiance fields from single-view and unlimited in-the-wild images.  IEEE Robotics and Automation Letters, 2024.
  * [42] Y. Liu, C. Lin, Z. Zeng, X. Long, L. Liu, T. Komura, and W. Wang.  Syncdreamer: Generating multiview-consistent images from a single-view image.  arXiv preprint arXiv:2309.03453, 2023.
  * [43] T. Lu, M. Yu, L. Xu, Y. Xiangli, L. Wang, D. Lin, and B. Dai.  Scaffold-gs: Structured 3d gaussians for view-adaptive rendering.  arXiv preprint arXiv:2312.00109, 2023.
  * [44] L. Melas-Kyriazi, I. Laina, C. Rupprecht, N. Neverova, A. Vedaldi, O. Gafni, and F. Kokkinos.  IM-3D: Iterative multiview diffusion and reconstruction for high-quality 3D generation.  arXiv preprint arXiv:2402.08682, 2024.
  * [45] G. Metzer, E. Richardson, O. Patashnik, R. Giryes, and D. Cohen-Or.  Latent-nerf for shape-guided generation of 3d shapes and textures.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12663–12673, 2023.
  * [46] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng.  Nerf: Representing Scenes As Neural Radiance Fields for View Synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [47] T. Müller, A. Evans, C. Schied, and A. Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM Transactions on Graphics (ToG), 41(4):1–15, 2022.
  * [48] T. Müller, A. Evans, C. Schied, and A. Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  arXiv:2201.05989, Jan. 2022.
  * [49] A. Nichol, H. Jun, P. Dhariwal, P. Mishkin, and M. Chen.  Point-E: A System for Generating 3D Point Clouds from Complex Prompts, 2022.
  * [50] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [51] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [52] G. Qian, J. Mai, A. Hamdi, J. Ren, A. Siarohin, B. Li, H.-Y. Lee, I. Skorokhodov, P. Wonka, S. Tulyakov, et al.  Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors.  arXiv preprint arXiv:2306.17843, 2023.
  * [53] C. Reiser, S. Peng, Y. Liao, and A. Geiger.  Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14335–14345, 2021.
  * [54] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [55] J. L. Schonberger and J.-M. Frahm.  Structure-from-motion revisited.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016.
  * [56] R. Shi, H. Chen, Z. Zhang, M. Liu, C. Xu, X. Wei, L. Chen, C. Zeng, and H. Su.  Zero123++: a single image to consistent multi-view diffusion base model.  arXiv preprint arXiv:2310.15110, 2023.
  * [57] Y. Shi, P. Wang, J. Ye, M. Long, K. Li, and X. Yang.  Mvdream: Multi-view diffusion for 3d generation.  arXiv preprint arXiv:2308.16512, 2023.
  * [58] M. Suhail, C. Esteves, L. Sigal, and A. Makadia.  Light field neural rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8269–8279, 2022.
  * [59] C. Sun, M. Sun, and H.-T. Chen.  Direct Voxel Grid Optimization: Super-Fast Convergence for Radiance Fields Reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459–5469, 2022.
  * [60] J. Sun, B. Zhang, R. Shao, L. Wang, W. Liu, Z. Xie, and Y. Liu.  Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior.  arXiv preprint arXiv:2310.16818, 2023.
  * [61] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446–2454, 2020.
  * [62] M. V. T, P. Wang, X. Chen, T. Chen, S. Venugopalan, and Z. Wang.  Is attention all that neRF needs?  In The Eleventh International Conference on Learning Representations, 2023.
  * [63] J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng.  DreamGaussian: Generative Gaussian splatting for efficient 3d content creation.  arXiv preprint arXiv:2309.16653, 2023.
  * [64] Ultralytics.  YOLOv8: A cutting-edge and state-of-the-art (sota) model that builds upon the success of previous yolo versions.  https://github.com/ultralytics/ultralytics?tab=readme-ov-file, 2023.
  * [65] D. Verbin, P. Hedman, B. Mildenhall, T. Zickler, J. T. Barron, and P. P. Srinivasan.  Ref-nerf: Structured view-dependent appearance for neural radiance fields.  arXiv preprint arXiv:2112.03907, 2021.
  * [66] C. Wang, M. Chai, M. He, D. Chen, and J. Liao.  Clip-nerf: Text-and-image driven manipulation of neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3835–3844, 2022.
  * [67] J. Wang, S. Manivasagam, Y. Chen, Z. Yang, I. A. Bârsan, A. J. Yang, W.-C. Ma, and R. Urtasun.  Cadsim: Robust and scalable in-the-wild 3d reconstruction for controllable sensor simulation.  arXiv preprint arXiv:2311.01447, 2023.
  * [68] P. Wang, Y. Liu, Z. Chen, L. Liu, Z. Liu, T. Komura, C. Theobalt, and W. Wang.  F2-nerf: Fast neural radiance field training with free camera trajectories.  CVPR, 2023.
  * [69] Q. Wang, Z. Wang, K. Genova, P. P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla, N. Snavely, and T. Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2021.
  * [70] P. Xiao, Z. Shao, S. Hao, Z. Zhang, X. Chai, J. Jiao, Z. Li, J. Wu, K. Sun, K. Jiang, et al.  Pandaset: Advanced sensor suite dataset for autonomous driving.  In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 3095–3101. IEEE, 2021.
  * [71] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  Advances in neural information processing systems, 34:12077–12090, 2021.
  * [72] Z. Xie, J. Zhang, W. Li, F. Zhang, and L. Zhang.  S-nerf: Neural radiance fields for street views.  arXiv preprint arXiv:2303.00749, 2023.
  * [73] A. Yu, V. Ye, M. Tancik, and A. Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578–4587, 2021.
  * [74] J. Zhang, G. Yang, S. Tulsiani, and D. Ramanan.  Ners: Neural reflectance surfaces for sparse-view 3d reconstruction in the wild.  Advances in Neural Information Processing Systems, 34:29835–29847, 2021.
  * [75] K. Zhang, N. Kolkin, S. Bi, F. Luan, Z. Xu, E. Shechtman, and N. Snavely.  Arf: Artistic radiance fields, 2022.
  * [76] C. Zheng and A. Vedaldi.  Free3D: Consistent novel view synthesis without 3D representation.  arXiv preprint arXiv:2312.04551, 2023.
  * [77] X. Zhou, Z. Lin, X. Shan, Y. Wang, D. Sun, and M.-H. Yang.  Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes.  arXiv preprint arXiv:2312.07920, 2023.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes]

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]
We ensure our dataset would not bring negative societal impacts.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [No]

    2. (b)

Did you include complete proofs of all theoretical results? [No]

  3. 3.

If you ran experiments (e.g. for benchmarks)…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
We provide a URL in the abstract for researchers to download our dataset with
detailed instructions.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes]  Training details are described in Section 4.
More details can be found in supplementary materials.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [No]

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]

    2. (b)

Did you mention the license of the assets? [No]  They are public and we only
use them for academic study.

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [No]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [Yes]  Our dataset is collected under the consent of
owners.

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [No]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [Yes]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [No]  Our dataset does not
contain any potential participant risks.

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [No]  We collect our dataset by
ourselves. We do not need to pay for participants.


# Open-Domain Text Evaluation via Meta Distribution Modeling

Sidi Lu1, Asli Celikyilmaz2, Tianlu Wang2, Nanyun Peng1
1University of California, Los Angeles; 2Meta AI Research
{sidilu<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Recent advances in open-domain text generation models powered by large pre-
trained language models (LLMs) have achieved remarkable performance. However,
evaluating and controlling these models for desired attributes remains a
challenge, as traditional reference-based metrics such as BLEU, ROUGE, and
METEOR are insufficient for open-ended generation tasks. Similarly, while
trainable discriminator-based evaluation metrics show promise, obtaining high-
quality training data is a non-trivial task. In this paper, we introduce a
novel approach to evaluate open-domain generation - the Meta-Distribution
Methods (MDM). Drawing on the correlation between the rising parameter counts
and the improving performance of LLMs, MDM creates a mapping from the contrast
of two probabilistic distributions – one known to be superior to the other –
to quality measures, which can be viewed as a distribution of distributions
_i.e._ Meta-Distribution. We investigate MDM for open-domain text generation
evaluation under two paradigms: 1) _Generative_ MDM, which leverages the Meta-
Distribution Methods to generate in-domain negative samples for training
discriminator-based metrics; 2) _Discriminative_ MDM, which directly uses
distribution discrepancies between two language models for evaluation. Our
experiments on multi-turn dialogue and factuality in abstractive summarization
demonstrate that MDMs correlate better with human judgment than existing
automatic evaluation metrics on both tasks, highlighting the strong
performance and generalizability of such methods.

##  1 Introduction

Over the past few years, open-domain text generation powered by large
pretrained generative language models (LLMs) has seen significant strides,
attracting substantial interest (Radford et al., 2018, 2019; Brown et al.,
2020; OpenAI, 2022, 2023). These systems exibit remarkable abilities,
including generating human-like responses, aiding in natural language
understanding, and accomplishing complex tasks such as programming and article
writing. Despite ongoing debates regarding theoretical indistinguishability of
near-optimal LLMs (Sadasivan et al., 2023), creating a reliable and scalable
automatic evaluation for these models is imperative but remains an open
challenge.

Existing metrics from pre-LLM eras have limitations in evaluating the quality
of the generated responses. Specifically, reference-based statistical metrics
(e.g. BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and
Lavie, 2005)) do not work well for open-ended generation problems with high
content diversity like storytelling (Yao et al., 2019) and dialogue systems
(Mesgar et al., 2019; Li et al., 2017; Wen et al., 2016), as evidenced by
their low correlation with human judgments (Liu et al., 2016; Hu et al.,
2020). For these tasks, it is challenging, if not impossible, to collect a
sufficiently large number of reference examples that represent the
distribution of all feasible outputs, thus limiting the effectiveness of
reference-based metrics. With recent progress in pretrained models, model-
based reference metrics like BERTScore (Zhang et al., 2019), Bluert (Sellam et
al., 2020) are proposed to facilitate automatic evaluation for text
generation. They alleviate the sample efficiency issue of statistical
reference-based methods by using pretrained models to compute the similarities
between texts based on higher-level semantics. However, the effectiveness of
such methods is still reliant on the representativeness of the reference set,
and thus falls short when the output semantic space is also highly diverse.

Reference-free evaluation metrics, which take in the evaluating text and
directly output a score, are thus proposed to provide a more reliable solution
for automatically evaluating open-domain generation. There are generally two
paradigms of training a model to evaluate text without references: 1)
_prediction-based_ approaches like ADEM (Lowe et al., 2017) and DEAM
(Ghazarian et al., 2022), which treat the problem as a prediction task, and
try to train a classifier or regressor to produce a score. These works either
require extensive human annotations or dedicated manual designs of the
manipulation process in order to automatically generate negative samples to
train the classifier. 2) _distribution/divergence-based_ approaches (Pillutla
et al., 2021; Pimentel et al., 2022) concern the target as obtaining a
continuous divergence score between distributions have shown promising
results. However, they generally lack the ability to accurately assign credit
to each data point to perform instance-level evaluations. Note that the
boundary between the two paradigms can sometimes be subtle, and there are
several existing model-based metrics (Zhong et al., 2022; Liu et al., 2023)
built from modules under both philosophical paradigms.

In this paper, we propose Meta-Distribution Methods (MDM), a general and
reference-free framework for evaluating open-domain text generation. MDM is
built on a simple but reasonable assumption: models with similar architectures
but different model sizes generally obtain better generation quality with
larger models. Therefore, MDM learns an energy function $E(p)$ to represent
the dynamics of model performances scale with increasing parameter counts.
With this energy function $E(p)$, which can also be viewed as a distribution
over distributions (thus a meta-distribution), we can perform inference in
both generative and discriminative fashion to build automatic evaluation
metrics. MDM improves over previous reference-free evaluation metrics in two
aspects: 1) The generative MDM as illustrated in Figure 1(a) produces
effective negative samples without the necessity of additional human
annotations or sophisticated design for the manipulation process, and thus
facilitates the learning of prediction-based evaluation metrics. 2) The
discriminative MDM as illustrated in Figure 1(b) provides a distribution-level
measurement without sacrificing the capability to assign credits to individual
sentences, and thus results in more reliable divergence-based evaluation
metrics. Results on dialogue and abstractive summarization evaluation show
that, MDMs achieve strong performance and outperform existing methods on all
datasets without much task-specific designs.

(a) Generative MDM

(b) Discriminative MDM

Figure 1: Conceptual illustration of the Meta-Distribution Methods. There are
two major paradigms of Meta-Distribution Methods: _generative MDM_ and
_discriminative MDM_. (a) _Generative MDM_ constructs fake negative samples
from positive ones for training a classifier-based metric. (b) _Discriminative
MDM_ directly evaluate the distribution/sequence in a Naive Bayes framework.

##  2 Background and Related Works

##### Open-Domain Text Evaluation

There has been a synchronously growing interest in developing robust
evaluation methods for open-domain text generation models. Traditional
evaluation metrics, such as BLEU and ROUGE, have been shown to be inadequate
for assessing the quality of complex, multi-sentence responses generated by
these models. As a result, researchers have explored alternative evaluation
methods, including human evaluation, adversarial evaluation, and unsupervised
metrics. Human evaluation remains the gold standard, but it is time-consuming
and costly. Adversarial evaluation, which involves testing models against a
set of challenging examples, has shown promise in identifying weaknesses in
current models. Unsupervised metrics, such as BERTScore and Perplexity,
provide quick and automated evaluation, but their correlation with human
judgments remains a topic of debate. The field of open-domain text evaluation
continues to evolve, and developing reliable evaluation methods will be
essential for advancing the state-of-the-art in this exciting area of
research.

##### Prediction-Based Metrics

ADEM (Lowe et al., 2017) is one of the first attempts at training a model to
evaluate machine-generated text. It deals with single-turn dialogue evaluation
problem, and uses the contextualized representation of the context in
interaction with that of the responses to train the model. DEAM (Ghazarian et
al., 2022) is a novel evaluation metric that aims to assess the coherence and
quality of multi-turn dialogue systems. Unlike traditional evaluation metrics,
DEAM uses manipulation techniques to construct negative samples from positive
samples, allowing for a more nuanced assessment of model performance. DEAM
operates by first parsing the sequence into an abstract meaning representation
(AMR), and then manipulating the AMR to introduce inconsistencies and
irrelevancies that undermine the coherence of the dialogue. The manipulated
AMR is then transformed back into text form for evaluation. This method
supports multi-turn dialogue evaluation and has achieved state-of-the-art
performance on various benchmark datasets. By using AMR-based semantic
manipulations, DEAM provides a promising approach for evaluating the quality
of dialogue systems in a more comprehensive and accurate manner. _Generative_
MDM shares a similar process , as it manipulates the positive true samples for
the generation of negative samples, serving the purpose of training a
classifier.

##### Distribution/Divergence-based Metrics

MAUVE and follow-up works (Pillutla et al., 2021; Pimentel et al., 2022)
analyse the quality gap between human-generated text and machine-generated
text by studying the divergence frontier of human-generated samples in
contrast to the learnt model. While their setup is not directly relevant to
our approach, it provides an insightful perspective of using the likelihood
predictions of LMs for evaluation purposes. Zhong et al. (2022) proposes a
multi-dimensional evaluation system for more robust automatic evaluation. It
ensembles the score from a set of _classifier-based_ metrics, each of which
trained to evaluate a specific aspect in intuition of the text quality.
GPTEval (Liu et al., 2023) tries to quantitatively exploit large language
models that are trained with strong human alignment. It uses the score
prediction from GPT-4 (OpenAI, 2023) to evaluate how well the given text
adheres to human opinion. _Discriminative_ MDM falls under this paradigm,
since it serves as a metric with more continuously distributed scores for the
evaluated text.

##### Contrastive Decoding and Contrastive Momentum

Contrastive decoding is a decoding algorithm that leverages the strengths of
two language models: a stronger expert model and a weaker amateur model. The
algorithm decodes towards the objective of maximizing the difference between
the log-probabilities of the expert and amateur models, resulting in high-
quality generated samples. Specifically, the algorithm tries to decode
sequences that maximize the _contrastive momentum_ :

$\log p_{\text{expert}}(x)-\log p_{\text{amateur}}(x),$ (1)

where $x$ is the generated sample and $p_{\text{expert}}$,
$p_{\text{amateur}}$ are the expert and amateur models, respectively. The
original paper (Li et al., 2022) demonstrates that this approach results in
even higher quality samples than decoding from the expert model alone.
Contrastive decoding provides an insightful way to study the dynamics of how
models’ capabilities scale up with larger parameter numbers. Under the
proposed framework of MDM, contrastive decoding can be viewed as leveraging
the two-point momentum for an approximate optimization towards the
maximization of $E(p)$.

##  3 Methodology

###  3.1 Notations and Problem Formulation

We use $\mathbf{s}$ to denote a sequence and $s_{i}$ to denote the $i-$th
token in $\mathbf{s}$. $p(\mathbf{s})$ denotes the probability of sequence
$\mathbf{s}$ under a model $p$. We assume model $p$ is a probabilistic
distribution defined on $\Sigma^{*}$, where $\Sigma$ is the set of valid
tokens and $\Sigma^{*}$ is the universal set of all sequences consisting of
such tokens.

Consider an energy function $E(p)$ which projects from a model’s likelihood
prediction $p(\mathbf{s})$ to “a measure of model performance” – a scalar.
This function does not necessarily have an analytical form, however, we assume
that we have access to some partial order relations it defines. If we
normalize this energy function using some activation functions such as the
softmax function, it becomes a “ _distribution of distributions_ ” hence a
Meta-Distribution. Intuitively, any evaluation metrics that aim to correlate
better with model performance, as evaluated through human judgments, can be
viewed as one version of $E(p)$.

In the ideal case, $E(p)$ can be used to perform:

  * •

_Discriminative_ inference: to evaluate any existing distribution by ranking
them according to $E(p)$, and/or quantify the contribution of a specific
sequence’s likelihood $p(\mathbf{s})$ _w.r.t._ $E(p)$;

  * •

_Generative_ inference: to improve/degenerate the model performance on the fly
by altering $p$ towards maximization/minimization of $E(p)$.

In this paper, we adapt both discriminative and generative inference of MDM to
either directly serve as the metric (_discriminative_) or produce negative
training samples (_generative_) as needed by the training of the metric model.

###  3.2 Connection with Contrastive Decoding

Contrastive decoding (Li et al., 2022) can be regarded as a simplified
concrete approach of _Generative_ inference with $E(p)$. The _contrastive
momentum_ can be viewed as a finite difference approximation of
$m=\frac{\partial{E(p)}}{\partial p}$, where it does the following two
simplification:

  * •

$m=\frac{\partial{E(p)}}{\partial p}$ is approximated by the finite difference
between two models that have a pre-assumed performance superiority of one to
the other (namely the _expert_ and _amateur_ model in the original paper (Li
et al., 2022) ).

  * •

It concerns the joint probability of each sequence as the product of step-wise
probabilities, _i.e._ the auto-regressive formulation.

Considering the 1-normalization constraint of probabilities, one feasible
solution is the following approximation in contrastive decoding:

$\log m(x|s)\approx\log~{}p_{1}(x|s)-\log~{}p_{0}(x|s)$

Searching towards this objective is equivalent to using $m(x|s)$ to optimize
$p$ towards the maximization of $E(p)$.

###  3.3 The Partial Order Assumption

We hereby discuss in detail how we can conduct contrastive methods for
evaluation purposes. While it is nontrivial, if not impossible, to come up
with analytical forms for $E(p)$, we can make some assumptions to obtain
partial orders from $E(p)$. Consider a _series_ of models that share similar
architectures and other pretraining/finetuning setups, but differ in model
sizes (e.g. T5-small/base/large, etc.). It is usually reasonable to assume
that, the model variant with a larger number of parameters usually would
perform better under the evaluation of most metrics. While it would still be
challenging for a more general superiority assumption on models from different
model classes, we can assume the following partial order (a linear order
within one concerned model class) for a distribution measure as is induced by
the energy function $E(p)$:

$E(p_{small})<E(p_{base})<E(p_{large})$

See Figure 2.

###  3.4 Meta-Distribution Methods

####  3.4.1 First Order Approximation of $E(p)$

As is previously mentioned in Sec 3.1, it could be intractable to estimate the
accurate version of $\frac{\partial{E(p)}}{\partial p}$. Following similar
approximations as in previous methods(Li et al., 2022), we approximate it
using a secant hyperplane between two distribution/model points. We denote
them as the _origin_ distribution $p_{o}$ and the _guide_ distribution
$p_{g}$. In a sense, this is equivalent to a first-order approximation of
$E(p)$ in the space $\\{p(\mathbf{s})|\mathbf{s}\in\Sigma*\\}$. Different
choices of $p_{o}$ and $p_{g}$ result in different quality of the first-order
approximations for $E(p)$, hence different performance of the evaluation
metric. We need to investigate the general principle for choosing the origin
and guide distributions.

Figure 2: (a) It’s difficult, if ever possible, to assume a total order under
the evaluation of $E(p)$ for models from different model classes. It’s still
possible to assume partial orders for models from the same model class. (b)
Generative MDM uses the degraded distribtution $p_{o}^{-}$ to synthesize fake
samples for training a discriminator as the metric. The warm/cold region
indicates the decision boundary of the resulting trainable metric induced by
fake samples from $p_{o}^{-}$. (c) Discriminative MDM directly determines the
decision boundary by pooling the values of the step-wise contrastive momentum.

####  3.4.2 Generative MDM: Synthetic Data Generation with MDM

Generative MDM follows prior works such as ADEM (Lowe et al., 2017) and DEAM
(Ghazarian et al., 2022) to formulate reference-free evaluation metrics as
prediction tasks. In order to evaluate generated texts, a discriminator can be
trained on positive and negative examples to serve as the evaluation
metric.111The discriminator does not necessarily need to provide binary
decisions, it can also produce scores. But we use binary examples for
simplicity. However, we usually only have human-written texts which are
considered positive examples while negative examples are non-trivial to
obtain. Randomly generating negative examples using a uniform distribution
over all possible sequences of tokens is not efficient, as the discriminator
would easily degenerate into a poor local minimum. On the other hand, simply
generating negative examples from pretrained large-language models (LLMs) may
not result in real low-quality texts, which would confuse the discriminator.

Generative MDM provides a controllable approach to reduce the performance of
existing pretrained models to generate “sufficiently deceptive and targeted
negative examples”. Following our previous analysis framework, this is
effectively equivalent to descending along the direction of $-\frac{\partial
E(p)}{\partial p}$ from an initial point $p_{o}$ — usually, a small (thus
worse-performing) model. Although we do not have a tractable explicit
estimation for $E(p)$, we can still utilize the previous approximation which
is to replace the differential operation with the difference between $p_{o}$
and a selected guide distribution $p_{g}$.

Specifically, we now consider an anchored _origin_ model $p_{o}$ to compute
the contrastive momentum towards several different superior _guide_ $p_{g}$
models. To control irrelevant factors, we obtain a guidance model $p_{g}$ from
a similar architecture but with more layers/parameters. By leveraging the
contrastive momentum $\log m_{o\rightarrow g}=\log p_{g}-\log p_{o}$ in the
reversed direction, we controllably degenerate from the _origin_ model.
Consequently, we obtain a probability distribution $\log p_{o}^{-}\propto\log
p_{o}-\gamma m_{o\rightarrow g}$ that disproportionately amplifies the
likelihood of “machine artifacts” in a controllable (by setting $\gamma$)
scale. Sampling from $p_{o}^{-}$ allows us to obtain suitable negative
examples.

We hereby discuss how to generate negative examples in a more targeted manner.
The whole process can be viewed as a controllable generation problem. We start
from existing positive examples and selectively decrease the overall text
quality without altering the majority of the context and meaning. In this
sense, the generated negative examples would be more targeted and misleading
compared to sampling from $p_{o}^{-}$ at the very beginning. Although for
differentiable data such as images this is simple, one can directly use
gradient-based approach to achieve it. However, this does not apply to non-
differentiable data like text. We hereby discuss how to construct a similar
process.

One feasible approach is to train a model to address a _segment
insertion/reconstruction_ problem. Given the context and the position at which
a segment is removed (randomly or strategically), we model a conditional
distribution that reconstructs the original segment. Once we have obtained
such a model and its degenerated form, we can utilize it to repeatedly operate
on existing positive examples, allowing us to construct negative examples in a
further well-controlled manner. This process enables us to generate negative
examples that are specifically targeted and deceptive. Although text data is
non-differentiable, this approach leverages the model’s ability to learn
conditional distributions and eventually constructs the desired ideal negative
examples. The full process can be summarized as follows:

####  3.4.3 Discriminative MDM: Directly using MDM as the metric

Although Generative MDM is a reasonably flexible and scalable framework, there
are still many variable factors in the generation process (e.g. how to choose
which segment to remove, the randomness in the process of segment
reconstruction, degradation strength factor $\gamma$ etc.) that may affect the
performance of the resulting metric. Therefore, it is desirable to remove the
generation subroutine completely. Consider why we need to train a
discriminator as a metric, because we usually do not have a tractable model
for the positive or negative distribution. However, under the MDM framework,
we do have a tractable model $p_{o}^{-}$ for the negative distribution, which
is composed from the origin model $p_{o}$ and the contrastive momentum $m$. In
light of this, we can consider directly deploying $m$ as a divergence-based
metric for evaluation.

For each sequence, we concern the step-wise likelihood prediction from the
_origin_ model $\log p_{o}(x|\mathbf{s}_{<t})$ and the _guide_ model $\log
p_{g}(x|\mathbf{s}_{<t})$. If a sequence is considered good, both models’
predicted likelihood will be relatively high, but $\sum_{t}\log
p_{g}(x|\mathbf{s}_{<t})$ should be more significant than $\sum_{t}\log
p_{o}(x|\mathbf{s}_{<t})$. If we strictly follow the definition, summing up
the step-wise contrastive momentum over the entire sequence (_i.e._ sum-
pooling) would be the metric to evaluate the generation quality. See Figure 2.

However, we argue that there would be a subtle discrepancy between theory and
practice. First, the sum-pooled score would still be numerically influenced by
the sequence length (even though not as severely as directly using the log-
likelihood). Moreover, sum-pooling overemphasizes the impact of extremely low
probability steps because the discrepancy between the origin model and guide
model predictions in those regions could be further amplified on the
logarithmic scale.

In the experiments, we compare different strategies to _pool_ the sequence of
step-wise contrastive momentum values into a sentence-level evaluation score.
We call this paradigm of using MDM as Discriminative MDM. There is no explicit
generation process in Discriminative MDM as we only treat the two models and
their contrastive momentum as likelihood value predictors.

##  4 Experiments

###  4.1 Dialogue Evaluation with MDM

The first part of our experiment is primarily focused on dialogue evaluation.
Given a set of annotated dialogues, each with human-annotated quality scores
ranging from $0.0$ to $1.0$, our objective is to assign scores to each
evaluated sequence that maximizes the correlation with human annotations.
Additionally, for dialogue evaluation, we assume we are not permitted to
perform any training on data within the same domain. Our training/fine-tuning
exercises are conducted on dialogues from both TopicalChatGopalakrishnan et
al. (2019) and PersonalChat datasets(Zhang et al., 2018). Subsequently, we
evaluate our methods on annotated dialogues from the FED (Mehri and Eskenazi,
2020) and DSTC9(Gunasekara et al., 2020) datasets.

##### Dataset and Experiment Setup

We adopt most experimental settings from DEAM (Ghazarian et al., 2022) to
verify the effectiveness of our method. The statistics of the involved
datasets in our experiments are shown as follows:

Table 1: Data usage in our experiments Dataset | size | Avg. len
---|---|---
TopicalChat + PersonalChat (Gopalakrishnan et al., 2019; Zhang et al., 2018) | 17567/2078 | 377
FED (test, w/ human annotation) (Mehri and Eskenazi, 2020) | 125 | 168
DSTC9 (test, w/ human annotation) (Gunasekara et al., 2020) | 2200 | 318

The two models in each contrastive and expert model from different sizes of T5
(Raffel et al., 2019; Wei et al., 2021) checkpoints respectively. For ablation
study of how different choices of such model pairs and other factors impact
the performance of MDM, please refer to Table 3.

####  4.1.1 Model Specification

There would be multiple strategies to construct the context-prediction pairs.
We study the following cases for Generative MDM:

  * •

Segment-Single: The manipulation of data is only applied once to a random
segment no longer than 20 tokens in a real dialogue.

  * •

Utterance-Single: The manipulation of data is only applied once to a random
utterance in a real dialogue.

  * •

Mixed-Single: The manipulation of data is only applied once to a random
utterance or a random segment no longer than 20 tokens in a real dialogue.

  * •

Mixed-Multi: The manipulation of data is applied randomly for a uniformly
random value from 1$\sim$4 times.

  * •

AMR-Multi: The location of data manipulation is guided by similar approach as
in DEAM Ghazarian et al. (2022).

Similarly, we study following aggregation strategies for discriminative MDM:

  * •

Pooling along the timestep axis (Avg-Pooled/Max-Pooled/Min-Pooled)

  * •

Classifier-Pooled: We train a small linear classifier to convert the sequence
of as a trainable pooler using annotated training data (from the original
dataset or as synthesized by DEAM(Ghazarian et al., 2022)).

####  4.1.2 Results and Analysis

We present quantitative results in Table 2. Given that MDM is designed to
model discrete comparison relations, our approach aligns with methodology
established by previous research (Mesgar et al., 2019; Vakulenko et al., 2018;
Zhang et al., 2021; Ghazarian et al., 2022) and report the Spearman
correlation to better evaluate MDM against these baselines.

Table 2: Spearman correlation of different approaches for dialogue evaluation. All reported results from our approach have $p$-value $<$ 1e-4. We show the best-performing results with bolded numbers and second-best with underlined numbers.  Model | FED | DSTC9
---|---|---
Coherence | Overall | Coherence | Overall
Mesgar et al. (2019) | 0.10 | -0.01 | 0.02 | 0.05
Vakulenko et al. (2018) | 0.13 | 0.10 | 0.00 | 0.00
DynaEval (Zhang et al., 2021) | -0.36 | -0.4 | -0.03 | -0.01
DEAM (Ghazarian et al., 2022) | 0.47 | 0.55 | 0.19 | 0.20
Generative MDM (Ours) | 0.51 | 0.52 | 0.19 | 0.23
Discriminative MDM (Ours) | 0.59 | 0.61 | 0.27 | 0.25

We find that DSTC9 favors segment-level manipulation while FED favors
utterance-level manipulation. Additionally, our findings indicate that larger
performance gap between the origin/guide models in general induces better
performance. Finally, Discriminative MDM methods present less bias across
datasets and offer more efficiency during training, as they eliminate the
necessity for training an additional classifier model.

Table 3: Ablation study on how different variants and hyperparameters of the proposed MDM impact the final performance of the methods. Model | FED | DSTC9
---|---|---
Coherence | Overall | Coherence | Overall
Generative MDM |  |  |  |
\- Segment-Single (small-large) | 0.12 | 0.07 | 0.11 | 0.10
\- Utterance-Single (small-large) | 0.29 | 0.36 | 0.05 | 0.08
\- Mixed-Single (small-large) | 0.32 | 0.35 | 0.14 | 0.12
\- Mixed-Multi (small-large) | 0.42 | 0.40 | 0.17 | 0.18
\- AMR-Multi (small-large) | 0.49 | 0.53 | 0.20 | 0.22
\- AMR-Multi (small-base) | 0.48 | 0.51 | 0.19 | 0.20
\- AMR-Multi (small-large) | 0.49 | 0.53 | 0.20 | 0.22
\- AMR-Multi (small-xl) | 0.51 | 0.52 | 0.19 | 0.23
Discriminative MDM |  |  |  |
\- Avg-Pooled (small-large) | 0.31 | 0.32 | 0.12 | 0.13
\- Min-Pooled (small-large) | 0.27 | 0.28 | 0.07 | 0.04
\- Max-Pooled (small-large) | 0.46 | 0.43 | 0.16 | 0.15
\- Classifier-Pooled (small-large) | 0.53 | 0.56 | 0.24 | 0.22
\- Classifier-Pooled (small-base) | 0.42 | 0.44 | 0.13 | 0.10
\- Classifier-Pooled (small-large) | 0.53 | 0.56 | 0.24 | 0.22
\- Classifier-Pooled (base-large) | 0.39 | 0.40 | 0.09 | 0.11
\- Classifier-Pooled (small-3b) | 0.59 | 0.61 | 0.27 | 0.25

###  4.2 Factuality Evaluation for Abstractive Summarization Models

We now concern a slightly different setup, where we use MDM for evaluating the
factuality of abstractive summarization models. Previous methods usually
consider the problem as an NLI problem with only the “ _entailment_ ” case of
annotations. We evaluate our model in comparison to existing works, such as
Falsesum(Utama et al., 2022), QAGS(Wang et al., 2020), Coco(Xie et al., 2021)
and FactCC(Kryściński et al., 2019).

####  4.2.1 Dataset and Experiment Setup

We adopt most experimental settings from existing works on factuality
evaluation of abstractive summarization. The following table lists some key
statistics of the datasets we use in this part of experiments.

Table 4: Data usage in our experiments Dataset | size
---|---
CNN/DailyMail (train) (Nallapati et al., 2016) | 1,003,355
QAGS-CNN/DailyMail (test, w/ human annotation) (Wang et al., 2020) | 235
QAGS-XSum (test, w/ human annotation) (Wang et al., 2020) | 239
SummEval (test, w/ human annotation) (Fabbri et al., 2021) | 1200/3600

We train our likelihood functions under both supervised and unsupervised
setups:

  * •

Unsupervised For annotated data, we only train the likelihood function to
_maximize_ the likelihood of positive samples. In this sense, the model is
more similar to the one we used in dialogue evaluation.

  * •

Supervised We train the likelihood function as factual-counterfactual label-
conditioned probabilities using synthesized pseudo labels following the
methods as described in FalseSum(Utama et al., 2022). When performing the
discriminative MDM, we use factual-conditoned larger model as the _guide_
model and counterfactual-conditioned smaller model as the _origin_.

####  4.2.2 Results and Analysis

We show results in Table 5. Previous works report their performance
inconsistently in either Spearman/Pearson correlation or an accuracy score
with 0/1 quantization of the annotations. We adopt 0/1-quantization and report
the accuracy of each baseline/model.

Table 5: Evaluation results for abstrative text summarization. For all models and datasets, we show the quantized accuracy scores. Results with * means ones from our re-implementation or re-evaluation so as to unify the metric. | SummEval | QAGS
---|---|---
| Overall | CNN/DM | XSum
Falsesum (Utama et al., 2022) | 65.18 | 75.05 | 94.89* | 55.65*
QAGS (Wang et al., 2020) | 59.82* | 72.15 | 88.08* | 56.48*
Coco (Xie et al., 2021) | 66.71* | 77.00* | 93.62* | 60.66*
FactCC (Kryściński et al., 2019) | 60.04 | 73.42 | 85.96* | 61.09*
Discriminative MDM (Ours.) |  |  |  |
\- Max-Pooled (small-large, Unsupervised) | 64.17 | 74.05 | 85.11 | 63.18
\- Classifier-Pooled (small-large, Unsupervised) | 64.74 | 74.47 | 90.21 | 59.00
\- Classifier-Pooled (small-large, Supervised) | 67.17 | 76.79 | 93.19 | 60.67
\- Classifier-Pooled (small-xl, Supervised) | 68.17 | 78.27 | 93.62 | 63.18

  * •

Learned pooling does not show as significant improvements as in dialogue for
factuality evaluation. This is probably due to that the majority of the output
semantics is already contained within the condition, and the statistical
properties of the step-wise contrastive momentum score are more consistently
distributed.

  * •

With the explicitly annotated data with pseudo labels, using the supervised
setting is in general performing better. This is probably due to that,
conditioning the likelihood function on the pseudo labels implicitly creates
an ensemble of MDM and the existing method Falsesum (Utama et al., 2022).

##  5 Conclusion and Future Work

This paper presents the Meta-Distribution Methods (MDM) as a general framework
for evaluating open-domain text generation models. MDM is constructed around
analyzing the correlation between model scales and the respective distribution
prediction, and how it can be exploited to alter the performance of a certain
model on-the-fly in inference. We demonstrate how MDM can be used for
evaluation purposes in two general paradigms: Generative MDM, which
manipulates existing positive samples to generate in-domain negative samples
and subsequently trains a classifier, and Discriminative MDM, which employs
the contrastive momentum as a direct metric for evaluation. Our experiments
results in multi-turn dialogue evaluation and factuality evaluation for
abstractive summarization illustrate that MDM correlates better with human
intuition than traditional metrics. In summary, the MDM method emerges as a
promising and scalable approach for evaluating open-domain text generation
systems, among others.

For future work, it is interesting to investigate higher order approximations
of the meta distribution that account for the dynamics of an extended series
of models across different scales. Furthermore, it presents an interesting
avenue to explore whether the Generative MDM approach can be extended for a
more effective ensemble of heterogeneous models that differ in scale,
architecture or even training data but operate under a reasonable assumption
of a partially ordered performance level.

## References

  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , pages 65–72.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Fabbri et al. (2021) Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021.  Summeval: Re-evaluating summarization evaluation.  _Transactions of the Association for Computational Linguistics_ , 9:391–409.
  * Ghazarian et al. (2022) Sarik Ghazarian, Nuan Wen, Aram Galstyan, and Nanyun Peng. 2022.  Deam: Dialogue coherence evaluation using amr-based semantic manipulations.  _arXiv preprint arXiv:2203.09711_.
  * Gopalakrishnan et al. (2019) Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. 2019.  Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations.  In _Proc. Interspeech 2019_ , pages 1891–1895.
  * Gunasekara et al. (2020) Chulaka Gunasekara, Seokhwan Kim, Luis Fernando D’Haro, Abhinav Rastogi, Yun-Nung Chen, Mihail Eric, Behnam Hedayatnia, Karthik Gopalakrishnan, Yang Liu, Chao-Wei Huang, et al. 2020.  Overview of the ninth dialog system technology challenge: Dstc9.  _arXiv preprint arXiv:2011.06486_.
  * Hu et al. (2020) Junjie Hu, Yu Cheng, Zhe Gan, Jingjing Liu, Jianfeng Gao, and Graham Neubig. 2020\.  What makes a good story? designing composite rewards for visual storytelling.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 7969–7976.
  * Kryściński et al. (2019) Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019\.  Evaluating the factual consistency of abstractive text summarization.  _arXiv preprint arXiv:1910.12840_.
  * Li et al. (2017) Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017.  Adversarial learning for neural dialogue generation.  _arXiv preprint arXiv:1701.06547_.
  * Li et al. (2022) Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2022.  Contrastive decoding: Open-ended text generation as optimization.  _arXiv preprint arXiv:2210.15097_.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016.  How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation.  _arXiv preprint arXiv:1603.08023_.
  * Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023\.  Gpteval: Nlg evaluation using gpt-4 with better human alignment.  _arXiv preprint arXiv:2303.16634_.
  * Lowe et al. (2017) Ryan Lowe, Michael Noseworthy, Iulian V Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017.  Towards an automatic turing test: Learning to evaluate dialogue responses.  _arXiv preprint arXiv:1708.07149_.
  * Mehri and Eskenazi (2020) Shikib Mehri and Maxine Eskenazi. 2020.  Unsupervised evaluation of interactive dialog with dialogpt.  _arXiv preprint arXiv:2006.12719_.
  * Mesgar et al. (2019) Mohsen Mesgar, Sebastian Bücker, and Iryna Gurevych. 2019.  Dialogue coherence assessment without explicit dialogue act labels.  _arXiv preprint arXiv:1908.08486_.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  _arXiv preprint arXiv:1602.06023_.
  * OpenAI (2022) OpenAI. 2022.  Gpt-3.5.  https://openai.com/blog/chatgpt.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Pillutla et al. (2021) Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021.  Mauve: Measuring the gap between neural text and human text using divergence frontiers.  _Advances in Neural Information Processing Systems_ , 34:4816–4828.
  * Pimentel et al. (2022) Tiago Pimentel, Clara Meister, and Ryan Cotterell. 2022.  On the usefulness of embeddings, clusters and strings for text generator evaluation.  _arXiv_.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Sadasivan et al. (2023) Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023.  Can ai-generated text be reliably detected?  _arXiv preprint arXiv:2303.11156_.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020.  Bleurt: Learning robust metrics for text generation.  _arXiv preprint arXiv:2004.04696_.
  * Utama et al. (2022) Prasetya Ajie Utama, Joshua Bambrick, Nafise Sadat Moosavi, and Iryna Gurevych. 2022\.  Falsesum: Generating document-level nli examples for recognizing factual inconsistency in summarization.  _arXiv preprint arXiv:2205.06009_.
  * Vakulenko et al. (2018) Svitlana Vakulenko, Maarten de Rijke, Michael Cochez, Vadim Savenkov, and Axel Polleres. 2018.  Measuring semantic coherence of a conversation.  In _The Semantic Web–ISWC 2018: 17th International Semantic Web Conference, Monterey, CA, USA, October 8–12, 2018, Proceedings, Part I 17_ , pages 634–651. Springer.
  * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.  Asking and answering questions to evaluate the factual consistency of summaries.  _arXiv preprint arXiv:2004.04228_.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_.
  * Wen et al. (2016) Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2016.  A network-based end-to-end trainable task-oriented dialogue system.  _arXiv preprint arXiv:1604.04562_.
  * Xie et al. (2021) Yuexiang Xie, Fei Sun, Yang Deng, Yaliang Li, and Bolin Ding. 2021.  Factual consistency evaluation for text summarization via counterfactual estimation.  _arXiv preprint arXiv:2108.13134_.
  * Yao et al. (2019) Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019.  Plan-and-write: Towards better automatic storytelling.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 7378–7385.
  * Zhang et al. (2021) Chen Zhang, Yiming Chen, Luis Fernando D’Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, and Haizhou Li. 2021.  Dynaeval: Unifying turn and dialogue level evaluation.  _arXiv preprint arXiv:2106.01112_.
  * Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018.  Personalizing dialogue agents: I have a dog, do you have pets too?  _arXiv preprint arXiv:1801.07243_.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019\.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_.
  * Zhong et al. (2022) Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022.  Towards a unified multi-dimensional evaluator for text generation.  _arXiv preprint arXiv:2210.07197_.