# Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models
in Engineering Design

Lyle Regenwetter1, Akash Srivastava2, Dan Gutfreund2, Faez Ahmed1
Massachusetts Institute of Technology1, MIT-IBM Watson AI Lab2
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Deep generative models, such as Variational Autoencoders (VAEs), Generative
Adversarial Networks (GANs), Diffusion Models, and Transformers, have shown
great promise in a variety of applications, including image and speech
synthesis, natural language processing, and drug discovery. However, when
applied to engineering design problems, evaluating the performance of these
models can be challenging, as traditional statistical metrics based on
likelihood may not fully capture the requirements of engineering applications.
This paper doubles as a review and a practical guide to evaluation metrics for
deep generative models (DGMs) in engineering design. We first summarize well-
accepted ‘classic’ evaluation metrics for deep generative models grounded in
machine learning theory and typical computer science applications. Using case
studies, we then highlight why these metrics seldom translate well to design
problems but see frequent use due to the lack of established alternatives.
Next, we curate a set of design-specific metrics which have been proposed
across different research communities and can be used for evaluating deep
generative models. These metrics focus on unique requirements in design and
engineering, such as constraint satisfaction, functional performance, novelty,
and conditioning. Since data-driven design problems are hugely varied in
nature, we discuss which evaluation metrics are best suited to different
design objectives and representation schemes. We structure our review and
discussion as a set of practical selection criteria and usage guidelines.
Throughout our discussion, we apply the metrics to models trained on simple-
to-visualize 2-dimensional example problems. Finally, to illustrate the
selection process and classic usage of the presented metrics, we evaluate
three deep generative models on a multifaceted bicycle frame design problem
considering performance target achievement, design novelty, and geometric
constraints. We publicly release the code for the datasets, models, and
metrics used throughout the paper at decode.mit.edu/projects/metrics/.

##  1 Introduction

Deep generative models (DGMs) have seen explosive growth across engineering
design disciplines in recent years. DGMs like Generative Adversarial Networks
(GAN) [1] and Variational Autoencoders (VAE)[2] have dominated image
generation problems since 2014, but only bridged the gap to the design
community in 2016 [3] and have since been applied across design domains to
problems such as optimal topology generation, airfoil synthesis, and
metamaterial design. As promising new methods for image synthesis like
diffusion models [4, 5] are introduced in other machine learning fields,
researchers in design adapt them to solve challenging design problems [6].
Similarly, transformers, a leading class of generative models for sequences,
have dominated natural language generation for years [7, 8, 9], and have seen
extensive use in the textual generation of design concepts [10, 11].

Deep generative models are powerful learners, boasting an unparalleled ability
to process and understand complex data distributions and mimic them through
batches of synthetic data. In the context of data-driven design, these
‘distributions’ are often comprised of a collection of existing designs that
lie in some multidimensional design manifold, in the same way that a
collection of points would form a density distribution in a Euclidean space
[3]. From this perspective, it’s clear why DGMs are promising data-driven
designers: They can study collections of existing designs, understand their
distribution, and generate new ones that, as the model understands, should
belong in the same manifold, but do not yet exist. Classically, the
distribution-matching capabilities of DGMs are measured using statistical
similarity (i.e, how similar is the distribution of generated designs to the
dataset?).

However, while DGMs’ amazing distribution-matching ability is often
beneficial, it is limited because, in many design problems, we desire designs
to be significantly unique or distinct from existing designs. Additionally,
even if distribution-matching is desirable, it is often secondary to meeting
problem constraints and achieving functional performance targets. As a result,
relying solely on similarity as an objective can lead to misguided design
efforts, since very similar designs can have drastically different
performance, as illustrated in Figure 1. Historically, design researchers have
failed to account for this gap when selecting evaluation metrics, often opting
for the classic similarity-based metrics that are prominent in other machine
learning domains. In this paper, we provide an exposition of evaluation
metrics for DGMs, which we show are important for design practitioners and
design automation researchers using deep generative models. We intend for the
metrics presented to apply to any given deep generative model for design, and
we do not go into detail about specific model architectures. For an
introduction to deep generative models and an overview of design engineering
research using deep generative models to date, we refer readers to a review
paper on deep generative models in engineering design [3].

(a) Deflection: $0.05\,cm$

(b) Deflection: $10.46\,cm$

Figure 1: Two very similar bike frames adapted from [12] with drastically
different structural performance. By most distance metrics, these bike frames
would be the most similar designs among a dataset of thousands. Yet, due to
the disconnected geometry highlighted, they experience deflections that differ
by over two orders of magnitude when subjected to the first loading scenario
in [12].

We broadly divide metrics into five categories which each address an important
facet of engineering design. The first of these is similarity, the class of
metric commonly used to evaluate DGMs. Though often overused, similarity is
still important, and we present several perspectives and metrics to consider
various aspects of similarity. The second category is diversity and novelty,
the idea that generated designs are often desired to be unique and varied.
Since novelty is often valued in design tasks, models should accordingly be
evaluated on their ability to generate novel and varied designs. The third
category is constraint satisfaction, the idea that generated designs must
often adhere to a set of explicit or implicit constraints in order to be
considered a valid design. The fourth is design performance, the idea that
generated designs often have associated functional performance attributes that
designers want to optimize. Designers may also want generated designs to meet
specific performance targets, necessitating a related set of metrics
pertaining to performance target achievement. Finally, the last category is
conditioning, the idea that conditional generative models should respect and
adhere to their conditioning information. There are several broader areas of
DGM evaluation that we deem out of scope and do not cover. These areas include
computational cost, data requirements, memory requirements, bias and fairness,
and privacy. Additionally, although we discuss a wide variety of evaluation
criteria within the five focus areas, this is not an exhaustive review.

##  2 Relevant Reviews of Evaluation Metrics

Several papers have discussed and contrasted methods for evaluating DGMs,
which are briefly discussed here. We will first mention a few review papers in
the dominant research thrusts for generative models, such as computer vision
and natural language processing. We will then discuss several papers with
noteworthy reviews of evaluation metrics relevant to DGMs, but not yet applied
to DGMs in engineering and design. Finally, we will discuss several papers
which partially address evaluation metrics for DGMs in specialized design
disciplines. To our knowledge, the existing body of research lacks a dedicated
discussion of applicable metrics for DGMs which generally apply across design
disciplines.

#### Reviews of Metrics in Vision and Natural Language

Image generation [2, 1, 13, 14, 15] and natural language generation [7, 8, 9],
remain dominant research thrusts for DGMs. As such, evaluation metrics for
generative models in these domains have received much attention and careful
consideration. Borji [16, 17] provides two complementary reviews which outline
a few evaluation metrics for GANs, though most of these metrics generalize to
other generative models in computer vision problems. Gatt & Krahmer [18] and
Dong et al. [19] review the state of natural language generation, including
detailed discussions of evaluation metrics. While the metrics discussed in the
mentioned papers focus on statistical similarity, this paper argues that
thinking beyond statistical similarity to include factors such as performance,
diversity, and constraints is crucial for design and engineering.

#### Reviews in Optimization and Design Methodology

Engineering optimization is a closely-related design discipline to generative
models. Metrics in this domain are often used to compare different
optimization algorithms or to quantify the strength of a set of optimized
solutions. As a result, they typically consider objective (or performance)
evaluation, constraints, and diversity of solutions. Riquelme et al. [20]
review evaluation metrics for multi-objective optimization, documenting their
popularity in optimization research and classifying them by type. Though
optimization metrics do not typically cover statistical similarity to a
training dataset, in this paper, we show that many evaluation metrics
developed for single and multi-objective optimization algorithms are
nonetheless relevant to DGMs and should be adopted by researchers.

Many design methodology papers have proposed and explored methods to
quantitatively evaluate specific aspects of designs. For example, Mueller &
Ochsendorf [21] analyze numerous metrics for design diversity quantification.
While useful, these design metrics are often set up to evaluate “generic”
designs and may not be sufficient to measure “AI” generated designs. In
particular, many metrics struggle with the extreme non-convexity and high
dimensionality of the complicated design spaces typically learned by DGMs.
This makes their use as an evaluation metric challenging.

#### DGM Evaluation Metrics in Engineering

Within certain specialized design disciplines, portions of papers have
included a dedicated discussion about evaluation metrics for DGMs. For
example, Bilodeau et al. [22] present an overview of evaluation methods for
generative models in their review of generative models for molecular
discovery. Shah et al. [23] review evaluation methods for generative models in
synthetic microstructure images. Lastly, Regenwetter et al. [12] propose
several performance-aware evaluation metrics for design problems.

#### Contributions

In the context of existing work, this paper has the following contributions:

  1. 1.

We provide a structured review and practical guide on metrics used to evaluate
deep generative models on engineering design problems.

  2. 2.

We highlight the fallacies of statistical similarity and propose a suite of
metrics to evaluate design exploration, design constraints, design quality,
and conditioning requirements. For each category, we propose multiple metrics,
many of which originate in other disciplines, such as multi-objective
optimization, natural image generation, and molecule synthesis.

  3. 3.

We train numerous deep generative models, including GANs, VAEs, MO-PaDGANs,
DTAI-GANs, cVAEs, and cGANs, on two-dimensional examples and real-world design
problems to demonstrate how deep generative models can be evaluated in
different circumstances.

  4. 4.

We present a design case study on bike frame synthesis focusing on
performance, diversity, and constraints and discuss how to select and apply
appropriate metrics.

##  3 Background

Many data-driven design practitioners are intimately familiar with the
challenges of model selection. Say a designer wants to train a model to
generate a set of optimal structural topologies that meet several
manufacturability constraints and performance targets. The designer trains a
BigGAN [15] and a StyleGAN2 [14] and generates thousands of design options
using each model. How should the designer select which model to use? The go-to
evaluation method for many practitioners using these types of GANs would be
the Fréchet Inception Distance (FID) [24], a metric that attempts to quantify
similarity to the training dataset. FID can indicate which sets of designs
generated by a model are most similar to the dataset, but it doesn’t provide
information on how well they meet specific design constraints, performance
targets, or diversity goals. Additionally, FID is calibrated for natural image
datasets, so it may not accurately measure similarity for other types of data
or structures.

For design researchers looking to train deep generative models (DGMs),
commonly used evaluation metrics should be revisited. In this paper, we
present a structured guide to help researchers select better evaluation
metrics for design problems, focusing on five main facets: similarity,
diversity, constraints, design performance, and conditioning. However, before
introducing individual metrics in Section 4, we will briefly discuss some
background concepts. First, we will discuss decisions on terminology
throughout the paper. Next, we will discuss common requirements and
prerequisites needed for evaluation, such as distance calculation and
hyperparameters. Finally, we will discuss classifications to categorize
metrics, such as point vs. set, and unary vs. binary. We include a tabular
summary of the metrics, their requirements, and their categorizations in Table
1.

Table 1: Overview of metrics discussed in the paper. For each metric, we
discuss: 1. The requirements and auxiliary computational cost necessary to
evaluate the metric (key: Aux – Auxiliary predictive task & training, CL –
Clustering method, CFC – Closed-Form Constraints, Cond – Condition parameter
calculation, Const – Constraint violation test, Dist – Distance metric, DP –
Differentiable method to calculate design performance, Emb – Vector embedding,
Inv – Dataset of invalid designs), Perf – Method to calculate design
performance. 2. Whether the metric characterizes points or sets of generated
samples. 3. Whether the metric uses a reference set (binary) or not (unary).
4. Relevant spaces to which the metric can reasonably apply. 5. Whether the
metric depends on hyperparameters. 6. The metric’s bounds. 7. The optimal
direction of the metric. Notes: *Assuming L2 Norm (Euclidean) Distance Kernel.
Distance Kernel impacts hyperparameters, bounds, and objective. For example,
switching to an RBF kernel would result in bounds of $[0,1]$ ($[-1,1]$ for
signed distances), flipping the direction of the objective, as well as adding
a hyperparameter to tune. ** Depending on the clustering method used, a
distance metric may suffice instead of a full embedding. *** Problem
parameters such as reference points or sets, objective weights and DTAI
priority parameters are not considered to be hyperparameters.

Category | Metric: |  | Requirements/
---
Eval Costs:
| Point/
---
Set:
| Unary/
---
Binary:
Space: |  | Hyperpara-
---
meters***:
| Bounds/
---
Values:
Direction:
Similarity and Distribution Matching | Statistical Distance/Divergence | Depends | Set | Binary | Both | Depends | $[0,\infty]$ | Minimize
Precision-Recall Curves | Emb**, CL | Set | Binary | Both | Yes | N/A | N/A
Precision ($F_{\beta<<1}$) | Emb**, CL | Set | Binary | Both | Yes | $[0-1]$ | Maximize
Nearest Datapoint* | Dist | Point | Binary | Both | No | $[0,\infty]$ | Minimize
Recall ($F_{\beta>>1}$) | Emb**, CL | Set | Binary | Both | Yes | $[0-1]$ | Maximize
Nearest Generated Sample* | Dist | Set | Binary | Both | Yes | $[0,\infty]$ | Minimize
Rediscovery* | Dist | Set | Binary | Both | Yes | $[0,\infty]$ | Minimize
ML Efficacy | Aux | Set | Binary | Both | Yes | Varies | Varies
Novelty & Diversity | Inter-Sample Distance* | Dist | Point | Unary | Both | No | $[0,\infty]$ | Maximize
Nearest Datapoint* | Dist | Point | Binary | Both | No | $[0,\infty]$ | Maximize
Distance to Centroid* | Emb | Point | Unary | Both | No | $[0,\infty]$ | Maximize
Entropy | Depends | Set | Unary | Both | Depends | $[0,\infty]$ | Minimize
DPP Diversity Score | Dist | Set | Unary | Both | Yes | $[0,\infty]$ | Minimize
Smallest Enclosing Hypersphere | EVAEmb | Set | Unary | Both | No | $[0,\infty]$ | Maximize
Convex Hull | Emb | Set | Unary | Both | No | $[0,\infty]$ | Maximize
Design Constraints | Constraint Satisfaction | Const | Point | Unary | N/A | No | {0,1} | Maximize
Constraint Satisfaction Rate | Const | Point | Unary | N/A | No | $[0-1]$ | Maximize
Signed Distance to Constraint Boundary* | CFC | Point | Unary | Design | No | $[-\infty,\infty]$ | Maximize
Predicted Constraint Satisfaction | Inv, Aux | Point | Unary | Both | Yes | $[0-1]$ | Maximize
Nearest Invalid Datapoint* | Inv, Dist | Point | Binary | Design | No | $[0,\infty]$ | Maximize
Performance and Target Achievement | Hypervolume | Perf | Set | Unary | Perf. | No | 0-k | Maximize
Target Achievement | Perf | Point | Unary | Perf. | No | {0,1} | Maximize
Target Achievement Rate | Perf | Point | Unary | Perf. | No | $[0-1]$ | Maximize
Signed Distance to Target* | Perf | Point | Unary | Perf. | No | $[-\infty,\infty]$ | Maximize
Design Target Achievement Index | Perf | Point | Unary | Perf. | No | 0-1 | Maximize
Generational Distance* | Perf | Point | Unary (?) | Perf. | No | $[0,\infty]$ | Minimize
Inst/Cum. Optimality Gap* | DP | Point | Unary | Perf. | No | $[0,\infty]$ | Minimize
Conditioning | Conditioning Adherence* | Cond | Point | Unary | Both | No | $[0,\infty]$ | Minimize
Conditioning Reconstruction | Aux | Point | Binary | Both | Yes | $[0,\infty]$ | Minimize

###  3.1 A Note on Terminology

In this paper, we broadly use the term ‘metrics’ to loosely refer to
‘evaluation criteria,’ not specifically distance metrics. To avoid loss of
generality, we broadly use the term ‘samples’ to refer to the output of a
generative model. We also refer to the original data entries as ‘datapoints.’
Though in generative design problems, ‘datapoints’ are often existing designs
and ‘samples’ are often generated designs, this may not be universally true.
In Section 6, we also refer to constraint-violating data entries as ‘invalid
datapoints,’ which typically correspond to existing design concepts that fail
to meet some set of constraints or design requirements.

###  3.2 Calculating Distance Between Designs

Many distance-based metrics presented in this paper require the ability to
calculate distances amongst and in between datapoints and samples. We indicate
which metrics require distance calculations in the requirements column of Tab.
1. Depending on the design representation used, calculating distances between
designs can be a significant challenge. We present some strategies to
calculate distances in different representation schemes in the appendix. In
general, practitioners can choose to directly calculate distances in the
original space and modality of the data, or can instead compute an embedding
over which to calculate a distance.

###  3.3 Hyperparameters

Most metrics depend on some parameters, which must be decided before using the
metric. This can be viewed as an opportunity for practitioners to carefully
consider the parameters they use and to report them clearly when evaluating
models. This allows for fair and consistent comparison of models.
Additionally, by standardizing parameter values, practitioners can ensure that
the evaluation metrics are being used in a fair and unbiased manner. We
indicate which metrics depend on hyperparameters in Table 1.

###  3.4 Point vs. Set Metrics

Certain metrics, such as statistical distance metrics, measure the properties
of a set while other metrics measure the properties of an individual point
(though point metrics can also be aggregated over a set of points to describe
the set). This distinction is particularly significant in design problems
where designers may desire to select a single ‘finalist’ or an elite group of
‘finalists’ from a set of designs generated by a DGM. Only a subset of the
metrics presented will be valuable in evaluating and contrasting individual
designs. Hence, we classify the metrics as point metrics or set metrics in
Table 1.

###  3.5 Unary vs. Binary Metrics

We also distinguish between unary and binary metrics. Unary metrics, such as
hypervolume, generate a score based on a single design or a set of designs. In
contrast, binary metrics such as statistical divergence utilize an additional
reference set in generating a score. Often, this reference set is the dataset
itself. Hence, we classify the metrics as unary or binary in Table 1.

###  3.6 Design Spaces and Performance Spaces

In design, we often frame a design as a point in some design space, without
loss of generality across data modalities. However, we also often care about
the functional performance of generated designs. We can similarly frame a
design’s multi-objective performance as a point in some multi-dimensional
performance space. In Table 1, we indicate which metrics make sense to apply
to the performance space and which make sense to apply to the design space.

##  4 Evaluating Statistical Similarity

Having touched on some broad classifications of metrics, we begin our detailed
discussion with the first of our five main categories: similarity. In classic
machine learning theory and many classic DGM tasks, such as image generation
and natural language generation, deep generative models have a single
overarching objective: To generate convincing samples that are new, but
generally indistinguishable from the dataset. We typically frame this as a
distribution-matching problem, i.e. is the distribution over generated samples
identical to the distribution over the dataset? Accordingly, the dominant
evaluation metrics in both image synthesis (FID [24], IS [25], KID [26], etc.)
and natural language generation (ROUGE [27], BLEU [28], METEOR [29], etc.)
have focused on similarity. Historically, similarity has also been the central
objective of deep generative models in design tasks [3], since it enforces
that any generated designs fall under the same general category as the designs
used to train the model. For example, if a hypothetical design practitioner
trains a model on a dataset of cars, similarity should enforce that their
model generates new car designs, rather than motorcycles (or simply
unrecognizable nonsense). However, in practice, the designer probably doesn’t
want to generate just any car, but rather a novel and functional car that
meets a variety of constraints and performance objectives. Broadly speaking,
mimicking the dataset is often not the only desirable objective in design.
Accordingly, designers must decide how to trade off similarity against other
design objectives, depending on the unique requirements of their problem. In
this section, we discuss metrics and tools to evaluate similarity and reserve
discussion about other objectives for later sections.

(a) Precision-Recall Curves

(b) Nearest Datapoint

(c) Nearest Gen. Sample

(d) Rediscovery

Figure 2: Illustrations of select similarity-related metrics.

#### Statistical Divergence/Distance Metrics

Quantifying the discrepancy between two random variables is one of the central
themes in machine learning, particularly in ML-based generative modeling [2,
1, 30, 31, 32, 33]. Within the field of deep generative modeling, the two
classes of statistical discrepancy measures have been popular, namely
$\phi$-divergences and Integral Probability Metrics (IPMs). Let $\mathbb{P}$
and $\mathbb{Q}$ be two probability distributions on a measurable space $M$,
such that $\mathbb{P}<<\mathbb{Q}$. Then, $\phi-$divergence is defined as,

$\displaystyle
D_{\phi}(\mathbb{P},\mathbb{Q})=\int_{M}\phi\Big{(}\frac{d\mathbb{P}}{d\mathbb{Q}}\Big{)}d\mathbb{Q}.$
(1)

Here, $\phi:\mathbb{R}^{+}\mapsto\mathbb{R}$ is a convex function such that
$\phi(1)=0$. A popular example of this class is the Kullbeck-Leibler (KL)
divergence, where $\phi(t)=t\log(t)$. Similarly, IPMs can be defined as

$\displaystyle
D_{\mathcal{F}}(\mathbb{P},\mathbb{Q})=\sup_{f\in\mathcal{F}}\Big{|}\int_{M}fd\mathbb{P}-\int_{M}fd\mathbb{Q}\Big{|}.$
(2)

A popular member of this class is the Maximum Mean Discrepancy (MMD), where
$\mathcal{F}$ is set to be the Reproducing Kernel Hilbert Space (RKHS). An
intuition behind these two classes of metrics is that the first one tries to
measure the ratio (where a ratio of one corresponds to identical sets), while
the second one measures the distance between two distributions (where a
distance of zero corresponds to identical sets).

Naturally, statistical distances are the optimal choice to measure the
similarity of the generated sample distribution to the true data-generating
distribution. However, for most high-dimensional problems of interest,
computation of such statistical similarity measures is intractable as it
requires estimation of the densities induced by the distributions. This has
led to the development of estimators of these measures [34, 35, 36, 37, 38]. A
popular class of plug-in estimators includes pre-training a classifier [37,
38, 36] to first estimate the log-ratio of the densities and then taking its
Monte-Carlo expectation. Another class of estimators relies on neural density
estimation [39], where individual densities are estimated using highly non-
linear bijective functions [40, 41] and then used to estimate the discrepancy.
Similar projection-based approaches also exist for IPMs [34, 42]. However, the
efficacy of such estimators is dependent on the modality and the
dimensionality of the problem’s data. Therefore domain-agnostic estimators of
statistical similarity remain largely an open problem.

Consequently, many of the leading similarity metrics are domain-specific,
leveraging certain advantages of the domain to calculate. In computer vision,
for example, countless domain-specific metrics have seen widespread adoption
[16]: Fréchet Inception distance (FID) uses a pre-trained Inception network to
calculate vector embeddings for images, assumes a Gaussian distribution over
this embedding space, then trivially calculates Wasserstein-2 Distance
(Fréchet distance under the Gaussian assumption) between the generated and
dataset distributions in this embedding space [24]111Other popular metrics
calculate different IPMs, such as Maximum Mean Discrepancy in Kernel Inception
Distance (KID) [26]. Other variants like Inception Score (IS) use the
statistical distance between marginal and true label distributions [25].. In
text generation methods, the commonly used perplexity metric is also a close
relative of statistical distance. Perplexity is defined as the exponential of
cross-entropy, which itself is the entropy of the data distribution plus the
KL divergence between the generated distribution and the data distribution.

FID and perplexity have been empirically found to correlate well with the
human perceptual evaluation of image and text realism. Since human perceptual
evaluation is relatively uniform in computer vision and natural language, FID
and perplexity are among the most commonly used metrics for model evaluation
in their respective fields. In other fields, like design, the perception of
realism is much more varied. This non-uniformity, alongside other challenges
like data modality, may preclude any generalizable design-specific statistical
distance metric from rising to prominence.

Instead, practitioners should evaluate the viability of applying domain-
specific methods to design problems on a case-by-case basis. For example, when
evaluating similarity in a structural topology generation problem, one may
consider using image-based metrics like FID, KID, and IS, which utilize a pre-
trained image classifier. These pre-trained image classifiers are often
trained on ImageNet, a large computer vision dataset [43]. Accordingly, these
metrics are highly biased towards ImageNet [44]. For example, even when
evaluating models trained on CIFAR-10, a very similar natural image dataset,
researchers have found that inception score significantly misrepresented model
performance [45]. At the beginning of Section 3, we shared an anecdote about a
practitioner considering whether to evaluate generated structural topologies
using FID. In a case like this, it is uncertain whether the low-dimensional
latent representations extracted from a network trained on animal or food
images in ImageNet would contain any useful information about structural
topologies, making metrics like FID a dubious choice for this type of problem.
Therefore, practitioners must be cautious about applying image-based metrics
in other domains and carefully evaluate their suitability.

In general, statistical distance metrics are excellent tools to evaluate
similarity, but are often challenging to estimate in high-dimensional
problems. They are often necessary for many design generation problems, to
ensure that the generated designs are similar to the training data. However,
when reliably estimating statistical distance is infeasible or when
practitioners desire more nuance in evaluating similarity, they can instead
turn to a variety of other methods, which we present in the following
subsections.

###  4.1 Decoupling ‘Realism’ and ‘Coverage’

Researchers often point to a key shortcoming of statistical distance metrics,
namely their inability to decouple two separate ideas in distribution
matching: ‘realism’ and ‘coverage.’ Realism is the idea that generated designs
should resemble the dataset222The term ‘realism’ can be misleading since it
implies that the dataset reflects reality (i.e., covers the entire space of
‘realistic’ data), which is typically untrue. Coverage is the idea that the
entire spread of the dataset should be represented by generated designs. While
any model that achieves perfect (zero) statistical distance must achieve both
perfect realism and coverage, an imperfect model can suffer from an unknown
balance of imperfect realism or coverage, which is difficult to diagnose with
only a single score. To combat this, methods like precision-recall curves
analyze generative models with an entire precision-recall tradeoff front
between these two factors.

#### Precision-Recall Curves(Fig. 2(a))

Precision-recall curves are borrowed concepts from supervised classification
but have been adapted as metrics for generative models. The concept was
originally proposed by Lucic et al. [46] as a method to benchmark generative
models on datasets with known data manifolds. The concept was extended to data
with unknown distributions by Sajjadi et al. [47], allowing for practical
evaluation of DGMs on ‘real’ datasets. In Sajjadi et al.’s framework,
generated data and original data are pooled, then clustered using k-means.
From the discrete distributions of the generated data and the training dataset
across bins, they calculate a precision-recall (PR) curve. Other methods have
also been proposed to generate PR curves from arbitrary distributions,
bypassing the need for discrete binning of the data [48]. This curve is
calculated by sweeping over a cutoff parameter which effectively dictates what
proportion of a cluster must belong to the generated or original data
distribution to be considered part of that distribution’s support. From the
curve, summary scores like Area-Under-the-Curve (AUC) and a maximum $F_{1}$
score can also be derived. We refer the reader to [47] for mathematical
reasoning and visual examples behind the PR curves for generative models.

###  4.2 Similarity of Generated Data to Dataset (‘Realism’)

Although precision-recall curves demonstrate how a DGM model tends to balance
realism and coverage, in some cases, we may care more for one or the other. In
problems where we particularly care about realism, we can calculate a ‘biased’
version of the $F_{1}$ score to estimate precision or use a simple distance-
to-dataset metric.

#### Precision

Precision for generative models captures the fraction of generated designs
falling within the support of the dataset. Extracting singular precision
values from PR curves is challenging since it’s unclear which value to select,
or how to average values. Instead, Sajjadi et al. [47] propose using the
maximum $F_{\beta}$ score for some $\beta<<1$ over all precision-recall pairs
in the PR curve as a proxy for precision.

#### Nearest Datapoint (Figure 2(b))

For a simple estimate of ‘realism,’ practitioners can calculate the distance
to the nearest datapoint for every generated sample. Despite its simplicity,
the score is an effective method to capture local proximity to the dataset.
Nearest datapoint can also be averaged over a set of samples to estimate the
‘realism’ performance of a DGM. This metric is particularly important for
applications of DGMs in data augmentation, as it demonstrates that generated
synthetic designs (samples) are similar to a dataset of existing designs
(datapoints).

###  4.3 Dataset Coverage

In some design applications, there might be a need to focus more on design
space coverage to ensure that all key modalities of the design space are
reflected in generated designs. In such cases, practitioners can instead
estimate recall or use other metrics to quantify coverage.

#### Recall

Like precision, recall is a metric introduced by Sajjadi et al. [47]. Though
extracting singular recall values from PR curves is similarly challenging,
practitioners can select some maximum $F_{\beta}$ score for some $\beta>>1$ to
estimate recall.

#### Nearest Generated Sample (Figure 2(c))

For a simple approach to capture dataset coverage, practitioners can calculate
the distance to the nearest generated sample for every original datapoint.
This score can be averaged over the dataset to evaluate a set of generated
samples. However, this score depends on the size of the generated sample set,
necessitating the selection of a set size tuning parameter for
standardization.

#### Rediscovery (Figure 2(d))

Rediscovery is an evaluation technique that evaluates an algorithm’s ability
to rediscover datapoints that were withheld during training. It has been used
in the molecule synthesis domain [22], to check whether a DGM could rediscover
a known molecule that was withheld from the training data. However, we
hypothesize that rediscovery can be a useful metric for many other design
problems. While rediscovery is originally used in discrete problems where the
rediscovery rate can be calculated as the exact proportion of withheld designs
rediscovered in a generated set [23], the metric can easily be relaxed to
support other data modalities. Instead of calculating a proportion of
datapoints exactly rediscovered, practitioners can instead calculate the
distance from a withheld datapoint to the nearest sample in a generated sample
set. Effectively, this performs a nearest generated sample evaluation over the
set of withheld designs. As such, this metric also depends on the size of the
generated set, necessitating a tuning parameter. Though it requires the
foresight to remove a split of the dataset before training, rediscovery
actually extends beyond the notion of pure ‘coverage’ and further quantifies
simple generalization capabilities of the model. As such, we feel that this
metric deserves consideration in other design disciplines beyond molecular
design.

###  4.4 Evaluating Effective use of Generated Data in Downstream Tasks

A common approach to measuring the similarity of a generated sample set to the
training dataset is to check whether the generated set serves as an effective
stand-in for some auxiliary task. If a practitioner is generating design data
with a specific downstream task in mind, it may be viable to directly evaluate
generated samples on this task as a metric. In other scenarios, artificially
constructed tasks can serve as an effective method to evaluate generated
samples.

#### ML Efficacy

Auxiliary machine learning tasks, such as classification, are often used to
evaluate generated sample sets in a process known as ML efficacy testing.
Several variations of these scores have been proposed, as in [49, 50]. Often a
supervised machine learning model is trained on the generated dataset and then
tested on the original data. Ideally, the performance on the original data
should be as strong as possible. Alternatively, one model is first trained on
the original dataset and tested on a split of the original data. Then, an
equivalent model is trained on a set of generated samples but still evaluated
on the same split of original data. In this case, the difference in
performance constitutes the ML efficacy score, which is minimized in the ideal
case where the two models have comparable performance.

###  4.5 Demonstration of Statistical Similarity Metrics

Thus far, we have introduced a variety of metrics to quantify a model’s
ability to generate distributions of designs that match the training dataset.
To showcase the use of these metrics, we evaluate two classic generative
models, a Variational Autoencoder and a Generative Adversarial Network333The
goal of this example is to compare any two DGM models; hence, the state-of-art
models and strong instantiations were intentionally not chosen as they
performed very well on a two-dimensional case. on a synthetic dataset. Details
on model architecture, metrics settings, and training are included in the
appendix. This synthetic example is commonly used in literature to compare
deep generative models [51]. Plots of the generated data are shown in Figure
3. Gray points denote training data, and red points denote samples. Scores for
a few selected metrics for these two models are shown in Table 3. If a
practitioner was using a single distribution-matching metric, such as MMD, to
compare the two models, they would think that the GAN is the stronger
performer among the two. However, when looking at an array of metrics, one can
note that the GAN is the stronger performer in accuracy-based metrics such as
F10 and nearest datapoint (better performance is highlighted by bold text). In
contrast, the VAE outperforms the GAN in many coverage-based metrics such as
nearest generated sample, rediscovery, and F0.1. In overall distribution-
matching metrics, results are mixed, with the GAN outperforming in MMD and
falling behind in F1 and AUC. The GAN also performed significantly better in
machine learning efficacy. Compared to a single metric, this suite presents a
more nuanced picture, and the final model selection could vary based on the
end goal. If coverage is more important, then the VAE may be preferred, while
the GAN may be preferred for realism. However, even the best coverage of the
design space does not guarantee the novelty or diversity of generated designs.
Therefore, in the following section, we introduce and discuss metrics to
evaluate the capabilities of DGMs to generate diverse and novel designs.

(a) VAE

(b) GAN

Figure 3: Distributions generated by a Variational Autoencoder and Generative Adversarial Network (red) overlaid over training data (gray). The GAN dominates in accuracy-related metrics while the VAE outperforms in coverage.  Table 2: Distribution-matching scores for models in Figure 3. The GAN dominates in accuracy-related metrics while the VAE outperforms in coverage. Point metrics are averaged over the generated set. (Avg.) Scores: | VAE | GAN
---|---|---
Nearest Datapoint | 0.043 | 0.018
Nearest Generated Sample | 0.090 | 0.100
Rediscovery | 0.093 | 0.099
Precision-Recall Curve F1 | 0.475 | 0.386
Precision-Recall Curve F10 | 0.907 | 0.934
Precision-Recall Curve F0.1 | 0.804 | 0.753
Precision-Recall Curve AUC | 0.468 | 0.379
Maximum Mean Discrepancy | 0.045 | 0.013
Machine Learning Efficacy | 0.675 | 0.765

##  5 Evaluating Design Exploration

Design exploration can be an important consideration in many fields, such as
product design, architecture, and engineering, where new and innovative
solutions are often sought after. A model that successfully explores the
design space will typically generate diverse sets of novel designs. Design
novelty refers to the degree to which a design is new or unique compared to
existing designs. Achieving novelty is particularly difficult for data-driven
generative models since it is somewhat contrary to its default objective of
learning and mimicking a dataset. In design methodology literature, novelty is
typically considered a point metric, whereas diversity (or variety) is
typically considered a set metric. In this section, we discuss several metrics
with which to quantify the novelty of generated samples and the overall
diversity of generated sample sets.

(a) Inter-Sample Distance

(b) Distance to Centroid

(c) Enclosing Hypersphere

(d) Convex Hull

(e) DPP Diversity Score

Figure 4: Illustrations of select diversity and novelty metrics.

###  5.1 Novelty

The design methodology community has a rich body of work classifying and
defining novelty. A common distinction is ‘psychological novelty’ (P-novelty)
versus ‘historical novelty’ (H-novelty). A design is considered P-novel if the
idea is new for the person who generated it. In contrast, a design is H-novel
if an idea has never appeared in history before [52]. If we treat our
generative model as the designer and consider the dataset it has seen as the
set of designs that our model knows, we argue that the P-novelty can be
estimated for a DGM. However, since we can’t assume that our dataset is
comprised of every design ever conceptualized, estimating H-novelty is
challenging, even if we use the dataset as a reference distribution.

#### Inter-Sample Distance (Figure 4(a))

A simple metric to measure the P-novelty of a generated sample is the distance
to the nearest other generated sample. In this simple form, the metric is a
strong assessment of local novelty. However, the metric has also been relaxed
to use the distance to the $n^{th}$ nearest neighbor [53].

#### Nearest Datapoint (Figure 2(b))

The nearest datapoint metric (previously introduced as a similarity metric in
Section 4.2) is a P-novelty metric, measuring how far from the nearest
datapoint a generated sample is. Whereas the metric was best minimized as a
metric for accuracy, when used to evaluate novelty, a larger score is
preferable. Nearest datapoint is a very insightful metric when checking for
data copying, where the DGM will overfit and memorize individual datapoints to
replicate while sampling. One limitation of the metric is its large
sensitivity to individual datapoints. Nevertheless, the nearest datapoint
metric has been used in design literature [51] to demonstrate the capability
of deep generative models to create novel designs.

#### Distance to Centroid (Figure 4(b))

Instead of using the distance to the nearest samples, practitioners can
instead quantify the novelty of a generated sample using the distance from the
sample to a single point which summarizes a set of generated samples, such as
the centroid or the geometric median. As [54] notes, the choice of centroid or
median can yield very different results. Though simple and cheap to compute,
the metric may carry implicit assumptions about convexity. For example, in a
torus-like distribution, the centroid may in fact be quite novel. [21] propose
a variant to calculate diversity using the maximum distance to the centroid or
median. Next, we discuss other dedicated metrics for diversity.

###  5.2 Diversity

Design diversity is closely related to the concept of “entropy” in information
theory and refers to the variety or range of different solutions or designs
that are generated for a given problem. In the context of design problems, it
can refer to the degree to which a set of solutions to a problem encompasses
different styles, forms, or variations. While novelty is a point metric,
diversity measures a property of a set of designs, though many averaged
novelty metrics may often closely correlate with diversity.

Diversity can be broken down into two components: uniformity and spread.
Uniformity measures the relative distance between designs. Spread, also known
as extent in multi-objective optimization literature, measures the range of
designs within the generated distribution [20]. Imagine the design space as a
balloon with small balls inside it. The spread can be thought of as the
diameter of the balloon. A bigger balloon will have a higher extent diversity
score. However, two balloons with the same diameter may have different
uniformity diversity scores, for example, if one balloon has all the balls
evenly distributed, while the other has most of the balls stuck in one corner.
Many diversity metrics, such as entropy, combine uniformity and spread into a
single value. Below, we discuss a few of these diversity metrics444Averaged
novelty metrics are also often more focused on either spread or uniformity.
For example, averaged inter-sample distance and averaged nearest datapoint are
often strong measures of uniformity, whereas averaged distance to centroid is
more focused on spread..

#### Entropy

In general, entropy scores are a metric for design diversity when evaluated on
a set of generated designs. Some popular metrics include the Shannon entropy
index, Herfindahl–Hirschman Index (HHI) [55], Gini-Simpson index, and inverse
Simpson index [54]. When working with continuous representations, since
practitioners typically only have access to samples, they must estimate
entropy from samples, a well-studied statistical problem [56]. When
practitioners do have discrete data, they can directly calculate the entropy.
Entropy captures both uniformity and spread, and its properties are well-
grounded in mathematics and information theory.

#### Smallest Enclosing Hypersphere (Figure 4(c))

The smallest enclosing hypersphere metric is a purely spread-focused metric
that identifies the hypervolume of the smallest hypersphere that encloses all
generated samples. Proposed by [57] for novelty measurement, the calculation
is nontrivial for high-dimensional data and is often approximated to reduce
cost [54]. Smallest enclosing hypersphere is highly sensitive to relative
scaling between parameters. It also makes a convexity assumption and is
sensitive to outliers.

#### Convex Hull (Figure 4(d))

The convex hull is another spread-focused metric defined as the smallest
convex set that includes a set of generated samples. [58] discusses the
application of convex hulls to diversity measurement. The total hypervolume
enclosed within the generated set’s convex hull is a common metric for
diversity. [54] found convex hull to agree with human assessment of novelty in
small low-dimensional design problems, compared to competing metrics. Like
smallest enclosing hypersphere, convex hull makes a convexity assumption and
is often sensitive to outliers.

#### DPP Diversity Score (Figure 4(e))

Determinantal Point Processes (DPP) can be used in conjunction with distance
metrics to evaluate a diversity score of a generated sample set. DPPs
calculate a score based on the eigenvalues of a matrix constructed using
distances between points from a generated sample set [59]. Like entropy, DPP
diversity captures both uniformity and spread. The benefit of DPP is its ease
of calculation for high-dimensional data, as it only requires a positive-
semidefinite kernel as an input. However, it is also sensitive to design
duplicates, as the determinant collapses to zero when any of the eigenvalues
is zero.

###  5.3 Demonstration of Design Exploration Metrics

In this section, we introduced a variety of diversity and novelty-related
metrics. To demonstrate their performance, we use the same representative
synthetic data problem introduced in Figure 3 to demonstrate diversity and
novelty-related metrics. At first glance, we might expect the VAE to have much
higher diversity and average novelty across the board since the generated
samples are more spaced apart and span regions of the space outside of the
dataset. Indeed, the VAE dominates in novelty metrics focused on uniformity,
like nearest datapoint and inter-sample distance. However, the GAN outperforms
the VAE in spread-related diversity and novelty metrics, like convex hull and
distance to centroid because samples generated by the GAN span a larger convex
space. This illustrates a potential shortcoming with certain metrics, where
the potentially novel samples in the center of the space generated by the VAE
do not contribute to the score. Therefore, if the goal is to generate designs
that are evenly distributed across the design space, the VAE model may be
preferred. However, if the goal is to generate designs that push the
boundaries of the design space, the GAN model may be preferred. The DPP
diversity score, which takes into account both uniformity and extent, slightly
favors the GAN model when using a Euclidean Radial Basis Function kernel. As
illustrated by this two-dimensional example, practitioners need to consider
both uniformity and spread and be aware of convexity assumptions when
evaluating a model’s diversity capabilities for a comprehensive understanding.
Next, we proceed to consider metrics that evaluate a model’s adherence to
design constraints.

Table 3: Diversity and novelty scores for generated distributions from Figure 3. The VAE dominates in average novelty, while the GAN generates in the more global diversity metrics. Point metrics are averaged over the generated set. (Avg.) Scores: | VAE | GAN
---|---|---
Nearest Datapoint | 0.043 | 0.018
Inter-Sample Distance | 0.030 | 0.019
Convex Hull | 5.651 | 7.173
DPP Diversity | 14.398 | 14.688
Distance to Centroid | 1.120 | 1.391

##  6 Evaluating Design Constraints

Many design problems have constraints, which are restrictions or limitations
placed on the possible designs within a given problem or task. These
constraints can be physical, such as materials or manufacturing limitations,
or functional, such as performance or cost requirements. They are used to
guide the design process and ensure that the final solution meets certain
requirements and is feasible to implement. Unlike performance targets, which
we discuss in Section 7.2, we must satisfy constraints for the design to be
valid. Appropriately handling constraints is extremely important for
generative models in design, and ensuring that DGMs can respect design
constraints is a critical consideration in evaluating their performance. In
this section, we propose several techniques for quantifying constraint
satisfaction in DGMs.

(a) Constraint
Satisfaction

(b) Constraint
Satisfaction Rate

(c) Predicted Constraint
Satisfaction

(d) Nearest Invalid
Datapoint

(e) Signed Distance to
Constraint Boundary

Figure 5: Illustrations of select constraint satisfaction metrics.

###  6.1 Leveraging Constraint Violation Tests

In many design problems, there exist known methods (using analytical
equations, rules, simulations, or physics knowledge) to test whether a design
is valid with respect to each of the problem constraints. This gives
practitioners the ability to directly evaluate generated designs. Below, we
list a few methods that can be used to measure the adherence of generated
samples to these constraint violation tests.

#### Constraint Satisfaction (Figure 5(a))

Provided that practitioners have access to some black-box constraint
satisfaction test, perhaps the simplest possible constraint violation metric
is a simple binary constraint satisfaction score (i.e., does a generated
sample simultaneously meet all constraints?). When averaged over a generated
sample set, this can effectively serve as an indicator for the proportion of
generated samples that are valid. In practice, more versatile scores are often
more informative, especially in problems with multiple constraints.

#### Constraint Satisfaction Rate (Figure 5(b))

A variant of the simple constraint satisfaction score is the constraint
satisfaction rate, which quantifies the proportion of all constraints met by a
single generated sample. If different constraints have different priority
weightings, this score can be weighted by the priority of the various
constraints.

###  6.2 Leveraging Mathematically-Defined Constraint Boundaries

When practitioners have access to a closed-form mathematically-defined
constraint boundary, they can calculate distances from generated design
samples to the constraint boundary.

#### Signed Distance to Constraint Boundary: (Figure 5(e))

In some design applications, it may be possible to calculate a distance to the
constraint boundary. The distance to constraint boundary metric is most
informative as a signed distance field (SDF), with generated samples
satisfying the constraints having positive distances and samples violating the
constraints having negative distances. Indicating by what margin samples are
satisfying or violating constraints can be significantly more informative than
a simple binary criterion or proportion of constraints met.

###  6.3 Metrics that Leverage Datasets of Invalid Designs

In some design applications, one may not have a black box constraint evaluator
or methods allowing the direct estimation of the distance to the constraint
boundary. In contrast, practitioners may have a large collection of infeasible
or invalid designs. These datasets of constraint-violating (invalid) designs
provide a tool to approximate the constraint adherence of generated designs.

#### Predicted Constraint Satisfaction

When practitioners have access to a reference set of constraint-violating
datapoints, they can use a classifier to predict the constraint satisfaction
of their generated samples. This classifier can be something complex like a
neural network, or something simple and robust like k-nearest neighbors. In
the case of k-nearest neighbors, the fraction of neighbors that are valid
provides a likelihood that a generated sample satisfies constraints, which
serves as the score, as shown in (Figure 5(c)).

#### Nearest Invalid Datapoint (Figure 5(d))

When practitioners have access to a reference set of constraint-violating
datapoints, they can calculate the distance from each generated sample to the
nearest known invalid datapoint. The underlying assumption is that samples
near constraint-violating datapoints are also likely to be constraint-
violating. This gives a rough approximation of the distance to the constraint
boundary.

###  6.4 Demonstration of Design Constraint Metrics

Having presented a variety of metrics for evaluating constraint satisfaction,
we again showcase these metrics on a simple 2-dimensional problem. To
demonstrate the constraint adherence task, we test a GAN and a VAE on a
variant of a concentric ring problem that we created for this task. In this
problem, we create a non-convex feasible design space consisting of a
concentric circle and ring, separated and surrounded by two infeasible
regions, respectively (Figure 6(a)). To support the nearest invalid datapoint
metric, we also include a dataset of invalid datapoints, shown in Figure 6(b).
Both the GAN and the VAE struggle to avoid the invalid area of the design
space, as seen by points overlapping with the infeasible regions in Figures
6(c) and 6(d). However, for this problem, we observe that the GAN outperforms
the VAE in every metric, showing that it is better suited to generate feasible
designs. Note that in cases when these metrics don’t align, the decision could
either depend on the use case or on whether a constraint satisfaction test is
available and whether constraints are closed-form. With this, we conclude our
discussion about constraint adherence metrics and move on to consider metrics
that evaluate design performance.

(a) Valid Datapoints

(b) Invalid Datapoints

(c) VAE

(d) GAN

Figure 6: Distributions generated by a Variational Autoencoder and Generative Adversarial Network. Both the VAE and GAN struggle to observe the non-convex constraint dividing the two valid regions of the design space. Generated samples that violate constraints are shown in black, while valid generated samples are shown in blue. The GAN demonstrates generally higher constraint satisfaction performance. Table 4: Constraint adherence scores averaged over the generated distributions in Figure 6. The GAN demonstrates generally higher constraint satisfaction performance. Point metrics are averaged over the generated set. Average scores: | VAE | GAN
---|---|---
Constraint Satisfaction | 0.713 | 0.833
Constraint Satisfaction Rate | 0.857 | 0.917
Predicted Constraint Satisfaction | 0.698 | 0.823
Nearest Invalid Datapoint | 0.17 | 0.173

##  7 Evaluating Design Quality

In problems like image synthesis, ‘quality’ and ‘realism’ are almost
synonymous. The more realistic the generated images, the higher their
‘quality’ and the higher the performance of the generator. This makes sense
for problems like human face generation, where if a generated image looks like
an existing human face, it might be considered a more realistic and higher-
quality image. In design, ‘quality’ is typically not associated with
similarity to existing designs. Instead, the quality of designs is often
governed by an associated set of functional performance characteristics, often
modeled as a mapping from the design space to some performance space. Let us
consider a bicycle design problem. As demonstrated in Fig. LABEL:twobikes
earlier, two frames that look visually similar in the pixel space still have
orders of magnitude different deflection values. Quality characteristics in
design may include factors such as cost, weight, efficiency, etc., and are
highly problem dependent. We’d like to note that a generated design’s innate
performance attributes can also serve as simple but effective evaluation
metrics for the design and the method that generated it.

(a) Hypervolume

(b) Generational Distance

(c) Optimality Gap

(d) Target Achievement

(e) Target Ach. Rate

(f) Signed Dist. to Target

(g) Design Target Achievement Index

Figure 7: Illustrations of select design quality and target achievement
metrics.

###  7.1 Performance Optimality

In many engineering problems, designers want to find a distributed set of
solutions that are performant across multiple objectives. This is especially
common in multi-objective optimization problems, where designers often seek to
identify a Pareto-optimal set of points. Pareto-optimal designs have the
property that any performance improvement in some objective must come at the
expense of performance in some other objective. This means that Pareto-optimal
designs cannot be ‘dominated’ by any other design, i.e., there exists no
design that has superior performance in every objective. Identifying a strong
approximation set for the Pareto-optimal front can be extremely valuable in
design problems because it effectively provides an optimal design given some
choice of objective priorities by the designer. If practitioners seek to
generate a diverse set of near-optimal designs, they may seek to quantify how
close a set of generated designs is to the true Pareto-optimal front. Below,
we discuss a few metrics that could be adopted by design researchers to
quantify performance optimality.

#### Hypervolume Metric (Figure 7(a))

The hypervolume metric, often simply referred to as ‘hypervolume,’ is a staple
metric in the multi-objective optimization field [20], which estimates the
proximity of a set of generated samples to the (often unknown) Pareto-optimal
front. In simple terms, the metric calculates the hypervolume comprised of all
points that are dominated by some point in the generated set but
simultaneously dominate some fixed reference point. In any problem where a
diverse near-Pareto-optimal set of samples is desired, hypervolume could be
used to quantify the performance of generated samples. For example, the metric
was used in [60] to compare the performance of different GAN models for
airfoil synthesis.

#### Generational Distance (Figure 7(b))

When a set of ‘optimal’ reference designs is known, Generational Distance
could be used to measure design optimality. Another staple of the multi-
objective optimization community [20], generational distance measures the
distance from a generated sample to the nearest point on the ‘optimal’
reference set [61]. This reference set may not consist of truly optimal
designs but is typically taken as a reliable approximation for a true Pareto-
optimal design set. Generational distance is not widely adopted as a metric
for deep generative models but can serve as an excellent metric for evaluating
the quality of generated designs in cases when an optimality frontier is known
or could be constructed from the training data using non-dominated ranking
methods.

#### Optimality Gap (Figure 7(c))

When practitioners don’t have a set of ‘optimal’ reference designs available
but still want to estimate the distance to an optimal design, they can instead
perform optimization on their design problem to attempt to discover those
‘optimal’ designs. The classic optimality gap metric uses gradient-based
optimization, which requires that gradients can be calculated over the
performance scores with respect to design parameters. The distance from the
generated design to the local minimum discovered by the optimizer is often
referred to as the ‘cumulative optimality gap’ or simply the ’optimality gap.’
A variant, the instantaneous optimality gap, has also been proposed to measure
the distance to the modified design after the first step of gradient descent
[62].

###  7.2 Target Achievement

While generating an entire set of Pareto-optimal designs can be helpful when
exact design goals are not yet decided, practitioners may also need to apply
generative models to design problems where performance targets are specified.
These types of problems necessitate a suite of metrics that quantify a model’s
ability to achieve performance targets. We would like to clarify that, in
contrast to hard design constraints, performance targets are intended to be
negotiable. They are also different from soft constraints, as exceeding a
target by a larger margin is often desirable, which is not the case with
constraints. While many of the metrics for constraint satisfaction can be
modified to quantify target achievement, the handling of design targets can
call for more nuanced metrics which reflect their flexibility.

#### Target Achievement Scores

Reformulating several of the previously discussed constraint satisfaction
metrics, we can define analogs for the target achievement case. Target
Achievement (Figure 7(d)), much like the constraint satisfaction score,
measures whether a design simultaneously meets all performance targets across
objectives. However, since simultaneously achieving all the targets in a given
problem may be difficult, more nuanced scores are typically more informative
in quantifying proximity to the target. Target Achievement Rate (Figure 7(e))
is analogous to the constraint satisfaction rate, quantifying the weighted
proportion of multi-objective design targets met by design. When practitioners
have a well-defined target criterion, they can calculate a Signed Distance to
Target (Figure 7(f)), indicating the degree to which designs are outperforming
or underperforming the set of multi-objective design targets.

#### Design Target Achievement Index (Figure 7(g))

In problems with particularly nuanced design goals and targets, practitioners
may want an even more flexible metric than the signed distance to the target.
Regenwetter et. al. [63] proposed the Design Target Achievement Index (DTAI),
which considers the relative importance of targets and the value of continued
optimization beyond the target. The key idea is to aggregate weighted soft
penalties when a target in any of the objectives is not achieved and combine
them with reward functions when the target is exceeded. DTAI is also bounded
and differentiable, making it viable as a training loss in generative design
problems where design performance values are provided in the dataset.

###  7.3 Demonstration of Design Quality Metrics

Having introduced a variety of performance and target achievement metrics for
DGMs in design, we once again present a 2-dimensional example demonstrating
the use of these metrics. We select the KNO1 as the objective function [64]
and add it to the 2-dimensional example discussed previously. We test a
standard GAN, which ignores performance, and a Multi-Objective Performance-
augmented Diverse GAN (MO-PaDGAN) [60], which attempts to generate a diverse
set of high-performing designs. For target-oriented metrics, we select a
fairly demanding performance target of $(0.5,0.5)$, which is only attainable
in small regions of the design space. The model and metric parameters are
included in the appendix, and the results are plotted and tabulated in Figure
8 and Table 5. The MO-PaDGAN largely ignores the two low-performance modes and
outperforms the standard GAN in every performance metric tested. It achieves a
stronger Pareto-optimal set as indicated by the hypervolume metric. It
generates designs that are, in general, closer to the known Pareto-optimal
reference set, as indicated by the generational distance metric. Its designs
achieve a greater fraction of the performance targets, lie closer to the
target boundary, and adhere more closely to the target overall, as indicated
by the weighted target achievement rate, signed distance to target, and design
target achievement index, respectively. These metrics match our intuition and
provide a way to quantify differences between different models: A standard GAN
underperforms in all metrics, as it only maximizes distribution similarity,
without any consideration for other factors such as objectives and targets.
Note that the GAN would dominate the MO-PaDGAN in almost every statistical
similarity metric. In design problems, higher functional performance often
comes at the expense of statistical similarity. Next, having explored metrics
to evaluate performance in DGMs, we move on to consider conditioning.

(a) Objective 1

(b) Objective 2

(c) Target-Ach. Region

(d) GAN

(e) MO-PaDGAN

Figure 8: Visual demonstration of a Generative Adversarial Network and Multi-Objective Performance-augmented Diverse GAN on the 2-dimensional KNO1 objective. The MO-PaDGAN, which considers functional performance samples predominantly from higher-performing modes. Table 5: Performance and target achievement scores for distributions in Figure 8.The MO-PaDGAN, which considers functional performance samples predominantly from higher-performing modes. Point metrics are averaged over the generated set. (Avg.) scores: | GAN | MO-PaDGAN
---|---|---
Hypervolume | 0.460 | 0.571
Generational Distance | 0.215 | 0.315
Weighted Target Ach. Rate | 0.162 | 0.430
Signed Distance to Target | -0.275 | -0.154
Minimum Target Ratio | 0.540 | 0.728
Design Target Ach. Index | 0.324 | 0.433

##  8 Evaluating Conditioning Requirements

In deep generative models, conditioning refers to the process of incorporating
additional information, such as labels or attributes, into the model when
generating new data. For example, in image generation, a deep generative model
can be conditioned on class labels, such as ‘dog’ or ‘cat’, in order to
generate images of specific animals, rather than randomly picking cats or dogs
(or training separate generators for each class of image). Conditional DGMs
are often more robust and data efficient than training many individual DGMs.
To address the conditioning task, conditional variants of many classic
generative models have been proposed [65, 66, 67, 68, 69].

Conditioning is especially important in design tasks. Unlike traditional
optimization approaches, the model is trained only once for all possible
conditions and at test time, a practitioner can query any target condition and
generate designs that meet that specific requirement. In many design tasks,
continuous conditions (as opposed to class conditions) are very common. For
example, in [68], Heyrani et al. generate airfoil shapes conditioned on a
specific lift-to-drag ratio.

In design, conditioning information is typically used to encode extra problem
information, but can sometimes be used used to encode constraints or
performance attributes. For example, it is common to condition a DGM on
boundary conditions (normally thought of as constraints), volume fraction
(often thought of as a functional performance attribute), and structural
loading (not typically considered a constraint or performance attribute) [70,
6].

(a) Conditioning
Adherence

(b) Conditioning
Reconstruction

(c) Conditional Nearest
Generated Sample

Figure 9: Illustrations of select conditioning metrics.

###  8.1 Conditioning Adherence (Figure 9(a))

When working with conditional models, practitioners may want to quantify the
degree to which their model respects these conditioning vectors. In some
problems, practitioners have a condition that they can directly calculate over
the generated samples. In class-conditional problems, this may be the ability
to evaluate the class of generated samples. In continuous conditioning
problems, this may be some parameter that practitioners want to fix to some
target value, such as the volume fraction in a topology generation problem. In
these cases, the distance to the condition is a viable approach to quantify
condition adherence.

###  8.2 Conditioning Reconstruction (Figure 9(b))

The ability to directly evaluate a class or condition parameter for a
generated sample can be difficult. Instead, practitioners must often turn to a
predictor to guess what the original conditioning information for generated
samples should have been. Ideally, in the case of class (discrete)
conditioning, the conditionally generated distribution should exactly match
the corresponding dataset distribution for the class. This problem decomposes
into a distribution-matching problem. Accordingly, practitioners can adapt a
technique used to estimate statistical distance and train a classifier to
predict the class condition based on a generated sample. In the case of
perfect constraint adherence, a predictor trained on the original dataset
should be equally confident making predictions on a hold-out set as on a
generated sample set. In class-conditional natural image generation problems,
the classic FID and IS metrics have been extended in [71]. In continuous
conditioning problems, a regressor can instead be used to predict a
conditioning vector from generated samples, from which any metric for
supervised regression can be leveraged as a conditioning reconstruction
metric.

###  8.3 Adapting Unary Metrics to Conditional Problems

Many of the metrics presented in previous sections must be adapted for
conditional problems. We first discuss strategies to adapt unary metrics to
conditional settings and discuss the more challenging task of adapting binary
metrics in the following section. Unary metrics must be calculated and
averaged over numerous conditions. For a fair comparison, these conditions
must be identical or equivalent when evaluating different models on a
particular problem. For class-conditional problems, this may mean reporting a
score for each class or reporting a weighted average across classes. For
continuous conditioning problems, this may mean averaging over many conditions
sampled from some repeatable parametric statistical distribution.

###  8.4 Adapting Binary Metrics to Conditional Problems

In unconditional problems, binary metrics typically compare some generated
posterior distribution to a prior distribution (often the dataset). In a
conditional setting, our posterior is instead a conditional distribution. In
this case, we typically have two options, each with its own strengths. We can
either compare the conditional posterior distribution to an equivalent
conditional prior distribution (usually the subset of the dataset which meets
the condition, which must often be estimated). Alternatively, we can compare
the conditional posterior distribution to the full marginal prior distribution
(the whole dataset). In either case, much like for unary metrics, we typically
average scores over a sweep of conditions for an overall condition-agnostic
score, or simply report metrics under several individual conditions.

#### Comparing Conditionally Posterior to a Conditional Prior

Comparing a conditional posterior to a conditional prior is the most natural
way to evaluate binary metrics in conditional problems. We have illustrated
the conditional nearest generated sample metric in Figure 9(c)), but this
principle applies broadly to many binary metrics. However, identifying a
conditional prior in conditional problems is often nontrivial. When the prior
is something other than samples from the dataset (a pareto-optimal subset in
generational distance, for example), we can’t trivially select a conditional
subset of the prior and proceed. Even when we are simply using the dataset as
a reference distribution, if we have a continuous conditioning space, we can’t
select a discrete subset of the dataset which exactly respects the condition.
Instead, we can approximate by selecting a subset of the dataset whose
conditions are proximal to the condition used to create the generated
distribution. However, this necessitates calculating some distance in the
conditioning space, such as the vicinal loss defined in literature [68, 72].
Calculating these vicinity-based values may again be nontrivial in problems
with high-dimensional or multimodal conditioning information.

#### Comparing Conditionally Generated Distributions to a Marginal
Distribution

In some cases, though less intuitive, we may actually prefer to compare
conditional posteriors to marginal priors. This can be the next best option
when calculating a conditional prior is intractable. But in certain cases,
comparing a conditional posterior to a marginal prior may actually be
preferable. Say, for example, that we are training a model to generate car
designs conditioned on a target weight of one ton. Our dataset consists of
sedans weighing between half a ton and two tons and SUVs weighing between two
and four tons. Model A generates many sedans weighing one ton, while model B
generates a variety of sedans and SUVs weighing one ton. Comparing the
conditional posterior to the marginal prior would favor model B. However,
comparing to the conditional prior would favor model A since the conditional
prior would only contain sedans. For certain applications, model B could be
considered the more valuable model. After all, it has essentially found a way
to generate lightweight SUVs even though there were none in the dataset. In
essence, the comparison to the marginal prior captures the ability of a model
to generate designs that meet the condition while still resembling designs in
the dataset that don’t meet the condition. However, A could just as reasonably
be argued as the better model, since it remained truer to its training data.
The comparison to the marginal or conditional prior effectively balances a
tradeoff between two possible training objectives for the model.

###  8.5 Demonstration of Conditioning Metrics

(a) Original data over
condition space

(b) Conditional
Prior

(c) cVAE vs.
marginal prior

(d) cVAE vs.
conditional prior

(e) cGAN vs.
marginal prior

(f) cGAN vs.
conditional prior

Figure 10: Visual demonstration of a conditional Variational Autoencoder and
conditional GAN on a continuous conditioning problem. Generated distributions
are overlaid over both the original dataset (marginal prior), as well as the
nearest 10% of the data to the condition (conditional prior).The cVAE better
matches the marginal prior, while the cGAN better matches the conditional
prior.

.

To demonstrate the conditioning of generative models, we construct a fairly
challenging continuous conditioning problem on the previously used dataset.
The goal is to generate samples that have a condition value of exactly 0.3.
Though we would typically care about performance across a variety of condition
values (hence the need for a conditional model), we will only examine this
particular conditioning value in this case study. In practice, we would likely
perform a weighted average across the conditioning variable to get a sense of
the overall conditional generational performance of the models. At this value
of 0.3, the centers of the leftmost and rightmost modes closely meet the
condition, while only outlying samples from the other four modes meet the
condition. We train a conditional VAE and a conditional GAN, the results of
which are plotted in Figure 10 and tabulated in 6. Details about the model
architecture, training, and metric settings are included in the appendix. We
take the conditional prior to be the 10% of the dataset that most closely
matches the condition, as shown in LABEL:CO2. Overall, the cGAN focuses
heavily on the leftmost and rightmost modes, struggles to accurately capture
other areas of the distribution, and ignores sparse areas of the distribution.
In contrast, the cVAE under-represents the two main modes, but much more
faithfully captures the remainder of the distribution. Overall, the cVAE
significantly outperforms the GAN in conditioning reconstruction and
adherence, indicating that it generated samples that, on average, much more
closely match the condition. In both conditional and marginal F10 and F0.1,
the cVAE and cGAN repeat trends from their unconditional counterparts in
Section 4.5. Due its focus on the main modes, the GAN significantly
outperforms in maximum mean discrepancy when comparing to the subset of the
dataset that most closely match the condition. However, when comparing to the
marginal prior, the cVAE is the clear leader in MMD, generating more samples
that lie closer to the four underrepresented modes. As illustrated, binary
metrics can tell very different stories when compared against conditional or
marginal priors. Next, we move on to consider miscellaneous DGM evaluation
metrics.

Table 6: Conditioning scores for distributions in Figure 10. The cVAE better matches the marginal prior, while the cGAN better matches the conditional prior. Point metrics are averaged over the generated set. (Avg.) scores: | cVAE | cGAN
---|---|---
Conditioning Reconstruction | 0.007 | 0.026
Conditioning Adherence | 0.008 | 0.026
Conditional F10 | 0.909 | 0.826
Conditional F0.1 | 0.879 | 0.960
Conditional MMD | 0.035 | 0.021
Marginal F10 | 0.949 | 0.933
Marginal F0.1 | 0.447 | 0.554
Marginal MMD | 0.068 | 0.097

##  9 Other Metrics for Evaluating Deep Generative Models

Thus far, we have discussed metrics to evaluate similarity, diversity,
constraints, performance, and conditioning. Below, we summarize a few more
types of metrics that may be important to evaluate the performance of DGMs in
certain problems.

###  9.1 Latent Disentanglement

Many deep generative models synthesize designs by taking randomized inputs
from some (usually multidimensional) latent variable space. Many design
researchers have attempted to link latent variables with the physical
properties of the generated designs [73, 74], which, ideally, would serve as a
tool for human designers to manually select or tune generated designs. If the
latent space is disentangled, the generation process becomes more
interpretable. Tasks like creating new functionally graded material or design
space exploration could also be enhanced by disentangled representations, as
moving in the latent space along certain dimensions corresponds to expected
changes.

Research has shown, however, that latent variables will not be strongly linked
to individual attributes of generated samples unless actively conditioned
during training. Hence, there is a need to measure the disentanglement of the
latent variable space of models to better understand which spaces can be
better interpreted or used for design space exploration. Disentanglement
metrics generally fall into one of three categories: Intervention-based
metrics such as Z-diff [75], predictor-based metrics such as attribute
predictability score (SAP) [76], and information-based metrics such as mutual
information gap (MIG) [77]. We refer readers to reviews like [78] and [79] for
more detailed discussions on disentanglement metrics.

###  9.2 Human Evaluation

Though automated evaluation metrics are often the most practical, they seldom
provide as valuable of an analysis as people. Human evaluation approaches can
roughly be divided into crowdsourced evaluation frameworks and expert
evaluation frameworks, where the primary tradeoff is cost versus evaluation
quality.

#### Crowdsourced Evaluation

Crowdsourced evaluation frameworks are common in computer vision fields since
untrained humans typically suffice for determining the ‘realism’ of images in
computer vision problems. Metrics like Human Eye Perceptual Evaluation (HYPE)
[80], for example, can approximate the length of time needed for average
humans to discern between real and fake, or simply quantify the proportion of
generated samples that fool crowdsourced evaluators given infinite time. While
such metrics may be viable for certain design problems, they have many
caveats. First and foremost, designs may be difficult for untrained
participants to properly evaluate since the feasibility of a given design may
not be apparent without engineering expertise. Additionally, evaluating
objectives beyond general ‘realism’ such as physical performance or novelty
would be challenging with such a crowdsourced evaluation framework. Finally,
designs may be challenging to visually represent in a way that a human can
quickly and accurately gauge the realism of the design.

#### Expert Evaluation

Expert evaluation methods are often used in various applications of deep
generative models to assess the quality of the generated samples. Domain
experts or linguists evaluate the samples based on specific criteria such as
realism, coherence, and relevance. The scores are then averaged over a set of
generated samples to compare different models. This approach is considered to
be a gold standard for evaluating the quality of generated samples. However,
it can be time-consuming and costly. Alternatively, human experts can also be
used to provide feedback to the model during the generation process, allowing
the model to learn from its mistakes and improve its performance over time.

Another approach to evaluate deep generative models is the Consensual
Assessment Technique (CAT) [81], which is commonly used in psychology design
methodology literature. CAT ratings are a subjective measure of creativity
based on the opinions of experts. It involves having experts rate designs on a
scale on criteria such as originality, feasibility, and potential impact. CAT
ratings can provide valuable insights into the perceived level of creativity
of a design. However, one of the key challenges of using these metrics is
scalability, as deep generative models are capable of generating millions of
samples in a short time, while the cost of evaluating each sample is very
high. In principle, most metrics (e.g., ranking, scoring) used to rate human-
generated designs could also be used to rate DGM-generated designs. Similarly,
many metrics in design, such as SVS [82] could be adapted for DGM output. We
refer the reader to a review [83] on these metrics in design literature for an
in-depth discussion. With this, we conclude our discussion on miscellaneous
metrics for DGM evaluation and move on to a case study on the application of
evaluation metrics in a real engineering design problem.

##  10 Application Study: Exploratory Bicycle Frame Design with Constraints
and Performance Targets

Throughout the paper, we have applied metrics to simple 2-dimensional problems
to visually demonstrate how the presented metrics function. However, real
design problems are typically higher-dimensional, often have more constraints
and objectives, and are generally highly non-convex. As such, we feel that a
practical case study on a real engineering design problem may be insightful.
Therefore, in this section, we present a challenging engineering design
problem that simultaneously captures similarity, diversity, constraints,
performance, and target achievement. In this problem, we want to design new
high-performing bicycle frames that meet a set of ten structural performance
targets and adhere to an unknown set of implicit design constraints. We want
to identify a DGM that consistently generates designs meeting performance
targets and constraints, but generates a wide enough variety to offer a broad
selection of design candidates.

This proposed data-driven approach using DGMs is viable thanks to the FRAMED
bicycle frame dataset introduced in [12]. FRAMED features roughly 4000
constraint-satisfying designs and roughly 300 constraint-violating designs.
Each frame design is inspired by a real bicycle design [84] and is
parameterized over 37 parameters. Each constraint-satisfying design is
‘labeled’ with a vector of 10 structural performance values, such as weight,
safety factors, and deflections under various loading conditions. This dataset
was previously used as a benchmark for target-seeking deep generative models
in [63] and we adapt the models tested for our demonstration. We also adopt
the objective weights and DTAI parameters from [63]. In this application
study, we leverage various metrics to evaluate the performance of their model,
as well as others on the dataset.

###  10.1 Models

For our case study, we train and test three Generative Adversarial Network
(GAN) variants. The first is a ‘vanilla’ GAN, a type of network that
adversarially trains two networks, one which generates new designs and one
which discriminates between existing designs and generated designs. The
vanilla GAN is blind to design performance, only implicitly observes
constraints, and does not promote exploration beyond the convex region of the
dataset. The second is a Multi-Objective Performance-augmented Diverse GAN
(MO-PaDGAN) which uses a performance-weighted DPP kernel to augment the loss
function of the GAN in an effort to simultaneously encourage higher
performance and greater diversity among generated designs. MO-PaDGAN generally
encourages diverse, higher-performing designs, but does not consider specific
performance targets or explicit constraint handling. The third is a DTAI-GAN
which further augments the MO-PaDGAN with a weighted target achievement loss
and classifier guidance to avoid constraint-violating designs. In this sense,
DTAI-GAN is the only model that ‘explicitly’ considers constraints, while the
other models ‘implicitly’ consider constraints by training only on constraint-
satisfying designs.

###  10.2 Selecting Metrics

We’d like to evaluate diversity, performance, target achievement, and
constraint satisfaction, alongside distribution matching. We discuss how we
select metrics for this problem and justify these choices.

#### Statistical Similarity

While the goal is not just to mimic existing designs, measuring similarity is
important to make sure that we are still generating bicycle frames. We would
also ideally like our generated designs to span as much of the design space as
possible, thereby representing as many key types of designs as possible. As
such, we care about both ‘realism’ and coverage, as well as general
similarity. For this, we select nearest datapoint (NDP), nearest generated
sample (NGS), and F1 to capture realism, coverage, and similarity,
respectively. Since we primarily care about these metrics in the design space,
we evaluate all three in the design space, rather than the performance space.

#### Design Exploration

As one of the stated objectives of the problem, we want to generate a diverse
set of designs. This is common in scenarios where a final design will be
selected by experts from a diverse set. While we could look to maximize
nearest datapoint as a diversity metric, we still want our designs to resemble
designs in the dataset. We instead calculate nearest datapoint in the
performance space, instead of the design space. Put plainly, while we want
generated designs to resemble existing designs, we would like them to have
different performance values. Since we also want generated designs to be
diverse, we select inter-sample distance and DPP diversity as two diversity
metrics of choice, which we evaluate in the design space.

#### Design Constraints

The FRAMED problem provides a method to evaluate constraint adherence for
generated samples, making the constraint satisfaction metric a natural choice.
Since closed-form constraint definitions are not available, more informative
choices such as signed distance to constraint boundary are not feasible in
this problem.

#### Design Quality and Target Achievement

In FRAMED, we do not have a known set of Pareto-optimal designs. While we
could infer an approximate set from the dataset, we have no basis for assuming
that dataset designs are near-optimal, in part due to a randomization step in
the FRAMED dataset generation pipeline [12]. Therefore, we can feasibly expect
to generate designs that dominate the previously non-dominated designs in the
dataset, making generational distance a poor choice. We also do not have a
differentiable solver which we could query to calculate the optimality gap. As
such, we use hypervolume to capture the overall optimality of generated
designs. Since we are given performance targets with priority weights, we use
a weighted target achievement rate as a simple metric, but we can also use
signed distance to target since we have access to simple closed-form target
criteria. Finally, we use the Design Target Achievement Index (DTAI) to
evaluate overall target achievement performance.

###  10.3 Results

We demonstrate the performance of the three DGMs and their scores on selected
metrics in Figure 11.

Figure 11: Evaluation of three models on FRAMED dataset. The first box shows a
2-D Principal Component Analysis embedding of generated designs overlaid over
the dataset. The second box shows select generated bike frames. The third box
shows select invalid bike frames. The last box shows kernel density plots of
structural performance scores of generated bikes (design target shown as the
dotted line; smaller is better).

#### Statistical Similarity

Across the board, the vanilla GAN achieved superior distribution-matching
performance to the GAN variants focused on design performance. As indicated by
the demonstrated Principal Component Analysis embeddings, MO-PaDGAN identified
a higher-perming region of the design space and consistently generated many
designs away from the original design distribution. Similarly, DTAI-GAN
identified a region of the design space where designs were likely to meet
performance targets and sampled heavily from this region. Since there were a
handful of dataset samples near this region, DTAI-GAN outperformed MO-PaDGAN
in nearest datapoint. However, since it very consistently generated designs
far away from the center of the dataset distribution, it scored much worse in
nearest generated sample. Both models achieved very low F1 scores. Though the
distribution-matching scores indicate that the performance-aware models are
deviating significantly from the dataset, we don’t necessarily find this
concerning since we primarily care about finding novel high-performing
designs.

#### Design Exploration

The three diversity metrics selected paint a mixed picture. The performance-
aware models seemed to attain slightly superior performance space diversity,
with the DTAI-GAN achieving marginally superior scores to MO-PaDGAN. In design
space diversity, the DTAI-GAN dominated in inter-sample distance, a metric
heavily focused on local diversity. In contrast, however, it drastically
trailed the other two models in DPP diversity, which considers diversity more
globally. This behavior can be seen when a generated set occupies a relatively
small convex space but is very well distributed throughout this space. This
was the case for the results generated by the DTAI-GAN, compared to the other
two models.

#### Design Constraints

Since we don’t have access to closed-form constraint tests, we are not able to
evaluate distance to constraint boundary or other more insightful constraint-
related metrics. Instead, we simply know that, as expected, the constraint-
aware DTAI-GAN is a significantly stronger performer at constraint
satisfaction than the two models that only implicitly considered constraints.

#### Design Quality and Target Achievement

Across all performance and target achievement metrics, DTAI-GAN scores the
highest, and the vanilla GAN scores the lowest. Kernel density plots for three
of the ten performance objectives are shown in the last panel of 11, and in
each, DTAI-GAN’s distribution is most favorable. Interestingly, the DTAI-GAN’s
average target success rate was barely higher than MO-PaDGAN’s. However, its
average signed distance to the target was much higher, indicating that DTAI-
GAN’s designs that exceeded the target did so more drastically and designs
that missed the target did so by a smaller margin.

###  10.4 Analysis

The metrics strongly demonstrate that the DTAI-GAN achieves its stated goal of
target achievement and constraint satisfaction while maintaining a reasonable
degree of diversity and novelty. The enhanced functional performance and
constraint satisfaction of generated designs detracted from the DTAI-GAN’s
ability to match the training dataset. However, this behavior was largely
expected and encouraged in order to discover higher-performing regions of the
design space than were previously represented in the dataset. All in all, the
results on this dataset support the claim made in [63] that DTAI-GAN achieves
superior target achievement and constraint satisfaction to baselines while
maintaining reasonable diversity and novelty. With this, we conclude our case
study and proceed to some final discussion on implications for engineering
design.

##  11 Implications for Engineering Design

Since the introduction of deep generative models in engineering design
research, the overwhelming majority of proposed models have been trained
exclusively to optimize for statistical similarity [3]. Since most of these
models were adopted from natural image synthesis problems, it’s no surprise
that design researchers have similarly adopted statistical similarity, the
commonplace class of metrics in the image synthesis domain. DGMs have shown
immense promise in numerous design fields, such as topology optimization,
airfoil design, photonic/phononic materials, molecular design, metamaterials,
and many more [3]. However, as design researchers continue to adapt and
enhance these models for design tasks, a similar need for modifications and
new additions to evaluation metrics grows more pressing by the day. This paper
highlighted this need for other evaluation metrics and identified a few
dimensions that are important for evaluating DGMs. We also listed metrics for
each of these dimensions that are either already in use or present the
potential to be used for evaluating deep generative models.

###  11.1 The Pitfall of Statistical Similarity

Early in this paper, we attempted to illustrate why relying solely upon
statistical similarity as a design evaluation metric can be misguided. In
short, statistical similarity is simply insufficient for most design
evaluation, since it ignores design constraints, novelty, diversity, and
performance, which are normally primary concerns in a design setting. This
limitation has been raised as a primary critique of DGMs in engineering. For
example, Woldseth et al. [85] demonstrate how DGMs applied to topology
optimization fail in very simple cases when neglecting to account for
performance or constraints due to the massive performance difference between
some statistically similar designs. Furthermore, over-optimizing for
statistical similarity can even be counterproductive and restrict the ability
of a model to discover truly useful designs that exhibit superior properties
to designs in the dataset. However, we nonetheless acknowledge that
statistical similarity is still an important facet of DGM performance. As
such, we presented new perspectives and techniques for examining statistical
similarity which have been underutilized in design research, such as the
notion of decoupling realism and coverage. We hope that design researchers
will both look beyond statistical similarity and be more tactical in the way
they approach statistical similarity when evaluating deep generative models.

###  11.2 Selecting Appropriate Metrics

What should practitioners consider beyond similarity? This depends on how they
seek to expand upon existing designs in the dataset, which typically requires
a careful examination of the design problem at hand. Some common goals are a
greater level of diversity in design candidates, a set of designs that meet a
challenging set of constraints, or a collection of designs that attain
superior performance attributes to existing designs. For each of these goals,
we have provided a suite of evaluation metrics for practitioners to consider.
We’ve discussed the strengths and weaknesses of these metrics, analyzed their
requirements, and discussed which metrics leverage potential resources like
constraint satisfaction tests, auxiliary datasets, or optimal reference
designs. Finally, though this paper has focused on presenting metrics for the
effective evaluation of DGMs, some of the proposed metrics can also be
incorporated into the training of DGMs with little to no modification, as with
the design target achievement index in [63] and DPP diversity metric in [60].

###  11.3 Responsible use of Evaluation Metrics

Having access to a wealth of evaluation metrics provides researchers with a
great deal of flexibility in showcasing the performance of their models.
However, in such a young field with few established common practices, this
flexibility may also cause confusion. For the good of the community, we
encourage researchers to leverage as many relevant metrics as possible to
showcase both the strengths and weaknesses of their models and provide a more
comprehensive picture. Finally, we encourage researchers to be diligent in
publishing any hyperparameters or evaluation metric settings and share their
training datasets. To properly compare models and establish clear progression
in the field, evaluation metric settings and training datasets must be fully
transparent in order to be replicated.

##  12 Conclusion

As deep generative models continue to expand their reach in engineering design
communities, the increasing need for effective and standardized metrics grows
more pronounced. This paper explored evaluation metrics for deep generative
models in engineering design and presented a curated set of design-focused
metrics to address this growing demand. We presented a wide variety of
evaluation metrics focusing on realism, coverage, novelty, constraint
satisfaction, performance, target achievement, and conditioning. We discussed
which metrics to select when evaluating different facets of model performance
and demonstrated the application of these metrics on easy-to-visualize
problems. Finally, we presented a practical application of the methods
discussed on a challenging bicycle frame design problem. We detailed how and
why we selected appropriate metrics and described the performance of three
deep generative models applied to the problem.

In writing this paper, our overarching goal has been to call attention to the
fallacies of statistical similarity metrics in engineering design problems,
inspire practitioners to consider the many alternatives that we discuss, and
provide them with the requisite knowledge to effectively apply these metrics
to their design problems. We publicly release our datasets, models, and
implementation code for the metrics used in our case studies at
decode.mit.edu/projects/metrics/ in order to facilitate easy adoption in
different domains. We sincerely hope that researchers will use these resources
to better understand their models and greatly improve their capabilities in
engineering design tasks.

## References

  * [1] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, in: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, MIT Press, Cambridge, MA, USA, 2014, p. 2672–2680.
  * [2] D. P. Kingma, M. Welling, Auto-encoding variational bayes, arXiv preprint arXiv:1312.6114 (2013).
  * [3] L. Regenwetter, A. H. Nobari, F. Ahmed, Deep generative models in engineering design: A review, Journal of Mechanical Design 144 (7) (2022) 071704.
  * [4] F.-A. Croitoru, V. Hondru, R. T. Ionescu, M. Shah, Diffusion models in vision: A survey, arXiv preprint arXiv:2209.04747 (2022).
  * [5] L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, Y. Shao, W. Zhang, B. Cui, M.-H. Yang, Diffusion models: A comprehensive survey of methods and applications, arXiv preprint arXiv:2209.00796 (2022).
  * [6] F. Mazé, F. Ahmed, Topodiff: A performance and constraint-guided diffusion model for topology optimization, arXiv preprint arXiv:2208.09591 (2022).
  * [7] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).
  * [8] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, et al., Exploring the limits of transfer learning with a unified text-to-text transformer., J. Mach. Learn. Res. 21 (140) (2020) 1–67.
  * [9] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901.
  * [10] Q. Zhu, J. Luo, Generative transformers for design concept generation, Journal of Computing and Information Science in Engineering 23 (4) (2023) 041003.
  * [11] L. Siddharth, L. Blessing, J. Luo, Natural language processing in-and-for design research, Design Science 8 (2022) e21.
  * [12] L. Regenwetter, C. Weaver, F. Ahmed, Framed: An automl approach for structural performance prediction of bicycle frames, Computer-Aided Design (2022) 103446\.
  * [13] T. Karras, S. Laine, T. Aila, A style-based generator architecture for generative adversarial networks, in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4396–4405.  doi:10.1109/CVPR.2019.00453.
  * [14] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, T. Aila, Analyzing and improving the image quality of stylegan, in: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 8107–8116.  doi:10.1109/CVPR42600.2020.00813.
  * [15] A. Brock, J. Donahue, K. Simonyan, Large scale gan training for high fidelity natural image synthesis, arXiv preprint arXiv:1809.11096 (2018).
  * [16] A. Borji, Pros and cons of gan evaluation measures, Computer Vision and Image Understanding 179 (2019) 41–65.
  * [17] A. Borji, Pros and cons of gan evaluation measures: New developments, Computer Vision and Image Understanding 215 (2022) 103329.
  * [18] A. Gatt, E. Krahmer, Survey of the state of the art in natural language generation: Core tasks, applications and evaluation, Journal of Artificial Intelligence Research 61 (2018) 65–170.
  * [19] C. Dong, Y. Li, H. Gong, M. Chen, J. Li, Y. Shen, M. Yang, A survey of natural language generation, ACM Computing Surveys 55 (8) (2022) 1–38.
  * [20] N. Riquelme, C. Von Lücken, B. Baran, Performance metrics in multi-objective optimization, in: 2015 Latin American Computing Conference (CLEI), 2015, pp. 1–11.  doi:10.1109/CLEI.2015.7360024.
  * [21] C. T. Mueller, J. A. Ochsendorf, Combining structural performance and designer preferences in evolutionary design space exploration, Automation in Construction 52 (2015) 70–82.
  * [22] C. Bilodeau, W. Jin, T. Jaakkola, R. Barzilay, K. F. Jensen, Generative models for molecular discovery: Recent advances and challenges, Wiley Interdisciplinary Reviews: Computational Molecular Science (2022) e1608.
  * [23] D. Shah, A. Suresh, A. Admasu, D. Upadhyay, K. Deb, A survey on evaluation metrics for synthetic material micro-structure images from generative models, arXiv preprint arXiv:2211.09727 (2022).
  * [24] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, Gans trained by a two time-scale update rule converge to a local nash equilibrium, Advances in neural information processing systems 30 (2017).
  * [25] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen, Improved techniques for training gans, in: Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, Curran Associates Inc., Red Hook, NY, USA, 2016, p. 2234–2242.
  * [26] M. Bińkowski, D. J. Sutherland, M. Arbel, A. Gretton, Demystifying mmd gans, arXiv preprint arXiv:1801.01401 (2018).
  * [27] C.-Y. Lin, Rouge: A package for automatic evaluation of summaries, in: Text summarization branches out, 2004, pp. 74–81.
  * [28] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318.
  * [29] S. Banerjee, A. Lavie, Meteor: An automatic metric for mt evaluation with improved correlation with human judgments, in: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005, pp. 65–72.
  * [30] M. Arjovsky, S. Chintala, L. Bottou, Wasserstein generative adversarial networks, in: Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, JMLR.org, 2017, p. 214–223.
  * [31] M. Arjovsky, L. Bottou, Towards principled methods for training generative adversarial networks, arXiv preprint arXiv:1701.04862 (2017).
  * [32] J. Ho, A. Jain, P. Abbeel, Denoising diffusion probabilistic models (2020).  arXiv:2006.11239.
  * [33] P. Dhariwal, A. Nichol, Diffusion models beat gans on image synthesis (2021).  arXiv:2105.05233.
  * [34] Z. Goldfeld, K. Greenewald, Sliced mutual information: A scalable measure of statistical dependence, Advances in Neural Information Processing Systems 34 (2021) 17567–17578.
  * [35] M. I. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Y. Bengio, A. Courville, R. D. Hjelm, Mine: mutual information neural estimation, arXiv preprint arXiv:1801.04062 (2018).
  * [36] B. Rhodes, K. Xu, M. U. Gutmann, Telescoping density-ratio estimation, Advances in neural information processing systems 33 (2020) 4905–4916.
  * [37] M. Sugiyama, T. Suzuki, T. Kanamori, Density-ratio matching under the bregman divergence: a unified framework of density-ratio estimation, Annals of the Institute of Statistical Mathematics 64 (5) (2012) 1009–1044.
  * [38] M. Gutmann, A. Hyvärinen, Noise-contrastive estimation: A new estimation principle for unnormalized statistical models, in: Proceedings of the thirteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, 2010, pp. 297–304.
  * [39] J. Alsing, T. Charnock, S. Feeney, B. Wandelt, Fast likelihood-free cosmology with neural density estimators and active learning, Monthly Notices of the Royal Astronomical Society 488 (3) (2019) 4440–4458.
  * [40] D. P. Kingma, P. Dhariwal, Glow: Generative flow with invertible 1x1 convolutions, Advances in neural information processing systems 31 (2018).
  * [41] I. Kobyzev, S. J. Prince, M. A. Brubaker, Normalizing flows: An introduction and review of current methods, IEEE transactions on pattern analysis and machine intelligence 43 (11) (2020) 3964–3979.
  * [42] I. Deshpande, Y.-T. Hu, R. Sun, A. Pyrros, N. Siddiqui, S. Koyejo, Z. Zhao, D. Forsyth, A. G. Schwing, Max-sliced wasserstein distance and its use for gans, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 10648–10656.
  * [43] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255.
  * [44] S. Barratt, R. Sharma, A note on the inception score, arXiv preprint arXiv:1801.01973 (2018).
  * [45] M. Rosca, B. Lakshminarayanan, D. Warde-Farley, S. Mohamed, Variational approaches for auto-encoding generative adversarial networks, arXiv preprint arXiv:1706.04987 (2017).
  * [46] M. Lucic, K. Kurach, M. Michalski, S. Gelly, O. Bousquet, Are gans created equal? a large-scale study, Advances in neural information processing systems 31 (2018).
  * [47] M. S. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, S. Gelly, Assessing generative models via precision and recall, Advances in neural information processing systems 31 (2018).
  * [48] L. Simon, R. Webster, J. Rabin, Revisiting precision recall definition for generative modeling, in: International Conference on Machine Learning, PMLR, 2019, pp. 5799–5808.
  * [49] S. Ravuri, O. Vinyals, Classification accuracy score for conditional generative models, Advances in neural information processing systems 32 (2019).
  * [50] L. Xu, M. Skoularidou, A. Cuesta-Infante, K. Veeramachaneni, Modeling tabular data using conditional gan, Advances in Neural Information Processing Systems 32 (2019).
  * [51] W. Chen, F. Ahmed, Padgan: Learning to generate high-quality novel designs, Journal of Mechanical Design 143 (3) (2021) 031703.
  * [52] M. A. Boden, Computer models of creativity, AI Magazine 30 (3) (2009) 23–23.
  * [53] J. Lehman, K. O. Stanley, Abandoning objectives: Evolution through the search for novelty alone, Evolutionary computation 19 (2) (2011) 189–223.
  * [54] N. C. Brown, C. T. Mueller, Quantifying diversity in parametric design: a comparison of possible metrics, AI EDAM 33 (1) (2019) 40–53.
  * [55] F. Ahmed, S. K. Ramachandran, M. Fuge, S. Hunter, S. Miller, Measuring and optimizing design variety using herfindahl index, in: International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 59278, American Society of Mechanical Engineers, 2019, p. V007T06A007.
  * [56] L. F. Kozachenko, N. N. Leonenko, Sample estimate of the entropy of a random vector, Problemy Peredachi Informatsii 23 (2) (1987) 9–16.
  * [57] S. Pavoine, S. Ollier, D. Pontier, Measuring diversity from dissimilarities with rao’s quadratic entropy: are any dissimilarities suitable?, Theoretical population biology 67 (4) (2005) 231–239.
  * [58] J. Podani, Convex hulls, habitat filtering, and functional diversity: mathematical elegance versus ecological interpretability, Community Ecology 10 (2) (2009) 244–250.
  * [59] A. Kulesza, B. Taskar, et al., Determinantal point processes for machine learning, Foundations and Trends® in Machine Learning 5 (2–3) (2012) 123–286.
  * [60] W. Chen, F. Ahmed, Mo-padgan: Reparameterizing engineering designs for augmented multi-objective optimization, Applied Soft Computing 113 (2021) 107909\.
  * [61] D. A. Van Veldhuizen, Multiobjective evolutionary algorithms: classifications, analyses, and new innovations, Air Force Institute of Technology, 1999.
  * [62] Q. Chen, J. Wang, P. Pope, M. Fuge, et al., Inverse design of two-dimensional airfoils using conditional generative models and surrogate log-likelihoods, Journal of Mechanical Design 144 (2) (2022).
  * [63] L. Regenwetter, F. Ahmed, Design target achievement index: A differentiable metric to enhance deep generative models in multi-objective inverse design, arXiv preprint arXiv:2205.03005 (2022).
  * [64] J. Knowles, Parego: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems, IEEE Transactions on Evolutionary Computation 10 (1) (2006) 50–66.
  * [65] M. Mirza, S. Osindero, Conditional generative adversarial nets, arXiv preprint arXiv:1411.1784 (2014).
  * [66] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, P. Abbeel, Infogan: Interpretable representation learning by information maximizing generative adversarial nets, in: Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016, pp. 2180–2188.
  * [67] K. Sohn, X. Yan, H. Lee, Learning structured output representation using deep conditional generative models, in: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS’15, MIT Press, Cambridge, MA, USA, 2015, p. 3483–3491.
  * [68] A. Heyrani Nobari, W. Chen, F. Ahmed, Pcdgan: A continuous conditional diverse generative adversarial network for inverse design, Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (Aug 2021).  doi:10.1145/3447548.3467414.
URL http://dx.doi.org/10.1145/3447548.3467414

  * [69] A. Heyrani Nobari, W. W. Chen, F. Ahmed, RANGE-GAN: Design Synthesis Under Constraints Using Conditional Generative Adversarial Networks, Journal of Mechanical Design (2021) 1–16arXiv:https://asmedigitalcollection.asme.org/mechanicaldesign/article-pdf/doi/10.1115/1.4052442/6756741/md-21-1243.pdf, doi:10.1115/1.4052442.
URL https://doi.org/10.1115/1.4052442

  * [70] Z. Nie, T. Lin, H. Jiang, L. B. Kara, Topologygan: Topology optimization using generative adversarial networks based on physical fields over the initial domain, Journal of Mechanical Design 143 (3) (2021) 031715.
  * [71] Y. Benny, T. Galanti, S. Benaim, L. Wolf, Evaluation metrics for conditional image generation, International Journal of Computer Vision 129 (5) (2021) 1712–1731.
  * [72] X. Ding, Y. Wang, Z. Xu, W. J. Welch, Z. J. Wang, Ccgan: Continuous conditional generative adversarial networks for image generation, in: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, OpenReview.net, 2021.
URL https://openreview.net/forum?id=PrzjugOsDeE

  * [73] L. Wang, Y.-C. Chan, F. Ahmed, Z. Liu, P. Zhu, W. Chen, Deep generative modeling for mechanistic-based learning and design of metamaterial systems, Computer Methods in Applied Mechanics and Engineering 372 (2020) 113377.
  * [74] W. Chen, M. Fuge, Synthesizing designs with interpart dependencies using hierarchical generative adversarial networks, Journal of Mechanical Design 141 (11) (2019) 111403.
  * [75] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, A. Lerchner, beta-vae: Learning basic visual concepts with a constrained variational framework, in: International conference on learning representations, 2017.
  * [76] A. Kumar, P. Sattigeri, A. Balakrishnan, Variational inference of disentangled latent concepts from unlabeled observations, arXiv preprint arXiv:1711.00848 (2017).
  * [77] R. T. Chen, X. Li, R. B. Grosse, D. K. Duvenaud, Isolating sources of disentanglement in variational autoencoders, Advances in neural information processing systems 31 (2018).
  * [78] J. Zaidi, J. Boilard, G. Gagnon, M.-A. Carbonneau, Measuring disentanglement: A review of metrics, arXiv preprint arXiv:2012.09276 (2020).
  * [79] K. Do, T. Tran, Theory and evaluation metrics for learning disentangled representations, arXiv preprint arXiv:1908.09961 (2019).
  * [80] S. Zhou, M. Gordon, R. Krishna, A. Narcomey, L. F. Fei-Fei, M. Bernstein, Hype: A benchmark for human eye perceptual evaluation of generative models, Advances in neural information processing systems 32 (2019).
  * [81] T. M. Amabile, Social psychology of creativity: A consensual assessment technique., Journal of personality and social psychology 43 (5) (1982) 997.
  * [82] J. J. Shah, S. M. Smith, N. Vargas-Hernandez, Metrics for measuring ideation effectiveness, Design studies 24 (2) (2003) 111–134.
  * [83] S. R. Miller, S. T. Hunter, E. Starkey, S. Ramachandran, F. Ahmed, M. Fuge, How should we measure creativity in engineering design? a comparison between social science and engineering approaches, Journal of Mechanical Design 143 (3) (2021).
  * [84] L. Regenwetter, B. Curry, F. Ahmed, Biked: A dataset for computational bicycle design with machine learning benchmarks, Journal of Mechanical Design 144 (3) (2022).
  * [85] R. V. Woldseth, N. Aage, J. A. Bærentzen, O. Sigmund, On the use of artificial neural networks in topology optimisation, Structural and Multidisciplinary Optimization 65 (10) (2022) 294.
  * [86] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE transactions on image processing 13 (4) (2004) 600–612.
  * [87] T. Nguyen, Q.-H. Pham, T. Le, T. Pham, N. Ho, B.-S. Hua, Point-set distances for learning representations of 3d point clouds, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10478–10487.
  * [88] M. Tantardini, F. Ieva, L. Tajoli, C. Piccardi, Comparing methods for comparing networks, Scientific reports 9 (1) (2019) 1–19.
  * [89] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning transferable visual models from natural language supervision (2021).  arXiv:2103.00020.
  * [90] V. Khrulkov, L. Mirvakhabova, E. Ustinova, I. Oseledets, V. Lempitsky, Hyperbolic image embeddings, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6418–6428.
  * [91] F. Faghri, D. J. Fleet, J. R. Kiros, S. Fidler, Vse++: Improving visual-semantic embeddings with hard negatives, arXiv preprint arXiv:1707.05612 (2017).
  * [92] G. Dai, J. Xie, Y. Fang, Siamese cnn-bilstm architecture for 3d shape representation learning., in: IJCAI, 2018, pp. 670–676.
  * [93] D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, et al., Universal sentence encoder, arXiv preprint arXiv:1803.11175 (2018).
  * [94] H. Cai, V. W. Zheng, K. C.-C. Chang, A comprehensive survey of graph embedding: Problems, techniques, and applications, IEEE Transactions on Knowledge and Data Engineering 30 (9) (2018) 1616–1637.
  * [95] S. Morozov, A. Voynov, A. Babenko, On self-supervised image representations for gan evaluation, in: International Conference on Learning Representations, 2020\.
  * [96] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).

##  13 Appendix

###  13.1 Calculating Distances between Designs

Many evaluation metrics necessitate the ability to calculate a distance
between designs. This task is often nontrivial. Below, we provide some methods
to calculate distances in various data modalities, as well as some strategies
to leverage embeddings to calculate distances.

  1. 1.

Continuous variables: Common distance metrics include Euclidean distance and
Manhattan distance. These distances are sensitive to the relative scaling
between parameters. As such, care must be taken to appropriately scale data.
Cosine similarity is another common approach to measuring ‘distance’ between
variables.

  2. 2.

Ordinal Discrete Variables: For ordinal discrete variables (such as the number
of teeth on a gear), $L_{1}$, $L_{2}$, and $L_{\infty}$ are all valid metrics.

  3. 3.

Categorical Variables: For discrete variables with no sequential significance
(i.e., categorical data), one-hot encoding can transform the data into boolean
data, to which the aforementioned distance methods apply. When directly
working in categorical spaces, Hamming distance is a widely used metric.

  4. 4.

Pixels: Calculating distances directly on images is challenging, largely due
to their high dimensionality. Depending on the nature of the images, directly
calculating pixel distance using a simple method like L2 may be a valid
approach. For example, this may work on topologies represented as images,
while on photographs of vehicles, pixel distance is unlikely to reflect
similarity in the content of images. As a strong alternative, distances
between images can be calculated using Structural Similarity (SSIM) [86].
Recently, calculating distances between images is most commonly done using
learned embeddings.

  5. 5.

3D geometry: For many common 3D geometry representation methods like point
clouds, meshes, and rasterized curves, point-set distances like Chamfer and
Hausdorff distance are commonly used. We refer the reader to papers like [87]
for further details on metrics used in computer graphics.

  6. 6.

Text: Directly calculating distances between pieces of textual data is
challenging. While simple categorical distances can be calculated on tokenized
text data using cosine distance, learned text embeddings are usually the
preferred approach to calculating distances on text data.

  7. 7.

Graphs: Measuring distance between graphs (networks) is a common problem known
as the ‘network comparison’ problem. Typically methods depend on the type of
graph (directed vs. undirected, weighted vs. unweighted, known vs. unknown
nodes, etc.). We refer readers interested in calculating distances between
graphs to [88].

  8. 8.

Multimodal Data Calculating distances between designs represented using
multimodal data is nontrivial and inherits the challenges of the individual
modalities. Though this problem requires further research, there are a few
strategies that researchers can select from. A simple approach takes the
weighted sum of distances across different modalities but requires tuning of
weights. Another approach constructs an embedding from the multimodal data
over which to measure distances [89].

While calculating distances in the original data modality is often viable, a
common alternative approach is to find features of the data and then calculate
distances between those features. These features are often learned using a
machine learning model instead of being hand-picked. These approaches are very
common when working with images [90, 91], 3D geometry [92], text [7, 93], and
graphs [94]. However, learned embeddings suffer from a potential risk of poor
generalizability. For example, a breadth of literature has documented how
image embeddings struggle to generalize even across similar natural image
datasets [95, 45, 44, 80]. In general, great care must be taken when using
learned embeddings to calculate distances, regardless of the modality of the
original data.

When working with multimodal data, calculating distances can be challenging.
Shared embeddings are a viable approach to calculating distances between
datapoints of different modalities. For example, Contrastive Language-Image
Pretraining (CLIP) [89] embeddings map images and text into a common embedding
space in which distances can be calculated, as showcased in Figure 12.

Figure 12: Cosine similarity between bicycle images from [84] and text prompts
in CLIP embedding space. Shared embeddings allow for distance calculation
between datapoints of different modalities.

###  13.2 Model and Hyperparameter Settings for Evaluation Experiments

We present the training settings for all models tested on the synthetic
datasets and all evaluation metrics used.

#### GAN and cGAN Models

  1. 1.

Discriminator and generator models are 2 hidden-layer neural networks with 100
parameters with leaky ReLU activation with slope coefficient $\alpha=0.2$.

  2. 2.

The latent vectors are four-dimensional

  3. 3.

Discriminator and generator optimizers are Adam [96] with learning rate
$1e-3$.

  4. 4.

The models are trained over 5000 randomized batches of 32 samples.

#### VAE Model

  1. 1.

Encoder and decoder models are 3 hidden-layer neural networks with 100
parameters with ReLU activation.

  2. 2.

The latent dimension is 4.

  3. 3.

The model is trained for 100 epochs of batches of 100.

  4. 4.

the model’s optimizer is Adam with learning rate $1e-3$.

#### MO-PaDGAN Model

  1. 1.

Discriminator an generator architectures and optimizers are the same as in the
GAN model. latent dimension and training epochs are also unchanged.

  2. 2.

All training parameters are identical to [60] except $\gamma_{0}=5$ and
$\gamma_{1}=2$.

#### Experiment 1: Evaluating statistical similarity on 2D data (Sec. 4.5

  1. 1.

Nearest datapoint, nearest generated sample, and rediscovery use Euclidean
distance.

  2. 2.

F1, F10, F0.1, and AUC-PR use 20 clusters, angle resolution of 1000, and 10
clustering runs.

  3. 3.

ML efficacy uses an auxiliary regression task of predicting the objectives
used in Sec. 7.3. A K-nearest-neighbors regressor with $K=5$ is used and
evaluated using coefficient of determination ($R^{2}$).

#### Experiment 2: Evaluating design exploration on 2D data (Sec. 5.3

  1. 1.

Nearest datapoint and inter-sample distance uses Euclidean distance.

  2. 2.

DPP Diversity uses a subset size of 10 and an exponentiation parameter of 0.1.

#### Experiment 3: Evaluating design constraints on 2D data (Sec. 6.4

  1. 1.

Predicted constraint satisfaction uses K-nearest-neighbors with $K=5$.

  2. 2.

Nearest Invalid Datapoint uses Euclidean Distance.

#### Experiment 4: Evaluating design quality on 2D data (Sec. 7.3

  1. 1.

The hypervolume reference point is selected as the 99th quantile performance
values from the dataset.

  2. 2.

The Pareto-optimal set for KNO1 is given as $x+y=.4705$.

  3. 3.

The target for all target-related scores is set as (0.5,0.5). Objectives are
weighted equally.

  4. 4.

DTAI parameters are $\alpha=\beta=[1,1]$.

#### Experiment 5: Evaluating conditioning adherence on 2D data (Sec. 8.5

  1. 1.

cVAE and cGAN architectures are identical to unconditional counterparts,
except for

  2. 2.

Both marginal and conditional F0.1 and F10 use 20 clusters, and angle
resolution of 1000, and 10 clustering runs.

  3. 3.

Conditional scores used the nearest 10% of the dataset as the constraint-
satisfying subset.

  4. 4.

Condition adherence and reconstruction use mean squared error.

  5. 5.

Condition reconstruction uses K-nearest-neighbors regression with $K=5$.

###  13.3 Model and Hyperparameter Settings for Bicycle Frame Design Problem

#### Evaluation Metric Settings

  1. 1.

Nearest datapoint, nearest generated sample, and inter-sample distance use
Euclidean distance.

  2. 2.

F1 score uses 20 clusters, angle resolution of 1000, and 10 clustering runs.

  3. 3.

DPP Diversity uses a subset size of 10 and an exponentiation parameter of 0.1.

  4. 4.

The target for all target-related scores is set as 0.75 quantile performance
value in each objective across the FRAMED dataset

  5. 5.

DTAI parameters are