# Drag-guided diffusion models for vehicle image generation

Nikos Aréchiga, Frank Permenter, Chenyang Yuan111Authors listed alphabetically
Toyota Research Institute
<EMAIL_ADDRESS>

Binyang Song
Massachusetts Institute of Technology
<EMAIL_ADDRESS>
###### Abstract

Denoising diffusion models trained at web-scale have revolutionized image
generation. The application of these tools to engineering design is an
intriguing possibility, but is currently limited by their inability to parse
and enforce concrete engineering constraints. In this paper, we take a step
towards this goal by proposing physics-based guidance, which enables
optimization of a performance metric (as predicted by a surrogate model)
during the generation process. As a proof-of-concept, we add drag guidance to
Stable Diffusion, which allows this tool to generate images of novel vehicles
while simultaneously minimizing their predicted drag coefficients.

##  1 Introduction

Diffusion models have produced state of the art results for high-resolution
image generation. One powerful aspect of diffusion models is the incorporation
of _text-based guidance_ , which steers the image generation process towards
an output that matches a user-provided description. Such guidance is
implemented in DALL-E 2 [25], Imagen [30] and Stable Diffusion [27]. Given
their capabilities, it is natural to ask how current image generation tools
can be applied to engineering design, and, in particular, if text-guidance is
sufficient to specify quantitative engineering constraints. Simple experiments
reveal significant challenges. For example, given a text prompt of “a house
with _two_ windows”, a text-guided diffusion model produces a house with
windows, but ignores the quantitative constraint on the specified _number_ of
windows (Figure 1(a)). The prompt “a car with a drag coefficient of 0.5” is
also completely misinterpreted (Figure 1(b)), showing an inability of the
generation process to even correctly parse the constraint.

To successfully incorporate engineering constraints into existing image
generation tools, it is clear that new guidance approaches must be developed.
In this paper, we implement _physics-based_ guidance that simultaneously
estimates and optimizes the performance of the generated object. As a proof-
of-concept, we implement _drag_ optimization of _vehicles_ , but our approach
can be adapted to any design task that admits a _surrogate model_ of
performance with the following characteristics:

  1. 1.

The input is an image, as produced by a diffusion process for image
generation.

  2. 2.

The model is robust to distributional shifts in the input.

  3. 3.

The model is differentiable, allowing for gradient-based optimization of the
output.

For the task of vehicle drag optimization, we train a surrogate model that
meets these specifications using the dataset [31], which consists of two-
dimensional (2D) vehicle renderings labeled with drag coefficients calculated
from fluid-dynamics simulations. Aiming for robustness to distributional
shifts, we build our model using feature-space embeddings from widely used
object recognition networks [23, 12, 9]. Finally, our model is implemented
using a deep-neural network and is hence differentiable.

(a) “a house with two windows”

(b) “a car with a drag coefficient of 0.5”

Figure 1: Illustrates the inability of text-guidance to realize quantitative
design specifications. Images were generated by craiyon.com

Leveraging our surrogate model, we integrate _drag guidance_ 222In this paper,
we use “drag” and “drag coefficient” interchangeably. into Stable Diffusion
(Figure 2),

Figure 2: Our pipeline for incorporating drag minimization into diffusion
models.

and show that this integrated system can generate car images that both match a
text prompt and minimize predicted drag. We also find that drag guidance can
produce more aesthetically streamlined outputs. Figure 3 contains illustrative
examples of our results compared to baseline images generated by an unmodified
version of Stable Diffusion.

Baseline | Drag optimized
---|---
|
|

Baseline | Drag optimized
---|---
|
|

Figure 3: Outputs of Stable Diffusion with and without our drag-guidance
technique.

##  2 Related Work

#### Guided diffusion

Guidance aims to minimize an auxiliary loss function $\phi(x)$ during the
image generation process [3]. This is done by incorporating the gradient of
$\phi(x)$ into each generation step. Guidance techniques can be differentiated
by the type of loss function. In _classifier-based_ guidance [8], the loss
function $\phi(x)$ is the logit probability of a specified class in a pre-
trained classifier. The aim of this guidance is to steer the output towards
this class. In _text-based_ guidance, $\phi(x)$ is a similarity score computed
in a latent space, e.g., the cosine similarity between the CLIP embedding of
the image $x$ and a given text-prompt [23, 7, 20, 15]. Guidance can also be
used to solve linear equations $Ax=b$ by setting $\phi(x)=\|b-Ax\|^{2}$. This
setup generalizes image inpainting, compressed sensing, and super-resolution
[6, 5]. Alternatively, one can implement _classifier-free_ guidance, in which
gradients of $\phi(x)$ are effectively learned during training. Such
approaches have been used for text guidance [13], inpainting [27] and single-
view reconstruction [17].

#### Diffusion for design

Prior applications of diffusion to engineering design include [11, 18], which
train _new_ diffusion models for a specific design task. In contrast, our work
focuses on adapting an _existing_ diffusion model (Stable Diffusion), aiming
to leverage the web-scale dataset used for its training.

#### Drag optimization and machine learning

Drag optimization is a classical problem that can be solved using PDE-
optimization techniques. These approaches require accurate solutions of the
Navier-Stokes equations, which are computationally expensive to obtain and
sensitive to shape representation. Circumventing these difficulties using ML-
techniques is an active area of research. One thread of research replaces
direct solution of the Navier-Stokes equations with a _surrogate model_. These
models are trained using computational fluid-dynamics (CFD) simulations and
either predict pressure or velocity fields [34, 26, 2, 10, 4, 1], and/or a
performance measure (e.g., drag coefficient) directly from model parameters
[31, 4]. Different model architectures have been used, including Gaussian
process regression [34], graph neural networks [2, 10], geodesic convolutional
neural networks [4, 1], variational autoencoders (VAEs) [29], and UNets [33,
14]. Our work continues this thread and develops a surrogate model for drag
coefficient estimation using _linear regression_ on features extracted from
pretrained neural-networks. Another thread concerns novel shape
representations that maintain end-to-end differentiability when cascaded with
a surrogate model. Such representations include Deep Implicit Fields [26],
PolyCubes [34], and 2D renderings [28, 33, 14, 31]. Our work uses 2D
renderings proposed by [31], which capture the point-wise depth and normal
information of three dimensional (3D) shapes using RGB color channels of pixel
data. This image representation allows us to easily implement a differentiable
surrogate-model that can predict drag coefficients directly from intermediate
Stable Diffusion outputs.

##  3 Denoising Diffusion Models

Diffusion models are a class of generative models inspired by the physical
process of diffusion. These models are trained by adding progressively larger
amounts of Gaussian noise to a given data set, mimicking a diffusion process.
Sampling reverses this process through _denoising_.

###  3.1 Training

Diffusion models estimate a _noise vector_ $\epsilon\in\mathbb{R}^{n}$ from a
given $y\in\mathbb{R}^{n}$ and _noise level_ $\sigma>0$ such that
$y=x+\sigma\epsilon$ approximately holds for some $x$ in the training set. The
learned estimator, denoted
$\epsilon_{\theta}:\mathbb{R}^{n}\times\mathbb{R}_{+}\rightarrow\mathbb{R}^{n}$,
is called a _denoiser_. The trainable parameters, denoted jointly by
$\theta\in\mathbb{R}^{m}$, are found by (approximately) minimizing

$\displaystyle
L(\theta):=\mathop{{}\mathbb{E}}_{x,\sigma,\epsilon}\left\lVert\epsilon_{\theta}(x+\sigma\epsilon,\sigma)-\epsilon\right\lVert^{2}$
(1)

when $x$ is drawn from the training-set distribution, $\sigma$ is drawn
uniformly from a finite set of positive numbers, and $\epsilon$ is drawn from
a Gaussian distribution $\mathcal{N}(0,I)$.

Current state-of-the-art diffusion models modify this training process in a
few ways. Models supporting classifier-free text guidance add text embeddings
(from CLIP, for example) to the denoiser input [23]. Models supporting high-
resolution image synthesis implement _latent diffusion_ [27], i.e., the
variable $x$ is not an element of pixel space, but, rather, an element of a
lower-dimensional latent space defined by a VAE.

###  3.2 Sampling

Let $\\{\sigma_{t}\\}^{T}_{t=0}$ denote a set of noise levels $\sigma$ used in
training and assume that $\sigma_{t}>\sigma_{t-1}$. The sequence $\sigma_{t}$
is used by _sampling algorithms_ to construct novel images using a trained
denoiser $\epsilon_{\theta}$. We use the DDIM sampler [32] which, given
$x_{T}\sim\mathcal{N}(0,\sigma_{T}^{2}I)$, generates an image $x_{0}$ via the
following recursion333We note [32] presents this recursion in coordinates
$z_{t}:=\sqrt{\alpha_{t}}x_{t}$ where $\alpha_{t}$ satisfies
$\sigma^{2}_{t}=(1-\alpha_{t})/\alpha_{t}$.

$\displaystyle
x_{t-1}=x_{t}-(\sigma_{t}-\sigma_{t-1})\epsilon_{\theta}(x_{t},t).$ (2)

Classifier-free text-guidance [13] executes this recursion using a modified
denoiser $\epsilon_{\theta}(x_{t},t,y)$ that takes a reference latent
embedding $y$ (e.g., from CLIP) as an optional third input. Concretely, it
constructs $\epsilon_{\theta}(x_{t},t)$ via

$\displaystyle\epsilon_{\theta}(x_{t},t)=(1-w)\epsilon_{\theta}(x_{t},t,\varnothing)+w(\epsilon_{\theta}(x_{t},t,y))$
(3)

where $w$ is a weighing factor and $\varnothing$ denotes the empty string.

##  4 Method and Implementation

In this section, we first describe the construction of a surrogate model for
drag coefficient estimation. We then describe our technique for drag guidance,
which incorporates gradients of this model into the sampling process.

###  4.1 A surrogate model for drag estimation

To estimate drag, we train a surrogate model that predicts drag coefficients
from side-view vehicle images (Figure 4). Training uses the dataset [31],
which consists of 2D vehicle renderings labeled with drag coefficients
calculated from CFD simulations. Our model consists of a trainable linear-
layer attached to a frozen feature extractor, a well-established technique in
transfer learning for improving robustness to out-of-distribution shifts [16].
More details of the training process can be found in Section 4.3. Table 1
compares model performance for different feature extractors, specifically,
CLIP [23], ResNet [12], Vision Transformers [9], and random convolutions [24].

Note that the surrogate model alone is not sufficient to perform drag
optimization: starting with an image of a car and minimizing the predicted
drag in pixel space produces results similar to Figure 5(a). We remedy this
using guided diffusion, which will interleave drag minimization steps with
denoising steps.

Figure 4: Training the drag estimator

(a) Naive drag optimization on pixel space

(b) Drag optimization interleaved with denoising Image manifold
---
Low predicted drag
---
$x_{t+1}$
---
$x_{t}$
---
$\hat{x}_{0}^{t}$
---
$\hat{x}_{\text{drag}}^{t}$
---
Optimization target
---
$x_{t-1}$
---

Figure 5: An illustration of how drag guidance works. Noisy images produced by naively minimizing the surrogate model (Figure 5(a)) are avoided by the inclusion of denoising steps (Figure 5(b)). Features | Dimension | Train $R^{2}$ | Test $R^{2}$ | Test MSE
---|---|---|---|---
CLIP | 512 | 0.639 | 0.545 | 0.00251
ViT | 768 | 0.639 | 0.549 | 0.00249
ResNet | 2048 | 0.630 | 0.509 | 0.00271
Random | 2560 | 0.640 | 0.554 | 0.00246
Table 1: Performance of drag estimator with different feature extractors.

###  4.2 Drag-guided diffusion

Let $\phi_{\text{drag}}:\mathbb{R}^{C\times H\times W}\rightarrow\mathbb{R}$
denote our surrogate model. To implement drag-guided diffusion, we follow [8,
Algorithm 2] and modify the sampling step (2) to

$\displaystyle x_{t-1}$
$\displaystyle=x_{t}-(\sigma_{t}-\sigma_{t-1})(\epsilon_{\theta}(x_{t},t)+\eta_{t}\nabla\phi_{\text{drag}}(\hat{x}_{0}^{t})),$
(4)

where $\eta_{t}$ denotes an iteration-dependent weight and $\hat{x}_{0}^{t}$
denotes the denoised output, defined as
$\hat{x}_{0}^{t}=x_{t}-\sigma_{t}\epsilon_{\theta}(x_{t},t)$. Per [21], we can
interpret construction of $\hat{x}_{0}^{t}$ as approximate projection onto the
image manifold. The iterations (4) in turn can be reinterpreted as a damped
form of projected gradient descent, an algorithm that alternates between
projection and gradient steps. Precisely, (4) implicitly constructs a shadow
iterate $\hat{x}_{\text{drag}}^{t}$ using a projected-gradient descent
iteration, and then dampens the update to $\hat{x}_{\text{drag}}^{t}$ using
the quantity $\alpha_{t}:=1-\sigma_{t-1}/\sigma_{t}$. Consider the following.

######  Proposition 4.1.

The iterations (4) are equivalent to

$\displaystyle\hat{x}^{t}_{0}=x_{t}-\sigma_{t}\epsilon_{\theta}(x_{t},t)~{}$
(5)
$\displaystyle\hat{x}_{\text{drag}}^{t}=\hat{x}^{t}_{0}-\gamma_{t}\nabla\phi_{\text{drag}}(\hat{x}^{t}_{0})$
$\displaystyle
x_{t-1}=(1-\alpha_{t})x_{t}+\alpha_{t}\hat{x}_{\text{drag}}^{t}$

where $\alpha_{t}=1-\frac{\sigma_{t-1}}{\sigma_{t}}$ and
$\gamma_{t}=\sigma_{t}\eta_{t}$.

###### Proof.

Expanding and simplifying the last equation gives

$\displaystyle x_{t-1}$
$\displaystyle=(1-\alpha_{t})x_{t}+\alpha_{t}[x_{t}-\sigma_{t}\epsilon_{\theta}(x_{t},t)-\gamma_{t}\nabla\phi_{\text{drag}}(\hat{x}^{t}_{0})]$
$\displaystyle=x_{t}+\alpha_{t}[\sigma_{t}\epsilon_{\theta}(x_{t},t)-\gamma_{t}\nabla\phi_{\text{drag}}(x^{t}_{0})].$

Equating with (4) shows that $\sigma_{t}-\sigma_{t-1}=\alpha_{t}\sigma_{t}$
and $(\sigma_{t}-\sigma_{t-1})\eta_{t}=\alpha_{t}\gamma_{t}$. Rearranging
proves the claim. ∎

In total, we can interpret construction of $\hat{x}^{t}_{0}$ as stepping
towards the image manifold, and construction of $\hat{x}_{\text{drag}}^{t}$ as
a drag-minimization step. This is illustrated by Figure 5(b).

###  4.3 Implementation details

To train our model, we used side-view renderings from the dataset [31]. For
each item in this dataset, we randomly generated 10 new training examples
using horizontal flips, vertical shifts (up to 25 pixels) and jitter (up to
five percent) in brightness, contrast, saturation and hue. The renderings were
also resized to 224 by 224 pixels to match the input size of the pretrained
feature extractors, which we obtained from the following Huggingface
repositories:

$\footnotesize\texttt{openai/clip-vit-base-patch32},\;\;\texttt{google/vit-
base-patch16-224-in21k},\;\;\texttt{microsoft/resnet-50}.$

We also compared performance to random convolutional features, which we
constructed using a kernel size of five and weights drawn from a standard
normal distribution. At the output of these convolutions, we applied a bias of
+2, a ReLU nonlinearity, and a pooling operation using a 55 by 55 window.
After extracting the features, we use ridge regression with regularization
parameter $\lambda=100$ and $\lambda=10$, for the pretrained and random
features respectively, to train the last linear layer.

Finally, drag guidance was incorporated into a checkpoint of Stable Diffusion
available at runwayml/stable-diffusion-v1-5, with
$\eta_{t}=\eta_{0}/\sqrt{1+1/\sigma_{t}^{2}}$, and $\eta_{0}=400$. During
sampling, we fix a reference image $x_{0}$, which is a side-view rendering of
a car, and set $T=80$. Then we let $x_{T}=x_{0}+\sigma_{T}\epsilon$, where
$\epsilon\sim\mathcal{N}(0,I)$, a commonly used technique in image-to-image
generation [19]. We also fix a text prompt and use a scale of 7.5 for
classifier-free text guidance.

##  5 Experimental Results

###  5.1 Generation of novel designs

Baseline | Drag optimized | Estimated Drag
---|---|---
|  |
|  |
Figure 6: Outputs of our image generation pipeline with drag optimization,
compared to baseline images with the drag optimization turned off.

We first consider generation of novel vehicle images with drag guidance.
Compared to baseline generation, we find that drag guidance can change the
generated image in subtle ways (e.g. artifact removal or surface smoothing as
in Figure 6) or through significant redesigns (e.g, adding a roof to a
convertible as in Figure 8). In all cases, however, a coherent vehicle shape
is preserved, as is the artistic style, showing that text-based guidance can
work in concordance with drag optimization. Sample results are shown in Figure
3 and Figure 6. Figure 6 also confirms that drag guidance indeed decreases
predicted drag. Appendix A contains more examples.

###  5.2 Drag-guided redesign

Recall that the DDIM sampler is initialized via
$x_{T}=x_{0}+\sigma_{T}\epsilon$, where $x_{0}$ denotes a provided reference
image and $\epsilon\sim\mathcal{N}(0,I)$. Deviation of the DDIM output from
$x_{0}$ is controlled by the size of the noise-level $\sigma_{T}$. For
$\sigma_{T}$ sufficiently small, the deviation preserves the basic structure
of $x_{0}$. This leads to drag-guided _redesigns_ of the reference. Figure 7
illustrates redesigns produced by this procedure. Note that for each redesign,
the initial predicted drag is the same and equal to the predicted drag of the
reference image.

Reference |  |
---|---|---
Design 1 |
Design 2 |
Design 3 |  |
Figure 7: Drag-guided redesigns of a reference image.

###  5.3 Drag-guided diffusion with a gradient-estimation sampler

A recent variation on DDIM is based on reinterpreting sampling as approximate
gradient-descent on the Euclidean distance function of the training set [21].
This leads to a gradient-estimation (GE) sampler that averages
$\epsilon_{\theta}(x_{t},t)$ over multiple time-steps, exploiting the fact
that the gradient of the distance function is constant along gradient-descent
trajectories [21, Section 4]. To use this sampler, one replaces
$\epsilon_{\theta}(x_{t},t)$ in (4) with
$\gamma\epsilon_{\theta}(x_{t},t)+(1-\gamma)\epsilon_{\theta}(x_{t+1},t+1)]$
for chosen $\gamma\in\mathbb{R}$. We compare outputs between GE and DDIM
samplers in Figure 8 using $\gamma=2$.

Baseline | Drag optimized (DDIM) | Drag optimized (GE)
---|---|---
|  |
|  |
|  |
Figure 8: Comparison of DDIM and the gradient-estimation (GE) sampler

###  5.4 Distributional robustness of drag estimator

Our drag estimator $\phi_{\text{drag}}$ should generalize to the set of
intermediate denoised images
$\hat{x}_{0}^{t}:=x_{t}-\sigma_{t}\epsilon_{\theta}(x_{t},t)$ generated by the
diffusion process. There are two forms of distributional shift that inhibit
good generalization. First, the dataset used to train the denoiser
$\epsilon_{\theta}$ differs from the dataset used to train
$\phi_{\text{drag}}$. Second, the distribution of the denoised outputs
$\hat{x}_{0}^{t}$ differs from that of the $\epsilon_{\theta}$-training set,
given that the denoiser is imperfect, i.e., given that
$\epsilon\neq\epsilon_{\theta}(x+\sigma_{t}\epsilon,t)$. While it is difficult
to evaluate this first type of distributional shift, robustness to the second
type can be evaluated with controlled experiments. We carry out two such
experiments to evaluate the robustness of $\phi_{\text{drag}}$ as a function
of the chosen feature extractor (Section 4.1). We also note that the first
type of shift may be mitigated by retraining $\phi_{\text{drag}}$ using
vehicle images from the Stable Diffusion training set, but defer this to
future research.

In our first experiment, we take an image used to train $\phi_{\text{drag}}$
and add varying levels of noise. We then predict the drag of the different
denoised images $\hat{x}_{0}^{t}$. A visualization of $\hat{x}_{0}^{t}$ and
drag predictions for different noise levels on a single example image is shown
in Figure 9. We see that the drag estimator can over or underestimate the
predicted drag, and that the error increases with the magnitude of added
noise.

|
---|---
Figure 9: Drag estimator performance on out of distribution data:
$\hat{x}_{0}^{t}$ with varying amount of noise added to a single image.

Next, we perform the same experiment over the entire training dataset, and
plot the results in Figure 10. The generalization error is measured with the
predicted mean-squared-error of the drag coefficient, which increases with the
amount of noise added. We see that the features obtained from pretrained
neural networks generalize much better than the random features, even though
they have similar performance on the training data.

|
---|---
Figure 10: Drag estimator performance on out of distribution data as measured
by mean-squared error on the entire dataset. The left plot is a “zoomed-in”
version of the right plot, showing that random features leads to a much higher
generalization error than features from pre-trained neural networks.

##  6 Conclusion, Limitations and Future Work

Generative AI tools that can synthesize engineering constraints and
performance metrics could rapidly speed-up the design process. This work is a
step towards building such tools and illustrated, as a proof of concept, that
current image generation techniques can be modified to optimize aerodynamic
drag. Other performance metrics that can be inferred from images could be
incorporated using appropriate modifications.

A current limitation of our pipeline is its difficulty controlling—or even
measuring—the error of the surrogate model on the final drag-optimized output,
since the ground-truth drag coefficient is not uniquely determined nor
computable from the generated 2D rendering. Future work could alter our
approach in a few ways to obtain a unique ground truth. For instance, one
could formulate the problem completely in 2D by replacing our surrogate
model—which predicts 3D drag from a 2D rendering—with a model that predicts 2D
drag from a planar car. Alternatively, one can formulate the problem
completely in 3D by replacing Stable Diffusion with a 3D generative model.
Another limitation of our approach is its 2D image domain formulation, which
may limit its design applications. Replacing images with CAD or 3D models is a
natural next step. We note combining parametric 3D-models with image diffusion
models is the central task of _score distillation sampling_ [22]. Adding
physics-based guidance to this technique is an interesting topic for future
research.

Despite unavailability of ground-truth, there are instances when inaccuracy of
the surrogate model is evident. For example, in Figure 11, the surrogate model
predicts dramatically different drag coefficients for two cars with similar
outlines. Furthermore, the predicted coefficient for one of the cars is
negative, which is not physically valid. To improve accuracy, future work
could develop active learning techniques that augment the training set by
computing ground-truth drag coefficients for generated vehicles. In addition,
one could investigate surrogate models with an implicit physical bias that,
for instance, only return non-negative drag coefficients and always make
similar predictions for similar shapes.

Baseline | Drag optimized | Estimated Drag
---|---|---
|  |
Figure 11: Images generated with and without drag guidance that demonstrate
incorrect predictions from the surrogate model. While the car outline is
largely unmodified by guidance, the surrogate model predicts large
improvements and even negative drag coefficients in the final iterations
(blue).

## Acknowledgements

The authors thank Faez Ahmed for several useful discussions.

## References

  * Abbas et al. [2022] A. Abbas, A. Rafiee, M. Haase, and A. Malcolm.  Geometrical deep learning for performance prediction of high-speed craft.  _Ocean Engineering_ , 258:111716, 2022.
  * Allen et al. [2022] K. R. Allen, T. Lopez-Guevara, K. Stachenfeld, A. Sanchez-Gonzalez, P. Battaglia, J. Hamrick, and T. Pfaff.  Physical design using differentiable learned simulators.  _arXiv preprint arXiv:2202.00728_ , 2022.
  * Bansal et al. [2023] A. Bansal, H.-M. Chu, A. Schwarzschild, S. Sengupta, M. Goldblum, J. Geiping, and T. Goldstein.  Universal guidance for diffusion models.  _arXiv preprint arXiv:2302.07121_ , 2023.
  * Baque et al. [2018] P. Baque, E. Remelli, F. Fleuret, and P. Fua.  Geodesic convolutional shape optimization.  In _International Conference on Machine Learning_ , pages 472–481. PMLR, 2018.
  * Chung et al. [2022a] H. Chung, J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye.  Diffusion posterior sampling for general noisy inverse problems.  _arXiv preprint arXiv:2209.14687_ , 2022a.
  * Chung et al. [2022b] H. Chung, B. Sim, D. Ryu, and J. C. Ye.  Improving diffusion models for inverse problems using manifold constraints.  _arXiv preprint arXiv:2206.00941_ , 2022b.
  * Crowson [2021] K. Crowson.  CLIP guided diffusion, 2021.  URL https://colab.research.google.com/drive/12a_Wrfi2_gwwAuN3VvMTwVMz9TfqctNj.
  * Dhariwal and Nichol [2021] P. Dhariwal and A. Nichol.  Diffusion models beat GANs on image synthesis.  _Advances in Neural Information Processing Systems_ , 34:8780–8794, 2021.
  * Dosovitskiy et al. [2020] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Durasov et al. [2021] N. Durasov, A. Lukoyanov, J. Donier, and P. Fua.  Debosh: Deep bayesian shape optimization.  _arXiv preprint arXiv:2109.13337_ , 2021.
  * Giannone and Ahmed [2023] G. Giannone and F. Ahmed.  Diffusing the optimal topology: A generative optimization approach.  _arXiv preprint arXiv:2303.09760_ , 2023.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Ho and Salimans [2022] J. Ho and T. Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Jacob et al. [2021] S. J. Jacob, M. Mrosek, C. Othmer, and H. Köstler.  Deep learning for real-time aerodynamic evaluations of arbitrary vehicle shapes.  _arXiv preprint arXiv:2108.05798_ , 2021.
  * Kim et al. [2021] G. Kim, T. Kwon, and J. C. Ye.  DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation.  10 2021.  doi: 10.48550/arxiv.2110.02711.  URL https://arxiv.org/abs/2110.02711v6.
  * Kumar et al. [2022] A. Kumar, A. Raghunathan, R. Jones, T. Ma, and P. Liang.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  _arXiv preprint arXiv:2202.10054_ , 2022.
  * Liu et al. [2023] R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  _arXiv preprint arXiv:2303.11328_ , 2023.
  * Mazé and Ahmed [2022] F. Mazé and F. Ahmed.  Topodiff: A performance and constraint-guided diffusion model for topology optimization.  _arXiv preprint arXiv:2208.09591_ , 2022.
  * Meng et al. [2021] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  In _International Conference on Learning Representations_ , 2021.
  * Nichol et al. [2021] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen.  GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.  12 2021.  doi: 10.48550/arxiv.2112.10741.  URL https://arxiv.org/abs/2112.10741v3.
  * Permenter and Yuan [2023] F. Permenter and C. Yuan.  Interpreting and improving diffusion models using the Euclidean distance function.  _arXiv preprint arXiv:2306.04848_ , 2023.
  * Poole et al. [2022] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv preprint arXiv:2209.14988_ , 2022.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Rahimi and Recht [2007] A. Rahimi and B. Recht.  Random features for large-scale kernel machines.  _Advances in neural information processing systems_ , 20, 2007.
  * Ramesh et al. [2022] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Remelli et al. [2020] E. Remelli, A. Lukoianov, S. Richter, B. Guillard, T. Bagautdinov, P. Baque, and P. Fua.  Meshsdf: Differentiable iso-surface extraction.  _Advances in Neural Information Processing Systems_ , 33:22468–22478, 2020.
  * Rombach et al. [2022] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Rosset et al. [2023] N. Rosset, G. Cordonnier, R. Duvigneau, and A. Bousseau.  Interactive design of 2d car profiles with aerodynamic feedback.  In _Computer Graphics Forum_ , volume 42, 2023.
  * Saha et al. [2021] S. Saha, T. Rios, L. L. Minku, B. V. Stein, P. Wollstadt, X. Yao, T. Back, B. Sendhoff, and S. Menzel.  Exploiting generative models for performance predictions of 3d car designs.  In _2021 IEEE Symposium Series on Computational Intelligence (SSCI)_ , pages 1–9. IEEE, 2021.
  * Saharia et al. [2022] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _arXiv preprint arXiv:2205.11487_ , 2022.
  * Song et al. [2023] B. Song, C. Yuan, N. Arechiga, F. Permenter, and F. Ahmed.  Surrogate modeling of car drag coefficient with depth and normal rendering.  In _Proceedings of the ASME 2023 IDETC/CIE_ , 2023.
  * Song et al. [2020] J. Song, C. Meng, and S. Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020.
  * Thuerey et al. [2020] N. Thuerey, K. Weißenow, L. Prantl, and X. Hu.  Deep learning methods for reynolds-averaged navier–stokes simulations of airfoil flows.  _AIAA Journal_ , 58(1):25–36, 2020.
  * Umetani and Bickel [2018] N. Umetani and B. Bickel.  Learning three-dimensional flow for interactive aerodynamic design.  _ACM Transactions on Graphics (TOG)_ , 37(4):1–10, 2018.

##  Appendix A Additional examples

Baseline | Drag optimized | Predicted Drag
---|---|---
|  |
|  |
|  |
|  |
Figure 12:  Images generated from the prompt “A car, high resolution studio photo, side view” with and without drag optimization. We also plot drag coefficients predicted by the surrogate model.  Baseline | Drag optimized | Predicted Drag
---|---|---
|  |
|  |
|  |
|  |
Figure 13:  Images generated from the prompt “An old fashioned car, high resolution studio photo, side view” with and without drag optimization. We also plot drag coefficients predicted by the surrogate model.  Baseline | Drag optimized | Predicted Drag
---|---|---
|  |
|  |
|  |
|  |
Figure 14:  Images generated from the prompt “An iconic car, high resolution
studio photo, side view” with and without drag optimization. We also plot drag
coefficients predicted by the surrogate model.