# 3D Shape Augmentation with Content-Aware Shape Resizing

Mingxiang Chen, Jian Zhang, Boli Zhou, Yang Song
Ant Group
Xihu District, Hangzhou, China
{mingxiang.cmx, zj134362, zhouboli.zbl<EMAIL_ADDRESS>

###### Abstract

Recent advancements in deep learning for 3D models have propelled
breakthroughs in generation, detection, and scene understanding. However, the
effectiveness of these algorithms hinges on large training datasets. We
address the challenge by introducing Efficient 3D Seam Carving (E3SC), a novel
3D model augmentation method based on seam carving, which progressively
deforms only part of the input model while ensuring the overall semantics are
unchanged. Experiments show that our approach is capable of producing diverse
and high-quality augmented 3D shapes across various types and styles of input
models, achieving considerable improvements over previous methods.
Quantitative evaluations demonstrate that our method effectively enhances the
novelty and quality of shapes generated by other subsequent 3D generation
algorithms.

Figure 1: Given an input model (rendered in red in the middle), our approach
has the capability to generate a variety of augmented 3D shapes characterized
by intricate structures and precise details.

##  1 Introduction

In recent years, there has been a rapid advancement in deep learning
technologies associated with three-dimensional models. The corresponding
algorithms have achieved significant breakthroughs in various tasks, including
the generation, recognition, detection, and understanding of three-dimensional
models and scenes. Despite the notable superiority of deep learning algorithms
over traditional machine learning methods, their effectiveness hinges on the
availability of large datasets for training. To address this challenge, the
adoption of data augmentation has become widespread, especially in domains
where substantial datasets are not readily accessible— a circumstance
frequently encountered in the context of many 3D tasks. The primary objective
of data augmentation is to generate supplementary data for training, and its
efficacy in enhancing performance has been empirically validated.

A traditional and straightforward way of 3D shape augmentation is axis
scaling, yet it is not content-aware, and the scaled 3D shape may be
distorted. Alternative methods have also indicated limited effectiveness in
addressing the issue of non-logical deformation. Though widely used in 2D
image generation, detection, recognition, and many other tasks, the
augmentation of 3D models has been relatively underexplored. On the other
hand, the significance of varied and high-quality 3D content is growing within
multiple industries, encompassing gaming, robotics, architecture, and social
platforms. Constricting training datasets to a narrow scope of 3D models poses
limitations on the algorithm’s potential in various respects. Consequently,
the development of an augmentation technique capable of generating diverse and
high-quality 3D models is imperative.

In this paper, we present a novel yet straightforward augmentation method that
produces diverse variations when given a 3D model as input. Our approach
leverages the content-aware 2D image resizing technique based on seam carving,
ensuring precise 3D seam prediction and enhanced computational efficiency.
Additionally, we mitigate the issue of diversity by introducing the ’anchor
points’ into our approach.

Overall, our contributions are summarized as follows:

  * •

We propose Efficient 3D Seam Carving, a content-aware 3D model augmentation
algorithm, that evaluates which parts of the model can be deformed along which
specific directions, thereby appropriately implementing different deformation
strategies for various regions of an input model.

  * •

We introduce beam search and anchor point selection techniques to ensure that
our method computes efficiently and outputs various 3D shapes.

  * •

We show compelling results and compare our method with previous methods. The
results show that our method produces high-quality and diverse augmented 3D
shapes among varied types of input models.

The paper is organized as follows. First, we introduce related works in
Section 2. The details of our method are explained in Section 3. The settings
of experiments and their results are discussed in Section 4. The conclusion is
presented in Section 5.

##  2 Related Works

Content-aware 2D image retargeting. Content-Aware Image Retargeting (CAIR)
techniques are important for displaying images or videos on various devices
with diverse aspect ratios [2]. Compared to naive resizing methods such as
uniform scaling and fixed-window cropping, CAIR methods preserve crucial
regions of the input image while minimizing the occurrence of artifacts and
distortion. Generally speaking, the CAIR methods can be classified into 4
categories [2]: 1) discrete methods [3, 27, 1, 14], 2) continuous methods [11,
18, 8], 3) multi-operator methods [28, 7], and 4) deep learning based methods
[32, 21, 19]. Numerous methodologies have been proposed within each category,
each with its own set of advantages and limitations. Among them, the Seam
Carving (SC) [3] algorithm is a set of discrete methods upon which many CAIR
techniques are based. The SC method first computes the energy of each pixel
based on a pre-defined energy function, typically indicating the pixel’s
significance. Subsequently, the algorithm identifies a low-energy seam and,
based on the target dimensions, determines whether the seam is to be removed
or inserted.

Image and 3D shape augmentation. Data augmentation is a technique commonly
used in computer vision and machine learning to artificially increase the size
of a training dataset and avoid overfitting by applying various
transformations to existing data. Image augmentation proves to be necessary
for numerous 2D image processing tasks [33, 30], including but not limited to
segmentation [17, 22], detection [6, 25, 12], unsupervised learning [15, 5,
13, 17], and various other applications. The 2D image augmentation techniques
can be grouped into three main categories [33]: model-free approaches, model-
based approaches, and optimizing policy-based. The model-free approach
leverages traditional image processing techniques such as transformations like
cropping, rotation, flipping, and scaling, whereas the model-based approach
capitalizes on image generation models [36, 10] to produce synthetic images.
On the other hand, the optimizing policy-based method seeks to balance and
find the most advantageous combination of both [9]. Moreover, image
augmentation techniques may vary depending on the target domain, for example,
medical images [20, 36, 10], agricultural images [34], and satellite images
[23].

As for the 3D shapes, to the best of our knowledge, random scaling [24, 31] is
probably the most widely adopted augmentation approach. Other methods include
piecewise warping [24], shape uniting [37], spectral augmentation [26], and
planar decimation [24, 31] (as documented in Blender’s manual
111https://docs.blender.org/manual/en/latest/modeling/modifiers/generate/decimate.html).
These methodologies lack content-awareness. Consequently, the execution of
major augmentation, such as a large scaling factor, may yield results with
noticeable artifacts and distortions (except for planar decimation).
Conversely, when minor augmentation is applied, the augmented model may differ
imperceptibly from the original model, thereby potentially being detrimental
to the algorithm’s generalizability. In the context of 2D images, cropping
serves as a straightforward remedy for the aforementioned distortion issues.
However, in the case of 3D models, integrity is important, and thus, cropping
is generally not considered a viable augmentation technique.

##  3 Method

###  3.1 Overview

A 3D shape can be represented as a discrete occupancy function
$o:z\in\mathcal{Z}\mapsto{0,1}$ which is defined on a 3D grid $\mathcal{Z}$.
$o(z)$ records the value of occupancy that $o(z)=1$ if the center of the grid
cell is inside of the 3D shape and $o(z)=0$ otherwise. Let $\mathbf{G_{o}}$ be
an $N_{i}\times N_{j}\times N_{k}$ grid containing the values of occupancies
given a water-tight 3D shape. A spatial seam is defined to be

$\mathbf{s^{z}}=\\{s^{z}_{i,j}\\}=\\{(i,j,z(i,j))\\}_{i,j},$ (1)

where $i,j,k$ are coordinate values on the x, y, and z axis, respectively.
$i,j,k\in\mathbb{Z}$, $1\leq i\leq N_{i}$, $1\leq j\leq N_{j}$, $1\leq k\leq
N_{k}$, and $\forall i,j$, $|z(i,j)-z(i-1,j)|\leq 1$, $|z(i,j)-z(i,j-1)|\leq
1$. Here, $z$ is a mapping
$z:[1,...,N_{i}]\times[1,...,N_{j}]\mapsto[1,...,N_{k}]$ representing the
coordinate values on the $z$ axis. That is, a plane splits the 3D shape to
two, from top to bottom, from left to right, and containing exactly one voxel
for every single coordinate $(i,j)$. In this example, the x-axis and y-axis
are associated with independent variables, while the z-axis is associated with
the dependent variables. Spatial seams that cut the 3D shape from different
directions (_e.g_. from top to bottom, from front to back) are defined in
similar ways.

Similar to the occupancy function, a discrete signed distance function (SDF)
$g:z\in\mathcal{Z}\mapsto\mathbb{R}$ is defined on a 3D grid $\mathcal{Z}$,
where the value records the signed distance from the center of the grid cell
to the surface of the 3D shape. Specifically, we can restrict the values of
the signed distance within a defined range, referred to as the Truncated
Signed Distance Function (TSDF). The grid containing the SDF and TSDF values
are denoted as $\mathbf{G_{s}}$ and $\mathbf{G_{t}}$, respectively. The
definition of the spatial seam in the signed distance grid is analogous to the
definition in the occupancy grid.

Given an energy function $e$, the cost of a spatial seam is

$E(s)=\sum_{i,j}e(\mathbf{G}(s^{z}_{i,j})),$ (2)

where we define the energy function as:

$e_{z}(\mathbf{G})=|\frac{\partial}{\partial z}\mathbf{G}|,$ (3)

and the variable $z$ in Eq. (3) changes following the direction of the spatial
seam. We name it as the cutting axis. Different from the energy function
designed for 2D images [27, 3], where the energy function is commonly
associated with the first-order derivatives of the image along the horizontal
and vertical axes, typically represented by the sum of their absolute values,
we mainly focus on capturing the gradient information of the model exclusively
along a specific axis. The rationale behind this observation stems from the
fact that when the gradient along a specific axis of a 3D shape reaches zero,
it commonly signifies the potential for scale variation along that particular
axis. To illustrate, take a handleless cup where the gradients along the
y-axis (representing the height direction) are zero in the central segment. If
we elongate the central segment along the y-axis, we would achieve a slightly
taller or shorter cup while preserving a common shape. In contrast, stretching
the cup along the x-axis or z-axis would result in an elliptical cup, a less
frequent occurrence.

$e_{3}(\mathbf{G})=|\frac{\partial}{\partial
x}\mathbf{G}|+|\frac{\partial}{\partial
y}\mathbf{G}|+|\frac{\partial}{\partial z}\mathbf{G}|.$ (4)

Figure 2: Comparison between different energy functions. Given the same input
model (a), we show three augmented results (b-d) using Eq. 3, and another
three results (e-g) using Eq. 4. The anchors are not limited to occupied cells
in both cases.

In Section 4, a comparative analysis will be conducted between the
aforementioned energy function and an alternative energy function considering
the first-order derivatives along all three axes. In short, aside from the
higher computational cost of using 4, the results of the two are similar. Our
augmentation algorithm follows the structure outlined in Algorithm 1. Notably,
the augmentation of a 3D shape involves adjustments along its three axes,
where the model is rotated to designate its x-axis, y-axis, and z-axis
successively as the cutting axis. Furthermore, a maximum scaling factor
denoted as $S_{max}$ ensures that the dimension $N$ of the model along a
specific axis remains within the range
$[N\times(1-S_{max}),N\times(1+S_{max})]$ after the augmentation.

Initialization: $\mathbf{G}$, $n$

// beam size

foreach  _$axis\in\\{\textbf{x},\textbf{y},\textbf{z}\\}$_ do

$\mathbf{G}\leftarrow$ Rotate($\mathbf{G},axis$)

$N_{i},N_{j},N_{k}\leftarrow$ Shape($\mathbf{G}$)

$delta\leftarrow$ RandInt(-$N_{k}*S_{max}$, $N_{k}*S_{max}$+1)

for  _$i=0;\ i <delta;\ i=i+1$_ do

$e\leftarrow$ Eq. (3)

$e_{x}\leftarrow$ Eq. (5)

$e_{y}\leftarrow$ Eq. (6)

$a\leftarrow$ SetAnchor($e$)

$p^{x}\leftarrow$ BeamSearch2D($n,e_{x},a$)

$p^{y}\leftarrow$ BeamSearch2D($n,e_{y},a$)

$seam^{x}\leftarrow$ BeamSearch3D($n,e,p^{x}$)

$seam^{y}\leftarrow$ BeamSearch3D($n,e,p^{y}$)

$seam\leftarrow$ MinCostSeam($seam^{x},seam^{y}$)

if  _$delta >0$_ then

$\mathbf{G}\leftarrow$ SeamInsertion($\mathbf{G},seam$)

else

$\mathbf{G}\leftarrow$ SeamRemoval($\mathbf{G},seam$)

end if

end for

$\mathbf{G}\leftarrow$ RotateInverse($\mathbf{G},axis$)

end foreach

Algorithm 1 Algorithm Overview

###  3.2 Beam Search

Solving the global optimal energy or minimal cut on a graph based on a 2D
image or 3D video volume is simply not feasible [27]. Hence, the beam search
algorithm is used to improve efficiency. This effective heuristic search
algorithm enables the generation of coherent sequences by exploring multiple
candidate hypotheses simultaneously. Firstly, an energy map is initialized
using Eq. (3). Subsequently, a 2D energy map is obtained by reducing the
original map on the x or y axis, which is named as the reducing axis:

$e_{x,2D}(j,k)=\sum_{i}e_{z}(i,j,k),$ (5)

$e_{y,2D}(i,k)=\sum_{j}e_{z}(i,j,k).$ (6)

The remaining axis, which is neither the cutting axis nor the reducing axis,
is hereby referred to as the main axis. Given an initial anchor point on the
2D energy map, three possible neighbors on top or bottom (along the main axis)
of it are extended. These extensions are then scored based on the sum of their
energy. Nevertheless, the cumulative inclusion of the three neighbors above or
below the current cell into the existing path will result in an exponential
increase in the overall number of plausible paths. Hence, pruning is then
applied to retain only the top-n candidate extensions, effectively reducing
the search space. The algorithm iteratively repeats the expansion, scoring,
and pruning steps until the candidate paths connect the pixels from the top to
the bottom of the 2D energy map. Finally, the lowest-cost path is selected as
the anchor path. Then the aforementioned procedure is repeated on the 3D
energy map, with the only distinction being that the initial anchor, which was
a single point, is now replaced by the anchor path obtained from the 2D energy
map.

Initialization: $n$, $e_{2D}$, $a$

$(x_{a},y_{a})\leftarrow a$

// anchor point

$h,w\leftarrow$ Shape($e_{2D}$)

$p\leftarrow$ list($h$)

// $p$ is a list of length $h$.

$p[x_{a}]\leftarrow y_{a}$

$beam\leftarrow[p]$

if  _$x_{a}=1$_ then

$x\leftarrow x_{a}+1$

else

$x\leftarrow x_{a}-1$

end if

while  _$x <h$_ do

$b\leftarrow[]$

foreach  _$p\in beam$_ do

if  _$x <x_{a}$_ then

$ys\leftarrow[p[x+1]-1,p[x+1],p[x+1]+1]$

else

$ys\leftarrow[p[x-1]-1,p[x-1],p[x-1]+1]$

end if

foreach  _$y\in ys$_ do

$p_{new}\leftarrow$ copy($p$)

$p_{new}[x]\leftarrow y$

$b$.append($p_{new}$)

end foreach

$beam\leftarrow$ top${}_{n}(b,n,e_{2D})$

end foreach

if  _$x=1$_ then

$x\leftarrow x_{a}+1$

else if  _$x <x_{a}$_ then

$x\leftarrow x-1$

else

$x\leftarrow x+1$

end if

end while

return beam[0]

Algorithm 2 Beam Search on the 2D Energy Map

###  3.3 Diversity

Figure 3: The selection of the anchor point can notably impact the trajectory
of the seam. Taking an occupancy grid of a plane as an example (a), the energy
maps are reduced along the y-axis (b-g). The anchors (illustrated as red dots)
vary along both the z-axis (b-d) and x-axis (e-g), and the cutting axis also
transitions from the z-axis (b-d) to the x-axis (e-g). Both modifications
contribute to a substantial divergence in the seams’ paths.

Figure 4: An illustration of seams computed using our algorithm, with
different cutting axis employed as input.

Beyond considerations of computational complexity, an additional rationale for
not using global optimal energy lies in our expectation for each augmentation
to yield distinct 3D shapes. As depicted in Fig. 3, the selection of anchor
points and the orientation of spatial seams exert a substantial influence on
the trajectory of the seam, thereby ensuring diversity in the resulting models
after each augmentation.

Note that for a relatively regular 3D shape, the partial derivatives of its
voxels along a particular axis are mostly zero or small values. Therefore,
when selecting an anchor point from a 2D energy map, a naive approach is to
randomly choose a pixel as the anchor. However, as the number of iterative
steps increases, the outcomes of augmentations may manifest only marginal
variations. To address this, the possible anchor locations are firstly
partitioned into several clusters using mini-batch $k$-means [29].
Subsequently, top $k$/3 plausible clusters are selected, ranked by their
average cost across $s$ simulations on the 2D energy map, with random points
within these clusters designated as anchors. At the beginning of each
augmentation, $m$ clusters are selected from the top $k$/3 clusters. For each
subsequent step within the augmentation, a cluster is chosen from the selected
subset, and a random point within the chosen cluster is assigned as the
anchor. In our method, the possible anchors are defined as the cells that are
both occupied (or in other words, inside the 3D shape) and the corresponding
energy on $e_{z}$ is less than a threshold $\epsilon$, which equals $1\times
10^{-3}$ in the experiments below.

In contrast to seam carving in images, for an occupancy grid or TSDF grid, the
beam search process yields numerous candidate paths with equivalent cumulative
gradients due to non-zero gradients occurring only on or near the model’s
surface during the search. To ensure that beam search can maximally expand the
search space, thereby finding solutions closer to the optimal, the beam search
is designed to maintain solutions with larger mutual distances when multiple
paths have equal costs during the seam search. More specifically, consider the
following scenario: suppose we have 12 paths and the beam size equals to 4. 1)
If exactly 4 paths have a cumulative gradient less than a specific threshold
T, then retain those 4 paths; 2) If there exists a set A containing three
paths, where the cumulative gradient of each path is less than the threshold
T, and another set B containing three paths, where the cumulative gradient of
each equals T, and the cumulative gradients of the remaining six paths are
greater than T, then select a path from B with the smallest total distance to
all paths in B and place it into set A, and retain the paths in set A; 3) If,
in the second case, the difference between the number of paths in set A and
the beam size is greater than 1, then first select a path from B with the
largest total distance to all paths in B and place it into a candidate set C,
and continuously select the path from B that has not been placed into C and
also has the largest minimum distance to all paths in C until the sum of the
sizes of A and C equals the beam size. Finally, retain the paths in set A and
set C. The distance between two paths $p_{1}$ and $p_{2}$ is defined as:

$D(p_{1},p_{2})=\sum_{x}|p_{1}[x]-p_{2}[x]|.$ (7)

###  3.4 Symmetry Check

Numerous methodologies are available for evaluating the symmetry of an input
3D shape. One approach involves examining the chamfer distance between sampled
surface points from each half and comparing it with a predetermined threshold.
In our proposed method, we quantify symmetry using the elementwise mismatch
rate between two halves. Taking inspiration from the $F_{1}$ score, which
appropriately balances precision and recall, we employ the harmonic mean of
mismatch rates for both occupied and unoccupied cells:

$m_{x}(i,j,k)=\mathbf{G_{o}}(i,j,k)\oplus\mathbf{G_{o}}(N_{i}-i,j,k)$ (8)

$rate_{o,x}=\frac{\sum_{i,j,k}m_{x}(i,j,k)\times\mathbf{G_{o}}(i,j,k)}{\sum_{i,j,k}\mathbf{G_{o}}(i,j,k)}$
(9)

$rate_{u,x}=\frac{\sum_{i,j,k}m_{x}(i,j,k)\times(1-\mathbf{G_{o}}(i,j,k))}{\sum_{i,j,k}1-\mathbf{G_{o}}(i,j,k)}$
(10)

$rate_{x}=2*\frac{rate_{o,x}*rate_{u,x}}{rate_{o,x}+rate_{u,x}}$ (11)

where $\oplus$ stands for the XOR operation, and $rate_{x}$ represents the
mismatch rate of the input 3D shape along the x-axis. Our definition of
mismatch rate along the y-axis and z-axis is analogous to that of $rate_{x}$,
a redundant elaboration is omitted at this juncture.

A 3D shape is considered symmetrical along a specific axis when the mismatch
rate is below a predefined threshold, denoted as $T_{s}$. If the condition is
satisfied, given that it would produce two results through mirror
transformations (as long as the path is not symmetrical along that axis), the
output is determined by selecting the seam with the lower total energy. If the
input model is symmetrical along multiple axes, an exhaustive exploration of
all feasible combinations of mirror transformations is conducted, and the
optimal seam is selected as the final output.

###  3.5 Performance Trade-offs

Solving the near-optimal seam is hard, particularly when dealing with high-
resolution grids. In our experiments, augmenting a 3D shape with a resolution
under 128 may require a few seconds, whereas resolutions exceeding 512 can
extend the process to several minutes. One plausible optimization involves
solely executing a 2D beam search, and stacking the identified path along the
reducing axis to establish the resulting seam. Despite receiving significant
performance improvement, this approach is accompanied by a substantial decline
in seam quality. Hence, we introduce an additional filtering mechanism that if
the average energy of the seam surpasses a threshold $T_{c}$, the seam is
discarded. In our conducted experiments, the algorithm gives satisfactory
results when:

$e_{avg}=\frac{\sum_{i,j,k}e(i,j,k)}{N_{i}\times N_{j}\times N_{k}}$ (12)

$T_{c}=\begin{cases}e_{avg}*0.25,&\text{if }e_{avg}>4\times 10^{-4}\\\ 1\times
10^{-4},&\text{otherwise}\end{cases}$ (13)

##  4 Experiments

Dataset. We employ ShapeNetV2 [4] as the evaluation set for our augmentation
method. In our experiments, we use Open3D to calculate the TSDF grid and the
occupancy grid, which integrates of multiple depth images to derive the TSDF
representation. Unless specified, the models in the following experiments are
represented by occupancy grids.

Qualitative demonstration. The augmented outcomes by our method are compared
with those obtained through axis scaling [24], piecewise linear warping [24],
and spectral augmentation technique [26]. In alignment with [24], we
independently scale each axis by uniformly sampling scaling factors within the
range $[0.75,1.25]$. For the piecewise linear warping, we first normalize the
vertices of the models to constrain the coordinate values to the interval
$[-1,1]$. Subsequently, we define a piecewise linear warping function by
partitioning the interval [-1, 1] into 6 even sub-intervals. The warping
factors are sampled from a log-normal distribution with a standard deviation
of 0.25. Consistent with the previous research, the warping factors exhibit
symmetry with respect to the x and z-axes. We adopt identical configurations
as outlined in [26] during the implementation of spectral augmentation.

As shown in Fig. 2, we compare the augmented results using Eq. 3 with an
alternative function taking the first-order derivatives along all three axes
into account using Eq. 4. In most cases, the results are similar. One
difference is that if the anchor cell is not an occupied cell (which is not
true in our method), the seam paths are sometimes likely to pass through the
tips of the 3D model where the sum of the energy is low if Eq. 4 is used. This
can result in a 3D shape deviating from common expectations, for example, the
flattened tips of the planes’ noses.

In Fig. 7 and Fig. 5, we present illustrative instances of augmented 3D shapes
spanning various genres and styles. We utilize occupancy grids and TSDF grids
to represent the models, respectively. The outcomes demonstrate the
effectiveness of our approach in augmenting a variety of 3D shapes that align
with common expectations.

Figure 5: Illustration of augmented models based on input 3D shapes
represented by TSDF grids. The first row of each column demonstrates the input
models. Table 1: Quantitative evaluation between the generated shapes produced
by Neural Wavelet-domain Diffusion [16] with (ours) and without (original) our
augmentation method. From the result, we can see that in most cases, our
augmentation method enhances the quality of the outcome especially when the
training data is insufficient.

| MMD $\downarrow$ | COV $\uparrow$ | 1-NNA $\sim 0.5$
---|---|---|---
| CD | EMD | CD | EMD | CD | EMD
| original | ours | original | ours | original | ours | original | ours | original | ours | original | ours
Airplane (2000) | 0.0026 | 0.0029 | 0.0958 | 0.0954 | 0.4630 | 0.4635 | 0.3790 | 0.3840 | 0.7423 | 0.7605 | 0.7878 | 0.7680
Bookshelf (452) | 0.0087 | 0.0089 | 0.1445 | 0.1442 | 0.5243 | 0.5377 | 0.5288 | 0.4823 | 0.5409 | 0.4878 | 0.5420 | 0.5265
Bed (233) | 0.0296 | 0.0134 | 0.2581 | 0.1950 | 0.3090 | 0.5837 | 0.3605 | 0.5494 | 0.8541 | 0.4485 | 0.8476 | 0.5386
Tower (132) | 0.0205 | 0.0098 | 0.1952 | 0.1613 | 0.3864 | 0.5227 | 0.3864 | 0.4697 | 0.7879 | 0.5568 | 0.7386 | 0.6326
Camera (113) | 0.0428 | 0.0196 | 0.2743 | 0.2058 | 0.3540 | 0.6106 | 0.3186 | 0.6106 | 0.8186 | 0.3938 | 0.7655 | 0.4248

Figure 6: Qualitative assessments across various metrics reveal that our
augmentation technique improves the outcomes’ quality in most cases. To
illustrate a more intuitive comparison of disparities, we computed the
absolute difference between 1-NNA results and 0.5.

Quantitative Evaluation. Intuitively, 3D model augmentation should enhance the
quality and diversity of results generated by shape-generation algorithms. We
employ Neural Wavelet-domain Diffusion [16] as the baseline model and compare
the outcomes with and without our augmentation method during training. It is
noteworthy that, apart from the airplane class models, which contain more than
4000 samples, the number of models in other categories is relatively limited.
Specifically, the categories of bookshelf, bed, tower, and camera contain 452,
233, 132, and 113 models, respectively. Prior to training, we augment each
model 8 times.

In line with the previous research [16, 35], we uniformly sample 2,048 points
on each generated shape and evaluate the shapes using three evaluation
metrics: 1) minimum matching distance (MMD), which assesses the fidelity of
the generated shapes; 2) coverage (COV), indicating the extent to which the
generated shapes encompass the given 3D repository; and 3) 1-NN classifier
accuracy (1-NNA), measuring the effectiveness of a classifier in
distinguishing the generated shapes from those in the repository. Generally, a
low MMD, a high COV, and a 1-NNA close to 50% signify satisfactory generation
quality. For the airplane models, we generated 2,000 models for evaluation
purposes. For other categories, we generated an equal number of models as the
training set for evaluation.

From Table 1, it is evident that our augmentation method significantly
enhances the performance of shape generation algorithms based on occupancy
grid, SDF grid, or TSDF grid, particularly when the training data is
insufficient.

Figure 7: Qualitative comparison with axis scaling [24], piecewise warping
[24], and spectral augmentation [26]. We show the results obtained by
augmenting each input model twice using the four methods, respectively.

Human preference study. Given that one of the primary goals of our approach is
to enlarge the 3D shape dataset, the augmented data holds potential for
various downstream applications, such as 3D shape generation, classification,
detection, and so on. Consequently, it is imperative for the augmented models
to exhibit both logical coherence and diversity. Otherwise, the training set
for downstream tasks may be contaminated, thereby yielding unsatisfying
outcomes. Evaluating the quality of 3D shapes is a complex task. We focus on
quantitative metrics derived from human preference evaluations, considering
two key dimensions: diversity and visual quality.

Using models sourced from ShapeNetV2 [4] as our initial seed, we apply various
methods to generate eight augmented 3D shapes for each model. A questionnaire
is then presented, displaying eight 3D shapes augmented by our method compared
to eight by a baseline. Both sets share identical rendering configurations and
originate from the same source model. Subsequently, we pose two binary choice
questions to assess preferences: ’Among the two groups of 3D models, which of
them do you think makes more sense to you? In other words, which one do you
think exhibits a higher quality?’ and ’Which group do you think is more
diverse? (Please exclude models that you consider to be of low quality when
evaluating.)’. Our study involves 150 participants, each compares 10 pairs of
results (2 models from each category).

As shown in Table 2, our proposed method shows competitive performance against
axis scaling in terms of model quality and is more preferred compared to other
baselines on the rest dimensions. It utilizes gradients to evaluate for which
parts and in which directions the model can be deformed. This capability is
likely the key factor contributing to its potential for achieving higher human
preference.

Table 2: Quantitative evaluation on human preference of our proposed method
over three baselines on diversity and quality.

| Quality | Diversity
---|---|---
Ours over axis scaling | 47.2% | 79.4%
Ours over axis piecewise warping | 78.6% | 63.6%
Ours over axis spectral augmentation | 91.4% | 85.0%

Figure 8: An illustration of our method fails to generate high-quality
augmentations for certain 3D models.

Limitations. Although the approach yields augmented 3D shapes of high quality,
we observed that the generated shapes may exhibit artifacts when applied to
pixel-style 3D shapes. Illustrated in Fig. 8, a genuine jet plane does not
feature square engines or a square body, introducing irregular gradients. Our
method struggles to accurately estimate the appropriate scaling direction for
these components. In some other instances, our approach generates augmented
desks and chairs with uncommon shapes. For instance, we observe tables with
unusually chunky tabletops and chairs with very thick backrests.

##  5 Conclusion

In this paper, we introduce a novel 3D model augmentation method based on 3D
seam carving. Our approach produces diverse and high-quality results across
various types of models in the form of both occupancy grids and SDF grids. The
proposed method may be useful for many tasks related to 3D models such as
model generation, detection, scene segmentation, 3D model recognition, and so
on. We expect that this method could offer more pronounced assistance for
training deep learning models with smaller datasets.

## References

  * [1] Radhakrishna Achanta and Sabine Süsstrunk.  Saliency detection for content-aware image resizing.  In 2009 16th IEEE international conference on image processing (ICIP), pages 1005–1008. IEEE, 2009.
  * [2] Bahareh Asheghi, Pedram Salehpour, Abdolhamid Moallemi Khiavi, and Mahdi Hashemzadeh.  A comprehensive review on content-aware image retargeting: From classical to state-of-the-art methods.  Signal Processing, 195:108496, 2022.
  * [3] Shai Avidan and Ariel Shamir.  Seam carving for content-aware image resizing.  In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 609–617. 2023.
  * [4] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [5] Mingxiang Chen, Zhanguo Chang, Haonan Lu, Bitao Yang, Zhuang Li, Liufang Guo, and Zhecheng Wang.  Augnet: End-to-end unsupervised visual representation learning with image augmentation.  arXiv preprint arXiv:2106.06250, 2021.
  * [6] Mingxiang Chen and Cihui Pan.  Parsing line segments of floor plan images using graph neural networks.  arXiv preprint arXiv:2303.03851, 2023.
  * [7] Weiming Dong, Ning Zhou, Jean-Claude Paul, and Xiaopeng Zhang.  Optimized image resizing using seam carving and scaling.  ACM Transactions on Graphics (TOG), 28(5):1–10, 2009.
  * [8] Huan Du, Zhi Liu, Jianliang Jiang, and Liquan Shen.  Stretchability-aware block scaling for image retargeting.  Journal of visual communication and image representation, 24(4):499–508, 2013.
  * [9] Alhussein Fawzi, Horst Samulowitz, Deepak Turaga, and Pascal Frossard.  Adaptive data augmentation for image classification.  In 2016 IEEE international conference on image processing (ICIP), pages 3688–3692. Ieee, 2016.
  * [10] Maayan Frid-Adar, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan.  Gan-based data augmentation for improved liver lesion classification.  2018\.
  * [11] Ran Gal, Olga Sorkine, and Daniel Cohen-Or.  Feature-aware texturing.  Rendering Techniques, 2006(17th):2, 2006.
  * [12] Ross Girshick.  Fast r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015.
  * [13] Philip Haeusser, Johannes Plapp, Vladimir Golkov, Elie Aljalbout, and Daniel Cremers.  Associative deep clustering: Training a classification network with no labels.  In Pattern Recognition: 40th German Conference, GCPR 2018, Stuttgart, Germany, October 9-12, 2018, Proceedings 40, pages 18–32. Springer, 2019.
  * [14] Mahdi Hashemzadeh, Bahareh Asheghi, and Nacer Farajzadeh.  Content-aware image resizing: An improved and shadow-preserving seam carving method.  Signal Processing, 155:233–246, 2019.
  * [15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  * [16] Ka-Hei Hui, Ruihui Li, Jingyu Hu, and Chi-Wing Fu.  Neural wavelet-domain diffusion for 3d shape generation.  In SIGGRAPH Asia 2022 Conference Papers, pages 1–9, 2022.
  * [17] Xu Ji, Joao F Henriques, and Andrea Vedaldi.  Invariant information clustering for unsupervised image classification and segmentation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9865–9874, 2019.
  * [18] Yun Liang, Zhuo Su, and Xiaonan Luo.  Patchwise scaling method for content-aware image resizing.  Signal Processing, 92(5):1243–1257, 2012.
  * [19] Jianxin Lin, Tiankuang Zhou, and Zhibo Chen.  Deepir: A deep semantics driven framework for image retargeting.  In 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pages 54–59. IEEE, 2019.
  * [20] Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez.  A survey on deep learning in medical image analysis.  Medical image analysis, 42:60–88, 2017.
  * [21] Si Liu, Zhen Wei, Yao Sun, Xinyu Ou, Junyu Lin, Bin Liu, and Ming-Hsuan Yang.  Composing semantic collage for image retargeting.  IEEE Transactions on Image Processing, 27(10):5032–5043, 2018.
  * [22] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [23] Sharada Prasanna Mohanty, Jakub Czakon, Kamil A Kaczmarek, Andrzej Pyskir, Piotr Tarasiewicz, Saket Kunwar, Janick Rohrbach, Dave Luo, Manjunath Prasad, Sascha Fleer, et al.  Deep learning for understanding satellite imagery: An experimental survey.  Frontiers in Artificial Intelligence, 3:534696, 2020.
  * [24] Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia.  Polygen: An autoregressive generative model of 3d meshes.  In International conference on machine learning, pages 7220–7229. PMLR, 2020.
  * [25] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [26] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or.  Texture: Text-guided texturing of 3d shapes.  arXiv preprint arXiv:2302.01721, 2023.
  * [27] Michael Rubinstein, Ariel Shamir, and Shai Avidan.  Improved seam carving for video retargeting.  ACM transactions on graphics (TOG), 27(3):1–9, 2008.
  * [28] Michael Rubinstein, Ariel Shamir, and Shai Avidan.  Multi-operator media retargeting.  ACM Transactions on graphics (TOG), 28(3):1–11, 2009.
  * [29] David Sculley.  Web-scale k-means clustering.  In Proceedings of the 19th international conference on World wide web, pages 1177–1178, 2010.
  * [30] Connor Shorten and Taghi M Khoshgoftaar.  A survey on image data augmentation for deep learning.  Journal of big data, 6(1):1–48, 2019.
  * [31] Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Nießner.  Meshgpt: Generating triangle meshes with decoder-only transformers.  arXiv preprint arXiv:2311.15475, 2023.
  * [32] Eungyeol Song, Minkyu Lee, and Sangyoun Lee.  Carvingnet: content-guided seam carving using deep convolution neural network.  IEEE Access, 7:284–292, 2018.
  * [33] Mingle Xu, Sook Yoon, Alvaro Fuentes, and Dong Sun Park.  A comprehensive survey of image augmentation techniques for deep learning.  Pattern Recognition, page 109347, 2023.
  * [34] Mingle Xu, Sook Yoon, Alvaro Fuentes, Jucheng Yang, and Dong Sun Park.  Style-consistent image translation: A novel data augmentation paradigm to improve plant disease recognition.  Frontiers in Plant Science, 12:3361, 2022.
  * [35] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan.  Pointflow: 3d point cloud generation with continuous normalizing flows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 4541–4550, 2019.
  * [36] Xin Yi, Ekta Walia, and Paul Babyn.  Generative adversarial network in medical imaging: A review.  Medical image analysis, 58:101552, 2019.
  * [37] Xin-Yang Zheng, Hao Pan, Peng-Shuai Wang, Xin Tong, Yang Liu, and Heung-Yeung Shum.  Locally attentional sdf diffusion for controllable 3d shape generation.  arXiv preprint arXiv:2305.04461, 2023.


White Paper for the 2024 Solar & Space Physics Decadal Survey

Advancing Theory and Modeling Efforts in Heliophysics

Main Category: Basic research
Subcategories 1: Fundamental space plasma theory,
Subcategories 2: Basic research that is driven by space weather user needs

Principal Author:
Fan Guo, Los Alamos National Laboratory
Email<EMAIL_ADDRESS>
Co-authors
Spiro Antiochos(U. of Michigan), Paul Cassak(West Virginia University), Bin
Chen(NJIT), Xiaohang Chen(U. of Arizona), Chuanfei Dong(PPPL/Princeton),
Cooper Downs(Predictive Science Inc.), Joe Giacalone(U. of Arizona), Colby C.
Haggerty(U. of Hawaii), Hantao Ji(PPPL/Princeton), Judith Karpen(NASA/GSFC),
James Klimchuk(NASA/GSFC), Wen Li(Boston University), Xiaocan Li(Dartmouth
College), Mitsuo Oka(U. of California Berkeley), Katharine K. Reeves(Harvard
Smithsonian CfA), Marc Swisdak(U. of Maryland), and Weichao Tu(West Virginia
University)

Synopsis
Heliophysics theory and modeling build understanding from fundamental
principles to motivate, interpret, and predict observations. Together with
observational analysis, they constitute a comprehensive scientific program in
heliophysics. As observations and data analysis become increasingly detailed,
it is critical that theory and modeling develop more quantitative predictions
and iterate with observations. Advanced theory and modeling can inspire and
greatly improve the design of new instruments and increase their chance of
success. In addition, in order to build physics-based space weather forecast
models, it is important to keep developing and testing new theories, and
maintaining constant communications with theory and modeling. Maintaining a
sustainable effort in theory and modeling is critically important to
heliophysics. We recommend that all funding agencies join forces and consider
expanding current and creating new theory and modeling programs–especially, 1.
NASA should restore the HTMS program to its original support level to meet the
critical needs of heliophysics science; 2. a Strategic Research Model program
needs to be created to support model development for next-generation basic
research codes; 3. new programs must be created for addressing mission-
critical theory and modeling needs; and 4. enhanced programs are urgently
required for training the next generation of theorists and modelers.

## 1 Introduction

Theory and modeling study heliophysical processes from fundamental principles
to understand, interpret, and predict observations. Together with
observational analysis, they constitute a comprehensive scientific program in
heliophysics. In response to observational discoveries, new theories are
developed and tested. Based on indirect observations at the time, Parker
theoretically predicted a supersonic solar wind (Parker, 1958, 1999), which is
directly related to the birth of heliophysics. A complete understanding of
heliophysics processes relies on close collaborations between theory/modeling
and observation/data analysis. Through such integrated efforts,
theory/modeling motivates further observational analysis, data-theory
comparison, and new missions/infrastructures. As observations and data
analysis become increasingly detailed, it is critical that theory/modeling
provides quantitative predictions and iterate with observations for further
progress. For example, recent observations by Parker Solar Probe and other
spacecraft have shown the prevalence of switchbacks in the inner heliosphere
(Bale et al., 2019; Kasper et al., 2019), leading to a new set of theories
proposed to explain the observations and increasingly stringent tests for
those theories (e.g., Squire et al., 2020; Zank et al., 2020; Ruffolo et al.,
2020; Drake et al., 2021; Schwadron & McComas, 2021). In addition, advances in
physics-based space weather models depend critically on the knowledge gained
from basic theory/modeling, and our ability to transition this knowledge into
those models. Therefore, maintaining a sustainable effort in theory and
modeling is essential to both basic and applied heliophysics.

Heliophysical plasma systems host a rich set of physical phenomena and are
often the only places that allow detailed remote sensing and in situ
observations. Many plasma processes, including magnetic reconnection and
collisionless shocks, were discovered or proposed to explain heliophysical
phenomena (e.g., Parker, 1957; Sweet, 1958; Blandford & Eichler, 1987).
Theoretical understanding of these phenomena has far-reaching impacts on our
insight into basic plasma physics, laboratory experiments, and planetary and
astrophysical plasmas. An incomplete list of processes (with corresponding
white papers) includes dynamos, the acceleration and transport of energetic
particles, magnetic reconnection (Ji et al., 2022; Chen et al., 2022),
collisionless shocks (Goodrich et al., 2022), turbulence and waves in fluid
and kinetic scales, wave-particle interaction (Thorne, 2010), plasma heating
and energy dissipation (Haggerty et al., 2022) and effects of partially
ionized plasmas. These processes underlie many important heliophysical
phenomena, including solar and heliospheric magnetic fields, energetic
particles and radiation (Shih et al., 2022; Chen et al., 2022; Turner et al.,
2022; Kollmann et al., 2022), coronal and solar wind heating (Klimchuk et al.,
2022; Mondal et al., 2022), the origin of solar eruptions and magnetospheric
storms and substorms (Chen et al., 2022; Green et al., 2022), the shape and
size of the heliosphere, ionospheric outflow, and the impact of solar activity
on planetary magnetospheres and atmospheres in our solar system (Cohen et al.,
2022; Crary et al., 2022) and beyond (Garcia-Sage et al., 2022).

## 2 The Importance of theory/modeling to observations and instruments

The exercise of theory/modeling leads to a significantly deeper understanding
of observations, identifies gaps in our understanding, and motivates new
observations, missions, and infrastructures. We point out that collaborations
between theory and observations in recent NASA missions, such as IBEX, MMS,
Van Allen Probes, THEMIS, MESSENGER, MAVEN, Juno, IRIS, SDO, Hinode, and
Parker Solar Probe, as well as ground-based facilities such as EOVSA and
DKIST, have led to major progress in our understanding of key heliophysics
phenomena. Current and future space and ground-based missions and
infrastructures would benefit greatly from strong theory/modeling
participation in augmented research programs and science teams, from the
design phase through post-launch data acquisition. In particular, comparing
theory/model predictions with the ground truth of observations can resolve
contentious issues and point to new directions for both observations and
theory. Figure 1 shows a model-data comparison for the solar flare on 2017
September 10, which was observed by multiple spacecraft/facilities including
EOVSA, Hinode, RHESSI, and SDO. The EOVSA observation measures the energetic
electron density and magnetic field strength (Chen et al., 2020; Fleishman et
al., 2020). The MHD model agrees well with the flare dynamics and the
distribution of the magnetic field strength, and the EOVSA radio image is
consistent with an MHD-particle-radiation model (Chen et al., 2020; Li et al.,
2022), suggesting that flare current-sheet reconnection and looptop sources
are crucial accelerators for energetic electrons. Combining advances in
theory/modeling of particle acceleration and flare dynamics and next-
generation radio imaging spectropolarimetry, as would be provided by FASR
(Chen et al., 2022), will further demystify the role of reconnection and
associated particle acceleration processes in flares.

Figure 1: Multi-wavelength observations and numerical modeling of an eruptive
solar flare event on 2017 Sept. 10. The simulation is based on gyrosynchrotron
radiation calculated from an input data-constrained MHD and particle model
shown in (a). (b) Spatial variation of the magnetic field strength along the
current sheet derived from the microwave data (black), which matches the model
predictions (red/blue) (Chen et al., 2020). The simulated EOVSA image using an
MHD-particle model (Li et al., 2022) in (c) well resembles the actually
observed EOVSA image at 5 GHz shown in (d) (see Chen et al. 2022).

## 3 The Importance of basic theory and modeling to space weather models

Physics-based space weather models condense our understanding of basic
heliophysical processes and attempt to predict and interpret space weather
events. However, in many areas of space weather modeling, the current physics
models are far from mature. These applied models need constant development and
input from basic theoretical and modeling studies, as the models often include
important physics uncertainties.

For example, solar energetic particle (SEP) events are thought to be generated
by two processes: shocks driven by coronal mass ejections (CMEs), and
magnetic-reconnection—driven acceleration in solar flares. However, their
relative contributions are still not well understood (Reames, 2020),
especially for particle acceleration associated with solar flares. For shock
acceleration, the relative contributions of quasi-parallel shocks (propagation
along the magnetic field) vs. quasi-perpendicular shocks remain unclear. In
addition, there have been numerous events that are not consistent with one-
dimensional diffusive shock acceleration theory, indicating that effects such
as multi-dimensional physics, downstream turbulence, and/or reconnection may
need to be included in our theoretical approach (Guo et al., 2021). To
accurately calculate particle acceleration at near-Sun CME shocks, the sharp
transition in physical properties at the shock must be resolved down to the
diffusion length of energetic particles. However, multidimensional MHD CME
models usually cannot resolve such scales, while kinetic models focused on the
shock front cannot reach the global scales of the full eruption. The new
generation of CME-SEP models need to address these uncertainties and overcome
the difficulties to achieve more reliable predictive capabilities.

Numerical models of space weather at Earth and other planets often rely on
global simulations of magnetospheric dynamics driven by interplanetary shocks
and CMEs, corotating interaction regions, magnetic storms, and energy transfer
to the ionosphere and thermosphere by electric fields and field-aligned
currents. The first generation of space weather forecasting tools are
primarily MHD-based. However, these fluid simulations cannot robustly model
key processes such as magnetic reconnection at kinetic scales, but rather rely
upon resistivity or viscosity (physical or numerical) to regulate sharp
structures in critically important boundary layers such as thin current sheets
associated with reconnection. Such traditional proxies are crude and have
limited predictive capability for space weather events. Extensive event
studies over the last decade (e.g., Burch et al., 2016, and the references
therein) have demonstrated that it is critical to include kinetic physics in
the narrow reconnection layers and magnetosphere-ionosphere coupling regions.

These gaps have not been fully resolved and need further development. Only
through persistent theoretical exploration can numerical models yield
quantitative observables and predictions with enough confidence
level/technology readiness level to compare with observations. There has been
great progress in fundamental comprehension of particle acceleration during
magnetic reconnection (Dahlin et al., 2017; Li et al., 2019; Arnold et al.,
2021; Zhang et al., 2021; Li et al., 2021, 2022), but particle acceleration in
solar flares, especially the enhanced acceleration of heavy ions and 3He
particles, and their contributions to large SEP events, are not well
understood. Moreover, the overall interaction between flares and CMEs, as well
as CME-CME interactions, in producing the most energetic SEPs remain a
mystery. In Earth’s and planetary magnetospheres, one notable attempt to
capture the physics of collisionless magnetic reconnection in the global
context is the ten-moment multi-fluid model that solves the continuity,
momentum, and pressure tensor equations for all ion and electron species, as
well as the full Maxwell equations (Wang et al., 2018; Dong et al., 2019; Wang
et al., 2020; see Fig. 2 for its application to Earth’s magnetosphere). In
addition, kinetic approaches such as hybrid (fluid electrons and kinetic ions)
models (Karimabadi et al., 2014; Le et al., 2021) and Vlasov-hybrid models
(von Alfthan et al., 2014) are promising candidates, especially by integrating
kinetic electron physics, for the next generation of global magnetosphere
simulations.

Figure 2: Perspective view of the Earth’s magnetosphere from a 3D ten-moment
multifluid simulation. The interplanetary magnetic field is southward. The
white streamlines are magnetic field lines in the meridional plane, indicating
the thinning of a current sheet on the night side, along with two flux ropes.
The blue-red contours in the $xz$-plane represent the $y$ component of the
magnetic field due to the Hall term in the model. The yellow-scale contours in
the $xy$-plane represent ion number density (Wang et al., 2020).

Attacking the problems outlined above with modeling advances, such as the
aforementioned examples, will enhance our physical insights and provide
significant theoretical inputs to space weather modeling and predictions.

## 4 The relation between theory and modeling efforts

Theory and modeling efforts are closely related but have important
differences. Theories are developed from fundamental principles to explain
poorly understood processes or phenomena, whereas modeling’s goal is to apply
theories to more complicated and realistic situations to compare with the
ground truth of observations. They cannot replace each other and both need to
be adequately supported, especially when basic theories are newly developed,
and not mature enough to directly compare with observations. For example,
basic studies of magnetic reconnection and associated particle acceleration
often use Harris-like current sheets and simplified boundary conditions. This
allows in-depth studies about basic processes and has been leading to
breakthroughs (e.g., Parker, 1957; Loureiro et al., 2007; Guo et al., 2014;
Liu et al., 2017; Cassak et al., 2017; Liu et al., 2022). At the same time,
one often needs to take into account realistic geometries such as applications
to solar eruptions, magnetospheric storms, and coronal heating, as they
require realistic 3D geometries and appropriate (e.g., line-tied) boundary
conditions. A similar situation applies to the theory and modeling of solar
eruptions. Our cartoons for the energy storage and release processes that
create CMEs are still debated in part because of the extraordinary complexity
of active regions observed on the sun. Similarly, the extent and kinematics of
CME shocks and ejecta may be strongly influenced by the instantaneous,
inhomogeneous 3D structure of the corona and inner heliosphere. Data-driven
and data-constrained modeling provides an avenue to bridge this gap (often
touted as the future of CME prediction and forecasting) but developing and
refining such approaches require dedicated investment with a parallel focus on
improving both the modeled physics (leveraging theory) and the mechanics of
the approach (e.g. proper boundary conditions, data assimilation,
incorporating subgrid physics, machine learning optimizations, etc).

Ultimately, it is important to recognize that theoretical understanding of
many heliophysical processes is immature and incomplete, and that theory and
model predictions often only address a selected set of scenarios and include
significant uncertainties. Achieving closure between theory and observations,
through modeling, is an iterative procedure that can take substantial effort
over years. Hence continuous progress in theory and modeling is still urgently
needed and federal agencies should protect against funding opportunities and
review panels from negatively biasing theory/modeling proposals that do not
incorporate observations.

## 5 Recommendations

  1. 1.

Restore the HTMS program to its original, inflation-adjusted level so that it
meets the critical needs of heliophysics science. More than 40 years ago, a
committee led by Stirling Colgate recommended that a dedicated program be
created to address large-scale, basic theory problems that required critical-
mass group efforts in order to make progress, which led to a theory program
now known as Heliophysics Theory, Modeling and Simulation (HTMS). The HTMS has
been a huge success and has been instrumental in transforming heliophysics
from a discovery-style, exploratory-science field, to a rigorous modern
discipline in which closure between models and observations has become the
norm. Many of the large-scale modeling groups that presently lead our field,
and many of the theory/modeling leaders in our field, owe their start to the
HTMS. The needs expressed in the Colgate report are even more important today,
because our observations and measurements have grown enormously in depth and
precision during the past decades, and require much more sophisticated
theory/modeling for advancing physical understanding. The funding for the HTMS
has remained constant throughout its history, however, so that now a typical
grant has only 1/3 the original buying power and is inadequate to support a
critical-mass effort. We recognize that other programs have been created to
tackle major unanswered questions in heliophysics, such as the NSF
ANSWERS/SWQU program, the NASA LWS TR&T, and the NASA DRIVE Centers, but none
is focused on attacking basic theory and modeling problems. Furthermore, such
programs support too few theory/modeling groups to make progress on the broad
range of basic theory challenges that are so important for advancing
heliophysics. Funding for the HTMS urgently needs to be increased by at least
a factor of three, so that critical-mass groups can once again be established
to attack our basic theory/modeling challenges. The original HTMS program had
a transformative impact on our field, elevating it to the next level as a
science discipline; a revitalized HTMS will have an equally transformative
impact.

  2. 2.

Begin a program of supporting model development for next-generation basic
research codes. Heliophysics has matured to the point that large-scale
numerical modeling is essential to every aspect of our science, from attacking
basic theory challenges to interpreting increasingly complex observations/data
and to improving space weather forecasting/nowcasting. The crucial role of
modeling to meet space weather objectives has long been recognized, and the
LWS Strategic Capability (SC) Program was created to respond directly to this
need. The SC Program has successfully provided many tools that the community
requires for advancing space weather science. No equivalent funding program
supports the development of basic research numerical tools, however, even
though the need is equally pressing. Given the lack of a dedicated program,
research codes are presently being developed haphazardly by cobbling together
funding from multiple sources, and are not generally available to the
community. Heliophysics urgently needs to develop next-generation numerical
codes to attack the problems of cross-scale and cross-domain coupling that are
at the heart of heliophysics science. Consequently, we recommend the creation
of a Strategic Research Model program, patterned after the LWS SC Program but
dedicated to funding the development of the next generation of basic-science
numerical codes. All models developed by this program will have the strict
requirement of being open-source and easily available to the whole
heliophysics community. As part of this Strategic Research Model program, the
developer groups will be supported to make the codes available through an
interface such as the CCMC or GitHub, and to maintain the codes if they are
sufficiently used by the community.

  3. 3.

Develop programs focused on addressing mission-critical theory and modeling
needs. As emphasized above, heliophysics is no longer a discovery-driven
field; developing and advancing new understanding now drives our science. Our
missions and instruments have achieved such a level of precision and
sophistication that theory and modeling are integral to their success.
Examples abound – in order to connect PSP solar-wind measurements to their
coronal origins, state-of-art models must be employed; in order to connect the
MMS particle distribution measurements to reconnection dynamics, theory/models
are essential; and in order to connect the exquisite DKIST photospheric
magnetic-field observations to solar activity, again theory/models are
necessary. This vital need of theory/modeling to derive science advances from
the observations will only increase with future missions. No current program
is focused on addressing this urgent need; consequently, we recommend the
development of a dedicated theory/modeling program to accompany all future
missions. These programs should begin roughly five years before mission launch
and be recompeted periodically, so that the theory and models will be in hand
when the data start arriving. Such mission-dedicated theory/modeling programs
would increase enormously the science return from future missions, and yet
have negligible cost compared to total mission funding.

  4. 4.

Develop joint theory/modeling programs with two or more basic research
Agencies: NSF, NASA, and DOE. Although instrument development and data
analysis are often highly specific to the particular Agency interests, such as
DOE laboratory experiments, NSF ground-based telescopes, or NASA space
missions, basic theory and modeling by its very nature cut across all
heliophysics science. For example, theories of magnetic reconnection and
particle acceleration are essential to the science goals of all Agencies.
Consequently, we recommend that joint-Agency programs be developed to address
common theory/modeling needs. Some of these joint programs could include the
revitalized HTMS and the Strategic Research Model Program recommended above.
Such joint programs would have the additional major benefit of maximizing the
effectiveness of Agency resources. For example, the DOE has uniquely powerful
resources in high-performance computing that could be made available to the
NASA community as well. Furthermore, the NSF has unique strengths in
educational programs, which would be of great value for building workforce
infrastructure for both NASA and the DOE (see #5). Inter-agency programs in
theory/modeling would be a win-win situation for all Agencies and for
heliophysics.

  5. 5.

Enhance programs for developing the next generation of theorists and modelers.
The vitality of a field is completely dependent on the strength of its
community infrastructure and especially on its upcoming generation of
researchers. New people bring in new ideas, and this rejuvenation is critical
for making major scientific progress. The dearth of university faculty who are
producing new researchers, particularly in basic theory and computation, is an
issue of grave concern in heliophysics. Programs are urgently needed that
address this issue, including joint programs as recommended above. A
successful example is the NSF faculty development in space sciences (FDSS)
program. The FDSS and similar programs are needed, but with increased
frequency of solicitations and increased emphasis on basic theory/modeling.
Furthermore, programs should be developed/enhanced for supporting Ph.D.
students who perform their thesis research at NASA and DOE Centers in close
collaboration with universities. It is essential that we develop a truly
unified, inclusive heliophysics community whose workforce infrastructure
integrates across science disciplines and institutions.

## 6 Summary

Theory and modeling need to be independent components of heliophysics science
and cannot be separated from integrated studies of heliophysics, or replaced
by data analysis and data science. Without theory and modeling, sophisticated
heliophysics understanding cannot be established, and empirical studies will
dominate. We recommend enhanced support that encourages the development and
growth of theoretical groups and young scientist communities, including but
not limited to the areas discussed above. All agencies should consider
expanding their current theory and modeling programs and creating new
heliophysics theory programs, to cover all four main areas of heliophysics. A
flagship theory and modeling program, such as a fully restored HTMS program,
is critically needed. Additional new programs need to be created to develop
next-generation basic research codes and address mission-critical theory and
modeling needs through collaborative and coordinated efforts of basic research
agencies such as NSF, NASA and DOE. We encourage the Decadal Survey Committee
to identify gaps in theory and modeling support, and suggest a strategy to
ensure the healthy development of theory and modeling efforts in heliophysics
research.

## References

  * Arnold et al. (2021) Arnold, H., Drake, J. F., Swisdak, M., et al. 2021, Phys. Rev. Lett., 126, 135101. doi:10.1103/PhysRevLett.126.135101
  * Bale et al. (2019) Bale, S. D., Badman, S. T., Bonnell, J. W., et al. 2019, Nature, 576, 237. doi:10.1038/s41586-019-1818-7
  * Blandford & Eichler (1987) Blandford, R. & Eichler, D. 1987, Phys. Rep., 154, 1. doi:10.1016/0370-1573(87)90134-7
  * Burch et al. (2016) Burch, J. L., Moore, T. E., Torbert, R. B. & Giles, B. L. 2016, Space Science Reviews volume 199, 5–21. doi:10.1007/s11214-015-0164-9
  * Cassak et al. (2017) Cassak, P. A., Liu, Y.-H., & Shay, M. A. 2017, Journal of Plasma Physics, 83, 715830501. doi:10.1017/S0022377817000666
  * Chen et al. (2020) Chen, B., Shen, C., Gary, D. E., et al. 2020, Nature Astronomy, 4, 1140. doi:10.1038/s41550-020-1147-7
  * Chen et al. (2022) Chen, B. et al. 2022, “Quantifying Solar Flare Energy Release”, White Paper submitted to the 2024 Solar and Space Physics Decadal Survey.
  * Cohen et al. (2022) Cohen, I. J. et al. 2022, “The case for studying other planetary magnetospheres and atmospheres in Heliophysics”, White Paper submitted to the 2024 Solar and Space Physics Decadal Survey.
  * Crary et al. (2022) Crary, F. et al. 2022, “The Magnetosphere of Jupiter: Moving from Discoveries Towards Understanding”, White Paper submitted to the 2024 Solar and Space Physics Decadal Survey.
  * Dahlin et al. (2017) Dahlin, J. T., Drake, J. F., & Swisdak, M. 2017, Physics of Plasmas, 24, 092110. doi:10.1063/1.4986211
  * Dong et al. (2019) Dong, C. et al. 2019, Geophys. Res. Lett., 46 (21), 11584-11596. doi:10.1029/2019GL083180
  * Drake et al. (2021) Drake, J. F., Agapitov, O., Swisdak, M., et al. 2021, A&A, 650, A2. doi:10.1051/0004-6361/202039432
  * Fleishman et al. (2020) Fleishman, G. D., Gary, D. E., Chen, B., et al. 2020, Science, 367, 278. doi:10.1126/science.aax6874
  * Garcia-Sage et al. (2022) Garcia-Sage, K. et al. 2022, Star-Exoplanet Interactions: An Emerging Interdisciplinary Field in Heliophysics, White Paper submitted to the 2024 Solar and Space Physics Decadal Survey.
  * Goodrich et al. (2022) Goodrich, C. et al. 2022, “The Persistent Mystery of Collisionless Shocks”, White Paper submitted to the 2024 Solar and Space Physics Decadal Survey.
  * Green et al. (2022) Green, J. L., Dong, C., Hesse, M., Young, C. A., Airapetian, V. 2022, Space Weather Observations, Modeling, and Alerts in Support of Human Exploration of Mars, White Paper submitted to the 2024 Solar and Space Physics Decadal Survey.
  * Guo et al. (2014) Guo, F., Li, H., Daughton, W., et al. 2014, Phys. Rev. Lett., 113, 155005. doi:10.1103/PhysRevLett.113.155005
  * Guo et al. (2021) Guo, F., Giacalone, J., & Zhao, L. 2021, Frontiers in Astronomy and Space Sciences, 8, 27. doi:10.3389/fspas.2021.644354
  * Haggerty et al. (2022) Haggerty, C. et al. 2022, “The Physics of Collisionless Dissipation in the Heliosphere”, White Paper submitted to the 2024 Solar and Space Physics Decadal Survey.
  * Ji et al. (2022) Ji, H. et al. 2022, “Major Scientific Challenges and Opportunities in Understanding Magnetic Reconnection and Related Explosive Phenomena in Heliophysics and Beyond”, White Paper submitted to the 2024 Solar and Space Physics Decadal Survey.
  * Karimabadi et al. (2014) Karimabadi, H., Roytershteyn, V., Vu, H. X., et al. 2014, Physics of Plasmas, 21, 062308. doi:10.1063/1.4882875
  * Kasper et al. (2019) Kasper, J. C., Bale, S. D., Belcher, J. W., et al. 2019, Nature, 576, 228. doi:10.1038/s41586-019-1813-z
  * Klimchuk et al. (2022) Klimchuk, J. 2022, Coronal Heating, White Paper submitted to the 2024 Solar and Space Physics Decadal Survey.
  * Kollmann et al. (2022) Kollmann, et al. 2022, Jupiter’s radiation belts as a target for NASA’s Heliophysics Division, White Paper submitted to the 2024 Solar and Space Physics Decadal Survey.
  * Le et al. (2021) Le, A., Winske, D., Stanier, A., et al. 2021, Journal of Geophysical Research (Space Physics), 126, e29125. doi:10.1029/2021JA029125
  * Li et al. (2019) Li, X., Guo, F., Li, H., et al. 2019, ApJ, 884, 118. doi:10.3847/1538-4357/ab4268
  * Li et al. (2021) Li, X., Guo, F., & Liu, Y.-H. 2021, Physics of Plasmas, 28, 052905. doi:10.1063/5.0047644
  * Li et al. (2022) Li, X., Guo, F., Chen, B., et al. 2022, ApJ, 932, 92. doi:10.3847/1538-4357/ac6efe
  * Liu et al. (2017) Liu, Y.-H., Hesse, M., Guo, F., et al. 2017, Phys. Rev. Lett., 118, 085101. doi:10.1103/PhysRevLett.118.085101
  * Liu et al. (2022) Liu, Y.-H., Cassak, P., Li, X., et al. 2022, Communications Physics, 5, 97. doi:10.1038/s42005-022-00854-x
  * Loureiro et al. (2007) Loureiro, N. F., Schekochihin, A. A., & Cowley, S. C. 2007, Physics of Plasmas, 14, 100703. doi:10.1063/1.2783986
  * Mondal et al. (2022) Mondal, S. et al. 2022, “Weak transients and the heating of the quiescent solar corona”, White Paper submitted to the 2024 Solar and Space Physics Decadal Survey.
  * Oka et al. (2022) Oka, M. et al. 2022, “Particle Acceleration in Solar Flares With Imaging-Spectroscopy in soft X-rays”, White Paper submitted to the 2024 Solar and Space Physics Decadal Survey
  * Parker (1957) Parker, E. N. 1957, J. Geophys. Res., 62, 509. doi:10.1029/JZ062i004p00509
  * Parker (1958) Parker, E. N. 1958, ApJ, 128, 664. doi:10.1086/146579
  * Parker (1999) Parker, E. N. 1999, ApJ, 525C, 792.
  * Reames (2020) Reames, D. V. 2020, Space Sci. Rev., 216, 20. doi:10.1007/s11214-020-0643-5
  * Ruffolo et al. (2020) Ruffolo, D., Matthaeus, W. H., Chhiber, R., et al. 2020, ApJ, 902, 94. doi:10.3847/1538-4357/abb594
  * Schwadron & McComas (2021) Schwadron, N. A. & McComas, D. J. 2021, ApJ, 909, 95. doi:10.3847/1538-4357/abd4e6
  * Shih et al. (2022) Shih, A. et al. 2022, High-energy Solar Physics, White Paper submitted to the 2024 Solar and Space Physics Decadal Survey.
  * Squire et al. (2020) Squire, J., Chandran, B. D. G., & Meyrand, R. 2020, ApJ, 891, L2. doi:10.3847/2041-8213/ab74e1
  * Sweet (1958) Sweet, P. A. 1958, Electromagnetic Phenomena in Cosmical Physics, 6, 123
  * Thorne (2010) Thorne, R. M. 2010, Geophys. Res. Lett., 37, L22107. doi:10.1029/2010GL044990
  * Turner et al. (2022) Turner, D. L. et al. 2022, Atmospheric and ionospheric impacts of energetic particle precipitation (EPP) from Earth’s ring current and radiation belts, White Paper submitted to the 2024 Solar and Space Physics Decadal Survey.
  * von Alfthan et al. (2014) von Alfthan, S., Pokhotelov, D., Kempf, Y., et al. 2014, Journal of Atmospheric and Solar-Terrestrial Physics, 120, 24. doi:10.1016/j.jastp.2014.08.012
  * Wang et al. (2018) Wang, L. et al. 2018, J. Geophys. Res. Space Phys., 123 (4), 2815-2830. doi:10.1002/2017JA024761
  * Wang et al. (2020) Wang, L. et al. 2020, J. Comput. Phys., 415, 109510. doi:10.1016/j.jcp.2020.109510
  * Zank et al. (2020) Zank, G. P., Nakanotani, M., Zhao, L.-L., et al. 2020, ApJ, 903, 1. doi:10.3847/1538-4357/abb828
  * Zhang et al. (2021) Zhang, Q., Guo, F., Daughton, W., et al. 2021, Phys. Rev. Lett., 127, 185101. doi:10.1103/PhysRevLett.127.185101