# Free-form Shape Modeling in XR: A Systematic Review

Shounak Chatterjee
Kolkata, India
<EMAIL_ADDRESS>
###### Abstract

Shape modeling research in Computer Graphics has been an active area for
decades. The ability to create and edit complex 3D shapes has been of key
importance in Computer-Aided Design, Animation, Architecture, and
Entertainment. With the growing popularity of Virtual and Augmented Reality,
new applications and tools have been developed for artistic content creation;
real-time interactive shape modeling has become increasingly important for a
continuum of virtual and augmented reality environments (eXtended Reality
(XR)). Shape modeling in XR opens new possibilities for intuitive design and
shape modeling in an accessible way. Artificial Intelligence (AI) approaches
generating shape information from text prompts are set to change how artists
create and edit 3D models. There has been a substantial body of research on
interactive 3D shape modeling. However, there is no recent extensive review of
the existing techniques and what AI shape generation means for shape modeling
in interactive XR environments. In this state-of-the-art paper, we fill this
research gap in the literature by surveying free-form shape modeling work in
XR, with a focus on sculpting and 3D sketching, the most intuitive forms of
free-form shape modeling. We classify and discuss these works across five
dimensions: contribution of the articles, domain setting, interaction tool,
auto-completion, and collaborative designing. The paper concludes by
discussing the disconnect between interactive 3D sculpting and sketching and
how this will likely evolve with the prevalence of AI shape-generation tools
in the future.

_K_ eywords Free-form Shape Modeling $\cdot$ Virtual Sculpting $\cdot$ 3D
sketching $\cdot$ Virtual Reality $\cdot$ Augmented Reality $\cdot$ Shape
Modeling $\cdot$ Graphics

##  1 Introduction

Extended reality, or XR, encompasses all immersive technologies, including
Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR). These
technologies extend reality by adding to or simulating the natural world
through digital materials, blending the virtual and ’real’ worlds, or creating
a fully immersive experience. In augmented reality, virtual objects and
information are overlaid in the real world. The user experiences this through
AR glasses or via screens such as a mobile phone or tablet. In virtual
reality, users can fully immerse themselves in a simulated digital environment
by using a VR headset or head-mounted display to get a 360-degree view of the
digital world. In mixed reality, digital and real-world objects co-exist and
can interact in real-time. This latest immersive technology is sometimes
referred to as hybrid reality.

Shape modeling is a significant aspect of computer graphics, with a rich
variety of modeling approaches including 3D shape reconstruction [1, 2, 3],
where a user creates or, more precisely, constructs a 3D model from one or
multiple RGB or depth images, and statistical shape modeling [4], where the
generated model is semantically similar to a pre-existing reference. This
survey paper focuses on free-form shape modeling, where the user creates and
carves a shape as an artistic process. In addition to this, we also analyze
the evolution of free-form shape modeling over the past 15 years, particularly
through the widespread adoption of virtual reality and augmented reality
interaction mechanisms. Devices such as Leap
Motion111https://www.ultraleap.com, Kinect222https://azure.microsoft.com/en-
us/products/kinect-dk, HTC Vive333https://www.vive.com, Oculus
Rift444https://www.oculus.com, and Hololens555https://www.microsoft.com/en-
us/hololens have greatly impacted free-from modeling, especially in virtual
sculpting and 3D sketching.

Previously, models were primarily visualized and interacted with on desktop
screens or holographic 2D screens [5]. Users could only see a 2D projection of
3D objects and had to rotate the scene to interact with it. Today, mixed
reality displays provide designers and artists with a full 3D view and an
immersive experience for designing and reviewing 3D shapes intuitively from
any viewpoint. Rendering algorithms have also improved over the years, and new
visualization techniques enable better modeling by enabling visual insights.
In the scope of this survey, we also cover several visualization techniques
related to shape modeling.

In this survey paper we give a comprehensive review of the state of the field
of free-form shape modeling. Since our research is focused on free-form shape
modeling, we discuss virtual sculpting and 3D sketching as our because these
two types of modeling approaches provide users with the freedom to intuitively
create models from as an artistic process.

###  1.1 Overview

Although numerous surveys [6, 7, 8] exist on sketch-based techniques, these
focus primarily on elucidating various aspects of both 2D and 3D sketching. We
explain these techniques in the following subsection. In our review, we expand
on these prior works by also incorporating sculpting techniques to discuss the
subject more comprehensively.

####  1.1.1 Related Work and Scope

In 2009, Olsen et al.[6] published a valuable survey on sketch-based modeling
systems. Their work included a systematic categorization, hinging on the
interpretation of sketches. Later, in 2016, Ding et al.[7] offered their
review of sketch-based modeling systems. The authors provided a thorough
exposition of different sketch-based modeling methods, including but not
limited to 2D sketches, 3D curve sketches, and surface drawings. They
classified sketch-based modeling systems based on factors such as input,
output, and the modeling approach utilized. More recently, in 2021, Wang et
al. [9] conducted a survey on AR and VR remote collaboration. Their work
synthesized information on remote collaboration in design, shape
reconstruction, and digital sculpting within AR and VR environments. In the
following year, 2022, Xu et al. [8] published their research on the state of
the art in free-hand sketching within the domain of deep learning. Their study
explored the characteristics, challenges, and opportunities related to working
with free-hand sketch data. In summary, our survey is positioned to provide a
combined view, weaving together the insights from the aforementioned research
to deliver a comprehensive understanding of both sculpting and 3D sketching
techniques.

####  1.1.2 Contributions

Free-form shape modeling which enables users to model any shape by sculpting
or sketching. We examine virtual sculpting and 3D sketching in detail and
their development in the last 16 years. We followed the guidelines proposed by
PRISMA methodology [10]666https://www.prisma-statement.org/ and Kitchenham
(2004), and we conducted a systematic literature review. In particular, (a) we
analyze how shape modeling methods (section 3) and their user interaction
techniques (section 4) evolved in the modern day. (b) we provide a detailed
taxonomy of both modeling techniques (section 3), their working environments,
and the tools used for interaction (section 4), and (c) we also discuss
collaborative shape modeling (section 6) and how artificial intelligence
(sections 5) is being used to model and refine a shape.

####  1.1.3 Structure of the review

The review is organized as follows:

  * •

Section 2 addresses our methodology that we followed in the manuscript. This
section states our research questions. Following that, it gives a rundown of
our literature selection process. After submitting our search queries, as
described in section 2, we received multiple results that needed to be
filtered out since not all of them were pertinent to our study. The selection
process is explained in subsection 2.1.

  * •

In section 3, we explain the methods of free-form shape modeling. We describe
various methods used for sculpting and sketching and how they evolved over
time.

  * •

In section 4 we describe the different interaction tools. We grouped these
into three subsections: gesture-based tool, pen-based tool, and 3D controller.
We explain their use and user preference for each type of tool for different
sculpting methods.

  * •

In sections 5 and 6, we elucidated the contemporary practices of free-form
shape modeling, which encompass collaboration among multiple users and the
utilization of artificial intelligence for refining and editing the model. The
survey reaches its conclusion in the section 8.

##  2 Methodology

Our study provides a systematic literature review of research on free-form
shape modeling in mixed reality. The research questions of this study are:

RQ1: What are the most preferred modeling techniques in extended reality?

RQ2: What methodologies/theories are being used to research 3D shape modeling
in extended reality?

RQ3: What are the research gaps in shape modeling in extended reality?

RQ4: Which hardware has been preferred for different modeling techniques?

To answer the research questions outlined above, we searched relevant research
articles published since 2007 in the following databases and digital
libraries: IEEE Xplore, ACM Digital Library, and SpringerLink. We formulated
multiple search strings and used primary terms such as ’virtual sculpting,’
’3D sketching,’ ’free-form shape modeling,’ ’mixed reality,’ ’augmented
reality,’ and ’virtual reality.’ We also considered alternative spellings and
other search terminologies, including related areas, and reviewed the
references in the articles. Our search process covered the time frame between
October 2021 and October 2023.

Record identified from databases (n = 431) Records removed before screening:
a) Duplicate Records (n=148) b) Records before 2007 (n=59) Records screened
(n=224) Records excluded (n=57) Records sought for retrieval (n=167) Records
not retrieved (n=9) Records assessed for eligibility (n=158) Records excluded:
Records not related to the scope (n=40) Studied included for review (n=118)
Identification Screening Included Identification of studies via databases and
registers Figure 1: PRISMA flow diagram for systematic records selection.
<200720072008200920102011201220132014201520162017201820192020202120222023$0$$5$$10$$8$$2$$1$$0$$1$$3$$1$$5$$0$$1$$2$$2$$4$$1$$2$$5$$5$$1$$10$$3$$5$$3$$3$$3$$3$$2$$1$$2$$5$$6$$5$$3$$4$$7$$1$$3$YearsPapers
publishedSculpting3D Sketching
---
Figure 2: Number of papers published over the years on Sculpting and 3D
sketching. Figure 3: Number of publications according to the publication venue
in our survey duration.

###  2.1 Selection Process

After performing the search queries, we reviewed the publications and applied
inclusion and exclusion on the following criteria. We included conference
proceedings and journal articles published from 2007 onward and assessed the
relevance of each paper based on its title and abstract. We only considered
papers related to free-form shape modeling in extended reality environments
(mixed, virtual, or augmented) or 2D screens (desktop, drawing tablets, etc.).

Further, we critically appraised the research papers and grouped these based
on the type of modeling techniques, the types of environments where the
modeling was performed, and the interaction tools used for modeling. Out of
431 papers retrieved using our provided keywords, 224 articles passed the
first stage, and 167 contained relevant studies. Finally, we selected 109
papers as primary sources for our review. In addition to these papers, we also
included some studies (around 15% of the total papers) published before 2007
that provided an underpinning foundational explanation behind the current
studies. In Figures 2 and 3 we have shown the statistics on the research
selected for our review, and in Figure 1 we have shown the article selection
process in a flow diagram. In Table 1 and 2 we have listed a few selected
papers from our study based on their popularity, i.e. the number of citations
and publishing venue, and impact, i.e., how their method is used in the future
research work repeatedly.

###  2.2 Types of literature contribution

In the first part of our study, we conducted a literature review based on the
shape modeling techniques. Free-from shape modeling can be categorized into
two primary techniques: sculpting and sketching. In sculpting, a user starts
from a primitive shape and carves out a desired shape. In contrast, sketching
involves starting from scratch to create a set of surface patches from 3D
strokes, which eventually lead to a shape. Irrespective of the specific
modeling techniques employed, the majority of the reviewed papers focused on
either introducing novel algorithms for shape representation and modeling,
proposing innovative interaction methods, or occasionally even both. In this
review, we have grouped and described each modeling technique according to its
free-form modeling type, mentioned in section 3. Additionally, we discussed
the involvement of the interactive tools, which we classified into three
categories and explained their use and preference for particular modeling
techniques in section 4.

Table 1: Categorization of different sculpting techniques published. These papers are sorted year-wise. Methods | Sculpting Method | Environment |  | Interaction Tool
---|---|---|---|---
Desktop | AR | VR |  | Hand-Gesture | Controller | Stylus
Sculpting [11] | Virtual Clay |  |  |  |  |  |  |
NURBS [12] | Surface Modeling |  |  |  |  |  |  |
Virtual 3D Sculpting [13] | Surface Modeling |  |  |  |  |  |  |
Surface Drawing [14] | Surface Modeling |  |  |  |  |  |  |
Virtual Clay [15] | Virtual Clay |  |  |  |  |  |  |
6DOF [16] | Surface Modeling |  |  |  |  |  |  |
V3DSPP [17] | Physical Proxy |  |  |  |  |  |  |
ARPottery [18] | Virtual Clay |  |  |  |  |  |  |
Hands on Virtual Clay [19] | Virtual Clay |  |  |  |  |  |  |
AFSC3DM [20] | Physical Proxy |  |  |  |  |  |  |
deForm [21] | Physical Proxy |  |  |  |  |  |  |
FreeD [22] | Physical Proxy |  |  |  |  |  |  |
SMNs [23] | Multi-layer |  |  |  |  |  |  |
MEMs [24] | Virtual Clay |  |  |  |  |  |  |
CSculpt [25] | AI Sculpting |  |  |  |  |  |  |
SLayer [26] | Multi-layer |  |  |  |  |  |  |
Auto-complete 3D Sculpting [27] | AI Sculpting |  |  |  |  |  |  |
DigiClay [28] | Virtual Clay |  |  |  |  |  |  |
RealPot [29] | Virtual Clay |  |  |  |  |  |  |
Mesh R-CNN [30] | AI Sculpting |  |  |  |  |  |  |
DeepSDF [31] | AI Sculpting |  |  |  |  |  |  |
DARLAT [32] | Physical Proxy |  |  |  |  |  |  |
PotteryVR [33] | Virtual Clay |  |  |  |  |  |  |
DreamFusion [34] | AI Sculpting |  |  |  |  |  |  |
NerfShop [35] | AI Sculpting |  |  |  |  |  |  |

##  3 Free-form Shape modeling

Research in computer graphics has produced various essential and powerful
techniques for image and shape synthesis. While shape modeling and processing
techniques are well understood, user-interface methods for creating and
manipulating computer models of 3D shapes are still relatively difficult in
the desktop environment than modeling in mixed reality space [36, 37].
Consequently, 3D shape modeling is a very laborious procedure that requires
considerable human time and effort in a 2D visualization. In our literature
review, we found that previous research indicates that users typically prefer
to perform modeling in a 3D space where they have enough freedom to make use
of the extra dimension. This extra dimension is helpful for modeling purposes
and is also an excellent way to demonstrate and review the model with others
[38, 39].

###  3.1 Sculpting

Sculpting is one of the key free-form shape modeling techniques. Here, a user
starts with a pre-existing basic shape of a 3D object. The user can perform
various operations such as extruding, indenting, and carving on the surface of
an object to give a shape of choice. Usually, users shape the object by hand
or with a special tool. Certain sculpting literature delves into various
aspects, such as algorithms, elucidating different approaches to representing
shapes. Another facet explored in the literature involves exploring
interaction tools and techniques, including traditional and unconventional
instruments employed to engage with the objects, sometimes within alternative
realities or environments. In our study, we found different types of sculpting
techniques that are explained below. In Table 1, we have chronologically
listed some standout studies from our review based on their impact metric
explained earlier.

Digital or virtual sculpting was introduced back in 1991 by Galyean et al.
[11]. The authors represented the primitive shape, i.e., the shape from which
the user started sculpting, in a mesh data structure and recorded the changes
made by the interaction tool (3D controller) in a voxel grid space. The voxel
data is converted to a mesh using the Marching Cubes algorithm [40]. In
contrast to this, Wong et al. [13] proposed a method to directly control the
shape using a control hand surface to fit open uniform B-spline surfaces.
Since the authors used an electronic glove as their input device, they took
hand points (joints and fingertips) to create a control surface. This control
surface directed the sculpting operations. A set of parametric surfaces
represented a shape, and it was updated by interpolating the changes in the
hand points received from the user.

####  3.1.1 Virtual Clay

McDonnell et al. [15] had a different approach to sculpting, where the
starting object was represented as a virtual 3D elastic clay. Virtual clay is
a digital sculpting technique that mimics the experience of working with
physical clay. Through this method, users can shape virtual objects using
tools like a virtual sculpting brush or input devices such as a 3D mouse or a
stylus. The authors propose a dynamic subdivision solid modeling approach,
where the object’s surface is dynamically divided into smaller polygons. This
allows users to sculpt specific portions of the object, resulting in localized
changes rather than altering the entire shape. In the sculpting workspace, the
virtual clay reacts to the user’s direct application of forces in an intuitive
and predictable way, and the haptic feedback can greatly improve the sense of
realism. Many researchers further adopted this virtual clay system [18, 19,
28, 29, 41, 33] to account for different environments and tools. Han et al.
[18] designed an AR-based pottery system based on the virtual clay system. The
users can make the pottery intuitive and use one or both hands to perform
different interaction functions like poke, push, pull, bend, stretch, etc.
Pihuit et al. [19] proposed a system with a passive tactile feedback device to
shape the virtual clay with hands. It allows users to apply finger pressure on
the pressure sensors, and it returns tactile feedback and realistic visual
behavior of the hand to provide a better immersion for the user. Milliez et
al. [24] and Gao et al. [28] proposed a similar virtual clay representation.
In the former, authors proposed to add virtual elastic clay models to create a
final shape instead of using a surface. The goal was to add separate models
and blend them. They incorporated automatic local adaptation into the
sculpting operation to accommodate model updates and deformations. The latter
used real-time mesh deformation of the virtual clay using hands to provide
users with an effortless and refined interface for art creation. Gao et al.
[29] proposed a virtual pottery system that is compatible with VR. They used a
cylinder-shaped clay mesh as their starting point and gave it a shape by
deforming the mesh and changing the radius at different heights. Users have
the option to update the height, thickness, and non-uniform radius. In Figure
4, we have shown different sculpting techniques in VR environment adapted from
[28, 29, 33].

(a) DigiClay [28]

(b) RealPot [29]

(c) PotteryVR [33]

Figure 4: Different techniques for VR sculpting. (a) Sculpting with hand
gestures; (b) & (c) sculpting the model using VR controllers.

Later, Dashti et al. [33] presented a scalable virtual pottery approach named
PotteryVR, which can create more realistic forms in the virtual world. Their
system used commercially available VR equipment, which is more accessible to
the users and it can accommodate multiple users. The authors also discussed
the evaluation methods for VR, focusing on visual feedback, gestural
interfaces, 3D modeling, and collaborative design. Their study aims to develop
modeling and practice skills, provide guidelines on designing 3D models, and
use of visual pottery for deformable shape modeling and rapid prototyping.
Barrerio et al. [41] presented another virtual clay modeling system with
natural tactile interaction. Their solution involved utilizing a clay
simulation method and an ultrasound-based rendering algorithm. This system
effectively handles extreme viscoplasticity and offers superior interaction
capabilities compared to previous methods. However, it has limitations,
including the high computational cost of the elastoplasticity constraint,
restricted rendering method coverage, and limited detailed finger modeling
dexterity due to low particle resolution and limited tactile stimulation( see
Figure 4).

####  3.1.2 Surface Modeling

In addition to the virtual clay system, another prevalent method involves
users modifying a hollow object’s surface. Lamousin et al. [12] proposed a
surface modeling method utilizing non-uniform rational B-splines (NURBS) to
modify curves and surfaces in 3D modeling. Unlike virtual clay, the NURBS-
based surface modeling method only modifies the object’s surface. In this
modeling approach, the objects utilized are typically represented using
implicit curves or surfaces [12]. Sometimes, the implicit surfaces are
discretized into mesh data structures [16]; sometimes, the curves are
tessellated with mesh as well. A set of control points defines the shape of
the surface, and a set of weights controls the influence of each control point
on the shape. Research by Liverani et al. [16] and Schkolne et al. [14] used
NURBS in different environments and interaction tools.

####  3.1.3 Physical Proxy

Sheng et al. [17] also showed mesh manipulation in their work by using NURBS
with a physical prop as a proxy to the virtual model for better control of the
interaction techniques. Later, researchers adopted the proxy method in various
ways. Marner et al. [20] proposed to physically sculpt a 3D model from foam
using traditional tools. In that process, the system digitally duplicates the
process by tracking the tools and the foam. A similar method was adopted by
Zoran et al. [22] with a better foam sculptor. There are many similar methods
for converting physical sculpting to virtual models [42, 43]. Follmer et al.
[21] and Natakuaithung et el. [32] proposed a similar approach but used
modeling clay instead of foam.

####  3.1.4 Multi-layer Sculpting

All the sculpting methods explained before had only one object, in either
virtual clay or mesh data structure. However, a model can be constructed with
multiple sub-models [23, 24], or it can be sculpted with multiple layered
materials [26]. Schmidt et al. [44] proposed the idea of fusing two meshes by
drag-and-drop methods. Their approach was to add another mesh by filling the
holes of a prior mesh. Stanculescun et al. [23] presented a systematic
approach to representing and deforming shapes with nested features to handle
free-form deformations. They used a hierarchical rule-based approach with
separate attributes for shape and geometry to adjust deformations in a
sculpting system for mesh features like regions, curves, and points. Milliez
et al. [24] introduced a deformation model for sculpting shapes with local
adjustments and geometry changes. This approach made the model more flexible
than traditional methods. The smart-clay application took advantage of this
flexibility, adjusting shapes through methods like stretching, merging, and
splitting in response to desired changes.

###  3.2 3D Sketching

3D sketching is another type of popular free-form shape modeling. To create a
surface or an object, the user scribbles some strokes in space, and those
become a collection of curves or surfaces. The user either starts from scratch
or adds something on top of a pre-existing shape without deforming it. Similar
to sculpting, different sketching techniques exist, especially depending on
the interactive tools. In Table 2, we have listed some of this research on
sketching, chronologically based on their popularity, i.e. the number of
citations and publishing venue, and impact, i.e., how their method is used in
the future research as we mentioned previously.

During the 1990s and early 2000s, 3D sketching was performed to create 3D
geometric shapes, which were manipulated using editing tools. The overall
process resembled sculpting, with initial ideas borrowed from sculpting
techniques [45, 5, 13, 11]. Deering et al. [45] introduced a 2D sketch
paradigm that could be expanded to 3D in VR, supporting complex animated 3D
scenes. However, at that time, drawing a 3D curve required users to examine
the projection of the curve from different viewpoints on a monitor [46]. Cohen
et al. [46] proposed a novel method for specifying 3D curves using 2D input
from a single viewpoint, utilizing the shadow of the curve on the floor.
Additionally, Igarashi et al. [47] conducted research on creating smooth 3D
shapes from 2D free-form sketches.

####  3.2.1 Generating 3D shapes from 2D sketches

There have been numerous research studies following the work of Igarashi et
al. [47] that utilized techniques to generate 3D shapes from 2D sketches,
representing the resulting shapes meshes [47, 48, 49, 50, 51, 52, 53], voxels
[54, 55], and parametric surfaces [56, 57, 58].

The research conducted by Igarashi et al.[47] proposed a system that takes a
2D sketch as input and inflates it into a 3D object. Additional details can be
added to the object by scribbling on it. Wang et al.[48] proposed modeling
complex 3D shapes by inflating 2D sketches, using layered meshes to combine
primitive shapes into a complex 3D structure. Gingold et al. [59] developed a
similar system for generating 3D models from 2D sketches, but with the
requirement for users to annotate and position the primitive shapes within the
sketch to provide semantic information, aiming to improve representation.
Their goal was to eliminate the constant need for rotating the sketch/model
during the sketching and modification process. Liu et al.[57] proposed a
method for constructing 3D free-form surface patches from 2D line drawings.
Their approach included techniques for curve interpolation from the boundary,
reconstruction of 3D curves from their 2D projections, and reconstruction of
surfaces from their backbones, which are curves inside the input boundary
sketch. Chen et al.[56] introduced a process for converting sketches into a
realistic 2.5D interpretation of an architectural design sketch with editing
options.

Ramos et al. [50] proposed a sketch-based modeling system that uses
convolution surfaces. The user draws their conceptual ideas in 2D planes
embedded in 3D space. The 2D drawing is then projected onto 3D surfaces, and
the depth in space is computed via ray-casting. Regarding 2D sketches, Xu et
al.[52] used 2D vector sketches to generate 3D curves and employed mesh
structures to generate a closed object. Guillard et al.[51] and Zhang et al.
[55] used 2D sketches and utilized neural networks to generate 3D objects in
mesh and voxel structures, respectively.

Table 2: Categorization of different sketching techniques published. These
papers are sorted year-wise.

Method | Sketching Technique | Envirionment |  | Interaction Tool
---|---|---|---|---
Desktop | AR | VR |  | Hand-Gesture | Controller | Stylus
An interface for sketching 3d curves [46] | 3D Curve Sketching |  |  |  |  |  |  |
CavePainting [5] | 3D Curve Sketching |  |  |  |  |  |  |
FreeDrawer [60] | 3D Curve Sketching |  |  |  |  |  |  |
Teddy [47] | 2D-to-3D Sketching |  |  |  |  |  |  |
Free-form Sketch [48] | 2D-to-3D Sketching |  |  |  |  |  |  |
Drawing in Air [61] | 3D Curve Sketching |  |  |  |  |  |  |
FiberMesh [62] | 3D Sketch-to-Shape |  |  |  |  |  |  |
Sketch3D [63] | 3D Curve Sketching |  |  |  |  |  |  |
ILoveSketch [64] | 3D Curve Sketching |  |  |  |  |  |  |
Sketching Reality [56] | 2D-to-3D Sketching |  |  |  |  |  |  |
Napkin sketch[65] | 3D Curve Sketching |  |  |  |  |  |  |
Structured annotations for 2D-to-3D modeling [59] | 2D-to-3D Sketching |  |  |  |  |  |  |
Analytic drawing of 3D scaffolds [44] | 3D Curve Sketching |  |  |  |  |  |  |
EverybodyLovesSketch [66] | 3D Curve Sketching |  |  |  |  |  |  |
3DCurveSketch [67] | 3D Curve Sketching |  |  |  |  |  |  |
OverCoat [68] | 3D Curve Sketching |  |  |  |  |  |  |
Single-view sketch based modeling [69] | 3D Sketch-to-Shape |  |  |  |  |  |  |
JustDrawIt [70] | 3D Curve Sketching |  |  |  |  |  |  |
Sketch-based shape retrieval [49] | 2D-to-3D Sketching |  |  |  |  |  |  |
SecondSkin [71] | 3D Curve Sketching |  |  |  |  |  |  |
3D Sketch-Based 3D Model Retrieval [72] | 3D Sketch-to-Shape |  |  |  |  |  |  |
SmartCanvas [73] | Model Guided Sketching |  |  |  |  |  |  |
Lift-Off [39] | Model Guided Sketching |  |  |  |  |  |  |
3D Shape Reconstruction from Sketches via MVCN [74] | 3D Sketch-to-Shape |  |  |  |  |  |  |
WireDraw [75] | Model Guided Sketching |  |  |  |  |  |  |
DreamSketch [54] | 2D-to-3D Sketching |  |  |  |  |  |  |
SweepCanvas [76] | 3D Curve Sketching |  |  |  |  |  |  |
EESSVR [37] | 3D Curve Sketching |  |  |  |  |  |  |
SymbosisSketch [77] | 3D Curve Sketching |  |  |  |  |  |  |
Model-Guided 3D Sketching [78] | Model Guided Sketching |  |  |  |  |  |  |
VRSketchPen [79] | 3D Curve Sketching |  |  |  |  |  |  |
VRSketchIn [80] | 3D Curve Sketching |  |  |  |  |  |  |
HandPainter [81] | 3D Curve Sketching |  |  |  |  |  |  |
Sketch2Mesh [51] | 2D-to-3D Sketching |  |  |  |  |  |  |
Sketch2Model [55] | 2D-to-3D Sketching |  |  |  |  |  |  |
CASSIE [82] | 3D Curve Sketching |  |  |  |  |  |  |
ScaffoldSketch [83] | 3D Curve Sketching |  |  |  |  |  |  |
Mid-Air Drawing of Curves on 3D Surfaces in VR [84] | 3D Curve Sketching |  |  |  |  |  |  |
PSFU3DS [85] | 3D Curve Sketching |  |  |  |  |  |  |
SKED [86] | AI Sketching |  |  |  |  |  |  |

####  3.2.2 3D curve Sketching

In this subsequent subsection, we explain different types of 3D sketching.
During our study on 3D sketching, we observed that these methods often involve
creating 3D sketches and sometimes enhancing the curves with virtual surfaces
or generating surfaces within a 3D space. Moreover, there are instances where
models are retrieved based on the 3D sketches provided by the users. We have
categorized the methods according to these specific techniques. There are a
variety of 3D curve sketching techniques. In some methods, the user designs an
object using 3D curves or surfaces, which may subsequently be tessellated with
meshes [14, 64, 82, 65, 81]. In other technique, users construct a scene on a
3D canvas employing both curves and surfaces [5, 76, 77, 80]. In Figure 5, we
compare some of the sketching methods in different environments adopted from
the methods [64, 65, 81].

In the year 1999, Cohen et al. [46] described a process to draw 3D curves
using a single 2D viewpoint. After that, we found several research works on 3D
sketching. As an early research, Wesche et al. [60] proposed a 3D sketching
workbench consisting of 3D tools for curve sketching and surface drawing. The
system was designed in a virtual environment and uses a stylus as the input
method. Another remarkable system was designed by Schkolne et al. [14], where
users can design 3D shapes from the surface marked by their hands in a 3D
virtual space. Keefe et al. [61] introduced haptic-aided drawing of 3D curves
in a virtual space. They demonstrated that for drawing in 3D space, haptic
feedback is necessary for precise control and accuracy. Their user study
showed that users of their system were able to draw complex shapes. Bae et al.
[64, 66] proposed a sketching technique on a 2D screen, i.e., drawing tablet.
Their system was in a tablet-stylus setting, so it could replicate the pen-
paper feeling to the professional designers. The authors maintain the accuracy
of the 3D curves by projecting objects in multiple 2D viewpoints. They have
incorporated on-screen gestures for fundamental operations. The system was
easily mastered and garnered positive feedback from professional users. A
similar method was adopted by Ma et al. [87] and Xin et al. [65] in their
research.

Another approach to 3D sketching/modeling allows users to draw a 3D sketch
using curves. Subsequently, closed areas are either immediately populated or
filled after a certain period with a surface mesh. Grimm et al.[70] pioneered
this domain, enabling users to transition from a 2D sketch to a 3D curve
sketch, followed by patch filling. However, the conversion from 2D to 3D
sketches occasionally lacks consistency. Researchers have gravitated towards
direct 3D sketching in VR environments as a solution. Proposals for systems
specifically designed for VR head-mounted displays (HMDs) have been put forth
by De Paoli et al.[71], Stanko et al.[88], and Yu et al.[82, 85].
Additionally, a variant of 3D curve sketching exists where an elementary
structure is laid out using straight lines, and then 3D curves are
incorporated to articulate the desired shape. Such methodologies have been
suggested by Schmidt et al.[44], and Yu et al.[83].

Research into drawing within a VR environment experienced a resurgence
following the introduction of VR HMD hardware,Arora et al.[37] developed a
sketching system in VR utilizing the HTC Vive, which allowed artists and
designers to design freely in an unconfined space. However, this rudimentary
system confronted challenges related to the absence of a physical surface,
orientation, and planarity. To facilitate sketching, the researchers employed
controllers and 3D pens. To address the concern of the missing physical
surface, Elasyed et al.[79] incorporated a haptic device within the stylus,
which alerts users when they are interacting with the virtual surface. To
tackle the planarity issue, Arora et al.[84] introduced an approach that
enabled drawing on any given surface. Meanwhile, Jiang et al.[81] presented a
sketching technique using a VR tracker to monitor hand positions within the VR
environment. They also employed a stylus integrated with optical trackers to
enhance positional accuracy.

(a) ILoveSketch [64]

(b) Napkin Sketch [65]

(c) HandPainter [81]

Figure 5: Different techniques of 3D sketching. (a) Sketching on top of a
drawing tablet screen; (b) Sketching in AR on a designated area; (c) Drawing
in VR.

Users interested in sketching sometimes opt for a virtual 3D canvas. Keefe et
al.[5] presented a method that allows users to craft 3D art utilizing 2D brush
strokes. Meanwhile, Zeleznik et al.[89] designed a 3D interface equipped with
functionalities such as panning, dragging, rotating, and focusing. Schmid et
al.[68] and Li et al.[76] advanced similar systems to construct 3D scenes from
RGB-D images that are further refined with sketches. Additionally, Arora et
al.[77], and Drey et al.[80] incorporated virtual reality into these systems
in their respective studies.

####  3.2.3 Generating 3D shapes from 3D sketches

Another approach to 3D modeling exists, which allows users to draw a sketch in
3D and then generate a corresponding 3D model. We have previously delved into
the work titled ‘Teddy’ by Igarashi et al. [47]. In that method, 2D sketches
are extrapolated to form 3D meshes. Another notable instance is FiberMesh
[62], presented by Nealen et al., a system for designing free-form surfaces
with a collection of 3D curves. In this system, users initiate a rudimentary
3D model and then add 3D curves on the model for modification. Andre et al.
[69] proposed an innovative system capable of producing complex objects from
3D strokes drawn from a unique viewpoint. Their methodology deconstructs each
object into smaller primitives defined by construction lines for object
generation. Some researchers have also worked on 3D model retrieval using
convolutional neural networks from 3D sketches. These research works focus on
retrieving 3D models that match the input sketches from the user. Depending on
the method, they may return a single object [72, 58, 74], or multiple models
that align with the initial sketch [90, 91].

####  3.2.4 Model guided 3D sketching

Another variant of 3D sketch-based modeling involves the creation of an object
within a 3D space, with guidance from a predetermined shape. Tsang et al.[92]
introduced an interface where a user can sketch within a 3D space and use a 2D
image as a guide. The strokes input by users also triggered geometric
suggestions, reducing the requirement to draw all parts of the new model.
Similarly, Zheng et al.[73] developed a system tailored for architectural or
design-oriented sketches. Here, users render a 2D sketch derived from an image
of an architecture, after which the system converts the 2D sketch into an
intricate 3D model based on some primitive shapes. Jackson et al.[39] proposed
an analogous system specifically for transmuting 2D sketches into 3D models
within a VR environment. In a distinct approach, Yue et al.[75] introduced a
technique for crafting a 3D sketch utilizing a specialized 3D extruder pen,
where users receive guidance from a virtual model projected through a VR
headset. Xu et al. [78] presented a novel interface specifically for 3D model-
guided sketching on 3D planes. This methodology strategically leverages the
geometry of an underlying 3D model to infer 3D planes on which 2D strokes were
drawn and around the mentioned 3D model.

##  4 Interaction Tools

For free-form modeling, the interaction tools take a significant portion of
the user experience. Our study of the literature reveals that various modeling
techniques highlight the impact of different interaction tools and methods on
user attention and expertise. We also discovered that certain devices are
preferred for sculpting while others are favored for sketching. Additionally,
there are tools such as pen-based tools, which are considered standard for
both modeling techniques, like pens for sketching. Furthermore, we came across
mentions of unique interactive tools such as 3D positional trackers, custom
stylus, etc. In this section, we will explore these different interaction
tools and examine how they provide users with advantages in free-form shape
modeling compared to other devices.

###  4.1 Gestures-based tools

Gestures, especially hand gestures, are among the most common forms of
interaction. For sculpting as they provide the users a natural experience to
interact with the shapes or objects [93, 15, 94, 28]. There are also
approaches for using hand gestures as the interactive medium for drawing [14,
95, 96, 97, 98, 5]. Nowadays, depth cameras such as Kinect and Leap Motion can
capture hand gestures accurately. However, back in 2004, Tosas et al. [99]
proposed a method that employed a camera, a black screen, and a virtual grid
to capture various gestures. It was not competent enough to capture moving
hand gestures, but many other researchers proposed methods based on this for
capturing hand gestures later [100, 101].

McDonnell et al. [15] proposed a modeling framework and haptic feedback based
on physics-based techniques. They used virtual clay as their sculpting
interface, with several tools like finger tracking and 3D pens for control.
Similar principles were applied by Kim et al. [93] and Gao et al. [28] where
they both used hand gestures in the air to sculpt objects virtually. Although
their methods were similar but, hand gesture recognition differed. Kim et al.
used pictures of hands taken similarly to Tosas et al. [99], and Gao et al.
used Kinect to recognize the gestures. Wesson et al. [94] proposed a sculpting
and evaluation technique using Kinect to mimic the interaction of physical
sculpting. Pihuit at al. [19] proposed a different method where, instead of
capturing hand gestures, they used a pressure-sensitive device to mimic the
hand pressure for sculpting. In this way, users were also getting haptic
feedback on their input. Later, Jacobs et al. [102] used a soft hand model for
virtual manipulation.

For 3D sketching, hand gestures are not always a preferred tool, as they lack
haptic feedback and may not produce accurate sketches without augmented
guidance. However, there are few papers [14, 96, 97, 36] on sketching with
hand gestures. Schkolne et al. [14] proposed a method for drawing surfaces
with several interaction tools, including hand gestures captured by an
electronic glove. Drawing surfaces may not require as much accuracy as drawing
curves or lines, making the hand a suitable choice, but, for some regions,
users may prefer specialized tools over hands [14]. Kim et al. [103] proposed
drawing a scaffold of an object where users do not have to be precise, using
the object as a guideline. There are some proposals [96, 97, 95] where hands,
especially finger motions, were used as the tool for drawing, where the user
uses fingers to draw in the air and has a guideline to lead the stroke. There
was also some research by Kwan et al. [36] and Dudley et al. [97] on 3D
sketching using hands in an AR environment.

###  4.2 Pen-based tools

Pen or stylus-based tools are another popular standard for interaction with
sketches and sculpting. This type of tool provides precise control to the
user. With pen-like tools, users can interact with the object accurately, as
it simulates the natural motion of sketching. In our study, we found that
different types of pen-based tools have been used in past research. In our
study, we learned that pen-based tools are mainly used for sketching and
surface drawing, but there exists a small body of research on sculpting with a
pen-based tool [11, 22].

The most common type of pen-based tool used for sketching is the stylus for
drawing pads. This method is widely used, even for 3D sketching, where users
sketch on a 2D tablet screen while working in 3D. Using a drawing tablet
provides users with a specific projection and haptic feedback. Keefe et al.
[5, 61], Bae et al. [64], and Xin et al. [65] have explained this technique in
their research, which demonstrates that users can represent and interact with
3D sketches on a 2D screen [66, 87]. Paczkowski et al. [104], Kazi et al.
[54], and Arora et al. [77] have extended this idea to a next level by adding
drawing in AR environment. In another method, Kim et al. [95] have proposed
that users start the sketch with an impression of the hand and then add
details using a pen.

Custom-made 3D stylus with 3D tracking sensors is another notable pen-based
tool that has gained prominence and is primarily designed for 3D sketching
purposes. Numerous articles [105, 39, 75, 106] delve into the functionalities
and applications of this tool. Its primary usage revolves around sketching and
drawing within VR environments. In some cases, the pen’s enclosure
incorporates multiple 3D tracking sensors, as proposed by Kamuro et al. [105]
and Jackson et al. [39]. Furthermore, Yue et al. [75] presented a compelling
proposal for a customized 3D extruder pen equipped with a 3D positioning
sensor. Additionally, the method introduced by Arora et al. [77] offers a 3D
sketching option, wherein the drawing tool operates based on a similar
conceptual framework. The continuous exploration and innovation in the field
of pen-based tools for three-dimensional artistic expression are reflected in
these advances.

###  4.3 3D Controller

In our study, we found that in some methods [107, 33, 29, 108], VR controllers
were used as an interaction tool. This specific type of shape modeling method
was designed for the VR environment so that the controllers gave native
support. There are also a few researchers who created a custom tracker for
sketching or sculpting. Eroglu et al. [109], Dong et al. [110], and Jiang et
al. [81] designed their custom controller based on the 3D positional trackers.
There were also methods proposed by Giunchi et al. [91] and Garcia et al.
[111] where they used the native VR controllers for interaction.

###  4.4 Other Tools

In our study, we found a few more interaction tools that are exclusive to the
aforementioned ones. There were some studies by Wang et al. [112], and Hagbi
et al. [113], where they designed sketching systems, which used a mouse as
their interaction tool. Although they used similar interaction tools, Wang et
al. [112] proposed a method to construct 3D shapes from 2D sketches in the
virtual world, whereas Hagbi et al. [113] worked on drawing in AR. Another
technique we found involves cutting [20] or sculpting [22] a piece of foam or
creating paper-crafts [43] to form a shape and fabricate it into a CAD model.

##  5 AI in modeling

With newer research, different AI techniques are now integrated into the space
of shape modeling. The AI modeling techniques we discovered in our research
can be categorized into suggestive modeling and AI-based shape generation. In
the former technique, the system suggests future edits, and in the latter, the
system generates a model based on specified constraints. Both approaches offer
unique advantages and potential applications in various domains.

###  5.1 Suggestive Modeling

It is widely acknowledged that creating drawings or sculptures in the air with
gestures without haptic feedback is challenging, as users do not experience
any tactile feedback [105]. Additionally, any kind of 3D modeling can be
difficult for a user with limited or no experience. To address these issues,
researchers have developed suggestive modeling. This technique uses machine
learning or AI to suggest users the object that they aim to create. These
suggestions are generated through the analysis of the user’s past modeling
efforts or by trying to recognize the shape that they are attempting to
create. The system not only gives suggestions for future sculpting actions,
but it can also correct any mistakes as well. Iaruss et al. [114] and Peng et
al. [115] proposed this type of approach, using deep learning to recognize
previous shapes and provide suggestions for improvement. Peng et al. [27] also
conducted research on automating the modeling process by mirroring or
predicting the user’s actions (as shown in Figure 6). For 3D sketching, there
are methods for model-guided sketching described by Xu et al. [78], and El
Sayed et al. [79] that guide users for the upcoming strokes. Guillard et al.
[51] and Zhang et al. [55] have also presented frameworks that take a sketch
input and generate a 3D mesh or model to represent the shape, allowing the
user to perform further edits.

###  5.2 Shape Generation using AI

Sculpting or sketching shapes can be challenging for novices. To address this,
researchers have explored using artificial intelligence to assist users in
modeling figures. Typically, an input is given in the form of a sketch or
image, and a more precise 3D shape is generated, represented in mesh [30,
116], voxel [30], or implicit [31, 117, 118, 119] data structures. From either
single-view or multi-view images, users can produce a 3D structure. On
occasions, these 3D models are editable [116, 119]. However, editing
capabilities are typically restricted to one model at a time.

In 2020, Mildenhall et al. [120] introduced the groundbreaking concept of the
Neural Radiance Field (NeRF). This innovative framework offered a novel
approach to 3D scene generation. A distinct advantage of the NeRF is its
ability to bypass the requirement of possessing data corresponding to each
individual frame. Concurrently, advancements in AI-driven image generation
techniques have facilitated the creation of novel views from singular image
via the application of the diffusion model [121]. An intriguing convergence of
technologies arises when one combines the capabilities of the diffusion model
with the foundational principles of the NeRF. By integrating the diffusion
model with NeRF, it becomes possible to instantiate an object solely from an
image or a textual prompt [34]. Subsequent to their inception, both techniques
went through refinements to enhance the fidelity of the output [122, 123, 124]
and to speed up [125, 126, 127] the scene generation process. Beyond these,
researchers have devised even more intricate methodologies for converting
images or textual prompts into 3D representations [128, 129] have been
developed. In a noteworthy contribution to this field was NeRFShop by Jambon
et al. [35], who proposed a methodology allowing users to interact directly
with the generated scenes. This not only streamlines the creation of a desired
object but also provides users the ability to modify it post-generation.
Mikhaeli et al. [86] proposed SKED, where a NeRF model is edited using
sketches and text prompts.

Figure 6: An example of automated sculpting. New strokes are automatically
added based on the previous stroke. [27].

##  6 Collaborative Editing

It is common for users and designers to want to present their models or
drawings to others for showcasing or review purposes. Collaborative modeling
enables multiple individuals to work on the same object simultaneously. It
presents its own challenges, such as conflicting edits to the same section of
the model, and latency issues when designers are working remotely. Hernández
et al. [130] proposing an innovative, easy-to-use, and optimized interface,
Denning et al. [131] presented a "diff and merge" method where multiple users
can work on separate parts of the model and merge their work into the main
model later. Calabrese et al. [25] (see Figure 7) proposed a unique
collaboration technique where changes made by two or more users can be merged
by taking either the average, a cumulative or weighted sum of the changes.
Hoppe et al. [132] designed a system that allows users to work both locally
and remotely simultaneously.

Figure 7: Collaborative modeling between two users [25].

##  7 Discussion

We have observed that free-form shape modeling has garnered significant
attention due to the advancement of affordable XR tracking and display
technologies. This has subsequently captivated the interest of numerous
researchers and practitioners in the domain. Earlier, we mentioned that we aim
to address several questions that emerged during our discussion on methodology
(see section 2). After conducting a thorough review, we have been able to
provide answers to these questions.

|
---|---
(a) Sculpting | (b) Sketching
Figure 8: The trend for research in different environments and interaction
tool for sculpting and sketching methods.

###  7.1 Preferred Techniques for modeling and their methods

Our findings suggest that 3D sketching is a preferred modeling technique
compared to virtual sculpting. Based on the articles analyzed for this study,
it can also be concluded that researchers exhibit a preference for the 3D
sketching method. This conclusion is derived from the abundant and diverse
research available on sketching techniques as opposed to that on sculpting
techniques. Furthermore, this preference may arise because users can start
from scratch with 3D sketching and gradually progress, while sculpting may be
unfamiliar or challenging to receive adequate feedback on. In contrast,
various methods have been developed for sketching with a stylus and tablet,
some of which replicate the pen and paper experience through vibrations.

During our research, we encountered diverse methodologies associated with
virtual sculpting and 3D sketching. Although we attempted to categorize the
majority of these methods, it remains plausible that certain techniques may
have been overlooked. In the realm of sculpting, we identified notable
techniques such as virtual clay, wherein users modify a filled shape.
Conversely, in surface modeling techniques, users adjust the surface of a
hollow form. Another identified method is sculpting with a physical proxy, in
which the user replicates editing actions on a tangible object. The material
is segregated into layers to incorporate intricate details in multi-layer
sculpting. Similarly, our study on 3D sketching unveiled methodologies like
generating 3D shapes either from 2D or 3D sketches, sketching curves within 3D
spaces, and employing a projected model as a guide during 3D sketching.
Additionally, our study touched upon AI shape modeling techniques utilizing
radiance fields. In Figure 8, we have shown the research trend for sculpting
and sketching. It is a metric that shows the involvement of different
environment and interaction tools.

###  7.2 Research Gaps

Upon scrutinizing our study, we identified several research gaps. However,
some of them seem to recur, appearing with notable frequency across our
observations and analyses. Although using VR or MR headsets in 3D shape
modeling undeniably offers discernible benefits, they fall short in
replicating the tactile feedback experienced in tangible, real-world
scenarios. Additionally, the occasional latency experienced within virtual
environments can considerably diminish immersive user experience. This can be
attributed to the inherent computational constraints of contemporary hardware.
Notably, there appears to be significant potential for advancements in AI
shape modeling, an area presently constrained by the limits of current
computational capabilities.

###  7.3 Preferred Hardware

Our analysis compellingly indicates that, for 3D sketching in any environment,
users exhibit a strong inclination towards tools that closely emulate the
tactile experience of drawing with a pen, capturing the intuitive essence of
the traditional pen-and-paper approach. On the other hand, gestures emerged as
the preferred tools for sculpting. They offer the dual advantage of
encompassing larger regions for shaping while still affording the user the
precision necessary for detailed work. Yet, it’s undeniable that auxiliary
tools are crucial for ultra-fine operations. It’s imperative that future
research robustly addresses the palpable challenges of delivering apt feedback
for both sculpting and sketching in a virtual 3D milieu. Additionally, the
seamless incorporation of AI and collaborative infrastructures into these
methods beckons more in-depth exploration. The intricacies of managing
simultaneous alterations by multiple contributors during cooperative modeling
persist as a monumental hurdle.

In recent years, there has been an increase in the attention given to the
field of AI model generation. This surge in interest reflects the potential
and importance of artificial intelligence in shaping the technological
landscape. However, as with many emerging technologies, it is not without its
challenges. Presently, the algorithms that drive the creation of AI models
tend to operate at a pace that some may find sub-optimal. This slow rate of
generation can hinder rapid progress, especially in applications that demand
quick turnarounds. Moreover, the AI models’ qualitative aspects pose a
significant concern. For VR, the experience is largely determined by the
fidelity and quality of the underlying models. The present AI models,
unfortunately, do not always meet the high standards required to create a
truly immersive VR experience.

Another aspect of the AI model generation process that currently lags behind
is the subsequent editing and modification phase. The tools and methodologies
currently available for this purpose exhibit a lack of intuitive design, even
in the desktop environments. Presently, many of these editing methods are
reliant on text-based prompts, which introduces another layer of complexity.
The use of text prompts for model editing is filled with challenges. For one,
with the text prompt, users lose artistic control. When relying on textual
instructions, an inherent ambiguity can limit the precision with which one can
modify the AI model. Moreover, a seemingly minor alteration or update to a
prompt can sometimes trigger a dramatic change in the resultant model. In
applications where continuity is paramount, such as in the creation of digital
assets, this unpredictability can be particularly problematic.

However, researchers are diligently working on developing high-quality and
rapid 3D representations that can be modified even after their initial
generation. Presently, for images, we possess the capability to modify them
through techniques like in-painting, dragging, and text prompts. Drawing
inspiration from these advancements, researchers are now striving to adapt and
implement similar modifications for 3D-generated models. It is anticipated
that in the near future, such manipulations of 3D models will become
commonplace and widely accessible.

##  8 Conclusion

In this survey, we presented state-of-the-art free-form shape modeling in
extended reality. We began by examining the desktop environments, then
expanded our survey to include augmented and virtual reality as mediums for
shape modeling. Our survey compares techniques of free-form shape modeling,
such as sculpting and 3D sketching, and discusses the research on both
algorithmic and interactive aspects. Based on the collected set of
publications that encompasses all relevant techniques, we derived a taxonomy
of free-form shape modeling consisting of three main categories: the field of
contribution, either on the algorithm side or the interaction side; the
working environment, such as a desktop, AR, or VR; and the type of interaction
method used, such as gestures, controllers or pen-like tools, even though gaps
in the taxonomy hint at the areas where work has been done, as well as hint at
opportunities to fill gaps in existing research literature. We also identified
significant challenges that could guide future research. With the
contributions mentioned, our aspiration is for an enhanced comprehension of
free-form shape modeling.

## References

  * [1] Andreas Geiger, Julius Ziegler, and Christoph Stiller.  Stereoscan: Dense 3d reconstruction in real-time.  In 2011 IEEE intelligent vehicles symposium (IV), pages 963–968. Ieee, 2011.
  * [2] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al.  Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera.  In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 559–568, 2011.
  * [3] Etienne Mouragnon, Maxime Lhuillier, Michel Dhome, Fabien Dekeyser, and Patrick Sayd.  Real time localization and 3d reconstruction.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 1, pages 363–370. IEEE, 2006.
  * [4] Rhodri H Davies, Timothy F Cootes, Carole J Twining, and Christopher J Taylor.  An information theoretic approach to statistical shape modelling.  In BMVC, pages 1–10, 2001.
  * [5] Daniel F. Keefe, Daniel Acevedo Feliz, Tomer Moscovich, David H. Laidlaw, and Joseph J. LaViola.  CavePainting: A fully immersive 3D artistic medium and interactive experience.  In Proceedings of the 2001 Symposium on Interactive 3D Graphics - SI3D ’01, pages 85–93. ACM Press, 2001.
  * [6] Luke Olsen, Faramarz F Samavati, Mario Costa Sousa, and Joaquim A Jorge.  Sketch-based modeling: A survey.  Computers & Graphics, 33(1):85–103, 2009.
  * [7] Chao Ding and Ligang Liu.  A survey of sketch based modeling systems.  Frontiers of Computer Science, 10:985–999, 2016.
  * [8] Peng Xu, Timothy M Hospedales, Qiyue Yin, Yi-Zhe Song, Tao Xiang, and Liang Wang.  Deep learning for free-hand sketch: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  * [9] Peng Wang, Xiaoliang Bai, Mark Billinghurst, Shusheng Zhang, Xiangyu Zhang, Shuxia Wang, Weiping He, Yuxiang Yan, and Hongyu Ji.  Ar/mr remote collaboration on physical tasks: A review.  Robotics and Computer-Integrated Manufacturing, 72:102071, 2021.
  * [10] Matthew J Page, David Moher, Patrick M Bossuyt, Isabelle Boutron, Tammy C Hoffmann, Cynthia D Mulrow, Larissa Shamseer, Jennifer M Tetzlaff, Elie A Akl, Sue E Brennan, Roger Chou, Julie Glanville, Jeremy M Grimshaw, Asbjørn Hróbjartsson, Manoj M Lalu, Tianjing Li, Elizabeth W Loder, Evan Mayo-Wilson, Steve McDonald, Luke A McGuinness, Lesley A Stewart, James Thomas, Andrea C Tricco, Vivian A Welch, Penny Whiting, and Joanne E McKenzie.  Prisma 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews.  BMJ, 372, 2021.
  * [11] Tinsley A Galyean and John F Hughes.  Sculpting: An interactive volumetric modeling technique.  ACM SIGGRAPH Computer Graphics, 25(4):267–274, 1991.
  * [12] H.J. Lamousin and N.N. Waggenspack.  NURBS-based free-form deformations.  IEEE Computer Graphics and Applications, 14(6):59–65, November 1994.
  * [13] Janis PY Wong, Rynson WH Lau, and Lizhuang Ma.  Virtual 3d sculpting.  The Journal of Visualization and Computer Animation, 11(3):155–166, 2000.
  * [14] Steven Schkolne, Michael Pruett, and Peter Schröder.  Surface drawing: Creating organic 3D shapes with the hand and tangible tools.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’01, pages 261–268. Association for Computing Machinery, 2001.
  * [15] Kevin T. McDonnell, Hong Qin, and Robert A. Wlodarczyk.  Virtual clay: A real-time sculpting system with haptic toolkits.  In Proceedings of the 2001 Symposium on Interactive 3D Graphics - SI3D ’01, pages 179–190. ACM Press, 2001.
  * [16] Alfredo Liverani and Serena Morigi.  Efficient 6DOF tools for free-form surface modelling.  Vis Comput, 20(8):554–564, November 2004.
  * [17] Jia Sheng, Ravin Balakrishnan, and Karan Singh.  An interface for virtual 3D sculpting via physical proxy.  In Proceedings of the 4th International Conference on Computer Graphics and Interactive Techniques in Australasia and Southeast Asia - GRAPHITE ’06, page 213, Kuala Lumpur, Malaysia, 2006. ACM Press.
  * [18] Gabjong Han, Jane Hwang, Seungmoon Choi, and Gerard Jounghyun Kim.  AR Pottery: Experiencing Pottery Making in the Augmented Space.  In Randall Shumaker, editor, Virtual Reality, Lecture Notes in Computer Science, pages 642–650. Springer, 2007.
  * [19] Adeline Pihuit, Paul G. Kry, and Marie-Paule Cani.  Hands on virtual clay.  In 2008 IEEE International Conference on Shape Modeling and Applications, pages 267–268, 2008.
  * [20] Michael R. Marner and Bruce H. Thomas.  Augmented foam sculpting for capturing 3D models.  In 2010 IEEE Symposium on 3D User Interfaces (3DUI), pages 63–70, 2010.
  * [21] Sean Follmer, Micah Johnson, Edward Adelson, and Hiroshi Ishii.  deForm: An interactive malleable surface for capturing 2.5D arbitrary objects, tools and touch.  In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST ’11, pages 527–536. Association for Computing Machinery, 2011.
  * [22] Amit Zoran and Joseph A. Paradiso.  FreeD: A freehand digital sculpting tool.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’13, 2013.
  * [23] Lucian Stanculescu, Raphaëlle Chaine, Marie-Paule Cani, and Karan Singh.  Sculpting multi-dimensional nested structures.  Computers & graphics, 37(6):753–763, 2013.
  * [24] Antoine Milliez, Michael Wand, M-P Cani, and H-P Seidel.  Mutable elastic models for sculpting structured shapes.  Computer Graphics Forum, 32(2pt1):21–30, 2013.
  * [25] Claudio Calabrese, Gabriele Salvati, Marco Tarini, and Fabio Pellacini.  cSculpt: A system for collaborative sculpting.  ACM Transactions on Graphics, 35(4):1–8, 2016.
  * [26] Calabrese Claudio, Fratarcangeli Marco, Fabio Pellacini, et al.  slayer: a system for multi-layered material sculpting.  EGSR’17: Proceedings of the Eurographics Symposium on Rendering: Experimental Ideas & Implementations, 2017.
  * [27] Mengqi Peng, Jun Xing, and Li-Yi Wei.  Autocomplete 3D sculpting.  ACM Transactions on Graphics, 37(4):132:1–132:15, 2018.
  * [28] Zihan Gao, Jingwen Li, Huiqiang Wang, and Guangsheng Feng.  DigiClay: An Interactive Installation for Virtual Pottery using Motion Sensing Technology.  In Proceedings of the 4th International Conference on Virtual Reality, ICVR 2018, pages 126–132. Association for Computing Machinery, 2018.
  * [29] Zihan Gao, Huiqiang Wang, Guangsheng Feng, Fangfang Guo, Hongwu Lv, and Bingyang Li.  RealPot: An immersive virtual pottery system with handheld haptic devices.  Multimedia Tools and Applications, 78(18):26569–26596, 2019.
  * [30] Georgia Gkioxari, Jitendra Malik, and Justin Johnson.  Mesh r-cnn.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9785–9795, 2019.
  * [31] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165–174, 2019.
  * [32] Phum Natakuaithung and Worapan Kusakunniran.  Development of AR Learning Assistance Tool for Clay-Sculpting 3D Model.  In 2020 12th International Conference on Knowledge and Smart Technology (KST), pages 109–114, 2020.
  * [33] Sarah Dashti, Edmond Prakash, Andres Adolfo Navarro-Newball, Fiaz Hussain, and Fiona Carroll.  PotteryVR: Virtual reality pottery.  The Visual Computer, 2022.
  * [34] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [35] Clément Jambon, Bernhard Kerbl, Georgios Kopanas, Stavros Diolatzis, George Drettakis, and Thomas Leimkühler.  Nerfshop: Interactive editing of neural radiance fields.  Proceedings of the ACM on Computer Graphics and Interactive Techniques, 6(1), 2023.
  * [36] Kin Chung Kwan and Hongbo Fu.  Mobi3DSketch: 3D Sketching in Mobile AR.  In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, pages 1–11. Association for Computing Machinery, 2019.
  * [37] Rahul Arora, Rubaiat Habib Kazi, Fraser Anderson, Tovi Grossman, Karan Singh, and George Fitzmaurice.  Experimental Evaluation of Sketching on Surfaces in VR.  In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 5643–5654. ACM, 2017.
  * [38] Daniel F. Keefe, Daniel Acevedo, Jadrian Miles, Fritz Drury, Sharon M. Swartz, and David H. Laidlaw.  Scientific Sketching for Collaborative VR Visualization Design.  IEEE Transactions on Visualization and Computer Graphics, 14(4):835–847, 2008.
  * [39] Bret Jackson and Daniel F. Keefe.  Lift-Off: Using Reference Imagery and Freehand Sketching to Create 3D Models in VR.  IEEE Transactions on Visualization and Computer Graphics, 22(4):1442–1451, 2016.
  * [40] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In Seminal graphics: pioneering efforts that shaped the field, pages 347–353, 1998.
  * [41] Héctor Barreiro, Joan Torres, and Miguel A. Otaduy.  Natural Tactile Interaction with Virtual Clay.  In 2021 IEEE World Haptics Conference (WHC), pages 403–408, 2021.
  * [42] Ammar Hattab and Gabriel Taubin.  3D Modeling by Scanning Physical Modifications.  In 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images, pages 25–32, 2015.
  * [43] Hyunjoo Oh, Tung D. Ta, Ryo Suzuki, Mark D. Gross, Yoshihiro Kawahara, and Lining Yao.  PEP (3D Printed Electronic Papercrafts): An Integrated Approach for 3D Sculpting Paper-Based Electronic Devices.  In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1–12. ACM, 2018.
  * [44] Ryan Schmidt, Azam Khan, Karan Singh, and Gord Kurtenbach.  Analytic drawing of 3D scaffolds.  In ACM SIGGRAPH Asia 2009 Papers, SIGGRAPH Asia ’09, pages 1–10. Association for Computing Machinery, 2009.
  * [45] Michael F. Deering.  HoloSketch: A virtual reality sketching/animation tool.  ACM Transactions on Computer-Human Interaction, 2(3):220–238, 1995.
  * [46] Jonathan M Cohen, Lee Markosian, Robert C Zeleznik, John F Hughes, and Ronen Barzel.  An interface for sketching 3d curves.  In Proceedings of the 1999 symposium on Interactive 3D graphics, pages 17–21, 1999.
  * [47] Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka.  Teddy: A sketching interface for 3D freeform design.  In ACM SIGGRAPH 2006 Courses, SIGGRAPH ’06, pages 11–es. Association for Computing Machinery, 2006.
  * [48] Haixiong Wang and Lee Markosian.  Free-form Sketch.  In Proceedings of the 4th Eurographics Workshop on Sketch-based Interfaces and Modeling, SBIM ’07, pages 53–58. Association for Computing Machinery, 2007.
  * [49] Mathias Eitz, Ronald Richter, Tamy Boubekeur, Kristian Hildebrand, and Marc Alexa.  Sketch-based shape retrieval.  ACM Transactions on Graphics, 31(4):1–10, 2012.  cites: eitzSketchbasedShapeRetrieval2012.
  * [50] Marcos Ramos, Leandro Teixeira, Vitor Martins, Anselmo Montenegro, Daniela G. Trevisan, and Cristina Nader Vasconcelos.  A New User-Friendly Sketch-Based Modeling Method Using Convolution Surfaces.  In 2016 XVIII Symposium on Virtual and Augmented Reality (SVR), pages 100–108, 2016.
  * [51] Benoit Guillard, Edoardo Remelli, Pierre Yvernay, and Pascal Fua.  Sketch2mesh: Reconstructing and editing 3d shapes from sketches.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13023–13032, 2021.
  * [52] Baoxuan Xu, William Chang, Alla Sheffer, Adrien Bousseau, James McCrae, and Karan Singh.  True2form: 3d curve networks from 2d sketches via selective regularization.  ACM Transactions on Graphics, 33(4), 2014.
  * [53] Olga A Karpenko and John F Hughes.  Smoothsketch: 3d free-form shapes from complex sketches.  In ACM SIGGRAPH 2006 Papers, pages 589–598. Association for Computing Machinery, 2006.
  * [54] Rubaiat Habib Kazi, Tovi Grossman, Hyunmin Cheong, Ali Hashemi, and George Fitzmaurice.  DreamSketch: Early Stage 3D Design Explorations with Sketching and Generative Design.  In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pages 401–414. ACM, 2017.
  * [55] Song-Hai Zhang, Yuan-Chen Guo, and Qing-Wen Gu.  Sketch2model: View-aware 3d modeling from single free-hand sketches.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6012–6021, 2021.
  * [56] Xuejin Chen, Sing Bing Kang, Ying-Qing Xu, Julie Dorsey, and Heung-Yeung Shum.  Sketching reality: Realistic interpretation of architectural designs.  ACM Transactions on Graphics, 27(2):1–15, 2008.
  * [57] Yang Liu and Yong Tsui Lee.  3D reconstruction of freeform shapes from 2D line drawings.  In Proceedings of the 9th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry, VRCAI ’10, pages 303–310. Association for Computing Machinery, 2010.
  * [58] Yuxiang Ye, Bo Li, and Yijuan Lu.  3D sketch-based 3D model retrieval with convolutional neural network ???  In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2936–2941, 2016.
  * [59] Yotam Gingold, Takeo Igarashi, and Denis Zorin.  Structured annotations for 2D-to-3D modeling.  In ACM SIGGRAPH Asia 2009 Papers, SIGGRAPH Asia ’09, pages 1–9. Association for Computing Machinery, 2009.
  * [60] Gerold Wesche and Hans-Peter Seidel.  FreeDrawer: A free-form sketching system on the responsive workbench.  In Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST ’01, pages 167–174, 2001.
  * [61] Daniel Keefe, Robert Zeleznik, and David Laidlaw.  Drawing on Air: Input Techniques for Controlled 3D Line Illustration.  IEEE Transactions on Visualization and Computer Graphics, 13(5):1067–1081, 2007.
  * [62] Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Marc Alexa.  FiberMesh: Designing freeform surfaces with 3D curves.  ACM Transactions on Graphics, 26(3):41, 2007.
  * [63] Guisheng Fang and Jianjun Wang.  An interactive sketching modeling system: Sketch3d.  In 2008 International Conference on Computer Science and Software Engineering, volume 2, pages 1049–1052. IEEE, 2008.
  * [64] Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh.  ILoveSketch: As-natural-as-possible sketching system for creating 3d curve models.  In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology, UIST ’08, pages 151–160. Association for Computing Machinery, 2008.
  * [65] Min Xin, Ehud Sharlin, and Mario Costa Sousa.  Napkin sketch: Handheld mixed reality 3D sketching.  In Proceedings of the 2008 ACM Symposium on Virtual Reality Software and Technology, VRST ’08, pages 223–226. Association for Computing Machinery, 2008.
  * [66] Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh.  EverybodyLovesSketch: 3D sketching for a broader audience.  In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology, UIST ’09, pages 59–68. Association for Computing Machinery, 2009.
  * [67] Ricardo Fabbri and Benjamin Kimia.  3D curve sketch: Flexible curve-based stereo reconstruction and calibration.  In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1538–1545, 2010.
  * [68] Johannes Schmid, Martin Sebastian Senn, Markus Gross, and Robert W. Sumner.  OverCoat: An implicit canvas for 3D painting.  In ACM SIGGRAPH 2011 Papers, SIGGRAPH ’11, pages 1–10. Association for Computing Machinery, 2011.
  * [69] Alexis Andre and Suguru Saito.  Single-view sketch based modeling.  In Proceedings of the Eighth Eurographics Symposium on Sketch-Based Interfaces and Modeling, SBIM ’11, pages 133–140. Association for Computing Machinery, 2011.
  * [70] Cindy Grimm and Pushkar Joshi.  Just Draw It! A 3D Sketching System.  Proceedings of the 9th Eurographics workshop on Sketch-based interfaces and modeling, page 16, 2012.
  * [71] Chris De Paoli and Karan Singh.  SecondSkin: Sketch-based construction of layered 3D models.  ACM Transactions on Graphics, 34(4):126:1–126:10, 2015.
  * [72] Bo Li, Yijuan Lu, Azeem Ghumman, Bradley Strylowski, Mario Gutierrez, Safiyah Sadiq, Scott Forster, Natacha Feola, and Travis Bugerin.  3D Sketch-Based 3D Model Retrieval.  In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, ICMR ’15, pages 555–558. Association for Computing Machinery, 2015.
  * [73] Youyi Zheng, Han Liu, Julie Dorsey, and Niloy J. Mitra.  SmartCanvas: Context-inferred Interpretation of Sketches for Preparatory Design Studies.  Computer Graphics Forum, pages 37–48, 2016.
  * [74] Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang.  3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks.  In 2017 International Conference on 3D Vision (3DV), pages 67–77, 2017.
  * [75] Ya-Ting Yue, Xiaolong Zhang, Yongliang Yang, Gang Ren, Yi-King Choi, and Wenping Wang.  WireDraw: 3D Wire Sculpturing Guided with Mixed Reality.  In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17, pages 3693–3704. Association for Computing Machinery, 2017.
  * [76] Yuwei Li, Xi Luo, Youyi Zheng, Pengfei Xu, and Hongbo Fu.  SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image.  In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, UIST ’17, pages 387–399. Association for Computing Machinery, 2017.
  * [77] Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and Karan Singh.  SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ.  In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, pages 1–15. Association for Computing Machinery, 2018.
  * [78] Pengfei Xu, Hongbo Fu, Youyi Zheng, Karan Singh, Hui Huang, and Chiew-Lan Tai.  Model-Guided 3D Sketching.  IEEE Transactions on Visualization and Computer Graphics, 25(10):2927–2939, 2019.
  * [79] Hesham Elsayed, Mayra Donaji Barrera Machuca, Christian Schaarschmidt, Karola Marky, Florian Müller, Jan Riemann, Andrii Matviienko, Martin Schmitz, Martin Weigel, and Max Mühlhäuser.  VRSketchPen: Unconstrained Haptic Assistance for Sketching in Virtual 3D Environments.  In 26th ACM Symposium on Virtual Reality Software and Technology, VRST ’20, pages 1–11. Association for Computing Machinery, 2020.
  * [80] Tobias Drey, Jan Gugenheimer, Julian Karlbauer, Maximilian Milo, and Enrico Rukzio.  VRSketchIn: Exploring the Design Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality.  In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, pages 1–14. Association for Computing Machinery, 2020.
  * [81] Ying Jiang, Congyi Zhang, Hongbo Fu, Alberto Cannavò, Fabrizio Lamberti, Henry Y K Lau, and Wenping Wang.  HandPainter - 3D Sketching in VR with Hand-based Physical Proxy.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, pages 1–13. Association for Computing Machinery, 2021.
  * [82] Emilie Yu, Rahul Arora, Tibor Stanko, J. Andreas Bærentzen, Karan Singh, and Adrien Bousseau.  CASSIE: Curve and Surface Sketching in Immersive Environments.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–14. ACM, 2021.
  * [83] Xue Yu, Stephen DiVerdi, Akshay Sharma, and Yotam Gingold.  ScaffoldSketch: Accurate Industrial Design Drawing in VR.  In The 34th Annual ACM Symposium on User Interface Software and Technology, UIST ’21, pages 372–384. Association for Computing Machinery, 2021.
  * [84] Rahul Arora and Karan Singh.  Mid-Air Drawing of Curves on 3D Surfaces in Virtual Reality.  ACM Transactions on Graphics, 40(3):1–17, 2021-07-04.
  * [85] Emilie Yu, Rahul Arora, J. Andreas Bærentzen, Karan Singh, and Adrien Bousseau.  Piecewise-smooth surface fitting onto unstructured 3D sketches.  ACM Transactions on Graphics, 41(4):1–16, 2022.
  * [86] Aryan Mikaeili, Or Perel, Mehdi Safaee, Daniel Cohen-Or, and Ali Mahdavi-Amiri.  Sked: Sketch-guided text-based 3d editing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14607–14619, 2023.
  * [87] Songhua Ma and Ling Tian.  A 3d sketch-based modeling design system with paper-and-pencil interface.  In 2013 International Conference on Computer Sciences and Applications, pages 446–453. IEEE, 2013.
  * [88] Tibor Stanko, Stefanie Hahmann, Georges-Pierre Bonneau, and Nathalie Saguin-Sprynski.  Smooth interpolation of curve networks with surface normals.  In Eurographics 2016 Short Papers, pages 21–24. Eurographics Association, 2016.
  * [89] Robert C. Zeleznik, Kenneth P. Herndon, and John F. Hughes.  SKETCH: An interface for sketching 3D scenes.  In ACM SIGGRAPH 2006 Courses, SIGGRAPH ’06, pages 9–es. Association for Computing Machinery, 2006.
  * [90] Fang Wang, Le Kang, and Yi Li.  Sketch-based 3d shape retrieval using convolutional neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1875–1883, 2015.
  * [91] Daniele Giunchi, Stuart James, and Anthony Steed.  3D sketching for interactive model retrieval in virtual reality.  In Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering, pages 1–12. ACM, 2018.
  * [92] Steve Tsang, Ravin Balakrishnan, Karan Singh, and Abhishek Ranjan.  A suggestive interface for image guided 3d sketching.  In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pages 591–598, 2004.
  * [93] Hyosun Kim, Georgia Albuquerque, Sven Havemann, and Dieter W Fellner.  Tangible 3d: Hand gesture interaction for immersive 3d modeling.  IPT/EGVE, 2005:191–9, 2005.
  * [94] Bradley Wesson and Brett Wilkinson.  Evaluating organic 3D sculpting using natural user interfaces with the Kinect.  In Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration, OzCHI ’13, pages 163–166. Association for Computing Machinery, 2013.
  * [95] Yongkwan Kim and Seok-Hyung Bae.  SketchingWithHands: 3D Sketching Handheld Products with First-Person Hand Posture.  In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST ’16, pages 797–808. Association for Computing Machinery, 2016.
  * [96] Judith Amores and Jaron Lanier.  HoloARt: Painting with Holograms in Mixed Reality.  In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA ’17, pages 421–424. Association for Computing Machinery, 2017.
  * [97] John J. Dudley, Hendrik Schuff, and Per Ola Kristensson.  Bare-Handed 3D Drawing in Augmented Reality.  In Proceedings of the 2018 Designing Interactive Systems Conference, DIS ’18, pages 241–252. Association for Computing Machinery, 2018.
  * [98] Miri Cohen, Daniele Regazzoni, and Chen Vrubel.  A 3D Virtual Sketching System Using NURBS Surfaces and Leap Motion Controller.  Computer-Aided Design and Applications, 17(1):167–177, 2019.
  * [99] Martin Tosas and Bai Li.  Virtual Touch Screen for Mixed Reality.  In Nicu Sebe, Michael Lew, and Thomas S. Huang, editors, Computer Vision in Human-Computer Interaction, Lecture Notes in Computer Science, pages 48–59. Springer, 2004.
  * [100] Martin Tosas and Li Bai.  Virtual touch screen: a vision-based interactive surface.  In 9th IASTED international conference on computer graphics and imaging, Innsbruck, Austria. ACTA Press, pages 81–86, 2007.
  * [101] Paulo Gallotti, Alberto Raposo, and Luciano Soares.  v-glove: A 3d virtual touch interface.  In 2011 XIII Symposium on Virtual Reality, pages 242–251. IEEE, 2011.
  * [102] Jan Jacobs and Bernd Froehlich.  A soft hand model for physically-based manipulation of virtual objects.  In 2011 IEEE Virtual Reality Conference, pages 11–18, 2011.
  * [103] Yongkwan Kim, Sang-Gyun An, Joon Hyub Lee, and Seok-Hyung Bae.  Agile 3D Sketching with Air Scaffolding.  In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, pages 1–12. Association for Computing Machinery, 2018.
  * [104] Patrick Paczkowski, Min H Kim, Yann Morvan, Julie Dorsey, Holly E Rushmeier, and Carol O’Sullivan.  Insitu: sketching architectural designs in context.  ACM Trans. Graph., 30(6):182, 2011.
  * [105] Sho Kamuro, Yuta Takeuchi, Kouta Minamizawa, and Susumu Tachi.  Haptic editor: Creation and editing system for touchable 3D content.  In SIGGRAPH Asia 2012 Technical Briefs, SA ’12, pages 1–4. Association for Computing Machinery, 2012.
  * [106] Philipp Pascal Hoffmann, Hesham Elsayed, Max Mühlhäuser, Rina R Wehbe, and Mayra Donaji Barrera Machuca.  Thermalpen: Adding thermal haptic feedback to 3d sketching.  In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–4, 2023.
  * [107] Ling Luo, Yulia Gryaditskaya, Yongxin Yang, Tao Xiang, and Yi-Zhe Song.  Towards 3D VR-Sketch to 3D Shape Retrieval.  In 2020 International Conference on 3D Vision (3DV), pages 81–90, 2020.
  * [108] Xinchi Xu, Yang Zhou, Bingchan Shao, Guihuan Feng, and Chun Yu.  Gesturesurface: Vr sketching through assembling scaffold surface with non-dominant hand.  IEEE Transactions on Visualization and Computer Graphics, 29(5):2499–2507, 2023.
  * [109] Sevinc Eroglu, Sascha Gebhardt, Patric Schmitz, Dominik Rausch, and Torsten Wolfgang Kuhlen.  Fluid Sketching—Immersive Sketching Based on Fluid Flow.  In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 475–482, 2018.
  * [110] Samuel Dong and Tobias Höllerer.  Real-Time Re-Textured Geometry Modeling Using Microsoft HoloLens.  In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 231–237, 2018.
  * [111] Maxime Garcia, Remi Ronfard, and Marie-Paule Cani.  Spatial Motion Doodles: Sketching Animation in VR Using Hand Gestures and Laban Motion Analysis.  In Motion, Interaction and Games, MIG ’19, pages 1–10. Association for Computing Machinery, 2019.
  * [112] Jing Wang, Feng Tian, and Hock Soon Seah.  Sketch-Up in the Virtual World.  In 2008 International Conference on Cyberworlds, pages 109–116, 2008.
  * [113] Nate Hagbi, Raphaël Grasset, Oriel Bergig, Mark Billinghurst, and Jihad El-Sana.  In-Place Sketching for content authoring in Augmented Reality games.  In 2010 IEEE Virtual Reality Conference (VR), pages 91–94, 2010.
  * [114] Emmanuel Iarussi, Adrien Bousseau, and Theophanis Tsandilas.  The drawing assistant: Automated drawing guidance and feedback from photographs.  In ACM Symposium on User Interface Software and Technology (UIST). ACM, 2013.
  * [115] Mengqi Peng, Li-yi Wei, Rubaiat Habib Kazi, and Vladimir G. Kim.  Autocomplete Animated Sculpting.  In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, pages 760–777. Association for Computing Machinery, 2020.
  * [116] Abhijit Kundu, Yin Li, and James M Rehg.  3d-rcnn: Instance-level 3d object reconstruction via render-and-compare.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3559–3568, 2018.
  * [117] Rohan Chabra, Jan E Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe.  Deep local shapes: Learning local sdf priors for detailed 3d reconstruction.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16, pages 608–625. Springer, 2020.
  * [118] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser.  Local deep implicit functions for 3d shape.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4857–4866, 2020.
  * [119] Petros Tzathas, Petros Maragos, and Anastasios Roussos.  3d neural sculpting (3dns): Editing neural signed distance functions.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4521–4530, 2023.
  * [120] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [121] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [122] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
  * [123] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
  * [124] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.
  * [125] Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  arXiv preprint arXiv:2112.05131, 2021.
  * [126] Marilena-Cătălina Pupezescu and Valentin Pupezescu.  Instant neural graphics primitives framework and its emerging applications in digitalization of dental care.  In 2022 E-Health and Bioengineering Conference (EHB), pages 1–4. IEEE, 2022.
  * [127] Ruilong Li, Matthew Tancik, and Angjoo Kanazawa.  Nerfacc: A general nerf acceleration toolbox.  arXiv preprint arXiv:2210.04847, 2022.
  * [128] Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi.  Realfusion: 360deg reconstruction of any object from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8446–8455, 2023.
  * [129] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300–309, 2023.
  * [130] Hamilton A. Hernández and Helmuth Trefftz.  WeSketch: A 3D Real Time Collaborative Virtual Environment that Improves the GUI Sketching Task.  In 2011 Eighth International Conference on Information Technology: New Generations, pages 163–168, 2011.
  * [131] Jonathan D. Denning and Fabio Pellacini.  MeshGit: Diffing and merging meshes for polygonal modeling.  ACM Transactions on Graphics, 32(4):35:1–35:10, 2013.
  * [132] Adrian H Hoppe, Florian van de Camp, and Rainer Stiefelhagen.  Shisha: Enabling shared perspective with face-to-face collaboration using redirected avatars in virtual reality.  Proceedings of the ACM on Human-Computer Interaction, 4(CSCW3):1–22, 2021.