Figure 1: Using conventional methods to train the network, the training
process of the network is a black box, and the training process is
uncontrollable. Our proposed method projects the high-dimensional features of
the data into a 2D workspace, where the user can manually edit the high-
dimensional features. Using this method to train the network can control the
training process of the network to a certain extent, which not only allows
people to better understand the training process of the network but also
integrates human knowledge into the training process of the network, thereby
improving the performance of the network.

# SpaceEditing: Integrating Human Knowledge into Deep Neural Networks via
Interactive Latent Space Editing

Jiafu Wei, Ding Xia, Haoran Xie, Chia-Ming Chang, Chuntao Li, and Xi Yang
Jiafu Wei, Chuntao Li, and Xi Yang (corresponding author) are with Jilin
University. E-mail<EMAIL_ADDRESS>{lct33<EMAIL_ADDRESS>Ding Xia and Chia-Ming Chang are with the University of Tokyo. E-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>Haoran Xie is with Japan
Advanced Institute of Science and Technology. E-mail<EMAIL_ADDRESS>
###### Abstract

We propose an interactive editing method that allows humans to help deep
neural networks (DNNs) learn a latent space more consistent with human
knowledge, thereby improving classification accuracy on indistinguishable
ambiguous data. Firstly, we visualize high-dimensional data features through
dimensionality reduction methods and design an interactive system SpaceEditing
to display the visualized data. SpaceEditing provides a 2D workspace based on
the idea of spatial layout. In this workspace, the user can move the
projection data in it according to the system guidance. Then, SpaceEditing
will find the corresponding high-dimensional features according to the
projection data moved by the user, and feed the high-dimensional features back
to the network for retraining, therefore achieving the purpose of
interactively modifying the high-dimensional latent space for the user.
Secondly, to more rationally incorporate human knowledge into the training
process of neural networks, we design a new loss function that enables the
network to learn user-modified information. Finally, We demonstrate how
SpaceEditing meets user needs through three case studies while evaluating our
proposed new method, and the results confirm the effectiveness of our method.

###### Index Terms:

Interaction, deep learning, latent space, spatial editing.

##  1 Introduction

Although deep neural networks (DNNs) maintain excellent results in
classification, they still struggle to distinguish similar ambiguous data. The
machine learning community has realized the disadvantage of networks in
dealing with abstract things [51], such as things like shapes and concepts. In
addition, when faced with datasets in some specialized fields, such as
archaeological-related datasets, the performance of the network is not
satisfactory. The domain dataset requires corresponding domain knowledge.
Therefore, the recognition of features by humans should help the learning of
deep learning networks.

However, the learning process of current deep learning networks is still
uncontrollable (Fig. 1). In general, whether in the training process or the
fine-tuning process, people can only judge the effect of network training
through the results of loss or metrics, but cannot directly process the data.
If people want to achieve better results, they can only start with traditional
fine-tuning methods such as adjusting hyperparameters, but such methods often
require multiple debugging to achieve better results. To let people
participate in the network training process more intuitively, the role of
high-dimensional features in network training has been paid more and more
attention[59]. High-dimensional features can be observed through projection
visualization [52, 31, 23], but existing methods cannot directly affect high-
dimensional features, and the latent space learned by existing deep learning
networks is still uncontrollable.

Interactive machine learning is a good direction to address the above problem,
which tries to let human knowledge help the network learn. For example, Sakata
et al. [44] introduce a network called CROWNN, which allows people to
participate in the network classification process, and then it leverages the
learned human strengths to better perform classification tasks.

The user interface is one of the important means to support interactive
machine learning. On the one hand, the user interface can help people observe
the relationship between data more intuitively. On the other hand, the user
can participate in the training of the network through the interactive
function of the user interface. Therefore, it is necessary to design a
suitable spatial layout structure for the user interface. For example, Chang
et al. [12] used the spatial layout idea to design a system for improving the
annotation quality of non-professional annotators.

Although in past work, we proposed a method to fine-tune the network using 2D
projections of the data[58], this method does not work directly on high-
dimensional features, and at the same time, the effect of this method is not
significant. Therefore, in this paper, we propose a novel interactive machine
learning method to address the above issues. Through our designed system,
users can intuitively observe the distribution of data in latent space. Then,
users can also modify the high-dimensional features in the latent space, and
the modified information will be used to retrain the network to improve the
performance of the network. Our method realizes the visualization of high-
dimensional features by projecting the high-dimensional features in the
network onto a 2D workspace, and the location information of the projected
point reflects the classification result of the point to a certain extent.
Based on the various interactive functions of our system, users can reclassify
the projection results based on their knowledge, and can also move the points
they think are misclassified to the positions they think are correctly
classified. Then, the system can automatically feed back the result of the
user’s movement to the network for retraining (Fig. 2).

To summarize, our contributions include:

  1. 1.

We propose a novel and effective method that enables users to interactively
edit the latent space based on their knowledge, thereby guiding the network’s
learning process. This method can incorporate human knowledge into the
training process of the network, which not only makes the network jump out of
the local minimum area and improves the performance of the network but also
allows users to obtain a more understandable latent space.

  2. 2.

We design a new interactive system, SpaceEditing, which can apply our proposed
method and allow users to synchronize their operations from two-dimensional
space to high-dimensional space. In addition, it also has various interactive
functions such as enlarge function, visual volume adjustment, interactive
movement, movement guidance, and history record, which provide feasibility for
manual editing of latent space.

  3. 3.

We conduct three case studies to evaluate the effect of SpaceEditing on
different types of machine learning tasks and users with different identities.
The case studies indicate: the usefulness of the system; the effectiveness of
the proposed method; the flexibility to adapt to different scenarios; the
user’s experience and evaluation.

Figure 2:  The workflow of our proposed method. (1) In the preprocessing
stage, the raw data is trained to obtain a network for visualizing the images.
(2) The data features output by the network are projected into a 2D workspace
for the user to observe. (3) In the 2D workspace, users can easily observe the
visualized data, and at the same time, they can use the functions provided by
the system to easily interact with the projection points. (4) When the user
edits the projection point in the 2D workspace, (5) the system automatically
performs calculations in a high-dimensional latent space. (6) The network will
learn according to the changes of projection points before and after moving,
(7) therefore achieving the purpose of the user helping the network to
retrain.

##  2 Related work

###  2.1 Interactive Machine Learning

With the rapid development of graphical interfaces, the importance of humans
in the working and learning process of machines has become increasingly
apparent [36, 9, 32]. The advantage of IML (Interactive Machine Learning) is
that the addition of humans can help machines complete abstract tasks that are
difficult for machines [44, 16, 37, 2, 17]. Machine learning can be used as a
creative tool, and human participation can help group machine learning in a
creative and artistic direction [19, 24]. Through an interactive method, it is
easier for the machine to generate the data and networks the user wants [14,
53, 64, 37]. In addition, some studies add brain interface to the process of
machine learning [34], which not only makes the system show stronger learning
ability [16] but also shortens the training time of the network [33].

Furthermore, IML systems pay special attention to user experience and user
understanding of the system due to human involvement, so it is necessary to
design robust and user-friendly systems [4, 49, 47, 65]. Human-oriented is the
premise to achieve this goal. Driven by the human-oriented design concept[21,
40, 62, 20], many meaningful new methods of interactive machine learning have
been born[22, 48].

There are many works based on the idea of IML, but there is no way to train
the network through human interaction with the latent space, and the potential
of the latent space has not been tapped. To this end, we propose a new deep
learning method for interacting with the latent space.

###  2.2 Conventional Fine-tuning Methodes

Fine-tuning is an important method in the field of machine learning. The time
required to train a new network can be greatly simplified by modifying part of
the network to the network required by the user [63]. Based on traditional
fine-tuning methods, many novel fine-tuning methods have been generated. For
example, by changing part of the network structure to retrieve and classify
data such as images, Radenović et al. [41] finally realized a fine-tuning
method without manual annotation. Rosa et al. [43] introduce harmony search
and some of its variants to fine-tune image classification, filling a gap in
CNN parameter optimization research. Observing that not all parameters need to
be updated during fine-tuning, Xu et al. [61] proposed an efficient fine-
tuning technique CHILD-TUNING, which masks the gradients of non-sub-networks
in the reverse process. In addition to the above methods, visualization
techniques are also applied to fine-tune the network. For example, Amershi et
al. [3] designed a visualization tool ModelTracker, which can analyze the
performance of the network and help users fine-tune the network.

Latent space plays a very important role in network training [59]. Although
the above works have achieved very good results in network fine-tuning, they
have not proposed new fine-tuning methods from the perspective of latent
space. In addition, in the datasets of some specialized fields, it is usually
necessary to combine a large amount of domain knowledge to achieve better
results [8], therefore the human-in-the-loop method is very suitable. For
example, to address the low accuracy of medical image segmentation, Wang et
al. [56] proposed a novel interactive segmentation framework. Based on the
above ideas, we propose a novel retraining method from latent space and human-
in-the-loop perspectives.

###  2.3 Spatial Layout

The concept of spatial layout is widely used in management because spatial
layout not only stores information but also reflects the relationship between
information to a certain extent [28, 30]. There are many meaningful types of
research based on spatial layouts, such as dimensionality reduction
visualization [54, 7]. By mapping high-dimensional data to a 2D or 3D
interface, people can more easily observe the layout relationship between the
data and then analyze the data [5, 52, 31, 15]. The spatial layout can also
reflect the relationship between icons, objects, and other information [46,
60]. Reasonable use of spatial layout can achieve the purpose of assisting
users. For example, to address the lack of professional annotators, Chang et
al. [12] utilize spatial layout to design a novel annotation interface to
improve the annotation quality of non-expert image annotations. Wang et al.
[55] used the hierarchical spatial structure to optimize the way the map,
which solved the tedious process of users zooming in to see map details and
zooming out to see an overview. Mai et al. [38] discussed and studied the
factors affecting the user’s spatial layout, and showed the factors affecting
the user’s related spatial layout. By combining the advantages of small
multiples and visual aggregation with interactive browsing, Lekschas et al.
[35] propose a structured design space to guide the design of visual-spatial
layouts.

Numerous studies have proven that information can be effectively managed
through spatial arrangement [27, 57, 25]. One example is a spatial search
system that makes it easy for users to search for desired information in 2D
space by interacting with the visualized data [10]. Chen et al. [13] designed
a system to facilitate bug discovery through semantic data search, in which
users interactively create a topology to convey information in a spatial
layout. Human-centric spatial layout techniques have also emerged, which
visualize time-series data through user interfaces to enhance human-to-human
collaboration [39]. Asai et al. [6] combine a code editor with an interactive
scatterplot editor enabling users to effectively understand the behavior of
statistical modeling algorithms. With the continuous development of
technology, spatial layout, and machine learning have been continuously
integrated, resulting in a large number of novel and excellent research
results [11, 42]. For example, Eisenstadt et al. [18] leveraged machine
learning techniques to process information representations about building room
types and the spatial layout of individual rooms.

##  3 System

The latent space means the representation of encoded data, however it is
incomprehensible in most cases. To further identify and understand the latent
space, and integrate human capabilities into the machine training process, we
design a novel interactive system SpaceEditing that allows users to adjust the
position of vectors in the latent space. Then, the system will retrain the
parameters of the network with moved vectors to improve the performance step
by step.

###  3.1 Design Goals

The primary design goal of SpaceEditing is to provide a novel method to
interact with the latent space for users. The system should highlight the
relationships and connections between different types of data. At the same
time, the basic requirement is that the user can visually observe the
distribution of the data in the latent space, and the user can easily interact
with the vectors in the latent space.

We proposed three primary designing goals for SpaceEditing:

  1. 1.

Visualize the data representation intuitively. A fundamental function is the
visualization of the latent space, where our system could clearly and
correctly describe the similarities and differences within data
representations. In this way, users can locate target data and make the right
modification without effort.

  2. 2.

Design an effective interactive system. A batch-selection mechanism is
critical for a dataset comprising millions of images. With this kind of
function, we can alleviate the burden that users need to process tons of data
one by one. Besides, to assist the batch-selection mechanism, we need to
develop a set of interaction mechanisms accordingly.

  3. 3.

Facilitate users by highlighting ambiguous data. A suggestion mechanism for
ambiguous data (data with wrong predicted values and data in unreasonable
locations) can substantially improve user experience. Therefore, we provide
corresponding movement guidance functions for users.

###  3.2 System Description

Figure 3:  System. The system can be divided into three parts: a) function
bar, b) workspace and c) history record module. In the workspace, projected
coordinates are combined with the data image itself. When the user mouses over
a certain point, an enlarged image of the point will be displayed, and at the
same time, a light purple guide line and guide circle will also be displayed
to provide a general direction for the user to move. The workspace shows a
total of four classes, corresponding to four colors. The history record module
is used to store the points operated by the user.

Our system consists of three parts: function bar, workspace, and history
record (Fig. 3).

####  3.2.1 Function Bar

Data classes and representative images are displayed in the function bar. When
the user clicks on the corresponding representative picture, the data of this
class in the workspace can be controlled to be displayed or hidden, and the
user can hide irrelevant data through this function (Fig. 4).

There are two buttons below, the reset button is used to restore user
operations, and the update button is used to control the network for
retraining. The bottom display boxes show the accuracy before and after the
update.

Figure 4:  Display and hide. Click the image representing the class on the
left to hide or display the corresponding class in the workspace.

####  3.2.2 Interactive User Workspace

The system uses a novel spatial layout that helps users obtain a more
comprehensible representation of the latent space (Fig. 3(b)) . We apply
Isomap [50] to latent representations of the data, resulting in a 2D visual
layout, the reason for applying Isomap as a dimensionality reduction method is
explained in Section 3.4. In order to display the data more intuitively on the
workspace, the data itself is displayed in the form of a thumbnail, and the
predicted class of the data is displayed in the form of a colored coordinate
point in the lower right corner of the thumbnail, where different colors
represent different predicted values. Through such a spatial layout design,
not only the prediction of the data is clear at a glance, but also the data
itself can be better presented to the user, which is convenient for the user
to compare and interact with the data.

The distribution of the data reflects the classification results of the
network. We used heatmaps of different colors as backgrounds for different
classes, which play the role of guidance for user movement. In the workspace,
users can explore and interactively modify the data in it. At the same time,
this system also has basic redo and undo functions, which brings convenience
to user operations.

Then enter the description of the main functions of the system.

Enlarge function. How to facilitate users to compare data is an important
consideration when we design the system. When the mouse hovers over a point,
the image represented by the point is enlarged (Fig. 3(b)). When the user is
faced with the situation of distinguishing different data, this function can
help the user distinguish mixed data, which is convenient for users to compare
adjacent data and make better movement judgments. At the same time, the system
also has basic interactive zoom and wheel pan functions.

Visual volume adjustment. If the amount of data is too large, it will
inevitably affect the user’s observation, therefore the system adds the
function of visual volume adjustment. The user can control the data range
displayed on the current screen by dragging the slider bar below the workspace
(Fig. 5). The slider bar indicates the importance of the data from left to
right. We define importance as how confident the network is about the outputs.
Therefore, we use the softmax function to sort the maximum value of the
network output in descending order, and the sorted result is the importance we
define. We associate importance with the range of data displayed. In addition,
the user can also control the number of data displayed on the current screen
through the input window on the right side of the slider bar. After entering a
number, the workspace will only display the corresponding number of projected
points. Users can hide over-displayed data according to the above functions.

Figure 5: Visual volume adjustment. An example of using the importance slider
bar to adjust the amount of displayed data.

Interactive movement. The most basic interactive function in the system is to
move the projected coordinates in the user’s workspace. The user can move a
single point, or use the Lasso tool to move multiple points at the same time
(Fig. 6).

Figure 6: Interactive movement. An example of multi-point movement using the
Lasso tool.

Movement guidance. To greatly facilitate the user’s operation, the system
should have some guidance functions. Our system uses a pink box to highlight
data with incorrect predictions to facilitate the user to select the data to
be moved (Fig. 3(b)). In addition, the system adds a guideline and a guide
circle to remind the user of the approximate location of the cluster formed by
the ground truth of the current moving point. We also use heatmaps to show the
distribution of each class, which can also be used as a reference for user
movement.

####  3.2.3 History Record

During the interaction process, it is inevitable for users to make movement
errors and want to modify the movement process. At the same time, users may
also find interesting patterns and want to record their movements. Therefore,
we provide the history record module (Fig. 3(c)). The history of user movement
will be stored in the history record module. By clicking on the corresponding
history record, the workspace can be traced back to the corresponding state.
Users can also use the display mechanism of the history module to make further
comparisons of the data. In addition, this module provides redo and undo
functions.

###  3.3 Feedback Calculation

Figure 7:  An example of simulating user movement. The user first observes the
data using the system functions, and finds the moving points $\mathit{m}$.
Then, the user moves the moving points $\mathit{m}$ to the place he thinks is
appropriate according to his knowledge. The system will select reference
positive points $\mathit{p}$ and reference negative points $\mathit{n}$
according to the position before and after the user moves.

In order to incorporate human knowledge into the training process of neural
networks, we design a new loss function to allow the user’s operations in the
2D workspace will be learned by high-dimensional hidden vectors, and the
network can learn a latent space that is more in line with human knowledge.

We strive to make the image features of the user’s movement in the latent
space closer to the position after the user’s movement in the latent space,
for which we draw on the idea of triplet loss [45]. The purpose of triplet
loss is to make the features of the same label as close as possible in spatial
position, while the features of different labels are as far away as possible
in spatial position. Our purpose is to make the user’s moving features closer
to the moved spatial positions, therefore we need to find a high-dimensional
feature $\mathit{P}$ in the high-dimensional space as a reference. In the 2D
workspace, we set the point the user moved to point $\mathit{m}$. We set the
$\mathit{k}$ points with the same label closest to point $\mathit{m}$ after
the user moves to point $\mathit{p_{i}}$ (Fig. 7), where
0$<$$\mathit{i}$$<$$\mathit{k}$. To calculate the high-dimensional feature
$\mathit{P}$ in the high-dimensional space, we perform a weighted sum over the
points $\mathit{pi}$ (0$<$$\mathit{i}$$<$$\mathit{k}$), where the weight is
the inverse of the distance between point $\mathit{m}$ and point
$\mathit{p_{i}}$ in the high-dimensional space (see Formula (1)).

$P=\sum\nolimits_{i}^{k}\frac{1}{||m-p_{i}||_{2}^{2}}\times p_{i}$ (1)

where $\mathit{m}$ and $\mathit{p}$i represent features in the high-
dimensional space, corresponding to points found in two-dimensional space.

At the same time, in order to suppress the aggregation phenomenon of different
classes of features to a certain extent, we define a high-dimensional feature
$\mathit{N}$. We set the $\mathit{k}$ points with different labels closest to
point $\mathit{m}$ before moving to point $\mathit{n_{i}}$, where
0$<$$\mathit{i}$$<$$\mathit{k}$. In the same way, the corresponding point
$\mathit{N}$ is obtained according to point $\mathit{n_{i}}$
(0$<$$\mathit{i}$$<$$\mathit{k}$) (see Formula (2)). What the loss function
needs to do is to move the point $\mathit{m}$ in the high-dimensional space
closer to the high-dimensional feature $\mathit{P}$, and make the distance
between point $\mathit{m}$ and high-dimensional feature $\mathit{N}$ in the
high-dimensional space further apart, therefore the network can learn a latent
space that is closer to the user’s movement.

$N=\sum\nolimits_{i}^{k}\frac{1}{||m-n_{i}||_{2}^{2}}\times n_{i}$ (2)

where $\mathit{n}$i represent features in the high-dimensional space,
corresponding to points found in 2D space.

The classification loss $\mathit{loss}$cls is obtained by performing a Cross-
entropy (CE) calculation between the predicted labels and the ground-truth
labels. The distance difference loss $\mathit{loss}$dis is calculated based on
the points the user moves (see Formula (3)).

$loss_{dis}=\sum\nolimits_{i}^{D}max(||m_{i}-P_{i}||_{2}^{2}-||m_{i}-N_{i}||_{2}^{2}+\delta,0)$
(3)

where $\mathit{D}$ represents the number of user moving points, $\mathit{m}$i,
$\mathit{P}$i and $\mathit{N}$i represent corresponding features in high-
dimensional space, and $\mathit{\delta}$ is the margin used to control the
distance between points $\mathit{P}$i and $\mathit{N}$i.

The total loss function is weighted by the classification loss and the
distance difference loss before and after adjustment (see Formula (4)). Using
the total loss function, the network can be retrained based on where the user
moves, and the output will be closer to what the user changed, therefore the
user can assist the network in better classification.

$Loss=w_{cls}\times loss_{cls}+w_{dis}\times loss_{dis}$ (4)

###  3.4 Network and Dimensionality Reduction Method Selection

After comparisons, we choose the classic ResNet18[29] as the basic
classification network. We used the pre-trained network of ResNet18, and to
avoid overfitting, we froze the first few layers of the network. The fully
connected layer of the network consists of two layers, and the number of nodes
in each layer is 2048 and 512 respectively. We apply a dimensionality
reduction method on layer 512 to visualize the data on the user workspace. The
network was trained with a learning rate between 0.001 and 0.0005 and used the
Adam optimizer with a fixed batch size of 128.

We tested four commonly used dimensionality reduction methods on the garbage
classification dataset, namely PCA, MDS, t-SNE, and Isomap (Fig. 8). Isomap
keeps the distance relationship between samples after dimensionality reduction
unchanged, and find the real manifold of high-dimensional data so that the
results after dimensionality reduction can more accurately correspond to high-
dimensional feature vectors. Therefore, we choose Isomap as the dimensionality
reduction method used by the system.

Figure 8:  Example of dimensionality reduction results for PCA, MDS, t-SNE,
and Isomap.

##  4 EVALUATION

###  4.1 Performance Metrics

In order to accurately evaluate the effectiveness of our method, we choose the
visualization results as the subjective evaluation metrics, and at the same
time, we choose the performance metrics of the network as the objective
evaluation metrics.

Subjective evaluation metrics. The visualization results of retraining are an
important consideration in evaluating our method. The visualization results
can not only reflect the quality of the network classification results but
also reflect whether the user’s editing of the latent space is effective. If
the user-edited retrained projection results are close to the user-edited
results, we can assume that the network has learned the user-edited latent
space.

Objective evaluation metrics. We use micro-F1 and ROC commonly used in the
classification field as performance metrics to evaluate the performance of the
network. For the final retraining results, we apply ROC for evaluation, and at
the same time, we plot the micro-F1 of each epoch as a micro-F1 curve.

###  4.2 Datasets Selection

We evaluate the proposed system using three datasets of varying difficulty,
namely bronze dataset (self-collected), garbage classification dataset [1],
and head pose dataset [26] (Fig. 9). And we invited users of different
identities to conduct case studies on these datasets. Each of the three types
of datasets selected for the experiments has four classes. The bronze dataset
for experiments consists of 800 images, of which about 400 are train set,
about 150 are validation set, and about 250 are test set. Both the garbage
classification dataset and the head pose dataset for experiments have 700
images, of which about 400 are train set, about 100 are validation set, and
the remaining about 200 are test set. The experimental results will be used to
demonstrate whether our designed system is useful in different scenarios.

Figure 9:  Example images for the three selected datasets.

The bronze dataset represents datasets in the specialized field, and the
similarities between bronzes from adjacent ages are so great that only
experienced archaeologists can accurately classify bronzes. There are multiple
sub-classes under each class in the garbage classification dataset, which
poses a certain challenge to the classification effect of the network. The
head pose dataset has orientation properties that are difficult for networks
to discern, and networks are particularly poor at distinguishing facing left
from facing right. When it comes to discriminating images in the three
datasets above, humans have an advantage that machines cannot.

##  5 CASE STUDY

###  5.1 Approach

We first train on raw data to obtain a network for visualizing images. Then,
the high-dimensional features are projected into a 2D workspace through the
network, which facilitates user interaction with high-dimensional hidden
vectors. The user can help network learning by adjusting the position of
projected points in the 2D workspace. When the user operates the projection
points on the 2D workspace, the network will learn according to the changes
before and after the projection points, therefore achieving the purpose of
artificially assisted network retraining.

###  5.2 Task

In order to verify the function of the system and the effectiveness of the
method, we invited users of different identities to evaluate. Their task is to
move the data in the 2D workspace based on their knowledge, then retrain the
network and evaluate the updated results. The conditions for stopping the
update are at the user’s discretion. The process of conducting a case study is
as follows:

  1. 1.

Preparation stage. Users watch a video explaining the principles and functions
of our system, after which we further explain to users how the system works
and their tasks, and answer their questions;

  2. 2.

Practice stage. Let users use the system for 10 minutes to familiarize
themselves with system functions;

  3. 3.

Test stage. The user officially started the test. For the training sets of
different difficulty, we stipulated different operation times, including 45
minutes for the bronze dataset, 25 minutes for the garbage classification
dataset, and 20 minutes for the head pose dataset;

  4. 4.

Interview stage. Interview with the user about the functionality of the system
and the results of the case study.

###  5.3 Bronze Dataset

The difficulty in dating bronzes lies in whether the judges can grasp the
characteristics of each age. At the same time, they are also required to
understand the styles, crafts, and casting methods formed in different
regions. Often only well-trained experts can make accurate judgments, and
experts also need to compare the features between images many times when
making judgments. In this case study, we invited a researcher in the field of
archaeology, $\mathit{E}$A, to use our system, who had seven years of studying
archaeology experience and had participated in the task of collation and
classification of large datasets of bronzes.

####  5.3.1 User Operation

We prepared the dataset and pretrained network in advance for $\mathit{E}$A as
needed. $\mathit{E}$A first looked at the classes of the dataset and found
that there were bronzes from the early Western Zhou and late Shang dynasties
in the data. Based on his expertise, he speculates that the classification
accuracy of the data from these two dynasties may be lower. Then, he carefully
observed the distribution of data projections and found that there was no
clear boundary between the data projections corresponding to these two
dynasties, which confirmed his conjecture. At the same time, he found that the
data far from the center of the class cluster is often located in the
transition period of age. This part of the data is often mixed with the common
characteristics of the two ages. The classification results of the network on
this part of the data are not very good, therefore he focused on this part of
the data. Then, he takes the center position of each class cluster as the
reference frame, and the data far from the center of the cluster as the main
mobile data. His method is to classify the data at the cluster center
according to a certain characteristic of bronzes, such as shape, therefore he
divides the data at the center of each class cluster into several sub-classes.
After that, he moves the data away from the cluster center to where he sees
fit in the cluster center. He said in a later interview that he wanted to
subdivide the interior of each class based on important features such as the
shape and decoration of the bronzes. Through the interactive features designed
by the system, the researcher can easily move the data where he seems
reasonable.

Figure 10:  Study results on the bronze dataset. (a) Comparison of retrained
projection distributions without and with user editing. (b) Comparison of
micro-F1 curves obtained by retraining without and with user editing. (c)
Comparison of ROC curves obtained from the final results of retraining without
and with user editing.

####  5.3.2 Results

There is no clear boundary between the retrained projection distribution
without $\mathit{E}$A editing, but sharp boundaries emerged between the
retrained projection distribution with $\mathit{E}$A editing (Fig. 10(a)).
Interestingly, we also find that the retrained projection distribution with
$\mathit{E}$A editing from left to right corresponds to the order of the
bronze age. From the results of micro-F1 curves (Fig. 10(b)) and ROC curves
(Fig. 10(c)), it can be seen that the performance of the network has been
significantly improved after $\mathit{E}$A editing.

####  5.3.3 User Feedback

$``$The system can prompt me with similarities and differences between data
that I would otherwise ignore.$"$ $\mathit{E}$A believes that the system can
help him discover relationships between data, which is ideal for his use case:
partitioning data by constantly comparing the differences between them. He can
easily observe the data using the capabilities of the system, and he can
further compare similarities and differences between the data using the
history record module. He even comes up with new usages, such as using the
spatial layout structure we designed to divide the task of archaeological
data, which can greatly reduce his workload. In addition, he also commented,
$``$Both shape and decoration are very important to the classification of
bronzes, and it is difficult for us to take into account both when we manually
classify these data, but the use of 2D layout structure solves this problem to
a certain extent.$"$ Placing data with multiple characteristics between
clusters, rather than just one of them, is an advantage of a 2D spatial
layout.

However, $\mathit{E}$A also said that people in the field of archaeology often
use various software to process the data. From his point of view, as a person
who is not in the field of deep learning, it is still difficult for our system
to get started, and a certain adaptation stage is required. At the same time,
archaeological datasets are inherently difficult, so more straightforward
systems tend to be more applicable.

###  5.4 Garbage Classification Dataset

The difficulty of garbage classification datasets is that there are many
subclasses under each class. We invited deep learning researcher $\mathit{E}$B
to conduct this case study, he has 7 years of computer learning experience and
4 years of deep learning experience, and his main research direction is fine-
grained classification. At the same time, he is proficient in basic garbage
sorting rules.

Figure 11:  Study results on the garbage classification dataset. (a)
Comparison of retrained projection distributions without and with three edits
by the user. (b) Comparison of micro-F1 curves obtained by retraining without
and with three edits by the user. (c) Comparison of ROC curves obtained from
the final results of retraining without and with three edits by the user.

####  5.4.1 Process

$\mathit{E}$B initially used the system’s adjust display points and zoom in
comparison functions. In post-use interviews, $\mathit{E}$B said he had
experience designing user interfaces, therefore he paid particular attention
to whether the system could clearly display data and compare data easily.
$\mathit{E}$B believes that the functions designed by the system are very
comprehensive, and the logic between each component is smooth, which can meet
his needs.

$\mathit{E}$B compared the relationship between the data in combination with
our system and current classification accuracy. He found that classes with low
classification accuracy were projected less densely, while classes with high
classification accuracy were projected more tightly. He speculates that the
reason may be that the network’s feature learning of this part of the data is
not thorough enough. Therefore, he focuses on predicting data that is not
dense enough. He first moved a small amount of deviated data to the vicinity
of the corresponding cluster, and then set the training period to 8 epochs.
But the result after retraining did not change much. He guessed it might be
because he was moving too little data. From the perspective of the
relationship between the fully connected layer and the latent space, he
intends to move the data with wrong predicted values to the vicinity of the
data with correct predicted values, and at the same time perform intra-class
aggregation operations on the data of each class. In a subsequent interview,
he stated that the fully connected layer of the network can be seen as a
function, therefore whether the predicted value is correct depends on the
independent variable features. If the wrongly predicted features are closer to
the correctly predicted features, the output of the network is also more
likely to change from wrongly predicted to correctly predicted. Then,
$\mathit{E}$B made a second retraining according to the above idea, this time
the training period is 12 epochs. The accuracy of this training has been
significantly improved. At the same time, $\mathit{E}$B believes that there is
a certain gap between the classes in the visualization results after
retraining, so he believes that his idea is feasible. Then, he continued the
above idea, focusing the third time on the purple class with no clear
boundaries. He mainly separated the purple class from the others and then did
a third retraining. After the third retraining, the accuracy rate improved
significantly. At the same time, there are obvious gaps between the classes in
the visualization results. $\mathit{E}$B is very satisfied with the result of
this movement, and then he ends the operation.

####  5.4.2 Results

Through the retrained visualization results without editing and with three
editing by $\mathit{E}$B (Fig. 11(a)), we can clearly feel the process of
separating the originally mixed data. We can also feel the changes brought by
user interaction to the network from the micro-F1 curves (Fig. 11(b)) and ROC
curves (Fig. 11(c)). At the same time, the metric curves and visualization
results demonstrate the effectiveness of our method.

####  5.4.3 User Feedback

$\mathit{E}$B praised the utility of the system and commented, $``$The system
is so novel that it is intelligible to be able to connect the latent space
with data so directly.$"$ At the same time, $\mathit{E}$B also said that he is
more concerned about how the system presents data to the user. He believes
that the system we designed is fully functional, not only has basic zooming,
viewing, and guiding functions, but also can adjust the display density, which
can display data well, and the system is also very user-friendly. In addition,
$\mathit{E}$B also repeatedly observed the results of retraining. He believes
that the network is gradually learning the latent space after his movement,
and the retrained network can also show better performance.

$\mathit{E}$B also suggested to the system that a 2D spatial layout might not
be sufficient when dealing with complex distributions of data points. Although
we use Isomap, a dimensionality reduction method that can correspond to low-
dimensional and high-dimensional, it cannot fully reflect the relationship
between features. In future work, we will try to elevate the spatial layout to
three-dimensional space.

Figure 12:  Study results on the head pose dataset. (a) Comparison of
retrained projection distributions without and with editing by six users. (b)
Comparison of mean micro-F1 curves obtained by retraining without and with
editing by six users. (c) Comparison of ROC curves obtained from the final
results of retraining without and with editing by six users.

###  5.5 Head Pose Dataset

The head pose dataset includes four classes, facing up, facing down, facing
left, and facing right. We invited 6 graduate students (3 males, 3 females) to
join our system evaluation, their majors include bioinformatics, computer
vision, and recommender systems, and all of them are deep learning beginners.

####  5.5.1 Process

After simple exercises, they can quickly master the skills of using the system
and understand the meaning of the spatial layout of the system.

They generally check the classification of the projection data from the
beginning. Four users found that the projection results of the facing up class
have the highest degree of aggregation, and the facing up class has the
highest classification accuracy. They feel that the density of the projection
data will affect the accuracy. They believe that aggregating scattered data
can be an efficient way to move. Another user found that the projected data
for the facing left and facing right classes were almost mixed, with the
lowest classification accuracy for both. Combining his knowledge, he assumed
that neural networks are not very distinguishable between the concepts of the
facing left and facing right classes, therefore he focused on dealing with
facing left and facing right classes. The last user knows some visualization
techniques, so she pays more attention to the distribution of projections
before and after moving. She observes that the data for the four classes are
more or less mixed together. $``$If I separate the data from the four classes
that are mixed, will the network learn such a distribution?$"$ With this idea
in mind, she applies the functions of the system to process the projection
data.

####  5.5.2 Results

Compared to the without editing visualization result, the six user
visualization results produced clear demarcations between all four classes
(Fig. 12(a)). In particular, the three classes of purple, red, and orange that
were originally mixed can have a clear cluster structure after editing and
retraining by the users. At the same time, it can be seen from the results of
micro-F1 curves (Fig. 12(b)) and ROC curves (Fig. 12(c)) that the users’
editing has significantly improved the performance of the network.

####  5.5.3 User Feedback

All six users expressed a positive view of the retrained results, they all
felt that the updated network learned their $``$knowledge$"$ and that by
moving the data process and results, they were able to further understand the
concept of latent space. One of the users commented, $``$The method used by
this system meets my needs in an application, and it solves many invisible
problems encountered in network training, making the whole process more
intuitive and clear.$"$ Making the originally uncontrollable training process
controllable has always been the direction of our efforts. When it comes to
the design of the system interface, they generally believe that such a spatial
layout allows them to grasp the spatial location information of the samples,
thumbnail images, and the overall distribution of the data, which is concise
and intuitive.

##  6 Discussion

The system facilitates the observation of data and helps to discover
relationships between data. The system can intuitively observe the
similarities and differences between data, and the user can use the spatial
layout of the system to dig out the similarities between adjacent data that
may be overlooked, and can also dig out the differences between data that are
far away. In addition, the system can help users organize data. For example,
archaeologists need to process each image when classifying archaeological
data, which is a tedious and tedious process. But by borrowing our spatial
layout, they can divide the data according to the projection results, which
greatly reduces the workload.

The system facilitates the partitioning of data. The combination of spatial
layout and interactive functions is not only beneficial to classify data
between classes but also to classify data within the same class according to
attributes such as shape or pattern. At the same time, such a spatial layout
can also provide a buffer for the user. For example, due to the continuity of
the ages, bronze wares often show the characteristics of two ages. Simply
dividing this type of data into a specific age will be biased. Using the two-
dimensional layout of our system, the user can solve this problem by placing
this part of the data between two age clusters.

This system allows users to obtain a more understandable latent space. The
system connects a high-dimensional latent space with a 2D workspace, allowing
users to apply their knowledge to the training process of the network, which
can make the network jump out of the local minimum area and improve the
performance of the network. At the same time, the original concept of latent
space is relatively abstract, but users can intuitively feel the changing
process of latent space in the process of using the system we designed.

This system allows users to speed up the learning process of the network. By
visualizing the changes before and after retraining, the user can see that the
updated results are closer to the results they moved. Meanwhile, the training
process of the original network is unknown, but our system exposes this
process to some extent.

The system functions are well-designed, easy to use, and can be displayed
clearly with data.  In the latter two case studies, they both rated the system
as easy to use, an assessment that differed from the researcher $\mathit{E}$A.
We can think that for users with a certain deep learning foundation, the
system is relatively easy to use. In addition, all users believed that the
interactive function settings of the system were very reasonable, the designed
functions could meet their needs in use, and the combination of system layout
and functions could clearly display the data.

##  7 CONCLUSION

In this paper, we introduce an interactive system that not only connects a
high-dimensional latent space with a 2D workspace but also allows the user to
interact with the visualized data and retrain the network, enabling humans to
help the network learn. The results of three case studies prove that latent
vectors can be edited by the user, and through the system we designed, human
knowledge in classification can be incorporated into the training process of
the network, thereby improving the performance of the network.

## Acknowledgments

This work is supported in part by the Young Scientists Fund of the National
Natural Science Foundation of China (Grant No.62206106) and Jilin University
(Grant No.419021422B08).

## References

  * [1] Domestic waste classification image dataset, 2021.
  * [2] S. Amershi, M. Cakmak, W. B. Knox, and T. Kulesza.  Power to the people: The role of humans in interactive machine learning.  Ai Magazine, 35(4):105–120, 2014.
  * [3] S. Amershi, M. Chickering, S. M. Drucker, B. Lee, P. Simard, and J. Suh.  Modeltracker: Redesigning performance analysis tools for machine learning.  In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 337–346, 2015.
  * [4] S. Amershi, J. Fogarty, A. Kapoor, and D. Tan.  Effective end-user interaction with machine learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 25, pp. 1529–1532, 2011.
  * [5] G. Armstrong, C. Martino, G. Rahman, A. Gonzalez, Y. Vázquez-Baeza, G. Mishne, and R. Knight.  Uniform manifold approximation and projection (umap) reveals composite patterns and resolves visualization artifacts in microbiome data.  Msystems, 6(5):e00691–21, 2021.
  * [6] K. Asai, T. Fukusato, and T. Igarashi.  Integrated development environment with interactive scatter plot for examining statistical modeling.  In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–7, 2020.
  * [7] S. Ayesha, M. K. Hanif, and R. Talib.  Overview and comparative study of dimensionality reduction techniques for high dimensional data.  Information Fusion, 59:44–58, 2020.
  * [8] C. Baumgartner, C. Böhm, and D. Baumgartner.  Modelling of classification rules on metabolic patterns including machine learning and expert knowledge.  Journal of biomedical informatics, 38(2):89–98, 2005.
  * [9] N. Boukhelifa, A. Bezerianos, and E. Lutton.  Evaluation of interactive machine learning systems.  In Human and Machine Learning, pp. 341–360. Springer, 2018.
  * [10] E. T. Brown, J. Liu, C. E. Brodley, and R. Chang.  Dis-function: Learning distance functions interactively.  In 2012 IEEE conference on visual analytics science and technology (VAST), pp. 83–92. IEEE, 2012.
  * [11] S. S. Bukhari, T. M. Breuel, A. Asi, and J. El-Sana.  Layout analysis for arabic historical document images using machine learning.  In 2012 International Conference on Frontiers in Handwriting Recognition, pp. 639–644. IEEE, 2012.
  * [12] C.-M. Chang, C.-H. Lee, and T. Igarashi.  Spatial labeling: leveraging spatial layout for improving label quality in non-expert image annotation.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–12, 2021.
  * [13] N.-C. Chen, J. Suh, J. Verwey, G. Ramos, S. Drucker, and P. Simard.  Anchorviz: Facilitating classifier error discovery through interactive semantic data exploration.  In 23rd International Conference on Intelligent User Interfaces, pp. 269–280, 2018.
  * [14] J. Costa, A. Bock, C. Emmart, C. Hansen, A. Ynnerman, and C. Silva.  Interactive visualization of atmospheric effects for celestial bodies.  IEEE Transactions on Visualization and Computer Graphics, 27(2):785–795, 2020.
  * [15] M. A. Cox and T. F. Cox.  Multidimensional scaling.  In Handbook of data visualization, pp. 315–347. Springer, 2008\.
  * [16] C. de la Torre-Ortiz, M. M. Spapé, L. Kangassalo, and T. Ruotsalo.  Brain relevance feedback for interactive image generation.  In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, pp. 1060–1070, 2020.
  * [17] J. J. Dudley and P. O. Kristensson.  A review of user interface design for interactive machine learning.  ACM Transactions on Interactive Intelligent Systems (TiiS), 8(2):1–37, 2018.
  * [18] V. Eisenstadt, H. Arora, C. Ziegler, J. Bielski, C. Langenhan, K.-D. Althoff, and A. Dengel.  Exploring optimal ways to represent topological and spatial features of building designs in deep learning methods and applications for architecture.  2021\.
  * [19] R. Fiebrink and B. Caramiaux.  The machine learning algorithm as creative musical tool.  arXiv preprint arXiv:1611.00379, 2016.
  * [20] R. Fiebrink, P. R. Cook, and D. Trueman.  Human model evaluation in interactive supervised learning.  In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 147–156, 2011.
  * [21] J. Françoise and F. Bevilacqua.  Motion-sound mapping through interaction: An approach to user-centered design of auditory feedback using machine learning.  ACM Transactions on Interactive Intelligent Systems (TiiS), 8(2):1–30, 2018.
  * [22] S. Gehrmann, H. Strobelt, R. Krüger, H. Pfister, and A. M. Rush.  Visual interaction with deep learning models through collaborative semantic inference.  IEEE transactions on visualization and computer graphics, 26(1):884–894, 2019.
  * [23] X. Geng, D.-C. Zhan, and Z.-H. Zhou.  Supervised nonlinear dimensionality reduction for visualization and classification.  IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 35(6):1098–1107, 2005.
  * [24] M. Gillies.  Understanding the role of interactive machine learning in movement interaction design.  ACM Transactions on Computer-Human Interaction (TOCHI), 26(1):1–34, 2019.
  * [25] D. Gotz, J. Zhang, W. Wang, J. Shrestha, and D. Borland.  Visual analysis of high-dimensional event sequence data via dynamic hierarchical aggregation.  IEEE transactions on visualization and computer graphics, 26(1):440–450, 2019.
  * [26] N. Gourier, D. Hall, and J. L. Crowley.  Estimating face orientation from robust detection of salient facial structures.  In FG Net workshop on visual observation of deictic gestures, vol. 6, p. 7. Citeseer, 2004.
  * [27] R. Hadsell, S. Chopra, and Y. LeCun.  Dimensionality reduction by learning an invariant mapping.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), vol. 2, pp. 1735–1742. IEEE, 2006.
  * [28] A. Harel, D. J. Kravitz, and C. I. Baker.  Deconstructing visual scenes in cortex: gradients of object and spatial layout information.  Cerebral Cortex, 23(4):947–957, 2013.
  * [29] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [30] T. Honcharenko, G. Ryzhakova, Y. Borodavka, V. Savenko, and O. Polosenko.  Method for representing spatial information of topological relations based on a multidimensional data model.  ARPN Journal of Engineering and Applied Sciences, 16(7):802–809, 2021.
  * [31] G. Ivosev, L. Burton, and R. Bonner.  Dimensionality reduction and visualization in principal component analysis.  Analytical chemistry, 80(13):4933–4944, 2008.
  * [32] M. I. Jordan and T. M. Mitchell.  Machine learning: Trends, perspectives, and prospects.  Science, 349(6245):255–260, 2015.
  * [33] N. Kosmyna, F. Tarpin-Bernard, and B. Rivet.  Adding human learning in brain–computer interfaces (bcis) towards a practical control modality.  ACM Transactions on Computer-Human Interaction (TOCHI), 22(3):1–37, 2015.
  * [34] N. Kosmyna, F. Tarpin-Bernard, and B. Rivet.  Conceptual priming for in-game bci training.  ACM Transactions on Computer-Human Interaction (TOCHI), 22(5):1–25, 2015.
  * [35] F. Lekschas, X. Zhou, W. Chen, N. Gehlenborg, B. Bach, and H. Pfister.  A generic framework and library for exploration of small multiples through interactive piling.  IEEE Transactions on Visualization and Computer Graphics, 27(2):358–368, 2020.
  * [36] W. Li, D. Sadigh, S. S. Sastry, and S. A. Seshia.  Synthesis for human-in-the-loop control systems.  In International conference on tools and algorithms for the construction and analysis of systems, pp. 470–484. Springer, 2014.
  * [37] Z. Luo, J. Zhou, H. Zhu, D. Du, X. Han, and H. Fu.  Simpmodeling: Sketching implicit field to guide mesh modeling for 3d animalmorphic head design.  In The 34th Annual ACM Symposium on User Interface Software and Technology, pp. 854–863, 2021.
  * [38] C. Mai, T. Wiltzius, F. Alt, and H. Hußmann.  Feeling alone in public: investigating the influence of spatial layout on users’ vr experience.  In Proceedings of the 10th Nordic conference on human-computer interaction, pp. 286–298, 2018.
  * [39] M. Mărăşoiu, A. F. Blackwell, A. Sarkar, and M. Spott.  Clarifying hypotheses by sketching data.  In Proceedings of the Eurographics/IEEE VGTC Conference on Visualization: Short Papers, pp. 125–129, 2016.
  * [40] A. T. Nguyen, A. Kharosekar, S. Krishnan, S. Krishnan, E. Tate, B. C. Wallace, and M. Lease.  Believe it or not: Designing a human-ai partnership for mixed-initiative fact-checking.  In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, pp. 189–199, 2018.
  * [41] F. Radenović, G. Tolias, and O. Chum.  Fine-tuning cnn image retrieval with no human annotation.  IEEE transactions on pattern analysis and machine intelligence, 41(7):1655–1668, 2018.
  * [42] M. Rahbar, M. Mahdavinejad, A. H. Markazi, and M. Bemanian.  Architectural layout design through deep learning and agent-based modeling: A hybrid approach.  Journal of Building Engineering, 47:103822, 2022.
  * [43] G. Rosa, J. Papa, A. Marana, W. Scheirer, and D. Cox.  Fine-tuning convolutional neural networks using harmony search.  In Iberoamerican Congress on Pattern Recognition, pp. 683–690. Springer, 2015.
  * [44] Y. Sakata, Y. Baba, and H. Kashima.  Crownn: Human-in-the-loop network with crowd-generated inputs.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7555–7559. IEEE, 2019.
  * [45] F. Schroff, D. Kalenichenko, and J. Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815–823, 2015.
  * [46] F. M. Shipman III, C. C. Marshall, and T. P. Moran.  Finding and using implicit structure in human-organized spatial layouts of information.  In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 346–353, 1995.
  * [47] T. Spinner, U. Schlegel, H. Schäfer, and M. El-Assady.  explainer: A visual analytics framework for interactive and explainable machine learning.  IEEE transactions on visualization and computer graphics, 26(1):1064–1074, 2019.
  * [48] T. Spinner, U. Schlegel, H. Schäfer, and M. El-Assady.  explainer: A visual analytics framework for interactive and explainable machine learning.  IEEE transactions on visualization and computer graphics, 26(1):1064–1074, 2019.
  * [49] J. Talbot, B. Lee, A. Kapoor, and D. S. Tan.  Ensemblematrix: interactive visualization to support machine learning with multiple classifiers.  In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 1283–1292, 2009.
  * [50] J. B. Tenenbaum, V. d. Silva, and J. C. Langford.  A global geometric framework for nonlinear dimensionality reduction.  science, 290(5500):2319–2323, 2000.
  * [51] M. Torres and F. Cantú.  Learning to see: Convolutional neural networks for the analysis of social science data.  Political Analysis, 30(1):113–131, 2022.
  * [52] L. Van der Maaten and G. Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [53] A. Verbraeck and E. Eisemann.  Interactive black-hole visualization.  IEEE Transactions on Visualization and Computer Graphics, 27(2):796–805, 2020.
  * [54] D. Wang and J. Gu.  Vasc: dimension reduction and visualization of single-cell rna-seq data by deep variational autoencoder.  Genomics, proteomics & bioinformatics, 16(5):320–331, 2018.
  * [55] F. Wang, Y. Li, D. Sakamoto, and T. Igarashi.  Hierarchical route maps for efficient navigation.  In Proceedings of the 19th international conference on Intelligent User Interfaces, pp. 169–178, 2014.
  * [56] G. Wang, W. Li, M. A. Zuluaga, R. Pratt, P. A. Patel, M. Aertsen, T. Doel, A. L. David, J. Deprest, S. Ourselin, et al.  Interactive medical image segmentation using deep learning with image-specific fine tuning.  IEEE transactions on medical imaging, 37(7):1562–1573, 2018.
  * [57] N. Watanabe, M. Washida, and T. Igarashi.  Bubble clusters: an interface for manipulating spatial aggregation of graphical objects.  In Proceedings of the 20th annual ACM symposium on User interface software and technology, pp. 173–182, 2007.
  * [58] J. Wei, H. Xie, C.-M. Chang, and X. Yang.  Fine-tuning deep neural networks by interactively refining the 2d latent space of ambiguous images.
  * [59] P. I. Wójcik and M. Kurdziel.  Training neural networks on high-dimensional data using random projection.  Pattern Analysis and Applications, 22(3):1221–1231, 2019.
  * [60] J. Wu, X. Zhang, J. Nichols, and J. P. Bigham.  Screen parsing: Towards reverse engineering of ui models from screenshots.  In The 34th Annual ACM Symposium on User Interface Software and Technology, pp. 470–483, 2021.
  * [61] R. Xu, F. Luo, Z. Zhang, C. Tan, B. Chang, S. Huang, and F. Huang.  Raise a child in large language model: Towards effective and generalizable fine-tuning.  arXiv preprint arXiv:2109.05687, 2021.
  * [62] W. Yang, X. Wang, J. Lu, W. Dou, and S. Liu.  Interactive steering of hierarchical clustering.  IEEE Transactions on Visualization and Computer Graphics, 27(10):3953–3967, 2020.
  * [63] X. Yin, W. Chen, X. Wu, and H. Yue.  Fine-tuning and visualization of convolutional neural networks.  In 2017 12th IEEE Conference on Industrial Electronics and Applications (ICIEA), pp. 1310–1315. IEEE, 2017.
  * [64] X. Zhang and P. J. Guo.  Mallard: Turn the web into a contextualized prototyping environment for machine learning.  In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, pp. 605–618, 2019.
  * [65] J. Zhao, M. Fan, and M. Feng.  Chartseer: Interactive steering exploratory visual analysis with machine intelligence.  IEEE Transactions on Visualization and Computer Graphics, 2020.